History of Evaluation of Title VII, Section 747 Programs

Summary of Previous Studies Assessing the Title VII, Section 747 Programs

Studies have demonstrated that Title VII, section 747 has contributed to the development of high-quality primary care education and training programs, established and maintained family medicine departments and, through the training of a primary care workforce, helped decrease the number of HPSAs. Specific studies and their findings are cited below.

Trends in the proportion of U.S. medical school graduates entering primary care in relationship to Title VII funding were explored by Rosenblatt, Whitcomb, and colleagues (1993). They found that Title VII, section 747 encouraged and sustained the development of primary care training programs but was not sufficient in itself to increase substantially the proportion of primary care physicians, given a climate favoring specialty care. Petersdorf (1993) suggested that this finding reflected the small amount of funding involved, and asserted that the Program is beneficial to the extent that it helps initiate and maintain family medicine departments.

In 1994, two General Accounting Office (GAO) reports offered perspectives on the Program. In the July 1994 report, the GAO acknowledged that although there were many challenges to evaluation, the Program was important for funding innovative projects and providing seed money for starting new programs, and that the Program was considered important in the creation and maintenance of family medicine departments (GAO, 1994). In a second report in October 1994, the GAO stated that “students who attended schools with family practice departments were 57% more likely to pursue primary care.” The same report goes on to state that “students attending medical schools with more highly funded family practice departments were 18% more likely to pursue primary care and students attending schools requiring a third-year family practice clerkship were [also] 18% more likely to pursue primary care.”

Baumgartner, Stenersen, and colleagues (1996) reviewed evaluations and studies of 30 Title VII and VIII programs, and noted problems hampering their ability to determine direct effects on provider supply, distribution, and minority representation. These programs included diverse program objectives; lack of common goals, data, and reporting requirements; and an inability to distinguish program effects from other funding sources or external influences.

Politzer, Horab, and colleagues (1997) found that establishment and maintenance of family medicine departments in private schools and their production of family physicians were positively associated with sustained receipt of Title VII, section 747 support. The same authors found less of a correlation between production of generalist physicians and Title VII, section 747 support at public schools. Their research concluded that it is unclear whether continuous receipt of Title VII, section 747 funds results in perpetuation of family medicine departments or the presence of such a department enhances the probability of receiving these funds. In 1999, Politzer, Hardwick, and colleagues simulated the impact that Title VII, section 747 support for generalist production had on reducing and eliminating HPSAs. Each scenario investigated showed that increasing the number of residency programs funded or the rate at which program graduates practice in underserved areas can decrease the time needed to eliminate HPSAs.

A survey examining the retention of recently graduated Title VII, section 747-funded family medicine faculty development fellows in academic medicine (Kohrs, Mainous, 1999) showed a 75 percent retention rate, though long-term retention was not measured. A follow-up survey measured the prevalence of service in underserved areas by Title VII, section 747-funded faculty development fellowship alumni, and demonstrated a high service rate in areas of need among this group (Kohrs, Mainous, et al., 2001).

Fryer, Meyers, and colleagues (2002) evaluated the practice specialty and location of 180,000 medical school graduates to analyze the effects of Title VII, section 747 funding between 1978 and 1993 on the intended outcomes to increase numbers of primary care physicians and to increase practice in rural and underserved areas. Title VII, section 747 funding was associated with higher rates of entry into family practice and practice in HPSAs, and pre-doctoral training and departmental development funding were strongly related to achievement of Title VII, section 747 objectives.

In 2002, a study by the Robert Graham Center for Policy Studies reported that medical schools that received Title VII, section 747 family medicine funds produced more medical students who practiced in family medicine or primary care, in a rural area, or in a HPSA (Fryer, Meyers, et al., 2002).

Edelstein, Krol, and colleagues (2003), found that Title VII, section 747 funding of pediatric dentistry training programs was successful in treating the underserved, in shaping careers dedicated to serving the underserved, and in recruiting underrepresented minority dentists.

A report by the Advisory Committee on Training in Primary Care Medicine and Dentistry (ACTPCMD, 2003) examined the role of Title VII, section 747 programs in educating and training healthcare professionals in cultural competency. The Committee concluded that Title VII, section 747 programs had a clear record of success and should prioritize training programs that address health disparities. In addition, the Committee noted that Title VII, section 747 grant programs have supported each of the ten Healthy People 2010 Leading Health Indicators. The Committee also noted that programs funded under Title VII, section 747 have significantly increased the number of underrepresented minority enrollees, graduates, and faculty in healthcare education (ACTPCMD, 2004).

The Undergraduate Medical Education for the 21st Century (UME-21) project was funded by BHPr’s Division of Medicine and Dentistry to address training challenges through the development and dissemination of innovations in medical education. In one evaluation of the UME-21 initiative, Veloski and Barzansky (2004) found that there was an increase in seniors’ ratings of instruction in the newer areas of evidence-based medicine, quality assurance, and cost-effectiveness in relation to National norms between 1999 and 2001. They concluded that even modest funding focused on specific goals can produce measurable results.

Proposals Within HRSA for Evaluating Program Success

BHPr is undergoing a comprehensive review of program-level logic models and performance measures. In addition to program-specific measures, which are designed to capture unique accomplishments of each BHPr program, BHPr is considering a set of five core measure areas, which are designed to capture accomplishments across five common purposes. These broader measures that use aggregated program data include:

• Diversity. Increase health workforce diversity.
• Primary care. Promote careers in primary care.
• Distribution. Improve the distribution of the healthcare workforce.
• Quality. Improve the quality of care.
• Infrastructure. Strengthen public health and healthcare infrastructure.

For each of these goals, the Bureau has proposed a set of performance measures and associated definitions, data sources, list of reporting programs, and a data analysis plan. The result of this work will be a comprehensive performance measurement framework that will improve assessment of each program’s contribution to BHPr goals. In developing this report, the Committee has worked with Bureau staff to ensure recommendations are consistent with program evaluation and performance measurement efforts within BHPr.

Historical and Recent Evaluations of Title VII, Section 747 Programs by OMB

Despite recent findings of beneficial impacts on certain education and clinical outcomes, the Office of Management and Budget’s (OMB’s) assessment of HRSA’s health professions training programs (including Title VII, section 747) for fiscal year 2003 resulted in an “ineffective” rating. OMB uses the Program Assessment Rating Tool (PART) to assess Government program performance and inform funding decisions. PART assessments examine program purpose and design; performance measurement, evaluations, and strategic planning; program management; and program results, with the goal of improving programs by identifying their strengths and weaknesses. The PART assessment of HRSA’s health professions training programs (including Title VII, section 747) for fiscal year 2003 resulted in an “ineffective” rating.

The assessment aggregated approximately 40 individual health professions programs, and most comments were about the programs as a group. Although the programs received positive assessments on a majority of the PART measures (receiving 60 points out of 100 for “purpose,” 71 for “planning,” and 73 for “management”), the rating for the programs’ “results/accountability” was only 13 out of a possible 100 points. Specific criticisms were leveled at the entire set of health professions programs as a group: their purposes are too diffuse and subject to varying interpretations by interested parties; the programs are under various authorities and not designed to have a significant impact on any one factor such as diversity, distribution, supply, or quality of health professionals; and the health professions programs were not aligned with program goals in a way that the impact of funding, policy, and legislative changes on performance could be readily known. In addition, OMB criticized HRSA for not using performance data to make decisions about the programs and for not supporting sufficient numbers of objective, independent evaluations of the health professions programs. Because it was able to cite only a few studies in this area, OMB answered “no” to the assessment question “do independent and quality evaluations of this program indicate that the program is effective and achieving results?”

As part of its criticism, OMB refers to a study that concludes that graduates of funded programs are only slightly more likely to serve in HPSAs. This is not a valid criticism of Title VII, section 747 programs because the purpose of the programs, as set forth in the legislation, is not specifically to place graduates in HPSAs. Instead, the legislation focuses on educating and training providers to enter primary care, including family medicine, and giving special consideration in the training to the needs of underserved populations. Although not mentioned in the OMB assessment, the same study concludes that “graduates of schools with Title VII, section 747 grants were significantly more likely to become family physicians and practice primary care than were graduates of schools without funding” (Fryer, Meyers, et al., 2002). According to that same study, 15.8 percent of graduates of funded programs entered family practice versus 10.2 percent of those from non-funded programs. In addition, 36.3 percent of graduates went on to practice primary care versus 30.9 percent of graduates of non-funded programs.

In aggregating 40 separate health professions programs in its assessment process, including lumping Title VII and Title VIII (which supports nursing grant programs) programs together, OMB was not able to isolate the specific effectiveness of the Title VII, section 747 programs, which are designed to bring about significantly different outcomes than other programs in HRSA’s health professions training program portfolio. Although Title VII, section 747 programs work synergistically with other HRSA programs, Title VII, section 747 programs are specifically designed to bring about significantly different outcomes than other programs. Collectively, the programs contribute to HRSA’s overall goals. The OMB assessment failed to evaluate the Title VII, section 747 programs based on the outcomes they were designed to bring about.

These findings point to the need for a clearer paradigm for evaluation of these programs as well as a systematic approach to applying relevant, reliable, and valid measures to the evaluation of Title VII, section 747 programs. Such an effort requires that measures be developed that are scientifically sound and programmatically relevant.

Overall Challenges in Evaluating Programs

The importance of performance measurement and program evaluation has long been recognized. In general, there are challenges to effective evaluation that have inhibited effective study implementation. These challenges include:

• Evaluation of burden of cost, effort and time, and resulting diversion from other goals. Activities associated with evaluation can place a significant burden on constituents. This is especially true when evaluation requires original data collection and data analysis. Grantees tend to have small budgets for evaluation. In addition, Title VII, section 747 is but a small portion of their overall budget, so resources required for evaluation can amount to a significant portion of program funding. Techniques for overcoming this challenge include developing data collection procedures that minimize the data collection burden, for example, through the use of automation, statistical sampling, and data mining of existing sources.
• Difficulty in tracking learners. Once learners have left grantee institutions, it is difficult to keep track of where they are practicing and the kinds of patients they are serving.
• Identification of appropriate measures and evaluation methodologies. The choice of outcome measures used must be scientifically sound and programmatically relevant. If inappropriate measures are chosen, performance measurement systems can encourage behaviors that are contrary to program goals. Measures that are insufficiently balanced can result in excessive focus on one measure at the expense of more important objectives.