Executive Summary

A well-prepared, effective primary care workforce reduces healthcare costs and can play a significant role in disease prevention and management of chronic illnesses. Title VII, section 747 programs have transformed the medical and dental education landscape with their increased emphasis on primary care education and training. The programs have helped to create a diverse, broadly competent primary care medicine and dentistry workforce. Title VII, section 747 programs have also improved the Nation’s health by training a quality primary care healthcare workforce, particularly for medically and dentally underserved populations.

Over the past 40 years, Title VII, section 747 has contributed to the development and expansion of education and training programs for primary care providers, promoted diversity in the workforce, and helped to ensure that health professions curricula respond to the changing demands and emerging needs of the U.S. population. Such demands and needs include addressing the disparities in health outcomes of vulnerable groups. The initial legislative purpose of the Program, created in 1963, was to increase the general supply of physicians. In successive reauthorizations, the focus of the Program shifted to production, education, and training of primary care providers, later to addressing geographic distribution problems of healthcare providers and, more recently, to education and training of primary care providers to serve medically and dentally underserved populations and high-risk groups.

Title VII, section 747 programs form the centerpiece of the Health Resources and Services Administration’s (HRSA’s) efforts to prepare the primary care health workforce. HRSA programs tend to work synergistically and complement one another. For example, Title VII, section 747 programs educate and train high-quality primary care providers, many of whom go on to join the National Health Service Corps (NHSC) and/or work in federally funded community, migrant, and rural health centers. In this way, Title VII, section 747 programs support the overall workforce goals of HRSA’s Bureau of Health Professions (BHPr).

The evaluation of Title VII, section 747 programs has been challenging for two major reasons. First, the ability to define meaningful longitudinal outcome measures for the programs is complicated by the changing nature of the legislation. Second, the myriad related programs funded by Title VII, section 747 and Title VIII (which supports nursing grant programs) under HRSA has created some blurring of programs goals, making it difficult to ascertain the discrete contributions of individual programs to specific outcomes.

Evaluations that have examined specific educational and clinical outcomes of Title VII, section 747 programs have found beneficial impact. Some studies have found that Title VII, section 747 encouraged and sustained development of primary care training programs. Others have shown that Program support leads to the establishment and maintenance of family medicine departments and associated production of family physicians. Others have developed analyses demonstrating that increasing the funding rate would reduce time required to eliminate Health Professional Shortage Areas (HPSAs). Some studies conclude that there are increased retention rates of family medicine faculty in funded institutions. Still other studies have demonstrated that increased service by alumni of Title VII programs was provided to those in need. However, the assessment of these programs done by the Office of Management and Budget (OMB) was more critical in substance and tone. In aggregating 40 separate health professions programs, including the lumping together of Title VII, section 747 and Title VIII programs in its assessment process, OMB was not able to isolate the specific effectiveness of Title VII, section 747 programs. Although Title VII, section 747 programs work synergistically with other programs in HRSA’s portfolio, they are specifically designed to bring about significantly different outcomes from those of other programs. Collectively, the programs contribute to HRSA’s overall goals. The OMB assessment failed to evaluate Title VII, section 747 programs based on the outcomes they were designed to bring about.

The Committee believes that OMB’s view, that health professions training programs are diffuse and subject to interpretation, points to the need to clarify the purpose and objectives of Title VII, section 747 programs. The purpose of this report is not only to clarify the purpose and objectives of the Program, but also to put forward recommendations regarding outcome measures for evaluating the impact of Title VII, section 747, based on the Program’s purpose and objectives. Based on the legislation, this purpose includes provision of funding for approved training of students, interns, and residents in family medicine, general internal medicine, and general pediatrics; training of physician assistants; training of residents in general dentistry and pediatric dentistry; and training of individuals who plan to teach in family medicine, internal medicine, pediatrics, and physician assistant training programs.

Based on a review of authorizing legislation and broader goals of BHPr and an examination of the past successes and capabilities of the Program, it is the Committee’s view that the primary purpose of the Title VII, section 747 Program is:

To educate and train physicians, pediatric and general dentists, and physician assistants to enhance the quality, capacity, and diversity of the Nation’s primary care workforce, giving special consideration to the healthcare needs of underserved populations and other high-risk groups.

Further, the Committee believes that there are seven key objectives for these programs.

Key Objectives

1. Improve the quality of education and training of the Nation’s primary care workforce.
2. Improve the capacity for education and training of the Nation’s primary care workforce, with special emphasis on individuals from disadvantaged backgrounds and underrepresented minorities.
3. Improve primary care education and training -curricula.
4. Improve primary care faculty development.
5. Identify, develop, and disseminate primary care education and training innovations and best practices among programs, accrediting bodies, and other constituents.
6. Improve the preparation of faculty, residents, and students (or learners) to work with medically and dentally underserved populations and build linkages to communities.
7. Improve the diversity and number of primary care faculty and students (or learners), with special emphasis on individuals from disadvantaged backgrounds and underrepresented minorities.

In this report, the term “disadvantaged” is used to refer to those individuals who are either economically or educationally disadvantaged. The term is broadly inclusive of racial minorities, ethnic minorities, and poor whites (Anglos). Individuals from disadvantaged backgrounds do not fit geographic boundaries. They can exist in rural and frontier communities as well as in urban and suburban communities.

The Committee’s Approach to Identifying Outcome Measures

To identify scientifically sound and programmatically relevant outcome measures for the seven key objectives above, the Committee conducted an assessment that included:

• Review of the literature on evaluation of education and training programs
• Development of a Title VII, section 747 Program logic model
• Review of BHPr’s goals and Title VII, section 747’s role, as part of a portfolio of programs, in meeting those goals

In the first step, the Committee conducted a review of the literature to identify and assess research on the evaluation of education and training programs with respect to education outcomes, workforce outcomes, and healthcare outcomes. The objectives of this step were to identify methodologically sound approaches for the evaluation of education and training programs and to identify candidate outcome measures for Title VII, section 747 objectives.

In the second step, the Committee developed a logic model for the Title VII, section 747 Program. A program logic model is a tool that provides a simplified visual representation of how a program or project is expected to work to achieve intended results (Schiller, 2004). A program logic model links outcomes (both near- and long-term) with program activities and processes as well as the theoretical assumptions and principles of the program (W.K. Kellogg Foundation, 2004). The logic model facilitated the identification of near- and long-term measures associated with the seven Program objectives. The chart on the next page (Figure 1) sets out the Committee’s model for how Title VII, section 747 programs achieve their outcomes. Each near- and longer-term outcome has been referenced to one of the seven key objectives for the programs.

In the third step, the Committee reviewed BHPr’s goals and Title VII, section 747’s role in helping to achieve those goals. BHPr is in the process of refining outcome measures to evaluate progress on the Bureau’s goals. To help ensure alignment between BHPr’s goals and recommended measures for Title VII, section 747 outcomes, the Committee took into consideration BHPr’s measures in identifying Title VII, section 747 measures.

In considering the findings of this assessment, the Committee identified and prioritized a set of 25 recommended outcome measures. The measures are categorized into two groups: near-term and longer-term. These measures are set out in the chart on the following page (Figure 2).

Conclusions

Evaluations of Title VII, section 747 programs in the past have not always been based on the same program outcomes and measures. In some cases, the use of different outcomes and measures has led to conflicting findings regarding the impact of these programs. Adoption of a consistent set of outcomes and measures, based on Program objectives, is important in order to help ensure that programs deliver the intended benefits to stakeholders and constituents. The Committee believes that adoption of the outcomes and measures recommended herein will facilitate more meaningful future assessment of the Program’s impact and foster continuous program improvement.

Recommendations

Consistent with the stated purpose:

To educate and train physicians, pediatric and general dentists, and physician assistants to enhance the quality, capacity, and diversity of the Nation’s primary care workforce, giving special consideration to the healthcare needs of underserved populations and other high-risk groups.

The Advisory Committee on Training in Primary Care Medicine and Dentistry (ACTPCMD) recommends that:

1. The outcome measures utilized to evaluate Title VII, section 747 programs should reflect the Program’s statutory focus on health professional education and training. Evaluation of this Program should focus on preparation of the primary care workforce. External evaluations of the Program should be designed to specifically capture educationally oriented outcomes. Meaningful evaluation should include quantitative and qualitative methods that address quality, capacity, and diversity. Outcome measures should encompass physician assistant training, pre-doctoral training, academic administrative units, primary care residency training, primary care faculty development, and general and pediatric dental residency training.
2. The proposed logic model and near- and longer-term outcome measures should guide the evaluation of Title VII, section 747 programs. Title VII, section 747 programs are diverse and have a variety of both near- and longer-term benefits for learners (students, residents, and faculty) and the populations served. Analysis, interpretation, and presentation of data should address various outcome measures as outlined in the report. The proposed logic model demonstrates the complex, multi-factorial set of interrelationships and influences on Program outcomes (see section entitled Description and Method for Reading the Logic Model, page 13) and provides a framework to identify and select programmatically relevant outcome measures.
3. Outcome measures should be identified that evaluate the synergistic role of Title VII, section 747 programs with other Bureau of Health Professions (BHPr) programs, especially the National Health Serv-ice Corps (NHSC), as well as the Health Careers Opportunity Program (HCOP), Area Health Education Centers (AHECs), and Rural Interdisciplinary Training. Evaluation of these programs should primarily focus on the preparation, education, and training of the primary care workforce. Evaluation should include the synergism between Title VII, section 747 and other BHPr programs such as HCOP, AHECs, and Rural Interdisciplinary Training. Although placement of program graduates in underserved settings is not a principal focus of Title VII, section 747 programs, it strongly impacts the preparation of healthcare professionals to assume roles for other BHPr programs. Evaluation could include how these programs have properly trained professionals to serve in underserved settings and in programs like the NHSC and community, migrant, and rural health centers, as well as to care for underserved populations and high-risk groups.
4. BHPr should develop procedures for data collection, analysis, and reporting of Program outcome measures. BHPr should oversee the process of collecting, analyzing, and reporting qualitative and quantitative data associated with outcomes. These outcome measures should be developed and implemented consistent with the logic model, objectives, and near- and longer-term outcomes presented in this report. The process developed must provide meaningful reporting on Program effectiveness without placing undue burden on grantees for collecting data.
5. Additional financial resources should be made available to BHPr to implement -effective evaluation processes for the programs in Title VII, section 747. The systematic evaluation of outcomes is critical to ensuring the effectiveness of Title VII, section 747 programs. However, establishing and sustaining effective evaluation methods is labor-intensive and expensive. Therefore, additional funds are necessary to develop an ongoing process for data collection, analysis, and reporting of program outcome measures. It should be noted that any plan to reassign funds from current Program allocations would strongly compromise this process and jeopardize the intended outcomes of these programs.

Figure text for Executive Summary-----

Figure 1. Title VII, Section 747 Logic Model

Inputs
Activities
Outputs
Near-Term Outcomes
Longer-Term Outcomes

Title VII, section 747 legislative authority
Title VII, section 747 funding
DHHS and BHPr infrastructure

Education and training grant management
Operation of contracts and co-op and collaborative agreements

Education and training grants, faculty development, and curricular innovations
Cost-effective management of funding mechanisms
Timely approvals, disbursements, and receipt of grantee submissions

O1: Improved primary care education and training curricula
O6: Increased ability of learner and faculty to serve high-risk, special needs, and vulnerable populations
O2: Improved primary care training capacity
O7: Increased diversity of primary care faculty, residents, and students
O3: Increased training in underserved communities

O1: Improved primary care workforce training and quality
O2: Improved capacity of primary care training infrastructure
O7: Improved diversity of graduates (including underrepresented minorities)
O3: Increased primary care providers serving high-risk and underserved populations
O1: Goals met from Healthy People 2010

ACTPCMD
COGME
Public and private partnerships

Partnership development
Primary care research and policy development
Partnerships
Workforce studies and training research
Annual reports – research results and recommendations
Feedback and consensus
Performance indicators/baseline performance data

O4: Published primary care training and primary care research in priority areas
O5: Recommendations on policy or National guidelines
O5: Dissemination of innovations and best practices
O5: Adoption of innovations and best practices by others
--------------

Figure 2. Recommended Outcome Measures

Key Objectives
Near-Term Measures
Longer-Term Measures

1. Improve the quality of education and training of the Nation’s primary care workforce.

• Evidence of competency of learners and faculty, as demonstrated by improvement in knowledge, skills, attitudes, etc.
• Goals met from Healthy People 2010

2. Improve the capacity for education and training of the Nation’s primary care workforce (with special emphasis on individuals from disadvantaged backgrounds and underrepresented minorities).

• Learners and faculty trained
• Retention in primary care
• Primary care training programs created, expanded, or affected as a result of Title VII, section 747 funding

3. Improve primary care education and training curricula.

• Grantees implementing curricula addressing critical healthcare needs as defined in the Title VII, section 747 Guidance
• Hours devoted to curriculum (can include training for culturally effective care)
• Grantees with rural and/or underserved populations training tracks, clinical rotations, etc.
• Programs that address emerging health care needs
• Graduates whose practice focuses on a specific underserved population or on a specific primary care problem

4. Improve primary care faculty development.

• Leadership roles and scholarly output of primary care faculty, as demonstrated by promotion and tenure, presentations and publications, research grants, advocacy, and public and professional service
• Primary care faculty in medical or dental educational institutions over a timeframe
• Primary care trained graduates in faculty positions

5. Identify, develop, and disseminate primary care education and training innovations and best practices among programs, accrediting bodies, and other constituents.

• Primary care education and training publications and research publications, including web-based publications
• Innovations, including use of new technology and best practices developed and adopted by accrediting bodies and others

6. Improve the preparation of faculty, residents and students (or learners) to work with medically and dentally underserved populations and build linkages to communities.

• Ambulatory and community-based training sites that serve primarily underserved populations
• Learners who are from disadvantaged backgrounds, who are from rural backgrounds, or who are underrepresented minorities or women
• Disadvantaged, high-risk, and special needs individuals served
• Graduates caring for underserved, uninsured, or special needs populations
• Where graduates practice

7. Improve the diversity and number of primary care faculty and students (or learners), with special emphasis on individuals from disadvantaged backgrounds and underrrepresented minorities.

• Learners who are from disadvantaged backgrounds, who are from rural backgrounds, or who are underrepresented minorities or women
• Learners among funded programs who indicate at matriculation and graduation that they intend to work in primary care
• Underrepresented minority faculty who have completed Title VII, section 747 faculty development programs, and who teach and/or serve underserved populations
• Underrepresented minority faculty involved in leadership or research positions
• Faculty, graduates, and practitioners trained in funded programs who are from disadvantaged backgrounds or who are underrepresented minorities or women