Conclusions

Historically, evaluations of Title VII, section 747 programs have not always been based on consistent Program outcomes and measures. In some cases, the use of different outcomes and measures leads to conflicting findings regarding the impact of these programs. Adoption of a consistent set of outcomes and measures, based on Program objectives, is a critical step in ensuring that programs deliver intended benefits to stakeholders and constituents. The Committee believes that adoption of the outcomes and measures recommended herein not only will facilitate more meaningful evaluation of Program impact, but also will foster continuous program improvement.

Recommendations

Consistent with the stated purpose:

To educate and train physicians, pediatric and general dentists, and physician assistants to enhance the quality, capacity, and diversity of the Nation’s primary care workforce, giving special consideration to the healthcare needs of underserved populations and other high-risk groups.

The Advisory Committee on Training in Primary Care Medicine and Dentistry (ACTPCMD) recommends that:

1. The outcome measures utilized to evaluate Title VII, section 747 programs should reflect the Program’s statutory focus on health professional education and training. Evaluation of this Program should focus on preparation of the primary care workforce. External evaluations of the Program should be designed to specifically capture educationally oriented outcomes. Meaningful evaluation should include quantitative and qualitative methods that address quality, capacity, and diversity. Outcome measures should encompass physician assistant training, pre-doctoral training, academic administrative units, primary care residency training, primary care faculty development, and general and pediatric dental residency training.
2. The proposed logic model and near- and longer-term outcome measures should guide the evaluation of Title VII, section 747 programs. Title VII, section 747 programs are diverse and have a variety of both near- and longer-term benefits for learners (students, residents, and faculty) and the populations served. Analysis, interpretation, and presentation of data should address various outcome measures as outlined in the report. The proposed logic model demonstrates the complex, multi-factorial set of interrelationships and influences on Program outcomes (see section entitled Description and Method for Reading the Logic Model, page 13) and provides a framework to identify and select programmatically relevant outcome measures.
3. Outcome measures should be identified that evaluate the synergistic role of Title VII, section 747 programs with other Bureau of Health Professions (BHPr) programs, especially the National Health Service Corps (NHSC), as well as Health Careers Opportunity Program (HCOP), Area Health Education Centers (AHECs), and Rural Interdisciplinary Training. Evaluation of these programs should primarily focus on the preparation, education, and training of the primary care workforce. Evaluation should include the synergism between Title VII, section 747 and other BHPr programs such as HCOP, AHECs, and Rural Interdisciplinary Training. Although placement of program graduates in underserved settings is not a principal focus of Title VII, section 747 programs, it strongly impacts the preparation of healthcare professionals to assume roles for other BHPr programs. Evaluation could include how these programs have properly trained professionals to serve in underserved settings and in programs like the NHSC and community, migrant, and rural health centers, as well as to care for underserved populations and high-risk groups.
4. BHPr should develop procedures for data collection, analysis, and reporting of Program outcome measures. BHPr should oversee the process of collecting, analyzing, and reporting qualitative and quantitative data associated with outcomes. These outcome measures should be developed and implemented consistent with the logic model, objectives, and near- and longer-term outcomes presented in this report. The process developed must provide meaningful reporting on Program effectiveness without placing undue burden on grantees for collecting data.
5. Additional financial resources should be made available to BHPr to implement effective evaluation processes for the programs in Title VII, section 747. The systematic evaluation of outcomes is critical to ensuring the effectiveness of Title VII, section 747 programs. However, establishing and sustaining effective evaluation methods is labor-intensive and expensive. Therefore, additional funds are necessary to develop an ongoing process for data collection, analysis, and reporting of program outcome measures. It should be noted that any plan to reassign funds from current Program allocations would strongly compromise this process and jeopardize the intended outcomes of these programs.

Figure text for main body-----

Figure 3. Title VII, Section 747 Strategic Framework

Purpose: To educate and train physicians, pediatric and general dentists, and physician assistants to enhance the quality, capacity, and diversity of the Nation’s primary care workforce, giving special consideration to the healthcare needs of underserved populations and other high-risk groups.

Stakeholders

• White House/OMB
• Congress
• U.S. Dept. of Health and Human Services
• Taxpayers/Public

Improve Health Outcomes of the Nation
Advance U.S. DHHS Primary Care Workforce Goals
Evaluate Impact of Title VII, Section 747 Programs
Improve the Nation’s Primary Care Training Quality and Capacity
Advise Primary  Care Workforce Policy
Disseminate Primary Care Delivery Innovations

Constituents

• Institutions ¹
• Learners ²

Improve Primary Care Education and Training
Serve as Change Agent
Grant Program Areas
Dental Residency
Physician Assistant
Academic Administrative
Faculty Development
Pre-doctoral
Primary Care Residency
Develop Training Innovations
Evaluate Innovations
Disseminate/Replicate Innovations
National Curricula and Training Programs

Operations

Operate Grant Programs
Develop Program Policy
Develop Partnerships
Manage Co-op and Collaborative Agreements and Contracts

¹ Instituions include departments, divisions, and programs.

² Learners include students, residents, and faculty.

Important
Very Important

--------------

Figure 4. Primary Care Education and Training Pipeline

Pre-college
College
Professional School
Residency
Exposure to Trained Faculty
Continuing Education

Market Place Influences
Lifestyle Preferences
Market Emphasis on Specialty Care

K - 12 Students
Trained Primary Care Providers
Improved Healthcare Outcomes

Title VII, Section 747
AHEC
HCOP
NHSC
Rural Interdisciplanary Training

--------------

Figure 5. Title VII, Section 747 Logic Model

Inputs
Activities
Outputs
Near-Term Outcomes
Longer-Term Outcomes

Title VII, section 747 legislative authority
Title VII, section 747 funding
DHHS and BHPr infrastructure

Education and training grant management
Operation of contracts and co-op and collaborative agreements
Education and training grants, faculty development, and curricular innovations
Cost-effective management of funding mechanisms
Timely approvals, disbursements, and receipt of grantee submissions

O1: Improved primary care education and training curricula
O6: Increased ability of learner and faculty to serve high-risk, special needs, and vulnerable populations
O2: Improved primary care training capacity
O7: Increased diversity of primary care faculty, residents, and students
O3: Increased training in underserved communities

O1: Improved primary care workforce training and quality
O2: Improved capacity of primary care training infrastructure
O7: Improved diversity of graduates (including underrepresented minorities)
O3: Increased primary care providers serving high-risk and underserved populations
O1: Goals met from Healthy People 2010

ACTPCMD
COGME
Public and private partnerships

Partnership development
Primary care research and policy development

Partnerships
Workforce studies and training research
Annual reports – research results and recommendations
Feedback and consensus
Performance indicators/baseline performance data

O4: Published primary care training and primary care research in priority areas
O5: Recommendations on policy or National guidelines
O5: Dissemination of innovations and best practices
O5: Adoption of innovations and best practices by others
--------------

Figure 6. Hierarchy of BHPr Outcomes

Health Care Outcomes

Workforce Outcomes
Supply
Diversity
Distribution

Medical/Dental Education Outcomes
Curricula
Faculty
Institutions ¹
Learners ²

Title VII, Section 747, 748 Programmatic Outputs
Primary Care Research and Policy Development
Partnerships
Grant Operations and Management
Contract Management

¹ Institutional outcomes include those related to departments, divisions, and programs.

² Learner outcomes include those related to students, residents, and faculty.

--------------

Figure 7. Recommended Outcome Measures

Key Objectives

Near-Term Measures
Longer-Term Measures

1. Improve the quality of education and training of the Nation’s primary care workforce.

• Evidence of competency of learners and faculty, as demonstrated by improvement in knowledge, skills, attitudes, etc.
• Goals met from Healthy People 2010

2. Improve the capacity for education and training of the Nation’s primary care workforce (with special emphasis on individuals from disadvantaged backgrounds and underrepresented minorities).

• Learners and faculty trained
• Retention in primary care
• Primary care training programs created, expanded, or affected as a result of Title VII, section 747 funding

3. Improve primary care education and training curricula.

• Grantees implementing curricula addressing critical healthcare needs as defined in the Title VII, section 747 Guidance
• Hours devoted to curriculum (can include training for culturally effective care)
• Grantees with rural and/or underserved populations training tracks, clinical rotations, etc.
• Programs that address emerging health care needs
• Graduates whose practice focuses on a specific underserved population or on a specific primary care problem

4. Improve primary care faculty development.

• Leadership roles and scholarly output of primary care faculty, as demonstrated by promotion and tenure, presentations and publications, research grants, advocacy, and public and professional service
• Primary care faculty in medical or dental educational institutions over a timeframe
• Primary care trained graduates in faculty positions

5. Identify, develop, and disseminate primary care education and training innovations and best practices among programs, accrediting bodies, and other constituents.

• Primary care education and training publications and research publications, including web-based publications
• Innovations, including use of new technology and best practices developed and adopted by accrediting bodies and others

6. Improve the preparation of faculty, residents and students (or learners) to work with medically and dentally underserved populations and build linkages to communities.

• Ambulatory and community-based training sites that serve primarily underserved populations
• Learners who are from disadvantaged backgrounds, who are from rural backgrounds, or who are underrepresented minorities or women
• Disadvantaged, high-risk, and special needs individuals served
• Graduates caring for underserved, uninsured, or special needs populations
• Where graduates practice

7. Improve the diversity and number of primary care faculty and students (or learners), with special emphasis on individuals from disadvantaged backgrounds and underrrepresented minorities.

• Learners who are from disadvantaged backgrounds, who are from rural backgrounds, or who are underrepresented minorities or women
• Learners among funded programs who indicate at matriculation and graduation that they intend to work in primary care
• Underrepresented minority faculty who have completed Title VII, section 747 faculty development programs, and who teach and/or serve underserved populations
• Underrepresented minority faculty involved in leadership or research positions
• Faculty, graduates, and practitioners trained in funded programs who are from disadvantaged backgrounds or who are underrepresented minorities or women

--------------

Figure 8. Evaluation Methodology

Develop Definitions for Outcomes Measured

• Align definitions with BHPr objectives and measures.
• Identify units of analysis and whether definitions are qualitative or quantitative.

Develop Data Collection Procedures

• Design data collection instruments.
• Establish collection procedures to minimize burden of response on grantees.
• Establish sampling approach.
• Determine which data are required and which data are optional.
• Establish quality assurance protocols.

Develop Data Analysis Procedures

• Design analysis.
• Establish baselines or standards.
• Identify bases of comparison.

Develop Reporting Approach

• Identify evaluation report audience.
• Design report content and format, and establish reporting frequency.

Evaluation Methodology