Appendices

Appendix A – Key Acronyms

Key acronyms used in the report include:

• ACTPCMD – Advisory Committee on Training in Primary Care Medicine and -Dentistry
• AHEC – Area Health Education Center
• BHPr – Bureau of Health Professions
• COGME – Council on Graduate Medical Education
• DHHS – U.S. Department of Health and Human Services
• D-HPSA – Dental Health Professional Shortage Area
• HCOP – Health Careers Opportunity Program
• HPSA – Health Professional Shortage Area
• HRSA – Health Resources and Services Administration
• MUC – Medically Underserved Community
• NHSC – National Health Service Corps
• OMB – Office of Management and Budget
• OSCE – Objective Structured Clinical Examinations
• PART – OMB’s Program Assessment Rating Tool
• PCSA – Primary Care Service Area
• UME-21 – Undergraduate Medical Education for the 21st Century

Appendix B – Examples of How to Read the Logic Model

Appendix B provides two simple examples (starting on the next page) that illustrate how to read logic models. The first example illustrates a funding scenario. The second illustrates the impact of Advisory Committee recommendations.

Funding

Funding levels influence the near- and longer-term outcomes of the Program. The shaded boxes in the logic model (Figure 9) on the next page illustrate how funding impacts factors such as the size and number of grants, which in turn impact the Program outcomes. When the logic model is read from left to right, it illustrates how funding contributes to the Program’s activities, outputs, and outcomes.

• Input. Title VII, section 747 funding is one of the inputs needed to operate the Program.
• Activities. The level of funding determines the level of activity such as education and training grants; operation of contracts, co-operative agreements, and collaborative agreements; partnership development; and primary care research and policy development. For example, more funding would allow for larger grants, less funding would require smaller grants.
• Outputs. The level of activity will in turn determine the amount of service delivered. The services include education and training grants and associated curricular innovations, partnerships, workforce studies, and annual reports.
• Near-term outcomes. The level of these services delivered will determine the level of outcomes for intended beneficiaries. The outcomes include improved primary care education and training curricula, increased ability of learner and faculty to serve, and improved primary care training capacity.
• Longer-term outcomes. These outcomes then provide long-term benefit to stakeholders and constituents. These outcomes include improved primary care workforce training and capacity, improved primary care training infrastructure, and improved diversity of graduates.

Impact of ACTPCMD Recommendations

The Committee’s recommendations influence the Program’s focus and priorities and thereby impact the types of innovations brought about by the Program. Adoption of these innovations helps to bring about the near- and longer-term outcomes of the Program. The shaded boxes in the logic model (Figure 10), depicted on the next page, when read from left to right, illustrate how the Committee’s recommendations contribute to the Program’s activities, outputs, and outcomes.

• Input. The ACTPCMD is one of the Program inputs. The Committee has a statutory obligation to make recommendations to the Congress and the Secretary of DHHS.
• Activities. A key Committee activity is primary care research resulting in policy development recommendations. For example, this year the Committee is developing policy recommendations on Program evaluation.
• Outputs. These policy recommendations are key to an annual report developed by the Committee.
• Near-term outcomes. The report makes a case for recommendations set out therein. In the case of this report, recommendations are made regarding policy for Program evaluation.
• Longer-term outcomes. Adoption of Program evaluation recommendations will not only enable policymakers to make better decisions about National healthcare policy, but will also facilitate continuous program improvement that will in turn enable improved primary care workforce education and training quality, improved capacity of training primary care infrastructure, and improved diversity of graduates.

Figure text for Appendix B-----

Figure 9. Logic Model – Funding Example
Inputs
Activities
Outputs
Near-Term Outcomes
Longer-Term Outcomes

Title VII, section 747 legislative authority
Title VII, section 747 funding
DHHS and BHPr infrastructure

Education and training grant management
Operation of contracts and co-op and collaborative agreements

Education and training grants, faculty development, and curricular innovations
Cost-effective management of funding mechanisms
Timely approvals, disbursements, and receipt of grantee submissions

O1: Improved primary care education and training curricula
O6: Increased ability of learner and faculty to serve high-risk, special needs, and vulnerable populations
O2: Improved primary care training capacity
O7: Increased diversity of primary care faculty, residents, and students
O3: Increased training in underserved communities

O1: Improved primary care workforce training and quality
O2: Improved capacity of primary care training infrastructure
O7: Improved diversity of graduates (including underrepresented minorities)
O3: Increased primary care providers serving high-risk and underserved populations
O1: Goals met from Healthy People 2010

ACTPCMD
COGME
Public and private partnerships

Partnership development
Primary care research and policy development

Partnerships
Workforce studies and training research
Annual reports – research results and recommendations
Feedback and consensus
Performance indicators/baseline performance data

O4: Published primary care training and primary care research in priority areas
O5: Recommendations on policy or National guidelines
O5: Dissemination of innovations and best practices

O5: Adoption of innovations and best practices by others

--------------

Figure 10. Logic Model – ACTPCMD Recommendation Example

Inputs
Activities
Outputs
Near-Term Outcomes
Longer-Term Outcomes

Title VII, section 747 legislative authority
Title VII, section 747 funding
DHHS and BHPr infrastructure

Education and training grant management
Operation of contracts and co-op and collaborative agreements

Education and training grants, faculty development, and curricular innovations
Cost-effective management of funding mechanisms
Timely approvals, disbursements, and receipt of grantee submissions

O1: Improved primary care education and training curricula
O6: Increased ability of learner and faculty to serve high-risk, special needs, and vulnerable populations
O2: Improved primary care training capacity
O7: Increased diversity of primary care faculty, residents, and students
O3: Increased training in underserved communities

O1: Improved primary care workforce training and quality
O2: Improved capacity of primary care training infrastructure
O7: Improved diversity of graduates (including underrepresented minorities)
O3: Increased primary care providers serving high-risk and underserved populations
O1: Goals met from Healthy People 2010

ACTPCMD
COGME
Public and private partnerships

Partnership development
Primary care research and policy development

Partnerships
Workforce studies and training research
Annual reports – research results and recommendations
Feedback and consensus
Performance indicators/baseline performance data

O4: Published primary care training and primary care research in priority areas
O5: Recommendations on policy or National guidelines
O5: Dissemination of innovations and best practices

O5: Adoption of innovations and best practices by others

Appendix C – BHPr Conceptual Framework and Core Performance Measures

The Bureau of Health Professions’ conceptual framework, depicted in Figure 11 on the next page, illustrates the relationship between its approximately 40 programs, the five common purposes of those programs, and the short-term, long-term, and ultimate outcomes of the Bureau’s program portfolio. The ultimate outcomes support various Healthy People 2010 goals. The purpose and objectives of Title VII, section 747 programs, as described in this report, are consonant with the common purposes and support the outcomes set out in the Bureau’s conceptual framework. Title VII, section 747 programs contribute directly to medical and dental education and training outcomes and, through these outcomes, contribute to BHPr’s common purposes and associated measures.

A summary of the Bureau-level core measures associated with the five common purpose areas is provided in Figure 12 on page 39. In addition, the figure identifies which of the seven Title VII, section 747 objectives identified in this report support each of the common purpose areas.

Title VII, Section 747 Objectives

1. Improve the quality of education and training of the Nation’s primary care workforce.
2. Improve the capacity for education and training of the Nation’s primary care workforce, with special emphasis on individuals from disadvantaged backgrounds and underrepresented minorities.
3. Improve primary care education and training -curricula.
4. Improve primary care faculty development.
5. Identify, develop, and disseminate primary care education and training innovations and best practices among programs, accrediting bodies, and other constituents.
6. Improve the preparation of faculty, residents, and students (or learners) to work with medically and dentally underserved populations and build linkages to communities.
7. Improve the diversity and number of primary care faculty and students (or learners), with special emphasis on individuals from disadvantaged backgrounds and underrepresented minorities.

Figure text for Appendix C-----

Figure 11. BHPr Conceptual Framework

Largely Measurable

Programs
Program Measures
BHPr Programs¹
Brief Description
Common Purpose
Short-Term Outcomes
Bureau-Level Performance Measures

Increase the diversity of the health professional population.
Increased Workforce Diversity
Matriculation and graduation rates for underrepresented minorities, and students from disadvantaged backgrounds; the proportion of minorities in the health professional workforce

Encourage the selection of a primary care career.
Primary Care Career Choice
Implementation of evidence-based strategies to promote careers in primary care, and the results of those strategies

Improve the distribution of health professionals in the United States.
Improved Workforce Distribution
Implementation of evidence-based strategies to improve workforce distribution, and the results of those strategies

Improve the quality of care, through education and training.
Improved Workforce Quality
The degree to which the Institute of Medicine’s 2003 core competencies are integrated into BHPr education and training programs; institutional commitment to addressing cultural competence and health literacy

Strengthen public health and health workforce infrastructure.
Improved Infrastructure for Health, Especially Primary Care Public Health
Improve timeliness and accessibility of data; the degree to which specific competencies related to public health are addressed in BHPr Programs

Less Measurable

Long-Term Outcome
Ultimate
National Outcome Measures

Improved Access to High-Quality Primary Care and Public Health Services
e.g., % with a usual source of care, receipt of preventive services

Reduced Morbidity
Reduced Mortality
Reduced Health Disparities

e.g., decreased mortality due to selected causes, decreased hospitalizations for primary care sensitive conditions, and improved black/white and Hispanic/white ratios for selected conditions

¹ BHPr programs include Centers of Excellence Program; Faculty Loan Repayment Program; Health Professional Student Loan Program; Loans for Disadvantaged Students Program; Minority Faculty Fellowship Program; Nursing Student Loan Program; Primary Care Loan Program; Scholarships Disadvantaged Students Program; Children’s Hospital Graduate Medical Education Payment Program; Dental Public Health Residency Program; Graduate Psychology Program; Training in Primary Care Medicine & Dentistry; National Research Service Award Program; Advanced Education Nursing Program; Comprehensive Geriatric Education Program-Nursing; Nursing Education Loan Repayment Program; Nursing Education, Practice, Retention Program; Nurse Faculty Loan Program; Nursing Workforce Diversity Program; National Health Service Corps Loan Repayment Program; Nursing Scholarship Program; National Health Service Corps Scholarship Program; Area Health Education Centers Program; Chiropractic Program; Public Health Traineeships Program; Public Health Training Center Program; Preventive Medicine Residency Program; State Loan Repayment Program; and Practitioner Databanks Programs.

--------------

Figure 12. Summary of BHPr Core Performance Measure

Common Purpose
Core Performance Measures
Title VII, Section 747 Objective

Diversity
Strategy: Increase health workforce diversity.

• Percentage of underrepresented minorities matriculating in health professions education and training programs
• Percentage of underrepresented minorities graduating from health professions education and training programs
• Percentage of disadvantaged students matriculating in health professions education and training programs
• Percentage of disadvantaged students graduating from health professions education and training programs
• Percentage of underrepresented minorities among health professionals younger than age 35 2, 6, and 7

Primary Care
Strategy: Promote careers in primary care.

• Degree to which Bureau-funded programs are implementing evidence-based strategies to promote selection of a primary care career among health professionals
• Percent change in the number of residency and traineeship positions filled annually in primary care medicine and dentistry
• Number of primary care health professionals as a percentage of all health professionals 1, 2, 3, 4, 5, 6, and 7

Distribution
Strategy: Improve the distribution of the primary care health workforce.

• Degree to which Bureau-funded programs are implementing evidence-based strategies to positively influence the distribution of the health professional workforce
• Level of disparity in the distribution of primary care physicians across Primary Care Service Areas (PCSAs) in the United States.
• Percent of the population living in areas below a set population to primary care provider ratio 2, 3, 6, and 7

Quality
Strategy: Improve the quality of care.

• Degree to which patient-centered care, health informatics, evidence-based decision-making, interdisciplinary team training, and quality measurement and improvement are integrated into BHPr-supported health professional education and training programs
• Degree to which BHPr education/training grantees include cultural competence in their programs
• Percentage of BHPR grantees whose organizations have an institutional policy addressing health literacy 1, 3, 4, 5, and 6

Infrastructure
Strategy: Strengthen public health and healthcare infrastructure.

• Accessibility of BHPr-developed data and information resources vital to health workforce analysis
• Degree to which Bureau-supported education and training programs contribute to the attainment of improved workforce competencies in population-based health
• National average score on Essential Public Health Function: Human Resource Development and Training in Public Health 1, 2, 3, 4, 5, 6, and 7

Appendix D – Description of Measures

Appendix D provides descriptions of the 24 recommended measures. Although the recommended process for developing definitions for the measures is described in the section entitled Develop Definitions for Outcome Measures, page 16, this appendix provides guidelines and examples for consideration for each measure.

Objective 1: Improve the quality of education and training of the Nation’s primary care workforce.

1. Evidence of competency of learners and faculty, as demonstrated by improvement in knowledge, skills, attitudes, etc. A traditional way of measuring an educational intervention is to assess the extent to which learners have advanced their knowledge, skills, or attitudes as a result of the intervention. Evidence of increased learner and faculty competency in the primary healthcare training setting demonstrates improved quality of education and training of the Nation’s primary care workforce. There is a need for the establishment of valid and reliable measures of educational program quality and outcomes to evaluate educational effectiveness (Heidenrich, Lye, et al., 2000; Bordage, Burack, et al., 1998). Examples for this measure may include traditional tests, pre- and post-test comparisons of trainees, Objective-Structured Clinical Examinations (OSCEs), tracking of results of the American Academy of Pediatrics’ Education in Quality Improvement for Pediatric Practice (eQIPP) Program, competency evaluations, surveys, and quasi-experiments.
2. Goals met from Healthy People 2010. Progress on the Healthy People 2010 goals is an indicator of overall National health outcomes to which the Title VII, section 747 programs contribute. In particular, Title VII, section 747 programs can contribute to the goal of eliminating health disparities among different segments of the population. For this measure, progress on relevant Healthy People 2010 goals should be evaluated. These Healthy People 2010 goals may include an:
   • Increase in the proportion of persons who have a specific source of ongoing care
   • Increase in the proportion of persons with a usual primary care provider
   • Increase in the proportion of schools of medicine, nursing, and health professional training schools whose basic curriculum for healthcare providers includes the core competencies in health promotion and disease prevention

Objective 2: Improve the capacity for education and training of the Nation’s primary care workforce, with special emphasis on individuals from disadvantaged backgrounds and underrepresented minorities.

3. Learners and faculty trained. Increasing capacity (e.g., more institutions and programs, faculty, training tracks, and clinical sites) for educating and training the Nation’s primary care workforce enables an increase in the number of learners and faculty trained. Examples of this measure include number or percentage of primary care learners graduated and number or percentage of faculty trained.
4. Retention in primary care. Increases in the retention of providers in primary care settings will lead to increased supply of providers in primary care settings. Examples of this measure include length of time providers remain in primary care settings, number of providers who remain in primary care practice for a certain period of time, and percentage of primary care providers remaining in primary care settings for a certain period of time.
5. Primary care training programs created, expanded, or affected as a result of Title VII, section 747 funding. Politzer, Hardwick, and colleagues (1999), suggest that measures such as the number of family medicine departments influenced is a good indicator of Title VII, section 747’s impact on capacity for training the primary care workforce. Examples of this measure include the number of training programs created in family practice, general pediatrics, general internal medicine, physician assistant, general dentistry, and pediatric dentistry.

Objective 3: Improve primary care education and training curricula.

6. Grantees implementing curricula addressing critical healthcare needs as defined in the Title VII, section 747 Guidance. An example of this measure is the number or percentage of grantees implementing curricula addressing critical healthcare needs as defined in the Title VII, section 747 Guidance.
7. Hours devoted to curriculum (can include training for culturally effective care). The time commitment as a measure of emphasis can be measured through hours spent on a given curriculum. Examples of this measure include the percentage of total hours (e.g., class time) spent on primary care curricula and training time for Title VII, section 747 residents relative to the time spent by all primary care residents.
8. Grantees with rural and/or underserved population training tracks, clinical rotations, etc. Campos-Outcalt, Senf, and colleagues (2004), indicate that an important factor in the increased production of primary care physicians is the adoption of a curriculum that maximizes clinical training with primary care physicians. They found a relationship between the number of family medicine clerkship sites to which students were exposed and the decision to practice family medicine. Meurer (1995) found that three types of curricular experiences may increase interest in primary care: third-year family medicine clerkships, continuity experiences in primary care settings, and primary care tracks. Direct experience working with rural or underserved populations through training tracks and clinical rotations improves the primary care curriculum by providing actual experiences that prepare physicians to serve those groups. Examples of this measure include the average number of family medicine clerkship sites to which students are exposed and the time spent in clinical rotations in primary care education.
9. Programs that address emerging healthcare needs. Emerging healthcare needs should be integrated into the primary care education and training curricula. Examples include bioterrorism preparedness, genomics, evidence-based guidelines, team-based care, use of technology, and use of data for quality improvement.
10. Graduates whose practice focuses on a specific underserved population or on a specific primary care problem. The long-term result of improved primary care education and training is an increase in graduates who are practicing primary care in primary care settings. Examples of this measure include the number or percentage of Title VII, section 747 graduates serving underserved populations relative to all primary care practitioners or number of medically compromised patients served (Atchison, Mito, et al., 2002).

Objective 4: Improve primary care faculty development.

11. Leadership roles and scholarly output of primary care faculty, as demonstrated by promotion and tenure, presentations and publications, research grants, advocacy, and public and professional serv-ice. Faculty productivity and leadership have been measured by teaching assessments, publication and dissemination of work, peer review, faculty in leadership positions (Rabinowitz, Babbott, et al., 2001), and community service. Examples of this measure include the number of primary care-related articles published, reports, and presentations delivered by funded faculty.
12. Primary care faculty in medical or dental educational institutions over a timeframe. The number of primary care faculty per year over a period of time is an indicator of change in education and training capacity. An example of this measure is growth in the number or percentage of primary care faculty at funded programs over a specific time period (e.g., 5 years). Another example is the number or percentage of primary care faculty at funded programs remaining in primary care teaching positions over a specific time period (e.g., 5 years).
13. Primary care trained graduates in faculty positions. The number of primary care graduates trained at funded institutions or programs who go on to take faculty positions in educational institutions is an indicator of Title VII, section 747 impact on education and training capacity resulting from faculty development. An example of this measure is the number or percentage of graduates from funded programs taking faculty positions.

Objective 5: Identify, develop, and disseminate primary care education and training -innovations and best practices among programs, accrediting bodies, and other -constituents.

14. Primary care education and training publications and primary care research publications, including web-based publications. To identify, develop, and disseminate innovations and best practices and their relative effectiveness, primary care education and training programs need to develop incentives. To benefit other programs, these best practices and innovations need to be shared with others who might adopt them through publications, presentations, web sites, and other vehicles. Examples of this measure include the number of education- and training-related publications, including web-based publications, and a qualitative assessment by an expert panel or an independent organization of the impact of publications.
15. Innovations, including use of new technology and best practices developed and adopted by accrediting bodies and others. In order to yield maximum benefit from innovations and best practices, accrediting bodies must adopt them. Adoption by accrediting bodies is also an indicator of quality or potential benefit provided by these innovations and best practices. Examples of this measure include the qualitative assessment of impact of innovations and best practices adopted, assessment made by an expert panel, and number of innovations and best practices developed and adopted.

Objective 6: Improve the preparation of faculty, residents, and students (or learners) to work with medically and dentally underserved populations and build linkages to -communities.

16. Ambulatory and community-based training sites that serve primarily underserved populations. There is evidence that exposure to training sites that serve underserved populations increases the likelihood that graduates will go on to work in those communities (Norris, Acosta, 1997). Over three-quarters of rural training track graduates practice in rural areas and feel prepared for rural practice (Rosenthal, 2000). Physicians who are prepared to become rural doctors and are prepared for small-town living tend to stay longer in rural practices (Pathman, Steiner, et al., 1999). Examples of this measure include the average number of hours of exposure to training sites that serve underserved populations and the average number of sites to which students are exposed that serve underserved populations.
17. Learners who are from disadvantaged backgrounds, who are from rural backgrounds, or who are underrepresented minorities or women. Pathman, Williams, and colleagues (1996), found that NHSC minority physicians tended to work in counties and practices with a greater proportion of minority residents and patients. Cantor, Miles, et al. (1996), indicated that women and minority physicians are more likely to serve poor, minority, and Medicaid populations. Rabinowitz and Paynter (2000) and Easterbrook, Godwin, and colleagues (1999), report that targeted selection of students from rural backgrounds is effective in producing physicians who practice in rural areas. Komaromy, Grumbach, et al. (1996), found that black and Hispanic physicians were more likely to provide healthcare for underserved populations. An example of this measure includes the number or percentage of learners who are economically disadvantaged, who are from rural backgrounds, or who are underrepresented minorities or women. (This measure applies to both Objectives 6 and 7.)
18. Disadvantaged, high-risk, and special needs individuals served. In addition to educating and training the primary care workforce, Title VII, section 747 training sites provide significant health services to those populations. An example of this measure includes the number of disadvantaged, high-risk, and special needs individuals who are served by Program learners or graduates.
19. Graduates caring for underserved, uninsured, or special needs populations. The presence and retention of primary care providers caring for underserved populations (Rosenblatt, Saunders, et al., 1996) enhances access for individuals in those areas. The number of graduates caring for these populations is an indicator of direct impact on access. An example of this measure includes the number of graduates who are caring for the underserved, graduates who are caring for uninsured patients, and graduates who are caring for special needs populations. Further examples of this measure include the number of primary care providers working in underserved areas, retention rates for primary care providers in underserved areas, and the number of primary care providers caring for underserved individuals, including those with special healthcare needs, Medicaid, State Children’s Health Insurance Program (SCHIP), and no insurance (pro-bono care).
20. Where graduates practice. Data from loan repayment programs, Association of American Medical Colleges (AAMC), American Academy of Family Physicians (AAFP), American Osteopathic Association (AOA), American Association of Colleges of Osteopathic Medicine (AACOM), and other professional organizations can be used to identify where practitioners are located. Examples of this measure include geographic distribution of graduates and proportion practicing in underserved versus non-underserved areas.

Objective 7: Improve the diversity and number of primary care faculty and students (or learners), with special emphasis on individuals from disadvantaged backgrounds and underrepresented minorities.

17.  Learners who are from disadvantaged backgrounds, who are from rural backgrounds, or who are underrepresented minorities or women. (This measure applies to both Objectives 6 and 7.)

21. Learners among funded programs who indicate at matriculation and graduation that they intend to work in primary care. Intention to serve at matriculation is a good indicator and valid measure of subsequent practice choice as a primary care provider. Examples of this measure include students indicating at matriculation their intention to practice primary care and students indicating at graduation their intention to practice primary care.
22. Underrepresented minority faculty who have completed Title VII, section 747 faculty development programs, and who teach and/or serve underserved populations. Examples of this measure include the number of Title VII, section 747 trained faculty who are from underrepresented groups and care for underserved populations or high-risk groups, and the number of Title VII, section 747 trained faculty who are engaged in primary care education and training.
23. Underrepresented minority faculty involved in leadership or research positions. Examples of this measure include the number of faculty from underrepresented groups who are involved in leadership positions, and the number of faculty from underrepresented groups who are involved in educational research, especially with regard to underserved populations.
24. Faculty, graduates, and practitioners trained in funded programs who are from disadvantaged backgrounds or are underrepresented minorities or are women. An example of this measure is the number or percentage of faculty, graduates, and practitioners who are underrepresented minorities, women, from disadvantaged backgrounds, or from rural or inner-city settings.