Approach for Developing Outcome Measures

In order to identify measures for the seven scientifically sound and programmatically relevant objectives, the Committee conducted an assessment that included:

• Review of the literature on evaluation of education and training programs (see section entitled Literature Review, page 11)
• Development of a Title VII, section 747 Program logic model (see section entitled Description and Method for Reading the Logic Model, page 13)
• Review of BHPr’s goals and Title VII, section 747’s role, as part of a portfolio of programs, in meeting those goals (see section entitled Title VII, Section 747’s Influence on the Workforce, page 3)

Literature Review

The Committee conducted a literature review of research published on evaluation of medical and dental education and training programs. The research was categorized into three outcome groups: education- and training-related, workforce-related, and healthcare-related.

Education - and Training - Related Outcomes

Education and training outcomes identified in the literature were further categorized into three subgroups: Learner, Curricular, and Institutional. Examples of outcome measures identified in these groups are provided below:

• Learner. Learner outcomes include those related to students, residents, and faculty who participated in the program being evaluated. This category could include outcomes such as student performance on the U.S. Medical Licensing Examination, learner surveys, and peer review. Examples of learner outcomes include:
  • Measured accomplishment of learning objectives such as changes in knowledge, skills, and attitudes (Olney, 1995; Arnold, Blank, et al., 1998; -Holmboe, Hawkins, 1998; Lawrence, Lindemann, et al., 1999; Association of American Medical Colleges, 2000; Peters, Greenberger, et al., 2000; Rabinowitz, Babbott, et al., 2001; Swing, 2002; Brasel, Bragg, et al., 2004; Papadakis, Hodgson, et al., 2004)
  • Participant and student evaluations and satisfaction ratings (Irby, Rakestraw, 1981; Davidson, Vega, et al., 1996; Whiteside, Pope, et al., 1997)
  • Program, residency, and fellowship completion rates (Wax, Donovan, 2000)
  • Examinations, grades, scores, and achievement tests (Elnicki, Halbritter, et al., 1999)
  • Student performance in standardized patient examinations (Haist, Griffith, et al., 2004)
  • Peer review ratings (Thomas, Gebo, et al., 1999)
  • Patient care provided by learner outcomes (Brook, Fink, et al., 1987; Norcini, Blank, et al., 1995; Evans, Rogers, et al., 1996; Wong, Hollenberg, et al., 1999)
  • Clinical productivity (Anderson, Stritter, et al., 1997; Taylor, Friedman, et al., 2001)
• Curricular. Curricular outcomes include those related to teaching methods, use of innovative curricula, and courses. Examples of curricular outcomes include:
  • Implementation of innovative curricula (Bowen, Stearns, et al., 1997; Kurth, Irigoyen, et al., 2000)
  • Currency and relevancy of curricular topics (Bowen, Stearns, et al., 1997; Kurth, Irigoyen, et al., 2000)
  • Teaching methods and use of structured or other teaching evaluations (Prislin, Fitzpatrick, et al., 1998; Zayas, James, et al., 1999)
  • Opportunities for on-site learning and feedback, and monitoring of student learning activities (-Whiteside, Mathias, 1996; Whiteside, Pope, et al., 1997)
  • Provision of multicultural curricula and cultural competence curricula (Pena, Munoz, et al., 2003)
• Institutional. Institutional outcomes include those related to schools and faculty. This outcome could include measures such as faculty in leadership positions. Examples of institutional outcomes include:
  • Growth in number of schools with accredited family medicine programs (Politzer, Hardwick, et al., 1999)
  • Increase in number of primary care and family medicine residency slots (Whitcomb, Cullen, et al., 1992; Rabinowitz, Diamond, et al., 1999; Pugno, Schmittling, et al., 2000; Campos-Outcalt, Senf, et al., 2004; Phillips, Starfield, 2004)
  • Growth in number of programs promoting primary care training and service in underserved areas (Ricketts, Hart, 2001; Campos-Outcalt, Senf, et al., 2004)
  • Rotations in primary care slots or rural residencies (Norris, Acosta, 1997; Easterbrook, Godwin, et al., 1999; Pathman, Steiner, et al., 1999; Geyman, Hart, et al., 2000; Rosenthal, 2000; Rosenblatt, Schneeweiss, et al., 2002)
  • Implementation of targeted selection and preferential admission of students likely to enter primary care or serve in underserved areas, as well as demographics of these students (Rosenthal, Rabinowitz, et al., 1996; Fryer, Stine, et al., 1997; Xu, Fields, et al., 1997; Shi, Samuels, et al., 1998; Easterbrook, Godwin, et al., 1999; Hart, 2001; Rabinowitz, Paynter, 2000; Grumbach, Coffman, et al., 2002)
  • Growth in number of qualified teachers (Irby, 1995; Zayas, James, et al., 1999; Heidenreich, Lye, et al., 2000)
  • Growth of faculty in leadership positions (-Rabinowitz, Babbott, et al., 2001)
  • Increase in learner satisfaction with the training program (Keitz, Holland, et al., 2003)

Workforce-Related Outcomes

In order to identify candidate outcomes and measures for the workforce outcomes linked to education and training, the Committee further categorized the workforce-related outcomes identified in the literature into three subgroups: Supply, Diversity, and Distribution. Examples of outcome measures identified in these groups are provided below:

• Supply. Supply outcomes include those related to numbers of learners, graduates, or providers. Examples of supply outcomes include:
  • Increased match rates for primary care specialties (Phillips, Starfield, 2004; Whitcomb, Cohen, 2004)
  • Increased percentage of graduates entering in and graduating from primary care medical disciplines (Whitcomb, Cullen, et al., 1992; Campos-Outcalt, Senf, et al., 2004; Phillips, Starfield, 2004)
  • Increased supply of primary care providers (Lurie, Goodman, et al., 2002; Meyers, Fryer, et al., 2002)
  • Retention of program graduates in underserved areas (Rosenblatt, Saunders, et al., 1996)
• Diversity. Diversity outcomes include those related to the demographics of learners, graduates, or providers. Examples of diversity outcomes include:
  • Increased supply of female and minority providers (Ellsbury, Doescher, et al., 2000)
  • Increased number of women and minorities practicing as generalists (Council on Graduate Medical Education, 1998)
  • Increased number of women practicing in rural areas (Doescher, Ellsbury, et al., 2000; Ellsbury, Baldwin, et al., 2002)
  • Demographics of students (Xu, Fields, et al., 1997; Shi, Samuels, et al., 1998; Grumbach, Coffman, et al., 2002)
• Distribution. Distribution outcomes include those related to placement of learners, graduates, or providers. Examples of distribution outcomes include:
  • Increased number of providers practicing in HPSAs (Chan, Hart, et al., 2004)
  • Increased number of providers practicing in underserved areas, rural areas, and inner cities as generalists (Komaromy, Grumbach, et al., 1996; Pathman, Williams, et al., 1996)
  • Increased number of providers practicing in rural areas (Easterbrook, Godwin, et al., 1999)
  • Increased number of providers practicing in underserved areas (Fink, Phillips, et al., 2003)
  • Decreased level of need of patients and communities in underserved areas (Pathman, Konrad, et al., 2004)

Healthcare-Related Outcomes

As discussed earlier, Title VII, section 747 programs are designed primarily to bring about education and training outcomes. Therefore, the Committee’s evaluation methodology efforts were focused primarily on education and training outcomes. However, the literature search yielded a small number of articles that linked healthcare-related outcomes to education and training. There are a number of challenges in examining the relationship between training and healthcare outcomes, including the latency of educational effect, individual variations, and the difficulty of controlling for any educational intervention (Chen, Bacuhner, et al., 2004). In addition, comparisons of clinical practice across different sites and healthcare systems are difficult to draw because they require relatively complex research designs or statistical techniques to adjust for variations in case mix among patient populations (Peabody, Luck, et al., 2004). Such confounders potentially dilute the ability to measure these outcomes. Given these considerations, two articles by Phillips, Starfield (2004), and Lishner, Rosenblatt, et al. (2000), for example, suggested correlations between primary care training and decreased mortality, reduced use of emergency departments, and better preventive care and screening.

Description and Method for Reading the Logic Model

A program logic model is a tool that provides a simplified visual representation of how a program or project is expected to work to achieve intended results (Schiller, 2004). According to the W.K. Kellogg Foundation, a program logic model links outcomes (both near- and longer-term) with program activities and processes as well as the theoretical assumptions and principles of the program (W.K. Kellogg Foundation, 2001). The process of developing a logic model is an opportunity for charting the course of planning, design, implementation, analysis, and knowledge generation for a program. Logic modeling can greatly enhance the participatory role and usefulness of evaluation as a management and learning tool.

Several types of logic models can be used to represent how a program accomplishes its objectives. These include Theory Approach Models, Outcomes Approach Models, and Activities Approach Models. For the Committee’s objectives, the Outcomes Approach Model was deemed the most appropriate. Outcomes Approach Models focus on program planning and identification of near-term (1–3 years) and longer-term (4–6 years) outcomes.

When read from left to right, the logic model describes how the program achieves results from the planning stage to outcomes. Reading a logic model means following the chain of reasoning of “if…then…” statements that connect the program’s parts (W.K. Kellogg Foundation, 2001).

When read from left to right, the major components of this type of logic model are:

• Inputs. These are resources (or barriers) that enable (or limit) program effectiveness or output. Resources may include funding, existing organizations, collaborating partners, staff, time, infrastructure, and others. Barriers may include regulations, geography, attitudes, or other limiting factors.
• Activities. These are processes, operations, and actions of the planned programs.
• Outputs. These are the direct results of program activities. They are usually described in terms of the size or scope, or both, of the services and products delivered or produced by the program.
• Outcomes. These are the impacts on intended beneficiaries. They may include specific changes in attitudes, behaviors, knowledge, skills, or level of function.

Appendix B, page 31, provides two illustrative examples of how to read the logic model. In describing the logic model for Title VII, section 747 programs, it is important to distinguish between program outputs and outcomes. Outputs represent the direct result of program operations and activities. An example of an output would include the grants administered by the Title VII, section 747 programs. Outcomes are the impacts on the intended beneficiaries resulting from the outputs. Examples of outcomes include training innovations or increased faculty competencies resulting from a grant.

The impact of outputs is more under the direct control of program managers, whereas realization of the outcomes will depend not only on the achievement of outputs, but also on the validity of the program’s logic model (whether the program’s outputs actually lead to the desired outcomes) and a set of exogenous factors. The logic model facilitated the Committee’s identification of near- and longer-term measures associated with program objectives.

The Committee then identified inputs, activities, and outputs necessary to bring about these near- and longer-term outcomes.

The chart (Figure 5) on the next page sets out the Committee’s Program Logic Model for Title VII, section 747 programs.

For each near- and longer-term outcome, the chart shown above indicates which of the seven key objectives, provided in the section entitled Objectives of Title VII, Section 747 Programs, page 2, are supported by the outcome. For example, Objective 5 is Identify, develop, and disseminate primary care education and training innovations and best practices among programs, accrediting bodies, and other constituents. A near-term outcome is Recommendations on policy or National guidelines. (O5 indicates Objective 5 in the chart). A longer-term outcome is Adoption of innovations and best practices by others.

Review of BHPr Objectives

The third step in the Committee’s process to develop scientifically sound and programmatically relevant measures was to review BHPr’s goals and Title VII, section 747’s role in achieving those goals. BHPr is in the process of refining performance measures to evaluate progress on the Bureau’s goals. To help ensure alignment between BHPr’s goals and Title VII, section 747 activities, the Committee took into consideration BHPr’s goals and associated measures in identifying the Title VII, section 747 measures. In addition to program--specific measures, which are designed to capture unique accomplishments of each of BHPr’s programs, BHPr is refining a set of five core measure areas, which are designed to capture accomplishments across five common purposes at the Bureau level. These broader measures, which use aggregated program data, include:

• Diversity. Increase health workforce diversity.
• Primary care. Promote careers in primary care.
• Distribution. Improve the distribution of the healthcare workforce.
• Quality. Improve the quality of care.
• Infrastructure. Strengthen public health and healthcare infrastructure.

In addition, BHPr is refining a set of National outcome measures, which relate to ultimate healthcare outcomes, including access to primary care and health status. The Title VII, section 747 programs contribute directly to medical and dental education and training outcomes and, through these outcomes, contribute to BHPr’s core measures and National healthcare outcome goals as depicted in the figure below.

The Committee’s objective was to identify medical and dental education and training outcome measures that were consistent with BHPr’s goals and measures. Appendix C, page 37, provides a summary of BHPr’s common purposes and associated core measures including a discussion of the consonance of these common purposes with the Title VII, section 747 objectives.