Recommended Outcome Measures

On the basis of the literature review, the Program’s logic model, and the review of BHPr’s goals and objectives, the Committee identified a list of candidate outcome measures. In order to prioritize measures in the candidate list and develop a set of recommended measures, the Committee established criteria for identifying the most appropriate measures. These criteria included:

1. The likelihood exists for obtaining accurate and reliable data for the measure at reasonable cost. This criterion is especially important because evaluation burden on grantees can have a significant impact on program viability.
2. The measure facilitates aggregation with other BHPr measures. This criterion helps ensure the ability to demonstrate the relevance of Title VII, section 747 programs in the context of the Bureau’s measures.
3. The measure is resistant to goal displacement. It is important that the measure not become the focus of program operations at the expense of other key objectives of Title VII, section 747.
4. The measure enables comparison over time to assess the degree of performance improvement.
5. The measure facilitates aggregation among the Title VII, section 747 program areas, supporting a comprehensive, integrated view of the programs.-
6. The measure reflects an evaluation method relevant to the needs of OMB and HRSA’s BHPr.
7. The measure facilitates easy assessment of perform-ance relative to targets.

The Committee then applied these criteria to the set of candidate outcome measures to develop a list of recommended outcome measures for each objective. The recommended measures are provided in the chart (Figure 7) on the following page. These measures reflect the direct impact of Title VII, section 747 programs. In addition, these programs bring about other indirect impacts through their complementary relationships with other BHPr programs, as discussed in the section entitled Title VII, Section 747’s Influence on the Workforce, page 3. In evaluating the impact of its portfolio of programs, BHPr should consider the synergy in these complementary relationships by identifying indicators that evaluate the synergy between Title VII, section 747 programs and other BHPr programs, especially the NHSC, as well as HCOP, AHECs, and Rural Interdisciplinary Training.

These outcome measures are described in Appendix D, Description of Measures, page 41.

Evaluation Methodology

This section presents an overall methodology for applying the measures discussed in the section entitled Recommended Outcome Measures, page 15. A well-constructed evaluation provides a rich set of data that not only can provide meaningful insights into program impact, but also can advance knowledge about education and training principles and approaches (see Figure 8 on page 18).

Develop Definitions for Outcome Measures

Developing appropriate definitions for measures is an important step for ensuring meaningful evaluation of the programs. Many of these definitions will be obvious and non-contentious. Some, however, will require careful consideration of Program and BHPr objectives. For example, definitions for measurement parameters such as “underserved” communities must be established to reflect Program and BHPr objectives. If Dental–HPSAs (D-HPSAs) were adopted as the definition for “underserved” communities with respect to dental care, and a State fails to designate adequately all potential D-HPSAs, then Title VII, section 747-trained pediatric dentists who locate in qualifying, but undesignated, communities will not be recognized.

In addition, some of the measures identified in the section entitled Recommended Outcome Measures, page 15, can effectively express the impact of the outcome in purely quantitative terms. For example, the measure: “Learners who are from disadvantaged backgrounds, who are from rural backgrounds, or who are underrepresented minorities or women” can be expressed in simple quantitative terms, and longitudinal comparisons are meaningful. However, the measure: “Innovations, including use of new technology and best practices developed and adopted by accrediting bodies and others” may have little significance in purely quantitative terms. To convey the programs’ impact with regard to this measure’s outcome, it may be necessary to develop a qualitative assessment of the impact of the innovations and best practices.

Develop Data Collection Procedures

The next step in evaluation methodology is to develop data collection procedures. Data can come from a variety of sources, including HRSA’s BHPr records, grantee applications, tests, surveys, or other data collection instruments. In some cases, there will be existing instruments that can be used. However, some new data collection instruments may need to be developed. It is important to ensure that the -reporting burden imposed on grantees is not excessive. Most grantees do not have significant resources for reporting, and as discussed earlier, Title VII, section 747 grants are typically a small portion of their overall program funding.

Some measures may be derived using a statistically significant sample rather than collecting data from all grantees or beneficiaries. For example, a measure such as: “Graduates caring for underserved, uninsured, or special needs populations” may be derived through sampling instead of collecting data from all grantees.

All data collection procedures require establishment of quality assurance protocols. Such protocols can include procedures to ensure reliability through uniform data collection and careful decisions about sampling strategies to ensure against bias.

Develop Data Analysis Procedures

The goal of developing data analysis procedures is to make the most effective use of the outcome measures. Establishing baselines, targets, and identifying what kinds of comparisons are most meaningful will be important evaluation efforts. Many of the recommended measures lend themselves to making comparisons over time and developing trend analysis. For example, the measures: “Ambulatory and community-based training sites that serve primarily underserved populations” and “Graduates caring for underserved, uninsured, or special needs populations” can be followed serially over time. It will also be useful to make comparisons among grantees. Such comparisons can help to identify best practices or innovations.

Develop a Reporting Approach

The outcome measures identified in the section entitled Recommended Outcome Measures, page 15, will provide a solid foundation for evaluating the impact of Title VII, section 747 programs. The goal of the reporting approach is to provide effective communication of program impact to constituents and stakeholders. In order to be effective, reporting mechanisms should express the results in terms relevant and understandable to the audience.

Summary

The systematic evaluation of outcomes is critical to ensuring the effectiveness of Title VII, section 747 programs. However, establishing and sustaining effective evaluation methods is labor-intensive and expensive. Therefore, additional funds are necessary to develop an ongoing process for data collection, analysis, and reporting of program outcome measures. Grantees have few resources for data collection and analysis. It should be noted that any plan to reassign funds from current Program allocations would strongly compromise this process and jeopardize the intended outcomes of these programs.