V. Recommendations on Outcomes and Performance Measures

The Committee recognizes and endorses the efforts that the Bureau of Health Professions (BHPr) has made relative to developing performance measures to help monitor progress of interdisciplinary programs and disseminate program outcomes to a wide variety of external constituencies. The Committee encourages BHPr to work with representatives from the interdisciplinary programs within its portfolio as they consider responding to the recommendations listed below. This will ensure that any new performance measures and processes/procedures associated with reporting of outcomes will better support the BHPr mission and more accurately reflect specific programmatic impact and relevance.

Recommendation 1: The Committee recommends inclusion of additional performance measures within the evaluation framework that more completely describe outcomes related to the changes of health status within communities and the economic impact of the various interdisciplinary programs.

Rationale

Endorsing a statement of common purpose and overarching goals related to changes in health status and economic impact at the community level may accomplish this. The existing evaluation framework focuses on measurements that track workforce diversity and the placement of health care workers in underserved communities. Examination of the effects of these processes represents a natural extension of this analysis. Performance measures should include changes in health status in underserved communities, the relationship of specific programming to changes in health status and direct services, and the quality and distribution of healthcare providers. Presently, there are no objective measurement tools that document changes in health care costs attributable to these programs. The evaluation of economic impact is a critical determinant of the value of interdisciplinary programs within the community. Tools need to be developed to illustrate cost savings with regard to: 1) health care expenditures for older adults receiving interdisciplinary care; 2) frequency of co-morbid conditions in older adults receiving interdisciplinary care; and 3) efficiency of health care professionals involved in interdisciplinary care settings. Additionally, performance measures should reflect demonstrated need for Federal support, leveraging of other sources of funds (State/local/private), and effectiveness of consortia and partnership efforts. This approach should be coordinated through the development of logic models in collaboration with BHPr.

Benefits

An additional and significant data set for the interdisciplinary programs will be obtained through a coordinated effort and approach that supports the dissemination of outcomes that more completely reflect direct community and economic impact. External constituencies will be able to link programmatic initiatives directly to underserved communities and vulnerable populations.

Recommendation 2: The Committee recommends that BHPr work with other Federal agencies such as the Office of the Management of the Budget (OMB) and Congressional Budget Office (CBO) to develop data collection processes for the interdisciplinary programs that include the use of qualitative data.

Rationale

To obtain a comprehensive evaluation of the interdisciplinary programs, one must consider both quantitative and qualitative data. Qualitative data provides information related to the end user and a more complete picture of community and economic impact. The current evaluation framework does not provide a mechanism to capture information that demonstrates the effects of education and training of health professionals on the target population. Similarly, it has not been possible to gauge the effectiveness of outreach activities targeted to pre-professional students and educational initiatives targeted to the general public.

Examples of qualitative data that can be considered are information obtained through end-user and community dialogue, description of program improvements and innovations, unexpected outcomes, and anecdotal accounts. These may be particularly relevant to the analysis of changes in cultural competency, workforce diversity, and the health professions pipeline. To obtain qualitative data it may be necessary to develop a qualitative instrument that encourages end user to submit their input as to the effectiveness of programs in their community.

Benefits

This approach may provide a better mechanism to communicate tertiary outcomes for the interdisciplinary programs. External constituencies may be able to more completely describe the effects of the interdisciplinary programs on individuals within communities. This would represent a significant and timely development in the evolution of the evaluation framework for these programs.

Recommendation 3: The Committee recommends development of a process for sharing data from all interdisciplinary programs within BHPr, among interested federal agencies, and across the programs.

Rationale

The Committee supports the BHPr concept of developing a data system for all interdisciplinary programs that is well defined, easily understood by all, and a minimal burden to the users. Additionally, the Committee supports the concept of the "one department" approach to evaluation and oversight of federally funded programs. However, joint assessment of program effectiveness should be based on evaluation criteria that produce acceptable and consistent documentation of outcomes. For the BHPr interdisciplinary programs, this should include a core of standardized reporting items (performance measures) across all programs. A web-based process could more easily facilitate access to and sharing of these standardized performance measures. A central repository for data from each program managed by BHPr would represent an important part of this web-based archive. Additionally, individuals or groups that access this data need to be provided with opportunities to comment on usefulness and relevance.

Benefits

A centralized source of information will ensure the availability of data and facilitate selection and use of data sets for specific purposes such as evaluation of program effectiveness and impact. This mechanism would more easily identify gaps in the data that indicate a need for additional strategies to collect specific information. Additionally, the existence of an accessible centralized data bank would make it easier to establish "best practices" models from the data. The data should be expressed in easily digestible short statements that are clear and concise and include a global glossary of evaluation and outcomes terms across all applicable Federal agencies.

Recommendation 4: Congress should appropriate funding for the purposes of evaluation, development of educational research models, and tracking long-term outcomes specific to BHPr interdisciplinary grant programs.

Rationale

Evaluation is required by BHPr. The state-of-the-art and the expectations for accurate and meaningful evaluation have evolved to the point that most programs need to hire specialists to design appropriate assessment tools and guide the overall evaluation process. The current size of program awards should be increased to allow for such support without sacrificing the number and quality of programmatic initiatives. The additional funds allocated for this important activity will allow for responses to the evolving expectations and demands of external constituencies regarding program evaluation.

Benefits

This approach would encourage grantees to embrace the importance of evaluation and become more outcome-oriented. Thus, grant-funded programs would have an increased ability to identify and address "gaps" in their programming. Additional types of evaluation could be accomplished including assessments associated with long-term tracking and educational research.