skip header and navigation
US Department of Health and Human ServicesHealth Resources and Services Adminstration
Health Resources and Services Adminstration   Health Resources and Services Administraton   Questions Search
 

Evaluating the Impact of Title VII, Section 747 Programs
5th Annual Report to the Secretary of the U.S. Department of Health and Human Services and to Congress

November 2005

 
Printer-friendly 5th Annual Report
Advisory Committee on Training in Primary Care Medicine and Dentistry

List of Figures

Acknowledgment of Public Comment
Abstract

Executive Summary

Background

Overview of Title VII, Section 747 Programs

Description of Title VII, Section 747 Programs

History of Evaluation of Title VII, Section 747 Programs

Overall Challenges in Evaluating Programs

Evaluation of Title VII, Section 747 Programs

Framework for Evaluating Title VII, Section 747 Programs

Challenges in Evaluating Title VII, Section 747 Programs

Approach for Developing Outcome Measures

Recommended Outcome Measures

Evaluation Methodology

Conclusions

Recommendations
References

Appendices

Appendix A – Key Acronyms

Appendix B – Examples of How to Read the Logic Model

Appendix C – BHPr Conceptual Framework and Core Performance Measures

Appendix D – Description of Measures

 

References

Advisory Committee on Training in Primary Care and Dentistry. (Nov 2003). Training Culturally Competent Primary Care Professionals to Provide High Quality Healthcare for All Americans: The Essential Role of Title VII, Section 747, in the Elimination of Healthcare Disparities. Washington, D.C.: U.S. Department of Health and Human Services.

Advisory Committee on Training in Primary Care and Dentistry. (Nov 2004). Preparing Primary Health Care Providers to Meet America’s Future Health Care Needs: The Critical Role of Title VII, Section 747. Washington, D.C.: U.S. Department of Health and Human Services, Washington, D.C.

Anderson WA, Stritter FT, Mygdal WK, Arndt JE, Reid A. (Mar 1997). Outcomes of three part-time faculty development fellowship programs. Fam Med: 29(3):204-208.

Arnold EL, Blank LL, Race KEH, Cipparrone N. (1998). Can professionalism be measured? The development of a scale for use in the medical environment. Acad Med: 73:1119-1121.

Association of American Medical Colleges. (2000). The Education of Medical Students: Ten Stories of Curricular Change. Milbank Memorial Fund.

Atchison, KA, Mito RS, Rosenberg, DJ, Lefever KH, Lin S, Englehardt R. (2002). PGD training and its impact on general dentistry practice patterns. J Dent Educ: 66:1348-1457.

Baumgartner L, Stenersen S, Campbell K. (1996). Overcoming barriers to evaluating federal health education and training programs. Eval of Health Professions: 19(3):293-310.

Bordage G, Burack JH, Irby DM, Stritter FT. (Jul 1998). Education in ambulatory settings: developing valid measures of educational outcomes, and other research priorities. Acad Med: 73(7):743-750.

Bowen JL, Stearns JA, Dohner C, Blackman J, -Simpson D. (Jun 1997). Defining and evaluating quality for ambulatory care educational programs. Acad Med: 72(6):506-510.

Brasel KJ, Bragg D, Simpson DE, Weigelt JA. (Jul 2004). Meeting the Accreditation Council for Graduate Medical Education competencies using established residency training program assessment tools. Am J Surg: 188(1):9-12.

Brook RH, Fink A, Kosecoff J, Linn LS, Watson WE, Davies AR, Clark VA, Kamberg C, Delbanco TL (Sep 1987). Educating physicians and treating patients in the ambulatory setting. Where are we going and how will we know when we arrive? Ann Intern Med: 107(3):392-398.

Campos-Outcalt D, Senf J, Kutob R. (Apr 2004). A comparison of primary care graduates from schools with increasing production of family physicians to those from schools with decreasing production. Fam Med: 36(4):260-264.

Cantor JC, Miles EL, Baker LC, Baker DC. (1996). Physician service to the underserved: implications for affirmative action in medical education. Inquiry: 33:167-180.

Chan L, Hart LG, Ricketts TC, Beaver SK. (2004). An analysis of Medicare’s incentive payment program for physicians in health professional shortage areas. J Rural Health: 20(20):109-117.

Chen FM, Bacuhner H, Burstin, H. (2004). A call for outcomes research in medical education. Acad Med: 79:955-960.

Council on Graduate Medical Education. (1998). Twelfth Report: Minorities in Medicine. Washington, D.C.: U.S. Department of Health and Human Services.

Davidson RA, Vega A, Edwards KI, Caranasos GJ. (Nov 1996). Introducing ambulatory care during the medicine clerkship: impact on evaluation and attitudes. Am J Med Sci: 312(5):206-213.

Doescher MP, Ellsbury KE, Hart LG. (Spring 2000). The distribution of rural female generalist physicians in the United States. J Rural Health: 16(2):111-118.

Dorsey ER, Jarjoura D, Rutecki G. (Sep 2003). Influence of controllable lifestyle on recent trends in specialty choice by US medical students. JAMA: 290:1173-1178.

Easterbrook M, Godwin M, Wilson R, Hodgetts G, Brown G, Pong R, Najgebauer E. (Apr 20, 1999). Rural background and clinical rural rotations during medical training: effect on practice location. CMAJ: 160(8):1159-1163.

Edelstein B, Krol D, Ingargiola P, De Biasi A. (May 2003). Assessing Pediatric Dentistry Title VII: Training Program Success. Children’s Dental Health Project final report to the American Academy of Pediatric Dentistry Foundation.

Ellsbury KE, Baldwin LM, Johnson KE, Runyan SJ, Hart LG. (Sep-Oct 2002). Gender-related factors in the recruitment of physicians to the rural Northwest. J Am Board Fam Pract: 15(5):391-400.

Ellsbury KE, Doescher MP, Hart LG. (May 2000). US medical schools and the rural family physician gender gap. Fam Med: 32(5):331-337.

Elnicki DM, Halbritter KA, Antonelli MA, Linger B. (Jun 1999). Educational and career outcomes of an internal medicine preceptorship for first-year medical students. J Gen Intern Med: 14(6):341-346.

Evans AT, Rogers LQ, Peden JG Jr, Seelig CB, Layne RD, Levine MA, Levin ML, Grossman RS, Darden PM, Jackson SM, Ammerman AS, Settle MB, Stritter FT, Fletcher SW. (Jul-Aug 1996). Teaching dietary counseling skills to residents: patient and physician outcomes. The CADRE Study Group. Am J Prev Med: 12(4):259-265.

Fink KS, Phillips RL Jr, Fryer GE, Koehn N. (Mar-Apr 2003). International medical graduates and the primary care workforce for rural underserved areas. Health Aff (Millwood): 22(2):255-262.

Fryer GE, Meyers DS, Krol DM, Phillips RL, Green LA, Dovey SM, Miyoshi TJ. (Jun 2002). The association of Title VII funding to departments of family medicine with choice of physician specialty and practice location. Fam Med: 34(6):436-440.

Fryer GE Jr, Stine C, Vojir C, Miller M. (Feb 1997). Predictors and profiles of rural versus urban family practice. Fam Med: 29(2):115-118.

General Accounting Office. (Jul 1994). Health Professions Education: Role of Title VII, Section 747/Title VIII Programs in Improving Access to Care is Unclear. HEHS-94-164. Washington, D.C.: Government Printing Office.

General Accounting Office. (Oct 1994). Medical Education: Curriculum and Financing Strategies Needed to Encourage Primary Care Training. HEHS-95-9. Washington, D.C.: Government Printing Office.

Geyman JP, Hart LG, Norris TE, Coombs JB, Lishner DM. (Winter 2000). Educating generalist physicians for rural practice: how are we doing? J Rural Health: 16(1):56-80.

Ginsberg PB. (Feb 2003). Payment and the future of primary care. Ann Intern Med: 138(3):233-234.

Grumbach K, Coffman J, Munoz C, Rosenoff E, -Gandara P, Sepulveda E. (2002). Strategies for Improving the Diversity of the Health Professions.Final Report. San Francisco: Center for California Health Workforce Studies.

Haist SA, Griffith CH, Hoellein AR, Talente G, -Montgomery T, Wilson JF. (2004). Improving students’ sexual history inquiry and HIV counseling with an interactive workshop using standardized patients. J Gen Intern Med: 19(5):549-553.

Hart, LG. (2001). “Health Care Workforce Supply in Underserved Rural Areas of the United States.” In Australian Medical Workforce Advisory Committee, Commonwealth Department of Health and Aged Care, 5th International Medical Workforce Conference Papers, November 2000. Sydney, Australia: Australian Medical Workforce Advisory Committee, 391-424.

Heidenreich C, Lye P, Simpson D, Lourich M. (Jan 2000). The search for effective and efficient ambulatory teaching methods through the literature. Pediatrics: 105(1 Pt 3):231-237.

Holmboe ES, Hawkins RE. (1998). Methods for evaluating the clinical competence of residents in internal medicine: a review. Ann Intern Med: 129:42-48.

Irby DM. (Oct 1995). Teaching and learning in ambulatory care settings: a thematic review of the literature. Acad Med: 70(10):898-931.

Irby D, Rakestraw P. (Mar 1981). Evaluating clinical teaching in medicine. J Med Educ: 56(3):181-186.

Keitz SA, Holland GJ, Melander EH, Bosworth HB, Pincus SH. (2003). The Veterans Affairs Learners’ Perceptions Survey: The Foundation for Educational Quality Improvement. Acad Med: 78:910-917.

Kohrs FP, Mainous AG 3rd, Fernandez ES, Matheny SC. (Feb 2001). Family medicine faculty development fellowships and the medically underserved. Fam Med: 33(2):124-127.

Kohrs FP, Mainous AG 3rd. (Jan 1999). Retention of family medicine faculty development fellows in academic medicine. Fam Med: 31(1):23-27.

Komaromy M, Grumbach K, Drake M, Vranizan K, Lurie N, Keane D, Bindman AB. (May 16, 1996). The role of black and Hispanic physicians in providing health care for underserved populations. N Engl J Med: 334(20):1305-1310.

Krebs NF, Jacobson MS, American Academy of Pediatrics Committee on Nutrition. (Aug 2003). Prevention of pediatric overweight and obesity. Policy Statement. Pediatrics: 112(2):424-430.

Kurth RJ, Irigoyen MM, Schmidt HJ, Li S. (Jun 2000). The effects of patient characteristics and practice settings on students’ participation in a primary care clerkship. Acad Med: 75(6):634-638.

Larson EB, Roberts KB, Grumback K. (Apr 2005). Primary care, generalism, public good: déjà vu? again! Ann Intern Med: 142(8):671-674.

Lawrence SL, Lindemann JC, Gottlieb M. (Jun 1999). What students value: learning outcomes in a required third-year ambulatory primary care clerkship. Acad Med: 74(6):715-717.

Lishner DL, Rosenblatt RA, Baldwin L-M, Hart LG. (2000). Emergency department use by the rural elderly. J Emerg Med: 18(3): 289-297.

Lurie JD, Goodman DC, Wennberg JE. (Mar-Apr 2002). Benchmarking the future generalist workforce. Eff Clin Pract: 5(2):58-66.

Macinko J, Starfield B, Shi L. (2003). The contribution of primary care systems to health outcomes within organization for Economic Cooperation and Development (OECD) countries, 1970-1998. Health Serv Res 38(3):831-865.

Meyers D, Fryer GE, Krol D, Phillips RL, Green LA, Dovey SM. (2002). Title VII funding is associated with more family physicians and more physicians serving the underserved. Am Fam Physician: 66:554.

Muerer LN. (May 1995). Influence of medical school curriculum on primary care specialty choice: analysis and synthesis of the literature. Acad Med: 70(5): 388-397.

Norcini JJ, Blank LL, Arnold GK, Kimball HR. (Nov 15, 1995). The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med: 123(10):795-799.

Norris TE, Acosta DA. (Jun 1997). A fellowship in rural family medicine: program development and outcomes. Fam Med: 29(6):414-420.

Olney C. (1995). Validation of a scale to measure development of social responsibility. Michigan Journal of Community Service Learning: 2145-2153.

Organisation for Economic Co-Operation and Development. (2003). Health at a Glance: OECD Indicators. France: OECD Publications.

Papadakis MA, Hodgson CS, Teherani A, Kohatsu ND. (2004). Unprofessional behavior in medical school is associated with subsequent disciplinary action by a State Medical Board. Acad Med: 79:244-249.

Pathman DE, Konrad TR, Dann R, Koch G. (Oct 2004). Retention of primary care physicians in rural health professional shortage areas. Am J Public Health: 94(10):1723-1729.

Pathman DE, Steiner BD, Jones BD, Konrad TR. (Jul 1999). Preparing and retaining rural physicians through medical education. Acad Med: 74(7):810-820.

Pathman DE, Williams ES, Konrad TR. (Fall 1996). Rural physician satisfaction: its sources and relationship to retention. J Rural Health: 12(5):366-377.

Peabody JW, Luck J, Glassman P, Jain S, Hansen J, Spell M, Lee M. (2004). Measuring the quality of physician practice by using clinical vignettes: a prospective validation study. Ann Intern Med: 141(10): 771-780.

Pena Dolhun E, Munoz C, Grumbach K. (Jun 2003). Cross-cultural education in U.S. medical schools: development of an assessment tool. Acad Med: 78(6):615-622.

Peters AS, Greenberger-Rosovsky R, Crowder C, Block SD, Moore GT. (2000). Long-term outcomes of the New Pathway Program at Harvard Medical School: a randomized controlled trial. Acad Med: 75:470-479.

Petersdorf RG. (Mar 1993). Commentary: primary care—medical students’ unpopular choice. Am J Public Health: 83(3):328-330.

Phillips RL Jr, Starfield B. (Aug 1, 2004). Why does a U.S. primary care physician workforce crisis matter? Am Fam Physician: 70(3):440, 442, 445-446.

Politzer RM, Hardwick KS, Cultice JM, Bazell C. (Winter 1999). Eliminating primary care health professional shortage areas: the impact of Title VII generalist physician education. J Rural Health: 15(1):11-20.

Politzer RM, Horab S, Fernandez E, Gamliel S, Kahn N, Mullan F. (Nov-Dec 1997). The impact of Title VII departmental and predoctoral support on the production of generalist physicians in private medical schools. Arch Fam Med: 6(6):531-535.

Prislin MD, Fitzpatrick C, Giglio M, Lie D, Radecki S. (Oct 1998). Initial experience with a multi-station objective structured teaching skills evaluation. Acad Med: 73(10):1116-1118.

Pugno P, Schmittling GT, McPherson DS, Kahn NB Jr. (Sep 2000). Entry of US medical school graduates into family practice residencies: 1999-2000 and 3-year summary. Fam Med: 32(8):534-542.

Rabinowitz HK, Babbott D, Bastacky S, Pascoe JM, Patel KK, Pye KL, Rodak J, Veit KJ, Wood DL. (2001). Innovative approaches to educating medical students for practice in a changing health care environment: The National UME-21 Project. Acad Med: 76: 587-597.

Rabinowitz HK, Diamond JJ, Hojat M, Hazelwood CE. (Spring 1999). Demographic, educational and economic factors related to recruitment and retention of physicians in rural Pennsylvania. J Rural Health: 15(2):212-218.

Rabinowitz HK, Paynter NP. (Summer 2000). The role of the medical school in rural graduate medical education: pipeline or control valve? J Rural Health: 16(3):249-253.

Ricketts TC, Hart LG, Pirani M. (Summer 2000). How many rural doctors do we have? J Rural Health: 16(3):198-207.

Rosenblatt RA, Saunders G, Shreffler J, Pirani MJ, Larson EH, Hart LG. (Jan-Feb 1996). Beyond retention: National Health Service Corps participation and subsequent practice locations of a cohort of rural family physicians. J Am Board Fam Pract: 9(1):23-30.

Rosenblatt RA, Schneeweiss R, Hart LG, Casey S, Andrilla CH, Chen FM. (2002). Family medicine training in rural America. JAMA: 288(9):1063-1064.

Rosenblatt RA, Whitcomb ME, Cullen TJ, Lishner DM, Hart LG. (Mar 1993). The effect of federal grants on medical schools’ production of primary care physicians. Am J Public Health: 83(3):322-328.

Rosenthal MP, Rabinowitz HK, Diamond JJ, Markham FW Jr. (Mar 1996). Medical students’ specialty choice and the need for primary care: our future. Prim Care: 23(1):155-167.

Rosenthal TC. (Summer 2000). Outcomes of rural training tracks: a review. J Rural Health: 16(3):213-216.

Schiller J. (Oct 2004). Update on Bureau Level Perform-ance Measures. Meeting of Advisory Committee on Training in Primary Care Medicine and Dentistry. Gaithersburg, Maryland.

Schneeweiss R, Rosenblatt RA, Dovey S, Hart LG, Chen FM, Casey S, Fryer GE Jr. (Feb 2003). The effects of the 1997 Balanced Budget Act on family practice residency training programs. Fam Med: 35(2):93-99.

Shi L, Samuels ME, Cochran CR, Glover S, Singh DA. (1998). Physician practice characteristics and satisfaction: a rural-urban comparison of medical directors at U.S. community and migrant health centers. J Rural Health: 14(4):346-356.

Starfield B. (1992). Primary Care: Concept, Evaluation, and Policy. New York: Oxford University Press.

Starfield B, Simpson L. (1993). Primary care as part of US health services report. JAMA: 269:3136-3139.

Swing SR. (Nov 2002). Assessing the ACGME general competencies: general considerations and assessment methods. Acad Emerg Med: 9(11):1278-1288.

Taylor JS, Friedman RH, Speckman JL, Ash AS, Moskowitz MA, Carr PL. (Apr 2001). Fellowship training and career outcomes for primary care physician-faculty. Acad Med: 76(4):366-372.

Thomas PA, Gebo KA, Hellmann DB. (1999). A pilot study of peer review in residency training. J Gen Intern Med: 14:551-554.

Veloski, JJ, Barzansky B. (2004). Evaluation of the UME-21 Initiative at 18 medical schools between 1999 and 2001. Fam Med: 36, Jan Suppl:S138-S145.

Wax PM, Donovan JW. (2000). Fellowship training in medical toxicology: characteristics, perceptions, and career impact. J Toxicol Clin Toxicol: 38(6):637-642, discussion 643-644.

Whitcomb ME, Cohen JJ. (Aug 12, 2004). The future of primary care medicine. N Engl J Med: 351(7):710-712.

Whitcomb ME, Cullen TJ, Hart LG, Lishner DM, Rosenblatt RA. (Sep 1992). Comparing the characteristics of schools that produce high percentages and low percentages of primary care physicians. Acad Med: 67(9):587-591.

Whiteside C, Mathias R. (Jun 1996). Training for rural practice. Are graduates of a University of British Columbia program well prepared? Can Fam Physician: 42:1113-1121.

Whiteside C, Pope A, Mathias R. (Aug 1997). Identifying the need for curriculum change: when a rural training program needs reform. Can Fam Physician: 43:1390-1394.

W.K. Kellogg Foundation. (2001). Logic Model Development Guide. Battle Creek, Michigan: W.K. Kellogg Foundation.

Wong MD, Hollenberg JP, Charlson ME. (1999). A comparison of clinical performance of primary care and traditional internal medicine residents. Medical Care: 37(8):773-784.

Xu G, Fields SK, Laine C, Veloski JJ, Barzansky B, Martini CJ. (May 1997). The relationship between the race/ethnicity of generalist physicians and their care for underserved populations. Am J Public Health: 87(5):817-822.

Zayas LE, James PA, Shipengrover JA, Schwartz DG, Osborne JW, Graham RP. (Oct 1999). Exploring instructional quality indicators in ambulatory medical settings: an ethnographic approach. Fam Med: 31(9):635-640.