Search tips
Search criteria 


Logo of jgmeLink to Publisher's site
J Grad Med Educ. 2012 March; 4(1): 16–22.
PMCID: PMC3312527

Assessing the Impact of Innovative Training of Family Physicians for the Patient-Centered Medical Home



New approaches to enhance access in primary care necessitate change in the model for residency education.


To describe instrument design, development and testing, and data collection strategies for residency programs, continuity clinics, residents, and program graduates participating in the Preparing the Personal Physician for Practice (P4) project.


We developed and pilot-tested surveys to assess demographic characteristics of residents, clinical and operational features of the continuity clinics and educational programs, and attitudes about and implementation status of Patient Centered Medical Home (PCMH) characteristics. Surveys were administered annually to P4 residency programs since the project started in 2007. Descriptive statistics were used to profile data from the P4 baseline year.


Most P4 residents were non-Hispanic white women (60.7%), married or partnered, attended medical school in the United States and were the first physicians in their families to attend medical school. Nearly 85% of residency continuity clinics were family health centers, and about 8% were federally qualified health centers. The most likely PCMH features in continuity clinics were having an electronic health record and having fully secure remote access available; both of which were found in more than 50% of continuity clinics. Approximately one-half of continuity clinics used the electronic health record for safety projects, and nearly 60% used it for quality-improvement projects.


We created a collaborative evaluation model in all 14 P4 residencies. Successful implementation of new surveys revealed important baseline features of residencies and residents that are pertinent to studying the effects of new training models for the PCMH.

What was known

Efforts are underway to redesign the training of primary care physicians although data on the outcomes are largely lacking to date.

What is new

Fourteen family medicine programs have implemented and sustained the P4 project in a collaborative manner, despite the lack of dedicated funding.


Self-selected sample with an interest in primary care; core measures are mostly survey data and self-reports.

Bottom line

The project has produced data on features of P4 residencies including use of EHRs and remote access. Lacking to date is an assessment of the effectiveness of the P4 program in preparing physicians for practice in the patient-centered medical home.


Primary care is undergoing transformative and rapid change in the United States, including new models of delivery such as the Patient Centered Medical Home (PCMH), and new standards for health information technology.1,8 These innovations are being implemented in the context of busy, complex, primary care health systems.6 In a time of transformation of primary care practices and societal demands for increased accountability, this suggests a need for educational research to study the outcomes of efforts to redesign primary care physician training.

The American Boards of Family Medicine, Internal Medicine, and Pediatrics are interested in directing attention and resources to inspire innovation in residency education.9,12 Studies have shown that evaluation of educational interventions often lack the rigor needed to draw meaningful conclusions.13,18 Studies with a longitudinal database from multiple educational sites with a larger sample of learners have the advantage of being able to detect outcomes over time and to provide greater generalizability than single-site studies.19 The extent to which residents are exposed to features of PCMH and whether their appreciation of those features is related to working with them during residency has not, to our knowledge, been studied to date. We describe the development of standardized core measures that capture longitudinal data at 14 Preparing the Personal Physician for Practice (P4) sites,20 and report baseline findings derived from those measures.


The P4 project (2007–2012) is a comparative case study of 14 residencies that are experimenting with changes in residency education. It seeks to produce physicians capable of effectively practicing in the PCMH model of care and to lead the transformation process to implement this new model. Detailed descriptions of individual programs' innovations, hypotheses, and site-specific measures are reported elsewhere.20,22

Core Measurement Development and Testing

Each program evaluated site-specific hypotheses, using locally selected measures and implementation schedules,22 and participated in a mixed-methods evaluation study using standardized core measures and an online diary system. To identify core measures, members of the P4 Executive Committee and Evaluation Team, located at Oregon Health & Science University, developed a list of process and outcome variables relevant for P4 evaluation, regardless of the innovations being tested. To ease the burden of data collection, the study used existing data sources, such as the Accreditation Council for Graduate Medical Education's Accreditation Data System and the National Resident Match Program. In addition, board scores and in-training exam scores will be supplied by the American Board of Family Medicine to examine changes in clinical knowledge over time.

Survey Development

The assessment of the P4 consists of 4 surveys. A resident survey is completed annually in November and assesses demographic information, the influence of P4 on residency ranking, satisfaction with rotations and faculty participation, how residents provide input into program revision, satisfaction with the quality of the program, and attitudes about the importance of PCMH features. A program survey, completed by program directors in the fall of each year, collects faculty and program characteristics, including financial data and residency match data, and asks the program directors to rate the health of the program. A continuity clinic survey, completed by clinic medical directors, collects information on patient demographics, practice characteristics, and aspects of the PCMH that have or have not been implemented during the project period. Finally, a graduate survey, completed by graduates of P4 programs 18 months after residency graduation, assesses practice characteristics, scope of practice, adequacy of residency training, and presence of PCMH features in graduates' clinical practices.23 All surveys were pilot-tested using cognitive interviewing techniques24 with program directors, medical directors at continuity clinics, and family medicine residents at both the University of Washington and Oregon Health & Science University (neither of which are involved in the P4 program).

The surveys were designed for historical cohort analyses, to allow comparisons of programs, continuity clinics, and residents participating in their respective innovations. The graduate survey will also create cohorts of pre-P4 and post-P4 resident graduates to facilitate the assessment of changes over time. Lastly, the P4 evaluation team launched an online diary system based at the Agency for Health Care Research and Quality that is designed to electronically prompt P4 program faculty, residents, and staff approximately every 6 months, allowing for collection of qualitative data on aspects of, and experiences with, implementing change. Online diary questions from the first 3 years will be reported elsewhere. Descriptive statistics were used to profile data from P4's baseline year, and Spearman correlation coefficient was used to assess baseline relationships between attitudes toward, and exposure to, PCMH features.


The response rates for the core surveys in the baseline year (2007) were 100% (14/14) for the program survey, 100% (24/24) for the continuity clinic survey, and 94% (310/330) for the resident survey. The baseline graduate survey was obtained in 2008 for those who graduated from residency in 2006 with a response rate of 89% (88/99). The survey data from program, resident, and continuity clinic are reported in tables 1, through through3,3, respectively. About 70% of visits done by residents are routine visits (chronic care and health maintenance), whereas about 25% are acute visits. Faculty clinicians, as a group at each continuity clinic, performed on average just more than 9000 visits per year, whereas residents performed nearly 8000 visits per year. In 2009, approximately one-half of the continuity clinics were using the electronic health record (EHR) for safety projects, and nearly 60% (14/24) were using it for clinical quality improvement projects. figures 1 and and22 illustrate the status of implementing electronic and nonelectronic features of the PCMH at the continuity clinics, indicating that the electronic features most likely to be mature (in 50% [12/24] of continuity clinics) were having an EHR and having fully secure remote-access available. The mean number of years an EHR was in place was 3.65 years with a range of 1 to 12 years. Only 42% (10/24) of clinics were using teams to manage patient care. In contrast, integrated behavioral health and integrated case management were present or mature in 71% (17/24) and 54% (13/24) of continuity clinics, respectively.

Status (%) of Electronic Features of the Patient-Centered Medical Home (n  = 24)
Status (%) of Nonelectronic Features of the Patient-Centered Medical Home (n  = 24)
Characteristics of Preparing the Personal Physician for Practice Programs
Preparing the Personal Physician for Practice (P4) Resident Characteristics (n  = 330)
Characteristics of Continuity Clinics (n  =  24 Clinics)

Residents exposed to the P4 innovations in the baseline period were primarily women with similar racial and ethnic diversity as has been reported to exist among current, practicing primary care physicians in the United States.25 Nearly one-third of P4 residents attended medical school outside the United States, and many were born outside the United States.

The predominant type of continuity clinic for P4 residents at baseline was a family health center “model” office; few were in federally qualified health centers. Most patients in all settings were assigned to a personal physician at baseline, which likely indicates the high commitment these programs had for patient-centered health care at the start of the project. Nearly half of the clinics were using an EHR for safety projects, and most were using this technology for clinical quality improvement projects. These high rates at baseline likely reflect the high commitment to PCMH features in these early adopters of innovation. Although a recent study comparing hospitals at various stages in the adoption of computerized health records found little difference in quality of care,26 continued study of how best to implement EHR systems to get the greatest gains in care is needed.


Evaluating innovation in the P4 residencies requires establishing baseline measures about essential features and individuals in the programs and prospectively studying how these key characteristics change over time. The P4 core evaluation instruments have been successfully designed and are currently being used in all 14 P4 residency programs, and we have attained nearly 100% response rates. Using a central evaluation core allowed for development and implementation standards for data exchange and the formation of a successful, residency-redesign collaborative around a common research agenda.

Program directors' global ratings of satisfaction with their programs were lower, relative to their ratings of faculty development programs and the morale of both the faculty and residents. This lower relative level of satisfaction may have motivated the program directors to apply to be a P4 program as an impetus for change. Overall, however, the means for all these ratings indicated there is room for improvement in faculty development and in the morale of faculty and residents. It will be important to see how these measures change over time.

We found that the continuity clinics for residency training varied in implemented features of the PCMH, with less than 50% having a mature EHR and fully secure remote access available, although other electronic features were in place in less than one-half the continuity clinics at baseline. Most of the continuity clinics were using integrated behavioral health and integrated social services to deliver care to their patients, and many were beginning to use a team-based care approach. Managing a multidisciplinary team is a skill physicians need if primary care is to be successful in achieving high quality outcomes.27

Ultimately, evaluation of the P4 approach may be effectively performed by Centers of Excellence in Educational Research. Such a model has been proposed previously19 because it is unlikely that every residency program and undergraduate medical school can afford to support experts in educational design, faculty development, measurement, and analytic/statistical expertise at the local level. Forming regional or national exchanges with a comprehensive longitudinal database from multiple collaborating programs strengthens our ability to answer important medical education research questions.28 As a result of the P4 initiative, we have registered a practice-based research network—PCERN (Primary Care Educational Research Network)—through AHRQ and are hopeful that our efforts will serve as a model for other schools and programs that could provide important, much appreciated expertise.


Fourteen family medicine programs selected to be a part of the P4 project have successfully implemented annual core study measures and achieved high response rates and continue to participate in P4 although they received no funding directly from P4 for their work. Our careful pilot-testing of core instruments resulted in robust interpretable data, as indicated by responses that were well characterized and usable in complex analyses, which they were in our assessment of the correlation between exposure to PCMH features and the residents' ratings of their importance. Most of our core measures are survey based and primarily quantitative. Although an online diary system was implemented to collect supplemental, qualitative data, entries tended to be brief because of respondents' busy schedules, and assessing change processes may be more challenging than we expected.

A collaborative evaluation model was created in all P4 residencies. Successful implementation of new surveys discerned important baseline features of residencies and residents that are pertinent to studying the effects of revised training for the PCMH.


Patricia A. Carney, PhD, is Professor of Family Medicine and of Public Health and Preventive Medicine at Oregon Health & Science University; M. Patrice Eiff, MD, is Professor and Vice Chair of Family Medicine at Oregon Health & Science University; John W. Saultz, MD, is Professor and Chair of Family Medicine at Oregon Health & Science University; Erik Lindbloom, MD, is Associate Professor of Family and Community Medicine at University of Missouri; Elaine Waller, BA, is Research Associate of Family Medicine at Oregon Health & Science University; Samuel Jones, MD, is Program Director of the Fairfax Residency Program at Virginia Commonwealth University and Chair-Elect of the American Board of Family Medicine; Jamie Osborn, MD, is Assistant Professor of Clinical Medicine at Loma Linda University School of Medicine; and Larry Green, MD, is Professor of Family Medicine at University of Colorado.

Funding: This work was supported by the Preparing the Personal Physician for Practice (P4) Project, which is jointly sponsored by the American Board of Family Medicine Foundation and the Association of Family Medicine Residency Directors, and the Family Medicine Research Program at Oregon Health & Science University.


1. Medicine in the Computer Age: The ACP and Medical Informatics, 2008. Accessed January 16, 2012.
2. Schabot MM. Medicine on the Internet, 2001. Accessed January 16, 2012.
3. Perkins FM. Health workforce planning and medical student career choice. JAMA. 2009;301(8):824;825–826. [PubMed]
4. Jiwa M, Carlsen J, Horner B. Healthcare innovation and market forces. Qual Prim Care. 2009;17(1):1–3. [PubMed]
5. Tracy J, Rheuban K, Waters RJ, DeVany M, Whitten P. Critical steps to scaling tele-health for national reform. Telemed J E Health. 2008;14(9):990–994. [PubMed]
6. Kilo CM, Wasson JH. Practice redesign and the patient-centered medical home: history, promises, and challenges. Health Aff (Millwood) 2010;29(5):773–778.2. [PubMed]
7. Friedberg MW, Safran DG, Coltin K, Dresser M, Schneider EC. Paying for performance in primary care: potential impact on practices and disparities. Health Aff (Millwood) 2010;29(5):926–932. [PubMed]
8. Centers for Medicare &Medicaid Services. CMS Proposes Definition of Meaningful Use of Certified Electronic Health Records Technology. Accessed November 29, 2010.
9. Robert Graham Center for Policy Studies in Family Medicine and Primary Care. The Patient Centered Medical Home: History, Seven Core Features, Evidence and Transformational Change, November 2007. Accessed January 16, 2012.
10. Jones MD, Jr, McGuinness GA, First LE, Leslie LK. Residency Review and Redesign in Pediatrics Committee. Linking process to outcome: are we training pediatricians to meet evolving health care needs. Pediatrics. 2009;123(suppl 1):S1–S7. Erratum in Pediatrics. 2009;123(4):1255. [PubMed]
11. Meyers FJ, Weinberger SE, Fitzgibbons JP, Glassroth J, Duffy FD, Clayton CP. Alliance for Academic Internal Medicine Education Redesign Task Force. Redesigning residency training in internal medicine: the consensus report of the Alliance for Academic Internal Medicine Education Redesign Task Force. Acad Med. 2007;82(12):1211–1219. [PubMed]
12. Nasca TJ. The CEO's first column—the next step in the outcomes-based accreditation project. ACGME Bull. May 2008:2–4.
13. Windish DM, Reed DA, Boonyasai RT, Chakraborti C, Bass EB. Methodological rigor of quality improvement curricula for physician trainees: a systematic review and recommendations for change. Acad Med. 2009;84(12):1677–1692. [PubMed]
14. Reed DA, Beckman TJ, Wright SM. An assessment of the methodologic quality of medical education research studies published in the American Journal of Surgery. Am J Surg. 2009;198(3):442–444. [PubMed]
15. Cook DA, Beckman TJ, Bordage G. Quality of reporting of experimental studies in medical education: a systematic review. Med Educ. 2007;41(8):737–745. [PubMed]
16. Wolf FM. Methodological quality, evidence, and Research in Medical Education (RIME) Acad Med. 2004;79(10, suppl):S68–S69. [PubMed]
17. Norman G. RCT  = results confounded and trivial: the perils of grand educational experiments. Med Educ. 2003;37(7):582–584. [PubMed]
18. Baernstein A, Liss H, Carney PA, Elmore JG. Trends in study methods used in undergraduate medical education research, 1969–2007. JAMA. 2007;298(9):1038–1045. [PMC free article] [PubMed]
19. Carney PA, Nierenberg DW, Pipas CF, Brooks WB, Stukel TA, Keller AM. Educational epidemiology: applying population-based design and analytic approaches to study medical education. JAMA. 2004;292(9):1044–1050. [PubMed]
20. Green LA, Jones SM, Fetter G, Jr, Pugno PA. Preparing the personal physician for practice: changing family medicine residency training to enable new model practice. Acad Med. 2007;82(12):1220–1207. [PubMed]
21. TransforMED. Transforming Medical Practices. Accessed October 21, 2010.
22. Carney PA, Eiff PM, Green LA, Lindbloom E, Jones S, Osborn J, et al. Preparing the personal physician for practice (P4): site-specific innovations, hypotheses, and measures at baseline. Fam Med. 2011;43(7):464–471. [PubMed]
23. Eiff M, Garvin R, Fogarty CT, LoPresti L, Young RA, Duane MR, et al. A model for a standardized national family medicine graduate survey. Fam Med. 2009;41(5):337–341. [PubMed]
24. Willis GB. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage Publications; 2005.
25. Komaromy M, Grumbach K, Drake M, Vranizan K, Lurie N, Keane D, et al. The role of black and Hispanic physicians in providing health care for underserved populations. NEJM. 1998;334(20):1305–1310. [PubMed]
26. Linder JA, Ma J, Bates DW, Middleton B, Stafford RS. Electronic health record use and the quality of ambulatory care in the United States. Arch Intern Med. 2007;9;167(13):1400–1405. [PubMed]
27. Bohmer RM. Managing the new primary care: the new skills that will be needed. Health Aff (Millwood) 2010;29(5):1010–1014. [PubMed]
28. Gruppen LD. Improving medical education research. Teach Learn Med. 2007;19(4):331–335. [PubMed]

Articles from Journal of Graduate Medical Education are provided here courtesy of Accreditation Council for Graduate Medical Education