PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jgimedspringer.comThis journalToc AlertsSubmit OnlineOpen Choice
 
J Gen Intern Med. Jan 2009; 24(1): 53–56.
Published online Oct 31, 2008. doi:  10.1007/s11606-008-0826-3
PMCID: PMC2607496

Physician Practice Behavior and Practice Guidelines: Using Unannounced Standardized Patients to Gather Data

Abstract

Background

Measuring actual practice behaviors of physicians, particularly as they relate to established clinical guidelines, is challenging. Standardized patients provide one method of collecting such data.

Objective

To demonstrate the use of unannounced standardized patients in gathering data that may address adherence to guidelines in an office setting.

Design

Unannounced standardized patients (SPs) simulating an initial type 2 diabetic visit presented to community offices of 32 internists as “real” patients to record physicians’ evaluation and management.

Participants

Unannounced SPs presented to the office of 32 internists as “real” patients.

Measurements

Unannounced SPs, simulating type 2 diabetics, completed a standardized assessment sheet, based on ADA guidelines to record physicians’ evaluation and management following an initial visit. Patient charts were also reviewed to determine if evaluation adhered to the guidelines.

Results

Unannounced SPs recorded 56 visits with 32 community internists; all SPs remained undetected. All physicians asked SPs about medications. At least 50% of physicians asked about home blood sugar monitoring, last eye exam, smoking, edema, and told patients to stop smoking. Less than 50% of physicians asked about parasthesias, performed fundoscopy, examined feet, referred the patient to a diabetic educator or ophthalmologist, or gave patients suggestions regarding glucose monitoring or exercise. HbA1c was ordered in 78%, metabolic profiles in 86%, and urinalysis/microalbumin in 41% of patients.

Conclusions

Unannounced standardized patients can successfully collect important data regarding physician practices in community settings. This method may be helpful in assessing physician adherence to established clinical practice guidelines.

KEY WORDS: standardized patients, practice guidelines

INTRODUCTION

The use of standardized patients (SPs) to assess clinical skills and behaviors has become a routine practice in the evaluation of medical students1,2. As lay people trained to simulate patients with real disorders, SPs can present in a consistent, reproducible manner. They can also be trained to effectively document and record participants’ interpersonal and clinical skills following each encounter3,4. The value of SPs in assessing professional performance of practicing physicians has also been demonstrated, but their use in evaluating adherence to guidelines for the management of common clinical problems by direct physician observation is limited510. To assess real practice behaviors of physicians, one would have to insert undetected (unannounced) standardized patients directly into practice settings. Few such reports are available1015, and most have focused on preventive care for otherwise asymptomatic adults. The use of “unannounced” standardized patients provides a unique opportunity to observe physician behavior in adhering to standard clinical practice guidelines for common chronic conditions such as diabetes.

The American Diabetes Association (ADA) publishes clear evidence-based goals for metabolic control and guidelines for patient management16. Yet previous studies have shown that adherence to these guidelines and achievement of metabolic goals in practice are suboptimal1720. Collection of data is generally limited to retrospective chart reviews or insurance claims data. To avoid these limitations, unannounced standardized patients, each representing a “typical” middle-aged male patient with type 2 diabetes, were used to assess physician behaviors of community-based internists as part of an educational research project. This study provides a unique look into the actual practice behaviors and adherence to ADA clinical practice guidelines of internists as they evaluated the first visit of a previously diagnosed uncomplicated type 2 diabetic.

METHODS

As part of an educational research project focused on training students and residents in non-academic, community-based ambulatory settings, internists in the greater New Orleans area were contacted and asked to participate and serve as preceptors. Internists who either expressed an interest or were recommended by peers agreed to enroll as either a participant or served as controls. All preceptors were subsequently assigned students and residents in their offices, while control physicians had no trainees. Sixteen physicians were enrolled over 2 consecutive years. Each year, eight of the internists served as participants and had the same student/resident team assigned to their office 1 afternoon every 1 to 2 weeks, while eight internists served as controls and had no trainees in their office. In year 1 of the project, nine of the physicians were recruited from one large multi-specialty group, two from other multi-specialty practices, and the remaining physicians were in private practices. Because of the logistics in dealing with multiple sites, in year 2, all physicians were recruited from the same large multi-specialty practice that participated in year 1. While approximately half of the recruited physicians were sub-specialists, all their practices included a substantial proportion of general internal medicine.

The study was submitted to the Institutional Review Board, which determined that informed consent was not necessary. However, all participating community physicians did sign a written agreement permitting the placement of SPs into their normal office schedules exactly as if they were “real” patients. To assure there would be no detection of SPs, a pseudo-healthcare plan (HMO) was created, from which all SP-related activity was coordinated. To prevent office personnel from inadvertently disclosing that a patient was an SP or that the HMO did not truly exist, information regarding the SPs was limited to one senior office manager at each practice location. Additionally, a letter was generated for other office personnel that indicated that the pseudo-HMO had been added as an accepted insurer several months in advance of scheduling any standardized patients. Once this information was provided, a coordinator from the standardized patient program began to schedule SP visits with the community physician offices. Each SP was given a pseudo-insurance card that requested that all visits be approved by calling a telephone number that went directly to the research project coordinator. When these patients presented to the physician offices, they registered by showing their pseudo-insurance card, and when approval was requested, the project coordinator, posing as the health-care provider employee, granted it. A post office box was obtained and listed as the HMO billing address for each SP. A flat fee payment was established and agreed to in advance for each SP visit to compensate for the loss of revenue caused by using a patient slot on the physician schedule.

The SPs entered each practice setting in an identical manner to existing patients, complete with the creation of a new patient office chart. Patients were seen and examined by either study or control preceptors in a manner no different than “real” patients. The original study design anticipated being able to evaluate the participating preceptors before and after they had student/resident teams in their offices, but making appointments at appropriately designated times proved logistically very difficult. Therefore, almost all SP visits for both participating and control preceptors were completed without the presence of trainees. If laboratory studies were requested, the registration card specifically requested that patients be sent to a large, commonly used reference laboratory. If other diagnostic studies were requested (radiographs, etc.), SPs asked to complete them at later dates. Similarly, if previous medical records were sought, the SP reported to the physician that he did not have the prior doctor’s name or address. All visits were planned as one-time only visits.

To assess the community physicians’ clinical behaviors, a standardized scenario was created and replicated by each SP at all of the office visits. A total of 15 SPs were trained to present as a middle-age, long-standing, type 2 diabetic male who had recently moved to the area and needed to establish a new health-care provider. Each “patient” was asymptomatic, but gave a history of type 2 diabetes and hypertension. SPs indicated that they smoked, did not monitor glucose, and did not exercise or watch their diets. All patients indicated that they were taking only metoprolol for hypertension and glipizide for their diabetes. All SPs were trained in advance to observe physician history-taking and physical examination behaviors included on a 64-item standardized assessment sheet, based on published clinical practice guidelines. Because the same assessment was originally intended also to evaluate students and included more basic aspects of the exam, not all 64 items were utilized in the final analysis of the practicing physicians. The checklist focused on items demonstrating that physicians sought evidence of preventive care and diabetic end-organ damage. Physicians were credited for any behaviors when they were addressed in any part of the history, or if it was possible that a physical exam maneuver was attempted. For example, physicians were credited with performing fundoscopy whenever an ophthalmoscope was used, while removal of socks was necessary for examination of the feet. Following each visit SPs were de-briefed about their experience by the SP trainer to maximize the completeness for each checklist. Inter-rater reliability of SPs was measured by having non-participating medicine residents complete scenarios, under the observation of the study co-investigators.

Office charges and diagnostic studies were also recorded. Following the completion of the research project, each patient chart was retrieved from the physician offices by the office manager, and the physician records were reviewed to determine what patient education and instructions were given, and what diagnostic studies were ordered. The cost of the visit, based on the Evaluation and Management level of service, and the cost of the diagnostic studies that were ordered were also determined. As a condition of participating, physician offices requested information regarding the systems and personnel utilized in registering new patients based on comprehensive notes on waiting times, attitudes and professionalism of the office and nursing staff observed by the SPs.

Data on all preceptor visits over a 2-year period are included in the present report. Due to complexities in the timing of these SP visits in relation to when physicians worked with a student/resident team, it was not possible, as originally planned, to evaluate the effect of precepting trainees on physician performance. Therefore, all physician data collected were pooled for evaluation. The frequency of physician behaviors, including adherence to published clinical practice guidelines while performing a first visit for a patient with uncomplicated type 2 diabetes, is presented with simple descriptive statistics.

RESULTS

Over 2 years, unannounced SPs scheduled appointments with 32 community internists. There were 56 SP visits, with no SP ever detected as not being a “real” patient. All visits were scheduled when learners were not expected to be present, though in several visits, a student did observe the preceptor. The maximum number of SP visits for any one physician was 2. Neither the presence of a student nor seeing another SP appeared to affect physician behaviors. Tables 1 and and22 present the data summarizing physician performance and the percent of physicians who adhered to the ADA guidelines for a comprehensive evaluation. Recommended features of the history and physical examination, laboratory studies, referrals, and patient education are shown. The items evaluated represent most components of the ADA guidelines on a comprehensive evaluation of a patient with diabetes. Many features of the history or examination were performed by less than half of the physicians, with less than 10% performing a sensory examination of the feet. There was also significant variation both among physicians and from the ADA guidelines for the laboratory studies that were ordered at this “first visit.” This was also demonstrated in the total charges for visits, including laboratory studies, which ranged from $150 to over $800.

Table 1
Features of the History and Physical Examination and Percent of Physicians Performing Recommended ADA Guidelines
Table 2
Laboratory Studies, Referrals, and Patient Education and Percent of Physicians Performing Recommended ADA Guidelines

DISCUSSION

This study demonstrates that unannounced standardized patients can gather useful data regarding physician compliance with published guidelines for care of persons with diabetes. SPs offer important advantages over assessments based on chart review and insurance claims7,9,2123. Chart abstraction is prone to false-negative results due to illegibility, lack of availability of the chart, or recording bias (physicians may not document every aspect of their exam)24. Insurance claim data do not reflect lack of follow through by a patient on essential advice, such as obtaining a complete ophthalmologic evaluation. Claim data also lack sensitivity and do not capture the process elements or details of the physician—patient interaction that are an essential aspect of the visit24. Unannounced SPs address these concerns by capturing the intent of the physician, thereby giving “credit” for all aspects of the encounter.

Though this study was limited by a small number of physicians, our results are consistent with recent reports from chart reviews using larger physician populations. We found that adherence to particular guidelines, such as foot examination, complete ophthalmologic examination, and metabolic control, is poor2527. Another limitation of our study is that physician performance was only assessed in the setting of an initial visit. Physicians may have been unfairly penalized for not performing interventions that they typically would accomplish at a follow-up visit as further data became available.

Accurate evaluation of physician performance is becoming even more critical as pay for performance measures are more widely adopted 28. Many have expressed concern about the use of chart reviews and insurance claims as the only basis for evaluating performance, as they cannot account for documented patient factors, such as socio-economic and life circumstances, and limited health literacy that clearly impact patient outcomes. The ability to accurately capture additional aspects of the patient encounter, such using unannounced SPs, is an important consideration for policy makers seeking valid and reliable methods to evaluate physician performance.

In summary, we found that unannounced standardized patients can effectively evaluate physician practice behaviors and their adherence to practice guidelines in a manner not achieved by more traditional methods. This manuscript provides the details of this methodology so that others can adapt this approach within their own setting.

Acknowledgements

This project was funded in part through an educational research grant from the Quality Care Research Fund, Aetna U.S. Healthcare, and has been presented in abstract form at the Southern Regional Meeting of the Southern Society of Clinical Investigation, February 2002, and the AAMC Annual Meeting, San Francisco, CA, November 2002.

Conflict of Interest None disclosed.

References

1. Colliver JA, Swartz MH. Assessing clinical performance with standardized patients. JAMA. 1997;278:790–4. [PubMed]
2. Adamo G. Simulated and standardized patients in OSCEs: achievements and challenges 1992–1993. Med Teach. 2003;25:262–70. [PubMed]
3. Nagoshi M, Williams S, Kasuya R, Sakai D, Masaki K, Blanchette PL. Using standardized patients to assess the geriatrics medicine skills of medical students, internal medicine residents, and geriatrics medicine fellows. Acad Med. 2004;79:698–702. [PubMed]
4. Epstein RM, Dannefer EF, Nofziger AC, Hansen JT, Schultz SH, Jospe N, et al. Comprehensive assessment of professional competence: the Rochester experiment. Teach Learn Med. 2004;16:186–96. [PubMed]
5. Beullens J, Rethans JJ, Goedhuys J, Buntinx F. The use of standardized patients in research in general practice. Fam Pract. 1997;14:58–62. [PubMed]
6. Tamblyn RM. The use of standardized patients in the assessment of medical practice. Can Med Assoc J. 1998;158:205–7. [PMC free article] [PubMed]
7. Peabody JW, Luck J, Glassman P, Dresselhaus TR, Lee M. Comparison of vignettes, standardized patients, and chart abstraction. A prospective validation study of 3 methods of measuring quality. JAMA. 2000;283:1715–22. [PubMed]
8. Beaulieu MD, Rivard M, Hudon E, Saucier D, Remondin M, Favreau R. Using standardized patients to measure professional performance of physicians. Int JQual Health Care. 2003;15:251–9. [PubMed]
9. Dresselhaus TR, Peabody JW, Lee M, Wang MM, Luck J. Measuring compliance with preventive care guidelines: standardized patients, clinical vignettes, and the medical record. J Gen Int Med. 2000;15:782–8. [PMC free article] [PubMed]
10. Carney PA, Dietrich AJ, Freeman DH Jr, Mott LA. The periodic health examination provided to asymptomatic older women: An assessment using standardized patients. Ann Int Med. 1993;119:129–35. [PubMed]
11. Carney PA, Ward DH.Using unannounced standardized patients to assess the HIV preventive practices of family nurse practitioners and family physicians. Nurse Pract. 23;56–8, 63, 67–8 passim; 1998. [PubMed]
12. Hutchinson B, Woodward CA, Norman GR, Abelson J, Brown JA. Provision of preventive care to unannounced standardized patients. Can Med Assoc J. 1998;158:185–93. [PMC free article] [PubMed]
13. Gorter S, Rethans JJ, van der Heijde D, Scherpbier A, Houben H, van der Vleuten C, et al. Reproducibility of clinical performance assessment in practice using incognito standardized patients. Med Educ. 2002;36:827–32. [PubMed]
14. Franz CE, Epstein R, Miller KN, Brown A, Song J, Feldman M, et al. Caught in the act? Prevalence, predictors, and consequences of physician detection of unannounced standardized patients. Health Serv Res. 2006;41:2290–302. [PMC free article] [PubMed]
15. Srinivasan M, Franks P, Meredith LS, Fiscella K, Epstein RM, Kravitz RL. Connoisseurs of care? Unannounced standardized patients’ ratings of physicians. Med Care. 2006;44:1092–8. [PubMed]
16. Standards of medical care in diabetes-07. Diabetes Care. 2007;30(Suppl 1):S4–41. [PubMed]
17. Saydah SH, Fradkin J, Cowie CC. Poor control of risk factors for vascular disease among adults with previously diagnosed diabetes. JAMA. 2004;291:335–42. [PubMed]
18. Saaddine JB, Engelgau MM, Beckles GL, Gregg EW, Thompson TJ, Narayan KM. A diabetes report card for the United States: quality of care in the 1990s. Ann Intern Med. 2002;136:565–74. [PubMed]
19. Peters AL, Legorreta AP, Ossorio RC, Davidson MB. Quality of outpatient care provided to diabetic patients. A health maintenance organization experience. Diabetes Care. 1996;19:601–6. [PubMed]
20. Suwatee P, Lynch JC, Pendergrass ML. Quality of care for diabetic patients in a large urban public hospital. Diabetes Care. 2003;26:563–8. [PubMed]
21. Gilbert EH, Lowenstein SR, Koziol-Mclain J, Barta DC, Steiner J. Chart reviews in emergency medical research: where are the methods. Ann Emerg Med. 1996;27:305–8. [PubMed]
22. Wu L, Ashton CM. Chart review: a need for reappraisal. Eval Health Prof. 1997;20:146–63. [PubMed]
23. Luck J, Peabody JW, Dresselhaus TR, Lee M, Glassman P. How well does chart abstraction measure quality? A prospective comparison of standardized patients with the medical record. Am J Med. 2000;108:642–9. [PubMed]
24. Garnick DW, Fowles J, Lawthers AG, Weiner JP, Parente ST, Palmer RH. Focus on quality: profiling physicians’ practice patterns. J Ambul Care Manage. 1994;17:44–75. [PubMed]
25. Tan KBH. Assessing Doctors’ Compliance with Guidelines on Diabetes Management. Int J Health Care Qual Assur. 2006;19:267–86. [PubMed]
26. Sequist TD, Gandhi TK, Karson AS, Fiskio J, Bugbee D, et al. A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. J Am Med Inform Assoc. 2005;12:431–7. [PMC free article] [PubMed]
27. Kirkman MS, Williams SR, Caffrey HH, Marrero DG. Impact of a program to improve adherence to diabetes guidelines by primary care physicians. Diabetes Care. 2002;25:1946–51. [PubMed]
28. Gulliford MC, Ashworth M, Robtham D, Mohiddin A. Achievement of metabolic targets for diabetes by English primary care practices under a new system of incentives. Diabet Med. 2007;24:505–11. [PubMed]

Articles from Journal of General Internal Medicine are provided here courtesy of Society of General Internal Medicine