The study was submitted to the Institutional Review Board, which determined that informed consent was not necessary. However, all participating community physicians did sign a written agreement permitting the placement of SPs into their normal office schedules exactly as if they were “real” patients. To assure there would be no detection of SPs, a pseudo-healthcare plan (HMO) was created, from which all SP-related activity was coordinated. To prevent office personnel from inadvertently disclosing that a patient was an SP or that the HMO did not truly exist, information regarding the SPs was limited to one senior office manager at each practice location. Additionally, a letter was generated for other office personnel that indicated that the pseudo-HMO had been added as an accepted insurer several months in advance of scheduling any standardized patients. Once this information was provided, a coordinator from the standardized patient program began to schedule SP visits with the community physician offices. Each SP was given a pseudo-insurance card that requested that all visits be approved by calling a telephone number that went directly to the research project coordinator. When these patients presented to the physician offices, they registered by showing their pseudo-insurance card, and when approval was requested, the project coordinator, posing as the health-care provider employee, granted it. A post office box was obtained and listed as the HMO billing address for each SP. A flat fee payment was established and agreed to in advance for each SP visit to compensate for the loss of revenue caused by using a patient slot on the physician schedule.

The SPs entered each practice setting in an identical manner to existing patients, complete with the creation of a new patient office chart. Patients were seen and examined by either study or control preceptors in a manner no different than “real” patients. The original study design anticipated being able to evaluate the participating preceptors before and after they had student/resident teams in their offices, but making appointments at appropriately designated times proved logistically very difficult. Therefore, almost all SP visits for both participating and control preceptors were completed without the presence of trainees. If laboratory studies were requested, the registration card specifically requested that patients be sent to a large, commonly used reference laboratory. If other diagnostic studies were requested (radiographs, etc.), SPs asked to complete them at later dates. Similarly, if previous medical records were sought, the SP reported to the physician that he did not have the prior doctor’s name or address. All visits were planned as one-time only visits.

To assess the community physicians’ clinical behaviors, a standardized scenario was created and replicated by each SP at all of the office visits. A total of 15 SPs were trained to present as a middle-age, long-standing, type 2 diabetic male who had recently moved to the area and needed to establish a new health-care provider. Each “patient” was asymptomatic, but gave a history of type 2 diabetes and hypertension. SPs indicated that they smoked, did not monitor glucose, and did not exercise or watch their diets. All patients indicated that they were taking only metoprolol for hypertension and glipizide for their diabetes. All SPs were trained in advance to observe physician history-taking and physical examination behaviors included on a 64-item standardized assessment sheet, based on published clinical practice guidelines. Because the same assessment was originally intended also to evaluate students and included more basic aspects of the exam, not all 64 items were utilized in the final analysis of the practicing physicians. The checklist focused on items demonstrating that physicians sought evidence of preventive care and diabetic end-organ damage. Physicians were credited for any behaviors when they were addressed in any part of the history, or if it was possible that a physical exam maneuver was attempted. For example, physicians were credited with performing fundoscopy whenever an ophthalmoscope was used, while removal of socks was necessary for examination of the feet. Following each visit SPs were de-briefed about their experience by the SP trainer to maximize the completeness for each checklist. Inter-rater reliability of SPs was measured by having non-participating medicine residents complete scenarios, under the observation of the study co-investigators.

Office charges and diagnostic studies were also recorded. Following the completion of the research project, each patient chart was retrieved from the physician offices by the office manager, and the physician records were reviewed to determine what patient education and instructions were given, and what diagnostic studies were ordered. The cost of the visit, based on the Evaluation and Management level of service, and the cost of the diagnostic studies that were ordered were also determined. As a condition of participating, physician offices requested information regarding the systems and personnel utilized in registering new patients based on comprehensive notes on waiting times, attitudes and professionalism of the office and nursing staff observed by the SPs.

Though this study was limited by a small number of physicians, our results are consistent with recent reports from chart reviews using larger physician populations. We found that adherence to particular guidelines, such as foot examination, complete ophthalmologic examination, and metabolic control, is poor^25–27. Another limitation of our study is that physician performance was only assessed in the setting of an initial visit. Physicians may have been unfairly penalized for not performing interventions that they typically would accomplish at a follow-up visit as further data became available.

Accurate evaluation of physician performance is becoming even more critical as pay for performance measures are more widely adopted ²⁸. Many have expressed concern about the use of chart reviews and insurance claims as the only basis for evaluating performance, as they cannot account for documented patient factors, such as socio-economic and life circumstances, and limited health literacy that clearly impact patient outcomes. The ability to accurately capture additional aspects of the patient encounter, such using unannounced SPs, is an important consideration for policy makers seeking valid and reliable methods to evaluate physician performance.