2.1 Introduction to USP Methodology
One of the major advantages of the USP methodology is the ability to introduce a simulated, standardized patient to enable difficult to capture clinical encounters and minimize response bias through the creation of a scheduled observable encounter. This methodology protects against bias because the USP presents as an actual patient. Because of these advantages, USPs are increasingly used to assess physician performance under actual practice conditions, which may differ from performance observed in structured examination conditions where physicians know they are being observed. To implement a USP protocol, physicians consent in advance to visits by USP(s) but do not know when a USP clinical encounter will occur. Recent communication studies using USPs have ranged from cancer to asthma patients and have focused on interviewing techniques, self-efficacy, and physician self-disclosure (23
In communication studies involving USPs, the most important threats to validity are detection of the USP and lack of role fidelity (31
). Physician detection of USPs has been found to range from 1–70% (33
). Luck (2003) carried out a validity study of USPs and reported successfully training 45 actors to represent a single USP role (34
). Other studies have reported similar results (35
). In general, detection rates average from 5–20% (7
Factors associated with USP detection are not well understood. Franz and colleagues (2006) report that type of practice predicting USP detection (detection least common in HMO settings) (7
). SP training, contextual, geographic, and cultural factors may also play an important role (7
). Careful USP preparation and advanced logistical management with the office manager is key to avoid detection. Three papers, the most recent published in 2001, have detailed strategies to introduce USPs into practices without detection (36
). However, with the advent of electronic records and instant access between patients insurance carriers and medical practices, the use of USPs has become more complex than what is currently reported in the literature. Today, there are many new challenges facing the introduction of unannounced standardized patients into physician practices. Innovative, creative, and often situation-specific solutions are needed to overcome these and other barriers.
Below we describe implementation of the USP methodology and provide an example of an on-going study of physician-patient communication using USPs to illustrate the methodology. We explain the steps taken to prevent detection, identify the major hurdles encountered during the study’s pre-implementation phase, and propose viable strategies to conquer the obstacles encountered when introducing USPs into practices. We also review how technological advances have changed the conduct of USP research, simultaneously making it easier and more challenging. The data presented is provided for illustrative purposes only and provides and example of application of strategies for implementing a USP study. presents a summary of the challenges and potential solutions to conducting a USP study.
Summary of USP logistical challenges
2.2 USP study methodology
To highlight the challenges and solutions to USP methodology, we present the challenges of a study that is examining physician-patient communication during a new primary care visit. This example simulates a 47-year old patient who presents a common set of symptoms to a primary care physician (general internist or family practitioner). The specific objective of the study is to understand how and whether physician-patient communications are affected by patient race and gender. USPs who vary by race and gender are trained to deliver the role in a standardized manner and to minimize outward expressions of racial or gender stereotypes. Each participating physician sees at least two USPs, each of a different gender and race (e.g., Caucasian male and African American female or African American male and White female), and each visit is recorded using a concealed recorder. USPs are scheduled as new patient visits and the practice is reimbursed for each visit. As an added incentive, physicians are offered the opportunity to receive feedback on how their communication style compares to that of their peers. Physicians are blinded to the set of symptoms that the USP will present and are blinded as to when the visit will occur. The first USP visit is scheduled at least four months after consent and the second visit occurs approximately three months later to avoid recognition of the patient case presentation. The resultant audiorecordings are evaluated by an independent rater to assess role fidelity and coded to examine the nature of the communication process. The medical chart is obtained to assess differential diagnosis and treatment/diagnostic recommendations.
2.3 USP Selection
USPs can be recruited from pools of actors or individuals who are professional standardized patients. The advantage of the latter is that these individuals regularly simulate patients, are familiar with the medical environment, and are usually comfortable with physical exams and how to simulate an illness condition. In our example study, USPs were selected from a pool of experienced SPs who work for a simulation program in a School of Medicine. Age, gender, and race were primary criteria, along with ability to perform the role. The number of USPs trained was based on the number of visits required by the study. Selecting at least two USPs of each gender-race type was necessary to ensure adequate availability of any given USP gender-race type and ensure consistency in role performance should any one USP leave the study unexpectedly. It was also important to choose individuals within a range of normal body mass index who were free of common chronic diseases such as high blood pressure to provide the physician with an accurate physical presentation free from introducing confounders into the communication surrounding the chief medical complaint.
2.4 Training of selected USPs
Adequate training of USPs is critical for a USP study’s success. Even when using seasoned USPs, a 3-day formal training period followed by practice and role refinement, should be budgeted at minimum. USPs participated in a three-day training program with two half-day follow-up trainings before going into the field. In addition, on-going monitoring is used to provide the SPs with reminders, tips and individual remediation as needed. The training was led by a standardized patient trainer with over 30 years experience in SP clinical skills teaching (GG). Each USP was trained to begin the encounter with a mandatory opening line, repeated verbatim, unless interrupted by the physician. Details were scripted regarding presenting illness history, associated symptoms (e.g., frequency, duration, intensity), and medical and social histories. USPs were trained to provide reasonable explanations for any unplanned abnormal physical findings. For example, a higher than normal blood pressure reading is explained as “white coat hypertension” or being rushed. In addition to the mandatory opening line, the USPs were trained to attach “feeling” words like “annoyed,” “concerned,” and “worried” to specific symptoms presented. They were also trained to give excuses (such as the need to return to work) to avoid immediate diagnostic tests (e.g., blood work).
One of the most difficult things to standardize in research with USPs is appropriate responses to physician prompts. USPs were trained to follow the physician’s lead and communication style. For example, USPs are required to respond to open-ended questions with open-ended responses and close-ended questions with one-word responses. Similarly, if a physician interrupts the USP or is discourteous, information is not divulged unless an appropriate question is re-directed by the physician. Verbal and non-verbal responses to the physical examination also require standardization, including response to palpation. Specific care was taken to standardize USPs non-verbal role presentation, including a discussion of dress, eye contact, posture, rate of speech, and expression of concern about problem. Following training, practiced encounters continued and a refresher session was held. After each visit, an experienced independent observer (GG) reviews the audiorecording and completes a role consistency form to give the USP feedback on performance, including the USP’s ability to provide accurate information, affect, and believability.
2.5 Creating an identity and detection prevention
Some USP studies are conducted within a small area while others require travel throughout a large region. In the study example, to maintain the identity and prevent detection, the USP is an individual who has moved to the area recently to take a new job. This allowed the USP to play an individual not native to the region. This can be especially advantageous when introducing USPs into rural practices and where regional speech and accents are especially pronounced.
Technological advances now allow researchers to easily generate logistical information about a USP. For example, this study’s USPs retain their first names but adopt last names randomly selected by an online name generator (e.g., http://www.fakenamegenerator.com/
). Because practices often require patients to present photo identification, the USPs are provided a photo identification card. Websites are available to design and create work identification cards and other forms of identification (e.g., http://www.easyidcard.com/
USP addresses are chosen using on-line mapping. Care is taken to ensure that no addresses are in actual use. Any addresses that belong to a residence are excluded. Using website satellite mapping technology, researchers and USPs have access to street-level imagery of the area surrounding the practice. This is particularly helpful in rural areas, where “small talk” with the physician may center on town landmarks. To further circumvent detection, we use another online product that enables us to assign USPs local working phone numbers for each visit with the correct area codes, without buying additional phone lines.
2.6 Responding to practice or physician phone calls to USP
Practices often telephone USPs to remind them of appointments, follow-up on a referral made by the physician, and/or ask billing questions. To deal with this practice, the example study uses phone numbers generated to mask the USP’s real identity. These numbers forward to a generic voicemail at the research office. When a voicemail is received from the physician’s office, it is electronically captured by an email that is sent to the email address associated with the account (typically the research coordinator). All voicemails can be easily accessed online using an embedded player in the email. For research staff, having access to messages from all SP telephone numbers through one email account facilitates the prompt handling of logistical issues that may arise throughout the pre- and post-visit process.
2.7 Obtaining audio-recordings of the visit while undercover
Technology has improved in recent years that facilitates good quality recordings using concealed digital voice recorders that are small enough to be concealed in a jacket pocket or purse, while still delivering high quality sound recordings. Programs are available to eliminate background noise from the recording post-production. Spy-type digital recording equipment that simulate everyday objects (e.g., pens, tie clips, credit cards.) is also available, but of poorer quality. In this study, a pen recorder is used as a back-up recording device in case of primary recorder failure. USPs are carefully trained in the use of both devices and the need to obtain audio files of the encounter for analysis of the communication process.
Despite these benefits, technological development over the past decade have also made conducting USP research more difficult. Electronic registration systems and electronic medical records (EMRs) present significant logistical challenges. The American Recovery and Reinvestment Act calls for EMRs for all patients by 2014 (see www.recovery.gov
), which will make this technology even more ubiquitous. Successful USP research in the United States will undoubtedly require an understanding of these systems and collaboration with key players to avoid USP detection outside of the clinical encounter itself.
2.8 Electronic registration system and insurance issues
Electronic registration systems, linking multiple private practices or many practices within a hospital system, place strict requirements on patient information entered to avoid record duplication. If data in specific record fields, such as name, or social security number are the same, the probability of detection rises. Therefore, in our example, an individual USP requires many separate identities as visits made within the same EMR/registration system. Specific care is taken in the selection of social security numbers that are not used by any US citizen (see http://ssa-custhelp.ssa.gov/app/answers/detail/a_id/425
). Use of these numbers ensure that USPs are not “sharing” identities with any real individuals. This approach ensures that the new registration file created for each USP individual visit is unique and lowers the chance of detection due to information overlap.
Many electronic systems have an automatic operation that verifies insurance eligibility and the validity of insurance information provided at the time of the visit. While other studies have used facsimile insurance ID cards, that approach is no longer possible because of the ability to instantly check these cards. To avoid this problem, the USPs in this study present as self-pay patients. USPs pay for office visits in cash, thus avoiding the logistical issues surrounding insurance verification, and claims processing. Practices are reimbursed for each new patient visit according to the amount set by major insurance carriers. The office managers are consulted about the amount the USP will be charged at the time of the visit so that the USP can bring enough money to pay cash for the visit. In countries with centralized health insurance systems, other arrangements will need to be made.
In addition, not all practices accept self-pay patients or cash, opting instead to bill or set them up for financial counseling. In this instance, research staff may need to meet with the health system’s financial services staff to establish special billing procedures. For this study, research staff and financial staff created group billing accounts within certain hospital systems. These special billing accounts were created under the names of two fictitious manufacturing companies where the USPs identified they were employed. When USPs check-in at practices of physicians affiliated with this billing system, they provide the administration staff with a letter from the manufacturing company outlining the agreement between the company and hospital billing; this letter includes the account number for billing. The two special accounts link directly to the department running the study and any charges incurred during the visit are billed directly to the study’s funding source. This approach may also be useful for centralized national or provincial health insurance systems.
2.9 Updating, inactivating, or deleting electronic medical records
EMRs generally include a section for the physician to order follow-up tests or specialist visits. In our example, an EMR systems expert was identified to monitor the USP charts for orders and visits beyond the initial visit and cancel them. This strategy alleviates the potential for a USP to occupy a specialist appointment time reserved for real patients and reduces the possibility of incurring charges outside of the initial visit. In systems that use paper records and more traditional systems, the Office Manager confederate is alerted and cancels the orders.
Because it is often impossible to delete a USP’s record from the practice’s EMR system, office managers are consulted to develop strategies to inactivate the patient file in an inconspicuous way as to not alert other practice physicians who have consented to the study. In order for the USP files to not interfere with real patient files, some systems allow records to be “flagged” as research or allow nurses or office managers to leave a comment on the chart that the patient is not real for audit purposes. Each EMR is different and an appropriate solution must be negotiated individually.
2.10 Role of the office manager in successful USP research
A confederate within the practice is an essential feature of successfully conducting this type of research. The confederate is usually an office manager or similarly situated employee who facilitates the entry of USPs into the practice for the clinical encounter and all aspects of pre- and post-study logistics. In fact, it would be impossible to conduct USP research without an inside collaborator. Our study research staff and the office manager meet at least once to discuss the logistical issues. Research staff dispel concerns about identity theft and insurance fraud and also assist the office manager with problems such as when the practice has restrictions on the individuals accepted as patients. Some practices will only accept individuals within a closed network or specific types of insurance. If problems arise once USPs arrive at the practice, such as visits that cost far more than originally planned, the confederate can assist the USP on-site. Office managers have also proven helpful in the physician close-out process. For this study, physicians who have seen both USPs are asked to complete a follow-up survey asking if they had identified any patients as USPs and a short demographics survey. The office manager serves as the intermediary, often providing the physician with the surveys, following up with them to ensure the surveys are completed, and faxed to the research staff. He/she will also provide copies of the medical records documenting the clinical encounter.