|Home | About | Journals | Submit | Contact Us | Français|
This paper provides an overview of the implementation of using Unannounced Standardized Patients (USPs) to conduct health communication research in clinical settings.
Certain types of health communication situations are difficult to capture because of their rarity or unpredictable nature. In primary care the real reasons for a visit are frequently unknown until the consultation is well under way. Therefore, it is logistically difficult for communication studies to capture many real-time communications between patients and their physicians. Although the USP methodology is ideal for capturing these communication behaviors, challenges to using this method include developing collaborative relationships with clinical practices, logistical issues such as safeguarding the identity of the USP, training USPs and creating their identities, maintaining fidelity to the role, and analyzing the resultant data.
This paper discusses the challenges and solutions to USP implementation. We provide an example of how to implement a USP study using an on-going study being conducted in primary care practices.
This paper explores the advantages and challenges as well as strategies to overcome obstacles to implementing a USP study.
Despite the challenges, USP methodology can contribute much to our understanding of health communication and practice.
Standardized patients (SPs) have been used to simulate clinical encounters for almost 50 years (1). SPs are commonly used in medical education to educate and evaluate students and practicing physicians, including skill certification via the SP-based US Medical Licensing Exam Clinical Skills Examination (2). A recent review of the SP literature highlights the types of studies conducted to date ranging from assessing physician knowledge, medical skills and a plethora of communications practice with acute and chronic care patients (3–13).
Unannounced standardized patients (USPs) is the method of choice to measure the effect of patient or physician characteristics on communication and decision making behaviors and to capture real-time physician-patient communication (14). Indirect methods to examine these behaviors have included medical chart review, physician and patient retrospective surveys, and paper and pencil or video case presentations (vignettes). These methods are limited in several ways. Medical charts lack completeness and do not record many sociological variables (15) and retrospective surveys often lack accuracy and detail. While vignettes are frequently used as a substitute for real interactions, this method’s obvious limitations are the inability to create vignettes with enough detail, their static nature, and high threat of response bias due to social desirability. Finally, the relational aspects of communication are difficult to measure reliably unless directly observed. Therefore, direct measurement methods are generally preferred to assess communication behaviors and patterns (16–22). Nonetheless, the unpredictability or rarity of when certain types of interactions will occur, or the difficulty of obtaining access, and the cost of case-finding, can make direct observation impractical. Furthermore, patients and physicians self-select according to demographics (e.g., gender) and interaction style; dissatisfied patients tend to switch doctors, and others may adapt their communication style to be in consort with that of the physician.
The objective of this review paper is to discuss the challenges associated with successful implementation of USP research methodology and provides feasible solutions to these barriers. This paper provides an overview of implementation challenges and strategies and provides examples from an on-going USP study being conducted in primary care settings.
One of the major advantages of the USP methodology is the ability to introduce a simulated, standardized patient to enable difficult to capture clinical encounters and minimize response bias through the creation of a scheduled observable encounter. This methodology protects against bias because the USP presents as an actual patient. Because of these advantages, USPs are increasingly used to assess physician performance under actual practice conditions, which may differ from performance observed in structured examination conditions where physicians know they are being observed. To implement a USP protocol, physicians consent in advance to visits by USP(s) but do not know when a USP clinical encounter will occur. Recent communication studies using USPs have ranged from cancer to asthma patients and have focused on interviewing techniques, self-efficacy, and physician self-disclosure (23–30).
In communication studies involving USPs, the most important threats to validity are detection of the USP and lack of role fidelity (31–32). Physician detection of USPs has been found to range from 1–70% (33). Luck (2003) carried out a validity study of USPs and reported successfully training 45 actors to represent a single USP role (34). Other studies have reported similar results (35). In general, detection rates average from 5–20% (7).
Factors associated with USP detection are not well understood. Franz and colleagues (2006) report that type of practice predicting USP detection (detection least common in HMO settings) (7). SP training, contextual, geographic, and cultural factors may also play an important role (7). Careful USP preparation and advanced logistical management with the office manager is key to avoid detection. Three papers, the most recent published in 2001, have detailed strategies to introduce USPs into practices without detection (36–38). However, with the advent of electronic records and instant access between patients insurance carriers and medical practices, the use of USPs has become more complex than what is currently reported in the literature. Today, there are many new challenges facing the introduction of unannounced standardized patients into physician practices. Innovative, creative, and often situation-specific solutions are needed to overcome these and other barriers.
Below we describe implementation of the USP methodology and provide an example of an on-going study of physician-patient communication using USPs to illustrate the methodology. We explain the steps taken to prevent detection, identify the major hurdles encountered during the study’s pre-implementation phase, and propose viable strategies to conquer the obstacles encountered when introducing USPs into practices. We also review how technological advances have changed the conduct of USP research, simultaneously making it easier and more challenging. The data presented is provided for illustrative purposes only and provides and example of application of strategies for implementing a USP study. Table 1 presents a summary of the challenges and potential solutions to conducting a USP study.
To highlight the challenges and solutions to USP methodology, we present the challenges of a study that is examining physician-patient communication during a new primary care visit. This example simulates a 47-year old patient who presents a common set of symptoms to a primary care physician (general internist or family practitioner). The specific objective of the study is to understand how and whether physician-patient communications are affected by patient race and gender. USPs who vary by race and gender are trained to deliver the role in a standardized manner and to minimize outward expressions of racial or gender stereotypes. Each participating physician sees at least two USPs, each of a different gender and race (e.g., Caucasian male and African American female or African American male and White female), and each visit is recorded using a concealed recorder. USPs are scheduled as new patient visits and the practice is reimbursed for each visit. As an added incentive, physicians are offered the opportunity to receive feedback on how their communication style compares to that of their peers. Physicians are blinded to the set of symptoms that the USP will present and are blinded as to when the visit will occur. The first USP visit is scheduled at least four months after consent and the second visit occurs approximately three months later to avoid recognition of the patient case presentation. The resultant audiorecordings are evaluated by an independent rater to assess role fidelity and coded to examine the nature of the communication process. The medical chart is obtained to assess differential diagnosis and treatment/diagnostic recommendations.
USPs can be recruited from pools of actors or individuals who are professional standardized patients. The advantage of the latter is that these individuals regularly simulate patients, are familiar with the medical environment, and are usually comfortable with physical exams and how to simulate an illness condition. In our example study, USPs were selected from a pool of experienced SPs who work for a simulation program in a School of Medicine. Age, gender, and race were primary criteria, along with ability to perform the role. The number of USPs trained was based on the number of visits required by the study. Selecting at least two USPs of each gender-race type was necessary to ensure adequate availability of any given USP gender-race type and ensure consistency in role performance should any one USP leave the study unexpectedly. It was also important to choose individuals within a range of normal body mass index who were free of common chronic diseases such as high blood pressure to provide the physician with an accurate physical presentation free from introducing confounders into the communication surrounding the chief medical complaint.
Adequate training of USPs is critical for a USP study’s success. Even when using seasoned USPs, a 3-day formal training period followed by practice and role refinement, should be budgeted at minimum. USPs participated in a three-day training program with two half-day follow-up trainings before going into the field. In addition, on-going monitoring is used to provide the SPs with reminders, tips and individual remediation as needed. The training was led by a standardized patient trainer with over 30 years experience in SP clinical skills teaching (GG). Each USP was trained to begin the encounter with a mandatory opening line, repeated verbatim, unless interrupted by the physician. Details were scripted regarding presenting illness history, associated symptoms (e.g., frequency, duration, intensity), and medical and social histories. USPs were trained to provide reasonable explanations for any unplanned abnormal physical findings. For example, a higher than normal blood pressure reading is explained as “white coat hypertension” or being rushed. In addition to the mandatory opening line, the USPs were trained to attach “feeling” words like “annoyed,” “concerned,” and “worried” to specific symptoms presented. They were also trained to give excuses (such as the need to return to work) to avoid immediate diagnostic tests (e.g., blood work).
One of the most difficult things to standardize in research with USPs is appropriate responses to physician prompts. USPs were trained to follow the physician’s lead and communication style. For example, USPs are required to respond to open-ended questions with open-ended responses and close-ended questions with one-word responses. Similarly, if a physician interrupts the USP or is discourteous, information is not divulged unless an appropriate question is re-directed by the physician. Verbal and non-verbal responses to the physical examination also require standardization, including response to palpation. Specific care was taken to standardize USPs non-verbal role presentation, including a discussion of dress, eye contact, posture, rate of speech, and expression of concern about problem. Following training, practiced encounters continued and a refresher session was held. After each visit, an experienced independent observer (GG) reviews the audiorecording and completes a role consistency form to give the USP feedback on performance, including the USP’s ability to provide accurate information, affect, and believability.
Some USP studies are conducted within a small area while others require travel throughout a large region. In the study example, to maintain the identity and prevent detection, the USP is an individual who has moved to the area recently to take a new job. This allowed the USP to play an individual not native to the region. This can be especially advantageous when introducing USPs into rural practices and where regional speech and accents are especially pronounced.
Technological advances now allow researchers to easily generate logistical information about a USP. For example, this study’s USPs retain their first names but adopt last names randomly selected by an online name generator (e.g., http://www.fakenamegenerator.com/). Because practices often require patients to present photo identification, the USPs are provided a photo identification card. Websites are available to design and create work identification cards and other forms of identification (e.g., http://www.easyidcard.com/).
USP addresses are chosen using on-line mapping. Care is taken to ensure that no addresses are in actual use. Any addresses that belong to a residence are excluded. Using website satellite mapping technology, researchers and USPs have access to street-level imagery of the area surrounding the practice. This is particularly helpful in rural areas, where “small talk” with the physician may center on town landmarks. To further circumvent detection, we use another online product that enables us to assign USPs local working phone numbers for each visit with the correct area codes, without buying additional phone lines.
Practices often telephone USPs to remind them of appointments, follow-up on a referral made by the physician, and/or ask billing questions. To deal with this practice, the example study uses phone numbers generated to mask the USP’s real identity. These numbers forward to a generic voicemail at the research office. When a voicemail is received from the physician’s office, it is electronically captured by an email that is sent to the email address associated with the account (typically the research coordinator). All voicemails can be easily accessed online using an embedded player in the email. For research staff, having access to messages from all SP telephone numbers through one email account facilitates the prompt handling of logistical issues that may arise throughout the pre- and post-visit process.
Technology has improved in recent years that facilitates good quality recordings using concealed digital voice recorders that are small enough to be concealed in a jacket pocket or purse, while still delivering high quality sound recordings. Programs are available to eliminate background noise from the recording post-production. Spy-type digital recording equipment that simulate everyday objects (e.g., pens, tie clips, credit cards.) is also available, but of poorer quality. In this study, a pen recorder is used as a back-up recording device in case of primary recorder failure. USPs are carefully trained in the use of both devices and the need to obtain audio files of the encounter for analysis of the communication process.
Despite these benefits, technological development over the past decade have also made conducting USP research more difficult. Electronic registration systems and electronic medical records (EMRs) present significant logistical challenges. The American Recovery and Reinvestment Act calls for EMRs for all patients by 2014 (see www.recovery.gov), which will make this technology even more ubiquitous. Successful USP research in the United States will undoubtedly require an understanding of these systems and collaboration with key players to avoid USP detection outside of the clinical encounter itself.
Electronic registration systems, linking multiple private practices or many practices within a hospital system, place strict requirements on patient information entered to avoid record duplication. If data in specific record fields, such as name, or social security number are the same, the probability of detection rises. Therefore, in our example, an individual USP requires many separate identities as visits made within the same EMR/registration system. Specific care is taken in the selection of social security numbers that are not used by any US citizen (see http://ssa-custhelp.ssa.gov/app/answers/detail/a_id/425). Use of these numbers ensure that USPs are not “sharing” identities with any real individuals. This approach ensures that the new registration file created for each USP individual visit is unique and lowers the chance of detection due to information overlap.
Many electronic systems have an automatic operation that verifies insurance eligibility and the validity of insurance information provided at the time of the visit. While other studies have used facsimile insurance ID cards, that approach is no longer possible because of the ability to instantly check these cards. To avoid this problem, the USPs in this study present as self-pay patients. USPs pay for office visits in cash, thus avoiding the logistical issues surrounding insurance verification, and claims processing. Practices are reimbursed for each new patient visit according to the amount set by major insurance carriers. The office managers are consulted about the amount the USP will be charged at the time of the visit so that the USP can bring enough money to pay cash for the visit. In countries with centralized health insurance systems, other arrangements will need to be made.
In addition, not all practices accept self-pay patients or cash, opting instead to bill or set them up for financial counseling. In this instance, research staff may need to meet with the health system’s financial services staff to establish special billing procedures. For this study, research staff and financial staff created group billing accounts within certain hospital systems. These special billing accounts were created under the names of two fictitious manufacturing companies where the USPs identified they were employed. When USPs check-in at practices of physicians affiliated with this billing system, they provide the administration staff with a letter from the manufacturing company outlining the agreement between the company and hospital billing; this letter includes the account number for billing. The two special accounts link directly to the department running the study and any charges incurred during the visit are billed directly to the study’s funding source. This approach may also be useful for centralized national or provincial health insurance systems.
EMRs generally include a section for the physician to order follow-up tests or specialist visits. In our example, an EMR systems expert was identified to monitor the USP charts for orders and visits beyond the initial visit and cancel them. This strategy alleviates the potential for a USP to occupy a specialist appointment time reserved for real patients and reduces the possibility of incurring charges outside of the initial visit. In systems that use paper records and more traditional systems, the Office Manager confederate is alerted and cancels the orders.
Because it is often impossible to delete a USP’s record from the practice’s EMR system, office managers are consulted to develop strategies to inactivate the patient file in an inconspicuous way as to not alert other practice physicians who have consented to the study. In order for the USP files to not interfere with real patient files, some systems allow records to be “flagged” as research or allow nurses or office managers to leave a comment on the chart that the patient is not real for audit purposes. Each EMR is different and an appropriate solution must be negotiated individually.
A confederate within the practice is an essential feature of successfully conducting this type of research. The confederate is usually an office manager or similarly situated employee who facilitates the entry of USPs into the practice for the clinical encounter and all aspects of pre- and post-study logistics. In fact, it would be impossible to conduct USP research without an inside collaborator. Our study research staff and the office manager meet at least once to discuss the logistical issues. Research staff dispel concerns about identity theft and insurance fraud and also assist the office manager with problems such as when the practice has restrictions on the individuals accepted as patients. Some practices will only accept individuals within a closed network or specific types of insurance. If problems arise once USPs arrive at the practice, such as visits that cost far more than originally planned, the confederate can assist the USP on-site. Office managers have also proven helpful in the physician close-out process. For this study, physicians who have seen both USPs are asked to complete a follow-up survey asking if they had identified any patients as USPs and a short demographics survey. The office manager serves as the intermediary, often providing the physician with the surveys, following up with them to ensure the surveys are completed, and faxed to the research staff. He/she will also provide copies of the medical records documenting the clinical encounter.
Using these procedures, the study has, to date, sent 39 USPs to 25 different primary care physicians at 19 different practices over a 5 month period, mostly in community-based nonacademic practices. In this study, once all recruited physicians in a practice have seen all USPs, the physician completes a post-study survey to assess potential detection of the USPs and then is told the identity of the USPs he or she saw. These procedures allow the researcher to assess whether the SP was detected and to debrief the physician. This is a necessary ethical step in a study that entails deception even though the physician subjects consent to these parameters.
To date, we have had only one known detection of a USP. This detection occurred not because of what occurred during the visit, but because the SP disclosed the purpose of his visit to the physician to a store clerk after the visit, who reported it to the physician. This demonstrates the need for absolute concealment about these visits. This is especially true in smaller communities. No physician has reported misidentifying a real patient as a USP.
Role fidelity evaluations assist the research team to maintain performance quality and fidelity. The study example has completed 16 USP evaluations. The evaluator determines if each key piece of information was divulged by the USP when appropriately prompted, not divulged when prompted, volunteered when not prompted, or not relevant (not volunteered, not prompted). The form divides the scripted role information into various categories, including presenting complaints, medical history, physical exam, social history, emotional state, authenticity, response to physician questioning, and overall performance. There is additional space on the form to provide qualitative (textual) data on accuracy within each category. Because the form is on-line, the USP can log on and see the “results” of his/her performance, receiving timely feedback that can improve the next visit’s performance. The trainer reviews the performance with the USP and may listen to an encounter together so that the trainer can specify how to improve performance. Table 2, below, provides selected examples and findings from the role consistency form. A large or consistent number of responses in the “did not tell when prompted” or “not asked, volunteered” column suggest areas of improvement for individual USPs. The USP trainer uses these results to move individual USPs towards the goal of receiving 100% of their checkboxes in the “told when prompted” or “not asked, not told” columns. In this study, USPs’ performance on individual items ranges from 81.3% or 93.8%, with the highest fidelity observed in the reporting of pain. Although the trainer evaluates the USPs using the role consistency form and may be biased towards indicating a higher adherence rating, the trainer is the most experienced and most familiar with the role. Furthermore, the ratings of role consistency ask for objective data, such as which conversation topics are discussed. These are chosen based on role and extracted from the recorded audio conversation.
USPs were also standardized to display and communicate similar levels of anxiety about their symptoms, to neither be too active nor too passive in their communications, and to follow the physician’s lead and divulge appropriate amounts of information depending on the nature of the question (open vs. close-ended). We rated USPs, for example, on how appropriate their responses to the physician were mean rating 6.2 out of 7; level of anxiety exhibited was 2.9 with an ideal level of 3–4. (See data in Table 3.)
USPs have been able to convincingly portray real patients. USPs were rated on a scale from 1–6 with 7 indicating the most convincing portrayal. ‘Believability’ was rated as 6.7 (SD=0.58), convincingness of portrayal, 6.8 (SD=0.54), and frequency of maintaining role, 6.9 (SD=0.34). To further evaluate authenticity of USP performance, we selected 6 questions relevant to USP research from the Maasstricht Assessment of Simulated Patients (MaSP) authenticity sub-scale (39). The results are shown in Table 4. Overall, USPs portrayed their roles authentically within the context of the patient-physician encounter. For example, of the16 USPs evaluated to date, all were rated as appearing authentic and 93.8% as answering questions in a natural manner.
A number of tools exist to assist in the coding of communication data. Approaches include examination of linguistic structures that are primarily descriptive of verbal content during a communication interaction and process models to examine interactional behaviors. Communication provides two streams of meaning: one that regards the denotative content of a message and another that reflects the affective tone, and therefore the relational quality of the interactants (40). In the example study, interactions between the USPs and the physicians are audiotaped, transcribed and coded by trained research assistants. To capture and analyze the communication between doctor and patient, we will use an observational scheme that captures both content and relational data. Therefore, we will use the Siminoff Communication Content & Affect Program (SCCAP), a comprehensive scheme for coding communication that accounts for both instrumental and relational communication and captures important contextual features with tailored coding templates (40). Once coded, this data can be used to conduct both descriptive and statistical analyses.
USPs methodology provides a “window” into what occurs in the practice encounter in a manner that is not possible with other indirect methods. As health reform in the US begins to focus on, among other issues, patient centered care, there will be a need for more studies that examine the critical nature of physician-patient communication. Thus, the need for more rigorous studies that employ the use of USPs will grow.
The USP process is maintained and improved throughout the course of the study in several ways. Communication with the study’s research coordinator immediately before and after each visit assists the USP to prepare for the visit and to debrief afterwards, including sharing any problems or concerns, getting timely feedback or, when necessary, immediate intervention by the research coordinator. Checking in with the office confederate also helps to “tie up” the loose ends of the visits (canceling ordered tests for example) and maintaining the study-healthcare practice relationship. Use of the fidelity role form that assesses each performance and feeds back both positive and corrective critiques to the USPs is also a necessary component of the research methodology. In this study, by using these procedures we have been able to obtain authentic role performance by USPs who stayed in their roles and responded to questions in a natural and appropriate manner.
This paper details the methods, challenges and strategies to conduct a USP study to illustrate the USP methodology, we presented how it is possible to apply these to a study of symptom presentation. A well-developed curriculum must be prepared to rigorously train USPs. Good training is required for a number of important reasons. The integrity and utility of the study hinges on the USPs’ ability to play the role convincingly and consistently. The audiotapes generated from such a study should be used to assess role fidelity and remediated as necessary. To be successful, USP research must be sensitive to the concerns of physicians and practice managers. USPs must fully understand their vital role as they become the “face” of the study to the practice; their fulfillment of their role critical to the integrity of the research. Well trained USPs who do not disrupt the regular practice flow contribute to practices’ satisfaction with a study and increase the likelihood that the practice will endorse this research methodology to their peers and participate in future research. Therefore, much advanced planning, communication and negotiation with the practices are required to help facilitate the process and the success of the USP’s encounter with the physician. Debriefing physicians and providing feedback is a critical part of the process. Physicians can provide important information, including whether or not they detected the USP. If detected, that encounter should be removed from the study. In addition, feedback to study subjects provides a concrete benefit to physicians who agree to participate in these studies and help anticipate how the findings of such studies can contribute to the improvement of physician-patient communication.
This methodology can guide how we train the next generation of physicians and locate the elements of communication that are vital to healthcare and healing processes. The relevance and application of the USP methodology in heath communication research are apparent. USP methodology can provide critical details that are necessary to fully examine the communication process. For example, well-trained USPs can write needed field notes that document additional details about the communication process, especially non-verbal communication that is missing from other methods. USP methodology can assist in investigating the roll of nonverbal communication behaviors such as affect. In addition, these studies can provide participating physicians with practical direct feedback. When this method is well developed it provides important insights into the clinician- patient communication process. Implementation of this methodology will become easier once its use is more widespread and understood within the healthcare community. Findings from USP studies can assist in the transformation of our current health system into a system that is responsive to the needs of patients and clinicians alike.
This study is funded by NIH/NCI grant #R01-CA124607.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.