PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
BMJ Qual Saf. Author manuscript; available in PMC 2013 August 1.
Published in final edited form as:
PMCID: PMC3680371
NIHMSID: NIHMS469764

Challenges of Making a Diagnosis in the Outpatient Setting: A Multi-Site Survey of Primary Care Physicians

Urmimala Sarkar, MD MPH,1,2 Doug Bonacum, MAS,3 William Strull, MD,3 Christiane Spitzmueller, PhD,4 Nancy Jin, MS,1,2 Andrea Lopez, BS,1,2 Traber Davis Giardina, MA, MSW,5,6 Ashley N.D. Meyer, PhD,5,6 and Hardeep Singh, MD, MPH5,6

Abstract

Background

Although misdiagnosis in the outpatient setting leads to significant patient harm and wasted resources, it is not well studied. We surveyed primary care physicians (PCPs) about barriers to timely diagnosis in the outpatient setting and assessed their perceptions of diagnostic difficulty.

Methods

We conducted a survey of general internists and family physicians practicing in an integrated health system across 10 geographically dispersed states in 2005. The survey elicited information on key cognitive failures (such as in clinical knowledge or judgment) for a specific case, and solicited strategies for reducing diagnostic delays. Content analysis was used to categorize cognitive failures and strategies for improvement. We examined the extent and predictors of diagnostic difficulty, defined as reporting >5% patients difficult to diagnose.

Results

Of 1817 physicians surveyed, 1054 (58%) responded; 848 (80%) respondents primarily practiced in outpatient settings and had an assigned patient panel (inclusion sample). Inadequate knowledge (19.9%) was the most commonly reported cognitive factor. Half reported >5% of their patients were difficult to diagnose; more experienced physicians reported less diagnostic difficulty. In adjusted analyses, problems with information processing (information availability and time to review it) and the referral process, were associated with greater diagnostic difficulty. Strategies for improvement most commonly involved workload issues (panel size, non-visit tasks).

Conclusions

PCPs report a variety of reasons for diagnostic difficulties in primary care practice. In our study, knowledge gaps appear to be a prominent concern. Interventions that address these gaps as well as practice level issues such as time to process diagnostic information and better subspecialty input may reduce diagnostic difficulties in primary care.

Keywords: diagnostic error, missed and delayed diagnosis, primary care, cognitive errors, patient safety

Introduction

Missed, delayed and incorrect diagnoses (also commonly labeled as misdiagnosis) lead to significant patient harm and wasted medical resources but remain under-studied, especially in the ambulatory setting.1-6 The diagnostic process is complex and involves multiple providers and settings over a period of time, and thus comprehensive investigation of this issue remains elusive. Prior work examining closed outpatient malpractice claims suggests that failures of clinical judgment, process breakdowns, and patient factors commonly contributed to missed and delayed diagnoses.7,8 However, claims-based studies are not always generalizable to primary care practice.9

Physician perspectives are essential in enhancing understanding of missed and delayed diagnosis. For instance, a recent physician survey solicited specific cases of diagnostic errors from internists, revealing that, across multiple diseases, process breakdowns in the testing phase were commonly implicated.4 In another study on outpatient safety, primary care physicians (PCPs) believed that poor processes of care involving test results contributed significantly to medical errors.10 While these physician perspectives provide useful information about the types of missed and delayed diagnosis and establish the relationship of errors with process breakdowns, workflow and process issues that lead to diagnostic difficulty are not well understood. For instance, difficulty in achieving timely and correct diagnosis might relate to factors that are disease-specific, such as atypical disease presentation,11 but may also result from factors related to availability and interpretation of diagnostic information, support from specialty consultants, and the PCP's work environment.

Physicians are often unaware of the diagnostic errors they make and lack systematic feedback systems to provide them with information about which of their patients experienced missed or delayed diagnosis.12 However, they can readily report difficulties they experience in the diagnostic processes,10 and this knowledge could further advance the understanding of missed and delayed diagnosis. Therefore, our objective was to survey PCPs practicing in an integrated health system across multiple U.S. regions about potential barriers to timely diagnosis in the outpatient setting and the diagnostic difficulty they faced in their practices. By reflecting on their own specific case of missed or delayed diagnosis, physicians reported key cognitive failures (such as in clinical knowledge or judgment) that occurred in the case. We also solicited their opinions on strategies to reduce diagnostic delays.

Methods

This survey was conducted as part of an internal quality improvement process within a large, integrated health care system. The goal of this program was to characterize the vulnerabilities in the outpatient diagnostic process. In addition to the survey discussed in this report, the project included focus groups of physicians and, separately, health plan member patients, discussing the diagnostic process. The information was used by the organizational leadership to attempt to improve the outpatient diagnostic process.

Study population

The target population included salaried, adult PCPs (family medicine or internal medicine) practicing within five distinct geographic regions of a large integrated health system. The survey was done as part of an internal quality improvement initiative in 2005. The research team identified potential participants through credentialing departments, continuing medical education departments, and published physician lists. Sub-specialists were not included.

Survey development and outcome measures

Because validated measures about diagnostic difficulty and barriers to timely diagnosis are underdeveloped, we developed questions based on a comprehensive literature review, and analyses of medical-legal claims. The survey instrument was pilot-tested through cognitive interviews with 15 physicians and modified accordingly.13-15

To determine the extent of perceived diagnostic difficulty, we used the item, “In the past year, about what percentage of your patients did you consider difficult to diagnose?” with 5 ordered responses of 0%, 1%-5%, 6%-10%, 11% to15%, and > 15%. This was our primary outcome of interest.

Respondents reported age, gender, region, and year in practice as well as years in the health system. Because we were mostly interested in PCP perspectives, we obtained information on time practiced in the outpatient vs. other (hospital, urgent care, nursing home) setting and whether participants managed an outpatient panel. The survey evaluated potential barriers to timely diagnosis related to three conceptual domains: (1) processing of diagnostic information such as from symptoms, signs and tests, (2) referral processes and (3) patient characteristics. We selected these domains because of their association with the diagnostic process and because several aspects of these domains were measurable using a PCP-focused structured survey instrument.

The survey also included two open-ended items that (1) elicited key failures in clinical knowledge or judgment that occurred in a specific case of delayed diagnosis recalled by the participant; and (2) requested suggestions to reduce delays in diagnosis, respectively. For the knowledge and judgment failures, the research team (with expertise in cognitive psychology and diagnostic error) used an established taxonomy of cognitive errors5,16 and used content analysis to categorize the described failures into one or more cognitive errors. Some respondents did not provide sufficient information for categorization and their responses were excluded. For the suggestions to reduce delays, a trained qualitative analyst used content analysis with open coding to describe the free-text responses, and the research team categorized them into several categories of interventions targeting systems and process issues to improve the diagnostic process.

Survey administration

An independent research firm administered the final 54-item confidential survey via mail to participants in March 2005. Physicians were offered a $10 incentive to participate, and non-responders received mailed reminders 7 and 14 days after the survey was distributed. The research firm performed all of the data collection and entry and de-identified results for analysis.

Measure Testing

We performed an explanatory factor analysis to ensure a single underlying construct for each scale related to the three conceptual domains (processing of diagnostic information, referral processes and patient characteristics). We calculated Cronbach's alpha for each measure to determine reliability levels. Based on the results, items mapped to 3 underlying factors, corresponding to three domains, as follows: (1) information processing, including availability of information and time to review it (5 items, range 1-5, 1 factor, eigenvalue=1.8, Cronbach alpha=0.70); (2) the subspecialty referral process, encompassing communication between physicians as well as logistical problems and delays in referrals (7 items, range 1-5, 1 factor, eigenvalue=2.7, Cronbach alpha=0.81); (3) patient characteristics, such as non-adherence to recommended follow-up (7 items, range 1-5,1 factor, eigenvalue=2.0, Cronbach alpha=0.72). For each domain, a higher score indicates a more desirable outcome.

Analysis

We tabulated physician characteristics and the frequency with which they reported diagnostic difficulty (our primary outcome). We then investigated whether physician characteristics and their ratings of potential barriers were associated with perceived diagnostic difficulty. We created a logistic regression model for the dependent variable of diagnostic difficulty, defined by perception that > 5% of cases in the prior year were difficult to diagnose. We chose this cut-off for frequent diagnostic difficulty based on the distribution of responses in our data (Figure 1). Variables significant at the level of p=0.05 in univariate models were retained in the multivariate model, and geographic region was included as pre-specified. We suspected that those reporting more frequent diagnostic difficulty would respond differently to questions about which specific factors served as barriers to timely and accurate diagnosis. Therefore, we also assessed how the overall pattern of responses to individual items in each domain differed between those reporting more versus less frequent diagnostic difficulty. We obtained IRB approval to analyze this data for publication.

Figure 1
Diagnostic difficulty: distribution of perceived frequency of difficult-to-diagnose cases

Results

Of 1817 surveys mailed, 1054 were completed (response rate 58%). We used American Association for Public Opinion Research response rate definition 617. For subsequent analysis, we restricted our study sample to 848 (80%) respondents who reported primarily practicing in the outpatient setting and had a patient panel. Respondents had been in practice for a mean of 13 years, and 32% reported prior work experience in private practice/non-group-model health maintenance organization. Their mean age was 43, and 62% were male. Overall, 50% of respondents reported that >5% of their cases were difficult to diagnose (Figure 1).

Regarding specific knowledge and judgment issues contributing to missed and delayed diagnoses, less than half of the respondents to this question (198 of 453) reported data that involved any cognitive factors; others reported system and patient factors instead. We report the most 10 common cognitive factors and the 10 most common systems factors described by participants, in Table 1.

Table 1
Cognitive factors related to missed and delayed diagnosis (N=251 responses from 198 respondents)

The most common cognitive issues reported were inadequate knowledge base and inadequate detection or perception of a clinical problem. Problems with interpretation of clinical information were also frequently reported, such as mis-identification of a symptom or sign or over-estimation of the reliability or saliency of a finding. Systems issues related to information availability and the negative impact of a heavy workload on the diagnostic process. Two patient-related factors emerged consistently from the responses: (1) lack of adherence to physician recommendations and diagnostic evaluation, and (2) failure to provide information about symptoms.

A variety of themes emerged in open-ended suggestions to reduce diagnostic delays, as shown in Table 2. The most common suggestions related to physician workload, which included lengthening patient visit times, reducing the size of each physician's panel, and off-loading non-clinical work to non-physicians. Respondents cited wait times for diagnostic tests, and for results to be conveyed to physicians and patients, as an area for improvement, mirroring results from other clinical settings.18 Respondents also reported that improved access to specialists would reduce diagnostic delays.

Table 2
Open-ended responses to improve diagnosis (Total responses=384)

As shown in Table 3, the only physician characteristic associated with reporting more frequent diagnostic difficulty was time since completion of residency (those with more experience reported less frequent diagnostic difficulty). In multivariate analyses, diagnostic difficulty was associated with worse ratings of information processing and more problems with the referral process.

Table 3
Factors associated with Higher Frequency of Diagnostic Difficulty

Within the domain of information processing, those who reported a higher frequency of diagnostic difficulty reported that clinically relevant information was not always available. In addition, they reported more frequent concerns about the adequacy of time to integrate clinical information and to think carefully about diagnostic issues (Table 4).

Table 4
Items associated with reporting more diagnostic difficulty

In terms of the referral process, those reporting diagnostic difficulty reported less subspecialty access, inadequate communication between PCPs and subspecialists, and insufficient guidance from subspecialists back to the referring provider (Table 4). In addition, those reporting difficulty were significantly more likely to report an unclear understanding between specialists and themselves regarding who is responsible for informing and treating patients when abnormal test results are reported (Table 4).

Conclusions

We report findings from a multi-center survey conducted to support quality improvement efforts to address ambulatory diagnostic problems faced by primary care physicians. We elicited several cognitive factors PCPs believed to be associated with diagnostic delays, as well as strategies for improvement. We found that half of respondents reporting that more than 5% of their cases were difficult to diagnose. Though this is not a high proportion for any individual clinician, it represents many difficult diagnoses overall. In our analysis, diagnostic difficulty appears to correlate with inadequate time to process diagnostic information and insufficient guidance from subspecialists.

Our study provides rich data regarding processes of care that make diagnosis vulnerable to errors in the primary care setting. Cognitive factors including inadequate knowledge base and faulty perception/detection were noted, but even when specifically asked about cognitive factors, respondents referenced system and patient factors, underscoring their importance. The most frequent cognitive factor reported was related to inadequate knowledge, in contrast to a previous study where knowledge problems were only infrequently related to diagnostic errors16. This was followed by faulty detection or perception of diagnostic information (such as from history/exam/record review), a factor also prominent in a recent survey of pediatricians19. In addition to improving information synthesis and reducing errors from biases and heuristics, efforts to reduce diagnostic errors should also focus on these two important factors. As with prior studies4,7, lack of timely information review was commonly cited as a contributor to missed and delayed diagnosis.

We also report specific improvement strategies that primary care physicians believe will improve missed and delayed diagnosis, for which empiric literature is generally lacking. Respondents' strategies for improvement often centered on physician workload; they identified several distinct links between physician workload and delays in diagnosis. First, they perceived longer visit time with patients, which may allow for more thorough history and physical examination, as important to the diagnostic process. Second, they specifically cited reducing workload in order to have protected time to review patient results, to ensure follow-up of clinical abnormalities. Other recent studies also demonstrate this to be a high-risk area despite improved information availability.20,21 For instance, studies reveal sub-optimal follow-up of abnormal test results, even in the presence of electronic health record-based clinician notification, suggesting that that missed or delayed diagnoses are not always related to lack of access to information. Other system and cognitive factors including time and workload are likely in play and need to be targeted for improvement.

In addition, participants suggested more time for physicians to think carefully about difficult cases, and to perform “cognitive work”.22 Moreover, they cited the burden associated with non-visit-related tasks. These tasks, such as following up patient test results, are increasingly becoming recognized as a significant contributor to physician workload,23 and will require specific, workflow targeted efforts to address. It is possible that the patient-centered medical home model, 24 with panel-based rather than visit-based reimbursement, 25 will allow for clinicians to allocate their time in a way that better supports timely, correct diagnoses. As an example, in the integrated health system in which this study took place, assessment of PCP performance is shifting to quality, service, and access metrics across a PCP's patient population, rather than the measurement of the number of visits per day provided by the PCP.

Despite the presence of an integrated health care system where subspecialists were readily accessible, we found that the referral process is perceived as a barrier to timely and complete diagnosis. The PCPs in this study who reported diagnostic difficulty acknowledged problems accessing and communicating with subspecialists. This problem is likely be worse in individual practices, where PCPs interact with subspecialists from several other health care systems using many types of communication methods.26 In addition, the issue of shared responsibility for a patient between a subspecialist and a PCP merits further inquiry. For instance, there is no clear consensus among clinicians about who should refer patients to a second specialist if the first specialty referral does not result in a diagnosis and/or treatment plan.27 Similarly, physician perceptions vary about who should inform patients about abnormal test results; studies suggest that usual clinic practice leaves many patients uninformed.18 Some degree of standardization of referral expectations and practices is likely to enhance the diagnostic process.26

Our study has several limitations. First, although conducted across five distinct U.S. geographic regions with significant variability in practice settings, it was originally designed as a quality improvement project within a single health system with salaried, staff physicians and thus, its generalizability might be limited. For instance, it might not be representative of communication between PCPs and subspecialists in U.S. ambulatory health care in general where per-consultation financial incentives for subspecialists are different.28 Second, our study was performed in 2005. Findings remain relevant despite the age of the data because: (a, cognitive factors described here are unlikely to have substantively altered; (b) system factors such as short visit length and problems processing information from multiple inputs are likely to persist. However, this study was conducted prior to comprehensive electronic health record (EHR) implementation at this health system. Barriers we found associated with perceived difficulty might not necessarily be mitigated by EHRs29 and most U.S. primary care practices do not currently have a comprehensive EHR. Third, this study was conducted prior to many of the landmark studies on diagnostic error and diagnostic processes; thus, the phrasing of the items might not at times follow taxonomies laid out in these papers4,7,16,30. However, the issues described in the survey are interpretable, have been described in the literature, and emanate from primary care workflow. Fourth, the patient population cared for by the PCPs, although demographically diverse,31 represents a group with health insurance, mostly provided by employers. As such, patients in this system may have fewer barriers than in other ambulatory health care settings. Fifth, our results reflect only the missed diagnoses that providers are aware of. There are likely other missed diagnosis that remain unknown to primary care providers, and our study cannot shed light on this type of missed diagnosis. Finally, as with all survey research, subjects who chose to respond to the survey are likely to differ from those who do not. We did not have sufficient demographic information on non-responders to make inferences about how non-response affected the reported results.

In summary, a significant proportion of primary care physicians report diagnostic difficulty involving at least 5% of their patients. Misdiagnosis relates to several system and cognitive factors but knowledge gaps might be more often responsible than previously estimated. Interventions that address practice level issues such as time to process diagnostic information and better subspecialty input might potentially reduce diagnostic safety concerns in primary care.

Acknowledgments

Dr. Sarkar is supported by Agency for Healthcare Research and Quality K08 HS017594. Dr. Singh is supported by an NIH K23 career development award (K23CA125585), and in part by the Houston VA HSR&D Center of Excellence (HFP90-020). Manuscript contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH or any of the other funders. None of the funders had any role in design and conduct of the study; collection, management, analysis, or interpretation of the data; or preparation, review, or approval of the manuscript.

Funding Statement: Dr. Sarkar is supported by Agency for Healthcare Research and Quality K08 HS017594. Dr. Singh is supported by an NIH K23 career development award (K23CA125585), and in part by the Houston VA HSR&D Center of Excellence (HFP90-020). Manuscript contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH or any of the other funders. None of the funders had any role in design and conduct of the study; collection, management, analysis, or interpretation of the data; or preparation, review, or approval of the manuscript.

Footnotes

Competing Interests Statement: Authors report no competing interests

Contributorship Statement: US developed analysis plan, oversaw quantitative analysis and wrote the first draft manuscript. DB conceived the study, oversaw data collection, owns data, and provided intellectual contribution in editing the manuscript. WS owns data and provided intellectual contribution in editing the manuscript. AL conducted literature review and assisted with drafting manuscript. CS and NJ conducted quantitative analysis. TG and ANDM conducted qualitative analysis. HS developed analysis plan, oversaw qualitative analysis and provided intellectual contribution in editing the manuscript.

References

1. Bishop TF, Ryan AM, Casalino LP. Paid malpractice claims for adverse events in inpatient and outpatient settings. Jama. 2011;305:2427–31. [PubMed]
2. Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med. 1991;324:377–84. [PubMed]
3. Thomas EJ, Studdert DM, Burstin HR, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care. 2000;38:261–71. [PubMed]
4. Schiff GD, Hasan O, Kim S, et al. Diagnostic error in medicine: analysis of 583 physician-reported errors. Arch Intern Med. 2009;169:1881–7. [PubMed]
5. Graber M. Diagnostic errors in medicine: a case of neglect. Jt Comm J Qual Patient Saf. 2005;31:106–13. [PubMed]
6. Newman-Toker DE, Pronovost PJ. Diagnostic errors--the next frontier for patient safety. JAMA. 2009;301:1060–2. [PubMed]
7. Gandhi TK, Kachalia A, Thomas EJ, et al. Missed and delayed diagnoses in the ambulatory setting: a study of closed malpractice claims. Ann Intern Med. 2006;145:488–96. [PubMed]
8. Phillips RL, Jr, Bartholomew LA, Dovey SM, et al. Learning from malpractice claims about negligent, adverse events in primary care in the United States. Qual Saf Health Care. 2004;13:121–6. [PMC free article] [PubMed]
9. Thomas EJ, Petersen LA. Measuring errors and adverse events in health care. J Gen Intern Med. 2003;18:61–7. [PMC free article] [PubMed]
10. Manwell LB, Williams ES, Babbott S, et al. Physician perspectives on quality and error in the outpatient setting. WMJ. 2009;108:139–44. [PubMed]
11. Kostopoulou O, Delaney BC, Munro CW. Diagnostic difficulty and error in primary care--a systematic review. Fam Pract. 2008;25:400–13. [PubMed]
12. Schiff GD. Minimizing diagnostic error: the importance of follow-up and feedback. Am J Med. 2008;121:S38–42. [PubMed]
13. Beatty P. The dynamics of cognitive interviewing. In: Presser SRJ, Couper MP, et al., editors. Methods for testing and evaluating survey questionnaires. Vol. 2004. Hoboken, N.J: John Wiley & Sons, Inc; pp. 45–66.
14. Carbone ET, Campbell MK, Honess-Morreale L. Use of cognitive interview techniques in the development of nutrition surveys and interactive nutrition messages for low-income populations. J Am Diet Assoc. 2002;102:690–6. [PubMed]
15. Collins D. Pretesting survey instruments: an overview of cognitive methods. Qual Life Res. 2003;12:229–38. [PubMed]
16. Graber ML, Franklin N, Gordon R. Diagnostic error in internal medicine. Arch Intern Med. 2005;165:1493–9. [PubMed]
17. The American Association for Public Opinion Research. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 7th. AAPOR; 2011.
18. Casalino LP, Dunham D, Chin MH, et al. Frequency of Failure to Inform Patients of Clinically Significant Outpatient Test Results. Archives of Internal Medicine. 2009;169:1123–29. [PubMed]
19. Singh H, Thomas EJ, Wilson L, et al. Errors of diagnosis in pediatric practice: a multisite survey. Pediatrics. 2010;126:70–9. [PMC free article] [PubMed]
20. Singh H, Thomas EJ, Mani S, et al. Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential? Arch Intern Med. 2009;169:1578–86. [PMC free article] [PubMed]
21. Singh H, Thomas EJ, Sittig DF, et al. Notification of abnormal lab test results in an electronic medical record: do any safety concerns remain? Am J Med. 2010;123:238–44. [PMC free article] [PubMed]
22. Singh H, Petersen LA, Thomas EJ. Understanding diagnostic errors in medicine: a lesson from aviation. Qual Saf Health Care. 2006;15:159–64. [PMC free article] [PubMed]
23. Doerr E, Galpin K, Jones-Taylor C, et al. Between-Visit Workload in Primary Care. J Gen Intern Med. 25:1289–92. [PMC free article] [PubMed]
24. Patient-Centered Primary Care Collaborative
25. Rittenhouse DR, Shortell SM. The patient-centered medical home: will it stand the test of health reform? JAMA. 2009;301:2038–40. [PubMed]
26. Singh H, Esquivel A, Sittig DF, et al. Follow-up actions on electronic referral communication in a multispecialty outpatient setting. J Gen Intern Med. 2011;26:64–9. [PMC free article] [PubMed]
27. American College of Physicians A Position Paper. The Patient-Centered Medical Home Neighbor: The Interface of the Patient-Centered Medical Home with Specialty/Subspecialty Practices. 2010
28. Pearson SD. Principles of generalist-specialist relationships. J Gen Intern Med. 1999;14(1):S13–20. [PMC free article] [PubMed]
29. Singh H, Graber M. Reducing diagnostic error through medical home-based primary care reform. Jama. 2010;304:463–4. [PMC free article] [PubMed]
30. Singh H, Thomas E, Khan M, et al. Identifying diagnostic errors in primary care using an electronic screening algorithm. Archives of Internal Medicine. 2007;167:302–8. [PubMed]
31. Krieger N. Overcoming the absence of socioeconomic data in medical records: validation and application of a census-based methodology. Am J Public Health. 1992;82:703–10. [PubMed]