|Home | About | Journals | Submit | Contact Us | Français|
To determine the effect of inaccurate communication of patient data—from clinicians caring for inpatients to prior-approval antimicrobial stewardship programs (ASP) staff (practitioners)—on the incidence of inappropriate antimicrobial recommendations by ASP practitioners
A retrospective cohort design was utilized. The accuracy of communicated patient data was evaluated against patients’ medical records for pre-determined, clinically significant inaccuracies. Inappropriate antimicrobial recommendations were defined when an expert panel unanimously rated the actual recommendations as inappropriate after reviewing vignettes derived from inpatients’ medical records.
The setting was an academic medical center with a prior-approval ASP.
All inpatient subjects of ASP prior-approval calls were eligible for inclusion.
The panel agreed on whether the antimicrobial recommendation was inappropriate or not in 163 (82%) of 200 ASP calls; the 163 calls were then the basis for further analyses. After controlling for confounders, inaccurate communications were associated with inappropriate antimicrobial recommendations with an odds ratio (OR) of 2.2 (p=0.03). In secondary analyses of specific data types, only inaccuracies of microbiological data were associated with the study outcome (OR 7.5, p=0.002). The most common reason given by panelists for an inappropriate rating was that antimicrobials were not indicated.
Inaccurate communication of patient data, particularly microbiological data, during prior-approval calls is associated with an increased risk of inappropriate antimicrobial recommendations from the ASP. Clinicians and ASP practitioners should work to confirm critical communicated data prior to use in prescribing decisions.
Hospital-acquired infections with antimicrobial-resistant pathogens are associated with significant morbidity and mortality . Antimicrobial use is a consistent risk factor for antimicrobial-resistant infections . However, much hospital antimicrobial use is unnecessary , and antimicrobial resistance has steadily risen over time .
Antimicrobial stewardship programs (ASPs) seek to improve antimicrobial prescribing through interventions such as antimicrobial formulary restriction, education, and prior-approval . In prior-approval programs, inpatient clinicians obtain permission from ASP practitioners (typically infectious disease-trained physicians or pharmacists) to prescribe restricted antimicrobials [6–8]. Prior-approval has been shown to improve antimicrobial prescribing and clinical outcomes, decrease costs, and limit the spread of antimicrobial resistance [6–8]. Prior-approval has been cited in recommendations from national infectious diseases societies as an important intervention to limit antimicrobial resistance , and is commonly employed by ASPs .
While prior-approval often depends on data communicated from clinicians, these data were found to contain inaccurate patient information in over 40% of calls made to practitioners in a prior study of our hospital’s ASP . If communication errors lead to inappropriate antimicrobial prescribing, clinical failures of therapy and antimicrobial overuse could limit the benefit of the program . In this retrospective cohort study, we determined the effect of communication inaccuracies during telephone interactions on the appropriateness of antimicrobials recommended by practitioners in the ASP.
This study was performed at the Hospital of the University of Pennsylvania, an academic medical center. The hospital’s prior-approval ASP has been largely described previously . In addition, restriction categories for antimicrobials are posted on the Internet . Although all ASP practitioners could electronically access laboratory and pharmacy data while in the hospital, the pharmacists, but not the ID fellows, elected to enroll in the hospital’s virtual private network to access patient records electronically from home.
All ASP calls from 2/4/03 to 5/7/03 were eligible for inclusion. Calls to fellows (vs. pharmacists) were oversampled to obtain an approximately equal number of calls for fellows and pharmacists to allow adequate measurement of both groups. Only the first call sampled for each patient was used. The study sample was distinct from the sample used in our previous ASP study .
We abstracted data to be evaluated for communication accuracy from the ASP forms and the patients’ medical records using a structured form. Only data available at the time of the call were abstracted. Data types included: microbiological results, abdominal/chest radiographic results, current antimicrobials, antimicrobial allergies, patient temperature, and renal function .
The following variables were abstracted for evaluation as potential confounders: patient age and sex, location (intensive care unit vs. ward), prior length of stay, location prior to admission, indication for therapy, Charlson comorbidity index (in which points are assigned for major chronic disease categories) [13, 14], caller specialty, caller training level, the individual ASP practitioner, weekday vs. weekend, and call time. Characteristics of the caller and ASP practitioner, patient location, and requested antimicrobials were abstracted from the ASP card; other elements were abstracted from patients’ medical records.
Structured case vignettes were created from the medical record data available to the primary service at the time of the ASP call. Upper-year medical housestaff and a certified physician assistant wrote the vignettes, which included: past medical history, admission reason, hospital course, use of central lines, temperature, use of vasopressors, intubation, reason for the call, allergies, current antimicrobials (and prior antimicrobials during the admission if applicable), and specific laboratory and radiographic data. Vignette writers had the reason for the ASP call, but were blinded to ASP communication accuracy.
The current study used the same operational list of definitions for inaccurate communications as a prior study at our institution (definition appendix published online ). Inaccurate communications were defined as clinically significant discrepancies between data elements abstracted from the form the ASP practitioner used to document the call and those in the medical record, using the medical record as the gold standard. Clinically significant discrepancies were those that the investigators a priori judged likely to influence antimicrobial prescribing. The definitions were designed to give the benefit of the doubt to the caller; inaccuracies were cases in which patient data recorded from the caller were directly contradicted by medical record data available at the time of the call. We did not define the absence of relevant data from the medical record as an inaccuracy.
Antimicrobial recommendations were evaluated for inappropriateness by a three-person ID expert panel blinded to the accuracy of the ASP call. The panel reviewed medical record vignettes and the antimicrobial recommendation made by the ASP. The experts were given published antimicrobial treatment guidelines (predominantly from the Infectious Disease Society of America ) as well as the hospital antimicrobial guidelines and antibiogram, all from the time of the ASP call. The panelists were instructed to rate an antimicrobial regimen as inappropriate (vs. uncertain or appropriate) if it was not recommended by hospital or national guidelines and they did not believe it was a reasonable choice. Antimicrobial recommendations were defined as inappropriate for the study outcome if the panelists unanimously rated them as inappropriate. The experts also indicated the reason(s) for inappropriate recommendations from a list and/or wrote in another reason.
Three rounds of ratings were performed using a methodology similar to that pioneered by the RAND Corporation (a non-profit institution that uses research and analysis to improve policy and decision making ) [17, 18]. Each panelist completed the first round of rating individually. Panelists then reconsidered their ratings after reviewing those of other panelists. Finally, the panelists met and attempted to reach a consensus on cases where ratings differed. Cases were excluded from analysis if the experts could not reach a consensus after the final round on whether recommendations were inappropriate or not.
We calculated weighted estimates of the frequency of inaccurate communications and the frequency of inappropriate recommendations. Weighted estimates were also used to determine if calls were more accurate when the ASP practitioners were in the hospital. Weighting was based on the proportion of calls taken by fellows vs. pharmacists in the study population. We also summarized the distribution of reasons for inappropriate recommendations.
The effects of inaccurate communications and each of the potential confounders on inappropriate recommendations were evaluated in bivariable analyses, which utilized selection strata weighting, with the appropriate parametric or non-parametric test. We used a conditional logistic regression model conditioning on the individual ASP practitioner to evaluate each potential confounder . The final conditional logistic regression model was constructed with inappropriateness of the antimicrobial recommendation as the outcome, inaccurate communication as one covariate and other covariates that had both a p-value of less than 0.20 for association with the study outcome and acted as confounders or effect modifiers for the primary study association. The adjusted odds ratio (OR) for the effect of inaccurate communication on the risk of inappropriate recommendations by the ASP was then calculated as the primary study outcome.
Multiple secondary analyses were performed using the main study outcome. First, we performed a Mantel-Haenszel test for homogeneity of the effect of inaccuracy on inappropriateness across the different ASP practitioners. We also repeated the main study analysis in a sample limited to calls in which all three panelists agreed on whether the call was inappropriate or not in the first rating round (when individual decisions could not be influenced by others). Finally, we calculated the rate of agreement and kappa for the relationship between the first vs. the final rating (inappropriate or not) for each expert panelist.
Additional secondary analyses were then conducted to examine the effect of inaccurate communication of each specific type of communicated patient data on inappropriate recommendations. Using the same methods as the primary analysis, separate multivariable conditional logistic regression models were used to examine the independent effect of subtypes of inaccurate data on the study outcome. The University of Pennsylvania institutional review board approved the study.
The panel unanimously agreed on whether the antimicrobial recommendation was inappropriate or not in 163 (82%) of 200 ASP calls after three rounds of reviews. These 163 calls are the basis for the study analyses (see Figure). 106 different clinicians made the 163 calls. 48% of the 163 study calls were made by 1st-year housestaff, 24.5% by 2nd-year housestaff, 21% by callers of unknown length of training, 6% by ≥ 3rd-year housestaff, and one (0.5%) by a nurse.
Overall, 42% [95% confidence interval (CI) 34%–50%] of the calls had inappropriate antimicrobial recommendations. Common selected reasons included: antimicrobial drugs not indicated (42%), overly broad spectrum of activity (26%), wrong spectrum of activity (12%), and overly narrow spectrum of activity (8%). The panelists wrote in a reason in 7% of cases, most commonly the need for a formal infectious disease consultation.
Forty-two percent (95% CI 35%–50%) of communications included inaccuracies: 37% in one type of data and 5% in two types of data. The levels of inaccuracies for specific types of data are listed in Table 1. Calls made to physician ASP practitioners inside the hospital (i.e., with access to patient data) were inaccurate in 50% of cases vs. 38% if outside, p=0.34. The results were similar for pharmacist ASP practitioners (who had data access while in the hospital and from home; inaccuracy 56% if inside the hospital vs. 35% if outside, p=0.10).
Fifty-five percent of calls with inaccurate communication resulted in inappropriate antimicrobial drug recommendations compared to 32% of calls without inaccurate communication (in the unconditional bivariable analyses, p=0.05). Patient age and prior patient location were also significantly associated with the outcome in these analyses (Table 2). In bivariable analyses conditioned on the individual ASP practitioner, the OR for the effect of inaccurate communications on inappropriate antimicrobial recommendations was 1.8 (95% CI 0.93–3.6, p=0.08). Day of call, prior patient location, and Charlson comorbidity index were also significantly associated with the outcome in these latter analyses (Table 3).
Using multivariable conditional logistic regression to control for confounding, communication inaccuracy was significantly associated with inappropriate antimicrobial recommendations with an OR of 2.2 (95% CI 1.0–4.4, p=0.05). Charlson comorbidity index and prior patient location of intermediate or long-term care center were also significant risk factors (Table 3).
The measure of association for the effect of inaccuracy on inappropriateness was consistently greater than or equal to one across all ASP practitioners (p-value of 0.89 for rejecting the null hypothesis of homogeneity). When the primary analysis (bivariable, conditional) was limited to calls in which the panelists were unanimous after the first round of ratings, the studied association increased to an OR of 4.50 (95% confidence interval 1.4–14.3, p=0.01). We also examined the how the pattern of ratings changed for each panelist during the three rounds of ratings (Table 4). The kappa for the agreement between the first and final rating was higher for Panelist A vs. Panelists B or C. If the final rating was inappropriate, Panelists B and C were more likely to have changed their rating than Panelist A. However, if the final rating was not inappropriate, then the difference in rates between the panelists for changing ratings was less.
In bivariable sub-analyses of the effect of inaccuracies of individual types of data, only inaccuracies of microbiological test results and patient temperature were significantly associated with inappropriate antimicrobial recommendations (Table 1). Multivariable modeling using patient temperature as a covariate was not possible due to sparse data. A conditional logistic regression model was constructed that included inaccurate communication of microbiological data. After controlling for potential confounding with multivariable modeling, we found that inaccuracies of microbiological data were associated with increased odds of inappropriate antimicrobial drug recommendations (OR of 7.5, 95% CI 2.1–27.0, p=0.002). As with the prior analysis, Charlson comorbidity index and prior patient location of intermediate or long-term care center were also associated with the study outcome in this model (Table 5).
We found that inaccuracies in patient data communicated from inpatient clinicians to ASP practitioners occurred frequently and were associated with inappropriate antimicrobial recommendations by the ASP practitioners. Inaccuracies of microbiological data were also individually associated with the study outcome. The most common reason for inappropriate antimicrobial recommendations was that antimicrobials were not indicated.
Prior-approval programs must often rely on communicated patient data. Our finding of a 42% frequency of communication inaccuracies is consistent with a prior study’s finding of 39% inaccuracy using the same definitions .
Our study is one of the first to apply methodology commonly used to evaluate the quality of medical decision-making in other disciplines to the area of antimicrobial drug prescribing. The study association did not appear to be driven by a small number of repeat individual callers or certain ASP practitioners. Callers were motivated to communicate data that they believed would support the use of the antimicrobial(s) they were requesting. Thus, it is plausible that inaccurate communications were of clinically important data and thus could cause inappropriate antimicrobial recommendations. We also found that inaccuracies of microbiological data were strongly associated with inappropriate recommendations.
One of our panelists was more likely to have his initial rating be the same as his final rating, particularly when the final consensus rating was inappropriate. However, the main study association persisted when limited to calls in which there was consensus prior to the panelists’ knowledge of each other’s opinions. Thus, the study findings were not driven by ratings from any one panelist.
We did not attempt to determine if inaccurate communications were intentional or unintentional. Others have described clinicians lowering their threshold for diagnosing infections to justify prescribing antimicrobials . Another study found medical students willing to lie to colleagues . Inaccuracies may also be unintentional. Housestaff report that cross-covering a large service and poor handoffs from other providers contribute to adverse events .
Inappropriate recommendations were most often due to antimicrobial overuse, consistent with antimicrobial utilization studies . While ASPs have been shown to be beneficial, our findings suggest significant potential for improvement.
Regardless of whether inaccurate communications are intentional or not, a technological solution may be required. Computer decision support systems may streamline ASP activities . Direct access to patient data by ASP practitioners is available outside some hospitals. ASP access to computerized data did not improve the accuracy of calls in our study. These unexpected results may suggest that available data were often not examined or that distractions outside the hospital were common.
Our study has several potential limitations. The outcome was a process measure (inappropriate prescribing) rather than a clinical outcome (e.g., treatment failure). However, we have now elucidated a critical step in the causal pathway between communication errors and adverse clinical events. The medical record may have been incomplete, leading us to misclassify the accuracy of some calls. We limited the potential for misclassification due to data abstraction error by using objective definitions applied to data gathered on structured data collection sheets. There was potential for outcome misclassification due to inaccurate recording of the call by the ASP practitioner. We limited this possibility by focusing on explicitly communicated and recorded data, not including relevant data that were inadequately communicated. A different group of panelists may potentially judge inappropriateness differently for some calls. However, any misclassification of inaccuracy or inappropriateness should be non-differential, biasing the results towards the null, since those assessing the exposure and study outcome were blinded to each other’s work. We controlled for confounding by measured variables using multivariable regression modeling. While our sample size was adequate for our primary aim, we may have been underpowered in our secondary analyses to detect small to moderate effects of inaccuracy of specific data types. Finally, our results may not be generalizable beyond academic medical centers with similar prior-approval ASPs.
We have shown that inaccurate communications are associated with inappropriate antimicrobial recommendations. Other factors, such as inadequate communication of patient data, may also play a yet undefined role. Studies are needed to test and extend our findings by evaluating other causes of inappropriate recommendations, downstream clinical outcomes, and the effect of technological interventions. Finally, clinicians and ASP practitioners should confirm critical communicated data before use in prescribing decisions.
The authors thank Edmund Weisberg, MS, for his editorial assistance in preparing the manuscript, Susan Walker, RN, MSN, for data entry, and Eunice Jeon, MS, PA-C, for constructing clinical vignettes. This work was supported by the Centers for Education and Research on Therapeutics (CERTs) grant (U18-HS10399) of the National Institutes of Health (NIH) from the Agency for Healthcare Research and Quality (AHRQ); the Ruth L. Kirschstein National Research Service Award (F32-HS-023982) of the NIH from the AHRQ (D.R.L.); a University Research Foundation grant from the University of Pennsylvania (N.O.F. and D.R.L.); and the Mentored Patient Oriented Research Career Development Award (K23-AI-060887) of the NIH from the National Institute of Allergy and Infectious Diseases (D.R.L.).
All authors report no conflicts of interest relevant to this article.