|Home | About | Journals | Submit | Contact Us | Français|
Introduction.Computerized collection of standardized measures of patient reported outcomes (PROs) provides a novel paradigm for data capture at the point of clinical care. Comparisons between data from PROs and Electronic Health Records (EHR) are lacking. We compare EHR and PRO for capture of depression and substance abuse and their relationship to adherence to antiretroviral therapy (ART).
Methods.This retrospective study includes HIV-positive patients at an HIV clinic who completed an initial PRO assessment April 2008–July 2009. The questionnaire includes measures of depression (PHQ-9) and substance abuse (ASSIST). Self-reported ART adherence was modeled using separate logistic regression analyses (EHR vs PRO).
Results.The study included 782 participants. EHR vs PRO diagnosis of current substance abuse was 13% (n = 99) vs 6% (n = 45) (P < .0001), and current depression was 41% (n = 317) vs 12% (n = 97) (P < .0001). In the EHR model, neither substance abuse (OR = 1.25; 95% CI = 0.70–2.21) nor depression (OR = 0.93; 95% CI = 0.62–1.40) was significantly associated with poor ART adherence. Conversely, in the PRO model, current substance abuse (OR = 2.78; 95% CI = 1.33–5.81) and current depression (OR = 1.93; 95% CI = 1.12–3.33) were associated with poor ART adherence.
Discussions.The explanatory characteristics of the PRO model correlated best with factors known to be associated with poor ART adherence (substance abuse; depression). The computerized capture of PROs as a part of routine clinical care may prove to be a complementary and potentially transformative health informatics technology for research and patient care.
The availability of timely, accurate patient data is critical to clinical decision making at the point of care and the ability to conduct comparative effectiveness research. Historically, cohort research has relied upon data captured by clinicians during routine medical care encounters, typically drawing upon data recorded in patients’ paper medical records or electronic health records (EHR). However, the quality of data captured in routine care settings is affected by a number of factors. At the provider level, high patient volume may impact the comprehensiveness and quality of data capture . At the patient level, factors such as comfort providing sensitive information to clinicians and social desirability bias may affect data capture [2–5]. As the validity of comparative effectiveness research ultimately depends on the precision and integrity of data collected as part of routine care, these issues can negatively impact cohort research efforts [6, 7].
In 2008, the University of Alabama at Birmingham 1917 HIV/AIDS Clinic implemented the routine capture of patient reported data or outcomes (PROs) across multiple healthcare domains. PROs consist of standardized, validated patient questionnaires administered directly to patients, a mode of capture promoted by the NIH in its PROMIS initiative . Using touch-screen computer-based administration, PROs are used for direct capture of substance use, depression, tobacco use, at-risk alcohol use, and other domains using validated instruments during routine care visits. This process allows for routine, systematic capture of standardized PRO data at the point of care. Furthermore, this practice represents a new clinical data capture paradigm and has augmented the traditional model of provider-led data capture on patients treated at our site.
Historically, in the pursuit of research initiatives, clinical cohorts such as ours have relied on clinician documentation in patients’ problem lists or diagnostic coding of data on psychiatric comorbidities like substance use, depression, tobacco use, and at-risk alcohol use. PROs offer an alternative method of data capture for these domains. In the present investigation, we evaluated psychiatric comorbidities that were captured by EHR problem lists versus those captured by PROs. Specifically, we sought to compare the rates of psychiatric comorbidities (such as substance use and depression) in data captured through EHR versus PROs. Furthermore, we compared the association of psychiatric comorbidity data captured via EHR versus PROs sources with adherence to antiretroviral therapy (ART), a comprehensively studied issue in HIV care [9–13]. We hypothesized that psychiatric comorbidities including substance use and depression, as measured by the PROs, would be more strongly associated with poor adherence to ART compared with these same domains captured in the EHR, suggesting more comprehensive and accurate capture of these psychiatric comorbidities by PROs.
The 1917 HIV/AIDS Clinic was named for its street address in the 1980s to avoid stigma and is the center of outpatient HIV/AIDS care on the campus of UAB. Currently, over 1800 patients receiving primary and subspecialty HIV care at the clinic participate in the Institutional Review Board (IRB)–approved observational, clinical cohort project. The UAB 1917 HIV/AIDS Clinic uses a locally developed EHR that imports all laboratory values from the central UAB laboratory, requires electronic prescription for all medications, and contains detailed provider encounter notes. The EHR and database are quality controlled, with all provider notes reviewed within 72 hours of entry to ensure appropriate data capture regarding changes (additions or deletions) in diagnoses, allergies, and medications, including start and stop dates for antiretroviral and all other prescribed medications. New and ongoing diagnoses are maintained in patients’ active problem lists, whereas resolved diagnoses discontinued by providers are removed from active problem lists and remain part of patients’ past medical histories. Our quality control process was recently commended by the Information Integrity Coalition with a Non-Profit Gold Award for Information Integrity (2007).
The 1917 Clinic is part of a the Center for AIDS Research (CFAR) Network of Integrated Clinical Systems (CNICS), an NIH-funded (R24AI067039) and IRB-approved collaborative research network that integrates a broad range of clinical data collected through point-of-care EHR systems from multiple academic medical centers providing longitudinal medical care to individuals with HIV/AIDS [14, 15]. The assessment of PROs was integrated into routine clinical care at UAB in April 2008 as part of the CNICS initiative.
This retrospective cross-sectional cohort study included HIV-infected patients seen at the UAB 1917 HIV/AIDS Clinic between 1 April 2008 and 1 July 2009 who:
Patients routinely complete the PROs approximately every 4 months; however, only data from the initial PRO assessment were included in this study. During our study period, PROs were gradually rolled out over the first six months to ultimately include all presenting patients. A total of 1142 patients completed their initial PRO during this time. Of these patients, 867 were taking ART and were included in the analyses (85 individuals were excluded because they did not meet inclusion criteria). Patients who were unable to use the touch-screen PROs because of cognitive impairment or medical instability were also excluded from the study. Research assistants aided patients who could not complete the PROs due to physical limitations, visual impairment, inadequate literacy, or difficulty operating the computer. These patients represent a relatively small segment of current patients (~3%).
An analytic dataset was generated by querying our EHR and computerized PRO databases using MS SQL. The source of data for conditions studied was EHR problem lists (recorded by providers) and PRO measures (recorded by patients).
Patient-level sociodemographic and laboratory characteristics included age, sex, race, HIV transmission risk factor, health insurance status (public, private, or uninsured) at cohort entry, plasma HIV viral load, and CD4 cell count closest to the date of PRO assessment (nearest value ±180 days). The conditions of interest included substance use, depression, tobacco use, and at-risk alcohol use.
EHR variables were extracted from diagnoses in patient problem lists. Criteria for diagnoses from PROs are as follows. Substance use was defined as use of cocaine, opiates, methamphetamines, and/or intravenous drugs. Depression was examined as a dichotomous variable by standardized, validated diagnostic scoring of the PHQ-9 to depressed or not depressed (depression = “major depression” and “other depression”) [21, 22]. Tobacco use was defined in PROs as current tobacco use of any amount, and absence of use was classified as having smoked fewer than 20 cigarettes. Patients were identified as having “at-risk alcohol use” if the AUDIT-C was scored as “at-risk” [23–26]. For both EHR and PROs, diagnoses of substance and/or tobacco use were classified as current, prior, or never. In EHR, a diagnosis was considered “current” if it appeared in a patient’s active problem list, “prior” if a diagnosis had been discontinued from the problem list, and “never” if a diagnosis was never on the patient’s problem list. Depression and at-risk alcohol use were dichotomized in terms of current status (“yes” vs “no”) as reported in the active EHR problem list or PRO. Essentially, these data represent diagnoses: one self-reported by patients using standardized instruments (PRO), the other drawing from traditional chart diagnoses (EHR) filled by providers.
Descriptive analyses were used to illustrate sociodemographic and clinical characteristics. McNemar’s test for 2x2 tables and Bowker’s test for symmetry for 3x3 tables were used to test the agreement between the conditions identified by PROs versus EHR [28, 29]. Separate logistic regression models (one model for PROs and another model for EHR) were fitted to evaluate the association between psychiatric comorbidities (substance use, depression, tobacco use, and at-risk alcohol use) with poor ART adherence while controlling for covariates. Both models included the same covariates except for the source of the selected comorbidities captured in EHR versus PRO formats. All statistical analyses were performed using SAS software, version 9.1.3 (SAS Institute).
Among 782 study patients, the mean age was 45 ± 10 years, 53% were white (n = 411), 78% were male (n = 607), and men who have sex with men (MSM) constituted the largest HIV transmission risk group (54%, n = 426). Most patients had a CD4 count >350 cells/mm3 (68%, n = 528) and a suppressed plasma viral load (<400 copies/mL; 82%, n = 644). All patients were on ART in accordance with eligibility criteria and 38% (n = 301) were on their first regimen (Table 1).
The frequency of psychiatric comorbidities documented in EHR versus those reported in the PRO assessment is detailed in Table 2. In our study sample, current substance use was reported over twice as frequently by EHR relative to PRO diagnoses (EHR 13%, n = 99 vs PRO 6%, n = 45), while prior substance use was reported much less frequently in the EHR relative to the PRO (EHR 6%, n = 49 vs PRO 31%, n = 246). Conversely, depression was reported more than three times as frequently in patients’ EHR as in the PRO assessment (EHR 41%, n = 317 vs PRO 12%, n = 97). Statistically significant differences (P < .05) in the documentation of substance use (current, prior, never), depression (yes, no), tobacco use (current, prior, never) and at-risk alcohol use (yes, no) between EHR and the PRO assessment were observed (Table 2).
In a multivariable logistic regression model including EHR data for capture of substance use, depression, tobacco use, and at-risk alcohol use, only minority race (black/other vs white: OR = 2.25; 95% CI = 1.46–3.46) and recent nonsuppressed VL (>400 copies/mL: OR = 1.92; 95% CI = 1.20–3.06) were associated with poor ART adherence. No other factors had a statistically significant association with poor ART adherence, including depression and substance use.
We conducted a separate, multivariable logistic regression model using the same covariates except PRO data was used to measure substance use, depression, tobacco use, and at-risk alcohol use. Race (black/other vs white: OR = 2.48; 95% CI = 1.59–3.86) and recent VL (suppressed vs nonsuppressed: OR = 1.88; 95% CI = 1.17–3.01) were again associated with poor ART adherence. In addition, substance use (current vs never: OR = 2.78; 95% CI = 1.33–5.81; and prior vs never: OR = 1.62; 95% CI = 1.01–2.61) and the presence of depression (OR = 1.93; 95% CI = 1.12–3.33) were associated with an increased risk for poor ART adherence when captured via the PRO assessment (Table 3).
The relative accuracy of documentation of psychiatric comorbidities used in biomedical research is rarely addressed. To our knowledge, our study is among the first to contrast the content of patient reported outcomes (PROs) data with traditional medical record diagnoses, which have traditionally been used in HIV clinical cohort studies. These psychiatric comorbidities have been linked to decreased ART adherence in the past and have been found to have important implications for morbidity and mortality. Our research suggests that PROs present an alternative method of capturing data on these domains. Indeed, PROs may provide additional benefits to provider capture because they call for updates of current versus previous status on each condition. Further, the decreased influence of social desirability bias may improve the reliability of data capture for sensitive domains such as substance use. PROs also offer the opportunity for intervention at clinical presentation because providers or other members of the health care team can be automatically alerted of issues detected by PROs. We have done so with the presence of suicidal ideation .
When comparing data captured through EHR to data from PROs, we found significant differences in the prevalence of substance use, depression, tobacco use, and at-risk alcohol use among HIV-infected patients in routine clinical care. These differences were found despite standardized provider training in the use of EHR and an additional layer of note review by a third party, routinely used for data entry quality control at our site. Provider, patient, and structural factors are likely to influence data capture across these domains in routine care. At the provider level, longstanding diagnoses such as depression and substance use may not undergo thorough reevaluation at every visit owing to time constraints or focus being distracted by competing clinical events. Thus, longstanding diagnoses may persist in active problem lists. At the patient level, some individuals may be uneasy discussing certain topics directly with their provider, resulting in incomplete data capture. Studies have shown that data capture for sensitive domains (substance use, sexual risks, and so on) is done more effectively by PROs than healthcare providers. In part due to social desirability bias, many patients report more truthful responses to computer-administered surveys than to face-to-face interviews across sensitive domains [4, 5, 30–35]. PRO assessments offer added value, revealing the state of the condition in real time and offering the provider an opportunity for intervention. Finally, on a structural level, PROs offer a more consistent and dynamically updated data capture platform that, if well integrated into clinic workflow, can provide efficient and standardized data capture. A benefit of such a system is that it is patient-dependent and would remain relatively unaffected by daily provider workload fluctuations and other forces that currently affect data capture at the point of care (eg, number of scheduled patients for a given clinic).
We also found significant differences in the association of these findings with poor ART adherence. The relationship between minority race, substance use, depression, and poor ART adherence has been extensively documented and well established in the HIV/AIDS literature in many prospective and retrospective trials [1, 9–13, 36]. While an increased risk for poor ART adherence for minority race and uncontrolled viral loads was found in both EHR and PRO models, the well-known associations between substance use, depression, and poor adherence were only found in the model utilizing self-reported PRO data. In essence, the model that employed PRO-captured variables for these domains corresponded with the extant literature.
We posit the concordance of the PRO model with the known risk factors for poor ART adherence suggests increased accuracy of these data across specific domains; it also suggests that the PROs present an opportunity to enhance the quality of data elements available for comparative effectiveness research and patient care. PROs provide an adjunct screening measure for data capture at the point of care for use in clinical practice to address barriers to ART adherence. While PRO data was immediately available to clinicians, it had not been fully implemented to routine care at the time of this study. To fully realize the benefits of PRO data capture, operational and implementation research must be conducted to find ways to integrate the capture of such instruments routinely in outpatient and potentially inpatient practice settings.
Our findings should be interpreted with respect to the limitations of our study. As an observational study from a single HIV cohort, our findings may not be applicable to other national or international settings, although our analysis may provide insights applicable to such settings. As with all observational studies, we are able to identify associations but cannot attribute causality. Although we controlled for measured confounders using multivariable models, potential for unmeasured confounding inherent to all observational studies persists. In another potential limitation, PROs at the 1917 Clinic are administered by a touch-screen system without an audio counterpart. Although assistance was provided to patients with visual impairment or inadequate literacy, these factors were not controlled for in this analysis and there is not a clinic-wide measure of patient literacy at this time. In addition, we are not able to directly assess the respective diagnostic accuracy of the EHR and PRO data through comparison to a third source; rather, we used associations of well established psychiatric comorbidities with poor ART adherence as indirect evidence of improved data quality and accuracy with PROs. Further research using structured clinical interviews as a gold standard to compare data from EHR and PROs may help to evaluate the relative accuracy of PRO findings.
In conclusion, our findings suggest that PROs represent an important resource to HIV/AIDS clinics at both the level of clinical care and cohort research. Ultimately, integrating PROs at the point of care in a routine clinical setting may represent a transformative health informatics for routine clinical care and comparative effectiveness research.
This work was supported by the National Institutes of Health (T35 HL007473-28 and K23MH082641) and the IDSA Medical Scholars program. The UAB 1917 HIV/AIDS Clinic Cohort Observational Database project receives financial support from the following: UAB Center for AIDS Research (P30-AI27767), Center for Aids Research Network of Integrated Clinical Systems (1 R24 AI067039-1), and the Mary Fisher CARE Fund.
M. J. M. has received recent research support from Tibotec Therapeutics, Merk Foundation, and Pfizer and has consulted for Bristol-Myers Squibb and Gilead Sciences. C. M. has received research funding and/or consulted for Boehringer Ingelheim Pharmaceuticals. J. E. S. has received research funding and/or consulted for Boehringer Ingelheim Pharmaceuticals. M. M. K. has received research funding and/or consulted for Gilead. M. S. S. has received research support or served as a consultant for Ardea, Avexa, Boehringer Ingelheim, Bristol-Myers Squibb, Gilead, Merck, Monogram Biosciences, Progenics, Tibotec Therapeutics, and Vertex. J. H. W. has received research support from the Bristol-Myers Squibb Virology Fellows Research Program for the 2006–2008 academic years, Pfizer, Tibotec Therapeutics, and Definicare, and has consulted for Bristol-Myers Squibb and Gilead Sciences.
All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.