PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of cidLink to Publisher's site
 
Clin Infect Dis. Jan 1, 2012; 54(1): 141–147.
Published online Oct 31, 2011. doi:  10.1093/cid/cir727
PMCID: PMC3243652
Patient Reported Outcomes in Routine Care: Advancing Data Capture for HIV Cohort Research
Michael S. Kozak,corresponding author1 Michael J. Mugavero,1 Jiatao Ye,2 Inmaculada Aban,2 Sarah T. Lawrence,1 Christa R. Nevin,1 James L. Raper,1 Cheryl McCullumsmith,3 Joseph E. Schumacher,4 Heidi M. Crane,5 Mari M. Kitahata,5 Michael S. Saag,1 and James H. Willig1
1Department of Internal Medicine, Division of Infectious Diseases
2Department of Biostatistics, School of Public Health
3Department of Psychiatry
4Department of Internal Medicine, Division of Preventative Medicine, University of Alabama at Birmingham
5Department of Medicine, University of Washington, Seattle
corresponding authorCorresponding author.
Correspondence: Michael S. Kozak, BA, Department of Internal Medicine, Division of Infectious Diseases, University of Alabama at Birmingham, CCB 174, 908 20th St So, Birmingham, AL 35294-2050 (mkozak/at/uab.edu).
Presented in part: International Association of Physicians in AIDS Care, Adherence Conference, Miami, Florida, 23–25 May 2010.
Received July 7, 2011; Accepted August 30, 2011.
Introduction. Computerized collection of standardized measures of patient reported outcomes (PROs) provides a novel paradigm for data capture at the point of clinical care. Comparisons between data from PROs and Electronic Health Records (EHR) are lacking. We compare EHR and PRO for capture of depression and substance abuse and their relationship to adherence to antiretroviral therapy (ART).
Methods. This retrospective study includes HIV-positive patients at an HIV clinic who completed an initial PRO assessment April 2008–July 2009. The questionnaire includes measures of depression (PHQ-9) and substance abuse (ASSIST). Self-reported ART adherence was modeled using separate logistic regression analyses (EHR vs PRO).
Results. The study included 782 participants. EHR vs PRO diagnosis of current substance abuse was 13% (n = 99) vs 6% (n = 45) (P < .0001), and current depression was 41% (n = 317) vs 12% (n = 97) (P < .0001). In the EHR model, neither substance abuse (OR = 1.25; 95% CI = 0.70–2.21) nor depression (OR = 0.93; 95% CI = 0.62–1.40) was significantly associated with poor ART adherence. Conversely, in the PRO model, current substance abuse (OR = 2.78; 95% CI = 1.33–5.81) and current depression (OR = 1.93; 95% CI = 1.12–3.33) were associated with poor ART adherence.
Discussions. The explanatory characteristics of the PRO model correlated best with factors known to be associated with poor ART adherence (substance abuse; depression). The computerized capture of PROs as a part of routine clinical care may prove to be a complementary and potentially transformative health informatics technology for research and patient care.
The availability of timely, accurate patient data is critical to clinical decision making at the point of care and the ability to conduct comparative effectiveness research. Historically, cohort research has relied upon data captured by clinicians during routine medical care encounters, typically drawing upon data recorded in patients’ paper medical records or electronic health records (EHR). However, the quality of data captured in routine care settings is affected by a number of factors. At the provider level, high patient volume may impact the comprehensiveness and quality of data capture [1]. At the patient level, factors such as comfort providing sensitive information to clinicians and social desirability bias may affect data capture [25]. As the validity of comparative effectiveness research ultimately depends on the precision and integrity of data collected as part of routine care, these issues can negatively impact cohort research efforts [6, 7].
In 2008, the University of Alabama at Birmingham 1917 HIV/AIDS Clinic implemented the routine capture of patient reported data or outcomes (PROs) across multiple healthcare domains. PROs consist of standardized, validated patient questionnaires administered directly to patients, a mode of capture promoted by the NIH in its PROMIS initiative [8]. Using touch-screen computer-based administration, PROs are used for direct capture of substance use, depression, tobacco use, at-risk alcohol use, and other domains using validated instruments during routine care visits. This process allows for routine, systematic capture of standardized PRO data at the point of care. Furthermore, this practice represents a new clinical data capture paradigm and has augmented the traditional model of provider-led data capture on patients treated at our site.
Historically, in the pursuit of research initiatives, clinical cohorts such as ours have relied on clinician documentation in patients’ problem lists or diagnostic coding of data on psychiatric comorbidities like substance use, depression, tobacco use, and at-risk alcohol use. PROs offer an alternative method of data capture for these domains. In the present investigation, we evaluated psychiatric comorbidities that were captured by EHR problem lists versus those captured by PROs. Specifically, we sought to compare the rates of psychiatric comorbidities (such as substance use and depression) in data captured through EHR versus PROs. Furthermore, we compared the association of psychiatric comorbidity data captured via EHR versus PROs sources with adherence to antiretroviral therapy (ART), a comprehensively studied issue in HIV care [913]. We hypothesized that psychiatric comorbidities including substance use and depression, as measured by the PROs, would be more strongly associated with poor adherence to ART compared with these same domains captured in the EHR, suggesting more comprehensive and accurate capture of these psychiatric comorbidities by PROs.
The 1917 HIV/AIDS Clinic was named for its street address in the 1980s to avoid stigma and is the center of outpatient HIV/AIDS care on the campus of UAB. Currently, over 1800 patients receiving primary and subspecialty HIV care at the clinic participate in the Institutional Review Board (IRB)–approved observational, clinical cohort project. The UAB 1917 HIV/AIDS Clinic uses a locally developed EHR that imports all laboratory values from the central UAB laboratory, requires electronic prescription for all medications, and contains detailed provider encounter notes. The EHR and database are quality controlled, with all provider notes reviewed within 72 hours of entry to ensure appropriate data capture regarding changes (additions or deletions) in diagnoses, allergies, and medications, including start and stop dates for antiretroviral and all other prescribed medications. New and ongoing diagnoses are maintained in patients’ active problem lists, whereas resolved diagnoses discontinued by providers are removed from active problem lists and remain part of patients’ past medical histories. Our quality control process was recently commended by the Information Integrity Coalition with a Non-Profit Gold Award for Information Integrity (2007).
The 1917 Clinic is part of a the Center for AIDS Research (CFAR) Network of Integrated Clinical Systems (CNICS), an NIH-funded (R24AI067039) and IRB-approved collaborative research network that integrates a broad range of clinical data collected through point-of-care EHR systems from multiple academic medical centers providing longitudinal medical care to individuals with HIV/AIDS [14, 15]. The assessment of PROs was integrated into routine clinical care at UAB in April 2008 as part of the CNICS initiative.
Study Sample and Procedures
This retrospective cross-sectional cohort study included HIV-infected patients seen at the UAB 1917 HIV/AIDS Clinic between 1 April 2008 and 1 July 2009 who:
  • Were receiving ART;
  • Successfully completed the PRO measures including substance use (Alcohol, Smoking and Substance Involve-ment Screening Test [ASSIST]), depression (Patient Health Questionnaire-9 [PHQ-9]), at-risk alcohol use (Alcohol Use Disorders Test-Consumption [AUDIT-C]), ART adherence (AIDS Clinical Trials Unit [ACTU-4]), and tobacco use domains; and
  • Had viral load (VL) and immunologic (CD4) testing within 6 months (nearest value ±180 days) of the PRO administration date [1620].
Patients routinely complete the PROs approximately every 4 months; however, only data from the initial PRO assessment were included in this study. During our study period, PROs were gradually rolled out over the first six months to ultimately include all presenting patients. A total of 1142 patients completed their initial PRO during this time. Of these patients, 867 were taking ART and were included in the analyses (85 individuals were excluded because they did not meet inclusion criteria). Patients who were unable to use the touch-screen PROs because of cognitive impairment or medical instability were also excluded from the study. Research assistants aided patients who could not complete the PROs due to physical limitations, visual impairment, inadequate literacy, or difficulty operating the computer. These patients represent a relatively small segment of current patients (~3%).
Study Variables
An analytic dataset was generated by querying our EHR and computerized PRO databases using MS SQL. The source of data for conditions studied was EHR problem lists (recorded by providers) and PRO measures (recorded by patients).
Independent Variables
Patient-level sociodemographic and laboratory characteristics included age, sex, race, HIV transmission risk factor, health insurance status (public, private, or uninsured) at cohort entry, plasma HIV viral load, and CD4 cell count closest to the date of PRO assessment (nearest value ±180 days). The conditions of interest included substance use, depression, tobacco use, and at-risk alcohol use.
EHR variables were extracted from diagnoses in patient problem lists. Criteria for diagnoses from PROs are as follows. Substance use was defined as use of cocaine, opiates, methamphetamines, and/or intravenous drugs. Depression was examined as a dichotomous variable by standardized, validated diagnostic scoring of the PHQ-9 to depressed or not depressed (depression = “major depression” and “other depression”) [21, 22]. Tobacco use was defined in PROs as current tobacco use of any amount, and absence of use was classified as having smoked fewer than 20 cigarettes. Patients were identified as having “at-risk alcohol use” if the AUDIT-C was scored as “at-risk” [2326]. For both EHR and PROs, diagnoses of substance and/or tobacco use were classified as current, prior, or never. In EHR, a diagnosis was considered “current” if it appeared in a patient’s active problem list, “prior” if a diagnosis had been discontinued from the problem list, and “never” if a diagnosis was never on the patient’s problem list. Depression and at-risk alcohol use were dichotomized in terms of current status (“yes” vs “no”) as reported in the active EHR problem list or PRO. Essentially, these data represent diagnoses: one self-reported by patients using standardized instruments (PRO), the other drawing from traditional chart diagnoses (EHR) filled by providers.
Dependent Variable
The primary outcome measure was adherence to ART, which was measured as part of the PRO assessment using questions from the ACTU-4 [20, 27]. Patients who, at the time of PRO administration, reported missing ≥1 ART dose within the prior 2 weeks were considered poorly adherent.
Statistical Analyses
Descriptive analyses were used to illustrate sociodemographic and clinical characteristics. McNemar’s test for 2x2 tables and Bowker’s test for symmetry for 3x3 tables were used to test the agreement between the conditions identified by PROs versus EHR [28, 29]. Separate logistic regression models (one model for PROs and another model for EHR) were fitted to evaluate the association between psychiatric comorbidities (substance use, depression, tobacco use, and at-risk alcohol use) with poor ART adherence while controlling for covariates. Both models included the same covariates except for the source of the selected comorbidities captured in EHR versus PRO formats. All statistical analyses were performed using SAS software, version 9.1.3 (SAS Institute).
Among 782 study patients, the mean age was 45 ± 10 years, 53% were white (n = 411), 78% were male (n = 607), and men who have sex with men (MSM) constituted the largest HIV transmission risk group (54%, n = 426). Most patients had a CD4 count >350 cells/mm3 (68%, n = 528) and a suppressed plasma viral load (<400 copies/mL; 82%, n = 644). All patients were on ART in accordance with eligibility criteria and 38% (n = 301) were on their first regimen (Table 1).
Table 1.
Table 1.
Demographic and Baseline Characteristics of 782 Patients on Antiretroviral Therapy at the Time of Their First Patient Reported Outcome Between April 2008 and July 2009
The frequency of psychiatric comorbidities documented in EHR versus those reported in the PRO assessment is detailed in Table 2. In our study sample, current substance use was reported over twice as frequently by EHR relative to PRO diagnoses (EHR 13%, n = 99 vs PRO 6%, n = 45), while prior substance use was reported much less frequently in the EHR relative to the PRO (EHR 6%, n = 49 vs PRO 31%, n = 246). Conversely, depression was reported more than three times as frequently in patients’ EHR as in the PRO assessment (EHR 41%, n = 317 vs PRO 12%, n = 97). Statistically significant differences (P < .05) in the documentation of substance use (current, prior, never), depression (yes, no), tobacco use (current, prior, never) and at-risk alcohol use (yes, no) between EHR and the PRO assessment were observed (Table 2).
Table 2.
Table 2.
Psychiatric Comorbidities as Captured by the Electronic Health Records or Measured by the Patient Reported Outcomes (PRO) Assessment in 782 Patients on Antiretroviral Therapy at the Time of Their First PRO Between April 2008 and July 2009
In a multivariable logistic regression model including EHR data for capture of substance use, depression, tobacco use, and at-risk alcohol use, only minority race (black/other vs white: OR = 2.25; 95% CI = 1.46–3.46) and recent nonsuppressed VL (>400 copies/mL: OR = 1.92; 95% CI = 1.20–3.06) were associated with poor ART adherence. No other factors had a statistically significant association with poor ART adherence, including depression and substance use.
We conducted a separate, multivariable logistic regression model using the same covariates except PRO data was used to measure substance use, depression, tobacco use, and at-risk alcohol use. Race (black/other vs white: OR = 2.48; 95% CI = 1.59–3.86) and recent VL (suppressed vs nonsuppressed: OR = 1.88; 95% CI = 1.17–3.01) were again associated with poor ART adherence. In addition, substance use (current vs never: OR = 2.78; 95% CI = 1.33–5.81; and prior vs never: OR = 1.62; 95% CI = 1.01–2.61) and the presence of depression (OR = 1.93; 95% CI = 1.12–3.33) were associated with an increased risk for poor ART adherence when captured via the PRO assessment (Table 3).
Table 3.
Table 3.
Logistic Regression of the Associations Between Poor Antiretroviral Therapy (ART) Adherence and Demographic and Clinical Characteristics Including Psychiatric Comorbidities Measured by Both the Electronic Health Records and the Patient Reported Outcomes (more ...)
The relative accuracy of documentation of psychiatric comorbidities used in biomedical research is rarely addressed. To our knowledge, our study is among the first to contrast the content of patient reported outcomes (PROs) data with traditional medical record diagnoses, which have traditionally been used in HIV clinical cohort studies. These psychiatric comorbidities have been linked to decreased ART adherence in the past and have been found to have important implications for morbidity and mortality. Our research suggests that PROs present an alternative method of capturing data on these domains. Indeed, PROs may provide additional benefits to provider capture because they call for updates of current versus previous status on each condition. Further, the decreased influence of social desirability bias may improve the reliability of data capture for sensitive domains such as substance use. PROs also offer the opportunity for intervention at clinical presentation because providers or other members of the health care team can be automatically alerted of issues detected by PROs. We have done so with the presence of suicidal ideation [15].
When comparing data captured through EHR to data from PROs, we found significant differences in the prevalence of substance use, depression, tobacco use, and at-risk alcohol use among HIV-infected patients in routine clinical care. These differences were found despite standardized provider training in the use of EHR and an additional layer of note review by a third party, routinely used for data entry quality control at our site. Provider, patient, and structural factors are likely to influence data capture across these domains in routine care. At the provider level, longstanding diagnoses such as depression and substance use may not undergo thorough reevaluation at every visit owing to time constraints or focus being distracted by competing clinical events. Thus, longstanding diagnoses may persist in active problem lists. At the patient level, some individuals may be uneasy discussing certain topics directly with their provider, resulting in incomplete data capture. Studies have shown that data capture for sensitive domains (substance use, sexual risks, and so on) is done more effectively by PROs than healthcare providers. In part due to social desirability bias, many patients report more truthful responses to computer-administered surveys than to face-to-face interviews across sensitive domains [4, 5, 3035]. PRO assessments offer added value, revealing the state of the condition in real time and offering the provider an opportunity for intervention. Finally, on a structural level, PROs offer a more consistent and dynamically updated data capture platform that, if well integrated into clinic workflow, can provide efficient and standardized data capture. A benefit of such a system is that it is patient-dependent and would remain relatively unaffected by daily provider workload fluctuations and other forces that currently affect data capture at the point of care (eg, number of scheduled patients for a given clinic).
We also found significant differences in the association of these findings with poor ART adherence. The relationship between minority race, substance use, depression, and poor ART adherence has been extensively documented and well established in the HIV/AIDS literature in many prospective and retrospective trials [1, 913, 36]. While an increased risk for poor ART adherence for minority race and uncontrolled viral loads was found in both EHR and PRO models, the well-known associations between substance use, depression, and poor adherence were only found in the model utilizing self-reported PRO data. In essence, the model that employed PRO-captured variables for these domains corresponded with the extant literature.
We posit the concordance of the PRO model with the known risk factors for poor ART adherence suggests increased accuracy of these data across specific domains; it also suggests that the PROs present an opportunity to enhance the quality of data elements available for comparative effectiveness research and patient care. PROs provide an adjunct screening measure for data capture at the point of care for use in clinical practice to address barriers to ART adherence. While PRO data was immediately available to clinicians, it had not been fully implemented to routine care at the time of this study. To fully realize the benefits of PRO data capture, operational and implementation research must be conducted to find ways to integrate the capture of such instruments routinely in outpatient and potentially inpatient practice settings.
Our findings should be interpreted with respect to the limitations of our study. As an observational study from a single HIV cohort, our findings may not be applicable to other national or international settings, although our analysis may provide insights applicable to such settings. As with all observational studies, we are able to identify associations but cannot attribute causality. Although we controlled for measured confounders using multivariable models, potential for unmeasured confounding inherent to all observational studies persists. In another potential limitation, PROs at the 1917 Clinic are administered by a touch-screen system without an audio counterpart. Although assistance was provided to patients with visual impairment or inadequate literacy, these factors were not controlled for in this analysis and there is not a clinic-wide measure of patient literacy at this time. In addition, we are not able to directly assess the respective diagnostic accuracy of the EHR and PRO data through comparison to a third source; rather, we used associations of well established psychiatric comorbidities with poor ART adherence as indirect evidence of improved data quality and accuracy with PROs. Further research using structured clinical interviews as a gold standard to compare data from EHR and PROs may help to evaluate the relative accuracy of PRO findings.
In conclusion, our findings suggest that PROs represent an important resource to HIV/AIDS clinics at both the level of clinical care and cohort research. Ultimately, integrating PROs at the point of care in a routine clinical setting may represent a transformative health informatics for routine clinical care and comparative effectiveness research.
Notes
Financial support.
This work was supported by the National Institutes of Health (T35 HL007473-28 and K23MH082641) and the IDSA Medical Scholars program. The UAB 1917 HIV/AIDS Clinic Cohort Observational Database project receives financial support from the following: UAB Center for AIDS Research (P30-AI27767), Center for Aids Research Network of Integrated Clinical Systems (1 R24 AI067039-1), and the Mary Fisher CARE Fund.
Potential conflicts of interest.
M. J. M. has received recent research support from Tibotec Therapeutics, Merk Foundation, and Pfizer and has consulted for Bristol-Myers Squibb and Gilead Sciences. C. M. has received research funding and/or consulted for Boehringer Ingelheim Pharmaceuticals. J. E. S. has received research funding and/or consulted for Boehringer Ingelheim Pharmaceuticals. M. M. K. has received research funding and/or consulted for Gilead. M. S. S. has received research support or served as a consultant for Ardea, Avexa, Boehringer Ingelheim, Bristol-Myers Squibb, Gilead, Merck, Monogram Biosciences, Progenics, Tibotec Therapeutics, and Vertex. J. H. W. has received research support from the Bristol-Myers Squibb Virology Fellows Research Program for the 2006–2008 academic years, Pfizer, Tibotec Therapeutics, and Definicare, and has consulted for Bristol-Myers Squibb and Gilead Sciences.
All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.
1. Pakhomov SV, Jacobsen SJ, Chute CG, Roger VL. Agreement between patient-reported symptoms and their documentation in the medical record. Am J Manag Care. 2008;14:530–9. [PMC free article] [PubMed]
2. Cruz CO, Meshberg EB, Shofer FS, McCusker CM, Chang AM, Hollander JE. Interrater reliability and accuracy of clinicians and trained research assistants performing prospective data collection in emergency department patients with potential acute coronary syndrome. Ann Emerg Med. 2009;54:1–7. [PubMed]
3. Goulet F, Jacques A, Gagnon R, Racette P, Sieber W. Assessment of family physicians’ performance using patient charts: Interrater reliability and concordance with chart-stimulated recall interview. Eval Health Prof. 2007;30:376–92. [PubMed]
4. Des Jarlais DC, Paone D, Milliken J, et al. Audio-computer interviewing to measure risk behaviour for HIV among injecting drug users: a quasi-randomised trial. Lancet. 1999;353:1657–61. [PubMed]
5. Metzger DS, Koblin B, Turner C, et al. HIVNET Vaccine Preparedness Study Protocol Team. Randomized controlled trial of audio computer-assisted self-interviewing: Utility and acceptability in longitudinal studies. Am J Epidemiol. 2000;152:99–106. [PubMed]
6. Commmittee on Comparative Effectiveness Research Prioritization, Board on Health Care Services, Institute of Medicine. Initial national priorities for comparitive effectiveness research. Washington, DC: The National Academies Press; 2009. Available at: http://www.nap.edu/catalog.php?record_id=12648.
7. Lau B, Gange SJ, Kirk GD, Moore RD. Evaluation of human immunodeficiency virus biomarkers: Inferences from interval and clinical cohort studies. Epidemiology. 2009;20:664–72. [PMC free article] [PubMed]
8. Patient-Reported Outcomes Measurement Systems. PROMIS mission, vision and goals. Available at: http://www.nihpromis.org/about/missionvisiongoals. Accessed 7 July 2011.
9. Gordillo V, del Amo J, Soriano V, Gonzalez-Lahoz J. Sociodemographic and psychological variables influencing adherence to antiretroviral therapy. AIDS. 1999;13:1763–9. [PubMed]
10. Tucker JS, Burnam MA, Sherbourne CD, Kung FY, Gifford AL. Substance use and mental health correlates of nonadherence to antiretroviral medications in a sample of patients with human immunodeficiency virus infection. Am J Med. 2003;114:573–80. [PubMed]
11. Mugavero M, Ostermann J, Whetten K, et al. Barriers to antiretroviral adherence: The importance of depression, abuse, and other traumatic events. AIDS Patient Care STDS. 2006;20:418–28. [PubMed]
12. Arnsten JH, Demas PA, Grant RW, et al. Impact of active drug use on antiretroviral therapy adherence and viral suppression in HIV-infected drug users. J Gen Intern Med. 2002;17:377–81. [PMC free article] [PubMed]
13. Hinkin CH, Hardy DJ, Mason KI, et al. Medication adherence in HIV-infected adults: Effect of patient age, cognitive status, and substance abuse. AIDS. 2004;18(Suppl 1):S19–25. [PMC free article] [PubMed]
14. Kitahata MM, Rodriguez B, Haubrich R, et al. Cohort profile: the Centers for AIDS Research Network of Integrated Clinical Systems. Int J Epidemiol. 2008;37:948–55. [PMC free article] [PubMed]
15. Lawrence ST, Willig JH, Crane HM, et al. Routine, self-administered, touch-screen, computer-based suicidal ideation assessment linked to automated response team notification in an HIV primary care setting. Clin Infect Dis. 2010;50:1165–73. [PMC free article] [PubMed]
16. Bradley KA, Bush KR, Epler AJ, et al. Two brief alcohol-screening tests from the Alcohol Use Disorders Identification Test (AUDIT): Validation in a female Veterans Affairs patient population. Arch Intern Med. 2003;163:821–9. [PubMed]
17. Newcombe DA, Humeniuk RE, Ali R. Validation of the World Health Organization Alcohol, Smoking and Substance Involvement Screening Test (ASSIST): Report of results from the Australian site. Drug Alcohol Rev. 2005;24:217–26. [PubMed]
18. Crane HM, Lober W, Webster E, et al. Routine collection of patient-reported outcomes in an HIV clinic setting: the first 100 patients. Curr HIV Res. 2007;5:109–18. [PubMed]
19. Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire. JAMA. 1999;282:1737–44. [PubMed]
20. Chesney MA, Ickovics JR, Chambers DB, et al. Self-reported adherence to antiretroviral medications among participants in HIV clinical trials: the AACTG adherence instruments. Patient Care Committee and Adherence Working Group of the Outcomes Committee of the Adult AIDS Clinical Trials Group (AACTG) AIDS Care. 2000;12:255–66. [PubMed]
21. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med. 2001;16:606–13. [PMC free article] [PubMed]
22. Kroenke K, Spitzer R. The PHQ-9: A new depression diagnostic and severity measure. Psychiatr Ann. 2002;32:509–21.
23. Saunders JB, Aasland OG, Barbor TF, de la Fuente JR. Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO Collaborative Project on Early Detection of Persons with Harmful Alcohol Consumption–II. Addiction. 1993;88:791–804. [PubMed]
24. Bradley KA, Bush KR, McDonell MB, Malone T, Fihn SD. Screening for problem drinking: Comparison of CAGE and AUDIT. J Gen Intern Med. 1998;13:379–88. [PMC free article] [PubMed]
25. Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA. The AUDIT alcohol consumption questions (AUDIT-C): An effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Alcohol Use Disorders Identification Test. Arch Intern Med. 1998;158:1789–95. [PubMed]
26. Gual A, Segura L, Contel M, Heather N, Colom J. Audit-3 and audit-4: Effectiveness of two short forms of the Alcohol Use Disorders Identification Test. Alcohol Alcohol. 2002;37:591–6. [PubMed]
27. Simoni JM, Kurth AE, Pearson CR, Pantalone DW, Merrill JO, Frick PA. Self-report measures of antiretroviral therapy adherence: A review with recommendations for HIV research and clinical management. AIDS Behav. 2006;10:227–45. [PubMed]
28. McNemar Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika. 1947;12:153–7. [PubMed]
29. Bowker AH. A test for symmetry in contingency tables. J Am Stat Assoc. 1948;43:572–4. [PubMed]
30. Kobak KA, Greist JH, Jefferson JW, Katzelnick DJ. Computer-administered clinical rating scales. A review. Psychopharmacology (Berl) 1996;127:291–301. [PubMed]
31. Lucas RW, Mullin PJ, Luna CB, McInroy DC. Psychiatrists and a computer as interrogators of patients with alcohol-related illnesses: a comparison. Br J Psychiatry. 1977;131:160–7. [PubMed]
32. Greist JH, Gustafson DH, Stauss FF, Rowse GL, Laughren TP, Chiles JA. A computer interview for suicide-risk prediction. Am J Psychiatry. 1973;130:1327–32. [PubMed]
33. Fann JR, Berry DL, Wolpin S, et al. Depression screening using the Patient Health Questionnaire-9 administered on a touch screen computer. Psychooncology. 2009;18:14–22. [PMC free article] [PubMed]
34. Rosenthal R, Persinger G, Fode KL. Experimenter bias, anxiety, and social desirability. Percept Mot Skills. 1962;15:73–4. [PubMed]
35. Ghanem KG, Hutton HE, Zenilman JM, Zimba R, Erbelding EJ. Audio computer assisted self interview and face to face interview modes in assessing response bias among STD clinic patients. Sex Transm Infect. 2005;81:421–5. [PMC free article] [PubMed]
36. Oh DL, Sarafian F, Silvestre A, et al. Evaluation of adherence and factors affecting adherence to combination antiretroviral therapy among white, Hispanic, and black men in the MACS Cohort. J Acquir Immune Defic Syndr. 2009;52:290–3. [PMC free article] [PubMed]
Articles from Clinical Infectious Diseases: An Official Publication of the Infectious Diseases Society of America are provided here courtesy of
Oxford University Press