PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Ann Emerg Med. Author manuscript; available in PMC 2011 November 11.
Published in final edited form as:
PMCID: PMC3213856
NIHMSID: NIHMS334419

The Six-Item Screener and AD8 for the Detection of Cognitive Impairment in Geriatric Emergency Department Patients

Abstract

Study objective

We evaluate the diagnostic test characteristics of the Six-Item Screener and the AD8 to detect cognitive dysfunction in adults older than 65 years and using the emergency department (ED) for any reason.

Methods

We conducted an observational cross-sectional cohort study at a single academic urban university-affiliated hospital. Subjects were consenting, non-critically ill, English-speaking adults older than 65 years and receiving care in the ED. We quantitatively assessed the diagnostic test characteristics of the Six-Item Screener and AD8 by using the Mini-Mental State Examination score less than 24 as the criterion standard for cognitive dysfunction.

Results

The prevalence of cognitive dysfunction was 35%, but only 6% of charts noted a pre-existing deficit. The Six-Item Screener was superior to either the caregiver-administered AD8 or the patient-administered AD8 for the detection of cognitive dysfunction.

Conclusion

The Six-Item Screener was superior to the caregiver- or patient-administered AD8 to identify older adults at increased risk for occult cognitive dysfunction.

INTRODUCTION

Background and Importance

Aging baby boomers will cause emergency departments (EDs) to care for unprecedented numbers of geriatric adults during the next 3 decades.1,2 The modern emergency care model is often poorly equipped to recognize and manage the unique constellation of aging pathology called geriatric syndromes.3,4 Cognitive dysfunction is one such geriatric syndrome and includes mild cognitive impairment, delirium, and dementia. Multiple observational trials have demonstrated that emergency physicians and nurses often fail to recognize cognitive dysfunction.5-9 Inpatient and outpatient physicians also fail to recognize impaired cognition.9-11 In fact, more than 70% of ED patients with cognitive dysfunction lack a previous diagnosis of dementia.8

Prompt recognition of cognitive dysfunction is essential for high-quality geriatric emergency care for several reasons. First, the prevalence of cognitive impairment is 23% to 40% in older ED patients.6,8,12-16 In the United States, 5.3 million persons experience Alzheimer’s dementia, at a health care cost of $148 billion per year.17 By 2050, 1 in 85 persons will live with Alzheimer’s dementia. If interventions could delay disease progression by just 1 year, 9.2 million fewer patients would require the expense of higher-level care.18 Second, unrecognized cognitive impairment may impede effective emergency care. Already, older adults use emergency care more frequently than younger populations and have longer lengths of stay while undergoing more diagnostic testing and more frequent admissions.19-22 Despite this disproportionate allocation of resources, geriatric adults have higher rates of ED recidivism and frequently report not understanding their emergency care instructions.19,22,23 Cognitive impairment may impede medical comprehension and accurate elicitation of details germane to the chief complaint.20,23-25

Goal of This Investigation

The importance of cognitive dysfunction screening within emergency medicine has recently been underscored by the development of geriatric quality indicators that include testing for cognitive dysfunction as a minimal standard of care.26 Furthermore, multiple stakeholders have also defined the recognition of cognitive impairment in older adults as a minimal core competency for emergency medicine residents.27 However, one of the barriers to effective cognitive screening is the lack of brief, ED-valid screening instruments.16,28 The Six-Item Screener (SIS) (Figure 1) is one instrument that can be administered to patients without the need for extra materials or equipment.29 Unfortunately, previous multicenter trials have failed to validate the SIS as a sufficiently sensitive instrument in the ED.15 The AD8 (Figure 2) is another paperless instrument developed to screen for dementia through 8 questions administered to the patient or the caregiver but which has not been validated in ED settings.30 The objective of this study was to evaluate the diagnostic test characteristics of the SIS and the AD8 in the ED to detect cognitive dysfunction in adults older than 65 years.

Figure 1
Six-Item Screener.

MATERIALS AND METHODS

Study Design

This was a prospective, cross-sectional, convenience-sampling study in the ED of one urban academic medical center.

Setting

Barnes Jewish Hospital is a Level I trauma center academic teaching hospital in St. Louis, MO, with more than 90,000 total visits annually, 20% of which are by patients aged 65 years or older.

Selection of Participants

According to the availability of 4 research assistants from July 1, 2008, to April 20, 2009, all ED patients aged 65 years or older were approached for enrollment in a convenience sampling. Enrollment occurred in the ED on weekdays and weekends during equally distributed day, evening, and overnight shifts. Potential participants were approached by a research assistant with the approval of the emergency physician caring for the patient. After potential participants received a scripted description of the study that did not include details of the study hypothesis, written consent was obtained. When subjects were physically or mentally incapacitated as judged by caregivers or clinicians, proxy consent was obtained from informed caregivers. We excluded patients who received medications that may have affected their mental status during the testing period (narcotics, benzodiazepines, antiemetics), were too critically ill to participate, as judged by the attending emergency physician, were unable to consent or cooperate with data acquisition, did not speak English, or refused to complete the questioning. Psychoactive medications administered before or during the cognitive testing interval were noted by a research assistant review of the electronic medical record. Our institution uses a computer physician order entry system, so all medications prescribed are electronically documented, including a nursing time stamp when the agents or interventions are actually administered. The study was approved by the Barnes-Jewish Hospital Institutional Review Board, with informed consent.

Data Collection and Processing

For eligible and consenting patients, the physician evaluating the patient (either senior resident or attending physician) administered the SIS at least 30 minutes before or after trained research personnel had administered the Mini-Mental State Examination (MMSE), using a standardized data collection form. The 30-minute delay was to provide a washout period for the 3-item recall, although the SIS used different items for the recall than the MMSE (apple, table, penny). In the majority of cases, the MMSE was administered first because physicians were not readily available to complete the SIS. The order of administration of MMSE and SIS was not randomized, nor was it recorded for subsequent analysis.

During their evaluation, physicians rated the patient’s level of consciousness as alert and attentive (normal), inattentive, hyperalert, lethargic, stuporous, or comatose. The physicians also performed and scored the SIS during their routine history and examination. On completion of the study sheet, the physician placed it in an opaque envelope, sealed it, and recorded the time on the outside of the envelope. Trained research personnel performed the MMSE according to standard instructions.31 When a caregiver was present at the bedside, the caregiver was asked to complete the AD8 (cAD8) while the research assistant administered the MMSE. If no caregiver was present at enrollment, the patient was asked to complete the AD8 (pAD8). Both the physicians performing the SIS and the research personnel completing the MMSE were blinded to the result of the other test. After completion of the MMSE and SIS, the ED medical record was reviewed for documentation of Alzheimer’s and other types of dementia in the medical history section.

Primary Data Analysis

Analysis was conducted according to Standards for Reporting of Diagnostic Accuracy criteria with SPSS (version 16.0; SPSS, Inc., Chicago, IL) and MEDCALC (version 11.2.1.0; MedCalc Software, Mariakerke, Belgium).32 The criterion standard for cognitive dysfunction was an MMSE score less than or equal to 23. Standard operating characteristics of diagnostic tests were calculated, including sensitivity, specificity, likelihood ratios, and receiver operating characteristics curves with area under the curve and 95% confidence intervals (CIs). In addition to computing the sensitivity and specificity at each score for the SIS and AD8, the receiver operating characteristics curve was visually inspected to determine the optimal cutoff point for each instrument that would simultaneously maximize sensitivity and specificity. The SIS sensitivity in the preliminary validation study was 94%.13 Verifying this sensitivity with 5% range for sensitivity (ie, if the point estimate were 94%, then the true value would reside somewhere between 89% and 99% or SD 5%) and a baseline prevalence of cognitive dysfunction of 23% would require 377 subjects to be enrolled with complete data collection.33

RESULTS

Characteristics of Study Subjects

Between June 2008 and April 2009, we approached 957 patients, excluded 586, and enrolled 371 subjects (Figure 3). A total of 52 patients did not complete the MMSE, SIS, or AD8 because they were discharged or admitted before the completion of data collection, so 319 subjects were included in this analysis.

Figure 3
Flow diagram for patient enrollment.

Enrolled subjects had mean age of 76 years, and 58% were women. Whites represented 42% of the cohort, whereas blacks were 58%. The prevalence of cognitive dysfunction was 35.4% (113/319; 95% CI 31% to 41%). Although cognitive dysfunction may represent mild cognitive impairment, dementia, or delirium, only 6% (19/319) of subjects had dementia listed in the ED medical records. The AD8 was administered to caregivers in 41% of cases and to patients in 59% of cases.

Main Results

As demonstrated in Table 1, the sensitivity of the SIS was superior to that of the cAD8 or pAD8 (74% versus 63% versus 37%, respectively). In addition, the SIS area under the curve was superior to that of the cAD8 and pAD8 (Figure 4A and B). The SIS receiver operating characteristics curve in Figure 4A and B differ because the respective curves represent the subset with (n=132) and without (n=187) a caregiver present at the bedside when cognitive testing occurred. Evaluation of the receiver operating characteristics did not identify a cutoff point by which to define abnormal cognitive function superior to those previously reported (SIS ≥2 errors, AD8 ≥2 affirmative responses). In isolation, neither the positive-nor the negative-likelihood ratios for the SIS, cAD8, or pAD8 would significantly alter the posttest probability of cognitive dysfunction.34 However, the combination of either an abnormal SIS or an abnormal cAD8 score demonstrated an increased sensitivity and specificity for cognitive dysfunction (Table 1). No combination of normal or abnormal SIS or cAD8 scores demonstrated better ability to discriminate normal from abnormal MMSE patients than the SIS alone (Table 2), although the combination of an abnormal SIS score and an abnormal cAD8 score had a positive likelihood ratio of 19.9 (95% CI 9.8 to 74.4). When compared with documented medical history, the MMSE, SIS, and cAD8 each identified an increased number of subjects with potential cognitive impairment (Table 3).

Figure 4
A, Receiver operating characteristic curve for pAD8* compared with SIS (n=187). B, Receiver operating characteristic curve for cAD8* compared with SIS (n=132).
Table 1
Diagnostic test characteristics of SIS and AD8.
Table 2
Diagnostic accuracy for various combinations of SIS and cAD8 (number of patients).
Table 3
Diagnostic accuracy for MMSE, SIS, and AD8 compared with documented medical history of dementia (number of patients).

LIMITATIONS

This study has several limitations. First, the MMSE is an imperfect criterion standard for cognitive impairment. The MMSE may be less specific in poorly educated and lower socioeconomic strata and less sensitive among highly educated populations.35-37 In addition, the MMSE is insensitive to mild cognitive impairment and does not differentiate dementia from delirium.38,39 If the SIS or AD8 were in fact superior to the MMSE as a criterion standard for cognitive dysfunction, then false-positive and false-negative rates would be erroneously inflated, misleadingly diminishing the diagnostic properties of the SIS or AD8.40-42 Furthermore, to differentiate dementia from delirium, a validated delirium screening tool such as the Confusion Assessment Method or the Confusion Assessment Method-ICU would need to be coadministered with the MMSE.43,44 Alternative screening instruments such as the Montreal Cognitive Assessment are superior tools to identify mild cognitive impairment.39 Future trials should assess the ability of brief screening instruments such as the SIS and cAD8 to detect mild cognitive impairment and delirium.

Second, this study was conducted at a single urban academic medical center. We excluded a substantial portion of patients who had received potentially sedating medications before enrollment, in addition to critically ill and non-English speaking subsets. We had a nonconsecutive convenience sampling of patients. Our results may not reflect the diagnostic performance of the SIS or AD8 in these excluded populations, although our analysis did not identify any significant differences in age or sex between enrolled and nonenrolled subjects.

Third, we failed to complete data collection on the a priori 377-subject sample size. However, the CIs for the SIS are within the preplanned 5% range. Our findings are also consistent with previous results with the SIS.15

Finally, we did not assess reliability of the SIS or AD8. Specifically, we cannot ascertain whether individual patients’ baseline cognitive status during their ED evaluation was confounded by their acute illness. Although the MMSE was administered before the SIS and AD8 in almost all cases, we cannot exclude a learning phenomenon for the 2 distinct 3-item recalls (MMSE and SIS) tested between the 2 instruments, which could affect observed diagnostic performance.45,46 In addition, the 30-minute washout period between the MMSE and SIS may have permitted cognitive deterioration or improvement in acutely ill patients, which would consequently increase false-positive or false-negative rates, respectively. We also did not assess for potential selection or ascertainment biases between research assistants by analyzing patient characteristic differences across research assistants. However, previous research has suggested that ED-based testing for cognitive function is reproducible.47 Furthermore, we have ongoing 3-week follow-up assessments under way to measure test-retest reliability for cognitive function in our population.

DISCUSSION

The sensitivity and specificity of the SIS to detect cognitive dysfunction in the ED are lower than initially reported.13,29 A previous multicenter trial had suggested significant variability in the SIS sensitivity between sites, with the possibility of a learning phenomenon when the same 3-item recall was used for the MMSE and the SIS.15,45,46 Our ED was site 2 in the previous multicenter SIS validation trial and had demonstrated SIS sensitivity of 60%. Our site had been the only one of 3 that used the identical 3-item recall for the SIS and MMSE. Because the MMSE was administered before the SIS, we offered a referenced hypothesis that subjects at our site learned the 3-item recall during criterion standard MMSE testing, resulting in better recall the second time with the SIS.15 If the hypothesis is correct, this learning phenomenon would increase false-negative results and decrease false-positive results, leading to decreased sensitivity and increased specificity, respectively. In support of this hypothesis, our site had the lowest sensitivity and highest specificity. In the current report, using unique 3-item recall items, the specificity is identical to that of the 2 other sites in the previous multicenter trial, and the sensitivity approaches their mean. The current study eliminates this bias by using distinct 3-item recall lists, although the sensitivity of the SIS would still fail to identify 25% of cognitively impaired patients.

The AD8 was developed to differentiate normal cognitive aging from mild dementia, using an informant-based interview.48 In outpatient clinic populations, this instrument has demonstrated reasonable sensitivity (74%), specificity (86%), and intrarater reliability.30,48,49 Combining the AD8 with performance-based measures has increased the sensitivity for the detection of cognitive impairment in patients referred to a dementia clinic.50 The AD8 has not previously been tested in ED populations and offers several potential advantages over other screening instruments. The AD8 consists of 8 brief questions that can be read to or by subjects. It can be administered to either patients or their caregivers when patients are incapacitated or unavailable. The results from this single-center trial do not support the use of the cAD8 or pAD8. The cAD8 is significantly inferior to the SIS, with nonoverlapping CIs for the receiver operating characteristics area under the curve, although the cAD8 is superior to the pAD8 in the ED setting.

An important finding of this study is the underrecognition of cognitive dysfunction in ED patients. According to our chart review, we found that only 6% of subjects had recognized dementia before their screening, whereas 35.4% had an abnormal MMSE result. This confirms previous research demonstrating that emergency nurses and physicians often fail to recognize cognitive impairment.5,8,9 Most do not routinely screen for cognitive dysfunction with validated instruments.51 Furthermore, inpatient and outpatient physicians often fail to identify cognitive impairment.9,10

In assessing diagnostic instruments, researchers and clinicians must face the reality that the perfect test does not exist. For a diagnostic test to be clinically useful, identification of patients with disease (true-positive results) should provide an opportunity to treat illness. Because false-positive and false-negative results also occur with diagnostic testing, clinicians must weigh the potential harm related to more invasive confirmatory testing or treatment of these erroneous results.52,53 How then do we define an acceptable level of diagnostic inaccuracy to appropriately balance the risks and harms associated with testing? For cognitive dysfunction, ED personnel fail to document recognition in the majority of cases, and only 6% of subjects in our trial presented with a history of dementia. If clinicians were to apply the SIS alone, 74% of patients with MMSE-defined cognitive impairment would be correctly identified within 1 minute, with a 23% false-positive rate. Although our research did not assess the potential benefits or adverse effects of cognitive screening, it is reasonable to surmise that with no cost-prohibitive criterion standard testing available in the ED, the “risks” of false-positive test results are minimal. In promptly recognizing a higher-risk subset of patients with the ED-friendly SIS, diagnostic good may outweigh the risk of harm while clinicians await an alternative cognitive dysfunction screening test for an increasingly geriatric society.54

As demonstrated in Table 1, the combination of a normal SIS and cAD8 result significantly reduces the probability of cognitive dysfunction (negative likelihood ratio=0.16) compared with either instrument in isolation. However, using both instruments requires more time and training, with the additional need to find consenting caregivers to complete the AD8. More important, adding the AD8 to the cognitive evaluation of geriatric ED patients does not improve one’s ability to increase the probability of dementia or delirium (positive likelihood ratio 3.0 for the combination compared with 3.3 for SIS alone). Therefore, the cAD8 may not be worth the added effort because the goal is to recognize (not necessarily exclude) those at increased risk for cognitive dysfunction.

Screening tools and clinical decision aids always require external validation in populations distinct from those used to derive them.55 Numerous examples of methodologically compelling tools that subsequently fail to be validated in unique patient groups have been recognized.56-58 Screening tools are statistical models that may not apply to alternative populations because of overfitting data to the derived model or individual variable instability.59 Several potential reasons could explain why the SIS and AD8 failed to be validated in our ED population. Patients and their caregivers in the ED present with acute or decompensated illnesses entailing labile emotions and chaotic information exchange. The increased prevalence of emotional and cognitive distracters, combined with diminished attentiveness associated with illness-related suffering, could reduce the accuracy and reliability of the patient and caregiver’s SIS and AD8 responses. This is precisely why such instruments need to be validated within the environments in which they could be used.60,61 Diagnostic tests are therefore subjected to a hierarchic outcomes approach progressing from technical value to diagnostic accuracy to clinical outcome efficacy and societal efficacy.62

Because the SIS evaluates only 2 domains (recall, orientation) of cognitive dysfunction, future assessment of screening tools in the ED should evaluate different or additional domains. Validating cognitive screening instruments that can be quickly administered without the need for paper, pencil, score cards, or specific equipment remains essential for ED screening efforts. A number of such instruments have been developed, but all await validation in the hectic ED environment.16,63

The SIS is more sensitive than the AD8 in identifying geriatric ED patients at increased risk for cognitive dysfunction, using the MMSE as the criterion standard. Future trials should assess equipment-free screening instruments that incorporate additional domains while maintaining ED-appropriate brevity. In addition, dementia and delirium should be distinguished, as should mild cognitive impairment and early dementia, using appropriate criterion standards.

Editor’s Capsule Summary

What is already known on this topic

Emergency physicians often fail to recognize cognitive dysfunction among emergency department (ED) patients older than 65 years.

What question this study addressed

Three hundred nineteen geriatric patients were screened for cognitive dysfunction, using the Six-Item Screener (SIS), the AD8 (caregiver-completed AD8 or patient-completed AD8), and the Mini-Mental State Examination (MMSE). The MMSE score was used as the criterion standard.

What this study adds to our knowledge

Cognitive dysfunction was observed in 35% of patients. Only 6% had pre-existing dysfunction noted in the medical record. The SIS was more sensitive than either AD8 but carried a 23% falsepositive rate.

How this is relevant to clinical practice

The SIS is a brief tool that may help physicians identify cognitive dysfunction among geriatric ED patients.

Acknowledgments

Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article as per ICMJE conflict of interest guidelines (see www.icmje.org). Dr. Carpenter was supported by Dennis W. Jahnigen Career Development Awards, which are funded by the American Geriatrics Society, the John A. Hartford Foundation, and Atlantic Philanthropies; and by the Washington University Goldfarb Patient Safety award. The content is solely the responsibility of the authors and does not necessarily represent the official views of the supporting societies and foundations or the funding agencies.

Footnotes

Author contributions: CRC coordinated the project, performed the statistical analysis, and drafted the article. BD, TNK, MS, and MR were responsible for recruitment and outcome measurements. CRC was responsible for the study concept and design and overall supervision of the project. All authors participated in the revision of the article. CRC takes responsibility for the paper as a whole.

Presented at the Society for Academic Emergency Medicine Annual Meeting, May 2009, New Orleans, LA; and at the American Geriatrics Society Annual Meeting, May 2009, Chicago, IL.

Reprints not available from the authors.

REFERENCES

1. Roberts DC, McKay MP, Shaffer A. Increasing rates of emergency department visits for elderly patients in the United States, 1993 to 2003. Ann Emerg Med. 2008;51:769–774. [PubMed]
2. Fitzgerald RT. American College of Emergency Physicians White Paper. The Future of Geriatric Care in Our Nation’s Emergency Departments: Impact and Implications. 2008 Available at http://www.acep.org/Work-Area/DownloadAsset.aspx?id=43376.
3. Hwang U, Morrison RS. The geriatric emergency department. J Am Geratr Soc. 2007;55:1873–1876. [PubMed]
4. Inouye SK, Studenski S, Tinetti ME, et al. Geriatric syndromes: clinical, research, and policy implications of a core geriatric concept. J Am Geratr Soc. 2007;55:780–791. [PMC free article] [PubMed]
5. Lewis LM, Miller DK, Morley JE, et al. Unrecognized delirium in ED geriatric patients. Am J Emerg Med. 1995;13:142–145. [PubMed]
6. Naughton BJ, Moran MB, Kadah H, et al. Delirium and other cognitive impairment in older adults in an emergency department. Ann Emerg Med. 1995;25:751–755. [PubMed]
7. Elie M, Rousseau F, Cole M, et al. Prevalence and detection of delirium in elderly emergency department patients. CMAJ. 2000;163:977–981. [PMC free article] [PubMed]
8. Hustey FM, Meldon SW. The prevalence and documentation of impaired mental status in elderly emergency department patients. Ann Emerg Med. 2002;39:248–253. [PubMed]
9. Heidt JW, Carpenter CR. Occult cognitive impairment in admitted older emergency department patients is not identified by admitting services. Ann Emerg Med. 2009;54:S81.
10. Callahan CM, Hendrie HC, Tierney MC. Documentation and evaluation of cognitive impairment in elderly primary care patients. Ann Intern Med. 1995;122:422–429. [PubMed]
11. Valcour VG, Masaki KH, Curb JD, et al. The detection of dementia in the primary care setting. Arch Intern Med. 2000;160:2964–2968. [PubMed]
12. Kakuma R, Galbaud du Fort G, Arsenault L, et al. Delirium in older emergency department patients discharged home: effect on survival. J Am Geriatr Soc. 2003;51:443–450. [PubMed]
13. Wilber ST, Lofgren SD, Mager TG, et al. An evaluation of two screening tools for cognitive impairment in older emergency department patients. Acad Emerg Med. 2005;12:612–616. [PubMed]
14. Hustey FM, Meldon SW, Smith MD, et al. The effect of mental status screening on the care of elderly emergency department patients. Ann Emerg Med. 2003;41:678–684. [PubMed]
15. Wilber ST, Carpenter CR, Hustey FM. The Six-Item Screener to detect cognitive impairment in older emergency department patients. Acad Emerg Med. 2008;15:613–616. [PubMed]
16. Carpenter CR. Does this patient have dementia? Ann Emerg Med. 2008;52:554–556. [PubMed]
17. Alzheimer’s Association Alzheimer’s facts and figures. Available at: http://www.alz.org/alzheimers_disease_facts_figures.asp.
18. Brookmeyer R, Johnson E, Ziegler-Graham K, et al. Forecasting the global burden of Alzheimer’s disease. Alzheimer Dement. 2007;3:186–191. [PubMed]
19. Lowenstein SR, Crescenzi CA, Kern DC, et al. Care of the elderly in the emergency department. Ann Emerg Med. 1986;15:528–535. [PubMed]
20. Baum SA, Rubenstein LZ. Old people in the emergency room: age-related differences in emergency department use and care. J Am Geriatr Soc. 1987;35:398–404. [PubMed]
21. Singal BM, Hedges JR, Rousseau EW, et al. Geriatric patient emergency visits. Part I: comparison of visits by geriatric and younger patients. Ann Emerg Med. 1992;21:802–807. [PubMed]
22. Aminzadeh F, Dalziel WB. Older adults in the emergency department: a systematic review of patterns of use, adverse outcomes, and effectiveness of interventions. Ann Emerg Med. 2002;39:238–247. [PubMed]
23. Hedges JR, Singal BM, Rousseau EW, et al. Geriatric patient emergency visits. Part II: perceptions of visits by geriatric and younger patients. Ann Emerg Med. 1992;21:808–813. [PubMed]
24. Hastings SN, Barrett A, Hocker M, et al. Older patients’ understanding of emergency department discharge information and its relationship with adverse outcomes. J Am Geriatr Soc. 2009;57:S71. [PubMed]
25. Engel KG, Heisler M, Smith DM, et al. Patient comprehension of emergency department care and instructions: are patients aware of when they do not understand? Ann Emerg Med. 2009;53:454–461. [PubMed]
26. Terrell KM, Hustey FM, Hwang U, et al. Quality indicators for geriatric emergency care. Acad Emerg Med. 2009;16:441–449. [PubMed]
27. Hogan TM, Losman ED, Carpenter CR, et al. Development of geriatric competencies for emergency medicine residents using an expert consensus process. Acad Emerg Med. 2010;17:316–324. [PMC free article] [PubMed]
28. Diner BM, Carpenter CR, O’Connell T, et al. Graduate medical education and knowledge translation: role models, information pipelines, and practice change thresholds. Acad Emerg Med. 2007;14:1008–1014. [PubMed]
29. Callahan CM, Unverzagt FW, Hui SL, et al. Six-Item Screener to identify cognitive impairment among potential subjects for clinical research. Med Care. 2002;40:771–781. [PubMed]
30. Galvin JE, Roe CM, Coats MA, et al. Patient’s rating of cognitive ability. Using the AD8, a brief informant interview, as a self-rating tool to detect dementia. Arch Neurol. 2007;64:725–730. [PubMed]
31. Folstein MF, Folstein SE, McHugh PR. Mini-Mental State: a practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res. 1975;12:189–198. [PubMed]
32. Bossuyt PMM, Reitsma JB, Bruns DE, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Int Med. 2003;138:W1–W12. [PubMed]
33. Obuchowski NA. Sample size calculations in studies of diagnostic accuracy. Stat Methods Med Res. 1998;7:371–392. [PubMed]
34. Hayden SR, Brown MD. Likelihood ratio: a powerful tool for incorporating the results of a diagnostic test into clinical decisionmaking. Ann Emerg Med. 1999;33:575–580. [PubMed]
35. Anthony JC, LeResche L, Niaz U, et al. Limits of the “Mini-Mental State” as a screening test for dementia and delirium among hospital patients. Psychol Med. 1982;12:397–408. [PubMed]
36. Ihl R, Frolich L, Dierks T, et al. Differential validity of pschometric tests in dementia of the Alzheimer type. Psychiatry Res. 1992;44:93–106. [PubMed]
37. Scazufca M, Almeida OP, Vallada HP, et al. Limitations of the Mini-Mental State Examination for screening dementia in a community with low socioeconomic status: results from the Sao Paulo Ageing & Health Study. Eur Arch Psychiatry Clin Neurosci. 2009;259:8–15. [PubMed]
38. Frank RM, Byrne GJ. The clinical utility of the Hopkins Verbal Learning Test as a screening test for mild dementia. Int J Geriatr Psychiatry. 2000;15:317–324. [PubMed]
39. Nasreddine ZS, Phillips NA, Bédirian V, et al. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc. 2005;53:695–699. [PubMed]
40. Hui SL, Zhou XH. Evaluation of diagnostic tests without gold standards. Stat Methods Med Res. 1998;7:354–370. [PubMed]
41. Mower WR. Evaluating bias and variability in diagnostic test reports. Ann Emerg Med. 1999;33:85–91. [PubMed]
42. Glasziou P, Irwig L, Deeks JJ. When should a new test become the current reference standard? Ann Intern Med. 2008;149:816–821. [PubMed]
43. Inouye SK, van Dyck CH, Slessi CA, et al. Clarifying confusion: the confusion assessment method. A new method for detection of delirium. Ann Intern Med. 1990;113:941–948. [PubMed]
44. Ely EW, Inouye SK, Bernard GR, et al. Delirium in mechanically ventilated patients: validity and reliability of the confusion assessment method for the intensive care unit (CAM-ICU) JAMA. 2001;286:2703–2710. [PubMed]
45. Cooper DB, Lacritz LH, Weiner MF, et al. Category fluency in mild cognitive impairment: reduced effect of practice in test-retest conditions. Alzheimer Dis Assoc Disord. 2004;18:120–122. [PubMed]
46. Galasko D, Abramson I, Corey-Bloom J, et al. Repeated exposure to the Mini-Mental State Examination and the Information-Memory-Concentration Test results in a practice effect on Alzheimer’s disease. Neurology. 1993;43:1559–1563. [PubMed]
47. McCusker J, Bellavance F, Cardin S, et al. Screening for geriatric problems in the emergency department: reliability and validity. Acad Emerg Med. 1998;5:883–893. [PubMed]
48. Galvin JE, Roe CM, Powlishta KK, et al. The AD8: a brief informant interview to detect dementia. Neurology. 2005;65:559–564. [PubMed]
49. Galvin JE, Roe CM, Xiong C, et al. Validity and reliability of the AD8 informant interview in dementia. Neurology. 2006;67:1942–1948. [PubMed]
50. Galvin JE, Roe CM, Morris JC. Evaluation of cognitive impairment in older adults: combining brief informant and performance measures. Arch Neurol. 2007;64:718–724. [PubMed]
51. Carpenter CR, Griffey RT, Stark S, et al. Geriatric syndrome screening in emergency medicine: a geriatric technician acceptability analysis. Ann Emerg Med. 2009;54:S82.
52. Pauker SG, Kassirer JP. The threshold approach to clinical decision making. N Engl J Med. 1980;302:1109–1117. [PubMed]
53. Lessler AL, Isserman JA, Agarwal R, et al. Testing low-risk patients for suspected pulmonary embolism: a decision analysis. Ann Emerg Med. 2010;55:316–326. [PubMed]
54. Body R, Foex B. On the philosophy of diagnosis: is doing more good than harm better than “primum non nocere”? Emerg Med J. 2009;26:238–240. [PubMed]
55. Stiell IG, Wells GA. Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med. 1999;33:437–447. [PubMed]
56. Stiell IG, Clement CM, McKnight RD, et al. The Canadian C-Spine Rule versus the NEXUS low-risk criteria in patients with trauma. N Engl J Med. 2003;349:2510–2518. [PubMed]
57. Chase M, Robey JL, Zogby KE, et al. Prospective validation of the thrombolysis in myocardial infarction risk score in the emergency department chest pain population. Ann Emerg Med. 2006;48:252–259. [PubMed]
58. Birnbaum A, Esses D, Bijur P, et al. Failure to validate the San Francisco Syncope Rule in an independent emergency department population. Ann Emerg Med. 2008;52:151–159. [PubMed]
59. Charlson ME, Ales KL, Simon R, et al. Why predictive indexes perform less well in validation studies. Is it magic or methods? Arch Intern Med. 1987;147:2155–2161. [PubMed]
60. Godwin M, Ruhland L, Casson I, et al. Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity. BMC Med Res Methodol. 2003;3:28. [PMC free article] [PubMed]
61. Knottnerus JA, Muris JW. Assessment of the accuracy of diagnostic tests: the cross-sectional study. J Clin Epidemiol. 2003;56:1118–1128. [PubMed]
62. Pearl WS. A hierarchical outcomes approach to test assessment. Ann Emerg Med. 1999;33:77–84. [PubMed]
63. Holsinger T, Deveau J, Boustani M, et al. Does this patient have dementia? JAMA. 2007;297:2391–2404. [PubMed]