PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
J Am Geriatr Soc. Author manuscript; available in PMC 2013 June 1.
Published in final edited form as:
PMCID: PMC3376010
NIHMSID: NIHMS364095

Identifying Indicators of Key Diagnostic Features of Delirium

Li-Wen Huang, AB,* Sharon K. Inouye, MD, MPH,* Richard N. Jones, ScD,* Tamara G. Fong, MD, PhD,*§ James L. Rudolph, MD, MS, Margaret G. O’Connor, PhD,§ Eran D. Metzger, MD,* Paul K. Crane, MD, MPH,# and Edward R. Marcantonio, MD, MS**

Abstract

Background

Delirium is a common, serious, and potentially treatable condition in older persons. Healthcare professionals often fail to recognize delirium. Our objective was to use an expert consensus process to identify indicators of key delirium features to help enhance bedside recognition of delirium.

Design

Modified Delphi consensus process to assign existing cognitive and delirium assessment items to delirium features in the Confusion Assessment Method (CAM) diagnostic algorithm.

Setting

Meetings of expert panel.

Participants

Panel of seven interdisciplinary clinical experts.

Measurements

Panelists’ assignments of each assessment item to indicate CAM features.

Results

From an initial pool of 119 assessment items, the panel assigned 66 items to at least one CAM feature, and many items were assigned to more than one feature. Experts achieved a high level of consensus, with a post-meeting kappa for agreement of 0.98. The study staff compiled the assignment results to create a comprehensive list of CAM feature indicators, consisting of 107 patient interview questions, cognitive tasks, and interviewer observations, with some items assigned to multiple features. A subpanel further shortened this list to 28 indicators of key delirium features.

Conclusion

We used a systematic, well-described qualitative methodology to create a list of indicators for delirium based on the features of the CAM diagnostic algorithm. This indicator list may be useful as a clinical tool for enhancing delirium recognition at the bedside and for aiding in the development of a brief delirium screening instrument.

Keywords: delirium, assessment, cognitive impairment, Confusion Assessment Method, aging

INTRODUCTION

Delirium, an acute change in mental status characterized by a fluctuating course, is a common, serious, yet potentially preventable and treatable complication in hospitalized older patients.16 Delirium is independently associated with poor outcomes, including increased risk of mortality, dementia, and institutionalization, as well as increased length of hospital stay and higher healthcare costs.2,3,7 Despite its clinical importance and economic impact, delirium remains a poorly understood condition, and health care professionals often fail to recognize delirium at the bedside.811

Delirium is difficult to effectively evaluate at the bedside for many reasons. First, delirium has a wide range of features. Some features, such as inattention, are essential for diagnosis, while others, such as hallucinations, are supportive but not necessary.12 Not all healthcare professionals recognize that different features have different diagnostic utilities or know how to evaluate the core features effectively. Second, the hypoactive form of delirium, characterized by reduced psychomotor activity and level of consciousness, is often overlooked or misdiagnosed, since a subdued, “compliant” patient may not be recognized to have delirium.8,13 Third, cognitive dysfunction due to delirium often cannot be detected without formal cognitive testing, yet healthcare professionals rarely administer such testing in day-to-day patient evaluations due to lack of time, training, or appreciation for its importance.8 A bedside evaluation of delirium requires a synthesis of information collected from the patient interview, formal cognitive testing, and clinical observations.12,14

Few tools provide practical assistance for delirium assessment in the clinical setting. The Confusion Assessment Method (CAM) diagnostic algorithm15 has been widely used in research and has been cited as the bedside delirium instrument with the best supportive evidence.16 While the CAM is an extremely useful tool, it is a diagnostic algorithm, not a structured patient interview designed for routine clinical care. The CAM has been used most successfully in the research setting, where in studies that involved more than 1000 patients and used trained staff and a formal assessment process, the CAM demonstrated an overall sensitivity of 94% (95% confidence interval [CI] = 91%–97%), specificity of 89% (95% CI = 85%–94%), and inter-rater reliability of 0.70–1.00.17 As these research studies demonstrate, optimal use of the CAM requires formal cognitive evaluation and some training.17,18 However, not all healthcare professionals have the training or expertise required to appropriately operationalize the CAM.

Our overall goal was to identify specific assessment items that indicate key CAM features of delirium. These indicators can help to systematize observations and ensure that clinicians assess for all the core features of delirium in a structured manner. The specific aims of this study were: (1) starting with a pool of standard delirium assessment items, to use a standard expert panel process to determine which CAM features are evaluated by each assessment item, and (2) to develop a list of indicators of key diagnostic features of delirium that would promote delirium recognition in the clinical, educational, and research settings.

METHODS

Overview

This study involved three distinct stages. First, the study team selected the initial pool of assessment items considered relevant for delirium and developed the assignment form to be used by experts. Second, an interdisciplinary panel of delirium experts from internal medicine, psychiatry, nursing, neurology, and neuropsychology determined which of the four key diagnostic features of the CAM were evaluated by each item using a two-round consensus process. Third, the study staff used the final consensus agreements to develop a list of indicators for delirium.

Delirium Assessment Selection and Assignment Form Development

Delirium Assessment Selection

The study team selected the initial pool of delirium assessment items from existing delirium and cognitive instruments, balancing competing goals of including the widest pool of items and having the ability to subsequently test items empirically. Ultimately, the team selected a pool of 119 delirium assessment items from the Blessed Information-Memory-Concentration (BIMC) test,19 the Digit Span test,20 and the Delirium Symptom Interview (DSI).21 Importantly, similar items were included in a structured delirium assessment to operationalize the CAM algorithm used extensively by the research team in previous work.22 Table 1 describes the initial pool of assessment items, which consists of patient interview questions, cognitive tasks, and interviewer observations.

Table 1
Initial Pool of Delirium Assessment Items

The study team chose to use the CAM diagnostic algorithm15 to define the key features of delirium due to its validity and clinical relevance.16,17 The CAM diagnostic algorithm evaluates four key features of delirium: 1) Acute Change in Mental Status with Fluctuating Course, 2) Inattention, 3) Disorganized Thinking, and 4) Altered Level of Consciousness. While Feature 1 was originally described as Acute Change and Fluctuating Course,15 it was broadened to Acute Change or Fluctuating Course for this study, as recommended in the CAM Training Manual for situations where maximal sensitivity for delirium detection is required.18

Assignment Form Development

In order to determine which CAM feature(s) can be informed by each assessment item, potentially informative responses to each item must be assigned to the four features of the CAM diagnostic algorithm. In general, four types of responses were thought to be potentially indicative of delirium symptoms: 1) incorrect response to an interview question or cognitive task, 2) patient report or interviewer observation that is positive for a delirium symptom, 3) no response (i.e., the patient failing to provide any answer to a question), or 4) a response of “don’t know.” Each item response could be assigned to one, more than one, or none of the CAM features. For each item, potentially informative responses were assigned to the CAM features as shown in Figure 1. Since correct responses to cognitive tasks or patient responses and interviewer observations negative for a delirium symptom indicate the absence of delirium features, they were not included for assignment.

Figure 1
Item Assignment Form Sample: Delirium Symptom Interview21 Question 6

The measurement units for this study included both items and indicators, which are important to distinguish. An item is a patient interview question, cognitive task, or interviewer observation. Each item may have several possible responses, such as incorrect answer, positive report/observation of a symptom, no response, or “don’t know.” An indicator is the set of responses to a single item that indicate a given CAM feature. Thus, items and item responses are not necessarily assigned exclusively to one feature, but indicators are. An item with responses assigned to multiple features will have multiple indicators. For example, in Figure 1, DSI question 6 (“Have you felt confused at any time during the past day?”)21 has four indicators, one for each CAM feature. A response of “yes” and “don’t know” are assigned to Feature 1 (Acute Change or Fluctuating Course), and thus these two item responses make up an indicator for Feature 1. The same set of responses is also an indicator for Feature 3 (Disorganized Thinking). Similarly, no response and “don’t know” make up the indicator for Feature 2 (Inattention), and no response alone is an indicator for Feature 4 (Altered Level of Consciousness).

Expert Panel Process

Panel Assignments, Meeting

The study team convened an interdisciplinary panel of seven experts chosen for their expertise in delirium assessment and knowledge of the CAM diagnostic algorithm. The panel included three experts from internal medicine/geriatric medicine and one each from geriatric psychiatry, geriatric nursing, neurology, and neuropsychology. The expert panel followed the modified Delphi method for the consensus process,2325 involving one round of individual, anonymous completion of the assignment form, followed by a face-to-face panel discussion, and finally, another round of individual assignment form completion.

In the first round, the study staff provided the experts with the assignment form and instructed them to assign each item response to one, more than one, or none of the CAM features. The panelists could also describe any difficulties or questions about the process in a “Notes” section on the form. The study staff combined the individual assignment results into a composite report that showed the number of times each item response was assigned to each CAM feature, along with any comments from the panelists. The staff then distributed the composite report to all panelists prior to the face-to-face meeting.

At the face-to-face consensus meeting, the panel reviewed the goals of achieving consensus and the clinical applicability of the results. The experts then discussed any areas of concern or disagreement. After the meeting, panel members immediately and independently reassigned all item responses to the CAM features. Consensus was determined by four or more agreeing votes among the seven experts.

Data Collection and Analysis

The study staff compiled the CAM feature assignments of all item responses and computed the resulting number of indicators for each CAM feature. The kappa for agreement was calculated for the pre- and post-meeting expert assignments for each CAM feature as well as all features combined. The staff also tallied the frequency of different response types (i.e., incorrect answer vs. positive report/observation vs. no response vs. “don’t know”) by feature to elucidate any patterns of CAM feature assignments according to response type.

Development of List of Delirium Indicators

The study staff used the final consensus results to create the comprehensive list of indicators for the CAM features of delirium. Because this list was too long for publication, the study staff employed a subpanel to select indicators from the comprehensive list for a brief list of indicators for key delirium features, with special consideration for minimizing items being assigned to multiple features. Using a similar process, a subpanel of five experts selected indicators that were representative of each CAM feature, with consensus determined by at least three out of five votes.

RESULTS

Expert Panel Process

From the initial pool of 119 assessment items, a total of 53 items were not found to be indicators of delirium. Of these 53, 33 items were not assigned to any CAM features with unanimous agreement from the panel, 18 items that were initially assigned to CAM features were subsequently combined with items on which they were logically dependent, and 2 items were dropped because they were repetitive with other assigned items (for details, see Table 1 footnote). For example, items scoring presence or absence of a symptom were combined with items rating severity of that symptom.

Table 2 shows the CAM feature assignments of the 66 assessment items remaining from the initial pool. The panel assigned several items to multiple CAM features, resulting in a total of 135 indicators. On average, each item was assigned to approximately two CAM features. In summary, the panel assigned 29 indicators to Feature 1 (Acute Change or Fluctuating Course), 37 indicators to Feature 2 (Inattention), 41 indicators to Feature 3 (Disorganized Thinking), and 28 indicators to Feature 4 (Altered Level of Consciousness). The kappa for agreement before the expert panel meeting was 0.22 for Feature 1, 0.29 for Feature 2, 0.34 for Feature 3, 0.31 for Feature 4, and 0.31 for all features combined. The post-meeting kappa for agreement was 0.98 for Feature 1, 0.97 for Feature 2, 0.98 for Feature 3, 0.98 for Feature 4, and 0.98 for all features combined. In summary, the panel achieved 100% agreement on 97% of the post-meeting assignment decisions.

Table 2
Final Assignment Summary by the Expert Panel

Table 3 breaks down the feature assignments by response type, demonstrating how certain response types preferentially indicate certain CAM features. While positive patient symptom reports or interviewer observations inform all features, incorrect responses to interview questions or cognitive tasks tend to indicate Inattention (Feature 2) and Disorganized Thinking (Feature 3). No response to patient questions and cognitive tasks exclusively indicates Inattention (Feature 2) or Altered Level of Consciousness (Feature 4). These results reflect the panel’s discussions that failing to respond to the interviewer’s questions could indicate either: 1) distractibility and trouble keeping track of or attending appropriately to the interview, or 2) a decreased level of consciousness. A response of “don’t know” mostly informs Inattention (Feature 2) or Disorganized Thinking (Feature 3).

Table 3
CAM Feature Assignments by Indicator Response Type

List of Delirium Indicators

The study staff adapted the 135 indicators from the final consensus assignments, rewording or combining certain indicators for the sake of brevity, to create a comprehensive list of delirium indicators. The resulting comprehensive list consists of 107 patient interview questions, cognitive tasks, and interviewer observations (Appendix for reviewers only, available to readers online or upon request). Items appear more than once if they were assigned to more than one feature, but the response(s) that indicate a particular feature vary from one to another.

Table 4 presents the briefer list of indicators created by the subpanel for publication. This list consists of 28 indicators in total—5 for Acute Change or Fluctuating Course, 8 for Inattention, 11 for Disorganized Thinking, and 4 for Altered Level of Consciousness. By consensus, the subpanel combined several indicators consisting of no response from the patient into one observational item for evaluating Altered Level of Consciousness: “Was the patient non-communicative or unresponsive to multiple questions during the interview?”

Table 4
Brief List of Indicators for the CAM Features of Delirium

DISCUSSION

Using a systematic, well-described qualitative methodology, a panel of seven interdisciplinary clinical experts achieved outstanding consensus in the assignment of standard delirium assessment items to key delirium features of the CAM diagnostic algorithm. A comprehensive and a brief list of indicators of key features of delirium were created. The delirium indicators include patient interview questions, cognitive tasks, and interviewer observations—each representing an important assessment modality for the evaluation of delirium. By providing specific assessment items for each CAM feature, our results may provide a valuable guide for healthcare professionals performing clinical assessments for delirium.

The panel determined that Feature 1 (Acute Change or Fluctuating Course) can be indicated, not surprisingly, by observed fluctuation of mental status domains but may also be informed by evidence of acute temporal, spatial, and personal disorientation, confusion, or perceptual disturbances. Feature 2 (Inattention) is indicated by errors on cognitive tasks designed to evaluate attention, and by observations of decreased awareness of surroundings (e.g. staring into space, losing track of interview) or a hyper-vigilant state (e.g. easily distracted). Of note, many indicators of Feature 2 are characterized by a response of “don’t know” or no response from the patient, suggesting that often a pattern of lack of response, rather than the specific question content, indicates the presence of inattention. Indicators of Feature 3 (Disorganized Thinking) include abnormalities of thought (e.g. illogical ideas, paucity of thought) or speech (e.g. disjointed words or phrases) as well as evidence of temporal, spatial, and personal disorientation, confusion, or perceptual disturbances. Feature 4 (Altered Level of Consciousness) is indicated by observations of decreased (e.g. sleepy, lethargic) or increased level of consciousness (e.g. easily startled) and a pattern of unresponsiveness to questions, which the subpanel added to the shortened list of indicators as its own summary indicator.

Some results of the item assignments deserve further comment. First, a large number of indicators in Table 2 do not imply increased usefulness in delirium assessment; it simply reflects a large number of items with related content that were selected for the initial pool of assessment items. For example, the DSI contained numerous items describing the manifestations of perceptual disturbances, accounting for the large number of perceptual disturbance indicators in Table 2. However, perceptual disturbances are relatively infrequent and poorly sensitive for delirium detection (sensitivity 19–30%).15 Second, consensus opinions from this study suggest that the type of response, in particular “don’t know” or no response, may provide information in addition to the actual question content. Thus, for the creation of future delirium instruments, explicit recording of no response and “don’t know” responses may be useful.

Previous studies have shown that variations in the use of formal cognitive testing and interpretation of interviewer observations affect the performance of the CAM, yet not all healthcare professionals have the training required for its optimal application.8,17,18 By listing specific assessment items, including cognitive tasks, that can be used to evaluate each delirium feature, our results will assist clinicians in the more effective implementation of the CAM diagnostic algorithm. In addition, the interviewer observations identified by our process may also help healthcare professionals put their observations, made either during routine clinical care or specific mental status evaluation, into the context of delirium symptoms. Identification of positive CAM features using these indicators should prompt a formal evaluation by an expert clinician and, if confirmed, appropriate steps to identify and address the underlying cause of delirium.

Strengths of this study include the use of a systematic, well-described qualitative methodology, the wide range of expertise represented by the panel, and the high level of consensus achieved after the panel meeting. Other strengths include a framework of key delirium features that will be easy to use in the clinical setting, a list of specific assessment items to guide the evaluation of each CAM feature, the inclusion of multiple assessment modalities, and the integration with the CAM diagnostic algorithm to enhance delirium recognition at the bedside.

Some limitations to this study must be noted. First, our initial item pool does not contain every neuropsychological test that can be used to assess delirium. However, it does contain tests of temporal, spatial, and personal orientation, many standard tests of attention, and commonly used tests for all pertinent domains of delirium. Secondly, items were frequently assigned to multiple CAM features in the comprehensive list of indicators. This finding reflects the inter-related nature of the CAM features and the challenge of isolating mental status domains in neuropsychological testing. The initial methodological decision to not restrict the number of CAM features to which each item could be assigned contributed to these multiple assignments. The subpanel took steps to limit multiple assignments in the brief list of indicators. Lastly, this study does not utilize any patient data, and the panel decisions represent consensus of expert opinions, not objective measures. However, the expert panel provided invaluable insights into this complex syndrome, allowing us to advance understanding in this area. We will take steps to validate this list of indicators using clinical data and to identify the best-performing assessment items for each delirium feature using advanced measurement techniques. The long-term goal is to use these indicators to create a brief delirium screening instrument for the clinical setting.

Prompt recognition of delirium and timely intervention may reduce its morbidity and mortality,5,6 yet its multifaceted nature makes diagnosis challenging, especially for those untrained in delirium and cognitive assessments.14 We have created both a comprehensive and a brief list of indicators for delirium based on the key delirium features in the CAM diagnostic algorithm to provide a framework for delirium assessment. By listing specific assessment items, response patterns, and linking these to the key CAM features, our results may be useful for helping healthcare professionals recognize delirium at the bedside. In addition, this work lays the groundwork for the creation of future brief screening instruments to improve delirium detection.

Supplementary Material

Supp App S1

ACKNOWLEDGMENTS

This work was supported in part by grants T32AG023480-06A1 (L-WH), T35AG038027-01 (L-WH, JLR), P01AG031720 (SKI), R01AG030618 (ERM), and K24AG035075 (ERM) from the National Institute on Aging.

Sponsor’s Role: The sponsors played no role in the design, methods, data collection, analysis, or preparation of the manuscript.

Footnotes

Meeting(s): An abstract of this work was presented at the 2011 American Geriatrics Society Annual Meeting.

Conflict of Interest:

Li-Wen Huang is a recipient of an award from the American Federation for Aging Research Medical Student Training in Aging Research (MSTAR) Program, the Doris Duke Charitable Foundation Clinical Research Fellowship at Harvard Medical School, and the American Academy of Neurology Medical Student Summer Research Scholarship. Dr. Marcantonio is the recipient of a Mid-career Investigator Award in Patient-Oriented Research from the National Institute on Aging. Dr. Inouye holds the Milton and Shirley F. Levy Family Chair.

Author Contributions: Huang: analysis and interpretation of data, preparation of manuscript, final approval of manuscript. Inouye: study concept and design, expert panel member, analysis and interpretation of data, preparation of manuscript, final approval of manuscript. Jones: study concept and design, acquisition of data, analysis and interpretation of data, revising drafts of manuscript, final approval of manuscript. Fong, Rudolph, O’Connor, Metzger: expert panel member, analysis and interpretation of data, revising drafts of manuscript, final approval of manuscript. Crane: analysis and interpretation of data, revising drafts of manuscript, final approval of manuscript. Marcantonio: study concept and design, expert panel member, acquisition of data, analysis and interpretation of data, preparation of manuscript, final approval of manuscript.

Human Subjects: An exemption application was submitted to our Institutional Review Board and this research was deemed suitable for IRB exempt status.

REFERENCES

1. Inouye SK, Bogardus ST, Charpentier PA, et al. A multicomponent intervention to prevent delirium in hospitalized older patients. N Engl J Med. 1999;340:669–676. [PubMed]
2. Witlox J, Eurelings LS, de Jonghe JF, et al. Delirium in elderly patients and the risk of postdischarge mortality, institutionalization, and dementia: A meta-analysis. JAMA. 2010;304:443–451. [PubMed]
3. Siddiqi N, House AO, Holmes JD. Occurrence and outcome of delirium in medical in-patients: A systematic literature review. Age Ageing. 2006;35:350–364. [PubMed]
4. Marcantonio ER, Flacker JM, Wright RJ, et al. Reducing delirium after hip fracture: A randomized trial. J Am Geriatr Soc. 2001;49:516–522. [PubMed]
5. Milisen K, Lemiengre J, Braes T, et al. Multicomponent intervention strategies for managing delirium in hospitalized older people: Systematic review. J Adv Nurs. 2005;52:79–90. [PubMed]
6. Lundström M, Edlund A, Karlsson S, et al. A multifactorial intervention program reduces the duration of delirium, length of hospitalization, and mortality in delirious patients. J Am Geriatr Soc. 2005;53:622–628. [PubMed]
7. Leslie DL, Marcantonio ER, Zhang Y, et al. One-year health care costs associated with delirium in the elderly population. Arch Intern Med. 2008;168:27–32. [PubMed]
8. Inouye SK, Foreman MD, Mion LC, et al. Nurses' recognition of delirium and its symptoms: Comparison of nurse and researcher ratings. Arch Intern Med. 2001;161:2467–2473. [PubMed]
9. Hustey FM, Meldon SW. The prevalence and documentation of impaired mental status in elderly emergency department patients. Ann Emerg Med. 2002;39:248–253. [PubMed]
10. Laurila JV, Pitkala KH, Strandberg TE, et al. Detection and documentation of dementia and delirium in acute geriatric wards. Gen Hosp Psychiatry. 2004;26:31–35. [PubMed]
11. Kales HC, Kamholz BA, Visnic SG, et al. Recorded Delirium in a National Sample of Elderly Inpatients: Potential Implications for Recognition. J Geriatr Psychiatry Neurol. 2003;16:32–38. [PubMed]
12. Gupta N, de Jonghe J, Schieveld J, et al. Delirium phenomenology: What can we learn from the symptoms of delirium? J Psychosom Res. 2008;65:215–222. [PubMed]
13. Kiely DK, Jones RN, Bergmann MA, et al. Association between psychomotor activity delirium subtypes and mortality among newly admitted postacute facility patients. J Gerontol A Biol Sci Med Sci. 2007;62:174–179. [PubMed]
14. Cole MG. Delirium in elderly patients. Am J Geriatr Psychiatry. 2004;12:7–21. [PubMed]
15. Inouye SK, van Dyck CH, Alessi CA, et al. Clarifying confusion: The confusion assessment method. Ann Intern Med. 1990;113:941. [PubMed]
16. Wong CL, Holroyd-Leduc J, Simel DL, et al. Does this patient have delirium?: Value of bedside instruments. JAMA. 2010;304:779–786. [PubMed]
17. Wei LA, Fearing MA, Sternberg EJ, et al. The Confusion Assessment Method: a systematic review of current usage. J Am Geriatr Soc. 2008;56:823–830. [PMC free article] [PubMed]
18. [Accessed August 5, 2011];The Confusion Assessment Method (CAM): Training Manual and Coding Guide (online) Available at: http://hospitalelderlifeprogram.org/pdf/The%20Confusion%20Assessment%20Method.pdf.
19. Blessed G, Tomlinson BE, Roth M. The association between quantitative measures of dementia and of senile change in the cerebral grey matter of elderly subjects. Br J Psychiatry. 1968;114:797–811. [PubMed]
20. Wechsler D. Wechsler Adult Intelligence Scale-Revised (WAIS-R) San Antonio: The Psychological Corporation; 1981.
21. Albert MS, Levkoff SE, Reilly C, et al. The delirium symptom interview: An interview for the detection of delirium symptoms in hospitalized patients. J Geriatr Psychiatry Neurol. 1992;5:14–21. [PubMed]
22. Simon SE, Bergmann MA, Jones RN, et al. Reliability of a structured assessment for nonclinicians to detect delirium among new admissions to postacute care. J Am Med Dir Assoc. 2006;7:412–415. [PubMed]
23. Jones J, Hunter D. Using the Delphi and nominal group technique in health services research. In: Mays N, Pope C, editors. Qualitative research in health care. 2nd Ed. London: BMJ Books; 2000. pp. 40–49.
24. [Accessed February 1, 2012];The Delphi Method, III: Use of Self Ratings to Improve Group Estimates (online) Available at: www.rand.org/pubs/research_memoranda/2006/RM6115.pdf.
25. [Accessed February 1, 2012];The RAND/UCLA Appropriateness Method User’s Manual (online) Available at: www.rand.org/content/dam/rand/pubs/monograph_reports/2011/MR1269.pdf.