PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
JAMA Pediatr. Author manuscript; available in PMC 2014 July 1.
Published in final edited form as:
PMCID: PMC3700595
NIHMSID: NIHMS463994

Informing the Uninformed: Optimizing the Consent Message Using a Fractional Factorial Design

Alan R. Tait, Ph.D.,*§ Terri Voepel-Lewis, M.S.N., R.N.,* Vijayan N. Nair, Ph.D.,** Naveen N. Narisetty, M.Stat,** and Angela Fagerlin, Ph.D.§

Abstract

Objective

Research information should be presented in a manner that promotes understanding. However, many parents and research subjects have difficulty understanding and making informed decisions. This study was designed to examine the effect of different communication strategies on parental understanding of research information.

Participants

640 parents of children scheduled for elective surgery

Design

Observational study using a fractional factorial design

Setting

Large tertiary care children's hospital

Interventions

Parents were randomized to receive information about a hypothetical pain trial presented in one of 16 consent documents containing different combinations of 5 selected communication strategies (i.e., length, readability, processability [formatting], graphical display, and supplemental verbal disclosure).

Main outcome measures

Parents were interviewed to determine their understanding of the study elements (e.g., protocol, alternatives etc.) and their gist (main point) and verbatim (actual) understanding of the risks and benefits.

Results

Main effects for understanding were found for processability, readability, message length, use of graphics, and verbal discussion. Consent documents with high processability, 8th grade reading level, and graphics resulted in significantly greater gist and verbatim understanding compared with forms without these attributes (mean difference, 95% CI = 0.57, 0.26–0.88, correct responses out of 7 and 0.54, 0.20–0.88 correct responses out of 4 for gist and verbatim, respectively).

Conclusions

Results identified several communication strategy combinations that improved parents' understanding of research information. Adoption of these active strategies by investigators, clinicians, IRBs, and study sponsors represents a simple, practical, and inexpensive means to optimize the consent message and enhance parental, participant, and patient understanding.

Introduction

As proxy decision-makers for their children, adequate parental informed consent is of paramount importance in the research and healthcare settings.1 For most studies, the standard vehicle for the disclosure of research information is the consent form, although the process of consent also requires a meaningful dialogue between the investigator and parent or participant.

As part of the consent process, Federal regulations mandate that research participants receive information regarding several core elements of consent (e.g., risks, protocol, etc.) in a manner that promotes understanding.2 Despite this, studies show that consent documents are often difficult for the average individual to understand and that verbal disclosure is quite variable.37 This is important since a lack of understanding may result in the unknowing acceptance of potential risk, non- participation, or may jeopardize the safety of participants who are unable to follow a research protocol.

The observations that understanding of consent information is often inadequate suggest that the current approaches to presenting study information are not always optimal. Indeed, although a variety of interventions to improve participant understanding have been tested, not all have been uniformly effective.8 The reasons for this are multifactorial but the fact that research and medical information are often presented in a “one size fits all” format ignores important differences in the way that individuals assimilate and understand complex information. Furthermore, many studies 912 describe just a single intervention (e.g., shorter forms) such that the potential benefits of multiple interventions on understanding have not been fully explored. Finally, many of these studies have focused solely on the consent document rather than the process as a whole.8 The hypothesis to be tested, therefore, was that different combinations of active communication strategies (written and verbal) would improve parents' understanding of consent information and thus, optimize the informed consent process.

Methods

Participants

The study was approved by the University of Michigan's Institutional Review Board (IRB) with a waiver of written informed consent. Parents (>18 yr) of children who were scheduled for elective surgery were included.

Consent form development

A series of 16 consent forms were developed that included different combinations of 5 selected communication strategies. Each form was based on the University of Michigan's IRB consent template and described a hypothetical clinical trial examining the efficacy and safety of transitioning children with moderate to severe pain from a short acting to a “new” long acting pain medication (“Painaway”). Prior to distribution, each consent form was reviewed by multiple experts in informed consent and lay individuals for consistency of content across documents.

Message selection

Based on the extant literature and our previous work, we selected 5 communication strategies that have been shown individually to improve subjects' understanding of consent information. The 5 selected strategies (each with two levels) included:

  • a)
    Readability (8th grade versus 12th grade reading level)13, 14
  • b)
    Processability/Formatting (low versus high)7,12 Techniques for improving processability included larger font (size 14), more white space, and highlighting using bulleting, bolding, and underlining. Processability was measured using the techniques developed by Irwin and Davis and adapted by Philipson et al. for use with consent forms.15, 16
  • c)
    Graphic display of risks and benefits (Graphs versus text). Pictographs were chosen to display risks and benefits based on previous research showing their effectiveness.1720
  • d)
    Verbal disclosure (yes versus no).7, 2123 Subjects received either a supplemental verbal description of the study using a standard script or no verbal description.
  • e)
    Message length (short versus long).7, 10, 24 Simplified versus more detail. On average, the longer forms were 3–4 pages longer than the shorter forms.

Regardless of which strategy combinations were used, each form contained all the elements of consent such that content between documents was not compromised.

Fractional factorial design (FFD)

Because a full factorial design for 5 factors would require an unwieldy 25 = 32 experimental groups, we employed instead a 16-group fractional factorial design in which parents were randomized to receive information about the pain study in one of 16 different consent forms each of which varied in terms of the message level combinations (see table 1). In order to determine the operational effect of each of the 5 strategies, half of the parents were randomized to one level of the factor and half to the other level and the outcomes (i.e., understanding) compared. FFDs have been used effectively in engineering applications,2527 however, their usefulness in behavioral studies has only recently been recognized.28 The advantage of FFDs is that they allow for the study of a large number of factors using fewer cells than required by a full factorial design (i.e. 16 versus 32). Thus, we chose to employ a Resolution V FFD with 16-groups that would allow us to estimate all main effects and all pre-specified 2-factor interactions among the 5 communication strategies.

Table 1
Sixteen-cell Resolution V Fractional Factorial Design with 5 two-level covariates

Procedures

Parents were approached in the preoperative waiting area by trained research assistants and told that we were evaluating the quality of our consent process. Parents were also told that their child would NOT be participating in an actual study but that they should consider the information as if real. Although the study represented a hypothetical study it was nevertheless presented to simulate our standard practice for consenting parents (i.e., place, time to read the consent, and environment). Parents received a $5 gift-card for their participation.

Primary outcome measures

Understanding of the consent elements

Parents were interviewed to measure their understanding of 11 required elements of consent (45.CFR.46).2 The semi-structured interview was based on the Deaconess Informed Consent Comprehension Test (DICCT)29 and followed the format described previously.3, 7, 30 Outcome measures included understanding of the study purpose, procedures, alternatives, direct and indirect benefits, risks, voluntariness, freedom to withdraw, study duration, contact information, and confidentiality. The open-ended responses were written down verbatim by trained research assistants. The interviewer was allowed to clarify individual questions and prompt parents for additional information but could not offer specific details. The transcribed responses were scored independently by two assessors who knew the study but were blinded as to which consent form was administered. The scoring system utilized a 0–2 scale based on the parent having no (0), partial (1), or complete (2) understanding.7

Gist and Verbatim understanding of risks and benefits

Gist understanding refers to the ability to understand the essential meaning of the risks and benefits.31, 32 Seven gist questions were included, one of which was: “If a child received Painaway which of the following is most likely: 1) good pain relief, 2) nausea and vomiting, 3) constipation, 4) slowed breathing. Gist understanding was defined as the ability to respond correctly to >5 questions out of 7. This threshold definition was based on previous studies,17,19 however, as validation, a sensitivity analysis was conducted and found qualitatively similar results using either a continuous measure of understanding or different thresholds e.g., 7 out of 7 correct answers.

Verbatim understanding refers to the ability to correctly report the actual risk/benefit statistics.31, 32 Four verbatim questions were included and required parents to respond to the question, “If 100 children took Painaway, how many would experience: 1) good pain relief, 2) nausea and vomiting, 3) constipation, and 4) excessive sleepiness?” Per previous work, verbatim understanding was defined as the ability to correctly respond to ≥ 3 out 4 verbatim questions.17, 19

Secondary outcomes and parent characteristics

Parents completed a questionnaire and two validated instruments regarding factors that influenced their understanding of the consent document.

Secondary outcome measures

  • 1)
    Perceptions of the risks, benefits, and study importance (0–10 numbers scale, where 10 = high).
  • 2)
    Satisfaction with the consent document format (5-point Likert scale of “extremely dissatisfied” to “extremely satisfied”).
  • 3)
    Consent form factors:
    • a)
      degree to which parents listened (if applicable) to the verbal disclosure and read the consent document (i.e., completely, partially, or not at all).
    • b)
      clarity of the information (i.e., not at all clear, fairly clear, or very clear),
    • c)
      amount of information (i.e., too little, just right, or too much)
    • d)
      readability (i.e., very difficult to read, about right, or very easy to read).

Parent characteristics

  • 1)
    Socio-demographics: Age, gender, race/ethnicity, education, parent/child prior research participation.
  • 2)
    Medical literacy: Measured using the validated Rapid Estimate of Adult Literacy in Medicine (REALM) instrument.33
  • 3)
    Numeracy: Measured using the validated 8-item subjective numeracy scale (SNS).34, 35
  • 4)
    Parental anxiety: Measured using a 0–10 numbers scale (10 = extremely anxious).36, 37

On completion of the study, parents were shown two consent documents representing two extremes of the written strategies (i.e. 8th grade, high processability, short, and graphics versus 12th grade, low processability, long, no graphics) and asked which they preferred (see eAppendix).

Statistical Analysis

Data were analyzed using PASW (Chicago, IL) and R software. Prior to the principal analyses, one-way ANOVA was used to determine if randomization was successful in creating equivalent groups. Because the threshold definitions of understanding e.g., ≥3 correct answers out of 4 are considered discrete data, they did not satisfy the usual assumptions for ANOVA and were thus, analyzed using logistic regression techniques. Although these definitions have been used previously,17,19 we performed sensitivity analyses to ensure that different threshold definitions produced qualitatively similar results. For example, the number of correct responses for verbatim understanding ranged from 0–4. Comparisons were thus made between 0–3 vs 4 correct answers; 0–2 vs 3–4; and 0–1 vs 2–4. Similar analyses were conducted for the other outcomes and confirmed that different thresholds produced qualitatively similar results.

The final models were determined using the Akaike information criterion (AIC) model-selection criterion.38 AIC selects the best fitting model from other candidate models in minimizing information loss. Inter-rater reliability for the assessors' scores of understanding was measured using the Kappa statistic. Kappa scores for each of the elements of consent ranged from 0.76 to 1.0 (P < 0.001) revealing very good/excellent agreement. Data are described as mean differences and odds ratios with 95% confidence intervals (CI).

Sample size was based on previous data comparing parental understanding of consent information using different individual communication strategies e.g., use of different graphics and high versus low processability.7, 20 Based on these data, we considered it important to detect an effect size of 0.5 between levels of each factor. This estimate required at least 640 subjects i.e., 40 subjects per cell in the 16-cell FFD (α = 0.05, β = 0.1, 2-sided).

Results

A total of 871 parents were approached to participate in this study of whom 209 declined and 22 withdrew. Data are thus presented for 640 parents. There were no differences in demographics between the 16 consent groups thus; table 2 provides a summary of the entire sample.

Table 2
Parental demographics

Overall, 182 (28.5%) parents reported that they had read “most” and 441 (69.0%) “all” of the consent form to which they were randomized. Short forms were more likely to be read completely compared with long forms (OR, 95% CI = 1.75, 1.23–2.49). Of those who did not read the entire form, 33 (5.2%) skipped the study purpose, 31 (4.9%) the procedures, 30 (4.7%) the benefits, 84 (13.2%) privacy issues, and 108 (17.0%) contact information.

Tables 3 and and44 describe the logistic regression analyses for verbatim and gist understanding, respectively. As shown in table 3, high processability and inclusion of graphs had significant positive effects on verbatim understanding as indicated by their positive coefficients. The odds ratio for inclusion of graphs, for example, showed that the odds of verbatim understanding were increased by 50% if graphs were used. Longer consent forms, on the other hand, had a negative effect on verbatim understanding. For gist understanding (table 4) high processability, inclusion of graphs, and verbal disclosure had significant positive effects, whereas readability (12th grade) had a negative effect. Logistic regression analyses for understanding of the individual elements of consent showed that high processability had significant positive effects on understanding of the study's purpose, procedures, risks, benefits, and study duration. Use of graphs had positive effects on the understanding of risks and alternatives and supplemental verbal disclosure had a positive effect on understanding of the procedures. In all regression models high numeracy and Caucasian race predicted improved understanding of the consent information.

Table 3
Final Logistic Regression Model for Verbatim Understanding* of Risks and Benefits
Table 4
Final Logistic Regression Model for Gist Understanding* of Risks and Benefits

Based on the results of the regression models, we assigned positive attributes to: 8th grade reading, high processability, graphics, verbal disclosure, and shortened forms. We then examined the effect of the number of positive attributes (1, 3, or 5) per consent form on parents' understanding and perceptions (Table 5). Results showed that although inclusion of all 5 positive attributes resulted in a trend towards greatest understanding, forms with at least 3 positive message attributes were sufficient to improve understanding significantly over those containing only one positive attribute. The number of positive attributes had no effect on understanding of voluntariness, freedom to withdraw, confidentiality, study duration, and contact information. Given that 3 positive attributes were sufficient to improve understanding, the effect of combining the 3 messages with the strongest main effects i.e., 8th grade reading, high processability, and graphics was examined. Forms containing these specific attributes resulted in significantly greater gist and verbatim understanding compared with forms without these attributes (mean difference, 95% CI = 0.57, 0.26–0.88 correct responses out of 7 and 0.54, 0.20–0.88 correct responses out of 4 for gist and verbatim, respectively). Similarly, understanding of the study purpose (0.22, 0.07–0.36 out of 2 correct responses), the risks (0.22, 0.07–0.38 out of 2), and benefits (0.17, 0.005–0.33 out of 2) were better understood when these 3 attributes were included.

Table 5
Effects of the Number of Positive Message Attributes on Parents' Understanding and Preferences

When shown the two documents with contrasting attributes (#8 vs #10, see table 1 and eAppendices 1 and 2), 460 (74.3%) parents preferred the consent form with the positive attributes (#8). Comments included “less words, easier to understand, simpler, like the bullet points, key points more obvious.” Those who preferred form #10 reported that they preferred more information and detail.

Discussion

Previously, Flory et al. conducted a systematic review of interventions designed to improve understanding of informed consent for research.8 This review highlighted several individual interventions used in the present study that improved understanding of consent including enhanced consent forms (i.e., improved readability, reduced length, and formatting), and verbal disclosure. 9, 10, 12, 23 However, this review also noted that these interventions were not consistently effective suggesting that any single intervention cannot be “all things to all people.” To date, there have been limited data with respect to the effectiveness of combining interventions, thus, our study is important in identifying specific strategies, that when combined, enhance understanding on a more consistent level.

The present study identified several active communication strategies that improved parental understanding of consent information. Furthermore, data showed that consent forms containing at least 3 active communication strategies resulted in enhanced understanding. In particular, consent forms written at 8th grade level with high processability and use of pictographs provided the best format for understanding risk/benefit statistics. Document processability was a strong and consistent predictor of understanding in our study. Improved processability was achieved simply using larger font, more white space and by bolding and underlining important information. Previously, we showed that consent/assent forms with high processability significantly improved understanding of research information and were preferred by 82% of parents and children compared with a standard form.7, 39 The use of risk/benefit graphics also resulted in improved understanding of risks. We chose pictographs as the graphic of choice as previous work suggests that this format is both preferred and better understood compared with text and other graphical formats.17, 19, 20 Graphical depiction of risks and benefits may work because they have greater visual salience, require less cognitive effort and thus, are easier to understand.4043

It was surprising that health literacy was not an independent predictor of understanding given that previous studies have shown the importance of health literacy in understanding health information. A likely reason for this was that our population was skewed towards those with good health literacy and this likely confounded the results. Despite this, the consent forms written at an 8th grade level were better understood compared with those written at the 12th grade level and thus, reemphasizes the importance of reducing the grade reading level of these forms.

Numeracy, however, was a significant predictor of understanding. Given that innumerate individuals are more likely to make risk/benefit trade-offs based on emotion or trust rather than numbers,44 these results are significant in emphasizing the importance of improving the presentation of risk/benefit statistics to those with low numeracy. Overall, Caucasian parents had higher numeracy and literacy abilities compared with minority parents and were more likely to have been educated beyond high school. However, even when controlling for these factors, Caucasian parents had better understanding compared with minority parents. These results are important in that a lack of understanding may contribute to greater distrust or disparities in research participation.45, 46

There are limitations to our approach that must be recognized. First, this study was based on a hypothetical study and as such may not fully replicate the real-life situation. To obviate this concern, the consent forms and the process of consent for this study mimicked our routine research practice. Furthermore, there is precedence for the use of simulated methodologies and strong evidence showing that behaviors based on real and simulated methodologies are highly correlated.47, 48 Second, while the supplemental verbal disclosure used in this study was standardized we recognize that, in practice, more or less information may be given depending on the investigator-subject interaction.

This study highlights the importance of employing several different active communication strategies in the design of clinical consent forms. Results suggest that incorporation of at least 3 of the identified active strategies, serves to significantly improve understanding of the information and perceptions of the message delivery. Although these results are based on parents' understanding of pediatric consent documents it is reasonable to assume that they would also be relevant to consent forms for adult research subjects and for patients who consent to treatment. We therefore recommend that investigators, clinicians, IRBs, and study sponsors consider inclusion of these active strategies as a simple, effective, and inexpensive means to optimize the consent message and enhance subject understanding and participation.

Acknowledgements

Supported by a grant to Dr. Tait from The National Institutes of Health, NICHD, (R01 HD053594). The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs.

References

1. Committee on Bioethics American Academy of Pediatrics. Informed consent, parental permission, and assent in pediatric practice. Pediatrics. 1995;95(2):314–7. [PubMed]
2. Department of Health and Human Services . Protection of Human Subjects. 1991. 45 CFR 46.
3. Tait AR, Voepel-Lewis T, Malviya S. Do they understand? (Part I): Parental consent for children participating in clinical anesthesia and surgery research. Anesthesiology. 2003;98(3):603–08. [PubMed]
4. Joffe S, Cook E, Cleary P, Clark J, Weeks J. Quality of informed consent: A new measure of understanding among research subjects. J Natl Cancer Inst. 2001;93(2):139–47. [PubMed]
5. Van Stuijvenberg M, Suur M, De Vos S, et al. Informed consent, parental awareness, and reasons for participating in a randomised controlled study. Arch Dis Child. 1998;79(2):120–25. [PMC free article] [PubMed]
6. Miller C, Searight H, Grable D, Schwartz R, Sowell C, Barbarash R. Comprehension and recall of the informational content of the informed consent document: an evaluation of 168 patients in a controlled clinical trial. J Clin Res Drug Dev. 1994;8(4):237–48.
7. Tait AR, Voepel-Lewis T, Malviya S, Philipson S. Improving the readability and processability of a pediatric informed consent document: effects on parents' understanding. Arch Pediatr Adolesc Med. 2005;159(4):347–52. [PubMed]
8. Flory J, Emanuel E. Interventions to improve research participants' understanding in informed consent for research. JAMA. 2004;202(13):1593–601. [PubMed]
9. Taub H, Baker M, Sturr J. Informed consent for research: Effects of readability, patient age, and education. Am Geriatr Soc. 1986;34(8):601–06. [PubMed]
10. Bjorn P, Rossel P, Holm S. Can the written information to research subjects be improved? - an empirical study. J Med Ethics. 1999;25(3):263–67. [PMC free article] [PubMed]
11. Agre P, Campbell F, Goldman B. Improving consent: the medium is not the message. IRB. 2003;25(5):S11–S19. [PubMed]
12. Dresden G, Levitt M. Modifying a standard industry clinical trial consent form improves patient information retention as part of the informed consent process. Acad Emerg Med. 2001;8(3):246–52. [PubMed]
13. Murgatroyd R, Cooper R. Readability of informed consent forms. Am J Hosp Pharm. 1991;48(12):2651–2. [PubMed]
14. Tarnowski K, Allen D, Mayhall C, Kelly P. Readability of pediatric biomedical research informed consent forms. Pediatrics. 1990;85(1):58–62. [PubMed]
15. Irwin J, Davis C. Assessing readability: A checklist approach. J Reading. 1980;24:124–30.
16. Philipson SJ, Doyle MA, Gabram SG, Nightingale C, Philipson EH. Informed consent for research: a study to evaluate readability and processability to effect change. J Investig Med. 1995;43(5):459–67. [PubMed]
17. Hawley S, Zikmund-Fisher B, Ubel P, Jancovic M, Lucas T, Fagerlin A. The impact of the format of graphical presentation on health-related knowledge and treatment choices. Patient Educ Counsel. 2008;73(3):448–55. [PubMed]
18. Houts P, Witmer J, Egeth H, Loscalzo M, Zabora J. Using pictographs to enhance recall of spoken medical instructions. Patient Educ Counsel. 2001;43(3):231–42. [PubMed]
19. Tait AR, Voepel-Lewis T, Zikmund-Fisher B, Fagerlin A. The effect of format on parents' understanding of the risks and benefits of clinical research: A comparison between text, tables, and graphics. J Hlth Comm. 2010;15(5):487–501. [PMC free article] [PubMed]
20. Tait AR, Voepel-Lewis T, Zikmund-Fisher B, Fagerlin A. Presenting research risks and benefits to parents: Does format matter? Anes Analg. 2010;111(3):718–23. [PMC free article] [PubMed]
21. Aaronson N, Visser-Pol E, Leenhouts G, et al. Telephone-based nursing intervention improves the effectiveness of the informed consent process in cancer clinical trials. J Clin Oncol. 1996;14(3):984–96. [PubMed]
22. Fitzgerald D, Marotte C, Verdier R, Johnson WJ, Pape J. Comprehension during informed consent in a less-developed country. Lancet. 2002;360(9342):1301–02. [PubMed]
23. Kucia A, Horowitz J. Is informed consent to clinical trials an “upside selective” process in acute coronary syndromes? Am Heart J. 2000;140(1):94–97. [PubMed]
24. Tait AR, Voepel-Lewis T, Malviya S. Factors that influence parents' assessments of the risks and benefits of research involving their children. Pediatrics. 2004;113(4):727–32. [PubMed]
25. Nair V, Pregibon D. Analyzing dispersion effects from replicated factorial experiments. Technometrics. 1988;30(3):247–57.
26. Sacks J, Welch W, Mitchell T, Wynn H. Design and analysis of computer experiments. Stat Sci. 1989;4(4):409–23.
27. Welch W, Buck R, Sacks J, Wynn H, Mitchell T, Morris M. Screening, predicting, and computer experiments. Technometrics. 1992;34(1):15–25.
28. Collins L, Murphy S, Nair V, Strecher V. A strategy for optimizing and evaluating behavioral interventions. Ann Behav Med. 2005;30(1):66–73. [PubMed]
29. Miller C, O'Donnell D, Searight H, Barbarash R. The Deaconess Informed Consent Comprehension Test: an assessment tool for clinical research subjects. Pharmacotherapy. 1996;16(5):872–8. [PubMed]
30. Tait AR, Voepel-Lewis T, Malviya S. Do they understand? (Part II): Assent of children participating in clinical anesthesia and surgical research. Anesthesiology. 2003;98(3):609–14. [PubMed]
31. Brainerd C, Reyna V. Gist is the grist: Fuzzy-trace theory and the new intuitionism. Devel Rev. 1990;10(3):3–47.
32. Reyna V. A theory of medical decision making in health: Fuzzy trace theory. Med Dec Making. 2008;28(6):850–65. [PMC free article] [PubMed]
33. Davis T, Long S, Jackson R, et al. Rapid estimate of adult literacy in medicine: A shortened screening instrument. Fam Med. 1993;25(6):391–95. [PubMed]
34. Fagerlin A, Zikmund-Fisher B, Ubel P, Jankovic A, Derry H, Smith D. Measuring numeracy without a math test: Development of the subjective numeracy scale (SNS) Med Dec Making. 2007;27(5):672–80. [PubMed]
35. Zikmund-Fisher B, Smith D, Ubel P, Fagerlin A. A validation of the subjective numeracy scale: effects of low numeracy on comprehension of risk communication and utility elicitations. Med Dec Making. 2007;27(5):663–71. [PubMed]
36. Cella D, Perry S. Reliability and concurrent validity of three visual analogue mood scales. Psychol Reports. 1986;59(2):827–33. [PubMed]
37. Folstein M, Luria R. Reliability, validity and clinical application of the visual analogue mood scale. Psychol Med. 1973;3(4):479–86. [PubMed]
38. Akaike H. A new look at the statistical model identification. IEEE Transactions on Automatic Control. 1974;19(6):716–23.
39. Tait AR, Voepel-Lewis T, Malviya S. Presenting research information to children: A tale of two methods. Anes Analg. 2007;105(2):358–64. [PubMed]
40. Price M, Cameron R, Butow P. Communicating risk information: The influence of graphical display format on quantitative information perception-accuracy, comprehension and preferences. Patient Educ and Counsel. 2007;69(1–3):121–28. [PubMed]
41. Burkell J. What are the chances? Evaluating risk and benefit information in consumer health materials. J Med Library Assoc. 2004;92(2):200–08. [PMC free article] [PubMed]
42. Feldman-Stewart D, Brundage M, Zotov V. Further insight into the perception of quantitaive information: Judgments of gist in treatment decisions. Med Dec Making. 2007;27(1):34–43. [PubMed]
43. Waters E, Weinstein N, Colditz G, Emmons K. Formats for improving risk communication in medical tradeoff decision. J Hlth Comm. 2006;11(2):167–82. [PubMed]
44. Peters E, Hibbard J, Slovic P, Dieckmann N. Numeracy skill and the communication, comprehension, and use of risk-benefit information. Hlth Affairs. 2007;26(3):741–48. [PubMed]
45. Gorelick P, Harris Y, Burnett B, Bonecutter F. The recruitment triangle: Reasons why African Americans enroll, refuse to enroll, or voluntarrily withdraw from a clinical trial. J Nat Med Assoc. 2000;90(3):141–45. [PMC free article] [PubMed]
46. Hardin R. Trustworthiness. Ethics. 1996;107(1):26–42.
47. Jago A, Vroom V. Predicting leader behavior from a measure of behavioral intent. Acad Manage J. 1978;21(4):715–21.
48. Robinson M, Clore G. Simulation, scenarios, and emotional appraisal: Testing the convergence of real and imagined reactions to emotional stimuli. Personal Social Psychol Bull. 2001;27(11):1520–32.