Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Arch Phys Med Rehabil. Author manuscript; available in PMC 2013 March 1.
Published in final edited form as:
PMCID: PMC3290736

Quality of care indicators for the rehabilitation of children with traumatic brain injury

Frederick P. Rivara, MD, MPH,1,2,3 Stephanie K. Ennis, MS, CCC-SLP,1,4 Rita Mangione-Smith, MD, MPH,2,3 Ellen J. MacKenzie, PhD,5,6 Kenneth M. Jaffe, MD,1,4 and The National Expert Panel for the Development of Pediatric Rehabilitation Quality Care Indicators



To develop measurement tools for assessing compliance with identifiable processes of inpatient care for children with traumatic brain injury that are reliable, valid, and amenable to implementation.


Literature review and expert panel using the RAND/UCLA Appropriateness Method and a Delphi technique.


Not applicable


Children with traumatic brain injury (TBI)


Not applicable

Main outcome measures

Quality of care indicators


A total of 119 indicators were developed across the domains of general management; family-centered care; cognitive-communication, speech, language and swallowing impairments; gross and fine motor skill impairments; neuropsychological, social and behavioral impairments; school re-entry; community integration. There was a high degree of agreement on these indicators as valid and feasible quality measures for children with TBI.


These indicators are an important step toward building a better base of evidence about the effectiveness and efficiency of the components of acute inpatient rehabilitation for pediatric patients with TBI.

MeSH keywords: brain injuries, quality of health care

Increasing attention is being given by providers, patients and policy makers to the quality of health care that is delivered. The Institute of Medicine (IOM) purports that quality depends on the degree to which health services promote desired health outcomes, reflect current professional knowledge and integrates six quality characteristics: safety, effectiveness, equity, timeliness, patient-centered, and efficiency.1 Delivering quality care requires indicators by which that quality can be measured. As stated by Duncan and Velozo2 in the proceedings from the State of the Science Conference on Rehabilitation, “establishing systems of care in which some performance indicators of good care are established and monitored, and for which health care providers are held accountable, contributes to improved quality.”

The structure-process-outcome paradigm for quality developed by Donabedian,3 was recently endorsed by Duncan and Velozo2 as an appropriate framework for assessing quality and outcomes of post acute care. The impact of the structure and organization of care on outcomes is largely through the processes of that care. While increasing emphasis has been placed on the functional outcomes after serious injury and illness, very little is known about how the processes of care relate to these outcomes.

The focus of this study was children with traumatic brain injury (TBI) because it is one of the most common reasons for acute inpatient pediatric rehabilitation and it involves a multidisciplinary approach that includes many different domains of care.4 While TBI is the most important cause of death from trauma to children,57 most children with TBI survive, and many of these children need rehabilitation in order to improve their functional outcomes. This article presents evidence-based and expert-endorsed measures of the processes of acute pediatric rehabilitation. In another article, we present the structural and organizational indicators of high quality care.8 Our goal was the development of measurement tools for assessing compliance with identifiable processes of care that are reliable (measurement results are reproducible), valid (measure elements of care that relate to outcome), and amenable to implementation (feasibility). Of primary interest were process measures that have been shown (based on a systematic literature review) or are hypothesized (using the collective judgment of experts) to have an impact on quality of patient care and outcomes. Development of these standards of care is an important step in reducing variations in care, and determining which care results in optimal outcomes


Our approach drew on methodology used by the Agency for Heath Care Policy and Research9 and RAND1013 in their development of appropriateness and quality of care measures (i.e., RAND/UCLA Appropriateness Method [RAM]), and combined a systematic review of scientific evidence regarding “best practices” with ratings by an expert panel of practitioners and researchers.

Systematic Review of the Literature

We started by first examining existing guidelines and prior reviews of the evidence for effectiveness of specific processes of care for the acute inpatient rehabilitation of children and adolescents with TBI.1422 We then conducted a systematic literature review using the major subject headings Rehabilitation and Traumatic Brain Injury paired with key words selected for each domain. Searches were conducted using the following databases: PubMed (1948–2009), CINHAL (1937–2009), Web of Science (1900–2009), PsycINFO (1806–2009), PEDro (1929–2009), Family Studies Abstracts (EBSCO; 1979–2009), ISI Web of Science (1900–2009), the Cochrane Library (2005–2009), REHABDATA (1956–2009), OTseeker (2002–2009), HAPI (Health & Psychosocial Instruments; 1985–2009, limited to children), Gale Academic OneFile (1980–2009), ProQuest eJournals (1998–2009) and ERIC (1966–2009). A manual search of bibliographies of key articles and reviews was completed, and the journals and websites of rehabilitation associations (e.g., American Speech Language and Hearing Association, American Therapeutic Recreation Association) were searched for relevant abstracts, proceedings or clinical guidelines. Search terms consisted of the major topic headings Rehabilitation and Brain Injury, combined with search strings specific to seven areas of rehabilitation care. We included randomized controlled trials, non-randomized experimental studies, case-control and cohort studies, case-series and expert opinion reports. We also recruited a project advisory committee comprising experts in pediatrics, physiatry, speech and language pathology, occupational therapy, physical therapy, social work, rehabilitation nursing, education, therapeutic recreation, and neuropsychology to supplement the database searches to identify relevant literature in the field. Articles retrieved from these search strategies were reviewed by a single research coordinator. The level of evidence was judged using the criteria of the Oxford Centre for Evidence-Based Medicine.23 A total of 226 references were reviewed in developing the indicators.

We were guided by a model of three 3 overarching categories to characterize a quality rehabilitation episode: (1) general attributes of coordinated, multidisciplinary, and individually targeted rehabilitation that include: baseline assessments, goal setting, treatment planning, ongoing monitoring of progress and the involvement and coordination of care across multiple disciplines; (2) adherence to specific management protocols (general and TBI specific) aimed at minimizing impairments and disabilities and preventing secondary conditions; and (3) involvement of the patient and family in all aspects of care including discharge planning and education. These categories were based on prior work of quality of care indicators for rehabilitation of adult patients after stroke.2527 The literature was summarized in seven domains chosen by the authors as logical categories of rehabilitation care for children with TBI: general management, family-centered care, cognition and communication, motor skills, neuropsychological assessment and social skills, school re-entry, and community integration.

We then constituted an interdisciplinary group of project advisors representing expertise in each of the seven domains of care to advise us in the development of the indicators. These individuals were prominent experts in their fields, and distinct from the National Expert Panel of pediatric physiatrists who rated the indicators as described below. Each project advisor was fully briefed on the purpose of the study and our conceptual framework, and then asked to read the background papers, review the items proposed by the study team, identify gaps, and suggest areas for improvement. The opinions elicited from the advisors were compiled by the study team and used to revise the set of dimensions and items within dimensions.

Delphi Process with National Expert Panel

We developed a National Expert Panel of nine, geographically distributed, pediatric physiatrists to conduct the next stage of indicator development. These individuals were selected based on their clinical and research leadership in rehabilitation of pediatric TBI, and national reputation. Consistent with the RAND/UCLA Appropriateness Method (RAM), a Delphi technique was used to draw out group opinion regarding these measures. The Delphi method is a collective rating process that has been widely used as a tool for solving problems in health and medicine and allows elicitation of expert opinion in an iterative and systematic manner.13, 2830 This method of selection of indicators has been shown to be reliable.31

In the first round, each panelist was sent electronic summaries of the key research in each domain, a list of potential indicators, and instructions for scoring of indicators. Panelists were asked to rate each indicator on a nine point scale (with nine being highest) for two characteristics: validity and feasibility. Validity was assessed by the panelists based on: (1) the scientific evidence or, when lacking, the collective expert opinion of the appropriateness of the indicator; (2) the perceived health benefits to patients; (3) the control of the health care provider over the processes of care for that indicator; and (4) their opinion that patients receiving the specified care would be receiving high quality care. Feasibility was based on the perceived ability to find the information in the medical record, and that decisions about adherence to the indicator based on medical record information would be reliable.

Measures of central tendency (median) and dispersion (mean absolute deviation from the median) subsequently were calculated for each indicator, and a conference call was arranged with all panelists to discuss those indicators for which there was substantial disagreement. In accordance with RAM guidelines, indicators with median ratings in the top third of the validity scale (score ≥ 7) were classified as appropriate, those with median ratings in the bottom third (score of 1 to 3) were classified as inappropriate, and those with intermediate median ratings (score of 4 to 6) were considered uncertain.32 In advance of the second rating round, panelists convened by phone to discuss any quality indicators classified as uncertain, or receiving a median feasibility score less than 4 or a dispersion measure greater than 1 on either scale. Ratings classified as inappropriate were omitted. The goal of this phone discussion was not to develop a consensus, but rather to allow panelists to share opinions, indentify any confusing wording, and suggest additions, omissions or modifications to the reviewed indicators. After the telephone call, having heard and participated in the discussion of expert opinion, the panelists were sent updated quality indicators and ratings sheets incorporating any suggested modifications and independently re-rated the indicators, using the same nine point scale for validity and feasibility.

The final set of indicators was selected using RAM procedures for conducting tests of hypotheses about the distribution of ratings in a hypothetical population of repeated ratings. Ratings were classified as “with agreement” or “with disagreement” using standard definitions for 9-member panels.32,33 In brief, to be classified as “with agreement”, no more than two panelist ratings could fall outside the tertile region containing the median score (7 to 9). An indicator was classified as “with disagreement” when three or more panelist ratings fell between 1 and 3 on the rating scale and three or more were between 7 and 9. Any indicator classified as “with disagreement” was rejected from the final set.


Initially, 133 indicators were developed. Thirty-nine indicators (30%) were restructured or reworded for clarity, 15 (11%) were omitted based on the rating criteria described above, 3 (2%) were consolidated and an additional 3 were added based on panelists’ suggestions. The final number of indicators in each domain is shown in Table 1; there were 119 in total. There was a high degree of agreement on these indicators as valid and feasible quality measures for children with TBI, as evidenced by an indicator retention rate of 89%, which is relatively high when compared with prior studies reporting indicator retention rates ranging from 70–87%.3436 As can be seen, many indicators were based on a combination of expert opinion and some scientific evidence. The full set of indicators is available in the online appendix. The indicators in each domain are briefly reviewed below.

Table 1
Number of process of care quality indicators for each domain

General management

These indicators focus on the components of care deemed necessary in the initial general medical assessment of children with TBI as well as the pre-injury functioning of the child and family. They assess whether the child is seen by a multidisciplinary team, the development of a treatment and monitoring plan to track the improvement of the child during the inpatient rehabilitation hospitalization, and for the coordination of care among a multidisciplinary team. Indicators were also developed for the management of the overall health of the child such as sleep and nutrition, and the prevention of secondary complications such as decubitus ulcers and nosocomial infections.

Family-centered care

Indicators for this and all the other domains were grouped into assessment, interventions and follow-up care. Assessment indicators were developed for both the initial and ongoing assessment of the impact of the TBI on the family, including needs, supports, coping skills and general family functioning. These provide the basis for intervention to educate the family about TBI and assist them with identified needs, both during the stay and post-discharge. Post-discharge services include community-based respites for social support, telephone counseling services and a contact person on the rehabilitation team.

Cognitive-communication, speech, language and swallowing impairments

Assessment indicators focused on impairments in cognitive-communication and language skills, motor speech, swallowing and aspiration risk. Intervention indicators primarily focused on functional strategies to be used in school and other natural contexts, such as strategies for social language use, memory and executive functions as well as a plan for continuing cognitive-communication skills development post-discharge.

Gross and fine motor skill impairments

Indicators were developed for the assessment of gait, mobility, balance, performance of activities of daily living (ADLs), vision and kitchen safety. Intervention indicators focused on treating impairments in range of motion, strength, tone mobility, ADLs, use of upper extremities. Detailed indicators were developed on the training of caregivers (e.g., to assist with transfers) as well as post-discharge referrals to physical and occupational therapists and durable equipment needs.

Neuropsychological, social and behavioral impairments

The process of care indicators for the assessment of neuropsychological function and impairments included documentation of pre-injury functioning and measures of social-emotional behavior. The indicators for intervention reflect processes for implementing and measuring the effectiveness of behavioral intervention plans for both children and their families and medications, as well as documented opportunities for practice of skill generalization.

School re-entry

Indicators were developed for the educational evaluation of patients by educational specialists in a simulated classroom setting, and coordination of re-entry with school personnel. Processes of care for school re-entry included provision of information about the child and educational placements/services, and ongoing communication with school staff, families and school liaisons.

Community integration

Indicators in this domain focused on assessment of pre-injury lifestyle, participation in leisure, play, and recreational activities, and social interactions with peers and family. It included structured outings during the inpatient stay, education about transportation and recreational safety, and plans for time to return safely to former community activities.


In 2003, the Guidelines for the Acute Management of TBI in Infants, Children and Adolescents were published,37 following similar work to establish guidelines for management of adult TBI.38 These guidelines were evidence-based recommendations made by a team of experts in the acute (emergency department and intensive care unit) care of patients with moderate and severe TBI. Recent studies indicate that full implementation of these guidelines can save a substantial number of lives while at the same time reducing health care costs.39

Unfortunately, similar work had not been done for the initial acute inpatient rehabilitation of children with TBI. There have been only limited efforts to develop process of care measures for adult rehabilitation. Hoenig et al24,25 examined structural and procedural elements of post-stroke rehabilitation centers within the Veterans Administration hospital system. Staffing ratios for nurses and physicians, diversity of physicians/ staff, presence of a simulated home environment, and total number of care settings on site, revealed positive independent associations with discharge location and total length of stay. Studies showed that processes of care as measured by this instrument, had a significant association with functional outcomes,34 and with greater patient satisfaction.35

This study extends the work of the Commission on Accreditation of Rehabilitation Facilities (CARF), an accreditor of health and human services including rehabilitation programs. CARF has developed standards that facilities must meet as part of the accreditation process. The CARF expert committee uses evidence made available by participants on the committee and receives informal input from users and consumers in the development of these stadnards. While there is an attempt to obtain consensus of the group, CARF does not use a formal Delphi process for arriving at consensus and the search of the literature for the development of standards may not be as formalized as that used by us. While CARF has endorsed identifying performance quality indicators,36 these are not available in the literature.


The limitations to these indicators must be acknowledged. While we conducted thorough reviews of the literature for indicators in each domain, it is possible that some studies were missed. Since few controlled trials of pediatric interventions were identified, we thus had to rely on expert consensus for many indicators. We sought to create a National Expert Panel that reflected geographic and institutional diversity. While there was good agreement of the panel on the choice of indicators, another panel might have rated the indicators differently. The 119 indicators that were developed must be revised and refined as new data become available and the association of these indicators with outcomes is determined and clarified.


Seven key domains of care related to the acute inpatient rehabilitation for pediatric TBi have been developed. The next step is to validate the validity and feasibility of implementing these domain-specific measures in individual acute pediatrics rehabilitation programs nationally and to measure variations in the care delivered to children with TBI hospitalized in rehabilitation centers in the US. These indicators are a necessary step toward building a better base of evidence about the effectiveness and efficiency of the components of rehabilitation for pediatric patients with TBI.

Supplementary Material



This work was supported by Grant number R21 HD059049-01A1 from the Eunice Kennedy Shriver National Institute of Child Health & Human Development.

We certify that no party having a direct interest in the results of the research supporting this article has or will confer a benefit on us or on any organization with which we are associated AND, if applicable, we certify that all financial and material support for this research (eg, NIH or NHS grants) and work are clearly identified in the title page of the manuscript.


RAND/UCLA Appropriateness Method
Traumatic Brain Injury
Activity of daily living


Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.


1. Institute Of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, D.C.: National Academies Press; 2001. [PubMed]
2. Duncan PW, Velozo CA. State-of-the-science on postacute rehabilitation: measurement and methodologies for assessing quality and establishing policy for postacute care. Arch Phys Med Rehabil. 2007;88(11):1482–1487. [PubMed]
3. Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q. 1966;44(3) Suppl:166–206. [PubMed]
4. Shi J, Xiang H, Wheeler K, Smith GA, Stallones L, Groner J, Wang Z. Costs, mortality likelihood and outcomes of hospitalized US children with traumatic brain injuries. Brain Inj. 2009;23:602–611. [PMC free article] [PubMed]
5. Sills MR, Libby AM, Orton HD. Prehospital and in-hospital mortality: a comparison of intentional and unintentional traumatic brain injuries in Colorado children. Arch Pediatr Adolesc Med. 2005;159:665–670. [PubMed]
6. Ventura T, Harrison-Felix C, Carlson N, et al. Mortality after discharge from acute care hospitalization with traumatic brain injury: a population-based study. Arch Phys Med Rehabil. 91:20–29. [PubMed]
7. Shi J, Xiang H, Wheeler K, et al. Costs, mortality likelihood and outcomes of hospitalized US children with traumatic brain injuries. Brain Inj. 2009;23:602–611. [PMC free article] [PubMed]
8. Zumsteg JM, Ennis SK, Jaffe KM, Mangione-Smith R, MacKenzie EJ, Rivara FP. Quality of care indicators for the structure and organization of inpatient rehabilitation care of children with traumatic brain injury. Archives of Physical Medicine and Rehabilitation [PMC free article] [PubMed]
9. Grady ML, Schwarz HA, editors. Medical Effectiveness Research Data Methods. Rockville, MD: AHCPR; 1992.
10. Rubin HR, Pronovost P, Diette GB. From a process of care to a measure: the development and testing of a quality indicator. Int J Qual Health Care. 2001;13(6):489–496. [PubMed]
11. Winslow CM, Kosecoff JB, Chassin M, Kanouse DE, Brook RH. The appropriateness of performing coronary artery bypass surgery. Jama. 1988;260(4):505–509. [PubMed]
12. Brook RH, McGlynn EA, Cleary PD. Quality of health care. Part 2: measuring quality of care. N Engl J Med. 1996;335(13):966–970. [PubMed]
13. Mangione-Smith R, DeCristofaro AH, Setodji CM, Keesey J, Klein DJ, Adams JL, et al. The quality of ambulatory care delivered to children in the United States. N Engl J Med. 2007;357(15):1515–1523. [PubMed]
14. Carney N, Chesnut RM, Maynard H, Mann NC, Patterson P, Helfand M. Effect of cognitive rehabilitation on outcomes for persons with traumatic brain injury: A systematic review. J Head Trauma Rehabil. 1999;14(3):277–307. [PubMed]
15. Chesnut RM, Carney N, Maynard H, Mann NC, Patterson P, Helfand M. Summary report: evidence for the effectiveness of rehabilitation for persons with traumatic brain injury. J Head Trauma Rehabil. 1999;14(2):176–188. [PubMed]
16. Cicerone KD, Dahlberg C, Kalmar K, Langenbahn DM, Malec JF, Bergquist TF, et al. Evidence-based cognitive rehabilitation: recommendations for clinical practice. Arch Phys Med Rehabil. 2000;81(12):1596–1615. [PubMed]
17. Cicerone KD, Dahlberg C, Malec JF, Langenbahn DM, Felicetti T, Kneipp S, et al. Evidence-based cognitive rehabilitation: updated review of the literature from 1998 through 2002. Arch Phys Med Rehabil. 2005;86(8):1681–1692. [PubMed]
18. Ylvisaker M, Turkstra L, Coehlo C, Yorkston K, Kennedy M, Sohlberg MM, et al. Behavioural interventions for children and adults with behaviour disorders after TBI: a systematic review of the evidence. Brain Inj. 2007;21(8):769–805. [PubMed]
19. Ylvisaker M, Turkstra LS, Coelho C. Behavioral and social interventions for individuals with traumatic brain injury: a summary of the research with clinical implications. Semin Speech Lang. 2005;26(4):256–267. [PubMed]
20. Turkstra LS, Coelho C, Ylvisaker M. The use of standardized tests for individuals with cognitive-communication disorders. Semin Speech Lang. 2005;26(4):215–222. [PubMed]
21. Turkstra L, Ylvisaker M, Coelho C, Kennedy M, Moore-Sohlberg M, Avery J, et al. Practice guidelines for standadized assessment for persons with TBI. J MedIcal Speech-Language Pathol. 2005:13.
22. Moore AD, Stambrook M. Coping strategies and locus of control following traumatic brain injury: relationship to long-term outcome. Brain Inj. 1992;6(1):89–94. [PubMed]
23. Centre for Evidence-Based Medicine. Oxford Centre for Evidence-Based Medicine Levels of Evidence [web] [cited 2010 October 14, 2010];2009 March; Available from: URL:
24. Hoenig H, Duncan PW, Horner RD, Reker DM, Samsa GP, Dudley TK, et al. Structure, process, and outcomes in stroke rehabilitation. Med Care. 2002;40(11):1036–1047. [PubMed]
25. Hoenig H, Sloane R, Horner RD, Zolkewitz M, Reker D. Differences in rehabilitation services and outcomes among stroke patients cared for in veterans hospitals. Health Serv Res. 2001;35(6):1293–1318. [PMC free article] [PubMed]
26. Reker DM, Duncan PW, Horner RD, Hoenig H, Samsa GP, Hamilton BB, et al. Postacute stroke guideline compliance is associated with greater patient satisfaction. Arch Phys Med Rehabil. 2002;83(6):750–756. [PubMed]
27. LaClair BJ, Reker DM, Duncan PW, Horner RD, Hoenig H. Stroke care: a method for measuring compliance with AHCPR guidelines. Am J Phys Med Rehabil. 2001;80:235–242. [PubMed]
28. Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care. 1986;2(1):53–63. [PubMed]
29. McGlynn EA, Kerr EA, Asch SM. New approach to assessing clinical quality of care for women: the QA Tool system. Womens Health Issues. 1999;9(4):184–192. [PubMed]
30. Kahn KL, Kosecoff J, Chassin MR, Flynn MF, Fink A, Pattaphongse N, et al. Measuring the clinical appropriateness of the use of a procedure. Can we do it? Med Care. 1988;26(4):415–422. [PubMed]
31. Kravitz RL, Park RE, Kahan JP. Measuring the clinical consistency of panelists' appropriateness ratings: the case of coronary artery bypass surgery. Health Policy. 1997;42(2):135–143. [PubMed]
32. Brook R. The RAND/UCLA Appropriateness Method. In: McCormick KA, Moore SR, Siegel RA, editors. Methodological Perspectives. Rockville, Md: US Department of Health and Human Services; 1994.
33. Fitch K, Bernstein SJ, Aguilar MS, et al. The RAND/UCLA Appropriateness Method User’s Manual. Santa Monica, CA: RAND; 2001. p. 2006.
34. McGory ML, Kao KK, Shekelle PG, Rubenstein LZ, Leonardi MJ, Parikh JA, et al. Developing quality indicators for elderly surgical patients. Ann Surg. 2009;250(2):338–347. [PubMed]
35. Schuster MA, Asch SM, McGlynn EA, Kerr EA, Hardy AM, Gifford DS. Development of a quality of care measurement system for children and adolescents. Methodological considerations and comparisons with a system for adult women. Arch Pediatr Adolesc Med. 1997;151(11):1085–1092. [PubMed]
36. Wilkerson DL. Accreditation and the use of outcomes-orineted information systems. Arch Phys Med Rehabil. 1997;78 Suppl 4:S31–S35. [PubMed]
37. Brain Trauma Foundation. Guidelines for the Acute Medical management of Severe TBI in infants, children and adolescents. NY: 2003.
38. Brain Trauma Foundation. Guidelines for the management of severe TBI, 3rd edition. NY: 1995.
39. Faul M, Wald MM, Rutland-Brown W, Sullivent EE, Sattin RW. Using a cost-benefit analysis to estimate outcomes of a clinical treatment guideline: testing the Brain Trauma Foundation guidelines for the treatment of severe traumatic brain injury. J Trauma. 2007;63(6):1271–1278. [PubMed]