PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of neuMary Ann Liebert, Inc.Mary Ann Liebert, Inc.JournalsSearchAlerts
Journal of Neurotrauma
 
J Neurotrauma. 2009 July; 26(7): 1069–1075.
PMCID: PMC2989840

The Influence of Enrollment Criteria on Recruitment and Outcome Distribution in Traumatic Brain Injury Studies: Results from the Impact Study

Bob Roozenbeek,1,,2 Andrew I.R. Maas,corresponding author2,,3 Anthony Marmarou,4 Isabella Butcher,5 Hester F. Lingsma,1 Juan Lu,4 Gillian S. McHugh,5 Gordon D. Murray,5 and Ewout W. Steyerberg1
1Department of Public Health, Center for Medical Decision Making, Rotterdam, The Netherlands.
2Department of Neurosurgery, Erasmus MC, Rotterdam, The Netherlands.
3Department of Neurosurgery, University Hospital Antwerp, Antwerp, Belgium.
4Department of Neurosurgery, Virginia Commonwealth University, Richmond, Virginia.
5Division of Community Health Sciences, University of Edinburgh, Edinburgh, Scotland.
The IMPACT* Study Group

Abstract

Substantial heterogeneity exists among patients who suffer from traumatic brain injury (TBI). Strict enrollment criteria may diminish heterogeneity in randomized controlled trials (RCTs), but will also decrease recruitment and may affect the outcome distribution. The aim of this study was to investigate the influences of commonly used enrollment criteria for RCTs in TBI on potential recruitment and on outcome distribution. We used individual patient data from the International Mission on Prognosis and Analysis of Clinical Trials in TBI (IMPACT) database, including six therapeutic phase III RCTs (n = 5816) and three surveys (n = 2217) in TBI. The primary outcome was the Glasgow Outcome Scale (GOS) at 6 months after injury, which we dichotomized as favorable/unfavorable. We investigated the influences of commonly used enrollment criteria on recruitment and outcome distribution: time window between injury and admission to study hospital  8 h; age at injury  65 years;  1 reactive pupil; motor score > 1; Glasgow Coma Scale  8. Application of all enrollment criteria resulted in a large reduction of recruitment in both the surveys (up to 65%) and the RCTs (up to 41%). Among the remaining patients, fewer had an unfavorable outcome in both the surveys (original, 60%; remaining, 44%) and the RCTs (original, 43%; remaining, 38%). Applying these enrollment criteria to patients from the surveys resulted in an outcome distribution that approximated the outcome observed in the RCTs. The use of strict enrollment criteria leads to substantial reductions in the recruitment of RCTs in TBI. The outcome in TBI studies depends strongly on the enrollment criteria.

Key words: enrollment criteria, heterogeneity, outcome distribution, randomized controlled trial, recruitment, traumatic brain injury

Introduction

Traumatic brain injury (TBI) represents a serious health problem worldwide, posing high costs to society. Much research is undertaken to improve outcome after TBI. Comparing outcome between different series of TBI patients and demonstrating efficacy of new approaches in randomized clinical trials (RCTs) is notoriously difficult due to the inherent heterogeneity of TBI populations (Maas et al., 1999). Heterogeneity in terms of type and severity of injury, as well as in terms of prognostic risk, may cause imbalances between treatment groups and dilute treatment effects.

Imbalances in baseline characteristics between treatment groups have been reported in some RCTs (Marshall et al., 1998). Such imbalances may be prevented by block randomization (Bullock, 2002), stratified randomization (Bullock, 2002; Choi and Bullock, 2001; Maas et al., 1999), and minimization. We may achieve similar numbers of patients in each arm of the trial by random allocation in blocks (“block randomization”). Stratified randomization may be used to ensure that equal numbers will be allocated to each arm with a characteristic thought to affect prognosis. Furthermore, minimization may be used to ensure balance between trial arms for several patient factors. The first patient is truly randomly allocated; for each subsequent patient, the treatment allocation is identified, which minimizes the imbalance between groups at that time. Approaches for dealing with the heterogeneity in the analysis phase (e.g., covariate adjustment) have also been employed in TBI trials (Hernández et al., 2006).

Heterogeneity amongst the population under study may also be reduced by using strict enrollment criteria. Enrollment criteria aim to select patients that are most likely to benefit from the studied treatment, both from a mechanistic and prognostic perspective (Maas et al., 1999; Machado et al., 1999; Narayan et al., 2002). Three main goals are aimed for: (1) exclusion of patients with a very good or very bad prognosis, because one may expect that these patients will do well without any treatment or will do poorly no matter what treatment is given; (2) inclusion of patients who are likely to benefit from the treatment, because of the pathophysiologic background of the injury and the studied treatment; (3) reaching a distribution of approximately 50% of patients with a favorable outcome and 50% of patients with an unfavorable outcome in the study population, to increase statistical power for the detection of a true treatment effect.

Machado et al. (1999) have demonstrated that targeting a clinical trial to patients with an intermediate prognostic risk would permit a potential reduction in sample size of approximately 30%. The disadvantage of this approach, however, is that it may substantially reduce recruitment.

The objective of this study is to investigate the influences of common enrollment criteria on the potential recruitment and on the outcome distribution of RCTs in TBI.

Methods

Study population

We used individual patient data from the International Mission on Prognosis and Analysis of Clinical Trials in TBI (IMPACT) database. This database contains data of patients with moderate (Glasgow Coma Scale [GCS], 9–12) and severe (GCS  8) TBI from eight RCTs and three surveys conducted between 1984 and 1996. IMPACT links researchers in Belgium, the Netherlands, the United Kingdom, and the United States in a project addressing methodological problems in design and analysis of RCTs in TBI. Details of the different studies and data management of the IMPACT database have been described previously (Marmarou et al., 2007). The primary outcome measurement was the Glasgow Outcome Scale (GOS) at 6 months after injury. For our analyses, we dichotomized the GOS into unfavorable (GOS 1–3) versus favorable outcome (GOS 4–5). We selected patients over 14 years of age at the time of injury. The data from two RCTs (HIT-I and SKB) were not used in our analysis, because the numbers of patients in these studies were too small for the purpose of our study. This resulted in a cohort of 8033 adults: 2217 patients from surveys and 5816 patients from RCTs.

Selection by enrollment criteria

We investigated the influences of five commonly used enrollment criteria for TBI studies: time window between injury and admission to study hospital  8 h; age at injury  65 years; ≥1 reactive pupil; motor score > 1; and GCS  8. We further explored the influence of other cut-off values for time window (≤4, ≤6, and ≤12 h) and for age (≤55, ≤60, and ≤70 years). The enrollment criteria were applied separately and simultaneously. The influence on the recruitment was studied by comparing the original sample sizes to the new sample sizes with the investigated enrollment criteria applied. The change in outcome distribution was studied by comparing unfavorable outcome fractions at 6 months after the injury.

Statistical analysis

Descriptive statistics included medians and percentages. The software used was SPSS, version 11.0.1 (SPSS Inc., Chicago, IL). Figure 1 was produced with R software version 2.6.0 (The R Foundation, 2007).

FIG. 1.
The influence of the application of the investigated enrollment criteria (time window  8 h; age at injury  65 years; ≥1 reactive pupil; motor score > 1; GCS  8) ...

Results

The baseline characteristics of the original study population are summarized in Table 1. These reflect the effects of the different initial enrollment criteria per study.

Table 1.
Distribution of Patient Characteristics Across Three Surveys and Six RCTs from the IMPACT Study

Effects of enrollment criteria on recruitment

Effects of enrollment criteria on recruitment were more pronounced in the surveys than in the RCTs, since these had less stringent initial enrollment criteria (Table 1). When each of the studied criteria was applied separately in the surveys, “at least one reactive pupil” led to the largest reduction in recruitment (32%), followed by “time window < 8 h” (20%), “motor score > 1” (18%), “GCS  8” (15%), and “age at injury  65” (12%; Table 2). In the RCTs, the reductions were less (Table 2).

Table 2.
Numbers of Excluded Patients Across Studies by Application of Enrollment Criteria

As expected, using shorter time windows led to a greater reduction of recruitment in both the surveys and the RCTs (Table 3). In the surveys, the recruitment reduction decreased gradually with the use of higher maximum ages: 22% (≤55 years); 17% (≤60 years); 12% (≤65 years); and 9% (≤70 years; Table 4). The RCTs showed less reduction: 10% (≤55 years); 5% (≤60 years); 2% (≤65 years); and 0% (≤70 years; Table 4).

Table 3.
Numbers of Excluded Patients by Selection on Time Window
Table 4.
Numbers of Excluded Patients by Selection on Age at Injury

Application of all five studied enrollment criteria simultaneously led to a mean recruitment reduction of 65% in the surveys (Table 2, Fig. 1). In the RCTs, the mean reduction of the recruitment was 41%.

Effects of enrollment criteria on outcome distribution

The application of more stringent enrollment criteria had profound effects on the outcome distribution (Table 2). Effects were greatest for age, pupils, motor score, and GCS. In contrast to all other selection criteria, the GCS criterion mainly excluded patients with a favorable outcome. Patients excluded for a high age, bilaterally unresponsive pupils or an absent motor score had more unfavorable outcomes. Only modest effects of different time windows were seen on the outcome distribution (Table 3).

Combining all enrollment criteria had a pronounced effect in the outcome distribution, which was most evident in the surveys, reflecting the greater initial heterogeneity. In the surveys the percentage of unfavorable outcome prior to applying the enrollment criteria was 60%, and decreased to 44% after applying the enrollment criteria, approximating the outcome distribution observed in the RCTs.

Effects of the application of all enrollment criteria on the recruitment and ouctome distribution are illustrated in Figure 1. For example, the first gray bar of Figure 1 shows that application of all investigated enrollment criteria on the population of the Traumatic Coma Data Bank (TCDB) survey, resulted in selection of 253 patients, from whom 43% had an unfavorable outcome. The first black bar depicts 351 excluded patients from the TCDB population with an unfavorable proportion of 81%.

Discussion

Our evaluations show substantial effects of enrollment criteria both on potential recruitment and on outcome. The five studied enrollment criteria led to a joint reduction of recruitment of 65% in the surveys and 41% in the RCTs, and substantially narrowed the differences in outcome between surveys and RCTs that were initially present. Clearly, effects in individual studies varied with the stringency of original criteria.

As a limitation of our study, it must be noticed that we only focused on the influences of five enrollment criteria applied separately and simultaneously. We did not investigate any interactions between the different enrollment criteria. Structural abnormalities as visualized by computed tomography (CT) examination are currently commonly used as selection criterion for enrollment in RCTs, but we did not include these in our analyses, as the available information was not detailed enough in all IMPACT studies.

In TBI, Phase III RCTs traditionally aim to target the population that is most likely to benefit from the therapy under investigation, both from a mechanistic and prognostic perspective. Mechanistic targeting is based upon our—often very poor—potential for identifying patients in whom specific pathophysiologic mechanisms are active. In prognostic targeting, enrollment criteria aim to obtain a homogeneous population with respect to expected outcome, excluding those with either a very poor or very good prognosis. For this reason, most trials have excluded patients over the age of 65, and patients with an absent motor response or unreactive pupils. Indeed, our results show a very high rate of unfavorable outcome in patients excluded for these reasons, indicating that this approach is appropriate.

Nevertheless, interpretation should be with caution: TBI is being seen more frequently in elderly patients and, with the ageing population, the average age of TBI patients is increasing (Maas et al., 2007). Elderly patients with TBI therefore represent an important subpopulation. The poor outcome in elderly patients observed in our series is primarily caused by severe TBI (GCS  8), and no inference may be made as to patients with moderate TBI.

With the current frequent use of early sedation, intubation and neuromuscular blockade, accurate assessment of the motor scale may be confounded, leading to erroneously recording the motor score as being absent. Likewise, small unreactive pupils may be caused by effects of deep sedation. Consequently, the use of these criteria in isolation may be hazardous. Taken in combination, however, reliability will be increased and for this reason we prefer to classify by prognosis. A limited number of admission characteristics can be used to establish the baseline prognostic risk in individual patients (Hukkelhoven et al., 2005, 2006).

Machado et al. (1999) have shown that targeting a trial towards patients with an intermediate prognosis may reduce the required sample size by 30%. These conclusions were reached from analyses performed on the European Brain Injury Consortium (EBIC) survey. Our results, including also analysis of the EBIC survey, however, show that the reduction in recruitment due to more stringent enrollment criteria may be even higher, thus obviating the potential benefit. This touches on the debate whether focused, targeted Phase III trials, or large mega trials (with inherent heterogeneity) should be preferred in the field of TBI. To date, only one mega-trial has been reported in TBI (Edwards et al., 2005).

We found that time window is a crucial factor in determining recruitment, but that little effect exists on the outcome distribution. Recruitment reductions by time window selection varied from 38% (≤4 h) to 15% (≤12 h) in the surveys. In the RCTs, there was a variation of 11% (≤4 h) to <1% (≤12 h) in reduction of recruitment.

Determination of the appropriate time window for a study should preferably be based on knowledge of the pathophysiologic mechanism targeted, and the time at which this mechanism may be active. Extrapolating results from experimental studies to the clinical situation may, however, be difficult. In practice, time windows are frequently determined based on “best guess” and the clinical perception as to the time frame within which a reasonable number of subjects may be recruited. Our studies show a substantial decline in recruitment when time windows are shortened from 6 to 4 hours, but less so on comparing a 6 hour time window to 8 hours. However, these effects of time window on recruitment were based on the available data in the IMPACT database about the time of injury and the time of admission to study hospital. In practice, the actual time windows to study drug administration are longer, due to organizational and logistical reasons such as informed consent procedures (Kompanje et al., 2005). Therefore, our observations are likely to be an underestimation of the actual decrease in recruitment by time window selection.

We also probably underestimate the actual recruitment reduction by application of selection criteria because we investigated the effect of selection criteria on pre-selected study populations. In the SAPHIR trial, for example, we have learned from the screening logs that only 19% of the number of patients initially screened was enrolled in the trial; 81% was excluded because of not meeting enrollment criteria, inability to obtain informed consent, or logistic reasons (e.g., unavailability of study personnel or medication) (Slieker et al., 2008).

The profound effect of enrollment criteria on outcome is perhaps not surprising, but has hitherto been insufficiently recognized. Enrollment criteria may be deliberately applied (as in RCTs), but may also inadvertently result from selective hospital admission policies in different institutions, or be caused by effects of local trauma organization, outside the control of investigators. A detailed understanding of such factors is, in our opinion, a prerequisite for comparing outcome results between series, originating from different settings or times.

It is remarkable that the application of common enrollment criteria to surveys resulted in an approximation of outcome distribution observed in RCTs.

Conclusion

Strict selection with common enrollment criteria leads to substantial recruitment reductions in TBI studies. The outcome distribution strongly depends on the enrollment criteria.

Acknowledgments

Grant support was provided by NIH NS-42691.

Author Disclosure Statement

No competing financial interests exist.

References

  • Bullock M.R. Merchant R.E. Choi S.C., et al. Outcome measures for clinical trials in neurotrauma. Neurosurg. Focus. 2002. www.aans.org/education/journal/neurosurgical/july02/13-1-nsf-toc.asp. [May 1;2009 ]. www.aans.org/education/journal/neurosurgical/july02/13-1-nsf-toc.asp [PubMed]
  • Choi S.C. Bullock R. Design and statistical issues in multicenter trials of severe head injury. Neurol. Res. 2001;23:190–192. [PubMed]
  • Edwards P. Arango M. Balica L., et al. Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroid in adults with head injury-outcomes at 6 months. Lancet. 2005;365:1957–1959. [PubMed]
  • Hernandez A.V. Steyerberg E.W. Butcher I., et al. Adjustment for strong predictors of outcome in traumatic brain injury trials: 25% reduction in sample size requirements in the IMPACT study. J. Neurotrauma. 2006;23:1295–1303. [PubMed]
  • Hukkelhoven C.W. Rampen A.J. Maas A.I., et al. Some prognostic models for traumatic brain injury were not valid. J. Clin. Epidemiol. 2006;59:132–143. [PubMed]
  • Hukkelhoven C.W. Steyerberg E.W. Habbema J.D., et al. Predicting outcome after traumatic brain injury: development and validation of a prognostic score based on admission characteristics. J. Neurotrauma. 2005;22:1025–1039. [PubMed]
  • Kompanje E.J. Maas A.I. Hilhorst M.T., et al. Ethical considerations on consent procedures for emergency research in severe and moderate traumatic brain injury. Acta Neurochir. (Wien) 2005;147:633–640. [PubMed]
  • Maas A.I. Marmarou A. Murray G.D., et al. Prognosis and clinical trial design in traumatic brain injury: the IMPACT study. J. Neurotrauma. 2007;24:232–238. [PubMed]
  • Maas A.I. Steyerberg E.W. Murray G.D., et al. Why have recent trials of neuroprotective agents in head injury failed to show convincing efficacy? A pragmatic analysis and theoretical considerations. Neurosurgery. 1999;44:1286–1298. [PubMed]
  • Machado S.G. Murray G.D. Teasdale G.M. Evaluation of designs for clinical trials of neuroprotective agents in head injury. European Brain Injury Consortium. J. Neurotrauma. 1999;16:1131–1138. [PubMed]
  • Marmarou A. Lu J. Butcher I., et al. IMPACT database of traumatic brain injury: design and description. J. Neurotrauma. 2007;24:239–250. [PubMed]
  • Marshall L.F. Maas A.I. Marshall S.B., et al. A multicenter trial on the efficacy of using tirilazad mesylate in cases of head injury. J. Neurosurg. 1998;89:519–525. [PubMed]
  • Narayan R.K. Michel M.E. Ansell C., et al. Clinical trials in head injury. J. Neurotrauma. 2002;19:503–557. [PMC free article] [PubMed]
  • Slieker F.J. Kompanje E.J. Murray G.D. Ohman J. Stocchetti N. Teasdale S.G. Maas A.I. SAPHIR and pharmos TBI investigators. Neurosurgery. 2008;62:1321–1328. [PubMed]

Articles from Journal of Neurotrauma are provided here courtesy of Mary Ann Liebert, Inc.