|Home | About | Journals | Submit | Contact Us | Français|
Design elements of clinical trials can introduce recruitment bias and reduce study efficiency. Trials involving the critically ill may be particularly prone to design-related inefficiencies. The VA/NIH Acute Renal Failure Trial Network (ATN) Study was designed to compare strategies of renal replacement therapy (RRT) in critically ill subjects with acute kidney injury (AKI).
Reasons for subject non-enrollment into the ATN Study were systematically monitored and categorized as modifiable or non-modifiable.
4339 subjects were screened; 2744 fulfilled inclusion criteria. Of these, 1034 were ineligible based on exclusion criteria. Of the remaining 1710 patients, 1124 (65.7%) enrolled. Impediments to informed consent excluded 21.4% of potentially eligible subjects; surrogate unavailability accounted for 1/3 of these exclusions. Delayed identification of potential subjects, physician refusal, and involvement in competing trials accounted for 4.4%, 2.7%, and 2.3% of exclusions. Comfort measures only (CMO) status, chronic illness, chronic kidney disease (CKD), and obesity excluded 11.8%, 7.8%, 7.6%, and 5.9% of potential subjects. Modification of an enrollment window reduced the loss of subjects from 6.6% to 2.3%.
The ATN Study’s enrollment efficiency compared favorably with previous ICU intervention trials and supports the representativeness of its enrolled population. Impediments to informed consent in the critically ill with AKI highlight the need for nontraditional acquisition methods. Restrictive enrollment windows may hamper subject recruitment, but can be effectively modified. The low rate of physician refusal acknowledges clinical equipoise in the study design. Underlying comorbidities are important design considerations for future trials involving the critically ill with AKI.
Elements in the design of randomized clinical trials (RCTs) can introduce systematic bias in subject recruitment and reduce efficiency of subject accrual. Because of the complexities of care of critically ill patients, where concurrent treatment trials are prevalent, rapid changes in patient census occur, informed consent limitations exist, and high mortality rates prevail, RCTs involving the critically ill may be particularly prone to design-related inefficiencies and bias in subject enrollment. Such bias can result in an exaggeration or diminution of treatment effects and ultimately confuse rather than inform clinical practice and healthcare policy (1).
Important information regarding the representativeness of clinical trial participants can be gleaned by evaluating characteristics of enrolled and non-enrolled subjects. Comparison of the two cohorts can provide a yardstick for gauging the external validity of a trial’s findings (2), although a detailed description of the clinical characteristics of non-enrolled subjects is often omitted or ignored in published reports (3,4).
The Consolidated Standards of Reporting Trials (CONSORT) statement calls for transparency in study design and improved quality of reporting (2, 5). Its recommendation to detail exclusion criteria allows for quantification of the effect of each criterion on subject enrollment. A cumulative assessment of reasons for exclusion yields valuable information about the trial’s generalizability.
In keeping with the CONSORT statement we detailed the process of subject selection for the VA/NIH Acute Renal Failure Trial Network (ATN) Study, a multi-center RCT comparing two strategies of renal replacement therapy (RRT) in critically ill subjects with acute kidney injury (AKI) (6). We quantified the study’s enrollment rate and reasons for exclusion using a screening log, which allowed us to assess the generalizability of the randomized subjects and to determine the impact of a change in exclusion criteria.
A detailed description of the ATN Study has been previously published (6). In brief, the ATN Study compares a less-intensive to a more intensive management strategy for renal replacement therapy (hemodialysis and hemodiafiltration) in critically ill patients with AKI. The primary study outcome is 60-day all-cause mortality. A cumulative screening log was compiled weekly by the data coordinating center using the data reported on study subject eligibility screening forms.
Potentially eligible subjects were critically ill patients who met inclusion criteria of 1) a clinical diagnosis of AKI secondary to acute tubular necrosis, 2) acute kidney injury [AKI: Δ serum creatinine (SCr) ≥ 2 mg/dL for men (≥ 1.5mg/dL for women) over a period of ≤ 4 days or oliguria > 24 hrs], 3) age ≥ 18 years, and 4) need for renal replacement therapy.
Potentially eligible subjects were excluded if they had chronic kidney disease (CKD; defined as baseline SCr > 2 mg/dL in men or 1.5 mg/dL in women); were kidney transplant recipients, incarcerated prisoners, pregnant, or were morbidly obese (weight >128.5 kg); were not candidates for support with RRT; were moribund or had care limited to comfort measures only (CMO); or were not expected to survive 28 days due to their underlying chronic medical illness. Physician refusal, concurrent participation in another interventional trial, delayed identification by study personnel (i.e., receipt of > 1 intermittent hemodialysis (IHD) session or > 24 hrs of continuous renal replacement therapy), or delayed start of initial renal replacement therapy beyond the enrollment window (initial window limit: blood urea nitrogen (BUN) > 60 mg/dL for ≥ 48 hrs; subsequently amended to BUN > 100 mg/dL for ≥ 72 hrs after meeting the definition of AKI) also resulted in exclusion.
Subjects meeting all inclusion and no exclusion criteria constituted the fully eligible cohort. Fully eligible study subjects were not enrolled if they or their proxies were unable or unwilling to provide informed consent. The enrollment rate was calculated as the ratio of enrolled and randomized subjects to fully eligible subjects. Potentially eligible subjects meeting all inclusion criteria were evaluated for reasons for non-enrollment. The more frequent reasons for non-enrollment were broadly categorized as either modifiable or non-modifiable; the percent of potentially eligible subjects excluded for each reason was determined.
In 44 months, 4339 subjects were screened, 2744 of whom satisfied all inclusion criteria (Figure 1). Of these potentially eligible subjects, 1034 (37.7%) were ineligible due to the presence of one or more exclusion criteria. Of the remaining 1710 fully eligible subjects (met all inclusion and no exclusion criteria), 1124 were enrolled, representing 25.9% of all subjects screened, 41.0% of potentially eligible subjects, and 65.7% of fully eligible subjects, respectively. The reasons for rejection of potentially eligible subjects are shown in Figure 1 and Table 1.
Limitations in acquiring informed consent were the most common barrier to enrollment. Five hundred eighty-six subjects were excluded due to absence of informed consent, representing 21.4% of potentially eligible subjects and 34.3% of the fully eligible cohort. Since fewer than 10% of subjects had decision-making capacity as the result of their acute illness, surrogate consent was required for the majority of enrolled subjects. Non-availability of a surrogate to provide informed consent excluded 193 patients (7.0% of potentially eligible subjects and 11.2% of all fully eligible subjects), and constituted nearly one third (32.9%) of subjects not enrolled due to absence of informed consent. Two hundred ninety-eight potentially eligible subjects (or their surrogates) declined to participate in the study (10.9% of potentially eligible subjects and 17.4% of the fully eligible cohort).
An eligibility window limiting the time-frame for subject recruitment was identified as a small but significant barrier to enrollment. Modification of the window after nine months of subject accrual reduced its negative impact on subject recruitment; decreasing the loss of potentially eligible from 6.6% to 2.3%.
Delayed identification of potential study subjects was responsible for 4.4% of missed enrollments in the potentially eligible cohort. Non-investigator physician refusal to permit participation of the subject in the trial was unusual, accounting for the exclusion of only 2.7% of the potentially eligible cohort. Similarly involvement in competing trials was an uncommon barrier to enrollment, excluding only 2.3% of potentially eligible subjects.
Non-modifiable exclusion criteria, including CMO status, chronic underlying medical illness, CKD, and morbid obesity, precluded enrollment of notable fractions of potentially eligible subjects (11.8%, 7.8%, 7.6%, and 5.9%, respectively). Subjects excluded for obesity had a mean body weight at screening of 154.3 ± 28.1 kg. The mean SCr of those excluded due to the presence of CKD was 2.9 ± 1.9 mg/dL.
Difficulty in the interpretation of RCT results due to obscurity in trial design has been raised as a significant concern in the methodologic literature of clinical research (7). The initial CONSORT statement called for improved transparency in study design (5) and even more explicit criteria for describing patient recruitment and reasons for exclusion were specified in the CONSORT II revision in order to further improve the readers’ understanding of a trial’s subject population (1). Closer scrutiny of patient recruitment and enrollment efficiency in clinical trials has also been encouraged for practical and ethical reasons since these data can be informative regarding feasibility of ongoing trials, timeliness of study completion, minimization of trial costs, and optimal use of limited resources (8).
In most trials, a marked surplus of potential subjects are screened for eligibility to offset impediments to enrollment (9). In the past decade recruitment efficiencies (defined as the percent of eligible subjects enrolled) for large, interventional critical care RCTs conducted have most commonly been reported to be <20% (10–14) with a few notable exceptions (15, 16). In contrast, the ATN study demonstrated excellent enrollment efficiency, recruiting 41% of potentially eligible subjects, and randomizing 66% of eligible subjects. A relatively high enrollment efficiency is not only beneficial from the standpoint of research costs, but also suggests greater generalizability of a trial’s results.
Gauging the true efficiency of reported trials has been difficult due to imprecision or absence of reporting of the trial enrollment process (3, 7). Adoption of a common lexicon and a requirement for quantifying not only enrolled subjects, but also those screened, determined potentially eligible and then truly eligible, would facilitate transparency of trial design, calculation of trial efficiency, and enhance interpretation of study results, particularly with regard to their generalizability.
Use of the CONSORT reporting criteria has highlighted the issue of subject exclusions as a barrier to study enrollment. A recent analysis of two series of large clinical oncology trials in the post-CONSORT era revealed that not only had the number of exclusion criteria nearly doubled over the past two decades, but this increase had occurred without a clear rationale, and was largely due to perpetuation of exclusion criteria in sequential trials (11). Consequently, exclusion criteria considered for the ATN study were carefully developed and closely scrutinized for their adverse impact on subject enrollment.
We used a screening log, as proposed by others, to pinpoint criteria that were highly leveraged with respect to subject accrual and evaluated corrective strategies to optimize enrollment (11, 17). Of the criteria adversely affecting enrollment efficiency, the nine most frequent were examined and broadly categorized as either modifiable or non-modifiable barriers to enrollment.
Limitations in acquiring informed consent were the most significant modifiable barriers to enrollment, contributing to the exclusion of 21.4% of potentially eligible subjects. One third of these non-enrolled subjects were excluded because the subject lacked decision-making capacity and the appropriate surrogate decision maker was not available to provide consent. These results are similar to observations in other critical care trials. In the ARDS Network’s FACTT Study, 10% of potentially eligible subjects were precluded from enrollment due to lack of consent, with 58% of these due to non-availability of the subject’s proxy (18). Similarly, in the observational, multi-center PICARD Study, absence of a proxy to provide consent accounted for up to 40% of excluded subjects, documenting that such obstacles to recruitment are not unique to RCTs.(19)
The validity of written informed consent in critical care has been debated, especially when obtained via proxy (20, 21); however signature documentation of informed consent is a regulatory requirement in the United States and the majority of other countries (22). Sufficient lead time to reasonably seek written informed consent from the patient or a proxy is therefore a prudent design consideration in future trials involving the critically ill to avoid the unnecessary exclusion of otherwise eligible subjects. In addition, the incorporation of alternative strategies for documentation of signature for informed consent, such as electronic transmittal, can facilitate the ability to enroll subjects with acutely impaired decision-making capacity by surrogates who are unable to be present at the study site during limited windows of eligibility.
The design of the ATN Study specified an “enrollment window” in an attempt to achieve a degree of uniformity in the timing of initiation of RRT in enrolled subjects. Periodic audits of the ATN Study screening log highlighted the negative impact that this enrollment window was having on subject accrual. It became readily apparent that there was no consensus in practice outside of the study setting regarding the indications for initiation of RRT. Patients were excluded both because of failure to fulfill the enrollment window’s criteria for initiation of renal support (early initiation) and because of failure to initiate therapy within 48 hours of meeting the study definition of ARF and reaching a BUN of > 60 mg/dL (late initiation). Nearly 7% of potentially eligible subjects in the trial’s first 10 months were excluded as a result of this enrollment window. The mean BUN at the initiation of RRT in these excluded subjects was 71 mg/dL, with 25% having a BUN > 103 mg/dL. Revision of the enrollment criteria, eliminating the “window” but excluding patients in whom the BUN was > 100 mg/dL for more than 72 hours after meeting the study definition of AKI removed an unintended barrier to subject recruitment while increasing the generalizability of the study population. After implementation of this change, fewer than 3% of potentially eligible subjects were excluded due to prolonged untreated azotemia.
Enrollment windows are criteria of precision and account for the greatest increase in eligibility criteria reported in the past several decades (8). A recent meta-analysis of acute stroke trials highlighted the importance of the enrollment window as a potential barrier to enrollment; a stringent window was the primary criterion that predicted reduced recruitment efficiency (23). While the merits of a precise enrollment window may be debated, at a minimum a clear biological basisshould be evident before invoking a restrictive enrollment window as an exclusion criterion.
Although elimination of the enrollment window was beneficial from the standpoint of subject recruitment, changing eligibility criteria during the conduct of a study does introduce the possibility of bias and may necessitate modification of the study analytic plan. In this case, the elimination of the enrollment window occurred relatively early in the course of the study, following enrollment of less than 20% of the overall study population. Since enrollment by treatment arm was well balanced both prior to and after this change was made, it is unlikely to have introduced bias as a result of imbalance in the application of the change across the two treatment arms.
Delayed identification of potentially eligible subjects by study personnel due to late notification by treating physicians excluded 4.4% of potentially eligible subjects. Privacy constraints under the Health Insurance Portability and Accountability Act of 1996 (HIPAA) can limit identification of potential study subjects and restrict screening of potentially eligible subjects by providers not directly participating in the potential subject’s care without a waiver of authorization. Late notification of study personnel increases the time constraints associated with obtaining informed consent and can preclude the opportunity to present a study to patients or their families while naïve to therapeutic interventions. It also leads to disproportionate attrition of subjects with the greatest mortality, and can result in a “death before consent” bias (19).
Non-investigator physician refusal was an infrequent obstacle to enrollment, accounting for non-enrollment of only 2.7% of potentially eligible subjects. In contrast, physician refusal was the second most common cause of non-enrollment of subjects in two recent critical care trials, accounting for 11% and 16% of non-enrolled subjects, respectively (12, 13), while in a trial of catheter management in the critically ill, treating physician refusal accounted for 50% of subject non-enrollment (15). Since treating physician “buy-in” was expected to be an important determinant of the ATN Study success, multiple strategies reported to foster protocol acceptance by non-investigator practitioners were employed at study sites, including involvement of multiple disciplines in the study team, repeated educational sessions for ICU staff, provision of research information, stakeholder involvement in protocol development, prospective protocol agreement from local clinicians, simplicity in study design, and ease in protocol implementation (11, 17). In addition, our low physician refusal rate supports the premise of widespread acknowledgement of clinical equipoise in the study’s design by non-investigator physicians.
Involvement in competing trials was an uncommon barrier to enrollment. Only a small fraction (2.3 %) of potentially eligible ATN subjects were excluded due to enrollment in concurrent interventional trials. In contrast, other investigators have reported that participation in competing studies excluded 13% of potential subjects (423/3245) (12). This markedly lower rate may reflect the fact that AKI commonly excludes subjects from other studies conducted in the ICU setting.
Various comorbidities constituted the non-modifiable barriers to enrollment. As the most extreme example of comorbidity, moribund patients and/or patients classified as CMO status accounted for 11.8% of potentially eligible subject exclusions, although co-selection of these criteria in screening may have yielded an over estimate of their effects on enrollment. Exclusion for hopelessness in critical care trials has been reported to account for 4 to 31% of subjects (12, 13, 16). Exclusion due to hopelessness may not only affect available subject pools, but may introduce survivorship bias and affect expected event rates and mortality, making this an important consideration in trial design (19). In designing the ATN Study, exclusion of these patients was considered appropriate as dialysis is futile as a life-preserving therapy in these settings.
Underlying life-limiting chronic illness excluded 7.8% of potentially eligible subjects. The rationale for this exclusion criterion is that management of RRT would have no impact on 60-day mortality, the primary study endpoint. This rate of study exclusion due to underlying chronic illness is similar to the rate of 10.6% of screened subjects reported in another recent critical care study (13). While it may be argued that exclusion of these patients reduces the generalizability of study results, the inclusion of such subjects could dilute the intervention’s effect and inflate the number of subjects required to detect a significant difference between groups, thereby decreasing study feasibility (24).
Nearly 8% of potentially eligible subjects were excluded from the ATN Study because of preexisting moderate to severe CKD (defined as a baseline serum creatinine > 2 mg/dL in men and > 1.5 mg/dL in women). While data from the PICARD study suggest that acute on chronic disease may be common, representing up to 30% of hospitalized patients with AKI (25), the exclusion of subjects with CKD from the ATN Study was deemed appropriate given differences in the natural history of acute on chronic kidney disease as compared to de novo AKI, with both lower mortality risk and lower probability of recovery of renal function in acute on chronic disease (25). Although exclusion of patients with moderate to advanced CKD may decrease the generalizability of the study’s findings, only 208 of the 2744 potentially eligible subjects (7.6%) were excluded on this basis.
Morbid obesity excluded 5.9% of potentially eligible subjects. Since the stipulated dosing of continuous venovenous hemodiafiltration (CVVHDF) was indexed to body mass, and the maximum combined flow-rate for dialysate and replacement fluid provided by the CRRT equipment most commonly utilized at participating study sites at the initiation of the study was limited to 4.5 liters per hour, the delivery of an effluent flow of 35 mL/kg per hour, as required in the intensive arm, was precluded in subjects with morbid obesity. Thus, the exclusion of patients weighing more than 128.5 kg was necessitated by the constraints of the existing medical technology at the study’s initiation.
In summary, enrollment efficiency in the ATN Study compared favorably with other intervention trials previously reported in the ICU and supports the representativeness of the study’s enrolled population. Periodic audits of a screening log permitted early identification of important barriers to enrollment and provided a gauge of the effectiveness of criteria modification on subsequent subject accrual. Impediments to acquiring informed consent were the major modifiable barriers to subject recruitment, and highlight the need for nontraditional methods of obtaining and documenting informed consent in future critical care trials.
The enrollment window was identified as an unintended barrier to subject recruitment and was readily modified to facilitate subject accrual. Excellent study acceptance or “buy-in” by non-investigator colleagues was evidenced by the low rate of non-investigator physician refusal and supports the premise of widespread acknowledgement of clinical equipoise in the study design. Underlying comorbidities were the basis for most non-modifiable barriers to enrollment and should be serious design considerations in future trials involving the critically ill.
The ATN Study is supported by the Cooperative Studies Program of the Department of Veterans Affairs Office of Research and Development and by the National Institute of Diabetes, Digestive and Kidney Diseases by interagency agreement Y1-DK-3508-01.
This research was presented at the 39th Annual Meeting and Scientific Exposition of the American Society of Nephrology in San Diego, CA, November 14–19, 2006.
Paul M. Palevsky, MD, Chairman, VA Pittsburgh Healthcare System, Pittsburgh, PA
Theresa Z. O’Connor, PhD, CSP Coordinating Center, VA Connecticut Healthcare System, West Haven, CT
Jane H. Zhang, PhD, CSP Coordinating Center, VA Connecticut Healthcare System, West Haven, CT
Glenn M. Chertow, MD, MPH, University of California, San Francisco, San Francisco, CA
Susan T. Crowley, MD, VA Connecticut Healthcare System, West Haven, CT
Devasmita Choudhury, MD, VA North Texas Healthcare System, Dallas, TX
John Kellum, MD, University of Pittsburgh, Pittsburgh, PA
Emil Paganini, MD, Cleveland Clinic Foundation, Cleveland, OH
Roland M.H. Schein, MD, Miami VA Medical Center, Miami, FL
B. Taylor Thompson, MD, Massachusetts General Hospital, Boston, MA
Mark W Smith, HERC, VA Palo Alto Healthcare System, Menlo Park, CA
Kathy Swanson, RPh, CSP Research Pharmacy Coordinating Center, New Mexico VA Healthcare System, Albuquerque, NM
Peter Peduzzi, PhD, (Ex Officio), Director VA CSP Coordinating Center, VA Connecticut Healthcare System, West Haven, CT
Robert Star, MD (Ex Officio), Senior Advisor, NIDDK, Bethesda, MD
VA Ann Arbor Healthcare System, Ann Arbor, MI (Eric Young, MD)
VA Western NY Healthcare System, Buffalo, NY (James Lohr, MD)
VA North Texas Healthcare System, Dallas, TX (Devasmita Choudhury, MD)
Houston VA Medical Center, Houston, TX (George Dolson, MD)
Richard L Roudebush VA Medical Center, Indianapolis, IN (Robert Bacallao, MD)
Central Arkansas Veterans Healthcare Center, Little Rock, AK (Mary Jo Shaver, MD)
West Los Angeles VA Healthcare Center, Los Angeles, CA (Jeffrey Kraut, MD)
Miami VA Medical Center, Miami, FL (Roland M. H. Schein, MD)
VA Tennessee Valley Healthcare System, Nashville, TN (T. Alp Ikizler, MD)
New Orleans VA Medical Center, New Orleans, LA (Vecihi Batuman, MD)
VA Pittsburgh Healthcare System, Pittsburgh, PA (Mohan Ramkumar, MD)
Portland VA Medical Center, Portland, OR (Suzanne Watnick, MD)
Hunter Holmes McGuire VA Medical Center, Richmond, VA (George Feldman, MD)
VA San Diego Healthcare System, San Diego, CA (Francis Gabbai, MD)
San Francisco VA Medical Center, San Francisco, CA (Kirsten Johansen, MD)
San Juan VA Medical Center, San Juan, PR (Carlos Rosado-Rodriguez, MD)
VA Puget Sound Healthcare System, Seattle, WA (Dennis Andress, MD)
VA Connecticut Healthcare System, West Haven, CT (Susan T. Crowley, MD)
Cleveland Clinic Foundation, Cleveland, OH (Emil Paganini, MD)
Johns Hopkins University (Hamid Rabb, MD)
Massachusetts General Hospital, Boston, MA (John Niles, MD)
University of California, San Francisco, San Francisco, CA (Glenn Chertow, MD)
University of Miami, Miami, FL (Gabriel Contreras, MD, MPH)
University of Pittsburgh, Pittsburgh, PA (Nabeel Aslam, MD)
University of Texas at Houston, Houston, TX (Kevin Finkel, MD, Andrew Shaw, MD)
Wake Forest University, Winston-Salem, NC (Michael Rocco, MD)
Washington University at St. Louis, St. Louis, MO (Anitha Vijayan)