Search tips
Search criteria 


Logo of jnciLink to Publisher's site
J Natl Cancer Inst. 2013 March 20; 105(6): 387–393.
Published online 2013 March 5. doi:  10.1093/jnci/djt001
PMCID: PMC3601950

Radiotherapy Protocol Deviations and Clinical Outcomes: A Meta-analysis of Cooperative Group Clinical Trials



Noncompliance with radiotherapy (RT) protocol guidelines has been linked to inferior clinical outcomes. We performed a meta-analysis of cooperative group trials to examine the association between RT quality assurance (QA) deviations and disease control and overall survival (OS).


We searched MEDLINE and the Cochrane Central Register of Controlled Trials for multi-institutional trials that reported clinical outcomes in relation to RT QA results. Hazard ratios (HRs) describing the association between RT protocol noncompliance and patient outcomes were extracted directly from the original studies or calculated from survival curves. Inverse variance meta-analyses were performed to assess the association between RT QA deviations and OS. A second meta-analysis tested the association between RT QA deviations and secondary outcomes, including local or locoregional control, event-free survival, and relapse. Random-effects models were used in cases of statistically significant (P < .10) effect heterogeneity. The Egger test was used to detect publication bias. All statistical tests were two-sided.


Eight studies (four pediatric, four adult) met all inclusion criteria and were incorporated into this analysis. The frequency of RT QA deviations ranged from 8% to 71% (median = 32%). In a random-effects model, RT deviations were associated with a statistically significant decrease in OS (HR of death = 1.74, 95% confidence interval [CI] = 1.28 to 2.35; P < .001). A similar effect was seen for secondary outcomes (HR of treatment failure = 1.79, 95% CI = 1.15 to 2.78; P = .009). No evidence of publication bias was detected.


In clinical trials, RT protocol deviations are associated with increased risks of treatment failure and overall mortality.

Quality improvement is an essential process in the field of medicine. On an individual level, adherence to quality assurance (QA) measures can help ensure the delivery of safe and effective care for each patient. Quality control is also critical for measuring the effectiveness of medical advances: QA measures are used to standardize therapy in clinical trials and ensure that the intended study questions are addressed (1).

In the field of radiation oncology, the reliable delivery of high-quality therapy and verification that such treatment has been delivered poses unique challenges (2,3). Cooperative research groups have progressively incorporated QA procedures into clinical radiotherapy (RT) protocols over the past 40 years (4). Some modern trials, for example, require that RT plans pass centralized review before study therapy begins. These measures have been shown to minimize treatment deviations (5,6) and may increase the likelihood that the study question can be answered (7). Although it seems intuitive that rigorous RT QA would improve patient outcomes, the benefits of such measures have not been established (8).

Several recent secondary analyses of clinical trials have demonstrated that RT QA deviations may be independent predictors of poor outcomes (9,10). We performed a meta-analysis of cooperative group trials to examine the association between RT QA deviations and clinical outcomes and estimate the magnitude of these effects.


Selection of Studies

We searched MEDLINE citations on February 7, 2012, for the terms “radiotherapy,” “quality assurance,” and “survival.” We did not apply any filter for publication date. We also searched the Cochrane Central Register of Controlled Trials for the terms “radiotherapy” and “quality assurance.” Publications that described clinical outcomes in multi-institutional clinical trials in relation to RT QA results were included in this analysis. Articles that detailed QA techniques but did not relate QA results to time-to-event outcome data were excluded. When more than one publication was identified from the same clinical trial, we used the most recent or most complete report of that trial.

Data Extraction and Clinical Endpoints

Data abstraction was conducted by the lead investigator (N. Ohri) according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement (11). For each study, we extracted the following information: name of the first author, year of publication, name of the clinical trial, study question, number of enrolled patients, number of patients included in QA analysis, definition of RT protocol deviation, and numbers of patients with and without RT deviations. The primary clinical outcome of interest for this analysis was overall survival (OS). Data for secondary outcomes, such as local control or event-free survival, were also recorded.

Hazard ratios (HRs) for time-to-event data (eg, OS, event-free survival) were extracted directly from the original studies or were estimated indirectly from the survival curves, as suggested by Parmar et al. (12). Data extraction from survival curves and estimations of effect sizes were performed using customized scripts in Matlab version 7.13 (The Mathworks, Natick, MA). The 95% confidence interval (CI) for each hazard ratio was extracted directly from the original report if available or estimated as described by Parmar et al. (12). All calculations were performed using study-level data because individual patient data were not available for this analysis. Because only two studies reported multivariable analyses examining the association between RT deviations and clinical outcomes, we used unadjusted hazard ratios.

Statistical Analysis

Meta-analyses were performed using the inverse variance method. Separate analyses were performed for OS and for the secondary outcomes, which were grouped together. For each meta-analysis, we calculated Cochran’s Q, which is a classical measure of heterogeneity of effect sizes across trials (13). The assumption of homogeneity was considered invalid for P values less than .10 (a conservative cutoff commonly used in meta-analyses). This prompted the use of the random-effects model to derive summary estimates for hazard ratios and 95% confidence intervals. Publication bias was evaluated visually with funnel plots and statistically as described by Egger et al. (14). A two-tailed P value of less than .10 was considered statistically significant.


Selection of Trials

Our initial searches yielded 600 results (523 from MEDLINE and 77 from the Cochrane Central Register of Controlled Trials). A total of 59 publications were found in both searches; thus there were 541 unique publications. After reviewing each abstract, 12 candidates for meeting our eligibility criteria were identified. After reading those 12 full-text articles, one paper (15) was excluded because updated results of the same trial were reported in another publication (16). A secondary analysis of data from five clinical trials was excluded because it did not report the impact of RT compliance on clinical outcomes (17). A third study was excluded because it only reported QA results for patients who experienced disease recurrence (18). A fourth was excluded because it did not report the association between RT QA results and clinical outcomes using time-to-event data (19). Trial selection results are depicted in Figure 1.

Figure 1.
Trial selection.

Eight studies met all inclusion criteria and were incorporated into this analysis (Table 1). These included two lung cancer trials (20,21), three trials for medulloblastoma or supratentorial primitive neuroectodermal tumor (16,22,23), and one trial each for Ewing sarcoma (24), pancreatic cancer (9), and head and neck cancer (10). In all but one trial (10), RT QA reviews were performed in a post hoc fashion. Six studies (9,10,16,20,21,23) reported the influence of RT QA deviations on OS. Six studies (9,10, 16,22–24) reported the effects of RT QA deviations on secondary endpoints, including local or locoregional control (10,24), event-free survival (16,23), and relapse (9,22). Various definitions of RT deviations were employed by the included studies and are summarized in Table 1. The frequency of RT deviations ranged from 8% to 71% (median = 32%).

Table 1.
Summary of trials included in the meta-analysis*

RT Deviations and OS

Meta-analysis results describing the association between RT deviations and OS are displayed in Figure 2. The Cochran Q statistic using a fixed-effects model was 19.7 (P = .001), prompting the use of a random-effects model. Using the random-effects model, RT deviations were associated with a statistically significant decrease in OS (HR = 1.74, 95% CI = 1.28 to 2.35). No evidence of publication bias was detected using the Egger test (P = .36).

Figure 2.
Forest plot of hazard ratios (HRs) describing the association between radiotherapy protocol deviations and overall survival. Hazard ratios for each trial are represented by the squares, the size of the square represents the weight of the trial in the ...

RT Deviations and Secondary Outcomes

Figure 3 depicts the association between RT deviations and secondary outcomes. When a fixed-effects model was used, the Cochran Q statistic was 16.8 (P = .005). We consequently performed a random-effects meta-analysis (Figure 2). Using that model, RT deviations were associated with statistically significantly worse secondary outcomes (HR = 1.79, 95% CI = 1.15 to 2.78). There was no evidence of publication bias using the Egger test (P = .47).

Figure 3.
Forest plot of hazard ratios (HRs) describing the association between radiotherapy protocol deviations and secondary outcomes. Hazard ratios for each trial are represented by the squares, the size of the square represents the weight of the trial in the ...


This analysis of published reports relating RT protocol deviations to patient outcomes in cooperative group clinical trials revealed that in the majority of cases, failure to meet RT QA measures was associated with inferior outcomes. Our meta-analysis demonstrated that RT protocol deviations occurred in 8% to 71% of cases and were associated with an approximately 75% increase in the risk of treatment failure and overall mortality. The magnitude of these effect sizes suggests that the delivery of high-quality RT is critical for the successful execution of clinical trials and for effective treatment of cancer patients.

Adherence to RT protocol guidelines is an important practical concern in the conduct of clinical trials because RT quality may influence the interpretability of study results. For example, many clinicians believe that identification of the optimal treatment approach following surgery for pancreatic cancer has been hindered by poor RT quality, outdated RT schedules, and a lack of RT quality control in landmark randomized trials (25). Data from Radiation Therapy Oncology Group (RTOG) 97-04 (a study included in this analysis) link adherence to RT guidelines in the adjuvant treatment of pancreatic cancer with improved disease control and OS (9). This finding is particularly striking because it has been difficult to prove that adjuvant RT itself is beneficial in the adjuvant setting.

The importance of incorporating QA measures into multi-institutional trials has been recognized for some time. Findings from a 2010 National Cancer Institute conference entitled “Methods and Issues for Redesigning Clinical Trial Quality Assurance in Radiotherapy” were recently published (8). Conference participants made four recommendations: 1) develop a tiered system and tailor intensity of QA to clinical trial objectives; 2) establish a case QA repository; 3) develop an evidence base for clinical trial QA; and 4) explore the feasibility of consolidating clinical trial QA in the United States. We agree with these recommendations, and we believe that our findings support the practice of incorporating real-time QA measures into clinical RT trials. As more advanced RT technologies are incorporated into clinical trials, the resources required to maintain a rigorous QA program will increase substantially (26). Consequently, judicious implementation of QA measures that are both effective and practical poses a challenge to current and future trialists.

Our findings may have implications outside the clinical trials arena. The effect sizes reported in this study surpass those typically seen in successful trials of novel treatment strategies. Although we acknowledge that the association between RT QA results and clinical outcomes may not be entirely causal, our findings suggest that the implementation of rigorous QA procedures may provide clinically significant benefits for all RT patients, including the vast majority of cancer patients who are not enrolled in clinical trials. This may be a worthy subject for future prospective study.

The definition, frequency, and clinical significance of RT QA deviations may all vary with treatment technique. The vast majority of the studies included in this analysis utilized two-dimensional RT. Three-dimensional RT planning was allowed in the most recent studies (9,10), but none included patients treated with intensity-modulated RT. Although the applicability of our findings to newer RT techniques is unclear, we believe that rigorous RT QA becomes even more critical as treatment complexity increases.

The delivery of RT is a complicated, multistep process. This analysis, however, focused solely on deviations that occurred during treatment planning. In the studies we examined, RT deviations were scored based on treatment fields and/or RT plans that were generated before the initiation of therapy. Subsequent changes in patient set-up or modifications to the treatment delivery were not accounted for. This practice is consistent with the QA procedures typically employed in modern RT trials. In certain situations, specialized imaging tools (27) or implantable dosimeters (28) may improve our ability to verify that the intended RT plan is actually delivered.

This study has several limitations that should be considered when interpreting our results. The QA data incorporated into this meta-analysis were generally collected retrospectively and were usually available for only a subset of patients from each study. Multivariable analyses examining the statistical significance of RT deviations along with other prognostic factors were reported in only a few papers. The definitions of RT deviations varied across studies, and bias could clearly have led authors to choose definitions that were statistically significantly correlated with outcomes. In addition, publication bias may have influenced our findings; “negative” studies that failed to link RT protocol noncompliance with inferior outcomes may have been performed within cooperative groups but never published. Although we did not find any statistical evidence of publication bias, our sensitivity to detect such an effect was limited by our sample size. Finally, the strong association between RT deviations and clinical outcomes may not truly represent causation; digression from protocol guidelines may be related to unfavorable features within the patient (eg, poor health or large tumor size) or the treatment team (eg, lack of experience).

In conclusion, in clinical trials, RT protocol deviations are associated with increased risk of treatment failure and overall mortality. The magnitude of these effect sizes indicates that rigorous RT QA is an important component of the clinical trial process. Additionally, these findings raise the possibility that strict adherence to RT planning guidelines might benefit all cancer patients who are treated with RT.


This research was supported in part by a grant from the National Cancer Institute Cancer Center Support (P30 CA56036).


The study sponsor did not play any role in the design of the study, the collection, analysis, or interpretation of the data, the writing of the study, or the decision to submit the study for publication. None of the authors have any conflicts of interest that are relevant to this publication.


1. Lomas J. Quality assurance and effectiveness in health care: an overview. Qual Assur Health Care. 1990; 2(1):5–12 [PubMed]
2. Williamson JF, Dunscombe PB, Sharpe MB, Thomadsen BR, Purdy JA, Deye JA, et al. Quality assurance needs for modern image-based radiotherapy: recommendations from 2007 interorganizational symposium on “quality assurance of radiation therapy: challenges of advanced technology” Int J Radiat Oncol Biol Phys. 2008; 71(1 Suppl):S2–12 [PubMed]
3. Crozier C, Erickson-Wittmann B, Movsas B, Khalid N, Wilson JF., et al. Shifting the focus to practice quality improvement in radiation oncology. J Healthc Qual. 2011;33(5)49–57 [PubMed]
4. Pawlicki T, Dunscombe P, Mundt AJ. Quality Assurance and Safety for Radiotherapy. Taylor & Francis Group; FL: Boca Raton; 2010.
5. Bentzen SM, Bernier J, Davis JB, et al. Clinical impact of dosimetry quality assurance programmes assessed by radiobiological modelling of data from the thermoluminescent dosimetry study of the European Organization for Research and Treatment of Cancer. Eur J Cancer. 2000; 36(5):615–620 [PubMed]
6. Ibbott GS, Hanson WF, O’Meara E, et al. Dose specification and quality assurance of radiation therapy oncology group protocol 95-17; a cooperative group study of iridium-192 breast implants as sole therapy. Int J Radiat Oncol Biol Phys. 2007; 69(5):1572–1578 [PMC free article] [PubMed]
7. Pettersen MN, Aird E, Olsen DR. Quality assurance of dosimetry and the impact on sample size in randomized clinical trials. Radiother Oncol. 2008; 86(2):195–199 [PubMed]
8. Bekelman JE, Deye JA, Vikram B, et al. Redesigning radiotherapy quality assurance: opportunities to develop an efficient, evidence-based system to support clinical trials—report of the National Cancer Institute Work Group on Radiotherapy Quality Assurance. Int J Radiat Oncol Biol Phys. 2012; 83(3):782–790 [PMC free article] [PubMed]
9. Abrams RA, Winter KA, Regine WF, et al. Failure to adhere to protocol specified radiation therapy guidelines was associated with decreased survival in RTOG 9704-A phase III trial of adjuvant chemotherapy and chemoradiotherapy for patients with resected adenocarcinoma of the pancreas. Int J Radiat Oncol Biol Phys. 2012; 82(2):809–816 [PMC free article] [PubMed]
10. Peters LJ, O’Sullivan B, Giralt J, et al. Critical impact of radiotherapy protocol compliance and quality in the treatment of advanced head and neck cancer: results from TROG 02.02. J Clin Oncol. 2010; 28(18):2996–3001 [PubMed]
11. Moher D, Liberati A, Tetzlaff J, Altman DG.PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009; 339: b2535 [PubMed]
12. Parmar MK, Torri V, Stewart L. Extracting summary statistics to perform meta-analyses of the published literature for survival endpoints. Stat Med. 1998; 17(24):2815–2834 [PubMed]
13. Cochran WG. The combination of estimates from different experiments. Biometrics. 1954; 10(1):101–129
14. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997; 315(7109):629–634 [PMC free article] [PubMed]
15. Taylor RE, Bailey CC, Robinson KJ, et al. Impact of radiotherapy parameters on outcome in the International Society of Paediatric Oncology/United Kingdom Children’s Cancer Study Group PNET-3 study of preradiotherapy chemotherapy for M0-M1 medulloblastoma. Int J Radiat Oncol Biol Phys. 2004; 58(4):1184–1193 [PubMed]
16. Taylor RE, Donachie PH, Weston CL, et al. Impact of radiotherapy parameters on outcome for patients with supratentorial primitive neuro-ectodermal tumours entered into the SIOP/UKCCSG PNET 3 study. Radiother Oncol. 2009; 92(1):83–88 [PubMed]
17. Khalil AA, Bentzen SM, Bernier J, et al. Compliance to the prescribed dose and overall treatment time in five randomized clinical trials of altered fractionation in radiotherapy for head-and-neck carcinomas. Int J Radiat Oncol Biol Phys. 2003; 55(3):568–575 [PubMed]
18. Million L, Anderson J, Breneman J, et al. Influence of noncompliance with radiation therapy protocol guidelines and operative bed recurrences for children with rhabdomyosarcoma and microscopic residual disease: a report from the Children’s Oncology Group. Int J Radiat Oncol Biol Phys. 2011; 80(2): 333–338 [PMC free article] [PubMed]
19. Eisbruch A, Harris J, Garden AS, et al. Multi-institutional trial of accelerated hypofractionated intensity-modulated radiation therapy for early-stage oropharyngeal cancer (RTOG 00-22). Int J Radiat Oncol Biol Phys. 2010; 76(5):1333–1338 [PMC free article] [PubMed]
20. Perez CA, Stanley K, Grundy G, et al. Impact of irradiation technique and tumor extent in tumor control and survival of patients with unresectable non-oat cell carcinoma of the lung: report by the Radiation Therapy Oncology Group. Cancer. 1982; 50(6):1091–1099 [PubMed]
21. White JE, Chen T, McCracken J, et al. The influence of radiation therapy quality control on survival, response and sites of relapse in oat cell carcinoma of the lung: preliminary report of a Southwest Oncology Group study. Cancer. 1982; 50(6):1084–1090 [PubMed]
22. Carrie C, Hoffstetter S, Gomez F, et al. Impact of targeting deviations on outcome in medulloblastoma: study of the French Society of Pediatric Oncology (SFOP). Int J Radiat Oncol Biol Phys. 1999; 45(2):435–439 [PubMed]
23. Miralbell R, Fitzgerald TJ, Laurie F, et al. Radiotherapy in pediatric medulloblastoma: quality assessment of Pediatric Oncology Group Trial 9031. Int J Radiat Oncol Biol Phys. 2006; 64(5):1325–1330 [PubMed]
24. Donaldson SS, Torrey M, Link MP, et al. A multidisciplinary study investigating radiotherapy in Ewing’s sarcoma: end results of POG #8346. Pediatric Oncology Group. Int J Radiat Oncol Biol Phys. 1998; 42(1): 125–135 [PubMed]
25. Picozzi VJ, Pisters PW, Vickers SM, Strasberg SM, et al. Strength of the evidence: adjuvant therapy for resected pancreatic cancer. J Gastrointest Surg. 2008; 12(4):657–661 [PubMed]
26. Weber DC, Poortmans PM, Hurkmans CW, Aird E, Gulyban A, Fairchild A, et al. Quality assurance for prospective EORTC radiation oncology trials: the challenges of advanced technology in a multicenter international setting. Radiother Oncol. 2011; 100(1):150–156 [PubMed]
27. Krejcarek SC, Grant PE, Henson JW, Tarbell NJ, Yock TI, et al. Physiologic and radiographic evidence of the distal edge of the proton beam in craniospinal irradiation. Int J Radiat Oncol Biol Phys. 2007; 68(3): 646–649 [PMC free article] [PubMed]
28. Den RB, Nowak K, Buzurovic I, et al. Implanted dosimeters identify radiation overdoses during IMRT for prostate cancer. Int J Radiat Oncol Biol Phys. 2012; 83(3):e371–e376 [PubMed]

Articles from JNCI Journal of the National Cancer Institute are provided here courtesy of Oxford University Press