PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of qualsafetyQuality and Safety in Health CareCurrent TOCInstructions for authors
 
Qual Saf Health Care. 2007 June; 16(3): 203–207.
PMCID: PMC2464981

Quality control of surgical and interventional procedures: a review of the CUSUM

Abstract

Background

The report of the CUSUM across surgical and interventional procedures has spawned a fair confusion in the literature.

Aim

To assess the use of the CUSUM and to clarify its utilisation in the perspective of future studies.

Nature of the study

Retrospective review.

Methods

A systematic literature search of Medline was carried out. From each article, data regarding the design of the study, the specialty, the performance criterion, the unit under control, the methodology and the model of the CUSUM used, the use of a graph, the use of a test and the type of test applied were retrieved.

Results

31 studies were found relevant. The design was mainly retrospective for the analysis of the learning curve. The main performance criteria under control were morbidity, mortality and success of the procedure. A graph was plotted in all studies as a CUSUM plot or as cumulative sums of non‐negative values. A test was used in 17 studies. Mislabelling of the plot and the test, and misuse of control limits were the most commonly reported mistakes.

Conclusion

The CUSUM tool is not yet properly reported in the surgical literature. Therefore, reporting of the CUSUM should be clarified and standardised before its use widens.

Quality control in medicine has generated considerable interest in the past decade issuing from public health authorities, doctors or patients themselves.1,2,3 The case of the general practitioner Harold Shipman and the Bristol Royal Infirmary Inquiry have brought the need for monitoring clinical performance into focus.4 More recently, it has found an important application in the control of surgical procedures, either in analysing the learning curve of a technique or in controlling a run of interventions.5,6,7 Lately, applications have been sought for controlling new technologies as they are initiated.8

The most commonly used methods of statistical process control are the Shewart chart, the sequential probability ratio test (SPRT), the exponentially weighted moving average and the cumulative sums (CUSUMs).4,9 Among all, the CUSUM has attracted more attention and disseminated in the medical literature due to its simple formulation and an intuitive representation, and to its capability of detecting small persistent changes.10,11 Numerous review papers have discussed methodological and statistical issues, and have given guidelines to perform such analyses.12,13 However, as for any new tool, its application spawns a fair confusion and, to the best of our knowledge, no one has yet reviewed how the CUSUM tool was, in fact, used across interventional and surgical literatures. Our aim was to bridge the gap between what should be done and what is actually done in the literature.

The objectives of the present work were to review the use of the CUSUM across different interventional and surgical specialities, and to clarify its use in the perspective of future studies.

Materials and methods

We performed a systematic literature search of Medline with the search terms “CUSUM or cumulative sums” and “surgery or procedure” on 15 March 2006, supplemented by cross‐bibliographic checks of reference lists. No restrictions were made regarding date of publication, language or publication status.

To be included in the review, studies had to analyse a manual procedure, surgical or other, by using the use of the CUSUM. Worked examples, methodological and statistical reviews were excluded.

Data retrieved from each article were: date of publication, specialty, objective (analysis of the learning curve or quality control) and design (retrospective or prospective) of the study, performance criterion (mortality, morbidity, success of the procedure, and so on), series size, numbers and types of units under control (care provider or centre), and the basis for definition of the reference value.

Data from the methodology and the applied model of the CUSUM were also recorded: the use of a graph as a CUSUM graph or as a CUSUM of non‐negative values; the use of a test (control limits or boundary lines) and the type of the applied test (CUSUM or SPRT); the reporting of enough information in the study to compute the graph if data were available; and the adjustment for patient's individual risk factors.

Different statistical methods have been used for continuous monitoring of a process: the CUSUM graph, the CUSUM test and the SPRT are the most commonly used methods (see Appendix).

The CUSUM graph should be differentiated from the CUSUM test, or tabular CUSUM. The CUSUM graph plots the cumulative sums of the deviation between the process and a target value. The graph provides a qualitative overview of the processed monitored (fig 1A1A).). The CUSUM test is a hypothesis test that relies on the comparison between computed values and a limit.14 It has a graphical representation where one plots the cumulative sums of the maximum between zero and a weighted value (sample weight) ((figsfigs 1B, 2A2A).). When the graph hits the limit, the process is claimed to be out of control, but it can never be considered as definitely in control. On the contrary, an inverted CUSUM is drawn below the zero horizontal axis and is used to detect quality improvements. A two‐sided CUSUM test allows detecting the deviation above or below a target when both are considered as suboptimal performances.

figure qc20776.f1
Figure 1 (A) A CUSUM graph and (B) a CUSUM test for monitoring limb alignment of total knee replacement using CT‐based navigation system.8 The CUSUM graph shows that learning curve lasted for the first 27 implants and then stabilised around ...
figure qc20776.f2
Figure 2 (A) A CUSUM test for monitoring death after paediatric cardiac operation.3 The test detected a cluster of surgical failures starting with patient 53. A comparison is drawn between the CUSUM test and (B) the sequential probability ratio ...

The SPRT is a hypothesis test in itself and a graphical representation could be displayed (fig 2B2B).). The SPRT presents two limits, an upper boundary line where one chooses to accept the alternative hypothesis (the process is in control) and a lower boundary line where one chooses to accept the null hypothesis.

A more detailed description of the use of the CUSUM can be found in dedicated reviews and are beyond the scope of this paper.12,13

Results

The search strategy generated 77 studies. In all, 31 studies were found relevant by title, abstract and on final complete retrieval of the article (table 11).). Publication years ranged from 1991 to 2005, with a recent increase in the number of publications (fig 33).). In all, nine studies were from Canada, eight from the UK, four from New Zealand, three from USA, two from France; and one publication each from five other countries.

figure qc20776.f3
Figure 3 The number of publications per year for procedures that are using the cumulative sum for the control of surgical or interventional procedures.
Table thumbnail
Table 1 Studies using the CUSUM as an evaluation tool for interventional and surgical procedures

The design was retrospective in 25 (80%) studies and prospective in 6 (20%); the objective was the analysis of the learning curve in 22 (71%) studies, quality control in 8 (26%) and both in 1 (3%) study. Specialties of interest were cardiovascular surgery in 14 (45%) studies, anaesthesia in 7 (23%), general surgery in 4 (13%), endoscopy in 3 (10%), otolaryngology‐head and neck surgery in 2 (6%) and orthopaedics in 1 (3%). The unit under control was the care provider in 22 (71%) studies and the centre in 9 (29%); the number of units under control per study ranged from 1 to 20. The median (range) number of procedures controlled per study was 299 (24–3983).

The main performance criterion under control was morbidity in 1 (3%) study, mortality in 7 (23%), mortality and morbidity in 7 (23%), and the success of the procedure in 16 (52%). The success of the procedure was defined as completion of the procedure, a procedure meeting the required standard or a procedure completed within a certain time. The data monitored was binomial in 28 (90%) studies and normally distributed in 3 (10%) studies. Target and failure rates were determined from consensus in the department in 5 (16%) studies, from reported rates in the literature in 9 (29%), from expert society requirements in 4 (13%) studies and from the observation during the monitoring of the procedure in 5 (16%). The basis for definition of target and failure rates was not reported in 7 (23%) studies.

A graph was plotted in all studies, but it was incorrectly labelled as “cumulative failures” in 2 (6%) studies; the plot was a CUSUM graph in 28 (90%) studies and a cumulative sum of non‐negative values in 3 (10%). A test was applied in 17 (55%) studies: 7 (41%) studies were SPRT, 6 of which were wrongly reported as CUSUM tests; 5 (29%) were standard CUSUM graphs with limits derived from a SPRT (ascending limits); 3 (18%) were CUSUM tests; 1 (6%) was the observed–expected cumulative values; and in 1 (6%) study, the boundary lines on the CUSUM graph were flat although they should have been ascending as the limits were those of a SPRT. Sufficient explanation for computing the CUSUM was given in 26 (84%) studies and the CUSUM was risk adjusted in 7 (23%) studies.

Discussion

Statistical process control has emerged in the medical literature after a wide expansion in the industry.15 In the 1920s, Shewart,16 who worked for Bell Telephone Laboratories, constituted a group of experts and established the scientific ground for quality control. Statistical process control and control charts emerged later, with Wald,17 who first developed the theory of SPRTs, and with Page14 who developed the CUSUM chart in the mid‐1950s. Control charts proved very useful to quality management in the industry and disseminated rapidly. Although the first publication of CUSUM in a general medical journal dates back to the mid‐1970s,18 its strong properties have not been exploited due to the confusion that arose in recent years together with the spread of the technique.

The CUSUM graph and the CUSUM test should be differentiated. Moreover, the CUSUM test and SPRT are usually confounded and this has yielded some confusion in the literature.3,19,20,21,22 The need for deciding whether a process is out of control requires the use of control limits. Some authors have used standard CUSUM graphs with limits being derived from a SPRT 23,24,25,26,27; however, a SPRT has two absorbing limits, the upper to reject H0 (the process is deemed out of control) and the lower to accept H0 (the process is above the required performance), and it does not make sense to continue monitoring once a limit has been hit.9 Indeed, if one continues plotting after the inferior limit has been hit, the CUSUM graph will build up credits with time and, therefore, it is unable to raise an alarm in due time if the process goes out of control after a period of good performance.25,26 The same applies for the upper limit, be it that of a CUSUM test or that of a SPRT test: when this limit is hit, the plot should be stopped, and the procedure may be investigated.

The CUSUM was usually used for two purposes: assessing a learning curve and quality control. The former has yet again generated more interest. However, the limits should be used with due care. Indeed, many have used a CUSUM test while assessing a learning curve, although it is not well adapted for this purpose.20,26,28 The CUSUM test was originally designed for monitoring a procedure that has reached a steady state, and was not designed to monitor a procedure that is supposed to be out of control from the start. In this case, the upper boundary limit of the CUSUM test is often crossed, sometimes as high as two or three times, and it then makes little sense for setting a limit.20,28 Reaching a steady state on the CUSUM graph may be enough assurance to conclude that the learning curve has settled down. Conversely, for quality control, a lower boundary line accepting the in‐control state, as in the SPRT, should not be used. Indeed, for continuous monitoring, when the purpose is the detection of changes that could occur during surveillance, one should never say that a procedure has reached an in‐control state and that monitoring can be safely stopped. To avoid this drawback, a resetting SPRT has been developed which resets by itself at 0, each time the lower boundary is hit.9

As for any new statistical tool, the CUSUM would benefit from clarification. Self‐contained explanations with adequate referencing to methodological or statistical reviews should be found in the text of each study. Authors willing to use the CUSUM should also clearly report on what was done. The design (retrospective or prospective) and the objective (analysis of learning curve or quality control) of the study, the performance criterion under control (mortality, morbidity, and so on) and how the reference value was determined (consensus from the department, literature, expert society, and so on) should be clearly enunciated in the methodology. The model of the CUSUM that was applied (CUSUM graph and CUSUM test), the type of data that were monitored (binomial, Poisson distribution and normally distributed data) and how the control limits were determined (log‐likelihood ratio) should also be clearly enunciated. Statistical considerations are beyond the scope of this work and can be found in specific reviews.4,29 However, with developments of the CUSUM in its prospective purpose, as for continuous monitoring and surveillance, other specific issues will soon arise: the determination of target and reference values, the determination of limits through the average run length and actions that should be taken when an alarm is raised.

Reports from the Institute of Medicine, Washington DC, USA have emphasised the need for building a safer health system.30 Inadequate monitoring of treatment is a cause for error, and the Institute of Medicine encouraged healthcare organisations to develop a culture of safety and create systems for continuously monitoring patient safety.30 The committee later proposed six aims for establishing the 21st century healthcare system and noted that today's healthcare system functions at far lower level than it can and should be.2 “Timely” was one of these key points and the report notified that “reducing waits and sometimes harmful delays for both those who receive and those who give care” would be far better at meeting patients' needs. The CUSUM tool has shown its power in detecting, in timely fashion, fatal medical errors, near misses and suboptimal clinical performance. Moreover, initiation of new surgical technologies, whose rapid development may render classic assessment inadequate, could benefit from continuous performance evaluation.8,31

Despite all the confusion in the use of the CUSUM, it is generating more and more interest in the surgical community, and is being handled more appropriately in recent years.8,32,33 Statistical control tools have shown their dramatic potential life‐saving impact through the retrospective study of Harold Shipman death certificates and the Royal Bristol infirmary annual mortality rates.4 However, their use in continuous prospective quality control is yet to be expanded. Continuous monitoring of failure rates (mortality after cardiac surgery, recurrence of hernia after repair, and so on) and clinical performance (rate of misses and near misses in cardiac surgery, conversion rate for laparoscopic surgery, and so on) at centre or care provider level would be of enormous help in improving patient care and cost effectiveness.34

Conclusion

The use of the CUSUM is generating more and more interest in the medical world and across the surgical community. However, despite extensive reviews on the topic, it is not yet properly reported in the surgical literature and the confusion remains, which may hamper its spread. Therefore, reporting of the CUSUM should be clarified and standardised. The shift from retrospective study and learning curve assessment to prospective quality control is at crossroads and promises great expectations.

Acknowledgments

We thank Dr Lysbeth Ravn for her help in translating her own manuscript. Her participation does not necessarily mean that she agrees with the content of the manuscript.

Abbreviations

SPRT - sequential probability ratio test

Appendix

The principle of CUSUM charts was developed for industrial quality control during the 1930s, and drew considerable attention and research during the 1950s. Assume that we wish to continuously monitor a process. The basic idea of CUSUM graphs is to plot the cumulative sum of the deviation between the process and a target value, instead of plotting the sequential values of the process themselves. If we note Xi as the ith measurement of the process and X0 the target value, the CUSUM Cn after the measurement of n has the simple expression: Cn = Σ(Xi–X0). When a target value cannot be defined, some first or last values of the process are taken (usually their mean) to define a target. Another expression of the CUSUM for Bernoulli data may be found where one plots Cn = Σ(Xi–s), where Xi–s is proportional to the log‐likelihood ratio.

The CUSUM test is a particular hypothesis test of the null hypothesis H0 “the process is in‐control”—that is, the performance is as expected, versus the alternative hypothesis H1 “the process is out of control”—that is, the level of performance has switched to a highly unacceptable level. The CUSUM test has an upper boundary line where one accepts the alternative hypothesis. When the graph hits this limit, the process is then claimed to be out of control. On the other side, it has a holding barrier at 0, which cannot be crossed; when the graph hits this limit it stays at 0 and the monitoring continues. With this procedure, acceptance that the process has reached an in‐control state is not possible. This is particularly well suited to quality control, where the process will possibly deviate in the future. For normally distributed data, two CUSUM tests are drawn simultaneously (one positive and one negative).

The sequential probability ratio test (SPRT) is a very close statistical method. However, the SPRT has a negative lower boundary. When it is crossed, enough evidence accumulates to say that the process is in control, and the monitoring should stop. Figure 11 presents an illustration of the CUSUM methodology.

Footnotes

Competing interests: None.

References

1. The Inquiry into the management of care of children receiving complex heart surgery at the Bristol Royal Infirmary http://www.bristol‐inquiry.org.uk/ (accessed 3 Apr 2007)
2. Institute Of Medicine Crossing the quality chasm: a new health system for the 21st Century. Washington, DC: National Academy Press, 2001
3. de Leval M R, Francois K, Bull C. et al Analysis of a cluster of surgical failures. Application to a series of neonatal arterial switch operations. J Thorac Cardiovasc Surg. 1994;107: 914–23; discussion 923–14,
4. Spiegelhalter D, Grigg O, Kinsman R. et al Risk‐adjusted sequential probability ratio tests: applications to Bristol, Shipman and adult cardiac surgery. Int J Qual Health Care 2003. 157–13.13 [PubMed]
5. Rogers C A, Ganesh J S, Banner N R. et al Cumulative risk adjusted monitoring of 30‐day mortality after cardiothoracic transplantation: UK experience. Eur J Cardiothorac Surg 2005. 271022–1029.1029 [PubMed]
6. Young A, Miller J P, Azarow K. Establishing learning curves for surgical residents using cumulative summation (CUSUM) analysis. Curr Surg 2005. 62330–334.334 [PubMed]
7. Bolsin S, Colson M. The use of the Cusum technique in the assessment of trainee competence in new procedures. Int J Qual Health Care 2000. 12433–438.438 [PubMed]
8. Nizard R S, Porcher R, Ravaud P. et al Use of the Cusum technique for evaluation of a CT‐based navigation system for total knee replacement. Clin Orthop Relat Res 2004. 180–188.188
9. Grigg O A, Farewell V T, Spiegelhalter D J. Use of risk‐adjusted CUSUM and RSPRT charts for monitoring in medical contexts. Stat Methods Med Res 2003. 12147–170.170 [PubMed]
10. Montgomery D C. Introduction to statistical quality control. 5th edn. New York, NY: John Wiley & Sons, 2005
11. Altman D G, Royston J P. The hidden effect of time. Stat Med . 1988;7629–637.637
12. Grunkemeier G L, Wu Y X, Furnary A P. Cumulative sum techniques for assessing surgical results. Ann Thorac Surg 2003. 76663–667.667 [PubMed]
13. Steiner S H, Cook R J, Farewell V T. Risk‐adjusted monitoring of binary surgical outcomes. Med Decis Making 2001. 21163–169.169 [PubMed]
14. Page E S. Continuous inspection schemes. Biometrika 1954. 41100–114.114
15. Williams S M, Parry B R, Schlup M M. Quality control: an application of the cusum. BMJ 1992. 3041359–1361.1361 [PMC free article] [PubMed]
16. Shewart W A. Economic control of quality manufactured product. New York: Van Nostrand, 1931
17. Wald A. Sequential analysis. New York, NY: John Wiley & Sons, 1947
18. de Saintonge D M, Vere D W. Why don't doctors use cusums? Lancet 1974. 1120–121.121 [PubMed]
19. de Oliveira Filho G R. The construction of learning curves for basic skills in anesthetic procedures: an application for the cumulative sum method. Anesth Analg 2002. 95411–416.416 [PubMed]
20. Naik V N, Devito I, Halpern S H. Cusum analysis is a useful tool to assess resident proficiency at insertion of labour epidurals. Can J Anaesth 2003. 50694–698.698 [PubMed]
21. Schlup M M, Williams S M, Barbezat G O. ERCP: a review of technical competency and workload in a small unit. Gastrointest Endosc 1997. 4648–52.52 [PubMed]
22. Schuepfer G, Johr M. Psoas compartment block (PCB) in children. Part II: generation of an institutional learning curve with a new technique. Paediatr Anaesth 2005. 15465–469.469 [PubMed]
23. Forbes T L, De Rose G, Harris K A. A CUSUM analysis of ruptured abdominal aortic aneurysm repair. Ann Vasc Surg 2002. 16527–533.533 [PubMed]
24. Forbes T L, DeRose G, Kribs S W. et al Cumulative sum failure analysis of the learning curve with endovascular abdominal aortic aneurysm repair. J Vasc Surg 2004. 39102–108.108 [PubMed]
25. Novick R J, Fox S A, Stitt L W. et al Cumulative sum failure analysis of a policy change from on‐pump to off‐pump coronary artery bypass grafting. Ann Thorac Surg 2001. 72S1016–S1021.S1021 [PubMed]
26. Novick R J, Stitt L W. The learning curve of an academic cardiac surgeon: use of the CUSUM method. J Card Surg. 1999;14: 312–20; discussion 321–2,
27. Song M H, Tajima K, Watanabe T. et al Learning curve of coronary surgery by a cardiac surgeon in Japan with the use of cumulative sum analysis. Jpn J Thorac Cardiovasc Surg 2005. 53551–556.556 [PubMed]
28. Kestin I G. A statistical approach to measuring the competence of anaesthetic trainees at practical procedures. Br J Anaesth 1995. 75805–809.809 [PubMed]
29. Grigg O A, Farewell V T. A risk‐adjusted sets method for monitoring adverse medical outcomes. Stat Med 2004. 231593–1602.1602 [PubMed]
30. Kohn L T, Corrigan J M, Donaldson M S. To err is human: building a safer health system. Washington, DC: National Academy Press, 2000
31. McCulloch P, Taylor I, Sasako M. et al Randomised trials in surgery: problems and possible solutions. BMJ 2002. 3241448–1451.1451 [PMC free article] [PubMed]
32. Harris J R, Forbes T L, Steiner S H. et al Risk‐adjusted analysis of early mortality after ruptured abdominal aortic aneurysm repair. J Vasc Surg 2005. 42387–391.391 [PubMed]
33. Sherlaw‐Johnson C, Morton A, Robinson M B. et al Real‐time monitoring of coronary care mortality: a comparison and combination of two monitoring tools. Int J Cardiol 2005. 100301–307.307 [PubMed]
34. Guth K A, Kleiner B. Quality assurance in the health care industry. J Health Care Finance 2005. 3133–40.40 [PubMed]
35. Parry B R, Williams S M. Competency and the colonoscopist: a learning curve. Aust N Z J Surg 1991. 61419–422.422 [PubMed]
36. Parry B R, Goh H S. Quality control in colonoscopy: a Singapore perspective. Int J Colorectal Dis 1993. 8139–141.141 [PubMed]
37. de Leval M R, Francois K, Bull C. et al Analysis of a cluster of surgical failures. Application to a series of neonatal arterial switch operations. J Thorac Cardiovasc Surg 1994. 107914–23 discussion 9234.23 discussion 9234 [PubMed]
38. Kestin I G. A statistical approach to measuring the competence of anaesthetic trainees at practical procedures. Br J Anaesth 1995. 75805–809.809 [PubMed]
39. Van Rij A M, McDonald J R, Pettigrew R A. et al Cusum as an aid to early assessment of the surgical trainee. Br J Surg 1995. 821500–1503.1503 [PubMed]
40. Atherton D P, O'Sullivan E, Lowe D. et al A ventilation‐exchange bougie for fibreoptic intubations with the laryngeal mask airway. Anaesthesia 1996. 511123–1126.1126 [PubMed]
41. Schlup M M, Williams S M, Barbezat G O. ERCP: a review of technical competency and workload in a small unit. Gastrointest Endosc 1997. 4648–52.52 [PubMed]
42. Poloniecki J, Valencia O, Littlejohns P. Cumulative risk adjusted mortality chart for detecting changes in death rate: observational study of heart surgery. BMJ 1998. 3161697–1700.1700 [PMC free article] [PubMed]
43. Novick R J, Stitt L W. The learning curve of an academic cardiac surgeon: use of the CUSUM method. J Card Surg 1999. 14312–20 discussion 3212.20 discussion 3212 [PubMed]
44. Bartlett A, Parry B. Cusum analysis of trends in operative selection and conversion rates for laparoscopic cholecystectomy. ANZ J Surg 2001. 71453–456.456 [PubMed]
45. Novick R J, Fox S A, Stitt L W. et al Cumulative sum failure analysis of a policy change from on‐pump to off‐pump coronary artery bypass grafting. Ann Thorac Surg 2001. 72S1016–S1021.S1021 [PubMed]
46. Ravn L I, Sprehn M, Pedersen C B. [The Cusum score. A tool for evaluation of clinical competence]. Ugeskr Laeger 2001. 1633644–3648.3648 [PubMed]
47. de Oliveira Filho G R. The construction of learning curves for basic skills in anesthetic procedures: an application for the cumulative sum method. Anesth Analg 2002. 95411–416.416 [PubMed]
48. Forbes T L, De Rose G, Harris K A. A CUSUM analysis of ruptured abdominal aortic aneurysm repair. Ann Vasc Surg 2002. 16527–533.533 [PubMed]
49. Novick R J, Fox S A, Stitt L W. et al Effect of off‐pump coronary artery bypass grafting on risk‐adjusted and cumulative sum failure outcomes after coronary artery surgery. J Card Surg 2002. 17520–528.528 [PubMed]
50. Novick R J, Fox S A, Stitt L W. et al Assessing the learning curve in off‐pump coronary artery surgery via CUSUM failure analysis. Ann Thorac Surg 2002. 73S358–S362.S362 [PubMed]
51. Robinson I A, Blackham R B, Cozens N J. et al Good practice in head and neck fine needle aspiration cytology as assessed by CUSUM. Cytopathology 2002. 13335–342.342 [PubMed]
52. Naik V N, Devito I, Halpern S H. Cusum analysis is a useful tool to assess resident proficiency at insertion of labour epidurals. Can J Anaesth 2003. 50694–698.698 [PubMed]
53. Novick R J, Fox S A, Kiaii B B. et al Analysis of the learning curve in telerobotic, beating heart coronary artery bypass grafting: a 90 patient experience. Ann Thorac Surg 2003. 76749–753.753 [PubMed]
54. Sharp J F, Cozens N, Robinson I. Assessment of surgical competence in parotid surgery using a CUSUM assessment tool. Clin Otolaryngol Allied Sci 2003. 28248–251.251 [PubMed]
55. Forbes T L, DeRose G, Kribs S W. et al Cumulative sum failure analysis of the learning curve with endovascular abdominal aortic aneurysm repair. J Vasc Surg 2004. 39102–108.108 [PubMed]
56. Nizard R S, Porcher R, Ravaud P. et al Use of the Cusum technique for evaluation of a CT‐based navigation system for total knee replacement. Clin Orthop Relat Res 2004. 425180–188.188 [PubMed]
57. Harris J R, Forbes T L, Steiner S H. et al Risk‐adjusted analysis of early mortality after ruptured abdominal aortic aneurysm repair. J Vasc Surg 2005. 42387–391.391 [PubMed]
58. Mace L, Bertrand S, Lucron H. et al [Paediatric cardiac surgery and autoevaluation: risk score, complexity score and graphic analysis]. Arch Mal Coeur Vaiss 2005. 98477–484.484 [PubMed]
59. Rogers C A, Ganesh J S, Banner N R. et al Cumulative risk adjusted monitoring of 30‐day mortality after cardiothoracic transplantation: UK experience. Eur J Cardiothorac Surg 2005. 271022–1029.1029 [PubMed]
60. Schuepfer G, Johr M. Psoas compartment block (PCB) in children: part II–generation of an institutional learning curve with a new technique. Paediatr Anaesth 2005. 15465–469.469 [PubMed]
61. Sherlaw‐Johnson C, Morton A, Robinson M B. et al Real‐time monitoring of coronary care mortality: a comparison and combination of two monitoring tools. Int J Cardiol 2005. 100301–307.307 [PubMed]
62. Song M H, Tajima K, Watanabe T. et al Learning curve of coronary surgery by a cardiac surgeon in Japan with the use of cumulative sum analysis. Jpn J Thorac Cardiovasc Surg 2005. 53551–556.556 [PubMed]
63. Tekkis P P, Fazio V W, Lavery I C. et al Evaluation of the learning curve in ileal pouch‐anal anastomosis surgery. Ann Surg 2005. 241262–268.268 [PubMed]
64. Tekkis P P, Senagore A J, Delaney C P. et al Evaluation of the learning curve in laparoscopic colorectal surgery: comparison of right‐sided and left‐sided resections. Ann Surg 2005. 24283–91.91 [PubMed]
65. Young A, Miller J P, Azarow K. Establishing learning curves for surgical residents using Cumulative Summation (CUSUM) Analysis. Curr Surg 2005. 62330–334.334 [PubMed]

Articles from Quality & Safety in Health Care are provided here courtesy of BMJ Publishing Group