Search tips
Search criteria 


Logo of bmjThis ArticleThe BMJ
BMJ. 2007 October 13; 335(7623): 730.
PMCID: PMC2018797

Data sources and performance measurement

Stephen Singleton, medical director

Measuring outcomes is necessary but difficult to get right

In this week's BMJ, Westaby and colleagues compare the value of two sources of data for determining mortality 30 days after congenital cardiac surgery—hospital episode statistics (HES) and the central cardiac audit database.1 They find that the central cardiac audit database is more complete than HES, but that individual centres need investment to improve the completeness and accuracy of their data. Their investigation follows a study published in the BMJ in 2004 that used HES to compare mortality from congenital heart surgery in different UK centres.2 The study suggested that Oxford had significantly higher mortality than the national average, and the results were reported widely by the media. So have we learnt anything new about the relative value of routinely collected versus specifically collected sources of data?

Routinely collected patient data are regularly analysed to investigate outcome. Equally regularly the results are contested by the specifically collected dataset, which is often designed to measure the very thing being looked for. So why use routinely collected data to draw clinical conclusions at all? The advantages include pragmatism, wide coverage, low cost, and easy access. The disadvantages include superficial or inaccurate coding and potentially damaging generalisations.

HES data from the National Health Service (NHS) are widely used to produce outcome information and more recently to publicise differences between hospitals. Data produced for administrative and financial purposes that are centred on the organisation not the patient may never be as complete as data derived from clinicians.

Huge datasets also invite misuse of statistical method—the significance of correlations is a product of the number of data points, not necessarily its relative importance. Chance findings will occur if many tests are done on the same data; association is not the same as causality. Nonetheless, the sheer scale of the HES database makes it attractive and it has become a rich source of hypothesis generation and evidence on outcomes. The database has been used to investigate associations between case volume and outcome (for example, oesophagectomy3 and repair of aortic aneurysms4), to search for potentially useful predictors of outcome (for example, excess mortality associated with delay in operation after hip fracture5), to carry out quasilongitudinal studies to track changes in outcome related to changes in clinical practice (for example, acute urinary retention and prostatectomy,6 follow-up after emergency admission,7 and changes in mortality after paediatric cardiac surgery2), and increasingly to predict individual outcomes in patients at high risk.

Results that conflict with HES data have often been reported—for example, the relation between hospital volume and outcome8 9 and the predictive factors for poorer outcome in certain patients.10 This is not just a contest of science and statistics but of politics and hearts and minds. As Westaby and colleagues note, the media rapidly picked up on the conclusion based on HES data that Oxford had significantly higher mortality after paediatric cardiac surgery than the national average.

While recognising that there are problems in special datasets too—timescales and numbers of episodes are often smaller, making it more likely to miss a rare event or true difference, and collecting outcome data on your own performance may bias the case mix of patients selected for intervention—Westaby and colleagues conclude that HES data should not be used for comparisons within specialties.

Patients do not necessarily trust official data sources.11 We need to know if they will trust information collected by doctors who analyse their own data and claim that their own performance is sound. The NHS has recently appointed a new medical director, Professor Sir Bruce Keogh, who is famous for his leadership of British cardiothoracic surgeons in measuring outcomes and making them public. This appointment sends a clear signal to staff, the public, and the media about the importance of measuring outcomes.

It is unclear how much patients change their choice of provider based on such knowledge, or how much employers manage their clinical staff with an eye on comparative performance, however intuitively it seems important. With all its potential problems, HES has more to offer than league tables of performance. Better knowledge will flow from a collaboration of all sound analyses, based on complete data, that are accurately coded by clinicians who have an interest in the outcome. This can only lead to more complete and contextualised data being released into the public domain.


Competing interests: None declared.

Provenance and peer review: Commissioned; not externally peer reviewed.


1. Westaby S, Archer N, Manning N, Adwani S, Grebenik C, Ormerod O, et al. Comparison of hospital episode statistics and central cardiac audit database in public reporting of congenital heart surgery mortality. BMJ 2007;335:759 doi: 10.1136/bmj.39318.644549.AE
2. Aylin P, Bottle A, Jarman B, Elliott P. Paediatric cardiac surgical mortality in England after Bristol: descriptive analysis of hospital episode statistics 1991-2002. BMJ 2004;329:825-9. [PMC free article] [PubMed]
3. Al-Sarira AA, David G, Willmott S, Slavin JP, Deakin M, Corless DJ. Oesophagectomy practice and outcomes in England. Br J Surg 2007;94:585-91. [PubMed]
4. Holt PJE, Poloniecki JD, Loftus IM, Michaels JA, Thompson MM. Epidemiological study of the relationship between volume and outcome after abdominal aortic aneurysm surgery in the UK from 2000 to 2005. Br J Surg 2007;94:441-8. [PubMed]
5. Bottle A, Aylin P. Mortality associated with delay in operation after hip fracture: observational study. BMJ 2006;332:947-51. [PMC free article] [PubMed]
6. Cathcart P, van der Meulen J, Armitage J, Emberton M. Incidence of primary and recurrent acute urinary retention between 1998 and 2003 in England. J Urol 2006;176:200-4. [PubMed]
7. Roland M, Dusheiko M, Gravelle H, Parker S. Follow-up of people aged 65 and over with a history of emergency admissions: analysis of routine admission data. BMJ 2005;330:289-92. [PMC free article] [PubMed]
8. Khuri SF, Henderson WG. The case against volume as a measure of quality of surgical care. World J Surg 2005;29:1222-9. [PubMed]
9. Birkhead JS, Weston C, Lowe D, and National Audit of Myocardial Infarction Project (MINAP) Steering Group. Impact of specialty of admitting physician and type of hospital on care and outcome for myocardial infarction in England and Wales during 2004-5: observational study. BMJ 2006;332:1306-11. [PMC free article] [PubMed]
10. Franzo A, Simon G, Francescutti C. Mortality associated with delay in operation after hip fracture: . . . but Italian data seem to contradict study findings. BMJ 2006;332:1093
11. Magee H, Davis LJ, Coulter A. Public views on healthcare performance indicators and patient choice. J R Soc Med 2003;96:338-42. [PMC free article] [PubMed]

Articles from The BMJ are provided here courtesy of BMJ Group