The Department of Health publishes data on MRSA infection in individual trusts obtained through its mandatory reporting scheme.4
Using data for financial years 2001-2, 2002-3, and 2003-4, I estimated how close the variation was to that expected by chance from the Poisson regression residuals around an individual trend line for each trust.5
This showed a significant over-dispersion of 1.76 (Pearson χ2
= 304.3, degrees of freedom = 167; P < 0.001). For example, Aintree Hospitals NHS Trust had 34 cases in 2001-2, rising to 66 cases in 2002-3, and falling to 48 in 2004-4, far more variability than would be expected by chance alone. Such over-dispersion is expected with an infectious disease, and means that all interval estimates of MRSA rates should be 33% wider (1.33 = √1.76) than those that assume simple random variability.
shows the MRSA rates in 2003-4 per 1000 bed days plotted against the number of bed days. Such funnel plots can be interpreted as control charts6,7
and have been used in a slightly different form by the Health Protection Agency when publishing MRSA rates.3
The control limits, expanded to allow for over-dispersion, are set around the overall rate observed for each trust type: 0.24 per 1000 bed days for specialist trusts, 0.16 for general acute, and 0.10 for single specialty. Although most trusts lie within the control limits, there are some clear outliers with high rates.
Fig 1 MRSA cases per 1000 bed days in 173 NHS acute trusts in 2003-4. The two funnels indicate where 95% and 99.8% of trusts would lie by chance alone, assuming no underlying variability within each type of trust. These boundaries correspond to 2 and 3 standard (more ...)
(left) shows a funnel plot of the change in rates between 2002-3 and 2003-4, pooling all types of trusts and using control limits around a hypothesis of no change (ratio = 1). All the trusts lie within the 3 standard deviation limits (adjusted for overdispersion), showing that any attempt at ranking trusts into a detailed league table of change would be entirely spurious. also emphasises the wide variability in change expected (and observed) in centres with low counts, which may occur through the trust either being small or having low MRSA rates, or both.
Fig 2 Ratio of MRSA rates in 2003-4 to those in 2002-3 in 169 NHS acute trusts plotted against (left) the average number of cases each year, with 95% and 99.8% control limits around an assumption of no change (ratio =1) and (right) the rate in 2002-3. Four (more ...)
(right) shows the ratio of the rates in 2003-4 to 2002-3 against the baseline 2002-3 rates. The clear negative relation between these (correlation = -0.43) shows the phenomenon of regression to the mean.8
Essentially, since high or low rates in 2002-3 are largely due to runs of chance events that are unlikely to be repeated, rates in the subsequent year will tend to be closer to the overall average rate. This immediately explains the finding reported by the media that “some of the hospitals with the lowest rates last year had a rise in MRSA cases this year,” and examples such as “York Health Services NHS Trust slipped 42 places in the ranking from having the lowest rate of MRSA cases for a general hospital last year.”9
(left) shows the chance limits for trusts with no true risk reduction: the 95% limits, for example, would then correspond to outcomes that show a “significant” (P < 0.05) change. Trusts with, say, less than 75 cases a year (comprising 82% of all trusts) would have a fair chance of observing a reduction in the shaded area—that is, achieving the target 20% reduction—by chance alone; for a median trust with 32 cases the probability of a false positive reduction is 25%. To prove a significant reduction, a median trust (marked M1) would need to achieve an observed risk ratio of 0.47 (53% observed risk reduction).
Fig 3 Funnel plots for a true annual risk reduction of 0% (left) and 20% (right). Trusts with data in the shaded area would achieve the 20% annual target. M1 corresponds to a median trust (32 cases a year) that achieves a significant reduction. M2 corresponds (more ...)
(right) shows that a trust with a true risk reduction of 20% has only a 50% chance of actually observing a reduction in the shaded area and hence achieving the target. However, if we were merely aiming to achieve a reduction compatible with the target (that is, not significantly different from the target), then a median trust (M2) would have to show only less than a 49% increase in order not to reject the hypothesis of a true 20% reduction. Detailed power calculations are provided on bmj.com