Over the past 10 years, the use of quality indicators (QIs) has been strongly encouraged as a means of assessing quality in hospitals. QIs are now a widely used tool in hospital regulation in developed countries (eg, in performance-based financing and the public disclosure of hospital comparisons).1
It is well established that key attributes of QIs are able to detect targeted areas for improvement on topics of importance, scientific soundness and feasibility.2
For national comparisons of healthcare organisations (HCO), a valid and standardised data collection process is also required, as any errors could affect a hospital's reputation and also have financial repercussions. Three main data sources are used to develop QIs: (1) ad hoc surveys (eg, patient's experience and satisfaction indicators), but these are costly and require recruitment of respondents and high hospital commitment,3
(2) medico-administrative data (eg, patient safety indicators), but these often capture limited information on complex care processes4
and (3) medical records (eg, clinical practice and organisational indicators) which are the preferred option for obtaining accurate and reliable clinical and organisational information.5–9
Most medical records are still paper medical records (PMRs) and entail difficulties in terms of data extraction that remains manual.10
The adoption of interoperable electronic medical record (EMR) systems could promote efficiency by developing an automated process of data extraction. However, it is expensive. Moreover, in the most highly developed countries that invest in this area, it remains arduous. For instance in US hospitals, only 13% reported use of a basic EMR system in 2008, according to a study by Jha et al
Although these numbers have significantly increased over the last few years—2011 data shows 35% adoption of basic EMR systems by US hospitals—the rates of adoption are still low.12
A recent national study shows that in France only 6% of medical records are fully electronic.13
In terms of data extraction for the purposes of quality measurement, a basic EMR system does not necessarily enable easy and automatic computation of aggregated data, nor does it preclude the use of partial paper charts, making some data completely inaccessible via the EMR.
We propose a pragmatic method for using PMRs to produce national QIs that display feasibility, reliability and discriminative power, and that enable PMR audits for hospital comparison. The method is based on data extraction from a random sample of PMRs in each hospital. It has been implemented in France since 2006, but could be adopted by other countries interested in assessing large-scale hospital performance.14
We describe the methods development, the PMR sampling strategy and the statistical procedures for ensuring robustness. Each step of the method is illustrated with appropriate examples. Last, we discuss the place of such a method in the context of development of EMR systems.