|Home | About | Journals | Submit | Contact Us | Français|
To study the effect of electronic medical record (EMR) implementation on preventive services covered by Ontario’s pay-for-performance program.
Prospective double-cohort study.
Twenty-seven community-based family physicians.
Eighteen physicians implemented EMRs, while 9 physicians continued to use paper records.
Provision of 4 preventive services affected by pay-for-performance incentives (Papanicolaou tests, screening mammograms, fecal occult blood testing, and influenza vaccinations) in the first 2 years of EMR implementation.
After adjustment, combined preventive services for the EMR group increased by 0.7%, a smaller increase than that seen in the non-EMR group (P = .55, 95% confidence interval −2.8 to 3.9).
When compared with paper records, EMR implementation had no significant effect on the provision of the 4 preventive services studied.
Vérifier l’effet de la mise en œuvre des dossiers médicaux électroniques (DMÉ) sur les services de prévention couverts par le programme ontarien de rémunération au rendement.
Étude prospective à double cohorte.
Vingt-sept médecins de famille pratiquant en milieu communautaire.
Dix-huit médecins avaient adopté les DMÉ alors que les 9 autres continuaient d’utiliser les dossiers papier.
Effet du programme incitatif de rémunération au rendement sur la prestation de 4 services préventifs (tests de Papanicolaou, dépistage par mammographies, recherche du sang occulte dans les selles et vaccination contre la grippe) au cours des 2 ans suivant l’introduction des DMÉ.
Après ajustement, l’ensemble des services préventifs pour le groupe DMÉ a augmenté de 0,7 %, une augmentation inférieure à celle observée pour le groupe non DMÉ (P = 0,55, intervalle de confiance à 95 % −2,8 à 3,9).
Par rapport aux dossiers papiers, la mise en œuvre des DMÉ n’a pas eu d’effet significatif sur la prestation des 4 services préventifs étudiés.
Electronic medical records (EMRs) have been identified as critical to quality improvement efforts,1–4 and policies favouring the establishment of EMRs are being implemented in the United States and in most Canadian provinces.2,5,6 In 2005, the Ontario government offered a subsidy to selected primary care physicians for the purchase of EMRs.5 Physicians who were eligible were those who had chosen a blended capitation payment system based largely on the age and sex of patients enrolled in their practices. By the end of the program in 2009, most physicians offered the subsidy had purchased EMRs.7 This provided an opportunity to compare groups of physicians in the same communities implementing and not implementing EMRs.
Although EMRs invite hope for improvement,8 there is still much that is unknown about the effect of EMRs on quality and performance. A recent systematic review found that computerized decision-support systems improved practitioner performance, especially with regard to immunizations.9 However, these systems were not commercial, off-the-shelf EMRs like those commonly found in primary care practices. Several studies have found that EMRs might not improve care10–15 and might even be a source of errors.15–17 Studies of EMR implementation have often been descriptive, with very few evaluating measurable outcomes.16 At present, the effect of EMR implementation on the quality of care in small community-based family practices is unknown.
Electronic medical record systems are complex, and their implementation involves changes to many processes; only some changes can be implemented early on. In this study, we focused on preventive services targeted by Ontario’s pay-for-performance program as markers of early progress in EMR implementation. These services were attached to financial incentives and were perceived as representing good care.17 The pay-for-performance target numbers varied by physician and were based on the percentage of enrolled family practice patients being provided with Papanicolaou smears, mammograms, influenza vaccinations, fecal occult blood screening, and primary vaccinations (in children younger than 2 years).18,19
In order to be eligible for funding in Ontario, an EMR system had to include the ability to manage the provision of such preventive services through point-of-care alerts and reminder letters. Although physicians using paper records were also offered financial incentives and were expected to produce an increase in preventive services, those using EMRs had electronic tools that could lead to a greater increase.
Our objective was to compare the differences in provision of these preventive services between community-based family physicians implementing EMRs and those continuing to use paper-based records.
We followed 2 cohorts of physicians: a group of 18 physicians implementing EMRs and a group of 9 physicians using paper records (the non-EMR cohort). Physicians in both cohorts were community-based, were affiliated with a local hospital, and were located in the Toronto area. All were members of the same local after-hours clinic. Physicians were signed out to the clinic and took turns providing walk-in care at a single location on evenings and weekends. No preventive services were offered at the clinic and no clinic data were included in the study.
The physicians in the EMR cohort had previously participated in a pay-for-performance study,20 so data on their characteristics were already available. They switched to the blended capitation model at the end of 2004 and began EMR implementation in the first 4 months of 2006 using the same EMR software. We studied the change in preventive services in the first 2 years of EMR implementation (2006 and 2007). The principal investigator was also a participant in this study (M.G.).
We recruited the non-EMR physicians through a letter of invitation and brief survey sent to all members of the after-hours clinic who were not part of the EMR cohort (125 physicians). These physicians were not eligible for the EMR subsidy. The recruitment process is shown in Figure 1.
The Ontario Ministry of Health and Long-Term Care provides family physicians with a list of patients in their practices who are eligible for each of the 4 preventive care services. We selected charts from these lists, using a random numbers table, from which we recorded the following: each patient’s age and sex, the presence or absence of a service within the required time period, and any record that a reminder letter had been sent to patients overdue for a service. We entered the data into an Epi Info database.21
We examined information on factors possibly associated with the provision of preventive services.22–32 Data were obtained through a questionnaire administered to all physicians. Following the methods of Glazier et al,33 practice-level data were also derived from linked administrative databases at the Institute for Clinical Evaluative Sciences (ICES). Income quintiles were derived from 2001 census data linked to patient postal codes. Neighbourhood income quintiles and recent immigration status were derived using published methods.33,34 Morbidity and comorbidity (adjusted diagnosis groups and resource utilization bands) were derived using the Johns Hopkins Adjusted Clinical Groups software,35 available from ICES, which has been described elsewhere.36–43 Numbers and percentages of patients with chronic diseases were obtained from linked ICES health databases.38–44
We audited paper charts for physicians in the non-EMR cohort and obtained data from EMR charts for physicians in the EMR cohort. When data were unavailable in the EMR, we retrieved and obtained data from the paper chart.
Five data auditors abstracted data from both paper charts and EMRs. The research coordinator initially audited 10 charts for each service in 2 practices and reviewed this with the principal investigator (M.G.). The coordinator then trained each data auditor, and reviewed at least 10 charts for each service from each auditor. The data were collected on paper forms and entered in the Epi Info database by 2 data entry clerks. Each clerk entered data from a training sample of at least 10 charts for each service. A randomly selected 10% data sample for each service, each year, and each physician was re-audited and entered in the database; we used the κ statistic to compare the 2 audits.
The study’s end point was whether or not a preventive service was provided and documented within a required time period for each eligible patient. The target patient population consisted of all eligible patients enrolled in the participating practices. Documentation that the patient received the service through another health care provider was acceptable. Information on services is presented in Table 1.
We calculated a composite process score44 by using the total number of charts audited for eligible patients for each service per physician as the denominator and the total number of services documented in the audits as the numerator. We could not obtain lists of eligible children for 2006 in the non-EMR cohort. As a result, children’s vaccinations were excluded from the composite score.
We compared the change in the proportion of patients who received preventive services between the EMR and non-EMR cohorts during EMR implementation. A 5% increase was considered the minimum clinically important difference.
We calculated the sample size required to have an 80% power to detect a clinically important increase in service provision of 5% or higher20 in the first 2 years of EMR implementation (assuming relatively stable rates in the non-EMR cohort and with rates of influenza vaccination ranging from 83% to 88%) using an α level of .05, and determined that 40 charts per service per provider would be required. To further increase statistical power, we audited 50 charts per year per service per physician. Physicians who practise in groups might influence one another; this would reduce the variation in practice and might inflate the observed effect.45 Based on our previous pay-for-performance study,20 the estimated intraclass correlation coefficient46 due to clustering of physicians within practices was 0.01. If we assume that, on average, the recruited physicians were clustered within groups of 4 physicians, and that the intraclass correlation coefficient was 0.01, the inflation factor would be 1.03 ([1 + (4 - 1) × 0.01]), which should have a negligible effect on the sample size required.
We compared unadjusted percentages using χ2 tests and compared adjusted percentages using multivariate logistic regression analysis. We used generalized estimating equations to adjust for the clustering structure of the data in regression models. When comparing the 2 groups, we adjusted for patient age for each individual patient,47 physician sex,20 years since each physician’s medical school graduation,22 and Certification in Family Medicine (ie, CCFP) status26 using multivariate logistic regression analysis. Administrative data could not be used for adjustment, as these data were not available to us at the patient level. Percentage of fecal occult blood tests provided might have been different from the other services provided; the provincial provision rate was low and there was a concurrent public health campaign in 2007.48,49 As a result, we re-analyzed the 2 cohorts with this screen excluded.
As the principal investigator was also a participant in the study, a sensitivity analysis was performed by excluding her practice data.
We performed analyses with SAS software, version 9.1. All tests were 2-sided using an α level of .05. The study was approved by the University of Toronto’s Research Ethics Board; the Sunnybrook Research Institute’s Research Ethics Board approved the use of ICES data. All participating physicians provided written informed consent.
Characteristics of the EMR and non-EMR physicians and their practice populations are presented in Table 2 and Table 3,22, 34–43 respectively. Physician characteristics between the 2 cohorts were similar. Patients in the EMR cohort were slightly younger and had lower levels of morbidity and comorbidity.
Results for individual services are presented in Table 4. Nine hundred charts were audited for each service for each year in the EMR cohort and 450 charts were audited in the non-EMR cohort. The number of Pap smears and mammograms increased more in the EMR cohort, while the rate of influenza vaccinations and fecal occult blood testing increased more in the non-EMR cohort. Composite process scores are shown in Table 5.
Female physician sex and younger patient age were positively correlated with the likelihood of receiving a service. There was no correlation with years since medical school graduation or Certification status. There was no significant difference in the change in service provision between the 2 groups; differences and confidence intervals were less than 5%. The results were not affected by the exclusion of fecal occult blood screening data.
Physicians in the EMR cohort mailed letters to 23 patients overdue for services in 2005, 265 patients in 2006, and 677 patients in 2007. Physicians in the non-EMR cohort mailed 1 letter in 2006 and none in 2007.
The results were unaffected by the exclusion of the principal investigator’s data. The κ statistic for the 10% of charts that were re-audited was 0.954, consistent with an acceptable level of agreement.
We found no statistically significant or clinically important difference in the change in preventive service provision between physicians implementing EMRs and those continuing to use paper records. Other studies have found no difference in the change in care of diabetes between EMR- and paper-based practices11 and no improvement when additional experience with EMR accrues over time.11,13,15
The EMR system we studied included elements that have been associated with an increase in the provision of preventive services, such as point-of-care alerts9,50,51 and the ability to generate reminder letters to patients.52 However, studies have shown that alerts or patient reminders for overdue services, even if available, might not be consistently implemented in practice.11,15,53,54 We found evidence of reminder letter mailings in the EMR cohort, but this did not appear to have had an effect on service uptake.
To frame and explain our results, a qualitative study presented in an accompanying article55 explores the context of this EMR implementation. Similar studies could help to determine what aspects of EMR implementation are associated with improved quality of care.
This study was limited to a group of selected physicians in Toronto. However, all physicians in this study were practising in community-based settings, similar to most family physician settings in Ontario.56 Physicians in our cohorts were similar to their colleagues in capitated and reformed fee–for-service models in Ontario urban centres.33
Nineteen percent of physicians using paper records responded to the study invitation; we do not know if their characteristics differed from those of the nonresponders. The participating physicians provided preventive services to a large proportion of their patients. Increases might have been limited by ceiling effects. Also, we studied a single EMR system; results might differ for physicians using other EMR systems.
This was an observational cohort study, and is therefore subject to both measured and unmeasured confounders. In this study, we measured confounders that have been reported in the literature to affect preventive services22–32,47,57,58 and used statistical adjustments to adjust for these factors. There were differences between our 2 cohorts in terms of physician funding mechanisms. However, a recent study using administrative data did not find any consistent difference in the provision of preventive services between Ontario physicians in capitated or reformed fee-for-service groups.59
Among the family physician practices studied, the first 2 years of EMR implementation were not associated with an increase in the provision of preventive services targeted by Ontario’s pay-for-performance program when compared with the continued use of paper records. It should not be assumed that EMR implementation improves care.
This study was funded by a research grant from the Ontario Ministry of Health and Long-Term Care Health System Strategy Division (grant no. 020709) and was supported by the Institute for Clinical Evaluative Sciences. The opinions, results, and conclusions reported in this paper are those of the authors and do not imply endorsement by the Ministry of Health and Long-Term Care or the Institute for Clinical Evaluative Sciences.
This article has been peer reviewed.
Cet article a fait l’objet d’une révision par des pairs.
All authors contributed to the concept and design of the study. Drs Greiver and Barnsley contributed to the data gathering. All authors contributed to the analysis and interpretation of the results and the preparation of the manuscript for publication.