|Home | About | Journals | Submit | Contact Us | Français|
The study objective is to determine whether transitioning from paper to electronic health records (EHR) impacted behavioral health screening rates in a large northeast pediatric practice.
The study setting is a pediatric practice with 7 pediatricians serving about 6000 patients. The population is diverse (54% non-white, 40% publicly insured or self-pay and 31% non-English speakers). An interrupted times series design (ITS) was used to evaluate the impact of EHR implementation on behavioral health screening rates. The main outcome measure was the rate of behavioral health screening 18 months before and 36 months after EHR implementation.
The rate of behavioral health screening increased from 70% to 91% during the baseline period. The training period (6 months) prior to EHR-based behavioral health screening was associated with a 28% decline in adjusted screening rates (84% to 64%). Only 50% of eligible youth were screened in the first month of EHR-based screening. It took more than three years for the screening rate to recover to baseline levels – climbing to 82% by April 2008.
Practice changes resulting from EHR adoption were highly disruptive to needed care and took several years to resolve completely. Transferring data input to medical assistants was necessary to improve compliance. It is likely that compliance with primary care HEDIS standards and Medicaid performance measures will be similarly affected as EHRs are implemented across the nation. While there are some benefits from implementing a fully automated medical record, the unintended effects on care following implementation must be acknowledged.
The American Recovery and Reinvestment Act of 2009 calls for the utilization of an electronic health record (EHR) for each person in the United States by 2014 (1). The legislation is based on the assumption that EHRs can improve the safety, timeliness, effectiveness, and efficiency of care (2). Health information technology has been shown to improve the quality of care in some circumstances. For example, embedded alert systems have increased the timeliness of childhood immunization and overall rates of immunization (3, 4) and an EHR add-on template for Attention Deficit Hyperactivity Disorder (ADHD) was reported to have increased follow-up visits, and documentation of symptoms (5). However, systematic reviews have concluded that the impact of EHRs on improving clinical outcomes is unproven (6-8).
The problems and harms associated with EHRs are also well documented. One report of the effects of computerized physician order entry (CPOE) showed an increase in pediatric mortality after a change in EHR software in a Pittsburg teaching hospital (9). Although a more recent time series analysis reported a decrease in hospital-wide mortality (10), this association may have been confounded by competing interventions (e.g., process and workflow changes, adjustments in ICU staffing, rollout of Rapid Response Teams) (11). Ash and colleagues have reported many unintended negative consequences of EHRs, such as increased time necessary to enter data, fragmentation of information across screens or tabs, inflexible processes, and physician overload (12-14). Several rigorous studies have reported no improvement in quality of care associated with the adoption of EHRs or even a deterioration in the quality of care (9, 15-19). It is also well known that transitioning from a paper to an EHR environment is associated with disruptions in workflow (20), increased patient cycle time (21), breakdowns in communication between clinicians (12), computer system crashes, and lack of communication between different EHR systems.
Despite the increasing evidence of unintended effects of EHRs (22), no extant studies have reported how specific pediatric screening or disease management processes may be affected by the transition from a paper to electronic environment. As EHR systems are implemented or modified across the country, clinicians and insurers may experience declines in the quality of care (as measured by compliance with mandated services) associated with the disruption of workflow and changes in protocols (17).
The purpose of this study was to determine whether the transition from paper to an electronic health record (EHR) impacted behavioral health screening in a large northeast pediatric practice setting. The increasing prevalence of mental illness in children and the scarcity of child mental health professionals suggest that pediatricians must improve detection and treatment of mental health disorders in primary care settings (23). National organizations are recommending such behavioral health screening in pediatric primary care to address this need (24-27). Behavioral health screens improve identification rates and referral numbers, and may impact timeliness of care (28-31). Behavioral health screening and related increases in referrals are also associated with improved behavioral health outcomes measured at subsequent preventive care visits (e.g., lower PSC scores) (32, 33). Thus, the positive and negative impacts of health information technology on behavioral health screening merit further attention. This study received approval from the Cambridge Health Alliance Institutional Review Board.
In 2002 a class action lawsuit Rosie D versus Romney (34) was filed in Massachusetts on behalf of eight children with serious psychiatric disabilities who were hospitalized or at risk for hospitalization due to the state’s failure to provide needed services to allow them to live at home and attend local schools. In 2007, the state was found in violation of the Medicaid Act (35). The court’s decision required that all primary care providers caring for MassHealth enrolled children aged less than 21 years, whose benefit package includes primary care service, begin using a clinically appropriate behavioral health screening tool at well-child visits.
At a large pediatric clinic site, behavioral health screening of children at well-child visits began in December 2003 – 5 years prior to the initiation of the mandatory statewide program. The clinic, which is part of a large public healthcare system, has 7 pediatricians and serves about 6000 patients of whom about 33% are under age 5. On average 180 children are eligible for behavioral health screening each month. While this is an academic teaching site, pediatricians generally are scheduled to see approximately 25 patients a day in 15 minute intervals. The population is economically and linguistically diverse (54% non-white, 40% publically insured or self-pay and 31% non-English speakers).
Screening was conducted using the Pediatric Symptom Checklist (PSC) (ages 4 years 11 months to age 16) and the Youth-PSC (Y-PSC) (ages 14-18) both of which are validated tools that have been translated into multiple languages (36, 37). On both tools, summary scores above a certain cut-point are considered positive screens for behavioral difficulties.
The screening process began with families/teens filling out the paper PSC/Y-PSC in the waiting room prior to the visit. They would then bring the paper form to the physician who would discuss it with them and score the instrument. Prior to implementation of the EHR, providers inserted the completed paper form into the medical record. Over the course of the first year of screening, nurse management and clinical leadership developed processes to enhance screening. Management reviewed charts the night before to identify children who were eligible for screening visits. Then they attached the appropriate PSC/Y-PSC to the chart for distribution by the receptionist when the patient arrived. In addition, the nurse manager was given regular updates on compliance and worked with front office staff to improve distribution while working with medical assistants to retrieve forms.
The go-live date for using the EHR (EPICare system) at this practice was March 29th 2005 but planning for implementation began six months earlier. Approximately 2 months before the go-live date, all staff went to full EPIC off-site training (40 hours each). Training occurred in a rolling manner with different elements so as not to overwhelm staff. To ease transition, patient visits were reduced by 50% in the first week and 25% in the second week after EHR implementation so providers could adjust to the EHR. There were also in-house trainers on site to help with implementation. In order to capture screening information electronically, a questionnaire for entering paper-based PSC data was programmed into the EHR in June 2005.
The PSC screening process remained the same using paper forms throughout; however, once the practice began using the EHR the results of the PSC were entered into the EHR via an electronic questionnaire with 4 required fields. These fields included the total score, responses to two questions; 1) Is your child (you) currently seeing a mental health counselor? 2) Does your child (you) have any emotional or behavioral problems for which she/he needs help? and indication of disposition (annual visit, return to primary care, referral to counseling). Initially, this was done by the physicians themselves and later transferred to medical assistants who also scanned the actual PSC form into the system. Reports were developed to provide monthly compliance rates by individual physicians to assist management with process improvement.
To measure other contextual factors that might have impacted screening rates and EHR transition, a review of past email communication, minutes of implementation meetings, and reports was conducted and discussions were held with two practice administrators to capture any environmental changes (such as staffing, scheduling, operations, and training information) which took place during the transitional period. In addition, these individuals were asked to describe the major challenges to EHR implementation.
To determine rates of screens, we drew data on preventive care eligible visits from the Meditech system monthly. Until March 2007, copies of PSCs were collected weekly and data was entered into a data base. The crude screening rate was calculated by dividing the number of paper screens by the number of eligible visits. Once the practice began using the electronic questionnaire, a monthly compliance report was also generated from the Meditech system to determine screening results in the EHR and identify the number of screens by provider by month. For the purposes of this study, we used the highest number of screens reported from either paper or electronic data until March 2007. Data was only available from the EHR after March 2007.
Interrupted times series (ITS) was used to evaluate the impact of EHR implementation on BH screening rates.(38) ITS is the strongest quasi experimental design for evaluating the effects of natural experiments.(39) Controlling for the baseline rate and trend of behavioral health screening, changes in the level and slope of the screening rate during the EHR planning period and post implementation were estimated. The regression equation was specified as:
We used maximum likelihood estimation to estimate the models and controlled for autocorrelation by including all significant autocorrelation parameters up to 12 months. Backward elimination was used to include covariates in the final model with a type 1 error threshold for inclusion of 0.05. All statistical analyses were performed using PROC AUTOREG in SAS 9.2 (40).
Figure 1 shows the baseline, training, and post-EHR implementation BH screening rates. The rate of screening increased during the baseline period, beginning at just below 70% in December 2003 and climbing to 91% in September 2004. The average screening rate during the baseline period was 81%. In the 6 month training period (October 2004 to March 2005) prior to EHR implementation, rates of screening dropped precipitously and significantly from 84% to 64% (p<.001). The switch from paper to EHR on March 29 of 2005 was associated with a further drop in screening rates. Only 50% of eligible youth were screened in the first month of EHR-based records. Rates recovered slowly over three years – climbing to 82% by April 2008, matching baseline rates. The monthly behavioral health screening rate also became much more variable after EHR implementation.
Table 1 shows the results of the segmented regression analysis. Both the time after training and time after implementation trend coefficients were significant. The 6 month training period was associated with a 4.9% monthly decrease in the adjusted behavioral health screening rate (p=.002). Behavioral health screening rates increased by 3.3% monthly (p = .006) between May 2005 and April 2008.
In our review of email, minutes, reports and conversations with practice management, we identified several important challenges occurring simultaneously to EPIC implementation. First, two months prior to implementation, the practice experienced a transition in nurse management leadership as well as several other personnel. This individual had been leading the EHR roll-out and with her departure several others became more involved with the implementation.
Second, according to staff, transitioning from paper to the EHR proved difficult. After EHR transition, the. PSC form was the only paper form which was handed over to parents of children coming in for their well child visits. Front office staff tended to forget to hand out the PSC forms particularly on days when there were few eligible visits. In addition, identifying children who met criteria was sometimes confusing. Thus, the percent of hard copy PSCs collected for eligible visits dropped substantially as seen in Figure 1. Later, health assistants were asked to back up front office staff and get a form if the parent was not carrying one into the exam room.
On the provider side, many providers lacked computer skills and had trouble keeping up with their work load during the transition. The additional burden of inputting PSC data into the EHR questionnaire proved problematic. As the medical director of the clinic noted “This computerized questionnaire is new to us here …, so I am not too surprised the completion rate is so poor”. Later, it became apparent that providers were entering the information via addendums after the visits were closed and it was eventually decided that the medical assistants should be responsible for inputting the PSC into the EHR questionnaire. This process significantly improved screening documentation. With the implementation of the Rosie D mandate (January 2008), additional changes were made for the purpose of billing including scanning forms and using specific billing codes.
Implementing an EHR is a complex process that demands attentive leadership engaged in ongoing solutions as providers adjust to the change (20, 41). In addition, the impact of this transition on provider behavior, practice operations, and administrative procedures cannot be underestimated.
In this study, we observed two negative phases of the changeover to EHR. First, the process of transitioning to an EHR, or pre-implementation, impacted actual screening practices—that is, fewer eligible children were actually screened in the months leading up to EHR implementation. We believe that this was due to both the preparation required for the transition (e.g. off-site training for all staff, and extraction of medical record data) and the changes in nursing management. Preparatory efforts were likely to distract staff towards EHR-related issues and have a cumulative impact during the roll-out period. However, a leadership void may have contributed to a progressive effect given that providers also lost administrative feedback on screening rates.
The second phase of screening decline was during the “go” implementation phase. Factors impacting this phase include a host of organizational changes necessary to adjust to the EHR. For example, the EHR changed front desk procedures which may have impacted the patient’s receipt of the PSC. Previously, children coming for well-child care were identified by the medical record forms but once the EHR went live, this was switched to a paperless system and the visual cue was lost. Systems had to be developed to identify which children needed a BH screen and these systems were monitored for improvement. Implementation proceeded in this trial and error manner until improvement was seen. Clinicians should think very carefully about how cues and alerts will be managed/maintained during the planning/training phase of EHR implementation. Whatever conventions are developed to identify the particular needs of individual patients, it is likely that these will need to be customized to the workflow of each practice. This challenges the assumptions of a national approach to EHR implementation in which commercial systems are used that are difficult to adapt to local practice styles or needs. This was a major reason for the failure and abandonment of the UK’s national health IT program (42).
Second, the process of shifting data capture tasks from paper to an EHR resulted in poorer documentation because some paper-based PSCs were not being entered into the EHR. It was only after processes were adapted including moving documentation responsibilities from physicians to medical assistants that documentation rates improved slowly over three years. Medical assistants may be particularly important in maintaining patient flow and satisfaction during EHR implementation (43, 44). It is our guess that handoff breakdowns occurred at many levels: between front desk and patient; between patient and physician, and between physician and EHR. Further research is needed to identify the actual step by step influences that this transition has on a practice setting and documentation protocols.
The persistence of reduced screening rates at this practice also appears to be consistent with the findings of previous studies. As noted in this study, the negative impact of transitioning from a paper to an EHR environment on workflow (20), increased patient cycle time (21), and communication breakdowns between clinicians (12) have been documented elsewhere. Bloom and Huntington (45) reported that the average time spent documenting during a clinical visit did not decrease between initial EHR implementation and follow-up 12 months later. Similarly, Samaan and colleagues (21) found that patient cycle time did not return to baseline for 2 and a half years following EHR implementation. Our findings further support the evidence that switching to an EHR has a major impact on a practice and recovery time may require up to 3 years of adaptation.
It is possible that transitional issues could have been avoided if the IT system had been capable of providing information about which children required screens and whether screening had been done within a prescribed time frame. A well constructed IT program should be able to prompt patients or patient’s parents to complete the screening directly into an automated system and monitor completion. IT systems that require the type of human effort noted in this implementation are doomed to degradation, at least temporarily, until staff learns how to make up for the system’s deficiencies. Unfortunately, the costs and the multiple demands for IT programming during EHR implementation, make the “perfect” IT system elusive.
The results of this analysis may serve as a warning for what may happen to health care quality as EHRs are rolled-out across the country. Quality of care (as measured by compliance rates) may decrease as clinicians and staff adapt to new information technology. Reducing the workload of clinicians during EHR implementation may not be enough to avoid lapses in providing mandatory care. In the practice we studied, EHRs changed how patients were identified for particular services, visual cues to prompt tasks, and processes by which information was entered into a medical record.
This study of a single heterogeneous practice comprised of only seven pediatricians in the Northeast may not be generalizable to other practices elsewhere. However, we believe that the issues encountered may be generalizable as practices nationwide transition to EHRs and few investigators have studied the unintended outcomes of EHRs (22). In addition, there was no control group available. It is therefore possible that other factors may have influenced how screening was being conducted during this time; however, this was not what practice administrators reported.
The declines in behavioral health screening resulting from the EHR adoption in the study practice were highly disruptive and took almost three years to resolve completely. It is likely that compliance with primary care HEDIS standards and Medicaid performance measures will be similarly affected. While there may be some benefits from implementing a fully automated medical record, the challenges inherent in implementation must be acknowledged. The effects of EHRs on patient behavioral health outcomes is still unclear.
The authors would like to acknowledge Dr. David Link, MD, chair of the Department of Pediatrics at the Cambridge Health Alliance and provider at Cambridge Pediatrics and the clinic staff who provided detailed information on EPIC implementation.
This study was supported in part by a grant through The Integrated Health and Behavioral Health Care for Children, Adolescents and Their Families grant program in HRSA’s Maternal and Child Health Bureau. Partial support also came from the Harvard Pilgrim Health Care Foundation, the National Institute of Mental Health and Health Maintenance Organization Mental Health Research Network (grant 1U19MH092201-01), and the Health Maintenance Organization Research Network Centers for Education and Research on Therapeutics, supported by the Agency for Healthcare Research and Quality (Grant No. U18HS010391) and the Harvard Pilgrim Health Care Institute.
Additional consultative support also came from Harvard Catalyst; The Harvard Clinical and Translational Science Center (NIH Award #UL1 RR 025758 and financial contributions from Harvard University and its affiliated academic health care centers). The content is solely the responsibility of the authors and does not necessarily represent the official views of Harvard Catalyst, Harvard University and its affiliated academic health care centers, the National Center for Research Resources, or the National Institutes of Health.
Disclosures of interest : None of the authors have any conflicts of interest
Robert Penfold, Group Health Research Institute & Department of Health Services Research, University of Washington.
Fang Zhang, Department of Population Medicine Harvard Medical School & Harvard Pilgrim Health Care Institute.
Stephen Soumerai, Department of Population Medicine Harvard Medical School & Harvard Pilgrim Health Care Institute.