PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Psychiatr Serv. Author manuscript; available in PMC 2013 July 8.
Published in final edited form as:
PMCID: PMC3704213
NIHMSID: NIHMS486119

Impact of Electronic Health Record Transition on Behavioral Health Screening

Abstract

Objective

The study objective is to determine whether transitioning from paper to electronic health records (EHR) impacted behavioral health screening rates in a large northeast pediatric practice.

Methods

The study setting is a pediatric practice with 7 pediatricians serving about 6000 patients. The population is diverse (54% non-white, 40% publicly insured or self-pay and 31% non-English speakers). An interrupted times series design (ITS) was used to evaluate the impact of EHR implementation on behavioral health screening rates. The main outcome measure was the rate of behavioral health screening 18 months before and 36 months after EHR implementation.

Results

The rate of behavioral health screening increased from 70% to 91% during the baseline period. The training period (6 months) prior to EHR-based behavioral health screening was associated with a 28% decline in adjusted screening rates (84% to 64%). Only 50% of eligible youth were screened in the first month of EHR-based screening. It took more than three years for the screening rate to recover to baseline levels – climbing to 82% by April 2008.

Conclusions

Practice changes resulting from EHR adoption were highly disruptive to needed care and took several years to resolve completely. Transferring data input to medical assistants was necessary to improve compliance. It is likely that compliance with primary care HEDIS standards and Medicaid performance measures will be similarly affected as EHRs are implemented across the nation. While there are some benefits from implementing a fully automated medical record, the unintended effects on care following implementation must be acknowledged.

Keywords: Electronic health record, behavioral health screening, interrupted time series

Introduction

The American Recovery and Reinvestment Act of 2009 calls for the utilization of an electronic health record (EHR) for each person in the United States by 2014 (1). The legislation is based on the assumption that EHRs can improve the safety, timeliness, effectiveness, and efficiency of care (2). Health information technology has been shown to improve the quality of care in some circumstances. For example, embedded alert systems have increased the timeliness of childhood immunization and overall rates of immunization (3, 4) and an EHR add-on template for Attention Deficit Hyperactivity Disorder (ADHD) was reported to have increased follow-up visits, and documentation of symptoms (5). However, systematic reviews have concluded that the impact of EHRs on improving clinical outcomes is unproven (6-8).

The problems and harms associated with EHRs are also well documented. One report of the effects of computerized physician order entry (CPOE) showed an increase in pediatric mortality after a change in EHR software in a Pittsburg teaching hospital (9). Although a more recent time series analysis reported a decrease in hospital-wide mortality (10), this association may have been confounded by competing interventions (e.g., process and workflow changes, adjustments in ICU staffing, rollout of Rapid Response Teams) (11). Ash and colleagues have reported many unintended negative consequences of EHRs, such as increased time necessary to enter data, fragmentation of information across screens or tabs, inflexible processes, and physician overload (12-14). Several rigorous studies have reported no improvement in quality of care associated with the adoption of EHRs or even a deterioration in the quality of care (9, 15-19). It is also well known that transitioning from a paper to an EHR environment is associated with disruptions in workflow (20), increased patient cycle time (21), breakdowns in communication between clinicians (12), computer system crashes, and lack of communication between different EHR systems.

Despite the increasing evidence of unintended effects of EHRs (22), no extant studies have reported how specific pediatric screening or disease management processes may be affected by the transition from a paper to electronic environment. As EHR systems are implemented or modified across the country, clinicians and insurers may experience declines in the quality of care (as measured by compliance with mandated services) associated with the disruption of workflow and changes in protocols (17).

The purpose of this study was to determine whether the transition from paper to an electronic health record (EHR) impacted behavioral health screening in a large northeast pediatric practice setting. The increasing prevalence of mental illness in children and the scarcity of child mental health professionals suggest that pediatricians must improve detection and treatment of mental health disorders in primary care settings (23). National organizations are recommending such behavioral health screening in pediatric primary care to address this need (24-27). Behavioral health screens improve identification rates and referral numbers, and may impact timeliness of care (28-31). Behavioral health screening and related increases in referrals are also associated with improved behavioral health outcomes measured at subsequent preventive care visits (e.g., lower PSC scores) (32, 33). Thus, the positive and negative impacts of health information technology on behavioral health screening merit further attention. This study received approval from the Cambridge Health Alliance Institutional Review Board.

PATIENTS AND METHODS

Setting

In 2002 a class action lawsuit Rosie D versus Romney (34) was filed in Massachusetts on behalf of eight children with serious psychiatric disabilities who were hospitalized or at risk for hospitalization due to the state’s failure to provide needed services to allow them to live at home and attend local schools. In 2007, the state was found in violation of the Medicaid Act (35). The court’s decision required that all primary care providers caring for MassHealth enrolled children aged less than 21 years, whose benefit package includes primary care service, begin using a clinically appropriate behavioral health screening tool at well-child visits.

At a large pediatric clinic site, behavioral health screening of children at well-child visits began in December 2003 – 5 years prior to the initiation of the mandatory statewide program. The clinic, which is part of a large public healthcare system, has 7 pediatricians and serves about 6000 patients of whom about 33% are under age 5. On average 180 children are eligible for behavioral health screening each month. While this is an academic teaching site, pediatricians generally are scheduled to see approximately 25 patients a day in 15 minute intervals. The population is economically and linguistically diverse (54% non-white, 40% publically insured or self-pay and 31% non-English speakers).

Screening was conducted using the Pediatric Symptom Checklist (PSC) (ages 4 years 11 months to age 16) and the Youth-PSC (Y-PSC) (ages 14-18) both of which are validated tools that have been translated into multiple languages (36, 37). On both tools, summary scores above a certain cut-point are considered positive screens for behavioral difficulties.

The screening process began with families/teens filling out the paper PSC/Y-PSC in the waiting room prior to the visit. They would then bring the paper form to the physician who would discuss it with them and score the instrument. Prior to implementation of the EHR, providers inserted the completed paper form into the medical record. Over the course of the first year of screening, nurse management and clinical leadership developed processes to enhance screening. Management reviewed charts the night before to identify children who were eligible for screening visits. Then they attached the appropriate PSC/Y-PSC to the chart for distribution by the receptionist when the patient arrived. In addition, the nurse manager was given regular updates on compliance and worked with front office staff to improve distribution while working with medical assistants to retrieve forms.

Practice changes during EHR implementation

The go-live date for using the EHR (EPICare system) at this practice was March 29th 2005 but planning for implementation began six months earlier. Approximately 2 months before the go-live date, all staff went to full EPIC off-site training (40 hours each). Training occurred in a rolling manner with different elements so as not to overwhelm staff. To ease transition, patient visits were reduced by 50% in the first week and 25% in the second week after EHR implementation so providers could adjust to the EHR. There were also in-house trainers on site to help with implementation. In order to capture screening information electronically, a questionnaire for entering paper-based PSC data was programmed into the EHR in June 2005.

The PSC screening process remained the same using paper forms throughout; however, once the practice began using the EHR the results of the PSC were entered into the EHR via an electronic questionnaire with 4 required fields. These fields included the total score, responses to two questions; 1) Is your child (you) currently seeing a mental health counselor? 2) Does your child (you) have any emotional or behavioral problems for which she/he needs help? and indication of disposition (annual visit, return to primary care, referral to counseling). Initially, this was done by the physicians themselves and later transferred to medical assistants who also scanned the actual PSC form into the system. Reports were developed to provide monthly compliance rates by individual physicians to assist management with process improvement.

Measures

To measure other contextual factors that might have impacted screening rates and EHR transition, a review of past email communication, minutes of implementation meetings, and reports was conducted and discussions were held with two practice administrators to capture any environmental changes (such as staffing, scheduling, operations, and training information) which took place during the transitional period. In addition, these individuals were asked to describe the major challenges to EHR implementation.

To determine rates of screens, we drew data on preventive care eligible visits from the Meditech system monthly. Until March 2007, copies of PSCs were collected weekly and data was entered into a data base. The crude screening rate was calculated by dividing the number of paper screens by the number of eligible visits. Once the practice began using the electronic questionnaire, a monthly compliance report was also generated from the Meditech system to determine screening results in the EHR and identify the number of screens by provider by month. For the purposes of this study, we used the highest number of screens reported from either paper or electronic data until March 2007. Data was only available from the EHR after March 2007.

Analysis

Interrupted times series (ITS) was used to evaluate the impact of EHR implementation on BH screening rates.(38) ITS is the strongest quasi experimental design for evaluating the effects of natural experiments.(39) Controlling for the baseline rate and trend of behavioral health screening, changes in the level and slope of the screening rate during the EHR planning period and post implementation were estimated. The regression equation was specified as:

Yt=β0+β1timet*+β2trainingt*+β3time after trainingt*+β4EHR implementationt*+β5time after EHRt*+et
(1)

We used maximum likelihood estimation to estimate the models and controlled for autocorrelation by including all significant autocorrelation parameters up to 12 months. Backward elimination was used to include covariates in the final model with a type 1 error threshold for inclusion of 0.05. All statistical analyses were performed using PROC AUTOREG in SAS 9.2 (40).

RESULTS

BH screening rates

Figure 1 shows the baseline, training, and post-EHR implementation BH screening rates. The rate of screening increased during the baseline period, beginning at just below 70% in December 2003 and climbing to 91% in September 2004. The average screening rate during the baseline period was 81%. In the 6 month training period (October 2004 to March 2005) prior to EHR implementation, rates of screening dropped precipitously and significantly from 84% to 64% (p<.001). The switch from paper to EHR on March 29 of 2005 was associated with a further drop in screening rates. Only 50% of eligible youth were screened in the first month of EHR-based records. Rates recovered slowly over three years – climbing to 82% by April 2008, matching baseline rates. The monthly behavioral health screening rate also became much more variable after EHR implementation.

Figure 1
Trends in the Pediatric Behavioral Health Screening Rate During the Baseline, Training, and EHR Periods in a Large Massachusetts Practice: December 2003 to May 2008

Table 1 shows the results of the segmented regression analysis. Both the time after training and time after implementation trend coefficients were significant. The 6 month training period was associated with a 4.9% monthly decrease in the adjusted behavioral health screening rate (p=.002). Behavioral health screening rates increased by 3.3% monthly (p = .006) between May 2005 and April 2008.

Table 1
Segmented Regression Coefficients

Environmental changes

In our review of email, minutes, reports and conversations with practice management, we identified several important challenges occurring simultaneously to EPIC implementation. First, two months prior to implementation, the practice experienced a transition in nurse management leadership as well as several other personnel. This individual had been leading the EHR roll-out and with her departure several others became more involved with the implementation.

Second, according to staff, transitioning from paper to the EHR proved difficult. After EHR transition, the. PSC form was the only paper form which was handed over to parents of children coming in for their well child visits. Front office staff tended to forget to hand out the PSC forms particularly on days when there were few eligible visits. In addition, identifying children who met criteria was sometimes confusing. Thus, the percent of hard copy PSCs collected for eligible visits dropped substantially as seen in Figure 1. Later, health assistants were asked to back up front office staff and get a form if the parent was not carrying one into the exam room.

On the provider side, many providers lacked computer skills and had trouble keeping up with their work load during the transition. The additional burden of inputting PSC data into the EHR questionnaire proved problematic. As the medical director of the clinic noted “This computerized questionnaire is new to us here …, so I am not too surprised the completion rate is so poor”. Later, it became apparent that providers were entering the information via addendums after the visits were closed and it was eventually decided that the medical assistants should be responsible for inputting the PSC into the EHR questionnaire. This process significantly improved screening documentation. With the implementation of the Rosie D mandate (January 2008), additional changes were made for the purpose of billing including scanning forms and using specific billing codes.

DISCUSSION

Implementing an EHR is a complex process that demands attentive leadership engaged in ongoing solutions as providers adjust to the change (20, 41). In addition, the impact of this transition on provider behavior, practice operations, and administrative procedures cannot be underestimated.

In this study, we observed two negative phases of the changeover to EHR. First, the process of transitioning to an EHR, or pre-implementation, impacted actual screening practices—that is, fewer eligible children were actually screened in the months leading up to EHR implementation. We believe that this was due to both the preparation required for the transition (e.g. off-site training for all staff, and extraction of medical record data) and the changes in nursing management. Preparatory efforts were likely to distract staff towards EHR-related issues and have a cumulative impact during the roll-out period. However, a leadership void may have contributed to a progressive effect given that providers also lost administrative feedback on screening rates.

The second phase of screening decline was during the “go” implementation phase. Factors impacting this phase include a host of organizational changes necessary to adjust to the EHR. For example, the EHR changed front desk procedures which may have impacted the patient’s receipt of the PSC. Previously, children coming for well-child care were identified by the medical record forms but once the EHR went live, this was switched to a paperless system and the visual cue was lost. Systems had to be developed to identify which children needed a BH screen and these systems were monitored for improvement. Implementation proceeded in this trial and error manner until improvement was seen. Clinicians should think very carefully about how cues and alerts will be managed/maintained during the planning/training phase of EHR implementation. Whatever conventions are developed to identify the particular needs of individual patients, it is likely that these will need to be customized to the workflow of each practice. This challenges the assumptions of a national approach to EHR implementation in which commercial systems are used that are difficult to adapt to local practice styles or needs. This was a major reason for the failure and abandonment of the UK’s national health IT program (42).

Second, the process of shifting data capture tasks from paper to an EHR resulted in poorer documentation because some paper-based PSCs were not being entered into the EHR. It was only after processes were adapted including moving documentation responsibilities from physicians to medical assistants that documentation rates improved slowly over three years. Medical assistants may be particularly important in maintaining patient flow and satisfaction during EHR implementation (43, 44). It is our guess that handoff breakdowns occurred at many levels: between front desk and patient; between patient and physician, and between physician and EHR. Further research is needed to identify the actual step by step influences that this transition has on a practice setting and documentation protocols.

The persistence of reduced screening rates at this practice also appears to be consistent with the findings of previous studies. As noted in this study, the negative impact of transitioning from a paper to an EHR environment on workflow (20), increased patient cycle time (21), and communication breakdowns between clinicians (12) have been documented elsewhere. Bloom and Huntington (45) reported that the average time spent documenting during a clinical visit did not decrease between initial EHR implementation and follow-up 12 months later. Similarly, Samaan and colleagues (21) found that patient cycle time did not return to baseline for 2 and a half years following EHR implementation. Our findings further support the evidence that switching to an EHR has a major impact on a practice and recovery time may require up to 3 years of adaptation.

It is possible that transitional issues could have been avoided if the IT system had been capable of providing information about which children required screens and whether screening had been done within a prescribed time frame. A well constructed IT program should be able to prompt patients or patient’s parents to complete the screening directly into an automated system and monitor completion. IT systems that require the type of human effort noted in this implementation are doomed to degradation, at least temporarily, until staff learns how to make up for the system’s deficiencies. Unfortunately, the costs and the multiple demands for IT programming during EHR implementation, make the “perfect” IT system elusive.

The results of this analysis may serve as a warning for what may happen to health care quality as EHRs are rolled-out across the country. Quality of care (as measured by compliance rates) may decrease as clinicians and staff adapt to new information technology. Reducing the workload of clinicians during EHR implementation may not be enough to avoid lapses in providing mandatory care. In the practice we studied, EHRs changed how patients were identified for particular services, visual cues to prompt tasks, and processes by which information was entered into a medical record.

Limitations

This study of a single heterogeneous practice comprised of only seven pediatricians in the Northeast may not be generalizable to other practices elsewhere. However, we believe that the issues encountered may be generalizable as practices nationwide transition to EHRs and few investigators have studied the unintended outcomes of EHRs (22). In addition, there was no control group available. It is therefore possible that other factors may have influenced how screening was being conducted during this time; however, this was not what practice administrators reported.

CONCLUSION

The declines in behavioral health screening resulting from the EHR adoption in the study practice were highly disruptive and took almost three years to resolve completely. It is likely that compliance with primary care HEDIS standards and Medicaid performance measures will be similarly affected. While there may be some benefits from implementing a fully automated medical record, the challenges inherent in implementation must be acknowledged. The effects of EHRs on patient behavioral health outcomes is still unclear.

Acknowledgements

The authors would like to acknowledge Dr. David Link, MD, chair of the Department of Pediatrics at the Cambridge Health Alliance and provider at Cambridge Pediatrics and the clinic staff who provided detailed information on EPIC implementation.

This study was supported in part by a grant through The Integrated Health and Behavioral Health Care for Children, Adolescents and Their Families grant program in HRSA’s Maternal and Child Health Bureau. Partial support also came from the Harvard Pilgrim Health Care Foundation, the National Institute of Mental Health and Health Maintenance Organization Mental Health Research Network (grant 1U19MH092201-01), and the Health Maintenance Organization Research Network Centers for Education and Research on Therapeutics, supported by the Agency for Healthcare Research and Quality (Grant No. U18HS010391) and the Harvard Pilgrim Health Care Institute.

Additional consultative support also came from Harvard Catalyst; The Harvard Clinical and Translational Science Center (NIH Award #UL1 RR 025758 and financial contributions from Harvard University and its affiliated academic health care centers). The content is solely the responsibility of the authors and does not necessarily represent the official views of Harvard Catalyst, Harvard University and its affiliated academic health care centers, the National Center for Research Resources, or the National Institutes of Health.

Abbreviations used

CPOE
Computerized Physician Order Entry
CHA
Cambridge Health Alliance
EHR
electronic health record
ITS
interrupted time series
BH
behavioral health
PSC
Pediatric Symptom Checklist

Footnotes

Disclosures of interest : None of the authors have any conflicts of interest

Contributor Information

Robert Penfold, Group Health Research Institute & Department of Health Services Research, University of Washington.

Fang Zhang, Department of Population Medicine Harvard Medical School & Harvard Pilgrim Health Care Institute.

Stephen Soumerai, Department of Population Medicine Harvard Medical School & Harvard Pilgrim Health Care Institute.

References

1. American Recovery and Reinvestment Act of 2009. Feb 17, 2009. Pub. L. No. 111-5, Stat. 123. 2009.
2. Corrigan JM, Donaldson MS, Kohn LT, et al. Crossing the Quality Chasm: A New Health System for the 21st Century. Institute of Medicine, National Academy of Sciences, National Academy Press; Washington, D.C.: 2001.
3. Fiks AG, Grundmeier RW, Biggs LM, et al. Impact of clinical alerts within an electronic health record on routine childhood immunization in an urban pediatric population. Pediatrics. 2007;120:707–14. [PubMed]
4. Fiks AG, Hunter KF, Localio AR, et al. Impact of electronic health record-based alerts on influenza vaccination for children with asthma. Pediatrics. 2009;124:159–69. [PubMed]
5. Co JP, Johnson SA, Poon EG, et al. Electronic health record decision support and quality of care for children with ADHD. Pediatrics. 2010;126:239–46. [PubMed]
6. Black AD, Car J, Pagliari C, et al. The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview. PLoS Medicine. 2011;8:e1000387. [PMC free article] [PubMed]
7. Eslami S, Abu-Hanna A, de Keizer NF. Evaluation of Outpatient Computerized Physician Medication Order Entry Systems: A Systematic Review. Journal of the American Medical Informatics Association. 2007;14:400–6. [PMC free article] [PubMed]
8. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. Journal of the American Medical Association. 2005;293:1223–38. [PubMed]
9. Han YY, Carcillo JA, Venkataraman ST, et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics. 2005;116:1506–12. [PubMed]
10. Longhurst CA, Parast L, Sandborg CI, et al. Decrease in hospital-wide mortality rate after implementation of a commercially sold computerized physician order entry system. Pediatrics. 2010;126:14–21. [PubMed]
11. Merrill M. Study correlates CPOE with decreased mortality rates. 2011 May 3; May 3, 2010. Available from: http://www.healthcareitnews.com/news/study-correlates-cpoe-decreased-mortality-rates.
12. Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. Journal of the American Medical Informatics Association. 2004;11:104–12. [PMC free article] [PubMed]
13. Ash JS, Sittig DF, Campbell EM, et al. Some unintended consequences of clinical decision support systems. American Medical Informatics Association Annual Symposium Proceedings. 2007:26–30. [PMC free article] [PubMed]
14. Ash JS, Sittig DF, Dykstra R, et al. Exploring the unintended consequences of computerized physician order entry. Studies in Health Technology and Informatics. 2007;129:198–202. [PubMed]
15. Aarts J, Koppel R. Implementation of computerized physician order entry in seven countries. Health Affairs (Millwood) 2009;28:404–14. [PubMed]
16. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors. Journal of the American Medical Association. 2005;293:1197–203. [PubMed]
17. Poissant L, Pereira J, Tamblyn R, et al. The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. Journal of the American Medical Informatics Association. 2005;12:505–16. [PMC free article] [PubMed]
18. Linder JA, Ma J, Bates DW, et al. Electronic health record use and the quality of ambulatory care in the United States. Archives of Internal Medicine. 2007;167:1400–5. [PubMed]
19. Poon EG, Wright A, Simon SR, et al. Relationship between use of electronic health record features and health care quality: results of a statewide survey. Medical Care. 2010;48:203–9. [PubMed]
20. McAlearney AS, Robbins J, Hirsch A, et al. Perceived efficiency impacts following electronic health record implementation: An exploratory study of an urban community health center network. International Journal of Medical Informatics. 2010;79:807–16. [PubMed]
21. Samaan ZM, Klein MD, Mansour ME, et al. The impact of the electronic health record on an academic pediatric primary care center. Journal of Ambulatory Care Management. 2009;32:180–7. [PubMed]
22. Bloomrosen M, Starren J, Lorenzi NM, et al. Anticipating and addressing the unintended consequences of health IT and policy: a report from the AMIA 2009 Health Policy Meeting. Journal of the American Medical Informatics Association. 2011;18:82–90. [PMC free article] [PubMed]
23. National Institute for Health Care Management Strategies to Support the Integraton of Mental Health into Pediatric Primary Care. 2010 Dec 3; Issue Paper. 2009. Available from: http://nihcm.org/pdf/PediatricMH-FINAL.pdf.
24. Suicide and suicide attempts in adolescents. Committee on Adolescents. American Academy of Pediatrics. Pediatrics. 2000;105:871–4. [PubMed]
25. New Freedom Commission on Mental Health . Achieving the Promise: Transforming Mental Health Care in America. Final Report. Substance Abuse and Mental Health Services Administration; Rockville, MD: 2003. DHHS Pub. No. SMA-03-3832.
26. Brugman E, Reijneveld SA, Verhulst FC, et al. Identification and management of psychosocial problems by preventive child health care. Archives of Pediatrics & Adolescent Medicine. 2001;155:462–9. [PubMed]
27. Murphy JM, Arnett HL, Bishop SJ, et al. Screening for psychosocial dysfunction in pediatric practice. A naturalistic study of the Pediatric Symptom Checklist. Clinical Pediatrics (Phila) 1992;31:660–7. [PubMed]
28. Chisolm DJ, Klima J, Gardner W, et al. Adolescent behavioral risk screening and use of health services. Administration and Policy in Mental Health. 2009;36:374–80. [PubMed]
29. Gardner W, Klima J, Chisolm D, et al. Screening, triage, and referral of patients who report suicidal thought during a primary care visit. Pediatrics. 2010;125:945–52. [PubMed]
30. Williams SB, O’Connor E, Eder M, et al. 2010/08/20 ed: Agency for Healthcare Research and Quality (US) 2009. Screening for Child and Adolescent Depression in Primary Care Settings: A Systematic Evidence Review for the U.S. Preventive Services Task Force. [PubMed]
31. Hacker KA, Myagmarjav E, Harris V, et al. Mental health screening in pediatric practice: factors related to positive screens and the contribution of parental/personal concern. Pediatrics. 2006;118:1896–906. [PubMed]
32. Hacker KA, Arsenault LN, Williams S, et al. Mental and behavioral health screening at preventive visits: opportunities for follow-up of patients who are nonadherent with the next preventive visit. The Journal of Pediatrics. 2011;158:666–71. [PubMed]
33. Hacker KA, Williams S, Myagmarjav E, et al. Persistence and change in pediatric symptom checklist scores over 10 to 18 months. Academic Pediatrics. 2009;9:270–7. [PubMed]
34. Rosie D. v. Romney, Stat. Supp. 2d 18 (D. Mass, 2006).
35. Social Security Act. Title XIX - Grants to States for Medical Assistance Programs. 1965
36. Navon M, Nelson D, Pagano M, et al. Use of the pediatric symptom checklist in strategies to improve preventive behavioral health care. Psychiatric Services. 2001;52:800–4. [PMC free article] [PubMed]
37. Vogels AG, Crone MR, Hoekstra F, et al. Comparing three short questionnaires to detect psychosocial dysfunction among primary school children: a randomized method. BMC Public Health. 2009;9:489. [PMC free article] [PubMed]
38. Wagner AK, Soumerai SB, Zhang F, et al. Segmented regression analysis of interrupted time series studies in medication use research. J Clin Pharm Ther. 2002;27:299–309. [PubMed]
39. Shadish W, Cook T, Campbell D. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin Company; Boston: 2002.
40. SAS Institute Inc. SAS OnlineDoc, Version 9.2. SAS Institute Inc.; Cary, NC: 2002-2006.
41. Castillo VH, Martinez-Garcia AI, Pulido JR. A knowledge-based taxonomy of critical factors for adopting electronic health record systems by physicians: a systematic literature review. BMC Medical Informatics and Decision Making. 2010;10:60. [PMC free article] [PubMed]
42. Soumerai SB, Avery T. Don’t Repeat the UK’s Electronic Health Records Failure. 2011 Feb 2; 2010. Available from: http://www.huffingtonpost.com/stephen-soumerai/dont-repeat-the-uks-elect_b_790470.html.
43. Nelson K, Pitaro M, Tzellas A, et al. Practice profile. Transforming the role of medical assistants in chronic disease management. Health Affairs (Millwood) 2010;29:963–5. [PubMed]
44. Tache S, Hill-Sakurai L. Medical assistants: the invisible “glue” of primary health care practices in the United States? Journal of Health Organization and Management. 2010;24:288–305. [PubMed]
45. Bloom MV, Huntington MK. Faculty, resident, and clinic staff’s evaluation of the effects of EHR implementation. Family Medicine. 2010;42:562–6. [PubMed]