Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
J Pediatr. Author manuscript; available in PMC 2013 May 1.
Published in final edited form as:
PMCID: PMC3586587

Comparative Effectiveness Research Using the Electronic Medical Record: An Emerging Area of Investigation in Pediatric Primary Care

Alexander G. Fiks, MD, MSCE,1,2,3,4 Robert W. Grundmeier, MD,1,2 Benyamin Margolis, PhD, MPH,5 Louis M. Bell, MD,1 Jennifer Steffes, MSW,6 James Massey, RN, MBA,1,2 and Richard C. Wasserman, MD, MPH1,7

Previous research demonstrated that as many as one-half of all clinical decisions are reached without adequate medical evidence to inform choices.1 In contrast to other areas of investigation, comparative effectiveness research (CER) directly addresses this problem. Specifically, CER uses varied study designs to generate and synthesize evidence demonstrating the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor clinical conditions or improve the delivery of care.1,2 The need for CER is especially urgent in pediatrics, a field in which clinicians and families frequently depend on the generalization of medical knowledge from adult trials despite limited evidence to support the use of these treatments.3 To close these knowledge gaps, the American Recovery and Reinvestment Act of 2009 (ARRA) allocated $1.1 billion for CER.4

CER to improve child health must address the needs of families treated by primary care clinicians in the community. After the newborn period, most children are infrequently admitted to hospitals and are only occasionally cared for by pediatric subspecialists.5 To aggregate data from multiple primary care settings and avoid bias from small samples of families or clinicians, pediatric primary care practice-based research networks have increasingly informed how pediatric clinicians understand child health issues.6 However, because of the costs of manually reviewing paper-based records and collecting other data from geographically dispersed sites, the research productivity of these networks could be dramatically increased through access to electronic medical record (EMR) data.7 This potential is enhanced because EMRs are becoming increasingly common in pediatric practices8,9 as a result of ARRA incentives, which devoted $19 billion to promote their adoption.4 Through examples from The Children’s Hospital of Philadelphia (CHOP) Pediatric Research Consortium (PeRC) and the American Academy of Pediatrics (AAP) Electronic Pediatric Research in Office Settings (ePROS) networks, this review highlights the challenges and opportunities in using EMRs to foster CER and improve the delivery and outcomes of pediatric care.

Strategies for Using EMR Data for CER

No single strategy currently exists to conduct CER with data from pediatric practices. To address the multiple barriers to using EMRs for CER (Table),10,11 the optimal approaches for performing this research will evolve with time. However, two large-scale pediatric primary care research networks, CHOP’s PeRC and the AAP’s ePROS, have been actively developing strategies for EMR-based CER. In PeRC, these approaches have been applied in a single implementation of one EMR system and resulted in approximately 50 publications in the last 8 years. In contrast, ePROS was launched in 2010 with funding from the Maternal and Child Health Bureau of the Health Resources and Services Administration and is designed to function with multiple implementations of multiple EMR systems. We highlight the diversity of approaches used for EMR-based CER in these two networks and present examples of successfully completed studies in PeRC and planned studies for ePROS. Our discussion will demonstrate both the strength of this increasingly important methodology and the need for future research to maximize medical progress.

Barriers to using EMRs for CER

Approaches Used in PeRC

With support from the Agency for Health Care Research and Quality, PeRC was created in 2003 with the goal of fully using clinical data to evaluate and improve child health outcomes. Because of local expertise, network leadership also prioritized the study of how best to use clinical decision support (CDS) at the point of care to achieve these goals. PeRC currently consists of 30 practices located in New Jersey and Pennsylvania and includes >160 attending physicians, 24 nurse practitioners, 144 residents, and >208 000 children. Consistent with Institute of Medicine priorities for conducting CER in diverse settings,1 PeRC includes urban practices serving primarily Medicaid-insured children and suburban practices caring for a largely privately insured population. EMR-based research is facilitated at CHOP by the presence of a common institutional review board that serves all sites, a team with research, database reporting, and medical informatics expertise in the hospital’s Center for Biomedical Informatics, and by the active use of the EMR at all network sites.12

Using the EMR to Support Distinct Types of CER: Examples from PeRC

At CHOP, distinct approaches are used for conducting CER, depending on whether a given study involves secondary data analysis, subject recruitment, or the implementation and evaluation of CDS. Because all CHOP’s practices share a common EMR (EpicCare, Verona, Wisconsin) with a single data structure, all data from clinical encounters flow to one data warehouse, Clarity, which is provided by the EMR vendor. We outline how PeRC researchers used the EMR and this data warehouse to study the effectiveness of antibiotic prophylaxis for urinary tract infection (UTI), to recruit subjects for CER, and to conduct research on the effectiveness of CDS to improve vaccination rates and asthma care for children. CER includes trials of CDS because they assess the relative benefits and risks of using EMR-based reminders and other tools versus usual care on clinical or patient outcomes.

Observational CER

In a 2007 publication, a team of PeRC researchers examined the association between the use of antimicrobial prophylaxis and the risk of recurrent UTI.13 Accurately defining subject eligibility and episodes of illness is critical in the analysis of EMR data. In this case, manual chart review verified that children had no earlier UTIs. Children were observed either until the child had a recurrent UTI or until the last clinic visit at the practice. Reflecting the scale of investigation facilitated by the use of the EMR in a practice-based research network, a population of 74 974 children was eligible for participation. From this group, the researchers identified 775 children with a first UTI on the basis of the microbiology results present in the EMR. Without the use of the EMR database, chart review on this scale would not have been feasible.

Aware of the need to validate the data, the researchers reviewed paper-based results and correspondence to ensure that the EMR did not lack documentation of UTIs treated outside of PeRC. Indicative of the completeness of the EMR in this clinical context, they found that the EMR agreed with data from detailed chart review with >95% sensitivity and specificity. These researchers then assessed the time to recurrent UTI for each child. They were able to assess exposures including age at first UTI, sex, race, voiding cystourethrogram result, and exposure to prophylactic antibiotics. Ultimately, the authors found that antimicrobial prophylaxis after a first UTI was not associated with a decreased risk of UTI, but did increase the subsequent risk of antimicrobial resistance.

Subject Recruitment

Research conducted within PeRC has also demonstrated the effectiveness of the EMR for subject recruitment. One EMR-based approach takes advantage of the relationship between pediatric clinicians and their patients’ families. In consultation with PeRC’s medical informatics group, the research team inserts on-screen recruitment prompts in the EMR when children meeting enrollment criteria have office visits (Figure 1). The prompt then reminds the clinician of the purpose of the study and the criteria for eligibility while they are face-to-face with a subject who meets inclusion criteria on the basis of age, sex, and clinical characteristics such as chronic medical problems or medication use. Once prompted, the clinician discusses the study with the family and indicates whether the family is eligible for the study and willing to be contacted by the research team, is ineligible for the study, or declines participation. This information is then automatically delivered to the research team. Early results with this system indicate that between 19% and 40% of clinicians actively referred patients to research studies.14 Once referred, as many as 25% of participants were enrolled when contacted by the study team. Because the earlier study did not address these issues, future research should be directed at developing strategies to improve rates of clinician referral and patient/family participation in research and understanding potential sources of selection bias in using EMR-based recruitment strategies.

Figure 1
Recruitment prompt. In PeRC, recruitment prompts appear onscreen when charts are opened for eligible patients. Through the prompts, the clinician is reminded of the study name, eligibility, purpose, and compensation to research subjects and reviews this ...

With institutional review board approval, PeRC also regularly generates rosters of eligible subjects who have upcoming appointments. On the basis of these reports, research assistants meet patients when they arrive for regularly scheduled office visits and invite them to participate. Results to date indicate that both clinicians and principal investigators have had favorable experiences with this approach, which was also more effective than clinician prompts for studies requiring large samples.14 However, the use of rosters may require extensive travel by research assistants when subjects are being recruited from a wide geographic area. Rosters may not be feasible when subjects are being recruited on the basis of sensitive information in their medical record. In that circumstance, the institutional review board may require a referral from clinicians before potential subjects are contacted. To minimize the burden on families and the research team, strategies such as assembling lists of patients with a general interest in participating in research in a particular area, such as autism, may be helpful. However, the increased efficiency of this type of approach must be balanced against the potential for selection bias.

Trials of Point-of-Care CDS

In addition to research that uses the EMR to answer clinical questions or enroll research subjects, the effectiveness of tools within the EMR to increase the effectiveness of clinical care is an emerging topic of investigation.15 Within PeRC, trials of CDS to improve pediatric immunization rates and asthma care have proven successful. Vaccination is ideally suited for CDS because immunization information is captured as discrete data within the EMR,and missing dataare minimized through the use of immunization registries. The availability of background information, such as the vaccination history, is essential to guide decision support logic. Because both Healthy People 2010 and network clinicians prioritized timely vaccination in children <24 months old, we created a system that delivered on-screen alerts to nurses or clinicians when any child in this age range arrived in the office and was due for immunization (Figure 2).16 To facilitate interpretation of the recommendations (for example, when earlier doses were given too early and were therefore invalid), PeRC researchers provided clarifying text. In a 1-year study involving 4 practices, alerts appeared at 15 928 visits.16 The investigators found that rates of captured immunization opportunities, defined as at least one vaccine given when vaccines were due at a visit, increased from 78.2% to 90.3% at well visits and nearly tripled from 11.3% to 32.0% at sick visits. Furthermore, up-to-date vaccination rates for children increased from 81.7% to 90.1% from the control to intervention period. Demonstrating the ability to translate EMR-based CER into practice, the vaccine alerts system has now been used at >1 000 000 visits at CHOP and provides a model for using the EMR to support clinical decision-making.

Figure 2
Appearance of vaccine alerts in the EMR. The PeRC vaccine decision-support system provides specific recommendations on which vaccines are due, contraindications because of allergy, and special considerations.

Unlike vaccine decision support that relies exclusively on data within the EMR, providing meaningful asthma decision support to clinicians depends on evaluating the care delivered in the context of children’s symptoms. To help clinicians collect this information, PeRC researchers created a form, called the asthma control tool, to help clinicians systematically capture information on asthma symptoms.17 In a 1-year study involving 12 practices, the CDS provided onscreen recommendations for medication changes and evidence-based office procedures with information from both the asthma control tool and EMR database. In the urban setting, the intervention resulted in a 6% increase in the use of asthma controller medications and 3% increase in spirometry at intervention versus control practices. Indicative of the importance of studying decision support in diverse clinical contexts, suburban practices had a 14% improvement in the use of asthma care plans to help families better manage symptoms at home and a 6% increase in spirometry with the intervention, but no change in prescribing.

Approaches Used in ePROS

Although only recently established and yet to yield any findings, the national reach and large scale CER potential of the AAP-based ePROS network merits its inclusion in this review. ePROS’ parent network, Pediatric Research in Office Settings, is the nation’s largest pediatric primary care research network and has in the past confined itself almost exclusively to paper-based data collection. ePROS is a subnetwork of Pediatric Research in Office Settings practices that use EMRs for clinical documentation and have been recruited under an ARRA grant from the Maternal and Child Health Bureau of the Health Resources and Services Administration. The innovation offered by ePROS is two-fold. First, ePROS is national in scope. Consequently, results from ePROS studies will be more generalizable to US children and pediatricians than studies from a regional network. Second, ePROS is not confined to practices and clinics using a single implementation of a single vendor’s EMR system, but can capture EMR data from various implementations of multiple EMR vendor systems. This is critically important, because according to the AAP Council on Clinical Information Technology, approximately 100 EMR systems are in use in pediatric practice to-day.18 Although it cannot access data from all these systems, through the use of its data extraction vendor, the Clinical Integration Networks of America, ePROS can extract data from 20 commonly used products while also accounting for the variability that occurs between different implementations of the same vendor’s product. ePROS employs a federated database scheme modeled after an existing family medicine EMR network,19 by which EMR data elements are extracted, aggregated, and standardized while remaining stored within each practice or clinic site. The data are subsequently de-identified before becoming available to researchers for selected secure queries (Figure 3). Consequently, no personally identifiable data leaves the practice.

Figure 3
ePROS data flow. Data are extracted locally at each practice through an on-site server. These data are then aggregated on a secure server in the federated ePROS database to allow for the analysis of pooled data from all participating practices. FTP, file ...

In contrast to PeRC, in which a single EMR system houses all data, in ePROS, a server with read-only access to the EMR must first be installed in each practice or clinic. Data mapping software in the ePROS practice server then extracts selected data elements from the EMR for storage in a clinical data repository in a standardized format suitable for point-of-care reports on specific practice patients. These data are then further standardized into the widely recognizable formats set by standards organizations, the Continuity of Care Record of the American Society for Testing and Materials International or the Continuity of Care Document. These formats can be anonymized and sufficiently standardized to allow comparisons of extracted variables in practices using different EMRs.

Data from the multiple ePROS practices can then be aggregated to create network-wide databases. Thus, even as data continue to reside in each practice’s dedicated server in the standardized clinical data repository, they can be aggregated and analyzed to perform observational CER. In addition to doing observational CER, the ePROS system offers the ability to deliver CDS. Unlike PeRC, CDS in ePROS will be delivered through paper-based reports summarizing recommendations for each child’s visit.

ePROS will begin with approximately 30 practice sites from around the country, with plans to quickly expand to >100 sites, with a patient population of approximately 1 million children. This will give ePROS the capacity to conduct CER, not only on common conditions such as obesity, asthma, and attention-deficit/hyperactivity disorder, but also on less common conditions. With this scale, data from the network will also help identify infrequent adverse outcomes of common treatments.

Initial ePROS Investigations

The first ePROS investigations will focus on two areas: (1) the prevalence of pediatric obesity and its accompanying co-morbidities; and (2) the rate of and indications for psycho-tropic medication use in children. These initial studies will demonstrate the feasibility of extracting data from ePROS practices and provide baseline data for future CER studies in these areas. The initial ePROS CER study will be a cluster randomized clinical trial of the effectiveness of point-of-care CDS in the medication management of children with attention-deficit/hyperactivity disorder who are being newly treated with stimulant medication.


Incentives from ARRA are helping to transform pediatric primary care through the expanding implementation of EMRs. Although multiple obstacles exist to the extraction of data from these systems, practice-based networks have begun to demonstrate the potential of using these systems for CER. Innovative investigation in these settings will increasingly address gaps in medical evidence that limit clinicians and families from reaching informed decisions in many contexts. The examples provided in this brief review highlight the diverse types of pediatric CER supported by the EMR. Although certain to evolve, the infrastructure and approaches now under evaluation in PeRC and ePROS are likely to provide a foundation to accelerate progress in pediatric research and clinical care in the next decade. Fortunately, federal funding agencies have recognized this opportunity to support research networks with expertise in clinical research, implementation science, and medical informatics as they work to improve child health by disseminating innovations to both practitioners and scholars.■


Supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development (award K23HD059919 to A.F.), Health Resources and Services Administration Maternal and Child Health Bureau (grant UB5MC20286), and National Eye Institute (R13EY019972 to R.W.). The content is solely the responsibility of the authors and does not necessarily represent the official views of the Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Institutes of Health, the Health Resources and Services Administration Maternal and Child Health Bureau, the National Eye Institute, or the US Department of Health and Human Services. A.F. and R.G. are co-inventors of, but have earned no income from, the “Care Assistant” web-based software capable of providing clinical decision support and assisting with research recruitment. They do not hold a patent on this invention nor is a software licensing agreement in place. The other authors declare no conflicts of interest.


American Academy of Pediatrics
American Recovery and Reinvestment Act
Clinical decision support
Comparative effectiveness research
Children’s Hospital of Philadelphia
Electronic medical record
Electronic Pediatric Research in Office Settings
Pediatric Research Consortium
Urinary tract infection


1. Institute of Medicine (US) Initial national priorities for comparative effectiveness research. National Academies Press; Washington, DC: 2009. Committee on Comparative Effectiveness Research Prioritization.
2. Sullivan P, Goldmann D. The promise of comparative effectiveness research. JAMA. 2011;305:400–1. [PubMed]
3. Simpson LA, Peterson L, Lannon CM, Murphy SB, Goodman C, Ren Z, et al. Special challenges in comparative effectiveness research on children’s and adolescents’ health. Health Aff (Millwood) 2010;29:1849–56. [PubMed]
4. American Recovery and Reinvestment Act of 2009: making supplemental appropriations for job preservation and creation, infrastructure investment, energy efficiency and science, assistance to the unemployed, and state and local fiscal stabilization, for fiscal year ending September 30, 2009. 2009 Feb 17;115 and for other purposes, Pub. L. No. 111-5 123 Stat.
5. Agency for Healthcare Research and Quality Healthcare Cost & Utilization Project (HCUP) [webpage on the Internet] Care of children and adolescents in US hospitals. Rockville: Agency for Healthcare Research and Quality: c2003; [Accessed May 31, 2011]. Available at
6. Wasserman R, Slora E, Bocian A. Current status of pediatric practice-based research networks. Curr Probl Pediatr Adolesc Health Care. 2003;33:115–23. [PubMed]
7. Lauer MS, Collins FS. Using science to improve the nation’s health system: NIH’s commitment to comparative effectiveness research. JAMA. 2010;303:2182–3. [PubMed]
8. DesRoches CM, Campbell EG, Rao SR, Donelan K, Ferris TG, Jha A, et al. Electronic health records in ambulatory care—a national survey of physicians. N Engl J Med. 2008;359:50–60. [PubMed]
9. Hsiao C-J, Beatty PC, Hing ES, Woodwell DA, Rechtsteiner E, Sisk J. Electronic medical records/electronic health record use by office-based physicians: United States, 2008 and preliminary 2009. Health E-Stat: National Center for Health Statistics; Hyattsville: 2009.
10. Fiks AG. Designing computerized decision support that works for clinicians and families. Curr Probl Pediatr Adolesc Health Care. 2011;41:60–88. [PMC free article] [PubMed]
11. Wasserman RC. Electronic medical records (EMRs), epidemiology, and epistemology: reflections on EMRs and future pediatric clinical research. Acad Pediatr. 2011;11:280–7. [PMC free article] [PubMed]
12. Fiks AG, Alessandrini EA, Forrest CB, Khan S, Localio AR, Gerber A. Electronic medical record use in pediatric primary care. J Am Med Inform Assoc. 2010;18:38–44. [PMC free article] [PubMed]
13. Conway PH, Cnaan A, Zaoutis T, Henry BV, Grundmeier RW, Keren R. Recurrent urinary tract infections in children: risk factors and association with prophylactic antimicrobials. JAMA. 2007;298:179–86. [PubMed]
14. Grundmeier R, Swietlik M, Bell L. Research subject enrollment by primary care pediatricians using an electronic health record. AMIA Annu Symp Proc. 2007:289–93. [PMC free article] [PubMed]
15. Garg AX, Adhikari NKJ, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293:1223–38. [PubMed]
16. Fiks AG, Grundmeier RW, Biggs LM, Localio AR, Alessandrini EA. Impact of clinical alerts within an electronic health record on routine childhood immunization in an urban pediatric population. Pediatrics. 2007;120:707–14. [PubMed]
17. Bell LM, Grundmeier R, Localio R, Zorc J, Fiks AG, Zhang X, et al. Electronic health record-based decision support to improve asthma care: a cluster-randomized trial. Pediatrics. 2010;125:e770–7. [PubMed]
18. AAP Council on Clinical Information Technology [webpage on the Internet] EMR review project. American Academy of Pediatrics: c2011; Elk Grove Village: [Accessed Jul 5, 2011]. Available at
19. Pace WD, Cifuentes M, Valuck RJ, Staton EW, Brandt EC, West DR. An electronic practice-based network for observational comparative effectiveness research. Ann Intern Med. 2009;151:338–40. [PubMed]