This was a cluster randomized clinical trial of provider behavior change. We measured prescribing behavior in both the intervention and control groups before and after the introduction in the intervention group of a DSS providing evidence at the time of electronic prescribing. This approach enabled us to measure the independent effects of the intervention, while controlling for baseline differences in prescribing behavior and other temporal trends unrelated to our intervention.
In this study the unit of intervention was the provider. We chose this design because a cluster randomized clinical trial is the strongest study design available to assess the effect of a DSS upon provider behavior, as it directly compares providers receiving the intervention to those not receiving the intervention. Direct contamination between providers was likely to be minimal because, as detailed below, the DSS provided privately viewed messages that only briefly engaged the practitioners and their patients. Had we randomized patients instead, it would have resulted in providers taking care of both patients receiving the intervention and others not receiving the intervention, with the strong possibility of diluting the intervention effect.
This study was conducted at two clinical sites. One was the Pediatric Care Center at the University of Washington (PCC), an outpatient teaching clinic for pediatric residents and a clinical practice site staffed by full-time pediatric providers. The other site was Skagit Pediatrics (SP), a primary care pediatric clinic serving a rural and semi-urban patient mix approximately 60 mi north of the Seattle metropolitan area.
At the start of the study period at PCC, care was provided by 29 resident physicians, two nurse practitioners, and seven attending physicians, each with their own patient panels. At SP, there were eight physicians and two nurse practitioners, also each with their own patient panels. Both clinics adopted a computerized patient flow manager developed by one of us (JAW); this system was described in detail in our earlier publication [35
]. At PCC, a computer workstation was placed in physician work areas and nursing stations and connected to a server via a local area network. At SP, because of limited space availability in exam rooms, providers were equipped with wireless handheld computers (either personal digital assistants or pen-based tablet computers) connected via the local area network to a server.
An electronic prescription writer was developed to interface with the computerized patient flow manager. To prescribe a medication, a provider first selected the patient's name and then the patient's medication, indication, dosage, and, finally, the duration. The patient's weight was entered by the nurse during check-in. A paper copy of the prescription was printed for the patient and was also attached to each medical record.
Providers were trained on the network, and then a 6-mo period prior to randomization was used at each site to collect baseline prescribing behavior for all providers. When the intervention started, paper prescriptions were either removed (at PCC) or actively discouraged (at SP).
The participants in the study were the 36 HCPs at PCC and the eight HCPs at SP (). Study investigators at each site were excluded from participation. The protocol for both sites was approved by the University of Washington Institutional Review Board. Consent to participate in the study was given by the providers; individual consent from patients was not required.
Recruitment, Randomization, and Analysis of Providers in Study
At each site, the unit of randomization was the HCP. We used a stratified randomization process to randomly assign providers to either the intervention arm or the control arm. Specifically, for each condition (otitis media, croup, etc.), providers were first stratified by the number of prescriptions they wrote in the baseline period, in order to roughly equalize the number of patients seen by providers and prescriptions written in the intervention and control arms. Then, within strata of high or low number of prescriptions written, HCPs were randomly assigned to receive evidence-based medicine prompts or not. In both clinics, providers could have been randomly assigned to receive anywhere from none to all of the evidence-based prompts; after randomization it turned out that all providers received at least one evidence-based prompt.
Random numbers for allocation were generated by computer, and were concealed until interventions were assigned. This process was overseen by the research coordinator (L. L.) in conjunction with the investigator in charge of the data structure (J. W.). L. L. enrolled participants and assigned participants to their groups based on the randomization. Participants were not informed in advance of the study as to which conditions were being investigated, and hence were blinded to the intervention. However, based on the nature of the intervention, HCPs were (theoretically) able to determine to which evidence screens they were randomized by discussions with other HCPs.
In 2000 and 2001, additional residents were randomized as they joined PCC as interns. After baseline data collection for these residents, they were randomized as the others to either the intervention or control groups, and then followed in the same manner as the other providers until study conclusion.
Interventions and Conditions Studied
For each condition studied, providers in the treatment arm were shown pop-up “alert” screens, based on their selection of medication, indication, or duration. The first screen contained a short summary of the evidence either supporting or refuting the current choice of medication, indication, or duration. The provider could then choose to (i) view more information about this evidence, (ii) view the abstract of the article from which the evidence was derived, (iii) view a scanned PDF version of the article, or (iv) have the reference E-mailed to them for later viewing. shows examples of the first screens shown to providers for otitis media and allergic rhinitis (a screenshot of how this message appeared is shown in Figure S1
). In the vast majority of cases, providers did not venture past the first screen during the process of writing a prescription.
Example Summaries of Actions Triggering First-Level Evidence Screen for Two Selected Conditions
The conditions included in the intervention were acute otitis media, allergic rhinitis, sinusitis, constipation, pharyngitis, croup, urticaria, and bronchiolitis. For the outcome of otitis media, a small percentage of the data includes data previously published in the earlier report by Christakis et al. Excluding these data did not affect our results or conclusions, and we have elected to include these data so that we are able to address the consistency of effect among the two intervention sites. The full details of the information provided in the evidence screens are available from the authors.
Our primary outcome was changed physician behavior in accordance with the intervention message screens. Our primary measure assessed all of the interventions considered together, in order to answer the question, “Can we influence provider prescribing behavior by providing ‘just-in-time,' evidence-based prompts?” We also looked at the effect of the message screens upon the separate outcomes of (i) otitis media, (ii) allergic rhinitis, and (iii) a combined category of the other (less common) conditions. These categories were chosen a priori, as these conditions were the ones found to be most frequently evaluated in the clinic during the study planning stages (and were not necessarily the conditions for which medications were most frequently later prescribed). All analyses were performed by “intention to treat.”
Bronchiolitis was studied separately. This was necessary because the “baseline” period (used for randomization and to measure change from baseline) did not include the most recent past bronchiolitis season, and hence change from baseline could not be measured in the analysis. Instead, for bronchiolitis we compared the behavior of providers receiving the intervention directly to the behavior of control providers.
We performed two subanalyses. In our first subanalysis we added a “one click” option for allergic rhinitis and otitis media that allowed a provider to rapidly accept a pre-written electronic prescription corresponding to the “correct” message presented on the screen. For example, a provider attempting to prescribe diphenhydramine for allergic rhinitis received an evidence-based alert screen recommending fluticasone for this indication. In the original intervention, a provider wishing to change from diphenhydramine to fluticasone would have had to close out the alert screen, cancel the diphenhydramine prescription, and then begin the fluticasone prescription. With the one-click option, a provider was able instead to click a button that closed both the alert page and the diphenhydramine prescription, and automatically completed a weight-based fluticasone prescription. This one-click option was estimated to save each provider approximately 11 keystrokes or mouse clicks for each prescription dispensed.
Our second subanalysis studied whether or not the intervention effectiveness faded over time, even with continued alerts. To assess whether providers were tiring of the intervention, we divided the time following the beginning of the intervention into five quarters (3-mo periods) following the introduction of the intervention; within each time period, the intervention effect was otherwise assessed exactly as in the main analysis. Dividing the study in this way allowed us to see whether the effect waned over the period of the study, and to test whether providers might be paying less attention to the intervention with repeated exposure over time.
As in the original study, provider behavior change was measured as the difference, by study arm, between the outcomes in the period before and after the trial (except for bronchiolitis, as mentioned above). Measuring each provider's behavior as a change from baseline served two functions. First, it controlled for each provider's individual prescribing practices, and second, it reduced the random variance in the outcome measure, affording our analysis greater power. Because we measured the outcomes as a change in individual behavior, it was not necessary to control for provider-specific potential confounders. To test the primary hypothesis we used weighted regression analyses, controlled for clustering by provider, to test the difference in behavior change between the treatment and control groups. Weighted regression analyses, again controlling for clustering by provider, were also used to assess the statistical significance of the behavior change within the treatment and control groups.
The provider panels were unbalanced because of different work styles and schedules, and some providers had many more visits than others. As a result, the outcomes (mean change in provider behavior) were estimated with a greater degree of precision for providers with many visits than for providers with fewer visits. To account for this in the regression analyses, we conducted weighted analyses, whereby each provider's behavior contributed information to the analysis proportional to the precision with which their mean was estimated. As in the original analysis, this method achieved greater precision in the intervention estimates than unweighted analyses.
Not all providers treated patients for each randomized intervention during both the before and after period, and therefore, the number of providers contributing information to each intervention varied slightly.
All analyses were conducted with Stata, version 6.0, statistical software (Statacorp,http://www.stata.com
The number of providers available to participate in this study was estimated a priori to be approximately 42. We calculated the power of the study adjusted for clustering effects using the method of Hayes and Bennett [38
]. At the time the study was conceived, approximately 75% of otitis media cases were being treated with antibiotics, and we considered a clinically significant goal to be that of lowering this proportion to 60%—an absolute change of 15%. With 42 providers, we calculated that this treatment effect could be detected with 90% power with alpha = 0.05, assuming a standard deviation of 15% across providers.