Of 3784 potentially relevant articles screened, 32 papers met all our inclusion criteria ().
Summary of 32 RCTs on disease specific clinical decision support systems
Fourteen (43%) of the trials were performed in the US, seven (21%) in the Netherlands, and four (12%) in the UK. Four of the trials were published in medical informatics journals, and the rest in medical journals. The trials included 181 702 patients and 7315 physicians. The majority (22 trials) were performed in primary care, including 897 general practitioner (GP) offices. Of the 11 trials performed at hospital level, two were performed in an outpatient department, three in internal medicine departments, one in a surgical department, one in an intensive care unit, two in emergency departments, one in a trauma unit, and one in various different departments. Asthma (n=4), diabetes (n=4), and hyperlipidemia (n=3) were the most common diseases addressed ().
General trial features
Twenty-six trials (81%) did not provide an RCT registration number (ie, http://Clinicaltrials.gov
and others), while only seven trials (21%) offered web access to the full trial protocol. One trial did not state funding sources (). In nine trials (28%), more than half of the authors were medical doctors; in 10 trials, information on the background and education of the author(s) was not provided. Twenty-two (68%) trials chose a cluster-randomized design, which was the most common design among trials in primary care (21 of 22). Of the nine trials performed in a hospital setting, four had a cluster-randomized design and in these cases the department was chosen as the clustering unit. Two trials provided information on changes to the trial protocol, and one trial addressed CONSORT guidelines.
Characteristics of RCTs of clinical decision support and impact on outcome and implementation
Less than half of the CDS tools were implemented in an electronic medical record, and 14 (43%) of the CDS tools provided automatic alerts (). Twenty-four (75%) of the developed CDS tools provided decision support at the time and location of the decision need. Eighteen (56%) of the CDS tools did not disrupt the natural workflow of the physician. None of these CDS features had a significant influence upon the primary endpoint or overall conclusions.
Addressing sequential phases of a complex intervention
None of the trials defined the intervention as complex or discussed the definition of a complex intervention.77
Four trials defined all phases of a complex intervention and these phases were described in detail ().
Trials reporting on long-term CDS implementation
Six trials reported on the long-term implementation of the CDS tool used in the RCT ().
Four of these trials addressed all phases of a complex intervention and had a statistically higher CONSORT score compared to trials not reporting long-term implementation (OR 1.64, p=0.04). Three of these trials were performed at a hospital level, with the largest trial including 87 000 patients.
Inter-rater reliability and CONSORT score
The intraclass correlation coefficient used to establish inter-rater reliability was 0.69 for the 22-item CONSORT scale. The mean CONSORT score was 30.75 (95% CI 27.0 to 34.5), median score 32, range 21–38.
CONSORT: title, abstract, and background
Five trials did not identify a randomized design in their title. All trials had a structured abstract and gave a solid background and rationale for the trial ().
The CONSORT checklist: scoring of 32 RCT trials of disease specific clinical decision support systems
CONSORT: materials and methods
One trial addressed the CONSORT guidelines in their Material and Methods section. Twenty-seven trials (84%) clearly defined their participants, eligibility, and ethics approval. Fourteen trials (43%) did not clearly define the study objective or hypothesis. Twenty-three trials (72%) had an adequate definition of outcome measures. Fourteen studies (37%) did not perform or had an inadequate sample size calculation ().
Most trials described mechanisms to generate random allocation (59%) and the method of implementing the random sequence (47%). In contrast, only five trials (15%) gave adequate information regarding blinding (whether or not blinding was necessary and if necessary, how it was performed) ().
Most trials (87%) provided a detailed description of statistical methods (). Five trials had no figure showing participant flow and four trials did not include a table showing demographics. Nine trials did not address exclusions during the trial, and 10 trials did not define the date of trial initiation and termination. Only two trials performed an interim analysis, and only one trial addressed the ‘harms or unintended effects’ of the intervention.
The interpretation of results was justified in 28 trials (87%). Four trials did not discuss limitations and six trials did not address generalizability or provide recommendations for the future ().
Thirteen trials (40%) were classified as superior quality trials (≥3 points). Nineteen (59%) described the study as randomized, and the sequence of randomization was explained and was appropriate. Twenty-seven (85%) did not describe blinding. Ten (32%) did not describe dropouts ().
The Jadad instrument: scoring of 32 RCT trials of disease specific clinical decision support systems