Authorization for Study
Prior to initiating the study, authors obtained human subjects approval from the Vanderbilt Institutional Review Board. As an exempt study evaluating an educational program, consent forms were not required from patients or clinicians. Through email to clinicians, project staff handing out explanatory cards during clinical rounds, and focused meetings with teams on each participating study unit, eligible clinicians obtained notification about the study and its randomization processes. Administrative and clinical leaders of each unit authorized the procedures for randomization and missing data prevention.
Vanderbilt University Hospital is a 658-bed, academic, tertiary care facility in Nashville, Tennessee that provides adult and pediatric care to local and referral populations. Based on established acceptance of the CICS service on individual patient care units, a similar level of experience providing CICS among the unit librarians, a stable history of use of the CICS service in those units, and the manner in which the CICS service had been used for patient-oriented questions, project staff selected four Vanderbilt University Hospital critical care units to participate in the study. For example, the Emergency Department was not included because it primarily uses the CICS for educational purposes (training residents or fellows), rather than patient-critical issues. The 35-bed Pediatric Intensive Care Unit (PICU), the 26-bed Medical Intensive Care Unit (MICU), the 60-bed Neonatal Intensive Care Unit (NICU), and the 31-bed Trauma Unit (TICU) were initially included as study units. However, the PICU was eliminated from the study after 9 consults because of a shortage of available CICS, PICU-trained personnel within the Eskind Library system.
Clinical Informatics Consult Service (CICS)
During the study, one specifically assigned CICS librarian on each study unit participated in clinical rounds three times per week with the critical care team. That individual completed the majority of the unit’s CICS consult requests, which were prioritized at their initiation by the requesting clinician: high (response required within 0.5 days), medium (3 days), or low (7 days). Priority time frames were established through consulting experienced CICS librarians and clinicians to establish a consensus optimal response time prior to the study’s initiation. Each consultation response included a documented bibliographic search strategy with corresponding references, a targeted list of full-text articles, and a written synthesis and critique of the relevant research materials. Typically, the CICS librarian shared this summary with the team verbally during a subsequent rounding session. For future reference by library staff and clinicians, CICS staff posted the consult summaries and related articles to the library’s intranet, web-based evidence repository.
Clinicians could, upon request to the rounding librarian, opt-out of the randomization process on a consult-by-consult basis if the evidence was deemed clinically critical and needed immediately, and if no resources were available to complete a search. Clinicians were immediately paged and emailed notification regarding the condition to which their consult had been randomized. Again, clinicians could opt out of the randomization at this point. Librarians entered the question received, unit, requestor, date of receipt, and priority level into a consult request database. Stratified by consult priority level, questions were immediately and automatically randomized via the computerized database to either the “CICS Provided” or “No CICS Provided” condition using a random number generator. For the “CICS Provided” condition, clinicians received typical consult results, whereas for the “No CICS Provided” condition, clinicians did not receive consult results. Clinicians in the Trauma Unit requested no paging within the study and only received the email. Because this was an effectiveness trial, regardless of the randomization status of a CICS consultation request, clinicians could conduct their own literature searches and reviews.
Consult Rating Procedures
Clinicians completed both a brief pre-consult request form and a post-consult evaluation form. The pre-consult forms were completed on paper by the clinician, acting either alone or aided by the CICS librarian. This pre-consult form included a standardized checklist of reasons for a clinical consultation request (e.g. update my knowledge) and a standardized checklist of clinical actions that consult results might influence (e.g. change drug dosage); participants could add additional reasons or actions if needed. Post-consult forms were emailed to clinicians 3 days after consult results became available, with the specific time frame for sending of the email based on the time frame (priority level) of the consult request. The question prompting the request and unit and clinician name were automatically inserted in the form by the consult request database. Clinicians completed the forms by following a hypertext link embedded in the email. For consultation requests that had been randomized to “No CICS Provided” status, clinicians received the URL for the evaluation form in an email sent 5 days after making the consult request. The email was delayed to allow clinicians time to review and potentially act on the information located on their own. Clinicians in the “No CICS Provided” condition who did not do their own search could not complete this form as there was no evidence or search to rate.
Reminders to clinicians helped to prevent missing response data. If clinicians did not complete their post-consult rating form within 3 days, they received a second (reminder) email request to complete the work, with a URL link to the form. A final (third) email reminder was sent out 3 days from that date. If the form was not completed within 3 days from the last reminder, project staff paged clinicians or contacted them by telephone to ask them to fill out the form. Based on prior agreement, the Trauma unit clinicians were never paged to fill out research forms. The last measure to prevent missing follow-up data involved mailing a printed copy of the form to the clinician. To address potential clinician “fatigue” in completing forms, a $5 gift card incentive was provided for each completed form.
Behaviors assessed on the 10-item post-consult forms included time spent searching for evidence by the clinician and associates/trainees, alternate sources of information such as formal and informal consults to colleagues, overall immediate and future impact of the consult results (or their own searches if completed), clinical actions that were influenced by the consult, barriers to implementation of the consult results, consult quality, and overall satisfaction. Response options were either a Likert-type format (range of 5 response options), free text, or select any number of relevant response options (e.g. “check all that apply”).
The main outcomes were analyzed using an Intention to Treat (ITT) approach using the 2 randomized groups (CICS provided/No CICS provided). Thus, for these analyses, data were analyzed only by the randomization (No CICS Provided/CICS Provided) condition even if the clinician opted out of the “No CICS Provided” status. Clinicians were requested to indicate if they wanted to opt out of the randomization process before they knew the condition of their consult. However, clinicians did not always indicate their opt out preference prior to knowing their condition and their requests to opt out were always honored. Therefore, in analyses we assumed that this opt out occurred after knowing the experimental condition and all opt outs were coded as No CICS Provided consults. This provides the most conservative estimate of opt out effects by putting all of the opt out consults (all of which actually did receive results) into the No CICS Provided for ITT analyses. When clinicians did not receive CICS search results, they completed only a subset of the assessment questions that would be relevant for their own searches (e.g. no questions asked regarding the research summary).
The primary analytic approach related each randomized condition to outcomes (immediate and potential future clinical decisions, action index, and number of research articles read, satisfaction, colleague consults, and time spent searching), using random coefficients hierarchical models to control for the clustering of consults within clinicians on outcomes. 35
If ratings within clinicians were correlated, traditional analysis of variance (ANOVA), assuming independence, might register specious significance. We used a two-level model in which each clinician’s intercept (mean of their own ratings) controlled for multiple ratings by clinicians.
To measure the clinical effects of each CICS consultation, the project created an “action index” – the number of specific clinical actions influenced by consult results. Actions included the initiation, addition, change, discontinuation, or inhibition of any clinician-initiated event related to diagnosing or treating the patient. The index, a simple sum of the number of actions that clinicians reported as impacted by the consult.
A secondary analytic approach examined each experimental group was divided into two subgroups based on whether clinicians initiated their own evidence searches, resulting in 4 “evidence source” groups: No Evidence (neither librarian nor clinician completed a search), Clinician Only, CICS Librarian Only, or CICS Librarian + Clinician. Only three “evidence source” groups had outcomes data and could be analyzed using this type of analysis: Clinician, CICS Librarian, and Clinician + CICS Librarian.
Multi-level analyses were conducted with SAS PROC MIXED (v9); all other analyses were conducted with SPSS (v14).