The four research participants were senior clinicians employed by the Posttraumatic Stress Disorder (PTSD) Clinical Team at the Salt Lake Veterans Affairs Medical Center. One was a clinical psychologist, one was a registered nurse, one was a social worker, and one was an addiction therapist. They have an average of 16 years of professional experience and have worked with the PTSD Clinical Team for an average of two years.
The PTSD Clinical Team, an outpatient clinic for patients with PTSD, was chosen as the study site for both design and practical reasons. The design consideration was a high incidence of MDD in patients with PTSD,22
which increased the opportunity to compare the two experimental methods in their effectiveness in documenting the DSM-IV diagnostic criteria for MDD that these patients met.
The PTSD Clinical Team was chosen as the study site for several practical reasons. First was the willingness of the PTSD Clinical Team staff to participate in the study, especially the willingness of the Coordinator of the PTSD Clinical Team to oversee the assignment of cases to experimental conditions. Second, the PTSD Clinical Team has permanent staffing, so the logistical and statistical problems associated with the rotation of residents and interns were avoided. Third was the location of the clinic, which was within a few feet of the offices of the investigator and his staff. The fourth reason was the availability of hardware and a local area network to support the computer arm of the study.
Two variables were used to assess the relative effectiveness of the two reminder methods. The first measure was the proportion of cases screened for mood disorder. The second measure was the proportion of cases for which the diagnosis of MDD was fully documented according to DSM-IV diagnostic criteria. For cases in the computer reminder condition, both measures were determined by checking the computer reminder database. In the paper condition, both measures were determined from the paper checklist.
Computer Reminder System
The computer reminder system was dubbed the CaseWalker, after the recommendation of the guideline developers that staff learn the logic of the guideline by “walking” cases through it. The CaseWalker generated reminders to screen patients for mood disorder, presented and scored the DSM-IV criteria for MDD, and created a progress notes based on answers given to questions derived from the guideline.
The CaseWalker platform was a Windows NT 4.0 local area network. The graphical user interface was written in Delphi 3. An Interbase database resided on the local network server. Expert C++, an inference engine and knowledge base, was used to encode the guideline algorithm.
Each clinician saw patients in his or her office, and each had a PC on his or her desk that ran the CaseWalker as well as the VHA electronic medical record (EMR). Use of the EMR increased over the course of the study. In the beginning, clinicians used it most often to view appointment schedules, and sometimes to view laboratory results. By the end of the study, they were using it for progress notes, laboratory orders, and medication orders. Thus, the clinicians were accustomed to using their computer in their clinical practice. However, it was not possible technically at the beginning of the study to integrate the EMR and the CaseWalker directly, so progress notes generated by CaseWalker had to be “cut and pasted” into the EMR.
Daily, the CaseWalker presented each clinician with a list of patients on the clinician's caseload who needed to be screened for mood disorder (). For each patient on the reminder list, the user had the option of selecting a new reminder date, terminating the reminders, or processing the guideline. Users who opted to process the guideline for a patient were asked whether a four-item screening test for mood disorder was positive. An optional pop-up form () was available to administer and score the screening test. Alternatively, the clinician could simply enter the result (positive or negative) of the screening test.
Reminder for patients requiring initial screening and assessment.
Checklist for initial screening for mood disorder.
If the mood disorder screening test was positive, an MDD diagnostic criteria checklist popped up automatically (). These criteria include the criteria for a diagnosis of major depressive episode (e.g., depressed mood, insomnia, weight change, fatigue) as well as the rule-out criteria for MDD (e.g., no psychotic or schizo-affective disorder, manic or mixed episodes, substance-induced mood disorder, or normal bereavement). In addition, the diagnosis of MDD requires a determination that the depressive symptoms caused clinically significant distress or impairment. The user was required to use the checklist in making the diagnosis of MDD, and the diagnostic criteria were scored automatically by the Delphi program.
First section of the checklist of DSM-IV criteria for major depressive disorder.
At the conclusion of a CaseWalker session, the clinician was given a chance to review the progress note that had been generated during the session. Each user entry during the session concatenated appropriate text to the progress note, which could be edited by the clinician. Then, the note could be either printed and filed in the paper medical record or pasted to an electronic progress note in the hospital information system.
The clinic clerk inserted the checklist into the assessment section of the paper medical record of each new patient assigned to the checklist arm of the study. Prior to each appointment, the paper medical record was to be pulled by clerical staff and made available to the clinician.
The paper checklist was three pages long. The first page contained the four-item mood disorder screening test. The next two pages contained the DSM-IV criteria for MDD in checklist form. If the screening test was positive, the clinician was to check all DSM-IV criteria that applied and determine whether the patient satisfied diagnostic criteria for MDD. These diagnostic criteria, of course, were exactly the same as those presented by the CaseWalker, and they were presented in the same order.
Consecutive admissions to the caseloads of the four participating clinicians between Jan 5, 1998, and Oct 7, 1998, were assigned randomly to one of the two experimental conditions. The coordinator of the PTSD Clinical Team assigned 108 patients newly referred to the clinic to one of the four PTSD Clinical Team clinicians on a nonrandom basis. The most common consideration in assigning a new patient was the current caseload of each clinician, but sometimes patient characteristics determined clinician assignment. For example, a female patient with PTSD secondary to sexual trauma would be assigned to the female therapist for clinical reasons. Even though the assignment of cases across clinicians was not random, potential bias of the comparison of the two reminder methods was controlled for by the random assignment of cases to experimental condition within clinician. Randomization was based on a table of random numbers.
Because there was not always time to screen for mood disorder during the first session with a patient, only those cases seen by the assigned clinician at least twice were included in the data analyses. A total of 83 cases met the two-visit-minimum criterion, but five cases that were supposed to be in the CaseWalker group were dropped from the study because of a clerical error that resulted in the CaseWalker procedure never being initiated for these cases. A total of 37 and 41 patients were assigned to the CaseWalker and paper checklist conditions, respectively. A chi-squared (χ2) analysis indicated no significant difference in the number of cases per clinician and experimental condition in the sample of 78 patients who had at least two visits (χ2 = 0.56, P = 0.91).
Analysis of variance (ANOVA)23
was used to analyze the results. Independent variables were experimental condition (i.e., CaseWalker vs. paper checklist), clinician, and the experimental condition-by-clinician interaction. Dependent variables were the absence or presence of the indicated measure, scored as 0 (absent) or 1 (present). The effect size of significant group differences was determined by means of the f
designates an f
value of 0.10 as small, 0.25 as medium, and 0.40 as large.