|Home | About | Journals | Submit | Contact Us | Français|
The present study developed and validated a configurable, adaptive, web-based version of the Structured Clinical Interview for DSM, the NetSCID. The validation included 24 clinicians who administered the SCID and 230 participants who completed the paper SCID and/or the NetSCID. Data-entry errors, branching errors, and clinician satisfaction were quantified. Relative to the paper SCID, the NetSCID resulted in far fewer data-entry and branching errors. Clinicians ‘preferred’ using the NetSCID and found that the NetSCID was easier to administer.
The Structured Clinical Interview for DSM (SCID) is currently the most reliable tool for assessing the presence or absence of the major DSM diagnoses, as defined by symptom clusters and time frames (1, 3). The structured format of the SCID with its direct adherence to DSM criteria accounts for its strong test-retest and inter-rater reliability for most diagnoses, and greatly improves the ability of researchers accurately to diagnose primary and comorbid DSM mental disorders (5, 7). Unfortunately, the full, printable SCID-Research Version (RV), available in PDF and MS Word formats, takes an average of 90 minutes to administer and requires considerable clinician training for reliable administration (2, 6). The SCID-5, released in 2014, consists of over 150 pages of heavily branched questions across 18 modules. Although it is easy to select individual modules for administration, more complex customizations of items and diagnoses within modules can be difficult to implement. In practice, once the paper SCID responses have been scored and diagnoses are finalized, the scoring sheets are usually archived for five years. Full response sets have rarely been available for research due to the complexity of entering SCID responses into a database.
In the current study, we sought to address some of the problems associated with paper SCID-RV administration in order to streamline its use in research and to make it possible for shorter versions of the SCID, including the SCID-Clinician Version (CV), to be used electronically in routine clinical practice. To do this, we worked closely with a panel of consultants including our second author, Dr. First (first author of the SCID), as well as several programmers and software testers, to implement the paper SCID-RV as a web-based software program.
Previous research has shown that administration of the SCID via telephone and paper produces similar results (4). Our goal was to create a software system that would branch instantaneously and automate calculations. We hypothesized that when compared with the paper SCID, a state-of-the-art software program for administering the SCID would be faster, require less training, decrease data-entry errors, and eliminate branching errors. We defined a data-entry error as an interviewer entering the wrong numeric response on the scoring sheet or screen. We defined a branching error as an interviewer’s entering a numeric response correctly, but accidentally skipping to an incorrect subsequent question. This error can occur as a result of an interviewer’s misreading skip instructions, miscounting endorsed criteria and therefore using the wrong skip instruction, or forgetting the correct instruction while searching for the next item on the paper form.
We also knew that a web-based software program would enable large numbers of detailed response sets to be collected in an SQL database and later downloaded, in a de-identified form, into SPSS or SAS. We anticipated that widespread use, particularly of a fixed or non-adaptive web-based SCID (one which enabled clinicians to turn off the ‘skip function’ for individual modules), could bring the benefits of ‘big data’ analysis to both clinical and mental health services research. For example, using a database it should be possible to determine the positive and negative predictive value of each DSM criterion. In addition, by selecting configurations that decrease branching, it should be possible retrospectively to identify novel symptom clusters that are associated with favorable outcomes, allowing clinicians to apply effective interventions to the individuals experiencing these specific symptom clusters. Lastly, we wanted to start with the most comprehensive version of the SCID, the SCID-RV, knowing that users would have the functionality to turn off branching as desired and to implement shortened clinical versions. Our goal was to create a software system that would support the DSM nosology, but one that would also help researchers to study quantitatively endorsement patterns in order to further refine the diagnostic nosology. An additional goal was to create a tool that would enable researchers to identify specific endorsement criteria that could be associated with individual bio-markers.
Although we were interested in making the SCID software program as user friendly as possible, we also wanted to be faithful to the format of the SCID to ensure that the published SCID validations would also apply to the software program. We wanted to try to assure that researchers could switch to the SCID software in the middle of long clinical trials without encountering adverse effect or confounding their previously collected SCID data. Thus, our challenge was to preserve the integrity of the paper SCID while adding the enhancements that only become possible when using a software program.
Over a two-year period, we reproduced the text and branching logic of the paper SCID for DSM-IV using C# in an ASP.NET MVC environment to create the ‘NetSCID’. We also performed extensive in-house automated testing, using over 160 de-identified clinical response sets that were made available for testing purposes. All known software errors were corrected. The software functionality and GUI were subsequently tested sequentially by 2 groups, each with 4 trained interviewers. Software improvements were made after each round of testing.
At Centerstone, a large not-for-profit provider of community mental health services in Tennessee, a total of 230 outpatient participants were recruited to test our branching algorithms across the 43 SCID diagnoses, to test inter-rater reliability, and to assess clinician satisfaction in two clinical studies. Centerstone study staff recruited an adult sample: age 18 to 75 (m = 44.54, sd = 12.28) years, 64% female, 3.3% Hispanic, 83.3% White, 10% African American, 1.3% Asian, and 1.3% American Indian/Alaskan Native; additionally 4.1% of participants reported being of more than one race. All participants underwent a full informed-consent process and provided written consent prior to engaging in any study procedures; all study and consent procedures were IRB-approved prior to the commencement of participant enrollment.
Administration of the SCID for DSM-IV and NetSCID-IV was conducted by eight clinicians. Four of the interviewers were (non-research) outpatient clinicians at Centerstone, a public sector facility, who volunteered to take a four-day training program to learn how to administer the SCID. Four additional clinicians were experienced SCID interviewers who had conducted research using the SCID for at least three years. All clinicians received four hours of NetSCID training. The paper SCID and NetSCID were simultaneously administered to the first 150 mental health outpatients recruited. Each outpatient was interviewed simultaneously by two clinicians in a private study room at Centerstone; one clinician interviewed the participant using the paper SCID and the other clinician followed the interview using the NetSCID. The clinician pairings were assigned randomly for each administration. Only the clinician using the paper SCID administered the SCID interview to the patient (i.e., read the questions aloud). We chose to have the paper-SCID interviewer ask the questions because the paper SCID represents the gold standard for behavioral health diagnosis and our goal was to assure that the electronic SCID achieved the same fidelity as the paper SCID. After each question, the paper SCID interviewer advised the second clinician of his or her rating, and the two clinicians discussed any disagreements. Once the final rating was agreed upon, both clinicians recorded the corresponding numeric value: the paper SCID interviewer on paper and the NetSCID interviewer on screen. The numeric value and the number of the new item were then reported by the paper interviewer and confirmed by the NetSCID interviewer. All differences in numeric response entry, and branching were discussed briefly until the source of the error was identified and recorded. Differences were categorized into 4 groups: (1) paper SCID data-entry error, (2) NetSCID data-entry error, (3) paper SCID branching/calculation error, and (4) NetSCID branching/calculation error. All errors were corrected before administration of the next item.
Out of 150 paired SCID administrations there were 90 total data-entry errors on the paper SCID compared to 80 total NetSCID data-entry errors. Most of these errors were made by newly trained SCID interviewers. There were also a total of 143 branching/calculation errors. Branching/calculation errors are the most prevalent type of error reported by SCID users and are the most likely to lead to misdiagnosis. A branching error was defined as a mismatch between the branch selected by the paper SCID interviewer and the NetSCID interviewer. In these situations both clinicians attempted to resolve the disagreement before moving on to the next item. Although there were an equal number of administrations by experienced and inexperienced clinicians, 139 (97%) of the 143 branching and calculation errors were made by interviewers administering the paper SCID. Of these 139 errors, 132 (95%) were made by recently trained SCID interviewers. Although the great majority of errors were made by recently trained SCID interviewers, still 7 (5%) of the branching and calculation errors were made by very experienced SCID interviewers. The TeleSage programmers subsequently fixed the four branching errors in the NetSCID code.
Interviews were conducted with 80 Centerstone participants by 16 clinicians. Eight of the interviewers were (non-research) outpatient clinicians at Centerstone, a public sector facility, who volunteered to take a four-day training program to learn how to administer the SCID. An additional eight individuals were experienced SCID interviewers who had conducted research using the SCID for at least three years. Each clinician was asked to administer the paper SCID for DSM-IV and the NetSCID to pairs of mental health outpatients who were matched based on their primary chart diagnosis. Since chart diagnoses were not necessarily deemed to be accurate and secondary diagnoses might also have varied, we also employed randomized participant assignments. This procedure was intended to assure that, in aggregate, the interview administration experiences for each modality were similar. After each interview, clinicians were asked to complete a brief Likert-scale satisfaction survey specific to the SCID administration modality used -- e.g. I found it easy to administer the ‘NetSCID’ or ‘Paper SCID’, (Strongly Agree – Strongly Disagree). In addition, clinicians were asked to complete a brief relative-satisfaction survey at the end of the study -- e.g. “Overall it was easier to use the…” (Paper SCID or NetSCID).
Specific administration modality satisfaction results gathered from the clinicians indicated a significant preference for the NetSCID over the paper SCID. Specifically, clinicians found the NetSCID easier to administer (t=2.71, p<.05), easier to navigate (t=2.62, p<.05), and easier to read (t=3.47, p<.01). Clinicians reported that it was easier to correct mistakes when using the NetSCID (t=2.98, p<.01). Regarding generation of diagnoses, clinicians found the automated NetSCID scoring system was simpler than the manual scoring option (t=5.90, p<.01). They also described the NetSCID’s highly formatted diagnostic report as easier to interpret than the paper SCID score sheet (t=3.09, p<.01). Overall, clinicians preferred the NetSCID (t=3.51, p<.01). Clinicians were also asked to complete a modality comparison survey after their last interview. When asked directly to compare the NetSCID and paper SCID, 14 out of 16 clinicians (88%; p<0.05) indicated that they preferred using the NetSCID over the paper SCID. The majority of clinicians reported that the NetSCID was easier to use (75%), easier to navigate (88%), and that they made fewer mistakes with the NetSCID (88%); these differences were all shown to be statistically significant (i.e., p≤0.05).
Across studies 1 and 2, we found that branching errors, the most common type of SCID-related error, were made overwhelmingly by recently trained interviewers. At the same time, experienced SCID interviewers with hundreds of hours of experience also made branching errors that could have led to misdiagnosis. The automated branching of the NetSCID was designed to eliminate the potential for branching errors. Newly trained interviewers also tended to make far fewer data-entry errors with the NetSCID when compared with the paper SCID. Both recently trained and experienced SCID interviewers overwhelmingly reported the NetSCID was easier to use than the paper SCID. Almost 90% of our raters preferred the NetSCID to using the paper SCID. The size of our sample, both raters and clients, and the real-life nature of the assessments that were conducted add to the strength of the current study. The principal limitations of this study include our lack of administration-time data and client satisfaction feedback.
The NetSCID reduces both data-entry and branching errors, which may also reduce the number of diagnostic errors associated with SCID administration. The NetSCID also leads to greater clinician satisfaction compared to the paper SCID. Importantly, if the NetSCID is widely used, and some researchers are willing to take the time necessary to administer individual unbranched NetSCID modules, the application of ‘big data’ analyses to the NetSCID database may ultimately facilitate improvements in diagnostic nosology.
Since completion of this study, trained researchers have completed over 4,000 NetSCID-IV administrations. Many of these administrations involved rapid data entry from existing paper forms by external research groups which served as independent validations of the branching algorithms. No additional NetSCID branching errors have been identified and all of the branching errors that were identified earlier were permanently corrected in the NetSCID-IV code. De-identified data from administrations has been made available to researchers through the NIMH Data Archive. Clinicians provided anecdotal reports that the NetSCID can reduce SCID administration times by 30%. The time savings is a result of automated branching and scoring that eliminate the need for clinicians to count symptoms, make calculations, or find the next item for administration; scoring is also instantaneous. Researchers who currently keep SCID data in an electronic format for analytic purposes report that they save about an hour in data-entry time with each administration. Lastly, the NetSCID has already been configured for use with DSM-5. In-house and external testing of the NetSCID-5 by independent research groups will be complete by November, 2015.
This research was supported in part by a grant from the National Institutes of Health (MH080463) awarded to TeleSage, Inc.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.