The present study consisted of two phases (). The first phase was aimed at selection of accurate and reproducible schizophrenia biomarkers from 181 assays comprising the Rules-Based Medicine DiscoveryMAP assay collection. Phase I resulted in the selection of 51 specific immunoassays to be used in assay validation. Phase II featured a refinement of the individual components of the multiplexed immunoassay, development of a decision rule for separating schizophrenia patients from normal controls, and validation of the decision rule using a cohort of 806 clinical samples. For biological validation of the decision rule, 480 of these samples were only analyzed during phase II of this study. The protocols for the study participants, clinical samples and test methods were carried out in compliance with the Standards for Reporting of Diagnostic Accuracy (STARD) initiative.10
Overview of the test development process.
Subjects were recruited from the Departments of Psychiatry at the Universities of Cologne (cohort 1), Muenster (cohort 2), Magdeburg (cohorts 3 and 4), Rotterdam (cohort 5) and the US military (n = 110 Bipolar Disorder patients and n = 110 controls). Cohorts used for the marker selection phase were comprised of 250 first- and recent-onset schizophrenia patients and 230 control subjects (). Schizophrenia patients of cohort 1 (n = 71), 2 (n = 46), 4 (n = 47) and 5 (n = 40) were antipsychotic-naïve and 32 out of 46 subjects from cohort 3 had not been treated with antipsychotic medication for more than 6 weeks prior to sample collection. Drug naïve patients are difficult to recruit since even large clinical facilities can only expect to diagnose about 20–30 such patients each year. To facilitate the future development of a test with differential diagnosis capability, we also carried out DiscoveryMAP analysis using samples from subjects within 30 days before their first contact with US military psychiatric services and who later received a confirmed diagnosis of bipolar disorder (BD) (n = 110, ). The cohort used to validate and implement the decision rule was comprised of samples from a mixture of first onset and chronic antipsychotic-treated schizophrenia patients along with healthy matched controls. The cohort originally consisted of a total of 838 subjects, 593 subjects diagnosed with schizophrenia and 245 matched healthy controls. During the laboratory testing of the samples, 32 samples were found to be of insufficient serum quantity, leaving a final validation population of 577 subjects diagnosed with schizophrenia and 229 healthy, matched subjects recruited at the Universities of Cologne, Muenster and Magdeburg ().
Demographic details of subjects included in phase I (biomarker selection).
Demographic details of pre-symptomatic bipolar disorder and control subjects.
Demographic details of 707 subjects used during phase II (51-plex validation). 99 patient follow up samples were available from cohort 2, yielding a total sample number of 806.
Schizophrenia was diagnosed based on the Structured Clinical Interview for Diagnostic and Statistical Manual (DSM)-IV. Patients used for phase I of this study fulfilled the criteria of the paranoid subtype (DSM-IV 295.30). All diagnoses and clinical tests were performed by psychiatrists following Good Clinical Practice guidelines. Patients whose clinical diagnosis required revision at a later stage were excluded from the study. Control subjects used in phase I of this study were matched to the schizophrenia patients for age, gender and social demographics and were recruited from the same economic and geographical area of the university districts. Controls with a family history of mental disease or with other medical conditions such as type II diabetes, hypertension, cardiovascular or autoimmune diseases were excluded from the study. Pre-symptomatic BD patients and respective controls (n = 110) were selected from a US military serum bank comprising approximately 43 million sera, which facilitated matching for age, gender, ethnicity and lifestyle.
The medical faculty ethical committees of the respective research facilities approved the protocols of the study. Informed consent was given in writing by all participants recruited at universities and clinical investigations were conducted according to the principles expressed in the Declaration of Helsinki. Blood samples were collected from all subjects between 8:00 and 12:00 hours into S-Monovette 7.5 mL serum tubes (Sarstedt; Numbrecht, Germany). The samples were left at room temperature for 2 hours to allow for blood coagulation and then centrifuged at 4000 × g for 5 minutes. The resulting supernatants were stored at −80 °C in Low Binding Eppendorf tubes (Hamburg, Germany).
DiscoveryMAP multiplex immunoassay profiling
Analytes were measured in 250 μL serum samples using the DiscoveryMAP multiplexed antigen immunoassays in the CLIA-certified laboratory at Rules-Based Medicine. Assays were calibrated using duplicate 8-point standard curves and raw intensity measurements were converted to absolute protein concentrations using proprietary software. Machine performance was verified using quality control samples at low, medium and high levels for each analyte. All standard and quality control samples were analyzed in a complex matrix to match the sample background. Serum samples were analyzed at optimized dilutions and analytes exceeding the highest concentrations on calibration curves were assigned the concentration of the highest standard, and those assayed below minimum concentrations were assigned the value 0.0. Assay reproducibility was assessed by reanalysis of the same samples approximately three months later by using Pearson’s correlation coefficients and monitoring the shift in average measurement levels.
Biomarker selection—phase I
The biomarker selection phase of the present study was aimed at identification of analytes that were altered reproducibly in schizophrenia compared to control subjects across independent cohorts (). Analytes were ranked based on the number of centers in which significant differences were observed using unpaired, two-tailed t-tests (P < 0.05). Analyte selection was guided by the following criteria: i) reproducibility (including the same directional change) in three or more centers, ii) high correlation (>0.8) and low average measurement shifts (<40%) in repeat measurement (see above) and iii) mean experimental values distant from the least detectable dose (>20 fold; LDD is defined as the average of the signal plus 3 standard deviations of 20 blank samples analyzed at the same time).
51-plex development and clinical validation—phase II
Efficient analysis of the 51 analytes required construction of new multiplexes. This procedure was guided by optimum dilution of serum and mixing of antibodies to give the most sensitive assays. The required dilutions of serum were 1:5, 1:50, 1:200, 1:10,000, and 1:200,000. The 1:5 dilution group consisted of 31 analytes which were divided into 4 multiplexes. For each higher dilution, only one multiplex was used, yielding a total of 8 new multiplexes for the 51 analytes. Once the multiplexes were created, large batches of reagents were manufactured to allow consistent testing of approximately 7000 samples. The reagents were validated using the following parameters: sensitivity, linearity, spike recovery, common serum matrix interferences, cross-reactivity, precision, correlation, freeze-thaw stability and short-term room temperature antigen stability.
Classification decision rule—design and optimization
To discriminate schizophrenia patients from controls using the markers selected in phase I of this study, we implemented a linear support vector machine (SVM) algorithm. This method minimized errors by counting each misclassified observation with a penalty parameter C. Specific penalty parameters were chosen for patients (CSZ) and controls (CNC), and the ratio F = CSZ/CNC was varied to modify the balance between sensitivity and specificity (visualised in a receiver operating characteristic (ROC) curve). Given a pair of parameters C and F, all elements of the data set were used to train the algorithm, and performance was measured using 10-fold cross validation (CV). The measured sensitivity and specificity calculated in each CV round were averaged and designated as the sensitivity and the specificity of the decision rule for the parameters C and F.
The optimization process was carried out using an in house developed code (Matlab 2009a). The search for optimal performance was performed among 3,100 pairs of parameters (C,F ) covering the following ranges: log2C = −10.0 to 0.0 with step 0.1 (100 values in total). log2F = −1.5 to +1.5 with step 0.1 (31 values in total).
We also computed the conditional probability C that a subject with a given score S is a schizophrenia patient. The computation of the conditional probability was based on the methodology developed by Vapnik
The conditional probabilities were used to augment the accuracy estimation of binary classification decision rules with various levels of confidence.