|Home | About | Journals | Submit | Contact Us | Français|
In this study, we examined the preliminary concurrent validity of a brief version of the Prodromal Questionnaire (PQ-B), a self-report screening measure for psychosis risk syndromes. Adolescents and young adults (N=141) who presented consecutively for clinical assessment to one of two early psychosis research clinics at the University of California, San Francisco and UC Los Angeles completed the PQ-B and the Structured Interview for Prodromal Syndromes (SIPS) at intake. Endorsement of three or more positive symptoms on the PQ-B differentiated between those with prodromal syndrome and psychotic syndrome diagnoses on the SIPS versus those with no SIPS diagnoses with 89% sensitivity, 58% specificity, and a positive Likelihood Ratio of 2.12. A Distress Score measuring the distress or impairment associated with endorsed positive symptoms increased the specificity to 68%, while retaining similar sensitivity of 88%. Agreement was very similar when participants with psychotic syndromes were excluded from the analyses. These results suggest that the PQ-B may be used as an effective, efficient self-report screen for prodromal psychosis syndromes when followed by diagnostic interview, in a two-stage evaluation process in help-seeking populations.
A growing body of research has demonstrated that individuals at “ultra-high-risk” (UHR) for psychosis can be reliably diagnosed using clinical interviews such as the Structured Interview for Prodromal Syndromes (SIPS) (Miller, et al, 2003) and the Comprehensive Assessment of At-Risk Mental States (CAARMS) (Yung, et al, 2005). Individuals diagnosed with UHR syndromes develop full psychotic disorders at a rate that ranges from 16% to 35% within 2 – 2.5 years (Cannon, et al, 2008; Yung, et al, 2007; Yung, et al., 2008). Although these interviews are indispensable in diagnosing prodromal psychosis, clinicians need specialized training to use them and they take several hours of clinicians’ and patients’ time. Currently, assessment with these instruments is only available in a small number of specialty clinics around the world.
In order to increase efficiency of identifying psychosis risk, we previously developed the Prodromal Questionnaire (PQ), a 92 item-self-report measure intended to be used in a two-stage screening process, followed by prodromal syndrome interviews. In a sample of young people referred to a prodromal psychosis research clinic, the PQ showed moderate concurrent validity with SIPS diagnoses, with 90% sensitivity and 49% specificity (Loewy, et al, 2005).
Recently, we modified the PQ to improve efficiency and accuracy. We focused on only positive symptom items, as those are the basis for interview-based diagnoses of symptomatic prodromal syndromes, and we assessed the frequency of each experience and presence of related distress or impairment. In the general population, psychotic-like experiences can be present in up to 20% of adults, often in the absence of a full psychotic disorder (Hanssen, et al, 2003). In that study, risk for later psychotic disorder was four to five times greater when individuals were distressed by the psychotic experience compared to those who were not. Undergraduate students endorsed PQ items at very high rates in our own study, but fewer endorsed items as distressing or impairing (Loewy, et al, 2007).
Although the ultimate target group for the PQ-B is the general help-seeking population, the first step of measure development is to assess preliminary validity of the PQ-B in a selected help-seeking group that is highly “enriched” for the target diagnoses (McGorry, et al, 2003). In the current study, we administered the PQ-B along with the SIPS to all adolescent and young adult patients consecutively presenting to two prodromal psychosis research clinics in California. We hypothesized that: 1) The PQ-B would show good concurrent validity with symptomatic syndromes on the SIPS, similar to the original PQ and 2) Assessing frequency of experiences and related distress/impairment would improve specificity of the PQ-B related to these SIPS diagnoses.
Study participants were 141 individuals age 12-35 who presented consecutively for evaluation at one of two prodromal psychosis research clinics: the Prodrome Assessment, Research and Treatment program at the University of California, San Francisco (UCSF) (N=47) and the Staglin Music Festival Center for Assessment and Prevention of Prodromal States at the University of California, Los Angeles (UCLA) (N=94). Subjects were referred from community clinicians, schools, family members, and self-referred from seeing information about the programs on the internet. Participants at the two sites did not significantly differ from each other on any demographic, psychosocial functioning or diagnostic grouping variables (see Table 1.)
A sample of age-matched healthy control participants (HCs) at both sites were recruited for comparison to the patient group through advertisements placed on websites and at local schools. HCs (N=46) were not significantly different from the patient group on age, ethnicity or socioeconomic status, as measured by years of parental education, but had a higher proportion of females than the patient group (p=.045). The control subjects at UCLA were assigned GAF scores, which were significantly higher than those of the patient group, as expected (p<.0001). Details of demographic characteristics are presented in Table 1.
The SIPS is semi-structured interview designed to be administered by trained clinicians (Miller, et al, 2003). The interview includes a biopsychosocial history and ratings along four major symptom dimensions on the Scale of Prodromal Symptoms (SOPS): positive, negative, disorganized and general/affective symptoms. The SIPS/SOPS diagnoses three types of prodromal syndromes, listed in order of typical sample prevalence: 1) Attenuated Positive Symptom Prodromal Syndrome (APS): Attenuated positive psychotic symptoms present at least once per week, started or worsened in that past year (unusual thought content/delusional ideas, suspiciousness/persecutory ideas, grandiosity, perceptual abnormalities/distortions, and conceptual disorganization; 2) Brief Intermittent Psychosis Prodromal Syndrome (BIPS): Brief and intermittent fully psychotic symptoms that have started recently; 3) Genetic Risk and Deterioration Prodromal Syndrome (GRDS): Either a family history of a psychotic disorder in any first-degree relative and decline of at least 30% in the past 12 months on the GAF scale, or, meets criteria for schizotypal personality disorder and has had a decline of 30% on the GAF in the past year.
After SIPS assessment, 44% of subjects were diagnosed with a UHR syndrome, 42% were diagnosed as being fully psychotic, and 13% received no psychotic-spectrum diagnosis. Among UHR subjects, 39 (95%) met APS criteria and two (5%) met BIPS criteria. One GRDS subject was excluded from analyses, as the PQ-B is intended to capture symptomatic at-risk syndromes.
The PQ-B was developed from the original 92-item Prodromal Questionnaire. First, we retained only positive symptom items, as these constitute the basis for symptomatic UHR diagnoses (APS & BIPS). Second, we analyzed the original clinic-referred UCLA sample and selected the positive symptom items with the greatest agreement with SIPS diagnoses. Third, we removed items endorsed by a large proportion of a general undergraduate university sample, as these items were assumed to be easily misunderstood and overendorsed (Loewy, et al, 2007). This resulted in 18 positive symptom items, two of which were slightly re-worded for clarity. We added five more positive symptom items to assess suspiciousness (2 items), grandiosity (2 items) and disorganized communication (1 item), which were under-represented on the PQ-B relative to items inquiring about unusual thinking and perceptual disturbances. Finally, we added one item on social functioning and one item on academic/occupational functioning. Following each individual item, we included two Likert scale follow-up questions that had been used previously in the undergraduate sample, inquiring about frequency and related distress or impairment. See Appendix A for a copy of the PQ-B and Appendix B for details on scoring.
Participants or their parents (for subjects age 12-17) completed a brief phone screen prior to being scheduled for a clinic intake in order to exclude cases of well-established psychosis, mental retardation, substance dependence, and neurological disorders such as temporal lobe epilepsy. Upon arrival at the clinic, participants provided informed consent or assent with parental consent for the study, then completed the PQ-B, followed by the SIPS. Whenever possible, collateral information was obtained by interviewing parents, significant others and relevant clinicians.
A sample of age-matched healthy control participants (HCs) at both sites completed the PQ-B and the SCID to rule out the presence of current Axis I diagnoses. Healthy control subjects at UCLA (N=26) also completed the SIPS.
Clinical interviewers at both sites were MA, PhD or MD- level clinicians who underwent a standard training procedure. Inter-rater reliability was excellent at both sites; ICCs for UCSF staff were 0.94 for SIPS diagnoses and 0.70 to 0.97 for SOPS ratings. All ICCs were above 0.80 at UCLA. Participant diagnoses were discussed in regular reliability rounds to limit rater drift. All study procedures were approved by the human subjects review committees at UCSF and UCLA.
Subjects with more than 6 items left unanswered on the PQ-B were excluded from the analyses (N=6; 4%). Next, remaining missing data were coded as no (0), based on informal questioning of patients across several studies of the PQ, which suggested that blank items nearly always indicated that participants had not experienced that symptom. Missing data for frequency and distress were also coded as 0, in accordance with how the measure would be used in actual practice. Distributions of PQ-B and SIPS scores were examined for violations of normality assumptions. All scores were skewed towards 0, as expected, and therefore non-parametric statistics were calculated as necessary.
Agreement between PQ-B scores and SIPS diagnoses was used to assess concurrent validity by generating receiver operating characteristic (ROC) curves and calculating areas under the curve (AUCs). Values for sensitivity, specificity, positive predictive value, negative predictive value and likelihood ratios were computed. Correlation analyses were performed between PQ-B scores and SOPS positive symptom scores using Spearman’s correlation coefficient. Cronbach’s coefficient alpha was used to examine internal consistency of the PQ-B. The Kruskal-Wallis test is a non-parametric rank test that was used to compare PQ-B scores across the three SIPS diagnostic groups (no SIPS diagnosis, prodromal syndrome, psychotic syndrome). When significant differences were detected across groups, post-hoc Mann-Whitney U tests then examined paired group contrasts.
All PQ-B scores predicted SIPS diagnoses of prodromal/psychotic syndromes versus no SIPS diagnosis with statistically significantly AUC values. The two functioning items did not improve prediction above and beyond the 23 positive symptom items. Furthermore, only 3 of the 5 positive symptom items that were added to the original 18 items (which emerged from the analyses described in section 2.2b above) provided additional predictive power above and beyond the 18 items. Therefore, the Total, Distress and Frequency scores were calculated using only the 21 positive symptom items. Content of the items on the measure that were excluded from scoring are included in Appendix B, along with scoring details.
A Total Score of 3 or more endorsed items balanced the greatest sensitivity (89%) with the greatest specificity (58%) and had a positive Likelihood Ratio of 2.12. Comparatively, the Frequency score lost substantial sensitivity, while the Distress score heightened specificity. Table 2 presents the cutoff values and accuracy of the PQ-B scores that showed the most agreement with SIPS symptomatic syndromes. Compared to the Total and Frequency scores, the Distress score with a cutoff of 6 or more showed the greatest specificity (68%) while retaining high sensitivity (88%); its performance was very similar when fully psychotic patients were excluded from the analysis. The Distress Score cutoff of 6 or more showed similar sensitivity across sites (87-90%) with some variation in specificity (63-73%), although all scores overlapped within their 95% confidence intervals. Figure 1 compares the ROC curves for the best-performing scores.
Total Score on the PQ-B was significantly correlated with all SIPS/SOPS scores at the level of p<.0001 (two-tailed, no correction for multiple comparisons), including positive symptoms (r=0.65), negative symptoms (r=0.50), disorganized symptoms (r=0.59), and general symptoms (r=0.60). The Distress Score was also significantly correlated with all SIPS/SOPS scores at the level of p<.0001 (two-tailed, no correction for multiple comparisons), including positive symptoms (r=0.60), negative symptoms (r=0.52), disorganized symptoms (r=0.59), and general/affective symptoms (r=0.65). Cronbach’s alpha for the Total score was 0.853. Finally, Kruskal-Wallis tests showed significant differences of PQ-B scores across SIPS/SOPS diagnostic groups for both the Total Score (p< .0001), and the Distress Score (p<.0001). Post-hoc paired contrasts revealed that Total Scores were significantly lower in the group with no SIPS diagnosis compared to the prodromal syndrome group (p<.0001) and psychotic group (p<.0001), but the prodromal and psychotic groups did not significantly differ from each other (p=.091). Similarly, the Distress Scores were significantly lower in the group with no SIPS diagnosis as compared to the prodromal syndrome group (p<.0001), and psychotic group (p<.0001), but the prodromal and psychotic groups did not significantly differ from each other (p=.10). Figures Figures33 and 4 show the mean PQ-B scores and their distributions across groups.
Six subjects out of 46 healthy control participants (13%) scored above the PQ-B Total Score cutoff of 3 or more endorsed items; the highest score amongst control subjects was seven. Five of these subjects (11%) also scored above the cutoff of 6 or more on the Distress Score. Five of the six Total Score high-scorers were seen at UCLA and none of them received a psychotic or prodromal syndrome diagnosis on the SIPS. Healthy controls at UCSF did not receive the SIPS.
Overall, the PQ-B showed good preliminary concurrent validity with interview-based SIPS diagnoses in our help-seeking sample of adolescents and young adults. It effectively differentiated between participants with SIPS diagnoses of prodromal and psychotic syndromes versus non-psychotic spectrum patients. The brief version of the PQ maintained the sensitivity of the original, while adding questions about related distress and impairment that improved specificity. However, assessing frequency of experiences or asking about functioning was not additionally helpful. The false-positive rate of 11% when using the Distress Score in the healthy control group suggests that performance of the PQ-B is similar across samples; however, this rate would result in a great number of interviews if screening large samples. Taking these results altogether, we recommend only using the PQ-B in help-seeking samples, especially with those individuals who are already suspected to have attenuated psychotic symptoms.
As discussed previously (Loewy, et al, 2005), this instrument is designed to function as the first step in a two-stage screening process that relies on clinician interview to obtain a diagnosis. Therefore, PQ-B users should be careful not to equate a high score with prodromal psychosis or unavoidable development of schizophrenia. In order to minimize unnecessary stigma and distress that may be associated with the diagnosis of a an attenuated psychosis syndrome (Yang, et al., 2010), we recommend discussing results in light of the need for a more thorough clinical interview and the importance of early detection and intervention in improving long term outcomes (Corcoran, et al., 2010).
In order for the PQ-B to be used in a general help-seeking population, future studies should assess concurrent validity of the two-stage screening process and in a general mental health sample. The results of the current study suggest that pursuing such a study is warranted. We have an ongoing study tracking 2-year outcomes of the SIPS-positive participants, to assess predictive validity of the PQ-B. Pending the outcome of future studies in unselected samples, we hope that the PQ-B can be used to increase access to care for help-seeking youth whose attenuated psychotic symptoms might otherwise go unnoticed or misdiagnosed.
Role of the Funding Source This work was supported by a Young Investigator award from the National Alliance for Research on Schizophrenia and Depression (NARSAD) and NIH grant K23 MH086618 to Dr. Loewy; a gift from the Lazslo N. Tauber Family Foundation to Dr. Vinogradov; and NIH Grants MH65079 and P50 MH066286 to Dr. Cannon. Additional support was provided by gifts to the UCLA Foundation by Garen and Shari Staglin and the International Mental Health Research Organization. The funding sources had no further role in study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the paper for publication.
We would like to thank Ashley Lee, Tara Niendam and Danielle Schlosser for their assistance in coordinating this project, and the reviewers for their thoughtful comments on the manuscript.
*Conflict of Interest Dr. Cannon is a consultant for Rules-Based Medicine on diagnostic biomarkers for mental disorders. The other authors declare that they have no conflicts of interest.
Contributors Dr. Loewy held primary responsibility for the study design, data collection, data analysis and manuscript writing. Ms. Pearson contributed to data analysis and manuscript writing. Dr. Vinogradov served as a senior mentor to Dr. Loewy and contributed to implementation of the study and manuscript writing. Dr. Bearden supervised data collection at UCLA and contributed to manuscript writing. Dr. Cannon contributed to the study design, manuscript writing and served as site PI at UCLA.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.