|Home | About | Journals | Submit | Contact Us | Français|
Individuals who report psychotic-like experiences are at increased risk of future clinical psychotic disorder. They constitute a unique “high-risk” group for studying the developmental trajectory to schizophrenia and related illnesses. Previous research has used screening instruments to identify this high-risk group, but the validity of these instruments has not yet been established. We administered a screening questionnaire with 7 items designed to assess psychotic-like experiences to 334 adolescents aged 11–13 years. Detailed clinical interviews were subsequently carried out with a sample of these adolescents. We calculated sensitivity and specificity and positive predictive value (PPV) and negative predictive value (NPV) for each screening question for the specific symptom it enquired about and also in relation to any psychotic-like experience. The predictive power varied substantially between items, with the question on auditory hallucinations (“Have you ever heard voices or sounds that no one else can hear?”) providing the best predictive power. For interview-verified auditory hallucinations specifically, this question had a PPV of 71.4% and an NPV of 90.4%. When assessed for its predictive power for any psychotic-like experience (including, but not limited to, auditory hallucinations), it provided a PPV of 100% and an NPV of 88.4%. Two further questions—relating to visual hallucinations and paranoid thoughts—also demonstrated good predictive power for psychotic-like experiences. Our results suggest that it may be possible to screen the general adolescent population for psychotic-like experiences with a high degree of accuracy using a short self-report questionnaire.
Psychotic-like experiences are more prevalent in the general population than clinical psychotic disorders, suggesting that these experiences may occur as part of a nonpathological phenotype, which lies on a continuum with psychotic disorder.1–3 A recent meta-analysis by van Os et al4 reported a median prevalence of about 5% for psychotic experiences in the general population. Data on adolescent populations indicate that rates of psychotic-like experiences may be even higher among this age group.1,5–8
Research on long-term outcomes suggests that individuals who report psychotic-like experiences are at significantly increased risk of clinical psychotic disorders.1,4,9 Among a New Zealand birth cohort, Poulton et al1 found that 11-year olds who reported psychotic symptoms were at a 5- to 16-fold specific increased risk of adult psychotic disorder. Welham et al10 have also documented that self-reported psychotic-like experiences (specifically, auditory hallucinations) at age 14 years are associated with increased risk for adulthood psychotic disorder, assessed aged 21 years. Hanssen et al9 found that, among a general population sample of 18- to 64-year olds, the 2-year transition rate to clinical psychotic disorder was more than 60 times higher for those who had previously reported psychotic-like experiences compared with those without incident psychotic experiences. Individuals who report psychotic-like experiences, then, represent a valuable population in studying the developmental trajectory to schizophrenia and related illnesses. This group can be conceptualized as representing a “pre-prodromal” population and provides a complementary approach to the ultra-high-risk prodromal strategy, which was pioneered in Melbourne, Australia, and has since been adopted in several centers worldwide.11–16 An accurate and rapid screening instrument for identifying this population would be valuable.
Many articles have reported on psychotic-like experiences as assessed by screening interviews or screening questionnaires.6,7,17–21 However, there have been mixed results from the 2 studies to date that have reported on the validity of these screening questions among adolescent populations. Laurens et al7 found only moderate agreement between their screener questionnaire and the results of a subsequent clinical interview in a random sample of their screened population. They reported kappa scores of between 0.16 and 0.65 for agreement between clinician and child on each of the screening questions, with most falling into either the “fair agreement” (0.21–0.40) or the “moderate agreement” range (0.41–0.60). Horwood et al21 also found that endorsement of their screener questions did not generally predict genuine psychotic symptoms when followed up by further probe questions.21 Positive predictive power for most questions was no more than 50%. One item, however, displayed some promise in terms of its positive predictive power; a screening question on “auditory hallucinations” attained a positive predictive value (PPV) of 70.6%, suggesting that such a question might be useful in identifying adolescents with psychotic-like experiences in the general population. However, due to the study design, they were not able to give any indication of the sensitivity of this question. Thus, the literature to date suggests a high rate of false positives when screening for psychotic experiences and gives no indication of the rate of false negatives. Further research is necessary to determine whether it is possible to sensitively and specifically screen for adolescents with psychotic-like experiences in the general population using self-report questionnaires.
Nineteen primary level schools in Dublin, Ireland, were contacted to take part in the study. The schools contacted were large, mixed-gender, state primary schools from the northern and western areas of the city. No private fee-paying schools were contacted. The principal teacher (headmaster or headmistress) of each school was contacted by phone and was given information about the study. Eight schools agreed to take part (42.1%), 7 schools did not wish to take part (36.8%), and 4 schools said that they might take part at another time (21.2%). The reasons that school principals gave for not participating were as follows: “teachers too busy/not interested” (n=3), “child in school currently very ill” (n=1), “would not get good response from pupils” (n=2), and “school currently taking part in another research study” (n=1). There were no significant differences between the schools that participated (n=8) and the schools that refused to participate (n=7) on number of pupils (605 [SD=162] vs 577 [SD=230], t=0.27 (df=13), P=0.8) or Dublin area postcode (proxy for socioeconomic status). However, 2 of the schools that declined to take part had “disadvantaged status,” indicating that the school was in a designated area of disadvantage (for details, see www.education.ie). None of the participating schools had “disadvantaged status.”
We wished to assess a variety of psychotic symptoms in a brief period of time and devised a 7-item Adolescent Psychotic-Like Symptom Screener (APSS). We chose 4 questions from the Diagnostic Interview Schedule for Children1,22 and added 3 further questions on visual hallucinations, delusions of control, and grandiosity in order to broaden the types of symptoms assessed.
For each question, there were 3 possible responses: “Yes, definitely,” “Maybe,” “No, never.” The questionnaire was scored as follows: yes, 1 point; maybe, 0.5 point; no, 0 point. Total scores were calculated for each child by summing the scores from each of the 7 questions. We made an a priori decision to rank those with scores of 2 or more as part of an “at-risk” group for psychotic-like experiences.
The screening took place in a 2-stage format. The study team visited the schools that agreed to take part and gave a short information session about brain development in childhood and adolescence to fifth and sixth classes (ie, pupils in the 2 most senior classes aged 11–13 y). Consent forms for the study were distributed for the children to take home to their parents. On the form, parents were asked to tick 1 box to indicate consent for their child to complete the questionnaire in the classroom. Parents were also asked to tick a second box and leave contact details if they would like to hear about the second stage of the study involving an interview and further testing. The study team returned to the school 1 week later to collect the completed forms and administer the questionnaire in the classrooms. The questionnaire took just a few minutes for each child to complete. They completed the questionnaires in the classrooms in small groups with the researcher present. For the sake of confidentiality, pupils were asked to complete the questionnaire without allowing anyone else to see their answers but could ask the researcher for clarification of any question that they did not understand.
The interview instrument used in this study was the Schedule for Affective Disorders and Schizophrenia for School-Aged Children, Present and Lifetime Versions (K-SADS).23 The K-SADS is a well-validated semistructured research diagnostic interview for the assessment of all Axis-1 psychiatric disorders in children and adolescents. Full K-SADS interviews were carried out with each adolescent and his/her parent. Children and parents were interviewed separately, both answering the same questions about the child. Interviews were conducted by 2 psychiatrists and 1 psychologist trained in the use of the K-SADS and involved detailed assessment of between 2 and 4hours, depending on the level of symptomatology reported.
The psychosis section of the K-SADS was used to enquire about psychotic-like experiences, both current and past. Positive responses to any of the screening questions were followed by detailed probing. All participants who reported psychotic symptoms during the screening section were subsequently assessed using the detailed psychosis supplement of the K-SADS, and extensive notes were taken during the interview.
A consensus meeting was held in order to evaluate the interview results for all participants once the interviews were completed. Participants were rated as having “definite” psychotic-like experiences if there was agreement between all 3 raters (I.K., M.H., and M.C.).
In order to compare the adolescents’ self-report responses with the interview results, we calculated the sensitivity, specificity, PPV, and negative predictive value (NPV) of each screening question for (a) the specific psychotic-like experience about which the question enquired (eg, auditory hallucinations) and (b) any psychotic-like experience at all. For the purposes of our calculations, endorsement of a question was an answer of “Yes, definitely” (as opposed to an answer of “Maybe” or “No”). Calculations were performed using STATA version 9.2 for Windows. These analyses were weighted to account for the greater proportion of the at-risk group who attended for interview compared with controls. All values reported in this article are weighted values.
In total, 725 consent forms were distributed, and 334 pupils completed the questionnaires (46.1% consent rate). Consent rates between schools varied between 27% and 79%. We have no information on the pupils who did not agree to take part in the study.
Of the 334 children, whose parents consented them to participate in the screening study, 82 did not give consent to be contacted about the second part of the study—the clinical interview—and 21 did not have English as a first language. Therefore, 231 children were available to be contacted about the interview part of the study. Of our 231 children, 77 had scores of 2 or more (the a priori at-risk group), and the remainder (n=154) had score of less than 2 (we called these the “control group”). We contacted the parents of all the children in the at-risk group by sending a letter inviting them to participate in the interview, part of the study. In total, 55 either refused or did not answer, and 22 (28.6% of the at-risk group) were interviewed. We contacted 90 randomly selected families in the control group of whom 68 either refused to participate or did not answer, and 22 agreed to take part in the interview study (24.4%) (see figure 1 for details).
There was no significant difference in consent rates for participation in interview between the at-risk group and the “control group” (χ2=0.36, df=2, P=.54). Within the a priori at-risk, group there was no significant difference in mean screener scores between those who agreed to come for interview and those who did not agree to come for interview (mean score=3.86 vs 3.31) (t=1.7, df=75, P=.093). Within the control group, there was no significant difference in mean screener scores between those who agreed to come for interview and those who did not agree to come for interview (mean score=0.5 vs 0.38) (t=1.08, df=88, P=.28). There was also no significant difference in mean screener scores between the control group and those who met criteria for control group who were not contacted for follow-up interview (mean score=0.5 vs 0.71) (t=−1.44, df=84, P=.15).
Overall, 38.3% of the total sample screened answered, “Yes definitely” to at least one screening question, and 73.7% answered “Yes definitely” or “Maybe,” to at least one screening question. A total of 50% answered, “Yes definitely” or “Maybe” to 2 or more questions, and 33% answered “Yes definitely” or “Maybe” to 3 or more questions. See figure 2 for distribution of scores on the questionnaire.
In total, 44 adolescents and their parents attended for interview—22 from the at-risk group and 22 control subjects (score less than 2). Nine children received a total of 12 lifetime diagnoses following K-SADS interview: social phobia (n=3), obsessive compulsive disorder (n=3), adjustment disorder with depressed mood (n=2), simple phobia (n=2), conduct disorder (n=1), and avoidant disorder of childhood (n=1). No participant received a diagnosis of a psychotic illness, but many disclosed psychotic-like experiences during the interview. Following the consensus meeting, 17 participants were considered to have had a definite psychotic-like experience and 22 to have had no psychotic-like experiences. No consensus was reached for 5 participants, and they were included with the “no psychotic-like experiences” group for the purpose of calculating how sensitive and specific the screening questionnaire was for identifying adolescents with definite psychotic-like experiences. For each participant who took part in the study, a parent was also interviewed using the K-SADS. In only one of the 17 cases of definite psychotic-like experiences, did a parent report that their child had experienced these types of symptoms during interview, indicating that parental report for psychotic-like experiences is inaccurate.
Table 1 displays the percentage of the “definite” group (the 17 participants found to have genuine psychotic-like experiences at interview) that endorsed each item on the screening questionnaire compared with the percentage confirmed to have these experiences at interview. Auditory hallucinations were the most common phenomenon reported by the participants with definite psychotic-like experiences at interview. The K-SADS includes parameters for excluding certain phenomena as nondiagnostic auditory hallucinations, which are believed to have little psychopathological importance, such as hearing footsteps, knocking, or one’s name. Diagnostic auditory hallucinations include the experience of hearing one or more voices saying at least one word other than one’s own name.24 Accordingly, 71% (n=12) of the participants in the definite group reported diagnostic auditory hallucinations (see table 2). These varied in frequency from a handful of discrete events over the course of their childhood to several times a week for several years. The most common phenomenon involved gedankenlautwerden (hearing one’s thoughts aloud) that was reported by two-thirds of this group (n=8). Other auditory hallucinations included command hallucinations, voices commenting on their behavior, and music.
Visual hallucinations were reported by 35% (n=6) of the adolescents in the definite group. This usually involved seeing not only people and faces (reported by 4 of the 6 participants with visual hallucinations) but also “aliens” (n=1) and strange shapes (n=2). All the adolescents who reported visual hallucinations had also experienced auditory hallucinations. Auditory hallucinations usually predated visual hallucinations. The most common attribution for these hallucinations (auditory and visual) involved “ghosts,” which was reported by 42% (n=5) of the participants with hallucinatory experiences.
All 17 adolescents in the definite group demonstrated at least some form of unusual thought content, eg, 53% (n=9) reported that they believed someone may be spying on them. Attributions for this were vague—participants were generally unable to make any suggestions as to who might be spying on them or for what reason they would do so. Rather, they reported a “feeling” that someone may be watching them, generally from within bushes or hedgerows. Nearly 30% (n=5) of participants with definite psychotic-like experiences reported that they had the ability to predict the future; this belief was held with unshakable conviction, and participants reported that their success in predicting the future could not have been a coincidence.
We analyzed the interview findings with respect to the questionnaire results and found that 14 of the 17 participants (82%) in the definite group had met the a priori at-risk score of 2 or more on the screener questionnaire. Four of the 22 (18.2%) who reported no psychotic-like experiences at interview had scored 2 or more, and 4 of the 5 participants (80%) in the “no consensus” group scored 2 or more. Therefore, our a priori score of 2 or more detected adolescents with definite psychotic-like experiences with a sensitivity of 70% (95% confidence interval [CI]=45.7–88.1) and a specificity of 82.6% (95% CI=68.6–92.2).
We calculated sensitivity, specificity, NPV, and PPV for an answer of “Yes, definitely” on each of the 7 items in our screener questionnaire. Table 3 shows the predictive power of each of the 7 items for the specific type of experience about which they ask (eg, did participants who endorsed the screening question on auditory hallucinations have clinically verifiable history of auditory hallucinations at interview). Table 3 also shows the predictive power of each of the 7 items for any psychotic-like experience (eg, did participants who endorsed the screening question on auditory hallucinations have clinically verifiable history of any psychotic-like experience, including, but not limited to, auditory hallucinations).
The single item with the greatest predictive power was question number 4 (Q4), “Have you ever heard voices or sounds that no one else can hear?” It provided a PPV of 71.4%, and an NPV of 90.4% for a clinically verifiable history of auditory hallucinations. This item was also the most predictive of any psychotic-like experience on interview. All participants who endorsed this item on the screener had clinically verifiable psychotic-like experiences at interview (PPV=100%). Furthermore, a relatively small number of adolescents with clinically verifiable psychotic-like experiences did not endorse this item (NPV=88.4%). This screening item was endorsed (answer of “Yes, definitely”) by 15.6% of the total screened sample.
Two further questions demonstrated moderately successful predictive power for interview-verifiable psychotic-like experiences: question 6 (“Have you ever seen things that other people could not see?”) demonstrated a PPV of 81.8% and an NPV of 80%. Question 3 (“Have you ever thought that people are following or spying on you?”) demonstrated a PPV of 80% and an NPV of 78.6%. The most poorly predictive item in the questionnaire was question 2 (“Have you ever had messages sent just to you through TV or radio?”), which demonstrated a PPV of 40% and an NPV of 71.4%.
Children and adolescents who experience psychotic-like experiences represent a valuable population for studying the developmental trajectory to clinical psychosis.1,4,9 However, conducting large numbers of clinical interviews in order to identify satisfactory numbers for study is problematic. Our results show that it may be possible to accurately identify adolescents who have experienced these types of symptoms using a rapidly administered, self-report questionnaire, particularly using a question on auditory hallucinations. By prescreening, researchers could quickly identify adolescents with a likely history of psychotic-like experiences.
As in the current study, Horwood et al21 found that a screening question on auditory hallucinations had the greatest predictive power of any of the screening questions (PPV=70.6%, compared with PPV=71.4% in the present study). Their analysis differed from ours, however, in that they looked only at the relationship between individual screening questions and their specific corresponding symptoms (that is, eg, did participants who endorsed the question on visual hallucinations actually report visual hallucinations when probed). Their interview schedule was not designed to assess for false negatives, unlike the current study. Our study design allowed us to calculate the predictive validity of the screener questions for any psychotic symptom at interview, which increased the ability of our screener to detect psychotic symptoms. Thus, while the question on auditory hallucinations had a PPV of 71.4% for auditory hallucinations, it had a PPV of 100% for psychotic-like experiences more generally.
Two further questions demonstrated relatively high positive and negative predictive values for participants with a genuine history of psychotic-like experiences: question number 3 (“Have you ever thought that people are following or spying on you?”), and question number 6 (“Have you ever seen things that other people could not see?”). However, the other screening questions were of more limited use in identifying adolescents in the general population with genuine psychotic-like experiences.
This study has a number of strengths. It involved detailed clinical interviews that allowed us to obtain rich data on the phenomenology of psychotic-like experiences in adolescents. The need for rich phenomenological data has recently been highlighted as vital to the progress of schizophrenia research.25 Little work has been carried out on developmental aspects of psychopathology. These detailed interviews enabled us to investigate the types of psychotic-like symptoms that children and adolescents experience, something that does not necessarily equate with adult psychotic-like experiences. The detailed interviews carried out for the present study also allowed us to assess for false negatives, as well as false positives, something that previous research on screening instruments so far has been unable to do. As a result, this is the first study to report the sensitivity and specificity of our screening questions.
Ascertainment bias, whereby psychosis-prone families are more likely to engage in psychosis studies, was minimized by the fact that this study was not presented to participants as a study of psychosis but rather as part of a study on adolescent brain development, improving the generalizability of our findings.26
A number of limitations warrant identification. Firstly, there was a relatively high refusal rate to take part in the interview, though this did not differ significantly between the at-risk and comparison groups. Secondly, the number of interviews conducted was relatively small. The confidence intervals, as a result, are relatively wide. Because the screening was conducted in schools, there is a potential bias against sickly children, who would be more likely to be absent on the day. Given that the 2 schools we approached that met criteria for “disadvantaged” status declined to participate, our screening procedure was not fully representative of the entire school-going adolescent population. For these reasons, our findings warrant replication.
The sensitivity, specificity, PPV, and NPV of screening questionnaires for psychotic-like experiences among the general adolescent population have not previously been tested. Our findings suggest that a screening questionnaire can provide good sensitivity and specificity in identifying young adolescents with psychotic-like experiences in the general population. In particular, an answer of “Yes, definitely” to our screening question on auditory hallucinations was very predictive of interview-verifiable psychotic-like experiences and may be useful for future research on this population. The questions on visual hallucinations and paranoid thoughts/spying also proved to have good predictive power for psychotic-like experiences. These questions could be added to other established screening instruments for general psychopathology and may facilitate identification of adolescents of interest without the need for large-scale, time-consuming clinical interviews.
Health Research Board, Ireland (Clinician Scientist Award CSA/2004/1 to M.C.).
We wish to thank the Clinical Research Centre, Royal College of Surgeons in Ireland, Beaumont Hospital, Dublin, Ireland, for use of their interviewing rooms.