PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
J Psychiatr Res. Author manuscript; available in PMC 2010 November 29.
Published in final edited form as:
PMCID: PMC2992983
NIHMSID: NIHMS164414

Computerized experience sampling method (ESMc): Assessing feasibility and validity among individuals with schizophrenia

Abstract

The Experience Sampling Method (ESM) is an ecologically valid, time-sampling of self-reports developed to study the dynamic process of person–environment interactions. ESM with digital wristwatch and booklets (paper-based ESM; ESMp) has been used extensively to study schizophrenia. The present study is designed to test the feasibility and validity of using Computerized ESM (ESMc) among individuals with schizophrenia. ESMc is advantageous in allowing for recording of precise time-stamps of responses. We used PDAs (“Personal Digital Assistant”; Palm handheld computers) to collect data on momentary psychotic symptoms, mood, and thoughts over a one day period among 10 hospitalized schizophrenia patients and 10 healthy controls. ESMc was equally acceptable to both groups, with similar ratings of comfort carrying the PDAs and operating them, interference with daily activities, as well as response rates. The schizophrenia patients reported significantly higher ratings of auditory and visual hallucinations, suspiciousness, sense of unreality, lack of thought control, fear of losing control, difficulty expressing thoughts, as well as depression/sadness, loneliness and less cheerfulness. Significant inverse relationships were found among both groups between ratings of feeling cheerful and being stressed, irritated, and sad/depressed. Among the schizophrenia subjects, the correlation between ratings of suspiciousness on ESMc and Scale for Assessment of Positive Symptoms (SAPS) approached significance, as well as the link between suspiciousness and stress. Our results support the feasibility and validity of using ESMc for assessment of momentary psychotic symptoms, mood, and experiences among individuals with schizophrenia. The authors discuss the potential applications of combining ESMc with ambulatory physiological measures.

Keywords: Schizophrenia, Psychosis, Experience sampling method (ESM), Palm handheld computers, Validity, Stress

1. Introduction

The Experience Sampling Method (ESM) is an ecologically valid, time-sampling of self-reports developed to study the dynamic process of person–environment interactions (Delespaul, 1995). Subjects in ESM studies are typically supplied with digital wristwatch and booklets (paper-based ESM; ESMp) containing questionnaires about momentary thoughts, mood, and environmental context. The subjects are instructed to complete the questionnaire upon hearing beeps from the wristwatches, which are preprogrammed to beep randomly a number of times a day to elicit experience samples. Over the past decade, ESMp has been used extensively to study diverse psychiatric populations including individuals with schizophrenia (Delespaul, 1995; Delespaul et al., 2002; Myin-Germeys et al., 2000, 2001, 2002), psychosis (Myin-Germeys et al., 2003a,b, 2004), psychosis–proneness (Husky et al., 2004; Tournier et al., 2003; Verdoux et al., 2003), depression and bipolar disorder (Barge-Schaapveld and Nicolson, 2002; Barge-Schaapveld et al., 1995, 1999; Peeters et al., 2003; Swendsen and Compagnone, 2000; Wang et al., 2004), criminal offenders and violent psychiatric patients (Hillbrand and Waite, 1994; Hillbrand et al., 2000), as well as cannabis users (Tournier et al., 2003; Verdoux et al., 2003).

ESMp offers a number of advantages over interviews and retrospective data gathering methodologies including: (1) minimizing the potential risk of memory biases; (2) allowing ecologically valid assessment of experiences in real-world, real-time environments (i.e., in situ and in vivo); as well as (3) enabling to assess daily fluctuations in thoughts, mood, and symptoms as part of the continuous ebb and flow of person–environment interactions. However, investigations using ESMp cannot be certain of the exact time that the subjects actually completed their responses. They also do not have control over the order of the responses a subject chooses to complete on the questionnaire and the length of time they spend responding to each question. Furthermore, ESMp methodology makes it cumbersome to use branching – the presentation of different questions based on the subject’s responses to previous items (i.e., if answer to question 4 is ‘yes’ then go to question 5, otherwise go to question 10).

The present study is designed to test the feasibility and validity of using computerized ESM (ESMc) among individuals suffering from schizophrenia. It uses PDAs (“Personal Digital Assistant”; Palm handheld computers) to collect data on psychotic symptoms, mood, thoughts, and experiences. ESMc offers a number of potential advantages over ESMp (Bolger et al., 2003; Stone and Shiffman, 2002) – (1) it provides direct measure of compliance by recording precise time-stamps for each response, as well as the time it took for the subject to complete each question; (2) ESMc facilitates the use of branching; (3) ESMc minimizes the cost and labor associated with transcription of data; and perhaps most importantly, (4) because of the availability of precise time-stamps, ESMc makes it feasible to investigate interactions between psychological and neurobiological mechanisms by synchronizing the times of ESMc and ambulatory physiological indexes (heart rate, electrodermal response, respiration, posture, movement, etc.). Additionally, Stone et al. (2003b) reported significantly higher compliance rates among users of electronic diaries, compared to paper-based diaries.

ESMc with PDAs has been used previously to assess clinical populations including individuals with chronic pain (Roelofs et al., 2004; Stone et al., 2003a), cancer (Gaertner et al., 2004), Parkinson’s disease (Nyholm et al., 2004), and anhedonia (Kwapil et al., 2005). It also has been used to study anxiety, ADHD symptoms, and smoking and alcohol use among adolescents (Henker et al., 2002; Whalen et al., 2002). However, to our knowledge, this methodology has not been employed with individuals with severe psychiatric disorders, and in particular with individuals with schizophrenia. Thus, our goal is to study the feasibility and validity of using ESMc among individuals with schizophrenia.

2. Method

2.1. Participants

Twenty individuals participated in this preliminary study at the New York State Psychiatric Institute (NYSPI). The participants’ demographic information is presented in Table 1. Ten patients with schizophrenia were recruited from the Schizophrenia Research Unit (SRU) – a psychiatric inpatient research unit. A sample of convenience of ten healthy controls was recruited from the medical center community. The healthy controls did not undergo assessment with the DIGS and SAPS. All subjects were aged between 18 and 55 and fluent in English. The study was approved by the NYSPI Institutional Review Board and all subjects provided written informed consent. Data were collected between July 2004 and January 2005.

Table 1
Demographics and clinical information

2.2. Instruments

Diagnosis of schizophrenia, schizophreniform disorder, schizoaffective disorder

Diagnoses were made using the Diagnostic Interview for Genetic Studies (DIGS; Nurnberger et al., 1994), a structured diagnostic interview and medical records review that is used to gather diagnostic and course of illness information for the mood, psychotic, and substance use DSM-IV Axis I disorders. DIGS interviews were completed by a Master’s level clinician blind to the present study.

Interview-based assessment of symptoms

Positive symptoms were assessed at admission to the SRU using the Scale for Assessment of Positive Symptoms (SAPS;Andreasen and Olsen, 1982) as part of standard research assessments at the SRU. The assessments were completed by a Master’s level clinicians blind to the present study.

Self-reports of momentary psychotic symptoms, mood, thoughts, and environmental contexts

Samples of psychotic symptoms, mood, thoughts, and environmental contexts were collected using self-report questionnaires completed on PDAs. The questionnaire was derived from previous ESM studies with individuals with schizophrenia and psychosis using ESMp (Delespaul, 1995; Myin-Germeys et al., 2000, 2001, 2002; Tournier et al., 2003; Verdoux et al., 2003). Table 2 presents the descriptions of the symptom and mood items assessed.

Table 2
Description of the ESMc symptom and mood items

2.3. Procedure

Evaluations took place over a one-day period and were conducted during weekdays. On the morning of the study, participants completed a questionnaire about their knowledge, exposure, and previous experiences using PDAs. A brief introduction session followed in which subjects were introduced to basic operations of PDAs and completed two full practice sets of questions on the PDAs. The introduction sessions typically lasted about 15–20 min. Subjects were then provided with a Palm Tungsten T3 PDA (Palm OS ® version 5.2.1) to carry with them throughout the day. We used the iESP software (version 3.3; Intel Research Center, Seattle, WA) to present questions and collect responses on the PDAs. The PDA was pre-programmed to beep randomly 10 times between 10:00 am and 10:00 pm to elicit information about current psychotic symptoms, mood, thoughts, and social context. A stratified time sampling scheme was used to minimize the probability that activations will be concentrated over a short period of time. The software was set up to divide the assessment period (12 h) by the number of activations (10) to create 10 equal time-windows of 72 min each (12 h × 60 min/10 beeps). It then scheduled one beep randomly within each time-window. Thus, the potential period of time between activations ranged from 1 to 143 min. In accordance with previous ESM schizophrenia studies (Delespaul, 1995; Myin-Germeys et al., 2001), subjects had to complete at least 33% of the experience sampling activations in order for their data to be considered valid and included in the data analyses. Upon hearing the beep, subjects were instructed to respond to a questionnaire presented on the screen of the PDA (i.e., “I feel stressed”; “My thoughts are suspicious”). For each symptom and mood question, subjects were asked to indicate on the PDA’s screen the quality of their current experience on a graphical slider similar to a visual analog scale (from “not at all” to “very much”; See Fig. 1). Responses were represented in the output as a value between 1 (“not at all”; leftmost extreme) and 100 (“very much”; rightmost extreme). Additionally, subjects were asked about their current social context and activities (i.e., “I am with…?”; “what am I doing?”; see Fig. 2).

Fig. 1
Screen shot of a sample question.
Fig. 2
Screen shot of a sample question (inpatients only).

As the aim of ESMc is to assess momentary “live” experiences, subjects were given 180 s to respond to the activating beeps, as well as 180 s to complete each of the questions. If they did not complete their responses within this time period, the PDA was programmed to automatically turn off until the next activation. This limitation was put in place to minimize the possibility of memory bias influencing the experiences reported. To minimize the potential of the PDAs interfering with the subjects’ normal daily activities and possibly impacting their responses, the PDAs were preprogrammed to be locked in-between activations for all basic PDA functions (calendars, to-do list, games, etc.), as well as access to the questions and previous responses. This locking procedure also ensured the privacy and confidentiality of subjects’ responses during the day of data collection. For safety reasons, healthy controls were instructed not to respond to beeps while driving home from work. Following the completion of the 10 experience samples, subjects completed a questionnaire assessing their experiences using the PDA throughout the study day. The PDAs were collected from the subjects during the morning following the study day and the data was downloaded to a PC using standard PDA–PC data synchronization.

3. Statistical analyses

ESM data has a hierarchical structure in which repeated observations are nested within subjects and therefore are not independent. Since observations from the same subject are more similar than observations from different subjects, the residuals are not independent. Thus, to assess the relationships among symptoms and moods we analyzed the data using multilevel linear modeling. This method is more appropriate than conventional unilevel analyses for analyzing nested data (Schwartz and Stone, 1998). Multilevel modeling techniques are a variant of unilevel regression analyses (Hox, 2000), and they are standard for the analysis of ESM data (Gable et al., 2000). The multilevel data analyses were conducted using SAS software for Windows (version 9). The correlations among symptoms were estimated using the regression model for each pair of symptoms. Since the measures on each subject are repeated at different time points, and the data includes unequal cluster sizes due to the missing values for certain time points, we used Generalized Estimation Equation (GEE; Liang and Zeger, 1986) adjustment within the regression model to estimate the correlations. The variables were first standardized using mean and standard deviation. The correlation between two symptoms (or moods), ai and aj, is represented by the regression model,

equation M1

The GEE-adjusted β coefficient is taken from the model, with β1 being the estimate for the correlation between the two symptoms. An autoregressive correlation structure is imposed since it is time-based within each individual. For comparisons between the schizophrenia and healthy controls group, the measurements on each subject’s symptoms and moods are assessed using z-tests with p-values based on two-sided tests. Similar to the assessment of relationships, the GEE-adjustment with an autoregressive correlation structure is imposed in order to take into account the dependence of the repeated measures within subject. This method is more effective and powerful than the t-test on the subjects’ means across the time points because it allows use of the information from all the repeated measurements rather than an aggregate of them. The SD was calculated based on the mean value of the measurement for each subject. Comparisons of the groups’ demographics and pre-and post-study response characteristics were calculated using t-tests.

4. Results

Thirteen individuals with schizophrenia were approached to participate in the study of which 11 agreed. In accordance with standards used in other ESM schizophrenia studies (Delespaul, 1995; Myin-Germeys et al., 2001), one subject’s data was excluded from data analysis for completing fewer than 33% of the experience sampling activations, resulting in 10 valid protocols. Eleven healthy controls were approached to participate in the study of which 10 agreed.

Table 3 presents the pre- and post-study response characteristics for each group, as well as differences between the groups. Subjects from the schizophrenia and healthy controls groups provided on average 8.10 (SD = 1.79, range = 5–10) and 8.00 responses (SD = 2.05, range = 5–10), respectively. This difference was not significant (t(18) = 0.12, p = 0.91). The number of uncompleted responses due to the 180 s time restriction was minimal (schizophrenia = 4, healthy controls = 1). Data on five responses (schizophrenia = 2, healthy controls = 3) were lost due to an early software malfunction that was later corrected. Among the healthy controls only one beep was not responded due to driving. The healthy controls average questionnaire completion time (from initial activating beep to completion time) was significantly faster (t(18) = 2.46, p = 0.02).

Table 3
Means, SD, distribution and differences in experiences between healthy controls and individual with schizophrenia

Prior to the study, subjects from both groups reported having limited familiarity with operating PDAs (M = 2.30, SD = 1.33 and M = 2.00, SD = 1.63, respectively) and moderate comfort levels operating them (M = 3.30, SD = 0.95 and M = 3.20, SD = 0.92, respectively). There were no significant differences between the groups in their familiarity with (t(18) = 0.45, p = 0.66) or comfort level of operating PDAs (t(18) = 0.24, p = 0.81). Likewise, following the study there were no significant differences between the groups in their reports of their ability to understand the presented questions, difficulties typing responses or operating the PDAs, or their level of comfort carrying the PDAs. The healthy controls characterized their participation in the study as significantly less challenging (t(18) = 2.63, p = 0.02).

Table 4 presents the means, standard deviation, and differences in symptom and moods between the healthy controls and individual with schizophrenia. As would be expected, individuals with schizophrenia reported significantly higher ratings of auditory and visual hallucinations, suspicious thinking, sense of unreality, lack of thought control, fear of losing control, difficulty expressing thoughts, as well as depression/sadness, lack of cheerfulness, and loneliness. There was no significant difference between the groups on ratings of level of stress, degree of being relaxed, and presence of racing thoughts. Among the schizophrenia patients, both psychotic symptoms and mood fluctuated considerably throughout the course of the day. Fig. 3 presents the temporal pattern of mean auditory hallucinations and sadness/depression symptom ratings in schizophrenia patients and healthy controls across time of day. Interestingly, auditory hallucinations appeared to subside during 11 am–12 pm and 5 pm–6 pm periods, corresponding with lunch and dinner time at the unit (respectively). Similarly, among healthy controls, ratings of sadness/depression appear to subside between 12 pm and 1 pm and again between 4 pm and 6 pm, corresponding with lunch time and the end of the work day period (respectively).

Fig. 3
Temporal pattern of auditory hallucinations and sadness/depression in schizophrenia patients and healthy control across time of day.
Table 4
Means, SD, and differences in symptoms and mood between healthy controls and individual with schizophrenia

As would be expected, inverse relationships were found for ratings of feeling cheerful and ratings of being stressed (r = −10.38, p < 0.001), irritated (r = −0.42, p < 0.0001), and sad/depressed (r = −0.53, p < 0.0001) among both groups. Admission symptom ratings (SAPS) were available for nine of the 10 schizophrenia patients. The correlation between aggregated ESM rating of suspiciousness and ratings of suspiciousness on the SAPS (item #8) at admission approached significance (r = 0.61, p = 0.07, n = 9). Similarly, there was a trend for a relationship between degree of suspiciousness and levels of stress among the schizophrenia group (r = 0.30, p = 0.10, n = 9).

5. Discussion

Our results support the feasibility and validity of using ESMc for assessment of momentary psychotic symptoms, mood, and experiences among hospitalized individuals with schizophrenia. Our findings are particularly significant given the fact our subjects had no previous experience using PDAs. The ESMc methodology was equally acceptable to the patients with schizophrenia and healthy controls, with similar ratings of comfort carrying the PDAs, operating them, degree of interference with daily activities, as well as overall levels of stress and difficulties participating in the study. Equally important, both groups had similar response rates to the beeps.

The subjective nature of ESM data (e.g., thoughts, moods, mental states) makes the use of customary reliability and validity techniques problematic. The assessment of reliability is complicated by the fact that experiences assessed by ESMc, such as “I feel lonely”, may not necessarily have behavioral expressions, thus making them difficult to verify. As a result, the ascertainment of reliability can be obtained only by evaluation of the validity (Delespaul, 1995). Once validity is demonstrated, reliability can be assumed. Delespaul (1995) suggested a number of methods to validate raw ESM data including the use of face validity, comparison of aggregated data between distinct groups, correlations between similar and dissimilar items, as well as determining associations with available behavioral/external referents.

In accord with these recommendations, we found ESMc to be valid. The items used in our assessment were designed to maximize face validity by using everyday vocabulary. A comparison of ratings of symptoms and mood between the subjects in the schizophrenia and healthy control groups indicate differences in the expected direction. Individuals with schizophrenia reported significantly higher ratings of auditory and visual hallucinations, suspicious thinking, sense of unreality, lack of thought control, fear of losing control, difficulty expressing thoughts, as well as depression/sadness, lack of cheerfulness, and loneliness. Similarly, the negative correlations between dissimilar items, such as feeling cheerful and feeling stressed, irritated, and sad/depressed support the discriminant validity of ESMc. The trend in the relationship between suspiciousness ratings on SAPS and ESMc give additional support for the validity of ESMc ratings among hospitalized patients with schizophrenia. However, the number of participants is small and should be considered preliminary.

The finding that there were no significant differences in ratings of stress between the schizophrenia and healthy control groups is noteworthy. Stress is common among individuals with schizophrenia (Kimhy et al., 2004; Steer et al., 2003) and has long been assumed to be associated with psychosis. Our findings suggest that schizophrenia patients are not necessarily experiencing more stress, but rather their sensitivity to stress leads to exacerbation of psychotic symptoms.

Overall, we believe ESMc offers a number of advantages in research methodology. The most significant contribution of ESMc to research is in providing precise time-stamps for each response. By synchronizing the times of PDAs and ambulatory physiological measures, this feature makes it feasible to investigate interactions between psychological and neurobiological mechanisms. Thus, it offers exciting new ways to understand human behavior and functioning. In particular, this methodology has the potential to offer new insights into the mechanisms involved in psychopathology by studying the dynamic processes that govern the interactions between subjective experiences, neurobiology, and the environment. ESMc also limits the cost and labor associated with transcription and cleaning of data – (Nyholm et al., 2004) reported on a study of 20 subjects in which the total time required for data entry from paper-diaries was 96 person–hours compared to less than 4 person–hours for PDA diaries. However, the authors did not report the amount of time spent programming the PDAs. It is also important to note that data entered on a PDA by a subject may not be necessarily less susceptible to data-entry errors compared to data entered by a trained data entry person. Furthermore, there is no way for researchers to verify the correctness of data entered by subjects using ESMc after the data was collected. Finally, ESMc offers advantages with regard to study design and data management. In particular, the ability to limit the response time available to participants as well as branching are important features.

Potential limitations of this study should be acknowledged. The sample size in the present study was small and the assessment period was shorter than what is typical in ESM schizophrenia studies. Thus, the results should be interpreted with caution. A number of factors may have contributed to the higher response rate with ESMc compared to ESMp. Studies of individuals with schizophrenia using ESMp typically assess subjects 10 times per day from 7:30 am to 10:30 pm over 6 days (60 beeps). In contrast, the assessment period in the present study lasted only one day (10 beeps) from 10:00 am to 10:00 pm. Delespaul (personal communication) found that the average response rate among 50 individuals with schizophrenia studied over six days using ESMp was 72% during the first day of assessment, with the response rate gradually declining to 61% at day four and stabilizing afterwards. However, the majority of the missed beeps occurred in the early morning and late night periods that were not sampled in the present study. Thus, the relatively higher response rate using ESMc may reflect the initial higher response rate associated with the novelty of participating in an ESM research protocol and/or the designation of an assessment period that is characterized by a higher response rate. Similarly, participants in our study were presented with 27–32 ESM questions per beep (depending on branching) compared to approximately 50 in typical ESMp studies with individuals with schizophrenia (Delespaul, 1995; Myin-Germeys et al., 2000, 2001, 2002), which may have influence the response rate. Additionally, the current study was carried out in an inpatient unit, whereas most ESMp studies were carried out among outpatient individuals with schizophrenia. Such environment may potentially increase the risk of PDAs having malfunctions or being damaged, lost, or stolen. However, Kwapil (personal communication) reported only two PDAs being broken, none lost/stolen, and limited technical malfunctions in an ESMc study among approximately 400 college students.

An important question is the feasibility to using ESMc with patients with severe psychotic symptoms. While this methodology will clearly not be suitable for sampling with some schizophrenia patients, the hospitalized schizophrenia patients who participated in this pilot study were quite symptomatic, with mean SAPS ratings of psychotic symptoms ranging from mild to moderate (see Table 1), with many displaying marked and severe psychotic symptoms. Consistent with this data, Delespaul (1995) and Myin-Germeys et al. (2001) has demonstrated extensively that ESM assessments are not restricted to small sub-samples of relatively asymptomatic schizophrenia patients.

In summary, assessment of momentary psychotic symptoms, mood, and experiences using ESMc is feasible among hospitalized schizophrenia patients. The methodology was acceptable to patients and caused minimal disruption in their daily activities and routines. The data also provide preliminary support for the validity of using ESMc in this population. Future studies should aim to replicate our findings with larger samples. Similarly, the feasibility of sampling experiences of over longer periods of time should be firmly established. Finally, future studies should also establish the feasibility of using ESMc with schizophrenia patients with more severe symptoms. Overall, ESMc is novel and exciting methodology that allows studying the dynamic processes that govern the interactions between subjective experiences, neurobiology, and the environment. Such knowledge may improve our understanding of mechanisms of development and resolution of psychopathology.

Acknowledgement

This work was supported by National Institute of Health Grant No. 1K24MH01699 (Dr. Dolores Malaspina).

References

  • Andreasen NC, Olsen S. Negative v positive schizophrenia. Definition and validation. Archives of General Psychiatry. 1982;39:789–94. [PubMed]
  • Barge-Schaapveld DQ, Nicolson NA. Effects of antidepressant treatment on the quality of daily life: an experience sampling study. Journal of Clinical Psychiatry. 2002;63:477–85. [PubMed]
  • Barge-Schaapveld DQ, Nicolson NA, van der Hoop RG, De Vries MW. Changes in daily life experience associated with clinical improvement in depression. Journal of Affective Disorders. 1995;34:139–54. [PubMed]
  • Barge-Schaapveld DQ, Nicolson NA, Berkhof J, DeVries MW. Quality of life in depression: daily life determinants and variability. Psychiatry Research. 1999;88:173–89. [PubMed]
  • Bolger N, Davis A, Rafaeli E. Diary methods: capturing life as it is lived. Annual Review of Psychology. 2003;54:579–616. [PubMed]
  • Delespaul P, deVries M, Van OJ. Determinants of occurrence and recovery from hallucinations in daily life. Social Psychiatry and Psychiatric Epidemiology. 2002;37:97–104. [PubMed]
  • Delespaul P. Assessing schizophrenia in daily life. ISPER; Maastricht (The Netherlands): 1995.
  • Gable SL, Reis HT, Elliot AJ. Behavioral activation and inhibition in everyday life. Journal of Personality and Social Psychology. 2000;78:1135–49. [PubMed]
  • Gaertner J, Eisner F, Pollmann-Dahmen K, Radbruch L, Sabatowski R. Electronic pain diary: a randomized crossover study. Journal of Pain and Symptom Management. 2004;28:259–67. [PubMed]
  • Henker B, Whalen CK, Jamner LD, Delfino RJ. Anxiety, affect, and activity in teenagers: monitoring daily life with electronic diaries. Journal of the American Academy of Child and Adolescent Psychiatry. 2002;41:660–70. [PubMed]
  • Hillbrand M, Waite BM. The everyday experience of an institutionalized sex offender: an idiographic application of the experience sampling method. Archives of Sex Behavior. 1994;23:453–63. [PubMed]
  • Hillbrand M, Waite BM, Miller DS, Spitz RT, Lingswiler VM. Serum cholesterol concentrations and mood states in violent psychiatric patients: an experience sampling study. Journal of Behavioral Medicine. 2000;23:519–29. [PubMed]
  • Hox JJ. Multilevel analysis: techniques and applications. Lawrence Erlbaum Associates; Mahwah (NJ): 2000.
  • Husky MM, Grondin OS, Swendsen JD. The relation between social behavior and negative affect in psychosis-prone individuals: an experience sampling investigation. European Psychiatry. 2004;19:1–7. [PubMed]
  • Kimhy D, Harkavy-Friedman JM, Nelson EA. Identifying life stressors of patients with schizophrenia at hospital discharge. Psychiatric Services. 2004;55:1444–5. [PubMed]
  • Kwapil TR, Silvia PJ, Myin-Germeys I, Brown LH, Anderson AJ, Coates SA. An experience sampling study of the relationship of social anhedonia with social contact, stress, and emotion. Schizophrenia Bulletin. 2005;31:203.
  • Liang KL, Zeger SL. Longitudinal data analysis using generalized linear models. 1986. pp. 13–22.
  • Myin-Germeys I, Delespaul PA, DeVries MW. Schizophrenia patients are more emotionally active than is assumed based on their behavior. Schizophrenia Bulletin. 2000;26:847–54. [PubMed]
  • Myin-Germeys I, Nicolson NA, Delespaul PA. The context of delusional experiences in the daily life of patients with schizophrenia. Psychological Medicine. 2001;31:489–98. [PubMed]
  • Myin-Germeys I, Krabbendam L, Jolles J, Delespaul PA, Van OJ. Are cognitive impairments associated with sensitivity to stress in schizophrenia? An experience sampling study. American Journal of Psychiatry. 2002;159:443–9. [PubMed]
  • Myin-Germeys I, Krabbendam L, Delespaul PA, Van OJ. Do life events have their effect on psychosis by influencing the emotional reactivity to daily life stress? Psychological Medicine. 2003a;33:327–33. [PubMed]
  • Myin-Germeys I, Peeters F, Havermans R, Nicolson NA, De Vries MW, Delespaul P, Van OJ. Emotional reactivity to daily life stress in psychosis and affective disorder: an experience sampling study. Acta Psychiatrica Scandinavica. 2003b;107:124–31. [PubMed]
  • Myin-Germeys I, Krabbendam L, Delespaul PA, Van OJ. Sex differences in emotional reactivity to daily life stress in psychosis. Journal of Clinical Psychiatry. 2004;65:805–9. [PubMed]
  • Nurnberger JI, Jr, Blehar MC, Kaufmann CA, York-Cooler C, Simpson SG, Harkavy-Friedman J, Severe JB, Malaspina D, Reich T. Diagnostic interview for genetic studies. Rationale, unique features, and training. NIMH Genetics Initiative. Archives of General Psychiatry. 1994;51:849–59. [PubMed]
  • Nyholm D, Kowalski J, Aquilonius SM. Wireless real-time electronic data capture for self-assessment of motor function and quality of life in Parkinson’s disease. Movement Disorders. 2004;19:446–51. [PubMed]
  • Peeters F, Nicolson NA, Berkhof J, Delespaul P, deVries M. Effects of daily events on mood states in major depressive disorder. Journal of Abnormal Psychology. 2003;112:203–11. [PubMed]
  • Roelofs J, Peters ML, Patijn J, Schouten EG, Vlaeyen JW. Electronic diary assessment of pain-related fear, attention to pain, and pain intensity in chronic low back pain patients. Pain. 2004;112:335–42. [PubMed]
  • Schwartz JE, Stone AA. Strategies for analyzing ecological momentary assessment data. Health Psychology. 1998;17:6–16. [PubMed]
  • Steer RA, Kumar G, Pinninti NR, Beck AT. Severity and internal consistency of self-reported anxiety in psychotic outpatients. Psychological Report. 2003;93:1233–8. [PubMed]
  • Stone AA, Shiffman S. Capturing momentary, self-report data: a proposal for reporting guidelines. Annals of Behavioral Medicine. 2002;24:236–43. [PubMed]
  • Stone AA, Broderick JE, Schwartz JE, Shiffman S, Litcher-Kelly L, Calvanese P. Intensive momentary reporting of pain with an electronic diary: reactivity, compliance, and patient satisfaction. Pain. 2003a;104:343–51. [PubMed]
  • Stone AA, Shiffman S, Schwartz JE, Broderick JE, Hufford MR. Patient compliance with paper and electronic diaries. Controlled Clinical Trials. 2003b;24:182–99. [PubMed]
  • Swendsen JD, Compagnone P. The expression of cognitive vulnerabilities for depression in daily life: a French-American study. European Psychiatry. 2000;15(Suppl. 1):22–8. [PubMed]
  • Tournier M, Sorbara F, Gindre C, Swendsen JD, Verdoux H. Cannabis use and anxiety in daily life: a naturalistic investigation in a non-clinical population. Psychiatry Research. 2003;118:1–8. [PubMed]
  • Verdoux H, Gindre C, Sorbara F, Tournier M, Swendsen JD. Effects of cannabis and psychosis vulnerability in daily life: an experience sampling test study. Psychological Medicine. 2003;33:23–32. [PubMed]
  • Wang PS, Beck AL, Berglund P, McKenas DK, Pronk NP, Simon GE, Kessler RC. Effects of major depression on moment-in-time work performance. American Journal of Psychiatry. 2004;161:1885–91. [PubMed]
  • Whalen CK, Jamner LD, Henker B, Delfino RJ, Lozano JM. The ADHD spectrum and everyday life: experience sampling of adolescent moods, activities, smoking, and drinking. Child Development. 2002;73:209–27. [PubMed]