|Home | About | Journals | Submit | Contact Us | Français|
Performance on a novel, virtual reality (VR) assessment of medication management skills, the Virtual Reality Apartment Medication Management Assessment (VRAMMA), was investigated in 25 patients with schizophrenia and 18 matched healthy controls. The VRAMMA is a virtual 4-room apartment consisting of a living room with an interactive clock and TV, a bedroom, a kitchen, and a bathroom with an interactive medicine cabinet. After an exploratory phase, participants were given a mock prescription regimen to be taken 15 minutes later from pill bottles located in the medicine cabinet in the bathroom of the virtual environment. The VRAMMA was administered with a validated measure of medication management skills, several neurocognitive tests, and a symptom scale. Results revealed that (1) schizophrenic patients made significantly more quantitative errors in the number of pills taken, were less accurate at taking the prescribed medications at the designated time, and checked the interactive clock less frequently than healthy controls; (2) in patients with schizophrenia, there was significant agreement in classification of adherence vs nonadherence between a validated measure of medication management skills and the VRAMMA; and (3) in patients with schizophrenia, years of education and a measure of verbal learning and memory were linked to quantitative errors on the VRAMMA, while positive symptoms, specifically delusional symptoms, were inversely linked to distance traveled within the VRAMMA. This is the first study, to our knowledge, to provide evidence for the utility of VR technology in the assessment of instrumental role functioning in patients with schizophrenia.
Schizophrenia is a chronic and profoundly disabling psychiatric disorder. Current estimates suggest that 70%–80% of patients with schizophrenia are unemployed at any one time, and only half of 1% patients with schizophrenia who receive Social Security Insurance (SSI/SSDI) ever remove themselves from entitlements.1,2 As many as 70% of patients have neurocognitive deficits,3 and some recent estimates suggest that the majority of patients with schizophrenia are noncompliant with their antipsychotic medication.4 Both of these factors have been linked to the poor functional outcome evident in these patients5,6 and thus have become important targets for intervention.
Until recently, most measures of functional or psychosocial status in studies of schizophrenia have relied on self-report7,8 with a few exceptions.9 The validity of these self-report measures, however, is dependent on intact emotional and intellectual function, as well as judgment skills that may be disrupted in schizophrenia. Schizophrenic patients often lack insight into the impact of their illness on role functioning,10 they frequently have thought disorder that disrupts coherent verbal expression, they may experience depressed or anxious moods that influence judgments of current activities and social relations,11 and their deficits in information-processing skills may influence their ability to accurately assess their functioning.
In response to these limitations, interest in the development of new performance-based scales of social and instrumental role functions for a more accurate assessment of everyday life-skills for patients with schizophrenia has intensified. Several paper-and-pencil, performance-based measures of psychosocial status have been validated in patients with severe mental illness, including the University of California at San Diego (UCSD) Performance-Based Skills Assessment12 and the Social Skills Performance Assessment.13 None of these performance-based scales have utilized virtual reality (VR) technology to produce multimodal simulated environments that afford evaluation of behavior in a more ecologically valid setting.
One area of instrumental role function in patients with schizophrenia that has received increased attention in recent years is medication management abilities, in light of both the importance of optimal pharmacotherapy for management of symptoms and the frequency with which patients do not adhere to their prescribed medication regimen. Estimates from the extant literature suggest that oral antipsychotic medication adherence rates in patients with psychotic disorders range widely, with divergent findings related to the method for assessing adherence and sample characteristics.14 Two recent studies indicate that adherence for patients treated with atypical antipsychotics is 40% or less, despite their less-burdensome side effect profile, relative to conventional antipsychotic medication.4,15 Poor medication adherence leads to increased frequency of relapse, emergency room visits, and rehospitalization.6,16
To our knowledge, only one study has validated a performance-based tool for medication management for patients with schizophrenia.17 In that study, 104 older patients with schizophrenia and 33 healthy controls completed a performance-based measure of medication management, the Medication Management Ability Assessment (MMAA), as well as a variety of clinical scales. Self-report and prescription refill records were evaluated to assess the relationship of the MMAA to other putative measures of adherence. Results revealed that the MMAA showed excellent test-retest stability, classified patients as significantly more nonadherent than healthy controls across a variety of definitions of adherence, was linked to cognitive status and other performance-based measures of everyday function, and was related to an external measure of adherence, prescription refill records, in a subsample of patients. While the MMAA includes a variety of elements crucial to more ecologically valid assessment of medication management skills in patients with schizophrenia, including a delay between the presentation of the medication regimen and time to take medications, as well as the availability of medication labels to help patients sequence their medication regimen correctly, the MMAA does not simulate the variety of environmental distractions or environmental prosthetics (written notes, use of a clock) that may be evident in a patient's natural environment that may influence adherence.
We have recently developed a new VR, performance-based instrument for the assessment of medication management skills in patients with schizophrenia, called the Virtual Reality Apartment Medication Management Assessment (VRAMMA). This instrument consists of a mock medication regimen and an apartment that the participant must navigate to take the appropriate type and dosage of medication at the appropriate time. The advantages of a VR instrument relative to interview-based and paper-and-pencil assessment of medication management skills is that (1) it dynamically engages a broad range of neurocognitive functions through the use of a complex multimodal environment that more closely approximates how neurocognitive skills are utilized in a patient's natural environment and that includes common environmental distractions, (2) in so doing it may provide a closer estimate of the actual community performance of medication management skills than that provided by clinic-administered capacity measures of medication management skills, and (3) it allows for careful measurement of how performance on this medication management task may break down by quantifying patterns of behavior within the virtual apartment.
Clearly, “real-world” medication adherence in patients with schizophrenia is influenced by a plethora of factors that have been traditionally grouped into environmental (eg, financial resources), medication (eg, side effect profile), and patient (eg, illness insight) factors.15 One patient factor, however, that has garnered increased attention in recent years is neurocognitive impairment, in light of increased recognition of its frequency in patients with schizophrenia.3 While several initial studies failed to show a relationship between neurocognitive function and self-reported medication adherence,18–20 one recent study suggested a close relationship between neurocognitive skills and a performance-based index of medication management skills. In that study, neurocognitive skills, especially conceptualization and memory functions, were more closely related to medication management skills in a sample of 110 middle-aged and older patients with schizophrenia than other patient factors such as attitudes toward medication, symptoms, or demographic factors.21
The goal of the current study was to provide preliminary validation of the VRAMMA as a tool for measuring medication management skill in patients with schizophrenia and to relate performance in the VRAMMA to neurocognitive skills and symptoms. Our specific hypotheses were as follows: (1) patients with schizophrenia would be more impaired in medication management skills than matched healthy controls as measured by overall accuracy in taking the correct medications, in the correct dose, at the correct time; (2) performance on the VR measure of medication management skills would relate to a standardized, validated paper-and-pencil measure of medication management, the MMAA,17 as well as neurocognitive measures of sustained visual vigilance, verbal memory and problem solving; and (3) negative symptoms would be linked to more sedentary behavior in the apartment, as measured by total distance traveled. A secondary hypothesis was that healthy controls would be more likely to use environmental prosthetics, specifically the interactive clock and a reminder note on the refrigerator in the kitchen documenting both the type and dose of pills to be taken, than patients with schizophrenia.
Twenty-five patients meeting Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV)22 criteria for schizophrenia or schizoaffective disorder as determined by the Structured Clinical Interview for DSM-IV23 and 18 community-dwelling healthy controls participated. Exclusion criteria for patients and matched controls included auditory or visual impairment, mental retardation (IQ < 70), traumatic brain injury, presence or history of any neurologic illness, lack of proficiency in English, and/or criteria met for concurrent substance abuse or dependence. For healthy controls, exclusion criteria also included first-degree relatives with significant history of schizophrenia or schizoaffective disorder. Healthy controls were recruited from the greater Hartford region via local advertisements and were screened with a medical health questionnaire. See table 1 for a summary of demographic characteristics of the patient and healthy samples and clinical characteristics of the patient sample. Patients had significantly less education than healthy controls (mean education was 12.7 vs 14.8 years) but were similar in other respects. All participants provided written informed consent, all procedures met institutional ethical approval, and all participants were paid $20 per hour for participation. All participants were able to complete the assessment.
The virtual apartment was programmed by the senior author and consisted of 4 rooms: a living room, bedroom, bathroom, and kitchen. The medicine cabinet where the target medications were stored was in the bathroom, a clock to check the time was located above the TV in the living room, and the reminder note, an interactive “post-it” note, was located on the kitchen refrigerator. The apartment consisted of high-resolution graphics, dynamic lighting, and three-dimensional stereo sound (see figure 1 for several screenshots from the virtual apartment).
An IBM-compatible computer with an SVGA color monitor was used for testing. Participants navigated through the apartment by manipulating a joystick. A speaker connected to the computer provided auditory feedback to the participants. After obtaining written informed consent, participants went through 2 phases of the experiment: (1) an exploration phase and (2) a testing phase.
The VRAMMA was administered in a standardized way as follows: In the exploration phase, the task was introduced by indicating that before the “test phase” of the task begins, the participant would be asked to do several activities in the apartment, so that they can get an understanding of their surroundings, as well as to practice moving with the joystick. Participants were also told that a prescription telling them to take certain pills would appear within a few seconds of beginning the practice session but that they should not try to memorize that regimen because different pills would be prescribed during the actual test phase of the task. Participants were told that this was the same form in which the prescription would be introduced in the actual task. At this juncture, participants were asked to walk to the TV and turn it on, to check the time by clicking the black box above the TV, to turn into the bedroom, and to turn on the light after entering it. Participants were also told to look around and see their bedroom. After participants explored the bedroom, they were asked to leave that room and go to the kitchen. Participants were told that the kitchen was the last door on the left and that they would be passing the bathroom on their way (to ensure all participants were familiar with the entire layout of the apartment). In the kitchen, participants told to turn on the light, turn off the stove, and check the reminder on the refrigerator. Prior to clicking on the reminder note, participants were again reminded not to memorize those medications. Participants were then asked to walk back into the bathroom, open the medicine cabinet, take out 4 pills, and then put 2 back in order to demonstrate what to do if too many pills were removed. Participants were told that in the actual task, 3 types of pills would be prescribed and that all 3 types must be taken out before hitting the “Take Pills” button because this would end the task.
In the “testing” phase, the task was introduced by telling the participant that several seconds into the task a post-it note would appear telling them which medications to take and when to take them and that they should use all the items in the apartment that were explored during the practice phase in order to take the right medications at the right time. The participant started out the test phase in the living room of the virtual environment. Soon after, a note appeared on the computer screen for 10 seconds describing a simulated medication regimen and when the prescribed pills were to be taken. For example, for some participants, it read, “Remember to take: 2 yellow, 4 red, and 1 blue pill in 15 minutes” (see figure 1). A 15-minute delay was selected based on pilot data indicating that healthy controls were able to complete the task successfully at this interval and to avoid an overly long assessment protocol for possible use of the VRAMMA as a clinical instrument. The virtual clock on top of the TV could be clicked to check the time; the time was displayed for 5 seconds after it was selected. There also was a list of the medications and correct dosages located on the refrigerator to use as a reminder if the participant forgot which medications to take; each time they checked the note, it displayed the medication reminder for 10 seconds.
The participant used the joystick to move through the virtual environment. The view on the screen was a first-person perspective so that if he/she pushed the joystick to the right, the view on the screen would pan to the right. If he/she pushed the joystick forward, the view gradually got larger and so on with other joystick movements. Additionally, at 4 junctures during the test session, an auditory distraction occurred. These were a phone ringing, a doorbell, a dog barking, and a police siren, always in the same order at 3-minute intervals. Each distracter lasted approximately 10 seconds.
If the participant attempted to take the medications prior to 12 minutes of elapsed time, a notice appeared indicating that their attempt was premature. This feature was included based on pilot work that indicated a small minority of patients would attempt to take their medications immediately. If an attempt to take medications was not initiated by 18 minutes, a notice appeared saying “time has expired, please go take your medication now.” Eighteen minutes was selected based on pilot data indicating that patients who failed to take their medications at 18 minutes after exposure to the written regimen failed to take their medications at all. At this point, the reminder notice on the refrigerator was deactivated. If 23 minutes elapsed and the participant still had not taken their medication, the program terminated in order to minimize the possibility of overly lengthy assessment periods.
Participant's location within the apartment was written to a data file with a time stamp at a sampling rate of 10 Hz. Whenever a significant event occurred (eg, participant turned on the TV, turned the light off, checked the clock, checked the reminder notice, took a pill), the action was written to the data file with a time stamp. Dependent measures included the following. (1) Quantitative errors: These were defined as any difference, higher or lower, between the total number of each specific pill prescribed and the total number selected by the participant and the total number of any pills that were not prescribed. For example, if the participant was prescribed 2 yellow pills but instead consumed 2 green pills, this would result in a quantitative error score of 4. (2) Qualitative errors: These were defined as errors in the type of pill selected by participants. For example, the same error described above would result in a qualitative error of 1. (3) Time discrepancy: the duration of absolute time, in minutes, that the participant was inaccurate from the target time of taking medication 15 minutes after presentation of the prescription regimen in the virtual apartment. (4) Total distance traveled: the total area traveled throughout the 4 rooms of the virtual apartment. (5) Clock checks: the number of times the interactive clock was checked in the living room. (6) Reminder note checks: the number of times the reminder note on the refrigerator in the kitchen was checked.
The MMAA17 is a role-play task that simulates a medication regimen. The experimenter provides the participant with 4 mock prescribed bottles of medications with simulated pills and instructions for dosage of each and number of times to be taken during a day. After a 60-minute delay, the participants are given the bottles back and instructed to give the experimenter the pills from the bottles as they would take them throughout the day in relationship to their daily meals. Participants are able to check the pill bottle labels as they wish, and no prompting by the experimenter occurs. Construct validity for this task has been supported by a link between task performance and pharmacy records for prescription refills.
The Positive and Negative Syndrome Scale (PANSS)24 was used to evaluate the severity of positive, negative, and general psychiatric symptoms in patients with schizophrenia. All PANSS ratings were conducted by a clinical research associate who had established a high degree of interrater reliability with senior clinicians at the Olin Neuropsychiatry Research Center.
The Continuous Performance Test—Identical Pairs25 version was used as a measure of sustained visual vigilance, the Short Category Test26 was administered to assess problem-solving skills, and the Hopkins Verbal Learning Test (HVLT)27 was selected as a measure of verbal learning and memory. We assessed these neurocognitive skills because they have been linked to a variety of measures of community function, social problem solving, and progress in psychosocial rehabilitation programs in patients with schizophrenia.5
Data were evaluated for normality and homogeneity of variance. We used Levene test for equality of variance. When violations of normality or homogeneity occurred, we substituted nonparametric tests (ie, Mann-Whitney U test for independent-samples t tests, Spearman rank-order correlation for Pearson product moment correlations). We used 2-group t tests to determine quantitative differences in performance in the virtual apartment between patients with schizophrenia and healthy controls. Construct validity was assessed by comparing the percentage of agreement for classifying patients as compliant vs noncompliant via the virtual apartment and the MMAA. Associations between performance in the virtual apartment and the MMAA to demographic, neurocognitive, and symptom measures in the patient sample were assessed with Pearson correlation coefficients. All statistical tests were 2 tailed, and alpha was set at .05.
As can be seen in table 2, patients with schizophrenia, compared with age- and sex-matched controls, made more quantitative errors (taking greater or lesser number of pills than prescribed), taking the pills at a time more discrepant from the expected time, and checking the clock less often in the virtual apartment. Patients were not significantly different from healthy controls on measures of qualitative errors (taking the wrong medication in the bathroom cabinet), number of checks of the reminder post-it note on the virtual refrigerator, or total distance traveled in the apartment.
Defining adherence as not being greater than 20% over or under the number of pills prescribed for the hypothetical medication regimen, 36% of our patient sample, as compared with 83% of healthy controls, were labeled compliant. There was a 72% agreement (χ2 = 4.8, df = 1; P < .05) in classification of patients into adherent vs nonadherent groups via the virtual apartment and the MMAA. Similar findings were evident when a more conservative adherence threshold of 10% over or under the prescribed number of pills was used.
As can be seen in table 3, years of education and verbal learning and memory, as measured by total scores from the HVLT, were related to the number of quantitative errors in medication selection in the virtual apartment in patients. Verbal learning and memory as measured by the HVLT was also linked to number correct on the MMAA. No other demographic or neurocognitive measures were related to performance in the virtual apartment or on the MMAA. Ratings of positive symptoms in patients were related to total distance traveled in the virtual apartment. There was no relationship between negative symptoms and any of the measures from the virtual apartment.
To further evaluate the relationship of positive symptoms with distance traveled in the VRAMMA performance, we evaluated the correlation of each of the 7 items from the positive symptom scale from the PANSS with distance traveled from the task. Only item 1, delusions, showed a relationship with distance traveled on the VRAMMA (Spearman rho = −.78, P < .001), ie, patients with higher delusion scores traveled lesser distances.
To our knowledge, this is the first study to utilize the technological innovation afforded by VR to provide a more naturalistic assessment of instrumental role function in patients with schizophrenia. More specifically, we used VR technology to assess the ability of patients with schizophrenia to manage a simulated medication regimen in a multiroom apartment that included a variety of environmental distractions, as well as several environmental prosthetics for helping to remember which medications to take at what time. It was hypothesized that the simulated, multimodal VR environment would engage a range of neurocognitive and sensorimotor skills that more closely approximated the manner in which these skills are utilized in a participant's natural environment. Results from this study suggest that this technology is well tolerated by patients and is sensitive to deficits in medication management skills previously documented in this population.17
With respect to specific hypotheses, patients with schizophrenia were less able to successfully complete the simulated medication regimen presented in the VRAMMA as measured by making a significantly higher number of errors in number of pills taken and showing a larger discrepancy from the appropriate time to take the assigned medication, as compared with matched healthy control performance. Preliminary construct validity was supported by significant agreement between classification of adherence vs nonadherence in the patient group by the VRAMMA and the MMAA, a validated measure of medication management skills in older patients with schizophrenia, when adherence was defined as not erring in more than 20% of the total number of pills assigned in the mock medication regimen. Preliminary construct validity was also supported by a significant relationship between verbal learning and memory, as measured by the HVLT, and the number of quantitative errors on the VRAMMA. A similar relationship was noted between the MMAA and verbal learning and memory. The hypothesis that increased negative symptoms would be associated with more stationary behavior in the apartment was not supported. There was evidence that healthy controls utilized environmental memory prosthetics in the virtual apartment more effectively than patients with schizophrenia; healthy controls checked the clock in the virtual living room more often than patients with schizophrenia, presumably to help guide them in taking the medication at the correct time. Two unexpected findings were that level of education and positive symptom ratings (delusions) were significantly related to quantitative errors and distance traveled in the VRAMMA, respectively, in patients with schizophrenia. These findings suggest that those patients with delusions were less likely to explore the virtual apartment, particularly as the delusions became greater in number and/or more crystallized and systematic. This finding provides behavioral evidence of decreased exploratory behavior in patients with delusional symptoms and demonstrates the way in which phenomenology in schizophrenia may be linked to behavior in virtual environments. Future studies should be conducted to evaluate whether this relationship extends to community behavior.
The link between neurocognition, specifically verbal learning and memory, and medication management skills on the VRAMMA is consistent with several studies that have investigated the relationship between neurocognition and either simulated, performance-based medication management skills or acquisition of medication management skills in psychosocial rehabilitation programs. For example, in a sample of 110 older outpatients with schizophrenia, Jeste et al.21 found a relationship between performance on the Mattis Dementia Rating Scale (MDRS) and performance on a validated measure of medication management skills (MMAA), with strongest relationships between memory and conceptualization subtests from the MDRS. Other studies have linked neurocognitive skills, but not symptoms, to progress in psychosocial skill training programs that include lessons in medication management skills. For example, Smith et al28 noted that hospitalized patients with a psychotic disorder with stronger verbal learning and memory performance showed greater improvement in community reintegration skills after 16 sessions of skills training that included training in understanding medication effects. Similarly, Kern et al,29 in a sample of 19 psychotic inpatients, noted that verbal learning and memory, along with sustained vigilance and susceptibility to distraction, was linked to acquisition of medication management and symptom identification skills after training in these skills, as well as on-task behavior during training groups. Taken together, these findings suggest that verbal learning and memory, among myriad neurocognitive deficits in patients with schizophrenia, may be rate limiting with respect to both performance and acquisition of medication management skills. Given the importance of medication management skills in disease management, verbal learning and memory may be a crucial target for the development of novel behavioral and pharmacologic neurocognitive treatment strategies for enhancing patient outcome.
Several caveats to the current study should be mentioned. First, as noted in the “Introduction” section, adherence is related to a variety of factors including attitudes toward medication, insight into illness, and medication side effect profiles. The VRAMMA is limited to identifying poor adherence related to the neurocognitive challenge inherent in remembering to take several medications at a specific point in time in a simulated natural environment. Patients may show intact performance on the VRAMMA but poor real-world compliance because of these additional factors. Second, the study would have been strengthened by the use of an objective measure of real-world adherence such as pharmacy records of prescription refills, pill counts, records of number of bottle cap openings, and/or medication blood levels. In the absence of such measures, it remains unclear (1) how effectively the VRAMMA predicts real-world medication compliance and (2) whether the VRAMMA has incremental validity for predicting real-world compliance beyond briefer and less technologically demanding paper-and-pencil measures of neurocognitive function. These issues are being explored in a larger scale follow-up study that includes measures of real-world compliance. Third, sample size was small, and meaningful relationships between demographics, symptoms, and neurocognitive skills and VRAMMA performance may have been overlooked secondary to limited power. Fourth, the schizophrenia sample consisted of highly stable outpatients with minimal symptom measurements (mean PANSS positive symptom scale = 14.7, mean PANSS negative symptom scale = 13.8). Thus, the sample may have been biased toward patients who were compliant with their medications, and potential relationships between symptom ratings and the VRAMMA in this patient population may have been overlooked given the restricted range of symptom rating scores. Fifth, the use of pills labeled by color and administered all at one time in the VRAMMA, without reference to meals, does not approximate the technical names and complex sequencing of medications in relation to a patient's daily meals that are characteristic of typical regimens.
It is important to note that while an assumption of the study was that prospective memory errors over a 15-minute delay in the apartment would mirror the likelihood of errors of greater duration over much longer delays, we acknowledge that taking medication immediately after exposure to the medication regimen is an effective strategy for remembering appropriate type and amounts of medication in the context of the VRAMMA, which could effect the validity of the measure for predicting real-world compliance, where such a strategy is not feasible. We note, however, that no patient in this study took their medication earlier than 7 minutes into the task, and on average, patients took their medication 13.5 minutes after medication list exposure, suggesting that this strategy was not adopted by patients in our study.
The VRAMMA will need to undergo additional evaluation for potential use as a clinical assessment instrument. First, acceptable reliability of the various measures from the VRAMMA for healthy participants and patients with schizophrenia will need to be established through repeated testing. We are currently collecting data to address this issue. Second, as noted above, validity will need to be strengthened through observed relationships between performance on the VRAMMA and a variety of actual community medication management skills such as pharmacy prescription refills, pill counts, and blood levels. Third, additional normative data with equivalent numbers of males and females and a range of ages and education levels will be necessary to adequately assess impairment in patient populations on this instrument. Lastly, further research will be necessary to determine to what degree the VRAMMA will be sensitive to medication compliance skills in patients with other psychiatric or medical disorders (eg, bipolar illness) for whom compliance is a common difficulty.
Only one other study, to our knowledge, has used VR technology for assessment of patients with schizophrenia. Sorkin et al30 in a study of 39 patients with schizophrenia and 21 healthy controls investigated the relationship of skill in navigating a virtual maze that required hypothesis generation and application, as well as rule shifting, to diagnosis and symptom ratings. Results of this study revealed that performance on the virtual maze accurately categorized 85% of the sample with schizophrenia, and scores in the maze were linked to several items from a standardized symptom rating scale. In combination with results from the current study, these findings suggest that VR technology with schizophrenia may be used effectively for assessment of both neurocognitive and instrumental role functioning skills.
This work was supported by grant K08 MH-69888 from the National Institute of Mental Health (NIMH), a Young Investigator Award from the National Alliance for Research on Schizophrenia and Depression, a Hartford Hospital Small Grants Initiative Award and gifts from the Spencer T. and Anne W. Olin Foundation to the Institute of Living, Hartford Hospital's Mental Health Network (Dr Kurtz), and grants NIMH R01 MH-43775 and MH-52886 and an NIMH MERIT award (Dr Pearlson).