|Home | About | Journals | Submit | Contact Us | Français|
Autism Spectrum Disorder (ASD) is a prevalent and costly neurodevelopmental disorder. Individuals with ASD often have deficits in social communication skills as well as adaptive behavior skills related to daily activities. We have recently designed a novel virtual reality (VR) based driving simulator for driving skill training for individuals with ASD. In this paper, we explored the feasibility of detecting engagement level, emotional states, and mental workload during VR-based driving using EEG as a first step towards a potential EEG-based Brain Computer Interface (BCI) for assisting autism intervention. We used spectral features of EEG signals from a 14-channel EEG neuroheadset, together with therapist ratings of behavioral engagement, enjoyment, frustration, boredom, and difficulty to train a group of classification models. Seven classification methods were applied and compared including Bayes network, naïve Bayes, Support Vector Machine (SVM), multilayer perceptron, K-nearest neighbors (KNN), random forest, and J48. The classification results were promising, with over 80% accuracy in classifying engagement and mental workload, and over 75% accuracy in classifying emotional states. Such results may lead to an adaptive closed-loop VR-based skill training system for use in autism intervention.
Autism Spectrum Disorder (ASD) is a grouping of neurodevelopmental disabilities characterized by pervasive impairments in social communication and behavioral functioning. The estimated prevalence of ASD is 1 in 68 . While researchers have extensively studied how to improve social skills, language development, and emotion recognition in young children with ASD [2, 3], far fewer studies have focused on meaningful skills related to adaptive adult independence such as driving. Many individuals fail to achieve typical milestones related to adult independence, with the ability to drive an automobile representing a particularly important skill for individuals with ASD as well as typically developing adults.
Recent literature has shown that individuals with ASD face difficulty in learning and maintaining driving skills [4–6]. Compared with their typically developed (TD) peers, individuals with ASD have difficulty in identifying driving hazards that include a person  and appear to evidence differences in gaze pattern and physiology signals while driving . In our previous work , we found that the average gaze positions on the driving field of view among the ASD group is significantly higher in the vertical direction and towards the right in the horizontal direction compared to the TD group, which was consistent with the results found in .
A growing number of studies have investigated Virtual Reality (VR)-based intervention for children with ASD . VR technology offers the potential to create an immersive, interactive, and realistic environment for behavioral learning and generalization in real world situations. VR-based intervention platforms in autism treatment have the advantages of providing precise control of complex stimuli, providing individualized treatment, and creating a structured and safe learning environment [8, 9]. We have recently developed a novel VR-based driving system for the purpose of addressing driving skills of adolescents with ASD . A future goal is to incorporate an EEG-based Brain Computer Interface (BCI) into this VR-based driving system. EEG-based BCI may provide a non-invasive way to estimate the engagement level, emotional states and mental workload of adolescents with ASD in order to facilitate individualized system adaptation for driving skill learning.
Traditionally, BCI applications have been developed as a new control and communication channel for individuals with severe motor disabilities . With the emergence of more affordable commercial EEG devices such as Emotiv EPOC  and NeuroSky Mindwave , BCI systems are becoming available for other applications such as video games, neuromarketing, etc. In this paper, we explored the feasibility of detecting engagement level, emotional states and mental workload of adolescents with ASD using EEG signals while they drive using a VR-based driving simulator. EEG signals have been previously used to recognize attentive state, engagement level, mental workload, and emotions in various applications [13–15]. The rest of the paper is organized as follows. Section II outlines the architecture of the VR-based driving system and the structure of the EEG-based BCI. Section III describes the experimental setup and procedure. Section IV presents the methods for EEG data analysis as well as classification results. Finally, section V discusses the implication of the results and highlights the future directions.
The architecture of the VR-based driving system is depicted in Fig. 1. There were five primary modules: VR Driving Module (VDM), Physiological Data Acquisition Module (PDM), EEG Data Acquisition Module (EDM), Gaze Data Acquisition Module (GDM), and Observer-based Assessment Module (OAM). The communication among the modules was implemented based on the client/server architecture through a local area network. VDM was composed of a virtual environment and a Logitech G27 controller. Participants operated a virtual vehicle inside the virtual environment using a steering wheel and pedal board. The virtual environment was built as a driving game with six difficulty levels. Each difficulty level contained three assignments, which were further decomposed into eight trials. The trials were designed to address specific driving skills such as decreasing speed in a school zone, lane merging, left/right turning, etc. During each driving session, participant’s EEG signals, physiology signals, eye gaze data and performance data were recorded by EDM, PDM, GDM and VDM, respectively. OAM was responsible for recording behavioral assessments of participants using a 0–9 rating scale with five categories: engagement, enjoyment, frustration, boredom, and difficulty. Three emotional states including enjoyment, frustration, and boredom were selected due to their importance in driving. Mental workload was found to increase linearly as level of difficulty increased in several tasks , and was shown to have been modulated by task difficulty in a closed-loop robot-assisted gait training system . Given the individual differences, we measured the perceived difficulty level of the game instead of the actual game difficulty level as the mental workload of adolescents with ASD. In this paper, we only focus on the EEG-based data analysis.
The proposed EEG-based BCI for the VR-based driving system consists of three main modules: Signal Preprocessing module, Feature Generation module, and Classification module. In this context, BCI serves as an implicit communication channel to enrich the human-computer interaction. The raw EEG signals collected from the scalp of the participant are first fed into a Signal Preprocessing module to remove outliers, correct EOG and EMG artifacts, and enhance the signal-to-noise ratio. Feature Generation module then transforms the time series signals into a set of meaningful features for Classification module to detect the engagement level, emotional states and mental workload of the participant. Feature Generation module is comprised of two steps: 1) extract features from the time series, and 2) reduce the number of features by either selecting a subset of useful features or representing the features in a lower-dimensional space. Finally, the classification results are sent to the VR driving module and could be used together with physiology signals, eye gaze data and performance data to make decisions on the adaption of the virtual driving game. In this work, the behavioral assessments data from OAM were used as ground truth to train a group of models that could be used in Classification module to classify engagement, enjoyment, frustration, boredom, and difficulty.
Each participant took part in six experimental sessions of approximately 60 minutes on different days. At the start of each session, the driving simulator seat was adjusted for the comfort of participants. Then, EEG sensors and physiology sensors were placed on the head and the body of each participant respectively, followed by an eye tracker calibration process. In the first session, participants watched a short tutorial that explained the basic rules and vehicle’s controls. Before the beginning of the task, three minutes of baseline data, including EEG signals and physiology signals were collected while the participants were asked to sit quietly in the simulator with their eyes opened. The driving part of the session consisted of a three-minute practice followed by three assignments. Practice mode was designed for the purpose of familiarizing the participants with the controlling of the vehicle in the game. There were no restrictions on participants’ driving behaviors. In contrast, in the assignment mode, participants followed GPS instructions and tried to obey traffic rules. The first and the last sessions were pre- and post-tests, respectively, and consisted of the same assignments from a variety of difficulty levels. The rest of the sessions selected assignments from the same difficulty level while the difficulty increased from one visit to the next.
Fig. 2 illustrates the experimental setup. A 14-channel Emotiv EPOC neuroheadset was used for collecting EEG signals from 0.2 to 45Hz at a sampling rate of 128Hz. These EEG sensors were placed at positions AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4, defined by the 10–20 system of electrode placement . Two additional reference sensors were located at positions P3 and P4. All the signals were measured with respect to reference sensor P3. We recorded the 14-channel EEG signals, headset rotational acceleration in X and Y directions, the contact quality of each sensor, and messages received from the VDM. The received messages were composed of timestamps and event descriptions and were used to align the EEG signals with driving situations and therapist’s rating. Examples of these messages were start/end of an assignment, start/end of a trail, and failure events.
In each session, a therapist observed the participants and their performance on the driving game from an adjacent room and used a novel program to provide online rating on the five categories. Two rating files were logged: 1) overall rating at the end of each assignment, and 2) periodic rating during each assignment on participants’ instantaneous states.
Sixteen male teenagers participated in the experiment. We did not exclusively recruit males, but ASD is estimated to occur with a 5:1 male to female ratio explaining this overrepresentation. The age of the participants ranged from 13 to 18 years (M = 15.24 years, SD = 1.63 years). All participants had a clinical diagnosis of ASD and scored at or above clinical cutoff on the Autism Diagnostic Observation Schedule . Twelve participants finished all six sessions, three participants finished two sessions, and the last participant finished four sessions. The final data set included 82 sessions. Participants were reimbursed for each visit and this study was approved by Vanderbilt University’s Institutional Review Board.
In this paper, we used a therapist’s overall subjective rating data to train classification models. One feature vector from each assignment was extracted and labeled with its corresponding rating scores. For each category, the continuous score in range [0, 9] were mapped to two classes, high or low, for binary classification.
The following procedure was used to reject and correct artifacts in the EEG signals, as well as to increase the signal-to-noise ratio. First, spikes in EEG signals were removed by slew rate limiting with a rising slew rate parameter of 50μV and a falling slew rate parameter of −50μV. Then, a 0.2–45Hz band-pass filter was applied on the EEG signals. Data from each assignment were then segmented into one-second epochs with 50% overlap. For an individual epoch to be included in the analyses, all sensors had to be in good contact with the subject’s scalp, there were no interpolated data points due to slew rate limiting, and less than 33% of the channels exceeded the voltage threshold of 150μV. Data in accepted epochs were re-referenced to the average reference. EOG and EMG artifact correction algorithms [19, 20] were then applied on the re-referenced data. We rejected the epoch again if any single channel in the epoch contains artifacts and thus failed to pass all four tests: channel deviation test, variance test, amplitude range test, and median gradient test . The remaining epochs (M = 268.48, SD = 147.72) were used to calculate the feature vector.
Mean-centering was performed on each extracted epoch followed by a hanning window function. Power spectral density variables (μV2) were then computed from the time series epoch data by summing power spectral density values across each 2Hz bin from 4Hz to 44Hz. Delta band activities (1–4Hz) were excluded due to potential frequency overlap with electrode-shift artifacts. Spectral features for each assignment were calculated by averaging the power spectral density variables across all the extracted epochs in that assignment, resulted in a 280 (14 channels × 20 frequency bins) dimensional feature vector. In the end, a total number of 201 out of 246 (82 sessions × 3 assignments) samples were used to train classification models.
The WEKA data mining software  was utilized to classify engagement, enjoyment, frustration, boredom, and difficulty as well as compare different classification methods. Preprocessing on the sample data included normalizing and standardizing the features and then balancing the number of instances in two classes. The principle component analysis (PCA) method was applied for dimensionality reduction to avoid overfitting. Seven supervised learning methods were compared for classification of each category: Bayes network with SimpleEstimator estimator and K2 search algorithm, naïve Bayes, Support Vector Machine (SVM) with a three degree radial basis function kernel, multilayer perceptron that has a number of (features + classes)/2 hidden layers and a 0.3 learning rate, K-nearest neighbors (KNN) with Euclidean distance function, random forest with 100 random trees and J48, a pruned C4.5 decision tree.
A 10-fold cross validation technique was used to estimate the accuracies of each classification method on all categories. Fig. 3 summarized and compared the classification accuracies. The highest accuracy for classifying engagement, enjoyment, frustration, boredom, and difficulty were 85.96% (KNN), 77.25% (KNN), 75.74% (KNN), 77.63% (random forest), and 81.46% (KNN), respectively. Given the complexity of the experimental setup and stimuli types, and variation in age and conditions of the subjects, such classification accuracies were very promising compared with the state of the art .
In this work we integrated an EEG data acquisition module into the VR-based driving system and explored the reliability of detecting engagement level, emotional states, and mental workload of adolescents with ASD while driving. We included three emotional states (enjoyment, frustration, and boredom) and used difficulty rating scores to represent mental workload. EEG signals and the therapist’s overall behavioral assessments were used to train a group of classification models for each category. We extracted spectral features from assignment related EEG signals and achieved classification accuracies of over 80% for detecting engagement and mental workload, and over 75% for detecting emotional states. In most cases, KNN classification method obtained the best accuracy. These results suggest that EEG-based BCI could be used in the VR-based driving system to enrich the human computer interaction, and more importantly, improve the system efficiency through individualized system adaptation based on multimodal sensory data and performance data. It will be promising and interesting to study the EEG signals of a TD group in reponse to the same system and potentially find the specific reaction pattern of the ASD group. However, in this work, our aim is to explore the use of EEG signals towards an adaptive closed-loop system for better driving skill training for individuals with ASD.
While the results are promising, note that we only included features in frequency domain for this study. Better classification results may be achieved by adding other types of features such as time domain features and synchronicity features. Examples of time domain features are entropy and autoregressive coefficients. Synchronicity features provide information on the relationship between two or more channels. Periodic ratings logged during the assignments are more informative and could augment overall ratings when training classification models. What’s more, constructing ensembles of classifiers may improve classification. In the future, we plan to implement the proposed structure of EEG-based BCI for the VR-based driving system. We will also incorporate physiology signals, eye gaze data, and performance data for individualized adaptation of the virtual driving game.
*Research supported in part by the National Institute of Health Grant 1R01MH091102-01A1, National Science Foundation Grant 0967170 and the Hobbs Society Grant from the Vanderbilt Kennedy Center.
Jing Fan, Electrical Engineering and Computer Science Department, Vanderbilt University, Nashville, TN 37212 USA.
Joshua W. Wade, Electrical Engineering and Computer Science Department, Vanderbilt University, Nashville, TN 37212 USA.
Dayi Bian, Electrical Engineering and Computer Science Department, Vanderbilt University, Nashville, TN 37212 USA.
Alexandra P. Key, Vanderbilt Kennedy Center for Research on Human Development and Department of Hearing and Speech Sciences, Vanderbilt University, 230 Appleton Place, PMB 74, Nashville, TN 37203 USA.
Zachary E. Warren, Treatment and Research Institute for Autism Spectrum Disorders (TRIAD), Pediatrics, Psychiatry and Special Education, Vanderbilt Kennedy Center, 230 Appleton Place, PMB 74, Nashville, TN 37203 USA.
Lorraine C. Mion, Vanderbilt University School of Nursing, Nashville, TN 37204 USA.
Nilanjan Sarkar, Mechanical Engineering Department, Electrical Engineering and Computer Science Department, Vanderbilt University, Nashville, TN 37212 USA.