BRAIN DATA AND DRIVER’S ATTENTION DURING SIMULATED DRIVE

. The attention of drivers is a serious issue and one of the critical factors of road safety. The question is whether the electrical activity of the human brain can be correctly measured/collected and utilized to monitor and interpret the driver’s attention during simulated driving. This article summarizes four experiments that have been designed, performed, and evaluated in the neuroinformatics laboratory at the University of West Bohemia. Simulated driving under various conditions in a car simulator was organized, and electrophysiology, mainly electroencephalography, data were collected from participants/drivers. The results include experience with the design of such experiments and the suitability of methods based on the collection and interpretation of electroencephalography data for driver attention detection.


Introduction
The attention of drivers is a serious issue and one of the critical factors of road safety.Inattentive drivers are dangerous not only to themselves but mainly to their surroundings and cause many accidents.However, the decline of attention, especially during long rides, is natural, and it is necessary to take this phenomenon into account.This paper summarizes some of the experiments that have been designed, performed, and evaluated in the neuroinformatics laboratory at the University of West Bohemia.Simulated driving under various conditions in a car simulator was organized, and electrophysiology (mainly electroencephalography, event-related potential) data were collected from participants/drivers. Electroencephalography (EEG) is a method and technique that records and evaluates the electrical activity of the (human) brain.A set of electrodes is placed at various (often predefined) locations on the scalp.These electrodes capture voltages (resulting from the firing of large neural circuits) that must be amplified and recorded.Event-related potentials (ERPs) are changes in the electrical brain activity that are time-locked to particular events (stimuli).ERPs are extracted from the underlying EEG data.
The main research objective of the presented set of four experiments was to find and verify if the electrical activity of the human brain (EEG/ERP data) could be correctly measured/collected and utilized to monitor and interpret the driver's attention during simulated driving.
The paper is organized as follows.The materials and methods section deals with the fundamentals of EEG/ERP methods and their benefits and drawbacks in investigating driver's attention.It further intro-duces the neuroinformatics lab at the University of West Bohemia and summarizes four performed experiments.The results section provides the results and conclusions from individual experiments.The conclusion section summarizes experience from these experiments and gives several cautious prospects for the future.

Materials and methods
This section provides a fundamental insight into the advantages and disadvantages of collecting, monitoring and analyzing the electrical activity of the human brain using the methods and techniques of electroencephalography and event-related potentials when interpreting the driver's attention during simulated driving.Then the neuroinformatics lab at the University of West Bohemia is introduced, and the description of four experiments dealing with driver's attention and using the EEG/ERP methods is provided.

Electroencephalography and event-related potentials
Electroencephalography has many advantages: affordability, non-invasiveness, routine examination protocols, and the opportunity to measure spontaneous activity.However, it also has a significant disadvantage, evident in scientific experiments.The resulting picture of brain activity (the EEG signal) is rough since it represents many sources of neuronal activity.Thus it is challenging to derive individual neurocognitive processes from the measured brain activity.Another disadvantage is the relatively difficult collection of a quality EEG signal outside the laboratory since only observable and clean data can be reasonably analyzed to interpret the driver's attention during simulated driving.
ERPs have two advantages compared to behavioral methods.They help determine which stage or stages of processing are influenced by a given experimental manipulation; a detailed set of examples can be seen, e.g. in [1].The second advantage of ERPs is that they can provide an online measure of stimuli processing, even when there is no behavioral response [2].
ERP components are obtained from the EEG signal by averaging the epochs around the events.The P300 (also labelled as P3) component depends entirely on the task performed by the subject and is not directly influenced by the physical properties of the stimulus [1].
The P300 component is sensitive to the probability of the target stimulus.P300 amplitude increases when the probability of the target stimulus class decreases.The amplitude of the P300 component becomes larger when a greater number of non-target stimuli precedes it.P300 amplitude is also larger when the subject pays more attention to a task.On the other hand, P300 amplitude is smaller if the subject does not know whether a given stimulus is/is not a target.It means that more complex tasks can increase P300 amplitude because the subject pays more attention to these tasks and simultaneously decrease P300 amplitude because the subject is not certain of the stimulus category [1].
P300 latency is associated with stimulus categorization; it increases when stimulus categorization is postponed.Since P300 latency does not depend on consequent processes (e.g.response selection, in our case, the driver's reaction), it can be used to determine if a performed experiment influences the processes of stimulus categorization or other processes related to a response (driver's reaction).
The experimental design of EEG/ERP experiments is a challenging and often critical step that influences other technical issues related to EEG/ERP research.An apparent and detailed description of ERP experimental design as a set of rules and strategies (e.g.focusing on specific, large, and easily isolated components or comparing only ERPs elicited by the same physical stimuli) is also provided in [1].Although these rules and strategies constrain the experimental design, they are not a standard part of the software tools used to design ERP experiments, and they are often violated in many published studies.Consequently, it was not easy to get inspired by the published studies when we designed the first experiments related to drivers' attention.

Related studies
While most studies examining the attention of drivers use behavioral methods (i.e. the behavior of the driver under certain conditions is investigated -e.g., how often he/she leaves his/her lane), not many experiments also use EEG and ERP methods and techniques.
The suitability of EEG-based techniques for recording drivers' activity during simulated driving was investigated in [3] where oscillations of brain electrical activity (frequency bands) were analyzed.As a result, an increase in alpha activity was interpreted as less attentional activity and a decrease in alpha activity as more attentional activity.Significant differences between drivers were observed.
The ERP method (P300 amplitude) is utilized in [4] where the impact of the secondary task performance (an oddball auditory task) on a primary driving task (lane keeping) was investigated.The study showed that when performing a simple secondary task during driving, the performance of the driving task and this secondary task were unaffected.However, analysis of brain activity showed reduced cortical processing of irrelevant and potentially distracting stimuli from the secondary task during driving.
The use of EEG data for the evaluation of driver fatigue was provided in [5].Energy parameters were computed, and finally, the evaluation model for driver fatigue was created based on the EEG data from the electrodes Fp1 and O1.Muscle artifacts were minimized by the experimental protocol design, and all trials showing artifacts linked to eye movements or blinks were removed before averaging ERPs.The model accuracy was about 92.3 %.
The impact of a surrogate forward collision warning system and its reliability according to the driver's attentional state was introduced in [6].Both behavioral and electrophysiological (amplitudes and latencies of several ERP components) data were recorded and evaluated.These results showed electrophysiological data as a valuable tool to complement behavioral data allowing a better understanding of how these systems impact the driver.
Using the ERP technique, it was found in [7] that the brain activity associated with processing the information necessary for the safe operation of a motor vehicle was suppressed when drivers were talking on a cell phone.The P300 amplitude was reduced, and the P300 latency was delayed when participants were engaged in phone conversations and reacted to intermittent lead vehicle deceleration.
The effect of regular and prior sleep restricted to five hours during simulated driving was studied in [8].
The study was carried out on 20 younger and 19 older healthy men.After a short sleep, younger drivers showed significantly more sleep-related deviations and greater 4 to 11 Hz EEG power, indicative of sleepiness [8].
A systematic framework for measuring and understanding cognitive distraction in three experiment settings (laboratory control, driving simulator, and instrumented vehicle) was presented in [9].Participants completed eight tasks commonly performed by the driver; primary, secondary, subjective, and physiological measures (the P300 component) were collected and integrated into a cognitive distraction scale.They vol.39/2022 Brain Data and Driver's Attention during Simulated Drive concluded that impairments in driving were directly related to the cognitive workload of these activities.
Simultaneous recording of EEG and eye-tracking to investigate situation awareness and working memory load in distracted driving was introduced in [10].The driver-vehicle-environment state-space related to drivers' conditions and environmental factors around the vehicle were described using (also) an ontology.
The amplitude of the P300 component reflecting individual differences in navigation performance in a driving task was investigated in [11].Two groups of navigators with good and poor navigation performance participated in a driving task; the P300 amplitude was measured while two types of triggers were presented (intersections and street signs).In general, poor navigators showed a larger P300 amplitude than good navigators.
To identify EEG oscillatory parameters for cognitive mechanisms involved in high and low controllability tasks, time spent on a task, task load, and cognitive controllability (using either re-active or pro-active driving task) in simulated driving scenarios were investigated in [12].The results on thirty healthy participants demonstrated that the controllability of a driving situation had a similar effect on oscillatory EEG activity like time on task and task load.
The drivers' fatigue was measured in [13] where EEG and forehead EOG were fused.The percentage of eye closure was calculated using eye movement data (recorded by eye tracking glasses as the indicator of drivers' fatigue level).The prediction correlation coefficient and root mean square error between the estimated and real fatigue levels were used to evaluate the performance of a single modality and fusion modality.The experimental results on twenty-one healthy subjects showed that this fusion could improve the performance of driving fatigue detection.
A real-time driving fatigue detection system based on a wireless dry EEG acquisition system was presented in [14].The prediction of fatigue in ten healthy subjects was consistent with the observation of reaction time recorded during simulated driving.
The study in [15] concluded that the EEG signal from the prefrontal brain region (forehead area) could be used to detect fatigue of drivers, although the signal classification accuracy (the SVM algorithm used for classification) was not high.
A survey on EEG-based driver state detection systems and their corresponding analysis algorithms over the last three decades was provided in [16].The authors concluded that the current EEG-based driver state monitoring algorithms were promising for safety applications, but many improvements were still required in EEG artefact reduction, real-time processing, and between-subject classification accuracy.
A two-level learning hierarchy radial basis function model was used to detect EEG-based driving fatigue in [17].The authors concluded that the proposed model achieved a better classification performance compared to other used artificial neural networks.
Multi-channel EEG recordings during a sustainedattention driving task were presented in [18].They included 62 sessions of 32-channel EEG data for 27 subjects driving on a four-lane highway.
A recurrent residual network used to analyze EEG data captured during simulated sustained attention driving tasks was presented in [19].The authors demonstrated the competitive results achieved by comparing this network with other benchmark models.
An attention-based multiscale convolutional neural network with a dynamical graph convolutional network proposed to detect driving fatigue was introduced in [20].The authors concluded that their model outperformed six widely used competitive EEG models with a high accuracy of 95.65 %.
An EEG-Based Spatio-Temporal Convolutional Neural Network for Driver Fatigue Evaluation was introduced in [21].Fatigue driving experiments were conducted to collect EEG signals from eight subjects alert and in fatigue states.The results indicated that the network fulfilled a better classification accuracy of 97.37 % than the eight methods it was compared with.
The study [22] investigated whether measures from low-cost devices monitoring peripheral physiological states (eye-tracker, heart rate monitor, and a highfidelity 32-channel quick-gel EEG system) were comparable to standard EEG measures in predicting lapses in attention to system failures.Twenty-five participants were engaged in a fully autonomous lane-changing driving task.The results showed that current lowfidelity technologies were not sensitive enough to reliably model reaction time to critical signals.
To summarize the related studies section, we can conclude that EEG and ERP methods and techniques are promising for detecting driver fatigue and the impact of various disturbing events and tasks on driver's attention.Although the studies usually work with fatigue, vigilance or cognitive performance, we will use the term attention to align with the fundamental explanation of EEG/ERP methods.We were surprised that many studies did not pay too much attention to design issues of such experiments since only a wellprepared design can lead to relatively clean data and reasonable interpretation.

Neuroinformatics laboratory equipment
The neuroinformatics laboratory at the University of West Bohemia is equipped with all necessary hardware infrastructure for driving simulations and recording EEG/ERP signals and additional biosignals.The experimental car simulator (a front part of a real Škoda Octavia car, Figure 1) is equipped with the Logitech G27 wheel, accelerator, and brake.These are connected to the control computer via the USB port.The tracks are prepared mainly using the World Racing 2 game produced by the Synetic Company [23].The track is projected on the wall in front of the car simulator.
The stimuli were presented either using a custom programmable hardware stimulator (for visual stimulation with LEDs) placed on the car windshield or the Presentation software tool (for auditory stimulation) produced by Neurobehavioral Systems, Inc. [24].
V-Amp produced by the Brain Products company was used as an EEG amplifier.It also served as an input of the sensors (also produced by the Brain Products company) capturing other biosignals.The BrainVision Recorder [25] was used for recording and local storing EEG/ERP and other biosignal data (e.g.respiration rate or skin conductance).MATLAB, EEGLAB, ERPLAB, BrainVision Analyzer software tools, and Python tools (especially the MNE library) were used to process and analyze experimental data.
Standard EEG caps (the 10-20 system defining locations of scalp electrodes) were used depending on the size of the participants' heads.The reference electrode was placed approximately 0.5-1 cm above the nose, and the ground electrode was placed on the ear.The biosignal sensors and the EEG cap were connected to the V-Amp amplifier.
Three computers were usually used: the first for the presentation of auditory stimuli (if they were presented), the second for storing recorded data, and the third for the track presentation.The diagram of the standard experimental setup is depicted in Figure 2.
EEG/ERP data were recorded with the sampling frequency 1 kHz; no filters were used during data recording.The resulting signal was stored in three files: .eegfile containing raw data; .vhdrfile containing metadata that describe raw data in .eegfile, and .avgfile containing the averaged signal around the used stimuli.IIR filter was applied to data from the Fz, Cz and Pz electrodes.These three electrodes were also selected for further processing.

Methodology -Experiments
Four already published EEG/ERP-based experiments investigating driver's attention during simulated driving are presented to show the evolution of our experimental work in EEG-based simulated driving.Although we could theoretically include dozens of mostly unpublished experiments in this work, various driving scenarios and overall driving conditions would confuse the article.
An appropriate experiment design (including the choice of the event-related component, type of stimuli, driven track, or entire data collected) has to be set and evaluated to achieve research objectives.When observable and clean data are collected, it is reasonable to interpret the driver's attention during simulated driving.The event-related component P300 was selected based on its nature, use in related studies, and the outcomes of our preliminary unpublished experiments.
In the first presented experiment, we proposed and tested the constraints related to the complexity of the experimental design.The experiment design was improved in the second presented paper.The third paper introduced more tiresome conditions for participants; not only a monotonous track but various daytime conditions and sleep deprivation were part of the experiment protocol.The last experiment used two signals collected in parallel to detect the driver's attention during a simulated drive -brain electrical activity and respiration.
EEG/ERP data were recorded with the sampling frequency of 1 kHz; no filters were used during data recording.The resulting signal was stored in three files: .eegfile containing raw data, .vhdrfile containing metadata that describe raw data in the .eegfile, and .avgfile containing the averaged signal around the used stimuli.Various IIR filters were applied to data from the Fz, Cz and Pz electrodes in the processing phase, typically in the frequency range 0.01-20 Hz.The Fz, Cz and Pz electrodes were also selected for further processing.The segmented data containing artifacts were first automatically and then manually rejected.

Experiment 1
The main objective of this work [26] was to determine if EEG/ERP methods and techniques and the event-related P300 component itself can be successfully utilized to investigate how the mental load of drivers affects their attention during a simulated drive.The second objective was to test the constraints of the experimental protocol complexity.
A relatively challenging virtual track containing sharp turns and allowing for potential collisions with oncoming cars (target stimuli) was designed and tested on two groups (nine volunteers) of drivers (affected/unaffected by alcohol).The occurrence of the P300 component was investigated.Then, an alternative protocol relying on simpler target stimuli (flashing diodes and simple sounds) was proposed, implemented, and tested.

Experiment 2
This paper's [27] objective was to detect changes in a driver's attention during a monotonous simulated drive.It was done using EEG/ERP techniques, performing auditory stimulation, and investigating properties of the P300 component related to attention.
The ERP experiment was designed to investigate the driver's attention changes during monotonous 40 minutes long simulated drive.The experiment was performed on 14 participants who drove a car simulator on a monotonous track (a motorway without disturbing obstacles).The participants were audibly stimulated (simple beeps of various duration and the sound of a crying child).The data were preprocessed (filtering, artefact rejection, epoch extraction, and baseline correction) and investigated for the P300 component's occurrence and peak latency in subsequent eight-minute intervals.

Experiment 3
The objective of the experiment [28] was to introduce more tiresome conditions for participants.Not only a monotonous track but also various daytime and sleep deprivation were part of the experiment protocol to detect changes in the driver's attention during a monotonous simulated drive.EEG and ERP techniques and auditory stimulation were utilized; changes in the peak latency of the P300 component over time were investigated.
The same hypothesis as in the previous experiments (the peak latency of the P300 component increases over time as the driver becomes more tired) was considered, but the experiment design differed and evolved in time.
A monotonous track was constructed in this experiment, and a simple auditory experiment was designed (three stimuli -simple target and non-target stimuli of 500 ms duration time and different frequencies, a rare stimulus of 1000 ms duration time).Each participant (11 participants in total) underwent four twenty-minute drives held over two days; each day, morning and afternoon sessions were held; on the first day, the participant usually slept, and on the second day, he/she suffered from sleep deprivation.Each drive was divided into time intervals of the same length (the stimulation was provided only in some time intervals to ensure relaxation time and prevent participants from familiarity with the presented stimuli).The latencies of the averaged P300 components were compared depending on the daytime and sleep deprivation.

Experiment 4
The paper's [29] objective was to consider two biosignals collected in parallel to detect the driver's attention during a simulated drive; these bio-signals were the electrical activity of the human brain (as in the previous experiments) and respiration.Moreover, the collected data were validated using a stacked autoencoder to guarantee their quality and annotated to make them publicly available and reusable.
The experiment was built on the previous experiments, considering the basic assumptions related to the P300 component and driving a car simulator on a monotonous track.The experiment (simulated drive) was prolonged to 60 minutes and divided into three sessions.Auditory stimulation was performed (target and non-target stimuli of the same length), and the background sound of drizzling imitated the natural environment.The respiration rate was captured parallel with the brain's electric activity using the same hardware amplifier.The data from a group of 15 participants were annotated, stored, preprocessed, validated, and partly analyzed (classified using a stacked autoencoder).

Results
The first experiment [26] brought significant resultsthe proposed track appeared too complex to provide experimenters with the observable occurrence of the P300 component, but its simplified alternative version was more promising.The P300 component was not observed in the grand average waveforms when the more complex target stimuli (sharp turns and possible collisions) were part of the virtual track.A similar trend of the grand average waveforms for both groups of subjects (affected/unaffected by alcohol) was observed.When target stimuli were elicited by external hardware devices (flashing diodes or sound generators), the P300 component was easily observable.
It was shown that the experiment design had to be relatively simple; otherwise, the event-related P300 component could not be identified.It is necessary to be sure if and when the P300 component is elicited.The subsequent experiments thus followed a more simple design.
The experiment design was improved in the second driving experiment [27].As a result, the P300 component was easily observable in the grand average waveforms.However, both variants of the experiment design (using a different location of the reference electrode) did not show gradually increasing peak latency concerning the length of the simulated drive.There were 30 % of artifacts in the epochs related to target stimuli.The target stimulus (a 600 ms long sound of a crying child) was too long and finally made the P300 component less analyzable.
Selecting a long target stimulus (a crying child) was probably not a good design solution since it stretched the P300 component length.Stretched components are less analyzable; their absolute coordinates are distorted.There were also not many target stimuli, and because of the removal of the artifacts, only 70 % of them entered the averaging process.The experiment's prolongation and introducing more tiresome conditions (e.g., exposing participants to heat, absence of sleep, or placing them in a darkened room) were further considered.
The P300 component was clearly identified in all experimental sessions of the third experiment [28].Figure 3 shows the grand averages on the electrode Fz (frontal central electrode) for all experimental sessions.On the other hand, the prolongation of the peak latency of the P300 component overtime was not observable when the grand average measure of each participant was investigated.However, the prolongation of the peak latency of the P300 component was observable when the techniques of peak latency and fractional 50 % area latency were applied to compute the grand average for each experimental session.The latency of the P300 component was not influenced by the time of day.The latency of the P300 component increased with sleep deprivation.
The experiment showed that the P300 component's prolongation was identifiable when the techniques of peak latency and fractional 50 % area latency were applied.We concluded that for the following experiments, it would be beneficial to have a larger number of target stimuli (i.e., the simulated drive has to be prolonged) and more participants.
In the fourth experiment [29], the P300 component was identified in most participants during all driving sessions.A high amplitude of the N2 component, when compared to the amplitude of the P300 component, was detected in some cases.Prolongation of the peak latency of the P300 component was evident in most participants when their peak latencies were averaged.However, prolongation of peak latency in time was not clearly observed when grand averages for all participants were investigated.Average respiration rate and respiration rates for most participants showed a decreasing trend during the experiment.
In this experiment, the trend of the increasing latency of the P300 component and decreasing respiratory rate was relatively clearly visible (Note: the data were not statistically evaluated).Since the experimental results were naturally affected (e.g., by different brain reactions of participated drivers, the sensibility of captured data to the environmental noise,  or participants' overall mental conditions), there was a significant effort to eliminate these circumstances by experiment design, setting of experimental conditions, and appropriate use of data preprocessing and processing methods.Moreover, the data were finally validated using the stacked autoencoder.

Conclusions
Various experiment protocols were designed (dozens of experimental protocols were tested, and four published ones were presented in this paper) to set and verify constraints significant for the successful investigation of drivers' attention utilizing the human brain's electrical activity and the EEG/ERP methods and techniques.
The initial experiments were over-specified; the experiment protocols were too complicated, naive, and unrealistic, given the method used.Over time, the experiment design was improved; the event-related P300 component was finally easily identifiable.The interesting results regarding the latency of the P300 component were observed, e.g., its increase with overall fatigue or sleep deprivation of drivers was detected.
On the other hand, we had to cope with the troubles related to participants' willingness to wear an EEG cap for a longer time, handle a longer drive in the simulator, or reduce unnecessary movements.The number of produced artifacts finally put time constraints on the designed experiments.
All the work related to the collection, long-term storage, processing, and interpretation of EEG/ERP data/metadata and publication of results also revealed the troubles with the data/metadata descriptions, procedures, and reproducibility of the findings.Even reproducing a single experiment by the same experimenter proved practically impossible in a few months.The need for standardized data annotations and formats, procedures, and workflows turned out to be enormous to share the data, procedures, and outcomes with the community and understand data, procedures, and conclusions after a particular time.
The investigation of driver's attention has finally become interesting for automotive in-car assistance systems and self-driving car development.The assistance systems that prevent drivers from causing accidents have been improved and can monitor drivers' behavior and vital signs (mainly related to attention) such as movements, eye blinks, and heartbeat.Moreover, self-driving cars are being developed to spare or even prevent people from driving.However, the data collection methods presented here are still complicated and impractical, and the results are not so reliable to be used in cars during actual driving.On the other hand, these methods can be used for verification purposes in laboratory conditions.

Figure 3 .
Figure3.Grand average on the electrode Fz -experimental sessions differ in the daytime and duration of sleep[28].