Comparing Eye-tracking System Effectiveness in Field and Driving Simulator Studies

3.1. Equipment

The equipment used for this research comprised an eye-tracking system, a driving simulator, and an instrumented vehicle specifically used for field surveys. The characteristics of the tools are described below.

3.1.2. Driving Simulator

The driving tests were performed using the fixed-based driving simulator of LASSTRE (Fig. 1b). The virtual reality laboratory is equipped with a STISIM driving simulator, placed in a full-cab Toyota Auris that has been converted into a driving simulator by removing all the unnecessary mechanical parts and replacing them with electronic systems connected to a workstation. The driver's feedback is controlled by a force-feedback steering wheel, brake, and accelerator pedals. Three projectors ensure a wide image with a 180° field of view on a curved screen located in front of the vehicle. The resolution of the visual scene is 1920 x 1200 pixels with a refresh rate of up to 60Hz. Road environmental sounds, such as the actual engine as well as those of the other vehicles, are reproduced in the vehicle to improve the realism of the test. The driving simulator is a very useful tool in the field of road and infrastructure engineering, capable of investigating how different factors, both external and internal to the driver, may affect the perception of driving risk [14].

Using a driving simulator has several benefits, as the costs are considerably lower than in field studies, and the tests are reproducible for all the samples of drivers with the same simulated events and in a controlled environment. The reliability of the tool has been completely validated in previous studies, typically in a rural environment, for speed and trajectory measures [15, 16]. However, further efforts should be directed toward the urban environment.

3.2. Case Study

The case study for the field survey and then reproduced in the simulated environment was conducted at an urban intersection located in Rome. The road cross-section was 18.00m, comprising a carriageway with one lane in each driving direction (each lane was about 4.00m) and both-side sidewalks of about 5.00m. In some sections, some parking areas (2.50 x 6.00m) were designed by restricting the sidewalks to 2.50m. In order to improve the level of realism of the simulation and its similarity with the real case, the intersection was implemented in the driving simulation environment by reproducing the same typology, the number of branches, the directions, and the markings and vertical signs. Moreover, concerning the same interference with the driver, all the vehicles that interacted with the drivers at the intersection, both in the on-field and simulated environment, were controlled by the experimenters. In the case of the field survey, the interfered vehicle was driven by an assistant experimenter. In addition, a series of elements and features were included in the simulated scenario, such as boundary conditions, markings and vertical signs, vegetation, buildings, green areas, and other vehicles. The entire route was 4500m, designed as a circular pattern; two road events, one static and another dynamic, were designed in order to assure the same driving conditions both in the field survey and the simulation experiment: i) the presence of a speed limit sign (two signs along the route), and ii) the presence of a pedestrian who crossed the road. Specifically, the speed limit was 30km/h, and the crossing pedestrian event was designed both in the field survey and the simulated environment in order to have the pedestrian out of the markings in the driver's view at a longitudinal distance of 300m and a lateral distance of 7.5m. Furthermore, when the driver was 100 meters from the pedestrian, he/she started crossing down the road at a speed of about 3.6km/h. However, the driver was not aware of the time when the pedestrian started crossing. It was preferred to study the case of a pedestrian crossing the road out of the marking in order to investigate the driver’s response in a more demanding condition. (Figs. 2 and 3) show the realism of the virtual reality simulation by referring to the two designed events: the presence of a speed limit sign and a crossing pedestrian.

3.3. Participants

For both the on-field and simulator studies, a sample of volunteer drivers was recruited at the Department of Engineering of Roma Tre University, with three selection criteria: over 18 years of age, at least 3 years of licensed driving experience, and unfamiliar with the test site. Twenty-two drivers participated in the real driving and simulation experiments (15 men and 7 women, ages ranging from 22 to 59 years, with a mean age of 31.9 years). In the real tests, eight participants were excluded from the analysis due to technical issues related to the equipment. According to the Chauvenet criterion [17] applied to the speed data, no drivers were identified as outliers and excluded from the analysis. As a result, the final sample included 14 drivers (9 males and 5 females) with an average age of 27.7 years (SD = 9.3). Instead, in the driving simulation tests, four participants were excluded from the further analysis of the data because they experienced symptoms related to simulation sickness. The Chauvenet criterion [17] applied to the speed data reported no outliers. As a result, the final sample included 18 drivers (12 males and 6 females) with an average age of 30.3 years (SD = 10.1).

3.4. Procedure

For each driver, the same test procedure was applied. Both driving tests (field and simulation survey) lasted approximately 50 minutes, with the order being randomized between participants to avoid bias in the results due to the sequence of the drives. During both experimental tests, the drivers were free to choose the speed and generally the driving behaviour, based on their safety perceptions and knowledge of the speed limit along the route. In the simulation test, the participants first drove a training scenario to familiarize themselves with the driving simulator. The training scenario included several situations to allow the driver to test the manoeuvres and obtain feedback from the driving controllers before performing the tests. In addition, the participants had to fill out two questionnaires: the first, before the training, to provide basic information, such as socio-demographic background, age, and driving experience; and the second, after completing the tests, to sign an informed consent form and provide any additional information on simulation sickness to investigate the driver’s discomfort during the test and gain insight into the experience of simulated driving.

3.5. Data Collection

Eye movement data, such as distance and duration of the first fixation, and driving speed profiles at the approach to the events were collected, analyzed, and compared between the two drives, in the real environment and the simulation environment, in order to study the differences in driver behavior in terms of both driving performance and eye movements. Due to the eye-tracking system, some acknowledged indicators [4, 6-10] were collected and studied. More specifically, the first fixation was investigated in terms of duration and distance. According to the literature, the first fixation is the first pause of the eye movement in a specific area of the visual field [4, 6]. For this purpose, two dynamic AOI were designed to enclose drivers’ views around the speed limit sign and the pedestrian and to monitor the eye movement parameters in these areas. It should be mentioned that a filter was applied at the first eye data processing to consider only the eye movements over a value of 30ms. According to the literature [4, 6], eye fixations differ from saccades, which are rapid movements between fixations (typically 30ms). Furthermore, speed is considered an effective safety indicator because of its positive correlation with crash risk [18]. According to the World Health Organization [19], as the average speed decreases, the risk of road crashes and their severity also decreases. As a result, speed data analysis is an effective and important tool for understanding and interpreting driving and eye movement behavior [20]. Consequently, drivers’ speed profiles were collected and analysed from 50 meters before the intersection in both the simulation and real driving.

4.1. Speed Limit Observation

The comparison between the number of times a speed limit sign (there were two speed limit signs along the route) has been looked at by the drivers (Fig. 4) revealed that in the field survey, in around 50% of the total cases (28), the drivers looked at the sign; in the simulation, this percentage was lower than 40% considering a higher number of total cases (36), being the sample composed by 18 valid participants instead of 14 resulted in the field study after the data validation phase. Therefore, the analyses of both the distribution of the values and the statistical validation of the average values were carried out, where at least one fixation on the sign per driver was recorded (Fig. 5). By comparing the distribution of fixation durations (Fig. 5a) and distances (Fig. 5b), it is possible to notice the following: the fixation duration is not considerably different between the two samples. The average value is 295ms in the real survey and 366ms in the simulation survey; accordingly, also the fixation distance is quite similar in the two samples. The average value is around 60 meters in both cases, namely 10 meters after the first point where the sign was visible; the standard deviation of the two samples is different for both the fixation duration and distance. Specifically, the standard deviations of the real survey are higher than in the simulation. This difference could be due to better visibility of the sign in the simulation scenario that allowed the driver to see it earlier and correctly interpret the sign. In addition, a t-test was carried out in order to check the statistical differences among the samples (Table 1). From preliminary analyses of boxplots, no outlier was identified in relation to the fixation durations. Furthermore, this parameter followed a normal distribution in both samples, as verified by the Shapiro-Wilk test (p > 5%).

The difference between the average values of the fixation duration was not statistically significant (t (27) = 0.77; p = 0.44). Accordingly, concerning the speed limit observation, no significant differences in the fixation duration have been recorded between the real and simulated experiments. In terms of the duration of the eye fixation, this finding verifies the validity of the eye-tracking technologies used in the virtual reality driving simulation experiment. Accordingly, the same analysis was carried out for the fixation distance (Table 2). From preliminary analyses of boxplots, no outlier was identified in relation to the fixation distances. Also, for this parameter, the fixation distance followed a normal distribution in both samples, as verified by the Shapiro-Wilk test (p > 5%). The difference between the average values of the fixation distance was not statistically significant (t (27) = 0.51; p = 0.62). Accordingly, also in this case, no significant differences have been revealed in the fixation distance between the real and simulated experiments. This finding confirmed the validation of the eye-tracking systems for the experiment in virtual reality driving simulation in terms of the distance of the eyes' fixation.

4.2. Crossing Pedestrian Analysis

In terms of the crossing pedestrian's fixation, a comparison of real and simulation surveys in terms of fixation durations and distances was conducted 100 meters before the crossing pedestrian's location. In contrast to the analysis of the static object (speed limit sign), the pedestrian, which could be considered a dynamic object, was observed by all the drivers, both in the real environment and in the simulation survey. In both the real and simulation surveys, Fig. (6) shows the distribution of fixation durations and distances on the crossing pedestrian. It is possible to notice the following by comparing the distribution of fixation durations (Fig. 6a) and distances (Fig. 6b): the average values of the fixation durations were different between the real and the simulation surveys (1848ms in the real survey and 6053ms in the simulation tests). This difference could be due to the uncertain behavior of the pedestrian in simulation as it was more difficult than in the real case to determine the intentions of the pedestrian by his/her movements; in fact, in the real case, the pedestrians were typically determined and focused on crossing the road, and it could be easily detectable by looking at their movements (head, legs, arms) and eyes fixation. Instead, the fixation distance was quite similar in the two samples (89m in the real survey and 95m in the simulation tests); the standard deviation of the two samples was different between the real and the simulation surveys; regarding the fixation durations, the standard deviation of the simulation survey was greater than the real one. In contrast, the real survey's standard deviation was higher than the simulation's regarding fixation distances. In addition, a t-test was carried out in order to check the statistical differences among the samples (Tables 3 and 4). From preliminary analyses of boxplots, 7% of outliers in fixation duration data were found in the real experiment. The fixation duration followed a normal distribution in both samples, as verified by the Shapiro-Wilk test (p > 5%). In this case, the difference between the average values of the fixation duration was found to be statistically significant (t (29) = 7.52; p = 0.00).

Accordingly, the same analysis was carried out for the fixation distances (Table 4). From preliminary analyses of boxplots, 14% of outliers in fixation distance data were found in the real experiments, while 6% of outliers were found in the simulation experiments. Furthermore, this parameter followed a normal distribution in both samples, as verified by the Shapiro-Wilk test (p > 5%). The difference between the average values of the fixation distance was not statistically significant (t (27) = 1.89; p = 0.10). Accordingly, concerning crossing pedestrian analysis, no significant differences in the fixation distance were recorded between the real and simulated experiments. This finding confirmed the validation of the eye-tracking systems for the experiment in virtual reality driving simulation in terms of the distance of the eyes' fixation and also for dynamic objects like pedestrians.

4.3. Speed Analysis

As mentioned before, some previous studies have addressed driving simulator validation in a rural context by comparing speed profiles in real and simulation surveys [15, 16]. However, few efforts have been made to validate the driving simulator in an urban context, specifically while approaching an intersection. in this regard, drivers' speed values have been plotted and studied in two different sections: 50 meters and 10 meters before the “intersection point”. it should be noted that the speed comparison was performed by choosing the two turning manoeuvres (manoeuvre 1 left-turn and manoeuvre 3 right-turn) that could be considered the most critical conditions in the context of an intersection. Fig. (7a and b) clearly show that the distributions of values between the two sections and the two manoeuvres were comparable. the following considerations can be applied: speed trends were approximately the same in the two manoeuvres; specifically, lower speeds were recorded close to the intersection. in both sections, the speed differences between the real and simulation surveys were small (5km/h). This finding demonstrated that risk perception is similar in the real environment and the simulation experiments. In the real survey, the standard deviation was very low (10km/h) in both the manoeuvres and the sections. On the contrary, in the simulation context, these values reached more than 35km/h. This could be due to drivers in the simulated driving environment having varying speed perceptions.

A t-test was carried out in order to check the statistical differences among the samples. From preliminary analyses of boxplots, outliers in speed data related to manoeuver 1 were found only in the real experiment, specifically 4% at 50 meters and 7% at 10 meters. Furthermore, this parameter followed a normal distribution in both samples, as verified by the Shapiro-Wilk test (p > 5%). The difference between the average values of speed data was not statistically significant in both sections (t (60) = 1.04; p = 0.31 at 50 meters; t (53) = 1.81; p = 0.08 at 10 meters), as reported in Table 5. Therefore, this finding confirmed that the simulator is a very efficient tool for investigating drivers’ behavior in terms of speed and also in approaching intersections that involve left-turn manoeuvres [14].

Accordingly, for manoeuvre 3, the same analyses were replicated, and the results were found to be comparable. Table 6 shows that data samples followed a normal distribution, as verified by the Shapiro-Wilk test (p > 5%). Also, in this case, the difference in the average values of speed data was not statistically significant in both the sections (t (29) = 1.66; p = 0.11 at 50 meters; t (26) = 1.31; p = 0.21 at 10 meters), confirming the previous validation obtained for manoeuvre 1.