All published articles of this journal are available on ScienceDirect.
Methodological Aspects of Field Operational Tests of After-Market and Nomadic Driver Support Systems and Impacts on Mobility
Abstract
Background:
This paper reports on the methodology undertaken and some results achieved within a study of drivers using aftermarket and nomadic devices (the TeleFOT project).
Objective:
To evaluate the methodology for conducting Field Operational Tests for Information and Communication Technology whilst also providing an example of the method applied in the context of mobility within the TeleFOT project.
Methods:
A ‘Top down, bottom up’ approach to the derivation of research questions and hypotheses is described. Statistical analysis has been undertaken on data collected through Field Operational Tests and Travel Diaries considering the impact of information functions (such as navigation, traffic information and green driving) upon journey length.
Results:
A summary of the results relating specifically to how the length of a journey can be affected by information functions indicates that Navigation and Traffic information can reduce the length of journeys whilst Green Driving functions tend to increase the journey length.
Conclusion:
The FOT methodology was successfully applied in the TeleFOT project as was the novel method for generating research questions. When turning the theoretical FOT method developed in FESTA into practice, several good innovations were made which can be recommended for future FOTs; collation of metadata, the use of comparable origin / destination pairs for analysis, centralised processing of raw data into legs in order to simplify the analysis of the huge datasets collected in the project.
1. INTRODUCTION
A Field Operational Test (FOT) is a relatively new concept in the field of road transportation and safety research. A definition of an FOT was made in the Field Operational Test Support Action (FESTA) handbook in 2008, now in its fifth revision [1] which defines FOTs as “studies undertaken to evaluate a function or functions under normal operating conditions in environments typically encountered by the host vehicle(s) using quasi-experimental methods”. Therefore, FOT’s are large-scale testing programmes aiming at a comprehensive assessment of the efficiency, quality, robustness and acceptance of new technologies and functions used for smarter, safer and cleaner and more comfortable transport solutions, such as navigation and traffic information, advanced driver assistance - and cooperative systems. Recent examples of completed FOTs include the ISA [2], EuroFOT [3] and Co-gistics [4] projects.
Many such functions (in particular Information and Communication Technologies – ICT) are available on portable devices such as navigators and smart phones and the market penetration of these has been increasing rapidly [5]. Nevertheless, no standards directly related to the use of aftermarket and nomadic devices in vehicles existed and there was little published knowledge about their overall impact on driver behaviour and the user-acceptance. Therefore the TeleFOT (Field Operational Tests of Aftermarket and Nomadic Devices in Vehicles) project was undertaken from 2008 to 2012 in order to understand in more detail how drivers interact with the devices and associated functions [6]. The TeleFOT project involved a number of large-scale pan-European FOTs with the collective aim to assess the impacts of driver support functions provided by nomadic devices on the driving task as well as on the transportation process as whole.
The purpose of this paper is to describe how the FOT method was conducted in the TeleFOT project and provide methodological recommendations for future FOTs. It should be noted that the extent of the data within TeleFOT is vast (involving logged data for over 10 million driven kilometres) and could not be analysed and reported on within one stand-alone study. Also, within the entire TeleFOT project, analysis was conducted for five individual ‘Impact Domains’ including Safety, Efficiency, Mobility, Environment and User-uptake. Therefore, subset analyses of the data were proposed. This paper presents a summary of the main results for the impact of ‘Mobility’. Therefore, the objectives of the current study are described below.
2. OBJECTIVES OF CURRENT STUDY
The main objective of this study was to provide Mobility impact assessment results from the TeleFOT data. Specifically, the objective was to summarize the findings related to individual research questions posed at mobility impact assessment domain, to assess the overall implications of the TeleFOT function on mobility and to draw conclusions from the findings. A second objective was to evaluate the methodology for conducting FOTs for Information and Communication Technology since this had not previously been implemented. A third objective was to generally explore the feasibility of using data from FOTs to address specific research questions since FOTs in general are a relatively new development and their suitability as a research tool is as yet under-explored.
3. METHODS
3.1. Field Operational Tests in TeleFOT
The TeleFOT approach was to compare the impacts that the use of a driver support function had on the driving behaviour compared with a baseline condition during which the function was not available. The FESTA definition of an FOT taking place “under normal operating conditions” was taken so the drivers used the vehicles during their daily routines, data logging took place automatically and the drivers did not receive special instructions about how and where to drive, i.e. the experiment was conducted applying the principles of naturalistic driving with no experimenter in the vehicle. A study period of up to 16 months was employed in order to see the longer term impacts and to allow for a period of functionality learning. Analysis of the logged data covered only the possibility to use the function. Typically the device providing the function also acted as the data logger, thus in order to log data, the device was switched on and in the vehicle. However, logging did not always include information on actual use of the function by the driver, thus the impacts of actual use could not be analysed, only of the possibility to use it.
In-vehicle data-loggers, generally the devices providing the functions, which sampled at 1Hz or greater (offering capability for logging GPS data and 3-axis accelerations) were used for automated data collection. Objective information management and data security were enabled by an automated system for Machine-to-Machine (M2M) communication from the data loggers of all participating test sites to a central database and from the central database to the individual data user (analyst).
3.2. Participants
The core of the TeleFOT research programme was based on conducting FOTs across Europe involving many participants (drivers) using a range of different driver support functions provided by after-market nomadic devices in their own vehicles. The participants in the tests were chosen so that they represented the intended user population. Therefore a description of the user population was an important first step in the recruitment and choice of participants. As a consequence, a targeted rather than a randomised sampling procedure was used in the different FOTs. Recruitment was aimed at participants who were keen to use the new in-vehicle information systems tested in the TeleFOT (SatNav, Green Driving Advisory, Speed Alert etc.) project but had not done so previously. The participants were initially contacted via a series of publicity events at each test-site including advertisements in local newspapers and notifications on public noticeboards (e.g. supermarkets, libraries etc.). Following this initial contact, participants were then recruited according to the following criteria;
• Aged between 25 to 65 years old (to include drivers with significant driver experience but to exclude novice and older drivers)
• Had three years’ driving experience
• Drive more than 10,000 km/year
In total, data were collected on 2,800 driver participants who complete over 10 million kilometres of driving during the project lifetime. In conjunction with the objective data recordings, web-based questionnaires were used to capture subjective and qualitative data. An initial background questionnaire was completed by each participant and then a series of ‘user uptake’ questionnaires, designed to identify the participant’s perceptions before, during and after having the driver support function available, were completed during each FOT. In total, 6,400 background and user uptake questionnaires were collected from the test participants. In addition, data on the mobility patterns of the participants was collected periodically using travel diaries sent to the participants for completion over the course of a week resulting in 1,760 travel diaries during the project’s duration (considered as semi-objective data).
Once recruited to take part in the FOT, the participants were informed about the overall purpose of the project and its organisation as well as about possible risks, the costs covered and not covered, whom to contact in case of system breakdown, etc. A formalised agreement about the arrangement between the involved organisations and the participants themselves was made.
3.3. Large-Scale and Detailed FOTs
The TeleFOT project consisted of both Large Scale FOTs (LFOTs) and Detailed FOTs (DFOTs). The FOTs were organised in three test communities based in Northern (Finland, Sweden), Central (Germany, UK, France) and Southern (Greece, Italy, Spain) Europe (Table 1). The LFOTs constituted the core of the TeleFOT project. The DFOTs were complementary to the LFOTs, providing additional information of a more experimental nature to enrich the analysis and interpretation of the results. The LFOTs were based upon the principle of a naturalistic driving study in the sense that they were studies which involved investigating the normal and everyday use of a set of different driver support functions provided by the platforms of nomadic and aftermarket devices (mobile phone or nomadic display). No control was made over the conditions in which the functions were used or over the way in which the driver should react to any relayed information. The LFOT data was collected over a long period of time from a large number of participants. The data from the FOTs were analysed statistically in order to find out the answers to questions and hypotheses posed.
The DFOTs differed from the LFOTs as they were carried out under experimental conditions. Within the DFOTs, the participants were asked to drive predetermined routes that represented predefined conditions. Furthermore, fewer participants were involved and experimental vehicles were used that were equipped with additional equipment allowing specialised data to be collected.
Across the whole project, twelve LFOTs and eleven DFOTs were carried out in seven European countries, grouped into the three TeleFOT test communities described above. In the LFOTs, the project partners recruited 2,382 test users in total, while data was collected from more than 9 million kilometres driven. A total of 452 participants completed DFOTs and data was collected for more than a million kilometres. The logged km accumulated by participants in each LFOT varied from 109 000 km to 4 155 000 km (Table 1) and driving hours from 3 800 hours to over 72 000 hours. The largest amount of data was from the Italian LFOT with 168 participants and the smallest amount of data was from the Swedish LFOT1 with 54 participants. It should be noted that the number of participants or logged kilometres used in the analysis may be smaller than the total number of participants or total kilometres due to e.g. early drop-outs and exclusions due to incomplete / missing data.
Table 1 shows also the functions under evaluation in each FOT. Due to the nature of the supporting platforms, often bundles, consisting of more than one function, were available to the participant. Thus, some of the analysis results from TeleFOT necessarily refer to the combined effects of multiple functions rather than a unique function.
| FOT | Function(s) | Number of Participants | Total Recorded Kilometres | Test Duration |
|---|---|---|---|---|
| Finland, LFOT | GD, SI/SA, TI | 140 | 343802 kilometres | September 2010 - October 2011 |
| Finland, DFOT2 | eCall from PSAP perspective |
2 | - | ½ hour on May 16 2011 |
| Finland, DFOT3 | Static navigation from usability perspective |
9 | - | 3 weeks, June - October 2009 |
| Finland, DFOT4 | Feedback on speeding for novice drivers |
81 | 593943 kilometres | July 2010 - December 2011 |
| Finland, DFOT5 | Green driving assistance, TeleBUS |
143 | 475000 kilometres | June 2010 - September 2011 |
| Sweden, LFOT1 | SA, GD | 54 | 109177 kilometres | February 2011 - December 2011 |
| Sweden LFOT2 | GD, NAV, TI | 96 | 622244 kilometres | November 2010 - June 2011 (baseline: November 2010- January 2011 & treatment: February-June 2011) |
| Sweden, LFOT3 | SI/SA, NAV | 657 | - | March - August 2009 |
| Sweden, LFOT4 | TI | 554 | 428092 kilometres | September 2010 - June 2011 (baseline: September 2010- December 2010 & treatment: January-June 2011) |
| UK, LFOT | NAV, SI/SA | 80 | 300000 kilometres (approximately) | December 2010 - November 2011 |
| UK, DFOT1 | NAV, SI/SA | 25 | 900 kilometres | December 2010 - November 2011 (following the LFOT conduction) |
| UK, DFOT2 | GD, LDW, FCW | 40 | 4910 kilometres | October 2011 - February 2012 |
| UK, DFOT3 | FCW, TTC, LDW | 23 | 1980 kilometres | January - February 2012 |
| France, LFOT | eCall | 233 | 244750 kilometres (approximately1) | July 2011 - November 2012 |
| Germany, DFOT | NAV, SI/SA, ADAS (ACC, FCW, LKA) |
9 | 11400 kilometres | August 2011 - April 2012 |
| Spain, LFOT1 | NAV, SI/SA | 120 | 837 729 kilometres | September 2010 - September 2011 |
| Spain, LFOT2 | NAV, GD | 97 |
Baseline: 863703.654 kilometres Treatment: 1189101.801 kilometres |
Baseline: January 2011 - May 2011 Treatment: June 2011 - December 2012 |
| NAV, TI | 35 |
Baseline: 526060.137 kilometres Treatment: 670665.483 kilometres |
Baseline: January 2011 - May 2011 Treatment: June 2011 - November 2012 |
|
| Spain, DFOT | NAV, SI/SA | 32 | 450 kilometres | Oct 2011 - Jan 2011 |
| Italy, LFOT | NAV, SI/SA | 168 | 4155133.254 kilometres | September 2010 - December 2011 |
| Italy, DFOT | NAV, GD, TI | 48 | 3800 kilometres | October 2011 - March 2012 |
| Greece, LFOT | NAV, SI, SA, TI | 148 | 806 776 kilometres | February - December 2011 |
| Greece, DFOT | NAV, SI, SA, ADAS (LDW, CAS) |
40 | 1305.6 kilometres | October 2011 - March 2012 |
| TOTAL | NAV, SI, SA, GD, TI, eCall, LDW, FCW (+ in vehicle ADAS from IKA and CERTH vehicles) |
LFOT: 2382 DFOT: 452 Total: 2834 |
More than 10 million kilometres recorded |
GD = Green Driving Advisory
1 Value based on participants’ estimation reported on background questionnaires
SI/SA = Speed Information/Speed Alert
TI = Traffic Information
NAV = Navigation Device
LDW = Lane Departure Warning
FCW = Forward Collision Warning
TTC = Time To Collision
ADAS = Advanced Driver Assistance System
LKA = Lane-Keeping Assist
ACC = Automatic Cruise Control
CAS = Collision Avoidance System
PSAP = Public Sector Answering Point
3.4. Study Design
Study design is an important consideration in relatively long tests such as those performed in TeleFOT. Wherever possible a within subjects design was used in order to reduce the variability between the baseline and experimental periods that could have been introduced by using between subjects. The test participants thus undertook their everyday mobility needs without the availability of the “function functions” (baseline period) and then with the functions available (experimental period). However, as the tests were conducted in a naturalistic manner, it was left up to participant, during the experimental phase, to decide when to actually use the functions that were available.
The data were collected in 3 different phases during the course of the project:
• Pre-test phase: Demographic background data about the participants were collected with questionnaires, e.g. gender, age, driving experience, experience with driving support systems or similar. Travel diary and user uptake questionnaire of the before condition.
• During the tests: The objective GPS data were logged. These data were complemented with additional semi-objective and subjective data. These were collected by means of travel diaries, user uptake questionnaires and individual and focus group interviews.
• Post-test phase: A subjective evaluation of the system by the user using questionnaires.
3.5. Development of Research Hypotheses
A key step-process in the implementation of the TeleFOT methodology (which involves the whole process from data collection through to data analysis and impact assessment) was the approach to selecting the research questions and associated hypotheses that were addressed using data collected by the FOTs. According to FESTA, hypotheses are essentially well defined suppositions in relation to the questions that the FOT is aiming to address at the beginning of the study that can be tested using the NHST (Null Hypothesis Significance Testing) approach. In general;
• The hypotheses state the researcher's expectations concerning the relationship between a number of variables within the FOT.
• The hypotheses are a refinement of the research problem and are specific statements of the problem that the FOT is addressing.
• The hypotheses state what the outcome of the FOT is expected to be.
• The hypotheses are formulated following the review of related literature and prior to the execution of the FOT.
• “A good hypothesis states as clearly and concisely as possible the expected relationship (or difference) between two variables and defines those variables in operational, measurable terms” and “A well-stated and defined hypothesis must be testable. It should be possible to support or not support the hypothesis by collecting and analysing data ” [7].
With these principles in mind, it was essential that the FOTs within TeleFOT were designed with clear hypotheses in order to aid the interpretation of the results. For example, an individual FOT may have been designed to answer the following research question “How does the use of a lateral warning system affect safety?” A resulting hypothesis could be formulated as “Drivers who use a lateral warning system have on average fewer lane deviations than those who do not”. Each hypothesis generated for the TeleFOT analysis dictated the selection of performance indicators to be collected, the data collection requirements and thus influenced the data acquisition systems that the study utilised, particularly where there was the opportunity for more specialised data collection using experimental vehicles in the DFOTs.
Within TeleFOT, a reference was made from the outset to the FESTA handbook for the purpose of selecting and developing research questions and hypotheses within the FOTs. According to FESTA, the practice to be followed was as follows:
Step 1 Formulation of the research hypothesis
Step 2 Testing of the hypothesis
Step 3 Acceptance of or rejection of the hypothesis
Step 4 Replication of the results or (in the case of rejection) refinement of the hypothesis.
Whilst the FESTA principles were useful and valid in the formulation of many of the research hypotheses, in some situations it was necessary within TeleFOT to develop an additional process for the formulation of research questions prior to defining specific hypotheses. This was to ensure that the study as a whole provided robust data that could address a range of issues related to future policy and regulation in respect of nomadic and after-market devices.
For this purpose a ‘top-down and bottom-up’ process was used to derive the final list of research questions and associated hypotheses that were relevant, targeted and could be feasibly addressed within the study.
Initially a top-down approach was required. This approach originated from the definition of the generic impacts to be determined within TeleFOT. These were driven by the theoretical basis used in the main assessment modalities within TeleFOT these being Safety, Mobility, Efficiency, Environment and User Uptake. These were also independent of function, device or use-case.
Secondly a bottom-up approach was required. This approach could only be taken once the functions, devices and use-cases were defined for testing. The detailed research questions were initially developed based on a consideration of four sets of factors namely:
1. Function: A functional description of the system (i.e. what it does) and the behaviour it may cause in a driver, guided by a simple cognitive model of the driving task;
2. Design: The implementation of the system (i.e. how it is designed), and the impact the design attributes have on the driver-system interaction;
3. Use-Case: The use-cases (i.e. the context of use-factors) and their relationship with consequences of use within the real world; and
4. Types of Impact: The types of impact that are being considered.
One of the main characteristics of the process was that it was iterative and took into account an integration of the top-down and bottom-up approaches and then considers the resultant research hypotheses in the context of the capability within the FOT to collect the required data. This is combined with a theoretical analysis of both the costs and benefits that are involved in the data collection process. Therefore it was considered important to conduct an appraisal of how necessary it was to answer the research hypotheses in the context of the costs involved in so-doing.
3.6. Data Management
In the large scale FOTs, all test vehicles were equipped with GPS loggers, generally incorporated within the device providing the function, which collected at least the coordinates, heading and speed. Most of them also collected altitude and number of satellites visible. Some LFOTs collected nomadic device and navigator usage logs such as function activation and traffic information messages received. However, this information was not collected for all FOTs.
For analysis of the LFOTs the FOT data were downloaded from a central server hosted in Finland. Some DFOT data were also hosted on the central server but where this was not possible (e.g. large video data-files) the data were stored locally and accessed according to analysis requirements. For ease of use, the Large-scale FOT data (GPS logger data) was organised into ‘Legs’ - a summary of travel data describing the driving behaviour on single trips. These were synthesised and summarised from the logged data ensuring that the data analyst did not need to process the raw data in order to undertake analysis. This also gave the analysts similar summary data sets for each logger and FOT. Legs with common origins and destinations were identified and many analyses were based on comparable journeys.
The questionnaire data and travel diary data was also hosted on the central server to facilitate the analysis work done by all partners.
3.7. Data Analysis
Within the TeleFOT project the impact of the functions was assessed in four different areas:
• Mobility
• Safety
• Environment
• Efficiency
Additionally, subjective data were used to evaluate user acceptance of the different functions.
Data analysis was made within the framework developed by the research question generation process. For example, for the mobility impact assessment, the research questions produced a framework shown in Fig. (1). Mobility was initially described as the amount of travel undertaken, the travel patterns of the user and the quality of the journey. Within these three areas measurables were defined; in the case of the amount of travel, measurables are the number of journeys, the length of the journey (distance) and the duration of the journey (time). Research Questions addressed the impact that each function had upon each of these measurables.
Each research question was dedicated to a cell in the left most part of the framework illustration. One project partner was responsible for the analysis of the hypotheses relating to a certain research question using the data of all the test sites and functions. The aim was to ensure the congruence of results between test sites and functions.
4. RESULTS
To provide an example of some analysis and results, the investigation of the research question related to length of journey from the mobility framework in Fig. (1) is presented here. The research question was assessed using all three data sources from the LFOTs; logged data, travel diaries and questionnaires. All test sites were included in the analysis. The analysis was based on frequently made journeys. In practice, same origin-destination pairs were selected in baseline and experimental periods for each participant.
In the logged data, a radius of 100 metres was used to determine origins and destinations based on GPS coordinates. All origin-destination pairs that a single test participant had at least once during the baseline period and at least once during the experimental period were included in the analysis. One origin-destination pair per participant contributed one observation. Participants contributed a varying number of these pairs. Mean distance driven between comparable origin-destination pairs were calculated for both the baseline and the experimental periods and compared. A paired-sample t-test was used to test for differences between baseline and experimental periods. The hypotheses tested were:
H0: There is no difference in the distance travelled comparing the base line to the experimental period
H1: There is a difference in the distance travelled comparing the base line to the experimental period
All functions were anticipated as likely to affect travel decisions of personal journeys, and thereby length of journeys. It was expected that distance could be affected especially by green driving support (increase) or navigation (decrease). Table 2 below summarises the results.
| FOT | Functions | N | Distance (km) | Distance Longer in Baseline (%) | Distance ≥10% Longer in Baseline (%) | Distance ≥10% Longer in Treatment (%) | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| Baseline | Treatment | Mean, Baseline | Mean, Treatment | St. dev., Baseline | St. dev., Treatment | |||||
| Finland LFOT | TI | TI | 12 | 13.7 | 13.8 | 7.3 | 8.0 | 41.7 | 8.3 | 16.7 |
| TI, GD, SI/SA | 8 | 30.8 | 29.7 | 31.2 | 31.4 | 37.5 | 25.0 | 12.5 | ||
| TI, SI/SA | TI | 50 | 19.4 | 20.1 | 13.4 | 12.9 | 50.0 | 26.0 | 34.0 | |
| TI, GD | 193 | 22.8 | 25.0 | 31.1 | 31.9 | 48.2 | 21.2 | 25.9 | ||
| TI, SI/SA, GD | 75 | 16.8 | 18.3 | 17.5 | 17.8 | 50.7 | 21.3 | 28.0 | ||
| TI, GD | TI, GD, SI/SA | 85 | 16.6 | 17.9 | 18.5 | 18.7 | 37.6 | 16.5 | 27.1 | |
| Spain LFOT1 | – | NA, SI/SA | 1174 | 15.9 | 15.5 | 15.0 | 14.6 | 51.6 | 12.2 | 10.2 |
| UK LFOT | – | NA, SI/SA | 291 | 13.2 | 13.5 | 19.4 | 20.3 | 48.8 | 24.7 | 21.0 |
| Italy LFOT | – | NA, SI/SA | 3897 | 7.56 | 7.62 | 15.5 | 15.8 | 43.5 | 10.5 | 17.1 |
| Sweden LFOT2 | – | GD, NA, TI | 1165 | 22.4 | 22.8 | 22.9 | 23.2 | 47.1 | 19.6 | 19.0 |
| Greece LFOT1 | – | NA | 414 | 11.7 | 11.9 | 16.2 | 16.4 | 60.6 | 27.8 | 16.7 |
| Greece LFOT2 | NA | NA, SI | 455 | 13.5 | 13.1 | 18.3 | 17.8 | 56.1 | 20.2 | 18.0 |
| Greece LFOT3 | NA | NA, TI | 430 | 13.7 | 15.5 | 18.7 | 37.2 | 66.1 | 30.1 | 16.0 |
| Greece LFOT4 | NA | NA, SA | 543 | 18.5 | 17.6 | 29.9 | 29.2 | 56.4 | 23.0 | 16.7 |
Proportions of origin-destination pairs with differences in distances were also calculated (Table 2). If there is no impact the distance should be longer in 50% of the origin-destination pairs in the treatment phase and 50% in the baseline phase. The results show that the journeys were longer in the treatment phase of the Finnish LFOT and in the baseline phase for the Spanish LFOT1 in line with statistically significant differences found.
The journey distance for the baseline vs the treatment phase were plotted for comparable origin /destination pairs for the FOTS that indicated a statistically significant impact on distance (Fig. (2) Finnish LFOT and Fig. (3) Spanish LFOT1). Regression equations indicate an overall strong association between the journey distance in the baseline and the treatment phases (r=0.87 Finnish LFOT, r=0.96 Spanish LFOT1). However, for both FOTS, most disagreement between the baseline and the treatment phases is observed for shorter journeys: excluding outliers, a higher proportion of data points lie above the regression line in the Finnish LFOT in accordance with the result that journeys are longer when using TI+GD compared to using TI+SI/SA. Considering shorter journeys in the Spanish LFOT1, a higher proportion of data points lie below the regression line, in this case indicating a reduction in journey time when using NA+SI/SA compared to no function.
The impacts in distance were assessed also taking the timing of journey into account. Journeys in logged data were divided into three groups: rush hour journeys (7-9 a.m. and 3-6 p.m.), day time journeys (9 a.m. to 3 p.m.) and night time journeys (6 p.m. to 7 a.m.).
The difference in distance between baseline and treatment means was smaller within any of these time frames than in all journeys for most LFOTs. The statistically significant increase of 2.2 km (9.6%) in distance seen in all journeys of the Finnish LFOT (TI + SI/SA in baseline and TI + GD in treatment) was seen also in rush hour journeys (2.3 km) and day time journeys (4.3 km) but not for night time journeys (Table 3). However, the significant decrease of 0.4 km (2.5%) in distance seen in Spanish LFOT1 was not present in rush hour or day time traffic but it was seen in night time journeys (0.4 km). None of the statistically non-significant differences became significant when only rush hour or day time journeys were studied. However, for night time traffic in Finnish LFOT (TI + SI/SA in baseline and TI + SI/SA + GD in treatment phase) the distance was statistically significantly 1.7 km (9.0%) longer in treatment than during the baseline phase and correspondingly in the Italian LFOT (NA + SI/SA) 0.3 km longer (3.5%) and in the Greek LFOT1 (NA) 1.3 km (9.5%) shorter in the treatment than during the baseline phase.
| FOT | Functions | Rush Hour | Day Time | Night Time | ||||
|---|---|---|---|---|---|---|---|---|
| BL | TR | BL | TR | BL | TR | BL | TR | |
| Finland LFOT | TI, SI/SA | TI | 21.2 | 21.2 | – | – | 21.4 | 19.9 |
| TI, SI/SA | TI, GD | 16.9 | 19.2 | 15.5 | 19.8 | 26.0 | 27.3 | |
| TI, SI/SA | TI, SI/SA, GD | 14.1 | 15.9 | 11.9 | 11.1 | 18.9 | 20.6 | |
| TI, GD | TI, GD, SI/SA | 13.0 | 12.0 | 12.8 | 13.1 | 18.8 | 20.1 | |
| Spain LFOT1 | – | NA, SI/SA | 17.6 | 17.6 | 12.0 | 11.6 | 14.4 | 14.0 |
| UK LFOT | – | NA, SI/SA | 19.0 | 18.8 | 7.0 | 6.6 | 9.4 | 9.6 |
| Italy LFOT | – | NA, SI/SA | 7.0 | 7.0 | 6.0 | 6.1 | 8.6 | 8.9 |
| Sweden LFOT2 |
– | GD, NA, TI | 11.1 | 11.5 | 8.6 | 8.7 | 15.2 | 15.4 |
| Greece LFOT1 | – | NA | 12.1 | 11.8 | 11.4 | 11.3 | 13.7 | 12.4 |
| Greece LFOT2 |
NA | NA, SI | 15.6 | 15.1 | 9.4 | 9.1 | 13.4 | 13.7 |
| Greece LFOT3 |
NA | NA, TI | 20.3 | 19.9 | 14.5 | 14.9 | 16.8 | 15.7 |
| Greece LFOT4 |
NA | NA, SA | 17.0 | 16.5 | 13.4 | 13.1 | 20.1 | 18.8 |
Correlation between logged data (all frequently made journeys) and questionnaire data was determined by comparing the assessed change direction (decrease or increase) in the questionnaire to the change direction in logged data. There were two statistically significant impacts found in the logged data: with function pair “traffic information and green driving” distance was 9.6% longer than for “traffic information and speed information/alert” in Finnish LFOT and distance was 2.5% shorter with the bundle “navigation and speed information/alert” than without any functions in the Spanish LFOT1. However, Finnish LFOT participants assessed no change in distance due to functions (93-96% of participants in traffic information, green driving and speed information/alert post-questionnaires). In the Spanish LFOT1, logger data was in concordance with the questionnaire data. In both of navigation and speed information/alert, the participants also assessed themselves that there was a decrease in distance due to functions.
The connection between logger and questionnaire data by participant groups was calculated for the Spanish LFOT1, where the logger data showed significant differences between treatment and baseline means. The logger data was in concordance with the questionnaire data also on participant group level for navigation. The participants who had replied “decrease” in the questionnaire had also a slightly smaller treatment mean than baseline mean (-0.80 km difference between baseline and treatment mean for SI/SA questionnaire and -0.42 km difference for NA questionnaire) and more participants had a decrease in logged distances (56% for NA). For speed information/alert there were more participants (60%) whose journey distance had increased even though in the questionnaire they reported a decrease.
5. DISCUSSION
Referring to the hypothesis stated above, overall the null hypothesis that there is no difference in distance travelled between comparable origin/destination journeys was supported across all FOTs with two exceptions. It was found that the bundle “navigation and speed information/alert” in the Spanish LFOT1, with logged distance 2.5% shorter than without any functions. In this FOT, logged data were in line with questionnaire data where participants also reported that there was a decrease in distance due to navigation/speed alert.
The other exception was for the Finnish LFOT where it was found that significant differences in length of journey (distance) occurred for the function pair “traffic information and green driving” with logged distance 9.6% longer than for the function pair “traffic information and speed information/alert”. This infers that speed information alert reduced travel distance compared to green-driving since traffic information was present in both conditions. This is perhaps a somewhat surprising result since intuitively there is no expectation that a speed information system would have an impact on distance travelled.
The remainder of the results were not significant. Green driving support (when traffic information and speed information/alert were the baseline, as in the Finnish LFOT) resulted in an increase in distance, whereas the bundle “green driving and speed information/alert” (traffic information as baseline, as in the Finnish LFOT) resulted in a decrease. Although not significant, these results may indicate that a green driving application affects the distance by increasing it. It can be assumed, however, that the speed information system does not impact the distance driven.
The UK and Italian LFOTs with the same bundle, “navigation and speed information/alert” gave contradictory results to those of the Spanish LFOT1. However, the UK and Italian results were also not statistically significant and were very small as absolute numbers (0.1-0.3 km difference). It is worth noting that if a navigator guides the driver to the fastest route, it may not always be the shortest. This also accounts for the increasing impact.
Travel diary results were not in line with the logged data results, as only the Greek LFOT and Swedish LFOT2 showed significant differences in distance. However, it is worth noting that only commuting journeys were analysed from the travel diary data, and these are likely to have more optimised distance during the baseline than would less frequent journeys. There may also be inaccuracies of estimated distance provided by the participants that hide potential small impacts.
What can be learnt from using FOTs as a tool to address specific research questions across various impact domains? With regard to Mobility, which is a focus of the paper, one lesson learned is that in addition to user up-take results, the impacts in driver behaviour should have been studied first (e.g. changes to journey length, changes in number of journeys) and the implications to different impact areas (mobility, efficiency, environment and safety) should have been studied second. There existed the same or very similar hypotheses in several impact areas (e.g. related to speed change which affects safety via accident and consequence risk, mobility and efficiency via travel time, and emissions). When the related driver behaviour analysis was done per impact area, overlapping work was done and occasionally contradicting results were obtained depending on methodological approach taken by each analyst. This can be (at least partly) overcome with cooperation, but it is not optimal use of resources. Therefore, for future FOTs the analysis of overall driver behaviour impacts is recommended before assessing the implications that they cause to different impact areas.
CONCLUSION
The purpose of this paper was to describe how the FOT method was implemented in the TeleFOT project and provide methodological recommendations for future FOTs. In addition, some examples were given on results obtained.
The FOT method applied in TeleFOT was successful in its objectives: to be able to utilise data collected from all of the test sites and to make conclusions based on harmonised analysis work over all test sites, datasets and functions. Objective (GPS), semi-objective (travel diary) and subjective (questionnaire) data was all used. When turning the theoretical FOT method developed in FESTA into practice, several good innovations were made and can be recommended for future FOTs.
The ‘top-down and bottom-up approach’ for research question and hypothesis development was relatively new at the time TeleFOT applied it. It turned out to be successful and, therefore, it is recommended be applied in future FOTs -even if such future studies do not involve tests of devices and functions similar to those tested in TeleFOT. The approach was iterative and took into account an integration of the top-down and bottom-up approaches and then considered the resultant research questions and hypotheses in the context of the capability within the FOT to collect the required data. This was combined with a theoretical analysis of both the costs and benefits that are involved in the data collection process. This combined ‘top-down and bottom-up’ approach of research question and hypothesis development can be recommended for future FOTs.
Centralised processing of raw data into legs was something that simplified the analysis of the huge datasets collected in the project. It ensured that the analysts located in different partner organisations all over Europe did not have to deal with different data formats coming from different loggers. The quantity of raw data would have required special analysis hardware and software but the leg data could be analysed with a regular PC and with any analysis software (e.g. SPSS). Harmonised leg data over all test sites supported the analysis and facilitated comparisons.
The division of work to make one organisation responsible for analysing an individual research question using the data from all of the test sites and considering all of the functions turned out to be successful, too. It ensured that the indicator selection, all methodological decisions and phenomena relating to specific issues were harmonised for all datasets and results could be compared with each other.
A good practice that can be recommended to future FOTs is the use of frequently made journeys (Origin / Destination pairs). When the FOT has been conducted applying the principles of naturalistic driving where the test organisers do not have control over the routes taken or origins and destinations the search for those in the resulting data facilitates the analysis.
Another good practise employed during TeleFOT was the collation of metadata which gave additional information about things that could have an influence on the results outside of the specific analysis variable. For example factors strongly affecting mobility (like changes in the work situation, family, location of home and work) were asked from time to time (here linked with travel diary collection). The more harmonised the mobility needs are during baseline and experimental periods of the study, the easier it is to see the true impacts of the tested functions.
CONSENT FOR PUBLICATION
Not applicable.
CONFLICT OF INTEREST
The authors declare no conflict of interest, financial or otherwise.
ACKNOWLEDGEMENTS
Declared none.
