Empirical Study on the Accuracy and Precision of Automatic Passenger Counting in European Bus Services

2.1. Measurement of Passenger Volumes

Nowadays, most advanced PTCs collect data on passenger volumes by using four methods. These methods are classified in Table 1 according to the possible use of the data collected with and without passenger participation and the adoption of manual or automatic counting strategies. Table 1 is adapted from [5].

On the one hand, passenger volumes may be measured using methods that require the participation of passengers who buy a ticket. They are the so-called ticket dependent methods.

Many worldwide PTCs determine daily passenger volumes by multiplying the Travel Document Sold (TDS) (ticket and passes, of course) by the average the number of potential trips per passenger. Unfortunately, this modus operandi does not provide insights on passenger volumes per route and time periods, when dedicated tickets are not issued. In addition, it is highly problematic to obtain disaggregated space-time passenger volumes from daily TDS data. Moreover, the recent increase in fare media and payment options has reduced the reliability for using the TDS method, even if it is still adopted in many PTCs worldwide.

Automatic Fare Collection (AFC) technologies help calculate space-time passenger volumes by smart cards [6]. Indeed, they record the number of tickets and passes tapped at specific points of the route either off-board (e.g., in turnstiles) or on-board (e.g., in validation ticket machines).

However, all ticket dependent methods have some drawbacks.

First, unlike other transportation systems issuing point-to-point tickets (e.g., airlines, ferries, and railways), these methods do not provide accurate knowledge of passenger volumes. Indeed, in not fully gated transit systems, which do not require passengers to tap-out their tickets before exiting, data on alighting and/or transfer stops are missing and need to be inferred. Moreover, in many worldwide PTCs, pass holders are not required to tap in/out their tickets, thus these passengers may be underestimated. This missing information may be inferred by e.g., Radio-Frequency Identification (RFID) technology to track both the origin and destination of passenger volumes in a passive way e.g., [7]. However, passengers may not have any RFID media.

Second, AFC systems track the access (and/or the exit) to the system, thus they provide atomic data for each passenger. However, as smart cards usually contain personal details, data privacy is a crucial issue to be addressed.

Third, ticket-dependent methods typically underestimate passenger volumes owing to possible fare evaders, who do not buy tickets or have invalid ones [8-10]. Therefore, ticket-dependent methods may not be extremely accurate, if data on passenger volumes are used to rearrange bus service planning and operations.

On the other hand, to overcome these drawbacks, vanguard PTCs also use methods that do not require the passenger’s participation. They are the so-called ticket independent methods.

Due to budget restrictions and lack of technology, data on passenger volumes are usually collected during some time periods by trained checkers. Thus, measurements are performed only at selected checkpoints (e.g., maximum load sections) or for a sample of trips over some hours for few days in a year. When a sample of trips is investigated, it is possible to build a load profile of the route by ride checks [11]. Unfortunately, these activities exhibit a significant level of empiricism and unpredictability because they force PTCs to operate with little, if any, data. Therefore, PTCs are not in an ideal position to revise service planning and operations accurately.

A continuous measurement of passenger volumes over space (at each bus stop) and time (at each period) would improve the characterization of the service and remove shortcomings that derive from pre-selected checkpoints or a sample of trips, and aggregating data over long time periods. This continuous measurement can be performed by APC systems.

2.2. APC System

APC systems are not a new topic for the ITS community and are not emerging technologies. Since the mid-1970s, several USA and Canadian PTCs have implemented APC systems [12-15]. The same applied in Europe, later. A wide range of competing APC systems have been developed. These systems may be classified according to the measurement of passenger volumes that can be indirect or direct.

In the case of an indirect measurement, two main systems exist: weight-based and mobile device-based.

By using the weight-based system, passenger volumes could be estimated by weighing all on board passengers by load sensors on the ground or on the suspensions or on the braking system [16, 17]. For instance, Nielsen et al. [17] analysed a new counting technique that exploits electronic weighing systems to control braking in rail systems. In this case, passenger volumes can be estimated by the total weight of passengers on the train.

By using the mobile device-based system, passenger volumes could be measured by counting their portable devices (e.g., phones and smartphones, tablets, and smartwatches). More precisely, the existing literature includes: (i) large-scale cell phone systems [18], (ii) Smartphone apps-based systems [19, 20] and (iii) Wi-Fi systems [21-23]. Systems (i) help collect data once the device is connected to the cellular network (e.g., a call made or received, a short message sent or received, a connection to the internet). Systems (ii) help track voluntary passengers or estimate the passenger volumes onboard by the participation of passengers. Systems (iii) help collect data once the device has an active Wi-Fi interface, regardless of the connection of the owner to a network.

However, all indirect APC systems present drawbacks limiting their successful application in bus transit services.

Weight-based systems provide the total weight of passengers on board but do not offer data on the flow of passengers. Therefore, if a bus weighs the same before and after a bus stop, this could be either due to no boarding or alighting passengers or the quasi-same number of boarding and alighting passengers. Thus, it is necessary to complete the electronic weighing equipment with a method to gather passenger volume information.

Mobile device-based systems present several pitfalls as well. Children or other people may not have mobile devices, or some passengers may carry on more than one device, thus, passenger volumes may be underestimated or overestimated. Systems (i) and (ii) need the collaboration with telco operators and/or the consent of passengers to estimate passenger volumes. Besides, as for system (ii), passenger volumes severely depend on the willingness of passengers to participate, and, this fact may not be compatible with APC systems that rely on crowd counting principles. Finally, due to the randomization of MAC addresses [24], tracking devices are no longer feasible using system (iii) [25].

In the case of a direct measurement, passenger volumes could be estimated by recognizing people when they board or alight the bus. Three main systems include old mats technologies (pressure sensitive or multiswitch), consolidated infrared technologies (passive or active) and recent video image technologies [13-16, 26].

For instance, multiswitch mats technologies measure passenger volumes by the patterns of footstep changes.

Infrared technologies measure passenger volumes by light beams. When the beams are interrupted, a count is registered. The sequence in which the beams are broken determines the direction of the movement of the passenger. To the authors’ knowledge, infrared technologies are the most adopted in buses and easily found in commerce.

Video image technologies measure passenger volumes by proper cameras in the bus that recognise the passenger. They use several algorithms to a) detect motion, b) estimate its direction, and c) validate the existence of a moving passenger. For instance, Chen et al. [26] used a zenithal camera to capture passenger volumes and the single image is divided in blocks classified according to its motion vector to distinguish between boarding and alighting passengers.

Nevertheless, all “direct” technologies present drawbacks as well. For instance, because mats need to be installed in/on the upper and lower steps of a stairwell, they cannot be adopted in recent low floor bus configurations, as it is not possible to differentiate if a passenger boards or alights the bus. Although one could argue that passengers can be distinguished by specialized doors, this operation is not appreciated by the bus drivers or the passengers. Passengers prefer to board/alight at the most convenient door (i.e., the closest to their waiting point); bus drivers need to accurately approach the bus stop area. A further drawback might be the capital cost due to the need to install at least more than one sensor per doorway.

However, apart from mats, the remaining direct APC systems appear easily applicable to recent transit services, as they can usually be installed on any kind of vehicle. As follows, the authors refer to these systems.

2.3. Accuracy and Precision of APC Systems

Usually, before using APC data in the operational management, two crucial factors are evaluated: the accuracy and precision.

The accuracy of raw count tends to be the focus of vendors and many PTCs. Its analysis evaluates the capability of APC systems to count passengers well. The accuracy measures the systematic over or undercounting of passenger volumes relative to the “ground truth”. Indeed, if the estimation of passenger volumes statistically differs from the ground truth, it is inaccurate. Thus, if the systematic error is known, correction factors need to be calculated to account for the biases.

The precision measures the distribution of errors between the measured and true value of passenger activity. The precision is always evaluated according to a level of confidence. For instance, a PTC would be 90% certain that passenger volumes collected by APC systems fall within ±5% of their true value.

Although accuracy and precision refer to errors in the measurement, the fundamental difference is on the nature of the error. Indeed, accuracy refers to systematic errors, whereas precision refers to random errors. Thus, unlike accuracy, a correction factor cannot be applied for low precision measurements.

Table 2 reports the outcomes of some applied research on direct APC systems largely focused on mats and infrared technologies on buses, because these technologies are among the most adopted in the real operation of buses, according to the authors’ knowledge.

A sample of boarding and alighting passenger data collected at bus stops is considered in order to measure the accuracy and precision. Moreover, unlike [32] and [33] that adopted video surveillance cameras as a comparison method, the evaluation of accuracy was done by comparing, at each stop, APC data with manual data taken by checkers on the bus (manual data are supposed to be error free).

Simple methods evaluated the accuracy by measuring the error rate of counting ¹ for boarding and alighting passengers or the error rate of a range of boarding/alighting conditions on routes exhibiting from 1 to 12 passengers at any given stop by cross-tabulation. More refined methods largely adopted confidence intervals and mean tests for paired data in order to evaluate if systematic errors existed. Other studies refined the accuracy by the Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) to evaluate the magnitude of the error [5].

As for precision, direct and inverse regression models were adopted to test the relative effect of measurement error as suggested by [35, 36].

In general, these studies revealed that APC systems undercounted passenger activity and were more accurate when they recorded boarding rather than alighting passengers. Moreover, even if discrepancies existed between manual counting and APCs, they were often not significant.

Studies [32, 33] showed that the amount of measurement error due to an APC system was statistically significant (at a 95% level of confidence) for boarding and alighting passengers and loads in all samples of buses examined. Moreover, the amount of estimation bias related to load estimates attributed to the APC system is 8.8% using passenger volumes obtained from video cameras as opposed to 26.8% obtained from manual counting.

2.4. Gaps in the Literature

Although all these studies added relevant contributions, some considerations arose.

First, even if it is common to use APC data for many purposes, to the best of the authors’ knowledge, research focusing on on-field evaluation of accuracy and precision of APC systems is largely found for American bus configurations, that differ from European ones. Unlike the American bus configurations, in Europe, buses are usually designed to accommodate many standing instead of seated passengers. Moreover, other differences may be observed owing to the location of sensors, the behaviour of passengers next to doors (e.g., passenger bunching) and so on. Thus, these differences might affect the analysis of accuracy and precision.

Second, even if the accuracy and the magnitude of the counting error were partially evaluated [5, 32, 33] were the only studies that assessed the precision.

Third, vendors often evaluate their products and usually provide data accuracy statistics. However, the accuracy rates provided by vendors tend to be higher than actual rates because they usually work on ideal conditions. Thus, their statistics is usually not based on the real operation of buses, which are affected by situations that are different from the ideal ones. For instance, sensors can be dirty, thus they may result in an inaccurate measurement of passenger activity.

Therefore, it is worth shedding light on the evaluation of accuracy and precision of APC systems also in a European bus market.

3.1. Experimental Setup

The experimentation was performed in the metropolitan area of Cagliari (Sardinia, Italy) having about 0.4 million inhabitants. The main local PTC, namely CTM, manages the public transportation with 271 vehicles (i.e., buses and trolleys) and serves approximately 38.9 million trips a year. Moreover, these vehicles travel over 12,3 million kilometres per year along 32 urban routes [37].

Nowadays, CTM collects passenger data by TDS and manual counting. However, because managers and planners have little trust in manually collected data, CTM is motivated to use APC systems. Moreover, there is a further motivation: their routes are certified according to EN 13816:2002 and are required to express measurements in passenger volumes [38-40].

A commercial infrared technology was tested. This technology uses active infrared sensors with beams in the invisible infrared spectrum. These sensors consist of a transmitter, which projects at least two beams vertically across the bus doorways to a light sensitive receiver. The sensors are installed unobtrusively over the vehicle doors and deliver passenger count data. A local unit mainly records data on boarding and alighting passengers, as well as the dwell and departure times of the bus at each bus stop. Moreover, this APC system includes a GPS module. As a result, because the GPS coordinates are available, the location data are matched to the APC data. At the end of each counting session, data are transmitted by a wireless connection to the data management system. Daily data are stored in a central database.

Evaluation tests were carried out on the most traveled high-frequency route of CTM to account for the worst operational conditions. This route presents the following characteristics:

• Their headways range from 5 minutes to 10 minutes, from 07.00 AM to 8.00 PM.
• The vehicles deployed have one typology.
• It is close to regional government offices, schools, hospitals, and shopping malls.

Due to budget constraints, only one bus was equipped with this technology.

CTM tested this APC system at the end of 2010².

Table 3 reports some characteristics of the bus selected for the experimentation. Table 3 is self-explanatory. Since, the number of counting sensors depends on the number of bus doors, 3 infrared sensors were installed in the vehicle.

Some arrangements were performed before collecting and analysing data.

The way bus drivers and checkers are treated is a relevant issue to consider when discussing the accuracy, because the APC system does not distinguish between bus drivers and passengers. Thus, the bus driver was included in boarding and alighting estimates as well as the checkers.

Second, no counting method helps provide 100% accuracy for any operational conditions, either automatic or manual. However, in this paper, manual counting is supposed to be error-free, even if this hypothesis might be doubtful [15, 32, 33]. Nevertheless, to limit this fact, checkers were accurately trained and one checker per bus door was adopted. More precisely, on-board checkers retrieved manual data as follows: Three checkers functioned as a team on the bus. Each checker monitored and recorded boarding and alighting passengers at each door, respectively. Moreover, the checker was responsible for recording his/her own observations and comments. At the end of each ride, data were merged to present the total boarding and alighting passengers at each bus stop. To summarize, the manual counting represented the ground truth.

A total of 950 stop-level data of boarding and alighting passengers on 24 rides over 3 weekdays were collected.

3.2. Evaluation Methods

Selected variables for the analysis are presented in Table 4. Passenger activity variables generated from manual data include boarding passengers (In) and alighting passengers (Out). Passenger activity variables generated from APC data are boarding passengers (APC_In) and alighting passengers (APC_Out). Finally, variables representing the difference between APC data and manual data are boarding differences (Diff_APC_In) and alighting differences (Diff_APC_Out), respectively.

Unlike [32] and [33], the variable load (i.e., accumulated boarding passengers minus accumulated alighting passengers) was not considered, for two reasons.

First, according to [5], in order to analyse load, proper processing and parsing is needed at vehicle-block and ride levels as well as for the correction of negative loads. Thus, only raw data (i.e., not ‘processed’) are considered.

Second, loads need to be inferred from raw counts. Thus, if errors are detected in raw data, further errors occur in loads, which can be of larger size. Moreover, inaccurate computations of loads may occur due to “drift” effect and may result in a significant overestimation of passenger volumes after many rides, if it is not correctly addressed [4, 5]. However, because raw data are considered, this effect may be neglected. Thus, whether boarding and alighting passenger volumes are accurate, the load estimation might be considered accurate as well.

3.2.1. Evaluation of the Accuracy

To evaluate the accuracy, a hierarchical approach was proposed: (i) the analysis of the presence/absence of the error and its direction; (ii) the analysis of the magnitude of the error, and (iii) the analysis of the nature of the error (i.e., random or systematic).

Table 4.

Variable	Description
In	Boarding passengers (Manual)
APC_In	Boarding passengers (Infrared)
Out	Alighting passengers (Manual)
APC_Out	Alighting passengers (Infrared)
Diff_APC_In	Boarding passenger difference (Infrared – Manual)
Diff_APC_Out	Alighting passenger difference (Infrared – Manual)

First, the difference between manual data and APC data was computed, for both boarding and alighting passengers, respectively. Next, the Total Error on Passenger Counts (TEPC), the Average Error for Counts of Boarding (AECB) and the Average Error for Counts of Alighting (AECA) were computed. Let:

• k be the index of the number of rides;
• j be the index of the number of bus doors;
• i be the index of the number of bus stops;
• r be the number of observed rides;
• d be the number of observed doors;
• n be the number of observed bus stops.

The computations of TEPC, AECB and AECA are as follows:

(1)

(2)

(3)

Next, the error rates were computed for classes of boarding and alighting passengers, respectively.

Second, a measure of the average magnitude of the errors was made in the set of APC data without considering its direction, to understand how large errors are. Thus, the MAE and the RMSE were calculated, for both boarding and alighting passengers, respectively. Let:

• Diff_APC_In_i and Diff_APC_Out_i be the difference in boarding and alighting passengers at bus stop i between data returned by the APC system and manual counting, respectively.

The computations of MAE_b and RMSE_b for boarding passengers and MAE_a and RMSE_a for alighting passengers are as follows:

(4)

(5)

(6)

(7)

Since they are negatively oriented scores, lower values are better. In our experimentation, APC data were taken as the forecast, while manual data were taken as the observed.

Third, according to [32] and [33], the evaluation of the accuracy was refined with mean tests and confidence intervals to understand the nature of the errors. In fact, even if differences were observed between APC data and manual data, one must understand if these differences are statistically significant.

Means tests were performed for the passenger activity variables expressed as differences between APC data and manual data. APC data are assumed to be accurate if the selected confidence intervals for boarding and alighting passengers encompass 0, respectively. The confidence intervals are denoted by Conf_Int_b and Conf_Int_a. respectively.

• Diff_APC_In and Diff_APC_Out be the average value of the difference of boarding passengers and alighting passengers measured automatically and manually, respectively;
• σ_b , σ_a the standard deviation of Diff_APC_In and Diff_APC_Out respectively;
• α be the significance level (i.e., the probability of rejecting the null hypothesis given that it is true);
• z be the z-score at the selected confidence interval.

The confidence intervals were computed as follows:

(8)

(9)

In the case of systematic undercounting or overcounting, the related correction factors for boarding and alighting passengers were computed as follows:

(10)

(11)

3.2.2. Evaluation of the Precision

The analysis of the precision was performed as follows.

First, two linear regression models were applied as in [32] and [33]. In the first model, APC data represented the explanatory variable and manual data represented the response variable. The opposite condition occurs for the second model.

Let:

In and Out be the variables of manual counting for boarding and alighting passengers, respectively.

APC_In, APC_Out be the variables of APC data for boarding and alighting passengers, respectively.

α₁,α₂,α₃,α₄ β_In,β_Out, β_{APC_Out} and γ₁,γ₂,γ₃,γ₄ be the intercepts, the regression coefficients and the stochastic noises of each model.

The regression models were formulated as follows, for boarding and alighting passengers, respectively.

(12)

(13)

(14)

(15)

Second, some t-tests at level of significance α determined whether the measurement error associated with the APC system was significantly different from 0 at a selected level of confidence. The tested variables were calculated for boarding and alighting passengers, respectively. The following formulas were adopted, which were taken from [32] and [33].

(16)

(17)

Third, according to [32] and [33], the amounts of estimation bias attributed to the APC system for boarding and alighting passengers were computed as follows:

(18)

(19)

4.1. APC System Accuracy Results

Values of TEPC, AECB and AECA were computed according to eqns. (1), (2) and (3), respectively. These results are reported in Table 6. Here, one notices that the APC system shows a slight tendency to undercount the total passengers and to undercount both boarding and alighting passengers, respectively. This is because all the percentage differences are negative.

These results contrast [29-33], where APC systems overcount passengers. In addition, the tendency to undercount is stronger for alighting passengers than for boarding ones. Furthermore, these results contrast the hypothesis that passengers board in a more orderly fashion (i.e., one passenger at time) than when they alight [41]. Conversely, our results (i.e., counts are more accurate for alighting passengers than boarding ones) confirms the hypothesis that people tend to rush more when boarding than when alighting a vehicle in a European bus market according to [3].

Our results are partially consistent with [34], that showed that an APC system undercounted boarding passengers and overcounted alighting passengers. However, this last evaluation was performed on rail vehicles, which have a different configuration with respect to buses (i.e., they have wider doors than buses allow simultaneous boarding and alighting movements).

Results of a cross-tabulation analysis of error rates for classes of boarding/alighting passengers are reported in Table 7.

They show that the APC performance achieved is good. Indeed, the APC system returns the truth for about 77% of boarding and alighting passengers, respectively. Even better, these percentages largely increase.

if a variance of ±1 passenger is included: the weighted averages provided 95% accuracy for boarding passengers and 97% for alighting passengers, respectively.

Next, the MAE, RMSE and the related difference were computed according to eqns. from (4) to (7) for both boarding and alighting passengers, respectively. These results are reported in Table 8.

Here, one notices that: (i) low values occurred; (ii) RMSE differs from MAE: therefore, all errors are not of the same magnitude. Moreover, (iii) because RMSE is always greater than MAE, there is some variation in the magnitude of the errors; and (iv) because the difference between RMSE-MAE is not so large, large errors are unlikely to have occurred.

As shown in Table 6, there are differences between APC data and manual data. However, the key question is to understand whether these differences are significant. Indeed, if no significant difference is found between APC data and manual data, only random errors occur, and the APC system is assumed to be accurate (or partially). Therefore, according to eqns. (8) and (9), the 95% Conf_Int_b and Conf_Int_a (p-value <0.05) were computed, respectively. These results are reported in Table 9 .

Here, results indicate that the APC system measures alighting passengers with statistical accuracy; conversely, the APC system systematically undercounts boarding passengers. The result of no statistically significant difference between APC data and manual data for alighting passengers contrasts [31-33], whereas it is consistent with [29] and [30]. This result is also consistent with [32] and [33] when low floor buses are analysed only.

Conversely, the result of a statistically significant difference for boarding passengers contrasts many studies of the American bus market ([29], [30], [31], [32] and [33]). Nevertheless, this result is consistent with [34] that showed that infrared technology significantly undercounts boarding passengers, even if on rail transit.

Moreover, the best accuracy for alighting passengers instead of boarding ones contrasts the hypothesis that standing passengers may obstruct the ride checkers view of the rear door where passengers are expected to alight [28].

According to Table 9, a calibration factor is required for boarding passengers only. This factor was calculated using eqn. (10) and results as 1.0494. Thus, this value should be applied to the boarding passengers collected by the APC system.

4.2. APC System Precision Results

The last analysis presents the results of precisions. They were computed according to eqns. from (12) to (19), respectively. These results are reported in Table 10.

Here, one notices that: (i) according to [32] and [33], the t-tests show that the amount of measurement error due to the APC system is statistically significant for both boarding and alighting passengers; (ii) unlike [32] and [33] for the case of low-floor buses, alighting passengers are shown to have higher measurement errors (i.e., the bias) than boarding passengers; (iii) the highest measurement error regards alighting passengers.

Finally, the relative amount of measurement error-related estimation bias owing to the APC system by using data collected by ride-checkers (i.e., the ground truth) is about 5% for boarding passengers and about 8% for alighting passengers, respectively. These results are consistent with [32] and [33] when low floor buses are analysed only. All studies observed that the amount of estimation biases is significant for both boarding and alighting passengers. However, the present study differs because: (i) manually collected data were compared instead of video surveillance camera data and (ii) the largest bias is reported for alighting passengers instead of boarding passengers.