The machine learning models are presented by establishing the average prediction accuracy that is compared by randomly splitting the data into a training subset (80% of the data) and a testing subset (20% of the data). The prediction errors were examined as well to compare the predictive power of the feature importance by machine learning models. The total error is calculated by dividing the number of mode preferences that are predicted to have the wrong mode choice by the total number of mode preferences. Meanwhile, the mode-specific prediction error is calculated by dividing the number of mode preferences wrongly predicted from the total number of mode preferences made by that particular mode. The analysis was conducted by using Orange software; an open-source data mining tool. This software provides a set of methods and algorithms that can perform effectively in data analysis.

The cross validation that is employed is the five-fold cross-validation technique to evaluate the performance of the classifiers from the training set. The importance of cross-validation is to standardise the repetition of validation and provide better insight after the data set has been reduced according to the data splitting as stated in the previous section. The fold of each iteration would give out the mean of overall accuracy and the standard deviation from each of the iterated train and test data.

The technique applied to measure the importance of features was feature importance by Random Forest. The most common explanations for classification models are feature importance [25]. The term feature importance is used to describe how important the feature was for the classification performance of the model. Random Forest can be used to rank the importance of features in a regression or classification problem in a natural way.

The Random Forest algorithm has built-in feature importance, which can be computed in two ways:

a) Gini importance (or mean decrease impurity), which is computed from the Random Forest structure.

b) Mean Decrease Accuracy is a method of computing the feature importance on permuted out-of-bag (OOB) samples based on the mean decrease in the accuracy.

Different machine learning models were investigated by incorporating the identified features, including Neural Network (NN), Random Forest (RF), Tree, and Support Vector Machine (SVM).

(a) Neural Network

The Neural Network Model is based on the wired net of the human brain that is intended to identify patterns of given datasets [26]. It consists of several layers that stem towards the predicted outcome or responses which are made of multiple nodes interconnected with subsequent layers. The connectivity of layers between input, hidden and output works in a chain form in which the initial features or input are being intensified by the weights commonly in the hidden layer, thereby producing a distinctive input function before submitting to an activation function. Fig. (3) illustrates the structure of Neural Network model.

Fig. (3). A simple diagram of neural network model.

These artificial networks may be used for predictive modeling, adaptive control, and applications where they can be trained via a dataset. Self-learning resulting from experience can occur within networks, which can derive conclusions from a complex and seemingly unrelated set of information. In the present study, the number of hidden neurons was set to 100 per hidden layer in a two-hidden layer configuration. The Rectified Linear Unit (ReLU) activation function was also employed. The solver used in the present study is Adam.

(b) Tree

Decision trees learn how to best split the dataset into smaller and smaller subsets to predict the target value. The condition, or test, is represented as the “leaf” (node) and the possible outcomes as “branches” (edges). This splitting process continues until no further gain can be made or a preset rule is met, e.g. the maximum depth of the tree is reached.

Decision trees normally suffer from the problem of overfitting if it is allowed to grow without any control. A single decision tree is faster in computation. When a dataset with features is taken as input by a decision tree, it will formulate some set of rules for prediction.

(1)

The parameters were set as given:

a) Minimum number of instances in leaves: 1

b) Do not split subsets smaller than: 5

c) Limit the maximal tree depth to: 100

(c) Random Forest

Random Forest is a flexible, easy to use machine learning algorithm that produces, even without hyperparameter tuning, a great result most of the time. Random Forest (RF) constructs many individual decision trees at training. Predictions from all trees are pooled to make the final prediction; the mode of the classes for classification or the mean prediction for regression. However, it is comparatively slower in computation as compared to trees. Random Forest does not use any set of formulas. As they use a collection of results to make a final decision, they are referred to as Ensemble techniques. Fig. (4) depicted the structures of Random Forest in making prediction.

The parameters were set as given:

a) Number of trees: 5,

b) Number of attributes considered at each split: 5,

c) Limit depth of individual trees: 3,

d) Do not split subsets smaller than 5.

(d) Support Vector Machine (SVM)

SVM is an administered learning model with related learning algorithms that break down information for arrangement and relapse examination. SVM is based on the boundaries defined by the largest distance between classes, commonly termed as margin, as visualized in Figs. (5-7). There are four basic concepts to the essence of SVM classification [27]. Firstly, the separating hyperplane. Second, the maximum- margin hyperplane. Third, the soft margin and lastly the kernel function.

Fig. (4). The best split among all features to split the node.

Fig. 5. The hyperplane separates a class and all others.

These hyperplanes can be described by the equations with a normalized or standardized dataset:

With label 1, anything on or above this boundary is of one class and

With label −1, anything on or below this boundary is of the other class,

The enactment of hyperplane to isolate the classes, in this case kernel trick classes, which are being utilized in this study as the SVM algorithm, stipulates the optimal marginal distance or box constraint between these classes. The kernel function employed within a training vector to measure its performance is given in the below equation:

(5)

The parameters were set as given:

a) SVM type with Cost (C): 0.10

b) Regression loss epsilon (ε):0.10

c) Kernel: Sigmoid tanh (gx. y+c)

d) Numerical tolerance: 0.0010

e) Iteration limit: 100

A confusion matrix is also known as an error matrix, a table that allows visualization of the performance of a classification model (or classifier) on a dataset for which the actual/predicted values of each row and column are known. Table 2 summarizes the row and column of the confusion matrices, correctly predicted choices (the diagonal cells), as well as the mischaracterized mode choice.

The confusion matrix reports the number of false positives, false negatives, true positives, and true negatives. In summary, the table yields (a) total prediction error or total error (b) the classification error or mode-specific prediction error, and (c) the percent correctly predicted or known as classification accuracy. The total prediction error is the sum of incorrectly predicted mode choices divided by the total mode choices of all modes. The classification error is the percentage of observed mode choices that are incorrectly predicted and divided by the total mode choices made by that particular mode. Meanwhile, the percent correctly predicted is the sum of correct observations found in the diagonal cells/ total true predictions divided by total mode choice of all modes. The listed metrics are important in evaluating the performance of each algorithm’s predictive power. The confusion matrix, as shown in Table 2, is an example of a binary classifier and it can be extended to the case of more than two classes. The terminology implemented in the confusion matrix can be explained as follows:

• True positives (TP): These are cases in which were predicted as yes (users will choose public transport), and users do choose public transport.

• True negatives (TN): The cases were predicted as no (users choose private vehicles instead), and users do not choose public transport.

• False positives (FP): The cases were predicted as yes, but users do not want to choose public transport. These cases are also known as a “Type I error”.

• False negatives (FN): The cases were predicted as no, but users actually will choose public transport. These cases are also known as a “Type II error”.

(6)

The outcome or target variable is dichotomous. Dichotomous can be explained as there are only two possible classes, fail/pass or labeled “0” and “1”. Logistic Regression predicts the probability of occurrence of a binary event utilizing a logit function. The following basic formula depicts the general Linear Regression equation:

(7)

Where, y is the dependent variable and X₁, X₂,.. and X_n are explanatory variables.

Meanwhile, the Sigmoid Function can be described as following:

Fig. (6). S-shape curve or sigmoidal curve.

Fig. (7). P-value and null hypothesis.

(8)

Logistic Regression is connected with utility theory, where a chosen mode is based on users’ desirability of certain variables. In other words, utility theory bases its beliefs upon individuals’ preferences, and it is widely used among economists to explain users’ behavior based on the ranking of their choices depending upon their preferences. The following equation shows the Logit Model derivation based on its sigmoid function that is applied on linear regression and is widely used in modeling travel mode choice:

(9)

where,

P(Y_i) is the predicted probability that Y is true for case i;

e is a mathematical constant;

B₀ is a constant estimated from the data;

B₁ is a B-coefficient estimated from the data;

X_i is the observed score on variable X for case i.

P-value or known as variables’ significant indicator is very important in statistical analysis.

The null hypothesis (H₀) in statistics explained that there is no relationship between the two variables being studied. In other words, in a hypothesis developed for a case study, a variable does not have any effect on the other. In this case, any result that reflects the null hypothesis means that the results are due to chance and insignificant to support the theory being investigated. Therefore, the null hypothesis presumes that the prediction of research being studied is failed to be proved.

Once the null hypothesis is concluded to be untrue, the alternative hypothesis (H₁) is adopted as the correct theory. The alternative hypothesis supports the theory that an independent variable affects the dependent variable; therefore, the results obtained are considered significant in terms of supporting the theory being investigated. The level of statistical significance is expressed as a p-value between the value of 0 and 1. The evidence is considered stronger with a smaller p-value; thus, the null hypothesis should be rejected.

A p-value less than 0.05 (or written as ≤ 0.05) is considered statistically significant. The value stipulates strong evidence against the null hypothesis. In other words, there is less than a 5% probability that the null hypothesis is correct. Thus, in such a case, researchers should reject the null hypothesis, and accept the alternative hypothesis. Meanwhile, a p-value higher than 0.05 (or written as > 0.05) is considered not statistically significant and stipulates a shred of strong evidence for the null hypothesis to be true. In this case, researchers should accept the null hypothesis and reject the alternative hypothesis.