All published articles of this journal are available on ScienceDirect.
Integral Assessment of the Level of Innovative Development of the Railway Industry Companies
Alexey Beloshitskiy1
Egor Shitov1
Dmitry Arkhipov1
Andrey Tulupov2
Authors Info & Affiliations
Abstract
Background:
NTI Competence Center, in the direction “Technologies of Storage and Analysis of Big Data,” has significant background in the development of integral indexes: a) IQ Cities (to Ministry of Construction of the Russian Federation), b) National Index of digital development of the subjects of the Russian Federation, c) Index of readiness of industries for the introduction of artificial intelligence, d) Index of innovative development for leading domestic companies, including in the sector of air transportation and oil production.
Objective:
Today, innovative, scientific, and technical development is one of the key factors in maintaining the competitiveness of the railway industry in the transport services market. At the same time, innovative development is a complex and dynamically changing process, due to the high rates of development and implementation of new technologies on the level of commercial use in various tech companies. Therefore, it is advisable to use an assessment of the level of innovative development of a company in comparison with the market competitors. Today in the railway industry, there is a need to assess and constantly monitor the level of innovative development of companies. The relevance of building a single tool for assessing the level of innovative development of railway companies is due to the lack of unified and comprehensive methods for calculating such indexes in the industry and the necessary efficiency of innovative railway companies to increase their competitiveness in the transport services market.
Methods:
The main research methods used were bibliometric analysis, qualitative methods of information analysis and quantitative methods for the development of integral indexes (methods for standardizing indicators, methods for assigning specific weights, methods for aggregating indicators). The data for the assessment of the indicators as in 2019 were collected using available sources of information (including official reports and strategic planning documents of railway companies, scientific literature from the Scopus/Web of Science databases) and the opinions of independent industry experts. There are three groups of indicators used to form an integral assessment:
- indicators for assessing the level of readiness of advanced innovative solutions and technologies (indicators are measured on a scale from 0 to 1 using the Technology Readiness Level scale, where 0 is no solution/ no supporting data; 0.5 is a solution at the development stage (TRL1-7), 0.75 is a solution at the approbation stage (TRL8), 1 is a solution at the production stage (TRL9).
- binary indicators for assessing the presence or absence of the use of advanced innovative solutions and technologies (indicators are measured on a scale from 0 to 1, where 0 is no relevant functional (specific characteristic/ capability), 0.5 is possible existence with consideration of indirect factors, 1 - confirmed existence).
- quantitative indicators for assessing the effectiveness and scope of innovation policy and the use of advanced innovative solutions and technologies (evaluated and ranked based on the minimum-maximum method).
To aggregate these groups of indicators at the subindex level, there was calculated a subindex of the average value of the indicators. All the weights of the indicators included in one subindex are the same.
Results:
The scientific result of the research is the development of a unique method (structure of the indicators) for calculating the integral index of innovative development of railway companies based on the analysis of the best domestic and foreign practices and approaches to the formation of integral indexes. The method for calculating the index was successfully tested as a part of the assessment of the level of innovative development of the Russian Railways holding.
Conclusion:
The main result of this work is integral index that has prospects in international practice as a tool for strategic planning of the activities of railway companies to improve efficiency and optimize work processes, as well as a tool for assessing the level of dynamics of the innovative development of the industry and increasing its competitiveness in comparison with other modes of transport.
1. INTRODUCTION
The railway industry is one of the backbone elements of the economic systems of most countries, while, both in passenger and freight traffic, it experiences significant competition from other modes of transport. In these conditions, the use of innovative and technological solutions aimed at increasing efficiency and optimization becomes the main factor in the development of the competitiveness of individual railway companies and the industry.
Considering the growing competition in the transport sector, the innovation policy of the public sector and the management of railway companies has to be based on assessment of the level of their innovative development by identifying the advantages and bottlenecks in technological development of companies relative to the main counterpart companies. Current market landscape demands a high-quality system for assessing the innovative development of railway companies. Such system allows to perform an effective comparative analysis and assessment of railway companies, but also companies providing transport services by other modes of transport. One of the main tools that make possible carrying out such an assessment and analysis, considering multidimensional indicators and parameters, are integral indexes.
2. MATERIALS AND METHODS
Integral indexes include coefficients that allow taking into account various indicators within the framework of a single calculation scale, differing both in units of measurement and in the nature of the assessment (quantitative and qualitative indicators). Their use makes it possible to significantly simplify the analysis of multidimensional and complex domains in cases where research objects are described by a wide range of different indicators, making it difficult to identify the differences between objects based on them [1-7]. For example, the article [3] describes ways to assess the complex environment “emerging technologies governments, and human preference work together to create a dynamic system loop” using an integral index. According to the study [5], “the complex index of air pollution and technological complex” can assess complex environmental systems with high accuracy, which is impossible by assessing individual indicators. The integral index combines several assessment parameters, i.e., indicators, ranging based on predefined weights.
Indicators are subdivided into several groups, the number of which is determined by the thematic focus of the integral index: high-level indicators (subindexes), reflecting the main-blocks of assessment; intermediate level indicators (distinguished in case of allocation of additional thematic assessment blocks in the structure of subindexes); lower-level indicators (distinguished if there are additional thematic assessment blocks in the structure of intermediate-level indicators).
The assessment of the indicators of the lower level (the lower level is determined depending on the component saturation of the assessed direction) is based on the values of specially calculated indicators and the degree of their influence on the indicators, determined in the form of specific weights. Schematically, the described system of forming the integral index is shown in the Fig. (1) below.
It is important to note that the terminological difference between the concepts presented in the framework of this structure, i.e., sub-index and indicator, indicator is quite relative and can be clearly traced only in cases of the multi-level integral indexes. The direct difference between a sub-index and an indicator from an indicator is that the latter is a normalized estimate of the real parameter of activity (of the country, city, company), while subindexes and indicators are an aggregated assessment of indicators.
The indicators presented within the framework of the index structure, as a rule, differ significantly from each other at the level of units of measurement and dimension, which significantly complicates the calculations and the formation of the result. To overcome this limitation while developing integral indexes based on a wide range of different indicators, a standardization mechanism is used, which is an adjustment of indicators in accordance with pre-selected conversion methods (Appendix A1, Table 1), allowing the indicators to be formed in a more convenient way of calculation and comparison.
A great variety of rationing methods defined by both advantages and specific individual disadvantages are used in the world practice. Shortcomings do not refute the practical applicability of these methods; they are only an indication of existing methodological “bottlenecks” that must be considered when choosing the specific method and analyzing the result of the application.
When calculating the integral index, each sub-index, index, and indicator has a representation of specific weights according to the methods of their assignment (Appendix A2, Table 2). The specific weights reflect the conditional degree and significance of the indicators for the formation of the final evaluation.
After the selection of the coefficients of the specific weights, the transition to the final stage of the formation of the index, the aggregation of indicators, is carried out. The main methods used for the aggregation are shown in Appendix A3, Table 3.
In order to identify the widest array of practical usage of integral indexes, including the areas of evaluation and used indicators, a review of research papers on the topic of assessing the level of development of the railway industry was carried out. The selection of relevant scientific articles and publications was made using databases, such as Scopus, Web of Science and eLibrary. Based on the results of the analysis of the research experience of using integral indexes for the assessment of the transport industry development, the key approaches used to form such indexes were identified (Appendix B1, Table 1).
The choice of the methodology for calculating the integral index determines the validity of the result and the reliability of the subsequent comparative analysis. The listed methods are the most commonly used for formulation of integral indexes for assessing various subject areas, including the level of development of transport and logistics and railway companies. The researcher examines existing and commonly used indexes that are applied to assess the urban transport systems or segments of the transport industry; they are described in the following list [8-12].
The carried-out analysis confirms the fact that the indexes that are actively used in world practice to assess the level of development of the transport sector and innovations, as a rule, are not sufficient to conduct a full analysis of the level of innovative development of transport companies, primarily railway operators. The greatest value in the context of developing an index of innovative development of railway companies is represented by indexes that assess the level of innovative development at the level of companies in the transport industry, since the assessment parameters used to calculate such indexes are largely comparable to the key indicators of innovativeness in the railway industry.
| Innovation Policy (Specific Weight - 9/35) | ||||||||
|---|---|---|---|---|---|---|---|---|
 Digital transformation strategy |
R&D center |
Availability of specialized innovation laboratories |
Availability of a business accelerator / business incubator |
Availability of a venture fund |
Organization of innovation-oriented events |
Intra-corporate entrepreneurship |
Number of articles by company employees published in journals indexed by the bibliographic and abstract database Scopus |
Number of company patents |
| Advanced Product and Technological Solutions, Technical Means (Specific Gravity - 26/35) | ||||||||
 Intelligent control systems for rolling stock (automation level GoA3 and higher) |
Hybrid traction rolling stock |
Hydrogen fuel cell powered rolling stock |
Natural gas rolling stock |
Biofuel rolling stock |
Drones |
«Machine vision» |
MaaS platforms («mobility as a service») |
Single digital «end-to-end» shipping document |
|
Robotic interaction systems with clients |
Intelligent indoor navigation systems for visitors to station complexes |
Comprehensive tracking systems and management of the movement of goods «from door to door» in real time |
Smart contracts |
Intelligent chat bots |
Platforms for organizing multimodal transportation |
Document management systems based on distributed ledger systems |
Intelligent systems for maintenance and repair of rolling stock |
Intelligent systems for maintenance and repair of infrastructure facilities |
|
«Digital twins» of rolling stock |
«Digital twin» of the infrastructure complex |
Additive technologies to produce spare parts and components for the repair of rolling stock and infrastructure facilities |
FRMCS railway radio communication standard based on 5G technology |
 Global ranking of company websites by SimilarWeb's |
Average rating of mobile applications for passenger transportation on Google Play and App Store |
Average rating of mobile applications for cargo transportation on Google Play and App Store |
Maximum operating speed of the operated rolling stock, km / h |
|
– indicators for assessing the level of readiness of advanced innovative solutions and technologies. – binary indicators for assessing the presence or absence of the use of advanced innovative solutions and technologies. – quantitative indicators for assessing the effectiveness and scope of innovation policy and the use of advanced innovative solutions and technologies.
1 θ - specific weight
2 Indicator 1 - intermediate level indicator
3 Indicator 2 - low level indicator
| Strengths | Weaknesses |
|---|---|
| - Comprehensive assessment of railway companies’ innovation activities. - The use of heterogeneous indicators (quantitative, binary, evaluation of innovative solutions). - Ability to modify the composition of indicators without changing the logic of the integral index construction. |
- There may be data misrepresentation due to the lack of complete and reliable information about the technical aspects of applying innovative solutions of some railway companies. |
| Strengths | Weaknesses |
| - Obtaining official data on TRL evaluation of innovative solutions from railway companies (besides Russian Railways holding). - Establishing the index as an official index in the UIC. |
- An innovative development paradigm shift. - Prohibition of data exchange between railway companies. |
In world practice, the indexes of innovative development of transport systems are actively used, including the BCG digital acceleration index [13], the digital transformation index [14], the airline digital development index [15], the urban innovative mobility index [16]. The listed indexes are characterized by a wide range of indicators that are relevant, among other things, for assessing the level of digital maturity of railway operators, actively developing digital services for customers, and optimizing internal management systems using advanced digital solutions. The most relevant is the urban innovative mobility index, which assesses the level of innovative development of the city's transport infrastructure, which, considering the trend towards the integration of the railway network into the urban transport system, determines the relevance of several parameters included in its structure for use in calculating the integral index of innovation development of railway companies.
Furthermore, in the Russian Federation, index of the development of the transport complex, which is compiled annually by the Lomonosov Moscow State University, is actively used, reflecting the results of comparison of the leading Russian and foreign cities in terms of the quality of transport services, their accessibility to the population, road safety, and the impact of transport on the environment, efficiency of freight logistics and other indicators [17]. It is designed to identify the strengths and weaknesses of the transport complex of large cities and determine the optimal ways for its further development.
Based on the results of the analysis of the practical experience of using integral indexes to assess the level of development of the transport industry and the readiness of innovations at the company level, the key approaches used to form such indexes were identified (Appendix B2, Table 2) [18-22].
Among the approaches analyzed in open sources, no mechanisms were identified that directly assess the level of innovative development of railway companies; either the overall efficiency of the railway industry at the level of individual countries is assessed, or innovative development/digital transformation without reference to a specific industry or related industries. According to the results of the study, it can be concluded that currently in the railway industry, there are no generally accepted methods for assessing the level of innovative development of railway companies, which necessitates the development of such a method.
Based on the results of the analysis of existing approaches to the formation of integral indexes, including integral indexes of the general and innovative development of companies in the transport sector and the railway transportation sector, the following conclusions were drawn:
(1) In the international practice of analyzing the innovative component of the transport sector, as well as at the level of thematic scientific works, the fact of using integral indexes, the object of which would be the level of technological, innovative, and digital development of railway companies, was not revealed. The existing indexes allow us to assess only the general level of innovative development of the transport system or the level of readiness of digital technologies of companies without specifying the industry. As a result, in the conditions of accelerating innovation cycles and active digitalization of the railway industry, the importance of a mechanism that makes it possible to assess the level of innovative development of railway companies on a regular basis, considering the main directions of technological transformations, increases.
(2) Selected methods for constructing these indexes, as well as their individual indicators, can be used to develop an integral index of the innovative development of railway companies. The methodology for constructing the considered indexes is based on a three-level structure, which is most appropriate for grouping the parameters used, since it allows to comprehensively cover all the main areas of activity while maintaining the relative simplicity of formation. The indicators that allow assessing the level of digital development of companies, in the use of specialized digital services and information and communication systems, are of the greatest importance for the formation of the index of innovative development of railway companies.
(3) The most common standardization method for calculating integral indexes of the overall development of transport systems is the minimum-maximum method (in 4 out of 10 analyzed indexes), which is explained by the relative ease of use with a sufficiently large number of heterogeneous variables. To assign specific weights, the method of equal weights is predominantly used, while in the case of full or partial base of the index on the results of surveys, the method of expert assessments is used. For aggregation, as a rule, a linear method is used due to the relative ease of implementation.
To form a unified toolkit for assessing the level of innovative development of railway companies, an integral index of innovative development of railway companies (hereinafter referred to as the Index) was developed, which is a comprehensive assessment of the achieved results of railway companies in the field of innovative development based on quantitatively and qualitatively measured indicators in the results of the reporting period. The index makes it possible to assess and monitor the level of innovative development of the Russian Railways holding. The methodology for calculating the Index was calculated considering the best practices in the field of constructing similar indexes.
As part of the approbation of the methodology for calculating the Index, 20 different approaches to building models were tested, which differ in the structure of grouping indicators at the level of subindexes, the method of standardizing quantitative indicators, and directly used indicators. The following were considered as the main methods for standardizing quantitative indicators:
- Minimum-maximum method;
- A method based on a categorical scale (based on thresholds).
The choice of these methods was due to their prevalence for calculating various integral indexes, including in the railway industry. In accordance with the standardization methods used, the models were subdivided into 2 main groups, within which further iterations were performed. Based on the results of the approbation, it was decided to apply the final version of the methods for calculating the Index of the minimum-maximum method, which, in contrast to standardization based on threshold values, allows the principle of flexibility of the developed methodology to be ensured. The fact that, if necessary, when analyzing indicators, considering changes in the scientific and technological structure, cycles and the emergence of new technological trends, the calculation methodology can be changed by adjusting the composition of indicators (excluding irrelevant indicators and/or adding new relevant indicators).
The initial concept envisaged the creation of a three-tier structure, including subindexes of the 1st level, subindexes of the 2nd level and indicators. Level 1 subindexes were based on the principles of grouping indicators characterizing the areas of activity of companies, including:
- Grouped indicators reflecting the innovative component in the field of cargo transportation;
- Grouped indicators reflecting the innovative component in the field of passenger transportation;
- Grouped cross-cutting indicators, the assessment of which can be sent without considering the activities of the analyzed companies.
Within the designated indexes of the type, there are subindexes of the 2nd level, grouping indicators according to a functional characteristic. In the future, to avoid the prevention of assessments due to the imbalance of performance indicators, it was decided to switch to the use of a two-tier structure, where the grouping of indicators at the level of subindexes is based on the principle of separation according to a functional criterion. Thus, the final structure ensures the use of the following subindexes: innovation policy; advanced product and technological solutions, technical means.
In the process of approbation, the composition of the indicators has also changed. The parameters related to the regional characteristics of innovative development were determined. The main reason for excluding several indicators from the final version of the methodology for calculating the Index is the lack of reliable data for their calculation in open sources for the most assessed railway companies. At the same time, within the framework of approbation, the indicators are included in the calculation of the Index for a limited sample of companies to verify the results; according to the results of the verification, no bias in the results of the index assessment was found.
The overall structure of the Index consists of indicators for assessing the level of readiness of advanced innovative solutions and technologies, binary indicators for assessing the presence or absence of the use of advanced innovative solutions and technologies, the number of indicators for assessing the effectiveness and scope of innovation policy and the use of advanced innovative solutions and technologies. In the structure of the Index, there are two basic averaged estimates of indicators, grouped according to a certain criterion in the general structure of the integral index (hereinafter referred to as subindexes, Table 1):
- Innovation policy, which includes indicators reflecting the level of organization of innovative activities within the framework of railway companies (9 indicators, of which 2 are quantitative, 7 are binary, the subindex weight in the total evaluation of the integral index is 9/35);
- Advanced products and technological solutions, technical means, which include indicators reflecting the results achieved in the field of innovative development of railway companies (26 indicators, of which 4 are quantitative, 22 indicators for assessing the level of technology maturity according to the Technology Readiness Scale, the subindex weight in the total evaluation of the integral index is 26/35).
The indicators specified in the sub-indices were selected on the basis of an analysis of strategic planning documents of railway companies and industry documents in the field of innovative development. The composition of the indicators reflects the priority criteria or ways of innovative development of most railway companies in the world until 2030.
The allocation of the listed areas of assessment is of a formal nature and is aimed at simplifying the perception of the general structure of the Index, reflected in the calculation model. Indicators for assessing the level of readiness of advanced innovative solutions and technologies among railway companies are assessed on a scale from 0 to 1, where 0 means no relevant solutions (or no information about solutions), 0,5 - solutions at the development stage, 0,75 - solutions at the stage approbation, 1 - solutions at the stage of replication. Likewise, binary indicators (both subindexes and indicators) are evaluated on a scale from 0 to 1 where 0 means the absence of the corresponding functionality (specific characteristics, capabilities), 0,5 means possible presence, considering indirect factors, 1 means confirmed presence. The assessment of the maturity level of solutions and/or technical means according to the Technology Readiness Level scale and binary indicators of the analyzed company and similar companies is assessed based on information provided in open sources of information: reports of railway companies, technical documentation, etc.
Quantitative indicators are brought into a comparable form due to the transition from absolute values to normalized values (from 0 to 1 or from 1 to 100) using the minimum-maximum method. The formula for converting an indicator into conditional points from 1 to 100, if a high value of this indicator is a good result (for example, revenue):
 |
- the normalized value of the k-th quantitative indicator of the i-th direction of innovative development1 of the j-th railway company;
1The directions of innovative development mean the conditional grouping of various indicators based on several common features. The grouping is of a formal nature and does not affect the formation of the normalized assessment, as well as the assessments of the sub-index and the final assessment.
- the current value of the k-th quantitative indicator of the i-th direction of innovative development of the j-th railway company;
- the minimum value of the k-th quantitative indicator of the i-th direction of innovative development across the entire sample of railway companies;
- the maximum value of the k-th quantitative indicator of the i-th direction of innovative development across the entire sample of railway companies.
The formula for converting an indicator to conditional points from 1 to 100, if a high value of this indicator is considered a bad result (for example, CO2 emissions):
 |
- normalized value of the k-th quantitative indicator of the i-th direction of innovative development of the j-th railway company;
- the current value of the k-th quantitative indicator of the i-th direction of innovative development of the j-th railway company;
- the minimum value of the k-th quantitative indicator of the i-th direction of innovative development across the entire sample of railway companies;
- the maximum value of the k-th quantitative indicator of the i-th direction of innovative development across the entire sample of railway companies.
To avoid distortions of the final estimate if a company provides services exclusively in one segment, quantitative indicators related to another segment at the level of this company are assigned a «zero» value (due to the methods of standardizing indicators, these values are excluded from the calculation of the index from a certain company). If data on quantitative indicators for an individual company are not available in the available information sources, while the parameters estimated using these indicators are potentially present in this company, such indicators are also assigned a «zero» value [23-25].
The final score of the Index for each of the railway companies is formed on a scale from 0 to 1 based on the results of aggregating subindexes using specific weights calculated as the ratio of the number of indicators included in the sub-index to the total number of indicators in the index structure. To aggregate the above groups of indicators (indicators for assessing the level of readiness of decisions, binary indicators, quantitative indicators) at the level of subindexes, the average value of the indicators included in a certain sub-index is calculated. To aggregate subindexes when calculating the final value of the integral index, a linear method is used, which consists of summing the normalized estimates of the variables, considering the predetermined specific weights:
 |
xi - value of the i-th subindex;
0i- share of the i-th sub-index.
The specific weights within the framework of the applied methodology for calculating the index are the values calculated based on the number of indicators included in each sub-index by the formulas:
 |
0i – specific weight of the i-th sub-index;
nxi – number of indicators in the structure of the i-th subindex;
N – the total number of indicators in the index structure.
For the assessment within the framework of the Index, 19 railway companies were selected that are close in the type of activity to the Russian Railways holding (comparability of the operating business model) and with a comparable or larger scale of activity compared to the Russian Railways holding. As a result of the assessment of the Index (based on information for 2020), a matrix was formed showing the values of the index for each of the railway companies (Fig. 2). Depending on the points received, railway companies can be further divided into the following 3 groups in terms of the level of innovative development: «leaders» (above 0,66 points); «Stable» (from 0,33 to 0,66 points); «Lagging behind» (below 0,33 points).
3. RESULTS
According to the results of the Index calculation, the companies Deutsche Bahn (Germany), SNCF (France), China Railway (China), Russian Railways holding (Russia), SBB CFF FFS (Switzerland), Japan Railways (Japan) are recognized as the leaders of innovative development. The mentioned companies have the largest number of innovative solutions which are currently used. For example, Deutsche Bahn и SNCF actively developed «digital twins» of rolling stock and infrastructure complex, drones, MaaS platforms, smart contracts, intelligent chat bots, platforms for organizing multimodal transportation. China Railway is a world leader in high-speed railways development and implementing of digital systems for railway process management. Russian Railways holding has one of the most developed intelligent control systems for rolling stock and for maintenance and repair of rolling stock and infrastructure complex, natural gas rolling stock. The leaders mentioned above are ahead of other companies by quantitative and binary indicators which are included in the subindexes. This is largely due to a similar companies’ business model and developed systems of internal corporate management focused on implementation of innovations.
Assessment of the level of innovative development of railway companies on an annual basis will make it possible to assess the position of railway companies in dynamics and to respond more quickly to new trends in scientific and technological development.
4. DISCUSSION
The conducted approbation proved the high level of the Index's applicability for the regular assessment of the level of innovative development of railway companies. The method for calculating the Index was successfully tested as a part of the assessment of the level of innovative development of the Russian Railways holding. The Index was verified by members of the Management Board of JSC Russian Railways. Table 2 shows the SWOT-analysis of the developed integral index.
SI1 - sub-index 1 «Innovation policy»
SI2 - subindex 2 «Advanced product and technological solutions, technical means»
Broad prospects for using and increasing the reliability of the Index calculations can be ensured by verifying the developed methodology in the International Union of Railways. The annual collection of official data from railway companies, including in terms of assessing the maturity of the technologies used, and the analysis of proposals for improving the index will eliminate the existing shortcomings of the index in terms of the use of statistical data on binary indicators and indicators for assessing the level of readiness of advanced innovative solutions and technologies in railway companies based on expert assessments, which will increase the accuracy of the Index calculation and the quality of the findings.
In addition, the international recognition of the Index will make it possible to expand the range of indicators used, for example, in terms of indicators, such as «Share of R&D costs (attributed to revenue), %», «Accuracy of adherence to passenger train timetables, %», «Share of freight shipments delivered in compliance with the established delivery time, %». These indicators were included in the calculation of the Index in one of its editions but were excluded due to the availability of data in open sources of information not for all railway companies.
CONCLUSION
The use of integral indices is an effective way to assess the complex innovation environment of railway companies. The index can be actively used in the practice of railway companies for annual monitoring of the effectiveness of innovation activities and identification of strengths and weaknesses for their further adjustment by strategic development programs. It will be possible to update the indicators of the developed integral index, taking into account changes in plans of innovative development of railway companies in the future. The developed methodological approaches can also be used to create other integral indices by railway companies to assess various aspects of their operation activity.
CONSENT FOR PUBLICATION
Not applicable.
AVAILABILITY OF DATA AND MATERIALS
Some or all data, models, or codes generated or used during the study are proprietary or confidential in nature and may only be provided with restrictions.
FUNDING
The work has been supported by the NTI Competence Center in the direction “Technologies of Storage and Analysis of Big Data”, Moscow State University named after M.V. Lomonosov, Moscow, Russia.
CONFLICT OF INTEREST
The authors declare no conflict of interest, financial or otherwise.
ACKNOWLEDGEMENTS
The authors are grateful to Trostyanskiy Sergey, associate Director of NTI Competence Center in the direction “Technologies of Storage and Analysis of Big Data”, Shitova Yulia, Deputy Head of Consulting and Expertise of NTI Competence Center in the direction “Technologies of Storage and Analysis of Big Data”, Rakov Dmitry, Deputy Head of Consulting and Expertise of NTI Competence Center in the direction “Technologies of Storage and Analysis of Big Data” for their very helpful suggestions.
APPENDICES
| Method Name | Description | Advantages | Disadvantages |
|---|---|---|---|
| Ranking method | Indicators are ranked according to their level of significance in descending order, by assigning numerical equivalents from 1 to n. | - Resistant to «outliers» in the data - Relative ease of implementation |
- The impossibility of assessing indicators in absolute terms |
| The classical method of valuation | It is calculated as the ratio of the value of an individual indicator to the average or maximum value of this indicator for the entire sample (for example, a group of companies), or, conversely, as the ratio of the average / maximum value of the indicator for the entire sample to a separate indicator. Calculated by the formulas:
|
- The spread of the values of indicators remains, therefore, the nature of differences for individual indicators is correctly reflected | - Significant differences in the values of indicators can strongly affect the final score of the index |
| Minimum-maximum method | Indicators are normalized according to 2 basic formulas (for example, a conditional company): 1. The formula for converting an indicator into conditional points from 1 to 100, if a high value of this indicator is considered a good result (for example, revenue):  2. The formula for converting an indicator to conditional points from 1 to 100, if a high value of this indicator is considered a bad result (for example, CO2 emissions): 
|
- Allows you to expand the range of values of indicators lying within a small interval | - Weak resistance to extreme values and «outliers» in the data |
| Standardization method (z-score) | Reduces indicators to a single scale, where: - average value = 0; - standard deviation = 1. Calculated by the formula:  где: xi – i-th indicator;  – arithmetic mean of indicators;  – standard deviation of indicators.When using this method, indicators with extreme values have a greater impact on the indicator formed on their basis. |
- Allows comparison of previously incomparable variables | - The presence of negative values - Not applicable when extremely good scores are better on multiple indicators than multiple average scores |
| Distance to reference method | Measures the position of a metric in relation to a predetermined reference value, for example, a target metric value that must be achieved in the analyzed time. | - Relative ease of implementation | - Weak resistance to extreme values and «outliers» in the data |
| Categorical scale method | Allows you to assign a value to each indicator in accordance with the specified categories, which can be both quantitative and qualitative. This method can also consist in assigning scores in accordance with getting into one or another percentile for an indicator. For example, the 10% of companies with the best performers receive 100 points, the next 10% receive 90 points, and so on. up to 0 points. | - The slightest changes in the real values of the indicator do not affect its normalized value | - Does not allow to take into account a large amount of information regarding the variance of the converted indicators |
| Method Name | Description | Advantages | Disadvantages |
|---|---|---|---|
| Equal weighting method | All variables in the index are assigned equal weights based on their quantity. | - Simplicity of formation | - Differences in the degree of influence of individual variables are leveled |
| Assigning Weights Based on Expert Assessment | Weights are assigned based on the positions of specially selected experts or according to the results of surveys among certain social groups | - Relative ease of formation | - Subjective nature of the assessment |
| Assigning Weights Based on Statistical Models | The reliability of the data for the variables is verified by statistical analysis. Higher weights are assigned to the most reliable data. | - Relatively reasonable nature of the assessment | - Relative complexity of implementation |
| Method Name | Description | Advantages | Disadvantages |
|---|---|---|---|
| Aggregation using the sum of weighted averages (linear method) | It consists of the summation of the normalized estimates of the variables, considering the given specific weights. Calculated by the formula:
|
- Ease of implementation | - Low objectivity of the method due to the ease of assessment |
| Aggregation using the product of weighted geometric group indicators | Calculated by the formula:
|
- Sufficiently high objectivity of the assessment since the specific gravity coefficients within the framework of this method are determined by calculation | - Relative complexity of implementation |
| Distance-to-Reference Aggregation | The index is calculated taking into account the distance at a certain benchmark value, for example, in the case of comparing companies to a high value of the integral index of a similar company. Calculated by the formula:
|
- Formalized calculation method | - Relative complexity of implementation |
| Multi-criteria aggregation | They are used when very different parameters are aggregated in the index. For example, production, financial and social indicators. | - Accounting for differences between parameters | - Complexity of calculation - Unlike other methods, this method is not characterized by the mutual compensability of the indicators included in it; the influence of indicators whose values of many indicators are slightly better will have a greater impact on the final assessment than indicators whose values of several indicators are much better. |
| Index Name | Organization | Analysis Object | Subject of Analysis | Data Collection Method | Standardization Method | The Method of Assigning Specific Weights | Aggregation Method |
|---|---|---|---|---|---|---|---|
| Indicators for benchmarking and monitoring transport infrastructure | Lulea University of Technology (Sweden) | Transport infrastructure | General level of productivity of railways | Collecting data from open sources | Minimum-maximum method | - Assigning equal weights - Correlation analysis - Expert assessments |
- Additive |
| Railway infrastructure and rolling stock reliability index | University of Tromso (Norway) | Railway infrastructure and rolling stock | The level of operability of railway infrastructure and rolling stock | Collecting data from open sources | n/a | N/a | n/a |
| Subway safety assessment index | Beijing Transport University, China Railway Research Institute Group of Companies, Chinese Academy of Sciences. | Subway infrastructure | Security level of stations and subway lines | Collecting data from open sources | n/a | Statistical | Cloud |
| Index Name | Organization | Analysis Object | Subject of Analysis | Data Collection Method | Standardization Method | The Method of Assigning Specific Weights | Aggregation Method |
|---|---|---|---|---|---|---|---|
| General indexes of development of transport systems | |||||||
| Urban Mobility Index | Center for Economic and Business Research, Qualcomm | Transport system of the city | Environmental friendliness | Querying statistics and collecting data from open sources | Minimum-maximum | Equal weights | Linear |
| Sustainable Urban Mobility Index | World Business Council for Sustainable Development | Transport system of the city | Mobility and efficiency | Querying statistics and collecting data from open sources | Minimum-maximum | Equal weights | Linear |
| Sustainable Urban Mobility Index | Arcadis | Transport system of the city | Mobility and efficiency | Querying statistics and collecting data from open sources | Minimum-maximum | Equal weight (subindexes), expert assessment (indicators and indicators) | Linear |
| Deloitte Urban Mobility Index | Deloitte | Transport system of the city | Mobility and efficiency | Querying statistics and collecting data from open sources | n/a | Equal weights | Linear |
| BCG European Railway Performance Index | BCG | Railway networks of European countries | General level of development | Based on quantitative and qualitative information from the International Union of Railways | n/a | Equal weight | Linear |
| Innovation development indexes | |||||||
| BCG Digital Acceleration Index | BCG | Companies of various industries | The ability to create value with digital technology | Company representatives survey | n/a | n/a | n/a |
| Digital Transformation Index | Dell Technologies | Companies of various industries | Company readiness for digital transformation | Company representatives survey | n/a | Expert assessment | n/a |
| Airline Digital Development Index | Lufthansa | Aviation companies | The level of development of digital solutions | Collecting data from open sources | n/a | n/a | n/a |
| Urban Innovation Mobility Index | Dubai Roads and Transport Authority, International Public Transport Union | Transport system of the city | Level of innovative development | Querying statistics and collecting data from open sources | n/a | n/a | n/a |
| European Digital Transformation Index | European Commission | Member States of the EU | Level of digital transformation | Statistics query | Minimum-maximum | n/a | n/a |
