Integral Assessment of the Level of Innovative Development of the Railway Industry Companies

NTI Competence Center, in the direction “Technologies of Storage and Analysis of Big Data,” has significant background in the development of integral indexes: a) IQ Cities (to Ministry of Construction of the Russian Federation), b) National Index of digital development of the subjects of the Russian Federation, c) Index of readiness of industries for the introduction of artificial intelligence, d) Index of innovative development for leading domestic companies, including in the sector of air transportation and oil production.


INTRODUCTION
The railway industry is one of the backbone elements of the economic systems of most countries, while, both in passenger and freight traffic, it experiences significant competition from other modes of transport. In these conditions, the use of innovative and technological solutions aimed at increasing efficiency and optimization becomes the main factor in the development of the competitiveness of individual railway companies and the industry.
Considering the growing competition in the transport sector, the innovation policy of the public sector and the management of railway companies has to be based on assessment of the level of their innovative development by identifying the advantages and bottlenecks in technological development of companies relative to the main counterpart companies. Current market landscape demands a high-quality system for assessing the innovative development of railway companies. Such system allows to perform an effective comparative analysis and assessment of railway companies, but also companies providing transport services by other modes of transport. One of the main tools that make possible carrying out such an assessment and analysis, considering multidimensional indicators and parameters, are integral indexes.

MATERIALS AND METHODS
Integral indexes include coefficients that allow taking into account various indicators within the framework of a single calculation scale, differing both in units of measurement and in the nature of the assessment (quantitative and qualitative indicators). Their use makes it possible to significantly simplify the analysis of multidimensional and complex domains in cases where research objects are described by a wide range of different indicators, making it difficult to identify the differences between objects based on them [1 -7]. For example, the article [3] describes ways to assess the complex environment "emerging technologies governments, and human preference work together to create a dynamic system loop" using an integral index. According to the study [5], "the complex index of air pollution and technological complex" can assess complex environmental systems with high accuracy, which is impossible by assessing individual indicators. The integral index combines several assessment parameters, i.e., indicators, ranging based on predefined weights.
Indicators are subdivided into several groups, the number of which is determined by the thematic focus of the integral index: high-level indicators (subindexes), reflecting the main The assessment of the indicators of the lower level (the lower level is determined depending on the component saturation of the assessed direction) is based on the values of specially calculated indicators and the degree of their influence on the indicators, determined in the form of specific weights. Schematically, the described system of forming the integral index is shown in the Fig. (1) below.
It is important to note that the terminological difference between the concepts presented in the framework of this structure, i.e., sub-index and indicator, indicator is quite relative and can be clearly traced only in cases of the multilevel integral indexes. The direct difference between a subindex and an indicator from an indicator is that the latter is a normalized estimate of the real parameter of activity (of the country, city, company), while subindexes and indicators are an aggregated assessment of indicators.
The indicators presented within the framework of the index structure, as a rule, differ significantly from each other at the level of units of measurement and dimension, which significantly complicates the calculations and the formation of the result. To overcome this limitation while developing integral indexes based on a wide range of different indicators, a standardization mechanism is used, which is an adjustment of indicators in accordance with pre-selected conversion methods (Appendix A1, Table 1), allowing the indicators to be formed in a more convenient way of calculation and comparison.
A great variety of rationing methods defined by both advantages and specific individual disadvantages are used in the world practice. Shortcomings do not refute the practical applicability of these methods; they are only an indication of existing methodological "bottlenecks" that must be considered when choosing the specific method and analyzing the result of the application.
When calculating the integral index, each sub-index, index, and indicator has a representation of specific weights according to the methods of their assignment (Appendix A2, Table 2). The specific weights reflect the conditional degree and significance of the indicators for the formation of the final evaluation.
After the selection of the coefficients of the specific weights, the transition to the final stage of the formation of the index, the aggregation of indicators, is carried out. The main methods used for the aggregation are shown in Appendix A3, Table 3.
In order to identify the widest array of practical usage of integral indexes, including the areas of evaluation and used indicators, a review of research papers on the topic of assessing the level of development of the railway industry was carried out. The selection of relevant scientific articles and publications was made using databases, such as Scopus, Web of Science and eLibrary. Based on the results of the analysis of the research experience of using integral indexes for the assessment of the transport industry development, the key approaches used to form such indexes were identified (Appendix B1, Table 1).
The choice of the methodology for calculating the integral index determines the validity of the result and the reliability of the subsequent comparative analysis. The listed methods are the most commonly used for formulation of integral indexes for assessing various subject areas, including the level of development of transport and logistics and railway companies. The researcher examines existing and commonly used indexes that are applied to assess the urban transport systems or segments of the transport industry; they are described in the following list [8 -12].
The carried-out analysis confirms the fact that the indexes that are actively used in world practice to assess the level of development of the transport sector and innovations, as a rule, are not sufficient to conduct a full analysis of the level of innovative development of transport companies, primarily railway operators. The greatest value in the context of developing an index of innovative development of railway companies is represented by indexes that assess the level of innovative development at the level of companies in the transport industry, since the assessment parameters used to calculate such indexes are largely comparable to the key indicators of innovativeness in the railway industry. -binary indicators for assessing the presence or absence of the use of advanced innovative solutions and technologies.
-quantitative indicators for assessing the effectiveness and scope of innovation policy and the use of advanced innovative solutions and technologies.  Table 2. SWOT-analysis of the innovation development index of railway companies.

Strengths Weaknesses
-Comprehensive assessment of railway companies' innovation activities. -The use of heterogeneous indicators (quantitative, binary, evaluation of innovative solutions). -Ability to modify the composition of indicators without changing the logic of the integral index construction.
-There may be data misrepresentation due to the lack of complete and reliable information about the technical aspects of applying innovative solutions of some railway companies.

Strengths Weaknesses
-Obtaining official data on TRL evaluation of innovative solutions from railway companies (besides Russian Railways holding).
-Establishing the index as an official index in the UIC.
-An innovative development paradigm shift. -Prohibition of data exchange between railway companies.
In world practice, the indexes of innovative development of transport systems are actively used, including the BCG digital acceleration index [13], the digital transformation index [14], the airline digital development index [15], the urban innovative mobility index [16]. The listed indexes are characterized by a wide range of indicators that are relevant, among other things, for assessing the level of digital maturity of railway operators, actively developing digital services for customers, and optimizing internal management systems using advanced digital solutions. The most relevant is the urban innovative mobility index, which assesses the level of innovative development of the city's transport infrastructure, which, considering the trend towards the integration of the railway network into the urban transport system, determines the relevance of several parameters included in its structure for use in calculating the integral index of innovation development of railway companies.
Furthermore, in the Russian Federation, index of the development of the transport complex, which is compiled annually by the Lomonosov Moscow State University, is actively used, reflecting the results of comparison of the leading Russian and foreign cities in terms of the quality of transport services, their accessibility to the population, road safety, and the impact of transport on the environment, efficiency of freight logistics and other indicators [17]. It is designed to identify the strengths and weaknesses of the transport complex of large cities and determine the optimal ways for its further development.
Based on the results of the analysis of the practical experience of using integral indexes to assess the level of development of the transport industry and the readiness of

Integral index
innovations at the company level, the key approaches used to form such indexes were identified (Appendix B2, Table 2) [18 -22].
Among the approaches analyzed in open sources, no mechanisms were identified that directly assess the level of innovative development of railway companies; either the overall efficiency of the railway industry at the level of individual countries is assessed, or innovative development/digital transformation without reference to a specific industry or related industries. According to the results of the study, it can be concluded that currently in the railway industry, there are no generally accepted methods for assessing the level of innovative development of railway companies, which necessitates the development of such a method.
Based on the results of the analysis of existing approaches to the formation of integral indexes, including integral indexes of the general and innovative development of companies in the transport sector and the railway transportation sector, the following conclusions were drawn: (1) In the international practice of analyzing the innovative component of the transport sector, as well as at the level of thematic scientific works, the fact of using integral indexes, the object of which would be the level of technological, innovative, and digital development of railway companies, was not revealed. The existing indexes allow us to assess only the general level of innovative development of the transport system or the level of readiness of digital technologies of companies without specifying the industry. As a result, in the conditions of accelerating innovation cycles and active digitalization of the railway industry, the importance of a mechanism that makes it possible to assess the level of innovative development of railway companies on a regular basis, considering the main directions of technological transformations, increases.
(2) Selected methods for constructing these indexes, as well as their individual indicators, can be used to develop an integral index of the innovative development of railway companies. The methodology for constructing the considered indexes is based on a three-level structure, which is most appropriate for grouping the parameters used, since it allows to comprehensively cover all the main areas of activity while maintaining the relative simplicity of formation. The indicators that allow assessing the level of digital development of companies, in the use of specialized digital services and information and communication systems, are of the greatest importance for the formation of the index of innovative development of railway companies.
(3) The most common standardization method for calculating integral indexes of the overall development of transport systems is the minimum-maximum method (in 4 out of 10 analyzed indexes), which is explained by the relative ease of use with a sufficiently large number of heterogeneous variables. To assign specific weights, the method of equal weights is predominantly used, while in the case of full or partial base of the index on the results of surveys, the method of expert assessments is used. For aggregation, as a rule, a linear method is used due to the relative ease of implementation.
To form a unified toolkit for assessing the level of innovative development of railway companies, an integral index of innovative development of railway companies (hereinafter referred to as the Index) was developed, which is a comprehensive assessment of the achieved results of railway companies in the field of innovative development based on quantitatively and qualitatively measured indicators in the results of the reporting period. The index makes it possible to assess and monitor the level of innovative development of the Russian Railways holding. The methodology for calculating the Index was calculated considering the best practices in the field of constructing similar indexes.
As part of the approbation of the methodology for calculating the Index, 20 different approaches to building models were tested, which differ in the structure of grouping indicators at the level of subindexes, the method of standardizing quantitative indicators, and directly used indicators. The following were considered as the main methods for standardizing quantitative indicators:

Minimum-maximum method;
A method based on a categorical scale (based on thresholds).
The choice of these methods was due to their prevalence for calculating various integral indexes, including in the railway industry. In accordance with the standardization methods used, the models were subdivided into 2 main groups, within which further iterations were performed. Based on the results of the approbation, it was decided to apply the final version of the methods for calculating the Index of the minimum-maximum method, which, in contrast to standardization based on threshold values, allows the principle of flexibility of the developed methodology to be ensured. The fact that, if necessary, when analyzing indicators, considering changes in the scientific and technological structure, cycles and the emergence of new technological trends, the calculation methodology can be changed by adjusting the composition of indicators (excluding irrelevant indicators and/or adding new relevant indicators).
The initial concept envisaged the creation of a three-tier structure, including subindexes of the 1 st level, subindexes of the 2 nd level and indicators. Level 1 subindexes were based on the principles of grouping indicators characterizing the areas of activity of companies, including: Grouped indicators reflecting the innovative component in the field of cargo transportation; Grouped indicators reflecting the innovative component in the field of passenger transportation; Grouped cross-cutting indicators, the assessment of which can be sent without considering the activities of the analyzed companies.
Within the designated indexes of the type, there are subindexes of the 2 nd level, grouping indicators according to a functional characteristic. In the future, to avoid the prevention of assessments due to the imbalance of performance indicators, it was decided to switch to the use of a two-tier structure, where the grouping of indicators at the level of subindexes is based on the principle of separation according to a functional criterion. Thus, the final structure ensures the use of the following subindexes: innovation policy; advanced product and technological solutions, technical means.
In the process of approbation, the composition of the indicators has also changed. The parameters related to the regional characteristics of innovative development were determined. The main reason for excluding several indicators from the final version of the methodology for calculating the Index is the lack of reliable data for their calculation in open sources for the most assessed railway companies. At the same time, within the framework of approbation, the indicators are included in the calculation of the Index for a limited sample of companies to verify the results; according to the results of the verification, no bias in the results of the index assessment was found.
The overall structure of the Index consists of indicators for assessing the level of readiness of advanced innovative solutions and technologies, binary indicators for assessing the presence or absence of the use of advanced innovative solutions and technologies, the number of indicators for assessing the effectiveness and scope of innovation policy and the use of advanced innovative solutions and technologies. In the structure of the Index, there are two basic averaged estimates of indicators, grouped according to a certain criterion in the general structure of the integral index (hereinafter referred to as subindexes, Table 1): Innovation policy, which includes indicators reflecting the level of organization of innovative activities within the framework of railway companies (9 indicators, of which 2 are quantitative, 7 are binary, the subindex weight in the total evaluation of the integral index is 9/35); Advanced products and technological solutions, technical means, which include indicators reflecting the results achieved in the field of innovative development of railway companies (26 indicators, of which 4 are quantitative, 22 indicators for assessing the level of technology maturity according to the Technology Readiness Scale, the subindex weight in the total evaluation of the integral index is 26/35). Quantitative indicators are brought into a comparable form due to the transition from absolute values to normalized values (from 0 to 1 or from 1 to 100) using the minimum-maximum method. The formula for converting an indicator into conditional points from 1 to 100, if a high value of this indicator is a good result (for example, revenue): -the normalized value of the k-th quantitative indicator of the i-th direction of innovative development 1 of the j-th railway company; 1 The directions of innovative development mean the conditional grouping of various indicators based on several common features. The grouping is of a formal nature and does not affect the formation of the normalized assessment, as well as the assessments of the sub-index and the final assessment.
-the current value of the k-th quantitative indicator of the i-th direction of innovative development of the j-th railway company; -the minimum value of the k-th quantitative indicator of the i-th direction of innovative development across the entire sample of railway companies; -the maximum value of the k-th quantitative indicator of the i-th direction of innovative development across the entire sample of railway companies.
The formula for converting an indicator to conditional points from 1 to 100, if a high value of this indicator is considered a bad result (for example, CO2 emissions): -normalized value of the k-th quantitative indicator of the i-th direction of innovative development of the j-th railway company; -the current value of the k-th quantitative indicator of the i-th direction of innovative development of the j-th railway company; -the minimum value of the k-th quantitative indicator of the i-th direction of innovative development across the entire sample of railway companies; -the maximum value of the k-th quantitative indicator of the i-th direction of innovative development across the entire sample of railway companies.
To avoid distortions of the final estimate if a company provides services exclusively in one segment, quantitative indicators related to another segment at the level of this company are assigned a «zero» value (due to the methods of standardizing indicators, these values are excluded from the calculation of the index from a certain company). If data on quantitative indicators for an individual company are not available in the available information sources, while the parameters estimated using these indicators are potentially present in this company, such indicators are also assigned a «zero» value [23 -25].
The final score of the Index for each of the railway companies is formed on a scale from 0 to 1 based on the results of aggregating subindexes using specific weights calculated as the ratio of the number of indicators included in the sub-index to the total number of indicators in the index structure. To aggregate the above groups of indicators (indicators for assessing the level of readiness of decisions, binary indicators, quantitative indicators) at the level of subindexes, the average value of the indicators included in a certain sub-index is calculated. To aggregate subindexes when calculating the final value of the integral index, a linear method is used, which consists of summing the normalized estimates of the variables, considering the predetermined specific weights: For the assessment within the framework of the Index, 19 railway companies were selected that are close in the type of activity to the Russian Railways holding (comparability of the operating business model) and with a comparable or larger scale of activity compared to the Russian Railways holding. As a result of the assessment of the Index (based on information for 2020), a matrix was formed showing the values of the index for each of the railway companies (Fig. 2). Depending on the points received, railway companies can be further divided into the following 3 groups in terms of the level of innovative development: «leaders» (above 0,66 points); «Stable» (from 0,33 to 0,66 points); «Lagging behind» (below 0,33 points).

RESULTS
According to the results of the Index calculation, the companies Deutsche Bahn (Germany), SNCF (France), China Railway (China), Russian Railways holding (Russia), SBB CFF FFS (Switzerland), Japan Railways (Japan) are recognized as the leaders of innovative development. The mentioned companies have the largest number of innovative solutions which are currently used. For example, Deutsche Bahn и SNCF actively developed «digital twins» of rolling stock and infrastructure complex, drones, MaaS platforms, smart contracts, intelligent chat bots, platforms for organizing multimodal transportation. China Railway is a world leader in high-speed railways development and implementing of digital systems for railway process management. Russian Railways holding has one of the most developed intelligent control systems for rolling stock and for maintenance and repair of rolling stock and infrastructure complex, natural gas rolling stock. The leaders mentioned above are ahead of other companies by quantitative and binary indicators which are included in the subindexes. This is largely due to a similar companies' business model and developed systems of internal corporate management focused on implementation of innovations.
Assessment of the level of innovative development of railway companies on an annual basis will make it possible to assess the position of railway companies in dynamics and to respond more quickly to new trends in scientific and technological development.

DISCUSSION
The conducted approbation proved the high level of the Index's applicability for the regular assessment of the level of innovative development of railway companies. The method for calculating the Index was successfully tested as a part of the assessment of the level of innovative development of the Russian Railways holding. The Index was verified by members of the Management Board of JSC Russian Railways. Table 2 shows the SWOT-analysis of the developed integral index. Fig. (2). Assessment of the innovation development index of railway companies in the context of the 20 largest companies. SI1 -sub-index 1 «Innovation policy» SI2 -subindex 2 «Advanced product and technological solutions, technical means» Broad prospects for using and increasing the reliability of the Index calculations can be ensured by verifying the developed methodology in the International Union of Railways. The annual collection of official data from railway companies, including in terms of assessing the maturity of the technologies used, and the analysis of proposals for improving the index will eliminate the existing shortcomings of the index in terms of the use of statistical data on binary indicators and indicators for assessing the level of readiness of advanced innovative solutions and technologies in railway companies based on expert assessments, which will increase the accuracy of the Index calculation and the quality of the findings.
In addition, the international recognition of the Index will make it possible to expand the range of indicators used, for example, in terms of indicators, such as «Share of R&D costs (attributed to revenue), %», «Accuracy of adherence to passenger train timetables, %», «Share of freight shipments delivered in compliance with the established delivery time, %». These indicators were included in the calculation of the Index in one of its editions but were excluded due to the availability of data in open sources of information not for all railway companies.

CONCLUSION
The use of integral indices is an effective way to assess the complex innovation environment of railway companies. The index can be actively used in the practice of railway companies for annual monitoring of the effectiveness of innovation activities and identification of strengths and weaknesses for their further adjustment by strategic development programs. It will be possible to update the indicators of the developed integral index, taking into account changes in plans of innovative development of railway companies in the future. The developed methodological approaches can also be used to create other integral indices by railway companies to assess various aspects of their operation activity.

CONSENT FOR PUBLICATION
Not applicable.

AVAILABILITY OF DATA AND MATERIALS
Some or all data, models, or codes generated or used during the study are proprietary or confidential in nature and may only be provided with restrictions.

FUNDING
The work has been supported by the NTI Competence Center in the direction "Technologies of Storage and Analysis of Big Data", Moscow State University named after M.V. Lomonosov, Moscow, Russia. Competence Center in the direction "Technologies of Storage and Analysis of Big Data", Rakov Dmitry, Deputy Head of Consulting and Expertise of NTI Competence Center in the direction "Technologies of Storage and Analysis of Big Data" for their very helpful suggestions.

Method Name Description Advantages Disadvantages
Ranking method Indicators are ranked according to their level of significance in descending order, by assigning numerical equivalents from 1 to n.
-Resistant to «outliers» in the data -Relative ease of implementation -The impossibility of assessing indicators in absolute terms The classical method of valuation It is calculated as the ratio of the value of an individual indicator to the average or maximum value of this indicator for the entire sample (for example, a group of companies), or, conversely, as the ratio of the average / maximum value of the indicator for the entire sample to a separate indicator. Calculated by the formulas: -

Method Name Description Advantages Disadvantages
Distance to reference method Measures the position of a metric in relation to a predetermined reference value, for example, a target metric value that must be achieved in the analyzed time.
-Relative ease of implementation -Weak resistance to extreme values and «outliers» in the data

Categorical scale method
Allows you to assign a value to each indicator in accordance with the specified categories, which can be both quantitative and qualitative. This method can also consist in assigning scores in accordance with getting into one or another percentile for an indicator. For example, the 10% of companies with the best performers receive 100 points, the next 10% receive 90 points, and so on. up to 0 points.

Method Name Description Advantages Disadvantages
Multi-criteria aggregation They are used when very different parameters are aggregated in the index. For example, production, financial and social indicators.
-Accounting for differences between parameters -Complexity of calculation -Unlike other methods, this method is not characterized by the mutual compensability of the indicators included in it; the influence of indicators whose values of many indicators are slightly better will have a greater impact on the final assessment than indicators whose values of several indicators are much better. transport infrastructure of the region.Bulletin of PSU, Series: Economics, vol. 4, no. 19, 2013. L.N. Rodionova, and R.N. Kidrachev, Transport component in [25] determining the integral index of the competitiveness of the (Appendix B2) contd.....