Safety KPIs Monitoring of safety performance

This paper aims to provide brief overview of aviation safety development focusing on modern trends represented by implementation of Safety Key Performance Indicators. Even though aviation is perceived as safe means of transport, it is still struggling with its complexity given by long-term growth and robustness which it has reached today. Thus nowadays safety issues are much more complex and harder to handle than ever before. We are more and more concerned about organizational factors and control mechanisms which have potential to further increase level of aviation safety. Within this paper we will not only introduce the concept of Key Performance Indicators in area of aviation safety as an efficient control mechanism, but also analyse available legislation and documentation. Finally we will propose complex set of indicators which could be applied to Czech Air Navigation Service Provider. KeywordsAviation safety, State Safety Programme (SSP), Safety Key Perfomrance Indicators (SKPIs),


INTRODUCTION
Safety has always been highlighted as the highest priority in aviation.It isn't accidental; unlike other means of transport consequences of risk situation in aviation led regularly to disasters and massive losses of lives.In order to keep interest in aviation development and taking advantage of its benefits a strict approach to safety had to be developed.The approach had changed itself many times during history; starting from understanding aircraft technology and its issues in the very beginning of commercial air transport and resulting into systematic point of view on whole aviation system nowadays: Figure 1: Evolution of approach to aviation safety [1] Over the time most significant change is introduction of proactive approach.Today it is simply no more sufficient to analyse incidents and accidents where many lives were lost or where serious risk has been present.First these accidents happen rarely proportionally to volume of traffic and secondly today we already know that there are always many barriers which have to be broken in order to allow such accident happen.There are much more of such occurrences, where not all the barriers were broken (and sometimes even involved people didn't know that some barriers were actually broken) but risk of incident has significantly arisen.
Thus we must get used to analyse each breaking of significant barrier.Gather and compare information from different sources and understand that this is the only way how to achieve constant and efficient improvements in aviation safety.Establishing control mechanisms is crucial step on the way to achieve the above.There must be harmonized way of gathering and assessing data agreed on national as well as international level so to produce comparable analysis and comprehensive output from all available data.The output shall be used then for senior management and respective authorities to support their decision making process.

II. KEY PERFORMANCE INDICATORS IN AREA OF SAFETY
Key Performance Indicators (KPIs) applied in any system are modern approach for assessing its actual performance regarding some characteristics in order to support further decision making process.They refer to defined problem or area, which they are supposed to describe more in depth.Generally, we apply them on systems which require complex decision process and where it is not clear how to properly control certain aspects of the system.
Generally indicators are expressed by number or some value.In many cases there are KPIs being used even without defining them explicitly.Doctor might use figures of lagging (output-based) indicators gained from his devices (e.g.temperature, blood pressure), which will provide him evidence of anyone's health status and thus support him in further decision process.More complicated could be lagging indicators used in economy: annual reports may provide evidence on company's performance status.The same works for safety.If a product or service is safe it means that it is safe for its users as well as for environment; it mustn't generate danger whilst being used.But due to the complexity of this area there will never be an absolute "safety assurance", i.e. zero probability of generating danger whilst being used.However, properly identified safety KPIs should solve 2P (Production vs. Protection) dilemma, what is in fact the most significant advantage for any (aviation) company: Figure 2. Safety space [2] In any organization production and safety are linked together.If we increase production, the safety risk will most likely proportionally rise.We would need additional resources and process enhancements in order to maintain safety risk at the same level.Safety KPIs work here as a warning tool, indicating unbalanced ratio between safety (protection) and volume of our activity (production).If safety KPIs are warning us we apparently don't invest enough into the protection.Similar KPIs could be established on the other side of Safety space, but they would be Cost-efficiency related.

III. GLOBAL AND EUROPEAN APPLICATION OF KPIS TO AVIATION SAFETY
Concept of KPIs in aviation is globally accepted and supported.They are being implemented not only in area of safety but also environment, cost-efficiency and capacity.There is still wider and wider application expected.
On a global level we have ICAO standards only.These are published in ICAO Annexes and ICAO Doc.9859.Apart from that ICAO recently (endorsed in October 2013) introduced Global Aviation Safety Plan (GASP) which sets up global safety objectives over next 15 years.It includes advanced safety oversight system with predictive risk management (i.e.system approach).
On European level we have so called "total system approach" [3] to aviation safety.It is based on fact that aviation consists of several components (products, operators, crews, aerodromes, ATM, ANS) connected into one single network.To establish and maintain high level of aviation safety a common approach and basic regulations are needed.It is nowadays based on three elements: the Strategy [4], the Programme [5] and the Safety Plan [6].Strategy covers sets of objectives from political authorities and through the Programme the Plan has been identified, containing specific action plans.The aim is clear: coordinate and work together to achieve one level of aviation safety within Europe.
EU Member States were mandated to implement their State Safety Programme (SSP) [7] while organizations within them were required to establish their Safety Management System (SMS) [8].Both elements are complementary and based on ICAO standards.
Respective regulations and directives are being issued, stakeholders are involved in the development and whole approach is being refined to achieve the synergy.However, there are still many issues to be solved and the application of safety KPIs is still rather in the beginning.

IV. TYPES OF SAFETY KPIS USED IN AVIATION
The safety KPIs must be ultimately established in a complex set.This is given considering 2 types of KPIsthe leading and lagging ones [9].Both of them can be efficiently used in area of safety.As mentioned, first type is input-oriented (in terms of safety also known as proactive) and the others are output-oriented (reactive).
In aviation the lagging (reactive) indicators would express number of certain events with regards to some baseline (time, volume of traffic etc.).Such an indicator can be fatal accidents but as we mentioned before, fatal accidents happen rarely.Thus we must search for occurrences, where not all the barriers were broken: in this case it can be 'Runway Incursion' (RI).Not all incursions are necessary causing accidents but they easily can some other time.This RI KPI would count number of incursions during some time at some place.
Obviously, having 5 runway incursions at London Heathrow airport during one year cannot be considered as the same level of safety as 5 runway incursions at some regional airport during the same time.Baseline is therefore very important here to be considered in the decision making process.But if we recognize long-term ascending trend of certain indicator, it might be good to have a look what may be causing it.Too high value of some reactive indicator is a warning of higher risk in the system and should attract attention of senior management.
On the other hand leading (proactive) indicators are aimed at prevention.Such an indicator can be 'Learning Culture', where we would count number of workshops or training within some organization and regarding some issue.Achieving higher value of such KPI we should recognize descending trends of related reactive KPIs.If the KPIs set it well established it can even assess effectiveness of prevention measurements.
Process of implementation of KPIs is usually initiated by establishment of reactive part which assesses actual performance or actual level of safety.Proactive part is then added to assess our investment and effort spent regarding the "protection" outlined in 2P dilemma.Of course, in such an ever changing environment like aviation is, the measured indicators might be changed over some time, extended or even replaced by other ones.Creating and maintaining such a set requires system-wide knowledge.We must be able to discover such 'hotspots' in the system which may ultimately lead to incidents or accidents if not well prevented.To facilitate the process of gaining such a knowledge one can learn a lot from investigations and subsequent recommendations, safety-related reports or external audits.There is in fact lot of information available, however it is still not used effectively or sometimes not used at all.

V. COMPLEX SET OF SAFETY INDICATORS
EU Member States were mandated to implement their SSPs which showed many ways how EU Member States approached area of safety.All of them included different safety KPIs.Required minimum is given by EU legislation which forces states to establish first of all the proactive indicators.Three Safety KPIs are mandatory not to only monitor, but to also achieve certain values at the end of year 2019 [10].
These indicators are: Effectiveness of Safety Management System (EoSM), use of Risk Analysis Tool (RAT) methodology and application of Just Culture.EoSM indicator is based on presumption that effective SMS will significantly prevent accidents, incidents or occurrences.RAT methodology aims to create unified assessing of occurrences so to not allow ambiguity in the interpretation.Just Culture is perceived as enabler of Safety Culture in any company, creating an atmosphere where safety related issues are widely reported and being properly analyzed.
However, reactive part is completely missing in European legislation, mainly due to differences across EU Member States.Unlike by the proactive part, one simply cannot apply certain set on all states and force them to achieve required level over some time.We would need here rather some flexible framework of principles, or some tools applicable in different conditions.
Still, the issued SPPs outlined some common problems.From our perspective, the best description of reactive part can be found in Finland's SSP, albeit it mixes the reactive part with proactive part required by EU legislation.Thus, some logical links between them are missing.But on the other hand this SSP outlined how safety KPIs could be used in tiers.
We analyzed the reactive part sorted into 3 tiers and discussed it with real Air Navigation Service Provider (ANSP; in this case ANS CR -ANSP in Czech Republic) to assess possible implementation and measurement.Based on the discussion we've tried to propose refined multi-tier system of safety KPIs applicable for ANS CR.Furthermore, we've added the missing logical links between reactive and proactive part and extended the proactive part with additional indicators from Safety Culture.As a result the system is clearly showing how the KPIs set can be established and how it can assess complex system in terms of safety.

VI. CONCLUSIONS
Safety KPIs are belonging to state-of-the-art but still lowcost control mechanisms which allow efficient safety performance evaluation.They require deeper understanding of evaluated system so one can identify critical links or critical system parts which may contribute in significant way to safety occurrences, incidents or accidents.Each of such identified 'hotspots' should be then observed and considered regarding some baseline (e.g.volume of traffic) in time.Number of unwanted events shall be counted within respective KPIs.The whole safety performance is then recorded not only in terms of unwanted events but also in terms of investment into prevention.
Therefore it is highly important to set the indicators in a correct way.This is main issue nowadays and we hope it is going to be soon resolved.To support this process at least in Czech Republic we have outlined our vision how such a KPIs set could be established.According to dialogue with ANS CR operation of the set would be definitely feasible.Ultimately it would create an overview of safety performance pointing out real issues.Decision making process regarding further action taking would be then facilitated.

Figure 3 .
Figure 3. Tiers of safety indicators But having complex issues it is certainly becoming more complicated to identify what is actually such a "key" indicator and what is not.