Automation of Taxiing

The article focuses on the possibility of automation of taxiing, which is the part of a flight, which, under adverse weather conditions, greatly reduces the operational usability of an airport, and is the only part of a flight that has not been affected by automation, yet. Taxiing is currently handled manually by the pilot, who controls the airplane based on information from visual perception. The article primarily deals with possible ways of obtaining navigational information, and its automatic transfer to the controls. Analyzed wand assessed were currently available technologies such as computer vision, Light Detection and Ranging and Global Navigation Satellite System, which are useful for navigation and their general implementation into an airplane was designed. Obstacles to the implementation were identified, too. The result is a proposed combination of systems along with their installation into airplane’s systems so that it is possible to use the automated taxiing.


Introduction
With decreasing separation between aircraft and the growing need to increase airspace capacity, great emphasis is put to navigation and automation in flight. The high utilization of airspace is, of course, accompanied by a high utilization of airports, which are, moreover, constantly being expanded in order to meet traffic demands, which makes them more and more complex. Despite the complicated system of taxiways, the land navigation is still made solely on the basis of visual perception and aircraft taxiing systems (throttle, steering, brakes) are operated manually by the crew, just as it has been done since the early days of aviation. The increased workload of the crew subsequently creates an environment, in which a critical situation can arise easily, both in terms of the traffic flow, as well as in terms of safety. This article assesses the possibility of introducing instrument navigation methods for ground operations, with emphasis on systems that do not rely on the Global Navigation Satellite System (GNSS).
The use of GNSS (with or without augmentation) as a single source of navigational data, for navigating on taxiways, has certain limitations.
It is mainly an insufficient navigational performance in terms of integrity and accuracy, which requirements are hardly or not at all satisfied. Besides the further development and improvement of GNSS navigation, it is also possible to deal with these limitations by using systems that are independent of GNSS data and can ensure improvements in integrity and accuracy, but furthermore provide new features, which would increase the safety of taxiing. One of the new features is obstacle detection in front of an airplane, which cannot be provided by the GNSS navigation itself.
To address the shortcomings of satellite navigation, technology independent of GNSS can be used. The main limitation of GNSS use for taxiing is the insufficient Time To Alert, therefore the primary requirement for such independent system would be the ability to follow navigational means (taxiway centerline) in a way, that a safe movement is ensured for sufficient amount of time [1,2]. Following the taxiway centerline is also very accurate, so the accuracy of the whole navigational system can be increased. Two systems were chosen for this purpose: Computer vision and Lidar.
Both Computer vision and Lidar replace the pilots vision, allowing for very precise following of airport marking (taxiway centerline) and also detection of obstacles in the field of vision. The Computer vision can also be used for detection and reading of airport signs and detection of active stop bars. Combination of these systems with satellite navigation addresses the insufficient performance of satellite navigation and combines the advantages of accurate following of taxiway centerline with determination of position at the airport. The main disadvantage is the dependency on weather conditions (fog, rain, snow). The next paragraphs describe both Computer vision and Lidar.

Computer vision
Computer vision, in general, addresses digital image processing in order to obtain information that can be further processed. It is quite common way of dealing with various issues in transportation, such as recognizing of road signs, analysis of traffic density or following of the lane. As a consequence of everincreasing computational power of today's computers and improving of image quality of cameras, the range of applications of the computer vision expands, and therefore it is desirable to consider the use of it in aviation.
The use of computer vision is essentially a direct substitute of a human, who reacts to situations based on looking out of the cockpit. The function of eyes is replaced by a camera mounted at a suitable location and the image evaluation is performed digitally using a computer program, which uses predefined algorithms. The analogy with human vision, however, introduces various setbacks to this solution (dependence on weather conditions, blind spots, etc.). The quality of evaluation of scanned situation depends on the quality of scanned image and on the sophistication of the algorithms.
Computer evaluation of an image is rather complex and demanding process, which can be divided into several steps: 1. Recording of an image and its digitalization 2. Preprocessing 3. Detection of predefined elements 4. Description of objects The image is first recorded by a camera (working in the appropriate region of the light spectrum) or a pair of cameras to ensure stereoscopic vision, which enables a relatively accurate calculation of distance between objects in the captured image and the camera. The image is further digitalized and as a result, there is a matrix defining the light intensity of each of the recorded points. The preprocessing is used for adjusting the whole image, right before the application of the algorithms. It includes, for example, contrast enhancement, noise reduction or adjustment of the color spectrum. After the preprocessing stage, the algorithms are applied and predefined objects, such as straight lines, edges of objects or curves, are recognized. Further algorithms are applied to these recognized elements of an image and the result is a description of the object of our interest in the form of information about concrete points (location and brightness) of this object. With the knowledge of physical location of the camera(s), it is possible to use this information for calculations of the actual values of distance, or to compare it with known objects (road signs, light color) and assign it a proper importance. [3] When considering the use of computer vision for taxiing, it is appropriate to return to the analogy with a human vision. The pilot determines the direction and speed of an airplane based on what he/she sees outside. The basic monitored elements of surroundings are airport signs and markings, on the basis of which the direction is controlled to ensure taxiing according to a given clearance. Furthermore, obstacles in the area are being observed, especially vehicles and other airplanes on the apron and taxiways, in order to provide for safe distance or to even prevent a collision. These activities could be replaced by the computer vision and its principle is shown in Fig. 1.

Image from camera
Computer vision Calculation of the required speed and direction Definition of objects Unlike the pilot's eyes, the camera can be placed at a location, where it is least affected by the meteorological situation, or multiple cameras can be used to ensure sufficient angle of view and redundancy. For ensuring movement along the taxiway centerline, it is necessary to identify the taxiway markings. The axis location on recorded image is clearly defined by the computer vision and given the known location of the camera, i.e. known distance between the camera and a taxiway, it is possible to, using the number of image points, calculate distance difference between a defined middle point and a taxiway centerline marking. This distance is consequently used to determine steering corrections in order to keep the nose wheel on the taxiway centerline. Intersections can be detected and a required trajectory set by detecting information markings. The issue of speed of taxiing is usually dealt with based on the operational situation. The computer vision enables detection of stop bars or runway holding points, on the basis of which a command to stop the airplane can be issued. When a pair of cameras is used for stereoscopic vision, the distance between the airplane and various objects can be calculated, therefor an obstacle can be detected and a collision avoided. This use is, however, limited by the meteorological conditions, because the obstacles are usually located in much greater distance then the markings and signs used for navigating.
Visibility is a major limiting factor for the use of the computer vision for taxiing. This restriction is the most limiting for detection airport signs, stop bars and runway holding points, which need to be detected in advance in order to allow for a safe stopping of an aircraft. On the other hand, it is not a limiting factor in case of taxiway centerline detection, because the location of the camera can be chosen in such way, that the ground is being recorded from relatively short distance and the surrounding area can be lit up by the taxi lights. With such short distance, a good visibility of markings can be ensured even in a thick fog. However, in all cases of the use of computer vision, it is necessary to ensure that all signs and markings are visible and not damaged. Figure 2 shows an example of taxiway centerline detection using only the basic algorithm probabilistic hough line transform on a relatively poor quality image in grayscale (left) without any filtering. On the right side of the figure is the resulting matrix of data, containing information about detected lines, converted to an image. The pink lines represent a detected object, which is in this case taxiway centerline. It is noticeable that despite the absence of any filtering, there is quite clear differentiation of the centerline from its surroundings. Using additional filters, such as color filter, the contrast can be further increased and the resulting values compared with the standard definition of the taxiway centerline. As well as the lines, curves can be detected and by their curvature, the turn radius can be determined in advance and optionally a speed can be determined based on the radius.
To summarize the computer vision, its advantages are a very accurate navigation along markings on a taxiway, ability to read signs, obstacle detection and a minimal Time to Alert [4]. Great potential for the use of computer vision on airports lies in the airport design philosophy itself, when great emphasis is given to contrast of signs and markings. However, there are disadvantages as well, such as the already mentioned dependency on meteorological conditions, especially visibility and the state of taxiways (layer of snow prohibiting the camera from scanning the markings). It is also impossible to determine airplane's location at the airport, but this could be dealt with by combining the computer vision with the GNSS.

Lidar
Light Detection and Ranging is an active system, using the reflection of laser beams to compute distances from surrounding objects by measuring the time between a beam emission and its return to the device. Historically, Lidar was developed solely for determining distances from objects. But, as the computational power increases, it finds use in much more demanding applications, such as creating 3D maps of airport obstacles, or detection of road and obstacles by an autonomous car. Regarding the fact that Lidar is already successfully being used in driver's assistant systems and in autonomous cars, it is possible to expect, that it can be used in the standardized conditions of airports with similarly successful results.
Lidar is usually composed of an evaluation unit and a head, containing all the sensors and laser beams transmitters (see Fig. 3). The device continuously emits invisible laser beams, which reflect off objects back to receivers placed in the head. Based on the information about intensity and an angle of reflected beam, the evaluation unit processes a 3D map of surroundings, which is continuously updated [5,6].
The Lidar principle suggests that the Lidar, although capable of creating an accurate model of objects, is not capable of reading airport signs. It is, however, possible to analyze the surface of a taxiway and detect markings based on their specific reflecting characteristics [7].
The principle of using Lidar technology for taxiing is similar to the computer vision with the difference that Lidar detects objects and markings based on their reflecting characteristics of laser beams. Figure 4 shows the principle of taxiing using Lidar.
The Lidar system is, just as the computer vision, installed on an airplane and has similar advantages and disadvantages. It detects obstacles and creates an accurate model of the airplane's surroundings and is also capable of determining the distance from these obstacles.  The main disadvantage is the dependency on meteorological conditions. Considerable limitation in comparison with computer vision is the inability to detect light emitting stop bars. On the other hand, Lidar does not require the illumination of detected objects, therefor is capable of detecting obstacles even at night.

Taxiing automation
The previous sections discussed systems that could enable automation of taxiing. The main problem is the navigational aspect of taxiing, when the objective is a reliable determination of airplane's location anywhere at the airport and its comparison with an ideal taxi route. Acquired deviations from ideal taxi route can be further used for airplane control on the ground, regardless of manual or automated control. For going forward, it is necessary to expect, that ideal navigation data and trajectory definition are available and the only task is to determine the deviation and its subsequent transformation to required corrections, which are used to control the airplane systems responsible for taxiing (steering, accelerating and decelerating). The goal is to point out the principles for realization of automated taxiing.
The current standardized way of controlling an airplane on the ground does not, with the exception of automated landing and braking, use any form of automation and its execution is the full responsibility of a crew. Simplified process is depicted in Fig. 5. Ideal taxi route is defined by clearance in the form of taxiway letter labels, which is given to the crew by the Air Traffic Control (ATC). The information about location and direction is acquired by looking out of the window and comparing the airport signs with taxi map. The speed is generally determined using an on board device (usually GNSS) and its rough value is especially important to avoid exceeding limits specified by the manufacturer or an airline. The value of acceleration is determined only on the basis of the crews' subjective feelings, so as to prevent any reducing of comfort of the passengers. This information is then, based on the pilot's training and experience, transferred to deflections of tiller (nose gear steering wheel), throttle lever and brake pedals [9,10].
The described process has a number of disadvantages that stem from the absolute dependence on human factor. Manual taxiing requires considerable amount of experience and takes up most of the mental capacity of the crew member controlling the aircraft, similar to when the flight is done without the use of autopilot. The need of automation in aviation was evident in the relatively distant past, when the first autopilot systems were developed, using feedback loops for controlling the airplane in the air. However, deployment of automated systems for taxiing could not take place, because there was not technology capable of replacing the human. Currently, such systems already exist and are successfully tested in transportation industry.
The automation of motion control technology is in general challenging and requires the integration of navigation systems and systems for actuators control (nose gear steering, braking intensity and engine power). The principle of automated taxiing is depicted in the following Fig. 6.
Input parameters for automated taxiing (ideal taxi route, information on position, speed, acceleration and direction) are identical to the case of manual control, as described in the previous section. Taxi route can be defined by the crew based on the received clearance or directly transmitted from ATC through a datalink (for example CPDLC). The rest of input parameters is, however, opposed to manual control, where their values is subjectively determined by the crew, determined precisely via navigation systems, speedometers and accelerometers, which itself eliminates the factor of human error in the process of obtaining them. These input parameters are then processed by digital computing systems in order to determine deviations of real location and speed from a situation defined by the ideal taxi route. Based on the information about these deviations, the value and an orientation (direction) of corrections is calculated. The corrections are then transformed by actuators to an angle of a nose gear, braking intensity and engine power with regard to a model of taxiing dynamics.  The automated process of corrections determination also uses feedback from engines and a nose gear to ensure smooth and precise maneuver execution. The described process is taking place constantly and thanks to the current computational power of computers and high sampling rate of sensors, the potential reaction time is several times lower than the reaction time of the flight crew.
The issue of implementing automated taxiing into airport ground operations has to be assessed from various perspectives. One of them is the actual control of those airplane parts responsible for direction and speed. The automation of current large passenger airplanes is quite significant a directly controls engines, brakes and a nose gear (during automated landing). There are two technical solutions in use to control the nose gear: 1. Control via cables, going from the steering wheel to hydraulic cylinders on the nose gear (for example Boeing 737). 2. Control of the hydraulic barrels via electric signals generated by computing unit. The control commands are generated based on the position of control elements in the cockpit and autopilot requirements during automated landing. This solution is used at the newest airplanes (for example A380). Figure 7 shows the first solution, which is now obsolete, but still in use. Its advantage is independence of electric power. The disadvantage is the impossibility of automation without further installation of additional devices (servomotors). Development, installation and certification of such devices would be financially and time demanding, therefore it is improbable that aircraft operators would accept it. Furthermore, airplanes using this solution are generally not able to land under CATI-IIc conditions, therefore the risk of zero visibility after runway   Figure 7. The older solution [11]. vacation is minimal. In conclusion, this arrangement is not suitable for automated taxiing and due to great weight and complexity, it will be completely replaced by electronic control systems.
The second, modern solution is shown in Fig. 8. It depicts a system used on Airbus A380 that controls the steering of nose gear and rear wheels of the main landing gear. The control elements located in the cockpit and hydraulic valves arranging the turning of wheels, are connected by electric lines by means of two redundant computing units (SCS), which evaluate the requirements for an angle of nose gear generated by autopilot (ECAM), brake pedals or the steering hand wheel. Such solution is, understandably, more suitable as it does not require installation of additional equipment. However, it is necessary to develop a software code, that generates proper demands on steering during taxiing.

Conclusion
The summary of the current systems for the control of an airplane on the ground shows that it is necessary to develop a new technological solution for automation of taxiing. Modern airplanes (generally the ones certified for CATIIIb and IIIc) are suitable for implementation of automated taxiing as they already have automated systems for brake and nose gear control. However, these functions are used only for slowing down after landing, therefor it is necessary to develop a new software solution. The older airplanes with mechanical controls cannot be modified for automated taxiing without installation of additional equipment, therefore it is reasonable to expect that costs would be disproportionate to the airplane's remaining flight time.
It is understandable that the automation of taxiing will be preceded by extensive studies, development and investments. However, it is a logical, and with increasing traffic, a needed step forward.