Prediction of Lower Extremity Movement by Cyclograms

Human gait is nowadays undergoing extensive analysis. Predictions of leg movements can be used for orthosis and prosthesis programming, and also for rehabilitation. Our work focuses on predicting human gait with the use of angle-angle diagrams, also called cyclograms. In conjunction with artiﬁcial intelligence, cyclograms oﬀer a wide area of medical applications. We have identiﬁed cyclogram characteristics such as the slope and the area of the cyclogram for a neural network learning algorithm. Neural networks learned by cyclograms oﬀer wide applications in prosthesis control systems.


Introduction
In medical practice, there is no appropriate widelyused application of a system based on artificial intelligence (AI) for identifying defects in the movement of human legs, or for controlling the actuators of prosthesis or rehabilitation facilities.Above all, it is difficult to evaluate the quality of walking.Medical decisions are often based only on the subjective views of physicians, and appropriate accurate methods and are not widely used in clinical practice.
Several methods can be used in medical practice and in physiotherapeutic research for identifying defects in the movement of the human body.The most widely-used method for studying gait behavior in clinical practice is gait phase analysis by gait time phase cycles [1,2].The time phase cycles of gait behavior have been used to analyze gait with the application of artificial intelligence methods [3][4][5][6][7][8][9], but the findings have not subsequently been applied in medical practice.Very intensive research is now being done on predicting leg movements by artificial intelligence and EMG signal measurements [10][11][12], mainly with a view to their possible application in myoelectrical prosthesis control systems.
For a study of gait we have used new methods based on an analysis of gait angles using cyclograms (also called angle-angle diagrams or cyclokinograms) and artificial intelligence to predict human motion of legs/prostheses.The concept of angle-angle diagrams, although known to the biomechanics community, has not been mentioned much in the recent literature.The first mention of the cyclogram [13] argued that a cyclic process such as walking is better understood if studied with a cyclic plot, e.g. an angle-angle diagram.The creation of cyclograms is based on gait angles that are objective, reliable and well suited for statistical study [14].This technique is strongly rooted in geometry, and the quantities are intuitively understandable [15].Depending on the cyclicity of the gait, cyclograms are closed trajectories generated by simultaneously plotting two (or more) joint quantities.In gait studies, most attention has traditionally been given to easily identifiable planar knee-hip cyclograms .In order to quantify the symmetry of human walking, we have obtained and studied a cyclogram for the same joint and two sides of the body [16].
Applications of cyclograms in conjunction with artificial intelligence can offer a wide range of medical applications, but this approach has not yet been studied or applied in practice.

Methods
To create and study angle-angle diagrams, we use a model of the human body created in MatLab Simulink and SimMechanic software.The movement of the model of a body is controlled by data measured by the motion capture system, which identifies the position of points/markers in the Cartesian coordinate system.We can generally use several methods for measuring movements in two/three dimensional space, e.g. an infrared (IR) camera with active markers, or a web camera, which is cheaper.For our application, we used an IR medical camera with active markers (Lukotronic AS 200 system) and placed LED diode markers on the following points on the person being measured: fifth metatarsophalangeal joint, malleolus lateralis, epicondylus lateralis, trochanter major, spina iliaca anterior superior, etc., Figure 1.
Using this method we can record the movement in a three-dimensional space, though we primarily study the movement in a two-dimensional sagittal plane.Human gait data commonly consists of the recorded positions of markers on the skin/dress at the extremities of the limb segments (the thigh, the shank, etc.) of a subject.If we have information on the movement of points/markers in space, and the points characterize parts of the human body, we can use these points to define the vectors of the positions of the body parts.The difference of the coordinates of two points in space defines a vector.The angles between each two segments are calculated by assuming the segments to be idealized rigid bodies.For the computing angles, we use the formula: for the two-dimensional system, because we assume that this knee or ankle joint has only one rotational degree of freedom in the case of simplified assumptions [17].u and v are vectors of body segments (thigh, shin, foot, etc.) represented by at least two points, i.e. markers.The calculation is performed in an equivalent way for three-dimensional space.If we are interested in the angle between a body segment and a physical horizontal we determine the angle between the horizontal vector (1, 0) and vector that is defined by the coordinates of the points on the body segments evaluated in the Cartesian coordinate system.The markers, i.e. the points, move in space together with the body segments, and the individual segments of the body move by the translational or angular movement per unit time.We therefore use numerical derivations to determine the translational and rotational speed and the acceleration of the individual segments, i.e. markers The aim of this study is not to evaluate the center of rotation of a joint, because even in practice, in the case of the motor control system of a prosthetic device, we do not expect complete conformity with the anatomical characteristics, and the deviations are negligible for our purpose, which is to provide verification of movement prediction methods.For the calculation we therefore used vectors based on the coordinates of the markers, and vector u and v in the equation does not necessarily correspond to the directional unit vectors of the body parts.
During the measurement, the measured data is grouped according to the age groups of the subjects, and could also be grouped according to the diagnoses of the subjects.We create cyclograms of the angles, angular velocities and angular acceleration: left knee -left hip, right knee -right hip, left hip -right hip, left knee -right knee, right ankle -right hip, left ankle -left hip, etc.We were also able to study the following characteristics of cyclograms: length of the trajectory, frequency of the loops, slope of the loops, maximum range, average speed, total circumscribed area of the loops.
The most important part of our work is on designing methods for applying cyclograms in practice in order to identify movement defects, and to apply cyclograms in the control system of the actuators of prosthesis or rehabilitation facilities.For this purpose, we use artificial intelligence methods which are implemented in MatLab toolboxes [18,19].For example, we can use Artificial Neural Networks (ANN) to predict the joint angles, i.e. for predicting cyclograms.
Artificial neural networks are based on the neural structure of the brain [20,21].They process records one at a time, and "learn" by comparing their prediction of the record with the known actual record [22,23].The input to the first layer consists of the values in a data record.The final layer is the output layer, where there is one node for each physical quantity.The prediction of time series using a neural network consists of teaching the net the history of the variable in a selected limited time and applying the taught information to the future.Data from the past is provided to the inputs of the neural network, and we expect future data from the outputs of the network.
Table 1: Learning process of artificial neural network according to the calculated joint angles

Input data
Target data Joint angles Joint angle Our learning method is based on the premise of the proposal of a table with m + 1 columns of states.We assume that five columns of previous states plus one column for the prediction will be sufficient, Table 1.The first row of the table records the first five angles computed from the measured data, and the sixth column indicates the next calculated value of the angle, as a target to which the artificial neural network learns by example.The second row records the second to sixth calculated value of the angle, and in the sixth column in the same row we insert the seventh value of the angle, as a target.This cascading method fills a table of n-4 rows, where n + 1 is the number of known values of the joint angles that we decided to use for the learning process, Table 1.Thus the method is generally based on information about a man walking.Walking is described by the cyclogram, and the cyclogram is used to learn neural networks.We chose the calculated values of the joint angles as an approach for selecting data for artificial neural network learning.For neural network learning we use part of the curve of the cyclogram that represents a set of states for learning.These states are divided into past states and prospective states.With each presentation, the output of the neural network was compared to the desired output, and the error was computed.This error was then fed back (backpropagated) to the neural network, and was used to adjust the weights in such a way that the error decreases with each iteration and the neural model gets closer to producing the desired output.
The table of input and target data can be extended to other parameters that are also very important for predicting the movement of the lower limbs.Appropriate parameters are, for example, the angular acceleration (i.e.four states of acceleration), and also the subject's weight in kilograms and age in years, Table 2.We tested several modifications to ANNs.The first designed and tested ANNs were for predicting the angle in only one joint of the left leg (hip, knee and ankle).We also designed ANNs for predicting the complete knee-hip cyclogram or the ankle-knee cyclogram, i.e. for predicting two angles.An ANN structure that we designed for predicting the angles of the hip-knee curve of a cyclogram is presented in Figure 2.
For learning the NN, we can also add the height of the subject in meters, or other additional parameters, e.g. a predefined code for an illness, operation of the musculoskeletal apparatus, etc.The backpropagation algorithm is used for training the neural networks.With backpropagation, the input data was repeatedly presented to the neural network.The above-described methods for predicting motion using learned neural networks reflect only a short part of the curve of the angle-angle diagram of a gait cycle.The values by which the neural network is learned reflect information on only a limited number of states within a short period of time.This method may not always adequately describe the stereotypes of human walking.The main problem is that the method does not describe the whole gait cycle and the transition from the current gait cycle into a new gait cycle.It is theoretically possible to design a number of input neurons of a neural network to describe the whole gait cycle, but the structure of the neural network will then be very complex, and the calculation will therefore be difficult and slow.For this reason, we proposed new ways to describe the gait cycle by characteristics of cyclogram.
The proposed method takes the distribution of the values less into account, but to a greater extent reflects the geometrical shape of the area A of one cycle.We can use several methods and we decided to use area moment of inertia to describe the property of a two-dimensional plane shape of the cyclogram.The reason for using second moment is that we can very easily describe the distribution of the circumscribed area of a cyclogram and we can determine the inclination angle θ of diagram.The value of angle θ, which is given a product moment of zero, is equal to: I xx is the second moment of the area about the xaxis, I xx is the second moment of the area about the y-axis and I xy is the product moment of area.The θ is the inclination angle between axes of the original coordinate system of the diagram and the principal axes of the area.The values of the inclination angle or the moments of the area can be used for learning the neural network.The neural network will be extended for neurons with regard to other input values, e.g. the inclination angle of one cycle before the actual value of the joint angles.The neural network will also be extended for new output neurons, so that the neural network learns by target values, e.g. the inclination angle of a subsequent one cycle after the actual value of the joint angles, Table 3.The patterns in Table 4 are used for learning the neural network using second moments and for predicting the one joint angle.For learning the ANNs, including the variables mentioned above, we can theoretically also use the area of one angle-angle diagram A, the area of the ellipse of inertia, the center of the area, etc. Description of the cyclogram by area, area moment and inclination angle has great advantage because calculated characteristics of the cyclo- the a 0 and a 1 are parameters identified by estimator.When y is the dependent variable of the first angle, e.g.knee angle, and the x is the independent variable of the second angle, e.g.hip angle, the direction (slope) of the principal axis, i.e. inclination angle θ of angle-angle diagram, is obtained from the parameter a 1 as follows tan θ = a 1 .
The trained neural networks prefer the typical changes from the previous cycle to the subsequent cycle, and avoid the use of atypical changes.In addition, we allow NN to estimate the expected gait cycle based on a specified slope of the angle-angle diagram of a gait cycle.Method based on the moment of inertia is proposed for training neural networks on the basis of individual data for a particular person.However, the method can be modified to take into account anthropometric data such as the weight, height or age of the subject, and the neural network can be expanded to include the appropriate number of input neurons, and can thus become universal.
The main object of our study was to predict the trajectory of cyclogram curves of on the basis of the current state of the lower extremities, and to make the prediction with the use of artificial intelligence.The set of data for training artificial neural networks was measured on 10 volunteers recruited from students of the Czech Technical University in Prague.The subjects were asked to walk properly on a treadmill at a variable walking speed.The main human walking speed was 1.5 km/h for studying and adjusting the proposed method.
We used the very accurate Lukotronic AS200 motion capture system to record the data.The IR camera with active markers was manufactured by Lutz Mechatronic Technology e.U.According to the CE-certification for medical products, EU-Directive 93/42/EEC, the manufacturer declares that the Lukotronic motion analysis system can be used for patient care in hospitals and rehabilitation centers.One camera system mounted in front of or behind the subject moving on the treadmill recorded the 3D motion of the lower limbs.Markers were placed in accordance with the manufacturer's recommendations for gait analysis by GaitLab software.The recommended marker set model is the same as the set defined by the Helen Hayes Hospital model [26] for Vicon Clinical Manager and the sagittal plane.Physicians defined the location of the markers by the Helen Hayes marker set model.The position of the markers on the foot was chosen mainly according to the requirements for a good record, because the movement of the feet is not usually measured in practice, and the manufacturer does not mention placing more than one marker on the foot.
After obtaining the measured data, we created the necessary cyclograms in MATLAB software.We obtained graphs of the changes of angles in all the main points in the lower part of human body and the sagittal plane.This was important for subsequent computation of the curve of the cyclograms and for predicting the motion.We used cyclograms as models/patterns for training the artificial neural networks.After training the neural networks we used them to predict the future states of the angles in the joints of the lower limbs.

Results
Our measured data indicates that the angle in the knee usually changes from 5 • (stretched leg) to 70 • (shrugged leg).In the hip, the angle is usually from 5 • to 40 • , Figure 3, and in the ankle the angle is usually from 60 • to 100 • , Figure 6.Under the assumption of neglecting inaccurate placement of the markers, and other simplifying assumptions, we can state that the angles correspond to the angle between the femur and the tibia and between the tibia and the metatarsus.The cyclogram in Figure 3 shows that the swing phase typically starts at a thigh extension angle of 0 • and knee flexion of about 80 % of the maximum.The subject weighed 65 kilograms and was 23 years old.After training the neural networks, we used parts of the cyclograms for predicting the future states of the joint angles.The result was that by using short sections of the cyclogram curve, which was loaded into a neural network, the trained neural network predicted the subsequent behavior of the gait by the predicted cyclogram curve.The two results of our proposed method for gait prediction are shown by the predicted knee-hip cyclogram (Figure 4) and the predicted ankle-knee cyclogram (Figure 7).For the prediction, we used NN learning without taking into account the inclination angles of the diagrams.The two-dimensional knee-hip and ankle-knee cyclogram shows the prediction cyclogram curves based only on measurements of past states, i.e. the remaining part of the curve is predicted by assuming knowledge of a short curve segment.We can see that the predicted curves correspond only partially to the usual form of cyclograms [13][14][15][16].The 11-7-2 artificial neural network, Figure 2, was used for predicting the cyclograms (Figure 4 and Figure 7).The results show that the prediction is inaccurate, especially in the ankleknee cyclogram, because of its complicated shape.The predicted knee-hip cyclogram is a relatively accurate, and is similar to the pattern, Figure 3.
We found that predictions of movement based only on an evaluation of a small section of the cyclogram are not suitable for predicting complex movements.In the case of gait, which often changes, the prediction is not always appropriate and accurate.Improved predictions are achieved by increasing the monitored section of the curve of the cyclogram, but the complexity of the neural network and the computing time also increase.These features are undesirable, and we had to modify the neural networks and the methods of learning.For ANN training and for the prediction, we used the variables identified by the moment of inertia method.The predicted kneehip cyclogram (Figure 5) and the predicted ankleknee cyclogram (Figure 8) show that the prediction is very accurate.We identified slight variability in the prediction of the angles, but the variability was negligible, because there are small variations in angles even in typical gait.This way of making predictions is suitable for situations where the movements often use stereotypes and typical gait changes between these stereotypes of human walking.
According to the method described here, the cyclograms inform us about the previous position of the lower limbs, and we can infer the expected conditions, i.e. the states during future predicted walking.This information is important in rehabilitation medicine, and also as an expected value of the angles used in control algorithms for lower limb prostheses.Cyclograms in conjunction with artificial intelligence are broadly applicable in medicine.We have described a method for predicting the motion of the lower extremities, and this predicted data can be used for evaluating human gait in physiotherapeutic practice, based on a study of angle-angle diagrams.Predictions of the joint angles of the leg are based on the principles of artificial intelligence.In our study we have used artificial backpropagation neural networks.By analogy with the analytical method based on NNs for a study of two-dimensional cyclograms, we designed and tested an methods based on NNs for a study of three-dimensional knee-hip-ankle cyclograms.The new methods can be applied in clinical practice for studies of disorders or characteristics in the motion function of the human body [27], and the method can be used in advanced control systems for controlled prostheses of the lower extremities [28].In the past, it was almost impossible to use complex algorithms based on artificial intelligence in the slow control systems of a controlled prosthesis, but today we can consider applying the methods described here in the algorithms for new prosthetic knee control systems [29,30].There is an obvious opportunity to continue in this research, and to use these methods to study and design a new hydraulic or pneumatic knee prosthesis.
A new robotic orthosis offers a second very important possibility of applications in rehabilitation medicine.We can use the proposed methods in algorithms for a driven robotic gait orthosis for the purposes of locomotion therapy [31,32].The therapy is based on simulating the movement of the lower limbs of healthy people by sophisticated robotic devices.An example of such a system is the Hocoma Lokomat system, which supports the rehabilitation of patients suffering from neurological diseases (multiple sclerosis, post-stroke), patients after spinal cord injury or after a traumatic brain injury resulting in partial loss of the ability to walk.The Lokomat has been on the market since 2001, and has crucially improved the art and science of locomotion therapy at the Motol University Hospital in Prague.We assume that our method will be applied in the control systems of such a device, because artificial intelligence applied in control systems could extend the possibilities of rehabilitation i.e. training possibilities.In addition, the identified and predicted gait pattern could be individually adjusted to the patient's needs [33,34], because our movement prediction method also takes into account the patient's weight and age.Moreover, the proposed new methods for identifying the technique of human walking, which is used for training neural networks, can be modified and used in other areas of artificial intelligence, such as reinforcement learning [35,36].
This work has not attempted to describe all potential ways of applying cyclograms in conjunction with artificial intelligence.We have shown new methods that have subsequently been proved by a number of simulations in MATLAB software.These methods based on cyclograms and ANNs could be suitable for a broad range of applications.

Fig. 1 :
Fig. 1: Location of IR markers and angles measured during the examination

Fig. 5 :Fig. 6 :
Fig. 5: Predicted knee-hip cyclogram (NN training, taking into account the inclination angles of the diagrams): predicted values of the angles -circular symbols; measured known values -cross symbols

Fig. 7 :
Fig. 7: Predicted ankle-knee cyclogram (NN training, not taking into account the inclination angles of the diagrams): predicted values of the angles -circular symbols; measured known values -cross symbols

Table 2 :
Learning process of artificial neural networks according to the angles and angular accelerations calculated in a joint, subject's weight and subject's age

Table 3 :
Learning process of artificial neural networks according to the joint angles and inclination angles of the angleangle diagram

Table 4 :
Learning process of artificial neural networks according to the joint angle, the moments of the area and the inclination angles of the one angle-angle diagram