A MYOELECTRIC PROSTHETIC ARM CONTROLLED BY A SENSOR-ACTUATOR LOOP

This paper describes new methods and systems designed for application in upper extremity prostheses. An artificial upper limb with this system is a robot arm controlled by EMG signals and a set of sensors. The new multi-sensor system is based on ultrasonic sensors, infrared sensors, Hall-effect sensors, a CO2 sensor and a relative humidity sensor. The multi-sensor system is used to update a 3D map of objects in the robot’s environment, or it directly sends information about the environment to the control system of the myoelectric arm. Occupancy grid mapping is used to build a 3D map of the robot’s environment. The multi-sensor system can identify the distance of objects in 3D space, and the information from the system is used in a 3D map to identify potential collisions or a potentially dangerous environment, which could damage the prosthesis or the user. Information from the sensors and from the 3D map is evaluated using a fuzzy expert system. The control system of the myoelectric prosthetic arm can choose an adequate reaction on the basis of information from the fuzzy expert system. The systems and methods were designed and verified using MatLab/Simulink. They are aimed for use as assistive technology for disabled people.


Introduction
Present-day control systems for robotic arms (RA) are based primarily on a point-to-point method [1], [2].Sensor systems designed for industrial manipulators use CCTV or a set of basic sensors (limit switches, optical sensors, etc.).An EMG signal is used for controlling an upper limb prosthesis.This RA is referred to as a myoelectric prosthesis.The sensors implemented in the myoelectric prosthesis are tactile sensors used for correcting the touch and the handgrip, an optical sensor with analog memory for motion detection [3], digital incremental encoders used in feedback for the PID controller, etc., or complex gyroscopic systems and/or inertia systems for information about movement acceleration, movement speed and movement direction [4], [5].The feedback information from the sensors is important for correct control using a PID controller.However, none of these sensors have been used in a commercially manufactured myoelectric prosthesis.Present-day commercially manufactured myoelectric prostheses use only direct control (i.e., forward kinematics) by the processed EMG signal, without additional information about the state of the RA and its environment.Commercially manufactured myoelectric prostheses do not use an additional signal that would provide feedback information about the position of the artificial RA in 3D space.This means that an EMG signal from one muscle is used to control one direction of rotation of one actuator [6][7][8][9].The EMG signals are scanned from the remaining mus-cles.However, it is difficult for the wearer to learn unnatural muscle contractions in order to wield more than three DoFs of the RA.For this reason, commercially manufactured prostheses are equipped with up to three actuators.Advanced RAs controlled by EMG are equipped with sensors that identify the state of each joint actuator [10], [11], but these sensors do not provide information about the robot's environment [12], [13].Some authors believe that a more complex sensory system may reduce the cognitive load on the wearer by taking away low level control, and an intelligent arm control system would remove the need to micromanage each action [14].In order to deal with the lack of information for a complex control system, we have to design a new multi-sensor system and control method based on creating environment maps.

Methods
We designed systems and methods for controlling the myoelectric RA based on a sensor-actuator loop, Fig. 1.The USER block is a proband with a myoelectric prosthetic arm.The EMG Signal Preprocessing Block creates the machine code for the Control Signal Generator Block.The Control Unit Block uses the control signal and information from the sensors and the 3D map, and creates sequences and a power signal for the RA actuators placed in the Robotic Arm Block.The Sensory Subsystem Block (SSB) is also installed on the RA.It is evident that the Robotic Arm Block and the SSB are mechanically interconnected.The Map Constructor Block (MCB) is used for object identification and for identifying the distances of objects from the RA.The Signal Comparator Block compares the control signals from the Control Signal Generator Block with a map of the environment from the MCB.The SSB consists of the sensors for monitoring the RA and its environment.Signals from the SSB are processed into maps of objects in the environment.The Signal Comparator and the Control Unit can differentiate between safe and dangerous objects and environments, and help to choose an adequate reaction.An artificial limb with the designed control system could be a semiautomatic robot, which does not need direct control by the patient.The proposed systems and methods are tested and installed on the myoelectric RA, which is controlled by MatLab (MathWorks, Inc.) software [15].The mechanical design of the RA is based on the AL5D robotic kit (Lynxmotion, Inc.) and seven electric rotary actuators (servomotors) to allow one translational motion (hand: 1DoF) and six rotational motions (wrist: 2DoF, elbow 1DoF, shoulder: 3DoF).

Preprocessing the EMG signal
EMG sensors with electrodes are used for recording and data preprocessing of the EMG signal.The signal is filtered and the EMG signal envelopes are calculated [16,17].The envelope is processed in accordance with the control method that is used.This can be based on forward kinematics (FK) or on inverse kinematics (IK).If the RA is controlled by the FK method, the EMG signal envelope is adjusted so that the angle or the angular velocity of the actuator is proportional to the magnitude of the envelope.The position control of the actuators using the IK method [15,18] is based on knowledge of the distance of the displacement or the translational velocity of the arm endpoint (in the defined 3D coordinate system).The distance of the displacement is proportional to the magnitude of the envelope.As in cases when FK control is applied, the envelope carries information about the angular velocity of the joint and the direction of rotation of the actuators [15].Two methods are used for controlling the actuators: a method with two levels of magnitude of the envelope, and a method with two levels of the slope of the envelope, Fig. 2. In both cases, the threshold value (1st Level) of the minimum magnitude of the envelope is used to control the start/stop of the actuators, Fig. 2. The application of the two methods will be described using FK.
The EMG signal envelope has various magnitudes (i.e., intensity), Fig. 2. The control unit uses the variable magnitude of the envelope.The expected range of magnitude of the EMG signal envelope is divided into two levels.The proband controls the actuators by varying the intensity of muscle activation.Low muscle contraction intensity creates an envelope of small magnitude, and high muscle contraction intensity creates an envelope of high magnitude [6].The intensity of the muscle contraction also determines the direction of rotation of the joint actuator (flexion, extension, etc.).The joint actuator performs the extension in a joint (i.e., the actuator at the elbow) after a pre-defined threshold value is exceeded, i.e., a defined level of magnitude of the envelope (2nd Level, Fig. 2).The joint actuator performs the flexion in a joint if the magnitude of the envelope is smaller than a pre-defined threshold value (2nd Level, Fig. 2).The instantaneous value for the magnitude of the envelope within the range defined by the pre-defined threshold values determines the angular velocity.
The second method is based on the level of the slope of the envelope.If the rate of the patient's muscle contraction is higher, the slope of the envelope of the EMG signal is steeper, and vice versa.The value for the magnitude of the slope of the envelope deter- mines the direction of rotation of the joint actuator.Again, the instantaneous value of the magnitude of the envelope determines the angular velocity.The joint actuator performs the extension in a joint (i.e., the actuator at the elbow) after a pre-defined level of the slope of the envelope has been exceeded.The joint actuator performs flexion in a joint if the magnitude of the slope of the envelope is smaller than a pre-defined level of the slope, Fig. 2.

Sensory subsystem
The important part of the sensor-actuator loop is the SSB, which is based on various types of sensors to identify potential collisions or a potentially dangerous environment.The most important sensors are ultrasonic range sensors (URS) for identifying objects in the surroundings.The sensory system is therefore based mainly on the set of ultrasonic sensors for distance measurement [15,[19][20][21].The set of sensors is composed of nine URSs (SRF05 type, Devantech Ltd.) and nine thermophile array sensors (TPAS) (TPA81 type, Devantech Ltd.).The sensors (i.e., nine URS and nine TPAS) are divided into three blocks: The Manus (hand) block contains 1xURS and 1xTPAS sensor (scanning the space before the hand); the Antebrachium (forearm) block contains 4xURS and 4xTPAS (scanning the space around the forearm); and the Brachium (upper arm) block contains 4xURS and 4xTPAS (scanning the space around the upper arm).In the Antebrachium and Brachium blocks, four pair of sensors (URS and TPAS) are placed around the perimeter of each segment.The scanned planes of Antebrachium and Brachium are perpendicular to each other, i.e., the sensors are oriented with the axes in the superior-inferior and anterior-posterior directions in the case of neutral-zero position of the joint angles (i.e., the so-called neutral posture of a healthy human subject).The 6-DoF myoelectric prosthetic arm is complemented by additional sensors near its endpoint, Fig. 3.The main reason for using other types of sensors is to identify a dangerous environment, which could damage the prosthesis or the user.The additional set of sensors is composed of two mutually perpendicular Hall-effect sensors (SS94A1, Honeywell Inc.), one CO 2 sensor (MG811, Hanwei Electronics Ltd.) and one relative humidity sensor (HS1101LF, Measurement Specialties Inc.)The Hall-effect sensors are used to measure the magnetic field strength in order to identify a potentially dangerous environment in which the electronic components of the prosthesis can be negatively affected.The carbon dioxide sensor (CO 2 sensor) is used for measuring the carbon dioxide gas concentration in order to identify a potentially dangerous environment in which the perception of the disabled patient could be negatively affected.The CO 2 sensor is used as a representative of the sensors for measuring air quality.The humidity sensor is used to measure the relative humidity and to identify a potentially dangerous amount of water vapour in the air, due to which the electronic components of the prosthesis might be damaged.

Real-time mapping of the environment
Sensory readings are sent to a PC via the RS232 bus and are then evaluated in MatLab software.A 3D map of the environment is gradually built up on the basis of these readings.The most important sensors used here are the URSs to avoid a collision between the arm and the surrounding objects.A characteristic of URSs that needs to be taken into account, is their wide beamwidth [22].An obstacle may be found anywhere in the cone at the measured range [23].However, the greatest probability of occurrence of an object is directly opposite to the sensor (on the axis of the sensor), and the probability decreases towards the edge of the cone.A quadratic model of URS according to Elfes [24] is therefore used.This model estimates the distribution of the occupancy probability from one URS reading.The model is visualized in Fig. 4. The x and y axes represent spatial coordinates in 2D space (in a real application, it is applied to a 3D space in a similar manner) and the z axis represents the probability of occurrence of an object.The plane at a value of 0.5 (Fig. 4) represents an unexplored space.
Values below this level stand for an empty space, while values above this level represent a higher probability of occurrence of an object.
To build a map, we decided to use occupancy grid mapping, which was first introduced by Elfes [24,25].In general, this method is based on decomposition of the proximate environment of RA into a 3D array of cubic cells.Each cell represents a section of space at given coordinates, and it holds the values of parameters describing this section.The most important information for RA navigation is the probability that the cell is occupied by some object.URSs are able to localize obstacles around the RA and to return values of their distance from the URSs.These are used to decide which cells are probably occupied and which are probably not occupied.A probabilistic model is essential, because there are many uncertainties that may occur [23].For example, there may be errors in the sensory readings, the beamwidth of the sensors, false reflections, etc.
As for the mapping itself, a Cartesian coordinate system x, y, z has been defined in the 3D space around the RA with an origin [0, 0, 0] located at the point of the shoulder joint (articulatio humeri), Fig. 3.A cell is a cube with dimensions 1 × 1 × 1 cm.This cell size was chosen as a compromise between the high resolution, the computational complexity and the accuracy of URSs.The map is therefore a discrete function of spatial coordinates Map(x, y, z).At the start of the mapping procedure, the map is a 3-dimensional matrix with each element set to 0.5 [26].This means that the states of the cells are unknown.After initialization of the SSB, the map is updated using sensory data.Using FK or IK, the exact locations of the URSs in the map, and also their orientation, is determined.This information is used to determine which cells are located within the cone of the ultrasonic beam and, therefore, their values have to be updated.The quadratic model (Fig. 4) is applied to the data in order to get the states of the cells based on sensory readings.The values stored in the cells are then updated by the Bayesian formula [26,27]: where Map(x, y, z) represents an updated value of the cell in position [x, y, z] after integration of a new sensory reading into the map, Map(x, y, z) represents the previous value, and is the probability that the cell is occupied (OCC) according to sensor range reading r, i.e., the sensor model (Fig. 4).It is essential to combine data from several measurements [24].At each time step, we obtain sensory readings from all URSs.A viable method for integrating data from independent measurements makes the map more accurate with each measurement.
In this way, we obtain the precise location of objects in relation to the articulatio humeri and arm segments with sensors.

Fuzzy expert system
A Fuzzy Expert System (FES) is designed to convert the measured data from SSB and/or MCB, and to identify potentially dangerous situations (PDS).The FES is used to interpret data from the multi-sensor, and, on the basis of sets of if-then rules, to infer an output.Expert knowledge based on information about the sensor characteristics and dimensions of the RA was used for setting up the FES implemented in Mat-Lab toolboxes [28].With fuzzy logic we are able to use natural language and words, instead of equations and numbers, [29].The possibility of a PDS can be described linguistically as: low, medium or high (see Table 1).A fuzzy if-then rule assumes the form: if x is A then y is B, where A and B are linguistic variables defined by fuzzy sets on universes of discourse X and Y, respectively (Tab. 1 and Tab. 2).The x can be distance, temperature, magnetic field, CO 2 concentration or relative humidity, and the y is the possibility of a PDS, Tab. 1.The if-then rules are derived from experimental findings and knowledge indicating the relation between the measured data from the sensors and the PDS.Fuzzy sets and membership functions of the FES are also designed [15].Thus, each variable is described by fuzzy sets.The output variable from the FES is the possibility (i.e., probability) of the PDS (having values in the interval from 0 to 1), Table 2.
The designed FES interprets an input, and, on the basis of sets of if-then rules, infers an output.The FES has five input variables and one output variable, Table 1.The Fuzzy Toolbox and the Mamdani Fuzzy Inference System (FIS) in MatLab were used to create the FED described here.Mamdani FIS is the best known and the most widely-used system in developing and applying fuzzy models [15].The designed FES uses PROD (product) for implication, SUM (simply the sum of each rule's output set) and BISECTOR for defuzzification.Each group of sensors has one FES [15].If the identified value of the PDS is more than 0.8, the control system with the sensor-actuator loop stops the motion of the RA, or a specific segment of the RA, and the proband is informed by a warning signal.Information from the 3D map is used to stop the whole RA, and the information from the specific sensor is used to stop the specific segment with the sensor

Results
Tests were carried out on the applicability of the control systems based on: level of magnitude of the EMG signal envelope, the level on the slope of the envelope, SSB, real-time mapping of the environment, and FES.The use of the level of magnitude of the EMG signal envelope and/or the level of the slope of the envelope is clear and easily applicable [15].Both methods are able to control the direction and the angular velocity of the rotation of the joint actuator by the intensity of the muscle contraction, i.e., the magnitude of the EMG signal envelope or the slope of the envelope of the EMG signal.The methods are computationally equivalent and equally complex, because the main parts (rectifications, filtering, etc.) of the signal processing were the same.
A more important task is to verify the SSB with the FES and the 3D map, because the SSB is the basis of the control of the myoelectric RA using the sensoractuator loop.The most important sensors used here are URSs to avoid a collision between the RA and objects in the surroundings.An experiment was performed to verify the functionality of the real-time mapping of the environment by SSB.The RA was in neutral-zero position, and specific objects were placed in its proximity.The data from, for example, the antebrachium (forearm) block contains four URSs sensors were sent to program using MatLab.The values of the measured ranges were subsequently processed and the software updated the 3D map.The output of the program is a 3D array of the probabilities that the cells of the occupancy grid are occupied.This can be visualized using MatLab software, preferably as a planar slice of the 3D space.Fig. 5. shows an example of a planar slice of the 3D map made in the plane of the four URS sensors.Fig. 5. shows a planar slice of the 3D map, which was updated several times while the RA was in the neutral-zero position.The cones of the four antebrachial sensors are clearly visible.The white cells (pixels) represent an empty space, and the black cells represent an occupied space.The gray area is an unexplored space.An upper cone (Fig. 5) arises from one URS sensor.Object A (Fig. 5) was standing in front of the sensor during the first several time steps (i.e., records), and then left.It is evident that there is no longer any obstacle in the mapped distance, because the cone goes out of the map.Only a gray strip can be seen in the distance, where object A had been standing.In the following time steps, it will turn white.The cone on the left (Fig. 5) arises from URSs that measured an object at a distance of 30 cm (Object B).The cone on the right arises from URSs that measured an object at a distance of 10 cm (Object C).The last cone (heading down) arises from a sensor that was heading towards an object with a rugged surface (Object D) at a distance of approximately 50 cm.There are noticeable errors from the rugged surface.In this particular case, the detection of the distance of an object with a rugged surface is not very accurate.In the other cases, with each sensory reading (i.e., each time step), the accuracy of the detected position is improved.Information about the distances of the objects is used to identify the PDS.Thus, the information from the 3D map is an input value for the FES.
Based on the above, FES is used to identify the PDS.If the PDS (greater than 0.8) is identified by the FES, the sensor-actuator loop of the myoelectric RA with the control system stops the motion of the RA or of a specific segment.The designed FES is sufficient to identify the possibility of the PDS.The one-input and one-output of FES can be depicted in a 2-D plot.The plot in Fig. 6 represents the identification of the PDS by one-input (distance of an object from the RA).

Discussion
The results of first experiments using the new methods and systems of the myoelectric RA based on a sensoractuator loop seem to be promising.The methods allow control of the angular velocity and the direction of rotation of the actuators by the EMG signal.The sensor system can identify objects in 3D space and their basic characteristics.The sensor system also provides information for the MCB with the 3D map for real-time mapping of the environment of RA.This map is used for identifying proximate objects.Thus, the map provides basic information about the objects near the RA, and this is used to avoid collisions with the objects.The FES interprets an input, and on the basis of sets of if-then rules it infers an output, i.e., PDS.
The SSB uses twenty-one sensors with a sample time of 1 second and less.This means that the software must work with more than 1 kB of data from the sensors.One second is a very long period for real-time working and for a quick reaction from the controlled the myoelectric RA.A suitable sample period is between 50-200 µs.When the system uses sample time 50 µs, the computer must work with 250 kB data per second.
The functionality of the real-time mapping of the environment has also been verified.A disadvantage of the mapping method used here is its computational complexity.Within each time step, the program has to go through the 3D array and update each cell in the cones of the URSs.Thus, this action may take several hundreds of milliseconds, or even seconds, depending on the number of cells in the 3D map.If the multisensor system is to be able to identify the possible collisions of RA with objects in the surroundings, the time needed for the computations has to be reduced at least to the order of tens of milliseconds for one time step.This may be done simply by reducing the number of cells in the map.One option is to choose an optimal maximum distance that will be mapped.This distance should be long enough to see objects with which the RA may potentially collide.Our estimate is that a distance of 60 cm may be enough to predict dangerous situations.This distance is also taken into account in the design of the FES.A second option is to choose an optimal resolution.We used a 1 × 1 × 1 cm cell.

Conclusion
The methods and systems presented here are original contributions to the investigation of new prosthesis control methods.The myoelectric RA based on a sensor-actuator loop uses methods to control the angular velocity and the direction of rotation of the actuators and to prevent PDSs that could damage the prosthesis or harm the user.The designed SSB can identify objects in 3D space and their basic characteristics.The system proposed in this paper also provides a 3D map of the environment of RA, based on real-time mapping of the environment by SSB.This map can be used for identifying proximate objects and the position of the arm in relation to these objects.The map provides basic information about obstacles near the RA.In the case of the expert system, we used fuzzy logic because the methods of artificial intelligence implemented in MATLAB offer great potential in control and signal processing.The FES is used to avoid PDSs, especially collisions with objects in the surroundings.
So far, the mapping algorithm uses only data from URS sensors.Future work can involve integrating data from TPASs.Then, the cells in the occupancy grid will be able to hold two values -occupancy probability and temperature.It may be useful to predict the PDS of touching very hot objects that could harm the person or damage the prosthesis.The semiautomatic myoelectric RA should be able to retain a safe distance from hot objects in order to prevent such situations.The next step in our work will be to test and adjust the FES by using new expert knowledge.
We have described methods for solving problems in the control of a myoelectric RA or a myoelectric prosthesis.All computational methods are implemented in MatLab software, and the data from the sensors is also processed in MatLab software.The proposed methods and systems are designed as an assistive technology for disabled people, and are to be understood as a contribution to the international debate on the design of new myoelectric prostheses and myoelectric RAs.

Figure 1 .
Figure 1.Block diagram of the standard control method for the myoelectric arm (I.) and a diagram of the new control method for a semiautomatic myoelectric arm (II.)

Figure 2 .
Figure 2. Method of two levels of magnitude of the EMG signal envelope (I.) and method of level of the slope of the envelope of the EMG signal (II.).

Figure 4 .
Figure 4. Quadratic model of an ultrasonic sensor visualized in MATLAB.

Figure 5 .
Figure 5. Example of a planar slice of the 3D map made in the plane of the antebrachium block (I.) and the experimental setting of the arm (II.).

Figure 6 .
Figure 6.Example of the 2-D plot of one input (Distance of an object from the RA) and one output (Potentially Dangerous Situation) of FES.

Table 1 .
General if-then rules to determine the possibility of a potentially dangerous situation.

Table 2 .
Fuzzy sets of variables, linguistic expressions and parameters for defining a triangular (a, b and c) membership function.