HYBRID REASONING FOR MULTI-ROBOT DRILL PLANNING IN OPEN-PIT MINES

A BSTRACT . Fleet automation often involves solving several, strongly correlated sub-problems, including task allocation, motion planning, and coordination. Solutions need to account for very speciﬁc, domain-dependent constraints. In addition, several aspects of the overall ﬂeet management problem become known only online. We propose a method for solving the ﬂeet-management problem grounded on a heuristically-guided search in the space of mutually feasible solutions to sub-problems. We focus on a mining application which requires online contingency handling and accommodating many domain-speciﬁc constraints. As contingencies occur, efﬁcient reasoning is performed to adjust the plan online for the entire ﬂeet.


INTRODUCTION
Autonomous vehicles are becoming key components in industrial automation.State-of-the-art methods for localization, mapping, control and motion planning have enabled the development and deployment of autonomous vehicles in logistics, material handling, and mining application.Yet important hindrances remain when it comes to employing fleets of autonomous vehicles [1].This paper is concerned with three of these hindrances: • Fleet automation often involves solving several, strongly correlated sub-problems -among these, allocating tasks to vehicles, planning vehicle motions, and vehicle coordination.This makes solving the overall problem very hard, as solutions should be found in the space of mutually-feasible solutions to sub-problems.
• Solutions often need to account for very specific, domain-dependent constraints.Thus, even if methods exist for solving individual sub-problems, these often need to be adapted to reflect the nuances of the particular domain.
• Several aspects of the overall fleet management problem become known only online.This makes it necessary to compute at least parts of the solution to the overall problem during execution and/or to adapt existing plans to contingencies.
In this paper, we propose a method to divide the overall fleet-management problem into sub-problems.We apply a general method for searching for a mutually feasible solution to the sub-problems of the overall problem.We focus on a mining application where fleets are composed of surface drilling machines.The aim is to plan and coordinate blast-hole pattern drilling with multiple drilling machines.Solutions consist of an executable plan for multiple vehicles operating concurrently within a common area of the open-pit mine, called a bench (see Figure 1).Several aspects of the problem become known only online: the duration of hole-drilling actions depends on the hardness of the rock, which is unknown at planning time; the durations of navigation actions between targets are also unknown at planning time, as they depend on the actual state of the terrain; and unplanned stops may occur due to the fact that drills may get stuck and need to be replaced.All three forms of contingencies are modeled as metric temporal constraints, and are posted online to a common representation that maintains the state of execution of the plan for the entire fleet.As temporal constraints become known during execution, efficient temporal reasoning is performed to adjust the plan and to provide an optimistic Time to Completion (TTC) of the overall plan for the entire fleet.In Section 2 we define the Drill Pattern Planning Problem and state the specific requirements of the application.Section 3 details how the problem is divided into sub-problems, and Section 4 describes the algorithm used to find mutually feasible solutions to the individual subproblems.Section 5 describes the domain-specific heuristics that capture the nuances of this particular mining application and how they can be easily plugged into the search for a solution.Section 6 is dedicated to the online aspects of the approach.A preliminary experimental evaluation is provided in Section 7, which evaluates the feasibility

PROBLEM DEFINITION AND REQUIREMENTS
We focus on a problem in a mining application, where a fleet of surface drills operates on a bench in an open-pit mine.A set of drill targets in the bench is given; at each target, a blast hole is to be drilled.The blast holes are then filled with explosive material that will be detonated after all targets have been drilled.After the explosion, the ore is taken away and processed for mineral extraction.
For each drill target, a machine can autonomously carry out a set of tasks: auto-tramming (navigating to the target from its current position), leveling (deploying jacks for horizontally leveling the machine), drilling, and de-leveling (retracting the jacks so the machine is placed back on its tracks).Each drilling machine has a square dust guard around its drill bit.Drilling produces piles of excess material around the hole, thus one side of the dust guard must be lifted after drilling; this allows the machine to navigate to the next target without colliding with the pile that has accumulated under it after drilling (see Figure 2).The Drill Pattern Planning Problem (DP 3 ) consists of computing a plan that involves machines reaching each drill target in a bench and performing the necessary operations to drill the blast hole.The plan is subject to the following requirements: • Machines should not collide with obstacles/each other; • Drilling a target leads to the creation of a pile under the dust guard of the machine; this pile constitutes an obstacle, hence no machine can drive over it; • Once drilling has been performed, the machine can only drive away from the target (backing up from the target pose) by raising the forward dust guard (the only part of the dust guard that can be actuated); • Motions should be executable by the machines i.e., motions should be kinematic feasible; • It should be possible to modify the plan online; • The plan should respect spatial constraints such as a virtual fence within which the vehicles are allowed to operate; this area is called a geofence.
The locations of drill targets and the geofence are given.These are based on a geological survey of the area and on the current production targets of the mine (see Figure 3).A set of machines R is also given, and the size of a fleet is based on the size of the bench.Also given are initial poses of all machines, as well as their desired final "parking" poses.The requirements above pose several problems e.g., task allocation (of machines to target poses), motion planning, and coordination.These problems cannot be treated separately, as the solutions of each problem depend on each other.For instance, coordination must lead to a sequence of target poses that accounts for the piles generated after drilling (which become obstacles that must be taken into account in motion planning).Hence, it is necessary to subject the possible choices made to solve one problem to the choices made in resolving the other problems -e.g., verifying through motion planning that a chosen sequence of targets to drill will be kinematically feasible.Because of these interdependencies, we face a hybrid reasoning problem.We propose an approach in which the overall problem is divided into sub-problems, and the solution to the overall problem is searched for in the joint search space of these sub-problems.In the next section, we define each sub-problem in detail, while in Section 4 we outline the algorithm used to search for plans that are mutually feasible with respect to all the sub-problems.

APPROACH
We divide the DP 3 into five sub-problems.
• The sequencing sub-problem consists of deciding a total ordering of targets i.e., sequencing every pair of targets.
• The moition planning sub-problem consists of deciding the pose of drilling machines at each target.Constraints on the orientation of the machine in certain target poses may be given (e.g., due to the presence of the geofence or other geographical constraints like cliffs and walls).The decisions are subject to kinematic constraints, obstacles and geofence, and must account for piles resulting from drilling, as well as the dust guard mechanism.
• The machine allocation sub-problem consists of allocating machines to targets given the available machines and their positions.Machine allocation also accounts for the need to reach a given end parking position.

DEMUR'15
http://robotics.fel.cvut.cz/demur15/Hybrid Reasoning for Multi-Robot Drill Planning in Open-Pit Mines • The coordination sub-problem consists of scheduling machines.Solutions to this sub-problem consider spatiotemporal overlap between machines and between machines and piles.
• The temporal sub-problem consists of deciding when machines should carry out motion, drilling, leveling and de-leveling operations, subject to temporal constraints arising from coordination, sequencing, maximum achievable speeds, etc.
A solution to DP 3 is obtained by reasoning upon these five different sub-problems jointly.Candidate solutions for a sub-problem are validated by dedicated solvers.Each solver focuses on a subset of aspects of the overall problem, e.g., a motion planner verifies kinematic feasibility and absence of collisions, while a temporal solver verifies that coordination choices are temporally feasible.Validated solutions for each sub-problem can be see as constraints that account for particular aspects of the overall problem.They are maintained in a common representation, which is sufficiently expressive to model the search space of all problems jointly.The common representation is a constraint network where variables represent missions.A mission is a tuple M = (gp, sp, r, P, m, S, T, A), where r represents the robot which should perform the set of activities A = {drilling, leveling, de-leveling} at, respectively, starting pose sp and goal pose gp.P is the path that r traverses to reach gp from sp, and is computed based on a map m of the environment.S is a set of polygons representing sweeps of the robot's footprint over P , and T is a set of time intervals representing when r will be in each polygon contained in S. Henceforth, we denote with M (•) an element of the mission tuple.
Let M be the set of all missions in the DP 3 (one for each drill target).A solution to the DP 3 is such that a value is decided for all elements of a mission, for each mission in M ∈ M. Each element is decided by solving one or more sub-problems.We view a mission M as a variable in a Constraint Satisfaction Problem (CSP, see [2]) whose domain represents all possible combinations of values that can be given to each element of M .Accordingly, we view a sub-problem as the problem of constraining the domains of missions so the requirements stated above are met.Hence, the solvers that validate solutions to the subproblems are seen as procedures that post constraints to the common constraint network.As we will see, adopting the CSP metaphor allows us to employ heuristic search strategies for solving the overall DP 3 .

SEQUENCING SUB-PROBLEM
The sequencing sub-problem consists of finding a total order of missions.A decision variable of this sub-problem is a mission M i ∈ M that does not have a preceding mission.A possible value that can be assigned to this decision variable is a precedence constraint M j precedes −−−−→ M i , asserting that mission M j ∈ M should occur before M i .M j is a mission for which it has not been already decided that it precedes another mission.A sequencing solver verifies that missions are totally ordered.Figure 4

MOTION PLANNING SUB-PROBLEM
The motion planning sub-problem consists of finding a goal pose gp for each mission M i ∈ M. A gp is a tuple x, y, θ in which x and y represent the position of a drill target, and θ is the orientation of the machine.The decision variables of the motion planning sub-problem are missions M i such that (1) M i (gp) does not have a defined orientation, i.e., θ is not assigned to an angle, (2) there exists M j precedes −−−−→ M i in the common constraint network, and (3) M j (gp) has been assigned an orientation.Possible values that can be assigned to a decision variable are a set of eight angles {θ 1 , . . ., θ 8 } ∈ [0, 2π).
A particular choice of approach angle for a mission is only feasible if the machine can drive away from the previous target M j (gp) and can navigate to the end pose of a mission M i (gp) considering piles created by all the preceding missions.For example, Figure 4(b) shows a selection of one feasible approach angle for several missions with respect to existing sequencing constraints.The approach angles are represented by pink arrows, and the machines drive away from the targets in the opposite direction of the pink arrows.
The eight possible assignments to the decision variable determine eight different possible end poses of the machine {M i (gp 1 ), . . ., M i (gp 8 )}, which differ only in the orientation of the machine in the goal pose.A possible assignment θ k is validated through a path planner, which is given the triple where the start pose is the goal pose of the preceding mission (M i (sp) = M j (gp)), and M i (m) is a map of the environment that contains obstacles and a geofence.The obstacles correspond to circular shapes centered in the goal poses of the preceding missions.Through the map M i (m), the path planner accounts for targets that have already been drilled prior to mission M i .
Since the path planner is invoked potentially several times while solving the motion planning sub-problem, computational efficiency is crucial.We employ a path planner for a car-like mobile robot based on cubic spirals [3].This path planner computes paths consisting of curvature-constrained curves constituted by few cubic spirals and straight lines.The output of the path planner is either fail, which indicates that a particular approach angle θ k cannot be achieved, or the spline M i (P ), representing a kinematically-feasible and obstacle-free motion from M i (sp) to M i (gp k ).

MACHINE ALLOCATION SUB-PROBLEM
In this sub-problem, a decision variable is a set M ⊆ M such that ∀M ∈ M , M (r) has not been decided and that is, a total order of missions has been decided, but machines have not been allocated.The values are com- plete assignments of robots to missions, i.e., an assignment M (r) = R for each missions in M .
Clearly, the machine allocation sub-problem has a huge space of possible solutions.Each possible solution has complex ramifications on other sub-problems: different allocations will affect the amount of coordination necessary; allocations must be such that the final mission of a machine is not surrounded by piles (drilled by other machines), which would make it impossible to navigate to its final parking pose.As we show in Section 5, heuristics with high pruning power are needed to explore the search space of this sub-problem, and these heuristics must account for other sub-problems.Solutions to the machine allocation sub-problem are indirectly validated in other problems, hence no particular solver is used for direct validation of possible values.

TEMPORAL SUB-PROBLEM
A mission's path P is segmented into a sequence of subpaths based on its curvature.Each segment is associated to a convex polygon s k resulting from the sweep of the machine's footprint along the sub-path.The resulting sequence {s 1 , . . ., s m } of convex polygons represents the areas occupied by a robot while navigating along the path (see an example in Figure 4(c)).Since the path planner used to obtain P is aware of the obstacles created by preceding missions, the convex polygons do not intersect the piles resulting from drilling (see Figure 4(d), where red circles represent piles, white polygons the motions of the machine, and the green line represents the geofence).
In addition to the polygons representing the motion of machines, the activities involved in a mission M (i.e., M (A) = {drilling, leveling, de-leveling}) have polygons associated to them.Since these activities all occur while the machine is idle in pose M (gp), their polygons coincide with the polygon that covers the last sub-path of M (P ).Hence, the set of all convex polygons of mission M is M (S) = {s 1 , . . ., s m }∪{s drilling , s leveling , s de-leveling }, where s drilling = s leveling = s The temporal sub-problem consists of deciding a start and end time for each interval I k .The temporal subproblem has a decision variable for every machine R ∈ R. Each decision variable is a set of missions M ⊆ M such that for all M i ∈ M , (1) M i (P ) has been decided, (2) M i (r) = R, and (3) the start and end times of M i (T ) have not been decided.We reduce the problem of deciding valid start/end times of the intervals to a Simple Temporal Problem (STP, [4]).The STP is formulated as a collection of temporal constraints as follows.First, for each M i ∈ M with intervals M i (T ) = {I 1 , . . ., I m+3 }, temporal constraints that reflect the order of the convex polygons along the path M i (P ) are imposed: Second, temporal constraints that force the possible start and end times of missions to adhere to the ordering decided by the sequencing solver are added.That is, for each pair of missions (M i , M j ) ∈ M × M that are subject to a sequencing constraint M i precedes −−−−→ M j , the following temporal constraint is added among the intervals M i (T ) = {I i 1 , . . ., I i m+3 } and M j (T ) = {I j 1 , . . ., I j m+3 }: reflecting the fact that the de-leveling polygon of mission M i occurs before the first motion of mission M j .Third, we impose minimum durations of the motions and activities of the machines: For every motion polygon s k , k ∈ {1, . . ., m}, an initial value for α is computed based on the maximum allowed velocity of machine R. Hence, the earliest time solution of the STP represents the fastest possible execution of all motions and activities in the plan, i.e., an "optimistic" estimate of the start and end times of all operations of all machines.During execution, further temporal constraints can be added to reflect contingencies such as machine maintenance, delays, and so on.The association of an interval per polygon allows us to predicate via temporal constraints on how long every movement or activity will take.Solving the STP is polynomial in the number of intervals in the temporal problem, namely Mi∈M |M i (T )| [4].
Note that there are no temporal constraints among intervals pertaining to different machines, hence the motions and activities of different machines may be concurrent.a consequence of decisions made in all previous subproblems, the common constraint network includes polygons, temporal intervals, and temporal constraints (eqs.( 1) to ( 3)) among them.The STP solver computes start and end times for each interval.This determines when machines will occupy motion and activity polygons in the various missions.If two polygons pertaining to different vehicles overlap, and their corresponding temporal intervals intersect, then the two vehicles may collide.Coordination avoids this by imposing additional constraints that eliminate temporal intersection where needed.Decision variables of the coordination sub-problem are pairs of polygons and intervals represented by quadruple (s i k , s j m , I i k , I j m ), of mission i and j respectively, that overlap both spatially and temporally, i.e., The value of a decision variable is one of two pos- , imposing either of which eliminates the temporal overlap between concurrent polygons.The STP solver will validate the sequencing in time of these two overlapping polygons accordingly.It will also compute the consequent shift in the occurrence of any other polygon whose interval is constrained with I j m or I i k by means of temporal constraint propagation within the common constraint network.We use a similar approach for the coordination of multiple vehicles as described in [1]. Figure 5 depicts the situation where missions of two machines are temporally and spatially overlapping.Polygons with the same color belong to one machine and for clarity, we omit the time intervals of polygons in the visualization.
The polygons involved in the decision variables represent two types of occupancy.The first type corresponds to the motions of machines as described in 3.4.The second corresponds to the piles created by drilling.By modeling both types of polygons in the common constraint network, collisions among machines and with piles are found and thus scheduled.Figure 6 illustrates a conflicting situation between the motions of a machine (depicted as yellow polygons) and a pile (depicted as a blue polygon).

BACKTRACKING SEARCH
The collection of decision variables for each sub-problem mentioned above constitutes a high-level CSP (henceforth called meta-CSP).Search in the meta-CSP consists in finding an assignment of values to decision variables that represent high-level requirements.Each of these requirements is, in our case, a sub-problem.Possible values among which these assignments are selected are verified by a specific solver for each sub-problem.Thanks to the common representation of the search space, each sub-problem solver accounts for the assignments made for decision variables of other sub-problems.For example, the path planner validates with respect to a map containing obstacles resulting from sequencing decisions; and the coordinator's decisions depend on the machine allocation as well as motion plans.The choices of values for decision variables in the various sub-problems contribute parts of the missions in the common representation, and the sub-problem solvers propagate the consequence of these decisions.
The sub-problem solvers used in our approach are denoted in the following with solve-p, where p ∈ {seq, alloc, time, coord, mp}.As we have explained, solve-seq disallows sequencing decisions that are not totally ordered; solve-mp verifies by means of a motion planner that motions are kinematically feasible and obstacle-free; solve-alloc accepts all candidate allocations, as the infeasible ones are discovered indirectly via coordination; solve-time is a STP solver which computes feasible start/end times of mission intervals subject to temporal constraints; solve-coord is also provided by the same STP solver, which validates and computes the consequences of temporal ordering decisions.
We use a CSP-style heuristically guided backtracking search to find values to assign to the decision variables.Henceforth, let the set of sub-problems be indicated by Function DP 3 -solver(M): success or failure ).Among the decision variables of a sub-problem, one is chosen according to a variable ordering heuristic h var i (line 4).For example, which target should be selected first among the decision variables D mp of the motion planning sub-problem.Among possible alternative values, one is chosen according to a value ordering heuristic h val i (lines 5-7).For instance, which approach angle has to be selected for a given target in the motion planning sub-problem.This value is added to the common representation (line 8).The sub-problem solver solve-p verifies that the assignment v is feasible.If so, DP 3 -solver is called recursively (line 10), which results in selecting another unassigned variable subject to the newly updated common representation M. Note that if all possible values are attempted for a decision variable d and all are rejected by solve-p, the algorithm returns failure (lines 6, 11-13).In the next section, we will explain the problem-, variable-and value-ordering heuristics that are used in DP 3 -solver.

HEURISTICS
DP 3 -solver must select a set of decision variables pertaining to a sub-problem from the union of all decision variables.This selection is guided by a heuristic h prob .Let D i ≺ D j indicate that the decision variables of problem i have a higher priority than those of problem j.The partial ordering based on which the h prob heuristic operates is Decision variables to branch on (within a chosen D i ) are ordered based on h var , and alternative values are chosen according to h val .
Variable ordering heuristics are provided for the sequencing sub-problem and for the coordination sub-problem.The latter heuristic is based on temporal flexibility and has been used for resource-constrained project scheduling [5].The former is based on an analysis of the drill target placements, and is described below.
Variable Ordering for Sequencing h var seq .The pattern of drill targets is analysed to reveal its topology and the possible principal directions of drill target sequencing (see Figure 7).To determine the former, we use a distance threshold; the latter are discovered via a K-Means clustering of the set of angular coefficients of topologically neighbouring drill targets.This yields clusters containing similarly oriented edges of the topology.These are used to group drill targets into roughly-parallel lines (see Figure 7).The topology and the groupings are used to rank drill targets in groups.Variable in these groups are first in the sequencing sub-problems.Value ordering heuristics are defined for the sequencing, allocation, motion planning, and coordination subproblems.As for variable ordering, the decision variable in the coordination sub-problem are branched upon using a heuristic based on temporal flexibility that is widely used in the scheduling literature [1].The remaining heuristics h val seq , h val alloc are explained below.
Value Ordering for Sequencing h val seq .A value for a decision variable of the sequencing sub-problem decides which drill target precedes a given target.There are many alternatives for this choice.Note that sequencing decision variable that are resolved first are those pertaining to drill targets along groups -for such decision variables, the heuristic prioritizes one of two possible predecessors, namely those adjacent to the current decision variable in the grouping.For example, the two values with highest heuristic score for decision variable M 141 in Figure 4( Also, this heuristic contributes to alleviating the computational burden of finding sequences in regions close to the geofence while transitioning between groups.Finding a feasible sequence in these situations is challenging because a machine has limited space to manoeuvre.These regions of highly-constrained motion span typically eight targets for every pair of adjacent groups, thus in the worst case sequencing requires verifying, through motion planning, 8 7 possible motions for each pair.For this reason, the heuristic uses given sequence patterns that reflect common practice by human operators.A sequence pattern FIGURE 8.An example of sequence pattern given by an operator.is a topological description of a human driving behavior, augmented with metric information that facilitates assessing whether the pattern is applicable in a given region.Specifically, a sequence pattern is a graph (V, E) where V is a set of nodes representing drill targets, and E is a sequence of precedence constraints among the nodes.A distance threshold d geofence is also given, and represents the minimum distance to the geofence required for the pattern to hold.Also, a ranking < of the nodes in terms of how far they lie from the geofence is provided.An example pattern is shown in Figure 8.If the search is considering a decision variable M i that is surrounded by targets that can be mapped to the nodes in the pattern, then the heuristic ranks possible predecessors of M i according to the edges in E.

Value Ordering for Motion Planning h val
mp .This heuristic suggests approach angles similar to those assigned to other drill targets in a same group.

DEMUR'15
http://robotics.fel.cvut.cz/demur15/Hybrid Reasoning for Multi-Robot Drill Planning in Open-Pit Mines Value Ordering for Machine Allocation h val alloc .A solution to the machine allocation sub-problem determines which drill target is drilled by which machine.Among all possible choices, those are preferred which have three properties: (1) each machine is assigned to a contiguous sequence; (2) start and end missions of a contiguous sequences are the drill targets close to an open area, i.e., not close to a geofence and not those that are entirely surrounded by other drill targets; and (3) targets are evenly distributed among machines.This heuristic not only contributes to the plan quality in terms of similarity to what a human planner would decide, but also improves the efficiency in planning time by suggesting a restriction on start and exit points for each machine.

ADAPTING SOLUTIONS ONLINE
Several aspects of the DP 3 are unknown at planning time.For example, the actual durations of activities only become apparent during execution.In a bench, various types of contingencies may occur, such as unexpected maintenance of machines, or increased drilling time due to unknown geological characteristics of the terrain.Therefore, we need to monitor the execution and reflect the contingencies in the common representation.In our approach, the nominal behavior of the machines is given by a solution of the DP 3 , obtained via Algorithm DP 3 -solver.The start and end times associated to the intervals M (T ) of every mission M are computed through temporal propagation.All the lower bounds represent the earliest possible times at which missions can be executed, and are used to compute the desired speeds at which the computed paths should be driven by the vehicle executives.A machine executive realizes the interface between machine controllers and the missions in the plan by instructing the machine controller to follow the given trajectories1 .It also updates the time intervals M (T ) of the current mission by posting into the common representation constraints representing the current progress of the machine.These constraints are used by the STP solver to propagate any mismatch between prescribed and executed missions of all machines in the fleet.
The STP solver plays a central role in execution monitoring.Machine executives update the common representation at a frequency of 1Hz to dispatch or end missions.A mission in ended by adding a temporal constraint into the common constraint network representing the finish time of the mission as the executive layer informs.The consequences of such updates can be easily computed within the period of one second because the STP solver performs polynomial inference.Also, due to the fact that adding constraints cannot "undo" other decisions, we can post unforeseen durations (e.g., encountering hard rock while drilling) at execution time.More precisely, the prolongation of an activity represented by a flexible time interval associated to a motion polygon cannot affect the sequencing, the approach angle, the particular motions, nor machine allocations.It only bears consequences on the coordination sub-problem, as delays may need to be propagated to other waiting machines.An example of this situation is described in the next Section.Note that if we want to minimize the TTC, then, prolongation of an activity should lead to the re-allocation of the machines, which in turn would result in updating the decisions in the sequencing sub-problems.This allows for re-balancing the workload among the machines.Considering the wide range of replanning strategies that can be used to minimize TTC is a topic for future work.
On-line temporal reasoning also caters to another important requirement of mining companies, namely the need to know an estimate of the Total Time to Completion (TTC).At planning time, we provide an optimistic TTC by initializing the duration constraints 3 with reasonable values: the durations of intervals corresponding to motion polygons are computed using the maximum allowed speed of machines; and the intervals corresponding to activity polygons are initialized with durations under nominal conditions (average rock density, and no maintenance).As execution proceeds, TTC is updated as a result of temporal reasoning to reflect the actual situation.

EXPERIMENTS AND EVALUATIONS
We carried out several experiments in different selections of benches and drill targets.In this section, we exemplify one of the most challenging ones.The resulting plan was then run by a Gazebo-simulated fleet of Pitviper-311s, and all interfaces between the DP 3 -solver and the platforms were realized as ROS [7] nodes.The DP 3 -solver ran on a 3.40GHz×4 Intel i7-3770M CPU with 8GB of memory.The difficulty of the example originates from the closeness of some targets to the geofence.It is also affected by the average distance between targets, which is 16 meters (only 1.8 meters larger than the length of the Pitviper-311).The drill target positions are taken from a real blast hole pattern recently drilled in an open-pit iron-ore mine in Western Australia.As noted earlier, problems become much more difficult if they contain drill targets that are close to the geofence, whereas when that is not the case problems are easier, since machines have enough space to manoeuvre regardless of how approach angles are chosen.This experiment contains 76 drill targets and 3 available machines.Figure 9 shows the final solution to the overall DP 3 for this specific bench.In the figure, robots and groups used by h var seq and h val seq are numbered to facilitate the explanation.As shown, machine 1 starts with row 1 and exits the field from row 2, machine 2 starts at row 3 and exits at row 4, and machine 3 starts its missions from row 6 and exits from row 5.The plan is found in 2.14 minutes.
Figure 10 shows three snapshots of a simulated run.Each snapshot shows the motions of each robot (the collection of all movement polygons).In Figure 10(a), the motions of robot 1 partly occupy row 3 and 4 for manoeuvring between row 1 and 2. This results in robot 2 having to wait just before the conflicting area, until robot 1 finishes its manoeuvre.As robot 2 also occupies some part of http://robotics.fel.cvut.cz/demur15/DEMUR'15 The fact that the motions of machine 2 do not conflict with row 1 and 2 is handled by the motion planner; the same holds for machine 3 and rows 3 and 4 (see Figure 10(c)).As machines start their execution concurrently, their motions would lead to collisions, were it not for the fact that the coordination sub-problem was solved as well.The temporal constraints that were selected by the algorithm resolve these conflicts by forcing machines 2 and 3 to yield to machine 1.
The plan obtained with Algorithm DP 3 -solver was run in simulation three more times.During the first run, we artificially injected delays in the drilling activities of robot 1 for a drill target in row 1.Through temporal reasoning, the delay was propagated on the start times of future missions of machines 2 and 3.The TTC drastically increased as a result, as robot 2 (and consequently robot 3) were forced to wait until robot 1 finished manoeuvring between rows 1 and 2. In the second run, we artificially delayed robot 1 while drilling a target in the second row.In this case, TTC only increased by the amount of the delay, since the delay does not affect robot 2 and robot 3. Finally, in the third run, we injected a delay in one of the initial drill targets of robots 2 and 3. Since these robots were scheduled to yield to robot 1 later on during execution, the delay did not contribute to increasing the TTC.In all three runs, the contingencies were accounted for by temporal reasoning and the resulting adjusted plan dispatched to the machines within the 1 second sampling period.We also evaluate the effectiveness of the sequence pat-  tern (sp, see Section 5) used by h val seq on a problems defined over the same drill pattern as used in the previous experiments.We consider three cases with a different number of drill targets (27, 53 and 76).Each case is run twice, once with and once without the sp heuristic.The search process is aborted after a 60 minute timeout.Table 1 shows the strong pruning power of this heuristic.
In the next experiment, the focus is to evaluate the TTC in a slightly bigger setup with 91 drill targets (see Figure 11).The TTC of the solution where machine 1 starts at row 1 and exits at row 4 and machine 2 starts at row 5 and exits at row 8 is 101 minutes, whereas the TTC is 61 min in the case that machine 1 starts at row 4 and exits at row 1 and machine 2 starts at row 5 and exits at row 8.In the latter case, machines do not need to wait for each other and all the operations can be done in parallel.In order to extract the latter solution from the search space, an ad-hoc sequencing heuristic built for this particular set of drill targets was employed.This proves that this solution is in the search space.As mentioned, devising a general heuristic that biases sequencing choices to minimize TTC is the topic of future work.Our approach consists in exploring the joint search space of different sub-problems via a heuristically informed CSPstyle backtracking search.The intelligence in our approach is distributed among several heuristics, each guiding the resolution of specific sub-problems.An obvious alternative approach is to define the state of the DP 3 as node in a traditional heuristic search, and to employ A* (or some other heuristic search algorithm) in conjunction with a heuristic that accounts for all aspects of the DP 3 .We would, however, have to build all aspects of the problem into this single heuristic -that is, we would have to make the heuristic capable of informing which are the best allocations of vehicles to targets, the best sequencing, grouping and scheduling decisions, etc.More importantly, it would DEMUR'15 http://robotics.fel.cvut.cz/demur15/Hybrid Reasoning for Multi-Robot Drill Planning in Open-Pit Mines be difficult to exploit the structure of the different subproblems to enhance performance.In our approach, scalability can be improved by identifying heuristics for specific sub-problems and/or heuristics that exploit the hybrid nature of the underlying representation.Another important factor in industrial domains is the ability to specify domaindependent requirements, e.g., in this specific bench, that drill targets 122 and 344 must be drilled at the end, or that machine 3 should not drill a set of drill targets.The modularity of our approach facilitates programming these types of requirements in the relevant decision variables of specific sub-problems.

RELATED WORK AND OUTLOOK
We divide the DP 3 into several sub-problems and the DP 3 -solver finds mutually feasible solutions to the individual sub-problems.The idea of problem decomposition and solution synthesis is not new in AI.There have been many studies in Multi Agent Systems (MAS) regarding cooperative problem solving [8].Although our work has not been done in the context of MAS, we reify this idea into a challenging robotic domain.
The DP -solver combines solutions from different sub-problems, each of which can be seen as a "classical" AI or robotics problem.The sequencing sub-problem can be seen as a Vehicle Routing Problem with Multiple Depots (MDVRP) where the drill targets are the customers.MDVRP is a combinatorial optimization which has been largely studied.However, solutions to MDVRPs often do not consider spatial, temporal and kinematic constraints, nor dynamic maps.The need to consider these problems destroys the assumptions based on which existing AI solutions are built.As this comprehensive survey describes [9] there are various modeling to the MDVRP including time windows, split delivery, heterogeneous fleet according to the single and multiple objectives.For future work, our main focus is to optimize the TTC while all the requirements are upheld, therefore, casting the sequencing subproblem to MDVRP is an option for further investigation.
Combined route and motion planning in the presence of strong spatial and temporal constraints has been studied [10].Although the application differs, the proposed approach is similar to ours in that it combines solutions from different sub-problems such as non-trivial motion planning and route planning.However, the solution assumes that sequencing of goal poses is given and does not handle on-line contingencies.
The coordination problem has to be addressed when we have a fleet of autonomous vehicles.Many approaches to this problem largely rely on fixed trajectories (e.g., [11] and the KIVA system [12,13]).This makes the coordination problem much easier than in our case, where vehicle paths are not known a priori.The coordination sub-problem could be addressed by using multi-robot motion planners, using a distributed approach [14] or solving the problem in a centralized fashion [15].The latter is similar to our work in terms of using a centralized approach.However, multirobot motion planners are not efficient enough for use within another search, and are unable to handle temporal contingencies that may occurred on-line.
In this work, we break down a given hybrid problem and identify interdependency among the sub-problems, and interleave reasoning within each sub-problem.A heuristically guided backtracking search finds a solution to the overall problem in the joint search spaces of these subproblems.This approach is general and can be used in other domains, such as task planning for mobile service robots [16] or warehouse management [17].
In addition to focus to the optimization issue, another aspect of our ongoing work is to broaden the range of possible online contingencies (e.g., machine breakdowns which require removing a machine) that can be dealt with by our DP 3 -solver.

FIGURE 1 .
FIGURE 1. Two AtlasCopco drilling machines (Pitviper-351) in the process of drilling targets in a bench.

FIGURE 2 .
FIGURE 2. The actuated dust guard and leveling jacks are highlighted on an Atlas Copco Pitviper-271.After drilling, piles of excess material accumulate under the dust guard, which requires the machine to actuate the dust guard before navigating away from the target.

FIGURE 3 .
FIGURE 3. A bench with drill targets (grey circles) and a geofence (green polygon).
(a) shows an example of decision in this sub-problem, namely M 166 precedes −−−−→ M 137 .White arrows in the figure represent precedence constraints.

FIGURE 4 .
FIGURE 4. Examples of different sub-problems with their decision variables and particular choices of values.
de-leveling = s m .Each convex polygon in M (S) is associated to a time interval in the set M (T ) = {I 1 , . . ., I m+3 }.Interval I k = [I s , I e ] is a flexible temporal interval within which robot M (r) is in s k , where I s = [l s , u s ], I e = [l e , u e ], l s/e , u s/e ∈ N represent, respectively, an interval of admissibility of the start and end times of the occurrence of polygon s k ∈ S.

FIGURE 5 .
FIGURE 5. Coordinating two machines to avoid spatiotemporal overlap.

FIGURE 6 .
FIGURE 6. Scheduling a machine to avoid spatio-temporal overlap with a pile {seq, alloc, time, coord, mp}.Given the set of missions M, Algorithm DP 3 -solver collects all the decision variables belonging to all the sub-problems (line 1), and terminates when no decision variables are left (lines 2 and 14).A particular sub-problem is then chosen according to a sub-problem ranking heuristic h prob (line 3), e.g., h prob prioritizes machine allocation decision variables over coordination decision variables, as the latter problem requires machines to be assigned to missions (see Sec-http://robotics.fel.cvut.cz/demur15/

FIGURE 7 .
FIGURE 7.An example of topology and group extraction.

FIGURE 9 .
FIGURE 9. A solution to the DP 3 problem in this bench which is visualized in the ROS visualization tool, white arrows depict sequencing, pink arrows represent vehicle poses at each drill target, and the green line shows a part of the geofence.

FIGURE 10 .
FIGURE 10.Motions of different robots represented on a separate figure as white convex polygons

FIGURE 11 .
FIGURE 11.An high quality solution with respect to the TTC for a bench with 91 drill targets

TABLE 1 .
A small quantitative evaluation h val seq