Acta Polytechnica

In this article, a specific production scheduling problem (PSP), the Parallel Machine Scheduling Problem (PMSP) with Job and Machine Sequence Setup Times, Due Dates and Maintenance Times is presented. In this article after the introduction and literature review the mathematical model of the Parallel Machines Scheduling Problem with Job and Machine Sequence Setup Times, Due Dates and Maintenance Times is presented. After that the Monte Carlo Tree Search and Simulated Annealing are detailed. Our representation technique and its evaluation are also introduced. After that, the efficiency of the algorithms is tested with benchmark data, which result, that algorithms are suitable for solving production scheduling problems. In this article, after the literature review, a suitable mathematical model is presented. The problem is solved with a specific Monte Carlo Tree Search (MCTS) algorithm, which uses a neighbourhood search method (2-opt). In the article, we present the efficiency of our Iterative Monte Carlo Tree Search (IMCTS) algorithm on randomly generated data sets.


Introduction
A cost-efficient production is one of the main goals of manufacturing companies, because production efficiency means higher profits for the company. A cost-efficient production means that as many jobs as possible are processed on time at the lowest cost. In this article, a specific production scheduling problem, the Parallel Machine Scheduling Problem with Job and Machine Sequence Setup Times, Due Dates and Maintenance Times is presented. In this specific problem, m jobs must be distributed among n machines. Setup time means transition time between jobs. The due date means the day by which the job has to be done. Maintenance time means a time window when the production stops because there is maintenance. In our problem, the objective function is the minimization of the setup times and maximization of the number of created jobs. The Production Scheduling Problems are NP hard, therefore, it is necessary to apply some heuristics to their solution. With heuristics, of course, there is little chance of finding a global optimum, but we can get a good local optimum within a reasonable amount of time. In this article, a specific Monte Carlo Tree Search (MCTS) algorithm is applied to the problem.
The paper is organized as follows: after the literature review, the mathematical model of our specific production scheduling problem is described. In section 4, the Monte Carlo Tree Search is detailed. In section 5, the representation and evaluation of Parallel Machine Scheduling Problem with Job and Machine Sequence Setup Times, Due Dates and Maintenance Times are presented. In section 5, our Iterative Monte Carlo Tree Search algorithm is also detailed. After that, the test results are discussed. In section 7, the conclusion and remarks are presented.

Literature review
Production Scheduling Problem (PSP) [1,2] is a common problem in manufacturing systems. The problem lies in scheduling n jobs between m machines. Over time, many types of PSP have evolved. In the following paragraph, some types of production scheduling problems are presented based on the literature.
In the case of Parallel Machine Scheduling Problem (PMSP) [3], m jobs must be distributed among n machines. Fuzzy Parallel Machine Problem [4] has some fuzzy parameter (setup time, capacity constraint, time windows etc.). Machine eligibility restriction [5] states if a machine is capable of processing a job (operation). In the case of production scheduling, the machines can be identical [6] or uniform [7]. The job can have due date [8], release time [9] and time window [10]. The due date means the day by which the job has to be done. Release date means that the job needs to be started at least in this predetermined time. The time window is the combination of the release time and due date. Jobs with priority level [11] should be created as soon as possible. The job processing time can be not only fixed but also variable [12]. The PMSP [9] can have several objective functions, for example, the minimization of the setup times [13], the minimization of the total earliness and tardiness time [14] or objective function, which takes into account environmental impacts (for example green scheduling problem) [7]. The constraints and components can  Table 2. The components of the jobs also be used together, for example, in article [15] the authors also use release and due dates, family set-ups, etc.

Parallel Machine Scheduling Problem with Job and Machine Sequence Dependent Setup Times, Due Dates and Maintenance Times
In this specific problem, m jobs must be distributed among n machines. Setup time means the transition time between jobs. The due date means the day by which the job has to be done. Maintenance time means a time window when the production stops because there is maintenance. In our problem, the objective function is the minimization of the setup times, and maximization of the number of created jobs. In the following, a simple example and its solution are presented, after that, the mathematical model of our problem is detailed. Table 1 illustrates the components of the machines, Table 2 presents the components of the jobs, while in Table 3, the setup times of Machine 1 can be seen.

The mathematical model
Assume that our problem contains n jobs and m machines. Each job has m * (n + 1) setup times because the setup time is job-and machine-dependent. Each  Table 4. The objective function is the minimization of the setup times: where Z 1 means the job-job setup time: and Z 2 means the machine-job setup time: Constraint 1: Each job can be assigned to one machine: Constraint 2: Production continuity: Constraint 3: Due date constraint: Constraint 4: Maintenance times: x i,j1,j0 = 1j 1 = 1...n∀j 0 = 1...n (8) occurs in at least one case vol.
no. / Parallel machine scheduling with Monte Carlo Tree Search

Monte Carlo Tree Search
The Monte Carlo Tree Search (MCTS) algorithm creates a tree structure to solve a specific problem. During the algorithm, the search tree grows gradually and asymmetrically. The algorithm starts at the root node and repeatedly picks a random child as a successor until it reaches a leaf node. The MCTS guides the search towards better candidate successors. It uses a balance between exploration and exploitation. The exploration means that the algorithm needs randomness to explore the search space. The exploitation means that the MCTS takes an advantage of the best option we know. The algorithm repeats the following four phases [16]: (1.) Selection: The algorithm starts from the root node and recursively chooses the best child based on the UCB formula until a leaf node is reached. This phase can be seen in Figure 2.
(2.) Expansion: When a leaf is reached, the algorithm further expands, unless there is no possible leaf. This phase can be seen in Figure 3.  values from the result. This phase can be seen in Figure 5.
The iteration of these four steps is called simulation.
The UCB formula provides a balance between exploration and exploitation:  where X j is the average win-rate (or fitness) of the j-th child, C is the UCB constant, most of the time C = 2. N j is the visit count of a child j, and N is the visit count of the parent.
Algorithm 1 presents the base Monte Carlo Tree Search technique. The Tree Policy means selecting or creating a leaf node from the nodes already contained within the search tree (i.e., the selection and expansion process). The Default Policy means a simulation (i.e., playing out the domain from a given non-terminal state to the estimating value). v 0 indicates the root   while Term. criterion is not met do 3: Create root node v 0 with state s 0 4: while Term. criterion is not met do 5: BACKUP(v, δ) 8: while Term. criterion is not met do 9: Application of 2-opt on v 10: while Term. criterion is not met do 13: Application of 2-opt on V . return V node, and its state is s 0 . v l is the last node reached during the tree policy. Its corresponding state is indicated with s l . δ means the gain of the terminal state reached with a random simulation from state s l .

Algorithm for Parallel Machine Scheduling Problem with Job and Machine Sequence Setup Times, Due Dates and Maintenance Times
Representation of our problem Our problem is represented with a permutation (Figure 6). In our approach, the permutation means a given sequence of job indices. Each element of the permutation is assigned to the first machine. If the constraints of any element (any job) of the permutation are not satisfied, then the other items (jobs) will be scheduled to the next machine. Edge swapping (2-opt) Edge swapping is the swapping of elements between two randomly selected permutation items.
Iterative Monte Carlo Tree Search (IMCTS) Our algorithm (Algorithm 2) is the modification of the Monte Carlo Tree Search.
We start from a random state (initial permutation). Switching from one state to another means an exchange (2-opt) operation. So by state, we mean a particular permutation (solution).
Our MCTS is an iterative algorithm, which means, that the tree is re-built iteratively. After a tree is built, in the result solution, the 2-opt operation is also performed.

Test results
The test results are described in this section. Our data set was artificially generated. A total of 7 different data sets were used. The first five data sets have 17 jobs and 5 machines, and the last two have 25 jobs and 10 machines. The best, worst, and the average results of the runs are illustrated in the tables, where P. means the problem, F. means the fitness value, T. means   Table 5, for smaller problems, our IMCTS algorithm proved to be efficient. The situation is similar for a larger data set, this is illustrated in Table 6. In summary, based on the test results, the IMCTS algorithm proved to be efficient based on our entire data set.

Conclusion
In this article, a specific production scheduling task, the Parallel Machine Scheduling Problem with Job and Machine Sequence Setup Times, Due Dates and Maintenance Times was introduced. After the introduction, the literature on production scheduling was presented. In section 3, the mathematical model of the Parallel Machine Scheduling Problem with Job and Machine Sequence Setup Times, Due Dates and Maintenance Times was detailed. After that, the Monte Carlo Tree Search algorithm is detailed. In section 6, our algorithms, their representation and evaluation were presented. In section 7, the efficiency of the Iterative Monte Carlo Tree Search was tested with artificially generated datasets. Based on the test results, the efficiency of our representation and evaluation is demonstrated. In this article, we presented an example of using the MCTS for scheduling.