AUGMENTED REALITY TO SUPPORT PARCEL HANDLING IN LAST-MILE LOGISTICS

. We have developed an augmented reality (AR) based system which keeps track of events during the parcel handling process for last-mile logistics. The system can retrieve and highlight the location of parcels in large piles with AR on the user’s smartphone. A camera array automatically detects the parcels laid down manually by an operator. New parcels are scanned and parcel fingerprints are generated semi-automatically. The system can detect and track the known parcels by fingerprint and can further highlight the location of the parcel using 3D visual clues, directly on the smartphone of the operator.


Introduction
The rapid growth in e-commerce challenges the lastmile logistics and further demands for more effective technical solutions and procedures [1].Last-mile logistics concerns the transport of goods from the final warehouse to the customers' home and can be separated into three main phases: (1) loading the truck at the warehouse, (2) transport from the warehouse to the delivery area, (3) the final delivery to the customers [2,3].
Augmented reality is expected to be one of the enabling key technologies in the age of industrial digitalization [4].Recent studies show warehouse operations can benefit from augmented reality technology through increasing the efficiency [5,6] and reducing the costs [7].
Several technical challenges reported in the literature [7,8] limit the adoption of augmented reality at the industrial level.So far, little work has been done to investigate the barriers of integrating augmented reality into logistics processes adapted to last-mile delivery.
This paper presents a technical solution which integrates augmented reality to support indoor as well as outdoor logistics processes for last-mile delivery of parcels.In order to show the field delivery agent the correct location of the parcel, it is important that the sensing system works successfully.The camera system tracks the parcel's precise location and presents this in augmented reality on the user's smartphone.The visual tracking uses parcel fingerprints with the characteristics of parcels, such as the dimensions, the shape and the texture or the bar-codes.We have implemented the AR system and further conducted a small-scale targeted experiment for testing the functional system prototype.This is the first approach to couple a camera sensor array which detects parcels in storage racks with a visualization in augmented reality on smartphones, to help localize parcels for parcel picking and delivery.The technical solution is called SOLiD and means Self Organized Logistics in Distribution.The research project represents the joint efforts of PrimeVision BV, DPD and the University of Groningen in The Netherlands.
The next section presents relevant research projects and related work on the automatic input sensing algorithms, tracking parcels and displaying information for building situational awareness in operational environments.
Section 3 discusses the aspects related to the architecture of the parcel tracking system.Section 4 presents and discusses the results of the experiments with the first functional system prototype applied in a scaled-down version of the first operational scenario.The last section addresses the conclusions of our research and pinpoints aspects regarding the future work.

Related work
The topic of augmented reality has been explored and found to be an interesting technology with the potential to improve the efficiency in logistics operations [3,9,10].The use of situation-specific information technology generally saves a significant amount of time due to the fact that less paperwork or manual checks have to be performed [2].Augmented reality technologies are usually offered in the form of apps for mobile phones, because these hand-held projectors are the best practice for introducing augmented reality to the market due to the low production costs and the ease of use [10,11].Recent laboratory experiments have shown that augmented reality can boost efficiency and accuracy for order picking in warehouse operations [5].Augmented reality graphic-based user interfaces can reduce job completion time and number of errors by 13 % and 59 % respectively, as concluded from a simulated warehouse job involving order picking and part assembly [6].Also, earlier work reported on the benefit of augmented reality over the traditional routines for warehouse operations [12].
In recent years, several system designs and architectures have been created to support the use of augmented reality.Essential elements in those architectures are a smart device (smart glasses or mobile phone), a projecting application system, camera vision, internet connection, privacy gates, command controls, modeling tools, back-end services and databases [13].This model has been used to develop the system design.Over the last years, a large number of open-source augmented reality frameworks (e.g.ARCore, ARkit, VR Juggle, Unity 3D) have been developed with tool kits that can be used easily by computer engineers [14,15].These tool-kits provide the computer engineers the tools to develop 2D or 3D content in an augmented form (e.g.arrow, text, cross etc.).The aforementioned frameworks help the computer engineers create the front-end part of the system, the part that is visible for the users and where the interaction with the system takes place.
There are also scholars that doubt the actual benefits that augmented reality can bring to the logistics operations because of multiple challenges during the implementation.Rejeb [8] conducted an extensive literature review of 43 papers from academic journals on the challenges of augmented reality in logistics and 74 % of these papers mentioned that technical challenges are an obstacle for implementing augmented reality systems.One key challenge is object recognition, because the digital world has to be in harmony with the real world in order to achieve high accuracy [16].Robustness of the system is continually improving despite computational costs are still high, but real-time tracking for pure vision-based approaches looks very promising [10].
Furthermore, software challenges follow no standardization of the programming language and this makes it difficult for practitioners to experiment and create the correct algorithm [17].
Next to the integration of augmented reality as a means to enhance the visualization, the integration of various sensors [18] and vision models [19,20] has been the focus in order to recognize, validate, and optimize the activities of workers in logistics and for warehouse automation.
In our research, we have integrated and built upon previous findings regarding input sensing models for activity recognition with augmented reality oriented user interface for supporting last-mile logistics operators and delivery agents.

System architecture
The system architecture consists of a front end and back end (Figure 1).The user interacts with the frontend part of the system, where the operator loads the truck at the distribution center.
The system is using cameras that are installed inside the truck, for registering and tracking the parcels inside the truck by using camera vision.The cameras provide video streams which extract knowledge about the layout of the parcel after loading and for rendering the AR layer.The algorithms for the detection and tracking of parcels are replicated on each camera of the camera array.The back end fuses together all the camera observations at the high semantic level so as to build global situational awareness on the parcels at the storage rack.
When the field delivery agent arrives at the customer, he/she runs the augmented reality application and sees a pop-up screen with the details about the customer and the parcel.
The 3D user interface displays the animated visual clue in the form of an animated arrow pointing to the parcel to be handled next by the field delivery agent.The 2D user interface displays information about the parcel to be handled.Moreover, the interface includes buttons to update the handling status of the parcel (see Figure 2).The detection of 2D markers from one or multiple cameras is used to reconstruct the 3D geometry of the physical shelves and storage rack.Following this, the camera parameters are computed.A second calibration relates to a manual alignment and is done through the mobile application by adjusting the 3D position of specific key-points of the shelf.This step is necessary for fine tuning the 3D rendering.The pattern match algorithm has a multi-threaded implementation and uses scale-invariant feature transform (SIFT) descriptors for visual pattern matching.The bar-code module relies on ZXing [21], an open-source, multi-format 1D/2D bar-code image processing library.In scan mode, the sensor component uses all information about the detected parcels to generate parcel fingerprints.The segmentation of the side images of a parcel uses the algorithm of Wang and Guo [22] which reconstructs the RGB-D indoor scene with plane primitives.The algorithm estimates the entire scene by adaptive planes without losing geometry details and preserves sharp features in the final mesh.In addition to the sensor component, the SOLiD system consists of an augmented reality-based visualization component.This component is a mobile application running on the smartphone of the field delivery agent (Android device).

Results
PrimeVision and the University of Groningen in The Netherlands have run a first experiment on a scaleddown version of the first scenario with the fixed storage rack at the premises of the parcel delivery company.Figure 2 illustrates a part of the experimental setup together with the AR layer highlighting the position of the parcel to be delivered next.
The experiment has shown that the combined accuracy of parcel detection and the precision of 3D parcel presentation in augmented reality for a smallscale scenario with the delivery agent and with tens of parcels in the parcel database, is 95 % (Figure 3).A limitation of the current system is the parcel detection by image recognition.The keypoint-based detector fails when a parcel has a few or no key points at all.On the other hand, the key point-based approach for parcel detection and tracking proved to be robust against variation in scale, illumination, rotation or occlusion.Two different tracking algorithms were tested in the lab test, namely state-based tracking with one single key frame and state-based tracking with multiple key frames.Our findings showed that when the augmented reality system used a multiple key-frame state-based algorithm was more robust compared to a single key-frame state-based algorithm because it was able to track 95 % of the parcels compared to 80 % respectively.Increasing the number of key frames per state has a positive impact on the performance of the system.The factors relate to lowered illumination and the associated noise sensed in the video frames.
One of the drawbacks of a single state-based tracking algorithm is that it only has one opportunity to find a parcel in contrast to a multi-frame state-based tracking algorithm.
The camera vision is subjected to the dynamic environment such as shadows, incidence of light or turbulence, and this creates noise for the algorithm to track the parcel.The system must be robust since it has to operate in a dynamic environment due to the fact that the cameras are installed inside a truck.Another reason for the lower success rate, was that parcels were only partially in the active Field Of View (FOV) of the camera.This made it more difficult for the algorithm to recognize the unique fingerprints of a parcel.Because the system only uses one key frame it sometimes was not able to find the parcel when it was partially in the active FOV.However, the higher success rate was achieved through a higher number of key frames that the tracking algorithm can use to find the parcels and thus increases the chances to find a parcel.The multiple key-frame state-based algorithm stored ten key frames every time a parcel was loaded into the truck.This increased the chances of finding the parcels by tenfold.The results of the preliminary study can be found in Figure 3.

Conclusions and future work
The first results from a small-scale targeted experiment of PrimeVision and the University of Groningen in The Netherlands has shown very promising results.The research has been carried out on a scaled-down version of the storage rack to comply with the new imposed governmental Covid-19 oriented regulations forbidding access to the research facilities.Our experiments have revealed that the SOLiD prototype can automatically determine the location of parcels during the parcel handling process in last-mile logistics.
A follow-up step of the current research project has already started by testing the one version of the system in a real-life production environment.We want to see the extent at which the system can support the parcel-handling activities in the storage location of a local store which has dedicated service for parcel handling.

Figure 2 .
Figure 2. Highlighting in AR the location of the parcel to be delivered next.

Figure 3 .
Figure 3. Results of the lab experiment.