U SMART ZONE – CREATING HIGHLY REALISTIC VIRTUAL ENVIRONMENT FOR VEHICLE-IN-THE-LOOP SIMULATIONS

. Developing Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicles (AV) based on machine learning is generally an extensive and costly process. Due to the increasing complexity of autonomous systems, the need for extensive testing and validation arises. In recent years, computer simulation has been used for these purposes. Performing realistic simulations, especially for the purpose of computer vision-based systems, requires a high-quality, almost photorealistic virtual environment. This paper introduces U SMART ZONE, a high-ﬁdelity virtual model of the Severní Terasa district in Ústí nad Labem, Czech Republic comprising of more than 7.5 km of drivable roads with a total area of approximately 1.4 km 2 for human-in-the-loop and hardware-in-the-loop (HiL) simulations.


Introduction
Autonomous driving is undoubtedly a technology that will, with its superiority, ultimately revolutionize the way we understand transportation and vehicle ownership.Many companies have been trying to develop this innovative technology for more than a decade.However, as the development progresses, it is becoming clear that a long way to fully autonomous vehicles still awaits us.Urban traffic is significantly more complicated, primarily due to the number of unpredictable or difficult-to-predict phenomena.The development of sufficiently complex systems for navigating in such a chaotic environment proved to be an issue principally because of the costly and time-consuming nature of machine learning.In addition, regarding security and legislative restrictions, it is often impossible to test experimental vehicle systems in live traffic.The range of learned scenarios is thus limited.As the complexity of autonomous driving increases, so does the need for high-quality and comprehensive testing.Autonomous driving systems are tested mainly on closed polygons, in virtual reality, or in a combination thereof, i.e., in augmented reality.While closed polygons offer almost laboratory conditions for testing and the possibility of reconstructing exact situations, virtual reality is significantly less limiting in the amount and scope of tests, which are, nonetheless, confined by preparation time, computing power, and other factors.Research in recent years indicates the increasingly efficacious use of virtual simulations for learning or validating of machine-learning-based vehicle systems.The supply of specialized software for "in-the-loop" testing has expanded significantly over the past years.
Based on our long-term understanding of vehicle simulations, predominantly in the human-in-the-loop domain, and our involvement in the commercial sector in developing Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicles (AV), we have identified one element as yet untapped.That is the photorealism and authenticity of the virtual environment.Virtual environments are still relatively simple and idealized.For complex ADAS and AV development and testing, improvements in the simulated environment are necessary to match the complexity of a general urban landscape.In addition, modern vehicle systems built on automatic image recognition systems are becoming progressively routine.And for these systems, it is critical in HiL learning to have a virtual landscape closely resembling the actual environment.For this reason, we decided to create an accurate photorealistic model of a sizeable part of the city Ústí nad Labem in the Czech Republic for virtual simulations with the possibility of validation via parallel testing.

Literature review
An extensive body of literature exists on developing simulation tools for HiL [1,3].Many of them consider their instruments highly realistic [2][3][4][5][6].Kang et al. [7] presented an overview of a publicly available virtual testing environment.This section introduces some approaches using high-quality, almost photorealistic computer-generated imagery.These computergenerated images work with objects with pre-assigned attributes.That in turn, gives the simulation tool the ability to display a semantic layer alongside the realistic scene, which can then be easily used for artificial intelligence learning or validation.This chapter  presents how to create such environment and what constitutes a high-quality environment.The first approach uses a direct reconstruction of the actual environment, for example, based on video footage.This method is utilized, for instance, by Gaidon et al. [1].Firstly, they acquire Real-World data.Consequently, they semi-automatically create a photorealistic dynamic 3D virtual world using off-the-shelf computer graphics engines and graphic assets (realistic 3D models and materials).In their model, main roads are positioned semi-automatically, while secondary roads and background objects (trees, buildings, etc.) are placed manually (the authors indicate these steps could further be automated).The resulting scenario can be seen in Figure 1.They then make minor adjustments to these virtual models, yielding a significantly broader range of scenarios for machine learning than they would get by simply annotating the underlying data.
Alternative approach operates with augmented reality.That is mainly used to validate already learned systems by extending the potential of closed test polygons.Authors use a virtual environment to generate road traffic.That includes vehicles, pedestrians, cyclists, or trains.Or they use it to add signs, obstacles and other static objects [4,10,11].Unfortunately, none of the mentioned-above authors on mixed reality testing showed the visual quality of the virtual elements.A different approach uses completely virtual systems.An example of such a system is a CAV HiL simulator [2].This simulator was employed for the development and testing Lane Keeping and Cooperative Adaptive Cruise Control.An illustration of the simulation can be seen in the Figure 2.
Another approach was demonstrated by Chen et al. [8].They verified the effectiveness of their HiL simulation platform by Path Planning and Decision-making Modules Verification, Reinforcement Learning-based Car-following Model Validation, and Autonomous Parking Algorithm.Their scenery was created using OpenStreetMap, a tool for creating terrain and basic building outlines.Various objects like buildings, trees, roads, and other object models were created by 3D modelling software Sketchup.The resulting scenario can be seen in Figure 3.
Similarly, Li et al. [5] build upon the open-source OpenStreetMap geographic data to create an elementary road network.Using CityEngine software, they import the attributes of the virtual road network (including type, width, and the number of lanes) and complete the surrounding environment (see Figure 4).
Introduced by Dosovitskiy et al., CARLA (see Fig-   ure 5) is complex HiL training and testing software [9]."The environment is composed of 3D models of static objects such as buildings, vegetation, traffic signs, and infrastructure, as well as dynamic objects such as vehicles and pedestrians.All models are carefully designed to reconcile visual quality and rendering speed: we use low-weight geometric models and textures but maintain visual realism by carefully crafting the materials and making use of variable level of detail."[9] They used these assets to create two urban environments, initially by modelling roads and sidewalks, then by manually placing buildings, vegetation, terrain, and other elements of transport infrastructure.The two city models have 2.9 and 1.4 km roads where the test vehicle can move.This system is modular and opensource and is used by many researchers for their HiL simulations.
Gomes et al. [12] took a different approach.In addition to autonomous driving testing, they also focused on the Quality of Experience of their users.For this reason, they aimed at a higher realism of the simulated environment, especially from the visual side.Their method used photogrammetry for all objects in the simulated environment, which you can see in Figure 6.
Continuously more authors use photogrammetry to create photorealistic objects, for instance, Shah et al. [3] uses their AirSim implemented in the Unreal engine (see Figure 7 -left) with its access to hundreds of pre-created models.Guerra et al. [6] took a similar approach with their FlightGoggles.They created their own 3D objects using photogrammetry (see Figure 7 -right).Both environments are generally used for development and testing of autonomous aerial vehicles.

Method
We decided to advance the current HiL simulation environment by creating a large-scale and highly realistic model of the Severní Terasa district in Ústí nad Labem, Czech Republic.That is a nearly 7.5 km of public roads with a total area of approximately 1.4 km 2 .This model contains a great variety of elements, including directionally divided and directionally undivided sections, a roundabout, a traffic light-controlled intersection, a dense public transport network, and frequent exits for integrated emergency vehicles (see Figure 8).Thus, it is ideal for running complex simulations.
The actual development of the model is divided into four successive steps.These are (i) elevation model, (ii) photogrammetric model, (iii) laser scanning, (iv) manual modelling.

Elevation model
The construction of the polygon commenced with an altitude map of the terrain.That defined the core element upon which more specific models were subsequently applied.Its visual quality primarily influences the quality of experience of human participants, as it also serves as a visual backdrop providing the model authenticity.At short distances, it is superseded by a more detailed model based on laser scanning or photogrammetry, described below.We compared several different sources of an altitude map.The first was OpenStreetMap (OSM).OSM (see Figure 9) is preferable in cases with lower standards for an accurate description of the relationship between real and virtual environments, as they have a relatively low resolution of elevation points for the vehicle or pedestrian simulation.Beyond that, they offer an additional option, for instance, the automatic generation of buildings.Nevertheless, the buildings and terrain generated are highly inaccurate, in particular, terrain lacks the subtle details (unevenness, etc.) found in an actual environment (see Figure 10).
In our country, a more accurate altimetry called ZABAGED® (short for "Basic Base of Geographic Data") is available.more authentic and makes it easier to situate objects in it.We further cultivate the visual aspect of the view through low-polygon model smoothing functions (e.g., Auto Smooth in Blender) (see Figure 12).

Photogrammetric model
The above elevation model delivers realistic and highvisual quality before a close-up.Then, it is insufficient from the driver's perspective.In such cases, photogrammetry can be beneficial as it provides, when used correctly, a relatively accurate point cloud.Consequently, accurate surface geometry can be generated, resulting in a high-quality and realistic simulation.And also, this step yields a high-quality texture that further enhances the realism of the overall environ- ment.Photogrammetry is a relatively simple and quickly deployable method for creating 3D models, but its use for longer sections still entails significant time effort for data collection.We are trying to solve this partly by placing the measuring equipment on a vehicle, which we then drive through the reconstructed area.The telescopic-modular design, shown in Figure 13, allows us to adjust the sensor positions to the nature of the site measured.It is adjustable in height between 1.5-3.0metres above the vehicle in any position, even when exceeding the vehicle outlines.That makes data collection quick and easy.This approach makes it possible to overcome the disadvantages of conventional fixed measuring devices (see Figure 14).However, this method brings several challenges for acquiring sufficiently clear images.Those range from dynamic shocks to the need for a synchronized shutter speed based on the vehicle speed.We use commercially available GoPro Hero 9 cameras to maintain this method cost-efficient.The difference between low and high quality images can be seen in Figure 15.Follow-up experiments showed that it is possible to achieve sufficiently high-quality image while driving.

Laser scan
The laser scan functions similarly within the framework of the constructed model as photogrammetry.Yet, it is a significantly more expensive technology employing, in our case, a high-speed high-performance dual scanner mobile mapping system Riegl VMX-2HA (Result shown in Figure 16).For this reason, we use the low-cost photogrammetry as an alternative.Consequently, we compare photogrammetric results with  result obtained via laser scanning.The key difference to be observed will be the quality of the geometric shape of the drivable area to allow the simulation of the realistic behaviour of the vehicle.The comparison will be performed using the professional vehicle physics simulation software CarSim.

Manual modelling
Many objects had to be modelled manually.There are many tools for simplifying complex geometric shapes obtained by photogrammetry or laser scanning (e.g., decimation, the process of discarding points from the data to improve performance) available.However, we have not come across any that would outperform manual modelling regarding object complexity (mesh LOD) and the resulting visual quality of the model, if a refined texture is used.In addition, many objects in our testing area that cannot be reconstructed without manual modelling (see Figure 17).We use many other tools to substitute surface details commonly lost using simplified object geometry.To illustrate, we apply the Shader effect via Vector Displacement based on the underlying texture (see Figures 17 and 18).This approach achieves simple yet photorealistic objects.For some objects, it is not yet possible to automate the creation of a virtual model with acceptable quality and complexity.An example of such an object can be seen in Figure 19.

Summary and future work
In this paper, we reviewed the current work on virtual environments for HiL simulation and presented our highly realistic digital model of a complex city environment for HiL simulation.The two main contributions of this model are (i) the highly heterogeneous environment for complex simulations, (ii) the possibility of parallel testing in virtual and actual environments to validate the results.
Using the real-world environment with all its flaws, including, for example, faulty, worn, or damaged roadway markings or contradictions between traffic signs and road markings, adds depth not previously used in HiL simulations.Thus, the resulting autonomous system derived from using the real-life environment is better equipped to rectify the randomness inherent to real road traffic.In the future, we intend to extend our model to include additional areas involving adjacent rural roads and highway segments.For modelling the rural roads, we have selected an area of approximately 8.5 km in length involving cramped villages and parts with limited visibility due to vegetation.That will allow simulations of emergencies associated with, for example, collisions with wildlife.Also, an approximately 9 km section directly adjacent to the current model area is selected for modelling the highway segments.

Figure 2 .
Figure 2. Simulation for testing Lane Keeping and Cooperative Adaptive Cruise Control [2].

Figure 3 .
Figure 3. Simulation for testing Path Planning and Decision-making Modules, Reinforcement Learning-based Car-following Model, and Autonomous Parking [8].

Figure 8 .
Figure 8. Extent of the reconstructed section in Ústí nad Labem, Czech Republic.

Figure 9 .
Figure 9. Example of terrain rendering by OSM.

Figure 10 .
Figure 10.Simple model crafted on low resolution OSM elevation map.
"Digital Terrain Model of the Czech Republic of the 5th generation (DMR 5G) represents the image of natural or man-modelled terrain in digital form as heights of discrete points in a triangulated irregular network (TIN) with a total mean error of the height 0.18 m in an open terrain and 0.3 m in a forested terrain."[13] The model was created using airborne laser scanning acquired data (see Figure 11).The availability of such an accurate elevation model makes the surrounding landscape look much

Figure 11 .
Figure 11.View of ZABAGED® elevation with basic texture (up) and without texture (down).

Figure 13 .
Figure 13.The measuring vehicle equipped with cameras for photogrammetry.

Figure 14 .
Figure 14.The disadvantage of a fixed scanner (in this case, a laser scanner) is the impossibility of collecting data on a site that is not directly visible to the scanner and Example of a model reconstructed from the passage of a measuring vehicle (down).

Figure 15 .
Figure 15.Image of the attainable quality of the photogrammetric model from a detailed environment shot with a static camera (up) and from a driving measuring vehicle (down).

Figure 16 .
Figure 16.Wired model from high-speed high-performance dual scanner mobile mapping system Riegl VMX-2HA.

Figure 17 .
Figure 17.Combination of multiple textures (plaster + graffiti) before (up) and after (down) application of displacement (focus on graffiti).

Figure 18 .
Figure 18.Real stop on modelled section (up) and manually modelled stop with texture from real photos (down).

Figure 19 .
Figure 19.Complicated model with no option automatic reconstruction.