DAVE
Data and processing model incl. prototype for digital twin for autonomous navigation of an inland vessel
| Duration: | 01.10.2022 - 30.09.2024 |
| Project Leader: | Prof. Dr.-Ing. Horst Hellbrück |
| Staff: | Jan-Philipp Schreiter |
Background
Inland waterway vessels have played a crucial role in global transport for centuries, as they are a cost-effective and efficient means of transporting goods and people across rivers and canals. In the future, inland waterway transport will become increasingly important as it is recognised as one of the least environmentally damaging modes of transport. However, European inland navigation faces challenges such as a lack of qualified personnel, increasing competition from alternative modes of transport and the need to minimise negative environmental impacts. To address these challenges, there is growing interest in the introduction of autonomous operation of inland vessels, which has already proven its worth in the automotive sector. Autonomous technology optimises ship navigation and planning, reducing the need for human intervention and increasing transport efficiency and safety. In addition, autonomous ships are designed to operate with fewer crew, further increasing safety and reducing costs. Despite the advanced technology of autonomous driving, integration into inland navigation is a complex task. Inland vessels have a long service life and many of them are still in operation after several decades. While this longevity is commendable, it also means that many vessels are equipped with outdated technology that may not be suitable for autonomous operation.
Objective and Approach
DAVE creates the digital twin of an inland waterway with all road users and the infrastructure. This enables remote operations, autonomous driving and simulations to analyse system behaviour and solve optimisation tasks. Communication and conflict resolution between road users on the inland waterway and with the traffic control centre are also implemented.
Task
The sub-project of the CoSA competence centre at Lübeck University of Applied Sciences comprises
- Communication between the sensors and with the IoT platform
- Connection of external information sources
- Collecting information about the entire ship
- Communication concept with collection of sensor data and efficient processing of this data
- Establishing rules of behaviour in the event of faulty and lost data transmissions
- Fusion of the independent sensor data of the digital twin
- Recurring testing of the concept with sensor data
- Realistic evaluation of the overall system
- Development of AI-supported algorithms to make decisions about the ship's control tasks, among other things
Communication
The communication concept for real-time communication between sensors and the IoT platform consistently integrates the previously developed sensor concept. It is divided into an internal and external view, with the internal view describing the components and interfaces on board the ship and the external view looking at the ship's connection to other participants. This concept is referred to as ‘Ship to X’. The internal view includes various sensors such as lidar, GPS, cameras and AIS. The external view, on the other hand, utilises technologies such as 5G and Starlink to ensure reliable communication with external partners.
The communication architecture is based on the Robot Operating System 2 (ROS2) middleware Data Distribution Service (DDS). The use of ROS2 enables the modular development and integration of software components for sensors, control systems and communication modules. The ROS2-DDS middleware ensures reliable data exchange within the system.
5G communication has been successfully integrated into the prototype box. A 5G modem from the manufacturer Quectel was used for this and the connection to the research network at the TH Lübeck was tested.
Assembly + Test
The ideas from the sensor and communication concept were combined in a prototype. The following sensors and components were used:
- LiDAR: RoboSense RS-LIDAR-M1
- Camera: Hik Vision DS-2CD2T47G2-LSU/SL
- NMEA matrix: Veinland 5NMEAto1-E
- Wind sensor: DigitalYacht WND100
- AIS: DigitalYacht AIS100 receiver
- GNSS: DigitalYacht GPS160
- Switch: NETGEAR GS308EPP PoE Switch
- Power: Ecoflow River 2 Max
- Supply: Mean Well NDR-75-24 transformer
The assembled prototype was subjected to various tests. A general function test in the laboratory, verification of the sensor data on land and a test trip on the River Trave. The synchronised delivery of several data streams from the sensors was checked in the first step using a recording in the laboratory. The data from each sensor was then verified individually. In the final test, the sensor platform was placed on an electric boat. With the outdoor measurement in October 2023 on the Trave in Lübeck, it was possible to test under realistic conditions. The focus of the measurements was on bridge crossings in order to limit the amount of data recorded.
AI in Project DAVE
The DAVE project focusses on object detection and segmentation in the field of artificial intelligence (AI). To this end, ROS2 nodes were developed that enable the inference of a YOLOv8 model.
Various models were trained, including the segmentation of water and bridges. This basis enables the successful implementation of autonomous control tasks. In combination with the fusion of the LiDAR sensor data, an object-based distance measurement is realised.
The sensor platform integrates an AI model from the project partner Titus Research, which enables the recognition and evaluation of traffic signs on waterways. The AI is based on YOLO version 7 and is similar to the object recognition system already integrated at TH Lübeck.
With this advanced technology, TH Lübeck is setting new standards in the use of artificial intelligence for autonomous systems in the maritime sector.
Sensordata Fusion
To improve sensor data processing, suitable methods for calibrating the camera and LiDAR sensors were selected and successfully implemented. The extrinsic calibration of camera to LiDAR was performed to ensure an accurate spatial mapping between the two sensor types.
Using the resulting projection matrix, transformation matrix and distortion matrix, it was possible to project the 3D point cloud of the LiDAR into the 2D camera image. This fusion of sensor data was then linked to the AI model for bridge/water segmentation to enable effective processing and interpretation of the data.
The determination of the distance of detected objects such as bridges was realised as a use case. This functionality provides significant added value for autonomous control tasks by enabling precise position determination and object size analysis. The implementation of these sensor calibration methods therefore represents significant progress in the development of autonomous systems.
Publications
Scientific Publications
- Inland Waterway Vessel Autonomy: Sensor and Communication Design - Ieee Oceans 2024 (Link)
- Sensor Fusion for Object Localization with ROS 2 - 5. KuVS-Fachgespräch zum Thema Lokalisierung (Link)
- Bridge Detection in Autonomous Shipping: A YOLOv8 Approach with Autodistill and GroundedSAM - MTEC-ICMASS-2024 (Link)
Further Reports
Sourcecode
Data
