Rolf Ernst

Autonomous cyber-physical systems depend on high-resolution sensors that generate multi-gigabyte-per-second data streams for accurate environmental perception and system safety. However, existing publish-subscribe middleware frameworks, such as ROS 2 with DDS, are optimized for small data objects and face significant latency and overhead challenges when handling large sensor data between distributed application nodes. Recognizing the need for more efficient data management, we propose a new companion middleware that prioritizes application-specific data relevance, enabling selective communication of critical information while reducing the burden on communication resources and maintaining interoperability with state-of-the-art publish-subscribe middleware. This companion middleware enables timely and effective sharing of sensor data by focusing on regions of interest relevant to specific tasks, such as traffic light detection in driving scenarios. Experimental evaluations of our open source implementation of this companion middleware on a Linux platform demonstrate that our protocol integrates efficiently with ROS 2, significantly enhancing data management and communication efficiency.

As semiconductor technology advances below 10 nm, the failure to capture the full range of system timing behaviors proposes a challenge for performance-demanding critical real-time applications such as neural networks. This article proposes a dual timing model for design, verification, and monitoring of real-time neural network systems.

In autonomous systems, growing sizes of application data, primarily related to perception tasks, have to be transmitted over communication infrastructures that provide higher data rates. Knowledge and exploitation of the clustered structure of multi-frame application data have proven to reduce the interference between different real-time critical communication streams, as recently shown for synchronous systems. However, the accompanying increase in frame numbers will foreseeably make the corresponding analysis impractical due to frame-dependent processing times. As a solution, based on the synchronous example mentioned, we demonstrate how to compose multi-frame transmissions to larger frame clusters in the analysis and decouple the analysis complexity from the application data sizes. Based on an automotive and industrial TSN use case, our results show comparable analytical response times and very low computation times at arbitrary data rates and sizes. It thus enables the deployment of efficient configuration methods that arbitrate large data samples transmitted through the network as one whole frame cluster. In addition, we made the developed analysis openly accessible.

Achieving reliable worst-case timing poses a challenge for modern, high-performance, commercial off-the-shelf hardware platforms deployed for industrial applications. Particularly for safety-critical industrial systems, e.g., robot-human collaboration using convolutional neural networks, timing must be considered to operate safely. Although state-of-the-art real-time operating systems and isolation techniques provide predictable timing, they restrict design decisions as many modern hardware platforms are not supported, introducing serious performance penalties. Besides traditional timing considerations, such as the number of cache misses, process variations due to chip manufacturing become more prominent, causing chips from the same model series to exhibit different timing behavior. This circumstance complicates achieving reliable timing on a system level even further. In this work, we present examples of physical variations using most recent hardware platforms, including 12th-generation Intel-based embedded hardware and GPU-based platforms using an Nvidia Jetson AGX Xavier. We elaborate on a potential solution from the avionics domain, called Timing Diversity, which allows for masking unexpected occurrences of worst-case timing behavior by exploiting modular redundancy inherent to safety-critical systems. A key result of our work is that Timing Diversity enables the safe usage of high-performance platforms such as the Nvidia Jetson AGX Xavier, consequently yielding a significant performance boost of nearly 6x.

In-Vehicle Networks (IVN) are under intense pressure to be resource-efficient at low cost and flexible enough for the future. The deployment of a large number of new sensors for autonomous driving and self-awareness significantly increases the demand for network resources with low latency. In addition, the dynamics of network traffic lead to large over-provisioning of resources due to the lack of flexibility in statically designed networks. Application-aware resource management addresses these issues by providing information about transferred application data, enabling control of applications, and managing resource allocation during runtime. Application-level information is used to classify data transmissions into traffic groups based on their characteristics. By focusing on resource-intensive applications, such as large sensor object transmissions composed of hundreds of individual packets, it becomes possible to shift from packet-level to object-level deadlines. This shift introduces slack, enabling new opportunities for end-to-end IVN reconfiguration, which optimizes resource utilization and provides the flexibility to handle future demands. The generated slack is used to synchronize on one side the network access for object transmissions and on the other side the reconfiguration process. The combination of these two mechanisms enhances the efficiency of the timestamp-based reconfiguration approach. As a result, it is possible to reconfigure the network at runtime while ensuring lossless and seamless data transmission for high-bandwidth connections from the application’s perspective. To demonstrate the effectiveness of our protocol under real-world conditions, we implemented it on our hardware test setup using commercial off-the-shelf, Linux-based computers to build an IVN.

Vehicle-to-everything (V2X) roadmaps envision future applications that require the reliable exchange of large sensor data over a wireless network in real time. Applications include sensor fusion for cooperative perception or remote vehicle control that are subject to stringent real-time and safety constraints. Real-time requirements result from end-to-end latency constraints, while reliability refers to the quest for loss-free sensor data transfer to reach maximum application quality. In wireless networks, both requirements are in conflict, because of the need for error correction. Notably, the established video coding standards are not suitable for this task, as demonstrated in experiments. This article shows that middleware-based backward error correction (BEC) in combination with application controlled selective data transmission is far more effective for this purpose. The mechanisms proposed in this article use application and context knowledge to dynamically adapt the data object volume at high error rates at sustained application resilience. We evaluate popular camera datasets and perception pipelines from the automotive domain and apply two complementary strategies. The results and comparisons show that this approach has great benefits, far beyond the state of the art. It also shows that there is no single strategy that outperforms the other in all use cases.

Future autonomous mobile systems will greatly benefit from cooperation and real-time sensor data exchange using V2X communication. In such applications, wireless communication has to cope with stringent real-time and safety constraints, a huge challenge given the inherently lossy wireless communication with highly dynamic channel and error conditions. To meet the safety and real-time goals, the use of state-of-the-art (5G and 802.11) V2X technologies focuses on reliable exchange of small data objects, as in URLLC. In contrast, reliable low-latency exchange of large data, such as camera frames, has received little attention, despite its predicted benefits in safe perception and cooperation. The LOTUS project, outlined in this paper, exploits the specific properties of large application data objects to develop novel, application-aware mechanisms for low-latency reliable large data exchange on top of existing and future V2X technologies. Evaluation with statistical analysis, simulation, and physical prototypes demonstrates the feasibility of low-latency large data object exchange at unprecedented levels of reliability.

There is a trend towards communication of larger data objects in wireless vehicle communication. In many cases, communication uses publish-subscribe protocols. Data rate requirements of such protocols are best addressed by wireless multicast protocols, but the existing protocols lack an error protection that is suitable for real-time and safety-critical applications. We present an application-aware protocol that supports the popular DDS (Data Distribution Service) middleware. By exploiting data object deadlines and slack for retransmissions and employing an adaptable, multicast-aware prioritization mechanism, the reliable exchange of large data objects is enabled. The protocol is sufficiently general to be used on top of different communication standards such as 802.11- and cellular-based V2X (Vehicle-to-Everything) technologies. The protocol was implemented in an OMNeT++ simulation model and evaluated against recent state-of-the-art alternatives using parameters and constraints taken from a motivational truck platooning example. Furthermore, the protocol was implemented using an open-source DDS implementation as the basis and tested on a physical wireless demonstrator setup. The evaluation shows that the presented multicast protocol substantially outperforms the alternatives keeping streaming applications operational even under high frame error rates.

With research trending towards increasing amounts of Vehicle-to-Everything (V2X) applications that use a shared wireless medium for data exchange, the coordination of those applications gains more and more importance. Applicationcentric Resource Management (RM) is a promising candidate for that coordination. It allows to address both application constraints as well as network properties in order to guarantee adherence to real-time and safety constraints of critical applications, such as cooperative perception based on large data object streaming. These applications require the reliable and fast exchange of large data objects. We use earlier work on error correcting real-time protocols with a safe control protocol for application-aware dynamic network (re-)configuration. The RM mechanisms were evaluated using a physical wireless prototype that used commodity of the shelf hardware that resembles an automated valet parking use case. The RM proved highly effective in preventing constraint violations. Meanwhile, it did so in a predictable and synchronized manner even if the wireless channel is subject to non-negligible error rates.