Presentation title

Impact of Machine Learning on Safety Monitor

Authors

Andrea Bondavalli and Lorenzo Strigini

Institution(s)

University of Florence

Presentation type

Presentation of a research group from one or more scientific institutions

Abstract

Machine-Learning components in safety-critical applications allows us to assign to computer systems very complex tasks that would be unfeasible otherwise. However they are also a weak point from the viewpoint of safety assurance, especially for critical systems with ultra-high dependability requirements of critical systems. Techniques and tools are needed to guarantee the safety of ML components, without degrading its performance. An aspect requiring study is how the interactions between components capable of learning, such as Neural Networks, and other components that are “static” in this sense, evolve during the training. It is theoretically possible that learning by the Neural Network may reduce the effectiveness of “static”, simple, and thus trusted, error checkers or safety monitors, creating a major problem for safety assurance.

We present an initial exploration of this problem focused on the automotive sector, where machine learning is heavily used with results. We considered a standard vehicle architecture, managed by a Controller backed by and a Safety Monitor: the Controller is a neural network trained with reinforcement learning techniques and is the primary component of this system. It senses the environment using 3 different cameras and its task is to drive the car from a start to a target position, obeying the traffic laws. The Safety Monitor performs a safety check on the action chosen by the Controller, using LiDAR data and accessing info on the system’s state such as speed and coordinates. If an obstacle is detected too close in front of the vehicle and the Controller’s action may not avoid collision, the Safety Monitor will brake. We tested this architecture by simulation, focusing on how the efficacy of the Safety Monitor changed as a result of the training of the Controller.

We documented instances in which the Safety Monitor was made less effective by the progressive improvement of the Controller. We discuss the relevance of these results and their implication for safe design and safety assurance.

The immediate implication is that the common architectures that pair machine learning components with safety monitors need joint empirical validation of the whole architectures, a much more onerous process than separate validation of these subsystems, as often advocated. This puts emphasis on the need for design and analysis techniques that address this issue effectively.

---

Additional material

• Presentation slides: [pdf]

For more details on this presentation please click the button below: