Machine Learning and the Unknown Unknowns

No ratings

Presented at High Confidence Software and Systems Conference 2022 by

Darren Cofer

One of the important certification objectives for airborne software is demonstrating the absence of unintended behavior. In current software development processes, unintended behavior is associated with some identifiable structural feature, such as specific lines of code or a model element. However, in machine learning approaches to system development, unintended behavior may emerge from the data used to train the system. New inputs not encountered during training may result in novel activations in a neural network, leading to unexpected (and potentially dangerous) outputs. In this talk we will review the rationale and methods for detecting unintended behavior in current airborne software systems, including the use of model based development techniques and formal methods for software verification. Then we will consider the challenges posed by machine learning and examine new techniques that are being developed to address these challenges, as well as how these techniques may shape new certification guidance. We will also present results from a recent flight demonstration in which run-time assurance techniques were used to guarantee the absence of unintented behaviors in neural network-based aircraft collision avoidance system