Runtime Safety Monitoring of Machine Learning with SafeML

May 9
3 min read

Safety of Machine Learning with SafeML

SafeML will be useful to you if you develop a dependable system with components that employ Machine Learning (ML).

A key question that you may wish to address is whether and when the reasoning of these components can be trusted.

Imagine that you design an autonomous car, and you wish to incorporate an ML system that uses a camera to read the state of traffic lights. You may have trained this system extensively with large volumes of data, but this system may fail in adverse environmental conditions with potentially catastrophic consequences.

In another scenario, an algorithm may learn to detect a disease by examining images of typical symptoms. The algorithm may be biased towards these typical symptoms and may give false negatives when encountering atypical symptoms of the disease.

In a third scenario, an algorithm may have been trained to detect features on a human face by examining pictures of a certain ethnic group. However, it may fail to detect similar features correctly when it encounters images of people of different ethnic origins.

This type of bias in ML systems is currently attracting research. In the University of Hull, we are developing a method called SafeML which uses statical techniques to detect and report on such biases.

To illustrate the problem and the principle that underpins SafeML as a solution, imagine a system that detects lung cancer and has been trained with images that show symptoms of the disease. The system classifies an image in three classes “negative” “early stages” and “advanced stage” of the disease. During training, the accuracy of the classifier can be estimated by validating the output of the algorithm using images where the outcome is already known. This accuracy can be predicted, and we assume that it is high before the system is deployed for operation on new unverified data. It is precisely the task of the classifier to diagnose and classify correctly such data. And the question arises: can we trust the predicted accuracy established during training in these new situations?

In SafeML we take the view that accuracy can be trusted only if the statistical distribution of inputs that affect a classification has been shown not to drift much from that of these inputs in the training set. Such drift can confuse an ML algorithm. The phenomenon is called “concept drift” and is illustrated in Fig.1.

Fig.1. Concept drift

From left to write you see statistical distributions of “training” “validation” and “operational” data in two cases. The second case shows concept drift between “training” and “operational” data is detected.

Let us imagine that in the example of the medical diagnosis of lung cancer, the algorithm has been trained using a verified dataset of common symptoms. We have used 80% of the dataset to train the algorithm and 20% to validate it. Because training and validation data are similar statistically, i.e. both sets include common symptoms, the algorithm performs well in validation with 93% accuracy.

In the “normal case” of operation, the algorithm encounters images that are very close statistically to the training data and performs well with accuracy that matches and exceeds the one established during validation.

However, in the case of “concept drift”, the algorithm encounters images that are statistically different from the training data, perhaps of atypical symptoms of the disease. This confuses the algorithm, and the classifier reports false negatives (no disease) and classifies wrongly in general. The accuracy drops below 70%.

To address this problem, SafeML uses well- known statistical empirical measures to measure this distributional shift between training and operational features that influence an output. Shifts detected are interpreted to a degree of confidence in the accuracy of classifications made by the ML algorithm.

When SafeML reports confidence on the reasoning, the output of the algorithm can be trusted, and any further decision is less likely to cause harm. Fig.2 shows the SafeML concept.