
SafeML: Safety Monitoring for Machine Learning
Runtime monitoring for machine learning classifiers
A machine learning classifier is validated on a dataset. In operation its inputs can drift away from that data, through new weather, a new sensor, a population the training set under-represented, or deliberate manipulation, and the classifier keeps giving confident answers that may be wrong. Accuracy measured at design time says nothing about that moment.
SafeML closes the gap with evidence produced while the system runs. It compares the statistical distribution of the data the classifier sees in operation with the distribution of the data it was trained on. When the two drift apart, the system knows its predictions are no longer backed by its validation and can respond: collect more data, warn an operator, or switch to a safer mode.
The approach is model-agnostic. It looks at the data rather than the model’s internals, so it works with many machine learning and deep learning classifiers. The original study found a meaningful correlation between the classifier’s decisions and these statistical distances, so the distance can act as a confidence signal.
SafeML was introduced in 2020 by Dr Koorosh Aslansefat, Ioannis Sorokos, Declan Whiting, Ramin Tavakoli Kolagari and Prof. Yiannis Papadopoulos.
How SafeML works
SafeML runs in two phases.
Offline, during training: the classifier is trained on trusted data, and SafeML records the statistical profile of that data for each class, including its cumulative distribution functions.
Online, during operation: incoming samples are buffered and grouped by the class the classifier predicted. Once enough samples have been collected, SafeML compares each group with its trusted reference using distances between empirical cumulative distribution functions: Kolmogorov-Smirnov, Kuiper, Anderson-Darling, Wasserstein, and a combined Wasserstein-Anderson-Darling measure.
The distance is checked against a threshold chosen during system design. If it is acceptable, the classifier’s output can be trusted; if not, the system raises an alarm, asks for more data, or hands over to an alternative system or a person.
SafeML has been extended in several directions since the first paper.
Human in the loop. SafeML II applied the distance measures in a human-in-the-loop procedure for autonomous vehicle software, tested on traffic sign recognition.
Continuous uncertainty. Work with Fraunhofer IESE replaced the yes-or-no trust decision with a continuous uncertainty estimate, removed assumptions about the shape of the data distribution, and made the computation faster on large datasets.
Security and explainability. The original study also applied SafeML to intrusion-detection data, and showed that the distance measures can help explain a classifier’s behaviour at runtime.
Trusted training data → reference distributions → live data buffer → statistical distance → trust, alarm or fallback
SafeML in runtime assurance
SafeML is one of the building blocks of DEIS’s wider work on runtime assurance. In the SESAME project, SafeML monitoring was one of the forms of runtime assurance built into Executable Digital Dependability Identities (EDDIs) on ROS-based robotic systems, alongside dependability guarantees and Bayesian-network monitoring.
In that setting a SafeML monitor does more than raise an alarm: its result can feed the EDDI’s assessment of whether the system can still provide the dependability guarantees its mission depends on.
Using SafeML
SafeML is open source under the MIT licence. The code, a Python package with examples plus MATLAB and R implementations, is on GitHub, with installation and quick-start guides on Read the Docs.
The method is described in SafeML: Safety Monitoring of Machine Learning Classifiers through Statistical Difference Measure, presented at IMBSA 2020.
Deploying a classifier in a safety-related system? We work with partners on runtime monitoring and assurance evidence for machine learning.
