top of page

Safety of AI

Research theme 01 · Led by Dr Koorosh Aslansefat

In short

Safety of AI asks whether a machine learning component can still be trusted once it’s deployed, and how to tell automatically when it can’t. A model is validated on the data it was given. Then the world changes.

Why it matters

Classical safety engineering assumes a system you can specify. You state what it should do, analyse how it could fail, and argue that the remaining risk is acceptable. A learned component breaks that assumption: its behaviour comes from data, not from a specification an engineer wrote, so there’s no design model to analyse in the usual way.

 

In practice, accuracy measured at validation time is a statement about the past. When the input distribution drifts — new weather, a new sensor, a population the training set under-represented — the model can keep giving confident outputs that are wrong. For a system whose failure can hurt someone, that silence is the hazard.

 

This theme builds the monitoring and assurance that closes the gap: evidence generated while the system runs, not just at design time.

Theme lead: Dr Koorosh Aslansefat

Assistant Professor · Co-Lead, DEIS

 

Works on dependability, AI safety and explainable AI.

 

Also contributing: Mohadeseh Mollapour · Razieh Arshadizadeh · Dr Debarati Chakraborty

 

View profile →

SafeML

Statistical distance measures that compare live inputs with the distribution a model was trained on, so a classifier can flag that it’s outside its validated envelope instead of answering anyway.

Out-of-distribution and dataset shift

Detecting when operating conditions have moved away from the training data, telling gradual drift from sudden change, and deciding what a safe response looks like in each case.

Interpretability

Explanation methods that help a safety engineer rather than a data scientist: tracing a failure to a cause that can be designed against, not just visualised.

SafeLLM

Extending assurance to large language models and the systems built on them, where outputs are open-ended and a “correct answer” is much harder to define than in classification.

AI assurance

Turning the above into an argument a regulator or safety case author can use: what evidence is needed, how it’s produced, how it’s kept current once the system is in service, and what it doesn’t cover.

Open questions

This is live research. These are the parts we consider unresolved:

 

• How much distribution shift is tolerable before a safety argument stops holding, and who decides.

• Whether runtime monitors can themselves be assured to the standard the system they protect requires.

• What an adequate safety case looks like for a generative model with an open-ended output.

 

Assuring an ML component? We work with industry partners on runtime monitoring and assurance evidence for deployed models.

 

Talk to this theme →

​

Related themes:

03 · Security and Safety-Security

05 · Robots & Autonomous Systems

06 · Safety and Assurance Tools

​

← All research themes

bottom of page