Can We Monitor Uncertainty Before an Agentic AI System Fails?
Multi-agent AI systems are rapidly moving from research prototypes into enterprise workflows. Instead of relying on a single Large Language Model (LLM), these systems use multiple specialised AI agents to plan tasks, retrieve information, evaluate evidence and generate final answers. This architecture is increasingly attractive for enterprise AI, Agentic RAG, automated decision support, AI assistants and business-critical workflows.
But there is an important problem.
What happens when one agent is uncertain, retrieves the wrong information or makes an unreliable decision?
In a sequential multi-agent system, that uncertainty does not necessarily remain with the agent that introduced it. As our recent research highlights, “uncertainty introduced during planning, retrieval or evaluation may propagate through the workflow,” potentially “leading to cascading errors and semantic drift.”
Researchers at the University of Hull have therefore investigated whether uncertainty can be monitored as it moves through an Agentic Retrieval-Augmented Generation pipeline. The resulting framework combines uncertainty signals from individual AI agents with a Bayesian Network to estimate the reliability of the overall workflow.
The aim is simple: rather than waiting until an AI system produces a questionable answer, can we identify where confidence begins to deteriorate while the answer is still being constructed?
The Hidden Risk of Multi-Agent AI
Multi-agent systems can improve AI capability by dividing a complicated problem among specialised agents. A Planner may decide what information is required, a retrieval system may search for evidence, an Evaluator may decide whether that evidence is sufficient, and a Generator may formulate the final answer.
For businesses, this resembles a team of specialists working together.
But it also introduces dependencies.
If an early agent provides uncertain or misleading information, later agents may treat that information as valid. The final response can therefore appear confident even when its reasoning process has gradually moved away from reliable evidence.
This problem is receiving increasing attention outside academic research as well. Recent practitioner discussions describe how retrieval mistakes can become part of the shared context passed between agents and subsequently influence downstream reasoning. Production-focused discussions also identify coordination complexity, reliability and limited observability as major challenges when multi-agent prototypes are deployed at scale.
For a CEO, CTO or AI product leader, this changes the question from:
“How accurate is my AI model?”
to:
“How reliable is the complete AI workflow, and can I see where it is becoming unreliable?”
That distinction becomes particularly important when Agentic AI supports decisions in healthcare, financial services, manufacturing, infrastructure, predictive maintenance, energy and other regulated or safety-related environments. The paper specifically motivates the work through the growing role of LLMs in business-critical and industrial processes.
Our Uncertainty-Aware Agentic RAG Framework
The framework developed at the University of Hull by Louis Donaldson, Connor Walker, Koorosh Aslansefat and Yiannis Papadopoulos introduces uncertainty monitoring directly into an Agentic RAG workflow.
Rather than assigning one confidence score only to the final answer, the framework examines uncertainty at different stages of the pipeline.
The architecture contains three key AI agents:
Planner Agent: Determines how the original question should be decomposed and what information needs to be retrieved.
Evaluator Agent: Examines retrieved evidence and determines whether sufficient information is available.
Generator Agent: Produces the final answer using the evidence collected during the preceding stages.
Each agent contributes information about its uncertainty. A Bayesian Network then connects these signals and estimates the overall state of the workflow. This provides both a system-level indication of reliability and a way of identifying which part of the pipeline contributed most strongly to the uncertainty.
This is particularly useful for AI observability, because monitoring only the final answer may hide where the underlying problem began.
How Does the Method Work?
The framework combines three complementary ideas.
1. Monitoring Semantic Divergence
An agent may gradually move away from the original information need while reformulating queries or reasoning across several steps.
The research measures this semantic divergence by comparing the meaning of the agent's selected output with alternative sampled outputs. Wasserstein distance is used within an embedding space to quantify this difference.
A large divergence suggests that the agent is less certain about the path it has selected.
For a multi-step AI workflow, this matters because small deviations early in the process can influence everything that follows.
2. Asking the Generator About Its Own Confidence
The Generator uses a P(True) self-evaluation mechanism.
After generating an answer, the model is effectively asked whether it believes its answer is correct. The probability associated with the affirmative response becomes an additional uncertainty signal.
This does not mean that an LLM's self-confidence should automatically be trusted. Instead, it provides another measurable signal that can be combined with evidence from other stages.
3. Propagating Uncertainty Through a Bayesian Network
The central component is the Bayesian Network.
Each stage of the Agentic RAG workflow becomes a node in the network. Dependencies between the stages are represented explicitly, allowing evidence observed at individual agents to influence the estimated reliability of the overall pipeline.
In the proof-of-concept implementation, the system can classify the overall workflow as OK or FAIL, providing a basis for actions such as accepting an answer, flagging it for review or abstaining from providing it.
This turns uncertainty from a hidden property of an AI model into something that can potentially be monitored across the complete AI decision process.
What Did the Experiments Show?
The framework was evaluated using StrategyQA and HotpotQA, two benchmarks requiring different levels of multi-step reasoning. Experiments used GPT-3.5-Turbo and GPT-4.1-Nano, with 200 sampled questions from each benchmark.
Four evaluation measures were considered: AUROC, AUARC, Expected Calibration Error and Brier Score.
The results reveal an important point.
More Complex Reasoning Makes Uncertainty Propagation More Valuable
On HotpotQA, where questions require stronger multi-hop reasoning, combining uncertainty across the complete Bayesian Network improved the ability to identify problematic responses.
For GPT-4.1-Nano, for example, the overall Bayesian Network achieved an AUROC of 0.7240, compared with 0.5919 for the Planner, 0.5369 for the Evaluator and 0.7145 for the Generator individually.
The researchers therefore found that Bayesian uncertainty propagation becomes particularly useful when uncertainty builds across multiple reasoning stages.
Individual Confidence Signals Still Matter
The Generator's P(True) signal was consistently one of the strongest individual indicators of uncertainty.
For simpler questions, this signal could sometimes perform better than combining weaker information from other agents. This is an important result because it shows that adding more uncertainty signals does not automatically produce a better system. Their quality matters.
Calibration Depends on the Task
The framework also behaved differently across the two benchmarks.
On the simpler StrategyQA benchmark, the Bayesian Network could be overly conservative and assign too much probability to failure. On HotpotQA, however, calibration improved considerably.
This suggests an important lesson for responsible enterprise AI:
An uncertainty monitoring system must itself be calibrated to the environment in which it operates.
Why This Matters for Enterprise Agentic AI
The significance of this research extends beyond question-answering benchmarks.
Companies are increasingly considering AI agents for customer support, engineering, financial analysis, supply-chain planning, scientific research, predictive maintenance and autonomous business processes.
In these settings, simply asking whether the final answer is correct is not enough.
Organisations increasingly need to understand:
where uncertainty entered the workflow;
how it affected downstream AI agents;
whether the final recommendation should be trusted;
when the AI system should abstain or request human review;
which agent should be investigated when something goes wrong.
These capabilities are directly relevant to AI governance, AI risk management, trustworthy AI, AI assurance and production AI observability.
For business leaders, the commercial issue is equally important. A sophisticated Agentic AI system that cannot indicate when it is uncertain creates operational risk, even if its average benchmark accuracy looks impressive.
This is why runtime AI monitoring may become as important as model capability as organisations move from experimental AI agents toward production systems.
From AI Accuracy to AI Observability
There is also a broader conceptual shift behind this work.
Traditional machine learning evaluation often asks:
How accurate is the model?
Agentic AI requires an additional question:
What is happening across the workflow when several AI components interact?
A final error might originate from retrieval, planning, evidence evaluation, reasoning or generation.
Monitoring these stages separately creates the possibility of moving from a single end-of-pipeline confidence score toward workflow-level AI observability.
For developers, this can support debugging.
For risk and governance teams, it can support auditing.
For management, it can provide a clearer view of whether an AI-enabled process is operating within an acceptable level of risk.
And for users, it creates the possibility of an AI system that knows when additional verification is required.
Current Limitations and Future Directions
The research is deliberately presented as a proof of concept, rather than a finished industrial assurance solution.
Three important limitations remain.
First, the current Bayesian Network uses a deterministic OR relationship, meaning unreliable upstream signals can sometimes have too much influence.
Second, the Generator's self-evaluation can be conservative and requires improved probability calibration.
Third, the present experiments use general question-answering benchmarks rather than domain-specific industrial data.
Future research will therefore investigate:
adaptive Bayesian Network structures;
better calibration of agent uncertainty;
learning different levels of importance for different AI agents;
domain-specific retrieval systems;
validation in real industrial decision-support environments.
One planned application is Offshore Wind maintenance decision support, where AI agents may need to combine maintenance records, operational information and technical knowledge while supporting decisions with real financial and safety implications.
Toward AI Agents That Know When Something Is Going Wrong
Agentic AI promises systems capable of performing increasingly complex tasks through cooperation between specialised AI agents.
But more capability also introduces more interaction.
And more interaction creates more opportunities for uncertainty to spread.
Our results indicate that Bayesian uncertainty propagation offers a promising way to observe this process, particularly for tasks involving genuine multi-stage reasoning. The goal is not simply to produce another confidence score. It is to begin understanding where uncertainty originates, how it moves through an AI workflow and when the complete system should be trusted.
As Agentic AI moves from demonstrations into business-critical applications, this type of runtime monitoring, AI assurance and uncertainty-aware decision support will become increasingly important for building AI systems that organisations can deploy responsibly.
Engage With Us
The full paper, “Bayesian Uncertainty Propagation for Agentic RAG Pipelines: A Proof-of-Concept Study on Multi-Hop Question Answering,” is available on arXiv.
Read the research paper on arXiv
The implementation, including the Agentic RAG pipeline, Bayesian Network inference and evaluation notebooks, is also publicly available through the repository identified in the paper.
Related discussions
The following two articles provide useful industry context around why error propagation, coordination and observability are becoming important issues for multi-agent AI systems:
Reference
Donaldson, L., Walker, C., Aslansefat, K., & Papadopoulos, Y. (2026). Bayesian uncertainty propagation for agentic RAG pipelines: A proof-of-concept study on multi-hop question answering. arXiv. https://doi.org/10.48550/arXiv.2607.00972
Comments