When AI Fails Quietly: Why Resilience, Not Prevention, Is the Real Challenge
Wed 1 Apr 2026
AI is changing how failure manifests inside the enterprise.
Traditional systems tend to fail visibly. AI systems often continue to operate, producing outputs and passing monitoring thresholds even as performance drifts, dependencies degrade, and decision quality becomes harder to assess. By the time anomalies are recognised, consequences may already have propagated across workflows.
Ahead of Tech Show Frankfurt, we sat down with Andrea Thu Le, Business Resilience Manager at Allianz Global Investors, who works at the intersection of governance, operations and incident response, where policy meets system behaviour in live environments.
From that vantage point, she observes a widening gap between formal compliance and operational readiness. As AI systems become embedded into decision-making processes, the challenge shifts from preventing failure entirely to managing it deliberately.
– – – – – –
You’ve been working at the intersection of AI and resilience. What’s changed recently that’s making organisations rethink how they approach failure?
For a long time, the dominant question in enterprise technology was “how do we prevent failure?” Security controls, governance frameworks, compliance programmes – they were all designed around the assumption that the right combination of policies and tooling could keep systems running as intended. AI has quietly made that assumption untenable.
What’s changed is that AI systems fail differently. A server either works or it doesn’t. An AI model can keep running and appear operational while producing outputs that are subtly wrong, gradually degraded, or increasingly misaligned with what the business actually needs. That kind of silent, invisible drift doesn’t trigger your incident management process. It doesn’t show up in your uptime metrics. And by the time someone notices, the downstream consequences have already accumulated.
At the same time, organisations are deploying AI faster than they’re building the operational structures to support it. Governance frameworks and model risk policies are being developed, but the question of what actually happens when something goes wrong, who owns the response, what’s the fallback, and how do you restore confidence in a system that’s been behaving unpredictably, is still largely unanswered for most firms.
In short, the shift I’m observing is from “prevent every failure” to “ensure AI-enabled services fail safely”. That reframe changes everything about how you design controls, test readiness, and define recovery.”
When AI-enabled services fail in practice, what does that failure actually look like inside an organisation, and where does it tend to start?
In my experience, AI failures in production rarely look like the dramatic outages we plan for. They tend to start quietly and subtly: an output that’s slightly off, a recommendation that doesn’t quite fit the context, a process that takes longer than expected because the model is behaving inconsistently. Because the system is still technically running, the standard monitoring and alerting infrastructure doesn’t catch it.
What makes this particularly challenging in financial services is the dependency chain underneath most AI-enabled services. A model might be functioning correctly, but the data pipeline feeding it has degraded. Or the third-party API the service relies on has started throttling requests. Or a cloud infrastructure change upstream has introduced latency that nobody downstream has been notified about. The failure is real and consequential, but its origin is three layers away from where the business owner is experiencing the problem.
The other pattern I see is what I’d call escalation ambiguity, meaning nobody is sure who owns the response when an AI system starts behaving unexpectedly. Is it an IT incident? A model risk issue? A third-party vendor problem? In the absence of pre-defined escalation criteria specific to AI, response is slower, less coordinated, and more likely to result in inconsistent decisions about whether to keep the service running or take it offline.
Beyond internal controls, AI-enabled services increasingly depend on third-party infrastructure, external model providers and data pipelines – many of which are exposed to geopolitical and regulatory pressures organisations can’t directly control. Where are the blind spots, and what should firms be doing about them?
The most significant blind spot is concentration risk at the infrastructure layer. The majority of enterprise AI services, regardless of the organisation running them, are built on a very small number of hyperscaler platforms and foundation model providers. That means a disruption at one of those providers isn’t an isolated incident; it’s potentially a sector-wide event. Most business continuity plans treat third-party AI failure as an edge case. For firms with AI-enabled services at the core of their operations, it should be a base case scenario they’ve actively modelled and tested.
A second exposure is what I would call embedded AI – capabilities that have been built into enterprise software platforms organisations already use for compliance, operations, or client servicing, and which arrived as a feature update rather than a deliberate AI adoption decision. Many firms don’t have a complete picture of where AI is actually operating within their technology estate. That makes it very difficult to assess the resilience of something you don’t know you’re depending on.
The third area, and the one I think is most underestimated right now, is geopolitical exposure. Firms operating across the US and EU jurisdictions are starting to ask hard questions about what happens if a vendor becomes restricted, restructures its European operations, or is subject to regulatory action that limits access. Those aren’t hypothetical scenarios anymore. And most business continuity plans were written before geopolitical fragmentation became a live operational variable.
To summarise, I ask five questions about every AI dependency: Do you know it exists? Do you know what it does? Do you know what happens if it fails? Have you tested that? And could it become inaccessible overnight due to factors outside your control? Most firms, unfortunately, can’t answer all five.
When organisations move from theory to implementation, what does an effective approach to AI resilience actually look like in practice?
The starting point is always the same: map your AI service chain before you try to protect it. That means documenting not just the AI model or platform, but everything it depends on: the data sources, the APIs, the cloud infrastructure, the human processes that sit alongside it. You can’t design resilience for something you haven’t fully described.
From there, the work divides into three areas. First, design your degraded modes deliberately. For every critical AI-enabled service, define what the fallback looks like: whether that’s human-in-the-loop review, a rule-based alternative process, or a feature kill switch that disables the AI component while keeping the broader service running. These decisions are much harder to make well under pressure; they need to be made in advance and documented clearly.
Second, define your AI-specific incident triggers. Traditional incident management criteria, such as system downtime, error rates, SLA breaches, and so on, don’t reliably catch AI-specific failure modes like model drift, output degradation, or silent data pipeline issues. Organisations need escalation criteria that are sensitive to the ways AI systems actually break, and they need those criteria agreed across security, operations, and business teams before an incident occurs.
Third, test it. Run scenario-based exercises that specifically cut an AI dependency and observe what actually happens: not only whether the technical failover works, but also the people involved know what to do, who has authority to make decisions, and how long the manual fallback can realistically be sustained. In my experience, that’s where most organisations discover the gaps they didn’t know they had.
Governance investment has accelerated in response to regulatory pressure, but there’s a growing gap between compliance readiness and operational readiness. How do organisations close that gap, and what does “genuine” AI resilience look like beyond the documentation?
Governance investment has been genuinely important. Model risk policies, AI registers, EU AI Act compliance programmes – these create the structural conditions for responsible AI deployment. The challenge is that governance frameworks are designed to prevent foreseeable problems. They’re not designed to tell you what to do at 2 pm on a trading day when a critical AI service starts behaving unexpectedly.
The gap I see most consistently is between documented controls and operational readiness. An organisation can have a comprehensive AI governance policy and a fully populated model risk register, and still have no clear answer to “who makes the call to suspend this AI service, and what do we do in the meantime?” That’s not a governance gap, but rather it’s a resilience gap. And the two are frequently conflated.
There’s also a tendency toward what I’d call control theatre – the accumulation of documentation, sign-offs, and compliance artefacts that demonstrate governance activity without building the operational capability to respond when things go wrong. It’s a natural consequence of regulatory pressure, but it can create a false sense of security. Genuine AI resilience means the people involved know what to do when the model fails, not just that the right documents exist. The test isn’t whether your framework looks complete on paper. It’s whether it holds up when something goes wrong at the worst possible moment.
Governance tells you what to control. Resilience tells you what to do when the controls aren’t enough. Organisations need both, but they’re not the same thing, and right now they’re often being treated as if they are.
DORA has raised the baseline for ICT resilience in financial services, but it was designed before AI-enabled services became central to operations. Where does it fall short for AI specifically, and what are firms having to work out for themselves?
DORA has been valuable in raising the baseline expectation for ICT resilience across financial services. It creates clear obligations around third-party risk management, incident reporting, and continuity planning that have forced many firms to mature their approach significantly. But it was designed with traditional ICT systems in mind, and there are meaningful gaps when you try to apply it to AI-enabled services.
The clearest gap is around recovery objectives. DORA requires firms to define RTOs and RPOs – how quickly a service can be restored and how much data loss is acceptable. For a traditional IT system, those are relatively straightforward to define and test. For an AI system, the question is much harder. You can restore the infrastructure, but if the model has been behaving unexpectedly, has it been restored to a trustworthy state? How do you validate that? How long does revalidation take? There’s no regulatory guidance on this yet, and firms are working it out independently, often with very different answers.
There’s also limited regulatory direction on AI-specific incident classification: what constitutes a reportable AI incident as distinct from a standard ICT incident, and where the thresholds sit. Most firms are making pragmatic decisions in the absence of clear guidance. That creates inconsistency across the sector and means that when something goes wrong, organisations are navigating both the incident itself and the ambiguity around whether and how to report it simultaneously. That’s an area where I expect we’ll see more regulatory specificity coming, but right now it’s genuinely unresolved territory.
As more decision-making becomes automated, how does the role of people change when something fails?
This is the resilience paradox that I think deserves more attention than it currently gets. The more AI automates routine decision-making, the less opportunity people have to practise making those decisions manually. And when the AI fails, and you need a human to step in, that human may not have the experience, the confidence, or the current information to do so effectively under pressure.
The answer isn’t to slow down AI adoption; instead, it’s to be deliberate about preserving human capability alongside it. That means designing fallback processes that are workable, not just theoretically available. It means training people not only on how to use AI tools, but on what to do when those tools are unavailable. And it means running exercises that put people in the position of making decisions without the AI layer, so that muscle memory exists when it’s needed.
It also means rethinking what “human-in-the-loop” actually means in practice. In many implementations, human oversight has become a formality – a sign-off step that happens too quickly and with too little genuine scrutiny to function as a meaningful control. Effective resilience requires human involvement that is substantive enough to catch problems, not just cosmetic enough to satisfy a governance requirement.
Resilience by design means preserving human capability, not just human presence. There’s a difference between a person who is technically “in the loop“ and a person who can actually take over when the AI fails.
