When a Cloud Provider Goes Down: Keep Your AI App Running

What you'll learn#

• Why an app that calls one AI model in one cloud goes down when that cloud does, and how to spot that weak part.
• Why waiting longer and trying again forever makes an outage worse, and what a safe retry looks like.
• How a timeout, a backup model and a plain fallback keep an app answering.
• When a backup is worth its cost, and when a clear sorry message is enough.

What to know first: what a large language model is ([What is an LLM?](/ai-engineering/what-is-an-llm/)) and that your app reaches one over the internet with an API call, a request your code sends to the model's company, which sends an answer back. That's all you need.

The problem: your app depends on someone else's computers#

You open Teams for a call, and it won't load. On the twelfth of November 2026, Microsoft's email, Teams and Copilot went down worldwide, for many hours. Microsoft said there was no sign of an attack, and it rolled back a recent network change whose timing matched the start; the full cause was promised in a post-incident report. For most people it was a lost morning. For anyone whose own app ran on those computers, it was a lost day of business.

Now meet Aaryan, who runs a small cake shop. Customers order on a chat in the shop's app. An AI model reads each message and fills in the order: which cake, which day, and the words for the icing. The model lives in one company's cloud: computers you rent over the internet instead of owning. Say that cloud goes down at six on a Friday evening, the busiest hour, and stays down until eight.

Nothing in Aaryan's code broke. The app did exactly what it was told: send every order to one model, in one cloud. The question this lesson answers is how to make the app keep taking orders when that cloud stops answering.

The big picture: four layers of defence#

Think of a dual-SIM phone. When one network has no signal, the phone switches to the other SIM, and you keep making calls. An app can do the same thing with its AI model. Here is the whole plan, from the order arriving to the order being saved.

Each layer catches what the one before it missed. The rest of this lesson takes them one at a time, each on the same Friday evening.

Stage 1: find the single point of failure#

A single point of failure is a part your system can't work without, and has no spare for. If it stops, everything stops. The first job is to list every part one order passes through and ask, for each one: if this goes down, who is hurt?

The column "Who is hurt" has a name engineers use: the blast radius, how much of the system one failure takes down. A broken phone has a blast radius of one person. The AI model's cloud has a blast radius of every customer, and no spare. That row is the single point of failure.

Stage 2: stop waiting with a timeout#

When a cloud is down, a request often doesn't fail at once. It hangs. Without a limit, Aaryan's app waits, the customer stares at a spinning wheel, and the app's own workers fill up with waiting requests. A timeout is the most a request may wait before your code gives up on it, like hanging up when nobody answers after ten rings.

Most model client libraries let you set a timeout on each call. Set it on purpose: the default is often long, because the library can't know how patient your customers are.

Stage 3: retry, but carefully#

Kelvin, who helped build the app, has a quick fix: make it try again until it works. Trying again is right for a one-second blip, when one request in a hundred fails by chance. It is wrong for a two-hour outage. Four tries of thirty seconds each is two minutes of spinning, and then the order still fails.

Worse, retries add load. If every app tries four times, a cloud that is already struggling gets four times the traffic. That is a retry storm, and it can keep a cloud down after the first fault is fixed. This isn't only theory: in November 2024, Microsoft said a change set off a flood of retry requests to its servers, and Outlook and Teams were disrupted for hours.

The safe version has three rules. Retry only a few times. Wait longer before each try, doubling the wait: that is exponential backoff (taught in full in [Retries and fallbacks](/ai-engineering/llm-retries-and-fallbacks/)). Add a little randomness to each wait, called jitter, so a thousand apps don't all retry in the same second. Then stop and move on.

Stage 4: switch to a backup model#

This is the dual-SIM move. When the main model fails, send the same order to a fallback model: a second model your app uses only when the first one can't answer. To protect you from a whole cloud going down, it has to run somewhere else, ideally at a different company.

Here the dual-SIM picture breaks. Two SIMs carry the same call; two models don't give the same answer. The backup may word the icing differently, read dates another way, or follow your prompt less closely. So Aaryan runs real past orders through the backup before the bad day comes, and fixes the prompt where its answers differ. A backup nobody tested is a second surprise, not a spare.

Stage 5: keep working without AI#

What if both models are down? The app can still do a smaller version of its job. That's called graceful degradation: when a part fails, the system keeps working in a simpler way instead of stopping, like a shop that takes cash when the card machine is down.

• A plain form: cake, day, icing words. No AI needed; the customer types the fields the model would have filled in.
• Save and reply later: put the message in a queue, a waiting list the app works through when the model is back, and tell the customer when to expect a reply.
• Saved answers: for common questions ("Are you open Sunday?"), reply from a short list of answers written in advance.
• An honest message: "Our assistant is down. Orders still work through the form." Better than a spinning wheel.

The point is to decide in advance which parts of your app need AI and which only use it to be nicer. For Aaryan, the AI fills in the order; it isn't the order. The form keeps the shop open.

Stage 6: know when it's down#

Every step above needs the app to notice the failure quickly. Three tools do that.

A circuit breaker works like the fuse box in a house. After a few failures in a row, it "trips": the app stops sending orders to the main model and goes straight to the backup, so no customer waits for a timeout. Every minute or so it lets one test request through; when that succeeds, it closes again. Don't rely on the provider's status page alone. During big outages these pages have sometimes updated late, so your own checks should decide.

A code sketch: try each model, then fall back#

def main_model(message):          # pretend the main cloud is down
    raise TimeoutError("no answer in 10 s")

def backup_model(message):        # a different company's cloud
    return {"cake": "chocolate", "day": "Saturday", "icing": "Happy birthday Riya"}

def take_order(message, models):
    for name, model in models:
        try:
            order = model(message)    # a real call sets timeout=10
            return f"Order from {name}: {order}"
        except Exception as error:    # down, too slow or refused
            print(f"{name} failed: {error}")
    return "Sorry, our assistant is down. Please use the order form."

message = "chocolate cake, Saturday, happy birthday Riya"
print(take_order(message, [("main", main_model), ("backup", backup_model)]))
print(take_order(message, [("main", main_model)]))

• main_model stands in for the call to the main provider. Here it always raises TimeoutError, the error a real client raises when the 10-second timeout runs out.
• backup_model stands in for a model at a different company. It returns the filled-in order.
• take_order gets a list of models in order of preference and tries each one.
• try and except catch any failure, so one model's error moves the loop on to the next model instead of crashing the app.
• The print inside except is the log line an engineer reads after the outage.
• If every model fails, the last line returns the plain-form message: graceful degradation.
• The first print call has a backup, so the order goes through. The second has only the main model, so the customer gets the form.

Common misconceptions#

• "Big clouds don't go down." Every cloud goes down sometimes. The twelfth of November 2026 proved it, and it wasn't Microsoft's first time. Big clouds are very reliable, which is exactly why few teams plan for the day they aren't.
• "Retrying until it works is the safe choice." Unlimited retries make customers wait and pile extra load on a cloud that's already struggling. Retry a few times with backoff and jitter, then switch.
• "Two regions of the same provider is a full backup." It covers a local fault, not a mistake the provider pushes everywhere at once. For that you need a second provider, or a way to work without AI.
• "The backup model will behave the same." Different models read prompts differently. Test the backup on real past inputs before you need it.
• "Every app needs two clouds." A weekend side project doesn't. A second provider means two bills, two sets of keys and twice the testing; a clear sorry message may be enough.

In real work#

AI engineers meet this the first time a provider has a bad day and the support team asks why the assistant is down. The work that follows is usually small and very worth it:

• Set timeouts on every model call, a little above your normal answer time.
• Add one fallback model at a second provider, behind the same function your app already calls. Some teams put an LLM gateway in front of all their models: one service that routes each request and switches providers on failure (taught in [Design an LLM gateway](/ai-engineering/design-llm-gateway/)).
• Test the fallback on a set of real past requests and compare its answers with the main model's.
• Decide the no-AI path: the form, the queue or the honest message, and show it in the app.
• Watch it: a health check, an alert when the circuit breaker trips, and a log of which model answered.

The trade-off is cost against risk. A backup provider costs little while unused if you pay per request, but it costs engineering time to keep its prompts working. Ask what one lost hour costs you. For Aaryan, a lost Friday evening is real money; for a hobby app, it isn't.

Recap#

1. Find the single point of failure: the part everyone depends on with no spare.
2. Set a timeout so a hanging cloud costs seconds, not minutes.
3. Retry carefully: a few tries, backoff and jitter, then stop.
4. Fall back to a tested backup model in a different cloud.
5. Degrade gracefully: a plain form, a queue or an honest message when both are down.
6. Notice fast: health checks and a circuit breaker, not only the status page.

Practice#

1. Recall: why does trying again until it works make a big outage worse?
2. Recall: why isn't a second region of the same cloud a full backup?
3. Challenge: list every part a request to your own app (or one you use) passes through. Mark each one's blast radius and whether it has a spare. Which row is your single point of failure?

Practice

Check your understanding

FAQ

Frequently asked questions

Sources

Where the facts come from

• GeekWire: Microsoft 365 starts to recover after a day of global email turmoil (Nov 2026)
• Euronews: Microsoft outage explained, what went wrong (13 Nov 2026)
• ThousandEyes: Microsoft outage analysis, November 25, 2024
• AWS Builders' Library: Timeouts, retries, and backoff with jitter
• Google SRE book: Addressing cascading failures
• AWS Well-Architected: Implement graceful degradation