Microsoft 365 went down on November 12 and stayed down for about a day. Outlook couldn’t reach mailboxes and email backed up for hours. The bug behind it wasn’t new. Microsoft had shipped it to its gateway servers in July, and it ran quietly for more than three months until two unrelated things happened at once: Teams traffic moved onto the gateways Exchange Online uses, and about 14% of gateway capacity was already offline for a different reason. That’s from Microsoft’s preliminary report, as summarized by Cyber Security News.

The bug explains why a few servers fell over. It doesn’t explain why the whole service did. When a gateway failed, Microsoft’s automation moved its connections to healthy gateways, which pushed them into the same bug, and in under two hours the failure had spread across regions. At the worst point more than 80% of servers had stopped responding. Engineers restarted them, and the restarted servers went straight back down as soon as the queued traffic reached them.

So the bug started the outage, and the systems built to recover from failures kept it going for a day. Distributed systems researchers have a name for that: a metastable failure.

What a metastable failure is

The term comes from a 2021 paper by Nathan Bronson and colleagues, Metastable Failures in Distributed Systems. A metastable failure is one where the system stays broken after whatever broke it has gone away. Every server is up and every CPU is busy, but almost nothing useful comes out the other end.

The paper describes three states a system can be in. In the stable state, load is comfortably below capacity and the system absorbs problems on its own. In the vulnerable state, the system is still healthy but close enough to its limits that a large enough trigger can tip it over. In the metastable state, it has tipped over and can’t get back by itself, because something inside the system keeps load above capacity even after the trigger is gone.

Stablerecovers on its ownVulnerablelooks healthyMetastablestays download risesload fallstriggertrigger removed: still stuckcut load sharply, then ramp back upMany production systems run in the middle box on purpose. It's efficient.
Figure 1. The three states from Bronson et al. The vulnerable state looks healthy on every dashboard until a trigger arrives.

The vulnerable state is the one to worry about, because it looks exactly like the stable one. Your dashboards are green, the daily peak goes through fine and the last load test passed. The paper points out that many production systems run there all the time on purpose, because it’s the efficient place to run. You only find out which state you were in when something pushes you.

The trigger is not the root cause

Every metastable failure has two parts. The trigger is the event that pushes the system over: a bad deploy, a load spike, a dependency that blips, a lost rack. The sustaining effect is the feedback loop that keeps the system overloaded once the trigger has passed.

Retries are the classic loop. Something slows the system for a few seconds, requests start to time out and clients retry them. Those retries are extra load on a system that’s already slow, so latency goes up again, more requests time out, and more retries arrive. By now the servers spend most of their time on requests whose clients have already given up. Throughput looks fine on the graphs, but goodput, the work that actually reaches someone still waiting for it, has dropped close to zero.

Triggerdeploy · blip · spikeLatency risesRequests time outClients retryLoad risesSUSTAINING LOOPkeeps running afterthe trigger is goneServers stay busy:throughput high,goodput near zero
Figure 2. The most common sustaining loop. The trigger starts it. The retries keep it going.

The paper’s argument, which I agree with, is that the loop is the real root cause. Marc Brooker, a distinguished engineer at AWS, put it plainly in his write-up of the paper: “We consider the root cause of a metastable failure to be the sustaining feedback loop, rather than the trigger.” Plenty of different triggers can push a system into the same loop. If your postmortem fixes the trigger and stops there, the next trigger will find the loop exactly where you left it.

That’s why “what broke?” is the wrong first question after an incident like this. It finds the trigger. The question that finds the loop is “why didn’t the system recover once the trigger was gone?”

How common this is

In 2022, Lexiang Huang and colleagues went through public incident reports from hyperscalers down to small companies and published Metastable Failures in the Wild, a study of 22 metastable failures at 11 organizations. A few of their findings stand out. At least 4 of the 15 major AWS outages in the previous decade were metastable failures. The outages lasted anywhere from 1.5 to 73.53 hours, most often four to ten. Retry policy was “by far” the most common sustaining effect, behind more than half of the incidents, and about 45% of the triggers were engineer errors such as bad configs, bad deploys and latent bugs. The most common way out was load shedding, used in over half of them.

0%25%50%75%100%share of incidents studiedRetries sustained the failuresustaining effect> 50%Load shedding used to recovermitigation> 50%Engineer error was a triggerconfig, deploy, latent bug~45%Load spike was a triggertrigger~35%
Figure 3. Findings from 22 metastable failures at 11 organizations, in Huang et al., OSDI 2022. Rows overlap: an incident can have several triggers.

The paper also splits the loops into two families. Workload amplification adds load, through retries or through each request becoming more expensive to serve. Capacity degradation takes capacity away, through garbage collection thrashing under memory pressure, caches going cold or healthy servers being pulled out of rotation. Both end up in the same place: the gap between load and capacity keeps widening instead of closing.

Case study: AWS us-east-1, October 2025

The best-documented example I know of is AWS’s post-event summary for the us-east-1 outage on October 19 and 20, 2025.

The trigger was a race condition in DynamoDB’s automated DNS management that left the regional DynamoDB endpoint with an empty DNS record. Errors started at 11:48 PM Pacific, and DNS was fixed by 2:25 AM. If the trigger were the whole story, the incident would have ended there.

It ran until the afternoon. EC2 has a subsystem called DropletWorkflow Manager (DWFM) that keeps a lease on every physical server in the fleet, and DWFM depends on DynamoDB. While DynamoDB was unreachable, those leases expired. When DynamoDB came back, DWFM tried to re-establish leases for the entire fleet at once, and the work took so long that leases timed out again before they finished, with retries piling up behind them. AWS’s own wording is that “DWFM had entered a state of congestive collapse.” The trigger had been fixed for almost two hours and the system still couldn’t make progress.

Engineers got out of it at 4:14 AM by doing what the research recommends: they “throttled incoming work and began selective restarts of DWFM hosts.” The restarts cleared the queues, and by 5:28 AM DWFM had re-established its leases.

Then a second loop started, this time in Network Load Balancer. Its health checks began putting new instances into service before their network configuration had propagated, so healthy nodes failed checks, passed, and failed again. All that flapping added load to the health check subsystem itself, which degraded and set off automatic DNS failover that pulled capacity out of the load balancers. The fix at 9:36 AM was to turn automatic failover off. EC2 was fully back at 1:50 PM, more than eleven hours after DNS was fixed.

12 AM3 AM6 AM9 AM12 PMDynamoDB DNSthe triggerEC2 lease managerDWFMLoad balancerhealth checksEC2 launchesand APIsempty recordleases expirecollapsethrottle + restartauto failover turned offflappingDNS fixed · 2:25 AMabout 11 more hours to recover
Figure 4. The us-east-1 outage, October 19–20, 2025, in Pacific time. Orange is time spent in a sustaining loop after the trigger was gone. Source: AWS post-event summary.

Both loops grew out of features meant to make the system more reliable, lease recovery in one case and health-check failover in the other. Retries, failover, health checks and automatic restarts all do their job when one thing fails. When a lot fails at once, they can be what keeps the system down.

How Microsoft 365’s failover spread the outage

The Microsoft 365 outage had a different loop with the same shape. The final report is due on November 20, so treat these details as preliminary.

The latent bug was lock contention inside a Windows process on the gateway servers. Under the right traffic it ate all the CPU and memory on the box. Microsoft’s testing missed it because the tests ran on weaker machines, and the contention got worse on newer hardware with more cores.

The loop was failover. Each gateway that fell over handed its connections to the healthy ones, which then had more traffic, hit the same contention and fell over in turn, handing their connections on to the next. The boundaries that should have contained the fault also failed, so it went global. Restarting a gateway didn’t help, because the queued traffic hit it the moment it came back.

1 · Triggerone server failsoutgatewaygatewaygatewaygateway2 · Failoverload moves onoutgatewaygateway3 · Restarttraffic returnsout~14% capacitymore load on fewer serversrestartedqueued traffic
Figure 5. How failover spread the Microsoft 365 gateway failure, simplified from Microsoft’s preliminary report as reported by Cyber Security News.

According to Microsoft’s status updates, the way out was more or less the opposite of failover. Engineers redirected traffic away from the broken infrastructure, worked on isolating regions so each could recover on its own, limited some types of traffic, rolled the fix out in phases and brought service back gradually, checking it was stable at each stage.

The loops to look for

Every incident has its own details, but the loops behind them come from a fairly short list.

Retries

A retry is a client asking the server to spend more time on its request, which is why AWS’s guide to timeouts and retries calls retries selfish. One client retrying once is harmless. The real damage comes from retries stacked across layers, because they multiply. Google’s SRE book gives the example of three layers that each make up to four attempts, which turns one user action into 64 attempts on the database. AWS works through five layers with three retries each and gets 243 times the normal database load, “making it unlikely to ever recover.”

EVERY LAYER RETRIESUser actionFrontendServiceData layerDatabase141664 attempts×4 per layerONE LAYER RETRIES, ON A BUDGETUser actionFrontendServiceData layerDatabaseretries hereevery other layer fails fast≤ 4 attempts
Figure 6. Retries multiply across layers. The example follows Google’s SRE book: four attempts at each of three layers.

Work nobody is waiting for

When a queue backs up, a server usually keeps working through it in the order requests arrived. By the time it reaches a request, that client timed out long ago and has already sent a retry, which joins the back of the same queue. Every request the server completes is wasted work and the queue never drains, which is how a system can show high throughput and almost no goodput.

Failover and health checks

Moving traffic off a sick server is the right call when one server is sick. When the whole fleet is overloaded, the same move concentrates load on fewer and fewer servers and knocks them over one at a time, which is exactly what happened to Microsoft’s gateways. Health checks that fail under load cause the same problem, because they pull overloaded but working servers out of rotation and push their share onto everyone else. AWS ended its NLB loop by turning automatic failover off.

Cold starts

A freshly restarted process has an empty cache, cold connection pools and code paths that haven’t been warmed up, so it serves each request more slowly than the process it replaced. If it gets a full share of traffic the moment it comes up, it falls straight back over. That’s what Microsoft’s restarted servers ran into.

Designing systems that recover

You won’t get rid of triggers. Deploys go wrong, dependencies blip and racks die. What you can control is whether a trigger causes a short dip or a stuck system, and that comes down to breaking the loops.

Retry at one layer, on a budget

AWS’s practice is “to retry at a single point in the stack,” and I’d make that a rule. Every other layer should fail fast and pass the error up instead of retrying on its own. Put the retries that remain on a budget. A token bucket lets them flow while the system is healthy and cuts them off when errors spike, and the AWS SDKs ship with one built in. The SRE book recommends both a per-request limit and a per-process budget. Back off exponentially and add jitter, so clients spread their retries out instead of all coming back at the same moment. And make sure anything you retry is idempotent, because otherwise the retry policy that protects you from a dropped request is the same one that charges a customer twice.

Shed load before it queues

An overloaded server should reject excess work immediately, with a 503, once in-flight requests or queue time pass a limit. A request rejected in a millisecond costs almost nothing. The same request sitting in a queue until its client gives up costs a full slot and produces nothing. Propagate deadlines through your calls too, so a server can drop a request whose caller has already stopped waiting.

Facebook’s Fail at Scale describes two queue policies built for exactly this. Controlled delay caps how long a request can sit in a queue, around 100 ms normally and around 5 ms once the queue hasn’t emptied in a while, so a standing queue never forms. Adaptive LIFO flips the queue to last in, first out when it backs up, so the newest requests, the ones whose users are still waiting, get served first.

Make failover aware of total capacity

Before moving load off a failing server, failover should check whether the servers that are left can actually take it. If they can’t, moving the load just moves the failure. Put a cap on how much capacity health checks and automatic failover can remove in a given window. When most of the fleet looks unhealthy at the same time, the cause is almost always overload or a shared dependency, and removing more servers only makes it worse. Treat fault-containment boundaries as real boundaries, too. A failing region shouldn’t be able to push its traffic into every other region.

Bring traffic back slowly

The SRE book’s recovery advice is blunt: fix the trigger, cut load sharply (it suggests as low as 1% of normal), let the servers get healthy, then ramp back up gradually. Restarting everything at full traffic recreates the exact conditions that caused the overload. Bring capacity back in stages, give caches time to warm, and check goodput at each step before taking the next one.

RESTART AT FULL TRAFFICtimeBRING TRAFFIC BACK IN STAGEStimeall waiting trafficfalls over, restarts, repeatscold capacitycapacity warming1%25%50%100%check goodput before each step
Figure 7. Why restarts fail during recovery. A cold server has less capacity than a warm one. The SRE book recommends cutting load sharply and ramping back up.

Build that throttle before you need it. In both of these outages, engineers had to work out how to limit traffic in the middle of the incident. A load-shedding switch, a per-region traffic cap and an admission rate limit should be tools you’ve already tested, not code you’re writing at 4 AM.

Measure the loop, not just the symptoms

Brooker asks the right question here: “Can you predict the next one of these metastable failures, rather than explain the last one?” You can’t predict a loop you aren’t measuring, so put its variables on your dashboards. Graph goodput next to throughput, because busy servers and unanswered users together mean you’re already in the loop. Track retries as a share of total traffic at every layer. Watch the age of the oldest request being served, not only queue length. And test recovery directly: inject a short dependency failure into a load test and time how long the system takes to come back. If it takes much longer than the fault lasted, you have a loop.

The short version