Retries as a Load Multiplier

A retry is another request. If the first one failed because the system had too much work, the second one arrives at exactly the wrong time.

That sounds obvious until you read the code. The retry sits in an error handler, next to a timeout and a reassuring name like resiliencePolicy. Locally, it gives one operation another chance. Across a fleet, it changes the traffic the dependency must survive.

GitHub’s August 17, 2026 outage is a recent example. Capacity pressure started the failure. Retry behavior made it harder to recover.

GitHub: recovery had its own bottleneck

According to GitHub’s own incident analysis, the incident ran from 13:28 to 21:15 UTC: 7 hours and 47 minutes. That does not mean every affected service was unavailable for the entire window. Most services recovered by 16:36 UTC; Actions remained degraded until approximately 18:03, and the Copilot Token Service fully recovered by 21:02.

A new traffic peak pushed infrastructure in the Central US datacenter beyond its limits. An Istio sidecar reached its concurrency limit but did not scale correctly: its autoscaling policy watched the host service, not the sidecar’s limits. The failure spread until four HAProxy nodes exhausted their network flow limits, degrading the shared authentication path.

GitHub’s optimistic retry logic added pressure to the overloaded load balancers. Pausing HAProxy on the affected nodes produced immediate broad recovery.

But recovery was not finished. Delayed replies from an internal endpoint triggered a latent VS Code retry bug. Copilot Token Service traffic rose from its normal 7,000-9,000 requests per second to 70,000-100,000. GitHub describes approximately tenfold traffic amplification. That figure belongs to this endpoint, not the whole platform or the total damage from the outage.

GitHub reduced gateway authentication retries, blocked inbound token requests with a 403, and restored traffic gradually. The specific status code was an incident mitigation, not a general recommendation to return authentication errors during overload.

The distinction matters: adding capacity or repairing the initial bottleneck would not, by itself, stop clients from producing the recovery load. GitHub had to interrupt the retry behavior too.

In its August 20 follow-up, GitHub described the outage as a capacity failure, not an incident triggered by a new code or configuration change. A previously inadequate scaling policy can be part of the cause without a configuration deployment being the trigger.

Three retries is not necessarily three attempts

Imagine a request passing through an application, a service client, and a database client. Each layer permits three retries after the initial attempt.

That is four attempts per layer. If each outer retry reruns the inner operation, and failures exhaust every layer’s allowance, one logical request can produce 64 database attempts.

This is a worst-case illustration, not a measurement of GitHub’s incident. Google’s SRE guidance uses the same example to explain why independently reasonable policies can become a combinatorial problem.

Even one retry layer can consume the headroom you thought you had. Suppose, purely as an illustration, 1,000 logical operations arrive each second and 20% need one extra attempt. That produces 1,200 attempts per second before accounting for further failures. If each attempt has the same cost and the dependency can sustain 1,100, the retry policy has pushed it beyond capacity.

The real system is less tidy. Some rejections are cheap; some failures happen after expensive work. Cache misses, connection setup, authentication, and database queries have different costs. Count attempts, but also measure the resource they exhaust.

This pattern is not specific to GitHub

During the AWS US-EAST-1 event on December 7, 2021, an automated scaling activity triggered unexpected client behavior and a surge of connections between AWS’s internal and main networks. The network devices became congested. Delays produced more connection attempts and retries, keeping the congestion going.

AWS reported that a latent issue prevented clients from backing off adequately. The affected code path had been in production for years. Recovery involved reducing traffic, isolating heavy sources, and adding network capacity. The incident also impaired monitoring, making the feedback loop harder to diagnose.

The lesson is not that AWS lacked a backoff policy. It is that having one was insufficient: the policy did not behave as expected under the conditions that mattered.

There are smaller, recent variations too. In GitHub’s August 6, 2026 Actions incident, capacity loss during a routine rollout caused saturation. After core services began recovering, a job-assignment bug handed runners jobs that had already been revoked. Runners kept retrying invalid work instead of acquiring valid jobs, creating a self-amplifying backlog.

Waiting longer would not make a revoked assignment valid. The system needed to stop retrying that operation.

In GitHub’s July 19-20, 2026 runner incident, an expired certificate disrupted connectivity for self-hosted and larger runners. Reconnection traffic increased load on GitHub APIs, adding three to four seconds of average request latency and elevated 5xx errors. The initial certificate problem and the secondary API load were different parts of the incident.

Recovery traffic is broader than retries. During Cloudflare’s June 12, 2025 outage, services began recovering as underlying storage returned, but cache repopulation caused continued errors and infrastructure rate limits. Cloudflare identified progressive re-enablement of namespaces as follow-up work to avoid overwhelming its own infrastructure during recovery. This is a cache-refill example, not evidence that retries caused the original storage failure.

A backlog, a reconnect wave, and a retry storm are not identical. They do share a capacity problem: work accumulated or multiplied during failure can overwhelm a dependency as it returns.

Backoff is necessary, but it is not a traffic budget

Exponential backoff spreads a client’s attempts over time. Jitter reduces the chance that many clients retry on the same schedule.

Neither guarantees that the fleet’s total traffic fits inside the dependency’s capacity. Enough clients, each retrying slowly, can still produce too much work. A capped backoff without a stopping rule can become a permanent background load.

The Amazon Builders’ Library recommends bounding retries and avoiding independent retry multiplication across layers. It also describes local token-bucket retry throttling. Google’s Handling Overload chapter distinguishes a per-request attempt limit from a per-client retry budget. One bounds what a single operation can do; the other bounds how much extra traffic the client can generate across operations.

For a service I own, I would make these decisions explicit:

• One retry owner for a given dependency call. Check the SDK, proxy, service mesh, and application. A wrapper can add retries to a client that already has them.
• An attempt limit and an overall deadline. Connection setup, attempts, and waiting all spend the same operation budget. Do not reset the deadline at every hop.
• A bounded retry allowance across operations. Define what happens when it is exhausted, and scope it to the dependency being protected. A per-process allowance grows when the fleet grows; it is not a global cap.
• Failure classification. Retry only failures the operation’s contract identifies as transient. Invalid input and revoked work need a different response. Respect server backpressure and Retry-After when applicable, without exceeding the overall deadline.
• Capped backoff with jitter. Spread attempts without making the request live forever.
• Safe side effects. A timeout means the caller did not receive a result. It does not prove a write never happened. Reuse an idempotency key for the same logical operation where the API supports it.
• Admission and concurrency limits. Protect the dependency even if a caller ignores the retry policy. Reject excess work early and cheaply where possible.

Idempotency prevents duplicate effects; it does not make duplicate attempts free. The request may still need parsing, authentication, and a deduplication lookup.

Circuit breakers can help, but they are not a universal substitute for these limits. Amazon also warns that breakers introduce operating modes that can be difficult to test and can delay recovery. If you use one, test how it opens, probes, and restores traffic. Do not let every instance probe the recovering dependency at once.

Back to the queue

In Skill Is a Dependency, I described a hypothetical queue consumer whose lag was growing while connection-pool wait time climbed. The tempting fix was more retries and more consumers.

If consumers are waiting for a saturated database pool, retries add more attempts to the same constrained resource. Adding consumers can add more concurrent waiters. Neither creates database capacity.

I would first compare logical messages completed with database attempts issued. Then I would check pool wait time, active connections, retry outcomes, and the age of the oldest message. If attempts rise while useful completions fall, more activity is not more progress.

Bound consumer concurrency against the dependency’s capacity. Give transient failures a delayed, bounded retry path, and give permanently invalid messages an explicit quarantine or dead-letter policy. Replaying that queue needs a rate limit too: otherwise the recovery procedure recreates the overload.

The right intervention depends on the bottleneck. A slow query needs a different fix from an unavailable partition or a broken assignment. The retry cannot make that diagnosis for you.

Test the return to service

A test that fails once and succeeds on the next call proves very little about overload.

Test a slow dependency, not only one that returns errors. Let many clients fail together. Leave their ordinary retry policies enabled, restore the dependency, and watch what happens. Repeat with cold caches and a backlog. Keep these experiments in a controlled environment or an explicitly bounded test slice.

Measure:

• Attempts per logical operation, separated by retry layer and dependency.
• Operations rescued by retries versus attempts that still fail.
• In-flight work, connection-pool wait time, and rejected work.
• End-to-end latency at p50, p95, and p99, including retry waits and terminal failures.
• Time for useful throughput and backlog age to return to normal after recovery.

Do not judge the policy from successful requests alone. Retries can improve the apparent success rate while increasing latency, consuming headroom, or delaying everyone else’s work.

Retries are useful. Slack’s January 4, 2021 postmortem explicitly credits retries, circuit breaking, and load-balancer panic mode with helping restore service while its network was still degraded. That is a useful counterexample to the claim that retrying a failing system is always wrong.

The question is whether another attempt has a reasonable chance of succeeding, whether its effects are safe, and whether the system can afford it.

Normal traffic tells you what the service costs when it works. Failure traffic tells you what your resilience policy costs when it doesn’t.

A retry policy should define both how to try again and when to stop.