Queueing Theory in One Picture

A service sits at 70% utilization and looks fine. Traffic creeps up, it reaches 90%, and latency is three times worse. Nobody deployed anything. Requests just started spending most of their time waiting for a turn.

That waiting is what this post is about, and one chart shows it.

The picture

The chart describes a deliberately simple queue: one server, one line. The horizontal axis is how busy the server is. The vertical axis is total response time, waiting plus service, as a multiple of the mean service time.

Read it left to right. Both curves are nearly flat, then bend, then go vertical. The p99 starts higher and bends sooner. The two dots mark where each curve crosses the 20 ms target.

One formula

Let s be mean service time and u be utilization: arrival rate divided by service rate. Mean response time is:

Mean response = s / (1 − u)

In this model response time is exponentially distributed, so the p99 is ln(100), about 4.6, times the mean.

Notice the baseline. Even with almost no load, the p99 is 4.6 times the mean. That comes from variation in service times, not from queueing.

The model assumes random arrivals, random service times, one server, an unbounded queue, and no retries. Real services break most of those. The end of this post covers what survives.

Your tail breaks first

Say your p99 target is 20 ms and service time is 1 ms. In this model the p99 crosses 20 ms at about 77% utilization. The mean doesn’t get there until 95%.

A dashboard that watches average latency will look fine for another 18 points of utilization after users are already seeing slow requests.

So the headroom you need comes from your latency target, not from a universal number. 70% isn’t automatically safe and 80% isn’t automatically doomed. It depends on what you promised and how variable the work is.

CPU can point the wrong way

Slack’s January 2021 postmortem shows the other trap. Degraded networking made backend calls slow, so web workers spent more time waiting on them. Waiting workers don’t burn CPU. CPU utilization dropped, and that triggered downscaling.

The thing that was actually saturated was worker availability, and the metric being watched pointed the other way. The utilization that matters is the one on your real bottleneck: worker threads, connections, a lock, a hot partition. Not necessarily CPU.

Retries spend the headroom

In Retries as a Load Multiplier, the problem was sending more work to a dependency that was already struggling. The chart shows why that hurts before anything fails outright.

At 90% utilization, 5% more work puts you at 94.5%. Mean response goes from 10x to about 18x service time, and the p99 goes from 46x to about 84x. A small change in load nearly doubles latency.

Real retries are correlated with failures, so this understates the feedback loop. It shows why the first extra requests are expensive, not how a full retry storm behaves.

Traffic isn’t the only way to move right on the chart. A slower dependency raises service time. Removing instances during a rollout lowers capacity. A cache miss makes each request more expensive. Any of them can push you toward the wall without a single extra request.

Where the picture stops being literal

Production services have multiple workers, uneven partitions, bursty arrivals, and requests of different sizes. The 4.6 ratio won’t hold, and even the mean depends on how variable service times are, as the M/G/1 result shows. What survives is the shape: flat, then a bend, then a wall.

Overload also doesn’t have to mean unbounded latency. A bounded queue can reject work, deadlines can expire it, and load shedding can protect the requests you accept by turning the rest away. Amazon’s guidance on load shedding covers that well.

When you measure, keep queue wait separate from service time. Keep fast rejections out of your success latency, or a failing service will look faster because it’s doing less work.

Find where your p99 crosses your target. That, not 100%, is your capacity.