Ship Fast. Recover Faster.

CI can prove that code passes the tests we wrote. It cannot prove how the code behaves under production traffic.

Production has uneven load, stale clients, slow dependencies, hot partitions, and user behavior we did not predict. Shipping exposes those conditions. That is why an early release can teach us more than another week of pre-release polish.

It is also where most of the risk sits. Google’s SRE teams report that roughly 70% of their outages come from changes to a live system. “Ship fast” only works when each release has a bounded blast radius. Without telemetry and a rollback path, it is just a faster way to move risk into production.

Release one hypothesis

Suppose we replace a queue consumer because the current one is too slow. The first release should answer one question: can the new consumer process real messages without increasing failures or database contention?

Deploy the new code, but do not send all traffic to it. Start with a small, fair slice of messages and compare it with the old consumer. If the result is healthy, widen the slice. If it is not, stop the rollout.

Do not combine that test with a schema rewrite, a new retry policy, and a queue migration. When several things change at once, a failure is harder to attribute and harder to reverse.

Small releases are easier to review, measure, and roll back.

Splitting a queue is not splitting HTTP traffic

A load balancer can send 1% of requests to a new version. A queue usually cannot, and the obvious approaches fail in different ways.

rollout / traffic split queue

With Kafka consumer groups, each partition belongs to exactly one consumer in the group. Starting the new consumer in a separate group does not split the traffic. Both groups read every message, and every side effect runs twice.

Partition ownership is a real split, but a coarse one. Give the new consumer one partition out of eight and the canary share is 12.5%, not 1%. That partition may also be hotter or colder than the others, which skews the comparison before the new code does anything.

A per-message flag is the most controllable option. One consumer group reads everything, and the handler routes each message by hashing a stable key: hash(key) % 100 < rollout_pct. The percentage can change at runtime, and each key stays on one path, so the same entity is not handled by both implementations at once. When the percentage rises, some keys move from the old path to the new one, so both paths must handle any state the other left behind.

Kafka’s share groups, production-ready since Kafka 4.2, let several consumers read the same partition with per-record acknowledgement. They do not give you a rollout dial either. If old and new consumers join one share group, the split depends on how many consumers there are and how fast each one fetches, not on a percentage you chose.

Whichever split you use, keep each unit of work in one pool. The SRE workbook makes the same point for multistage pipelines: if a work unit moves between canary and control workers, the signals mix and neither side can be measured cleanly.

Separate deployment from release

Deployment puts code in production. Release sends production work to that code. Treat them as separate operations.

The new consumer can be deployed at zero traffic. A feature flag or rollout controller can then increase traffic without another deployment. The same control should send traffic back to the old path.

rollout / deploy vs release flag

A kill switch that requires a code change, review, build, and deployment is not a kill switch.

A flag is fast, but it is not instant. Check three details before you depend on it:

• Propagation. Many server SDKs poll for changes. Unleash’s server SDKs, for example, refresh every 15 seconds by default. Streaming SDKs are faster, but every SDK caches. Measure how long “off” takes to reach every instance.
• The default value. Under the OpenFeature specification, a failed evaluation returns the default value supplied in code. If the flag service is unreachable, or the SDK has not finished initializing, that default decides where traffic goes. Make the default the old path.
• In-flight work. Turning a flag off affects messages evaluated after the change. Messages already inside the new handler finish there. Read the flag per message, not once at startup, and know how long your slowest message takes.

Useful release controls include:

• A server-side feature flag with a known owner.
• A canary percentage or explicit tenant allowlist.
• Automatic rollout pauses when a threshold is breached.
• A tested command that sends traffic back to the previous path, with a measured time to take effect.

Define the telemetry first

Before activation, define the metrics that allow the rollout to continue and the metrics that stop it.

For the queue consumer, I would watch:

• Message failure and retry rates.
• Queue depth and age of the oldest message.
• Processing latency at p50, p95, and p99.
• Database connection-pool utilization and wait time.
• Throughput split by partition and tenant.

Break every metric down by path: canary and control. A fleet-wide number hides a broken canary. The SRE workbook’s example is a 5% canary failing 20% of its requests, which shows up as only a 1% error rate for the whole service.

Set the threshold and evaluation window before the release. For example: stop if the canary’s failure rate exceeds the control by one percentage point for five minutes. A vague instruction to “watch the dashboard” is not a stop condition.

rollout / production streaming

$ deploy --canary

deploy → 1% → 5% → 25% → 100%

↘ threshold breached

rollback_

The window needs enough messages

A threshold is only as good as the sample behind it. Suppose the control fails 0.5% of messages and the canary actually fails 2.5%, five times as often. Under the rule above, here is how a single five-minute window behaves:

Canary messages in the window	Bad canary passes	Healthy canary is stopped
200	about 26%	about 2%
2,000	under 0.2%	effectively never

At low volume, a canary that fails five times as often still passes about one window in four. Size the window by message count, not only by minutes. A low-volume queue needs a longer window, a larger canary, or a proper statistical test. Netflix’s Kayenta, for example, compares canary and baseline metrics using the Mann-Whitney U test instead of a fixed gap.

Two timing rules from the SRE workbook also apply. A metric’s aggregation interval should be no longer than the canary step, because an errors-per-hour metric cannot judge a 30-minute step. And each step should last at least as long as one unit of work, which matters for consumers that process long-running jobs.

Some failures only appear at higher load

A 1% canary finds crashes, parse errors, and bad configuration. It rarely finds contention. Connection-pool saturation, lock conflicts, and consumer lag grow with load, so a rollout can look perfect at 1% and fail at 50%.

rollout / ramp hold per step

Hold each ramp step long enough to evaluate it, and expect the larger steps to surface different problems than the first one. The SRE workbook notes that performance defects typically appear only under heavy load, so a ramp that runs entirely off-peak can pass without ever meeting the conditions that matter.

Compare against absolute limits too

The canary and control often share a database. If the new consumer saturates the shared connection pool, the control slows down as well. A relative rule, canary versus control, then sees no difference while both paths get worse.

Pair each relative threshold with an absolute one tied to your SLO, such as the age of the oldest message or p99 processing latency. The rollout stops if either one is breached.

Watch the tail

Average latency is not enough. A consumer can look healthy on average while one hot partition falls behind or requests wait for a saturated connection pool. The tail is often the first evidence that the rollout is unsafe.

telemetry / p99 by partition breach

Rollback must include state

Rolling back a binary is easy. Rolling back state is not.

If the new consumer writes an event that the old consumer cannot parse, restoring the old binary will not recover the system. The same problem appears with destructive schema migrations, duplicate payments, and one-way calls to external systems.

A log makes this sharper. Rolling back the code does not remove the events the new version already wrote. They stay in the topic until retention expires, and anything that replays the topic will read them again.

Keep the old and new versions compatible during the rollout:

• Expand a schema before removing or changing existing fields.
• Make readers accept both event versions before writers emit the new one.
• Use idempotency keys for side effects that may be retried.
• Remove the old path in a later release, after the new path is stable.

This is parallel change, also called expand and contract. The order of the steps decides which rollbacks are safe.

schema / parallel change readers first

The version you roll back to must already understand everything the version you are leaving has written. Schema registries encode the same rule. Under Confluent Schema Registry’s default BACKWARD compatibility, you upgrade all consumers before producing events with the new schema. Contract only when no v1 event remains that a consumer might reread.

Moving traffic replays messages

Every rollout step and every rollback moves work between consumers. In Kafka, that usually means a rebalance, and a rebalance exposes uncommitted work.

kafka / rebalance at-least-once

A Kafka commit records the next offset to read. The new owner of a partition resumes from that committed offset, not from the last message the previous owner processed. Anything processed but not yet committed runs again. This is ordinary at-least-once behavior, and the rollback path hits it just as often as the rollout does.

That makes idempotency part of the rollback plan:

• Derive the idempotency key from the message, such as its event ID, not from the attempt. A retry must produce the same key.
• Use the same key derivation in the old and new versions. If the new consumer builds keys differently, a message replayed across versions looks new to the downstream system.
• Keep keys longer than the longest possible replay. Stripe, for instance, may prune idempotency keys once they are 24 hours old, and a request reusing a pruned key is treated as new.
• Commit offsets only after the side effect completes, and stop processing a partition as soon as its ownership is revoked.

A five-minute deployment with a two-hour recovery procedure is not fast delivery. It is fast activation followed by slow failure.

Coding agents increase release pressure

Coding agents reduce implementation time. They do not reduce production uncertainty.

DORA’s research on AI-assisted development found that higher AI adoption is associated with higher delivery throughput and higher delivery instability. More code reaches production sooner, and the release process has to absorb it.

A small request can arrive with retries, compatibility branches, and synthetic resilience added around it. Those paths increase the review surface and create behavior that may only run during a failure.

Keep generated changes narrow. Review every fallback and side effect. Look hardest at anything that touches idempotency keys, event schemas, or offset commits, because those decide whether rollback works. Release behind the same canary and rollback controls used for human-written code.

Faster code generation should produce smaller releases more often, not larger unreviewed releases.

Match the first release to the risk

“Ship first” does not mean every experiment should reach every customer.

Money movement, security controls, safety systems, and destructive data changes need stronger controls. The first release might be an internal workflow, a read-only mode, or shadow traffic.

Shadow traffic is only safe when duplicate execution cannot create side effects. For a queue consumer, that usually means a separate consumer group whose writes go to a scratch store or nowhere. Even then, the shadow shares infrastructure. It adds load to the same database and can warm the same caches. The SRE workbook warns that when a teed copy of traffic shares a cache with production, the inflated hit rate invalidates the performance measurements.

The goal is still early feedback. The allowed environment and blast radius depend on the cost of failure.

The p99 of shipping

Most deployments are uneventful. The new code receives traffic, the metrics stay within limits, and the rollout completes.

The p99 deployment finds the undocumented dependency, hot partition, or incompatible event. That release tests the controls around the code: detection, rollout pause, traffic shift, replay safety, and state compatibility.

DORA treats recovery as part of delivery speed. It renamed “time to restore service” to failed deployment recovery time in 2023, and in 2024 it moved that metric from the stability group into throughput. Recovering from a bad change is itself a change reaching production.

Ship a small change. Define the stop condition. Start with a fair canary. Keep the old path compatible until the new path is stable.

Shipping speed gets code into production. Recovery speed determines whether you can do it safely again tomorrow.