Jan 29, 2026·14 min read

Share|

Category:.NET

Polly retries making outages worse: stop retry storms with backoff and jitter

When retries amplify failures instead of fixing them: how retry storms happen in .NET, how to prove it, and the four components that stop cascading failures.

Free download: Polly Retry Policies package. Jump to the download section.

Paid pack available. Jump to the Axiom pack.

The Real Cost: A 45-Minute Outage from One Missing Parameter

The incident: A team running a critical service added Polly retries to handle transient network blips. They used the simplest form: Retry(3). It looked safe. But one Tuesday night, an external API they depend on went down for 2 minutes. Their service should have degraded gracefully. Instead, it cascaded into a full outage lasting 45 minutes.

Here's why: With no backoff policy, every failed request retried immediately. Three times. All in microseconds. With 500 concurrent users hitting the service across 10 instances, they generated 15,000 failed calls to an API that was already struggling. When the API recovered 2 minutes later, it got hammered by the retry backlog and fell over again. The service that was supposed to be resilient became the problem amplifier.

The cost: 45 minutes of downtime (full outage for external customers), 8 engineers firefighting, post-mortems, customer trust erosion. All because of one line of code that looked reasonable.

This is a retry storm, and it's one of the most dangerous resilience mistakes in .NET. What makes it dangerous is that it feels safe. "Retries are resilience, right?" Wrong: retries without boundaries are chaos amplification.

Fast triage table (what to check first)

Why retries amplify load during outages instead of recovering

Before implementing any retry policy, you need to understand what happens when you retry without care.

The mental model most teams have: "If a request fails, ask again. Simple."

What actually happens: You amplify load on a system that's already struggling. Here's the cascade:

1. A downstream dependency slows down or fails (database, API, cache, whatever).
2. Your service gets a failure. Retry immediately.
3. All requests retry at once (synchronized failure).
4. The downstream system gets 2x, 3x, 10x the load it was already choking on.
5. Downstream goes slower or dies completely.
6. More retries.
7. System spirals into unrecoverable state.

The worst part: Your observability sees "lots of retries happening" but misses that retries are causing the cascade, not recovering from it.

Why this happens: Most teams think retries are free. They are not. Each retry consumes concurrency. Each attempt holds resources (threads, sockets, connection pools). When you multiply load during a downstream failure, you can exhaust your own service first.

Why teams choose naive retries: They've heard "retries are good," they want resilience, and the simplest Polly code is easy to write. There's no alarm bell telling you that Retry(3) is dangerous until you're in the middle of an outage. This pattern appears frequently in background jobs and bots where "just retry" feels safe but causes silent cascades.

When retries make failures worse: transient vs permanent errors

Here's what nobody tells you about retries: They're only safe if you know what you're retrying and why.

Transient failures (network hiccup, temporary unavailability, 503 Service Unavailable) benefit from retries. Waiting a moment and trying again often works.

Permanent failures (404 Not Found, 401 Unauthorized, malformed request) should never be retried. The same failure will happen every time. Retrying does not help. It just wastes time and resources.

Cascading failures (downstream service is down) should never be retried with aggressive policies. If Database is gone, retrying your query 3 times instantly doesn't bring it back. It just kills your own thread pool faster.

The decision of when to retry is business logic. Getting it wrong costs you. Teams often don't distinguish between these cases, which is why they end up with policies that hurt instead of help.

Another tradeoff: Retries delay failure. If you are going to fail anyway (say, a vendor API is down for 10 minutes), retrying for 5 minutes means your customer waits 5+ minutes for a failure message instead of getting it immediately. Some paths should fail fast, not retry hopefully.

What to retry and what to fail fast: decision checklist

Not all retries are created equal. Here's how to think about what to retry:

Retry if:

• The error is transient (networking blip, temporary slowness, 503 Service Unavailable, connection timeout).
• The error is not your fault (you didn't send a bad request; the system is just busy).
• There's a reasonable chance waiting a bit will help (yes for network, maybe for 429, no for 404).

Don't retry if:

• The error is permanent (404, 401, 400, validation failure).
• The error indicates the downstream system is down and won't recover quickly.
• Retrying will consume resources faster than failure would (e.g., database is fully locked; retrying queries makes it worse).

This distinction is critical. A single retry policy for "all errors" is how you end up in the 45-minute outage scenario.

Four components that stop retry storms: backoff, jitter, caps, stop rules

There are four components to a safe retry policy. Miss any one, and you are back in trouble.

Component 1: Exponential backoff: Wait longer with each retry, doubling each time. First retry waits 1 second, second waits 2 seconds, third waits 4 seconds. Why? Because if the downstream system is struggling, hammering it immediately does not help. Waiting gives it time to recover. The first retry is fast (good for transient blips), later retries are slower (good for real outages where recovery takes time).

Component 2: Jitter: Add randomness to the wait time. Instead of "wait 1 second", wait "1 second + random 0 to 1 second". Why? If 100 requests all fail at the exact same moment, you do not want all 100 retrying at the exact same moment 1 second later. That is a thundering herd, and it kills recovery. Jitter spreads retries across a time range, so the downstream system gets requests more steadily instead of in synchronized waves.

Component 3: Caps: Do not let exponential backoff grow forever. Cap the max wait at something reasonable (30 to 60 seconds). After you have waited that long for a single request, you are not recovering. You are just tying up resources.

Component 4: Stop rules: After N retries (usually 3 to 5), give up. If it has not recovered by then, it probably will not for this request. Fail fast and let upstream callers handle it (with their circuit breaker, fallback, etc.).

All four matter. Any one missing and you're vulnerable.

Shipped asset: production-ready Polly package

A policy shape that works in production

The exact code varies by client and SLA. The shape does not.

1. Timeout budget first. One call has a maximum budget including retries, or you convert dependency failure into your own timeout cascade.
2. Retry only what is plausibly transient. 429, 5xx, timeouts, connection resets.
3. Backoff + jitter + cap + stop. Spread retries and stop after a small number.
4. Circuit breaker for cascades. If failures cluster, stop hammering and fail fast.
5. Structured retry logs. Retrying without a queryable event is how storms hide.

This is the smallest excerpt worth copying into a post (the full version is in the shipped package):

// Exponential backoff + jitter (capped)
var retry = Policy<HttpResponseMessage>
  .Handle<HttpRequestException>()
  .OrResult(r => (int)r.StatusCode >= 500 || (int)r.StatusCode == 429)
  .WaitAndRetryAsync(
    retryCount: 3,
    sleepDurationProvider: attempt =>
    {
      var exponential = Math.Pow(2, attempt - 1); // 1, 2, 4
      var jitterSeconds = Random.Shared.NextDouble();
      var seconds = Math.Min(exponential + jitterSeconds, 30);
      return TimeSpan.FromSeconds(seconds);
    },
    onRetry: (outcome, delay, attempt, _) =>
    {
      // log attempt, delay, endpoint, reason, correlation id
    });

What to log: make retries observable

Every retry should log context. You can't fix what you can't see. Log these fields every time:

{
  "timestamp": "2026-01-27T14:30:20.123Z",
  "event": "api_call_retry",
  "url": "https://api.example.com/orders",
  "failure_reason": "timeout / 503 / 429",
  "attempt_number": 2,
  "max_attempts": 3,
  "wait_seconds_before_retry": 2.456,
  "total_elapsed_seconds_so_far": 4.200,
  "elapsed_since_original_request_seconds": 4.200,
  "correlation_id": "req-abc123"
}

With these fields, you can:

• See which endpoints are flaky (high retry rates).
• Spot retry storms (many retries for same endpoint, all failing).
• Correlate retries with downstream outages.
• Measure recovery time ("how long did retries take before success?").

One additional metric matters in incident response: retry rate per instance. If it jumps, you are in the amplification phase.

Use correlation IDs to trace retries across services so you can see which original request spawned which retry attempts and how retries cascade through your dependency chain.

How to stop a retry storm in production right now

This is a production rescue lane, not a best-practices checklist. When you are paging, you need actions that reduce load now.

1. Confirm it is retries. Look for a jump in retry logs, especially clustered by one dependency and one endpoint.
2. Reduce concurrency first. Cap outbound concurrency per dependency. Fewer in-flight requests means fewer retries.
3. Tighten stop rules. Lower retry count temporarily. Lower cap. Shorten circuit breaker sampling windows.
4. Enforce a total budget. If the downstream is down, do not spend minutes retrying per request.
5. Degrade intentionally. Return partial data, cached data, or an explicit 503 quickly. Do not hold threads.
6. After stabilization: classify which statuses/exceptions are retried, and add tests so a future change cannot reintroduce Retry(3).

If you are also seeing requests timing out with normal CPU, treat that as thread pool starvation and fix time budgets and cancellation across the call path first.

Tradeoffs

• Retries can hide deterministic failures. Poison inputs, auth failures, and invalid requests must stop and surface immediately.
• Retries add latency. That latency must fit inside a request budget, or you convert a dependency outage into your own outage.
• Circuit breakers change failure modes. That is the point. Make sure the caller path is designed to handle fast failures.

Troubleshooting Questions Engineers Search

Additional Questions

Resources

• Polly Documentation: Retry
• Polly Documentation: Circuit Breaker
• AWS: Exponential Backoff and Jitter
• Related: Thread pool starvation: the silent killer of ASP.NET performance
• Related: Timeouts first: why infinite waits create recurring outages in .NET

Checklist (copy/paste)

• Total time budget is defined per dependency call path (not just “retry 3 times”).
• Per-attempt timeouts exist and are smaller than the total budget.
• Retryable errors are explicitly classified (timeouts/429/selected 5xx); permanent failures fail fast.
• Backoff is exponential and capped (no unbounded waits).
• Jitter is applied (no synchronized retry waves).
• Max attempts is small and enforced (start 2–4).
• Circuit breaker opens on sustained failure clusters (fast failure during outages).
• Concurrency is capped around flaky dependencies (bulkhead) to prevent amplification.
• Retry logs include: endpoint/dependency, attempt, delay, reason, total elapsed, correlation id.
• Alert exists for retry rate spikes per dependency/endpoint.

Coming soon

If you want more runbooks like this (plus logging schemas and decision trees), that is what Axiom is becoming.

Join to get notified as we ship new operational assets for reliability.

Key takeaways

• Naive retries cause retry storms. Retry(3) without backoff amplifies load and cascades failures.
• Classify what you retry. Transient errors benefit from retries; permanent errors do not.
• Backoff + jitter + caps + stop rules are not optional.
• Circuit breakers prevent cascades by forcing fast failure.
• Observable retries are debuggable retries. Log every attempt and alert on retry rate.

If you want a fast diagnosis of a retry storm, see .NET Production Rescue or contact me and include one example retry log event plus the endpoint and status distribution.

Related posts

.NETJan 26, 2026

Retries making outages worse: when resilience policies multiply failures in .NET

Retry storms don't look like a bug—they look like good engineering until retries amplify failures and multiply in-flight requests during backpressure.

.NETFeb 04, 2026

Idempotency keys for APIs: stop duplicate orders, emails, and writes

When retries create duplicate side effects, idempotency keys are the only safe fix. This playbook shows how to design keys, store results, and prove duplicates cannot recur.

.NETJan 30, 2026

HttpClient keeps getting 429s: why retries amplify rate limiting in .NET

When retries multiply 429 errors instead of fixing them: how retry amplification happens, how to prove it, and how to honor Retry-After with budgets.