Idempotency keys for APIs: stop duplicate orders, emails, and writes

Feb 04, 20267 min read

Category:.NET

Idempotency keys for APIs: stop duplicate orders, emails, and writes

When retries create duplicate side effects, idempotency keys are the only safe fix. This playbook shows how to design keys, store results, and prove duplicates cannot recur.

Download available. Jump to the shipped asset.

You see the worst kind of incident: “everything is green” but customers are harmed. A gateway returns a 502/504, the client retries (correctly), and your API executes the side effect twice. Support now has duplicate orders, duplicate emails, and a backlog of manual cleanup — and engineering can’t prove what happened without reconstructing it from partial logs.

That is not a client bug. It’s a server-side safety gap. If retries exist anywhere in the chain (client SDK, gateway, queue consumer, operator replays), side effects must be protected against duplication. Idempotency keys are the smallest fix that works in a live .NET API without a rewrite.

If you only do three things
  • Require an idempotency key on any endpoint with side effects.
  • Store the key and the result before returning success.
  • Return the same response for repeated keys within a safe TTL.

Why retries create duplicate side effects

Retries are normal in production. Networks drop packets. Dependencies time out. Clients retry because they should. The problem is that most APIs treat every retry as a new request and execute the side effect again.

If an order creation endpoint inserts a record and sends a confirmation email, a retry can insert a second order and send a second email. This is not a bug in the client. It is a missing safety guard in the server.

Idempotency keys solve this by turning multiple identical requests into one logical operation. The server recognizes a repeated key and returns the stored result instead of executing the side effect again.

The incident pattern this playbook targets

  • Duplicate orders after a timeout or 500.
  • Duplicate emails or notifications after a retry.
  • Partial success where the client is unsure if it should retry.
  • Manual cleanup after a production incident.

If any of those are real in your system, idempotency is not optional.

Mini incident timeline

  • A partial outage causes intermittent 502/504 at the edge.
  • Clients and upstream services retry. Some original requests actually succeeded, but the response never made it back.
  • The retry replays the write and you now have duplicate rows / duplicate side effects.
  • Cleanup starts without evidence: which retries were replays vs. genuinely new operations.

A single idempotency key would have turned the retry into a replayed response instead of a second side effect.

Diagnosis ladder for duplicate side effects

  1. Do retries exist. If the client retries on timeout or 5xx, duplicates are possible.
  2. Is there a server side idempotency key. If not, duplication is guaranteed under failure.
  3. Is the key persisted before side effects. If not, a crash can still duplicate.
  4. Is the response cached and returned on repeats. If not, clients will keep retrying.
  5. Are keys scoped to a caller. If not, one client can block another.

Common misconceptions that cause duplicate orders

The bad fixes are predictable:

  • “Disable retries.” That trades duplicate harm for downtime and timeouts. Retries are normal; you need to make them safe.
  • “POST can’t be safe.” POST can be safe when the server treats repeated requests as one logical operation.
  • “A unique constraint is enough.” Constraints help, but they don’t replay the original response and they don’t cover multi-step side effects. You still need a key store + stored result.
  • “Idempotency is a big redesign.” The minimal version is bounded: validate key, store result, replay response for repeats within a TTL.

A minimal idempotency design for .NET APIs

Key rules

  • Clients generate a unique key per logical operation.
  • The key is sent in a header such as Idempotency-Key.
  • The server stores the key and response before returning success.
  • Repeated keys return the stored response, not a new side effect.

Key scope

  • Scope the key by client or account.
  • This prevents one client from blocking another by reusing keys.

TTL

  • Keep idempotency records long enough to cover retries and delayed responses.
  • A common starting point is 24 hours, adjusted by business risk.

Example: a simple idempotency contract

text
Idempotency-Key: 1c7f8c4a-5b4a-4d9d-9dd9-948f92f2c5b4

Store it with a hash of the request body and the response metadata. When a repeat request arrives with the same key and same body hash, return the stored response. If the body hash does not match, return a 409 and stop the duplicate.

Fix plan: roll out idempotency without breaking production

Phase 1: Require keys on the most painful endpoint

  • Start with the endpoint that caused duplicates.
  • Accept the key but log when it is missing.
  • Do not block requests yet.

Phase 2: Enforce keys and store results

  • Reject requests without a key for the protected endpoint.
  • Store the response and return it on repeat.
  • Add metrics for duplicate requests prevented.

Phase 3: Expand and standardize

  • Roll out to other side effect endpoints.
  • Add a shared middleware for key validation and storage.
  • Add a runbook for incidents involving duplicates.

What to log so duplicates are provable

If you do not log the key, you will not be able to prove duplicates or prevent them. These fields are the minimum:

  • idempotency_key
  • idempotency_result (stored, replayed, conflict)
  • request_hash
  • status_code
  • duration_ms
  • client_id

Log these fields on every request that uses a key. This turns duplicate detection from guesswork into evidence.

Tradeoffs and limits

Idempotency is not free. You need storage and a TTL policy. You must decide how long to keep the record and how to handle conflicts. But it is bounded work compared to the cost of duplicate orders and manual cleanup.

The other limit is partial side effects. If your system has multiple side effects, you may need a ledger or outbox. The minimal approach here still reduces duplicates but may not cover every cross system scenario. Use it to stop the bleeding, then harden further.

Shipped asset

Download
Free

Idempotency key contract template for .NET APIs

Key format, storage checklist, and response replay rules (free, email delivery)

What you get (4 files):

  • idempotency-contract-template.md
  • idempotency-key-strategy.md
  • request-cache-schema.sql
  • README.md
Axiom Pack
$79

Idempotency Implementation Playbook

Dealing with duplicate writes across multiple services or side effects? Get schemas, outbox patterns, and verification steps that prove retries can’t re-run the operation.

  • Storage schema + TTL guidance for safe replay at scale
  • Outbox / ledger patterns for multi-step side effects
  • Verification harness ideas (prove duplicates can’t recur, not “seems fixed”)
Get Idempotency Playbook →

Resources

Internal: status="comingSoon" ctaLabel="Coming soon"

External:

Yes. A retry repeats the request. If the server does not protect the side effect, it will run again. The only safe fix is an idempotency key with stored results.

Only for endpoints that create or mutate state. Read only endpoints do not need them. Start with the most painful endpoints and expand carefully.

Log the missing key first. Then enforce it once clients are updated. For public APIs, return a clear error with a short explanation and a link to docs.

Long enough to cover retries and delayed responses. A common starting point is 24 hours. Adjust based on business risk and cost.

Not always. The minimal approach uses a key store and cached response. If you have multiple side effects that must be consistent, you will likely need an outbox or ledger.

Treat it as a conflict. Return a 409 or similar error and log the mismatch. This prevents accidental or malicious reuse of keys across different requests.

Coming soon

If you are dealing with duplicates across multiple services, the idempotency playbook includes schemas, outbox patterns, and verification steps to make retries safe at scale.

Coming soon

Axiom .NET Rescue (Coming Soon)

Get notified when we ship idempotency templates, outbox patterns, and production runbooks for .NET services.

Key takeaways

  • Retries create duplicates unless the server enforces idempotency.
  • A key, a store, and a replayed response is the minimum safe design.
  • Log idempotency fields so duplicates are provable and preventable.
  • Roll out in phases to avoid breaking clients.
  • Use the minimal approach to stop the bleeding, then harden.

Recommended resources

Download the shipped checklist/templates for this post.

A key format, storage checklist, and replay rules to prevent duplicate side effects in .NET APIs.

resource

Related posts