Exchange API ban prevention runbook

This runbook is meant for "it’s 2am and the bot is getting errors" situations.

A) Identify what’s happening (fast triage)

• Are we seeing 429 (rate limit) or 401/403 (auth/ban) or 5xx (exchange instability)?
• Did error rates start right after deploy/config change? (regression)
• Did request volume spike? (reconnect storm, retry storm, fan-out)

B) Rate limit hardening (429)

• Enforce a per-exchange limiter (token bucket / leaky bucket).
• Respect Retry-After when present.
• Add jitter to avoid synchronized waves.
• Reduce concurrency under sustained 429.

C) Auth hygiene (401/403/signature)

• Verify API key permissions match endpoints used.
• Check clock drift (see timestamp resource).
• Ensure nonce/timestamp is monotonic per key when required.
• Never log secret keys; log key id / last 4 + request id.

D) Protect the exchange (and yourself)

• Circuit breaker: stop calling the exchange when error budget is exceeded.
• Backoff + jitter on retryable failures.
• Separate “market data” from “trading” traffic paths.

E) What to log (minimum)

• Exchange, endpoint, status code, error body (redacted), request id.
• Rate limiter state (tokens/queue depth), concurrency.
• Retry attempt number and delay.

F) Containment steps during incident

1. Reduce traffic: lower polling frequency and concurrency.
2. Disable non-critical endpoints.
3. If auth is failing, stop retries and page.
4. Use allowlists only if the exchange explicitly requires it, and work with exchange support. Do not try to evade bans.