Timestamp drift runbook + logging schema

What you get

2 operational files:

time-sync-runbook.md - Incident response procedure

• Detection: what symptoms mean it's a timestamp issue (signature failures on valid requests)
• Diagnosis: which systems to check first (NTP status, clock sources, system clock)
• Verification: how to confirm it's drift (compare local vs exchange time)
• Mitigation: immediate actions (sync clocks, restart, escalate)
• Prevention: post-incident setup (NTP monitoring, alerting, constraints)
• Recovery: how to handle requests that failed due to drift

logging-fields-for-signature-incidents.md - Required observability

• Timestamp fields: request_time, server_time, local_time, drift_ms
• Context: exchange, endpoint, key_id, request_signature
• Diagnostics: ntp_offset_ms, system_clock_source, drift_direction
• Resolution: action_taken, time_corrected, incident_duration_ms
• Integration: how to query these fields in Datadog, Axiom, CloudWatch

How to use

1. Download the runbook
2. Post it where your on-call engineers can find it (wiki, handbook, Slack)
3. Integrate logging schema into your bot's signature error events
4. Set up alerts for: drift > 5 seconds OR NTP unsynchronized
5. Test it: intentionally skew your system clock and run through the runbook

What this solves

This package helps you:

• Reduce time spent guessing during signature incidents
• Give on call a clear procedure that starts with drift checks
• Debug with queryable logs instead of log hunting

Real-world: Timestamp drift incident

Without logging:

10:30 - Alarm: API signatures failing on Binance
10:35 - Engineer: "Maybe API keys are wrong? Let me rotate them..."
10:45 - Still failing. "Maybe signature algorithm changed?"
11:00 - Tries new keys. Still fails.
11:30 - Finally: "Wait, what time is it on the server?" 
        Checks. Server is 45 seconds behind. Syncs NTP.
        Signatures work again.

MTTR: 60 minutes. Lost orders. Wasted engineer time.

With this runbook + logging:

10:30 - Alarm: API signatures failing
10:32 - Engineer runs runbook. First check: system clock vs NTP.
        Logs show drift_ms: -45000 (45 seconds behind)
10:33 - Run: `ntpd -q -g` to resync
10:34 - Signatures working again.

MTTR: 4 minutes. Clear diagnosis. Actionable.

Logging example

{
  "timestamp": "2026-01-27T14:30:20.123Z",
  "event_type": "signature_verification_failed",
  "exchange": "binance",
  "endpoint": "/api/v3/order",
  "api_key_id": "key_prod_1",
  "local_time_ms": 1706347820000,
  "server_time_ms": 1706347775000,
  "drift_ms": -45000,
  "ntp_status": "unsynchronized",
  "system_clock_source": "hardware_rtc",
  "action_taken": "signature_error_logged_for_investigation"
}

Runbook excerpt

Step 1: Detect drift

Check if this is drift:
- Signatures fail on valid requests (not auth failure)
- Multiple keys fail simultaneously (not a single key issue)
- All exchanges affected, not just one
Then it is likely timestamp drift.

Step 2: Verify

$ timedatectl status
       Local time: Mon 2026-01-27 14:30:20 UTC
       Universal time: Mon 2026-01-27 14:30:20 UTC
   RTC time: Mon 2026-01-27 14:30:15 UTC  ← LAGGING 5 SECONDS
   
$ curl -s https://api.binance.com/api/v3/time | jq '.serverTime'
1706347775000  ← Server says 14:30:15
Local is 5 seconds ahead. Signatures will fail.

Step 3: Fix

# Force sync with NTP
sudo ntpd -q -g

# Verify
timedatectl status
It should match server time within 1 second.

When you need this

• You're running crypto bots or integration services (signatures required)
• You've had mysterious signature failures in production
• You have multiple servers (they drift against each other)
• You want your on-call to diagnose fast, not guess
• You need observability into clock health

NTP setup (prevention)

# Install NTP daemon
sudo apt-get install ntp
 
# Config: use multiple time sources
# /etc/ntp.conf
server 0.ubuntu.pool.ntp.org iburst
server 1.ubuntu.pool.ntp.org iburst
server 2.ubuntu.pool.ntp.org iburst
server 3.ubuntu.pool.ntp.org iburst
 
# Restart
sudo systemctl restart ntp
 
# Verify it's syncing
ntpstat

Related resources

• Why crypto bots fail (and how to build one that works)
• Exchange API bans: how to prevent them
• Exchange rate limiting: fixed window vs leaky bucket
• Backoff + jitter: the simplest reliability win

Services

Back to resources

Read the full article