Resources/Timestamp drift runbook + logging schema

Timestamp drift runbook + logging schema

On-call runbook for signature error incidents. Logging schema makes clock drift diagnostics fast. Know what to check in minutes, not hours.

FreeJan 27, 2026

What’s inside

Individual files are accessible (best for SEO/AI), plus you can download the full ZIP.

logging-fields-for-signature-incidents.mdmd time-sync-runbook.mdmd

Source code

This resource is backed by a public GitHub repository with source code, templates, and documentation you can fork, review, and integrate.

View on GitHub

From this article

Browse all

Signature invalid but bot was working: why clock drift breaks auth suddenly

When bot gets signature invalid or 401 after working fine for hours: why clock drift breaks exchange auth suddenly, and the time calibration that prevents it.

Also referenced in

Binance error -1021: timestamp for this request is outside the recvWindow

What you get

2 operational files:

time-sync-runbook.md - Incident response procedure

Detection: what symptoms mean it's a timestamp issue (signature failures on valid requests)
Diagnosis: which systems to check first (NTP status, clock sources, system clock)
Verification: how to confirm it's drift (compare local vs exchange time)
Mitigation: immediate actions (sync clocks, restart, escalate)
Prevention: post-incident setup (NTP monitoring, alerting, constraints)
Recovery: how to handle requests that failed due to drift

logging-fields-for-signature-incidents.md - Required observability

Timestamp fields: request_time, server_time, local_time, drift_ms
Context: exchange, endpoint, key_id, request_signature
Diagnostics: ntp_offset_ms, system_clock_source, drift_direction
Resolution: action_taken, time_corrected, incident_duration_ms
Integration: how to query these fields in Datadog, Axiom, CloudWatch

How to use

Download the runbook
Post it where your on-call engineers can find it (wiki, handbook, Slack)
Integrate logging schema into your bot's signature error events
Set up alerts for: drift > 5 seconds OR NTP unsynchronized
Test it: intentionally skew your system clock and run through the runbook

What this solves

This package helps you:

Reduce time spent guessing during signature incidents
Give on call a clear procedure that starts with drift checks
Debug with queryable logs instead of log hunting

Real-world: Timestamp drift incident

Without logging:

code

10:30 - Alarm: API signatures failing on Binance
10:35 - Engineer: "Maybe API keys are wrong? Let me rotate them..."
10:45 - Still failing. "Maybe signature algorithm changed?"
11:00 - Tries new keys. Still fails.
11:30 - Finally: "Wait, what time is it on the server?" 
        Checks. Server is 45 seconds behind. Syncs NTP.
        Signatures work again.

MTTR: 60 minutes. Lost orders. Wasted engineer time.

With this runbook + logging:

code

10:30 - Alarm: API signatures failing
10:32 - Engineer runs runbook. First check: system clock vs NTP.
        Logs show drift_ms: -45000 (45 seconds behind)
10:33 - Run: `ntpd -q -g` to resync
10:34 - Signatures working again.

MTTR: 4 minutes. Clear diagnosis. Actionable.

Logging example

json

{
  "timestamp": "2026-01-27T14:30:20.123Z",
  "event_type": "signature_verification_failed",
  "exchange": "binance",
  "endpoint": "/api/v3/order",
  "api_key_id": "key_prod_1",
  "local_time_ms": 1706347820000,
  "server_time_ms": 1706347775000,
  "drift_ms": -45000,
  "ntp_status": "unsynchronized",
  "system_clock_source": "hardware_rtc",
  "action_taken": "signature_error_logged_for_investigation"
}

Runbook excerpt

Step 1: Detect drift

code

Check if this is drift:
- Signatures fail on valid requests (not auth failure)
- Multiple keys fail simultaneously (not a single key issue)
- All exchanges affected, not just one
Then it is likely timestamp drift.

Step 2: Verify

code

$ timedatectl status
       Local time: Mon 2026-01-27 14:30:20 UTC
       Universal time: Mon 2026-01-27 14:30:20 UTC
   RTC time: Mon 2026-01-27 14:30:15 UTC  ← LAGGING 5 SECONDS
   
$ curl -s https://api.binance.com/api/v3/time | jq '.serverTime'
1706347775000  ← Server says 14:30:15
Local is 5 seconds ahead. Signatures will fail.

Step 3: Fix

code

# Force sync with NTP
sudo ntpd -q -g

# Verify
timedatectl status
It should match server time within 1 second.

When you need this

You're running crypto bots or integration services (signatures required)
You've had mysterious signature failures in production
You have multiple servers (they drift against each other)
You want your on-call to diagnose fast, not guess
You need observability into clock health

NTP setup (prevention)

bash

# Install NTP daemon
sudo apt-get install ntp
 
# Config: use multiple time sources
# /etc/ntp.conf
server 0.ubuntu.pool.ntp.org iburst
server 1.ubuntu.pool.ntp.org iburst
server 2.ubuntu.pool.ntp.org iburst
server 3.ubuntu.pool.ntp.org iburst
 
# Restart
sudo systemctl restart ntp
 
# Verify it's syncing
ntpstat