Six-clock order latency

In plain English: when a trading system reacts to the market, the delay between "the market moved" and "our order was filled" is money. But a single end-to-end number can't tell you where time was lost — was the data feed slow, did the strategy think too long, did the database write stall, or did the broker take ages to acknowledge? AlphaSwarm answers this by stamping a clock at six named checkpoints along every order's journey, so operators can see exactly which segment got slower and fix that one.

The six clocks

Clock	Segment measured	Typical failure it exposes
md.event_lag	Exchange event time → feed hand-off (ingest_ns stamped at the feed boundary)	Slow or backlogged market-data feed
strategy.dispatch_wait	Feed hand-off → strategy callback starts	Session event-loop congestion
strategy.decide	Strategy callback runtime	Expensive signal logic, model inference stalls
oms.gate	Pre-trade risk/compliance checks	Gate chain regressions
oms.persist	Order-intent ledger write	Database latency (measured p50 ≈ 34.5 ms on real Postgres vs ~0.04 ms with a noop sink — this segment dominates)
broker.submit_ack	Submit → broker acknowledgement	Broker/API degradation
ams.apply_fill	Fill receipt → position/account state applied	Accounting backlog

An additional end-to-end decision-to-fill measurement spans the whole chain, and an OrderClockTrace carries the per-order stamps.

How it's exported

• Clocks are OpenTelemetry histograms defined in alphaswarm/trading/order_clock_metrics.py with the export path in alphaswarm/observability/six_clock_export.py.
• Celery workers each get their own MeterProvider; series flow over OTLP into the metrics stack (VictoriaMetrics/Prometheus).
• PromQL note: query the unprefixed series that prometheusremotewrite actually emits — oms_gate_seconds_bucket, oms_persist_seconds_bucket, etc., not alphaswarm_oms_gate_*. The Grafana dashboard (fixed UID as-six-clocks) is already aligned.

When it's on

Unusually for this platform, six-clock metrics are auto-on in paper, dev, local, test, and CI environments — the overhead is negligible and the diagnostic value is high.

Setting	Default	Effect
order_clock_metrics_enabled	auto-on (paper/dev/local/test/CI)	Enables clock stamping + export.
order_clock_metrics_force_disabled	off	Hard off-switch that wins over auto-on.

Operator quick checks

# Confirm series are arriving (VictoriaMetrics example)
curl -s 'http://localhost:8428/api/v1/series?match[]=oms_persist_seconds_bucket' | head

# p95 of the ledger-write segment over 5 minutes
# histogram_quantile(0.95, sum(rate(oms_persist_seconds_bucket[5m])) by (le))

Open Grafana → dashboard as-six-clocks for the per-segment latency panels and the decision-to-fill overview.

See also

• Observability — the OTEL → Jaeger tracing and structured-logs baseline.
• Observability stack — the collector / storage / dashboard topology these histograms flow through.
• Paper trading — the session loop the paper-path clocks instrument.
• Execution paths — where the OMS gate and broker submit sit in the order flow.