Runbook — quota-exhaustion

A bucket has fired AlphaSwarmRatelimitBucketAt80Percent, AlphaSwarmRatelimitBucketAt95Percent, or AlphaSwarmRatelimitBucketExhausted (see alphaswarm_platform/deployments/kubernetes/base-services/monitoring/alphaswarm_ratelimit_alerts.yaml).

Diagnosis (5 min)

1. No /data/ratelimit dashboard route was found in alphaswarm_ui as of this review; use alphaswarm ratelimit status --key-id <id> or query rl_ledger directly (below) to find the over-consuming (user_id, service, key_id).

2. Inspect the rl_ledger partition for the last hour:

   SELECT decision, count(*), sum(tokens_consumed)
     FROM rl_ledger
    WHERE ts > now() - interval '1 hour'
      AND key_id = :key_id
    GROUP BY decision;

3. Cross-reference audit_log for the calling tool_id — data.ingest.materialize or data.ingest.preview_source are the usual culprits.

Decision tree (10 min)

Cause	Action
Misconfigured backfill	alphaswarm materialize cancel <reservation_id> is not an implemented command; cancel via DELETE /reservations/{reservation_id} on the ratelimit API instead. The reservation auto-releases.
Vendor downgrade	Mint a higher-tier key via alphaswarm keys mint --service polygon --rps 100 --burst 1000.
Stuck connector loop	alphaswarm ratelimit status --key-id <id> shows the call rate; halt the offending Dagster sensor via the topbar kill-switch.
Legitimate traffic	Raise the policy via data.ratelimit.policy.update (Tier-P + step-up MFA).

Recovery (15 min)

1. Once the cause is addressed, the bucket refills at the policy's refill_rate; no manual reset is required.

2. If the operator wants an immediate reset: note that alphaswarm ratelimit admin reset is not an implemented command — alphaswarm_ratelimit.cli.ratelimit_cmd only exposes status and policies, with no admin subgroup. Coordinate with the ratelimit subsystem owner on the current reset procedure until this verb ships:

   alphaswarm ratelimit admin reset --user-id <uid> --service polygon --key-id primary

3. Verify recovery in Grafana:

   rl_bucket_remaining{service="polygon.aggregates"} > 50

Postmortem

Every quota-exhaustion alert that requires manual intervention must produce a postmortem PR within 72 hours. Template: alphaswarm_docs/docs/how-to/runbooks/templates/postmortem.md (to be authored).