Skip to main content

alphaswarm-worker

Catalog date: 2026-06-24.

Celery orchestration worker pod that drains the light / coordination queues produced by alphaswarm-core.

As of the Phase 4c worker/executor split it has its own slim image (target worker) carrying only the task-dispatch + lineage surface — not the API stage's visualization / dev / Dash deps it used to inherit. Heavy compute (backtest / training / ML / agents / factors / RAG) is offloaded to the sibling alphaswarm-executor. See worker vs executor images for the full rationale and dependency matrix.

Identity​

FieldValue
Service idalphaswarm-worker
Roleworker
Packagealphaswarm_worker/
Image (key)worker
Built fromalphaswarm_platform/Dockerfile (target worker, multi-arch) or the standalone build/docker/alphaswarm_worker/Dockerfile

Wire​

FieldValue
Protocolnone (no HTTP listener)
HealthCelery broker connection probe + Prometheus metrics on :9100 (when enabled)
Public URL—
Brokerredis://redis:6379/0 (settings.redis_url / ALPHASWARM_REDIS_URL)
Result backendredis://redis:6379/0 — same URL as the broker in alphaswarm/tasks/celery_app.py (broker=settings.redis_url, backend=settings.redis_url); redis://redis:6379/1 is a separate pub/sub channel (ALPHASWARM_REDIS_PUBSUB_URL), not the Celery result backend

Deployment surfaces​

SurfaceWhere
Composeservice worker in alphaswarm_platform/compose/docker-compose.yml; alphaswarm-worker in deployments/compose/docker-compose.local.yml
Kustomizedeployments/kubernetes/base/alphaswarm-worker/ — Deployment + HPA + PDB
AlphaSwarm CRfolded into AlphaSwarmMonolith — AQPMonolithSpec scales it via workerReplicas; there is no spec.workers.queues field
Terraform modulealphaswarm_platform/terraform/modules/faas/ — Celery + KEDA per-queue ScaledObjects

Queue families​

The orchestration worker drains the light / coordination queues only. The heavy compute queues (backtest, training, ml, agents, factors, rag) are drained by alphaswarm-executor. KEDA scales each queue family independently. Default queue map:

QueueDrivesScale-to-zeroNotes
defaultmisc tasks, callbacks, lookupsyesalways-on min=1 in prod
paperpaper trading session ticksnosub-second latency required
terraformTerraformRuntime celery wrappersyes
ingestionAirbyte / Dagster / connector pullsnouses long-lived workers
workflowsWorkflowRuntime orchestrationyes
hftHFT hot-path event handlersnopinned to hft-nodes (compose/legacy)
note

The faas KEDA module keys per-queue Deployments off local.heavy_queues — heavy queues run the alphaswarm-executor image, everything else runs this alphaswarm-worker image. The two image sets never share a queue.

Dependencies​

Upstream:

  • redis — broker + result backend.
  • postgres — task lookups, ledger writes.
  • alphaswarm-core — progress emit callbacks, lookup APIs.
  • All data-plane services the alphaswarm-core pod depends on (the same code paths run inside Celery).

Downstream:

  • Beat schedules tasks; the worker is the consumer.
  • HFT-tagged tasks land on the hft-nodes/ workload (PTP-tuned).

Operations​

  • Scaling: KEDA ScaledObject per queue; idle queues scale to zero. The per-queue min/max lives in the faas Terraform module.
  • Concurrency: the orchestration worker runs concurrency 4 (light, IO-bound dispatch work); 1 for HFT (single-threaded pinning).
  • Drain on shutdown: deployments/kubernetes/base/alphaswarm-worker/deployment.yaml sets no terminationGracePeriodSeconds override (Kubernetes default, 30s) and no preStop hook. The terminationGracePeriodSeconds: 600 + preStop drain behavior described in an earlier version of this doc belongs to the sibling alphaswarm-executor Deployment, not this one.
  • Audit: WorkloadRuntime actions land workload_runs rows; the worker pod respects the kill-switch Redis key the same way the API does.

See also​