Portable Dagster and Prefect orchestration

This is the canonical technical guide for AlphaSwarm's portable orchestration subsystem. It describes the rollout target implemented across alphaswarm_core, alphaswarm_orchestration, alphaswarm_config, alphaswarm_worker, the monolith, and alphaswarm_platform.

Rollout state: the subsystem is additive and default-off. Celery remains the compatibility transport until each migrated schedule has exactly one verified engine owner. See ADR 025 and the rollout runbook.

The pinned framework baseline is Dagster 1.13.13, Dagster integration packages 0.29.13, Prefect 3.7.8, and prefect-kubernetes 0.7.10. The design follows Dagster's documented op, job, schedule, and sensor boundaries. The Prefect adapter follows the documented API client, custom worker, custom event, and self-hosted Helm boundaries.

Non-negotiable invariants

1. One logical run has exactly one engine, retry owner, concurrency governor, and active trigger owner.
2. Dagster and Prefect are co-equal native authorities. AlphaSwarm never reads either framework's internal database tables.
3. The AlphaSwarm gateway owns customer authentication, authorization, ownership checks, step-up, audit, and normalized cross-engine metadata.
4. Framework tags, labels, and native UI metadata are correlation aids, never authorization evidence.
5. Heavy or distributed work delegates through dependency-light alphaswarm_worker.WorkSubmissionClient; engine processes do not import Ray, Dask, Spark, or the monolith to submit a WorkRequest.
6. agents.WorkflowRuntime remains the semantic runtime for agent workflows. Dagster or Prefect controls the containing portable run.
7. Portability preserves native features. extensions.dagster and extensions.prefect either compile explicitly or produce a diagnostic; there is no silent degradation.
8. Definitions and their versions are immutable. Native engine state is retained alongside, not instead of, the canonical projection.
9. The monolith is the durable session-resource manager authority. The controller owns Kubernetes mutations behind an authenticated, versioned internal API; neither its request payload nor Kubernetes metadata grants tenant authorization.

System context

Open the full C4 context SVG. The system boundary deliberately keeps native UIs internal. Hosted operations, CLI, and DataMCP all use the tenant-aware gateway.

Portable definition model

TaskDefinition is the engine-neutral root. Its deterministic content hash creates an immutable TaskDefinitionVersion. The graph is a declaration-order stable DAG of TaskStep records with handler references, dependencies, parameters, execution profiles, step retries, cache policy, artifact specs, and safe metadata.

Contract	Purpose
ExecutionProfile	Workload kind, image, queue, service account, resource requests, timeout, and worker delegation intent
RunRetryPolicy	Run-level attempt cap, backoff, jitter, and retryable canonical states
StepRetryPolicy	Step-level retry behavior compiled into the selected engine
ConcurrencyPolicy	Portable run or key-scoped concurrency limit
CachePolicy	Deterministic cache key, expiry, and refresh behavior
ArtifactSpec	Required output key, media type, URI policy, and checksum expectation
ScheduleTrigger	Five-field cron, IANA timezone, and enabled state
EventTrigger	Event types, resource match, reactive or proactive posture, threshold, and window
PollingTrigger	Evaluator reference, Draft 2020-12 cursor schema, interval, and deduplication expression

The compiler returns a CompileReport even when compilation fails. Diagnostics identify unsupported capability, unregistered handler, non-portable native extension, invalid trigger, or invalid definition without mutating a control plane. EngineCapabilities makes preservation explicit for cancellation, retries, schedules, events, polling, artifacts, partitions, backfills, and native extensions.

Engine adapter lifecycle

Every adapter implements validate, compile, register, launch, get_run, cancel, retry, paged event and artifact access, reconcile, and capabilities. Two synchronous rehydration seams are required after process restart:

• bind_registration(native_registration_id, compilation) restores the compiled registration index without registering again.
• bind_run(logical_run_id, native_run_id, attempt, native_state, created_at, registration_id) restores logical/native run indexes without launching or mutating the native engine.

The control plane recompiles the persisted immutable version before binding a registration. This keeps adapters stateless across deployment restarts without turning read recovery into an external write.

Dagster-owned run

Open the full Dagster sequence SVG.

Portable steps compile into typed ops and a job backed by AlphaSwarmContextResource. Schedules compile to Dagster schedules; event and polling triggers compile to sensors where the registered evaluator contract is available. Native assets, partitions, checks, automation conditions, and backfills remain Dagster extensions.

The control adapter uses the public Dagster client plus version-gated GraphQL documents for launch, terminate, re-execution, event logs, asset metadata, checks, partitions, schedules, sensors, and backfills. Event cursors are opaque AlphaSwarm envelopes containing both the native checkpoint and the next normalized sequence. Persist and return them unchanged.

K8sRunLauncher remains the supported process launcher. Portable steps submit heavy work to alphaswarm_worker; the implementation does not rely on an unstable custom executor API.

Prefect-owned run

Open the full Prefect sequence SVG.

The same graph compiles into a flow and tasks. Hidden parameters carry the signed context reference, digest, and envelope so task workers do not lose tenant, session, environment, or trace state. PrefectClientGateway creates Kubernetes work pools, environment/workload queues, deployments, schedules, and automations through supported APIs.

AlphaSwarmKubernetesWorker.prepare_for_flow_run verifies context before creating a Kubernetes Job. It applies safe labels, environment and deployment version, service account, resource requests, and the W3C trace carrier. Reference-only config and secret Blocks resolve values through the existing credential resolver; Blocks do not become a second secret store.

Canonical run lifecycle

Open the full lifecycle SVG.

The normalized states are SCHEDULED, PENDING, RUNNING, SUCCEEDED, FAILED, CANCELLED, and CRASHED. Every RunRecord and event may retain the native state and a bounded native summary. A checksummed NativePayloadReference points to large raw payloads in object storage.

Retry transitions return FAILED or CRASHED work to PENDING only through the engine already recorded as retry_owner. A retry produces a new attempt; it does not create a second logical schedule owner.

Execution context and security

Open the full security-boundary SVG.

ExecutionContextV1 unifies identity, tenancy, runtime, and trace state:

• user, organization, team, workspace, project, lab, cell, region, role, and tenancy strategy;
• environment, deployment version, session, request, correlation, parent run, experiment, test, and run IDs;
• W3C traceparent, tracestate, and bounded baggage;
• non-secret labels, sanitized extras, ConfigRef, and SecretRef; and
• issued and expiry timestamps.

The immutable context is serialized into a versioned, expiring HMAC envelope. Launch validation checks signature, key ID, digest, expiry, and required user identity. The gateway stores a trusted reference and digest; canonical engine metadata contains only:

• alphaswarm/run_id;
• alphaswarm/context_ref;
• alphaswarm/context_digest; and
• alphaswarm/context_schema.

Authorization is re-evaluated at the gateway and database boundary. A valid context proves integrity and supplies execution state; it does not confer a permission by itself.

Session-scoped distributed execution

Portable runs can acquire execution capabilities without making Dagster, Prefect, Ray, Dask, Spark, Polars, Vaex, or an interactive kernel part of the session identity model. The dependency direction is deliberate:

1. alphaswarm_core defines immutable SessionResourceSpecV1, SessionResourceLeaseV1, SessionEndpointRefV1, and SessionResourceManager wire contracts.
2. alphaswarm_orchestration.SessionResourceCoordinator owns reservation, reference-count, renewal, release, failure compensation, and reconciliation policy while importing no Kubernetes client.
3. The monolith embeds the durable manager, owns forced-RLS coordinator state, authenticated customer APIs, durable idempotency outcomes, startup health, reconciliation, and audit.
4. The dependency-light alphaswarm_orchestration.HttpSessionResourceProvider calls the controller's authenticated, versioned lifecycle API without importing a Kubernetes, Ray, Dask, or Spark SDK.
5. alphaswarm_controller.KubernetesSessionResourceProvider exclusively owns native RayCluster, DaskCluster, SparkConnect, NetworkPolicy, Service, and tombstone mutations behind that API.
6. alphaswarm_worker verifies the signed context and authoritative lease at every driver and remote-worker boundary before resolving symbolic endpoints.

Open the full session-resource sequence SVG.

Manager and controller boundary

The public /orchestration/session-resources routes call the manager embedded in the monolith. The manager persists the coordinator's reservation, lease, consumer, operation, and event state before crossing a process boundary. It uses the HTTP provider to call only these controller-internal operations:

• GET /internal/session-resources/health;
• POST /internal/session-resources/v1/provision;
• POST /internal/session-resources/v1/renew;
• POST /internal/session-resources/v1/release; and
• POST /internal/session-resources/v1/cleanup-abandoned.

The controller requires its service-to-service bearer credential with the manage:infrastructure scope before validating the versioned payload. The payload contains a spec, current lease or reservation, operation ID, timestamp, and fence as needed, but it is never an authorization source. The controller returns a validated lease with symbolic endpoint references or a bounded cleanup acknowledgement. The HTTP provider re-validates spec and native lease authority before the manager commits it.

Resource classes and cache authority

Kinds	Resource class	Provisioning authority	Reuse boundary
Ray, Dask, Spark	provisioned_cluster	Kubernetes provider	Explicit session, user, organization, local-instance, or disabled cache scope
Polars, Vaex	local_runtime	Worker process	Same validated cache scope and runtime identity
Prefect, Dagster	orchestration_binding	Existing engine adapter	Existing registration/run authority; never a second scheduler
Interactive kernel	interactive	Interactive runtime provider	Session-bound unless an explicit narrower policy disables reuse

Every specification pins user, organization, workspace, optional project, environment, deployment version, session, signed-context reference and digest, idle and maximum TTLs, cache scope, safe labels, and reference-only config or secret dependencies. Deterministic canonical JSON and SHA-256 digests make replay and comparison stable. Secret-shaped labels, endpoint values, and credential material fail validation.

The cache namespace is derived from the declared scope and identity; callers cannot substitute a broader namespace. Local reuse additionally requires a symbolic local_instance_id. A cached native resource is reusable only while its spec authority, context digest, deployment version, TTLs, and provider fence still match.

Lease lifecycle and fencing

Open the full session-resource lifecycle SVG.

The public lifecycle is REQUESTED, PROVISIONING, READY, DEGRADED, RELEASING, RELEASED, FAILED, and EXPIRED. The coordinator also keeps a durable internal reservation state and a per-consumer compare-and-swap token. Provisioning heartbeats do not grant takeover authority; an ambiguous or cancelled provision is cleaned up before its reservation can be made terminal. Late provider success cannot publish READY after the consumer is detached.

Native resources carry a monotonically increasing fencing version bound to the lease endpoint authority. Kubernetes updates use resourceVersion compare and swap with bounded conflict retries. A stale lease cannot renew, release, or delete a newer resource. Post-create tombstone checks compensate races where a release wins while the provider is creating the cluster. Provider-native status is reduced to an allowlisted, non-secret summary.

Reference counts are consumer records, not a mutable integer supplied by an API caller. A run consumer is derived from logical run ID, attempt, and sole engine owner. Terminal run state detaches that consumer; only the final detach may release the native resource. Reconciliation repairs fenced renewal, release, and abandoned-cleanup operations without inventing a new native authority.

Worker and remote-engine boundary

WorkRequest.resource_bindings carries the immutable spec and lease, never a raw connection URL. ExecutionContextReceiver first verifies the HMAC envelope, expiry, digest, tenant/run projections, and trace carrier. It then resolves each lease through a trusted repository and requires byte-for-byte canonical agreement before assert_usable checks identity, state, TTL, fence, and endpoint authority.

The driver resolves a symbolic endpoint only after those checks. Ray and Dask workers re-verify the context with their configured keyring and return a runtime inventory containing Python, AlphaSwarm package, engine, required module, and context-digest versions. Execution fails closed on missing inventory, a package mismatch, an incompatible engine major/minor, a missing module, or a context mismatch. Spark preserves the same request, binding, deadline, cancellation, and point-in-time DataBinding contract through its native submission boundary.

Normalized orchestration and session-execution metadata

Open the full metadata ERD SVG.

The monolith owns ten additive orchestration tables:

Table	Authority and critical constraints
orchestration_definitions	Workspace-unique logical key and default engine
orchestration_definition_versions	Immutable version and content hash per definition
orchestration_engine_registrations	Unique native registration and one registration per version/engine
orchestration_trigger_bindings	Unique native trigger and partial unique active owner per version/trigger key
orchestration_runs	Logical run, one engine/retry owner, context projection, canonical/native state, active-run index
orchestration_run_attempts	Unique attempt number and native engine attempt binding
orchestration_events	Append-only, unique engine/native ID and run sequence
orchestration_artifacts	Checksummed URI, size, media type, and native metadata
orchestration_engine_bindings	Logical/native IDs for definition, registration, trigger, run, attempt, and artifact
orchestration_reconciliation_cursors	Workspace/engine/stream/scope checkpoint and watermark

Every table carries the existing project/workspace scope columns. PostgreSQL forces row-level security and the application uses the pooled app_runtime role. Repository queries also apply explicit scope predicates as defense in depth. Indexed foreign keys, workspace/state/time indexes, cursor pagination, partial idempotency indexes, and the partial active-run index support control plane queries.

orchestration_events is not partitioned initially. Partitioning is deferred until measured volume approaches the existing 100-million-row threshold. Update and delete triggers make the event ledger append-only.

The durable session manager adds eight forced-RLS tables without storing raw endpoints or credential values:

Table	Authority and critical constraints
execution_sessions	Verified context/session authority, parent linkage, state, and expiry
session_resources	Immutable spec digest plus current state, fence, revision, and derived consumer count
session_resource_reservations	Exclusive provision/release claim, heartbeat, fence, and tombstone state
session_resource_leases	Current symbolic endpoint and DB-protected native authority
session_resource_consumers	Per-consumer compare-and-swap token and attach/detach state
session_resource_events	Append-only, cursor-addressable lifecycle projection
session_resource_operations	Durable acquire/renew/release idempotency claim and redacted response
session_cache_entries	Scoped, checksummed object reference for session/user/org/local reuse

The session event table follows the same measured 100-million-row partitioning threshold. Database triggers protect append-only events, immutable lease authority, monotonic fences, and derived reference counts. Advisory locks plus row locks serialize coordinator transitions across API replicas; process-local locks are not the durability boundary.

API, streaming, and compatibility

The authenticated /orchestration API covers:

• definitions, immutable versions, validation, and compilation reports;
• engine registrations and trigger bindings;
• runs, attempts, events, logs, and artifacts;
• session resource acquire, renew, release, list, and lookup;
• health, workers, and pools; and
• authorized cancellation and retry.

Mutations require write scopes, ownership, idempotency keys, context signature verification, and audit events. Destructive actions require step-up. SSE uses the browser's Last-Event-ID header when it is present; that reconnect cursor must take precedence over any stale after query parameter captured when the stream URL was created. A first connection may use after. The stream emits normalized events with the compatibility projection:

{
  "task_id": "logical-run-id",
  "stage": "portable-step-or-state",
  "message": "human-readable event",
  "timestamp": "2026-07-11T12:00:00Z"
}

The /workflows routes and stable Celery task names remain facades. With rollout flags off they preserve the legacy path. When compatibility routing is enabled they create a normalized logical run, select the configured engine, and preserve TaskAccepted, task IDs, stream URLs, replay, and halt behavior. The agents versus workflows queue names must match before this path is enabled.

Data and metadata flow

Open the full data-flow SVG.

Airbyte remains the ingestion control plane. Dagster retains first-class asset, dbt, partition, and check semantics. A generic pipeline definition may also run through Prefect against the same manifest contract. Iceberg holds governed tables, Polaris provides catalog semantics, and DataHub receives ingestion, transformation, and dataset lineage. Orchestration events are operational metadata; they do not replace DataHub's data-lineage authority.

Self-hosted deployment

Open the full deployment SVG.

Compose provides an orchestration-smoke profile with pinned custom images, health checks, resource ceilings, PostgreSQL 16, authenticated noeviction Redis, and OTel/Jaeger. Kubernetes production defaults are:

• Prefect API ×2, background services ×1, Kubernetes workers ×2;
• Dagster webserver ×2, daemon ×1, code location ×1;
• separate Dagster, Prefect, and AlphaSwarm application databases and users;
• pg_trgm in the Prefect database;
• External Secrets, service accounts, probes, resource bounds, NetworkPolicies, topology spread, and PodDisruptionBudgets where replicas support them; and
• internal-only native UIs.

Acceptance uses a dedicated Prefect work pool and worker image. It must not replace a production pool's base job template, and the image must contain the reviewed alphaswarm_orchestration artifact before the smoke begins. Ad-hoc package installation into a running worker is not acceptance evidence. Acceptance metadata is bootstrapped before the isolated worker starts because Prefect workers cache the work-pool base Job template. The observer is scoped to the pod namespace so its watches match the namespace-scoped worker Role.

Dagster acceptance uses a distinct Helm release with SyncInMemoryRunCoordinator and no daemon. Its webserver synchronously submits the one acceptance run to its own K8sRunLauncher; it cannot dequeue a production run from the shared PostgreSQL-backed queue. The rendered profile fails closed unless it contains exactly one synchronous coordinator and zero acceptance daemon resources.

The 2026-07-15 Julia k3s acceptance proved both engine execution paths, adapter-enforced deletion of an actively cancelled Prefect Job, namespace- scoped observer watches, cross-namespace OTLP reachability, platform dependency health, isolated-resource cleanup, and restoration of production replica counts. The exact run IDs, Job evidence, revision, and reproduction commands are retained in the rollout runbook.

Kubernetes session providers require the relevant CRDs/operators before their kinds can be enabled: KubeRay for RayCluster, the Dask Kubernetes operator for DaskCluster, and Spark Operator for SparkConnect. Namespace allowlists, service accounts, egress NetworkPolicies, pinned images and resource profiles, and context-key/endpoint-reference Secrets remain environment configuration; the generic contracts contain no cluster credentials.

The default alphaswarm-session-runtime service account is namespace scoped and cannot read Secrets. Spark Connect servers retain the projected token only to manage executor pods, Services, ConfigMaps, and optional PVCs through the reviewed Role. The controller explicitly disables token automount in Ray and Dask pod templates, which do not need that Kubernetes API authority.

Placement is resource-kind specific and validated before cluster mutation. Because the pinned rayproject/ray:2.9.0 image is amd64-only, Ray head and worker PodSpecs require kubernetes.io/arch In [amd64] by default. Dask and Spark remain architecture-portable unless their own allowlist or node selector is configured. The Ray head keeps client-readiness and dashboard-liveness probes; workers do not expose or probe the head-only dashboard port, leaving worker lifecycle and availability to KubeRay.

The default Dagster values preserve its existing database authority. The PostgreSQL 16 cutover uses a separate values overlay and an explicit migration and rollback procedure; it is not implicit in the control-plane rollout.

Representative migration

The first wave is intentionally narrow:

Definition	Default owner	Required parity proof
system.context_echo	Both, test-only	Identical normalized context and output on Dagster and Prefect
data.pipeline_manifest_materialization	Dagster	Prefect runs the same manifest and produces matching artifact checksums
agents.workflow_runtime	Prefect	Dagster compilation remains valid while Prefect wraps WorkflowRuntime

Open the full migration SVG.

No Celery Beat entry is disabled until its replacement trigger is registered, disabled, shadow-verified, and ready for an ownership handoff. The handoff order is always legacy off, then one native trigger on. Rollback reverses that order.

C4 container and component views

Open the full C4 container SVG.

Open the full C4 component SVG.

Renderer-neutral semantic model

The architecture source of truth is semantic-model.v1.json. It is renderer-neutral and contains:

• stable node, edge, group, source, and view IDs;
• typed nodes and labeled directed relationships;
• source provenance for every entity and relationship;
• view membership, audience, abstraction level, question, and long text alternative;
• layered-layout direction, routing, and stability hints; and
• Mermaid source, SVG export, and regeneration paths.

The hosted React Flow/ELK surface should consume this model or a versioned projection of it. React owns semantic selection, filters, and URL state; ELK owns layered/orthogonal placement; XYFlow owns rendering and viewport state. Do not maintain a second hard-coded architecture graph.

For desktop, use a synchronized outline/filter rail, scrollable central canvas, and inspector. For mobile portrait, start at the selected run or focused DAG and move outline/filters/inspector into Sheets; landscape is the preferred wide DAG inspection mode. Search, selection, fit/reset, zoom, export, and structured table alternatives must remain keyboard accessible.

Diagram provenance and regeneration

Every .mmd source starts with machine-validated metadata for its stable ID, audience, abstraction level, source paths, generation date, text alternative, and exact regeneration command. SVG exports preserve searchable text and add role="img", <title>, and <desc>.

pnpm diagrams:validate
pnpm diagrams:render
pnpm diagrams:check
python3 -m pytest -q scripts/tests/test_orchestration_diagrams.py

diagrams:check renders into a temporary directory and compares exact bytes. The pinned Mermaid CLI, deterministic IDs, and fixed hand-drawn seed prevent layout drift from being committed unnoticed. The SVG exports are an explicit exception to the general Mermaid-only convention because this architecture package requires durable vector exports; Mermaid remains the editable source.

Stable view ID	Audience	Level	Source / export
portable-orchestration-c4-context-v1	Executives, architects, security	C4 L1	mermaid/c4-context.mmd / c4-context.svg
portable-orchestration-c4-container-v1	Architects, platform, operations	C4 L2	mermaid/c4-container.mmd / c4-container.svg
portable-orchestration-c4-component-v1	Backend maintainers	C4 L3	mermaid/c4-component.mmd / c4-component.svg
portable-orchestration-dagster-sequence-v1	Backend, SRE, incident response	Runtime sequence	mermaid/dagster-sequence.mmd / dagster-sequence.svg
portable-orchestration-prefect-sequence-v1	Backend, SRE, incident response	Runtime sequence	mermaid/prefect-sequence.mmd / prefect-sequence.svg
portable-orchestration-lifecycle-v1	Developers, operators, auditors	State model	mermaid/run-lifecycle.mmd / run-lifecycle.svg
portable-orchestration-metadata-erd-v1	Backend, database, audit	Relational projection	mermaid/metadata-erd.mmd / metadata-erd.svg
portable-orchestration-context-security-v1	Security, backend, operations	Trust boundaries	mermaid/context-security.mmd / context-security.svg
portable-orchestration-data-flow-v1	Data, platform, governance	Logical data flow	mermaid/data-flow.mmd / data-flow.svg
portable-orchestration-deployment-topology-v1	SRE, platform, security	Deployment	mermaid/deployment-topology.mmd / deployment-topology.svg
portable-orchestration-celery-migration-v1	Release, SRE, service owners	Rollout flow	mermaid/celery-migration.mmd / celery-migration.svg
portable-orchestration-session-resource-sequence-v1	Backend, platform, SRE, security	Runtime sequence	mermaid/session-resource-sequence.mmd / session-resource-sequence.svg
portable-orchestration-session-resource-lifecycle-v1	Backend, platform, SRE, audit	Resource state model	mermaid/session-resource-lifecycle.mmd / session-resource-lifecycle.svg

All paths in the table are beneath static/architecture/orchestration/ for sources and static/img/architecture/orchestration/ for exports. The complete source-path and text-alternative metadata lives in the semantic model and each Mermaid file.

Source map

The primary implementation evidence is:

• alphaswarm_core/src/alphaswarm_core/execution/context.py;
• alphaswarm_core/src/alphaswarm_core/execution/session.py;
• alphaswarm_orchestration/src/alphaswarm_orchestration/contracts.py and adapter.py;
• alphaswarm_orchestration/src/alphaswarm_orchestration/session_resources.py;
• alphaswarm_orchestration/src/alphaswarm_orchestration/http_session_provider.py;
• alphaswarm_orchestration/src/alphaswarm_orchestration/adapters/dagster/;
• alphaswarm_orchestration/src/alphaswarm_orchestration/adapters/prefect/;
• alphaswarm_worker/src/alphaswarm_worker/submission.py;
• alphaswarm_worker/src/alphaswarm_worker/execution/{context,resources,remote}.py;
• alphaswarm_controller/src/alphaswarm_controller/api/routers/session_resources.py and providers/session_resources.py;
• alphaswarm/alphaswarm/orchestration/, alphaswarm/alphaswarm/api/routes/orchestration.py, and alphaswarm/alphaswarm/persistence/models_orchestration.py;
• alphaswarm/alembic/versions/0124_portable_orchestration_ledger.py;
• alphaswarm/alembic/versions/0130_session_distributed_execution.py;
• alphaswarm/alembic/versions/0131_session_resource_operations.py;
• alphaswarm_platform/deployments/compose/docker-compose.orchestration.yml plus deployments/kubernetes/mlops/{dagster,prefect,session-runtime}/.

Treat checked-in source and supported native APIs as final authority. Diagrams record the architecture as reviewed on 2026-07-15; regenerate and review them whenever a boundary, ownership rule, table relationship, or deployment replica count changes.