Dagster sandbox

In plain English: operators can try a Dagster component (and an Airbyte connection rendered as a component) without writing into the production cache or warehouse. Each try is a session: its own folder, its own Redis key prefix, and a short-lived swap of six named endpoints while sandbox code is running. That isolation lives inside the sandbox package. Other modules that still read settings.* directly are not swapped. The live Dagster defs loader is refused while a sandbox env is active, because Dagster 1.13.13 has no endpoint-injection seam.

Session lifecycle

Every session is created through SandboxRuntime.create_session. The REST prefix is /dagster/sandbox. The router defaults to data:read; mutators additionally require data:write. Missing and foreign sessions return the same 404 sandbox session not found.

Step	Route	Scope	What happens
Create	POST /sessions	data:read (authenticated)	Unique tempdir, Redis namespace, default TTL 60 minutes (max 720). Owner / workspace / project stamped from the request context.
Write component	POST /sessions/{id}/components	data:write	YAML body written under the session folder after path-name validation.
Write Airbyte	POST /sessions/{id}/airbyte	data:write	Loads an Airbyte row with a tenancy filter (workspace_id or owner_user_id). Missing or foreign connection id is 404. Validated component name still applies.
Load	POST /sessions/{id}/load	data:write	SandboxExecutor.load() under enter_sandbox_env(...).
Execute	POST /sessions/{id}/execute	data:write	Enqueues execute_sandbox_session.delay(session_id). Response is {task_id, stream_url} where stream_url is /chat/stream/{task_id}.
Inspect	GET /sessions, GET /sessions/{id}	data:read	List is the in-process _registry filtered by tenancy. Get-by-id uses get(...) or rehydrate(...).
Teardown	DELETE /sessions/{id}	data:write	See Inspect and teardown.
Janitor	POST /janitor	platform:admin	Registry-wide reap of in-process expired sessions.

Frontend: /data/sandbox (SandboxConsole). Stream frames reuse useChatStream.

Four isolation mechanisms

These are the four boxes rule 32 names. Progress frames are the rule-4 contract, documented under Celery and progress frames, not a fifth isolation box.

1. Unique tempdir. tempfile.mkdtemp(prefix=f"alphaswarm_sandbox_{session_id}_"). Rehydrate allocates a new tempdir on the worker that rebuilds the session.
2. Redis prefix alphaswarm:sandbox:<session_id>:* via SandboxRedisNamespace. Never alphaswarm:cache:*. Teardown is KEYS + DEL under that prefix.
3. SandboxEnvResolver ContextVar for the six named endpoints (see Endpoint isolation). Does not mutate the cached settings singleton.
4. Expiry + janitor. Each session stores expires_at / ttl_minutes. SandboxRuntime.janitor() tears down expired entries still in this process _registry (folder + Redis namespace). There is no Celery beat entry and no CLI flag; operators call POST /janitor.

Isolation is not whole-process or whole-platform. It is session folder + Redis prefix + ContextVar overrides on those six names inside the sandbox package, plus path-name validation on writes, plus cross-process rehydrate through the persisted row.

Component-name validation

write_component (and write_airbyte_connection, which writes through it) reject path escapes: ../, absolute paths (POSIX and Windows), NUL bytes, and empty names. The runtime raises SandboxPathError("invalid component name").

Routes map that to HTTP 400 with body "invalid component name". The rejected name, session folder, and resolved filesystem path are not returned.

Cross-process rehydrate

Workers do not share the in-process _registry. SandboxRuntime.rehydrate(session_id) reads DagsterSandboxSessionRow, preserves owner / workspace / project / expiry / TTL / env metadata, rematerializes components through write_component() (the path guards stay on), and uses a new per-worker tempdir. Closed or already-expired rows are not rebuilt.

Celery execute_sandbox_session and REST _get_visible_runtime both do get(...) or rehydrate(...). Foreign sessions still return identical 404 sandbox session not found.

Alembic 0161_dagster_sandbox_rehydrate_state is the persistence seam: it added env_overrides_json and ttl_minutes on dagster_sandbox_sessions. Row writes from the REST layer are best-effort (a persistence failure does not fail the HTTP handler).

Endpoint isolation (partial)

Covered names (SANDBOX_ENDPOINT_NAMES):

• iceberg_rest_uri
• iceberg_warehouse
• polaris_base_url
• alpha_vantage_base_url
• datahub_gms_url
• kafka_bootstrap

Accessors: sandbox_aware_endpoint(name) / sandbox_aware_endpoints(). Active SandboxEnvResolver ContextVar first; else from alphaswarm.config import settings. The cached Settings object is not mutated.

SandboxExecutor.load() under an active sandbox env refuses the live Dagster defs loader (1.13.13 has no endpoint-injection seam) and uses the manifest fallback with metadata code sandbox_endpoint_isolation_unavailable. stream_execute() drives load() inside enter_sandbox_env(...) and carries that refusal on the start frame (error_code, live_loader, fallback_loader, endpoint names/counts). Endpoint values are not surfaced.

Not isolated. Out-of-package call sites still read settings.* directly, including:

• alphaswarm/data/iceberg_catalog.py (iceberg_rest_uri, iceberg_warehouse)
• alphaswarm/services/service_manager.py (polaris_base_url, iceberg_rest_uri)
• alphaswarm/services/polaris_client.py (polaris_base_url)
• alphaswarm/tasks/visualization_tasks.py (datahub_gms_url)
• alphaswarm/data/datahub/client.py, aspect_puller.py (datahub_gms_url)
• alphaswarm/data/sources/alpha_vantage/datahub.py (datahub_gms_url)
• alphaswarm/trading/feeds/kafka_feed.py (kafka_bootstrap)
• alphaswarm/streaming/kafka_producer.py, clusters.py, admin/kafka_admin.py (kafka_bootstrap)
• alphaswarm/data/lakehouse/hudi/hudi_streamer.py (kafka_bootstrap)
• alphaswarm/data/fetchers/stream/kafka_fetcher.py, sinks/kafka_sink.py (kafka_bootstrap)

Do not treat the sandbox as a platform-wide endpoint firewall. Credentials still resolve through CredentialResolver; the ContextVar only swaps those six URL-shaped settings inside sandbox accessors.

Celery and progress frames

execute_sandbox_session streams through _progress.emit. Frames stay {task_id, stage, message, timestamp, **extras}. Extras include session_id, asset_key, and metadata (including the live-loader refusal code when a sandbox env is active). The frontend reuses useChatStream.

Inspect and teardown

Live HTTP only — there is no sandbox CLI.

DELETE /sessions/{id} calls SandboxRuntime.teardown() then marks the persisted row status="closed" (the row is not deleted):

• Redis keys under alphaswarm:sandbox:<session_id>:*
• the session tempdir (shutil.rmtree)
• the in-process _registry entry

POST /janitor (platform:admin) walks this process _registry and tears down sessions past expires_at the same way (folder + Redis). It does not scan Postgres for expired rows on other workers, and it does not itself persist closed. Rehydrate already refuses expired or closed rows, so a worker that never had the session in _registry will not rebuild it after expiry.

GET /sessions lists only the in-process registry (tenancy-filtered). Use GET /sessions/{id} to resolve a persisted session via rehydrate.

Production jobs, schedules, and instance config live on the Dagster operations page.