Skip to main content

Chainguard base migration runbook

Phase 2 §5.1 + §5.2 + §5.3 + §5.4 of RESTRUCTURING_PLAN.md. Owns the cutover from Debian-slim base images to Chainguard Wolfi bases, the cosign + SBOM signing pipeline, and the Kyverno admission policies that gate signed-only images in production.

Scope​

Four AlphaSwarm-owned images are targeted to move to Chainguard Wolfi in Phase 2 §5.1. As of this writing only alphaswarm-client has actually landed on Chainguard bases on the mainline branch; the other three still build from Debian-based images (alphaswarm-api/alphaswarm-worker/ alphaswarm-controller are on a pinned python:3.12-slim, and alphaswarm-ui is on node:20-bookworm/node:20-bookworm-slim, not node:20-alpine). A Chainguard migration for these three exists on an unmerged branch; treat the "Base after" column below as the target state this runbook produces, not the current state, until that branch lands:

ImageDockerfileBase before (current)Base after (target)
alphaswarm-api / alphaswarm-worker (shared api target)alphaswarm_platform/Dockerfile (or alphaswarm_platform/build/docker/alphaswarm-runtime/Dockerfile, used by the current build-publish.yml)python:3.12-slimcgr.dev/chainguard/python:3.11-dev
alphaswarm-controlleralphaswarm_platform/build/docker/alphaswarm_controller/Dockerfilepython:3.12-slim (pinned by digest, via BUILD_BASE/RUNTIME_BASE build args)cgr.dev/chainguard/python:3.11-dev (builder) + cgr.dev/chainguard/python:3.11 (runtime)
alphaswarm-clientalphaswarm_platform/build/docker/alphaswarm_client/Dockerfile— (already migrated)cgr.dev/chainguard/node:20-dev + cgr.dev/chainguard/python:3.11-dev (builders) + cgr.dev/chainguard/python:3.11 (runtime) — confirmed in place
alphaswarm-uialphaswarm_platform/build/docker/alphaswarm_ui/Dockerfilenode:20-bookworm (builder) + node:20-bookworm-slim (runtime)cgr.dev/chainguard/node:20-dev (builder) + cgr.dev/chainguard/node:20 (runtime)

Two images carry documented exemptions and stay on their current bases:

ImageDockerfileReason
alphaswarm-bots standardalphaswarm_bots/DockerfileAlready on gcr.io/distroless/python3-debian12:nonroot — smaller and more locked-down than Chainguard Python, no shell at all. Builder stage stays on python:3.12-slim-bookworm for build-essential availability.
alphaswarm-bots HFTalphaswarm_bots/Dockerfile.hftKernel-bypass libs (DPDK, Onload, Mellanox OFED) require kernel headers + libnuma1 + linuxptp + ethtool + kmod which Chainguard's nonroot Wolfi runtime image does not ship. Per ADR 007.

Two future-phase scaffolds are created in Phase 2 §5.6:

ImageDockerfileActivation phase
alphaswarm-edge (Envoy cell router)alphaswarm_platform/build/docker/alphaswarm-edge/DockerfilePhase 3 §6.4 (cell topology)
alphaswarm-agent-sandbox (gVisor target)alphaswarm_platform/build/docker/alphaswarm-agent-sandbox/DockerfilePhase 5 §8 (per-tenant MCP + agent sandbox)

Why Chainguard Wolfi​

  • glibc, not musl — keeps native wheel compatibility for numpy, pyarrow, torch, psycopg2, etc. The RESTRUCTURING_PLAN footnote at §5.1 explicitly notes that Alpine/musl-style minimalism breaks the native-wheel toolchain.
  • Continuously rebuilt — Chainguard ships a fresh image set every ~24 hours, so CVE patches land without us doing anything beyond a rebuild. Pair with Renovate (Phase 1 §4.7) to re-trigger the build matrix on a base-image bump.
  • No CVEs in the base — Chainguard runs distroless-style scans and publishes a daily-zero-CVE SLO for latest tags. Application-level CVEs are still our responsibility: the current build-sign-push composite action runs a Trivy scan (severity: HIGH,CRITICAL) that blocks the build on findings. The grype/syft commands below are a manual/local verification path — neither tool is currently wired into CI (.github/workflows and .github/actions have no grype or syft step).
  • Single nonroot UID convention (65532) — matches the Phase 2 §5.4 PSS restricted profile. The runtime stages never run as root; the -dev builder runs as root only for apk add.

Build verification​

Local one-off build (no push, no signing — for inner-loop dev):

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --target api \
  --file alphaswarm_platform/Dockerfile \
  --tag alphaswarm-api:dev \
  .

Multi-arch build via build-publish.yml (CI canonical path — the workflow actually named build-multi-arch.yml does not exist; alphaswarm_platform and alphaswarm each carry their own build-publish.yml, triggered on version tags and via workflow_dispatch):

gh workflow run build-publish.yml \
  --ref feat/phase-2-supply-chain

The workflow signs every pushed image with cosign keyless OIDC via the build-sign-push composite action (alphaswarm_platform/.github/actions/build-sign-push/action.yml), which also attaches build provenance + an SBOM (docker/build-push-action's provenance: true / sbom: true) and runs a Trivy vulnerability scan (HIGH/CRITICAL findings block the build). There is currently no separate inspect job — verify signatures/attestations locally with cosign verify as below.

Verify cosign signature locally​

cosign verify \
  --certificate-identity-regexp 'https://github.com/Alpha-Swarm-ai/alphaswarm_platform/.github/workflows/build-publish\.yml@refs/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  docker.io/julianwiley/alphaswarm-api:latest

Expected exit code: 0. The output prints the signature payload including the Rekor transparency log entry index.

Verify CycloneDX SBOM attestation locally​

cosign verify-attestation \
  --certificate-identity-regexp 'https://github.com/Alpha-Swarm-ai/alphaswarm_platform/.github/workflows/build-publish\.yml@refs/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --type cyclonedx \
  docker.io/julianwiley/alphaswarm-api:latest > sbom-attestation.json

The predicate field of the attestation is the base64-encoded CycloneDX document.

Re-run grype against the SBOM​

syft docker.io/julianwiley/alphaswarm-api:latest -o cyclonedx-json=sbom.json
grype sbom:sbom.json --fail-on high

Exit code 0 = no HIGH or CRITICAL CVEs; non-zero = the gate fires in CI.

Kyverno audit-to-enforce ratchet​

The six Phase 2 §5.3 cluster policies below (deployments/kubernetes/security/kyverno/cluster-policies/00-* through 05-* in alphaswarm_platform) have already been ratcheted from Audit to validationFailureAction: Enforce in the current tree — including 02-require-runtime-class.yaml, which each in-tree policy comment now notes was flipped once the referenced follow-up phase landed. The workflow below is kept for reference (e.g. for onboarding a new policy through the same ratchet), not because these six are still pending:

PolicyAudit-mode soakEnforce gate
00-verify-signatures.yaml7 days zero violations across all AlphaSwarm-owned namespacesPhase 2.5 — done, now Enforce
01-require-pss-restricted.yaml7 days zero violationsPhase 2.5 — done, now Enforce
02-require-runtime-class.yamlWas held until Phase 5 §8.3 landed the gVisor RuntimeClassPhase 5.1 — done, now Enforce
03-no-host-network.yaml7 days zero violations after alphaswarm-edge namespace carries alphaswarm.io/host-network-allowed: "true"Phase 2.5 — done, now Enforce
04-no-privilege-escalation.yaml7 days zero violationsPhase 2.5 — done, now Enforce
05-required-labels.yaml7 days zero violations on namespaces that carry alphaswarm.io/componentPhase 2.5 — done, now Enforce

Three additional policies (06-money-plane-progressive-delivery.yaml, 07-restrict-image-registries.yaml, 08-deployment-version-labels.yaml) now also exist in the same directory; they postdate Phase 2 §5.3 and are out of scope for this runbook.

Operator workflow to flip Audit → Enforce​

# 1. Verify zero violations for the target policy:
kubectl get clusterpolicyreport -o jsonpath='{range .items[*].results[?(@.policy=="alphaswarm-verify-image-signatures")]}{.result}{"\n"}{end}' \
  | sort | uniq -c

# Expected output: only "pass" lines. Any "fail" lines block the ratchet.

# 2. Patch the policy in place:
kubectl patch clusterpolicy alphaswarm-verify-image-signatures \
  --type=merge \
  -p '{"spec":{"validationFailureAction":"Enforce"}}'

# 3. Update the YAML in tree so the audit-only state is preserved:
sed -i 's/validationFailureAction: Audit/validationFailureAction: Enforce/' \
  alphaswarm_platform/deployments/kubernetes/security/kyverno/cluster-policies/00-verify-signatures.yaml

# 4. Commit + open PR with `[Phase 2.5 ratchet]` in the title.

Rollback​

The Chainguard migration is reversible per Dockerfile. Each Dockerfile carries a Phase 2 §5.1 comment at the top documenting the previous base image. To roll back a single image:

  1. Revert that file in alphaswarm_platform/Dockerfile or alphaswarm_platform/build/docker/<service>/Dockerfile to its pre-Phase-2 state.
  2. Trigger build-multi-arch.yml on the revert branch.
  3. The cosign keyless signature still applies (it signs by digest, not base image). The grype scan may fail differently because the Debian-slim base ships different CVEs.

Cosign signing on PRs​

The Phase 2 §5.2 cosign + SBOM + grype steps gate on if: github.event_name != 'pull_request' because cosign keyless requires OIDC, which is unavailable on PRs from forked repositories. PRs from internal branches still build (and pull- through cache), but they neither push nor sign. The inspect job that runs cosign verify on :latest tags is only useful for merged commits.

If you need to verify a signature on a PR build, push to a feature branch in the canonical repo (not a fork) and check the registry manually:

docker pull docker.io/julianwiley/alphaswarm-api:feat-phase-2-supply-chain-<sha>
cosign verify \
  --certificate-identity-regexp 'https://github.com/Alpha-Swarm-ai/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  docker.io/julianwiley/alphaswarm-api:feat-phase-2-supply-chain-<sha>