fix(ci): universe push retry is shallow-clone-safe (fetch+reset+reapply, not rebase)

v1.0.7's `git pull --rebase` retry never worked: the universe clone is
`--depth 1` shallow, so rebase has no merge base and fails immediately →
push stayed rejected under concurrent rolls, hard-failing the Deploy step
with the expired kubeconfig (hanzoai/world v2.4.44 stuck at 2.4.43).

Replace rebase with a shallow-safe loop: on a non-fast-forward reject,
`git fetch --depth 1 origin main` + `reset --hard FETCH_HEAD`, re-apply the
one-file CR tag change, re-commit, retry (6×). A no-op after reset (remote
already carries our tag) counts as recorded. Never force-push. Verified
against a real shallow clone with a concurrent racing push: our CR change
lands, the racer's other-file commit is preserved.

Claude-Session: https://claude.ai/code/session_01NP4ehjWW2h98FkE4YjUKuJ

.github/workflows/build.yml

@@ -15,7 +15,19 @@ on:
           the build, metered as build minutes). Bring-your-own: pass the labels
           of your own self-hosted arc runners instead.
         type: string
-        default: '["self-hosted","linux","amd64"]'
+        default: '["hanzo-build-linux-amd64"]'
+      mode:
+        description: >-
+          Build execution mode. `buildx` (default) runs the full buildx →
+          test → deploy pipeline ON the arc runner. `delegate` instead POSTs the
+          build to platform.hanzo.ai (`/v1/arcd/enqueue`) — platform builds
+          in-cluster with BuildKit and rolls the operator Service CR itself, so
+          the GitHub job finishes in seconds with no runner buildx. A repo opts
+          in by passing `with: { mode: delegate }`; everything else is unchanged.
+          Requires the `PLATFORM_BUILD_CALLBACK_TOKEN` secret (via secrets:
+          inherit).
+        type: string
+        default: buildx
 
 permissions:
   contents: read
@@ -27,7 +39,88 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
+      - name: Provision parse toolchain (jq + PyYAML)
+        # This reusable parses the caller's hanzo.yml with python3 + PyYAML and
+        # slices JSON with jq. The stock arc runner image
+        # (ghcr.io/actions/actions-runner:latest) is minimal and ships NEITHER,
+        # so provision them here. Guarded (a no-op the moment a runner image bakes
+        # them in) — this keeps the reusable self-contained: any org can import it
+        # onto a bare runner and it just works.
+        run: |
+          set -e
+          # Sudo-free FIRST (bare arc nodes often lack passwordless sudo / apt
+          # network — `sudo apt-get` then dies with no captured logs). Install jq
+          # as a static binary and PyYAML via pip --user into ~/.local/bin; fall
+          # back to apt only if those are unavailable. Works on bare AND baked nodes.
+          export PATH="$HOME/.local/bin:$PATH"
+          mkdir -p "$HOME/.local/bin"
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+          if ! command -v jq >/dev/null 2>&1; then
+            curl -fsSL https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64 \
+              -o "$HOME/.local/bin/jq" && chmod +x "$HOME/.local/bin/jq" \
+            || { sudo apt-get update -qq && sudo apt-get install -y -qq jq; }
+          fi
+          # yq (mikefarah) — static Go binary, curl-installed like jq. Replaces the
+          # PyYAML/python3 YAML parse that could not be provisioned on locked-down
+          # arc nodes (sudo blocked by no_new_privs, pip/PyPI unavailable).
+          if ! command -v yq >/dev/null 2>&1; then
+            curl -fsSL https://github.com/mikefarah/yq/releases/download/v4.44.3/yq_linux_amd64 \
+              -o "$HOME/.local/bin/yq" && chmod +x "$HOME/.local/bin/yq"
+          fi
+          jq --version
+          yq --version
+
+      - name: Delegate build to platform (mode=delegate)
+        # The GHA-escape fast path: instead of running buildx on this runner, POST
+        # each image in hanzo.yml to platform.hanzo.ai's direct-enqueue webhook
+        # (`/v1/arcd/enqueue`). Platform creates a build_job row, launches an
+        # in-cluster BuildKit Job on its own pool, pushes to the registry, and —
+        # for a system service — patches the operator Service CR to roll it. The
+        # downstream is IDENTICAL to the platform GitHub-App webhook path (one
+        # build path, two front doors), so a delegated build behaves exactly like
+        # a platform-native one. This job then exits in seconds — no buildx, no
+        # KMS, no runner-side deploy.
+        if: inputs.mode == 'delegate'
+        env:
+          ENQUEUE_URL: ${{ vars.PLATFORM_ENQUEUE_URL || 'https://platform.hanzo.ai/v1/arcd/enqueue' }}
+          ENQUEUE_TOKEN: ${{ secrets.PLATFORM_BUILD_CALLBACK_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${ENQUEUE_TOKEN:-}" ]; then
+            echo "::error::mode=delegate needs the PLATFORM_BUILD_CALLBACK_TOKEN secret (secrets: inherit)"; exit 1
+          fi
+          REPO="${{ github.repository }}"
+          SHA="${{ github.sha }}"
+          SHORT=$(echo "$SHA" | cut -c1-7)
+          REF="${{ github.ref }}"
+          BRANCH="${{ github.ref_name }}"
+          # One enqueue per (image, platform), mirroring the buildx tag shape the
+          # deploy path expects (`sha-<short>-<arch>[-<suffix>]`). Default arch is
+          # amd64 (single-arch), so an existing repo's tag shape is unchanged.
+          yq -o=json -I=0 '.images' hanzo.yml | jq -c '.[]' | while read -r img; do
+            name=$(echo "$img"|jq -r .name); repo=$(echo "$img"|jq -r .repo)
+            ctx=$(echo "$img"|jq -r .context); df=$(echo "$img"|jq -r '.dockerfile // (.context+"/Dockerfile")')
+            sfx=$(echo "$img"|jq -r '."tag-suffix" // ""')
+            echo "$img" | jq -r '(.platforms // ["linux/amd64"])[]' | while read -r plat; do
+              arch="${plat##*/}"
+              image="${repo}:sha-${SHORT}-${arch}${sfx:+-$sfx}"
+              body=$(jq -nc \
+                --arg repo "$REPO" --arg sha "$SHA" --arg image "$image" \
+                --arg ref "$REF" --arg branch "$BRANCH" \
+                --arg dockerfile "$df" --arg context "$ctx" --arg arch "$arch" \
+                '{repo:$repo,sha:$sha,image:$image,ref:$ref,branch:$branch,dockerfile:$dockerfile,context:$context,os:"linux",arch:$arch}')
+              echo "::group::delegate $name → $image"
+              code=$(curl -sS -o /tmp/enqueue.out -w '%{http_code}' -X POST "$ENQUEUE_URL" \
+                -H "Authorization: Bearer $ENQUEUE_TOKEN" -H 'Content-Type: application/json' -d "$body")
+              cat /tmp/enqueue.out; echo
+              # 202 Accepted = queued; 409 = no live runner for the pool (surface it loud).
+              if [ "$code" != "202" ]; then echo "::error::enqueue $image failed (HTTP $code)"; exit 1; fi
+              echo "::endgroup::"
+            done
+          done
+
       - name: Authenticated git for go modules (rate-limit + any private repo)
+        if: inputs.mode != 'delegate'
         # luxfi/hanzoai/zooai Go modules are PUBLIC, so `go` resolves them through
         # the default public proxy (proxy.golang.org) + checksum db (sum.golang.org)
         # — canonical, IMMUTABLE hashes that a force-moved tag can no longer break.
@@ -63,22 +156,39 @@ jobs:
             && echo "git auth OK" \
             || echo "::warning::GH_PAT set but repo probe failed"
 
-      - name: Log in to GHCR (automatic workflow token)
-        # GHCR push needs no stored credential: GitHub injects a per-job
-        # GITHUB_TOKEN scoped to THIS repo, and `permissions: packages: write`
-        # (above) lets it push the repo's own package (e.g. zooai/node →
-        # ghcr.io/zooai/node, same org). First push creates the package
-        # repo-linked. No KMS, no PAT for the push path.
-        run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
+      - name: Log in to GHCR (GH_PAT when present, else automatic token)
+        if: inputs.mode != 'delegate'
+        env:
+          GH_PAT: ${{ secrets.GH_PAT }}
+        # Prefer the KMS-backed GH_PAT (admin:org + write:packages): it can push
+        # or CREATE any <org> package regardless of which repo the package is
+        # linked to. The per-job GITHUB_TOKEN only writes a package linked to THIS
+        # repo, so it 403s on a package created/linked elsewhere (e.g.
+        # ghcr.io/hanzoai/cms). Fall back to the automatic token when GH_PAT is
+        # absent (public forks like zooai/node that create their own repo-linked
+        # package on first push).
+        run: |
+          if [ -n "${GH_PAT:-}" ]; then
+            echo "$GH_PAT" | docker login ghcr.io -u hanzo-dev --password-stdin
+          else
+            echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
+          fi
 
       - name: Fetch deploy credentials from KMS
         id: kms
+        if: inputs.mode != 'delegate'
         env:
           KMS_CLIENT_ID: ${{ secrets.KMS_CLIENT_ID }}
           KMS_CLIENT_SECRET: ${{ secrets.KMS_CLIENT_SECRET }}
           KMS_ENDPOINT: ${{ vars.KMS_ENDPOINT || 'https://kms.hanzo.ai' }}
           KMS_ORG: ${{ vars.KMS_ORG }}
         run: |
+          # This step is BEST-EFFORT (see below): GHCR push already works via the
+          # workflow token, and deploy creds are optional. GitHub wraps `run:` in
+          # `bash -eo pipefail`, so we MUST explicitly `set +e` — otherwise an
+          # unguarded curl (e.g. a KMS secret 404 at this org/path) aborts the
+          # step and fails the whole build. Keep -u/-o pipefail; drop -e.
+          set +e
           set -uo pipefail
           # Canonical luxfi/kms surface — /v1/kms (the Infisical /api/* surface was
           # removed when KMS migrated to luxfi/kms, MPC-rooted). Auth = an IAM
@@ -93,10 +203,26 @@ jobs:
           #     repo-scoped automatic token cannot read. Handed to buildx as the
           #     `gh_token` BuildKit secret.
           #   KUBECONFIG  — cluster access for the deploy step.
-          ORG="$(python3 -c "import yaml;print((yaml.safe_load(open('hanzo.yml')).get('kms') or {}).get('org','') )" 2>/dev/null || true)"
+          ORG="$(yq -r '.kms.org // ""' hanzo.yml 2>/dev/null || true)"
           : "${ORG:=${KMS_ORG:-}}"
-          ENV="$(python3 -c "import yaml;print((yaml.safe_load(open('hanzo.yml')).get('kms') or {}).get('environment','prod'))" 2>/dev/null || echo prod)"
-          PATHQ="$(python3 -c "import yaml;print((yaml.safe_load(open('hanzo.yml')).get('kms') or {}).get('path','/deploy').strip('/'))" 2>/dev/null || echo deploy)"
+          # Systemic default: when neither hanzo.yml `kms.org` nor the KMS_ORG var
+          # is set, derive the KMS org from the GitHub owner. The org → KMS-org
+          # relation is a small, stable first-party fact; keeping it here (ONE
+          # place) is why every repo gets the canonical KMS deploy-cred path with
+          # zero per-repo config — before this, ORG was always empty and this whole
+          # step short-circuited, so no build ever fetched its private-dep git
+          # credential from KMS (it silently fell back to GH_PAT and failed on
+          # private hanzoai/cloud). hanzo.yml `kms.org` and KMS_ORG still override.
+          if [ -z "$ORG" ]; then
+            case "${{ github.repository_owner }}" in
+              hanzoai) ORG=hanzo ;;
+              luxfi)   ORG=lux ;;
+              zooai)   ORG=zoo ;;
+              *)       ORG="${{ github.repository_owner }}" ;;
+            esac
+          fi
+          ENV="$(yq -r '.kms.environment // "prod"' hanzo.yml 2>/dev/null || echo prod)"
+          PATHQ="$(yq -r '.kms.path // "/deploy"' hanzo.yml | sed 's#^/##; s#/$##' 2>/dev/null || echo deploy)"
           if [ -z "${KMS_CLIENT_ID:-}" ] || [ -z "$ORG" ]; then
             echo "::notice::KMS not configured (no KMS_CLIENT_ID or org) — skipping; GHCR push uses the workflow token"; exit 0
           fi
@@ -111,59 +237,415 @@ jobs:
           if [ -n "$GIT_TOKEN" ]; then echo "::add-mask::$GIT_TOKEN"; echo "GIT_TOKEN=$GIT_TOKEN" >> "$GITHUB_ENV"; fi
           KUBECONFIG_B64=$(get KUBECONFIG)
           if [ -n "$KUBECONFIG_B64" ]; then echo "$KUBECONFIG_B64" | base64 -d > "$RUNNER_TEMP/kubeconfig"; echo "kubeconfig=$RUNNER_TEMP/kubeconfig" >> "$GITHUB_OUTPUT"; fi
+          # Build-time secrets → --build-arg. A repo declares per-image
+          # `build_secrets: [NAME, ...]` in hanzo.yml; each NAME is fetched from
+          # the SAME org/path/env and exported (masked) so the build step bakes
+          # it in as `--build-arg NAME=value`. The KMS key name IS the build-arg
+          # name — one name, one place. For publishable client tokens a Vite SPA
+          # must embed at build (e.g. VITE_MAPBOX_TOKEN). Undeclared → no-op, so
+          # every existing repo is byte-for-byte unchanged.
+          for bs in $(yq -r '[(.images // [])[] | (.build_secrets // [])[]] | unique | .[]' hanzo.yml 2>/dev/null || true); do
+            case "$bs" in ''|*[!A-Za-z0-9_]*) echo "::warning::skipping invalid build_secret name '$bs'"; continue;; esac
+            v=$(get "$bs" || true)
+            if [ -n "$v" ]; then
+              echo "::add-mask::$v"
+              { echo "$bs<<__KMS_BUILDARG_EOF__"; echo "$v"; echo "__KMS_BUILDARG_EOF__"; } >> "$GITHUB_ENV"
+            else echo "::warning::build_secret $bs not in KMS ($ORG/$PATHQ env=$ENV) — build-arg will be empty"; fi
+          done
+
+      - name: GHCR push credential (KMS write:packages token)
+        if: inputs.mode != 'delegate'
+        env:
+          KUBECONFIG: ${{ steps.kms.outputs.kubeconfig }}
+        # The per-job GITHUB_TOKEN — and a package-less GH_PAT — can only push a
+        # package LINKED to this repo, so they 403 on a package created/linked
+        # elsewhere (ghcr.io/hanzoai/cms). Upgrade the ghcr login to the org's
+        # ghcr push token from the cluster (buildx-ghcr-auth, admin write:packages,
+        # KMS-synced) when a kubeconfig is present — it pushes/creates ANY <org>
+        # package. Fail-safe: keep the earlier login if anything is unavailable
+        # (public forks with no KMS keep their repo-linked GITHUB_TOKEN push).
+        run: |
+          set -uo pipefail
+          [ -z "${KUBECONFIG:-}" ] && { echo "::notice::no kubeconfig — keeping the earlier GHCR login"; exit 0; }
+          command -v kubectl >/dev/null 2>&1 || {
+            KVER=$(curl -fsSL https://dl.k8s.io/release/stable.txt)
+            mkdir -p "$HOME/.local/bin"
+            curl -fsSL "https://dl.k8s.io/release/${KVER}/bin/linux/amd64/kubectl" -o "$HOME/.local/bin/kubectl" && chmod +x "$HOME/.local/bin/kubectl"
+            export PATH="$HOME/.local/bin:$PATH"
+          }
+          CFG=$(kubectl -n hanzo get secret buildx-ghcr-auth -o jsonpath='{.data.\.dockerconfigjson}' 2>/dev/null | base64 -d || true)
+          [ -z "$CFG" ] && { echo "::notice::buildx-ghcr-auth not readable — keeping the earlier GHCR login"; exit 0; }
+          UP=$(echo "$CFG" | jq -r '.auths | to_entries[] | select(.key|test("ghcr")) | .value.auth' | head -1 | base64 -d 2>/dev/null || true)
+          [ -z "$UP" ] && { echo "::notice::no ghcr auth in buildx-ghcr-auth — keeping the earlier login"; exit 0; }
+          echo "::add-mask::${UP#*:}"
+          if echo "${UP#*:}" | docker login ghcr.io -u "${UP%%:*}" --password-stdin; then
+            echo "::notice::GHCR login upgraded to the KMS write:packages token"
+          else
+            echo "::notice::KMS GHCR token login failed — keeping the earlier login"
+          fi
+
+      - name: Mirror credential (registry.hanzo.ai)
+        if: inputs.mode != 'delegate'
+        env:
+          KUBECONFIG: ${{ steps.kms.outputs.kubeconfig }}
+          REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
+          REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
+        run: |
+          set -uo pipefail
+          # Direct credential first (repo/org secret — works on private repos,
+          # where the Free plan hides org secrets... including these; private
+          # repos set them at REPO level). KMS-kubeconfig read is the fallback.
+          if [ -n "${REGISTRY_USER:-}" ] && [ -n "${REGISTRY_PASSWORD:-}" ]; then
+            # Best-effort: a registry.hanzo.ai login FAILURE (not just a missing
+            # cred) must never fail the run — the image still pushes to GHCR, the
+            # primary. Without this guard, bash -e aborts the step and SKIPS the
+            # build entirely (a registry hiccup takes the whole lane red).
+            if echo "$REGISTRY_PASSWORD" | docker login registry.hanzo.ai -u "$REGISTRY_USER" --password-stdin; then
+              echo "MIRROR_OK=1" >> "$GITHUB_ENV"
+            else
+              echo "::notice::registry.hanzo.ai login failed — mirror skipped (GHCR-only push)"
+            fi
+            exit 0
+          fi
+          [ -z "${KUBECONFIG:-}" ] && { echo "::notice::no registry secret and no kubeconfig — mirror skipped"; exit 0; }
+          # Bare arc runners ship no kubectl — same static provision the deploy
+          # step uses.
+          command -v kubectl >/dev/null 2>&1 || {
+            KVER=$(curl -fsSL https://dl.k8s.io/release/stable.txt)
+            mkdir -p "$HOME/.local/bin"
+            curl -fsSL "https://dl.k8s.io/release/${KVER}/bin/linux/amd64/kubectl"               -o "$HOME/.local/bin/kubectl" && chmod +x "$HOME/.local/bin/kubectl"
+            echo "$HOME/.local/bin" >> "$GITHUB_PATH"; export PATH="$HOME/.local/bin:$PATH"
+          }
+          CFG=$(kubectl -n hanzo get secret registry-credentials                   -o jsonpath='{.data.\.dockerconfigjson}' 2>/dev/null | base64 -d || true)
+          if [ -z "$CFG" ]; then
+            echo "::notice::registry-credentials not readable from this kubeconfig — mirror skipped (GHCR-only push)"; exit 0
+          fi
+          USERPASS=$(echo "$CFG" | jq -r '.auths["registry.hanzo.ai"].auth // empty' | base64 -d)
+          [ -z "$USERPASS" ] && { echo "::notice::no registry.hanzo.ai auth in dockerconfig — mirror skipped"; exit 0; }
+          echo "::add-mask::${USERPASS#*:}"
+          # Best-effort: login failure → skip mirror, never fail the run (see above).
+          if echo "${USERPASS#*:}" | docker login registry.hanzo.ai -u "${USERPASS%%:*}" --password-stdin; then
+            echo "MIRROR_OK=1" >> "$GITHUB_ENV"
+          else
+            echo "::notice::registry.hanzo.ai login failed — mirror skipped (GHCR-only push)"
+          fi
 
       - name: Build & push images (per hanzo.yml)
+        if: inputs.mode != 'delegate'
+        env:
+          GH_PAT: ${{ secrets.GH_PAT }}
         run: |
           set -euo pipefail
+          # Test-only callers (hanzo.yml without `images:` — e.g. a repo whose
+          # image lane lives in its own release.yml, or a pure library) skip the
+          # build step entirely instead of exploding on a null .images.
+          if [ "$(yq -r '.images // [] | length' hanzo.yml 2>/dev/null || echo 0)" = "0" ]; then
+            echo "::notice::no images: in hanzo.yml — test-only caller, skipping build"; exit 0
+          fi
+          # Build-time private cross-org Go module read (the buildx `gh_token`
+          # secret): prefer the KMS-fetched GIT_TOKEN, else fall back to the org
+          # GH_PAT — the SAME BuildKit gh_token cloud's release.yml uses (proven
+          # working). Keeps image builds green when the KMS deploy-cred fetch is
+          # unavailable (a repo with a private cross-org dep like hanzoai/cloud
+          # otherwise fails `go mod tidy` with git exit 128 in the buildx stage).
+          # No-op for public-only builds when both are empty. Exported so the
+          # `--secret id=gh_token,env=GIT_TOKEN` below reads it from the env.
+          export GIT_TOKEN="${GIT_TOKEN:-${GH_PAT:-}}"
           SHORT=$(echo "${{ github.sha }}" | cut -c1-7)
           IS_TAG=$([ "${{ github.ref_type }}" = "tag" ] && echo 1 || echo 0)
           VER="${{ github.ref_name }}"; VER="${VER#v}"
-          python3 -c "import yaml,json;print(json.dumps(yaml.safe_load(open('hanzo.yml'))['images']))" | jq -c '.[]' | while read -r img; do
+          yq -o=json -I=0 '.images' hanzo.yml | jq -c '.[]' | while read -r img; do
             name=$(echo "$img"|jq -r .name); ctx=$(echo "$img"|jq -r .context)
             df=$(echo "$img"|jq -r '.dockerfile // (.context+"/Dockerfile")'); repo=$(echo "$img"|jq -r .repo)
             # tag-suffix is OPTIONAL (most repos ship a single variant). When set
             # (e.g. "ce"/"ee") it qualifies every tag; when absent the tags are
             # clean (no trailing dash). Build + deploy must agree on this shape.
             sfx=$(echo "$img"|jq -r '."tag-suffix" // ""')
-            TAGS="-t $repo:sha-${SHORT}-amd64${sfx:+-$sfx} -t $repo:${sfx:+$sfx-}latest"
-            [ "$IS_TAG" = 1 ] && TAGS="$TAGS -t $repo:${VER}-amd64${sfx:+-$sfx}"
-            echo "::group::build $name → $repo (${sfx})"
+            # platforms is OPT-IN per image in hanzo.yml (default: amd64 only, so
+            # every existing repo's tag shape "-amd64" is UNCHANGED). Set e.g.
+            #   platforms: [linux/amd64, linux/arm64]
+            # to emit a multi-arch MANIFEST LIST — one digest serving both arches.
+            # DOKS has no arm64 nodes, so arm64 builds via buildx QEMU emulation
+            # (binfmt set up below); pure-Go (CGO_ENABLED=0) Dockerfiles that honor
+            # $TARGETARCH cross-compile natively (fast, no emulation). For true
+            # native-speed arm64, register a bare-metal arm64 host (spark/GB10) as
+            # the hanzo-build-linux-arm64 self-hosted runner (values-build-arm64.yaml).
+            plats=$(echo "$img"|jq -r '(.platforms // ["linux/amd64"]) | join(",")')
+            if [ "$plats" = "linux/amd64" ]; then
+              # single-arch: keep the exact legacy tag shape (-amd64) deploys expect.
+              TAGS="-t $repo:sha-${SHORT}-amd64${sfx:+-$sfx} -t $repo:${sfx:+$sfx-}latest"
+              # Release (tag) build: publish the CANONICAL bare semver tag (what the
+              # universe CR pins to — matches world 2.4.10 / cloud v1.801.62) AND the
+              # legacy -amd64 semver alias (back-compat for CRs still on that shape).
+              [ "$IS_TAG" = 1 ] && TAGS="$TAGS -t $repo:${VER}${sfx:+-$sfx} -t $repo:${VER}-amd64${sfx:+-$sfx}"
+            else
+              # multi-arch: one arch-neutral manifest-list tag (no -amd64 suffix).
+              docker run --privileged --rm tonistiigi/binfmt --install arm64 >/dev/null 2>&1 || true
+              TAGS="-t $repo:sha-${SHORT}${sfx:+-$sfx} -t $repo:${sfx:+$sfx-}latest"
+              [ "$IS_TAG" = 1 ] && TAGS="$TAGS -t $repo:${VER}${sfx:+-$sfx}"
+            fi
+            echo "::group::build $name → $repo (${sfx}) [$plats]"
+            # --build-arg assembly: static hanzo.yml `args` (a fixed value, e.g. a
+            # pinned base image tag) + `build_secrets` (KMS values the KMS step
+            # exported into the env above). Empty when a repo declares neither, so
+            # the buildx line is unchanged for every existing repo.
+            BUILD_ARGS=""
+            while IFS= read -r kv; do [ -n "$kv" ] && BUILD_ARGS="$BUILD_ARGS --build-arg $kv"; done \
+              < <(echo "$img" | jq -r '(.args // {}) | to_entries[] | "\(.key)=\(.value)"')
+            for bs in $(echo "$img" | jq -r '(.build_secrets // [])[]'); do
+              v=$(printenv "$bs" 2>/dev/null || true); [ -n "$v" ] && BUILD_ARGS="$BUILD_ARGS --build-arg $bs=$v"
+            done
             # GIT_TOKEN (from KMS, via GITHUB_ENV) is passed as the `gh_token`
             # BuildKit secret so Dockerfiles can clone private Go modules; omitted
             # cleanly when absent (public-only builds unaffected).
-            docker buildx build --platform linux/amd64 ${GIT_TOKEN:+--secret id=gh_token,env=GIT_TOKEN} --push $TAGS -f "$df" "$ctx"
+            docker buildx build --platform "$plats" $BUILD_ARGS ${GIT_TOKEN:+--secret id=gh_token,env=GIT_TOKEN} --push $TAGS -f "$df" "$ctx"
+            # Dual-host: mirror the exact tag set to registry.hanzo.ai (server-
+            # side manifest copy — no rebuild). ghcr.io/<org>/<name> →
+            # registry.hanzo.ai/<org>/<name>; public consumers keep ghcr, the
+            # fleet DEPLOYS from our registry.
+            if [ "${MIRROR_OK:-}" = "1" ]; then
+              # crane, not buildx imagetools: the IAM token realm doesn't answer
+              # buildx's multi-scope token request (spec gap, tracked).
+              command -v crane >/dev/null 2>&1 || {
+                mkdir -p "$HOME/.local/bin"
+                curl -fsSL https://github.com/google/go-containerregistry/releases/download/v0.20.2/go-containerregistry_Linux_x86_64.tar.gz \
+                  | tar -xz -C "$HOME/.local/bin" crane
+                export PATH="$HOME/.local/bin:$PATH"
+              }
+              mrepo="registry.hanzo.ai/${repo#*/}"
+              echo "$TAGS" | tr ' ' '\n' | grep -v '^-t$' | grep -v '^$' | while read -r ref; do
+                crane copy "$ref" "${mrepo}:${ref##*:}" \
+                  || echo "::warning::mirror of $ref failed (ghcr push unaffected)"
+              done
+            fi
             echo "::endgroup::"
           done
 
+      - name: Provision Go toolchain (go test gates on bare runners)
+        # hanzo.yml `test:` gates (e.g. `go vet ./...`, `go test ...`) run
+        # DIRECTLY on the runner, NOT inside a build container — but the stock
+        # arc runner image (ghcr.io/actions/actions-runner) ships no Go, so a Go
+        # gate dies with `go: command not found` (exit 127). Provision the repo's
+        # OWN Go version from go.mod so the toolchain matches the module exactly.
+        # Guarded to Go repos (go.mod present) so pure-JS/TS callers are
+        # unaffected; harmless if a future runner image bakes Go in.
+        if: inputs.mode != 'delegate' && hashFiles('go.mod') != ''
+        uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+          cache: false
+
+      - name: Provision C toolchain (cgo test gates)
+        # CGO_ENABLED=1 gates (e.g. go-sqlite3, which bundles the sqlite
+        # amalgamation) need a C compiler; the minimal arc runner ships none
+        # (`cgo: gcc not found`). Same guarded provision the parse-toolchain step
+        # above uses — a no-op when gcc is already present. If apt-get update
+        # fails (arc snapshot mirror rot: "no longer has a Release file"),
+        # repoint archive.ubuntu.com at the DO mirror — both sources.list and
+        # noble's deb822 ubuntu.sources — and retry once.
+        if: inputs.mode != 'delegate' && hashFiles('go.mod') != ''
+        run: |
+          command -v gcc >/dev/null 2>&1 && exit 0
+          sudo apt-get update -qq || {
+            sudo find /etc/apt -name '*.list' -o -name '*.sources' | \
+              xargs -r sudo sed -i 's|https\?://archive.ubuntu.com/ubuntu|http://mirrors.digitalocean.com/ubuntu|g'
+            sudo apt-get update -qq
+          }
+          sudo apt-get install -y -qq gcc
+
+      - name: Provision Node toolchain (JS test gates)
+        # JS/TS `test:` gates (e.g. `pnpm install --frozen-lockfile && pnpm lint`)
+        # also run DIRECTLY on the runner, and the stock arc runner image ships
+        # no Node — so the gate dies at `corepack: command not found` (exit 127)
+        # before it ever reads package.json. Same guarded provision as the Go
+        # step above: only for JS callers (package.json present), harmless if a
+        # future runner image bakes Node in. `corepack enable` shims the repo's
+        # own packageManager (pnpm/yarn) at its pinned version.
+        if: inputs.mode != 'delegate' && hashFiles('package.json') != ''
+        uses: actions/setup-node@v4
+        with:
+          node-version: 22
+      - name: Enable corepack (pnpm/yarn shims for JS test gates)
+        if: inputs.mode != 'delegate' && hashFiles('package.json') != ''
+        run: corepack enable
+
+      - name: Authenticate runner git for private Go modules (test gates)
+        # The Test step runs `go vet`/`go test` ON the runner (not in buildx), so
+        # `go` fetches private hanzoai/* modules (GOPRIVATE → direct) through the
+        # runner's git, which needs a credential. The IMAGE build authenticates
+        # via the KMS `gh_token` inside buildx (GIT_TOKEN); the earlier GH_PAT
+        # git-auth step is a no-op for repos that rely on that KMS token (GH_PAT
+        # unset) — so `go vet` dies with `could not read Username for github.com`.
+        # Reuse the SAME token here for the runner's git (GIT_TOKEN, set by the
+        # KMS step above; GH_PAT fallback). No-op when neither is present.
+        if: inputs.mode != 'delegate' && hashFiles('go.mod') != ''
+        env:
+          GH_PAT: ${{ secrets.GH_PAT }}
+        run: |
+          set -uo pipefail
+          TOKEN="${GIT_TOKEN:-${GH_PAT:-}}"
+          if [ -z "$TOKEN" ]; then echo "no git token — public modules only"; exit 0; fi
+          CFG="$RUNNER_TEMP/gitconfig-go-test"; : > "$CFG"
+          GIT_CONFIG_GLOBAL="$CFG" git config --global \
+            url."https://x-access-token:${TOKEN}@github.com/".insteadOf "https://github.com/"
+          { echo "GIT_CONFIG_GLOBAL=$CFG"; echo "GIT_CONFIG_NOSYSTEM=1"; } >> "$GITHUB_ENV"
+          echo "runner git authenticated for private Go modules"
+
       - name: Test (per hanzo.yml)
+        if: inputs.mode != 'delegate'
         run: |
           set -euo pipefail
-          python3 -c "import yaml,json;print(json.dumps(yaml.safe_load(open('hanzo.yml')).get('test') or []))" | jq -c '.[]' | while read -r t; do
+          yq -o=json -I=0 '.test // []' hanzo.yml | jq -c '.[]' | while read -r t; do
             name=$(echo "$t"|jq -r .name); cmd=$(echo "$t"|jq -r .run)
             echo "::group::test $name"; bash -c "$cmd"; echo "::endgroup::"
           done
 
       - name: Deploy (per hanzo.yml)
-        if: github.event_name != 'pull_request' && steps.kms.outputs.kubeconfig != ''
+        if: inputs.mode != 'delegate' && github.event_name != 'pull_request' && steps.kms.outputs.kubeconfig != ''
         env:
           KUBECONFIG: ${{ steps.kms.outputs.kubeconfig }}
         run: |
           set -euo pipefail
           REF="${{ github.ref_name }}"
-          ON=$(python3 -c "import yaml,json;print(json.dumps((yaml.safe_load(open('hanzo.yml')).get('deploy') or {}).get('on') or []))")
+          ON=$(yq -o=json -I=0 '.deploy.on // []' hanzo.yml)
           IS_TAG=$([ "${{ github.ref_type }}" = "tag" ] && echo 1 || echo 0)
+          VER="${REF#v}"   # canonical semver for tag releases (ref_name without the v)
           if [ "$IS_TAG" != 1 ] && ! echo "$ON" | jq -e --arg b "$REF" 'index($b)' >/dev/null; then
             echo "branch $REF not in deploy.on — skipping deploy"; exit 0
           fi
+          # Bare arc runners ship no kubectl — provision the static binary
+          # (same sudo-free pattern as jq/yq above).
+          command -v kubectl >/dev/null 2>&1 || {
+            mkdir -p "$HOME/.local/bin"; export PATH="$HOME/.local/bin:$PATH"
+            KVER=$(curl -fsSL https://dl.k8s.io/release/stable.txt)
+            curl -fsSL "https://dl.k8s.io/release/${KVER}/bin/linux/amd64/kubectl" \
+              -o "$HOME/.local/bin/kubectl" && chmod +x "$HOME/.local/bin/kubectl"
+          }
           SHORT=$(echo "${{ github.sha }}" | cut -c1-7)
-          NS=$(python3 -c "import yaml;print(yaml.safe_load(open('hanzo.yml'))['deploy']['namespace'])")
-          python3 -c "import yaml,json;print(json.dumps(yaml.safe_load(open('hanzo.yml'))['images']))" > /tmp/imgs.json
-          python3 -c "import yaml,json;print(json.dumps(yaml.safe_load(open('hanzo.yml'))['deploy']['services']))" | jq -c '.[]' | while read -r s; do
+          NS=$(yq -r '.deploy.namespace' hanzo.yml)
+          RTO=$(yq -r '.deploy."rollout-timeout" // "600s"' hanzo.yml)
+          yq -o=json -I=0 '.images' hanzo.yml > /tmp/imgs.json
+          # Desired state lives in the universe repo: the in-cluster
+          # gitops-reconcile job re-applies its CRs every ~5 minutes, so any
+          # direct patch that is not ALSO recorded there is reverted on the
+          # next cycle. Record first (durable), then patch (accelerates the
+          # roll). Universe write rides the same KMS git token as module reads.
+          UNIVERSE=""
+          if [ -n "${GIT_TOKEN:-}" ]; then
+            git clone -q --depth 1 \
+              "https://x-access-token:${GIT_TOKEN}@github.com/hanzoai/universe.git" \
+              /tmp/universe 2>/dev/null && UNIVERSE=/tmp/universe \
+              || echo "::warning::universe clone failed — rolls below are transient until recorded there"
+          fi
+          yq -o=json -I=0 '.deploy.services' hanzo.yml | jq -c '.[]' | while read -r s; do
             svc=$(echo "$s"|jq -r .name); imgname=$(echo "$s"|jq -r .image)
             repo=$(jq -r --arg n "$imgname" '.[]|select(.name==$n)|.repo' /tmp/imgs.json)
             sfx=$(jq -r --arg n "$imgname" '.[]|select(.name==$n)|."tag-suffix" // ""' /tmp/imgs.json)
-            ref="$repo:sha-${SHORT}-amd64${sfx:+-$sfx}"
+            plats=$(jq -r --arg n "$imgname" '.[]|select(.name==$n)|(.platforms // ["linux/amd64"])|join(",")' /tmp/imgs.json)
+            # A TAGGED release pins the CANONICAL bare semver (VER = ref_name w/o the
+            # v) — the very image the build step published above; a branch build stays
+            # on its transient per-commit sha tag (arch-matched: bare for multi-arch).
+            if [ "$IS_TAG" = 1 ]; then
+              tag="${VER}${sfx:+-$sfx}"
+            elif [ "$plats" = "linux/amd64" ]; then
+              tag="sha-${SHORT}-amd64${sfx:+-$sfx}"
+            else
+              tag="sha-${SHORT}${sfx:+-$sfx}"
+            fi
+            ref="$repo:$tag"
+            CR="$UNIVERSE/infra/k8s/operator/crs/$svc.yaml"
+            # Backward-clobber guard (semver): a transient BRANCH build must NEVER
+            # overwrite a service pinned to a semver RELEASE (rolls prod back to a dev
+            # image). Complements the sha-ancestry guard below, which only sees sha-
+            # tags. Releases are tagged; a branch push leaves the semver pin untouched.
+            cur=""
+            if [ -n "$UNIVERSE" ] && [ -f "$CR" ]; then cur="$(yq -r '.spec.image.tag // ""' "$CR")"; fi
+            if [ "$IS_TAG" != 1 ] && printf '%s' "$cur" | grep -qE '^v?[0-9]+\.[0-9]+\.[0-9]+'; then
+              echo "::notice::$svc pinned to release $cur — branch build ${SHORT} leaves it (tag a release to deploy)"; continue
+            fi
             echo "rolling $svc → $ref"
-            kubectl -n "$NS" set image "deployment/$svc" "*=$ref"
-            kubectl -n "$NS" rollout status "deployment/$svc" --timeout=180s
+            # recorded=1 once the desired tag is durably in universe: the
+            # in-cluster operator reconciles that every ~5min, so an expired
+            # runner kubeconfig must not fail a deploy it will complete. Stays 0
+            # for bare-Deployment repos (no CR) where kubectl is the only path.
+            recorded=0
+            if [ -n "$UNIVERSE" ] && [ -f "$CR" ]; then
+              # Never roll a pin BACKWARD: builds finish out of order, and a slow
+              # build of an older commit must not overwrite a newer roll. If the
+              # CR's current sha is a descendant of ours, ours is stale — skip.
+              CURSHA=$(yq -r '.spec.image.tag // ""' "$CR" | sed -nE 's/^sha-([a-f0-9]{7}).*/\1/p')
+              git fetch -q --unshallow origin 2>/dev/null || true
+              if [ -n "$CURSHA" ] && [ "$CURSHA" != "$SHORT" ] \
+                 && git cat-file -e "$CURSHA" 2>/dev/null \
+                 && git merge-base --is-ancestor "$SHORT" "$CURSHA" 2>/dev/null; then
+                echo "::notice::$svc CR already at descendant $CURSHA — not rolling back to $SHORT"
+                recorded=1   # newer release already recorded + reconciling
+              else
+                yq -i ".spec.image.tag = \"$tag\"" "$CR"
+                # Sweep EVERY same-repo image reference in the CR (sidecars,
+                # initContainers pinned to this image) — the tag field alone
+                # left sidecars on stale tags every roll.
+                repoEsc=$(printf '%s' "$repo" | sed 's/[.[\*^$]/\\&/g')
+                sed -i -E "s|(${repoEsc}):sha-[A-Za-z0-9-]+|\1:${tag}|g" "$CR"
+                if git -C "$UNIVERSE" diff --quiet; then
+                  recorded=1   # CR already pins $tag (recorded on a prior run)
+                else
+                  commitCR() { git -C "$UNIVERSE" -c user.name=hanzo-ci -c user.email=dev@hanzo.ai \
+                    commit -qam "deploy($svc): $tag (${GITHUB_REPOSITORY}@${SHORT})"; }
+                  commitCR
+                  # EVERY service's deploy pushes to universe/main, so under
+                  # concurrent rolls a plain push loses the race with a
+                  # non-fast-forward reject (THE reason deploys stall when many
+                  # land at once). The clone is shallow (--depth 1) so `pull
+                  # --rebase` has no merge base and fails — instead fetch the moved
+                  # tip, hard-reset onto it, and re-apply our one-file CR change,
+                  # then retry the push. A no-op after reset (remote already
+                  # carries our tag) counts as recorded. We never force-push, so a
+                  # newer roll of THIS service is preserved (its tag survives the
+                  # reset; our re-apply is last-writer only when tags differ).
+                  for _try in 1 2 3 4 5 6; do
+                    if git -C "$UNIVERSE" push -q; then recorded=1; break; fi
+                    git -C "$UNIVERSE" fetch -q --depth 1 origin main || break
+                    git -C "$UNIVERSE" reset -q --hard FETCH_HEAD
+                    yq -i ".spec.image.tag = \"$tag\"" "$CR"
+                    sed -i -E "s|(${repoEsc}):sha-[A-Za-z0-9-]+|\1:${tag}|g" "$CR"
+                    if git -C "$UNIVERSE" diff --quiet; then recorded=1; break; fi
+                    commitCR
+                  done
+                  [ "$recorded" = 1 ] || echo "::warning::universe push failed for $svc after retries — roll is transient (runner kubectl must land it)"
+                fi
+              fi
+            fi
+            # Runner-side kubectl ACCELERATES the operator roll + smoke-tests it.
+            # When recorded=1 the operator owns the rollout, so an expired runner
+            # kubeconfig warns instead of failing the deploy; when recorded=0
+            # kubectl is the only path and stays fatal.
+            if [ "$recorded" = 1 ]; then
+              kc() { kubectl "$@" || echo "::warning::$svc: runner kubectl failed (expired kubeconfig?) — universe record stands; operator reconciles"; }
+            else
+              kc() { kubectl "$@"; }
+            fi
+            # Operator-managed services (Service CR) reconcile the Deployment —
+            # patch the CR when it exists; bare Deployments get set-image.
+            if kubectl -n "$NS" get "services.hanzo.ai/$svc" >/dev/null 2>&1; then
+              kc -n "$NS" patch "services.hanzo.ai/$svc" --type=merge \
+                -p "{\"spec\":{\"image\":{\"repository\":\"$repo\",\"tag\":\"$tag\"}}}"
+            else
+              # Bare Deployment: set the new image ONLY on containers already running
+              # THIS repo's image — never the '*' wildcard, which clobbers foreign
+              # sidecars (e.g. an rclone S3-mirror) with the app image and wedges the
+              # rollout. Fall back to the service-named container on first deploy.
+              mapfile -t CS < <(kubectl -n "$NS" get "deployment/$svc" \
+                -o jsonpath='{range .spec.template.spec.containers[*]}{.name} {.image}{"\n"}{end}' \
+                | awk -v r="$repo" 'index($2, r"@")==1 || index($2, r":")==1 {print $1}')
+              [ "${#CS[@]}" -eq 0 ] && CS=("$svc")
+              args=(); for c in "${CS[@]}"; do args+=("$c=$ref"); done
+              kc -n "$NS" set image "deployment/$svc" "${args[@]}"
+            fi
+            # Timeout scales with the image: a service vendoring GB-scale model/node
+            # packs (e.g. studio) pulls multi-GB layers on a cold node + waits for the
+            # operator to reconcile — legitimately minutes. 180s failed mid-pull and
+            # is THE reason studio deploys silently failed since 0.15.8. Override
+            # per-repo with deploy.rollout-timeout.
+            kc -n "$NS" rollout status "deployment/$svc" --timeout="$RTO"
           done

README.md

@@ -52,6 +52,33 @@ build minutes). To run on **your own** self-hosted arc runners, pass their label
     secrets: inherit
 ```
 
+## Delegate to platform (skip runner buildx)
+
+By default the build runs buildx **on** the arc runner. To instead hand the build
+to **platform.hanzo.ai** — which builds in-cluster with BuildKit and rolls the
+service itself — pass `mode: delegate`:
+
+```yaml
+    uses: hanzoai/ci/.github/workflows/build.yml@v1
+    with:
+      mode: delegate
+    secrets: inherit
+```
+
+The GitHub job then just POSTs each image in `hanzo.yml` to platform's direct
+build webhook (`/v1/arcd/enqueue`) and exits in **seconds** — no runner buildx,
+no KMS, no runner-side deploy. Platform creates the build job, launches an
+in-cluster BuildKit Job on its own pool, pushes to the registry, and patches the
+operator `Service` CR to roll it. It's the same build path as the platform
+GitHub-App webhook — one build path, two front doors.
+
+Requires one extra secret, `PLATFORM_BUILD_CALLBACK_TOKEN` (org- or repo-level,
+picked up via `secrets: inherit`). Override the endpoint with the
+`PLATFORM_ENQUEUE_URL` repo/org variable (default `https://platform.hanzo.ai/v1/arcd/enqueue`).
+
+`mode: buildx` (the default) is unchanged — existing repos keep running buildx on
+arc, so delegation is strictly opt-in.
+
 ## Credentials
 
 The only GitHub secrets a repo sets are `KMS_CLIENT_ID` / `KMS_CLIENT_SECRET`