binaries: publish to a bucket, and gate the bits once instead of twice

`binaries:` published only to a GitHub release, which is storage on a quota we
do not own — 400MiB per cloud release, hundreds of releases. A top-level
`bucket:` now sends the same artifacts and the same binaries.json to hanzoai/s3
instead. Nothing about the format changes; only the url the index carries.

Credentials are the S3_ADMIN_* names the services already read (clients/s3admin),
pulled from KMS at run time — not a CI-only copy nobody remembers to rotate.
A declared bucket with no credential fails the publish rather than shipping an
index whose artifacts are absent, and the lane proves the index is readable
UNAUTHENTICATED before calling it published: a host fetches it with no
credentials, so a private object is an index that resolves every app to a 403.
Granting that read is the bucket's job, one policy once, which is why this
checks rather than sets it. Artifacts upload first and the index last, so it
never names an object that is not there yet.

Also adds `tests:` (default true, so every existing caller is byte-identical).
false asserts the gate already ran on this exact commit — it does not mean ship
untested. hanzoai/cloud mints its v* tag only after that SHA passed the gate on
main and built and smoked (clients/platform/release.go), so a tag build re-tests
a proven commit and pays a 3108-package link storm for no new information. The
publish-after-gating invariant is unchanged: it is enforced once, not twice.

Verified end to end against a real hanzoai/s3: both platforms cross-compiled,
signed SigV4 PUTs, index last, anonymous GET 200, digest of the fetched bits
equal to the index. Four paths exercised — non-tag builds without publishing,
missing credential refuses, private bucket refuses naming the remedy, public
bucket publishes.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>

.hanzo/workflows/build.yml

@@ -28,9 +28,21 @@ on:
           inherit).
         type: string
         default: buildx
+      tests:
+        description: >-
+          Run the `test:` block. Default true, and a caller should leave it that
+          way. false asserts that THIS EXACT COMMIT was already gated before this
+          run — it does not mean "ship untested". The one shape that holds today
+          is hanzoai/cloud's release: clients/platform/release.go mints the v* tag
+          only after that SHA passed the gate on main AND built AND smoked, so the
+          tag build would re-test a commit already proven, at the cost of a
+          3108-package link storm. Passing false there runs the gate once instead
+          of twice; passing it anywhere else runs it zero times.
+        type: boolean
+        default: true
 
 permissions:
-  contents: read
+  contents: write # release assets for `binaries:`; read is enough without it
   packages: write
 
 jobs:
@@ -156,14 +168,23 @@ jobs:
             && echo "git auth OK" \
             || echo "::warning::GH_PAT set but repo probe failed"
 
-      - name: Log in to GHCR (automatic workflow token)
+      - name: Log in to GHCR (GH_PAT when present, else automatic token)
         if: inputs.mode != 'delegate'
-        # GHCR push needs no stored credential: GitHub injects a per-job
-        # GITHUB_TOKEN scoped to THIS repo, and `permissions: packages: write`
-        # (above) lets it push the repo's own package (e.g. zooai/node →
-        # ghcr.io/zooai/node, same org). First push creates the package
-        # repo-linked. No KMS, no PAT for the push path.
-        run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
+        env:
+          GH_PAT: ${{ secrets.GH_PAT }}
+        # Prefer the KMS-backed GH_PAT (admin:org + write:packages): it can push
+        # or CREATE any <org> package regardless of which repo the package is
+        # linked to. The per-job GITHUB_TOKEN only writes a package linked to THIS
+        # repo, so it 403s on a package created/linked elsewhere (e.g.
+        # ghcr.io/hanzoai/cms). Fall back to the automatic token when GH_PAT is
+        # absent (public forks like zooai/node that create their own repo-linked
+        # package on first push).
+        run: |
+          if [ -n "${GH_PAT:-}" ]; then
+            echo "$GH_PAT" | docker login ghcr.io -u hanzo-dev --password-stdin
+          else
+            echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
+          fi
 
       - name: Fetch deploy credentials from KMS
         id: kms
@@ -228,8 +249,70 @@ jobs:
           if [ -n "$GIT_TOKEN" ]; then echo "::add-mask::$GIT_TOKEN"; echo "GIT_TOKEN=$GIT_TOKEN" >> "$GITHUB_ENV"; fi
           KUBECONFIG_B64=$(get KUBECONFIG)
           if [ -n "$KUBECONFIG_B64" ]; then echo "$KUBECONFIG_B64" | base64 -d > "$RUNNER_TEMP/kubeconfig"; echo "kubeconfig=$RUNNER_TEMP/kubeconfig" >> "$GITHUB_OUTPUT"; fi
+          # Object-store credential for a repo that publishes `binaries:` to a
+          # bucket:. Same four names the services read at run time, from the same
+          # org/path/env — a CI-only copy of a credential is a second thing to
+          # rotate and the one nobody remembers.
+          if [ -n "$(yq -r '.bucket // ""' hanzo.yml 2>/dev/null || true)" ]; then
+            for k in S3_ADMIN_ACCESS_KEY S3_ADMIN_SECRET_KEY; do
+              v=$(get "$k")
+              if [ -n "$v" ]; then echo "::add-mask::$v"; echo "$k=$v" >> "$GITHUB_ENV"; fi
+            done
+            # Endpoint and region are NOT masked: both appear in every published
+            # URL, and a redacted host turns the printed index into noise.
+            for k in S3_PUBLIC_ENDPOINT S3_REGION S3_PUBLIC_SECURE; do
+              v=$(get "$k")
+              if [ -n "$v" ]; then echo "$k=$v" >> "$GITHUB_ENV"; fi
+            done
+          fi
+          # Build-time secrets → --build-arg. A repo declares per-image
+          # `build_secrets: [NAME, ...]` in hanzo.yml; each NAME is fetched from
+          # the SAME org/path/env and exported (masked) so the build step bakes
+          # it in as `--build-arg NAME=value`. The KMS key name IS the build-arg
+          # name — one name, one place. For publishable client tokens a Vite SPA
+          # must embed at build (e.g. VITE_MAPBOX_TOKEN). Undeclared → no-op, so
+          # every existing repo is byte-for-byte unchanged.
+          for bs in $(yq -r '[(.images // [])[] | (.build_secrets // [])[]] | unique | .[]' hanzo.yml 2>/dev/null || true); do
+            case "$bs" in ''|*[!A-Za-z0-9_]*) echo "::warning::skipping invalid build_secret name '$bs'"; continue;; esac
+            v=$(get "$bs" || true)
+            if [ -n "$v" ]; then
+              echo "::add-mask::$v"
+              { echo "$bs<<__KMS_BUILDARG_EOF__"; echo "$v"; echo "__KMS_BUILDARG_EOF__"; } >> "$GITHUB_ENV"
+            else echo "::warning::build_secret $bs not in KMS ($ORG/$PATHQ env=$ENV) — build-arg will be empty"; fi
+          done
 
-      - name: Mirror credential (registry.hanzo.ai)
+      - name: GHCR push credential (KMS write:packages token)
+        if: inputs.mode != 'delegate'
+        env:
+          KUBECONFIG: ${{ steps.kms.outputs.kubeconfig }}
+        # The per-job GITHUB_TOKEN — and a package-less GH_PAT — can only push a
+        # package LINKED to this repo, so they 403 on a package created/linked
+        # elsewhere (ghcr.io/hanzoai/cms). Upgrade the ghcr login to the org's
+        # ghcr push token from the cluster (buildx-ghcr-auth, admin write:packages,
+        # KMS-synced) when a kubeconfig is present — it pushes/creates ANY <org>
+        # package. Fail-safe: keep the earlier login if anything is unavailable
+        # (public forks with no KMS keep their repo-linked GITHUB_TOKEN push).
+        run: |
+          set -uo pipefail
+          [ -z "${KUBECONFIG:-}" ] && { echo "::notice::no kubeconfig — keeping the earlier GHCR login"; exit 0; }
+          command -v kubectl >/dev/null 2>&1 || {
+            KVER=$(curl -fsSL https://dl.k8s.io/release/stable.txt)
+            mkdir -p "$HOME/.local/bin"
+            curl -fsSL "https://dl.k8s.io/release/${KVER}/bin/linux/amd64/kubectl" -o "$HOME/.local/bin/kubectl" && chmod +x "$HOME/.local/bin/kubectl"
+            export PATH="$HOME/.local/bin:$PATH"
+          }
+          CFG=$(kubectl -n hanzo get secret buildx-ghcr-auth -o jsonpath='{.data.\.dockerconfigjson}' 2>/dev/null | base64 -d || true)
+          [ -z "$CFG" ] && { echo "::notice::buildx-ghcr-auth not readable — keeping the earlier GHCR login"; exit 0; }
+          UP=$(echo "$CFG" | jq -r '.auths | to_entries[] | select(.key|test("ghcr")) | .value.auth' | head -1 | base64 -d 2>/dev/null || true)
+          [ -z "$UP" ] && { echo "::notice::no ghcr auth in buildx-ghcr-auth — keeping the earlier login"; exit 0; }
+          echo "::add-mask::${UP#*:}"
+          if echo "${UP#*:}" | docker login ghcr.io -u "${UP%%:*}" --password-stdin; then
+            echo "::notice::GHCR login upgraded to the KMS write:packages token"
+          else
+            echo "::notice::KMS GHCR token login failed — keeping the earlier login"
+          fi
+
+      - name: Native registry credential (oci.hanzo.ai)
         if: inputs.mode != 'delegate'
         env:
           KUBECONFIG: ${{ steps.kms.outputs.kubeconfig }}
@@ -241,10 +324,18 @@ jobs:
           # where the Free plan hides org secrets... including these; private
           # repos set them at REPO level). KMS-kubeconfig read is the fallback.
           if [ -n "${REGISTRY_USER:-}" ] && [ -n "${REGISTRY_PASSWORD:-}" ]; then
-            echo "$REGISTRY_PASSWORD" | docker login registry.hanzo.ai -u "$REGISTRY_USER" --password-stdin
-            echo "MIRROR_OK=1" >> "$GITHUB_ENV"; exit 0
+            # Best-effort: a registry.hanzo.ai login FAILURE (not just a missing
+            # cred) must never fail the run — the image still pushes to GHCR, the
+            # primary. Without this guard, bash -e aborts the step and SKIPS the
+            # build entirely (a registry hiccup takes the whole lane red).
+            if echo "$REGISTRY_PASSWORD" | docker login oci.hanzo.ai -u "$REGISTRY_USER" --password-stdin; then
+              echo "MIRROR_OK=1" >> "$GITHUB_ENV"
+            else
+              echo "::notice::registry.hanzo.ai login failed — mirror skipped (GHCR-only push)"
+            fi
+            exit 0
           fi
-          [ -z "${KUBECONFIG:-}" ] && { echo "::notice::no registry secret and no kubeconfig — mirror skipped"; exit 0; }
+          [ -z "${KUBECONFIG:-}" ] && { echo "::warning::no registry credential and no kubeconfig — image will NOT reach oci.hanzo.ai (ghcr only)"; exit 0; }
           # Bare arc runners ship no kubectl — same static provision the deploy
           # step uses.
           command -v kubectl >/dev/null 2>&1 || {
@@ -255,13 +346,17 @@ jobs:
           }
           CFG=$(kubectl -n hanzo get secret registry-credentials                   -o jsonpath='{.data.\.dockerconfigjson}' 2>/dev/null | base64 -d || true)
           if [ -z "$CFG" ]; then
-            echo "::notice::registry-credentials not readable from this kubeconfig — mirror skipped (GHCR-only push)"; exit 0
+            echo "::warning::registry-credentials not readable from this kubeconfig — image will NOT reach oci.hanzo.ai (ghcr only)"; exit 0
           fi
           USERPASS=$(echo "$CFG" | jq -r '.auths["registry.hanzo.ai"].auth // empty' | base64 -d)
-          [ -z "$USERPASS" ] && { echo "::notice::no registry.hanzo.ai auth in dockerconfig — mirror skipped"; exit 0; }
+          [ -z "$USERPASS" ] && { echo "::warning::no registry auth in dockerconfig — image will NOT reach oci.hanzo.ai (ghcr only)"; exit 0; }
           echo "::add-mask::${USERPASS#*:}"
-          echo "${USERPASS#*:}" | docker login registry.hanzo.ai -u "${USERPASS%%:*}" --password-stdin
-          echo "MIRROR_OK=1" >> "$GITHUB_ENV"
+          # Best-effort: login failure → skip mirror, never fail the run (see above).
+          if echo "${USERPASS#*:}" | docker login oci.hanzo.ai -u "${USERPASS%%:*}" --password-stdin; then
+            echo "MIRROR_OK=1" >> "$GITHUB_ENV"
+          else
+            echo "::notice::registry.hanzo.ai login failed — mirror skipped (GHCR-only push)"
+          fi
 
       - name: Build & push images (per hanzo.yml)
         if: inputs.mode != 'delegate'
@@ -286,7 +381,8 @@ jobs:
           export GIT_TOKEN="${GIT_TOKEN:-${GH_PAT:-}}"
           SHORT=$(echo "${{ github.sha }}" | cut -c1-7)
           IS_TAG=$([ "${{ github.ref_type }}" = "tag" ] && echo 1 || echo 0)
-          VER="${{ github.ref_name }}"; VER="${VER#v}"
+          REL="${{ github.ref_name }}"   # the git tag, verbatim: v1.26.19
+          VER="${REL#v}"                 # v-stripped alias: 1.26.19
           yq -o=json -I=0 '.images' hanzo.yml | jq -c '.[]' | while read -r img; do
             name=$(echo "$img"|jq -r .name); ctx=$(echo "$img"|jq -r .context)
             df=$(echo "$img"|jq -r '.dockerfile // (.context+"/Dockerfile")'); repo=$(echo "$img"|jq -r .repo)
@@ -307,25 +403,44 @@ jobs:
             if [ "$plats" = "linux/amd64" ]; then
               # single-arch: keep the exact legacy tag shape (-amd64) deploys expect.
               TAGS="-t $repo:sha-${SHORT}-amd64${sfx:+-$sfx} -t $repo:${sfx:+$sfx-}latest"
-              # Release (tag) build: publish the CANONICAL bare semver tag (what the
-              # universe CR pins to — matches world 2.4.10 / cloud v1.801.62) AND the
-              # legacy -amd64 semver alias (back-compat for CRs still on that shape).
-              [ "$IS_TAG" = 1 ] && TAGS="$TAGS -t $repo:${VER}${sfx:+-$sfx} -t $repo:${VER}-amd64${sfx:+-$sfx}"
             else
               # multi-arch: one arch-neutral manifest-list tag (no -amd64 suffix).
               docker run --privileged --rm tonistiigi/binfmt --install arm64 >/dev/null 2>&1 || true
               TAGS="-t $repo:sha-${SHORT}${sfx:+-$sfx} -t $repo:${sfx:+$sfx-}latest"
-              [ "$IS_TAG" = 1 ] && TAGS="$TAGS -t $repo:${VER}${sfx:+-$sfx}"
+            fi
+            # Release (tag) build. The git tag IS the release name, so publish it
+            # VERBATIM (v1.26.19) — that is the shape a universe CR pins, and
+            # stripping the v is why releases were finished by hand-`crane copy`ing
+            # sha-<sha7> onto the semver a human typed. Identity in, identity out.
+            # The v-stripped alias stays for CRs already pinned that way (world
+            # 2.4.51), and is skipped when a repo tags without a v. The old
+            # `<ver>-amd64` alias is deleted: no CR in the fleet pinned it.
+            if [ "$IS_TAG" = 1 ]; then
+              TAGS="$TAGS -t $repo:${REL}${sfx:+-$sfx}"
+              [ "$REL" != "$VER" ] && TAGS="$TAGS -t $repo:${VER}${sfx:+-$sfx}"
             fi
             echo "::group::build $name → $repo (${sfx}) [$plats]"
+            # --build-arg assembly: static hanzo.yml `args` (a fixed value, e.g. a
+            # pinned base image tag) + `build_secrets` (KMS values the KMS step
+            # exported into the env above). Empty when a repo declares neither, so
+            # the buildx line is unchanged for every existing repo.
+            BUILD_ARGS=""
+            while IFS= read -r kv; do [ -n "$kv" ] && BUILD_ARGS="$BUILD_ARGS --build-arg $kv"; done \
+              < <(echo "$img" | jq -r '(.args // {}) | to_entries[] | "\(.key)=\(.value)"')
+            for bs in $(echo "$img" | jq -r '(.build_secrets // [])[]'); do
+              v=$(printenv "$bs" 2>/dev/null || true); [ -n "$v" ] && BUILD_ARGS="$BUILD_ARGS --build-arg $bs=$v"
+            done
             # GIT_TOKEN (from KMS, via GITHUB_ENV) is passed as the `gh_token`
             # BuildKit secret so Dockerfiles can clone private Go modules; omitted
             # cleanly when absent (public-only builds unaffected).
-            docker buildx build --platform "$plats" ${GIT_TOKEN:+--secret id=gh_token,env=GIT_TOKEN} --push $TAGS -f "$df" "$ctx"
+            docker buildx build --platform "$plats" $BUILD_ARGS ${GIT_TOKEN:+--secret id=gh_token,env=GIT_TOKEN} --push $TAGS -f "$df" "$ctx"
             # Dual-host: mirror the exact tag set to registry.hanzo.ai (server-
             # side manifest copy — no rebuild). ghcr.io/<org>/<name> →
-            # registry.hanzo.ai/<org>/<name>; public consumers keep ghcr, the
-            # fleet DEPLOYS from our registry.
+            # oci.hanzo.ai/<org>/<name>; public consumers keep ghcr, the fleet
+            # is migrating to pull from ours. A skip here is now a WARNING, not
+            # a notice: an image that never reaches our registry is the reason
+            # a deploy still depends on GitHub, and that should be visible in
+            # the run, not buried.
             if [ "${MIRROR_OK:-}" = "1" ]; then
               # crane, not buildx imagetools: the IAM token realm doesn't answer
               # buildx's multi-scope token request (spec gap, tracked).
@@ -335,10 +450,10 @@ jobs:
                   | tar -xz -C "$HOME/.local/bin" crane
                 export PATH="$HOME/.local/bin:$PATH"
               }
-              mrepo="registry.hanzo.ai/${repo#*/}"
+              mrepo="oci.hanzo.ai/${repo#*/}"
               echo "$TAGS" | tr ' ' '\n' | grep -v '^-t$' | grep -v '^$' | while read -r ref; do
                 crane copy "$ref" "${mrepo}:${ref##*:}" \
-                  || echo "::warning::mirror of $ref failed (ghcr push unaffected)"
+                  || echo "::warning::$ref did not reach oci.hanzo.ai (ghcr push unaffected)"
               done
             fi
             echo "::endgroup::"
@@ -415,7 +530,7 @@ jobs:
           echo "runner git authenticated for private Go modules"
 
       - name: Test (per hanzo.yml)
-        if: inputs.mode != 'delegate'
+        if: inputs.mode != 'delegate' && inputs.tests
         run: |
           set -euo pipefail
           yq -o=json -I=0 '.test // []' hanzo.yml | jq -c '.[]' | while read -r t; do
@@ -423,6 +538,188 @@ jobs:
             echo "::group::test $name"; bash -c "$cmd"; echo "::endgroup::"
           done
 
+      - name: Build & publish binaries (per hanzo.yml)
+        # The plugin lane. `images:` ships an OCI image a CLUSTER runs;
+        # `binaries:` ships an EXECUTABLE a RUNNING HOST installs — a zip plugin
+        # (`zip.Load(zip.Plugin{URL, Sum})`), which fetches the artifact,
+        # verifies its SHA-256 BEFORE making it executable, and caches it by
+        # digest. So a plugin is built ONCE per OS/arch and every host picks up
+        # the same bits: nobody rebuilds the world to ship a plugin, and no host
+        # is trusted to have built it right.
+        #
+        # It rides THIS pipeline, not a second one — same hanzo.yml, same job,
+        # same runner, same KMS token — and adds two rules of its own:
+        #   - BUILT on every push, PUBLISHED only on a tag. A cross-compile that
+        #     breaks arm64 fails the PR that broke it, not the release.
+        #   - published AFTER the bits are GATED, because a host installs an
+        #     artifact unattended. Normally that gate is the `test:` step above,
+        #     in this same run. A caller passing `tests: false` asserts it ran
+        #     EARLIER on this exact commit: hanzoai/cloud mints its v* tag only
+        #     after that SHA passed the gate on main and built and smoked
+        #     (clients/platform/release.go), so the tag build would re-test a
+        #     commit already proven. The invariant did not weaken — it is
+        #     enforced once instead of twice, and the assertion is the input.
+        #     (Images push before tests; an image is rolled out by a deliberate,
+        #     reviewed pin — an artifact is not.)
+        # dist/binaries.json is uploaded with them: url + sha256 for every
+        # artifact, so the bits and the digest that authorizes them are the same
+        # release, and a host reads url+sum from ONE place.
+        if: inputs.mode != 'delegate'
+        env:
+          GH_PAT: ${{ secrets.GH_PAT }}
+          GH_AUTOMATIC: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ "$(yq -r '.binaries // [] | length' hanzo.yml 2>/dev/null || echo 0)" = "0" ]; then
+            echo "::notice::no binaries: in hanzo.yml — nothing to publish"; exit 0
+          fi
+          TAG="${{ github.ref_name }}"; REPO="${{ github.repository }}"
+          IS_TAG=$([ "${{ github.ref_type }}" = "tag" ] && echo 1 || echo 0)
+          # The runner tells us which forge it is talking to — github.com on the
+          # arc pool, a GHES/forge front door elsewhere. Reading it instead of
+          # hardcoding the hostname is what keeps ONE lane serving both.
+          SERVER="${GITHUB_SERVER_URL:-https://github.com}"
+          API="${GITHUB_API_URL:-https://api.github.com}"
+          case "$API" in
+            https://api.github.com) UPLOADS=https://uploads.github.com ;;
+            *) UPLOADS="$SERVER/api/uploads" ;;
+          esac
+          # A repo naming a bucket: publishes to hanzoai/s3 instead of a release.
+          # Same artifacts, same index, one url per artifact — only where it
+          # points changes, and it is decided HERE because the url is baked into
+          # binaries.json at build time.
+          BUCKET="$(yq -r '.bucket // ""' hanzo.yml 2>/dev/null || true)"
+          if [ -n "$BUCKET" ]; then
+            # Host and scheme are the two clients/s3admin reads, so CI publishes
+            # to the same URL the services resolve.
+            SCHEME=https
+            if [ "${S3_PUBLIC_SECURE:-true}" = "false" ]; then SCHEME=http; fi
+            BASE="$SCHEME://${S3_PUBLIC_ENDPOINT:-s3.hanzo.ai}/$BUCKET/$REPO/$TAG"
+          else
+            BASE="$SERVER/$REPO/releases/download/$TAG"
+          fi
+          rm -rf dist; mkdir -p dist
+          # CGO_ENABLED=0 is not a preference: the host that installs this runs
+          # it on WHATEVER base image the host happens to be, so a binary linked
+          # against that runner's glibc is a plugin that starts on the runner and
+          # nowhere else. -trimpath keeps the digest a function of the source,
+          # not of the checkout path.
+          yq -o=json -I=0 '.binaries' hanzo.yml | jq -c '.[]' | while read -r b; do
+            name=$(echo "$b"|jq -r .name); main=$(echo "$b"|jq -r '.main // ""')
+            lf=$(echo "$b"|jq -r '.ldflags // "-s -w"')
+            run=$(echo "$b"|jq -r '.run // ""'); out=$(echo "$b"|jq -r '.out // ""')
+            # `run:` + `out:` is the SAME lane for every toolchain that is not Go:
+            # the command that builds, and the glob of what it produced. It is why
+            # a repo with no Dockerfile and no Go can still publish an artifact —
+            # an npm tarball, a wheel, a Rust binary — through this one block.
+            # (`image:` also appears on such an entry: it names the toolchain
+            # container the PLATFORM lane runs the command in. Here the toolchain
+            # is the runner, so this lane reads past it.)
+            if [ -n "$run" ]; then
+              [ -n "$out" ] || { echo "::error::binaries[$name] declares run: without out: — nothing names what it produced"; exit 1; }
+              echo "::group::build $name ($run)"
+              sh -c "$run"
+              n=0
+              for f in $out; do
+                [ -f "$f" ] || continue
+                base=$(basename "$f"); cp "$f" "dist/$base"
+                sha=$(sha256sum "dist/$base" | cut -d' ' -f1)
+                # os/arch are "any": an npm tarball or a wheel is not per-platform,
+                # and an index entry that claimed one would be a lie a host acts on.
+                jq -nc --arg n "$name" --arg os any --arg arch any \
+                       --arg url "$BASE/$base" --arg sha "$sha" \
+                  '{name:$n,os:$os,arch:$arch,url:$url,sha256:$sha}' >> dist/index.jsonl
+                echo "$base  $sha"; n=$((n+1))
+              done
+              echo "::endgroup::"
+              [ "$n" -gt 0 ] || { echo "::error::binaries[$name] out: '$out' matched no file the recipe produced"; exit 1; }
+              continue
+            fi
+            main="${main:-.}"
+            echo "$b" | jq -r '(.platforms // ["linux/amd64"])[]' | while read -r plat; do
+              os="${plat%%/*}"; arch="${plat##*/}"
+              file="${name}-${os}-${arch}"; [ "$os" = windows ] && file="${file}.exe"
+              echo "::group::build $name $os/$arch"
+              CGO_ENABLED=0 GOOS="$os" GOARCH="$arch" \
+                go build -trimpath -ldflags "$lf" -o "dist/$file" "$main"
+              sha=$(sha256sum "dist/$file" | cut -d' ' -f1)
+              jq -nc --arg n "$name" --arg os "$os" --arg arch "$arch" \
+                     --arg url "$BASE/$file" --arg sha "$sha" \
+                '{name:$n,os:$os,arch:$arch,url:$url,sha256:$sha}' >> dist/index.jsonl
+              echo "$file  $sha"
+              echo "::endgroup::"
+            done
+          done
+          jq -s --arg repo "$REPO" --arg tag "$TAG" \
+            '{repo:$repo,tag:$tag,binaries:.}' dist/index.jsonl > dist/binaries.json
+          rm dist/index.jsonl
+          if [ "$IS_TAG" != 1 ]; then
+            echo "::notice::binaries built for $(jq '.binaries|length' dist/binaries.json) platform(s) — publishing happens on a tag"; exit 0
+          fi
+          if [ -n "$BUCKET" ]; then
+            # Fail closed. An index published without its artifacts is a host
+            # that resolves every app to a 404, which is worse than a host with
+            # no index at all.
+            if [ -z "${S3_ADMIN_ACCESS_KEY:-}" ] || [ -z "${S3_ADMIN_SECRET_KEY:-}" ]; then
+              echo "::error::hanzo.yml names bucket: $BUCKET but KMS holds no S3_ADMIN_ACCESS_KEY/SECRET_KEY for this org — refusing to publish"; exit 1
+            fi
+            # PUT is idempotent, so re-running a release converges on the object
+            # rather than colliding with the one already there.
+            put() { curl -fsS -X PUT --aws-sigv4 "aws:amz:${S3_REGION:-us-east-1}:s3" \
+              -u "$S3_ADMIN_ACCESS_KEY:$S3_ADMIN_SECRET_KEY" \
+              -H 'Content-Type: application/octet-stream' \
+              --data-binary @"$1" "$BASE/$(basename "$1")" >/dev/null
+              echo "published $BASE/$(basename "$1")"; }
+            # Artifacts first, index LAST. The index is the only file a host
+            # reads, so between the two writes it must never name an object that
+            # is not there yet.
+            for f in dist/*; do
+              [ "$f" = dist/binaries.json ] || put "$f"
+            done
+            put dist/binaries.json
+            # A host reads the index with NO credentials, so prove one can
+            # before calling this published — a private object is an index that
+            # resolves every app to a 403. Granting read is the BUCKET's job
+            # (one policy, once), which is why this checks rather than sets it.
+            code=$(curl -s -o /dev/null -w '%{http_code}' "$BASE/binaries.json")
+            if [ "$code" != 200 ]; then
+              echo "::error::published, but $BASE/binaries.json answers $code unauthenticated — grant s3:GetObject on $BUCKET/* to Principal \"*\" with a bucket policy"; exit 1
+            fi
+            echo "### Plugin artifacts — \`$TAG\`" >> "$GITHUB_STEP_SUMMARY"
+            echo "Index: \`$BASE/binaries.json\`" >> "$GITHUB_STEP_SUMMARY"
+            exit 0
+          fi
+          # Same token ladder the rest of this file uses: the KMS org token, the
+          # org PAT, then the per-job automatic one (which can write a release on
+          # its OWN repo, so a repo with no KMS still publishes).
+          TOKEN="${GIT_TOKEN:-${GH_PAT:-$GH_AUTOMATIC}}"
+          api() { curl -fsS -H "Authorization: Bearer $TOKEN" -H 'Accept: application/vnd.github+json' "$@"; }
+          rel=$(api "$API/repos/$REPO/releases/tags/$TAG" 2>/dev/null | jq -r '.id // empty' || true)
+          if [ -z "$rel" ]; then
+            rel=$(api -X POST "$API/repos/$REPO/releases" \
+              -d "$(jq -nc --arg t "$TAG" '{tag_name:$t,name:$t,generate_release_notes:true}')" | jq -r .id)
+          fi
+          for f in dist/*; do
+            n=$(basename "$f")
+            # Re-running a release must converge, not 422: drop a same-named
+            # asset first. The digest in binaries.json is regenerated with it,
+            # so the pair can never disagree.
+            old=$(api "$API/repos/$REPO/releases/$rel/assets" | jq -r --arg n "$n" '.[]?|select(.name==$n)|.id')
+            for id in $old; do api -X DELETE "$API/repos/$REPO/releases/assets/$id" >/dev/null; done
+            api -X POST -H 'Content-Type: application/octet-stream' --data-binary @"$f" \
+              "$UPLOADS/repos/$REPO/releases/$rel/assets?name=$n" >/dev/null
+            echo "published $BASE/$n"
+          done
+          # The job prints what a host pastes. A digest a human retypes is a
+          # digest a human gets wrong.
+          {
+            echo "### Plugin artifacts — \`$TAG\`"
+            echo '```go'
+            jq -r '.binaries[]|"zip.Load(zip.Plugin{\n    Name: \"\(.name)\",   // \(.os)/\(.arch)\n    URL:  \"\(.url)\",\n    Sum:  \"\(.sha256)\",\n}, \"/v1/\(.name)\")"' dist/binaries.json
+            echo '```'
+            echo "Index: \`$BASE/binaries.json\`"
+          } >> "$GITHUB_STEP_SUMMARY"
+
       - name: Deploy (per hanzo.yml)
         if: inputs.mode != 'delegate' && github.event_name != 'pull_request' && steps.kms.outputs.kubeconfig != ''
         env:
@@ -430,9 +727,17 @@ jobs:
         run: |
           set -euo pipefail
           REF="${{ github.ref_name }}"
+          # A caller with no `deploy:` declares its rollout elsewhere (hanzoai/git
+          # pins its own CR by a reviewed universe change). Leave before reading
+          # deploy.services, which is null there — `jq '.[]'` over null aborts the
+          # step, and on a TAG build the deploy.on gate below is bypassed, so this
+          # used to surface only on a release. Same guard the build step has for
+          # `images:`.
+          if [ "$(yq -r '.deploy | type' hanzo.yml 2>/dev/null || echo '!!null')" != '!!map' ]; then
+            echo "::notice::no deploy: in hanzo.yml — build-only caller, skipping deploy"; exit 0
+          fi
           ON=$(yq -o=json -I=0 '.deploy.on // []' hanzo.yml)
           IS_TAG=$([ "${{ github.ref_type }}" = "tag" ] && echo 1 || echo 0)
-          VER="${REF#v}"   # canonical semver for tag releases (ref_name without the v)
           if [ "$IS_TAG" != 1 ] && ! echo "$ON" | jq -e --arg b "$REF" 'index($b)' >/dev/null; then
             echo "branch $REF not in deploy.on — skipping deploy"; exit 0
           fi
@@ -465,11 +770,11 @@ jobs:
             repo=$(jq -r --arg n "$imgname" '.[]|select(.name==$n)|.repo' /tmp/imgs.json)
             sfx=$(jq -r --arg n "$imgname" '.[]|select(.name==$n)|."tag-suffix" // ""' /tmp/imgs.json)
             plats=$(jq -r --arg n "$imgname" '.[]|select(.name==$n)|(.platforms // ["linux/amd64"])|join(",")' /tmp/imgs.json)
-            # A TAGGED release pins the CANONICAL bare semver (VER = ref_name w/o the
-            # v) — the very image the build step published above; a branch build stays
-            # on its transient per-commit sha tag (arch-matched: bare for multi-arch).
+            # A TAGGED release pins the git tag VERBATIM (REF) — the very image the
+            # build step published above; a branch build stays on its transient
+            # per-commit sha tag (arch-matched: bare for multi-arch).
             if [ "$IS_TAG" = 1 ]; then
-              tag="${VER}${sfx:+-$sfx}"
+              tag="${REF}${sfx:+-$sfx}"
             elif [ "$plats" = "linux/amd64" ]; then
               tag="sha-${SHORT}-amd64${sfx:+-$sfx}"
             else
@@ -487,6 +792,11 @@ jobs:
               echo "::notice::$svc pinned to release $cur — branch build ${SHORT} leaves it (tag a release to deploy)"; continue
             fi
             echo "rolling $svc → $ref"
+            # recorded=1 once the desired tag is durably in universe: the
+            # in-cluster operator reconciles that every ~5min, so an expired
+            # runner kubeconfig must not fail a deploy it will complete. Stays 0
+            # for bare-Deployment repos (no CR) where kubectl is the only path.
+            recorded=0
             if [ -n "$UNIVERSE" ] && [ -f "$CR" ]; then
               # Never roll a pin BACKWARD: builds finish out of order, and a slow
               # build of an older commit must not overwrite a newer roll. If the
@@ -497,6 +807,7 @@ jobs:
                  && git cat-file -e "$CURSHA" 2>/dev/null \
                  && git merge-base --is-ancestor "$SHORT" "$CURSHA" 2>/dev/null; then
                 echo "::notice::$svc CR already at descendant $CURSHA — not rolling back to $SHORT"
+                recorded=1   # newer release already recorded + reconciling
               else
                 yq -i ".spec.image.tag = \"$tag\"" "$CR"
                 # Sweep EVERY same-repo image reference in the CR (sidecars,
@@ -504,18 +815,48 @@ jobs:
                 # left sidecars on stale tags every roll.
                 repoEsc=$(printf '%s' "$repo" | sed 's/[.[\*^$]/\\&/g')
                 sed -i -E "s|(${repoEsc}):sha-[A-Za-z0-9-]+|\1:${tag}|g" "$CR"
-                git -C "$UNIVERSE" diff --quiet || {
-                  git -C "$UNIVERSE" -c user.name=hanzo-ci -c user.email=dev@hanzo.ai \
-                    commit -qam "deploy($svc): $tag (${GITHUB_REPOSITORY}@${SHORT})"
-                  git -C "$UNIVERSE" push -q \
-                    || echo "::warning::universe push failed for $svc — roll is transient"
-                }
+                if git -C "$UNIVERSE" diff --quiet; then
+                  recorded=1   # CR already pins $tag (recorded on a prior run)
+                else
+                  commitCR() { git -C "$UNIVERSE" -c user.name=hanzo-ci -c user.email=dev@hanzo.ai \
+                    commit -qam "deploy($svc): $tag (${GITHUB_REPOSITORY}@${SHORT})"; }
+                  commitCR
+                  # EVERY service's deploy pushes to universe/main, so under
+                  # concurrent rolls a plain push loses the race with a
+                  # non-fast-forward reject (THE reason deploys stall when many
+                  # land at once). The clone is shallow (--depth 1) so `pull
+                  # --rebase` has no merge base and fails — instead fetch the moved
+                  # tip, hard-reset onto it, and re-apply our one-file CR change,
+                  # then retry the push. A no-op after reset (remote already
+                  # carries our tag) counts as recorded. We never force-push, so a
+                  # newer roll of THIS service is preserved (its tag survives the
+                  # reset; our re-apply is last-writer only when tags differ).
+                  for _try in 1 2 3 4 5 6; do
+                    if git -C "$UNIVERSE" push -q; then recorded=1; break; fi
+                    git -C "$UNIVERSE" fetch -q --depth 1 origin main || break
+                    git -C "$UNIVERSE" reset -q --hard FETCH_HEAD
+                    yq -i ".spec.image.tag = \"$tag\"" "$CR"
+                    sed -i -E "s|(${repoEsc}):sha-[A-Za-z0-9-]+|\1:${tag}|g" "$CR"
+                    if git -C "$UNIVERSE" diff --quiet; then recorded=1; break; fi
+                    commitCR
+                  done
+                  [ "$recorded" = 1 ] || echo "::warning::universe push failed for $svc after retries — roll is transient (runner kubectl must land it)"
+                fi
               fi
             fi
+            # Runner-side kubectl ACCELERATES the operator roll + smoke-tests it.
+            # When recorded=1 the operator owns the rollout, so an expired runner
+            # kubeconfig warns instead of failing the deploy; when recorded=0
+            # kubectl is the only path and stays fatal.
+            if [ "$recorded" = 1 ]; then
+              kc() { kubectl "$@" || echo "::warning::$svc: runner kubectl failed (expired kubeconfig?) — universe record stands; operator reconciles"; }
+            else
+              kc() { kubectl "$@"; }
+            fi
             # Operator-managed services (Service CR) reconcile the Deployment —
             # patch the CR when it exists; bare Deployments get set-image.
             if kubectl -n "$NS" get "services.hanzo.ai/$svc" >/dev/null 2>&1; then
-              kubectl -n "$NS" patch "services.hanzo.ai/$svc" --type=merge \
+              kc -n "$NS" patch "services.hanzo.ai/$svc" --type=merge \
                 -p "{\"spec\":{\"image\":{\"repository\":\"$repo\",\"tag\":\"$tag\"}}}"
             else
               # Bare Deployment: set the new image ONLY on containers already running
@@ -527,12 +868,12 @@ jobs:
                 | awk -v r="$repo" 'index($2, r"@")==1 || index($2, r":")==1 {print $1}')
               [ "${#CS[@]}" -eq 0 ] && CS=("$svc")
               args=(); for c in "${CS[@]}"; do args+=("$c=$ref"); done
-              kubectl -n "$NS" set image "deployment/$svc" "${args[@]}"
+              kc -n "$NS" set image "deployment/$svc" "${args[@]}"
             fi
             # Timeout scales with the image: a service vendoring GB-scale model/node
             # packs (e.g. studio) pulls multi-GB layers on a cold node + waits for the
             # operator to reconcile — legitimately minutes. 180s failed mid-pull and
             # is THE reason studio deploys silently failed since 0.15.8. Override
             # per-repo with deploy.rollout-timeout.
-            kubectl -n "$NS" rollout status "deployment/$svc" --timeout="$RTO"
+            kc -n "$NS" rollout status "deployment/$svc" --timeout="$RTO"
           done

.hanzo/workflows/sync-from-github.yml

@@ -0,0 +1,78 @@
+name: Sync from GitHub
+
+# git.hanzo.ai is CANONICAL and builds natively; development also lands on
+# github.com/hanzoai/ci. This job carries GitHub's commits to the forge.
+#
+# WITHOUT IT THIS REPO DRIFTS, and it had: the forge sat 3 commits behind main
+# with nothing to close the gap. ci is one of only 13 non-mirror repos in the
+# hanzoai org, and non-mirror is exactly the gap between the two mechanisms that
+# keep the forge current:
+#
+#   hanzoai/mirrors reconcile.py — makes qualifying GitHub repos pull-mirrors on
+#     a 6h schedule, and the server then syncs them. It is idempotent and
+#     "existing mirrors are left alone", so a repo that already exists as a
+#     NON-mirror is skipped forever.
+#   this file — the per-repo pull, for repos that are native-canonical and so
+#     must not be mirrors.
+#
+# A non-mirror repo without this file has neither, and drifts silently. Sibling
+# repos that carry it sit 0-1 commits behind; the two that lacked it, ci and
+# cloud, sat 3 and 10 behind.
+#
+# ONE deterministic direction: an in-cluster PULL. The runner reaches both ends,
+# so the sync has no ingress dependency and nothing has to push INTO the forge
+# from outside — which matters, because GitHub org secrets are inert on this
+# Free-plan org and a GitHub-side push has no working credential.
+#
+# NO CREDENTIAL. hanzoai/ci is public, so the fetch is anonymous. Private
+# siblings need GH_PAT from the forge's own secret store; this one must never
+# grow a token it has no use for.
+#
+# Fast-forward ONLY. A divergence fails LOUDLY rather than force-pushing either
+# side and destroying whichever history lost the race.
+
+on:
+  schedule:
+    - cron: '*/10 * * * *'
+  workflow_dispatch: {}
+
+concurrency:
+  group: sync-from-github
+  cancel-in-progress: false
+
+jobs:
+  ff-main:
+    runs-on: [hanzo-build-linux-amd64]
+    steps:
+      - name: Checkout main (full history for the ancestry check)
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          persist-credentials: true
+
+      - name: Fast-forward main from github.com/hanzoai/ci
+        run: |
+          set -euo pipefail
+          git fetch --quiet https://github.com/hanzoai/ci.git main
+          LOCAL="$(git rev-parse HEAD)"
+          REMOTE="$(git rev-parse FETCH_HEAD)"
+          if [ "$LOCAL" = "$REMOTE" ]; then
+            echo "in sync at $LOCAL"
+            exit 0
+          fi
+          if git merge-base --is-ancestor "$LOCAL" "$REMOTE"; then
+            echo "fast-forwarding $LOCAL -> $REMOTE"
+            git push origin "$REMOTE:refs/heads/main"
+            # A push made with the workflow token does NOT trigger other workflows
+            # (loop prevention), so synced commits would never build. Dispatch it
+            # explicitly — a real fast-forward means real commits arrived.
+            curl -fsS --max-time 20 -X POST \
+              -H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
+              -H "Content-Type: application/json" \
+              "${{ github.server_url }}/v1/repos/${{ github.repository }}/actions/workflows/build.yml/dispatches" \
+              -d '{"ref":"main"}' || echo "build dispatch failed (non-fatal)"
+          else
+            echo "DIVERGED: native $LOCAL is not an ancestor of GitHub $REMOTE." >&2
+            echo "Resolve by hand; this job will not force-push either side." >&2
+            exit 1
+          fi

Dockerfile

@@ -0,0 +1,31 @@
+# syntax=docker/dockerfile:1
+#
+# ci — the ci.hanzo.ai dashboard. Pure-Go, no cgo, no assets: the page is
+# server-rendered from a template compiled into the binary, so the image is the
+# binary and a CA bundle. Nothing to serve from disk, nothing to go stale
+# against the code.
+FROM golang:1.24-alpine AS builder
+WORKDIR /build
+# Resolve through the module proxy: proxy.golang.org and sum.golang.org agree
+# and neither can change under us, which a direct fetch against a moved tag
+# cannot promise.
+ENV GOPROXY=https://proxy.golang.org,direct
+COPY go.mod ./
+RUN --mount=type=cache,id=ci-gomod,target=/go/pkg/mod go mod download
+COPY . .
+RUN --mount=type=cache,id=ci-gomod,target=/go/pkg/mod \
+    --mount=type=cache,target=/root/.cache/go-build \
+    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /build/ci .
+
+FROM alpine:3.21
+RUN apk add --no-cache ca-certificates tzdata \
+    && addgroup -S hanzo && adduser -S hanzo -G hanzo
+COPY --from=builder /build/ci /app/ci
+USER hanzo
+EXPOSE 8080
+# Liveness only. Readiness deliberately does not gate on having a snapshot — see
+# the /healthz comment in main.go: a Hanzo Git outage must render as a dashboard
+# saying so, not as this pod leaving the load balancer as well.
+HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --retries=3 \
+    CMD wget -qO- http://127.0.0.1:8080/healthz || exit 1
+ENTRYPOINT ["/app/ci"]

README.md

@@ -40,6 +40,74 @@ jobs:
 
 That's it. The build/test/deploy logic lives here, once.
 
+## `binaries:` — publish a plugin once, install it everywhere
+
+`images:` ships an OCI image a **cluster** runs. `binaries:` ships an
+executable a **running host** installs: a [zip](https://github.com/zap-proto/zip)
+plugin, fetched at run time by URL and verified against its SHA-256 before it is
+ever made executable. Build it once per OS/arch here; every host picks up the
+same bits, and nobody rebuilds the world to ship a plugin.
+
+```yaml
+binaries:
+  - name: billing
+    main: ./cmd/billing              # the Go package; default "."
+    platforms: [linux/amd64, linux/arm64]   # default [linux/amd64]
+    ldflags: "-s -w"                 # default
+```
+
+`main:` is the zero-config **Go** lane. Every other toolchain uses the same block
+with `run:` (the command that builds) and `out:` (the glob of what it produced) —
+which is how a repo with no Dockerfile and no Go still publishes an artifact:
+
+```yaml
+binaries:
+  - name: sdk
+    run: npm install && npm run build && npm pack --pack-destination .
+    out: "*.tgz"
+    image: node:22-bookworm          # the toolchain — see below
+```
+
+`image:` names the container the **platform** lane runs `run:` in
+(`POST /v1/runner`, one initContainer per entry, in-cluster). Here the toolchain
+IS the runner, so this workflow reads past it. It is not a second recipe: both
+lanes read the same `binaries:` block out of the same `hanzo.yml` and publish the
+same `binaries.json` at the same URL.
+
+Artifacts land under `<name>` in the index regardless of lane; a `run:` entry is
+`os: any, arch: any`, because an npm tarball or a wheel is not per-platform and
+an index entry that claimed one would be a lie a host acts on.
+
+Built on every push (an arm64 cross-compile that breaks fails the PR that broke
+it) and **published on a tag**, after the `test:` gate — a host installs an
+artifact unattended, so the tests gate the bits. Each artifact lands on the
+GitHub Release for that tag:
+
+```
+https://github.com/<owner>/<repo>/releases/download/<tag>/<name>-<os>-<arch>
+```
+
+plus `binaries.json` beside them — `{name, os, arch, url, sha256}` for every
+artifact, so the bits and the digest that authorizes them ship as one release
+and a host reads both from one place. The job summary prints the
+`zip.Load(zip.Plugin{URL, Sum})` a host pastes.
+
+Add a top-level `bucket:` and they publish to **hanzoai/s3** instead — same
+artifacts, same index, only the url changes:
+
+```yaml
+bucket: plugins   # → https://s3.hanzo.ai/plugins/<owner>/<repo>/<tag>/binaries.json
+```
+
+Credentials are the `S3_ADMIN_*` names the services already read, pulled from
+KMS at run time; a declared bucket with no credential fails the publish rather
+than shipping an index whose artifacts are missing. Use it for anything large or
+frequent — a GitHub release stores it on a quota we do not own.
+
+Builds are `CGO_ENABLED=0 -trimpath`: the host that installs this runs it on
+whatever base image the host is, and the digest must be a function of the
+source, not of the checkout path.
+
 ## Runners — our cloud or your own
 
 By default the build runs on the **Hanzo cloud** arc pool (we run it; metered as

go.mod

@@ -0,0 +1,3 @@
+module github.com/hanzoai/ci
+
+go 1.24

main.go

@@ -0,0 +1,481 @@
+// ci — the dashboard behind ci.hanzo.ai.
+//
+// It owns no build state. Run truth lives in Hanzo Git (git.hanzo.ai), which
+// schedules the jobs and holds every log; this reads that and presents it. The
+// alternative — a CI service with its own run database — would put two answers
+// to "did the build pass" in the fleet, and the one users look at would be the
+// one that can drift. So: git.hanzo.ai is the store, ci.hanzo.ai is the view.
+//
+// This is the CI half of the pair. cd.hanzo.ai reconciles image pins from
+// hanzoai/universe and is the delivery view; the two are deliberately separate
+// surfaces over separate systems, not one console pretending build and deploy
+// are the same event.
+//
+// Tenancy is the same value everywhere: an org slug. Hanzo Git namespaces repos
+// by org, IAM issues that slug in the `owner` claim, and Hanzo CD fences
+// projects by it. Filtering here by `org` is therefore the same boundary those
+// enforce, not a parallel notion of who-sees-what.
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"log/slog"
+	"net/http"
+	"net/url"
+	"os"
+	"os/signal"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+)
+
+func main() {
+	logger := slog.New(slog.NewJSONHandler(os.Stderr, nil))
+
+	cfg, err := loadConfig()
+	if err != nil {
+		logger.Error("config", "err", err)
+		os.Exit(1)
+	}
+
+	src := &gitSource{base: cfg.gitBase, token: cfg.gitToken, http: &http.Client{Timeout: 20 * time.Second}}
+	cache := &runCache{}
+
+	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
+	defer stop()
+
+	// One poller, one cache. Every viewer reads the same snapshot, so N open
+	// dashboards cost Hanzo Git exactly as much as one — a dashboard that
+	// fanned each page load into upstream calls is how a status page takes the
+	// system it reports on down.
+	go poll(ctx, logger, src, cache, cfg)
+
+	mux := http.NewServeMux()
+	mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
+		// Liveness only: up means "serving". Readiness deliberately does NOT
+		// gate on having a snapshot — a Hanzo Git outage must show as a stale
+		// dashboard saying so, not as ci.hanzo.ai disappearing from the LB too.
+		writeJSON(w, http.StatusOK, map[string]any{"status": "ok"})
+	})
+	mux.HandleFunc("/v1/runs", func(w http.ResponseWriter, r *http.Request) {
+		snap := cache.get()
+		writeJSON(w, http.StatusOK, map[string]any{
+			"runs":      filterByOrg(snap.Runs, r.URL.Query().Get("org")),
+			"fetchedAt": snap.FetchedAt,
+			"stale":     snap.stale(cfg.staleAfter),
+			"sourceErr": snap.errString(),
+			"repos":     snap.Repos,
+			"orgs":      orgsOf(snap.Runs),
+		})
+	})
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path != "/" {
+			http.NotFound(w, r)
+			return
+		}
+		renderDashboard(w, cache.get(), r.URL.Query().Get("org"), cfg)
+	})
+
+	srv := &http.Server{
+		Addr:              cfg.listen,
+		Handler:           mux,
+		ReadHeaderTimeout: 10 * time.Second,
+	}
+	go func() {
+		<-ctx.Done()
+		sh, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+		defer cancel()
+		_ = srv.Shutdown(sh)
+	}()
+
+	logger.Info("ci dashboard listening", "addr", cfg.listen, "source", cfg.gitBase, "refresh", cfg.refresh.String())
+	if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
+		logger.Error("serve", "err", err)
+		os.Exit(1)
+	}
+}
+
+// ───────────────────────────── config ─────────────────────────────
+
+type config struct {
+	listen     string
+	gitBase    string
+	gitToken   string
+	refresh    time.Duration
+	staleAfter time.Duration
+	scanRepos  int
+	runsPer    int
+}
+
+func loadConfig() (config, error) {
+	c := config{
+		listen:    env("CI_LISTEN", ":8080"),
+		gitBase:   strings.TrimRight(env("CI_GIT_BASE", "https://git.hanzo.ai"), "/"),
+		gitToken:  os.Getenv("CI_GIT_TOKEN"),
+		scanRepos: envInt("CI_SCAN_REPOS", 60),
+		runsPer:   envInt("CI_RUNS_PER_REPO", 8),
+	}
+	c.refresh = time.Duration(envInt("CI_REFRESH_SECONDS", 45)) * time.Second
+	// Stale is a multiple of refresh, not its own knob: the only meaningful
+	// definition of stale is "we have missed several refreshes", and deriving
+	// it means the two can never be configured into contradiction.
+	c.staleAfter = 4 * c.refresh
+	if c.gitToken == "" {
+		return c, errors.New("CI_GIT_TOKEN required (Hanzo Git API token)")
+	}
+	return c, nil
+}
+
+func env(k, def string) string {
+	if v := os.Getenv(k); v != "" {
+		return v
+	}
+	return def
+}
+
+func envInt(k string, def int) int {
+	if v, err := strconv.Atoi(os.Getenv(k)); err == nil && v > 0 {
+		return v
+	}
+	return def
+}
+
+// ───────────────────────────── model ─────────────────────────────
+
+// Run is the projection of a Hanzo Git workflow run this dashboard shows. It is
+// deliberately a SUBSET: the upstream object carries a dozen more fields, and
+// copying them all would make this a second schema to maintain against theirs.
+type Run struct {
+	ID       int64  `json:"id"`
+	Org      string `json:"org"`
+	Repo     string `json:"repo"`
+	Workflow string `json:"workflow"`
+	Title    string `json:"title"`
+
+	// Status and Conclusion are BOTH required to know how a run went, and
+	// reading only one is wrong in a way that looks fine. Status answers
+	// "is it over" (queued | in_progress | completed); Conclusion answers
+	// "how did it end" and is empty until it is over. A view that buckets on
+	// Status alone sees `completed` and cannot tell a pass from a failure —
+	// which is exactly the bug this pair replaced: every finished run,
+	// including successes and cancellations, was being drawn as failing.
+	Status     string `json:"status"`
+	Conclusion string `json:"conclusion"`
+
+	Event     string    `json:"event"`
+	Branch    string    `json:"branch"`
+	SHA       string    `json:"sha"`
+	Actor     string    `json:"actor"`
+	Number    int       `json:"number"`
+	URL       string    `json:"url"`
+	StartedAt time.Time `json:"startedAt"`
+	EndedAt   time.Time `json:"endedAt"`
+}
+
+// Duration is zero-valued rather than negative when a run has not finished —
+// callers render "running", and a negative duration would print as one.
+func (r Run) Duration() time.Duration {
+	if r.StartedAt.IsZero() || r.EndedAt.IsZero() || r.EndedAt.Before(r.StartedAt) {
+		return 0
+	}
+	return r.EndedAt.Sub(r.StartedAt)
+}
+
+type snapshot struct {
+	Runs      []Run     `json:"runs"`
+	Repos     int       `json:"repos"`
+	FetchedAt time.Time `json:"fetchedAt"`
+	Err       error     `json:"-"`
+}
+
+func (s snapshot) stale(after time.Duration) bool {
+	return s.FetchedAt.IsZero() || time.Since(s.FetchedAt) > after
+}
+
+func (s snapshot) errString() string {
+	if s.Err == nil {
+		return ""
+	}
+	return s.Err.Error()
+}
+
+type runCache struct {
+	mu   sync.RWMutex
+	snap snapshot
+}
+
+func (c *runCache) get() snapshot {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	return c.snap
+}
+
+// put keeps the LAST GOOD run list when a refresh fails, recording the error
+// alongside it. A failed poll must not blank the dashboard: "Hanzo Git is
+// unreachable, here is what we last saw" is strictly more useful than an empty
+// page, which reads as "nothing is building".
+func (c *runCache) put(s snapshot) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if s.Err != nil && len(s.Runs) == 0 && len(c.snap.Runs) > 0 {
+		prev := c.snap
+		prev.Err = s.Err
+		c.snap = prev
+		return
+	}
+	c.snap = s
+}
+
+// ───────────────────────────── source ─────────────────────────────
+
+type gitSource struct {
+	base  string
+	token string
+	http  *http.Client
+}
+
+func (g *gitSource) getJSON(ctx context.Context, path string, out any) error {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, g.base+path, nil)
+	if err != nil {
+		return err
+	}
+	req.Header.Set("Authorization", "token "+g.token)
+	req.Header.Set("Accept", "application/json")
+	resp, err := g.http.Do(req)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return fmt.Errorf("%s: %s", path, resp.Status)
+	}
+	return json.NewDecoder(resp.Body).Decode(out)
+}
+
+type repoRef struct {
+	FullName string `json:"full_name"`
+}
+
+// repos returns the most recently ACTIVE repositories. Sorting by activity and
+// taking a window is the whole scan strategy: the instance mirrors ~1400 repos
+// and almost none of them built in the last hour, so walking all of them would
+// spend the entire refresh budget confirming silence.
+func (g *gitSource) repos(ctx context.Context, limit int) ([]string, error) {
+	var body struct {
+		Data []repoRef `json:"data"`
+	}
+	q := url.Values{}
+	q.Set("sort", "updated")
+	q.Set("order", "desc")
+	q.Set("limit", strconv.Itoa(limit))
+	if err := g.getJSON(ctx, "/v1/repos/search?"+q.Encode(), &body); err != nil {
+		return nil, err
+	}
+	names := make([]string, 0, len(body.Data))
+	for _, r := range body.Data {
+		if r.FullName != "" {
+			names = append(names, r.FullName)
+		}
+	}
+	return names, nil
+}
+
+type apiRun struct {
+	ID           int64  `json:"id"`
+	DisplayTitle string `json:"display_title"`
+	Path         string `json:"path"`
+	Event        string `json:"event"`
+	Status       string `json:"status"`
+	Conclusion   string `json:"conclusion"`
+	HeadBranch   string `json:"head_branch"`
+	HeadSHA      string `json:"head_sha"`
+	RunNumber    int    `json:"run_number"`
+	HTMLURL      string `json:"html_url"`
+	StartedAt    string `json:"started_at"`
+	CompletedAt  string `json:"completed_at"`
+	Actor        struct {
+		Login string `json:"login"`
+	} `json:"actor"`
+}
+
+func (g *gitSource) runs(ctx context.Context, fullName string, limit int) ([]Run, error) {
+	var body struct {
+		WorkflowRuns []apiRun `json:"workflow_runs"`
+	}
+	path := fmt.Sprintf("/v1/repos/%s/actions/runs?limit=%d", fullName, limit)
+	if err := g.getJSON(ctx, path, &body); err != nil {
+		return nil, err
+	}
+	org, repo := splitFullName(fullName)
+	out := make([]Run, 0, len(body.WorkflowRuns))
+	for _, r := range body.WorkflowRuns {
+		out = append(out, Run{
+			ID:         r.ID,
+			Org:        org,
+			Repo:       repo,
+			Workflow:   workflowOf(r.Path),
+			Title:      r.DisplayTitle,
+			Status:     r.Status,
+			Conclusion: r.Conclusion,
+			Event:      r.Event,
+			Branch:     r.HeadBranch,
+			SHA:        shortSHA(r.HeadSHA),
+			Actor:      r.Actor.Login,
+			Number:     r.RunNumber,
+			URL:        r.HTMLURL,
+			StartedAt:  parseTime(r.StartedAt),
+			EndedAt:    parseTime(r.CompletedAt),
+		})
+	}
+	return out, nil
+}
+
+// poll refreshes the snapshot on an interval, forever.
+func poll(ctx context.Context, logger *slog.Logger, src *gitSource, cache *runCache, cfg config) {
+	refresh := func() {
+		rctx, cancel := context.WithTimeout(ctx, 90*time.Second)
+		defer cancel()
+
+		names, err := src.repos(rctx, cfg.scanRepos)
+		if err != nil {
+			logger.Warn("repo scan failed", "err", err)
+			cache.put(snapshot{FetchedAt: time.Now().UTC(), Err: err})
+			return
+		}
+
+		// Fan out, bounded. The cap is small on purpose: this is a read against
+		// the forge that schedules every build in the fleet, and a dashboard is
+		// never worth degrading it.
+		const workers = 6
+		var (
+			mu   sync.Mutex
+			all  []Run
+			errs []string
+			wg   sync.WaitGroup
+		)
+		jobs := make(chan string)
+		for i := 0; i < workers; i++ {
+			wg.Add(1)
+			go func() {
+				defer wg.Done()
+				for name := range jobs {
+					rs, err := src.runs(rctx, name, cfg.runsPer)
+					mu.Lock()
+					if err != nil {
+						// A repo with Actions disabled 404s. That is normal and
+						// not worth surfacing as a dashboard-level failure, so
+						// it is counted, not shown.
+						errs = append(errs, name)
+					} else {
+						all = append(all, rs...)
+					}
+					mu.Unlock()
+				}
+			}()
+		}
+		for _, n := range names {
+			select {
+			case jobs <- n:
+			case <-rctx.Done():
+			}
+		}
+		close(jobs)
+		wg.Wait()
+
+		sort.Slice(all, func(i, j int) bool { return all[i].StartedAt.After(all[j].StartedAt) })
+		cache.put(snapshot{Runs: all, Repos: len(names) - len(errs), FetchedAt: time.Now().UTC()})
+		logger.Info("refreshed", "repos", len(names), "withRuns", len(names)-len(errs), "runs", len(all))
+	}
+
+	refresh()
+	t := time.NewTicker(cfg.refresh)
+	defer t.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-t.C:
+			refresh()
+		}
+	}
+}
+
+// ───────────────────────────── helpers ─────────────────────────────
+
+func splitFullName(s string) (org, repo string) {
+	if i := strings.IndexByte(s, '/'); i > 0 {
+		return s[:i], s[i+1:]
+	}
+	return "", s
+}
+
+// workflowOf reduces "e2e.yml@refs/heads/main" to "e2e.yml".
+func workflowOf(path string) string {
+	if i := strings.IndexByte(path, '@'); i > 0 {
+		return path[:i]
+	}
+	return path
+}
+
+func shortSHA(s string) string {
+	if len(s) > 7 {
+		return s[:7]
+	}
+	return s
+}
+
+func parseTime(s string) time.Time {
+	if s == "" {
+		return time.Time{}
+	}
+	t, err := time.Parse(time.RFC3339, s)
+	if err != nil {
+		return time.Time{}
+	}
+	// Hanzo Git reports an unset timestamp as the Unix epoch rather than null;
+	// treated as absent so the UI shows "—" instead of 1970.
+	if t.Year() < 2000 {
+		return time.Time{}
+	}
+	return t.UTC()
+}
+
+func filterByOrg(runs []Run, org string) []Run {
+	if org == "" {
+		return runs
+	}
+	out := make([]Run, 0, len(runs))
+	for _, r := range runs {
+		if r.Org == org {
+			out = append(out, r)
+		}
+	}
+	return out
+}
+
+func orgsOf(runs []Run) []string {
+	seen := map[string]bool{}
+	for _, r := range runs {
+		if r.Org != "" {
+			seen[r.Org] = true
+		}
+	}
+	out := make([]string, 0, len(seen))
+	for o := range seen {
+		out = append(out, o)
+	}
+	sort.Strings(out)
+	return out
+}
+
+func writeJSON(w http.ResponseWriter, status int, v any) {
+	w.Header().Set("Content-Type", "application/json")
+	w.WriteHeader(status)
+	_ = json.NewEncoder(w).Encode(v)
+}

render.go

@@ -0,0 +1,225 @@
+package main
+
+import (
+	"fmt"
+	"html/template"
+	"net/http"
+	"strings"
+	"time"
+)
+
+// render.go — the HTML view. Server-rendered on purpose: this page is a table
+// of build results, and a client-side app would ship a bundle, a fetch layer
+// and a loading state to show the same rows a second later. The dashboard also
+// has to be readable when the thing it reports on is broken, which is exactly
+// when a build pipeline for its own frontend is the wrong dependency.
+
+func renderDashboard(w http.ResponseWriter, snap snapshot, org string, cfg config) {
+	runs := filterByOrg(snap.Runs, org)
+	if len(runs) > 200 {
+		runs = runs[:200]
+	}
+
+	data := struct {
+		Runs      []Run
+		Orgs      []string
+		Org       string
+		Repos     int
+		FetchedAt time.Time
+		Age       string
+		Stale     bool
+		SourceErr string
+		Source    string
+		Counts    map[string]int
+	}{
+		Runs:      runs,
+		Orgs:      orgsOf(snap.Runs),
+		Org:       org,
+		Repos:     snap.Repos,
+		FetchedAt: snap.FetchedAt,
+		Age:       humanAge(snap.FetchedAt),
+		Stale:     snap.stale(cfg.staleAfter),
+		SourceErr: snap.errString(),
+		Source:    cfg.gitBase,
+		Counts:    countByOutcome(runs),
+	}
+
+	w.Header().Set("Content-Type", "text/html; charset=utf-8")
+	if err := tmpl.Execute(w, data); err != nil {
+		http.Error(w, err.Error(), http.StatusInternalServerError)
+	}
+}
+
+// countByOutcome buckets runs for the summary strip.
+func countByOutcome(runs []Run) map[string]int {
+	c := map[string]int{"success": 0, "failure": 0, "running": 0, "cancelled": 0}
+	for _, r := range runs {
+		c[outcome(r)]++
+	}
+	return c
+}
+
+// outcome collapses (status, conclusion) into the four states worth a colour.
+//
+// Status alone is NOT enough and getting this wrong is silent: Hanzo Git
+// reports every finished run as `completed` regardless of how it went, so
+// bucketing on status painted successes and cancellations as failures — on the
+// live instance that was 15 of 20 runs mislabelled red.
+//
+// `cancelled` gets its own bucket rather than folding into failure. On this
+// fleet cancellations are the single largest category (superseded pushes cancel
+// the in-flight run), and a board that shows them as broken is a board nobody
+// trusts, which is worse than no board.
+func outcome(r Run) string {
+	if !strings.EqualFold(r.Status, "completed") {
+		return "running" // queued | in_progress | waiting | blocked
+	}
+	switch strings.ToLower(r.Conclusion) {
+	case "success":
+		return "success"
+	case "cancelled", "canceled", "skipped":
+		return "cancelled"
+	case "":
+		// Completed with no conclusion should not happen; if it does, say
+		// "running" rather than inventing a verdict the data does not support.
+		return "running"
+	default:
+		return "failure" // failure | timed_out | action_required
+	}
+}
+
+func humanAge(t time.Time) string {
+	if t.IsZero() {
+		return "never"
+	}
+	d := time.Since(t)
+	switch {
+	case d < time.Minute:
+		return fmt.Sprintf("%ds ago", int(d.Seconds()))
+	case d < time.Hour:
+		return fmt.Sprintf("%dm ago", int(d.Minutes()))
+	default:
+		return fmt.Sprintf("%dh ago", int(d.Hours()))
+	}
+}
+
+func humanDur(d time.Duration) string {
+	if d <= 0 {
+		return "—"
+	}
+	if d < time.Minute {
+		return fmt.Sprintf("%ds", int(d.Seconds()))
+	}
+	return fmt.Sprintf("%dm%02ds", int(d.Minutes()), int(d.Seconds())%60)
+}
+
+var tmpl = template.Must(template.New("ci").Funcs(template.FuncMap{
+	"outcome": outcome,
+	"dur":     func(r Run) string { return humanDur(r.Duration()) },
+	"ago":     humanAge,
+}).Parse(`<!doctype html>
+<html lang="en"><head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width,initial-scale=1">
+<title>Hanzo CI</title>
+<meta http-equiv="refresh" content="60">
+<style>
+:root{--bg:#0b0b0d;--panel:#141417;--line:#25252b;--fg:#e8e8ea;--dim:#8b8b95;
+      --ok:#3fb950;--fail:#f85149;--run:#d29922;--accent:#8B5CF6}
+*{box-sizing:border-box}
+body{margin:0;background:var(--bg);color:var(--fg);
+     font:14px/1.5 ui-sans-serif,-apple-system,"Segoe UI",Roboto,sans-serif}
+header{display:flex;align-items:center;gap:16px;padding:16px 24px;
+       border-bottom:1px solid var(--line);background:var(--panel)}
+h1{margin:0;font-size:16px;font-weight:600;letter-spacing:-.01em}
+h1 span{color:var(--accent)}
+.meta{margin-left:auto;color:var(--dim);font-size:12px;text-align:right}
+.strip{display:flex;gap:8px;padding:16px 24px;flex-wrap:wrap}
+.chip{padding:6px 12px;border:1px solid var(--line);border-radius:8px;
+      background:var(--panel);font-size:12px;color:var(--dim)}
+.chip b{color:var(--fg);font-weight:600}
+.chip.ok b{color:var(--ok)} .chip.fail b{color:var(--fail)} .chip.run b{color:var(--run)}
+.chip.cancel b{color:var(--dim)}
+nav{display:flex;gap:6px;padding:0 24px 16px;flex-wrap:wrap}
+nav a{padding:5px 11px;border:1px solid var(--line);border-radius:999px;
+      background:var(--panel);color:var(--dim);text-decoration:none;font-size:12px}
+nav a.on{border-color:var(--accent);color:var(--fg)}
+.warn{margin:0 24px 16px;padding:10px 14px;border:1px solid var(--run);
+      border-radius:8px;background:#221b0c;color:#f0d58c;font-size:13px}
+table{width:100%;border-collapse:collapse}
+th{position:sticky;top:0;background:var(--panel);text-align:left;font-size:11px;
+   text-transform:uppercase;letter-spacing:.06em;color:var(--dim);
+   padding:10px 12px;border-bottom:1px solid var(--line);font-weight:600}
+td{padding:10px 12px;border-bottom:1px solid var(--line);vertical-align:top}
+tr:hover td{background:#111114}
+a{color:inherit}
+.dot{display:inline-block;width:8px;height:8px;border-radius:50%;margin-right:8px}
+.dot.success{background:var(--ok)} .dot.failure{background:var(--fail)}
+.dot.running{background:var(--run);animation:p 1.4s ease-in-out infinite}
+.dot.cancelled{background:#4a4a52}
+@keyframes p{50%{opacity:.35}}
+.repo{font-weight:600}
+.org{color:var(--dim)}
+.title{color:var(--dim);max-width:42ch;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
+.mono{font-family:ui-monospace,SFMono-Regular,Menlo,monospace;font-size:12px;color:var(--dim)}
+.empty{padding:48px 24px;text-align:center;color:var(--dim)}
+footer{padding:16px 24px;color:var(--dim);font-size:12px;border-top:1px solid var(--line)}
+@media(max-width:760px){.hide-sm{display:none}}
+</style></head><body>
+
+<header>
+  <h1>Hanzo <span>CI</span></h1>
+  <div class="meta">
+    {{.Repos}} repos &middot; refreshed {{.Age}}<br>
+    source {{.Source}}
+  </div>
+</header>
+
+<div class="strip">
+  <span class="chip ok">passing <b>{{index .Counts "success"}}</b></span>
+  <span class="chip fail">failing <b>{{index .Counts "failure"}}</b></span>
+  <span class="chip run">running <b>{{index .Counts "running"}}</b></span>
+  <span class="chip cancel">cancelled <b>{{index .Counts "cancelled"}}</b></span>
+</div>
+
+<nav>
+  <a href="/" {{if eq .Org ""}}class="on"{{end}}>all orgs</a>
+  {{range .Orgs}}<a href="/?org={{.}}" {{if eq $.Org .}}class="on"{{end}}>{{.}}</a>{{end}}
+</nav>
+
+{{if .Stale}}<div class="warn">
+  Snapshot is stale — last successful refresh {{.Age}}.
+  {{if .SourceErr}}Hanzo Git said: {{.SourceErr}}{{else}}Hanzo Git is not answering.{{end}}
+  These rows are the last good read, not current state.
+</div>{{end}}
+
+{{if .Runs}}
+<table>
+<thead><tr>
+  <th>Repository</th><th>Workflow</th><th class="hide-sm">Commit</th>
+  <th class="hide-sm">Actor</th><th>Started</th><th>Took</th>
+</tr></thead>
+<tbody>
+{{range .Runs}}
+<tr>
+  <td><span class="dot {{outcome .}}"></span><a href="{{.URL}}"><span class="org">{{.Org}}/</span><span class="repo">{{.Repo}}</span></a></td>
+  <td>{{.Workflow}} <span class="mono">#{{.Number}}</span><div class="title">{{.Title}}</div></td>
+  <td class="hide-sm mono">{{.Branch}}@{{.SHA}}<div>{{.Event}}</div></td>
+  <td class="hide-sm mono">{{.Actor}}</td>
+  <td class="mono">{{ago .StartedAt}}</td>
+  <td class="mono">{{dur .}}</td>
+</tr>
+{{end}}
+</tbody></table>
+{{else}}
+<div class="empty">
+  No runs in the scanned window.<br>
+  <span class="mono">Builds land here from {{.Source}} — this view holds no state of its own.</span>
+</div>
+{{end}}
+
+<footer>
+  Build truth lives in Hanzo Git; this is a view over it.
+  Delivery is <a href="https://cd.hanzo.ai">cd.hanzo.ai</a>.
+</footer>
+</body></html>`))