Compare commits

..
37 Commits
Author SHA1 Message Date
hanzo-dev 7077cec764 fix(ci): mirror-credential login is best-effort — a registry.hanzo.ai hiccup must not skip the build
Both `docker login registry.hanzo.ai` calls ran unguarded under bash -e, so a
login FAILURE (registry down / transient) aborted the step and SKIPPED the whole
build — the image never reached GHCR (the primary). A missing cred was already
fail-safe; a failed login now is too: warn + skip the mirror, push GHCR-only.
Same best-effort principle as the cloud release lane.
2026-07-17 23:46:54 -07:00
hanzo-dev 1aae2b65a3 ci: pin universe CRs to canonical semver on tag releases (not sha-<short>-amd64)
Deploy roll now pins spec.image.tag to the BARE semver (VER = ref_name w/o v)
on a tagged release — matching world 2.4.10 / cloud v1.801.62 — instead of the
sha-<short>-amd64 it hardcoded on every build. Branch/main pushes keep their
per-commit sha tag (continuous dev path), now arch-matched (bare sha for
multi-arch, fixing a latent -amd64 mismatch).

Build step also publishes the bare semver tag on tag builds (kept the -amd64
alias for back-compat). New semver backward-clobber guard: a branch build never
overwrites a service already pinned to a semver release — complements the
sha-ancestry guard (which only sees sha- tags), so neither kind of pin rolls
backward.

Emitter of the 'deploy(<svc>): sha-...' hanzo-ci commits. Consumed as @v1.
2026-07-17 23:16:48 -07:00
Hanzo AI 93eafe4f55 roll: never backward, sweep every same-repo image in the CR
Builds finish out of order — a slow build of an older commit overwrote newer
rolls (observed live: studio pinned back one merge). The roll now skips when
the CR's sha is a descendant of the builder's. And the tag field alone left
same-image sidecars on stale tags every roll (reconciled by hand four times
tonight); every same-repo image reference in the CR now moves together.
2026-07-17 13:02:41 -07:00
Hanzo AI 4fcc2e02ce fix(ci): provision Node + corepack for JS test gates (exit-127 corepack-not-found on bare arc runners)
The reusable workflow provisions Go and C toolchains for hanzo.yml test
gates but never Node — any JS caller's gate (pnpm install && pnpm lint)
died at 'corepack: command not found' before reading package.json.
Mirror the Go step: guarded to package.json callers, setup-node 22 +
corepack enable so the repo's pinned packageManager shims resolve.
2026-07-16 10:58:04 -07:00
Hanzo AI 2866795e2d fix(ci): deploy rollout-timeout configurable, default 600s (180s failed mid-pull on GB-scale images → studio deploys silently failed since 0.15.8); set-image on repo-matching containers only (never '*' wildcard clobbering rclone sidecars on bare deployments) 2026-07-15 21:59:57 -07:00
zeekay 3083524803 ci: mirror via crane (IAM token realm vs buildx multi-scope) 2026-07-15 02:59:43 -07:00
zeekay 313180e201 ci: mirror prefers direct REGISTRY_USER/PASSWORD (private repos can't see org KMS secrets on Free); kubeconfig fallback retained 2026-07-15 02:43:31 -07:00
zeekay 76cf840edb ci: remove bisect artifacts (root cause: caller-org default workflow permissions must be write for permissions: packages: write reusables) 2026-07-14 23:37:05 -07:00
zeekay 28f1664092 ci: min5/6/7 single-variable bisect 2026-07-14 23:35:39 -07:00
zeekay f3acdc8cf7 ci: min3/min4 bisect 2026-07-14 23:33:00 -07:00
zeekay 02370735a0 ci: min2 header bisect 2026-07-14 23:32:16 -07:00
zeekay 39553ee364 ci: min reusable (cross-org bisect) 2026-07-14 23:31:29 -07:00
zeekay 5ee1b857de fix(ci): test-only callers (no images:) skip the build step cleanly 2026-07-14 23:28:00 -07:00
zeekay e628b788c4 feat(ci): dual-host image push — ghcr.io + registry.hanzo.ai mirror
Public identity stays ghcr.io (GitHub imports keep working); every built tag
is also mirrored server-side (imagetools create) to OUR fleet registry so the
cluster never depends on GHCR to deploy. Credential = the cluster-synced
registry-credentials dockerconfig read via the KMS-fetched kubeconfig;
gracefully skips (GHCR-only) when unavailable. No rebuild, no extra minutes.
2026-07-14 23:26:48 -07:00
hanzo-dev 56e83218ca fix(ci): deploys record desired state in universe, then accelerate
gitops-reconcile re-applies universe CRs every ~5min, so a CR patch (or
set-image) alone is reverted on the next cycle. The deploy step now
bumps infra/k8s/operator/crs/<svc>.yaml in universe (no-op-safe commit,
same KMS git token) and keeps the CR patch/set-image only to make the
roll immediate.

Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
2026-07-14 17:26:57 -07:00
hanzo-dev b5df8baca7 fix(ci): deploy patches the operator Service CR, not the Deployment
The hanzo operator reconciles Deployments from the Service CR — a bare
'kubectl set image' gets reverted on the next reconcile (observed on
world: rolled, served, reverted minutes later). Patch the CR's
spec.image when one exists; keep the Deployment fallback for
non-operator services.

Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
2026-07-14 17:22:08 -07:00
hanzo-dev 3668bb9ebd fix(ci): provision static kubectl for the deploy step
Bare arc runners ship no kubectl; the deploy step died with exit 127
right after the image push. Same sudo-free static-binary pattern as
jq/yq.

Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
2026-07-14 17:16:46 -07:00
hanzo-dev 86428e2f8a fix(ci): survive apt mirror rot in the cgo provision step
arc snapshot images intermittently lose archive.ubuntu.com Release files
(apt-get update exit 100 → every Go repo's build dies before its gates).
Repoint to the DO mirror (sources.list + deb822) and retry once; still a
no-op when gcc is baked in.

Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
2026-07-14 17:04:19 -07:00
hanzo-dev 9f4fd9a959 fix(ci): static yq provisioning — unblocks builds on locked-down arc nodes
PyYAML can't install on arc nodes (sudo blocked by no_new_privs, no pip). Parse
hanzo.yml with a curl-installed static yq binary + jq instead. Proven by the cms
build (past provision→GHCR→KMS). Robust on bare AND pre-baked nodes.
2026-07-13 10:39:40 -07:00
f7426add1d fix(ci): provision Node for JS/TS test gates on bare arc runners (#9)
A hanzo.yml `test:` gate for a JS/TS repo (e.g. `corepack … && pnpm lint`) runs
directly on the runner, but the minimal arc runner image ships no user-PATH Node
(its bundled node is for the runner's own action execution only), so the gate died
`corepack: command not found` (exit 127) — e.g. hanzo.ai's lint gate.

Add a `Provision Node toolchain` step (the JS twin of the existing Go-toolchain
provision): `actions/setup-node@v4` pinned to LTS 22, guarded to repos with a
package.json (`hashFiles('package.json') != ''`) so non-JS callers are unaffected
and harmless if a future runner image bakes Node in. corepack then activates the
exact pnpm/npm the gate requests.

Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ

Co-authored-by: hanzo-dev <dev@hanzo.ai>
2026-07-10 11:20:08 -07:00
0770a28962 fix(ci): make images: optional in hanzo.yml (lint-only / deploy-elsewhere repos) (#8)
The reusable hard-subscripted `hanzo.yml['images']` in the delegate, buildx, and
deploy steps, so a repo that ships no container image (e.g. a static site deployed
via its own Cloudflare Pages deploy.yml, importing this reusable only for the
`test:` lint gate) failed with a Python KeyError before any build ran.

Use `.get('images') or []` in all three places: absent → empty list → the build
loop runs zero times and no GHCR push is attempted. Backward-compatible (every
repo with `images:` is unchanged) and it stops a cross-org repo (e.g.
hanzo-apps/hanzo.ai) from hitting `denied: permission_denied` on a vestigial push.

Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ

Co-authored-by: hanzo-dev <dev@hanzo.ai>
2026-07-10 11:08:51 -07:00
zeekayandClaude Opus 4.8 5106bdeada ci: authenticate runner git for private Go modules in the Test step
The Test step runs `go vet`/`go test` on the runner, so `go` fetches
private hanzoai/* modules (GOPRIVATE → direct) via the runner's git. Repos
that authenticate builds with the KMS `gh_token` (GIT_TOKEN) rather than an
org GH_PAT had no runner-git credential, so the gate failed with
`fatal: could not read Username for 'https://github.com'` on
hanzoai/dbx, hanzoai/tasks, hanzoai/pubsub-go, etc.

Add a guarded step (before Test) that configures git `insteadOf` with the
SAME token the image build uses — GIT_TOKEN (set by the KMS step), GH_PAT
fallback — so the runner's git can clone private modules. Gated on go.mod;
no-op when no token is present.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 10:49:15 -07:00
zeekayandClaude Opus 4.8 0339fc7dee ci: provision Go + C toolchain before the Test step on bare arc runners
hanzo.yml `test:` gates run directly on the runner (not in a build
container), but the stock arc runner image ships no Go and no C compiler.
Any Go test gate therefore died with `go: command not found` (exit 127) —
and CGO_ENABLED=1 gates would next hit `cgo: gcc not found`. This was
latent because most callers never reached the Test step (image build failed
first); hanzoai/commerce is the first to build clean and reach a Go gate.

Add two guarded steps before Test, mirroring the existing jq/PyYAML
provisioning: `actions/setup-go@v5` pinned to the repo's own go.mod version,
and a guarded gcc install. Both gated on `hashFiles('go.mod')` so pure JS/TS
callers are unaffected, and both no-op when the toolchain is already present.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 10:42:14 -07:00
hanzo-dev 0542e97a06 fix(ci): derive KMS org from GitHub owner so the deploy-cred fetch actually runs
The KMS deploy-cred step read the org only from hanzo.yml kms.org or the
KMS_ORG var. No repo sets either, so ORG was always empty and the step
short-circuited with 'KMS not configured' — /v1/kms/auth/login was never
called. Every private-dep build then fell back to the org GH_PAT, which
cannot read private hanzoai/cloud, so go mod tidy failed with git exit 128
(commerce/ai/chat images never built; #70 enforcement could not deploy).

Derive ORG from github.repository_owner (hanzoai->hanzo, luxfi->lux,
zooai->zoo; owner as-is otherwise) when unset. hanzo.yml kms.org and
KMS_ORG still override. One place, zero per-repo config. The full path
(login -> token -> GET deploy/GITHUB_TOKEN -> read hanzoai/cloud) is
verified live against kms.hanzo.ai.
2026-07-04 21:56:36 -07:00
265f807635 ci: add opt-in mode: delegate — hand the build to platform.hanzo.ai (#6)
The default `mode: buildx` path is unchanged: buildx → test → deploy ON the
arc runner. `mode: delegate` instead POSTs each hanzo.yml image to platform's
direct build webhook (POST /v1/arcd/enqueue, bearer PLATFORM_BUILD_CALLBACK_TOKEN,
body {repo, sha, image, ref, branch, dockerfile, context, os, arch}) and exits
in seconds — platform builds in-cluster with BuildKit on its own pool, pushes to
the registry, and rolls the operator Service CR. Same downstream as platform's
GitHub-App webhook (one build path, two front doors); no runner buildx, no KMS,
no runner-side deploy.

- new `mode` workflow_call input (default `buildx`) — delegation is opt-in, so
  existing repos are untouched.
- new "Delegate build to platform" step (mode == delegate): parses hanzo.yml,
  enqueues one build per (image, platform) with the buildx tag shape the deploy
  path expects (sha-<short>-<arch>[-<suffix>]); fails loud on a non-202.
- buildx/GHCR/KMS/test/deploy steps gated `mode != 'delegate'`; checkout + parse
  toolchain still run (needed to read hanzo.yml).
- endpoint override via the PLATFORM_ENQUEUE_URL repo/org var.
- README documents the delegate opt-in.

Co-authored-by: hanzo-dev <dev@hanzo.ai>
2026-07-04 21:11:03 -07:00
zandGitHub c06948b467 Merge pull request #7 from hanzoai/fix/buildx-ghpat-fallback
fix(ci): fall back buildx gh_token to GH_PAT when KMS token is empty
2026-07-04 21:09:07 -07:00
hanzo-dev 2ef5d47f94 fix(ci): fall back buildx gh_token to GH_PAT when KMS token is empty
The buildx `gh_token` secret (for reading private cross-org Go modules like
github.com/hanzoai/cloud during `go mod tidy`) was mounted ONLY from the
KMS-fetched GIT_TOKEN. When the KMS deploy-cred login fails (or KMS is
unconfigured), GIT_TOKEN is empty, the --secret is omitted, and the buildx
stage's `go mod tidy` hits the private repo unauthenticated -> git exit 128 ->
image build fails. This broke every repo with a private cross-org dep
(hanzoai/commerce, hanzoai/ai) while the KMS login was down.

Fall back to the existing valid GH_PAT org secret — the SAME BuildKit gh_token
cloud/release.yml already uses successfully. Guarded + exported; no-op for
public-only builds when both are empty. One credential path, proven working.
2026-07-04 21:08:47 -07:00
3a9e055c45 feat(ci): opt-in multi-arch builds (amd64+arm64) via hanzo.yml platforms (#5)
DOKS has no arm64 nodes. Add per-image opt-in `platforms:` in hanzo.yml:
default stays [linux/amd64] (every existing repo's -amd64 tag shape UNCHANGED,
zero behavior change). Set [linux/amd64, linux/arm64] → buildx emits a
multi-arch manifest list (one digest, both arches); binfmt/QEMU installed for
arm64 emulation, and pure-Go CGO_ENABLED=0 Dockerfiles honoring $TARGETARCH
cross-compile natively (fast). For native-speed arm64, register a bare-metal
arm64 host (spark/GB10) as the hanzo-build-linux-arm64 runner (values exist).

Co-authored-by: hanzo-dev <dev@hanzo.ai>
2026-07-04 19:47:09 -07:00
hanzo-dev 8ae1c46625 fix(ci): KMS step must set +e (GitHub wraps run in bash -e)
The step is documented BEST-EFFORT (GHCR push uses the workflow token;
deploy creds are optional), and its 'set -uo pipefail' deliberately omits
-e. But GitHub runs 'run:' under 'bash -eo pipefail', so errexit is active
regardless — an unguarded curl (a KMS secret 404 at the caller's org/path)
aborted the step (exit 22) and failed the whole build. Explicitly 'set +e'
so KMS degrades gracefully (missing kubeconfig -> deploy simply skipped).
2026-07-03 15:42:25 -07:00
hanzo-dev 9d18d4a6ba fix(ci): provision jq + PyYAML on the runner before parsing hanzo.yml
The stock arc runner image (ghcr.io/actions/actions-runner:latest) is
minimal and ships neither jq nor python3-yaml, but the build step parses
hanzo.yml with python3+PyYAML under 'set -euo pipefail' and fails:
  ModuleNotFoundError: No module named 'yaml'
(The KMS step swallowed the same error via '|| true'.)

Add a guarded setup step (apt-get python3-yaml jq) right after checkout —
idempotent, a no-op once a runner image bakes them in. Keeps the reusable
self-contained so any org can import it onto a bare runner.
2026-07-03 15:34:24 -07:00
ecc3da75f0 fix(ci): build.yml runner defaults to online ARC pool, not offline evo (same disease as .github#11) (#3)
Co-authored-by: Hanzo AI <ai@hanzo.ai>
2026-07-02 23:38:52 -07:00
z d425a88e9a docs(brand): add hero banner 2026-06-28 20:06:09 -07:00
z fb096091b3 chore(brand): dynamic hero banner 2026-06-28 20:06:08 -07:00
hanzo-dev aadb7a04c0 ci: drop blanket GOPRIVATE — public modules use the immutable proxy+sumdb
luxfi/hanzoai/zooai Go modules are public (verified: dex, precompile, authz, pq,
age, keys all resolve unauthenticated). Setting GOPRIVATE for the whole orgs
routed them 'direct' and bypassed sum.golang.org, so a force-moved tag silently
poisoned every downstream go.sum (the recurring 'checksum mismatch' build
breakage). Dropping it lets go resolve them via proxy.golang.org + sumdb —
canonical IMMUTABLE hashes a re-publish cannot break. The GH_PAT git credential
stays for authenticated direct fallback + any genuinely-private repo (re-add via
a NARROW GOPRIVATE, never whole-org).
2026-06-28 14:10:59 -07:00
hanzo-dev 599667e100 fix(ci): tag-suffix optional — clean image tags when absent
The reusable build read ."tag-suffix" unconditionally, so any repo without it
(every repo today) got broken tags like sha-xxx-amd64-null. Default to empty in
both build and deploy so single-variant repos get clean tags; multi-variant
repos still qualify (…-ce, ce-latest). Unblocks adopting the reusable workflow.
2026-06-28 05:38:06 -07:00
Hanzo DevandGitHub 2bc3b2e1a9 build: KMS secret fetch over canonical /v1/kms + buildx gh_token secret (#2)
* build: KMS secret fetch over canonical /v1/kms + buildx gh_token secret

The Infisical /api/* surface was removed when KMS migrated to luxfi/kms, so the
old /api/v1/auth/universal-auth/login + /api/v3/secrets/raw calls now 404 and
every repo's CI fails to fetch GHCR_TOKEN/KUBECONFIG. Rewrite to /v1/kms:
- auth: POST /v1/kms/auth/login {clientId,clientSecret} (IAM client_credentials
  via the <org>-kms app) -> accessToken
- fetch: per-secret GET /v1/kms/orgs/<org>/secrets/<path>/<name>?env=<env>
- org from hanzo.yml kms.org or KMS_ORG var
Also fetch GIT_TOKEN and pass it to buildx as the gh_token BuildKit secret so
Dockerfiles can clone private Go modules (luxfi/dex, luxfi/precompile). Empty-safe.

* build: one GITHUB_TOKEN in KMS for GHCR push + private-module clone (DRY)

A single GitHub PAT with repo + write:packages does both jobs, so collapse the
split GHCR_TOKEN/GIT_TOKEN to one KMS key GITHUB_TOKEN. KUBECONFIG stays separate
(different concern). Shell var GHTOKEN avoids the reserved GITHUB_ env prefix.

* build: GHCR push via automatic workflow token; KMS only for cross-org clone + kubeconfig

GitHub injects a per-job GITHUB_TOKEN scoped to the running repo; with
permissions: packages: write it can push the repo's own GHCR package (same-org),
so the push path needs no stored credential. KMS is now best-effort, supplying
only what the automatic token can't: a cross-org PAT to clone other orgs' private
Go modules (luxfi/dex, luxfi/precompile) + KUBECONFIG for deploy. Public repos
with no cross-org deps build with zero KMS dependency.
2026-06-24 10:43:03 -07:00
Hanzo DevandGitHub 572c712140 ci: private Go module access (luxfi/hanzoai/zooai), guarded by GH_PAT (#1)
* ci: configure private Go module access (luxfi/hanzoai/zooai)

Any go-based `test:` step in a consumer's hanzo.yml needs to fetch
private github.com/{luxfi,hanzoai,zooai}/* modules; without auth go
hits 'git ls-remote … exit 128'. Add a guarded step that wires GH_PAT
via git insteadOf + sets GOPRIVATE job-wide. No-op when GH_PAT is
absent, so non-Go consumers (bootnode etc.) are unaffected.

* ci: make private-module auth runner-state-immune (fresh GIT_CONFIG_GLOBAL)

Upgrade the naive 'git config --global' to the robust pattern proven in
hanzoai/iam: fresh per-job GIT_CONFIG_GLOBAL + GIT_CONFIG_NOSYSTEM=1, probe
from a neutral dir. A plain global config is overridden on shared arc
runners by stale ~/.gitconfig state and actions/checkout's persisted
extraheader. Still a no-op without GH_PAT.
2026-06-22 17:21:04 -07:00
7 changed files with 40 additions and 1255 deletions
@@ -28,21 +28,9 @@ on:
inherit).
type: string
default: buildx
tests:
description: >-
Run the `test:` block. Default true, and a caller should leave it that
way. false asserts that THIS EXACT COMMIT was already gated before this
run — it does not mean "ship untested". The one shape that holds today
is hanzoai/cloud's release: clients/platform/release.go mints the v* tag
only after that SHA passed the gate on main AND built AND smoked, so the
tag build would re-test a commit already proven, at the cost of a
3108-package link storm. Passing false there runs the gate once instead
of twice; passing it anywhere else runs it zero times.
type: boolean
default: true
permissions:
contents: write # release assets for `binaries:`; read is enough without it
contents: read
packages: write
jobs:
@@ -168,23 +156,14 @@ jobs:
&& echo "git auth OK" \
|| echo "::warning::GH_PAT set but repo probe failed"
- name: Log in to GHCR (GH_PAT when present, else automatic token)
- name: Log in to GHCR (automatic workflow token)
if: inputs.mode != 'delegate'
env:
GH_PAT: ${{ secrets.GH_PAT }}
# Prefer the KMS-backed GH_PAT (admin:org + write:packages): it can push
# or CREATE any <org> package regardless of which repo the package is
# linked to. The per-job GITHUB_TOKEN only writes a package linked to THIS
# repo, so it 403s on a package created/linked elsewhere (e.g.
# ghcr.io/hanzoai/cms). Fall back to the automatic token when GH_PAT is
# absent (public forks like zooai/node that create their own repo-linked
# package on first push).
run: |
if [ -n "${GH_PAT:-}" ]; then
echo "$GH_PAT" | docker login ghcr.io -u hanzo-dev --password-stdin
else
echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
fi
# GHCR push needs no stored credential: GitHub injects a per-job
# GITHUB_TOKEN scoped to THIS repo, and `permissions: packages: write`
# (above) lets it push the repo's own package (e.g. zooai/node →
# ghcr.io/zooai/node, same org). First push creates the package
# repo-linked. No KMS, no PAT for the push path.
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
- name: Fetch deploy credentials from KMS
id: kms
@@ -249,70 +228,8 @@ jobs:
if [ -n "$GIT_TOKEN" ]; then echo "::add-mask::$GIT_TOKEN"; echo "GIT_TOKEN=$GIT_TOKEN" >> "$GITHUB_ENV"; fi
KUBECONFIG_B64=$(get KUBECONFIG)
if [ -n "$KUBECONFIG_B64" ]; then echo "$KUBECONFIG_B64" | base64 -d > "$RUNNER_TEMP/kubeconfig"; echo "kubeconfig=$RUNNER_TEMP/kubeconfig" >> "$GITHUB_OUTPUT"; fi
# Object-store credential for a repo that publishes `binaries:` to a
# bucket:. Same four names the services read at run time, from the same
# org/path/env — a CI-only copy of a credential is a second thing to
# rotate and the one nobody remembers.
if [ -n "$(yq -r '.bucket // ""' hanzo.yml 2>/dev/null || true)" ]; then
for k in S3_ADMIN_ACCESS_KEY S3_ADMIN_SECRET_KEY; do
v=$(get "$k")
if [ -n "$v" ]; then echo "::add-mask::$v"; echo "$k=$v" >> "$GITHUB_ENV"; fi
done
# Endpoint and region are NOT masked: both appear in every published
# URL, and a redacted host turns the printed index into noise.
for k in S3_PUBLIC_ENDPOINT S3_REGION S3_PUBLIC_SECURE; do
v=$(get "$k")
if [ -n "$v" ]; then echo "$k=$v" >> "$GITHUB_ENV"; fi
done
fi
# Build-time secrets → --build-arg. A repo declares per-image
# `build_secrets: [NAME, ...]` in hanzo.yml; each NAME is fetched from
# the SAME org/path/env and exported (masked) so the build step bakes
# it in as `--build-arg NAME=value`. The KMS key name IS the build-arg
# name — one name, one place. For publishable client tokens a Vite SPA
# must embed at build (e.g. VITE_MAPBOX_TOKEN). Undeclared → no-op, so
# every existing repo is byte-for-byte unchanged.
for bs in $(yq -r '[(.images // [])[] | (.build_secrets // [])[]] | unique | .[]' hanzo.yml 2>/dev/null || true); do
case "$bs" in ''|*[!A-Za-z0-9_]*) echo "::warning::skipping invalid build_secret name '$bs'"; continue;; esac
v=$(get "$bs" || true)
if [ -n "$v" ]; then
echo "::add-mask::$v"
{ echo "$bs<<__KMS_BUILDARG_EOF__"; echo "$v"; echo "__KMS_BUILDARG_EOF__"; } >> "$GITHUB_ENV"
else echo "::warning::build_secret $bs not in KMS ($ORG/$PATHQ env=$ENV) — build-arg will be empty"; fi
done
- name: GHCR push credential (KMS write:packages token)
if: inputs.mode != 'delegate'
env:
KUBECONFIG: ${{ steps.kms.outputs.kubeconfig }}
# The per-job GITHUB_TOKEN — and a package-less GH_PAT — can only push a
# package LINKED to this repo, so they 403 on a package created/linked
# elsewhere (ghcr.io/hanzoai/cms). Upgrade the ghcr login to the org's
# ghcr push token from the cluster (buildx-ghcr-auth, admin write:packages,
# KMS-synced) when a kubeconfig is present — it pushes/creates ANY <org>
# package. Fail-safe: keep the earlier login if anything is unavailable
# (public forks with no KMS keep their repo-linked GITHUB_TOKEN push).
run: |
set -uo pipefail
[ -z "${KUBECONFIG:-}" ] && { echo "::notice::no kubeconfig — keeping the earlier GHCR login"; exit 0; }
command -v kubectl >/dev/null 2>&1 || {
KVER=$(curl -fsSL https://dl.k8s.io/release/stable.txt)
mkdir -p "$HOME/.local/bin"
curl -fsSL "https://dl.k8s.io/release/${KVER}/bin/linux/amd64/kubectl" -o "$HOME/.local/bin/kubectl" && chmod +x "$HOME/.local/bin/kubectl"
export PATH="$HOME/.local/bin:$PATH"
}
CFG=$(kubectl -n hanzo get secret buildx-ghcr-auth -o jsonpath='{.data.\.dockerconfigjson}' 2>/dev/null | base64 -d || true)
[ -z "$CFG" ] && { echo "::notice::buildx-ghcr-auth not readable — keeping the earlier GHCR login"; exit 0; }
UP=$(echo "$CFG" | jq -r '.auths | to_entries[] | select(.key|test("ghcr")) | .value.auth' | head -1 | base64 -d 2>/dev/null || true)
[ -z "$UP" ] && { echo "::notice::no ghcr auth in buildx-ghcr-auth — keeping the earlier login"; exit 0; }
echo "::add-mask::${UP#*:}"
if echo "${UP#*:}" | docker login ghcr.io -u "${UP%%:*}" --password-stdin; then
echo "::notice::GHCR login upgraded to the KMS write:packages token"
else
echo "::notice::KMS GHCR token login failed — keeping the earlier login"
fi
- name: Native registry credential (oci.hanzo.ai)
- name: Mirror credential (registry.hanzo.ai)
if: inputs.mode != 'delegate'
env:
KUBECONFIG: ${{ steps.kms.outputs.kubeconfig }}
@@ -328,14 +245,14 @@ jobs:
# cred) must never fail the run — the image still pushes to GHCR, the
# primary. Without this guard, bash -e aborts the step and SKIPS the
# build entirely (a registry hiccup takes the whole lane red).
if echo "$REGISTRY_PASSWORD" | docker login oci.hanzo.ai -u "$REGISTRY_USER" --password-stdin; then
if echo "$REGISTRY_PASSWORD" | docker login registry.hanzo.ai -u "$REGISTRY_USER" --password-stdin; then
echo "MIRROR_OK=1" >> "$GITHUB_ENV"
else
echo "::notice::registry.hanzo.ai login failed — mirror skipped (GHCR-only push)"
fi
exit 0
fi
[ -z "${KUBECONFIG:-}" ] && { echo "::warning::no registry credential and no kubeconfig — image will NOT reach oci.hanzo.ai (ghcr only)"; exit 0; }
[ -z "${KUBECONFIG:-}" ] && { echo "::notice::no registry secret and no kubeconfig — mirror skipped"; exit 0; }
# Bare arc runners ship no kubectl — same static provision the deploy
# step uses.
command -v kubectl >/dev/null 2>&1 || {
@@ -346,13 +263,13 @@ jobs:
}
CFG=$(kubectl -n hanzo get secret registry-credentials -o jsonpath='{.data.\.dockerconfigjson}' 2>/dev/null | base64 -d || true)
if [ -z "$CFG" ]; then
echo "::warning::registry-credentials not readable from this kubeconfig — image will NOT reach oci.hanzo.ai (ghcr only)"; exit 0
echo "::notice::registry-credentials not readable from this kubeconfig — mirror skipped (GHCR-only push)"; exit 0
fi
USERPASS=$(echo "$CFG" | jq -r '.auths["registry.hanzo.ai"].auth // empty' | base64 -d)
[ -z "$USERPASS" ] && { echo "::warning::no registry auth in dockerconfig — image will NOT reach oci.hanzo.ai (ghcr only)"; exit 0; }
[ -z "$USERPASS" ] && { echo "::notice::no registry.hanzo.ai auth in dockerconfig — mirror skipped"; exit 0; }
echo "::add-mask::${USERPASS#*:}"
# Best-effort: login failure → skip mirror, never fail the run (see above).
if echo "${USERPASS#*:}" | docker login oci.hanzo.ai -u "${USERPASS%%:*}" --password-stdin; then
if echo "${USERPASS#*:}" | docker login registry.hanzo.ai -u "${USERPASS%%:*}" --password-stdin; then
echo "MIRROR_OK=1" >> "$GITHUB_ENV"
else
echo "::notice::registry.hanzo.ai login failed — mirror skipped (GHCR-only push)"
@@ -381,8 +298,7 @@ jobs:
export GIT_TOKEN="${GIT_TOKEN:-${GH_PAT:-}}"
SHORT=$(echo "${{ github.sha }}" | cut -c1-7)
IS_TAG=$([ "${{ github.ref_type }}" = "tag" ] && echo 1 || echo 0)
REL="${{ github.ref_name }}" # the git tag, verbatim: v1.26.19
VER="${REL#v}" # v-stripped alias: 1.26.19
VER="${{ github.ref_name }}"; VER="${VER#v}"
yq -o=json -I=0 '.images' hanzo.yml | jq -c '.[]' | while read -r img; do
name=$(echo "$img"|jq -r .name); ctx=$(echo "$img"|jq -r .context)
df=$(echo "$img"|jq -r '.dockerfile // (.context+"/Dockerfile")'); repo=$(echo "$img"|jq -r .repo)
@@ -403,44 +319,25 @@ jobs:
if [ "$plats" = "linux/amd64" ]; then
# single-arch: keep the exact legacy tag shape (-amd64) deploys expect.
TAGS="-t $repo:sha-${SHORT}-amd64${sfx:+-$sfx} -t $repo:${sfx:+$sfx-}latest"
# Release (tag) build: publish the CANONICAL bare semver tag (what the
# universe CR pins to — matches world 2.4.10 / cloud v1.801.62) AND the
# legacy -amd64 semver alias (back-compat for CRs still on that shape).
[ "$IS_TAG" = 1 ] && TAGS="$TAGS -t $repo:${VER}${sfx:+-$sfx} -t $repo:${VER}-amd64${sfx:+-$sfx}"
else
# multi-arch: one arch-neutral manifest-list tag (no -amd64 suffix).
docker run --privileged --rm tonistiigi/binfmt --install arm64 >/dev/null 2>&1 || true
TAGS="-t $repo:sha-${SHORT}${sfx:+-$sfx} -t $repo:${sfx:+$sfx-}latest"
fi
# Release (tag) build. The git tag IS the release name, so publish it
# VERBATIM (v1.26.19) — that is the shape a universe CR pins, and
# stripping the v is why releases were finished by hand-`crane copy`ing
# sha-<sha7> onto the semver a human typed. Identity in, identity out.
# The v-stripped alias stays for CRs already pinned that way (world
# 2.4.51), and is skipped when a repo tags without a v. The old
# `<ver>-amd64` alias is deleted: no CR in the fleet pinned it.
if [ "$IS_TAG" = 1 ]; then
TAGS="$TAGS -t $repo:${REL}${sfx:+-$sfx}"
[ "$REL" != "$VER" ] && TAGS="$TAGS -t $repo:${VER}${sfx:+-$sfx}"
[ "$IS_TAG" = 1 ] && TAGS="$TAGS -t $repo:${VER}${sfx:+-$sfx}"
fi
echo "::group::build $name → $repo (${sfx}) [$plats]"
# --build-arg assembly: static hanzo.yml `args` (a fixed value, e.g. a
# pinned base image tag) + `build_secrets` (KMS values the KMS step
# exported into the env above). Empty when a repo declares neither, so
# the buildx line is unchanged for every existing repo.
BUILD_ARGS=""
while IFS= read -r kv; do [ -n "$kv" ] && BUILD_ARGS="$BUILD_ARGS --build-arg $kv"; done \
< <(echo "$img" | jq -r '(.args // {}) | to_entries[] | "\(.key)=\(.value)"')
for bs in $(echo "$img" | jq -r '(.build_secrets // [])[]'); do
v=$(printenv "$bs" 2>/dev/null || true); [ -n "$v" ] && BUILD_ARGS="$BUILD_ARGS --build-arg $bs=$v"
done
# GIT_TOKEN (from KMS, via GITHUB_ENV) is passed as the `gh_token`
# BuildKit secret so Dockerfiles can clone private Go modules; omitted
# cleanly when absent (public-only builds unaffected).
docker buildx build --platform "$plats" $BUILD_ARGS ${GIT_TOKEN:+--secret id=gh_token,env=GIT_TOKEN} --push $TAGS -f "$df" "$ctx"
docker buildx build --platform "$plats" ${GIT_TOKEN:+--secret id=gh_token,env=GIT_TOKEN} --push $TAGS -f "$df" "$ctx"
# Dual-host: mirror the exact tag set to registry.hanzo.ai (server-
# side manifest copy — no rebuild). ghcr.io/<org>/<name> →
# oci.hanzo.ai/<org>/<name>; public consumers keep ghcr, the fleet
# is migrating to pull from ours. A skip here is now a WARNING, not
# a notice: an image that never reaches our registry is the reason
# a deploy still depends on GitHub, and that should be visible in
# the run, not buried.
# registry.hanzo.ai/<org>/<name>; public consumers keep ghcr, the
# fleet DEPLOYS from our registry.
if [ "${MIRROR_OK:-}" = "1" ]; then
# crane, not buildx imagetools: the IAM token realm doesn't answer
# buildx's multi-scope token request (spec gap, tracked).
@@ -450,10 +347,10 @@ jobs:
| tar -xz -C "$HOME/.local/bin" crane
export PATH="$HOME/.local/bin:$PATH"
}
mrepo="oci.hanzo.ai/${repo#*/}"
mrepo="registry.hanzo.ai/${repo#*/}"
echo "$TAGS" | tr ' ' '\n' | grep -v '^-t$' | grep -v '^$' | while read -r ref; do
crane copy "$ref" "${mrepo}:${ref##*:}" \
|| echo "::warning::$ref did not reach oci.hanzo.ai (ghcr push unaffected)"
|| echo "::warning::mirror of $ref failed (ghcr push unaffected)"
done
fi
echo "::endgroup::"
@@ -530,7 +427,7 @@ jobs:
echo "runner git authenticated for private Go modules"
- name: Test (per hanzo.yml)
if: inputs.mode != 'delegate' && inputs.tests
if: inputs.mode != 'delegate'
run: |
set -euo pipefail
yq -o=json -I=0 '.test // []' hanzo.yml | jq -c '.[]' | while read -r t; do
@@ -538,188 +435,6 @@ jobs:
echo "::group::test $name"; bash -c "$cmd"; echo "::endgroup::"
done
- name: Build & publish binaries (per hanzo.yml)
# The plugin lane. `images:` ships an OCI image a CLUSTER runs;
# `binaries:` ships an EXECUTABLE a RUNNING HOST installs — a zip plugin
# (`zip.Load(zip.Plugin{URL, Sum})`), which fetches the artifact,
# verifies its SHA-256 BEFORE making it executable, and caches it by
# digest. So a plugin is built ONCE per OS/arch and every host picks up
# the same bits: nobody rebuilds the world to ship a plugin, and no host
# is trusted to have built it right.
#
# It rides THIS pipeline, not a second one — same hanzo.yml, same job,
# same runner, same KMS token — and adds two rules of its own:
# - BUILT on every push, PUBLISHED only on a tag. A cross-compile that
# breaks arm64 fails the PR that broke it, not the release.
# - published AFTER the bits are GATED, because a host installs an
# artifact unattended. Normally that gate is the `test:` step above,
# in this same run. A caller passing `tests: false` asserts it ran
# EARLIER on this exact commit: hanzoai/cloud mints its v* tag only
# after that SHA passed the gate on main and built and smoked
# (clients/platform/release.go), so the tag build would re-test a
# commit already proven. The invariant did not weaken — it is
# enforced once instead of twice, and the assertion is the input.
# (Images push before tests; an image is rolled out by a deliberate,
# reviewed pin — an artifact is not.)
# dist/binaries.json is uploaded with them: url + sha256 for every
# artifact, so the bits and the digest that authorizes them are the same
# release, and a host reads url+sum from ONE place.
if: inputs.mode != 'delegate'
env:
GH_PAT: ${{ secrets.GH_PAT }}
GH_AUTOMATIC: ${{ secrets.GITHUB_TOKEN }}
run: |
set -euo pipefail
if [ "$(yq -r '.binaries // [] | length' hanzo.yml 2>/dev/null || echo 0)" = "0" ]; then
echo "::notice::no binaries: in hanzo.yml — nothing to publish"; exit 0
fi
TAG="${{ github.ref_name }}"; REPO="${{ github.repository }}"
IS_TAG=$([ "${{ github.ref_type }}" = "tag" ] && echo 1 || echo 0)
# The runner tells us which forge it is talking to — github.com on the
# arc pool, a GHES/forge front door elsewhere. Reading it instead of
# hardcoding the hostname is what keeps ONE lane serving both.
SERVER="${GITHUB_SERVER_URL:-https://github.com}"
API="${GITHUB_API_URL:-https://api.github.com}"
case "$API" in
https://api.github.com) UPLOADS=https://uploads.github.com ;;
*) UPLOADS="$SERVER/api/uploads" ;;
esac
# A repo naming a bucket: publishes to hanzoai/s3 instead of a release.
# Same artifacts, same index, one url per artifact — only where it
# points changes, and it is decided HERE because the url is baked into
# binaries.json at build time.
BUCKET="$(yq -r '.bucket // ""' hanzo.yml 2>/dev/null || true)"
if [ -n "$BUCKET" ]; then
# Host and scheme are the two clients/s3admin reads, so CI publishes
# to the same URL the services resolve.
SCHEME=https
if [ "${S3_PUBLIC_SECURE:-true}" = "false" ]; then SCHEME=http; fi
BASE="$SCHEME://${S3_PUBLIC_ENDPOINT:-s3.hanzo.ai}/$BUCKET/$REPO/$TAG"
else
BASE="$SERVER/$REPO/releases/download/$TAG"
fi
rm -rf dist; mkdir -p dist
# CGO_ENABLED=0 is not a preference: the host that installs this runs
# it on WHATEVER base image the host happens to be, so a binary linked
# against that runner's glibc is a plugin that starts on the runner and
# nowhere else. -trimpath keeps the digest a function of the source,
# not of the checkout path.
yq -o=json -I=0 '.binaries' hanzo.yml | jq -c '.[]' | while read -r b; do
name=$(echo "$b"|jq -r .name); main=$(echo "$b"|jq -r '.main // ""')
lf=$(echo "$b"|jq -r '.ldflags // "-s -w"')
run=$(echo "$b"|jq -r '.run // ""'); out=$(echo "$b"|jq -r '.out // ""')
# `run:` + `out:` is the SAME lane for every toolchain that is not Go:
# the command that builds, and the glob of what it produced. It is why
# a repo with no Dockerfile and no Go can still publish an artifact —
# an npm tarball, a wheel, a Rust binary — through this one block.
# (`image:` also appears on such an entry: it names the toolchain
# container the PLATFORM lane runs the command in. Here the toolchain
# is the runner, so this lane reads past it.)
if [ -n "$run" ]; then
[ -n "$out" ] || { echo "::error::binaries[$name] declares run: without out: — nothing names what it produced"; exit 1; }
echo "::group::build $name ($run)"
sh -c "$run"
n=0
for f in $out; do
[ -f "$f" ] || continue
base=$(basename "$f"); cp "$f" "dist/$base"
sha=$(sha256sum "dist/$base" | cut -d' ' -f1)
# os/arch are "any": an npm tarball or a wheel is not per-platform,
# and an index entry that claimed one would be a lie a host acts on.
jq -nc --arg n "$name" --arg os any --arg arch any \
--arg url "$BASE/$base" --arg sha "$sha" \
'{name:$n,os:$os,arch:$arch,url:$url,sha256:$sha}' >> dist/index.jsonl
echo "$base $sha"; n=$((n+1))
done
echo "::endgroup::"
[ "$n" -gt 0 ] || { echo "::error::binaries[$name] out: '$out' matched no file the recipe produced"; exit 1; }
continue
fi
main="${main:-.}"
echo "$b" | jq -r '(.platforms // ["linux/amd64"])[]' | while read -r plat; do
os="${plat%%/*}"; arch="${plat##*/}"
file="${name}-${os}-${arch}"; [ "$os" = windows ] && file="${file}.exe"
echo "::group::build $name $os/$arch"
CGO_ENABLED=0 GOOS="$os" GOARCH="$arch" \
go build -trimpath -ldflags "$lf" -o "dist/$file" "$main"
sha=$(sha256sum "dist/$file" | cut -d' ' -f1)
jq -nc --arg n "$name" --arg os "$os" --arg arch "$arch" \
--arg url "$BASE/$file" --arg sha "$sha" \
'{name:$n,os:$os,arch:$arch,url:$url,sha256:$sha}' >> dist/index.jsonl
echo "$file $sha"
echo "::endgroup::"
done
done
jq -s --arg repo "$REPO" --arg tag "$TAG" \
'{repo:$repo,tag:$tag,binaries:.}' dist/index.jsonl > dist/binaries.json
rm dist/index.jsonl
if [ "$IS_TAG" != 1 ]; then
echo "::notice::binaries built for $(jq '.binaries|length' dist/binaries.json) platform(s) — publishing happens on a tag"; exit 0
fi
if [ -n "$BUCKET" ]; then
# Fail closed. An index published without its artifacts is a host
# that resolves every app to a 404, which is worse than a host with
# no index at all.
if [ -z "${S3_ADMIN_ACCESS_KEY:-}" ] || [ -z "${S3_ADMIN_SECRET_KEY:-}" ]; then
echo "::error::hanzo.yml names bucket: $BUCKET but KMS holds no S3_ADMIN_ACCESS_KEY/SECRET_KEY for this org — refusing to publish"; exit 1
fi
# PUT is idempotent, so re-running a release converges on the object
# rather than colliding with the one already there.
put() { curl -fsS -X PUT --aws-sigv4 "aws:amz:${S3_REGION:-us-east-1}:s3" \
-u "$S3_ADMIN_ACCESS_KEY:$S3_ADMIN_SECRET_KEY" \
-H 'Content-Type: application/octet-stream' \
--data-binary @"$1" "$BASE/$(basename "$1")" >/dev/null
echo "published $BASE/$(basename "$1")"; }
# Artifacts first, index LAST. The index is the only file a host
# reads, so between the two writes it must never name an object that
# is not there yet.
for f in dist/*; do
[ "$f" = dist/binaries.json ] || put "$f"
done
put dist/binaries.json
# A host reads the index with NO credentials, so prove one can
# before calling this published — a private object is an index that
# resolves every app to a 403. Granting read is the BUCKET's job
# (one policy, once), which is why this checks rather than sets it.
code=$(curl -s -o /dev/null -w '%{http_code}' "$BASE/binaries.json")
if [ "$code" != 200 ]; then
echo "::error::published, but $BASE/binaries.json answers $code unauthenticated — grant s3:GetObject on $BUCKET/* to Principal \"*\" with a bucket policy"; exit 1
fi
echo "### Plugin artifacts — \`$TAG\`" >> "$GITHUB_STEP_SUMMARY"
echo "Index: \`$BASE/binaries.json\`" >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
# Same token ladder the rest of this file uses: the KMS org token, the
# org PAT, then the per-job automatic one (which can write a release on
# its OWN repo, so a repo with no KMS still publishes).
TOKEN="${GIT_TOKEN:-${GH_PAT:-$GH_AUTOMATIC}}"
api() { curl -fsS -H "Authorization: Bearer $TOKEN" -H 'Accept: application/vnd.github+json' "$@"; }
rel=$(api "$API/repos/$REPO/releases/tags/$TAG" 2>/dev/null | jq -r '.id // empty' || true)
if [ -z "$rel" ]; then
rel=$(api -X POST "$API/repos/$REPO/releases" \
-d "$(jq -nc --arg t "$TAG" '{tag_name:$t,name:$t,generate_release_notes:true}')" | jq -r .id)
fi
for f in dist/*; do
n=$(basename "$f")
# Re-running a release must converge, not 422: drop a same-named
# asset first. The digest in binaries.json is regenerated with it,
# so the pair can never disagree.
old=$(api "$API/repos/$REPO/releases/$rel/assets" | jq -r --arg n "$n" '.[]?|select(.name==$n)|.id')
for id in $old; do api -X DELETE "$API/repos/$REPO/releases/assets/$id" >/dev/null; done
api -X POST -H 'Content-Type: application/octet-stream' --data-binary @"$f" \
"$UPLOADS/repos/$REPO/releases/$rel/assets?name=$n" >/dev/null
echo "published $BASE/$n"
done
# The job prints what a host pastes. A digest a human retypes is a
# digest a human gets wrong.
{
echo "### Plugin artifacts — \`$TAG\`"
echo '```go'
jq -r '.binaries[]|"zip.Load(zip.Plugin{\n Name: \"\(.name)\", // \(.os)/\(.arch)\n URL: \"\(.url)\",\n Sum: \"\(.sha256)\",\n}, \"/v1/\(.name)\")"' dist/binaries.json
echo '```'
echo "Index: \`$BASE/binaries.json\`"
} >> "$GITHUB_STEP_SUMMARY"
- name: Deploy (per hanzo.yml)
if: inputs.mode != 'delegate' && github.event_name != 'pull_request' && steps.kms.outputs.kubeconfig != ''
env:
@@ -727,17 +442,9 @@ jobs:
run: |
set -euo pipefail
REF="${{ github.ref_name }}"
# A caller with no `deploy:` declares its rollout elsewhere (hanzoai/git
# pins its own CR by a reviewed universe change). Leave before reading
# deploy.services, which is null there — `jq '.[]'` over null aborts the
# step, and on a TAG build the deploy.on gate below is bypassed, so this
# used to surface only on a release. Same guard the build step has for
# `images:`.
if [ "$(yq -r '.deploy | type' hanzo.yml 2>/dev/null || echo '!!null')" != '!!map' ]; then
echo "::notice::no deploy: in hanzo.yml — build-only caller, skipping deploy"; exit 0
fi
ON=$(yq -o=json -I=0 '.deploy.on // []' hanzo.yml)
IS_TAG=$([ "${{ github.ref_type }}" = "tag" ] && echo 1 || echo 0)
VER="${REF#v}" # canonical semver for tag releases (ref_name without the v)
if [ "$IS_TAG" != 1 ] && ! echo "$ON" | jq -e --arg b "$REF" 'index($b)' >/dev/null; then
echo "branch $REF not in deploy.on — skipping deploy"; exit 0
fi
@@ -770,11 +477,11 @@ jobs:
repo=$(jq -r --arg n "$imgname" '.[]|select(.name==$n)|.repo' /tmp/imgs.json)
sfx=$(jq -r --arg n "$imgname" '.[]|select(.name==$n)|."tag-suffix" // ""' /tmp/imgs.json)
plats=$(jq -r --arg n "$imgname" '.[]|select(.name==$n)|(.platforms // ["linux/amd64"])|join(",")' /tmp/imgs.json)
# A TAGGED release pins the git tag VERBATIM (REF) — the very image the
# build step published above; a branch build stays on its transient
# per-commit sha tag (arch-matched: bare for multi-arch).
# A TAGGED release pins the CANONICAL bare semver (VER = ref_name w/o the
# v) — the very image the build step published above; a branch build stays
# on its transient per-commit sha tag (arch-matched: bare for multi-arch).
if [ "$IS_TAG" = 1 ]; then
tag="${REF}${sfx:+-$sfx}"
tag="${VER}${sfx:+-$sfx}"
elif [ "$plats" = "linux/amd64" ]; then
tag="sha-${SHORT}-amd64${sfx:+-$sfx}"
else
@@ -792,11 +499,6 @@ jobs:
echo "::notice::$svc pinned to release $cur — branch build ${SHORT} leaves it (tag a release to deploy)"; continue
fi
echo "rolling $svc → $ref"
# recorded=1 once the desired tag is durably in universe: the
# in-cluster operator reconciles that every ~5min, so an expired
# runner kubeconfig must not fail a deploy it will complete. Stays 0
# for bare-Deployment repos (no CR) where kubectl is the only path.
recorded=0
if [ -n "$UNIVERSE" ] && [ -f "$CR" ]; then
# Never roll a pin BACKWARD: builds finish out of order, and a slow
# build of an older commit must not overwrite a newer roll. If the
@@ -807,7 +509,6 @@ jobs:
&& git cat-file -e "$CURSHA" 2>/dev/null \
&& git merge-base --is-ancestor "$SHORT" "$CURSHA" 2>/dev/null; then
echo "::notice::$svc CR already at descendant $CURSHA — not rolling back to $SHORT"
recorded=1 # newer release already recorded + reconciling
else
yq -i ".spec.image.tag = \"$tag\"" "$CR"
# Sweep EVERY same-repo image reference in the CR (sidecars,
@@ -815,48 +516,18 @@ jobs:
# left sidecars on stale tags every roll.
repoEsc=$(printf '%s' "$repo" | sed 's/[.[\*^$]/\\&/g')
sed -i -E "s|(${repoEsc}):sha-[A-Za-z0-9-]+|\1:${tag}|g" "$CR"
if git -C "$UNIVERSE" diff --quiet; then
recorded=1 # CR already pins $tag (recorded on a prior run)
else
commitCR() { git -C "$UNIVERSE" -c user.name=hanzo-ci -c user.email=dev@hanzo.ai \
commit -qam "deploy($svc): $tag (${GITHUB_REPOSITORY}@${SHORT})"; }
commitCR
# EVERY service's deploy pushes to universe/main, so under
# concurrent rolls a plain push loses the race with a
# non-fast-forward reject (THE reason deploys stall when many
# land at once). The clone is shallow (--depth 1) so `pull
# --rebase` has no merge base and fails — instead fetch the moved
# tip, hard-reset onto it, and re-apply our one-file CR change,
# then retry the push. A no-op after reset (remote already
# carries our tag) counts as recorded. We never force-push, so a
# newer roll of THIS service is preserved (its tag survives the
# reset; our re-apply is last-writer only when tags differ).
for _try in 1 2 3 4 5 6; do
if git -C "$UNIVERSE" push -q; then recorded=1; break; fi
git -C "$UNIVERSE" fetch -q --depth 1 origin main || break
git -C "$UNIVERSE" reset -q --hard FETCH_HEAD
yq -i ".spec.image.tag = \"$tag\"" "$CR"
sed -i -E "s|(${repoEsc}):sha-[A-Za-z0-9-]+|\1:${tag}|g" "$CR"
if git -C "$UNIVERSE" diff --quiet; then recorded=1; break; fi
commitCR
done
[ "$recorded" = 1 ] || echo "::warning::universe push failed for $svc after retries — roll is transient (runner kubectl must land it)"
fi
git -C "$UNIVERSE" diff --quiet || {
git -C "$UNIVERSE" -c user.name=hanzo-ci -c user.email=dev@hanzo.ai \
commit -qam "deploy($svc): $tag (${GITHUB_REPOSITORY}@${SHORT})"
git -C "$UNIVERSE" push -q \
|| echo "::warning::universe push failed for $svc — roll is transient"
}
fi
fi
# Runner-side kubectl ACCELERATES the operator roll + smoke-tests it.
# When recorded=1 the operator owns the rollout, so an expired runner
# kubeconfig warns instead of failing the deploy; when recorded=0
# kubectl is the only path and stays fatal.
if [ "$recorded" = 1 ]; then
kc() { kubectl "$@" || echo "::warning::$svc: runner kubectl failed (expired kubeconfig?) — universe record stands; operator reconciles"; }
else
kc() { kubectl "$@"; }
fi
# Operator-managed services (Service CR) reconcile the Deployment —
# patch the CR when it exists; bare Deployments get set-image.
if kubectl -n "$NS" get "services.hanzo.ai/$svc" >/dev/null 2>&1; then
kc -n "$NS" patch "services.hanzo.ai/$svc" --type=merge \
kubectl -n "$NS" patch "services.hanzo.ai/$svc" --type=merge \
-p "{\"spec\":{\"image\":{\"repository\":\"$repo\",\"tag\":\"$tag\"}}}"
else
# Bare Deployment: set the new image ONLY on containers already running
@@ -868,12 +539,12 @@ jobs:
| awk -v r="$repo" 'index($2, r"@")==1 || index($2, r":")==1 {print $1}')
[ "${#CS[@]}" -eq 0 ] && CS=("$svc")
args=(); for c in "${CS[@]}"; do args+=("$c=$ref"); done
kc -n "$NS" set image "deployment/$svc" "${args[@]}"
kubectl -n "$NS" set image "deployment/$svc" "${args[@]}"
fi
# Timeout scales with the image: a service vendoring GB-scale model/node
# packs (e.g. studio) pulls multi-GB layers on a cold node + waits for the
# operator to reconcile — legitimately minutes. 180s failed mid-pull and
# is THE reason studio deploys silently failed since 0.15.8. Override
# per-repo with deploy.rollout-timeout.
kc -n "$NS" rollout status "deployment/$svc" --timeout="$RTO"
kubectl -n "$NS" rollout status "deployment/$svc" --timeout="$RTO"
done
-78
View File
@@ -1,78 +0,0 @@
name: Sync from GitHub
# git.hanzo.ai is CANONICAL and builds natively; development also lands on
# github.com/hanzoai/ci. This job carries GitHub's commits to the forge.
#
# WITHOUT IT THIS REPO DRIFTS, and it had: the forge sat 3 commits behind main
# with nothing to close the gap. ci is one of only 13 non-mirror repos in the
# hanzoai org, and non-mirror is exactly the gap between the two mechanisms that
# keep the forge current:
#
# hanzoai/mirrors reconcile.py — makes qualifying GitHub repos pull-mirrors on
# a 6h schedule, and the server then syncs them. It is idempotent and
# "existing mirrors are left alone", so a repo that already exists as a
# NON-mirror is skipped forever.
# this file — the per-repo pull, for repos that are native-canonical and so
# must not be mirrors.
#
# A non-mirror repo without this file has neither, and drifts silently. Sibling
# repos that carry it sit 0-1 commits behind; the two that lacked it, ci and
# cloud, sat 3 and 10 behind.
#
# ONE deterministic direction: an in-cluster PULL. The runner reaches both ends,
# so the sync has no ingress dependency and nothing has to push INTO the forge
# from outside — which matters, because GitHub org secrets are inert on this
# Free-plan org and a GitHub-side push has no working credential.
#
# NO CREDENTIAL. hanzoai/ci is public, so the fetch is anonymous. Private
# siblings need GH_PAT from the forge's own secret store; this one must never
# grow a token it has no use for.
#
# Fast-forward ONLY. A divergence fails LOUDLY rather than force-pushing either
# side and destroying whichever history lost the race.
on:
schedule:
- cron: '*/10 * * * *'
workflow_dispatch: {}
concurrency:
group: sync-from-github
cancel-in-progress: false
jobs:
ff-main:
runs-on: [hanzo-build-linux-amd64]
steps:
- name: Checkout main (full history for the ancestry check)
uses: actions/checkout@v4
with:
fetch-depth: 0
persist-credentials: true
- name: Fast-forward main from github.com/hanzoai/ci
run: |
set -euo pipefail
git fetch --quiet https://github.com/hanzoai/ci.git main
LOCAL="$(git rev-parse HEAD)"
REMOTE="$(git rev-parse FETCH_HEAD)"
if [ "$LOCAL" = "$REMOTE" ]; then
echo "in sync at $LOCAL"
exit 0
fi
if git merge-base --is-ancestor "$LOCAL" "$REMOTE"; then
echo "fast-forwarding $LOCAL -> $REMOTE"
git push origin "$REMOTE:refs/heads/main"
# A push made with the workflow token does NOT trigger other workflows
# (loop prevention), so synced commits would never build. Dispatch it
# explicitly — a real fast-forward means real commits arrived.
curl -fsS --max-time 20 -X POST \
-H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
-H "Content-Type: application/json" \
"${{ github.server_url }}/v1/repos/${{ github.repository }}/actions/workflows/build.yml/dispatches" \
-d '{"ref":"main"}' || echo "build dispatch failed (non-fatal)"
else
echo "DIVERGED: native $LOCAL is not an ancestor of GitHub $REMOTE." >&2
echo "Resolve by hand; this job will not force-push either side." >&2
exit 1
fi
-31
View File
@@ -1,31 +0,0 @@
# syntax=docker/dockerfile:1
#
# ci — the ci.hanzo.ai dashboard. Pure-Go, no cgo, no assets: the page is
# server-rendered from a template compiled into the binary, so the image is the
# binary and a CA bundle. Nothing to serve from disk, nothing to go stale
# against the code.
FROM golang:1.24-alpine AS builder
WORKDIR /build
# Resolve through the module proxy: proxy.golang.org and sum.golang.org agree
# and neither can change under us, which a direct fetch against a moved tag
# cannot promise.
ENV GOPROXY=https://proxy.golang.org,direct
COPY go.mod ./
RUN --mount=type=cache,id=ci-gomod,target=/go/pkg/mod go mod download
COPY . .
RUN --mount=type=cache,id=ci-gomod,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /build/ci .
FROM alpine:3.21
RUN apk add --no-cache ca-certificates tzdata \
&& addgroup -S hanzo && adduser -S hanzo -G hanzo
COPY --from=builder /build/ci /app/ci
USER hanzo
EXPOSE 8080
# Liveness only. Readiness deliberately does not gate on having a snapshot — see
# the /healthz comment in main.go: a Hanzo Git outage must render as a dashboard
# saying so, not as this pod leaving the load balancer as well.
HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --retries=3 \
CMD wget -qO- http://127.0.0.1:8080/healthz || exit 1
ENTRYPOINT ["/app/ci"]
-68
View File
@@ -40,74 +40,6 @@ jobs:
That's it. The build/test/deploy logic lives here, once.
## `binaries:` — publish a plugin once, install it everywhere
`images:` ships an OCI image a **cluster** runs. `binaries:` ships an
executable a **running host** installs: a [zip](https://github.com/zap-proto/zip)
plugin, fetched at run time by URL and verified against its SHA-256 before it is
ever made executable. Build it once per OS/arch here; every host picks up the
same bits, and nobody rebuilds the world to ship a plugin.
```yaml
binaries:
- name: billing
main: ./cmd/billing # the Go package; default "."
platforms: [linux/amd64, linux/arm64] # default [linux/amd64]
ldflags: "-s -w" # default
```
`main:` is the zero-config **Go** lane. Every other toolchain uses the same block
with `run:` (the command that builds) and `out:` (the glob of what it produced) —
which is how a repo with no Dockerfile and no Go still publishes an artifact:
```yaml
binaries:
- name: sdk
run: npm install && npm run build && npm pack --pack-destination .
out: "*.tgz"
image: node:22-bookworm # the toolchain — see below
```
`image:` names the container the **platform** lane runs `run:` in
(`POST /v1/runner`, one initContainer per entry, in-cluster). Here the toolchain
IS the runner, so this workflow reads past it. It is not a second recipe: both
lanes read the same `binaries:` block out of the same `hanzo.yml` and publish the
same `binaries.json` at the same URL.
Artifacts land under `<name>` in the index regardless of lane; a `run:` entry is
`os: any, arch: any`, because an npm tarball or a wheel is not per-platform and
an index entry that claimed one would be a lie a host acts on.
Built on every push (an arm64 cross-compile that breaks fails the PR that broke
it) and **published on a tag**, after the `test:` gate — a host installs an
artifact unattended, so the tests gate the bits. Each artifact lands on the
GitHub Release for that tag:
```
https://github.com/<owner>/<repo>/releases/download/<tag>/<name>-<os>-<arch>
```
plus `binaries.json` beside them — `{name, os, arch, url, sha256}` for every
artifact, so the bits and the digest that authorizes them ship as one release
and a host reads both from one place. The job summary prints the
`zip.Load(zip.Plugin{URL, Sum})` a host pastes.
Add a top-level `bucket:` and they publish to **hanzoai/s3** instead — same
artifacts, same index, only the url changes:
```yaml
bucket: plugins # → https://s3.hanzo.ai/plugins/<owner>/<repo>/<tag>/binaries.json
```
Credentials are the `S3_ADMIN_*` names the services already read, pulled from
KMS at run time; a declared bucket with no credential fails the publish rather
than shipping an index whose artifacts are missing. Use it for anything large or
frequent — a GitHub release stores it on a quota we do not own.
Builds are `CGO_ENABLED=0 -trimpath`: the host that installs this runs it on
whatever base image the host is, and the digest must be a function of the
source, not of the checkout path.
## Runners — our cloud or your own
By default the build runs on the **Hanzo cloud** arc pool (we run it; metered as
-3
View File
@@ -1,3 +0,0 @@
module github.com/hanzoai/ci
go 1.24
-481
View File
@@ -1,481 +0,0 @@
// ci — the dashboard behind ci.hanzo.ai.
//
// It owns no build state. Run truth lives in Hanzo Git (git.hanzo.ai), which
// schedules the jobs and holds every log; this reads that and presents it. The
// alternative — a CI service with its own run database — would put two answers
// to "did the build pass" in the fleet, and the one users look at would be the
// one that can drift. So: git.hanzo.ai is the store, ci.hanzo.ai is the view.
//
// This is the CI half of the pair. cd.hanzo.ai reconciles image pins from
// hanzoai/universe and is the delivery view; the two are deliberately separate
// surfaces over separate systems, not one console pretending build and deploy
// are the same event.
//
// Tenancy is the same value everywhere: an org slug. Hanzo Git namespaces repos
// by org, IAM issues that slug in the `owner` claim, and Hanzo CD fences
// projects by it. Filtering here by `org` is therefore the same boundary those
// enforce, not a parallel notion of who-sees-what.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"net/http"
"net/url"
"os"
"os/signal"
"sort"
"strconv"
"strings"
"sync"
"syscall"
"time"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stderr, nil))
cfg, err := loadConfig()
if err != nil {
logger.Error("config", "err", err)
os.Exit(1)
}
src := &gitSource{base: cfg.gitBase, token: cfg.gitToken, http: &http.Client{Timeout: 20 * time.Second}}
cache := &runCache{}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
// One poller, one cache. Every viewer reads the same snapshot, so N open
// dashboards cost Hanzo Git exactly as much as one — a dashboard that
// fanned each page load into upstream calls is how a status page takes the
// system it reports on down.
go poll(ctx, logger, src, cache, cfg)
mux := http.NewServeMux()
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
// Liveness only: up means "serving". Readiness deliberately does NOT
// gate on having a snapshot — a Hanzo Git outage must show as a stale
// dashboard saying so, not as ci.hanzo.ai disappearing from the LB too.
writeJSON(w, http.StatusOK, map[string]any{"status": "ok"})
})
mux.HandleFunc("/v1/runs", func(w http.ResponseWriter, r *http.Request) {
snap := cache.get()
writeJSON(w, http.StatusOK, map[string]any{
"runs": filterByOrg(snap.Runs, r.URL.Query().Get("org")),
"fetchedAt": snap.FetchedAt,
"stale": snap.stale(cfg.staleAfter),
"sourceErr": snap.errString(),
"repos": snap.Repos,
"orgs": orgsOf(snap.Runs),
})
})
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/" {
http.NotFound(w, r)
return
}
renderDashboard(w, cache.get(), r.URL.Query().Get("org"), cfg)
})
srv := &http.Server{
Addr: cfg.listen,
Handler: mux,
ReadHeaderTimeout: 10 * time.Second,
}
go func() {
<-ctx.Done()
sh, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
_ = srv.Shutdown(sh)
}()
logger.Info("ci dashboard listening", "addr", cfg.listen, "source", cfg.gitBase, "refresh", cfg.refresh.String())
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
logger.Error("serve", "err", err)
os.Exit(1)
}
}
// ───────────────────────────── config ─────────────────────────────
type config struct {
listen string
gitBase string
gitToken string
refresh time.Duration
staleAfter time.Duration
scanRepos int
runsPer int
}
func loadConfig() (config, error) {
c := config{
listen: env("CI_LISTEN", ":8080"),
gitBase: strings.TrimRight(env("CI_GIT_BASE", "https://git.hanzo.ai"), "/"),
gitToken: os.Getenv("CI_GIT_TOKEN"),
scanRepos: envInt("CI_SCAN_REPOS", 60),
runsPer: envInt("CI_RUNS_PER_REPO", 8),
}
c.refresh = time.Duration(envInt("CI_REFRESH_SECONDS", 45)) * time.Second
// Stale is a multiple of refresh, not its own knob: the only meaningful
// definition of stale is "we have missed several refreshes", and deriving
// it means the two can never be configured into contradiction.
c.staleAfter = 4 * c.refresh
if c.gitToken == "" {
return c, errors.New("CI_GIT_TOKEN required (Hanzo Git API token)")
}
return c, nil
}
func env(k, def string) string {
if v := os.Getenv(k); v != "" {
return v
}
return def
}
func envInt(k string, def int) int {
if v, err := strconv.Atoi(os.Getenv(k)); err == nil && v > 0 {
return v
}
return def
}
// ───────────────────────────── model ─────────────────────────────
// Run is the projection of a Hanzo Git workflow run this dashboard shows. It is
// deliberately a SUBSET: the upstream object carries a dozen more fields, and
// copying them all would make this a second schema to maintain against theirs.
type Run struct {
ID int64 `json:"id"`
Org string `json:"org"`
Repo string `json:"repo"`
Workflow string `json:"workflow"`
Title string `json:"title"`
// Status and Conclusion are BOTH required to know how a run went, and
// reading only one is wrong in a way that looks fine. Status answers
// "is it over" (queued | in_progress | completed); Conclusion answers
// "how did it end" and is empty until it is over. A view that buckets on
// Status alone sees `completed` and cannot tell a pass from a failure —
// which is exactly the bug this pair replaced: every finished run,
// including successes and cancellations, was being drawn as failing.
Status string `json:"status"`
Conclusion string `json:"conclusion"`
Event string `json:"event"`
Branch string `json:"branch"`
SHA string `json:"sha"`
Actor string `json:"actor"`
Number int `json:"number"`
URL string `json:"url"`
StartedAt time.Time `json:"startedAt"`
EndedAt time.Time `json:"endedAt"`
}
// Duration is zero-valued rather than negative when a run has not finished —
// callers render "running", and a negative duration would print as one.
func (r Run) Duration() time.Duration {
if r.StartedAt.IsZero() || r.EndedAt.IsZero() || r.EndedAt.Before(r.StartedAt) {
return 0
}
return r.EndedAt.Sub(r.StartedAt)
}
type snapshot struct {
Runs []Run `json:"runs"`
Repos int `json:"repos"`
FetchedAt time.Time `json:"fetchedAt"`
Err error `json:"-"`
}
func (s snapshot) stale(after time.Duration) bool {
return s.FetchedAt.IsZero() || time.Since(s.FetchedAt) > after
}
func (s snapshot) errString() string {
if s.Err == nil {
return ""
}
return s.Err.Error()
}
type runCache struct {
mu sync.RWMutex
snap snapshot
}
func (c *runCache) get() snapshot {
c.mu.RLock()
defer c.mu.RUnlock()
return c.snap
}
// put keeps the LAST GOOD run list when a refresh fails, recording the error
// alongside it. A failed poll must not blank the dashboard: "Hanzo Git is
// unreachable, here is what we last saw" is strictly more useful than an empty
// page, which reads as "nothing is building".
func (c *runCache) put(s snapshot) {
c.mu.Lock()
defer c.mu.Unlock()
if s.Err != nil && len(s.Runs) == 0 && len(c.snap.Runs) > 0 {
prev := c.snap
prev.Err = s.Err
c.snap = prev
return
}
c.snap = s
}
// ───────────────────────────── source ─────────────────────────────
type gitSource struct {
base string
token string
http *http.Client
}
func (g *gitSource) getJSON(ctx context.Context, path string, out any) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, g.base+path, nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "token "+g.token)
req.Header.Set("Accept", "application/json")
resp, err := g.http.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("%s: %s", path, resp.Status)
}
return json.NewDecoder(resp.Body).Decode(out)
}
type repoRef struct {
FullName string `json:"full_name"`
}
// repos returns the most recently ACTIVE repositories. Sorting by activity and
// taking a window is the whole scan strategy: the instance mirrors ~1400 repos
// and almost none of them built in the last hour, so walking all of them would
// spend the entire refresh budget confirming silence.
func (g *gitSource) repos(ctx context.Context, limit int) ([]string, error) {
var body struct {
Data []repoRef `json:"data"`
}
q := url.Values{}
q.Set("sort", "updated")
q.Set("order", "desc")
q.Set("limit", strconv.Itoa(limit))
if err := g.getJSON(ctx, "/v1/repos/search?"+q.Encode(), &body); err != nil {
return nil, err
}
names := make([]string, 0, len(body.Data))
for _, r := range body.Data {
if r.FullName != "" {
names = append(names, r.FullName)
}
}
return names, nil
}
type apiRun struct {
ID int64 `json:"id"`
DisplayTitle string `json:"display_title"`
Path string `json:"path"`
Event string `json:"event"`
Status string `json:"status"`
Conclusion string `json:"conclusion"`
HeadBranch string `json:"head_branch"`
HeadSHA string `json:"head_sha"`
RunNumber int `json:"run_number"`
HTMLURL string `json:"html_url"`
StartedAt string `json:"started_at"`
CompletedAt string `json:"completed_at"`
Actor struct {
Login string `json:"login"`
} `json:"actor"`
}
func (g *gitSource) runs(ctx context.Context, fullName string, limit int) ([]Run, error) {
var body struct {
WorkflowRuns []apiRun `json:"workflow_runs"`
}
path := fmt.Sprintf("/v1/repos/%s/actions/runs?limit=%d", fullName, limit)
if err := g.getJSON(ctx, path, &body); err != nil {
return nil, err
}
org, repo := splitFullName(fullName)
out := make([]Run, 0, len(body.WorkflowRuns))
for _, r := range body.WorkflowRuns {
out = append(out, Run{
ID: r.ID,
Org: org,
Repo: repo,
Workflow: workflowOf(r.Path),
Title: r.DisplayTitle,
Status: r.Status,
Conclusion: r.Conclusion,
Event: r.Event,
Branch: r.HeadBranch,
SHA: shortSHA(r.HeadSHA),
Actor: r.Actor.Login,
Number: r.RunNumber,
URL: r.HTMLURL,
StartedAt: parseTime(r.StartedAt),
EndedAt: parseTime(r.CompletedAt),
})
}
return out, nil
}
// poll refreshes the snapshot on an interval, forever.
func poll(ctx context.Context, logger *slog.Logger, src *gitSource, cache *runCache, cfg config) {
refresh := func() {
rctx, cancel := context.WithTimeout(ctx, 90*time.Second)
defer cancel()
names, err := src.repos(rctx, cfg.scanRepos)
if err != nil {
logger.Warn("repo scan failed", "err", err)
cache.put(snapshot{FetchedAt: time.Now().UTC(), Err: err})
return
}
// Fan out, bounded. The cap is small on purpose: this is a read against
// the forge that schedules every build in the fleet, and a dashboard is
// never worth degrading it.
const workers = 6
var (
mu sync.Mutex
all []Run
errs []string
wg sync.WaitGroup
)
jobs := make(chan string)
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for name := range jobs {
rs, err := src.runs(rctx, name, cfg.runsPer)
mu.Lock()
if err != nil {
// A repo with Actions disabled 404s. That is normal and
// not worth surfacing as a dashboard-level failure, so
// it is counted, not shown.
errs = append(errs, name)
} else {
all = append(all, rs...)
}
mu.Unlock()
}
}()
}
for _, n := range names {
select {
case jobs <- n:
case <-rctx.Done():
}
}
close(jobs)
wg.Wait()
sort.Slice(all, func(i, j int) bool { return all[i].StartedAt.After(all[j].StartedAt) })
cache.put(snapshot{Runs: all, Repos: len(names) - len(errs), FetchedAt: time.Now().UTC()})
logger.Info("refreshed", "repos", len(names), "withRuns", len(names)-len(errs), "runs", len(all))
}
refresh()
t := time.NewTicker(cfg.refresh)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
refresh()
}
}
}
// ───────────────────────────── helpers ─────────────────────────────
func splitFullName(s string) (org, repo string) {
if i := strings.IndexByte(s, '/'); i > 0 {
return s[:i], s[i+1:]
}
return "", s
}
// workflowOf reduces "e2e.yml@refs/heads/main" to "e2e.yml".
func workflowOf(path string) string {
if i := strings.IndexByte(path, '@'); i > 0 {
return path[:i]
}
return path
}
func shortSHA(s string) string {
if len(s) > 7 {
return s[:7]
}
return s
}
func parseTime(s string) time.Time {
if s == "" {
return time.Time{}
}
t, err := time.Parse(time.RFC3339, s)
if err != nil {
return time.Time{}
}
// Hanzo Git reports an unset timestamp as the Unix epoch rather than null;
// treated as absent so the UI shows "—" instead of 1970.
if t.Year() < 2000 {
return time.Time{}
}
return t.UTC()
}
func filterByOrg(runs []Run, org string) []Run {
if org == "" {
return runs
}
out := make([]Run, 0, len(runs))
for _, r := range runs {
if r.Org == org {
out = append(out, r)
}
}
return out
}
func orgsOf(runs []Run) []string {
seen := map[string]bool{}
for _, r := range runs {
if r.Org != "" {
seen[r.Org] = true
}
}
out := make([]string, 0, len(seen))
for o := range seen {
out = append(out, o)
}
sort.Strings(out)
return out
}
func writeJSON(w http.ResponseWriter, status int, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(v)
}
-225
View File
@@ -1,225 +0,0 @@
package main
import (
"fmt"
"html/template"
"net/http"
"strings"
"time"
)
// render.go — the HTML view. Server-rendered on purpose: this page is a table
// of build results, and a client-side app would ship a bundle, a fetch layer
// and a loading state to show the same rows a second later. The dashboard also
// has to be readable when the thing it reports on is broken, which is exactly
// when a build pipeline for its own frontend is the wrong dependency.
func renderDashboard(w http.ResponseWriter, snap snapshot, org string, cfg config) {
runs := filterByOrg(snap.Runs, org)
if len(runs) > 200 {
runs = runs[:200]
}
data := struct {
Runs []Run
Orgs []string
Org string
Repos int
FetchedAt time.Time
Age string
Stale bool
SourceErr string
Source string
Counts map[string]int
}{
Runs: runs,
Orgs: orgsOf(snap.Runs),
Org: org,
Repos: snap.Repos,
FetchedAt: snap.FetchedAt,
Age: humanAge(snap.FetchedAt),
Stale: snap.stale(cfg.staleAfter),
SourceErr: snap.errString(),
Source: cfg.gitBase,
Counts: countByOutcome(runs),
}
w.Header().Set("Content-Type", "text/html; charset=utf-8")
if err := tmpl.Execute(w, data); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
// countByOutcome buckets runs for the summary strip.
func countByOutcome(runs []Run) map[string]int {
c := map[string]int{"success": 0, "failure": 0, "running": 0, "cancelled": 0}
for _, r := range runs {
c[outcome(r)]++
}
return c
}
// outcome collapses (status, conclusion) into the four states worth a colour.
//
// Status alone is NOT enough and getting this wrong is silent: Hanzo Git
// reports every finished run as `completed` regardless of how it went, so
// bucketing on status painted successes and cancellations as failures — on the
// live instance that was 15 of 20 runs mislabelled red.
//
// `cancelled` gets its own bucket rather than folding into failure. On this
// fleet cancellations are the single largest category (superseded pushes cancel
// the in-flight run), and a board that shows them as broken is a board nobody
// trusts, which is worse than no board.
func outcome(r Run) string {
if !strings.EqualFold(r.Status, "completed") {
return "running" // queued | in_progress | waiting | blocked
}
switch strings.ToLower(r.Conclusion) {
case "success":
return "success"
case "cancelled", "canceled", "skipped":
return "cancelled"
case "":
// Completed with no conclusion should not happen; if it does, say
// "running" rather than inventing a verdict the data does not support.
return "running"
default:
return "failure" // failure | timed_out | action_required
}
}
func humanAge(t time.Time) string {
if t.IsZero() {
return "never"
}
d := time.Since(t)
switch {
case d < time.Minute:
return fmt.Sprintf("%ds ago", int(d.Seconds()))
case d < time.Hour:
return fmt.Sprintf("%dm ago", int(d.Minutes()))
default:
return fmt.Sprintf("%dh ago", int(d.Hours()))
}
}
func humanDur(d time.Duration) string {
if d <= 0 {
return "—"
}
if d < time.Minute {
return fmt.Sprintf("%ds", int(d.Seconds()))
}
return fmt.Sprintf("%dm%02ds", int(d.Minutes()), int(d.Seconds())%60)
}
var tmpl = template.Must(template.New("ci").Funcs(template.FuncMap{
"outcome": outcome,
"dur": func(r Run) string { return humanDur(r.Duration()) },
"ago": humanAge,
}).Parse(`<!doctype html>
<html lang="en"><head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Hanzo CI</title>
<meta http-equiv="refresh" content="60">
<style>
:root{--bg:#0b0b0d;--panel:#141417;--line:#25252b;--fg:#e8e8ea;--dim:#8b8b95;
--ok:#3fb950;--fail:#f85149;--run:#d29922;--accent:#8B5CF6}
*{box-sizing:border-box}
body{margin:0;background:var(--bg);color:var(--fg);
font:14px/1.5 ui-sans-serif,-apple-system,"Segoe UI",Roboto,sans-serif}
header{display:flex;align-items:center;gap:16px;padding:16px 24px;
border-bottom:1px solid var(--line);background:var(--panel)}
h1{margin:0;font-size:16px;font-weight:600;letter-spacing:-.01em}
h1 span{color:var(--accent)}
.meta{margin-left:auto;color:var(--dim);font-size:12px;text-align:right}
.strip{display:flex;gap:8px;padding:16px 24px;flex-wrap:wrap}
.chip{padding:6px 12px;border:1px solid var(--line);border-radius:8px;
background:var(--panel);font-size:12px;color:var(--dim)}
.chip b{color:var(--fg);font-weight:600}
.chip.ok b{color:var(--ok)} .chip.fail b{color:var(--fail)} .chip.run b{color:var(--run)}
.chip.cancel b{color:var(--dim)}
nav{display:flex;gap:6px;padding:0 24px 16px;flex-wrap:wrap}
nav a{padding:5px 11px;border:1px solid var(--line);border-radius:999px;
background:var(--panel);color:var(--dim);text-decoration:none;font-size:12px}
nav a.on{border-color:var(--accent);color:var(--fg)}
.warn{margin:0 24px 16px;padding:10px 14px;border:1px solid var(--run);
border-radius:8px;background:#221b0c;color:#f0d58c;font-size:13px}
table{width:100%;border-collapse:collapse}
th{position:sticky;top:0;background:var(--panel);text-align:left;font-size:11px;
text-transform:uppercase;letter-spacing:.06em;color:var(--dim);
padding:10px 12px;border-bottom:1px solid var(--line);font-weight:600}
td{padding:10px 12px;border-bottom:1px solid var(--line);vertical-align:top}
tr:hover td{background:#111114}
a{color:inherit}
.dot{display:inline-block;width:8px;height:8px;border-radius:50%;margin-right:8px}
.dot.success{background:var(--ok)} .dot.failure{background:var(--fail)}
.dot.running{background:var(--run);animation:p 1.4s ease-in-out infinite}
.dot.cancelled{background:#4a4a52}
@keyframes p{50%{opacity:.35}}
.repo{font-weight:600}
.org{color:var(--dim)}
.title{color:var(--dim);max-width:42ch;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
.mono{font-family:ui-monospace,SFMono-Regular,Menlo,monospace;font-size:12px;color:var(--dim)}
.empty{padding:48px 24px;text-align:center;color:var(--dim)}
footer{padding:16px 24px;color:var(--dim);font-size:12px;border-top:1px solid var(--line)}
@media(max-width:760px){.hide-sm{display:none}}
</style></head><body>
<header>
<h1>Hanzo <span>CI</span></h1>
<div class="meta">
{{.Repos}} repos &middot; refreshed {{.Age}}<br>
source {{.Source}}
</div>
</header>
<div class="strip">
<span class="chip ok">passing <b>{{index .Counts "success"}}</b></span>
<span class="chip fail">failing <b>{{index .Counts "failure"}}</b></span>
<span class="chip run">running <b>{{index .Counts "running"}}</b></span>
<span class="chip cancel">cancelled <b>{{index .Counts "cancelled"}}</b></span>
</div>
<nav>
<a href="/" {{if eq .Org ""}}class="on"{{end}}>all orgs</a>
{{range .Orgs}}<a href="/?org={{.}}" {{if eq $.Org .}}class="on"{{end}}>{{.}}</a>{{end}}
</nav>
{{if .Stale}}<div class="warn">
Snapshot is stale — last successful refresh {{.Age}}.
{{if .SourceErr}}Hanzo Git said: {{.SourceErr}}{{else}}Hanzo Git is not answering.{{end}}
These rows are the last good read, not current state.
</div>{{end}}
{{if .Runs}}
<table>
<thead><tr>
<th>Repository</th><th>Workflow</th><th class="hide-sm">Commit</th>
<th class="hide-sm">Actor</th><th>Started</th><th>Took</th>
</tr></thead>
<tbody>
{{range .Runs}}
<tr>
<td><span class="dot {{outcome .}}"></span><a href="{{.URL}}"><span class="org">{{.Org}}/</span><span class="repo">{{.Repo}}</span></a></td>
<td>{{.Workflow}} <span class="mono">#{{.Number}}</span><div class="title">{{.Title}}</div></td>
<td class="hide-sm mono">{{.Branch}}@{{.SHA}}<div>{{.Event}}</div></td>
<td class="hide-sm mono">{{.Actor}}</td>
<td class="mono">{{ago .StartedAt}}</td>
<td class="mono">{{dur .}}</td>
</tr>
{{end}}
</tbody></table>
{{else}}
<div class="empty">
No runs in the scanned window.<br>
<span class="mono">Builds land here from {{.Source}} — this view holds no state of its own.</span>
</div>
{{end}}
<footer>
Build truth lives in Hanzo Git; this is a view over it.
Delivery is <a href="https://cd.hanzo.ai">cd.hanzo.ai</a>.
</footer>
</body></html>`))