Red flagged NewCircuitBreaker's unbounded context.Background() filer read as a latent
CrashLoopBackOff: on a K8s cold/co-restart the s3 gateway dials the filer Service (TCP
accepted by kube endpoints) before the filer answers RPCs, the unbounded read blocks
~4min, :8333 never binds, liveness kills the pod. Confirmed via goroutine dump; the fix
is a bounded context (zap-proto muxConn.CallContext honors ctx.Done()).
The startup path has THREE reads of this anti-pattern, all of which block :8333 — fixing
only the circuit breaker leaves the gateway hanging at the first one:
- command/s3.go GetFilerConfiguration: REQUIRED read, already in a for{}+Sleep(1s) retry
loop, but the unbounded ctx defeats the retry (first call hangs forever). Bound each
attempt (10s) so the retry loop actually retries until the filer answers.
- s3api/s3api_circuit_breaker.go: OPTIONAL read. Bound (10s); the existing fail-open path
(circuit breaker disabled) runs on deadline.
- s3api/bucket_metadata.go BucketRegistry.init: OPTIONAL warm-up whose error path was
glog.Fatal (itself a CrashLoop trigger). Bound (10s) + warn-and-proceed instead of
Fatal; the registry warms lazily via GetBucketMetadata -> LoadBucketMetadataFromFiler.
Empirically verified on a local single-node cluster: the bounded ctx is honored
(muxConn.CallContext returns "context deadline exceeded" at 10s), :8333 binds (~10s vs an
indefinite hang before), and a SigV4 create-bucket / PUT / GET (byte-identical) / DELETE
round-trip succeeds. GOWORK=off go build ./s3/... = 0; s3api bucket registry/metadata
tests pass. Keeps the fail-open intent; only bounds the wait.
go mod tidy (GOWORK=off) removes go.mod/go.sum entries no longer referenced by
any package, under any build tag:
- github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.1 — unimported (grep all .go = 0)
- github.com/luxfi/container v0.0.4 — unimported (grep all .go = 0)
- modernc.org/sqlite marked // indirect (pulled transitively via hanzoai/sqlite;
no direct import remains)
Default build and full-tag build (elastic gocdk sqlite ydb tarantool tikv rclone)
both green. go.sum: the checksums for the two dropped modules are removed.
NOTE: github.com/hanzos3/go-sdk is deliberately NOT dropped — `go mod why` shows
it is a required transitive dep of github.com/luxfi/zapdb (the filer zapdb
metadata store) via s3/filer/luxdb; tidy keeps it.
- remove stress: a 32MB Mach-O arm64 (macOS) binary committed to the repo root,
unreferenced by any build/CI/script — an accidentally-committed build artifact.
- LLM.md: the gRPC-rip section listed volume RPC, raft, cert hot-reload and the
core dial/server as still grpc-bound; all were since ported to ZAP. The main
module now has zero google.golang.org/grpc imports (grep -rl google.golang.org/grpc s3/ = 0).
Record the true residual: google.golang.org/grpc // indirect is pulled only by
the optional Google Cloud client family (kms/pubsub/storage apiv*), which are
grpc-transport SDKs never dialed in a no-GCP deployment; zeroing it is a
CTO-gated GCP-surface rip, not an RPC-transport change.
* fix(sqlite): use hanzoai/sqlite (the one driver), drop direct modernc import
* fix(sqlite): use hanzoai/sqlite (the one driver), drop direct modernc import
---------
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The public s3 repo called a reusable workflow in the PRIVATE hanzoai/.github repo
(uses: hanzoai/.github/.github/workflows/docker-build.yml@main). GitHub forbids a
public repo from calling a private repo's reusable workflow — it resolves as
'workflow was not found', so every docker-build failed at 0s (not a runner or YAML
issue; the shared file and org access were both fine).
Replace with a self-contained build that:
- pushes to ghcr.io/hanzoai/s3 with GITHUB_TOKEN (no custom registry secret, no
cross-repo reusable dependency),
- builds linux/amd64 + linux/arm64 (was amd64-only) via buildx/QEMU — the Dockerfile
is pure-Go (CGO_ENABLED=0) so it cross-builds cleanly,
- runs on the self-hosted amd64 runner (evo).
Co-authored-by: Hanzo AI <ai@hanzo.ai>
* fix: correct crude seaweedfs->hanzo rename artifacts + drop legacy minio
- s3 update: GitHubLatestRelease("hanzo","hanzo") -> ("hanzoai","s3"); the
blunt rename pointed updates at a nonexistent repo (help text already said
hanzoai/s3). Now 's3 update' pulls real Hanzo S3 releases. (go build clean.)
- LLM.md: fix upstream attribution (was github.com/hanzo/hanzo) to credit the real
SeaweedFS upstream, matching NOTICE; correct 'renamed from weed'; note goexif +
go-fuse are the only external deps still under the seaweedfs org.
- LLM.md: add S-Chain section cross-linking datastore (OLAP) + the S-Chain storage
role (reverse of hanzoai/datastore's link), proven-vs-direction scoped.
- drop legacy docker/compose/local-minio-gateway-compose.yml (unreferenced MinIO-
gateway demo).
main is already ZAP-native + internally renamed; these are residual-artifact fixes.
* fix: correct seaweedfs->hanzo rename artifacts + datastore cross-link
- s3 update: GitHubLatestRelease("hanzo","hanzo") -> ("hanzoai","s3") so updates
pull real Hanzo S3 releases (help text already said hanzoai/s3). go build clean.
- LLM.md: correct upstream attribution to SeaweedFS (matches NOTICE); note goexif +
go-fuse are the only remaining seaweedfs-org deps; add S-Chain datastore cross-link.
---------
Co-authored-by: Hanzo AI <ai@hanzo.ai>
LICENSE-attested or repo-attested upstream is now called out at the
top of LLM.md alongside the project description, so downstream
audits (universe/docs/PRODUCTS.md) can rely on a single canonical
location for the OSS lineage statement.
Required for hanzoai/.github/.github/workflows/docker-build.yml@main —
without it the workflow_call dies as startup_failure with no jobs
dispatched. Caller permissions are a CEILING.
Replaces all ghcr.io/hanzoai/<svc>:latest pins with the latest published
semver tag for each service. Per CLAUDE.md auto-bump policy: mutable
branch tags (:latest, :main, :dev) are deprecated for cluster pins —
only immutable semver permitted.
Bulk update across services with published v* tags. Services without a
published semver remain on :latest until their release pipeline cuts a
v* tag.
Trigger builds on main, test, dev branches. Add type=ref,event=branch
and type=ref,event=pr to metadata-action tags so branch names map to
image tags automatically.
Drop vendor-specific prefix from internal S3 storage headers.
30 files across cmd/ and internal/ — all header constants,
metadata keys, replication headers, and test fixtures updated.
- Update Dockerfile.hanzo build stage to golang:1.26-bookworm
- Update go.mod toolchain to go1.26.1
- Update all 12 CI workflow files to use go-version 1.26.x
- Change s3ReservedBucket constant from "minio" to "s3" so health
endpoints register at /s3/health/* instead of /minio/health/*
- Update Prometheus metrics namespace from "minio" to "s3"
- Update API response DisplayName from "minio" to "s3"
- Update release binary name and download URL references
- Multi-stage Dockerfile.hanzo with distroless base for CI builds
- Update publish workflow with build-args for version injection
- Update .gitignore for new binary naming convention
- 780+ files touched by automated rebrand script
- Override EnvPrefix to "S3_" instead of importing MINIO_ from madmin-go
- Add EnvLegacyPrefix "MINIO_" constant for backwards compat
- Add MINIO_ fallback in ResolveConfigParam for subsystem env vars
- Add MINIO_ROOT_USER/PASSWORD fallback in credential loading
- Add MINIO_ access key fallback in credential loading
- Skip legacy MINIO_ vars in bootstrap peer sync
Register health check handlers at both /minio/health/* (legacy) and /health/*
(short path) so /health/live and /health/ready work without the minio prefix.
- Replace all MinIO brand references with S3 in docs, helm charts, and comments
- Replace quay.io/minio images with ghcr.io/hanzoai/s3 and ghcr.io/hanzos3/*
- Replace min.io domains: docs→docs.hanzo.ai, dl→dl.hanzo.space, play→play.hanzo.space
- Replace slack.min.io with hanzo.ai/discord
- Replace subnet.min.io with subnet.hanzo.ai
- Update Makefile REPO to ghcr.io/hanzoai
- Update helm values: mc image, OIDC clientId, service account name
- Update Go source user-facing URLs and messages
- Fix test configs and grafana dashboard descriptions
- Both docker and binaries jobs now run on [self-hosted, hanzo-build]
- Build runner deployed to hanzo-k8s (actions-runner-system namespace)
- Buildx uses docker-container driver (no DinD needed)
- Publish workflow now pushes to both ghcr.io/hanzoai/s3 and ghcr.io/hanzoai/storage
- Trigger on main branch (not master)
- Build linux/amd64 + linux/arm64
- Dockerfile label points to github.com/hanzoai/s3
- publish.yml: push to both ghcr.io/hanzoai/s3 and ghcr.io/hanzoai/storage
- publish.yml: build linux/darwin/windows binaries on tag push
- workers/dl/: Cloudflare Worker to proxy GitHub release assets at dl.hanzo.space
Complete rename of all environment variable references from MINIO_
prefix to S3_ prefix across the entire codebase:
- internal/config/constants.go: EnvPrefix and all Env* constants
- All Go source files: string literals, error messages, comments
- Prometheus metrics: minio_* → s3_* metric name prefix
- Dockerfiles: ENV declarations
- Build scripts, CI workflows, helm charts, docs
- Grafana dashboards: metric queries updated
Go import paths and package names unchanged.
Build verified: go build ./cmd/... passes clean.
Hanzo Dev