s3's SQLite filer already imports github.com/hanzoai/sqlite on main. v0.3.2 is
self-contained (vendored pure-Go engine), so modernc.org/{sqlite,libc,memory,
mathutil} are no longer needed and are dropped from go.mod (verified: go build ./...
stays green, cgo + nocgo, without them). csqlite + sqlcipher (v0.3.2's cgo backend)
enter as indirect.
modernc removal is done via go mod edit rather than go mod tidy: a pre-existing,
unrelated go.sum re-tag drift on github.com/luxfi/pq@v1.0.3 blocks full-graph ops
on this branch's base (origin/main) — out of scope for the sqlite migration.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Red flagged NewCircuitBreaker's unbounded context.Background() filer read as a latent
CrashLoopBackOff: on a K8s cold/co-restart the s3 gateway dials the filer Service (TCP
accepted by kube endpoints) before the filer answers RPCs, the unbounded read blocks
~4min, :8333 never binds, liveness kills the pod. Confirmed via goroutine dump; the fix
is a bounded context (zap-proto muxConn.CallContext honors ctx.Done()).
The startup path has THREE reads of this anti-pattern, all of which block :8333 — fixing
only the circuit breaker leaves the gateway hanging at the first one:
- command/s3.go GetFilerConfiguration: REQUIRED read, already in a for{}+Sleep(1s) retry
loop, but the unbounded ctx defeats the retry (first call hangs forever). Bound each
attempt (10s) so the retry loop actually retries until the filer answers.
- s3api/s3api_circuit_breaker.go: OPTIONAL read. Bound (10s); the existing fail-open path
(circuit breaker disabled) runs on deadline.
- s3api/bucket_metadata.go BucketRegistry.init: OPTIONAL warm-up whose error path was
glog.Fatal (itself a CrashLoop trigger). Bound (10s) + warn-and-proceed instead of
Fatal; the registry warms lazily via GetBucketMetadata -> LoadBucketMetadataFromFiler.
Empirically verified on a local single-node cluster: the bounded ctx is honored
(muxConn.CallContext returns "context deadline exceeded" at 10s), :8333 binds (~10s vs an
indefinite hang before), and a SigV4 create-bucket / PUT / GET (byte-identical) / DELETE
round-trip succeeds. GOWORK=off go build ./s3/... = 0; s3api bucket registry/metadata
tests pass. Keeps the fail-open intent; only bounds the wait.
Remove a 32MB macOS Mach-O binary junk-committed at repo root, 2 unused
indirect deps, and correct the stale gRPC section in LLM.md. Zero Go source
change. Red-verified: grpc-free ZAP stubs kept load-bearing, grpc//indirect
structural to the GCP client family, full-tag build green, go-sdk retained.
go mod tidy (GOWORK=off) removes go.mod/go.sum entries no longer referenced by
any package, under any build tag:
- github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.1 — unimported (grep all .go = 0)
- github.com/luxfi/container v0.0.4 — unimported (grep all .go = 0)
- modernc.org/sqlite marked // indirect (pulled transitively via hanzoai/sqlite;
no direct import remains)
Default build and full-tag build (elastic gocdk sqlite ydb tarantool tikv rclone)
both green. go.sum: the checksums for the two dropped modules are removed.
NOTE: github.com/hanzos3/go-sdk is deliberately NOT dropped — `go mod why` shows
it is a required transitive dep of github.com/luxfi/zapdb (the filer zapdb
metadata store) via s3/filer/luxdb; tidy keeps it.
- remove stress: a 32MB Mach-O arm64 (macOS) binary committed to the repo root,
unreferenced by any build/CI/script — an accidentally-committed build artifact.
- LLM.md: the gRPC-rip section listed volume RPC, raft, cert hot-reload and the
core dial/server as still grpc-bound; all were since ported to ZAP. The main
module now has zero google.golang.org/grpc imports (grep -rl google.golang.org/grpc s3/ = 0).
Record the true residual: google.golang.org/grpc // indirect is pulled only by
the optional Google Cloud client family (kms/pubsub/storage apiv*), which are
grpc-transport SDKs never dialed in a no-GCP deployment; zeroing it is a
CTO-gated GCP-surface rip, not an RPC-transport change.
* fix(sqlite): use hanzoai/sqlite (the one driver), drop direct modernc import
* fix(sqlite): use hanzoai/sqlite (the one driver), drop direct modernc import
---------
Co-authored-by: Hanzo AI <ai@hanzo.ai>
The public s3 repo called a reusable workflow in the PRIVATE hanzoai/.github repo
(uses: hanzoai/.github/.github/workflows/docker-build.yml@main). GitHub forbids a
public repo from calling a private repo's reusable workflow — it resolves as
'workflow was not found', so every docker-build failed at 0s (not a runner or YAML
issue; the shared file and org access were both fine).
Replace with a self-contained build that:
- pushes to ghcr.io/hanzoai/s3 with GITHUB_TOKEN (no custom registry secret, no
cross-repo reusable dependency),
- builds linux/amd64 + linux/arm64 (was amd64-only) via buildx/QEMU — the Dockerfile
is pure-Go (CGO_ENABLED=0) so it cross-builds cleanly,
- runs on the self-hosted amd64 runner (evo).
Co-authored-by: Hanzo AI <ai@hanzo.ai>
* fix: correct crude seaweedfs->hanzo rename artifacts + drop legacy minio
- s3 update: GitHubLatestRelease("hanzo","hanzo") -> ("hanzoai","s3"); the
blunt rename pointed updates at a nonexistent repo (help text already said
hanzoai/s3). Now 's3 update' pulls real Hanzo S3 releases. (go build clean.)
- LLM.md: fix upstream attribution (was github.com/hanzo/hanzo) to credit the real
SeaweedFS upstream, matching NOTICE; correct 'renamed from weed'; note goexif +
go-fuse are the only external deps still under the seaweedfs org.
- LLM.md: add S-Chain section cross-linking datastore (OLAP) + the S-Chain storage
role (reverse of hanzoai/datastore's link), proven-vs-direction scoped.
- drop legacy docker/compose/local-minio-gateway-compose.yml (unreferenced MinIO-
gateway demo).
main is already ZAP-native + internally renamed; these are residual-artifact fixes.
* fix: correct seaweedfs->hanzo rename artifacts + datastore cross-link
- s3 update: GitHubLatestRelease("hanzo","hanzo") -> ("hanzoai","s3") so updates
pull real Hanzo S3 releases (help text already said hanzoai/s3). go build clean.
- LLM.md: correct upstream attribution to SeaweedFS (matches NOTICE); note goexif +
go-fuse are the only remaining seaweedfs-org deps; add S-Chain datastore cross-link.
---------
Co-authored-by: Hanzo AI <ai@hanzo.ai>
Completes the ZAP-native migration on main's leaderless base (schain
pinned-writer is canonical — NO consensus engine introduced). Every
non-test s3 package is now gRPC-free (`google.golang.org/grpc` importers:
16 -> 0); all gRPC-importing test files converted too, so the root module
has ZERO direct grpc importers.
Source (ported from feat/grpc-final's grpc-free versions onto main, which
do NOT touch the leaderless rewrite):
- pb/grpc_client_server.go: dialoption param thread, ZAP pools only.
- mq/broker/*: status.Errorf(codes.X) -> fmt.Errorf("X: ...").
- server/volume_grpc_admin.go: keeps the v2026.3.1 authz-bypass fix
(fail-closed over peerless ZAP) + grpc-free.
- server/volume_grpc_erasure_coding.go, server/common.go (request_id),
svc/master/stream_server.go (drop grpc.ServerStream embed),
wdclient/masterclient.go, operation/assign_file_id.go (string-match
classification).
- security/tls.go (drop LoadServerTLS/advancedtls/SNIStripping),
security/tls_reload.go + certreload/certreload.go (native mtime/size
keypair watcher, no grpc certprovider/pemfile),
util/http/client/http_client.go.
command/master.go: kept main's leaderless schain coordinator; stripped
only the vestigial reflection-only gRPC server (no master RPC rode it).
Tests: codes/status -> fmt/strings; grpc.ServerStream/metadata mocks ->
grpc-free seams (Send/Recv/Context); integration tests dial ZAP
(transport.Dial + pb.NewZapFilerClient / master.New(...ToMasterZapAddress)).
volume_grpc_admin_zap_authz_test.go still PASSES and still asserts the
admin bypass is CLOSED. Deleted tls_sni_test.go (tested removed SNIStripping).
go-bip39: upstream github.com/tyler-smith/go-bip39 repo is deleted (fetch
fails under GOPROXY=direct), breaking `go mod tidy`. Redirect to the luxfi
fork via an in-tree drop-in module (third_party/go-bip39) under the original
path delegating to github.com/luxfi/go-bip39 v1.1.2 — a plain replace is
impossible (the fork renamed its module path and self-imports
.../wordlists, "used for two different module paths"). `go mod tidy` now
succeeds.
go.mod: google.golang.org/grpc/security/advancedtls DROPPED; grpc demoted
direct -> // indirect. grpc now survives ONLY as a transitive of Google
Cloud SDKs (cloud.google.com/go/{kms,storage,pubsub}) used by the GCP
KMS / GCS storage / Pub/Sub integrations — external API-client transport,
not s3 internal RPC. Fully removing it would mean ripping out all GCP
cloud support (out of scope).
Removes hashicorp/raft entirely from the S3 master — the last consensus that
wasn't native Lux. Replaces the leader-election + serialized-writer FSM with a
leaderless topology.Coordinator interface:
- AllocateVolumeId / NextFileId / topology read-write, no IsLeader gate, no election.
- LocalCoordinator: single-node dev/test impl (keeps the suite green, standalone
master works).
- schainCoordinator: production backend — delegates allocation to the schain
storage VM's owner-gated, ML-DSA-verified AllocateTx (luxfi/chains, the native
pinned-writer over lux/consensus). Fails CLOSED until the schain endpoint is
wired (ErrSchainNotWired) — never silently single-writer in prod.
Deleted: raft_hashicorp.go, raft_server.go, raft_server_handlers.go, raft_common.go,
master_grpc_server_raft.go, consensus_server.go (+ tests), the cluster_raft_* shell
cmds, masterNewRaft.html. Dropped hashicorp/raft + raft-boltdb from go.mod. Removed
IsLeader gating from master_server, assign/volume/collection/admin handlers,
meta_aggregator, kafka coordinator, telemetry, UI.
Assign hot-path no longer requires being leader — any master serves; allocation is
serialized by the Coordinator (pinned writer), not an elected leader. Build exit 0;
server/topology/command/mq/admin suites all green.
VERIFY: hashicorp/raft in prod=0, go.mod=0; remaining 'isLeader' hits are
explanatory comments + the filer meta_aggregator's own aggregation flag (not master
raft).
Red review of released v2026.3.0 found a CRITICAL deploy-gating authz bypass I
introduced in dcd09e8b4:
- [CRITICAL] checkGrpcAdminAuth returned nil (ALLOW) on no gRPC peer — but the ZAP
backend always dispatches with context.Background() (peerless) and the guard is
always non-nil, so the IP whitelist was bypassed on 100% of ZAP admin RPCs. On
plaintext ZAP with a configured whitelist, ANY unauthenticated caller could
DeleteCollection/AllocateVolume/VolumeDelete. FIX: fail closed — allow on no-peer
ONLY when the whitelist is empty (IsWhiteListed("")==true, allow-all/dev); deny
when a whitelist is configured (IP can't be evaluated over ZAP; use the PQ-mTLS
allowed_commonNames CN gate instead). Red's TestZapAdminAuthBypassesWhitelist
flips FAIL→PASS; control + EC e2e stay green.
- [HIGH] kafka gateway dialed the broker with grpc.mq while the broker server +
canonical client use grpc.msg_broker — an mTLS rollout would sever kafka↔broker.
Unified on grpc.msg_broker.
- [LOW] ReceiveFile zero-Send CloseSend opened a doomed empty-oneof stream (server
'unknown message type'); now a clean no-op, CloseAndRecv handles the nil stream.
Keeps Red's regression tests.
All internal S3 service transport is now native ZAP (github.com/zap-proto/go v1.8.1):
- master/volume/mq/filer/object servers retired gRPC, serve over transport.ListenStream(TLS)
- PQ X-Wing (X25519MLKEM768) by construction; per-service ZAP client pools
- svc/{master,volume,mq,filer,object} packages (Pike-style)
- Red-verified (4 HIGH + MEDIUM fixed); EC e2e green; raft is the only remaining gRPC
server (slated for leaderless pinned-writer replacement, not migration)
Suite: green (1 known-flaky perf test).
The ReceiveFile client-streaming adapter opened the wire stream with an
empty ReceiveFileRequestInput{} (Data=0, oneof-unset) as the init frame,
on the false assumption the server would "drain it without acting." But
serverStream.Recv replays the init frame as the server's FIRST Recv, so
the production handler's oneof switch (server/volume_grpc_copy.go) saw a
request with no variant set and hit its default → "unknown message type",
breaking EC shard distribution.
Root cause: s3/svc/volume/client.go ReceiveFile + receiveFileClientStream
— a protocol contract mismatch with serverStream.Recv (wire), not a
codec/offset bug. The wire layer was always correct (its own roundtrip
test opens with the real Info as init).
Fix: defer the wire OpenStream until the first Send, using that frame as
the init payload — so the real info frame leads the stream, exactly like
every server-streaming RPC rides its first message as init. No empty
opener. CloseSend opens a (genuinely empty) stream only if nothing was
ever sent, keeping the method total.
No generated files touched — fix is entirely in the hand-written adapter.
Verify:
- new focused unit test s3/svc/volume/receivefile_oneof_test.go drives the
exact caller path (ReceiveFile→Send(Info)→Send(content)→CloseAndRecv)
against a strict server mirroring the production handler's default;
FAILS before, PASSES after.
- e2e TestEcEncodeJulorLayoutConverges: PASS (51.2s), 17 shards distributed.
The volume admin gate (checkGrpcAdminAuth) read the caller IP via gRPC
peer.FromContext and denied on 'no peer info' — but over the native ZAP mesh there
is no gRPC peer, so every admin RPC (AllocateVolume, DeleteCollection, EC ops...)
was denied, breaking EC shard distribution. Over ZAP the caller is authenticated
at the TRANSPORT by PQ-mTLS (pb.ServerTLSConfig requires+verifies the client cert
CN and fails closed without a CA), so the IP whitelist is a gRPC-era fallback that
can't apply. Treat absence of a gRPC peer as the ZAP path → transport mTLS is the
authorization boundary. Also fixes the e2e test's stale *grpc.ClientConn slice type.
(Exposed by TestEcEncodeJulorLayoutConverges; a separate ReceiveFile streaming-oneof
bug remains, handed off.)
masterzap→svc/master, volumezap→svc/volume, mqzap→svc/mq, filerzap→svc/filer,
zapsvc→svc/object. Simple one-word packages nested under s3/svc/, parallel to
s3/wire/<name>; the zap suffix was redundant (ZAP is the only transport) and
compound. Dual-import collisions (s3/mq, s3/filer domain pkgs) aliased filersvc/
mqsvc in the few files that need both. Bare-name-vs-var clash (master param vs
master pkg) fixed by renaming the param to masterAddr. Swept test/ stragglers +
an e2e test's stale *grpc.ClientConn→transport.Conn. Suite green (66 ok).
Red found that a mesh component with grpc.<c>.cert/.key set but grpc.ca empty
silently served one-way TLS — a server cert presented, ANY client accepted, no
mutual auth (mTLS downgrade). Internal ZAP services are mTLS-only, so refuse to
serve: glog.Fatalf with an actionable message instead of dropping client-cert
verification. Same for an unreadable CA. Applies uniformly to master/volume/mq/
filer (one shared pb.ServerTLSConfig). Green: pb, security, command.
Merge worktree-agent-aee73f310721c9a9e: adopt the masterzap package (server.go/
server_rpc.go/stream_server.go — 21 unary + 3 bidi over ZAP) as the master server
adapter, matching filerzap/volumezap/mqzap (one way). DELETE s3/server/master_zap_bridge.go
+ master_zap_stream.go (architecture A: s3server.NewMasterZapBackend) per owner decision.
command/master.go + master_follower.go serve the master over transport.ListenStream/
ListenStreamTLS; WithMasterClient pooled via masterPool. Red HIGH fixes included
(assign retry-over-ZAP string classifier; KeepConnected/SendHeartbeat abandon-path
CloseSend). Consumers (filer_discovery, coordinator_registry, volume_metrics) routed
through pb.WithMasterClient. de-grpc'd master_pb gets a stdlib UnimplementedHanzoServer;
volume Query gets an Unimplemented stub (volume_grpc_query.go) replacing the retired embed.
Reconciled against main's WIP + the volume/mq integrations: pb-local ClientTLSConfig/
ServerTLSConfig (no pb<->security cycle), pb.DialOption type throughout.
GATES: zero gRPC server registrations (master/volume/mq/filer all retired); no conflict
markers; no pb->security import cycle. Suite: 50 pkgs ok; 1 known-flaky perf-timing test
(TestConcurrencyIneffectiveOnStreamBottleneck) passes 3/3 in isolation.
Eliminate direct goleveldb use from the on-disk volume needle-map index and
the on-disk chunk cache, routing both through luxfi/database with the
ZAP-native zapdb backend (factory.New("zapdb", ...)), matching the proven
filer luxdb store. The needle-id -> (offset,size) byte layout, the watermark
batched-write semantics, and the rebuild-from-.idx recovery path are
preserved byte-for-byte, so existing .ldb directories keep working.
- needle_map_leveldb.go: LevelDbNeedleMap now holds a database.Database.
Get/Put/Delete, watermark get/set, generate-from-idx, increment loading,
and lazy load/unload all reimplemented over database.Database. BadgerDB
(zapdb) auto-recovers on open, subsuming leveldb RecoverFile. Freshness is
now determined by the db dir's newest file mtime vs the .idx (uncertainty
rebuilds — always safe).
- recordCount made atomic.Uint64. It is bumped on every Put/Delete which run
under a shared RLock (lazy path) or no lock (resident path); the prior plain
uint64 was a latent data race. Add() gives each writer a consistent count.
- needle_map.go / needle_map_memory.go: drop *opt.Options from the
TempNeedleMapper.UpdateNeedleMap surface (goleveldb-specific, meaningless to
zapdb which self-tunes).
- volume_loading.go: collapse the small/medium/large NeedleMapLevelDb cases
into one zapdb-backed map; remove the goleveldb open-file-cache consts.
- chunk_cache_on_disk.go: open the cache needle map without leveldb opts.
- Tests: port the stale-watermark rebuild + concurrency tests off goleveldb;
add round-trip (256 ids, exact offset/size), batch+reopen-persistence tests.
Functional needle-map tests pass under -race.
Left untouched: needle_map/memdb.go (in-memory MemDb) — it is an in-memory
sort buffer, not on-disk embedded KV, and its DescendingVisit needs reverse
iteration that database.Iterator (forward-only) cannot provide. It is now the
sole direct goleveldb importer in s3. No bbolt exists in s3.
luxfi/database v1.19.0 and luxfi/log v1.4.3 promoted indirect -> direct.
Merge worktree-agent-aef6acb034ce75a0b: mqzap server adapter (14 unary + 7
streaming), command/mq_broker.go serves over ZAP (gRPC HanzoMessaging retired),
WithBrokerGrpcClient flipped to brokerPool, kafka broker client + follower legs
migrated, Red HIGH mTLS-downgrade fix. Reconciled against main's WIP:
- broker dialer unified into pb.DialBrokerZapAddr (exported, mirrors DialMasterZapAddr);
deleted mqzap/dial.go and pointed both follower legs (broker_grpc_sub, local_partition)
at it — fixes the pb<->mqzap<->security import cycle the same way master avoids it.
- ClientTLSConfig/ServerTLSConfig calls retargeted security.X -> pb.X (main relocated
them into pb). DialOption type aligned to pb.DialOption.
Green: mqzap, all mq/*, wire/mq_broker, pb.
filer, master, iam, plugin and worker no longer use google.golang.org/grpc on
the client path.
- gRPC-free client interfaces: the generated *_grpc.pb.go client interfaces drop
"opts ...grpc.CallOption" (no caller ever passed one) and their streaming
methods return the shared seam types in s3/pb/rpc (ServerStream[T] /
BidiStream[Req,Resp]) instead of grpc.ServerStreamingClient/BidiStreamingClient.
The gRPC server scaffolding (Unimplemented*Server, Register*Server, ServiceDesc,
handler trampolines) is deleted; servers serve over the ZAP wire Dispatch.
- The ZAP adapters (filerzap/masterzap/...) implement the gRPC-free interfaces;
grpc.DialOption becomes pb.DialOption (the dial is the ZAP transport).
- Unary calls honor context deadlines: the wire client threads ctx to
transport.Conn.CallContext (v1.8.1), so a call to an unresponsive peer aborts
on the caller's deadline instead of blocking forever.
- Error semantics: the filer tags each failure with its code name ("NotFound: …",
"Internal: …") in the message, which crosses the ZAP wire and is classified by
mount/error_classifier.go — replacing grpc status/codes.
- master ZAP listener port: grpcPort+10000 could exceed 65535; made overflow-safe.
- test doubles updated to the gRPC-free interfaces (0 tests skipped).
- dead cmd/s3-db and cmd/s3-sql (postgres-wire / SQL-engine entry points whose
backends were already ripped) deleted; finished the backend-rip import cleanup.
go build ./... green; go test ./s3/... green (heavy stream benches run under
their existing -short gates).
Still on gRPC (next wave): volume_server + mq (reverted — servers/adapters not
ready), the remaining ~23 plumbing files, and the go.mod dependency itself.
Preserving the working-tree WIP as a recoverable checkpoint before integrating the
parallel agent rips (master/volume/mq). This is architecture A for the master rip
(s3server.NewMasterZapBackend in master_zap_bridge.go); builds clean. Volume + mq
not ripped here. Recoverable: revert this commit to restore the prior HEAD.
Red found embedMessage was the PRE-FIX template: it copied the sub-buffer
verbatim (header included) and returned base+rootOff. For a present-but-empty
nested message (zero-field, e.g. QueryRequest InputSerialization.ParquetInput{},
buffer header-only with rootOff==HeaderSize==len) that offset landed one past the
copied region -> out of bounds -> Object.IsNull() -> the variant was silently
DROPPED on the wire. Same defect class the filer wire already fixed
(child[HeaderSize:] copy, not verbatim).
Fix:
- copy only the payload region sub[HeaderSize:] and return base+(rootOff-HeaderSize)
— the filer's fix; all internal field-relative pointers shift by the same
HeaderSize so deeper nested objects/lists/strings stay valid.
- AND preserve present-but-empty presence: WriteBytes(empty) returns 0 (which the
filer fix would also read back as null), so for an empty body we write one
aligned anchor byte, giving the size-0 object (which reads no fields) a non-null,
in-bounds root. Presence survives instead of collapsing.
Red's TestQuery_InputSerializationVariants/parquet (was FAIL: ParquetInput->nil)
now PASSES; csv/json still pass; the 43-field FetchAndWriteNeedle RemoteConf graph
and the VolumeServerStatus nested float32 graph still round-trip faithfully.
Added wire-level TestEmbedEmptyMessagePresenceParquet +
TestEmbedNonEmptyMessageStillFaithful to lock both branches in.
NOTE: helpers_zap.go is generated (DO NOT EDIT). The proto->zap codegen MUST carry
this embedMessage form on regen — same standing requirement recorded for the
other generated hand-fixes in this migration (embedMessage whole-segment copy,
Dispatch err-in-body). Flag upstream zap-proto/go: a builder-native empty-embedded
encoding (StartObject(0) yielding a referenceable root) would remove the anchor-
byte workaround.
Red found the streaming server-stream adapters handed the engine
context.Background() instead of a context cancelled on client conn-drop — a
goroutine leak for any handler that gates an idle wait on stream.Context().Done()
(and lost request-scoped cancellation on the checkGrpcAdminAuth(stream.Context())
streaming-admin paths).
Root cause: wire/volume_server.serverStream exposed no Context(), unlike the
filer template (filerstream gives every server-stream type a Context() ->
transport.Stream.Context(), the v1.6.2 cancel-on-disconnect fix).
Fix mirrors filerstream exactly:
- serverStream.Context() -> s.Context() (transport per-stream ctx, cancelled on
stream end OR peer conn-drop).
- each of the 11 *ServerStream types gets Context() delegating to z.Context().
- the volumezap Send/Recv adapters return out.Context()/in.Context() and drop the
ctx=context.Background() field entirely.
Red's TestVolumeTailSender_ContextCancelOnConnDrop (was a 3s-timeout LEAK) now
PASSES: the handler blocked on stream.Context().Done() releases on client conn
drop. This also gives the streaming admin-auth paths a real cancellable context
(red fix-priority #3, satisfied by this change).
Red found the two follower-replication legs (SubscribeFollowMe in broker_grpc_sub.go,
PublishFollowMe in local_partition.go) regressed from pb.GrpcDial(...,
security.LoadClientTLS(grpc.msg_broker)) to unconditional plaintext transport.Dial —
a silent mTLS downgrade (vs a PQ-mTLS follower listener: handshake fails, replication
breaks; vs plaintext: payloads in cleartext). Add mqzap.DialBroker — the SAME gate as
pb.dialBrokerZapAddr / kafka.dialBrokerZap (PQ-TLS when grpc.msg_broker certs, else
plaintext) — and route both legs through it. Also drop the now-dead grpc.DialOption
param from MaybeConnectToFollowers (gating moved to viper), removing grpc from
local_partition.go entirely. Keeps Red's red_vectors_test.go (6 round-trip proofs).
WithVolumeServerClient now reuses one pooled ZAP transport.Conn per volume
address (transport.NewPool over dialVolumeZapAddr — PQ-TLS via transport.DialTLS +
PQTLSConfig when grpc.volume certs are set, else plaintext) and runs fn with
volumezap.New(conn). Signature is unchanged, so all ~40 consumers (operation,
shell, storage, topology, server replication/EC paths) migrate to ZAP through
this single seam — no call-site edits. Mirrors the proven filerPool.
startGrpcService now serves the volume server over transport.ListenStream /
ListenStreamTLS (PQ X-Wing mTLS when grpc.volume.cert/.key is set, plaintext
otherwise) via volumezap.NewServerBackend + vsw.Dispatch/StreamHandler — the same
listener carries the 37 unary + 11 streaming RPCs. The gRPC path is gone:
volume_server_pb.RegisterVolumeServerServer, reflection.Register,
pb.NewGrpcServer, pb.ServeGrpcOnLocalSocket, and the grpc/reflection imports all
removed. shutdown closes the *transport.Server (Close, not GracefulStop).
Adapt the gRPC-shaped volume_server_pb.VolumeServerServer to the generated
vsw.VolumeServerStore so the whole volume server answers over the canonical
github.com/zap-proto/go transport — the volume analogue of filerzap.NewServerBackend.
- server.go: 37 unary methods, one hop each way (Wrap req buffer -> pb -> engine
-> pb -> New resp buffer).
- stream_server.go: 11 streaming methods (10 server-stream via Send-adapters
embedding grpc.ServerStream for the unused interface methods; 1 client-stream
ReceiveFile via Recv/SendAndClose).
- server_rpc.go / server_convert.go: per-RPC + leaf converters, the exact inverse
of rpc.go/convert.go (request decode, response encode).
- server_roundtrip_test.go: proto-shaped fake served over real TCP ListenStream,
reached by the volumezap client — unary scalar, unary nested/repeated/oneof,
server-stream, and client-stream all proven over ZAP. No protobuf framing,
no gRPC transport.
fakeEngine implements mq_pb.HanzoMessagingServer; served via mq_brokerwire.Serve
through the real NewServerBackend/NewStreamServer and dialed with mqzap.New. Covers
unary (FindBrokerLeader/ConfigureTopic with retention+assignments/FetchMessage/
GetPartitionRangeInfo), bidi PublishMessage + SubscribeMessage (oneof resp), and
server-stream GetUnflushedMessages. All green.
- broker SubscribeMessage: dial the follower broker over transport.Dial and open
the SubscribeFollowMe client-stream via OpenStream(...Ordinal), forwarding Ack/
Close as zero-copy wire frames (mirrors PublishFollowMe in local_partition.go);
drops pb.GrpcDial + mq_pb.NewHanzoMessagingClient.
- kafka integration BrokerClient: replace grpc.DialContext + NewHanzoMessagingClient
with dialBrokerZap (PQ-TLS/plaintext) + mqzap.New; conn is now transport.Conn.
dialBrokerZapAddr (PQ-TLS when grpc.msg_broker.cert/.key, else plaintext) + a
transport.NewPool, mirroring filerPool. WithBrokerGrpcClient now pools one ZAP
conn per broker address and runs fn with mqzap.New(conn) — the broker fully cut
over so its address IS the ZAP endpoint (no port offset). DialOption now inert.
Replaces both pb.RegisterHanzoMessagingServer sites with transport.ListenStream /
ListenStreamTLS over mq_brokerwire.Dispatch(mqzap.NewServerBackend) +
mqzap.NewStreamServer. PQ-secured mTLS (X25519MLKEM768) when grpc.msg_broker.cert/.key
is set, plaintext otherwise — same gate the gRPC broker enforced, no downgrade.
Main + localhost listeners; blocks until interrupt closes every ZAP listener.
NewServerBackend (14 unary) + NewStreamServer (7 streaming) adapt the existing
broker engine to the generated mq_brokerwire.Broker/StreamBroker, mirroring the
proven filerzap template. server_convert.go owns the request-view->*mq_pb and
*mq_pb-response->ResponseInput direction; stream_server_convert.go the streaming
mirror; both reuse the leaf converters in convert.go (added partitionOffsetFromWire,
brokerStatsFromView, publisher/subscriber/range ToWire). Zero protobuf on the wire.
Three stragglers left the tree red:
- cmd/s3-db and cmd/s3-sql were the PostgreSQL-wire and SQL-engine entry
points; their backing packages (s3/server/postgres, s3/query/engine) were
ripped in 5b7369380, leaving the commands importing nothing. Deleted.
- s3/server/filer_server.go and s3/command/imports.go still imported the 20
non-SQLite filer backends deleted in 448e751f1. Dropped the dead imports.
- test/plugin_workers/framework.go passed *transport.Stream where v1.7.0 made
Stream an interface (transport.Stream). Fixed the handler signature.
go build ./... now exits 0.
HeartbeatReqFromWire decoded Volumes/EcShards/LocationUuids/MaxVolumeCounts
but silently omitted NewVolumes, DeletedVolumes, NewEcShards, DeletedEcShards
and DiskTags — which the encode side writes and the wire view exposes. That is
the volume->master incremental topology-churn channel, so a wired adapter would
leave the master blind to incremental volume/EC-shard adds and deletes.
Add the five decode blocks (reusing ecShardInfoFromView; new volumeShortInfoFromView
and diskTagFromView), and a round-trip test that pins all five — the prior
TestSendHeartbeatBidi only exercised Ip/Port/Id.
Deleted filer backends arangodb/cassandra(2)/etcd/hbase/leveldb2/leveldb3/
mongodb/mysql(2)/postgres(2)/redis(1,2,3)/tarantool/tikv/ydb/elastic and their
blank imports in command/imports.go + server/filer_server.go. Kept sqlite (the
only selectable metadata store) + abstract_sql (sqlite base) + leveldb (FUSE
mount meta_cache uses it directly, internal — not a backend choice). Per:
SQLite-only scoped per tenant.
