s3's SQLite filer already imports github.com/hanzoai/sqlite on main. v0.3.2 is
self-contained (vendored pure-Go engine), so modernc.org/{sqlite,libc,memory,
mathutil} are no longer needed and are dropped from go.mod (verified: go build ./...
stays green, cgo + nocgo, without them). csqlite + sqlcipher (v0.3.2's cgo backend)
enter as indirect.
modernc removal is done via go mod edit rather than go mod tidy: a pre-existing,
unrelated go.sum re-tag drift on github.com/luxfi/pq@v1.0.3 blocks full-graph ops
on this branch's base (origin/main) — out of scope for the sqlite migration.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
go mod tidy (GOWORK=off) removes go.mod/go.sum entries no longer referenced by
any package, under any build tag:
- github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.1 — unimported (grep all .go = 0)
- github.com/luxfi/container v0.0.4 — unimported (grep all .go = 0)
- modernc.org/sqlite marked // indirect (pulled transitively via hanzoai/sqlite;
no direct import remains)
Default build and full-tag build (elastic gocdk sqlite ydb tarantool tikv rclone)
both green. go.sum: the checksums for the two dropped modules are removed.
NOTE: github.com/hanzos3/go-sdk is deliberately NOT dropped — `go mod why` shows
it is a required transitive dep of github.com/luxfi/zapdb (the filer zapdb
metadata store) via s3/filer/luxdb; tidy keeps it.
* fix(sqlite): use hanzoai/sqlite (the one driver), drop direct modernc import
* fix(sqlite): use hanzoai/sqlite (the one driver), drop direct modernc import
---------
Co-authored-by: Hanzo AI <ai@hanzo.ai>
Completes the ZAP-native migration on main's leaderless base (schain
pinned-writer is canonical — NO consensus engine introduced). Every
non-test s3 package is now gRPC-free (`google.golang.org/grpc` importers:
16 -> 0); all gRPC-importing test files converted too, so the root module
has ZERO direct grpc importers.
Source (ported from feat/grpc-final's grpc-free versions onto main, which
do NOT touch the leaderless rewrite):
- pb/grpc_client_server.go: dialoption param thread, ZAP pools only.
- mq/broker/*: status.Errorf(codes.X) -> fmt.Errorf("X: ...").
- server/volume_grpc_admin.go: keeps the v2026.3.1 authz-bypass fix
(fail-closed over peerless ZAP) + grpc-free.
- server/volume_grpc_erasure_coding.go, server/common.go (request_id),
svc/master/stream_server.go (drop grpc.ServerStream embed),
wdclient/masterclient.go, operation/assign_file_id.go (string-match
classification).
- security/tls.go (drop LoadServerTLS/advancedtls/SNIStripping),
security/tls_reload.go + certreload/certreload.go (native mtime/size
keypair watcher, no grpc certprovider/pemfile),
util/http/client/http_client.go.
command/master.go: kept main's leaderless schain coordinator; stripped
only the vestigial reflection-only gRPC server (no master RPC rode it).
Tests: codes/status -> fmt/strings; grpc.ServerStream/metadata mocks ->
grpc-free seams (Send/Recv/Context); integration tests dial ZAP
(transport.Dial + pb.NewZapFilerClient / master.New(...ToMasterZapAddress)).
volume_grpc_admin_zap_authz_test.go still PASSES and still asserts the
admin bypass is CLOSED. Deleted tls_sni_test.go (tested removed SNIStripping).
go-bip39: upstream github.com/tyler-smith/go-bip39 repo is deleted (fetch
fails under GOPROXY=direct), breaking `go mod tidy`. Redirect to the luxfi
fork via an in-tree drop-in module (third_party/go-bip39) under the original
path delegating to github.com/luxfi/go-bip39 v1.1.2 — a plain replace is
impossible (the fork renamed its module path and self-imports
.../wordlists, "used for two different module paths"). `go mod tidy` now
succeeds.
go.mod: google.golang.org/grpc/security/advancedtls DROPPED; grpc demoted
direct -> // indirect. grpc now survives ONLY as a transitive of Google
Cloud SDKs (cloud.google.com/go/{kms,storage,pubsub}) used by the GCP
KMS / GCS storage / Pub/Sub integrations — external API-client transport,
not s3 internal RPC. Fully removing it would mean ripping out all GCP
cloud support (out of scope).
Preserving the working-tree WIP as a recoverable checkpoint before integrating the
parallel agent rips (master/volume/mq). This is architecture A for the master rip
(s3server.NewMasterZapBackend in master_zap_bridge.go); builds clean. Volume + mq
not ripped here. Recoverable: revert this commit to restore the prior HEAD.
zap-proto/go v1.7.0 makes Conn and Stream interfaces (one contract every
transport realises natively) and adds native-QUIC — one QUIC stream per ZAP op,
zero head-of-line blocking — with PQ X-Wing (X25519MLKEM768) the default and no
backwards-compat shims. Mechanical follow-through here: *transport.Conn ->
transport.Conn and *transport.Stream -> transport.Stream across 40 files (zapsvc,
pb, filerstream, masterstream, mq*, volumezap, worker, mount, all wire/* zapsvc
adapters). No PQ path downgraded. Full s3 suite green: 75 ok, 0 fail.
Pool.With(addr, fn) encapsulates get->run->evict-if-died, so WithGrpcFilerClient/
WithOneOfGrpcFilerClients drop the borrow/evict boilerplate. The full
generalization now lives upstream; s3 supplies only the dial choice.
Replace the hand-rolled per-address filer conn cache with transport.Pool — the
pooling (dial-or-reuse, evict-dead-on-IsClosed) is a generic transport concern
now living in github.com/zap-proto/go, reusable by every ZAP client (iam,
master, volume, mq), not duplicated per service. s3 injects only the dial choice
(dialFilerZapAddr: PQ-TLS when grpc.filer certs are set, else plaintext). Filer
RPCs reuse one Conn per address — no TCP/PQ handshake per call.
zap-proto/go v1.6.2 gives each stream a Context cancelled when the stream ends
or the conn drops. filerstream server stream types expose Context(); the filerzap
streaming adapters now pass s.Context() (was context.Background) to the engine
methods, so an idle SubscribeMetadata whose peer disconnects is released instead
of leaking its goroutine + subscription. Proven by TestStreamServerSubscribe
ReleaseOnDisconnect (idle handler freed on client conn drop).
Red flagged the cutover as a security downgrade: the gRPC filer enforced
security.LoadServerTLS("grpc.filer", RequireClientCert) but the ZAP cutover was
plaintext, and transport v1.5.0 had no streaming+TLS listener. Fixed end to end:
- zap-proto/go v1.6.1 adds transport.ListenStreamTLS (PQ-TLS + server streaming;
TLS analogue of ListenStream). PQTLSConfig pins X25519MLKEM768 (PQ X-Wing).
- security.ServerTLSConfig/ClientTLSConfig: raw *tls.Config from the SAME
grpc.filer.cert/.key + grpc.ca + allowed_commonNames config LoadServerTLS used
(mTLS: RequireAndVerifyClientCert + CN/wildcard gate). nil when unset.
- command/filer.go: when grpc.filer certs are set, serve via ListenStreamTLS over
PQTLSConfig — no downgrade; else plaintext (loopback/dev), as gRPC was when
uncerted.
- pb.WithGrpcFilerClient/WithOneOfGrpcFilerClients dial via dialFilerZap: DialTLS
+ PQTLSConfig when certs are set, else plain Dial. Both sides gate identically.
Plaintext path (no certs) unchanged — filerzap/policy/mount/pb/security green.
A non-PQ peer fails the handshake (X25519MLKEM768 is the only offered curve).
mount.configure shell client + WFS.Configure handler + mount server now ride
mountwire over the local unix socket (transport.Serve on the existing
listener); UnimplementedHanzoMountServer embed dropped. mount_pb deleted —
first _pb service package fully eliminated. (s3/mount still imports grpc only
for Option.GrpcDialOption, swept later.)
ConsensusServer wires the master's HA coordination onto luxfi/consensus via
replog.Log over a linear consensus.Chain — the master's replicated FSM (a
monotonic max-volume-id + topology id) now commits through Lux consensus
instead of a Raft log:
- Do(MaxVolumeIdCommand) -> replog.Commit (finalize + apply, drop-in for
raft Server.Do); applyMaxVolumeId applies finalized commands to topology.
- Peers/AddPeer/RemovePeer (validator set), Name, Stop.
- ZAP-native (votes gossip over the zap-proto transport, no gRPC),
post-quantum-final (Quasar BLS + ML-DSA), no separate leader-election FSM.
go.mod: github.com/luxfi/consensus pinned to the public pushed replog commit
(v1.25.22-0.20260624053430-ad7d73d8f739) — no local replace. Builds green
(cgo via SDKROOT).
Next: swap NewRaftServer's callers (s.raftServer.Do/Leader/Peers...) to
ConsensusServer, then drop the 4 seaweedfs/hashicorp raft go.mod lines.
- Pin github.com/zap-proto/go to the public v1.5.0 release (go.sum from the
module proxy) and DELETE the local 'replace => /Users/z/work/zap/go'. The
wire layer now resolves entirely from public modules — zero local-path
replaces (only the public apache/thrift v0.22.0 32-bit-overflow pin stays).
- Delete the dead '// replace github.com/seaweedfs/raft => /Users/chrislu/...'
comment (SeaweedFS author's local path; lineage scrub).
Next: master HA consensus seaweedfs/raft -> luxfi/consensus (ZAP-native),
which drops the last SeaweedFS-lineage require.
we had a chicken and egg problem with this feature even
when used with kes the credentials generation would
not work in correct sequence causing setup/deployment
disruptions.
This PR streamlines all of this properly to ensure that
this functionality works as advertised.
