* fix(sqlite): use hanzoai/sqlite (the one driver), drop direct modernc import
* fix(sqlite): use hanzoai/sqlite (the one driver), drop direct modernc import
---------
Co-authored-by: Hanzo AI <ai@hanzo.ai>
* replica: extract single-writer election to hanzoai/ha
Election (Owner/IsOwner/Replicas/Member) was storage-agnostic pure Go with
zero vfs-internal callers — it lived here only historically. It is now its
own coordination primitive at github.com/hanzoai/ha so any singleton (cron,
queue drain, billing sweep) gets exactly-once WITHOUT importing a SQLite
library. vfs/replica keeps what it is: SQLite replication (Replicator,
SQLiteDB, BackendStore). One concern per package.
Consumers (visor, cloud/internal/org) migrate to hanzoai/ha in lockstep;
they pin vfs v0.6.2 so this main-only removal does not disturb them until
they bump. Removes owner.go + its two election tests; doc points at ha.
* ci: enable cgo for the -race steps (pre-existing red)
go test -race requires cgo, but the self-hosted hanzo-build-linux-amd64
runner has no C compiler and defaulted CGO_ENABLED=0 (vfs is CGO-free in
production), so both race steps failed — first with 'go: -race requires
cgo', then 'C compiler gcc not found'. Every main run has been red on this
for many commits. Fix: install gcc + set CGO_ENABLED=1 on the two -race
steps only; the production build stays CGO_ENABLED=0. Now the race detector
actually runs (real coverage for the concurrent replicator/ship loops).
Unrelated to the ha extraction but folded in so this PR lands green rather
than override a known-bad check.
* replica: round-fenced writes (FencedStore) — close split-brain double-write (v0.6.3)
Election (hanzoai/ha) is coordination-free, so two replicas with divergent
membership views can each elect themselves the single writer for one org, and
the object store's unconditional Put ('overwriting is allowed') lets both
clobber the mirror. FencedStore closes this at the storage layer over a
ConditionalStore (S3 If-Match / GCS generation-match) CAS:
- one object per key carries a monotone round header
- admit a Put iff round >= recorded; reject (ErrStaleRound) when below, so a
deposed/partitioned writer is fenced once a successor advanced the round
- '>=' (not one-shot '>') because the owner re-ships at its STABLE lease round
every tick; only a LOWER round is stale
- CarryForward is the safe takeover seal: re-reads and carries the
predecessor's last landed write forward atomically via CAS, so a successor
can never clobber an acknowledged write
- round header + payload in ONE object / ONE conditional write: no window
between 'round admitted' and 'data written'
Round is a plain uint64 (the storage boundary). The coordination value
ha.Lease{Owner,Round} and the linearizable round source stay in ha and its
consumers. Pure over the ConditionalStore seam; concrete minio store lives with
the S3-client owner. Tests: deposed-fenced, same-round re-ship, handoff
lock-out, carry-forward-no-loss, and concurrent race proofs (20x race-stable).
---------
Co-authored-by: hanzo <z@hanzo.ai>
Election (hanzoai/ha) is coordination-free, so two replicas with divergent
membership views can each elect themselves the single writer for one org, and
the object store's unconditional Put ('overwriting is allowed') lets both
clobber the mirror. FencedStore closes this at the storage layer over a
ConditionalStore (S3 If-Match / GCS generation-match) CAS:
- one object per key carries a monotone round header
- admit a Put iff round >= recorded; reject (ErrStaleRound) when below, so a
deposed/partitioned writer is fenced once a successor advanced the round
- '>=' (not one-shot '>') because the owner re-ships at its STABLE lease round
every tick; only a LOWER round is stale
- CarryForward is the safe takeover seal: re-reads and carries the
predecessor's last landed write forward atomically via CAS, so a successor
can never clobber an acknowledged write
- round header + payload in ONE object / ONE conditional write: no window
between 'round admitted' and 'data written'
Round is a plain uint64 (the storage boundary). The coordination value
ha.Lease{Owner,Round} and the linearizable round source stay in ha and its
consumers. Pure over the ConditionalStore seam; concrete minio store lives with
the S3-client owner. Tests: deposed-fenced, same-round re-ship, handoff
lock-out, carry-forward-no-loss, and concurrent race proofs (20x race-stable).
* replica: extract single-writer election to hanzoai/ha
Election (Owner/IsOwner/Replicas/Member) was storage-agnostic pure Go with
zero vfs-internal callers — it lived here only historically. It is now its
own coordination primitive at github.com/hanzoai/ha so any singleton (cron,
queue drain, billing sweep) gets exactly-once WITHOUT importing a SQLite
library. vfs/replica keeps what it is: SQLite replication (Replicator,
SQLiteDB, BackendStore). One concern per package.
Consumers (visor, cloud/internal/org) migrate to hanzoai/ha in lockstep;
they pin vfs v0.6.2 so this main-only removal does not disturb them until
they bump. Removes owner.go + its two election tests; doc points at ha.
* ci: enable cgo for the -race steps (pre-existing red)
go test -race requires cgo, but the self-hosted hanzo-build-linux-amd64
runner has no C compiler and defaulted CGO_ENABLED=0 (vfs is CGO-free in
production), so both race steps failed — first with 'go: -race requires
cgo', then 'C compiler gcc not found'. Every main run has been red on this
for many commits. Fix: install gcc + set CGO_ENABLED=1 on the two -race
steps only; the production build stays CGO_ENABLED=0. Now the race detector
actually runs (real coverage for the concurrent replicator/ship loops).
Unrelated to the ha extraction but folded in so this PR lands green rather
than override a known-bad check.
---------
Co-authored-by: hanzo <z@hanzo.ai>
xorm/GORM services can't share a handle with SQLiteDB (Restore swaps the file). Expose
SnapshotFile(path) (transient read handle, VACUUM INTO) + RestoreFile(path,data) (atomic
swap) so a service using its OWN engine adopts HA: RestoreFile before opening, SnapshotFile
in the push loop. SQLiteDB now wraps these (one and one way). Tests still green.
Completes the shared HA-SQLite lib: NewBackendStore(be) adapts any vfs backend.Backend
(file locally, s3/SeaweedFS in prod) to the Replicator Store, with a cheap content-version
sidecar (<key>.ver) so a reader answers Version() without fetching the whole DB. So a
service wires HA in one line: NewReplicator(key, NewBackendStore(be), OpenSQLite(path)).
Test: full owner->vfs-backend->reader path over the real file backend (SQLite rows
survive push/pull, incl. updates + the version-skip).
The ONE importable durability library — promoted from cloud/internal/org (proven,
but internal + unimportable + only a memDB test) into hanzoai/vfs so EVERY service
adopts the same code, never re-invents HA:
- Replicator: per-org SQLite <-> vfs object store, whole-object WAL-checkpointed
snapshots, per-org sealed. Push (owner) / Pull (reader) / PushLoop.
- owner.go: deterministic single-writer election via Rendezvous/HRW over IAM
membership — no coordinator, no lock service, no Postgres, no Redis.
- sqlite.go: the REAL SQLite DB impl the cloud replicator never had — Snapshot via
VACUUM INTO (consistent, WAL-checkpointed), Restore via atomic file swap+reopen.
CGO-free (modernc). This is what makes HA SQLite actually work on disk.
Tests: real SQLite snapshot/restore round-trip (rows preserved, usable after
restore), full owner->store->reader Push/Pull cycle + redundant-pull skip, HRW
determinism/exclusivity/spread. Adoption contract (hydrate-on-open, single-writer
gate, post-commit ship) documented for visor/iam/commerce/base rollout.
* vfs: gate Mount() + cmd/vfsd behind cloud_mount build tag
The Mount(), SetInstance(), Shutdown() HTTP-server surface drags in
hanzoai/cloud + hanzoai/zip. Downstream services that consume the
core block + FS API (vfs.VFS, vfs.FS, vfs.File, NewCrypto) directly
— ATS shard layer, KMS, BD — don't need the HTTP wire and
should not pull in the cloud module graph.
Without -tags cloud_mount: vfs builds with only backend/cache/crypto/
fs/file/block. With it: Mount() + cmd/vfsd are compiled in, exactly
as before.
vfs_test, fs_test, sqlite_test, production_test still run + pass on
the default build (verified locally).
* chore: bump go to 1.26.4
Picks up upstream security fixes for crypto/x509, mime, net/textproto,
crypto/fips140. No dependency bumps — only the toolchain directive.
go test -race ./... is green.
Three new primitives, all brand-neutral:
BlockTimeWorkload(blockTime, avgBlockBytes, monthsRetention,
readAmp, objectBytes) -> Workload
Generic chain-style workload constructor parameterised by
block time so a single sweep covers 1ms..1s with one call.
Tiered{RatePerSecond, HotWindow, ColdRetention, HotBackend,
ColdBackend, PutsPerByte, GetsPerByte, EgressGBPerMonth}
Splits cost between a hot tier (PVC/SSD, no per-op fees) and a
cold tier (object store). The hot window is configurable so
operators can dial query latency vs cost.
Cluster{Writers, Readers, Base, ReadsPerReaderRatio,
EgressPerReaderGBPerMonth}
Models a shared-archive fleet. Content-addressed dedup means
Writers does not multiply storage/PUT cost (the dedup
invariant). Readers each contribute their GET share linearly.
Tests pin three invariants:
TestBlockTimeSweep_PrintsCostCurve — 1ms..1s sweep + hot sizes
TestTiered_24h_Hot_R2_Cold — tiering math sanity
TestCluster_DedupKeepsWriterCostFlat — 1w vs 11w cost identical
TestCluster_ReadersScaleGetCost — 1, 11, 100, 1000 readers
Adds pkg/cost with a brand-neutral pricing model:
Workload { StorageGB, PutsPerMonth, GetsPerMonth, EgressGBPerMonth }
Catalog: aws-s3-{standard,ia,glacier-instant}, cloudflare-r2,
do-spaces, gcs-{standard,coldline}, k8s-pvc-block
EstimateFor, CompareBackends, Report, FromBenchmark
Plus cmd/vfs-cost — a thin CLI over the package that reports the
per-month cost of a workload on every backend, sorted cheapest first.
The package contains no application-specific defaults. Validator
archive shapes, sqlite snapshot pipelines, indexer workloads etc.
compose Workload themselves in their own repos; cost stays the
primitive.
Adds five Go tests covering the four code-level concerns that gate
using VFS as the durable store for stateful services like luxd /
zapdb:
TestCrashRecovery_Synced_Persists (#1)
TestCrashRecovery_UnsyncedWriteIsLost_NotCorrupted (#1)
TestCompaction_SustainedWrite (#4, opt-in)
TestBackendOutage_PutErrorBubblesUp (#5)
TestBackendOutage_GetErrorBubblesUp (#5)
TestKeyRotation_MultiRecipientAllowsRollingRotation (#6)
Adds docs/PRODUCTION-READINESS.md addressing all seven concerns —
the four with code, plus three operational runbooks (#2 empty-cache
bootstrap timing, #3 warm-cache bootstrap timing, #7 rollback path
to PVC-local state).
Verdict for blockchain state: VFS is production-grade as the
finalized-block archive tier — finalized blocks are immutable, so
they stream into VFS once (no per-write fsync overhead) and dedup
content-addressed across replicas. Hot tier (zapdb WAL + memtable)
stays on local SSD. The §4 6.4 fsync'd-ops/s benchmark is the
worst case (per-write Sync); the finalized-SST stream pattern looks
nothing like this and is bounded by network bandwidth not metadata
churn.
#2 and #3 are deliberately runbook-only — bootstrap timing needs
a real cluster + S3 backend to measure honestly, not a Go test.
#7 ditto — the procedure is straightforward (vfsd export → tar →
PVC → restore) but execution is operational.
Switch from gcr.io/distroless/static:nonroot to FROM scratch. The
static Go binary gets CA certs, tzdata, /etc/passwd and /etc/group
(nonroot uid/gid 65532) copied from the build stage.
The `=> ../cloud` and `=> ../zip` replace directives were a parallel-dev
hack. Cloud is now published at v0.1.0 (HIP-0106 unified binary) and
zip at v0.2.0 (circuit-breaker primitive + multi-App isolation).
Resolve from GOPROXY — no local-dir pins remain.
* feat: cloud Mount() per HIP-0106
Adds pkg/vfs.Mount(*zip.App, cloud.Deps) error so the unified Hanzo
Cloud binary can blank-import and register VFS alongside every
other Hanzo subsystem.
VFS has no public HTTP surface today — it's an S3-backed virtual
block filesystem driven through FUSE/CLI/sidecar. Mount() exposes
liveness/readiness probes plus a minimal block-CRUD HTTP API so
co-resident subsystems can talk to VFS without falling back to the
CLI:
- GET /v1/vfs/health — always 200
- GET /v1/vfs/readyz — 200 only when an instance is
attached via vfs.SetInstance
- PUT /v1/vfs/blocks — payload upload
- GET /v1/vfs/blocks/:id — payload download
- DELETE /v1/vfs/blocks/:id — backend delete
- GET /v1/vfs/stats — cache/backend counters
cmd/vfsd is a HIP-0106 thin shim that wires backend + age keys from
env (VFSD_BACKEND, VFSD_AGE_KEY) and listens on cloud.Config.ListenAddr.
The existing cmd/vfs CLI (put/get/stats/mount) is untouched.
init() registers with cloud.Register("vfs", 20, …). schema/vfs.zap
adds the minimal Health + Block ZAP interface; full multi-block
File surface lands in follow-up PRs.
Tests:
- pkg/vfs: TestMount_HealthReadyz + TestMount_PutGetRoundtrip green
- existing TestRoundTrip + TestStats unchanged
* refactor: collapse pkg/vfs/ → root for clean import path
Subsystem code lives in package vfs at repo root. Importers now use
`github.com/hanzoai/vfs` (no /pkg/vfs/ nesting). Standalone shim moved
to cmd/vfs/ where applicable.
Per Hanzo Go-stdlib pattern: repo IS the package.
* refactor: flatten import path — github.com/hanzoai/cloud (drop /pkg/cloud)
Subsystem code lives in package vfs at repo root. Importers now use
`github.com/hanzoai/vfs` (no /pkg/vfs/ nesting). Standalone shim moved
to cmd/vfs/ where applicable.
Per Hanzo Go-stdlib pattern: repo IS the package.
Adds pkg/vfs.Mount(*zip.App, cloud.Deps) error so the unified Hanzo
Cloud binary can blank-import and register VFS alongside every
other Hanzo subsystem.
VFS has no public HTTP surface today — it's an S3-backed virtual
block filesystem driven through FUSE/CLI/sidecar. Mount() exposes
liveness/readiness probes plus a minimal block-CRUD HTTP API so
co-resident subsystems can talk to VFS without falling back to the
CLI:
- GET /v1/vfs/health — always 200
- GET /v1/vfs/readyz — 200 only when an instance is
attached via vfs.SetInstance
- PUT /v1/vfs/blocks — payload upload
- GET /v1/vfs/blocks/:id — payload download
- DELETE /v1/vfs/blocks/:id — backend delete
- GET /v1/vfs/stats — cache/backend counters
cmd/vfsd is a HIP-0106 thin shim that wires backend + age keys from
env (VFSD_BACKEND, VFSD_AGE_KEY) and listens on cloud.Config.ListenAddr.
The existing cmd/vfs CLI (put/get/stats/mount) is untouched.
init() registers with cloud.Register("vfs", 20, …). schema/vfs.zap
adds the minimal Health + Block ZAP interface; full multi-block
File surface lands in follow-up PRs.
Tests:
- pkg/vfs: TestMount_HealthReadyz + TestMount_PutGetRoundtrip green
- existing TestRoundTrip + TestStats unchanged
bazil.org/fuse dropped macOS support (commit 65cc252+). 0.3.0
shipped Linux-only. This drop adds a parallel pkg/mount/fuse_darwin.go
using github.com/jacobsa/fuse, which still supports both macFUSE
(kext) and fuse-t (userspace, no kext, SIP-friendly).
Same Mount(*vfs.FS, mountpoint) signature; the `fuse` build tag
picks the right driver per GOOS at compile time.
What's new
----------
- pkg/mount/fuse_darwin.go (`-tags fuse && darwin`): full
fuseutil.FileSystem implementation using jacobsa/fuse. Maps every
POSIX op the kernel sends:
LookUpInode → FS.Lookup
GetInodeAttributes → FS.Lookup + inode→attrs
SetInodeAttributes(size) → File.Truncate + Sync
MkDir → FS.Mkdir
CreateFile → FS.Create + FS.Open + handle alloc
Unlink/RmDir → FS.Remove
OpenDir/ReadDir/ReleaseDirHandle → FS.ReadDir
OpenFile/ReadFile/WriteFile → FS.Open + File.{ReadAt,WriteAt}
SyncFile/FlushFile → File.Sync
ReleaseFileHandle → File.Close
StatFS → ~1 PiB pseudo-quota (backend is unlimited)
- pkg/vfs/fs.go: new FS.PathOfInode(InodeID) for the macOS driver,
which gets ops as (parent_id, name) tuples instead of bazil's
node-pointer style. O(depth) walk up the parent chain.
- pkg/mount/fuse_sqlite_test.go: build tag widened to
`fuse && (linux || darwin)`. Adds `runtime.GOOS` switch to invoke
the right unmount tool (`fusermount`/`fusermount3` on Linux,
`umount`/`diskutil unmount` on macOS).
Verified
--------
On macOS with fuse-t installed:
VFS_FUSE_E2E=1 go test -tags fuse -timeout 60s \
-run TestFUSESQLite ./pkg/mount/ -v
→ PASS: TestFUSESQLite (1.16s)
→ 100 rows survived umount+remount, integrity_check=ok
Real FUSE mount, real SQLite opening test.db directly on the
mountpoint, 100 INSERTs, unmount, remount, PRAGMA integrity_check.
Zero copy step.
The mount layer is no longer a stub. With `-tags fuse` on Linux,
SQLite (and any POSIX consumer) opens files directly on the
mountpoint — no intermediate copy. Default builds stay
dependency-light: bazil.org/fuse + bazil.org/fuse/fs are gated
behind the build tag.
What's new
----------
- pkg/mount/fuse_unix.go (`-tags fuse && linux`): full FS/Node/Handle
bridges. Maps every POSIX op the kernel sends:
Read → File.ReadAt
Write → File.WriteAt
Fsync → File.Sync (durable through age + backend)
Flush → File.Sync (close-time flush)
Release → File.Close
Setattr.Size → File.Truncate
Lookup → FS.Lookup
ReadDirAll → FS.ReadDir
Create → FS.Create + Open
Mkdir → FS.Mkdir
Remove → FS.Remove
Open(O_TRUNC) → File.Truncate(0)
- pkg/mount/fuse_stub.go: same Mount/MountWithVFSAdapter signatures
for the default no-fuse build; returns a build-tag error.
- pkg/mount/fuse_sqlite_test.go: gated by VFS_FUSE_E2E=1 +
`-tags fuse`. Mounts in tempdir, creates `test.db`, 100 INSERTs,
unmounts, remounts, runs PRAGMA integrity_check. Verifies the DB
survives the full encrypt → backend → decrypt round trip with
zero copy.
- cmd/vfs: mount subcommand wires through MountWithVFSAdapter.
CI
--
.github/workflows/build.yml runs three test passes on Ubuntu:
1. default `go test -race ./...`
2. `go vet -tags fuse ./...`
3. apt-get fuse + `VFS_FUSE_E2E=1 go test -race -tags fuse
-run TestFUSESQLite ./pkg/mount/ -v`
Caveats
-------
- macOS: bazil.org/fuse dropped macOS support upstream (commit
65cc252). 0.3.1 will add a parallel mount_darwin.go using
github.com/jacobsa/fuse so devs on macOS get the same e2e flow.
Until then, macOS devs use the in-process API + the SQLite
byte-roundtrip test in pkg/vfs/sqlite_test.go (already shipped).
- Local NVMe disk cache (write-back tier) lands in 0.4.0; today the
cache is in-memory LRU.
Adds the byte-addressed File API and a multi-file FS over the block
VFS — the layer SQLite (and any other POSIX consumer) needs. FUSE
mount lands in 0.3.0; until then the API is in-process Go.
What's new
----------
- pkg/vfs/fs.go — FS with inode tree, directory ops (Mkdir, Create,
Lookup, ReadDir, Remove). Tree persisted as one encrypted JSON blob
at metadata/root.zap.age (sharded post-1.0 if/when needed).
- pkg/vfs/file.go — File handle implementing io.ReaderAt + io.WriterAt
with read-modify-write for partial-block writes, dirty buffer per
handle, Truncate (shrink and grow with zero-padding), Sync to flush
+ persist metadata, Close auto-syncs.
Testing
-------
- fs_test.go (5 tests): create/stat, dir ops, persist-across-remount,
multi-block (16 KiB / 4 blocks), partial-block read-modify-write
semantics.
- sqlite_test.go (1 test): proves a real SQLite database written
through encrypt → backend → decrypt is byte-identical and passes
`PRAGMA integrity_check ok`. Reference DB: 200 INSERTs across
4-row schema → 20480 bytes / 5 blocks. modernc.org/sqlite (pure
Go) eliminates the need for cgo in CI.
What this proves
----------------
- 4 KiB block alignment matches SQLite's default page size — zero
amplification for full-page writes.
- Partial-page writes (SQLite header updates, WAL frame headers)
hit the read-modify-write path correctly.
- File size tracking is exact bytes (not block-rounded), so SQLite
Stat() returns the right size for its `pragma page_size` math.
- Cross-mount durability: write file, close FS, re-open — bytes
survive the metadata-blob serialization round-trip.
What's next
-----------
- 0.3.0: bazil.org/fuse mount. Kernel POSIX VFS calls land at
File.{ReadAt,WriteAt,Sync,Truncate}; SQLite opens DB directly on
the mountpoint with no copy step. Linux + macOS.
- 0.4.0: NVMe disk write-back cache (configurable, survives restart).
- 0.5.0: gcs + azureblob backends.
- 0.6.0: K8s sidecar mode + Helm chart.
- 1.0.0: chunked uploads, multipart resume, GC for orphan blocks
via a refcount map at metadata/refs.zap.age, Prom/OTel metrics.
Hanzo Dev
