vfs/LLM.md

hanzoai/vfs — AI Engineering Guide

What this is

S3-backed virtual block filesystem. Local-NVMe hot tier + S3 (or any object store) cold tier with PQ encryption per block. Stateful services (IAM, BD, ATS, TA, KMS, lqd snapshots) get unlimited disk via S3 without needing K8s PVC sizing decisions.

Companion to ~/work/hanzo/replicate — replicate streams SQLite WAL frames to S3 (file-level granularity); vfs is block-level granularity that transparently spills cold pages.

replica/ subpackage — HA-SQLite substrate (HIP-0107)

github.com/hanzoai/vfs/replica is the shared per-org SQLite HA substrate: local SQLite for speed, snapshot replication to the object store, hydrate-on-open, single-writer gated by github.com/hanzoai/ha election. Replicator (Push/Pull), DB (Snapshot/Restore), Store, DBPath.

Fencing (replica/fence.go, since v0.6.3) — election alone is coordination-free, so two replicas with divergent membership views can each elect themselves writer for one org (split-brain) and unconditional Store.Put lets both overwrite the mirror. FencedStore closes this over a ConditionalStore (S3 If-Match / GCS generation-match) CAS: one object per key carries a monotone round header, admitted iff round >= recorded, rejected (ErrStaleRound) when below — so a deposed writer's ship is refused once a successor advanced the round. Put(key,data,round) ships; CarryForward(key,round,hydrate) is the safe takeover seal (re-reads + carries the predecessor's last landed write forward atomically via CAS, so no acknowledged write is lost); Get/Round read. Round is a plain uint64 (the storage boundary value); the coordination value ha.Lease{Owner,Round} and the round SOURCE live in ha and its consumers, never here. The concrete ConditionalStore (minio If-Match) lives with the S3-client owner (cloud); this package stays dependency-free over the seam.

Architecture

                  ┌──────────────────────────────┐
                  │   FUSE / WASI mount point    │  /data/
                  └──────────────┬───────────────┘
                                 │ POSIX read/write
                  ┌──────────────▼───────────────┐
                  │   Block layer (4 KiB pages)  │
                  │   Content-addressable        │
                  │   blake3 hash → block ID     │
                  └──────────────┬───────────────┘
                                 │
                       ┌─────────┴─────────┐
                       ▼                   ▼
            ┌──────────────────┐  ┌──────────────────┐
            │  Local NVMe      │  │  Object backend  │
            │  hot cache       │  │  (s3/gcs/azure/  │
            │  (configurable)  │  │   file://)       │
            │  LRU eviction    │  │  PQ-encrypted    │
            └──────────────────┘  └──────────────────┘

Encryption: every block is age-encrypted with hybrid X25519 + ML-KEM-768 recipients (via luxfi/age) before leaving the local cache. Backends never see plaintext.

Content-addressable: each block stored at blocks/<blake3-prefix>/<hash>.zap.age. Identical blocks dedupe naturally.

Layout

vfs/
├── cmd/
│   └── vfs/                 # CLI entry point
│       └── main.go
├── pkg/
│   ├── vfs/
│   │   ├── vfs.go           # Top-level FS interface
│   │   ├── block.go         # 4 KiB block + content-addressable hashing
│   │   ├── cache.go         # LRU write-back cache
│   │   └── crypto.go        # luxfi/age PQ encryption per block
│   ├── backend/
│   │   ├── backend.go       # Backend interface (Get/Put/Delete/List)
│   │   ├── file/file.go     # file:// backend (local dev)
│   │   ├── s3/s3.go         # AWS S3 / S3-compatible backend
│   │   ├── gcs/             # (TODO) Google Cloud Storage
│   │   └── azure/           # (TODO) Azure Blob
│   ├── mount/
│   │   ├── fuse_unix.go     # bazil.org/fuse (build tag: fuse + !windows)
│   │   ├── fuse_stub.go     # stub for builds without FUSE
│   │   └── wasi.go          # (TODO) WASI guest mode
│   └── sidecar/
│       └── sidecar.go       # (TODO) K8s sidecar mode
├── tests/                   # E2E tests
├── docs/                    # Architecture docs
├── go.mod
├── Makefile
├── Dockerfile
└── VERSION

CLI

# Mount an S3-backed VFS (FUSE)
vfs mount /tmp/v \
    --backend s3://bucket/prefix \
    --age-recipient age1xyz... \
    --age-key /etc/vfs/age.key \
    --cache-dir /var/cache/vfs \
    --cache-size 10Gi

# One-shot put/get (no mount; useful for testing)
vfs put /path/to/file --backend file:///tmp/store
vfs get hash --backend file:///tmp/store > /tmp/out

# Stats (cache hit ratio, backend bytes, block count)
vfs stats --cache-dir /var/cache/vfs

K8s sidecar pattern

Same shape as hanzo/replicate — runs alongside Base/SQLite services:

spec:
  containers:
    - name: tenant-bd
      volumeMounts:
        - name: data
          mountPath: /data        # FUSE-mounted VFS
    - name: vfs-sidecar
      image: ghcr.io/hanzoai/vfs:0.1.0
      args:
        - mount
        - /data
        - --backend=s3://tenant-vfs/{env}/bd
        - --age-key=/etc/vfs/age.key
      securityContext:
        privileged: true       # FUSE needs CAP_SYS_ADMIN
      volumeMounts:
        - name: data
          mountPath: /data
          mountPropagation: Bidirectional
        - name: vfs-cache
          mountPath: /var/cache/vfs
        - name: vfs-keys
          mountPath: /etc/vfs
          readOnly: true
  volumes:
    - name: data
      emptyDir: {}              # the VFS overlays this
    - name: vfs-cache
      ephemeral:
        volumeClaimTemplate:
          spec:
            accessModes: [ReadWriteOnce]
            storageClassName: premium-rwo
            resources:
              requests:
                storage: 10Gi   # NVMe hot tier
    - name: vfs-keys
      secret:
        secretName: vfs-age-keys

Backend URL grammar

Scheme	Form	Notes
file://	file:///tmp/store	Local dev; no network.
s3://	s3://bucket/prefix	Region from AWS_REGION env or IRSA. Endpoint override via AWS_ENDPOINT_URL for S3-compat (Hanzo Storage, MinIO, R2).
gcs://	gcs://bucket/prefix	(TODO)
azureblob://	azureblob://account/container/prefix	(TODO)

Encryption

Each 4 KiB block is age-encrypted with one or more recipients (X25519 + optional ML-KEM-768 for hybrid PQ). Recipients are configured at mount time via --age-recipient (repeatable). The decryption key is loaded from --age-key (file path) or VFS_AGE_KEY env var.

Hybrid PQ recipients use luxfi/age extensions (see ~/work/lux/age/internal/x25519 and ~/work/lux/age/pq.go). Plain classical X25519 also works; mix-and-match is supported.

Build tags

• fuse — compile FUSE mount support (requires bazil.org/fuse, Linux/macOS; not Windows)
• wasi — compile WASI guest mode (TODO)

Default go build produces a CLI that supports put/get/stats but not mount. Use make build-fuse for the full mount-capable binary.

Roadmap

Phase	Scope	Status
0.1.0	Block layer, file:// + s3 backends, age PQ crypto, LRU cache, CLI (put/get/stats), tests	✅ shipped
0.2.0	Multi-block File API (ReadAt/WriteAt/Truncate/Sync), FS with inode tree + dirs, persisted metadata blob, SQLite roundtrip proven (20 KiB DB → encrypted blocks → restore → PRAGMA integrity_check ok)	✅ shipped
0.3.0	bazil.org/fuse mount on Linux: kernel POSIX read/write/fsync/setattr land at File.{ReadAt,WriteAt,Sync,Truncate}; SQLite opens DB directly on the mountpoint with no copy step. End-to-end FUSE+SQLite test verifies create→100 INSERTs→close→umount→remount→PRAGMA integrity_check ok, 100 rows survived. macOS via jacobsa/fuse lands in 0.3.1.	✅ shipped
0.3.1	macOS FUSE via github.com/jacobsa/fuse — verified live on macOS with fuse-t: TestFUSESQLite … 100 rows survived umount+remount, integrity_check=ok. Linux uses bazil/fuse, macOS uses jacobsa/fuse, same Mount(*vfs.FS, mountpoint) signature. The fuse build tag picks the right driver per GOOS.	✅ shipped
0.4.0	NVMe disk write-back cache: spill LRU evictions to a local fs cache (configurable via --cache-dir /var/cache/vfs --cache-size 10Gi). Survives process restarts.
0.5.0	gcs + azureblob backends
0.6.0	K8s sidecar mode + Helm chart
0.7.0	WASI guest mode (browser/dev embedding)
1.0.0	Production-hardened: chunked uploads, multipart resume, GC for orphan blocks (block reference-count map at metadata/refs.zap.age), metrics (Prometheus / OTel) + SLO targets

Rules

1. NEVER store plaintext blocks on the backend. Encryption happens before write, decryption after read; the backend interface only sees ciphertext bytes.
2. NEVER allocate disk space proactively for the backend. The whole point is "unlimited" — let S3 deal with capacity.
3. NEVER mix block sizes within one filesystem. 4 KiB is canonical; future formats bump the on-disk magic byte.
4. NEVER skip age recipient verification on read. A block decryption failure is a hard error, not a silent zero.
5. CLAUDE.md / AGENTS.md are symlinks to LLM.md — never commit them.