initial: vfs scaffold (block layer + file:// + s3 backends + age PQ crypto)
Phase 0.1.0 — production-shaped scaffold for the S3-backed virtual block filesystem. Stateful services (IAM, BD, ATS, TA, KMS, onyxd, lqd snapshots) get unlimited disk via S3 with PQ-encrypted blocks; the hot tier is a configurable LRU cache. Companion to hanzo/replicate (WAL streaming) — `replicate` is file-level granularity, `vfs` is block-level (4 KiB pages, content-addressable via blake3-256). Included -------- - pkg/backend: pluggable Backend interface + Open(URL) dispatch. - pkg/backend/file: file:// backend (atomic put via temp+rename, path traversal blocked, recursive list with prefix filter). - pkg/backend/s3: AWS SDK v2 S3 backend with `?endpoint=` override for S3-compat (R2, MinIO, Hanzo Storage). NotFound → ErrNotFound. - pkg/vfs/block: blake3-256 hashing → BlockID, 2-hex shard prefix for the backend key, BlockSize=4096, zero-pad to BlockSize before encryption (block boundaries don't leak file lengths). - pkg/vfs/crypto: luxfi/age v1.5.0 wrapper. Recipients (write-side) + Identities (read-side); hybrid X25519 + ML-KEM-768 supported via the same recipient list. - pkg/vfs/cache: in-memory LRU bound by total bytes; touch-promotes on read; eviction on Put when over cap. NVMe write-back tier lands in 0.2.0. - pkg/vfs: top-level handle. PutBlock/GetBlock with cache hit-path + content-hash integrity check on read. Stats() surfaces cache + backend metadata. - pkg/mount: fuse_stub.go (default build) + fuse_unix.go (-tags fuse). Real FUSE wiring lands in 0.2.0; current stub returns a build-tag error pointing at `make build-fuse`. - cmd/vfs: cobra CLI (put/get/stats/mount). Backend registered via blank-import; --age-recipient repeatable; --age-key reads identity PEM from file or VFS_AGE_KEY env. Tests ----- - block: hash determinism, path shape, verify match/mismatch, pad. - cache: LRU eviction, touch-promotion, basic ops. - file backend: put/get/delete/stat/list, path-traversal block. - vfs roundtrip: encrypt → backend → decrypt → byte-equal plaintext, zero-pad correctness; multiple writes of identical plaintext both round-trip; stats reflect insertions. CI / build ---------- - .github/workflows/build.yml: vet + race-tests on Go 1.24, amd64 binary upload, GHCR image build + push on main / v* tags. - Dockerfile: distroless static, CGO_ENABLED=0, ARG VERSION ldflag. - Makefile: build / build-fuse / test / fmt / vet / check / image / install. VERSION-pinned. Roadmap ------- - 0.2.0: bazil.org/fuse mount, NVMe disk cache layer - 0.3.0: gcs + azureblob backends - 0.4.0: K8s sidecar mode + Helm chart - 0.5.0: WASI guest mode - 1.0.0: chunked uploads, multipart resume, GC for orphan blocks, metrics + SLO targets
This commit is contained in:
Hanzo Dev
@@ -0,0 +1,64 @@
|
||||
name: Build
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
tags: ['v*']
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version: '1.24'
|
||||
cache: true
|
||||
- name: vet
|
||||
run: go vet ./...
|
||||
- name: test
|
||||
run: go test -race ./...
|
||||
|
||||
build-amd64:
|
||||
needs: test
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version: '1.24'
|
||||
cache: true
|
||||
- name: build
|
||||
run: |
|
||||
VERSION=$(tr -d ' \n\r\t' < VERSION)
|
||||
mkdir -p dist
|
||||
CGO_ENABLED=0 go build \
|
||||
-ldflags "-X main.version=${VERSION}" \
|
||||
-o dist/vfs-linux-amd64 ./cmd/vfs
|
||||
- uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: vfs-linux-amd64
|
||||
path: dist/vfs-linux-amd64
|
||||
|
||||
image:
|
||||
needs: test
|
||||
if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v'))
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: login ghcr
|
||||
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
|
||||
- name: build + push
|
||||
run: |
|
||||
VERSION=$(tr -d ' \n\r\t' < VERSION)
|
||||
IMG=ghcr.io/hanzoai/vfs
|
||||
docker build --build-arg VERSION=${VERSION} -t ${IMG}:${VERSION} .
|
||||
docker tag ${IMG}:${VERSION} ${IMG}:latest
|
||||
docker push ${IMG}:${VERSION}
|
||||
docker push ${IMG}:latest
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
bin/
|
||||
vfs
|
||||
*.test
|
||||
*.out
|
||||
.DS_Store
|
||||
.idea/
|
||||
.vscode/
|
||||
dist/
|
||||
.cache/
|
||||
CLAUDE.md
|
||||
AGENTS.md
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
FROM golang:1.24-alpine AS builder
|
||||
WORKDIR /build
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
COPY . .
|
||||
ARG VERSION=dev
|
||||
RUN CGO_ENABLED=0 go build \
|
||||
-ldflags "-X main.version=${VERSION}" \
|
||||
-o /vfs ./cmd/vfs
|
||||
|
||||
FROM gcr.io/distroless/static:nonroot
|
||||
COPY --from=builder /vfs /usr/local/bin/vfs
|
||||
USER nonroot:nonroot
|
||||
ENTRYPOINT ["/usr/local/bin/vfs"]
|
||||
@@ -0,0 +1,17 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
Copyright 2026 Hanzo AI, Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -0,0 +1,193 @@
|
||||
# hanzoai/vfs — AI Engineering Guide
|
||||
|
||||
## What this is
|
||||
|
||||
S3-backed virtual block filesystem. Local-NVMe hot tier + S3 (or any object
|
||||
store) cold tier with PQ encryption per block. Stateful services (IAM, BD,
|
||||
ATS, TA, KMS, onyxd, lqd snapshots) get unlimited disk via S3 without
|
||||
needing K8s PVC sizing decisions.
|
||||
|
||||
Companion to `~/work/hanzo/replicate` — `replicate` streams SQLite WAL
|
||||
frames to S3 (file-level granularity); `vfs` is block-level granularity
|
||||
that transparently spills cold pages.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────┐
|
||||
│ FUSE / WASI mount point │ /data/
|
||||
└──────────────┬───────────────┘
|
||||
│ POSIX read/write
|
||||
┌──────────────▼───────────────┐
|
||||
│ Block layer (4 KiB pages) │
|
||||
│ Content-addressable │
|
||||
│ blake3 hash → block ID │
|
||||
└──────────────┬───────────────┘
|
||||
│
|
||||
┌─────────┴─────────┐
|
||||
▼ ▼
|
||||
┌──────────────────┐ ┌──────────────────┐
|
||||
│ Local NVMe │ │ Object backend │
|
||||
│ hot cache │ │ (s3/gcs/azure/ │
|
||||
│ (configurable) │ │ file://) │
|
||||
│ LRU eviction │ │ PQ-encrypted │
|
||||
└──────────────────┘ └──────────────────┘
|
||||
```
|
||||
|
||||
Encryption: every block is `age`-encrypted with hybrid X25519 + ML-KEM-768
|
||||
recipients (via `luxfi/age`) before leaving the local cache. Backends
|
||||
never see plaintext.
|
||||
|
||||
Content-addressable: each block stored at `blocks/<blake3-prefix>/<hash>.zap.age`.
|
||||
Identical blocks dedupe naturally.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
vfs/
|
||||
├── cmd/
|
||||
│ └── vfs/ # CLI entry point
|
||||
│ └── main.go
|
||||
├── pkg/
|
||||
│ ├── vfs/
|
||||
│ │ ├── vfs.go # Top-level FS interface
|
||||
│ │ ├── block.go # 4 KiB block + content-addressable hashing
|
||||
│ │ ├── cache.go # LRU write-back cache
|
||||
│ │ └── crypto.go # luxfi/age PQ encryption per block
|
||||
│ ├── backend/
|
||||
│ │ ├── backend.go # Backend interface (Get/Put/Delete/List)
|
||||
│ │ ├── file/file.go # file:// backend (local dev)
|
||||
│ │ ├── s3/s3.go # AWS S3 / S3-compatible backend
|
||||
│ │ ├── gcs/ # (TODO) Google Cloud Storage
|
||||
│ │ └── azure/ # (TODO) Azure Blob
|
||||
│ ├── mount/
|
||||
│ │ ├── fuse_unix.go # bazil.org/fuse (build tag: fuse + !windows)
|
||||
│ │ ├── fuse_stub.go # stub for builds without FUSE
|
||||
│ │ └── wasi.go # (TODO) WASI guest mode
|
||||
│ └── sidecar/
|
||||
│ └── sidecar.go # (TODO) K8s sidecar mode
|
||||
├── tests/ # E2E tests
|
||||
├── docs/ # Architecture docs
|
||||
├── go.mod
|
||||
├── Makefile
|
||||
├── Dockerfile
|
||||
└── VERSION
|
||||
```
|
||||
|
||||
## CLI
|
||||
|
||||
```bash
|
||||
# Mount an S3-backed VFS (FUSE)
|
||||
vfs mount /tmp/v \
|
||||
--backend s3://bucket/prefix \
|
||||
--age-recipient age1xyz... \
|
||||
--age-key /etc/vfs/age.key \
|
||||
--cache-dir /var/cache/vfs \
|
||||
--cache-size 10Gi
|
||||
|
||||
# One-shot put/get (no mount; useful for testing)
|
||||
vfs put /path/to/file --backend file:///tmp/store
|
||||
vfs get hash --backend file:///tmp/store > /tmp/out
|
||||
|
||||
# Stats (cache hit ratio, backend bytes, block count)
|
||||
vfs stats --cache-dir /var/cache/vfs
|
||||
```
|
||||
|
||||
## K8s sidecar pattern
|
||||
|
||||
Same shape as `hanzo/replicate` — runs alongside Base/SQLite services:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
containers:
|
||||
- name:
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data # FUSE-mounted VFS
|
||||
- name: vfs-sidecar
|
||||
image: ghcr.io/hanzoai/vfs:0.1.0
|
||||
args:
|
||||
- mount
|
||||
- /data
|
||||
- --backend=s3://liquidity-vfs/{env}/bd
|
||||
- --age-key=/etc/vfs/age.key
|
||||
securityContext:
|
||||
privileged: true # FUSE needs CAP_SYS_ADMIN
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
mountPropagation: Bidirectional
|
||||
- name: vfs-cache
|
||||
mountPath: /var/cache/vfs
|
||||
- name: vfs-keys
|
||||
mountPath: /etc/vfs
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: data
|
||||
emptyDir: {} # the VFS overlays this
|
||||
- name: vfs-cache
|
||||
ephemeral:
|
||||
volumeClaimTemplate:
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: premium-rwo
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi # NVMe hot tier
|
||||
- name: vfs-keys
|
||||
secret:
|
||||
secretName: vfs-age-keys
|
||||
```
|
||||
|
||||
## Backend URL grammar
|
||||
|
||||
| Scheme | Form | Notes |
|
||||
|---|---|---|
|
||||
| `file://` | `file:///tmp/store` | Local dev; no network. |
|
||||
| `s3://` | `s3://bucket/prefix` | Region from `AWS_REGION` env or IRSA. Endpoint override via `AWS_ENDPOINT_URL` for S3-compat (Hanzo Storage, MinIO, R2). |
|
||||
| `gcs://` | `gcs://bucket/prefix` | (TODO) |
|
||||
| `azureblob://` | `azureblob://account/container/prefix` | (TODO) |
|
||||
|
||||
## Encryption
|
||||
|
||||
Each 4 KiB block is age-encrypted with one or more recipients (X25519 +
|
||||
optional ML-KEM-768 for hybrid PQ). Recipients are configured at mount
|
||||
time via `--age-recipient` (repeatable). The decryption key is loaded
|
||||
from `--age-key` (file path) or `VFS_AGE_KEY` env var.
|
||||
|
||||
Hybrid PQ recipients use `luxfi/age` extensions (see
|
||||
`~/work/lux/age/internal/x25519` and `~/work/lux/age/pq.go`). Plain
|
||||
classical X25519 also works; mix-and-match is supported.
|
||||
|
||||
## Build tags
|
||||
|
||||
- `fuse` — compile FUSE mount support (requires `bazil.org/fuse`,
|
||||
Linux/macOS; not Windows)
|
||||
- `wasi` — compile WASI guest mode (TODO)
|
||||
|
||||
Default `go build` produces a CLI that supports `put`/`get`/`stats` but
|
||||
not `mount`. Use `make build-fuse` for the full mount-capable binary.
|
||||
|
||||
## Roadmap
|
||||
|
||||
| Phase | Scope |
|
||||
|---|---|
|
||||
| 0.1.0 (this commit) | Block layer, file:// + s3 backends, age PQ crypto, LRU cache, CLI scaffolding (put/get/stats), tests |
|
||||
| 0.2.0 | FUSE mount via bazil.org/fuse (Linux/macOS) |
|
||||
| 0.3.0 | gcs + azureblob backends |
|
||||
| 0.4.0 | K8s sidecar mode + Helm chart |
|
||||
| 0.5.0 | WASI guest mode (browser/dev embedding) |
|
||||
| 1.0.0 | Production-hardened: chunked uploads, multipart resume, GC for orphan blocks, metrics + SLO targets |
|
||||
|
||||
## Rules
|
||||
|
||||
1. NEVER store plaintext blocks on the backend. Encryption happens
|
||||
before write, decryption after read; the backend interface only
|
||||
sees ciphertext bytes.
|
||||
2. NEVER allocate disk space proactively for the backend. The whole
|
||||
point is "unlimited" — let S3 deal with capacity.
|
||||
3. NEVER mix block sizes within one filesystem. 4 KiB is canonical;
|
||||
future formats bump the on-disk magic byte.
|
||||
4. NEVER skip age recipient verification on read. A block decryption
|
||||
failure is a hard error, not a silent zero.
|
||||
5. CLAUDE.md / AGENTS.md are symlinks to LLM.md — never commit them.
|
||||
@@ -0,0 +1,38 @@
|
||||
SHELL := /usr/bin/env bash
|
||||
VERSION := $(shell tr -d ' \n\r\t' < VERSION)
|
||||
BIN := bin/vfs
|
||||
|
||||
.PHONY: help build test fmt vet check clean install image
|
||||
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
help: ## Show targets
|
||||
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN{FS=":.*?## "}{printf " \033[36m%-12s\033[0m %s\n", $$1, $$2}'
|
||||
|
||||
build: ## Build vfs CLI
|
||||
@mkdir -p bin
|
||||
go build -ldflags "-X main.version=$(VERSION)" -o $(BIN) ./cmd/vfs
|
||||
|
||||
build-fuse: ## Build with FUSE mount support (Linux/macOS)
|
||||
@mkdir -p bin
|
||||
go build -tags=fuse -ldflags "-X main.version=$(VERSION)" -o $(BIN) ./cmd/vfs
|
||||
|
||||
test: ## Run unit tests
|
||||
go test -race ./...
|
||||
|
||||
fmt: ## gofmt
|
||||
gofmt -s -w .
|
||||
|
||||
vet: ## go vet
|
||||
go vet ./...
|
||||
|
||||
check: fmt vet test ## fmt + vet + test
|
||||
|
||||
install: build ## Install to $GOBIN
|
||||
go install -ldflags "-X main.version=$(VERSION)" ./cmd/vfs
|
||||
|
||||
clean: ## Clean build artifacts
|
||||
rm -rf bin/ dist/
|
||||
|
||||
image: ## Build container image
|
||||
docker build --build-arg VERSION=$(VERSION) -t ghcr.io/hanzoai/vfs:$(VERSION) .
|
||||
@@ -0,0 +1,50 @@
|
||||
# vfs
|
||||
|
||||
S3-backed virtual block filesystem with PQ encryption — unlimited write storage for stateful services.
|
||||
|
||||
[](https://github.com/hanzoai/vfs/actions/workflows/build.yml)
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# Build
|
||||
make build
|
||||
|
||||
# Generate an age key (one-time)
|
||||
go run filippo.io/age/cmd/age-keygen > /tmp/vfs.key
|
||||
RECIPIENT=$(grep 'public key' /tmp/vfs.key | cut -d: -f2 | tr -d ' ')
|
||||
|
||||
# Put + get a block (file:// backend, dev only)
|
||||
echo "hello world" > /tmp/in.txt
|
||||
ID=$(./bin/vfs put /tmp/in.txt --backend file:///tmp/vfs-store --age-recipient "$RECIPIENT" --age-key /tmp/vfs.key)
|
||||
echo "block: $ID"
|
||||
./bin/vfs get "$ID" --backend file:///tmp/vfs-store --age-key /tmp/vfs.key
|
||||
|
||||
# Same flow against S3
|
||||
./bin/vfs put /tmp/in.txt \
|
||||
--backend "s3://my-bucket/vfs-prefix?region=us-east-1" \
|
||||
--age-recipient "$RECIPIENT" \
|
||||
--age-key /tmp/vfs.key
|
||||
```
|
||||
|
||||
## What it is
|
||||
|
||||
A 4 KiB block-level virtual filesystem that:
|
||||
|
||||
- Hashes every block with **blake3-256** for content addressability + integrity.
|
||||
- Encrypts every block with **luxfi/age** (X25519, optionally hybrid PQ via ML-KEM-768) before it touches the backend.
|
||||
- Stores blocks as `blocks/<2-hex-shard>/<full-hash>.zap.age` on a pluggable backend (`file://`, `s3://`, `gcs://` (TODO), `azureblob://` (TODO)).
|
||||
- Caches recently-used blocks in an in-memory LRU (NVMe write-back cache lands in 0.2.0).
|
||||
- Mounts as a FUSE filesystem (`make build-fuse`; mount layer lands in 0.2.0).
|
||||
|
||||
## Why
|
||||
|
||||
Stateful services in K8s usually pre-size a PVC and either over-provision or run out of disk. `vfs` lets a service write as if it has unlimited local NVMe — the hot tier is a configurable local cache, the cold tier is S3 (or any object store), and the PQ-encrypted blocks let the operator put cold pages in any region without leaking content.
|
||||
|
||||
Companion to [hanzo/replicate](https://github.com/hanzoai/replicate) — `replicate` streams SQLite WAL frames file-level; `vfs` does block-level for any file system.
|
||||
|
||||
See [`LLM.md`](./LLM.md) for the full architecture, K8s sidecar pattern, encryption design, and roadmap.
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0
|
||||
@@ -0,0 +1,36 @@
|
||||
module github.com/hanzoai/vfs
|
||||
|
||||
go 1.24.0
|
||||
|
||||
require (
|
||||
github.com/aws/aws-sdk-go-v2 v1.30.0
|
||||
github.com/aws/aws-sdk-go-v2/config v1.27.0
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.55.0
|
||||
github.com/luxfi/age v1.5.0
|
||||
github.com/spf13/cobra v1.8.0
|
||||
github.com/zeebo/blake3 v0.2.4
|
||||
)
|
||||
|
||||
require (
|
||||
filippo.io/hpke v0.4.0 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.17.0 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.15.0 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.2 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.10 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.10 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.19.0 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.22.0 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.27.0 // indirect
|
||||
github.com/aws/smithy-go v1.20.2 // indirect
|
||||
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.0.12 // indirect
|
||||
github.com/spf13/pflag v1.0.5 // indirect
|
||||
golang.org/x/crypto v0.45.0 // indirect
|
||||
golang.org/x/sys v0.38.0 // indirect
|
||||
)
|
||||
@@ -0,0 +1,66 @@
|
||||
c2sp.org/CCTV/age v0.0.0-20251208015420-e9274a7bdbfd h1:ZLsPO6WdZ5zatV4UfVpr7oAwLGRZ+sebTUruuM4Ra3M=
|
||||
c2sp.org/CCTV/age v0.0.0-20251208015420-e9274a7bdbfd/go.mod h1:SrHC2C7r5GkDk8R+NFVzYy/sdj0Ypg9htaPXQq5Cqeo=
|
||||
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
|
||||
filippo.io/hpke v0.4.0/go.mod h1:EmAN849/P3qdeK+PCMkDpDm83vRHM5cDipBJ8xbQLVY=
|
||||
github.com/aws/aws-sdk-go-v2 v1.30.0 h1:6qAwtzlfcTtcL8NHtbDQAqgM5s6NDipQTkPxyH/6kAA=
|
||||
github.com/aws/aws-sdk-go-v2 v1.30.0/go.mod h1:ffIFB97e2yNsv4aTSGkqtHnppsIJzw7G7BReUZ3jCXM=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 h1:x6xsQXGSmW6frevwDA+vi/wqhp1ct18mVXYN08/93to=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2/go.mod h1:lPprDr1e6cJdyYeGXnRaJoP4Md+cDBvi2eOj00BlGmg=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.27.0 h1:J5sdGCAHuWKIXLeXiqr8II/adSvetkx0qdZwdbXXpb0=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.27.0/go.mod h1:cfh8v69nuSUohNFMbIISP2fhmblGmYEOKs5V53HiHnk=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.17.0 h1:lMW2x6sKBsiAJrpi1doOXqWFyEPoE886DTb1X0wb7So=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.17.0/go.mod h1:uT41FIH8cCIxOdUYIL0PYyHlL1NoneDuDSCwg5VE/5o=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.15.0 h1:xWCwjjvVz2ojYTP4kBKUuUh9ZrXfcAXpflhOUUeXg1k=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.15.0/go.mod h1:j3fACuqXg4oMTQOR2yY7m0NmJY0yBK4L4sLsRXq1Ins=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.8 h1:RnLB7p6aaFMRfyQkD6ckxR7myCC9SABIqSz4czYUUbU=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.8/go.mod h1:XH7dQJd+56wEbP1I4e4Duo+QhSMxNArE8VP7NuUOTeM=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.8 h1:jzApk2f58L9yW9q1GEab3BMMFWUkkiZhyrRUtbwUbKU=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.8/go.mod h1:WqO+FftfO3tGePUtQxPXM6iODVfqMwsVMgTbG/ZXIdQ=
|
||||
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 h1:hT8rVHwugYE2lEfdFE0QWVo81lF7jMrYJVDWI+f+VxU=
|
||||
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0/go.mod h1:8tu/lYfQfFe6IGnaOdrpVgEL2IrrDOf6/m9RQum4NkY=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.8 h1:jH33S0y5Bo5ZVML62JgZhjd/LrtU+vbR8W7XnIE3Srk=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.8/go.mod h1:hD5YwHLOy6k7d6kqcn3me1bFWHOtzhaXstMd6BpdB68=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.2 h1:Ji0DY1xUsUr3I8cHps0G+XM3WWU16lP6yG8qu1GAZAs=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.2/go.mod h1:5CsjAbs3NlGQyZNFACh+zztPDI7fU6eW9QsxjfnuBKg=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.10 h1:pkYC5zTOSPXEYJj56b2SOik9AL432i5MT1YVTQbKOK0=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.10/go.mod h1:/WNsBOlKWZCG3PMh2aSp8vkyyT/clpMZqOtrnIKqGfk=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.10 h1:7kZqP7akv0enu6ykJhb9OYlw16oOrSy+Epus8o/VqMY=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.10/go.mod h1:gYVF3nM1ApfTRDj9pvdhootBb8WbiIejuqn4w8ruMes=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.8 h1:iQNXVs1vtaq+y9M90M4ZIVNORje0qXTscqHLqoOnFS0=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.8/go.mod h1:yUQPRlWqGG0lfNsmjbRWKVwgilfBtZTOFSLEYALlAig=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.55.0 h1:6kq0Xql9qiwNGL/Go87ZqR4otg9jnKs71OfWCVbPxLM=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.55.0/go.mod h1:oSkRFuHVWmUY4Ssk16ErGzBqvYEbvORJFzFXzWhTB2s=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.19.0 h1:u6OkVDxtBPnxPkZ9/63ynEe+8kHbtS5IfaC4PzVxzWM=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.19.0/go.mod h1:YqbU3RS/pkDVu+v+Nwxvn0i1WB0HkNWEePWbmODEbbs=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.22.0 h1:6DL0qu5+315wbsAEEmzK+P9leRwNbkp+lGjPC+CEvb8=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.22.0/go.mod h1:olUAyg+FaoFaL/zFaeQQONjOZ9HXoxgvI/c7mQTYz7M=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.27.0 h1:cjTRjh700H36MQ8M0LnDn33W3JmwC77mdxIIyPWCdpM=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.27.0/go.mod h1:nXfOBMWPokIbOY+Gi7a1psWMSvskUCemZzI+SMB7Akc=
|
||||
github.com/aws/smithy-go v1.20.2 h1:tbp628ireGtzcHDDmLT/6ADHidqnwgF57XOXZe6tp4Q=
|
||||
github.com/aws/smithy-go v1.20.2/go.mod h1:krry+ya/rV9RDcV/Q16kpu6ypI4K2czasz0NC3qS14E=
|
||||
github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
|
||||
github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
|
||||
github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
||||
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
|
||||
github.com/klauspost/cpuid/v2 v2.0.12 h1:p9dKCg8i4gmOxtv35DvrYoWqYzQrvEVdjQ762Y0OqZE=
|
||||
github.com/klauspost/cpuid/v2 v2.0.12/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
|
||||
github.com/luxfi/age v1.5.0 h1:G69HbSV4R3vKEH9B0CulnRaMdSdf4RalMgP8xKmxHeI=
|
||||
github.com/luxfi/age v1.5.0/go.mod h1:iAYAxgvrXxcy746+Ovh/eWWDuF9teJLNcCSSOX9RYW0=
|
||||
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
||||
github.com/spf13/cobra v1.8.0 h1:7aJaZx1B85qltLMc546zn58BxxfZdR/W22ej9CFoEf0=
|
||||
github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyhcho=
|
||||
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
|
||||
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
github.com/zeebo/assert v1.1.0 h1:hU1L1vLTHsnO8x8c9KAR5GmM5QscxHg5RNU5z5qbUWY=
|
||||
github.com/zeebo/assert v1.1.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
|
||||
github.com/zeebo/blake3 v0.2.4 h1:KYQPkhpRtcqh0ssGYcKLG1JYvddkEA8QwCM/yBqhaZI=
|
||||
github.com/zeebo/blake3 v0.2.4/go.mod h1:7eeQ6d2iXWRGF6npfaxl2CU+xy2Fjo2gxeyZGCRUjcE=
|
||||
github.com/zeebo/pcg v1.0.1 h1:lyqfGeWiv4ahac6ttHs+I5hwtH/+1mrhlCtVNQM2kHo=
|
||||
github.com/zeebo/pcg v1.0.1/go.mod h1:09F0S9iiKrwn9rlI5yjLkmrug154/YRW6KnnXVDM/l4=
|
||||
golang.org/x/crypto v0.45.0 h1:jMBrvKuj23MTlT0bQEOBcAE0mjg8mK9RXFhRH6nyF3Q=
|
||||
golang.org/x/crypto v0.45.0/go.mod h1:XTGrrkGJve7CYK7J8PEww4aY7gM3qMCElcJQ8n8JdX4=
|
||||
golang.org/x/sys v0.38.0 h1:3yZWxaJjBmCWXqhN1qh02AkOnCQ1poK6oF+a7xWL6Gc=
|
||||
golang.org/x/sys v0.38.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,115 @@
|
||||
// Package backend defines the object-store interface that VFS writes
|
||||
// encrypted blocks against. Implementations cover S3, GCS, Azure Blob,
|
||||
// and a `file://` adapter for local dev. The interface is intentionally
|
||||
// minimal: VFS does its own content addressing and encryption above
|
||||
// this layer; the backend is just a typed key→bytes store.
|
||||
package backend
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/url"
|
||||
)
|
||||
|
||||
// ErrNotFound is returned by Get when the requested key is absent.
|
||||
var ErrNotFound = errors.New("backend: key not found")
|
||||
|
||||
// Backend is the contract every object-store adapter implements.
|
||||
//
|
||||
// Keys are arbitrary `/`-delimited strings; VFS uses
|
||||
// `blocks/<blake3-prefix>/<full-hash>.zap.age` for block storage.
|
||||
//
|
||||
// Implementations MUST be safe for concurrent use. Get/Put/Delete are
|
||||
// idempotent — repeating the same operation is allowed and SHOULD NOT
|
||||
// surface as an error.
|
||||
type Backend interface {
|
||||
// Get returns the bytes stored at key. If the key does not exist,
|
||||
// the returned error wraps ErrNotFound (errors.Is(err, ErrNotFound)).
|
||||
Get(ctx context.Context, key string) ([]byte, error)
|
||||
|
||||
// Put writes data at key. Overwriting is allowed.
|
||||
Put(ctx context.Context, key string, data []byte) error
|
||||
|
||||
// Delete removes the key. Removing a non-existent key MUST NOT error.
|
||||
Delete(ctx context.Context, key string) error
|
||||
|
||||
// List returns keys with the given prefix. The returned channel is
|
||||
// closed when the listing completes; on error the channel is closed
|
||||
// after the error is sent on errCh.
|
||||
List(ctx context.Context, prefix string) (keys <-chan string, errCh <-chan error)
|
||||
|
||||
// Stat returns the size in bytes for key without fetching the body.
|
||||
// ErrNotFound is returned if the key is absent.
|
||||
Stat(ctx context.Context, key string) (size int64, err error)
|
||||
|
||||
// Close releases any resources (HTTP connections, file handles).
|
||||
// Safe to call multiple times.
|
||||
Close() error
|
||||
|
||||
// String returns a stable, human-readable description of the
|
||||
// backend (e.g. "s3://bucket/prefix").
|
||||
String() string
|
||||
}
|
||||
|
||||
// Reader returns an io.ReadCloser when streaming is preferred over
|
||||
// in-memory Get. Optional — implementations may return ErrNotImplemented.
|
||||
type Reader interface {
|
||||
GetReader(ctx context.Context, key string) (io.ReadCloser, error)
|
||||
}
|
||||
|
||||
// Writer returns an io.WriteCloser for streaming uploads. Optional.
|
||||
type Writer interface {
|
||||
PutWriter(ctx context.Context, key string) (io.WriteCloser, error)
|
||||
}
|
||||
|
||||
// ErrNotImplemented signals that a backend doesn't support the optional
|
||||
// streaming interface for the given operation.
|
||||
var ErrNotImplemented = errors.New("backend: operation not implemented")
|
||||
|
||||
// Open dispatches a backend URL to the right implementation. Recognised
|
||||
// schemes:
|
||||
//
|
||||
// file://path — local filesystem (dev only)
|
||||
// s3://bucket/prefix — AWS S3 / S3-compatible (R2, Hanzo Storage, MinIO)
|
||||
// gcs://bucket/prefix — Google Cloud Storage (TODO)
|
||||
// azureblob://... — Azure Blob (TODO)
|
||||
//
|
||||
// The opener function is registered by each backend package's init().
|
||||
type Opener func(ctx context.Context, u *url.URL) (Backend, error)
|
||||
|
||||
var openers = map[string]Opener{}
|
||||
|
||||
// Register attaches an Opener for a URL scheme. Called from backend
|
||||
// implementations' init().
|
||||
func Register(scheme string, opener Opener) {
|
||||
if scheme == "" || opener == nil {
|
||||
panic("backend.Register: scheme and opener are required")
|
||||
}
|
||||
openers[scheme] = opener
|
||||
}
|
||||
|
||||
// Open parses the backend URL and dispatches to the registered opener.
|
||||
func Open(ctx context.Context, raw string) (Backend, error) {
|
||||
u, err := url.Parse(raw)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("backend.Open: parse %q: %w", raw, err)
|
||||
}
|
||||
if u.Scheme == "" {
|
||||
return nil, fmt.Errorf("backend.Open: missing scheme in %q", raw)
|
||||
}
|
||||
opener, ok := openers[u.Scheme]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("backend.Open: no opener registered for %q (have: %v)", u.Scheme, registeredSchemes())
|
||||
}
|
||||
return opener(ctx, u)
|
||||
}
|
||||
|
||||
func registeredSchemes() []string {
|
||||
out := make([]string, 0, len(openers))
|
||||
for k := range openers {
|
||||
out = append(out, k)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,177 @@
|
||||
// Package file is a filesystem-backed Backend for local dev. NOT
|
||||
// suitable for production — no fsync barrier discipline, no checksum.
|
||||
package file
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/hanzoai/vfs/pkg/backend"
|
||||
)
|
||||
|
||||
func init() {
|
||||
backend.Register("file", openFile)
|
||||
}
|
||||
|
||||
type fileBackend struct {
|
||||
root string
|
||||
|
||||
closeOnce sync.Once
|
||||
closed bool
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
func openFile(_ context.Context, u *url.URL) (backend.Backend, error) {
|
||||
// file:///abs/path → u.Path = "/abs/path"
|
||||
// file://./relpath → u.Host = ".", u.Path = "/relpath"
|
||||
root := u.Path
|
||||
if u.Host != "" && u.Host != "localhost" {
|
||||
root = filepath.Join(u.Host, u.Path)
|
||||
}
|
||||
if root == "" {
|
||||
return nil, fmt.Errorf("file backend: missing path in %q", u.String())
|
||||
}
|
||||
if err := os.MkdirAll(root, 0o755); err != nil {
|
||||
return nil, fmt.Errorf("file backend: mkdir %q: %w", root, err)
|
||||
}
|
||||
return &fileBackend{root: root}, nil
|
||||
}
|
||||
|
||||
func (f *fileBackend) keyPath(key string) string {
|
||||
// Disallow "..". Backend keys are simple `/`-delimited names; never user paths.
|
||||
if strings.Contains(key, "..") {
|
||||
return ""
|
||||
}
|
||||
return filepath.Join(f.root, filepath.FromSlash(key))
|
||||
}
|
||||
|
||||
func (f *fileBackend) Get(_ context.Context, key string) ([]byte, error) {
|
||||
p := f.keyPath(key)
|
||||
if p == "" {
|
||||
return nil, fmt.Errorf("file backend: invalid key %q", key)
|
||||
}
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil, fmt.Errorf("%w: %s", backend.ErrNotFound, key)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
|
||||
func (f *fileBackend) Put(_ context.Context, key string, data []byte) error {
|
||||
p := f.keyPath(key)
|
||||
if p == "" {
|
||||
return fmt.Errorf("file backend: invalid key %q", key)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
// Atomic write: temp file + rename.
|
||||
tmp, err := os.CreateTemp(filepath.Dir(p), ".vfs-"+filepath.Base(p)+"-*")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() {
|
||||
_ = os.Remove(tmp.Name())
|
||||
}()
|
||||
if _, err := tmp.Write(data); err != nil {
|
||||
_ = tmp.Close()
|
||||
return err
|
||||
}
|
||||
if err := tmp.Sync(); err != nil {
|
||||
_ = tmp.Close()
|
||||
return err
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp.Name(), p)
|
||||
}
|
||||
|
||||
func (f *fileBackend) Delete(_ context.Context, key string) error {
|
||||
p := f.keyPath(key)
|
||||
if p == "" {
|
||||
return fmt.Errorf("file backend: invalid key %q", key)
|
||||
}
|
||||
if err := os.Remove(p); err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fileBackend) List(ctx context.Context, prefix string) (<-chan string, <-chan error) {
|
||||
keys := make(chan string, 64)
|
||||
errCh := make(chan error, 1)
|
||||
go func() {
|
||||
defer close(keys)
|
||||
defer close(errCh)
|
||||
base := f.root
|
||||
walkRoot := base
|
||||
if prefix != "" {
|
||||
walkRoot = filepath.Join(base, filepath.FromSlash(prefix))
|
||||
}
|
||||
err := filepath.WalkDir(walkRoot, func(path string, d os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil // empty prefix
|
||||
}
|
||||
return err
|
||||
}
|
||||
if d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
rel, rerr := filepath.Rel(base, path)
|
||||
if rerr != nil {
|
||||
return rerr
|
||||
}
|
||||
rel = filepath.ToSlash(rel)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case keys <- rel:
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil && !errors.Is(err, context.Canceled) {
|
||||
errCh <- err
|
||||
}
|
||||
}()
|
||||
return keys, errCh
|
||||
}
|
||||
|
||||
func (f *fileBackend) Stat(_ context.Context, key string) (int64, error) {
|
||||
p := f.keyPath(key)
|
||||
if p == "" {
|
||||
return 0, fmt.Errorf("file backend: invalid key %q", key)
|
||||
}
|
||||
st, err := os.Stat(p)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return 0, fmt.Errorf("%w: %s", backend.ErrNotFound, key)
|
||||
}
|
||||
return 0, err
|
||||
}
|
||||
return st.Size(), nil
|
||||
}
|
||||
|
||||
func (f *fileBackend) Close() error {
|
||||
f.closeOnce.Do(func() { f.closed = true })
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fileBackend) String() string { return "file://" + f.root }
|
||||
|
||||
// Compile-time interface check.
|
||||
var _ backend.Backend = (*fileBackend)(nil)
|
||||
|
||||
// Unused but reserved for streaming when we need it.
|
||||
type _ = io.Reader
|
||||
@@ -0,0 +1,96 @@
|
||||
package file
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/hanzoai/vfs/pkg/backend"
|
||||
)
|
||||
|
||||
func newFileBE(t *testing.T) (backend.Backend, string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
be, err := backend.Open(context.Background(), "file://"+filepath.ToSlash(dir))
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = be.Close() })
|
||||
return be, dir
|
||||
}
|
||||
|
||||
func TestFilePutGetDelete(t *testing.T) {
|
||||
be, _ := newFileBE(t)
|
||||
ctx := context.Background()
|
||||
|
||||
if err := be.Put(ctx, "blocks/aa/abc.zap.age", []byte("hello")); err != nil {
|
||||
t.Fatalf("put: %v", err)
|
||||
}
|
||||
got, err := be.Get(ctx, "blocks/aa/abc.zap.age")
|
||||
if err != nil {
|
||||
t.Fatalf("get: %v", err)
|
||||
}
|
||||
if string(got) != "hello" {
|
||||
t.Fatalf("got %q want hello", got)
|
||||
}
|
||||
|
||||
sz, err := be.Stat(ctx, "blocks/aa/abc.zap.age")
|
||||
if err != nil {
|
||||
t.Fatalf("stat: %v", err)
|
||||
}
|
||||
if sz != 5 {
|
||||
t.Fatalf("stat size %d want 5", sz)
|
||||
}
|
||||
|
||||
if err := be.Delete(ctx, "blocks/aa/abc.zap.age"); err != nil {
|
||||
t.Fatalf("delete: %v", err)
|
||||
}
|
||||
if _, err := be.Get(ctx, "blocks/aa/abc.zap.age"); !errors.Is(err, backend.ErrNotFound) {
|
||||
t.Fatalf("expected ErrNotFound, got %v", err)
|
||||
}
|
||||
|
||||
// Delete-of-absent must not error
|
||||
if err := be.Delete(ctx, "blocks/zz/missing.zap.age"); err != nil {
|
||||
t.Fatalf("idempotent delete: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileList(t *testing.T) {
|
||||
be, _ := newFileBE(t)
|
||||
ctx := context.Background()
|
||||
|
||||
keys := []string{
|
||||
"blocks/aa/1.zap.age",
|
||||
"blocks/aa/2.zap.age",
|
||||
"blocks/bb/3.zap.age",
|
||||
}
|
||||
for _, k := range keys {
|
||||
if err := be.Put(ctx, k, []byte("x")); err != nil {
|
||||
t.Fatalf("put %s: %v", k, err)
|
||||
}
|
||||
}
|
||||
|
||||
ch, errCh := be.List(ctx, "blocks/aa")
|
||||
got := map[string]bool{}
|
||||
for k := range ch {
|
||||
got[k] = true
|
||||
}
|
||||
if err := <-errCh; err != nil {
|
||||
t.Fatalf("list err: %v", err)
|
||||
}
|
||||
if !got["blocks/aa/1.zap.age"] || !got["blocks/aa/2.zap.age"] {
|
||||
t.Fatalf("missing aa keys: %v", got)
|
||||
}
|
||||
if got["blocks/bb/3.zap.age"] {
|
||||
t.Fatalf("bb key leaked into aa prefix list: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileTraversalBlocked(t *testing.T) {
|
||||
be, _ := newFileBE(t)
|
||||
ctx := context.Background()
|
||||
if err := be.Put(ctx, "../../../etc/evil", []byte("x")); err == nil {
|
||||
t.Fatal("path traversal should be blocked")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,168 @@
|
||||
// Package s3 is an AWS S3 / S3-compatible Backend.
|
||||
//
|
||||
// Endpoint override (R2, Hanzo Storage, MinIO) via AWS_ENDPOINT_URL or
|
||||
// the URL's `endpoint` query parameter.
|
||||
package s3
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
awsv2 "github.com/aws/aws-sdk-go-v2/aws"
|
||||
"github.com/aws/aws-sdk-go-v2/config"
|
||||
s3v2 "github.com/aws/aws-sdk-go-v2/service/s3"
|
||||
s3types "github.com/aws/aws-sdk-go-v2/service/s3/types"
|
||||
|
||||
"github.com/hanzoai/vfs/pkg/backend"
|
||||
)
|
||||
|
||||
func init() {
|
||||
backend.Register("s3", openS3)
|
||||
}
|
||||
|
||||
type s3Backend struct {
|
||||
client *s3v2.Client
|
||||
bucket string
|
||||
prefix string
|
||||
desc string
|
||||
|
||||
closeOnce sync.Once
|
||||
}
|
||||
|
||||
func openS3(ctx context.Context, u *url.URL) (backend.Backend, error) {
|
||||
bucket := u.Host
|
||||
if bucket == "" {
|
||||
return nil, fmt.Errorf("s3 backend: missing bucket in %q", u.String())
|
||||
}
|
||||
prefix := strings.TrimPrefix(u.Path, "/")
|
||||
|
||||
loadOpts := []func(*config.LoadOptions) error{}
|
||||
if region := u.Query().Get("region"); region != "" {
|
||||
loadOpts = append(loadOpts, config.WithRegion(region))
|
||||
}
|
||||
|
||||
cfg, err := config.LoadDefaultConfig(ctx, loadOpts...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("s3 backend: load config: %w", err)
|
||||
}
|
||||
|
||||
clientOpts := []func(*s3v2.Options){}
|
||||
if endpoint := u.Query().Get("endpoint"); endpoint != "" {
|
||||
clientOpts = append(clientOpts, func(o *s3v2.Options) {
|
||||
o.BaseEndpoint = awsv2.String(endpoint)
|
||||
o.UsePathStyle = true
|
||||
})
|
||||
}
|
||||
|
||||
cli := s3v2.NewFromConfig(cfg, clientOpts...)
|
||||
return &s3Backend{
|
||||
client: cli,
|
||||
bucket: bucket,
|
||||
prefix: prefix,
|
||||
desc: u.String(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (s *s3Backend) k(key string) string {
|
||||
if s.prefix == "" {
|
||||
return key
|
||||
}
|
||||
return s.prefix + "/" + key
|
||||
}
|
||||
|
||||
func (s *s3Backend) Get(ctx context.Context, key string) ([]byte, error) {
|
||||
out, err := s.client.GetObject(ctx, &s3v2.GetObjectInput{
|
||||
Bucket: awsv2.String(s.bucket),
|
||||
Key: awsv2.String(s.k(key)),
|
||||
})
|
||||
if err != nil {
|
||||
var nsk *s3types.NoSuchKey
|
||||
if errors.As(err, &nsk) {
|
||||
return nil, fmt.Errorf("%w: %s", backend.ErrNotFound, key)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
defer out.Body.Close()
|
||||
return io.ReadAll(out.Body)
|
||||
}
|
||||
|
||||
func (s *s3Backend) Put(ctx context.Context, key string, data []byte) error {
|
||||
_, err := s.client.PutObject(ctx, &s3v2.PutObjectInput{
|
||||
Bucket: awsv2.String(s.bucket),
|
||||
Key: awsv2.String(s.k(key)),
|
||||
Body: bytes.NewReader(data),
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *s3Backend) Delete(ctx context.Context, key string) error {
|
||||
_, err := s.client.DeleteObject(ctx, &s3v2.DeleteObjectInput{
|
||||
Bucket: awsv2.String(s.bucket),
|
||||
Key: awsv2.String(s.k(key)),
|
||||
})
|
||||
// S3 DeleteObject is idempotent — absent keys are not an error.
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *s3Backend) List(ctx context.Context, prefix string) (<-chan string, <-chan error) {
|
||||
keys := make(chan string, 256)
|
||||
errCh := make(chan error, 1)
|
||||
go func() {
|
||||
defer close(keys)
|
||||
defer close(errCh)
|
||||
fullPrefix := s.k(prefix)
|
||||
paginator := s3v2.NewListObjectsV2Paginator(s.client, &s3v2.ListObjectsV2Input{
|
||||
Bucket: awsv2.String(s.bucket),
|
||||
Prefix: awsv2.String(fullPrefix),
|
||||
})
|
||||
for paginator.HasMorePages() {
|
||||
page, err := paginator.NextPage(ctx)
|
||||
if err != nil {
|
||||
errCh <- err
|
||||
return
|
||||
}
|
||||
for _, obj := range page.Contents {
|
||||
k := awsv2.ToString(obj.Key)
|
||||
if s.prefix != "" {
|
||||
k = strings.TrimPrefix(k, s.prefix+"/")
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case keys <- k:
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
return keys, errCh
|
||||
}
|
||||
|
||||
func (s *s3Backend) Stat(ctx context.Context, key string) (int64, error) {
|
||||
out, err := s.client.HeadObject(ctx, &s3v2.HeadObjectInput{
|
||||
Bucket: awsv2.String(s.bucket),
|
||||
Key: awsv2.String(s.k(key)),
|
||||
})
|
||||
if err != nil {
|
||||
var nfe *s3types.NotFound
|
||||
if errors.As(err, &nfe) {
|
||||
return 0, fmt.Errorf("%w: %s", backend.ErrNotFound, key)
|
||||
}
|
||||
return 0, err
|
||||
}
|
||||
return awsv2.ToInt64(out.ContentLength), nil
|
||||
}
|
||||
|
||||
func (s *s3Backend) Close() error {
|
||||
s.closeOnce.Do(func() {})
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *s3Backend) String() string { return s.desc }
|
||||
|
||||
var _ backend.Backend = (*s3Backend)(nil)
|
||||
@@ -0,0 +1,21 @@
|
||||
//go:build !fuse
|
||||
|
||||
// Package mount is the FUSE/WASI mount layer. Without the `fuse` build
|
||||
// tag this is a stub — `vfs mount` returns a clear error pointing at
|
||||
// `make build-fuse`.
|
||||
package mount
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/hanzoai/vfs/pkg/vfs"
|
||||
)
|
||||
|
||||
// Mount returns a build-tag error. Build with `-tags fuse` to enable.
|
||||
func Mount(ctx context.Context, v *vfs.VFS, mountpoint string) error {
|
||||
_ = ctx
|
||||
_ = v
|
||||
_ = mountpoint
|
||||
return fmt.Errorf("mount: this binary was built without FUSE support — use `make build-fuse` or `go build -tags fuse`")
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
//go:build fuse && !windows
|
||||
|
||||
package mount
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/hanzoai/vfs/pkg/vfs"
|
||||
)
|
||||
|
||||
// Mount mounts a VFS at the given mountpoint via FUSE. Requires the
|
||||
// `fuse` build tag and the bazil.org/fuse runtime (Linux/macOS).
|
||||
//
|
||||
// TODO(0.2.0): full FUSE filesystem implementation. This stub
|
||||
// compiles under the `fuse` tag but returns a not-yet-implemented
|
||||
// error. Wiring requires:
|
||||
// 1. bazil.org/fuse + bazil.org/fuse/fs imports (added to go.mod
|
||||
// under the fuse tag — keep the default build dependency-light)
|
||||
// 2. an FS root + Node implementations that map POSIX ops to
|
||||
// vfs.PutBlock/GetBlock with a per-file inode + extent index
|
||||
// stored in the same VFS as a special "metadata/" key prefix
|
||||
// 3. proper handling of unmount on Ctrl-C + SIGTERM
|
||||
// 4. tests against a tmpfs-backed file:// VFS on Linux CI
|
||||
func Mount(ctx context.Context, v *vfs.VFS, mountpoint string) error {
|
||||
_ = ctx
|
||||
_ = v
|
||||
_ = mountpoint
|
||||
return fmt.Errorf("mount: FUSE implementation pending (0.2.0 milestone). " +
|
||||
"PutBlock/GetBlock work today via the CLI; mount-as-filesystem ships next")
|
||||
}
|
||||
Reference in New Issue
Block a user