StartList returns ListBuilder by value (mirrors the value-typed ObjectBuilder):
Add*/Finish keep pointer receivers, every caller binds to an addressable local so
Go auto-addresses it. Dropped the dead elemSize field (never read — stride is
implicit in which Add* is called). Byte-identical wire (golden+fuzz+all round-trips
green). NOTE: not a measured alloc win — escape analysis already stack-allocated
the &ListBuilder{} at all 9 call sites (composite stays 583ns/5allocs, the 5 being
mandatory writeEnvelopePrefix pool-egress copies). Value is code-cleanliness +
consistency.
Also fixes zap/v1 subpackage: v1.2.4's ObjectBuilder value change left v1/*.go
referencing *zap.ObjectBuilder → the subpackage didn't build (latent; only the
root package is imported downstream so it went unnoticed). Converted to the value
type. Full module (root + v1) builds + tests green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Decomplect that unlocks a perf win. SetBytes deferred its data into an offsets
[]offsetEntry slice replayed in Finish — machinery that recorded intentions to
replay later, and heap-allocated a slice per object-with-a-bytes-field. Replace
it with directness: StartObject RESERVES the fixed section eagerly, so SetBytes
appends its tail + patches its own pointer on the spot; Finish just returns the
offset. Removing the offsets field makes ObjectBuilder's methods mutate only
ob.b (through the pointer) and never ob itself — so ObjectBuilder becomes a
VALUE type and StartObject stops heap-allocating &ObjectBuilder per call (it was
65% of composite-build allocs).
Byte-identical wire (golden + fuzz + pool byte-equality + nested-proof suites all
green). Real X-tx wire composite: 922ns/11allocs -> 655ns/5allocs (1.4x, <half
the allocs); vs the original byte-blob 2551ns/37allocs that's 3.9x / 7.4x fewer
allocs. Speeds EVERY zap object build node-wide, not just X.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
AddObjectPtr (build a repeated-message field as pointer slots) had no matching
typed reader — callers had to hand-deref the 4-byte signed rel. List.ObjectPtr(i)
is that reader: absOffset = slotPos + int32(rel), same rule as Object.Object,
with the HeaderSize/bounds guards. This is the primitive that lets a tx be built
as ONE native-nested ZAP object instead of byte-blob envelope concat.
nested_tx_proof_test.go: reference design for decomplecting utxo/wire — an
X-chain 2in/2out money-move built as a single nested object (SetObject +
AddObjectPtr out/in lists + inline uint16 fx-discriminator preserving fx
polymorphism). Measured vs the byte-blob envelope pattern it replaces:
build 1345ns/19allocs -> 291ns/0allocs (4.6x, ZERO alloc via pooled builder)
parse 4114ns/26allocs -> 104ns/2allocs (~40x; 2 allocs are result slices)
Round-trip + pooled byte-identical proven.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
SetBytes was heap-copying every payload (append([]byte(nil), v...)) at every
nesting level — the dominant cost of deep wire composition (X-chain tx build
runs ~11 nested builders). It now retains the caller's slice; the caller keeps
it valid until Finish (every in-tree constructor builds+finishes in one frame).
Byte output is unchanged: Finish writes payloads in the same SetBytes order.
GetBuilder/PutBuilder: exported sync.Pool for Builders (write-side counterpart
to the read bufpool). Reuse is unconditionally byte-safe — ensureField zero-
fills every extended span and align zero-fills padding; proven by
TestBuilderPool_ByteIdentical (dirty pooled builder emits bytes identical to a
fresh one). go.sum: re-record luxfi/pq v1.0.3 (upstream re-tag drifted content).
Micro: sample object build 354ns/9allocs -> 257ns/6allocs; the real win
compounds in utxo/wire + xvm envelope composition (next commits).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Decouples WHO (Destination) from HOW (Interface) from the message (Payload):
Send picks the cheapest Interface whose CanReach(dst) is true. When the dst
lives in this binary, InProcessInterface (Cost 0) delivers the LIVE Value to a
local handler — zero serialize, zero copy, zero socket; Encode runs only on a
network hop. NodeInterface adapts the existing *Node wire (unchanged) as the
Cost-10 fallback. Named Router to stay orthogonal to the Transport wire enum
and the transport/ GPU subpackage. First consumer: hanzo/cloud o11y telemetry.
- router.go: Router, Destination, Payload, Interface, LocalHandler, ErrNoRoute
- interface.go: InProcessInterface (Cost 0) + NodeInterface (Cost 10, ErrNotEncodable)
- router_test.go: 7 tests incl. zero-serialize proof (Encode never called in-process)
Executable core of LP-202 §5.2 (three-class stream demultiplexing) and
§6.1 (post-quantum channel). One QUIC connection carries the unified
transport's three stream classes — 'R' RPC, 'C' consensus-P2P, 'Z'
ZAP-native — each routed to its own dispatcher by the one-byte class
prefix (the only new byte on the wire), and asserts the connection
negotiated X25519MLKEM768 (IANA 0x11EC) on both sides.
Additive test only; exercises luxfi/zap alone, touches nothing in luxd
consensus or proposer VM. Runs green (CGO_ENABLED=0 in this env; the
default-CGO failure is an unrelated accel/SDK-header issue in the dep
graph, not this test).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Sets Schema.SkipRegistry so a per-service schema set (e.g. Hanzo S3's
Get/PutObject kinds) does NOT register into the shared zapv1.DefaultRegistry —
their kind bytes are unique only within the service, not globally. Needed for
the ZAP service-mesh pattern where every service owns a private kind space.
The repeated-message wire shape: a list whose elements carry their own
variable-length tails (string/bytes/nested), which the fixed-stride inline
List cannot hold. Completes proto field-shape coverage in the runtime —
map<k,v> reduces to a repeated {key,value} entry, so this is its keystone too.
Composition, not a new wire concept: ListNested = List ∘ NestedAt. The list
slots are a contiguous stride-4 object-pointer array; each pointer is
followed exactly as a singular nested pointer to an out-of-line tailed View.
Runtime:
- builder.go: ListBuilder.AddObjectPtr(targetPos) — appends a signed 4-byte
relative object pointer (the repeated-message element kind).
- v1/list_nested.go: ListNested[N] (Len/At/All/IsZero), ListNestedAt[S,N]
(read), WriteListNested[S,N] + NestedElemSetter[N] (write). Same audited
bounds discipline as Object.Object / ListStride (RED-HIGH-1/2 inherited).
Codegen:
- elemIsTailed: a list element with a string/bytes/nested/list sub-field
auto-routes to ListNested/WriteListNested/ListNestedAt; scalar-only
elements stay in the tighter inline List/WriteList/ListAt. One Elem
declaration, the generator picks the correct encoding.
Proven end-to-end: runtime roundtrip (3 elements each with its OWN string
tail, via At and All, + empty), emit routing test (tailed->ListNested,
scalar-only->List), and a generated-code wire roundtrip (testpkg/repeatwire).
Root zap + full v1+codegen suite green; scalar listwire unchanged (no regression).
Completes the variable-length codegen (strings/bytes/lists/nested) — the
last shape needed before protoc-gen-zap can map every proto message field
mechanically.
Runtime (v1/nested.go):
- NestedAt[S,N]: read an out-of-line 4-byte object pointer as a typed
View[N]; null/OOB -> zero View (proto3 unset).
- WriteNested[S,N]: build a flat nested object into the parent tail and
patch the pointer; the singular peer of WriteList. Composes recursively
(nested may carry scalars, strings, bytes, lists, further nesting).
Codegen (schema.go, emit.go):
- Field.Nested *NestedMsg (4-byte object pointer, distinct from the 8-byte
list/tail pointer); IsNested().
- Constructor takes *Value (nil => null), nil-guarded WriteNested build.
- emitNestedSupport: value struct + NestedAt read accessor.
- emitElemWrites: shared scalar/string/bytes element-write body, so list
elements now also support string/bytes sub-fields (repeated-of-message).
- ListElem/NestedMsg carry Wire (element WireName) for Offset<Wire>_<F>.
Proven end-to-end: runtime roundtrip (nested w/ own string tail + null
case), generator-output tests (go/parser-valid), and a generated-code wire
roundtrip (testpkg/nestwire). Full v1+codegen suite green, no regression.
Completes the variable-length codegen (strings/bytes already shipped; lists now
join them). A list field of a fixed-size element schema generates: a per-element
value-input struct, a typed constructor param []Elem, an inline WriteList build
(bridged onto the hand-rolled ObjectBuilder by the new zapv1.SetterFrom, one
zapv1.Write per element field), and a zero-copy ListAt read accessor returning
zapv1.List[E] (range-over-func ready) — mechanically reproducing examples/
batch_tx.go.
Adds: Schema.Element (flat list-element slots carry no kind byte — fields may
start at offset 0; Wrap skips the kind check); runtime helpers SetterFrom /
WriteString / WriteBytes bridging the hand-rolled fast path to the typed writers.
Proven end-to-end: testpkg/listwire round-trips a BatchTx{scalar + []Item}
through build -> wrap -> ListAt.All() with every element read zero-copy
(TestRoundTrip_List), plus generator-output tests (valid Go via go/parser +
construct assertions). Existing golden tests unchanged.
There is one perfected typed ZAP layer and it lives at github.com/luxfi/zap/v1 —
forwards-only, no backwards-compat baggage, never a breaking major. Mechanical,
behavior-preserving: v2/ -> v1/ (git mv), package zapv2 -> zapv1, import path
.../zap/v2 -> .../zap/v1 (incl. the codegen's emitted banner + import so
generated files reference /v1). Codegen golden tests pass; typed layer vets
clean. (Pre-existing CGO 'resolv' link failures in the root package are
unrelated env noise.)
Field.Type now accepts "string" and "bytes" (no <N>) for variable-length
tail fields alongside the existing scalar + fixed "bytes<N>" kinds. They take
an 8-byte tail pointer in the fixed payload; the constructor emits
SetText/SetBytes and a standalone zero-copy accessor over Object.Text/Bytes.
Purely additive — existing fixed-scalar/byte-array schemas unchanged.
Tests: new TestEmit_VarLength + all existing green (TestEmit_UnsupportedType
retargeted to 'complex128' since 'string' is now valid). Generated RemoteStorage
Location (3 strings + int32) verified downstream: 44 ns/op, 0 allocs/op reads.
This is what Hanzo S3 needs to define its messages as ZAP schemas (no protobuf,
no _pb, no marshal).
FEE-ON-TRANSFER id divergence (LOW/INFO #2c). On-chain DeriveIntentID binds the OBSERVED-DELTA
locked amount (the value the 0x9999 vault actually received), not the requested amount. For
native LUX + standard ERC-20s, locked == amountIn so this off-chain pre-derivation equals the
on-chain id exactly. For a FEE-ON-TRANSFER token, locked < amountIn so the pre-derived id
DIVERGES — but that is only a missed pre-registration optimization, not a safety or correlation
break: the on-chain IntentSubmitted event carries the REAL id as its INDEXED topic (and the real
locked amount), so a keeper correlates via that topic — the authoritative binding — and builds
the D order from the event, never a guessed id. Scoped the DeriveIntentID doc to state this
precisely (pre-registration valid iff locked == amountIn; otherwise correlate via the indexed
topic).
CROSS-REPO id golden (LOW/INFO #3b). Added TestIntentID_OffChainEqualsOnChain: the off-chain
DeriveIntentID over fixed inputs must hash to a shared golden the precompile side ALSO pins
(its TestIntentID_CrossRepoGolden, native_intentid_parity_test). dexsession cannot import
precompile/dex (EVM+geth+cgo + a forbidden compile edge to C Verify), so the two id
implementations are anchored to ONE golden — either drifting fails its half. Closes the missing
cross-repo equivalence test.
CGO_ENABLED=0 pure-Go (GOWORK=off); full dexsession suite green.
THE SETTLE CALLDATA REVERTED (byte-identity). EncodeSettlementHookData emitted a 64-byte
Phase-B body (outputID|amount), but the precompile's decodeSettlementBody requires 96
(outputID|amount|intentID) — the intentID word binds the credit to the taker's intent record
(the per-taker cap + deadline gate). So every settle tx built off this calldata REVERTED
on-chain (ErrSettleBodyMalformed). Add the intentID word so the body is byte-identical with
precompile/dex/settle_hookdata.go; settlementBodyLen is now 96. The DExportRef already carries
IntentID, so the production importSettlement path and every ResolveExport supplies it.
calldata_parity_test pins the full 96-byte body against a hardcoded vector.
THE MULTI-HOP ROUTE REVERTED (RT01 scoping). prepareRouteLocal signed an on-chain hookData of
DI01|RT01|hopCount|path. The precompile has NO route-marker awareness: it classifies the DI01
tag as a Phase-A intent, then decodeIntentBody rejects the RT01 body width (not in {0,32,64})
-> every multi-hop swap REVERTED on-chain. Fix by scoping the route OFF the on-chain calldata:
the on-chain leg is now a PLAIN DI01 intent on the entry market (nonce=CallIndex, matching the
id this session derives — DeriveIntentID binds the entry market + nonce, NOT the path, so the
id is unchanged). The PATH travels to the D router over the ZAP control plane (the RouteRequest
envelope, buildRouteRequest/NotifyRoute), where it belongs — it is D-matcher orchestration that
moves no C-side value (one input object, one final output object regardless of hop count).
Removed EncodeRouteIntentHookData/DecodeRouteIntentHookData (the on-chain RT01 wire the
precompile never supported — a bug generator); the path's one canonical wire is the
RouteRequest envelope. Reworked the route parity test to pin the new property (on-chain route
leg is a precompile-accepted DI01 body carrying no RT01/DS01 tag) and the multi-hop e2e test to
read the path from the session (control plane), not the signed calldata.
CGO_ENABLED=0 pure-Go (GOWORK=off); full dexsession suite green.
The off-chain side derived the intent id with a ZERO txID (pre-landing) while the chain
used the REAL txID, so the ids never matched and the keeper's watch could not correlate
against a live chain (the swap-seam watch-correlation break). FIX, byte-identical to
precompile/dex: DeriveIntentID drops txID+callIndex and binds a NONCE instead
(networkID,cChainID,dChainID,account,assetIn,amountIn,marketID,nonce). The nonce is the
taker's disambiguator carried in the swap's DI01 hookData; EncodeIntentHookData now emits
deadline[32]|nonce[32] (minimal-width, matching the precompile decoder), so off-chain and
on-chain derive the IDENTICAL id from the IDENTICAL calldata.
- SwapIntentRequest.CallIndex -> Nonce (uint64); wire reflowed (sirNonce uint64).
- session.go/server.go/v4session.go derive the id without txID; the swap prepare carries
(deadline, nonce) in hookData. v4 LP/route paths pass their CallIndex as the nonce.
- TestParity_DeriveIntentID / TestParity_VectorPin updated to the new canonical algorithm
(NO txID; nonce trails marketID) — the cross-repo byte-identity contract holds.
Full dexsession suite green (GOWORK=off, CGO_ENABLED=0). Pairs with precompile
fix/swap-rail-intent-record (DeriveIntentID nonce) — keep the two byte-identical.
The retag path hand-patched the ZAP frame flag word at fixed header offset
[6:8] — the sole deviation from the FinishWithFlags idiom used everywhere
else, brittle against any wire-layout change. buildPreparedIntent now takes
the msgType and finalizes under it directly, so NotifyIntent builds an
MsgNotify frame in one step instead of building MsgPrepare then re-tagging
finished bytes. retag() is deleted (no other callers).
TestWire_PreparedIntent_RouteFlags asserts the route flag (Flags()>>8) round-
trips for both MsgPrepare and MsgNotify and the body is untouched by the route
choice. Full dexsession suite green.
Commits the V4 session files that were built + red-verified in the working tree
but never git-added (only v4route.go landed in 9719fcc, so that commit did not
compile). Adds V4Swap/Route/Liquidity/Collect/State sessions + the bidirectional
V4_* message set + per-action scoped caps, wired into server/wire. Orchestration
only — no compile edge to the value path; money-plane invariant intact.
L1: bound hopCount via division (len/32) not hopCount*32 — overflow-safe even on
a 32-bit int build (latent; deploy targets are 64-bit).
L2: zero RouteStatus.Ref unless phase is terminal (Committed/Refunded) in the
direct Poll() reader, mirroring streamRoute's gate — orchestration-surface
hygiene (the chain re-binds on import regardless). Non-money-plane hardening.
Orchestration ONLY — ZAP is NEVER settlement authority. A ZAP response is never
sufficient to move money; money moves only when C consumes a real D->C atomic
export object or D consumes a real C->D object. The layer holds no StateDB/
AtomicState/shared-memory and has no compile edge to precompile/dex or the EVM:
it cannot be imported into the C Verify path, and Verify does not import it.
- DexSession capabilities (Cap'n-Proto): bootstrap -> restricted session;
unforgeable, grant-scoped QuoteCap/IntentCap/WatchCap/SettlementCap/AdminCap.
No cap exposes a value-moving method (proof by exhaustion).
- prepareSwapIntent returns 0x9999 CALLDATA the user signs (no funds reserved,
no enforceable amountOut). notifyIntent triggers a D scan (cannot validate a
match without a D block). importSettlement points at a DExportRef pointer; the
chain binds recipient/asset/amount to the real object.
- Wire format + DeriveIntentID/DeriveUTXOID + V4 swap selector + hookData phase
tags reproduced as pure values (no EVM/cgo dep) and pinned byte-identical to
precompile/dex + chains/dexvm by parity tests (three-homes pattern).
- Promise pipelining (thenAsync + SwapFlow) overlaps network latency end-to-end.
- 26 tests incl. TestRED_ZAP_ResponseAloneCannotMoveMoney. CGO_ENABLED=0, pure Go.
WIP main was an orphaned old base; remote main is the canonical refactor and
already contains the WIP intent, advanced further: PQ handshake (bigger
handshake/), the full transport/ suite WITH C kernels (dpdk_pmd/gpudirect_rdma/
uma_cuda) and QUIC (quic-go in go.mod), plus larger node.go/zap.go. All 15
modified source files + go.mod/go.sum in the WIP were the older base and are
superseded (accept remote). LLM.md/MIGRATION.md: remote canonical, no unique
local notes to fold. The 416 mdns/rust/target/* build artifacts the WIP had
committed are dropped (compiled output, never committed).
Genuinely-new, ported: mcp/ package (MCP<->ZAP bridge for fast tool calling),
absent from remote. Builds clean against the refactored zap API (go build
./... OK).
Full WIP preserved on branch wip/local + tag backup/pre-reconcile-main.
Two defects made the canonical zapclient native-TCP path non-functional
(undetected — no test did a real Client→Server TCP round-trip; zapweb
works via Dispatch, inter-shard uses raw zap.Node):
1. Server.Register bound the node handler under the full opcode, but the
Node routes inbound by Flags()>>8 (the high byte). Mismatch → handler
never found → call hung. Register now uses op>>8; dispatch still reads
the full Flags() to select the procedure.
2. Connect with a static Discovery set the node NoDiscovery, so it never
learned peer addresses and could not dial. Added WithStaticPeers + a
static Discovery, ConnectDirectID (returns the handshake-learned
NodeID), and client-side dial-by-address (memoized) so static peers
(placeholder NodeID + real Address — the cloud/K8s Service-DNS path)
are reachable. mDNS path unchanged.
Adds TestNativeTCPRoundTrip as the regression guard.
Start() closed the listener and failed the whole node when mDNS
discovery couldn't start. But mDNS is zero-config LAN convenience, not a
serving requirement — on networks without multicast (most Kubernetes pod
overlays, restricted hosts) the node must keep accepting on its listener;
peers reach it by address (Service DNS at the fixed port). Now a
discovery-start failure logs a warning and continues. This makes 'mDNS on
by default' safe everywhere: LAN gets zero-config discovery, cloud serves
on the fixed port regardless.
The browser half of one protocol. A TypeScript port of the ZAP wire
codec (zap.go + builder.go) byte-identical to Go — proven by golden
vectors emitted from golden_test.go (12/12 encode+decode) — plus a
zapweb WebSocket client mirroring client.go (opcode stamp, [reqID][flag]
correlation), validated by a live TS-client to Go-echoserver round-trip
(cmd/echoserver). procedureOpcode matches Go FNV-1a. Zero runtime deps:
browser-global WebSocket, Node native type-stripping for tests.
Browsers cannot open raw TCP/QUIC, so the external surface could not be
native ZAP. zapweb bridges a WebSocket to zapclient.Server.Dispatch — the
exact same procedure registry + PeerVerifier the native node uses. One
dispatch path, two transports (native ZAP service↔service, zapweb
browser↔service). No REST. Adds the dependency-free coder/websocket.
Server.Dispatch is the new exported, transport-agnostic entry point.
ServerOptions.Port existed but had no functional-option setter, so every
zapclient.Server bound an ephemeral port reachable only via mDNS — which
is not forwarded across most Kubernetes pod overlays. WithServerPort lets
a server bind a fixed port reachable by Service DNS (the production
discovery model). Pairs with WithNoDiscovery.
package mcp (an MCP-over-ZAP tool-calling bridge from the initial commit) has zero importers anywhere in the tree and its Bridge is never instantiated, so the handlers never register. Dead code in a core wire library; remove it rather than carry/harden an unreachable path.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
HandshakeTimeoutSec was defined but referenced nowhere and no SetReadDeadline existed, so a peer could stall the pre-auth handshake indefinitely (goroutine/FD exhaustion). Initiator.Run and Responder.Run now set a read deadline of HandshakeTimeoutSec when conn is a net.Conn, cleared on return; bare io.ReadWriter callers remain responsible for their own timeout.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ZAP is the wire protocol (FIX, HTTPS, WS, QUIC, X-Wing adaptors) — pure
bytes in / bytes out. Internal serialization is not its concern.
The codec subpackage was added in v0.8.0 by a misguided relocation of
luxfi/codec; revert. Application-level struct serialization belongs in
each VM (hand-rolled like luxfi/net's binio.go pattern), not in ZAP.
Move the entire luxfi/codec module content into luxfi/zap/codec as the
canonical home for the ZAP wire codec. luxfi/codec module is now
redundant and queued for archival; consumers should import
github.com/luxfi/zap/codec directly.
This decomplects the alias chain proto/zap_codec → codec/zapcodec into
one canonical home. The 'codec' name lives because the package IS the
wire codec; 'zap' is its namespace because ZAP is the wire it speaks.
Contents:
- codec.go, errors.go, packing.go (root codec.Manager + helpers)
- codecmock/ (go.uber.org/mock generated Manager mock)
- linearcodec/ (big-endian linear codec, legacy compat)
- reflectcodec/ (struct fielder + reflection type codec)
- wrappers/ (packer/closer wrappers)
- zapcodec/ (little-endian ZAP-native codec, sourced from the
cleaned luxfi/zapcodec@v1.0.1 extraction)
All import paths rewritten github.com/luxfi/codec/* and
github.com/luxfi/zapcodec → github.com/luxfi/zap/codec/*.
Build: GOWORK=off go build ./codec/... — PASS
Tests: GOWORK=off go test ./codec/... — PASS (wrappers, zapcodec)
Three orthogonal additions, one wire format.
v2 package — generic ad-hoc schema dispatch (~/work/lux/zap/v2):
- README + IMPOSSIBILITY + BENCH_RESULTS document the v1↔v2 split.
- builder/field/iter/list/pool/registry/schema/view + codegen package
emit v1-equivalent fast paths from declarative .zap sources.
- _compile_fail_test/ + examples/ cover negative and positive cases.
- v2_test.go exercises the generic dispatch path.
- doc.go in v1 marks v1 deprecated for *new* schema authoring; existing
v1 buffers stay parseable by both v1 and v2.
bufpool — pooled read slabs (bufpool.go + bufpool_test.go):
- Message gains optional *bufRef; Release() returns the slab to its
pool when sourced from the pooled read path, GC-managed otherwise.
- ParseHeader returns (data, rootOffset) without allocating *Message,
used by zapv2 to avoid two allocations per parse.
- bench_read_test.go + node_quic_stream_test.go cover both paths.
Transport allocator + UMA/RDMA cgo (transport/*):
- Transport.Caps()/Buffer interface: capability + slab-typed buffer
contract. Bytes() vs DevicePtr() decoupled so UMA and GPUDirect
can share the same call surface.
- DPDK PMD + GPUDirect RDMA + CUDA UMA real cgo backends gated behind
build tags; stub paths remain default on darwin / non-Linux.
- transport_test.go covers Caps reporting + Buffer lifecycle on the
default transport.
LLM.md + go.mod follow the additions.
LP-023 Red round 3 follow-ups:
NEW-V1 follow-up: tighter per-element stride clamp via Object.ListStride.
The bare Object.List uses the permissive `length <= len(data)` baseline
(wire layer cannot know per-accessor stride). Callers that know the
element width can opt into the tighter `length * minStride <= bufRem`
clamp via ListStride(off, stride), rejecting poisoned length values up
front instead of relying on per-element bounds checks.
NEW-V1 docstring: List.Len() now carries the SAFETY caveat — callers MUST
NOT pre-allocate via make([]T, l.Len()) without an independent bound.
FuzzParse (Red follow-up #3): round-trip property fuzzer pinning the
Parse↔Bytes contract: msg.Bytes() == data[:msg.Size()], Parse is
idempotent on its own output, Version() always in {Version1, Version2}.
1M+ execs clean on M1 Max with 20s fuzztime; seed corpus includes
adversarial buffers that exercised RED-HIGH-1/2/3.
Tests:
- TestNewV1_ListStrideTighterClamp — poisoned length=100 passes bare
baseline but ListStride(0, 4) rejects (400 > bufRem)
- TestNewV1_ListStrideAcceptsHonestLength — honest length=5 stride=4
buffer must pass
All existing tests still pass (RedRound2 HIGH-1/2/3, MEDIUM-1, V18 + base).
Wire format unchanged — purely a tightened acceptance test; backward
compatible at the wire level. Callers update obj.List(off) →
obj.ListStride(off, minStride) per element type.
RED-HIGH-1: Object.List length is now clamped against len(data) at parse.
Attacker-set length=0xFFFFFFFF used to flow through to consumers, where
`for i := 0; i < l.Len()` would iterate 4G times.
RED-HIGH-2: Object.Object, Object.List, and Object.Bytes now reject any
absOffset < HeaderSize. A negative relOffset can no longer alias the wire
header (Magic/Version/Flags/RootOffset/Size). The signed-cast remains so
honest builders can finalize nested items before their parent.
RED-MEDIUM-1: bump wire Version from 1 to 2 (Version1 still accepted at
Parse for backward compat). NewBuilder emits Version2 by default;
NewBuilderV1 is kept for explicit-legacy paths. Message.Version() exposes
the parsed version so v3 platformvm Wrap*Tx accessors can gate on
Version2 to reject v1-vs-v2 schema confusion (e.g. NetworkID=11 byte 0
collision with TxKindBaseFull).
RED-V18: Parse now rejects size < HeaderSize. A buffer with size=0
previously slipped through and panicked on subsequent Root()/Flags().
Regression tests pinned in-tree at zap_test.go:
TestRedRound2_HIGH1_UncappedListLength
TestRedRound2_HIGH2_BackwardListPointer
TestRedRound2_HIGH2_BackwardObjectPointer
TestRedRound2_HIGH2_BackwardBytesPointer
TestRedRound2_MEDIUM1_VersionParse
TestRedRound2_MEDIUM1_NewBuilderEmitsV2
TestRedRound2_V18_SizeZeroRejected
TestRedRound2_F1_NegativeBitPatternSweep
Originals at /tmp/red_zap_attacks/*_test.go.
Object.Bytes was casting the relative offset to int32 before bounds
checking. A bit-pattern with the high bit set (e.g. 0xFFFFFFE0)
sign-extended to -32, slipped past the absPos+length > size guard, and
let a Bytes accessor alias bytes BACK inside the same object's fixed
section. Once a TxID = hash(buffer) lands, two distinct buffers (legit
vs aliased-Memo) hash differently while decoding to identical logical
content — classic transaction malleability.
Fix: drop the int32 cast in Object.Bytes. SetBytes always defers
payload writing to ObjectBuilder.Finish, which writes after the fixed
section — Bytes relOffsets are always forward-pointing. Switching to
uint32→int conversion routes negative bit-patterns through the size
bounds check as ~4 GiB positions, which the guard now rejects.
Object.Object and Object.List keep the signed-int32 decoding: builders
may finalize a nested object/list BEFORE the outer object (TestList,
TestNestedObject exercise exactly this), so legitimate negative
relOffsets exist for those types.
Regression tests:
TestBytesNegativeRelOffsetRejected (0xFFFFFFE0)
TestBytesMaxUintRelOffsetRejected (0xFFFFFFFF)
Coordinates with luxfi/node LP-023 Phase 1 batch 2 Red review (F1).
New package github.com/luxfi/zap/quic provides the canonical QUIC
transport for the ZAP messaging substrate.
Wire format is identical to the TCP transport (length-prefixed
Cap'n Proto frames), so every consumer's wire-level expectations
are preserved.
What QUIC adds beyond TCP+TLS:
- Multiplexed bidi/uni streams (one ZAP RPC per QUIC stream,
no head-of-line blocking between RPCs)
- Connection migration on local-IP changes (Wi-Fi to LTE,
NAT rebind, validator re-homing)
- 0-RTT resumption via TLS 1.3 session tickets
- TLS 1.3 hybrid post-quantum key exchange X25519MLKEM768
(IANA NamedGroup 0x11ec) as the default
- ALPN allowlist 'zap/1'
Cryptography:
- We use the IANA-registered hybrid X25519MLKEM768 verbatim;
no custom combiner, no z-wing fork. Go stdlib's crypto/tls
performs HKDF-Extract(X25519_ss || MLKEM_ss) internally.
- Server cert sig stays classical (ECDSA / RSA) — Go stdlib
TLS does not yet ship ML-DSA. The hybrid KEM defends against
harvest-now/decrypt-later; the cert is the orthogonal
auth-time concern, gated by today's trusted CA.
- 0-RTT replay is documented; non-idempotent handlers must
set RejectEarlyData=true.
Back-compat is sacred: TransportTCP is the default for
NodeConfig.Transport, so every existing consumer keeps working
untouched. TransportQUIC is opt-in and requires:
import _ "github.com/luxfi/zap/quic"
so the QUIC TransportFactory registers itself at init.
Tests prove X25519MLKEM768 actually negotiates end-to-end:
$ go test ./quic/... -run TestHandshake -v
TestHandshake_NegotiatesX25519MLKEM768
client negotiated CurveID = 0x11ec, ALPN = "zap/1"
--- PASS
Also covered: classical-fallback negotiation when server excludes
the hybrid, ALPN allowlist rejection, multiplex of concurrent
streams, 0-RTT round-trip, unidirectional broadcast streams,
frame-size enforcement, QUICConfig propagation through Listen.
Benchmarks on Apple M1 Max (1-RPC ping-pong, single connection):
RTT_QUIC/256B 0.40 MB/s 634 us/op
RTT_QUIC/1MiB 44.94 MB/s 23 ms/op
RTT_TCP_Baseline/256B 3.16 MB/s 81 us/op (no TLS, no PQ)
RTT_TCP_Baseline/1MiB 838.88 MB/s 1.2 ms/op
Multistream concurrent (16 parallel): 5.02 MB/s @ 50us/op
QUIC pays for TLS 1.3 + PQ KEM relative to the plaintext-TCP
baseline; small-message latency is 8x, large-message throughput
is 18x slower. The multistream concurrency case is where QUIC
wins — independent RPCs do not block each other.
Files:
quic/conn.go transport-level Conn (Send/Recv/streams)
quic/server.go Listener wrapping quic-go Listener/EarlyListener
quic/client.go Dialer with 0-RTT-attempting DialEarly
quic/tls.go TLS defaults + self-signed test cert helper
quic/factory.go registers QUIC TransportFactory with parent zap pkg
quic/config.go Config wrapper exposed via NodeConfig.QUICConfig
quic/doc.go package documentation
quic/README.md deployment guide + threat model
quic/handshake_test.go X25519MLKEM768 negotiation + multiplex + 0-RTT
quic/benchmark_test.go QUIC vs TCP baseline RTT + throughput
transport.go new Transport enum + TransportFactory hook
node_quic.go QUIC-path Send/Call/ConnectDirect dispatch
node.go additive: Transport field on NodeConfig, branch
in Start/Send/Call/ConnectDirect
LLM.md QUIC transport section under PQ-TLS Support
go.mod / go.sum add github.com/quic-go/quic-go v0.59.1
Document the canonical storage backends in luxd today and the
migration path per surface. ZAP-native ancient store (in luxfi/geth)
is the first surface shipped; P-Chain/X-Chain/chainData migrations are
tracked separately (#186, #187).
The EVM ancient store migration is one-shot via
`luxfi/geth/cmd/migrate-ancient`. The luxfi/operator NodeFleet CRD
defaults archive pods to --ancient-store-backend=zap.
zeekay