StartList returns ListBuilder by value (mirrors the value-typed ObjectBuilder):
Add*/Finish keep pointer receivers, every caller binds to an addressable local so
Go auto-addresses it. Dropped the dead elemSize field (never read — stride is
implicit in which Add* is called). Byte-identical wire (golden+fuzz+all round-trips
green). NOTE: not a measured alloc win — escape analysis already stack-allocated
the &ListBuilder{} at all 9 call sites (composite stays 583ns/5allocs, the 5 being
mandatory writeEnvelopePrefix pool-egress copies). Value is code-cleanliness +
consistency.
Also fixes zap/v1 subpackage: v1.2.4's ObjectBuilder value change left v1/*.go
referencing *zap.ObjectBuilder → the subpackage didn't build (latent; only the
root package is imported downstream so it went unnoticed). Converted to the value
type. Full module (root + v1) builds + tests green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Decomplect that unlocks a perf win. SetBytes deferred its data into an offsets
[]offsetEntry slice replayed in Finish — machinery that recorded intentions to
replay later, and heap-allocated a slice per object-with-a-bytes-field. Replace
it with directness: StartObject RESERVES the fixed section eagerly, so SetBytes
appends its tail + patches its own pointer on the spot; Finish just returns the
offset. Removing the offsets field makes ObjectBuilder's methods mutate only
ob.b (through the pointer) and never ob itself — so ObjectBuilder becomes a
VALUE type and StartObject stops heap-allocating &ObjectBuilder per call (it was
65% of composite-build allocs).
Byte-identical wire (golden + fuzz + pool byte-equality + nested-proof suites all
green). Real X-tx wire composite: 922ns/11allocs -> 655ns/5allocs (1.4x, <half
the allocs); vs the original byte-blob 2551ns/37allocs that's 3.9x / 7.4x fewer
allocs. Speeds EVERY zap object build node-wide, not just X.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
AddObjectPtr (build a repeated-message field as pointer slots) had no matching
typed reader — callers had to hand-deref the 4-byte signed rel. List.ObjectPtr(i)
is that reader: absOffset = slotPos + int32(rel), same rule as Object.Object,
with the HeaderSize/bounds guards. This is the primitive that lets a tx be built
as ONE native-nested ZAP object instead of byte-blob envelope concat.
nested_tx_proof_test.go: reference design for decomplecting utxo/wire — an
X-chain 2in/2out money-move built as a single nested object (SetObject +
AddObjectPtr out/in lists + inline uint16 fx-discriminator preserving fx
polymorphism). Measured vs the byte-blob envelope pattern it replaces:
build 1345ns/19allocs -> 291ns/0allocs (4.6x, ZERO alloc via pooled builder)
parse 4114ns/26allocs -> 104ns/2allocs (~40x; 2 allocs are result slices)
Round-trip + pooled byte-identical proven.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
SetBytes was heap-copying every payload (append([]byte(nil), v...)) at every
nesting level — the dominant cost of deep wire composition (X-chain tx build
runs ~11 nested builders). It now retains the caller's slice; the caller keeps
it valid until Finish (every in-tree constructor builds+finishes in one frame).
Byte output is unchanged: Finish writes payloads in the same SetBytes order.
GetBuilder/PutBuilder: exported sync.Pool for Builders (write-side counterpart
to the read bufpool). Reuse is unconditionally byte-safe — ensureField zero-
fills every extended span and align zero-fills padding; proven by
TestBuilderPool_ByteIdentical (dirty pooled builder emits bytes identical to a
fresh one). go.sum: re-record luxfi/pq v1.0.3 (upstream re-tag drifted content).
Micro: sample object build 354ns/9allocs -> 257ns/6allocs; the real win
compounds in utxo/wire + xvm envelope composition (next commits).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Executable core of LP-202 §5.2 (three-class stream demultiplexing) and
§6.1 (post-quantum channel). One QUIC connection carries the unified
transport's three stream classes — 'R' RPC, 'C' consensus-P2P, 'Z'
ZAP-native — each routed to its own dispatcher by the one-byte class
prefix (the only new byte on the wire), and asserts the connection
negotiated X25519MLKEM768 (IANA 0x11EC) on both sides.
Additive test only; exercises luxfi/zap alone, touches nothing in luxd
consensus or proposer VM. Runs green (CGO_ENABLED=0 in this env; the
default-CGO failure is an unrelated accel/SDK-header issue in the dep
graph, not this test).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Sets Schema.SkipRegistry so a per-service schema set (e.g. Hanzo S3's
Get/PutObject kinds) does NOT register into the shared zapv1.DefaultRegistry —
their kind bytes are unique only within the service, not globally. Needed for
the ZAP service-mesh pattern where every service owns a private kind space.
The repeated-message wire shape: a list whose elements carry their own
variable-length tails (string/bytes/nested), which the fixed-stride inline
List cannot hold. Completes proto field-shape coverage in the runtime —
map<k,v> reduces to a repeated {key,value} entry, so this is its keystone too.
Composition, not a new wire concept: ListNested = List ∘ NestedAt. The list
slots are a contiguous stride-4 object-pointer array; each pointer is
followed exactly as a singular nested pointer to an out-of-line tailed View.
Runtime:
- builder.go: ListBuilder.AddObjectPtr(targetPos) — appends a signed 4-byte
relative object pointer (the repeated-message element kind).
- v1/list_nested.go: ListNested[N] (Len/At/All/IsZero), ListNestedAt[S,N]
(read), WriteListNested[S,N] + NestedElemSetter[N] (write). Same audited
bounds discipline as Object.Object / ListStride (RED-HIGH-1/2 inherited).
Codegen:
- elemIsTailed: a list element with a string/bytes/nested/list sub-field
auto-routes to ListNested/WriteListNested/ListNestedAt; scalar-only
elements stay in the tighter inline List/WriteList/ListAt. One Elem
declaration, the generator picks the correct encoding.
Proven end-to-end: runtime roundtrip (3 elements each with its OWN string
tail, via At and All, + empty), emit routing test (tailed->ListNested,
scalar-only->List), and a generated-code wire roundtrip (testpkg/repeatwire).
Root zap + full v1+codegen suite green; scalar listwire unchanged (no regression).
Completes the variable-length codegen (strings/bytes/lists/nested) — the
last shape needed before protoc-gen-zap can map every proto message field
mechanically.
Runtime (v1/nested.go):
- NestedAt[S,N]: read an out-of-line 4-byte object pointer as a typed
View[N]; null/OOB -> zero View (proto3 unset).
- WriteNested[S,N]: build a flat nested object into the parent tail and
patch the pointer; the singular peer of WriteList. Composes recursively
(nested may carry scalars, strings, bytes, lists, further nesting).
Codegen (schema.go, emit.go):
- Field.Nested *NestedMsg (4-byte object pointer, distinct from the 8-byte
list/tail pointer); IsNested().
- Constructor takes *Value (nil => null), nil-guarded WriteNested build.
- emitNestedSupport: value struct + NestedAt read accessor.
- emitElemWrites: shared scalar/string/bytes element-write body, so list
elements now also support string/bytes sub-fields (repeated-of-message).
- ListElem/NestedMsg carry Wire (element WireName) for Offset<Wire>_<F>.
Proven end-to-end: runtime roundtrip (nested w/ own string tail + null
case), generator-output tests (go/parser-valid), and a generated-code wire
roundtrip (testpkg/nestwire). Full v1+codegen suite green, no regression.
Completes the variable-length codegen (strings/bytes already shipped; lists now
join them). A list field of a fixed-size element schema generates: a per-element
value-input struct, a typed constructor param []Elem, an inline WriteList build
(bridged onto the hand-rolled ObjectBuilder by the new zapv1.SetterFrom, one
zapv1.Write per element field), and a zero-copy ListAt read accessor returning
zapv1.List[E] (range-over-func ready) — mechanically reproducing examples/
batch_tx.go.
Adds: Schema.Element (flat list-element slots carry no kind byte — fields may
start at offset 0; Wrap skips the kind check); runtime helpers SetterFrom /
WriteString / WriteBytes bridging the hand-rolled fast path to the typed writers.
Proven end-to-end: testpkg/listwire round-trips a BatchTx{scalar + []Item}
through build -> wrap -> ListAt.All() with every element read zero-copy
(TestRoundTrip_List), plus generator-output tests (valid Go via go/parser +
construct assertions). Existing golden tests unchanged.
There is one perfected typed ZAP layer and it lives at github.com/luxfi/zap/v1 —
forwards-only, no backwards-compat baggage, never a breaking major. Mechanical,
behavior-preserving: v2/ -> v1/ (git mv), package zapv2 -> zapv1, import path
.../zap/v2 -> .../zap/v1 (incl. the codegen's emitted banner + import so
generated files reference /v1). Codegen golden tests pass; typed layer vets
clean. (Pre-existing CGO 'resolv' link failures in the root package are
unrelated env noise.)
Field.Type now accepts "string" and "bytes" (no <N>) for variable-length
tail fields alongside the existing scalar + fixed "bytes<N>" kinds. They take
an 8-byte tail pointer in the fixed payload; the constructor emits
SetText/SetBytes and a standalone zero-copy accessor over Object.Text/Bytes.
Purely additive — existing fixed-scalar/byte-array schemas unchanged.
Tests: new TestEmit_VarLength + all existing green (TestEmit_UnsupportedType
retargeted to 'complex128' since 'string' is now valid). Generated RemoteStorage
Location (3 strings + int32) verified downstream: 44 ns/op, 0 allocs/op reads.
This is what Hanzo S3 needs to define its messages as ZAP schemas (no protobuf,
no _pb, no marshal).
FEE-ON-TRANSFER id divergence (LOW/INFO #2c). On-chain DeriveIntentID binds the OBSERVED-DELTA
locked amount (the value the 0x9999 vault actually received), not the requested amount. For
native LUX + standard ERC-20s, locked == amountIn so this off-chain pre-derivation equals the
on-chain id exactly. For a FEE-ON-TRANSFER token, locked < amountIn so the pre-derived id
DIVERGES — but that is only a missed pre-registration optimization, not a safety or correlation
break: the on-chain IntentSubmitted event carries the REAL id as its INDEXED topic (and the real
locked amount), so a keeper correlates via that topic — the authoritative binding — and builds
the D order from the event, never a guessed id. Scoped the DeriveIntentID doc to state this
precisely (pre-registration valid iff locked == amountIn; otherwise correlate via the indexed
topic).
CROSS-REPO id golden (LOW/INFO #3b). Added TestIntentID_OffChainEqualsOnChain: the off-chain
DeriveIntentID over fixed inputs must hash to a shared golden the precompile side ALSO pins
(its TestIntentID_CrossRepoGolden, native_intentid_parity_test). dexsession cannot import
precompile/dex (EVM+geth+cgo + a forbidden compile edge to C Verify), so the two id
implementations are anchored to ONE golden — either drifting fails its half. Closes the missing
cross-repo equivalence test.
CGO_ENABLED=0 pure-Go (GOWORK=off); full dexsession suite green.
THE SETTLE CALLDATA REVERTED (byte-identity). EncodeSettlementHookData emitted a 64-byte
Phase-B body (outputID|amount), but the precompile's decodeSettlementBody requires 96
(outputID|amount|intentID) — the intentID word binds the credit to the taker's intent record
(the per-taker cap + deadline gate). So every settle tx built off this calldata REVERTED
on-chain (ErrSettleBodyMalformed). Add the intentID word so the body is byte-identical with
precompile/dex/settle_hookdata.go; settlementBodyLen is now 96. The DExportRef already carries
IntentID, so the production importSettlement path and every ResolveExport supplies it.
calldata_parity_test pins the full 96-byte body against a hardcoded vector.
THE MULTI-HOP ROUTE REVERTED (RT01 scoping). prepareRouteLocal signed an on-chain hookData of
DI01|RT01|hopCount|path. The precompile has NO route-marker awareness: it classifies the DI01
tag as a Phase-A intent, then decodeIntentBody rejects the RT01 body width (not in {0,32,64})
-> every multi-hop swap REVERTED on-chain. Fix by scoping the route OFF the on-chain calldata:
the on-chain leg is now a PLAIN DI01 intent on the entry market (nonce=CallIndex, matching the
id this session derives — DeriveIntentID binds the entry market + nonce, NOT the path, so the
id is unchanged). The PATH travels to the D router over the ZAP control plane (the RouteRequest
envelope, buildRouteRequest/NotifyRoute), where it belongs — it is D-matcher orchestration that
moves no C-side value (one input object, one final output object regardless of hop count).
Removed EncodeRouteIntentHookData/DecodeRouteIntentHookData (the on-chain RT01 wire the
precompile never supported — a bug generator); the path's one canonical wire is the
RouteRequest envelope. Reworked the route parity test to pin the new property (on-chain route
leg is a precompile-accepted DI01 body carrying no RT01/DS01 tag) and the multi-hop e2e test to
read the path from the session (control plane), not the signed calldata.
CGO_ENABLED=0 pure-Go (GOWORK=off); full dexsession suite green.
The off-chain side derived the intent id with a ZERO txID (pre-landing) while the chain
used the REAL txID, so the ids never matched and the keeper's watch could not correlate
against a live chain (the swap-seam watch-correlation break). FIX, byte-identical to
precompile/dex: DeriveIntentID drops txID+callIndex and binds a NONCE instead
(networkID,cChainID,dChainID,account,assetIn,amountIn,marketID,nonce). The nonce is the
taker's disambiguator carried in the swap's DI01 hookData; EncodeIntentHookData now emits
deadline[32]|nonce[32] (minimal-width, matching the precompile decoder), so off-chain and
on-chain derive the IDENTICAL id from the IDENTICAL calldata.
- SwapIntentRequest.CallIndex -> Nonce (uint64); wire reflowed (sirNonce uint64).
- session.go/server.go/v4session.go derive the id without txID; the swap prepare carries
(deadline, nonce) in hookData. v4 LP/route paths pass their CallIndex as the nonce.
- TestParity_DeriveIntentID / TestParity_VectorPin updated to the new canonical algorithm
(NO txID; nonce trails marketID) — the cross-repo byte-identity contract holds.
Full dexsession suite green (GOWORK=off, CGO_ENABLED=0). Pairs with precompile
fix/swap-rail-intent-record (DeriveIntentID nonce) — keep the two byte-identical.
The retag path hand-patched the ZAP frame flag word at fixed header offset
[6:8] — the sole deviation from the FinishWithFlags idiom used everywhere
else, brittle against any wire-layout change. buildPreparedIntent now takes
the msgType and finalizes under it directly, so NotifyIntent builds an
MsgNotify frame in one step instead of building MsgPrepare then re-tagging
finished bytes. retag() is deleted (no other callers).
TestWire_PreparedIntent_RouteFlags asserts the route flag (Flags()>>8) round-
trips for both MsgPrepare and MsgNotify and the body is untouched by the route
choice. Full dexsession suite green.
Commits the V4 session files that were built + red-verified in the working tree
but never git-added (only v4route.go landed in 9719fcc, so that commit did not
compile). Adds V4Swap/Route/Liquidity/Collect/State sessions + the bidirectional
V4_* message set + per-action scoped caps, wired into server/wire. Orchestration
only — no compile edge to the value path; money-plane invariant intact.
L1: bound hopCount via division (len/32) not hopCount*32 — overflow-safe even on
a 32-bit int build (latent; deploy targets are 64-bit).
L2: zero RouteStatus.Ref unless phase is terminal (Committed/Refunded) in the
direct Poll() reader, mirroring streamRoute's gate — orchestration-surface
hygiene (the chain re-binds on import regardless). Non-money-plane hardening.
Orchestration ONLY — ZAP is NEVER settlement authority. A ZAP response is never
sufficient to move money; money moves only when C consumes a real D->C atomic
export object or D consumes a real C->D object. The layer holds no StateDB/
AtomicState/shared-memory and has no compile edge to precompile/dex or the EVM:
it cannot be imported into the C Verify path, and Verify does not import it.
- DexSession capabilities (Cap'n-Proto): bootstrap -> restricted session;
unforgeable, grant-scoped QuoteCap/IntentCap/WatchCap/SettlementCap/AdminCap.
No cap exposes a value-moving method (proof by exhaustion).
- prepareSwapIntent returns 0x9999 CALLDATA the user signs (no funds reserved,
no enforceable amountOut). notifyIntent triggers a D scan (cannot validate a
match without a D block). importSettlement points at a DExportRef pointer; the
chain binds recipient/asset/amount to the real object.
- Wire format + DeriveIntentID/DeriveUTXOID + V4 swap selector + hookData phase
tags reproduced as pure values (no EVM/cgo dep) and pinned byte-identical to
precompile/dex + chains/dexvm by parity tests (three-homes pattern).
- Promise pipelining (thenAsync + SwapFlow) overlaps network latency end-to-end.
- 26 tests incl. TestRED_ZAP_ResponseAloneCannotMoveMoney. CGO_ENABLED=0, pure Go.
Two defects made the canonical zapclient native-TCP path non-functional
(undetected — no test did a real Client→Server TCP round-trip; zapweb
works via Dispatch, inter-shard uses raw zap.Node):
1. Server.Register bound the node handler under the full opcode, but the
Node routes inbound by Flags()>>8 (the high byte). Mismatch → handler
never found → call hung. Register now uses op>>8; dispatch still reads
the full Flags() to select the procedure.
2. Connect with a static Discovery set the node NoDiscovery, so it never
learned peer addresses and could not dial. Added WithStaticPeers + a
static Discovery, ConnectDirectID (returns the handshake-learned
NodeID), and client-side dial-by-address (memoized) so static peers
(placeholder NodeID + real Address — the cloud/K8s Service-DNS path)
are reachable. mDNS path unchanged.
Adds TestNativeTCPRoundTrip as the regression guard.
Start() closed the listener and failed the whole node when mDNS
discovery couldn't start. But mDNS is zero-config LAN convenience, not a
serving requirement — on networks without multicast (most Kubernetes pod
overlays, restricted hosts) the node must keep accepting on its listener;
peers reach it by address (Service DNS at the fixed port). Now a
discovery-start failure logs a warning and continues. This makes 'mDNS on
by default' safe everywhere: LAN gets zero-config discovery, cloud serves
on the fixed port regardless.
The browser half of one protocol. A TypeScript port of the ZAP wire
codec (zap.go + builder.go) byte-identical to Go — proven by golden
vectors emitted from golden_test.go (12/12 encode+decode) — plus a
zapweb WebSocket client mirroring client.go (opcode stamp, [reqID][flag]
correlation), validated by a live TS-client to Go-echoserver round-trip
(cmd/echoserver). procedureOpcode matches Go FNV-1a. Zero runtime deps:
browser-global WebSocket, Node native type-stripping for tests.
Browsers cannot open raw TCP/QUIC, so the external surface could not be
native ZAP. zapweb bridges a WebSocket to zapclient.Server.Dispatch — the
exact same procedure registry + PeerVerifier the native node uses. One
dispatch path, two transports (native ZAP service↔service, zapweb
browser↔service). No REST. Adds the dependency-free coder/websocket.
Server.Dispatch is the new exported, transport-agnostic entry point.
ServerOptions.Port existed but had no functional-option setter, so every
zapclient.Server bound an ephemeral port reachable only via mDNS — which
is not forwarded across most Kubernetes pod overlays. WithServerPort lets
a server bind a fixed port reachable by Service DNS (the production
discovery model). Pairs with WithNoDiscovery.
ObjectBuilder.SetText/SetBytes stored deferred entries but Finish()
never wrote the actual text/bytes data or patched the relative offsets.
Now Finish() writes deferred data after the fixed section and patches
each field's relative offset correctly.
Adds 3 round-trip tests: single text, multiple text, nested objects
with text fields.
zeekay