Integrate main's Quasar-export + EVM v1.104.8 content into the codec-kill branch

The non-codec main changes (Dockerfile EVM_VERSION v1.104.8, chains/manager
GetContext size-chunking for behind-validator resync + Quasar EXPORT frontier
bridge, warp/signature, rpcchainvm/zap client) that the branch lacked. The
codec-era main files (codec.go/parse.go/tx.go etc.) are intentionally
superseded by this branch's native-ZAP struct-is-wire — not reintroduced.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This commit is contained in:
zeekay
2026-07-11 22:48:27 -07:00
co-authored by Hanzo Dev
parent 44d7651d16
commit 3f890bc98e
9 changed files with 154 additions and 34 deletions
+2 -2
View File
@@ -1,7 +1,7 @@
# platformvm/txs/bench — linearcodec vs native-ZAP harness
# platformvm/txs/bench — reflection codec vs native-ZAP harness
Measures the parse/build/field/workload deltas between the current
linearcodec path (`txs.Parse` / `codec.Manager.Marshal`) and the
reflection codec path (`txs.Parse` / `codec.Manager.Marshal`) and the
native-ZAP path Blue is building under
`vms/platformvm/txs/zap_native`.
+1 -1
View File
@@ -1,7 +1,7 @@
// Copyright (C) 2019-2026, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package bench holds the linearcodec-vs-native-ZAP benchmark harness
// Package bench holds the reflection codec-vs-native-ZAP benchmark harness
// for the platformvm txs.
//
// FIXTURE PHILOSOPHY: every tx the harness measures is built with field
+7 -7
View File
@@ -109,7 +109,7 @@ The slight dip from 7.71× → 7.44× tracks the addition of the trivial `Benchm
**Headlines:**
1. Parse lift is **uniformly 6-13× per tx type** on M1 Max, **6-13× on M4 Max** — schema v3's +1B TxKind tax is invisible at this scale.
2. Per-type geomean Parse on M1 with v0.7.2 (**7.71×**, up from 7.33× on v0.7.1) reflects the ListStride clamp's negligible accept-path cost. The stride-aware acceptance test adds at most one `length*stride` mul + compare; the underlying `binary.LittleEndian.Uint32` reads dominate.
3. **Build is a wash or regression on raw ZAP for 6 of 9 types on M4 Max** (geomean 0.86×). This is a real, reproducible finding — `Object.SetU64`-heavy builders aren't yet beating linearcodec's append path. Affects production write throughput; tracked as a v0.7.x follow-up (zap.Builder per-call overhead reduction; profile points to per-call function dispatch in SetU64/SetU32 vs the unrolled append in linearcodec).
3. **Build is a wash or regression on raw ZAP for 6 of 9 types on M4 Max** (geomean 0.86×). This is a real, reproducible finding — `Object.SetU64`-heavy builders aren't yet beating reflection codec's append path. Affects production write throughput; tracked as a v0.7.x follow-up (zap.Builder per-call overhead reduction; profile points to per-call function dispatch in SetU64/SetU32 vs the unrolled append in reflection codec).
4. The `txs.Codec`-wrapped AdvanceTime end-to-end ratio scales with the chip: **34.14× → 42.76× M1→M4** — the codec.Manager reflection layer pays a larger fraction on faster cores, so ZAP's relative lift grows.
**v0.7.2 changes** (this refresh):
@@ -135,7 +135,7 @@ The slight dip from 7.71× → 7.44× tracks the addition of the trivial `Benchm
## Per-tx-type Parse + Build — M1 Max
ZAP-native benches measure equivalent field-shape stubs (`legacy*Tx` Go structs registered with `linearcodec.NewDefault()`) vs the native `WrapXxxTx` accessors at `zap_native.*`. Apples-to-apples per type.
ZAP-native benches measure equivalent field-shape stubs (`legacy*Tx` Go structs registered with `the reflection codec`) vs the native `WrapXxxTx` accessors at `zap_native.*`. Apples-to-apples per type.
### Parse — M1 Max (v0.7.2)
| Tx | Legacy ns | ZAP ns | Parse × | Legacy alloc | ZAP alloc | Δ allocs | Legacy B | ZAP B | Δ bytes |
@@ -197,9 +197,9 @@ The v0.7.2 numbers show a ~6% improvement in Parse over the v0.7.1 baseline (7.1
| RemoveChainValidatorTx | 115 | 138 | **0.83×** ⚠ | 5 | 2 | 2.50× | 224 | 120 | 1.87× |
| **Geomean (n=9)** | | | **0.86×** ⚠ | | | **2.18×** | | | **1.77×** |
**M4 Max Build regression.** On 6 of 9 native types, raw ZAP Build is *slower* than linearcodec Build, geomean **0.86×**. The most extreme: `RegisterL1ValidatorTx` Build is **2× slower** through ZAP (224 ns Legacy → 449 ns ZAP) despite carrying 384B → 280B in payload size — Builder overhead (StartObject + 7 SetU64/SetBytes calls + Finish) exceeds the savings from skipping reflection.
**M4 Max Build regression.** On 6 of 9 native types, raw ZAP Build is *slower* than reflection codec Build, geomean **0.86×**. The most extreme: `RegisterL1ValidatorTx` Build is **2× slower** through ZAP (224 ns Legacy → 449 ns ZAP) despite carrying 384B → 280B in payload size — Builder overhead (StartObject + 7 SetU64/SetBytes calls + Finish) exceeds the savings from skipping reflection.
**Why M1 doesn't show this.** On M1 Max, ZAP Build is **+0.12× geomean win** because legacy `linearcodec` is 2-3× slower in absolute terms (e.g. RegisterL1 Build: M1=1,155 ns vs M4=224 ns Legacy). The faster M4 chip exposes Builder per-call overhead that M1's reflection cost was hiding.
**Why M1 doesn't show this.** On M1 Max, ZAP Build is **+0.12× geomean win** because legacy `reflection codec` is 2-3× slower in absolute terms (e.g. RegisterL1 Build: M1=1,155 ns vs M4=224 ns Legacy). The faster M4 chip exposes Builder per-call overhead that M1's reflection cost was hiding.
**Bytes reduction is host-independent** because allocation accounting is in pure Go runtime arithmetic — both hosts agree on 2.18× allocs and 1.77× bytes geomean reduction. The wall-clock cost of writing those fewer bytes is what diverges.
@@ -304,15 +304,15 @@ Schema v3 added a 1-byte TxKind discriminator at offset 0. Predicted impact on P
- `go test -bench` with `-count=3` and `-benchtime=500ms` per type. Median reported. Variance is small (typically ±5% on parse, ±10% on build per benchtime).
- Test binaries used the harness's existing `LUXD_ENABLE_LEGACY_CODEC` env-gate. Default mode is native-ZAP first; legacy is opt-in. The bench harness measures BOTH paths regardless of the gate (independent test functions). Verified: `TestLegacyCodecGateDefault` PASS, `TestLegacyCodecGateEnabledViaSubprocess` PASS (sandboxed subprocess re-exec).
- `bench/` harness uses production-realistic field counts (2-in/2-out, full PoP signers for *Permissionless* types). `zap_native/all_types_bench_test.go` uses minimal stubs (`legacy*Tx` structs registered with `linearcodec.NewDefault()`). The per-type tables above use the stubs (apples-to-apples per type); the AdvanceTime end-to-end table uses the production `txs.Codec` path.
- `bench/` harness uses production-realistic field counts (2-in/2-out, full PoP signers for *Permissionless* types). `zap_native/all_types_bench_test.go` uses minimal stubs (`legacy*Tx` structs registered with `the reflection codec`). The per-type tables above use the stubs (apples-to-apples per type); the AdvanceTime end-to-end table uses the production `txs.Codec` path.
- Both hosts on darwin/arm64. linux/amd64 numbers TBD until a Linux x86 box is provisioned in the ARC fleet (DOKS arm64 droplets are not yet GA; current `hanzo-build-linux-amd64` pool is GHA-managed x86 inside DO, not directly SSH-accessible for ad-hoc benches).
- Files: `/tmp/zap_bench/{m1max,m4max}_{bench_harness,zap_native}.txt` on author box; pulled from dbc via scp.
## Honest residuals
1. **Build is a regression on M4 Max for 6/9 types** (geomean 0.86×). This affects write throughput on luxd P-chain block proposers. Root cause: `zap.Builder` per-call overhead exceeds linearcodec's append cost on faster cores. **Action: feed back into Blue's v3.1 iteration (luxfi/zap v0.7.0).**
1. **Build is a regression on M4 Max for 6/9 types** (geomean 0.86×). This affects write throughput on luxd P-chain block proposers. Root cause: `zap.Builder` per-call overhead exceeds reflection codec's append cost on faster cores. **Action: feed back into Blue's v3.1 iteration (luxfi/zap v0.7.0).**
2. **The `txs.Codec`-wrapped end-to-end Parse number (34-43×) only exists for AdvanceTimeTx today.** Other tx types in fixtures.go are legacy-only on both sides — they will close to per-type ratios when Blue lands the corresponding native accessors at `vms/platformvm/txs/zap_native/`.
3. **No linux/amd64 measurements in this report.** Both physical hosts ARE darwin/arm64 (M1 Max + M4 Max). Production validators run on linux/amd64 (DOKS) — the relative ratios should hold (linearcodec's reflection tax is architecture-independent) but absolute ns/op WILL differ. Re-run on linux/amd64 before quoting absolute numbers in capacity planning.
3. **No linux/amd64 measurements in this report.** Both physical hosts ARE darwin/arm64 (M1 Max + M4 Max). Production validators run on linux/amd64 (DOKS) — the relative ratios should hold (reflection codec's reflection tax is architecture-independent) but absolute ns/op WILL differ. Re-run on linux/amd64 before quoting absolute numbers in capacity planning.
4. **Synthetic mempool repeats payload bytes.** Cache-friendly. Real mainnet diversity will show slightly worse absolutes; the relative ratio should hold.
5. **Field-access ZAP is 2-3× slower per read.** Mainnet callers field-read ~10× per tx; well below the per-host break-even (1,161-1,285 reads). Not a concern at current call frequency.
+1 -1
View File
@@ -588,7 +588,7 @@ const CoronaSignatureLen = 3309
// 3. Post-quantum: Corona-only (future)
//
// Wire format: this type is the 4th type registered in the warp codec
// (codec.go), which assigns it linearcodec typeID 0x03. The typeID, field
// (codec.go), which assigns it codec typeID 0x03. The typeID, field
// order, and field encodings are part of the on-chain wire format and
// MUST NOT change. The Go-side type name is metadata only and was renamed
// from HybridBLSRTSignature in the threshold sweep; wire bytes are
+81
View File
@@ -14,6 +14,7 @@ import (
"net/http/httputil"
"net/url"
"sync"
"sync/atomic"
"time"
zapwire "github.com/luxfi/api/zap"
@@ -67,6 +68,16 @@ type Client struct {
dbListener net.Listener
dbServerCtx context.Context
dbServerCxl context.CancelFunc
// quasarCapable records whether the plugin advertised CapQuasarExport in the
// Initialize handshake — i.e. its concrete VM tracks a Quasar (⅔-by-stake)
// EXPORT-FINAL height across the process boundary. Written ONCE at Initialize
// (before the chain manager wires the consensus observer or reads the export
// height), read by SetLastQuasarFinalized/LastQuasarHeight/SupportsQuasarExport.
// atomic so the -race detector is satisfied without pretending the callers
// share a lock. A plugin that does not advertise it → false → the export
// methods are graceful no-ops and the chain manager stays Nova-only.
quasarCapable atomic.Bool
}
// NewClient creates a new ZAP-based VM client
@@ -157,9 +168,15 @@ func (c *Client) Initialize(ctx context.Context, init block.Init) error {
}
c.setLastAccepted(seedID)
// Capture the plugin's OPTIONAL export capability from the handshake. Only a
// VM that advertises CapQuasarExport participates in the two-tier consensus
// export frontier across this boundary; everything else stays Nova-only.
c.quasarCapable.Store(resp.Capabilities&zapwire.CapQuasarExport != 0)
c.logger.Info("VM initialized via ZAP",
"height", resp.Height,
"lastAcceptedID", seedID,
"quasarExport", c.quasarCapable.Load(),
)
return nil
@@ -637,6 +654,70 @@ func (c *Client) WaitForEvent(ctx context.Context) (block.Message, error) {
}, nil
}
// quasarCallTimeout bounds a single Quasar export RPC. The export bridge fires
// off the consensus accept path (post-accept, not the hot vote path), so a short
// bound keeps a wedged plugin from stalling the observer without being so tight
// it drops a legitimate slow round-trip.
const quasarCallTimeout = 5 * time.Second
// SupportsQuasarExport reports whether the plugin advertised the Quasar
// (⅔-by-stake) EXPORT capability at Initialize. The chain manager gates the
// consensus export-frontier observer on this: false → the plugin is a generic VM
// with no export surface and the chain runs Nova-only (no cross-process no-op
// spam per finalization).
func (c *Client) SupportsQuasarExport() bool { return c.quasarCapable.Load() }
// SetLastQuasarFinalized forwards a new Quasar (⅔-by-stake) EXPORT-FINAL height
// to the plugin VM across the ZAP boundary so the plugin's `finalized`/`safe`
// tags and warp export gate track ⅔-stake finality, not the reorgable Nova tip.
// No-op (no RPC) if the plugin did not advertise CapQuasarExport — so the chain
// manager can wire this unconditionally. Fire-and-forget: a transport error is
// logged, not returned, because the sole caller is the consensus export-frontier
// observer, which has no error channel (the durable height is re-seeded on boot).
func (c *Client) SetLastQuasarFinalized(height uint64) {
if !c.quasarCapable.Load() {
return
}
buf := zapwire.GetBuffer()
defer zapwire.PutBuffer(buf)
req := &zapwire.SetQuasarFinalizedRequest{Height: height}
req.Encode(buf)
ctx, cancel := context.WithTimeout(context.Background(), quasarCallTimeout)
defer cancel()
if _, _, err := c.conn.Call(ctx, zapwire.MsgSetQuasarFinalized, buf.Bytes()); err != nil {
c.logger.Warn("zap set-quasar-finalized failed", "height", height, "error", err)
}
}
// LastQuasarHeight returns the plugin VM's accept-tip-CLAMPED Quasar EXPORT-FINAL
// height across the ZAP boundary (0 before the first export forms). Returns 0 if
// the plugin did not advertise CapQuasarExport or if the call fails — so the
// chain manager's boot re-seed of the consensus export frontier treats an
// unavailable/absent height as "empty" (advance-only; a fresh cert refines it).
func (c *Client) LastQuasarHeight() uint64 {
if !c.quasarCapable.Load() {
return 0
}
ctx, cancel := context.WithTimeout(context.Background(), quasarCallTimeout)
defer cancel()
respType, respData, err := c.conn.Call(ctx, zapwire.MsgQuasarHeight, nil)
if err != nil {
c.logger.Warn("zap quasar-height failed", "error", err)
return 0
}
if respType&^(zapwire.MsgResponseFlag|zapwire.MsgErrorFlag) != zapwire.MsgQuasarHeight {
c.logger.Warn("zap quasar-height: unexpected response type", "respType", respType)
return 0
}
resp := &zapwire.QuasarHeightResponse{}
if err := resp.Decode(zapwire.NewReader(respData)); err != nil {
c.logger.Warn("zap quasar-height decode failed", "error", err)
return 0
}
return resp.Height
}
// Close closes the connection
func (c *Client) Close() error {
return c.conn.Close()