main pinned consensus v1.36.9, which contains the silent-rebuild-storm bug that
took devnet down for four and a half days. Any network that upgraded from HEAD
would have wedged the same way.
The bug: buildBlocksLocked keeps pendingBuildBlocks after a failed build, and
proposervm truncates the block timestamp to a whole second — so every rebuild
inside that second re-mints an IDENTICAL blkID. consensus.AddBlock rejects it as
"already exists" and a bare `continue` skips Propose/RequestVotes entirely. The
block is never sent, never voted, never decided, while pending txs keep
re-notifying. Measured on devnet: one node re-built the same blkID 2,287 times
at height 512, with ZERO "proposed block to validators" and ZERO "finalized
block via quorum cert" lines across 200k log lines fleet-wide.
It also leaves a durable trap. The on-disk vote-once guard binds those storm
blkIDs to heights that can never be re-proposed, so reserveSlotForSign refuses
every later block at those heights — the chain stays wedged even after the
binary is fixed. Devnet needed a hand-repaired guard on all five nodes to
recover; that repair has no code path yet (the voteguard docstring points at
engine/chain/lock_migration.go, which does not exist).
This is a REGRESSION, not a stale pin: v1.36.24 and v1.36.25 both pinned
v1.36.10 and finalize normally — devnet runs v1.36.25 today and is producing
blocks. v1.36.26, v1.36.27, v1.36.28 and HEAD went back to v1.36.9.
v1.36.10 is the latest consensus release, so this moves forward. Verified
against an EMPTY module cache (the cold path the in-cluster builder takes), and
./vms/... and ./chains/... build clean.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestCurrentRPCChainVMCompatible has been red since v1.36.11: every release
since then bumped defaultPatch without registering the version against
RPCChainVMProtocol 42, so version.Current was absent from its own
compatibility set. Protocol 42 has not moved across that range — the list was
simply never updated. Backfilled through v1.36.31 (v1.36.29 omitted: that tag
was pushed off a pre-rebase commit and deleted, it is not a release).
v1.36.30 carries the health fix but was tagged before this backfill, so its
tree still fails ./version. v1.36.31 is the green tag and the rollout target.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.36.29 was tagged off a pre-rebase commit that never reached main and has
been deleted. Forward only: the release is v1.36.30, on main.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/health's `bootstrapped` check publishes chains.Nets.Bootstrapping()
verbatim as its message. Nets.chains is keyed by NET id, and the aggregate
appended the map key rather than asking the net which of its chains had not
converged. So every stuck primary-network chain surfaced as one ID,
11111111111111111111111111111111LpoYY — constants.PrimaryNetworkID, i.e.
ids.Empty — a "chain" the chain manager has never heard of. An operator who
went looking for it got "there is no chain with alias/ID", and N stuck chains
collapsed into a single indistinguishable entry that named none of them.
Measured on lux-devnet luxd-0 (v1.36.23), POST health.health:
"bootstrapped":{"message":["11111111111111111111111111111111LpoYY"],
"error":"chains not bootstrapped","contiguousFailures":3213}
The net owns the bootstrapping set, so the net is what can name those chains:
nets.Net grows Bootstrapping() []ids.ID and chains.Nets aggregates those
instead of its own keys. One place holds the set; one place reads it.
The existing TestNetsBootstrapping asserted the defect (require.Contains
bootstrapping, netID) — that assertion is why it survived review. It now
demands the chain ID and refuses the net ID.
This also cuts the first tag containing dada5a31, the GET /v1/health encoder
fix: apihealth.APIReply carries a time.Duration per check, jsonv2 has no
default representation for it, so every GET degraded to
{"healthy":…,"error":"health reply encode failed"} while the status code still
looked right. Reproduced here directly —
json: cannot marshal from Go time.Duration within "/checks/network/duration"
That fix landed on main on 2026-07-25 but no tag ever carried it: v1.36.28
points at 52de3948, seven commits behind. The fleet runs v1.36.2/23/24, all of
which predate it.
Tests (GOWORK=off CGO_ENABLED=0):
chains ok — TestNetsBootstrappingReportsChainsNotNets fails against the
old aggregate with exactly ids.Empty in the list
nets ok — TestNetBootstrappingNamesTheChains
health ok — the three handler tests fail against the jsonv2 encoder
(checks decode empty) and pass against encoding/json
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The Go 1.26 runtimesecret experiment SIGSEGVs at startup under the WSL2 kernel
(go1.26.3, go1.26.4). Detect WSL from /proc/sys/kernel/osrelease and drop the
experiment only there; all other platforms keep stack/register zeroing.
The Go 1.26 runtimesecret experiment (stack/register zeroing for forward
secrecy) SIGSEGVs at startup on the WSL2 kernel, confirmed on go1.26.3 and
go1.26.4. Detect WSL via /proc/sys/kernel/osrelease and select GOEXPERIMENT=none
there; every other platform keeps forward secrecy. Plain `make build` now
produces a working luxd on WSL.
Two endpoints were answering with the right status code and an empty
truth, so every fleet looked instrumented while reporting nothing.
GET /v1/health encoded apihealth.APIReply through jsonv2, which has no
default representation for time.Duration — and every Result carries one.
The marshal therefore failed on every node, every time, and the handler
fell back to {"healthy":…,"error":"health reply encode failed"}. The
status code is written before the encode, so k8s probes and dashboards
stayed green while the body carried no checks at all. Measured on Zoo,
Hanzo and Pars mainnet, node v1.34.9 and v1.36.2 alike.
The type is defined with encoding/json tags and the POST (jsonrpc) path
already encodes it through that codec, which is why POST returned the
full check set while GET returned nothing. One wire type deserves one
encoder: GET now uses encoding/json too, so the two paths agree by
construction. encoding/json also maps invalid UTF-8 in check Details to
U+FFFD instead of failing, which is the behaviour the old jsontext
AllowInvalidUTF8 option was reaching for. The buffer stays: it keeps a
partial encode from shipping a torn body.
/v1/metrics answered 200 with zero bytes for the same shape of reason.
luxfi/metric resolves NewRegistry() to a no-op registry unless the
binary is built with -tags metrics (registry_noop.go, //go:build
!metrics), and only the `full` profile passed it. Images build with the
default `minimal` profile, so every metric in luxd registered into a
black hole while --api-metrics-enabled=true still advertised metrics as
on. Whether metrics are served is the runtime flag's call; a build tag
must not silently overrule it. `metrics` becomes a base tag on every
profile.
Verified: the three new handler tests fail against the jsonv2 encoder
(checks decode empty, error field nil — the exact production symptom)
and pass after. `go tool nm` shows (*registry).Gather absent from a
default build and present once the tag is set.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
M-Chain's committee is the validator set, and RunKeygen refuses a policy
needing more parties than the committee has. mainnet's mchain.json asked
for 7-of-10 against five genesis validators, so a custody key could never
have been generated — the failure would have surfaced the first time
someone tried to bridge, not at deploy.
Adds the invariant as a test over every network's built genesis, and
picks up luxfi/genesis v1.16.4 where the policies are sized to fit
(3-of-5 on mainnet/testnet/devnet, 2-of-3 on localnet).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The build installed M-Chain's plugin binary as
tGVBwRxpmD2aFdg3iYjgRvrCe8Jcmq9UNKxyHMus2NZ8WcD8t — the CB58 of the
retired `thresholdvm` identifier — while the genesis builder declares the
chain with constants.MPCVMID (qCURact1n41FcoNBch8iMVBwc9AWie48D118ZNJ5tBdWrvryS).
The plugin registry resolves a CreateChainTx's vmID to an implementation
by filename, so the two never met: M-Chain was declared in genesis and no
node could start it. The Dockerfile comment had already been renamed to
"mpcvm" without the CB58 being recomputed, which is why it read as correct.
Fixes all five sites (plugin build target, the build-verify list, the
runtime COPY, the comment, and publish_plugin_set.sh) and adds
genesis/builder/mchain_test.go, which pins the vmID literally and asserts
M-Chain is present in the height-0 chain set for mainnet, testnet and
local. A vmID is an immutable one-way door once a chain is created with
it, so it is now covered by a test rather than by five copies of a string.
Bumps luxfi/genesis to v1.16.3, where mchain.json states its quorum as
"policy": "7-of-10" instead of a bare mpcThreshold that reads as the
signer count to an operator and as the polynomial degree to a library.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two things kept the cevm backend from ever reaching a running node.
The fetch pointed at luxcpp/cevm v0.19.0, which does not exist — the libs
publish from the private lux-private/cevm repo, and unauthenticated downloads
404 there. Point it at v0.51.10 in that repo and authenticate with the same
`ghtok` secret the private go modules and lux-accel already use. It stays
best-effort, so a build without the token behaves exactly as before.
Separately, the C-Chain plugin builds with CGO_ENABLED=0 and no tags, so it is
always the pure-Go EVM regardless of what luxd itself links — which is why
production images carry no cevm at all. That build now honours EVM_CGO and
EVM_TAGS, defaulting to today exact behaviour; EVM_CGO=1 EVM_TAGS=cevm is what
makes AutoEVM resolve to CppEVM. The libraries are already in this stage (the
plugin section shares the builder stage rather than starting a new FROM).
Defaults unchanged, so the in-flight v1.36.12 fleet build is byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Each removed package was a SECOND way to do something that already has a
canonical first way. Zero importers workspace-wide; verified identical build
output before/after (pre-existing keyutil breakage unchanged).
- chains/rpc 1250 lines whose own header says 'This replaces lines
941-990' of chains/manager.go. The replacement never
happened; manager.go still registers handlers inline.
- node/config.go 265-line shadow of the canonical node/config/node/config.go.
- vms/mpcvm self-described 'thin backward-compatibility alias' wrappers
- vms/dexvm over github.com/luxfi/chains/*. No backwards compatibility,
only forwards perfection.
Simple made easy: one name, one home, one path.
- LAUNCH_CHECKLIST.md: pre-launch plan for v1.24.11, 154/154 boxes
unchecked, zero references; superseded by LLM.md/CHANGELOG.md/RELEASE.md
and NETWORKS.yaml + genesis/configs for the chain-ID/port tables.
- rename_app.sh, replace_imports.sh: spent one-off sed migrations. Both
are now actively harmful — replace_imports.sh would rewrite the live
github.com/luxfi/vm/manager import in vms/manager.go.
- gen_zoo_addr: stray 3.4MB darwin/arm64 build artifact committed at root;
source preserved at cmd/gen_zoo_addr/gen_zoo_addr.go (.gitignore already
covers the intended output path).
- .ci-status-check.md, .ci-trigger: dated CI-poke stamps, no workflow reads
them.
Also removed (untracked): .claude/worktrees/ agent scratch — a 73MB
whole-tree duplicate that polluted every grep. Detached via git worktree
remove; node HEAD cf8e4c6f51 was an ancestor of main, and the nested
lux/evm worktree HEAD 7156c44f6 is release tag v1.104.12, so no work lost.
Deleted the two fully-merged worktree-agent-* branches it left behind.
No Go source, config, manifest, or .github/ file was touched.
Disconnect/updateUptimeLocked now return (mutated bool, err); VM.Disconnected commits only when the flush wrote uptime state. Before StartTracking (bootstrap churn) and for non-validator peers the flush is a no-op, so the prior unconditional Commit was an empty full-write+fsync on every such disconnect. Skipping it is correct: every writer under stateLock commits its own diff, so no orphaned write depends on the disconnect path.
RED round-2 verdict: SHIP-READY (H1 phantom verified, mutated-gate correct, H2 pre-existing/not-worsened, block-height orthogonal). Isolated from the uncommitted staking-KMS WIP, which is quarantined on feat/staking-kms-native pending its own Blue->Red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The uptime event-delivery plumbing (node/chain_router.go + chains/manager.go
blockHandler) dropped two invariants avalanchego's handler upholds — the
consensus lock and the peer version — each a mainnet-fleet crash. The uptime
tracker itself (vms/platformvm/uptime_tracker.go) is RED-cleared and UNCHANGED.
CRITICAL #2 — P-chain state race (concurrent map writes -> fatal):
chainRouter dispatched Connected/Disconnected on the peer-lifecycle goroutine,
where VM.Disconnected -> tracker.Disconnect (state.SetUptime) + state.Commit
(state.write) ran concurrently with the block acceptor's state.CommitBatch
(state.write) on the engine accept goroutine — no shared lock, so ordinary peer
churn triggered Go "concurrent map writes" and crashed the P-chain node.
Fix: one VM-owned stateLock serializes every commit of shared platform state
that originates outside the engine's lock-free VM.Accept call-out —
- block DECISION: block/executor Block.Accept/Reject hold &vm.stateLock
(supplied to executor.NewManager) around the whole acceptor visit;
- peer/lifecycle: VM.Disconnected and the onReady/onBootstrapStarted/Shutdown
Start/StopTracking uptime flushes hold vm.stateLock.
This is avalanchego's ctx.Lock invariant (accept serialized with
engine.Connected/Disconnected), scoped to the state the platform VM owns and
implemented at the VM: the Lux engine invokes VM.Accept as a lock-free call-out
(no ctx.Lock over accept exists) and a chain-agnostic blockHandler cannot reach a
per-VM accept lock, so routing "through the engine" is not feasible.
CRITICAL #1 — C-Chain nil-version panic on state sync:
blockHandler.Connected passed connector.Connected(ctx, nodeID, nil). proposervm
promotes Connected to coreth, whose state-sync peer tracker compares peer
versions; a nil version deref panics any C-Chain node running state sync (a fresh
join OR a validator rejoining after falling behind — the launch's core invariant).
Fix: plumb the REAL peer version through a versionedConnector capability —
chainRouter.Connected converts the node peer version (luxfi/node/version) to the
VM boundary type (luxfi/version = chain.VersionInfo) and delivers it via
blockHandler.ConnectedWithVersion -> connector.Connected. Audit: geth in-process
Connected is a no-op stub (nil-safe); xvm stores the pointer without deref
(nil-safe); the real coreth plugin derefs -> fixed by feeding the real version.
Tests (SDKROOT + CGO_ENABLED=1, -race):
- vms/platformvm/uptime_state_race_test.go: state.Commit (VM.Disconnected path)
concurrent with state.CommitBatch (acceptor path) under the shared lock — no
race/fatal; verified meaningful (an unlocked probe trips DATA RACE).
- chains/blockhandler_connected_version_test.go + node/chain_router_connected_version_test.go:
the real, converted version reaches the connector non-nil (dedup covered).
Uptime tracker 13/13 and the block/executor reward gate untouched and green;
go build ./... and go vet clean.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
All 5 mainnet validators read uptime=0.0000 / connected=null even for peers
connected 24h+, so prefersCommit (block/executor/options.go) always saw
0% < threshold and withheld staking rewards (~165M LUX gate).
Root causes (four, all fixed):
1. StartTracking never existed/called. The custom uptimeTracker was created
late (onReady) and never baselined validator records, so a long-running
validator's stored upDuration stayed 0 and CalculateUptimePercentFrom
returned 0/total = 0.
2. upDuration flushed only on Disconnect/Shutdown, so a continuously-connected
validator's persisted uptime never grew.
3. service.go left `connected` hard-nil ("IsConnected no longer exists").
4. ROOT: VM.Connected never fired. network.Connected -> chainRouter.Connected
only added to a connectedPeers set and logged; it never dispatched to chain
handlers, and blockHandler.Connected was a no-op. So tracker.Connect was
never called and the connected map was always empty — nothing to accrue.
Fix (faithful port of avalanchego snow/uptime.Manager semantics, luxfi pkgs):
- vms/platformvm/uptime_tracker.go: rewrite as a startedTracking-gated tracker.
StartTracking/StopTracking/StartedTracking/IsConnected added; CalculateUptime
folds the live connected session forward to now using the persisted
lastUpdated (reconstructed from the second-granular uptime.State encoding), so
a connected validator accrues uptime WITHOUT a Disconnect. Before tracking, a
validator is assumed online since its last update (avalanchego baseline);
after tracking, only genuine sessions accrue. Second-granular clock matches
storage. CalculateUptimePercentFrom is the clean upDuration/(now-from) form,
clamped to [0,1].
- vms/platformvm/vm.go: create+register the tracker at Initialize (so bootstrap
Connect events are captured), StartTracking(primary validators) at onReady,
StopTracking on re-bootstrap and Shutdown. Mirrors avalanchego's
onNormalOperationsStarted lifecycle.
- vms/platformvm/service.go: populate `connected` via tracker.IsConnected.
- node/chain_router.go: chainRouter.Connected/Disconnected now dispatch to every
registered chain handler (snapshot under lock, call outside).
- chains/manager.go: blockHandler forwards Connected/Disconnected to its chain's
VM (engineVM) exactly once (dedup set), so the P-chain uptime tracker — and
every VM's peer set — finally observes connectivity.
Consensus safety: prefersCommit is a per-node PREFERENCE feeding oracle-block
voting, not a deterministic value. Fixing the local uptime measurement (which
was uniformly 0) changes only which reward option each node prefers; consensus
still converges by preference voting. No staking/reward math changed.
Tests (vms/platformvm/uptime_tracker_test.go, -race green):
- TestUptimeTrackerLongRunningValidatorAccruesUptime: a 30-day validator reports
~100%. Uses only the shared Calculator API; run against the OLD tracker it
returns 0.0 (FAIL — the exact mainnet symptom), against the fix 1.0 (PASS).
- Continuously-connected-climbs, StartTracking-baselines, flush-on-disconnect,
never-connected-gets-zero, StopTracking-flushes, idempotent double-connect,
dedup, concurrency. block/executor (reward gate) suite green.
Follow-on (NOT in this commit): devnet 5-node uptime-climb verification ->
gated release -> owner-gated one-at-a-time mainnet roll. No image built, no
deploy, no live validator touched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Propagates two just-published upstream fixes into the node (luxd) binary:
- consensus v1.36.7 → v1.36.9: the chain engine no longer os.Exit(1)-kills a
validator on a SetPreference orphan-refusal — it reconciles the VM to the
certified block when the diverged tip is provably uncertified, and halts
fail-closed only on a genuine double-finalization (v1.36.8). Also unifies
consensus's transitive geth pin to v1.20.1, matching node/evm (v1.36.9). The
new PreferenceReconciler VM interface is optional, so node's VMs are
unaffected (they take the non-fatal defer path); no code change required here.
- chains v1.7.7 → v1.7.9: every VM (aivm/keyvm/quantumvm/dexvm) now serializes
through native luxfi/zap struct-is-wire, with canonical-id hardening.
go.mod/go.sum only — dependency-graph resolution, no node code change. Transitive
indirects move forward within-major (bft, cockroachdb/errors, sentry-go,
prometheus/common, go-internal); no major-version bumps. geth stays v1.20.1
(already node's pin). Verified: GOWORK=off CGO_ENABLED=0 go build ./... clean.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The proposervm block wire stored the timestamp as Unix SECONDS (timestamp.Unix()), so the
sub-second window/granularity work was silently truncated to whole seconds when written to the
block: buildChild computed the proposer slot from sub-second time, but verify read back
second-resolution and computed a DIFFERENT slot → errUnexpectedProposer verify-drops at any
window < 1s. Store timestamp.UnixMilli() and read time.UnixMilli() so sub-second timestamps
round-trip and build/verify agree on the slot. Measured at a 100ms window: errUnexpectedProposer
frequent→0, 108→149 TPS, 2.8s→2.2s cadence, 5/5 byte-identical. Forward-only wire change (the
outer proposervm timestamp; the inner EVM block timestamp is independent). Remaining sub-second
floor is now the EVM targetBlockRate, not the proposervm.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two coupled knobs for differentiated cadence — co-located D-Chain/DEX (fast) vs public
C-Chain (standard):
1. MinBlkDelay was hardcoded to the 1s DefaultMinBlockDelay in chains/manager.go, ignoring
its own --proposervm-min-block-delay flag. Now wired node.Config → ManagerConfig →
proposervm.Config (0 ⇒ 1s default). High-throughput nets set it low.
2. Block timestamps were Truncate(time.Second) at 4 sites, quantizing cadence to 1s
regardless of WindowDuration — so a sub-second window inflated slot numbers without finer
time resolution and could NOT produce blocks faster than 1/s. Truncation now tracks
proposer.TimestampGranularity() = min(1s, WindowDuration): mainnet (>=1s) keeps exact
1-second block times; sub-second windows get matching sub-second timestamps.
Measured: 1s window/delay → ~95-104 TPS, byte-identical 5/5 finality (unchanged from before,
no regression). NOTE: true sub-second cadence additionally requires a slot-handoff fix — at
100ms the slot recomputed in buildChild from a drifted 'now' can differ from the slot
timeToBuild scheduled, causing errUnexpectedProposer verify drops; that + multi-sender
saturation is the next step. These knobs are the necessary foundation.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two changes, both node-local:
1. CADENCE — proposer.WindowDuration (the proposer-slot spacing, a validator's min build
delay = slot index x WindowDuration) was a hardcoded 5s const tuned for mainnet-scale
validator sets, flooring small/local block cadence at 5s per slot. It is now a
startup-configurable var (default 5s, unchanged for mainnet) set via the new
--proposervm-window-duration flag → node.Config → ManagerConfig → proposervm.Config →
proposer.SetWindowDuration at VM init. Read by both the windower delay math and
TimeToSlot, so they stay consistent. Measured on a 5-node strict-PQ net: 5s→1s took
sustained C-Chain from 44 TPS / 14s-per-block to 104 TPS / 3.8s-per-block, still 5/5
byte-identical finality.
2. SECURITY (cryptographer MEDIUM-1) — postForkCommonComponents.Verify now refuses a block
carrying a CLASSICAL secp256k1 proposer identity when the chain is strict-PQ
(StakingMLDSASigner set), UNCONDITIONALLY (before the consensusState==Ready gate, so it
also holds during bootstrap/state-sync). Fills the documented-but-unwired 'proposer'
SchemeGate site: the downgrade defense is now an explicit fail-closed in-perimeter gate,
not merely emergent from 20-byte NodeID collision-resistance + upstream enforcement.
Adds SignedBlock.HasClassicalProposer(). Mirrors contract.RefuseUnderStrictPQ.
Pins consensus v1.36.7 (consume-on-error build loop — kills the non-leader BuildBlock spin).
Block + proposervm VM tests green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Under strict-PQ the canonical NodeID is ML-DSA-65-derived (config/node DeriveNodeID →
DeriveMLDSA), so the P-chain validator set and the proposervm windower are ML-DSA-keyed.
But proposervm signed post-fork blocks with the classical TLS leaf and derived the block
Proposer() via ids.NodeIDFromCert — a different value than DeriveMLDSA — so every signed
block (height ≥ 2) failed verifyPostDurangoBlockDelay with errUnexpectedProposer, was
dropped, and rebuilt in an unbounded storm (block 1 survived only because the unsigned
transition block skips the proposer comparison).
The block's offCert slot now carries a scheme-tagged proposer identity [scheme:1B|identity]:
0x90 classical (cert DER → NodeIDFromCert, ECDSA verify) or 0x42 strict-PQ (raw ML-DSA-65
pubkey → DeriveMLDSA(ids.Empty,pub), ML-DSA verify). ML-DSA signing/verification uses a
FIPS 204 §5.2 domain-separation context ('lux-proposervm-block-v1') so a proposer signature
can never be replayed as another ML-DSA message. proposervm.Config gains StakingMLDSASigner
/StakingMLDSAPub, plumbed from StakingConfig through ManagerConfig; exactly one scheme is
active per chain. K=1 nets are unaffected (transition + no-window blocks are unsigned).
Proven on a 5-node strict-PQ local net: sustained multi-block production, every block
built once, gossiped, and finalized byte-identical on all 5 (no automine). Pins consensus
v1.36.6 (engine logger + logged build-drops) which made this diagnosable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The 9 VM lifecycle ops (Initialize, Linearize, SetState, CreateHandlers,
router AddChain, state-sync) were each bounded at a hardcoded 30s. A cold
coreth 'Regenerate historical state' pass after an unclean shutdown takes
67-134s on the mainnet C-Chain, blowing that budget → context cancelled
mid-init → VM marked failed → C-Chain route never registered → the recurring
post-restart 404 that required a second restart to clear. One named bounded
constant (10m) covers regen with margin while still surfacing a truly-hung VM.
Stop stays 10s. This is the third stacked restart-fragility bug after
skip-bootstrap frontier (v1.36.11) and rejoin discriminator (v1.36.13).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The #66/#74 durable rejoin fix was INERT for the C-Chain — the chain whose 40h
mainnet freeze motivated it. chains/manager.go gated expectsStakedBeacons on
ids.IsNativeChain(chainParams.ID) (the blockchain ID), but ids.IsNativeChain only
matches the symbolic 111...C alias; every deployed C/X/Q has a HASH blockchain ID
(devnet 21HieZng, mainnet 2wRdZG), so isNativeChain was ALWAYS false and under the
production --skip-bootstrap=true the beacons were emptied -> a behind C-Chain named
its stale local tip the frontier and never caught up. Verified on devnet v1.36.12:
C-Chain wedged at height 0 ('using empty beacons for single-node mode').
Fix: discriminate on the VALIDATING NET (chainParams.ChainID == PrimaryNetworkID)
via new chainValidatesOnPrimaryNetwork — PrimaryNetworkID for C/X/Q, the sovereign
net ID for L2s. C/X/Q now keep staked beacons under --skip-bootstrap (peer-sync a
behind validator); L2s keep the empty-beacon single-node path. New regression
TestChainValidatesOnPrimaryNetwork_RealHashChainID exercises the real discriminator
with hash IDs; TestRED_EmptyStakedSetFailsSafe still green (forged-frontier gate intact).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Captures the devnet-canary findings so the gated rollout is resumable: the
v1.36.11 EVM-plugin api skew (fixed in v1.36.12), the v1.36.2->v1.36.x P-Chain
codec migration (one-time wipe + proven cross-version re-bootstrap), the durable
rejoin mechanism, per-net RewardManager addresses/config shape, and the
one-at-a-time verify-tip roll protocol. Also folds in the pre-existing
RewardManager->DAO-Safe C-Chain design note.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.36.11's node binary is correct (durable rejoin fix fe23efd1f2), but its image
baked VM plugins built against a stale luxfi/api: the C-Chain EVM plugin from
luxfi/evm@v1.104.8 and the D-Chain dexvm plugin from luxfi/dex@v1.5.15 both resolve
api v1.0.15, while the node pins api v1.0.16. api v1.0.16 APPENDED
InitializeResponse.Capabilities (uint64) for the Quasar-export handshake (api
1f2dc5a). The node decodes that field; the stale plugins never encode it, so
vms/rpcchainvm/zap/client.go fails every EVM VM Initialize with
'zap decode initialize response: unexpected EOF'. Native VMs (P/X/Q) are unaffected;
every EVM chain (C, D, and the L2 EVMs) fails to boot. Verified on devnet (published
v1.36.11 digest c3cf92a6): P/X re-bootstrap fine, C+D deterministically EOF.
Fix (image-only; node source unchanged beyond the version bump):
- EVM_VERSION v1.104.8 -> v1.104.9 (api v1.0.15 -> v1.0.16, indirect via luxfi/vm)
- force luxfi/api@v1.0.16 in the dexvm build stage (no dex release pins v1.0.16 yet)
- CHAINS_REF v1.7.6 already carries api v1.0.16 (the other 10 VMs were fine)
The api bump is code-free for plugins (chains v1.7.4->v1.7.5 adopted it go.mod-only).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Node has ZERO .proto files and ZERO .pb.go — the wire is hand-written ZAP schemas
(proto/{platformvm,vm,p2p,sync}/*_zap.go). The buf/protoc tooling around it was
orphaned: removed proto/{buf.yaml,buf.gen.yaml,Dockerfile.buf,buf.md,README.md},
scripts/protobuf_codegen.sh, the Taskfile generate-protobuf + check-generate-protobuf
targets, ci.yml buf-lint + check_generated_protobuf jobs (the latter ran the deleted
script → would fail CI), and the buf-lint.yml workflow. defaultPatch 10→11 (dev
version string; image already correct via ldflags). Binary unchanged — tooling/CI only.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Durable root-cause fix for the behind-validator rejoin wedge (mainnet luxd-0:
C-Chain frozen ~40h at a stale height, only a manual chaindata wipe unstuck it).
Root cause: buildChain computed the bootstrap frontier-sync discriminator as
`expectsStakedBeacons := !m.SkipBootstrap && native && !platform` AND emptied the
beacon set whenever `m.SkipBootstrap`. Production validators hardcode
--skip-bootstrap=true (to skip the initial bootstrap WAIT), so a real multi-
validator native chain (C/X/Q) got expectsStakedBeacons=false + EMPTY beacons.
FrontierTip then reported FrontierNoBeacons ("nothing to sync to"), the node named
its STALE local last-accepted the network frontier, transitioned the VM to normal
operation there, and never fetched the gap from its 4 healthy peers — the wedge
survived every restart because --skip-bootstrap is persistent config. The entire
peer-sync/self-heal machinery (bootstrap_sync.go) was dead code under skip-bootstrap.
Fix: drive the discriminator from SYBIL PROTECTION (the true "real staked network"
signal, already wired into ManagerConfig), not --skip-bootstrap. A sybil-protected
native non-platform chain now keeps its staked beacon set and expectsStakedBeacons
even under --skip-bootstrap, so a behind validator always catches up from peers.
A genuine single-node / dev net runs sybil-protection OFF and still takes the
empty-beacon immediate-start path. A single-VALIDATOR staked net (self-only set) is
handled by a new FrontierTip hasExternalBeacons rule (placed after the P-ready gate),
so it immediate-starts without a Connecting hang while a >=2 set runs the quorum.
This matches the proven live remediation (flipping skip-bootstrap=false via the
.allow-bootstrap marker caught luxd-0 up to tip in ~90s) and preserves every RED
safety invariant (empty staked set still fails safe — TestRED_EmptyStakedSetFailsSafe).
Tests: TestChainExpectsStakedBeacons_SybilDrivesNotSkipBootstrap,
TestNodeBootstrap_SelfOnlyStakedSet_ReportsNoBeacons, and the headline
TestNodeBootstrap_BehindValidator_StakedSet_CatchesUpNoWipe (N-blocks-behind →
restart → catches up from peers to tip, no wipe). Full chains suite green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both executor backends re-declared a local SharedMemory interface with the
weak Apply(map[ids.ID]interface{}, ...interface{}) signature — but nothing
referenced it. The real atomic path uses Runtime.SharedMemory (= the narrow
atomic.SharedMemory: Apply(map[ids.ID]*atomic.Requests, ...database.Batch)).
Pure dead code; removed both. 28 platformvm packages green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fx set is a CLOSED sum type (secp256k1fx | nftfx | propertyfx), fixed at
compile time — no hot-loading, no genesis-configured fx. So the runtime
'which fx owns this value' dispatch needs no reflection: it is a total match on
the variant tag. Replaced reflect.TypeOf(val) + map[reflect.Type]int with:
- fxKindOf(val): a Go type switch over the closed set of fx primitive types
(compiler-checked exhaustive; lowers to a jump on the interface type tag),
returning the value's wire.TypeKind — the same family tag the wire envelope
already carries.
- FxIndex: a dense [16]int array indexed by that TypeKind (one bounds-checked
load; -1 = unregistered), filled by the SAME fx.(type) switch NewCustomParser
already ran — no separate reflect registration.
getFx (semantic verifier + tx_init) is now fxKindOf → array index: zero reflect,
zero map-hash, compile-time-checked. Deleted registerFxTypes + all
map[reflect.Type]int fields/params (parser, block/parser, vm, backend). Node-only
(uses already-imported utxo fx types + wire.TypeKind); no dep cascade.
Full xvm suite green (11 pkgs — secp/nft/property verify dispatch exercised).
This removes the LAST reflection from the X-chain tx/verify path.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Superset of transitive module hashes go resolves at v1.36.7; -mod=readonly build
+ full 155-package test suite green. No go.mod change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zap v1.2.2 (zero-copy SetBytes + Builder pool) + utxo v0.5.3 (pooled wire
builders + SetBytesFixed) cut X-chain tx wire composition ~1.9x (2551->1345ns,
37->19 allocs on the isolated composite; the X build path benefits proportionally
without touching P-chain parse, which stays at 705ns/3allocs). P/X/xvm/
components-lux suites all green against the new deps.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Supersedes v1.36.4's build recipe which used a build-time 'replace node => /build'
(forbidden: no local replace directives). Instead, chains v1.7.5 pins the real,
published node v1.36.4 tag, so the baked VM plugins resolve node the normal way —
Dockerfile just clones chains and builds; CHAINS_REF v1.7.4 -> v1.7.5; go.mod
chains v1.7.4 -> v1.7.5; genproto realigned. Zero replace directives anywhere.
tidy + -mod=readonly + cold 'go mod download all' clean.
Note: luxfi node/geth/utxo git tags are being periodically wiped by an external
tag-sync (root cause of the 'unknown revision' failures); all four pinned tags
(node v1.36.4, chains v1.7.5, utxo v0.5.1, geth v1.17.12) re-verified present
immediately before this build races the sync window.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plugin stage cloned chains and built each VM's cmd/plugin standalone, which
resolved chains' pinned github.com/luxfi/node@v1.30.6 — a wiped/disjoint release
tag (325 orphan commits, no merge-base with main) absent from the remote, so the
hermetic build died with 'unknown revision v1.30.6' and the required bridgevm/
mpcvm/zkvm plugins went missing (FATAL).
Fix: inject 'replace github.com/luxfi/node => /build' (the exact vX.Y.Z node source
being built) into every /tmp/chains go.mod before building. This is what the
existing 'plugins in lockstep with the host node' intent actually requires — the
baked plugins now match the node they run in instead of an ancient pin. chains is
the main module during the plugin build, so node(/build)'s own require of chains
resolves back to /tmp/chains (main-module-wins) — no version fetch, no loop. Also
bumped CHAINS_REF default v1.7.2 -> v1.7.4 to match node's go.mod chains pin.
Verified: all 10 required plugins build cold (fresh GOMODCACHE, CGO_ENABLED=0) into
real binaries against local node v1.36.4.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The xvm codec kill depends on utxo/wire's TransferableOut/In + nftfx/propertyfx
envelopes, which were only in the local ~/work/lux/utxo working tree (pinned via
'replace github.com/luxfi/utxo => ../utxo'). That local-path replace works for a
local build but breaks the hermetic Docker/CI build ('open /utxo/go.mod: no such
file or directory'). Published those two additive wire commits as utxo v0.5.1
(clean ff on utxo main, +2 over v0.5.0) and pin it here — the exact code node was
built+tested against. Dropped the replace. Cold-cache 'go mod download all' is now
clean; xvm/components/lux/wallet tests green against v0.5.1.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The codec kill is complete. Every node VM (platformvm, xvm, warp, proposervm,
components/lux) and every chains app-chain VM (bridgevm, zkvm, mpcvm — now native
in chains v1.7.4) marshals via native ZAP struct-is-wire. Nothing imports
node/vms/pcodecs anymore, so the package — the last reflection/serialize-tag
dispatcher (a thin alias over proto/zap_codec's LinearCodec) — is deleted.
- rm vms/pcodecs + vms/pcodecs/pcodecsmock (zero consumers; verified whole tree).
- go.mod: chains v1.7.2 -> v1.7.4 (the pcodecs-free chains), precompile
v0.19.0 -> v0.19.1. Realigned genproto so the split googleapis/rpc module
resolves by longest-prefix (no monolith ambiguity); tidy clean, -mod=readonly
build clean.
- version -> v1.36.4 (constants.go defaultPatch, compatibility.json under the
same RPCChainVM protocol as v1.36.3 — no protocol change, only a codec rip).
There is one and only one way to put a struct on the wire: ZAP.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Tail of the xvm codec kill (ddb3fbca93): the X-chain wallet builder + signer
now rebuild signed wire bytes as unsigned ‖ fx credential envelopes over the
native luxfi/utxo/wire form, and components/lux parses fx Inputs from their wire
envelope by concrete type. No linearcodec, no reflection on the wallet path.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The non-codec main changes (Dockerfile EVM_VERSION v1.104.8, chains/manager
GetContext size-chunking for behind-validator resync + Quasar EXPORT frontier
bridge, warp/signature, rpcchainvm/zap client) that the branch lacked. The
codec-era main files (codec.go/parse.go/tx.go etc.) are intentionally
superseded by this branch's native-ZAP struct-is-wire — not reintroduced.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
RED-verified consensus-safety fix: zap.Parse truncates to the header size
field, so a block buffer with extra tail bytes wraps the SAME message but
ID=hash(bytes) differs — a block-hash malleability / fork vector. setID now
rejects msg.Size() != len(bytes) with ErrExtraSpace. (The P-chain TX envelope
already guards this at tx.go:66; only block Parse had the gap.)
Renamed codec_determinism_test.go -> block_determinism_test.go and converted
its assertions to native ZAP: New*Block + Parse(b) (no Codec), native golden
AbortBlock bytes (65B: zap header + kind/parent/height/time object), byte-
stability + BlockID-stability + trailing-bytes-rejected. Block + txs green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
defaultMinor/Patch -> 36/3 (Dockerfile injects no version ldflags, so the
default IS the released binary's self-reported version; v1.36.0-2 shipped
self-reporting 1.32.11). v1.36.3 registered under RPCChainVM protocol 42.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Node builds + full test sweep green (155 ok / 0 FAIL) against PUBLISHED
modules only: consensus v1.36.1, zap v1.2.0, utxo v0.3.7. The local zap/utxo
replaces are gone; the upstream nftfx/propertyfx wire (utxo e790d39) ships as
v0.3.8 with the xvm struct-is-wire patch, which is the only remaining pcodecs
consumer and lands in the next release.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Route registration, client URL builders, and log lines all move to /v1/
(wallet primary api, chains/rpc handler_manager+chain_integration, xvm
client+wallet_client, xsvm api client, multi-network example, manager logs).
Zero /ext/ literals remain. One endpoint namespace, no transition alias.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zeekay