/v1/health's `bootstrapped` check publishes chains.Nets.Bootstrapping()
verbatim as its message. Nets.chains is keyed by NET id, and the aggregate
appended the map key rather than asking the net which of its chains had not
converged. So every stuck primary-network chain surfaced as one ID,
11111111111111111111111111111111LpoYY — constants.PrimaryNetworkID, i.e.
ids.Empty — a "chain" the chain manager has never heard of. An operator who
went looking for it got "there is no chain with alias/ID", and N stuck chains
collapsed into a single indistinguishable entry that named none of them.
Measured on lux-devnet luxd-0 (v1.36.23), POST health.health:
"bootstrapped":{"message":["11111111111111111111111111111111LpoYY"],
"error":"chains not bootstrapped","contiguousFailures":3213}
The net owns the bootstrapping set, so the net is what can name those chains:
nets.Net grows Bootstrapping() []ids.ID and chains.Nets aggregates those
instead of its own keys. One place holds the set; one place reads it.
The existing TestNetsBootstrapping asserted the defect (require.Contains
bootstrapping, netID) — that assertion is why it survived review. It now
demands the chain ID and refuses the net ID.
This also cuts the first tag containing dada5a31, the GET /v1/health encoder
fix: apihealth.APIReply carries a time.Duration per check, jsonv2 has no
default representation for it, so every GET degraded to
{"healthy":…,"error":"health reply encode failed"} while the status code still
looked right. Reproduced here directly —
json: cannot marshal from Go time.Duration within "/checks/network/duration"
That fix landed on main on 2026-07-25 but no tag ever carried it: v1.36.28
points at 52de3948, seven commits behind. The fleet runs v1.36.2/23/24, all of
which predate it.
Tests (GOWORK=off CGO_ENABLED=0):
chains ok — TestNetsBootstrappingReportsChainsNotNets fails against the
old aggregate with exactly ids.Empty in the list
nets ok — TestNetBootstrappingNamesTheChains
health ok — the three handler tests fail against the jsonv2 encoder
(checks decode empty) and pass against encoding/json
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Each removed package was a SECOND way to do something that already has a
canonical first way. Zero importers workspace-wide; verified identical build
output before/after (pre-existing keyutil breakage unchanged).
- chains/rpc 1250 lines whose own header says 'This replaces lines
941-990' of chains/manager.go. The replacement never
happened; manager.go still registers handlers inline.
- node/config.go 265-line shadow of the canonical node/config/node/config.go.
- vms/mpcvm self-described 'thin backward-compatibility alias' wrappers
- vms/dexvm over github.com/luxfi/chains/*. No backwards compatibility,
only forwards perfection.
Simple made easy: one name, one home, one path.
The uptime event-delivery plumbing (node/chain_router.go + chains/manager.go
blockHandler) dropped two invariants avalanchego's handler upholds — the
consensus lock and the peer version — each a mainnet-fleet crash. The uptime
tracker itself (vms/platformvm/uptime_tracker.go) is RED-cleared and UNCHANGED.
CRITICAL #2 — P-chain state race (concurrent map writes -> fatal):
chainRouter dispatched Connected/Disconnected on the peer-lifecycle goroutine,
where VM.Disconnected -> tracker.Disconnect (state.SetUptime) + state.Commit
(state.write) ran concurrently with the block acceptor's state.CommitBatch
(state.write) on the engine accept goroutine — no shared lock, so ordinary peer
churn triggered Go "concurrent map writes" and crashed the P-chain node.
Fix: one VM-owned stateLock serializes every commit of shared platform state
that originates outside the engine's lock-free VM.Accept call-out —
- block DECISION: block/executor Block.Accept/Reject hold &vm.stateLock
(supplied to executor.NewManager) around the whole acceptor visit;
- peer/lifecycle: VM.Disconnected and the onReady/onBootstrapStarted/Shutdown
Start/StopTracking uptime flushes hold vm.stateLock.
This is avalanchego's ctx.Lock invariant (accept serialized with
engine.Connected/Disconnected), scoped to the state the platform VM owns and
implemented at the VM: the Lux engine invokes VM.Accept as a lock-free call-out
(no ctx.Lock over accept exists) and a chain-agnostic blockHandler cannot reach a
per-VM accept lock, so routing "through the engine" is not feasible.
CRITICAL #1 — C-Chain nil-version panic on state sync:
blockHandler.Connected passed connector.Connected(ctx, nodeID, nil). proposervm
promotes Connected to coreth, whose state-sync peer tracker compares peer
versions; a nil version deref panics any C-Chain node running state sync (a fresh
join OR a validator rejoining after falling behind — the launch's core invariant).
Fix: plumb the REAL peer version through a versionedConnector capability —
chainRouter.Connected converts the node peer version (luxfi/node/version) to the
VM boundary type (luxfi/version = chain.VersionInfo) and delivers it via
blockHandler.ConnectedWithVersion -> connector.Connected. Audit: geth in-process
Connected is a no-op stub (nil-safe); xvm stores the pointer without deref
(nil-safe); the real coreth plugin derefs -> fixed by feeding the real version.
Tests (SDKROOT + CGO_ENABLED=1, -race):
- vms/platformvm/uptime_state_race_test.go: state.Commit (VM.Disconnected path)
concurrent with state.CommitBatch (acceptor path) under the shared lock — no
race/fatal; verified meaningful (an unlocked probe trips DATA RACE).
- chains/blockhandler_connected_version_test.go + node/chain_router_connected_version_test.go:
the real, converted version reaches the connector non-nil (dedup covered).
Uptime tracker 13/13 and the block/executor reward gate untouched and green;
go build ./... and go vet clean.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
All 5 mainnet validators read uptime=0.0000 / connected=null even for peers
connected 24h+, so prefersCommit (block/executor/options.go) always saw
0% < threshold and withheld staking rewards (~165M LUX gate).
Root causes (four, all fixed):
1. StartTracking never existed/called. The custom uptimeTracker was created
late (onReady) and never baselined validator records, so a long-running
validator's stored upDuration stayed 0 and CalculateUptimePercentFrom
returned 0/total = 0.
2. upDuration flushed only on Disconnect/Shutdown, so a continuously-connected
validator's persisted uptime never grew.
3. service.go left `connected` hard-nil ("IsConnected no longer exists").
4. ROOT: VM.Connected never fired. network.Connected -> chainRouter.Connected
only added to a connectedPeers set and logged; it never dispatched to chain
handlers, and blockHandler.Connected was a no-op. So tracker.Connect was
never called and the connected map was always empty — nothing to accrue.
Fix (faithful port of avalanchego snow/uptime.Manager semantics, luxfi pkgs):
- vms/platformvm/uptime_tracker.go: rewrite as a startedTracking-gated tracker.
StartTracking/StopTracking/StartedTracking/IsConnected added; CalculateUptime
folds the live connected session forward to now using the persisted
lastUpdated (reconstructed from the second-granular uptime.State encoding), so
a connected validator accrues uptime WITHOUT a Disconnect. Before tracking, a
validator is assumed online since its last update (avalanchego baseline);
after tracking, only genuine sessions accrue. Second-granular clock matches
storage. CalculateUptimePercentFrom is the clean upDuration/(now-from) form,
clamped to [0,1].
- vms/platformvm/vm.go: create+register the tracker at Initialize (so bootstrap
Connect events are captured), StartTracking(primary validators) at onReady,
StopTracking on re-bootstrap and Shutdown. Mirrors avalanchego's
onNormalOperationsStarted lifecycle.
- vms/platformvm/service.go: populate `connected` via tracker.IsConnected.
- node/chain_router.go: chainRouter.Connected/Disconnected now dispatch to every
registered chain handler (snapshot under lock, call outside).
- chains/manager.go: blockHandler forwards Connected/Disconnected to its chain's
VM (engineVM) exactly once (dedup set), so the P-chain uptime tracker — and
every VM's peer set — finally observes connectivity.
Consensus safety: prefersCommit is a per-node PREFERENCE feeding oracle-block
voting, not a deterministic value. Fixing the local uptime measurement (which
was uniformly 0) changes only which reward option each node prefers; consensus
still converges by preference voting. No staking/reward math changed.
Tests (vms/platformvm/uptime_tracker_test.go, -race green):
- TestUptimeTrackerLongRunningValidatorAccruesUptime: a 30-day validator reports
~100%. Uses only the shared Calculator API; run against the OLD tracker it
returns 0.0 (FAIL — the exact mainnet symptom), against the fix 1.0 (PASS).
- Continuously-connected-climbs, StartTracking-baselines, flush-on-disconnect,
never-connected-gets-zero, StopTracking-flushes, idempotent double-connect,
dedup, concurrency. block/executor (reward gate) suite green.
Follow-on (NOT in this commit): devnet 5-node uptime-climb verification ->
gated release -> owner-gated one-at-a-time mainnet roll. No image built, no
deploy, no live validator touched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two coupled knobs for differentiated cadence — co-located D-Chain/DEX (fast) vs public
C-Chain (standard):
1. MinBlkDelay was hardcoded to the 1s DefaultMinBlockDelay in chains/manager.go, ignoring
its own --proposervm-min-block-delay flag. Now wired node.Config → ManagerConfig →
proposervm.Config (0 ⇒ 1s default). High-throughput nets set it low.
2. Block timestamps were Truncate(time.Second) at 4 sites, quantizing cadence to 1s
regardless of WindowDuration — so a sub-second window inflated slot numbers without finer
time resolution and could NOT produce blocks faster than 1/s. Truncation now tracks
proposer.TimestampGranularity() = min(1s, WindowDuration): mainnet (>=1s) keeps exact
1-second block times; sub-second windows get matching sub-second timestamps.
Measured: 1s window/delay → ~95-104 TPS, byte-identical 5/5 finality (unchanged from before,
no regression). NOTE: true sub-second cadence additionally requires a slot-handoff fix — at
100ms the slot recomputed in buildChild from a drifted 'now' can differ from the slot
timeToBuild scheduled, causing errUnexpectedProposer verify drops; that + multi-sender
saturation is the next step. These knobs are the necessary foundation.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two changes, both node-local:
1. CADENCE — proposer.WindowDuration (the proposer-slot spacing, a validator's min build
delay = slot index x WindowDuration) was a hardcoded 5s const tuned for mainnet-scale
validator sets, flooring small/local block cadence at 5s per slot. It is now a
startup-configurable var (default 5s, unchanged for mainnet) set via the new
--proposervm-window-duration flag → node.Config → ManagerConfig → proposervm.Config →
proposer.SetWindowDuration at VM init. Read by both the windower delay math and
TimeToSlot, so they stay consistent. Measured on a 5-node strict-PQ net: 5s→1s took
sustained C-Chain from 44 TPS / 14s-per-block to 104 TPS / 3.8s-per-block, still 5/5
byte-identical finality.
2. SECURITY (cryptographer MEDIUM-1) — postForkCommonComponents.Verify now refuses a block
carrying a CLASSICAL secp256k1 proposer identity when the chain is strict-PQ
(StakingMLDSASigner set), UNCONDITIONALLY (before the consensusState==Ready gate, so it
also holds during bootstrap/state-sync). Fills the documented-but-unwired 'proposer'
SchemeGate site: the downgrade defense is now an explicit fail-closed in-perimeter gate,
not merely emergent from 20-byte NodeID collision-resistance + upstream enforcement.
Adds SignedBlock.HasClassicalProposer(). Mirrors contract.RefuseUnderStrictPQ.
Pins consensus v1.36.7 (consume-on-error build loop — kills the non-leader BuildBlock spin).
Block + proposervm VM tests green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Under strict-PQ the canonical NodeID is ML-DSA-65-derived (config/node DeriveNodeID →
DeriveMLDSA), so the P-chain validator set and the proposervm windower are ML-DSA-keyed.
But proposervm signed post-fork blocks with the classical TLS leaf and derived the block
Proposer() via ids.NodeIDFromCert — a different value than DeriveMLDSA — so every signed
block (height ≥ 2) failed verifyPostDurangoBlockDelay with errUnexpectedProposer, was
dropped, and rebuilt in an unbounded storm (block 1 survived only because the unsigned
transition block skips the proposer comparison).
The block's offCert slot now carries a scheme-tagged proposer identity [scheme:1B|identity]:
0x90 classical (cert DER → NodeIDFromCert, ECDSA verify) or 0x42 strict-PQ (raw ML-DSA-65
pubkey → DeriveMLDSA(ids.Empty,pub), ML-DSA verify). ML-DSA signing/verification uses a
FIPS 204 §5.2 domain-separation context ('lux-proposervm-block-v1') so a proposer signature
can never be replayed as another ML-DSA message. proposervm.Config gains StakingMLDSASigner
/StakingMLDSAPub, plumbed from StakingConfig through ManagerConfig; exactly one scheme is
active per chain. K=1 nets are unaffected (transition + no-window blocks are unsigned).
Proven on a 5-node strict-PQ local net: sustained multi-block production, every block
built once, gossiped, and finalized byte-identical on all 5 (no automine). Pins consensus
v1.36.6 (engine logger + logged build-drops) which made this diagnosable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The 9 VM lifecycle ops (Initialize, Linearize, SetState, CreateHandlers,
router AddChain, state-sync) were each bounded at a hardcoded 30s. A cold
coreth 'Regenerate historical state' pass after an unclean shutdown takes
67-134s on the mainnet C-Chain, blowing that budget → context cancelled
mid-init → VM marked failed → C-Chain route never registered → the recurring
post-restart 404 that required a second restart to clear. One named bounded
constant (10m) covers regen with margin while still surfacing a truly-hung VM.
Stop stays 10s. This is the third stacked restart-fragility bug after
skip-bootstrap frontier (v1.36.11) and rejoin discriminator (v1.36.13).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The #66/#74 durable rejoin fix was INERT for the C-Chain — the chain whose 40h
mainnet freeze motivated it. chains/manager.go gated expectsStakedBeacons on
ids.IsNativeChain(chainParams.ID) (the blockchain ID), but ids.IsNativeChain only
matches the symbolic 111...C alias; every deployed C/X/Q has a HASH blockchain ID
(devnet 21HieZng, mainnet 2wRdZG), so isNativeChain was ALWAYS false and under the
production --skip-bootstrap=true the beacons were emptied -> a behind C-Chain named
its stale local tip the frontier and never caught up. Verified on devnet v1.36.12:
C-Chain wedged at height 0 ('using empty beacons for single-node mode').
Fix: discriminate on the VALIDATING NET (chainParams.ChainID == PrimaryNetworkID)
via new chainValidatesOnPrimaryNetwork — PrimaryNetworkID for C/X/Q, the sovereign
net ID for L2s. C/X/Q now keep staked beacons under --skip-bootstrap (peer-sync a
behind validator); L2s keep the empty-beacon single-node path. New regression
TestChainValidatesOnPrimaryNetwork_RealHashChainID exercises the real discriminator
with hash IDs; TestRED_EmptyStakedSetFailsSafe still green (forged-frontier gate intact).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Durable root-cause fix for the behind-validator rejoin wedge (mainnet luxd-0:
C-Chain frozen ~40h at a stale height, only a manual chaindata wipe unstuck it).
Root cause: buildChain computed the bootstrap frontier-sync discriminator as
`expectsStakedBeacons := !m.SkipBootstrap && native && !platform` AND emptied the
beacon set whenever `m.SkipBootstrap`. Production validators hardcode
--skip-bootstrap=true (to skip the initial bootstrap WAIT), so a real multi-
validator native chain (C/X/Q) got expectsStakedBeacons=false + EMPTY beacons.
FrontierTip then reported FrontierNoBeacons ("nothing to sync to"), the node named
its STALE local last-accepted the network frontier, transitioned the VM to normal
operation there, and never fetched the gap from its 4 healthy peers — the wedge
survived every restart because --skip-bootstrap is persistent config. The entire
peer-sync/self-heal machinery (bootstrap_sync.go) was dead code under skip-bootstrap.
Fix: drive the discriminator from SYBIL PROTECTION (the true "real staked network"
signal, already wired into ManagerConfig), not --skip-bootstrap. A sybil-protected
native non-platform chain now keeps its staked beacon set and expectsStakedBeacons
even under --skip-bootstrap, so a behind validator always catches up from peers.
A genuine single-node / dev net runs sybil-protection OFF and still takes the
empty-beacon immediate-start path. A single-VALIDATOR staked net (self-only set) is
handled by a new FrontierTip hasExternalBeacons rule (placed after the P-ready gate),
so it immediate-starts without a Connecting hang while a >=2 set runs the quorum.
This matches the proven live remediation (flipping skip-bootstrap=false via the
.allow-bootstrap marker caught luxd-0 up to tip in ~90s) and preserves every RED
safety invariant (empty staked set still fails safe — TestRED_EmptyStakedSetFailsSafe).
Tests: TestChainExpectsStakedBeacons_SybilDrivesNotSkipBootstrap,
TestNodeBootstrap_SelfOnlyStakedSet_ReportsNoBeacons, and the headline
TestNodeBootstrap_BehindValidator_StakedSet_CatchesUpNoWipe (N-blocks-behind →
restart → catches up from peers to tip, no wipe). Full chains suite green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The non-codec main changes (Dockerfile EVM_VERSION v1.104.8, chains/manager
GetContext size-chunking for behind-validator resync + Quasar EXPORT frontier
bridge, warp/signature, rpcchainvm/zap client) that the branch lacked. The
codec-era main files (codec.go/parse.go/tx.go etc.) are intentionally
superseded by this branch's native-ZAP struct-is-wire — not reintroduced.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Route registration, client URL builders, and log lines all move to /v1/
(wallet primary api, chains/rpc handler_manager+chain_integration, xvm
client+wallet_client, xsvm api client, multi-network example, manager logs).
Zero /ext/ literals remain. One endpoint namespace, no transition alias.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.36 upstreamed round-scoped view-change INTO the engine (internal prevote/POL
in attestation/reconcile/cert), dropping the external hooks config.Parameters.
ViewChange + Runtime.HandleIncomingPrevote. Ripped the node's now-redundant
external plumbing: the ViewChange enable block (LUX_CONSENSUS_VIEW_CHANGE gate),
the quorumKindPrevote gossip routing + HandleIncomingPrevote call, the dead
BroadcastPrevote gossiper method, and the quorumKindPrevote envelope kind. The
engine owns view-change natively now (fail-secure halt on 2a-n>f unchanged).
Whole node builds EXIT 0 on v1.36.1; luxd binary compiles (55M).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Wire the two-tier consensus export boundary through the node so the EVM `finalized`/`safe` tags and
the warp export gate track the ⅔-stake Quasar tip, never the reorgable Nova/accept tip:
- Set NetworkConfig.QuasarObserver to push each EXPORT (Quasar) frontier advance into the raw inner
VM (SetLastQuasarFinalized) — the eth/warp backends live there, not on the proposervm wrapper.
Interface-gated: only a VM exposing the export sink (the C-Chain EVM) participates.
- On boot, re-seed the consensus export frontier from the VM's DURABLE Quasar height
(SyncQuasarFrontier) so GetQuasarTip/QuasarHeight do not regress on restart.
Also remove the node's dead references to the v1.36-deleted Tendermint braid (the consensus engine
dropped it in 174af3c31), which no longer compile against the v1.36 engine:
- the LUX_CONSENSUS_VIEW_CHANGE opt-in (params.ViewChange is gone — Nova is the sole decider),
- the quorumKindPrevote gossip kind + BroadcastPrevote + HandleIncomingPrevote routing (no prevotes;
the ⅔ Quasar attestation rides the ordinary accept-vote gossip). Keep the braid dead.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Benchmark-proven live mainnet bug (250-trader DEX load): a validator that falls behind cannot
resync the C-Chain. GetContext (the catch-up "context response", wire = Ancestors) bounded the
response ONLY by block COUNT (maxContextBlocks=256). Under heavy DEX load 256 blocks summed to
3.4-5.7 MB, exceeding the 2 MB peer message cap (the zstd compressor refuses uncompressed input
above constants.DefaultMaxMessageSize to prevent a decompression bomb), so msgCreator.Ancestors
FAILED to build and the behind validator received NOTHING — permanently stuck while the tip
advanced (luxd-3 stuck at 256 while tip went 293→300, looping empty-block builds).
Fix (piece 1/3 of the layered design — defense in depth): GetContext now ALSO bounds the
response by serialized SIZE. It stops adding blocks before the accumulated payload would exceed
byteBudget (cap − 128 KiB envelope margin), but ALWAYS includes at least one block so a behind
node makes progress every round; the requester re-requests the remaining gap (context fetch is
already a multi-round oldest-first fill). A single block that alone exceeds the budget is still
served (best-effort) so the walk never deadlocks — the forthcoming trust-tiered validator cap
(pieces 2/3) gives such a block the send headroom; a stranger's tight cap correctly rejects it.
Tests (chains/context_chunk_test.go): TestGetContext_ChunksBySize_FitsUnderCap (100×150 KiB
chain → 12 blocks / 1.84 MB, under budget, vs ~15 MB packed by count — fail-on-old);
TestGetContext_SingleOversizeBlock_StillServed (one oversize block still served, no deadlock).
Pieces 2/3 to follow: trust-tiered message cap (validator peers get headroom, strangers keep
2 MB) + confirm the inbound throttler/benchlist penalizes oversized-message senders.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Removes the forbidden Avalanche term from code comments in chains/manager.go,
chains/quorum.go, and vms/proposervm/proposer/windower_determinism_test.go. Reworded to
preserve exact meaning; comments only, no identifier/behavior change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The "round-scoped view-change ENABLED for chain K=21 alpha=15" line read as if K=21/α=15
were the finality committee — misleading. It is the Snowman SAMPLE preset. The α-of-K
cert and the view-change POL/precommit are sized to the LIVE validator set at runtime
(effectiveCommittee/bftCommittee; 5 validators → K=5/α=4), and the engine already logs
the effective (K,α) on each committee re-clamp. Relabel the fields presetK/presetAlpha
and add a note pointing at the runtime committee-clamp log. Log-only; no behavior change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Restores mainnet C-Chain finality. The prior stall: with 5 live validators but
MainnetParams K=21/alpha=15, BOTH finality gates (assembleCertLocked cert-alpha
AND view-change POL) required 15 distinct votes — impossible from 5 validators,
so the chain froze (safe, no fork). consensus v1.35.33 sizes both gates from the
live validator count via effectiveCommittee (alpha=4 for n=5), inheriting the
minBFTCommittee K=4/alpha=3 floor (1085013 self-finality guard preserved);
Snowman K=21 sample untouched. node chains/quorum.go adds
validatorStakeSource.ValidatorCount (height-indexed, deterministic). EVM stays
v1.104.7 (head-state pin). Test: 5-validator MainnetParams control freezes,
fix converges in 0.35s.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The recurring freeze: a validator that fell behind could never rejoin, spamming
"sentTo=0" and "failed to fetch preferred block" until bounced. Two node-layer
defects, both BFT-safe (compared against ava avalanchego proposervm):
#1 proposervm.SetPreference (vms/proposervm/vm.go): assigned vm.preferred BEFORE
fetching, so a build-tip the node doesn't hold POISONED vm.preferred forever
(BuildBlock errors on every attempt; Quasar cert-finality has no re-converge
poll). Now: getBlock (both post-fork + inner stores) FIRST, delegate to inner,
adopt the preference only on success; on a total miss keep the last held tip
and stay live. Identical to ava in every case its single-store invariant
produces; only adds recovery for Lux's build-tip steering (never bricks).
SetPreference is not an acceptance gate, so safety/agreement is untouched.
#3 peer selection (chains/manager.go): consensus signals "fetch a certified-but-
untracked block" via ids.EmptyNodeID; the old code Add(EmptyNodeID)+Send, so
GetAncestors went to ZERO peers and the cert-verified gap was never fetched.
Now: when nodeID is Empty, sample real connected peers that track this chain's
network (same selection pollFrontierOnce uses); the cert already gated the ask.
Adds vm_rejoin_wedge_test.go. (Defect #2 — the engine build-tip steering in
consensus/engine/chain — is the upstream root cause, tracked separately.)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Harden the bootstrap checkpoint — the one path that bypasses the beacon quorum —
into a cryptographically SIGNED weak-subjectivity anchor.
- Checkpoint carries an authority Signature over a domain-separated canonical
(id,height); CheckpointVerifier (injected, backed by a proven primitive —
Ed25519/BLS, never custom crypto) authenticates it. The policy stays crypto-free
exactly like AncestrySource/heightOf.
- AcceptsFrontier trusts a pinned checkpoint ONLY when the configured authority
signed that exact (id,height). Present-but-unsigned, signed-by-a-non-authority,
signature-transplanted-to-another-(id,height), and no-verifier-wired all FAIL
CLOSED. A compromised flag cannot inject a false sync anchor without also forging
the authority's signature. It is a cryptographic vouch, NEVER a ⅔-live-stake tally.
- No production path pins a checkpoint yet (opt-in operator escape hatch), so the
invariant is enforced for when it is activated; nothing to wire now.
Folds consensus v1.35.16 (single ⅔ formula + leaf-lock doctrine) into the node.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Make the bootstrap-trust vs live-finality boundary a HARD API split so neither
can impersonate the other — the mass-recovery deadlock was bootstrap reusing the
finality quorum (>2/3 of CURRENT stake), which is unsatisfiable when the
recovery targets are themselves down validators.
- FinalityQuorum.HasFinality(weight,total): the live rule, strict >2/3 of current
stake (unchanged). Renamed from ConsensusQuorum to the reviewer's exact spec.
- BootstrapTrust.AcceptsFrontier(replies): selects a weak-subjective sync anchor
from AUTHENTICATED CONFIGURED beacons — a response FLOOR (MinResponses) plus a
supermajority over the RESPONDERS, NEVER over the whole set. Distinct type, so
bootstrap cannot call the finality predicate (INVARIANT 3: acceptance != finality;
the node re-executes every synced block before re-entering consensus).
- Contrast proven: 3-of-5 bootstrap ACCEPTS a frontier, yet HasFinality(3/5) is
false and STAYS false after sync — bootstrap is not a finality bypass.
Folds consensus v1.35.15 (the global unlock-before-call-out invariant + the
n=1..10 quorum acceptance matrix) into the node binary.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Adds the per-deployment opt-in flag: when LUX_CONSENSUS_VIEW_CHANGE=true and K>1, sets
consensusParams.ViewChange=true so devnet/testnet activate the round-scoped view-change
without a mainnet default (mainnet owner-gated). The engine fail-secure HALTS if the committee
fails 2a-n>f, so enabling never weakens safety. Bumps consensus v1.35.6->v1.35.7 (removes dead
safeConfig dup). This is what makes v1.32.13's prevote transport actually activatable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Adds the p2p transport for the consensus round-scoped view-change:
- quorumKindPrevote (kind 3) in the quorum-gossip envelope + decodeQuorumGossip accepts it.
- blockHandler.Gossip demux routes kind 3 -> engine.HandleIncomingPrevote(payload) (the engine
decodes+verifies height/round/canonical/sig from the payload, domain LUX/chain/prevote/v1,
and tallies toward a POL).
- networkGossiper.BroadcastPrevote(chainID,networkID,height,round,canonical,voteBytes) frames
kind 3 + broadcasts to ALL validators, mirroring BroadcastVote.
Type-checks against the round-scoped consensus (local replace, chains pkg builds clean).
Requires: consensus tag (with ViewChange + HandleIncomingPrevote + BroadcastPrevote) -> node
go.mod bump -> ARC build -> node v1.32.13 (also carries v1.32.12 evm v1.101.3 descent fix +
isBootstrapped truth fix). ViewChange is opt-in per network (default off); enable on devnet/
testnet only, NO mainnet.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Definitive probe: does the descent's served ancestry batch contain the requested
tip (want), and what are the parsed ids vs the serve-side ids — pins whether the
tip survives the Bytes()->ParseBlock round-trip (proposervm wrapping).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
INFO-level logging on both sides of the descent so we can isolate send-vs-serve:
- SEND: sampleAncestorBeacons (weights/connected/ahead/sample), Ancestors SENT
(sampleSize/sentTo), GOT batch / TIMEOUT / msgBuild-fail.
- SERVE: GetContext RECV, walk-miss (which block + i), SERVED 0 (firstFound),
SERVING (containers).
- ROUTING: deliverBootstrapAncestors (dataLen/decodedBlocks/chFound/delivered).
To be reverted once the root cause is pinned. Devnet diagnostic build only.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Documentation-only; no behavior change (v1.32.11 image unaffected). Red review of
the v1.32.11 diff: 0 critical/high/medium, recommendation SHIP.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two node/bootstrap-layer fixes for a clean rolling validator upgrade. Consensus
engine untouched (v1.35.5 boot-seed from v1.32.10 carried forward).
[HIGH — correctness] info.isBootstrapped(C) tracks REAL state, not premature true.
manager.IsBootstrapped(id) returned true the instant a chain merely EXISTED in
m.chains (set right after createChain launched the async, possibly-stalling
bootstrap goroutine) — so a C-Chain stalled at genesis (head 0x0) reported
info.isBootstrapped(C)=true, masking the stall from any wait-for-healthy gate.
Now keys on the SAME sb.Bootstrapped signal the readiness health check uses
(m.Nets.IsChainBootstrapped), set only after runInitialSync reaches the named
frontier and the VM goes to normal operation (head advanced, eth-RPC live).
GetChains().Bootstrapped fixed identically. New per-chain query on nets.Net +
chains.Nets (nil-safe). Regression test TestIsBootstrappedTracksRealConvergence.
[MEDIUM — robustness] WIPE-path GetAncestors fetch hardened for ≥2 peers at genesis.
Applying the proven avalanchego contract (studied in snowman/bootstrap +
getter): (a) Ancestors buffers the whole sample and SKIPS empty batches,
returning the first NON-EMPTY one — a size-1 channel let a fast empty reply from
a genesis peer win the race and starve a peer that actually held the ancestry;
(b) sampleAncestorBeacons PREFERS beacons the frontier round found genuinely
ahead (they hold the ancestry) — ava's PeerTracker "ask a prover" bias sourced
from the replies we already have, no separate tracker. Tests:
TestNodeBootstrap_WipePath_MixedGenesisPeers_ObtainsAncestry,
TestSampleAncestorBeacons_PrefersAheadBeacons.
Build: main+chains+nets+service/info compile CGO_ENABLED=0 (ARC/Dockerfile path);
full chains+nets suites green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Drop the Avalanche-heritage /ext prefix; /v1 is the single canonical route
surface (one way, no backward compat). The node's baseURL is the source of
truth; clients, SDKs, CLI, indexer, maker, genesis, netrunner, and the
k8s/compose/gateway/explorer configs are updated to match.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Pick up consensus v1.33.3 (durable + epoch-keyed + funnel-pruned per-height
vote-once guard) and ACTIVATE its durability in production: buildChain now opens a
per-chain VoteGuardStore (chain.OpenVoteGuard at <chainDataDir>/vote-guard) for
every K>1 signing chain and passes it to the engine via NetworkConfig.VoteGuard.
This closes Red's HIGH-1 on the wire that matters: a signing validator's
(height,epoch)→canonical bindings are fsync'd before it votes and reloaded on
startup, so a crash between casting a vote and finalizing the height cannot forget
the binding and let the restarted node sign a conflicting sibling — the cross-node
fork with zero Byzantine intent under a rolling upgrade / eviction / OOM. The store
is opened at the node boundary so the fail-closed-on-corrupt decision (a signer
refusing to start with equivocation memory it cannot trust) lives here, not in the
engine. luxd builds against pinned v1.33.3.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
RED CRITICAL: the frontier caught-up shortcut gated on CONNECTIVITY
(fullyConnectedBeacons) while the tally keyed on REPLIES (CaughtUp over
withSelfVote). The two diverge for a beacon that is CONNECTED but SILENT
(TCP up, frontier reply delayed/withheld) — a capability strictly weaker
than a full eclipse, and one that occurs naturally when an ahead beacon
replays state on a mass co-restart. Such a beacon counts as fully
connected yet is absent from the tally, so self's heavy weight backfills
the naming floor and the node goes live at a forged STALE height.
Add repliedCovers() set-containment guard: every connected beacon must
have answered THIS frontier round before the self-vote shortcut can fire.
A silent ahead-beacon now blocks caught-up -> node fails safe to
FrontierConnecting (keeps waiting); a genuine fresh net (every connected
beacon answers genesis) still completes.
Keep RED's zz_red_probe_test.go as a permanent regression guard:
connected-but-silent ahead beacon -> status=FrontierConnecting (was
FrontierCaughtUp). Fresh-net + equal-stake self-vote tests still pass.
On a FRESH net a validator whose own stake makes the PEER-ONLY responder weight
fall below the stake-majority NAMING floor (a heavy validator, a small beacon set
like the P-chain's CustomBeacons, or skewed stake: peers = total-self, so
peers < total/2+1 ⟺ self > total/2-1) could NEVER name a frontier even with the
WHOLE validator set connected — FrontierTip returned FrontierConnecting forever
("bootstrap waiting for beacon connectivity before naming the frontier") and the
node hung, blocking uniform block production.
The node is itself a beacon and knows its OWN accepted tip with certainty. Add the
SELF-VOTE: in the below-floor case, when the node has reached its ENTIRE beacon set
(fullyConnectedBeacons — no eclipse can hide an ahead-tip) and that full set PLUS
itself unanimously hold a tip it has ALREADY ACCEPTED with nobody ahead, return the
new chainbootstrap.FrontierCaughtUp -> the loop completes at the node's own tip.
Safety (proven by tests): self is counted ONLY under FULL connectivity, so an
eclipse suppressing any beacon breaks full-set and falls back to the
partition-capture floor (fail safe) — self can never tip a PARTIAL view into a
false caught-up (stale-go-live guard holds). AcceptsFrontier still tallies PEERS
ONLY, so a behind node's ahead-frontier is named undiluted and it still syncs; a
peer above our height defeats CaughtUp. self only vouches for its OWN accepted tip
so C1 (forged-frontier naming) is untouched.
- selfNodeID threaded from m.NodeID into the blockHandler.
- helpers fullyConnectedBeacons + withSelfVote.
- bump luxfi/consensus v1.29.0 -> v1.29.1 (FrontierCaughtUp status).
Tests: TestNodeBootstrap_FreshNet_SelfVoteUnderFullConnectivity_CaughtUp,
TestNodeBootstrap_SelfVote_PeerAheadDefeatsCaughtUp,
TestNodeBootstrap_SelfVote_PartialConnectivityFailsSafe. Full chains suite green.
Completes the multi-validator block-production fix on top of the v1.30.96
proposervm wrap. Three subtle correctness gaps that left single-proposer
NOT actually holding:
CRITICAL-1 (pre-fork→post-fork transition): the FIRST post-fork block (which a
fresh-genesis chain like devnet hits immediately) was forwarded to the inner VM
and built by EVERY validator unconditionally — forking the chain at its very
start. timeToBuildPreForkTransitionLocked now windows the transition block the
same way post-fork blocks are windowed (MinDelayForProposer), so a non-leader
waits its slot and adopts the elected leader's gossiped transition block; a down
leader can't stall (the eligible set widens as the slot advances).
CRITICAL-3 (validator-set ID): the windower hardcoded PrimaryNetworkID, so a
sovereign L1 (Zoo/Hanzo/Pars, validators under their own networkID==chainID) got
an EMPTY set → ErrAnyoneCanPropose → equivocation exactly like the unfixed
C-Chain, AND diverged from the cert's set. config.NetworkID (resolved once in
manager.go: primary for native, else the L1's chainID, fallback to primary if
empty) now binds the windower to the SAME set the cert uses. Zero value
(ids.Empty) IS PrimaryNetworkID, so native chains are byte-identical. newWindower
encapsulates this.
proposervm pkg + full luxd binary build clean; go vet clean (chains, proposervm).
The prior commit (9d9d2a8785) captured only manager.go, which CALLS
shouldWrapInProposerVM but did not include its definition, so HEAD did not
compile (chains/manager.go:1205: undefined: shouldWrapInProposerVM). This adds:
- chains/quorum.go: shouldWrapInProposerVM(k, chainID, innerIsDAGNative) —
the single pure policy gate (K>1 AND not P-Chain AND not DAG-native) that
decides whether a linear chain.ChainVM is wrapped in proposervm for
single-proposer-per-height block production.
- chains/proposervm_wrap_test.go: table tests pinning the gate (C-Chain
devnet/mainnet wrapped; P-Chain, X-Chain, K==1 excluded) and that the K
flows from selectConsensusParams.
- chains/manager.go: gofmt import ordering.
Restores a green build. No push, no tag.
The deeper half of the block-production fix. Lux had commented out the
proposervm wrapper and hand-rolled a WaitForEvent→toEngine bridge that dropped
the single-proposer invariant: every validator's engine called BuildBlock
UNCONDITIONALLY at every height off a slightly-different mempool, so two nodes
could each gather an α-of-K cert for DIFFERENT blocks at the same height →
reportCertEquivocation → Logger.Crit → crash (and the DEX-tx nondeterminism
wedge). That, together with the evm builder-ready race (evm v1.99.51), is why
every multi-validator C-Chain froze.
Re-wrap multi-validator linear chains in proposervm so block production follows
the Snowman++ proposer schedule — exactly ONE validator builds height H, the
rest wait and vote. proposervm.WaitForEvent only returns PendingTxs inside THIS
node's proposer window, collapsing the race at its source.
Gated (all three): K>1 (single-proposer only matters for a quorum), not the
P-Chain (its height-indexed validators.State is published during its own
createChain, after the snapshot used here), not DAG-native (X-Chain's Linearize
bridge doesn't compose with proposervm's pull/window model). Engine, WaitForEvent
bridge, block handler, SetState, and HTTP registration all route through the
wrapped engineVM; unwrapped chains keep the original path byte-for-byte.
chains pkg builds + go vet clean. Validated end-to-end on devnet (5 validators).
A validator holding gossiped-but-unaccepted blocks (in the VM store, lastAccepted
below them) concluded caught-up at its stale height forever — the caught-up check
conflated store presence (Has/heldHeight via vm.GetBlock) with acceptance.
Fix (consensus v1.25.36): Has→Accepted (a stored-but-unaccepted named frontier
descends+ACCEPTS through the cert-gated per-height guard); recognize convergence
off the in-process consensus finalized ledger, not the frozen VM.LastAccepted
cache; un-freeze the zap client's lastAcceptedID on Accept. Proven on mainnet
luxd-2 (1082780→1082796). C1/VerifyWeighted/cert-gate intact. blue→red→blue→red.
The v1.30.84 bootstrap-vm-ready ordering fix passed 50 unit tests + the BFT
proof but FAILED the real mainnet canary: a STALE spare (luxd-2) at C-Chain
accepted height 1082780 went Ready at 1082780 instead of fetching the 16 blocks
to the producers' frontier 1082796, then sat frozen. The mocks injected the
FULL, STABLE validator set from t=0 — masking the production trigger.
ROOT CAUSE (primary). A native chain's runBootstrapThenPoll starts right after
its VM.Initialize (dispatchChainCreator), NOT after the P-chain has finished
its own initial sync. The P-chain populates m.Validators as it replays blocks
(genesis stakers + every AddValidatorTx), so FrontierTip can run while
GetMap(PrimaryNetworkID) is a PARTIAL set. The MinResponseWeight stake-majority
floor (total/2+1) is fail-secure ONLY when `total` is the TRUE full-set stake;
under a partial denominator a degenerate at-or-below responder subset (the
genuinely-ahead producers not yet loaded) clears the floor and the exact-fast-
path / CaughtUp concludes "caught up" at the stale local height. Proven by
elimination: with the full set EVERY accurate-reply path is fail-safe (wait /
sync), so the freeze requires a non-representative responder set.
FIX (primary): gate a native non-platform chain's frontier-trust on the P-chain
having COMPLETED initial sync (m.pChainBootstrapped, published by
monitorBootstrap; read via blockHandler.primaryNetworkReady). Until then
FrontierTip returns FrontierConnecting (WAIT), so every branch judges the TRUE
full validator set and the existing floor logic holds. Deadlock-free: the
P-chain converges independently from its own configured CustomBeacons.
FIX (self-heal / the "ALSO FIX"): pollFrontierOnce filtered peers on b.chainID —
the IDENTICAL chainID/networkID confusion already fixed in connectedBeacons.
Peers advertise the NET they track, never an individual native chain id, so the
sample was always empty and a behind-but-Ready validator never discovered it was
behind. Filter on b.networkID so the NormalOp catch-up poller actually fires.
INSTRUMENTATION: FrontierTip now logs (Debug) the full beacon-set size + total
stake (the floor DENOMINATOR — a partial value is the smoking gun), connected
count, and every reply's tip + locally-resolved height + held-or-not, plus the
decision. Set the C-chain log level to Debug on the canary to capture ground
truth.
TESTS (production-modeling, not the ideal full set): TestRED_PartialStakedSet_
BehindNodeMustNotGoLiveStale reproduces the freeze (without the gate it NAMES the
stale own tip; with it, WAITS; then converges on the full set) — proven
load-bearing (disabling the gate fails the test). Plus not-held-tips → sync,
empty-ancestry → stays Bootstrapping (never Ready stale), and peer-connect-delay
→ wait-then-converge. Full chains suite green (C1, co-restart, M1,
mass-recovery, skewed-weight all preserved), race-clean.
nameFrontier's ancestor-tolerant filter was `ref.Height < MinFrontierHeight`
(MinFrontierHeight = the node's own last-accepted), so a block AT the node's
own height passed the filter and could be NAMED. Red's M1 eclipse: with the
genuinely-ahead responders throttled below the ⅔ naming threshold (R_a < ⅔R)
but the at-height responders let through, the node's OWN height accrues ⅔
purely as the shared ANCESTOR of the ahead N+5 tips → nameFrontier names it →
FrontierNamed at own height → the node goes Ready STALE (5 blocks behind a
finalized N+5). nameFrontier ran BEFORE CaughtUp in FrontierTip, so it
short-circuited the stricter CaughtUp determination that already refuses this.
FIX (one operator): `ref.Height <= MinFrontierHeight` — own height is excluded
from naming, routing the at-own-height case to CaughtUp, which alone
distinguishes a legit all-at-N fleet (Ready at N) from an eclipse with ahead
tips the node lacks (refuse → sync). The two paths compose with no gap:
nameFrontier names STRICTLY ABOVE own height (behind nodes); CaughtUp decides
at-or-below own height (tip-holders); the fast path (active ⅔-direct report)
stays exempt for the legit unanimous-at-tip case.
Safe for layer-5 + C1: a behind node names HIGHER than its own height (survives
both the old `<` and new `<=`); the ⅔-of-responder-weight naming requirement is
untouched (only height eligibility tightens — strictly safer). Round-1
behind-node convergence is unchanged.
Faithful test correction (manager.go: GetAcceptedFrontier answers vm.LastAccepted
— the last-ACCEPTED block, NEVER an un-finalized preferred tip): the two
ancestor-tolerant convergence integration tests modeled a beacon reporting an
un-finalized tip FOREVER (a gap-1 instance of this very M1 stale-go-live). They
are restated to model finalization PROPAGATION — the node names the ⅔-common as
a behind node, then RATCHETS to the true finalized tip as the laggards' accepted
frontier advances — converging to the real frontier (stronger guarantee), never
falsely caught-up below it.
Tests (RED-before by reverting the one-operator hunk, GREEN-after):
- TestBootstrapTrust_EclipseOwnHeightNotNamedRoutesToCaughtUp (policy, deterministic
+ own-height boundary)
- TestRED_EclipseOwnHeightNotNamedRoutesToSync (integration: FrontierNoQuorum +
full-loop fails safe at stale N + eclipse-lifts→syncs-up)
GOWORK=off CGO_ENABLED=0 go test ./chains/ — all green (layer-5 convergence, C1
forged/eclipse/partition, caught-up co-restart, fast-path, self-heal).
RED found a CRITICAL liveness regression in the VM-ready gate (b313912413): a
tip-holding producer on a mixed-height co-restart cannot satisfy its own go-live
gate and fails safe DOWN at its own tip — the OPPOSITE of the stale-go-live bug the
gate fixed, and just as wrong.
Mechanism (PoC reproduced): a producer at N sees its 4 peers split
{2 producers@N, luxd-2@N-16, luxd-0@genesis}. The tip-holders are only ½ of the
responders (< ⅔) so no frontier is NAMED; the ⅔-backed common ancestor N-16 is
below the node's own last-accepted (MinFrontierHeight filters it). → ErrNoBeaconQuorum
→ runInitialSync false → C-Chain STOPPED. The loop's Has(tip)→caughtUp shortcut is
only reached AFTER FrontierNamed, which a tip-holder never gets.
PRIMARY FIX — a CAUGHT-UP determination (BootstrapPolicy.CaughtUp), the dual of
AcceptsFrontier ("nobody is ahead" vs "here is the block ahead"). On the
ErrNoBootstrapQuorum branch, FrontierTip concludes caught-up and names the node's
OWN held tip (→ the loop's existing Has()-shortcut → Ready at own height) iff:
(a) the SAME response floor AcceptsFrontier uses is met (MinResponses count AND
MinResponseWeight stake-majority) — an eclipse hiding the ahead-nodes drops
below the floor → fail safe, no partition-capture;
(b) every responder's reported ACCEPTED tip is at height ≤ lastAccepted — a stale
node has an honest responder ahead → still syncs (stale-go-live stays fixed);
(c) the node HOLDS every reported tip (heightOf injected, VM-free) — never declares
caught-up to a sibling/fork it lacks.
floorMet + tallyResponders are extracted so naming and caught-up share ONE floor and
ONE eligibility rule (DRY). No consensus change — reuses FrontierNamed + Has, builds
GOWORK=off against the deployed consensus v1.25.35.
SELF-HEAL (RED MEDIUM #2) — the K8s probes all poll the always-green
/ext/health/liveness, so a fail-safe-DOWN node is NEVER restarted (a permanent brick).
runInitialSync now RE-ATTEMPTS a transient connectivity fail-safe (bootstrapMaxAttempts
≤ 0 ⇒ until the quorum returns or shutdown), staying in Bootstrapping (never live at
stale height) and converging the instant the quorum returns. bootstrapConnecting tells
monitorBootstrap's no-progress watchdog this is a deliberate WAIT (not a stall) so it
does not force-STOP a node correctly waiting for its quorum. Structural failures (deep
gap → state-sync) are NOT retried.
LOW — FinishBootstrap now runs BEFORE transitionVMReady (close the no-cert
AcceptBootstrapBlock gate before the VM can build); transitionVMReady's SetState timeout
is derived from the shutdown ctx (was context.Background()).
MEDIUM — createDAG (X/Q) go-live left UNCONDITIONAL, now documented as a DELIBERATE
linear-only scope (mainnet froze on linear C/D/B/T/Z; DAG bootstrap convergence is a
separate task). createDAG does not regress.
C1, VerifyWeighted, the ⅔-stake cert, and "no VerifiedQuorumCert no finality" are
untouched. Tests (chains): TestRED_TipHolderCoRestartGoesReadyAtOwnTip reproduces the
regression — RED before on b313912413 (FrontierNoQuorum, live==false), GREEN after
(FrontierNamed at own tip, vmReady once at N). Plus 6 CaughtUp unit tests (incl. both
adversarial fake-caught-up cases), TestRED_MajorityOutageSelfHealsWhenQuorumReturns,
and TestWatchBootstrapProgress_ConnectingNotStalled. Full C1/forged/eclipse/partition
suite re-run green (41 bootstrap tests, 0 fail).
Root cause of the restart-freeze (mainnet luxd-2: C-Chain stuck at 1082780
while producers finalized 1082796): buildChain called SetState(vm.Ready)
UNCONDITIONALLY right after Initialize — at the LOCAL last-accepted height —
and ran the node-layer initial sync (runBootstrapThenPoll) afterward as a
detached, non-gating goroutine. vm.Ready fires the EVM's
onNormalOperationsStarted (block building, mempool gossip, validator dispatch);
a restarted STALE validator therefore went live at its stale height and never
converged to the finalized frontier. The proper vm.Bootstrapping state (the
EVM's onBootstrapStarted: snapshots only, no building) was skipped entirely.
Fix — restore the correct VM lifecycle, gated:
- buildChain now SetState(vm.Bootstrapping) after init (not Ready). The VM
fetch+executes the gap in Bootstrapping, serving nothing as head.
- The Ready transition moves into the bootstrap goroutine (runInitialSync,
split out of runBootstrapThenPoll for unit-testability) and fires ONLY after
bs.Run reaches the named ⅔-by-stake beacon frontier — VM live + engine
cert-gate (FinishBootstrap) go together at the frontier, never at a stale
height. Wired via blockHandler.vmReady (captures the VM's SetState).
- Fresh-genesis / single-node (FrontierNoBeacons) and at-the-tip restart
(caught-up) go Ready promptly — no regression, no hang.
- Eclipse / isolation: bs.Run's bounded ConnectDeadline is the retry that
self-heals when beacons return; past it the node fails SAFE (stays
Bootstrapping, records the reason for monitorBootstrap) — never serves stale
state, never an unbounded hang. The orchestrator restart now converges.
C1 forged-chain defense, VerifyWeighted, and the ⅔-stake quorum cert are
untouched — the frontier is still named only by the configured-beacon
⅔-of-responders quorum + content-addressed descent (all RED/BootstrapTrust
tests pass). AcceptedFrontier/Ancestors replies route to the bootstrap channels
via bsActive throughout the phase (verified).
Tests (chains): TestRED_StaleValidatorGoesReadyOnlyAfterReachingFrontier
reproduces the production scenario — RED before (VM goes live at stale N=30),
GREEN after (only at frontier N+K=46). Plus fresh-genesis no-regression,
bounded eclipse fail-safe, and at-the-tip producer fast-Ready. Race-clean.
NOTE (flagged for review, NOT in this change): the createDAG path (X/Q DAG
chains, consensusdag.Engine) has the same unconditional SetState(Ready) at
manager.go:1792 but a different engine without the node-layer bootstrap-sync
wiring — a separate fix if any production chain uses it.
Mutation-proven gap: H/D2 construct policies directly, so nothing caught the
production constructor silently dropping the stake-majority floor. This asserts
bootstrapPolicy() emits MinResponseWeight=⌈total/2⌉ (skewed), the count floor,
the AncestrySource, no-re-deadlock for equal-weight 3-of-5, and disabled-on-empty.
Re-red verdict was SHIP; this closes the only (LOW) residual.
The MinResponses COUNT floor and the ⅔-of-responders WEIGHT agreement diverge under
SKEWED validator stake: an attacker eclipsing the HEAVY honest beacons while passing
enough LIGHT honest ones to clear the count can shrink the responder-weight denominator
until <⅓-of-total Byzantine stake names a forged frontier (re-red PoC: w={3,3,13,1,1,1},
Byz 27%). Fix: bootstrapPolicy() now sets MinResponseWeight=⌈total/2⌉ (the field was
wired into AcceptsFrontier but left 0). Proven safe AND deadlock-free: equal-weight
recovery needs only >½ (3/5=0.6≥0.5), not the >⅔ that caused the original deadlock.
Closes the re-red HIGH + INFO-2 (same root cause). Tests H (skewed→reject, load-bearing)
+ D2 (non-configured swarm→nothing). Required for non-equal-weight validator sets
(validators-come-and-go). C1/A-G/finality all green.
THE DEADLOCK (mainnet, node v1.30.79): 5 equal-weight validators (each
0.5e18, total 2.5e18, ⅔ floor 1.667e18). The 2 stranded recovery targets
ARE 2 of the 5; with them down only 3 producers connect = 1.5e18 = 60% < ⅔.
The prior bootstrap frontier quorum required ⅔ of CURRENT TOTAL stake to be
CONNECTED before naming a sync frontier — mathematically unsatisfiable during
a mass outage when the down nodes are themselves validators — so no node could
ever recover. Bootstrap trust was braided into consensus finality, and
finality's ⅔ rule cannot be met when ⅓+ of the set is the thing recovering.
THE FIX — a SEPARATE type with a SEPARATE threat model, not a renamed
threshold (chains/bootstrap_trust.go):
- ConsensusQuorum.HasFinality(weight,total): > ⅔ CURRENT stake — UNCHANGED;
present only to NAME the live rule and CONTRAST it.
- BootstrapTrust.AcceptsFrontier(ctx,replies): selects a weak-subjective sync
anchor from authenticated CONFIGURED beacons. NOT a finality oracle.
- BootstrapPolicy: TrustedBeacons (the configured/checkpoint/genesis anchor —
never peer self-report), AgreementThreshold (⅔ of RESPONDERS), MinResponses
(a response FLOOR), MinResponseWeight, Checkpoint (operator override).
Three invariants:
1. NON-CIRCULAR eligibility — only NodeIDs in TrustedBeacons count; peers
never define who is a beacon.
2. A response FLOOR prevents partition-capture — MinResponses (default: a
MAJORITY of the configured set) authenticated beacons must respond; below
it, REJECT unless an operator checkpoint is pinned. For the 5-validator
mainnet (MinResponses=3) the 3 reachable producers recover while a
2-beacon partition is rejected.
3. Acceptance ≠ finality — the named Frontier is "safe to begin sync from",
re-executed during descent; live acceptance stays governed by
ConsensusQuorum alone.
The acceptance gate (FrontierTip) now routes through BootstrapPolicy: the
⅔-of-CURRENT-TOTAL connect gate is GONE; agreement is ⅔ of the RESPONDERS over
a MinResponses floor. The ancestor-tolerant common-ancestor tally is reused
(now a global cross-anchor union tally so a sibling split converges to the
shared committed block — case F), bounded below by MinFrontierHeight (the
node's last-accepted height) so a fork sharing only history BELOW the node
fails safe instead of false-completing at the deep ancestor. Consensus
(v1.25.35) is UNTOUCHED — the FrontierStatus interface is unchanged and its
engine/chain suite stays green.
Tests (chains green; consensus engine/chain green): A mass-recovery success,
B one-beacon capture rejected, C two-beacon partition rejected, D
non-configured peer ignored, E minority-configured forgery rejected (forger
ratifies the real block, never names its forgery), F split reachable ancestry
selects the common ancestor, G finality unchanged (3-of-5 accepts for
bootstrap but is NOT HasFinality). Plus checkpoint override and a load-bearing
fork-at-shared-genesis fail-safe guard for the global tally. The reframed
SubAlpha→SubQuorum test replaces the ⅔-of-total assertion that WAS the bug.
Each guard proven load-bearing by revert: restore the ⅔-of-total gate and A
deadlocks; drop the configured-beacon filter and D captures; drop
MinFrontierHeight and the shared-genesis fork false-completes.
Branch only; not deployed. Re-canary luxd-2.
The exact-tip frontier quorum required ⅔-by-stake of beacons to report the
IDENTICAL accepted tip. On a live chain validators are legitimately ±1 block
apart at the bleeding edge, so the freshest tip splits and no single id draws
⅔. The mainnet canary (luxd-2): 2 producers had accepted block N, the third
producer's un-finalized PENDING block was N+1; neither tip alone cleared ⅔, so
FrontierTip returned NoQuorum forever and the healthy stale node failed safe at
its stale height instead of converging. The chain was idle, so the split was
STABLE and never resolved.
FIX (node-only — the stake quorum lives where the stake lives; consensus stays
stake-agnostic and just descends from whatever block is named): make the tally
ANCESTOR-TOLERANT. A beacon reporting tip N+1 also has N in its accepted chain,
so the named frontier is the HIGHEST block a ⅔-by-stake supermajority SHARES
(B == their tip OR B is an ancestor of their tip), not the highest proposal. N
draws all three producers (300 > 200) and is named; N+1 draws only the lone
producer and is not. The node syncs to N (the ⅔-agreed committed height); live
consensus catch-up handles N+1.
Ancestry is learned by CONTENT (reusing the parent-link descent the sync loop
trusts): for each candidate tip, fetch its ancestry, rebuild the parent-linked
chain, and cumulatively tally — the highest block clearing the floor across
anchors is named. A credit is truthful because a block's parent id is fixed by
its content, so a peer cannot fake the linkage below an honestly-reported tip.
C1 preserved EXACTLY: a block is named only with > ⅔ of the TOTAL real
staked-beacon weight and ≥ required distinct voters; a forged or sub-⅔ tip on a
side-fork shares only the honest prefix and can never raise the named block above
the genuine ⅔-common height. Only the agreement RELATION changed (exact-tip →
in-the-accepted-chain), never the ⅔-by-stake-of-the-real-set requirement. The
content-addressed descent, per-height guard, re-execution, weak-subjectivity
anchor, F2 bounded-NoQuorum retry, deep-gap fail-safe and single-node completion
are all untouched (consensus engine/chain/... unchanged + green).
Also: resolve the quorum the instant ALL connected beacons have answered (no
exact ⅔), instead of waiting out the full frontier window — so a live-frontier
split converges in one round instead of stalling a window each pass.
Tests (chains green; consensus engine/chain/... green): TestRED_TipSplitConverges-
ToCommonHeight (the canary — converges to N, not the minority N+1, not fail-safe;
proven load-bearing: 2-of-3 = 200 is NOT > the 200 floor, so the exact tally
cannot name N), TestRED_ForgedHighTipFromMinorityNotNamed (a Byzantine beacon's
forged block built ON real N ratifies only N, never names itself — C1),
TestNodeBootstrap_ExactQuorumUsesFastPath (unanimous ⅔ still names the exact tip
with NO fetch), TestNodeBootstrap_BeaconsSplitNoQuorum (disjoint 3/3 partition,
ancestry served, STILL NoQuorum — ancestor tolerance does not manufacture a
quorum). NOT deployed — re-canary luxd-2.
THE CANARY (luxd-2, node v1.30.77): a healthy STALE node (C-Chain 1082780,
gap-17 to producers at 1082797) rolled to the convergence fix logged
"bootstrap waiting for beacon connectivity..." x10 then failed SAFE at the
connect deadline — "beacon set never reached the connectivity needed to form a
⅔ quorum ... eclipsed/partitioned" — despite connectedPeers=9 and the
producers being right there. Correct fail-safe, but it did NOT converge.
DIAGNOSIS: the beacon set was ALREADY the stake-weighted validator set, not
the --bootstrap-nodes endpoints. For a native chain (C-Chain) manager.go wires
beacons = m.Validators (n.vdrs, populated by the bootstrapped P-chain's
state.initValidatorSets under PrimaryNetworkID); the --bootstrap-nodes endpoint
beacons are a SEPARATE weight-1 set used only as the P-chain's CustomBeacons.
FrontierConnecting (not FrontierNoBeacons) confirms m.Validators was non-empty.
The real bug was the CONNECTIVITY filter: connectedBeacons admitted a connected
beacon only if p.TrackedChains.Contains(b.chainID). But a peer advertises the
NETS it tracks (network/peer/peer.go adds constants.PrimaryNetworkID + every
tracked L1 net ID to peer.Info.TrackedChains) — it NEVER advertises an
individual native chain ID. So the filter matched ZERO real peers,
connectedStake stayed 0, the ⅔ floor was never reachable, and FrontierTip
reported FrontierConnecting until the deadline. The tests passed only because
the mock modeled TrackedChains as set.Of(chainID) — agreeing with the bug, not
reality.
FIX (node-only; consensus FrontierStatus logic unchanged and correct):
- connectedBeacons filters on b.networkID (the NET the beacon set is anchored
to — PrimaryNetworkID for native chains, the L1 net ID for sovereign chains),
the value real peers actually advertise. C1 PRESERVED: a peer is still
admitted ONLY if it is in the staked set (weights); the ⅔-by-stake threshold,
the TOTAL-stake denominator, and the content-addressed descent are untouched.
A non-staked peer that tracks the network is still ignored.
- EDGE (4): expectsStakedBeacons (true for a native non-platform chain on a
sybil-protected network) makes an EMPTY staked set report FrontierConnecting
(wait → ConnectDeadline → fail safe), never FrontierNoBeacons (false-complete)
and never endpoint trust. Distinguishes "validator set not yet loaded → wait
→ converge" from "genuinely empty → fail safe". The P-chain (CustomBeacons may
be empty under endpoint-only --bootstrap-nodes) and single-node keep
"empty ⇒ nothing to sync to".
TESTS (chains):
- TestRED_PeersTrackNetNotChain_StaleNodeConverges: 3 producers holding > ⅔
stake, connected + advertising the NET (not the chain), name the frontier and
the stale node converges. Proven load-bearing: reverting the filter to
b.chainID makes connectedBeacons return nil and this test fails exactly as the
canary did. Mock now models TrackedChains = the NET (matches peer.go).
- TestRED_EmptyStakedSetFailsSafe: empty staked set + expectsStakedBeacons →
Connecting → ErrBeaconsUnreachable, no endpoint trust; the flag is the proven
discriminator vs FrontierNoBeacons.
- C1 (ForgedFrontierFromNonBeacon / HonestBeaconQuorumIgnoresMalicious /
SubAlphaStakeCannotName), F2/NoQuorum, deep-gap, weak-subjectivity,
single-node all green; consensus engine/chain/... unchanged and green.
NOT deployed — re-canary luxd-2.
When the initial-sync loop RETURNS a fail-safe error (ErrBeaconsUnreachable /
ErrNoBeaconQuorum / ErrGapTooLarge), runBootstrapThenPoll exited leaving the chain
in a DEAD WINDOW: neither bootstrapping (the loop is gone) nor stopped
(monitorBootstrap only stops the engine at the +5-min no-progress watchdog). The
operator saw a chain that had already given up but was not marked failed for up to
5 minutes.
runBootstrapThenPoll now records the fail-safe reason (bootstrapFailed atomic +
BootstrapFailed/BootstrapFailure accessors). watchBootstrapProgress takes a failed()
predicate and returns the new bootstrapFailed outcome the next tick; monitorBootstrap
stops the engine + registers a failing health check IMMEDIATELY, carrying the real
reason (eclipsed/partitioned/too-far-behind) rather than the generic no-progress
timeout. The stop+health logic is DRY'd into one m.failBootstrapChain shared by the
stall and fail-safe outcomes. Combined with the consensus-side bounded NoQuorum retry
(F2), a transient live-frontier split converges and a genuine eclipse fails fast and
visibly — making the mainnet 5/5 roll deterministic.
Tests: TestWatchBootstrapProgress_FailSafeSurfacesPromptly (failed -> bootstrapFailed
next tick, not after the window), TestWatchBootstrapProgress_ReadyBeatsFailed (success
never masked), TestBootstrapFailure_AccessorSurfacesReason (driver<->monitor plumbing).
Existing watchdog + node bootstrap (C1 forged-frontier finalizes ZERO, stale-converge,
split-no-quorum) all still green. NOTE: build the combined tree with a temp
go.mod replace github.com/luxfi/consensus=../consensus (consensus f93162d15); not committed.
Node side of the stale-node convergence fix. blockHandler.FrontierTip now returns
a chainbootstrap.FrontierStatus instead of a bare ok bool, decomplecting the THREE
reasons a frontier may not be named so the bootstrap loop can WAIT for beacon
connectivity instead of false-completing at the local stale height:
- no beacon set configured -> FrontierNoBeacons (single-node: complete)
- beacons configured, < ⅔ stake up -> FrontierConnecting (the boot race: WAIT)
- ⅔ stake connected, no agreement -> FrontierNoQuorum (eclipse: fail safe)
- ⅔-by-stake quorum agrees on a tip -> FrontierNamed (C1 gate: sync)
The discriminator is the SAME ⅔ floor (config.TwoThirdsStakeFloor over the FULL
beacon stake) used by the quorum tally: if even unanimous CONNECTED beacon stake
could not clear the floor (or too few distinct beacons are up), connectivity is
still coming up -> FrontierConnecting (the canary boot race, where FrontierTip ran
before any beacon handshaked). Only once enough beacon stake is connected do we
query + tally; a split that clears none -> FrontierNoQuorum.
The C1 forged-chain gate is UNCHANGED: the query + weighted ⅔-stake tally is
extracted verbatim into queryFrontierQuorum (a non-beacon peer, a single peer, or
any sub-⅔ set can still NEVER reach FrontierNamed). connectedBeaconsTrackingChain
is replaced by the smaller connectedBeacons helper (shared with
sampleAncestorBeacons — DRY). monitorBootstrap/watchBootstrapProgress need no
change: they already gate the chain-ready signal on BootstrapComplete (Run
success), which now returns only on genuine caught-up.
Tests: reproduces the canary over the REAL GetAcceptedFrontier/GetAncestors
transport (TestNodeBootstrap_StaleNodeWaitsForBeaconConnect — beacons connect after
a delay; the stale node WAITS through the connecting passes, then names the ⅔
frontier and converges, never false-completing at the stale height); the
forged-frontier-from-non-beacon case now FAILS SAFE (ErrBeaconsUnreachable) instead
of false-completing while still finalizing ZERO forged blocks; split-beacons report
FrontierNoQuorum; a no-beacon-set node reports FrontierNoBeacons (single-node
completes). All existing C1 quorum, sub-⅔, invalid-halt, framing, deliver-gating,
and H2 watchdog tests stay green.
Dev integration: node go.mod stays pinned consensus v1.25.34 (NO committed
replace); build/test against the consensus fix branch via
`GOWORK=off go mod edit -replace github.com/luxfi/consensus=../consensus`
(the go.work path is blocked by an unrelated ./compliance go.sum mismatch). Tag
consensus + bump node go.mod at the coordinated merge.
