Documentation-only; no behavior change (v1.32.11 image unaffected). Red review of
the v1.32.11 diff: 0 critical/high/medium, recommendation SHIP.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two node/bootstrap-layer fixes for a clean rolling validator upgrade. Consensus
engine untouched (v1.35.5 boot-seed from v1.32.10 carried forward).
[HIGH — correctness] info.isBootstrapped(C) tracks REAL state, not premature true.
manager.IsBootstrapped(id) returned true the instant a chain merely EXISTED in
m.chains (set right after createChain launched the async, possibly-stalling
bootstrap goroutine) — so a C-Chain stalled at genesis (head 0x0) reported
info.isBootstrapped(C)=true, masking the stall from any wait-for-healthy gate.
Now keys on the SAME sb.Bootstrapped signal the readiness health check uses
(m.Nets.IsChainBootstrapped), set only after runInitialSync reaches the named
frontier and the VM goes to normal operation (head advanced, eth-RPC live).
GetChains().Bootstrapped fixed identically. New per-chain query on nets.Net +
chains.Nets (nil-safe). Regression test TestIsBootstrappedTracksRealConvergence.
[MEDIUM — robustness] WIPE-path GetAncestors fetch hardened for ≥2 peers at genesis.
Applying the proven avalanchego contract (studied in snowman/bootstrap +
getter): (a) Ancestors buffers the whole sample and SKIPS empty batches,
returning the first NON-EMPTY one — a size-1 channel let a fast empty reply from
a genesis peer win the race and starve a peer that actually held the ancestry;
(b) sampleAncestorBeacons PREFERS beacons the frontier round found genuinely
ahead (they hold the ancestry) — ava's PeerTracker "ask a prover" bias sourced
from the replies we already have, no separate tracker. Tests:
TestNodeBootstrap_WipePath_MixedGenesisPeers_ObtainsAncestry,
TestSampleAncestorBeacons_PrefersAheadBeacons.
Build: main+chains+nets+service/info compile CGO_ENABLED=0 (ARC/Dockerfile path);
full chains+nets suites green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
consensus v1.35.5 is a patch-only fix (go.mod byte-identical to v1.35.2..v1.35.4): the
decided-height sign-gate floor is now seeded DIRECTLY from vm.LastAccepted at engine Start
(before signing) and in SyncState, so a node upgrading IN PLACE from a legacy v1 vote-guard
file (floor 0) has a real decided floor from the first instant of boot — closing the
mainnet v1->v2 upgrade window where the floor was 0 until the first post-upgrade finalize.
Sign-gate-only (never enters byHeight/ledger.Height; PART-A intact; only refuses more).
Supersedes the un-rolled node v1.32.9 (consensus v1.35.4) for the mainnet-first roll. No
node code change (fix is internal to the consensus engine). go mod tidy drops an orphaned
luxfi/bft indirect (pre-existing). Full node builds clean on linux/amd64 (cgo off).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
RECOVERY BUILD for the frozen testnet C-Chain. Carries BOTH fresh-multi-node fixes:
- STATE halt (missing trie node) — evm v1.101.2 MaterializeState, from the v1.32.7 tree.
- CONSENSUS double-finalization + liveness stall — consensus v1.35.2 (epoch-blind
vote-once + per-height vote convergence + RED-round hardening).
Deliberately based on v1.32.7, NOT node v1.33.0: v1.33.0 is a mis-numbered DISPROVEN
build (older commit, MISSING the evm v1.101.2 MaterializeState commit, pins the buggy
consensus v1.33.3) — basing on it would regress the state-halt fix. This branch also
EXCLUDES node-main's /ext->/v1 route migration to keep the incident recovery minimal
and avoid breaking /ext/bc/C/rpc tooling mid-recovery.
Recommended tag: node v1.33.1 (forward past the v1.33.0 revert-magnet so the image
tooling cannot revert testnet to the disproven build). NOT tagged here — cutting the
tag may auto-trigger a live deploy; that is a live-incident action requiring explicit
owner authorization.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Drop the Avalanche-heritage /ext prefix; /v1 is the single canonical route
surface (one way, no backward compat). The node's baseURL is the source of
truth; clients, SDKs, CLI, indexer, maker, genesis, netrunner, and the
k8s/compose/gateway/explorer configs are updated to match.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
evm v1.101.1's accept-path guard was confirmed a no-op live (halt reproduced at
block 8 on v1.32.6). v1.101.2 rebuilds pruned parent state on demand at BuildBlock,
which is where the post-accept state loss manifests. Isolated EVM_VERSION bump on
the v1.32.4 baseline.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Isolated bump of EVM_VERSION v1.99.52 -> v1.101.1 on the v1.32.4 baseline.
v1.101.1 core/plugin code == v1.99.52 (verified byte-identical) + the accept-path
equivocation-commit-gap backstop (BlockChain.Accept re-materializes an accepted
block's state if !HasState, on the parent state). Fixes the fresh-genesis testnet/
devnet block-2 'missing trie node' halt. CHAINS_REF/DEX_REF unchanged. Devnet test
build; RED-gated before mainnet.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Pick up consensus v1.33.3 (durable + epoch-keyed + funnel-pruned per-height
vote-once guard) and ACTIVATE its durability in production: buildChain now opens a
per-chain VoteGuardStore (chain.OpenVoteGuard at <chainDataDir>/vote-guard) for
every K>1 signing chain and passes it to the engine via NetworkConfig.VoteGuard.
This closes Red's HIGH-1 on the wire that matters: a signing validator's
(height,epoch)→canonical bindings are fsync'd before it votes and reloaded on
startup, so a crash between casting a vote and finalizing the height cannot forget
the binding and let the restarted node sign a conflicting sibling — the cross-node
fork with zero Byzantine intent under a rolling upgrade / eviction / OOM. The store
is opened at the node boundary so the fail-closed-on-corrupt decision (a signer
refusing to start with equivocation memory it cannot trust) lives here, not in the
engine. luxd builds against pinned v1.33.3.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
consensus v1.33.2 closes the CRITICAL cross-node fork Red found in v1.33.1: the
finality layer now enforces per-height vote-once (an honest validator signs at most
one canonical per height, both signing sites through reserveSlotForSign), so two
conflicting siblings can no longer both gather a >=2/3-stake cert. All 3 Red fork
repros GREEN, emergent probe 15x no fork, engine/chain suite green under -race.
luxd builds clean. Ships with the proposervm liveness fix + luxd-3 ZAP block-id guard.
node v1.32.3 = v1.32.2 + the idle-builder bounded-wake fix, delivered as the
baked C-Chain EVM plugin. The node Dockerfile git-clones luxfi/evm@EVM_VERSION
and builds ./plugin into the image; v1.99.52 = v1.99.51 + startPendingTxPoll
(500ms mempool re-poll so a tx submitted to an idle demand-driven chain wakes
the builder within a bounded window — closes the lost-wakeup/subscribe-gap).
Patch bump only (x.x.x+1 on the pinned v1.99.x line, NOT a jump to v1.100.x);
cherry-picked clean onto v1.99.51 and RED-cleared. Deterministic: wake timing
only, block contents unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Adopts consensus v1.33.1: the build-path re-solicit of an undecided own
proposal (avalanche repoll alignment) plus the v1.33.0 QCv2 canonical-
commitment finality + incident-1082814 verify-before-slash. Pulls transitive
ids v1.3.0, warp v1.24.0, pulsar v1.9.0, threshold v1.12.0. luxd builds
(GOWORK=off, pinned deps). No node code changes required — consensus public
API (NewRuntime/NetworkConfig/Runtime) is stable across the bump.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two pre-existing reliability bugs surfaced by the v1.32.1 roll (both present
identically in v1.31.5 — NOT caused by QCv2), plus a snapshot-inconsistency
diagnostic.
BUG 1 (HIGH) — concurrent-map crash in getPostForkBlock. The consensus engine
drives the proposervm from multiple handler goroutines concurrently (a
PullQuery/Put verifying a block WRITES verifiedBlocks via verifyAndRecordInnerBlk
while a Qbit handler READS it via GetBlock -> getPostForkBlock; the retry loop in
PullQuery even documents "allow concurrent Put to complete"). The verifiedBlocks
map was read/written with no lock -> the Go runtime aborts with "fatal error:
concurrent map read and map write" (exit 2) under heavy verify load. Fix: a
dedicated verifiedBlocksLock (RWMutex) behind three accessors (cachedVerifiedBlock
/ recordVerifiedBlock / forgetVerifiedBlock) at ALL seven sites; the sibling
Tree.nodes map (touched on the same Verify/Accept paths) gets its own RWMutex,
with Accept restructured to make its inner-VM Accept/Reject callouts OUTSIDE the
lock. Both are leaf locks, never nested, never held across a callout — the lock
graph is a strict DAG (vm.lock > verifiedBlocksLock; tree.lock disjoint), so
deadlock-free. innerBlkCache (lru.SizedCache) is already internally locked.
Regression: TestVerifiedBlocksConcurrentAccess + TestTreeConcurrentAccess hammer
readers vs writers under -race; both FAIL on the pre-fix code with the exact
production signatures ("concurrent map read and map write" / "concurrent map
writes") and pass after.
BUG 3 — a proposervm height index BELOW the inner VM tip (the devnet-C
"index 7 < inner 8" from a snapshot restored inconsistently across the proposervm
and EVM databases) had a cryptic fatal ("should never be lower"). It is now a
LOUD, ACTIONABLE fatal with a recovery runbook. It is deliberately NOT
"self-healed" by dropping the finality pointer: after DeleteLastAccepted,
proposervm.LastAccepted() falls back to the INNER-namespace id, whose ParentID is
contiguity-incompatible with the network's OUTER wrappers — that permanently
wedges bootstrap (first-block anchor), catch-up (parent==tip guard) and live
Verify (parent lookup) at the inner tip (blocks at height <= tip are skipped, so
the missing wrapper is never rebuilt): a SILENT wedge strictly worse than the
loud stop. The only correct remedy is operator action (restore a consistent
snapshot or full resync), which the error now states. The reconciliation decision
is decomplected into a pure classifyHeightRepair(pro, inner) -> heightRelation,
regression-locked by TestClassifyHeightRepair so a revert to a silent reset fails
the test; the fork height is read lazily (only the ahead/rollback path needs it),
so the match and behind paths gain no new failure mode.
RED-reviewed: CHANGE 1 (locks) cleared (deadlock-free DAG, fail-on-old proven);
the original bug-3 silent-reset was caught as a wedge and reworked to this
fail-loud form. Full vms/proposervm/... suite green under -race.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two pre-existing reliability bugs surfaced by the v1.32.1 roll (both present
identically in v1.31.5 — NOT caused by QCv2), plus a snapshot-recovery self-heal.
BUG 1 (HIGH) — concurrent-map crash in getPostForkBlock. The consensus engine
drives the proposervm from multiple handler goroutines concurrently (a
PullQuery/Put verifying a block WRITES verifiedBlocks via verifyAndRecordInnerBlk
while a Qbit handler READS it via GetBlock -> getPostForkBlock); the retry loop
in PullQuery even documents the concurrency ("allow concurrent Put to complete").
The verifiedBlocks map was read/written with no lock -> the Go runtime aborts
with "fatal error: concurrent map read and map write" (exit 2) under heavy
verify load. Fix: a dedicated verifiedBlocksLock (RWMutex) behind three accessors
(cachedVerifiedBlock / recordVerifiedBlock / forgetVerifiedBlock) at ALL seven
sites; the sibling Tree.nodes map (touched on the same Verify/Accept paths) gets
its own RWMutex, with Accept restructured to make its inner-VM Accept/Reject
callouts OUTSIDE the lock. Both are leaf locks, never nested, never held across a
callout — provably deadlock-free. innerBlkCache (lru.SizedCache) is already
internally locked. Regression: TestVerifiedBlocksConcurrentAccess +
TestTreeConcurrentAccess hammer readers vs writers under -race; both FAIL on the
pre-fix code with the exact production signatures ("concurrent map read and map
write" / "concurrent map writes") and pass after.
BUG 3 — a proposervm height index BELOW the inner VM tip (the devnet-C
"index 7 < inner 8" from a partially-restored snapshot) was a FATAL init error
that bricked the whole chain ("error creating required chain" -> exit 1). The
proposervm cannot fabricate the missing wrapper blocks, so it now drops the stale
finality pointer and re-bootstraps from peers via the catch-up transport (the
SAME recovery the forkHeight-rollback branch already uses) instead of crashing.
The reconciliation decision is decomplected into a PURE planHeightRepair
(proHeight, innerHeight, forkHeight) -> repairAction, regression-locked by
TestPlanHeightRepair so a revert to fatal fails the test.
Full vms/proposervm/... suite green under -race.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Asserts the safety/liveness boundary that makes the consensus down/wedged/forked-
proposer fix BFT-safe — the proposer-schedule half of avalanchego Snowman++
(windower.go is byte-identical to ava):
- DETERMINISM (safety): two independent windower instances over the same set, and
the same inputs across 64 seeds, compute byte-identical ExpectedProposer for
every (chainID, height, slot). Honest nodes cannot disagree on the eligible
proposer, so an out-of-turn signed block is rejected by EVERY honest node and an
attacker cannot flood competing accepted blocks / fork.
- OUT-OF-TURN REJECTION (safety): exactly ONE validator is in-turn per slot; every
other is out-of-turn (the windower half of verifyPostDurangoBlockDelay's
errUnexpectedProposer) — no early/out-of-turn acceptance hole.
- SLOT ROTATION (liveness): consecutive slots designate different proposers, so a
down/wedged/forked designated proposer is routed around within a few slots — a
faulty leader cannot halt the chain.
Tests the EXISTING windower (no production change); proves the boundary the
consensus liveness fix relies on holds over the input space.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
arm64 validators (spark GB10, arm64 nodes generally) could not pull
ghcr.io/luxfi/node — the Docker workflow built linux/amd64 only. The
Dockerfile already cross-compiles via TARGETARCH with CGO_ENABLED=0, so
buildx produces arm64 on the amd64 lux-build pool without QEMU-emulating
the compile. Adds a workflow_dispatch `tag` input to (re)build an existing
release tag (e.g. v1.32.1) as a multi-arch manifest.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Pins the incident-1082814 durable fix. consensus v1.32.1 supersedes both the
down-leader line (v1.31.1, what main pinned) and the QCv2-core-only line
(v1.31.2, what node v1.31.5 shipped). Finality keys on the canonical inner
execution commitment; equivocation evidence requires a VERIFIED alpha-of-K cert
over a DIFFERENT canonical, removing the false-slash fatal-exit crash that
destabilized mainnet under DEX-fill load on v1.31.5.
Adds indirect github.com/luxfi/bft v0.1.5 (cert verify gate). Full node module
compiles clean; consensus proof suite 377 subtests + race + vet + fuzz green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
consensus v1.31.1 fixes the down-leader HALT: when a slot-leader proposer is
down, validators built competing siblings at the same height, the alpha-of-K
votes split, no cert assembled, and the chain stalled (devnet stuck at height
36 with 4/5 healthy). The fix converges all validators onto one deterministic
build tip (PreferredBuildTip) so blocks finalize through a validator going down.
Transitive: corona v0.8.0->v0.10.2, dkg v0.3.0->v0.3.5, pulsar v1.2.0->v1.7.1
(the dealerless PQ line consensus v1.31.1 is built against). luxd builds clean.
luxfi/pq's v1.0.3 git tag was force-moved upstream after this go.sum was
recorded. go.sum kept the originally-published zip hash
h1:pFlQm1+5FuKTDUh2y/23bXWkN4I2Rc5iuxJypwDFFMs= (still served by
proxy.golang.org / sum.golang.org). The release builds fetch luxfi modules
direct from GitHub (GOPRIVATE=github.com/luxfi/* => GONOPROXY) on a cold
cache, bypassing the immutable proxy; the moved tag now hashes to
h1:ksw1dmfTR0dqqNMRS7BjGcprCO2Fhc+3Iiq2/NMuONw=, so the goreleaser
darwin_amd64 build (.github/workflows/build.yml) failed strict
-mod=readonly verification:
verifying github.com/luxfi/pq@v1.0.3: checksum mismatch
downloaded: h1:ksw1dmfTR0dqqNMRS7BjGcprCO2Fhc+3Iiq2/NMuONw=
go.sum: h1:pFlQm1+5FuKTDUh2y/23bXWkN4I2Rc5iuxJypwDFFMs=
SECURITY ERROR
=> v* tags produced no macOS artifacts. (The "downloading luxfi/log" line in
the CI log was a concurrent-download red herring; pq is the culprit.) The
Docker image escapes this because the Dockerfile strips first-party go.sum
lines and rebuilds them under -mod=mod.
Refresh only pq's content hash to the current tag. Version stays v1.0.3 and
the /go.mod hash is unchanged (pq's deps did not change). Keeps the release
build strict and consistent with the Docker image.
Verified: fresh direct fetch verifies readonly against the new go.sum
(exit 0); the exact goreleaser target
(CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -trimpath -ldflags='-s -w'
./main) compiles to a Mach-O x86_64 binary; cold clean-cache darwin
`go mod download` no longer raises SECURITY ERROR.
consensus v1.31.0 on top of the v1.30.0 sibling-tolerant finality already shipped in
node v1.31.1: the finality decomplect (pure Ledger,Cert,DAG->Ledger',Plan fold;
markFinalized unwriteable), topological.go preference layer, the RED HIGH-1 clone()
O(window) perf fix (was O(chain height) — 86.7ms/100MB per finalize at 10^6 heights),
and RED LOW-1/LOW-2 hardening. Backward-compatible: node compiles with zero API drift;
proposervm + chains green. No transitive crypto-dep bumps.
Consume the sibling-tolerant finality fix: cert-driven FinalizeBranch reorg
replaces the per-height admission gate that deadlocked the proposervm
pre-fork->post-fork transition (devnet stuck at finalized height 1 while
blocks produced past 10). Base = v1.31.0 (= v1.30.101 union quasar PQ-finality
+ frontier self-vote) + the repliedCovers eclipse-defense cherry-pick.
Verified: go build ./... clean; chains + proposervm tests green against v1.30.0.
RED CRITICAL: the frontier caught-up shortcut gated on CONNECTIVITY
(fullyConnectedBeacons) while the tally keyed on REPLIES (CaughtUp over
withSelfVote). The two diverge for a beacon that is CONNECTED but SILENT
(TCP up, frontier reply delayed/withheld) — a capability strictly weaker
than a full eclipse, and one that occurs naturally when an ahead beacon
replays state on a mass co-restart. Such a beacon counts as fully
connected yet is absent from the tally, so self's heavy weight backfills
the naming floor and the node goes live at a forged STALE height.
Add repliedCovers() set-containment guard: every connected beacon must
have answered THIS frontier round before the self-vote shortcut can fire.
A silent ahead-beacon now blocks caught-up -> node fails safe to
FrontierConnecting (keeps waiting); a genuine fresh net (every connected
beacon answers genesis) still completes.
Keep RED's zz_red_probe_test.go as a permanent regression guard:
connected-but-silent ahead beacon -> status=FrontierConnecting (was
FrontierCaughtUp). Fresh-net + equal-stake self-vote tests still pass.
Supersede the stale Quasar-wrapper notes with the actual consensus/quasar verify
gate: dormant-by-default safety contract, files/tests, and the owner-gated
remaining-work map (producer+encoder export, gossip ingest, per-epoch validator
provider, config wiring, grace-window runbook) + mainnet activation order.
Red adversarial review of the PQ-finality gate. Dormant default verified a true
no-op; these harden the ACTIVATED path before any forward-dated activation:
H1 (HIGH) — bindCheck now binds cert.Epoch == cp.Epoch (and cert.Round). The
gate resolves verification keys from the cert's epoch; an unbound epoch let a
cert signed under a DIFFERENT validator-set era (e.g. a compromised RETIRED
committee key) certify the current block, nullifying KeyEra rotation as a
blast-radius bound. Honest certs match (producer signs Subject.Epoch==cp.Epoch).
+ wrong-epoch / wrong-round anti-replay test cases.
M2 (MED) — activation is now HEIGHT-ONLY, deterministic. Removed time.Now() (and
the ActivationConfig.Time field) from the accept path: wall-clock gating split
finalization across validators with skewed clocks (some halting on a missing
cert, others finalizing without one). A height is consensus-agreed; timestamp
forward-dating is expressed by choosing the activation height.
L5 (LOW) — an activated gate with a nil store/validator provider now fails
closed with ErrGateMisconfigured instead of panicking in the accept hook (a
panic would halt the chain uncontrollably). Off the dormant path. + test.
M3 (doc) — made the StateRoot=0 transitive-through-BlockHash contract explicit in
bindCheck (non-zero cert StateRoot already rejected + tested) so the producer
follow-on cannot silently ship state-committing certs that self-halt.
proposervm: verifyQuasarFinality short-circuits on nil gate BEFORE building the
Checkpoint, so the default path makes zero block-accessor calls.
13 gate tests green under -race; full proposervm suite green; go build ./... exit 0.
Deferred to follow-ons (gate is dormant + unwired in production): M4 cert-
unavailability halt runbook + grace window, L6 MemCertStore eviction (no ingest
caller yet), proposervm Accept-path integration test, positive valid-cert test
(needs consensus to export cert encoders).
Node-side integration of luxfi/consensus v1.29.0 Quasar compact-cert finality.
luxd finalizes on classical Snow every block; at CHECKPOINTS a sampled committee
QuasarCert over the finalized digest is VERIFIED. This wires the verify half as
an OPTIONAL, FORWARD-DATED, DORMANT-BY-DEFAULT check in the block-accept path.
consensus/quasar (new package):
- Gate.VerifyAccepted: the accept-path safety boundary. nil gate / zero
activation / below-activation-height / non-checkpoint => no-op (classical
Snow unchanged). Post-activation at a checkpoint => REQUIRE a valid cert
bound to the finalized block; fail closed (missing/mismatch/invalid -> error).
- ActivationConfig: forward-dated dormant switch (Height==0 => never; optional
wall-clock Time gate). The safety contract: this branch changes NOTHING about
live finality until the owner sets a real activation height.
- bindCheck: anti-replay binding (cert chain/height/block/state == finalized).
- policyStore: default posture HYBRID_PQ = Beam(BLS) ^ Pulsar (Pulsar verify
~140us < BLS, cert ~27KB); STRICT_DUAL_PQ / POLARIS configurable. The cert
can never select its own (weaker) policy (I1/I2 enforced by consensus).
- ValidatorSet/MemCertStore: the per-leg key + cert-lookup seams (HYBRID_PQ
BLS+Pulsar keys now; Corona/Magnetar/weighted-set + per-epoch resolution are
the activation-wiring follow-on).
- Producer (scaffolding): the committee-sign service contract + MaybeProduce
request site, sharing ONE checkpoint cadence with verify. nil producer =
verify-only (the default). Concrete pulsard signer is the follow-on; it needs
consensus to export the (currently package-private) cert/payload ENCODERS.
proposervm: post_fork_block.Accept calls vm.verifyQuasarFinality(b) BEFORE the
accept commits, so an un-PQ-certified checkpoint halts without persisting. nil
gate (default) => unchanged classical accept. SetQuasarGate installs it.
Tests: 12 gate tests green (dormant no-op, nil-safe, below-activation, time-gate,
non-checkpoint, missing-fails-closed, 4x mismatch/anti-replay, real-verifier
delegation, validator-set-unavailable, producer nil-safety/dormant/active). Full
proposervm suite green; go build ./... exit 0.
On a FRESH net a validator whose own stake makes the PEER-ONLY responder weight
fall below the stake-majority NAMING floor (a heavy validator, a small beacon set
like the P-chain's CustomBeacons, or skewed stake: peers = total-self, so
peers < total/2+1 ⟺ self > total/2-1) could NEVER name a frontier even with the
WHOLE validator set connected — FrontierTip returned FrontierConnecting forever
("bootstrap waiting for beacon connectivity before naming the frontier") and the
node hung, blocking uniform block production.
The node is itself a beacon and knows its OWN accepted tip with certainty. Add the
SELF-VOTE: in the below-floor case, when the node has reached its ENTIRE beacon set
(fullyConnectedBeacons — no eclipse can hide an ahead-tip) and that full set PLUS
itself unanimously hold a tip it has ALREADY ACCEPTED with nobody ahead, return the
new chainbootstrap.FrontierCaughtUp -> the loop completes at the node's own tip.
Safety (proven by tests): self is counted ONLY under FULL connectivity, so an
eclipse suppressing any beacon breaks full-set and falls back to the
partition-capture floor (fail safe) — self can never tip a PARTIAL view into a
false caught-up (stale-go-live guard holds). AcceptsFrontier still tallies PEERS
ONLY, so a behind node's ahead-frontier is named undiluted and it still syncs; a
peer above our height defeats CaughtUp. self only vouches for its OWN accepted tip
so C1 (forged-frontier naming) is untouched.
- selfNodeID threaded from m.NodeID into the blockHandler.
- helpers fullyConnectedBeacons + withSelfVote.
- bump luxfi/consensus v1.29.0 -> v1.29.1 (FrontierCaughtUp status).
Tests: TestNodeBootstrap_FreshNet_SelfVoteUnderFullConnectivity_CaughtUp,
TestNodeBootstrap_SelfVote_PeerAheadDefeatsCaughtUp,
TestNodeBootstrap_SelfVote_PartialConnectivityFailsSafe. Full chains suite green.
The node declared n.StakingTLSSigner (passed to the chain manager → proposervm
StakingLeafSigner) but NEVER assigned it, so proposervm received a nil signer.
The pre-fork→post-fork transition block (height 1) is unsigned and built fine,
but the first SIGNED post-fork block (height 2) called block.Build → key.Sign on
a nil key → SIGSEGV nil-pointer panic → node crash (exit 2), flushing the
mempool and dropping the DEX deploy. Assign n.StakingTLSSigner from the staking
TLS cert's private key (already extracted at this point for the TLS config),
mirroring avalanchego node.go. Now the elected proposer signs post-fork blocks.
Block production + 5-node α-of-K finality verified converging on devnet
(identical block-1 hash across all 5) once this lands.
Addresses RED's DO-NOT-SHIP review of the proposervm re-wrap. Four fixes:
CRITICAL-2 (consensus v1.25.36→v1.25.37): the equivocation handler downgraded
Logger.Crit→Error. The per-height guard already rejects the second conflicting
cert (safety preserved); Crit→os.Exit converted a handled safety event into a
fleet-wide liveness kill (every honest node that observed the gossiped cert
exited) AND made the slashing-evidence loop dead code. Now: log Error, reject,
record DoubleVote evidence, keep running.
CRITICAL-1 (pre_fork_block.go + vm.go): window the pre-fork→post-fork TRANSITION
block. It must stay unsigned (verifyPostForkChild rejects a signed transition),
but WHO builds it is now gated by the windower's ExpectedProposer — only the
elected leader builds the unsigned height-1/N+1 block; non-leaders drop it and
adopt the gossiped block (timeToBuildPreForkTransitionLocked windows the wait).
Without this, every validator built its own transition block stamped with its
local wall-clock second → divergent height-1 blocks → fork at chain start.
CRITICAL-3 (manager.go + config.go + vm.go): thread the resolved networkID into
proposervm.Config and the windower (newWindower) instead of hardcoding
PrimaryNetworkID. On a sovereign L1 (Zoo/Hanzo/Pars, networkID==chainID) the
hardcoded primary set was empty → ErrAnyoneCanPropose → single-proposer
silently did not hold. Resolved ONCE in manager.go so windower and cert side
share the identical set.
Fresh-chain init tolerance (vm.go): repairAcceptedChainByHeight returns nil when
the inner last-accepted block is not retrievable (brand/feature VMs Q/A/G/K
whose genesis references an empty parent → GetBlock 'not found' even though the
ID is not ids.Empty). Nothing to repair; previously crashed the whole node at
init. Complements the ids.Empty guard.
proposervm tests pass.
Completes the multi-validator block-production fix on top of the v1.30.96
proposervm wrap. Three subtle correctness gaps that left single-proposer
NOT actually holding:
CRITICAL-1 (pre-fork→post-fork transition): the FIRST post-fork block (which a
fresh-genesis chain like devnet hits immediately) was forwarded to the inner VM
and built by EVERY validator unconditionally — forking the chain at its very
start. timeToBuildPreForkTransitionLocked now windows the transition block the
same way post-fork blocks are windowed (MinDelayForProposer), so a non-leader
waits its slot and adopts the elected leader's gossiped transition block; a down
leader can't stall (the eligible set widens as the slot advances).
CRITICAL-3 (validator-set ID): the windower hardcoded PrimaryNetworkID, so a
sovereign L1 (Zoo/Hanzo/Pars, validators under their own networkID==chainID) got
an EMPTY set → ErrAnyoneCanPropose → equivocation exactly like the unfixed
C-Chain, AND diverged from the cert's set. config.NetworkID (resolved once in
manager.go: primary for native, else the L1's chainID, fallback to primary if
empty) now binds the windower to the SAME set the cert uses. Zero value
(ids.Empty) IS PrimaryNetworkID, so native chains are byte-identical. newWindower
encapsulates this.
proposervm pkg + full luxd binary build clean; go vet clean (chains, proposervm).
Deploying the proposervm re-wrap crash-looped the whole node on a fresh
re-genesis: repairAcceptedChainByHeight read the inner VM's last-accepted as
ids.Empty (some Lux VMs report the empty ID before their first acceptance) and
called GetBlock(ids.Empty) → 'failed to get block 111...LpoYY: not found' →
VM initialization failed → 'error creating required chain' → node exit(1).
A fresh inner chain has no accepted chain to roll the proposervm height index
back against, so there is nothing to repair. Guard the empty-ID case with an
early return, mirroring the existing 'nothing to repair' returns (and the
empty-handling setLastAcceptedMetadata already does for the same fresh case).
ava never hits this because its inner VMs return a valid genesis last-accepted;
Lux's brand/feature VMs can return empty.
The prior commit (9d9d2a8785) captured only manager.go, which CALLS
shouldWrapInProposerVM but did not include its definition, so HEAD did not
compile (chains/manager.go:1205: undefined: shouldWrapInProposerVM). This adds:
- chains/quorum.go: shouldWrapInProposerVM(k, chainID, innerIsDAGNative) —
the single pure policy gate (K>1 AND not P-Chain AND not DAG-native) that
decides whether a linear chain.ChainVM is wrapped in proposervm for
single-proposer-per-height block production.
- chains/proposervm_wrap_test.go: table tests pinning the gate (C-Chain
devnet/mainnet wrapped; P-Chain, X-Chain, K==1 excluded) and that the K
flows from selectConsensusParams.
- chains/manager.go: gofmt import ordering.
Restores a green build. No push, no tag.
The deeper half of the block-production fix. Lux had commented out the
proposervm wrapper and hand-rolled a WaitForEvent→toEngine bridge that dropped
the single-proposer invariant: every validator's engine called BuildBlock
UNCONDITIONALLY at every height off a slightly-different mempool, so two nodes
could each gather an α-of-K cert for DIFFERENT blocks at the same height →
reportCertEquivocation → Logger.Crit → crash (and the DEX-tx nondeterminism
wedge). That, together with the evm builder-ready race (evm v1.99.51), is why
every multi-validator C-Chain froze.
Re-wrap multi-validator linear chains in proposervm so block production follows
the Snowman++ proposer schedule — exactly ONE validator builds height H, the
rest wait and vote. proposervm.WaitForEvent only returns PendingTxs inside THIS
node's proposer window, collapsing the race at its source.
Gated (all three): K>1 (single-proposer only matters for a quorum), not the
P-Chain (its height-indexed validators.State is published during its own
createChain, after the snapshot used here), not DAG-native (X-Chain's Linearize
bridge doesn't compose with proposervm's pull/window model). Engine, WaitForEvent
bridge, block handler, SetState, and HTTP registration all route through the
wrapped engineVM; unwrapped chains keep the original path byte-for-byte.
chains pkg builds + go vet clean. Validated end-to-end on devnet (5 validators).
P0: all Lux C-Chains (mainnet/testnet/devnet) froze at their imported frontier —
no live block produced. Root cause in the EVM plugin block-build trigger:
1. WaitForEvent blocked forever when the node forwarder called it before the
builder initialized (nil-builder race) — so PendingTxs never reached the
engine and no block ever built.
2. A block built faster than parent.Time+TargetBlockRate has a higher required
fee; a small-tip/legacy tx can't cover it, verifyBlockFee rejects it, stall.
evm v1.99.51 fixes both (builderReady select + target-rate pacing, with a
nextBuildTime pure helper + regression test). chains v1.4.8 rebuilds the
C-Chain VM plugin against it.
Dockerfile: EVM_VERSION v1.99.50->v1.99.51, chains pin + CHAINS_REF
v1.4.7->v1.4.8. node go.mod chains v1.4.7->v1.4.8 (+ tidy for the new
genesis/security transitive). luxd binary builds clean.
Carries evm v1.99.50: paces block building to the target block rate, fixing
(1) small transactions never mining because the rapid-block gas cost exceeds
their tips, and (2) the equivocation crash-loop from validators racing to
finalize conflicting blocks at the same height. Together with v1.99.49
(WaitForEvent builder-ready race), a fresh multi-validator EVM chain now builds
and finalizes blocks under full BFT consensus and accepts ordinary traffic.
Carries evm v1.99.49: fixes the fresh-start block-production stall where the
C-Chain EVM plugin's WaitForEvent blocks forever when the node-side notification
bridge calls it before SetState(NormalOp) initializes the block builder. Without
this, a freshly-booted (or restarted) EVM chain accepts txs into the mempool but
never builds a block. See luxfi/evm#fix/waitforevent-builder-ready-race.
The C-Chain EVM plugin (mgj786NP, where 0x9999 lives) is built from EVM_VERSION,
NOT node's go.mod. v1.30.91 bumped the binary deps but the plugin still built from
the stale v1.99.40 (precompile v0.12.0, no DEX fix). Bump EVM_VERSION→v1.99.47
(precompile v0.15.0 active-only DEX money path) + chains v1.3.21→v1.3.22 to match.
Now the deployed plugin actually carries the real-fill DEX code.
warp v1.23.0 = typed finality-evidence model + Pulse->Corona relabel.
No node package references the renamed warp symbols: go build ./... is green
and every warp test passes (vms/platformvm/warp{,/message,/payload,/zwarp},
vms/platformvm/network, vms/platformvm, vms/example/xsvm). Pure dependency bump.
Note: vms/platformvm/txs test binary has a pre-existing, warp-unrelated compile
error (add_validator_test.go:373 undefined: fmt) that predates v1.30.88 and is
not addressed here.
A validator holding gossiped-but-unaccepted blocks (in the VM store, lastAccepted
below them) concluded caught-up at its stale height forever — the caught-up check
conflated store presence (Has/heldHeight via vm.GetBlock) with acceptance.
Fix (consensus v1.25.36): Has→Accepted (a stored-but-unaccepted named frontier
descends+ACCEPTS through the cert-gated per-height guard); recognize convergence
off the in-process consensus finalized ledger, not the frozen VM.LastAccepted
cache; un-freeze the zap client's lastAcceptedID on Accept. Proven on mainnet
luxd-2 (1082780→1082796). C1/VerifyWeighted/cert-gate intact. blue→red→blue→red.
The v1.30.84 bootstrap-vm-ready ordering fix passed 50 unit tests + the BFT
proof but FAILED the real mainnet canary: a STALE spare (luxd-2) at C-Chain
accepted height 1082780 went Ready at 1082780 instead of fetching the 16 blocks
to the producers' frontier 1082796, then sat frozen. The mocks injected the
FULL, STABLE validator set from t=0 — masking the production trigger.
ROOT CAUSE (primary). A native chain's runBootstrapThenPoll starts right after
its VM.Initialize (dispatchChainCreator), NOT after the P-chain has finished
its own initial sync. The P-chain populates m.Validators as it replays blocks
(genesis stakers + every AddValidatorTx), so FrontierTip can run while
GetMap(PrimaryNetworkID) is a PARTIAL set. The MinResponseWeight stake-majority
floor (total/2+1) is fail-secure ONLY when `total` is the TRUE full-set stake;
under a partial denominator a degenerate at-or-below responder subset (the
genuinely-ahead producers not yet loaded) clears the floor and the exact-fast-
path / CaughtUp concludes "caught up" at the stale local height. Proven by
elimination: with the full set EVERY accurate-reply path is fail-safe (wait /
sync), so the freeze requires a non-representative responder set.
FIX (primary): gate a native non-platform chain's frontier-trust on the P-chain
having COMPLETED initial sync (m.pChainBootstrapped, published by
monitorBootstrap; read via blockHandler.primaryNetworkReady). Until then
FrontierTip returns FrontierConnecting (WAIT), so every branch judges the TRUE
full validator set and the existing floor logic holds. Deadlock-free: the
P-chain converges independently from its own configured CustomBeacons.
FIX (self-heal / the "ALSO FIX"): pollFrontierOnce filtered peers on b.chainID —
the IDENTICAL chainID/networkID confusion already fixed in connectedBeacons.
Peers advertise the NET they track, never an individual native chain id, so the
sample was always empty and a behind-but-Ready validator never discovered it was
behind. Filter on b.networkID so the NormalOp catch-up poller actually fires.
INSTRUMENTATION: FrontierTip now logs (Debug) the full beacon-set size + total
stake (the floor DENOMINATOR — a partial value is the smoking gun), connected
count, and every reply's tip + locally-resolved height + held-or-not, plus the
decision. Set the C-chain log level to Debug on the canary to capture ground
truth.
TESTS (production-modeling, not the ideal full set): TestRED_PartialStakedSet_
BehindNodeMustNotGoLiveStale reproduces the freeze (without the gate it NAMES the
stale own tip; with it, WAITS; then converges on the full set) — proven
load-bearing (disabling the gate fails the test). Plus not-held-tips → sync,
empty-ancestry → stays Bootstrapping (never Ready stale), and peer-connect-delay
→ wait-then-converge. Full chains suite green (C1, co-restart, M1,
mass-recovery, skewed-weight all preserved), race-clean.
nameFrontier's ancestor-tolerant filter was `ref.Height < MinFrontierHeight`
(MinFrontierHeight = the node's own last-accepted), so a block AT the node's
own height passed the filter and could be NAMED. Red's M1 eclipse: with the
genuinely-ahead responders throttled below the ⅔ naming threshold (R_a < ⅔R)
but the at-height responders let through, the node's OWN height accrues ⅔
purely as the shared ANCESTOR of the ahead N+5 tips → nameFrontier names it →
FrontierNamed at own height → the node goes Ready STALE (5 blocks behind a
finalized N+5). nameFrontier ran BEFORE CaughtUp in FrontierTip, so it
short-circuited the stricter CaughtUp determination that already refuses this.
FIX (one operator): `ref.Height <= MinFrontierHeight` — own height is excluded
from naming, routing the at-own-height case to CaughtUp, which alone
distinguishes a legit all-at-N fleet (Ready at N) from an eclipse with ahead
tips the node lacks (refuse → sync). The two paths compose with no gap:
nameFrontier names STRICTLY ABOVE own height (behind nodes); CaughtUp decides
at-or-below own height (tip-holders); the fast path (active ⅔-direct report)
stays exempt for the legit unanimous-at-tip case.
Safe for layer-5 + C1: a behind node names HIGHER than its own height (survives
both the old `<` and new `<=`); the ⅔-of-responder-weight naming requirement is
untouched (only height eligibility tightens — strictly safer). Round-1
behind-node convergence is unchanged.
Faithful test correction (manager.go: GetAcceptedFrontier answers vm.LastAccepted
— the last-ACCEPTED block, NEVER an un-finalized preferred tip): the two
ancestor-tolerant convergence integration tests modeled a beacon reporting an
un-finalized tip FOREVER (a gap-1 instance of this very M1 stale-go-live). They
are restated to model finalization PROPAGATION — the node names the ⅔-common as
a behind node, then RATCHETS to the true finalized tip as the laggards' accepted
frontier advances — converging to the real frontier (stronger guarantee), never
falsely caught-up below it.
Tests (RED-before by reverting the one-operator hunk, GREEN-after):
- TestBootstrapTrust_EclipseOwnHeightNotNamedRoutesToCaughtUp (policy, deterministic
+ own-height boundary)
- TestRED_EclipseOwnHeightNotNamedRoutesToSync (integration: FrontierNoQuorum +
full-loop fails safe at stale N + eclipse-lifts→syncs-up)
GOWORK=off CGO_ENABLED=0 go test ./chains/ — all green (layer-5 convergence, C1
forged/eclipse/partition, caught-up co-restart, fast-path, self-heal).
RED found a CRITICAL liveness regression in the VM-ready gate (b313912413): a
tip-holding producer on a mixed-height co-restart cannot satisfy its own go-live
gate and fails safe DOWN at its own tip — the OPPOSITE of the stale-go-live bug the
gate fixed, and just as wrong.
Mechanism (PoC reproduced): a producer at N sees its 4 peers split
{2 producers@N, luxd-2@N-16, luxd-0@genesis}. The tip-holders are only ½ of the
responders (< ⅔) so no frontier is NAMED; the ⅔-backed common ancestor N-16 is
below the node's own last-accepted (MinFrontierHeight filters it). → ErrNoBeaconQuorum
→ runInitialSync false → C-Chain STOPPED. The loop's Has(tip)→caughtUp shortcut is
only reached AFTER FrontierNamed, which a tip-holder never gets.
PRIMARY FIX — a CAUGHT-UP determination (BootstrapPolicy.CaughtUp), the dual of
AcceptsFrontier ("nobody is ahead" vs "here is the block ahead"). On the
ErrNoBootstrapQuorum branch, FrontierTip concludes caught-up and names the node's
OWN held tip (→ the loop's existing Has()-shortcut → Ready at own height) iff:
(a) the SAME response floor AcceptsFrontier uses is met (MinResponses count AND
MinResponseWeight stake-majority) — an eclipse hiding the ahead-nodes drops
below the floor → fail safe, no partition-capture;
(b) every responder's reported ACCEPTED tip is at height ≤ lastAccepted — a stale
node has an honest responder ahead → still syncs (stale-go-live stays fixed);
(c) the node HOLDS every reported tip (heightOf injected, VM-free) — never declares
caught-up to a sibling/fork it lacks.
floorMet + tallyResponders are extracted so naming and caught-up share ONE floor and
ONE eligibility rule (DRY). No consensus change — reuses FrontierNamed + Has, builds
GOWORK=off against the deployed consensus v1.25.35.
SELF-HEAL (RED MEDIUM #2) — the K8s probes all poll the always-green
/ext/health/liveness, so a fail-safe-DOWN node is NEVER restarted (a permanent brick).
runInitialSync now RE-ATTEMPTS a transient connectivity fail-safe (bootstrapMaxAttempts
≤ 0 ⇒ until the quorum returns or shutdown), staying in Bootstrapping (never live at
stale height) and converging the instant the quorum returns. bootstrapConnecting tells
monitorBootstrap's no-progress watchdog this is a deliberate WAIT (not a stall) so it
does not force-STOP a node correctly waiting for its quorum. Structural failures (deep
gap → state-sync) are NOT retried.
LOW — FinishBootstrap now runs BEFORE transitionVMReady (close the no-cert
AcceptBootstrapBlock gate before the VM can build); transitionVMReady's SetState timeout
is derived from the shutdown ctx (was context.Background()).
MEDIUM — createDAG (X/Q) go-live left UNCONDITIONAL, now documented as a DELIBERATE
linear-only scope (mainnet froze on linear C/D/B/T/Z; DAG bootstrap convergence is a
separate task). createDAG does not regress.
C1, VerifyWeighted, the ⅔-stake cert, and "no VerifiedQuorumCert no finality" are
untouched. Tests (chains): TestRED_TipHolderCoRestartGoesReadyAtOwnTip reproduces the
regression — RED before on b313912413 (FrontierNoQuorum, live==false), GREEN after
(FrontierNamed at own tip, vmReady once at N). Plus 6 CaughtUp unit tests (incl. both
adversarial fake-caught-up cases), TestRED_MajorityOutageSelfHealsWhenQuorumReturns,
and TestWatchBootstrapProgress_ConnectingNotStalled. Full C1/forged/eclipse/partition
suite re-run green (41 bootstrap tests, 0 fail).
