mirror of
https://github.com/luxfi/node.git
synced 2026-07-27 03:39:39 +00:00
51a304804c51f522c245551aff3ee5b7fb96b62b
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 zeekayandHanzo Dev
|
51a304804c |
chore: migrate luxd HTTP routes /ext -> /v1
Drop the Avalanche-heritage /ext prefix; /v1 is the single canonical route surface (one way, no backward compat). The node's baseURL is the source of truth; clients, SDKs, CLI, indexer, maker, genesis, netrunner, and the k8s/compose/gateway/explorer configs are updated to match. Co-authored-by: Hanzo Dev <dev@hanzo.ai> |
||
|
zeekay
|
774dd82c63 |
fix(bootstrap): self-vote caught-up requires every connected beacon replied
RED CRITICAL: the frontier caught-up shortcut gated on CONNECTIVITY (fullyConnectedBeacons) while the tally keyed on REPLIES (CaughtUp over withSelfVote). The two diverge for a beacon that is CONNECTED but SILENT (TCP up, frontier reply delayed/withheld) — a capability strictly weaker than a full eclipse, and one that occurs naturally when an ahead beacon replays state on a mass co-restart. Such a beacon counts as fully connected yet is absent from the tally, so self's heavy weight backfills the naming floor and the node goes live at a forged STALE height. Add repliedCovers() set-containment guard: every connected beacon must have answered THIS frontier round before the self-vote shortcut can fire. A silent ahead-beacon now blocks caught-up -> node fails safe to FrontierConnecting (keeps waiting); a genuine fresh net (every connected beacon answers genesis) still completes. Keep RED's zz_red_probe_test.go as a permanent regression guard: connected-but-silent ahead beacon -> status=FrontierConnecting (was FrontierCaughtUp). Fresh-net + equal-stake self-vote tests still pass. |
||
|
zeekay
|
d70a407a3a |
fix(chains): fresh-net frontier-naming — self-vote under full connectivity (consume consensus v1.29.1 FrontierCaughtUp)
On a FRESH net a validator whose own stake makes the PEER-ONLY responder weight
fall below the stake-majority NAMING floor (a heavy validator, a small beacon set
like the P-chain's CustomBeacons, or skewed stake: peers = total-self, so
peers < total/2+1 ⟺ self > total/2-1) could NEVER name a frontier even with the
WHOLE validator set connected — FrontierTip returned FrontierConnecting forever
("bootstrap waiting for beacon connectivity before naming the frontier") and the
node hung, blocking uniform block production.
The node is itself a beacon and knows its OWN accepted tip with certainty. Add the
SELF-VOTE: in the below-floor case, when the node has reached its ENTIRE beacon set
(fullyConnectedBeacons — no eclipse can hide an ahead-tip) and that full set PLUS
itself unanimously hold a tip it has ALREADY ACCEPTED with nobody ahead, return the
new chainbootstrap.FrontierCaughtUp -> the loop completes at the node's own tip.
Safety (proven by tests): self is counted ONLY under FULL connectivity, so an
eclipse suppressing any beacon breaks full-set and falls back to the
partition-capture floor (fail safe) — self can never tip a PARTIAL view into a
false caught-up (stale-go-live guard holds). AcceptsFrontier still tallies PEERS
ONLY, so a behind node's ahead-frontier is named undiluted and it still syncs; a
peer above our height defeats CaughtUp. self only vouches for its OWN accepted tip
so C1 (forged-frontier naming) is untouched.
- selfNodeID threaded from m.NodeID into the blockHandler.
- helpers fullyConnectedBeacons + withSelfVote.
- bump luxfi/consensus v1.29.0 -> v1.29.1 (FrontierCaughtUp status).
Tests: TestNodeBootstrap_FreshNet_SelfVoteUnderFullConnectivity_CaughtUp,
TestNodeBootstrap_SelfVote_PeerAheadDefeatsCaughtUp,
TestNodeBootstrap_SelfVote_PartialConnectivityFailsSafe. Full chains suite green.
|
||
|
zeekay
|
e155575597 |
fix(chains): luxd-2 behind-node freeze — caught-up by ACCEPTANCE not store + convergence off the finalized ledger
A validator holding gossiped-but-unaccepted blocks (in the VM store, lastAccepted below them) concluded caught-up at its stale height forever — the caught-up check conflated store presence (Has/heldHeight via vm.GetBlock) with acceptance. Fix (consensus v1.25.36): Has→Accepted (a stored-but-unaccepted named frontier descends+ACCEPTS through the cert-gated per-height guard); recognize convergence off the in-process consensus finalized ledger, not the frozen VM.LastAccepted cache; un-freeze the zap client's lastAcceptedID on Accept. Proven on mainnet luxd-2 (1082780→1082796). C1/VerifyWeighted/cert-gate intact. blue→red→blue→red. |
||
|
zeekay
|
33f4c8f22c |
fix(chains): close the mainnet luxd-2 freeze — gate native-chain frontier-trust on the FULL staked set + self-heal poller (canary)
The v1.30.84 bootstrap-vm-ready ordering fix passed 50 unit tests + the BFT proof but FAILED the real mainnet canary: a STALE spare (luxd-2) at C-Chain accepted height 1082780 went Ready at 1082780 instead of fetching the 16 blocks to the producers' frontier 1082796, then sat frozen. The mocks injected the FULL, STABLE validator set from t=0 — masking the production trigger. ROOT CAUSE (primary). A native chain's runBootstrapThenPoll starts right after its VM.Initialize (dispatchChainCreator), NOT after the P-chain has finished its own initial sync. The P-chain populates m.Validators as it replays blocks (genesis stakers + every AddValidatorTx), so FrontierTip can run while GetMap(PrimaryNetworkID) is a PARTIAL set. The MinResponseWeight stake-majority floor (total/2+1) is fail-secure ONLY when `total` is the TRUE full-set stake; under a partial denominator a degenerate at-or-below responder subset (the genuinely-ahead producers not yet loaded) clears the floor and the exact-fast- path / CaughtUp concludes "caught up" at the stale local height. Proven by elimination: with the full set EVERY accurate-reply path is fail-safe (wait / sync), so the freeze requires a non-representative responder set. FIX (primary): gate a native non-platform chain's frontier-trust on the P-chain having COMPLETED initial sync (m.pChainBootstrapped, published by monitorBootstrap; read via blockHandler.primaryNetworkReady). Until then FrontierTip returns FrontierConnecting (WAIT), so every branch judges the TRUE full validator set and the existing floor logic holds. Deadlock-free: the P-chain converges independently from its own configured CustomBeacons. FIX (self-heal / the "ALSO FIX"): pollFrontierOnce filtered peers on b.chainID — the IDENTICAL chainID/networkID confusion already fixed in connectedBeacons. Peers advertise the NET they track, never an individual native chain id, so the sample was always empty and a behind-but-Ready validator never discovered it was behind. Filter on b.networkID so the NormalOp catch-up poller actually fires. INSTRUMENTATION: FrontierTip now logs (Debug) the full beacon-set size + total stake (the floor DENOMINATOR — a partial value is the smoking gun), connected count, and every reply's tip + locally-resolved height + held-or-not, plus the decision. Set the C-chain log level to Debug on the canary to capture ground truth. TESTS (production-modeling, not the ideal full set): TestRED_PartialStakedSet_ BehindNodeMustNotGoLiveStale reproduces the freeze (without the gate it NAMES the stale own tip; with it, WAITS; then converges on the full set) — proven load-bearing (disabling the gate fails the test). Plus not-held-tips → sync, empty-ancestry → stays Bootstrapping (never Ready stale), and peer-connect-delay → wait-then-converge. Full chains suite green (C1, co-restart, M1, mass-recovery, skewed-weight all preserved), race-clean. |
||
|
zeekay
|
bc0cff5730 |
fix(chains): close M1 eclipse-stale own-height naming — frontier names STRICTLY ABOVE last-accepted (red fast-follow)
nameFrontier's ancestor-tolerant filter was `ref.Height < MinFrontierHeight` (MinFrontierHeight = the node's own last-accepted), so a block AT the node's own height passed the filter and could be NAMED. Red's M1 eclipse: with the genuinely-ahead responders throttled below the ⅔ naming threshold (R_a < ⅔R) but the at-height responders let through, the node's OWN height accrues ⅔ purely as the shared ANCESTOR of the ahead N+5 tips → nameFrontier names it → FrontierNamed at own height → the node goes Ready STALE (5 blocks behind a finalized N+5). nameFrontier ran BEFORE CaughtUp in FrontierTip, so it short-circuited the stricter CaughtUp determination that already refuses this. FIX (one operator): `ref.Height <= MinFrontierHeight` — own height is excluded from naming, routing the at-own-height case to CaughtUp, which alone distinguishes a legit all-at-N fleet (Ready at N) from an eclipse with ahead tips the node lacks (refuse → sync). The two paths compose with no gap: nameFrontier names STRICTLY ABOVE own height (behind nodes); CaughtUp decides at-or-below own height (tip-holders); the fast path (active ⅔-direct report) stays exempt for the legit unanimous-at-tip case. Safe for layer-5 + C1: a behind node names HIGHER than its own height (survives both the old `<` and new `<=`); the ⅔-of-responder-weight naming requirement is untouched (only height eligibility tightens — strictly safer). Round-1 behind-node convergence is unchanged. Faithful test correction (manager.go: GetAcceptedFrontier answers vm.LastAccepted — the last-ACCEPTED block, NEVER an un-finalized preferred tip): the two ancestor-tolerant convergence integration tests modeled a beacon reporting an un-finalized tip FOREVER (a gap-1 instance of this very M1 stale-go-live). They are restated to model finalization PROPAGATION — the node names the ⅔-common as a behind node, then RATCHETS to the true finalized tip as the laggards' accepted frontier advances — converging to the real frontier (stronger guarantee), never falsely caught-up below it. Tests (RED-before by reverting the one-operator hunk, GREEN-after): - TestBootstrapTrust_EclipseOwnHeightNotNamedRoutesToCaughtUp (policy, deterministic + own-height boundary) - TestRED_EclipseOwnHeightNotNamedRoutesToSync (integration: FrontierNoQuorum + full-loop fails safe at stale N + eclipse-lifts→syncs-up) GOWORK=off CGO_ENABLED=0 go test ./chains/ — all green (layer-5 convergence, C1 forged/eclipse/partition, caught-up co-restart, fast-path, self-heal). |
||
|
zeekay
|
8e0815cf7b |
fix(chains): caught-up go-live for tip-holders — close the co-restart freeze regression (RED CRITICAL)
RED found a CRITICAL liveness regression in the VM-ready gate ( |
||
|
zeekay
|
b313912413 |
fix(chains): gate VM normal-op transition on initial sync reaching the frontier
Root cause of the restart-freeze (mainnet luxd-2: C-Chain stuck at 1082780
while producers finalized 1082796): buildChain called SetState(vm.Ready)
UNCONDITIONALLY right after Initialize — at the LOCAL last-accepted height —
and ran the node-layer initial sync (runBootstrapThenPoll) afterward as a
detached, non-gating goroutine. vm.Ready fires the EVM's
onNormalOperationsStarted (block building, mempool gossip, validator dispatch);
a restarted STALE validator therefore went live at its stale height and never
converged to the finalized frontier. The proper vm.Bootstrapping state (the
EVM's onBootstrapStarted: snapshots only, no building) was skipped entirely.
Fix — restore the correct VM lifecycle, gated:
- buildChain now SetState(vm.Bootstrapping) after init (not Ready). The VM
fetch+executes the gap in Bootstrapping, serving nothing as head.
- The Ready transition moves into the bootstrap goroutine (runInitialSync,
split out of runBootstrapThenPoll for unit-testability) and fires ONLY after
bs.Run reaches the named ⅔-by-stake beacon frontier — VM live + engine
cert-gate (FinishBootstrap) go together at the frontier, never at a stale
height. Wired via blockHandler.vmReady (captures the VM's SetState).
- Fresh-genesis / single-node (FrontierNoBeacons) and at-the-tip restart
(caught-up) go Ready promptly — no regression, no hang.
- Eclipse / isolation: bs.Run's bounded ConnectDeadline is the retry that
self-heals when beacons return; past it the node fails SAFE (stays
Bootstrapping, records the reason for monitorBootstrap) — never serves stale
state, never an unbounded hang. The orchestrator restart now converges.
C1 forged-chain defense, VerifyWeighted, and the ⅔-stake quorum cert are
untouched — the frontier is still named only by the configured-beacon
⅔-of-responders quorum + content-addressed descent (all RED/BootstrapTrust
tests pass). AcceptedFrontier/Ancestors replies route to the bootstrap channels
via bsActive throughout the phase (verified).
Tests (chains): TestRED_StaleValidatorGoesReadyOnlyAfterReachingFrontier
reproduces the production scenario — RED before (VM goes live at stale N=30),
GREEN after (only at frontier N+K=46). Plus fresh-genesis no-regression,
bounded eclipse fail-safe, and at-the-tip producer fast-Ready. Race-clean.
NOTE (flagged for review, NOT in this change): the createDAG path (X/Q DAG
chains, consensusdag.Engine) has the same unconditional SetState(Ready) at
manager.go:1792 but a different engine without the node-layer bootstrap-sync
wiring — a separate fix if any production chain uses it.
|
||
|
zeekay
|
e869b0d7f5 |
fix(bootstrap): MinResponseWeight stake-majority floor — close skewed-weight partition-capture (re-red HIGH)
The MinResponses COUNT floor and the ⅔-of-responders WEIGHT agreement diverge under
SKEWED validator stake: an attacker eclipsing the HEAVY honest beacons while passing
enough LIGHT honest ones to clear the count can shrink the responder-weight denominator
until <⅓-of-total Byzantine stake names a forged frontier (re-red PoC: w={3,3,13,1,1,1},
Byz 27%). Fix: bootstrapPolicy() now sets MinResponseWeight=⌈total/2⌉ (the field was
wired into AcceptsFrontier but left 0). Proven safe AND deadlock-free: equal-weight
recovery needs only >½ (3/5=0.6≥0.5), not the >⅔ that caused the original deadlock.
Closes the re-red HIGH + INFO-2 (same root cause). Tests H (skewed→reject, load-bearing)
+ D2 (non-configured swarm→nothing). Required for non-equal-weight validator sets
(validators-come-and-go). C1/A-G/finality all green.
|
||
|
zeekay
|
b1e792c0ed |
fix(chains): BootstrapTrust policy — decomplect bootstrap trust from consensus finality to break the mass-recovery deadlock
THE DEADLOCK (mainnet, node v1.30.79): 5 equal-weight validators (each
0.5e18, total 2.5e18, ⅔ floor 1.667e18). The 2 stranded recovery targets
ARE 2 of the 5; with them down only 3 producers connect = 1.5e18 = 60% < ⅔.
The prior bootstrap frontier quorum required ⅔ of CURRENT TOTAL stake to be
CONNECTED before naming a sync frontier — mathematically unsatisfiable during
a mass outage when the down nodes are themselves validators — so no node could
ever recover. Bootstrap trust was braided into consensus finality, and
finality's ⅔ rule cannot be met when ⅓+ of the set is the thing recovering.
THE FIX — a SEPARATE type with a SEPARATE threat model, not a renamed
threshold (chains/bootstrap_trust.go):
- ConsensusQuorum.HasFinality(weight,total): > ⅔ CURRENT stake — UNCHANGED;
present only to NAME the live rule and CONTRAST it.
- BootstrapTrust.AcceptsFrontier(ctx,replies): selects a weak-subjective sync
anchor from authenticated CONFIGURED beacons. NOT a finality oracle.
- BootstrapPolicy: TrustedBeacons (the configured/checkpoint/genesis anchor —
never peer self-report), AgreementThreshold (⅔ of RESPONDERS), MinResponses
(a response FLOOR), MinResponseWeight, Checkpoint (operator override).
Three invariants:
1. NON-CIRCULAR eligibility — only NodeIDs in TrustedBeacons count; peers
never define who is a beacon.
2. A response FLOOR prevents partition-capture — MinResponses (default: a
MAJORITY of the configured set) authenticated beacons must respond; below
it, REJECT unless an operator checkpoint is pinned. For the 5-validator
mainnet (MinResponses=3) the 3 reachable producers recover while a
2-beacon partition is rejected.
3. Acceptance ≠ finality — the named Frontier is "safe to begin sync from",
re-executed during descent; live acceptance stays governed by
ConsensusQuorum alone.
The acceptance gate (FrontierTip) now routes through BootstrapPolicy: the
⅔-of-CURRENT-TOTAL connect gate is GONE; agreement is ⅔ of the RESPONDERS over
a MinResponses floor. The ancestor-tolerant common-ancestor tally is reused
(now a global cross-anchor union tally so a sibling split converges to the
shared committed block — case F), bounded below by MinFrontierHeight (the
node's last-accepted height) so a fork sharing only history BELOW the node
fails safe instead of false-completing at the deep ancestor. Consensus
(v1.25.35) is UNTOUCHED — the FrontierStatus interface is unchanged and its
engine/chain suite stays green.
Tests (chains green; consensus engine/chain green): A mass-recovery success,
B one-beacon capture rejected, C two-beacon partition rejected, D
non-configured peer ignored, E minority-configured forgery rejected (forger
ratifies the real block, never names its forgery), F split reachable ancestry
selects the common ancestor, G finality unchanged (3-of-5 accepts for
bootstrap but is NOT HasFinality). Plus checkpoint override and a load-bearing
fork-at-shared-genesis fail-safe guard for the global tally. The reframed
SubAlpha→SubQuorum test replaces the ⅔-of-total assertion that WAS the bug.
Each guard proven load-bearing by revert: restore the ⅔-of-total gate and A
deadlocks; drop the configured-beacon filter and D captures; drop
MinFrontierHeight and the shared-genesis fork false-completes.
Branch only; not deployed. Re-canary luxd-2.
|
||
|
zeekay
|
cd17d27c62 |
fix(chains): ancestor-tolerant ⅔ frontier quorum — a ±1-block tip split converges to the common committed height (mainnet canary luxd-2)
The exact-tip frontier quorum required ⅔-by-stake of beacons to report the IDENTICAL accepted tip. On a live chain validators are legitimately ±1 block apart at the bleeding edge, so the freshest tip splits and no single id draws ⅔. The mainnet canary (luxd-2): 2 producers had accepted block N, the third producer's un-finalized PENDING block was N+1; neither tip alone cleared ⅔, so FrontierTip returned NoQuorum forever and the healthy stale node failed safe at its stale height instead of converging. The chain was idle, so the split was STABLE and never resolved. FIX (node-only — the stake quorum lives where the stake lives; consensus stays stake-agnostic and just descends from whatever block is named): make the tally ANCESTOR-TOLERANT. A beacon reporting tip N+1 also has N in its accepted chain, so the named frontier is the HIGHEST block a ⅔-by-stake supermajority SHARES (B == their tip OR B is an ancestor of their tip), not the highest proposal. N draws all three producers (300 > 200) and is named; N+1 draws only the lone producer and is not. The node syncs to N (the ⅔-agreed committed height); live consensus catch-up handles N+1. Ancestry is learned by CONTENT (reusing the parent-link descent the sync loop trusts): for each candidate tip, fetch its ancestry, rebuild the parent-linked chain, and cumulatively tally — the highest block clearing the floor across anchors is named. A credit is truthful because a block's parent id is fixed by its content, so a peer cannot fake the linkage below an honestly-reported tip. C1 preserved EXACTLY: a block is named only with > ⅔ of the TOTAL real staked-beacon weight and ≥ required distinct voters; a forged or sub-⅔ tip on a side-fork shares only the honest prefix and can never raise the named block above the genuine ⅔-common height. Only the agreement RELATION changed (exact-tip → in-the-accepted-chain), never the ⅔-by-stake-of-the-real-set requirement. The content-addressed descent, per-height guard, re-execution, weak-subjectivity anchor, F2 bounded-NoQuorum retry, deep-gap fail-safe and single-node completion are all untouched (consensus engine/chain/... unchanged + green). Also: resolve the quorum the instant ALL connected beacons have answered (no exact ⅔), instead of waiting out the full frontier window — so a live-frontier split converges in one round instead of stalling a window each pass. Tests (chains green; consensus engine/chain/... green): TestRED_TipSplitConverges- ToCommonHeight (the canary — converges to N, not the minority N+1, not fail-safe; proven load-bearing: 2-of-3 = 200 is NOT > the 200 floor, so the exact tally cannot name N), TestRED_ForgedHighTipFromMinorityNotNamed (a Byzantine beacon's forged block built ON real N ratifies only N, never names itself — C1), TestNodeBootstrap_ExactQuorumUsesFastPath (unanimous ⅔ still names the exact tip with NO fetch), TestNodeBootstrap_BeaconsSplitNoQuorum (disjoint 3/3 partition, ancestry served, STILL NoQuorum — ancestor tolerance does not manufacture a quorum). NOT deployed — re-canary luxd-2. |
||
|
zeekay
|
f1a247d5ae |
fix(chains): bootstrap beacon connectivity matches the NET, not the chain ID — stale node converges against the staked validator set (mainnet canary)
THE CANARY (luxd-2, node v1.30.77): a healthy STALE node (C-Chain 1082780, gap-17 to producers at 1082797) rolled to the convergence fix logged "bootstrap waiting for beacon connectivity..." x10 then failed SAFE at the connect deadline — "beacon set never reached the connectivity needed to form a ⅔ quorum ... eclipsed/partitioned" — despite connectedPeers=9 and the producers being right there. Correct fail-safe, but it did NOT converge. DIAGNOSIS: the beacon set was ALREADY the stake-weighted validator set, not the --bootstrap-nodes endpoints. For a native chain (C-Chain) manager.go wires beacons = m.Validators (n.vdrs, populated by the bootstrapped P-chain's state.initValidatorSets under PrimaryNetworkID); the --bootstrap-nodes endpoint beacons are a SEPARATE weight-1 set used only as the P-chain's CustomBeacons. FrontierConnecting (not FrontierNoBeacons) confirms m.Validators was non-empty. The real bug was the CONNECTIVITY filter: connectedBeacons admitted a connected beacon only if p.TrackedChains.Contains(b.chainID). But a peer advertises the NETS it tracks (network/peer/peer.go adds constants.PrimaryNetworkID + every tracked L1 net ID to peer.Info.TrackedChains) — it NEVER advertises an individual native chain ID. So the filter matched ZERO real peers, connectedStake stayed 0, the ⅔ floor was never reachable, and FrontierTip reported FrontierConnecting until the deadline. The tests passed only because the mock modeled TrackedChains as set.Of(chainID) — agreeing with the bug, not reality. FIX (node-only; consensus FrontierStatus logic unchanged and correct): - connectedBeacons filters on b.networkID (the NET the beacon set is anchored to — PrimaryNetworkID for native chains, the L1 net ID for sovereign chains), the value real peers actually advertise. C1 PRESERVED: a peer is still admitted ONLY if it is in the staked set (weights); the ⅔-by-stake threshold, the TOTAL-stake denominator, and the content-addressed descent are untouched. A non-staked peer that tracks the network is still ignored. - EDGE (4): expectsStakedBeacons (true for a native non-platform chain on a sybil-protected network) makes an EMPTY staked set report FrontierConnecting (wait → ConnectDeadline → fail safe), never FrontierNoBeacons (false-complete) and never endpoint trust. Distinguishes "validator set not yet loaded → wait → converge" from "genuinely empty → fail safe". The P-chain (CustomBeacons may be empty under endpoint-only --bootstrap-nodes) and single-node keep "empty ⇒ nothing to sync to". TESTS (chains): - TestRED_PeersTrackNetNotChain_StaleNodeConverges: 3 producers holding > ⅔ stake, connected + advertising the NET (not the chain), name the frontier and the stale node converges. Proven load-bearing: reverting the filter to b.chainID makes connectedBeacons return nil and this test fails exactly as the canary did. Mock now models TrackedChains = the NET (matches peer.go). - TestRED_EmptyStakedSetFailsSafe: empty staked set + expectsStakedBeacons → Connecting → ErrBeaconsUnreachable, no endpoint trust; the flag is the proven discriminator vs FrontierNoBeacons. - C1 (ForgedFrontierFromNonBeacon / HonestBeaconQuorumIgnoresMalicious / SubAlphaStakeCannotName), F2/NoQuorum, deep-gap, weak-subjectivity, single-node all green; consensus engine/chain/... unchanged and green. NOT deployed — re-canary luxd-2. |
||
|
zeekay
|
590123c34f |
fix(chains): surface a bootstrap fail-safe return promptly — stop the chain when Run gives up, not 5 min later (F5)
When the initial-sync loop RETURNS a fail-safe error (ErrBeaconsUnreachable / ErrNoBeaconQuorum / ErrGapTooLarge), runBootstrapThenPoll exited leaving the chain in a DEAD WINDOW: neither bootstrapping (the loop is gone) nor stopped (monitorBootstrap only stops the engine at the +5-min no-progress watchdog). The operator saw a chain that had already given up but was not marked failed for up to 5 minutes. runBootstrapThenPoll now records the fail-safe reason (bootstrapFailed atomic + BootstrapFailed/BootstrapFailure accessors). watchBootstrapProgress takes a failed() predicate and returns the new bootstrapFailed outcome the next tick; monitorBootstrap stops the engine + registers a failing health check IMMEDIATELY, carrying the real reason (eclipsed/partitioned/too-far-behind) rather than the generic no-progress timeout. The stop+health logic is DRY'd into one m.failBootstrapChain shared by the stall and fail-safe outcomes. Combined with the consensus-side bounded NoQuorum retry (F2), a transient live-frontier split converges and a genuine eclipse fails fast and visibly — making the mainnet 5/5 roll deterministic. Tests: TestWatchBootstrapProgress_FailSafeSurfacesPromptly (failed -> bootstrapFailed next tick, not after the window), TestWatchBootstrapProgress_ReadyBeatsFailed (success never masked), TestBootstrapFailure_AccessorSurfacesReason (driver<->monitor plumbing). Existing watchdog + node bootstrap (C1 forged-frontier finalizes ZERO, stale-converge, split-no-quorum) all still green. NOTE: build the combined tree with a temp go.mod replace github.com/luxfi/consensus=../consensus (consensus f93162d15); not committed. |
||
|
zeekay
|
da8f57e6e9 |
fix(chains): FrontierTip reports beacon-connectivity status — wait for beacons before concluding caught-up (mainnet canary)
Node side of the stale-node convergence fix. blockHandler.FrontierTip now returns a chainbootstrap.FrontierStatus instead of a bare ok bool, decomplecting the THREE reasons a frontier may not be named so the bootstrap loop can WAIT for beacon connectivity instead of false-completing at the local stale height: - no beacon set configured -> FrontierNoBeacons (single-node: complete) - beacons configured, < ⅔ stake up -> FrontierConnecting (the boot race: WAIT) - ⅔ stake connected, no agreement -> FrontierNoQuorum (eclipse: fail safe) - ⅔-by-stake quorum agrees on a tip -> FrontierNamed (C1 gate: sync) The discriminator is the SAME ⅔ floor (config.TwoThirdsStakeFloor over the FULL beacon stake) used by the quorum tally: if even unanimous CONNECTED beacon stake could not clear the floor (or too few distinct beacons are up), connectivity is still coming up -> FrontierConnecting (the canary boot race, where FrontierTip ran before any beacon handshaked). Only once enough beacon stake is connected do we query + tally; a split that clears none -> FrontierNoQuorum. The C1 forged-chain gate is UNCHANGED: the query + weighted ⅔-stake tally is extracted verbatim into queryFrontierQuorum (a non-beacon peer, a single peer, or any sub-⅔ set can still NEVER reach FrontierNamed). connectedBeaconsTrackingChain is replaced by the smaller connectedBeacons helper (shared with sampleAncestorBeacons — DRY). monitorBootstrap/watchBootstrapProgress need no change: they already gate the chain-ready signal on BootstrapComplete (Run success), which now returns only on genuine caught-up. Tests: reproduces the canary over the REAL GetAcceptedFrontier/GetAncestors transport (TestNodeBootstrap_StaleNodeWaitsForBeaconConnect — beacons connect after a delay; the stale node WAITS through the connecting passes, then names the ⅔ frontier and converges, never false-completing at the stale height); the forged-frontier-from-non-beacon case now FAILS SAFE (ErrBeaconsUnreachable) instead of false-completing while still finalizing ZERO forged blocks; split-beacons report FrontierNoQuorum; a no-beacon-set node reports FrontierNoBeacons (single-node completes). All existing C1 quorum, sub-⅔, invalid-halt, framing, deliver-gating, and H2 watchdog tests stay green. Dev integration: node go.mod stays pinned consensus v1.25.34 (NO committed replace); build/test against the consensus fix branch via `GOWORK=off go mod edit -replace github.com/luxfi/consensus=../consensus` (the go.work path is blocked by an unrelated ./compliance go.sum mismatch). Tag consensus + bump node go.mod at the coordinated merge. |
||
|
zeekay
|
06487c67c9 |
fix(chains): beacon + α-weighted-stake quorum names the bootstrap frontier — close C1 forged-chain naming
THE VULN (red PoC): an empty/behind node discovered the network frontier from
samplePeerTrackingChain = net.PeerInfo(nil) (ALL connected peers, filtered only by
self-reported TrackedChains — no stake/beacon filter) and FrontierTip returned the
FIRST reply (no quorum). An eclipse/malicious peer (no validator keys needed) named a
forged-but-Verify-passing frontier from genesis; the node fetched+executed it and
finalized the forged chain cert-lessly (proposervm disabled => Verify is just the EVM
state transition), bricking against the real chain. red finalized 40 forged blocks.
THE FIX — the frontier is named ONLY by a 2/3-BY-STAKE quorum of the configured BEACONS:
- beacons (validators.Manager) is wired into the blockHandler (was computed in buildChain
then discarded as unused). For native chains it is the primary-network validator set
under networkID = PrimaryNetworkID.
- FrontierTip queries ONLY connected beacons (PeerInfo is beacon-restricted), tallies
their accepted-tip replies WEIGHTED by stake (replies now carry the responder nodeID),
and returns a tip only once its agreeing stake clears the shared live floor
(config.TwoThirdsStakeFloor over the FULL beacon stake) with >=2 distinct beacons. A
non-beacon peer, a single peer, or any sub-2/3-stake set can NEVER name the frontier =>
fail-closed (the loop has nothing to sync to). Mirrors VerifyWeighted exactly, so the
bootstrap anchor can never drift from live finality.
- Optional weak-subjectivity checkpoint (wsCheckpoint{ID,Height}) wired into the loop.
- M1: Ancestors fetches from a ROTATED beacon sample (bsRotor) so no beacon monopolizes
the descent; safety is independent of which beacon serves (content-addressed in
consensus).
- H2: monitorBootstrap is now PROGRESS-BASED — the once-set 5-minute hard timer that
killed a healthy-but-slow sync is replaced by a no-progress stall window (reset on
height advance via BootstrapHeight()). Decomplected into watchBootstrapProgress (pure,
unit-tested).
Tests: TestRED_ForgedFrontierFromNonBeaconRejected (headline — 0 forged finalized, was
40), honest-beacon-quorum-ignores-malicious, sub-2/3-stake-cannot-name, progress-watchdog
(slow-but-advancing NOT killed / genuine stall fails). Existing transport + framing +
gating tests stay green. (consensus side: content-addressed descent + M2 + WS + M3.)
Convention: node go.mod stays pinned consensus v1.25.32; built via go.work in dev,
consensus tagged + go.mod bumped at the coordinated merge (same as
|
||
|
zeekay
|
58f1e04928 |
fix(chains): drive REAL initial sync — empty/behind C-Chain node fetches+executes to the frontier
The node declared a chain 'bootstrapped' the instant the engine started, at
whatever LOCAL last-accepted height it had: monitorBootstrap polled
engine.IsBootstrapped() which Start() sets true immediately (genesis for an empty
DB, the partial height for a behind node). No block-fetch sync ever ran, so an
empty luxd-0 stuck at C-Chain height 0 ('finished bootstrapping pollCount=1',
fetched nothing) and a behind node stuck at its partial height while the producers
held the full chain.
Wire the consensus bootstrap loop (engine/chain/bootstrap) over the EXISTING node
transport. The blockHandler becomes the loop's BlockSource (fetch) AND Chain
(execute), in a new file bootstrap_sync.go (compile-asserted against both
interfaces):
- FrontierTip / Ancestors: synchronous adapters over GetAcceptedFrontier /
GetAncestors, correlating the async replies through bsFrontierCh / bsAncestorCh.
- ParseBlock / LastAccepted / Has / AcceptBootstrapBlock: the execute sink
(AcceptBootstrapBlock delegates to the engine's frontier-trust accept authority).
- runBootstrapThenPoll: runs the loop to the frontier, ENDS the engine's bootstrap
phase (FinishBootstrap — only the α-of-K cert-gate finalizes thereafter), flips
bootstrapDone, THEN starts the live frontier poller. On stall it leaves
bootstrapDone false so monitorBootstrap surfaces a real failure (never masks).
manager.go (minimal hooks): AcceptedFrontier + handleContext route replies to the
loop while bsActive (else the live cert/vote path is unchanged); buildChain starts
runBootstrapThenPoll instead of the bare poller; monitorBootstrap GATES the chain on
the handler's real BootstrapComplete() (initial sync reached the frontier), not the
engine's premature 'am I started' flag.
Tests (node, via the go.work workspace -> local consensus): an EMPTY node converges
genesis->50 over the real GetAcceptedFrontier/GetAncestors path; an invalid block
HALTS sync at the block below it; framing round-trip; reply gating. Existing chains +
op-table suites stay green.
Depends on the consensus feat/chain-bootstrap-fetch-execute branch (new
AcceptBootstrapBlock / FinishBootstrap / bootstrap pkg). Integrated for dev via
go.work (use ./consensus); DEPLOY = tag consensus (next patch) + bump this require +
keep go.mod pinned (no committed local replace).
|