mirror of
https://github.com/luxfi/node.git
synced 2026-07-27 03:39:39 +00:00
chains: skip-bootstrap must not disable peer-sync on a staked network (#66/#74)
Durable root-cause fix for the behind-validator rejoin wedge (mainnet luxd-0:
C-Chain frozen ~40h at a stale height, only a manual chaindata wipe unstuck it).
Root cause: buildChain computed the bootstrap frontier-sync discriminator as
`expectsStakedBeacons := !m.SkipBootstrap && native && !platform` AND emptied the
beacon set whenever `m.SkipBootstrap`. Production validators hardcode
--skip-bootstrap=true (to skip the initial bootstrap WAIT), so a real multi-
validator native chain (C/X/Q) got expectsStakedBeacons=false + EMPTY beacons.
FrontierTip then reported FrontierNoBeacons ("nothing to sync to"), the node named
its STALE local last-accepted the network frontier, transitioned the VM to normal
operation there, and never fetched the gap from its 4 healthy peers — the wedge
survived every restart because --skip-bootstrap is persistent config. The entire
peer-sync/self-heal machinery (bootstrap_sync.go) was dead code under skip-bootstrap.
Fix: drive the discriminator from SYBIL PROTECTION (the true "real staked network"
signal, already wired into ManagerConfig), not --skip-bootstrap. A sybil-protected
native non-platform chain now keeps its staked beacon set and expectsStakedBeacons
even under --skip-bootstrap, so a behind validator always catches up from peers.
A genuine single-node / dev net runs sybil-protection OFF and still takes the
empty-beacon immediate-start path. A single-VALIDATOR staked net (self-only set) is
handled by a new FrontierTip hasExternalBeacons rule (placed after the P-ready gate),
so it immediate-starts without a Connecting hang while a >=2 set runs the quorum.
This matches the proven live remediation (flipping skip-bootstrap=false via the
.allow-bootstrap marker caught luxd-0 up to tip in ~90s) and preserves every RED
safety invariant (empty staked set still fails safe — TestRED_EmptyStakedSetFailsSafe).
Tests: TestChainExpectsStakedBeacons_SybilDrivesNotSkipBootstrap,
TestNodeBootstrap_SelfOnlyStakedSet_ReportsNoBeacons, and the headline
TestNodeBootstrap_BehindValidator_StakedSet_CatchesUpNoWipe (N-blocks-behind →
restart → catches up from peers to tip, no wipe). Full chains suite green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This commit is contained in:
zeekay
@@ -201,6 +201,19 @@ func (b *blockHandler) fullyConnectedBeacons(weights map[ids.NodeID]uint64, conn
|
||||
return external > 0 && len(connected) >= external
|
||||
}
|
||||
|
||||
// hasExternalBeacons reports whether the staked beacon set contains any validator OTHER than this
|
||||
// node. False for a self-only (single-validator) set — there is then no peer to sync from, so the
|
||||
// node's own tip IS the frontier (FrontierTip returns FrontierNoBeacons). The set is read from
|
||||
// P-chain STATE, not connectivity, so this is a TOPOLOGY fact (a genuine single-validator net),
|
||||
// never an eclipse artifact (an eclipse drops peers from `connected`, never from `weights`).
|
||||
func (b *blockHandler) hasExternalBeacons(weights map[ids.NodeID]uint64) bool {
|
||||
external := len(weights)
|
||||
if _, selfIsBeacon := weights[b.selfNodeID]; selfIsBeacon {
|
||||
external--
|
||||
}
|
||||
return external > 0
|
||||
}
|
||||
|
||||
// withSelfVote returns `replies` plus the node's OWN accepted frontier as a beacon reply — the
|
||||
// SELF-VOTE. The node is itself a beacon (selfNodeID in `weights`, the trust anchor) and knows its
|
||||
// own accepted tip (lastID) with certainty, so it vouches for it exactly as a connected peer's reply
|
||||
@@ -311,6 +324,21 @@ func (b *blockHandler) FrontierTip(ctx context.Context) (ids.ID, chainbootstrap.
|
||||
return ids.Empty, chainbootstrap.FrontierConnecting
|
||||
}
|
||||
|
||||
// SELF-ONLY staked set: the configured validator set is exactly THIS node — a genuine
|
||||
// single-validator network (a sybil-protected devnet, or the sole validator of an L1). There
|
||||
// is no OTHER beacon to sync from, so the node's own accepted tip IS the network frontier →
|
||||
// immediate start (FrontierNoBeacons). This is NOT an eclipse: the staked set is read from
|
||||
// P-chain STATE, not connectivity, so a hidden peer would still appear in `weights`; only a
|
||||
// genuine single-validator topology reaches here. Placed AFTER the P-ready gate so a boot-race
|
||||
// partial set (transiently self-only mid-P-replay) WAITS above rather than false-completing here.
|
||||
// This is the sybil-protected single-validator counterpart to the empty-set FrontierNoBeacons
|
||||
// path: a multi-validator staked set (the ≥2 case) still runs the ⅔-by-stake quorum below.
|
||||
if !b.hasExternalBeacons(weights) {
|
||||
b.logger.Debug("bootstrap frontier: self-only staked set (single validator) — NoBeacons, nothing to sync to",
|
||||
log.Stringer("chainID", b.chainID))
|
||||
return ids.Empty, chainbootstrap.FrontierNoBeacons
|
||||
}
|
||||
|
||||
// THE MASS-RECOVERY FIX. The acceptance decision is the BootstrapPolicy (a SEPARATE object
|
||||
// with a SEPARATE threat model — bootstrap_trust.go), NOT the ⅔-of-CURRENT-total-stake
|
||||
// consensus floor. The prior code required a ⅔-by-stake quorum of the WHOLE validator set to
|
||||
|
||||
@@ -2032,3 +2032,96 @@ func TestRED_BehindNode_PeerConnectDelay_NeverFalseCompletesAtStale(t *testing.T
|
||||
require.Greater(t, net.peerInfoCalls, net.connectAfterCalls,
|
||||
"the loop must have WAITED through the connecting passes before naming the frontier")
|
||||
}
|
||||
|
||||
// TestChainExpectsStakedBeacons_SybilDrivesNotSkipBootstrap pins the ROOT-CAUSE decision of the
|
||||
// rejoin wedge (tasks #66/#74): whether a chain syncs its bootstrap frontier against the STAKED
|
||||
// primary-network set is driven by SYBIL PROTECTION, NOT --skip-bootstrap. Production validators
|
||||
// set --skip-bootstrap (to skip the initial bootstrap WAIT); the prior `!m.SkipBootstrap && ...`
|
||||
// made that flag also EMPTY the beacon set, so a behind native chain (C/X/Q) reported
|
||||
// FrontierNoBeacons, named its STALE local tip the network frontier, went live there, and never
|
||||
// caught up across restarts until a manual chaindata wipe. skip-bootstrap is not even an input to
|
||||
// the fixed decision — that is the point.
|
||||
func TestChainExpectsStakedBeacons_SybilDrivesNotSkipBootstrap(t *testing.T) {
|
||||
require.True(t, chainExpectsStakedBeacons(true, true, false),
|
||||
"sybil-protected native non-platform chain (C/X/Q) MUST expect staked beacons → peer-sync a behind validator (regardless of skip-bootstrap)")
|
||||
require.False(t, chainExpectsStakedBeacons(false, true, false),
|
||||
"non-sybil (dev / single-node) native chain → NO staked beacons: the empty-beacon immediate-start path")
|
||||
require.False(t, chainExpectsStakedBeacons(true, false, false),
|
||||
"a non-native chain does not sync against the primary staked set here")
|
||||
require.False(t, chainExpectsStakedBeacons(true, true, true),
|
||||
"the platform chain anchors to its OWN CustomBeacons, never the staked-set frontier quorum")
|
||||
}
|
||||
|
||||
// TestNodeBootstrap_SelfOnlyStakedSet_ReportsNoBeacons covers the single-VALIDATOR counterpart of
|
||||
// the fix: a sybil-protected chain whose staked set is EXACTLY this node (a single-validator
|
||||
// devnet, or the sole validator of an L1) has no OTHER beacon to sync from, so FrontierTip reports
|
||||
// FrontierNoBeacons (immediate start at the node's own tip) rather than hanging in FrontierConnecting.
|
||||
// This is what lets skip-bootstrap stop emptying native beacons WITHOUT bricking a single-validator
|
||||
// net. It is NOT an eclipse: the staked set is read from P-chain STATE (hasExternalBeacons), so a
|
||||
// hidden peer would still appear in `weights`.
|
||||
func TestNodeBootstrap_SelfOnlyStakedSet_ReportsNoBeacons(t *testing.T) {
|
||||
const N = 10
|
||||
chain, byID := buildBSChain(N, -1)
|
||||
vm := newBSVMAt(chain, N) // the sole validator IS at the frontier
|
||||
self := ids.GenerateTestNodeID()
|
||||
bh, chainID := newBSHandlerWeighted(t, vm, map[ids.NodeID]uint64{self: 100})
|
||||
bh.selfNodeID = self // the staked set is EXACTLY this node — a genuine single-validator net
|
||||
bh.net = &bsBeaconNet{bh: bh, chainID: chainID, byID: byID, tip: chain[N]}
|
||||
bh.msgCreator = bsMsgBuilder{}
|
||||
|
||||
bh.bsActive.Store(true)
|
||||
tip, status := bh.FrontierTip(context.Background())
|
||||
bh.bsActive.Store(false)
|
||||
require.Equal(t, chainbootstrap.FrontierNoBeacons, status,
|
||||
"self-only staked set (single validator) → NoBeacons (nothing to sync to), never a FrontierConnecting hang")
|
||||
require.Equal(t, ids.Empty, tip)
|
||||
|
||||
// Discriminator: add ONE other validator and the SAME node must now run the quorum (has a peer
|
||||
// to sync from) — proving the self-only shortcut fires ONLY for a genuinely alone validator.
|
||||
require.True(t, bh.hasExternalBeacons(map[ids.NodeID]uint64{self: 100, ids.GenerateTestNodeID(): 100}),
|
||||
"a ≥2-validator staked set has an external beacon → runs the ⅔-by-stake quorum, not the self-only shortcut")
|
||||
}
|
||||
|
||||
// TestNodeBootstrap_BehindValidator_StakedSet_CatchesUpNoWipe is THE REJOIN INVARIANT (tasks
|
||||
// #66/#74), the code reproduction of the mainnet luxd-0 wedge: a staked validator that fell behind
|
||||
// must sync from its peers on restart and reach the network frontier WITHOUT a chaindata wipe.
|
||||
//
|
||||
// The node reloads its STALE on-disk tip at height M (the "restart" — no wipe) and holds a
|
||||
// sybil-protected native-chain staked beacon set (expectsStakedBeacons=true — exactly what
|
||||
// buildChain now leaves in place under --skip-bootstrap, instead of emptying it). The 4 healthy
|
||||
// producers name+serve the frontier N over the real GetAcceptedFrontier/GetAncestors transport, so
|
||||
// the behind node fetch+executes M+1..N and ends ACCEPTED at N — never stuck at the stale M.
|
||||
func TestNodeBootstrap_BehindValidator_StakedSet_CatchesUpNoWipe(t *testing.T) {
|
||||
const N = 60 // network frontier (the healthy producers)
|
||||
const M = 42 // our STALE local height after a restart — behind by N-M, NO wipe
|
||||
chain, byID := buildBSChain(N, -1)
|
||||
vm := newBSVMAt(chain, M) // reload the stale on-disk tip (the restart)
|
||||
|
||||
// A 5-validator staked set (equal stake): this node is one, the other 4 are the connected,
|
||||
// serving producers at the frontier N. This mirrors lux-mainnet's 5-validator C-Chain.
|
||||
weights := map[ids.NodeID]uint64{}
|
||||
producers := make([]ids.NodeID, 4)
|
||||
for i := range producers {
|
||||
producers[i] = ids.GenerateTestNodeID()
|
||||
weights[producers[i]] = 100
|
||||
}
|
||||
self := ids.GenerateTestNodeID()
|
||||
weights[self] = 100
|
||||
bh, chainID := newBSHandlerWeighted(t, vm, weights)
|
||||
bh.selfNodeID = self
|
||||
require.True(t, bh.expectsStakedBeacons,
|
||||
"a native sybil chain expects staked beacons even under skip-bootstrap — the fix that keeps peer-sync alive")
|
||||
|
||||
bh.net = &bsBeaconNet{bh: bh, chainID: chainID, connected: producers, byID: byID, tip: chain[N], serveAncestors: true}
|
||||
bh.msgCreator = bsMsgBuilder{}
|
||||
|
||||
ctx := context.Background()
|
||||
require.NoError(t, runBS(t, bh),
|
||||
"behind validator must converge to the frontier from its peers — no wipe, chain keeps finalizing")
|
||||
|
||||
last, _ := vm.LastAccepted(ctx)
|
||||
require.Equal(t, chain[N].id, last,
|
||||
"REJOIN: behind validator caught up from peers to frontier N=%d (was stuck at stale M=%d, no wipe)", N, M)
|
||||
require.True(t, bh.Accepted(ctx, chain[N].id),
|
||||
"node holds + ACCEPTED the frontier tip after the rejoin (genuine catch-up, not a stale false-complete)")
|
||||
}
|
||||
|
||||
+34
-9
@@ -1109,18 +1109,21 @@ func (m *manager) buildChain(chainParams ChainParameters, sb nets.Net) (*chainIn
|
||||
}
|
||||
|
||||
// expectsStakedBeacons gates the EMPTY-beacon-set behavior of the bootstrap frontier
|
||||
// quorum (see blockHandler.expectsStakedBeacons). True ONLY for a native NON-platform
|
||||
// chain (C/X/Q/...) on a sybil-protected network — those sync against the STAKED
|
||||
// quorum (see blockHandler.expectsStakedBeacons). True for a native NON-platform chain
|
||||
// (C/X/Q/...) on a SYBIL-PROTECTED (real staked) network — those sync against the STAKED
|
||||
// primary-network validator set (m.Validators, populated by the already-bootstrapped
|
||||
// P-chain), so an empty set means "not yet loaded / misconfig → wait then fail safe",
|
||||
// NOT "single-node → done". The P-chain (CustomBeacons may be empty under endpoint-only
|
||||
// --bootstrap-nodes) and skip-bootstrap keep the "empty ⇒ nothing to sync to" behavior.
|
||||
// Computed BEFORE the skip-bootstrap override so it is false in single-node mode.
|
||||
expectsStakedBeacons := !m.SkipBootstrap && ids.IsNativeChain(chainParams.ID) && !isPlatformChain
|
||||
// NOT "single-node → done". Driven by sybil-protection, NOT --skip-bootstrap: a production
|
||||
// validator sets skip-bootstrap and that must NOT make it masquerade as single-node and
|
||||
// wedge behind at its stale local tip (tasks #66/#74). See chainExpectsStakedBeacons.
|
||||
expectsStakedBeacons := chainExpectsStakedBeacons(m.SybilProtectionEnabled, ids.IsNativeChain(chainParams.ID), isPlatformChain)
|
||||
|
||||
// In skip-bootstrap mode, use empty beacons for all chains
|
||||
// This enables single-node development mode
|
||||
if m.SkipBootstrap {
|
||||
// skip-bootstrap forces empty beacons (single-node immediate-start) for every chain EXCEPT
|
||||
// one that syncs against the staked set — those MUST always catch a behind validator up from
|
||||
// its peers, so emptying their beacons (the prior UNCONDITIONAL override) is precisely the
|
||||
// rejoin wedge this fixes. A genuine single-node / dev net runs sybil-protection OFF, so its
|
||||
// native chains still fall here and get the empty-beacon immediate-start path.
|
||||
if m.SkipBootstrap && !expectsStakedBeacons {
|
||||
beacons = &emptyValidatorManager{}
|
||||
m.Log.Info("skip-bootstrap enabled - using empty beacons for single-node mode")
|
||||
}
|
||||
@@ -2654,6 +2657,28 @@ func (m *manager) getOrMakeVMGatherer(vmID ids.ID) (metrics.MultiGatherer, error
|
||||
return vmGatherer, nil
|
||||
}
|
||||
|
||||
// chainExpectsStakedBeacons reports whether a chain must sync its bootstrap frontier against the
|
||||
// STAKED primary-network validator set — the ⅔-by-stake quorum that names the network frontier
|
||||
// (blockHandler.expectsStakedBeacons, the C1 forged-chain gate). When true, --skip-bootstrap does
|
||||
// NOT empty the chain's beacon set (buildChain below), so a behind validator always catches up
|
||||
// from its peers.
|
||||
//
|
||||
// It is driven by SYBIL PROTECTION — the true "this is a real staked network" signal — and NOT by
|
||||
// --skip-bootstrap. Production validators set --skip-bootstrap to skip the initial bootstrap WAIT,
|
||||
// but that flag must NEVER disable peer-sync on a staked network. Keying this decision off
|
||||
// --skip-bootstrap (the prior `!m.SkipBootstrap && ...`) is exactly what wedged a behind native
|
||||
// chain (C/X/Q...): with skip-bootstrap set, the chain got expectsStakedBeacons=false + empty
|
||||
// beacons, so FrontierTip reported FrontierNoBeacons ("nothing to sync to"), the node named its
|
||||
// STALE local last-accepted the network frontier, went live there, and never caught up across
|
||||
// restarts until a manual chaindata wipe (tasks #66/#74). A genuine single-node / dev network runs
|
||||
// sybil-protection OFF, so it still takes the empty-beacon immediate-start path. The platform chain
|
||||
// anchors to its own CustomBeacons (not the staked set), so it is excluded here as before; a
|
||||
// single-VALIDATOR staked net (self-only set) is handled downstream by FrontierTip's
|
||||
// hasExternalBeacons rule, which reads the LOADED set.
|
||||
func chainExpectsStakedBeacons(sybilProtectionEnabled, isNativeChain, isPlatformChain bool) bool {
|
||||
return sybilProtectionEnabled && isNativeChain && !isPlatformChain
|
||||
}
|
||||
|
||||
// emptyValidatorManager implements validators.Manager with no validators
|
||||
type emptyValidatorManager struct{}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user