fix(chains): gate VM normal-op transition on initial sync reaching the frontier

Root cause of the restart-freeze (mainnet luxd-2: C-Chain stuck at 1082780 while producers finalized 1082796): buildChain called SetState(vm.Ready) UNCONDITIONALLY right after Initialize — at the LOCAL last-accepted height — and ran the node-layer initial sync (runBootstrapThenPoll) afterward as a detached, non-gating goroutine. vm.Ready fires the EVM's onNormalOperationsStarted (block building, mempool gossip, validator dispatch); a restarted STALE validator therefore went live at its stale height and never converged to the finalized frontier. The proper vm.Bootstrapping state (the EVM's onBootstrapStarted: snapshots only, no building) was skipped entirely. Fix — restore the correct VM lifecycle, gated: - buildChain now SetState(vm.Bootstrapping) after init (not Ready). The VM fetch+executes the gap in Bootstrapping, serving nothing as head. - The Ready transition moves into the bootstrap goroutine (runInitialSync, split out of runBootstrapThenPoll for unit-testability) and fires ONLY after bs.Run reaches the named ⅔-by-stake beacon frontier — VM live + engine cert-gate (FinishBootstrap) go together at the frontier, never at a stale height. Wired via blockHandler.vmReady (captures the VM's SetState). - Fresh-genesis / single-node (FrontierNoBeacons) and at-the-tip restart (caught-up) go Ready promptly — no regression, no hang. - Eclipse / isolation: bs.Run's bounded ConnectDeadline is the retry that self-heals when beacons return; past it the node fails SAFE (stays Bootstrapping, records the reason for monitorBootstrap) — never serves stale state, never an unbounded hang. The orchestrator restart now converges. C1 forged-chain defense, VerifyWeighted, and the ⅔-stake quorum cert are untouched — the frontier is still named only by the configured-beacon ⅔-of-responders quorum + content-addressed descent (all RED/BootstrapTrust tests pass). AcceptedFrontier/Ancestors replies route to the bootstrap channels via bsActive throughout the phase (verified). Tests (chains): TestRED_StaleValidatorGoesReadyOnlyAfterReachingFrontier reproduces the production scenario — RED before (VM goes live at stale N=30), GREEN after (only at frontier N+K=46). Plus fresh-genesis no-regression, bounded eclipse fail-safe, and at-the-tip producer fast-Ready. Race-clean. NOTE (flagged for review, NOT in this change): the createDAG path (X/Q DAG chains, consensusdag.Engine) has the same unconditional SetState(Ready) at manager.go:1792 but a different engine without the node-layer bootstrap-sync wiring — a separate fix if any production chain uses it.
2026-07-27 03:39:39 +00:00 · 2026-06-27 02:47:48 -07:00
parent c1035aada9
commit b313912413
3 changed files with 318 additions and 31 deletions
@@ -548,23 +548,60 @@ func (b *blockHandler) deliverBootstrapAncestors(requestID uint32, data []byte)
 	return true
 }

-// runBootstrapThenPoll is the chain's startup sync driver. It (1) runs the
-// fetch+execute bootstrap loop to converge to the network frontier, (2) on success
-// ENDS the engine's bootstrap phase (FinishBootstrap — only the α-of-K cert-gate
-// finalizes thereafter) and flips bootstrapDone (so monitorBootstrap marks the chain
-// ready), then (3) hands off to the live frontier poller (runtime catch-up via the
-// cert path). On bootstrap failure it leaves bootstrapDone false so monitorBootstrap
-// surfaces a real failure (timeout) rather than masking it. Node-lifetime goroutine
-// (started in buildChain); exits when ctx is done (shutdown).
+// runBootstrapThenPoll is the chain's startup sync driver (the goroutine buildChain
+// launches). It runs INITIAL SYNC to the network frontier with the VM's normal-operation
+// transition GATED on that completion (runInitialSync), then — ONLY if the chain actually
+// went live — hands off to the live frontier poller (runtime cert-carry catch-up). On
+// fail-safe runInitialSync returns false and the poller is NOT started: the chain stays
+// not-ready for monitorBootstrap to surface, and the VM stays in Bootstrapping (it never
+// serves a stale height). Exits when ctx is done (shutdown). The gating logic lives in
+// runInitialSync so it is unit-testable without the blocking poller hand-off.
 func (b *blockHandler) runBootstrapThenPoll(ctx context.Context) {
+	if b.runInitialSync(ctx) {
+		b.runFrontierPoller(ctx)
+	}
+}
+
+// runInitialSync drives the fetch+execute bootstrap loop to the network frontier and, ON
+// SUCCESS, transitions the VM to normal operation (transitionVMReady → vm.Ready), ENDS the
+// engine's bootstrap phase (FinishBootstrap — only the α-of-K cert-gate finalizes
+// thereafter), and flips bootstrapDone — IN THAT ORDER, so "VM serves / builds" and "engine
+// cert-gates finality" go live TOGETHER at the named frontier. Returns true iff the chain
+// went live.
+//
+// THE ORDERING IS THE FIX. Previously buildChain put the VM into normal operation
+// UNCONDITIONALLY right after Initialize — at the LOCAL last-accepted height — and ran this
+// sync afterward as a detached, non-gating goroutine. A restarted STALE validator therefore
+// went live (block building, mempool, validator dispatch via the EVM's
+// onNormalOperationsStarted) at its stale height and never converged to the finalized
+// frontier. Gating the normal-op transition on reaching the frontier makes "VM live at a
+// stale height" structurally impossible: the VM is in Bootstrapping until it has converged.
+//
+// FAIL-SAFE (eclipse / isolated node). bs.Run is internally BOUNDED: it WAITS for beacon
+// connectivity (re-sampling the beacon set every RetryInterval up to ConnectDeadline) — the
+// bounded retry that self-heals the instant the beacons return — then returns rather than
+// hanging. If the deadline elapses with no ⅔-by-stake beacon quorum reachable (genuine
+// eclipse / partition / deep gap), runInitialSync returns false WITHOUT going Ready: the VM
+// stays in Bootstrapping (it serves nothing as head), and bootstrapFailed records the reason
+// so monitorBootstrap stops the chain and surfaces it. The orchestrator restarts the node,
+// which re-syncs and converges (nothing pins it live at the stale height anymore). The node
+// NEVER false-completes at its local stale height.
+func (b *blockHandler) runInitialSync(ctx context.Context) bool {
 	if b.engine == nil || b.net == nil || b.msgCreator == nil {
-		// No transport/engine to drive sync — nothing to bootstrap; treat as ready so a
-		// degenerate handler does not pin the chain unbootstrapped.
+		// Degenerate handler (no transport/engine to drive sync): nothing to converge to.
+		// Go live immediately so a single-node / transport-less chain is not pinned
+		// unbootstrapped — the same fast-path the no-beacon-set case takes.
 		if b.engine != nil {
 			b.engine.FinishBootstrap()
 		}
+		if err := b.transitionVMReady(); err != nil {
+			b.logger.Error("degenerate chain: VM SetState(Ready) failed — NOT marking bootstrapped",
+				log.Stringer("chainID", b.chainID), log.Err(err))
+			b.bootstrapFailed.Store(&bsFailure{err: err})
+			return false
+		}
 		b.bootstrapDone.Store(true)
-		return
+		return true
 	}

 	b.bsActive.Store(true)
@@ -572,6 +609,11 @@ func (b *blockHandler) runBootstrapThenPoll(ctx context.Context) {
 		Source: b,
 		Chain:  b,
 		Log:    b.logger,
+		// Bounded beacon-connect WAIT + re-sample pause (zero ⇒ library defaults 3m / 1s).
+		// ConnectDeadline is the bounded retry: bs.Run re-samples the beacon set every
+		// RetryInterval up to this deadline, converging the instant the beacons return.
+		ConnectDeadline: b.bootstrapConnectDeadline,
+		RetryInterval:   b.bootstrapRetryInterval,
 		// Optional operator-pinned weak-subjectivity anchor: the α-agreed frontier must
 		// descend from this id at this height (defense-in-depth for empty-genesis). Zero
 		// ⇒ disabled (the beacon + ⅔-stake quorum is the primary anchor).
@@ -583,27 +625,52 @@ func (b *blockHandler) runBootstrapThenPoll(ctx context.Context) {

 	if err != nil {
 		if ctx.Err() != nil {
-			return // shutdown — not a bootstrap failure
+			return false // shutdown — not a bootstrap failure
 		}
-		// Initial sync did not complete (eclipse / partition / gap-too-large). Do NOT mark the
-		// chain ready. Record the fail-safe reason so monitorBootstrap surfaces it PROMPTLY (F5)
-		// — stopping the chain the moment Run returns instead of leaving it in the dead window
-		// until the 5-min no-progress watchdog. The operator state-syncs (deep gap) or fixes
-		// peering, then restarts.
-		b.logger.Warn("chain initial sync did not complete — chain NOT marked bootstrapped",
+		// Initial sync did not reach the frontier within the bounded deadline (eclipse /
+		// partition / deep gap). DO NOT transition to normal operation — leaving the VM in
+		// Bootstrapping is the fail-safe: it does not serve / build at the stale local height
+		// (that was the freeze defect). Record the reason so monitorBootstrap stops the chain
+		// PROMPTLY (F5) and surfaces it; the orchestrator restarts and the node re-syncs. bs.Run
+		// already retried beacon connectivity for its full ConnectDeadline, so this is the
+		// bounded fail-safe, not a hang.
+		b.logger.Warn("chain initial sync did not reach the frontier — VM stays bootstrapping (NOT serving normal-op), failing safe",
 			log.Stringer("chainID", b.chainID),
 			log.Err(err))
 		b.bootstrapFailed.Store(&bsFailure{err: err})
-		return
+		return false
 	}

-	// Reached the frontier: end the bootstrap phase (fail-close the bootstrap accept
-	// authority) and mark the chain ready, THEN run the live frontier poller.
+	// Reached the frontier (or no beacon set / already at the tip): transition the VM to
+	// normal operation, THEN end the engine's bootstrap phase and mark ready — so the two
+	// go-live transitions happen TOGETHER at the named frontier, never at a stale height. A VM
+	// that REFUSES normal-op is a real failure: do NOT mark ready (fail safe).
+	if err := b.transitionVMReady(); err != nil {
+		b.logger.Error("chain reached the frontier but VM SetState(Ready) failed — NOT marking bootstrapped",
+			log.Stringer("chainID", b.chainID), log.Err(err))
+		b.bootstrapFailed.Store(&bsFailure{err: err})
+		return false
+	}
 	b.engine.FinishBootstrap()
 	b.bootstrapDone.Store(true)
-	b.logger.Info("chain initial sync complete — entering live consensus",
+	b.logger.Info("chain initial sync complete — VM live (normal operation) at the network frontier",
 		log.Stringer("chainID", b.chainID))
-	b.runFrontierPoller(ctx)
+	return true
+}
+
+// transitionVMReady moves the VM to NORMAL OPERATION (vm.Ready) through the gated callback
+// buildChain wired from the VM's SetState. It is the SINGLE place the VM goes live, called
+// by runInitialSync ONLY after initial sync has reached the named frontier. No-op when no
+// SetState VM is wired (degenerate / test handlers — nothing to transition). Bounded by a
+// 30s timeout (decoupled from the sync ctx) so a wedged VM transition cannot hang the
+// goroutine, matching the original buildChain transition budget.
+func (b *blockHandler) transitionVMReady() error {
+	if b.vmReady == nil {
+		return nil
+	}
+	sctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+	return b.vmReady(sctx)
 }

 // decodeContextBlocks extracts the raw block bytes (oldest-first) from a framed
@@ -1025,3 +1025,180 @@ func TestBootstrapFailure_AccessorSurfacesReason(t *testing.T) {
 	require.True(t, bh.BootstrapFailed(), "after a fail-safe return BootstrapFailed must be true")
 	require.ErrorIs(t, bh.BootstrapFailure(), chainbootstrap.ErrNoBeaconQuorum, "the precise fail-safe reason must surface for the health check")
 }
+
+// ----- VM normal-operation GATING (the restart-freeze fix) -------------------
+//
+// These prove the ORDERING fix in runInitialSync: the VM transitions to NORMAL OPERATION
+// (vm.Ready → the EVM's onNormalOperationsStarted: block building / mempool / validator
+// dispatch) ONLY AFTER initial sync has fetch+executed up to the network frontier — never at
+// the LOCAL (possibly stale) height. Before the fix buildChain put the VM into normal
+// operation UNCONDITIONALLY right after Initialize, so a restarted STALE validator served /
+// built at its stale height and never converged (the luxd-2 mainnet freeze: stuck at 1082780
+// while producers finalized 1082796). vmHeightAtReady records the height the VM was at the
+// instant it was told to go live; the gate makes that the frontier, not the stale tip.
+
+// recordVMReady wires bh.vmReady to capture (a) how many times the VM was transitioned to
+// normal operation and (b) the VM's accepted height at the FIRST such transition — the two
+// facts the gating invariant turns on.
+func recordVMReady(bh *blockHandler, vm *bsTestVM) (calls *int, heightAtReady *uint64) {
+	calls = new(int)
+	heightAtReady = new(uint64)
+	bh.vmReady = func(context.Context) error {
+		if *calls == 0 {
+			*heightAtReady = vm.all[vm.lastAcc].height
+		}
+		*calls++
+		return nil
+	}
+	return calls, heightAtReady
+}
+
+// TestRED_StaleValidatorGoesReadyOnlyAfterReachingFrontier reproduces the EXACT production
+// scenario: a node with stale local state (height N) restarts against a live network whose
+// finalized frontier is N+k, validator set loaded. It MUST fetch+execute to N+k BEFORE the VM
+// is transitioned to normal operation — never go live at the stale N.
+//
+// FAILS before the fix (the VM was transitioned to normal operation at N, in buildChain,
+// before this sync ran) and PASSES after (the transition is gated on reaching N+k).
+func TestRED_StaleValidatorGoesReadyOnlyAfterReachingFrontier(t *testing.T) {
+	const N, K = 30, 16 // stale at 30; live frontier at 46 — the luxd-2 16-block gap shape
+	chain, byID := buildBSChain(N+K, -1)
+	vm := newBSVMAt(chain, N) // node ALREADY at height N (restarted spare), gap N+1..N+K ahead
+
+	// 5-validator equal-stake beacon set (the mainnet shape), all connected and serving — the
+	// "validator set IS loaded, 9 peers incl. producers" live condition.
+	bIDs := make([]ids.NodeID, 5)
+	weights := map[ids.NodeID]uint64{}
+	for i := range bIDs {
+		bIDs[i] = ids.GenerateTestNodeID()
+		weights[bIDs[i]] = 100
+	}
+	bh, chainID := newBSHandlerWeighted(t, vm, weights)
+	bh.bootstrapRetryInterval = 2 * time.Millisecond
+	bh.bootstrapConnectDeadline = 2 * time.Second
+	bh.net = &bsBeaconNet{bh: bh, chainID: chainID, connected: bIDs, byID: byID, tip: chain[N+K], serveAncestors: true}
+	bh.msgCreator = bsMsgBuilder{}
+
+	calls, heightAtReady := recordVMReady(bh, vm)
+
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	live := bh.runInitialSync(ctx)
+
+	require.True(t, live, "node must reach the frontier and go live")
+	require.True(t, bh.bootstrapDone.Load(), "chain must be marked bootstrapped after reaching the frontier")
+	require.False(t, bh.BootstrapFailed(), "a converged sync must not record a fail-safe reason")
+
+	last, _ := vm.LastAccepted(ctx)
+	require.Equal(t, chain[N+K].id, last, "VM must fetch+execute up to the network frontier (height %d)", N+K)
+
+	require.Equal(t, 1, *calls, "VM must be transitioned to normal operation exactly once")
+	require.Equal(t, uint64(N+K), *heightAtReady,
+		"THE FIX: VM must go to normal operation ONLY at the frontier (height %d), never at the stale local height %d", N+K, N)
+}
+
+// TestNodeBootstrap_FreshGenesisNoBeaconsReadyImmediately is the NO-REGRESSION guard for a
+// fresh network / single node: NO beacon set (FrontierNoBeacons) means there is nothing ahead
+// to sync to, so the VM must go to normal operation PROMPTLY at genesis. The gate must not pin
+// a legitimate fresh-genesis / dev node unbootstrapped.
+func TestNodeBootstrap_FreshGenesisNoBeaconsReadyImmediately(t *testing.T) {
+	chain, byID := buildBSChain(0, -1) // genesis only
+	vm := newBSVM(chain)
+	bh, chainID, _ := newBSHandlerAndEngine(t, vm, 0) // 0 beacons, expectsStakedBeacons=false
+	bh.bootstrapRetryInterval = 2 * time.Millisecond
+	bh.bootstrapConnectDeadline = 2 * time.Second
+	bh.net = &bsBeaconNet{bh: bh, chainID: chainID, connected: nil, byID: byID, tip: chain[0]}
+	bh.msgCreator = bsMsgBuilder{}
+
+	calls, heightAtReady := recordVMReady(bh, vm)
+
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	start := time.Now()
+	live := bh.runInitialSync(ctx)
+
+	require.True(t, live, "a no-beacon (fresh genesis / single node) chain must go live")
+	require.True(t, bh.bootstrapDone.Load(), "fresh-genesis chain must be marked bootstrapped")
+	require.Less(t, time.Since(start), 2*time.Second, "no-beacon fast path must not wait out the connect deadline")
+	require.Equal(t, 1, *calls, "VM must be transitioned to normal operation once")
+	require.Equal(t, uint64(0), *heightAtReady, "fresh-genesis node goes live at genesis (height 0)")
+}
+
+// TestRED_EclipsedValidatorNeverGoesReadyAtStaleHeight is the FAIL-SAFE proof: a stale node
+// whose beacons are UNREACHABLE (eclipse / isolation) must NOT go to normal operation at its
+// stale height (that was the freeze defect). Within the bounded connect deadline it fails safe
+// — VM stays in Bootstrapping (vmReady never called), bootstrapFailed records the reason — and
+// it does NOT hang (the deadline bounds it). monitorBootstrap then stops the chain; the
+// orchestrator restarts and the node re-syncs.
+func TestRED_EclipsedValidatorNeverGoesReadyAtStaleHeight(t *testing.T) {
+	const N, K = 30, 16
+	chain, byID := buildBSChain(N+K, -1)
+	vm := newBSVMAt(chain, N)
+
+	bIDs := make([]ids.NodeID, 5)
+	weights := map[ids.NodeID]uint64{}
+	for i := range bIDs {
+		bIDs[i] = ids.GenerateTestNodeID()
+		weights[bIDs[i]] = 100
+	}
+	bh, chainID := newBSHandlerWeighted(t, vm, weights)
+	bh.bootstrapRetryInterval = 2 * time.Millisecond
+	bh.bootstrapConnectDeadline = 300 * time.Millisecond // bounded fail-safe window for the test
+	// connected: nil — the beacon set is configured (weights known) but NONE are reachable.
+	bh.net = &bsBeaconNet{bh: bh, chainID: chainID, connected: nil, byID: byID, tip: chain[N+K], serveAncestors: true}
+	bh.msgCreator = bsMsgBuilder{}
+
+	calls, _ := recordVMReady(bh, vm)
+
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	start := time.Now()
+	live := bh.runInitialSync(ctx)
+	elapsed := time.Since(start)
+
+	require.False(t, live, "an eclipsed node must NOT go live")
+	require.Equal(t, 0, *calls, "THE FIX: VM must NEVER be transitioned to normal operation at the stale height when beacons are unreachable")
+	require.False(t, bh.bootstrapDone.Load(), "an eclipsed node must not be marked bootstrapped")
+	require.True(t, bh.BootstrapFailed(), "the fail-safe reason must be recorded for monitorBootstrap")
+	require.ErrorIs(t, bh.BootstrapFailure(), chainbootstrap.ErrBeaconsUnreachable, "eclipse must surface as ErrBeaconsUnreachable")
+
+	last, _ := vm.LastAccepted(ctx)
+	require.Equal(t, chain[N].id, last, "the VM must stay at its stale height (it served nothing, never advanced)")
+	require.Less(t, elapsed, 3*time.Second, "fail-safe must be BOUNDED — no unbounded hang")
+}
+
+// TestNodeBootstrap_ProducerAtTipReadyFast is the PRODUCER hard-constraint guard: a producer
+// at (or at most one block from) the frontier restarts. It must re-bootstrap QUICKLY and go
+// live — not hang waiting to sync blocks it already has. Here the node is at N and the beacon
+// quorum names N (it already holds the tip): the loop is caught-up on the first round and the
+// VM goes live at N promptly.
+func TestNodeBootstrap_ProducerAtTipReadyFast(t *testing.T) {
+	const N = 40
+	chain, byID := buildBSChain(N, -1)
+	vm := newBSVMAt(chain, N) // already at the tip
+
+	bIDs := make([]ids.NodeID, 5)
+	weights := map[ids.NodeID]uint64{}
+	for i := range bIDs {
+		bIDs[i] = ids.GenerateTestNodeID()
+		weights[bIDs[i]] = 100
+	}
+	bh, chainID := newBSHandlerWeighted(t, vm, weights)
+	bh.bootstrapRetryInterval = 2 * time.Millisecond
+	bh.bootstrapConnectDeadline = 2 * time.Second
+	bh.net = &bsBeaconNet{bh: bh, chainID: chainID, connected: bIDs, byID: byID, tip: chain[N], serveAncestors: true}
+	bh.msgCreator = bsMsgBuilder{}
+
+	calls, heightAtReady := recordVMReady(bh, vm)
+
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	start := time.Now()
+	live := bh.runInitialSync(ctx)
+
+	require.True(t, live, "a producer at the tip must go live")
+	require.Less(t, time.Since(start), 2*time.Second, "an at-the-tip restart must be FAST — not hang")
+	require.Equal(t, 1, *calls, "VM transitioned to normal operation once")
+	require.Equal(t, uint64(N), *heightAtReady, "producer goes live at the frontier (== its tip, height %d)", N)
+	require.True(t, bh.bootstrapDone.Load(), "producer chain marked bootstrapped")
+}
@@ -1225,22 +1225,31 @@ func (m *manager) buildChain(chainParams ChainParameters, sb nets.Net) (*chainIn
 			linCancel()
 		}

-		// Transition VM to normal operation after initialization
-		// For genesis-based networks with pre-configured validators, this is required
-		// to make the VM APIs available immediately
+		// Place the VM in BOOTSTRAPPING state (NOT normal operation) after init. The
+		// node-layer initial sync (runBootstrapThenPoll → runInitialSync) drives it to the
+		// network frontier; ONLY when that completes does the VM transition to Ready (see
+		// blockHandler.vmReady, wired below). Going straight to Ready here was the
+		// restart-freeze defect: SetState(Ready) fires the EVM's onNormalOperationsStarted
+		// (block building, mempool gossip, validator dispatch) at the LOCAL last-accepted
+		// height — so a restarted STALE validator served / built at its stale height while
+		// the real sync ran detached and non-gating. Bootstrapping fires onBootstrapStarted
+		// (snapshots only, no building) — the correct state while the node fetch+executes the
+		// gap. The two "go live" transitions (VM serves + engine cert-gates) now happen
+		// TOGETHER at the named frontier, never at a stale height.
 		if stateVM, ok := vmTyped.(interface {
 			SetState(context.Context, uint32) error
 		}); ok {
-			m.Log.Info("transitioning VM to normal operation",
+			m.Log.Info("transitioning VM to bootstrapping (initial sync gates normal operation)",
 				log.Stringer("chainID", chainParams.ID))
 			stateCtx, stateCancel := context.WithTimeout(context.Background(), 30*time.Second)
-			defer stateCancel()
-			if err := stateVM.SetState(stateCtx, uint32(vm.Ready)); err != nil {
-				m.Log.Error("failed to transition VM to normal operation",
+			if err := stateVM.SetState(stateCtx, uint32(vm.Bootstrapping)); err != nil {
+				stateCancel()
+				m.Log.Error("failed to transition VM to bootstrapping",
 					log.Stringer("chainID", chainParams.ID),
 					log.Err(err))
-				return nil, fmt.Errorf("failed to transition VM to normal operation: %w", err)
+				return nil, fmt.Errorf("failed to transition VM to bootstrapping: %w", err)
 			}
+			stateCancel()
 		}

 		// Create integrated consensus engine - the ONE right way to set up chain consensus
@@ -1465,6 +1474,22 @@ func (m *manager) buildChain(chainParams ChainParameters, sb nets.Net) (*chainIn
 		// on K==1), so the container bytes it parses match the bytes the engine
 		// framed — one codec, no raw-vs-wrapped split.
 		bh := newBlockHandler(blockBuilder, m.Log, consensusEngine, m.Net, m.MsgCreator, chainParams.ID, networkID, beacons, expectsStakedBeacons)
+		// Wire the gated normal-operation transition. The bootstrap driver calls this ONCE
+		// initial sync has reached the named frontier (runInitialSync) — never before, so the
+		// VM cannot serve / build at a stale height. nil when the VM exposes no SetState
+		// (degenerate / non-EVM paths): runInitialSync then just marks ready (nothing to
+		// transition). The closure captures vmTyped (the real VM, which owns SetState), not the
+		// block-builder wrapper the handler holds as b.vm.
+		if stateVM, ok := vmTyped.(interface {
+			SetState(context.Context, uint32) error
+		}); ok {
+			chainID := chainParams.ID
+			bh.vmReady = func(ctx context.Context) error {
+				m.Log.Info("initial sync reached the frontier — transitioning VM to normal operation",
+					log.Stringer("chainID", chainID))
+				return stateVM.SetState(ctx, uint32(vm.Ready))
+			}
+		}
 		// Late-bind the handler as the engine's catch-up wire: it owns requestContext
 		// (send GetAncestors) and handleContext -> Put -> engine (deliver the fetched
 		// ancestors), so a follower that falls behind now self-heals instead of
@@ -2506,6 +2531,16 @@ type blockHandler struct {
 	bsAncestorCh    map[uint32]chan [][]byte // requestID -> ancestors reply for the current Ancestors
 	bsRotor         gatomic.Uint32           // round-robins the Ancestors peer sample (M1: no monopoly)

+	// vmReady transitions the VM to NORMAL OPERATION (vm.Ready → the EVM's
+	// onNormalOperationsStarted: block building, mempool gossip, validator dispatch).
+	// runInitialSync calls it EXACTLY ONCE, and ONLY after initial sync has fetch+executed
+	// up to the named network frontier — never at the local (possibly stale) height. That
+	// ordering is the whole fix: a restarted stale validator stays in Bootstrapping (serving
+	// nothing as head) until it has converged, so it can never go live at a stale height.
+	// nil for handlers over a VM with no SetState (degenerate / test) → runInitialSync just
+	// marks ready. Set in buildChain right after construction.
+	vmReady func(context.Context) error
+
 	// beacons is the BEACON / staked-validator set the accepted-frontier quorum is
 	// anchored to (m.Validators for native chains, CustomBeacons for the P-chain). The
 	// C1 forged-chain gate: ONLY a node in this set, weighted by stake, can contribute to
@@ -2548,6 +2583,14 @@ type blockHandler struct {
 	bootstrapMinResponses int
 	bootstrapAgreement    Ratio
 	bootstrapCheckpoint   *Checkpoint
+
+	// bootstrapConnectDeadline / bootstrapRetryInterval tune the initial-sync loop
+	// (chainbootstrap.Config): how long runInitialSync WAITS for beacon connectivity before
+	// failing safe (ConnectDeadline — the bounded retry that self-heals when beacons return),
+	// and the re-sample pause between rounds (RetryInterval). Zero ⇒ library defaults (3m / 1s).
+	// Operator-overridable; the bootstrap tests set them small to bound the fail-safe path.
+	bootstrapConnectDeadline time.Duration
+	bootstrapRetryInterval   time.Duration
 }

 // bsFrontierReply is one beacon's accepted-frontier answer, tagged with the responder so