mirror of https://github.com/luxfi/node.git synced 2026-07-27 03:39:39 +00:00

Files

T

zeekayandHanzo Dev 439f2768e0 docs/postmortem: mark residual #1 (proposer-preference restart loop) FIXED

BuildBlock last-accepted fallback landed in 2dc15620e4 (ships v1.34.26).

Co-authored-by: Hanzo Dev <dev@hanzo.ai>

2026-07-08 23:27:42 -07:00

8.4 KiB

Raw Blame History

Postmortem: C-Chain accepted-head state GC eviction (mainnet freeze)

Status: Resolved. Mainnet recovered and accepted at 5/5 validators, C-Chain height 1085412, hash 0xd957eae6cb0bbef37174…, all validators in agreement, explorer at tip, treasury and Genesis NFT state verified.

Severity: Critical (mainnet C-Chain unable to build blocks; no state loss).

Accepted permanent invariant

The accepted C-Chain head's state root must never be GC/pruning eligible — across idle windows, duplicate empty-block state roots, cold snapshot/cache layers, small state history, and restarts.

accepted head ⇒ accepted head state root is pinned ⇒ GC/pruning cannot evict the execution base for H+1.

The specific accepted-head GC eviction failure is structurally prevented by the head-state pin, and production evidence confirms the fleet no longer exhibits the prior failure signature. (A formal long-idle ritual was not completed to termination during the incident window; the sign-off rests on the structural invariant plus production evidence: a head idle for 3h04m was built on cleanly, multiple 5–15 minute zero-traffic windows passed with tip state readable, and zero eviction/materialize canaries appeared fleet-wide after the fix.)

Failure mode (exact)

The C-Chain EVM (coreth-lineage, luxfi/evm) in pruning mode manages trie memory with cappedMemoryTrieWriter (core/state_manager.go):

Accepted state roots are held in a tipBuffer (BoundedBuffer) of depth state-history (default 32); as roots age out of the buffer they are Dereferenced.
Dirty trie nodes are only committed to disk at commit-interval boundaries (default 4096 blocks), with optimistic Cap flushes near the boundary.
The insert-time triedb.Reference(root, {}) in writeBlockAndSetHead is refcount-balanced: it is consumed as the block ages through the tipBuffer (or via RejectTrie). It therefore does not protect an idle head.

On an idle chain, consecutive empty blocks share identical state roots. The tipBuffer's aging Dereference for an old entry then lands on the live head root (duplicate key), dropping its reference count to zero. At any height that is not a commit boundary the head root has never been persisted, so the next Cap/flush evicts it from the dirty cache. The subsequent BuildBlock cannot open the parent (head) state:

failed to materialize parent state for build: … StateAt: missing trie node
<head state root> … is not available, not found

and the chain wedges. RPC reads at latest fail with the same error (any StateAt(root) caller). Consensus is unaffected — all validators agree on the head block; only the local execution base for H+1 is gone. State is always deterministically re-derivable from durable blocks, so a restart re-executes and recovers — restart is recovery evidence, not a fix: the head could still be evicted again in the next idle window.

Preconditions (all defaults on affected mainnet validators)

pruning-enabled: true
state-history: 32
commit-interval: 4096
idle or bursty-then-idle traffic (heartbeat pause, low organic flow)
duplicate empty-block state roots at the tip
current height not at a commit boundary

Any C-Chain deployment matching these preconditions is exposed on affected versions — this bug class will recur wherever the EVM runs with pruning, small state history, long commit intervals, and idle traffic.

Observed occurrences

Mainnet froze at height 1085200 after a ~10 minute heartbeat pause (2026-07-07). All five validators had the block, none could serve or build on its state.
An earlier fleet-wide variant contributed to the 1082879→1085012 incident window (mixed with a separate proposervm/consensus issue documented in the consensus fault-recovery audit).

Affected / fixed versions

Component	Affected	Fixed
`luxfi/evm` (C-Chain plugin)	≤ v1.104.6 (all pruning-mode deployments; the balanced insert-time Reference in v1.104.3-hotfix was insufficient)	v1.104.7
`luxfi/node` image	v1.34.14 – v1.34.23 (carry affected EVM plugins)	v1.34.24 (interim), v1.34.25 (canonical: identical fix, proper semver, clean go.mod)

Fix commits (luxfi/evm, branch evm-main-bugb): 9bab20f7a (pin), 58b90490c (non-fatal degradation), e3781c35c (vm v1.2.6 parity).

The fix (structural)

core/blockchain.go: a dedicated unbalanced GC reference held on the accepted head's state root — headStatePinRoot + pinAcceptedHead(root):

Transferred head-to-head: Reference the new head root first, then Dereference the previous pinned root; exactly one live head pin exists at all times, on lastAccepted.Root().
Established in Accept, in SetLastAcceptedBlockDirect, and on the loaded head at startup (loadLastState), so the invariant holds across restarts and the restart-then-idle path.
Skips when the root is unchanged (duplicate empty-block roots keep exactly one reference) and when state is not yet materialized (bootstrapping / state-sync; the first Accept then establishes it).
Deliberately not balanced against InsertTrie/AcceptTrie/RejectTrie — its lifetime is "is the accepted head", nothing else.
Non-fatal on backend error (pathdb Reference/Dereference are no-ops / "not supported"): a failed pin degrades to pre-fix behavior with a WARN rather than wedging Accept or startup.

No archive-mode workaround and no state-sync hack. Disk growth is unchanged (one extra referenced root).

Recovery recipe (what actually worked)

Wedged-at-tip (state evicted, DB otherwise consistent): restart the node. Boot re-executes from the last committed root and re-materializes the head state deterministically. Valid as recovery; deploy the fixed version so it cannot recur.
proposervm/EVM height split (proposervm finality index … is BEHIND the inner VM tip; produced here by crash-churn on affected versions — the fail-closed guard then correctly refuses to mount): restarts cannot heal a split. Restore the node's PVC from a VolumeSnapshot of a currently healthy peer. Per-ordinal staking keys are installed by startup.sh from the luxd-staking secret, so cross-node volume clones are safe (distinct NodeIDs).
PVC swaps must happen at StatefulSet replicas=0. A live single-pod PVC delete/recreate always loses the race to the StatefulSet controller, which recreates a blank PVC first.
If a fleet-consistent EVM rewind is needed instead (no healthy peer): evm/cmd/repair-cchain rewinds the standalone EVM lastAccepted to the proposervm floor; on boot the heightAhead branch self-heals (used in the 1084996 recovery). Zero re-execution; never a re-genesis.
Retain evidence snapshots before every destructive step.

Residual follow-ups (non-blocking; restart/churn liveness, not consensus or state-loss)

Proposer-preference restart loop: after heavy sibling churn, a node's proposervm preference can reference a never-persisted outer block; every BuildBlock then fails not found in a tight loop and the node's voter goes mute (observed ~170 err/s). Restart clears it. Fix: fall back to last-accepted when the preferred parent is not fetchable. FIXED (commit 8001bc5179, branch ship/node-v1.34.24, ships in v1.34.26): vms/proposervm/vm.go BuildBlock now builds the child on last-accepted (always held — committed state) when vm.preferred is unfetchable, instead of hard-erroring; it surfaces the original error only when last-accepted is itself the unfetchable id. Build-side companion to the already-shipped defect #1 SetPreference validate-before-assign hardening. Tests: vms/proposervm/vm_buildblock_fallback_test.go.
Ancestor-fetch liveness: the finality guard refuses certs with "ancestor … is not tracked (behind; fetch and retry)" but the fetch never fires, so the node loops instead of catching up. Restart clears it. Fix: actually schedule the ancestor fetch on this path.

Monitoring (keep active)

Eviction/materialize canaries: STATE-MATERIALIZE, missing trie node, ACCEPT-BACKSTOP log lines — expect zero.
Accepted height/hash equality across all validators.
Explorer tip parity with chain head.
is BEHIND the inner (heightBehind) occurrences — expect zero.
Do not treat the heartbeat as a safety mechanism; it is a liveness nicety.

8.4 KiB Raw Blame History Unescape Escape