Compare commits

...
21 Commits
Author SHA1 Message Date
zeekayandHanzo Dev bd2edc135f fix(chains): chunked ancestry descent — a halted fleet must still name its frontier
bootstrapNamingWindow was one constant doing THREE jobs: the per-fetch
transport size, the total ancestry budget, and a health heuristic. That
conflation wedged mainnet 96369 with no way out.

nameFrontier is already ava-equivalent — it does not count identical tip
IDs, it builds a union index from each reported tip's ancestry and gives
"global credit" (every tip vouches for every block on its chain), which
is GetAccepted-at-height semantics derived LOCALLY from hash-linked
ancestry instead of trusting a peer's assertion. Strictly stronger: a
peer can lie in a reply, it cannot forge a parent chain.

The defect was that the vouching walk stopped after ONE 256-block fetch.
Measured live, per-node in-pod:

  luxd-1 -> 1098726 (1 responder)   luxd-3, luxd-4 -> 1098191 (2)
  1098191 IS an ancestor of 1098726; luxd-3/luxd-4 `latest` IS 1098191,
  so no competing branch exists anywhere.

The ⅔-of-RESPONDER floor is 2, and 1098191 should have earned 3 — its
two direct backers plus luxd-1 vouching via ancestry. But the gap is
535 blocks, the single 256-block fetch never reached down that far, so
it earned only 2, and `> floor` rejected it. Nothing named, every retry,
forever. The node sat at height 0 and the fleet could not regain quorum.

The health heuristic ("a ⅔-common height this far below the highest tip
is not a healthy bleeding-edge split") is true for a LIVE chain and
false for a HALTED one, where the skew between a straggler and a node
that ran on alone is arbitrarily large and perfectly healthy. Recovery
from a halt is exactly when this path matters most, and it was disabled
precisely then.

  - NamingWindow stays 256, now documented as the per-FETCH transport
    bound only (Ancestry returns full blocks; one fetch must fit a
    network message).
  - MaxNamingDepth (32768) is the new TOTAL walk budget, in window-sized
    chunks. Purely a resource bound: safety comes from hash-verified
    ancestry, MinResponders distinct voters, the ⅔-of-responder floor,
    and full re-Verify on descent — none of which depend on it.
  - The walk now checks ctx each chunk, so the deadline binds the WALK
    and not merely each fetch (a peer serving a long fabricated chain
    cheaply would otherwise run the whole budget before anyone looked
    at the clock).

Tests — the matrix, all executed:
  H_HaltSkewDeeperThanOneWindow      535-block gap (asserts gap > one
                                     window, else it proves nothing)
  H_HaltSkewBeyondDepthStillFailsSafe  skew past the budget still names
                                     nothing rather than guessing
  1 high + 2 low + 2 unavailable  -> names the common ancestor
  3 high + 2 unavailable          -> names the high tip, not an ancestor
  fabricated tall tip + 2 low     -> SAFE HALT. A 9,000,000-height chain
                                     that does not link earns credit only
                                     on its own chain and can never win
                                     on height; the honest pair reach
                                     exactly the floor and strict `>`
                                     refuses. 1 of 3 responders can cost
                                     LIVENESS, never SAFETY.
  2 conflicting equal-height branches -> halts, never picks by height

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 11:03:06 -07:00
zeekayandHanzo Dev bdd7179164 docs(LLM.md): the dedup-capability branch DOES touch proposervm — scope the no-upstream-help claim to the build path
The upstream-review entry claimed nothing in the delta or its branches
touches proposervm. The delta half is proven (0 of 56 files); the branch
half was false: containerman17/proposervm-dedup-capability rewrites the
block store (inner-bytes dedup) and adds a boot-repair arm for the same
unclean-shutdown window OuterCommittedInnerNot pins. Storage-layer work,
not the build-path preferred fetch — the vm.go:380 verdict stands, but
whoever attacks it should know the one adjacent branch by name.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 09:26:05 -07:00
zeekayandHanzo Dev 50fe597bf7 v1.36.37: pin evm v1.104.23 — C-Chain plugin gets the luxfi/vm v1.3.3 map-race fix
The v1.104.22 plugin (vm v1.3.1) dies with a concurrent-map fatal in
components/chain/state.go getCachedBlock under gossip ParseBlock load, and the
dead plugin is never restarted: the node stays Running while the C-Chain is
dead (zap: connection closed, 'cannot vote correctly'). Observed live on
devnet luxd-1 during the v1.36.36 rolling-upgrade drill.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 09:25:13 -07:00
zeekayandHanzo Dev 8873e231e8 test(proposervm): crash-copy cold-boot recovery armor
Copy the committed bytes out from under a RUNNING proposervm (no
Shutdown), boot a second, cold VM over the copy, and require: a clean
boot in Initialize order, source-equality of the finality pointer, fork
height and every height's envelope opened cold, and a successful build
whose outer parent is the recovered tip and whose inner parent is that
tip's inner block. The matrix covers nothing-accepted, first-accept,
a longer run, and a copy taken while the source keeps accepting; a
dedicated test pins the one crash window the accept path leaves open
(outer batch committed, inner accept lost) booting through the
roll-back arm and re-proposing.

Harness seams testVMOnBase + acceptRangeThroughProposervm added to
height_lag_repro_test.go; everything else reuses the real VM, State and
accept path.

Verified: full vms/proposervm package green; negative control (dropping
the accept-path db.Commit) fails all four persistence-bearing scenarios.

Prompted by reviewing the upstream reference delta
bcc851822d..c5d3c8aafe: zero code ports apply (nothing in it touches
proposervm), but its crash-recovery test discipline was worth carrying
over. Verdict recorded in LLM.md.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 09:13:48 -07:00
zeekayandHanzo Dev 2a724fb090 v1.36.36: no-op patch for rolling-upgrade drill — sync stale version.txt fallback (1.32.11 -> 1.36.36)
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 09:12:04 -07:00
zeekayandHanzo Dev 23d93bf60b docs(LLM.md): testnet 96368 rolled to v1.36.35 — and the two defects it took to get there
The proposervm repair landed exactly as devnet predicted (1453->1491 rebuilt on
first boot, twice), and restoring a 4th live C-Chain broke the 1779 freeze without
touching --consensus-quorum-size. But two blockers devnet never hit had to be
fixed first, and one of them is a landmine for every future image:

luxfi/evm main pinned luxfi/vm v1.3.1 while node v1.36.35 pinned v1.3.3, so the
map-race fix shipped in luxd and NOT in the C-Chain plugin that actually verifies
blocks. It killed testnet luxd-0 five minutes into the roll, and the readiness
probe could not see it: pod Ready, restarts=0, isBootstrapped(C)=true, RPC dead.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 09:03:30 -07:00
zeekayandHanzo Dev 8a797f4805 docs(LLM.md): record the v1.36.35 devnet roll result and the third, pre-existing proposervm stall
Both fixed defects are gone fleet-wide (0 CONSENSUS-CERTIFIED fatals, 0 map races,
restarts=0 on all five for 20+ min) and a transaction reached identical receipts on
all five nodes. Five-way tip parity is NOT met: luxd-2 and luxd-4 freeze on
proposervm vm.go:380 'failed to fetch preferred block', which is proven pre-existing
(mainnet v1.36.2 luxd-1: 2223 occurrences; testnet v1.36.24 luxd-3: 35693) and is
survivable — the stalled node still votes, so the chain kept finalizing to 1913.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 03:45:59 -07:00
zeekayandHanzo Dev 9f4819cf77 v1.36.35: repair the main-branch build break (zap-proto/http v0.3.0)
011e9bf99d bumped github.com/zap-proto/http from a pseudo-version to v0.3.0 and
pushed to main without a build. v0.3.0 is a breaking API change:

  Server.Handler   net/http.Handler   ->  fasthttp.RequestHandler
  NewTransport(addr)                  ->  Dial(network, addr)

so server/http/zap_listener.go stopped compiling and MAIN HAS BEEN UNBUILDABLE
SINCE. The on-cluster image build for v1.36.34 failed on exactly that line, which
is how it surfaced; v1.36.34 produced no image and its tag is deleted.

Repair: bridge the SAME net/http handler chain the HTTP listener serves with
fasthttpadaptor.NewFastHTTPHandler, so the two transports stay behaviourally
identical and only the wire encoding differs — one handler, one place. The
round-trip test now drives a real ZAP request over the wire through that bridge
using the fasthttp request/response pair.

  ok github.com/luxfi/node/server/http
     TestZapRPCListenAddr / TestStartZapRPCListener_Disabled /
     TestStartZapRPCListener_RoundTrip

Carries the v1.36.34 payload (consensus v1.36.12 + vm v1.3.3) unchanged.
Binary self-reports luxd/1.36.35.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 03:17:36 -07:00
zeekayandHanzo Dev 84a08c2b9c v1.36.34: consensus v1.36.12 + vm v1.3.3 — the two P0s the v1.36.33 roll unmasked
consensus v1.36.12 — five devnet validators os.Exit(1)'d on a benign state:
certified = head - 1 with no fork anywhere. Two defects, one crash: the finalize
path steered the VM with a stale local blockID (backwards, into the EVM's correct
accepted-irreversibility refusal), and the classifier called "head is certified at
its own height" a double-finalization without ever checking that `certified` was at
that same height. Fixed at the producer (steer at the live build anchor) and the
classifier (a certified head is only orphaned for a block our ledger does not
certify at its height). The EVM guard and the fail-closed halt are both intact.

vm v1.3.3 — the EVM plugin process died on an unrecoverable Go map fatal in
chain.State.getCachedBlock, reached from the ZAP RPC ParseBlock handler with no
chain lock. State now owns the lock for verifiedBlocks + lastAcceptedBlock. luxd
survives that fatal and keeps reporting healthy while its chain is gone, so it is
invisible to /v1/health and pod-Ready alike.

Binary self-reports luxd/1.36.34.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 03:13:14 -07:00
zeekayandClaude Opus 5 011e9bf99d deps: drop go.sum lines that disagree with the checksum log
These hashes were recorded from direct VCS fetches made while GOPRIVATE covered
github.com/luxfi/*, which switches off the module proxy and checksum
verification together. Tags moved afterwards, so what was written down no longer
matches what sum.golang.org holds, and the build refuses to verify.

proxy.golang.org still serves the originally-published bytes and those verify
against the log, so dropping only the disagreeing lines and re-tidying restores
the build with no version change. Lines that agree were left alone — this is not
a regenerated go.sum.

The mismatch followed the machine, not the repository: the same commit built
wherever the cache was clean. Any machine or CI runner that fetched luxfi
modules under the old GOPRIVATE holds the same poisoned entries.

Also moves zap-proto/http off the 2026-05 pseudo-version to v0.3.0. The listener
already wrapped its handler with fasthttpadaptor, which is what v0.3.0 Server
takes; the pinned version still declared net/http.Handler, so the two disagreed
and the package would not compile. The go.sum repair had to land first — go get
could not run while the stale luxfi/vm lines were still being verified.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 02:14:22 -07:00
zeekayandHanzo Dev c39a85522d docs(LLM.md): correct the pChainHeight claim in the v1.36.33 entry
"Nothing in the verify path reads pChainHeight before the parent check" is wrong:
postForkCommonComponents.Verify reads child.PChainHeight() at block.go:144 and
returns errPChainHeightNotMonotonic BEFORE the inner-parent check at block.go:151.
The conclusion is unchanged and now stated exactly — that read is monotonicity only
(0 < 0 is false, so it passes), and every P-Chain-DEPENDENT check (epoch,
GetCurrentHeight, proposer window) is gated behind consensusState == Ready and sits
AFTER the parent check, so pChainHeight=0 cannot produce errInnerParentMismatch.

Also name the ids so the LpoYY reading is checkable: constants.PlatformChainID =
ids.PChainID ("111…P"), PrimaryNetworkID = ids.Empty ("111…LpoYY") —
luxfi/constants@v1.6.2 network_ids.go:64-65.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 01:06:32 -07:00
zeekayandHanzo Dev 1c92026c7e fix(admin): setLoggerLevel/getLoggerLevel actually move and report a logger's level
Both endpoints answered 200 OK having done nothing. SetLoggerLevel computed the
logger names and threw them away — `loggerNames := a.getLoggerNames(...); _ =
loggerNames` — and getLogLevels returned an empty map unconditionally, so
getLoggerLevel reported `{}` for every logger no matter what had been set. There
was no way to raise a running node's log level, which is why the 2026-07-28
devnet/testnet build-loop diagnosis had to be run off boot logs.

log.Factory already exposes exactly what is needed at the pinned v1.4.3:
SetLogLevel/SetDisplayLevel/GetLogLevel/GetDisplayLevel, all addressed BY NAME,
plus log.ToLevel to parse the argument. So:

- SetLoggerLevel parses both levels BEFORE taking the lock (a rejected level
  leaves every logger untouched) and applies only the levels the caller supplied,
  so omitting one keeps its value instead of resetting it to the zero Level.
- getLogLevels reads the factory it is reporting on.
- getLoggerNames is the ONE place either endpoint decides what it addresses, and
  it now refuses an empty name explicitly: loggers are addressed by name and
  log.Factory exposes no enumeration, so the "every logger" form cannot be
  served. Answering 200 OK for it is the bug, not the contract.

Known remaining gap, in the dependency not here: factory.SetLogLevel silently
ignores an unregistered name and GetLogLevel answers InfoLevel for it, so a
typo'd loggerName still succeeds quietly. Closing that needs luxfi/log to report
an unknown logger; it is not reachable from this repo.

logger_level_test.go drives the REAL log.Factory the node builds, not a double,
and asserts against the factory as well as the API reply, so a getLoggerLevel
that merely echoed the request could not pass. Both behavioural tests fail with
the pre-fix bodies restored; the arg-validation test passes on both sides.

No version bump and no new tag: v1.36.33 stays the exact artifact under review
for the devnet/testnet/mainnet roll. This rides the next release.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 01:04:02 -07:00
zeekayandHanzo Dev f7e501d021 docs(LLM): the v1.36.33 roll surface is the StatefulSet, and mainnet is not a clean control
Verified live 2026-07-28T07:5xZ while red-reviewing the v1.36.33 hand-off:

- lux-operator and lux-operator-devnet are 0/0, so luxnetworks.lux.cloud/luxd
  reconciles nothing and its tags are stale (mainnet v1.34.0 — in no registry;
  testnet v1.32.12). The live image is on the StatefulSet (mainnet v1.36.2 via
  kubectl-patch 07-25, devnet v1.36.25@sha256:ca497eff, testnet
  v1.36.24@sha256:91e2542b), all OnDelete. Rolling via the CR then deleting a pod
  reboots it on the OLD image — the C-Chain-dies-on-boot case.
- mainnet luxd-0/3/4 are frozen at 1098191 (block ts 2026-07-24T15:46:19Z) with
  3998/4000 log lines rebuilding height=1098196, luxd-2 has no C-Chain, and only
  luxd-1 mines (1098341, receipt status 0x1). Same hash at 1098191 on luxd-0 and
  luxd-1, so it is not a fork. The drop line is missing from the v1.36.2 BINARY
  (grep -c "built block failed verification" /luxd/build/luxd: 0 on mainnet,
  1 on devnet v1.36.25) — which is the only reason mainnet looked clean.
- The 4 broken mainnet pods are exactly those on ControllerRevision rev 147;
  luxd-1, the one that mines, is the only pod still on rev 144. Deleting it
  recreates it on the revision the others broke on.
- ghcr.io/luxfi/node:v1.36.2 corresponds to no git tag, so what mainnet runs is
  not reproducible from this repo.

Docs only: no source, no version change, no live change.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 01:00:46 -07:00
zeekayandHanzo Dev a57251c318 fix(proposervm): anchor the inner build parent, so a node stops rejecting the block it just built
devnet 96367 and testnet 96368 spent hours in a build→self-verify-fail→drop loop:
every proposer emitted "built block … height=1047" and immediately "built block
failed verification — dropping / inner parentID didn't match expected parent",
83–456 drops/min per node, with the accepted tip frozen two heights BELOW what the
builder kept proposing.

buildChild asked the inner VM for a block without first pointing the inner VM at the
parent's inner block. The inner VM builds on ITS OWN head (luxfi/evm: the miner reads
bc.CurrentBlock()); postForkCommonComponents.Verify requires
child.innerBlk.Parent() == parent.innerBlk.ID(). Build and verify read two different
pointers, one required to equal the other, and nothing asserted it at the moment it
was needed.

The head drifts without the proposervm's knowledge: verifying a GOSSIPED block whose
parent is the current head optimistically makes it the head (evm core/blockchain.go
writeBlockAndSetHead → newTip → writeCanonicalBlockWithLogs → writeHeadBlock), no
accept and no proposervm involvement. SetPreference cannot undo it either — it
short-circuits on an unchanged outer preference, so re-affirming the same tip never
re-pushes the inner preference. Nothing self-corrects; the loop is forever.

VM.anchorInnerBuildParent asserts the invariant where it is required: one inner
SetPreference immediately before delegating, from BOTH build sites
(postForkCommonComponents.buildChild and preForkBlock.buildChild — the transition
block is verified by the same check, and at the fork height every validator may emit
its own candidate, so the drift is the norm there). On a healthy node the inner
setPreference early-returns on current.Hash() == block.Hash(): one lookup, no state
change, no behaviour change. When it fails, the head is provably not the parent's
inner block, so refusing to build is strictly better than emitting a block this node
is guaranteed to drop.

This is NOT the P-Chain. info.isBootstrapped{"chain":"P"} is true on 15/15 nodes and
platform.getHeight is 0 on 15/15 including mainnet, whose built blocks also carry
pChainHeight=0; the health blob's "111…LpoYY" is the primary-network NET id, not the
P-Chain. Nothing in the verify path reads pChainHeight before the parent check.

build_inner_parent_test.go models the inner VM's three real head semantics
(build-on-head, verify-advances-head, SetPreference-reorgs-head) and fails without
this change on both build paths; TestBuildChild_HealthyHead_NoReorg pins the
no-behaviour-change half.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-28 00:43:07 -07:00
zeekayandHanzo Dev 2f6ef061ab chore(version): patch-bump 1.36.31 -> 1.36.32
Carries the proposervm finality-index fix. defaultPatch and the RPCChainVM
protocol-42 compatibility list move together, so a binary built without
ldflags reports the same version as the tag and TestCurrentRPCChainVMCompatible
stays green.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-27 19:45:05 -07:00
zeekayandHanzo Dev 7d2f01eb0c fix(proposervm): finality index can no longer fall behind the inner VM tip, and a node that is already behind boots and repairs itself
ROOT CAUSE. Every post-fork accept commits the outer envelope, its height
index entry and the last-accepted pointer in ONE versiondb batch BEFORE the
inner block is accepted, so the index can only run AHEAD. But the proposervm
has one accept path that moves the inner VM and writes nothing:
preForkBlock.Accept, whose acceptOuterBlk() is a no-op. Post-fork it was
reachable because BOTH proposervm block constructors fall back to it
silently — getBlock() when the id is not an OUTER envelope id (and the id the
consensus ledger records as canonical IS the inner block's id, see
postForkCommonComponents.CanonicalID), and ParseBlock() when the envelope
does not parse. Accepting one such block advanced the inner VM and left the
index stranded. Nothing complained, because nothing asserted the invariant at
the moment it was violated — it was only checked at the NEXT boot, where
repairAcceptedChainByHeight refused to start and killed the chain:

  VM initialization failed error="failed to repair accepted chain by height:
  proposervm finality index (height 6, id ns5qGN4i...) is BEHIND the inner VM
  tip (height 98, id TUTR74eA...)"
  non-critical chain failed to initialize ... chainAlias=C

The writes were never issued at all — this was never a persistence problem.

PREVENTION. One predicate, vm.refusePreForkAfterFork(height), used by both
seams so they cannot disagree: post-fork, a block at or above the recorded
fork height is never a pre-fork block, so getPreForkBlock refuses to build one
and preForkBlock.acceptOuterBlk refuses to accept one. Before the fork
(no fork height recorded) both are unchanged no-ops. acceptPostForkBlock also
warns on a non-contiguous index so a hole is named where it opens.

RECOVERY (height_backfill.go), outer-only — no inner re-execution, no EVM
rollback, no resync:
  1. rebuildOuterIndexFromStore re-derives the index at boot from envelopes
     already in this node's block store, binding each candidate to the inner
     block WE accepted at that height and to the envelope at height-1.
  2. If that cannot reach the tip the chain STARTS anyway, loud and
     build-gated (a node with a hole must not propose), and heals through
     BackfillOuterBlock — which is also offered every envelope arriving via
     ordinary ParseBlock traffic, so a damaged node repairs itself from peers
     with no new transport and no operator step.
The finality pointer is only ever moved FORWARD onto a proven envelope and is
never dropped: DeleteLastAccepted would make LastAccepted() fall back to an
inner-namespace id whose ParentID is contiguity-incompatible with the
network's outer wrappers, silently wedging the node at the inner tip forever.

TESTS. height_lag_repro_test.go reproduces the lag using only pre-existing
API; on the unfixed code it prints the divergence and then the verbatim
production init error, and passes after the fix. height_backfill_test.go
proves both recovery paths, the build gate, the self-heal from parse traffic,
and the two rejection cases (wrong inner block, out of order).

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-27 19:43:37 -07:00
zeekayandHanzo Dev c76fcda71d fix: point keyutil at kms/pkg/mnemonic, the canonical loader
keys.LoadMnemonicFromKMS was deleted from luxfi/keys at v1.3.0 and this
call site never moved, so the example helper has not compiled since:

    keyutil.go:76:25: undefined: keys.LoadMnemonicFromKMS

keyutil backs eight wallet examples, so the break is not contained to
one program.

The loader was not lost — it moved to kms/pkg/mnemonic deliberately.
keys must not import kms, because kms already imports keys for
ServiceIdentity envelopes, and the back edge would close an import
cycle. mnemonic.LoadFromKMS keeps the same behaviour and adds a
ServiceIdentity parameter; nil is the documented value for a caller
trusting the network boundary rather than a signed application-layer
envelope, which is what this path already relied on.

luxfi/kms was already an indirect dependency (v1.12.4); this makes it
direct at v1.12.10. The MNEMONIC-env local is renamed to `phrase` so it
stops shadowing the package inside the same function.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-27 13:47:38 -07:00
zeekayandHanzo Dev e1e3917772 Dockerfile: pin EVM v1.104.22 — realign plugin api/vm/geth with node, land the fee-split seam
Two defects in one ARG.

1. api mismatch. Node main pins luxfi/api v1.1.1, vm v1.3.1, geth v1.20.1.
   EVM_VERSION was pinned at v1.104.9, whose go.mod is api v1.0.16 / vm v1.2.6
   / geth v1.17.12. That is the same InitializeResponse decode mismatch this
   file already documents for v1.104.8, just inverted: host and plugin must sit
   on one api line or every mgj786 chain fails to initialize. v1.104.22 is
   api v1.1.1 / vm v1.3.1 / geth v1.20.1 — an exact match.

2. fee split absent. The C-Chain fee-split seam (core/fee_split.go creditTxFee,
   extras.FeeSplitTimestamp, extras.FeeRewardVault 0x0100..0002) first ships in
   evm v1.104.14. Every node image up to and including v1.36.25 bakes an evm
   below that line, so encoding/json silently discards the genesis
   "feeSplitTimestamp" key. Measured on devnet 96367 (node v1.36.25, genesis
   feeSplitTimestamp 1785133547, long past): a mined transfer credited 100% of
   its fee to the block coinbase and left the reward vault at exactly 0 — no
   reward accrual, no burn. Confirmed against the running plugin binary:
   feeSplitTimestamp / creditTxFee / FeeRewardVault all grep 0 times, while
   feeConfig and cancunTime hit.

The split remains dormant wherever feeSplitTimestamp is absent (mainnet), where
creditTxFee takes the unchanged legacy coinbase path, so this bump is
behaviour-preserving there.

Note for operators upgrading a LIVE chain that already carries a past-dated
feeSplitTimestamp: extras.checkConfigCompatible rejects nil -> set once the
timestamp is behind head, so forward-date feeSplitTimestamp (or re-genesis)
in the same change as the image bump.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-27 11:47:15 -07:00
zeekay e7137ed28d Merge remote-tracking branch 'origin/main'
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-26 20:06:15 -07:00
hanzo-devandHanzo Dev 3398024a19 build: reconcile tools go.sum with the module graph
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-26 20:05:15 -07:00
zeekayandHanzo Dev e77b696662 fix(deps): re-pin luxfi/consensus to v1.36.10 — v1.36.9 silently wedges a chain
main pinned consensus v1.36.9, which contains the silent-rebuild-storm bug that
took devnet down for four and a half days. Any network that upgraded from HEAD
would have wedged the same way.

The bug: buildBlocksLocked keeps pendingBuildBlocks after a failed build, and
proposervm truncates the block timestamp to a whole second — so every rebuild
inside that second re-mints an IDENTICAL blkID. consensus.AddBlock rejects it as
"already exists" and a bare `continue` skips Propose/RequestVotes entirely. The
block is never sent, never voted, never decided, while pending txs keep
re-notifying. Measured on devnet: one node re-built the same blkID 2,287 times
at height 512, with ZERO "proposed block to validators" and ZERO "finalized
block via quorum cert" lines across 200k log lines fleet-wide.

It also leaves a durable trap. The on-disk vote-once guard binds those storm
blkIDs to heights that can never be re-proposed, so reserveSlotForSign refuses
every later block at those heights — the chain stays wedged even after the
binary is fixed. Devnet needed a hand-repaired guard on all five nodes to
recover; that repair has no code path yet (the voteguard docstring points at
engine/chain/lock_migration.go, which does not exist).

This is a REGRESSION, not a stale pin: v1.36.24 and v1.36.25 both pinned
v1.36.10 and finalize normally — devnet runs v1.36.25 today and is producing
blocks. v1.36.26, v1.36.27, v1.36.28 and HEAD went back to v1.36.9.

v1.36.10 is the latest consensus release, so this moves forward. Verified
against an EMPTY module cache (the cold path the in-cluster builder takes), and
./vms/... and ./chains/... build clean.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-26 19:51:24 -07:00
25 changed files with 2550 additions and 131 deletions
+14 -1
View File
@@ -277,7 +277,20 @@ RUN . ./build_env.sh && \
ARG EVM_CGO=0
ARG EVM_TAGS=""
ARG EVM_VERSION=v1.104.9
# v1.104.22 realigns the plugin with THIS node's own go.mod: api v1.0.16 ->
# v1.1.1, vm v1.2.6 -> v1.3.1, geth v1.17.12 -> v1.20.1. Node main pins api
# v1.1.1, so pinning an evm at api v1.0.16 reintroduces the exact
# InitializeResponse decode mismatch described above, just in the opposite
# direction — keep this ARG and node's go.mod on the same api/vm/geth line.
#
# v1.104.14 is also the FIRST evm tag carrying the C-Chain fee-split seam
# (core/fee_split.go creditTxFee + extras.FeeSplitTimestamp/FeeRewardVault).
# Below it, encoding/json silently DISCARDS the genesis "feeSplitTimestamp"
# key: a chain configured for the 50/50 split instead routes 100% of every fee
# to the block coinbase and the reward vault 0x0100..0002 stays 0 forever, with
# nothing burned. The split stays dormant wherever feeSplitTimestamp is absent
# (mainnet), so this bump is behaviour-preserving there.
ARG EVM_VERSION=v1.104.23
ARG EVM_VM_ID=mgj786NP7uDwBCcq6YwThhaN8FLyybkCa4zBWTQbNgmK6k9A6
# the pinned evm go.mod may pin a dead luxfi/upgrade pseudo-version
# (v1.0.1-0.20260603055252-f51810805436 — commit pruned from origin). Heal it to
+269
View File
@@ -253,6 +253,275 @@ when v1.36.12 image is ready → set devnet sts image v1.36.12, delete luxd-1, c
C-Chain inits + reaches tip; then finish devnet (durable-fix proof + RewardManager),
then gated testnet→mainnet→zoo.
## Crash-boot recovery armor + upstream delta verdict (vms/proposervm, test-only, 2026-07-28)
**Upstream review (avalanchego master `bcc851822d..c5d3c8aafe`, 6 commits): ZERO code
ports.** grpc bump (no named bug; would cascade through luxfi/vm), README typo, typed
sync-client scaffolding (client-side only, no serving handlers even upstream; our sync
direction is ZAP-native — reference for task #66 only), a Firewood `debug_intermediateRoots`
enabler (our tracers API is `.disabled`, twice N/A), and two streaming-VM-only mempool
guards whose failure modes we already bracket (`miner/worker.go:62 targetTxsSize=1800KiB`
build cap + txpool `ErrGasLimit`/128KB caps under the 2MiB wire cap). **Nothing in the
delta touches proposervm — the open `vm.go:380` preferred-fetch failure gets no upstream
help.** One upstream BRANCH does touch proposervm — `containerman17/proposervm-dedup-capability`
(+72 vm.go): inner-bytes dedup in the block store plus an extra boot-repair arm for the
unclean-shutdown window where the outer index lands above the inner survivor (the very
window `OuterCommittedInnerNot` below pins) — storage-layer work, not the build-path
preferred fetch, so it changes nothing about `vm.go:380` either. What WAS worth taking is
their recovery-test discipline, ported as:
**`vm_crashboot_test.go`** — copy the COMMITTED bytes out from under a RUNNING proposervm
(no Shutdown), boot a SECOND, cold VM over the copy (repair + metadata, Initialize order),
assert source-equality (finality pointer, fork height, every height's envelope openable
cold) and then BUILD on the recovered tip (outer parent == recovered envelope, inner
parent == its inner block — the errInnerParentMismatch invariant from a cold boot).
Matrix: nothing-accepted / first-accept / longer run / copy-while-source-runs-on, plus
`OuterCommittedInnerNot`: the one crash window the accept path leaves open
(`acceptPostForkBlock` commits BEFORE the inner accept) must boot via the roll-back arm
and re-propose. Harness seams added in height_lag_repro_test.go (`testVMOnBase`,
`acceptRangeThroughProposervm`). **Negative control executed**: deleting the
`vm.db.Commit()` from `acceptPostForkBlock` fails every persistence-bearing scenario
(4 of 6) — the armor bites on flush-order regressions, not just on green paths.
## v1.36.35 — the certified-descendant false halt + the plugin-killing map fatal (devnet 96367, 2026-07-28)
> Shipped as v1.36.35, not v1.36.34: the v1.36.34 tag inherited a build break that
> `011e9bf99d` ("deps: drop go.sum lines that disagree with the checksum log") had
> already pushed to main — it bumped `zap-proto/http` from a pseudo-version to v0.3.0,
> whose `Server.Handler` is a `fasthttp.RequestHandler` and whose `NewTransport` is now
> `Dial(network, addr)`, so `server/http/zap_listener.go` no longer compiled. **main was
> unbuildable from that commit until this one.** Repaired here by bridging the one
> net/http handler chain with `fasthttpadaptor.NewFastHTTPHandler` (the two transports
> stay behaviourally identical — only the wire encoding differs) and moving the
> round-trip test onto the fasthttp request/response pair. v1.36.34 produced no image
> and its tag was deleted.
Rolling devnet to v1.36.33 fixed the build→verify-fail→drop loop and unmasked two
DIFFERENT failures. Both are fixed here; each has a regression test that fails before
and passes after.
**P0-1 — five validators `os.Exit(1)` on a benign state.** Each devnet node hit, once:
```
error VM accepted head is CONSENSUS-CERTIFIED and conflicts with the newly finalized
block — refusing to orphan it orphanedHeight=1364
fatal SetPreference would orphan a CONSENSUS-CERTIFIED block — refusing (fail-closed)
error="cannot orphan finalized block at height: 1364 to common block at height: 1363"
```
at three distinct height pairs (1258/1259, 1363/1364, 1364/1365), ALWAYS with
`certified = head 1`, with every surviving node holding byte-identical blocks at all
five heights — no fork anywhere. (The "1168/1169" pair in the original report never
appears in any fatal line; those two blocks are a normal parent/child present on all
live nodes.)
Two defects in `luxfi/consensus`, one crash — fixed in **consensus v1.36.12**:
- *Producer.* `acceptWithCertCore` releases `t.mu` across every VM call-out, then steers
the VM with its STALE local `blockID`. A finalize that completed in that window has
already advanced the ledger AND the EVM to `blockID+1`, so the steer is BACKWARDS and
the EVM's accepted-irreversibility guard (`evm/core/blockchain.go:1987`,
`commonBlock < lastAccepted`) correctly refuses it. **That guard is right and is
untouched.** Now steers at the live build anchor (`PreferredBuildTip`
`ledger.BuildAnchor`), which the accept ordering (ApplyCert BEFORE VM.Accept) keeps at
or above the VM's own accepted head. Same value the build path already uses.
- *Classifier.* `reconcileVMToCertified` asked only "is the head the ledger's certified
canonical at ITS OWN height?" — trivially true for every healthy node whose head is
certified — and called that a two-blocks-at-one-height double-finalization. It never
established that `certified` was at that same height. The ledger holds one canonical
per height along one contiguous chain, so when both are certified at their own heights
they lie on that one chain and the head merely DESCENDS from the target: nothing is
orphaned. **The fail-closed halt is not weakened** — it now fires on the state that is
actually unsafe (steering off a certified head onto a block our own ledger does not
certify at its height).
Direction: NEITHER roll the head back NOR certify forward. The VM head is legitimately
ahead and already CONTAINS the certified block; `FinalityLedger.BuildAnchor` already
documents `head > certified` as the designed state, and rolling back is exactly what
`blockchain.go:1987` exists to refuse. The correct action is no action — plus not issuing
the backwards steer at all.
**P0-2 — the EVM plugin process dies and never comes back.** devnet luxd-1:
```
fatal error: concurrent map read and map write
vm/components/chain.(*State).getCachedBlock state.go:216
vm/components/chain.(*State).ParseBlock state.go:267
evm/plugin/evm.(*VM).ParseBlock vm.go:340
vm/rpc.(*zapVMServer).handleParseBlock vm_server_zap.go:477
```
`chain.State` was written against avalanchego's contract that the consensus engine holds
the chain lock across every VM call. The ZAP VM server does NOT reinstate it — it
dispatches ParseBlock/GetBlock and the Verify/Accept/Reject wrappers concurrently.
`verifiedBlocks` (plain map) and `lastAcceptedBlock` (pointer) are the only State that is
not self-synchronising; every `cache.Cacher` carries its own mutex. Fixed in **vm v1.3.3**
by giving State one RWMutex for exactly those two, never held across a call into the
inner VM. A Go map fatal is unrecoverable: it kills the plugin, **luxd survives and keeps
answering `info.getNodeVersion` while its chain is gone** — pod-Ready and `/v1/health`
both stay green — and there is no self-heal.
**Devnet roll result (10:2310:45Z).** All five pods on v1.36.35, `restarts=0` on every one
for 20+ minutes. Both fixed defects are GONE fleet-wide: `CONSENSUS-CERTIFIED` fatal = 0
(previously all five died within minutes), `concurrent map` = 0. luxd-1, whose C-Chain had
been dead ~45 min with no self-heal, came back at tip parity on first boot; luxd-3, stuck at
1524, caught up immediately. A transaction of ours got a receipt with identical status,
blockHash and resulting balance on all five nodes (`0xcb5e9a06…`, block 1636, status 0x1).
**NOT accepted — a THIRD, pre-existing defect blocks five-way parity.** Two nodes (luxd-2 at
1789, luxd-4 at 1825) freeze their tip while 0/1/3 advance in lockstep, emitting
```
error unexpected build block failure error="not found"
reason="failed to fetch preferred block; no distinct last-accepted fallback"
```
from `vms/proposervm/vm.go:380`. That branch is reached when the preferred block is
unfetchable AND last-accepted is either unreadable or the same id — a local storage/index
condition, not a steering choice. It is **NOT a regression from this release**: the identical
line appears on binaries built long before it — mainnet luxd-1 on **v1.36.2** (2,223
occurrences, and that is the one mainnet node still producing) and testnet luxd-3 on
**v1.36.24** (35,693). It is also survivable: the stalled node keeps VOTING, so quorum holds
and the chain keeps finalizing (devnet reached 1913 with two nodes frozen), and devnet luxd-1
self-recovered from it once (1609 → 1634). Left unfixed and unchained-to — it needs its own
diagnosis at the proposervm layer.
**Quorum arithmetic (reported, not changed).** Devnet and testnet both run
`--consensus-sample-size=5 --consensus-quorum-size=4`. With only 3 live C-Chains the
quorum is arithmetically unreachable; devnet demonstrated both directions in one session
(pinned at 1378 on 3 live, 1378→1395→1396 the moment luxd-4 restored a 4th). This is a
config value, not a code defect, and lowering it lowers the safety margin — left as is.
### Testnet 96368 roll (15:3216:00Z) — accepted, after two blockers the devnet roll never hit
Testnet was frozen at **1779** with only three live C-Chains, below `--consensus-quorum-size=4`
(`ceil(2·5/3)+1`), so no block could be accepted. luxd-1 and luxd-4 had no C-Chain at all —
`/v1/bc/C/rpc` 404 — on `failed to repair accepted chain by height: proposervm finality index
(height 1453 / 1463) is BEHIND the inner VM tip (height 1491)`. v1.36.35 turns that fatal into
a repair, and it worked on the first boot of each: `proposervm finality index REBUILT from the
local block store — index and inner tip agree fromHeight=1453 toHeight=1491`. The freeze broke
the instant a **fourth** C-Chain came up. Final: five nodes on v1.36.35, every binary
self-reporting `luxd/1.36.35`, `1779 → 1888` and climbing, tips in exact agreement, and one
real transaction (`0x333d5bbd…`, block `0x728`) returning `status=0x1` with **identical
blockHash `0x07fd0c68…` and `gasUsed=0x5208` on all five nodes**. α was NOT lowered.
Two defects had to be fixed first. Both are invisible on devnet and both apply to any fleet.
**1. The RLP startup import is fatal on re-run — it kills the C-Chain on EVERY boot.**
Testnet's startup script passes `--import-chain-data` on every boot, relying on
`isNothingToImportError` to make re-importing an already-imported chain a no-op. That guard
only ever existed on `luxfi/evm` `hotfix/v1.104.9` (commit `c58d307e`, tags
`v1.104.9-hotfix.2/3/4`); `git merge-base --is-ancestor c58d307e main` = **false**. Only the
*other* half of that commit was forward-ported. So images built from evm main die with
`startup import failed: no blocks imported (parsed=0)`. Proven from the two plugin binaries
in-cluster, with a positive control:
| string in `plugins/mgj786NP7…` | v1.36.24 | v1.36.35 |
|---|---|---|
| `ImportChain: resuming from current head` | 1 | 1 ← control |
| `nothing to import` | 1 | **0** ← the guard |
| `no blocks imported (parsed=` | 1 | 1 |
Fixed **twice, on purpose**: the guard is restored in `luxfi/evm` main (with
`startup_import_idempotency_test.go` locking the contract), and the flag is now gated on a
per-PVC sentinel in `universe/k8s/lux-testnet/luxd-startup.yaml` — a completed one-time
migration must not re-run forever. Sentinel pre-seeded on all five PVCs before the ConfigMap
was patched. **Mainnet is NOT exposed**: its `luxd-startup` ConfigMap (39,719 bytes, contains
`consensus-quorum-size` twice as a control) has zero `--import-chain-data` and no `.rlp` on disk.
**2. 🚨 The `luxfi/vm` map-race fix never reached the C-Chain.** node v1.36.35 bumped
`luxfi/vm` to v1.3.3 for it, but the C-Chain is a **plugin built from `luxfi/evm`**, which
still pinned **v1.3.1**. Read off the two binaries inside one v1.36.35 pod:
```
/luxd/build/luxd github.com/luxfi/vm@v1.3.3
/luxd/build/plugins/mgj786NP7… github.com/luxfi/vm@v1.3.1 ← verifies the blocks
```
It fired on testnet luxd-0 five minutes after the roll: `fatal error: concurrent map writes`
in `luxfi/vm@v1.3.1/components/chain/block.go:44 (*BlockWrapper).Verify` via
`rpc/vm_server_zap.go:580 handleBlockVerify`. v1.3.3 is precisely the fix for that line — it
takes `state.blocksLock` around `verifiedBlocks[blkID] = bw`, which v1.3.1 wrote unlocked from
every concurrent ZAP RPC handler. `luxfi/evm` main is now on v1.3.3; **the next node image
must be built after that bump, or this race ships again.**
⚠️ **The readiness probe cannot see this.** The plugin dies while luxd survives, so the pod
stays `ready=true`, `restarts=0`, and `info.isBootstrapped(C)` keeps answering `true` while
`eth_blockNumber` times out — a dead C-Chain still in the Service, the exact failure the probe
was redesigned to catch. Only a per-node **tip** probe sees it. Recovery is a pod delete.
## v1.36.33 — the build→self-verify-fail→drop loop (devnet 96367 / testnet 96368, 2026-07-28)
**Symptom.** Every proposer built a block and then rejected the block it had just
built, forever: `built block … height=1047` immediately followed by
`built block failed verification — dropping error="inner parentID didn't match
expected parent"`, 83456 drops/min per node, accepted tip frozen two heights BELOW
what the builder kept proposing (devnet tip 1045, builds 1047).
**Root cause (one line).** `buildChild` asked the inner VM for a block without first
pointing the inner VM at the parent's inner block. The inner VM builds on ITS OWN
head (`luxfi/evm` miner reads `bc.CurrentBlock()`); the verify path requires
`child.innerBlk.Parent() == parent.innerBlk.ID()`. Two different pointers, one
required equal to the other, nothing asserting it. The head drifts on its own:
verifying a GOSSIPED block whose parent is the current head optimistically sets the
head (`core/blockchain.go writeBlockAndSetHead → newTip → writeCanonicalBlockWithLogs`),
and `VM.SetPreference` short-circuits on an unchanged outer preference so it never
re-pushes the inner preference to drag the head back. Non-self-correcting by
construction.
**Fix.** `VM.anchorInnerBuildParent` (vms/proposervm/vm.go) — one inner
`SetPreference` at the point of use, called from BOTH build delegations
(`postForkCommonComponents.buildChild`, `preForkBlock.buildChild`). On a healthy node
the inner `setPreference` early-returns on `current.Hash() == block.Hash()`, so it is
a lookup and no state change; when it fails, the head is provably not the parent's
inner block, so refusing to build beats emitting a block we would drop.
Regression proof: `vms/proposervm/build_inner_parent_test.go` (models the three evm
head semantics — build-on-head, verify-advances-head, SetPreference-reorgs-head).
**NOT the cause, measured:** the P-Chain. `info.isBootstrapped{"chain":"P"}` = true on
15/15 nodes and `platform.getHeight` = 0 on 15/15 **including mainnet**, whose built
blocks also carry `pChainHeight=0`. `bootstrapped.message:["111…LpoYY"]` is the
primary-network NET id (`chains/chains.go Nets.Bootstrapping` keys by net), not the
P-Chain — the P-Chain's chain id prints `111…P` (`constants.PlatformChainID =
ids.PChainID`, while `PrimaryNetworkID = ids.Empty`). The verify path DOES read
pChainHeight before the parent check, but only as monotonicity
(`childPChainHeight < parentPChainHeight`, and 0 < 0 is false, so it passes); every
P-Chain-DEPENDENT validation — epoch, `GetCurrentHeight`, proposer window — is gated
behind `consensusState == Ready` and sits AFTER the parent check. So `pChainHeight=0`
cannot produce `errInnerParentMismatch`.
**Sibling failure modes on the same fleets (already fixed in 1.36.32, needs the roll):**
outer index BEHIND the inner tip ⇒ `refusing to build`/boot repair
(`height_backfill.go`, 7d2f01eb), and preferred-absent-locally ⇒ last-accepted
fallback (`vm.go BuildBlock`). All three are the same invariant seen from three sides.
**Roll surface — measured 2026-07-28T07:5xZ, do not use the LuxNetwork CR.**
`lux-operator` and `lux-operator-devnet` are **0/0** in `lux-system`, so nothing
reconciles `luxnetworks.lux.cloud/luxd`; its tags are stale garbage (mainnet
`v1.34.0`, testnet `v1.32.12` — v1.34.0 exists in no registry). The live image is on
the **StatefulSet**, hand-maintained: mainnet `ghcr.io/luxfi/node:v1.36.2`
(`kubectl-patch` 07-25T00:27:04Z), devnet `v1.36.25@sha256:ca497eff…`, testnet
`v1.36.24@sha256:91e2542b…`, all three `updateStrategy: OnDelete`. Change the image in
the universe manifest and apply, then delete one pod. Editing the CR and deleting a pod
reboots it on the OLD image.
**Mainnet 96369 is NOT a clean control — 3 of 5 nodes have Mode B, invisibly.**
luxd-0/3/4 `eth_blockNumber` = `0x10c1cf` (1098191, block ts 2026-07-24T15:46:19Z) and
have not moved in 3+ days while luxd-1 mines (1098341, tx status 0x1, 07:58Z); luxd-2
serves 404 (no C-Chain). Not a fork — 1098191 hashes identically on luxd-0 and luxd-1.
3998 of luxd-0's last 4000 log lines are the same `built block … height=1098196`. The
drop line is absent because the **binary** lacks it: `grep -c "built block failed
verification" /luxd/build/luxd` = **0** on mainnet v1.36.2, **1** on devnet v1.36.25.
`/v1/health` says `{"healthy":true,"error":"health reply encode failed"}` on the frozen
node and the mining node alike — never gate on it, use tip parity.
Two hazards for the roll: the 4 broken mainnet pods are exactly the ones on
ControllerRevision `luxd-596857c9d6` (rev 147, kubectl-patch 07-25: GOMEMLIMIT=6GiB,
mem limit 16Gi→12Gi, request 4Gi→7Gi) and the one mining node, luxd-1, is the only pod
still on rev 144 — **deleting luxd-1 recreates it on the revision every other node
broke on**. And `ghcr.io/luxfi/node:v1.36.2` has no git tag at all
(`git ls-remote origin refs/tags/v1.36.2` → empty; v1.36.24/25 → present), so what
mainnet runs is not reproducible from this repo.
## Essential Commands
### Release & build (canonical) — via platform.hanzo.ai, NOT GitHub Actions
+21 -7
View File
@@ -57,14 +57,28 @@ const (
// tip, so a single (even >⅔-stake) validator cannot alone determine the frontier.
// Capped at the beacon-set size for tiny networks.
bootstrapMinAgreeingBeacons = 2
// bootstrapNamingWindow bounds the ancestry the ANCESTOR-TOLERANT frontier tally fetches
// per candidate anchor to resolve the bleeding-edge skew between honest beacons. On a live
// chain that skew is tiny (the canary was ONE block: 2 producers at N, 1 at N+1), so a
// single 256-block window covers it with enormous margin. A ⅔-common height more than this
// far below the highest tip is not a healthy bleeding-edge split — the tally names nothing
// (the loop then retries / fails safe), never a wrong block. Matches the consensus descent's
// per-round fetch size.
// bootstrapNamingWindow is the PER-FETCH ancestry chunk of the ANCESTOR-TOLERANT frontier
// tally. Ancestry returns FULL BLOCKS, so one fetch must stay inside a network message —
// this is a TRANSPORT bound, matching the consensus descent's per-round fetch size.
//
// It was previously ALSO doing two other jobs: the total ancestry budget, and a HEALTH
// HEURISTIC ("a ⅔-common height more than this far below the highest tip is not a healthy
// bleeding-edge split"). That heuristic is true for a LIVE chain and FALSE for a HALTED
// one, where the skew between a straggler and a node that ran on alone is arbitrarily large
// and perfectly healthy — no fork, just a stopped chain. Conflating the three WEDGED
// mainnet 96369: responders were split 1098726 (1 node) / 1098191 (2 nodes), 1098191 IS an
// ancestor of 1098726, and the ⅔-of-responder floor would have named it — but the gap was
// 535 blocks, the single 256-block fetch never reached down far enough for the high tip to
// vouch for it, and the tally named nothing on every retry, forever. Recovery from a halt
// is exactly when this path matters most, and it was disabled precisely then.
// The total budget is now maxNamingDepth; the health heuristic is gone.
bootstrapNamingWindow = 256
// bootstrapMaxNamingDepth is the TOTAL ancestry one anchor may be walked down, in
// bootstrapNamingWindow-sized chunks. Purely a resource bound (with maxNamingAnchors and
// bootstrapNamingTimeout): 32768 covers ~9h of 1s blocks of halt-skew, and the common
// healthy case never fetches at all — a single tip clearing the floor takes the exact fast
// path with zero fetches. Safety does NOT come from this number; see maxNamingDepth().
bootstrapMaxNamingDepth = 32768
// maxNamingAnchors bounds how many DISTINCT reported tips the ancestor-tolerant tally will
// fetch ancestry for in one round. Honest beacons cluster on a handful of adjacent tips, so
// this is never reached in practice; it caps the work a Byzantine swarm reporting many
+52 -11
View File
@@ -243,10 +243,13 @@ type BootstrapPolicy struct {
// CheckpointVerifier authenticates the Checkpoint's authority signature (INVARIANT 4). nil ⇒ a
// configured Checkpoint is NOT trusted (fail closed) — a bare (id,height) is never enough.
CheckpointVerifier CheckpointVerifier
// NamingWindow bounds the ancestry fetched per anchor; MaxAnchors bounds how many distinct
// reported tips are resolved. Both default to the package constants when zero.
NamingWindow int
MaxAnchors int
// NamingWindow is the per-FETCH ancestry chunk (Ancestry returns FULL blocks, so one fetch
// must stay inside a network message); MaxNamingDepth is the TOTAL ancestry a single anchor
// may be walked down in NamingWindow-sized chunks; MaxAnchors bounds how many distinct
// reported tips are resolved. All default to the package constants when zero.
NamingWindow int
MaxNamingDepth int
MaxAnchors int
// NamingTimeout TOTAL-bounds the ancestor-tolerant resolution (all anchor fetches combined) so
// a partition that ANSWERS the frontier query but WITHHOLDS ancestry cannot make the decision
// hang — it returns what it found (or nothing → ErrNoBootstrapQuorum) and the caller's bounded
@@ -299,6 +302,17 @@ func (p *BootstrapPolicy) namingWindow() int {
return bootstrapNamingWindow
}
// maxNamingDepth is the TOTAL ancestry one anchor may be walked down, in namingWindow-sized
// chunks. It is a RESOURCE bound and nothing else. Safety comes from hash-verified ancestry
// (a forged chain cannot link), MinResponders distinct voters, the ⅔-of-RESPONDER-stake floor,
// and the full re-Verify every block gets on the descent — none of which depend on this number.
func (p *BootstrapPolicy) maxNamingDepth() int {
if p.MaxNamingDepth > 0 {
return p.MaxNamingDepth
}
return bootstrapMaxNamingDepth
}
func (p *BootstrapPolicy) maxAnchors() int {
if p.MaxAnchors > 0 {
return p.MaxAnchors
@@ -505,14 +519,41 @@ func (p *BootstrapPolicy) nameFrontier(ctx context.Context, stakeOnTip map[ids.I
break
}
fetches++
refs, err := p.Source.Ancestry(ctx, tip, p.namingWindow())
if err != nil {
continue
}
for _, ref := range refs {
if _, ok := index[ref.ID]; !ok {
index[ref.ID] = ref
// CHUNKED DESCENT. namingWindow is the per-FETCH size (Ancestry returns FULL blocks, so
// one fetch must fit a network message) — NOT the total ancestry worth walking. Keep
// following the parent chain in window-sized chunks up to maxNamingDepth, so a fleet
// that HALTED with a straggler far below the highest tip still resolves its ⅔-backed
// common ancestor. A single un-chunked window silently capped this at 256 and wedged
// mainnet at a 535-block gap. ctx already TOTAL-bounds every fetch (namingTimeout).
cur := tip
for depth := 0; depth < p.maxNamingDepth(); {
// The deadline must bind the WALK, not just each fetch: a Byzantine peer serving a
// long fabricated chain cheaply (or an in-process Source) would otherwise run the
// full depth budget before anyone checked the clock.
if ctx.Err() != nil {
break
}
refs, err := p.Source.Ancestry(ctx, cur, p.namingWindow())
if err != nil || len(refs) == 0 {
break
}
deepest, first := BlockRef{}, true
for _, ref := range refs {
if _, ok := index[ref.ID]; !ok {
index[ref.ID] = ref
}
if first || ref.Height < deepest.Height {
deepest, first = ref, false
}
}
depth += len(refs) // len(refs) >= 1 here, so depth strictly advances -> terminates
if deepest.Parent == ids.Empty {
break // reached genesis
}
if _, covered := index[deepest.Parent]; covered {
break // an earlier anchor's chain already covers everything below
}
cur = deepest.Parent
}
}
if len(index) == 0 {
+172
View File
@@ -916,3 +916,175 @@ func TestBootstrapTrust_EclipseOwnHeightNotNamedRoutesToCaughtUp(t *testing.T) {
require.Equal(t, refs[N].ID, f.ID, "boundary: N is named iff its height is STRICTLY ABOVE MinFrontierHeight")
require.Equal(t, uint64(N), f.Height)
}
// ----- H: HALT-SKEW RECOVERY (mainnet 96369 wedge) ---------------------------
// TestBootstrapTrust_H_HaltSkewDeeperThanOneWindow reproduces the live mainnet 96369 wedge and
// proves the chunked descent fixes it.
//
// SHAPE (measured per-node, in-pod, 2026-07-28): the fleet HALTED below its α=4-of-5 threshold.
// One node ran on alone to 1098726 while two stayed at 1098191; 1098191 IS an ancestor of
// 1098726 (no fork — luxd-3/luxd-4's `latest` IS 1098191, they hold nothing competing), so the
// ⅔-of-RESPONDER floor is satisfied at 1098191 by all three responders and it MUST be named.
//
// It was not. nameFrontier fetched one bootstrapNamingWindow (256) of ancestry per anchor, and
// the gap is 535 — so the high tip's ancestry never reached down far enough to vouch for the
// common block. 1098191 held only its 2 direct responders (2 of 3 = below the ⅔ floor of 2,
// which the strict `>` rejects), the tally named NOTHING, and every retry did the same. The
// node sat at height 0 forever and mainnet could not regain quorum.
//
// The gap here (535) is deliberately > one window (256) and < maxNamingDepth. With the
// single-fetch window this FAILS (frontier is nil → ErrNoBootstrapQuorum); with the chunked
// descent the walk continues past the first chunk and names the common ancestor.
func TestBootstrapTrust_H_HaltSkewDeeperThanOneWindow(t *testing.T) {
const w uint64 = 100
const gap = 535 // luxd-1 1098726 - luxd-3/luxd-4 1098191, the measured mainnet skew
// common is the last block the whole fleet accepted; `ahead` extends it by `gap`.
refs, byID := refChain(1000)
common := refs[1000]
prev := common
for i := 0; i < gap; i++ {
c := childRef(prev)
byID[c.ID] = c
prev = c
}
ahead := prev
require.Equal(t, common.Height+gap, ahead.Height)
require.Greater(t, gap, bootstrapNamingWindow, "gap MUST exceed one fetch window or this proves nothing")
require.Less(t, gap, bootstrapMaxNamingDepth, "gap must stay inside the total depth budget")
beacons := nodeIDs(3) // the three responders with a live C-Chain
policy := &BootstrapPolicy{
TrustedBeacons: equalBeacons(beacons, w),
MinResponses: 3,
Source: &stubAncestry{byID: byID},
}
replies := []BeaconReply{
reply(beacons[0], ahead.ID, w), // luxd-1, ran on alone
reply(beacons[1], common.ID, w), // luxd-3
reply(beacons[2], common.ID, w), // luxd-4
}
f, err := policy.AcceptsFrontier(context.Background(), replies)
require.NoError(t, err, "a fleet split across ONE chain must name a frontier, not wedge")
require.NotNil(t, f)
require.Equal(t, common.ID, f.ID,
"must name the ⅔-backed COMMON ancestor: the high tip vouches for it via ancestry")
require.Equal(t, common.Height, f.Height)
}
// TestBootstrapTrust_H_HaltSkewBeyondDepthStillFailsSafe pins the resource bound's edge: a skew
// DEEPER than maxNamingDepth still names nothing rather than guessing. Depth is a work bound, so
// exhausting it must degrade to the SAME fail-safe as before, never to a wrong block.
func TestBootstrapTrust_H_HaltSkewBeyondDepthStillFailsSafe(t *testing.T) {
const w uint64 = 100
refs, byID := refChain(10)
common := refs[10]
prev := common
for i := 0; i < 64; i++ {
c := childRef(prev)
byID[c.ID] = c
prev = c
}
ahead := prev
beacons := nodeIDs(3)
policy := &BootstrapPolicy{
TrustedBeacons: equalBeacons(beacons, w),
MinResponses: 3,
NamingWindow: 4, // tiny chunk...
MaxNamingDepth: 8, // ...and a depth budget far shallower than the 64-block skew
Source: &stubAncestry{byID: byID},
}
_, err := policy.AcceptsFrontier(context.Background(), []BeaconReply{
reply(beacons[0], ahead.ID, w),
reply(beacons[1], common.ID, w),
reply(beacons[2], common.ID, w),
})
require.Error(t, err, "a skew beyond the depth budget must fail SAFE, never name a guess")
}
// TestBootstrapTrust_H_AcceptanceMatrix pins the owner-specified recovery matrix at the policy
// layer. Each case is the SAME five-validator mainnet shape with a different responder pattern.
//
// The rule being pinned is "highest verifiably-vouched descendant wins", NOT "numerically highest
// tip advertised". A tip whose ancestry does not link into the fleet's chain earns NO credit no
// matter how high it claims to be, so a Byzantine node cannot pull the fleet onto a fabricated
// chain by advertising a tall one.
func TestBootstrapTrust_H_AcceptanceMatrix(t *testing.T) {
const w uint64 = 100
mk := func() (BlockRef, BlockRef, map[ids.ID]BlockRef) {
refs, byID := refChain(600)
common := refs[600]
prev := common
for i := 0; i < 535; i++ { // the measured mainnet skew, > one 256 window
c := childRef(prev)
byID[c.ID] = c
prev = c
}
return common, prev, byID
}
t.Run("1_high_2_low_2_unavailable__names_common_ancestor", func(t *testing.T) {
common, ahead, byID := mk()
b := nodeIDs(3)
p := &BootstrapPolicy{TrustedBeacons: equalBeacons(b, w), MinResponses: 3, Source: &stubAncestry{byID: byID}}
f, err := p.AcceptsFrontier(context.Background(), []BeaconReply{
reply(b[0], ahead.ID, w), reply(b[1], common.ID, w), reply(b[2], common.ID, w)})
require.NoError(t, err)
require.Equal(t, common.ID, f.ID)
})
t.Run("3_high_2_unavailable__names_high_tip", func(t *testing.T) {
_, ahead, byID := mk()
b := nodeIDs(3)
p := &BootstrapPolicy{TrustedBeacons: equalBeacons(b, w), MinResponses: 3, Source: &stubAncestry{byID: byID}}
f, err := p.AcceptsFrontier(context.Background(), []BeaconReply{
reply(b[0], ahead.ID, w), reply(b[1], ahead.ID, w), reply(b[2], ahead.ID, w)})
require.NoError(t, err)
require.Equal(t, ahead.ID, f.ID, "unanimous high tip must be named, not an ancestor")
})
t.Run("fabricated_tall_tip_2_low__rejects_fake_names_common", func(t *testing.T) {
common, _, byID := mk()
// A Byzantine node advertises a tip on a chain of its own that never links into the
// fleet's history. Its ancestry earns credit only for ITS OWN blocks, never for the
// fleet's — so it cannot outvote the two honest low responders.
fakeRefs, fakeByID := refChain(9_000_000)
fake := fakeRefs[9_000_000]
for id, r := range fakeByID {
byID[id] = r
}
b := nodeIDs(3)
p := &BootstrapPolicy{TrustedBeacons: equalBeacons(b, w), MinResponses: 3, Source: &stubAncestry{byID: byID}}
f, err := p.AcceptsFrontier(context.Background(), []BeaconReply{
reply(b[0], fake.ID, w), reply(b[1], common.ID, w), reply(b[2], common.ID, w)})
// The fake tip earns credit ONLY on its own disjoint chain (100), far below the ⅔ floor
// of 200, so it can never be named however tall it claims to be. `common` earns exactly
// 200 from its two honest backers — and the floor is STRICT (`> floor`), so 200 is not
// enough either: the Byzantine node withheld the third vouch by sitting on a chain that
// does not link. Correct outcome is a SAFE HALT, not "fall back to the low tip".
// This is the honest BFT trade: 1 of 3 responders can cost LIVENESS, never SAFETY.
require.Error(t, err, "must halt safely; must NOT follow a taller unvouched chain")
require.Nil(t, f)
})
t.Run("two_conflicting_branches_same_height__halts_safely", func(t *testing.T) {
refs, byID := refChain(600)
common := refs[600]
// Two disjoint branches of equal height off the common block, each backed by one node,
// and NO responder majority on either. Nothing may be named by height tiebreak.
l, r := childRef(common), childRef(common)
byID[l.ID], byID[r.ID] = l, r
b := nodeIDs(3)
p := &BootstrapPolicy{
TrustedBeacons: equalBeacons(b, w), MinResponses: 3,
MinFrontierHeight: common.Height, // node already holds `common`; only a tip AHEAD may be named
Source: &stubAncestry{byID: byID},
}
_, err := p.AcceptsFrontier(context.Background(), []BeaconReply{
reply(b[0], l.ID, w), reply(b[1], r.ID, w), reply(b[2], common.ID, w)})
require.Error(t, err, "conflicting equal-height branches must halt safely, never pick by height")
})
}
+23 -19
View File
@@ -26,7 +26,7 @@ require (
github.com/huin/goupnp v1.3.0
github.com/jackpal/gateway v1.1.1
github.com/jackpal/go-nat-pmp v1.0.2
github.com/luxfi/consensus v1.36.9
github.com/luxfi/consensus v1.36.12
github.com/luxfi/crypto v1.20.2
github.com/luxfi/database v1.21.1
github.com/luxfi/ids v1.3.2
@@ -56,13 +56,13 @@ require (
go.opentelemetry.io/otel/trace v1.44.0 // indirect
go.uber.org/goleak v1.3.0
go.uber.org/mock v0.6.0
golang.org/x/crypto v0.52.0
golang.org/x/crypto v0.54.0
golang.org/x/exp v0.0.0-20260529124908-c761662dc8c9 // indirect
golang.org/x/mod v0.36.0
golang.org/x/net v0.55.0
golang.org/x/sync v0.20.0
golang.org/x/mod v0.37.0
golang.org/x/net v0.57.0
golang.org/x/sync v0.22.0
golang.org/x/time v0.15.0
golang.org/x/tools v0.45.0
golang.org/x/tools v0.47.0
gonum.org/v1/gonum v0.17.0
google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
@@ -90,7 +90,7 @@ require (
github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/klauspost/compress v1.18.6
github.com/klauspost/compress v1.19.1
github.com/kr/pretty v0.3.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
@@ -107,8 +107,8 @@ require (
github.com/yusufpapurcu/wmi v1.2.4 // indirect
go.opentelemetry.io/otel/metric v1.44.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/sys v0.45.0 // indirect
golang.org/x/text v0.37.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/text v0.40.0 // indirect
)
require (
@@ -130,7 +130,7 @@ require (
github.com/luxfi/genesis/pkg/genesis/security v1.13.8
github.com/luxfi/geth v1.20.1
github.com/luxfi/go-bip39 v1.2.0
github.com/luxfi/keys v1.4.1
github.com/luxfi/kms v1.12.10
github.com/luxfi/math/safe v0.0.1
github.com/luxfi/net v0.1.1
github.com/luxfi/p2p v1.22.1
@@ -145,33 +145,34 @@ require (
github.com/luxfi/utils v1.3.1
github.com/luxfi/utxo v0.5.8
github.com/luxfi/validators v1.3.1
github.com/luxfi/vm v1.3.1
github.com/luxfi/vm v1.3.3
github.com/luxfi/warp v1.24.1
github.com/luxfi/zap v1.2.6
github.com/luxfi/zwing v0.6.1
github.com/nbutton23/zxcvbn-go v0.0.0-20210217022336-fa2cb2858354
github.com/zap-proto/http v0.0.0-20260506200741-fd6047874433
github.com/valyala/fasthttp v1.73.0
github.com/zap-proto/http v0.3.0
go.uber.org/zap v1.27.1
)
require (
capnproto.org/go/capnp/v3 v3.0.1-alpha.2 // indirect
filippo.io/edwards25519 v1.2.0 // indirect
filippo.io/hpke v0.4.0 // indirect
github.com/andybalholm/brotli v1.2.2 // indirect
github.com/aws/aws-sdk-go-v2 v1.41.5 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.5 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 // indirect
github.com/aws/aws-sdk-go-v2/config v1.32.13 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.19.13 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.21 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.6 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.18 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.10 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.18 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.2 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.3 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.9 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.14 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.18 // indirect
@@ -180,7 +181,6 @@ require (
github.com/btcsuite/btcd/btcec/v2 v2.5.0 // indirect
github.com/btcsuite/btcd/chainhash/v2 v2.0.0 // indirect
github.com/cenkalti/backoff v2.2.1+incompatible // indirect
github.com/colega/zeropool v0.0.0-20230505084239-6fb4a4f75381 // indirect
github.com/decred/dcrd/crypto/blake256 v1.1.0 // indirect
github.com/go-ini/ini v1.67.0 // indirect
github.com/goccy/go-yaml v1.19.2 // indirect
@@ -191,9 +191,11 @@ require (
github.com/hanzos3/go-sdk v1.0.2 // indirect
github.com/klauspost/crc32 v1.3.0 // indirect
github.com/luxfi/age v1.6.0 // indirect
github.com/luxfi/bft v0.1.5 // indirect
github.com/luxfi/corona v0.10.4 // indirect
github.com/luxfi/crypto/ipa v1.2.4 // indirect
github.com/luxfi/dkg v0.3.5 // indirect
github.com/luxfi/keys v1.4.1 // indirect
github.com/luxfi/lattice/v7 v7.1.4 // indirect
github.com/luxfi/lens v0.2.1 // indirect
github.com/luxfi/light v1.0.0 // indirect
@@ -214,6 +216,8 @@ require (
github.com/philhofer/fwd v1.2.0 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/tinylib/msgp v1.6.4 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/zap-proto/go v1.1.0 // indirect
go.mongodb.org/mongo-driver v1.17.9 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
)
+50 -28
View File
@@ -1,7 +1,5 @@
c2sp.org/CCTV/age v0.0.0-20251208015420-e9274a7bdbfd h1:ZLsPO6WdZ5zatV4UfVpr7oAwLGRZ+sebTUruuM4Ra3M=
c2sp.org/CCTV/age v0.0.0-20251208015420-e9274a7bdbfd/go.mod h1:SrHC2C7r5GkDk8R+NFVzYy/sdj0Ypg9htaPXQq5Cqeo=
capnproto.org/go/capnp/v3 v3.0.1-alpha.2 h1:W/cf+XEArUSwcBBE/9wS2NpWDkM5NLQOjmzEiHZpYi0=
capnproto.org/go/capnp/v3 v3.0.1-alpha.2/go.mod h1:2vT5D2dtG8sJGEoEKU17e+j7shdaYp1Myl8X03B3hmc=
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
@@ -19,10 +17,14 @@ github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA
github.com/StephenButtolph/canoto v0.17.3 h1:lvsnYD4b96vD1knnmp1xCmZqfYpY/jSeRozGdOfdvGI=
github.com/StephenButtolph/canoto v0.17.3/go.mod h1:IcnAHC6nJUfQFVR9y60ko2ecUqqHHSB6UwI9NnBFZnE=
github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII=
github.com/andybalholm/brotli v1.2.2 h1:HzTuoo2ErYQqf5qvcJInB8uvqSVxRttzkFexPWtnceM=
github.com/andybalholm/brotli v1.2.2/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/aws/aws-sdk-go-v2 v1.41.5 h1:dj5kopbwUsVUVFgO4Fi5BIT3t4WyqIDjGKCangnV/yY=
github.com/aws/aws-sdk-go-v2 v1.41.5/go.mod h1:mwsPRE8ceUUpiTgF7QmQIJ7lgsKUPQOUl3o72QBrE1o=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.5 h1:zWFmPmgw4sveAYi1mRqG+E/g0461cJ5M4bJ8/nc6d3Q=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.5/go.mod h1:nVUlMLVV8ycXSb7mSkcNu9e3v/1TJq2RTlrPwhYWr5c=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 h1:eBMB84YGghSocM7PsjmmPffTa+1FBUeNvGvFou6V/4o=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8/go.mod h1:lyw7GFp3qENLh7kwzf7iMzAxDn+NzjXEAGjKS2UOKqI=
github.com/aws/aws-sdk-go-v2/config v1.32.13 h1:5KgbxMaS2coSWRrx9TX/QtWbqzgQkOdEa3sZPhBhCSg=
github.com/aws/aws-sdk-go-v2/config v1.32.13/go.mod h1:8zz7wedqtCbw5e9Mi2doEwDyEgHcEE9YOJp6a8jdSMY=
github.com/aws/aws-sdk-go-v2/credentials v1.19.13 h1:mA59E3fokBvyEGHKFdnpNNrvaR351cqiHgRg+JzOSRI=
@@ -37,16 +39,24 @@ github.com/aws/aws-sdk-go-v2/internal/ini v1.8.6 h1:qYQ4pzQ2Oz6WpQ8T3HvGHnZydA72
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.6/go.mod h1:O3h0IK87yXci+kg6flUKzJnWeziQUKciKrLjcatSNcY=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.18 h1:eZioDaZGJ0tMM4gzmkNIO2aAoQd+je7Ug7TkvAzlmkU=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.18/go.mod h1:CCXwUKAJdoWr6/NcxZ+zsiPr6oH/Q5aTooRGYieAyj4=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22 h1:rWyie/PxDRIdhNf4DzRk0lvjVOqFJuNnO8WwaIRVxzQ=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22/go.mod h1:zd/JsJ4P7oGfUhXn1VyLqaRZwPmZwg44Jf2dS84Dm3Y=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7 h1:5EniKhLZe4xzL7a+fU3C2tfUN4nWIqlLesfrjkuPFTY=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7/go.mod h1:x0nZssQ3qZSnIcePWLvcoFisRXJzcTVvYpAAdYX8+GI=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.10 h1:fJvQ5mIBVfKtiyx0AHY6HeWcRX5LGANLpq8SVR+Uazs=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.10/go.mod h1:Kzm5e6OmNH8VMkgK9t+ry5jEih4Y8whqs+1hrkxim1I=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13 h1:JRaIgADQS/U6uXDqlPiefP32yXTda7Kqfx+LgspooZM=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13/go.mod h1:CEuVn5WqOMilYl+tbccq8+N2ieCy0gVn3OtRb0vBNNM=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21 h1:c31//R3xgIJMSC8S6hEVq+38DcvUlgFY0FM6mSI5oto=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21/go.mod h1:r6+pf23ouCB718FUxaqzZdbpYFyDtehyZcmP5KL9FkA=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.18 h1:/A/xDuZAVD2BpsS2fftFRo/NoEKQJ8YTnJDEHBy2Gtg=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.18/go.mod h1:hWe9b4f+djUQGmyiGEeOnZv69dtMSgpDRIvNMvuvzvY=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21 h1:ZlvrNcHSFFWURB8avufQq9gFsheUgjVD9536obIknfM=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21/go.mod h1:cv3TNhVrssKR0O/xxLJVRfd2oazSnZnkUeTf6ctUwfQ=
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.2 h1:M1A9AjcFwlxTLuf0Faj88L8Iqw0n/AJHjpZTQzMMsSc=
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.2/go.mod h1:KsdTV6Q9WKUZm2mNJnUFmIoXfZux91M3sr/a4REX8e0=
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.3 h1:HwxWTbTrIHm5qY+CAEur0s/figc3qwvLWsNkF4RPToo=
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.3/go.mod h1:uoA43SdFwacedBfSgfFSjjCvYe8aYBS7EnU5GZ/YKMM=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.9 h1:QKZH0S178gCmFEgst8hN0mCX1KxLgHBKKY/CLqwP8lg=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.9/go.mod h1:7yuQJoT+OoH8aqIxw9vwF+8KpvLZ8AWmvmUWHsGQZvI=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.14 h1:GcLE9ba5ehAQma6wlopUesYg/hbcOhFNWTjELkiWkh4=
@@ -111,8 +121,6 @@ github.com/cockroachdb/redact v1.1.8 h1:8eVLLj6juKxiKrAEw2b8cJvNqWq++U8WOfQFuL7K
github.com/cockroachdb/redact v1.1.8/go.mod h1:GceHHpJ0rMDpYARL5In88Alq/xMBUtVlz7Qxix6ZVkw=
github.com/cockroachdb/tokenbucket v0.0.0-20250429170803-42689b6311bb h1:3bCgBvB8PbJVMX1ouCcSIxvsqKPYM7gs72o0zC76n9g=
github.com/cockroachdb/tokenbucket v0.0.0-20250429170803-42689b6311bb/go.mod h1:7nc4anLGjupUW/PeY5qiNYsdNXj7zopG+eqsS7To5IQ=
github.com/colega/zeropool v0.0.0-20230505084239-6fb4a4f75381 h1:d5EKgQfRQvO97jnISfR89AiCCCJMwMFoSxUiU0OGCRU=
github.com/colega/zeropool v0.0.0-20230505084239-6fb4a4f75381/go.mod h1:OU76gHeRo8xrzGJU3F3I1CqX1ekM8dfJw0+wPeMwnp0=
github.com/consensys/gnark-crypto v0.20.1 h1:PXDUBvk8AzhvWowHLWBEAfUQcV1/aZgWIqD6eMpXmDg=
github.com/consensys/gnark-crypto v0.20.1/go.mod h1:RBWrSgy+IDbGR69RRV313th3M/aZU1ubk2om+qHuTSc=
github.com/cosmos/go-bip39 v1.0.0 h1:pcomnQdrdH22njcAatO0yWojsUnCO3y2tNoV1cb6hHY=
@@ -280,8 +288,8 @@ github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlT
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4=
github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao=
github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
@@ -305,6 +313,8 @@ github.com/luxfi/api v1.1.1 h1:CXD7m0quPmUm+Qw35TrF+E7b0Fq4qz9gHfrZ5gyrjHU=
github.com/luxfi/api v1.1.1/go.mod h1:g6J0iohVqaIj2aO1u/ZJPqjiX2tog0NM3/SBf7wJ4cA=
github.com/luxfi/atomic v1.0.0 h1:xUV60MuzRvXngaQ1sM0yVC2v4TRoLlUGkkH7M9PS4yw=
github.com/luxfi/atomic v1.0.0/go.mod h1:0G2mTlQ6TXWHICUHrUUPu1/qAiIyR4gSZ2tva9ci/bI=
github.com/luxfi/bft v0.1.5 h1:5xVLPkog4e5LTgaVlb9pgxA0EWE6tkrKwHPZVRz+RZw=
github.com/luxfi/bft v0.1.5/go.mod h1:5I8Ft8yA69xZlDe3RB0i4MgbqFKLZe65o/sha8JuKvU=
github.com/luxfi/cache v1.3.1 h1:grQhi/B5GKypG7avDMeY143QTgFbfEvQICKNIh1Cw6U=
github.com/luxfi/cache v1.3.1/go.mod h1:2MokdbeNUy/9O3mdREWkE6BiN7tRvePkXiKkcb+4M7g=
github.com/luxfi/chains v1.7.9 h1:sDF/WQn56o/4i1/uCS8i5ENDMuyWi8+iVcnYGAstMDM=
@@ -315,8 +325,10 @@ github.com/luxfi/compress v0.1.1 h1:cQjRYQFRrw8HinjW1T8FmKgqdRwrFYDCdecc1t0i+uQ=
github.com/luxfi/compress v0.1.1/go.mod h1:d4CkRRmwPzafIp57Jxra3RmAmNwNwU8Oc0QBBRlTqGo=
github.com/luxfi/concurrent v0.1.1 h1:LkAmNXybOsAm97nFILqMDBp/YLgleuLmyGomFT7YNxU=
github.com/luxfi/concurrent v0.1.1/go.mod h1:GlkfiJtPBpatNxvROf7hkPi4gsnmdHGmqhDnVWFrkZE=
github.com/luxfi/consensus v1.36.9 h1:un4iRecrXQqePGEtVNJO4uc76n4GhxOeq+YAaoQB0TE=
github.com/luxfi/consensus v1.36.9/go.mod h1:iwlx11D7BRdEaUqyUvueAe2Fit1K1k26avBK2eLYdnc=
github.com/luxfi/consensus v1.36.10 h1:LJPvHc2zL7VFhbqfYu11rA50yvxzTK17Z4mRN04HQhU=
github.com/luxfi/consensus v1.36.10/go.mod h1:iwlx11D7BRdEaUqyUvueAe2Fit1K1k26avBK2eLYdnc=
github.com/luxfi/consensus v1.36.12 h1:MJKq6j9+04Y2NC+NArGQgv5QNcjprYCk+gppCz1PdSg=
github.com/luxfi/consensus v1.36.12/go.mod h1:mCtA6k31FVW/H6rsYQ7V8VvS198WsFS8oFo/z3IMI+A=
github.com/luxfi/constants v1.6.2 h1:pXHdKIFbfE9qX4xOjq2LxYvagNhhNvspUVEbPcIEKfA=
github.com/luxfi/constants v1.6.2/go.mod h1:r0oH8C/+r/XFYBq1AJxt6zWRKKRKgDzrEMop/CCs9rI=
github.com/luxfi/container v0.2.1 h1:MTnfKXzS5+oxV5jKZerdOxSA6iMPaQI9/FWGufizzaw=
@@ -351,6 +363,8 @@ github.com/luxfi/keychain v1.1.1 h1:dTYEPy6CGVC1sogMci4iJogUvW6VdTmemplQdzRqnAs=
github.com/luxfi/keychain v1.1.1/go.mod h1:hAzBcwxGumtoYrM5hfhwdt8wE0p7r2JCd5AxswqfkoY=
github.com/luxfi/keys v1.4.1 h1:2Zcoovaz9OLPz7m7VGXfRrGnrlqt0GeUpJclsPBi4EU=
github.com/luxfi/keys v1.4.1/go.mod h1:P8EUP5DKrR1SUZBGZjDT3rWcp2P1miUlVh7IBRNBphU=
github.com/luxfi/kms v1.12.10 h1:fXgisnBkixFSrqhOFbZVRQkf8cX4q4vS3Gix1qjL3PQ=
github.com/luxfi/kms v1.12.10/go.mod h1:CCcWDXIlDT2TwfYAUxedLyzShFRJAlVlp15zQ2L3CrU=
github.com/luxfi/lattice/v7 v7.1.4 h1:hQR02M6cHTAV5+joOPi9gb9Gm+z/hKJnhJF4IlciIJs=
github.com/luxfi/lattice/v7 v7.1.4/go.mod h1:DmIQFi3mJiehVsR235l1NKYEU0JhU649OX5p7gMEW2c=
github.com/luxfi/lens v0.2.1 h1:5Qd0GdjbM+XUVgwDbZ452tKkR7yeE8QnBTHHaH8fJNY=
@@ -423,8 +437,10 @@ github.com/luxfi/validators v1.3.1 h1:+/7j0CTXlMKyaSLFM+gd6Fq64/edORJfrD/xGnf7Xc
github.com/luxfi/validators v1.3.1/go.mod h1:sIQyUZOvXoJ/9/RCOhHSzjumZIoxiqacNOn5mQWVozU=
github.com/luxfi/version v1.0.1 h1:T/1KYWEMmsrNQk7pN7PFPAwh/7XbeX7cFAKLBqI37Sk=
github.com/luxfi/version v1.0.1/go.mod h1:Y5fPkQ2DB0XRBCxgSPXp4ISzL1/jptKnmFknShRJCyg=
github.com/luxfi/vm v1.3.1 h1:rLCbygaajehVkUoJ5czwhpUAJaC5J5okq3p+j2QSPSo=
github.com/luxfi/vm v1.3.1/go.mod h1:uViH3COP8hhCbj42v1MTohkPCDwYQCZanNIOb/StWqY=
github.com/luxfi/vm v1.3.1 h1:U2Bv7IDRFv8JtjUARfY53ZnOPRB2iPrPHSMDCQ+T0Js=
github.com/luxfi/vm v1.3.1/go.mod h1:6YR/uFV2FoofmogQShv+HM7V8alz5rbrQYjp5w0bNVA=
github.com/luxfi/vm v1.3.3 h1:BEBamD9VUcPG8mtL2Ga0Us1lTQ+ubaZqng/PZjNqzPc=
github.com/luxfi/vm v1.3.3/go.mod h1:v2XjW0yymxHwy1xPsSz338s0J8nRqOIb6RiDf0wUEgk=
github.com/luxfi/warp v1.24.1 h1:9F+z6fy4sxmXp+3LlR9m3TiZ8Dfthh+s5KOeewSJL30=
github.com/luxfi/warp v1.24.1/go.mod h1:kRGQDt6EB3oRLYo71aXqnmJuCBVfND3oQ5/2miaLoyg=
github.com/luxfi/zap v1.2.6 h1:NBpbm9Gib41Oi/XAkAZKQ3hb+xCafo7JsrUjw+bKiAc=
@@ -583,27 +599,33 @@ github.com/tidwall/sjson v1.2.5 h1:kLy8mja+1c9jlljvWTlSazM7cKDRfJuR/bOJhcY5NcY=
github.com/tidwall/sjson v1.2.5/go.mod h1:Fvgq9kS/6ociJEDnK0Fk1cpYF4FIW6ZF7LAe+6jwd28=
github.com/tinylib/msgp v1.6.4 h1:mOwYbyYDLPj35mkA2BjjYejgJk9BuHxDdvRnb6v2ZcQ=
github.com/tinylib/msgp v1.6.4/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA=
github.com/tj/assert v0.0.3 h1:Df/BlaZ20mq6kuai7f5z2TvPFiwC3xaWJSDQNiIS3Rk=
github.com/tj/assert v0.0.3/go.mod h1:Ne6X72Q+TB1AteidzQncjw9PabbMp4PBMZ1k+vd1Pvk=
github.com/tklauser/go-sysconf v0.4.0 h1:7H0uAN+7RkwWRaxhYXDLqa5V3LPrJeV8wmD9dRUgPQU=
github.com/tklauser/go-sysconf v0.4.0/go.mod h1:8mTNWyog7H+MpKijp4VmKJAd2bbYQ2zuUwkYRbUArPI=
github.com/tklauser/numcpus v0.12.0 h1:NR85qdvHA9pFse3x3weVZ0r0ST8R6l5RHbZrlRaqob4=
github.com/tklauser/numcpus v0.12.0/go.mod h1:ABHeXzJnr/qqwguhClkZKT1/8VABcYrsyUiUGobwWJg=
github.com/urfave/cli/v2 v2.27.7 h1:bH59vdhbjLv3LAvIu6gd0usJHgoTTPhCFib8qqOwXYU=
github.com/urfave/cli/v2 v2.27.7/go.mod h1:CyNAG/xg+iAOg0N4MPGZqVmv2rCoP267496AOXUZjA4=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.73.0 h1:ocTOORnBWtJ+P8t/6wAjdkchMzdfHmWx2VD/DPbgZ7s=
github.com/valyala/fasthttp v1.73.0/go.mod h1:EtXQDHaR+5P18p8wqDRFpUhxr108Ga9mXvVJXHRrN2k=
github.com/wlynxg/anet v0.0.5 h1:J3VJGi1gvo0JwZ/P1/Yc/8p63SoW98B5dHkYDmpgvvU=
github.com/wlynxg/anet v0.0.5/go.mod h1:eay5PRQr7fIVAMbTbchTnO9gG65Hg/uYGdc7mguHxoA=
github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
github.com/xrash/smetrics v0.0.0-20250705151800-55b8f293f342 h1:FnBeRrxr7OU4VvAzt5X7s6266i6cSVkkFPS0TuXWbIg=
github.com/xrash/smetrics v0.0.0-20250705151800-55b8f293f342/go.mod h1:Ohn+xnUBiLI6FVj/9LpzZWtj1/D6lUovWYBkxHVV3aM=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.4.1/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
github.com/zap-proto/http v0.0.0-20260506200741-fd6047874433 h1:7WsCr/pZvWozimdYNffL3B9K6gLr8w0Z7WAi1+eZWtc=
github.com/zap-proto/http v0.0.0-20260506200741-fd6047874433/go.mod h1:xySyVTIwjknmVE+p+6rukkX86rFZtXeAu/QY7KXMYqw=
github.com/zap-proto/go v1.1.0 h1:DDGhuTBNqkmNax7QV/VvkInJR7hiOcybwUjN3YBt8fg=
github.com/zap-proto/go v1.1.0/go.mod h1:914SNGTH6Rv3Yu1MweWJBPEN8FZlo5C39QyhaB0C7Q0=
github.com/zap-proto/http v0.3.0 h1:l7DvlngiYqmzNY6fzyRYw2ZIAhF35FqwOe6mvAOqpMg=
github.com/zap-proto/http v0.3.0/go.mod h1:UYfGhDDCetgxs65XSev8Lpf65COg5vKQK+cWwZGh4zQ=
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
github.com/zeebo/blake3 v0.2.4 h1:KYQPkhpRtcqh0ssGYcKLG1JYvddkEA8QwCM/yBqhaZI=
@@ -642,16 +664,16 @@ golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnf
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/exp v0.0.0-20260529124908-c761662dc8c9 h1:4d4PbuBNwaxMXkXI8yiIYjydtMU+04RHeuSxJdgKftM=
golang.org/x/exp v0.0.0-20260529124908-c761662dc8c9/go.mod h1:d2fgXJLVs4dYDHUk5lwMIfzRzSrWCfGZb0ZqeLa/Vcw=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.5.1/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro=
golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
golang.org/x/net v0.0.0-20180719180050-a680a1efc54d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
@@ -666,15 +688,15 @@ golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT
golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/net v0.0.0-20220607020251-c690dde0001d/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -697,8 +719,8 @@ golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -706,8 +728,8 @@ golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -717,8 +739,8 @@ golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roY
golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.1.8/go.mod h1:nABZi5QlRsZVlzPpHl034qft6wpY4eDcsTt5AaioBiU=
golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8=
golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0=
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+5 -1
View File
@@ -19,6 +19,7 @@ import (
"net/http"
"os"
"github.com/valyala/fasthttp/fasthttpadaptor"
zaphttp "github.com/zap-proto/http"
log "github.com/luxfi/log"
@@ -53,7 +54,10 @@ func startZapRPCListener(logger log.Logger, handler http.Handler, addr string) *
if addr == "" {
return nil
}
srv := &zaphttp.Server{Addr: addr, Handler: handler}
// zap-proto/http >= v0.2.0 takes a fasthttp.RequestHandler. Adapt the very
// same net/http handler chain the HTTP listener serves, so the two
// transports stay behaviourally identical — only the wire encoding differs.
srv := &zaphttp.Server{Addr: addr, Handler: fasthttpadaptor.NewFastHTTPHandler(handler)}
go func() {
logger.Info("ZAP-RPC API listening", log.UserString("addr", addr))
if err := srv.ListenAndServe(); err != nil {
+24 -9
View File
@@ -9,6 +9,7 @@ import (
"testing"
"time"
"github.com/valyala/fasthttp"
zaphttp "github.com/zap-proto/http"
log "github.com/luxfi/log"
@@ -22,8 +23,8 @@ func TestZapRPCListenAddr(t *testing.T) {
val string
want string
}{
{set: false, want: ""}, // unset → disabled (default)
{set: true, val: "", want: ""}, // empty → disabled
{set: false, want: ""}, // unset → disabled (default)
{set: true, val: "", want: ""}, // empty → disabled
{set: true, val: "off", want: ""},
{set: true, val: "OFF", want: ""},
{set: true, val: "0", want: ""},
@@ -74,18 +75,32 @@ func TestStartZapRPCListener_RoundTrip(t *testing.T) {
// Give the goroutine a beat to bind.
time.Sleep(150 * time.Millisecond)
client := &http.Client{Transport: zaphttp.NewTransport(addr)}
resp, err := client.Post("http://"+addr+"/v1/bc/C/rpc", "application/json", nil)
if err != nil {
// zap-proto/http >= v0.2.0 speaks fasthttp on both ends: Transport.Do takes
// a fasthttp request/response pair rather than implementing
// http.RoundTripper. The round trip being asserted is unchanged — a real
// ZAP request over the wire must reach the net/http handler the listener
// was given, through the fasthttpadaptor bridge in startZapRPCListener.
// v0.3.0 replaced NewTransport with Dial, mirroring net.Dial: the network is
// a VALUE ("tcp", "unix") rather than a family of constructors.
transport := zaphttp.Dial("tcp", addr)
defer transport.CloseIdleConnections()
req, resp := fasthttp.AcquireRequest(), fasthttp.AcquireResponse()
defer fasthttp.ReleaseRequest(req)
defer fasthttp.ReleaseResponse(resp)
req.SetRequestURI("http://" + addr + "/v1/bc/C/rpc")
req.Header.SetMethod(http.MethodPost)
req.Header.SetContentType("application/json")
if err := transport.Do(req, resp); err != nil {
t.Fatalf("ZAP round-trip POST failed: %v", err)
}
defer resp.Body.Close()
got, _ := io.ReadAll(resp.Body)
if string(got) != body {
if got := string(resp.Body()); got != body {
t.Fatalf("ZAP round-trip body = %q, want %q", got, body)
}
if ct := resp.Header.Get("Content-Type"); ct != "application/json" {
if ct := string(resp.Header.ContentType()); ct != "application/json" {
t.Fatalf("ZAP round-trip Content-Type = %q, want application/json", ct)
}
}
+126
View File
@@ -0,0 +1,126 @@
// Copyright (C) 2019-2026, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// logger_level_test.go — admin.setLoggerLevel / admin.getLoggerLevel must actually
// move a live logger's level.
//
// They did not. SetLoggerLevel computed the logger names and threw them away
// (`loggerNames := a.getLoggerNames(...); _ = loggerNames`) and getLogLevels returned
// an empty map unconditionally, so BOTH endpoints answered 200 OK having done nothing.
// That is why the 2026-07-28 devnet/testnet build-loop diagnosis had to be run off boot
// logs: raising a running node's log level was impossible.
//
// The tests drive the REAL log.Factory (the same one the node builds), not a double,
// so a passing SetLoggerLevel means the level the logger actually filters on moved.
package admin
import (
"context"
"testing"
"github.com/stretchr/testify/require"
apiadmin "github.com/luxfi/api/admin"
"github.com/luxfi/log"
)
// newLevelTestService returns an admin service over a real log.Factory that already
// holds one registered logger — the shape the node runs in.
func newLevelTestService(t *testing.T, loggerName string) *Service {
t.Helper()
factory := log.NewFactory()
t.Cleanup(factory.Close)
_, err := factory.Make(loggerName)
require.NoError(t, err)
return &Service{Config: Config{Log: log.Noop(), LogFactory: factory}}
}
// TestSetLoggerLevel_MovesTheLevelAndGetReportsIt is the regression: set, then read
// back through the API. Before the fix the set was discarded and the get returned an
// empty map, so both halves silently lied.
func TestSetLoggerLevel_MovesTheLevelAndGetReportsIt(t *testing.T) {
require := require.New(t)
ctx := context.Background()
svc := newLevelTestService(t, "C")
_, err := svc.SetLoggerLevel(ctx, &apiadmin.SetLoggerLevelArgs{
LoggerName: "C",
LogLevel: "debug",
DisplayLevel: "error",
})
require.NoError(err)
reply, err := svc.GetLoggerLevel(ctx, &apiadmin.GetLoggerLevelArgs{LoggerName: "C"})
require.NoError(err)
require.Equal(
map[string]apiadmin.LogAndDisplayLevels{"C": {
LogLevel: log.DebugLevel.String(),
DisplayLevel: log.ErrorLevel.String(),
}},
reply.LoggerLevels,
"setLoggerLevel must move the live logger's level and getLoggerLevel must report it",
)
// The factory is the single source of truth — assert against it directly too, so a
// getLoggerLevel that merely echoed the request back could not pass this test.
logLevel, err := svc.LogFactory.GetLogLevel("C")
require.NoError(err)
require.Equal(log.DebugLevel, logLevel)
}
// TestSetLoggerLevel_OneLevelAtATime pins that omitting a level leaves it alone rather
// than resetting it to the zero Level.
func TestSetLoggerLevel_OneLevelAtATime(t *testing.T) {
require := require.New(t)
ctx := context.Background()
svc := newLevelTestService(t, "node")
_, err := svc.SetLoggerLevel(ctx, &apiadmin.SetLoggerLevelArgs{
LoggerName: "node", LogLevel: "trace", DisplayLevel: "warn",
})
require.NoError(err)
// Only displayLevel this time — logLevel must survive.
_, err = svc.SetLoggerLevel(ctx, &apiadmin.SetLoggerLevelArgs{
LoggerName: "node", DisplayLevel: "fatal",
})
require.NoError(err)
reply, err := svc.GetLoggerLevel(ctx, &apiadmin.GetLoggerLevelArgs{LoggerName: "node"})
require.NoError(err)
require.Equal(log.TraceLevel.String(), reply.LoggerLevels["node"].LogLevel)
require.Equal(log.FatalLevel.String(), reply.LoggerLevels["node"].DisplayLevel)
}
// TestLoggerLevel_RejectsUnservableAndInvalidArgs — every refusal is explicit. log.Factory
// addresses loggers BY NAME and exposes no enumeration, so an empty name cannot be served;
// returning 200 OK for it is the bug, not the contract.
func TestLoggerLevel_RejectsUnservableAndInvalidArgs(t *testing.T) {
require := require.New(t)
ctx := context.Background()
svc := newLevelTestService(t, "C")
_, err := svc.SetLoggerLevel(ctx, &apiadmin.SetLoggerLevelArgs{LogLevel: "debug"})
require.ErrorIs(err, errNoLoggerName)
_, err = svc.GetLoggerLevel(ctx, &apiadmin.GetLoggerLevelArgs{})
require.ErrorIs(err, errNoLoggerName)
_, err = svc.SetLoggerLevel(ctx, &apiadmin.SetLoggerLevelArgs{LoggerName: "C"})
require.ErrorIs(err, errNoLogLevel)
// An unparseable level must be refused BEFORE anything is mutated.
before, err := svc.LogFactory.GetLogLevel("C")
require.NoError(err)
_, err = svc.SetLoggerLevel(ctx, &apiadmin.SetLoggerLevelArgs{LoggerName: "C", LogLevel: "loud"})
require.Error(err)
after, err := svc.LogFactory.GetLogLevel("C")
require.NoError(err)
require.Equal(before, after, "a rejected level must leave the logger untouched")
}
+61 -8
View File
@@ -40,6 +40,7 @@ const (
var (
errAliasTooLong = errors.New("alias length is too long")
errNoLogLevel = errors.New("need to specify either displayLevel or logLevel")
errNoLoggerName = errors.New("need to specify loggerName: loggers are addressed by name and cannot be enumerated")
)
// ChainTracker is the interface for tracking chains at runtime.
@@ -209,11 +210,39 @@ func (a *Service) SetLoggerLevel(ctx context.Context, args *apiadmin.SetLoggerLe
return nil, errNoLogLevel
}
// Parse before mutating: a rejected level must leave every logger untouched.
var logLevel, displayLevel log.Level
if args.LogLevel != "" {
var err error
if logLevel, err = log.ToLevel(args.LogLevel); err != nil {
return nil, err
}
}
if args.DisplayLevel != "" {
var err error
if displayLevel, err = log.ToLevel(args.DisplayLevel); err != nil {
return nil, err
}
}
loggerNames, err := a.getLoggerNames(args.LoggerName)
if err != nil {
return nil, err
}
a.lock.Lock()
defer a.lock.Unlock()
loggerNames := a.getLoggerNames(args.LoggerName)
_ = loggerNames
for _, name := range loggerNames {
// Only the levels the caller supplied — an omitted level keeps its value
// instead of being reset to the zero Level.
if args.LogLevel != "" {
a.LogFactory.SetLogLevel(name, logLevel)
}
if args.DisplayLevel != "" {
a.LogFactory.SetDisplayLevel(name, displayLevel)
}
}
return &apiadmin.EmptyReply{}, nil
}
@@ -225,10 +254,14 @@ func (a *Service) GetLoggerLevel(ctx context.Context, args *apiadmin.GetLoggerLe
log.String("loggerName", args.LoggerName),
)
loggerNames, err := a.getLoggerNames(args.LoggerName)
if err != nil {
return nil, err
}
a.lock.RLock()
defer a.lock.RUnlock()
loggerNames := a.getLoggerNames(args.LoggerName)
loggerLevels, err := a.getLogLevels(loggerNames)
if err != nil {
return nil, err
@@ -401,15 +434,35 @@ func (a *Service) GetTrackedChains(ctx context.Context) (*apiadmin.GetTrackedCha
return &apiadmin.GetTrackedChainsReply{TrackedChains: trackedChains}, nil
}
func (a *Service) getLoggerNames(loggerName string) []string {
if len(loggerName) == 0 {
return []string{}
// getLoggerNames resolves the loggerName argument to the loggers to act on — the one
// place either logger-level endpoint decides what it is addressing.
//
// log.Factory addresses loggers BY NAME and exposes no enumeration, so the "every
// logger" form (an empty name) cannot be served. Refuse it explicitly: answering 200 OK
// while doing nothing is what made these endpoints unusable.
func (a *Service) getLoggerNames(loggerName string) ([]string, error) {
if loggerName == "" {
return nil, errNoLoggerName
}
return []string{loggerName}
return []string{loggerName}, nil
}
func (a *Service) getLogLevels(loggerNames []string) (map[string]apiadmin.LogAndDisplayLevels, error) {
loggerLevels := make(map[string]apiadmin.LogAndDisplayLevels)
loggerLevels := make(map[string]apiadmin.LogAndDisplayLevels, len(loggerNames))
for _, name := range loggerNames {
logLevel, err := a.LogFactory.GetLogLevel(name)
if err != nil {
return nil, err
}
displayLevel, err := a.LogFactory.GetDisplayLevel(name)
if err != nil {
return nil, err
}
loggerLevels[name] = apiadmin.LogAndDisplayLevels{
LogLevel: logLevel.String(),
DisplayLevel: displayLevel.String(),
}
}
return loggerLevels, nil
}
+3
View File
@@ -12,6 +12,7 @@ github.com/Ladicle/tabwriter v1.0.0 h1:DZQqPvMumBDwVNElso13afjYLNp0Z7pHqHnu0r4t9
github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
github.com/OpenPeeDeeP/depguard/v2 v2.2.1 h1:vckeWVESWp6Qog7UZSARNqfu/cZqvki8zsuj3piCMx4=
github.com/StephenButtolph/canoto v0.17.3 h1:lvsnYD4b96vD1knnmp1xCmZqfYpY/jSeRozGdOfdvGI=
github.com/StephenButtolph/canoto v0.17.3/go.mod h1:IcnAHC6nJUfQFVR9y60ko2ecUqqHHSB6UwI9NnBFZnE=
github.com/alecthomas/assert/v2 v2.11.0 h1:2Q9r3ki8+JYXvGsDyBXwH3LcJ+WK5D0gc5E8vS6K3D0=
github.com/alecthomas/chroma/v2 v2.17.2 h1:Rm81SCZ2mPoH+Q8ZCc/9YvzPUN/E7HgPiPJD8SLV6GI=
github.com/alecthomas/go-check-sumtype v0.3.1 h1:u9aUvbGINJxLVXiFvHUlPEaD7VDULsrxJb4Aq31NLkU=
@@ -56,6 +57,7 @@ github.com/dominikbraun/graph v0.23.0 h1:TdZB4pPqCLFxYhdyMFb1TBdFxp8XLcJfTTBQucV
github.com/ettle/strcase v0.2.0 h1:fGNiVF21fHXpX1niBgk0aROov1LagYsOwV/xqKDKR/Q=
github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
github.com/fatih/structtag v1.2.0 h1:/OdNE99OxoI/PqaW/SuSK9uxxT3f/tcSZgon/ssNSx4=
github.com/fatih/structtag v1.2.0/go.mod h1:mBJUNpUnHmRKrKlQQlmCrh5PuhftFbNv8Ys4/aAZl94=
github.com/firefart/nonamedreturns v1.0.6 h1:vmiBcKV/3EqKY3ZiPxCINmpS431OcE1S47AQUwhrg8E=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
@@ -247,6 +249,7 @@ go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93 h1:fQsdNF2N+/YewlRZiricy4P1iimyPKZ/xwniHj8Q2a0=
golang.org/x/exp/typeparams v0.0.0-20250210185358-939b2ce775ac h1:TSSpLIG4v+p0rPv1pNOQtl1I8knsO4S9trOxNMOLVP4=
golang.org/x/mod v0.31.0 h1:HaW9xtz0+kOcWKwli0ZXy79Ix+UW/vOfmWI5QVd2tgI=
golang.org/x/mod v0.31.0/go.mod h1:43JraMp9cGx1Rx3AqioxrbrhNsLl2l/iNAvuBkrezpg=
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
+1 -1
View File
@@ -1 +1 @@
1.32.11
1.36.36
+3 -1
View File
@@ -92,7 +92,9 @@
"v1.36.27",
"v1.36.28",
"v1.36.30",
"v1.36.31"
"v1.36.31",
"v1.36.32",
"v1.36.33"
],
"41": [
"v1.13.2"
+1 -1
View File
@@ -77,7 +77,7 @@ var (
const (
defaultMajor = 1
defaultMinor = 36
defaultPatch = 31
defaultPatch = 35
)
func init() {
+6
View File
@@ -301,6 +301,12 @@ func (p *postForkCommonComponents) buildChild(
_ = pChainHeight
contextPChainHeight := epoch.PChainHeight
// The inner VM builds on ITS OWN head, and Verify above requires the child's inner
// parent to be exactly p.innerBlk. Anchor the two together before delegating.
if err := p.vm.anchorInnerBuildParent(ctx, parentID, p.innerBlk.ID()); err != nil {
return nil, err
}
var innerBlock chain.Block
if p.vm.blockBuilderVM != nil {
builtBlock, err := p.vm.blockBuilderVM.BuildBlockWithRuntime(ctx, &runtime.Runtime{
+321
View File
@@ -0,0 +1,321 @@
// Copyright (C) 2019-2026, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// build_inner_parent_test.go — the deterministic repro of the SELF-REJECTION LOOP
// measured on devnet 96367 and testnet 96368 on 2026-07-28:
//
// info built block blkID=2srokbs… height=1047 innerBlkID=1DMGHAoo… pChainHeight=0
// error built block failed verification — dropping
// error="inner parentID didn't match expected parent" height=1047 parentID=2V6mi4tX…
//
// 83456 drops/min per node, forever, with the externally visible tip frozen two heights
// BELOW what the builder kept proposing. Every node rejected the block it had just built.
//
// The invariant being violated is postForkCommonComponents.Verify's third check
// (block.go, errInnerParentMismatch): a child's inner parent MUST be the inner block
// wrapped by the outer parent it extends. buildChild did not establish it — it asked the
// inner VM to build, and the inner VM builds on ITS OWN head. These tests model the inner
// VM with the luxfi/evm head semantics that make the two pointers diverge, and assert the
// invariant on the block buildChild actually returns.
package proposervm
import (
"context"
"fmt"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/luxfi/database/memdb"
"github.com/luxfi/database/versiondb"
"github.com/luxfi/ids"
"github.com/luxfi/log"
validatorstest "github.com/luxfi/validators/validatorstest"
vmchain "github.com/luxfi/vm/chain"
"github.com/luxfi/vm/chain/blocktest"
"github.com/luxfi/node/vms/proposervm/block"
"github.com/luxfi/node/vms/proposervm/proposer"
"github.com/luxfi/node/vms/proposervm/state"
)
// evmLikeInner models the inner execution VM's HEAD, which is the only part of the inner
// VM this defect involves. Three behaviours are copied from luxfi/evm:
//
// - BuildBlock builds on the head (miner/worker.go: parent = chain.CurrentBlock())
// - verifyGossiped advances the head (core/blockchain.go writeBlockAndSetHead →
// newTip(block) → writeCanonicalBlockWithLogs → writeHeadBlock)
// - SetPreference reorgs the head, and is a no-op when it already matches
// (core/blockchain.go setPreference: early return on current.Hash() == block.Hash(),
// otherwise writeKnownBlock)
//
// Nothing else is stubbed: the test drives the REAL postForkBlock.buildChild.
type evmLikeInner struct {
vmchain.ChainVM
blocks map[ids.ID]*blocktest.Block // every block inserted into the inner chain
head ids.ID
setPrefCalls int
reorgs int
}
func newEVMLikeInner(genesis *blocktest.Block) *evmLikeInner {
return &evmLikeInner{
blocks: map[ids.ID]*blocktest.Block{genesis.IDV: genesis},
head: genesis.IDV,
}
}
func (c *evmLikeInner) insert(b *blocktest.Block) { c.blocks[b.IDV] = b }
// child mints a block extending [parent] without touching the head.
func (c *evmLikeInner) child(parent *blocktest.Block) *blocktest.Block {
id := ids.GenerateTestID()
b := &blocktest.Block{
IDV: id,
ParentV: parent.IDV,
HeightV: parent.HeightV + 1,
BytesV: id[:],
TimestampV: parent.TimestampV.Add(time.Second),
}
c.insert(b)
return b
}
// verifyGossiped is the drift mechanism: verifying a peer's block whose parent is the
// current head OPTIMISTICALLY makes it the head — no accept, no proposervm involvement.
func (c *evmLikeInner) verifyGossiped(b *blocktest.Block) {
c.insert(b)
if b.ParentV == c.head { // core/blockchain.go newTip
c.head = b.IDV
}
}
func (c *evmLikeInner) BuildBlock(context.Context) (vmchain.Block, error) {
return c.child(c.blocks[c.head]), nil
}
func (c *evmLikeInner) SetPreference(_ context.Context, id ids.ID) error {
c.setPrefCalls++
blk, ok := c.blocks[id]
if !ok {
return fmt.Errorf("evm-like inner VM: block %s not in chain", id)
}
if c.head == blk.IDV { // setPreference early return — already the head
return nil
}
c.head = blk.IDV
c.reorgs++
return nil
}
func (c *evmLikeInner) GetBlock(_ context.Context, id ids.ID) (vmchain.Block, error) {
blk, ok := c.blocks[id]
if !ok {
return nil, fmt.Errorf("evm-like inner VM: block %s not found", id)
}
return blk, nil
}
// anyoneCanProposeWindower makes every slot open, so buildChild takes the UNSIGNED path
// and needs no staking key. The proposer election is orthogonal to the parent invariant.
type anyoneCanProposeWindower struct{}
func (anyoneCanProposeWindower) Proposers(context.Context, uint64, uint64, int) ([]ids.NodeID, error) {
return nil, proposer.ErrAnyoneCanPropose
}
func (anyoneCanProposeWindower) Delay(context.Context, uint64, uint64, ids.NodeID, int) (time.Duration, error) {
return 0, proposer.ErrAnyoneCanPropose
}
func (anyoneCanProposeWindower) ExpectedProposer(context.Context, uint64, uint64, uint64) (ids.NodeID, error) {
return ids.EmptyNodeID, proposer.ErrAnyoneCanPropose
}
func (anyoneCanProposeWindower) MinDelayForProposer(context.Context, uint64, uint64, ids.NodeID, uint64) (time.Duration, error) {
return 0, proposer.ErrAnyoneCanPropose
}
// newBuildParentTestVM returns a proposervm whose outer PREFERRED envelope wraps
// innerParent, plus the inner chain, at the moment before a build attempt.
func newBuildParentTestVM(t *testing.T, innerParent *blocktest.Block, inner *evmLikeInner) (*VM, *postForkBlock) {
t.Helper()
vm := &VM{
verifiedBlocks: map[ids.ID]PostForkBlock{},
logger: log.Noop(),
}
vm.State = state.New(versiondb.New(memdb.New()))
vm.ChainVM = inner
vm.Windower = anyoneCanProposeWindower{}
vm.validatorState = &validatorstest.State{
GetCurrentHeightF: func(context.Context) (uint64, error) { return 0, nil },
}
// Deterministic clock, one second after the parent envelope: slot 0, so the child
// timestamp is not window-snapped and the parent check is the only thing under test.
parentTimestamp := innerParent.TimestampV
vm.Clock.Set(parentTimestamp.Add(time.Second))
statelessParent, err := block.BuildUnsigned(
ids.GenerateTestID(), // outer grandparent
parentTimestamp,
0, // pChainHeight — 0 fleet-wide, mainnet included; orthogonal to this defect
block.Epoch{},
innerParent.BytesV,
)
require.NoError(t, err)
return vm, &postForkBlock{
SignedBlock: statelessParent,
postForkCommonComponents: postForkCommonComponents{vm: vm, innerBlk: innerParent},
}
}
// TestBuildChild_InnerHeadDriftedByGossipVerify_ChildExtendsOuterParentsInner is the
// repro. The node's outer preferred wraps inner height 1045 (the accepted tip); it then
// VERIFIES a gossiped inner 1046, which optimistically becomes the EVM head. The next
// build must still produce 1046-off-1045 — the block its own Verify will accept.
//
// BEFORE THE FIX buildChild delegated straight to the inner VM, which built off the
// drifted head: a child at height 1047 whose inner parent is 1046, while the outer parent
// wraps 1045 ⇒ errInnerParentMismatch on the node's own block, dropped, repeat forever.
// That is the live devnet signature exactly: built 1047 against an accepted tip of 1045.
func TestBuildChild_InnerHeadDriftedByGossipVerify_ChildExtendsOuterParentsInner(t *testing.T) {
require := require.New(t)
ctx := context.Background()
genesis := &blocktest.Block{
IDV: ids.GenerateTestID(),
HeightV: 1044,
BytesV: []byte("1044"),
TimestampV: time.Unix(1_785_216_000, 0),
}
inner := newEVMLikeInner(genesis)
// The accepted inner tip, wrapped by this node's outer preferred envelope.
acceptedTip := inner.child(genesis) // height 1045
inner.head = acceptedTip.IDV
_, outerParent := newBuildParentTestVM(t, acceptedTip, inner)
// DRIFT: a gossiped inner block at 1046 verifies and optimistically takes the head.
gossiped := inner.child(acceptedTip) // height 1046
inner.verifyGossiped(gossiped)
require.Equal(gossiped.IDV, inner.head, "precondition: the EVM head drifted off the accepted tip")
built, err := outerParent.buildChild(ctx)
require.NoError(err)
child, ok := built.(*postForkBlock)
require.True(ok)
// THE INVARIANT — verbatim the condition postForkCommonComponents.Verify enforces
// (expectedInnerParentID := p.innerBlk.ID(); innerParentID := child.innerBlk.Parent()).
require.Equal(acceptedTip.ID(), child.innerBlk.Parent(),
"the built child's inner parent must be the outer parent's inner block, or the node "+
"rejects its own block with errInnerParentMismatch and drops it forever")
require.Equal(acceptedTip.HeightV+1, child.Height(),
"the child must be built one above the outer parent's inner block, not above the drifted head")
// And the head was reorged back — the drift is repaired, not merely tolerated.
require.Equal(acceptedTip.IDV, inner.head)
require.Equal(1, inner.reorgs)
}
// TestBuildChild_HealthyHead_NoReorg pins the no-behaviour-change half: when the inner
// head already IS the outer parent's inner block (every healthy node, every block), the
// re-anchor is a no-op inside the inner VM — no reorg, no head movement.
func TestBuildChild_HealthyHead_NoReorg(t *testing.T) {
require := require.New(t)
ctx := context.Background()
genesis := &blocktest.Block{
IDV: ids.GenerateTestID(),
HeightV: 100,
BytesV: []byte("100"),
TimestampV: time.Unix(1_785_216_000, 0),
}
inner := newEVMLikeInner(genesis)
acceptedTip := inner.child(genesis)
inner.head = acceptedTip.IDV
_, outerParent := newBuildParentTestVM(t, acceptedTip, inner)
built, err := outerParent.buildChild(ctx)
require.NoError(err)
require.Equal(acceptedTip.ID(), built.(*postForkBlock).innerBlk.Parent())
require.Equal(acceptedTip.IDV, inner.head, "head must not move on a healthy build")
require.Zero(inner.reorgs, "no reorg on a healthy node — the inner SetPreference early-returns")
}
// TestBuildChild_PreForkTransition_ChildExtendsThisBlock covers the OTHER build
// delegation, preForkBlock.buildChild, which produces the pre-fork→post-fork transition
// block and is verified by preForkBlock.verifyPostForkChild's identical inner-parent check
// ("Make sure [b] is the parent of [child]'s inner block", pre_fork_block.go).
//
// At the fork height every validator may emit its own unsigned transition candidate, so a
// gossiped sibling verifying here is the norm, not an edge case — and it drifts the inner
// head exactly as on a live chain. Without the anchor the transition wedges at chain
// start, which is every fresh net's first two blocks.
func TestBuildChild_PreForkTransition_ChildExtendsThisBlock(t *testing.T) {
require := require.New(t)
ctx := context.Background()
genesis := &blocktest.Block{
IDV: ids.GenerateTestID(),
HeightV: 0,
BytesV: []byte("genesis"),
TimestampV: time.Unix(1_785_216_000, 0),
}
inner := newEVMLikeInner(genesis)
vm, _ := newBuildParentTestVM(t, genesis, inner)
preFork := &preForkBlock{Block: genesis, vm: vm}
// A peer's competing transition candidate verifies and takes the inner head.
sibling := inner.child(genesis)
inner.verifyGossiped(sibling)
require.Equal(sibling.IDV, inner.head)
built, err := preFork.buildChild(ctx)
require.NoError(err)
require.Equal(genesis.ID(), built.(*postForkBlock).innerBlk.Parent(),
"the transition block's inner parent must be the pre-fork block itself")
require.Equal(genesis.HeightV+1, built.Height())
require.Equal(genesis.IDV, inner.head)
}
// TestBuildChild_InnerParentNotInInnerChain_FailsInsteadOfBuildingADoomedBlock covers the
// error case: if the inner VM cannot anchor on the outer parent's inner block, then the
// head is provably NOT that block, so any child built would fail this node's own Verify.
// Returning the error is strictly better than emitting a block we will drop.
func TestBuildChild_InnerParentNotInInnerChain_FailsInsteadOfBuildingADoomedBlock(t *testing.T) {
require := require.New(t)
ctx := context.Background()
genesis := &blocktest.Block{
IDV: ids.GenerateTestID(),
HeightV: 100,
BytesV: []byte("100"),
TimestampV: time.Unix(1_785_216_000, 0),
}
inner := newEVMLikeInner(genesis)
// An inner parent the inner VM does not hold (never inserted).
orphanID := ids.GenerateTestID()
orphan := &blocktest.Block{
IDV: orphanID,
ParentV: genesis.IDV,
HeightV: 101,
BytesV: orphanID[:],
TimestampV: genesis.TimestampV.Add(time.Second),
}
_, outerParent := newBuildParentTestVM(t, orphan, inner)
_, err := outerParent.buildChild(ctx)
require.ErrorContains(err, "failed to anchor inner build parent")
}
+302
View File
@@ -0,0 +1,302 @@
// Copyright (C) 2019-2026, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// height_backfill.go — RECOVERY for a proposervm finality index that sits BELOW
// the inner VM's accepted tip.
//
// THE INVARIANT. Every post-fork accept commits the outer envelope, its height
// index entry and the last-accepted pointer in ONE versiondb batch BEFORE the
// inner block is accepted (vm.acceptPostForkBlock → postForkBlock.Accept), so the
// proposervm index can only ever be AHEAD of the inner VM, never behind.
//
// HOW IT BREAKS ANYWAY. The proposervm has one accept path that advances the inner
// VM while leaving the outer index untouched: preForkBlock.Accept, whose
// acceptOuterBlk() is a no-op by construction. Post-fork that path was reachable
// because BOTH of the proposervm's block constructors fall back to it silently —
// ParseBlock() falls back to parsePreForkBlock when the outer envelope does not
// parse, and getBlock() falls back to getPreForkBlock when the id is not an outer
// id (which is exactly what the CANONICAL id recorded by the consensus ledger is:
// postForkCommonComponents.CanonicalID() == innerBlk.ID()). Either fallback hands
// the engine a preForkBlock wrapping a post-fork inner block; accepting it moves
// the inner VM and NOT the index. Nothing complained, because nothing asserted the
// invariant at the moment it was violated — it was only ever checked at the NEXT
// boot, by which point the node could not start. That is why the lag looked like
// a mysterious "persistence" problem: the writes were never issued at all.
//
// The prevention half lives at the two fallbacks + preForkBlock.acceptOuterBlk
// (see pre_fork_block.go and vm.getPreForkBlock): post-fork, a pre-fork block is
// never constructed and never accepted.
//
// THE RECOVERY half lives here, and it must work for nodes that are ALREADY
// damaged. It is two escalating steps, both OUTER-ONLY — they never re-execute,
// re-verify or re-accept an inner block, so a node whose EVM is already at the tip
// is repaired without touching the EVM:
//
// 1. rebuildOuterIndexFromStore: re-derive the height index and last-accepted
// pointer from the outer envelopes ALREADY in this node's block store, binding
// each candidate to the inner block this node itself accepted at that height.
// Fully local, no network, no operator. This is what turns "refuses to start"
// into "starts, repairs itself, and keeps its finality pointer".
//
// 2. If step 1 cannot reach the inner tip, the node still STARTS, in an explicit
// backfill-pending state: loud, fail-safe (it will not BUILD a block while its
// finality index is incomplete) and repairable through BackfillOuterBlock,
// which accepts the missing envelopes from any source and applies the SAME
// binding checks. The alternative — the previous behaviour — was a dead
// C-Chain and a destructive resync.
//
// WHAT IS NEVER DONE, and why: the finality pointer is never dropped
// (DeleteLastAccepted). proposervm.LastAccepted would then fall back to an
// INNER-namespace id whose ParentID is contiguity-incompatible with the network's
// OUTER wrappers, permanently wedging bootstrap, catch-up and live Verify at the
// inner tip — a silent wedge strictly worse than the loud stop it would replace.
// Every path below only ever moves the pointer FORWARD, onto an envelope that was
// proven to wrap the inner block this node already accepted at that height.
package proposervm
import (
"bytes"
"context"
"crypto/sha256"
"errors"
"fmt"
"github.com/luxfi/ids"
"github.com/luxfi/log"
statelessblock "github.com/luxfi/node/vms/proposervm/block"
"github.com/luxfi/node/vms/proposervm/state"
)
var (
// ErrOuterBackfillNotPending is returned by BackfillOuterBlock when the index
// is already whole — the repair is idempotent, not an error to call twice.
ErrOuterBackfillNotPending = errors.New("proposervm: no outer-index backfill is pending")
// ErrOuterBackfillRejected is returned when a candidate envelope does not bind
// to this node's own accepted chain. Fail-closed: an envelope is only indexed
// when it provably wraps the inner block WE accepted at that height and chains
// off the envelope WE already hold at height-1, so a peer (or a stale file)
// cannot steer the finality pointer.
ErrOuterBackfillRejected = errors.New("proposervm: outer backfill candidate rejected")
// errPreForkAfterFork is the prevention guard: post-fork, a block at or above
// the recorded fork height is NEVER a pre-fork block. Constructing or accepting
// one is what silently strands the finality index below the inner tip.
errPreForkAfterFork = errors.New("proposervm: refusing a pre-fork block at or above the fork height")
)
// outerBackfill is the state of an incomplete outer index. Zero value = whole.
//
// It is guarded by its own leaf mutex (vm.backfillMu) rather than vm.lock: the
// accept path, the build path and the operator repair path all read it, and it is
// never held across a callout, so it cannot participate in a deadlock.
type outerBackfill struct {
// next is the FIRST height whose outer envelope is missing. It advances as
// envelopes land, so it doubles as the resume point across restarts.
next uint64
// tip is the inner VM's accepted height — the height the index must reach.
tip uint64
// innerTipID is the inner block at [tip], recorded for diagnostics.
innerTipID ids.ID
}
// NeedsOuterBackfill reports the still-missing outer height range [from, to] and
// whether a backfill is pending. It is the seam an operator tool or a node-layer
// fetch driver reads; when pending is false the index is whole.
func (vm *VM) NeedsOuterBackfill() (from uint64, to uint64, pending bool) {
vm.backfillMu.RLock()
defer vm.backfillMu.RUnlock()
if vm.backfill == nil {
return 0, 0, false
}
return vm.backfill.next, vm.backfill.tip, true
}
// outerBackfillPending is the internal fast check used by the build gate.
func (vm *VM) outerBackfillPending() bool {
vm.backfillMu.RLock()
defer vm.backfillMu.RUnlock()
return vm.backfill != nil
}
// innerBlockAtHeight returns the inner VM's ACCEPTED block at [height] together
// with its bytes — the binding target every repair candidate must match. It reads
// the inner VM's own height index, so it is the inner VM's truth, not ours.
func (vm *VM) innerBlockAtHeight(ctx context.Context, height uint64) (ids.ID, []byte, error) {
innerID, err := vm.ChainVM.GetBlockIDAtHeight(ctx, height)
if err != nil {
return ids.Empty, nil, err
}
innerBlk, err := vm.ChainVM.GetBlock(ctx, innerID)
if err != nil {
return ids.Empty, nil, err
}
return innerID, innerBlk.Bytes(), nil
}
// indexOuterAtHeight commits ONE outer envelope into the finality index: the
// envelope itself, its height entry, and the last-accepted pointer, in a single
// versiondb batch — byte-for-byte what acceptPostForkBlock writes on the live
// path. It deliberately does NOT touch the inner VM.
func (vm *VM) indexOuterAtHeight(height uint64, blk statelessblock.Block) error {
if err := vm.State.PutBlock(blk); err != nil {
return err
}
if err := vm.State.SetBlockIDAtHeight(height, blk.ID()); err != nil {
return err
}
if err := vm.State.SetLastAccepted(blk.ID()); err != nil {
return err
}
return vm.db.Commit()
}
// rebuildOuterIndexFromStore walks the gap (proHeight, innerHeight] forward,
// re-indexing each height from an outer envelope ALREADY present in this node's
// block store.
//
// Soundness. A candidate at height h is accepted ONLY if
//
// envelope.Block() == the bytes of the inner block WE accepted at h, and
// envelope.ParentID() == the envelope we just indexed at h-1
//
// so the rebuilt chain is the unique outer chain that wraps our own accepted inner
// chain. Nothing is inferred and nothing is fabricated: an unmatched height stops
// the walk immediately.
//
// Returns the height the index reached (>= proHeight). A return equal to
// innerHeight means the index is whole again.
func (vm *VM) rebuildOuterIndexFromStore(
ctx context.Context,
proHeight uint64,
proID ids.ID,
innerHeight uint64,
) (uint64, error) {
// One pass over the block store: inner-bytes digest -> outer envelope. Keyed by
// digest rather than by inner id because deriving an inner id from bytes would
// mean parsing untrusted bytes through the inner VM; the digest comparison is
// exact and side-effect free.
byInnerDigest := make(map[[sha256.Size]byte]statelessblock.Block)
if err := state.ScanBlocks(vm.db, func(blk statelessblock.Block) error {
byInnerDigest[sha256.Sum256(blk.Block())] = blk
return nil
}); err != nil {
return proHeight, fmt.Errorf("failed to scan the proposervm block store: %w", err)
}
reached := proHeight
parentID := proID
for h := proHeight + 1; h <= innerHeight; h++ {
_, innerBytes, err := vm.innerBlockAtHeight(ctx, h)
if err != nil {
vm.logger.Warn("proposervm outer-index rebuild: inner block unreadable, stopping walk",
log.Uint64("height", h), log.Err(err))
break
}
candidate, ok := byInnerDigest[sha256.Sum256(innerBytes)]
if !ok || candidate.ParentID() != parentID {
break
}
if err := vm.indexOuterAtHeight(h, candidate); err != nil {
return reached, fmt.Errorf("failed to re-index outer block at height %d: %w", h, err)
}
reached = h
parentID = candidate.ID()
}
return reached, nil
}
// enterOuterBackfill records an incomplete index and lets the VM start anyway.
// Fail-safe, not fail-open: BuildBlock is gated off while pending (a node whose
// finality index has a hole must never PROPOSE), the height index still reports
// ErrNotFound inside the gap so nothing reads a wrong id, and the last-accepted
// pointer is left exactly where the rebuild got to.
func (vm *VM) enterOuterBackfill(from, tip uint64, innerTipID ids.ID) {
vm.backfillMu.Lock()
vm.backfill = &outerBackfill{next: from, tip: tip, innerTipID: innerTipID}
vm.backfillMu.Unlock()
vm.logger.Warn("proposervm STARTING WITH AN INCOMPLETE FINALITY INDEX — outer backfill pending",
log.Uint64("firstMissingHeight", from),
log.Uint64("innerTipHeight", tip),
log.Stringer("innerTipID", innerTipID),
log.String("effect", "this chain will NOT build blocks until the missing outer envelopes are supplied"),
log.String("recovery", "feed the outer proposervm envelopes for the missing heights via "+
"BackfillOuterBlock (cmd/repair-proposervm backfill) — each is bound to the inner block "+
"this node already accepted, so no EVM re-execution and no resync are required"),
)
}
// BackfillOuterBlock supplies ONE missing outer envelope, oldest-first, and is the
// operator/driver half of the recovery. It is fail-closed and idempotent:
//
// - not pending -> ErrOuterBackfillNotPending
// - envelope below the next height -> nil (already indexed; tolerate replay)
// - signature/format invalid -> error (parsed WITH verification)
// - not the next height, wrong parent,
// or wrapping a different inner block-> ErrOuterBackfillRejected
//
// On the final height it clears the pending state and refreshes the in-memory
// last-accepted metadata, after which the chain behaves exactly like one that
// booted whole.
func (vm *VM) BackfillOuterBlock(ctx context.Context, outerBytes []byte) error {
vm.backfillMu.Lock()
defer vm.backfillMu.Unlock()
if vm.backfill == nil {
return ErrOuterBackfillNotPending
}
height := vm.backfill.next
// Parse WITH proposer-signature verification: the envelope must be a real
// block of THIS chain, not merely well-formed bytes.
blk, err := statelessblock.Parse(outerBytes, vm.rt.ChainID)
if err != nil {
return fmt.Errorf("%w: envelope did not parse/verify: %w", ErrOuterBackfillRejected, err)
}
// BIND 1 — the envelope must chain off the one we already hold.
parentID, err := vm.State.GetLastAccepted()
if err != nil {
return fmt.Errorf("%w: cannot read the current finality pointer: %w", ErrOuterBackfillRejected, err)
}
if blk.ParentID() != parentID {
return fmt.Errorf("%w: envelope parent %s != current finality pointer %s (feed oldest-first)",
ErrOuterBackfillRejected, blk.ParentID(), parentID)
}
// BIND 2 — the envelope must wrap the inner block WE accepted at this height.
// This is what makes accepting bytes from an untrusted source safe.
innerID, innerBytes, err := vm.innerBlockAtHeight(ctx, height)
if err != nil {
return fmt.Errorf("%w: inner block at height %d unreadable: %w", ErrOuterBackfillRejected, height, err)
}
if !bytes.Equal(blk.Block(), innerBytes) {
return fmt.Errorf("%w: envelope at height %d does not wrap our accepted inner block %s",
ErrOuterBackfillRejected, height, innerID)
}
if err := vm.indexOuterAtHeight(height, blk); err != nil {
return err
}
vm.logger.Info("proposervm outer backfill: height repaired",
log.Uint64("height", height),
log.Stringer("outerID", blk.ID()),
log.Uint64("remaining", vm.backfill.tip-height),
)
if height >= vm.backfill.tip {
tip := vm.backfill.tip
vm.backfill = nil
if err := vm.setLastAcceptedMetadata(ctx); err != nil {
return fmt.Errorf("outer backfill completed at height %d but last-accepted metadata failed: %w", tip, err)
}
vm.logger.Info("proposervm FINALITY INDEX FULLY REPAIRED — backfill complete",
log.Uint64("height", tip))
return nil
}
vm.backfill.next = height + 1
return nil
}
+253
View File
@@ -0,0 +1,253 @@
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// height_backfill_test.go — RECOVERY: a node whose proposervm finality index is
// ALREADY behind the inner VM tip must BOOT and REPAIR itself instead of killing
// the chain ("non-critical chain failed to initialize chainAlias=C" — a dead
// C-Chain on a pod that still reports 1/1 Ready).
//
// Two paths, both OUTER-ONLY (no inner re-execution, no EVM rollback, no resync):
// - the envelopes for the gap are still in the local block store -> rebuilt at
// boot with no network and no operator;
// - they are not -> the chain STARTS anyway, loud, build-gated, and heals from
// supplied envelopes through BackfillOuterBlock.
//
// The shared harness lives in height_lag_repro_test.go.
package proposervm
import (
"context"
"errors"
"testing"
"time"
"github.com/luxfi/ids"
statelessblock "github.com/luxfi/node/vms/proposervm/block"
)
// ---------------------------------------------------------------------------
// 2. RECOVERY — a node whose index is ALREADY behind boots and heals.
// ---------------------------------------------------------------------------
// TestOuterIndexRebuild_FromLocalStore_BootsAndHeals: the damaged node still holds
// the outer envelopes for the gap in its own block store (the truncated-index /
// inconsistently-restored-snapshot case). Boot must rebuild the index from them —
// no network, no operator, no resync — and must NOT return an error.
func TestOuterIndexRebuild_FromLocalStore_BootsAndHeals(t *testing.T) {
ctx := context.Background()
ic := newInnerChain(t, 8)
vm := testVM(t, ic)
outers := acceptThroughProposervm(t, vm, ic, 8)
// DAMAGE: roll the finality pointer + height index back to 5 while the inner VM
// stays at 8 — byte-for-byte the on-disk state luxd-3 booted with (index 6,
// inner 98), except the envelopes survive in the block store.
if err := vm.State.SetLastAccepted(outers[5].ID()); err != nil {
t.Fatalf("SetLastAccepted: %v", err)
}
for h := uint64(6); h <= 8; h++ {
if err := vm.State.DeleteBlockIDAtHeight(h); err != nil {
t.Fatalf("DeleteBlockIDAtHeight(%d): %v", h, err)
}
}
if err := vm.db.Commit(); err != nil {
t.Fatalf("commit damage: %v", err)
}
if got := indexHeight(t, vm); got != 5 {
t.Fatalf("precondition: damaged index must read 5, got %d", got)
}
// BOOT.
if err := vm.repairAcceptedChainByHeight(ctx); err != nil {
t.Fatalf("boot with a behind index must REPAIR, not fail: %v", err)
}
if _, _, pending := vm.NeedsOuterBackfill(); pending {
t.Fatal("the local store held every missing envelope — no backfill should be pending")
}
if got := indexHeight(t, vm); got != 8 {
t.Fatalf("index must be rebuilt to the inner tip 8, got %d", got)
}
lastAccepted, err := vm.State.GetLastAccepted()
if err != nil || lastAccepted != outers[8].ID() {
t.Fatalf("finality pointer must be the canonical envelope at 8 (%s), got %s (err %v)",
outers[8].ID(), lastAccepted, err)
}
for h := uint64(6); h <= 8; h++ {
got, err := vm.State.GetBlockIDAtHeight(h)
if err != nil || got != outers[h].ID() {
t.Fatalf("height index at %d must be %s, got %s (err %v)", h, outers[h].ID(), got, err)
}
}
}
// TestOuterBackfill_HealsFromSuppliedEnvelopes: the envelopes for the gap are NOT
// in the local store (the pre-fork-fallback case — they were never persisted).
// The node must still START — the old code killed the C-Chain here — come up
// explicitly backfill-pending, refuse to build, and then heal from supplied
// envelopes with no EVM re-execution.
func TestOuterBackfill_HealsFromSuppliedEnvelopes(t *testing.T) {
ctx := context.Background()
ic := newInnerChain(t, 8)
vm := testVM(t, ic)
acceptThroughProposervm(t, vm, ic, 5)
lastGood, err := vm.State.GetLastAccepted()
if err != nil {
t.Fatalf("GetLastAccepted: %v", err)
}
// DAMAGE: the inner VM ran on to 8 without the index (exactly what the pre-fork
// fallback did in production). No envelopes for 6..8 exist locally.
ic.accept(8)
// BOOT — must not error.
if err := vm.repairAcceptedChainByHeight(ctx); err != nil {
t.Fatalf("boot with a behind index must START the chain, got: %v", err)
}
from, to, pending := vm.NeedsOuterBackfill()
if !pending || from != 6 || to != 8 {
t.Fatalf("want backfill pending 6..8, got from=%d to=%d pending=%v", from, to, pending)
}
// FAIL-SAFE: a node with a hole in its finality index must not propose.
if _, err := vm.BuildBlock(ctx); err == nil {
t.Fatal("BuildBlock must refuse while the finality index is incomplete")
}
// NEGATIVE 1 — an envelope wrapping a DIFFERENT inner block than the one we
// accepted at this height is rejected (a peer cannot steer the pointer).
bogus, err := statelessblock.BuildUnsigned(
lastGood, time.Unix(6, 0), 0, statelessblock.Epoch{}, []byte("not-our-inner-block"))
if err != nil {
t.Fatalf("BuildUnsigned: %v", err)
}
if err := vm.BackfillOuterBlock(ctx, bogus.Bytes()); !errors.Is(err, ErrOuterBackfillRejected) {
t.Fatalf("an envelope that does not wrap our accepted inner block must be rejected, got %v", err)
}
// NEGATIVE 2 — out of order (height 7 before 6) is rejected: it does not chain
// off the current finality pointer.
outOfOrder := outerFor(t, ic, 7, lastGood)
if err := vm.BackfillOuterBlock(ctx, outOfOrder.Bytes()); !errors.Is(err, ErrOuterBackfillRejected) {
t.Fatalf("an out-of-order envelope must be rejected, got %v", err)
}
// HEAL — feed 6, 7, 8 oldest-first, as a healthy peer or the repair tool would.
parent := lastGood
want := map[uint64]ids.ID{}
for h := uint64(6); h <= 8; h++ {
sb := outerFor(t, ic, h, parent)
if err := vm.BackfillOuterBlock(ctx, sb.Bytes()); err != nil {
t.Fatalf("BackfillOuterBlock(h=%d): %v", h, err)
}
want[h] = sb.ID()
parent = sb.ID()
}
if _, _, pending := vm.NeedsOuterBackfill(); pending {
t.Fatal("backfill must be complete after the last height")
}
if got := indexHeight(t, vm); got != 8 {
t.Fatalf("index must reach the inner tip 8, got %d", got)
}
for h := uint64(6); h <= 8; h++ {
got, err := vm.State.GetBlockIDAtHeight(h)
if err != nil || got != want[h] {
t.Fatalf("height index at %d must be %s, got %s (err %v)", h, want[h], got, err)
}
}
// Idempotent: nothing pending, so a replayed call says so rather than corrupting.
if err := vm.BackfillOuterBlock(ctx, outOfOrder.Bytes()); !errors.Is(err, ErrOuterBackfillNotPending) {
t.Fatalf("want ErrOuterBackfillNotPending after completion, got %v", err)
}
// The chain is a full participant again.
if _, _, pending := vm.NeedsOuterBackfill(); pending {
t.Fatal("build gate must have lifted")
}
// And a fresh boot on the repaired state is a clean no-op.
if err := vm.repairAcceptedChainByHeight(ctx); err != nil {
t.Fatalf("boot after repair must be a no-op, got %v", err)
}
}
// TestBootWithBehindIndex_NeverKillsTheChain is the direct regression lock on the
// production symptom: repairAcceptedChainByHeight must NEVER return an error for a
// behind index, because chains/manager.go turns that error into
// "non-critical chain failed to initialize chainAlias=C" — a dead C-Chain on a pod
// that still reports 1/1 Ready.
func TestBootWithBehindIndex_NeverKillsTheChain(t *testing.T) {
ctx := context.Background()
for _, gap := range []uint64{1, 2, 6, 91} {
ic := newInnerChain(t, 100)
vm := testVM(t, ic)
acceptThroughProposervm(t, vm, ic, 100-gap)
ic.accept(100)
if err := vm.repairAcceptedChainByHeight(ctx); err != nil {
t.Fatalf("gap %d: boot must not fail, got %v", gap, err)
}
from, to, pending := vm.NeedsOuterBackfill()
if !pending || from != 100-gap+1 || to != 100 {
t.Fatalf("gap %d: want pending %d..100, got %d..%d pending=%v", gap, 100-gap+1, from, to, pending)
}
}
}
func mustForkHeight(t *testing.T, vm *VM) uint64 {
t.Helper()
h, err := vm.State.GetForkHeight()
if err != nil {
t.Fatalf("GetForkHeight: %v", err)
}
return h
}
// TestOuterBackfill_SelfHealsFromOrdinaryParseTraffic proves the loop closes with
// no new transport and no operator: every envelope a peer serves passes through
// VM.ParseBlock, and a backfill-pending node absorbs exactly the ones it needs.
// This is what lets an ALREADY-DAMAGED node come back by itself.
func TestOuterBackfill_SelfHealsFromOrdinaryParseTraffic(t *testing.T) {
ctx := context.Background()
ic := newInnerChain(t, 8)
vm := testVM(t, ic)
acceptThroughProposervm(t, vm, ic, 5)
lastGood, err := vm.State.GetLastAccepted()
if err != nil {
t.Fatalf("GetLastAccepted: %v", err)
}
ic.accept(8) // the inner VM ran on without the index
if err := vm.repairAcceptedChainByHeight(ctx); err != nil {
t.Fatalf("boot must start the chain, got %v", err)
}
if _, _, pending := vm.NeedsOuterBackfill(); !pending {
t.Fatal("precondition: backfill must be pending")
}
// Peer traffic, oldest-first, delivered through the ORDINARY parse path.
parent := lastGood
for h := uint64(6); h <= 8; h++ {
sb := outerFor(t, ic, h, parent)
if _, err := vm.ParseBlock(ctx, sb.Bytes()); err != nil {
t.Fatalf("ParseBlock(h=%d): %v", h, err)
}
parent = sb.ID()
}
if _, _, pending := vm.NeedsOuterBackfill(); pending {
t.Fatal("the node must have healed itself from ordinary parse traffic")
}
if got := indexHeight(t, vm); got != 8 {
t.Fatalf("index must reach the inner tip 8, got %d", got)
}
// The build gate lifted with the heal (the gate itself is asserted in
// TestOuterBackfill_HealsFromSuppliedEnvelopes; driving the real builder here
// would need a validator set this harness deliberately does not stand up).
if _, _, pending := vm.NeedsOuterBackfill(); pending {
t.Fatal("build gate must have lifted")
}
}
+280
View File
@@ -0,0 +1,280 @@
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// height_lag_repro_test.go — the DETERMINISTIC REPRODUCTION of the proposervm
// finality-index lag, written against the PRE-EXISTING proposervm API only (no
// symbol introduced by the fix), so it fails on the unfixed code and passes on the
// fixed code without being tautological.
//
// The production failure it reproduces (lux-devnet luxd-3, C-Chain, verbatim):
//
// VM initialization failed error="failed to repair accepted chain by height:
// proposervm finality index (height 6, id ns5qGN4i…) is BEHIND the inner VM tip
// (height 98, id TUTR74eA…)" chainID=21HieZng…
// non-critical chain failed to initialize … chainAlias=C
//
// THE MECHANISM. Every post-fork accept commits the envelope, its height index
// entry and the last-accepted pointer in ONE versiondb batch BEFORE the inner
// block is accepted, so the index can only run AHEAD. But the proposervm has one
// accept path that moves the inner VM and writes NOTHING: preForkBlock.Accept,
// whose acceptOuterBlk() is a no-op. Post-fork it was reachable because getBlock()
// silently falls back to constructing a preForkBlock whenever the id is not an
// OUTER envelope id — and the id this fork's consensus ledger records as canonical
// IS the inner block's id (postForkCommonComponents.CanonicalID). One such accept
// and the index is stranded: invisible while running, fatal at the next boot.
//
// This file also holds the shared harness the recovery tests use.
package proposervm
import (
"context"
"errors"
"testing"
"time"
"github.com/luxfi/database"
"github.com/luxfi/database/memdb"
"github.com/luxfi/database/versiondb"
"github.com/luxfi/ids"
"github.com/luxfi/log"
"github.com/luxfi/metric"
"github.com/luxfi/runtime"
vmchain "github.com/luxfi/vm/chain"
"github.com/luxfi/vm/chain/blocktest"
"github.com/luxfi/node/cache/lru"
statelessblock "github.com/luxfi/node/vms/proposervm/block"
"github.com/luxfi/node/vms/proposervm/state"
"github.com/luxfi/node/vms/proposervm/tree"
)
// innerChain is a stand-in for the inner execution VM (the C-Chain EVM): a
// contiguous accepted chain 1..tip answering LastAccepted / GetBlock /
// GetBlockIDAtHeight / ParseBlock exactly as the real one does.
type innerChain struct {
byHeight map[uint64]*blocktest.Block
byID map[ids.ID]*blocktest.Block
byBytes map[string]*blocktest.Block
tip uint64
}
func newInnerChain(t *testing.T, tip uint64) *innerChain {
t.Helper()
ic := &innerChain{
byHeight: map[uint64]*blocktest.Block{},
byID: map[ids.ID]*blocktest.Block{},
byBytes: map[string]*blocktest.Block{},
}
parent := ids.Empty
for h := uint64(1); h <= tip; h++ {
id := ids.GenerateTestID()
blk := &blocktest.Block{
IDV: id,
ParentV: parent,
HeightV: h,
BytesV: append([]byte("inner"), id[:]...),
TimestampV: time.Unix(int64(h), 0),
}
ic.byHeight[h] = blk
ic.byID[id] = blk
ic.byBytes[string(blk.BytesV)] = blk
parent = id
}
ic.tip = tip
return ic
}
// accept models the inner VM accepting a block: its head advances. This is the
// ONLY way the inner tip moves, exactly as in the real EVM.
func (ic *innerChain) accept(h uint64) { ic.tip = h }
func (ic *innerChain) vm() *blocktest.VM {
return &blocktest.VM{
LastAcceptedF: func(context.Context) (ids.ID, error) {
return ic.byHeight[ic.tip].IDV, nil
},
GetBlockF: func(_ context.Context, id ids.ID) (vmchain.Block, error) {
blk, ok := ic.byID[id]
if !ok {
return nil, errors.New("inner: block not found")
}
return blk, nil
},
GetBlockIDAtHeightF: func(_ context.Context, h uint64) (ids.ID, error) {
blk, ok := ic.byHeight[h]
if !ok {
return ids.Empty, errors.New("inner: height not indexed")
}
return blk.IDV, nil
},
ParseBlockF: func(_ context.Context, b []byte) (vmchain.Block, error) {
blk, ok := ic.byBytes[string(b)]
if !ok {
return nil, errors.New("inner: unparseable bytes")
}
return blk, nil
},
}
}
// testVM builds a proposervm over memdb wrapping [ic] — the real VM struct with
// the real State, so every write below goes through the production code paths.
func testVM(t *testing.T, ic *innerChain) *VM {
t.Helper()
return testVMOnBase(t, ic, memdb.New())
}
// testVMOnBase is testVM over a caller-supplied base database — the seam the
// crash-boot tests (vm_crashboot_test.go) use to boot a SECOND, cold VM over
// bytes copied out of a running one.
func testVMOnBase(t *testing.T, ic *innerChain, base database.Database) *VM {
t.Helper()
db := versiondb.New(base)
metrics := metric.New("")
vm := &VM{
ChainVM: ic.vm(),
db: db,
State: state.New(db),
Tree: tree.New(),
verifiedBlocks: map[ids.ID]PostForkBlock{},
innerBlkCache: lru.NewSizedCache(innerBlkCacheSize, cachedBlockSize),
logger: log.Noop(),
rt: &runtime.Runtime{ChainID: ids.GenerateTestID()},
}
vm.lastAcceptedTimestampGaugeVec = metrics.NewGaugeVec(
"last_accepted_timestamp", "timestamp of the last block accepted", []string{"block_type"})
return vm
}
// outerFor wraps the inner block at [h] in a real proposervm envelope chained off
// [parentOuter] — the same shape a proposer builds.
func outerFor(t *testing.T, ic *innerChain, h uint64, parentOuter ids.ID) statelessblock.SignedBlock {
t.Helper()
sb, err := statelessblock.BuildUnsigned(
parentOuter, time.Unix(int64(h), 0), 0, statelessblock.Epoch{}, ic.byHeight[h].BytesV)
if err != nil {
t.Fatalf("BuildUnsigned(h=%d): %v", h, err)
}
return sb
}
// acceptThroughProposervm drives heights 1..through through the REAL post-fork
// accept path, so index and inner head advance in lock-step. This is a healthy
// node, and it is what records the fork height.
func acceptThroughProposervm(t *testing.T, vm *VM, ic *innerChain, through uint64) map[uint64]statelessblock.SignedBlock {
t.Helper()
return acceptRangeThroughProposervm(t, vm, ic, 1, through, ids.Empty)
}
// acceptRangeThroughProposervm is acceptThroughProposervm for heights
// from..through, chaining the first envelope off parentOuter — so a test can
// keep a source node accepting AFTER a crash copy was taken.
func acceptRangeThroughProposervm(t *testing.T, vm *VM, ic *innerChain, from, through uint64, parentOuter ids.ID) map[uint64]statelessblock.SignedBlock {
t.Helper()
outers := map[uint64]statelessblock.SignedBlock{}
for h := from; h <= through; h++ {
sb := outerFor(t, ic, h, parentOuter)
blk := &postForkBlock{
SignedBlock: sb,
postForkCommonComponents: postForkCommonComponents{vm: vm, innerBlk: ic.byHeight[h]},
}
vm.Tree.Add(ic.byHeight[h])
if err := blk.Accept(context.Background()); err != nil {
t.Fatalf("post-fork Accept(h=%d): %v", h, err)
}
ic.accept(h)
outers[h] = sb
parentOuter = sb.ID()
}
return outers
}
// indexHeight reads the height the PERSISTED finality index sits at — the exact
// quantity repairAcceptedChainByHeight compares against the inner tip at boot.
func indexHeight(t *testing.T, vm *VM) uint64 {
t.Helper()
id, err := vm.State.GetLastAccepted()
if err != nil {
t.Fatalf("GetLastAccepted: %v", err)
}
blk, err := vm.getPostForkBlock(context.Background(), id)
if err != nil {
t.Fatalf("getPostForkBlock(%s): %v", id, err)
}
return blk.Height()
}
// TestFinalityIndexLag_IndexMustNeverFallBehindInnerTip is THE reproduction.
//
// It asks the proposervm for the block at the CANONICAL id of height 6 — the id
// the consensus ledger records for a proposervm-wrapped block — accepts whatever
// comes back, and then asserts the two things a correct proposervm guarantees:
//
// 1. the persisted finality index is never below the inner VM tip, and
// 2. a boot-time reconciliation of that state never kills the chain.
//
// UNFIXED, getBlock hands back a preForkBlock, its Accept is a silent no-op on the
// outer index, the inner VM advances to 6, and this test fails on (1) — printing
// the same divergence luxd-3 died of. FIXED, the pre-fork construction/accept is
// refused, nothing diverges, and both assertions hold.
//
// It uses NO symbol introduced by the fix, so it is a genuine before/after probe.
func TestFinalityIndexLag_IndexMustNeverFallBehindInnerTip(t *testing.T) {
ctx := context.Background()
ic := newInnerChain(t, 8)
vm := testVM(t, ic)
acceptThroughProposervm(t, vm, ic, 5)
if got, tip := indexHeight(t, vm), ic.tip; got != 5 || tip != 5 {
t.Fatalf("precondition: want index=5 inner=5, got index=%d inner=%d", got, tip)
}
forkHeight, err := vm.State.GetForkHeight()
if err != nil {
t.Fatalf("GetForkHeight: %v", err)
}
// The engine asks for height 6 by its CANONICAL (inner) id.
if blk, err := vm.getBlock(ctx, ic.byHeight[6].IDV); err == nil {
if _, isPreFork := blk.(*preForkBlock); isPreFork {
t.Logf("getBlock(canonical id @6) returned a preForkBlock — fork height is %d", forkHeight)
}
if err := blk.Accept(ctx); err == nil {
ic.accept(6) // the inner VM applied it
}
}
// ASSERTION 1 — the invariant.
if got := indexHeight(t, vm); got < ic.tip {
// Errorf, not Fatalf: assertion 2 below must also run, so one failing run
// shows the WHOLE production symptom chain (lag now -> dead chain at boot).
t.Errorf("FINALITY INDEX LAG REPRODUCED: index is at height %d but the inner VM tip is %d "+
"(fork height %d). An accept advanced the inner VM without writing the proposervm index; "+
"this node is now unbootable", got, ic.tip, forkHeight)
}
// ASSERTION 2 — boot must not kill the chain.
if err := vm.repairAcceptedChainByHeight(ctx); err != nil {
t.Fatalf("BOOT KILLED THE CHAIN: %v", err)
}
}
// TestPreForkAcceptStillWorksBeforeFork is the positive control for the guard: on
// a chain that has NOT forked there is no fork height, every block is legitimately
// pre-fork, and both construction and Accept must still work. A guard that broke
// this would brick every chain before its transition block.
func TestPreForkAcceptStillWorksBeforeFork(t *testing.T) {
ctx := context.Background()
ic := newInnerChain(t, 3)
vm := testVM(t, ic)
if _, err := vm.State.GetForkHeight(); err == nil {
t.Fatal("precondition: a fresh chain must have no fork height")
}
pre := &preForkBlock{Block: ic.byHeight[1], vm: vm}
if err := pre.Accept(ctx); err != nil {
t.Fatalf("pre-fork accept before the fork must succeed, got %v", err)
}
if _, err := vm.getPreForkBlock(ctx, ic.byHeight[1].IDV); err != nil {
t.Fatalf("pre-fork getBlock before the fork must succeed, got %v", err)
}
}
+24 -2
View File
@@ -65,8 +65,20 @@ func (b *preForkBlock) Accept(ctx context.Context) error {
return b.acceptInnerBlk(ctx)
}
func (*preForkBlock) acceptOuterBlk() error {
return nil
// acceptOuterBlk is a NO-OP for a genuine pre-fork block — a pre-fork block has no
// outer envelope, so there is nothing to index — and a hard REFUSAL once the chain
// has forked.
//
// The refusal is the second half of the finality-index root-cause fix (the first is
// vm.getPreForkBlock). Accepting a pre-fork block at or above the fork height
// advances the inner VM while leaving the proposervm index exactly where it was:
// the index silently falls behind the inner tip during normal operation, and the
// NEXT boot cannot start the chain. Refusing here converts that invisible,
// deferred, fatal corruption into an immediate, attributable error at the exact
// accept that would have caused it — and the engine's accept path is fail-closed,
// so finality stops rather than diverging.
func (b *preForkBlock) acceptOuterBlk() error {
return b.vm.refusePreForkAfterFork(b.Height())
}
func (b *preForkBlock) acceptInnerBlk(ctx context.Context) error {
@@ -275,6 +287,16 @@ func (b *preForkBlock) buildChild(ctx context.Context) (Block, error) {
}
// else: we ARE the elected proposer for this slot — build the unsigned block.
// The inner VM builds on ITS OWN head, and verifyPostForkChild below requires the
// transition block's inner parent to be exactly THIS block's inner block. Anchor the
// two together before delegating (see VM.anchorInnerBuildParent): at the fork height
// every validator may emit its own unsigned transition candidate, so a gossiped
// sibling that verifies here would otherwise drift the inner head and wedge the
// transition — the same self-rejection loop, at chain start.
if err := b.vm.anchorInnerBuildParent(ctx, parentID, b.Block.ID()); err != nil {
return nil, err
}
var innerBlock chain.Block
if b.vm.blockBuilderVM != nil {
// VM supports BuildBlockWithRuntime
+45
View File
@@ -0,0 +1,45 @@
// Copyright (C) 2019-2026, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package state
import (
"github.com/luxfi/database/prefixdb"
"github.com/luxfi/database/versiondb"
"github.com/luxfi/node/vms/proposervm/block"
)
// ScanBlocks visits every OUTER proposervm block persisted in the block store of
// [db], in key order.
//
// It is a REPAIR-PATH reader, deliberately kept OUT of the State interface: the
// hot path addresses blocks by id, and only the boot-time index rebuild
// (proposervm.rebuildOuterIndexFromStore) needs to enumerate them. Keeping it a
// free function over the same [blockStatePrefix] the store writes under means the
// prefix stays defined exactly once, and no mock/interface churn is imposed on
// callers that will never scan.
//
// Records that fail to decode are SKIPPED, not fatal: a rebuild must extract every
// usable wrapper from a store that may be partially damaged — that is the whole
// point of running it. Only an iterator-level error aborts the scan.
func ScanBlocks(db *versiondb.Database, visit func(block.Block) error) error {
blockDB := prefixdb.New(blockStatePrefix, db)
it := blockDB.NewIterator()
defer it.Release()
for it.Next() {
wrapper := blockWrapper{}
if err := parseBlockWrapper(it.Value(), &wrapper); err != nil {
continue
}
blk, err := block.ParseWithoutVerification(wrapper.Block)
if err != nil {
continue
}
if err := visit(blk); err != nil {
return err
}
}
return it.Error()
}
+177 -27
View File
@@ -121,6 +121,16 @@ type VM struct {
// [postForkBlock] and its inner block.
lastAcceptedTimestampGaugeVec metric.GaugeVec
// backfillMu guards [backfill]. It is a LEAF lock — never held across a
// callout — so it cannot deadlock against [lock], the [Tree] or the inner VM.
backfillMu sync.RWMutex
// backfill is non-nil ONLY while this node booted with a finality index that
// the local block store could not fully rebuild (see height_backfill.go). While
// it is set the chain runs read-only-ish: it refuses to BUILD blocks, and
// BackfillOuterBlock is the seam that completes the index. nil = index whole,
// which is the state of every healthy node.
backfill *outerBackfill
// quasarGate is the OPTIONAL post-quantum finality-cert gate. nil (the
// default) means PQ-finality verification is OFF — the accept path is
// unchanged classical Snow. When set AND forward-dated activation is reached,
@@ -332,6 +342,18 @@ func (vm *VM) SetState(ctx context.Context, newState uint32) error {
}
func (vm *VM) BuildBlock(ctx context.Context) (vmchain.Block, error) {
// FAIL-SAFE while the finality index has a hole. A node that booted with an
// incomplete outer index (height_backfill.go) does not know the canonical
// envelope at its own tip, so anything it proposed would extend a parent the
// network does not recognise. It stays a follower until BackfillOuterBlock
// closes the gap. Serving RPC and following the chain are unaffected — this is
// exactly the difference between the old dead C-Chain and a live one.
if from, to, pending := vm.NeedsOuterBackfill(); pending {
return nil, fmt.Errorf(
"proposervm: refusing to build — finality index incomplete, outer envelopes for heights %d..%d are missing",
from, to)
}
preferredBlock, err := vm.getBlock(ctx, vm.preferred)
if err == nil {
return preferredBlock.buildChild(ctx)
@@ -383,6 +405,17 @@ func (vm *VM) BuildBlock(ctx context.Context) (vmchain.Block, error) {
}
func (vm *VM) ParseBlock(ctx context.Context, b []byte) (vmchain.Block, error) {
// SELF-HEAL SEAM. Every outer envelope this node receives — gossip, GetAncestors,
// catch-up — arrives here. If this node booted with a hole in its finality index
// (height_backfill.go), offer the bytes to the backfill: it takes ONLY the exact
// next missing height, only if the envelope verifies AND wraps the inner block we
// already accepted there, so a wrong or hostile block cannot land. That makes a
// damaged node repair itself from ordinary peer traffic with no new transport and
// no operator step. Errors are expected and ignored (almost every block is not the
// one we need); when nothing is pending this is one RLock and out.
if vm.outerBackfillPending() {
_ = vm.BackfillOuterBlock(ctx, b)
}
if blk, err := vm.parsePostForkBlock(ctx, b, true); err == nil {
return blk, nil
}
@@ -474,6 +507,56 @@ func (vm *VM) SetPreference(ctx context.Context, preferred ids.ID) error {
return nil
}
// anchorInnerBuildParent points the inner VM at [innerParentID] — the inner block wrapped
// by the outer parent a build is about to extend — and is called from both build
// delegations (postForkCommonComponents.buildChild and preForkBlock.buildChild) as the
// last step before asking the inner VM for a block.
//
// THE INVARIANT it establishes is the one the verify path enforces:
// child.innerBlk.Parent() == parent.innerBlk.ID(), else errInnerParentMismatch
// (block.go, and pre_fork_block.go for the transition block). The inner VM never reads
// the proposervm's parent — it builds on ITS OWN head (luxfi/evm: the miner reads
// bc.CurrentBlock()) — so build and verify read two different pointers, and the builder's
// is not ours to assume.
//
// WHY MAINTAINING IT IN SetPreference ALONE IS NOT ENOUGH. The inner head moves for
// reasons the proposervm never observes: verifying a GOSSIPED block whose parent is the
// current head optimistically makes it the head (evm core/blockchain.go
// writeBlockAndSetHead → newTip → writeCanonicalBlockWithLogs → writeHeadBlock), with no
// proposervm involvement and no accept. SetPreference cannot undo that drift either — it
// short-circuits on an unchanged outer preference (above), so re-affirming the same tip
// never re-pushes the inner preference. Every subsequent build then extends the drifted
// head, the node REJECTS THE BLOCK IT JUST BUILT, drops it, and repeats forever:
// devnet 96367 / testnet 96368 on 2026-07-28, "built block failed verification —
// dropping / inner parentID didn't match expected parent", 83456 drops/min per node,
// the accepted tip frozen while the builder ran two heights ahead of it. Never
// self-corrects, on any node, ever.
//
// SetPreference on the INNER VM is the right primitive and the only one: it is that VM's
// own head-reorg entry point (evm VM.SetPreference → BlockChain.SetPreference →
// writeKnownBlock, which performs the reorg side effects). Asserting it at the point of
// use — rather than trusting a distant caller to have done it — is what makes the
// requirement local to the code that depends on it.
//
// COST AND SAFETY. On a healthy node the head already IS the parent's inner block and the
// inner SetPreference returns early on that identity (evm setPreference:
// `current.Hash() == block.Hash()`), so this is one lookup and no state change — zero
// behaviour change in the common case. When it fails, the head is provably NOT the
// parent's inner block, so any child built would fail this node's own Verify; refusing to
// build is strictly better than emitting a block we are guaranteed to drop.
func (vm *VM) anchorInnerBuildParent(ctx context.Context, parentID, innerParentID ids.ID) error {
if err := vm.ChainVM.SetPreference(ctx, innerParentID); err != nil {
vm.logger.Error("unexpected build block failure",
log.String("reason", "failed to anchor the inner VM on the parent's inner block"),
log.Stringer("parentID", parentID),
log.Stringer("innerParentID", innerParentID),
log.Err(err),
)
return fmt.Errorf("failed to anchor inner build parent %s: %w", innerParentID, err)
}
return nil
}
func (vm *VM) WaitForEvent(ctx context.Context) (vmcore.Message, error) {
for {
if err := ctx.Err(); err != nil {
@@ -783,35 +866,51 @@ func (vm *VM) repairAcceptedChainByHeight(ctx context.Context) error {
return nil
case heightBehind:
// INVARIANT VIOLATION and UNRECOVERABLE LOCALLY: the proposervm's finality
// index sits BELOW the inner VM's accepted tip. In a correct system this
// never happens — the proposervm last-accepted pointer and height index
// commit in the SAME versiondb batch as every inner accept, so they cannot
// lag. Reaching here means the on-disk proposervm state was truncated
// relative to the inner EVM — e.g. a snapshot restored inconsistently across
// the two databases (the devnet-C "index 7 < inner 8").
// INVARIANT VIOLATION: the proposervm's finality index sits BELOW the inner
// VM's accepted tip. Every post-fork accept commits the envelope, its height
// entry and the last-accepted pointer in ONE versiondb batch BEFORE the inner
// block is accepted, so this can only mean some accept advanced the inner VM
// WITHOUT going through acceptPostForkBlock (the pre-fork fallback — now
// prevented, see height_backfill.go) or that the on-disk proposervm state was
// truncated relative to the inner EVM.
//
// We FAIL LOUD rather than "self-heal", because there is no correct local
// heal: the proposervm cannot fabricate the missing outer wrapper blocks for
// heights (pro, inner]. In particular, dropping the finality pointer
// (DeleteLastAccepted) is NOT a heal — proposervm.LastAccepted() would then
// fall back to the inner-namespace id (see LastAccepted), whose ParentID is
// contiguity-incompatible with the network's OUTER wrappers, permanently
// wedging bootstrap (the first-block anchor), catch-up (the parent==tip
// guard) and live Verify (the parent lookup) at the inner tip — and since
// every path skips blocks at height <= the tip, the missing wrapper is never
// rebuilt. That silent wedge is strictly worse than this loud, actionable
// stop. The correct remedy is operator action; surface it explicitly.
return fmt.Errorf(
"proposervm finality index (height %d, id %s) is BEHIND the inner VM tip (height %d, id %s): "+
"the on-disk proposervm state is truncated/inconsistent relative to the inner EVM "+
"(e.g. a snapshot restored inconsistently across the proposervm and EVM databases). "+
"This cannot be repaired locally — the proposervm cannot rebuild the missing outer wrapper "+
"blocks. RECOVERY: restore a snapshot that is consistent across BOTH databases, or fully "+
"resync this node from peers (wipe this chain's db and re-bootstrap). Refusing to auto-reset "+
"the finality pointer, which would silently wedge this node at the inner tip forever",
proLastAcceptedHeight, proLastAcceptedID, innerLastAcceptedHeight, innerLastAcceptedID,
// This used to be a hard init failure, which killed the chain on the node
// ("non-critical chain failed to initialize chainAlias=C") and made every
// restart fatal. It is now REPAIRED — outer-only, with no inner
// re-execution:
//
// 1. re-derive the index from the outer envelopes already in this node's
// block store, each bound to the inner block WE accepted at that height;
// 2. if that cannot reach the tip, START ANYWAY in an explicit, loud,
// build-gated backfill-pending state that BackfillOuterBlock completes.
//
// The finality pointer is only ever moved FORWARD onto a proven envelope and
// is NEVER dropped: a DeleteLastAccepted here would make LastAccepted() fall
// back to an inner-namespace id whose ParentID is contiguity-incompatible
// with the network's outer wrappers, silently wedging bootstrap/catch-up/live
// Verify at the inner tip forever.
vm.logger.Warn("proposervm finality index is BEHIND the inner VM tip — repairing",
log.Uint64("indexHeight", proLastAcceptedHeight),
log.Stringer("indexID", proLastAcceptedID),
log.Uint64("innerTipHeight", innerLastAcceptedHeight),
log.Stringer("innerTipID", innerLastAcceptedID),
)
reached, err := vm.rebuildOuterIndexFromStore(
ctx, proLastAcceptedHeight, proLastAcceptedID, innerLastAcceptedHeight)
if err != nil {
return err
}
if reached >= innerLastAcceptedHeight {
vm.logger.Info("proposervm finality index REBUILT from the local block store — index and inner tip agree",
log.Uint64("fromHeight", proLastAcceptedHeight),
log.Uint64("toHeight", reached),
)
return nil
}
vm.enterOuterBackfill(reached+1, innerLastAcceptedHeight, innerLastAcceptedID)
return nil
}
// heightAhead: the inner vm is BEHIND the proposer vm (the inner rolled back or
@@ -1028,16 +1127,67 @@ func (vm *VM) getPreForkBlock(ctx context.Context, blkID ids.ID) (*preForkBlock,
if err != nil {
return nil, err
}
// ROOT-CAUSE GUARD (the finality-index lag). getBlock() falls back here
// whenever [blkID] is not an OUTER envelope id — and the id the consensus
// ledger records as canonical IS the inner block's id
// (postForkCommonComponents.CanonicalID). Without this check, asking the
// proposervm for a canonical id post-fork silently returns a preForkBlock
// wrapping a post-fork inner block, whose Accept advances the inner VM and
// leaves the outer index untouched (preForkBlock.acceptOuterBlk is a no-op).
// That is how the index ends up BEHIND the inner tip while everything looks
// healthy, and why the NEXT boot could not start the chain.
//
// Post-fork, a block at or above the recorded fork height is by definition NOT
// a pre-fork block: there is no such thing as a legitimate pre-fork block at a
// height the proposervm has already claimed. Refuse to construct it.
if err := vm.refusePreForkAfterFork(engineBlk.Height()); err != nil {
return nil, err
}
return &preForkBlock{
Block: engineBlk,
vm: vm,
}, nil
}
// refusePreForkAfterFork returns errPreForkAfterFork when [height] is at or above
// the recorded proposervm fork height. Before the fork (no fork height recorded)
// every block is legitimately pre-fork and this is a no-op, so a chain that has
// not yet transitioned is completely unaffected. ONE predicate, used by both the
// construction guard (getPreForkBlock) and the accept guard
// (preForkBlock.acceptOuterBlk), so the two can never disagree.
func (vm *VM) refusePreForkAfterFork(height uint64) error {
forkHeight, err := vm.State.GetForkHeight()
if err == database.ErrNotFound {
return nil // chain has not forked; everything is pre-fork
}
if err != nil {
return err
}
if height < forkHeight {
return nil
}
return fmt.Errorf("%w: height %d >= fork height %d", errPreForkAfterFork, height, forkHeight)
}
func (vm *VM) acceptPostForkBlock(blk PostForkBlock) error {
height := blk.Height()
blkID := blk.ID()
// EARLY DETECTOR for the finality-index lag. The index must advance one height
// at a time; a jump means some accept moved the inner VM without moving the
// index (the pre-fork fallback the guards above now refuse) and the gap will be
// fatal at the next boot. Surfacing it HERE names the height where the hole
// opened instead of leaving a post-mortem for the next restart. Not fatal: the
// accept itself is correct and the boot-time rebuild can close a hole; refusing
// finality on a warning would be the worse trade.
if vm.lastAcceptedHeight != 0 && height != vm.lastAcceptedHeight+1 {
vm.logger.Warn("proposervm finality index is NOT contiguous — a height was accepted without being indexed",
log.Uint64("previousIndexedHeight", vm.lastAcceptedHeight),
log.Uint64("acceptedHeight", height),
log.Stringer("blkID", blkID),
)
}
vm.lastAcceptedHeight = height
vm.forgetVerifiedBlock(blkID)
+299
View File
@@ -0,0 +1,299 @@
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// vm_crashboot_test.go — CRASH-COPY RECOVERY ARMOR.
//
// Every recovery test before this file repaired the SAME warm VM value: its
// Tree, verifiedBlocks map and caches were still populated, so a repair that
// silently leaned on in-memory state would still have passed. These tests
// remove that crutch. They copy the COMMITTED bytes out from under a RUNNING
// proposervm — no Shutdown, no extra Commit beyond what the accept path itself
// performed — and boot a SECOND, cold VM over the copy: a process kill plus
// restart, byte-faithful. The recovered VM must
//
// 1. BOOT — repairAcceptedChainByHeight then setLastAcceptedMetadata, the
// Initialize order — without error;
// 2. AGREE with the source's committed view at the copy instant: finality
// pointer, fork height, and an openable envelope at EVERY height; and
// 3. BUILD — produce a child whose outer parent is the recovered tip and
// whose inner parent is that tip's inner block. This is the
// errInnerParentMismatch invariant (build_inner_parent_test.go) proven
// from a cold boot on persisted state, not just at steady state.
//
// The matrix pins the persistence boundaries: nothing accepted yet, exactly
// one accept (the fork-height record), a longer run, and a copy taken while
// the source keeps accepting (the copy must be an immutable snapshot).
// TestCrashBoot_OuterCommittedInnerNot pins the ONE window the accept path
// leaves open: acceptPostForkBlock commits the outer batch BEFORE the inner
// VM accepts, so a crash between the two strands the index one ahead — boot
// must roll the pointer back onto the inner survivor and keep building.
//
// The shared harness lives in height_lag_repro_test.go.
package proposervm
import (
"context"
"errors"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/luxfi/database"
"github.com/luxfi/database/memdb"
"github.com/luxfi/ids"
validatorstest "github.com/luxfi/validators/validatorstest"
vmchain "github.com/luxfi/vm/chain"
"github.com/luxfi/vm/chain/blocktest"
)
// crashCopy copies every committed key/value out of a live base database into
// a fresh memdb — a byte-level snapshot taken with the source still running.
// This is exactly what a crash leaves behind: versiondb buffers uncommitted
// writes in memory, and they die with the process.
func crashCopy(t *testing.T, base database.Database) *memdb.Database {
t.Helper()
cp := memdb.New()
it := base.NewIterator()
defer it.Release()
for it.Next() {
if err := cp.Put(it.Key(), it.Value()); err != nil {
t.Fatalf("crashCopy Put: %v", err)
}
}
if err := it.Error(); err != nil {
t.Fatalf("crashCopy iterator: %v", err)
}
return cp
}
// snapshot clones the inner chain as its own database would survive a crash at
// this instant: every block ever inserted (verified-but-unaccepted included,
// as the real EVM persists bodies on insert) with the acceptance tip pinned at
// [tip]. The clone is immune to the source's subsequent progress.
func (ic *innerChain) snapshot(tip uint64) *innerChain {
cp := &innerChain{
byHeight: make(map[uint64]*blocktest.Block, len(ic.byHeight)),
byID: make(map[ids.ID]*blocktest.Block, len(ic.byID)),
byBytes: make(map[string]*blocktest.Block, len(ic.byBytes)),
tip: tip,
}
for h, b := range ic.byHeight {
cp.byHeight[h] = b
}
for id, b := range ic.byID {
cp.byID[id] = b
}
for s, b := range ic.byBytes {
cp.byBytes[s] = b
}
return cp
}
// buildableVM is ic.vm() plus the two calls a build needs. SetPreference is
// anchorInnerBuildParent's seam — it moves the head exactly as the EVM does.
// BuildBlock mints a FRESH child of the head (building is not accepting, and a
// replacement proposal after a crash is a new block, not the lost one).
func (ic *innerChain) buildableVM() *blocktest.VM {
vm := ic.vm()
vm.SetPreferenceF = func(_ context.Context, id ids.ID) error {
blk, ok := ic.byID[id]
if !ok {
return errors.New("inner: preference not in chain")
}
ic.tip = blk.HeightV
return nil
}
vm.BuildBlockF = func(context.Context) (vmchain.Block, error) {
parent := ic.byHeight[ic.tip]
id := ids.GenerateTestID()
child := &blocktest.Block{
IDV: id,
ParentV: parent.IDV,
HeightV: parent.HeightV + 1,
BytesV: append([]byte("inner"), id[:]...),
TimestampV: parent.TimestampV.Add(time.Second),
}
ic.byID[id] = child
ic.byBytes[string(child.BytesV)] = child
return child, nil
}
return vm
}
// bootRecoveredVM stands a COLD VM up over copied bytes and runs the boot
// sequence in Initialize order. Windower/validator-state/no-op wiring matches
// build_inner_parent_test.go: every slot open, so builds take the unsigned
// path and the recovery properties are the only thing under test.
func bootRecoveredVM(t *testing.T, ic *innerChain, base database.Database) *VM {
t.Helper()
vm := testVMOnBase(t, ic, base)
vm.ChainVM = ic.buildableVM()
vm.Windower = anyoneCanProposeWindower{}
vm.validatorState = &validatorstest.State{
GetCurrentHeightF: func(context.Context) (uint64, error) { return 0, nil },
}
if err := vm.repairAcceptedChainByHeight(context.Background()); err != nil {
t.Fatalf("BOOT KILLED THE CHAIN (repair): %v", err)
}
if err := vm.setLastAcceptedMetadata(context.Background()); err != nil {
t.Fatalf("BOOT KILLED THE CHAIN (metadata): %v", err)
}
return vm
}
// mustBuildOnRecoveredTip drives the recovered VM's REAL public build path —
// SetPreference(last accepted) then BuildBlock, what the engine does after a
// restart — and asserts the built child extends the recovered tip on BOTH
// layers: outer parent == the tip envelope, inner parent == the tip's inner
// block. A recovered node failing either would reject its own proposal.
func mustBuildOnRecoveredTip(t *testing.T, vm *VM, ic *innerChain) {
t.Helper()
require := require.New(t)
ctx := context.Background()
// Deterministic clock strictly after the recovered tip's timestamp
// (harness blocks carry time.Unix(height, 0)).
vm.Clock.Set(time.Unix(int64(ic.tip)+2, 0))
lastAccepted, err := vm.LastAccepted(ctx)
require.NoError(err)
require.NoError(vm.SetPreference(ctx, lastAccepted),
"the recovered tip must be fetchable — SetPreference validates before assigning")
built, err := vm.BuildBlock(ctx)
require.NoError(err, "a crash-recovered node must be able to build")
child, ok := built.(*postForkBlock)
require.True(ok, "the child of a recovered tip is a post-fork envelope")
wantInnerParent := ic.byHeight[ic.tip].IDV
require.Equal(wantInnerParent, child.innerBlk.Parent(),
"the built child's inner parent must be the recovered tip's inner block "+
"(errInnerParentMismatch invariant, from a cold boot)")
require.Equal(ic.tip+1, child.Height())
require.Equal(lastAccepted, child.Parent(),
"the built child's outer parent must be the recovered finality pointer")
}
// TestCrashBoot_ColdVMOverDirtyCopy_AgreesAndBuilds is the persistence-boundary
// matrix: commit-state copies taken with NO shutdown, each booted by a second,
// cold VM that must agree with the source's committed view at the copy instant
// and then build on it.
func TestCrashBoot_ColdVMOverDirtyCopy_AgreesAndBuilds(t *testing.T) {
for _, tt := range []struct {
name string
accepted uint64 // heights accepted through the proposervm before the copy
runOnAfter uint64 // heights the SOURCE keeps accepting after the copy
}{
// Nothing ever accepted through the proposervm: no fork height, no
// outer state. Boot must be a no-op and the pre-fork world intact.
{name: "nothing_accepted", accepted: 0, runOnAfter: 0},
// Exactly one accept — the boundary that records the fork height.
{name: "first_accept_records_fork", accepted: 1, runOnAfter: 0},
{name: "several_accepted", accepted: 8, runOnAfter: 0},
// The copy is taken and the source RUNS ON: later accepts must not
// leak into the recovered view (the copy is an immutable snapshot).
{name: "copy_while_source_runs_on", accepted: 5, runOnAfter: 3},
} {
t.Run(tt.name, func(t *testing.T) {
require := require.New(t)
ctx := context.Background()
ic := newInnerChain(t, tt.accepted+tt.runOnAfter+2)
base := memdb.New()
src := testVMOnBase(t, ic, base)
outers := acceptThroughProposervm(t, src, ic, tt.accepted)
// THE COPY — no Shutdown, the source stays live. Snapshot both
// layers at the same instant, as one machine's disk would be.
cp := crashCopy(t, base)
icAtCopy := ic.snapshot(ic.tip)
if tt.runOnAfter > 0 {
acceptRangeThroughProposervm(
t, src, ic, tt.accepted+1, tt.accepted+tt.runOnAfter, outers[tt.accepted].ID())
if got := indexHeight(t, src); got != tt.accepted+tt.runOnAfter {
t.Fatalf("source must have run on to %d, at %d", tt.accepted+tt.runOnAfter, got)
}
}
vm2 := bootRecoveredVM(t, icAtCopy, cp)
if tt.accepted == 0 {
// Pre-fork world: no outer state exists, none may be invented.
_, err := vm2.State.GetLastAccepted()
require.ErrorIs(err, database.ErrNotFound)
_, err = vm2.State.GetForkHeight()
require.ErrorIs(err, database.ErrNotFound)
// LastAccepted falls through to the inner chain.
la, err := vm2.LastAccepted(ctx)
require.NoError(err)
require.Equal(icAtCopy.byHeight[icAtCopy.tip].IDV, la)
} else {
// The recovered finality pointer is the copy-instant tip —
// not genesis, not the source's later progress.
got, err := vm2.State.GetLastAccepted()
require.NoError(err)
require.Equal(outers[tt.accepted].ID(), got)
require.Equal(mustForkHeight(t, src), mustForkHeight(t, vm2))
// Every height's envelope is indexed AND openable cold.
for h := uint64(1); h <= tt.accepted; h++ {
id, err := vm2.State.GetBlockIDAtHeight(h)
require.NoError(err, "height %d must be indexed", h)
require.Equal(outers[h].ID(), id, "height %d", h)
blk, err := vm2.getPostForkBlock(ctx, id)
require.NoError(err, "envelope at height %d must be openable on the recovered VM", h)
require.Equal(h, blk.Height())
}
}
if _, _, pending := vm2.NeedsOuterBackfill(); pending {
t.Fatal("a lock-step copy has no gap — no backfill may be pending")
}
mustBuildOnRecoveredTip(t, vm2, icAtCopy)
})
}
}
// TestCrashBoot_OuterCommittedInnerNot_RollsBackAndBuilds pins the one crash
// window the accept path leaves open. acceptPostForkBlock commits the envelope,
// height index and finality pointer in ONE batch BEFORE the inner VM accepts
// (that ordering is why the index can never legitimately fall behind). A crash
// inside the window therefore persists an index one AHEAD of the inner
// survivor. Boot must take repairAcceptedChainByHeight's roll-back arm — move
// the pointer back onto the height the inner VM survived at — and the node
// must then build its replacement proposal there.
func TestCrashBoot_OuterCommittedInnerNot_RollsBackAndBuilds(t *testing.T) {
require := require.New(t)
ctx := context.Background()
ic := newInnerChain(t, 8)
base := memdb.New()
src := testVMOnBase(t, ic, base)
outers := acceptThroughProposervm(t, src, ic, 4)
require.Equal(uint64(4), ic.tip, "precondition: lock-step through 4")
// Height 5's outer accept COMMITS…
sb := outerFor(t, ic, 5, outers[4].ID())
blk := &postForkBlock{
SignedBlock: sb,
postForkCommonComponents: postForkCommonComponents{vm: src, innerBlk: ic.byHeight[5]},
}
src.Tree.Add(ic.byHeight[5])
require.NoError(blk.Accept(ctx))
// …and the process dies BEFORE the inner VM accepts 5 (no ic.accept(5)).
cp := crashCopy(t, base)
icAtCopy := ic.snapshot(4)
vm2 := bootRecoveredVM(t, icAtCopy, cp)
got, err := vm2.State.GetLastAccepted()
require.NoError(err)
require.Equal(outers[4].ID(), got,
"boot must roll the finality pointer back onto the height the inner VM survived at")
// The node re-proposes at 5: a NEW envelope over a NEW inner child of 4 —
// the lost block is gone, not resurrected.
mustBuildOnRecoveredTip(t, vm2, icAtCopy)
}
@@ -17,7 +17,7 @@ import (
"github.com/luxfi/crypto/secp256k1"
"github.com/luxfi/go-bip32"
"github.com/luxfi/go-bip39"
"github.com/luxfi/keys"
"github.com/luxfi/kms/pkg/mnemonic"
)
const (
@@ -31,10 +31,10 @@ const (
// MustLoadKey loads a secp256k1 private key from (in order of priority):
// 1. MNEMONIC environment variable (BIP-39 mnemonic phrase)
// 2. via native ZAP — when KMS_ADDR + KMS_ENV +
// 2. via native ZAP — when KMS_ADDR + KMS_ENV +
// KMS_MNEMONIC_PATH are set in the environment. Uses the canonical
// luxfi/kms keys.LoadMnemonic loader so every Lux-derived
// service resolves keys the same way.
// kms/pkg/mnemonic loader so every Lux-derived service resolves
// keys the same way.
// 3. Key name provided as first command-line argument
// 4. ~/.lux/keys/default/ if it exists
//
@@ -58,28 +58,31 @@ func MustLoadKey() *secp256k1.PrivateKey {
// LoadKey attempts to load a private key using the priority order above.
func LoadKey() (*secp256k1.PrivateKey, error) {
// 1. MNEMONIC env var — local dev + CI test seam.
if mnemonic := strings.TrimSpace(os.Getenv("MNEMONIC")); mnemonic != "" {
return keyFromMnemonic(mnemonic)
if phrase := strings.TrimSpace(os.Getenv("MNEMONIC")); phrase != "" {
return keyFromMnemonic(phrase)
}
// 2. via native ZAP — production path for any Lux-derived
// service running under a KMS-projected env. The canonical loader
// lives in luxfi/keys (alongside the BIP-39 derivation primitives)
// so luxd, netrunner, lux/cli, and every descending L1's bootstrap
// all resolve mnemonics the same way.
// Trust at the network boundary (NetworkPolicy + ZAP wire) — the
// staking material itself comes from the KMS-held operational
// mnemonic at KMS_MNEMONIC_PATH (default /mnemonic).
// lives in kms/pkg/mnemonic, not luxfi/keys: keys must not import
// kms, because kms already imports keys for ServiceIdentity, and
// the back edge would close an import cycle. luxd, netrunner,
// lux/cli, and every descending L1's bootstrap all resolve
// mnemonics through it.
// Trust at the network boundary (NetworkPolicy + ZAP wire) — hence
// a nil ServiceIdentity — the staking material itself comes from
// the KMS-held operational mnemonic at KMS_MNEMONIC_PATH.
if addr := os.Getenv("KMS_ADDR"); addr != "" {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
mnemonic, err := keys.LoadMnemonicFromKMS(ctx, addr,
phrase, err := mnemonic.LoadFromKMS(ctx, addr,
os.Getenv("KMS_ENV"),
envOr("KMS_MNEMONIC_PATH", "/mnemonic"))
envOr("KMS_MNEMONIC_PATH", "/mnemonic"),
nil)
if err != nil {
return nil, fmt.Errorf("load mnemonic from KMS: %w", err)
}
return keyFromMnemonic(mnemonic)
return keyFromMnemonic(phrase)
}
// 3. Key name from command-line arguments.