mirror of
https://github.com/luxfi/node.git
synced 2026-07-28 18:57:33 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bd2edc135f | ||
|
|
bdd7179164 | ||
|
|
50fe597bf7 | ||
|
|
8873e231e8 | ||
|
|
2a724fb090 | ||
|
|
23d93bf60b | ||
|
|
8a797f4805 | ||
|
|
9f4819cf77 | ||
|
|
84a08c2b9c | ||
|
|
011e9bf99d | ||
|
|
c39a85522d | ||
|
|
1c92026c7e | ||
|
|
f7e501d021 | ||
|
|
a57251c318 | ||
|
|
2f6ef061ab | ||
|
|
7d2f01eb0c | ||
|
|
c76fcda71d | ||
|
|
e1e3917772 | ||
|
|
e7137ed28d | ||
|
|
3398024a19 | ||
|
|
e77b696662 |
+14
-1
@@ -277,7 +277,20 @@ RUN . ./build_env.sh && \
|
||||
ARG EVM_CGO=0
|
||||
ARG EVM_TAGS=""
|
||||
|
||||
ARG EVM_VERSION=v1.104.9
|
||||
# v1.104.22 realigns the plugin with THIS node's own go.mod: api v1.0.16 ->
|
||||
# v1.1.1, vm v1.2.6 -> v1.3.1, geth v1.17.12 -> v1.20.1. Node main pins api
|
||||
# v1.1.1, so pinning an evm at api v1.0.16 reintroduces the exact
|
||||
# InitializeResponse decode mismatch described above, just in the opposite
|
||||
# direction — keep this ARG and node's go.mod on the same api/vm/geth line.
|
||||
#
|
||||
# v1.104.14 is also the FIRST evm tag carrying the C-Chain fee-split seam
|
||||
# (core/fee_split.go creditTxFee + extras.FeeSplitTimestamp/FeeRewardVault).
|
||||
# Below it, encoding/json silently DISCARDS the genesis "feeSplitTimestamp"
|
||||
# key: a chain configured for the 50/50 split instead routes 100% of every fee
|
||||
# to the block coinbase and the reward vault 0x0100..0002 stays 0 forever, with
|
||||
# nothing burned. The split stays dormant wherever feeSplitTimestamp is absent
|
||||
# (mainnet), so this bump is behaviour-preserving there.
|
||||
ARG EVM_VERSION=v1.104.23
|
||||
ARG EVM_VM_ID=mgj786NP7uDwBCcq6YwThhaN8FLyybkCa4zBWTQbNgmK6k9A6
|
||||
# the pinned evm go.mod may pin a dead luxfi/upgrade pseudo-version
|
||||
# (v1.0.1-0.20260603055252-f51810805436 — commit pruned from origin). Heal it to
|
||||
|
||||
@@ -253,6 +253,275 @@ when v1.36.12 image is ready → set devnet sts image v1.36.12, delete luxd-1, c
|
||||
C-Chain inits + reaches tip; then finish devnet (durable-fix proof + RewardManager),
|
||||
then gated testnet→mainnet→zoo.
|
||||
|
||||
## Crash-boot recovery armor + upstream delta verdict (vms/proposervm, test-only, 2026-07-28)
|
||||
|
||||
**Upstream review (avalanchego master `bcc851822d..c5d3c8aafe`, 6 commits): ZERO code
|
||||
ports.** grpc bump (no named bug; would cascade through luxfi/vm), README typo, typed
|
||||
sync-client scaffolding (client-side only, no serving handlers even upstream; our sync
|
||||
direction is ZAP-native — reference for task #66 only), a Firewood `debug_intermediateRoots`
|
||||
enabler (our tracers API is `.disabled`, twice N/A), and two streaming-VM-only mempool
|
||||
guards whose failure modes we already bracket (`miner/worker.go:62 targetTxsSize=1800KiB`
|
||||
build cap + txpool `ErrGasLimit`/128KB caps under the 2MiB wire cap). **Nothing in the
|
||||
delta touches proposervm — the open `vm.go:380` preferred-fetch failure gets no upstream
|
||||
help.** One upstream BRANCH does touch proposervm — `containerman17/proposervm-dedup-capability`
|
||||
(+72 vm.go): inner-bytes dedup in the block store plus an extra boot-repair arm for the
|
||||
unclean-shutdown window where the outer index lands above the inner survivor (the very
|
||||
window `OuterCommittedInnerNot` below pins) — storage-layer work, not the build-path
|
||||
preferred fetch, so it changes nothing about `vm.go:380` either. What WAS worth taking is
|
||||
their recovery-test discipline, ported as:
|
||||
|
||||
**`vm_crashboot_test.go`** — copy the COMMITTED bytes out from under a RUNNING proposervm
|
||||
(no Shutdown), boot a SECOND, cold VM over the copy (repair + metadata, Initialize order),
|
||||
assert source-equality (finality pointer, fork height, every height's envelope openable
|
||||
cold) and then BUILD on the recovered tip (outer parent == recovered envelope, inner
|
||||
parent == its inner block — the errInnerParentMismatch invariant from a cold boot).
|
||||
Matrix: nothing-accepted / first-accept / longer run / copy-while-source-runs-on, plus
|
||||
`OuterCommittedInnerNot`: the one crash window the accept path leaves open
|
||||
(`acceptPostForkBlock` commits BEFORE the inner accept) must boot via the roll-back arm
|
||||
and re-propose. Harness seams added in height_lag_repro_test.go (`testVMOnBase`,
|
||||
`acceptRangeThroughProposervm`). **Negative control executed**: deleting the
|
||||
`vm.db.Commit()` from `acceptPostForkBlock` fails every persistence-bearing scenario
|
||||
(4 of 6) — the armor bites on flush-order regressions, not just on green paths.
|
||||
|
||||
## v1.36.35 — the certified-descendant false halt + the plugin-killing map fatal (devnet 96367, 2026-07-28)
|
||||
|
||||
> Shipped as v1.36.35, not v1.36.34: the v1.36.34 tag inherited a build break that
|
||||
> `011e9bf99d` ("deps: drop go.sum lines that disagree with the checksum log") had
|
||||
> already pushed to main — it bumped `zap-proto/http` from a pseudo-version to v0.3.0,
|
||||
> whose `Server.Handler` is a `fasthttp.RequestHandler` and whose `NewTransport` is now
|
||||
> `Dial(network, addr)`, so `server/http/zap_listener.go` no longer compiled. **main was
|
||||
> unbuildable from that commit until this one.** Repaired here by bridging the one
|
||||
> net/http handler chain with `fasthttpadaptor.NewFastHTTPHandler` (the two transports
|
||||
> stay behaviourally identical — only the wire encoding differs) and moving the
|
||||
> round-trip test onto the fasthttp request/response pair. v1.36.34 produced no image
|
||||
> and its tag was deleted.
|
||||
|
||||
Rolling devnet to v1.36.33 fixed the build→verify-fail→drop loop and unmasked two
|
||||
DIFFERENT failures. Both are fixed here; each has a regression test that fails before
|
||||
and passes after.
|
||||
|
||||
**P0-1 — five validators `os.Exit(1)` on a benign state.** Each devnet node hit, once:
|
||||
|
||||
```
|
||||
error VM accepted head is CONSENSUS-CERTIFIED and conflicts with the newly finalized
|
||||
block — refusing to orphan it orphanedHeight=1364
|
||||
fatal SetPreference would orphan a CONSENSUS-CERTIFIED block — refusing (fail-closed)
|
||||
error="cannot orphan finalized block at height: 1364 to common block at height: 1363"
|
||||
```
|
||||
|
||||
at three distinct height pairs (1258/1259, 1363/1364, 1364/1365), ALWAYS with
|
||||
`certified = head − 1`, with every surviving node holding byte-identical blocks at all
|
||||
five heights — no fork anywhere. (The "1168/1169" pair in the original report never
|
||||
appears in any fatal line; those two blocks are a normal parent/child present on all
|
||||
live nodes.)
|
||||
|
||||
Two defects in `luxfi/consensus`, one crash — fixed in **consensus v1.36.12**:
|
||||
|
||||
- *Producer.* `acceptWithCertCore` releases `t.mu` across every VM call-out, then steers
|
||||
the VM with its STALE local `blockID`. A finalize that completed in that window has
|
||||
already advanced the ledger AND the EVM to `blockID+1`, so the steer is BACKWARDS and
|
||||
the EVM's accepted-irreversibility guard (`evm/core/blockchain.go:1987`,
|
||||
`commonBlock < lastAccepted`) correctly refuses it. **That guard is right and is
|
||||
untouched.** Now steers at the live build anchor (`PreferredBuildTip` →
|
||||
`ledger.BuildAnchor`), which the accept ordering (ApplyCert BEFORE VM.Accept) keeps at
|
||||
or above the VM's own accepted head. Same value the build path already uses.
|
||||
- *Classifier.* `reconcileVMToCertified` asked only "is the head the ledger's certified
|
||||
canonical at ITS OWN height?" — trivially true for every healthy node whose head is
|
||||
certified — and called that a two-blocks-at-one-height double-finalization. It never
|
||||
established that `certified` was at that same height. The ledger holds one canonical
|
||||
per height along one contiguous chain, so when both are certified at their own heights
|
||||
they lie on that one chain and the head merely DESCENDS from the target: nothing is
|
||||
orphaned. **The fail-closed halt is not weakened** — it now fires on the state that is
|
||||
actually unsafe (steering off a certified head onto a block our own ledger does not
|
||||
certify at its height).
|
||||
|
||||
Direction: NEITHER roll the head back NOR certify forward. The VM head is legitimately
|
||||
ahead and already CONTAINS the certified block; `FinalityLedger.BuildAnchor` already
|
||||
documents `head > certified` as the designed state, and rolling back is exactly what
|
||||
`blockchain.go:1987` exists to refuse. The correct action is no action — plus not issuing
|
||||
the backwards steer at all.
|
||||
|
||||
**P0-2 — the EVM plugin process dies and never comes back.** devnet luxd-1:
|
||||
|
||||
```
|
||||
fatal error: concurrent map read and map write
|
||||
vm/components/chain.(*State).getCachedBlock state.go:216
|
||||
vm/components/chain.(*State).ParseBlock state.go:267
|
||||
evm/plugin/evm.(*VM).ParseBlock vm.go:340
|
||||
vm/rpc.(*zapVMServer).handleParseBlock vm_server_zap.go:477
|
||||
```
|
||||
|
||||
`chain.State` was written against avalanchego's contract that the consensus engine holds
|
||||
the chain lock across every VM call. The ZAP VM server does NOT reinstate it — it
|
||||
dispatches ParseBlock/GetBlock and the Verify/Accept/Reject wrappers concurrently.
|
||||
`verifiedBlocks` (plain map) and `lastAcceptedBlock` (pointer) are the only State that is
|
||||
not self-synchronising; every `cache.Cacher` carries its own mutex. Fixed in **vm v1.3.3**
|
||||
by giving State one RWMutex for exactly those two, never held across a call into the
|
||||
inner VM. A Go map fatal is unrecoverable: it kills the plugin, **luxd survives and keeps
|
||||
answering `info.getNodeVersion` while its chain is gone** — pod-Ready and `/v1/health`
|
||||
both stay green — and there is no self-heal.
|
||||
|
||||
**Devnet roll result (10:23–10:45Z).** All five pods on v1.36.35, `restarts=0` on every one
|
||||
for 20+ minutes. Both fixed defects are GONE fleet-wide: `CONSENSUS-CERTIFIED` fatal = 0
|
||||
(previously all five died within minutes), `concurrent map` = 0. luxd-1, whose C-Chain had
|
||||
been dead ~45 min with no self-heal, came back at tip parity on first boot; luxd-3, stuck at
|
||||
1524, caught up immediately. A transaction of ours got a receipt with identical status,
|
||||
blockHash and resulting balance on all five nodes (`0xcb5e9a06…`, block 1636, status 0x1).
|
||||
|
||||
**NOT accepted — a THIRD, pre-existing defect blocks five-way parity.** Two nodes (luxd-2 at
|
||||
1789, luxd-4 at 1825) freeze their tip while 0/1/3 advance in lockstep, emitting
|
||||
|
||||
```
|
||||
error unexpected build block failure error="not found"
|
||||
reason="failed to fetch preferred block; no distinct last-accepted fallback"
|
||||
```
|
||||
|
||||
from `vms/proposervm/vm.go:380`. That branch is reached when the preferred block is
|
||||
unfetchable AND last-accepted is either unreadable or the same id — a local storage/index
|
||||
condition, not a steering choice. It is **NOT a regression from this release**: the identical
|
||||
line appears on binaries built long before it — mainnet luxd-1 on **v1.36.2** (2,223
|
||||
occurrences, and that is the one mainnet node still producing) and testnet luxd-3 on
|
||||
**v1.36.24** (35,693). It is also survivable: the stalled node keeps VOTING, so quorum holds
|
||||
and the chain keeps finalizing (devnet reached 1913 with two nodes frozen), and devnet luxd-1
|
||||
self-recovered from it once (1609 → 1634). Left unfixed and unchained-to — it needs its own
|
||||
diagnosis at the proposervm layer.
|
||||
|
||||
**Quorum arithmetic (reported, not changed).** Devnet and testnet both run
|
||||
`--consensus-sample-size=5 --consensus-quorum-size=4`. With only 3 live C-Chains the
|
||||
quorum is arithmetically unreachable; devnet demonstrated both directions in one session
|
||||
(pinned at 1378 on 3 live, 1378→1395→1396 the moment luxd-4 restored a 4th). This is a
|
||||
config value, not a code defect, and lowering it lowers the safety margin — left as is.
|
||||
|
||||
### Testnet 96368 roll (15:32–16:00Z) — accepted, after two blockers the devnet roll never hit
|
||||
|
||||
Testnet was frozen at **1779** with only three live C-Chains, below `--consensus-quorum-size=4`
|
||||
(`ceil(2·5/3)+1`), so no block could be accepted. luxd-1 and luxd-4 had no C-Chain at all —
|
||||
`/v1/bc/C/rpc` 404 — on `failed to repair accepted chain by height: proposervm finality index
|
||||
(height 1453 / 1463) is BEHIND the inner VM tip (height 1491)`. v1.36.35 turns that fatal into
|
||||
a repair, and it worked on the first boot of each: `proposervm finality index REBUILT from the
|
||||
local block store — index and inner tip agree fromHeight=1453 toHeight=1491`. The freeze broke
|
||||
the instant a **fourth** C-Chain came up. Final: five nodes on v1.36.35, every binary
|
||||
self-reporting `luxd/1.36.35`, `1779 → 1888` and climbing, tips in exact agreement, and one
|
||||
real transaction (`0x333d5bbd…`, block `0x728`) returning `status=0x1` with **identical
|
||||
blockHash `0x07fd0c68…` and `gasUsed=0x5208` on all five nodes**. α was NOT lowered.
|
||||
|
||||
Two defects had to be fixed first. Both are invisible on devnet and both apply to any fleet.
|
||||
|
||||
**1. The RLP startup import is fatal on re-run — it kills the C-Chain on EVERY boot.**
|
||||
Testnet's startup script passes `--import-chain-data` on every boot, relying on
|
||||
`isNothingToImportError` to make re-importing an already-imported chain a no-op. That guard
|
||||
only ever existed on `luxfi/evm` `hotfix/v1.104.9` (commit `c58d307e`, tags
|
||||
`v1.104.9-hotfix.2/3/4`); `git merge-base --is-ancestor c58d307e main` = **false**. Only the
|
||||
*other* half of that commit was forward-ported. So images built from evm main die with
|
||||
`startup import failed: no blocks imported (parsed=0)`. Proven from the two plugin binaries
|
||||
in-cluster, with a positive control:
|
||||
|
||||
| string in `plugins/mgj786NP7…` | v1.36.24 | v1.36.35 |
|
||||
|---|---|---|
|
||||
| `ImportChain: resuming from current head` | 1 | 1 ← control |
|
||||
| `nothing to import` | 1 | **0** ← the guard |
|
||||
| `no blocks imported (parsed=` | 1 | 1 |
|
||||
|
||||
Fixed **twice, on purpose**: the guard is restored in `luxfi/evm` main (with
|
||||
`startup_import_idempotency_test.go` locking the contract), and the flag is now gated on a
|
||||
per-PVC sentinel in `universe/k8s/lux-testnet/luxd-startup.yaml` — a completed one-time
|
||||
migration must not re-run forever. Sentinel pre-seeded on all five PVCs before the ConfigMap
|
||||
was patched. **Mainnet is NOT exposed**: its `luxd-startup` ConfigMap (39,719 bytes, contains
|
||||
`consensus-quorum-size` twice as a control) has zero `--import-chain-data` and no `.rlp` on disk.
|
||||
|
||||
**2. 🚨 The `luxfi/vm` map-race fix never reached the C-Chain.** node v1.36.35 bumped
|
||||
`luxfi/vm` to v1.3.3 for it, but the C-Chain is a **plugin built from `luxfi/evm`**, which
|
||||
still pinned **v1.3.1**. Read off the two binaries inside one v1.36.35 pod:
|
||||
|
||||
```
|
||||
/luxd/build/luxd github.com/luxfi/vm@v1.3.3
|
||||
/luxd/build/plugins/mgj786NP7… github.com/luxfi/vm@v1.3.1 ← verifies the blocks
|
||||
```
|
||||
|
||||
It fired on testnet luxd-0 five minutes after the roll: `fatal error: concurrent map writes`
|
||||
in `luxfi/vm@v1.3.1/components/chain/block.go:44 (*BlockWrapper).Verify` via
|
||||
`rpc/vm_server_zap.go:580 handleBlockVerify`. v1.3.3 is precisely the fix for that line — it
|
||||
takes `state.blocksLock` around `verifiedBlocks[blkID] = bw`, which v1.3.1 wrote unlocked from
|
||||
every concurrent ZAP RPC handler. `luxfi/evm` main is now on v1.3.3; **the next node image
|
||||
must be built after that bump, or this race ships again.**
|
||||
|
||||
⚠️ **The readiness probe cannot see this.** The plugin dies while luxd survives, so the pod
|
||||
stays `ready=true`, `restarts=0`, and `info.isBootstrapped(C)` keeps answering `true` while
|
||||
`eth_blockNumber` times out — a dead C-Chain still in the Service, the exact failure the probe
|
||||
was redesigned to catch. Only a per-node **tip** probe sees it. Recovery is a pod delete.
|
||||
|
||||
## v1.36.33 — the build→self-verify-fail→drop loop (devnet 96367 / testnet 96368, 2026-07-28)
|
||||
|
||||
**Symptom.** Every proposer built a block and then rejected the block it had just
|
||||
built, forever: `built block … height=1047` immediately followed by
|
||||
`built block failed verification — dropping error="inner parentID didn't match
|
||||
expected parent"`, 83–456 drops/min per node, accepted tip frozen two heights BELOW
|
||||
what the builder kept proposing (devnet tip 1045, builds 1047).
|
||||
|
||||
**Root cause (one line).** `buildChild` asked the inner VM for a block without first
|
||||
pointing the inner VM at the parent's inner block. The inner VM builds on ITS OWN
|
||||
head (`luxfi/evm` miner reads `bc.CurrentBlock()`); the verify path requires
|
||||
`child.innerBlk.Parent() == parent.innerBlk.ID()`. Two different pointers, one
|
||||
required equal to the other, nothing asserting it. The head drifts on its own:
|
||||
verifying a GOSSIPED block whose parent is the current head optimistically sets the
|
||||
head (`core/blockchain.go writeBlockAndSetHead → newTip → writeCanonicalBlockWithLogs`),
|
||||
and `VM.SetPreference` short-circuits on an unchanged outer preference so it never
|
||||
re-pushes the inner preference to drag the head back. Non-self-correcting by
|
||||
construction.
|
||||
|
||||
**Fix.** `VM.anchorInnerBuildParent` (vms/proposervm/vm.go) — one inner
|
||||
`SetPreference` at the point of use, called from BOTH build delegations
|
||||
(`postForkCommonComponents.buildChild`, `preForkBlock.buildChild`). On a healthy node
|
||||
the inner `setPreference` early-returns on `current.Hash() == block.Hash()`, so it is
|
||||
a lookup and no state change; when it fails, the head is provably not the parent's
|
||||
inner block, so refusing to build beats emitting a block we would drop.
|
||||
Regression proof: `vms/proposervm/build_inner_parent_test.go` (models the three evm
|
||||
head semantics — build-on-head, verify-advances-head, SetPreference-reorgs-head).
|
||||
|
||||
**NOT the cause, measured:** the P-Chain. `info.isBootstrapped{"chain":"P"}` = true on
|
||||
15/15 nodes and `platform.getHeight` = 0 on 15/15 **including mainnet**, whose built
|
||||
blocks also carry `pChainHeight=0`. `bootstrapped.message:["111…LpoYY"]` is the
|
||||
primary-network NET id (`chains/chains.go Nets.Bootstrapping` keys by net), not the
|
||||
P-Chain — the P-Chain's chain id prints `111…P` (`constants.PlatformChainID =
|
||||
ids.PChainID`, while `PrimaryNetworkID = ids.Empty`). The verify path DOES read
|
||||
pChainHeight before the parent check, but only as monotonicity
|
||||
(`childPChainHeight < parentPChainHeight`, and 0 < 0 is false, so it passes); every
|
||||
P-Chain-DEPENDENT validation — epoch, `GetCurrentHeight`, proposer window — is gated
|
||||
behind `consensusState == Ready` and sits AFTER the parent check. So `pChainHeight=0`
|
||||
cannot produce `errInnerParentMismatch`.
|
||||
|
||||
**Sibling failure modes on the same fleets (already fixed in 1.36.32, needs the roll):**
|
||||
outer index BEHIND the inner tip ⇒ `refusing to build`/boot repair
|
||||
(`height_backfill.go`, 7d2f01eb), and preferred-absent-locally ⇒ last-accepted
|
||||
fallback (`vm.go BuildBlock`). All three are the same invariant seen from three sides.
|
||||
|
||||
**Roll surface — measured 2026-07-28T07:5xZ, do not use the LuxNetwork CR.**
|
||||
`lux-operator` and `lux-operator-devnet` are **0/0** in `lux-system`, so nothing
|
||||
reconciles `luxnetworks.lux.cloud/luxd`; its tags are stale garbage (mainnet
|
||||
`v1.34.0`, testnet `v1.32.12` — v1.34.0 exists in no registry). The live image is on
|
||||
the **StatefulSet**, hand-maintained: mainnet `ghcr.io/luxfi/node:v1.36.2`
|
||||
(`kubectl-patch` 07-25T00:27:04Z), devnet `v1.36.25@sha256:ca497eff…`, testnet
|
||||
`v1.36.24@sha256:91e2542b…`, all three `updateStrategy: OnDelete`. Change the image in
|
||||
the universe manifest and apply, then delete one pod. Editing the CR and deleting a pod
|
||||
reboots it on the OLD image.
|
||||
|
||||
**Mainnet 96369 is NOT a clean control — 3 of 5 nodes have Mode B, invisibly.**
|
||||
luxd-0/3/4 `eth_blockNumber` = `0x10c1cf` (1098191, block ts 2026-07-24T15:46:19Z) and
|
||||
have not moved in 3+ days while luxd-1 mines (1098341, tx status 0x1, 07:58Z); luxd-2
|
||||
serves 404 (no C-Chain). Not a fork — 1098191 hashes identically on luxd-0 and luxd-1.
|
||||
3998 of luxd-0's last 4000 log lines are the same `built block … height=1098196`. The
|
||||
drop line is absent because the **binary** lacks it: `grep -c "built block failed
|
||||
verification" /luxd/build/luxd` = **0** on mainnet v1.36.2, **1** on devnet v1.36.25.
|
||||
`/v1/health` says `{"healthy":true,"error":"health reply encode failed"}` on the frozen
|
||||
node and the mining node alike — never gate on it, use tip parity.
|
||||
Two hazards for the roll: the 4 broken mainnet pods are exactly the ones on
|
||||
ControllerRevision `luxd-596857c9d6` (rev 147, kubectl-patch 07-25: GOMEMLIMIT=6GiB,
|
||||
mem limit 16Gi→12Gi, request 4Gi→7Gi) and the one mining node, luxd-1, is the only pod
|
||||
still on rev 144 — **deleting luxd-1 recreates it on the revision every other node
|
||||
broke on**. And `ghcr.io/luxfi/node:v1.36.2` has no git tag at all
|
||||
(`git ls-remote origin refs/tags/v1.36.2` → empty; v1.36.24/25 → present), so what
|
||||
mainnet runs is not reproducible from this repo.
|
||||
|
||||
## Essential Commands
|
||||
|
||||
### Release & build (canonical) — via platform.hanzo.ai, NOT GitHub Actions
|
||||
|
||||
@@ -57,14 +57,28 @@ const (
|
||||
// tip, so a single (even >⅔-stake) validator cannot alone determine the frontier.
|
||||
// Capped at the beacon-set size for tiny networks.
|
||||
bootstrapMinAgreeingBeacons = 2
|
||||
// bootstrapNamingWindow bounds the ancestry the ANCESTOR-TOLERANT frontier tally fetches
|
||||
// per candidate anchor to resolve the bleeding-edge skew between honest beacons. On a live
|
||||
// chain that skew is tiny (the canary was ONE block: 2 producers at N, 1 at N+1), so a
|
||||
// single 256-block window covers it with enormous margin. A ⅔-common height more than this
|
||||
// far below the highest tip is not a healthy bleeding-edge split — the tally names nothing
|
||||
// (the loop then retries / fails safe), never a wrong block. Matches the consensus descent's
|
||||
// per-round fetch size.
|
||||
// bootstrapNamingWindow is the PER-FETCH ancestry chunk of the ANCESTOR-TOLERANT frontier
|
||||
// tally. Ancestry returns FULL BLOCKS, so one fetch must stay inside a network message —
|
||||
// this is a TRANSPORT bound, matching the consensus descent's per-round fetch size.
|
||||
//
|
||||
// It was previously ALSO doing two other jobs: the total ancestry budget, and a HEALTH
|
||||
// HEURISTIC ("a ⅔-common height more than this far below the highest tip is not a healthy
|
||||
// bleeding-edge split"). That heuristic is true for a LIVE chain and FALSE for a HALTED
|
||||
// one, where the skew between a straggler and a node that ran on alone is arbitrarily large
|
||||
// and perfectly healthy — no fork, just a stopped chain. Conflating the three WEDGED
|
||||
// mainnet 96369: responders were split 1098726 (1 node) / 1098191 (2 nodes), 1098191 IS an
|
||||
// ancestor of 1098726, and the ⅔-of-responder floor would have named it — but the gap was
|
||||
// 535 blocks, the single 256-block fetch never reached down far enough for the high tip to
|
||||
// vouch for it, and the tally named nothing on every retry, forever. Recovery from a halt
|
||||
// is exactly when this path matters most, and it was disabled precisely then.
|
||||
// The total budget is now maxNamingDepth; the health heuristic is gone.
|
||||
bootstrapNamingWindow = 256
|
||||
// bootstrapMaxNamingDepth is the TOTAL ancestry one anchor may be walked down, in
|
||||
// bootstrapNamingWindow-sized chunks. Purely a resource bound (with maxNamingAnchors and
|
||||
// bootstrapNamingTimeout): 32768 covers ~9h of 1s blocks of halt-skew, and the common
|
||||
// healthy case never fetches at all — a single tip clearing the floor takes the exact fast
|
||||
// path with zero fetches. Safety does NOT come from this number; see maxNamingDepth().
|
||||
bootstrapMaxNamingDepth = 32768
|
||||
// maxNamingAnchors bounds how many DISTINCT reported tips the ancestor-tolerant tally will
|
||||
// fetch ancestry for in one round. Honest beacons cluster on a handful of adjacent tips, so
|
||||
// this is never reached in practice; it caps the work a Byzantine swarm reporting many
|
||||
|
||||
+52
-11
@@ -243,10 +243,13 @@ type BootstrapPolicy struct {
|
||||
// CheckpointVerifier authenticates the Checkpoint's authority signature (INVARIANT 4). nil ⇒ a
|
||||
// configured Checkpoint is NOT trusted (fail closed) — a bare (id,height) is never enough.
|
||||
CheckpointVerifier CheckpointVerifier
|
||||
// NamingWindow bounds the ancestry fetched per anchor; MaxAnchors bounds how many distinct
|
||||
// reported tips are resolved. Both default to the package constants when zero.
|
||||
NamingWindow int
|
||||
MaxAnchors int
|
||||
// NamingWindow is the per-FETCH ancestry chunk (Ancestry returns FULL blocks, so one fetch
|
||||
// must stay inside a network message); MaxNamingDepth is the TOTAL ancestry a single anchor
|
||||
// may be walked down in NamingWindow-sized chunks; MaxAnchors bounds how many distinct
|
||||
// reported tips are resolved. All default to the package constants when zero.
|
||||
NamingWindow int
|
||||
MaxNamingDepth int
|
||||
MaxAnchors int
|
||||
// NamingTimeout TOTAL-bounds the ancestor-tolerant resolution (all anchor fetches combined) so
|
||||
// a partition that ANSWERS the frontier query but WITHHOLDS ancestry cannot make the decision
|
||||
// hang — it returns what it found (or nothing → ErrNoBootstrapQuorum) and the caller's bounded
|
||||
@@ -299,6 +302,17 @@ func (p *BootstrapPolicy) namingWindow() int {
|
||||
return bootstrapNamingWindow
|
||||
}
|
||||
|
||||
// maxNamingDepth is the TOTAL ancestry one anchor may be walked down, in namingWindow-sized
|
||||
// chunks. It is a RESOURCE bound and nothing else. Safety comes from hash-verified ancestry
|
||||
// (a forged chain cannot link), MinResponders distinct voters, the ⅔-of-RESPONDER-stake floor,
|
||||
// and the full re-Verify every block gets on the descent — none of which depend on this number.
|
||||
func (p *BootstrapPolicy) maxNamingDepth() int {
|
||||
if p.MaxNamingDepth > 0 {
|
||||
return p.MaxNamingDepth
|
||||
}
|
||||
return bootstrapMaxNamingDepth
|
||||
}
|
||||
|
||||
func (p *BootstrapPolicy) maxAnchors() int {
|
||||
if p.MaxAnchors > 0 {
|
||||
return p.MaxAnchors
|
||||
@@ -505,14 +519,41 @@ func (p *BootstrapPolicy) nameFrontier(ctx context.Context, stakeOnTip map[ids.I
|
||||
break
|
||||
}
|
||||
fetches++
|
||||
refs, err := p.Source.Ancestry(ctx, tip, p.namingWindow())
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, ref := range refs {
|
||||
if _, ok := index[ref.ID]; !ok {
|
||||
index[ref.ID] = ref
|
||||
// CHUNKED DESCENT. namingWindow is the per-FETCH size (Ancestry returns FULL blocks, so
|
||||
// one fetch must fit a network message) — NOT the total ancestry worth walking. Keep
|
||||
// following the parent chain in window-sized chunks up to maxNamingDepth, so a fleet
|
||||
// that HALTED with a straggler far below the highest tip still resolves its ⅔-backed
|
||||
// common ancestor. A single un-chunked window silently capped this at 256 and wedged
|
||||
// mainnet at a 535-block gap. ctx already TOTAL-bounds every fetch (namingTimeout).
|
||||
cur := tip
|
||||
for depth := 0; depth < p.maxNamingDepth(); {
|
||||
// The deadline must bind the WALK, not just each fetch: a Byzantine peer serving a
|
||||
// long fabricated chain cheaply (or an in-process Source) would otherwise run the
|
||||
// full depth budget before anyone checked the clock.
|
||||
if ctx.Err() != nil {
|
||||
break
|
||||
}
|
||||
refs, err := p.Source.Ancestry(ctx, cur, p.namingWindow())
|
||||
if err != nil || len(refs) == 0 {
|
||||
break
|
||||
}
|
||||
deepest, first := BlockRef{}, true
|
||||
for _, ref := range refs {
|
||||
if _, ok := index[ref.ID]; !ok {
|
||||
index[ref.ID] = ref
|
||||
}
|
||||
if first || ref.Height < deepest.Height {
|
||||
deepest, first = ref, false
|
||||
}
|
||||
}
|
||||
depth += len(refs) // len(refs) >= 1 here, so depth strictly advances -> terminates
|
||||
if deepest.Parent == ids.Empty {
|
||||
break // reached genesis
|
||||
}
|
||||
if _, covered := index[deepest.Parent]; covered {
|
||||
break // an earlier anchor's chain already covers everything below
|
||||
}
|
||||
cur = deepest.Parent
|
||||
}
|
||||
}
|
||||
if len(index) == 0 {
|
||||
|
||||
@@ -916,3 +916,175 @@ func TestBootstrapTrust_EclipseOwnHeightNotNamedRoutesToCaughtUp(t *testing.T) {
|
||||
require.Equal(t, refs[N].ID, f.ID, "boundary: N is named iff its height is STRICTLY ABOVE MinFrontierHeight")
|
||||
require.Equal(t, uint64(N), f.Height)
|
||||
}
|
||||
|
||||
// ----- H: HALT-SKEW RECOVERY (mainnet 96369 wedge) ---------------------------
|
||||
|
||||
// TestBootstrapTrust_H_HaltSkewDeeperThanOneWindow reproduces the live mainnet 96369 wedge and
|
||||
// proves the chunked descent fixes it.
|
||||
//
|
||||
// SHAPE (measured per-node, in-pod, 2026-07-28): the fleet HALTED below its α=4-of-5 threshold.
|
||||
// One node ran on alone to 1098726 while two stayed at 1098191; 1098191 IS an ancestor of
|
||||
// 1098726 (no fork — luxd-3/luxd-4's `latest` IS 1098191, they hold nothing competing), so the
|
||||
// ⅔-of-RESPONDER floor is satisfied at 1098191 by all three responders and it MUST be named.
|
||||
//
|
||||
// It was not. nameFrontier fetched one bootstrapNamingWindow (256) of ancestry per anchor, and
|
||||
// the gap is 535 — so the high tip's ancestry never reached down far enough to vouch for the
|
||||
// common block. 1098191 held only its 2 direct responders (2 of 3 = below the ⅔ floor of 2,
|
||||
// which the strict `>` rejects), the tally named NOTHING, and every retry did the same. The
|
||||
// node sat at height 0 forever and mainnet could not regain quorum.
|
||||
//
|
||||
// The gap here (535) is deliberately > one window (256) and < maxNamingDepth. With the
|
||||
// single-fetch window this FAILS (frontier is nil → ErrNoBootstrapQuorum); with the chunked
|
||||
// descent the walk continues past the first chunk and names the common ancestor.
|
||||
func TestBootstrapTrust_H_HaltSkewDeeperThanOneWindow(t *testing.T) {
|
||||
const w uint64 = 100
|
||||
const gap = 535 // luxd-1 1098726 - luxd-3/luxd-4 1098191, the measured mainnet skew
|
||||
|
||||
// common is the last block the whole fleet accepted; `ahead` extends it by `gap`.
|
||||
refs, byID := refChain(1000)
|
||||
common := refs[1000]
|
||||
prev := common
|
||||
for i := 0; i < gap; i++ {
|
||||
c := childRef(prev)
|
||||
byID[c.ID] = c
|
||||
prev = c
|
||||
}
|
||||
ahead := prev
|
||||
require.Equal(t, common.Height+gap, ahead.Height)
|
||||
require.Greater(t, gap, bootstrapNamingWindow, "gap MUST exceed one fetch window or this proves nothing")
|
||||
require.Less(t, gap, bootstrapMaxNamingDepth, "gap must stay inside the total depth budget")
|
||||
|
||||
beacons := nodeIDs(3) // the three responders with a live C-Chain
|
||||
policy := &BootstrapPolicy{
|
||||
TrustedBeacons: equalBeacons(beacons, w),
|
||||
MinResponses: 3,
|
||||
Source: &stubAncestry{byID: byID},
|
||||
}
|
||||
replies := []BeaconReply{
|
||||
reply(beacons[0], ahead.ID, w), // luxd-1, ran on alone
|
||||
reply(beacons[1], common.ID, w), // luxd-3
|
||||
reply(beacons[2], common.ID, w), // luxd-4
|
||||
}
|
||||
|
||||
f, err := policy.AcceptsFrontier(context.Background(), replies)
|
||||
require.NoError(t, err, "a fleet split across ONE chain must name a frontier, not wedge")
|
||||
require.NotNil(t, f)
|
||||
require.Equal(t, common.ID, f.ID,
|
||||
"must name the ⅔-backed COMMON ancestor: the high tip vouches for it via ancestry")
|
||||
require.Equal(t, common.Height, f.Height)
|
||||
}
|
||||
|
||||
// TestBootstrapTrust_H_HaltSkewBeyondDepthStillFailsSafe pins the resource bound's edge: a skew
|
||||
// DEEPER than maxNamingDepth still names nothing rather than guessing. Depth is a work bound, so
|
||||
// exhausting it must degrade to the SAME fail-safe as before, never to a wrong block.
|
||||
func TestBootstrapTrust_H_HaltSkewBeyondDepthStillFailsSafe(t *testing.T) {
|
||||
const w uint64 = 100
|
||||
refs, byID := refChain(10)
|
||||
common := refs[10]
|
||||
prev := common
|
||||
for i := 0; i < 64; i++ {
|
||||
c := childRef(prev)
|
||||
byID[c.ID] = c
|
||||
prev = c
|
||||
}
|
||||
ahead := prev
|
||||
|
||||
beacons := nodeIDs(3)
|
||||
policy := &BootstrapPolicy{
|
||||
TrustedBeacons: equalBeacons(beacons, w),
|
||||
MinResponses: 3,
|
||||
NamingWindow: 4, // tiny chunk...
|
||||
MaxNamingDepth: 8, // ...and a depth budget far shallower than the 64-block skew
|
||||
Source: &stubAncestry{byID: byID},
|
||||
}
|
||||
_, err := policy.AcceptsFrontier(context.Background(), []BeaconReply{
|
||||
reply(beacons[0], ahead.ID, w),
|
||||
reply(beacons[1], common.ID, w),
|
||||
reply(beacons[2], common.ID, w),
|
||||
})
|
||||
require.Error(t, err, "a skew beyond the depth budget must fail SAFE, never name a guess")
|
||||
}
|
||||
|
||||
// TestBootstrapTrust_H_AcceptanceMatrix pins the owner-specified recovery matrix at the policy
|
||||
// layer. Each case is the SAME five-validator mainnet shape with a different responder pattern.
|
||||
//
|
||||
// The rule being pinned is "highest verifiably-vouched descendant wins", NOT "numerically highest
|
||||
// tip advertised". A tip whose ancestry does not link into the fleet's chain earns NO credit no
|
||||
// matter how high it claims to be, so a Byzantine node cannot pull the fleet onto a fabricated
|
||||
// chain by advertising a tall one.
|
||||
func TestBootstrapTrust_H_AcceptanceMatrix(t *testing.T) {
|
||||
const w uint64 = 100
|
||||
mk := func() (BlockRef, BlockRef, map[ids.ID]BlockRef) {
|
||||
refs, byID := refChain(600)
|
||||
common := refs[600]
|
||||
prev := common
|
||||
for i := 0; i < 535; i++ { // the measured mainnet skew, > one 256 window
|
||||
c := childRef(prev)
|
||||
byID[c.ID] = c
|
||||
prev = c
|
||||
}
|
||||
return common, prev, byID
|
||||
}
|
||||
|
||||
t.Run("1_high_2_low_2_unavailable__names_common_ancestor", func(t *testing.T) {
|
||||
common, ahead, byID := mk()
|
||||
b := nodeIDs(3)
|
||||
p := &BootstrapPolicy{TrustedBeacons: equalBeacons(b, w), MinResponses: 3, Source: &stubAncestry{byID: byID}}
|
||||
f, err := p.AcceptsFrontier(context.Background(), []BeaconReply{
|
||||
reply(b[0], ahead.ID, w), reply(b[1], common.ID, w), reply(b[2], common.ID, w)})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, common.ID, f.ID)
|
||||
})
|
||||
|
||||
t.Run("3_high_2_unavailable__names_high_tip", func(t *testing.T) {
|
||||
_, ahead, byID := mk()
|
||||
b := nodeIDs(3)
|
||||
p := &BootstrapPolicy{TrustedBeacons: equalBeacons(b, w), MinResponses: 3, Source: &stubAncestry{byID: byID}}
|
||||
f, err := p.AcceptsFrontier(context.Background(), []BeaconReply{
|
||||
reply(b[0], ahead.ID, w), reply(b[1], ahead.ID, w), reply(b[2], ahead.ID, w)})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, ahead.ID, f.ID, "unanimous high tip must be named, not an ancestor")
|
||||
})
|
||||
|
||||
t.Run("fabricated_tall_tip_2_low__rejects_fake_names_common", func(t *testing.T) {
|
||||
common, _, byID := mk()
|
||||
// A Byzantine node advertises a tip on a chain of its own that never links into the
|
||||
// fleet's history. Its ancestry earns credit only for ITS OWN blocks, never for the
|
||||
// fleet's — so it cannot outvote the two honest low responders.
|
||||
fakeRefs, fakeByID := refChain(9_000_000)
|
||||
fake := fakeRefs[9_000_000]
|
||||
for id, r := range fakeByID {
|
||||
byID[id] = r
|
||||
}
|
||||
b := nodeIDs(3)
|
||||
p := &BootstrapPolicy{TrustedBeacons: equalBeacons(b, w), MinResponses: 3, Source: &stubAncestry{byID: byID}}
|
||||
f, err := p.AcceptsFrontier(context.Background(), []BeaconReply{
|
||||
reply(b[0], fake.ID, w), reply(b[1], common.ID, w), reply(b[2], common.ID, w)})
|
||||
// The fake tip earns credit ONLY on its own disjoint chain (100), far below the ⅔ floor
|
||||
// of 200, so it can never be named however tall it claims to be. `common` earns exactly
|
||||
// 200 from its two honest backers — and the floor is STRICT (`> floor`), so 200 is not
|
||||
// enough either: the Byzantine node withheld the third vouch by sitting on a chain that
|
||||
// does not link. Correct outcome is a SAFE HALT, not "fall back to the low tip".
|
||||
// This is the honest BFT trade: 1 of 3 responders can cost LIVENESS, never SAFETY.
|
||||
require.Error(t, err, "must halt safely; must NOT follow a taller unvouched chain")
|
||||
require.Nil(t, f)
|
||||
})
|
||||
|
||||
t.Run("two_conflicting_branches_same_height__halts_safely", func(t *testing.T) {
|
||||
refs, byID := refChain(600)
|
||||
common := refs[600]
|
||||
// Two disjoint branches of equal height off the common block, each backed by one node,
|
||||
// and NO responder majority on either. Nothing may be named by height tiebreak.
|
||||
l, r := childRef(common), childRef(common)
|
||||
byID[l.ID], byID[r.ID] = l, r
|
||||
b := nodeIDs(3)
|
||||
p := &BootstrapPolicy{
|
||||
TrustedBeacons: equalBeacons(b, w), MinResponses: 3,
|
||||
MinFrontierHeight: common.Height, // node already holds `common`; only a tip AHEAD may be named
|
||||
Source: &stubAncestry{byID: byID},
|
||||
}
|
||||
_, err := p.AcceptsFrontier(context.Background(), []BeaconReply{
|
||||
reply(b[0], l.ID, w), reply(b[1], r.ID, w), reply(b[2], common.ID, w)})
|
||||
require.Error(t, err, "conflicting equal-height branches must halt safely, never pick by height")
|
||||
})
|
||||
}
|
||||
|
||||
@@ -26,7 +26,7 @@ require (
|
||||
github.com/huin/goupnp v1.3.0
|
||||
github.com/jackpal/gateway v1.1.1
|
||||
github.com/jackpal/go-nat-pmp v1.0.2
|
||||
github.com/luxfi/consensus v1.36.9
|
||||
github.com/luxfi/consensus v1.36.12
|
||||
github.com/luxfi/crypto v1.20.2
|
||||
github.com/luxfi/database v1.21.1
|
||||
github.com/luxfi/ids v1.3.2
|
||||
@@ -56,13 +56,13 @@ require (
|
||||
go.opentelemetry.io/otel/trace v1.44.0 // indirect
|
||||
go.uber.org/goleak v1.3.0
|
||||
go.uber.org/mock v0.6.0
|
||||
golang.org/x/crypto v0.52.0
|
||||
golang.org/x/crypto v0.54.0
|
||||
golang.org/x/exp v0.0.0-20260529124908-c761662dc8c9 // indirect
|
||||
golang.org/x/mod v0.36.0
|
||||
golang.org/x/net v0.55.0
|
||||
golang.org/x/sync v0.20.0
|
||||
golang.org/x/mod v0.37.0
|
||||
golang.org/x/net v0.57.0
|
||||
golang.org/x/sync v0.22.0
|
||||
golang.org/x/time v0.15.0
|
||||
golang.org/x/tools v0.45.0
|
||||
golang.org/x/tools v0.47.0
|
||||
gonum.org/v1/gonum v0.17.0
|
||||
google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
@@ -90,7 +90,7 @@ require (
|
||||
github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674
|
||||
github.com/hashicorp/golang-lru v1.0.2 // indirect
|
||||
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
||||
github.com/klauspost/compress v1.18.6
|
||||
github.com/klauspost/compress v1.19.1
|
||||
github.com/kr/pretty v0.3.1 // indirect
|
||||
github.com/kr/text v0.2.0 // indirect
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
||||
@@ -107,8 +107,8 @@ require (
|
||||
github.com/yusufpapurcu/wmi v1.2.4 // indirect
|
||||
go.opentelemetry.io/otel/metric v1.44.0 // indirect
|
||||
go.uber.org/multierr v1.11.0 // indirect
|
||||
golang.org/x/sys v0.45.0 // indirect
|
||||
golang.org/x/text v0.37.0 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
golang.org/x/text v0.40.0 // indirect
|
||||
)
|
||||
|
||||
require (
|
||||
@@ -130,7 +130,7 @@ require (
|
||||
github.com/luxfi/genesis/pkg/genesis/security v1.13.8
|
||||
github.com/luxfi/geth v1.20.1
|
||||
github.com/luxfi/go-bip39 v1.2.0
|
||||
github.com/luxfi/keys v1.4.1
|
||||
github.com/luxfi/kms v1.12.10
|
||||
github.com/luxfi/math/safe v0.0.1
|
||||
github.com/luxfi/net v0.1.1
|
||||
github.com/luxfi/p2p v1.22.1
|
||||
@@ -145,33 +145,34 @@ require (
|
||||
github.com/luxfi/utils v1.3.1
|
||||
github.com/luxfi/utxo v0.5.8
|
||||
github.com/luxfi/validators v1.3.1
|
||||
github.com/luxfi/vm v1.3.1
|
||||
github.com/luxfi/vm v1.3.3
|
||||
github.com/luxfi/warp v1.24.1
|
||||
github.com/luxfi/zap v1.2.6
|
||||
github.com/luxfi/zwing v0.6.1
|
||||
github.com/nbutton23/zxcvbn-go v0.0.0-20210217022336-fa2cb2858354
|
||||
github.com/zap-proto/http v0.0.0-20260506200741-fd6047874433
|
||||
github.com/valyala/fasthttp v1.73.0
|
||||
github.com/zap-proto/http v0.3.0
|
||||
go.uber.org/zap v1.27.1
|
||||
)
|
||||
|
||||
require (
|
||||
capnproto.org/go/capnp/v3 v3.0.1-alpha.2 // indirect
|
||||
filippo.io/edwards25519 v1.2.0 // indirect
|
||||
filippo.io/hpke v0.4.0 // indirect
|
||||
github.com/andybalholm/brotli v1.2.2 // indirect
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.5 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.5 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.13 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.13 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.21 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.6 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.18 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.10 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.18 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.2 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.3 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.0.9 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.14 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.18 // indirect
|
||||
@@ -180,7 +181,6 @@ require (
|
||||
github.com/btcsuite/btcd/btcec/v2 v2.5.0 // indirect
|
||||
github.com/btcsuite/btcd/chainhash/v2 v2.0.0 // indirect
|
||||
github.com/cenkalti/backoff v2.2.1+incompatible // indirect
|
||||
github.com/colega/zeropool v0.0.0-20230505084239-6fb4a4f75381 // indirect
|
||||
github.com/decred/dcrd/crypto/blake256 v1.1.0 // indirect
|
||||
github.com/go-ini/ini v1.67.0 // indirect
|
||||
github.com/goccy/go-yaml v1.19.2 // indirect
|
||||
@@ -191,9 +191,11 @@ require (
|
||||
github.com/hanzos3/go-sdk v1.0.2 // indirect
|
||||
github.com/klauspost/crc32 v1.3.0 // indirect
|
||||
github.com/luxfi/age v1.6.0 // indirect
|
||||
github.com/luxfi/bft v0.1.5 // indirect
|
||||
github.com/luxfi/corona v0.10.4 // indirect
|
||||
github.com/luxfi/crypto/ipa v1.2.4 // indirect
|
||||
github.com/luxfi/dkg v0.3.5 // indirect
|
||||
github.com/luxfi/keys v1.4.1 // indirect
|
||||
github.com/luxfi/lattice/v7 v7.1.4 // indirect
|
||||
github.com/luxfi/lens v0.2.1 // indirect
|
||||
github.com/luxfi/light v1.0.0 // indirect
|
||||
@@ -214,6 +216,8 @@ require (
|
||||
github.com/philhofer/fwd v1.2.0 // indirect
|
||||
github.com/rs/xid v1.6.0 // indirect
|
||||
github.com/tinylib/msgp v1.6.4 // indirect
|
||||
github.com/valyala/bytebufferpool v1.0.0 // indirect
|
||||
github.com/zap-proto/go v1.1.0 // indirect
|
||||
go.mongodb.org/mongo-driver v1.17.9 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
|
||||
)
|
||||
|
||||
@@ -1,7 +1,5 @@
|
||||
c2sp.org/CCTV/age v0.0.0-20251208015420-e9274a7bdbfd h1:ZLsPO6WdZ5zatV4UfVpr7oAwLGRZ+sebTUruuM4Ra3M=
|
||||
c2sp.org/CCTV/age v0.0.0-20251208015420-e9274a7bdbfd/go.mod h1:SrHC2C7r5GkDk8R+NFVzYy/sdj0Ypg9htaPXQq5Cqeo=
|
||||
capnproto.org/go/capnp/v3 v3.0.1-alpha.2 h1:W/cf+XEArUSwcBBE/9wS2NpWDkM5NLQOjmzEiHZpYi0=
|
||||
capnproto.org/go/capnp/v3 v3.0.1-alpha.2/go.mod h1:2vT5D2dtG8sJGEoEKU17e+j7shdaYp1Myl8X03B3hmc=
|
||||
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
|
||||
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
|
||||
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
|
||||
@@ -19,10 +17,14 @@ github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA
|
||||
github.com/StephenButtolph/canoto v0.17.3 h1:lvsnYD4b96vD1knnmp1xCmZqfYpY/jSeRozGdOfdvGI=
|
||||
github.com/StephenButtolph/canoto v0.17.3/go.mod h1:IcnAHC6nJUfQFVR9y60ko2ecUqqHHSB6UwI9NnBFZnE=
|
||||
github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII=
|
||||
github.com/andybalholm/brotli v1.2.2 h1:HzTuoo2ErYQqf5qvcJInB8uvqSVxRttzkFexPWtnceM=
|
||||
github.com/andybalholm/brotli v1.2.2/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.5 h1:dj5kopbwUsVUVFgO4Fi5BIT3t4WyqIDjGKCangnV/yY=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.5/go.mod h1:mwsPRE8ceUUpiTgF7QmQIJ7lgsKUPQOUl3o72QBrE1o=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.5 h1:zWFmPmgw4sveAYi1mRqG+E/g0461cJ5M4bJ8/nc6d3Q=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.5/go.mod h1:nVUlMLVV8ycXSb7mSkcNu9e3v/1TJq2RTlrPwhYWr5c=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 h1:eBMB84YGghSocM7PsjmmPffTa+1FBUeNvGvFou6V/4o=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8/go.mod h1:lyw7GFp3qENLh7kwzf7iMzAxDn+NzjXEAGjKS2UOKqI=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.13 h1:5KgbxMaS2coSWRrx9TX/QtWbqzgQkOdEa3sZPhBhCSg=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.13/go.mod h1:8zz7wedqtCbw5e9Mi2doEwDyEgHcEE9YOJp6a8jdSMY=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.13 h1:mA59E3fokBvyEGHKFdnpNNrvaR351cqiHgRg+JzOSRI=
|
||||
@@ -37,16 +39,24 @@ github.com/aws/aws-sdk-go-v2/internal/ini v1.8.6 h1:qYQ4pzQ2Oz6WpQ8T3HvGHnZydA72
|
||||
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.6/go.mod h1:O3h0IK87yXci+kg6flUKzJnWeziQUKciKrLjcatSNcY=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.18 h1:eZioDaZGJ0tMM4gzmkNIO2aAoQd+je7Ug7TkvAzlmkU=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.18/go.mod h1:CCXwUKAJdoWr6/NcxZ+zsiPr6oH/Q5aTooRGYieAyj4=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22 h1:rWyie/PxDRIdhNf4DzRk0lvjVOqFJuNnO8WwaIRVxzQ=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22/go.mod h1:zd/JsJ4P7oGfUhXn1VyLqaRZwPmZwg44Jf2dS84Dm3Y=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7 h1:5EniKhLZe4xzL7a+fU3C2tfUN4nWIqlLesfrjkuPFTY=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7/go.mod h1:x0nZssQ3qZSnIcePWLvcoFisRXJzcTVvYpAAdYX8+GI=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.10 h1:fJvQ5mIBVfKtiyx0AHY6HeWcRX5LGANLpq8SVR+Uazs=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.10/go.mod h1:Kzm5e6OmNH8VMkgK9t+ry5jEih4Y8whqs+1hrkxim1I=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13 h1:JRaIgADQS/U6uXDqlPiefP32yXTda7Kqfx+LgspooZM=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13/go.mod h1:CEuVn5WqOMilYl+tbccq8+N2ieCy0gVn3OtRb0vBNNM=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21 h1:c31//R3xgIJMSC8S6hEVq+38DcvUlgFY0FM6mSI5oto=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21/go.mod h1:r6+pf23ouCB718FUxaqzZdbpYFyDtehyZcmP5KL9FkA=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.18 h1:/A/xDuZAVD2BpsS2fftFRo/NoEKQJ8YTnJDEHBy2Gtg=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.18/go.mod h1:hWe9b4f+djUQGmyiGEeOnZv69dtMSgpDRIvNMvuvzvY=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21 h1:ZlvrNcHSFFWURB8avufQq9gFsheUgjVD9536obIknfM=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21/go.mod h1:cv3TNhVrssKR0O/xxLJVRfd2oazSnZnkUeTf6ctUwfQ=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.2 h1:M1A9AjcFwlxTLuf0Faj88L8Iqw0n/AJHjpZTQzMMsSc=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.2/go.mod h1:KsdTV6Q9WKUZm2mNJnUFmIoXfZux91M3sr/a4REX8e0=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.3 h1:HwxWTbTrIHm5qY+CAEur0s/figc3qwvLWsNkF4RPToo=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.3/go.mod h1:uoA43SdFwacedBfSgfFSjjCvYe8aYBS7EnU5GZ/YKMM=
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.0.9 h1:QKZH0S178gCmFEgst8hN0mCX1KxLgHBKKY/CLqwP8lg=
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.0.9/go.mod h1:7yuQJoT+OoH8aqIxw9vwF+8KpvLZ8AWmvmUWHsGQZvI=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.14 h1:GcLE9ba5ehAQma6wlopUesYg/hbcOhFNWTjELkiWkh4=
|
||||
@@ -111,8 +121,6 @@ github.com/cockroachdb/redact v1.1.8 h1:8eVLLj6juKxiKrAEw2b8cJvNqWq++U8WOfQFuL7K
|
||||
github.com/cockroachdb/redact v1.1.8/go.mod h1:GceHHpJ0rMDpYARL5In88Alq/xMBUtVlz7Qxix6ZVkw=
|
||||
github.com/cockroachdb/tokenbucket v0.0.0-20250429170803-42689b6311bb h1:3bCgBvB8PbJVMX1ouCcSIxvsqKPYM7gs72o0zC76n9g=
|
||||
github.com/cockroachdb/tokenbucket v0.0.0-20250429170803-42689b6311bb/go.mod h1:7nc4anLGjupUW/PeY5qiNYsdNXj7zopG+eqsS7To5IQ=
|
||||
github.com/colega/zeropool v0.0.0-20230505084239-6fb4a4f75381 h1:d5EKgQfRQvO97jnISfR89AiCCCJMwMFoSxUiU0OGCRU=
|
||||
github.com/colega/zeropool v0.0.0-20230505084239-6fb4a4f75381/go.mod h1:OU76gHeRo8xrzGJU3F3I1CqX1ekM8dfJw0+wPeMwnp0=
|
||||
github.com/consensys/gnark-crypto v0.20.1 h1:PXDUBvk8AzhvWowHLWBEAfUQcV1/aZgWIqD6eMpXmDg=
|
||||
github.com/consensys/gnark-crypto v0.20.1/go.mod h1:RBWrSgy+IDbGR69RRV313th3M/aZU1ubk2om+qHuTSc=
|
||||
github.com/cosmos/go-bip39 v1.0.0 h1:pcomnQdrdH22njcAatO0yWojsUnCO3y2tNoV1cb6hHY=
|
||||
@@ -280,8 +288,8 @@ github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlT
|
||||
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
|
||||
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
|
||||
github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4=
|
||||
github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao=
|
||||
github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
|
||||
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
|
||||
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
||||
@@ -305,6 +313,8 @@ github.com/luxfi/api v1.1.1 h1:CXD7m0quPmUm+Qw35TrF+E7b0Fq4qz9gHfrZ5gyrjHU=
|
||||
github.com/luxfi/api v1.1.1/go.mod h1:g6J0iohVqaIj2aO1u/ZJPqjiX2tog0NM3/SBf7wJ4cA=
|
||||
github.com/luxfi/atomic v1.0.0 h1:xUV60MuzRvXngaQ1sM0yVC2v4TRoLlUGkkH7M9PS4yw=
|
||||
github.com/luxfi/atomic v1.0.0/go.mod h1:0G2mTlQ6TXWHICUHrUUPu1/qAiIyR4gSZ2tva9ci/bI=
|
||||
github.com/luxfi/bft v0.1.5 h1:5xVLPkog4e5LTgaVlb9pgxA0EWE6tkrKwHPZVRz+RZw=
|
||||
github.com/luxfi/bft v0.1.5/go.mod h1:5I8Ft8yA69xZlDe3RB0i4MgbqFKLZe65o/sha8JuKvU=
|
||||
github.com/luxfi/cache v1.3.1 h1:grQhi/B5GKypG7avDMeY143QTgFbfEvQICKNIh1Cw6U=
|
||||
github.com/luxfi/cache v1.3.1/go.mod h1:2MokdbeNUy/9O3mdREWkE6BiN7tRvePkXiKkcb+4M7g=
|
||||
github.com/luxfi/chains v1.7.9 h1:sDF/WQn56o/4i1/uCS8i5ENDMuyWi8+iVcnYGAstMDM=
|
||||
@@ -315,8 +325,10 @@ github.com/luxfi/compress v0.1.1 h1:cQjRYQFRrw8HinjW1T8FmKgqdRwrFYDCdecc1t0i+uQ=
|
||||
github.com/luxfi/compress v0.1.1/go.mod h1:d4CkRRmwPzafIp57Jxra3RmAmNwNwU8Oc0QBBRlTqGo=
|
||||
github.com/luxfi/concurrent v0.1.1 h1:LkAmNXybOsAm97nFILqMDBp/YLgleuLmyGomFT7YNxU=
|
||||
github.com/luxfi/concurrent v0.1.1/go.mod h1:GlkfiJtPBpatNxvROf7hkPi4gsnmdHGmqhDnVWFrkZE=
|
||||
github.com/luxfi/consensus v1.36.9 h1:un4iRecrXQqePGEtVNJO4uc76n4GhxOeq+YAaoQB0TE=
|
||||
github.com/luxfi/consensus v1.36.9/go.mod h1:iwlx11D7BRdEaUqyUvueAe2Fit1K1k26avBK2eLYdnc=
|
||||
github.com/luxfi/consensus v1.36.10 h1:LJPvHc2zL7VFhbqfYu11rA50yvxzTK17Z4mRN04HQhU=
|
||||
github.com/luxfi/consensus v1.36.10/go.mod h1:iwlx11D7BRdEaUqyUvueAe2Fit1K1k26avBK2eLYdnc=
|
||||
github.com/luxfi/consensus v1.36.12 h1:MJKq6j9+04Y2NC+NArGQgv5QNcjprYCk+gppCz1PdSg=
|
||||
github.com/luxfi/consensus v1.36.12/go.mod h1:mCtA6k31FVW/H6rsYQ7V8VvS198WsFS8oFo/z3IMI+A=
|
||||
github.com/luxfi/constants v1.6.2 h1:pXHdKIFbfE9qX4xOjq2LxYvagNhhNvspUVEbPcIEKfA=
|
||||
github.com/luxfi/constants v1.6.2/go.mod h1:r0oH8C/+r/XFYBq1AJxt6zWRKKRKgDzrEMop/CCs9rI=
|
||||
github.com/luxfi/container v0.2.1 h1:MTnfKXzS5+oxV5jKZerdOxSA6iMPaQI9/FWGufizzaw=
|
||||
@@ -351,6 +363,8 @@ github.com/luxfi/keychain v1.1.1 h1:dTYEPy6CGVC1sogMci4iJogUvW6VdTmemplQdzRqnAs=
|
||||
github.com/luxfi/keychain v1.1.1/go.mod h1:hAzBcwxGumtoYrM5hfhwdt8wE0p7r2JCd5AxswqfkoY=
|
||||
github.com/luxfi/keys v1.4.1 h1:2Zcoovaz9OLPz7m7VGXfRrGnrlqt0GeUpJclsPBi4EU=
|
||||
github.com/luxfi/keys v1.4.1/go.mod h1:P8EUP5DKrR1SUZBGZjDT3rWcp2P1miUlVh7IBRNBphU=
|
||||
github.com/luxfi/kms v1.12.10 h1:fXgisnBkixFSrqhOFbZVRQkf8cX4q4vS3Gix1qjL3PQ=
|
||||
github.com/luxfi/kms v1.12.10/go.mod h1:CCcWDXIlDT2TwfYAUxedLyzShFRJAlVlp15zQ2L3CrU=
|
||||
github.com/luxfi/lattice/v7 v7.1.4 h1:hQR02M6cHTAV5+joOPi9gb9Gm+z/hKJnhJF4IlciIJs=
|
||||
github.com/luxfi/lattice/v7 v7.1.4/go.mod h1:DmIQFi3mJiehVsR235l1NKYEU0JhU649OX5p7gMEW2c=
|
||||
github.com/luxfi/lens v0.2.1 h1:5Qd0GdjbM+XUVgwDbZ452tKkR7yeE8QnBTHHaH8fJNY=
|
||||
@@ -423,8 +437,10 @@ github.com/luxfi/validators v1.3.1 h1:+/7j0CTXlMKyaSLFM+gd6Fq64/edORJfrD/xGnf7Xc
|
||||
github.com/luxfi/validators v1.3.1/go.mod h1:sIQyUZOvXoJ/9/RCOhHSzjumZIoxiqacNOn5mQWVozU=
|
||||
github.com/luxfi/version v1.0.1 h1:T/1KYWEMmsrNQk7pN7PFPAwh/7XbeX7cFAKLBqI37Sk=
|
||||
github.com/luxfi/version v1.0.1/go.mod h1:Y5fPkQ2DB0XRBCxgSPXp4ISzL1/jptKnmFknShRJCyg=
|
||||
github.com/luxfi/vm v1.3.1 h1:rLCbygaajehVkUoJ5czwhpUAJaC5J5okq3p+j2QSPSo=
|
||||
github.com/luxfi/vm v1.3.1/go.mod h1:uViH3COP8hhCbj42v1MTohkPCDwYQCZanNIOb/StWqY=
|
||||
github.com/luxfi/vm v1.3.1 h1:U2Bv7IDRFv8JtjUARfY53ZnOPRB2iPrPHSMDCQ+T0Js=
|
||||
github.com/luxfi/vm v1.3.1/go.mod h1:6YR/uFV2FoofmogQShv+HM7V8alz5rbrQYjp5w0bNVA=
|
||||
github.com/luxfi/vm v1.3.3 h1:BEBamD9VUcPG8mtL2Ga0Us1lTQ+ubaZqng/PZjNqzPc=
|
||||
github.com/luxfi/vm v1.3.3/go.mod h1:v2XjW0yymxHwy1xPsSz338s0J8nRqOIb6RiDf0wUEgk=
|
||||
github.com/luxfi/warp v1.24.1 h1:9F+z6fy4sxmXp+3LlR9m3TiZ8Dfthh+s5KOeewSJL30=
|
||||
github.com/luxfi/warp v1.24.1/go.mod h1:kRGQDt6EB3oRLYo71aXqnmJuCBVfND3oQ5/2miaLoyg=
|
||||
github.com/luxfi/zap v1.2.6 h1:NBpbm9Gib41Oi/XAkAZKQ3hb+xCafo7JsrUjw+bKiAc=
|
||||
@@ -583,27 +599,33 @@ github.com/tidwall/sjson v1.2.5 h1:kLy8mja+1c9jlljvWTlSazM7cKDRfJuR/bOJhcY5NcY=
|
||||
github.com/tidwall/sjson v1.2.5/go.mod h1:Fvgq9kS/6ociJEDnK0Fk1cpYF4FIW6ZF7LAe+6jwd28=
|
||||
github.com/tinylib/msgp v1.6.4 h1:mOwYbyYDLPj35mkA2BjjYejgJk9BuHxDdvRnb6v2ZcQ=
|
||||
github.com/tinylib/msgp v1.6.4/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA=
|
||||
github.com/tj/assert v0.0.3 h1:Df/BlaZ20mq6kuai7f5z2TvPFiwC3xaWJSDQNiIS3Rk=
|
||||
github.com/tj/assert v0.0.3/go.mod h1:Ne6X72Q+TB1AteidzQncjw9PabbMp4PBMZ1k+vd1Pvk=
|
||||
github.com/tklauser/go-sysconf v0.4.0 h1:7H0uAN+7RkwWRaxhYXDLqa5V3LPrJeV8wmD9dRUgPQU=
|
||||
github.com/tklauser/go-sysconf v0.4.0/go.mod h1:8mTNWyog7H+MpKijp4VmKJAd2bbYQ2zuUwkYRbUArPI=
|
||||
github.com/tklauser/numcpus v0.12.0 h1:NR85qdvHA9pFse3x3weVZ0r0ST8R6l5RHbZrlRaqob4=
|
||||
github.com/tklauser/numcpus v0.12.0/go.mod h1:ABHeXzJnr/qqwguhClkZKT1/8VABcYrsyUiUGobwWJg=
|
||||
github.com/urfave/cli/v2 v2.27.7 h1:bH59vdhbjLv3LAvIu6gd0usJHgoTTPhCFib8qqOwXYU=
|
||||
github.com/urfave/cli/v2 v2.27.7/go.mod h1:CyNAG/xg+iAOg0N4MPGZqVmv2rCoP267496AOXUZjA4=
|
||||
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
|
||||
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
|
||||
github.com/valyala/fasthttp v1.73.0 h1:ocTOORnBWtJ+P8t/6wAjdkchMzdfHmWx2VD/DPbgZ7s=
|
||||
github.com/valyala/fasthttp v1.73.0/go.mod h1:EtXQDHaR+5P18p8wqDRFpUhxr108Ga9mXvVJXHRrN2k=
|
||||
github.com/wlynxg/anet v0.0.5 h1:J3VJGi1gvo0JwZ/P1/Yc/8p63SoW98B5dHkYDmpgvvU=
|
||||
github.com/wlynxg/anet v0.0.5/go.mod h1:eay5PRQr7fIVAMbTbchTnO9gG65Hg/uYGdc7mguHxoA=
|
||||
github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
|
||||
github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
|
||||
github.com/xrash/smetrics v0.0.0-20250705151800-55b8f293f342 h1:FnBeRrxr7OU4VvAzt5X7s6266i6cSVkkFPS0TuXWbIg=
|
||||
github.com/xrash/smetrics v0.0.0-20250705151800-55b8f293f342/go.mod h1:Ohn+xnUBiLI6FVj/9LpzZWtj1/D6lUovWYBkxHVV3aM=
|
||||
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
|
||||
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
|
||||
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
|
||||
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
|
||||
github.com/yuin/goldmark v1.4.1/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
|
||||
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
|
||||
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
|
||||
github.com/zap-proto/http v0.0.0-20260506200741-fd6047874433 h1:7WsCr/pZvWozimdYNffL3B9K6gLr8w0Z7WAi1+eZWtc=
|
||||
github.com/zap-proto/http v0.0.0-20260506200741-fd6047874433/go.mod h1:xySyVTIwjknmVE+p+6rukkX86rFZtXeAu/QY7KXMYqw=
|
||||
github.com/zap-proto/go v1.1.0 h1:DDGhuTBNqkmNax7QV/VvkInJR7hiOcybwUjN3YBt8fg=
|
||||
github.com/zap-proto/go v1.1.0/go.mod h1:914SNGTH6Rv3Yu1MweWJBPEN8FZlo5C39QyhaB0C7Q0=
|
||||
github.com/zap-proto/http v0.3.0 h1:l7DvlngiYqmzNY6fzyRYw2ZIAhF35FqwOe6mvAOqpMg=
|
||||
github.com/zap-proto/http v0.3.0/go.mod h1:UYfGhDDCetgxs65XSev8Lpf65COg5vKQK+cWwZGh4zQ=
|
||||
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
|
||||
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
|
||||
github.com/zeebo/blake3 v0.2.4 h1:KYQPkhpRtcqh0ssGYcKLG1JYvddkEA8QwCM/yBqhaZI=
|
||||
@@ -642,16 +664,16 @@ golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnf
|
||||
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
|
||||
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
|
||||
golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
|
||||
golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
|
||||
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
|
||||
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
|
||||
golang.org/x/exp v0.0.0-20260529124908-c761662dc8c9 h1:4d4PbuBNwaxMXkXI8yiIYjydtMU+04RHeuSxJdgKftM=
|
||||
golang.org/x/exp v0.0.0-20260529124908-c761662dc8c9/go.mod h1:d2fgXJLVs4dYDHUk5lwMIfzRzSrWCfGZb0ZqeLa/Vcw=
|
||||
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
|
||||
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
|
||||
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
|
||||
golang.org/x/mod v0.5.1/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro=
|
||||
golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
|
||||
golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
|
||||
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
|
||||
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
|
||||
golang.org/x/net v0.0.0-20180719180050-a680a1efc54d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||
@@ -666,15 +688,15 @@ golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT
|
||||
golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||
golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
|
||||
golang.org/x/net v0.0.0-20220607020251-c690dde0001d/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
|
||||
golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
|
||||
golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
|
||||
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
|
||||
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
|
||||
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
|
||||
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
|
||||
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
@@ -697,8 +719,8 @@ golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBc
|
||||
golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
|
||||
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
||||
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
@@ -706,8 +728,8 @@ golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
|
||||
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
|
||||
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
|
||||
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
|
||||
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
||||
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
|
||||
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
|
||||
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
@@ -717,8 +739,8 @@ golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roY
|
||||
golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
|
||||
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
|
||||
golang.org/x/tools v0.1.8/go.mod h1:nABZi5QlRsZVlzPpHl034qft6wpY4eDcsTt5AaioBiU=
|
||||
golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8=
|
||||
golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0=
|
||||
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
|
||||
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
|
||||
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
|
||||
@@ -19,6 +19,7 @@ import (
|
||||
"net/http"
|
||||
"os"
|
||||
|
||||
"github.com/valyala/fasthttp/fasthttpadaptor"
|
||||
zaphttp "github.com/zap-proto/http"
|
||||
|
||||
log "github.com/luxfi/log"
|
||||
@@ -53,7 +54,10 @@ func startZapRPCListener(logger log.Logger, handler http.Handler, addr string) *
|
||||
if addr == "" {
|
||||
return nil
|
||||
}
|
||||
srv := &zaphttp.Server{Addr: addr, Handler: handler}
|
||||
// zap-proto/http >= v0.2.0 takes a fasthttp.RequestHandler. Adapt the very
|
||||
// same net/http handler chain the HTTP listener serves, so the two
|
||||
// transports stay behaviourally identical — only the wire encoding differs.
|
||||
srv := &zaphttp.Server{Addr: addr, Handler: fasthttpadaptor.NewFastHTTPHandler(handler)}
|
||||
go func() {
|
||||
logger.Info("ZAP-RPC API listening", log.UserString("addr", addr))
|
||||
if err := srv.ListenAndServe(); err != nil {
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/valyala/fasthttp"
|
||||
zaphttp "github.com/zap-proto/http"
|
||||
|
||||
log "github.com/luxfi/log"
|
||||
@@ -22,8 +23,8 @@ func TestZapRPCListenAddr(t *testing.T) {
|
||||
val string
|
||||
want string
|
||||
}{
|
||||
{set: false, want: ""}, // unset → disabled (default)
|
||||
{set: true, val: "", want: ""}, // empty → disabled
|
||||
{set: false, want: ""}, // unset → disabled (default)
|
||||
{set: true, val: "", want: ""}, // empty → disabled
|
||||
{set: true, val: "off", want: ""},
|
||||
{set: true, val: "OFF", want: ""},
|
||||
{set: true, val: "0", want: ""},
|
||||
@@ -74,18 +75,32 @@ func TestStartZapRPCListener_RoundTrip(t *testing.T) {
|
||||
// Give the goroutine a beat to bind.
|
||||
time.Sleep(150 * time.Millisecond)
|
||||
|
||||
client := &http.Client{Transport: zaphttp.NewTransport(addr)}
|
||||
resp, err := client.Post("http://"+addr+"/v1/bc/C/rpc", "application/json", nil)
|
||||
if err != nil {
|
||||
// zap-proto/http >= v0.2.0 speaks fasthttp on both ends: Transport.Do takes
|
||||
// a fasthttp request/response pair rather than implementing
|
||||
// http.RoundTripper. The round trip being asserted is unchanged — a real
|
||||
// ZAP request over the wire must reach the net/http handler the listener
|
||||
// was given, through the fasthttpadaptor bridge in startZapRPCListener.
|
||||
// v0.3.0 replaced NewTransport with Dial, mirroring net.Dial: the network is
|
||||
// a VALUE ("tcp", "unix") rather than a family of constructors.
|
||||
transport := zaphttp.Dial("tcp", addr)
|
||||
defer transport.CloseIdleConnections()
|
||||
|
||||
req, resp := fasthttp.AcquireRequest(), fasthttp.AcquireResponse()
|
||||
defer fasthttp.ReleaseRequest(req)
|
||||
defer fasthttp.ReleaseResponse(resp)
|
||||
|
||||
req.SetRequestURI("http://" + addr + "/v1/bc/C/rpc")
|
||||
req.Header.SetMethod(http.MethodPost)
|
||||
req.Header.SetContentType("application/json")
|
||||
|
||||
if err := transport.Do(req, resp); err != nil {
|
||||
t.Fatalf("ZAP round-trip POST failed: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
got, _ := io.ReadAll(resp.Body)
|
||||
if string(got) != body {
|
||||
if got := string(resp.Body()); got != body {
|
||||
t.Fatalf("ZAP round-trip body = %q, want %q", got, body)
|
||||
}
|
||||
if ct := resp.Header.Get("Content-Type"); ct != "application/json" {
|
||||
if ct := string(resp.Header.ContentType()); ct != "application/json" {
|
||||
t.Fatalf("ZAP round-trip Content-Type = %q, want application/json", ct)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
// Copyright (C) 2019-2026, Lux Industries Inc. All rights reserved.
|
||||
// See the file LICENSE for licensing terms.
|
||||
|
||||
// logger_level_test.go — admin.setLoggerLevel / admin.getLoggerLevel must actually
|
||||
// move a live logger's level.
|
||||
//
|
||||
// They did not. SetLoggerLevel computed the logger names and threw them away
|
||||
// (`loggerNames := a.getLoggerNames(...); _ = loggerNames`) and getLogLevels returned
|
||||
// an empty map unconditionally, so BOTH endpoints answered 200 OK having done nothing.
|
||||
// That is why the 2026-07-28 devnet/testnet build-loop diagnosis had to be run off boot
|
||||
// logs: raising a running node's log level was impossible.
|
||||
//
|
||||
// The tests drive the REAL log.Factory (the same one the node builds), not a double,
|
||||
// so a passing SetLoggerLevel means the level the logger actually filters on moved.
|
||||
package admin
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
apiadmin "github.com/luxfi/api/admin"
|
||||
"github.com/luxfi/log"
|
||||
)
|
||||
|
||||
// newLevelTestService returns an admin service over a real log.Factory that already
|
||||
// holds one registered logger — the shape the node runs in.
|
||||
func newLevelTestService(t *testing.T, loggerName string) *Service {
|
||||
t.Helper()
|
||||
|
||||
factory := log.NewFactory()
|
||||
t.Cleanup(factory.Close)
|
||||
|
||||
_, err := factory.Make(loggerName)
|
||||
require.NoError(t, err)
|
||||
|
||||
return &Service{Config: Config{Log: log.Noop(), LogFactory: factory}}
|
||||
}
|
||||
|
||||
// TestSetLoggerLevel_MovesTheLevelAndGetReportsIt is the regression: set, then read
|
||||
// back through the API. Before the fix the set was discarded and the get returned an
|
||||
// empty map, so both halves silently lied.
|
||||
func TestSetLoggerLevel_MovesTheLevelAndGetReportsIt(t *testing.T) {
|
||||
require := require.New(t)
|
||||
ctx := context.Background()
|
||||
|
||||
svc := newLevelTestService(t, "C")
|
||||
|
||||
_, err := svc.SetLoggerLevel(ctx, &apiadmin.SetLoggerLevelArgs{
|
||||
LoggerName: "C",
|
||||
LogLevel: "debug",
|
||||
DisplayLevel: "error",
|
||||
})
|
||||
require.NoError(err)
|
||||
|
||||
reply, err := svc.GetLoggerLevel(ctx, &apiadmin.GetLoggerLevelArgs{LoggerName: "C"})
|
||||
require.NoError(err)
|
||||
require.Equal(
|
||||
map[string]apiadmin.LogAndDisplayLevels{"C": {
|
||||
LogLevel: log.DebugLevel.String(),
|
||||
DisplayLevel: log.ErrorLevel.String(),
|
||||
}},
|
||||
reply.LoggerLevels,
|
||||
"setLoggerLevel must move the live logger's level and getLoggerLevel must report it",
|
||||
)
|
||||
|
||||
// The factory is the single source of truth — assert against it directly too, so a
|
||||
// getLoggerLevel that merely echoed the request back could not pass this test.
|
||||
logLevel, err := svc.LogFactory.GetLogLevel("C")
|
||||
require.NoError(err)
|
||||
require.Equal(log.DebugLevel, logLevel)
|
||||
}
|
||||
|
||||
// TestSetLoggerLevel_OneLevelAtATime pins that omitting a level leaves it alone rather
|
||||
// than resetting it to the zero Level.
|
||||
func TestSetLoggerLevel_OneLevelAtATime(t *testing.T) {
|
||||
require := require.New(t)
|
||||
ctx := context.Background()
|
||||
|
||||
svc := newLevelTestService(t, "node")
|
||||
|
||||
_, err := svc.SetLoggerLevel(ctx, &apiadmin.SetLoggerLevelArgs{
|
||||
LoggerName: "node", LogLevel: "trace", DisplayLevel: "warn",
|
||||
})
|
||||
require.NoError(err)
|
||||
|
||||
// Only displayLevel this time — logLevel must survive.
|
||||
_, err = svc.SetLoggerLevel(ctx, &apiadmin.SetLoggerLevelArgs{
|
||||
LoggerName: "node", DisplayLevel: "fatal",
|
||||
})
|
||||
require.NoError(err)
|
||||
|
||||
reply, err := svc.GetLoggerLevel(ctx, &apiadmin.GetLoggerLevelArgs{LoggerName: "node"})
|
||||
require.NoError(err)
|
||||
require.Equal(log.TraceLevel.String(), reply.LoggerLevels["node"].LogLevel)
|
||||
require.Equal(log.FatalLevel.String(), reply.LoggerLevels["node"].DisplayLevel)
|
||||
}
|
||||
|
||||
// TestLoggerLevel_RejectsUnservableAndInvalidArgs — every refusal is explicit. log.Factory
|
||||
// addresses loggers BY NAME and exposes no enumeration, so an empty name cannot be served;
|
||||
// returning 200 OK for it is the bug, not the contract.
|
||||
func TestLoggerLevel_RejectsUnservableAndInvalidArgs(t *testing.T) {
|
||||
require := require.New(t)
|
||||
ctx := context.Background()
|
||||
|
||||
svc := newLevelTestService(t, "C")
|
||||
|
||||
_, err := svc.SetLoggerLevel(ctx, &apiadmin.SetLoggerLevelArgs{LogLevel: "debug"})
|
||||
require.ErrorIs(err, errNoLoggerName)
|
||||
|
||||
_, err = svc.GetLoggerLevel(ctx, &apiadmin.GetLoggerLevelArgs{})
|
||||
require.ErrorIs(err, errNoLoggerName)
|
||||
|
||||
_, err = svc.SetLoggerLevel(ctx, &apiadmin.SetLoggerLevelArgs{LoggerName: "C"})
|
||||
require.ErrorIs(err, errNoLogLevel)
|
||||
|
||||
// An unparseable level must be refused BEFORE anything is mutated.
|
||||
before, err := svc.LogFactory.GetLogLevel("C")
|
||||
require.NoError(err)
|
||||
_, err = svc.SetLoggerLevel(ctx, &apiadmin.SetLoggerLevelArgs{LoggerName: "C", LogLevel: "loud"})
|
||||
require.Error(err)
|
||||
after, err := svc.LogFactory.GetLogLevel("C")
|
||||
require.NoError(err)
|
||||
require.Equal(before, after, "a rejected level must leave the logger untouched")
|
||||
}
|
||||
@@ -40,6 +40,7 @@ const (
|
||||
var (
|
||||
errAliasTooLong = errors.New("alias length is too long")
|
||||
errNoLogLevel = errors.New("need to specify either displayLevel or logLevel")
|
||||
errNoLoggerName = errors.New("need to specify loggerName: loggers are addressed by name and cannot be enumerated")
|
||||
)
|
||||
|
||||
// ChainTracker is the interface for tracking chains at runtime.
|
||||
@@ -209,11 +210,39 @@ func (a *Service) SetLoggerLevel(ctx context.Context, args *apiadmin.SetLoggerLe
|
||||
return nil, errNoLogLevel
|
||||
}
|
||||
|
||||
// Parse before mutating: a rejected level must leave every logger untouched.
|
||||
var logLevel, displayLevel log.Level
|
||||
if args.LogLevel != "" {
|
||||
var err error
|
||||
if logLevel, err = log.ToLevel(args.LogLevel); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
if args.DisplayLevel != "" {
|
||||
var err error
|
||||
if displayLevel, err = log.ToLevel(args.DisplayLevel); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
loggerNames, err := a.getLoggerNames(args.LoggerName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
a.lock.Lock()
|
||||
defer a.lock.Unlock()
|
||||
|
||||
loggerNames := a.getLoggerNames(args.LoggerName)
|
||||
_ = loggerNames
|
||||
for _, name := range loggerNames {
|
||||
// Only the levels the caller supplied — an omitted level keeps its value
|
||||
// instead of being reset to the zero Level.
|
||||
if args.LogLevel != "" {
|
||||
a.LogFactory.SetLogLevel(name, logLevel)
|
||||
}
|
||||
if args.DisplayLevel != "" {
|
||||
a.LogFactory.SetDisplayLevel(name, displayLevel)
|
||||
}
|
||||
}
|
||||
return &apiadmin.EmptyReply{}, nil
|
||||
}
|
||||
|
||||
@@ -225,10 +254,14 @@ func (a *Service) GetLoggerLevel(ctx context.Context, args *apiadmin.GetLoggerLe
|
||||
log.String("loggerName", args.LoggerName),
|
||||
)
|
||||
|
||||
loggerNames, err := a.getLoggerNames(args.LoggerName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
a.lock.RLock()
|
||||
defer a.lock.RUnlock()
|
||||
|
||||
loggerNames := a.getLoggerNames(args.LoggerName)
|
||||
loggerLevels, err := a.getLogLevels(loggerNames)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -401,15 +434,35 @@ func (a *Service) GetTrackedChains(ctx context.Context) (*apiadmin.GetTrackedCha
|
||||
return &apiadmin.GetTrackedChainsReply{TrackedChains: trackedChains}, nil
|
||||
}
|
||||
|
||||
func (a *Service) getLoggerNames(loggerName string) []string {
|
||||
if len(loggerName) == 0 {
|
||||
return []string{}
|
||||
// getLoggerNames resolves the loggerName argument to the loggers to act on — the one
|
||||
// place either logger-level endpoint decides what it is addressing.
|
||||
//
|
||||
// log.Factory addresses loggers BY NAME and exposes no enumeration, so the "every
|
||||
// logger" form (an empty name) cannot be served. Refuse it explicitly: answering 200 OK
|
||||
// while doing nothing is what made these endpoints unusable.
|
||||
func (a *Service) getLoggerNames(loggerName string) ([]string, error) {
|
||||
if loggerName == "" {
|
||||
return nil, errNoLoggerName
|
||||
}
|
||||
return []string{loggerName}
|
||||
return []string{loggerName}, nil
|
||||
}
|
||||
|
||||
func (a *Service) getLogLevels(loggerNames []string) (map[string]apiadmin.LogAndDisplayLevels, error) {
|
||||
loggerLevels := make(map[string]apiadmin.LogAndDisplayLevels)
|
||||
loggerLevels := make(map[string]apiadmin.LogAndDisplayLevels, len(loggerNames))
|
||||
for _, name := range loggerNames {
|
||||
logLevel, err := a.LogFactory.GetLogLevel(name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
displayLevel, err := a.LogFactory.GetDisplayLevel(name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
loggerLevels[name] = apiadmin.LogAndDisplayLevels{
|
||||
LogLevel: logLevel.String(),
|
||||
DisplayLevel: displayLevel.String(),
|
||||
}
|
||||
}
|
||||
return loggerLevels, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@ github.com/Ladicle/tabwriter v1.0.0 h1:DZQqPvMumBDwVNElso13afjYLNp0Z7pHqHnu0r4t9
|
||||
github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
|
||||
github.com/OpenPeeDeeP/depguard/v2 v2.2.1 h1:vckeWVESWp6Qog7UZSARNqfu/cZqvki8zsuj3piCMx4=
|
||||
github.com/StephenButtolph/canoto v0.17.3 h1:lvsnYD4b96vD1knnmp1xCmZqfYpY/jSeRozGdOfdvGI=
|
||||
github.com/StephenButtolph/canoto v0.17.3/go.mod h1:IcnAHC6nJUfQFVR9y60ko2ecUqqHHSB6UwI9NnBFZnE=
|
||||
github.com/alecthomas/assert/v2 v2.11.0 h1:2Q9r3ki8+JYXvGsDyBXwH3LcJ+WK5D0gc5E8vS6K3D0=
|
||||
github.com/alecthomas/chroma/v2 v2.17.2 h1:Rm81SCZ2mPoH+Q8ZCc/9YvzPUN/E7HgPiPJD8SLV6GI=
|
||||
github.com/alecthomas/go-check-sumtype v0.3.1 h1:u9aUvbGINJxLVXiFvHUlPEaD7VDULsrxJb4Aq31NLkU=
|
||||
@@ -56,6 +57,7 @@ github.com/dominikbraun/graph v0.23.0 h1:TdZB4pPqCLFxYhdyMFb1TBdFxp8XLcJfTTBQucV
|
||||
github.com/ettle/strcase v0.2.0 h1:fGNiVF21fHXpX1niBgk0aROov1LagYsOwV/xqKDKR/Q=
|
||||
github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
|
||||
github.com/fatih/structtag v1.2.0 h1:/OdNE99OxoI/PqaW/SuSK9uxxT3f/tcSZgon/ssNSx4=
|
||||
github.com/fatih/structtag v1.2.0/go.mod h1:mBJUNpUnHmRKrKlQQlmCrh5PuhftFbNv8Ys4/aAZl94=
|
||||
github.com/firefart/nonamedreturns v1.0.6 h1:vmiBcKV/3EqKY3ZiPxCINmpS431OcE1S47AQUwhrg8E=
|
||||
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
|
||||
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
|
||||
@@ -247,6 +249,7 @@ go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
|
||||
golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93 h1:fQsdNF2N+/YewlRZiricy4P1iimyPKZ/xwniHj8Q2a0=
|
||||
golang.org/x/exp/typeparams v0.0.0-20250210185358-939b2ce775ac h1:TSSpLIG4v+p0rPv1pNOQtl1I8knsO4S9trOxNMOLVP4=
|
||||
golang.org/x/mod v0.31.0 h1:HaW9xtz0+kOcWKwli0ZXy79Ix+UW/vOfmWI5QVd2tgI=
|
||||
golang.org/x/mod v0.31.0/go.mod h1:43JraMp9cGx1Rx3AqioxrbrhNsLl2l/iNAvuBkrezpg=
|
||||
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
|
||||
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
|
||||
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
|
||||
|
||||
+1
-1
@@ -1 +1 @@
|
||||
1.32.11
|
||||
1.36.36
|
||||
|
||||
@@ -92,7 +92,9 @@
|
||||
"v1.36.27",
|
||||
"v1.36.28",
|
||||
"v1.36.30",
|
||||
"v1.36.31"
|
||||
"v1.36.31",
|
||||
"v1.36.32",
|
||||
"v1.36.33"
|
||||
],
|
||||
"41": [
|
||||
"v1.13.2"
|
||||
|
||||
@@ -77,7 +77,7 @@ var (
|
||||
const (
|
||||
defaultMajor = 1
|
||||
defaultMinor = 36
|
||||
defaultPatch = 31
|
||||
defaultPatch = 35
|
||||
)
|
||||
|
||||
func init() {
|
||||
|
||||
@@ -301,6 +301,12 @@ func (p *postForkCommonComponents) buildChild(
|
||||
_ = pChainHeight
|
||||
contextPChainHeight := epoch.PChainHeight
|
||||
|
||||
// The inner VM builds on ITS OWN head, and Verify above requires the child's inner
|
||||
// parent to be exactly p.innerBlk. Anchor the two together before delegating.
|
||||
if err := p.vm.anchorInnerBuildParent(ctx, parentID, p.innerBlk.ID()); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var innerBlock chain.Block
|
||||
if p.vm.blockBuilderVM != nil {
|
||||
builtBlock, err := p.vm.blockBuilderVM.BuildBlockWithRuntime(ctx, &runtime.Runtime{
|
||||
|
||||
@@ -0,0 +1,321 @@
|
||||
// Copyright (C) 2019-2026, Lux Industries Inc. All rights reserved.
|
||||
// See the file LICENSE for licensing terms.
|
||||
|
||||
// build_inner_parent_test.go — the deterministic repro of the SELF-REJECTION LOOP
|
||||
// measured on devnet 96367 and testnet 96368 on 2026-07-28:
|
||||
//
|
||||
// info built block blkID=2srokbs… height=1047 innerBlkID=1DMGHAoo… pChainHeight=0
|
||||
// error built block failed verification — dropping
|
||||
// error="inner parentID didn't match expected parent" height=1047 parentID=2V6mi4tX…
|
||||
//
|
||||
// 83–456 drops/min per node, forever, with the externally visible tip frozen two heights
|
||||
// BELOW what the builder kept proposing. Every node rejected the block it had just built.
|
||||
//
|
||||
// The invariant being violated is postForkCommonComponents.Verify's third check
|
||||
// (block.go, errInnerParentMismatch): a child's inner parent MUST be the inner block
|
||||
// wrapped by the outer parent it extends. buildChild did not establish it — it asked the
|
||||
// inner VM to build, and the inner VM builds on ITS OWN head. These tests model the inner
|
||||
// VM with the luxfi/evm head semantics that make the two pointers diverge, and assert the
|
||||
// invariant on the block buildChild actually returns.
|
||||
package proposervm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/luxfi/database/memdb"
|
||||
"github.com/luxfi/database/versiondb"
|
||||
"github.com/luxfi/ids"
|
||||
"github.com/luxfi/log"
|
||||
validatorstest "github.com/luxfi/validators/validatorstest"
|
||||
vmchain "github.com/luxfi/vm/chain"
|
||||
"github.com/luxfi/vm/chain/blocktest"
|
||||
|
||||
"github.com/luxfi/node/vms/proposervm/block"
|
||||
"github.com/luxfi/node/vms/proposervm/proposer"
|
||||
"github.com/luxfi/node/vms/proposervm/state"
|
||||
)
|
||||
|
||||
// evmLikeInner models the inner execution VM's HEAD, which is the only part of the inner
|
||||
// VM this defect involves. Three behaviours are copied from luxfi/evm:
|
||||
//
|
||||
// - BuildBlock builds on the head (miner/worker.go: parent = chain.CurrentBlock())
|
||||
// - verifyGossiped advances the head (core/blockchain.go writeBlockAndSetHead →
|
||||
// newTip(block) → writeCanonicalBlockWithLogs → writeHeadBlock)
|
||||
// - SetPreference reorgs the head, and is a no-op when it already matches
|
||||
// (core/blockchain.go setPreference: early return on current.Hash() == block.Hash(),
|
||||
// otherwise writeKnownBlock)
|
||||
//
|
||||
// Nothing else is stubbed: the test drives the REAL postForkBlock.buildChild.
|
||||
type evmLikeInner struct {
|
||||
vmchain.ChainVM
|
||||
blocks map[ids.ID]*blocktest.Block // every block inserted into the inner chain
|
||||
head ids.ID
|
||||
setPrefCalls int
|
||||
reorgs int
|
||||
}
|
||||
|
||||
func newEVMLikeInner(genesis *blocktest.Block) *evmLikeInner {
|
||||
return &evmLikeInner{
|
||||
blocks: map[ids.ID]*blocktest.Block{genesis.IDV: genesis},
|
||||
head: genesis.IDV,
|
||||
}
|
||||
}
|
||||
|
||||
func (c *evmLikeInner) insert(b *blocktest.Block) { c.blocks[b.IDV] = b }
|
||||
|
||||
// child mints a block extending [parent] without touching the head.
|
||||
func (c *evmLikeInner) child(parent *blocktest.Block) *blocktest.Block {
|
||||
id := ids.GenerateTestID()
|
||||
b := &blocktest.Block{
|
||||
IDV: id,
|
||||
ParentV: parent.IDV,
|
||||
HeightV: parent.HeightV + 1,
|
||||
BytesV: id[:],
|
||||
TimestampV: parent.TimestampV.Add(time.Second),
|
||||
}
|
||||
c.insert(b)
|
||||
return b
|
||||
}
|
||||
|
||||
// verifyGossiped is the drift mechanism: verifying a peer's block whose parent is the
|
||||
// current head OPTIMISTICALLY makes it the head — no accept, no proposervm involvement.
|
||||
func (c *evmLikeInner) verifyGossiped(b *blocktest.Block) {
|
||||
c.insert(b)
|
||||
if b.ParentV == c.head { // core/blockchain.go newTip
|
||||
c.head = b.IDV
|
||||
}
|
||||
}
|
||||
|
||||
func (c *evmLikeInner) BuildBlock(context.Context) (vmchain.Block, error) {
|
||||
return c.child(c.blocks[c.head]), nil
|
||||
}
|
||||
|
||||
func (c *evmLikeInner) SetPreference(_ context.Context, id ids.ID) error {
|
||||
c.setPrefCalls++
|
||||
blk, ok := c.blocks[id]
|
||||
if !ok {
|
||||
return fmt.Errorf("evm-like inner VM: block %s not in chain", id)
|
||||
}
|
||||
if c.head == blk.IDV { // setPreference early return — already the head
|
||||
return nil
|
||||
}
|
||||
c.head = blk.IDV
|
||||
c.reorgs++
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *evmLikeInner) GetBlock(_ context.Context, id ids.ID) (vmchain.Block, error) {
|
||||
blk, ok := c.blocks[id]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("evm-like inner VM: block %s not found", id)
|
||||
}
|
||||
return blk, nil
|
||||
}
|
||||
|
||||
// anyoneCanProposeWindower makes every slot open, so buildChild takes the UNSIGNED path
|
||||
// and needs no staking key. The proposer election is orthogonal to the parent invariant.
|
||||
type anyoneCanProposeWindower struct{}
|
||||
|
||||
func (anyoneCanProposeWindower) Proposers(context.Context, uint64, uint64, int) ([]ids.NodeID, error) {
|
||||
return nil, proposer.ErrAnyoneCanPropose
|
||||
}
|
||||
|
||||
func (anyoneCanProposeWindower) Delay(context.Context, uint64, uint64, ids.NodeID, int) (time.Duration, error) {
|
||||
return 0, proposer.ErrAnyoneCanPropose
|
||||
}
|
||||
|
||||
func (anyoneCanProposeWindower) ExpectedProposer(context.Context, uint64, uint64, uint64) (ids.NodeID, error) {
|
||||
return ids.EmptyNodeID, proposer.ErrAnyoneCanPropose
|
||||
}
|
||||
|
||||
func (anyoneCanProposeWindower) MinDelayForProposer(context.Context, uint64, uint64, ids.NodeID, uint64) (time.Duration, error) {
|
||||
return 0, proposer.ErrAnyoneCanPropose
|
||||
}
|
||||
|
||||
// newBuildParentTestVM returns a proposervm whose outer PREFERRED envelope wraps
|
||||
// innerParent, plus the inner chain, at the moment before a build attempt.
|
||||
func newBuildParentTestVM(t *testing.T, innerParent *blocktest.Block, inner *evmLikeInner) (*VM, *postForkBlock) {
|
||||
t.Helper()
|
||||
|
||||
vm := &VM{
|
||||
verifiedBlocks: map[ids.ID]PostForkBlock{},
|
||||
logger: log.Noop(),
|
||||
}
|
||||
vm.State = state.New(versiondb.New(memdb.New()))
|
||||
vm.ChainVM = inner
|
||||
vm.Windower = anyoneCanProposeWindower{}
|
||||
vm.validatorState = &validatorstest.State{
|
||||
GetCurrentHeightF: func(context.Context) (uint64, error) { return 0, nil },
|
||||
}
|
||||
// Deterministic clock, one second after the parent envelope: slot 0, so the child
|
||||
// timestamp is not window-snapped and the parent check is the only thing under test.
|
||||
parentTimestamp := innerParent.TimestampV
|
||||
vm.Clock.Set(parentTimestamp.Add(time.Second))
|
||||
|
||||
statelessParent, err := block.BuildUnsigned(
|
||||
ids.GenerateTestID(), // outer grandparent
|
||||
parentTimestamp,
|
||||
0, // pChainHeight — 0 fleet-wide, mainnet included; orthogonal to this defect
|
||||
block.Epoch{},
|
||||
innerParent.BytesV,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
return vm, &postForkBlock{
|
||||
SignedBlock: statelessParent,
|
||||
postForkCommonComponents: postForkCommonComponents{vm: vm, innerBlk: innerParent},
|
||||
}
|
||||
}
|
||||
|
||||
// TestBuildChild_InnerHeadDriftedByGossipVerify_ChildExtendsOuterParentsInner is the
|
||||
// repro. The node's outer preferred wraps inner height 1045 (the accepted tip); it then
|
||||
// VERIFIES a gossiped inner 1046, which optimistically becomes the EVM head. The next
|
||||
// build must still produce 1046-off-1045 — the block its own Verify will accept.
|
||||
//
|
||||
// BEFORE THE FIX buildChild delegated straight to the inner VM, which built off the
|
||||
// drifted head: a child at height 1047 whose inner parent is 1046, while the outer parent
|
||||
// wraps 1045 ⇒ errInnerParentMismatch on the node's own block, dropped, repeat forever.
|
||||
// That is the live devnet signature exactly: built 1047 against an accepted tip of 1045.
|
||||
func TestBuildChild_InnerHeadDriftedByGossipVerify_ChildExtendsOuterParentsInner(t *testing.T) {
|
||||
require := require.New(t)
|
||||
ctx := context.Background()
|
||||
|
||||
genesis := &blocktest.Block{
|
||||
IDV: ids.GenerateTestID(),
|
||||
HeightV: 1044,
|
||||
BytesV: []byte("1044"),
|
||||
TimestampV: time.Unix(1_785_216_000, 0),
|
||||
}
|
||||
inner := newEVMLikeInner(genesis)
|
||||
|
||||
// The accepted inner tip, wrapped by this node's outer preferred envelope.
|
||||
acceptedTip := inner.child(genesis) // height 1045
|
||||
inner.head = acceptedTip.IDV
|
||||
|
||||
_, outerParent := newBuildParentTestVM(t, acceptedTip, inner)
|
||||
|
||||
// DRIFT: a gossiped inner block at 1046 verifies and optimistically takes the head.
|
||||
gossiped := inner.child(acceptedTip) // height 1046
|
||||
inner.verifyGossiped(gossiped)
|
||||
require.Equal(gossiped.IDV, inner.head, "precondition: the EVM head drifted off the accepted tip")
|
||||
|
||||
built, err := outerParent.buildChild(ctx)
|
||||
require.NoError(err)
|
||||
|
||||
child, ok := built.(*postForkBlock)
|
||||
require.True(ok)
|
||||
|
||||
// THE INVARIANT — verbatim the condition postForkCommonComponents.Verify enforces
|
||||
// (expectedInnerParentID := p.innerBlk.ID(); innerParentID := child.innerBlk.Parent()).
|
||||
require.Equal(acceptedTip.ID(), child.innerBlk.Parent(),
|
||||
"the built child's inner parent must be the outer parent's inner block, or the node "+
|
||||
"rejects its own block with errInnerParentMismatch and drops it forever")
|
||||
require.Equal(acceptedTip.HeightV+1, child.Height(),
|
||||
"the child must be built one above the outer parent's inner block, not above the drifted head")
|
||||
|
||||
// And the head was reorged back — the drift is repaired, not merely tolerated.
|
||||
require.Equal(acceptedTip.IDV, inner.head)
|
||||
require.Equal(1, inner.reorgs)
|
||||
}
|
||||
|
||||
// TestBuildChild_HealthyHead_NoReorg pins the no-behaviour-change half: when the inner
|
||||
// head already IS the outer parent's inner block (every healthy node, every block), the
|
||||
// re-anchor is a no-op inside the inner VM — no reorg, no head movement.
|
||||
func TestBuildChild_HealthyHead_NoReorg(t *testing.T) {
|
||||
require := require.New(t)
|
||||
ctx := context.Background()
|
||||
|
||||
genesis := &blocktest.Block{
|
||||
IDV: ids.GenerateTestID(),
|
||||
HeightV: 100,
|
||||
BytesV: []byte("100"),
|
||||
TimestampV: time.Unix(1_785_216_000, 0),
|
||||
}
|
||||
inner := newEVMLikeInner(genesis)
|
||||
acceptedTip := inner.child(genesis)
|
||||
inner.head = acceptedTip.IDV
|
||||
|
||||
_, outerParent := newBuildParentTestVM(t, acceptedTip, inner)
|
||||
|
||||
built, err := outerParent.buildChild(ctx)
|
||||
require.NoError(err)
|
||||
|
||||
require.Equal(acceptedTip.ID(), built.(*postForkBlock).innerBlk.Parent())
|
||||
require.Equal(acceptedTip.IDV, inner.head, "head must not move on a healthy build")
|
||||
require.Zero(inner.reorgs, "no reorg on a healthy node — the inner SetPreference early-returns")
|
||||
}
|
||||
|
||||
// TestBuildChild_PreForkTransition_ChildExtendsThisBlock covers the OTHER build
|
||||
// delegation, preForkBlock.buildChild, which produces the pre-fork→post-fork transition
|
||||
// block and is verified by preForkBlock.verifyPostForkChild's identical inner-parent check
|
||||
// ("Make sure [b] is the parent of [child]'s inner block", pre_fork_block.go).
|
||||
//
|
||||
// At the fork height every validator may emit its own unsigned transition candidate, so a
|
||||
// gossiped sibling verifying here is the norm, not an edge case — and it drifts the inner
|
||||
// head exactly as on a live chain. Without the anchor the transition wedges at chain
|
||||
// start, which is every fresh net's first two blocks.
|
||||
func TestBuildChild_PreForkTransition_ChildExtendsThisBlock(t *testing.T) {
|
||||
require := require.New(t)
|
||||
ctx := context.Background()
|
||||
|
||||
genesis := &blocktest.Block{
|
||||
IDV: ids.GenerateTestID(),
|
||||
HeightV: 0,
|
||||
BytesV: []byte("genesis"),
|
||||
TimestampV: time.Unix(1_785_216_000, 0),
|
||||
}
|
||||
inner := newEVMLikeInner(genesis)
|
||||
|
||||
vm, _ := newBuildParentTestVM(t, genesis, inner)
|
||||
preFork := &preForkBlock{Block: genesis, vm: vm}
|
||||
|
||||
// A peer's competing transition candidate verifies and takes the inner head.
|
||||
sibling := inner.child(genesis)
|
||||
inner.verifyGossiped(sibling)
|
||||
require.Equal(sibling.IDV, inner.head)
|
||||
|
||||
built, err := preFork.buildChild(ctx)
|
||||
require.NoError(err)
|
||||
|
||||
require.Equal(genesis.ID(), built.(*postForkBlock).innerBlk.Parent(),
|
||||
"the transition block's inner parent must be the pre-fork block itself")
|
||||
require.Equal(genesis.HeightV+1, built.Height())
|
||||
require.Equal(genesis.IDV, inner.head)
|
||||
}
|
||||
|
||||
// TestBuildChild_InnerParentNotInInnerChain_FailsInsteadOfBuildingADoomedBlock covers the
|
||||
// error case: if the inner VM cannot anchor on the outer parent's inner block, then the
|
||||
// head is provably NOT that block, so any child built would fail this node's own Verify.
|
||||
// Returning the error is strictly better than emitting a block we will drop.
|
||||
func TestBuildChild_InnerParentNotInInnerChain_FailsInsteadOfBuildingADoomedBlock(t *testing.T) {
|
||||
require := require.New(t)
|
||||
ctx := context.Background()
|
||||
|
||||
genesis := &blocktest.Block{
|
||||
IDV: ids.GenerateTestID(),
|
||||
HeightV: 100,
|
||||
BytesV: []byte("100"),
|
||||
TimestampV: time.Unix(1_785_216_000, 0),
|
||||
}
|
||||
inner := newEVMLikeInner(genesis)
|
||||
|
||||
// An inner parent the inner VM does not hold (never inserted).
|
||||
orphanID := ids.GenerateTestID()
|
||||
orphan := &blocktest.Block{
|
||||
IDV: orphanID,
|
||||
ParentV: genesis.IDV,
|
||||
HeightV: 101,
|
||||
BytesV: orphanID[:],
|
||||
TimestampV: genesis.TimestampV.Add(time.Second),
|
||||
}
|
||||
|
||||
_, outerParent := newBuildParentTestVM(t, orphan, inner)
|
||||
|
||||
_, err := outerParent.buildChild(ctx)
|
||||
require.ErrorContains(err, "failed to anchor inner build parent")
|
||||
}
|
||||
@@ -0,0 +1,302 @@
|
||||
// Copyright (C) 2019-2026, Lux Industries Inc. All rights reserved.
|
||||
// See the file LICENSE for licensing terms.
|
||||
|
||||
// height_backfill.go — RECOVERY for a proposervm finality index that sits BELOW
|
||||
// the inner VM's accepted tip.
|
||||
//
|
||||
// THE INVARIANT. Every post-fork accept commits the outer envelope, its height
|
||||
// index entry and the last-accepted pointer in ONE versiondb batch BEFORE the
|
||||
// inner block is accepted (vm.acceptPostForkBlock → postForkBlock.Accept), so the
|
||||
// proposervm index can only ever be AHEAD of the inner VM, never behind.
|
||||
//
|
||||
// HOW IT BREAKS ANYWAY. The proposervm has one accept path that advances the inner
|
||||
// VM while leaving the outer index untouched: preForkBlock.Accept, whose
|
||||
// acceptOuterBlk() is a no-op by construction. Post-fork that path was reachable
|
||||
// because BOTH of the proposervm's block constructors fall back to it silently —
|
||||
// ParseBlock() falls back to parsePreForkBlock when the outer envelope does not
|
||||
// parse, and getBlock() falls back to getPreForkBlock when the id is not an outer
|
||||
// id (which is exactly what the CANONICAL id recorded by the consensus ledger is:
|
||||
// postForkCommonComponents.CanonicalID() == innerBlk.ID()). Either fallback hands
|
||||
// the engine a preForkBlock wrapping a post-fork inner block; accepting it moves
|
||||
// the inner VM and NOT the index. Nothing complained, because nothing asserted the
|
||||
// invariant at the moment it was violated — it was only ever checked at the NEXT
|
||||
// boot, by which point the node could not start. That is why the lag looked like
|
||||
// a mysterious "persistence" problem: the writes were never issued at all.
|
||||
//
|
||||
// The prevention half lives at the two fallbacks + preForkBlock.acceptOuterBlk
|
||||
// (see pre_fork_block.go and vm.getPreForkBlock): post-fork, a pre-fork block is
|
||||
// never constructed and never accepted.
|
||||
//
|
||||
// THE RECOVERY half lives here, and it must work for nodes that are ALREADY
|
||||
// damaged. It is two escalating steps, both OUTER-ONLY — they never re-execute,
|
||||
// re-verify or re-accept an inner block, so a node whose EVM is already at the tip
|
||||
// is repaired without touching the EVM:
|
||||
//
|
||||
// 1. rebuildOuterIndexFromStore: re-derive the height index and last-accepted
|
||||
// pointer from the outer envelopes ALREADY in this node's block store, binding
|
||||
// each candidate to the inner block this node itself accepted at that height.
|
||||
// Fully local, no network, no operator. This is what turns "refuses to start"
|
||||
// into "starts, repairs itself, and keeps its finality pointer".
|
||||
//
|
||||
// 2. If step 1 cannot reach the inner tip, the node still STARTS, in an explicit
|
||||
// backfill-pending state: loud, fail-safe (it will not BUILD a block while its
|
||||
// finality index is incomplete) and repairable through BackfillOuterBlock,
|
||||
// which accepts the missing envelopes from any source and applies the SAME
|
||||
// binding checks. The alternative — the previous behaviour — was a dead
|
||||
// C-Chain and a destructive resync.
|
||||
//
|
||||
// WHAT IS NEVER DONE, and why: the finality pointer is never dropped
|
||||
// (DeleteLastAccepted). proposervm.LastAccepted would then fall back to an
|
||||
// INNER-namespace id whose ParentID is contiguity-incompatible with the network's
|
||||
// OUTER wrappers, permanently wedging bootstrap, catch-up and live Verify at the
|
||||
// inner tip — a silent wedge strictly worse than the loud stop it would replace.
|
||||
// Every path below only ever moves the pointer FORWARD, onto an envelope that was
|
||||
// proven to wrap the inner block this node already accepted at that height.
|
||||
package proposervm
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"github.com/luxfi/ids"
|
||||
"github.com/luxfi/log"
|
||||
|
||||
statelessblock "github.com/luxfi/node/vms/proposervm/block"
|
||||
"github.com/luxfi/node/vms/proposervm/state"
|
||||
)
|
||||
|
||||
var (
|
||||
// ErrOuterBackfillNotPending is returned by BackfillOuterBlock when the index
|
||||
// is already whole — the repair is idempotent, not an error to call twice.
|
||||
ErrOuterBackfillNotPending = errors.New("proposervm: no outer-index backfill is pending")
|
||||
|
||||
// ErrOuterBackfillRejected is returned when a candidate envelope does not bind
|
||||
// to this node's own accepted chain. Fail-closed: an envelope is only indexed
|
||||
// when it provably wraps the inner block WE accepted at that height and chains
|
||||
// off the envelope WE already hold at height-1, so a peer (or a stale file)
|
||||
// cannot steer the finality pointer.
|
||||
ErrOuterBackfillRejected = errors.New("proposervm: outer backfill candidate rejected")
|
||||
|
||||
// errPreForkAfterFork is the prevention guard: post-fork, a block at or above
|
||||
// the recorded fork height is NEVER a pre-fork block. Constructing or accepting
|
||||
// one is what silently strands the finality index below the inner tip.
|
||||
errPreForkAfterFork = errors.New("proposervm: refusing a pre-fork block at or above the fork height")
|
||||
)
|
||||
|
||||
// outerBackfill is the state of an incomplete outer index. Zero value = whole.
|
||||
//
|
||||
// It is guarded by its own leaf mutex (vm.backfillMu) rather than vm.lock: the
|
||||
// accept path, the build path and the operator repair path all read it, and it is
|
||||
// never held across a callout, so it cannot participate in a deadlock.
|
||||
type outerBackfill struct {
|
||||
// next is the FIRST height whose outer envelope is missing. It advances as
|
||||
// envelopes land, so it doubles as the resume point across restarts.
|
||||
next uint64
|
||||
// tip is the inner VM's accepted height — the height the index must reach.
|
||||
tip uint64
|
||||
// innerTipID is the inner block at [tip], recorded for diagnostics.
|
||||
innerTipID ids.ID
|
||||
}
|
||||
|
||||
// NeedsOuterBackfill reports the still-missing outer height range [from, to] and
|
||||
// whether a backfill is pending. It is the seam an operator tool or a node-layer
|
||||
// fetch driver reads; when pending is false the index is whole.
|
||||
func (vm *VM) NeedsOuterBackfill() (from uint64, to uint64, pending bool) {
|
||||
vm.backfillMu.RLock()
|
||||
defer vm.backfillMu.RUnlock()
|
||||
if vm.backfill == nil {
|
||||
return 0, 0, false
|
||||
}
|
||||
return vm.backfill.next, vm.backfill.tip, true
|
||||
}
|
||||
|
||||
// outerBackfillPending is the internal fast check used by the build gate.
|
||||
func (vm *VM) outerBackfillPending() bool {
|
||||
vm.backfillMu.RLock()
|
||||
defer vm.backfillMu.RUnlock()
|
||||
return vm.backfill != nil
|
||||
}
|
||||
|
||||
// innerBlockAtHeight returns the inner VM's ACCEPTED block at [height] together
|
||||
// with its bytes — the binding target every repair candidate must match. It reads
|
||||
// the inner VM's own height index, so it is the inner VM's truth, not ours.
|
||||
func (vm *VM) innerBlockAtHeight(ctx context.Context, height uint64) (ids.ID, []byte, error) {
|
||||
innerID, err := vm.ChainVM.GetBlockIDAtHeight(ctx, height)
|
||||
if err != nil {
|
||||
return ids.Empty, nil, err
|
||||
}
|
||||
innerBlk, err := vm.ChainVM.GetBlock(ctx, innerID)
|
||||
if err != nil {
|
||||
return ids.Empty, nil, err
|
||||
}
|
||||
return innerID, innerBlk.Bytes(), nil
|
||||
}
|
||||
|
||||
// indexOuterAtHeight commits ONE outer envelope into the finality index: the
|
||||
// envelope itself, its height entry, and the last-accepted pointer, in a single
|
||||
// versiondb batch — byte-for-byte what acceptPostForkBlock writes on the live
|
||||
// path. It deliberately does NOT touch the inner VM.
|
||||
func (vm *VM) indexOuterAtHeight(height uint64, blk statelessblock.Block) error {
|
||||
if err := vm.State.PutBlock(blk); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := vm.State.SetBlockIDAtHeight(height, blk.ID()); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := vm.State.SetLastAccepted(blk.ID()); err != nil {
|
||||
return err
|
||||
}
|
||||
return vm.db.Commit()
|
||||
}
|
||||
|
||||
// rebuildOuterIndexFromStore walks the gap (proHeight, innerHeight] forward,
|
||||
// re-indexing each height from an outer envelope ALREADY present in this node's
|
||||
// block store.
|
||||
//
|
||||
// Soundness. A candidate at height h is accepted ONLY if
|
||||
//
|
||||
// envelope.Block() == the bytes of the inner block WE accepted at h, and
|
||||
// envelope.ParentID() == the envelope we just indexed at h-1
|
||||
//
|
||||
// so the rebuilt chain is the unique outer chain that wraps our own accepted inner
|
||||
// chain. Nothing is inferred and nothing is fabricated: an unmatched height stops
|
||||
// the walk immediately.
|
||||
//
|
||||
// Returns the height the index reached (>= proHeight). A return equal to
|
||||
// innerHeight means the index is whole again.
|
||||
func (vm *VM) rebuildOuterIndexFromStore(
|
||||
ctx context.Context,
|
||||
proHeight uint64,
|
||||
proID ids.ID,
|
||||
innerHeight uint64,
|
||||
) (uint64, error) {
|
||||
// One pass over the block store: inner-bytes digest -> outer envelope. Keyed by
|
||||
// digest rather than by inner id because deriving an inner id from bytes would
|
||||
// mean parsing untrusted bytes through the inner VM; the digest comparison is
|
||||
// exact and side-effect free.
|
||||
byInnerDigest := make(map[[sha256.Size]byte]statelessblock.Block)
|
||||
if err := state.ScanBlocks(vm.db, func(blk statelessblock.Block) error {
|
||||
byInnerDigest[sha256.Sum256(blk.Block())] = blk
|
||||
return nil
|
||||
}); err != nil {
|
||||
return proHeight, fmt.Errorf("failed to scan the proposervm block store: %w", err)
|
||||
}
|
||||
|
||||
reached := proHeight
|
||||
parentID := proID
|
||||
for h := proHeight + 1; h <= innerHeight; h++ {
|
||||
_, innerBytes, err := vm.innerBlockAtHeight(ctx, h)
|
||||
if err != nil {
|
||||
vm.logger.Warn("proposervm outer-index rebuild: inner block unreadable, stopping walk",
|
||||
log.Uint64("height", h), log.Err(err))
|
||||
break
|
||||
}
|
||||
candidate, ok := byInnerDigest[sha256.Sum256(innerBytes)]
|
||||
if !ok || candidate.ParentID() != parentID {
|
||||
break
|
||||
}
|
||||
if err := vm.indexOuterAtHeight(h, candidate); err != nil {
|
||||
return reached, fmt.Errorf("failed to re-index outer block at height %d: %w", h, err)
|
||||
}
|
||||
reached = h
|
||||
parentID = candidate.ID()
|
||||
}
|
||||
return reached, nil
|
||||
}
|
||||
|
||||
// enterOuterBackfill records an incomplete index and lets the VM start anyway.
|
||||
// Fail-safe, not fail-open: BuildBlock is gated off while pending (a node whose
|
||||
// finality index has a hole must never PROPOSE), the height index still reports
|
||||
// ErrNotFound inside the gap so nothing reads a wrong id, and the last-accepted
|
||||
// pointer is left exactly where the rebuild got to.
|
||||
func (vm *VM) enterOuterBackfill(from, tip uint64, innerTipID ids.ID) {
|
||||
vm.backfillMu.Lock()
|
||||
vm.backfill = &outerBackfill{next: from, tip: tip, innerTipID: innerTipID}
|
||||
vm.backfillMu.Unlock()
|
||||
|
||||
vm.logger.Warn("proposervm STARTING WITH AN INCOMPLETE FINALITY INDEX — outer backfill pending",
|
||||
log.Uint64("firstMissingHeight", from),
|
||||
log.Uint64("innerTipHeight", tip),
|
||||
log.Stringer("innerTipID", innerTipID),
|
||||
log.String("effect", "this chain will NOT build blocks until the missing outer envelopes are supplied"),
|
||||
log.String("recovery", "feed the outer proposervm envelopes for the missing heights via "+
|
||||
"BackfillOuterBlock (cmd/repair-proposervm backfill) — each is bound to the inner block "+
|
||||
"this node already accepted, so no EVM re-execution and no resync are required"),
|
||||
)
|
||||
}
|
||||
|
||||
// BackfillOuterBlock supplies ONE missing outer envelope, oldest-first, and is the
|
||||
// operator/driver half of the recovery. It is fail-closed and idempotent:
|
||||
//
|
||||
// - not pending -> ErrOuterBackfillNotPending
|
||||
// - envelope below the next height -> nil (already indexed; tolerate replay)
|
||||
// - signature/format invalid -> error (parsed WITH verification)
|
||||
// - not the next height, wrong parent,
|
||||
// or wrapping a different inner block-> ErrOuterBackfillRejected
|
||||
//
|
||||
// On the final height it clears the pending state and refreshes the in-memory
|
||||
// last-accepted metadata, after which the chain behaves exactly like one that
|
||||
// booted whole.
|
||||
func (vm *VM) BackfillOuterBlock(ctx context.Context, outerBytes []byte) error {
|
||||
vm.backfillMu.Lock()
|
||||
defer vm.backfillMu.Unlock()
|
||||
|
||||
if vm.backfill == nil {
|
||||
return ErrOuterBackfillNotPending
|
||||
}
|
||||
height := vm.backfill.next
|
||||
|
||||
// Parse WITH proposer-signature verification: the envelope must be a real
|
||||
// block of THIS chain, not merely well-formed bytes.
|
||||
blk, err := statelessblock.Parse(outerBytes, vm.rt.ChainID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: envelope did not parse/verify: %w", ErrOuterBackfillRejected, err)
|
||||
}
|
||||
|
||||
// BIND 1 — the envelope must chain off the one we already hold.
|
||||
parentID, err := vm.State.GetLastAccepted()
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: cannot read the current finality pointer: %w", ErrOuterBackfillRejected, err)
|
||||
}
|
||||
if blk.ParentID() != parentID {
|
||||
return fmt.Errorf("%w: envelope parent %s != current finality pointer %s (feed oldest-first)",
|
||||
ErrOuterBackfillRejected, blk.ParentID(), parentID)
|
||||
}
|
||||
|
||||
// BIND 2 — the envelope must wrap the inner block WE accepted at this height.
|
||||
// This is what makes accepting bytes from an untrusted source safe.
|
||||
innerID, innerBytes, err := vm.innerBlockAtHeight(ctx, height)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: inner block at height %d unreadable: %w", ErrOuterBackfillRejected, height, err)
|
||||
}
|
||||
if !bytes.Equal(blk.Block(), innerBytes) {
|
||||
return fmt.Errorf("%w: envelope at height %d does not wrap our accepted inner block %s",
|
||||
ErrOuterBackfillRejected, height, innerID)
|
||||
}
|
||||
|
||||
if err := vm.indexOuterAtHeight(height, blk); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
vm.logger.Info("proposervm outer backfill: height repaired",
|
||||
log.Uint64("height", height),
|
||||
log.Stringer("outerID", blk.ID()),
|
||||
log.Uint64("remaining", vm.backfill.tip-height),
|
||||
)
|
||||
|
||||
if height >= vm.backfill.tip {
|
||||
tip := vm.backfill.tip
|
||||
vm.backfill = nil
|
||||
if err := vm.setLastAcceptedMetadata(ctx); err != nil {
|
||||
return fmt.Errorf("outer backfill completed at height %d but last-accepted metadata failed: %w", tip, err)
|
||||
}
|
||||
vm.logger.Info("proposervm FINALITY INDEX FULLY REPAIRED — backfill complete",
|
||||
log.Uint64("height", tip))
|
||||
return nil
|
||||
}
|
||||
vm.backfill.next = height + 1
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,253 @@
|
||||
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
|
||||
// See the file LICENSE for licensing terms.
|
||||
|
||||
// height_backfill_test.go — RECOVERY: a node whose proposervm finality index is
|
||||
// ALREADY behind the inner VM tip must BOOT and REPAIR itself instead of killing
|
||||
// the chain ("non-critical chain failed to initialize chainAlias=C" — a dead
|
||||
// C-Chain on a pod that still reports 1/1 Ready).
|
||||
//
|
||||
// Two paths, both OUTER-ONLY (no inner re-execution, no EVM rollback, no resync):
|
||||
// - the envelopes for the gap are still in the local block store -> rebuilt at
|
||||
// boot with no network and no operator;
|
||||
// - they are not -> the chain STARTS anyway, loud, build-gated, and heals from
|
||||
// supplied envelopes through BackfillOuterBlock.
|
||||
//
|
||||
// The shared harness lives in height_lag_repro_test.go.
|
||||
package proposervm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/ids"
|
||||
|
||||
statelessblock "github.com/luxfi/node/vms/proposervm/block"
|
||||
)
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 2. RECOVERY — a node whose index is ALREADY behind boots and heals.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// TestOuterIndexRebuild_FromLocalStore_BootsAndHeals: the damaged node still holds
|
||||
// the outer envelopes for the gap in its own block store (the truncated-index /
|
||||
// inconsistently-restored-snapshot case). Boot must rebuild the index from them —
|
||||
// no network, no operator, no resync — and must NOT return an error.
|
||||
func TestOuterIndexRebuild_FromLocalStore_BootsAndHeals(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ic := newInnerChain(t, 8)
|
||||
vm := testVM(t, ic)
|
||||
|
||||
outers := acceptThroughProposervm(t, vm, ic, 8)
|
||||
|
||||
// DAMAGE: roll the finality pointer + height index back to 5 while the inner VM
|
||||
// stays at 8 — byte-for-byte the on-disk state luxd-3 booted with (index 6,
|
||||
// inner 98), except the envelopes survive in the block store.
|
||||
if err := vm.State.SetLastAccepted(outers[5].ID()); err != nil {
|
||||
t.Fatalf("SetLastAccepted: %v", err)
|
||||
}
|
||||
for h := uint64(6); h <= 8; h++ {
|
||||
if err := vm.State.DeleteBlockIDAtHeight(h); err != nil {
|
||||
t.Fatalf("DeleteBlockIDAtHeight(%d): %v", h, err)
|
||||
}
|
||||
}
|
||||
if err := vm.db.Commit(); err != nil {
|
||||
t.Fatalf("commit damage: %v", err)
|
||||
}
|
||||
if got := indexHeight(t, vm); got != 5 {
|
||||
t.Fatalf("precondition: damaged index must read 5, got %d", got)
|
||||
}
|
||||
|
||||
// BOOT.
|
||||
if err := vm.repairAcceptedChainByHeight(ctx); err != nil {
|
||||
t.Fatalf("boot with a behind index must REPAIR, not fail: %v", err)
|
||||
}
|
||||
|
||||
if _, _, pending := vm.NeedsOuterBackfill(); pending {
|
||||
t.Fatal("the local store held every missing envelope — no backfill should be pending")
|
||||
}
|
||||
if got := indexHeight(t, vm); got != 8 {
|
||||
t.Fatalf("index must be rebuilt to the inner tip 8, got %d", got)
|
||||
}
|
||||
lastAccepted, err := vm.State.GetLastAccepted()
|
||||
if err != nil || lastAccepted != outers[8].ID() {
|
||||
t.Fatalf("finality pointer must be the canonical envelope at 8 (%s), got %s (err %v)",
|
||||
outers[8].ID(), lastAccepted, err)
|
||||
}
|
||||
for h := uint64(6); h <= 8; h++ {
|
||||
got, err := vm.State.GetBlockIDAtHeight(h)
|
||||
if err != nil || got != outers[h].ID() {
|
||||
t.Fatalf("height index at %d must be %s, got %s (err %v)", h, outers[h].ID(), got, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestOuterBackfill_HealsFromSuppliedEnvelopes: the envelopes for the gap are NOT
|
||||
// in the local store (the pre-fork-fallback case — they were never persisted).
|
||||
// The node must still START — the old code killed the C-Chain here — come up
|
||||
// explicitly backfill-pending, refuse to build, and then heal from supplied
|
||||
// envelopes with no EVM re-execution.
|
||||
func TestOuterBackfill_HealsFromSuppliedEnvelopes(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ic := newInnerChain(t, 8)
|
||||
vm := testVM(t, ic)
|
||||
|
||||
acceptThroughProposervm(t, vm, ic, 5)
|
||||
lastGood, err := vm.State.GetLastAccepted()
|
||||
if err != nil {
|
||||
t.Fatalf("GetLastAccepted: %v", err)
|
||||
}
|
||||
|
||||
// DAMAGE: the inner VM ran on to 8 without the index (exactly what the pre-fork
|
||||
// fallback did in production). No envelopes for 6..8 exist locally.
|
||||
ic.accept(8)
|
||||
|
||||
// BOOT — must not error.
|
||||
if err := vm.repairAcceptedChainByHeight(ctx); err != nil {
|
||||
t.Fatalf("boot with a behind index must START the chain, got: %v", err)
|
||||
}
|
||||
from, to, pending := vm.NeedsOuterBackfill()
|
||||
if !pending || from != 6 || to != 8 {
|
||||
t.Fatalf("want backfill pending 6..8, got from=%d to=%d pending=%v", from, to, pending)
|
||||
}
|
||||
|
||||
// FAIL-SAFE: a node with a hole in its finality index must not propose.
|
||||
if _, err := vm.BuildBlock(ctx); err == nil {
|
||||
t.Fatal("BuildBlock must refuse while the finality index is incomplete")
|
||||
}
|
||||
|
||||
// NEGATIVE 1 — an envelope wrapping a DIFFERENT inner block than the one we
|
||||
// accepted at this height is rejected (a peer cannot steer the pointer).
|
||||
bogus, err := statelessblock.BuildUnsigned(
|
||||
lastGood, time.Unix(6, 0), 0, statelessblock.Epoch{}, []byte("not-our-inner-block"))
|
||||
if err != nil {
|
||||
t.Fatalf("BuildUnsigned: %v", err)
|
||||
}
|
||||
if err := vm.BackfillOuterBlock(ctx, bogus.Bytes()); !errors.Is(err, ErrOuterBackfillRejected) {
|
||||
t.Fatalf("an envelope that does not wrap our accepted inner block must be rejected, got %v", err)
|
||||
}
|
||||
|
||||
// NEGATIVE 2 — out of order (height 7 before 6) is rejected: it does not chain
|
||||
// off the current finality pointer.
|
||||
outOfOrder := outerFor(t, ic, 7, lastGood)
|
||||
if err := vm.BackfillOuterBlock(ctx, outOfOrder.Bytes()); !errors.Is(err, ErrOuterBackfillRejected) {
|
||||
t.Fatalf("an out-of-order envelope must be rejected, got %v", err)
|
||||
}
|
||||
|
||||
// HEAL — feed 6, 7, 8 oldest-first, as a healthy peer or the repair tool would.
|
||||
parent := lastGood
|
||||
want := map[uint64]ids.ID{}
|
||||
for h := uint64(6); h <= 8; h++ {
|
||||
sb := outerFor(t, ic, h, parent)
|
||||
if err := vm.BackfillOuterBlock(ctx, sb.Bytes()); err != nil {
|
||||
t.Fatalf("BackfillOuterBlock(h=%d): %v", h, err)
|
||||
}
|
||||
want[h] = sb.ID()
|
||||
parent = sb.ID()
|
||||
}
|
||||
|
||||
if _, _, pending := vm.NeedsOuterBackfill(); pending {
|
||||
t.Fatal("backfill must be complete after the last height")
|
||||
}
|
||||
if got := indexHeight(t, vm); got != 8 {
|
||||
t.Fatalf("index must reach the inner tip 8, got %d", got)
|
||||
}
|
||||
for h := uint64(6); h <= 8; h++ {
|
||||
got, err := vm.State.GetBlockIDAtHeight(h)
|
||||
if err != nil || got != want[h] {
|
||||
t.Fatalf("height index at %d must be %s, got %s (err %v)", h, want[h], got, err)
|
||||
}
|
||||
}
|
||||
// Idempotent: nothing pending, so a replayed call says so rather than corrupting.
|
||||
if err := vm.BackfillOuterBlock(ctx, outOfOrder.Bytes()); !errors.Is(err, ErrOuterBackfillNotPending) {
|
||||
t.Fatalf("want ErrOuterBackfillNotPending after completion, got %v", err)
|
||||
}
|
||||
// The chain is a full participant again.
|
||||
if _, _, pending := vm.NeedsOuterBackfill(); pending {
|
||||
t.Fatal("build gate must have lifted")
|
||||
}
|
||||
// And a fresh boot on the repaired state is a clean no-op.
|
||||
if err := vm.repairAcceptedChainByHeight(ctx); err != nil {
|
||||
t.Fatalf("boot after repair must be a no-op, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestBootWithBehindIndex_NeverKillsTheChain is the direct regression lock on the
|
||||
// production symptom: repairAcceptedChainByHeight must NEVER return an error for a
|
||||
// behind index, because chains/manager.go turns that error into
|
||||
// "non-critical chain failed to initialize chainAlias=C" — a dead C-Chain on a pod
|
||||
// that still reports 1/1 Ready.
|
||||
func TestBootWithBehindIndex_NeverKillsTheChain(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
for _, gap := range []uint64{1, 2, 6, 91} {
|
||||
ic := newInnerChain(t, 100)
|
||||
vm := testVM(t, ic)
|
||||
acceptThroughProposervm(t, vm, ic, 100-gap)
|
||||
ic.accept(100)
|
||||
|
||||
if err := vm.repairAcceptedChainByHeight(ctx); err != nil {
|
||||
t.Fatalf("gap %d: boot must not fail, got %v", gap, err)
|
||||
}
|
||||
from, to, pending := vm.NeedsOuterBackfill()
|
||||
if !pending || from != 100-gap+1 || to != 100 {
|
||||
t.Fatalf("gap %d: want pending %d..100, got %d..%d pending=%v", gap, 100-gap+1, from, to, pending)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func mustForkHeight(t *testing.T, vm *VM) uint64 {
|
||||
t.Helper()
|
||||
h, err := vm.State.GetForkHeight()
|
||||
if err != nil {
|
||||
t.Fatalf("GetForkHeight: %v", err)
|
||||
}
|
||||
return h
|
||||
}
|
||||
|
||||
// TestOuterBackfill_SelfHealsFromOrdinaryParseTraffic proves the loop closes with
|
||||
// no new transport and no operator: every envelope a peer serves passes through
|
||||
// VM.ParseBlock, and a backfill-pending node absorbs exactly the ones it needs.
|
||||
// This is what lets an ALREADY-DAMAGED node come back by itself.
|
||||
func TestOuterBackfill_SelfHealsFromOrdinaryParseTraffic(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ic := newInnerChain(t, 8)
|
||||
vm := testVM(t, ic)
|
||||
|
||||
acceptThroughProposervm(t, vm, ic, 5)
|
||||
lastGood, err := vm.State.GetLastAccepted()
|
||||
if err != nil {
|
||||
t.Fatalf("GetLastAccepted: %v", err)
|
||||
}
|
||||
ic.accept(8) // the inner VM ran on without the index
|
||||
|
||||
if err := vm.repairAcceptedChainByHeight(ctx); err != nil {
|
||||
t.Fatalf("boot must start the chain, got %v", err)
|
||||
}
|
||||
if _, _, pending := vm.NeedsOuterBackfill(); !pending {
|
||||
t.Fatal("precondition: backfill must be pending")
|
||||
}
|
||||
|
||||
// Peer traffic, oldest-first, delivered through the ORDINARY parse path.
|
||||
parent := lastGood
|
||||
for h := uint64(6); h <= 8; h++ {
|
||||
sb := outerFor(t, ic, h, parent)
|
||||
if _, err := vm.ParseBlock(ctx, sb.Bytes()); err != nil {
|
||||
t.Fatalf("ParseBlock(h=%d): %v", h, err)
|
||||
}
|
||||
parent = sb.ID()
|
||||
}
|
||||
|
||||
if _, _, pending := vm.NeedsOuterBackfill(); pending {
|
||||
t.Fatal("the node must have healed itself from ordinary parse traffic")
|
||||
}
|
||||
if got := indexHeight(t, vm); got != 8 {
|
||||
t.Fatalf("index must reach the inner tip 8, got %d", got)
|
||||
}
|
||||
// The build gate lifted with the heal (the gate itself is asserted in
|
||||
// TestOuterBackfill_HealsFromSuppliedEnvelopes; driving the real builder here
|
||||
// would need a validator set this harness deliberately does not stand up).
|
||||
if _, _, pending := vm.NeedsOuterBackfill(); pending {
|
||||
t.Fatal("build gate must have lifted")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,280 @@
|
||||
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
|
||||
// See the file LICENSE for licensing terms.
|
||||
|
||||
// height_lag_repro_test.go — the DETERMINISTIC REPRODUCTION of the proposervm
|
||||
// finality-index lag, written against the PRE-EXISTING proposervm API only (no
|
||||
// symbol introduced by the fix), so it fails on the unfixed code and passes on the
|
||||
// fixed code without being tautological.
|
||||
//
|
||||
// The production failure it reproduces (lux-devnet luxd-3, C-Chain, verbatim):
|
||||
//
|
||||
// VM initialization failed error="failed to repair accepted chain by height:
|
||||
// proposervm finality index (height 6, id ns5qGN4i…) is BEHIND the inner VM tip
|
||||
// (height 98, id TUTR74eA…)" chainID=21HieZng…
|
||||
// non-critical chain failed to initialize … chainAlias=C
|
||||
//
|
||||
// THE MECHANISM. Every post-fork accept commits the envelope, its height index
|
||||
// entry and the last-accepted pointer in ONE versiondb batch BEFORE the inner
|
||||
// block is accepted, so the index can only run AHEAD. But the proposervm has one
|
||||
// accept path that moves the inner VM and writes NOTHING: preForkBlock.Accept,
|
||||
// whose acceptOuterBlk() is a no-op. Post-fork it was reachable because getBlock()
|
||||
// silently falls back to constructing a preForkBlock whenever the id is not an
|
||||
// OUTER envelope id — and the id this fork's consensus ledger records as canonical
|
||||
// IS the inner block's id (postForkCommonComponents.CanonicalID). One such accept
|
||||
// and the index is stranded: invisible while running, fatal at the next boot.
|
||||
//
|
||||
// This file also holds the shared harness the recovery tests use.
|
||||
package proposervm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/database"
|
||||
"github.com/luxfi/database/memdb"
|
||||
"github.com/luxfi/database/versiondb"
|
||||
"github.com/luxfi/ids"
|
||||
"github.com/luxfi/log"
|
||||
"github.com/luxfi/metric"
|
||||
"github.com/luxfi/runtime"
|
||||
vmchain "github.com/luxfi/vm/chain"
|
||||
"github.com/luxfi/vm/chain/blocktest"
|
||||
|
||||
"github.com/luxfi/node/cache/lru"
|
||||
statelessblock "github.com/luxfi/node/vms/proposervm/block"
|
||||
"github.com/luxfi/node/vms/proposervm/state"
|
||||
"github.com/luxfi/node/vms/proposervm/tree"
|
||||
)
|
||||
|
||||
// innerChain is a stand-in for the inner execution VM (the C-Chain EVM): a
|
||||
// contiguous accepted chain 1..tip answering LastAccepted / GetBlock /
|
||||
// GetBlockIDAtHeight / ParseBlock exactly as the real one does.
|
||||
type innerChain struct {
|
||||
byHeight map[uint64]*blocktest.Block
|
||||
byID map[ids.ID]*blocktest.Block
|
||||
byBytes map[string]*blocktest.Block
|
||||
tip uint64
|
||||
}
|
||||
|
||||
func newInnerChain(t *testing.T, tip uint64) *innerChain {
|
||||
t.Helper()
|
||||
ic := &innerChain{
|
||||
byHeight: map[uint64]*blocktest.Block{},
|
||||
byID: map[ids.ID]*blocktest.Block{},
|
||||
byBytes: map[string]*blocktest.Block{},
|
||||
}
|
||||
parent := ids.Empty
|
||||
for h := uint64(1); h <= tip; h++ {
|
||||
id := ids.GenerateTestID()
|
||||
blk := &blocktest.Block{
|
||||
IDV: id,
|
||||
ParentV: parent,
|
||||
HeightV: h,
|
||||
BytesV: append([]byte("inner"), id[:]...),
|
||||
TimestampV: time.Unix(int64(h), 0),
|
||||
}
|
||||
ic.byHeight[h] = blk
|
||||
ic.byID[id] = blk
|
||||
ic.byBytes[string(blk.BytesV)] = blk
|
||||
parent = id
|
||||
}
|
||||
ic.tip = tip
|
||||
return ic
|
||||
}
|
||||
|
||||
// accept models the inner VM accepting a block: its head advances. This is the
|
||||
// ONLY way the inner tip moves, exactly as in the real EVM.
|
||||
func (ic *innerChain) accept(h uint64) { ic.tip = h }
|
||||
|
||||
func (ic *innerChain) vm() *blocktest.VM {
|
||||
return &blocktest.VM{
|
||||
LastAcceptedF: func(context.Context) (ids.ID, error) {
|
||||
return ic.byHeight[ic.tip].IDV, nil
|
||||
},
|
||||
GetBlockF: func(_ context.Context, id ids.ID) (vmchain.Block, error) {
|
||||
blk, ok := ic.byID[id]
|
||||
if !ok {
|
||||
return nil, errors.New("inner: block not found")
|
||||
}
|
||||
return blk, nil
|
||||
},
|
||||
GetBlockIDAtHeightF: func(_ context.Context, h uint64) (ids.ID, error) {
|
||||
blk, ok := ic.byHeight[h]
|
||||
if !ok {
|
||||
return ids.Empty, errors.New("inner: height not indexed")
|
||||
}
|
||||
return blk.IDV, nil
|
||||
},
|
||||
ParseBlockF: func(_ context.Context, b []byte) (vmchain.Block, error) {
|
||||
blk, ok := ic.byBytes[string(b)]
|
||||
if !ok {
|
||||
return nil, errors.New("inner: unparseable bytes")
|
||||
}
|
||||
return blk, nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// testVM builds a proposervm over memdb wrapping [ic] — the real VM struct with
|
||||
// the real State, so every write below goes through the production code paths.
|
||||
func testVM(t *testing.T, ic *innerChain) *VM {
|
||||
t.Helper()
|
||||
return testVMOnBase(t, ic, memdb.New())
|
||||
}
|
||||
|
||||
// testVMOnBase is testVM over a caller-supplied base database — the seam the
|
||||
// crash-boot tests (vm_crashboot_test.go) use to boot a SECOND, cold VM over
|
||||
// bytes copied out of a running one.
|
||||
func testVMOnBase(t *testing.T, ic *innerChain, base database.Database) *VM {
|
||||
t.Helper()
|
||||
db := versiondb.New(base)
|
||||
metrics := metric.New("")
|
||||
vm := &VM{
|
||||
ChainVM: ic.vm(),
|
||||
db: db,
|
||||
State: state.New(db),
|
||||
Tree: tree.New(),
|
||||
verifiedBlocks: map[ids.ID]PostForkBlock{},
|
||||
innerBlkCache: lru.NewSizedCache(innerBlkCacheSize, cachedBlockSize),
|
||||
logger: log.Noop(),
|
||||
rt: &runtime.Runtime{ChainID: ids.GenerateTestID()},
|
||||
}
|
||||
vm.lastAcceptedTimestampGaugeVec = metrics.NewGaugeVec(
|
||||
"last_accepted_timestamp", "timestamp of the last block accepted", []string{"block_type"})
|
||||
return vm
|
||||
}
|
||||
|
||||
// outerFor wraps the inner block at [h] in a real proposervm envelope chained off
|
||||
// [parentOuter] — the same shape a proposer builds.
|
||||
func outerFor(t *testing.T, ic *innerChain, h uint64, parentOuter ids.ID) statelessblock.SignedBlock {
|
||||
t.Helper()
|
||||
sb, err := statelessblock.BuildUnsigned(
|
||||
parentOuter, time.Unix(int64(h), 0), 0, statelessblock.Epoch{}, ic.byHeight[h].BytesV)
|
||||
if err != nil {
|
||||
t.Fatalf("BuildUnsigned(h=%d): %v", h, err)
|
||||
}
|
||||
return sb
|
||||
}
|
||||
|
||||
// acceptThroughProposervm drives heights 1..through through the REAL post-fork
|
||||
// accept path, so index and inner head advance in lock-step. This is a healthy
|
||||
// node, and it is what records the fork height.
|
||||
func acceptThroughProposervm(t *testing.T, vm *VM, ic *innerChain, through uint64) map[uint64]statelessblock.SignedBlock {
|
||||
t.Helper()
|
||||
return acceptRangeThroughProposervm(t, vm, ic, 1, through, ids.Empty)
|
||||
}
|
||||
|
||||
// acceptRangeThroughProposervm is acceptThroughProposervm for heights
|
||||
// from..through, chaining the first envelope off parentOuter — so a test can
|
||||
// keep a source node accepting AFTER a crash copy was taken.
|
||||
func acceptRangeThroughProposervm(t *testing.T, vm *VM, ic *innerChain, from, through uint64, parentOuter ids.ID) map[uint64]statelessblock.SignedBlock {
|
||||
t.Helper()
|
||||
outers := map[uint64]statelessblock.SignedBlock{}
|
||||
for h := from; h <= through; h++ {
|
||||
sb := outerFor(t, ic, h, parentOuter)
|
||||
blk := &postForkBlock{
|
||||
SignedBlock: sb,
|
||||
postForkCommonComponents: postForkCommonComponents{vm: vm, innerBlk: ic.byHeight[h]},
|
||||
}
|
||||
vm.Tree.Add(ic.byHeight[h])
|
||||
if err := blk.Accept(context.Background()); err != nil {
|
||||
t.Fatalf("post-fork Accept(h=%d): %v", h, err)
|
||||
}
|
||||
ic.accept(h)
|
||||
outers[h] = sb
|
||||
parentOuter = sb.ID()
|
||||
}
|
||||
return outers
|
||||
}
|
||||
|
||||
// indexHeight reads the height the PERSISTED finality index sits at — the exact
|
||||
// quantity repairAcceptedChainByHeight compares against the inner tip at boot.
|
||||
func indexHeight(t *testing.T, vm *VM) uint64 {
|
||||
t.Helper()
|
||||
id, err := vm.State.GetLastAccepted()
|
||||
if err != nil {
|
||||
t.Fatalf("GetLastAccepted: %v", err)
|
||||
}
|
||||
blk, err := vm.getPostForkBlock(context.Background(), id)
|
||||
if err != nil {
|
||||
t.Fatalf("getPostForkBlock(%s): %v", id, err)
|
||||
}
|
||||
return blk.Height()
|
||||
}
|
||||
|
||||
// TestFinalityIndexLag_IndexMustNeverFallBehindInnerTip is THE reproduction.
|
||||
//
|
||||
// It asks the proposervm for the block at the CANONICAL id of height 6 — the id
|
||||
// the consensus ledger records for a proposervm-wrapped block — accepts whatever
|
||||
// comes back, and then asserts the two things a correct proposervm guarantees:
|
||||
//
|
||||
// 1. the persisted finality index is never below the inner VM tip, and
|
||||
// 2. a boot-time reconciliation of that state never kills the chain.
|
||||
//
|
||||
// UNFIXED, getBlock hands back a preForkBlock, its Accept is a silent no-op on the
|
||||
// outer index, the inner VM advances to 6, and this test fails on (1) — printing
|
||||
// the same divergence luxd-3 died of. FIXED, the pre-fork construction/accept is
|
||||
// refused, nothing diverges, and both assertions hold.
|
||||
//
|
||||
// It uses NO symbol introduced by the fix, so it is a genuine before/after probe.
|
||||
func TestFinalityIndexLag_IndexMustNeverFallBehindInnerTip(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ic := newInnerChain(t, 8)
|
||||
vm := testVM(t, ic)
|
||||
|
||||
acceptThroughProposervm(t, vm, ic, 5)
|
||||
if got, tip := indexHeight(t, vm), ic.tip; got != 5 || tip != 5 {
|
||||
t.Fatalf("precondition: want index=5 inner=5, got index=%d inner=%d", got, tip)
|
||||
}
|
||||
forkHeight, err := vm.State.GetForkHeight()
|
||||
if err != nil {
|
||||
t.Fatalf("GetForkHeight: %v", err)
|
||||
}
|
||||
|
||||
// The engine asks for height 6 by its CANONICAL (inner) id.
|
||||
if blk, err := vm.getBlock(ctx, ic.byHeight[6].IDV); err == nil {
|
||||
if _, isPreFork := blk.(*preForkBlock); isPreFork {
|
||||
t.Logf("getBlock(canonical id @6) returned a preForkBlock — fork height is %d", forkHeight)
|
||||
}
|
||||
if err := blk.Accept(ctx); err == nil {
|
||||
ic.accept(6) // the inner VM applied it
|
||||
}
|
||||
}
|
||||
|
||||
// ASSERTION 1 — the invariant.
|
||||
if got := indexHeight(t, vm); got < ic.tip {
|
||||
// Errorf, not Fatalf: assertion 2 below must also run, so one failing run
|
||||
// shows the WHOLE production symptom chain (lag now -> dead chain at boot).
|
||||
t.Errorf("FINALITY INDEX LAG REPRODUCED: index is at height %d but the inner VM tip is %d "+
|
||||
"(fork height %d). An accept advanced the inner VM without writing the proposervm index; "+
|
||||
"this node is now unbootable", got, ic.tip, forkHeight)
|
||||
}
|
||||
|
||||
// ASSERTION 2 — boot must not kill the chain.
|
||||
if err := vm.repairAcceptedChainByHeight(ctx); err != nil {
|
||||
t.Fatalf("BOOT KILLED THE CHAIN: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPreForkAcceptStillWorksBeforeFork is the positive control for the guard: on
|
||||
// a chain that has NOT forked there is no fork height, every block is legitimately
|
||||
// pre-fork, and both construction and Accept must still work. A guard that broke
|
||||
// this would brick every chain before its transition block.
|
||||
func TestPreForkAcceptStillWorksBeforeFork(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ic := newInnerChain(t, 3)
|
||||
vm := testVM(t, ic)
|
||||
|
||||
if _, err := vm.State.GetForkHeight(); err == nil {
|
||||
t.Fatal("precondition: a fresh chain must have no fork height")
|
||||
}
|
||||
pre := &preForkBlock{Block: ic.byHeight[1], vm: vm}
|
||||
if err := pre.Accept(ctx); err != nil {
|
||||
t.Fatalf("pre-fork accept before the fork must succeed, got %v", err)
|
||||
}
|
||||
if _, err := vm.getPreForkBlock(ctx, ic.byHeight[1].IDV); err != nil {
|
||||
t.Fatalf("pre-fork getBlock before the fork must succeed, got %v", err)
|
||||
}
|
||||
}
|
||||
@@ -65,8 +65,20 @@ func (b *preForkBlock) Accept(ctx context.Context) error {
|
||||
return b.acceptInnerBlk(ctx)
|
||||
}
|
||||
|
||||
func (*preForkBlock) acceptOuterBlk() error {
|
||||
return nil
|
||||
// acceptOuterBlk is a NO-OP for a genuine pre-fork block — a pre-fork block has no
|
||||
// outer envelope, so there is nothing to index — and a hard REFUSAL once the chain
|
||||
// has forked.
|
||||
//
|
||||
// The refusal is the second half of the finality-index root-cause fix (the first is
|
||||
// vm.getPreForkBlock). Accepting a pre-fork block at or above the fork height
|
||||
// advances the inner VM while leaving the proposervm index exactly where it was:
|
||||
// the index silently falls behind the inner tip during normal operation, and the
|
||||
// NEXT boot cannot start the chain. Refusing here converts that invisible,
|
||||
// deferred, fatal corruption into an immediate, attributable error at the exact
|
||||
// accept that would have caused it — and the engine's accept path is fail-closed,
|
||||
// so finality stops rather than diverging.
|
||||
func (b *preForkBlock) acceptOuterBlk() error {
|
||||
return b.vm.refusePreForkAfterFork(b.Height())
|
||||
}
|
||||
|
||||
func (b *preForkBlock) acceptInnerBlk(ctx context.Context) error {
|
||||
@@ -275,6 +287,16 @@ func (b *preForkBlock) buildChild(ctx context.Context) (Block, error) {
|
||||
}
|
||||
// else: we ARE the elected proposer for this slot — build the unsigned block.
|
||||
|
||||
// The inner VM builds on ITS OWN head, and verifyPostForkChild below requires the
|
||||
// transition block's inner parent to be exactly THIS block's inner block. Anchor the
|
||||
// two together before delegating (see VM.anchorInnerBuildParent): at the fork height
|
||||
// every validator may emit its own unsigned transition candidate, so a gossiped
|
||||
// sibling that verifies here would otherwise drift the inner head and wedge the
|
||||
// transition — the same self-rejection loop, at chain start.
|
||||
if err := b.vm.anchorInnerBuildParent(ctx, parentID, b.Block.ID()); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var innerBlock chain.Block
|
||||
if b.vm.blockBuilderVM != nil {
|
||||
// VM supports BuildBlockWithRuntime
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
// Copyright (C) 2019-2026, Lux Industries Inc. All rights reserved.
|
||||
// See the file LICENSE for licensing terms.
|
||||
|
||||
package state
|
||||
|
||||
import (
|
||||
"github.com/luxfi/database/prefixdb"
|
||||
"github.com/luxfi/database/versiondb"
|
||||
|
||||
"github.com/luxfi/node/vms/proposervm/block"
|
||||
)
|
||||
|
||||
// ScanBlocks visits every OUTER proposervm block persisted in the block store of
|
||||
// [db], in key order.
|
||||
//
|
||||
// It is a REPAIR-PATH reader, deliberately kept OUT of the State interface: the
|
||||
// hot path addresses blocks by id, and only the boot-time index rebuild
|
||||
// (proposervm.rebuildOuterIndexFromStore) needs to enumerate them. Keeping it a
|
||||
// free function over the same [blockStatePrefix] the store writes under means the
|
||||
// prefix stays defined exactly once, and no mock/interface churn is imposed on
|
||||
// callers that will never scan.
|
||||
//
|
||||
// Records that fail to decode are SKIPPED, not fatal: a rebuild must extract every
|
||||
// usable wrapper from a store that may be partially damaged — that is the whole
|
||||
// point of running it. Only an iterator-level error aborts the scan.
|
||||
func ScanBlocks(db *versiondb.Database, visit func(block.Block) error) error {
|
||||
blockDB := prefixdb.New(blockStatePrefix, db)
|
||||
it := blockDB.NewIterator()
|
||||
defer it.Release()
|
||||
|
||||
for it.Next() {
|
||||
wrapper := blockWrapper{}
|
||||
if err := parseBlockWrapper(it.Value(), &wrapper); err != nil {
|
||||
continue
|
||||
}
|
||||
blk, err := block.ParseWithoutVerification(wrapper.Block)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if err := visit(blk); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return it.Error()
|
||||
}
|
||||
+177
-27
@@ -121,6 +121,16 @@ type VM struct {
|
||||
// [postForkBlock] and its inner block.
|
||||
lastAcceptedTimestampGaugeVec metric.GaugeVec
|
||||
|
||||
// backfillMu guards [backfill]. It is a LEAF lock — never held across a
|
||||
// callout — so it cannot deadlock against [lock], the [Tree] or the inner VM.
|
||||
backfillMu sync.RWMutex
|
||||
// backfill is non-nil ONLY while this node booted with a finality index that
|
||||
// the local block store could not fully rebuild (see height_backfill.go). While
|
||||
// it is set the chain runs read-only-ish: it refuses to BUILD blocks, and
|
||||
// BackfillOuterBlock is the seam that completes the index. nil = index whole,
|
||||
// which is the state of every healthy node.
|
||||
backfill *outerBackfill
|
||||
|
||||
// quasarGate is the OPTIONAL post-quantum finality-cert gate. nil (the
|
||||
// default) means PQ-finality verification is OFF — the accept path is
|
||||
// unchanged classical Snow. When set AND forward-dated activation is reached,
|
||||
@@ -332,6 +342,18 @@ func (vm *VM) SetState(ctx context.Context, newState uint32) error {
|
||||
}
|
||||
|
||||
func (vm *VM) BuildBlock(ctx context.Context) (vmchain.Block, error) {
|
||||
// FAIL-SAFE while the finality index has a hole. A node that booted with an
|
||||
// incomplete outer index (height_backfill.go) does not know the canonical
|
||||
// envelope at its own tip, so anything it proposed would extend a parent the
|
||||
// network does not recognise. It stays a follower until BackfillOuterBlock
|
||||
// closes the gap. Serving RPC and following the chain are unaffected — this is
|
||||
// exactly the difference between the old dead C-Chain and a live one.
|
||||
if from, to, pending := vm.NeedsOuterBackfill(); pending {
|
||||
return nil, fmt.Errorf(
|
||||
"proposervm: refusing to build — finality index incomplete, outer envelopes for heights %d..%d are missing",
|
||||
from, to)
|
||||
}
|
||||
|
||||
preferredBlock, err := vm.getBlock(ctx, vm.preferred)
|
||||
if err == nil {
|
||||
return preferredBlock.buildChild(ctx)
|
||||
@@ -383,6 +405,17 @@ func (vm *VM) BuildBlock(ctx context.Context) (vmchain.Block, error) {
|
||||
}
|
||||
|
||||
func (vm *VM) ParseBlock(ctx context.Context, b []byte) (vmchain.Block, error) {
|
||||
// SELF-HEAL SEAM. Every outer envelope this node receives — gossip, GetAncestors,
|
||||
// catch-up — arrives here. If this node booted with a hole in its finality index
|
||||
// (height_backfill.go), offer the bytes to the backfill: it takes ONLY the exact
|
||||
// next missing height, only if the envelope verifies AND wraps the inner block we
|
||||
// already accepted there, so a wrong or hostile block cannot land. That makes a
|
||||
// damaged node repair itself from ordinary peer traffic with no new transport and
|
||||
// no operator step. Errors are expected and ignored (almost every block is not the
|
||||
// one we need); when nothing is pending this is one RLock and out.
|
||||
if vm.outerBackfillPending() {
|
||||
_ = vm.BackfillOuterBlock(ctx, b)
|
||||
}
|
||||
if blk, err := vm.parsePostForkBlock(ctx, b, true); err == nil {
|
||||
return blk, nil
|
||||
}
|
||||
@@ -474,6 +507,56 @@ func (vm *VM) SetPreference(ctx context.Context, preferred ids.ID) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// anchorInnerBuildParent points the inner VM at [innerParentID] — the inner block wrapped
|
||||
// by the outer parent a build is about to extend — and is called from both build
|
||||
// delegations (postForkCommonComponents.buildChild and preForkBlock.buildChild) as the
|
||||
// last step before asking the inner VM for a block.
|
||||
//
|
||||
// THE INVARIANT it establishes is the one the verify path enforces:
|
||||
// child.innerBlk.Parent() == parent.innerBlk.ID(), else errInnerParentMismatch
|
||||
// (block.go, and pre_fork_block.go for the transition block). The inner VM never reads
|
||||
// the proposervm's parent — it builds on ITS OWN head (luxfi/evm: the miner reads
|
||||
// bc.CurrentBlock()) — so build and verify read two different pointers, and the builder's
|
||||
// is not ours to assume.
|
||||
//
|
||||
// WHY MAINTAINING IT IN SetPreference ALONE IS NOT ENOUGH. The inner head moves for
|
||||
// reasons the proposervm never observes: verifying a GOSSIPED block whose parent is the
|
||||
// current head optimistically makes it the head (evm core/blockchain.go
|
||||
// writeBlockAndSetHead → newTip → writeCanonicalBlockWithLogs → writeHeadBlock), with no
|
||||
// proposervm involvement and no accept. SetPreference cannot undo that drift either — it
|
||||
// short-circuits on an unchanged outer preference (above), so re-affirming the same tip
|
||||
// never re-pushes the inner preference. Every subsequent build then extends the drifted
|
||||
// head, the node REJECTS THE BLOCK IT JUST BUILT, drops it, and repeats forever:
|
||||
// devnet 96367 / testnet 96368 on 2026-07-28, "built block failed verification —
|
||||
// dropping / inner parentID didn't match expected parent", 83–456 drops/min per node,
|
||||
// the accepted tip frozen while the builder ran two heights ahead of it. Never
|
||||
// self-corrects, on any node, ever.
|
||||
//
|
||||
// SetPreference on the INNER VM is the right primitive and the only one: it is that VM's
|
||||
// own head-reorg entry point (evm VM.SetPreference → BlockChain.SetPreference →
|
||||
// writeKnownBlock, which performs the reorg side effects). Asserting it at the point of
|
||||
// use — rather than trusting a distant caller to have done it — is what makes the
|
||||
// requirement local to the code that depends on it.
|
||||
//
|
||||
// COST AND SAFETY. On a healthy node the head already IS the parent's inner block and the
|
||||
// inner SetPreference returns early on that identity (evm setPreference:
|
||||
// `current.Hash() == block.Hash()`), so this is one lookup and no state change — zero
|
||||
// behaviour change in the common case. When it fails, the head is provably NOT the
|
||||
// parent's inner block, so any child built would fail this node's own Verify; refusing to
|
||||
// build is strictly better than emitting a block we are guaranteed to drop.
|
||||
func (vm *VM) anchorInnerBuildParent(ctx context.Context, parentID, innerParentID ids.ID) error {
|
||||
if err := vm.ChainVM.SetPreference(ctx, innerParentID); err != nil {
|
||||
vm.logger.Error("unexpected build block failure",
|
||||
log.String("reason", "failed to anchor the inner VM on the parent's inner block"),
|
||||
log.Stringer("parentID", parentID),
|
||||
log.Stringer("innerParentID", innerParentID),
|
||||
log.Err(err),
|
||||
)
|
||||
return fmt.Errorf("failed to anchor inner build parent %s: %w", innerParentID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (vm *VM) WaitForEvent(ctx context.Context) (vmcore.Message, error) {
|
||||
for {
|
||||
if err := ctx.Err(); err != nil {
|
||||
@@ -783,35 +866,51 @@ func (vm *VM) repairAcceptedChainByHeight(ctx context.Context) error {
|
||||
return nil
|
||||
|
||||
case heightBehind:
|
||||
// INVARIANT VIOLATION and UNRECOVERABLE LOCALLY: the proposervm's finality
|
||||
// index sits BELOW the inner VM's accepted tip. In a correct system this
|
||||
// never happens — the proposervm last-accepted pointer and height index
|
||||
// commit in the SAME versiondb batch as every inner accept, so they cannot
|
||||
// lag. Reaching here means the on-disk proposervm state was truncated
|
||||
// relative to the inner EVM — e.g. a snapshot restored inconsistently across
|
||||
// the two databases (the devnet-C "index 7 < inner 8").
|
||||
// INVARIANT VIOLATION: the proposervm's finality index sits BELOW the inner
|
||||
// VM's accepted tip. Every post-fork accept commits the envelope, its height
|
||||
// entry and the last-accepted pointer in ONE versiondb batch BEFORE the inner
|
||||
// block is accepted, so this can only mean some accept advanced the inner VM
|
||||
// WITHOUT going through acceptPostForkBlock (the pre-fork fallback — now
|
||||
// prevented, see height_backfill.go) or that the on-disk proposervm state was
|
||||
// truncated relative to the inner EVM.
|
||||
//
|
||||
// We FAIL LOUD rather than "self-heal", because there is no correct local
|
||||
// heal: the proposervm cannot fabricate the missing outer wrapper blocks for
|
||||
// heights (pro, inner]. In particular, dropping the finality pointer
|
||||
// (DeleteLastAccepted) is NOT a heal — proposervm.LastAccepted() would then
|
||||
// fall back to the inner-namespace id (see LastAccepted), whose ParentID is
|
||||
// contiguity-incompatible with the network's OUTER wrappers, permanently
|
||||
// wedging bootstrap (the first-block anchor), catch-up (the parent==tip
|
||||
// guard) and live Verify (the parent lookup) at the inner tip — and since
|
||||
// every path skips blocks at height <= the tip, the missing wrapper is never
|
||||
// rebuilt. That silent wedge is strictly worse than this loud, actionable
|
||||
// stop. The correct remedy is operator action; surface it explicitly.
|
||||
return fmt.Errorf(
|
||||
"proposervm finality index (height %d, id %s) is BEHIND the inner VM tip (height %d, id %s): "+
|
||||
"the on-disk proposervm state is truncated/inconsistent relative to the inner EVM "+
|
||||
"(e.g. a snapshot restored inconsistently across the proposervm and EVM databases). "+
|
||||
"This cannot be repaired locally — the proposervm cannot rebuild the missing outer wrapper "+
|
||||
"blocks. RECOVERY: restore a snapshot that is consistent across BOTH databases, or fully "+
|
||||
"resync this node from peers (wipe this chain's db and re-bootstrap). Refusing to auto-reset "+
|
||||
"the finality pointer, which would silently wedge this node at the inner tip forever",
|
||||
proLastAcceptedHeight, proLastAcceptedID, innerLastAcceptedHeight, innerLastAcceptedID,
|
||||
// This used to be a hard init failure, which killed the chain on the node
|
||||
// ("non-critical chain failed to initialize chainAlias=C") and made every
|
||||
// restart fatal. It is now REPAIRED — outer-only, with no inner
|
||||
// re-execution:
|
||||
//
|
||||
// 1. re-derive the index from the outer envelopes already in this node's
|
||||
// block store, each bound to the inner block WE accepted at that height;
|
||||
// 2. if that cannot reach the tip, START ANYWAY in an explicit, loud,
|
||||
// build-gated backfill-pending state that BackfillOuterBlock completes.
|
||||
//
|
||||
// The finality pointer is only ever moved FORWARD onto a proven envelope and
|
||||
// is NEVER dropped: a DeleteLastAccepted here would make LastAccepted() fall
|
||||
// back to an inner-namespace id whose ParentID is contiguity-incompatible
|
||||
// with the network's outer wrappers, silently wedging bootstrap/catch-up/live
|
||||
// Verify at the inner tip forever.
|
||||
vm.logger.Warn("proposervm finality index is BEHIND the inner VM tip — repairing",
|
||||
log.Uint64("indexHeight", proLastAcceptedHeight),
|
||||
log.Stringer("indexID", proLastAcceptedID),
|
||||
log.Uint64("innerTipHeight", innerLastAcceptedHeight),
|
||||
log.Stringer("innerTipID", innerLastAcceptedID),
|
||||
)
|
||||
|
||||
reached, err := vm.rebuildOuterIndexFromStore(
|
||||
ctx, proLastAcceptedHeight, proLastAcceptedID, innerLastAcceptedHeight)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if reached >= innerLastAcceptedHeight {
|
||||
vm.logger.Info("proposervm finality index REBUILT from the local block store — index and inner tip agree",
|
||||
log.Uint64("fromHeight", proLastAcceptedHeight),
|
||||
log.Uint64("toHeight", reached),
|
||||
)
|
||||
return nil
|
||||
}
|
||||
|
||||
vm.enterOuterBackfill(reached+1, innerLastAcceptedHeight, innerLastAcceptedID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// heightAhead: the inner vm is BEHIND the proposer vm (the inner rolled back or
|
||||
@@ -1028,16 +1127,67 @@ func (vm *VM) getPreForkBlock(ctx context.Context, blkID ids.ID) (*preForkBlock,
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// ROOT-CAUSE GUARD (the finality-index lag). getBlock() falls back here
|
||||
// whenever [blkID] is not an OUTER envelope id — and the id the consensus
|
||||
// ledger records as canonical IS the inner block's id
|
||||
// (postForkCommonComponents.CanonicalID). Without this check, asking the
|
||||
// proposervm for a canonical id post-fork silently returns a preForkBlock
|
||||
// wrapping a post-fork inner block, whose Accept advances the inner VM and
|
||||
// leaves the outer index untouched (preForkBlock.acceptOuterBlk is a no-op).
|
||||
// That is how the index ends up BEHIND the inner tip while everything looks
|
||||
// healthy, and why the NEXT boot could not start the chain.
|
||||
//
|
||||
// Post-fork, a block at or above the recorded fork height is by definition NOT
|
||||
// a pre-fork block: there is no such thing as a legitimate pre-fork block at a
|
||||
// height the proposervm has already claimed. Refuse to construct it.
|
||||
if err := vm.refusePreForkAfterFork(engineBlk.Height()); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &preForkBlock{
|
||||
Block: engineBlk,
|
||||
vm: vm,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// refusePreForkAfterFork returns errPreForkAfterFork when [height] is at or above
|
||||
// the recorded proposervm fork height. Before the fork (no fork height recorded)
|
||||
// every block is legitimately pre-fork and this is a no-op, so a chain that has
|
||||
// not yet transitioned is completely unaffected. ONE predicate, used by both the
|
||||
// construction guard (getPreForkBlock) and the accept guard
|
||||
// (preForkBlock.acceptOuterBlk), so the two can never disagree.
|
||||
func (vm *VM) refusePreForkAfterFork(height uint64) error {
|
||||
forkHeight, err := vm.State.GetForkHeight()
|
||||
if err == database.ErrNotFound {
|
||||
return nil // chain has not forked; everything is pre-fork
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if height < forkHeight {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("%w: height %d >= fork height %d", errPreForkAfterFork, height, forkHeight)
|
||||
}
|
||||
|
||||
func (vm *VM) acceptPostForkBlock(blk PostForkBlock) error {
|
||||
height := blk.Height()
|
||||
blkID := blk.ID()
|
||||
|
||||
// EARLY DETECTOR for the finality-index lag. The index must advance one height
|
||||
// at a time; a jump means some accept moved the inner VM without moving the
|
||||
// index (the pre-fork fallback the guards above now refuse) and the gap will be
|
||||
// fatal at the next boot. Surfacing it HERE names the height where the hole
|
||||
// opened instead of leaving a post-mortem for the next restart. Not fatal: the
|
||||
// accept itself is correct and the boot-time rebuild can close a hole; refusing
|
||||
// finality on a warning would be the worse trade.
|
||||
if vm.lastAcceptedHeight != 0 && height != vm.lastAcceptedHeight+1 {
|
||||
vm.logger.Warn("proposervm finality index is NOT contiguous — a height was accepted without being indexed",
|
||||
log.Uint64("previousIndexedHeight", vm.lastAcceptedHeight),
|
||||
log.Uint64("acceptedHeight", height),
|
||||
log.Stringer("blkID", blkID),
|
||||
)
|
||||
}
|
||||
|
||||
vm.lastAcceptedHeight = height
|
||||
vm.forgetVerifiedBlock(blkID)
|
||||
|
||||
|
||||
@@ -0,0 +1,299 @@
|
||||
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
|
||||
// See the file LICENSE for licensing terms.
|
||||
|
||||
// vm_crashboot_test.go — CRASH-COPY RECOVERY ARMOR.
|
||||
//
|
||||
// Every recovery test before this file repaired the SAME warm VM value: its
|
||||
// Tree, verifiedBlocks map and caches were still populated, so a repair that
|
||||
// silently leaned on in-memory state would still have passed. These tests
|
||||
// remove that crutch. They copy the COMMITTED bytes out from under a RUNNING
|
||||
// proposervm — no Shutdown, no extra Commit beyond what the accept path itself
|
||||
// performed — and boot a SECOND, cold VM over the copy: a process kill plus
|
||||
// restart, byte-faithful. The recovered VM must
|
||||
//
|
||||
// 1. BOOT — repairAcceptedChainByHeight then setLastAcceptedMetadata, the
|
||||
// Initialize order — without error;
|
||||
// 2. AGREE with the source's committed view at the copy instant: finality
|
||||
// pointer, fork height, and an openable envelope at EVERY height; and
|
||||
// 3. BUILD — produce a child whose outer parent is the recovered tip and
|
||||
// whose inner parent is that tip's inner block. This is the
|
||||
// errInnerParentMismatch invariant (build_inner_parent_test.go) proven
|
||||
// from a cold boot on persisted state, not just at steady state.
|
||||
//
|
||||
// The matrix pins the persistence boundaries: nothing accepted yet, exactly
|
||||
// one accept (the fork-height record), a longer run, and a copy taken while
|
||||
// the source keeps accepting (the copy must be an immutable snapshot).
|
||||
// TestCrashBoot_OuterCommittedInnerNot pins the ONE window the accept path
|
||||
// leaves open: acceptPostForkBlock commits the outer batch BEFORE the inner
|
||||
// VM accepts, so a crash between the two strands the index one ahead — boot
|
||||
// must roll the pointer back onto the inner survivor and keep building.
|
||||
//
|
||||
// The shared harness lives in height_lag_repro_test.go.
|
||||
package proposervm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/luxfi/database"
|
||||
"github.com/luxfi/database/memdb"
|
||||
"github.com/luxfi/ids"
|
||||
validatorstest "github.com/luxfi/validators/validatorstest"
|
||||
vmchain "github.com/luxfi/vm/chain"
|
||||
"github.com/luxfi/vm/chain/blocktest"
|
||||
)
|
||||
|
||||
// crashCopy copies every committed key/value out of a live base database into
|
||||
// a fresh memdb — a byte-level snapshot taken with the source still running.
|
||||
// This is exactly what a crash leaves behind: versiondb buffers uncommitted
|
||||
// writes in memory, and they die with the process.
|
||||
func crashCopy(t *testing.T, base database.Database) *memdb.Database {
|
||||
t.Helper()
|
||||
cp := memdb.New()
|
||||
it := base.NewIterator()
|
||||
defer it.Release()
|
||||
for it.Next() {
|
||||
if err := cp.Put(it.Key(), it.Value()); err != nil {
|
||||
t.Fatalf("crashCopy Put: %v", err)
|
||||
}
|
||||
}
|
||||
if err := it.Error(); err != nil {
|
||||
t.Fatalf("crashCopy iterator: %v", err)
|
||||
}
|
||||
return cp
|
||||
}
|
||||
|
||||
// snapshot clones the inner chain as its own database would survive a crash at
|
||||
// this instant: every block ever inserted (verified-but-unaccepted included,
|
||||
// as the real EVM persists bodies on insert) with the acceptance tip pinned at
|
||||
// [tip]. The clone is immune to the source's subsequent progress.
|
||||
func (ic *innerChain) snapshot(tip uint64) *innerChain {
|
||||
cp := &innerChain{
|
||||
byHeight: make(map[uint64]*blocktest.Block, len(ic.byHeight)),
|
||||
byID: make(map[ids.ID]*blocktest.Block, len(ic.byID)),
|
||||
byBytes: make(map[string]*blocktest.Block, len(ic.byBytes)),
|
||||
tip: tip,
|
||||
}
|
||||
for h, b := range ic.byHeight {
|
||||
cp.byHeight[h] = b
|
||||
}
|
||||
for id, b := range ic.byID {
|
||||
cp.byID[id] = b
|
||||
}
|
||||
for s, b := range ic.byBytes {
|
||||
cp.byBytes[s] = b
|
||||
}
|
||||
return cp
|
||||
}
|
||||
|
||||
// buildableVM is ic.vm() plus the two calls a build needs. SetPreference is
|
||||
// anchorInnerBuildParent's seam — it moves the head exactly as the EVM does.
|
||||
// BuildBlock mints a FRESH child of the head (building is not accepting, and a
|
||||
// replacement proposal after a crash is a new block, not the lost one).
|
||||
func (ic *innerChain) buildableVM() *blocktest.VM {
|
||||
vm := ic.vm()
|
||||
vm.SetPreferenceF = func(_ context.Context, id ids.ID) error {
|
||||
blk, ok := ic.byID[id]
|
||||
if !ok {
|
||||
return errors.New("inner: preference not in chain")
|
||||
}
|
||||
ic.tip = blk.HeightV
|
||||
return nil
|
||||
}
|
||||
vm.BuildBlockF = func(context.Context) (vmchain.Block, error) {
|
||||
parent := ic.byHeight[ic.tip]
|
||||
id := ids.GenerateTestID()
|
||||
child := &blocktest.Block{
|
||||
IDV: id,
|
||||
ParentV: parent.IDV,
|
||||
HeightV: parent.HeightV + 1,
|
||||
BytesV: append([]byte("inner"), id[:]...),
|
||||
TimestampV: parent.TimestampV.Add(time.Second),
|
||||
}
|
||||
ic.byID[id] = child
|
||||
ic.byBytes[string(child.BytesV)] = child
|
||||
return child, nil
|
||||
}
|
||||
return vm
|
||||
}
|
||||
|
||||
// bootRecoveredVM stands a COLD VM up over copied bytes and runs the boot
|
||||
// sequence in Initialize order. Windower/validator-state/no-op wiring matches
|
||||
// build_inner_parent_test.go: every slot open, so builds take the unsigned
|
||||
// path and the recovery properties are the only thing under test.
|
||||
func bootRecoveredVM(t *testing.T, ic *innerChain, base database.Database) *VM {
|
||||
t.Helper()
|
||||
vm := testVMOnBase(t, ic, base)
|
||||
vm.ChainVM = ic.buildableVM()
|
||||
vm.Windower = anyoneCanProposeWindower{}
|
||||
vm.validatorState = &validatorstest.State{
|
||||
GetCurrentHeightF: func(context.Context) (uint64, error) { return 0, nil },
|
||||
}
|
||||
if err := vm.repairAcceptedChainByHeight(context.Background()); err != nil {
|
||||
t.Fatalf("BOOT KILLED THE CHAIN (repair): %v", err)
|
||||
}
|
||||
if err := vm.setLastAcceptedMetadata(context.Background()); err != nil {
|
||||
t.Fatalf("BOOT KILLED THE CHAIN (metadata): %v", err)
|
||||
}
|
||||
return vm
|
||||
}
|
||||
|
||||
// mustBuildOnRecoveredTip drives the recovered VM's REAL public build path —
|
||||
// SetPreference(last accepted) then BuildBlock, what the engine does after a
|
||||
// restart — and asserts the built child extends the recovered tip on BOTH
|
||||
// layers: outer parent == the tip envelope, inner parent == the tip's inner
|
||||
// block. A recovered node failing either would reject its own proposal.
|
||||
func mustBuildOnRecoveredTip(t *testing.T, vm *VM, ic *innerChain) {
|
||||
t.Helper()
|
||||
require := require.New(t)
|
||||
ctx := context.Background()
|
||||
|
||||
// Deterministic clock strictly after the recovered tip's timestamp
|
||||
// (harness blocks carry time.Unix(height, 0)).
|
||||
vm.Clock.Set(time.Unix(int64(ic.tip)+2, 0))
|
||||
|
||||
lastAccepted, err := vm.LastAccepted(ctx)
|
||||
require.NoError(err)
|
||||
require.NoError(vm.SetPreference(ctx, lastAccepted),
|
||||
"the recovered tip must be fetchable — SetPreference validates before assigning")
|
||||
|
||||
built, err := vm.BuildBlock(ctx)
|
||||
require.NoError(err, "a crash-recovered node must be able to build")
|
||||
child, ok := built.(*postForkBlock)
|
||||
require.True(ok, "the child of a recovered tip is a post-fork envelope")
|
||||
|
||||
wantInnerParent := ic.byHeight[ic.tip].IDV
|
||||
require.Equal(wantInnerParent, child.innerBlk.Parent(),
|
||||
"the built child's inner parent must be the recovered tip's inner block "+
|
||||
"(errInnerParentMismatch invariant, from a cold boot)")
|
||||
require.Equal(ic.tip+1, child.Height())
|
||||
require.Equal(lastAccepted, child.Parent(),
|
||||
"the built child's outer parent must be the recovered finality pointer")
|
||||
}
|
||||
|
||||
// TestCrashBoot_ColdVMOverDirtyCopy_AgreesAndBuilds is the persistence-boundary
|
||||
// matrix: commit-state copies taken with NO shutdown, each booted by a second,
|
||||
// cold VM that must agree with the source's committed view at the copy instant
|
||||
// and then build on it.
|
||||
func TestCrashBoot_ColdVMOverDirtyCopy_AgreesAndBuilds(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
accepted uint64 // heights accepted through the proposervm before the copy
|
||||
runOnAfter uint64 // heights the SOURCE keeps accepting after the copy
|
||||
}{
|
||||
// Nothing ever accepted through the proposervm: no fork height, no
|
||||
// outer state. Boot must be a no-op and the pre-fork world intact.
|
||||
{name: "nothing_accepted", accepted: 0, runOnAfter: 0},
|
||||
// Exactly one accept — the boundary that records the fork height.
|
||||
{name: "first_accept_records_fork", accepted: 1, runOnAfter: 0},
|
||||
{name: "several_accepted", accepted: 8, runOnAfter: 0},
|
||||
// The copy is taken and the source RUNS ON: later accepts must not
|
||||
// leak into the recovered view (the copy is an immutable snapshot).
|
||||
{name: "copy_while_source_runs_on", accepted: 5, runOnAfter: 3},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
require := require.New(t)
|
||||
ctx := context.Background()
|
||||
|
||||
ic := newInnerChain(t, tt.accepted+tt.runOnAfter+2)
|
||||
base := memdb.New()
|
||||
src := testVMOnBase(t, ic, base)
|
||||
outers := acceptThroughProposervm(t, src, ic, tt.accepted)
|
||||
|
||||
// THE COPY — no Shutdown, the source stays live. Snapshot both
|
||||
// layers at the same instant, as one machine's disk would be.
|
||||
cp := crashCopy(t, base)
|
||||
icAtCopy := ic.snapshot(ic.tip)
|
||||
|
||||
if tt.runOnAfter > 0 {
|
||||
acceptRangeThroughProposervm(
|
||||
t, src, ic, tt.accepted+1, tt.accepted+tt.runOnAfter, outers[tt.accepted].ID())
|
||||
if got := indexHeight(t, src); got != tt.accepted+tt.runOnAfter {
|
||||
t.Fatalf("source must have run on to %d, at %d", tt.accepted+tt.runOnAfter, got)
|
||||
}
|
||||
}
|
||||
|
||||
vm2 := bootRecoveredVM(t, icAtCopy, cp)
|
||||
|
||||
if tt.accepted == 0 {
|
||||
// Pre-fork world: no outer state exists, none may be invented.
|
||||
_, err := vm2.State.GetLastAccepted()
|
||||
require.ErrorIs(err, database.ErrNotFound)
|
||||
_, err = vm2.State.GetForkHeight()
|
||||
require.ErrorIs(err, database.ErrNotFound)
|
||||
// LastAccepted falls through to the inner chain.
|
||||
la, err := vm2.LastAccepted(ctx)
|
||||
require.NoError(err)
|
||||
require.Equal(icAtCopy.byHeight[icAtCopy.tip].IDV, la)
|
||||
} else {
|
||||
// The recovered finality pointer is the copy-instant tip —
|
||||
// not genesis, not the source's later progress.
|
||||
got, err := vm2.State.GetLastAccepted()
|
||||
require.NoError(err)
|
||||
require.Equal(outers[tt.accepted].ID(), got)
|
||||
require.Equal(mustForkHeight(t, src), mustForkHeight(t, vm2))
|
||||
|
||||
// Every height's envelope is indexed AND openable cold.
|
||||
for h := uint64(1); h <= tt.accepted; h++ {
|
||||
id, err := vm2.State.GetBlockIDAtHeight(h)
|
||||
require.NoError(err, "height %d must be indexed", h)
|
||||
require.Equal(outers[h].ID(), id, "height %d", h)
|
||||
blk, err := vm2.getPostForkBlock(ctx, id)
|
||||
require.NoError(err, "envelope at height %d must be openable on the recovered VM", h)
|
||||
require.Equal(h, blk.Height())
|
||||
}
|
||||
}
|
||||
if _, _, pending := vm2.NeedsOuterBackfill(); pending {
|
||||
t.Fatal("a lock-step copy has no gap — no backfill may be pending")
|
||||
}
|
||||
|
||||
mustBuildOnRecoveredTip(t, vm2, icAtCopy)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestCrashBoot_OuterCommittedInnerNot_RollsBackAndBuilds pins the one crash
|
||||
// window the accept path leaves open. acceptPostForkBlock commits the envelope,
|
||||
// height index and finality pointer in ONE batch BEFORE the inner VM accepts
|
||||
// (that ordering is why the index can never legitimately fall behind). A crash
|
||||
// inside the window therefore persists an index one AHEAD of the inner
|
||||
// survivor. Boot must take repairAcceptedChainByHeight's roll-back arm — move
|
||||
// the pointer back onto the height the inner VM survived at — and the node
|
||||
// must then build its replacement proposal there.
|
||||
func TestCrashBoot_OuterCommittedInnerNot_RollsBackAndBuilds(t *testing.T) {
|
||||
require := require.New(t)
|
||||
ctx := context.Background()
|
||||
|
||||
ic := newInnerChain(t, 8)
|
||||
base := memdb.New()
|
||||
src := testVMOnBase(t, ic, base)
|
||||
outers := acceptThroughProposervm(t, src, ic, 4)
|
||||
require.Equal(uint64(4), ic.tip, "precondition: lock-step through 4")
|
||||
|
||||
// Height 5's outer accept COMMITS…
|
||||
sb := outerFor(t, ic, 5, outers[4].ID())
|
||||
blk := &postForkBlock{
|
||||
SignedBlock: sb,
|
||||
postForkCommonComponents: postForkCommonComponents{vm: src, innerBlk: ic.byHeight[5]},
|
||||
}
|
||||
src.Tree.Add(ic.byHeight[5])
|
||||
require.NoError(blk.Accept(ctx))
|
||||
// …and the process dies BEFORE the inner VM accepts 5 (no ic.accept(5)).
|
||||
|
||||
cp := crashCopy(t, base)
|
||||
icAtCopy := ic.snapshot(4)
|
||||
vm2 := bootRecoveredVM(t, icAtCopy, cp)
|
||||
|
||||
got, err := vm2.State.GetLastAccepted()
|
||||
require.NoError(err)
|
||||
require.Equal(outers[4].ID(), got,
|
||||
"boot must roll the finality pointer back onto the height the inner VM survived at")
|
||||
|
||||
// The node re-proposes at 5: a NEW envelope over a NEW inner child of 4 —
|
||||
// the lost block is gone, not resurrected.
|
||||
mustBuildOnRecoveredTip(t, vm2, icAtCopy)
|
||||
}
|
||||
@@ -17,7 +17,7 @@ import (
|
||||
"github.com/luxfi/crypto/secp256k1"
|
||||
"github.com/luxfi/go-bip32"
|
||||
"github.com/luxfi/go-bip39"
|
||||
"github.com/luxfi/keys"
|
||||
"github.com/luxfi/kms/pkg/mnemonic"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -31,10 +31,10 @@ const (
|
||||
|
||||
// MustLoadKey loads a secp256k1 private key from (in order of priority):
|
||||
// 1. MNEMONIC environment variable (BIP-39 mnemonic phrase)
|
||||
// 2. via native ZAP — when KMS_ADDR + KMS_ENV +
|
||||
// 2. via native ZAP — when KMS_ADDR + KMS_ENV +
|
||||
// KMS_MNEMONIC_PATH are set in the environment. Uses the canonical
|
||||
// luxfi/kms keys.LoadMnemonic loader so every Lux-derived
|
||||
// service resolves keys the same way.
|
||||
// kms/pkg/mnemonic loader so every Lux-derived service resolves
|
||||
// keys the same way.
|
||||
// 3. Key name provided as first command-line argument
|
||||
// 4. ~/.lux/keys/default/ if it exists
|
||||
//
|
||||
@@ -58,28 +58,31 @@ func MustLoadKey() *secp256k1.PrivateKey {
|
||||
// LoadKey attempts to load a private key using the priority order above.
|
||||
func LoadKey() (*secp256k1.PrivateKey, error) {
|
||||
// 1. MNEMONIC env var — local dev + CI test seam.
|
||||
if mnemonic := strings.TrimSpace(os.Getenv("MNEMONIC")); mnemonic != "" {
|
||||
return keyFromMnemonic(mnemonic)
|
||||
if phrase := strings.TrimSpace(os.Getenv("MNEMONIC")); phrase != "" {
|
||||
return keyFromMnemonic(phrase)
|
||||
}
|
||||
|
||||
// 2. via native ZAP — production path for any Lux-derived
|
||||
// service running under a KMS-projected env. The canonical loader
|
||||
// lives in luxfi/keys (alongside the BIP-39 derivation primitives)
|
||||
// so luxd, netrunner, lux/cli, and every descending L1's bootstrap
|
||||
// all resolve mnemonics the same way.
|
||||
// Trust at the network boundary (NetworkPolicy + ZAP wire) — the
|
||||
// staking material itself comes from the KMS-held operational
|
||||
// mnemonic at KMS_MNEMONIC_PATH (default /mnemonic).
|
||||
// lives in kms/pkg/mnemonic, not luxfi/keys: keys must not import
|
||||
// kms, because kms already imports keys for ServiceIdentity, and
|
||||
// the back edge would close an import cycle. luxd, netrunner,
|
||||
// lux/cli, and every descending L1's bootstrap all resolve
|
||||
// mnemonics through it.
|
||||
// Trust at the network boundary (NetworkPolicy + ZAP wire) — hence
|
||||
// a nil ServiceIdentity — the staking material itself comes from
|
||||
// the KMS-held operational mnemonic at KMS_MNEMONIC_PATH.
|
||||
if addr := os.Getenv("KMS_ADDR"); addr != "" {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
mnemonic, err := keys.LoadMnemonicFromKMS(ctx, addr,
|
||||
phrase, err := mnemonic.LoadFromKMS(ctx, addr,
|
||||
os.Getenv("KMS_ENV"),
|
||||
envOr("KMS_MNEMONIC_PATH", "/mnemonic"))
|
||||
envOr("KMS_MNEMONIC_PATH", "/mnemonic"),
|
||||
nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load mnemonic from KMS: %w", err)
|
||||
}
|
||||
return keyFromMnemonic(mnemonic)
|
||||
return keyFromMnemonic(phrase)
|
||||
}
|
||||
|
||||
// 3. Key name from command-line arguments.
|
||||
|
||||
Reference in New Issue
Block a user