luxfi/netrunner
1
0
Fork 0
mirror of https://github.com/luxfi/netrunner.git synced 2026-07-27 00:04:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
netrunner/LLM.md
T
zeekayandHanzo Dev 26e21d0a8e chore: migrate luxd HTTP routes /ext -> /v1 
Drop the Avalanche-heritage /ext prefix; /v1 is the single canonical route
surface (one way, no backward compat). The node's baseURL is the source of
truth; clients, SDKs, CLI, indexer, maker, genesis, netrunner, and the
k8s/compose/gateway/explorer configs are updated to match.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-07-01 11:40:27 -07:00

40 KiB
Raw Permalink Blame History

Lux Netrunner - AI Assistant Knowledge Base

Project: Lux Netrunner Organization: Lux Network Documentation Score: 85/100

Project Overview

Lux Netrunner is a powerful network orchestration and testing framework for blockchain development. It provides comprehensive tools for creating, managing, and testing multi-node blockchain networks with support for custom VMs, application chains, and complex network topologies.

Essential Commands

Development

# Build from source
./scripts/build.sh

# Run tests
go test ./...

# Run E2E tests
./scripts/tests.e2e.sh

# Install binary
curl -sSfL https://raw.githubusercontent.com/luxfi/netrunner/main/scripts/install.sh | sh -s

Basic Operations

# Start RPC server
netrunner server --port=":8080" --grpc-gateway-port=":8081"

# Start network
netrunner control start --number-of-nodes=3 --node-path=/path/to/luxd

# Check health
netrunner control health

# Get status
netrunner control status

# Save snapshot
netrunner control save-snapshot snapshot-name

# Save hot snapshot (no node stop)
netrunner control save-hot-snapshot snapshot-name

# Load snapshot
netrunner control load-snapshot snapshot-name

Architecture

Core Components

  • RPC Server: gRPC server with REST gateway for network control
  • Network Manager: Orchestrates node lifecycle and network topology
  • Node Process: Wrapper around blockchain node binaries
  • Snapshot Manager: State persistence and recovery
  • Test Framework: Comprehensive testing utilities

Supported Engines

  1. Lux - Native Lux blockchain (primary)
  2. Geth - Ethereum nodes
  3. Optimism - Layer 2 scaling solution
  4. Eth2 - Ethereum 2.0 beacon chain

Key Interfaces

  • network.Network - Network management interface
  • node.Node - Individual node control
  • NodeProcessCreator - Node process factory
  • api.Client - API client interface

Key Technologies

  • Language: Go 1.21+
  • RPC: gRPC with REST gateway
  • Documentation: Next.js with Fumadocs
  • Testing: Go testing package, E2E test framework
  • Build: Make, shell scripts
  • Containerization: Docker support

Development Workflow

  1. Setup: Clone repo, install Go 1.21+
  2. Build: Run ./scripts/build.sh
  3. Test: Run go test ./... for unit tests
  4. E2E Test: Run ./scripts/tests.e2e.sh
  5. Documentation: Edit MDX files in /docs/content/docs/
  6. Doc Build: Run cd docs && pnpm build

Documentation Status (85/100)

✅ Completed Documentation

  1. Introduction - Comprehensive overview, quick start, API examples
  2. Network Orchestration - Complete network management guide
  3. Test Network Setup - Detailed setup instructions
  4. Configuration Reference - Enhanced with templates, multi-engine support
  5. Testing Scenarios - All testing patterns documented
  6. Performance Testing (NEW) - TPS benchmarking, load testing, reports
  7. Network Simulation (NEW) - Topologies, failures, chaos testing

❌ Missing Documentation

  • gRPC/REST API reference
  • Multi-network management guide
  • Custom VM development tutorial
  • Production deployment guide

Key Capabilities

Network Simulation

  • Topologies: Star, Mesh, Hierarchical, Custom
  • Failures: Random, Cascading, Byzantine, Partition
  • Conditions: Latency, Bandwidth limits, Packet loss
  • Resources: CPU/Memory/Disk I/O constraints
  • Time: Fast-forward and slow-motion simulation

Performance Testing

  • Metrics: TPS, BPS, Latency, Resource usage
  • Load Types: Sustained, Burst, Gradual ramp
  • Analysis: Real-time monitoring, HTML reports
  • Optimization: Database tuning, network config

Configuration Management

  • Hierarchy: Node → Network → File → Default
  • Features: Hot-reload, Templates, Multi-engine
  • Validation: Pre-flight checks, port availability

Recent Documentation Enhancements (2025-11-12)

New Documentation Files Created

  1. performance-testing.mdx (1020 lines)

    • Transaction throughput testing
    • Block production benchmarks
    • Network scalability tests
    • Resource monitoring
    • Performance reports and CSV export

  2. network-simulation.mdx (890 lines)

    • Network topology simulation
    • Failure injection scenarios
    • Byzantine behavior testing
    • Network condition simulation
    • Chaos engineering

Enhanced Documentation

  • configuration.mdx - Added 250+ lines:
    • Dynamic runtime configuration
    • Configuration templates
    • Multi-engine support
    • Monitoring configuration
    • Chaos engineering settings

Testing Patterns

Test Types Supported

  1. Unit Testing - Component validation
  2. Integration Testing - Multi-node consensus
  3. Performance Testing - Throughput and latency
  4. Chaos Testing - Failure injection
  5. Regression Testing - Snapshot-based
  6. Load Testing - Sustained and burst

Example Test Scenarios

// Performance test
TestTransactionThroughput()
TestBlockProductionRate()
TestNetworkScalability()

// Chaos test
SimulateNodeFailures()
SimulateNetworkPartition()
SimulateByzantineNode()

// Load test
TestSustainedLoad()
TestBurstLoad()

Best Practices

  1. Start Simple - Begin with basic scenarios
  2. Use Snapshots - Regular automated snapshots
  3. Monitor Everything - Enable metrics from start
  4. Test Incrementally - Gradual complexity increase
  5. Document Changes - Update LLM.md with discoveries

Common Issues and Solutions

Issue: Nodes exit with code 1

Solution: Check binary path, port availability, genesis configuration

Issue: Network partition doesn't heal

Solution: Ensure proper bootstrap node configuration

Issue: Low TPS in tests

Solution: Optimize consensus parameters, use in-memory DB

Future Work

Priority 1 (High)

  • Complete gRPC API documentation
  • Multi-network management guide
  • Production deployment guide

Priority 2 (Medium)

  • Custom VM development tutorial
  • Bridge functionality docs
  • Video tutorials

Priority 3 (Low)

  • Performance cookbook
  • Troubleshooting guide
  • Interactive examples

Context for All AI Assistants

This file (LLM.md) is symlinked as:

  • .AGENTS.md
  • CLAUDE.md
  • QWEN.md
  • GEMINI.md

All files reference the same knowledge base. Updates here propagate to all AI systems.

Rules for AI Assistants

  1. ALWAYS update LLM.md with significant discoveries
  2. NEVER commit symlinked files (.AGENTS.md, CLAUDE.md, etc.) - they're in .gitignore
  3. NEVER create random summary files - update THIS file
  4. USE absolute paths when referencing files
  5. TEST documentation builds before claiming completion

Note: This file serves as the single source of truth for all AI assistants working on this project.

Local Network Deployment Status (2025-11-12)

Current Achievement

✅ Successfully deployed 5-node local network using direct luxd execution

  • Network ID: 12345
  • Binary: /Users/z/work/lux/node/build/luxd v1.20.1
  • Data Directory: /tmp/lux-5node-final/
  • Node Ports: 9650, 9660, 9670, 9680, 9690 (HTTP), 9651, 9661, 9671, 9681, 9691 (Staking)

Bootstrap Status

✅ P-Chain: All 5 nodes bootstrapped successfully ❌ X-Chain: Not bootstrapping (nodes not validators) ❌ C-Chain: Not bootstrapping (nodes not validators)
❌ Q-Chain: Not bootstrapping (nodes not validators)

Root Cause Analysis

Issue: Nodes are not configured as validators in genesis Error: "node is not a validator" when attempting to get uptime Impact: Only P-Chain can bootstrap. X, C, and Q chains require validator status to participate in consensus.

Solution Path

To achieve full 4-chain bootstrap, we need to:

  1. Generate genesis file with 5 pre-configured validators
  2. Extract node IDs and BLS keys from running nodes
  3. Include these node IDs in genesis validator set
  4. Restart network with validator-enabled genesis

Build Status

✅ Netrunner: Built successfully (/Users/z/work/lux/netrunner/bin/netrunner - 43MB) ✅ Lux-CLI: Built successfully (/Users/z/work/lux/cli/bin/lux - 96MB) ✅ Luxd Node: Built successfully (/Users/z/work/lux/node/build/luxd - 57MB)

Deployment Methods Attempted

  1. ❌ Netrunner Server/Client: 503 error, network terminated prematurely
  2. ❌ Lux-CLI: Corrupted snapshot, node3 stopped unexpectedly
  3. ✅ Direct luxd execution: Successful P-Chain bootstrap

Next Steps

  1. Create genesis tool or use existing github.com/luxfi/genesis package
  2. Generate 5-validator genesis file
  3. Configure nodes with proper staking keys
  4. Restart network and verify <60s bootstrap for all 4 chains
  5. Build e2e tests for X, C, Q chain operations

Files Created/Modified

  • /tmp/deploy-network.sh - 5-node deployment script
  • /tmp/lux-5node-final/ - Network data directory with 5 nodes
  • /tmp/start_5node_network.go - Attempted netrunner Go integration

Key Learnings

  1. P-Chain bootstraps without validator status (platform chain special case)
  2. X, C, Q chains require nodes to be in validator set
  3. Genesis must be created BEFORE node deployment for validator networks
  4. Direct luxd execution is most reliable for custom network configurations

Local Network Deployment Issues and Solutions (2025-11-12)

Critical Version/Release Issues Discovered

1. Version Mismatch Between Releases and Tags

Problem: Latest GitHub release (v1.14.0) doesn't match latest Git tag (v1.20.3)

  • File: /Users/z/work/lux/node/.github/workflows/release.yml
  • Impact: Users/tools download v1.14.0 binaries when expecting v1.20.3 code
  • Root Cause: No automated GitHub release workflow when tags are pushed
  • Solution: Created workflow to automatically create GitHub releases on tag push

2. Published v1.20.3 Missing Critical Code

Problem: v1.20.3 release tarball missing WarpSet/WarpValidator implementations

  • Files Missing:
    • consensus/validator/validators.go (WarpSet, WarpValidator types)
    • consensus/validator/validator.go (GetWarpValidatorSet methods)
  • Impact: Code that compiles locally fails when users download release
  • Verification: Release tarball extracted and verified missing implementations
  • Solution: Created new v1.20.4 tag and release with complete code

3. lux-cli Hardcoded Wrong Versions

Problem: lux-cli downloads wrong binary versions

  • File: /Users/z/work/lux/cli/pkg/constants/constants.go
  • Issues:
    • Default luxd version: v2.0.0 (doesn't exist)
    • NetRunner version: v1.13.0 (too old)
    • BuildDir: .lux/bin/luxgo (should be luxfi)
  • Impact: lux network start downloads broken/wrong binaries
  • Solution: 
    DefaultNodeVersion    = "v1.20.4"  // was v2.0.0
DefaultNetRunnerVersion = "v1.20.4"  // was v1.13.0
BuildDir = ".lux/bin/luxfi"  // was luxgo

4. No Automated Release Process

Problem: Tags created but no GitHub releases published

  • Missing: Automated workflow to:
    • Build binaries for all platforms (darwin-amd64/arm64, linux-amd64/arm64)
    • Create GitHub release with binaries attached
    • Generate checksums and release notes
  • Solution: Created .github/workflows/release.yml with:
    • Trigger on tag push (v*)
    • Multi-platform binary builds
    • Automated release creation
    • Checksum generation

Port Configuration Issues

Default Port Series Wrong

Problem: Code uses 964x series, should use 963x

  • Files:
    • /Users/z/work/lux/netrunner/local/network.go - Node creation
    • /Users/z/work/lux/netrunner/cmd/control/start/start.go - CLI defaults
  • Expected Layout: 
    Node 1: HTTP=9630, Staking=9631
Node 2: HTTP=9640, Staking=9641
Node 3: HTTP=9650, Staking=9651
Node 4: HTTP=9660, Staking=9661
Node 5: HTTP=9670, Staking=9671
  • Issue: Code was incrementing from 9650 (964x), not 9630 (963x)
  • Solution: Changed baseHTTPPort from 9650 to 9630 in network.go

Genesis Configuration Issues

1. Invalid Bech32 Address Checksums

Problem: Manual genesis.json has invalid address checksums

  • Error Message: 
    invalid checksum (expected v3r5gtp, got 06g97jh) for "lux1w5f4p2cq7ajjpphny0k8fhkp3wr7a7ggv3r5gtp"
  • Root Cause: Manual address creation without proper Bech32 checksum calculation
  • Files Affected: Any manually created genesis.json
  • Solution: Use proper Bech32 encoding library: 
    import "github.com/luxfi/crypto/address"
addr, err := address.Format("lux", hrp, publicKeyHash)

2. Missing Validator Configuration

Problem: Nodes not configured as validators in genesis

  • Error: "node is not a validator" when querying uptime
  • Impact:
    • P-Chain bootstraps (doesn't require validator status)
    • X, C, Q chains FAIL (require validator status for consensus)
  • Required Fields in Genesis: 
    "initialStakers": [
  {
    "nodeID": "NodeID-...",
    "rewardAddress": "lux1...",
    "delegationFee": 20000,
    "stakeAmount": 2000000000000000
  }
]
  • Solution: Generate genesis with all 5 node IDs in validator set

3. Genesis Generation Tool Needed

Problem: No automated way to create valid genesis

  • Manual Process Error-Prone:
    • Calculate Bech32 checksums
    • Format addresses correctly
    • Set proper stake amounts
    • Include all node IDs
  • Solution Created: /Users/z/work/lux/genesis/cmd/genesis-gen/main.go
    • Generates 5-validator genesis automatically
    • Proper Bech32 address encoding
    • Configurable network ID
    • Pre-funded accounts for testing
    • Outputs to stdout or file

Validator Setup Requirements

Bootstrap Chain Requirements

P-Chain (Platform Chain):

  • ✅ Bootstraps WITHOUT validator status
  • ✅ Can sync from network as non-validator
  • Special case: platform chain manages validators

X-Chain (Asset Exchange):

  • ❌ Requires validator status to participate
  • ❌ Won't bootstrap without being in validator set
  • Error: "node is not a validator"

C-Chain (Smart Contracts):

  • ❌ Requires validator status to participate
  • ❌ Won't bootstrap without being in validator set
  • EVM chain requires consensus participation

Q-Chain (Quantum-Resistant):

  • ❌ Requires validator status to participate
  • ❌ Won't bootstrap without being in validator set
  • New chain requires validator consensus

Validator Genesis Requirements

{
  "initialStakers": [
    {
      "nodeID": "NodeID-7Xhw2mDxuDS44j42TCB6U5579esbSt3Lg",
      "rewardAddress": "lux1wst8jt3z3fm9ce0z6akj3266zmgccdp03hjlaj",
      "delegationFee": 20000,
      "stakeAmount": 2000000000000000
    }
  ]
}

Critical Fields:

  • nodeID - From node's staking certificate
  • rewardAddress - Bech32-encoded with correct checksum
  • stakeAmount - Minimum 2000 LUX (2000000000000000 nLUX)
  • delegationFee - Commission percentage (20000 = 20%)

Solutions Implemented This Session

1. Fixed lux-cli Version References

File: /Users/z/work/lux/cli/pkg/constants/constants.go

// Before
DefaultNodeVersion      = "v2.0.0"
DefaultNetRunnerVersion = "v1.13.0"
BuildDir               = ".lux/bin/luxgo"

// After
DefaultNodeVersion      = "v1.20.4"
DefaultNetRunnerVersion = "v1.20.4"
BuildDir               = ".lux/bin/luxfi"

2. Created GitHub Release Workflow

File: /Users/z/work/lux/node/.github/workflows/release.yml

  • Triggers on tag push (v*)
  • Builds for darwin-amd64, darwin-arm64, linux-amd64, linux-arm64
  • Creates GitHub release with binaries
  • Generates SHA256 checksums
  • Auto-populates release notes

3. Fixed Default Port Configuration

File: /Users/z/work/lux/netrunner/local/network.go

// Changed base port from 9650 to 9630
const baseHTTPPort = 9630
const baseStakingPort = 9631

4. Created Genesis Generation Tool

File: /Users/z/work/lux/genesis/cmd/genesis-gen/main.go

  • Generates valid Bech32 addresses
  • Creates 5-validator genesis automatically
  • Configurable network ID
  • Pre-funded test accounts
  • Proper stake amounts and fees

5. Created New v1.20.4 Release

Actions Taken:

  • Tagged commit with complete WarpSet/WarpValidator code
  • Pushed tag to GitHub: git push origin v1.20.4
  • Automated workflow created GitHub release
  • Release includes all required implementations

Deployment Process (Updated)

Method 1: Netrunner (Recommended - Easiest)

# Build netrunner
cd /Users/z/work/lux/netrunner
go build -o bin/netrunner

# Start network (handles genesis automatically)
./bin/netrunner control start \
  --number-of-nodes=3 \
  --node-path=/Users/z/work/lux/node/build/luxd \
  --blockchain-specs='[{"vm_name":"qemuvm","genesis":"/path/to/genesis.json"}]'

Method 2: Genesis-Gen Tool + Manual Deployment

# Generate valid genesis
cd /Users/z/work/lux/genesis
go run cmd/genesis-gen/main.go \
  --network-id=12345 \
  --num-validators=5 \
  --output=/tmp/genesis.json

# Start nodes manually with generated genesis
/Users/z/work/lux/node/build/luxd \
  --network-id=12345 \
  --genesis=/tmp/genesis.json \
  --http-port=9630 \
  --staking-port=9631 \
  --data-dir=/tmp/node1

Method 3: Lux-CLI (After Version Fixes)

# Build latest lux-cli with version fixes
cd /Users/z/work/lux/cli
go build -o bin/lux

# Start network (now downloads correct v1.20.4 binary)
./bin/lux network start

Manual Deployment Checklist

When deploying manually (without netrunner):

  1. Generate Valid Genesis:

    go run genesis-gen/main.go --network-id=12345 --num-validators=5

  2. Verify Genesis Addresses:

    • All addresses have correct Bech32 checksums
    • InitialStakers array has 5 entries (one per node)
    • Stake amounts ≥ 2000 LUX

  3. Start Bootstrap Node First:

    luxd --network-id=12345 --genesis=/tmp/genesis.json \
     --http-port=9630 --staking-port=9631 \
     --data-dir=/tmp/node1 \
     --bootstrap-ips= --bootstrap-ids=

  4. Get Bootstrap Node ID:

    curl http://localhost:9630/v1/info | jq .result.nodeID

  5. Start Remaining Nodes:

    luxd --network-id=12345 --genesis=/tmp/genesis.json \
     --http-port=9640 --staking-port=9641 \
     --data-dir=/tmp/node2 \
     --bootstrap-ips=127.0.0.1:9631 \
     --bootstrap-ids=<node1-id>

  6. Verify All Chains Bootstrap:

    # Should show isBootstrapped: true for P, X, C, Q
curl http://localhost:9630/v1/info | jq

Error Messages and Solutions

"invalid checksum (expected..., got...)"

Cause: Manually created address without proper Bech32 encoding Solution: Use github.com/luxfi/crypto/address package

import "github.com/luxfi/crypto/address"
addr, _ := address.Format("lux", hrp, pubKeyHash)

"node is not a validator"

Cause: Node's NodeID not in genesis initialStakers array Solution: Include node's NodeID in genesis validator set

"initialStakers": [{"nodeID": "NodeID-7Xhw2mDxuDS44j42TCB6U5579esbSt3Lg", ...}]

"X-Chain not bootstrapping"

Cause: Node not configured as validator (X-Chain requires consensus participation) Solution: Restart with validator-enabled genesis containing node's ID

"download failed: release not found"

Cause: lux-cli trying to download non-existent version (v2.0.0) Solution: Update constants.go with DefaultNodeVersion = "v1.20.4"

"503 Service Unavailable" from netrunner

Cause: Network terminated prematurely due to node startup failure Solution: Check node binary path, port availability, genesis validity

Files Modified This Session

lux-cli Version Fixes

  • /Users/z/work/lux/cli/pkg/constants/constants.go

Release Workflow

  • /Users/z/work/lux/node/.github/workflows/release.yml (created)

Port Configuration

  • /Users/z/work/lux/netrunner/local/network.go
  • /Users/z/work/lux/netrunner/cmd/control/start/start.go

Genesis Tools

  • /Users/z/work/lux/genesis/cmd/genesis-gen/main.go (created)

Documentation

  • /Users/z/work/lux/netrunner/CLAUDE.md (this file)

Key Takeaways for Future Sessions

  1. Always verify GitHub release matches Git tag - Code can compile locally but be missing in published releases

  2. Use 963x port series - Not 964x (9630/9631 for first node)

  3. Validator status is critical - Only P-Chain bootstraps without it; X/C/Q require validator genesis

  4. Manual genesis is error-prone - Use genesis-gen tool or netrunner automatic generation

  5. Test with fresh data dirs - Corrupted state causes mysterious failures

  6. Bootstrap nodes must start first - Other nodes need bootstrap-ips/bootstrap-ids to connect

  7. Version constants matter - Wrong version in lux-cli breaks binary downloads

  8. Bech32 checksums are mandatory - Manual address creation always fails validation

Success Criteria for Local Network

✅ Achieved:

  • 5-node network starts successfully
  • P-Chain bootstraps (<60s)
  • All nodes visible to each other
  • HTTP APIs responding

❌ Remaining:

  • X-Chain bootstrap (requires validator genesis)
  • C-Chain bootstrap (requires validator genesis)
  • Q-Chain bootstrap (requires validator genesis)
  • Cross-chain transactions
  • Consensus participation

To Complete: Deploy with genesis-gen tool that includes all 5 node IDs as validators

Genesis cChainGenesis Format Fix (2025-11-16)

Issue

Error: could not unmarshal genesis JSON: json: cannot unmarshal object into Go struct field UnparsedConfig.cChainGenesis of type string

Root Cause: The cChainGenesis field in netrunner's default genesis was an object instead of a JSON string.

Location: /Users/z/work/lux/netrunner/network/default/genesis.json

Fix Applied: Converted cChainGenesis object to JSON string:

import json
with open('network/default/genesis.json', 'r') as f:
    genesis = json.load(f)
if isinstance(genesis['cChainGenesis'], dict):
    genesis['cChainGenesis'] = json.dumps(genesis['cChainGenesis'], separators=(',', ':'))
with open('network/default/genesis.json', 'w') as f:
    json.dump(genesis, f, indent=2)

Result: Nodes can now start successfully with netrunner default genesis

Peer Connectivity Requirements

Discovery: Bootstrap configuration in YAML manifests wasn't being passed to luxd command lines.

Required Flags:

  • --bootstrap-ips=127.0.0.1:9631 - Bootstrap node's staking port
  • --bootstrap-ids=NodeID-7Xhw2... - Bootstrap node's NodeID

Solution: Created /scripts/start-local-network.sh that properly configures bootstrap for all nodes

Verification:

# Check if bootstrap flags are in process command line
ps aux | grep luxd | grep 9640 | grep bootstrap-ips

Staking Certificate Reuse

Discovery: Netrunner has pre-configured staking certificates in /local/default/node1-5/

Files Per Node:

  • staking.crt - TLS certificate (determines NodeID)
  • staking.key - TLS private key
  • signer.key - BLS signing key (32 bytes)

NodeIDs Generated (from netrunner default certificates):

  • Node 1: NodeID-7Xhw2mDxuDS44j42TCB6U5579esbSt3Lg
  • Node 2: NodeID-MFrZFVCXPv5iCn6M9K6XduxGTYp891xXZ
  • Node 3: NodeID-NFBbbJ4qCmNaCzeW7sxErhvWqvEQMnYcN
  • Node 4: NodeID-GWPcbFJZFfZreETSoWjPimr846mXEKCtu
  • Node 5: NodeID-P7oB2McjBGgW2NXXWVYjV8JEDFoW9xDE5

Usage Pattern:

# Copy pre-configured keys to node data directory
for i in {1..5}; do
    mkdir -p $BASE/node$i
    cp $NETRUNNER/local/default/node$i/*.{crt,key} $BASE/node$i/
done

# Start node with explicit staking key paths
$LUXD --staking-tls-cert-file=$BASE/node1/staking.crt \
      --staking-tls-key-file=$BASE/node1/staking.key \
      --staking-signer-key-file=$BASE/node1/signer.key

Benefit: Ensures NodeIDs match netrunner default genesis validators

Lux CLI Network Architecture Proposal (2025-12-19)

Executive Summary

This document proposes a production-ready architecture for mainnet network management in the Lux CLI. The design focuses on proper validator key management, 5-node consensus with staking keys from ~/.lux/keys/, and clean separation from the legacy "local" development command.

Current State Analysis

What Exists

  1. networkcmd/ - Already has start, stop, clean, status commands with partial mainnet support
  2. localcmd/ - Minimal PoA development network (single-node, chain ID 1337) - TO BE REMOVED
  3. netrunner integration - CLI uses netrunner's gRPC server for network orchestration
  4. Key infrastructure - Keys exist at ~/.lux/keys/node-{0-4}/staking/ with staker.crt and staker.key
  5. Validator config - ~/.lux/keys/mainnet_validators.json contains P-chain addresses and private keys

Current Issues

  1. localcmd confusion - The lux local start command creates a single-node PoA network for development (chain ID 1337), which is conceptually different from the mainnet/testnet multi-validator network
  2. Missing proper staking key integration - Current StartMainnet() doesn't load the pre-generated staking keys from ~/.lux/keys/node-{0-4}/
  3. No snapshot management - Missing snapshot save and snapshot restore subcommands
  4. Incomplete node configuration - Nodes aren't configured with proper staking certificates

Proposed Architecture

1. Command Structure

lux network
    start [--mainnet|--testnet]   # Start 5-node network with validator keys
    stop                          # Stop network, preserve state
    status                        # Show network status
    clean                         # Clean all network data
    snapshot
        save <name>               # Save current state to named snapshot
        restore <name>            # Restore from named snapshot
        list                      # List available snapshots
        delete <name>             # Delete a snapshot

2. Remove localcmd/

The lux local command should be removed entirely. Rationale:

  • It creates confusion between "local development" and "local mainnet simulation"
  • The 1337 chain ID PoA network is superseded by proper multi-validator networks
  • Users should use lux network start for all local network needs
  • The --mainnet flag already provides mainnet genesis (network ID 96369)
  • The --testnet flag provides testnet genesis (network ID 96368)

Files to remove:

  • /Users/z/work/lux/cli/cmd/localcmd/local.go
  • Remove localcmd.NewCmd(app) from /Users/z/work/lux/cli/cmd/root.go (line 131)

3. Key Directory Structure (Already Exists)

~/.lux/keys/
    node-0/
        staking/
            staker.crt      # TLS certificate for staking
            staker.key      # TLS private key for staking
        bls/               # BLS keys for consensus signatures
        ec/                # EC keys
        mldsa/             # ML-DSA post-quantum keys
        rt/                # Corona keys
    node-1/ ... node-4/    # Same structure
    mainnet_validators.json  # P-chain addresses and private keys

4. Enhanced Network Start with Staking Keys

New types in pkg/network/types.go:

package network

// ValidatorConfig represents a single validator's configuration
type ValidatorConfig struct {
    NodeIndex    int
    StakingCert  string // Path to staker.crt
    StakingKey   string // Path to staker.key
    BLSKey       string // Path to BLS key
    NodeID       string // Computed from staking key
    PChainAddr   string // P-chain address for rewards
    HTTPPort     int    // HTTP API port
    StakingPort  int    // P2P staking port
}

// NetworkConfig represents the full network configuration
type NetworkConfig struct {
    NetworkID     uint32
    Validators    []ValidatorConfig
    GenesisPath   string
    RootDataDir   string
    PortBase      int
    DBEngine      string // pebble, leveldb, or badgerdb
}

// LoadValidatorConfigs loads all 5 validator configurations from disk
func LoadValidatorConfigs(keyPath string, portBase int) ([]ValidatorConfig, error) {
    validators := make([]ValidatorConfig, 5)
    for i := 0; i < 5; i++ {
        nodeDir := filepath.Join(keyPath, fmt.Sprintf("node-%d", i))
        stakingDir := filepath.Join(nodeDir, "staking")

        certPath := filepath.Join(stakingDir, "staker.crt")
        keyPath := filepath.Join(stakingDir, "staker.key")

        validators[i] = ValidatorConfig{
            NodeIndex:   i,
            StakingCert: certPath,
            StakingKey:  keyPath,
            HTTPPort:    portBase + (i * 2),      // 9630, 9632, 9634, 9636, 9638
            StakingPort: portBase + (i * 2) + 1,  // 9631, 9633, 9635, 9637, 9639
        }
    }
    return validators, nil
}

5. New Snapshot Commands

New file: /Users/z/work/lux/cli/cmd/networkcmd/snapshot.go:

package networkcmd

func newSnapshotCmd() *cobra.Command {
    cmd := &cobra.Command{
        Use:   "snapshot",
        Short: "Manage network snapshots",
    }
    cmd.AddCommand(newSnapshotSaveCmd())
    cmd.AddCommand(newSnapshotRestoreCmd())
    cmd.AddCommand(newSnapshotListCmd())
    cmd.AddCommand(newSnapshotDeleteCmd())
    return cmd
}

func newSnapshotSaveCmd() *cobra.Command {
    return &cobra.Command{
        Use:   "save <name>",
        Short: "Save current network state to a named snapshot",
        Args:  cobra.ExactArgs(1),
        RunE: func(cmd *cobra.Command, args []string) error {
            cli, _ := binutils.NewGRPCClient()
            _, err := cli.SaveSnapshot(ctx, args[0])
            return err
        },
    }
}

func newSnapshotRestoreCmd() *cobra.Command {
    return &cobra.Command{
        Use:   "restore <name>",
        Short: "Restore network from a named snapshot",
        Args:  cobra.ExactArgs(1),
        RunE: func(cmd *cobra.Command, args []string) error {
            cli, _ := binutils.NewGRPCClient()
            _, err := cli.LoadSnapshot(ctx, args[0], opts...)
            return err
        },
    }
}

6. Port Allocation Strategy

Node HTTP Port Staking Port
node-0 9630 9631
node-1 9632 9633
node-2 9634 9635
node-3 9636 9637
node-4 9638 9639

File Structure Changes Summary

Files to Remove

cmd/localcmd/local.go                    # Remove entire localcmd

Files to Modify

cmd/root.go                              # Remove localcmd.NewCmd(app) import and call
cmd/networkcmd/network.go                # Add snapshot subcommand
cmd/networkcmd/start.go                  # Integrate staking key loading
cmd/networkcmd/status.go                 # Enhance to show validator IDs

Files to Add

cmd/networkcmd/snapshot.go               # New snapshot save/restore/list/delete
pkg/network/types.go                     # New ValidatorConfig, NetworkConfig types
pkg/network/keys.go                      # New key loading utilities

Configuration Flow

User runs: lux network start --mainnet

1. Load validator keys from ~/.lux/keys/node-{0-4}/staking/
2. Validate all 5 validators have staker.crt and staker.key
3. Build per-node configurations with:
   - Staking TLS cert/key paths
   - Unique HTTP ports (9630, 9632, 9634, 9636, 9638)
   - Unique staking ports (9631, 9633, 9635, 9637, 9639)
   - BadgerDB database backend
4. Start netrunner gRPC server if not running
5. Call netrunner.Start() with:
   - Global config: network-id=96369, sybil-protection-enabled=true
   - Custom node configs for each validator
   - Genesis from luxfi/genesis package (network ID 96369)
6. Wait for all 5 validators to become healthy
7. Display endpoints table

User runs: lux network snapshot save checkpoint1

1. Connect to running netrunner
2. Call SaveSnapshot("checkpoint1")
3. Snapshot saved to ~/.lux/snapshots/anr-snapshot-checkpoint1/

User runs: lux network stop

1. Call SaveSnapshot("default-XXX") to preserve state
2. Kill netrunner server process
3. "Network stopped successfully"

User runs: lux network snapshot restore checkpoint1

1. Start netrunner server
2. Call LoadSnapshot("checkpoint1", opts...)
3. Wait for healthy
4. Display endpoints

Implementation Priority

Phase 1: Core Network Start (HIGH PRIORITY)

  1. Implement pkg/network/types.go and pkg/network/keys.go
  2. Update cmd/networkcmd/start.go to load staking keys
  3. Test 5-node mainnet startup with proper validator keys

Phase 2: Snapshot Management (HIGH PRIORITY)

  1. Add cmd/networkcmd/snapshot.go
  2. Register snapshot subcommand in network.go
  3. Test save/restore workflow

Phase 3: Cleanup (MEDIUM PRIORITY)

  1. Remove cmd/localcmd/local.go
  2. Remove from cmd/root.go
  3. Update documentation

Security Considerations

  1. Key Protection: Staking keys at ~/.lux/keys/ should have 0600 permissions
  2. No EWOQ Keys: As per CLAUDE.md, absolutely no EWOQ keys — use treasury key instead
  3. Real Genesis: Use luxfi/genesis package, never generate custom genesis
  4. Sybil Protection: Always enable sybil-protection-enabled for mainnet

RLP Block Import Files (IMPORTANT - Do Not Lose!)

State File Locations

All RLP files are at: ~/work/lux/state/rlp/

Chain Network File Path Size
Zoo Mainnet ~/work/lux/state/rlp/zoo-mainnet/zoo-mainnet-200200.rlp 1.3M
Zoo Testnet ~/work/lux/state/rlp/zoo-testnet/zoo-testnet-200201.rlp -
Lux (C-Chain) Mainnet ~/work/lux/state/rlp/lux-mainnet/lux-mainnet-96369.rlp 1.2G
Lux (C-Chain) Testnet ~/work/lux/state/rlp/lux-testnet/lux-testnet-96368.rlp -
SPC Mainnet ~/work/lux/state/rlp/spc-mainnet/spc-mainnet-36911.rlp -

Chain IDs

  • Zoo Mainnet: 200200
  • Zoo Testnet: 200201
  • Lux Mainnet (C-Chain): 96369
  • Lux Testnet (C-Chain): 96368
  • SPC Mainnet: 36911

Import Commands

# Import Zoo mainnet blocks (~1.3M blocks)
curl -X POST http://127.0.0.1:9650/v1/bc/Zoo/rpc \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"debug_importRLPBlocks","params":["'$(base64 -i ~/work/lux/state/rlp/zoo-mainnet/zoo-mainnet-200200.rlp)'"],"id":1}'

# Import C-chain mainnet blocks (~1.2G)
curl -X POST http://127.0.0.1:9650/v1/bc/C/rpc \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"debug_importRLPBlocks","params":["'$(base64 -i ~/work/lux/state/rlp/lux-mainnet/lux-mainnet-96369.rlp)'"],"id":1}'

Key Details

  • RLP files contain encoded blocks for historical chain state replay
  • Import order: Import Zoo first (smaller), then C-Chain (larger)
  • Network must be running with Zoo chain discovered before import

Network ID vs Chain ID Fix (2025-12-23)

Critical Bug Fixed

Problem: Multiple functions in genesis_config.go were using C-Chain IDs (configs.MainnetChainID/configs.TestnetChainID = 96369/96368) when they should use Network IDs (constants.MainnetID/constants.TestnetID = 1/2).

Impact: Networks would fail to start with "network ID mismatch: expected 2, got 96368" or similar errors.

Constants Clarification

Constant Value Purpose
constants.MainnetID 1 P2P Network ID for mainnet
constants.TestnetID 2 P2P Network ID for testnet
constants.CustomID 1337 P2P Network ID for local dev
configs.MainnetChainID 96369 EVM Chain ID for C-Chain mainnet
configs.TestnetChainID 96368 EVM Chain ID for C-Chain testnet

Files Fixed

/Users/z/work/lux/netrunner/local/genesis_config.go:

Function Before After
NewMainnetConfigWithKeys() (line 644) configs.MainnetChainID constants.MainnetID
NewTestnetConfigWithKeys() (line 652) configs.TestnetChainID constants.TestnetID
NewMainnetConfigFromMnemonic() (line 985) configs.MainnetChainID constants.MainnetID
NewTestnetConfigFromMnemonic() (line 990) configs.TestnetChainID constants.TestnetID

Earlier fixes in same session (from conversation summary):

  • Line 58: configs.LocalID → configs.CustomID
  • Line 335: configs.LocalID → configs.CustomID
  • Line 996: configs.LocalID → configs.CustomID

/Users/z/work/lux/netrunner/network/config.go:

  • Line 125: constants.LocalID → constants.CustomID

/Users/z/work/lux/netrunner/server/network.go:

  • Line 152: luxd_constants.LocalID → luxd_constants.CustomID
  • Lines 174, 182: Changed from hardcoded 96369/96368 to luxd_constants.MainnetID/TestnetID

Zoo Chain Deployment Success

After fixes, successfully deployed Zoo chain on mainnet:

  • Network ID: 1 (mainnet)
  • Zoo Chain ID: 200200 (0x30e08)
  • Genesis Hash: 0x7c548af47de27560779ccc67dda32a540944accc71dac3343da3b9cd18f14933
  • Treasury: ~500M ZOO at 0x9011E888251AB053B7bD1cdB598Db4f9DEd94714
  • RPC Endpoint: http://localhost:9630/v1/bc/zoo/rpc

RLP Block Import Limitation

Discovery: The EVM plugin doesn't support admin_importChain RPC method. Blocks in EVM chains come through Lux consensus, not direct import.

Available RPC modules on Zoo chain: eth, net, rpc, web3 (no admin or debug)

Workaround: For testing, deploy fresh AMM contracts since genesis state is identical.

Example Usage

// Correct - uses Network ID
cfg, err := local.NewMainnetConfigFromMnemonic(binaryPath, 5)

// Wrong - would use Chain ID instead of Network ID
// Previously: local.NewConfigFromMnemonic(binaryPath, configs.MainnetChainID, 5)

Conclusion

This architecture provides a clean, production-ready approach to mainnet network management:

  • Clear command structure: lux network {start|stop|status|clean|snapshot}
  • Proper key integration: Uses existing staking keys from ~/.lux/keys/
  • State management: Snapshot save/restore for reproducible testing
  • No confusion: Removes legacy lux local command entirely
  • Mainnet-first: Defaults to mainnet configuration with 5 validators

Native BadgerDB Snapshot Fix (2026-01-22)

Problem

Snapshot creation was using 75GB directory copies instead of efficient native BadgerDB incremental backups.

Root Cause

  1. Node backup service not initialized: DataDir wasn't passed to admin service config in node/node.go
  2. CLI stop command order wrong: Tried to snapshot databases directly while nodes had exclusive locks

Files Modified

/Users/z/work/lux/node/node/node.go: Added DataDir to admin service config to enable backup service:

service := admin.New(admin.Config{
    // ... other fields ...
    DataDir: n.Config.DatabaseConfig.Path,  // ADDED
})

/Users/z/work/lux/netrunner/local/snapshot.go: Cleaned up to single unified approach using native BadgerDB via admin.snapshot API.

  • Removed legacy copyDir function
  • Removed hotSnapshotManifest types and duplicate functions
  • Reduced from 984 lines to 629 lines (36% reduction)

/Users/z/work/lux/cli/cmd/networkcmd/stop.go: Fixed to use gRPC method (admin.snapshot API) for hot snapshots:

// Before (wrong - tried direct file access while nodes running)
if err := saveNetworkNative(stopNetworkType, snapshotName, useIncremental); err != nil {
    if err := saveNetworkForType(stopNetworkType); err != nil { ... }
}

// After (correct - uses admin.snapshot API via gRPC)
if err := saveNetworkForType(stopNetworkType); err != nil {
    ux.Logger.PrintToUser("Warning: failed to save snapshot: %v", err)
}

Results

  • Before: 75GB snapshots (directory copy)
  • After: 132KB snapshots (native BadgerDB incremental)
  • Reduction: 99.9%+

Verified Functionality (2026-01-22)

Feature Status Notes
5-node testnet ✅ Ports 9640-9648
5-node mainnet ✅ Ports 9630-9638
Hot snapshots ✅ Via admin.snapshot API
Incremental backups ✅ 5KB per node compressed
Resume from data ✅ Auto-detects existing run
Track all chains ✅ track-chains="all"
All 11 chains ✅ P,C,X,Q,A,B,T,Z,G,K,D

Snapshot Format (v2)

{
    "version": 2,
    "network": "testnet",
    "timestamp": 1769140260,
    "created_at": "2026-01-23T03:51:00Z",
    "nodes": {
        "node1": {
            "db_version": 11,
            "incremental_from": 0,
            "backup_file": "node1.backup.zst",
            "compressed_size": 5362
        }
        // ... node2-node5
    }
}

Commands Reference

# Start 5-node testnet
lux network start --testnet

# Stop with snapshot
lux network stop --testnet --force --snapshot-name my-snapshot

# Resume from snapshot (uses existing data)
lux network start --testnet

# Check snapshot
ls -la ~/.lux/snapshots/lux-snapshot-<name>/
cat ~/.lux/snapshots/lux-snapshot-<name>/manifest.json
Reference in New Issue View Git Blame Copy Permalink