netrunner/MULTI_NETWORK_ARCHITECTURE.md

Multi-Network Architecture Reference

Overview

The Lux CLI and netrunner support running up to 4 independent primary networks simultaneously:

• mainnet (Network ID: 96369)
• testnet (Network ID: 96368)
• devnet (Network ID: 96370)
• local (Network ID: 1337)

Each network operates in complete isolation with its own:

• gRPC control server
• Node processes (5 validators default)
• Data directories
• Plugin directories
• State files

---

1. Complete Port Allocation Table

Control Layer (gRPC Servers)

Network	gRPC Server	gRPC Gateway	Description
mainnet	8369	8379	Primary production network
testnet	8370	8380	Test network
devnet	8371	8381	Development network
local	8372	8382	Local ephemeral network

Node Layer (per network, 5 nodes default)

Each network uses a base port with 2 ports per node (HTTP API + Staking P2P).

Network	Base Port	Node 1	Node 2	Node 3	Node 4	Node 5
mainnet	9630	9630/9631	9632/9633	9634/9635	9636/9637	9638/9639
testnet	9640	9640/9641	9642/9643	9644/9645	9646/9647	9648/9649
devnet	9650	9650/9651	9652/9653	9654/9655	9656/9657	9658/9659
local	9660	9660/9661	9662/9663	9664/9665	9666/9667	9668/9669

Port Formula

HTTP_PORT  = BASE_PORT + (node_index * 2)
STAKING_PORT = BASE_PORT + (node_index * 2) + 1

where node_index = 0..4 for nodes 1-5

Full Port Matrix (All Networks, All Nodes)

Network   Node  HTTP   Staking  gRPC   Gateway
-------   ----  -----  -------  -----  -------
mainnet   1     9630   9631     8369   8379
mainnet   2     9632   9633       |      |
mainnet   3     9634   9635       |      |
mainnet   4     9636   9637       |      |
mainnet   5     9638   9639       v      v

testnet   1     9640   9641     8370   8380
testnet   2     9642   9643       |      |
testnet   3     9644   9645       |      |
testnet   4     9646   9647       |      |
testnet   5     9648   9649       v      v

devnet    1     9650   9651     8371   8381
devnet    2     9652   9653       |      |
devnet    3     9654   9655       |      |
devnet    4     9656   9657       |      |
devnet    5     9658   9659       v      v

local     1     9660   9661     8372   8382
local     2     9662   9663       |      |
local     3     9664   9665       |      |
local     4     9666   9667       |      |
local     5     9668   9669       v      v

---

2. Directory Structure

~/.lux/
├── bin/                              # Shared binaries
│   └── luxd                          # Node binary
│
├── plugins/                          # VM plugin management
│   └── current/                      # Active VM plugins (symlinks)
│       ├── srEXiWaHuhNyGwPUi444Tu47ZEDwxTWrbQiuD7FmgSAQ6X7Dy  # evm
│       └── <other-vm-ids>
│
├── chains/                           # Canonical chain definitions
│   └── <chain-name>/
│       ├── genesis.json              # Chain genesis
│       ├── config.json               # Chain config
│       └── sidecar.json              # Chain metadata
│
├── keys/                             # Cryptographic keys
│   ├── staker.crt                    # Staking certificate
│   ├── staker.key                    # Staking key
│   └── *.pk                          # Private keys
│
├── runs/                             # Per-network run directories
│   ├── mainnet/
│   │   └── run_20251223_143022/      # Timestamped run
│   │       ├── node1/
│   │       │   ├── db/               # Node database
│   │       │   ├── logs/             # Node logs
│   │       │   ├── staking.crt       # Node certificate
│   │       │   ├── staking.key       # Node TLS key
│   │       │   ├── signer.key        # BLS signer key
│   │       │   └── config.json       # Node config
│   │       ├── node2/
│   │       ├── node3/
│   │       ├── node4/
│   │       ├── node5/
│   │       └── genesis.json          # Network genesis
│   │
│   ├── testnet/
│   │   └── run_<timestamp>/          # Same structure as mainnet
│   │
│   ├── devnet/
│   │   └── run_<timestamp>/
│   │
│   └── local/
│       └── run_<timestamp>/
│
├── server/                           # gRPC server logs
│   ├── mainnet/
│   │   └── server_<timestamp>/
│   │       └── lux-server.log
│   ├── testnet/
│   ├── devnet/
│   └── local/
│
├── snapshots/                        # Network snapshots
│   └── anr-snapshot-<name>/
│
└── config/                           # CLI configuration
    ├── mainnet_network_state.json    # Mainnet state
    ├── testnet_network_state.json    # Testnet state
    ├── devnet_network_state.json     # Devnet state
    ├── local_network_state.json      # Local state
    ├── mainnet.run                   # Mainnet gRPC PID file
    ├── testnet.run                   # Testnet gRPC PID file
    ├── devnet.run                    # Devnet gRPC PID file
    └── local.run                     # Local gRPC PID file

---

3. Process Hierarchy

Starting All Networks

User Shell
│
├── lux network start --mainnet
│   └── lux-server (gRPC @ 8369)          # Backend controller
│       └── netrunner.Start()
│           ├── luxd node1 (9630/9631)    # Validator 1
│           ├── luxd node2 (9632/9633)    # Validator 2
│           ├── luxd node3 (9634/9635)    # Validator 3
│           ├── luxd node4 (9636/9637)    # Validator 4
│           └── luxd node5 (9638/9639)    # Validator 5
│
├── lux network start --testnet
│   └── lux-server (gRPC @ 8370)
│       └── netrunner.Start()
│           └── [5 luxd nodes @ 9640-9649]
│
├── lux network start --devnet
│   └── lux-server (gRPC @ 8371)
│       └── netrunner.Start()
│           └── [5 luxd nodes @ 9650-9659]
│
└── lux network start (local)
    └── lux-server (gRPC @ 8372)
        └── netrunner.Start()
            └── [5 luxd nodes @ 9660-9669]

Process Types

Process	Count per Network	Role
lux-server	1	gRPC control server, manages network lifecycle
luxd	5 (default)	Validator node, runs consensus

---

4. State File Locations

Run Files (gRPC Server PID)

Network	Run File	Contents
mainnet	~/.lux/config/mainnet.run	{"pid": 12345, "networkType": "mainnet", "grpcPort": 8369}
testnet	~/.lux/config/testnet.run	{"pid": 12346, "networkType": "testnet", "grpcPort": 8370}
devnet	~/.lux/config/devnet.run	{"pid": 12347, "networkType": "devnet", "grpcPort": 8371}
local	~/.lux/config/local.run	{"pid": 12348, "networkType": "local", "grpcPort": 8372}

Network State Files

Network	State File	Purpose
mainnet	~/.lux/config/mainnet_network_state.json	Tracks deployed chains, endpoints
testnet	~/.lux/config/testnet_network_state.json	Tracks deployed chains, endpoints
devnet	~/.lux/config/devnet_network_state.json	Tracks deployed chains, endpoints
local	~/.lux/config/local_network_state.json	Tracks deployed chains, endpoints

State File Format

{
  "networkType": "mainnet",
  "networkID": 96369,
  "httpPort": 9630,
  "grpcPort": 8369,
  "gatewayPort": 8379,
  "rootDataDir": "~/.lux/runs/mainnet/run_20251223_143022"
}

---

5. Shared vs Isolated Resources

Shared Resources

Resource	Path	Description
luxd binary	~/.lux/bin/luxd	Single binary, shared by all networks
VM plugins	~/.lux/plugins/current/	Shared VM binaries (by VM ID)
Chain definitions	~/.lux/chains/<name>/	Genesis, config, sidecar for each chain
Keys	~/.lux/keys/	Cryptographic keys

Isolated Resources

Resource	Path Pattern	Description
Node databases	~/.lux/runs/<network>/run_*/node*/db/	Per-network, per-node state
Node logs	~/.lux/runs/<network>/run_*/node*/logs/	Per-network, per-node logs
Node config	~/.lux/runs/<network>/run_*/node*/config.json	Per-network node configuration
Genesis	~/.lux/runs/<network>/run_*/genesis.json	Per-network genesis
gRPC PID	~/.lux/config/<network>.run	Per-network control process
Network state	~/.lux/config/<network>_network_state.json	Per-network state

---

6. Critical Code Changes Needed

6.1 Current State (Already Implemented)

The following are already implemented:

1. Centralized Port Constants (luxfi/constants/server.go):

   • GetGRPCPorts(networkType) - Returns network-specific gRPC ports
   • GetNetworkPorts(networkType) - Returns full port configuration

2. Network-Aware gRPC Server (cli/pkg/binutils/processes.go):

   • StartServerProcessForNetwork(app, networkType) - Starts network-specific server
   • IsServerProcessRunningForNetwork(app, networkType) - Checks server status
   • KillgRPCServerProcessForNetwork(app, networkType) - Stops specific server

3. Network-Aware CLI (cli/cmd/networkcmd/start.go):

   • --mainnet, --testnet flags select network type
   • startPublicNetwork(cfg) handles network-specific configuration

6.2 Changes Required for Full Multi-Network Support

A. Port Base Flag Fix

File: cli/cmd/networkcmd/start.go

Current behavior uses portBase flag defaulting to 9630 for all networks. Need to respect network-specific defaults:

// Current (line 456-459)
effectivePortBase := cfg.portBase
if effectivePortBase == 0 {
    effectivePortBase = 9630  // Wrong: same for all networks
}

// Required
effectivePortBase := cfg.portBase
if effectivePortBase == 0 {
    effectivePortBase = constants.GetNetworkPorts(cfg.networkName).NodeBase
}

B. Network ID in Configuration

File: netrunner/server/network.go

Ensure network ID is properly propagated from config:

// Line 173-191 - Already handles this via switch on networkID
switch networkID {
case 96369: // LUX Mainnet
    cfg, err = local.NewMainnetConfigFromMnemonic(...)
case 96368: // LUX Testnet
    cfg, err = local.NewTestnetConfigFromMnemonic(...)
default:
    cfg, err = local.NewDefaultConfigNNodes(...)
}

C. Run Directory Isolation

File: cli/pkg/chain/run.go (needs creation or modification)

Ensure EnsureNetworkRunDir is used consistently:

func EnsureNetworkRunDir(baseDir, networkName string) (string, error) {
    // Creates: <baseDir>/runs/<networkName>/run_<timestamp>/
    // Reuses existing run if node data present
}

D. Devnet Support

File: cli/cmd/networkcmd/start.go

Add --devnet flag and handler:

var devnet bool

cmd.Flags().BoolVar(&devnet, "devnet", false, "start a devnet node with 5 validators")

func StartDevnet() error {
    return startPublicNetwork(networkConfig{
        networkID:   96370,
        networkName: "devnet",
        portBase:    9650,
    })
}

---

7. Testing Strategy for Parallel Operation

7.1 Unit Tests

File: cli/pkg/binutils/constants_test.go (exists)

func TestGetGRPCPortsNoOverlap(t *testing.T) {
    networks := []string{"mainnet", "testnet", "devnet", "local"}
    ports := make(map[int]string)
    
    for _, net := range networks {
        p := GetGRPCPorts(net)
        if existing, ok := ports[p.Server]; ok {
            t.Errorf("Port %d used by both %s and %s", p.Server, existing, net)
        }
        ports[p.Server] = net
        ports[p.Gateway] = net
    }
}

7.2 Integration Tests

Test Scenario 1: Start All Networks

#!/bin/bash
# Start all networks in parallel
lux network start --mainnet &
lux network start --testnet &
lux network start --devnet &
lux network start &  # local

# Wait for all to become healthy
sleep 60

# Verify each network is accessible
curl http://localhost:9630/ext/health  # mainnet
curl http://localhost:9640/ext/health  # testnet
curl http://localhost:9650/ext/health  # devnet
curl http://localhost:9660/ext/health  # local

Test Scenario 2: Deploy Chain to Each Network

# Deploy same chain definition to each network
lux chain create mychain --evm

lux chain deploy mychain --network mainnet
lux chain deploy mychain --network testnet
lux chain deploy mychain --network devnet
lux chain deploy mychain --network local

# Verify chain is deployed on each
lux chain list --network mainnet
lux chain list --network testnet
lux chain list --network devnet
lux chain list --network local

Test Scenario 3: Stop/Start Individual Networks

# Stop only testnet
lux network stop --testnet

# Verify other networks still running
curl http://localhost:9630/ext/health  # mainnet - should succeed
curl http://localhost:9640/ext/health  # testnet - should fail
curl http://localhost:9650/ext/health  # devnet - should succeed

# Restart testnet
lux network start --testnet

# Verify testnet back up
curl http://localhost:9640/ext/health  # should succeed

7.3 Stress Test

# Create 20 chains across 4 networks (5 each)
for net in mainnet testnet devnet local; do
    for i in {1..5}; do
        lux chain create chain-${net}-${i} --evm
        lux chain deploy chain-${net}-${i} --network ${net} &
    done
done

wait

# Verify all chains deployed
for net in mainnet testnet devnet local; do
    count=$(lux chain list --network ${net} | grep -c "chain-${net}")
    if [ "$count" -ne 5 ]; then
        echo "FAIL: Expected 5 chains on ${net}, got ${count}"
    fi
done

7.4 Resource Isolation Test

# Start mainnet with specific data
lux network start --mainnet

# Create transactions on mainnet
# ...

# Stop mainnet, start testnet
lux network stop --mainnet
lux network start --testnet

# Verify testnet has clean state (no mainnet data)
# Verify mainnet data still exists on disk

# Restart mainnet, verify state persisted
lux network start --mainnet

---

8. CLI Commands Reference

Network Management

# Start networks
lux network start                    # Start local network
lux network start --mainnet          # Start mainnet (96369)
lux network start --testnet          # Start testnet (96368)
lux network start --devnet           # Start devnet (96370)

# Stop networks
lux network stop                     # Stop local network
lux network stop --mainnet           # Stop mainnet
lux network stop --testnet           # Stop testnet
lux network stop --devnet            # Stop devnet

# Status
lux network status                   # Status of local network
lux network status --mainnet         # Status of mainnet
lux network status --all             # Status of all networks

Chain Deployment

# Deploy to specific network
lux chain deploy <chain> --network mainnet
lux chain deploy <chain> --network testnet
lux chain deploy <chain> --network devnet
lux chain deploy <chain> --network local

---

9. Environment Variables

Variable	Description	Example
LUX_NETWORK_TYPE	Network type for backend	mainnet
LUX_MNEMONIC	HD wallet seed for key derivation	word1 word2 ...
LUX_PRIVATE_KEY	Direct private key (hex)	0x123...

---

10. Troubleshooting

Port Conflict Detection

# Check for port conflicts
for port in 8369 8370 8371 8372 8379 8380 8381 8382; do
    if lsof -i :$port > /dev/null 2>&1; then
        echo "Port $port in use"
    fi
done

Process Cleanup

# Kill all gRPC servers
pkill -f "lux-server"

# Kill all luxd nodes
pkill -f "luxd"

# Clean run files
rm -f ~/.lux/config/*.run

# Clean state files (WARNING: loses deployed chain info)
rm -f ~/.lux/config/*_network_state.json

Log Locations

Network	Server Log	Node Logs
mainnet	~/.lux/server/mainnet/*/lux-server.log	~/.lux/runs/mainnet/*/node*/logs/
testnet	~/.lux/server/testnet/*/lux-server.log	~/.lux/runs/testnet/*/node*/logs/
devnet	~/.lux/server/devnet/*/lux-server.log	~/.lux/runs/devnet/*/node*/logs/
local	~/.lux/server/local/*/lux-server.log	~/.lux/runs/local/*/node*/logs/

---

Document Version: 1.0.0 Last Updated: 2025-12-23 Compatible with: CLI v1.16.x, Netrunner v1.x.x