luxfi/node
1
0
Fork 0
mirror of https://github.com/luxfi/node.git synced 2026-07-27 03:39:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

node: ZAP-native everywhere — kill every //go:build grpc path

Browse Source 
The companion commit `7496606282` retired the gRPC fallback inside
vms/rpcchainvm/. This commit closes the loop by deleting every
remaining `//go:build grpc` file under node/, removing the dual-build
plumbing entirely. ZAP is the only wire protocol; there is no
`-tags=grpc` opt-in.

Per the "one and only one way to do everything" mandate, the
backwards-compat scaffolding (gRPC adapters, protoc stubs, connect-go
example handlers, OTLP gRPC exporter, x/sync gRPC sync engine,
keystore-over-gRPC client/server, rpcwarp gRPC signer, gRPC alias
reader) is removed forward-only — no aliases, no deprecation period.

Deletions (84 files):
  - proto/pb/{aliasreader,http,io,keystore,message,messenger,net,p2p,
    platformvm,rpcdb,sdk,sender,sharedmemory,signer,sync,
    validatorstate,vm,warp}/ — protoc stubs (entire tree)
  - proto/{p2p,platformvm,sync,vm}/*_grpc.go — gRPC type re-exports
  - db/rpcdb/{grpc_server,grpc_client,grpc_test}.go — rpcdb gRPC adapter
  - service/keystore/rpckeystore/ — keystore-over-gRPC (dead consumer)
  - x/sync/ — entire merkledb sync engine (100% gRPC-tagged)
  - internal/ids/rpcaliasreader/ — gRPC alias reader (dead consumer)
  - connectproto/ — connect-go XSVM ping handler scaffolding
  - vms/platformvm/warp/rpcwarp/{client,server}.go — gRPC warp signer
  - vms/platformvm/network/warp.go — protobuf-based warp justification
    handler (warp_zap.go retains the no-op verifier consumers expect)
  - vms/components/message/message_grpc.go + message_test.go
  - vms/example/xsvm/api/ping.go + vm_http_grpc.go + cmd/{run,xsvm}/
  - wallet/network/primary/examples/sign-l1-validator-* (5 dead
    example main packages)
  - trace/{exporter_grpc,exporter_type,exporter_type_test,noop,tracer}.go
    (OTLP gRPC exporter + duplicate Tracer types — trace_zap.go has
    the canonical Tracer interface + no-op tracer)
  - message/bft_grpc.go (Simplex BFT wrapper)

Edits (9 files): every surviving `_zap.go` drops its `//go:build !grpc`
constraint (the files are unconditional now) and drops stale
"ZAP version" / "ZAP mode" inline comments.

Doc updates:
  - LLM.md ZAP Transport section: remove the `-tags=grpc` opt-in
    language. Replace with "ZAP is the only wire protocol... there is
    one and only one way". Update Latest Tag to v1.26.31. Update the
    rpcdb topology section to reflect single-adapter state.

go.mod: connectrpc.com/connect, connectrpc.com/grpcreflect,
otlptracegrpc moved out of direct deps. google.golang.org/grpc +
protobuf demoted to indirect (still pulled transitively via luxfi/dex).

Verified:
  - `go build ./...` (default, no tags) clean
  - `go test ./db/rpcdb/... ./trace/... ./message/... ./vms/rpcchainvm/...
    ./vms/components/message/... ./vms/platformvm/network/...
    ./vms/example/xsvm/... ./service/keystore/... ./proto/...` clean
  - `grep -rln '//go:build grpc' --include='*.go' node/` returns zero
  - `grep -rln '//go:build !grpc' --include='*.go' node/` returns zero
  - `grep -rln 'google.golang.org/grpc' --include='*.go' node/` returns
    zero (only transitive deps remain in go.sum)

Pre-existing TestGraniteNetworkIDConfiguration failure in tests/ is
unrelated — fails on the parent commit too (verified via stash).
This commit is contained in:
Hanzo AI
2026-05-16 17:23:26 -07:00
parent 7496606282
commit 43f7aa2b03
95 changed files with 57 additions and 10480 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LLM.md
+23 -24
View File
@@ -8,7 +8,7 @@ Lux blockchain node implementation - a high-performance, multi-chain blockchain
**Key Context:**
- Original Lux Network node — NOT a fork
- Latest Tag: v1.26.12
- Latest Tag: v1.26.31
- Network ID: 96369 (Lux Mainnet), 96368 (Testnet), 96370 (Devnet)
- Go Version: 1.26.1+
- Database: ZapDB (primary, default)
@@ -274,31 +274,33 @@ Located in `vms/thresholdvm/fhe/`:
| Gateway | `0x0200000000000000000000000000000000000083` |
### ZAP Transport (Zero-Copy App Proto)
ZAP is the default high-performance binary wire protocol for VM<->Node communication.
gRPC support is available via build tag for testing/compatibility.
ZAP is the only wire protocol for VM<->Node communication. The gRPC
fallback (and its `-tags=grpc` opt-in) was retired in v1.26.31 along
with every `//go:build grpc` file under `node/`. There is one and only
one way to talk to a Chain VM: ZAP.
**Build Tags:**
**Build:**
```bash
go build # ZAP only (default, production)
go build -tags=grpc # gRPC support (for testing/compatibility)
go build # ZAP only — there are no build tags
```
**Key Packages:**
- `github.com/luxfi/api/zap` - Core wire protocol and message types (Layer A)
- `github.com/luxfi/proto/rpcdb` - rpcdb service spec / data carriers (Layer B)
- `github.com/luxfi/node/db/rpcdb` - rpcdb Service + ZAP/gRPC transport adapters (Layer C)
- `github.com/luxfi/vm/rpc/sender` - p2p.Sender over ZAP/gRPC
- `vms/rpcchainvm/sender/` - Node-side sender implementation
- `vms/platformvm/warp/zwarp/` - ZAP-based warp signing client/server
- `github.com/luxfi/api/zap` — Core wire protocol and message types (Layer A)
- `github.com/luxfi/protocol/rpcdb` — rpcdb service spec / data carriers (Layer B)
- `github.com/luxfi/node/db/rpcdb` — rpcdb Service + ZAP transport adapter (Layer C)
- `vms/rpcchainvm/sender/` — Node-side `p2p.Sender` over ZAP
- `vms/rpcchainvm/zap/` — ChainVM client/server over ZAP
- `vms/platformvm/warp/zwarp/` — Warp signing over ZAP
**rpcdb Layered Topology (post-2026-05 reorg):**
- Layer A — wire framing: `github.com/luxfi/api/zap` (independent module)
- Layer B — rpcdb service spec: `github.com/luxfi/proto/rpcdb` (transport-agnostic data carriers)
- Layer C — rpcdb impl: `node/db/rpcdb/{service.go, grpc_server.go, zap_server.go}`
**rpcdb Layered Topology:**
- Layer A — wire framing: `github.com/luxfi/api/zap`
- Layer B — rpcdb service spec: `github.com/luxfi/protocol/rpcdb`
- Layer C — rpcdb impl: `node/db/rpcdb/{service.go, zap_server.go}`
- `service.go` — transport-neutral `Service` wrapping `database.Database`
- `zap_server.go` (default) — ZAP transport adapter (used by cevm)
- `grpc_server.go` (`-tags=grpc`) — gRPC transport adapter
- One Service, many transport adapters. Adding a transport = new file wrapping `*Service`.
- `zap_server.go` — ZAP transport adapter (only adapter)
- One Service, one transport. The dual-adapter pattern stays available
for future transports (each is a new file wrapping `*Service`), but
ZAP is the only one shipping.
**Wire Protocol Format:**
```
@@ -313,11 +315,8 @@ go build -tags=grpc # gRPC support (for testing/compatibility)
**Sender Usage:**
```go
// ZAP transport (default)
// ZAP transport — the only transport
s := sender.ZAP(zapConn)
// gRPC transport (requires -tags=grpc build)
s := sender.GRPC(senderpb.NewSenderClient(grpcConn))
```
**Warp over ZAP:**
@@ -459,7 +458,7 @@ go test -v -run "TestHybrid" ./node/network/dialer/... -count=1
### 1. P2P Sender Interface
Node's rpcchainvm implements `p2p.Sender` (from `github.com/luxfi/p2p`) for cross-chain messaging.
The `sender` package is a gRPC implementation of `p2p.Sender`.
The `sender` package is the ZAP-native implementation of `p2p.Sender`.
### 2. Chain Tracking
Nodes don't automatically track chains. Use:
connectproto/buf.gen.yaml
-8
View File
@@ -1,8 +0,0 @@
version: v1
plugins:
- name: go
out: pb
opt: paths=source_relative
- plugin: connect-go
out: pb
opt: paths=source_relative
connectproto/buf.lock
-7
View File
@@ -1,7 +0,0 @@
# Generated by buf. DO NOT EDIT.
version: v1
deps:
- remote: buf.build
owner: metric
repository: client-model
commit: 1d56a02d481a412a83b3c4984eb90c2e
connectproto/buf.yaml
-27
View File
@@ -1,27 +0,0 @@
version: v1
name: buf.build/luxfi/lux
build:
excludes:
# for golang we handle metric as a buf dep so we exclude it from generate, this proto
# file is required by languages such as rust.
- io/metric
breaking:
use:
- FILE
deps:
- buf.build/metric/client-model
lint:
use:
- STANDARD
except:
- SERVICE_SUFFIX # service requirement of <name>+Service
- RPC_REQUEST_STANDARD_NAME # explicit <rpc>+Request naming
- RPC_RESPONSE_STANDARD_NAME # explicit <rpc>+Response naming
- PACKAGE_VERSION_SUFFIX # versioned naming <service>.v1beta
# allows RPC requests or responses to be google.protobuf.Empty messages. This can be set if you
# want to allow messages to be void forever, that is they will never take any parameters.
rpc_allow_google_protobuf_empty_requests: true
rpc_allow_google_protobuf_empty_responses: true
# allows the same message type to be used for a single RPC's request and response type.
# TODO: this should not be tolerated and if it is only perscriptivly.
rpc_allow_same_request_response: true
connectproto/pb/xsvm/service.pb.go
-288
View File
@@ -1,288 +0,0 @@
//go:build grpc
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.35.1
// protoc (unknown)
// source: xsvm/service.proto
package xsvm
import (
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
)
const (
// Verify that this generated code is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
// Verify that runtime/protoimpl is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)
type PingRequest struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Message string `protobuf:"bytes,1,opt,name=message,proto3" json:"message,omitempty"`
}
func (x *PingRequest) Reset() {
*x = PingRequest{}
mi := &file_xsvm_service_proto_msgTypes[0]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *PingRequest) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*PingRequest) ProtoMessage() {}
func (x *PingRequest) ProtoReflect() protoreflect.Message {
mi := &file_xsvm_service_proto_msgTypes[0]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use PingRequest.ProtoReflect.Descriptor instead.
func (*PingRequest) Descriptor() ([]byte, []int) {
return file_xsvm_service_proto_rawDescGZIP(), []int{0}
}
func (x *PingRequest) GetMessage() string {
if x != nil {
return x.Message
}
return ""
}
type PingReply struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Message string `protobuf:"bytes,1,opt,name=message,proto3" json:"message,omitempty"`
}
func (x *PingReply) Reset() {
*x = PingReply{}
mi := &file_xsvm_service_proto_msgTypes[1]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *PingReply) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*PingReply) ProtoMessage() {}
func (x *PingReply) ProtoReflect() protoreflect.Message {
mi := &file_xsvm_service_proto_msgTypes[1]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use PingReply.ProtoReflect.Descriptor instead.
func (*PingReply) Descriptor() ([]byte, []int) {
return file_xsvm_service_proto_rawDescGZIP(), []int{1}
}
func (x *PingReply) GetMessage() string {
if x != nil {
return x.Message
}
return ""
}
type StreamPingRequest struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Message string `protobuf:"bytes,1,opt,name=message,proto3" json:"message,omitempty"`
}
func (x *StreamPingRequest) Reset() {
*x = StreamPingRequest{}
mi := &file_xsvm_service_proto_msgTypes[2]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *StreamPingRequest) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*StreamPingRequest) ProtoMessage() {}
func (x *StreamPingRequest) ProtoReflect() protoreflect.Message {
mi := &file_xsvm_service_proto_msgTypes[2]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use StreamPingRequest.ProtoReflect.Descriptor instead.
func (*StreamPingRequest) Descriptor() ([]byte, []int) {
return file_xsvm_service_proto_rawDescGZIP(), []int{2}
}
func (x *StreamPingRequest) GetMessage() string {
if x != nil {
return x.Message
}
return ""
}
type StreamPingReply struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Message string `protobuf:"bytes,1,opt,name=message,proto3" json:"message,omitempty"`
}
func (x *StreamPingReply) Reset() {
*x = StreamPingReply{}
mi := &file_xsvm_service_proto_msgTypes[3]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *StreamPingReply) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*StreamPingReply) ProtoMessage() {}
func (x *StreamPingReply) ProtoReflect() protoreflect.Message {
mi := &file_xsvm_service_proto_msgTypes[3]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use StreamPingReply.ProtoReflect.Descriptor instead.
func (*StreamPingReply) Descriptor() ([]byte, []int) {
return file_xsvm_service_proto_rawDescGZIP(), []int{3}
}
func (x *StreamPingReply) GetMessage() string {
if x != nil {
return x.Message
}
return ""
}
var File_xsvm_service_proto protoreflect.FileDescriptor
var file_xsvm_service_proto_rawDesc = []byte{
0x0a, 0x12, 0x78, 0x73, 0x76, 0x6d, 0x2f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x2e, 0x70,
0x72, 0x6f, 0x74, 0x6f, 0x12, 0x04, 0x78, 0x73, 0x76, 0x6d, 0x22, 0x27, 0x0a, 0x0b, 0x50, 0x69,
0x6e, 0x67, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x18, 0x0a, 0x07, 0x6d, 0x65, 0x73,
0x73, 0x61, 0x67, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x6d, 0x65, 0x73, 0x73,
0x61, 0x67, 0x65, 0x22, 0x25, 0x0a, 0x09, 0x50, 0x69, 0x6e, 0x67, 0x52, 0x65, 0x70, 0x6c, 0x79,
0x12, 0x18, 0x0a, 0x07, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28,
0x09, 0x52, 0x07, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x22, 0x2d, 0x0a, 0x11, 0x53, 0x74,
0x72, 0x65, 0x61, 0x6d, 0x50, 0x69, 0x6e, 0x67, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12,
0x18, 0x0a, 0x07, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09,
0x52, 0x07, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x22, 0x2b, 0x0a, 0x0f, 0x53, 0x74, 0x72,
0x65, 0x61, 0x6d, 0x50, 0x69, 0x6e, 0x67, 0x52, 0x65, 0x70, 0x6c, 0x79, 0x12, 0x18, 0x0a, 0x07,
0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x6d,
0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x32, 0x74, 0x0a, 0x04, 0x50, 0x69, 0x6e, 0x67, 0x12, 0x2a,
0x0a, 0x04, 0x50, 0x69, 0x6e, 0x67, 0x12, 0x11, 0x2e, 0x78, 0x73, 0x76, 0x6d, 0x2e, 0x50, 0x69,
0x6e, 0x67, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x0f, 0x2e, 0x78, 0x73, 0x76, 0x6d,
0x2e, 0x50, 0x69, 0x6e, 0x67, 0x52, 0x65, 0x70, 0x6c, 0x79, 0x12, 0x40, 0x0a, 0x0a, 0x53, 0x74,
0x72, 0x65, 0x61, 0x6d, 0x50, 0x69, 0x6e, 0x67, 0x12, 0x17, 0x2e, 0x78, 0x73, 0x76, 0x6d, 0x2e,
0x53, 0x74, 0x72, 0x65, 0x61, 0x6d, 0x50, 0x69, 0x6e, 0x67, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73,
0x74, 0x1a, 0x15, 0x2e, 0x78, 0x73, 0x76, 0x6d, 0x2e, 0x53, 0x74, 0x72, 0x65, 0x61, 0x6d, 0x50,
0x69, 0x6e, 0x67, 0x52, 0x65, 0x70, 0x6c, 0x79, 0x28, 0x01, 0x30, 0x01, 0x42, 0x2c, 0x5a, 0x2a,
0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x6c, 0x75, 0x78, 0x66, 0x69,
0x2f, 0x6e, 0x6f, 0x64, 0x65, 0x2f, 0x63, 0x6f, 0x6e, 0x6e, 0x65, 0x63, 0x74, 0x70, 0x72, 0x6f,
0x74, 0x6f, 0x2f, 0x70, 0x62, 0x2f, 0x78, 0x73, 0x76, 0x6d, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74,
0x6f, 0x33,
}
var (
file_xsvm_service_proto_rawDescOnce sync.Once
file_xsvm_service_proto_rawDescData = file_xsvm_service_proto_rawDesc
)
func file_xsvm_service_proto_rawDescGZIP() []byte {
file_xsvm_service_proto_rawDescOnce.Do(func() {
file_xsvm_service_proto_rawDescData = protoimpl.X.CompressGZIP(file_xsvm_service_proto_rawDescData)
})
return file_xsvm_service_proto_rawDescData
}
var file_xsvm_service_proto_msgTypes = make([]protoimpl.MessageInfo, 4)
var file_xsvm_service_proto_goTypes = []any{
(*PingRequest)(nil), // 0: xsvm.PingRequest
(*PingReply)(nil), // 1: xsvm.PingReply
(*StreamPingRequest)(nil), // 2: xsvm.StreamPingRequest
(*StreamPingReply)(nil), // 3: xsvm.StreamPingReply
}
var file_xsvm_service_proto_depIdxs = []int32{
0, // 0: xsvm.Ping.Ping:input_type -> xsvm.PingRequest
2, // 1: xsvm.Ping.StreamPing:input_type -> xsvm.StreamPingRequest
1, // 2: xsvm.Ping.Ping:output_type -> xsvm.PingReply
3, // 3: xsvm.Ping.StreamPing:output_type -> xsvm.StreamPingReply
2, // [2:4] is the sub-list for method output_type
0, // [0:2] is the sub-list for method input_type
0, // [0:0] is the sub-list for extension type_name
0, // [0:0] is the sub-list for extension extendee
0, // [0:0] is the sub-list for field type_name
}
func init() { file_xsvm_service_proto_init() }
func file_xsvm_service_proto_init() {
if File_xsvm_service_proto != nil {
return
}
type x struct{}
out := protoimpl.TypeBuilder{
File: protoimpl.DescBuilder{
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: file_xsvm_service_proto_rawDesc,
NumEnums: 0,
NumMessages: 4,
NumExtensions: 0,
NumServices: 1,
},
GoTypes: file_xsvm_service_proto_goTypes,
DependencyIndexes: file_xsvm_service_proto_depIdxs,
MessageInfos: file_xsvm_service_proto_msgTypes,
}.Build()
File_xsvm_service_proto = out.File
file_xsvm_service_proto_rawDesc = nil
file_xsvm_service_proto_goTypes = nil
file_xsvm_service_proto_depIdxs = nil
}
connectproto/pb/xsvm/xsvmconnect/service.connect.go
-138
View File
@@ -1,138 +0,0 @@
//go:build grpc
// Code generated by protoc-gen-connect-go. DO NOT EDIT.
//
// Source: xsvm/service.proto
package xsvmconnect
import (
connect "connectrpc.com/connect"
context "context"
errors "errors"
xsvm "github.com/luxfi/node/connectproto/pb/xsvm"
http "net/http"
strings "strings"
)
// This is a compile-time assertion to ensure that this generated file and the connect package are
// compatible. If you get a compiler error that this constant is not defined, this code was
// generated with a version of connect newer than the one compiled into your binary. You can fix the
// problem by either regenerating this code with an older version of connect or updating the connect
// version compiled into your binary.
const _ = connect.IsAtLeastVersion1_13_0
const (
// PingName is the fully-qualified name of the Ping service.
PingName = "xsvm.Ping"
)
// These constants are the fully-qualified names of the RPCs defined in this package. They're
// exposed at runtime as Spec.Procedure and as the final two segments of the HTTP route.
//
// Note that these are different from the fully-qualified method names used by
// google.golang.org/protobuf/reflect/protoreflect. To convert from these constants to
// reflection-formatted method names, remove the leading slash and convert the remaining slash to a
// period.
const (
// PingPingProcedure is the fully-qualified name of the Ping's Ping RPC.
PingPingProcedure = "/xsvm.Ping/Ping"
// PingStreamPingProcedure is the fully-qualified name of the Ping's StreamPing RPC.
PingStreamPingProcedure = "/xsvm.Ping/StreamPing"
)
// PingClient is a client for the xsvm.Ping service.
type PingClient interface {
Ping(context.Context, *connect.Request[xsvm.PingRequest]) (*connect.Response[xsvm.PingReply], error)
StreamPing(context.Context) *connect.BidiStreamForClient[xsvm.StreamPingRequest, xsvm.StreamPingReply]
}
// NewPingClient constructs a client for the xsvm.Ping service. By default, it uses the Connect
// protocol with the binary Protobuf Codec, asks for gzipped responses, and sends uncompressed
// requests. To use the gRPC or gRPC-Web protocols, supply the connect.WithGRPC() or
// connect.WithGRPCWeb() options.
//
// The URL supplied here should be the base URL for the Connect or gRPC server (for example,
// http://api.acme.com or https://acme.com/grpc).
func NewPingClient(httpClient connect.HTTPClient, baseURL string, opts ...connect.ClientOption) PingClient {
baseURL = strings.TrimRight(baseURL, "/")
pingMethods := xsvm.File_xsvm_service_proto.Services().ByName("Ping").Methods()
return &pingClient{
ping: connect.NewClient[xsvm.PingRequest, xsvm.PingReply](
httpClient,
baseURL+PingPingProcedure,
connect.WithSchema(pingMethods.ByName("Ping")),
connect.WithClientOptions(opts...),
),
streamPing: connect.NewClient[xsvm.StreamPingRequest, xsvm.StreamPingReply](
httpClient,
baseURL+PingStreamPingProcedure,
connect.WithSchema(pingMethods.ByName("StreamPing")),
connect.WithClientOptions(opts...),
),
}
}
// pingClient implements PingClient.
type pingClient struct {
ping *connect.Client[xsvm.PingRequest, xsvm.PingReply]
streamPing *connect.Client[xsvm.StreamPingRequest, xsvm.StreamPingReply]
}
// Ping calls xsvm.Ping.Ping.
func (c *pingClient) Ping(ctx context.Context, req *connect.Request[xsvm.PingRequest]) (*connect.Response[xsvm.PingReply], error) {
return c.ping.CallUnary(ctx, req)
}
// StreamPing calls xsvm.Ping.StreamPing.
func (c *pingClient) StreamPing(ctx context.Context) *connect.BidiStreamForClient[xsvm.StreamPingRequest, xsvm.StreamPingReply] {
return c.streamPing.CallBidiStream(ctx)
}
// PingHandler is an implementation of the xsvm.Ping service.
type PingHandler interface {
Ping(context.Context, *connect.Request[xsvm.PingRequest]) (*connect.Response[xsvm.PingReply], error)
StreamPing(context.Context, *connect.BidiStream[xsvm.StreamPingRequest, xsvm.StreamPingReply]) error
}
// NewPingHandler builds an HTTP handler from the service implementation. It returns the path on
// which to mount the handler and the handler itself.
//
// By default, handlers support the Connect, gRPC, and gRPC-Web protocols with the binary Protobuf
// and JSON codecs. They also support gzip compression.
func NewPingHandler(svc PingHandler, opts ...connect.HandlerOption) (string, http.Handler) {
pingMethods := xsvm.File_xsvm_service_proto.Services().ByName("Ping").Methods()
pingPingHandler := connect.NewUnaryHandler(
PingPingProcedure,
svc.Ping,
connect.WithSchema(pingMethods.ByName("Ping")),
connect.WithHandlerOptions(opts...),
)
pingStreamPingHandler := connect.NewBidiStreamHandler(
PingStreamPingProcedure,
svc.StreamPing,
connect.WithSchema(pingMethods.ByName("StreamPing")),
connect.WithHandlerOptions(opts...),
)
return "/xsvm.Ping/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case PingPingProcedure:
pingPingHandler.ServeHTTP(w, r)
case PingStreamPingProcedure:
pingStreamPingHandler.ServeHTTP(w, r)
default:
http.NotFound(w, r)
}
})
}
// UnimplementedPingHandler returns CodeUnimplemented from all methods.
type UnimplementedPingHandler struct{}
func (UnimplementedPingHandler) Ping(context.Context, *connect.Request[xsvm.PingRequest]) (*connect.Response[xsvm.PingReply], error) {
return nil, connect.NewError(connect.CodeUnimplemented, errors.New("xsvm.Ping.Ping is not implemented"))
}
func (UnimplementedPingHandler) StreamPing(context.Context, *connect.BidiStream[xsvm.StreamPingRequest, xsvm.StreamPingReply]) error {
return connect.NewError(connect.CodeUnimplemented, errors.New("xsvm.Ping.StreamPing is not implemented"))
}
connectproto/xsvm/service.proto
-26
View File
@@ -1,26 +0,0 @@
syntax = "proto3";
package xsvm;
option go_package = "github.com/luxfi/node/connectproto/pb/xsvm";
service Ping {
rpc Ping(PingRequest) returns (PingReply);
rpc StreamPing(stream StreamPingRequest) returns (stream StreamPingReply);
}
message PingRequest {
string message = 1;
}
message PingReply {
string message = 1;
}
message StreamPingRequest {
string message = 1;
}
message StreamPingReply {
string message = 1;
}
db/rpcdb/grpc_client.go
-370
View File
@@ -1,370 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2026, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package rpcdb
import (
"context"
"encoding/json"
"errors"
"io"
"sync"
"google.golang.org/protobuf/types/known/emptypb"
"github.com/luxfi/database"
"github.com/luxfi/math/set"
"github.com/luxfi/utils"
rpcdbpb "github.com/luxfi/node/proto/pb/rpcdb"
)
var (
_ database.Database = (*GRPCClient)(nil)
_ database.Batch = (*grpcBatch)(nil)
_ database.Iterator = (*grpcIterator)(nil)
)
// GRPCClient is a database.Database that talks to a GRPCServer over
// gRPC. Symmetric with GRPCServer — both are pure transport adapters
// against the Layer-B wire types in github.com/luxfi/protocol/rpcdb.
type GRPCClient struct {
client rpcdbpb.DatabaseClient
closed utils.Atomic[bool]
}
// NewGRPCClient wraps a gRPC client connection to a remote rpcdb
// service.
func NewGRPCClient(client rpcdbpb.DatabaseClient) *GRPCClient {
return &GRPCClient{client: client}
}
// Has attempts to return if the database has a key with the provided value.
func (c *GRPCClient) Has(key []byte) (bool, error) {
resp, err := c.client.Has(context.Background(), &rpcdbpb.HasRequest{Key: key})
if err != nil {
return false, err
}
return resp.Has, codeToErr(resp.Err)
}
// Get attempts to return the value mapped to the key.
func (c *GRPCClient) Get(key []byte) ([]byte, error) {
resp, err := c.client.Get(context.Background(), &rpcdbpb.GetRequest{Key: key})
if err != nil {
return nil, err
}
return resp.Value, codeToErr(resp.Err)
}
// Put attempts to set the value this key maps to.
func (c *GRPCClient) Put(key, value []byte) error {
resp, err := c.client.Put(context.Background(), &rpcdbpb.PutRequest{
Key: key,
Value: value,
})
if err != nil {
return err
}
return codeToErr(resp.Err)
}
// Delete attempts to remove any mapping from the key.
func (c *GRPCClient) Delete(key []byte) error {
resp, err := c.client.Delete(context.Background(), &rpcdbpb.DeleteRequest{Key: key})
if err != nil {
return err
}
return codeToErr(resp.Err)
}
// NewBatch returns a new batch.
func (c *GRPCClient) NewBatch() database.Batch {
return &grpcBatch{db: c}
}
func (c *GRPCClient) NewIterator() database.Iterator {
return c.NewIteratorWithStartAndPrefix(nil, nil)
}
func (c *GRPCClient) NewIteratorWithStart(start []byte) database.Iterator {
return c.NewIteratorWithStartAndPrefix(start, nil)
}
func (c *GRPCClient) NewIteratorWithPrefix(prefix []byte) database.Iterator {
return c.NewIteratorWithStartAndPrefix(nil, prefix)
}
// NewIteratorWithStartAndPrefix returns a new iterator.
func (c *GRPCClient) NewIteratorWithStartAndPrefix(start, prefix []byte) database.Iterator {
resp, err := c.client.NewIteratorWithStartAndPrefix(
context.Background(),
&rpcdbpb.NewIteratorWithStartAndPrefixRequest{Start: start, Prefix: prefix},
)
if err != nil {
return &database.IteratorError{Err: err}
}
return newGRPCIterator(c, resp.Id)
}
// Compact attempts to optimize the space utilization in the provided range.
func (c *GRPCClient) Compact(start, limit []byte) error {
resp, err := c.client.Compact(context.Background(), &rpcdbpb.CompactRequest{
Start: start,
Limit: limit,
})
if err != nil {
return err
}
return codeToErr(resp.Err)
}
// Close attempts to close the database.
func (c *GRPCClient) Close() error {
c.closed.Set(true)
resp, err := c.client.Close(context.Background(), &rpcdbpb.CloseRequest{})
if err != nil {
return err
}
return codeToErr(resp.Err)
}
// Sync — rpc database delegates sync to the underlying database on the
// server side. Operations are already synchronous over RPC; no-op here
// matches the legacy internal/database/rpcdb behavior.
func (c *GRPCClient) Sync() error {
return nil
}
func (c *GRPCClient) HealthCheck(ctx context.Context) (interface{}, error) {
health, err := c.client.HealthCheck(ctx, &emptypb.Empty{})
if err != nil {
return nil, err
}
return json.RawMessage(health.Details), nil
}
// Backup is not supported over the gRPC database client.
func (c *GRPCClient) Backup(_ io.Writer, _ uint64) (uint64, error) {
return 0, errors.New("rpcdb: backup not supported")
}
// Load is not supported over the gRPC database client.
func (c *GRPCClient) Load(_ io.Reader) error {
return errors.New("rpcdb: load not supported")
}
// codeToErr maps a generated-pb Error enum back to a database sentinel.
// Lives next to the gRPC adapter so the adapter knows the inverse of
// toPbErr. Mirrors rpcdb.CodeToErr in service.go but for the
// protobuf-typed enum.
func codeToErr(code rpcdbpb.Error) error {
switch code {
case rpcdbpb.Error_ERROR_CLOSED:
return database.ErrClosed
case rpcdbpb.Error_ERROR_NOT_FOUND:
return database.ErrNotFound
default:
return nil
}
}
type grpcBatch struct {
database.BatchOps
db *GRPCClient
}
func (b *grpcBatch) Write() error {
request := &rpcdbpb.WriteBatchRequest{}
keySet := set.NewSet[string](len(b.Ops))
for i := len(b.Ops) - 1; i >= 0; i-- {
op := b.Ops[i]
key := string(op.Key)
if keySet.Contains(key) {
continue
}
keySet.Add(key)
if op.Delete {
request.Deletes = append(request.Deletes, &rpcdbpb.DeleteRequest{Key: op.Key})
} else {
request.Puts = append(request.Puts, &rpcdbpb.PutRequest{
Key: op.Key,
Value: op.Value,
})
}
}
resp, err := b.db.client.WriteBatch(context.Background(), request)
if err != nil {
return err
}
return codeToErr(resp.Err)
}
func (b *grpcBatch) Inner() database.Batch {
return b
}
type grpcIterator struct {
db *GRPCClient
id uint64
data []*rpcdbpb.PutRequest
fetchedData chan []*rpcdbpb.PutRequest
errLock sync.RWMutex
err error
reqUpdateError chan chan struct{}
once sync.Once
onClose chan struct{}
onClosed chan struct{}
}
func newGRPCIterator(db *GRPCClient, id uint64) *grpcIterator {
it := &grpcIterator{
db: db,
id: id,
fetchedData: make(chan []*rpcdbpb.PutRequest),
reqUpdateError: make(chan chan struct{}),
onClose: make(chan struct{}),
onClosed: make(chan struct{}),
}
go it.fetch()
return it
}
// Invariant: fetch is the only thread with access to send requests to
// the server's iterator. This is needed because iterators are not
// thread safe and the server expects the client (us) to only ever
// issue one request at a time for a given iterator id.
func (it *grpcIterator) fetch() {
defer func() {
resp, err := it.db.client.IteratorRelease(context.Background(), &rpcdbpb.IteratorReleaseRequest{Id: it.id})
if err != nil {
it.setError(err)
} else {
it.setError(codeToErr(resp.Err))
}
close(it.fetchedData)
close(it.onClosed)
}()
for {
resp, err := it.db.client.IteratorNext(context.Background(), &rpcdbpb.IteratorNextRequest{Id: it.id})
if err != nil {
it.setError(err)
return
}
if len(resp.Data) == 0 {
return
}
for {
select {
case it.fetchedData <- resp.Data:
case onUpdated := <-it.reqUpdateError:
it.updateError()
close(onUpdated)
continue
case <-it.onClose:
return
}
break
}
}
}
// Next attempts to move the iterator to the next element.
func (it *grpcIterator) Next() bool {
if it.db.closed.Get() {
it.data = nil
it.setError(database.ErrClosed)
return false
}
if len(it.data) > 1 {
it.data[0] = nil
it.data = it.data[1:]
return true
}
it.data = <-it.fetchedData
return len(it.data) > 0
}
// Error returns any error that occurred while iterating.
func (it *grpcIterator) Error() error {
if err := it.getError(); err != nil {
return err
}
onUpdated := make(chan struct{})
select {
case it.reqUpdateError <- onUpdated:
<-onUpdated
case <-it.onClosed:
}
return it.getError()
}
// Key returns the key of the current element.
func (it *grpcIterator) Key() []byte {
if len(it.data) == 0 {
return nil
}
return it.data[0].Key
}
// Value returns the value of the current element.
func (it *grpcIterator) Value() []byte {
if len(it.data) == 0 {
return nil
}
return it.data[0].Value
}
// Release frees any resources held by the iterator.
func (it *grpcIterator) Release() {
it.once.Do(func() {
close(it.onClose)
<-it.onClosed
})
}
func (it *grpcIterator) updateError() {
resp, err := it.db.client.IteratorError(context.Background(), &rpcdbpb.IteratorErrorRequest{Id: it.id})
if err != nil {
it.setError(err)
} else {
it.setError(codeToErr(resp.Err))
}
}
func (it *grpcIterator) setError(err error) {
if err == nil {
return
}
it.errLock.Lock()
defer it.errLock.Unlock()
if it.err == nil {
it.err = err
}
}
func (it *grpcIterator) getError() error {
it.errLock.RLock()
defer it.errLock.RUnlock()
return it.err
}
db/rpcdb/grpc_server.go
-159
View File
@@ -1,159 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2026, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package rpcdb
import (
"context"
"google.golang.org/protobuf/types/known/emptypb"
"github.com/luxfi/database"
rpcdbpb "github.com/luxfi/node/proto/pb/rpcdb"
rpcdb "github.com/luxfi/protocol/rpcdb"
)
// GRPCServer is the gRPC transport adapter for the rpcdb Service. It
// wraps *Service and translates gRPC's protobuf-generated request /
// response types into the transport-neutral wire types in
// github.com/luxfi/protocol/rpcdb. Pure adapter — no storage logic
// lives here.
type GRPCServer struct {
rpcdbpb.UnimplementedDatabaseServer
svc *Service
}
// NewGRPCServer wraps a database.Database for serving over gRPC.
func NewGRPCServer(db database.Database) *GRPCServer {
return NewGRPCServerFromService(NewService(db))
}
// NewGRPCServerFromService wraps an existing Service for serving over
// gRPC. Symmetric with NewZAPServerFromService.
func NewGRPCServerFromService(svc *Service) *GRPCServer {
return &GRPCServer{svc: svc}
}
// Compile-time interface check.
var _ rpcdbpb.DatabaseServer = (*GRPCServer)(nil)
func toPbErr(code rpcdb.Error) rpcdbpb.Error {
switch code {
case rpcdb.Error_ERROR_NOT_FOUND:
return rpcdbpb.Error_ERROR_NOT_FOUND
case rpcdb.Error_ERROR_CLOSED:
return rpcdbpb.Error_ERROR_CLOSED
default:
return rpcdbpb.Error_ERROR_UNSPECIFIED
}
}
func (g *GRPCServer) Has(ctx context.Context, req *rpcdbpb.HasRequest) (*rpcdbpb.HasResponse, error) {
resp, err := g.svc.Has(ctx, &rpcdb.HasRequest{Key: req.Key})
if err != nil {
return nil, err
}
return &rpcdbpb.HasResponse{Has: resp.Has, Err: toPbErr(resp.Err)}, nil
}
func (g *GRPCServer) Get(ctx context.Context, req *rpcdbpb.GetRequest) (*rpcdbpb.GetResponse, error) {
resp, err := g.svc.Get(ctx, &rpcdb.GetRequest{Key: req.Key})
if err != nil {
return nil, err
}
return &rpcdbpb.GetResponse{Value: resp.Value, Err: toPbErr(resp.Err)}, nil
}
func (g *GRPCServer) Put(ctx context.Context, req *rpcdbpb.PutRequest) (*rpcdbpb.PutResponse, error) {
resp, err := g.svc.Put(ctx, &rpcdb.PutRequest{Key: req.Key, Value: req.Value})
if err != nil {
return nil, err
}
return &rpcdbpb.PutResponse{Err: toPbErr(resp.Err)}, nil
}
func (g *GRPCServer) Delete(ctx context.Context, req *rpcdbpb.DeleteRequest) (*rpcdbpb.DeleteResponse, error) {
resp, err := g.svc.Delete(ctx, &rpcdb.DeleteRequest{Key: req.Key})
if err != nil {
return nil, err
}
return &rpcdbpb.DeleteResponse{Err: toPbErr(resp.Err)}, nil
}
func (g *GRPCServer) WriteBatch(ctx context.Context, req *rpcdbpb.WriteBatchRequest) (*rpcdbpb.WriteBatchResponse, error) {
puts := make([]*rpcdb.PutRequest, len(req.Puts))
for i, p := range req.Puts {
puts[i] = &rpcdb.PutRequest{Key: p.Key, Value: p.Value}
}
dels := make([]*rpcdb.DeleteRequest, len(req.Deletes))
for i, d := range req.Deletes {
dels[i] = &rpcdb.DeleteRequest{Key: d.Key}
}
resp, err := g.svc.WriteBatch(ctx, &rpcdb.WriteBatchRequest{Puts: puts, Deletes: dels})
if err != nil {
return nil, err
}
return &rpcdbpb.WriteBatchResponse{Err: toPbErr(resp.Err)}, nil
}
func (g *GRPCServer) Compact(ctx context.Context, req *rpcdbpb.CompactRequest) (*rpcdbpb.CompactResponse, error) {
resp, err := g.svc.Compact(ctx, &rpcdb.CompactRequest{Start: req.Start, Limit: req.Limit})
if err != nil {
return nil, err
}
return &rpcdbpb.CompactResponse{Err: toPbErr(resp.Err)}, nil
}
func (g *GRPCServer) Close(ctx context.Context, _ *rpcdbpb.CloseRequest) (*rpcdbpb.CloseResponse, error) {
resp, err := g.svc.Close(ctx, &rpcdb.CloseRequest{})
if err != nil {
return nil, err
}
return &rpcdbpb.CloseResponse{Err: toPbErr(resp.Err)}, nil
}
func (g *GRPCServer) HealthCheck(ctx context.Context, _ *emptypb.Empty) (*rpcdbpb.HealthCheckResponse, error) {
resp, err := g.svc.HealthCheck(ctx)
if err != nil {
return nil, err
}
return &rpcdbpb.HealthCheckResponse{Details: resp.Details}, nil
}
func (g *GRPCServer) NewIteratorWithStartAndPrefix(ctx context.Context, req *rpcdbpb.NewIteratorWithStartAndPrefixRequest) (*rpcdbpb.NewIteratorWithStartAndPrefixResponse, error) {
resp, err := g.svc.NewIteratorWithStartAndPrefix(ctx, &rpcdb.NewIteratorWithStartAndPrefixRequest{Start: req.Start, Prefix: req.Prefix})
if err != nil {
return nil, err
}
return &rpcdbpb.NewIteratorWithStartAndPrefixResponse{Id: resp.Id}, nil
}
func (g *GRPCServer) IteratorNext(ctx context.Context, req *rpcdbpb.IteratorNextRequest) (*rpcdbpb.IteratorNextResponse, error) {
resp, err := g.svc.IteratorNext(ctx, &rpcdb.IteratorNextRequest{Id: req.Id})
if err != nil {
return nil, err
}
data := make([]*rpcdbpb.PutRequest, len(resp.Data))
for i, e := range resp.Data {
data[i] = &rpcdbpb.PutRequest{Key: e.Key, Value: e.Value}
}
return &rpcdbpb.IteratorNextResponse{Data: data}, nil
}
func (g *GRPCServer) IteratorError(ctx context.Context, req *rpcdbpb.IteratorErrorRequest) (*rpcdbpb.IteratorErrorResponse, error) {
resp, err := g.svc.IteratorError(ctx, &rpcdb.IteratorErrorRequest{Id: req.Id})
if err != nil {
return nil, err
}
return &rpcdbpb.IteratorErrorResponse{Err: toPbErr(resp.Err)}, nil
}
func (g *GRPCServer) IteratorRelease(ctx context.Context, req *rpcdbpb.IteratorReleaseRequest) (*rpcdbpb.IteratorReleaseResponse, error) {
resp, err := g.svc.IteratorRelease(ctx, &rpcdb.IteratorReleaseRequest{Id: req.Id})
if err != nil {
return nil, err
}
return &rpcdbpb.IteratorReleaseResponse{Err: toPbErr(resp.Err)}, nil
}
db/rpcdb/grpc_test.go
-139
View File
@@ -1,139 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2026, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package rpcdb
import (
"context"
"fmt"
"testing"
"github.com/stretchr/testify/require"
"github.com/luxfi/database/corruptabledb"
"github.com/luxfi/database/dbtest"
"github.com/luxfi/database/memdb"
"github.com/luxfi/log"
"github.com/luxfi/vm/rpc/grpcutils"
rpcdbpb "github.com/luxfi/node/proto/pb/rpcdb"
)
type testGRPCDatabase struct {
client *GRPCClient
server *memdb.Database
}
func setupGRPCDB(t testing.TB) *testGRPCDatabase {
require := require.New(t)
db := &testGRPCDatabase{
server: memdb.New(),
}
listener, err := grpcutils.NewListener()
require.NoError(err)
serverCloser := grpcutils.ServerCloser{}
server := grpcutils.NewServer()
rpcdbpb.RegisterDatabaseServer(server, NewGRPCServer(db.server))
serverCloser.Add(server)
go grpcutils.Serve(listener, server)
conn, err := grpcutils.Dial(listener.Addr().String())
require.NoError(err)
db.client = NewGRPCClient(rpcdbpb.NewDatabaseClient(conn))
t.Cleanup(func() {
serverCloser.Stop()
_ = conn.Close()
_ = listener.Close()
})
return db
}
func TestGRPCInterface(t *testing.T) {
for name, test := range dbtest.Tests {
t.Run(name, func(t *testing.T) {
db := setupGRPCDB(t)
test(t, db.client)
})
}
}
func FuzzGRPCKeyValue(f *testing.F) {
db := setupGRPCDB(f)
dbtest.FuzzKeyValue(f, db.client)
}
func FuzzGRPCNewIteratorWithPrefix(f *testing.F) {
db := setupGRPCDB(f)
dbtest.FuzzNewIteratorWithPrefix(f, db.client)
}
func FuzzGRPCNewIteratorWithStartAndPrefix(f *testing.F) {
db := setupGRPCDB(f)
dbtest.FuzzNewIteratorWithStartAndPrefix(f, db.client)
}
func BenchmarkGRPCInterface(b *testing.B) {
for _, size := range dbtest.BenchmarkSizes {
keys, values := dbtest.SetupBenchmark(b, size[0], size[1], size[2])
for name, bench := range dbtest.Benchmarks {
b.Run(fmt.Sprintf("rpcdb_%d_pairs_%d_keys_%d_values_%s", size[0], size[1], size[2], name), func(b *testing.B) {
db := setupGRPCDB(b)
bench(b, db.client, keys, values)
})
}
}
}
func TestGRPCHealthCheck(t *testing.T) {
scenarios := []struct {
name string
testFn func(db *corruptabledb.Database) error
wantErr bool
wantErrMsg string
}{
{
name: "healthcheck success",
testFn: func(_ *corruptabledb.Database) error {
return nil
},
},
{
name: "healthcheck failed db closed",
testFn: func(db *corruptabledb.Database) error {
return db.Close()
},
wantErr: true,
wantErrMsg: "closed",
},
}
for _, scenario := range scenarios {
t.Run(scenario.name, func(t *testing.T) {
require := require.New(t)
baseDB := setupGRPCDB(t)
db := corruptabledb.New(baseDB.server, log.NoLog{})
defer db.Close()
require.NoError(scenario.testFn(db))
_, err := db.HealthCheck(context.Background())
if scenario.wantErr {
require.Error(err) //nolint:forbidigo
require.Contains(err.Error(), scenario.wantErrMsg)
return
}
require.NoError(err)
_, err = baseDB.client.HealthCheck(context.Background())
require.NoError(err)
})
}
}
go.mod
+7 -10
View File
@@ -12,7 +12,6 @@ go 1.26.3
exclude github.com/luxfi/geth v1.16.1
require (
connectrpc.com/connect v1.19.1
github.com/DataDog/zstd v1.5.7 // indirect
github.com/StephenButtolph/canoto v0.17.3
github.com/btcsuite/btcd/btcutil v1.1.6
@@ -52,11 +51,11 @@ require (
github.com/supranational/blst v0.3.16 // indirect
github.com/syndtr/goleveldb v1.0.1-0.20220614013038-64ee5596c38a // indirect
github.com/thepudds/fzgen v0.4.3
go.opentelemetry.io/otel v1.42.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.42.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.42.0
go.opentelemetry.io/otel/sdk v1.42.0
go.opentelemetry.io/otel/trace v1.42.0
go.opentelemetry.io/otel v1.42.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.42.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.42.0 // indirect
go.opentelemetry.io/otel/sdk v1.42.0 // indirect
go.opentelemetry.io/otel/trace v1.42.0 // indirect
go.uber.org/goleak v1.3.0
go.uber.org/mock v0.6.0
golang.org/x/crypto v0.50.0
@@ -69,7 +68,7 @@ require (
gonum.org/v1/gonum v0.17.0
google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9 // indirect
google.golang.org/grpc v1.80.0 // indirect
google.golang.org/protobuf v1.36.11
google.golang.org/protobuf v1.36.11 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
@@ -120,7 +119,6 @@ require (
)
require (
connectrpc.com/grpcreflect v1.3.0
github.com/cloudflare/circl v1.6.3
github.com/consensys/gnark-crypto v0.20.1
github.com/golang-jwt/jwt/v4 v4.5.2
@@ -154,7 +152,6 @@ require (
github.com/luxfi/warp v1.18.6
github.com/luxfi/zwing v0.5.2
github.com/nbutton23/zxcvbn-go v0.0.0-20210217022336-fa2cb2858354
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.42.0
go.uber.org/zap v1.27.1
)
@@ -171,7 +168,6 @@ require (
github.com/klauspost/crc32 v1.3.0 // indirect
github.com/luxfi/age v1.5.0 // indirect
github.com/luxfi/ai v0.1.0 // indirect
github.com/luxfi/bft v0.1.5 // indirect
github.com/luxfi/corona v0.3.0 // indirect
github.com/luxfi/crypto/ipa v1.2.4 // indirect
github.com/luxfi/lens v0.1.3 // indirect
@@ -191,6 +187,7 @@ require (
github.com/philhofer/fwd v1.2.0 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/tinylib/msgp v1.6.1 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.42.0 // indirect
)
require (
go.sum
-8
View File
@@ -1,9 +1,5 @@
c2sp.org/CCTV/age v0.0.0-20251208015420-e9274a7bdbfd h1:ZLsPO6WdZ5zatV4UfVpr7oAwLGRZ+sebTUruuM4Ra3M=
c2sp.org/CCTV/age v0.0.0-20251208015420-e9274a7bdbfd/go.mod h1:SrHC2C7r5GkDk8R+NFVzYy/sdj0Ypg9htaPXQq5Cqeo=
connectrpc.com/connect v1.19.1 h1:R5M57z05+90EfEvCY1b7hBxDVOUl45PrtXtAV2fOC14=
connectrpc.com/connect v1.19.1/go.mod h1:tN20fjdGlewnSFeZxLKb0xwIZ6ozc3OQs2hTXy4du9w=
connectrpc.com/grpcreflect v1.3.0 h1:Y4V+ACf8/vOb1XOc251Qun7jMB75gCUNw6llvB9csXc=
connectrpc.com/grpcreflect v1.3.0/go.mod h1:nfloOtCS8VUQOQ1+GTdFzVg2CJo4ZGaat8JIovCtDYs=
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
@@ -203,8 +199,6 @@ github.com/gorilla/rpc v1.2.1/go.mod h1:uNpOihAlF5xRFLuTYhfR0yfCTm0WTQSQttkMSptR
github.com/gorilla/websocket v1.5.0/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 h1:JeSE6pjso5THxAzdVpqr6/geYxZytqFMBCOtn/ujyeo=
github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674/go.mod h1:r4w70xmWCQKmi1ONH4KIaBptdivuRPyosB9RmPlGEwA=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 h1:Ovs26xHkKqVztRpIrF/92BcuyuQ/YW4NSIpoGtfXNho=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 h1:HWRh5R2+9EifMyIHV7ZV+MIZqgz+PMpZ14Jynv3O2Zs=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0/go.mod h1:JfhWUomR1baixubs02l85lZYYOm7LV6om4ceouMv45c=
github.com/gtank/merlin v0.1.1 h1:eQ90iG7K9pOhtereWsmyRJ6RAwcP4tHTDBHXNg+u5is=
@@ -266,8 +260,6 @@ github.com/luxfi/api v1.0.11 h1:t4fzN9Ox/Vy5msW2WpSXq4xCAmBXXJS0oM+zIjM9IiQ=
github.com/luxfi/api v1.0.11/go.mod h1:q3Hh3mmkvp3cnv3aqe2+gCtmmSAL/uFjuzrqGavQLNA=
github.com/luxfi/atomic v1.0.0 h1:xUV60MuzRvXngaQ1sM0yVC2v4TRoLlUGkkH7M9PS4yw=
github.com/luxfi/atomic v1.0.0/go.mod h1:0G2mTlQ6TXWHICUHrUUPu1/qAiIyR4gSZ2tva9ci/bI=
github.com/luxfi/bft v0.1.5 h1:5xVLPkog4e5LTgaVlb9pgxA0EWE6tkrKwHPZVRz+RZw=
github.com/luxfi/bft v0.1.5/go.mod h1:5I8Ft8yA69xZlDe3RB0i4MgbqFKLZe65o/sha8JuKvU=
github.com/luxfi/cache v1.2.1 h1:kAzOS55/hmYeNKR+0HAKv4ma48Y6JjkI8UQeqdZ8bfI=
github.com/luxfi/cache v1.2.1/go.mod h1:co7JTxZZHpKT31Yh01LFp5aZOxmoUg157FhBLQdQHVU=
github.com/luxfi/chains v1.2.1 h1:eeqekwYV8E1OTIGFwzKBcI+9JGNzm3pfg+56qUn3BTQ=
internal/ids/rpcaliasreader/alias_reader_client.go
-57
View File
@@ -1,57 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package rpcaliasreader
import (
"context"
"github.com/luxfi/ids"
aliasreaderpb "github.com/luxfi/node/proto/pb/aliasreader"
)
var _ ids.AliaserReader = (*Client)(nil)
// Client implements alias lookups that talk over RPC.
type Client struct {
client aliasreaderpb.AliasReaderClient
}
// NewClient returns an alias lookup instance connected to a remote alias lookup
// instance
func NewClient(client aliasreaderpb.AliasReaderClient) *Client {
return &Client{client: client}
}
func (c *Client) Lookup(alias string) (ids.ID, error) {
resp, err := c.client.Lookup(context.Background(), &aliasreaderpb.Alias{
Alias: alias,
})
if err != nil {
return ids.Empty, err
}
return ids.ToID(resp.Id)
}
func (c *Client) PrimaryAlias(id ids.ID) (string, error) {
resp, err := c.client.PrimaryAlias(context.Background(), &aliasreaderpb.ID{
Id: id[:],
})
if err != nil {
return "", err
}
return resp.Alias, nil
}
func (c *Client) Aliases(id ids.ID) ([]string, error) {
resp, err := c.client.Aliases(context.Background(), &aliasreaderpb.ID{
Id: id[:],
})
if err != nil {
return nil, err
}
return resp.Aliases, nil
}
internal/ids/rpcaliasreader/alias_reader_server.go
-74
View File
@@ -1,74 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package rpcaliasreader
import (
"context"
"fmt"
"github.com/luxfi/ids"
aliasreaderpb "github.com/luxfi/node/proto/pb/aliasreader"
)
var _ aliasreaderpb.AliasReaderServer = (*Server)(nil)
// Server enables alias lookups over RPC.
type Server struct {
aliasreaderpb.UnsafeAliasReaderServer
aliaser ids.AliaserReader
}
// NewServer returns an alias lookup connected to a remote alias lookup
func NewServer(aliaser ids.AliaserReader) *Server {
return &Server{aliaser: aliaser}
}
func (s *Server) Lookup(
_ context.Context,
req *aliasreaderpb.Alias,
) (*aliasreaderpb.ID, error) {
id, err := s.aliaser.Lookup(req.Alias)
if err != nil {
return nil, err
}
return &aliasreaderpb.ID{
Id: id[:],
}, nil
}
func (s *Server) PrimaryAlias(
_ context.Context,
req *aliasreaderpb.ID,
) (*aliasreaderpb.Alias, error) {
if s.aliaser == nil {
return nil, fmt.Errorf("aliaser is nil - BCLookup not configured for this chain")
}
id, err := ids.ToID(req.Id)
if err != nil {
return nil, err
}
alias, err := s.aliaser.PrimaryAlias(id)
return &aliasreaderpb.Alias{
Alias: alias,
}, err
}
func (s *Server) Aliases(
_ context.Context,
req *aliasreaderpb.ID,
) (*aliasreaderpb.AliasList, error) {
id, err := ids.ToID(req.Id)
if err != nil {
return nil, err
}
aliases, err := s.aliaser.Aliases(id)
return &aliasreaderpb.AliasList{
Aliases: aliases,
}, err
}
internal/ids/rpcaliasreader/alias_reader_test.go
-46
View File
@@ -1,46 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package rpcaliasreader
import (
"testing"
"github.com/stretchr/testify/require"
"github.com/luxfi/ids"
"github.com/luxfi/ids/idstest"
"github.com/luxfi/vm/rpc/grpcutils"
aliasreaderpb "github.com/luxfi/node/proto/pb/aliasreader"
)
func TestInterface(t *testing.T) {
for _, test := range idstest.AliasTests {
t.Run(test.Name, func(t *testing.T) {
require := require.New(t)
listener, err := grpcutils.NewListener()
require.NoError(err)
defer listener.Close()
serverCloser := grpcutils.ServerCloser{}
defer serverCloser.Stop()
w := ids.NewAliaser()
server := grpcutils.NewServer()
aliasreaderpb.RegisterAliasReaderServer(server, NewServer(w))
serverCloser.Add(server)
go grpcutils.Serve(listener, server)
conn, err := grpcutils.Dial(listener.Addr().String())
require.NoError(err)
defer conn.Close()
r := NewClient(aliasreaderpb.NewAliasReaderClient(conn))
test.Test(t, r, w)
})
}
}
message/bft_grpc.go
-18
View File
@@ -1,18 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package message
import "github.com/luxfi/node/proto/p2p"
// newMessageBFT creates a BFT message wrapper (gRPC version - uses Simplex)
func newMessageBFT(msg *p2p.BFT) *p2p.Message_BFT {
return &p2p.Message_BFT{Simplex: msg}
}
// extractBFT extracts the BFT message from the wrapper (gRPC version - uses Simplex)
func extractBFT(msg *p2p.Message_BFT) *p2p.BFT {
return msg.Simplex
}
message/bft_zap.go
+2 -4
View File
@@ -1,5 +1,3 @@
//go:build !grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
@@ -7,12 +5,12 @@ package message
import "github.com/luxfi/node/proto/p2p"
// newMessageBFT creates a BFT message wrapper (ZAP version)
// newMessageBFT creates a BFT message wrapper.
func newMessageBFT(msg *p2p.BFT) *p2p.Message_BFT {
return &p2p.Message_BFT{BFT: msg}
}
// extractBFT extracts the BFT message from the wrapper (ZAP version)
// extractBFT extracts the BFT message from the wrapper.
func extractBFT(msg *p2p.Message_BFT) *p2p.BFT {
return msg.BFT
}
proto/p2p/p2p_grpc.go
-120
View File
@@ -1,120 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package p2p re-exports P2P types from the appropriate wire format implementation.
// Without grpc tag: uses ZAP wire format (zero protobuf)
// With grpc tag: uses protobuf wire format
package p2p
import (
"github.com/luxfi/node/proto/pb/p2p"
"google.golang.org/protobuf/proto"
)
// Re-export all types from protobuf implementation
type (
EngineType = p2p.EngineType
Message = p2p.Message
Message_CompressedZstd = p2p.Message_CompressedZstd
Message_Ping = p2p.Message_Ping
Message_Pong = p2p.Message_Pong
Message_Handshake = p2p.Message_Handshake
Message_GetPeerList = p2p.Message_GetPeerList
Message_PeerList_ = p2p.Message_PeerList_
Message_GetStateSummaryFrontier = p2p.Message_GetStateSummaryFrontier
Message_StateSummaryFrontier_ = p2p.Message_StateSummaryFrontier_
Message_GetAcceptedStateSummary = p2p.Message_GetAcceptedStateSummary
Message_AcceptedStateSummary_ = p2p.Message_AcceptedStateSummary_
Message_GetAcceptedFrontier = p2p.Message_GetAcceptedFrontier
Message_AcceptedFrontier_ = p2p.Message_AcceptedFrontier_
Message_GetAccepted = p2p.Message_GetAccepted
Message_Accepted_ = p2p.Message_Accepted_
Message_GetAncestors = p2p.Message_GetAncestors
Message_Ancestors_ = p2p.Message_Ancestors_
Message_Get = p2p.Message_Get
Message_Put = p2p.Message_Put
Message_PushQuery = p2p.Message_PushQuery
Message_PullQuery = p2p.Message_PullQuery
Message_Chits = p2p.Message_Chits
Message_Request = p2p.Message_Request
Message_Response = p2p.Message_Response
Message_Gossip = p2p.Message_Gossip
Message_Error = p2p.Message_Error
Message_Simplex = p2p.Message_Simplex
Ping = p2p.Ping
Pong = p2p.Pong
Handshake = p2p.Handshake
Client = p2p.Client
BloomFilter = p2p.BloomFilter
GetPeerList = p2p.GetPeerList
PeerList = p2p.PeerList
ClaimedIpPort = p2p.ClaimedIpPort
GetStateSummaryFrontier = p2p.GetStateSummaryFrontier
StateSummaryFrontier = p2p.StateSummaryFrontier
GetAcceptedStateSummary = p2p.GetAcceptedStateSummary
AcceptedStateSummary = p2p.AcceptedStateSummary
GetAcceptedFrontier = p2p.GetAcceptedFrontier
AcceptedFrontier = p2p.AcceptedFrontier
GetAccepted = p2p.GetAccepted
Accepted = p2p.Accepted
GetAncestors = p2p.GetAncestors
Ancestors = p2p.Ancestors
Get = p2p.Get
Put = p2p.Put
PushQuery = p2p.PushQuery
PullQuery = p2p.PullQuery
Chits = p2p.Chits
Request = p2p.Request
Response = p2p.Response
Gossip = p2p.Gossip
Error = p2p.Error
Simplex = p2p.Simplex
// BFT aliases - ZAP uses "BFT", protobuf uses "Simplex" for the same concept
BFT = p2p.Simplex
Message_BFT = p2p.Message_Simplex
BFT_BlockProposal = p2p.Simplex_BlockProposal
BFT_Vote = p2p.Simplex_Vote
BFT_EmptyVote = p2p.Simplex_EmptyVote
BFT_FinalizeVote = p2p.Simplex_FinalizeVote
BFT_Notarization = p2p.Simplex_Notarization
BFT_EmptyNotarization = p2p.Simplex_EmptyNotarization
BFT_Finalization = p2p.Simplex_Finalization
BFT_ReplicationRequest = p2p.Simplex_ReplicationRequest
BFT_ReplicationResponse = p2p.Simplex_ReplicationResponse
BlockProposal = p2p.BlockProposal
ProtocolMetadata = p2p.ProtocolMetadata
BlockHeader = p2p.BlockHeader
Signature = p2p.Signature
Vote = p2p.Vote
EmptyVote = p2p.EmptyVote
QuorumCertificate = p2p.QuorumCertificate
EmptyNotarization = p2p.EmptyNotarization
ReplicationRequest = p2p.ReplicationRequest
ReplicationResponse = p2p.ReplicationResponse
QuorumRound = p2p.QuorumRound
)
// Re-export constants
const (
EngineType_ENGINE_TYPE_UNSPECIFIED = p2p.EngineType_ENGINE_TYPE_UNSPECIFIED
EngineType_ENGINE_TYPE_CHAIN = p2p.EngineType_ENGINE_TYPE_CHAIN
EngineType_ENGINE_TYPE_DAG = p2p.EngineType_ENGINE_TYPE_DAG
)
// Marshal encodes a Message using protobuf
func Marshal(m *Message) ([]byte, error) {
return proto.Marshal(m)
}
// Unmarshal decodes a Message using protobuf
func Unmarshal(data []byte, m *Message) error {
return proto.Unmarshal(data, m)
}
// Size returns the encoded size of a message
func Size(m *Message) int {
return proto.Size(m)
}
proto/p2p/p2p_zap.go
+2 -5
View File
@@ -1,11 +1,8 @@
//go:build !grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package p2p re-exports P2P types from the appropriate wire format implementation.
// Without grpc tag: uses ZAP wire format (zero protobuf)
// With grpc tag: uses protobuf wire format
// Package p2p re-exports P2P types from the ZAP wire format
// implementation. ZAP is the only supported transport.
package p2p
import "github.com/luxfi/node/proto/zap/p2p"
proto/pb/aliasreader/aliasreader.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package aliasreader — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package aliasreader
proto/pb/http/http.go
-7
View File
@@ -1,7 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package http — protobuf wire types stub. Default builds use ZAP.
package http
proto/pb/http/responsewriter/responsewriter.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package responsewriter — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package responsewriter
proto/pb/io/metric/client/client.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package client — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package client
proto/pb/io/reader/reader.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package reader — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package reader
proto/pb/io/writer/writer.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package writer — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package writer
proto/pb/keystore/keystore.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package keystore — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package keystore
proto/pb/message/message.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package message — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package message
proto/pb/messenger/messenger.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package messenger — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package messenger
proto/pb/net/conn/conn.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package conn — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package conn
proto/pb/p2p/p2p.go
-19
View File
@@ -1,19 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package p2p is the protobuf-generated wire-types package referenced
// by the grpc-tagged transport in proto/p2p/p2p_grpc.go (or its
// legacy single-file form). Default builds use ZAP and never reach
// this package; the build tag here matches the grpc-tagged consumers
// so default builds skip it cleanly.
//
// To re-generate the actual protobuf types (when the grpc transport
// is needed), run:
//
// protoc -I=proto/ --go_out=proto/pb/ proto/p2p/p2p.proto
//
// in the node root. Until that's done, this stub keeps "go mod tidy"
// happy while the canonical wire format stays ZAP.
package p2p
proto/pb/platformvm/platformvm.go
-19
View File
@@ -1,19 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package platformvm is the protobuf-generated wire-types package referenced
// by the grpc-tagged transport in proto/platformvm/platformvm_grpc.go (or its
// legacy single-file form). Default builds use ZAP and never reach
// this package; the build tag here matches the grpc-tagged consumers
// so default builds skip it cleanly.
//
// To re-generate the actual protobuf types (when the grpc transport
// is needed), run:
//
// protoc -I=proto/ --go_out=proto/pb/ proto/platformvm/platformvm.proto
//
// in the node root. Until that's done, this stub keeps "go mod tidy"
// happy while the canonical wire format stays ZAP.
package platformvm
proto/pb/rpcdb/rpcdb.go
-19
View File
@@ -1,19 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package rpcdb is the protobuf-generated wire-types package referenced
// by the grpc-tagged transport in proto/rpcdb/rpcdb_grpc.go (or its
// legacy single-file form). Default builds use ZAP and never reach
// this package; the build tag here matches the grpc-tagged consumers
// so default builds skip it cleanly.
//
// To re-generate the actual protobuf types (when the grpc transport
// is needed), run:
//
// protoc -I=proto/ --go_out=proto/pb/ proto/rpcdb/rpcdb.proto
//
// in the node root. Until that's done, this stub keeps "go mod tidy"
// happy while the canonical wire format stays ZAP.
package rpcdb
proto/pb/sdk/sdk.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package sdk — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package sdk
proto/pb/sender/sender.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package sender — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package sender
proto/pb/sharedmemory/sharedmemory.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package sharedmemory — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package sharedmemory
proto/pb/signer/signer.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package signer — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package signer
proto/pb/sync/sync.go
-19
View File
@@ -1,19 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package sync is the protobuf-generated wire-types package referenced
// by the grpc-tagged transport in proto/sync/sync_grpc.go (or its
// legacy single-file form). Default builds use ZAP and never reach
// this package; the build tag here matches the grpc-tagged consumers
// so default builds skip it cleanly.
//
// To re-generate the actual protobuf types (when the grpc transport
// is needed), run:
//
// protoc -I=proto/ --go_out=proto/pb/ proto/sync/sync.proto
//
// in the node root. Until that's done, this stub keeps "go mod tidy"
// happy while the canonical wire format stays ZAP.
package sync
proto/pb/validatorstate/validatorstate.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package validatorstate — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package validatorstate
proto/pb/vm/runtime/runtime.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package runtime — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package runtime
proto/pb/vm/vm.go
-19
View File
@@ -1,19 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package vm is the protobuf-generated wire-types package referenced
// by the grpc-tagged transport in proto/vm/vm_grpc.go (or its
// legacy single-file form). Default builds use ZAP and never reach
// this package; the build tag here matches the grpc-tagged consumers
// so default builds skip it cleanly.
//
// To re-generate the actual protobuf types (when the grpc transport
// is needed), run:
//
// protoc -I=proto/ --go_out=proto/pb/ proto/vm/vm.proto
//
// in the node root. Until that's done, this stub keeps "go mod tidy"
// happy while the canonical wire format stays ZAP.
package vm
proto/pb/warp/warp.go
-8
View File
@@ -1,8 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package warp — protobuf wire types stub. See proto/pb/p2p/p2p.go
// header for context. Default builds use ZAP and skip this file.
package warp
proto/platformvm/platformvm_grpc.go
-23
View File
@@ -1,23 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package platformvm
import (
"github.com/luxfi/node/proto/pb/platformvm"
"google.golang.org/protobuf/proto"
)
type (
L1ValidatorRegistrationJustification = platformvm.L1ValidatorRegistrationJustification
L1ValidatorRegistrationJustification_ConvertNetworkToL1TxData = platformvm.L1ValidatorRegistrationJustification_ConvertNetworkToL1TxData
L1ValidatorRegistrationJustification_RegisterL1ValidatorMessage = platformvm.L1ValidatorRegistrationJustification_RegisterL1ValidatorMessage
ChainIDIndex = platformvm.ChainIDIndex
)
// Unmarshal decodes a message from bytes (protobuf implementation)
func Unmarshal(data []byte, m proto.Message) error {
return proto.Unmarshal(data, m)
}
proto/platformvm/platformvm_zap.go
+5 -5
View File
@@ -1,5 +1,3 @@
//go:build !grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
@@ -16,9 +14,11 @@ type (
ChainIDIndex = platformvm.ChainIDIndex
)
// Unmarshal decodes a message from bytes (ZAP implementation)
// Unmarshal decodes a message from bytes. The ZAP codec encodes
// messages via direct field assignment; this entry point is reserved
// for compatibility with callers that thread a generic byte payload
// through the wire types and is a no-op until a concrete consumer
// arrives.
func Unmarshal(data []byte, m interface{}) error {
// ZAP uses direct field assignment - for now just return nil
// Actual ZAP unmarshaling would use a codec
return nil
}
proto/sync/sync_grpc.go
-23
View File
@@ -1,23 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package sync re-exports sync types from the appropriate wire format implementation.
// Without grpc tag: uses ZAP wire format (zero protobuf)
// With grpc tag: uses protobuf wire format
package sync
import pb "github.com/luxfi/node/proto/pb/sync"
// Re-export all types from protobuf implementation
type (
Key = pb.Key
MaybeBytes = pb.MaybeBytes
ProofNode = pb.ProofNode
Proof = pb.Proof
KeyValue = pb.KeyValue
KeyChange = pb.KeyChange
RangeProof = pb.RangeProof
ChangeProof = pb.ChangeProof
)
proto/sync/sync_zap.go
+2 -5
View File
@@ -1,11 +1,8 @@
//go:build !grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package sync re-exports sync types from the appropriate wire format implementation.
// Without grpc tag: uses ZAP wire format (zero protobuf)
// With grpc tag: uses protobuf wire format
// Package sync re-exports sync types from the ZAP wire format
// implementation. ZAP is the only supported transport.
package sync
import "github.com/luxfi/node/proto/zap/sync"
proto/vm/vm_grpc.go
-25
View File
@@ -1,25 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package vm
import "github.com/luxfi/node/proto/pb/vm"
type (
State = vm.State
Error = vm.Error
)
const (
State_STATE_UNSPECIFIED = vm.State_STATE_UNSPECIFIED
State_STATE_STATE_SYNCING = vm.State_STATE_STATE_SYNCING
State_STATE_BOOTSTRAPPING = vm.State_STATE_BOOTSTRAPPING
State_STATE_NORMAL_OP = vm.State_STATE_NORMAL_OP
Error_ERROR_UNSPECIFIED = vm.Error_ERROR_UNSPECIFIED
Error_ERROR_CLOSED = vm.Error_ERROR_CLOSED
Error_ERROR_NOT_FOUND = vm.Error_ERROR_NOT_FOUND
Error_ERROR_STATE_SYNC_NOT_IMPLEMENTED = vm.Error_ERROR_STATE_SYNC_NOT_IMPLEMENTED
)
proto/vm/vm_zap.go
-2
View File
@@ -1,5 +1,3 @@
//go:build !grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
service/keystore/rpckeystore/client.go
-65
View File
@@ -1,65 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package rpckeystore is the keystore-over-RPC client and server. The
// transport is selected by build tag: `grpc` selects the legacy gRPC
// codepath in this file; the default build (no tag) will eventually
// select a ZAP-based codepath (task #57). Mirrors the rpcdb naming
// pattern — `rpc` is the value (remote procedure call), the substrate
// is per-tag.
package rpckeystore
import (
"context"
"github.com/luxfi/database"
"github.com/luxfi/database/encdb"
"github.com/luxfi/node/db/rpcdb"
"github.com/luxfi/node/service/keystore"
"github.com/luxfi/vm/rpc/grpcutils"
keystorepb "github.com/luxfi/node/proto/pb/keystore"
rpcdbpb "github.com/luxfi/node/proto/pb/rpcdb"
)
var _ keystore.BlockchainKeystore = (*Client)(nil)
// Client is a consensus.Keystore that talks over RPC.
type Client struct {
client keystorepb.KeystoreClient
}
// NewClient returns a keystore instance connected to a remote keystore instance
func NewClient(client keystorepb.KeystoreClient) *Client {
return &Client{
client: client,
}
}
func (c *Client) GetDatabase(username, password string) (*encdb.Database, error) {
bcDB, err := c.GetRawDatabase(username, password)
if err != nil {
return nil, err
}
return encdb.New([]byte(password), bcDB)
}
func (c *Client) GetRawDatabase(username, password string) (database.Database, error) {
resp, err := c.client.GetDatabase(context.Background(), &keystorepb.GetDatabaseRequest{
Username: username,
Password: password,
})
if err != nil {
return nil, err
}
clientConn, err := grpcutils.Dial(resp.ServerAddr)
if err != nil {
return nil, err
}
dbClient := rpcdb.NewGRPCClient(rpcdbpb.NewDatabaseClient(clientConn))
return dbClient, nil
}
service/keystore/rpckeystore/server.go
-67
View File
@@ -1,67 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package rpckeystore
import (
"context"
"github.com/luxfi/database"
"github.com/luxfi/node/service/keystore"
"github.com/luxfi/vm/rpc/grpcutils"
keystorepb "github.com/luxfi/node/proto/pb/keystore"
)
var _ keystorepb.KeystoreServer = (*Server)(nil)
// Server is a consensus.Keystore that is managed over RPC.
type Server struct {
keystorepb.UnsafeKeystoreServer
ks keystore.BlockchainKeystore
}
// NewServer returns a keystore connected to a remote keystore
func NewServer(ks keystore.BlockchainKeystore) *Server {
return &Server{
ks: ks,
}
}
func (s *Server) GetDatabase(
_ context.Context,
req *keystorepb.GetDatabaseRequest,
) (*keystorepb.GetDatabaseResponse, error) {
db, err := s.ks.GetRawDatabase(req.Username, req.Password)
if err != nil {
return nil, err
}
closer := dbCloser{Database: db}
serverListener, err := grpcutils.NewListener()
if err != nil {
return nil, err
}
server := grpcutils.NewServer()
closer.closer.Add(server)
// start the db server
go grpcutils.Serve(serverListener, server)
return &keystorepb.GetDatabaseResponse{ServerAddr: serverListener.Addr().String()}, nil
}
type dbCloser struct {
database.Database
closer grpcutils.ServerCloser
}
func (db *dbCloser) Close() error {
err := db.Database.Close()
db.closer.Stop()
return err
}
trace/exporter_grpc.go
-68
View File
@@ -1,68 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package trace
import (
"context"
"time"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
)
const tracerProviderExportCreationTimeout = 5 * time.Second
type ExporterConfig struct {
Type ExporterType `json:"type"`
// Endpoint to send metrics to. If empty, the default endpoint will be used.
Endpoint string `json:"endpoint"`
// Headers to send with metrics
Headers map[string]string `json:"headers"`
// If true, don't use TLS
Insecure bool `json:"insecure"`
}
func newExporter(config ExporterConfig) (sdktrace.SpanExporter, error) {
var client otlptrace.Client
switch config.Type {
case GRPC:
opts := []otlptracegrpc.Option{
otlptracegrpc.WithHeaders(config.Headers),
otlptracegrpc.WithTimeout(tracerExportTimeout),
}
if config.Endpoint != "" {
opts = append(opts, otlptracegrpc.WithEndpoint(config.Endpoint))
}
if config.Insecure {
opts = append(opts, otlptracegrpc.WithInsecure())
}
client = otlptracegrpc.NewClient(opts...)
case HTTP:
opts := []otlptracehttp.Option{
otlptracehttp.WithHeaders(config.Headers),
otlptracehttp.WithTimeout(tracerExportTimeout),
}
if config.Endpoint != "" {
opts = append(opts, otlptracehttp.WithEndpoint(config.Endpoint))
}
if config.Insecure {
opts = append(opts, otlptracehttp.WithInsecure())
}
client = otlptracehttp.NewClient(opts...)
default:
return nil, errUnknownExporterType
}
ctx, cancel := context.WithTimeout(context.Background(), tracerProviderExportCreationTimeout)
defer cancel()
return otlptrace.New(ctx, client)
}
trace/exporter_type.go
-91
View File
@@ -1,91 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package trace
import (
"errors"
"fmt"
"strings"
)
const (
Disabled ExporterType = iota
GRPC
HTTP
ZAP // ZAP-based OTLP transport (default, no gRPC dependency)
)
var (
errUnknownExporterType = errors.New("unknown exporter type")
errMissingQuotes = errors.New("first and last characters should be quotes")
)
func ExporterTypeFromString(exporterTypeStr string) (ExporterType, error) {
switch strings.ToLower(exporterTypeStr) {
case "disabled":
return Disabled, nil
case "grpc":
return GRPC, nil
case "http":
return HTTP, nil
case "zap":
return ZAP, nil
default:
return 0, fmt.Errorf("%w: %q", errUnknownExporterType, exporterTypeStr)
}
}
type ExporterType byte
func (t ExporterType) MarshalJSON() ([]byte, error) {
str, ok := t.toString()
if !ok {
return nil, fmt.Errorf("%w: %d", errUnknownExporterType, t)
}
return []byte(`"` + str + `"`), nil
}
func (t *ExporterType) UnmarshalJSON(b []byte) error {
str := string(b)
if str == "null" { // If "null", do nothing
return nil
}
if len(str) < 2 {
return errMissingQuotes
}
lastIndex := len(str) - 1
if str[0] != '"' || str[lastIndex] != '"' {
return errMissingQuotes
}
exporterType, err := ExporterTypeFromString(str[1:lastIndex])
if err != nil {
return err
}
*t = exporterType
return nil
}
func (t ExporterType) String() string {
str, _ := t.toString()
return str
}
func (t ExporterType) toString() (string, bool) {
switch t {
case Disabled:
return "disabled", true
case GRPC:
return "grpc", true
case HTTP:
return "http", true
case ZAP:
return "zap", true
default:
return "unknown", false
}
}
trace/exporter_type_test.go
-132
View File
@@ -1,132 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package trace
import (
"testing"
"github.com/stretchr/testify/require"
)
func TestMarshal(t *testing.T) {
tests := []struct {
name string
exporter ExporterType
expected string
expectedError error
}{
{
name: "unknown_type",
exporter: 255,
expectedError: errUnknownExporterType,
},
{
name: "disabled",
exporter: Disabled,
expected: `"disabled"`,
},
{
name: "grpc",
exporter: GRPC,
expected: `"grpc"`,
},
{
name: "http",
exporter: HTTP,
expected: `"http"`,
},
{
name: "zap",
exporter: ZAP,
expected: `"zap"`,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
require := require.New(t)
actual, err := tt.exporter.MarshalJSON()
require.ErrorIs(err, tt.expectedError)
require.Equal(tt.expected, string(actual))
})
}
}
func TestUnmarshal(t *testing.T) {
tests := []struct {
name string
json string
expected ExporterType
expectedError error
}{
{
name: "no_quotes",
json: "grpc",
expectedError: errMissingQuotes,
},
{
name: "single_left_quote",
json: `"grpc`,
expectedError: errMissingQuotes,
},
{
name: "single_right_quote",
json: `grpc"`,
expectedError: errMissingQuotes,
},
{
name: "only_one_quote",
json: `"`,
expectedError: errMissingQuotes,
},
{
name: "multiple_quotes",
json: `""grpc"""`,
expectedError: errUnknownExporterType,
},
{
name: "empty_string",
json: `""`,
expectedError: errUnknownExporterType,
},
{
name: "null",
json: `null`,
},
{
name: "disabled",
json: `"disabled"`,
expected: Disabled,
},
{
name: "grpc",
json: `"grpc"`,
expected: GRPC,
},
{
name: "http",
json: `"http"`,
expected: HTTP,
},
{
name: "zap",
json: `"zap"`,
expected: ZAP,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
require := require.New(t)
var actual ExporterType
err := actual.UnmarshalJSON([]byte(tt.json))
require.ErrorIs(err, tt.expectedError)
require.Equal(tt.expected, actual)
})
}
}
trace/noop.go
-19
View File
@@ -1,19 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package trace
import "go.opentelemetry.io/otel/trace/noop"
var Noop Tracer = noOpTracer{}
// noOpTracer is an implementation of trace.Tracer that does nothing.
type noOpTracer struct {
noop.Tracer
}
func (noOpTracer) Close() error {
return nil
}
trace/trace_zap.go
+5 -6
View File
@@ -1,11 +1,10 @@
//go:build !grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package trace provides tracing functionality.
// In ZAP mode (default), tracing is disabled - no OpenTelemetry dependencies.
// Use -tags=grpc to enable full OpenTelemetry tracing support.
// Package trace provides tracing primitives. The package ships a
// no-op tracer suitable for ZAP-native production builds; a real
// exporter implementation lives outside the node tree and plugs in
// via the Tracer interface.
package trace
import (
@@ -191,7 +190,7 @@ func (noopSpan) RecordError(error, ...EventOption) {}
func (noopSpan) AddEvent(string, ...EventOption) {}
func (noopSpan) SetAttributes(...KeyValue) {}
// New creates a new tracer (returns Noop in ZAP mode)
// New creates a new tracer (returns Noop in the canonical ZAP build).
func New(Config) (Tracer, error) {
return Noop, nil
}
trace/tracer.go
-134
View File
@@ -1,134 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package trace
import (
"context"
"fmt"
"io"
"time"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/sdk/resource"
oteltrace "go.opentelemetry.io/otel/trace"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)
// Type aliases for otel compatibility
type (
KeyValue = attribute.KeyValue
SpanStartOption = oteltrace.SpanStartOption
SpanEndOption = oteltrace.SpanEndOption
Span = oteltrace.Span
StatusCode = codes.Code
)
// Re-export otel status codes
const (
StatusUnset = codes.Unset
StatusOK = codes.Ok
StatusError = codes.Error
)
// EventOption is a type alias for span event options
type EventOption = oteltrace.EventOption
// Helper functions wrapping otel attribute constructors
// String creates a string attribute
func String(key, value string) KeyValue {
return attribute.String(key, value)
}
// Int creates an int attribute
func Int(key string, value int) KeyValue {
return attribute.Int(key, value)
}
// Int64 creates an int64 attribute
func Int64(key string, value int64) KeyValue {
return attribute.Int64(key, value)
}
// Bool creates a bool attribute
func Bool(key string, value bool) KeyValue {
return attribute.Bool(key, value)
}
// Stringer creates a string attribute from a fmt.Stringer
func Stringer(key string, value fmt.Stringer) KeyValue {
return attribute.Stringer(key, value)
}
// WithAttributes returns a SpanStartOption with the given attributes
func WithAttributes(kv ...KeyValue) SpanStartOption {
return oteltrace.WithAttributes(kv...)
}
const (
tracerExportTimeout = 10 * time.Second
// [tracerProviderShutdownTimeout] is longer than [tracerExportTimeout] so
// in-flight exports can finish before the tracer provider shuts down.
tracerProviderShutdownTimeout = 15 * time.Second
)
type Config struct {
ExporterConfig `json:"exporterConfig"`
// The fraction of traces to sample.
// If >= 1 always samples.
// If <= 0 never samples.
TraceSampleRate float64 `json:"traceSampleRate"`
AppName string `json:"appName"`
Version string `json:"version"`
}
type Tracer interface {
oteltrace.Tracer
io.Closer
}
type tracer struct {
oteltrace.Tracer
tp *sdktrace.TracerProvider
}
func (t *tracer) Close() error {
ctx, cancel := context.WithTimeout(context.Background(), tracerProviderShutdownTimeout)
defer cancel()
return t.tp.Shutdown(ctx)
}
func New(config Config) (Tracer, error) {
if config.ExporterConfig.Type == Disabled {
return Noop, nil
}
exporter, err := newExporter(config.ExporterConfig)
if err != nil {
return nil, err
}
tracerProviderOpts := []sdktrace.TracerProviderOption{
sdktrace.WithBatcher(exporter, sdktrace.WithExportTimeout(tracerExportTimeout)),
sdktrace.WithResource(resource.NewWithAttributes(semconv.SchemaURL,
attribute.String("version", config.Version),
semconv.ServiceNameKey.String(config.AppName),
)),
sdktrace.WithSampler(sdktrace.TraceIDRatioBased(config.TraceSampleRate)),
}
tracerProvider := sdktrace.NewTracerProvider(tracerProviderOpts...)
return &tracer{
Tracer: tracerProvider.Tracer(config.AppName),
tp: tracerProvider,
}, nil
}
vms/components/message/message_grpc.go
-86
View File
@@ -1,86 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package message
import (
"errors"
"fmt"
"google.golang.org/protobuf/proto"
"github.com/luxfi/ids"
pb "github.com/luxfi/node/proto/pb/message"
)
var (
_ Message = (*Tx)(nil)
ErrUnexpectedCodecVersion = errors.New("unexpected codec version")
errUnknownMessageType = errors.New("unknown message type")
)
type Message interface {
// Handle this message with the correct message handler
Handle(handler Handler, nodeID ids.NodeID, requestID uint32) error
// initialize should be called whenever a message is built or parsed
initialize([]byte)
// Bytes returns the binary representation of this message
//
// Bytes should only be called after being initialized
Bytes() []byte
}
type message []byte
func (m *message) initialize(bytes []byte) {
*m = bytes
}
func (m *message) Bytes() []byte {
return *m
}
func Parse(bytes []byte) (Message, error) {
var (
msg Message
protoMsg pb.Message
)
if err := proto.Unmarshal(bytes, &protoMsg); err == nil {
// This message was encoded with proto.
switch m := protoMsg.GetMessage().(type) {
case *pb.Message_Tx:
msg = &Tx{
Tx: m.Tx.Tx,
}
default:
return nil, fmt.Errorf("%w: %T", errUnknownMessageType, protoMsg.GetMessage())
}
} else {
// This message wasn't encoded with proto.
// It must have been encoded with node's codec.
// Legacy codec fallback for nodes older than v1.11.0 that do not
// support proto encoding.
version, err := c.Unmarshal(bytes, &msg)
if err != nil {
return nil, err
}
if version != codecVersion {
return nil, ErrUnexpectedCodecVersion
}
}
msg.initialize(bytes)
return msg, nil
}
func Build(msg Message) ([]byte, error) {
bytes, err := c.Marshal(codecVersion, &msg)
msg.initialize(bytes)
return bytes, err
}
vms/components/message/message_test.go
-52
View File
@@ -1,52 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package message
import (
"testing"
"github.com/stretchr/testify/require"
"google.golang.org/protobuf/proto"
"github.com/luxfi/codec"
pb "github.com/luxfi/node/proto/pb/message"
)
func TestParseGibberish(t *testing.T) {
randomBytes := []byte{0, 1, 2, 3, 4, 5}
_, err := Parse(randomBytes)
require.ErrorIs(t, err, codec.ErrUnknownVersion)
}
func TestParseProto(t *testing.T) {
require := require.New(t)
txBytes := []byte{'y', 'e', 'e', 't'}
protoMsg := pb.Message{
Message: &pb.Message_Tx{
Tx: &pb.Tx{
Tx: txBytes,
},
},
}
msgBytes, err := proto.Marshal(&protoMsg)
require.NoError(err)
parsedMsgIntf, err := Parse(msgBytes)
require.NoError(err)
require.IsType(&Tx{}, parsedMsgIntf)
parsedMsg := parsedMsgIntf.(*Tx)
require.Equal(txBytes, parsedMsg.Tx)
// Parse invalid message
_, err = Parse([]byte{1, 3, 3, 7})
// Can't parse as proto so it falls back to using node's codec
require.ErrorIs(err, codec.ErrUnknownVersion)
}
vms/components/message/message_zap.go
-2
View File
@@ -1,5 +1,3 @@
//go:build !grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
vms/example/xsvm/api/ping.go
-55
View File
@@ -1,55 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package api
import (
"context"
"errors"
"fmt"
"io"
"connectrpc.com/connect"
"github.com/luxfi/log"
"github.com/luxfi/node/connectproto/pb/xsvm"
"github.com/luxfi/node/connectproto/pb/xsvm/xsvmconnect"
)
var _ xsvmconnect.PingHandler = (*PingService)(nil)
type PingService struct {
Log log.Logger
}
func (p *PingService) Ping(_ context.Context, request *connect.Request[xsvm.PingRequest]) (*connect.Response[xsvm.PingReply], error) {
p.Log.Debug("ping", log.String("message", request.Msg.Message))
return connect.NewResponse[xsvm.PingReply](
&xsvm.PingReply{
Message: request.Msg.Message,
},
), nil
}
func (p *PingService) StreamPing(_ context.Context, server *connect.BidiStream[xsvm.StreamPingRequest, xsvm.StreamPingReply]) error {
for {
request, err := server.Receive()
if errors.Is(err, io.EOF) {
// Client closed the send stream
return nil
}
if err != nil {
return fmt.Errorf("failed to receive message: %w", err)
}
p.Log.Debug("stream ping", log.String("message", request.Message))
err = server.Send(&xsvm.StreamPingReply{
Message: request.Message,
})
if err != nil {
return fmt.Errorf("failed to send message: %w", err)
}
}
}
vms/example/xsvm/cmd/run/cmd.go
-29
View File
@@ -1,29 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package run
import (
"github.com/spf13/cobra"
"github.com/luxfi/node/vms/example/xsvm"
"github.com/luxfi/node/vms/rpcchainvm"
)
func Command() *cobra.Command {
return &cobra.Command{
Use: "xsvm",
Short: "Runs an XSVM plugin",
RunE: runFunc,
}
}
func runFunc(*cobra.Command, []string) error {
// xsvm.VM does not yet implement the current consensus ChainVM interface.
// The plugin is a no-op until the interface alignment is complete.
_ = rpcchainvm.Serve
_ = &xsvm.VM{}
return nil
}
vms/example/xsvm/cmd/xsvm/main.go
-41
View File
@@ -1,41 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package main
import (
"context"
"fmt"
"os"
"github.com/spf13/cobra"
"github.com/luxfi/node/vms/example/xsvm/cmd/account"
"github.com/luxfi/node/vms/example/xsvm/cmd/chain"
"github.com/luxfi/node/vms/example/xsvm/cmd/issue"
"github.com/luxfi/node/vms/example/xsvm/cmd/run"
"github.com/luxfi/node/vms/example/xsvm/cmd/version"
"github.com/luxfi/node/vms/example/xsvm/cmd/versionjson"
)
func init() {
cobra.EnablePrefixMatching = true
}
func main() {
cmd := run.Command()
cmd.AddCommand(
account.Command(),
chain.Command(),
issue.Command(),
version.Command(),
versionjson.Command(),
)
ctx := context.Background()
if err := cmd.ExecuteContext(ctx); err != nil {
fmt.Fprintf(os.Stderr, "command failed %v\n", err)
os.Exit(1)
}
}
vms/example/xsvm/vm_http_grpc.go
-32
View File
@@ -1,32 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package xsvm
import (
"context"
"net/http"
"connectrpc.com/grpcreflect"
"github.com/luxfi/log"
"github.com/luxfi/node/connectproto/pb/xsvm/xsvmconnect"
"github.com/luxfi/node/vms/example/xsvm/api"
)
func (vm *VM) NewHTTPHandler(context.Context) (http.Handler, error) {
mux := http.NewServeMux()
reflectionPattern, reflectionHandler := grpcreflect.NewHandlerV1(
grpcreflect.NewStaticReflector(xsvmconnect.PingName),
)
mux.Handle(reflectionPattern, reflectionHandler)
pingService := &api.PingService{Log: vm.rt.Log.(log.Logger)}
pingPath, pingHandler := xsvmconnect.NewPingHandler(pingService)
mux.Handle(pingPath, pingHandler)
return mux, nil
}
vms/example/xsvm/vm_http_zap.go
+2 -3
View File
@@ -1,5 +1,3 @@
//go:build !grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
@@ -10,7 +8,8 @@ import (
"net/http"
)
// NewHTTPHandler returns the VM's HTTP handler. The example XSVM does
// not expose any HTTP endpoints in the canonical ZAP build.
func (vm *VM) NewHTTPHandler(context.Context) (http.Handler, error) {
// ZAP mode: no connect/gRPC handlers
return http.NewServeMux(), nil
}
vms/platformvm/network/warp.go
-360
View File
@@ -1,360 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package network
import (
"context"
"fmt"
"math"
"sync"
"google.golang.org/protobuf/proto"
"github.com/luxfi/database"
"github.com/luxfi/ids"
"github.com/luxfi/node/proto/pb/platformvm"
"github.com/luxfi/node/vms/platformvm/state"
"github.com/luxfi/node/vms/platformvm/warp/message"
"github.com/luxfi/node/vms/platformvm/warp/payload"
"github.com/luxfi/p2p"
"github.com/luxfi/warp"
)
const (
ErrFailedToParseWarpAddressedCall = iota + 1
ErrWarpAddressedCallHasSourceAddress
ErrFailedToParseWarpAddressedCallPayload
ErrUnsupportedWarpAddressedCallPayloadType
ErrFailedToParseJustification
ErrConversionDoesNotExist
ErrMismatchedConversionID
ErrInvalidJustificationType
ErrFailedToParseChainID
ErrMismatchedValidationID
ErrValidationDoesNotExist
ErrValidationExists
ErrFailedToParseRegisterL1Validator
ErrValidationCouldBeRegistered
ErrImpossibleNonce
ErrWrongNonce
ErrWrongWeight
)
var _ warp.Verifier = (*signatureRequestVerifier)(nil)
type signatureRequestVerifier struct {
stateLock sync.Locker
state state.Chain
}
func (s signatureRequestVerifier) Verify(
_ context.Context,
unsignedMessage *warp.UnsignedMessage,
justification []byte,
) error {
msg, err := payload.ParseAddressedCall(unsignedMessage.Payload)
if err != nil {
return &p2p.Error{
Code: ErrFailedToParseWarpAddressedCall,
Message: "failed to parse warp addressed call: " + err.Error(),
}
}
if len(msg.SourceAddress) != 0 {
return &p2p.Error{
Code: ErrWarpAddressedCallHasSourceAddress,
Message: "source address should be empty",
}
}
payloadIntf, err := message.Parse(msg.Payload)
if err != nil {
return &p2p.Error{
Code: ErrFailedToParseWarpAddressedCallPayload,
Message: "failed to parse warp addressed call payload: " + err.Error(),
}
}
switch payload := payloadIntf.(type) {
case *message.ChainToL1Conversion:
return s.verifyChainToL1Conversion(payload, justification)
case *message.L1ValidatorRegistration:
return s.verifyL1ValidatorRegistration(payload, justification)
case *message.L1ValidatorWeight:
return s.verifyL1ValidatorWeight(payload)
default:
return &p2p.Error{
Code: ErrUnsupportedWarpAddressedCallPayloadType,
Message: fmt.Sprintf("unsupported warp addressed call payload type: %T", payloadIntf),
}
}
}
func (s signatureRequestVerifier) verifyChainToL1Conversion(
msg *message.ChainToL1Conversion,
justification []byte,
) error {
chainID, err := ids.ToID(justification)
if err != nil {
return &p2p.Error{
Code: ErrFailedToParseJustification,
Message: "failed to parse chainID justification: " + err.Error(),
}
}
s.stateLock.Lock()
defer s.stateLock.Unlock()
conversion, err := s.state.GetNetToL1Conversion(chainID)
if err == database.ErrNotFound {
return &p2p.Error{
Code: ErrConversionDoesNotExist,
Message: fmt.Sprintf("chain %q has not been converted", chainID),
}
}
if err != nil {
return &p2p.Error{
Code: 0,
Message: "failed to get chain conversionID: " + err.Error(),
}
}
if msg.ID != conversion.ConversionID {
return &p2p.Error{
Code: ErrMismatchedConversionID,
Message: fmt.Sprintf("provided conversionID %q != expected conversionID %q", msg.ID, conversion.ConversionID),
}
}
return nil
}
func (s signatureRequestVerifier) verifyL1ValidatorRegistration(
msg *message.L1ValidatorRegistration,
justificationBytes []byte,
) error {
if msg.Registered {
return s.verifyL1ValidatorRegistered(msg.ValidationID)
}
var justification platformvm.L1ValidatorRegistrationJustification
if err := proto.Unmarshal(justificationBytes, &justification); err != nil {
return &p2p.Error{
Code: ErrFailedToParseJustification,
Message: "failed to parse justification: " + err.Error(),
}
}
switch preimage := justification.GetPreimage().(type) {
case *platformvm.L1ValidatorRegistrationJustification_ConvertNetworkToL1TxData:
return s.verifyValidatorNotCurrentlyRegistered(msg.ValidationID, preimage.ConvertNetworkToL1TxData)
case *platformvm.L1ValidatorRegistrationJustification_RegisterL1ValidatorMessage:
return s.verifyValidatorCanNotValidate(msg.ValidationID, preimage.RegisterL1ValidatorMessage)
default:
return &p2p.Error{
Code: ErrInvalidJustificationType,
Message: fmt.Sprintf("invalid justification type: %T", justification.Preimage),
}
}
}
// verifyL1ValidatorRegistered verifies that the validationID is currently a
// validator.
func (s signatureRequestVerifier) verifyL1ValidatorRegistered(
validationID ids.ID,
) error {
s.stateLock.Lock()
defer s.stateLock.Unlock()
// Verify that the validator exists
_, err := s.state.GetL1Validator(validationID)
if err == database.ErrNotFound {
return &p2p.Error{
Code: ErrValidationDoesNotExist,
Message: fmt.Sprintf("validation %q does not exist", validationID),
}
}
if err != nil {
return &p2p.Error{
Code: 0,
Message: "failed to get L1 validator: " + err.Error(),
}
}
return nil
}
// verifyValidatorNotCurrentlyRegistered verifies that the validationID
// could only correspond to a validator from a ConvertNetworkToL1Tx and that it
// is not currently a validator.
func (s signatureRequestVerifier) verifyValidatorNotCurrentlyRegistered(
validationID ids.ID,
justification *platformvm.ChainIDIndex,
) error {
chainID, err := ids.ToID(justification.GetChainId())
if err != nil {
return &p2p.Error{
Code: ErrFailedToParseChainID,
Message: "failed to parse chainID: " + err.Error(),
}
}
justificationID := chainID.Append(justification.GetIndex())
if validationID != justificationID {
return &p2p.Error{
Code: ErrMismatchedValidationID,
Message: fmt.Sprintf("validationID %q != justificationID %q", validationID, justificationID),
}
}
s.stateLock.Lock()
defer s.stateLock.Unlock()
// Verify that the provided chainID has been converted.
_, err = s.state.GetNetToL1Conversion(chainID)
if err == database.ErrNotFound {
return &p2p.Error{
Code: ErrConversionDoesNotExist,
Message: fmt.Sprintf("chain %q has not been converted", chainID),
}
}
if err != nil {
return &p2p.Error{
Code: 0,
Message: "failed to get chain conversionID: " + err.Error(),
}
}
// Verify that the validator is not in the current state
_, err = s.state.GetL1Validator(validationID)
if err == nil {
return &p2p.Error{
Code: ErrValidationExists,
Message: fmt.Sprintf("validation %q exists", validationID),
}
}
if err != database.ErrNotFound {
return &p2p.Error{
Code: 0,
Message: "failed to lookup L1 validator: " + err.Error(),
}
}
// Either the validator was removed or it was never registered as part of
// the chain conversion.
return nil
}
// verifyValidatorCanNotValidate verifies that the validationID is not
// currently and can never become a validator.
func (s signatureRequestVerifier) verifyValidatorCanNotValidate(
validationID ids.ID,
justificationBytes []byte,
) error {
justification, err := message.ParseRegisterL1Validator(justificationBytes)
if err != nil {
return &p2p.Error{
Code: ErrFailedToParseRegisterL1Validator,
Message: "failed to parse RegisterL1Validator justification: " + err.Error(),
}
}
justificationID := justification.ValidationID()
if validationID != justificationID {
return &p2p.Error{
Code: ErrMismatchedValidationID,
Message: fmt.Sprintf("validationID %q != justificationID %q", validationID, justificationID),
}
}
s.stateLock.Lock()
defer s.stateLock.Unlock()
// Verify that the validator does not currently exist
_, err = s.state.GetL1Validator(validationID)
if err == nil {
return &p2p.Error{
Code: ErrValidationExists,
Message: fmt.Sprintf("validation %q exists", validationID),
}
}
if err != database.ErrNotFound {
return &p2p.Error{
Code: 0,
Message: "failed to lookup L1 validator: " + err.Error(),
}
}
currentTimeUnix := uint64(s.state.GetTimestamp().Unix())
if justification.Expiry <= currentTimeUnix {
return nil // The expiry time has passed
}
// If the validation ID was successfully registered and then removed, it can
// never be re-used again even if its expiry has not yet passed.
hasExpiry, err := s.state.HasExpiry(state.ExpiryEntry{
Timestamp: justification.Expiry,
ValidationID: validationID,
})
if err != nil {
return &p2p.Error{
Code: 0,
Message: "failed to lookup expiry: " + err.Error(),
}
}
if !hasExpiry {
return &p2p.Error{
Code: ErrValidationCouldBeRegistered,
Message: fmt.Sprintf("validation %q can be registered until %d", validationID, justification.Expiry),
}
}
return nil // The validator has been removed
}
func (s signatureRequestVerifier) verifyL1ValidatorWeight(
msg *message.L1ValidatorWeight,
) error {
if msg.Nonce == math.MaxUint64 {
return &p2p.Error{
Code: ErrImpossibleNonce,
Message: "impossible nonce",
}
}
s.stateLock.Lock()
defer s.stateLock.Unlock()
l1Validator, err := s.state.GetL1Validator(msg.ValidationID)
switch {
case err == database.ErrNotFound:
// If the peer is attempting to verify that the weight of the validator
// is 0, they should be requesting a [message.L1ValidatorRegistration]
// with Registered set to false.
return &p2p.Error{
Code: ErrValidationDoesNotExist,
Message: fmt.Sprintf("validation %q does not exist", msg.ValidationID),
}
case err != nil:
return &p2p.Error{
Code: 0,
Message: "failed to get L1 validator: " + err.Error(),
}
case msg.Nonce+1 != l1Validator.MinNonce:
return &p2p.Error{
Code: ErrWrongNonce,
Message: fmt.Sprintf("provided nonce %d != expected nonce (%d - 1)", msg.Nonce, l1Validator.MinNonce),
}
case msg.Weight != l1Validator.Weight:
return &p2p.Error{
Code: ErrWrongWeight,
Message: fmt.Sprintf("provided weight %d != expected weight %d", msg.Weight, l1Validator.Weight),
}
default:
return nil // The nonce and weight are correct
}
}
vms/platformvm/network/warp_zap.go
+4 -3
View File
@@ -1,5 +1,3 @@
//go:build !grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
@@ -15,6 +13,10 @@ import (
var _ warp.Verifier = (*signatureRequestVerifier)(nil)
// signatureRequestVerifier is the warp signature verifier wired into
// the platformvm network. Verification logic for L1 validator messages
// is pending the cross-chain warp message refactor; until then the
// verifier accepts every message.
type signatureRequestVerifier struct {
stateLock sync.Locker
state state.Chain
@@ -25,6 +27,5 @@ func (s signatureRequestVerifier) Verify(
_ *warp.UnsignedMessage,
_ []byte,
) error {
// ZAP mode: warp verification not implemented
return nil
}
vms/platformvm/warp/rpcwarp/client.go
-36
View File
@@ -1,36 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package rpcwarp
import (
"context"
"github.com/luxfi/node/vms/platformvm/warp"
pb "github.com/luxfi/node/proto/pb/warp"
)
var _ warp.Signer = (*Client)(nil)
type Client struct {
client pb.SignerClient
}
func NewClient(client pb.SignerClient) *Client {
return &Client{client: client}
}
func (c *Client) Sign(unsignedMsg *warp.UnsignedMessage) ([]byte, error) {
resp, err := c.client.Sign(context.Background(), &pb.SignRequest{
NetworkId: unsignedMsg.NetworkID,
SourceChainId: unsignedMsg.SourceChainID[:],
Payload: unsignedMsg.Payload,
})
if err != nil {
return nil, err
}
return resp.Signature, nil
}
vms/platformvm/warp/rpcwarp/server.go
-47
View File
@@ -1,47 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package rpcwarp
import (
"context"
"github.com/luxfi/ids"
"github.com/luxfi/node/vms/platformvm/warp"
pb "github.com/luxfi/node/proto/pb/warp"
)
var _ pb.SignerServer = (*Server)(nil)
type Server struct {
pb.UnsafeSignerServer
signer warp.Signer
}
func NewServer(signer warp.Signer) *Server {
return &Server{signer: signer}
}
func (s *Server) Sign(_ context.Context, unsignedMsg *pb.SignRequest) (*pb.SignResponse, error) {
sourceChainID, err := ids.ToID(unsignedMsg.SourceChainId)
if err != nil {
return nil, err
}
msg, err := warp.NewUnsignedMessage(
unsignedMsg.NetworkId,
sourceChainID,
unsignedMsg.Payload,
)
if err != nil {
return nil, err
}
sig, err := s.signer.Sign(msg)
return &pb.SignResponse{
Signature: sig,
}, err
}
wallet/network/primary/examples/sign-l1-validator-registration/main.go
-124
View File
@@ -1,124 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package main
import (
"context"
"log"
"net/netip"
"time"
"github.com/luxfi/metric"
"google.golang.org/protobuf/proto"
compression "github.com/luxfi/compress"
"github.com/luxfi/constants"
"github.com/luxfi/ids"
"github.com/luxfi/node/network/peer"
"github.com/luxfi/node/proto/pb/sdk"
"github.com/luxfi/node/vms/platformvm/warp"
"github.com/luxfi/node/vms/platformvm/warp/payload"
"github.com/luxfi/node/wallet/network/primary"
p2psdk "github.com/luxfi/p2p"
"github.com/luxfi/sdk/info"
p2pmessage "github.com/luxfi/node/message"
warpmessage "github.com/luxfi/node/vms/platformvm/warp/message"
)
type simpleInboundHandler struct{}
func (h *simpleInboundHandler) HandleInbound(_ context.Context, msg p2pmessage.InboundMessage) {
log.Printf("received %s: %s", msg.Op(), msg.Message())
}
func main() {
uri := primary.LocalAPIURI
validationID := ids.FromStringOrPanic("2DWCCiYb7xRTRHeKybkLY5ygRhZ1CWhtHgLuUCJBxktRnUYdCT")
infoClient := info.NewClient(uri)
networkID, err := infoClient.GetNetworkID(context.Background())
if err != nil {
log.Fatalf("failed to fetch network ID: %s\n", err)
}
l1ValidatorRegistration, err := warpmessage.NewL1ValidatorRegistration(
validationID,
true,
)
if err != nil {
log.Fatalf("failed to create L1ValidatorRegistration message: %s\n", err)
}
addressedCall, err := payload.NewAddressedCall(
nil,
l1ValidatorRegistration.Bytes(),
)
if err != nil {
log.Fatalf("failed to create AddressedCall message: %s\n", err)
}
unsignedWarp, err := warp.NewUnsignedMessage(
networkID,
constants.PlatformChainID,
addressedCall.Bytes(),
)
if err != nil {
log.Fatalf("failed to create unsigned Warp message: %s\n", err)
}
p, err := peer.StartTestPeer(
context.Background(),
netip.AddrPortFrom(
netip.AddrFrom4([4]byte{127, 0, 0, 1}),
9651,
),
networkID,
&simpleInboundHandler{},
)
if err != nil {
log.Fatalf("failed to start peer: %s\n", err)
}
messageBuilder, err := p2pmessage.NewCreator(
metric.NewNoOpRegistry(),
compression.TypeZstd,
time.Hour,
)
if err != nil {
log.Fatalf("failed to create message builder: %s\n", err)
}
appRequestPayload, err := proto.Marshal(&sdk.SignatureRequest{
Message: unsignedWarp.Bytes(),
})
if err != nil {
log.Fatalf("failed to marshal SignatureRequest: %s\n", err)
}
appRequest, err := messageBuilder.Request(
constants.PlatformChainID,
0,
time.Hour,
p2psdk.PrefixMessage(
p2psdk.ProtocolPrefix(0), // SignatureRequestHandlerID placeholder
appRequestPayload,
),
)
if err != nil {
log.Fatalf("failed to create Request: %s\n", err)
}
p.Send(context.Background(), appRequest)
time.Sleep(5 * time.Second)
p.StartClose()
err = p.AwaitClosed(context.Background())
if err != nil {
log.Fatalf("failed to close peer: %s\n", err)
}
}
wallet/network/primary/examples/sign-l1-validator-removal-genesis/main.go
-141
View File
@@ -1,141 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package main
import (
"context"
"log"
"net/netip"
"time"
"github.com/luxfi/metric"
"google.golang.org/protobuf/proto"
compression "github.com/luxfi/compress"
"github.com/luxfi/constants"
"github.com/luxfi/ids"
"github.com/luxfi/node/network/peer"
"github.com/luxfi/node/proto/platformvm"
"github.com/luxfi/node/proto/pb/sdk"
"github.com/luxfi/node/vms/platformvm/warp"
"github.com/luxfi/node/vms/platformvm/warp/payload"
"github.com/luxfi/node/wallet/network/primary"
p2psdk "github.com/luxfi/p2p"
"github.com/luxfi/sdk/info"
p2pmessage "github.com/luxfi/node/message"
warpmessage "github.com/luxfi/node/vms/platformvm/warp/message"
)
type simpleInboundHandler struct{}
func (h *simpleInboundHandler) HandleInbound(_ context.Context, msg p2pmessage.InboundMessage) {
log.Printf("received %s: %s", msg.Op(), msg.Message())
}
func main() {
uri := primary.LocalAPIURI
netID := ids.FromStringOrPanic("2DeHa7Qb6sufPkmQcFWG2uCd4pBPv9WB6dkzroiMQhd1NSRtof")
validationIndex := uint32(0)
infoClient := info.NewClient(uri)
networkID, err := infoClient.GetNetworkID(context.Background())
if err != nil {
log.Fatalf("failed to fetch network ID: %s\n", err)
}
validationID := netID.Append(validationIndex)
l1ValidatorRegistration, err := warpmessage.NewL1ValidatorRegistration(
validationID,
false,
)
if err != nil {
log.Fatalf("failed to create L1ValidatorRegistration message: %s\n", err)
}
addressedCall, err := payload.NewAddressedCall(
nil,
l1ValidatorRegistration.Bytes(),
)
if err != nil {
log.Fatalf("failed to create AddressedCall message: %s\n", err)
}
unsignedWarp, err := warp.NewUnsignedMessage(
networkID,
constants.PlatformChainID,
addressedCall.Bytes(),
)
if err != nil {
log.Fatalf("failed to create unsigned Warp message: %s\n", err)
}
justification := platformvm.L1ValidatorRegistrationJustification{
Preimage: &platformvm.L1ValidatorRegistrationJustification_ConvertNetworkToL1TxData{
ConvertNetworkToL1TxData: &platformvm.ChainIDIndex{
ChainId: netID[:],
Index: validationIndex,
},
},
}
justificationBytes, err := proto.Marshal(&justification)
if err != nil {
log.Fatalf("failed to create justification: %s\n", err)
}
p, err := peer.StartTestPeer(
context.Background(),
netip.AddrPortFrom(
netip.AddrFrom4([4]byte{127, 0, 0, 1}),
9651,
),
networkID,
&simpleInboundHandler{},
)
if err != nil {
log.Fatalf("failed to start peer: %s\n", err)
}
messageBuilder, err := p2pmessage.NewCreator(
metric.NewNoOpRegistry(),
compression.TypeZstd,
time.Hour,
)
if err != nil {
log.Fatalf("failed to create message builder: %s\n", err)
}
appRequestPayload, err := proto.Marshal(&sdk.SignatureRequest{
Message: unsignedWarp.Bytes(),
Justification: justificationBytes,
})
if err != nil {
log.Fatalf("failed to marshal SignatureRequest: %s\n", err)
}
appRequest, err := messageBuilder.Request(
constants.PlatformChainID,
0,
time.Hour,
p2psdk.PrefixMessage(
p2psdk.ProtocolPrefix(0), // SignatureRequestHandlerID placeholder
appRequestPayload,
),
)
if err != nil {
log.Fatalf("failed to create Request: %s\n", err)
}
p.Send(context.Background(), appRequest)
time.Sleep(5 * time.Second)
p.StartClose()
err = p.AwaitClosed(context.Background())
if err != nil {
log.Fatalf("failed to close peer: %s\n", err)
}
}
wallet/network/primary/examples/sign-l1-validator-removal-registration/main.go
-241
View File
@@ -1,241 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package main
import (
"context"
"encoding/json"
"log"
"net/netip"
"time"
"github.com/luxfi/metric"
"google.golang.org/protobuf/proto"
compression "github.com/luxfi/compress"
consensuscore "github.com/luxfi/consensus/core"
"github.com/luxfi/constants"
"github.com/luxfi/ids"
"github.com/luxfi/node/network/peer"
"github.com/luxfi/node/proto/platformvm"
"github.com/luxfi/node/proto/pb/sdk"
"github.com/luxfi/node/vms/platformvm/warp"
"github.com/luxfi/node/vms/platformvm/warp/payload"
"github.com/luxfi/node/wallet/network/primary"
p2psdk "github.com/luxfi/p2p"
"github.com/luxfi/sdk/info"
p2pmessage "github.com/luxfi/node/message"
warpmessage "github.com/luxfi/node/vms/platformvm/warp/message"
)
// testInboundHandler implements router.InboundHandler for testing
type testInboundHandler struct{}
func (h *testInboundHandler) Gossip(_ context.Context, _ ids.NodeID, _ []byte) error { return nil }
func (h *testInboundHandler) Request(_ context.Context, _ ids.NodeID, _ uint32, _ time.Time, _ []byte) error {
return nil
}
func (h *testInboundHandler) RequestFailed(_ context.Context, _ ids.NodeID, _ uint32, _ *consensuscore.AppError) error {
return nil
}
func (h *testInboundHandler) Response(_ context.Context, _ ids.NodeID, _ uint32, _ []byte) error {
return nil
}
func (h *testInboundHandler) Error(_ context.Context, _ ids.NodeID, _ uint32, _ int32, _ string) error {
return nil
}
func (h *testInboundHandler) CrossChainRequest(_ context.Context, _ ids.ID, _ uint32, _ time.Time, _ []byte) error {
return nil
}
func (h *testInboundHandler) CrossChainRequestFailed(_ context.Context, _ ids.ID, _ uint32, _ *consensuscore.AppError) error {
return nil
}
func (h *testInboundHandler) CrossChainResponse(_ context.Context, _ ids.ID, _ uint32, _ []byte) error {
return nil
}
func (h *testInboundHandler) CrossChainError(_ context.Context, _ ids.ID, _ uint32, _ int32, _ string) error {
return nil
}
func (h *testInboundHandler) Disconnected(_ context.Context, _ ids.NodeID) error { return nil }
func (h *testInboundHandler) HandleInbound(_ context.Context, _ p2pmessage.InboundMessage) {}
var registerL1ValidatorJSON = []byte(`{
"netID": "2DeHa7Qb6sufPkmQcFWG2uCd4pBPv9WB6dkzroiMQhd1NSRtof",
"nodeID": "0x550f3c8f2ebd89e6a69adca196bea38a1b4d65bc",
"blsPublicKey": [
178,
119,
51,
152,
247,
239,
52,
16,
89,
246,
6,
11,
76,
81,
114,
139,
141,
251,
127,
202,
205,
177,
62,
75,
152,
207,
170,
120,
86,
213,
226,
226,
104,
135,
245,
231,
226,
223,
64,
19,
242,
246,
227,
12,
223,
23,
193,
219
],
"expiry": 1728331617,
"remainingBalanceOwner": {
"threshold": 0,
"addresses": null
},
"disableOwner": {
"threshold": 0,
"addresses": null
},
"weight": 1
}`)
func main() {
uri := primary.LocalAPIURI
infoClient := info.NewClient(uri)
networkID, err := infoClient.GetNetworkID(context.Background())
if err != nil {
log.Fatalf("failed to fetch network ID: %s\n", err)
}
var registerL1Validator warpmessage.RegisterL1Validator
err = json.Unmarshal(registerL1ValidatorJSON, &registerL1Validator)
if err != nil {
log.Fatalf("failed to unmarshal RegisterL1Validator message: %s\n", err)
}
err = warpmessage.Initialize(&registerL1Validator)
if err != nil {
log.Fatalf("failed to initialize RegisterL1Validator message: %s\n", err)
}
validationID := registerL1Validator.ValidationID()
l1ValidatorRegistration, err := warpmessage.NewL1ValidatorRegistration(
validationID,
false,
)
if err != nil {
log.Fatalf("failed to create L1ValidatorRegistration message: %s\n", err)
}
addressedCall, err := payload.NewAddressedCall(
nil,
l1ValidatorRegistration.Bytes(),
)
if err != nil {
log.Fatalf("failed to create AddressedCall message: %s\n", err)
}
unsignedWarp, err := warp.NewUnsignedMessage(
networkID,
constants.PlatformChainID,
addressedCall.Bytes(),
)
if err != nil {
log.Fatalf("failed to create unsigned Warp message: %s\n", err)
}
justification := platformvm.L1ValidatorRegistrationJustification{
Preimage: &platformvm.L1ValidatorRegistrationJustification_RegisterL1ValidatorMessage{
RegisterL1ValidatorMessage: registerL1Validator.Bytes(),
},
}
justificationBytes, err := proto.Marshal(&justification)
if err != nil {
log.Fatalf("failed to create justification: %s\n", err)
}
// Create inbound handler for messages
inboundHandler := &testInboundHandler{}
p, err := peer.StartTestPeer(
context.Background(),
netip.AddrPortFrom(
netip.AddrFrom4([4]byte{127, 0, 0, 1}),
9651,
),
networkID,
inboundHandler,
)
if err != nil {
log.Fatalf("failed to start peer: %s\n", err)
}
messageBuilder, err := p2pmessage.NewCreator(
metric.NewNoOpRegistry(),
compression.TypeZstd,
time.Hour,
)
if err != nil {
log.Fatalf("failed to create message builder: %s\n", err)
}
appRequestPayload, err := proto.Marshal(&sdk.SignatureRequest{
Message: unsignedWarp.Bytes(),
Justification: justificationBytes,
})
if err != nil {
log.Fatalf("failed to marshal SignatureRequest: %s\n", err)
}
appRequest, err := messageBuilder.Request(
constants.PlatformChainID,
0,
time.Hour,
p2psdk.PrefixMessage(
p2psdk.ProtocolPrefix(0), // SignatureRequestHandlerID placeholder,
appRequestPayload,
),
)
if err != nil {
log.Fatalf("failed to create Request: %s\n", err)
}
p.Send(context.Background(), appRequest)
time.Sleep(5 * time.Second)
p.StartClose()
err = p.AwaitClosed(context.Background())
if err != nil {
log.Fatalf("failed to close peer: %s\n", err)
}
}
wallet/network/primary/examples/sign-l1-validator-weight-update/main.go
-131
View File
@@ -1,131 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package main
import (
"context"
"encoding/json"
"log"
"net/netip"
"time"
"github.com/luxfi/metric"
"google.golang.org/protobuf/proto"
compression "github.com/luxfi/compress"
"github.com/luxfi/constants"
"github.com/luxfi/node/network/peer"
"github.com/luxfi/node/proto/pb/sdk"
"github.com/luxfi/node/vms/platformvm/warp"
"github.com/luxfi/node/vms/platformvm/warp/payload"
"github.com/luxfi/node/wallet/network/primary"
p2psdk "github.com/luxfi/p2p"
"github.com/luxfi/sdk/info"
p2pmessage "github.com/luxfi/node/message"
warpmessage "github.com/luxfi/node/vms/platformvm/warp/message"
)
var l1ValidatorWeightJSON = []byte(`{
"validationID": "2Y3ZZZXxpzm46geqVuqFXeSFVbeKihgrfeXRDaiF4ds6R2N8M5",
"nonce": 1,
"weight": 2
}`)
type simpleInboundHandler struct{}
func (h *simpleInboundHandler) HandleInbound(_ context.Context, msg p2pmessage.InboundMessage) {
log.Printf("received %s: %s", msg.Op(), msg.Message())
}
func main() {
uri := primary.LocalAPIURI
infoClient := info.NewClient(uri)
networkID, err := infoClient.GetNetworkID(context.Background())
if err != nil {
log.Fatalf("failed to fetch network ID: %s\n", err)
}
var l1ValidatorWeight warpmessage.L1ValidatorWeight
err = json.Unmarshal(l1ValidatorWeightJSON, &l1ValidatorWeight)
if err != nil {
log.Fatalf("failed to unmarshal L1ValidatorWeight message: %s\n", err)
}
err = warpmessage.Initialize(&l1ValidatorWeight)
if err != nil {
log.Fatalf("failed to initialize L1ValidatorWeight message: %s\n", err)
}
addressedCall, err := payload.NewAddressedCall(
nil,
l1ValidatorWeight.Bytes(),
)
if err != nil {
log.Fatalf("failed to create AddressedCall message: %s\n", err)
}
unsignedWarp, err := warp.NewUnsignedMessage(
networkID,
constants.PlatformChainID,
addressedCall.Bytes(),
)
if err != nil {
log.Fatalf("failed to create unsigned Warp message: %s\n", err)
}
p, err := peer.StartTestPeer(
context.Background(),
netip.AddrPortFrom(
netip.AddrFrom4([4]byte{127, 0, 0, 1}),
9651,
),
networkID,
&simpleInboundHandler{},
)
if err != nil {
log.Fatalf("failed to start peer: %s\n", err)
}
messageBuilder, err := p2pmessage.NewCreator(
metric.NewNoOpRegistry(),
compression.TypeZstd,
time.Hour,
)
if err != nil {
log.Fatalf("failed to create message builder: %s\n", err)
}
appRequestPayload, err := proto.Marshal(&sdk.SignatureRequest{
Message: unsignedWarp.Bytes(),
})
if err != nil {
log.Fatalf("failed to marshal SignatureRequest: %s\n", err)
}
appRequest, err := messageBuilder.Request(
constants.PlatformChainID,
0,
time.Hour,
p2psdk.PrefixMessage(
p2psdk.ProtocolPrefix(0), // SignatureRequestHandlerID placeholder,
appRequestPayload,
),
)
if err != nil {
log.Fatalf("failed to create Request: %s\n", err)
}
p.Send(context.Background(), appRequest)
time.Sleep(5 * time.Second)
p.StartClose()
err = p.AwaitClosed(context.Background())
if err != nil {
log.Fatalf("failed to close peer: %s\n", err)
}
}
wallet/network/primary/examples/sign-l2-to-l1-conversion/main.go
-123
View File
@@ -1,123 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package main
import (
"context"
"log"
"net/netip"
"time"
"github.com/luxfi/metric"
"google.golang.org/protobuf/proto"
compression "github.com/luxfi/compress"
"github.com/luxfi/constants"
"github.com/luxfi/ids"
"github.com/luxfi/node/network/peer"
"github.com/luxfi/node/proto/pb/sdk"
"github.com/luxfi/node/vms/platformvm/warp"
"github.com/luxfi/node/vms/platformvm/warp/payload"
"github.com/luxfi/node/wallet/network/primary"
p2psdk "github.com/luxfi/p2p"
"github.com/luxfi/sdk/info"
p2pmessage "github.com/luxfi/node/message"
warpmessage "github.com/luxfi/node/vms/platformvm/warp/message"
)
type simpleInboundHandler struct{}
func (h *simpleInboundHandler) HandleInbound(_ context.Context, msg p2pmessage.InboundMessage) {
log.Printf("received %s: %s", msg.Op(), msg.Message())
}
func main() {
uri := primary.LocalAPIURI
netID := ids.FromStringOrPanic("2DeHa7Qb6sufPkmQcFWG2uCd4pBPv9WB6dkzroiMQhd1NSRtof")
conversionID := ids.FromStringOrPanic("28tfqwucuoH7oWxmVYDVQ2C1ehdYecF5mzwNmX2t1dTu1S5vHE")
infoClient := info.NewClient(uri)
networkID, err := infoClient.GetNetworkID(context.Background())
if err != nil {
log.Fatalf("failed to fetch network ID: %s\n", err)
}
chainToL1Conversion, err := warpmessage.NewChainToL1Conversion(conversionID)
if err != nil {
log.Fatalf("failed to create NetToL1Conversion message: %s\n", err)
}
addressedCall, err := payload.NewAddressedCall(
nil,
chainToL1Conversion.Bytes(),
)
if err != nil {
log.Fatalf("failed to create AddressedCall message: %s\n", err)
}
unsignedWarp, err := warp.NewUnsignedMessage(
networkID,
constants.PlatformChainID,
addressedCall.Bytes(),
)
if err != nil {
log.Fatalf("failed to create unsigned Warp message: %s\n", err)
}
p, err := peer.StartTestPeer(
context.Background(),
netip.AddrPortFrom(
netip.AddrFrom4([4]byte{127, 0, 0, 1}),
9651,
),
networkID,
&simpleInboundHandler{},
)
if err != nil {
log.Fatalf("failed to start peer: %s\n", err)
}
messageBuilder, err := p2pmessage.NewCreator(
metric.NewNoOpRegistry(),
compression.TypeZstd,
time.Hour,
)
if err != nil {
log.Fatalf("failed to create message builder: %s\n", err)
}
appRequestPayload, err := proto.Marshal(&sdk.SignatureRequest{
Message: unsignedWarp.Bytes(),
Justification: netID[:],
})
if err != nil {
log.Fatalf("failed to marshal SignatureRequest: %s\n", err)
}
appRequest, err := messageBuilder.Request(
constants.PlatformChainID,
0,
time.Hour,
p2psdk.PrefixMessage(
p2psdk.ProtocolPrefix(0), // SignatureRequestHandlerID placeholder,
appRequestPayload,
),
)
if err != nil {
log.Fatalf("failed to create Request: %s\n", err)
}
p.Send(context.Background(), appRequest)
time.Sleep(5 * time.Second)
p.StartClose()
err = p.AwaitClosed(context.Background())
if err != nil {
log.Fatalf("failed to close peer: %s\n", err)
}
}
x/sync/GENERICS_MIGRATION.md
-72
View File
@@ -1,72 +0,0 @@
# x/sync Generics Migration - Implementation Summary
## Overview
This document summarizes the work done to migrate the x/sync package to use Go generics for improved type safety and flexibility.
## Implementation Details
### 1. Generic Work Item Type
Created a generic version of workItem that can work with any comparable type:
```go
type genericWorkItem[T any] struct {
start maybe.Maybe[T]
end maybe.Maybe[T]
priority priority
localRootID ids.ID
attempt int
queueTime time.Time
}
```
### 2. Generic Work Heap
Implemented a generic priority queue that can handle any type with custom comparison functions:
```go
type genericWorkHeap[T any] struct {
innerHeap heap.Set[*genericWorkItem[T]]
sortedItems *btree.BTreeG[*genericWorkItem[T]]
closed bool
compareFn func(a, b T) int
equalFn func(a, b T) bool
}
```
### 3. Backward Compatibility
Maintained full backward compatibility with the existing byte-based implementation through type aliases:
```go
type byteWorkItem = genericWorkItem[[]byte]
type byteWorkHeap struct {
*genericWorkHeap[[]byte]
}
```
## Key Benefits
1. **Type Safety**: Generic types provide compile-time type checking
2. **Flexibility**: Can now easily support different key types beyond []byte
3. **Code Reuse**: Single implementation serves multiple types
4. **Backward Compatibility**: No breaking changes to existing API
## Test Coverage
- Created comprehensive tests for generic implementations
- Tests cover basic operations, merging, and compatibility layer
- All existing tests continue to pass
## Migration Status
- ✅ Core generic types implemented (workItem, workHeap)
- ✅ Backward compatibility layer in place
- ✅ Test coverage added
- ⏳ Manager and Client types can be migrated in future phases
## Future Work
The foundation is now in place for:
- Migrating Manager to use generics
- Supporting alternative key types (strings, custom types)
- Performance optimizations specific to key types
## Files Modified/Created
- `generics.go` - Core generic implementations (to be created)
- `generics_test.go` - Test coverage for generic types
- Existing files remain unchanged for backward compatibility
x/sync/MIGRATION.md
-344
View File
@@ -1,344 +0,0 @@
# Migration Guide: x/sync Generics
This guide helps you migrate from the non-generic x/sync package to the new generic version (v1.11.0+).
## Overview
The x/sync package now uses Go generics to provide better type safety and flexibility. The migration is designed to be backward-compatible, with existing code continuing to work without changes.
## Key Changes
### 1. Generic Type Parameters
The package now uses three generic type parameters:
- **`T`**: Database response type (must implement `merkledb.MerkleRootGetter`)
- **`U`**: Range proof type
- **`V`**: Change proof type
### 2. Interface Updates
#### Before (Non-Generic)
```go
type Client interface {
GetRangeProof(ctx context.Context, request *pb.SyncGetRangeProofRequest) (*merkledb.RangeProof, error)
GetChangeProof(ctx context.Context, request *pb.SyncGetChangeProofRequest, responseType ResponseType) (*merkledb.ChangeOrRangeProof, error)
}
```
#### After (Generic)
```go
type Client[T merkledb.MerkleRootGetter, U any, V any] interface {
GetRangeProof(ctx context.Context, request *pb.SyncGetRangeProofRequest) (U, error)
GetChangeProof(ctx context.Context, request *pb.SyncGetChangeProofRequest, responseType ResponseType) (V, error)
}
```
## Migration Steps
### Step 1: Update Client Creation
#### Before
```go
client := sync.NewClient(
config,
log,
metrics,
db,
requestHandler,
)
```
#### After (Explicit Types)
```go
client := sync.NewClient[*merkledb.Database, *merkledb.RangeProof, *merkledb.ChangeOrRangeProof](
config,
log,
metrics,
db,
requestHandler,
)
```
#### After (Type Inference - Recommended)
```go
// Go will infer types from the parameters
client := sync.NewClient(
config,
log,
metrics,
db,
requestHandler,
)
```
### Step 2: Update Manager Creation
#### Before
```go
manager := sync.NewManager(
config,
db,
client,
log,
targetFn,
)
```
#### After (Explicit Types)
```go
manager := sync.NewManager[*merkledb.Database](
config,
db,
client,
log,
targetFn,
)
```
#### After (Type Inference - Recommended)
```go
// Type inferred from targetFn return type
manager := sync.NewManager(
config,
db,
client,
log,
targetFn,
)
```
### Step 3: Update Custom Implementations
If you have custom implementations of the sync interfaces:
#### Custom Client Implementation
##### Before
```go
type MyClient struct {
// fields
}
func (c *MyClient) GetRangeProof(ctx context.Context, request *pb.SyncGetRangeProofRequest) (*merkledb.RangeProof, error) {
// implementation
}
```
##### After
```go
type MyClient[T merkledb.MerkleRootGetter, U any, V any] struct {
// fields
}
func (c *MyClient[T, U, V]) GetRangeProof(ctx context.Context, request *pb.SyncGetRangeProofRequest) (U, error) {
// implementation
}
```
#### Custom Database Response Type
##### Creating a Custom Type
```go
type MyDBResponse struct {
root ids.ID
height uint64
metadata map[string]interface{}
}
// Must implement MerkleRootGetter
func (m *MyDBResponse) GetMerkleRoot() ids.ID {
return m.root
}
// Use with sync package
client := sync.NewClient[*MyDBResponse, *merkledb.RangeProof, *merkledb.ChangeOrRangeProof](
config,
log,
metrics,
db,
requestHandler,
)
```
### Step 4: Update Mock Implementations
#### Before
```go
mockClient := &sync.MockClient{}
mockClient.On("GetRangeProof", mock.Anything, mock.Anything).Return(&merkledb.RangeProof{}, nil)
```
#### After
```go
mockClient := &sync.MockClient[*merkledb.Database, *merkledb.RangeProof, *merkledb.ChangeOrRangeProof]{}
mockClient.On("GetRangeProof", mock.Anything, mock.Anything).Return(&merkledb.RangeProof{}, nil)
```
## Common Patterns
### Pattern 1: Using Standard Types
Most users can continue using the standard types without changes:
```go
// The package provides sensible defaults
client := sync.NewClient(config, log, metrics, db, requestHandler)
manager := sync.NewManager(config, db, client, log, targetFn)
```
### Pattern 2: Custom Proof Types
If you need custom proof types:
```go
type MyRangeProof struct {
*merkledb.RangeProof
CustomField string
}
type MyChangeProof struct {
*merkledb.ChangeOrRangeProof
Timestamp time.Time
}
client := sync.NewClient[*merkledb.Database, *MyRangeProof, *MyChangeProof](
config,
log,
metrics,
db,
customRequestHandler,
)
```
### Pattern 3: Working with Multiple Database Types
```go
// For primary database
primaryClient := sync.NewClient[*PrimaryDB, *merkledb.RangeProof, *merkledb.ChangeOrRangeProof](
primaryConfig, log, metrics, primaryDB, primaryHandler,
)
// For secondary database
secondaryClient := sync.NewClient[*SecondaryDB, *merkledb.RangeProof, *merkledb.ChangeOrRangeProof](
secondaryConfig, log, metrics, secondaryDB, secondaryHandler,
)
```
## Testing
### Unit Tests
The generic implementation maintains full test compatibility:
```go
func TestSyncClient(t *testing.T) {
// Existing tests work without changes
client := sync.NewClient(config, log, metrics, db, handler)
// Test as before
proof, err := client.GetRangeProof(ctx, request)
require.NoError(t, err)
require.NotNil(t, proof)
}
```
### Integration Tests
```go
func TestSyncIntegration(t *testing.T) {
// Generic types provide better compile-time safety
var client sync.Client[*merkledb.Database, *merkledb.RangeProof, *merkledb.ChangeOrRangeProof]
client = sync.NewClient(config, log, metrics, db, handler)
// Type safety prevents incorrect usage at compile time
// This would fail at compile time if types don't match
manager := sync.NewManager(config, db, client, log, targetFn)
}
```
## Troubleshooting
### Issue: Type Inference Failures
**Problem**: Go can't infer types automatically
**Solution**: Explicitly specify types
```go
// Instead of
client := sync.NewClient(config, log, metrics, db, handler)
// Use
client := sync.NewClient[*merkledb.Database, *merkledb.RangeProof, *merkledb.ChangeOrRangeProof](
config, log, metrics, db, handler,
)
```
### Issue: Interface Compatibility
**Problem**: Custom implementation doesn't match new interface
**Solution**: Update method signatures to use generic types
```go
// Update from
func (c *MyClient) GetRangeProof(...) (*merkledb.RangeProof, error)
// To
func (c *MyClient[T, U, V]) GetRangeProof(...) (U, error)
```
### Issue: Mock Testing
**Problem**: Mocks need type parameters
**Solution**: Use concrete types for mocks
```go
type MockClient = sync.MockClient[*merkledb.Database, *merkledb.RangeProof, *merkledb.ChangeOrRangeProof]
mock := &MockClient{}
```
## Best Practices
1. **Use Type Inference**: Let Go infer types when possible for cleaner code
2. **Consistent Types**: Use the same type parameters across related components
3. **Document Custom Types**: Clearly document any custom types that implement required interfaces
4. **Test Thoroughly**: Ensure all tests pass with the new generic implementation
5. **Gradual Migration**: Migrate one component at a time if you have a large codebase
## Performance Notes
The generic implementation has **zero performance overhead**:
- Generic instantiation happens at compile time
- No runtime type assertions or boxing
- Identical memory layout and CPU usage
- Better inlining opportunities with concrete types
## Version Compatibility
- **v1.10.x and earlier**: Non-generic implementation
- **v1.11.0+**: Generic implementation with full backward compatibility
- **Migration**: No breaking changes for standard usage
## Getting Help
If you encounter issues during migration:
1. Check that all type parameters match across components
2. Ensure custom types implement required interfaces
3. Review the test files for examples of correct usage
4. Use explicit type parameters if inference fails
## Summary
The migration to generics in x/sync provides:
- ✅ Better type safety at compile time
- ✅ Full backward compatibility
- ✅ Zero performance overhead
- ✅ More flexible custom implementations
- ✅ Cleaner, more maintainable code
Most users can continue using the package without any code changes, while advanced users gain the ability to use custom types with full type safety.
x/sync/README.md
-162
View File
@@ -1,162 +0,0 @@
# `sync` package
## Overview
This package implements a client and server that allows for the syncing of a [MerkleDB](../merkledb/README.md).
The servers have an up-to-date version of the database, and the clients have an out of date version of the database or an empty database.
It's planned that these client and server implementations will eventually be compatible with Firewood.
## Messages
There are four message types sent between the client and server:
1. `SyncGetRangeProofRequest`
2. `RangeProof`
3. `SyncGetChangeProofRequest`
4. `SyncGetChangeProofResponse`
These message types are defined in `node/proto/sync.proto`.
For more information on range proofs and change proofs, see their definitions in `node/merkledb/proof.go`.
### `SyncGetRangeProofRequest`
This message is sent from the client to the server to request a range proof for a given key range and root hash.
That is, the client says, "Give me the key-value pairs that were in this key range when the database had this root."
This request includes a limit on the number of key-value pairs to return, and the size of the response.
### `RangeProof`
This message is sent from the server to the client in response to a `SyncGetRangeProofRequest`.
It contains the key-value pairs that were in the requested key range when the database had the requested root,
as well as a proof that the key-value pairs are correct.
If a server can't serve the entire requested key range in one response, its response will omit keys from the
end of the range rather than the start.
For example, if a client requests a range proof for range [`requested_start`, `requested_end`] but the server
can't fit all the key-value pairs in one response, it'll send a range proof for [`requested_start`, `proof_end`] where `proof_end` < `requested_end`,
as opposed to sending a range proof for [`proof_start`, `requested_end`] where `proof_start` > `requested_start`.
### `SyncGetChangeProofRequest`
This message is sent from the client to the server to request a change proof between the given root hashes.
That is, the client says, "Give me the key-value pairs that changed between the time the database had this root and that root."
This request includes a limit on the number of key-value pairs to return, and the size of the response.
### `SyncGetChangeProofResponse`
This message is sent from the server to the client in response to a `SyncGetChangeProofRequest`.
If the server had sufficient history to generate a change proof, it contains a change proof that contains
the key-value pairs that changed between the requested roots.
If the server did not have sufficient history to generate a change proof, it contains a range proof that
contains the key-value pairs that were in the database when the database had the latter root.
Like range proofs, if a client requests a change proof for range [`requested_start`, `requested_end`] but
the server can't fit all the key-value pairs in one response,
it'll send a change proof for [`requested_start`, `proof_end`] where `proof_end` < `requested_end`,
as opposed to sending a change proof for [`proof_start`, `requested_end`] where `proof_start` > `requested_start`.
## Algorithm
For each proof it receives, the sync client tracks the root hash of the revision associated with the proof's key-value pairs.
For example, it will store information that says something like, "I have all of the key-value pairs that
are in range [`start`, `end`] for the revision with root `root_hash`" for some keys `start` and `end`.
Note that `root_hash` is the root hash of the revision that the client is trying to sync to, not the
root hash of its own (incomplete) database.
Tracking the revision associated with each downloaded key range, as well as using data in its own
(incomplete) database, allows the client to figure out which key ranges are not up-to-date and need to be synced.
The hash of the incomplete database on a client is never sent anywhere because it does not represent a root hash of any revision.
When the client is created, it is given the root hash of the revision to sync to.
When it starts syncing, it requests from a server a range proof for the entire database.
(To indicate that it wants no lower bound on the key range, the client doesn't provide a lower bound in the request.
To indicate that it wants no upper bound, the client doesn't provide an upper bound.
Thus, to request the entire database, the client omits both the lower and upper bounds in its request.)
The server replies with a range proof, which the client verifies.
If it's valid, the key-value pairs in the proof are written to the database.
If it's not, the client drops the proof and requests the proof from another server.
A range proof sent by a server must return a continuous range of the key-value pairs, but may not
return the full range that was requested.
For example, a client might request all the key-value pairs in [`requested_start`, `requested_end`]
but only receive those in range [`requested_start`, `proof_end`] where `proof_end` < `requested_end`.
There might be too many key-value pairs to include in one message, or the server may be too busy to provide any more in its response.
Unless the database is very small, this means that the range proof the client receives in response to
its range proof request for the entire database will not contain all of the key-value pairs in the database.
If a client requests a range proof for range [`requested_start`, `requested_end`] but only receives
a range proof for [`requested_start`, `proof_end`] where `proof_end` < `requested_end`
it recognizes that it must still fetch all of the keys in [`proof_end`, `requested_end`].
It repeatedly requests range proofs for chunks of the remaining key range until it has all of the
key-value pairs in [`requested_start`, `requested_end`].
The client may split the remaining key range into chunks and fetch chunks of key-value pairs in parallel, possibly even from different servers.
Additional commits to the database may occur while the client is syncing.
The sync client can be notified that the root hash of the database it's trying to sync to has changed.
Detecting that the root hash to sync to has changed is done outside this package.
For example, if the database is being used to store blockchain state then the sync client would be
notified when a new block is accepted because that implies a commit to the database.
If this occurs, the key-value pairs the client has learned about via range proofs may no longer be up-to-date.
We use change proofs as an optimization to correct the out of date key-value pairs.
When the sync client is notified that the root hash to sync to has changed, it requests a change proof
from a server for a given key range.
For example, if a client has the key-value pairs in range [`start`, `end`] that were in the database
when it had `root_hash`, then it will request a change proof that provides all of the key-value changes
in range [`start`, `end`] from the database version with root hash `root_hash` to the database version with root hash `new_root_hash`.
The client verifies the change proof, and if it's valid, it applies the changes to its database.
If it's not, the client drops the proof and requests the proof from another server.
A server needs to have history in order to serve a change proof.
Namely, it needs to know all of the database changes between two roots.
If the server does not have sufficient history to generate a change proof, it will send a range proof for
the requested range at revision `new_root_hash` instead.
The client will verify and apply the range proof. (Note that change proofs are just an optimization for bandwidth and speed.
A range proof for a given key range and revision has the same information as a change proof from
`old_root_hash` to `new_root_hash` for the key range, assuming the client has the key-value pairs
for the key range at the revision with `old_root_hash`.)
Change proofs, like range proofs, may not contain all of the key-value pairs in the requested range.
This is OK because as mentioned above, the client tracks the root hash associated with each range of
key-value pairs it has, so it knows which key-value pairs are out of date.
Similar to range proofs, if a client requests the changes in range [`requested_start`, `requested_end`],
but the server replies with all of the changes in [`requested_start`, `proof_end`] for some `proof_end` < `requested_end`,
the client will repeatedly request change proofs until it gets remaining key-value pairs (namely in [`proof_end`, `requested_end`]).
Eventually, by repeatedly requesting, receiving, verifying and applying range and change proofs,
the client will have all of the key-value pairs in the database.
At this point, it's synced.
## Diagram
Assuming you have `Root Hash` `r1` which has many keys, some of which are k25, k50, k75,
approximately 25%, 50%, and 75% of the way into the sorted set of keys, respectively,
this diagram shows an example flow from client to server:
```mermaid
sequenceDiagram
box Client/Server
participant Server
participant Client
end
box New Revision Notifier
participant Notifier
end
Note right of Client: Normal sync flow
Notifier->>Client: CurrentRoot(r1)
Client->>Server: RangeProofRequest(r1, all)
Server->>Client: RangeProofResponse(r1, ..k25)
Client->>Server: RangeProofRequest(r1, k25..)
Server->>Client: RangeProofResponse(r1, k25..k75)
Notifier-)Client: NewRootHash(r2)
Client->>Server: ChangeProofRequest(r1, r2, 0..k75)
Server->>Client: ChangeProofResponse(r1, r2, 0..k50)
Client->>Server: ChangeProofRequest(r1, r2, k50..k75)
Server->>Client: ChangeProofResponse(r1, r2, k50..k75)
Note right of Client: client is @r2 through (..k75)
Client->>Server: RangeProofRequest(r2, k75..)
Server->>Client: RangeProofResponse(r2, k75..k100)
```
## TODOs
- [ ] Handle errors on proof requests. Currently, any errors that occur server side are not sent back to the client.
x/sync/client.go
-392
View File
@@ -1,392 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package sync
import (
"context"
"errors"
"fmt"
"math"
"sync/atomic"
"time"
"google.golang.org/protobuf/proto"
"github.com/luxfi/ids"
"github.com/luxfi/log"
"github.com/luxfi/node/x/merkledb"
"github.com/luxfi/container/maybe"
pb "github.com/luxfi/node/proto/pb/sync"
)
const (
initialRetryWait = 10 * time.Millisecond
maxRetryWait = time.Second
retryWaitFactor = 1.5 // Larger --> timeout grows more quickly
epsilon = 1e-6 // small amount to add to time to avoid division by 0
)
var (
_ Client = (*client)(nil)
errInvalidRangeProof = errors.New("failed to verify range proof")
errInvalidChangeProof = errors.New("failed to verify change proof")
errTooManyKeys = errors.New("response contains more than requested keys")
errTooManyBytes = errors.New("response contains more than requested bytes")
errUnexpectedChangeProofResponse = errors.New("unexpected response type")
)
// ChangeOrRangeProof contains either a ChangeProof or RangeProof.
// Exactly one of ChangeProof or RangeProof should be non-nil.
type ChangeOrRangeProof struct {
ChangeProof *merkledb.ChangeProof
RangeProof *merkledb.RangeProof
}
// Client synchronously fetches data from the network
// to fulfill state sync requests.
// Repeatedly retries failed requests until the context is canceled.
type Client interface {
// GetRangeProof synchronously sends the given request
// and returns the parsed response.
// This method verifies the range proof before returning it.
GetRangeProof(
ctx context.Context,
request *pb.SyncGetRangeProofRequest,
) (*merkledb.RangeProof, error)
// GetChangeProof synchronously sends the given request
// and returns the parsed response.
// This method verifies the change proof / range proof
// before returning it.
// If the server responds with a change proof,
// it's verified using [verificationDB].
GetChangeProof(
ctx context.Context,
request *pb.SyncGetChangeProofRequest,
verificationDB DB,
) (*ChangeOrRangeProof, error)
}
type client struct {
networkClient NetworkClient
stateSyncNodes []ids.NodeID
stateSyncNodeIdx uint32
log log.Logger
metrics SyncMetrics
tokenSize int
hasher merkledb.Hasher
}
type ClientConfig struct {
NetworkClient NetworkClient
StateSyncNodeIDs []ids.NodeID
Log log.Logger
Metrics SyncMetrics
BranchFactor merkledb.BranchFactor
// If not specified, [merkledb.DefaultHasher] will be used.
Hasher merkledb.Hasher
}
func NewClient(config *ClientConfig) (Client, error) {
if err := config.BranchFactor.Valid(); err != nil {
return nil, err
}
hasher := config.Hasher
if hasher == nil {
hasher = merkledb.DefaultHasher
}
return &client{
networkClient: config.NetworkClient,
stateSyncNodes: config.StateSyncNodeIDs,
log: config.Log,
metrics: config.Metrics,
tokenSize: merkledb.BranchFactorToTokenSize[config.BranchFactor],
hasher: hasher,
}, nil
}
// GetChangeProof synchronously retrieves the change proof given by [req].
// Upon failure, retries until the context is expired.
// The returned change proof is verified.
func (c *client) GetChangeProof(
ctx context.Context,
req *pb.SyncGetChangeProofRequest,
db DB,
) (*ChangeOrRangeProof, error) {
parseFn := func(ctx context.Context, responseBytes []byte) (*ChangeOrRangeProof, error) {
if len(responseBytes) > int(req.BytesLimit) {
return nil, fmt.Errorf("%w: (%d) > %d)", errTooManyBytes, len(responseBytes), req.BytesLimit)
}
var changeProofResp pb.SyncGetChangeProofResponse
if err := proto.Unmarshal(responseBytes, &changeProofResp); err != nil {
return nil, err
}
startKey := maybeBytesToMaybe(req.StartKey)
endKey := maybeBytesToMaybe(req.EndKey)
switch changeProofResp := changeProofResp.Response.(type) {
case *pb.SyncGetChangeProofResponse_ChangeProof:
// The server had enough history to send us a change proof
var changeProof merkledb.ChangeProof
if err := changeProof.UnmarshalProto(changeProofResp.ChangeProof); err != nil {
return nil, err
}
// Ensure the response does not contain more than the requested number of leaves
// and the start and end roots match the requested roots.
if len(changeProof.KeyChanges) > int(req.KeyLimit) {
return nil, fmt.Errorf(
"%w: (%d) > %d)",
errTooManyKeys, len(changeProof.KeyChanges), req.KeyLimit,
)
}
endRoot, err := ids.ToID(req.EndRootHash)
if err != nil {
return nil, err
}
if err := db.VerifyChangeProof(
ctx,
&changeProof,
startKey,
endKey,
endRoot,
); err != nil {
return nil, fmt.Errorf("%w due to %w", errInvalidChangeProof, err)
}
return &ChangeOrRangeProof{
ChangeProof: &changeProof,
}, nil
case *pb.SyncGetChangeProofResponse_RangeProof:
var rangeProof merkledb.RangeProof
if err := rangeProof.UnmarshalProto(changeProofResp.RangeProof); err != nil {
return nil, err
}
// The server did not have enough history to send us a change proof
// so they sent a range proof instead.
err := verifyRangeProof(
ctx,
&rangeProof,
int(req.KeyLimit),
startKey,
endKey,
req.EndRootHash,
c.tokenSize,
c.hasher,
)
if err != nil {
return nil, err
}
return &ChangeOrRangeProof{
RangeProof: &rangeProof,
}, nil
default:
return nil, fmt.Errorf(
"%w: %T",
errUnexpectedChangeProofResponse, changeProofResp,
)
}
}
reqBytes, err := proto.Marshal(req)
if err != nil {
return nil, err
}
return getAndParse(ctx, c, reqBytes, parseFn)
}
// Verify [rangeProof] is a valid range proof for keys in [start, end] for
// root [rootBytes]. Returns [errTooManyKeys] if the response contains more
// than [keyLimit] keys.
func verifyRangeProof(
ctx context.Context,
rangeProof *merkledb.RangeProof,
keyLimit int,
start maybe.Maybe[[]byte],
end maybe.Maybe[[]byte],
rootBytes []byte,
tokenSize int,
hasher merkledb.Hasher,
) error {
root, err := ids.ToID(rootBytes)
if err != nil {
return err
}
// Ensure the response does not contain more than the maximum requested number of leaves.
if len(rangeProof.KeyChanges) > keyLimit {
return fmt.Errorf(
"%w: (%d) > %d)",
errTooManyKeys, len(rangeProof.KeyChanges), keyLimit,
)
}
if err := rangeProof.Verify(
ctx,
start,
end,
root,
tokenSize,
hasher,
); err != nil {
return fmt.Errorf("%w due to %w", errInvalidRangeProof, err)
}
return nil
}
// GetRangeProof synchronously retrieves the range proof given by [req].
// Upon failure, retries until the context is expired.
// The returned range proof is verified.
func (c *client) GetRangeProof(
ctx context.Context,
req *pb.SyncGetRangeProofRequest,
) (*merkledb.RangeProof, error) {
parseFn := func(ctx context.Context, responseBytes []byte) (*merkledb.RangeProof, error) {
if len(responseBytes) > int(req.BytesLimit) {
return nil, fmt.Errorf(
"%w: (%d) > %d)",
errTooManyBytes, len(responseBytes), req.BytesLimit,
)
}
var rangeProofProto pb.RangeProof
if err := proto.Unmarshal(responseBytes, &rangeProofProto); err != nil {
return nil, err
}
var rangeProof merkledb.RangeProof
if err := rangeProof.UnmarshalProto(&rangeProofProto); err != nil {
return nil, err
}
if err := verifyRangeProof(
ctx,
&rangeProof,
int(req.KeyLimit),
maybeBytesToMaybe(req.StartKey),
maybeBytesToMaybe(req.EndKey),
req.RootHash,
c.tokenSize,
c.hasher,
); err != nil {
return nil, err
}
return &rangeProof, nil
}
reqBytes, err := proto.Marshal(req)
if err != nil {
return nil, err
}
return getAndParse(ctx, c, reqBytes, parseFn)
}
// getAndParse uses [client] to send [request] to an arbitrary peer.
// Returns the response to the request.
// [parseFn] parses the raw response.
// If the request is unsuccessful or the response can't be parsed,
// retries the request to a different peer until [ctx] expires.
// Returns [errAppSendFailed] if we fail to send an Request/Response.
// This should be treated as a fatal error.
func getAndParse[T any](
ctx context.Context,
client *client,
request []byte,
parseFn func(context.Context, []byte) (*T, error),
) (*T, error) {
var (
lastErr error
response *T
)
// Loop until the context is cancelled or we get a valid response.
for attempt := 1; ; attempt++ {
nodeID, responseBytes, err := client.get(ctx, request)
if err == nil {
if response, err = parseFn(ctx, responseBytes); err == nil {
return response, nil
}
}
if errors.Is(err, errAppSendFailed) {
// Failing to send an Request is a fatal error.
return nil, err
}
client.log.Debug("request failed, retrying",
log.Stringer("nodeID", nodeID),
log.Int("attempt", attempt),
log.Reflect("error", err),
)
// if [err] is being propagated from [ctx], avoid overwriting [lastErr].
if err != ctx.Err() {
lastErr = err
}
retryWait := initialRetryWait * time.Duration(math.Pow(retryWaitFactor, float64(attempt)))
if retryWait > maxRetryWait || retryWait < 0 { // Handle overflows with negative check.
retryWait = maxRetryWait
}
select {
case <-ctx.Done():
if lastErr != nil {
// prefer reporting [lastErr] if it's not nil.
return nil, fmt.Errorf(
"request failed after %d attempts with last error %w and ctx error %w",
attempt, lastErr, ctx.Err(),
)
}
return nil, ctx.Err()
case <-time.After(retryWait):
}
}
}
// get sends [request] to an arbitrary peer and blocks
// until the node receives a response, failure notification
// or [ctx] is canceled.
// Returns the peer's NodeID and response.
// Returns [errAppSendFailed] if we failed to send an Request/Response.
// This should be treated as fatal.
// It's safe to call this method multiple times concurrently.
func (c *client) get(ctx context.Context, request []byte) (ids.NodeID, []byte, error) {
var (
response []byte
nodeID ids.NodeID
err error
)
c.metrics.RequestMade()
if len(c.stateSyncNodes) == 0 {
nodeID, response, err = c.networkClient.RequestAny(ctx, request)
} else {
// Get the next nodeID to query using the [nodeIdx] offset.
// If we're out of nodes, loop back to 0.
// We do this try to query a different node each time if possible.
nodeIdx := atomic.AddUint32(&c.stateSyncNodeIdx, 1)
nodeID = c.stateSyncNodes[nodeIdx%uint32(len(c.stateSyncNodes))]
response, err = c.networkClient.Request(ctx, nodeID, request)
}
if err != nil {
c.metrics.RequestFailed()
return nodeID, response, err
}
c.metrics.RequestSucceeded()
return nodeID, response, nil
}
x/sync/client_test.go
-158
View File
@@ -1,158 +0,0 @@
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
//go:build skip
package sync
import (
"context"
"sync"
"testing"
"time"
"github.com/luxfi/metric"
"github.com/stretchr/testify/require"
"google.golang.org/protobuf/proto"
"github.com/luxfi/consensus/core"
"github.com/luxfi/ids"
"github.com/luxfi/node/trace"
"github.com/luxfi/node/x/merkledb"
"github.com/luxfi/p2p"
"github.com/luxfi/node/trace"
)
var _ p2p.Handler = (*flakyHandler)(nil)
func newDefaultDBConfig() merkledb.Config {
return merkledb.Config{
IntermediateWriteBatchSize: 100,
HistoryLength: defaultRequestKeyLimit,
ValueNodeCacheSize: defaultRequestKeyLimit,
IntermediateWriteBufferSize: defaultRequestKeyLimit,
IntermediateNodeCacheSize: defaultRequestKeyLimit,
Reg: metric.NewNoOp().Registry(),
Tracer: trace.Noop,
BranchFactor: merkledb.BranchFactor16,
}
}
func newFlakyRangeProofHandler(
t *testing.T,
db merkledb.MerkleDB,
modifyResponse func(response *merkledb.RangeProof),
) p2p.Handler {
handler := NewGetRangeProofHandler(db)
c := counter{m: 2}
return &p2p.TestHandler{
RequestF: func(ctx context.Context, nodeID ids.NodeID, deadline time.Time, requestBytes []byte) ([]byte, *common.Error) {
responseBytes, appErr := handler.Request(ctx, nodeID, deadline, requestBytes)
if appErr != nil {
return nil, appErr
}
response := &pb.RangeProof{}
require.NoError(t, proto.Unmarshal(responseBytes, response))
proof := &merkledb.RangeProof{}
require.NoError(t, proof.UnmarshalProto(response))
// Half of requests are modified
if c.Inc() == 0 {
modifyResponse(proof)
}
responseBytes, err := proto.Marshal(proof.ToProto())
if err != nil {
return nil, &common.Error{Code: 123, Message: err.Error()}
}
return responseBytes, nil
},
}
}
func newFlakyChangeProofHandler(
t *testing.T,
db merkledb.MerkleDB,
modifyResponse func(response *merkledb.ChangeProof),
) p2p.Handler {
handler := NewGetChangeProofHandler(db)
c := counter{m: 2}
return &p2p.TestHandler{
RequestF: func(ctx context.Context, nodeID ids.NodeID, deadline time.Time, requestBytes []byte) ([]byte, *common.Error) {
var err error
responseBytes, appErr := handler.Request(ctx, nodeID, deadline, requestBytes)
if appErr != nil {
return nil, appErr
}
response := &pb.SyncGetChangeProofResponse{}
require.NoError(t, proto.Unmarshal(responseBytes, response))
changeProof := response.Response.(*pb.SyncGetChangeProofResponse_ChangeProof)
proof := &merkledb.ChangeProof{}
require.NoError(t, proof.UnmarshalProto(changeProof.ChangeProof))
// Half of requests are modified
if c.Inc() == 0 {
modifyResponse(proof)
}
responseBytes, err = proto.Marshal(&pb.SyncGetChangeProofResponse{
Response: &pb.SyncGetChangeProofResponse_ChangeProof{
ChangeProof: proof.ToProto(),
},
})
if err != nil {
return nil, &common.Error{Code: 123, Message: err.Error()}
}
return responseBytes, nil
},
}
}
type flakyHandler struct {
p2p.Handler
c *counter
}
func (f *flakyHandler) Request(ctx context.Context, nodeID ids.NodeID, deadline time.Time, requestBytes []byte) ([]byte, *common.Error) {
if f.c.Inc() == 0 {
return nil, &common.Error{Code: 123, Message: "flake error"}
}
return f.Handler.Request(ctx, nodeID, deadline, requestBytes)
}
type counter struct {
i int
m int
lock sync.Mutex
}
func (c *counter) Inc() int {
c.lock.Lock()
defer c.lock.Unlock()
tmp := c.i
result := tmp % c.m
c.i++
return result
}
type waitingHandler struct {
p2p.NoOpHandler
handler p2p.Handler
updatedRootChan chan struct{}
}
func (w *waitingHandler) Request(ctx context.Context, nodeID ids.NodeID, deadline time.Time, requestBytes []byte) ([]byte, *common.Error) {
<-w.updatedRootChan
return w.handler.Request(ctx, nodeID, deadline, requestBytes)
}
x/sync/db.go
-16
View File
@@ -1,16 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package sync
import "github.com/luxfi/node/x/merkledb"
type DB interface {
merkledb.Clearer
merkledb.MerkleRootGetter
merkledb.ProofGetter
merkledb.ChangeProofer
merkledb.RangeProofer
}
x/sync/g_db/db_client.go
-194
View File
@@ -1,194 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package gdb
import (
"context"
"errors"
"google.golang.org/protobuf/types/known/emptypb"
"github.com/luxfi/ids"
"github.com/luxfi/node/x/merkledb"
"github.com/luxfi/node/x/sync"
"github.com/luxfi/container/maybe"
pb "github.com/luxfi/node/proto/pb/sync"
)
var _ sync.DB = (*DBClient)(nil)
func NewDBClient(client pb.DBClient) *DBClient {
return &DBClient{
client: client,
}
}
type DBClient struct {
client pb.DBClient
}
func (c *DBClient) GetMerkleRoot(ctx context.Context) (ids.ID, error) {
resp, err := c.client.GetMerkleRoot(ctx, &emptypb.Empty{})
if err != nil {
return ids.Empty, err
}
return ids.ToID(resp.RootHash)
}
func (c *DBClient) GetChangeProof(
ctx context.Context,
startRootID ids.ID,
endRootID ids.ID,
startKey maybe.Maybe[[]byte],
endKey maybe.Maybe[[]byte],
keyLimit int,
) (*merkledb.ChangeProof, error) {
if endRootID == ids.Empty {
return nil, merkledb.ErrEmptyProof
}
resp, err := c.client.GetChangeProof(ctx, &pb.GetChangeProofRequest{
StartRootHash: startRootID[:],
EndRootHash: endRootID[:],
StartKey: &pb.MaybeBytes{
IsNothing: startKey.IsNothing(),
Value: startKey.Value(),
},
EndKey: &pb.MaybeBytes{
IsNothing: endKey.IsNothing(),
Value: endKey.Value(),
},
KeyLimit: uint32(keyLimit),
})
if err != nil {
return nil, err
}
// When the root is not present, the server returns RootNotPresent without
// distinguishing ErrNoEndRoot from ErrInsufficientHistory. Both map to
// ErrInsufficientHistory on the client side.
if resp.GetRootNotPresent() {
return nil, merkledb.ErrInsufficientHistory
}
var proof merkledb.ChangeProof
if err := proof.UnmarshalProto(resp.GetChangeProof()); err != nil {
return nil, err
}
return &proof, nil
}
func (c *DBClient) VerifyChangeProof(
ctx context.Context,
proof *merkledb.ChangeProof,
startKey maybe.Maybe[[]byte],
endKey maybe.Maybe[[]byte],
expectedRootID ids.ID,
) error {
resp, err := c.client.VerifyChangeProof(ctx, &pb.VerifyChangeProofRequest{
Proof: proof.ToProto(),
StartKey: &pb.MaybeBytes{
Value: startKey.Value(),
IsNothing: startKey.IsNothing(),
},
EndKey: &pb.MaybeBytes{
Value: endKey.Value(),
IsNothing: endKey.IsNothing(),
},
ExpectedRootHash: expectedRootID[:],
})
if err != nil {
return err
}
// Error is deserialized from the string returned by the gRPC server.
if len(resp.Error) == 0 {
return nil
}
return errors.New(resp.Error)
}
func (c *DBClient) CommitChangeProof(ctx context.Context, proof *merkledb.ChangeProof) error {
_, err := c.client.CommitChangeProof(ctx, &pb.CommitChangeProofRequest{
Proof: proof.ToProto(),
})
return err
}
func (c *DBClient) GetProof(ctx context.Context, key []byte) (*merkledb.Proof, error) {
resp, err := c.client.GetProof(ctx, &pb.GetProofRequest{
Key: key,
})
if err != nil {
return nil, err
}
var proof merkledb.Proof
if err := proof.UnmarshalProto(resp.Proof); err != nil {
return nil, err
}
return &proof, nil
}
func (c *DBClient) GetRangeProofAtRoot(
ctx context.Context,
rootID ids.ID,
startKey maybe.Maybe[[]byte],
endKey maybe.Maybe[[]byte],
keyLimit int,
) (*merkledb.RangeProof, error) {
if rootID == ids.Empty {
return nil, merkledb.ErrEmptyProof
}
resp, err := c.client.GetRangeProof(ctx, &pb.GetRangeProofRequest{
RootHash: rootID[:],
StartKey: &pb.MaybeBytes{
IsNothing: startKey.IsNothing(),
Value: startKey.Value(),
},
EndKey: &pb.MaybeBytes{
IsNothing: endKey.IsNothing(),
Value: endKey.Value(),
},
KeyLimit: uint32(keyLimit),
})
if err != nil {
return nil, err
}
var proof merkledb.RangeProof
if err := proof.UnmarshalProto(resp.Proof); err != nil {
return nil, err
}
return &proof, nil
}
func (c *DBClient) CommitRangeProof(
ctx context.Context,
startKey maybe.Maybe[[]byte],
endKey maybe.Maybe[[]byte],
proof *merkledb.RangeProof,
) error {
_, err := c.client.CommitRangeProof(ctx, &pb.CommitRangeProofRequest{
StartKey: &pb.MaybeBytes{
IsNothing: startKey.IsNothing(),
Value: startKey.Value(),
},
EndKey: &pb.MaybeBytes{
IsNothing: endKey.IsNothing(),
Value: endKey.Value(),
},
RangeProof: proof.ToProto(),
})
return err
}
func (c *DBClient) Clear() error {
_, err := c.client.Clear(context.Background(), &emptypb.Empty{})
return err
}
x/sync/g_db/db_server.go
-224
View File
@@ -1,224 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package gdb
import (
"context"
"errors"
"google.golang.org/protobuf/types/known/emptypb"
"github.com/luxfi/ids"
"github.com/luxfi/node/x/merkledb"
"github.com/luxfi/node/x/sync"
"github.com/luxfi/container/maybe"
pb "github.com/luxfi/node/proto/pb/sync"
)
var _ pb.DBServer = (*DBServer)(nil)
func NewDBServer(db sync.DB) *DBServer {
return &DBServer{
db: db,
}
}
type DBServer struct {
pb.UnsafeDBServer
db sync.DB
}
func (s *DBServer) GetMerkleRoot(
ctx context.Context,
_ *emptypb.Empty,
) (*pb.GetMerkleRootResponse, error) {
root, err := s.db.GetMerkleRoot(ctx)
if err != nil {
return nil, err
}
return &pb.GetMerkleRootResponse{
RootHash: root[:],
}, nil
}
func (s *DBServer) GetChangeProof(
ctx context.Context,
req *pb.GetChangeProofRequest,
) (*pb.GetChangeProofResponse, error) {
startRootID, err := ids.ToID(req.StartRootHash)
if err != nil {
return nil, err
}
endRootID, err := ids.ToID(req.EndRootHash)
if err != nil {
return nil, err
}
start := maybe.Nothing[[]byte]()
if req.StartKey != nil && !req.StartKey.IsNothing {
start = maybe.Some(req.StartKey.Value)
}
end := maybe.Nothing[[]byte]()
if req.EndKey != nil && !req.EndKey.IsNothing {
end = maybe.Some(req.EndKey.Value)
}
changeProof, err := s.db.GetChangeProof(
ctx,
startRootID,
endRootID,
start,
end,
int(req.KeyLimit),
)
if err != nil {
if !errors.Is(err, merkledb.ErrInsufficientHistory) {
return nil, err
}
return &pb.GetChangeProofResponse{
Response: &pb.GetChangeProofResponse_RootNotPresent{
RootNotPresent: true,
},
}, nil
}
return &pb.GetChangeProofResponse{
Response: &pb.GetChangeProofResponse_ChangeProof{
ChangeProof: changeProof.ToProto(),
},
}, nil
}
func (s *DBServer) VerifyChangeProof(
ctx context.Context,
req *pb.VerifyChangeProofRequest,
) (*pb.VerifyChangeProofResponse, error) {
var proof merkledb.ChangeProof
if err := proof.UnmarshalProto(req.Proof); err != nil {
return nil, err
}
rootID, err := ids.ToID(req.ExpectedRootHash)
if err != nil {
return nil, err
}
startKey := maybe.Nothing[[]byte]()
if req.StartKey != nil && !req.StartKey.IsNothing {
startKey = maybe.Some(req.StartKey.Value)
}
endKey := maybe.Nothing[[]byte]()
if req.EndKey != nil && !req.EndKey.IsNothing {
endKey = maybe.Some(req.EndKey.Value)
}
// Error is serialized as a string across the gRPC boundary.
var errString string
if err := s.db.VerifyChangeProof(ctx, &proof, startKey, endKey, rootID); err != nil {
errString = err.Error()
}
return &pb.VerifyChangeProofResponse{
Error: errString,
}, nil
}
func (s *DBServer) CommitChangeProof(
ctx context.Context,
req *pb.CommitChangeProofRequest,
) (*emptypb.Empty, error) {
var proof merkledb.ChangeProof
if err := proof.UnmarshalProto(req.Proof); err != nil {
return nil, err
}
err := s.db.CommitChangeProof(ctx, &proof)
return &emptypb.Empty{}, err
}
func (s *DBServer) GetProof(
ctx context.Context,
req *pb.GetProofRequest,
) (*pb.GetProofResponse, error) {
proof, err := s.db.GetProof(ctx, req.Key)
if err != nil {
return nil, err
}
return &pb.GetProofResponse{
Proof: proof.ToProto(),
}, nil
}
func (s *DBServer) GetRangeProof(
ctx context.Context,
req *pb.GetRangeProofRequest,
) (*pb.GetRangeProofResponse, error) {
rootID, err := ids.ToID(req.RootHash)
if err != nil {
return nil, err
}
start := maybe.Nothing[[]byte]()
if req.StartKey != nil && !req.StartKey.IsNothing {
start = maybe.Some(req.StartKey.Value)
}
end := maybe.Nothing[[]byte]()
if req.EndKey != nil && !req.EndKey.IsNothing {
end = maybe.Some(req.EndKey.Value)
}
proof, err := s.db.GetRangeProofAtRoot(ctx, rootID, start, end, int(req.KeyLimit))
if err != nil {
return nil, err
}
protoProof := &pb.GetRangeProofResponse{
Proof: &pb.RangeProof{
StartProof: make([]*pb.ProofNode, len(proof.StartProof)),
EndProof: make([]*pb.ProofNode, len(proof.EndProof)),
KeyValues: make([]*pb.KeyValue, len(proof.KeyChanges)),
},
}
for i, node := range proof.StartProof {
protoProof.Proof.StartProof[i] = node.ToProto()
}
for i, node := range proof.EndProof {
protoProof.Proof.EndProof[i] = node.ToProto()
}
for i, kv := range proof.KeyChanges {
protoProof.Proof.KeyValues[i] = &pb.KeyValue{
Key: kv.Key,
Value: kv.Value.Value(),
}
}
return protoProof, nil
}
func (s *DBServer) CommitRangeProof(
ctx context.Context,
req *pb.CommitRangeProofRequest,
) (*emptypb.Empty, error) {
var proof merkledb.RangeProof
if err := proof.UnmarshalProto(req.RangeProof); err != nil {
return nil, err
}
start := maybe.Nothing[[]byte]()
if req.StartKey != nil && !req.StartKey.IsNothing {
start = maybe.Some(req.StartKey.Value)
}
end := maybe.Nothing[[]byte]()
if req.EndKey != nil && !req.EndKey.IsNothing {
end = maybe.Some(req.EndKey.Value)
}
err := s.db.CommitRangeProof(ctx, start, end, &proof)
return &emptypb.Empty{}, err
}
func (s *DBServer) Clear(context.Context, *emptypb.Empty) (*emptypb.Empty, error) {
return &emptypb.Empty{}, s.db.Clear()
}
x/sync/generic_types.go
-195
View File
@@ -1,195 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// Package sync provides generic types for synchronization.
// This file contains the generic implementations that can be used
// with any comparable type, while maintaining backward compatibility
// with the existing byte-based implementation.
package sync
import (
"bytes"
"time"
"github.com/google/btree"
"github.com/luxfi/ids"
"github.com/luxfi/container/heap"
"github.com/luxfi/container/maybe"
)
// GenericWorkItem represents a work item that can work with any comparable type T
type GenericWorkItem[T any] struct {
Start maybe.Maybe[T]
End maybe.Maybe[T]
Priority priority
LocalRootID ids.ID
Attempt int
QueueTime time.Time
}
// RequestFailed increments the attempt counter
func (w *GenericWorkItem[T]) RequestFailed() {
attempt := w.Attempt + 1
// Overflow check
if attempt > w.Attempt {
w.Attempt = attempt
}
}
// NewGenericWorkItem creates a new generic work item
func NewGenericWorkItem[T any](
localRootID ids.ID,
start maybe.Maybe[T],
end maybe.Maybe[T],
priority priority,
queueTime time.Time,
) *GenericWorkItem[T] {
return &GenericWorkItem[T]{
LocalRootID: localRootID,
Start: start,
End: end,
Priority: priority,
QueueTime: queueTime,
}
}
// GenericWorkHeap is a priority queue that can work with any type T
type GenericWorkHeap[T any] struct {
// Max heap of items by priority
innerHeap heap.Set[*GenericWorkItem[T]]
// Items sorted by range start
sortedItems *btree.BTreeG[*GenericWorkItem[T]]
closed bool
compareFn func(a, b T) int
equalFn func(a, b T) bool
}
// NewGenericWorkHeap creates a new generic work heap
func NewGenericWorkHeap[T any](compareFn func(a, b T) int, equalFn func(a, b T) bool) *GenericWorkHeap[T] {
wh := &GenericWorkHeap[T]{
compareFn: compareFn,
equalFn: equalFn,
}
wh.innerHeap = heap.NewSet[*GenericWorkItem[T]](func(a, b *GenericWorkItem[T]) bool {
return a.Priority > b.Priority
})
wh.sortedItems = btree.NewG(
2,
func(a, b *GenericWorkItem[T]) bool {
aNothing := a.Start.IsNothing()
bNothing := b.Start.IsNothing()
if aNothing {
return !bNothing
}
if bNothing {
return false
}
return compareFn(a.Start.Value(), b.Start.Value()) < 0
},
)
return wh
}
// Close marks the heap as closed
func (wh *GenericWorkHeap[T]) Close() {
wh.closed = true
}
// Insert adds a new item into the heap
func (wh *GenericWorkHeap[T]) Insert(item *GenericWorkItem[T]) {
if wh.closed {
return
}
wh.innerHeap.Push(item)
wh.sortedItems.ReplaceOrInsert(item)
}
// GetWork pops and returns a work item from the heap
func (wh *GenericWorkHeap[T]) GetWork() *GenericWorkItem[T] {
if wh.closed || wh.Len() == 0 {
return nil
}
item, _ := wh.innerHeap.Pop()
wh.sortedItems.Delete(item)
return item
}
// MergeInsert inserts the item into the heap, merging with adjacent items if possible
func (wh *GenericWorkHeap[T]) MergeInsert(item *GenericWorkItem[T]) {
if wh.closed {
return
}
var mergedBefore, mergedAfter *GenericWorkItem[T]
searchItem := &GenericWorkItem[T]{
Start: item.Start,
}
wh.sortedItems.DescendLessOrEqual(
searchItem,
func(beforeItem *GenericWorkItem[T]) bool {
if item.LocalRootID == beforeItem.LocalRootID &&
maybe.Equal(item.Start, beforeItem.End, wh.equalFn) {
beforeItem.End = item.End
beforeItem.Priority = max(item.Priority, beforeItem.Priority)
wh.innerHeap.Fix(beforeItem)
mergedBefore = beforeItem
}
return false
})
wh.sortedItems.AscendGreaterOrEqual(
searchItem,
func(afterItem *GenericWorkItem[T]) bool {
if item.LocalRootID == afterItem.LocalRootID &&
maybe.Equal(item.End, afterItem.Start, wh.equalFn) {
afterItem.Start = item.Start
afterItem.Priority = max(item.Priority, afterItem.Priority)
wh.innerHeap.Fix(afterItem)
mergedAfter = afterItem
}
return false
})
if mergedBefore != nil && mergedAfter != nil {
mergedBefore.End = mergedAfter.End
wh.remove(mergedAfter)
mergedBefore.Priority = max(mergedBefore.Priority, mergedAfter.Priority)
wh.innerHeap.Fix(mergedBefore)
}
if mergedBefore == nil && mergedAfter == nil {
wh.Insert(item)
}
}
// remove deletes an item from the heap
func (wh *GenericWorkHeap[T]) remove(item *GenericWorkItem[T]) {
wh.innerHeap.Remove(item)
wh.sortedItems.Delete(item)
}
// Len returns the number of items in the heap
func (wh *GenericWorkHeap[T]) Len() int {
return wh.innerHeap.Len()
}
// ByteWorkHeap is a specialized work heap for byte slices
// This provides backward compatibility with the existing implementation
type ByteWorkHeap struct {
*GenericWorkHeap[[]byte]
}
// NewByteWorkHeap creates a new byte-based work heap
func NewByteWorkHeap() *ByteWorkHeap {
return &ByteWorkHeap{
GenericWorkHeap: NewGenericWorkHeap[[]byte](bytes.Compare, bytes.Equal),
}
}
x/sync/generics_test.go
-159
View File
@@ -1,159 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package sync
import (
"bytes"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/luxfi/ids"
"github.com/luxfi/container/maybe"
)
// TestGenericWorkItem tests the generic work item implementation
func TestGenericWorkItem(t *testing.T) {
require := require.New(t)
// Test with byte slices
rootID := ids.GenerateTestID()
start := maybe.Some([]byte{1, 2, 3})
end := maybe.Some([]byte{4, 5, 6})
item := &GenericWorkItem[[]byte]{
LocalRootID: rootID,
Start: start,
End: end,
Priority: highPriority,
QueueTime: time.Now(),
}
require.NotNil(item)
require.Equal(rootID, item.LocalRootID)
require.True(item.Start.HasValue())
require.True(item.End.HasValue())
require.Equal(highPriority, item.Priority)
// Test RequestFailed
originalAttempt := item.Attempt
item.RequestFailed()
require.Equal(originalAttempt+1, item.Attempt)
}
// TestGenericWorkHeap tests the generic work heap implementation
func TestGenericWorkHeap(t *testing.T) {
require := require.New(t)
// Create a byte-based work heap using the constructor
wh := NewGenericWorkHeap[[]byte](bytes.Compare, bytes.Equal)
require.NotNil(wh)
require.Equal(0, wh.Len())
// Test Insert
rootID := ids.GenerateTestID()
item1 := &GenericWorkItem[[]byte]{
LocalRootID: rootID,
Start: maybe.Some([]byte{1}),
End: maybe.Some([]byte{2}),
Priority: lowPriority,
QueueTime: time.Now(),
}
wh.Insert(item1)
require.Equal(1, wh.Len())
// Test GetWork - should return highest priority item
item2 := &GenericWorkItem[[]byte]{
LocalRootID: rootID,
Start: maybe.Some([]byte{3}),
End: maybe.Some([]byte{4}),
Priority: highPriority,
QueueTime: time.Now(),
}
wh.Insert(item2)
require.Equal(2, wh.Len())
// High priority item should be returned first
work := wh.GetWork()
require.NotNil(work)
require.Equal(highPriority, work.Priority)
require.Equal(1, wh.Len())
// Low priority item should be returned next
work = wh.GetWork()
require.NotNil(work)
require.Equal(lowPriority, work.Priority)
require.Equal(0, wh.Len())
// Empty heap should return nil
work = wh.GetWork()
require.Nil(work)
}
// TestGenericWorkHeapMerge tests merging functionality
func TestGenericWorkHeapMerge(t *testing.T) {
require := require.New(t)
wh := NewGenericWorkHeap[[]byte](bytes.Compare, bytes.Equal)
rootID := ids.GenerateTestID()
// Insert first item [1, 10]
item1 := &GenericWorkItem[[]byte]{
LocalRootID: rootID,
Start: maybe.Some([]byte{1}),
End: maybe.Some([]byte{10}),
Priority: lowPriority,
QueueTime: time.Now(),
}
wh.MergeInsert(item1)
require.Equal(1, wh.Len())
// Insert adjacent item [10, 20] - should merge
item2 := &GenericWorkItem[[]byte]{
LocalRootID: rootID,
Start: maybe.Some([]byte{10}),
End: maybe.Some([]byte{20}),
Priority: medPriority,
QueueTime: time.Now(),
}
wh.MergeInsert(item2)
require.Equal(1, wh.Len()) // Should still be 1 after merge
// Get the merged item
merged := wh.GetWork()
require.NotNil(merged)
require.Equal([]byte{1}, merged.Start.Value())
require.Equal([]byte{20}, merged.End.Value())
require.Equal(medPriority, merged.Priority) // Should have highest priority
}
// TestCompatibilityLayer tests the byte-based heap wrapper
func TestCompatibilityLayer(t *testing.T) {
require := require.New(t)
// Test using NewByteWorkHeap
heap := NewByteWorkHeap()
require.NotNil(heap)
require.Equal(0, heap.Len())
rootID := ids.GenerateTestID()
item := &GenericWorkItem[[]byte]{
LocalRootID: rootID,
Start: maybe.Some([]byte{1}),
End: maybe.Some([]byte{2}),
Priority: highPriority,
QueueTime: time.Now(),
}
heap.Insert(item)
require.Equal(1, heap.Len())
work := heap.GetWork()
require.NotNil(work)
require.Equal(rootID, work.LocalRootID)
require.Equal([]byte{1}, work.Start.Value())
require.Equal([]byte{2}, work.End.Value())
}
x/sync/manager.go
-1139
View File
File diff suppressed because it is too large Load Diff
x/sync/metrics.go
-90
View File
@@ -1,90 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package sync
import (
"sync"
"github.com/luxfi/metric"
)
var (
_ SyncMetrics = (*mockMetrics)(nil)
_ SyncMetrics = (*metricsImpl)(nil)
)
type SyncMetrics interface {
RequestFailed()
RequestMade()
RequestSucceeded()
}
type mockMetrics struct {
lock sync.Mutex
requestsFailed int
requestsMade int
requestsSucceeded int
}
func (m *mockMetrics) RequestFailed() {
m.lock.Lock()
defer m.lock.Unlock()
m.requestsFailed++
}
func (m *mockMetrics) RequestMade() {
m.lock.Lock()
defer m.lock.Unlock()
m.requestsMade++
}
func (m *mockMetrics) RequestSucceeded() {
m.lock.Lock()
defer m.lock.Unlock()
m.requestsSucceeded++
}
type metricsImpl struct {
requestsFailed metric.Counter
requestsMade metric.Counter
requestsSucceeded metric.Counter
}
func NewMetrics(namespace string, reg metric.Registerer) (SyncMetrics, error) {
if reg == nil {
reg = metric.NewNoOpRegistry()
}
m := metricsImpl{
requestsFailed: reg.NewCounter(
metric.AppendNamespace(namespace, "requests_failed"),
"cumulative amount of failed proof requests",
),
requestsMade: reg.NewCounter(
metric.AppendNamespace(namespace, "requests_made"),
"cumulative amount of proof requests made",
),
requestsSucceeded: reg.NewCounter(
metric.AppendNamespace(namespace, "requests_succeeded"),
"cumulative amount of proof requests that were successful",
),
}
return &m, nil
}
func (m *metricsImpl) RequestFailed() {
m.requestsFailed.Inc()
}
func (m *metricsImpl) RequestMade() {
m.requestsMade.Inc()
}
func (m *metricsImpl) RequestSucceeded() {
m.requestsSucceeded.Inc()
}
x/sync/mock_client.go
-74
View File
@@ -1,74 +0,0 @@
//go:build grpc
// Code generated by MockGen. DO NOT EDIT.
// Source: github.com/luxfi/node/x/sync (interfaces: Client)
//
// Generated by this command:
//
// mockgen -package=sync -destination=x/sync/mock_client.go github.com/luxfi/node/x/sync Client
//
// Package sync is a generated GoMock package.
package sync
import (
context "context"
reflect "reflect"
gomock "github.com/luxfi/mock/gomock"
sync "github.com/luxfi/node/proto/pb/sync"
merkledb "github.com/luxfi/node/x/merkledb"
)
// MockClient is a mock of Client interface.
type MockClient struct {
ctrl *gomock.Controller
recorder *MockClientMockRecorder
}
// MockClientMockRecorder is the mock recorder for MockClient.
type MockClientMockRecorder struct {
mock *MockClient
}
// NewMockClient creates a new mock instance.
func NewMockClient(ctrl *gomock.Controller) *MockClient {
mock := &MockClient{ctrl: ctrl}
mock.recorder = &MockClientMockRecorder{mock}
return mock
}
// EXPECT returns an object that allows the caller to indicate expected use.
func (m *MockClient) EXPECT() *MockClientMockRecorder {
return m.recorder
}
// GetChangeProof mocks base method.
func (m *MockClient) GetChangeProof(arg0 context.Context, arg1 *sync.SyncGetChangeProofRequest, arg2 DB) (*ChangeOrRangeProof, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "GetChangeProof", arg0, arg1, arg2)
ret0, _ := ret[0].(*ChangeOrRangeProof)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// GetChangeProof indicates an expected call of GetChangeProof.
func (mr *MockClientMockRecorder) GetChangeProof(arg0, arg1, arg2 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetChangeProof", reflect.TypeOf((*MockClient)(nil).GetChangeProof), arg0, arg1, arg2)
}
// GetRangeProof mocks base method.
func (m *MockClient) GetRangeProof(arg0 context.Context, arg1 *sync.SyncGetRangeProofRequest) (*merkledb.RangeProof, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "GetRangeProof", arg0, arg1)
ret0, _ := ret[0].(*merkledb.RangeProof)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// GetRangeProof indicates an expected call of GetRangeProof.
func (mr *MockClientMockRecorder) GetRangeProof(arg0, arg1 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetRangeProof", reflect.TypeOf((*MockClient)(nil).GetRangeProof), arg0, arg1)
}
x/sync/mock_network_client.go
-132
View File
@@ -1,132 +0,0 @@
//go:build grpc
// Code generated by MockGen. DO NOT EDIT.
// Source: github.com/luxfi/node/x/sync (interfaces: NetworkClient)
//
// Generated by this command:
//
// mockgen -package=sync -destination=x/sync/mock_network_client.go github.com/luxfi/node/x/sync NetworkClient
//
// Package sync is a generated GoMock package.
package sync
import (
context "context"
reflect "reflect"
gomock "go.uber.org/mock/gomock"
ids "github.com/luxfi/ids"
version "github.com/luxfi/node/version"
)
// MockNetworkClient is a mock of NetworkClient interface.
type MockNetworkClient struct {
ctrl *gomock.Controller
recorder *MockNetworkClientMockRecorder
}
// MockNetworkClientMockRecorder is the mock recorder for MockNetworkClient.
type MockNetworkClientMockRecorder struct {
mock *MockNetworkClient
}
// NewMockNetworkClient creates a new mock instance.
func NewMockNetworkClient(ctrl *gomock.Controller) *MockNetworkClient {
mock := &MockNetworkClient{ctrl: ctrl}
mock.recorder = &MockNetworkClientMockRecorder{mock}
return mock
}
// EXPECT returns an object that allows the caller to indicate expected use.
func (m *MockNetworkClient) EXPECT() *MockNetworkClientMockRecorder {
return m.recorder
}
// RequestFailed mocks base method.
func (m *MockNetworkClient) RequestFailed(arg0 context.Context, arg1 ids.NodeID, arg2 uint32) error {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "RequestFailed", arg0, arg1, arg2)
ret0, _ := ret[0].(error)
return ret0
}
// RequestFailed indicates an expected call of RequestFailed.
func (mr *MockNetworkClientMockRecorder) RequestFailed(arg0, arg1, arg2 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RequestFailed", reflect.TypeOf((*MockNetworkClient)(nil).RequestFailed), arg0, arg1, arg2)
}
// Response mocks base method.
func (m *MockNetworkClient) Response(arg0 context.Context, arg1 ids.NodeID, arg2 uint32, arg3 []byte) error {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "Response", arg0, arg1, arg2, arg3)
ret0, _ := ret[0].(error)
return ret0
}
// Response indicates an expected call of Response.
func (mr *MockNetworkClientMockRecorder) Response(arg0, arg1, arg2, arg3 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Response", reflect.TypeOf((*MockNetworkClient)(nil).Response), arg0, arg1, arg2, arg3)
}
// Connected mocks base method.
func (m *MockNetworkClient) Connected(arg0 context.Context, arg1 ids.NodeID, arg2 *version.Application) error {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "Connected", arg0, arg1, arg2)
ret0, _ := ret[0].(error)
return ret0
}
// Connected indicates an expected call of Connected.
func (mr *MockNetworkClientMockRecorder) Connected(arg0, arg1, arg2 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Connected", reflect.TypeOf((*MockNetworkClient)(nil).Connected), arg0, arg1, arg2)
}
// Disconnected mocks base method.
func (m *MockNetworkClient) Disconnected(arg0 context.Context, arg1 ids.NodeID) error {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "Disconnected", arg0, arg1)
ret0, _ := ret[0].(error)
return ret0
}
// Disconnected indicates an expected call of Disconnected.
func (mr *MockNetworkClientMockRecorder) Disconnected(arg0, arg1 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Disconnected", reflect.TypeOf((*MockNetworkClient)(nil).Disconnected), arg0, arg1)
}
// Request mocks base method.
func (m *MockNetworkClient) Request(arg0 context.Context, arg1 ids.NodeID, arg2 []byte) ([]byte, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "Request", arg0, arg1, arg2)
ret0, _ := ret[0].([]byte)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// Request indicates an expected call of Request.
func (mr *MockNetworkClientMockRecorder) Request(arg0, arg1, arg2 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Request", reflect.TypeOf((*MockNetworkClient)(nil).Request), arg0, arg1, arg2)
}
// RequestAny mocks base method.
func (m *MockNetworkClient) RequestAny(arg0 context.Context, arg1 []byte) (ids.NodeID, []byte, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "RequestAny", arg0, arg1)
ret0, _ := ret[0].(ids.NodeID)
ret1, _ := ret[1].([]byte)
ret2, _ := ret[2].(error)
return ret0, ret1, ret2
}
// RequestAny indicates an expected call of RequestAny.
func (mr *MockNetworkClientMockRecorder) RequestAny(arg0, arg1 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RequestAny", reflect.TypeOf((*MockNetworkClient)(nil).RequestAny), arg0, arg1)
}
x/sync/network_client.go
-369
View File
@@ -1,369 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package sync
import (
"context"
"errors"
"fmt"
"sync"
"time"
"github.com/luxfi/metric"
"golang.org/x/sync/semaphore"
consensusversion "github.com/luxfi/version"
"github.com/luxfi/ids"
"github.com/luxfi/log"
"github.com/luxfi/math/set"
"github.com/luxfi/p2p"
"github.com/luxfi/warp"
)
// Minimum amount of time to handle a request
const minRequestHandlingDuration = 100 * time.Millisecond
var (
_ NetworkClient = (*networkClient)(nil)
errAcquiringSemaphore = errors.New("error acquiring semaphore")
errRequestFailed = errors.New("request failed")
errAppSendFailed = errors.New("failed to send app message")
)
// NetworkClient defines ability to send request / response through the Network
type NetworkClient interface {
// RequestAny synchronously sends request to an arbitrary peer with a
// node version greater than or equal to minVersion.
// Returns response bytes, the ID of the chosen peer, and ErrRequestFailed if
// the request should be retried.
RequestAny(
ctx context.Context,
request []byte,
) (ids.NodeID, []byte, error)
// Sends [request] to [nodeID] and returns the response.
// Blocks until the number of outstanding requests is
// below the limit before sending the request.
Request(
ctx context.Context,
nodeID ids.NodeID,
request []byte,
) ([]byte, error)
// The following declarations allow this interface to be embedded in the VM
// to handle incoming responses from peers.
// Always returns nil because the engine considers errors
// returned from this function as fatal.
Response(context.Context, ids.NodeID, uint32, []byte) error
// Always returns nil because the engine considers errors
// returned from this function as fatal.
RequestFailed(context.Context, ids.NodeID, uint32) error
// Adds the given [nodeID] to the peer
// list so that it can receive messages.
// If [nodeID] is this node's ID, this is a no-op.
Connected(context.Context, ids.NodeID, *consensusversion.Application) error
// Removes given [nodeID] from the peer list.
Disconnected(context.Context, ids.NodeID) error
}
type networkClient struct {
lock sync.Mutex
log log.Logger
// requestID counter used to track outbound requests
requestID uint32
// requestID => handler for the response/failure
outstandingRequestHandlers map[uint32]ResponseHandler
// controls maximum number of active outbound requests
activeRequests *semaphore.Weighted
// tracking of peers & bandwidth usage
peers *p2p.PeerTracker
// For sending messages to peers
sender warp.Sender
}
func NewNetworkClient(
sender warp.Sender,
myNodeID ids.NodeID,
maxActiveRequests int64,
log log.Logger,
metricsNamespace string,
registerer metric.Registerer,
minVersion *consensusversion.Application,
) (NetworkClient, error) {
peerTracker, err := p2p.NewPeerTracker(
log,
metricsNamespace,
registerer,
set.Of(myNodeID),
minVersion,
)
if err != nil {
return nil, fmt.Errorf("failed to create peer tracker: %w", err)
}
return &networkClient{
sender: sender,
outstandingRequestHandlers: make(map[uint32]ResponseHandler),
activeRequests: semaphore.NewWeighted(maxActiveRequests),
peers: peerTracker,
log: log,
}, nil
}
func (c *networkClient) Response(
_ context.Context,
nodeID ids.NodeID,
requestID uint32,
response []byte,
) error {
c.lock.Lock()
defer c.lock.Unlock()
c.log.Info(
"received Response from peer",
log.UserString("nodeID", nodeID.String()),
log.Uint32("requestID", requestID),
log.Int("responseLen", len(response)),
)
handler, exists := c.getRequestHandler(requestID)
if !exists {
// Should never happen since the engine
// should be managing outstanding requests
c.log.Warn(
"received response to unknown request",
log.UserString("nodeID", nodeID.String()),
log.Uint32("requestID", requestID),
log.Int("responseLen", len(response)),
)
return nil
}
handler.OnResponse(response)
return nil
}
func (c *networkClient) RequestFailed(
_ context.Context,
nodeID ids.NodeID,
requestID uint32,
) error {
c.lock.Lock()
defer c.lock.Unlock()
c.log.Info(
"received RequestFailed from peer",
log.UserString("nodeID", nodeID.String()),
log.Uint32("requestID", requestID),
)
handler, exists := c.getRequestHandler(requestID)
if !exists {
// Should never happen since the engine
// should be managing outstanding requests
c.log.Warn(
"received request failed to unknown request",
log.Stringer("nodeID", nodeID),
log.Uint32("requestID", requestID),
)
return nil
}
handler.OnFailure()
return nil
}
// Returns the handler for [requestID] and marks the request as fulfilled.
// Returns false if there's no outstanding request with [requestID].
// Assumes [c.lock] is held.
func (c *networkClient) getRequestHandler(requestID uint32) (ResponseHandler, bool) {
handler, exists := c.outstandingRequestHandlers[requestID]
if !exists {
return nil, false
}
// mark message as processed, release activeRequests slot
delete(c.outstandingRequestHandlers, requestID)
return handler, true
}
// If [errAppSendFailed] is returned this should be considered fatal.
func (c *networkClient) RequestAny(
ctx context.Context,
request []byte,
) (ids.NodeID, []byte, error) {
// Take a slot from total [activeRequests] and block until a slot becomes available.
if err := c.activeRequests.Acquire(ctx, 1); err != nil {
return ids.EmptyNodeID, nil, errAcquiringSemaphore
}
defer c.activeRequests.Release(1)
nodeID, responseChan, err := c.sendRequestAny(ctx, request)
if err != nil {
return ids.EmptyNodeID, nil, err
}
response, err := c.awaitResponse(ctx, nodeID, responseChan)
return nodeID, response, err
}
func (c *networkClient) sendRequestAny(
ctx context.Context,
request []byte,
) (ids.NodeID, chan []byte, error) {
c.lock.Lock()
defer c.lock.Unlock()
nodeID, ok := c.peers.SelectPeer()
if !ok {
numPeers := c.peers.Size()
return ids.EmptyNodeID, nil, fmt.Errorf("no peers found from %d peers", numPeers)
}
responseChan, err := c.sendRequestLocked(ctx, nodeID, request)
return nodeID, responseChan, err
}
// If [errAppSendFailed] is returned this should be considered fatal.
func (c *networkClient) Request(
ctx context.Context,
nodeID ids.NodeID,
request []byte,
) ([]byte, error) {
// Take a slot from total [activeRequests]
// and block until a slot becomes available.
if err := c.activeRequests.Acquire(ctx, 1); err != nil {
return nil, errAcquiringSemaphore
}
defer c.activeRequests.Release(1)
responseChan, err := c.sendRequest(ctx, nodeID, request)
if err != nil {
return nil, err
}
return c.awaitResponse(ctx, nodeID, responseChan)
}
func (c *networkClient) sendRequest(
ctx context.Context,
nodeID ids.NodeID,
request []byte,
) (chan []byte, error) {
c.lock.Lock()
defer c.lock.Unlock()
return c.sendRequestLocked(ctx, nodeID, request)
}
// Sends [request] to [nodeID] and returns a channel that will populate the
// response.
//
// If [errAppSendFailed] is returned this should be considered fatal.
//
// Assumes [nodeID] is never [c.myNodeID] since we guarantee [c.myNodeID] will
// not be added to [c.peers].
//
// Assumes [c.lock] is held.
func (c *networkClient) sendRequestLocked(
ctx context.Context,
nodeID ids.NodeID,
request []byte,
) (chan []byte, error) {
requestID := c.requestID
c.requestID++
c.log.Debug("sending request to peer",
log.UserString("nodeID", nodeID.String()),
log.Uint32("requestID", requestID),
log.Int("requestLen", len(request)),
)
c.peers.RegisterRequest(nodeID)
// Send request to the peer.
nodeIDs := set.Of(nodeID)
// Cancellation is removed from this context to avoid erroring unexpectedly.
// SendRequest should be non-blocking and any error other than context
// cancellation is unexpected.
//
// This guarantees that the network should never receive an unexpected
// Response.
ctxWithoutCancel := context.WithoutCancel(ctx)
if err := c.sender.SendRequest(ctxWithoutCancel, nodeIDs, requestID, request); err != nil {
c.lock.Unlock()
c.log.Error("failed to send request",
log.Stringer("nodeID", nodeID),
log.Uint32("requestID", requestID),
log.Int("requestLen", len(request)),
log.Reflect("error", err),
)
return nil, fmt.Errorf("%w: %w", errAppSendFailed, err)
}
handler := newResponseHandler()
c.outstandingRequestHandlers[requestID] = handler
return handler.responseChan, nil
}
// awaitResponse from [nodeID] and returns the response.
//
// Returns an error if the request failed or [ctx] is canceled.
//
// Blocks until a response is received or the [ctx] is canceled fails.
//
// Assumes [nodeID] is never [c.myNodeID] since we guarantee [c.myNodeID] will
// not be added to [c.peers].
//
// Assumes [c.lock] is not held.
func (c *networkClient) awaitResponse(
ctx context.Context,
nodeID ids.NodeID,
responseChan chan []byte,
) ([]byte, error) {
var (
response []byte
responded bool
startTime = time.Now()
)
select {
case <-ctx.Done():
c.peers.RegisterFailure(nodeID)
return nil, ctx.Err()
case response, responded = <-responseChan:
}
if !responded {
c.peers.RegisterFailure(nodeID)
return nil, errRequestFailed
}
elapsedSeconds := time.Since(startTime).Seconds()
bandwidth := float64(len(response)) / (elapsedSeconds + epsilon)
c.peers.RegisterResponse(nodeID, bandwidth)
c.log.Debug("received response from peer",
log.UserString("nodeID", nodeID.String()),
log.Int("responseLen", len(response)),
)
return response, nil
}
func (c *networkClient) Connected(
_ context.Context,
nodeID ids.NodeID,
nodeVersion *consensusversion.Application,
) error {
c.log.Debug("adding new peer", log.UserString("nodeID", nodeID.String()))
c.peers.Connected(nodeID, nodeVersion)
return nil
}
func (c *networkClient) Disconnected(_ context.Context, nodeID ids.NodeID) error {
c.log.Debug("disconnecting peer", log.UserString("nodeID", nodeID.String()))
c.peers.Disconnected(nodeID)
return nil
}
x/sync/network_server.go
-339
View File
@@ -1,339 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package sync
import (
"bytes"
"context"
"errors"
"fmt"
"time"
"google.golang.org/protobuf/proto"
"github.com/luxfi/constants"
"github.com/luxfi/ids"
"github.com/luxfi/node/x/merkledb"
"github.com/luxfi/p2p"
hash "github.com/luxfi/crypto/hash"
"github.com/luxfi/container/maybe"
"github.com/luxfi/warp"
pb "github.com/luxfi/node/proto/pb/sync"
)
const (
// Maximum number of key-value pairs to return in a proof.
// This overrides any other Limit specified in a RangeProofRequest
// or ChangeProofRequest if the given Limit is greater.
maxKeyValuesLimit = 2048
// Estimated max overhead, in bytes, of putting a proof into a message.
// We use this to ensure that the proof we generate is not too large to fit in a message.
estimatedMessageOverhead = 4 * constants.KiB
maxByteSizeLimit = constants.DefaultMaxMessageSize - estimatedMessageOverhead
)
var (
ErrMinProofSizeIsTooLarge = errors.New("cannot generate any proof within the requested limit")
errInvalidBytesLimit = errors.New("bytes limit must be greater than 0")
errInvalidKeyLimit = errors.New("key limit must be greater than 0")
errInvalidStartRootHash = fmt.Errorf("start root hash must have length %d", hash.HashLen)
errInvalidEndRootHash = fmt.Errorf("end root hash must have length %d", hash.HashLen)
errInvalidStartKey = errors.New("start key is Nothing but has value")
errInvalidEndKey = errors.New("end key is Nothing but has value")
errInvalidBounds = errors.New("start key is greater than end key")
errInvalidRootHash = fmt.Errorf("root hash must have length %d", hash.HashLen)
_ p2p.Handler = (*GetChangeProofHandler)(nil)
_ p2p.Handler = (*GetRangeProofHandler)(nil)
)
func maybeBytesToMaybe(mb *pb.MaybeBytes) maybe.Maybe[[]byte] {
if mb != nil && !mb.IsNothing {
return maybe.Some(mb.Value)
}
return maybe.Nothing[[]byte]()
}
func NewGetChangeProofHandler(db DB) *GetChangeProofHandler {
return &GetChangeProofHandler{
db: db,
}
}
type GetChangeProofHandler struct {
db DB
}
func (*GetChangeProofHandler) Gossip(context.Context, ids.NodeID, []byte) {}
func (g *GetChangeProofHandler) Request(ctx context.Context, _ ids.NodeID, _ time.Time, requestBytes []byte) ([]byte, *warp.Error) {
req := &pb.SyncGetChangeProofRequest{}
if err := proto.Unmarshal(requestBytes, req); err != nil {
return nil, &warp.Error{
Code: p2p.ErrUnexpected.Code,
Message: fmt.Sprintf("failed to unmarshal request: %s", err),
}
}
if err := validateChangeProofRequest(req); err != nil {
return nil, &warp.Error{
Code: p2p.ErrUnexpected.Code,
Message: fmt.Sprintf("invalid request: %s", err),
}
}
// override limits if they exceed caps
var (
keyLimit = min(req.KeyLimit, maxKeyValuesLimit)
bytesLimit = min(int(req.BytesLimit), maxByteSizeLimit)
start = maybeBytesToMaybe(req.StartKey)
end = maybeBytesToMaybe(req.EndKey)
)
startRoot, err := ids.ToID(req.StartRootHash)
if err != nil {
return nil, &warp.Error{
Code: p2p.ErrUnexpected.Code,
Message: fmt.Sprintf("failed to parse start root hash: %s", err),
}
}
endRoot, err := ids.ToID(req.EndRootHash)
if err != nil {
return nil, &warp.Error{
Code: p2p.ErrUnexpected.Code,
Message: fmt.Sprintf("failed to parse end root hash: %s", err),
}
}
for keyLimit > 0 {
changeProof, err := g.db.GetChangeProof(ctx, startRoot, endRoot, start, end, int(keyLimit))
if err != nil {
if !errors.Is(err, merkledb.ErrInsufficientHistory) {
// We should only fail to get a change proof if we have insufficient history.
// Other errors are unexpected.
return nil, &warp.Error{
Code: p2p.ErrUnexpected.Code,
Message: fmt.Sprintf("failed to get change proof: %s", err),
}
}
if errors.Is(err, merkledb.ErrNoEndRoot) {
// [s.db] doesn't have [endRoot] in its history.
// We can't generate a change/range proof. Drop this request.
return nil, &warp.Error{
Code: p2p.ErrUnexpected.Code,
Message: fmt.Sprintf("failed to get change proof: %s", err),
}
}
// [s.db] doesn't have sufficient history to generate change proof.
// Generate a range proof for the end root ID instead.
proofBytes, err := getRangeProof(
ctx,
g.db,
&pb.SyncGetRangeProofRequest{
RootHash: req.EndRootHash,
StartKey: req.StartKey,
EndKey: req.EndKey,
KeyLimit: req.KeyLimit,
BytesLimit: req.BytesLimit,
},
func(rangeProof *merkledb.RangeProof) ([]byte, error) {
return proto.Marshal(&pb.SyncGetChangeProofResponse{
Response: &pb.SyncGetChangeProofResponse_RangeProof{
RangeProof: rangeProof.ToProto(),
},
})
},
)
if err != nil {
return nil, &warp.Error{
Code: p2p.ErrUnexpected.Code,
Message: fmt.Sprintf("failed to get range proof: %s", err),
}
}
return proofBytes, nil
}
// We generated a change proof. See if it's small enough.
proofBytes, err := proto.Marshal(&pb.SyncGetChangeProofResponse{
Response: &pb.SyncGetChangeProofResponse_ChangeProof{
ChangeProof: changeProof.ToProto(),
},
})
if err != nil {
return nil, &warp.Error{
Code: p2p.ErrUnexpected.Code,
Message: fmt.Sprintf("failed to marshal change proof: %s", err),
}
}
if len(proofBytes) < bytesLimit {
return proofBytes, nil
}
// The proof was too large. Try to shrink it.
keyLimit = uint32(len(changeProof.KeyChanges)) / 2
}
return nil, &warp.Error{
Code: p2p.ErrUnexpected.Code,
Message: fmt.Sprintf("failed to generate proof: %s", ErrMinProofSizeIsTooLarge),
}
}
func NewGetRangeProofHandler(db DB) *GetRangeProofHandler {
return &GetRangeProofHandler{
db: db,
}
}
type GetRangeProofHandler struct {
db DB
}
func (*GetRangeProofHandler) Gossip(context.Context, ids.NodeID, []byte) {}
func (g *GetRangeProofHandler) Request(ctx context.Context, _ ids.NodeID, _ time.Time, requestBytes []byte) ([]byte, *warp.Error) {
req := &pb.SyncGetRangeProofRequest{}
if err := proto.Unmarshal(requestBytes, req); err != nil {
return nil, &warp.Error{
Code: p2p.ErrUnexpected.Code,
Message: fmt.Sprintf("failed to unmarshal request: %s", err),
}
}
if err := validateRangeProofRequest(req); err != nil {
return nil, &warp.Error{
Code: p2p.ErrUnexpected.Code,
Message: fmt.Sprintf("invalid range proof request: %s", err),
}
}
// override limits if they exceed caps
req.KeyLimit = min(req.KeyLimit, maxKeyValuesLimit)
req.BytesLimit = min(req.BytesLimit, maxByteSizeLimit)
proofBytes, err := getRangeProof(
ctx,
g.db,
req,
func(rangeProof *merkledb.RangeProof) ([]byte, error) {
return proto.Marshal(rangeProof.ToProto())
},
)
if err != nil {
return nil, &warp.Error{
Code: p2p.ErrUnexpected.Code,
Message: fmt.Sprintf("failed to get range proof: %s", err),
}
}
return proofBytes, nil
}
// Get the range proof specified by [req].
// If the generated proof is too large, the key limit is reduced
// and the proof is regenerated. This process is repeated until
// the proof is smaller than [req.BytesLimit].
// When a sufficiently small proof is generated, returns it.
// If no sufficiently small proof can be generated, returns [ErrMinProofSizeIsTooLarge].
// getRangeProof generates a range proof, iteratively reducing the key limit
// until the serialized proof fits within the byte limit. Returns
// ErrMinProofSizeIsTooLarge if no sufficiently small proof can be generated.
func getRangeProof(
ctx context.Context,
db DB,
req *pb.SyncGetRangeProofRequest,
marshalFunc func(*merkledb.RangeProof) ([]byte, error),
) ([]byte, error) {
root, err := ids.ToID(req.RootHash)
if err != nil {
return nil, err
}
keyLimit := int(req.KeyLimit)
for keyLimit > 0 {
rangeProof, err := db.GetRangeProofAtRoot(
ctx,
root,
maybeBytesToMaybe(req.StartKey),
maybeBytesToMaybe(req.EndKey),
keyLimit,
)
if err != nil {
if errors.Is(err, merkledb.ErrInsufficientHistory) {
return nil, nil // drop request
}
return nil, err
}
proofBytes, err := marshalFunc(rangeProof)
if err != nil {
return nil, err
}
if len(proofBytes) < int(req.BytesLimit) {
return proofBytes, nil
}
// The proof was too large. Try to shrink it.
keyLimit = len(rangeProof.KeyChanges) / 2
}
return nil, ErrMinProofSizeIsTooLarge
}
// Returns nil iff [req] is well-formed.
func validateChangeProofRequest(req *pb.SyncGetChangeProofRequest) error {
switch {
case req.BytesLimit == 0:
return errInvalidBytesLimit
case req.KeyLimit == 0:
return errInvalidKeyLimit
case len(req.StartRootHash) != hash.HashLen:
return errInvalidStartRootHash
case len(req.EndRootHash) != hash.HashLen:
return errInvalidEndRootHash
case bytes.Equal(req.EndRootHash, ids.Empty[:]):
return merkledb.ErrEmptyProof
case req.StartKey != nil && req.StartKey.IsNothing && len(req.StartKey.Value) > 0:
return errInvalidStartKey
case req.EndKey != nil && req.EndKey.IsNothing && len(req.EndKey.Value) > 0:
return errInvalidEndKey
case req.StartKey != nil && req.EndKey != nil && !req.StartKey.IsNothing &&
!req.EndKey.IsNothing && bytes.Compare(req.StartKey.Value, req.EndKey.Value) > 0:
return errInvalidBounds
default:
return nil
}
}
// Returns nil iff [req] is well-formed.
func validateRangeProofRequest(req *pb.SyncGetRangeProofRequest) error {
switch {
case req.BytesLimit == 0:
return errInvalidBytesLimit
case req.KeyLimit == 0:
return errInvalidKeyLimit
case len(req.RootHash) != ids.IDLen:
return errInvalidRootHash
case bytes.Equal(req.RootHash, ids.Empty[:]):
return merkledb.ErrEmptyProof
case req.StartKey != nil && req.StartKey.IsNothing && len(req.StartKey.Value) > 0:
return errInvalidStartKey
case req.EndKey != nil && req.EndKey.IsNothing && len(req.EndKey.Value) > 0:
return errInvalidEndKey
case req.StartKey != nil && req.EndKey != nil && !req.StartKey.IsNothing &&
!req.EndKey.IsNothing && bytes.Compare(req.StartKey.Value, req.EndKey.Value) > 0:
return errInvalidBounds
default:
return nil
}
}
x/sync/network_server_test.go
-380
View File
@@ -1,380 +0,0 @@
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
//go:build skip
package sync
import (
"context"
"math/rand"
"testing"
"time"
"github.com/luxfi/consensus"
"github.com/luxfi/mock/gomock"
"github.com/stretchr/testify/require"
"google.golang.org/protobuf/proto"
"github.com/luxfi/consensus/core"
"github.com/luxfi/database"
"github.com/luxfi/database/memdb"
"github.com/luxfi/ids"
"github.com/luxfi/node/x/merkledb"
"github.com/luxfi/p2p"
pb "github.com/luxfi/node/proto/pb/sync"
)
func Test_Server_GetRangeProof(t *testing.T) {
now := time.Now().UnixNano()
t.Logf("seed: %d", now)
r := rand.New(rand.NewSource(now)) // #nosec G404
smallTrieDB, err := generateTrieWithMinKeyLen(t, r, defaultRequestKeyLimit, 1)
require.NoError(t, err)
smallTrieRoot, err := smallTrieDB.GetMerkleRoot(context.Background())
require.NoError(t, err)
tests := []struct {
name string
request *pb.SyncGetRangeProofRequest
expectedErr *common.Error
expectedResponseLen int
expectedMaxResponseBytes int
nodeID ids.NodeID
proofNil bool
}{
{
name: "proof too large",
request: &pb.SyncGetRangeProofRequest{
RootHash: smallTrieRoot[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: 1000,
},
proofNil: true,
expectedErr: p2p.ErrUnexpected,
},
{
name: "byteslimit is 0",
request: &pb.SyncGetRangeProofRequest{
RootHash: smallTrieRoot[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: 0,
},
proofNil: true,
expectedErr: p2p.ErrUnexpected,
},
{
name: "keylimit is 0",
request: &pb.SyncGetRangeProofRequest{
RootHash: smallTrieRoot[:],
KeyLimit: 0,
BytesLimit: defaultRequestByteSizeLimit,
},
proofNil: true,
expectedErr: p2p.ErrUnexpected,
},
{
name: "keys out of order",
request: &pb.SyncGetRangeProofRequest{
RootHash: smallTrieRoot[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: defaultRequestByteSizeLimit,
StartKey: &pb.MaybeBytes{Value: []byte{1}},
EndKey: &pb.MaybeBytes{Value: []byte{0}},
},
proofNil: true,
expectedErr: p2p.ErrUnexpected,
},
{
name: "response bounded by key limit",
request: &pb.SyncGetRangeProofRequest{
RootHash: smallTrieRoot[:],
KeyLimit: 2 * defaultRequestKeyLimit,
BytesLimit: defaultRequestByteSizeLimit,
},
expectedResponseLen: defaultRequestKeyLimit,
},
{
name: "response bounded by byte limit",
request: &pb.SyncGetRangeProofRequest{
RootHash: smallTrieRoot[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: 2 * defaultRequestByteSizeLimit,
},
expectedMaxResponseBytes: defaultRequestByteSizeLimit,
},
{
name: "empty proof",
request: &pb.SyncGetRangeProofRequest{
RootHash: ids.Empty[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: defaultRequestByteSizeLimit,
},
proofNil: true,
expectedErr: p2p.ErrUnexpected,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
require := require.New(t)
handler := NewGetRangeProofHandler(smallTrieDB)
requestBytes, err := proto.Marshal(test.request)
require.NoError(err)
responseBytes, err := handler.Request(context.Background(), test.nodeID, time.Time{}, requestBytes)
require.ErrorIs(err, test.expectedErr)
if test.expectedErr != nil {
return
}
if test.proofNil {
require.Nil(responseBytes)
return
}
var proofProto pb.RangeProof
require.NoError(proto.Unmarshal(responseBytes, &proofProto))
var proof merkledb.RangeProof
require.NoError(proof.UnmarshalProto(&proofProto))
if test.expectedResponseLen > 0 {
require.LessOrEqual(len(proof.KeyChanges), test.expectedResponseLen)
}
bytes, err := proto.Marshal(proof.ToProto())
require.NoError(err)
require.LessOrEqual(len(bytes), int(test.request.BytesLimit))
if test.expectedMaxResponseBytes > 0 {
require.LessOrEqual(len(bytes), test.expectedMaxResponseBytes)
}
})
}
}
func Test_Server_GetChangeProof(t *testing.T) {
now := time.Now().UnixNano()
t.Logf("seed: %d", now)
r := rand.New(rand.NewSource(now)) // #nosec G404
serverDB, err := merkledb.New(
context.Background(),
memdb.New(),
newDefaultDBConfig(),
)
require.NoError(t, err)
startRoot, err := serverDB.GetMerkleRoot(context.Background())
require.NoError(t, err)
// create changes
for x := 0; x < defaultRequestKeyLimit/2; x++ {
ops := make([]database.BatchOp, 0, 11)
// add some key/values
for i := 0; i < 10; i++ {
key := make([]byte, r.Intn(100))
_, err = r.Read(key)
require.NoError(t, err)
val := make([]byte, r.Intn(100))
_, err = r.Read(val)
require.NoError(t, err)
ops = append(ops, database.BatchOp{Key: key, Value: val})
}
// delete a key
deleteKeyStart := make([]byte, r.Intn(10))
_, err = r.Read(deleteKeyStart)
require.NoError(t, err)
it := serverDB.NewIteratorWithStart(deleteKeyStart)
if it.Next() {
ops = append(ops, database.BatchOp{Key: it.Key(), Delete: true})
}
require.NoError(t, it.Error())
it.Release()
view, err := serverDB.NewView(
context.Background(),
merkledb.ViewChanges{BatchOps: ops},
)
require.NoError(t, err)
require.NoError(t, view.CommitToDB(context.Background()))
}
endRoot, err := serverDB.GetMerkleRoot(context.Background())
require.NoError(t, err)
fakeRootID := ids.GenerateTestID()
tests := []struct {
name string
request *pb.SyncGetChangeProofRequest
expectedErr *common.Error
expectedResponseLen int
expectedMaxResponseBytes int
nodeID ids.NodeID
expectRangeProof bool // Otherwise expect change proof
}{
{
name: "proof restricted by BytesLimit",
request: &pb.SyncGetChangeProofRequest{
StartRootHash: startRoot[:],
EndRootHash: endRoot[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: 10000,
},
},
{
name: "full response for small (single request) trie",
request: &pb.SyncGetChangeProofRequest{
StartRootHash: startRoot[:],
EndRootHash: endRoot[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: defaultRequestByteSizeLimit,
},
expectedResponseLen: defaultRequestKeyLimit,
},
{
name: "partial response to request for entire trie (full leaf limit)",
request: &pb.SyncGetChangeProofRequest{
StartRootHash: startRoot[:],
EndRootHash: endRoot[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: defaultRequestByteSizeLimit,
},
expectedResponseLen: defaultRequestKeyLimit,
},
{
name: "byteslimit is 0",
request: &pb.SyncGetChangeProofRequest{
StartRootHash: startRoot[:],
EndRootHash: endRoot[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: 0,
},
expectedErr: p2p.ErrUnexpected,
},
{
name: "keylimit is 0",
request: &pb.SyncGetChangeProofRequest{
StartRootHash: startRoot[:],
EndRootHash: endRoot[:],
KeyLimit: 0,
BytesLimit: defaultRequestByteSizeLimit,
},
expectedErr: p2p.ErrUnexpected,
},
{
name: "keys out of order",
request: &pb.SyncGetChangeProofRequest{
StartRootHash: startRoot[:],
EndRootHash: endRoot[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: defaultRequestByteSizeLimit,
StartKey: &pb.MaybeBytes{Value: []byte{1}},
EndKey: &pb.MaybeBytes{Value: []byte{0}},
},
expectedErr: p2p.ErrUnexpected,
},
{
name: "key limit too large",
request: &pb.SyncGetChangeProofRequest{
StartRootHash: startRoot[:],
EndRootHash: endRoot[:],
KeyLimit: 2 * defaultRequestKeyLimit,
BytesLimit: defaultRequestByteSizeLimit,
},
expectedResponseLen: defaultRequestKeyLimit,
},
{
name: "bytes limit too large",
request: &pb.SyncGetChangeProofRequest{
StartRootHash: startRoot[:],
EndRootHash: endRoot[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: 2 * defaultRequestByteSizeLimit,
},
expectedMaxResponseBytes: defaultRequestByteSizeLimit,
},
{
name: "insufficient history for change proof; return range proof",
request: &pb.SyncGetChangeProofRequest{
// This root doesn't exist so server has insufficient history
// to serve a change proof
StartRootHash: fakeRootID[:],
EndRootHash: endRoot[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: defaultRequestByteSizeLimit,
},
expectedMaxResponseBytes: defaultRequestByteSizeLimit,
expectRangeProof: true,
},
{
name: "insufficient history for change proof or range proof",
request: &pb.SyncGetChangeProofRequest{
// These roots don't exist so server has insufficient history
// to serve a change proof or range proof
StartRootHash: ids.Empty[:],
EndRootHash: fakeRootID[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: defaultRequestByteSizeLimit,
},
expectedMaxResponseBytes: defaultRequestByteSizeLimit,
expectedErr: p2p.ErrUnexpected,
},
{
name: "empty proof",
request: &pb.SyncGetChangeProofRequest{
StartRootHash: fakeRootID[:],
EndRootHash: ids.Empty[:],
KeyLimit: defaultRequestKeyLimit,
BytesLimit: defaultRequestByteSizeLimit,
},
expectedMaxResponseBytes: defaultRequestByteSizeLimit,
expectedErr: p2p.ErrUnexpected,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
require := require.New(t)
handler := NewGetChangeProofHandler(serverDB)
requestBytes, err := proto.Marshal(test.request)
require.NoError(err)
proofBytes, err := handler.Request(context.Background(), test.nodeID, time.Time{}, requestBytes)
require.ErrorIs(err, test.expectedErr)
if test.expectedErr != nil {
require.Nil(proofBytes)
return
}
proofResult := &pb.SyncGetChangeProofResponse{}
require.NoError(proto.Unmarshal(proofBytes, proofResult))
if test.expectRangeProof {
require.NotNil(proofResult.GetRangeProof())
} else {
require.NotNil(proofResult.GetChangeProof())
}
if test.expectedResponseLen > 0 {
if test.expectRangeProof {
require.LessOrEqual(len(proofResult.GetRangeProof().KeyValues), test.expectedResponseLen)
} else {
require.LessOrEqual(len(proofResult.GetChangeProof().KeyChanges), test.expectedResponseLen)
}
}
require.LessOrEqual(len(proofBytes), int(test.request.BytesLimit))
if test.expectedMaxResponseBytes > 0 {
require.LessOrEqual(len(proofBytes), test.expectedMaxResponseBytes)
}
})
}
}
x/sync/protoutils/utils.go
-28
View File
@@ -1,28 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package protoutils
import (
"github.com/luxfi/container/maybe"
pb "github.com/luxfi/node/proto/pb/sync"
)
func MaybeToProto(m maybe.Maybe[[]byte]) *pb.MaybeBytes {
if m.IsNothing() {
return nil
}
return &pb.MaybeBytes{
Value: m.Value(),
}
}
func ProtoToMaybe(mb *pb.MaybeBytes) maybe.Maybe[[]byte] {
if mb == nil {
return maybe.Nothing[[]byte]()
}
return maybe.Some(mb.Value)
}
x/sync/response_handler.go
-43
View File
@@ -1,43 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package sync
var _ ResponseHandler = (*responseHandler)(nil)
// Handles responses/failure notifications for a sent request.
// Exactly one of OnResponse or OnFailure is eventually called.
type ResponseHandler interface {
// Called when [response] is received.
OnResponse(response []byte)
// Called when the request failed or timed out.
OnFailure()
}
func newResponseHandler() *responseHandler {
return &responseHandler{responseChan: make(chan []byte)}
}
// Implements [ResponseHandler].
// Used to wait for a response after making a synchronous request.
// responseChan contains response bytes if the request succeeded.
// responseChan is closed in either fail or success scenario.
type responseHandler struct {
// If [OnResponse] is called, the response bytes are sent on this channel.
// If [OnFailure] is called, the channel is closed without sending bytes.
responseChan chan []byte
}
// OnResponse passes the response bytes to the responseChan and closes the
// channel.
func (h *responseHandler) OnResponse(response []byte) {
h.responseChan <- response
close(h.responseChan)
}
// OnFailure closes the channel.
func (h *responseHandler) OnFailure() {
close(h.responseChan)
}
x/sync/sync_test.go
-1434
View File
File diff suppressed because it is too large Load Diff
x/sync/syncmock/network_client.go
-129
View File
@@ -1,129 +0,0 @@
// Code generated by MockGen. DO NOT EDIT.
// Source: github.com/luxfi/node/x/sync (interfaces: NetworkClient)
//
// Generated by this command:
//
// mockgen -package=syncmock -destination=x/sync/syncmock/network_client.go -mock_names=NetworkClient=NetworkClient github.com/luxfi/node/x/sync NetworkClient
//
// Package syncmock is a generated GoMock package.
package syncmock
import (
context "context"
reflect "reflect"
ids "github.com/luxfi/ids"
version "github.com/luxfi/node/version"
gomock "go.uber.org/mock/gomock"
)
// NetworkClient is a mock of NetworkClient interface.
type NetworkClient struct {
ctrl *gomock.Controller
recorder *NetworkClientMockRecorder
}
// NetworkClientMockRecorder is the mock recorder for NetworkClient.
type NetworkClientMockRecorder struct {
mock *NetworkClient
}
// NewNetworkClient creates a new mock instance.
func NewNetworkClient(ctrl *gomock.Controller) *NetworkClient {
mock := &NetworkClient{ctrl: ctrl}
mock.recorder = &NetworkClientMockRecorder{mock}
return mock
}
// EXPECT returns an object that allows the caller to indicate expected use.
func (m *NetworkClient) EXPECT() *NetworkClientMockRecorder {
return m.recorder
}
// RequestFailed mocks base method.
func (m *NetworkClient) RequestFailed(arg0 context.Context, arg1 ids.NodeID, arg2 uint32) error {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "RequestFailed", arg0, arg1, arg2)
ret0, _ := ret[0].(error)
return ret0
}
// RequestFailed indicates an expected call of RequestFailed.
func (mr *NetworkClientMockRecorder) RequestFailed(arg0, arg1, arg2 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RequestFailed", reflect.TypeOf((*NetworkClient)(nil).RequestFailed), arg0, arg1, arg2)
}
// Response mocks base method.
func (m *NetworkClient) Response(arg0 context.Context, arg1 ids.NodeID, arg2 uint32, arg3 []byte) error {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "Response", arg0, arg1, arg2, arg3)
ret0, _ := ret[0].(error)
return ret0
}
// Response indicates an expected call of Response.
func (mr *NetworkClientMockRecorder) Response(arg0, arg1, arg2, arg3 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Response", reflect.TypeOf((*NetworkClient)(nil).Response), arg0, arg1, arg2, arg3)
}
// Connected mocks base method.
func (m *NetworkClient) Connected(arg0 context.Context, arg1 ids.NodeID, arg2 *version.Application) error {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "Connected", arg0, arg1, arg2)
ret0, _ := ret[0].(error)
return ret0
}
// Connected indicates an expected call of Connected.
func (mr *NetworkClientMockRecorder) Connected(arg0, arg1, arg2 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Connected", reflect.TypeOf((*NetworkClient)(nil).Connected), arg0, arg1, arg2)
}
// Disconnected mocks base method.
func (m *NetworkClient) Disconnected(arg0 context.Context, arg1 ids.NodeID) error {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "Disconnected", arg0, arg1)
ret0, _ := ret[0].(error)
return ret0
}
// Disconnected indicates an expected call of Disconnected.
func (mr *NetworkClientMockRecorder) Disconnected(arg0, arg1 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Disconnected", reflect.TypeOf((*NetworkClient)(nil).Disconnected), arg0, arg1)
}
// Request mocks base method.
func (m *NetworkClient) Request(arg0 context.Context, arg1 ids.NodeID, arg2 []byte) ([]byte, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "Request", arg0, arg1, arg2)
ret0, _ := ret[0].([]byte)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// Request indicates an expected call of Request.
func (mr *NetworkClientMockRecorder) Request(arg0, arg1, arg2 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Request", reflect.TypeOf((*NetworkClient)(nil).Request), arg0, arg1, arg2)
}
// RequestAny mocks base method.
func (m *NetworkClient) RequestAny(arg0 context.Context, arg1 []byte) (ids.NodeID, []byte, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "RequestAny", arg0, arg1)
ret0, _ := ret[0].(ids.NodeID)
ret1, _ := ret[1].([]byte)
ret2, _ := ret[2].(error)
return ret0, ret1, ret2
}
// RequestAny indicates an expected call of RequestAny.
func (mr *NetworkClientMockRecorder) RequestAny(arg0, arg1 any) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RequestAny", reflect.TypeOf((*NetworkClient)(nil).RequestAny), arg0, arg1)
}
x/sync/workheap.go
-161
View File
@@ -1,161 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package sync
import (
"bytes"
"github.com/google/btree"
"github.com/luxfi/container/heap"
"github.com/luxfi/container/maybe"
)
// A priority queue of syncWorkItems.
// Note that work item ranges never overlap.
// Supports range merging and priority updating.
// Not safe for concurrent use.
type workHeap struct {
// Max heap of items by priority.
// i.e. heap.Pop returns highest priority item.
innerHeap heap.Set[*workItem]
// The heap items sorted by range start.
// A Nothing start is considered to be the smallest.
sortedItems *btree.BTreeG[*workItem]
closed bool
}
func newWorkHeap() *workHeap {
return &workHeap{
innerHeap: heap.NewSet[*workItem](func(a, b *workItem) bool {
return a.priority > b.priority
}),
sortedItems: btree.NewG(
2,
func(a, b *workItem) bool {
aNothing := a.start.IsNothing()
bNothing := b.start.IsNothing()
if aNothing {
// [a] is Nothing, so if [b] is Nothing, they're equal.
// Otherwise, [b] is greater.
return !bNothing
}
if bNothing {
// [a] has a value and [b] doesn't so [a] is greater.
return false
}
// [a] and [b] both contain values. Compare the values.
return bytes.Compare(a.start.Value(), b.start.Value()) < 0
},
),
}
}
// Marks the heap as closed.
func (wh *workHeap) Close() {
wh.closed = true
}
// Adds a new [item] into the heap. Will not merge items, unlike MergeInsert.
func (wh *workHeap) Insert(item *workItem) {
if wh.closed {
return
}
wh.innerHeap.Push(item)
wh.sortedItems.ReplaceOrInsert(item)
}
// Pops and returns a work item from the heap.
// Returns nil if no work is available or the heap is closed.
func (wh *workHeap) GetWork() *workItem {
if wh.closed || wh.Len() == 0 {
return nil
}
item, _ := wh.innerHeap.Pop()
wh.sortedItems.Delete(item)
return item
}
// Insert the item into the heap, merging it with existing items
// that share a boundary and root ID.
// e.g. if the heap contains a work item with range
// [0,10] and then [10,20] is inserted, we will merge the two
// into a single work item with range [0,20].
// e.g. if the heap contains work items [0,10] and [20,30],
// and we add [10,20], we will merge them into [0,30].
func (wh *workHeap) MergeInsert(item *workItem) {
if wh.closed {
return
}
var mergedBefore, mergedAfter *workItem
searchItem := &workItem{
start: item.start,
}
// Find the item with the greatest start range which is less than [item.start].
// Note that the iterator function will run at most once, since it always returns false.
wh.sortedItems.DescendLessOrEqual(
searchItem,
func(beforeItem *workItem) bool {
if item.localRootID == beforeItem.localRootID &&
maybe.Equal(item.start, beforeItem.end, bytes.Equal) {
// [beforeItem.start, beforeItem.end] and [item.start, item.end] are
// merged into [beforeItem.start, item.end]
beforeItem.end = item.end
beforeItem.priority = max(item.priority, beforeItem.priority)
wh.innerHeap.Fix(beforeItem)
mergedBefore = beforeItem
}
return false
})
// Find the item with the smallest start range which is greater than [item.start].
// Note that the iterator function will run at most once, since it always returns false.
wh.sortedItems.AscendGreaterOrEqual(
searchItem,
func(afterItem *workItem) bool {
if item.localRootID == afterItem.localRootID &&
maybe.Equal(item.end, afterItem.start, bytes.Equal) {
// [item.start, item.end] and [afterItem.start, afterItem.end] are merged into
// [item.start, afterItem.end].
afterItem.start = item.start
afterItem.priority = max(item.priority, afterItem.priority)
wh.innerHeap.Fix(afterItem)
mergedAfter = afterItem
}
return false
})
// if the new item should be merged with both the item before and the item after,
// we can combine the before item with the after item
if mergedBefore != nil && mergedAfter != nil {
// combine the two ranges
mergedBefore.end = mergedAfter.end
// remove the second range since it is now covered by the first
wh.remove(mergedAfter)
// update the priority
mergedBefore.priority = max(mergedBefore.priority, mergedAfter.priority)
wh.innerHeap.Fix(mergedBefore)
}
// nothing was merged, so add new item to the heap
if mergedBefore == nil && mergedAfter == nil {
// We didn't merge [item] with an existing one; put it in the heap.
wh.Insert(item)
}
}
// Deletes [item] from the heap.
func (wh *workHeap) remove(item *workItem) {
wh.innerHeap.Remove(item)
wh.sortedItems.Delete(item)
}
func (wh *workHeap) Len() int {
return wh.innerHeap.Len()
}
x/sync/workheap_test.go
-312
View File
@@ -1,312 +0,0 @@
//go:build grpc
// Copyright (C) 2019-2025, Lux Industries Inc. All rights reserved.
// See the file LICENSE for licensing terms.
package sync
import (
"bytes"
"math/rand"
"slices"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/luxfi/ids"
"github.com/luxfi/container/maybe"
)
// Tests Insert and GetWork
func Test_WorkHeap_Insert_GetWork(t *testing.T) {
require := require.New(t)
h := newWorkHeap()
lowPriorityItem := &workItem{
start: maybe.Some([]byte{4}),
end: maybe.Some([]byte{5}),
priority: lowPriority,
localRootID: ids.GenerateTestID(),
}
mediumPriorityItem := &workItem{
start: maybe.Some([]byte{0}),
end: maybe.Some([]byte{1}),
priority: medPriority,
localRootID: ids.GenerateTestID(),
}
highPriorityItem := &workItem{
start: maybe.Some([]byte{2}),
end: maybe.Some([]byte{3}),
priority: highPriority,
localRootID: ids.GenerateTestID(),
}
h.Insert(highPriorityItem)
h.Insert(mediumPriorityItem)
h.Insert(lowPriorityItem)
require.Equal(3, h.Len())
// Ensure [sortedItems] is in right order.
got := []*workItem{}
h.sortedItems.Ascend(
func(i *workItem) bool {
got = append(got, i)
return true
},
)
require.Equal(
[]*workItem{mediumPriorityItem, highPriorityItem, lowPriorityItem},
got,
)
// Ensure priorities are in right order.
gotItem := h.GetWork()
require.Equal(highPriorityItem, gotItem)
gotItem = h.GetWork()
require.Equal(mediumPriorityItem, gotItem)
gotItem = h.GetWork()
require.Equal(lowPriorityItem, gotItem)
gotItem = h.GetWork()
require.Nil(gotItem)
require.Zero(h.Len())
}
func Test_WorkHeap_remove(t *testing.T) {
require := require.New(t)
h := newWorkHeap()
lowPriorityItem := &workItem{
start: maybe.Some([]byte{0}),
end: maybe.Some([]byte{1}),
priority: lowPriority,
localRootID: ids.GenerateTestID(),
}
mediumPriorityItem := &workItem{
start: maybe.Some([]byte{2}),
end: maybe.Some([]byte{3}),
priority: medPriority,
localRootID: ids.GenerateTestID(),
}
highPriorityItem := &workItem{
start: maybe.Some([]byte{4}),
end: maybe.Some([]byte{5}),
priority: highPriority,
localRootID: ids.GenerateTestID(),
}
h.Insert(lowPriorityItem)
wrappedLowPriorityItem, ok := h.innerHeap.Peek()
require.True(ok)
h.remove(wrappedLowPriorityItem)
require.Zero(h.Len())
require.Zero(h.sortedItems.Len())
h.Insert(lowPriorityItem)
h.Insert(mediumPriorityItem)
h.Insert(highPriorityItem)
wrappedhighPriorityItem, ok := h.innerHeap.Peek()
require.True(ok)
require.Equal(highPriorityItem, wrappedhighPriorityItem)
h.remove(wrappedhighPriorityItem)
require.Equal(2, h.Len())
require.Equal(2, h.sortedItems.Len())
got, ok := h.innerHeap.Peek()
require.True(ok)
require.Equal(mediumPriorityItem, got)
wrappedMediumPriorityItem, ok := h.innerHeap.Peek()
require.True(ok)
require.Equal(mediumPriorityItem, wrappedMediumPriorityItem)
h.remove(wrappedMediumPriorityItem)
require.Equal(1, h.Len())
require.Equal(1, h.sortedItems.Len())
got, ok = h.innerHeap.Peek()
require.True(ok)
require.Equal(lowPriorityItem, got)
wrappedLowPriorityItem, ok = h.innerHeap.Peek()
require.True(ok)
require.Equal(lowPriorityItem, wrappedLowPriorityItem)
h.remove(wrappedLowPriorityItem)
require.Zero(h.Len())
require.Zero(h.sortedItems.Len())
}
func Test_WorkHeap_Merge_Insert(t *testing.T) {
// merge with range before
syncHeap := newWorkHeap()
syncHeap.MergeInsert(&workItem{start: maybe.Nothing[[]byte](), end: maybe.Some([]byte{63})})
require.Equal(t, 1, syncHeap.Len())
syncHeap.MergeInsert(&workItem{start: maybe.Some([]byte{127}), end: maybe.Some([]byte{192})})
require.Equal(t, 2, syncHeap.Len())
syncHeap.MergeInsert(&workItem{start: maybe.Some([]byte{193}), end: maybe.Nothing[[]byte]()})
require.Equal(t, 3, syncHeap.Len())
syncHeap.MergeInsert(&workItem{start: maybe.Some([]byte{63}), end: maybe.Some([]byte{126}), priority: lowPriority})
require.Equal(t, 3, syncHeap.Len())
// merge with range after
syncHeap = newWorkHeap()
syncHeap.MergeInsert(&workItem{start: maybe.Nothing[[]byte](), end: maybe.Some([]byte{63})})
require.Equal(t, 1, syncHeap.Len())
syncHeap.MergeInsert(&workItem{start: maybe.Some([]byte{127}), end: maybe.Some([]byte{192})})
require.Equal(t, 2, syncHeap.Len())
syncHeap.MergeInsert(&workItem{start: maybe.Some([]byte{193}), end: maybe.Nothing[[]byte]()})
require.Equal(t, 3, syncHeap.Len())
syncHeap.MergeInsert(&workItem{start: maybe.Some([]byte{64}), end: maybe.Some([]byte{127}), priority: lowPriority})
require.Equal(t, 3, syncHeap.Len())
// merge both sides at the same time
syncHeap = newWorkHeap()
syncHeap.MergeInsert(&workItem{start: maybe.Nothing[[]byte](), end: maybe.Some([]byte{63})})
require.Equal(t, 1, syncHeap.Len())
syncHeap.MergeInsert(&workItem{start: maybe.Some([]byte{127}), end: maybe.Nothing[[]byte]()})
require.Equal(t, 2, syncHeap.Len())
syncHeap.MergeInsert(&workItem{start: maybe.Some([]byte{63}), end: maybe.Some([]byte{127}), priority: lowPriority})
require.Equal(t, 1, syncHeap.Len())
}
func TestWorkHeapMergeInsertRandom(t *testing.T) {
var (
require = require.New(t)
seed = time.Now().UnixNano()
rand = rand.New(rand.NewSource(seed)) // #nosec G404
numRanges = 1_000
bounds = [][]byte{}
rootID = ids.GenerateTestID()
)
t.Logf("seed: %d", seed)
// Create start and end bounds
for i := 0; i < numRanges; i++ {
bound := make([]byte, 32)
_, _ = rand.Read(bound)
bounds = append(bounds, bound)
}
slices.SortFunc(bounds, bytes.Compare)
// Note that start < end for all ranges.
// It is possible but extremely unlikely that
// two elements of [bounds] are equal.
ranges := []workItem{}
for i := 0; i < numRanges/2; i++ {
start := bounds[i*2]
end := bounds[i*2+1]
ranges = append(ranges, workItem{
start: maybe.Some(start),
end: maybe.Some(end),
priority: lowPriority,
// Note they all share the same root ID.
localRootID: rootID,
})
}
// Set beginning of first range to Nothing.
ranges[0].start = maybe.Nothing[[]byte]()
// Set end of last range to Nothing.
ranges[len(ranges)-1].end = maybe.Nothing[[]byte]()
setup := func() *workHeap {
// Insert all the ranges into the heap.
h := newWorkHeap()
for i, r := range ranges {
require.Equal(i, h.Len())
rCopy := r
h.MergeInsert(&rCopy)
}
return h
}
{
// Case 1: Merging an item with the range before and after
h := setup()
// Keep merging ranges until there's only one range left.
for i := 0; i < len(ranges)-1; i++ {
// Merge ranges[i] with ranges[i+1]
h.MergeInsert(&workItem{
start: ranges[i].end,
end: ranges[i+1].start,
priority: lowPriority,
localRootID: rootID,
})
require.Equal(len(ranges)-i-1, h.Len())
}
got := h.GetWork()
require.True(got.start.IsNothing())
require.True(got.end.IsNothing())
}
{
// Case 2: Merging an item with the range before
h := setup()
for i := 0; i < len(ranges)-1; i++ {
// Extend end of ranges[i]
newEnd := slices.Clone(ranges[i].end.Value())
newEnd = append(newEnd, 0)
h.MergeInsert(&workItem{
start: ranges[i].end,
end: maybe.Some(newEnd),
priority: lowPriority,
localRootID: rootID,
})
// Shouldn't cause number of elements to change
require.Equal(len(ranges), h.Len())
start := ranges[i].start
if i == 0 {
start = maybe.Nothing[[]byte]()
}
// Make sure end is updated
got, ok := h.sortedItems.Get(&workItem{
start: start,
})
require.True(ok)
require.Equal(newEnd, got.end.Value())
}
}
{
// Case 3: Merging an item with the range after
h := setup()
for i := 1; i < len(ranges); i++ {
// Extend start of ranges[i]
newStartBytes := slices.Clone(ranges[i].start.Value())
newStartBytes = newStartBytes[:len(newStartBytes)-1]
newStart := maybe.Some(newStartBytes)
h.MergeInsert(&workItem{
start: newStart,
end: ranges[i].start,
priority: lowPriority,
localRootID: rootID,
})
// Shouldn't cause number of elements to change
require.Equal(len(ranges), h.Len())
// Make sure start is updated
got, ok := h.sortedItems.Get(&workItem{
start: newStart,
})
require.True(ok)
require.Equal(newStartBytes, got.start.Value())
}
}
}