Hanzo Stream
Kafka wire protocol adapter for Hanzo PubSub (NATS JetStream). Accepts standard Kafka clients and translates requests to NATS underneath, allowing any Kafka producer/consumer to work against Hanzo infrastructure without code changes.
Features
- Topic creation, produce, and consume via the Kafka binary protocol
- Cluster mode with Raft consensus and Serf membership
- Compression support
- Multiple log segments per partition
- Compatible with standard Kafka CLI tools and client libraries
Quick Start
Start a single-node broker on port 9092:
go run main.go --bootstrap --node-id 1
Test with any Kafka client:
# Create topic
kafka-topics.sh --create --topic events --bootstrap-server localhost:9092
# Produce
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic events
# Consume
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic events --from-beginning
Cluster Mode
Cluster mode uses Raft for distributed state and HashiCorp Serf for membership management.
# Bootstrap first node
go run main.go --bootstrap --node-id 1 --serf-addr 127.0.0.1:3331
# Join additional nodes
go run main.go --node-id 2 --broker-port 9093 --raft-addr localhost:2222 --serf-addr 127.0.0.1:3332 --serf-join "127.0.0.1:3331"
go run main.go --node-id 3 --broker-port 9094 --raft-addr localhost:2223 --serf-addr 127.0.0.1:3333 --serf-join "127.0.0.1:3331"
Kubernetes
In Hanzo infrastructure, Stream runs as a deployment in the hanzo namespace and connects to NATS via --pubsub-url nats://pubsub.hanzo.svc:4222.
Two instances are typically deployed:
| Deployment | Service | Purpose |
|---|---|---|
insights-kafka |
insights-kafka:9092 |
Dedicated to Insights pipeline |
stream |
stream:9092 |
General purpose |
Running Tests
Tests use Kafka CLI tools for end-to-end verification:
export KAFKA_BIN_DIR=/path/to/kafka_2.13-3.9.0/bin
go test -v ./...
Credits
Based on MonKafka by cefboud. Cluster mode inspired by Jocko.
License
MIT