Removed /v4 suffix from module path and all internal imports.
Go module is now github.com/luxfi/zapdb (no major version suffix).
Import as: import "github.com/luxfi/zapdb"
No code changes — only module path and import rewrite.
Replace proto.Marshal/Unmarshal with native pb.Marshal/pb.Unmarshal to
match the wire format used by luxfi/badger/v4. This ensures databases
written by nodes using the badger fork (v1.23.4) can be read by nodes
using zapdb (v1.23.5), preventing "proto: cannot parse invalid
wire-format data" errors during rolling upgrades.
Added types_zap.go and marshal_zap.go from luxfi/badger fork, which
provide custom LittleEndian binary encoding for ManifestChangeSet,
KV, Checksum, and other internal types. The standard protobuf
generated code (badgerpb4.pb.go) is now behind a grpc build tag.
- Rename module from github.com/dgraph-io/badger/v4 to github.com/luxfi/zapdb/v4
- Update all internal imports to use new module path
- Preserve all ZAP enhancements from badger fork
* Let's use allocator again
* Switch to z.NumAllocBytes
* Make stream work with both another Badger DB or with a file to backup.
* Add a test for Allocator
* Use allocator for Backup
* Bring in latest Ristretto
Co-authored-by: Daniel Mai <daniel@dgraph.io>
Stream.Send now sends out z.Buffer instead of pb.KVList. z.Buffer marshals each KV as a separate slice. This significantly reduces the memory requirement by the Stream framework. Stream no longer uses z.Allocator or tries to put pb.KV struct on the Allocator for memory safety reasons.
Bring back the z.AllocatorPool for table.Builder.
Changes:
* Use z.Buffer for stream.Send
* Only use 8 streams in write bench
* Revert "Bug Fix: Fix up how we use z.Allocator"
This reverts commit d6defb637d.
* Bring allocator back. Use z.Buffer for send
* Add BufferToKVList function
* Print jemalloc while stream
* Bring in latest Ristretto
* Fix memory leak and benchmark read test
Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
We removed the global z.Allocator pool from z package. Instead, we now use a new z.AllocatorPool class in the places which need a pool. In this case, we brought it to TableBuilder and Stream.
Fix up a memory leak in Stream.
Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
Run tests in parallel. Break up the tests into 3 functions: manual, packages and root. Run those via GNU parallel. Moved some expensive tests to manual mode.
With these changes, the badger tests can now run in under 6 minutes on a fast machine.
Co-authored-by: Manish R Jain <manish@dgraph.io>
KVs can take up a lot of memory in the stream framework. With this change, we allocate them using z.Allocator, and allow callers to KeyToList to use the allocator to generate KVs as well. After we call Send, we release them.
Also change Stream Framework to spit out StreamDone markers.
Currently, there's no limit to how many KVs can be included in a single
batch. This leads to issues when the size is over a hard limit. For
example, a batch of size > 2GB will cause issues if it has to be sent
over gRPC.
Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
The iterator has a new field ThreadId that is set by the stream framework. This allows
the KeyToList function which thread it's running on and can be used to simulate
thread-local storage.
As explained in https://discuss.dgraph.io/t/go-modules-on-badger-and-dgraph/4662,
Go Modules force an import path rename that currently breaks support for most other
dependency management systems, including dep.
It seems an effort is being made to add support for this import path renaming by
patching the go tool itself, but until the community hasn't settled and Go Modules
is an easier option to adopt, we are simply not supporting them.
For those willing to use badger v2 from Go modules it is still very simple, you might
notice that once you set the version to be v2.0.0 a +incompatible will be added to it
indicating the dependency has not opted-in into Go Modules. That's normal and not to
be a concern.
* Add functions to create, modify Entry
Currently no functions are exposed to create and modify Entry, to
be set in a transaction. Functions Set, SetWithMeta, SetWithDiscard,
SetWithTTL, all are part of Txn struct. These functions internally
create an entry and then call txn.SetEntry. All of Set functions,
set a specific property of Entry apart from key and value. This
restricts us from setting multiple property for same entry such as
meta and TTL.
This PR adds functions to set different properties on an existing
Entry. After setting desired properties, user can call txn.SetEntry
to add entry in the transaction. It also removes entry related functions
from Txn Struct except SetEntry and Set.
* Fix all compilation failure due to above change using
new Entry methods.
* Fix comments from GolangCI.
- Fix a race condition in DropAll, caused if concurrent writes are going on.
- Use `blockWrites` during DB.Close as well, so users don't see channel panics when doing writes after DB.Close.
- Port tests over for Stream framework from Dgraph.
Changes:
* Starting to add a test for stream
* Add a test for Stream. Allow repeated calls to Orchestrate on the same Stream object.
* Fix various race conditions with concurrent writes during DropAll.
* Use blockWrites during Close as well, to avoid the chan panics users see when doing writes after Close.
* Add license for stream_test.go
* 100 chars
Hanzo AI