Removed /v4 suffix from module path and all internal imports.
Go module is now github.com/luxfi/zapdb (no major version suffix).
Import as: import "github.com/luxfi/zapdb"
No code changes — only module path and import rewrite.
- Rename module from github.com/dgraph-io/badger/v4 to github.com/luxfi/zapdb/v4
- Update all internal imports to use new module path
- Preserve all ZAP enhancements from badger fork
The write amplification with value log can be unpredictably high. It's better to use value log only for really big values, and keep as many values as possible within the LSM tree.
SinceTs can be used to read data above a particular timestamp. All data
with version less than or equal to the sinceTs will be ignored.
SinceTs is always less than the readTs and when sinceTs is set, the
data between sinceTs and readTs will be read.
sinceTs < data to be read <= readTs
Fixes DGRAPH-2958
Implement multiple ideas for speeding up compactions:
1. Dynamic Level Sizes: https://rocksdb.org/blog/2015/07/23/dynamic-level.html
2. L0 to L0 compactions: https://rocksdb.org/blog/2017/06/26/17-level-based-changes.html
3. Sub Compactions: Split up one compaction into multiple sub-compactions using key ranges, which can be run concurrently.
4. If a table being generated at Li overlaps with >= 10 tables at Li+1, finish the table. This helps avoid big overlaps and expensive compactions later.
5. Update compaction priority based on the priority of the next level prioritizing compactions of lower levels over upper levels, resulting in an always healthy LSM tree structure.
With these changes, we can load 1B entries (160GB of data) into Badger (without the Stream framework) in 1h25m at 31 MB/s. This is a significant improvement over current master.
Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
This PR significantly improves Badger's disk usage behavior.
Breaking: This PR increases the magic version from 7 to 8. So, no older Badger directories would work with this change.
With this PR, we no longer use value log as write-ahead log. Instead, each MemTable has its own WAL. Value logs now only write values which are greater than ValueThreshold, while MemTable WAL only writes smaller values and value pointers.
On a crash and restart, the MemTable WALs are replayed to apply updates to Skiplist. When MemTables are flushed to L0, the corresponding WALs are deleted.
This PR makes big changes to how value log GC works:
- Discard stats are now stored in a separate file, instead of within the LSM tree.
- GC only picks up value logs based off discard stats.
- GC no longer does sampling, it uses discard stats to inform when a value log needs to be GCed.
- Value log would now no longer grow indefinitely, because of the shift to MemTable WAL.
- Removed the `badger gc` tool.
- Value Log Head pointer tracking is removed.
- Only the last value log file is replayed on every start, and truncated as necessary.
This PR also makes a bunch of other changes:
- Removes ValueLogLoadingMode (always uses mmap now).
- Removes TableLoadingMode (always uses mmap now).
- Removes Truncate option.
- Removes KeepL0InMemory option.
Increase value threshold from 32 bytes to 1 KB to allow easier space
reclaimation. There have been multiple reports about value log garbage
collection being unsatisfactory. This commit increases the default value
of threshold from 32 bytes to 1 KB which would allow keys and values
upto 1 KB to be co-allocated in the LSM tree. This would allow GC to
reclaim space easily for values less than 1 KB.
A bunch of tests would start goroutines and never stop them. This PR fixes
that. There should not be any goroutine left after all the tests are
completed.
This PR also adds a db.cleanup method which would be called in case badger.Open
call fails. The cleanup method will stop all the goroutines started by Open
call.
When duplicate entries with different versions are inserted, the
current implementation would store only the latest update and
drop all the previous ones. This PR fixes it. We keep all
duplicate entries in ManagedWriteBatch.
This PR adds support for setting different versions for different
keys in write batch. The existing implementation of write batch
allows setting only a single version for all keys in the write batch.
With this PR, user can do the following
```
wb := db.NewManagedWriteBatch()
wb.SetEntryAt(e, ts)
wb.Flush()
```
Also, the existing behavior of `txn.Commit()` in `un-managed
(normal) mode` is to panic, the new behavior would be to
return an error.
This PR introduces in-memory mode in badger. The in-memory mode
can be enabled by setting options.InMemory=true. When badger is
running in in-memory mode no files are created and everything
is stored in memory.
On DB close, all stored data is lost.
NOTE - An existing DB cannot be opened in in-memory mode.
Fixes - https://github.com/dgraph-io/badger/issues/1001
This commit adds ristretto to badger. All the reads are done through
the block cache. The block cache holds blocks in a decompressed and
unencrypted state. The memory usage of cache can be changed by
`MaxCacheSize` option. The default size of the cache is 1 GB.
This commit fixes windows build issues. There were 4 issues
1. Trying to truncate a mmapped file.
On linux, we can truncate a mmapped file but on windows this
operation is invalid. We should unmap the file before truncating it.
2. Removing a file that has open file descriptor.
On windows we cannot delete a file if it has an open file descriptor.
The following functions would increment the ref for a table but never
decrease it.
- getTableInfo function in levels.go
The getTableInfo function would create an iterator but never
close it and because of this we would have issues trying to
delete the file.
- CreateTable function in stream writer
The createTable function calls OpenTable which sets the ref of a
table to 1 but we would never decrement this ref.
3. AppVeyor file checksum mismatch error (iterator errors)
These errors were caused because we were trying to create a file with
a random name but on appVeyor the `rand.Int63()` would return the same
value for multiple calls. The problem was that `rand.Seed` was called
with a `time.Now()` timestamp which wasn't always unique on AppVeyor.
4. AppVeyor out of disk space error.
In some of the tests we weren't closing the DB and trying to remove
the directory. This meant the directory would never get deleted and
thus AppVeyor would run out of disk space.
With this commit, the AppVeyor build should succeed.
As explained in https://discuss.dgraph.io/t/go-modules-on-badger-and-dgraph/4662,
Go Modules force an import path rename that currently breaks support for most other
dependency management systems, including dep.
It seems an effort is being made to add support for this import path renaming by
patching the go tool itself, but until the community hasn't settled and Go Modules
is an easier option to adopt, we are simply not supporting them.
For those willing to use badger v2 from Go modules it is still very simple, you might
notice that once you set the version to be v2.0.0 a +incompatible will be added to it
indicating the dependency has not opted-in into Go Modules. That's normal and not to
be a concern.
* Remove SetWithTTL from WriteBatch as part of New EntryAPIs.
As we have removed SetWith functions from Txn struct. We can
do same for WriteBatch. Anyone can create Entry using new Entry
APIs and call SetEntry on WriteBatch.
* Change WriteBatch Set method to have same syntax as Txn.Set
* Fix DOCs for Entry APIs
* Add functions to create, modify Entry
Currently no functions are exposed to create and modify Entry, to
be set in a transaction. Functions Set, SetWithMeta, SetWithDiscard,
SetWithTTL, all are part of Txn struct. These functions internally
create an entry and then call txn.SetEntry. All of Set functions,
set a specific property of Entry apart from key and value. This
restricts us from setting multiple property for same entry such as
meta and TTL.
This PR adds functions to set different properties on an existing
Entry. After setting desired properties, user can call txn.SetEntry
to add entry in the transaction. It also removes entry related functions
from Txn Struct except SetEntry and Set.
* Fix all compilation failure due to above change using
new Entry methods.
* Fix comments from GolangCI.
This API would allow Badger to completely get rid of the given prefix, P. It does this by:
1. Flushing memtables to disk (skipping over the keys with P).
2. Doing normal compaction from L0 -> L1 (skipping keys with P).
3. Doing same level compaction on all the rest of the levels, picking tables which might have the prefix and rewriting them to new tables (again, skipping keys with P).
Before starting, all the writes are paused, the usual bits which apply to `DropAll` apply here as well.
Note: This PR has a breaking change for the Logger interface. It adds a Debugf API.
Changes:
* Working drop prefix
* Make the drop prefix more robust. Add a new Debugf API for logging.
* Add parallel tests for DropPrefix
* Avoid a test failure due to removing p dir.
* Only test DropPrefix in DropPrefix test.
* Address Martin's comments
- Fix a race condition in DropAll, caused if concurrent writes are going on.
- Use `blockWrites` during DB.Close as well, so users don't see channel panics when doing writes after DB.Close.
- Port tests over for Stream framework from Dgraph.
Changes:
* Starting to add a test for stream
* Add a test for Stream. Allow repeated calls to Orchestrate on the same Stream object.
* Fix various race conditions with concurrent writes during DropAll.
* Use blockWrites during Close as well, to avoid the chan panics users see when doing writes after Close.
* Add license for stream_test.go
* 100 chars
This PR ports over a Stream framework from Dgraph to Badger. This framework allows users to concurrently iterate over Badger, converting data to key-value lists, which then get sent out serially. In Dgraph we use this framework for shipping snapshots from leader to followers, doing periodic delta merging of updates, moving predicates from one Alpha group to another, and so on.
However, the framework is general enough that it could lie within Badger, and that's what this PR achieves. In addition, the Backup API of Badger now uses this framework to make taking backups faster.
Changes:
* Port Stream from Dgraph to Badger.
* Switch Backup to use the new Stream framework.
* Update godocs.
* Remove a t.Logf data race.
* Self-review
- Do not use deletion markers during DropAll, because that causes writes at lower timestamps to be ignored (in managed DB mode).
- Instead, delete all SSTables and all log files, and reset value log and value log head to zero.
- Make DropAll open to all users of Badger, including normal mode.
Changelog:
* Do not use delete markers in DropAll. Instead, delete the entire LSM tree and all the value logs.
* Moved DropAll to db.go, so it is available to all users of Badger. Fixed a bug where we were not catching memtable flushes during DropAll.
* Make DropAll test more races. Catch another place where writes should fail due to blockedWrite.
* Self-review
* Remove the extra ManagedDB struct, which was causing maintenance headaches, due to the way Go handles polymorphism. Instead just expose the ManagedTxns option, and use it directly.
* Bring back the OpenManaged function for convenience.
This method drops all tables from the LSM tree, except one to maintain the persistence of badgerHead key. This is important to avoid value log being replayed from scratch.
It then iterates over the rest of the keys and marks them as deleted. The end result should be that all keys are considered deleted.
This method would perform a lot better than deleting every key via iteration.
NOTE to users: The timestamp used for writes must be greater than the max timestamp of writes before DropAll, to ensure that new writes are not lower than the delete markers in terms of versioning.
Hanzo AI