In the existing implementation of oracle, if you happen to always have at least
one write transaction open the memory usage of the transaction oracle is
unbounded. It is actually relatively easy to hit when batch importing data. If
you have more than one WriteBatch active during the import the transaction
oracle will never be cleaned up.
This commit fixes it. The core idea is to avoid increasing contention on purely
read transactions; so only clean up the transaction oracle when write
transactions are committed even if technically we could free memory sooner;
Split the big `oracle.commit` map into one map per previously committed
transaction; (this allows Go to release memory sooner than when performing
deletes on a single map);
Take advantage of the fact that we have acquired the oracle lock in
oracle.newCommitTs to do the cleanup I am assuming here that the number of
committed-but-still-tracked transactions is small, which makes an
implementation based on a simple slice reasonable. If that's not the case we
will need some form of a sorted data-structure (i.e. a b-tree) here.
Co-authored-by: Damien Tournoud <damien@platform.sh>
When duplicate entries with different versions are inserted, the
current implementation would store only the latest update and
drop all the previous ones. This PR fixes it. We keep all
duplicate entries in ManagedWriteBatch.
This PR adds support for setting different versions for different
keys in write batch. The existing implementation of write batch
allows setting only a single version for all keys in the write batch.
With this PR, user can do the following
```
wb := db.NewManagedWriteBatch()
wb.SetEntryAt(e, ts)
wb.Flush()
```
Also, the existing behavior of `txn.Commit()` in `un-managed
(normal) mode` is to panic, the new behavior would be to
return an error.
- Do not use deletion markers during DropAll, because that causes writes at lower timestamps to be ignored (in managed DB mode).
- Instead, delete all SSTables and all log files, and reset value log and value log head to zero.
- Make DropAll open to all users of Badger, including normal mode.
Changelog:
* Do not use delete markers in DropAll. Instead, delete the entire LSM tree and all the value logs.
* Moved DropAll to db.go, so it is available to all users of Badger. Fixed a bug where we were not catching memtable flushes during DropAll.
* Make DropAll test more races. Catch another place where writes should fail due to blockedWrite.
* Self-review
**Breaking change**
- This change removes the ability to provide a callback in Commit. Instead, it creates another function called CommitWith, which takes a callback.
- Previously, a user-specified callback might or might not get executed. This PR fixes that by guaranteeing that the commit callback is always executed.
- Create a WriteBatch API to batch up multiple updates into a single txn. The txn gets Committed via callback, so the user gets a simple efficient way to do a lot of writes.
- Run only a few goroutines to execute these user callbacks. Previously, we were shooting off one goroutine per Commit callback.
* Remove the extra ManagedDB struct, which was causing maintenance headaches, due to the way Go handles polymorphism. Instead just expose the ManagedTxns option, and use it directly.
* Bring back the OpenManaged function for convenience.
This method drops all tables from the LSM tree, except one to maintain the persistence of badgerHead key. This is important to avoid value log being replayed from scratch.
It then iterates over the rest of the keys and marks them as deleted. The end result should be that all keys are considered deleted.
This method would perform a lot better than deleting every key via iteration.
NOTE to users: The timestamp used for writes must be greater than the max timestamp of writes before DropAll, to ensure that new writes are not lower than the delete markers in terms of versioning.
Add a new SetWithDiscard method, which indicates discarding of earlier versions. This information is used by compaction to discard versions. Value log GC can then discard key-value pairs which are no longer in the LSM tree (due to removal by compactions). Thus, this simple change removes the need to have PurgeBelowTs and other PurgeOlderVersions methods. Note that PurgeOlderVersions method is already made obsolete by NumVersionsToKeep option.
Only the flip side, this would make the prediction of which value log can be removed harder. Though, compaction can tackle that by updating the discard stats for value log. Maybe in another PR.
Also simplifies MergeOperator code.
- Add a GetSequence() method, which can hand out Sequence object.
- Add a Next() method on Sequence object, which is thread-safe.
- Panic for this function on ManagedDB.
- Add corresponding docs in README.
This is a bit of a cleanup of #280 (committed as
8fa1d7f577)
* Make ManagedTxns option private, so that it can be set only within the
package.
* Introduce a separate struct ManagedDB, which embeds badger.DB.
* Introduce ManagedDB.NewTransaction() and ManagedDB.NewTransactionAt()
methods.
* Introduce Txn.CommitAt, applicable only to ManageDB.
