Fixes https://github.com/dgraph-io/badger/issues/1962
We have assumed that index won't be zero for a WaterMark but in badger's
unmanaged mode we start transactions with readTs = 0. This affects
oracle.readMark that could have values starting at 0.
This commit fails incremental stream writer if memtable has data. It
also allows calling incremental stream writer more times than the number
of levels we have. This is done by calling Flatten when we have reached
L0 and incremental stream writer can be used again after Flatten.
Vlog needs to stay open while the compaction is being run. With the CompactL0OnClose option, it becomes necessary to close the vlog after all the compactions have been completed.
(cherry picked from commit 921a6e2a86)
z.Allocator imposes an upper bound of 1GB on the size of the allocator. In builder, we double the allocation while reallocating to amortize the cost over the future allocations.
Consider a scenario where the size of KV is 700MB. The size of the allocator would be 700MB. Now we want to finish the block, so we will need some bytes for the metadata. That can easily fit into the remaining 324MB. But we were earlier doing over-allocation that causes panic.
(cherry picked from commit 3c1b8bcde2)
With the introduction of SinceTs, a bug was introduced #1653 that skips the pending entries.
The default value of SinceTs is zero. And for the transaction made at readTs 0, the pending entries have version set to 0. So they were also getting skipped.
(cherry picked from commit 50276c44e2)
We provide few special options along with badger options.
// Special Handling: compression, cache-mb, cache-percentage sub-flags.
// Valid values for special flags:
// compression: valid options are {none,snappy,zstd:<level>}
// cache-mb: uint
// cache-percentage: 2 comma-separated values summing upto 100. Format: blockCache,indexCache
// goroutines: alias for numgoroutines.
// Example: compression=zstd:3; cache-mb=2048; cache-percentage=70,30; goroutines=8;
Adds functionality to ban a namespace so that no read/write is allowed to a key belonging
to any of the banned namespaces:
- Add BanNamespcae() and BannedNamespace() API.
- Set(), Get() operations on the keys belonging to any of these banned namespaces is denied.
- The iterator will skip these namespaces.
- The keys for these banned namespaces will be dropped at the time of compaction as the
iterator will not even pick them.
For details: check PR #1638
When you're scanning a lot, but sending little data, the stream framework seems to work really slowly. But, that's deceiving, because it is scanning really fast.
We calculate the rough uncompressed size of the data the stream framework will send. We also periodically log the amount of data scanned by produceKVs.
This PR adds debug information for the crash in `initIndex`. We suspect that it may be due to unsynced mmap (#1625), or because of mishandling of SST files.
SST files were not sync'ed to disk by default. Machine crash may happen before the explicit sync (which happens at DB close), leading to an inconsistent state and badger not able to open. It is better to explicitly do a sync.
There may be a case when the table contains [0x0000,...., 0xffff]. If we are searching for k=0x0011, we should not directly infer that k is present. It may not be present. This leads to the triggering of compaction at the time of DropPrefix which is quite expensive.
This PR fixes the issue by confirming that case by searching inside the table for the prefix.
We currently do not check if there exist some data to drop for prefixes in DropPrefix(). This can make drop prefix to be expensive because it has to do a bunch of compactions, etc.
This PR optimizes that behavior by not doing any of that in case no data exist for prefixes.
If a table has a mixture of value log pointers and embedded values,
badger will carry over the last length from a value log entry into
the subsequent embedded entries.
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
