Fixes#2067
**Description**
This PR fixes an issue where Go's `append` returns `nil` when appending
an empty slice to a nil slice.
I added a check in `SafeCopy` to ensure we always return `[]byte{}` in
this case, along with a new unit test.
I also verified it using the test offered in the issue
**Checklist**
- [x] Code compiles correctly and linting passes locally
- [ ] For all _code_ changes, an entry added to the `CHANGELOG.md` file
describing and linking to
this PR
- [x] Tests added for new functionality, or regression tests for bug
fixes added as applicable
---------
Co-authored-by: Matthew McNeely <matthew.mcneely@gmail.com>
**Description**
Resolves the following error :
./test.sh: line 8: [: =: unary operator expected
**Checklist**
- [x] Code compiles correctly and linting passes locally
- [ ] For all _code_ changes, an entry added to the `CHANGELOG.md` file
describing and linking to
this PR
- [ ] Tests added for new functionality, or regression tests for bug
fixes added as applicable
- [ ] For public APIs, new features, etc., PR on [docs
repo](https://github.com/hypermodeinc/docs)
staged and linked here
Stream.Send now sends out z.Buffer instead of pb.KVList. z.Buffer marshals each KV as a separate slice. This significantly reduces the memory requirement by the Stream framework. Stream no longer uses z.Allocator or tries to put pb.KV struct on the Allocator for memory safety reasons.
Bring back the z.AllocatorPool for table.Builder.
Changes:
* Use z.Buffer for stream.Send
* Only use 8 streams in write bench
* Revert "Bug Fix: Fix up how we use z.Allocator"
This reverts commit d6defb637d.
* Bring allocator back. Use z.Buffer for send
* Add BufferToKVList function
* Print jemalloc while stream
* Bring in latest Ristretto
* Fix memory leak and benchmark read test
Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
We removed the global z.Allocator pool from z package. Instead, we now use a new z.AllocatorPool class in the places which need a pool. In this case, we brought it to TableBuilder and Stream.
Fix up a memory leak in Stream.
Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
Badger compactions are split into sub compactions if there are more than 3 tables
bottom tables are involved. The current implementation would drop keys if the top
table would overlap with a bottom table. Consider the following scenario
```
Top table => [C (deleted)]
bot table => [ A ] [ B ] [ C (valid) ] [ D ]
```
When this range is compacted, the result was `[ A B C(valid) D ] `
(only `C(deleted)` was dropped). The expected result was `[A B D]` .
This PR fixes the above-mentioned issue. The test in this PR fails on master
but works with this change.
The parallel command runs in function in a subprocess and the `packages`variable
declared at the start of the file wouldn't be visible to the subprocess unless exported.
This PR exports the `packages` variable so that it's visible to the child process as well.
Run `go list` BEFORE setting GOFLAGS="-json" so that the output is in the right
format for grep. Otherwise, this error happens:
===> Testing "github.com/dgraph-io/badger/v2/options",
go: finding module for package "github.com/dgraph-io/badger/v2/options",
can't load package: package "github.com/dgraph-io/badger/v2/options",: malformed module path "\"github.com/dgraph-io/badger/v2/options\",": invalid char '"'
Avoid Go memory spikes by using z.Allocator to allocate blocks during table generation. Also avoid having to copy data over, unless necessary -- instead write the data directly to the mmapped file.
With these changes, we're able to load 1 billion randomly generated 32-byte keys, 128-byte values in 1.5 hrs, with memory usage maxing out to 5 GB.
Co-authored-by: Manish R Jain <manish@dgraph.io>
Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
Run tests in parallel. Break up the tests into 3 functions: manual, packages and root. Run those via GNU parallel. Moved some expensive tests to manual mode.
With these changes, the badger tests can now run in under 6 minutes on a fast machine.
Co-authored-by: Manish R Jain <manish@dgraph.io>
KVs can take up a lot of memory in the stream framework. With this change, we allocate them using z.Allocator, and allow callers to KeyToList to use the allocator to generate KVs as well. After we call Send, we release them.
Also change Stream Framework to spit out StreamDone markers.
This PR significantly improves Badger's disk usage behavior.
Breaking: This PR increases the magic version from 7 to 8. So, no older Badger directories would work with this change.
With this PR, we no longer use value log as write-ahead log. Instead, each MemTable has its own WAL. Value logs now only write values which are greater than ValueThreshold, while MemTable WAL only writes smaller values and value pointers.
On a crash and restart, the MemTable WALs are replayed to apply updates to Skiplist. When MemTables are flushed to L0, the corresponding WALs are deleted.
This PR makes big changes to how value log GC works:
- Discard stats are now stored in a separate file, instead of within the LSM tree.
- GC only picks up value logs based off discard stats.
- GC no longer does sampling, it uses discard stats to inform when a value log needs to be GCed.
- Value log would now no longer grow indefinitely, because of the shift to MemTable WAL.
- Removed the `badger gc` tool.
- Value Log Head pointer tracking is removed.
- Only the last value log file is replayed on every start, and truncated as necessary.
This PR also makes a bunch of other changes:
- Removes ValueLogLoadingMode (always uses mmap now).
- Removes TableLoadingMode (always uses mmap now).
- Removes Truncate option.
- Removes KeepL0InMemory option.
Teamcity build fails intermittently because of the 10-minute timeout. The
10-minute timeout is for the complete test suite. This PR increases the
test timeout from 10 minutes to 15 minutes. We might need to increase
the timeout in the future when more tests are added.
This could possibly be a bug in `go test` command
https://github.com/golang/go/issues/36527 .
The `go test` command would skip tests in sub-packages if
the top-level package has a `custom flag` defined and the
sub-packages don't define it.
Issue https://github.com/golang/go/issues/36527#issue-548887632
has an example of this.
This PR also removes the code from the test that would unnecessary
start a web server. I see two problems here
1. An unnecessary web server running.
2. We cannot run multiple tests are the same time since the second
run of the test would try to start a web server and
crash saying `port already in use`.
This PR introduces in-memory mode in badger. The in-memory mode
can be enabled by setting options.InMemory=true. When badger is
running in in-memory mode no files are created and everything
is stored in memory.
On DB close, all stored data is lost.
NOTE - An existing DB cannot be opened in in-memory mode.
Fixes - https://github.com/dgraph-io/badger/issues/1001
This API would allow Badger to completely get rid of the given prefix, P. It does this by:
1. Flushing memtables to disk (skipping over the keys with P).
2. Doing normal compaction from L0 -> L1 (skipping keys with P).
3. Doing same level compaction on all the rest of the levels, picking tables which might have the prefix and rewriting them to new tables (again, skipping keys with P).
Before starting, all the writes are paused, the usual bits which apply to `DropAll` apply here as well.
Note: This PR has a breaking change for the Logger interface. It adds a Debugf API.
Changes:
* Working drop prefix
* Make the drop prefix more robust. Add a new Debugf API for logging.
* Add parallel tests for DropPrefix
* Avoid a test failure due to removing p dir.
* Only test DropPrefix in DropPrefix test.
* Address Martin's comments
In a specific scenario, where a FS truncates the value log entries perfectly so that a txn's FIN entry is lost, while the rest of the transaction is not; AND this is the first transaction in the system; Badger can skip truncating this transaction on the first Replay (run when Badger is Open). But, will truncate after it has a few valid transactions. The reason is that when there's only one invalid txn, the valid offset of file becomes zero. It is hard to differentiate between a valid offset of zero, v/s an uninitialized offset.
To fix this, we use the file size as a guide. If the file size is greater than the valid offset, we would truncate the file. This interferes with value log GC (and how we grow a value log file to double it's size due to how Windows treats mmapped files). So, we move the truncate logic to within replay function, and out of iterate, so it does not interfere with the functionality of value log GC.
Also, move memory expensive tests to manual mode, and the tests write to disk instead of tmpfs.
Fixes#592 .
