mirror of https://github.com/luxfi/zapdb.git synced 2026-07-27 06:54:45 +00:00

Files

T

History

Aman MangalandAman Mangal 0785f5219f opt(stream): add option to directly copy over tables from lower levels (#1700 )

Also takes a bug fix from PR #1712, commit 5a8d344

This PR adds FullCopy option in Stream. This allows sending the
table entirely to the writer. If this option is set to true we
directly copy over the tables from the last 2 levels. This option
increases the stream speed while also lowering the memory
consumption on the DB that is streaming the KVs.

For 71GB, compressed and encrypted DB we observed 3x improvement
in speed. The DB contained ~65GB in the last 2 levels while
remaining in the above levels.

To use this option, the following options should be set in Stream.

stream.KeyToList = nil
stream.ChooseKey = nil
stream.SinceTs = 0
db.managedTxns = true

If we use stream writer for receiving the KVs, the encryption mode
has to be the same in sender and receiver. This will restrict
db.StreamDB() to use the same encryption mode in both input and
output DB. Added TODO for allowing different encryption modes.

2025-06-27 19:03:42 +05:30

builder_test.go

chore: align repo conventions (#2158 )

2025-02-05 17:00:32 -05:00

builder.go

chore(deps): remove dependency on github.com/pkg/errors (#2184 )

2025-04-07 22:02:35 +05:30

iterator.go

chore: align repo conventions (#2158 )

2025-02-05 17:00:32 -05:00

merge_iterator_test.go

chore: align repo conventions (#2158 )

2025-02-05 17:00:32 -05:00

merge_iterator.go

chore: align repo conventions (#2158 )

2025-02-05 17:00:32 -05:00

README.md

chore: align repo conventions (#2158 )

2025-02-05 17:00:32 -05:00

table_test.go

chore: align repo conventions (#2158 )

2025-02-05 17:00:32 -05:00

table.go

opt(stream): add option to directly copy over tables from lower levels (#1700 )

2025-06-27 19:03:42 +05:30

README.md

Size of table is 123,217,667 bytes for all benchmarks.

BenchmarkRead

$ go test -bench ^BenchmarkRead$ -run ^$ -count 3
goos: linux
goarch: amd64
pkg: github.com/dgraph-io/badger/table
BenchmarkRead-16              10       154074944 ns/op
BenchmarkRead-16              10       154340411 ns/op
BenchmarkRead-16              10       151914489 ns/op
PASS
ok       github.com/dgraph-io/badger/table       22.467s

Size of table is 123,217,667 bytes, which is ~118MB.

The rate is ~762MB/s using LoadToRAM (when table is in RAM).

To read a 64MB table, this would take ~0.084s, which is negligible.

BenchmarkReadAndBuild

$ go test -bench BenchmarkReadAndBuild -run ^$ -count 3
goos: linux
goarch: amd64
pkg: github.com/dgraph-io/badger/table
BenchmarkReadAndBuild-16              1       1026755231 ns/op
BenchmarkReadAndBuild-16              1       1009543316 ns/op
BenchmarkReadAndBuild-16              1       1039920546 ns/op
PASS
ok       github.com/dgraph-io/badger/table       12.081s

The rate is ~123MB/s. To build a 64MB table, this would take ~0.56s. Note that this does NOT include the flushing of the table to disk. All we are doing above is reading one table (which is in RAM) and write one table in memory.

The table building takes 0.56-0.084s ~ 0.4823s.

BenchmarkReadMerged

Below, we merge 5 tables. The total size remains unchanged at ~122M.

$ go test -bench ReadMerged -run ^$ -count 3
goos: linux
goarch: amd64
pkg: github.com/dgraph-io/badger/table
BenchmarkReadMerged-16              2       977588975 ns/op
BenchmarkReadMerged-16              2       982140738 ns/op
BenchmarkReadMerged-16              2       962046017 ns/op
PASS
ok       github.com/dgraph-io/badger/table       27.433s

The rate is ~120MB/s. To read a 64MB table using merge iterator, this would take ~0.53s.

BenchmarkRandomRead

go test -bench BenchmarkRandomRead$ -run ^$ -count 3
goos: linux
goarch: amd64
pkg: github.com/dgraph-io/badger/table
BenchmarkRandomRead-16              500000       2645 ns/op
BenchmarkRandomRead-16              500000       2648 ns/op
BenchmarkRandomRead-16              500000       2614 ns/op
PASS
ok       github.com/dgraph-io/badger/table       50.850s

For random read benchmarking, we are randomly reading a key and verifying its value.

DB Open benchmark

Create badger DB with 2 billion key-value pairs (about 380GB of data)
```
badger fill -m 2000 --dir="/tmp/data" --sorted
```

Clear buffers and swap memory

free -mh && sync && echo 3 | sudo tee /proc/sys/vm/drop_caches && sudo swapoff -a && sudo swapon -a && free -mh

Also flush disk buffers

blockdev --flushbufs /dev/nvme0n1p4

Run the benchmark

go test -run=^$ github.com/dgraph-io/badger -bench ^BenchmarkDBOpen$ -benchdir="/tmp/data" -v

badger 2019/06/04 17:15:56 INFO: 126 tables out of 1028 opened in 3.017s
badger 2019/06/04 17:15:59 INFO: 257 tables out of 1028 opened in 6.014s
badger 2019/06/04 17:16:02 INFO: 387 tables out of 1028 opened in 9.017s
badger 2019/06/04 17:16:05 INFO: 516 tables out of 1028 opened in 12.025s
badger 2019/06/04 17:16:08 INFO: 645 tables out of 1028 opened in 15.013s
badger 2019/06/04 17:16:11 INFO: 775 tables out of 1028 opened in 18.008s
badger 2019/06/04 17:16:14 INFO: 906 tables out of 1028 opened in 21.003s
badger 2019/06/04 17:16:17 INFO: All 1028 tables opened in 23.851s
badger 2019/06/04 17:16:17 INFO: Replaying file id: 1998 at offset: 332000
badger 2019/06/04 17:16:17 INFO: Replay took: 9.81µs
goos: linux
goarch: amd64
pkg: github.com/dgraph-io/badger
BenchmarkDBOpen-16              1       23930082140 ns/op
PASS
ok       github.com/dgraph-io/badger       24.076s

It takes about 23.851s to open a DB with 2 billion sorted key-value entries.