- Fix format string vet errors in db2_test.go and key_registry.go
for Go 1.26 stricter printf checks
- Update pb/gen.sh to add //go:build grpc tag after protoc generation
so protobuf types only compile with grpc tag (default uses ZAP
native binary encoding from types_zap.go)
Replace proto.Marshal/Unmarshal with native pb.Marshal/pb.Unmarshal to
match the wire format used by luxfi/badger/v4. This ensures databases
written by nodes using the badger fork (v1.23.4) can be read by nodes
using zapdb (v1.23.5), preventing "proto: cannot parse invalid
wire-format data" errors during rolling upgrades.
Added types_zap.go and marshal_zap.go from luxfi/badger fork, which
provide custom LittleEndian binary encoding for ManifestChangeSet,
KV, Checksum, and other internal types. The standard protobuf
generated code (badgerpb4.pb.go) is now behind a grpc build tag.
This PR adds a way to match a prefix, while ignoring certain portions of the key (aka holes).
This is useful to implement multi-tenancy in Dgraph, where namespace is stored on the byte index 3-11 in the prefix key, and those bytes need to be ignored to subscribe to updates across all the namespaces.
KVs can take up a lot of memory in the stream framework. With this change, we allocate them using z.Allocator, and allow callers to KeyToList to use the allocator to generate KVs as well. After we call Send, we release them.
Also change Stream Framework to spit out StreamDone markers.
This PR
- Uses flatbuffers instead of protobufs for table index and directly stores byte slices in the cache.
- Uses MaxVersion to pick the oldest tables for compaction first.
- Uses leveldb/bloom so that we can test it without unmarshal
- Adds uncompressed size and key count in table index.
- Updates write bench tool to use managed mode.
If a Go program imports both badger (v1) and badger/v2, a warning will
be produced at init time:
WARNING: proto: file "pb.proto" is already registered
A future release will panic on registration conflicts. See:
https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict
The problem is the "pb.proto" filename; it's registered globally, which
makes it very likely to cause conflicts with other protobuf-generated
packages in a Go binary.
Coincidentally, this is a problem with badger's pb package in the v1 module,
since that too uses the name "pb.proto". Instead, call the file
"badgerpb2.proto", which should be unique enough, and it's also the name
we use for the Protobuf package.
Finally, update gen.sh to work out of the box via just "bash gen.sh"
without needing extra tweaks, thanks to the "paths=source_relative"
option. It forces output files to be produced next to the input files,
which is what we want.
This PR renames badger protobuf package from `dgraph.badger.v2.pb`
to `badgerpb2`.
The `pb.pb.go` file has been regenerated using the `pb/gen.sh` script.
There were two instances of init-time work being incompatible with v1.
That is, if one imported both v1 and v2 as part of a Go build, the
resulting binary would end up panicking before main could run. The
examples below are done with the latest versions of v1 and v2, and a
main.go as follows:
package main
import (
_ "github.com/dgraph-io/badger"
_ "github.com/dgraph-io/badger/v2"
)
func main() {}
First, the protobuf package used "pb" as its proto package name. This is
a problem, because types are registered globally with their fully
qualified names, like "pb.Foo". Since both badger/pb and badger/v2/pb
tried to globally register types with the same qualified names, we'd get
a panic:
$ go run .
panic: proto: duplicate enum registered: pb.ManifestChange_Operation
goroutine 1 [running]:
github.com/golang/protobuf/proto.RegisterEnum(...)
.../go/pkg/mod/github.com/golang/protobuf@v1.3.1/proto/properties.go:459
github.com/dgraph-io/badger/v2/pb.init.0()
.../badger/pb/pb.pb.go:638 +0x459
To fix this, make v2's proto package fully qualified. Since the
namespace is global, just "v2.pb" wouldn't suffice; it's not unique
enough. "dgraph.badger.v2.pb" seems good, since it follows the Go module
path pretty closely.
The second issue was with expvar, which too uses globally registered
names:
$ go run .
2020/04/08 22:59:20 Reuse of exported var name: badger_disk_reads_total
panic: Reuse of exported var name: badger_disk_reads_total
goroutine 1 [running]:
log.Panicln(0xc00010de48, 0x2, 0x2)
.../src/log/log.go:365 +0xac
expvar.Publish(0x906fcc, 0x17, 0x9946a0, 0xc0000b0318)
.../src/expvar/expvar.go:278 +0x267
expvar.NewInt(...)
.../src/expvar/expvar.go:298
github.com/dgraph-io/badger/v2/y.init.1()
.../badger/y/metrics.go:55 +0x65
exit status 2
This time, replacing the "badger_" var prefix with "badger_v2_" seems
like it's simple enough as a fix.
Fixes#1208.
This PR stores the total key-value size in a table in the table
footer. The key-value size can be accessed via db.Tables(..) call.
It returns the list of all tables along with the total size of key-values.
The proto files in the current master are not compatible with the ones in dgraph.
See build for PR https://github.com/dgraph-io/dgraph/pull/4200
```
[Step 3/5] # github.com/dgraph-io/dgraph/protos/pb
[11:02:11] [Step 3/5] ../protos/pb/pb.pb.go:7146:29: m.Kv[iNdEx].MarshalToSizedBuffer undefined (type *pb.KV has no field or method MarshalToSizedBuffer)
[11:02:11] [Step 3/5] ../protos/pb/pb.pb.go:7209:29: m.Kv[iNdEx].MarshalToSizedBuffer undefined (type *pb.KV has no field or method MarshalToSizedBuffer)
[11:02:40] [Step 3/5] Makefile:66: recipe for target 'install' failed
[11:02:40] [Step 3/5] make[1]: *** [install] Error 2
[11:02:40] [Step 3/5] make[1]: Leaving directory '/home/pawan0201/go/src/github.com/dgraph-io/dgraph/dgraph'
[11:02:40] [Step 3/5] make: *** [install] Error 2
[11:02:40] [Step 3/5] Makefile:43: recipe for target 'install' failed
[11:02:40] [Step 3/5] Process exited with code 2
```
https://teamcity.dgraph.io/viewLog.html?buildId=29232&buildTypeId=Dgraph_Blockade&tab=buildLog
This PR updates the proto files in badger. I've run `./gen.sh` script to regenerate these.
This PR introduces, a way to tell StreamWriter to close a stream. Previously Streams were always open until Flush is called on StreamWriter. This resulted in memory utilisation, because of underlying TableBuilder to a sortedWriter. Also closing all sorted writer in single call resulted in
more memory allocation(during Flush()). This can be useful in some case such as bulk loader in
Dgraph, where only one stream is active at a time.
This commit adds support for compression in badger. Two compression
algorithms - Snappy and ZSTD are supported for now. The compression algorithm information
is stored in the manifest file. We compress blocks (typically of size 4KB) stored in the SST.
The compression algorithm can be specified via the CompressionType option to badger.
This PR will add support for encrypting data which going to be on disk.
Two components are being encrypted, one is sst and another one is
vlog. In sst, each block is encrypted with seperate IV using AES CTR
mode. In vlog, each entry is being encrypted. Each vlog will have base
IV of 12 bytes. IV for the each entry is generated by merging base IV
with entry offset. Data are encrypted using datakey, which is generated by
the badger. The datakey is further encrypted using user provided key
and stored in disk. So that user can change key. In order to change
key user has to provide old key and new key. we'll decrypt using old
key and store the datakey back to disk by encrypting using the new
key. By this mechanism, it'll simplfy the key change.
* Add key-offset index to the end of SST table
This commit adds the key-offset index used for searching the blocks in
a table to the end the SST file. The length of the index is stored in
the last 4 bytes of the file.
Changes:
* Start work on a sorted stream writer which can avoid paying the cost of compactions.
* Logic for sorted stream writer is all written.
* Add a stream id to each key in the output from Stream framework.
* Make StreamWriter work. Performance is incredible, able to write at 100MBps.
* Add Tests for Stream Writer
* Update Oracle after Stream Writer is done
* Mods in stream framework so we can decrease the number of key ranges
* Add awareness of stream id in sortedWriter.
* Sync the directories when StreamWriter is done.
* Add licenses
* Moving builder.Finish within the goroutine improves throughput by 15%. Getting 112MBps write speed.
* Change Stream Writer Tests to have managed modes also
* Rename files to stream_writer.go and the corresponding test file.
* Fix a bug caused by marking an index done below current done until, causing a panic. Now we just recreate the oracle.
* Update protos
Add SHA256 checksums for SSTables in MANIFEST. If a table no longer matches this checksum, that table would be skipped over with an error.
Tested that it works with previous badger directories. As new tables get created, Badger would store their checksums in MANIFEST.
Modified `badger info` to show the checksums stored in MANIFEST, so user can manually compare the output from `sha256sum <filename>` if needed.
Fixes#680 .
- Flatten function can now be called from live code path, i.e. while Badger is running.
- Refactor stop and start compactions code, so it can be used in Flatten, DropAll, and other places.
- Rename protos package to pb. Consolidate both the proto files into one pb.proto.
- Rename KeyToKVList to just KeyToList in Stream.
- Rename KVPair to just KV.
Changes:
* Rename protos to pb.
* Rename KeyToKVList to KeyToList.
* Refactor stop and start compactions, so we can use it in various places.
* Rename KVPair to KV
* Defer startCompactions right next to stopCompactions.
* Catch all usages of KVPair. Rename to KV.
zeekay