mirror of
https://github.com/luxfi/zapdb.git
synced 2026-07-27 06:54:45 +00:00
Bill ProvinceandGitHub
9db7255529
fixes DGRAPHCORE-234 ## Problem The purpose of TestPersistLFDiscardStats is to make sure that if you make changes to the file such that we need to maintain the discardStats in the value log, and if you close the DB, then when you reopen the DB, you remember the same status. Note that the DiscardStats are updated when we perform compaction, and in our test cases we have 4 compactors concurrently running (with ids 0, 1, 2, and 3). Most of the time, we were fine when we captured a single compaction cycle -- and then closed the DB and then reopened. However, there was a race condition wherein we waited for some arbitrary amount of time, then captured the current discardStats, then closed the DB. The problem is that between the time that we capture the discardStatus and close the DB, another compaction cycle may have started! Hence, every once in a while, we would find that the saved copy of the discard stats would not match what we picked up later when we reopened the file. ## Solution As noted in the problem statement, we ended up waiting an "arbitrary amount of time" instead of waiting for specific events and then reacting to those events. Specifically, we wanted to wait until at least "some stats" had been generated, and then we waited for compaction to complete. Unfortunately, there did not exist a clear way (previously) to capture the event of "some stats having been generated", nor was there a way to capture the discardStats upon closure of the database. The solution then was to add in two things: first, a test channel in the database where we can log messages to this channel, but only when the channel has been specified. Second, we add in a means (via options.go) of specifying an "onCloseDiscardCapture" map. This map will be populated (assuming it was initialized and is not nil) when we close the db and specifically when we close the valueLog. We no longer rely on time.Sleep, but instead rely on specific events.
80 lines
3.1 KiB
Go
80 lines
3.1 KiB
Go
/*
|
|
* Copyright 2023 Dgraph Labs, Inc. and Contributors
|
|
*
|
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
* you may not use this file except in compliance with the License.
|
|
* You may obtain a copy of the License at
|
|
*
|
|
* http://www.apache.org/licenses/LICENSE-2.0
|
|
*
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
* See the License for the specific language governing permissions and
|
|
* limitations under the License.
|
|
*/
|
|
|
|
package badger
|
|
|
|
// Important: Do NOT import the "testing" package, as otherwise, that
|
|
// will pull in imports into the production class that we do not want.
|
|
|
|
// TODO: Consider using this with specific compilation tags so that it only
|
|
// shows up when performing testing (e.g., specify build tag=unit).
|
|
// We are not yet ready to do that, as it may impact customer usage as
|
|
// well as requiring us to update the CI build flags. Moreover, the
|
|
// current model does not actually incur any significant cost.
|
|
// If we do this, we will also want to introduce a parallel file that
|
|
// overrides some of these structs and functions with empty contents.
|
|
|
|
// String constants for messages to be pushed to syncChan.
|
|
const (
|
|
updateDiscardStatsMsg = "updateDiscardStats iteration done"
|
|
endVLogInitMsg = "End: vlog.init(db)"
|
|
)
|
|
|
|
// testOnlyOptions specifies an extension to the type Options that we want to
|
|
// use only in the context of testing.
|
|
type testOnlyOptions struct {
|
|
// syncChan is used to listen for specific messages related to activities
|
|
// that can occur in a DB instance. Currently, this is only used in
|
|
// testing activities.
|
|
syncChan chan string
|
|
}
|
|
|
|
// testOnlyDBExtensions specifies an extension to the type DB that we want to
|
|
// use only in the context of testing.
|
|
type testOnlyDBExtensions struct {
|
|
syncChan chan string
|
|
|
|
// onCloseDiscardCapture will be populated by a DB instance during the
|
|
// process of performing the Close operation. Currently, we only consider
|
|
// using this during testing.
|
|
onCloseDiscardCapture map[uint64]uint64
|
|
}
|
|
|
|
// logToSyncChan sends a message to the DB's syncChan. Note that we expect
|
|
// that the DB never closes this channel; the responsibility for
|
|
// allocating and closing the channel belongs to the test module.
|
|
// if db.syncChan is nil or has never been initialized, ths will be
|
|
// silently ignored.
|
|
func (db *DB) logToSyncChan(msg string) {
|
|
if db.syncChan != nil {
|
|
db.syncChan <- msg
|
|
}
|
|
}
|
|
|
|
// captureDiscardStats will copy the contents of the discardStats file
|
|
// maintained by vlog to the onCloseDiscardCapture map specified by
|
|
// db.opt. Of couse, if db.opt.onCloseDiscardCapture is nil (as expected
|
|
// for a production system as opposed to a test system), this is a no-op.
|
|
func (db *DB) captureDiscardStats() {
|
|
if db.onCloseDiscardCapture != nil {
|
|
db.vlog.discardStats.Lock()
|
|
db.vlog.discardStats.Iterate(func(id, val uint64) {
|
|
db.onCloseDiscardCapture[id] = val
|
|
})
|
|
db.vlog.discardStats.Unlock()
|
|
}
|
|
}
|