Compare commits

..
188 Commits
Author SHA1 Message Date
547400a923 Add release notes entry for Valkey 9.1.0 GA (#3682)
Add GA release note entry
Update release version

---------

Signed-off-by: Lucas Yang <lucasyonge@gmail.com>
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
2026-05-19 11:03:27 -04:00
Roshan KhatriandMadelyn Olson 9f4e9924fa Update the datasize to match the automated perfs (#3753)
Update the data-size to match the automated perf benchmarking as 96 is
also embedded so moving to 128 to test for non-embedded.

I have also update the wfs to use the config files from the
`valkey-perf-benchmark` so we don't need to track the configs in 2
places, everything will be pulled from source `valkey-perf-benchmark`
and the configs will be standard throughout all runs

---------

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2026-05-18 20:51:48 -07:00
93af567b54 Fix deferred freeClient clobbering replication state after replicaof (#3719)
## Summary

This PR addresses 2 race conditions where deferred `freeClient`
(introduced by #3324) clobbers replication state set by `REPLICAOF`
commands.

Found while triaging the recurring `test-ubuntu-tls-io-threads` daily CI
failure in `tests/unit/wait.tcl`.

## Root Cause

Since PR #3324 ("Redesign IO threading communication model"),
`freeClient()` on a primary client with pending IO is deferred via
`freeClientAsync` (gated on `clientHasPendingIO`). When the deferred
free eventually executes, it chains through `replicationCachePrimary()`
-> `replicationHandlePrimaryDisconnection()`, which unconditionally sets
`server.repl_state = REPL_STATE_CONNECT`.

This causes two bugs:

### Bug 1: REPLICAOF NO ONE (SIGSEGV)

`replicationUnsetPrimary()` sets `primary_host = NULL` before calling
`freeClient`. The deferred free runs later, sets `repl_state =
REPL_STATE_CONNECT` while `primary_host` is still NULL.
`replicationCron` then calls `connectWithPrimary()` which passes NULL to
`connTLSConnect()` -> `inet_pton(AF_INET, NULL, ...)` -> SIGSEGV.

### Bug 2: REPLICAOF newhost newport (connection leak)

`replicationSetPrimary()` calls `freeClient(old_primary)` (deferred),
then sets `primary_host` to the new IP and progresses `repl_state` to
`REPL_STATE_CONNECTING` with a new connection handle in
`server.repl_transfer_s`. The deferred free runs later, clobbers
`repl_state` back to `REPL_STATE_CONNECT`. `replicationCron` then calls
`connectWithPrimary()` again, overwriting `server.repl_transfer_s`
without closing the previous connection -- an FD leak.

## Fix

Make `replicationHandlePrimaryDisconnection()` only transition to
`REPL_STATE_CONNECT` when `repl_state` is still `REPL_STATE_CONNECTED`
and `primary_host` is set. This means the disconnection is genuine and
no other state transition has already occurred. If `repl_state` has
already moved on (CONNECT, CONNECTING, NONE, etc.), the deferred free is
stale and the function leaves the state untouched.

Additionally:
- `connTLSConnect()`: Return `C_ERR` if `addr` is NULL (defense in
depth).
- `tests/unit/wait.tcl`: Add 10s timeout to the blocking WAITAOF test,
and add dedicated tests for the repoint scenario.

## Reproduction

Reproduced locally by establishing TLS replication and executing
`REPLICAOF NO ONE`:
- Without fix: server crashes with signal 11, accessing address 0x0
- With fix: server continues operating normally

## Testing

- Full `unit/wait` test suite passes (51/51) with IO threads enabled
- New tests "Repoint replica between primaries does not leak connections
or crash" and "Rapid repoint does not crash or leak" pass
- Crash reproduced locally over TLS (SIGSEGV without fix, graceful
handling with fix)

---------

Signed-off-by: Yaron Sananes <yaron.sananes@gmail.com>
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2026-05-18 20:51:48 -07:00
9818c15d06 Restore ERR prefix and error-code preservation in Lua scripting (#3678)
Closes #3663.

## Why

Three callers in `src/modules/lua/script_lua.c` lost error-code handling
during the #2858 move of the Lua scripting engine into a module. On the
wire, `redis.pcall()` (no args), `redis.setresp(4)`, `pcall(redis.log,
1)`, `pcall(redis.log, 10, 'msg')`, and `pcall(redis.set_repl)` now
return the bare message instead of `ERR <message>`. OOM and WRONGTYPE
happened to keep working because their code letters survived elsewhere
by accident.

Reproducer:

```
$ ./src/valkey-cli eval "return redis.pcall()" 0
"Please specify at least one argument for this redis lib call"   # before this PR
"ERR Please specify at least one argument for this redis lib call"  # with this PR
```

## What changed

In `src/modules/lua/script_lua.c`:

1. `luaPushErrorBuff` — bare-message branch (case 2): prepend `ERR `
unless the message already starts with a code letter (defensive — five
existing callers hardcode `"ERR ..."` strings; double-prefixing them
would also be wrong).
2. `luaProcessReplyError` push_error branch: re-add the leading `-`
before delegating to `luaPushError` so the case-1 code-extraction path
handles `OOM`, `WRONGTYPE`, `READONLY`, etc.
3. `errorCallback` errno==0 branch: same re-add-`-` approach.

41 LoC across the three functions.

## Test

`tests/unit/scripting.tcl` regression at line 1028 covers:

- All five direct-API errors above — asserts `ERR ` prefix
- WRONGTYPE error path — asserts the code survives and is not
double-prefixed
- Runs 4 times under the `foreach is_eval × foreach
script_compatibility_api` matrix

Fails on `unstable`, passes with this PR.

---------

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-05-18 20:51:48 -07:00
BinbinandMadelyn Olson 3c9a49f51f Use full hash-seed bytes when deriving SipHash seed (#3654)
The hash-seed config is an sds string and may contain embedded NUL
bytes (sdssplitargs preserves \xNN escapes inside double quotes).

In the old code getHashSeedFromString() used strlen() on it, so two
hash-seed values differing only after a \x00 byte collapsed to the
same SipHash seed, can break it.

hash-seed was added in #2608.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-05-18 20:51:48 -07:00
d10f076f50 Fix invalid memory access in RESTORE with malformed zipmap (CVE-2026-25243) (#3619)
Root cause: zipmapValidateIntegrity() and zipmapNext() use different
methods to calculate pointer advancement for length-encoded fields.
Validation reads the actual encoded size via
zipmapGetEncodedLengthSize() (which returns 5 for the 0xFE prefix), but
zipmapRawKeyLength() (used by zipmapNext during hash conversion)
recalculates via zipmapEncodeLength() which returns 1 for decoded
lengths < 254. A crafted zipmap with an overlong 5-byte encoding for a
small length passes validation but causes a 4-byte pointer mismatch in
zipmapNext(), leading to heap buffer over-reads during the
zipmap-to-listpack conversion.

Fix: add sanity checks in zipmapValidateIntegrity() to reject entries
where the decoded length < ZIPMAP_BIGLEN (254) but the encoding uses
more than 1 byte. This is applied to both field-name and value lengths.

Test: added a regression test in tests/unit/dump.tcl that crafts a
RESTORE payload with a 2-entry zipmap where the first field uses an
overlong 5-byte length encoding for value 3. Post-patch, this is cleanly
rejected by zipmapValidateIntegrity(). Pre-patch, the misaligned
zipmapNext() reads garbage (confirmed via server log: "Hash zipmap with
dup elements, or big length (0)") which also produces an error, so the
test serves as a defense-in-depth regression anchor rather than a strict
pass/fail differentiator. The actual heap over-read is detectable with
AddressSanitizer builds.

Signed-off-by: ikolomi <ikolomin@amazon.com>
Co-authored-by: ikolomi <ikolomin@amazon.com>
2026-05-18 20:51:48 -07:00
RaghavandMadelyn Olson bb0dcc9028 CLUSTERSCAN MATCH pattern maps to a specific slot optimizations (#3380)
When a MATCH pattern maps to a specific slot, `CLUSTERSCAN` can skip
directly to that slot
instead of walking through all slots one by one.

- On `cursor 0`, starts directly at the matching slot
- If cursor is behind the matching slot, jumps forward
- If cursor is ahead of the matching slot, we conclude the scan as we
cannot match keys
- If both SLOT and MATCH are provided but target different slots,
returns 0 immediately

Signed-off-by: nmvk <r@nmvk.com>
2026-05-18 20:51:48 -07:00
Madelyn OlsonandGitHub 6671099036 Revert libvalkey to 0.4.x to fix sentinel crash (#3752)
Reverts the libvalkey portion of #3697 (0.5.0 update).

The 0.5.0 update introduced a `DICT_INCLUDE_DIR` feature whose
implementation is broken: `#include "dict.h"` in `async.c` and
`cluster.c` resolves to libvalkey's local `dict.h` (5-field `dictType`)
instead of valkey's (9-field `dictType`), causing a struct layout
mismatch with the linked `dictCreate`. This crashes every sentinel
instance on startup.

On unstable this works by coincidence because `dict` is typedef'd to
`hashtable` and the memory layout happens to have zeros at the critical
offset. On 9.1 with the traditional dict implementation, it reads
garbage and crashes.

The fix should be done upstream in libvalkey (the `DICT_INCLUDE_DIR`
feature needs to use `<dict.h>` instead of `"dict.h"`). Once that's
fixed upstream, we can re-update.

**Crash signature:**
```
signal 11, accessing address 0x90
sentinelReconnectInstance -> valkeyAsyncConnectBind -> dictCreate
```

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2026-05-18 14:24:02 -07:00
Björn SvenssonandLucas Yang 122873973b Update deps/libvalkey to version 0.5.0 (#3697)
Squashed 'deps/libvalkey/' changes from 45c2ed15c..cb5ff91aa

cb5ff91aa Release 0.5.0
cd3d5d8ff Fix potential heap-buffer-overflow in cluster error reply parsing (#305)
32c7c5d09 Refactor error handling to use valkeySetErrorFromErrno and valkeyClearError (#303)
e4b548b32 Remove support for external dict while still supporting external sds (#302)
61b27c453 Try all addresses from DNS before failing to connect (#300)
8aff94fab RDMA: Fix lost EPOLLIN/POLLIN events
b9285b5fc Optimize read buffer compaction and reduce copying (#294)
0fa809877 Additional overflow protection for MAP/ATTR
b26c56e87 Fix gcc warnings (#287)

git-subtree-dir: deps/libvalkey
git-subtree-split: cb5ff91aa5816807f0a549bd2b36e611253dfeb6

Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
2026-05-14 13:35:04 +00:00
66b8910e18 Add check to disallow CLUSTER SETSLOT to target self (#3689)
Currently, the `CLUSTER SETSLOT MIGRATING/IMPORTING` commands
only check the ownership of the slot. So if the user mistyped node IDs while
writing these commands, it will be pretty confusing to debug, since the
`valkey-cli --cluster check` command will still report the correct status.

This PR adds a simple check to verify if the node ID in the command
points to itself, reducing confusion when doing manual hash slot
migration.

For IMPORTING, myself is the target node, so the "node" must be the
source node (not myself).

For MIGRATING, myself is the source node, so the "node" must be the
target node (not myself).

Signed-off-by: hieu2102 <hieund2102@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2026-05-14 13:34:24 +00:00
BinbinandLucas Yang bf18c824bb Improve CLUSTERSCAN error handling test with broader coverage (#3674)
Rewrite the CLUSTERDOWN test to use a 3-node cluster and cover four
distinct degraded-cluster scenarios:
1. Node down with full-coverage enabled -> CLUSTERDOWN on all slots
2. Node down with full-coverage disabled -> MOVED for unreachable
   slots, local slots handled normally
3. Slot unassigned with full-coverage enabled -> "Hash slot not served"
   for the unassigned slot, CLUSTERDOWN for others
4. Slot unassigned with full-coverage disabled -> "Hash slot not served"
   for the unassigned slot, MOVED for remote slots

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-05-14 13:33:44 +00:00
BinbinandLucas Yang 53f416d629 Minor cleanup and fix typo in bzmpop get_keys_function (#3667)
This should be bzmpopGetKeys, however, it is harmless because
blmpopGetKeys and bzmpopGetKeys are implemented exactly the same.
This typo was introduced in the very beginning.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-05-14 13:32:59 +00:00
Sarthak AggarwalandLucas Yang 0d57c6c3aa Fix weekly workflow startup_failure caused by permissions mismatch (#3684)
The weekly workflow has been broken since May 10.

#3358 added a `consolidate-test-failures` job to `daily.yml` that
needs `actions: write` to delete per-job artifacts. `weekly.yml` calls
`daily.yml` as a reusable workflow but only grants `actions: read`

Verified on my fork:
`determine-release-branches` ran, the nested `daily.yml` matrix
expanded, and the child jobs were started. Cancelled after that.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-05-14 13:32:19 +00:00
334f7a87b0 IO-Threads redesign cleanup work (#3544)
Follow-ups to the IO-threads redesign (#3324 / #3367). Bundles several
independent fixes and refactors:

- tryOffloadFreeObjToIOThreads: only offload the free of the sds buffer
(which the I/O thread allocated). The robj itself is main-thread-owned
and is now freed on the main thread.
- trySendWriteToIOThreads: defer clearing last_header until after a
successful enqueue, so the main thread can't mutate a header the I/O
thread is concurrently reading.
- evictClients: drop the pending_freed / close_asap bookkeeping and rely
on freeClient's return value to bump stat_evictedclients. The old
accounting double-counted protected clients and could under-evict.
- Make MPSC/SPMC/SPSC queue sizes a runtime parameter on *Init(q, size)
instead of compile-time *_QUEUE_SIZE macros in queues.h. The I/O-threads
sizes (16384 / 4096 / 4096) now live as constants in io_threads.c; tests
pass their own sizes.
- updateIOThreads uses io_shared_outbox.queue_size instead of the macro.
- Add function-level doc comments to queues.h.
- Rename IOThreadFreeArgv/IOThreadPoll → ioThreadFreeArgv/ioThreadPoll
for consistency.
- Remove unused job_handler typedef; fix static_assert message (7 → 8).
clang-format pass; Makefile tab→spaces nit.
- Replace the getrusage(RUSAGE_THREAD) sys/user CPU sampling (and its
#ifdef RUSAGE_THREAD fallback) with server.stat_active_time. One
portable signal, one threshold (IO_IGNITION_MAIN_THREAD_ACTIVE_PERCENT =
30) replaces the dual sys>30% / sys>5%+user>50% rules.
- Renames STATS_METRIC_MAIN_THREAD_CPU_{SYS,USER} →
STATS_METRIC_MAIN_THREAD_ACTIVE_TIME.

---------

Signed-off-by: akash kumar <akumdev@amazon.com>
Signed-off-by: Akash Kumar <45854686+akashkgit@users.noreply.github.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2026-05-12 17:38:22 +00:00
Hanxi ZhangandLucas Yang 6ed880d592 Automatically create github issues for test failures from daily CI runs (#3358)
Continuation of #3315 (accidentally closed)
Part of #2670

## Summary
Automatically detect test failures from daily CI runs and create/update
GitHub issues.

## What it does
- After each daily CI run, detects test failures from all test
environments
- Creates a new GitHub issue if the failure is not already reported
- Comments on existing issues if the failure is already reported
- Local usage: Developers can generate a JSON report of test failures
locally by passing --failures-output:
example:
```./runtest --single unit/auth --failures-output results.json --verbose```
Without the flag, no file is created.

## Changes
- `tests/test_helper.tcl` — add `--failures-output` flag to write valkey/moduleapi failures to a specified JSON file, filter TIMEOUT/Sanitizer/Valgrind/Can't start/check for memory leaks
- `tests/instances.tcl` — add failure tracking and `--failures-output` support for sentinel/cluster tests
- `.github/workflows/daily.yml` — pass `--failures-output` to all test commands, one artifact upload per job, consolidation job to merge all artifacts
- `.github/workflows/test-failure-detector.yml` — new workflow triggered on Daily completion to create/update GitHub issues
- `.github/actions/upload-test-failures/action.yml` — reusable composite action for uploading test failure artifacts

## Testing
Ran multiple daily workflow dispatches with dummy tests and verified:
- Failure JSON files created correctly for valkey, moduleapi, sentinel, cluster
- Artifacts uploaded and consolidated into single report
- Issues created and commented on for repeated failures: 
- - (valkey)https://github.com/hanxizh9910/valkey/issues/158
- - (moduleapi)https://github.com/hanxizh9910/valkey/issues/76
- - (cluster)https://github.com/hanxizh9910/valkey/issues/157
- - (sentinel)https://github.com/hanxizh9910/valkey/issues/156

Note: Previous test issues have been closed. Here's what it looks like when failures are detected (the sentinel and cluster dummy test failures are intentional):

<img width="1559" height="515" alt="Multiple test failure issues created automatically" src="https://github.com/user-attachments/assets/ce4e1ffa-83f2-44dd-a6e2-13b07f0507a5" />

- Example of running daily: https://github.com/hanxizh9910/valkey/actions/runs/23165826266
Result: https://github.com/hanxizh9910/valkey/issues:
<img width="1447" height="349" alt="Screenshot 2026-03-17 at 11 45 36 AM" src="https://github.com/user-attachments/assets/f8b18fb8-5541-4421-b30e-f14e16e82ce7" />

---------

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
2026-05-12 17:36:54 +00:00
Rain ValentineandLucas Yang ed7f7c8c44 Big Endian: add daily workflow UT job and fix UTs (#3330)
Big endian support on Valkey is "best effort" and not guaranteed, but we
haven't been doing any regular testing at all afaik. This PR adds a job
to the daily workflow to run UTs on an emulated big endian platform.
Integration tests failed excessively because of how slow emulation is.

I fixed several problems with tests and improved UT coverage of key
points where endian byte order matters - and fwiw I didn't find any
bugs. I think the main coverage gap remaining after this is RDB
serialization (maybe little endian <-> big endian round trips?)

There are couple lines of endian-specific code for #3166 and this change
can test it.

Signed-off-by: Rain Valentine <rsg000@gmail.com>
2026-05-12 17:36:12 +00:00
Ran ShidlansikandGitHub 78c2b3d3ed fix(cluster): Remove per-call srand in valkey-cli (9.1 backport) (#3668)
Backport of #3586 to the 9.1 branch.

## Problem

`clusterManagerNodePrimaryRandom()` called `srand(time(NULL))` on every
invocation, then immediately `rand() % primary_count`. When called in a
tight loop for uncovered slots, all calls within the same wall-clock
second produce the identical seed, causing every uncovered slot to be
assigned to the same primary node.

The same issue existed in `clusterManagerOptimizeAntiAffinity`,
`pipeMode`, and `LRUTestMode`.

## Solution

Remove all per-call `srand()` invocations and consolidate into a single
`srand(time(NULL) ^ getpid())` at the start of `main()`. This allows
`rand()` to advance its state across calls, distributing uncovered slots
randomly across available primaries.

(cherry picked from commit 6f6cf5a612)

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2026-05-12 20:01:55 +03:00
Rain ValentineandLucas Yang 944495ca05 extra UT
Signed-off-by: Rain Valentine <rsg000@gmail.com>
2026-05-12 15:34:58 +00:00
Rain ValentineandLucas Yang 1cb3335fbb hashtable iterator safety: invalidate on exhaustion
Signed-off-by: Rain Valentine <rsg000@gmail.com>
2026-05-12 15:34:28 +00:00
Ping XieandLucas Yang e75edefdda Fix verify-provenance action pin (#3594) 2026-05-12 15:33:08 +00:00
Ping XieandLucas Yang cda65eedee Update provenance action to refine layer2 exemption policies (#3593) 2026-05-12 15:32:38 +00:00
Ping XieandLucas Yang 29791ce9a7 Implement Provenance Guard (#3109)
This PR bootstraps Valkey's provenance guard integration.

The provenance guard is a content-based similarity detection system that helps maintain proper code provenance by comparing incoming PR changes against fingerprint databases built from Redis commits and PRs. The matching logic now lives in the external `valkey-io/verify-provenance` action repository; this PR wires Valkey to that action and seeds the required database branch.

Key features:
  * Content-based detection: Uses normalized diff fingerprints and fuzzy matching to detect similar changes, including cases where files have moved or been refactored.
  * Externalized action logic: The check and refresh implementation is maintained in `valkey-io/verify-provenance` and is pinned by exact commit SHA from Valkey workflows.
  * Provenance Guard workflow: Runs on PR activity to check incoming changes against the provenance databases and report potential matches.
  * Daily Refresh workflow: Runs daily to refresh PR fingerprints and commits updated data back to `verify-provenance-db`.
  * Dedicated DB branch: Stores provenance databases on the orphan `verify-provenance-db` branch, separate from Valkey source code.
  * Privacy-first storage: Stores compressed non-reversible fingerprints, not source code.

The initial `verify-provenance-db` branch has been bootstrapped with fingerprints of Redis commits and PRs.

  ---------

  Signed-off-by: Ping Xie <pingxie@outlook.com>
2026-05-12 15:32:03 +00:00
BinbinandLucas Yang 55a1003476 Use wait_for_condition instead of hardcoding in dump.tcl test (#3647)
This after causes the test to take 15 seconds, but it might only
require just over 10 seconds since the MIGRATE_SOCKET_CACHE_TTL
is 10 seconds.
```
/* Cleanup expired MIGRATE cached sockets. */
run_with_period(1000) {
    migrateCloseTimedoutSockets();
}
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-05-12 15:31:22 +00:00
24ccc63f24 Speed up cluster startup by 10 seconds (#3606)
Replicas with replication offset 0 are flagged as "loading" in CLUSTER
SHARDS and are hidden in CLUSTER SLOTS. These commands assume that the
replicas have not yet synched from their primary, but the replication
offset is zero also when the primary hasn't had any commands written to
the replication stream yet.

The replication offset becomes non-zero when the primary sends a PING
on the replication stream, which it does every 10 seconds by default.
This eventually marks the replicas "online" in CLUSTER SHARDS / CLUSTER
SLOTS.

This change sends a PING on the replication stream when the first
replica connects, to trigger a non-zero replication offset and make the
replicas show up as "online" immediately.

The test framework's "start_cluster" waits for the all nodes to be
marked "online" before running test cases. Production deployment tools
may have similar checks. With this change, starting a cluster gets up to
10 seconds faster.

Some tests with large clusters need these extra 10 seconds for the
cluster to stabilize, so test framework's wait_for_cluster_propagation
is allowed 10 extra seconds (60 instead of 50).

**Additional test framework change:** Catch assertions and print the
duration of setting up the cluster in start_cluster.

This change catches asserts and timeouts in wait_for_condition during
start_cluster's setup phase, prints the error and skips the test cases
within the cluster_cluster body, instead of crashing the whole test
framework with a Tcl exception and stack trace. This is implemented
as a test fixture with a setup and teardown code, which is a new
construct added to the test framework.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2026-05-12 15:30:47 +00:00
chzhooandLucas Yang ed2a1fba7b Reduce latency spikes during rehashing via incremental page release (#3481)
Incrementally release memory to the OS using `madvice(MADV_DONTNEED)`
when rehashing. This reduces the latency of the `free()` call at the end
of the rehashing.

### Problem

Upon rehashing completion, `rehashingCompleted()` frees the old table
via `zfree()`. For large tables (e.g., tens of millions of keys), this
operation can take tens of milliseconds, causing noticeable latency
spikes in the server.

### Solution

After each bucket migration, if 16 pages worth of buckets have been
processed, we immediately release those pages to the OS via
`madvise(MADV_DONTNEED)`. This incremental approach ensures that by the
time `rehashingCompleted()` is called, most physical pages have already
been returned to the OS, making the final cleanup fast and predictable.

---------

Signed-off-by: chzhoo <czawyx@163.com>
2026-05-12 15:30:13 +00:00
03dd56148e CLUSTERSCAN range bounded scanning across contiguous slots (#3391)
Instead of scanning one slot at a time `CLUSTERSCAN` now scans the
entire contiguous range of slots owned by the current node.

Implementation details

* Re-sharding safe as the hash slot is updated based on the local cursor
  position.
* Fingerprint remains stable across the entire contiguous slot range
  instead of being reset per slot.
* Parsing/validation of parameters for the SCAN commands is refactored
  and moved to a separate function.

```
   > CLUSTERSCAN 0
   "0-{06S}-0"                      # start at slot 0

   > CLUSTERSCAN 0-{06S}-0
   "aBcDeF-{06S}-48"                # scanning slot 0...

   > CLUSTERSCAN aBcDeF-{06S}-48
   "aBcDeF-{1Y7}-16"                 # slot 0 done, continues to slot 6 (same node hence FP is unchanged)

   > CLUSTERSCAN aBcDeF-{1Y7}-16
   "aBcDeF-{0or}-32"                # slot 6 done, continues to slot 100  (same node hence FP is unchanged)
   ...
   > CLUSTERSCAN aBcDeF-{...}-64
   "0-{8YG}-0"                      # Current continuous slot boundary reached hence cross-node transition 
```

Follow-up of #2934

---------

Signed-off-by: nmvk <r@nmvk.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-05-12 15:29:29 +00:00
Viktor SöderqvistandLucas Yang 0e74303f82 Add null check in updateSSLPendingFlag (#3641)
Fixes #3607

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-05-12 15:28:50 +00:00
Sarthak AggarwalandLucas Yang d55c0956b9 Skip deferred_reply test in req/res log validation (#3642)
Introduced in test here #3578

The `deferred_reply` module regression test can emit two top-level RESP
replies for a single client command: the normal `SET` reply followed by
the module callback reply.

The req/res log validator assumes each logged argv is followed by one
response.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-05-12 15:28:14 +00:00
eifrah-awsandLucas Yang 2f575d0dce Fix Deferred Reply Placeholders in Active Deferred Buffers (#3578)
Ensure deferred-length reply placeholders are created and resolved in
the same reply list that subsequent nested replies use when the deferred
reply buffer is active. This prevents malformed responses when module
callbacks build postponed-length arrays while a client is already
deferring output.

Add a regression test module and unit test that reproduce the issue
through keyspace notifications and verify the nested array reply is
serialized correctly. Register the new module with the module test
build.

Here is a small illustration of what happens with and without the fix:

## With the fix — placeholder goes to `c->deferred_reply`:

```
   c->buf:            (empty)
   c->reply:          (empty)
   c->deferred_reply: [+OK\r\n] [*2\r\n] [+first\r\n] [*2\r\n] [+a\r\n] [+b\r\n]
                       ^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                       SET reply  reply callback (outer array, inner array via POSTPONED_LEN)
   → commitDeferredReplyBuffer joins into c->reply
   → Wire: +OK\r\n *2\r\n +first\r\n *2\r\n +a\r\n +b\r\n  ✓
            OK      [ "first",        ["a",   "b"] ]
```

##    Without the fix — placeholder goes to `c->reply`:
```
   c->buf:            (empty)
   c->reply:          [*2\r\n]          ← placeholder filled by setDeferredArrayLen (WRONG LIST!)
   c->deferred_reply: [+OK\r\n] [*2\r\n] [+first\r\n] [+a\r\n] [+b\r\n]
                       ^^^^^^^^  outer array  "first"     "a"      "b"
   → commitDeferredReplyBuffer appends deferred_reply AFTER reply
   → Wire: *2\r\n +OK\r\n *2\r\n +first\r\n +a\r\n +b\r\n  ✗
           [ OK,          ["first",   "a"] ]  "b"  ← orphaned   
```

* networking
* tests/modules
* tests/unit/moduleapi

**Generated by CodeLite**

---------

Signed-off-by: Eran Ifrah <eifrah@amazon.com>
2026-05-12 15:27:39 +00:00
Raghav MuddurandLucas Yang 5ffa81abbf Fix the memory leak in valkey-benchmark (#3643)
CI caught ip and name SDS allocations being leaked in
fetchClusterConfiguration. The ip SDS was copied again via sdsnew()
before being passed to createClusterNode(), leaking the original. The
name SDS was leaked when the node already existed in the dict.

Free ip and name on all exit paths in fetchClusterConfiguration. Remove
stale guard in freeClusterNode, no longer needed since #1392

CI Error -
```
Direct leak of 33 byte(s) in 3 object(s) allocated from:
      #0 0x7f4c3a0fd9c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
      #1 0x5564620c124a in ztrymalloc_usable_internal /home/runner/work/valkey/valkey/src/zmalloc.c:172
      #2 0x5564620c124a in zmalloc_usable /home/runner/work/valkey/valkey/src/zmalloc.c:268
      #3 0x5564620dfbe6 in _sdsnewlen.constprop.0 /home/runner/work/valkey/valkey/src/sds.c:102
      #4 0x556462050996 in sdsnewlen /home/runner/work/valkey/valkey/src/sds.c:169
      #5 0x556462050996 in sdsnew /home/runner/work/valkey/valkey/src/sds.c:185
      #6 0x556462050996 in fetchClusterConfiguration /home/runner/work/valkey/valkey/src/valkey-benchmark.c:1477
```

Issue was reproduceable locally using `leaks --atExit`

Signed-off-by: nmvk <r@nmvk.com>
2026-05-12 15:26:57 +00:00
Quanye YangandLucas Yang ef4e56c870 Fixes server crash when RDMA benchmark clients disconnect (#3448)
Fixes server crash when RDMA benchmark clients disconnect (part of
#3345).

This PR focuses on the server-side crash fix. The benchmark
client-side fix will be submitted in a separate PR.

## Server Crash on Client Disconnect 

​ When interrupting valkey-benchmark with Ctrl+C, the server would crash
with a segmentation fault in handleReadResult, accessing address 0x8
(NULL pointer dereference).

### Root Cause

The same RDMA connection could be added to pending_list multiple times,
leading to use-after-free:

1. When a client disconnects, rdmaHandleDisconnect() adds the connection
to pending_list and sets conn->state = CONN_STATE_CLOSED
2. Before rdmaProcessPendingData() processes it, event-driven callbacks
(e.g., connRdmaSetWriteHandler) could add the same connection again to
pending_list
3. In rdmaProcessPendingData(), the first iteration processes and may
free the connection, but the second iteration accesses the already freed
connection → SIGSEGV at address 0x8 (NULL + offset)

The problem was exacerbated because:

  - valkey-benchmark creates multiple concurrent connections
  - On Ctrl+C, these connections disconnect nearly simultaneously
  - Event callbacks can re-add connections before processing completes

 **Stack trace from crash:**

  	handleReadResult+0x100
  	readQueryFromClient+0x63
  	rdmaProcessPendingData (0x1b1eb9)
  	beforeSleep+0x82

### Solution

 Four targeted fixes in `src/rdma.c`:

1. `rdmaHandleDisconnect (line 535)`: Check if pending_list_node is NULL
before adding to pending_list to prevent duplicates

   ```C
/* we can't close connection now, let's mark this connection as closed
state */
   if (rdma_conn->pending_list_node == NULL) {
       listAddNodeTail(pending_list, conn);
       rdma_conn->pending_list_node = listLast(pending_list);
   }
   ```

2. `connRdmaSetWriteHandler (line 956)`: Add same check before adding to
pending_list to prevent duplicates during write handler updates

   ```C
   /* does this connection has pending write data? */
   if (func) {
       if (rdma_conn->pending_list_node == NULL) {
           listAddNodeTail(pending_list, conn);
           rdma_conn->pending_list_node = listLast(pending_list);
       }
   } else if (rdma_conn->pending_list_node) {
       listDelNode(pending_list, rdma_conn->pending_list_node);
       rdma_conn->pending_list_node = NULL;
   }
   ```

3. `rdmaProcessPendingData (line 1786)`: Use iterator node 'ln' instead
of rdma_conn->pending_list_node when deleting, as the latter may have
been overwritten if the connection was re-added

   ```C
listDelNode(pending_list, ln); // Use ln, not
rdma_conn->pending_list_node
   rdma_conn->pending_list_node = NULL;
   ```

4. `rdmaProcessPendingData (line 1782-1787)`: Reorder operations to call
handlers before deleting from list, ensuring connection structure
remains valid during handler execution

   ```C
if (conn->state == CONN_STATE_ERROR || conn->state == CONN_STATE_CLOSED)
{
       /* Invoke handlers first, then remove from list */
       if (callHandler(conn, conn->read_handler)) {
           callHandler(conn, conn->write_handler);
       }

       listDelNode(pending_list, ln);
       rdma_conn->pending_list_node = NULL;

       ++processed;
       continue;
   }
   ```

The fix uses **pending_list_node** as both a list pointer and a
"**already in list**" flag:

  - NULL = not in list, safe to add
  - non-NULL = already in list, skip adding

This ensures each connection appears in pending_list at most once,
preventing use-after-free crashes.

Signed-off-by: Ada-Church-Closure <nunotabashinobu066@gmail.com>
2026-05-12 15:26:27 +00:00
Quanye YangandLucas Yang 80a1d98668 valkey-benchmark: centralize RDMA WRITABLE kick via createFileEvent (#3492)
Add createFileEvent(): for non-RDMA, it is just aeCreateFileEvent.
For RDMA, when registering AE_WRITABLE, register the event and call the
handler once (same as the old direct writeHandler kick).
Do not speculatively call readHandler after registering AE_READABLE
(my previous approach): that interacts badly with libvalkey’s
valkey-io/libvalkey#301 wakeup path and can stall the benchmark after
the first request.

Problem fixed:

`valkey-benchmark --rdma` could hang or stall (e.g. GET-heavy
workloads). Part of that was lost EPOLLIN when libvalkey drained the CQ
and processed inbound data on the write path: the outer poll never saw a
“new” edge. That belongs in libvalkey and is addressed by
valkey-io/libvalkey#301 (eventfd re-arm / nested epoll integration).

Separately, RDMA is not driven by kernel POLLOUT on the benchmark’s fd
the way TCP is, so AE_WRITABLE may never fire unless we explicitly run
the write path once after registration. The code previously did that
with ad-hoc writeHandler calls (issueFirstRequestForClients,
resetClient).

Benchmark RDMA runs should use a libvalkey tree that includes
valkey-io/libvalkey#301 (or equivalent fix).

---------

Signed-off-by: Ada-Church-Closure <nunotabashinobu066@gmail.com>
2026-05-12 15:24:55 +00:00
sananesandLucas Yang 2c94512309 Fix SIGSEGV in VM_GetLRU/SetLRU/GetLFU/SetLFU on NULL key (#3610)
## Fix SIGSEGV in VM_GetLRU, VM_SetLRU, VM_GetLFU, VM_SetLFU on NULL key

### Description

`VM_GetLRU`, `VM_SetLRU`, `VM_GetLFU`, and `VM_SetLFU` crash with
SIGSEGV when passed a NULL `ValkeyModuleKey` pointer. This happens
because all four functions dereference `key->value` without first
checking if `key` itself is NULL.

When a module opens a nonexistent key in `VALKEYMODULE_READ` mode,
`VM_OpenKey` returns NULL. If a module passes that NULL pointer into any
of these functions, the server crashes.

### Reproduction

```
valkey-server --loadmodule tests/modules/misc.so
valkey-cli test.getlru nonexistent_key
# Server crashes: SIGSEGV (signal 11)
```

### Fix

**`src/module.c`** — Add a `!key` guard before dereferencing
`key->value` in all four functions:

```c
// Before:
if (!key->value) return VALKEYMODULE_ERR;

// After:
if (!key || !key->value) return VALKEYMODULE_ERR;
```

**`tests/modules/misc.c`** — Add early return after
`open_key_or_reply()` in `test_getlru`, `test_setlru`, `test_getlfu`,
and `test_setlfu`. The helper already sends the error reply to the
client when the key is not found, so the command handler just needs to
stop processing:

```c
ValkeyModuleKey *key = open_key_or_reply(ctx, argv[1], VALKEYMODULE_READ|VALKEYMODULE_OPEN_KEY_NOTOUCH);
if (!key) return VALKEYMODULE_OK;
```

### After fix

```
valkey-cli test.getlru nonexistent_key
(error) key not found
# Server stays up
```

Signed-off-by: Yaron Sananes <yaron.sananes@gmail.com>
2026-05-12 15:24:07 +00:00
Brad BebeeandLucas Yang 4258c0edf2 Fix checkPrefixCollisionsOrReply returning non-zero on self-overlap (#3583) 2026-05-12 14:27:20 +00:00
Sarthak AggarwalandLucas Yang dee1dfcee1 Run ASan Tests on run-extra-tests label (#3512)
It's important to enabled ASAN on run-extra-tests label so we can
catch some of the bugs in the PRs before they are merged into unstable.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-05-12 14:26:42 +00:00
1e74f42b2c Handle NULL pointer in streamTrim listpack delta calculation (#3591)
When XTRIM marks the last entry in a listpack node as deleted, lpNext()
returns NULL after the lp-count field (EOF). The delta calculation (p -
lp) on a NULL pointer is undefined behavior and produces a garbage
pointer, corrupting the listpack. A subsequent XREAD hitting the
corrupted node triggers the lpValidateNext assertion failure and crashes
the server.

Guard the delta calculation with a NULL check so the while(p) loop
terminates naturally when the last entry is reached.

Fixes #3569

Signed-off-by: Saurabh Kher <saurabh@amazon.com>
Co-authored-by: Saurabh Kher <saurabh@amazon.com>
2026-05-12 14:26:03 +00:00
FAN PEIandLucas Yang 911541e905 Fix off-by-one boundary in lpEncodeBacklen() for 3 values (#3601)
The function lpEncodeBacklen() uses `<= 127` for the 1-byte case but `<
16383`, `< 2097151`, and `< 268435455` for the subsequent cases. This
means the exact values 16383, 2097151, and 268435455 (i.e. 2^14-1,
2^21-1, 2^28-1) unnecessarily use one extra byte than needed:

- `l < 16383` → `16383` (2^14-1) uses 3 bytes instead of 2
- `l < 2097151` → `2097151` (2^21-1) uses 4 bytes instead of 3
- `l < 268435455` → `268435455` (2^28-1) uses 5 bytes instead of 4

The decoding side (`lpDecodeBacklen`) is unaffected since it parses
continuation bits continuously without discrete range checks.

This is a correctness issue and has no impact on data integrity since
encoding and decoding use the same function boundaries, but it wastes up
to 1 byte per affected entry.

Signed-off-by: fanpei91 <fanpei91@gmail.com>
2026-05-12 14:25:16 +00:00
Jeff DuffyandLucas Yang 5e1b98f000 Fix compilation error: replace deprecated je_calloc with zcalloc_num (#3592)
Fixes #1905

## Summary
The direct use of `je_calloc` in `src/allocator_defrag.c` causes
compilation failures on systems (e.g., Arch Linux with GCC 14.2.1) where
`calloc` is marked as deprecated and `-Werror=deprecated` is enabled.

## Changes
Replace the two `je_calloc` calls in `allocatorDefragInit()` with
`zcalloc_num`, which is the proper Valkey allocation wrapper that
provides the same semantics (num × size with zero-fill) without directly
invoking the deprecated `calloc` symbol.

## Testing
- Build compiles cleanly
- Integration tests pass (unit/memefficiency, defrag, unit/other — 51
passed, 0 failed)

Signed-off-by: jaduffy <jaduffy@amazon.com>
2026-05-12 14:24:36 +00:00
Jacob MurphyandLucas Yang e648751f45 fix: validate key count before allocating result in keyspec (#3598)
In `getKeysUsingKeySpecs`, when extracting keys based on the
`KSPEC_FK_KEYNUM `spec (like in the `EVAL` command), the server read the
number of keys from the arguments and calculated the expected end index.

However, it called `getKeysPrepareResult` to allocate memory for the
result array before validating whether last was within the bounds of the
actual arguments provided.

If a client sent a command with a huge declared number of keys (e.g.,
`COMMAND GETKEYS EVAL "return 1" 2147483647 key1`), the server would
allocate a massive amount of memory. Since `vm.overcommit_memory` is
recommended, this allocation would NOT normally have triggered OOM (we
never wrote to it so there is no physical memory allocated), but if you
disable overcommit, this could trigger an OOM.

You can reproduce it with:

```
$ prlimit --as=1073741824 src/valkey-server --save ""
...
384270:M 30 Apr 2026 04:27:24.456 * Ready to accept connections tcp

...
<in valkey-cli>
127.0.0.1:6379> command getkeys eval "return 1" 2147483647 key1

...
<in server log>
384270:M 30 Apr 2026 04:29:26.950 # Out Of Memory allocating 17179869176 bytes!
```

## Solution

* Moved the bounds check `if (last >= argc || last < first || first >=
argc)` to execute before the call to `getKeysPrepareResult`, preventing
the large allocation on invalid input.
* To further catch issues like this, protected against integer overflow
during the calculation of last by using a long long temporary variable.
If it exceeds INT_MAX or falls below INT_MIN, the spec is marked invalid
immediately.

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2026-05-12 14:23:57 +00:00
chenshiandLucas Yang 4ed961e928 Fix: prevent NULL dereference crash in connectSlotExportJob when target node disappears (#3596)
### Summary

This PR fixes a NULL pointer dereference (SIGSEGV) in
`connectSlotExportJob()`
(`src/cluster_migrateslots.c`) that can crash a Valkey cluster node,
causing a
denial-of-service condition.

### Root Cause

When `CLUSTER MIGRATESLOTS` is issued, a migration job is created with
state
`SLOT_EXPORT_CONNECTING`. On the next `clusterCron()` tick,
`proceedWithSlotMigration()` calls `connectSlotExportJob()`, which looks
up the
target node via `clusterLookupNode()`.

`clusterLookupNode()` can legitimately return `NULL` — for example, if
the
target node is removed from the cluster (e.g. via `CLUSTER FORGET`)
between the
time the migration job is created and the time the cron fires. This is a
realistic race condition in any cluster topology change scenario.

The return value was **never checked**, so the subsequent call to
`getNodeDefaultReplicationPort(n)` immediately dereferences the NULL
pointer,
crashing the process:

```c
// Before fix — vulnerable
clusterNode *n = clusterLookupNode(job->target_node_name, CLUSTER_NAMELEN);
int port = getNodeDefaultReplicationPort(n);  // SIGSEGV if n == NULL
serverLog(..., n->ip, port);                  // second dereference

Signed-off-by: chenshi5012 <chenshi5012@163.com>
2026-05-12 14:23:22 +00:00
Jim BrunnerandLucas Yang 341060a566 fix LTO compilation warning in eval (#3584)
I noticed this LTO compile warning in the eval code. Looks like it's
getting confused about an sds length, even though checked above. Just
added an assert to clarify.

The warning:
```c
    LINK valkey-server
eval.c: In function ‘evalExtractShebangFlags’:
eval.c:263:27: warning: argument 1 value ‘18446744073709551615’ exceeds maximum object size 9223372036854775807 [-Walloc-size-larger-than=]
             *out_engine = zcalloc(engine_name_len + 1);
                           ^
zmalloc.c:324:7: note: in a call to allocation function ‘valkey_calloc’ declared here
 void *zcalloc(size_t size) {
       ^
```

Signed-off-by: Jim Brunner <brunnerj@amazon.com>
2026-05-12 14:22:46 +00:00
e1a2c35620 Fix GEOSEARCH BYPOLYGON leak on invalid COUNT (#3568)
Free BYPOLYGON points before returning from invalid COUNT parsing paths
in GEOSEARCH/GEOSEARCHSTORE.

Closes #3567

---------

Signed-off-by: Su Ko <rhtn1128@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2026-05-12 14:21:44 +00:00
f4092eec8b Set errno on EOF in syncRead and propagate it in logs (#3580)
When read() returns 0 (EOF/connection closed) in syncRead(), errno is
not set by POSIX, so it retains a stale value (typically 0). This causes
callers using connGetLastError() to log strerror(0) which is the
misleading string "Success".

Set errno = ECONNRESET on EOF in syncRead(), matching the existing
pattern used for the timeout case (errno = ETIMEDOUT).

Also set conn->last_errno = errno in connSocketSyncWrite,
connSocketSyncRead, and connSocketSyncReadLine wrappers, matching the
pattern used by their async counterparts connSocketWrite and
connSocketRead.

After this fix, replica logs will show:
  "I/O error reading bulk count from PRIMARY: Connection reset by peer"
instead of the misleading:
  "I/O error reading bulk count from PRIMARY: Success"

---------

Signed-off-by: Abhishek Mathur <matshek@amazon.com>
Signed-off-by: djk1027 <djk9510271@gmail.com>
Co-authored-by: Abhishek Mathur <matshek@amazon.com>
Co-authored-by: Daejun Kim <djk9510271@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2026-05-12 14:21:08 +00:00
Jim BrunnerandLucas Yang 41c33a407f fix compile warning in util.c (#3585)
Address this compile warning:
```c
    CC util.o
util.c:638:1: warning: ‘no_sanitize’ attribute directive ignored [-Wattributes]
 __attribute__((no_sanitize_address, no_sanitize("thread"), used)) static int (*string2ll_resolver(void))(const char *, size_t, long long *) {
 ^~~~~~~~~~~~~
```
Addresses portability concerns around these attributes.

---------

Signed-off-by: Jim Brunner <brunnerj@amazon.com>
2026-05-12 14:20:29 +00:00
BinbinandLucas Yang e335c9495a Skip cluster resharding test under valgrind (#3574)
This change was introduced in #3382. This test is already very slow on
its own. Under valgrind it gets slow enough that the per-node restart
step lets primaries be marked FAIL and triggers failovers, after which
"Verify slaves consistency" no longer holds since it assumes the original
topology.

It was never run under valgrind before and exercises nothing valgrind
meaningfully covers, so just tag it valgrind:skip.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-05-12 14:19:59 +00:00
Daejun KimandLucas Yang 17b73588c3 Remove redundant count division in genericHgetallCommand (#3573)
The argument `count /= 2` modifies `count` as a side effect, and the
following `count /= 2` divides it again unnecessarily.
Since `count` is not used after this point, fix it by using `count / 2`
without the side effect and remove the redundant second assignment.

Signed-off-by: djk1027 <djk9510271@gmail.com>
2026-05-12 14:19:07 +00:00
6ad6bd38d2 Migrate the remaining cluster tests to the new framework and remove legacy files (#2297) (#3382)
Migrated the remaining cluster tests to tests/unit/cluster/ to use the same
framework for all cluster tests. Cleaned up the obsolete cluster test framework
files and updated the CI workflows to use the new unified test runner.

Changes:
  Moved and mapped 6 test files:
  - 03-failover-loop.tcl → Merged into existing failover.tcl
  - 04-resharding.tcl → resharding.tcl
  - 12-replica-migration-2.tcl + 12.1-replica-migration-3.tcl →
  replica-migration-slow.tcl
  - 07-replica-migration.tcl → Merged into existing replica-migration.tcl
  - 28-cluster-shards.tcl → Merged into existing cluster-shards.tcl

Other changes:
  - Converted old framework APIs (e.g., K, RI) to new framework APIs (e.g., R, srv)
  - Added process_is_alive check in cluster_util.tcl to fix an exception in
  failover tests caused by executing ps on dead processes
  - Heavy tests (resharding, replica-migration-slow) marked with slow tag and
  wrapped in run_solo to prevent resource contention in sanitizer environments
  - replica-migration-slow marked with valgrind:skip tag since it is very slow
  - Removed the entire tests/cluster/ directory including run.tcl, cluster.tcl,
  includes/, and helpers/
  - Kept runtest-cluster as a wrapper script (exec ./runtest --cluster "$@")
  - Removed ./runtest-cluster calls from .github/workflows/daily.yml as cluster
  tests are now included in ./runtest

Closes #2297.

Signed-off-by: Jun Yeong Kim <junyeonggim5@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2026-05-12 14:18:04 +00:00
Viktor SöderqvistandLucas Yang 1b96c06256 Unique samples in hashtableSampleEntries (#3460)
Instead of "scanning" random bucket chains using a random cursor for
each scan call, start at a random cursor and then continue sampling
buckets in scan order. The scan stops when we have sampled all elements,
so the cursor never wraps around to sample the same buckets again. This
ensures that we don't get any duplicate samples.

The functions hashtableRandomEntry and hashtableFairRandomEntry keeps
the old behavior using another sampling function preserving their
behavior. The fairness tests fail if we use the modified
hashtableSampleEntries.

This restores the behavior of dictGetSomeKeys (which is now an alias of
hashtableSampleEntries) and deflakes the test case:

Gossip count scales with higher percentage of
`cluster-message-gossip-perc`
    in tests/unit/cluster/packet.tcl

Fixes #3454

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-05-12 14:17:05 +00:00
06a188eba9 Add cluster bus network traffic usage metric in bytes (#3396)
Adds cluster bus byte metrics to `CLUSTER INFO`, split by admin,
pub/sub, and module traffic. The change tracks sent and received bytes
on the cluster bus, exposes them as
`cluster_stats_*_bytes_{sent,received}`.

Example:

```
> src/valkey-cli CLUSTER INFO
cluster_state:fail
...
cluster_stats_bytes_sent:46088
cluster_stats_bytes_received:46080
cluster_stats_pubsub_bytes_sent:0
cluster_stats_pubsub_bytes_received:0
cluster_stats_module_bytes_sent:0
cluster_stats_module_bytes_received:0
total_cluster_links_buffer_limit_exceeded:0
```

Fixes: https://github.com/valkey-io/valkey/issues/1929

---------

Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-05-12 14:13:55 +00:00
dc84eedd18 Fix UAF in unblockClientOnKey when reprocessed command frees the client (CVE-2026-23479) (#3614)
When a blocked client is woken up on a key, unblockClientOnKey()
re-executes its pending command via processCommandAndResetClient(). That
function returns C_ERR when the client was freed as a side effect of the
re-execution (it detects this by server.current_client becoming NULL
inside freeClient()).

The pre-fix code ignored the return value and unconditionally accessed
c->flag.blocked / c->flag.module afterwards -- a use-after-free on the
freed client.

The fix mirrors the existing dead-client handling in
processPendingCommandAndInputBuffer() (src/networking.c): on C_ERR,
unwind the execution unit, restore server.current_client, and return
without touching c.

Signed-off-by: ikolomi <ikolomin@amazon.com>
Co-authored-by: ikolomi <ikolomin@amazon.com>
2026-05-07 08:06:46 +03:00
b9ca1a6eda Fix invalid memory access in RESTORE with malformed zipmap (CVE-2026-25243) (#3620)
1. src/zipmap.c: Add sanity checks in zipmapValidateIntegrity() to
reject entries where the decoded length < ZIPMAP_BIGLEN (254) but the
encoding uses more than 1 byte. This prevents a pointer arithmetic
mismatch between validation and zipmapNext() that leads to heap buffer
over-reads.

2. src/rdb.c (hash zipmap conversion): Reorder the
hashtableAdd()/lpSafeToAdd() checks so lpSafeToAdd() is evaluated before
hashtableAdd() takes ownership of the field SDS. Add missing lpFree(lp)
in the error path to fix a memory leak when the conversion fails.

Signed-off-by: ikolomi <ikolomin@amazon.com>
Co-authored-by: ikolomi <ikolomin@amazon.com>
2026-05-05 16:44:34 -07:00
60ba7dbf88 Delay full sync during yielding Lua scripts to prevent use-after-free (CVE-2026-23631) (#3626)
During a full sync, the functions/scripting engine is freed right before
loading the RDB from the primary. If a Lua script is still running and
yielding via the long-command mechanism at that moment, the freed engine
can be accessed when the script resumes, causing a use-after-free.

Add a guard at the top of replicaReceiveRDBFromPrimaryToMemory() to
check isInsideYieldingLongCommand() and return early, deferring the sync
processing until the script completes.

Signed-off-by: ikolomi <ikolomin@amazon.com>
Co-authored-by: ikolomi <ikolomin@amazon.com>
2026-05-05 15:47:55 -07:00
81bed3ec00 Update version to 9.1.0-rc2 and add release notes (#3521)
Bump `VALKEY_RELEASE_STAGE` from `rc1` to `rc2` and add release notes
for changes backported from `unstable` in #3519

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthakaggarwal97@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2026-04-28 09:32:37 -07:00
Sarthak AggarwalandGitHub 80022b4c2c Revert "Fail fast on invalid certificates at TLS config load (#2999)" (#3572)
This reverts commit dafa73164a.

As discussed in the TSC Core Meeting, this PR reverts #2999 from Valkey
9.1. We plan to do in next major version as it's a breaking change.

**Note**: Merge conflicts in `src/tls.c` includes were resolved due to
overlapping changes from PR #2913, which added
`tlsUpdateCertInfoFromDir()` with shared header dependencies.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-04-27 13:36:03 -07:00
eifrah-awsandMadelyn Olson a3e1fe0cd8 Fix remove cached eval scripts on engine unregister (#3503)
Remove eval script cache entries that belong to a scripting engine when
that engine is unregistered. This prevents the eval cache from retaining
dangling engine pointers and keeps the tracked script memory in sync
after engine shutdown.

The scripting engine unregister path now invokes a new eval cleanup
helper, which scans the cached scripts, drops matching entries from the
LRU list and dictionary, and adjusts cache memory accounting accordingly.

* scripting engine
* eval cache

Signed-off-by: Eran Ifrah <eifrah@amazon.com>
2026-04-27 13:35:11 -07:00
Jacob MurphyandMadelyn Olson 01ee9f276d Ensure client slot migration pointer is cleared during reset (#3554)
If not cleared, the job may no longer be valid by the time the client
goes to cleanup. This dangling reference could cause a crash if you set
slot-migration-log-max-len to 0 and are very unlucky.

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 754e908e14 Fix lua-enable-insecure-api default value cannot be changed to yes (#3548)
The default value of lua-enable-insecure-api cannot be safely changed
from no to yes due to two issues:

1. In createEngineContext(), lua_enable_insecure_api was hardcoded to 0
before initializing Lua states, so deprecated APIs (newproxy, setfenv,
   getfenv) were never registered in the global table regardless of the
   actual config value. Once the global table is locked, the config
   change has no effect.

2. lua_insecure_api_current was initialized to 0 (struct zero-init) and
   never synced with the final config value. If the default was changed
   to yes(1), a subsequent CONFIG SET no would see both values as 0 and
   skip the evalReset() call in updateLuaEnableInsecureApi().

Fix by reading the real config via isLuaInsecureAPIEnabled() in
createEngineContext() before Lua state initialization, and syncing
lua_insecure_api_current after all config sources (default, config file,
command-line args) are applied.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 69bdda9fb8 Fix rdmaServer leaks when create listen cm id error (#3557)
In here we should go to error to free the resources:
```
error:
    if (listen_cmid) rdma_destroy_id(listen_cmid);
    if (listen_channel) rdma_destroy_event_channel(listen_channel);
    ret = ANET_ERR;

end:
    freeaddrinfo(servinfo);
    return ret;
}
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-04-27 13:35:11 -07:00
charsyamandMadelyn Olson 7decd4f4b7 hashtable: fix dismissHashtable madvise size (#3533)
The bug was in dismissHashtable(), which computes the size passed to
zmadvise_dontneed() for the top-level hashtable
  tables.

ht->tables[i] points to a contiguous array of bucket objects, but the
code used sizeof(bucket *) instead of
sizeof(bucket) when calculating the length. That means it treated the
allocation like an array of pointers rather than an
  array of buckets.

As a result, the advised range was much smaller than the actual table
allocation. On 64-bit builds, bucket is 64 bytes
while bucket * is 8 bytes, so only about one eighth of the table was
covered. This does not usually break correctness,
but it defeats the purpose of the function: after a fork, we want to
tell the kernel that the hashtable pages are no
longer needed so we reduce copy-on-write overhead. With the wrong size,
most of the table memory was never included in
  that hint.

The fix is to use sizeof(bucket) so the full top-level bucket array is
passed to zmadvise_dontneed().

Signed-off-by: DaeMyung Kang <charsyam@gmail.com>
2026-04-27 13:35:11 -07:00
Hanxi ZhangandMadelyn Olson cb824a9c29 Strip LTO flags from static Lua module build (#3555)
### Summary

The daily CI sanitizer jobs with clang are failing during the build
step.
When the static Lua module is built with `-flto`, the `.o` files contain
LLVM bitcode that gets archived into `libvalkeylua.a`. The system linker
cannot read this bitcode, causing build failures:

`/usr/bin/ld:
/home/runner/work/valkey/valkey/src/modules/lua/libvalkeylua.a: member
/home/runner/work/valkey/valkey/src/modules/lua/libvalkeylua.a(debug_lua.o)
in archive is not an object`

The previous fix (#3546) pinned clang to version 17, but this was
insufficient, the issue is not just a version mismatch but that the
system linker fundamentally cannot read LTO bitcode from `.a` archives.

Example failure:
https://github.com/valkey-io/valkey/actions/runs/24865821147/job/72801509768

### Fix

Strip LTO flags from OPTIMIZATION in the Lua module Makefile using
  `override`


Tested: https://github.com/hanxizh9910/valkey/actions/runs/24913834442

---------

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
2026-04-27 13:35:11 -07:00
Sarthak AggarwalandMadelyn Olson 1aedfa455a Fix module commandresult event cleanup during unsubscribe and module unload (#3545)
This follows up on the commandresult API work and fixes cleanup around
unsubscribe and module unload.

The main issue was that command-result event listeners could leave stale
state behind. On unload, we removed the listeners themselves but didn’t
fully update the fast-path listener counters. Separately, unsubscribing
with a NULL callback could behave badly if the listener wasn’t present
anymore. In practice, that meant later commands could still walk into
command-result event handling after the module was supposed to be
cleaned up.

Failed in Daily as well yesterday:
https://github.com/valkey-io/valkey/actions/runs/24753491944/job/72421581610#step:10:852
Related Failures:
https://github.com/valkey-io/valkey/pull/2936#issuecomment-4290490199

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-04-27 13:35:11 -07:00
Sarthak AggarwalandMadelyn Olson 98790669e5 Add zmalloc_aligned() and fix SPMC queue buffer alignment (#3504)
The SPMC queue from #3324 needs each `spmcCell` to be cache-line
aligned, but plain `zmalloc()` does not guarantee that in all build
configurations.

This change introduces `zmalloc_cache_aligned()` and uses it for the
SPMC queue buffer allocation in `spmcInit()`.

Failing CI: https://github.com/valkey-io/valkey/actions/runs/24374139344

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-04-27 13:35:11 -07:00
charsyamandMadelyn Olson cc73cf1615 Optimize HGETDEL to pause auto shrink when deleting multiple items (#3535)
Match HGETDEL with the existing batch-delete pattern used by HDEL.

HDEL already pauses hashtable auto-shrink while deleting multiple fields
so shrink evaluation is deferred until the batch completes. HGETDEL was
missing the same optimization even though it also deletes fields in a loop.

Pause auto-shrink for hashtable-encoded hashes before the HGETDEL delete
loop and resume it once afterwards. This preserves observable behavior
and reduces redundant shrink work for multi-field deletes.

Same as #3144.

Signed-off-by: DaeMyung Kang <charsyam@gmail.com>
2026-04-27 13:35:11 -07:00
Madelyn Olson 8550b82ef2 Fix FD leak in connSocketBlockingConnect on timeout (#3541)
## Summary
Fix a file descriptor leak in `connSocketBlockingConnect()` when
`aeWait()` times out.

## Bug
When `anetTcpNonBlockConnect()` succeeds but `aeWait()` times out (e.g.,
MIGRATE to an unreachable host), the fd is leaked because it was never
assigned to `conn->fd`. The caller's `connClose()` checks `conn->fd !=
-1` and skips cleanup.

## Fix
Assign `conn->fd = fd` immediately after `anetTcpNonBlockConnect()`
succeeds, before `aeWait()`. This way the caller's normal `connClose()`
cleanup path handles the fd on any error, which is consistent with how
the rest of the connection lifecycle works.

TLS connections also benefit since `connTLSBlockingConnect` delegates to
this function for the TCP layer.

## Reproducer
```
valkey-cli SET key hello
# Repeat against unreachable host:
for i in $(seq 1 30); do valkey-cli MIGRATE 192.0.2.1 6379 key 0 500; done
# Check: /proc/<pid>/fd shows 30 leaked socket fds
```

*This issue was generated by AI but verified, with love, by a human.*

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 1427741f3b Fix double free in stream consumer PEL loading with corrupt RDB data (#3498)
There is a double free issue in the code. The error handling path called
both decrRefCount(o) and streamFreeNACK(nack), but the nack was obtained
from cgroup->pel via raxFind and is still referenced there. decrRefCount(o)
frees it through freeStream -> streamFreeCG -> raxFreeWithCallback(cg->pel, zfree),
so the explicit streamFreeNACK(nack) causes a double free.

Remove the redundant streamFreeNACK(nack) call and add a regression
test with a crafted corrupt payload that triggers the duplicate consumer
PEL entry path.

This was introduced in 6ec241d934.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-04-27 13:35:11 -07:00
c883ca9a08 Deflake many-slot-migration under valgrind (#3462)
## Problem
`Fix cluster` in `tests/unit/cluster/many-slot-migration.tcl` has been
timing out daily on valgrind jobs since April 3, 2026. The test runs 10
cluster nodes under valgrind, migrating 40,000 keys across 1,000 slots —
too much work for valgrind-instrumented builds.

The slowdown is caused by #3366 (dict→hashtable wrapper). Under `-O0`
(valgrind builds), the `static inline` wrappers become real function
calls that valgrind instruments, adding ~75% overhead to hot paths like
`dictSize`. This compounds across 10 valgrind processes over a 20-minute
migration test. No impact on production builds (`-O2` inlines everything).

## Fix
Scale the test workload down under valgrind: 10,000 keys / 250 slots
instead of 40,000 / 1,000. Normal runs are unchanged. Still exercises
the same cluster repair path.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Co-authored-by: sarthakaggarwal97 <sarthakaggarwal97@users.noreply.github.com>
2026-04-27 13:35:11 -07:00
Deepak NandihalliandMadelyn Olson 8e360c7eb7 Fix race condition during async client freeing with IO threading enabled (#3458)
When close_asap flag is set, set bytes read to 0
    
In the readToQueryBuf, the c->nread represents the number of bytes read.
When close_asap flag is set, there is a bug where the c->nread isn't
reset to 0 and this breaks the invariant. IOThreads then incorrectly
think there is data to read and results in a crash. This change fixes
this bug.

To elaborate on the race possible:

1. Let's say that a IO thread job for reading query from a client got
enqueued as part of a epoll -
https://github.com/valkey-io/valkey/blob/unstable/src/io_threads.c#L417.
2. Later the client gets freed async and is marked as close_asap -
https://github.com/valkey-io/valkey/blob/unstable/src/networking.c#L2175
3. While processing the io_thread job for the client, it invokes
iothreadReadQueryFromClient. Here,
[`readToQueryBuf`](https://github.com/valkey-io/valkey/blob/unstable/src/networking.c#L6497)
returns as a no-op since the client is marked close-asap. Also, the
c->nread is not reset to 0 and count contain the value from a previous
read.
4. Later parseInputBuffer [gets
invoked](https://github.com/valkey-io/valkey/blob/unstable/src/networking.c#L6514).
5. The parseInputBuffer then [accesses the
query_buf](https://github.com/valkey-io/valkey/blob/unstable/src/networking.c#L3864).
The query_buf here would be null in resetSharedQueryBuf as part of
beforeNextClient.

Signed-off-by: Deepak Nandihalli <deepak.nandihalli@gmail.com>
2026-04-27 13:35:11 -07:00
3e2fd3c800 Stabilize diskless no-drop replication test (#3511)
This deflakes all variants of `diskless replicas drop during rdb pipe`.

The main issue turned out to be that the test was too sensitive to
timing and log ordering under TLS, not that the core behavior was wrong.
This keeps the same five subcases (no, slow, fast, all, timeout) but
makes them much less CI-fragile.

CI passes 200 times:
https://github.com/sarthakaggarwal97/valkey/actions/runs/24547258515

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <25262500+sarthakaggarwal97@users.noreply.github.com>
Co-authored-by: Sarthak Aggarwal <25262500+sarthakaggarwal97@users.noreply.github.com>
2026-04-27 13:35:11 -07:00
Yang ZhaoandMadelyn Olson 40b32ac4a8 Fix VLA warning in io_threads (#3518)
https://github.com/valkey-io/valkey/pull/3324 introduced `BATCH_SIZE` as
a const int local variable and used it as an array bound. Clang 17
rejects this with:
```
io_threads.c:305:22: error: variable length array folded to constant array as an extension [-Werror,-Wgnu-folding-constant]
  305 |     void *batch_jobs[BATCH_SIZE];
      |                      ^~~~~~~~~~
1 error generated.
make[1]: *** [io_threads.o] Error 1
make: *** [all] Error 2

```
Old Clang versions do not emit this warning, maybe that is why the CI
passed. Fix by promoting `BATCH_SIZE` to a file-scope `#define`.

Signed-off-by: Yang Zhao <zymy701@gmail.com>
2026-04-27 13:35:11 -07:00
7464d45eaa Module command result callback addition (#2936)
## Add Command Result Event Notifications for Modules

### Summary

1. Adds new server events `ValkeyModuleEvent_CommandResultSuccess` and
`ValkeyModuleEvent_CommandResultFailure` for that can notify subscribed
modules after command execution. This enables modules to implement audit
logging, error monitoring, performance tracking, and observability
without modifying core server code.
2. Adds new server event `ValkeyModuleEvent_CommandResultACLDenied` for
commands rejected by ACL. Together with PR #2237 this covers auditing of
authentication and authorisation.

### Motivation

There is currently no module API to observe command outcomes after
execution or to capture ACL denied commands. Modules that need audit
logging or error monitoring have no mechanism to be notified when
commands succeed or fail, what arguments were used, how long they took,
or how many keys were modified. This feature fills that gap using the
existing `ValkeyModule_SubscribeToServerEvent()` infrastructure.

### API

#### Events

| Event | Description |
|---|---|
| `ValkeyModuleEvent_CommandResultSuccess` | Fired after a command
completes successfully |
| `ValkeyModuleEvent_CommandResultFailure` | Fired after a command
returns an error |
| `ValkeyModuleEvent_CommandACLDenied` | Fired after a command is
rejected by ACL |

These are separate events (not sub-events), so modules can for example
only subscribe to failures without incurring any callback overhead for
successful commands.

#### Event Data: `ValkeyModuleCommandResultInfo`

The `data` pointer passed to the callback can be cast to
`ValkeyModuleCommandResultInfo`:

```c
typedef struct ValkeyModuleCommandResultInfo {
    uint64_t version;           /* Version of this structure for ABI compat. */
    const char *command_name;   /* Full command name (e.g., "SET", "CLIENT|LIST"). */
    long long duration_us;      /* Execution duration in microseconds. */
    long long dirty;            /* Number of keys modified. */
    uint64_t client_id;         /* Client ID that executed the command. */
    int is_module_client;       /* 1 if command was from RM_Call, 0 otherwise. */
    int argc;                   /* Number of command arguments. */
    ValkeyModuleString **argv;  /* Command arguments array (zero-copy, read-only). */
    int acl_deny_reason;        /* ACL_DENIED_CMD/KEY/CHANNEL/AUTH; 0 for non-ACL events */
    const char *acl_object;     /* Denied resource name (key/channel); NULL for CMD/AUTH */
} ValkeyModuleCommandResultInfoV1;
```

The struct is versioned (`VALKEYMODULE_COMMANDRESULTINFO_VERSION`) for
forward-compatible API evolution.

### Usage Example

```c
/* Callback receives events for whichever event(s) you subscribed to */
void OnCommandResult(ValkeyModuleCtx *ctx, ValkeyModuleEvent eid,
                     uint64_t subevent, void *data) {
    VALKEYMODULE_NOT_USED(ctx);
    VALKEYMODULE_NOT_USED(subevent);

    ValkeyModuleCommandResultInfo *info = (ValkeyModuleCommandResultInfo *)data;
    if (info->version != VALKEYMODULE_COMMANDRESULTINFO_VERSION) return;

    int failed = (eid.id == VALKEYMODULE_EVENT_COMMAND_RESULT_FAILURE);

    /* Access fields directly */
    printf("command=%s status=%s duration=%lldus dirty=%lld client=%llu\n",
           info->command_name,
           failed ? "FAIL" : "OK",
           info->duration_us,
           info->dirty,
           info->client_id);

    /* Access argv (read-only, zero-copy) */
    for (int i = 0; i < info->argc; i++) {
        size_t len;
        const char *arg = ValkeyModule_StringPtrLen(info->argv[i], &len);
        printf("  argv[%d] = %.*s\n", i, (int)len, arg);
    }
}

/* Subscribe in ValkeyModule_OnLoad or at runtime */

/* Option A: command failures only (recommended for audit logging) */
ValkeyModule_SubscribeToServerEvent(ctx,
    ValkeyModuleEvent_CommandResultFailure, OnCommandResult);

/* Option B: command successes only */
ValkeyModule_SubscribeToServerEvent(ctx,
    ValkeyModuleEvent_CommandResultSuccess, OnCommandResult);

/* Option C: both command outcomes*/
ValkeyModule_SubscribeToServerEvent(ctx,
    ValkeyModuleEvent_CommandResultSuccess, OnCommandResult);
ValkeyModule_SubscribeToServerEvent(ctx,
    ValkeyModuleEvent_CommandResultFailure, OnCommandResult);

/* Subscribe to ACL Denied */
ValkeyModule_SubscribeToServerEvent(ctx,
        ValkeyModuleEvent_CommandResultACLDenied, onCommandResult);

/* Unsubscribe pass NULL callback */
ValkeyModule_SubscribeToServerEvent(ctx,
    ValkeyModuleEvent_CommandResultFailure, NULL);
```

### Design Decisions

- **Separate events instead of sub-events**: Modules subscribing only to
failures have zero overhead for successful commands (~2ns listener-list
check vs ~30ns callback invocation per command). This is critical since
success events fire on the hot path of every command.
- **Stack-allocated info struct**: The `ValkeyModuleCommandResultInfoV1`
is built on the stack ΓÇö no heap allocation per event.
- **Zero-copy argv**: Arguments are passed directly from the client's
argv array. Any integer-encoded arguments (from `tryObjectEncoding()`
during command execution) are decoded to string-encoded objects before
being passed to the callback, ensuring compatibility with
`ValkeyModule_StringPtrLen()`.
- **Early exit**: If no modules are subscribed to any server events, the
event firing function returns immediately before building the info
struct.
- **Uses existing server event infrastructure**: Follows the
`ValkeyModule_SubscribeToServerEvent()` pattern used by all other server
events, rather than introducing a new callback mechanism.

### Files Changed

| File | Change |
|---|---|
| `src/valkeymodule.h` | Event IDs, event constants,
`ValkeyModuleCommandResultInfoV1` struct |
| `src/module.c` | `moduleFireCommandResultEvent()`, event
documentation, event version entries |
| `src/module.h` | Function declaration |
| `src/server.c` | Call `moduleFireCommandResultEvent()` from `call()`
after command execution |
| `src/server.c` | Call to `moduleFireCommandACLDeniedEvent` in
`processCommand` after ACL rejection |
| `tests/modules/commandresult.c` | Test module exercising the full API
|
| `tests/unit/moduleapi/commandresult.tcl` | Integration tests |

---------

Signed-off-by: martinrvisser <mvisser@hotmail.com>
Signed-off-by: martinrvisser <martinrvisser@users.noreply.github.com>
Co-authored-by: Ricardo Dias <rjd15372@gmail.com>
2026-04-27 13:35:11 -07:00
eifrah-awsandMadelyn Olson 37242267c9 Add Static Module Support (#3392)
Add a build option to compile the Lua scripting engine as a static
module and wire the server to load it directly at startup when enabled.
The module load path now resolves on-load and on-unload entry points
from the main binary, and the module lifecycle keeps those callbacks so
unload works without a shared library handle.

The Lua module build was updated to support both static and shared
variants, with the static path exporting visible wrapper symbols and
linking the server with the module archive. While touching the Lua code,
a few internal symbols were renamed for consistency and the monotonic
time helper was clarified.

Note that this PR addresses the LUA module, but it can be applied to
other "core" modules (like: Bloom, Json, Search and others). With this
change, it will be easier to ship Valkey bundle with modules.

Areas touched:

* CMake
* Makefile
* Lua scripting module
* Core module loading

**Generated by CodeLite**

---------

Signed-off-by: Eran Ifrah <eifrah@amazon.com>
2026-04-27 13:35:11 -07:00
Daniil KashapovandMadelyn Olson 28f1cc9b29 Improve COB memory tracking with copy avoidance (#3306)
This improves COB memory tracking when using copy avoidance for bulk
string replies. This fix addresses underestimation of client memory
usage that occurred when reply buffers stored pointers to shared `robj`
instead of copying data.
IO threads calculate actual reply sizes by calling `sdslen()` on strings
before writing, for that we need atomic `tracked_for_cob` flag in
payload headers to prevent race conditions and double accounting.

See #2396

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
2026-04-27 13:35:11 -07:00
Madelyn Olson 60fbe16c63 Fix HPERSIST RESP protocol violation on wrong-type key (#3516)
`hpersistCommand` calls `addReplyArrayLen` before `lookupKeyWrite` +
`checkType`. When HPERSIST targets a non-hash key, the server writes a
RESP array header followed by a WRONGTYPE error — a malformed response
that permanently desynchronizes the client connection.

This moves `lookupKeyWrite` + `checkType` before `addReplyArrayLen`,
matching the pattern used by every other HFE command (e.g.
`hgetdelCommand`, `hexpireGenericCommand`).

Added a test for HPERSIST on a wrong-type key.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2026-04-27 13:35:11 -07:00
a09e12e27b [Flaky Tests] Avoid re-triggering io-thread activation (#3509)
The test was accidentally waking the IO threads while trying to check
that they had gone idle.

After the recent IO-thread refactor in #3324, the
[test](https://github.com/valkey-io/valkey/pull/3324/changes#diff-21314ec3a338f739eab1536f91f528d1efe7c6a93935a71b9c02f77a3858f121R112)
started forcing `io-threads-always-active`, and its repeated `INFO`
polling counted as fresh activity. So instead of just observing the
worker threads, the test kept reactivating them and then flaked.

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <25262500+sarthakaggarwal97@users.noreply.github.com>
Co-authored-by: Sarthak Aggarwal <25262500+sarthakaggarwal97@users.noreply.github.com>
2026-04-27 13:35:11 -07:00
Roshan KhatriandMadelyn Olson 728595058a Fix use-after-unload crash in test auth module's blocking thread (#3464)
## Problem

The test `Test module aof save on server start from empty` in
`tests/unit/moduleapi/hooks.tcl` sporadically crashes with `I/O error
reading reply`.

**Frequency:** 2 out of 15 days (March 26 on
`centosstream9-tls-module-no-tls`, April 8 on `fedorarawhide-jemalloc`).

**Example failing run:**
https://github.com/valkey-io/valkey/actions/runs/24110987718/job/70345236353

## Root Cause

The crash is a **use-after-unload** in the auth test module's blocking
authentication thread, NOT a timing issue in the AOF test.

The crash log from April 8 shows:
```
71112:M 00:42:59.710 * Module testacl unloaded
71112:M 00:42:59.711 # crashed by signal: 11, si_code: 1
71112:M 00:42:59.711 # Crashed running the instruction at: 0x7f9dc717384b
```

The sequence:
1. `blocking_auth_cb` spawns a background thread
(`AuthBlock_ThreadMain`) that sleeps 500ms
2. Thread wakes, calls `ValkeyModule_UnblockClient()` → main thread
processes unblock, decrements `module->blocked_clients`
3. Auth command completes, test calls `r module unload testacl`
4. `moduleUnloadInternal` checks `blocked_clients == 0` if true,
proceeds with `dlclose()`
5. **But the background thread is still executing cleanup code**
(freeing strings, returning from function)
6. Thread returns into unmapped memory → **SIGSEGV**

The `invalidFunctionWasCalled` in the stack trace is the crash handler's
safety stub, and the crashing address `0x7f9dc717384b` is in the
unmapped auth.so address space.

## Fix

Track the background thread ID and `pthread_join()` it in
`ValkeyModule_OnUnload` before the module is dlclose'd. This ensures the
thread has fully exited before the code is unmapped.

The key insight is that `ValkeyModule_UnblockClient()` signals "auth is
done" but not "thread is done" — the thread still has cleanup code to
execute after that call. `pthread_join()` is the correct synchronization
point because it only returns after the thread has fully exited.

No mutex is needed since both `blocking_auth_cb` (which creates the
thread) and `OnUnload` (which joins it) run on the main event loop
thread.

Changes to `tests/modules/auth.c`:
- Add global `blocking_auth_tid` and `blocking_auth_tid_valid` flag
- Set `blocking_auth_tid_valid = 1` after successful `pthread_create`
- In `OnUnload`, `pthread_join` the thread if one was created

## Testing

Ran `unit/moduleapi/hooks` 100 loops on rpm-distros and ubuntu runners —
**all passed**:
- **Workflow run:**
https://github.com/roshkhatri/valkey/actions/runs/24164276124
- **Config:** `--loops 100 --single unit/moduleapi/hooks` on
`almalinux8`, `almalinux9`, `fedoralatest`, `fedorarawhide`,
`centosstream9`, `ubuntu-jemalloc`, `ubuntu-arm`
- **Result:** 7/7 jobs , zero failures across 700 total test iterations

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 6a6b6c8746 Rewrite the faster failover test case (#3495)
The test introduced in #2227 is fragile because it unconditionally asserted
that "best ranked replica" would be logged for every shard's replica. This
assumption was incorrect for two reasons:

1. myselfIsBestRankedReplica() requires failover_failed_primary_rank == 0.
   When two primaries fail simultaneously, only the shard with the lowest
   shard_id gets failed_primary_rank == 0. The other shard's replicas can
   only trigger "Myself become the best ranked replica" after the first
   shard completes failover and the result propagates via gossip.

2. clusterAllReplicasThinkPrimaryIsFail() requires all sibling replicas
   to have their MY_PRIMARY_FAIL flag propagated via gossip. If the
   election delay is shorter than the gossip propagation time, the
   replica may start the election before receiving the flag.

It's also too strict and checks too many things. Like we can check that there
is no delay before starting the failover, but we shouldn't check that the rank
0 replica actually wins, because maybe it doesn't. Also we're not sure the other
replica does a psync afterwards, it can also do a full sync.

Restructure the test into three focused scenarios with retry logic:

- Test 0 (3 primaries + 1 replica): The sole replica is deterministically
  the best ranked replica. All conditions are trivially satisfied.

- Test 1 (3 primaries + 2 replicas): Single primary failure with two
  competing replicas. failed_primary_rank is deterministically 0, but
  MY_PRIMARY_FAIL gossip timing may prevent triggering in some runs.
  Uses internal retry with cluster recovery to ensure at least one
  successful trigger.

- Test 2 (5 primaries + 7 replicas): Two primaries failure. Verifies no
  failover timeout and uses retry to cover the "best ranked replica"
  path for at least one shard.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 0d297bddb4 Change rdbSaveStreamConsumers return type from size_t to ssize_t (#3499)
Minor cleanup, the function will return -1 on error.
It worked anyway before because it is implicitly converted:
-1 => SIZE_MAX => -1 again when it's implicitly cast back to
ssize_t.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-04-27 13:35:11 -07:00
267e3a54f9 Redesign IO threading communication model (#3324)
#### Summary
This PR redesigns the IO threading communication model, replacing the
inefficient client-list polling approach with a high-performance,
lock-free queue architecture. This change improves throughput by
**8–17%** across various workloads and lays the groundwork for
offloading command execution to IO threads in following PRs.



### Performance Comparison: Unstable vs New IO Queues

| Type | Operation | Unstable Branch (M TPS) | New IO Queues (M TPS)|
Difference (%) |
| :--- | :--- | :--- | :--- | :--- |
| **CME**<sup>1</sup> | SET | 1.02 | 1.19 | **+16.67%** |
| **CME** | GET | 1.30 | 1.47 | **+13.08%** |
| **CMD**<sup>2</sup> | SET | 1.15 | 1.35 | **+17.39%** |
| **CMD** | GET | 1.52 | 1.64 | **+7.89%** |

<sup>1</sup> Amazon terminology for cluster mode
<sup>2</sup> Amazon termonology for standalone mode, i.e. config
`cluster-enabled no`

- Test Configuration: 8 IO threads • 400 clients • 512-byte values • 3M
keys

#### Motivation
The previous IO model had several limitations that created performance
bottlenecks:
* **Inefficient Polling:** The main thread lacks a direct notification
mechanism for completed work. Instead, it must constantly iterate
through a list of all pending clients to check their state, wasting
significant CPU cycles.
* **Manual Load Balancing:** Jobs are assigned to specific threads
upfront. This requires the main thread to predict which thread to use,
often leaving some threads idle while others are overloaded.
* **Static Scaling:** Thread activation relies on a fixed heuristic
(e.g., 1 thread per 2 events). This approach fails to adapt to varying
workloads, such as TLS connections or differing read/write sizes.

### The Solution
To address these inefficiencies, this PR replaces the single SPSC queue
used currently with three specialized queues to handle communication and
load balancing more effectively.

#### 1. Main > IO: Shared Queue (Single Producer Multi Consumer)
Single queue from the main-thread to IO threads.
* **Automatic Load Balancing:** All threads pull from the same source.
Busy threads take less work, and idle threads take more, so we don't
need to manually select a thread.
* **Adaptive Scaling:** We now use the queue depth to decide when to add
or remove threads. If the queue is full, we scale up; if it's empty, we
scale down.
* *Ignition:* To get things started before the queue fills up, we
monitor the main thread's CPU. If usage goes over 30%, we wake up the
first IO thread.
* **Implementation:** To prevent contention among consumers, each item
in the ring buffer is padded to reside in its own cache line. Sequence
numbers are utilized to indicate whether a cell is empty or populated,
allowing threads to safely claim work.

#### 2. IO > Main: The Response Channel (MPSC Queue)
We replaced the old polling loop with a response queue.
* ** Faster Completion:** IO threads push completed jobs into this
queue. The main thread detects new data simply by checking if the queue
is not empty, removing the need to scan pending clients.
* **Contention Management:** To avoid lock contention, each thread
reserves a slot by atomically incrementing the tail index. In the rare
event that the queue is full, pending jobs are buffered in a local
temporary list until space becomes available.

#### 3. MAIN > IO (Thread-Specific): Private Inbox (SPSC Queue)
We kept the existing Single-Producer Single-Consumer (SPSC) queues for
tasks that must happen on a specific thread (like freeing memory
allocated by that thread). IO threads always check their private inbox
before looking at the shared queue.

### Changes Required
* **Async client release**
The main thread no longer busy-waits for IO threads to finish with a
client. Since the client must be popped from the multi-producer queue
before it can be released, clients with pending IO are now marked for
asynchronous closure.
* **eviction clients logic**
Updated evictClients() to account for memory pending release (clients
marked close_asap). freeClient() now returns a status code (1 for freed,
0 for async-close) to ensure the eviction loop does not over-evict by
ignoring memory that is about to be reclaimed.
* **events-per-io-thread config**
Replaced the `events-per-io-thread` configuration with
`io-threads-always-active`. as we no longer track events, since this
config is use only for tests no backward compatibility issue arises.
* **packed job instead of handlers**
Jobs are now represented as tagged pointers (using lower 3 bits for job
type) instead of separate `{handler, data}` structs. This reduces memory
overhead and allows jobs to be passed through the queues as single
pointers.
* **head caching in spsc queue**
The SPSC queue now caches the `head` index on the producer side
(`head_cache`) to avoid frequent atomic loads. The producer only
refreshes from the atomic `head` when the cache indicates the queue
might be full, reducing cross-thread cache-line bouncing.

* **deferred commit in SPSC queue**.   
`spscEnqueue()` supports batching via a `commit` flag. Multiple jobs can
be enqueued with `commit=false`, then flushed with a single
`spscCommit()` call, reducing atomic operations and cache-line bouncing.
* **rollback on fullness check failure**
When `spmcEnqueue()` fails due to a full queue, the client state is
rolled back (e.g., `io_write_state` reset to `CLIENT_IDLE`). This
rollback approach removes the need to call an expensive `isFull` check
before every enqueue, we just attempt the enqueue and revert if it
fails.

* **epoll offloading via SPSC at high thread counts**.  
When `active_io_threads_num > 9`, poll jobs are sent to per-thread SPSC
queues (round-robin). Since threads check their private queue first,
this ensures poll jobs are processed promptly without waiting behind
jobs in the shared SPMC queue.
* **avoid offload write before read comes back**
Added a check `if (c->io_read_state == CLIENT_PENDING_IO) return C_OK`
in `trySendWriteToIOThreads()`. In the previous per-thread SPSC
implementation, we could send consecutive read and write jobs for the
same client knowing a single thread would handle them in order. With the
shared SPMC queue, different threads may pick up the jobs, so we must
wait for the read to complete before sending a write to avoid 2 threads
handling the same client.
* **removing pending_read_list_node from client and
clients_pending_io_read/write lists from server**
Removed `pending_read_list_node` from the `client` struct and
`clients_pending_io_read`/`clients_pending_io_write` lists from
`valkeyServer`. as the new mpsc eliminates the need for these tracking
structures.
* **added inst metrics for pending io jobs**
Added `instantaneous_io_pending_jobs` metric via `STATS_METRIC_IO_WAIT`
to track average queue depth over time.
* **added stat for current active threads number**
Added `active_io_threads_num` to the INFO stats output for better
visibility.
* **added internal inst metric for main-thread cpu (non apple
compliant)**
Added `STATS_METRIC_MAIN_THREAD_CPU_SYS` to track main thread CPU usage
via `getrusage(RUSAGE_THREAD)`. This powers the "ignition" policy, when
CPU exceeds 30%, the first IO thread is activated. `RUSAGE_THREAD` is
Linux-specific, so macOS falls back to event-count heuristics.
* **added stat for pending read and writes for io**
Added `io_threaded_reads_pending` and `io_threaded_writes_pending` stats
to track how many read/write jobs are currently in-flight to IO threads.
* **added volatile for crashed**
Changed `server.crashed` from `int` to `volatile int` to ensure the
crash flag is visible across threads immediately, allowing IO threads to
detect a crash and stop sending responses back to the main thread to
avoid deadlock on crash.

---------

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
Signed-off-by: akash kumar <akumdev@amazon.com>
Co-authored-by: Uri Yagelnik <uriy@amazon.com>
Co-authored-by: Dan Touitou <dan.touitou@gmail.com>
2026-04-27 13:35:11 -07:00
Ricardo DiasandMadelyn Olson 2f351c1b2b Adds new (hidden) ValkeyModule_CallArgv API functions (#3122)
This PR introduces `ValkeyModule_CallArgv`, a new low-level module API
for calling server commands. It is designed as a complement to the
existing `ValkeyModule_Call` covering two scenarios where
`ValkeyModule_Call` is suboptimal:

1. **The module already holds the arguments as an argv array.**
`ValkeyModule_Call` always allocates a new argv array, creates a new
string object for the command name, and calls
`incrRefCount`/`decrRefCount` on every argument. `ValkeyModule_CallArgv`
borrows the caller's argv array directly — no allocation, no refcount
manipulation. Ownership of the array stays with the caller.

2. **The module wants to forward the reply to its own client without
inspecting it.** `ValkeyModule_Call` always parses the raw RESP bytes
into an intermediate `ValkeyModuleCallReply` tree and then re-serializes
it back to RESP to send to the caller. `ValkeyModule_CallArgv` exposes
the raw RESP bytes directly through a callback, allowing a pass-through
module command to copy bytes from the inner command's output buffer
straight into the outer client's output buffer.

---

## New API

### `ValkeyModule_CallArgv`

```c
int ValkeyModule_CallArgv(ValkeyModuleCtx *ctx,
                          ValkeyModuleString **argv,
                          int argc,
                          int flags,
                          ValkeyModuleReplyHandlers *reply_handlers,
                          void *reply_ctx);
```

Calls a server command using an existing argv array. The caller retains
ownership of `argv` and its elements — they are never freed or
ref-counted by the function.

Returns `VALKEYMODULE_OK` on success or `VALKEYMODULE_ERR` on error,
with `errno` set to the same values as `ValkeyModule_Call`. The function
does **not** return a `ValkeyModuleCallReply`; the reply is delivered
through `reply_handlers` instead.

`flags` is a bitwise OR of one or more `VALKEYMODULE_CALL_ARGV_*`
constants, which correspond directly to the format-string specifiers of
`ValkeyModule_Call`:

| Flag | Equivalent format char | Meaning |
|---|---|---|
| `VALKEYMODULE_CALL_ARGV_REPLICATE` | `!` | Propagate to replicas and
AOF |
| `VALKEYMODULE_CALL_ARGV_NO_AOF` | `A` | Skip AOF propagation |
| `VALKEYMODULE_CALL_ARGV_NO_REPLICAS` | `R` | Skip replica propagation
|
| `VALKEYMODULE_CALL_ARGV_RESP_3` | `3` | Force RESP3 for the inner call
|
| `VALKEYMODULE_CALL_ARGV_RESP_AUTO` | `0` | Match the calling client's
protocol (RESP2 or RESP3) |
| `VALKEYMODULE_CALL_ARGV_RUN_AS_USER` | `C` | Apply ACL checks for the
context user |
| `VALKEYMODULE_CALL_ARGV_SCRIPT_MODE` | `S` | Mark as script execution
|
| `VALKEYMODULE_CALL_ARGV_NO_WRITES` | `W` | Disallow write commands |
| `VALKEYMODULE_CALL_ARGV_ERRORS_AS_REPLIES` | `E` | Return error
replies instead of raising errno |
| `VALKEYMODULE_CALL_ARGV_RESPECT_DENY_OOM` | `M` | Honour deny-oom
policy |
| `VALKEYMODULE_CALL_ARGV_DRY_RUN` | `D` | Dry-run mode (implies
`ERRORS_AS_REPLIES`) |
| `VALKEYMODULE_CALL_ARGV_ALLOW_BLOCK` | `K` | Allow the inner command
to block |
| `VALKEYMODULE_CALL_ARGV_REPLY_EXACT` | `X` | Disable reply type
coercion |

---

### `ValkeyModuleReplyHandlers`

```c
typedef struct ValkeyModuleReplyHandlers {
    /* Scalar types */
    void (*null)(void *ctx);
    void (*bulkString)(void *ctx, const char *str, size_t len);
    void (*simpleString)(void *ctx, const char *str, size_t len);
    void (*error)(void *ctx, const char *msg, size_t len);
    void (*integer)(void *ctx, long long val);
    void (*doubleVal)(void *ctx, double val);
    void (*boolVal)(void *ctx, int val);
    void (*bigNumber)(void *ctx, const char *str, size_t len);
    void (*verbatimString)(void *ctx, const char *str, size_t len, const char *fmt);
    /* Collection types — paired start/end callbacks */
    void (*arrayStart)(void *ctx, size_t len);
    void (*arrayEnd)(void *ctx);
    void (*mapStart)(void *ctx, size_t len);
    void (*mapEnd)(void *ctx);
    void (*setStart)(void *ctx, size_t len);
    void (*setEnd)(void *ctx);
    void (*attributeStart)(void *ctx, size_t len);
    void (*attributeEnd)(void *ctx);
    /* Invoked on parse error */
    void (*replyParsingError)(void *ctx);
    /* Raw-bytes intercept — called before per-type callbacks */
    int (*onRespAvailable)(void *ctx, ValkeyModuleCtx *module_ctx, const char *proto, size_t proto_len);
    /* Blocking command support */
    void (*onBlocked)(void *ctx, ValkeyModuleCtx *module_ctx, ValkeyModuleCallArgvBlockedHandle *handle);

    void *context;   /* Passed as first argument to every callback */
} ValkeyModuleReplyHandlers;
```

All fields are optional; set unused ones to `NULL`.

**`onRespAvailable`** is called first, before any per-type callback,
with the complete raw RESP reply in the `proto`/`proto_len` arguments.
Return `1` to continue into the per-type callbacks; return `0` to stop
after the raw bytes. Pass `NULL` to skip and go straight to per-type
dispatching.

**`onBlocked`** is only called when `VALKEYMODULE_CALL_ARGV_ALLOW_BLOCK`
is set and the inner command blocks. The `handle` can be passed to
`ValkeyModule_CallArgvAbort` to cancel the call. The handle is valid
until either `onRespAvailable` is invoked or
`ValkeyModule_CallArgvAbort` is called, whichever comes first.

---

### `ValkeyModule_CallArgvAbort`

```c
int ValkeyModule_CallArgvAbort(ValkeyModuleCallArgvBlockedHandle *handle);
```

Cancels a blocking call whose `onBlocked` callback has already fired.
Returns `VALKEYMODULE_OK` if the abort succeeded (neither
`onRespAvailable` nor any other reply callback will be invoked), or
`VALKEYMODULE_ERR` if the call has already completed.

---

### `ValkeyModule_ReplyRaw`

```c
int ValkeyModule_ReplyRaw(ValkeyModuleCtx *ctx, const char *proto, size_t proto_len);
```

Appends raw RESP bytes directly to the calling client's output buffer,
with no parsing or re-serialization. Intended for use inside an
`onRespAvailable` callback to implement zero-copy reply forwarding.

---

## Usage Example: `ValkeyModule_Call` vs `ValkeyModule_CallArgv`

The following two implementations of a generic pass-through proxy
command are functionally equivalent, but use different code paths
internally.

### With `ValkeyModule_Call` (existing API)

```c
int ProxyCommand(ValkeyModuleCtx *ctx, ValkeyModuleString **argv, int argc) {
    if (argc < 2) return ValkeyModule_WrongArity(ctx);

    /* VM_Call builds a new argv array internally, creates a string object for
     * the command name, and calls incrRefCount/decrRefCount on every argument.
     * The reply is parsed into a CallReply tree, then re-serialized to RESP. */
    ValkeyModuleCallReply *reply =
        ValkeyModule_Call(ctx, ValkeyModule_StringPtrLen(argv[1], NULL), "v",
                          argv + 2, (size_t)(argc - 2));

    return ValkeyModule_ReplyWithCallReply(ctx, reply);
}
```

### With `ValkeyModule_CallArgv` and `onRespAvailable` (new API —
pass-through)

```c
static int forwardRawReply(void *ctx, ValkeyModuleCtx *mctx,
                           const char *proto, size_t proto_len) {
    (void)ctx;
    /* Write the raw RESP bytes directly to the outer client — no parse, no
     * re-serialize, no intermediate allocation. */
    ValkeyModule_ReplyRaw(mctx, proto, proto_len);
    return 0; /* stop — do not invoke per-type callbacks */
}

static ValkeyModuleReplyHandlers passthrough_handlers = {
    .onRespAvailable = forwardRawReply,
};

int ProxyCommand(ValkeyModuleCtx *ctx, ValkeyModuleString **argv, int argc) {
    if (argc < 2) return ValkeyModule_WrongArity(ctx);

    /* VM_CallArgv borrows argv directly (no copy, no refcount changes).
     * The reply is written straight into the outer client's output buffer
     * through the onRespAvailable callback above. */
    if (ValkeyModule_CallArgv(ctx, argv + 1, argc - 1,
                              VALKEYMODULE_CALL_ARGV_RESP_AUTO,
                              &passthrough_handlers,
                              NULL) == VALKEYMODULE_ERR) {
        return ValkeyModule_ReplyWithError(ctx, "ERR command failed");
    }
    return VALKEYMODULE_OK;
}
```

Use `VALKEYMODULE_CALL_ARGV_RESP_AUTO` to ensure the inner command
replies in the same protocol version as the outer client (RESP2 or
RESP3), so the raw bytes can be forwarded without modification.

---

## Internal changes

### `argv_borrowed` client flag

A new `argv_borrowed` bit is added to the client flags struct. When
`VM_CallArgv` runs the inner command on a temporary client, it sets this
flag to signal that the temp client does not own the argv array.
`freeClientArgv` and `freeClientOriginalArgv` respect the flag and skip
freeing. `resetClient` clears it unconditionally after each command.
Several string commands (`SET`, `MSET`, `APPEND`, …) that previously
encoded or reused argv objects in-place are guarded so they do not
mutate argv elements they do not own.

A debug-only assertion (enabled when `enable-debug-assert yes` is set in
the config) verifies at runtime that no command modifies the argv array
or decrements any element's ref-count when `argv_borrowed` is set.

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
Signed-off-by: Ricardo Dias <rjd15372@gmail.com>
2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 7661f53622 Add cluster-config-save-behavior option to control nodes.conf save behavior (#3372)
This commit introduces a new configuration option `cluster-config-save-behavior`
that controls how the cluster handles nodes.conf file save failures.

The option supports two modes:
- `sync` (default): Synchronously save the config file. If the save fails,
  the process exits. This maintains backward compatibility with the old
  behavior (before 9.1).
- `best-effort`: Synchronously save the config file. If the save fails,
  only log a warning and continue running. This allows the node to survive
  disk failures (e.g., disk full, read-only filesystem) without exitting,
  giving administrators time to address the issue.

Note that this modifies the behavior of #1032, whereas #1032 was "best-effort",
we have now introduced a configuration option that defaults to "sync."
See #1032 discussion for more details.

Background:
When a disk becomes read-only or full, any cluster metadata change would
trigger a nodes.conf save attempt. With the old behavior, the node would
immediately exit via clusterSaveConfigOrDie(), potentially causing multiple
nodes on the same machine to crash simultaneously, leading to cluster
unavailability.

The new `best-effort` mode addresses this by allowing nodes to continue
operating even when disk writes fail. This is particularly useful in cloud
environments where disk failures are more common due to scale.

Note: Startup-time config saves (in clusterInit and verifyClusterConfigWithData)
still use clusterSaveConfigOrDie() since disk issues at startup should cause
immediate failure.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-04-27 13:35:11 -07:00
Ricardo DiasandMadelyn Olson 3cbf0abb9d Defer argument redaction by storing indices instead of rewriting argv (#3471)
Previously, `redactClientCommandArgument()` forced a full copy of
`c->argv` into `original_argv` immediately, then replaced the sensitive
slot with `shared.redacted`. This meant any call to redact an argument
triggered an eager argv backup even when no rewriting was needed.

The new approach stores the indices of arguments to be redacted in a
small dynamic array (`redact_args`) on the client struct. Redaction is
applied lazily when the argv is actually consumed (e.g. by the command
log), avoiding the unnecessary argv copy on the hot path.

This change is required for the implementation of
`ValkeyModule_CallArgv` module API function (#3122) , because CallArgv
owns the `argv` array, which can't be modified by the redact function.

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2026-04-27 13:35:11 -07:00
Roshan KhatriandMadelyn Olson 34ce284e3d Increase timeouts in faster-failover test for slow CI runners (#3463)
Two changes to tests/unit/cluster/faster-failover.tcl:

1. FAIL detection timeout: `wait_for_condition 1000 10` → `1000 50`
   (10s → 50s)
2. psync_max_retries: 1200 → 2400 normal (120s → 240s),
   6000 → 12000 valgrind (600s → 1200s)

The test `The best replica can initiate an election immediately in an
automatic failover` in `tests/unit/cluster/faster-failover.tcl` has been
flaky since it was introduced on March 27, 2026 by #2227.

**Frequency:** 8 out of 15 days (Mar 27 – Apr 8), across valgrind,
sanitizer, and slow CI runners.

**Common errors:**
- `log message of "Successful partial resynchronization with primary"
  not found` (timeout waiting for psync)
- `expected pattern found in srv -N log file: *best ranked replica*`
  (timeout waiting for FAIL propagation)

The test spins up a 12-node cluster (5 primaries + 7 replicas), pauses
nodes, and waits for FAIL detection to propagate across all nodes before
failover + partial resync. 

A previous fix attempt #3424 increased the psync timeout from 50s to
120s (600s valgrind), which reduced frequency but did not eliminate it.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2026-04-27 13:35:11 -07:00
charsyamandMadelyn Olson 43f39b08d9 Fix trivial double-free issue in rdbLoadObject (#3453)
In rdbLoadObject, when sdstrynewlen is OK and hashtableAdd is OK,
but lpSafeToAdd is FAIL, field will be double-freed.

But lpSafeToAdd will only fail when len + add > LISTPACK_MAX_SAFETY_SIZE(1GB),
so it can rarely happened and trivial.

Signed-off-by: charsyam <charsyam@naver.com>
2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson f3b496193b Fix config rewrite producing negative values for unsigned memory configs (#3440)
rewriteConfigFormatMemory() and rewriteConfigBytesOption() used long long
parameters to represent memory bytes, but configs like maxmemory are stored
as unsigned long long and allow values up to ULLONG_MAX.

When the value exceeds LLONG_MAX (e.g. 9223372036854775808), it overflows to
negative when passed as long long, causing config rewrite to produce incorrect
output like "maxmemory -8589934592gb".

Change both functions to use unsigned long long, matching the actual semantics
of memory byte values. This is consistent with how numericConfigGet() already
handles MEMORY_CONFIG using ull2string().

There are some MEMORY_CONFIG are signed and can be negative:
1. maxmemory-clients is a PERCENT_CONFIG
2. slot-migration-max-failover-repl-bytes is a SIGNED_MEMORY_CONFIG (after #3443) 

None of them were affected:
```
static void numericConfigRewrite(standardConfig *config, const char *name, struct rewriteConfigState *state) {
    ...
    if (config->data.numeric.flags & PERCENT_CONFIG && value < 0) {
        rewriteConfigPercentOption(state, name, -value, config->data.numeric.default_value);
    } else if (config->data.numeric.flags & SIGNED_MEMORY_CONFIG && value < 0) {
        rewriteConfigNumericalOption(state, name, value, config->data.numeric.default_value);
    } else if (config->data.numeric.flags & MEMORY_CONFIG) {
        rewriteConfigBytesOption(state, name, value, config->data.numeric.default_value);
    ...
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-04-27 13:35:11 -07:00
Viktor SöderqvistandMadelyn Olson 15d6dc347f Attempt to deflake 'diskless no replicas drop during rdb pipe' (#3461)
Increase time to wait for bgsave to start. This wait has been seen
failing in these test cases.

The test case loops with 'no', 'slow', 'fast', 'all' and 'timeout'
replicas, in tests/integration/replication.tcl

Example:

*** [err]: diskless fast replicas drop during rdb pipe in
tests/integration/replication.tcl
    rdb child didn't terminate

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-04-27 13:35:11 -07:00
110dc03968 Fix slot-migration-max-failover-repl-bytes unable to accept -1 (#3443)
In valkey.conf, slot-migration-max-failover-repl-bytes allows setting
to -1 to disable the limit.
```
Setting this to -1 will disable this limit
```

But slot-migration-max-failover-repl-bytes is defined as MEMORY_CONFIG
and memtoull() rejects negative inputs, making it impossible to set the
value to -1 via config file or CONFIG SET.
```
>>> 'slot-migration-max-failover-repl-bytes "-1"'
argument must be a memory value
```

Introduce SIGNED_MEMORY_CONFIG flag for memory configs that also accept
plain negative number. When memtoull() fails and this flag is set, fall back to string2ll()
for parsing. Use ll2string() for CONFIG GET and rewriteConfigNumericalOption() for
CONFIG REWRITE when the value is negative.

Add a serverAssert in initConfigValues() to enforce that PERCENT_CONFIG
and SIGNED_MEMORY_CONFIG are never combined on the same config, since
both use negative values with different semantics.

This means we have had this issue since it was introduced in #1949.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-04-27 13:35:11 -07:00
RaghavandMadelyn Olson fe11fcb2a8 Skip faster-failover test under TLS (#3444)
The faster-failover.tcl test starts 12 cluster nodes 5 times with TLS,
which is CPU intensive due to the volume of TLS handshakes. Running
alongside other tests on CI runners causes random I/O timeouts in
unrelated tests.

Tagging it as `tls:skip`

Signed-off-by: nmvk <r@nmvk.com>
2026-04-27 13:35:11 -07:00
Viktor SöderqvistandMadelyn Olson 99333f9a69 Fix some flaky tests using CLIENT REPLY OFF (#3452)
Some test cases write thoughsands of commands in a pipeline and
afterwards read the replies. This can lead to TCP ACK being dropped and
the connection broken. CLIENT REPLY OFF prevents this.

"Main db not affected when fail to diskless load" in
cluster/diskless-load-swapdb has been observed to be flaky.

The others are just defensive but they follow the same pattern.

Similar fixes in the past: #3430, #2483.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-04-27 13:35:11 -07:00
27f23b4d0e Fix RDB expiry write length and leak when loading zipmap (#3422)
Fix two things:

1. Incorrect RDB object size reported when writing hash fields with
expiration, because of wrong position of parentheses. Affects the
serialized size reported in DEBUG OBJECT.

2. Memory leak when loading a zipmap (only used in really old RDB
versions).

Signed-off-by: charsyam <charsyam@naver.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
2026-04-27 13:35:11 -07:00
Nikhil MangloreandMadelyn Olson 8a909a06e6 Increase embedded string threshold from 64 to 128 bytes (#3397)
This PR increases the embedded string threshold to 128 bytes (2 cache
lines) in order to improve memory overhead. I also fixed the tests that
pertained to embedded values.

Closes #3025

---------

Signed-off-by: Nikhil Manglore <nmanglor@amazon.com>
2026-04-27 13:35:11 -07:00
Lucas YangandMadelyn Olson 6c0bf7c539 Update maintainer affiliation (#3449)
Update Lucas Affiliation to Percona

Signed-off-by: Lucas Yang <lucasyonge@gmail.com>
2026-04-27 13:35:11 -07:00
24a0721daf Grammar, spelling, and punctuation fixes across the summary fields for Valkey commands. (#3309)
I was fixing grammar issues in the valkey-swift client
(https://github.com/valkey-io/valkey-swift/pull/357), and would like to
apply the same fixes - missing words, full sentence punctuation, and
imperative verb forms - to the breadth of the commands.

---------

Signed-off-by: Joe Heck <j_heck@apple.com>
Signed-off-by: Joseph Heck <j_heck@apple.com>
Co-authored-by: Sarthak Aggarwal <sarthakaggarwal97@gmail.com>
Co-authored-by: Lucas Yang <lucasyonge@gmail.com>
2026-04-27 13:35:11 -07:00
Ran ShidlansikandMadelyn Olson 04cf1f8550 Avoid having server.h being included by cli and benchmark (#3420)
valkey-cli and valkey-benchmark link only a small subset of .o files and
do not include cluster_migrateslots.o. At -O3 this works because the
compiler either inlines or discards the unused static function —
the symbol reference never reaches the linker. At -O0 (no inlining, no
dead-code elimination), the compiler emits the full body of
getClientType into every .o that includes server.h, producing an
unresolved
reference to _isImportSlotMigrationJob at link time.

fix is to avoid including server.h as part of external application
compilation dependency

fixes: https://github.com/valkey-io/valkey/issues/3415

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2026-04-27 13:35:11 -07:00
Madelyn Olson f94c1a7edf Fix VLA warning in linenoise and enable -Werror (#3439)
Replace 'const int seqBufferMaxLength' with a #define to avoid a
variable-length array warning (-Wgnu-folding-constant) in C.

Also add -Werror to the linenoise Makefile so future warnings are caught
at build time.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2026-04-27 13:35:11 -07:00
da8d047a9b Big endian bitmap byte order mismatch fix (#3401)
Found while implementing `clusterNodeGetSlotRangeEnd` for `CLUSTERSCAN`
range bounded scanning, which uses the simlilar approach.

When tested on s390x via Docker and QEMU. Slot boundary came as 5440
instead of 5461.

```
127.0.0.1:6001> clusterscan 0-{06S}-0
1) "0-{4HD}-0"
2) 1) "{06S}key1"
127.0.0.1:6001> keyslot 0-{4HD}-0
(error) ERR unknown command 'keyslot', with args beginning with: '0-{4HD}-0'
127.0.0.1:6001> cluster keyslot 0-{4HD}-0
(integer) 5440
127.0.0.1:6001> cluster nodes
08e28d7e8dcfc731ac537d0518bfa32577da6ec7 127.0.0.1:6002@16002 master - 0 1774415944347 2 connected 5461-10922
53f6d4e61eee13ea98441eec05b3c4c95c6c83a4 127.0.0.1:6003@16003 master - 0 1774415943314 3 connected 10923-16383
46664ac624f001cc0708e8ddcfcc9e45e8f444a0 127.0.0.1:6001@16001 myself,master - 0 0 1 connected 0-5460
```

Updating here as it is same approach and does not follow `memrev64ifbe`
followed by `memcpy`

Signed-off-by: nmvk <r@nmvk.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2026-04-27 13:35:11 -07:00
Viktor SöderqvistandMadelyn Olson 1ddaefbb93 Fix some flaky tests (#3430)
Fixing multiple flaky tests.

    slave buffer are counted correctly in tests/unit/maxmemory.tcl
    Memory efficiency with values in range * in tests/unit/memefficiency.tcl

These tests send large numbers of pipelined commands using deferring
clients without reading replies, causing the server's client output
buffer to grow. On slow CI runners, this leads to TCP backpressure and
I/O errors that crash the test runner. Fix: Use CLIENT REPLY OFF to
suppress reply generation, matching the pattern from commit 972f94953c.

---

    Sub-replica reports zero repl offset and rank, and fails to win election
    in tests/unit/cluster/replica-migration.tcl
    New non-empty replica reports zero repl offset and rank, and fails to
    win election in tests/unit/cluster/replica-migration.tcl

In the replica-migration tests, a MOVED errors results in an Tcl
exception. After failover, wait_for_condition blocks issue GET commands
to cluster nodes that may not have fully updated their slot routing. An
unhandled MOVED exception crashes the test runner. Fix: Wrap the
condition in catch so MOVED errors are retried. Also wrap debug prints
in the else clause. Fixes the following tests:

---

    Replica can update the config epoch when trigger the failover -
    automatic in tests/unit/cluster/failover2.tcl

Increase wait timeout for failover expiry. The test waits 10 seconds for
"Failover attempt expired", but the default cluster-node-timeout in
start_cluster is 3000ms, making auth_timeout 6 seconds plus ~3
seconds for failure detection — barely fitting in 10 seconds and failing
on slow CI runners. Fix: Increase wait from 1000×10ms to 1200×50ms
(60 seconds).

---

    dual-channel-replication lazyfree test

The test looks up the replica's main-channel connection id after writing
50MB of data. On slow CI runners, the replica connection may have been
disconnected by the output buffer soft limit (64MB/60s) before the
lookup, causing get_client_id_by_last_cmd to return empty. Two changes:

1. Move the connection id lookup before the write loop, while the sync
   is known to be in progress.
2. Reduce writes from 50 x 1MB to 10 x 1MB. The test only needs enough
   data to exceed the lazyfree threshold (64 blocks ~= 1MB). 10MB is
   sufficient and avoids approaching the output buffer limit.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-04-27 13:35:11 -07:00
d7d85ea831 Enhance cluster stale packet detection to prevent sub-replica and empty primary (#2811)
## Summary
This PR handles a network race condition that would cause a replica to read stale cluster
packet and then incorrectly promote itself to an empty primary within an existing shard.

## Issue
"Migrated replica reports zero repl offset and rank, and fails to win election - sigstop"
the test case in `replica-migration.tcl` is a known example where the network race condition
can occur.

Here's the timeline:
- T1: Slot migration — R7 and R3 both try to replicate from R0. Only R3 succeeds, triggering
  a BGSAVE on R0, while R7 blocks in `receiveSynchronousResponse()`.
- T2: While R7 is blocked, R0 is SIGSTOP'd by the test. R4 wins the election and becomes the
  new primary. R4 sends PINGs to R7 - These PINGs land in R7's kernel TCP receive buffer but
  are not read by R7 yet.
- T3 (5s after T1, due to receiveSynchronousResponse): R7 wakes up:
  - It reads from inbound links and finds out the remote nodes have already closed their end
    (they detected R7 as FAIL), getting "I/O error: connection closed" on each. R7 calls `accept()`
    for the new connections that were established during the block. These connections carry data
    that was sent seconds ago while R7 was dead.
  - R7's outbound links are still valid, so it sends PING to R4.
- T4: R7 receives and processes fresh PONG packet from R4 on outbound link, reconfiguring itself
  to follow R4.
- T5: R7 reads stale PING packet of R4 via inbound link. This packet was generated R4 was following
  R0, so R7 incorrectly believes R4 is still following R0 now. And Reconfiguring itself as a replica
  of R0 from R4.
- T6: R7 finds R0 is FAIL, so it starts an election and wins, and becomes an empty-primary.

## Analyze
So in T4, R4 is the new primary, and R7 is reconfiguring itself as a replica. So in R7's view,
R4 is the primary, and myself (R7) is a replica.

And in T5, there is a stale packet from sender (R4), the stale packet is saying: sender (R4) is
a replica and R0 is the primary.

We originally had a logic for stale packet, meaning we would try to ignore stale packet that
would cause exceptions. So in T5:
- sender_claims_to_be_primary is false since R4 is saying it is a replica.
- sender_last_reported_as_primary is true since in R7's view, R4 is a primary.
- sender_claimed_primary is R0, and sender (R4) and sender_claimed_primary (R0) is in the same shard.
- nodeEpoch(sender_claimed_primary) is R0's epoch. R0 is an old and dead (not yet) primary.
- sender_claimed_config_epoch is R0's epoch since R4 is a replica, and R0 is R4's primary.
- nodeEpoch(sender_claimed_primary) == sender_claimed_config_epoch, so the logic fail and we process
  a stale packet. So in this point, the packet is not a stale packet in R4 and R0's view.
- But it is a stale packet in myself (R7) view. In R7's local view, R4 is the new primary and it
  should have a bigger epoch, that is nodeEpoch(sender) should > sender_claimed_config_epoch.

## Fix
The PR fixes the issue by enhancing the existing guardrail logic against stale packet. Previously
that logic only detects `nodeEpoch(sender_claimed_primary) > sender_claimed_config_epoch` as stale
packet, now it also checks `nodeEpoch(sender) > sender_claimed_config_epoch` to make sure we have
up-to-date primary-replica chain.

Signed-off-by: Zhijun <dszhijun@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2026-04-27 13:35:11 -07:00
Viktor SöderqvistandMadelyn Olson 3c9b27b476 Increase timeout in flaky "failover immediately" test case (#3424)
The test case "The best replica can initiate an election immediately
test" has been failing in CI jobs.

Increase the timeout to account for slow runners.

Old waiting time: 50 seconds. New waiting time: 120 seconds, with
valgrind: 600 seconds.

Intoduced in #2227.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-04-27 13:35:11 -07:00
Viktor SöderqvistandMadelyn Olson 5b25c93731 Reduce corrupt-dump-fuzzer tests from 10 to 1 minute (#3425)
In Daily test runs with the `--accurate` flag, the corrupt-dump-fuzzer
test runs for 10 minutes (600 seconds) with "sanitize_dump: no" and then
another 10 minutes with "sanitize_dump: yes". This causes the runner to
time out the whole test job to be aborted with a sigterm from the
runner.

Example:

13697:signal-handler (1774917673) Received SIGTERM scheduling
shutdown...

This change reduces this hard-coded fuzzer run from 10 to 1 minute. We
have many tests jobs so the fuzzer gets plenty of time to run anyway.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson d598dbb991 Optimize WATCH duplicate key check from O(N) to O(1) using per-db hashtable (#3360)
Previously, watchForKey() checked for duplicate watched keys by iterating
through the client's entire watched_keys list with O(N) complexity, where
N is the total number of keys watched by the client. So the time complexity
for the WATCH command could be quite poor and become a slow command.

This commit introduces a per-db hashtable (watched_keys_by_db) in the
client's multiState structure to enable O(1) duplicate key detection.
The hashtable is lazily allocated only when the client starts watching
keys, minimizing memory overhead for clients that don't use WATCH.

The per-db hashtable stores watchedKey* directly as the hashtable entry
since it already contains the key, so no custom destructors are needed.
Memory management remains centralized in the watched_keys list.

This optimization is especially beneficial when a client watches many
keys across different databases, as the check no longer scales with
the total watched key count.

This might be a minor scenario, but there's no harm in optimizing it.

There is a test in multi.tcl, before this patch, it took 15s, and after
this patch, it only took 50ms.
```
        set elements {}
        for {set i 0} {$i < 50000} {incr i} {
            lappend elements key-$i
        }
        r watch {*}$elements
        r watch {*}$elements
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 84240f51c6 Handle EAGAIN in clusterWriteHandler (#3421)
To avoid freeing the cluster link when EAGAIN occurs,
so that we can try again and keeping the send messages.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-04-27 13:35:11 -07:00
Ran ShidlansikandMadelyn Olson ab7498e020 fix test_entry to consider diffrerent allocator size classes (#3416)
fixes: https://github.com/valkey-io/valkey/issues/3200

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2026-04-27 13:35:11 -07:00
76b4b4b47e Improve listpack threshold guidance in valkey.conf (#3419)
Fixes #3299

Add brief guidance in valkey.conf explaining what the listpack
thresholds control and the memory/CPU tradeoff when tuning them.

---------

Signed-off-by: Tarte <emprimula@gmail.com>
Signed-off-by: KimHuiSu <101166683+Tarte12@users.noreply.github.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-04-27 13:35:11 -07:00
zhenwei piandMadelyn Olson 3b1ae0afd0 Show uname -a in RDMA CI job (#3418)
The RXE project should keep the same version with the CI machine,
showing uname in RDMA CI job to find out the reason of kmod installing
failure.

Signed-off-by: zhenwei pi <zhenwei.pi@linux.dev>
2026-04-27 13:35:11 -07:00
Nikhil MangloreandMadelyn Olson 2c5a748ba3 Fix race condition in diskless swapdb RedisModuleEvent_ReplAsyncLoad tests (#3404)
These failures seem to be attributed to a race condition in the Aborted
test case.

`rdb-key-save-delay` 10000 was being set after `$master exec` triggered
the full resync. Since repl-diskless-sync-delay 0 was set, the master
would immediately start streaming the RDB to the replica once it
reconnected. On the ARM runner, when it's fast enough, the entire RDB
generation and transfer could complete in ~78ms, before the delay was
ever applied. This meant the replica would complete the swap and have
1010 keys instead of the expected 200 and there would be no
async_loading window to observe or abort which led to the failures.

We saw this in the daily test failure logs
```
92948:S * RDB memory usage when created 110.85 Mb
92948:S * Done loading RDB, keys loaded: 1010, keys expired: 0
```

The fix moves `rdb-key-save-delay 10000` to before `$master exec` to
guarantee the delay is in effect on the master before the RDB generation
begins.

Closes #3394, closes #3395.

Signed-off-by: Nikhil Manglore <nmanglor@amazon.com>
2026-04-27 13:35:11 -07:00
3d80bf6e6c Do the failover immediately if the replica is the best ranked replica (#2227)
In #2023 (#2209, etc.), we are exploring ways to make failover faster,
that is, to minimize the delay.

When a node is marked as FAIL and before the failover starts, there is a
delay of 500-1000ms. The original purpose of this delay:

1. Allow FAIL to propagate to at least a majority of the primaries. This
   makes sure they will vote when a replica sends failover auth request.
2. Allow replicas to exchange their offsets, so they will have a correct
   view of their own rank.

We want to minimize this delay while ensuring safety. It is useful for
example in these cases:

1. If there is only one replica, then we don't need any delay, or
2. If there are more replicas, with a fast network, the replicas can
   exchange the offsets very quickly and start the failover within a few
   milliseconds instead of 500-1000ms.

In this PR, when we can be sure that the replica is the best ranked
replica, we let
it initiate a failover immediately and completely remove the delay.

### How to ensure safety?

1. To make sure this replica has the best rank, it only skips the delay
   if it is sure that it have the best rank and that all replicas in the
   same shard agree that the primary is failing. A new flag
   `CLUSTER_NODE_MY_PRIMARY_FAIL` is introduced to indicate that each
   replica has marked its primary as FAIL. If all replicas say that the
   primary is failing, we also know that the offset is not updated,
   because the offset is not incrementing when the primary is failing.
   We can skip the delay only if we have received a message from all
   replicas and they all have set this flag.

2. To make sure the primaries will vote even if they didn't receive the
   FAIL yet, we use the `CLUSTERMSG_FLAG0_FORCEACK` to make sure they
   will vote. This is equivalent to broadcasting a FAIL message to all
   primaries before we broadcast the failover auth request (but
   cheaper).

The race between FAIL (broadcast by A) and AUTH REQUEST (broadcast by R)
is illustrated in the following sequence diagram:

```
A      R        B      C
|      |        |      |
| FAIL |        |      |
|----->| AUTH R.|      |
|      |------->|      |
| FAIL |        |      |
|-------------->|      |
|      | AUTH R.|      |
|      |-------------->|
| FAIL |        |      |
|--------------------->|
```

### Details

This is the how the failover is initiated, with new steps marked with
**(new)**:

1. A majority of primaries have marked another primary as PFAIL.

2. Some nodes counts failure reports and marks the failing primary as
   FAIL. The node
   that detects FAIL broadcasts it to all nodes in the cluster.

3. When a replica receives FAIL (or detects FAIL itself by counting
   PFAIL reports) it schedules a failover:
   
   a. It sets a timeout (500ms + random 0-500ms).
   b. It broadcasts pong to the other replicas in the same shard.   
   c. **(new)** The pong (actually the clusterMsg header) has a new flag
      `CLUSTER_NODE_MY_PRIMARY_FAIL`. When the replicas broadcast pong
      to each other here, this flag is set.

4. **(new)** When the following conditions are met, skip the remaining
   delay and start the failover using AUTH REQUEST with the FORCE ACK
   flag set, that is if
   
   a. a PONG is received from every other replica in the same shard
      (broadcast within the shard) and
   b. all replicas have marked that its primary is FAIL in their last
      message (the new `CLUSTER_NODE_MY_PRIMARY_FAIL` flag is set) and
   c. this is the best replica (rank = 0) and
   d. my replication offset != 0.

5. When the delay has passed and no other replica has initiated
   failover, then initiate failover.

Notes:

* With 3(c), we don't need to wait for FAIL to propagate to all voting
  primaries. At this point, a FAIL has already been broadcast by some
  node, but there is a race so our auth request may arrive to some node
  before the FAIL. Using the FORCE ACK flag ensures the primaries will
  vote for us. (It is equivalent to broacasting another FAIL just before
  broadcasting auth request.)
  
* 4(b) ensures that we have received the replication offset from all
  other replicas and that it's up to date. If a replica says that it's
  primary is failing, it also means that the replication from the
  primary to that replica has stopped.

* 4(c) is to avoid a special bad case. It can happen that not all
  replicas know about each other. In this case, two replicas can think
  they are both the best replica and start the failover at the same
  time. This can already happen without this PR. When it happens, it
  usually means that a new replica has just joined and it has no data
  (offset = 0) and if it wins the election, there is a problem of
  dataloss (discussed and partially mitigated for the replica migration
  case in #885). To avoid this case, skip this fast failover path if the
  replica has offset = 0.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-04-27 13:35:11 -07:00
75bb8bd3e4 ARM NEON SIMD optimization for pvFind() in vset.c (#3033)
**Title:** ARM NEON SIMD optimization for pvFind() in vset.c

**Description:**

This PR resolves #2806. Thanks to @ranshid for guidance on testing
methodology and workload design.

### Summary

This PR adds ARM64 NEON SIMD optimization to the `pvFind()` function in
vset.c, which performs linear pointer search in pVector. The pVector is
used internally by vset to track expired fields in hash objects (HFE).
The optimization processes 4 pointers per iteration using 128-bit NEON
vector instructions.


### Implementation Details

- Added `pvFindSIMD_NEON64()` static inline helper function using NEON
intrinsics
- Modified `pvFind()` to use SIMD path when `len >= 8` on ARM64
- Added `#include <arm_neon.h>` guarded by `HAVE_ARM_NEON`
- No changes to function signatures or external behavior
- Scalar fallback remains for non-ARM64 platforms and small vectors

### Benchmark Results

#### Micro benchmark (Apple M4 Pro, 50M iterations)

Scalar version:
```
len16_mid              | len= 16 pos=  8 |    2.8 ns
len16_last             | len= 16 pos= 15 |    5.0 ns
len32_mid              | len= 32 pos= 16 |    5.1 ns
len64_last             | len= 64 pos= 63 |   15.7 ns
len127_last            | len=127 pos=126 |   34.0 ns
len127_notfound        | len=127 pos= -1 |   33.8 ns
```

NEON version:
```
len16_mid              | len= 16 pos=  8 |    2.0 ns | 1.42x
len16_last             | len= 16 pos= 15 |    2.7 ns | 1.85x
len32_mid              | len= 32 pos= 16 |    2.8 ns | 1.79x
len64_last             | len= 64 pos= 63 |    9.2 ns | 1.71x
len127_last            | len=127 pos=126 |   18.1 ns | 1.88x
len127_notfound        | len=127 pos= -1 |   17.8 ns | 1.90x
```

#### Production HFE benchmark (stress profile, 120s duration)

Workload profile:
- 5000 hashes with 64-127 fields each
- Short TTLs (2-10 seconds) to trigger expiration
- 30% deletes, 60% updates (both trigger pvFind)
- 4 concurrent threads

```
Platform        Mode      Avg Time    Throughput   Speedup
-----------     ------    --------    ----------   -------
Apple M4        scalar     94.3 ns   10.61 M/s     baseline
Apple M4        NEON       74.4 ns   13.44 M/s     1.27x (21% faster)
```

SIMD utilization (M4 NEON):
```
- 100% of calls used SIMD path
- 99.3% of elements scanned via SIMD
- 74.7% of matches found in SIMD section
- 25.3% found in scalar tail (last 0-3 elements)
```



### Platform Support

- **ARM64 (aarch64)**: SIMD enabled via `HAVE_ARM_NEON`
- **x86_64 / other**: Falls back to scalar implementation, no behavior
change

---------

Signed-off-by: Ahmad Belbeisi <ahmadbelb@gmail.com>
Signed-off-by: Ahmad Belbeisi <ahmad.belbeisi@tum.de>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2026-04-27 13:35:11 -07:00
Alina LiuandMadelyn Olson 178e9e99d2 Fix EntryTest.entryUpdate failure on macOS due to allocator differences (#3398)
## Problem

The `EntryTest.entryUpdate` unit test fails on macOS with (mentioned in
#3200):

Expected: (entryMemUsage(e10)) < (current_embedded_allocation_size * 3 /
4)
      actual: 48 vs 48

## Root Cause

`entryMemUsage` for embedded entries reflects the actual zmalloc
allocation size, which depends on the platform allocator's bucket sizes.

Valkey's bundled jemalloc is configured with `LG_QUANTUM=3` (8-byte
granularity), giving size classes: 8, 16, 24, 32, 40, 48, 56, 64, ...
However, macOS libc uses 16-byte aligned buckets: 16, 32, 48, 64, 80,
...

The test's value10 (21 chars) produces an entryReqSize of 40 bytes.
Jemalloc has a 40-byte size class, so entryMemUsage returns 40. macOS
rounds up to 48, which equals 3/4 of e9's 64-byte allocation, causing
the strict less-than assertion to fail.

## Fix

Shrink value10 from 21 to 13 characters, reducing entryReqSize from 40
to 32 bytes. Both allocators have a 32-byte bucket, and 32 < 48 holds on
both platforms.

## Test

All tests pass on macOS.

Signed-off-by: Alina Liu <liusalisa6363@gmail.com>
2026-04-27 13:35:11 -07:00
harrylin98andMadelyn Olson 1de84d9d4f ci: include gtests in code coverage report
Signed-off-by: harrylin98 <harrylin980107@gmail.com>
2026-04-27 13:35:11 -07:00
Yang ZhaoandMadelyn Olson 8d062cc469 Fix valkey-cli --cluster del-node for unreachable nodes (#3209)
The `valkey-cli --cluster del-node` command fails when attempting to
delete unreachable or failed nodes, reporting `No such node ID` even
though the node exists in the cluster topology.
The root cause is the command only loads information about reachable
nodes, causing the lookup to fail. This PR added a new function for
loading all nodes information to solve this.


### Implementation

1. Loading all nodes from gossip:
- Added `clusterManagerLoadAllInfoFromNode()` that loads both reachable
and unreachable nodes from cluster gossip
- Extracts common logic into `clusterManagerLoadInfoCommon()` with an
`include_unreachable` flag
- Keeps the original `clusterManagerLoadInfoFromNode()` unchanged to
avoid affecting existing callers
2. Added success message to be consistent with other cluster commands:
`[OK] Node <id> removed from the cluster.`
3. Added test coverage for `del-node` which previously had none.
4. Load slot information from gossip for unreachable nodes in
`clusterManagerNodeLoadInfo()`
5. Skip unreachable primaries in `clusterManagerNodeWithLeastReplicas()`


### Testing
```
./runtest --single unit/cluster/cli

[ok]: del-node: Cannot delete node with slots (9 ms)
[ok]: del-node: Delete reachable node without slots (23 ms)
[ok]: del-node: Delete unreachable node without slots (1333 ms)
[ok]: del-node: Cannot delete unreachable primary with slots (3368 ms)
```
```
valkey-cli --cluster del-node 127.0.0.1:7000 eb837ea7c48908e5304eafd8b1b3ced57147c448

Could not connect to Valkey at 127.0.0.1:7002: Connection refused
>>> Removing node eb837ea7c48908e5304eafd8b1b3ced57147c448 from cluster 127.0.0.1:7000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> WARNING: Could not connect to node 127.0.0.1:7002, unable to send CLUSTER RESET.
[OK] Node eb837ea7c48908e5304eafd8b1b3ced57147c448 removed from the cluster.
```

### Behavior change

Before
```
$ valkey-cli --cluster del-node <entry-node-ip>:<entry-node-port> <failed-node-id>
Could not connect to Valkey at <target-node-ip>:<target-node-port>: Connection refused
>>> Removing node <id> from cluster <entry-node-ip>:<entry-node-port>
[ERR] No such node ID <id>
```
After
```
$ valkey-cli --cluster del-node <entry-node-ip>:<entry-node-port> <failed-node-id>
Could not connect to Valkey at <target-node-ip>:<target-node-port>: Connection refused
>>> Removing node <id> from cluster <entry-node-ip>:<entry-node-port>
>>> Sending CLUSTER FORGET messages to the cluster...
>>> WARNING: Could not connect to node <target-node-ip>:<target-node-port>, unable to send CLUSTER RESET.
[OK] Node <id> removed from the cluster.
```

### Related Issue
Fixes https://github.com/valkey-io/valkey/issues/3208

---------

Signed-off-by: Yang Zhao <zymy701@gmail.com>
2026-04-27 13:35:11 -07:00
Roshan KhatriandMadelyn Olson b454c95776 Upload all benchmark artifacts including server logs (#3388)
Upload the entire results directory instead of only metrics JSON files.
This includes server logs which are useful for debugging benchmark
failures.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2026-04-27 13:35:11 -07:00
Roshan KhatriandMadelyn Olson 2eadce4ff5 Pin workflow pip/go/npm dependencies for OpenSSF compliance (#3276)
Pin package manager dependencies in CI workflows to improve the Pinned-Dependencies
score in OpenSSF Scorecard.

Changes:
- benchmark-on-label.yml, benchmark-release.yml: add `--require-hashes`
  to `pip install` adding on valkey-perf-benchmark repo:
  https://github.com/valkey-io/valkey-perf-benchmark/pull/44
- ci.yml: pin `yamlfmt` to `v0.21.0` instead of `@latest`
- reply-schemas-linter.yml: use npm ci with `package-lock.json` instead
  of unpinned npm install, package files in `utils/reply-schema-linter/`

Signed-off-by: Roshaan Khatri <rvkhatri@amazon.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2026-04-27 13:35:11 -07:00
Roshan KhatriandMadelyn Olson 218ad732a0 fix benchmark queue and reduce the total duration (#3387)
Previously, our workflow used a global concurrency group, which
effectively limited execution to one running job and one pending job.
Any additional requests were automatically canceled, preventing a true
queue from forming.

We are now shifting to a model where we remove the concurrency
restriction and allow jobs to queue directly on the self-hosted runner.
This enables multiple workflow runs to be accepted and queued instead of
being dropped.

While GitHub can accept workflow triggers at a high rate (e.g., hundreds
per minute), the actual execution is still constrained by runner
capacity, in our case, a single runner processing one job at a time.

However, queued jobs are subject to GitHub’s 24-hour timeout policy.
This means any job that waits in the queue for more than 24 hours before
starting will be automatically canceled (timedout).

In practical terms, this approach improves reliability by eliminating
premature cancellations, but the effective queue size is still bounded
by how many jobs the runner can process within a 24-hour window. we
could increase the number of runners to run these in parallel.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2026-04-27 13:35:11 -07:00
RaghavandMadelyn Olson 258060ddbb CLUSTERSCAN MATCH pattern maps to a specific slot optimizations (#3380)
When a MATCH pattern maps to a specific slot, `CLUSTERSCAN` can skip
directly to that slot
instead of walking through all slots one by one.

- On `cursor 0`, starts directly at the matching slot
- If cursor is behind the matching slot, jumps forward
- If cursor is ahead of the matching slot, we conclude the scan as we
cannot match keys
- If both SLOT and MATCH are provided but target different slots,
returns 0 immediately

Signed-off-by: nmvk <r@nmvk.com>
2026-04-27 13:35:11 -07:00
Harry LinandMadelyn Olson 9142991068 Ensure the daily workflow uses gtest-parallel to run unit tests in isolation (#3375)
The daily workflow was directly invoking the `valkey-unit-gtests` executable.
The intended invocation is to use `gtest-parallel` to ensure that the tests are executed in isolation.

Signed-off-by: harrylin98 <harrylin980107@gmail.com>
2026-04-27 13:35:11 -07:00
Ran ShidlansikandMadelyn Olson 6c4bb03dc5 remove duplicated lline (#3379)
Probably added by mistake during some merge of #1566

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2026-04-27 13:35:11 -07:00
Harkrishn PatroandMadelyn Olson ca82cdde9c Add AGENTS.md file for agentic coding assistant steering (#3371)
Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
2026-04-27 13:35:11 -07:00
Jacob MurphyandMadelyn Olson 0a33b19ea2 Add design-docs folder and README. (#3300)
These are forked from the RFC instructions created by
@zuiderkwast and @hpatro in
https://github.com/valkey-io/valkey-rfc/pulls/1 and
https://github.com/valkey-io/valkey-rfc/pulls/6. It also includes the Atomic Slot Migration design to bootstrap the folder.

---------

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2026-04-27 13:35:11 -07:00
secwallandMadelyn Olson 4147dd14f8 Make scripting debug test skippable (#3368)
I'm developing a module to provide luajit as lua execution engine.
See #1229 for details.

Unfortunately said module doesn't support debugging yet. So in order to
test it with valkey tests scripting debug tests need to be skipped. This
patch makes an anonymous test skippable by name.

Signed-off-by: secwall <secwall@yandex-team.ru>
2026-04-27 13:35:11 -07:00
Harkrishn PatroandMadelyn Olson c91cd8c926 Make macOS leaks check skippable (#3370) 2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 347e1168b5 Fix incorrect memory overhead calculation for watched keys (#3359)
The multiStateMemOverhead() function was incorrectly calculating the
memory overhead for watched keys. It used sizeof(c->mstate->watched_keys)
which is the size of the list structure itself, instead of sizeof(watchedKey)
which is the actual per-key overhead.

This was introduced in #1405.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-04-27 13:35:11 -07:00
Madelyn OlsonandGitHub 3f9fc9ff69 Update version to 9.1.0-rc1 and add release notes (#3333)
Add release notes for Valkey 9.1 for review and bumping the version so we can stamp it.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2026-03-16 16:39:41 -07:00
Sarthak AggarwalandGitHub 803fc6e7c6 Weekly tests branches are not honored on scheduled workflow (#3340)
`weekly.yml` calls `daily.yml` with `use_git_ref` set to each release
branch (for example 7.2). But the checkout logic in `daily.yml` only
used `inputs.use_git_ref` when `github.event_name` was
`workflow_dispatch` or `workflow_call`. otherwise it fell back to
`github.ref`.

For reusable workflows, GitHub keeps the caller workflow’s github
context. That means when `weekly.yml` is triggered by schedule, the
called `daily.yml` still sees `github.event_name == 'schedule'` and
`github.ref` for the caller branch (unstable). As a result, jobs labeled
as release-branch runs could still check out unstable.

Added a guard for Gtest Unit Tests. It will skip the job if gtest is not
available / supported.

Run with CI Issue:
https://github.com/valkey-io/valkey/actions/runs/22815380713

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-03-13 14:50:00 -07:00
nesty92andGitHub b3ae643246 Optimize streams range hot path (#3002)
This PR improves stream performance in the range iteration and reply
generation paths, benefits xadd, xrange, xrevrange, xreadgroup.

- ull2string memcpy optimization
- streamID struct + streamCompareID
- streamID2string + reply path
- getClientType inline + cache locality

Inspired by the high-level description (not the code) of
https://github.com/redis/redis/pull/14480.

---------

Signed-off-by: Ernesto Alejandro Santana Hidalgo <ernesto.alejandrosantana@gmail.com>
2026-03-13 16:33:49 +01:00
Rain ValentineandGitHub 5a2509b743 Fix OOM aborts in large-memory ASAN tests on GitHub runners (#3263)
Carries on from where #3161 left off. The test-sanitizer-address-large-memory
jobs were being OOM-killed on GitHub-hosted runners (15.6GB RAM) due to
ASAN's 2-3x memory overhead.

Changes:
- Skip 4GB quicklist compression test under ASAN (requires ~16-24GB with
dual buffers + ASAN overhead)
- Reduce integration test sizes from 5GB to 4.1GB (preserves >4GB 32-bit
boundary coverage)
- Reduce XADD iterations from 10 to 3
- Add memory monitoring to track minimum free memory during CI runs

Signed-off-by: Rain Valentine <rsg000@gmail.com>
2026-03-12 12:33:33 +08:00
Rain ValentineandGitHub dc31c5f91a [DEFLAKE] Deflake replica selection test by relaxing cluster configurations (#3261)
The "New Master down consecutively" test was sometimes failing under
Valgrind by timing out. The new overrides match those used for the
cluster in the first part of the file - see #2672

Under Valgrind's 10-20x slowdown, a single failover requiring ~15
seconds of server time can exceed the test's 100-second wall-clock wait.

Error text:
```
*** [err]: New Master down consecutively in tests/unit/cluster/slave-selection.tcl
No failover detected when master 12 fails
```

Daily run failure:
https://github.com/valkey-io/valkey/actions/runs/22421982161/job/64921545936#logs

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
Signed-off-by: Rain Valentine <rainval@amazon.com>
2026-03-11 17:52:02 -07:00
Ray CaoandGitHub d9b200a02e Reuse RDB file as AOF preamble on replica after disk-based full sync (#1901)
When a replica finishes a **disk-based** full sync and both `appendonly
yes` and `aof-use-rdb-preamble yes` are enabled, this PR reuses the
received `dump.rdb` as the new AOF base file, instead of triggering
`BGREWRITEAOF`.

This avoids generating an almost identical base snapshot and removes
redundant CPU/IO work.

Before:

1. Replica receives RDB via disk-based full sync
2. Replica loads RDB into memory
3. Replica may delete synced RDB (`rdb-del-sync-files`)
4. `BGREWRITEAOF` runs and generates a new base snapshot
5. New snapshot becomes AOF base

After:

1. Replica receives RDB via disk-based full sync
2. Replica loads RDB into memory
3. Synced RDB is renamed to the new AOF base file
4. A new incremental AOF is created
5. No `BGREWRITEAOF` is needed

Tests:

- [x] Disk-based sync + preamble on: RDB reused as AOF base
- [x] New writes after sync: incr AOF works correctly
- [x] Replica restart: reused AOF loads correctly
- [x] Disk-based sync + preamble off: fallback to `BGREWRITEAOF`
- [x] Diskless sync + preamble on: fallback to `BGREWRITEAOF`
- [x] Diskless sync with stale local `dump.rdb`: stale file is not
       reused

---------

Signed-off-by: Ray Cao <zisong.cw@alibaba-inc.com>
2026-03-11 18:28:26 +01:00
BinbinandGitHub df1738b4d7 Add availability-zone to the reply schema for CLUSTER SHARDS/SLOTS (#3352)
`availability_zone` was added to the HELLO command in #1487, and
it was missing the reply schema and the test was wrongly marked
with `logreqres:skip`.

`availability-zone` was added to CLUSTER SHARDS and CLUSTER SLOTS
commands in #3156, and it was missing the reply schema.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-03-11 15:59:55 +01:00
Sarthak AggarwalandGitHub 8848a9ca9c Deflake io-threads test by generating enough work (#3354)
Fixes the failing run:

```
[err]: Force the use of IO threads and assert active IO thread usage in tests/unit/io-threads.tcl
Expected '0.000000' to be more than '0' (context: type source line 49 file /Users/runner/work/valkey/valkey/tests/unit/io-threads.tcl cmd {assert_morethan $used_active_time 0} proc ::test)
```

The change deflakes the test by generating enough I/O-thread work.

The test previously created 16 deferred clients but only sent 15 total
`INCR` commands. On fast runners, that burst could be short enough that
`used_active_time_io_thread_*` was still reported as `0.000000`, causing
the test to fail even though I/O threads had run.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-03-11 12:39:40 +01:00
ff1c26159a Add unsigned 64-bit numeric config in module API (#1546)
- Supports `VALKEYMODULE_CONFIG_UNSIGNED` and `UNSIGNED_CONFIG` for a
   wider range of configurations.
- Supported the API for `ValkeyModule_RegisterUnsignedNumericConfig`
- `string2ull` method supports the length parameter, making it more
   secure

---------

Signed-off-by: artikell <739609084@qq.com>
Signed-off-by: skyfirelee <739609084@qq.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2026-03-11 12:31:46 +01:00
Daniel LemireandGitHub eed79cb6cc Replace fast_float (C++) with ffc.h (#3329)
There is now a port of fast_float in C. So instead of having an optional
fast_float dependency, we can just use ffc instead, unconditionally.

https://github.com/kolemannix/ffc.h

It is a high quality port. The performance should be the same or
improved.

Note : I am the maintainer and main author of fast_float.

---------

Signed-off-by: Daniel Lemire <daniel@lemire.me>
2026-03-11 12:26:44 +01:00
Remi ColletandGitHub e00129eb87 Inherit LDFLAGS for TLS and RDMA modules (#3344)
With current Makefile, `LDFLAGS` are not used for modules.

This results in security options not applied.

```
$ annocheck /usr/lib64/valkey/modules/rdma.so 
annocheck: Version 12.99.
Hardened: rdma.so: FAIL: bind-now test because not linked with -Wl,-z,now 
Hardened: Rerun annocheck with --verbose to see more information on the tests.
Hardened: rdma.so: Overall: FAIL.
```

With this patch

```
$ annocheck /usr/lib64/valkey/modules/rdma.so 
annocheck: Version 12.99.
Hardened: rdma.so: PASS.
```

Signed-off-by: Remi Collet <remi@remirepo.net>
2026-03-11 11:40:15 +01:00
BinbinandGitHub 9239bf28db Optimize SET key value EX/PX/EXAT ttl to reduce calls of rewriteClientCommandVector (#3279)
If the command is in the form of "SET key value EX/PX/EXAT ttl",
then we don't need to rewrite the entire command vector.

In addition to saving the rewriting of the command vector, we also
save one command table lookup in this case ince we don't need to
lookup SET command again.

We already have many relevant test coverages in expire.tcl. This
PR does not change anything around the propagate.

Somehow like 8c03ef7d06.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-03-11 10:35:14 +08:00
BinbinandGitHub 837a48cef1 Fix timing issue in RDB abort test (#3343)
In flushall we will call killRDBChild to kill the RDB child, but
we need to wait for checkChildrenDone and waitpid to kick in, so
that we can call resetChildState to reset the state. The previous
wait_for_condition 10 100 (1s) might not be enough, changed it to
wait_for_condition 50 100 (5s).
```
[err]: Test FLUSHALL aborts bgsave in tests/integration/rdb.tcl
bgsave not aborted
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-03-11 10:34:35 +08:00
d3240a9b8d Fix ZDIFF algorithm 2 memory leak on early exit (#3342)
`zdiffAlgorithm2()` can break out early once the destination cardinality
reaches zero. In that path, a temporary SDS created by `zuiSdsFromValue()`
was left dirty and never released, because that cleanup normally happens on
the next iterator step.

This patch explicitly discards the dirty value before the early `break`.
```
==11724== 11 bytes in 1 blocks are definitely lost in loss record 36 of 1,442
==11724==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==11724==    by 0x2FA974: ztrymalloc_usable_internal (zmalloc.c:156)
==11724==    by 0x2FAAEA: zmalloc_usable (zmalloc.c:200)
==11724==    by 0x282D25: _sdsnewlen (sds.c:102)
==11724==    by 0x2830B2: sdsnewlen (sds.c:169)
==11724==    by 0x2DB537: zuiSdsFromValue (t_zset.c:2290)
==11724==    by 0x2DBE00: zdiffAlgorithm2 (t_zset.c:2502)
==11724==    by 0x2DC085: zdiff (t_zset.c:2568)
==11724==    by 0x2DD01F: zunionInterDiffGenericCommand (t_zset.c:2817)
==11724==    by 0x2DD574: zdiffCommand (t_zset.c:2898)
==11724==    by 0x29F5AE: call (server.c:3883)
==11724==    by 0x2A1598: processCommand (server.c:4569)
```

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthakaggarwal97@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-03-11 10:34:15 +08:00
bandalgomsuandGitHub cd67bc9494 Add availability-zone field support to CLUSTER SHARDS / CLUSTER SLOTS (#3156)
Implemented a way to propagate az through gossip and obtain az
information
through the CLUSTER SHARDS and CLUSTER SLOTS commands. It will only be
displayed if the node is configured with it.

Closes #3110

---------

Signed-off-by: Su Ko <rhtn1128@gmail.com>
2026-03-10 19:56:05 +01:00
BinbinandGitHub 5d27f8335a New MSETEX command allows to set an expiration time with MSET (#3121)
It's a combination of MSET and EXPIRE, which allows us to set an
expiration
time simultaneously with the MSET operation.

Syntax for the new MSETEX command:
```
/* MSETEX numkeys key value [key value ...] [NX | XX]
 *     [EX seconds | PX milliseconds |
 *      EXAT seconds-timestamp | PXAT milliseconds-timestamp | KEEPTTL] */
```

The MSETEX command supports a set of options (like SET command):
- EX seconds -- Set the specified expire time, in seconds (a positive
integer).
- PX milliseconds -- Set the specified expire time, in milliseconds (a
positive
  integer).
- EXAT timestamp-seconds -- Set the specified Unix time at which the
keys will
  expire, in seconds (a positive integer).
- PXAT timestamp-milliseconds -- Set the specified Unix time at which
the keys
  will expire, in milliseconds (a positive integer).
- KEEPTTL -- Retain the time to live associated with the keys.
- NX -- Only set the keys if all keys do not already exist.
- XX -- Only set the keys if all keys already exists.

Return Value (like MSETNX command):
- Integer reply: 0 if no key was set due to NX or XX options.
- Integer reply: 1 if all the keys were set.

Closes #2592

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-03-10 19:09:12 +02:00
cjx-zarandGitHub 8c0c6b2044 fix alignment different between c and c++ in gtest (#3331)
I found that on 32-bit platforms, the gtest fails if I add a declaration
`int aof_rewrite_for_replication;` between
https://github.com/valkey-io/valkey/blob/971e5f5c30a27bf73d3ad59a8b24c7f3128db128/src/server.h#L1843
and
https://github.com/valkey-io/valkey/blob/971e5f5c30a27bf73d3ad59a8b24c7f3128db128/src/server.h#L2108
The crash is caused by
https://github.com/valkey-io/valkey/blob/971e5f5c30a27bf73d3ad59a8b24c7f3128db128/src/unit/wrappers.h#L40
stripping the _Atomic qualifier in C++ builds. This makes struct
valkeyServer have different alignment/field offsets in C-compiled code
(libvalkey.a) vs C++-compiled tests (gtest). As a result, fields after
these atomics are accessed at different offsets, leading to corrupted
reads/writes and a segfault.

I added VALKEY_ATOMIC, and it works as follows:
- C builds (production): VALKEY_ATOMIC(uint64_t) -> _Atomic uint64_t
(unchanged behavior)
- C++ builds (tests via wrappers.h): VALKEY_ATOMIC(uint64_t) ->
alignas(8) uint64_t (preserves 8-byte alignment)
- All existing atomic_load_explicit / atomic_store_explicit calls in C
code continue to work because _Atomic is present in C mode
- C++ test code only does direct field access (never uses atomic
operations), so alignas(N) type is sufficient

---------

Signed-off-by: cjx-zar <jxchenczar@foxmail.com>
2026-03-10 07:02:24 -07:00
Harkrishn PatroandGitHub 1303dc2fbb Use dictGetSomeKeys instead of dictGetRandomKey for gossip node population (#3258)
Observation around `dictGetSomeKeys(N)` performance is that it is around
10x faster than randomly picking `N` elements over N*3 attempts.
Further, this avoids chances of not picking N elements due to duplicate
element selection with individual random pick. Overall, it should help
picking gossip node(s) for cluster bus message faster.

---------

Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
2026-03-10 01:48:40 -07:00
Sarthak AggarwalandGitHub 259921486b [flaky-tests] atomically snapshot dual-channel memory stats (#3336)
This fixes a flaky dual-channel replication integration test:
https://github.com/valkey-io/valkey/actions/runs/22810251608/job/66165776198#step:8:7701

`INFO memory` field `used_memory_overhead` and `MEMORY STATS` field
`overhead.total` can change during dual-channel sync if replica's
pending replication buffer is still changing. This is probably more
visible in slower environments.

The test now collects `INFO` and `MEMORY STATS` in a single `MULTI/EXEC`
on both the primary and replica, so the compared values come from the
same snapshot.

Passing here:
https://github.com/sarthakaggarwal97/valkey/actions/runs/22864585326/job/66327772967

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-03-09 15:45:29 -07:00
Harkrishn PatroandGitHub fa232d72bb Make the number of healthy nodes in gossip section configurable (#2746)
This PR introduces a new config `cluster-message-gossip-perc` which
allows an operator to modify the amount of gossip node information to be
sent per ping/pong/meet message. It can be modified dynamically (Related
to https://github.com/valkey-io/valkey/issues/2291). The default value
is 10% i.e. 10% of peer node information would be gossiped along with
each ping/pong/meet packet. Users can tune this configuration, setting
the value higher allows faster information dissemination whereas setting
it lower would lead to direct PING messages if no information was
received about a node with the `server.cluster_node_timeout/2` period.

Note: the behavior for partially failed gossip nodes still remains
intact where all the `pfail` nodes are part of the message for faster
propagation of information and faster transition of `PFAIL` to `FAIL`.

---------

Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
2026-03-09 14:57:58 -07:00
Roshan KhatriandGitHub c16c83bd34 Adds cluster benchmark support for benchmark-on-label (#3338)
Now we will be able to add a `run-cluster-benchmark` label to run a
benchmark with cluster-mode enabled valkey-server

It will use the config
https://github.com/valkey-io/valkey/blob/unstable/.github/benchmark_configs/benchmark-config-arm.json
modified for for cluster mode with a single clustermode enabled instance of
valkey.

It uses the same single instance for the benchmark as for run-benchmark.
 
If both labels are used, they are sequential in the same concurrency group `group:
ec2-al-2023-pr-benchmarking-arm64`.

---------

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2026-03-09 22:48:29 +01:00
2b594a69f0 Replace long long timestamp references with more specific mstime_t/ustime_t (#3252)
Addresses issue #2350

As noted in the issue, much of the expire code uses raw long long for
timestamps, which provides no semantic meaning about the unit or purpose
of the value. Valkey already defines mstime_t (milliseconds) and
ustime_t (microseconds) typedefs — this PR replaces bare long long
declarations with the appropriate typedef wherever the value represents
an expiration timestamp or time duration.

This PR only fixes a small subset in the codebase, but it is an
incremental step toward fully replacing the bare long long references.

---------

Signed-off-by: curious-george-rk <r.ebu@gmail.com>
Co-authored-by: curious-george-rk <r.ebu@gmail.com>
2026-03-09 20:03:26 +02:00
eifrah-awsandGitHub ed85c734c2 Changed LUA module dependency (#3325)
Before this change, if you built `valkey-server` (e.g. `make
valkey-server`) the LUA module was not built. With this change, the LUA
module is now a direct a dependency of `valkey-server`
- (unless `BUILD_LUA=no` is passed)

Added some colors to the Lua module & Lua lib `Makefile`s to they blend
nicely in the build output.

Signed-off-by: Eran Ifrah <eifrah@amazon.com>
2026-03-09 20:01:29 +02:00
bb89628d19 CLUSTERSCAN Command (#2934)
Implemented CLUSTERSCAN command for topology-aware scanning

Unlike `SCAN` which is local to a single node, `CLUSTERSCAN` provides a
mechanism that helps clients iterate across slot boundaries and handles
`MOVED` redirections.

**Key details**

* Global cluster iteration via `fingerprint-{hashtag}-cursor`
* Scan one slot at a time
* Start the CLUSTERSCAN with 0
* SLOT argument for parallel scanning of multiple slots
* Re-use scanGenericCommand for the response

**Cursor format:** `fingerprint-{hashtag}-localcursor`
 - Fingerprint is a hash of the node's DB seed that identifies the
   current memory layout. On mismatch, scan restarts from cursor 0
   rather than returning an error.
 - Fingerprint 0 indicates a cross slot cursor (e.g., initial cursor
   or slot transition) where validation is skipped.
 - Hashtag encodes the target slot
 - Local cursor tracks position within the slot

**Usage:**

```
CLUSTERSCAN <cursor> [MATCH pattern] [COUNT count] [TYPE type] [SLOT number]
```

```
  CLUSTERSCAN 0                    # Start scanning from slot 0
  CLUSTERSCAN <cursor>             # Continue from cursor
  CLUSTERSCAN 0 SLOT 1000          # Start scanning specific slot
  CLUSTERSCAN <cursor> MATCH user:* COUNT 100
```

---------

Signed-off-by: nmvk <r@nmvk.com>
Signed-off-by: Raghav <r@nmvk.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2026-03-09 07:41:36 -07:00
BinbinandGitHub 971e5f5c30 Update cluster config file when NOFAILOVER changes (#3280)
Currently after changing cluster-replica-no-failover, we will
call clusterUpdateMyselfFlags to trigger CLUSTER_TODO_SAVE_CONFIG.
But in gossip when the sender's NOFAILOVER changes, we won't
trigger the save, this cause nodes.conf to not save the latest
nofailover flag.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-03-08 00:01:55 +08:00
Daniil KashapovandGitHub 9568ee4f05 Fixed point arithmetic for x86 TSC (#3314)
Replace integer mono_ticksPerMicrosecond with fixed-point arithmetic in
x86 TSC monotonic clock for better accuracy (and performance because of
multiplication + shift instead of division).
Calibration now uses double precision to compute multiplier.

**Comparison with TSC @ 2400.5 ticks/us**

**Old Method:**
Calibration: 2400 (truncated from 2400.5)
Converting 1 second of ticks: 2,400,500,000 / 2400 = 1,000,208 us
Error: +208 us per second

**New Method:**
Calibration: sample_ticks_per_us = 2400.5 (double stores exactly in this
case)
Multiplier: 2^24 / 2400.5 = 6989.942 -> stored as 6989 in uint64_t
Converting 1 second: (2,400,500,000 * 6989) >> 24 = 999,992 us
Error: -8 us per second

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
2026-03-06 17:57:03 +01:00
Yang ZhaoandGitHub 653e330af4 Recommend passwordless users for mTLS certificate-based authentication (#3311)
Closes https://github.com/valkey-io/valkey/issues/3286

---------

Signed-off-by: Yang Zhao <zymy701@gmail.com>
2026-03-06 14:39:05 +01:00
Björn SvenssonandGitHub f6ac10dcef Use ar archiver installed by brew in CI build-macos-latest (#3317)
Since there is some mismatch between the already installed `ar` tool on
a macOS runner
and Clang 22, installed by brew; lets use the brew installed `llvm-ar`.

Expected to fix the issue in CI job `build-macos-latest`.

---------

Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
2026-03-06 12:37:29 +01:00
BinbinandGitHub fe7e4392ef Fix timing issue in LATENCY GRAPH test (#3265)
In #3260 we try to deflake it by using a assert_range, but the upper
limit of the range is low so the test is still flaky. Increase the upper
limit to 200ms more than the expected latency, e.g. from 550 to 700 when
the expected latency is 500.

```
*** [err]: LATENCY GRAPH can output the event graph in tests/unit/latency-monitor.tcl
Expected '625' to be between to '450' and '550' (context: type source line 143 file /Users/runner/work/valkey/valkey/tests/unit/latency-monitor.tcl cmd {assert_range $high 450 550} proc ::test)
```

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-03-05 20:03:32 +01:00
ffaa9ad1d7 Let script continue if busy-reply-threshold is zero (#3307)
Before we added LUA as a module we had a logic to NOT register the
luaHook when the value of `busy-reply-threshold` config is set to 0.

Now we ALWAYS register the hook and in order to keep aligned with
old behavior we will let the execution of the script continue from
the interrupt hook when `busy-reply-threshold` config is set == 0.

And in value.conf, we are saying the config can be negative, but in
fact the config is minimum at 0, fix the valkey.conf as well.
```
# The default is 5 seconds. It is possible to set it to 0 or a negative value
# to disable this mechanism (uninterrupted execution)
```

It was introduced in #2858.

Signed-off-by: Alon Arenberg <alonare@amazon.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2026-03-05 23:52:47 +08:00
Sarthak AggarwalandGitHub 255b827f35 fix Codecov v5 input from file to files (#3308)
This PR fixes a Codecov workflow misconfiguration introduced when
upgrading codecov/codecov-action from v4 to v5 (in #3185).
In v5, the action expects files (plural), but the workflow still used
file.

The coverage shown is 0 right now:
https://app.codecov.io/gh/valkey-io/valkey


Documentation from -
https://github.com/codecov/codecov-action/tree/v5?tab=readme-ov-file#arguments

```
The following arguments have been changed

    file (this has been deprecated in favor of files)
    plugin (this has been deprecated in favor of plugins)

```

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-03-05 13:49:38 +01:00
Ran ShidlansikandGitHub aa06454d3f Fix zset to operate defrag scan lexicographically (#3316)
as part of #2508 we introduce a defrag optimization to avoid matching
the replaced defrag entry.
Even though defrag replacements does not impact the skip list order,
when scores are equal, we MUST compare elements lexicographically to
maintain correct skip list ordering.
Otherwise we might miss locating the entry.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2026-03-05 13:51:47 +02:00
Roshan KhatriandGitHub 3550a651fd Fix compatibility for OpenSSL < 3.0 and Almalinux version mismatch for daily tests (#3303)
`SSL_get0_peer_certificate()` was introduced in OpenSSL 3.0. The recent
commit 136852efc (Support TLS authentication using SAN URI) used it in
`tlsGetPeerUser()` without a version guard, breaking builds against
`OpenSSL 1.1.x.`

Use `SSL_get_peer_certificate()` on OpenSSL < 3.0 with the corresponding
`X509_free()` since the older API increments the reference count.

Fixes build failure: implicit declaration of function
`SSL_get0_peer_certificate [-Werror=implicit-function-declaration]`

Also fixes the version mismatch for almalinux 9 daily tests.
Closes #3304.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2026-03-05 19:34:17 +08:00
Kurt McKeeandGitHub 276cbd7c43 Update the CodeQL action to resolve v3 deprecation warnings (#3310)
The CodeQL workflow is currently throwing a deprecation warning
regarding use of v3.

> CodeQL Action v3 will be deprecated in December 2026. Please update
all occurrences of the CodeQL Action in your workflow files to v4.

This PR introduces the following changes:
* References to CodeQL v3 have been updated to the SHA of the latest
CodeQL release, [v4.32.5].

Signed-off-by: Kurt McKee <contactme@kurtmckee.org>
2026-03-05 10:55:47 +08:00
Harry LinandGitHub ef1da0f81c Fix compatibility issue for death test with Valgrind (#3301) 2026-03-03 19:03:57 -08:00
Yang ZhaoandGitHub 136852efcc Support TLS authentication using SAN URI (#3078)
Closes https://github.com/valkey-io/valkey/issues/3077

### Overview 
URI in SAN is used to represent client identities in modern mTLS
deployments where CN may be empty or deprecated. See the
https://github.com/valkey-io/valkey/issues/3077#issuecomment-3775615304
for more details.
When `tls-auth-clients-user URI` is configured, during the TLS
handshake, the server iterates through the URIs in the client
certificate and authenticates the client as the first enabled user whose
name matches one of those URIs.

### Implementation
- Introduced a new value `URI` for `tls-auth-clients-user`
- Added new function `getCertSanUri` that:
- Extracts URI entries from the certificate's SAN extension
- Checks each URI against existing Valkey users
- Returns the first URI that matches an enabled user
- Renamed `getCertFieldByName` → `getCertSubjectFieldByName` for clarity
- Modified `tlsGetPeerUsername` to support both CN and URI
authentication modes

### Example behavior
Common setup
``` 
# client certificate X509v3 Subject Alternative Name
URI:urn:valkey:user:first, URI:urn:valkey:user:second

# valkey.conf
tls-auth-clients-user URI
hide-user-data-from-log no
```
Use case 1: multiple enabled users
```
user urn:valkey:user:first on >clientpass allcommands allkeys
user urn:valkey:user:second on >clientpass allcommands allkeys

39762:M 26 Jan 2026 22:06:25.122 - TLS: Auto-authenticated client as urn:valkey:user:first
```
Use case 2: first URI disabled, second enabled
```
user urn:valkey:user:first off >clientpass allcommands allkeys
user urn:valkey:user:second on >clientpass allcommands allkeys

39792:M 26 Jan 2026 22:07:08.006 - TLS: Auto-authenticated client as urn:valkey:user:second
```
Use case 3: all matching users disabled or no matching user
```
user urn:valkey:user:first off >clientpass allcommands allkeys
user urn:valkey:user:second off >clientpass allcommands allkeys

39812:M 26 Jan 2026 22:07:34.174 * TLS: No matching user found in certificate SAN URI fields
127.0.0.1:6379> acl whoami
"default"
127.0.0.1:6379> acl log
1)  1) "count"
    2) (integer) 1
    3) "reason"
    4) "tls-cert"
    5) "context"
    6) "toplevel"
    7) "object"
    8) ""
    9) "username"
   10) "urn:valkey:user:second"
   11) "age-seconds"
   12) "17.381"
   13) "client-info"
   14) "id=3 addr=127.0.0.1:57236 laddr=127.0.0.1:6379 fd=8 name= age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=NULL user=default redir=-1 resp=2 lib-name= lib-ver= tot-net-in=0 tot-net-out=0 tot-cmds=0"
   15) "entry-id"
   16) (integer) 0
   17) "timestamp-created"
   18) (integer) 1771963041866
   19) "timestamp-last-updated"
   20) (integer) 1771963041866
127.0.0.1:6379>
```

---------

Signed-off-by: Yang Zhao <zymy701@gmail.com>
2026-03-03 14:38:04 +01:00
Rain ValentineandGitHub 2622aa8e03 [DEFLAKE] Fix flaky block_keyspace_notification pipelining test (#3294)
The test "Blocking keyspace notification with pipelining hset after
hget" was recently failing intermittently with two different errors:

1. `Expected [expr {114 * 10 < 1114}]` - timing assertion failed under Valgrind
2. `Timeout waiting for blocked clients` - race condition on normal runs

The test used wall-clock timing to verify that hget (non-blocking)
completed faster than hset (blocking). This is unreliable because:
- Valgrind slows execution 10-50x, making timing ratios meaningless
- Fast systems may complete both operations in <10ms, causing ratio failures

This fix replaces timing assertions with blocked client count checks,
which directly verify the blocking mechanism rather than inferring it
from timing. The test now confirms hget's response is available before
hset blocks, then waits for the blocked client count to transition
through the expected states.

Signed-off-by: Rain Valentine <rsg000@gmail.com>
2026-03-03 15:56:28 +08:00
9a51cc390a Fix TSAN compatibility for module loading (#3257)
Made few changes to support TSAN build in valkey-search repository.
Related: https://github.com/valkey-io/valkey-search/issues/703

Signed-off-by: Baswanth Vegunta <baswanth@amazon.com>
Co-authored-by: Baswanth Vegunta <baswanth@amazon.com>
2026-03-03 15:54:37 +08:00
Rain ValentineandGitHub be048d4246 rename functions - prefix should be zset not zsl (#3278)
Simple function renaming. The zsl prefix denotes the function is
specific to the zset skiplist, but zslFreeLexRange and zslParseLexRange
are not specific to zset encoding. `zset` is a more accurate prefix.

I am working on #3166 but this is worthwhile on its own.

Signed-off-by: Rain Valentine <rainval@amazon.com>
2026-03-02 16:18:35 -08:00
Sarthak AggarwalandGitHub c1b6c248b3 Fix weekly release runs checking out unstable in daily workflow (#3295)
Honors `workflow_call` inputs rather than checking out the
`GITHUB_HEAD_REF` always.

```
Determining the checkout info
  /usr/bin/git branch --list --remote origin/8.0
    origin/8.0
/usr/bin/git sparse-checkout disable
/usr/bin/git config --local --unset-all extensions.worktreeConfig
Checking out the ref
  /usr/bin/git checkout --progress --force -B 8.0 refs/remotes/origin/8.0
  Switched to a new branch '8.0'
  branch '8.0' set up to track 'origin/8.0'.

```


Now the workflow checks out the right branch. 
Link:
https://github.com/sarthakaggarwal97/valkey/actions/runs/22599943936/job/65479450708#step:3:83

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-03-02 16:13:26 -08:00
BinbinandGitHub 05fb04465a Show replica dual-channel replication buffer memory in INFO MEMORY and MEMORY STATS (#2924)
After introducing dual channel replication in #60, we added two new info
fields `replicas_repl_buffer_size` and `replicas_repl_buffer_peak` to the
info replication section.

However, we did not display the actual memory usage of the replication
buffer in info memory section. Furthermore, since the replicas replication
buffer was not previously included in memory overhead section, this caused
the `used_memory_dataset` and `used_memory_overhead` fields in INFO MEMORY
to display incorrect values, and also caused the `overhead.total` field in
`MEMORY STATS` to display an incorrect value. Previously, on replica nodes,
the replicas replication buffer was included in dataset memory.

Changes:
- `mem_total_replication_buffers` should include all replication buffers, so
  now it will also contain `mem_replicas_repl_buffer`.
- Added a new `mem_replicas_repl_buffer` field in INFO MEMORY section. During
  the dual channel replication, it show total memory consumed by the replicas
  replication buffer on replica side.
- Added a `new replicas.repl.buffer` field in MEMORY STATS. It is the same
  metric as `mem_replicas_repl_buffer`.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-03-02 10:44:45 +08:00
Ran ShidlansikandGitHub 4969ccc08a Deflake ttl persistence in aof test (#3285)
Sometimes it might miss setting a field with short expiration

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2026-03-02 10:40:30 +08:00
Tony WoosterandGitHub 811abbf18f Fix use-after-free in generateSyncSlotsEstablishCommand (#3283)
Noticed a use-after-free/double-free bug while load-testing manual slot
migrations. Was able to reliably trigger a process crash.

Original trace:

```
Backtrace:
0   libsystem_platform.dylib            0x00000001917496a4 _sigtramp + 56
1   libsystem_pthread.dylib             0x000000019170f848 pthread_kill + 296
2   libsystem_c.dylib                   0x00000001916189e4 abort + 124
3   libsystem_malloc.dylib              0x000000019151c174 malloc_vreport + 892
4   libsystem_malloc.dylib              0x000000019151fc90 malloc_report + 64
5   libsystem_malloc.dylib              0x000000019152421c ___BUG_IN_CLIENT_OF_LIBMALLOC_POINTER_BEING_FREED_WAS_NOT_ALLOCATED + 32
6   valkey-server                       0x0000000104fc2174 _sdsMakeRoomFor + 780
7   valkey-server                       0x0000000104fc30a0 sdscatfmt + 168
8   valkey-server                       0x00000001050318b8 generateSyncSlotsEstablishCommand + 176
9   valkey-server                       0x000000010503156c createSlotImportJob + 324
10  valkey-server                       0x0000000105034fd4 clusterCommandSyncSlots + 1572
11  valkey-server                       0x000000010502f370 clusterCommandSpecial + 2432
12  valkey-server                       0x0000000105021e80 clusterCommand + 2992
13  valkey-server                       0x0000000104fb59a8 call + 320
14  valkey-server                       0x0000000104fcc708 processCommandAndResetClient + 3088
15  valkey-server                       0x0000000104fc9c3c processInputBuffer + 440
16  valkey-server                       0x0000000104fc9198 readQueryFromClient + 108
17  valkey-server                       0x000000010508073c connSocketEventHandler + 136
18  valkey-server                       0x0000000104fa49b4 aeProcessEvents + 316
19  valkey-server                       0x0000000104fc081c main + 18408
20  dyld                                0x000000019136eb98 start + 6076
```

The bug seems to be that the result of `sdscatfmt` is being ignored.
Since `sdscatfmt` potentially reallocs a new buffer when it needs to,
and deallocs the old one, this can cause a use-after-free/double-free if
the loop triggers a buffer-size increase.

This PR fixes the bug.

Signed-off-by: Tony Wooster <twooster@gmail.coM>
2026-03-02 10:40:01 +08:00
bandalgomsuandGitHub 6b2411da04 Fix use zfree in cli --eval path (#3281)
In `evalMode()`, `argv2` is allocated with `zmalloc()` but freed with `free()`.
On builds using jemalloc/tcmalloc, this can cause `free(): invalid pointer`
Closes #3273

Signed-off-by: Su Ko <rhtn1128@gmail.com>
2026-03-01 21:27:44 +08:00
Alina LiuandGitHub e97d127fb2 Suppress valgrind for BIO jobs (#3277)
Suppress valgrind for BIO jobs valgrind errors:
https://github.com/valkey-io/valkey/actions/runs/21969557125/job/63467641572#step:6:8648

## Changes

- Add `allocBioJob()` with `__attribute__((noinline))` as a centralized
allocation function for all BIO jobs. The `noinline` attribute ensures
it appears as a distinct frame in valgrind stack traces.
- Replace all direct `zmalloc` calls in `bioCreate*Job` functions
with`allocBioJob()`.
- Add a valgrind suppression in `src/valgrind.sup` matching definite
leaks with `allocBioJob` in the call stack.
- Remove `current_job` introduced in #3256

---------

Signed-off-by: Alina Liu <liusalisa6363@gmail.com>
2026-02-27 22:36:20 -08:00
Gagan H RandGitHub da206c609a Set read permission in workflow to improve OpenSSF score (#3267)
This PR fixes #3264. Score is currently 6. It will get improved to
around 6.9 once this PR is merged.

Signed-off-by: Gagan H R <hrgagan4@gmail.com>
2026-02-27 23:23:31 +01:00
Harry LinandGitHub ce291b3613 Add gtest dependencies to unit test workflows (#3270)
Install gtest dependencies in daily.yml.

Signed-off-by: harrylin98 <harrylin980107@gmail.com>
2026-02-27 13:39:51 -08:00
Madelyn OlsonandGitHub c84990bf56 Add Jim Brunner as a committer (#3272)
---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2026-02-27 13:17:54 -08:00
Hanxi ZhangandGitHub 3110d910ef Deflake Restart target replica during migration (without save) causes success test (#3226)
Fix #3222

- Replace assert_equal with wait_for_condition for replica DBSIZE checks
- Fix a duplicate assert_equal line that checked $not_owning_repl twice
instead of checking both $not_owning_prim and $not_owning_repl.

## Analysis

wait_for_countkeysinslot did work, the keys were there. The issue is
that COUNTKEYSINSLOT and DBSIZE read from different sources.

COUNTKEYSINSLOT reads the actual hashtable size:
```
// countKeysInSlotForDb (cluster.c, line 831)
kvstoreHashtableSize(db->keys, hashslot)
    → hashtableSize(ht)    // actual keys in the slot
```
DBSIZE reads a cached counter:
```
// dbsizeCommand (db.c)
kvstoreSize(c->db->keys) // This can be found in line 358, kvstore.c)
    → kvs->key_count       // cached total
```
When a slot is marked as importing, keys are excluded from key_count:
```
// cumulativeKeyCountAdd (kvstore.c, line 157)
if (kvstoreIsImporting(kvs, didx)) {
    kvs->importing_key_count += delta;  // hidden from DBSIZE
    return;                              // key_count NOT updated
}
kvs->key_count += delta;                // visible to DBSIZE
```

key_count will be updated when:
```
finishSlotMigrationJob (cluster_migrateslots.c, line 2302)
-> call setSlotImportingStateInAllDbs(slots, 0) (cluster_migrateslots.c, line 2340)
  -> call setSlotImportingStateInDb() (cluster_migrateslots.c, line 249)
    -> call kvstoreSetIsImporting() (cluster_migrates.c, line 238)
      -> call cmulativeKeyCountAdd() (kvstore.c, line 955)
        -> kvs->key_count += delta;
```

When slots are marked as importing, the keys exist in the hashtables but
are excluded from metrics like key_count. As documented in
kvstoreSetIsImporting (kvstore.c):

"Importing hashtables are not included in hashtable metrics and are
excluded from scanning and random key lookup."

After the replica restarts and resyncs, there is a brief window where
the migrated slots are still marked as importing. During this window,
COUNTKEYSINSLOT sees the keys (reads actual hashtable) but DBSIZE
returns 0 (reads key_count which excludes importing keys). Once the
importing flag is cleared, key_count is updated and DBSIZE returns the
correct value.

wait_for_condition handles this by retrying until DBSIZE reflects the
updated key_count.

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
2026-02-27 10:42:50 +08:00
3107c4eb21 Introduce GoogleTest for Valkey unit testing (#3241)
This PR adds GoogleTest (gtest) support to Valkey to enable 
writing modern unit tests,as mentioned in
https://github.com/valkey-io/valkey/issues/2878

**Motivation**: 
GoogleTest provides richer assertions, test fixtures, mocking
support, and improved diagnostics, helping improve test coverage
and maintainability over time.

For more details, see `src/gtest/README.md`.

**Changes**

This PR integrates the GoogleTest framework and migrates all
existing C unit tests to GoogleTest.

---------

Signed-off-by: Harry Lin <harrylhl@amazon.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: Alina Liu <liusalisa6363@gmail.com>
Signed-off-by: Jacob Murphy <jkmurphy@google.com>
Signed-off-by: Jim Brunner <brunnerj@amazon.com>
Co-authored-by: Harry Lin <harrylhl@amazon.com>
Co-authored-by: Jim Brunner <brunnerj@amazon.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Jacob Murphy <jkmurphy@google.com>
Co-authored-by: Alina Liu <liusalisa6363@gmail.com>
2026-02-26 14:14:23 -08:00
Gagan H RandGitHub 1b52016873 Adds scorecard workflow to publish OpenSSF scores (#3163)
Publish OpenSSF Scorecard results, which means users and downstream
consumers can easily discover the project’s security best-practice
signals via Scorecard API.

Publishing Scorecard results:

- Improves transparency for users and integrators
- Provides early visibility into missing or improvable security
practices

Fixes #3162

---------

Signed-off-by: Gagan H R <hrgagan4@gmail.com>
2026-02-26 21:24:06 +01:00
Shinobu NunotabaandGitHub ea942753c4 Lazy loading of RDMA libs in CLI/Benchmark when building as module (#3072)
This PR decouples the mandatory RDMA library dependency from
`valkey-cli` and `valkey-benchmark` by implementing a **runtime dynamic
loading (lazy loading)** mechanism.

When `BUILD_RDMA=module` is used, the CLI tools are no longer
hard-linked to `librdmacm` and `libibverbs`. Instead, these libraries
are only loaded via `dlopen` when the `--rdma` flag is explicitly
invoked by the user.

- **Dynamic Symbol Loader**: Implemented a symbol loader in libvalkey
using dlopen() and dlsym(), here:
https://github.com/valkey-io/libvalkey/pull/284. That was merged and
libvalkey was updated in valkey to include this change.
- **Build System Propagation**:
- **Makefile**: When building as a module, pass `USE_DLOPEN_RDMA` flag
to libvalkey's makefile.
- **CMake**: When building as a module, pass `ENABLE_DLOPEN_RDMA` flag
to libvalkey's CMake build.

Fixes #3070

Signed-off-by: Ada-Church-Closure <nunotabashinobu066@gmail.com>
2026-02-26 13:57:16 +01:00
Yana MolodetskyandGitHub aa39375bdf Deflake test 'LATENCY GRAPH can output the event graph' (#3260)
The test was using `assert_morethan_equal` with fixed minimum thresholds
(500ms for high, 300ms for low), which caused flaky failures when the
actual latency values were slightly below these thresholds due to timing
variations.

Example failure:

```
Expected '499' to be more than or equal to '500' (context: type eval line 12 cmd {assert_morethan_equal $high 500} proc ::test)
```

Changes:

- Replaced assert_morethan_equal $high 500 with assert_range $high 450
550
- Replaced assert_morethan_equal $low 300 with assert_range $low 250 350

The new ranges (450-550ms for high, 250-350ms for low) align with the
expected latency values from the debug sleep commands (0.5s and 0.3s
respectively) while providing tolerance for timing variations,
consistent with the approach used elsewhere in the test file.

Signed-off-by: Yana Molodetsky <yamolodu@amazon.com>
2026-02-26 13:13:41 +02:00
Alina LiuandGitHub c3020e0a11 Replace mutexQueuePeek with current_job field for BIO memcheck fix (#3256)
## Problem
The peek-then-pop pattern introduced in
https://github.com/valkey-io/valkey/pull/3178 solved the valgrind
false-positive leak report
(https://github.com/valkey-io/valkey/actions/runs/21969557125/job/63467641572#step:6:8648)
for in-flight BIO jobs, but `peek` is a problematic API on a mutexqueue:
multiple readers can peek the same item, and one reader can pop what
another peeked.

## Fix
Instead, store the in-flight job pointer in
`bio_worker_data.current_job` before processing and clear it after
freeing. Since `bio_workers[]` is a static array, valgrind can always
trace from global memory to the job allocation, even after the worker
thread is cancelled.

Removed `mutexQueuePeek` from mutexqueue.{c,h} and its test.

## Test
Manually run Daily workflow: All four valgrind jobs are green:
`test-valgrind-test` passed, `test-valgrind-misc` passed,
`test-valgrind-no-malloc-usable-size-test` passed,
`test-valgrind-no-malloc-usable-size-misc` passed.

Signed-off-by: Alina Liu <liusalisa6363@gmail.com>
2026-02-25 15:59:16 -08:00
Hanxi ZhangandGitHub 617643f35e Fix type of now parameter in rdbLoadObject() to use mstime_t instead of time_t (#3255)
Fix #3254 
### Problem
The now parameter in rdbLoadObject() was declared as time_t, which
truncates millisecond timestamps on 32-bit systems, causing hash field
expiry checks to fail during RDB load.
Reproduced:
https://github.com/hanxizh9910/valkey/actions/runs/22382739246/job/64786946078

### Fix

Changed time_t now → mstime_t now in the declaration and definition of
rdbLoadObject().

### Test run
https://github.com/hanxizh9910/valkey/actions/runs/22383363092

---------

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
2026-02-25 10:56:20 +02:00
Hanxi ZhangandGitHub 63f37ee960 Fix flake dual-channel-replication primary gets cob overrun before established psync test (#3242)
Fix #3231: Increase wait timeout for RDB load log detection in test

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
2026-02-25 10:10:13 +08:00
Rain ValentineandGitHub 851ca54c66 Bugfix for GIL deadlock while unloading script engine, reenable memory test in crash report (#3029)
The memory test was commented out in #2858 and should have been
reenabled. On further investigation I found that the server hangs during
shutdown inside the `bioDrainWorker(BIO_LAZY_FREE)` call. This causes
deadlock because the lock was acquired for shutdown but lazy free jobs
require the GIL too:

- main thread: `serverCron()` acquires GIL via `afterSleep()` then calls
`finishShutdown()`, which eventually calls our script module unload code
that calls `bioDrainWorker()`.
- bio threads: Pending lazy free jobs such as `lazyFreeEvalScripts()`
call `scriptingEngineCallFreeFunction()` which requires the GIL.

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
2026-02-24 19:50:31 +01:00
b32c68a66f Abort and swap the tables if ht1 is very full during the hashtable shrink rehashing (#3175)
During hashtable shrinking, all keys are inserted into ht1.
This PR adds a mechanism to the hashtable: when there are severe hash
collisions with the new ht1 during the shrink rehashing process, the
hashtable will stop shrinking by swapping ht0 and ht1 to avoid excessive
performance degradation of the new hashtable during this period.

In extreme cases, for example, if ht0 is very large and is reduced to
only one entry, and the new ht1 is very small after the resize, the ht0
rehash process is very slow since we have a lot of empty buckets. If a
large number of elements are added into ht1 at this time, it will lead
to severe hash collisions in ht1.

During the add operation during hashtable shrinking, we check the fill
percentage of ht1. If it exceeds MAX_FILL_PERCENT_HARD, we swap ht0 and
ht1, and abort the shrinking process, and then set rehash_idx back to 0
and restart the rehash.

This PR also added a new debug hashtable-can-abort-shrink subcommand to
control this behavior.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
2026-02-24 12:58:39 +01:00
6b81e36c9d Skip expired hash fields when loading non-preamble RDB on primary (#3091)
## Summary

Fixes #2620

Skip loading expired hash fields when a non-preamble RDB is being loaded
on a primary server. Propagate `HDEL` to replicas when expired fields
are skipped.

## Changes

- Updated `rdbLoadObject` signature to accept `rdbflags` and `now`
parameters
- Added logic to skip expired hash fields during RDB load on primary
- Propagate `HDEL` to replicas when `RDBFLAGS_FEED_REPL` is set
- Updated all callers of `rdbLoadObject`
- Added unit test

---------

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2026-02-24 13:24:47 +02:00
BinbinandGitHub af7bac374b Logging fix or improvement around new shard ID generation (#3192)
When a cluster reset is performed on a replica node, a new shard ID is generated
because the node is about to become an empty primary node, see #2283.

However, the log added in #2510 caused some confusions. In clusterSetNodeAsPrimary
we will print:
```
serverLog(LL_NOTICE, "Reconfiguring node %.40s (%s) as primary for shard %.40s", n->name, humanNodename(n), n->shard_id);
```

In clusterReset, we first call clusterSetNodeAsPrimary and then generate a new
shard ID, which causes us to print an error shard ID log first.

There is an exmaple, when a replica node performs a cluster reset, we will print:
```
xxx * Cluster reset (user request from 'xxx').
xxx * Reconfiguring node af76a3e0ffcd77bd14fa47ce4d07ab2bdc78702f (xxx) as primary for shard ea528667634af8beed83adac2b9af8360769a1b4
```

But the node shard id is actually:
```
xxx> cluster myshardid
"52ede26d1554dd203161ba09011af14574b2cc84"
```

Now after a new shard ID is generated we will print a log, and we also move the
call to clusterSetNodeAsPrimary after the new shard id, so that we can have the
right one. After this PR:
```
xxx * Cluster reset (user request from 'xxx').
xxx * Moving myself to a new shard bd31870ce73f5977084e6a46e337a4a1ad38fc66.
xxx * Reconfiguring node 1d54b904efd30cd9d7d1abbfd63c8fafbb62e1c8 (xxx) as primary for shard bd31870ce73f5977084e6a46e337a4a1ad38fc66
```

This is part of #2989, but i guess we won't merge the extension fix in a short
time, so i am gonna extracting it separately as a log fix (or improvement).

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-02-24 11:13:15 +08:00
Roshan KhatriandGitHub fd65d487eb Fix Tcl 9 compatibility in cluster packet test (#3251)
Remove -encoding binary from fconfigure as it is no longer supported in
Tcl 9 (Fedora Rawhide and Daily / test-fedoralatest-jemalloc
(pull_request)). -translation binary alone is sufficient.
```
https://github.com/valkey-io/valkey/actions/runs/22324297258/job/64590589777?pr=3225
===== Start of server log (pid 17617) =====

### Starting server for test 
17617:M 23 Feb 2026 21:21:47.846 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
17617:M 23 Feb 2026 21:21:47.846 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
17617:M 23 Feb 2026 21:21:47.846 * Valkey version=9.0.3, bits=64, commit=00000000, modified=0, pid=17617, just started
17617:M 23 Feb 2026 21:21:47.846 * Configuration loaded
17617:M 23 Feb 2026 21:21:47.847 * monotonic clock: POSIX clock_gettime
                .+^+.                                                
            .+#########+.                                            
        .+########+########+.           Valkey 9.0.3 (00000000/0) 64 bit
    .+########+'     '+########+.                                    
 .########+'     .+.     '+########.    Running in cluster mode
 |####+'     .+#######+.     '+####|    Port: 25121
 |###|   .+###############+.   |###|    PID: 17617                     
 |###|   |#####*'' ''*#####|   |###|                                 
 |###|   |####'  .-.  '####|   |###|                                 
 |###|   |###(  (@@@)  )###|   |###|          https://valkey.io/      
 |###|   |####.  '-'  .####|   |###|                                 
 |###|   |#####*.   .*#####|   |###|                                 
 |###|   '+#####|   |#####+'   |###|                                 
 |####+.     +##|   |#+'     .+####|                                 
 '#######+   |##|        .+########'                                 
    '+###|   |##|    .+########+'                                    
        '|   |####+########+'                                        
             +#########+'                                            
                '+v+'                                                

17617:M 23 Feb 2026 21:21:47.849 * No cluster configuration found, I'm 5501916ccdf76dee6b652cab10402cab3a8f9152
17617:M 23 Feb 2026 21:21:47.852 * Server initialized
17617:M 23 Feb 2026 21:21:47.852 * Ready to accept connections tcp
17617:M 23 Feb 2026 21:21:47.852 * Ready to accept connections unix
17617:M 23 Feb 2026 21:21:47.963 - Accepted 127.0.0.1:37791
17617:M 23 Feb 2026 21:21:47.963 - Client closed connection id=2 addr=127.0.0.1:37791 laddr=127.0.0.1:25121 fd=12 name= age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37504 events=r cmd=ping user=default redir=-1 resp=2 lib-name= lib-ver= tot-net-in=7 tot-net-out=7 tot-cmds=1
17617:M 23 Feb 2026 21:21:47.969 - Accepted 127.0.0.1:39251
17617:M 23 Feb 2026 21:21:47.970 * configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH
17617:M 23 Feb 2026 21:21:47.975 # Missing implement of connection type tls
17617:M 23 Feb 2026 21:21:47.976 # DEBUG LOG: ========== I am primary 0 ==========
17617:M 23 Feb 2026 21:21:49.872 * Cluster state changed: ok
### Starting test Packet with missing gossip messages don't cause invalid read in tests/unit/cluster/packet.tcl
17617:M 23 Feb 2026 21:21:49.879 - Accepting cluster node connection from 127.0.0.1:33701
17617:M 23 Feb 2026 21:21:49.880 - Client closed connection id=3 addr=127.0.0.1:39251 laddr=127.0.0.1:25121 fd=12 name= age=2 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=426 obl=0 oll=0 omem=0 tot-mem=22144 events=r cmd=cluster|info user=default redir=-1 resp=2 lib-name= lib-ver= tot-net-in=1363 tot-net-out=17324 tot-cmds=47
17617:signal-handler (1771881709) Received SIGTERM scheduling shutdown...
17617:M 23 Feb 2026 21:21:49.973 * User requested shutdown...
17617:M 23 Feb 2026 21:21:49.973 * Removing the pid file.
17617:M 23 Feb 2026 21:21:49.973 * Saving the cluster configuration file before exiting.
17617:M 23 Feb 2026 21:21:49.978 * Removing the unix socket file.
17617:M 23 Feb 2026 21:21:49.978 # Valkey is now ready to exit, bye bye...
===== End of server log (pid 17617) =====


===== Start of server stderr log (pid 17617) =====


===== End of server stderr log (pid 17617) =====

[exception]: Executing test client: unknown encoding "binary": No longer supported.
	please use either "-translation binary" or "-encoding iso8859-1".
unknown encoding "binary": No longer supported.
	please use either "-translation binary" or "-encoding iso8859-1"
    while executing
"fconfigure $sock -translation binary -encoding binary -buffering none -blocking 1"
    ("uplevel" body line 18)
    invoked from within
"uplevel 1 $code"
    (procedure "test" line 62)
    invoked from within
"test "Packet with missing gossip messages don't cause invalid read" {
        set base_port [srv 0 port]
        set cluster_port [expr {$base_port + ..."
    ("uplevel" body line 2)
    invoked from within
"uplevel 1 $code"
    (procedure "cluster_setup" line 41)
    invoked from within
"cluster_setup 1 0 1 continuous_slot_allocation default_replica_allocation {
    test "Packet with missing gossip messages don't cause invalid read" {
..."
    ("uplevel" body line 1)
    invoked from within
"uplevel 1 $code "
    (procedure "start_server" line 2)
    invoked from within
"start_server {overrides {cluster-enabled yes cluster-ping-interval 100 cluster-node-timeout 3000 cluster-databases 16 cluster-slot-stats-enabled yes} ..."
    ("uplevel" body line 1)
    invoked from within
"uplevel 1 $code"
    (procedure "start_multiple_servers" line 5)
    invoked from within
"start_multiple_servers $node_count $options $code"
    (procedure "start_cluster" line 17)
    invoked from within
"start_cluster 1 0 {tags {external:skip cluster tls:skip}} {
    test "Packet with missing gossip messages don't cause invalid read" {
        set base..."
    (file "tests/unit/cluster/packet.tcl" line 84)
    invoked from within
"source $path"
    (procedure "execute_test_file" line 4)
    invoked from within
"execute_test_file $data"
    (procedure "test_client_main" line 10)
    invoked from within
"test_client_main $::test_server_port "
Killing still running Valkey server 10700
Killing still running Valkey server 11659
Killing still running Valkey server 11705
Killing still running Valkey server 12735
Killing still running Valkey server 12775
Killing still running Valkey server 12808
Killing still running Valkey server 12858
Killing still running Valkey server 12901
Killing still running Valkey server 12940
Killing still running Valkey server 12972
Killing still running Valkey server 13004
Killing still running Valkey server 13036
Killing still running Valkey server 13073
Killing still running Valkey server 14952
Killing still running Valkey server 16631
Killing still running Valkey server 16670
Killing still running Valkey server 16686
Killing still running Valkey server 16702
Killing still running Valkey server 16718
Killing still running Valkey server 16734
Killing still running Valkey server 16753
Killing still running Valkey server 16771
Killing still running Valkey server 17094
Killing still running Valkey server 17112
Killing still running Valkey server 17133
Killing still running Valkey server 17262
Killing still running Valkey server 17278
Killing still running Valkey server 17316
Killing still running Valkey server 17349
Killing still running Valkey server 17379
Killing still running Valkey server 17410
Killing still running Valkey server 17430
Killing still running Valkey server 17443
Killing still running Valkey server 17459
Killing still running Valkey server 17478
Killing still running Valkey server 17491
Killing still running Valkey server 17507
Killing still running Valkey server 17526
Killing still running Valkey server 17545
Killing still running Valkey server 17561
Killing still running Valkey server 17577
Killing still running Valkey server 17716
Killing still running Valkey server 17778
Killing still running Valkey server 17825
Killing still running Valkey server 17848
Killing still running Valkey server 17896
Killing still running Valkey server 17931
Killing still running Valkey server 17953
Killing still running Valkey server 17973
Killing still running Valkey server 17991
Killing still running Valkey server 18010
Killing still running Valkey server 18064
Killing still running Valkey server 18082
Killing still running Valkey server 18134
Killing still running Valkey server 18118
Killing still running Valkey server 18150
Killing still running Valkey server 18166
Killing still running Valkey server 18188
Killing still running Valkey server 18209
```

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2026-02-23 15:19:25 -08:00
Madelyn Olson a737a11d66 Reset request type after handling empty requests
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2026-02-23 11:26:53 -08:00
Roshan KhatriandMadelyn Olson 77ff9eb0e3 Fix for [CVE-2025-67733] RESP Protocol Injectiton via Lua error_reply
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2026-02-23 09:56:06 -08:00
Roshan KhatriandMadelyn Olson 5a97f83b5b Fix for [CVE-2026-21863] Remote DoS with malformed Valkey Cluster bus message
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2026-02-23 09:56:06 -08:00
BinbinandGitHub 274deca160 Fix getKeysUsingKeySpecs keynum case ignoring keystep when calc the last key (#3197)
This issue was encountered while processing #3121. Currently in all our
commands with KSPEC_FK_KEYNUM, key step is 1. So this bug does not
currently
affect any core commands.

If we have commands with different key step values, calculting the last
key
in here will casue problems since we are not including step in the
calculation.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-02-23 09:07:22 -08:00
Joseph HeyburnandGitHub bdefaa54ff Dual-channel-replication announces itself at replica-announce-ip if configured (#2846)
When dual-channel-replication is enabled, and replica-announce-ip is
set, the RDB/AOF channel does not announce itself at this endpoint. This
defaults to the IP address behind the NAT, or the Kubernetes Pod IP in
our case.

This means that if Sentinel is polling the primary for connected
replicas, it will first see the ephemeral pod IP, then revert to the
announce-ip - leaving behind the pod IP as a down replica.

This PR configures the RDB/AOF channel to also announce itself at the
announce-ip to prevent the stale replica.

## Testing

I evaluated writing unit tests for this, but I am not sure of a way we
can test an IP address different to localhost (127.0.0.1) that would
fail without the fix. I did test on Kubernetes against 9.0 tag and
verified the fix there too.

### Status quo

On 9.0 image tag:

```
$ kubectl get pods -n valkey-baseline -o custom-columns=NAME:.metadata.name,POD-IP:.status.podIP
NAME                              POD-IP
valkey-primary-5bd78c8566-llb6k   10.244.0.25
valkey-replica-0                  10.244.0.17
valkey-replica-1                  10.244.0.13

$ kubectl get services -n valkey-baseline -o custom-columns=NAME:.metadata.name,CLUSTER-IP:.spec.clusterIP
NAME               CLUSTER-IP
valkey-primary     10.96.147.28
valkey-replica-0   10.96.66.233
valkey-replica-1   10.96.57.230
```

Logs below show that pod IP for valkey-primary-5bd78c8566-llb6k
`10.244.0.25:6379` is being used for dual-channel replication. This
should be its cluster IP `10.96.147.28` as this is what is set in
replica-announce-ip.

```
1:M 14 Nov 2025 17:57:51.750 * Replica 10.96.147.28:6379 asks for synchronization
1:M 14 Nov 2025 17:57:51.751 * Replica 10.244.0.25:6379 asks for synchronization
1:M 14 Nov 2025 17:57:56.135 * Dual channel replication: Sending to replica 10.244.0.25:6379 RDB end offset 1763269 and client-id 35
1:M 14 Nov 2025 17:57:56.140 * Replica 10.96.147.28:6379 asks for synchronization
```

### This fix

```
$ kubectl get pods -n valkey-test -o custom-columns=NAME:.metadata.name,CLUSTER-IP:.status.podIP  
NAME                              POD-IP
valkey-primary-594c9597b5-qqvdk   10.244.0.26
valkey-replica-0                  10.244.0.10
valkey-replica-1                  10.244.0.18

$ kubectl get services -n valkey-test -o custom-columns=NAME:.metadata.name,CLUSTER-IP:.spec.clusterIP
NAME               CLUSTER-IP
valkey-primary     10.96.125.142
valkey-replica     None
valkey-replica-0   10.96.155.74
valkey-replica-1   10.96.64.111
valkey-sentinel    None
```

Logs show that the Cluster IP is now being used for dual-channel
replication.

```
1:M 14 Nov 2025 17:57:49.923 * Replica 10.96.125.142:6379 asks for synchronization
1:M 14 Nov 2025 17:57:49.924 * Replica 10.96.125.142:6379 asks for synchronization
1:M 14 Nov 2025 17:57:54.913 * Dual channel replication: Sending to replica 10.96.125.142:6379 RDB end offset 1771247 and client-id 36
1:M 14 Nov 2025 17:57:54.916 * Replica 10.96.125.142:6379 asks for synchronization
```

Fixes #2338

Signed-off-by: Joseph Heyburn <jdheyburn@gmail.com>
2026-02-23 18:16:48 +02:00
Björn SvenssonandGitHub d3ae62cc9c Update deps/libvalkey to version 0.4.0 (#3216)
Update deps/libvalkey to version 0.4.0

Squashed 'deps/libvalkey/' changes from b012f8e85..45c2ed15c

45c2ed15c Release 0.4.0 (#286)
40d6590d7 Implement runtime dynamic loading for RDMA libraries (#284)
62e757d17 Release 0.3.0 (#283)
a554f0942 Fix potential uint32_t underflow issue (#280)
8f9051ae0 Correcting command parser bug (#277)
29023eb36 Add valkey-json, valkey-bloom, valkey-search to cmddef.h
ae756bc89 Update cmddef.h to Valkey 9.0.0
21abd737e Replace problematic alloca() with fixed stack alloc
38191079c Fix compilation on Solaris with Sun/Solaris Studio
ef5de0312 Make libvalkey initialization thread-safe
ae341dea5 Support slotmap updates using CLUSTER NODES in RESP3 (#262)
36f6e2292 Fix the long-blocking read for Valkey RDMA. (#233)
c090c28be Use a uintptr_t hop for casting pointers to ints
daa7f11ac Avoid heap buffer overflow in valkeyAsyncFormattedCommand (#245)
15974930d Add option to select a logical database (#244)
983d67e4f Install the macosx adapter on Apple platforms only
...

git-subtree-dir: deps/libvalkey
git-subtree-split: 45c2ed15cab9fa0ea1a6cabc8460f5eea6240de5

Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
2026-02-23 15:28:30 +01:00
721 changed files with 56175 additions and 43017 deletions
+5 -1
View File
@@ -5,7 +5,7 @@ extend-exclude = [
".git/",
"deps/",
# crc16_slottable is primarily pre-generated random strings.
"src/crc16_slottable.h",
"src/crc16_slottable.c",
]
ignore-hidden = false
@@ -14,6 +14,7 @@ exat = "exat"
optin = "optin"
smove = "smove"
Parth = "Parth" # seems like the spellchecker does not like it is similar to "Path"
NAM = "NAM" # Name similar to Name
nd = "nd"
[default]
@@ -71,3 +72,6 @@ ake = "ake"
[type.tcl.extend-words]
fo = "fo"
tre = "tre"
[type.cpp.extend-words]
fo = "fo"
+5 -5
View File
@@ -1,17 +1,17 @@
blank_issues_enabled: true
contact_links:
- name: Questions?
url: https://github.com/kv-io/kv/discussions
url: https://github.com/valkey-io/valkey/discussions
about: Ask and answer questions on GitHub Discussions.
- name: Chat with us on Discord?
url: https://discord.gg/zbcPa5umUB
about: We are on Discord!
- name: Chat with us on Matrix?
url: https://matrix.to/#/#kv:matrix.org
url: https://matrix.to/#/#valkey:matrix.org
about: We are on Matrix too!
- name: Chat with us on Slack?
url: https://join.slack.com/t/kv-oss-developer/shared_invite/zt-2nxs51chx-EB9hu9Qdch3GMfRcztTSkQ
url: https://join.slack.com/t/valkey-oss-developer/shared_invite/zt-2nxs51chx-EB9hu9Qdch3GMfRcztTSkQ
about: We are on Slack too!
- name: Documentation issue?
url: https://github.com/kv-io/kv-doc/issues
about: Report it on the kv-doc repo.
url: https://github.com/valkey-io/valkey-doc/issues
about: Report it on the valkey-doc repo.
@@ -0,0 +1,14 @@
name: 'Upload Test Failures'
description: 'Upload test failure artifacts'
inputs:
job-name:
description: 'Unique name for the artifact'
required: true
runs:
using: 'composite'
steps:
- name: Upload test failures
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: test-failures-${{ inputs.job-name }}
path: test-failures/
@@ -1,20 +0,0 @@
[
{
"duration": 180,
"keyspacelen": [3000000],
"data_sizes": [16,96],
"pipelines": [1, 10],
"clients": [1600],
"commands": [
"SET",
"GET"
],
"cluster_mode": false,
"tls_mode": false,
"warmup": 30,
"io-threads": [1,9],
"benchmark-threads": 90,
"server_cpu_range": "0-8",
"client_cpu_range": "96-191"
}
]
@@ -1,20 +0,0 @@
[
{
"duration": 180,
"keyspacelen": [3000000],
"data_sizes": [16,96],
"pipelines": [1, 10],
"clients": [1600],
"commands": [
"SET",
"GET"
],
"cluster_mode": false,
"tls_mode": false,
"warmup": 30,
"io-threads": [1,9],
"benchmark-threads": 90,
"server_cpu_range": "0-8",
"client_cpu_range": "144-191,48-95"
}
]
+4 -4
View File
@@ -1,6 +1,6 @@
# KV Project Instructions
# Valkey Project Instructions
You are an expert code reviewer for the KV project. Provide helpful, constructive feedback on code quality, safety, and adherence to project standards.
You are an expert code reviewer for the Valkey project. Provide helpful, constructive feedback on code quality, safety, and adherence to project standards.
## 1. Review Tone & Focus
- **Tone:** Be professional, direct, constructive, and empathetic.
@@ -9,7 +9,7 @@ You are an expert code reviewer for the KV project. Provide helpful, constructiv
## 2. Critical Checks
- **DCO:** **Flag missing** `Signed-off-by: Name <email>` in commits. Every commit needs it.
- **Security:** If PR fixes a security vulnerability, flag it: "Security fixes should be reported privately to security@hanzo.ai, not via public PRs."
- **Security:** If PR fixes a security vulnerability, flag it: "Security fixes should be reported privately to security@lists.valkey.io, not via public PRs."
## 3. Major Decision Detection
Flag PRs that appear to be "Technical Major Decisions" requiring TSC consensus:
@@ -23,7 +23,7 @@ Flag PRs that appear to be "Technical Major Decisions" requiring TSC consensus:
## 4. Documentation Reminder
If PR changes user-facing behavior (new commands, changed semantics, new config):
- **Remind** author that docs at [kv-doc](https://github.com/hanzoai/kv-doc) may need updating.
- **Remind** author that docs at [valkey-doc](https://github.com/valkey-io/valkey-doc) may need updating.
- **Suggest** linking PR to related Issue with "Fixes #xyz" pattern if applicable.
## 5. Governance Changes
@@ -1,10 +1,10 @@
---
applyTo:
- "src/**/*.{c,h}"
- "kv.conf"
- "valkey.conf"
---
# KV Core Engine Review Standards
# Valkey Core Engine Review Standards
Apply these standards to core engine C code. Do NOT apply to `deps/` (vendored dependencies).
@@ -36,12 +36,12 @@ Apply these standards to core engine C code. Do NOT apply to `deps/` (vendored d
- **PR Scope:** Separate refactoring from functional changes for easier backporting.
## 5. Testing & Documentation
- **Unit Tests:** Required for data structures in `src/unit/`. Test files should follow `test_*.c` naming.
- **Unit Tests:** Required for data structures in `src/unit/`. Test files should follow `test_*.cpp` naming.
- **Integration Tests:** Required for commands in `tests/`.
- **Command Changes:** New/modified commands need corresponding updates in `src/commands/*.json`.
- **New C Files:** Remind to update `CMakeLists.txt` when adding new `.c` source files.
- **License:** New files need BSD-3-Clause header. Material changes (>100 lines) also require it.
- **Documentation:** User-facing changes need docs at [kv-doc](https://github.com/hanzoai/kv-doc).
- **Documentation:** User-facing changes need docs at [valkey-doc](https://github.com/valkey-io/valkey-doc).
## 6. Critical Escalation
- **Trigger:** Changes to `cluster*.c`, `replication.c`, `rdb.c`, `aof.c`.
@@ -3,7 +3,7 @@ applyTo:
- "tests/**/*.tcl"
---
# KV Integration Test Review Standards
# Valkey Integration Test Review Standards
Apply these standards to Tcl-based integration tests (from DEVELOPMENT_GUIDE.md).
+1 -1
View File
@@ -3,7 +3,7 @@ applyTo:
- "utils/**/*"
---
# KV Utilities Review Standards
# Valkey Utilities Review Standards
Apply these standards to utility scripts and tools.
+60 -42
View File
@@ -5,8 +5,8 @@ on:
types: [labeled]
concurrency:
group: ec2-al-2023-pr-benchmarking-arm64
cancel-in-progress: false
group: benchmark-${{ github.event.pull_request.number }}-${{ github.event.label.name }}
cancel-in-progress: true
defaults:
run:
@@ -20,34 +20,35 @@ permissions:
jobs:
benchmark:
if: |
github.event.action == 'labeled' && github.event.label.name == 'run-benchmark' &&
github.repository == 'kv-io/kv'
github.event.action == 'labeled' &&
(github.event.label.name == 'run-benchmark' || github.event.label.name == 'run-cluster-benchmark') &&
github.repository == 'valkey-io/valkey'
runs-on: ["self-hosted", "ec2-al-2023-pr-benchmarking-arm64"]
timeout-minutes: 7200
steps:
- name: Checkout kv
- name: Checkout valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
path: kv
path: valkey
fetch-depth: 0
ref: ${{ github.event.pull_request.merge_commit_sha }}
persist-credentials: false
- name: Checkout kv-perf-benchmark
- name: Checkout valkey-perf-benchmark
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ github.repository_owner }}/kv-perf-benchmark
path: kv-perf-benchmark
repository: ${{ github.repository_owner }}/valkey-perf-benchmark
path: valkey-perf-benchmark
fetch-depth: 1
persist-credentials: false
- name: Checkout kv for latest benchmark.
- name: Checkout valkey for latest benchmark.
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: kv-io/kv
repository: valkey-io/valkey
ref: "unstable"
path: kv_latest
path: valkey_latest
fetch-depth: 0
persist-credentials: false
@@ -58,56 +59,72 @@ jobs:
cache: "pip"
- name: Install dependencies
working-directory: kv-perf-benchmark
working-directory: valkey-perf-benchmark
run: |
sudo dnf groupinstall "Development Tools" -y
sudo dnf install -y gcc gcc-c++ make \
python3-devel \
openssl-devel \
bzip2-devel \
libffi-devel
pip install -r requirements.txt
libffi-devel \
jq
pip install --require-hashes -r requirements.txt
- name: Build latest kv_latest
working-directory: kv_latest
- name: Build latest valkey_latest
working-directory: valkey_latest
run: |
echo "Building latest kv-benchmark for latest benchmark executable..."
echo "Building latest valkey-benchmark for latest benchmark executable..."
make distclean || true
make -j
if [[ -f "src/kv-benchmark" ]]; then
echo "Successfully built latest kv-benchmark"
ls -la src/kv-benchmark
./src/kv-benchmark --version || echo "Version check completed"
if [[ -f "src/valkey-benchmark" ]]; then
echo "Successfully built latest valkey-benchmark"
ls -la src/valkey-benchmark
./src/valkey-benchmark --version || echo "Version check completed"
else
echo "Failed to build kv-benchmark"
echo "Failed to build valkey-benchmark"
exit 1
fi
KV_BENCHMARK_PATH="$(pwd)/src/kv-benchmark"
echo "KV_BENCHMARK_PATH=$KV_BENCHMARK_PATH" >> $GITHUB_ENV
echo "Latest kv-benchmark path: $KV_BENCHMARK_PATH"
VALKEY_BENCHMARK_PATH="$(pwd)/src/valkey-benchmark"
echo "VALKEY_BENCHMARK_PATH=$VALKEY_BENCHMARK_PATH" >> $GITHUB_ENV
echo "Latest valkey-benchmark path: $VALKEY_BENCHMARK_PATH"
- name: Filter data_sizes > 1024 from benchmark config
working-directory: valkey-perf-benchmark
run: |
CONFIG_FILE="configs/benchmark-config-arm.json"
jq '[.[] | .data_sizes = [.data_sizes[] | select(. <= 1024)]]' "$CONFIG_FILE" > tmp.json && mv tmp.json "$CONFIG_FILE"
- name: Enable cluster mode in benchmark config
working-directory: valkey-perf-benchmark
if: github.event.label.name == 'run-cluster-benchmark'
run: |
CONFIG_FILE="configs/benchmark-config-arm.json"
sed -i 's/"cluster_mode": false/"cluster_mode": true/g' "$CONFIG_FILE"
echo "Updated config for cluster mode:"
cat "$CONFIG_FILE"
- name: Run benchmarks
working-directory: kv-perf-benchmark
working-directory: valkey-perf-benchmark
run: |
CONFIG_FILE="../kv/.github/benchmark_configs/benchmark-config-arm.json"
CONFIG_FILE="configs/benchmark-config-arm.json"
# Base benchmark arguments
BENCHMARK_ARGS=(
--config "$CONFIG_FILE"
--commits "${{ github.event.pull_request.merge_commit_sha }}"
--baseline "${{ github.event.pull_request.base.ref }}"
--kv-benchmark-path "$KV_BENCHMARK_PATH"
--valkey-benchmark-path "$VALKEY_BENCHMARK_PATH"
--target-ip ${{ secrets.EC2_ARM64_IP }}
--kv-path "../kv"
--valkey-path "../valkey"
--results-dir "results"
--runs 5
--runs 3
)
# Run benchmark
python ./benchmark.py "${BENCHMARK_ARGS[@]}"
- name: Compare results
working-directory: kv-perf-benchmark
working-directory: valkey-perf-benchmark
run: |
python ./utils/compare_benchmark_results.py \
--baseline ./results/${{ github.event.pull_request.base.ref }}/metrics.json \
@@ -120,10 +137,9 @@ jobs:
continue-on-error: true
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: benchmark-results
name: ${{ github.event.label.name }}-results
path: |
./kv-perf-benchmark/results/${{ github.event.pull_request.merge_commit_sha }}/metrics.json
./kv-perf-benchmark/results/${{ github.event.pull_request.base.ref }}/metrics.json
./valkey-perf-benchmark/results/
comparison.md
- name: Comment PR with results
@@ -137,30 +153,32 @@ jobs:
const sha = '${{ github.event.pull_request.head.sha }}';
const short = sha.slice(0,7);
const link = `[\`${short}\`](https://github.com/${owner}/${repo}/commit/${sha})`
const label = '${{ github.event.label.name }}';
const prefix = label === 'run-cluster-benchmark' ? 'Cluster Benchmark' : 'Benchmark';
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner,
repo,
body: `**Benchmark ran on this commit:** ${link}\n\n${body}`
body: `**${prefix} ran on this commit:** ${link}\n\n${body}`
});
- name: Cleanup any running kv processes
- name: Cleanup any running valkey processes
if: always()
continue-on-error: true
run: |
rm -rf comparison.md kv*
pkill -f kv
rm -rf comparison.md valkey*
pkill -f valkey
exit_code=$?
if [ $exit_code -eq 0 ]; then
echo "Killed running kv processes"
echo "Killed running valkey processes"
elif [ $exit_code -eq 1 ]; then
echo "No kv processes found to kill"
echo "No valkey processes found to kill"
else
echo "Warning: pkill failed with exit code $exit_code"
fi
- name: Remove ${{ github.event.label.name }} label
if: always() && github.event.label.name == 'run-benchmark'
if: always() && (github.event.label.name == 'run-benchmark' || github.event.label.name == 'run-cluster-benchmark')
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
+45 -47
View File
@@ -1,4 +1,4 @@
name: Compare KV Versions
name: Compare Valkey Versions
on:
workflow_dispatch:
@@ -32,16 +32,14 @@ permissions:
jobs:
benchmark:
if: github.repository == 'kv-io/kv'
if: github.repository == 'valkey-io/valkey'
strategy:
matrix:
include:
- arch: x86
machine: ec2-al-2023-pr-benchmarking-x86
config: benchmark-config-x86.json
- arch: arm64
machine: ec2-al-2023-pr-benchmarking-arm64
config: benchmark-config-arm.json
concurrency:
group: ${{ matrix.machine }}
cancel-in-progress: false
@@ -54,7 +52,6 @@ jobs:
echo "Version 2: ${{ github.event.inputs.version2 }}"
echo "Issue ID: ${{ github.event.inputs.issue_id }}"
echo "Architecture: ${{ matrix.arch }}"
echo "Config: ${{ matrix.config }}"
echo "Runs: ${{ github.event.inputs.runs }}"
# Validate issue ID is numeric
@@ -69,25 +66,25 @@ jobs:
exit 1
fi
- name: Checkout kv for latest benchmark.
- name: Checkout valkey for latest benchmark.
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: kv-io/kv
repository: valkey-io/valkey
ref: "unstable"
path: kv_latest
path: valkey_latest
fetch-depth: 0
persist-credentials: false
- name: Checkout kv
- name: Checkout valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
path: kv
path: valkey
- name: Checkout kv-perf-benchmark
- name: Checkout valkey-perf-benchmark
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ github.repository_owner }}/kv-perf-benchmark
path: kv-perf-benchmark
repository: ${{ github.repository_owner }}/valkey-perf-benchmark
path: valkey-perf-benchmark
fetch-depth: 1
persist-credentials: false
@@ -98,77 +95,79 @@ jobs:
cache: "pip"
- name: Install dependencies
working-directory: kv-perf-benchmark
working-directory: valkey-perf-benchmark
run: |
sudo dnf groupinstall "Development Tools" -y
sudo dnf install -y gcc gcc-c++ make \
python3-devel \
openssl-devel \
bzip2-devel \
libffi-devel
pip install -r requirements.txt
libffi-devel \
jq
pip install --require-hashes -r requirements.txt
- name: Build latest kv_latest
working-directory: kv_latest
- name: Build latest valkey_latest
working-directory: valkey_latest
run: |
echo "Building latest kv-benchmark for latest benchmark executable..."
echo "Building latest valkey-benchmark for latest benchmark executable..."
# Clean any previous builds
make distclean || true
# Build kv-benchmark with latest code
# Build valkey-benchmark with latest code
make -j$(nproc)
# Verify the binary was created
if [[ -f "src/kv-benchmark" ]]; then
echo "✓ Successfully built latest kv-benchmark"
ls -la src/kv-benchmark
if [[ -f "src/valkey-benchmark" ]]; then
echo "✓ Successfully built latest valkey-benchmark"
ls -la src/valkey-benchmark
# Test the binary
echo "Testing kv-benchmark binary..."
./src/kv-benchmark --version || echo "Version check completed"
echo "Testing valkey-benchmark binary..."
./src/valkey-benchmark --version || echo "Version check completed"
else
echo "Failed to build kv-benchmark"
echo "Failed to build valkey-benchmark"
exit 1
fi
# Store the absolute path for later use
KV_BENCHMARK_PATH="$(pwd)/src/kv-benchmark"
echo "KV_BENCHMARK_PATH=$KV_BENCHMARK_PATH" >> $GITHUB_ENV
echo "Latest kv-benchmark path: $KV_BENCHMARK_PATH"
VALKEY_BENCHMARK_PATH="$(pwd)/src/valkey-benchmark"
echo "VALKEY_BENCHMARK_PATH=$VALKEY_BENCHMARK_PATH" >> $GITHUB_ENV
echo "Latest valkey-benchmark path: $VALKEY_BENCHMARK_PATH"
- name: Run benchmarks
working-directory: kv-perf-benchmark
working-directory: valkey-perf-benchmark
env:
EC2_X86_IP: ${{ secrets.EC2_X86_IP }}
EC2_ARM64_IP: ${{ secrets.EC2_ARM64_IP }}
run: |
# Set the target IP based on the matrix architecture
CONFIG_FILE="configs/benchmark-config-arm.json"
# Set the target IP and override config based on architecture
if [[ "${{ matrix.arch }}" == "x86" ]]; then
TARGET_IP=$EC2_X86_IP
echo "Using x86 machine IP"
# Override client_cpu_range for x86 due to different NUMA node core allocation
jq '[.[] | .client_cpu_range = "144-191,48-95"]' "$CONFIG_FILE" > tmp.json && mv tmp.json "$CONFIG_FILE"
elif [[ "${{ matrix.arch }}" == "arm64" ]]; then
TARGET_IP=$EC2_ARM64_IP
echo "Using ARM64 machine IP"
else
echo "Error: Unknown architecture: ${{ matrix.arch }}"
exit 1
fi
CONFIG_FILE="../kv/.github/benchmark_configs/${{ matrix.config }}"
# Verify our custom kv-benchmark exists
if [[ ! -f "$KV_BENCHMARK_PATH" ]]; then
echo "Custom kv-benchmark not found at: $KV_BENCHMARK_PATH"
# Verify our custom valkey-benchmark exists
if [[ ! -f "$VALKEY_BENCHMARK_PATH" ]]; then
echo "Custom valkey-benchmark not found at: $VALKEY_BENCHMARK_PATH"
exit 1
fi
echo "Using custom kv-benchmark from: $KV_BENCHMARK_PATH"
echo "Using custom valkey-benchmark from: $VALKEY_BENCHMARK_PATH"
# Base benchmark arguments with custom kv-benchmark path
# Base benchmark arguments with custom valkey-benchmark path
BENCHMARK_ARGS=(
--config "$CONFIG_FILE"
--commits ${{ github.event.inputs.version1 }} ${{ github.event.inputs.version2 }}
--kv-benchmark-path "$KV_BENCHMARK_PATH"
--valkey-benchmark-path "$VALKEY_BENCHMARK_PATH"
--target-ip $TARGET_IP
--results-dir "results"
--runs ${{ github.event.inputs.runs }}
@@ -178,15 +177,14 @@ jobs:
echo "- Version 1: ${{ github.event.inputs.version1 }}"
echo "- Version 2: ${{ github.event.inputs.version2 }}"
echo "- Architecture: ${{ matrix.arch }}"
echo "- Config: ${{ matrix.config }}"
echo "- Using latest kv-benchmark: $KV_BENCHMARK_PATH"
echo "- Using latest valkey-benchmark: $VALKEY_BENCHMARK_PATH"
echo "- This ensures both versions use the same (latest) benchmark tool for consistent results"
# Run benchmark with custom kv-benchmark executable
# Run benchmark with custom valkey-benchmark executable
python ./benchmark.py "${BENCHMARK_ARGS[@]}"
- name: Compare results
working-directory: kv-perf-benchmark
working-directory: valkey-perf-benchmark
run: |
# Find the actual result directories (they might have different names than input)
VERSION1_DIR=$(find ./results -maxdepth 1 -type d -name "*${{ github.event.inputs.version1 }}*" | head -1)
@@ -224,14 +222,14 @@ jobs:
with:
name: benchmark-results-${{ matrix.arch }}-${{ github.event.inputs.issue_id }}
path: |
./kv-perf-benchmark/results/*
./valkey-perf-benchmark/results/*
comparison.md
- name: Cleanup any running kv processes and files
- name: Cleanup any running valkey processes and files
if: always()
continue-on-error: true
run: |
pkill -f kv || echo "No kv processes found to kill"
pkill -f valkey || echo "No valkey processes found to kill"
rm -rf *
combine-results:
+66 -30
View File
@@ -30,12 +30,14 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
# Fail build if there are warnings
# build with TLS just for compilation coverage
run: make -j4 all-with-unit-tests SERVER_CFLAGS='-Werror' BUILD_TLS=yes USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
run: |
sudo apt-get install pkg-config libgtest-dev libgmock-dev
make -j4 all-with-unit-tests SERVER_CFLAGS='-Werror' BUILD_TLS=yes USE_LIBBACKTRACE=yes
- name: test
run: |
sudo apt-get install tcl8.6 tclx
@@ -50,7 +52,7 @@ jobs:
if [[ ! -z "$dirty" ]]; then echo "$dirty"; exit 1; fi
- name: unit tests
run: |
./src/kv-unit-tests
make test-unit
test-ubuntu-latest-compatibility:
runs-on: ubuntu-latest
@@ -58,10 +60,13 @@ jobs:
fail-fast: false
matrix:
server:
- {version: "7.2.11", file: "kv-7.2.11-noble-x86_64.tar.gz"}
- {version: "8.0.6", file: "kv-8.0.6-noble-x86_64.tar.gz"}
- {version: "8.1.4", file: "kv-8.1.4-noble-x86_64.tar.gz"}
- {version: "7.2.11", file: "valkey-7.2.11-noble-x86_64.tar.gz"}
- {version: "8.0.6", file: "valkey-8.0.6-noble-x86_64.tar.gz"}
- {version: "8.1.4", file: "valkey-8.1.4-noble-x86_64.tar.gz"}
steps:
- name: Install gtest
run: |
sudo apt-get install pkg-config libgtest-dev libgmock-dev
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -69,30 +74,30 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
# Fail build if there are warnings
# build with TLS just for compilation coverage
run: make -j4 all-with-unit-tests SERVER_CFLAGS='-Werror' BUILD_TLS=yes USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
run: make -j4 all-with-unit-tests SERVER_CFLAGS='-Werror' BUILD_TLS=yes USE_LIBBACKTRACE=yes
- name: Install old server (${{ matrix.server.version }}) for compatibility testing
run: |
mkdir -p tests/tmp
cd tests/tmp
wget https://download.kv.io/releases/${{ matrix.server.file }}
wget https://download.valkey.io/releases/${{ matrix.server.file }}
tar -xvf ${{ matrix.server.file }}
- name: Run compatibility tests against ${{ matrix.server.version }}
run: |
sudo apt-get install -y tcl8.6 tclx
./runtest --verbose --tags "-slow needs:other-server" --dump-logs \
--other-server-path tests/tmp/kv-${{ matrix.server.version }}-noble-x86_64/bin/kv-server
--other-server-path tests/tmp/valkey-${{ matrix.server.version }}-noble-x86_64/bin/valkey-server
- name: Module API tests against ${{ matrix.server.version }}
run: |
CFLAGS='-Werror' ./runtest-moduleapi --tags needs:other-server --verbose --dump-logs \
--other-server-path tests/tmp/kv-${{ matrix.server.version }}-noble-x86_64/bin/kv-server
--other-server-path tests/tmp/valkey-${{ matrix.server.version }}-noble-x86_64/bin/valkey-server
test-ubuntu-latest-cmake-tls:
runs-on: ubuntu-latest
@@ -103,7 +108,7 @@ jobs:
sudo apt-get install -y cmake libssl-dev
mkdir -p build-release
cd build-release
cmake -DCMAKE_BUILD_TYPE=Release .. -DBUILD_TLS=yes -DBUILD_UNIT_TESTS=yes
cmake -DCMAKE_BUILD_TYPE=Release .. -DBUILD_TLS=yes -DBUILD_UNIT_GTESTS=yes
make -j$(nproc)
- name: test
run: |
@@ -111,8 +116,7 @@ jobs:
./utils/gen-test-certs.sh
./build-release/runtest --verbose --tags -slow --dump-logs --tls
- name: unit tests
run: |
./build-release/bin/kv-unit-tests
run: make -C build-release test-unit
test-sanitizer-address:
runs-on: ubuntu-latest
@@ -124,11 +128,13 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
# build with TLS module just for compilation coverage
run: make -j4 all-with-unit-tests SANITIZER=address SERVER_CFLAGS='-Werror' BUILD_TLS=module USE_LIBBACKTRACE=yes
run: |
sudo apt-get install pkg-config libgtest-dev libgmock-dev
make -j4 all-with-unit-tests SANITIZER=address SERVER_CFLAGS='-Werror' BUILD_TLS=module USE_LIBBACKTRACE=yes
- name: testprep
run: sudo apt-get install tcl8.6 tclx -y
- name: test
@@ -136,7 +142,8 @@ jobs:
- name: module api test
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs
- name: unit tests
run: ./src/kv-unit-tests
run: |
make test-unit
test-rdma:
runs-on: ubuntu-latest
@@ -148,7 +155,7 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: prepare-development-libraries
run: sudo apt-get install librdmacm-dev libibverbs-dev
@@ -160,6 +167,7 @@ jobs:
make -j4 BUILD_RDMA=yes USE_LIBBACKTRACE=yes
- name: clone-rxe-kmod
run: |
uname -a
mkdir -p tests/rdma/rxe
git clone https://github.com/pizhenwei/rxe.git tests/rdma/rxe
make -C tests/rdma/rxe
@@ -203,7 +211,7 @@ jobs:
run: |
apt-get update && apt-get install -y build-essential
cd libbacktrace && ./configure && make && make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
run: make -j4 SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes
@@ -218,11 +226,19 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install build dependencies
run: brew install llvm googletest
- name: make
# Build with additional upcoming features
run: make -j3 all-with-unit-tests SERVER_CFLAGS='-Werror' USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
run: |
export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
export CC=/opt/homebrew/opt/llvm/bin/clang
export CXX=/opt/homebrew/opt/llvm/bin/clang++
export AR=/opt/homebrew/opt/llvm/bin/llvm-ar
export RANLIB=/opt/homebrew/opt/llvm/bin/llvm-ranlib
make -j3 all-with-unit-tests SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes LIBBACKTRACE_PREFIX=/usr/local
build-32bit:
runs-on: ubuntu-latest
@@ -238,18 +254,38 @@ jobs:
sudo apt-get update
sudo apt-get install libc6-dev-i386 libstdc++-11-dev-i386-cross gcc-multilib g++-multilib
cd libbacktrace && ./configure CFLAGS="-m32" --prefix=/usr/local/libbacktrace32 && make && sudo make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install gtest
run: |
sudo apt-get update
sudo apt-get install libgtest-dev
mkdir -p /tmp/gtest32
cd /tmp/gtest32
cmake -B build32 \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_FLAGS="-m32" \
-DCMAKE_CXX_FLAGS="-m32" \
-DCMAKE_EXE_LINKER_FLAGS="-m32" \
/usr/src/googletest
cmake --build build32 --parallel
sudo cp build32/lib/*.a /usr/lib32/
cd $GITHUB_WORKSPACE
- name: make
# Fast float requires C++ 32-bit libraries to compile on 64-bit ubuntu
# machine i.e. "-cross" suffixed version. Cross-compiling c++ to 32-bit
# also requires multilib support for g++ compiler i.e. "-multilib"
# suffixed version of g++. g++-multilib generally includes libstdc++.
# *cross version as well, but it is also added explicitly just in case.
run: make -j4 SERVER_CFLAGS='-Werror' 32bit USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes LIBBACKTRACE_PREFIX=/usr/local/libbacktrace32
run: |
make -j4 SERVER_CFLAGS='-Werror' 32bit USE_LIBBACKTRACE=yes LIBBACKTRACE_PREFIX=/usr/local/libbacktrace32 \
GTEST_CFLAGS="-I/usr/src/googletest/googletest/include -I/usr/src/googletest/googlemock/include" \
GTEST_LIBS="/usr/lib32/libgtest.a /usr/lib32/libgmock.a"
- name: unit tests
run: |
./src/kv-unit-tests
make test-unit \
GTEST_CFLAGS="-I/usr/src/googletest/googletest/include -I/usr/src/googletest/googlemock/include" \
GTEST_LIBS="/usr/lib32/libgtest.a /usr/lib32/libgmock.a"
build-libc-malloc:
runs-on: ubuntu-latest
@@ -261,10 +297,10 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
run: make -j4 SERVER_CFLAGS='-Werror' MALLOC=libc USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
run: make -j4 SERVER_CFLAGS='-Werror' MALLOC=libc USE_LIBBACKTRACE=yes
build-almalinux8-jemalloc:
runs-on: ubuntu-latest
@@ -278,12 +314,12 @@ jobs:
path: libbacktrace
- name: Build libbacktrace
run: |
dnf -y install epel-release gcc gcc-c++ make procps-ng which
dnf -y install epel-release gcc gcc-c++ make procps-ng which git cmake
cd libbacktrace && ./configure && make && make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
run: make -j4 SERVER_CFLAGS='-Werror' USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
run: make -j4 SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes
format-yaml:
runs-on: ubuntu-latest
@@ -298,7 +334,7 @@ jobs:
- name: Setup YAML formatter
run: |
go install github.com/google/yamlfmt/cmd/yamlfmt@latest
go install github.com/google/yamlfmt/cmd/yamlfmt@v0.21.0
- name: Run yamlfmt
id: yamlfmt
+4 -1
View File
@@ -1,5 +1,8 @@
name: Clang Format Check
permissions:
contents: read
on:
push:
paths-ignore:
@@ -39,7 +42,7 @@ jobs:
# Run clang-format and capture the diff
cd src
shopt -s globstar
clang-format-18 -i **/*.c **/*.h
clang-format-18 -i **/*.c **/*.h **/*.cpp **/*.hpp
# Capture the diff output
DIFF=$(git diff)
if [ ! -z "$DIFF" ]; then
+11 -2
View File
@@ -1,5 +1,8 @@
name: "Codecov"
permissions:
contents: read
# Enabling on each push is to display the coverage changes in every PR,
# where each PR needs to be compared against the coverage of the head commit
on:
@@ -30,8 +33,14 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install gtest
run: |
git clone --depth 1 -b v1.17.x https://github.com/google/googletest.git
cmake -S googletest -B googletest/build
cmake --build googletest/build -- -j$(nproc)
sudo cmake --install googletest/build
- name: Install lcov and run test
run: |
sudo apt-get install lcov tclx
@@ -41,4 +50,4 @@ jobs:
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2
with:
token: ${{ secrets.CODECOV_TOKEN }}
file: ./src/kv.info
files: ./src/valkey.info
+4 -4
View File
@@ -26,7 +26,7 @@ jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
if: github.event_name != 'schedule' || github.repository == 'kv-io/kv'
if: github.event_name != 'schedule' || github.repository == 'valkey-io/valkey'
permissions:
security-events: write
@@ -40,12 +40,12 @@ jobs:
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Initialize CodeQL
uses: github/codeql-action/init@b5ebac6f4c00c8ccddb7cdcd45fdb248329f808a # v3.32.2
uses: github/codeql-action/init@c793b717bc78562f491db7b0e93a3a178b099162 # v4.32.5
with:
languages: ${{ matrix.language }}
- name: Autobuild
uses: github/codeql-action/autobuild@b5ebac6f4c00c8ccddb7cdcd45fdb248329f808a # v3.32.2
uses: github/codeql-action/autobuild@c793b717bc78562f491db7b0e93a3a178b099162 # v4.32.5
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@b5ebac6f4c00c8ccddb7cdcd45fdb248329f808a # v3.32.2
uses: github/codeql-action/analyze@c793b717bc78562f491db7b0e93a3a178b099162 # v4.32.5
+5 -5
View File
@@ -16,7 +16,7 @@ permissions:
jobs:
coverity:
if: github.repository == 'kv-io/kv'
if: github.repository == 'valkey-io/valkey'
runs-on: ubuntu-latest
steps:
- name: Install libbacktrace
@@ -26,14 +26,14 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Download and extract the Coverity Build Tool
run: |
wget -q https://scan.coverity.com/download/cxx/linux64 --post-data "token=${{ secrets.COVERITY_SCAN_TOKEN }}&project=kv-io%2Fkv" -O cov-analysis-linux64.tar.gz
wget -q https://scan.coverity.com/download/cxx/linux64 --post-data "token=${{ secrets.COVERITY_SCAN_TOKEN }}&project=valkey-io%2Fvalkey" -O cov-analysis-linux64.tar.gz
mkdir cov-analysis-linux64
tar xzf cov-analysis-linux64.tar.gz --strip 1 -C cov-analysis-linux64
- name: Install KV dependencies
- name: Install Valkey dependencies
run: sudo apt install -y gcc procps libssl-dev
- name: Build with cov-build
run: cov-analysis-linux64/bin/cov-build --dir cov-int make USE_LIBBACKTRACE=yes
@@ -44,4 +44,4 @@ jobs:
--form email=${{ secrets.COVERITY_SCAN_EMAIL }} \
--form token=${{ secrets.COVERITY_SCAN_TOKEN }} \
--form file=@cov-int.tgz \
https://scan.coverity.com/builds?project=kv-io%2Fkv
https://scan.coverity.com/builds?project=valkey-io%2Fvalkey
File diff suppressed because it is too large Load Diff
-58
View File
@@ -1,58 +0,0 @@
name: Build and Deploy Hanzo KV
on:
push:
branches: [main]
tags: ['v*']
paths:
- 'src/**'
- 'deps/**'
- 'Dockerfile'
- '.github/workflows/deploy.yml'
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Image metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/hanzoai/kv
tags: |
type=raw,value=latest,enable={{is_default_branch}}
type=raw,value=9,enable={{is_default_branch}}
type=sha,prefix=
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
- name: Build and push
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha,scope=kv
cache-to: type=gha,mode=max,scope=kv
+13 -13
View File
@@ -24,7 +24,7 @@ permissions:
jobs:
test-external-standalone:
runs-on: ubuntu-latest
if: github.event_name != 'schedule' || github.repository == 'kv-io/kv'
if: github.event_name != 'schedule' || github.repository == 'valkey-io/valkey'
timeout-minutes: 1440
steps:
- name: Install libbacktrace
@@ -34,13 +34,13 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Build
run: make SERVER_CFLAGS=-Werror USE_LIBBACKTRACE=yes
- name: Start kv-server
- name: Start valkey-server
run: |
./src/kv-server --daemonize yes --save "" --logfile external-server.log \
./src/valkey-server --daemonize yes --save "" --logfile external-server.log \
--enable-protected-configs yes --enable-debug-command yes --enable-module-command yes
- name: Run external test
run: |
@@ -57,7 +57,7 @@ jobs:
test-external-cluster:
runs-on: ubuntu-latest
if: github.event_name != 'schedule' || github.repository == 'kv-io/kv'
if: github.event_name != 'schedule' || github.repository == 'valkey-io/valkey'
timeout-minutes: 1440
steps:
- name: Install libbacktrace
@@ -67,17 +67,17 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Build
run: make SERVER_CFLAGS=-Werror USE_LIBBACKTRACE=yes
- name: Start kv-server
- name: Start valkey-server
run: |
./src/kv-server --cluster-enabled yes --cluster-databases 16 --daemonize yes \
./src/valkey-server --cluster-enabled yes --cluster-databases 16 --daemonize yes \
--save "" --logfile external-server.log \
--enable-protected-configs yes --enable-debug-command yes --enable-module-command yes
- name: Create a single node cluster
run: ./src/kv-cli cluster addslots $(for slot in {0..16383}; do echo $slot; done); sleep 5
run: ./src/valkey-cli cluster addslots $(for slot in {0..16383}; do echo $slot; done); sleep 5
- name: Run external test
run: |
./runtest \
@@ -94,7 +94,7 @@ jobs:
test-external-nodebug:
runs-on: ubuntu-latest
if: github.event_name != 'schedule' || github.repository == 'kv-io/kv'
if: github.event_name != 'schedule' || github.repository == 'valkey-io/valkey'
timeout-minutes: 1440
steps:
- name: Install libbacktrace
@@ -104,13 +104,13 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Build
run: make SERVER_CFLAGS=-Werror USE_LIBBACKTRACE=yes
- name: Start kv-server
- name: Start valkey-server
run: |
./src/kv-server --daemonize yes --save "" --logfile external-server.log
./src/valkey-server --daemonize yes --save "" --logfile external-server.log
- name: Run external test
run: |
./runtest \
+30
View File
@@ -0,0 +1,30 @@
name: Provenance Guard
on:
pull_request_target:
types: [opened, synchronize, reopened, ready_for_review]
permissions:
contents: read
pull-requests: write
issues: write
jobs:
check:
if: github.event.pull_request.base.repo.full_name == 'valkey-io/valkey'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 0
- name: Run Provenance Check
uses: valkey-io/verify-provenance@05ee217ecf96b948ce996dbc965f285cbe73752a
with:
source_repo: "redis/redis"
target_repo: "${{ github.repository }}"
branding_pairs: "Redis:Valkey"
prefix_pairs: "RM_:VM_,REDISMODULE_:VALKEYMODULE_"
exclude_dirs: "deps/"
github_token: "${{ secrets.GITHUB_TOKEN }}"
db_branch: "verify-provenance-db"
+61
View File
@@ -0,0 +1,61 @@
name: Refresh Provenance Data
on:
schedule:
- cron: "0 2 * * *"
workflow_dispatch:
permissions:
contents: write
jobs:
refresh:
if: github.repository == 'valkey-io/valkey'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 0
- name: Refresh Fingerprints
uses: valkey-io/verify-provenance@05ee217ecf96b948ce996dbc965f285cbe73752a
with:
mode: "refresh"
source_repo: "redis/redis"
target_repo: "${{ github.repository }}"
branding_pairs: "Redis:Valkey,KeyDB:Valkey"
prefix_pairs: "RM_:VM_,REDISMODULE_:VALKEYMODULE_"
github_token: "${{ secrets.GITHUB_TOKEN }}"
db_branch: "verify-provenance-db"
- name: Commit Updated PR DB
run: |
set -euo pipefail
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
DB_BRANCH="verify-provenance-db"
# Create the DB branch on first run, otherwise reuse it.
if git ls-remote --exit-code --heads origin "$DB_BRANCH" >/dev/null 2>&1; then
git fetch origin "+$DB_BRANCH:$DB_BRANCH"
git checkout "$DB_BRANCH"
else
git checkout --orphan "$DB_BRANCH"
git rm -rf . >/dev/null 2>&1 || true
fi
# Overwrite with refreshed file (the action places it at .refreshed_pr_db.json.gz)
if [ -f .refreshed_pr_db.json.gz ]; then
mv .refreshed_pr_db.json.gz pr_fingerprints.json.gz
git add pr_fingerprints.json.gz
if ! git diff --cached --quiet; then
git commit -m "Automated PR fingerprint refresh [skip ci]"
git push origin "$DB_BRANCH"
else
echo "No changes to PR database."
fi
else
echo "Error: Refreshed DB file not found."
exit 1
fi
+3 -2
View File
@@ -23,7 +23,8 @@ jobs:
- name: Setup nodejs
uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # v6.2.0
- name: Install packages
run: npm install ajv
working-directory: utils/reply-schema-linter
run: npm ci --ignore-scripts
- name: linter
run: node ./utils/reply_schema_linter.js
run: NODE_PATH=utils/reply-schema-linter/node_modules node ./utils/reply_schema_linter.js
+41
View File
@@ -0,0 +1,41 @@
name: OpenSSF Scorecard supply-chain security
on:
push:
branches: [unstable]
schedule:
- cron: '0 0 * * 1'
permissions: read-all
jobs:
analysis:
name: Scorecard analysis
runs-on: ubuntu-latest
permissions:
security-events: write
id-token: write
steps:
- name: "Checkout code"
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd #v6.0.2
with:
persist-credentials: false
- name: "Run analysis"
uses: ossf/scorecard-action@4eaacf0543bb3f2c246792bd56e8cdeffafb205a #v2.4.3
with:
results_file: results.sarif
results_format: sarif
publish_results: true
- name: "Upload artifact"
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f #v6.0.0
with:
name: SARIF file
path: results.sarif
retention-days: 5
- name: "Upload to code-scanning"
uses: github/codeql-action/upload-sarif@6bc82e05fd0ea64601dd4b465378bbcf57de0314 #v4.32.1
with:
sarif_file: results.sarif
+273
View File
@@ -0,0 +1,273 @@
name: Test Failure Detector
on:
workflow_run:
workflows: ["Daily"]
branches:
- unstable
types:
- completed
permissions:
contents: read
issues: write
actions: read
jobs:
detect-test-failures:
runs-on: ubuntu-latest
if: >
(github.event.workflow_run.event == 'workflow_dispatch' ||
(github.event.workflow_run.event == 'schedule' && github.repository == 'valkey-io/valkey')) &&
github.event.workflow_run.conclusion != 'cancelled'
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
# Step 1: Download consolidated artifact
- name: Download failures
id: download
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
with:
script: |
const fs = require('fs');
const runId = context.payload.workflow_run.id;
console.log(`Fetching artifacts from run ${runId}`);
const artifacts = await github.paginate(
github.rest.actions.listWorkflowRunArtifacts,
{
owner: context.repo.owner,
repo: context.repo.repo,
run_id: runId,
},
(response) => response.data
);
const matched = artifacts.find(a => a.name === 'all-test-failures');
if (!matched) {
console.log('No all-test-failures artifact found');
core.setOutput('found', 'false');
return;
}
console.log(`Downloading: ${matched.name}`);
const download = await github.rest.actions.downloadArtifact({
owner: context.repo.owner,
repo: context.repo.repo,
artifact_id: matched.id,
archive_format: 'zip',
});
fs.writeFileSync('all-test-failures.zip', Buffer.from(download.data));
core.setOutput('found', 'true');
- name: Unzip artifact
if: steps.download.outputs.found == 'true'
run: |
unzip -o all-test-failures.zip
echo "=== Content ==="
cat all-test-failures.json
# Step 2: Get per-job URLs
- name: Get job URLs
if: steps.download.outputs.found == 'true'
id: jobs
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
with:
script: |
const fs = require('fs');
const runId = context.payload.workflow_run.id;
const jobs = await github.paginate(
github.rest.actions.listJobsForWorkflowRun,
{
owner: context.repo.owner,
repo: context.repo.repo,
run_id: runId,
},
(response) => response.data
);
console.log(`Found ${jobs.length} jobs`);
const jobUrlMap = {};
for (const job of jobs) {
jobUrlMap[job.name] = job.html_url;
const normalized = job.name
.replace(/\s*\(([^)]+)\)/g, '-$1')
.replace(/\s+/g, '-');
if (normalized !== job.name) {
jobUrlMap[normalized] = job.html_url;
console.log(`${job.name} -> ${job.html_url} (also: ${normalized})`);
} else {
console.log(`${job.name} -> ${job.html_url}`);
}
}
fs.writeFileSync('job-urls.json', JSON.stringify(jobUrlMap, null, 2));
# Step 3: Parse and merge failures
- name: Merge failures
if: steps.download.outputs.found == 'true'
id: merge
run: |
python3 << 'EOF'
import json, os, sys
with open("job-urls.json") as f:
job_urls = json.load(f)
with open("all-test-failures.json") as f:
all_failures = json.load(f)
grouped = {}
for job_name, suites in all_failures.items():
for suite_name, entries in suites.items():
for entry in entries:
name = f"{entry['test_name']} in {entry['test_file']}"
if name not in grouped:
grouped[name] = {
"name": name,
"test_name": entry["test_name"],
"test_file": entry["test_file"],
"error": entry.get("error", ""),
"jobs": []
}
# Deduplicate: skip if this job already recorded for this test
if job_name not in [j["job"] for j in grouped[name]["jobs"]]:
grouped[name]["jobs"].append({
"job": job_name,
"suite": suite_name,
"url": job_urls.get(job_name, "")
})
print(f"{name} in {job_name}/{suite_name}")
unique_failures = list(grouped.values())
print(f"\nTotal unique failures: {len(unique_failures)}")
with open("failures.json", "w") as f:
json.dump(unique_failures, f, indent=2)
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
f.write(f"has_failures={'true' if unique_failures else 'false'}\n")
EOF
# Step 4: Create or update issues
- name: Create or update issues
if: steps.merge.outputs.has_failures == 'true'
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
with:
script: |
const fs = require('fs');
const failures = JSON.parse(fs.readFileSync('failures.json', 'utf8'));
const label = 'test-failure';
// Ensure label exists
try {
await github.rest.issues.getLabel({
owner: context.repo.owner,
repo: context.repo.repo,
name: label,
});
} catch (e) {
if (e.status === 404) {
await github.rest.issues.createLabel({
owner: context.repo.owner,
repo: context.repo.repo,
name: label,
color: 'e11d48',
description: 'Test failure detected by CI',
});
}
}
const existingIssues = await github.paginate(
github.rest.issues.listForRepo,
{
owner: context.repo.owner,
repo: context.repo.repo,
labels: label,
state: 'open',
},
(response) => response.data
);
const today = new Date().toISOString().split('T')[0];
for (const failure of failures) {
const title = `[TEST-FAILURE] ${failure.test_name} in ${failure.test_file}`;
const envList = failure.jobs.map(j => j.job);
const ciLinks = failure.jobs.map(j => `- \`${j.job}\`: [CI link](${j.url})`).join('\n');
const envLine = `**Environments:** ${envList.map(e => '`' + e + '`').join(', ')}`;
const existing = existingIssues.find(i => i.title === title);
if (existing) {
console.log(`Found existing issue #${existing.number} for ${failure.name}`);
const envMatch = existing.body.match(/\*\*Environments:\*\*\s*(.+)/);
const envInner = envMatch ? envMatch[1].match(/`([^`]+)`/g) : null;
const existingEnvs = envInner
? envInner.map(e => e.replace(/`/g, ''))
: [];
const newEnvs = envList.filter(e => !existingEnvs.includes(e));
if (newEnvs.length > 0) {
console.log(` New environments: ${newEnvs.join(', ')}`);
const allEnvs = [...existingEnvs, ...newEnvs];
const newEnvLine = `**Environments:** ${allEnvs.map(e => '`' + e + '`').join(', ')}`;
const updatedBody = existing.body.replace(/\*\*Environments:\*\*\s*.+/, newEnvLine);
await github.rest.issues.update({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: existing.number,
body: updatedBody,
});
}
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: existing.number,
body: `Test failed again on ${today}.\n\n**Failed in:**\n${ciLinks}`,
});
} else {
console.log(`Creating issue for ${failure.name}`);
await github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
title: title,
labels: [label],
body: [
`**Summary**`,
``,
`\`${failure.test_name}\` in \`${failure.test_file}\` is failing in CI.`,
``,
`**Failing test(s)**`,
``,
`- Test name: \`${failure.test_name}\``,
`- Test file: \`${failure.test_file}\``,
`- CI link(s):`,
ciLinks,
``,
`**Error stack trace**`,
``,
'```',
failure.error || 'N/A',
'```',
``,
envLine,
``,
`---`,
`*Auto-created by Test Failure Detector*`,
].join('\n'),
});
}
}
console.log(`Done. Processed ${failures.length} failure(s).`);
+5 -2
View File
@@ -1,12 +1,15 @@
name: Trigger Build Release
permissions:
contents: read
on:
release:
types: [published]
workflow_dispatch:
inputs:
version:
description: Version of KV to build
description: Version of Valkey to build
required: true
environment:
description: Environment to build
@@ -41,7 +44,7 @@ jobs:
uses: peter-evans/repository-dispatch@28959ce8df70de7be546dd1250a005dd32156697 # v4.0.1
with:
token: ${{ secrets.AUTOMATION_PAT }}
repository: ${{ github.repository_owner }}/kv-release-automation
repository: ${{ github.repository_owner }}/valkey-release-automation
event-type: build-release
client-payload: >
{
+5 -5
View File
@@ -4,7 +4,7 @@ on:
- cron: '0 6 * * 0'
workflow_dispatch: {}
permissions:
actions: read
actions: write
contents: read
pull-requests: read
concurrency:
@@ -12,7 +12,7 @@ concurrency:
cancel-in-progress: false
jobs:
determine-release-branches:
if: github.repository == 'kv-io/kv'
if: github.repository == 'valkey-io/valkey'
runs-on: ubuntu-latest
outputs:
branches: ${{ steps.release-branches.outputs.branches }}
@@ -49,9 +49,9 @@ jobs:
core.setOutput('has-branches', releaseBranches.length > 0 ? 'true' : 'false');
run-daily-for-release-branches:
needs: determine-release-branches
if: needs.determine-release-branches.outputs.has-branches == 'true' && github.repository == 'kv-io/kv'
if: needs.determine-release-branches.outputs.has-branches == 'true' && github.repository == 'valkey-io/valkey'
permissions:
actions: read
actions: write
contents: read
pull-requests: read
strategy:
@@ -61,7 +61,7 @@ jobs:
max-parallel: 1
uses: ./.github/workflows/daily.yml
with:
use_repo: kv-io/kv
use_repo: valkey-io/valkey
use_git_ref: ${{ matrix.release-branch }}
skipjobs: ""
skiptests: ""
+3 -1
View File
@@ -15,7 +15,7 @@ dump*.rdb
*-cli
*-sentinel
*-server
*-unit-tests
*-unit-gtests
doc-tools
release
misc/*
@@ -56,3 +56,5 @@ build-debug/
build-release/
cmake-build-debug/
cmake-build-release/
__pycache__
src/unit/.flags
+225 -11
View File
@@ -1,16 +1,230 @@
Hello! This file is just a placeholder, since this is the "unstable" branch
of KV, the place where all the development happens.
Valkey 9.1 release notes
========================
There is no release notes for this branch, it gets forked into another branch
every time there is a partial feature freeze in order to eventually create
a new stable release.
Upgrade urgency levels:
Usually "unstable" is stable enough for you to use it in development environments
however you should never use it in production environments. It is possible
to download the latest stable release here:
| Level | Meaning |
|----------|---------------------------------------------------------------------|
| LOW | No need to upgrade unless there are new features you want to use. |
| MODERATE | Program an upgrade of the server, but it's not urgent. |
| HIGH | There is a critical bug that may affect a subset of users. Upgrade! |
| CRITICAL | There is a critical bug affecting MOST USERS. Upgrade ASAP. |
| SECURITY | There are security fixes in the release. |
https://kv.io/download/
Valkey 9.1.0 GA - Released Tue May 19 2026
---------------------
More information is available at https://kv.io
Upgrade urgency LOW: This is the first stable release of Valkey 9.1.
Happy hacking!
### Security fixes
* (CVE-2026-23479) Use-After-Free in unblock client flow
* (CVE-2026-25243) Invalid Memory Access in RESTORE command
* (CVE-2026-23631) Use-after-free when full sync occurs during a yielding Lua/function execution
### New Features and enhanced behavior
* Add cluster bus network traffic usage metric in bytes by @hpatro (#3396)
* Reduce latency spikes during rehashing via incremental page release by @chzhoo (#3481)
### Bug Fixes
* Fix(syncio): Set errno on EOF in syncRead and propagate to conn->last by @abmathur-ie (#3580)
* Fix GEOSEARCH BYPOLYGON leak on invalid COUNT by @bandalgomsu (#3568)
* Handle NULL pointer in streamTrim listpack delta calculation by @smkher (#3591)
* Fixes server crash when RDMA benchmark clients disconnect by @quanyeyang (#3448)
* Fix the memory leak in valkey-benchmark by @nmvk (#3643)
Valkey 9.1.0-rc2 - Released Tue Apr 28 00:00:00 2026
-------------------------------------
Upgrade urgency LOW: This is the second release candidate of Valkey 9.1.0.
### Behavior Changes
* Revert strict TLS certificate validation at config load as it is a breaking change, deferred to next major version (#3572)
### New Features and enhanced behavior
* Do the failover immediately if the replica is the best ranked replica by @enjoy-binbin (#2227)
* Add `cluster-config-save-behavior` option to control nodes.conf save behavior by @enjoy-binbin (#3372)
* Lua scripting engine is now statically linked by default instead of dynamically linked by @eifrah-aws (#3392)
* Module command result callback addition by @martinrvisser (#2936)
### Performance and Efficiency improvements
* Redesign IO threading communication model with lock-free queues (8-17% throughput gain) by @akashkgit (#3324)
* Increase embedded string threshold from 64 to 128 bytes (30% GET throughput gain) by @Nikhil-Manglore (#3397)
* ARM NEON SIMD optimization for pvFind() in vset.c (2-3x speedup) by @ahmadbelb (#3033)
* Optimize WATCH duplicate key check from O(N) to O(1) using per-db hashtable by @enjoy-binbin (#3360)
* Optimize `CLUSTERSCAN MATCH` so that it uses a specific slot if given by @nmvk (#3380)
* Improve COB memory tracking with copy avoidance by @dvkashapov (#3306)
### Bug Fixes
* Fix `valkey-cli --cluster del-node` for unreachable nodes by @yang-z-o (#3209)
* Enhance cluster stale packet detection to prevent sub-replica and empty primary by @zhijun42 (#2811)
* Big endian bitmap byte order mismatch fix by @nmvk (#3401)
* Fix slot-migration-max-failover-repl-bytes unable to accept -1 by @enjoy-binbin (#3443)
* Fix config rewrite producing negative values for unsigned memory configs by @enjoy-binbin (#3440)
* Fix HPERSIST RESP protocol violation on wrong-type key by @madolson (#3516)
* Fix lua-enable-insecure-api default value cannot be changed to yes by @enjoy-binbin (#3548)
Valkey 9.1.0-rc1 - Released Tue Mar 17 00:00:00 2026
-------------------------------------
Upgrade urgency LOW: This is the first release candidate of Valkey 9.1.0, with
new features, performance improvements, and bug fixes.
### New Features and enhanced behavior
* Database-level access control by @dvkashapov (#2309)
* Move Lua scripting engine into a Valkey module by @rjd15372 (#2858)
* Support cross node consistency for `SCAN` commands through configurable DB hash seed by @sarthakaggarwal97 (#2608)
* Support automatic TLS reload by @yang-z-o (#3020)
* Support TLS authentication using SAN URI by @yang-z-o (#3078)
* Prevent invalid TLS certificates from being loaded by @yang-z-o (#2999)
* Failing to save the cluster config file will no longer exit the process by @enjoy-binbin (#1032)
### Command and API updates
* Makes `CLUSTER KEYSLOT` available in standalone mode by @stockholmux (#3040)
* Update HSETEX so that it always issue keyspace notifications after validation by @ranshid (#3001)
* Adds HGETDEL command by @roshkhatri (#2851)
* Support NX/XX flag in HSETEX command by @hanxizh9910 (#2668)
* New `CLUSTERSCAN` command for cluster-wide key scanning across nodes by @nmvk (#2934)
* New `MSETEX` command to set multiple keys with a shared expiration by @enjoy-binbin (#3121)
* `CLUSTER SHARDS`/`CLUSTER SLOTS` now include an `availability-zone` field by @bandalgomsu (#3156)
### Performance and Efficiency improvements
* Optimize zset memory usage by embedding element in skiplist by @chzhoo (#2508)
* Remove internal server object pointer overhead in small strings by @rainsupreme (#2516)
* Optimize skiplist query efficiency by embedding the skiplist header by @chzhoo (#2867)
* Improve performance during rehashing by @chzhoo (#3073)
* Optimize SREM/ZREM/HDEL to pause auto shrink when deleting multiple items by @enjoy-binbin (#3144)
* Abort and swap the tables if ht1 is very full during the hashtable shrink rehashing by @enjoy-binbin (#3175)
* Improve performance of copy avoidance when command reply tracking is disabled by @dvkashapov (#3086)
* Enable hardware clock by default by @dvkashapov (#3103)
* Improve `COMMAND` performance by caching responses by @ebarskiy (#2839)
* Add support for asynchronous freeing of keys on writable replicas by @Scut-Corgis (#2849)
* Faster `XRANGE`/`XREVRANGE` via stream range hot-path optimization by @nesty92 (#3002)
* Replicas can reuse the RDB file as AOF preamble after disk-based full sync by @RayaCoo (#1901)
### Module API changes
* Add ValkeyModule_ClusterKeySlotC by @bandalgomsu (#2984)
* Add more client info flags to module API by @martinrvisser (#2868)
* Add prefix-aware ACL permission checks and new module API by @eifrah-aws (#2796)
* Support unsigned 64-bit numeric config values in module API by @artikell (#1546)
### Observability and logging
* Cumulative metrics for active I/O threads usage by @deepakrn (#2463)
* Cumulative metric for active main thread usage by @dvkashapov (#2931)
* Support whole cluster info for INFO command in cluster_info section by @soloestoy, @ranshid (#2876, #2964)
* Add remaining_repl_size field in `CLUSTER GETSLOTMIGRATIONS` output by @enjoy-binbin (#3135)
* Add logging helper function to print node's ip:port when nodename not explicitly set by @zhijun42 (#2777)
* Dual-channel-replication announces itself at replica-announce-ip if configured by @jdheyburn (#2846)
* Show replica dual-channel replication buffer memory in `INFO MEMORY` and `MEMORY STATS` by @enjoy-binbin (#2924)
* Add `rdb_transmitted` state to replica state in `INFO` by @enjoy-binbin (#2833)
* New INFO section for scripting engines by @rjd15372 (#2738)
* Adding json support for log-format config by @jbergstroem (#1791)
* Add server side TLS certificate expiry tracking and INFO telemetry by @YiwenZhang12 (#2913)
* Add option to use libbacktrace for backtraces in crash reports by @rainsupreme (#3034)
### Build and Tooling
* Show RPS histogram in valkey-benchmark by @hanxizh9910 (#2471)
* Add --warmup and --duration parameters to valkey-benchmark by @rainsupreme (#2581)
* Lazy loading of RDMA libs in CLI/Benchmark when building as module by @Ada-Church-Closure (#3072)
* Add support for atomic slot migration to valkey-cli by @murphyjacob4 (#2755)
* Replace C++ fast_float dependency with pure C implementation (ffc) by @lemire (#3329)
### Bug Fixes
* Strictly check CRLF when parsing querybuf by @enjoy-binbin (#2872)
### Contributors
* Abhishek Mathur @abmathur-ie
* Adam Fowler @adam-fowler
* Aditya Teltia @AdityaTeltia
* Akash Kumar @akashkgit
* Alina Liu @asagege
* Allen Samuels @allenss-amazon
* Alon Arenberg @alon-arenberg
* aradz44 @aradz44
* Arthur Lee @arthurkiller
* bandalgomsu @bandalgomsu
* Baswanth @baswanth09
* Benson-li @li-benson
* Binbin @enjoy-binbin
* Björn Svensson @bjosv
* bpint @bpint
* chenshi @chenshi5012
* chzhoo @chzhoo
* cjx-zar @cjx-zar
* Daejun Kim @djk1027
* Daniil Kashapov @dvkashapov
* Deepak Nandihalli @deepakrn
* Diego Ciciani @diegociciani
* eifrah-aws @eifrah-aws
* Evgeny Barskiy @ebarskiy
* FAN PEI @fanpei91
* Gabi Ganam @gabiganam
* Gagan H R @gaganhr94
* Hanxi Zhang @hanxizh9910
* Harkrishn Patro @hpatro
* Harry Lin @harrylin98
* hieu2102 @hieu2102
* Jacob Murphy @murphyjacob4
* Jeff Duffy @jaduffy
* jiegang0219 @jiegang0219
* Jim Brunner @JimB123
* Johan Bergström @jbergstroem
* John @johnufida
* Joseph Heyburn @jdheyburn
* Jun Yeong Kim @junyeong0619
* Katie Holly @Fusl
* Ken @otherscase
* korjeek @korjeek
* Kurt McKee @kurtmckee
* Kyle J. Davis @stockholmux
* Kyle Kim @kyle-yh-kim
* Leon Anavi @leon-anavi
* Madelyn Olson @madolson
* Mangat Singh Toor @immangat
* Marc Jakobi @mrcjkb
* martinrvisser @martinrvisser
* Marvin Rösch @marvinroesch
* Murad Shahmammadli @MuradSh
* NAM UK KIM @namuk2004
* Nikhil Manglore @Nikhil-Manglore
* Ouri Half @ouriamzn
* Patrik Hermansson @phermansson
* Ping Xie @PingXie
* Quanye Yang @Ada-Church-Closure
* Quanye Yang @quanyeyang
* Raghav Muddur @nmvk
* Rain Valentine @rainsupreme
* Ran Shidlansik @ranshid
* Ricardo Dias @rjd15372
* Ritoban Dutta @ritoban23
* Roshan Khatri @roshkhatri
* ruihong123 @ruihong123
* Sachin Venkatesha Murthy @sachinvmurthy
* sananes @yaronsananes
* Sarthak Aggarwal @sarthakaggarwal97
* Satheesha CH Gowda @satheesha
* Saurabh K @smkher
* Seungmin Lee @sungming2
* Shinobu Nunotaba @Ada-Church-Closure
* Simon Baatz @gmbnomis
* skyfirelee @artikell
* Sourav Singh Rawat @frostzt
* stydxm @stydxm
* Ted Lyngmo @TedLyngmo
* Tony Wooster @twooster
* uriyage @uriyage
* Vadym Khoptynets @poiuj
* Venkat Pamulapati @ChiliPaneer
* Viktor Söderqvist @zuiderkwast
* Vitah Lin @vitahlin
* Vitali @VitalyAR
* withRiver @withRiver
* wxmzy88 @wxmzy88
* xbasel @xbasel
* Yair Gottdenker @yairgott
* Yana Molodetsky @yanamolo
* Yang Zhao @yang-z-o
* Yiwen Zhang @YiwenZhang12
* yzc-yzc @yzc-yzc
* zhaozhao.zz @soloestoy
* Zhijun Liao @zhijun42
-1
View File
@@ -1 +0,0 @@
LLM.md
+56
View File
@@ -0,0 +1,56 @@
# AGENTS.md
## Scope
- These instructions apply to the entire repository unless a deeper `AGENTS.md` overrides them.
## Repo overview
- This is the Valkey server codebase.
- The main implementation lives under `src/`.
- Unit tests live under `src/unit`
- Integration tests live under `tests/`.
- Top-level `Makefile` forwards most targets into `src/Makefile`.
## Working guidelines
- Keep changes minimal and easy to backport.
- Match the style of the surrounding code instead of introducing new patterns.
- Avoid unrelated refactors in the same change.
## Build
- Default build: `make`
- Clean rebuild when build settings or bundled deps change: `make distclean && make`
## Unit tests
- Unit tests live under `src/unit/` and use GoogleTest (gtest/gmock).
- Build and run all unit tests: `make -C src test-unit`
- Unit tests cover data-structure and low-level logic changes.
- A single test filter can be run with: `make -C src test-unit && ./src/unit/valkey-unit-gtests --gtest_filter='<TestSuite>.<TestName>'`
## Integration tests
- Integration tests live under `tests/` and are written in Tcl.
- Run the full integration suite: `make test` (from the repo root).
- Run a single test file: `./runtest --single <path/to/test.tcl>`
- Additional specialized suites:
- Cluster tests: `./runtest-cluster`
- Sentinel tests: `./runtest-sentinel`
- Module API tests: `./runtest-moduleapi`
- For targeted validation, run the smallest relevant test scope first before broader suites.
## Code style
- Follow the repository conventions described in `DEVELOPMENT_GUIDE.md`.
- Most formatting is enforced by `clang-format`.
- CI uses `clang-format-18` across `*.c`, `*.h`, `*.cpp`, and `*.hpp` files.
- When touching C/C++ sources or headers, run `clang-format-18 -i` on the modified files before finalizing when the tool is available.
- Use comments for non-obvious behavior and rationale, not for restating code.
## Tests
- Code changes should include relevant tests when the repo already has a matching test location.
- Data-structure and low-level logic changes usually belong in `src/unit/` (C++ gtest).
- End-to-end behavior changes usually belong in `tests/` (Tcl integration tests).
- If behavior or commands change, check whether related documentation also needs updating.
## Files to avoid touching unless required
- Do not commit local runtime artifacts such as `dump.rdb`, `nodes.conf`, `*.log`, or ad hoc cluster directories unless the task explicitly requires them.
- Treat vendored dependency code under `deps/` as special-case changes; modify it only when the task clearly requires it.
## Pull Requests
Always push to the user's fork. Never push to the upstream valkey-io/valkey repository. Never push directly to unstable. If a user fork does not exist, ask the contributor to create one.
-1
View File
@@ -1 +0,0 @@
LLM.md
+6 -6
View File
@@ -13,20 +13,20 @@ if (APPLE)
endif ()
# Options
option(BUILD_LUA "Build KV Lua scripting engine" ON)
option(BUILD_UNIT_TESTS "Build kv-unit-tests" OFF)
set(BUILD_LUA "static" CACHE STRING "Build Valkey Lua scripting engine: static (default), module, no")
option(BUILD_UNIT_GTESTS "Build valkey-unit-gtests" OFF)
option(BUILD_TEST_MODULES "Build all test modules" OFF)
option(BUILD_EXAMPLE_MODULES "Build example modules" OFF)
set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake/Modules/")
project("kv")
project("valkey")
add_compile_options(-Wundef)
set(CMAKE_C_STANDARD 11)
set(CMAKE_C_STANDARD_REQUIRED ON)
set(CMAKE_C_EXTENSIONS ON)
include(KVSetup)
include(ValkeySetup)
add_subdirectory(src)
add_subdirectory(tests)
@@ -40,7 +40,7 @@ unset(CLANGPP CACHE)
unset(CLANG CACHE)
unset(BUILD_RDMA_MODULE CACHE)
unset(BUILD_TLS_MODULE CACHE)
unset(BUILD_UNIT_TESTS CACHE)
unset(BUILD_UNIT_GTESTS CACHE)
unset(BUILD_TEST_MODULES CACHE)
unset(BUILD_EXAMPLE_MODULES CACHE)
unset(USE_TLS CACHE)
@@ -63,7 +63,7 @@ function(copy_runtest_script script_name)
set(insert_content
"# Most tests assume running from the project root
cd ${CMAKE_SOURCE_DIR}
export KV_BIN_DIR=\"${CMAKE_BINARY_DIR}/bin\"
export VALKEY_BIN_DIR=\"${CMAKE_BINARY_DIR}/bin\"
")
# Reconstruct the full script
set(new_contents "${shebang_line}${insert_content}${script_body}")
+1 -1
View File
@@ -49,7 +49,7 @@ representative at an online or offline event.
Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
this email address: maintainers@hanzo.ai.
this email address: maintainers@lists.valkey.io.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
+16 -16
View File
@@ -1,23 +1,23 @@
Contributing to KV
Contributing to Valkey
======================
Welcome and thank you for wanting to contribute!
# Project governance
The KV project is led by a Technical Steering Committee, whose responsibilities are laid out in [GOVERNANCE.md](GOVERNANCE.md).
The Valkey project is led by a Technical Steering Committee, whose responsibilities are laid out in [GOVERNANCE.md](GOVERNANCE.md).
## Get started
* Have a question? Ask it on
[GitHub Discussions](https://github.com/hanzoai/kv/discussions)
or [KV's Discord](https://discord.gg/zbcPa5umUB)
or [KV's Matrix](https://matrix.to/#/#kv:matrix.org)
* Found a bug? [Report it here](https://github.com/hanzoai/kv/issues/new?template=bug_report.md&title=%5BBUG%5D)
* KV crashed? [Submit a crash report here](https://github.com/hanzoai/kv/issues/new?template=crash_report.md&title=%5BCRASH%5D+%3Cshort+description%3E)
* Suggest a new feature? [Post your detailed feature request here](https://github.com/hanzoai/kv/issues/new?template=feature_request.md&title=%5BNEW%5D)
* Report a test failure? [Report it here](https://github.com/hanzoai/kv/issues/new?template=test-failure.md)
* Want to help with documentation? [Move on to kv-doc](https://github.com/hanzoai/kv-doc)
[GitHub Discussions](https://github.com/valkey-io/valkey/discussions)
or [Valkey's Discord](https://discord.gg/zbcPa5umUB)
or [Valkey's Matrix](https://matrix.to/#/#valkey:matrix.org)
* Found a bug? [Report it here](https://github.com/valkey-io/valkey/issues/new?template=bug_report.md&title=%5BBUG%5D)
* Valkey crashed? [Submit a crash report here](https://github.com/valkey-io/valkey/issues/new?template=crash_report.md&title=%5BCRASH%5D+%3Cshort+description%3E)
* Suggest a new feature? [Post your detailed feature request here](https://github.com/valkey-io/valkey/issues/new?template=feature_request.md&title=%5BNEW%5D)
* Report a test failure? [Report it here](https://github.com/valkey-io/valkey/issues/new?template=test-failure.md)
* Want to help with documentation? [Move on to valkey-doc](https://github.com/valkey-io/valkey-doc)
* Report a vulnerability? See [SECURITY.md](SECURITY.md)
## Developer Certificate of Origin
@@ -58,7 +58,7 @@ By making a contribution to this project, I certify that:
involved.
```
We require that every contribution to KV to be signed with a DCO. We require the
We require that every contribution to Valkey to be signed with a DCO. We require the
usage of known identity (such as a real or preferred name). We do not accept anonymous
contributors nor those utilizing pseudonyms. A DCO signed commit will contain a line like:
@@ -72,7 +72,7 @@ user.name and user.email are set in your git configs, you can use `git commit` w
or `--signoff` to add the `Signed-off-by` line to the end of the commit message. We also
require revert commits to include a DCO.
If you're contributing code to the KV project in any other form, including
If you're contributing code to the Valkey project in any other form, including
sending a code fragment or patch via private email or public discussion groups,
you need to ensure that the contribution is in accordance with the DCO.
@@ -86,7 +86,7 @@ features to be accepted. Here you can see if there is consensus about your idea.
2. If in step 1 you get an acknowledgment from the project leaders, use the following
procedure to submit a patch:
1. Fork KV on GitHub ([HOWTO](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo))
1. Fork Valkey on GitHub ([HOWTO](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo))
1. Create a topic branch (`git checkout -b my_branch`)
1. Make the needed changes and commit with a DCO. (`git commit -s`)
1. Push to your branch (`git push origin my_branch`)
@@ -101,7 +101,7 @@ try to popularize it, have other users commenting and sharing their point of
view, and so forth. This helps.
4. While developing code, make sure to refer to our [DEVELOPMENT_GUIDE.md](DEVELOPMENT_GUIDE.md),
which includes documentation about various best practices for writing KV code.
which includes documentation about various best practices for writing Valkey code.
5. For minor fixes, open a pull request on GitHub.
@@ -117,7 +117,7 @@ Use [`.github/workflows/daily.yml`](.github/workflows/daily.yml) with
2. Click **Run workflow**.
3. In the **Branch** dropdown, select the branch that contains the workflow file you want to use.
4. In the input fields, set:
* `use_repo` to your fork (for example, `your-user/kv`)
* `use_repo` to your fork (for example, `your-user/valkey`)
* `use_git_ref` to your branch name (or a specific commit SHA)
5. Optionally set `skipjobs`, `skiptests`, `test_args`, and `cluster_test_args`.
6. Click **Run workflow**.
@@ -125,7 +125,7 @@ Use [`.github/workflows/daily.yml`](.github/workflows/daily.yml) with
Notes:
* To run the full matrix, set `skipjobs` and `skiptests` to `none`.
Do not leave them empty, since the workflow input defaults may be applied.
* The scheduled part of this workflow is gated to `hanzoai/kv`, but manual
* The scheduled part of this workflow is gated to `valkey-io/valkey`, but manual
`workflow_dispatch` runs work for forks.
Thanks!
+1 -1
View File
@@ -4,7 +4,7 @@ SPDX-License-Identifier: BSD-3-Clause
BSD 3-Clause License
Copyright (c) 2024-present, KV contributors
Copyright (c) 2024-present, Valkey contributors
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+11 -11
View File
@@ -1,5 +1,5 @@
# KV development guidelines
This document provides a general overview for writing and designing code for KV.
# Valkey development guidelines
This document provides a general overview for writing and designing code for Valkey.
During our long development history, we've made a lot of inconsistent decisions, but we strive to get incrementally better.
## General Best practices
@@ -7,7 +7,7 @@ During our long development history, we've made a lot of inconsistent decisions,
We do a lot of backporting as a project, and the more lines changed, the higher the chance of having to resolve merge conflicts.
Please separate refactoring and functional changes into separate PRs, to make it easier to handle backporting.
1. Avoid adding configuration when a feature can be fully controlled by heuristics.
We want KV to work correctly out of the box without much tuning.
We want Valkey to work correctly out of the box without much tuning.
Configurations can be added to provide additional tuning of features.
When the workload characteristics can't be inferred or imply a tradeoff (CPU vs memory), then provide a configuration.
@@ -33,7 +33,7 @@ Most of the style guidelines are enforced by clang format, but some additional c
For historical reasons, some functions used the integer type, and they are kept as is to make it easier to backport changes.
## Naming conventions
KV has a long history of inconsistent naming conventions.
Valkey has a long history of inconsistent naming conventions.
Generally follow the style of the surrounding code, but you can also always use the following conventions for variable and structure names:
- Variable names: `snake_case` or all lower case for short names (e.g. `cached_reply` or `keylen`).
@@ -45,7 +45,7 @@ Generally follow the style of the surrounding code, but you can also always use
When creating new source code files, use the following snippet to indicate the license:
```
/*
* Copyright (c) KV Contributors
* Copyright (c) Valkey Contributors
* All rights reserved.
* SPDX-License-Identifier: BSD-3-Clause
*/
@@ -55,7 +55,7 @@ If you are making material changes to a file that has a different license at the
There isn't a well defined test for what is considered a material change, but a good rule of thumb is that material changes are more than 100 lines of code.
## Test coverage
KV uses two types of tests: unit and integration tests.
Valkey uses two types of tests: unit and integration tests.
All contributions should include a test of some form.
Unit tests are present in the `src/unit` directory, and are intended to test individual structures or files.
@@ -66,10 +66,10 @@ Adding new commands should come with corresponding integration tests.
When writing cluster mode tests, do not use the legacy `tests/cluster` framework, which has been deprecated, and instead write tests in `unit/cluster`.
## Documentation
KV keeps most of the user documentation in the [kv-doc](https://github.com/hanzoai/kv-doc) repository in a few areas:
1. Major functionality is documented in the [topics](https://github.com/hanzoai/kv-doc/tree/main/topics) section.
1. Specific command behavior is documented in the [commands](https://github.com/hanzoai/kv-doc/tree/main/commands) section.
Command history is also documented in the [command json file](https://github.com/hanzoai/kv/tree/unstable/src/commands).
1. Server info fields are documented in the [INFO](https://github.com/hanzoai/kv-doc/blob/main/commands/info.md) command.
Valkey keeps most of the user documentation in the [valkey-doc](https://github.com/valkey-io/valkey-doc) repository in a few areas:
1. Major functionality is documented in the [topics](https://github.com/valkey-io/valkey-doc/tree/main/topics) section.
1. Specific command behavior is documented in the [commands](https://github.com/valkey-io/valkey-doc/tree/main/commands) section.
Command history is also documented in the [command json file](https://github.com/valkey-io/valkey/tree/unstable/src/commands).
1. Server info fields are documented in the [INFO](https://github.com/valkey-io/valkey-doc/blob/main/commands/info.md) command.
When a PR is opened that requires documentation to be updated, the `needs-doc-pr` should be added until the corresponding documentation PR is open.
-31
View File
@@ -1,31 +0,0 @@
ARG KV_VERSION=9
# Hanzo KV: High-performance key-value store
FROM kv/kv:${KV_VERSION}-alpine AS base
FROM base
LABEL maintainer="dev@hanzo.ai"
LABEL org.opencontainers.image.source="https://github.com/hanzoai/kv"
LABEL org.opencontainers.image.description="Hanzo KV - High-performance key-value store"
LABEL org.opencontainers.image.vendor="Hanzo AI"
# Install Hanzo KV CLI tools
# Primary names are kv-* ; legacy kv-* names remain as symlinks
RUN cp /usr/local/bin/kv-server /usr/local/bin/kv-server \
&& cp /usr/local/bin/kv-cli /usr/local/bin/kv-cli \
&& ln -sf /usr/local/bin/kv-cli /usr/local/bin/kv \
&& cp /usr/local/bin/kv-sentinel /usr/local/bin/kv-sentinel 2>/dev/null; \
cp /usr/local/bin/kv-benchmark /usr/local/bin/kv-benchmark 2>/dev/null; \
cp /usr/local/bin/kv-check-aof /usr/local/bin/kv-check-aof 2>/dev/null; \
cp /usr/local/bin/kv-check-rdb /usr/local/bin/kv-check-rdb 2>/dev/null; \
true
EXPOSE 6379
HEALTHCHECK --interval=15s --timeout=3s --start-period=10s --retries=3 \
CMD kv ping | grep -q PONG || exit 1
ENTRYPOINT ["kv-server"]
CMD ["--bind", "0.0.0.0", "--dir", "/data", "--maxmemory-policy", "allkeys-lru", "--protected-mode", "no"]
+14 -14
View File
@@ -1,17 +1,17 @@
# Project Governance
The KV project is managed by a Technical Steering Committee (TSC) composed of the maintainers of the KV repository.
The KV project includes all of the current and future repositories under the hanzoai organization.
The Valkey project is managed by a Technical Steering Committee (TSC) composed of the maintainers of the Valkey repository.
The Valkey project includes all of the current and future repositories under the Valkey-io organization.
Committers are defined as individuals with write access to the code within a repository.
Maintainers are defined as individuals with full access to a repository and own its governance.
Both maintainers and committers shall be clearly listed in the MAINTAINERS.md file in a given project's repository.
Maintainers of other repositories within the KV project are not members of the TSC unless explicitly added.
Maintainers of other repositories within the Valkey project are not members of the TSC unless explicitly added.
## Technical Steering Committee
The TSC is responsible for oversight of all technical, project, approval, and policy matters for KV.
The TSC is responsible for oversight of all technical, project, approval, and policy matters for Valkey.
The TSC members are listed in the [MAINTAINERS.md](MAINTAINERS.md) file in the KV repository.
The TSC members are listed in the [MAINTAINERS.md](MAINTAINERS.md) file in the Valkey repository.
At any time, no more than one third (1/3) of the TSC members may be employees, contractors, or representatives of the same organization or affiliated organizations.
For the purposes of this document, “organization” includes companies, corporations, universities, research institutes, non-profits, governmental institutions, and any of their subsidiaries or affiliates.
@@ -22,8 +22,8 @@ The TSC shall strive to resolve the situation within 30 days of notification, an
The TSC shall appoint a Chair responsible for organizing TSC meetings.
If the TSC Chair is removed from the TSC (or the Chair steps down from that role), it is the responsibility of the TSC to appoint a new Chair.
The TSC may, at its discretion, add or remove members who are not maintainers of the main KV repository.
The TSC may, at its discretion, add or remove maintainers from other repositories within the KV project.
The TSC may, at its discretion, add or remove members who are not maintainers of the main Valkey repository.
The TSC may, at its discretion, add or remove maintainers from other repositories within the Valkey project.
## Voting
@@ -33,12 +33,12 @@ Rather, the TSC shall determine consensus based on their good faith consideratio
The TSC shall document evidence of consensus in accordance with these requirements.
If consensus cannot be reached, the TSC shall make the decision by a vote.
A vote shall also be called when an issue or pull request is marked as a major decision, which are decisions that have a significant impact on the KV architecture or design.
A vote shall also be called when an issue or pull request is marked as a major decision, which are decisions that have a significant impact on the Valkey architecture or design.
### Technical Major Decisions
Technical major decisions include:
* Fundamental changes to the KV core datastructures
* Fundamental changes to the Valkey core datastructures
* Adding a new data structure or API
* Changes that affect backward compatibility
* New user visible fields that need to be maintained
@@ -56,7 +56,7 @@ Governance major decisions include:
* Adding TSC members or involuntary removal of TSC members
* Modifying this governance document
* Delegation of maintainership for projects or governance authority
* Creating, modifying, or removing roles within the KV project
* Creating, modifying, or removing roles within the Valkey project
* Any change that alters voting rules, TSC responsibilities, or project oversight
* Structural changes to the TSC, including composition limits
@@ -79,13 +79,13 @@ A maintainer's access (and accordingly, their position on the TSC) will be remov
* Resignation: Written notice of resignation to the TSC.
* Unreachable Member: If a member is unresponsive for more than six months, the remaining active members of the TSC may vote to remove the unreachable member by a simple majority.
## Technical direction for other KV projects
## Technical direction for other Valkey projects
The TSC may delegate decision making for other projects within the KV organization to the maintainers responsible for those projects.
The TSC may delegate decision making for other projects within the Valkey organization to the maintainers responsible for those projects.
Delegation of decision making for a project is considered a [Governance Major Decision](#governance-major-decisions).
Projects within the KV organization must indicate the individuals with commit permissions by updating the MAINTAINERS.md within their repositories.
Projects within the Valkey organization must indicate the individuals with commit permissions by updating the MAINTAINERS.md within their repositories.
The TSC may, at its discretion, overrule the decisions made by other projects within the KV organization, although they shall show restraint in doing so.
The TSC may, at its discretion, overrule the decisions made by other projects within the Valkey organization, although they shall show restraint in doing so.
## License of this document
-7
View File
@@ -1,7 +0,0 @@
# kv — AI Assistant Context
<p align="center">
<strong>Hanzo KV</strong>
</p>
<p align="center">
+2 -1
View File
@@ -16,7 +16,7 @@ Maintainers listed in alphabetical order by their github ID.
| ------------------- | ------------- | ----------- |
| Binbin Zhu | @enjoy-binbin | Tencent |
| Harkrishn Patro | @hpatro | Amazon |
| Lucas Yang | @lucasyonge | - |
| Lucas Yang | @lucasyonge | Percona |
| Madelyn Olson | @madolson | Amazon |
| Jacob Murphy | @murphyjacob4 | Google |
| Ping Xie | @pingxie | Oracle |
@@ -30,6 +30,7 @@ Committers listed in alphabetical order by their github ID.
| Committer | GitHub ID | Affiliation |
| ------------------- | ------------- | ----------- |
| Jim Brunner | @JimB123 | Amazon |
| Ricardo Dias | @rjd15372 | Percona |
## Former Maintainers and Committers
+369 -120
View File
@@ -1,145 +1,394 @@
<p align="center">
<strong>Hanzo KV</strong>
</p>
[![codecov](https://codecov.io/gh/valkey-io/valkey/graph/badge.svg?token=KYYSJAYC5F)](https://codecov.io/gh/valkey-io/valkey)
[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/valkey-io/valkey/badge)](https://securityscorecards.dev/viewer/?uri=github.com/valkey-io/valkey)
<p align="center">
High-performance key-value store for the Hanzo ecosystem.<br/>
In-memory data store used as database, cache, streaming engine, and message broker.
</p>
This project was forked from the open source Redis project right before the transition to their new source available licenses.
<p align="center">
<a href="https://github.com/hanzoai/kv/actions"><img src="https://github.com/hanzoai/kv/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
<a href="https://github.com/hanzoai/kv/releases"><img src="https://img.shields.io/github/v/release/hanzoai/kv" alt="Release"></a>
<a href="https://github.com/hanzoai/kv/blob/main/LICENSE"><img src="https://img.shields.io/github/license/hanzoai/kv" alt="License"></a>
</p>
This README is just a fast *quick start* document. More details can be found under [valkey.io](https://valkey.io/)
---
# What is Valkey?
## Features
Valkey is a high-performance data structure server that primarily serves key/value workloads.
It supports a wide range of native structures and an extensible plugin system for adding new data structures and access patterns.
- **In-memory key-value store** -- sub-millisecond reads and writes
- **Redis-compatible protocol** -- drop-in replacement for existing Redis clients
- **Persistence** -- RDB snapshots and AOF (append-only file) for durability
- **Replication** -- primary-replica with automatic failover via Sentinel
- **Lua scripting** -- server-side scripting for atomic operations
- **Pub/Sub** -- publish and subscribe messaging
- **Streams** -- append-only log data structure for event sourcing
- **Cluster mode** -- horizontal scaling with automatic sharding
- **Modules** -- extensible plugin system for custom data structures
- **ZAP native** -- built-in [ZAP binary protocol](https://github.com/luxfi/zap) on port 9653 (17x faster than JSON-RPC)
- **Multi-arch** -- linux/amd64 and linux/arm64
# Building Valkey using `Makefile`
## Quick Start
Valkey can be compiled and used on Linux, macOS, OpenBSD, NetBSD, FreeBSD.
We support big endian and little endian architectures, and both 32 bit
and 64 bit systems.
### Docker
It may compile on Solaris derived systems (for instance SmartOS) but our
support for this platform is *best effort* and Valkey is not guaranteed to
work as well as in Linux, macOS, and \*BSD.
It is as simple as:
% make
To build with TLS support, you'll need OpenSSL development libraries (e.g.
libssl-dev on Debian/Ubuntu).
To build TLS support as Valkey built-in:
% make BUILD_TLS=yes
To build TLS as Valkey module:
% make BUILD_TLS=module
Note that sentinel mode does not support TLS module.
To build with experimental RDMA support you'll need RDMA development libraries
(e.g. librdmacm-dev and libibverbs-dev on Debian/Ubuntu).
To build RDMA support as Valkey built-in:
% make BUILD_RDMA=yes
To build RDMA as Valkey module:
% make BUILD_RDMA=module
To build with systemd support, you'll need systemd development libraries (such
as libsystemd-dev on Debian/Ubuntu or systemd-devel on CentOS) and run:
% make USE_SYSTEMD=yes
To build with enhanced stack traces that include file names and line numbers
for all functions (including static functions), use libbacktrace:
% make USE_LIBBACKTRACE=yes
To build Valkey without the Lua engine:
% make BUILD_LUA=no
To append a suffix to Valkey program names, use:
% make PROG_SUFFIX="-alt"
You can build a 32 bit Valkey binary using:
% make 32bit
After building Valkey, it is a good idea to test it using:
% make test
The above runs the main integration tests. Additional tests are started using:
% make test-unit # Unit tests
% make test-modules # Tests of the module API
% make test-sentinel # Valkey Sentinel integration tests
% make test-cluster # Valkey Cluster integration tests
More about running the integration tests can be found in
[tests/README.md](tests/README.md) and for unit tests, see
[src/unit/README.md](src/unit/README.md).
## Performance monitoring
Valkey Performance Dashboards provide a consolidated view of throughput trends across versions, helping contributors validate improvements and identify regressions quickly.
- [Performance Overview](https://valkey.io/performance/) - Compare throughput across Valkey versions
- [Unstable Branch Dashboard](https://perf-dashboard.valkey.io/public-dashboards/3e45bf8ded3043edaa941331cd1a94e2) - Track performance of all commits in the unstable branch
## Fixing build problems with dependencies or cached build options
Valkey has some dependencies which are included in the `deps` directory.
`make` does not automatically rebuild dependencies even if something in
the source code of dependencies changes.
When you update the source code with `git pull` or when code inside the
dependencies tree is modified in any other way, make sure to use the following
command in order to really clean everything and rebuild from scratch:
% make distclean
This will clean: jemalloc, lua, libvalkey, linenoise and other dependencies.
Also if you force certain build options like 32bit target, no C compiler
optimizations (for debugging purposes), and other similar build time options,
those options are cached indefinitely until you issue a `make distclean`
command.
## Fixing problems building 32 bit binaries
If after building Valkey with a 32 bit target you need to rebuild it
with a 64 bit target, or the other way around, you need to perform a
`make distclean` in the root directory of the Valkey distribution.
In case of build errors when trying to build a 32 bit binary of Valkey, try
the following steps:
* Install the package libc6-dev-i386 (also try g++-multilib).
* Try using the following command line instead of `make 32bit`:
`make CFLAGS="-m32 -march=native" LDFLAGS="-m32"`
## Allocator
Selecting a non-default memory allocator when building Valkey is done by setting
the `MALLOC` environment variable. Valkey is compiled and linked against libc
malloc by default, with the exception of jemalloc being the default on Linux
systems. This default was picked because jemalloc has proven to have fewer
fragmentation problems than libc malloc.
To force compiling against libc malloc, use:
% make MALLOC=libc
To compile against jemalloc on Mac OS X systems, use:
% make MALLOC=jemalloc
## Monotonic clock
By default, Valkey uses the processor's internal instruction clock (TSC on x86,
CNTVCT on ARM) for monotonic time tracking, which provides approximately 3x
faster time access compared to POSIX clock_gettime (~10-30ns vs ~100ns).
This is enabled by default on supported architectures (x86_64 Linux and aarch64)
and automatically falls back to POSIX clock_gettime on unsupported systems.
For more information about processor clock usage, see:
http://oliveryang.net/2015/09/pitfalls-of-TSC-usage/
To disable the processor clock and force POSIX clock_gettime, use:
% make CFLAGS="-DNO_PROCESSOR_CLOCK"
## Verbose build
Valkey will build with a user-friendly colorized output by default.
If you want to see a more verbose output, use the following:
% make V=1
# Running Valkey
To run Valkey with the default configuration, just type:
% cd src
% ./valkey-server
If you want to provide your valkey.conf, you have to run it using an additional
parameter (the path of the configuration file):
% cd src
% ./valkey-server /path/to/valkey.conf
It is possible to alter the Valkey configuration by passing parameters directly
as options using the command line. Examples:
% ./valkey-server --port 9999 --replicaof 127.0.0.1 6379
% ./valkey-server /etc/valkey/6379.conf --loglevel debug
All the options in valkey.conf are also supported as options using the command
line, with exactly the same name.
# Running Valkey with TLS:
## Running manually
To manually run a Valkey server with TLS mode (assuming `./utils/gen-test-certs.sh`
was invoked so sample certificates/keys are available):
* TLS built-in mode:
```
./src/valkey-server --tls-port 6379 --port 0 \
--tls-cert-file ./tests/tls/valkey.crt \
--tls-key-file ./tests/tls/valkey.key \
--tls-ca-cert-file ./tests/tls/ca.crt
```
* TLS module mode:
```
./src/valkey-server --tls-port 6379 --port 0 \
--tls-cert-file ./tests/tls/valkey.crt \
--tls-key-file ./tests/tls/valkey.key \
--tls-ca-cert-file ./tests/tls/ca.crt \
--loadmodule src/valkey-tls.so
```
Note that you can disable TCP by specifying `--port 0` explicitly.
It's also possible to have both TCP and TLS available at the same time,
but you'll have to assign different ports.
Use `valkey-cli` to connect to the Valkey server:
```
./src/valkey-cli --tls \
--cert ./tests/tls/valkey.crt \
--key ./tests/tls/valkey.key \
--cacert ./tests/tls/ca.crt
```
Specifying `--tls-replication yes` makes a replica connect to the primary.
Using `--tls-cluster yes` makes Valkey Cluster use TLS across nodes.
# Running Valkey with RDMA:
Note that Valkey Over RDMA is an experimental feature.
It may be changed or removed in any minor or major version.
Currently, it is only supported on Linux.
* RDMA built-in mode:
```
./src/valkey-server --protected-mode no \
--rdma-bind 192.168.122.100 --rdma-port 6379
```
* RDMA module mode:
```
./src/valkey-server --protected-mode no \
--loadmodule src/valkey-rdma.so --rdma-bind 192.168.122.100 --rdma-port 6379
```
It's possible to change bind address/port of RDMA by runtime command:
192.168.122.100:6379> CONFIG SET rdma-port 6380
It's also possible to have both RDMA and TCP available, and there is no
conflict of TCP(6379) and RDMA(6379), Ex:
% ./src/valkey-server --protected-mode no \
--loadmodule src/valkey-rdma.so --rdma-bind 192.168.122.100 --rdma-port 6379 \
--port 6379
Note that the network card (192.168.122.100 of this example) should support
RDMA. To test a server supports RDMA or not:
% rdma res show (a new version iproute2 package)
Or:
% ibv_devices
# Playing with Valkey
You can use valkey-cli to play with Valkey. Start a valkey-server instance,
then in another terminal try the following:
% cd src
% ./valkey-cli
valkey> ping
PONG
valkey> set foo bar
OK
valkey> get foo
"bar"
valkey> incr mycounter
(integer) 1
valkey> incr mycounter
(integer) 2
valkey>
# Installing Valkey
In order to install Valkey binaries into /usr/local/bin, just use:
% make install
You can use `make PREFIX=/some/other/directory install` if you wish to use a
different destination.
_Note_: For compatibility with Redis, we create symlinks from the Redis names (`redis-server`, `redis-cli`, etc.) to the Valkey binaries installed by `make install`.
The symlinks are created in same directory as the Valkey binaries.
The symlinks are removed when using `make uninstall`.
The creation of the symlinks can be skipped by setting the makefile variable `USE_REDIS_SYMLINKS=no`.
`make install` will just install binaries in your system, but will not configure
init scripts and configuration files in the appropriate place. This is not
needed if you just want to play a bit with Valkey, but if you are installing
it the proper way for a production system, we have a script that does this
for Ubuntu and Debian systems:
% cd utils
% ./install_server.sh
_Note_: `install_server.sh` will not work on macOS; it is built for Linux only.
The script will ask you a few questions and will setup everything you need
to run Valkey properly as a background daemon that will start again on
system reboots.
You'll be able to stop and start Valkey using the script named
`/etc/init.d/valkey_<portnumber>`, for instance `/etc/init.d/valkey_6379`.
# Building using `CMake`
In addition to the traditional `Makefile` build, Valkey supports an alternative, **experimental**, build system using `CMake`.
To build and install `Valkey`, in `Release` mode (an optimized build), type this into your terminal:
```bash
docker run -d --name hanzo-kv -p 6379:6379 ghcr.io/hanzoai/kv
mkdir build-release
cd $_
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/valkey
sudo make install
# Valkey is now installed under /opt/valkey
```
### Connect
Other options supported by Valkey's `CMake` build system:
## Special build flags
- `-DBUILD_TLS=<yes|no>` enable TLS build for Valkey. Default: `no`
- `-DBUILD_RDMA=<no|module>` enable RDMA module build (only module mode supported). Default: `no`
- `-DBUILD_MALLOC=<libc|jemalloc|tcmalloc|tcmalloc_minimal>` choose the allocator to use. Default on Linux: `jemalloc`, for other OS: `libc`
- `-DBUILD_SANITIZER=<address|thread|undefined>` build with address sanitizer enabled. Default: disabled (no sanitizer)
- `-DBUILD_UNIT_GTESTS=[yes|no]` when set, the build will produce unit tests executable `valkey-unit-gtests`. Default: `no`
- `-DBUILD_TEST_MODULES=[yes|no]` when set, the build will include the modules located under the `tests/modules` folder. Default: `no`
- `-DBUILD_EXAMPLE_MODULES=[yes|no]` when set, the build will include the example modules located under the `src/modules` folder. Default: `no`
## Common flags
- `-DCMAKE_BUILD_TYPE=<Debug|Release...>` define the build type, see CMake manual for more details
- `-DCMAKE_INSTALL_PREFIX=/installation/path` override this value to define a custom install prefix. Default: `/usr/local`
- `-G"<Generator Name>"` generate build files for "Generator Name". By default, CMake will generate `Makefile`s.
## Verbose build
`CMake` generates a user-friendly colorized output by default.
If you want to see a more verbose output, use the following:
```bash
docker exec -it hanzo-kv kv
127.0.0.1:6379> SET hello world
OK
127.0.0.1:6379> GET hello
"world"
make VERBOSE=1
```
Any Redis-compatible CLI also works out of the box.
## Troubleshooting
### Build from Source
During the `CMake` stage, `CMake` caches variables in a local file named `CMakeCache.txt`. All variables generated by Valkey
are removed from the cache once consumed (this is done by calling to `unset(VAR-NAME CACHE)`). However, some variables,
like the compiler path, are kept in cache. To start a fresh build either remove the cache file `CMakeCache.txt` from the
build folder, or delete the build folder completely.
**It is important to re-run `CMake` when adding new source files.**
## Integration with IDE
During the `CMake` stage of the build, `CMake` generates a JSON file named `compile_commands.json` and places it under the
build folder. This file is used by many IDEs and text editors for providing code completion (via `clangd`).
A small caveat is that these tools will look for `compile_commands.json` under the Valkey's top folder.
A common workaround is to create a symbolic link to it:
```bash
make
make test
make install
cd /path/to/valkey/
# We assume here that your build folder is `build-release`
ln -sf $(pwd)/build-release/compile_commands.json $(pwd)/compile_commands.json
```
## CLI Tools
Restart your IDE and voila
| Command | Description |
|---------|-------------|
| `kv` | Interactive CLI (default) |
| `kv-server` | Start KV server |
| `kv-cli` | Command-line client |
| `kv-sentinel` | High-availability sentinel |
| `kv-benchmark` | Performance benchmarking |
| `kv-check-aof` | AOF file integrity check |
| `kv-check-rdb` | RDB file integrity check |
# Code contributions
## Configuration
Please see the [CONTRIBUTING.md][2]. For security bugs and vulnerabilities, please see [SECURITY.md][3].
Pass a config file at startup:
# Valkey is an open community project under LF Projects
```bash
kv-server /etc/kv/kv.conf
```
Valkey a Series of LF Projects, LLC
2810 N Church St, PMB 57274
Wilmington, Delaware 19802-4447
Or set options via command line:
```bash
kv-server --port 6379 --maxmemory 256mb --appendonly yes
```
## ZAP Binary Protocol
Hanzo KV speaks [ZAP](https://github.com/luxfi/zap) natively on port **9653** — no sidecar needed.
ZAP is a zero-copy binary protocol that's 17x faster than JSON-RPC with 11x less memory usage.
### Enable ZAP
ZAP is enabled by default. Load the module:
```bash
kv-server --loadmodule /path/to/zap.so
# or with custom port:
kv-server --loadmodule /path/to/zap.so PORT 9653
```
### ZAP Operations
| Path | Body | Description |
|------|------|-------------|
| `/get` | `{"key":"mykey"}` | GET a key |
| `/set` | `{"key":"mykey","value":"myval"}` | SET a key |
| `/del` | `{"key":"mykey"}` | DEL a key |
| `/cmd` | `{"cmd":"PING","args":[]}` | Execute any command |
### Module API
Develop custom modules using the KV Module API:
```c
#include "kvmodule.h"
int KVModule_OnLoad(KVModuleCtx *ctx, KVModuleString **argv, int argc) {
if (KVModule_Init(ctx, "mymod", 1, KVMODULE_APIVER_1) == KVMODULE_ERR)
return KVMODULE_ERR;
// register commands...
return KVMODULE_OK;
}
```
## Client SDKs
| Language | Package | Install |
|----------|---------|---------|
| Python | [hanzo-kv](https://pypi.org/project/hanzo-kv) | `pip install hanzo-kv` |
| Go | [hanzo/kv-go](https://github.com/hanzoai/kv-go) | `go get github.com/hanzoai/kv-go` |
| Node.js | [@hanzo/kv](https://github.com/hanzoai/kv-client) | `npm install @hanzo/kv` |
Any Redis-compatible client library will also work.
## Documentation
Full documentation is available at [docs.hanzo.ai](https://docs.hanzo.ai).
## License
BSD-3-Clause
Copyright (c) 2024-2026 Hanzo AI Inc. All rights reserved.
[1]: https://github.com/valkey-io/valkey/blob/unstable/COPYING
[2]: https://github.com/valkey-io/valkey/blob/unstable/CONTRIBUTING.md
[3]: https://github.com/valkey-io/valkey/blob/unstable/SECURITY.md
+3 -3
View File
@@ -1,6 +1,6 @@
## Reporting a Vulnerability
If you believe you've discovered a security vulnerability, please contact the KV team at security@hanzo.ai.
If you believe you've discovered a security vulnerability, please contact the Valkey team at security@lists.valkey.io.
Please *DO NOT* create an issue.
We follow a responsible disclosure procedure, so depending on the severity of the issue we may notify KV vendors about the issue before releasing it publicly.
If you would like to be added to our list of vendors, please reach out to the KV team at maintainers@hanzo.ai.
We follow a responsible disclosure procedure, so depending on the severity of the issue we may notify Valkey vendors about the issue before releasing it publicly.
If you would like to be added to our list of vendors, please reach out to the Valkey team at maintainers@lists.valkey.io.
+7 -7
View File
@@ -1,23 +1,23 @@
set(CPACK_PACKAGE_NAME "kv")
set(CPACK_PACKAGE_NAME "valkey")
kv_parse_version(CPACK_PACKAGE_VERSION_MAJOR CPACK_PACKAGE_VERSION_MINOR CPACK_PACKAGE_VERSION_PATCH)
valkey_parse_version(CPACK_PACKAGE_VERSION_MAJOR CPACK_PACKAGE_VERSION_MINOR CPACK_PACKAGE_VERSION_PATCH)
set(CPACK_PACKAGE_CONTACT "maintainers@lists.kv.io")
set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "KV is an open source (BSD) high-performance key/value datastore")
set(CPACK_PACKAGE_CONTACT "maintainers@lists.valkey.io")
set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "Valkey is an open source (BSD) high-performance key/value datastore")
set(CPACK_RESOURCE_FILE_LICENSE "${CMAKE_SOURCE_DIR}/COPYING")
set(CPACK_RESOURCE_FILE_README "${CMAKE_SOURCE_DIR}/README.md")
set(CPACK_STRIP_FILES TRUE)
kv_get_distro_name(DISTRO_NAME)
valkey_get_distro_name(DISTRO_NAME)
message(STATUS "Current host distro: ${DISTRO_NAME}")
if (DISTRO_NAME MATCHES ubuntu
OR DISTRO_NAME MATCHES debian
OR DISTRO_NAME MATCHES mint)
message(STATUS "Adding target package for ${DISTRO_NAME}")
set(CPACK_PACKAGING_INSTALL_PREFIX "/opt/kv")
set(CPACK_PACKAGING_INSTALL_PREFIX "/opt/valkey")
# Debian related parameters
set(CPACK_DEBIAN_PACKAGE_MAINTAINER "KV contributors")
set(CPACK_DEBIAN_PACKAGE_MAINTAINER "Valkey contributors")
set(CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON)
set(CPACK_DEBIAN_FILE_NAME DEB-DEFAULT)
set(CPACK_GENERATOR "DEB")
+21 -15
View File
@@ -2,8 +2,8 @@
# Define the sources to be built
# -------------------------------------------------
# kv-server source files
set(KV_SERVER_SRCS
# valkey-server source files
set(VALKEY_SERVER_SRCS
${CMAKE_SOURCE_DIR}/src/threads_mngr.c
${CMAKE_SOURCE_DIR}/src/adlist.c
${CMAKE_SOURCE_DIR}/src/vector.c
@@ -48,6 +48,7 @@ set(KV_SERVER_SRCS
${CMAKE_SOURCE_DIR}/src/cluster_legacy.c
${CMAKE_SOURCE_DIR}/src/cluster_slot_stats.c
${CMAKE_SOURCE_DIR}/src/crc16.c
${CMAKE_SOURCE_DIR}/src/crc16_slottable.c
${CMAKE_SOURCE_DIR}/src/commandlog.c
${CMAKE_SOURCE_DIR}/src/eval.c
${CMAKE_SOURCE_DIR}/src/bio.c
@@ -66,8 +67,9 @@ set(KV_SERVER_SRCS
${CMAKE_SOURCE_DIR}/src/hyperloglog.c
${CMAKE_SOURCE_DIR}/src/latency.c
${CMAKE_SOURCE_DIR}/src/sparkline.c
${CMAKE_SOURCE_DIR}/src/kv-check-rdb.c
${CMAKE_SOURCE_DIR}/src/kv-check-aof.c
${CMAKE_SOURCE_DIR}/src/valkey-check-rdb.c
${CMAKE_SOURCE_DIR}/src/valkey-check-aof.c
${CMAKE_SOURCE_DIR}/src/valkey_strtod.c
${CMAKE_SOURCE_DIR}/src/geo.c
${CMAKE_SOURCE_DIR}/src/lazyfree.c
${CMAKE_SOURCE_DIR}/src/module.c
@@ -119,18 +121,20 @@ set(KV_SERVER_SRCS
${CMAKE_SOURCE_DIR}/src/entry.c
${CMAKE_SOURCE_DIR}/src/vset.c
${CMAKE_SOURCE_DIR}/src/fifo.c
${CMAKE_SOURCE_DIR}/src/mutexqueue.c)
${CMAKE_SOURCE_DIR}/src/mutexqueue.c
${CMAKE_SOURCE_DIR}/src/queues.c)
# kv-cli
set(KV_CLI_SRCS
# valkey-cli
set(VALKEY_CLI_SRCS
${CMAKE_SOURCE_DIR}/src/anet.c
${CMAKE_SOURCE_DIR}/src/adlist.c
${CMAKE_SOURCE_DIR}/src/dict.c
${CMAKE_SOURCE_DIR}/src/sds.c
${CMAKE_SOURCE_DIR}/src/sha256.c
${CMAKE_SOURCE_DIR}/src/util.c
${CMAKE_SOURCE_DIR}/src/kv-cli.c
${CMAKE_SOURCE_DIR}/src/valkey-cli.c
${CMAKE_SOURCE_DIR}/src/valkey_strtod.c
${CMAKE_SOURCE_DIR}/src/zmalloc.c
${CMAKE_SOURCE_DIR}/src/release.c
${CMAKE_SOURCE_DIR}/src/ae.c
@@ -146,14 +150,15 @@ set(KV_CLI_SRCS
${CMAKE_SOURCE_DIR}/src/strl.c
${CMAKE_SOURCE_DIR}/src/cli_commands.c)
# kv-benchmark
set(KV_BENCHMARK_SRCS
# valkey-benchmark
set(VALKEY_BENCHMARK_SRCS
${CMAKE_SOURCE_DIR}/src/ae.c
${CMAKE_SOURCE_DIR}/src/anet.c
${CMAKE_SOURCE_DIR}/src/sds.c
${CMAKE_SOURCE_DIR}/src/sha256.c
${CMAKE_SOURCE_DIR}/src/util.c
${CMAKE_SOURCE_DIR}/src/kv-benchmark.c
${CMAKE_SOURCE_DIR}/src/valkey-benchmark.c
${CMAKE_SOURCE_DIR}/src/valkey_strtod.c
${CMAKE_SOURCE_DIR}/src/adlist.c
${CMAKE_SOURCE_DIR}/src/dict.c
${CMAKE_SOURCE_DIR}/src/zmalloc.c
@@ -164,6 +169,7 @@ set(KV_BENCHMARK_SRCS
${CMAKE_SOURCE_DIR}/src/crc64.c
${CMAKE_SOURCE_DIR}/src/siphash.c
${CMAKE_SOURCE_DIR}/src/crc16.c
${CMAKE_SOURCE_DIR}/src/crc16_slottable.c
${CMAKE_SOURCE_DIR}/src/monotonic.c
${CMAKE_SOURCE_DIR}/src/cli_common.c
${CMAKE_SOURCE_DIR}/src/mt19937-64.c
@@ -171,8 +177,8 @@ set(KV_BENCHMARK_SRCS
${CMAKE_SOURCE_DIR}/src/fuzzer_client.c
${CMAKE_SOURCE_DIR}/src/fuzzer_command_generator.c)
# kv-rdma module
set(KV_RDMA_MODULE_SRCS ${CMAKE_SOURCE_DIR}/src/rdma.c)
# valkey-rdma module
set(VALKEY_RDMA_MODULE_SRCS ${CMAKE_SOURCE_DIR}/src/rdma.c)
# kv-tls module
set(KV_TLS_MODULE_SRCS ${CMAKE_SOURCE_DIR}/src/tls.c)
# valkey-tls module
set(VALKEY_TLS_MODULE_SRCS ${CMAKE_SOURCE_DIR}/src/tls.c)
+7 -7
View File
@@ -1,5 +1,5 @@
# Return the current host distro name. For example: ubuntu, debian, amzn etc
function (kv_get_distro_name DISTRO_NAME)
function (valkey_get_distro_name DISTRO_NAME)
if (LINUX AND NOT APPLE)
execute_process(
COMMAND /bin/bash "-c" "cat /etc/os-release |grep ^ID=|cut -d = -f 2"
@@ -26,20 +26,20 @@ function (kv_get_distro_name DISTRO_NAME)
endif ()
endfunction ()
function (kv_parse_version OUT_MAJOR OUT_MINOR OUT_PATCH)
function (valkey_parse_version OUT_MAJOR OUT_MINOR OUT_PATCH)
# Read and parse package version from version.h file
file(STRINGS ${CMAKE_SOURCE_DIR}/src/version.h VERSION_LINES)
foreach (LINE ${VERSION_LINES})
string(FIND "${LINE}" "#define KV_VERSION " VERSION_STR_POS)
string(FIND "${LINE}" "#define VALKEY_VERSION " VERSION_STR_POS)
if (VERSION_STR_POS GREATER -1)
string(REPLACE "#define KV_VERSION " "" LINE "${LINE}")
string(REPLACE "#define VALKEY_VERSION " "" LINE "${LINE}")
string(REPLACE "\"" "" LINE "${LINE}")
# Change "." to ";" to make it a list
string(REPLACE "." ";" LINE "${LINE}")
list(GET LINE 0 _MAJOR)
list(GET LINE 1 _MINOR)
list(GET LINE 2 _PATCH)
message(STATUS "KV version: ${_MAJOR}.${_MINOR}.${_PATCH}")
message(STATUS "Valkey version: ${_MAJOR}.${_MINOR}.${_PATCH}")
# Set the output variables
set(${OUT_MAJOR}
${_MAJOR}
@@ -66,7 +66,7 @@ endfunction ()
# - `yes` | `1` | `on` => return `1`
# - `module` => return `2`
# ~~~
function (kv_parse_build_option OPTION_VALUE OUT_ARG_ENUM)
function (valkey_parse_build_option OPTION_VALUE OUT_ARG_ENUM)
list(APPEND VALID_OPTIONS "yes")
list(APPEND VALID_OPTIONS "1")
list(APPEND VALID_OPTIONS "on")
@@ -101,7 +101,7 @@ function (kv_parse_build_option OPTION_VALUE OUT_ARG_ENUM)
endif ()
endfunction ()
function (kv_pkg_config PKGNAME OUT_VARIABLE)
function (valkey_pkg_config PKGNAME OUT_VARIABLE)
if (NOT FOUND_PKGCONFIG)
# Locate pkg-config once
find_package(PkgConfig REQUIRED)
@@ -9,11 +9,11 @@ set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/lib")
# Generate compile_commands.json file for IDEs code completion support
set(CMAKE_EXPORT_COMPILE_COMMANDS 1)
processorcount(KV_PROCESSOR_COUNT)
message(STATUS "Processor count: ${KV_PROCESSOR_COUNT}")
processorcount(VALKEY_PROCESSOR_COUNT)
message(STATUS "Processor count: ${VALKEY_PROCESSOR_COUNT}")
# Installed executables will have this permissions
set(KV_EXE_PERMISSIONS
set(VALKEY_EXE_PERMISSIONS
OWNER_EXECUTE
OWNER_WRITE
OWNER_READ
@@ -22,22 +22,22 @@ set(KV_EXE_PERMISSIONS
WORLD_EXECUTE
WORLD_READ)
set(KV_SERVER_CFLAGS "")
set(KV_SERVER_LDFLAGS "")
set(VALKEY_SERVER_CFLAGS "")
set(VALKEY_SERVER_LDFLAGS "")
# ----------------------------------------------------
# Helper functions & macros
# ----------------------------------------------------
macro (add_kv_server_compiler_options value)
set(KV_SERVER_CFLAGS "${KV_SERVER_CFLAGS} ${value}")
macro (add_valkey_server_compiler_options value)
set(VALKEY_SERVER_CFLAGS "${VALKEY_SERVER_CFLAGS} ${value}")
endmacro ()
macro (add_kv_server_linker_option value)
list(APPEND KV_SERVER_LDFLAGS ${value})
macro (add_valkey_server_linker_option value)
list(APPEND VALKEY_SERVER_LDFLAGS ${value})
endmacro ()
macro (get_kv_server_linker_option return_value)
list(JOIN KV_SERVER_LDFLAGS " " ${value} ${return_value})
macro (get_valkey_server_linker_option return_value)
list(JOIN VALKEY_SERVER_LDFLAGS " " ${value} ${return_value})
endmacro ()
set(IS_FREEBSD 0)
@@ -45,11 +45,11 @@ if (CMAKE_SYSTEM_NAME MATCHES "^.*BSD$|DragonFly")
message(STATUS "Building for FreeBSD compatible system")
set(IS_FREEBSD 1)
include_directories("/usr/local/include")
add_kv_server_compiler_options("-DUSE_BACKTRACE")
add_valkey_server_compiler_options("-DUSE_BACKTRACE")
endif ()
# Helper function for creating symbolic link so that: link -> source
macro (kv_create_symlink source link)
macro (valkey_create_symlink source link)
add_custom_command(
TARGET ${source} POST_BUILD
COMMAND ${CMAKE_COMMAND} -E create_symlink
@@ -60,18 +60,18 @@ macro (kv_create_symlink source link)
endmacro ()
# Install a binary
macro (kv_install_bin target)
macro (valkey_install_bin target)
# Install cli tool and create a redis symbolic link
install(
TARGETS ${target}
DESTINATION ${CMAKE_INSTALL_BINDIR}
PERMISSIONS ${KV_EXE_PERMISSIONS}
COMPONENT "kv")
PERMISSIONS ${VALKEY_EXE_PERMISSIONS}
COMPONENT "valkey")
endmacro ()
# Helper function that defines, builds and installs `target` In addition, it creates a symbolic link between the target
# and `link_name`
macro (kv_build_and_install_bin target sources ld_flags libs link_name)
macro (valkey_build_and_install_bin target sources ld_flags libs link_name)
add_executable(${target} ${sources})
if (USE_JEMALLOC
@@ -83,15 +83,15 @@ macro (kv_build_and_install_bin target sources ld_flags libs link_name)
# Place this line last to ensure that ${ld_flags} is placed last on the linker line
target_link_libraries(${target} ${libs} ${ld_flags})
target_link_libraries(${target} kv::kv)
target_link_libraries(${target} valkey::valkey)
if (USE_TLS)
# Add required libraries needed for TLS
target_link_libraries(${target} OpenSSL::SSL kv::kv_tls)
target_link_libraries(${target} OpenSSL::SSL valkey::valkey_tls)
endif ()
if (USE_RDMA)
# Add required libraries needed for RDMA
target_link_libraries(${target} kv::kv_rdma)
target_link_libraries(${target} valkey::valkey_rdma)
endif ()
if (IS_FREEBSD)
@@ -102,18 +102,18 @@ macro (kv_build_and_install_bin target sources ld_flags libs link_name)
target_compile_options(${target} PRIVATE -Werror -Wall)
# Install cli tool and create a redis symbolic link
kv_install_bin(${target})
kv_create_symlink(${target} ${link_name})
valkey_install_bin(${target})
valkey_create_symlink(${target} ${link_name})
endmacro ()
# Determine if we are building in Release or Debug mode
if (CMAKE_BUILD_TYPE MATCHES Debug OR CMAKE_BUILD_TYPE MATCHES DebugFull)
set(KV_DEBUG_BUILD 1)
set(KV_RELEASE_BUILD 0)
set(VALKEY_DEBUG_BUILD 1)
set(VALKEY_RELEASE_BUILD 0)
message(STATUS "Building in debug mode")
else ()
set(KV_DEBUG_BUILD 0)
set(KV_RELEASE_BUILD 1)
set(VALKEY_DEBUG_BUILD 0)
set(VALKEY_RELEASE_BUILD 1)
message(STATUS "Building in release mode")
endif ()
@@ -138,21 +138,21 @@ if (BUILD_MALLOC)
if ("${BUILD_MALLOC}" STREQUAL "jemalloc")
set(MALLOC_LIB "jemalloc")
set(ALLOCATOR_LIB "jemalloc")
add_kv_server_compiler_options("-DUSE_JEMALLOC")
add_valkey_server_compiler_options("-DUSE_JEMALLOC")
set(USE_JEMALLOC 1)
elseif ("${BUILD_MALLOC}" STREQUAL "libc")
set(MALLOC_LIB "libc")
elseif ("${BUILD_MALLOC}" STREQUAL "tcmalloc")
set(MALLOC_LIB "tcmalloc")
kv_pkg_config(libtcmalloc ALLOCATOR_LIB)
valkey_pkg_config(libtcmalloc ALLOCATOR_LIB)
add_kv_server_compiler_options("-DUSE_TCMALLOC")
add_valkey_server_compiler_options("-DUSE_TCMALLOC")
set(USE_TCMALLOC 1)
elseif ("${BUILD_MALLOC}" STREQUAL "tcmalloc_minimal")
set(MALLOC_LIB "tcmalloc_minimal")
kv_pkg_config(libtcmalloc_minimal ALLOCATOR_LIB)
valkey_pkg_config(libtcmalloc_minimal ALLOCATOR_LIB)
add_kv_server_compiler_options("-DUSE_TCMALLOC")
add_valkey_server_compiler_options("-DUSE_TCMALLOC")
set(USE_TCMALLOC_MINIMAL 1)
else ()
message(FATAL_ERROR "BUILD_MALLOC can be one of: jemalloc, libc, tcmalloc or tcmalloc_minimal")
@@ -163,7 +163,7 @@ message(STATUS "Using ${MALLOC_LIB}")
# TLS support
if (BUILD_TLS)
kv_parse_build_option(${BUILD_TLS} USE_TLS)
valkey_parse_build_option(${BUILD_TLS} USE_TLS)
if (USE_TLS EQUAL 1)
# Only search for OpenSSL if needed
find_package(OpenSSL REQUIRED)
@@ -173,8 +173,8 @@ if (BUILD_TLS)
endif ()
if (USE_TLS EQUAL 1)
add_kv_server_compiler_options("-DUSE_OPENSSL=1")
add_kv_server_compiler_options("-DBUILD_TLS_MODULE=0")
add_valkey_server_compiler_options("-DUSE_OPENSSL=1")
add_valkey_server_compiler_options("-DBUILD_TLS_MODULE=0")
else ()
# Build TLS as a module RDMA can only be built as a module. So disable it
message(WARNING "BUILD_TLS can be one of: [ON | OFF | 1 | 0], but '${BUILD_TLS}' was provided")
@@ -191,22 +191,22 @@ if (BUILD_RDMA)
set(BUILD_RDMA_MODULE 0)
# RDMA support (Linux only)
if (LINUX AND NOT APPLE)
kv_parse_build_option(${BUILD_RDMA} USE_RDMA)
valkey_parse_build_option(${BUILD_RDMA} USE_RDMA)
find_package(PkgConfig REQUIRED)
# Locate librdmacm & libibverbs, fail if we can't find them
kv_pkg_config(librdmacm RDMACM_LIBS)
kv_pkg_config(libibverbs IBVERBS_LIBS)
valkey_pkg_config(librdmacm RDMACM_LIBS)
valkey_pkg_config(libibverbs IBVERBS_LIBS)
message(STATUS "${RDMACM_LIBS};${IBVERBS_LIBS}")
list(APPEND RDMA_LIBS "${RDMACM_LIBS};${IBVERBS_LIBS}")
if (USE_RDMA EQUAL 2) # Module
message(STATUS "Building RDMA as module")
add_kv_server_compiler_options("-DUSE_RDMA=2")
add_valkey_server_compiler_options("-DUSE_RDMA=2")
set(BUILD_RDMA_MODULE 2)
elseif (USE_RDMA EQUAL 1) # Builtin
message(STATUS "Building RDMA as builtin")
add_kv_server_compiler_options("-DUSE_RDMA=1")
add_kv_server_compiler_options("-DBUILD_RDMA_MODULE=0")
add_valkey_server_compiler_options("-DUSE_RDMA=1")
add_valkey_server_compiler_options("-DBUILD_RDMA_MODULE=0")
list(APPEND SERVER_LIBS "${RDMA_LIBS}")
endif ()
else ()
@@ -231,53 +231,53 @@ endif ()
message(STATUS "Building on ${CMAKE_HOST_SYSTEM_NAME}")
if (BUILDING_ARM64)
message(STATUS "Compiling kv for ARM64")
add_kv_server_linker_option("-funwind-tables")
message(STATUS "Compiling valkey for ARM64")
add_valkey_server_linker_option("-funwind-tables")
endif ()
if (APPLE)
add_kv_server_linker_option("-rdynamic")
add_kv_server_linker_option("-ldl")
add_valkey_server_linker_option("-Wl,-export_dynamic")
add_valkey_server_linker_option("-ldl")
elseif (UNIX)
add_kv_server_linker_option("-rdynamic")
add_kv_server_linker_option("-pthread")
add_kv_server_linker_option("-ldl")
add_kv_server_linker_option("-lm")
add_valkey_server_linker_option("-rdynamic")
add_valkey_server_linker_option("-pthread")
add_valkey_server_linker_option("-ldl")
add_valkey_server_linker_option("-lm")
endif ()
if (KV_DEBUG_BUILD)
if (VALKEY_DEBUG_BUILD)
# Debug build, use enable "-fno-omit-frame-pointer"
add_kv_server_compiler_options("-fno-omit-frame-pointer")
add_valkey_server_compiler_options("-fno-omit-frame-pointer")
endif ()
# Check for Atomic
check_include_files(stdatomic.h HAVE_C11_ATOMIC)
if (HAVE_C11_ATOMIC)
add_kv_server_compiler_options("-std=gnu11")
add_valkey_server_compiler_options("-std=gnu11")
else ()
add_kv_server_compiler_options("-std=c99")
add_valkey_server_compiler_options("-std=c99")
endif ()
# Sanitizer
if (BUILD_SANITIZER)
# Common CFLAGS
list(APPEND KV_SANITAIZER_CFLAGS "-fno-sanitize-recover=all")
list(APPEND KV_SANITAIZER_CFLAGS "-fno-omit-frame-pointer")
list(APPEND VALKEY_SANITAIZER_CFLAGS "-fno-sanitize-recover=all")
list(APPEND VALKEY_SANITAIZER_CFLAGS "-fno-omit-frame-pointer")
if ("${BUILD_SANITIZER}" STREQUAL "address")
list(APPEND KV_SANITAIZER_CFLAGS "-fsanitize=address")
list(APPEND KV_SANITAIZER_LDFLAGS "-fsanitize=address")
list(APPEND VALKEY_SANITAIZER_CFLAGS "-fsanitize=address")
list(APPEND VALKEY_SANITAIZER_LDFLAGS "-fsanitize=address")
elseif ("${BUILD_SANITIZER}" STREQUAL "thread")
list(APPEND KV_SANITAIZER_CFLAGS "-fsanitize=thread")
list(APPEND KV_SANITAIZER_LDFLAGS "-fsanitize=thread")
list(APPEND VALKEY_SANITAIZER_CFLAGS "-fsanitize=thread")
list(APPEND VALKEY_SANITAIZER_LDFLAGS "-fsanitize=thread")
elseif ("${BUILD_SANITIZER}" STREQUAL "undefined")
list(APPEND KV_SANITAIZER_CFLAGS "-fsanitize=undefined")
list(APPEND KV_SANITAIZER_LDFLAGS "-fsanitize=undefined")
list(APPEND VALKEY_SANITAIZER_CFLAGS "-fsanitize=undefined")
list(APPEND VALKEY_SANITAIZER_LDFLAGS "-fsanitize=undefined")
else ()
message(FATAL_ERROR "Unknown sanitizer: ${BUILD_SANITIZER}")
endif ()
endif ()
include_directories("${CMAKE_SOURCE_DIR}/deps/libkv/include")
include_directories("${CMAKE_SOURCE_DIR}/deps/libvalkey/include")
include_directories("${CMAKE_SOURCE_DIR}/src/modules/lua")
include_directories("${CMAKE_SOURCE_DIR}/deps/linenoise")
include_directories("${CMAKE_SOURCE_DIR}/deps/hdr_histogram")
@@ -291,7 +291,7 @@ if (USE_JEMALLOC)
endif ()
# Common compiler flags
add_kv_server_compiler_options("-pedantic")
add_valkey_server_compiler_options("-pedantic")
if (NOT BUILD_LUA)
message(STATUS "Lua scripting engine is disabled")
@@ -330,22 +330,10 @@ if (PYTHON_EXE)
COMMAND touch ${CMAKE_BINARY_DIR}/fmtargs_generated
WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}/src")
add_custom_target(generate_fmtargs_h DEPENDS ${CMAKE_BINARY_DIR}/fmtargs_generated)
# Rule for generating test_files.h
message(STATUS "Adding target generate_test_files_h")
file(GLOB UNIT_TEST_SRCS "${CMAKE_SOURCE_DIR}/src/unit/*.c")
add_custom_command(
OUTPUT ${CMAKE_BINARY_DIR}/test_files_generated
DEPENDS "${UNIT_TEST_SRCS};${CMAKE_SOURCE_DIR}/utils/generate-unit-test-header.py"
COMMAND ${PYTHON_EXE} ${CMAKE_SOURCE_DIR}/utils/generate-unit-test-header.py
COMMAND touch ${CMAKE_BINARY_DIR}/test_files_generated
WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}/src")
add_custom_target(generate_test_files_h DEPENDS ${CMAKE_BINARY_DIR}/test_files_generated)
else ()
# Fake targets
add_custom_target(generate_commands_def)
add_custom_target(generate_fmtargs_h)
add_custom_target(generate_test_files_h)
endif ()
# Generate release.h file (always)
@@ -365,8 +353,8 @@ include(SourceFiles)
# Clear the below variables from the cache
unset(CMAKE_C_FLAGS CACHE)
unset(KV_SERVER_LDFLAGS CACHE)
unset(KV_SERVER_CFLAGS CACHE)
unset(VALKEY_SERVER_LDFLAGS CACHE)
unset(VALKEY_SERVER_CFLAGS CACHE)
unset(PYTHON_EXE CACHE)
unset(HAVE_C11_ATOMIC CACHE)
unset(USE_TLS CACHE)
+10 -6
View File
@@ -3,7 +3,7 @@ if (USE_JEMALLOC)
endif ()
add_subdirectory(lua)
# Set libkv options. We need to disable the defaults set in the OPTION(..) we do this by setting them in the CACHE
# Set libvalkey options. We need to disable the defaults set in the OPTION(..) we do this by setting them in the CACHE
set(BUILD_SHARED_LIBS
OFF
CACHE BOOL "Build shared libraries")
@@ -11,27 +11,31 @@ set(DISABLE_TESTS
ON
CACHE BOOL "If tests should be compiled or not")
if (USE_TLS) # Module or no module
message(STATUS "Building kv_tls")
message(STATUS "Building valkey_tls")
set(ENABLE_TLS
ON
CACHE BOOL "If TLS support should be compiled or not")
endif ()
if (USE_RDMA) # Module or no module
message(STATUS "Building kv_rdma")
message(STATUS "Building valkey_rdma")
set(ENABLE_RDMA
ON
CACHE BOOL "If RDMA support should be compiled or not")
if (USE_RDMA EQUAL 2) # Module
set(ENABLE_DLOPEN_RDMA ON CACHE BOOL "Build valkey_rdma with dynamic loading")
endif()
endif ()
# Let libkv use sds and dict provided by kv.
# Let libvalkey use sds and dict provided by valkey.
set(DICT_INCLUDE_DIR ${CMAKE_SOURCE_DIR}/src)
set(SDS_INCLUDE_DIR ${CMAKE_SOURCE_DIR}/src)
add_subdirectory(libkv)
add_subdirectory(libvalkey)
add_subdirectory(linenoise)
add_subdirectory(fpconv)
add_subdirectory(hdr_histogram)
add_subdirectory(fast_float)
# Clear any cached variables passed to libkv from the cache
# Clear any cached variables passed to libvalkey from the cache
unset(BUILD_SHARED_LIBS CACHE)
unset(DISABLE_TESTS CACHE)
unset(ENABLE_TLS CACHE)
+16 -12
View File
@@ -36,31 +36,33 @@ ifneq ($(shell sh -c '[ -f .make-ldflags ] && cat .make-ldflags || echo none'),
endif
distclean:
-(cd libkv && $(MAKE) clean) > /dev/null || true
-(cd libvalkey && $(MAKE) clean) > /dev/null || true
-(cd linenoise && $(MAKE) clean) > /dev/null || true
-(cd lua && $(MAKE) clean) > /dev/null || true
-(cd jemalloc && [ -f Makefile ] && $(MAKE) distclean) > /dev/null || true
-(cd hdr_histogram && $(MAKE) clean) > /dev/null || true
-(cd fpconv && $(MAKE) clean) > /dev/null || true
-(cd fast_float_c_interface && $(MAKE) clean) > /dev/null || true
-(rm -f .make-*)
.PHONY: distclean
LIBKV_MAKE_FLAGS = SDS_INCLUDE_DIR=../../src/ DICT_INCLUDE_DIR=../../src/
LIBVALKEY_MAKE_FLAGS = SDS_INCLUDE_DIR=../../src/ DICT_INCLUDE_DIR=../../src/
ifneq (,$(filter $(BUILD_TLS),yes module))
LIBKV_MAKE_FLAGS += USE_TLS=1
LIBVALKEY_MAKE_FLAGS += USE_TLS=1
endif
ifneq (,$(filter $(BUILD_RDMA),yes module))
LIBKV_MAKE_FLAGS += USE_RDMA=1
LIBVALKEY_MAKE_FLAGS += USE_RDMA=1
ifneq (,$(filter $(BUILD_RDMA),module))
LIBVALKEY_MAKE_FLAGS += USE_DLOPEN_RDMA=1
endif
endif
libkv: .make-prerequisites
libvalkey: .make-prerequisites
@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
cd libkv && $(MAKE) static $(LIBKV_MAKE_FLAGS)
cd libvalkey && $(MAKE) static $(LIBVALKEY_MAKE_FLAGS)
.PHONY: libkv
.PHONY: libvalkey
linenoise: .make-prerequisites
@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
@@ -123,8 +125,10 @@ jemalloc: .make-prerequisites
.PHONY: jemalloc
fast_float_c_interface: .make-prerequisites
gtest-parallel: .make-prerequisites
@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
cd fast_float_c_interface && $(MAKE)
.PHONY: fast_float_c_interface
@if [ ! -f gtest-parallel/gtest_parallel.py ]; then \
echo "Downloading gtest-parallel..."; \
rm -rf gtest-parallel; \
git clone --depth 1 https://github.com/google/gtest-parallel.git gtest-parallel; \
fi
+59 -28
View File
@@ -1,12 +1,13 @@
This directory contains all KV dependencies, except for the libc that
This directory contains all Valkey dependencies, except for the libc that
should be provided by the operating system.
* **Jemalloc** is our memory allocator, used as replacement for libc malloc on Linux by default. It has good performances and excellent fragmentation behavior. This component is upgraded from time to time.
* **libkv** is the official C client library for KV. It is used by kv-cli, kv-benchmark and KV Sentinel. It is managed in a separate project and updated as needed.
* **linenoise** is a readline replacement. It is developed by the same authors of KV but is managed as a separated project and updated as needed.
* **libvalkey** is the official C client library for Valkey. It is used by valkey-cli, valkey-benchmark and Valkey Sentinel. It is managed in a separate project and updated as needed.
* **linenoise** is a readline replacement. It is developed by the same authors of Valkey but is managed as a separated project and updated as needed.
* **lua** is Lua 5.1 with minor changes for security and additional libraries.
* **hdr_histogram** Used for per-command latency tracking histograms.
* **fast_float** is a replacement for strtod to convert strings to floats efficiently.
* **ffc.h** is a C99 port of the fast_float library, used as a replacement for strtod to convert strings to floats efficiently.
* **gtest-parallel** is a script for running googletest tests in parallel.
How to upgrade the above dependencies
===
@@ -14,10 +15,10 @@ How to upgrade the above dependencies
Jemalloc
---
Jemalloc is modified with changes that allow us to implement the KV
active defragmentation logic. However this feature of KV is not mandatory
and KV is able to understand if the Jemalloc version it is compiled
against supports such KV-specific modifications. So in theory, if you
Jemalloc is modified with changes that allow us to implement the Valkey
active defragmentation logic. However this feature of Valkey is not mandatory
and Valkey is able to understand if the Jemalloc version it is compiled
against supports such Valkey-specific modifications. So in theory, if you
are not interested in the active defragmentation, you can replace Jemalloc
just following these steps:
@@ -29,7 +30,7 @@ just following these steps:
Jemalloc configuration script is broken and will not work nested in another
git repository.
However note that we change Jemalloc settings via the `configure` script of Jemalloc using the `--with-lg-quantum` option, setting it to the value of 3 instead of 4. This provides us with more size classes that better suit the KV data structures, in order to gain memory efficiency.
However note that we change Jemalloc settings via the `configure` script of Jemalloc using the `--with-lg-quantum` option, setting it to the value of 3 instead of 4. This provides us with more size classes that better suit the Valkey data structures, in order to gain memory efficiency.
If you want to upgrade Jemalloc while also providing support for
active defragmentation, in addition to the above steps you need to perform
@@ -39,7 +40,7 @@ the following additional steps:
to add `#define JEMALLOC_FRAG_HINT`.
6. Implement the function `je_get_defrag_hint()` inside `src/jemalloc.c`. You
can see how it is implemented in the current Jemalloc source tree shipped
with KV, and rewrite it according to the new Jemalloc internals, if they
with Valkey, and rewrite it according to the new Jemalloc internals, if they
changed, otherwise you could just copy the old implementation if you are
upgrading just to a similar version of Jemalloc.
@@ -59,13 +60,13 @@ cd deps/jemalloc
4. Update jemalloc's version in `deps/Makefile`: search for "`--with-version=<old-version-tag>-0-g0`" and update it accordingly.
5. Commit the changes (VERSION,configure,Makefile).
Libkv
Libvalkey
---
Libkv is used by Sentinel, `kv-cli` and `kv-benchmark`.
The library is built without its own version of the sds and dict type and uses the KV provided variant instead.
Libvalkey is used by Sentinel, `valkey-cli` and `valkey-benchmark`.
The library is built without its own version of the sds and dict type and uses the Valkey provided variant instead.
1. `git subtree pull --prefix deps/libkv https://github.com/hanzoai/kv.git <version-tag> --squash`<br>
1. `git subtree pull --prefix deps/libvalkey https://github.com/valkey-io/libvalkey.git <version-tag> --squash`<br>
This should hopefully merge the local changes into the new version.
2. Commit the changes.
@@ -73,7 +74,7 @@ Linenoise
---
Linenoise is rarely upgraded as needed. The upgrade process is trivial since
KV uses a non modified version of linenoise, so to upgrade just do the
Valkey uses a non modified version of linenoise, so to upgrade just do the
following:
1. Remove the linenoise directory.
@@ -83,11 +84,11 @@ Lua
---
We use Lua 5.1 and no upgrade is planned currently, since we don't want to break
Lua scripts for new Lua features: in the context of KV Lua scripts the
Lua scripts for new Lua features: in the context of Valkey Lua scripts the
capabilities of 5.1 are usually more than enough, the release is rock solid,
and we definitely don't want to break old scripts.
So upgrading of Lua is up to the KV project maintainers and should be a
So upgrading of Lua is up to the Valkey project maintainers and should be a
manual procedure performed by taking a diff between the different versions.
Currently we have at least the following differences between official Lua 5.1
@@ -107,17 +108,47 @@ We use a customized version based on master branch commit e4448cf6d1cd08fff51981
2. Copy updated files from newer version onto files in /hdr_histogram.
3. Apply the changes from 1 above to the updated files.
fast_float
ffc.h
---
The fast_float library provides fast header-only implementations for the C++ from_chars functions for `float` and `double` types as well as integer types. These functions convert ASCII strings representing decimal values (e.g., `1.3e10`) into binary types. The functions are much faster than comparable number-parsing functions from existing C++ standard libraries.
Specifically, `fast_float` provides the following function to parse floating-point numbers with a C++17-like syntax (the library itself only requires C++11):
template <typename T, typename UC = char, typename = FASTFLOAT_ENABLE_IF(is_supported_float_type<T>())>
from_chars_result_t<UC> from_chars(UC const *first, UC const *last, T &value, chars_format fmt = chars_format::general);
ffc.h is a pure C99 port of the fast_float library, providing fast string-to-double
conversion without requiring a C++ compiler.
To upgrade the library,
1. Check out https://github.com/fastfloat/fast_float/tree/main
2. cd fast_float
3. Invoke "python3 ./script/amalgamate.py --output fast_float.h"
4. Copy fast_float.h file to "deps/fast_float/".
1. Download the latest ffc.h from https://github.com/kolemannix/ffc.h/releases
2. Copy ffc.h to "deps/fast_float/".
gtest-parallel
---
The `deps/gtest-parallel` directory is imported from the upstream
https://github.com/google/gtest-parallel repository as a subtree snapshot (not a real Git subtree).
Current upstream version:
- Upstream commit: `cd488bd` (from google/gtest-parallel)
Updating gtest-parallel
Run the following from the repository root.
1. Add the remote and fetch upstream:
```sh
git remote add gtest-parallel https://github.com/google/gtest-parallel.git
git fetch gtest-parallel master
```
2. Remove any previous import and commit (commit A):
```sh
rm -rf deps/gtest-parallel
```
3. Update the subtree from upstream:
```sh
git subtree add --prefix=deps/gtest-parallel gtest-parallel master --squash
```
4. Reset back to commit A with proper sign-off:
```sh
git reset --soft <commit-A-hash>
```
5. Commit the changes.
+2
View File
@@ -0,0 +1,2 @@
add_library(ffc INTERFACE)
target_include_directories(ffc INTERFACE ${CMAKE_CURRENT_SOURCE_DIR})
-3912
View File
File diff suppressed because it is too large Load Diff
+3235
View File
File diff suppressed because one or more lines are too long
-37
View File
@@ -1,37 +0,0 @@
CCCOLOR:="\033[34m"
SRCCOLOR:="\033[33m"
ENDCOLOR:="\033[0m"
CXX?=c++
# we need = instead of := so that $@ in QUIET_CXX gets evaluated in the rule and is assigned appropriate value.
TEMP:=$(CXX)
QUIET_CXX=@printf ' %b %b\n' $(CCCOLOR)C++$(ENDCOLOR) $(SRCCOLOR)$@$(ENDCOLOR) 1>&2;
CXX=$(QUIET_CXX)$(TEMP)
WARN=-Wall -W -Wno-missing-field-initializers
STD=-pedantic -std=c++11
OPT?=-O3
CLANG := $(findstring clang,$(shell sh -c '$(CC) --version | head -1'))
ifeq ($(OPT),-O3)
ifeq (clang,$(CLANG))
OPT+=-flto
else
OPT+=-flto=auto -ffat-lto-objects
endif
endif
# 1) Today src/Makefile passes -m32 flag for explicit 32-bit build on 64-bit machine, via CFLAGS. For 32-bit build on
# 32-bit machine and 64-bit on 64-bit machine, CFLAGS are empty. No other flags are set that can conflict with C++,
# therefore let's use CFLAGS without changes for now.
# 2) FASTFLOAT_ALLOWS_LEADING_PLUS allows +inf to be parsed as inf, instead of error.
CXXFLAGS=$(STD) $(OPT) $(WARN) -static -fPIC -fno-exceptions $(CFLAGS) -D FASTFLOAT_ALLOWS_LEADING_PLUS
.PHONY: all clean
all: fast_float_strtod.o
clean:
rm -f *.o || true;
-24
View File
@@ -1,24 +0,0 @@
/*
* Copyright (c) Valkey Contributors
* All rights reserved.
* SPDX-License-Identifier: BSD-3-Clause
*/
#include "../fast_float/fast_float.h"
#include <cerrno>
extern "C"
{
double fast_float_strtod(const char *str, const char** endptr)
{
double temp = 0;
auto answer = fast_float::from_chars(str, str + strlen(str), temp);
if (answer.ec != std::errc()) {
errno = (answer.ec == std::errc::result_out_of_range) ? ERANGE : EINVAL;
}
if (endptr) {
*endptr = answer.ptr;
}
return temp;
}
}
+2
View File
@@ -0,0 +1,2 @@
.*.swp
*.py[co]
+4
View File
@@ -0,0 +1,4 @@
[style]
based_on_style = pep8
indent_width = 2
column_limit = 80
+28
View File
@@ -0,0 +1,28 @@
# How to Contribute
We'd love to accept your patches and contributions to this project. There are
just a few small guidelines you need to follow.
## Contributor License Agreement
Contributions to this project must be accompanied by a Contributor License
Agreement. You (or your employer) retain the copyright to your contribution;
this simply gives us permission to use and redistribute your contributions as
part of the project. Head over to <https://cla.developers.google.com/> to see
your current agreements on file or to sign a new one.
You generally only need to submit a CLA once, so if you've already submitted one
(even if it was for a different project), you probably don't need to do it
again.
## Code reviews
All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose. Consult
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
information on using pull requests.
## Community Guidelines
This project follows [Google's Open Source Community
Guidelines](https://opensource.google.com/conduct/).
+202
View File
@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+90
View File
@@ -0,0 +1,90 @@
# gtest-parallel
_This is not an official Google product._
`gtest-parallel` is a script that executes [Google
Test](https://github.com/google/googletest) binaries in parallel, providing good
speedup for single-threaded tests (on multi-core machines) and tests that do not
run at 100% CPU (on single- or multi-core machines).
The script works by listing the tests of each binary, and then executing them on
workers in separate processes. This works fine so long as the tests are self
contained and do not share resources (reading data is fine, writing to the same
log file is probably not).
## Basic Usage
_For a full list of options, see `--help`._
$ ./gtest-parallel path/to/binary...
This shards all enabled tests across a number of workers, defaulting to the
number of cores in the system. If your system uses Python 2, but you have no
python2 binary, run `python gtest-parallel` instead of `./gtest-parallel`.
To run only a select set of tests, run:
$ ./gtest-parallel path/to/binary... --gtest_filter=Foo.*:Bar.*
This filter takes the same parameters as Google Test, so -Foo.\* can be used for
test exclusion as well. This is especially useful for slow tests (that you're
not working on), or tests that may not be able to run in parallel.
## Flakiness
Flaky tests (tests that do not deterministically pass or fail) often cause a lot
of developer headache. A test that fails only 1% of the time can be very hard to
detect as flaky, and even harder to convince yourself of having fixed.
`gtest-parallel` supports repeating individual tests (`--repeat=`), which can be
very useful for flakiness testing. Some tests are also more flaky under high
loads (especially tests that use realtime clocks), so raising the number of
`--workers=` well above the number of available core can often cause contention
and be fruitful for detecting flaky tests as well.
$ ./gtest-parallel out/{binary1,binary2,binary3} --repeat=1000 --workers=128
The above command repeats all tests inside `binary1`, `binary2` and `binary3`
located in `out/`. The tests are run `1000` times each on `128` workers (this is
more than I have cores on my machine anyways). This can often be done and then
left overnight if you've no initial guess to which tests are flaky and which
ones aren't. When you've figured out which tests are flaky (and want to fix
them), repeat the above command with `--gtest_filter=` to only retry the flaky
tests that you are fixing.
Note that repeated tests do run concurrently with themselves for efficiency, and
as such they have problem writing to hard-coded files, even if they are only
used by that single test. `tmpfile()` and similar library functions are often
your friends here.
### Flakiness Summaries
Especially for disabled tests, you might wonder how stable a test seems before
trying to enable it. `gtest-parallel` prints summaries (number of passed/failed
tests) when `--repeat=` is used and at least one test fails. This can be used to
generate passed/failed statistics per test. If no statistics are generated then
all invocations tests are passing, congratulations!
For example, to try all disabled tests and see how stable they are:
$ ./gtest-parallel path/to/binary... -r1000 --gtest_filter=*.DISABLED_* --gtest_also_run_disabled_tests
Which will generate something like this at the end of the run:
SUMMARY:
path/to/binary... Foo.DISABLED_Bar passed 0 / 1000 times.
path/to/binary... FooBar.DISABLED_Baz passed 30 / 1000 times.
path/to/binary... Foo.DISABLED_Baz passed 1000 / 1000 times.
## Running Tests Within Test Cases Sequentially
Sometimes tests within a single test case use globally-shared resources
(hard-coded file paths, sockets, etc.) and cannot be run in parallel. Running
such tests in parallel will either fail or be flaky (if they happen to not
overlap during execution, they pass). So long as these resources are only shared
within the same test case `gtest-parallel` can still provide some parallelism.
For such binaries where test cases are independent, `gtest-parallel` provides
`--serialize_test_cases` that runs tests within the same test case sequentially.
While generally not providing as much speedup as fully parallel test execution,
this permits such binaries to partially benefit from parallel execution.
+18
View File
@@ -0,0 +1,18 @@
#!/usr/bin/env python3
# Copyright 2017 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import gtest_parallel
import sys
sys.exit(gtest_parallel.main())
+955
View File
@@ -0,0 +1,955 @@
# Copyright 2013 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import errno
from functools import total_ordering
import gzip
import io
import json
import multiprocessing
import optparse
import os
import re
import shutil
import signal
import subprocess
import sys
import tempfile
import threading
import time
if sys.version_info.major >= 3:
long = int
import _pickle as cPickle
import _thread as thread
else:
import cPickle
import thread
from pickle import HIGHEST_PROTOCOL as PICKLE_HIGHEST_PROTOCOL
if sys.platform == 'win32':
import msvcrt
else:
import fcntl
# An object that catches SIGINT sent to the Python process and notices
# if processes passed to wait() die by SIGINT (we need to look for
# both of those cases, because pressing Ctrl+C can result in either
# the main process or one of the subprocesses getting the signal).
#
# Before a SIGINT is seen, wait(p) will simply call p.wait() and
# return the result. Once a SIGINT has been seen (in the main process
# or a subprocess, including the one the current call is waiting for),
# wait(p) will call p.terminate() and raise ProcessWasInterrupted.
class SigintHandler(object):
class ProcessWasInterrupted(Exception):
pass
sigint_returncodes = {
-signal.SIGINT, # Unix
-1073741510, # Windows
}
def __init__(self):
self.__lock = threading.Lock()
self.__processes = set()
self.__got_sigint = False
signal.signal(signal.SIGINT, lambda signal_num, frame: self.interrupt())
def __on_sigint(self):
self.__got_sigint = True
while self.__processes:
try:
self.__processes.pop().terminate()
except OSError:
pass
def interrupt(self):
with self.__lock:
self.__on_sigint()
def got_sigint(self):
with self.__lock:
return self.__got_sigint
def wait(self, p, timeout_per_test):
with self.__lock:
if self.__got_sigint:
p.terminate()
self.__processes.add(p)
try:
code = p.wait(timeout_per_test)
except subprocess.TimeoutExpired :
p.terminate()
self.__processes.remove(p)
code = -errno.ETIME
with self.__lock:
self.__processes.discard(p)
if code in self.sigint_returncodes:
self.__on_sigint()
if self.__got_sigint:
raise self.ProcessWasInterrupted
return code
sigint_handler = SigintHandler()
# Return the width of the terminal, or None if it couldn't be
# determined (e.g. because we're not being run interactively).
def term_width(out):
if not out.isatty():
return None
try:
p = subprocess.Popen(["stty", "size"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
(out, err) = p.communicate()
if p.returncode != 0 or err:
return None
return int(out.split()[1])
except (IndexError, OSError, ValueError):
return None
# Output transient and permanent lines of text. If several transient
# lines are written in sequence, the new will overwrite the old. We
# use this to ensure that lots of unimportant info (tests passing)
# won't drown out important info (tests failing).
class Outputter(object):
def __init__(self, out_file):
self.__out_file = out_file
self.__previous_line_was_transient = False
self.__width = term_width(out_file) # Line width, or None if not a tty.
def transient_line(self, msg):
if self.__width is None:
self.__out_file.write(msg + "\n")
self.__out_file.flush()
else:
self.__out_file.write("\r" + msg[:self.__width].ljust(self.__width))
self.__previous_line_was_transient = True
def flush_transient_output(self):
if self.__previous_line_was_transient:
self.__out_file.write("\n")
self.__previous_line_was_transient = False
def permanent_line(self, msg):
self.flush_transient_output()
self.__out_file.write(msg + "\n")
if self.__width is None:
self.__out_file.flush()
def get_save_file_path():
"""Return path to file for saving transient data."""
if sys.platform == 'win32':
default_cache_path = os.path.join(os.path.expanduser('~'), 'AppData',
'Local')
cache_path = os.environ.get('LOCALAPPDATA', default_cache_path)
else:
# We don't use xdg module since it's not a standard.
default_cache_path = os.path.join(os.path.expanduser('~'), '.cache')
cache_path = os.environ.get('XDG_CACHE_HOME', default_cache_path)
if os.path.isdir(cache_path):
return os.path.join(cache_path, 'gtest-parallel')
else:
sys.stderr.write('Directory {} does not exist'.format(cache_path))
return os.path.join(os.path.expanduser('~'), '.gtest-parallel-times')
@total_ordering
class Task(object):
"""Stores information about a task (single execution of a test).
This class stores information about the test to be executed (gtest binary and
test name), and its result (log file, exit code and runtime).
Each task is uniquely identified by the gtest binary, the test name and an
execution number that increases each time the test is executed.
Additionaly we store the last execution time, so that next time the test is
executed, the slowest tests are run first.
"""
def __init__(self, test_binary, test_name, test_command, execution_number,
last_execution_time, output_dir):
self.test_name = test_name
self.output_dir = output_dir
self.test_binary = test_binary
self.test_command = test_command
self.execution_number = execution_number
self.last_execution_time = last_execution_time
self.exit_code = None
self.runtime_ms = None
self.test_id = (test_binary, test_name)
self.task_id = (test_binary, test_name, self.execution_number)
self.log_file = Task._logname(self.output_dir, self.test_binary, test_name,
self.execution_number)
def __sorting_key(self):
# Unseen or failing tests (both missing execution time) take precedence over
# execution time. Tests are greater (seen as slower) when missing times so
# that they are executed first.
return (1 if self.last_execution_time is None else 0,
self.last_execution_time)
def __eq__(self, other):
return self.__sorting_key() == other.__sorting_key()
def __ne__(self, other):
return not (self == other)
def __lt__(self, other):
return self.__sorting_key() < other.__sorting_key()
@staticmethod
def _normalize(string):
return re.sub('[^A-Za-z0-9]', '_', string)
@staticmethod
def _logname(output_dir, test_binary, test_name, execution_number):
# Store logs to temporary files if there is no output_dir.
if output_dir is None:
(log_handle, log_name) = tempfile.mkstemp(prefix='gtest_parallel_',
suffix=".log")
os.close(log_handle)
return log_name
log_name = '%s-%s-%d.log' % (Task._normalize(os.path.basename(test_binary)),
Task._normalize(test_name), execution_number)
return os.path.join(output_dir, log_name)
def run(self, timeout_per_test):
begin = time.time()
with open(self.log_file, 'w') as log:
task = subprocess.Popen(self.test_command, stdout=log, stderr=log)
try:
self.exit_code = sigint_handler.wait(task, timeout_per_test)
except sigint_handler.ProcessWasInterrupted:
thread.exit()
self.runtime_ms = int(1000 * (time.time() - begin))
self.last_execution_time = None if self.exit_code else self.runtime_ms
class TaskManager(object):
"""Executes the tasks and stores the passed, failed and interrupted tasks.
When a task is run, this class keeps track if it passed, failed or was
interrupted. After a task finishes it calls the relevant functions of the
Logger, TestResults and TestTimes classes, and in case of failure, retries the
test as specified by the --retry_failed flag.
"""
def __init__(self, times, logger, test_results, task_factory, times_to_retry,
initial_execution_number):
self.times = times
self.logger = logger
self.test_results = test_results
self.task_factory = task_factory
self.times_to_retry = times_to_retry
self.initial_execution_number = initial_execution_number
self.global_exit_code = 0
self.passed = []
self.failed = []
self.started = {}
self.timed_out = []
self.execution_number = {}
self.lock = threading.Lock()
def __get_next_execution_number(self, test_id):
with self.lock:
next_execution_number = self.execution_number.setdefault(
test_id, self.initial_execution_number)
self.execution_number[test_id] += 1
return next_execution_number
def __register_start(self, task):
with self.lock:
self.started[task.task_id] = task
def register_exit(self, task):
self.logger.log_exit(task)
self.times.record_test_time(task.test_binary, task.test_name,
task.last_execution_time)
if self.test_results:
self.test_results.log(task.test_name, task.runtime_ms / 1000.0,
task.exit_code)
with self.lock:
self.started.pop(task.task_id)
if task.exit_code == 0:
self.passed.append(task)
elif task.exit_code == -errno.ETIME:
self.timed_out.append(task)
else:
self.failed.append(task)
def run_task(self, task, timeout_per_test):
for try_number in range(self.times_to_retry + 1):
self.__register_start(task)
task.run(timeout_per_test)
self.register_exit(task)
if task.exit_code == 0:
break
if try_number < self.times_to_retry:
execution_number = self.__get_next_execution_number(task.test_id)
# We need create a new Task instance. Each task represents a single test
# execution, with its own runtime, exit code and log file.
task = self.task_factory(task.test_binary, task.test_name,
task.test_command, execution_number,
task.last_execution_time, task.output_dir)
with self.lock:
if task.exit_code != 0:
self.global_exit_code = task.exit_code
class FilterFormat(object):
def __init__(self, output_dir):
if sys.stdout.isatty():
# stdout needs to be unbuffered since the output is interactive.
if isinstance(sys.stdout, io.TextIOWrapper):
# workaround for https://bugs.python.org/issue17404
sys.stdout = io.TextIOWrapper(sys.stdout.detach(),
line_buffering=True,
write_through=True,
newline='\n')
else:
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
self.output_dir = output_dir
self.total_tasks = 0
self.finished_tasks = 0
self.out = Outputter(sys.stdout)
self.stdout_lock = threading.Lock()
def move_to(self, destination_dir, tasks):
if self.output_dir is None:
return
destination_dir = os.path.join(self.output_dir, destination_dir)
os.makedirs(destination_dir)
for task in tasks:
shutil.move(task.log_file, destination_dir)
def print_tests(self, message, tasks, print_try_number, print_test_command):
self.out.permanent_line("%s (%s/%s):" %
(message, len(tasks), self.total_tasks))
for task in sorted(tasks):
runtime_ms = 'Interrupted'
if task.runtime_ms is not None:
runtime_ms = '%d ms' % task.runtime_ms
if print_test_command:
try:
cmd_str = " ".join(task.test_command)
except TypeError:
cmd_str = task.test_command
self.out.permanent_line(
"%11s: %s%s" %
(runtime_ms, cmd_str,
(" (try #%d)" % task.execution_number) if print_try_number else ""))
else:
self.out.permanent_line(
"%11s: %s %s%s" %
(runtime_ms, task.test_binary, task.test_name,
(" (try #%d)" % task.execution_number) if print_try_number else ""))
def log_exit(self, task):
with self.stdout_lock:
self.finished_tasks += 1
self.out.transient_line("[%d/%d] %s (%d ms)" %
(self.finished_tasks, self.total_tasks,
task.test_name, task.runtime_ms))
if task.exit_code != 0:
signal_name = None
if task.exit_code < 0:
try:
signal_name = signal.Signals(-task.exit_code).name
except ValueError:
pass
with open(task.log_file) as f:
for line in f.readlines():
self.out.permanent_line(line.rstrip())
if task.exit_code is None:
self.out.permanent_line("[%d/%d] %s aborted after %d ms" %
(self.finished_tasks, self.total_tasks,
task.test_name, task.runtime_ms))
elif task.exit_code == -errno.ETIME:
self.out.permanent_line(
"\033[31m[ TIMEOUT ]\033[0m %s timed out after %d s"
% (task.test_name, task.runtime_ms/1000))
elif signal_name is not None:
self.out.permanent_line(
"[%d/%d] %s killed by signal %s (%d ms)" %
(self.finished_tasks, self.total_tasks, task.test_name,
signal_name, task.runtime_ms))
else:
self.out.permanent_line(
"[%d/%d] %s returned with exit code %d (%d ms)" %
(self.finished_tasks, self.total_tasks, task.test_name,
task.exit_code, task.runtime_ms))
if self.output_dir is None:
# Try to remove the file 100 times (sleeping for 0.1 second in between).
# This is a workaround for a process handle seemingly holding on to the
# file for too long inside os.subprocess. This workaround is in place
# until we figure out a minimal repro to report upstream (or a better
# suspect) to prevent os.remove exceptions.
num_tries = 100
for i in range(num_tries):
try:
os.remove(task.log_file)
except OSError as e:
if e.errno is not errno.ENOENT:
if i is num_tries - 1:
self.out.permanent_line('Could not remove temporary log file: ' +
str(e))
else:
time.sleep(0.1)
continue
break
def log_tasks(self, total_tasks):
self.total_tasks += total_tasks
self.out.transient_line("[0/%d] Running tests..." % self.total_tasks)
def summarize(self, passed_tasks, failed_tasks, interrupted_tasks):
stats = {}
def add_stats(stats, task, idx):
task_key = (task.test_binary, task.test_name)
if not task_key in stats:
# (passed, failed, interrupted) task_key is added as tie breaker to get
# alphabetic sorting on equally-stable tests
stats[task_key] = [0, 0, 0, task_key]
stats[task_key][idx] += 1
for task in passed_tasks:
add_stats(stats, task, 0)
for task in failed_tasks:
add_stats(stats, task, 1)
for task in interrupted_tasks:
add_stats(stats, task, 2)
self.out.permanent_line("SUMMARY:")
for task_key in sorted(stats, key=stats.__getitem__):
(num_passed, num_failed, num_interrupted, _) = stats[task_key]
(test_binary, task_name) = task_key
total_runs = num_passed + num_failed + num_interrupted
if num_passed == total_runs:
continue
self.out.permanent_line(" %s %s passed %d / %d times%s." %
(test_binary, task_name, num_passed, total_runs,
"" if num_interrupted == 0 else
(" (%d interrupted)" % num_interrupted)))
def flush(self):
self.out.flush_transient_output()
class CollectTestResults(object):
def __init__(self, json_dump_filepath):
self.test_results_lock = threading.Lock()
self.json_dump_file = open(json_dump_filepath, 'w')
self.test_results = {
"interrupted": False,
"path_delimiter": ".",
# Third version of the file format. See the link in the flag description
# for details.
"version": 3,
"seconds_since_epoch": int(time.time()),
"num_failures_by_type": {
"PASS": 0,
"FAIL": 0,
"TIMEOUT": 0,
},
"tests": {},
}
def log(self, test, runtime_seconds, exit_code):
if exit_code is None:
actual_result = "TIMEOUT"
elif exit_code == 0:
actual_result = "PASS"
else:
actual_result = "FAIL"
with self.test_results_lock:
self.test_results['num_failures_by_type'][actual_result] += 1
results = self.test_results['tests']
for name in test.split('.'):
results = results.setdefault(name, {})
if results:
results['actual'] += ' ' + actual_result
results['times'].append(runtime_seconds)
else: # This is the first invocation of the test
results['actual'] = actual_result
results['times'] = [runtime_seconds]
results['time'] = runtime_seconds
results['expected'] = 'PASS'
def dump_to_file_and_close(self):
json.dump(self.test_results, self.json_dump_file)
self.json_dump_file.close()
# Record of test runtimes. Has built-in locking.
class TestTimes(object):
class LockedFile(object):
def __init__(self, filename, mode):
self._filename = filename
self._mode = mode
self._fo = None
def __enter__(self):
self._fo = open(self._filename, self._mode)
# Regardless of opening mode we always seek to the beginning of file.
# This simplifies code working with LockedFile and also ensures that
# we lock (and unlock below) always the same region in file on win32.
self._fo.seek(0)
try:
if sys.platform == 'win32':
# We are locking here fixed location in file to use it as
# an exclusive lock on entire file.
msvcrt.locking(self._fo.fileno(), msvcrt.LK_LOCK, 1)
else:
fcntl.flock(self._fo.fileno(), fcntl.LOCK_EX)
except IOError:
self._fo.close()
raise
return self._fo
def __exit__(self, exc_type, exc_value, traceback):
# Flush any buffered data to disk. This is needed to prevent race
# condition which happens from the moment of releasing file lock
# till closing the file.
self._fo.flush()
try:
if sys.platform == 'win32':
self._fo.seek(0)
msvcrt.locking(self._fo.fileno(), msvcrt.LK_UNLCK, 1)
else:
fcntl.flock(self._fo.fileno(), fcntl.LOCK_UN)
finally:
self._fo.close()
return exc_value is None
def __init__(self, save_file):
"Create new object seeded with saved test times from the given file."
self.__times = {} # (test binary, test name) -> runtime in ms
# Protects calls to record_test_time(); other calls are not
# expected to be made concurrently.
self.__lock = threading.Lock()
try:
with TestTimes.LockedFile(save_file, 'rb') as fd:
times = TestTimes.__read_test_times_file(fd)
except IOError:
# We couldn't obtain the lock.
return
# Discard saved times if the format isn't right.
if type(times) is not dict:
return
for ((test_binary, test_name), runtime) in times.items():
if (type(test_binary) is not str or type(test_name) is not str
or type(runtime) not in {int, long, type(None)}):
return
self.__times = times
def get_test_time(self, binary, testname):
"""Return the last duration for the given test as an integer number of
milliseconds, or None if the test failed or if there's no record for it."""
return self.__times.get((binary, testname), None)
def record_test_time(self, binary, testname, runtime_ms):
"""Record that the given test ran in the specified number of
milliseconds. If the test failed, runtime_ms should be None."""
with self.__lock:
self.__times[(binary, testname)] = runtime_ms
def write_to_file(self, save_file):
"Write all the times to file."
try:
with TestTimes.LockedFile(save_file, 'a+b') as fd:
times = TestTimes.__read_test_times_file(fd)
if times is None:
times = self.__times
else:
times.update(self.__times)
# We erase data from file while still holding a lock to it. This
# way reading old test times and appending new ones are atomic
# for external viewer.
fd.seek(0)
fd.truncate()
with gzip.GzipFile(fileobj=fd, mode='wb') as gzf:
cPickle.dump(times, gzf, PICKLE_HIGHEST_PROTOCOL)
except IOError:
pass # ignore errors---saving the times isn't that important
@staticmethod
def __read_test_times_file(fd):
try:
with gzip.GzipFile(fileobj=fd, mode='rb') as gzf:
times = cPickle.load(gzf)
except Exception:
# File doesn't exist, isn't readable, is malformed---whatever.
# Just ignore it.
return None
else:
return times
def find_tests(binaries, additional_args, options, times):
test_count = 0
tasks = []
for test_binary in binaries:
command = [test_binary] + additional_args
if options.gtest_also_run_disabled_tests:
command += ['--gtest_also_run_disabled_tests']
list_command = command + ['--gtest_list_tests']
if options.gtest_filter != '':
list_command += ['--gtest_filter=' + options.gtest_filter]
try:
test_list = subprocess.check_output(list_command,
stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
sys.exit("%s: %s\n%s" % (test_binary, str(e), e.output))
try:
test_list = test_list.split('\n')
except TypeError:
# subprocess.check_output() returns bytes in python3
test_list = test_list.decode(sys.stdout.encoding).split('\n')
command += ['--gtest_color=' + options.gtest_color]
test_group = ''
for line in test_list:
if not line.strip():
continue
if line[0] != " ":
# Remove comments for typed tests and strip whitespace.
test_group = line.split('#')[0].strip()
continue
# Remove comments for parameterized tests and strip whitespace.
line = line.split('#')[0].strip()
if not line:
continue
test_name = test_group + line
if not options.gtest_also_run_disabled_tests and 'DISABLED_' in test_name:
continue
# Skip PRE_ tests which are used by Chromium.
if '.PRE_' in test_name:
continue
last_execution_time = times.get_test_time(test_binary, test_name)
if options.failed and last_execution_time is not None:
continue
test_command = command + ['--gtest_filter=' + test_name]
if (test_count - options.shard_index) % options.shard_count == 0:
for execution_number in range(options.repeat):
tasks.append(
Task(test_binary, test_name, test_command, execution_number + 1,
last_execution_time, options.output_dir))
test_count += 1
# Sort the tasks to run the slowest tests first, so that faster ones can be
# finished in parallel.
return sorted(tasks, reverse=True)
def execute_tasks(tasks, pool_size, task_manager, timeout_seconds,
timeout_per_test, serialize_test_cases):
class WorkerFn(object):
def __init__(self, tasks, running_groups, timeout_per_test):
self.tasks = tasks
self.running_groups = running_groups
self.timeout_per_test = timeout_per_test
self.task_lock = threading.Lock()
def __call__(self):
while True:
with self.task_lock:
for task_id in range(len(self.tasks)):
task = self.tasks[task_id]
if self.running_groups is not None:
test_group = task.test_name.split('.')[0]
if test_group in self.running_groups:
# Try to find other non-running test group.
continue
else:
self.running_groups.add(test_group)
del self.tasks[task_id]
break
else:
# Either there is no tasks left or number or remaining test
# cases (groups) is less than number or running threads.
return
task_manager.run_task(task, self.timeout_per_test)
if self.running_groups is not None:
with self.task_lock:
self.running_groups.remove(test_group)
def start_daemon(func):
t = threading.Thread(target=func)
t.daemon = True
t.start()
return t
timeout = None
try:
if timeout_seconds:
timeout = threading.Timer(timeout_seconds, sigint_handler.interrupt)
timeout.start()
running_groups = set() if serialize_test_cases else None
worker_fn = WorkerFn(tasks, running_groups, timeout_per_test)
workers = [start_daemon(worker_fn) for _ in range(pool_size)]
for worker in workers:
worker.join()
finally:
if timeout:
timeout.cancel()
for task in list(task_manager.started.values()):
task.runtime_ms = timeout_seconds * 1000
task_manager.register_exit(task)
def default_options_parser():
parser = optparse.OptionParser(
usage='usage: %prog [options] binary [binary ...] -- [additional args]')
parser.add_option('-d',
'--output_dir',
type='string',
default=None,
help='Output directory for test logs. Logs will be '
'available under gtest-parallel-logs/, so '
'--output_dir=/tmp will results in all logs being '
'available under /tmp/gtest-parallel-logs/.')
parser.add_option('-r',
'--repeat',
type='int',
default=1,
help='Number of times to execute all the tests.')
parser.add_option('--retry_failed',
type='int',
default=0,
help='Number of times to repeat failed tests.')
parser.add_option('--failed',
action='store_true',
default=False,
help='run only failed and new tests')
parser.add_option('-w',
'--workers',
type='int',
default=multiprocessing.cpu_count(),
help='number of workers to spawn')
parser.add_option('--gtest_color',
type='string',
default='yes',
help='color output')
parser.add_option('--gtest_filter',
type='string',
default='',
help='test filter')
parser.add_option('--gtest_also_run_disabled_tests',
action='store_true',
default=False,
help='run disabled tests too')
parser.add_option(
'--print_test_times',
action='store_true',
default=False,
help='list the run time of each test at the end of execution')
parser.add_option(
'--print_test_command',
action='store_true',
default=False,
help='Print full test command instead of name')
parser.add_option('--shard_count',
type='int',
default=1,
help='total number of shards (for sharding test execution '
'between multiple machines)')
parser.add_option('--shard_index',
type='int',
default=0,
help='zero-indexed number identifying this shard (for '
'sharding test execution between multiple machines)')
parser.add_option(
'--dump_json_test_results',
type='string',
default=None,
help='Saves the results of the tests as a JSON machine-'
'readable file. The format of the file is specified at '
'https://www.chromium.org/developers/the-json-test-results-format')
parser.add_option('--timeout',
type='int',
default=None,
help='Interrupt all remaining processes after the given '
'time (in seconds).')
parser.add_option('--timeout_per_test',
type='int',
default=None,
help='Interrupt single processes after the given '
'time (in seconds).')
parser.add_option('--serialize_test_cases',
action='store_true',
default=False,
help='Do not run tests from the same test '
'case in parallel.')
return parser
def main():
# Remove additional arguments (anything after --).
additional_args = []
for i in range(len(sys.argv)):
if sys.argv[i] == '--':
additional_args = sys.argv[i + 1:]
sys.argv = sys.argv[:i]
break
parser = default_options_parser()
(options, binaries) = parser.parse_args()
if (options.output_dir is not None and not os.path.isdir(options.output_dir)):
parser.error('--output_dir value must be an existing directory, '
'current value is "%s"' % options.output_dir)
# Append gtest-parallel-logs to log output, this is to avoid deleting user
# data if an user passes a directory where files are already present. If a
# user specifies --output_dir=Docs/, we'll create Docs/gtest-parallel-logs
# and clean that directory out on startup, instead of nuking Docs/.
if options.output_dir:
options.output_dir = os.path.join(options.output_dir, 'gtest-parallel-logs')
if binaries == []:
parser.print_usage()
sys.exit(1)
if options.shard_count < 1:
parser.error("Invalid number of shards: %d. Must be at least 1." %
options.shard_count)
if not (0 <= options.shard_index < options.shard_count):
parser.error("Invalid shard index: %d. Must be between 0 and %d "
"(less than the number of shards)." %
(options.shard_index, options.shard_count - 1))
# Check that all test binaries have an unique basename. That way we can ensure
# the logs are saved to unique files even when two different binaries have
# common tests.
unique_binaries = set(os.path.basename(binary) for binary in binaries)
assert len(unique_binaries) == len(binaries), (
"All test binaries must have an unique basename.")
if options.output_dir:
# Remove files from old test runs.
if os.path.isdir(options.output_dir):
shutil.rmtree(options.output_dir)
# Create directory for test log output.
try:
os.makedirs(options.output_dir)
except OSError as e:
# Ignore errors if this directory already exists.
if e.errno != errno.EEXIST or not os.path.isdir(options.output_dir):
raise e
test_results = None
if options.dump_json_test_results is not None:
test_results = CollectTestResults(options.dump_json_test_results)
save_file = get_save_file_path()
times = TestTimes(save_file)
logger = FilterFormat(options.output_dir)
task_manager = TaskManager(times, logger, test_results, Task,
options.retry_failed, options.repeat + 1)
tasks = find_tests(binaries, additional_args, options, times)
logger.log_tasks(len(tasks))
execute_tasks(tasks, options.workers, task_manager, options.timeout,
options.timeout_per_test, options.serialize_test_cases)
print_try_number = options.retry_failed > 0 or options.repeat > 1
if task_manager.passed:
logger.move_to('passed', task_manager.passed)
if options.print_test_times:
logger.print_tests('PASSED TESTS', task_manager.passed, print_try_number, options.print_test_command)
if task_manager.failed:
logger.print_tests('FAILED TESTS', task_manager.failed, print_try_number, options.print_test_command)
logger.move_to('failed', task_manager.failed)
if task_manager.timed_out:
logger.print_tests('TIMED OUT TESTS', task_manager.timed_out, print_try_number, options.print_test_command)
logger.move_to('timed_out', task_manager.timed_out)
if task_manager.started:
logger.print_tests('INTERRUPTED TESTS', task_manager.started.values(),
print_try_number, options.print_test_command)
logger.move_to('interrupted', task_manager.started.values())
if options.repeat > 1 and (task_manager.failed or task_manager.started):
logger.summarize(task_manager.passed, task_manager.failed,
task_manager.started.values())
logger.flush()
times.write_to_file(save_file)
if test_results:
test_results.dump_to_file_and_close()
if sigint_handler.got_sigint():
return -signal.SIGINT
return task_manager.global_exit_code
if __name__ == "__main__":
sys.exit(main())
+173
View File
@@ -0,0 +1,173 @@
# Copyright 2017 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import collections
import threading
import time
class LoggerMock(object):
def __init__(self, test_lib):
self.test_lib = test_lib
self.runtimes = collections.defaultdict(list)
self.exit_codes = collections.defaultdict(list)
self.last_execution_times = collections.defaultdict(list)
self.execution_numbers = collections.defaultdict(list)
def log_exit(self, task):
self.runtimes[task.test_id].append(task.runtime_ms)
self.exit_codes[task.test_id].append(task.exit_code)
self.last_execution_times[task.test_id].append(task.last_execution_time)
self.execution_numbers[task.test_id].append(task.execution_number)
def assertRecorded(self, test_id, expected, retries):
self.test_lib.assertIn(test_id, self.runtimes)
self.test_lib.assertListEqual(expected['runtime_ms'][:retries],
self.runtimes[test_id])
self.test_lib.assertListEqual(expected['exit_code'][:retries],
self.exit_codes[test_id])
self.test_lib.assertListEqual(expected['last_execution_time'][:retries],
self.last_execution_times[test_id])
self.test_lib.assertListEqual(expected['execution_number'][:retries],
self.execution_numbers[test_id])
class TestTimesMock(object):
def __init__(self, test_lib, test_data=None):
self.test_lib = test_lib
self.test_data = test_data or {}
self.last_execution_times = collections.defaultdict(list)
def record_test_time(self, test_binary, test_name, last_execution_time):
test_id = (test_binary, test_name)
self.last_execution_times[test_id].append(last_execution_time)
def get_test_time(self, test_binary, test_name):
test_group, test = test_name.split('.')
return self.test_data.get(test_binary, {}).get(test_group,
{}).get(test, None)
def assertRecorded(self, test_id, expected, retries):
self.test_lib.assertIn(test_id, self.last_execution_times)
self.test_lib.assertListEqual(expected['last_execution_time'][:retries],
self.last_execution_times[test_id])
class TestResultsMock(object):
def __init__(self, test_lib):
self.results = []
self.test_lib = test_lib
def log(self, test_name, runtime_ms, actual_result):
self.results.append((test_name, runtime_ms, actual_result))
def assertRecorded(self, test_id, expected, retries):
test_results = [
(test_id[1], runtime_ms / 1000.0, exit_code)
for runtime_ms, exit_code in zip(expected['runtime_ms'][:retries],
expected['exit_code'][:retries])
]
for test_result in test_results:
self.test_lib.assertIn(test_result, self.results)
class TaskManagerMock(object):
def __init__(self):
self.running_groups = []
self.check_lock = threading.Lock()
self.had_running_parallel_groups = False
self.total_tasks_run = 0
self.started = {}
def __register_start(self, task):
self.started[task.task_id] = task
def register_exit(self, task):
self.started.pop(task.task_id)
def run_task(self, task, timeout_per_test):
self.__register_start(task)
test_group = task.test_name.split('.')[0]
with self.check_lock:
self.total_tasks_run += 1
if test_group in self.running_groups:
self.had_running_parallel_groups = True
self.running_groups.append(test_group)
# Delay as if real test were run.
time.sleep(0.001)
with self.check_lock:
self.running_groups.remove(test_group)
class TaskMockFactory(object):
def __init__(self, test_data):
self.data = test_data
self.passed = []
self.failed = []
def get_task(self, test_id, execution_number=0):
task = TaskMock(test_id, execution_number, self.data[test_id])
if task.exit_code == 0:
self.passed.append(task)
else:
self.failed.append(task)
return task
def __call__(self, test_binary, test_name, test_command, execution_number,
last_execution_time, output_dir):
return self.get_task((test_binary, test_name), execution_number)
class TaskMock(object):
def __init__(self, test_id, execution_number, test_data):
self.test_id = test_id
self.execution_number = execution_number
self.runtime_ms = test_data['runtime_ms'][execution_number]
self.exit_code = test_data['exit_code'][execution_number]
self.last_execution_time = (
test_data['last_execution_time'][execution_number])
if 'log_file' in test_data:
self.log_file = test_data['log_file'][execution_number]
else:
self.log_file = None
self.test_command = None
self.output_dir = None
self.test_binary = test_id[0]
self.test_name = test_id[1]
self.task_id = (test_id[0], test_id[1], execution_number)
def run(self, timeout_per_test):
pass
class SubprocessMock(object):
def __init__(self, test_data=None):
self._test_data = test_data
self.last_invocation = None
def __call__(self, command, **kwargs):
self.last_invocation = command
binary = command[0]
test_list = []
tests_for_binary = sorted(self._test_data.get(binary, {}).items())
for test_group, tests in tests_for_binary:
test_list.append(test_group + ".")
for test in sorted(tests):
test_list.append(" " + test)
return '\n'.join(test_list)
+656
View File
@@ -0,0 +1,656 @@
#!/usr/bin/env python
# Copyright 2017 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import contextlib
import os.path
import random
import shutil
import sys
import tempfile
import threading
import unittest
import gtest_parallel
from gtest_parallel_mocks import LoggerMock
from gtest_parallel_mocks import SubprocessMock
from gtest_parallel_mocks import TestTimesMock
from gtest_parallel_mocks import TestResultsMock
from gtest_parallel_mocks import TaskManagerMock
from gtest_parallel_mocks import TaskMockFactory
from gtest_parallel_mocks import TaskMock
@contextlib.contextmanager
def guard_temp_dir():
try:
temp_dir = tempfile.mkdtemp()
yield temp_dir
finally:
shutil.rmtree(temp_dir)
@contextlib.contextmanager
def guard_temp_subdir(temp_dir, *path):
assert path, 'Path should not be empty'
try:
temp_subdir = os.path.join(temp_dir, *path)
os.makedirs(temp_subdir)
yield temp_subdir
finally:
shutil.rmtree(os.path.join(temp_dir, path[0]))
@contextlib.contextmanager
def guard_patch_module(import_name, new_val):
def patch(module, names, val):
if len(names) == 1:
old = getattr(module, names[0])
setattr(module, names[0], val)
return old
else:
return patch(getattr(module, names[0]), names[1:], val)
try:
old_val = patch(gtest_parallel, import_name.split('.'), new_val)
yield old_val
finally:
patch(gtest_parallel, import_name.split('.'), old_val)
class TestTaskManager(unittest.TestCase):
def setUp(self):
self.passing_task = (('fake_binary', 'Fake.PassingTest'), {
'runtime_ms': [10],
'exit_code': [0],
'last_execution_time': [10],
})
self.failing_task = (('fake_binary', 'Fake.FailingTest'), {
'runtime_ms': [20, 30, 40],
'exit_code': [1, 1, 1],
'last_execution_time': [None, None, None],
})
self.fails_once_then_succeeds = (('another_binary', 'Fake.Test.FailOnce'), {
'runtime_ms': [21, 22],
'exit_code': [1, 0],
'last_execution_time': [None, 22],
})
self.fails_twice_then_succeeds = (('yet_another_binary',
'Fake.Test.FailTwice'), {
'runtime_ms': [23, 25, 24],
'exit_code': [1, 1, 0],
'last_execution_time':
[None, None, 24],
})
def execute_tasks(self, tasks, retries, expected_exit_code):
repeat = 1
times = TestTimesMock(self)
logger = LoggerMock(self)
test_results = TestResultsMock(self)
task_mock_factory = TaskMockFactory(dict(tasks))
task_manager = gtest_parallel.TaskManager(times, logger, test_results,
task_mock_factory, retries,
repeat)
for test_id, expected in tasks:
task = task_mock_factory.get_task(test_id)
task_manager.run_task(task, timeout_per_test=30)
expected['execution_number'] = list(range(len(expected['exit_code'])))
logger.assertRecorded(test_id, expected, retries + 1)
times.assertRecorded(test_id, expected, retries + 1)
test_results.assertRecorded(test_id, expected, retries + 1)
self.assertEqual(len(task_manager.started), 0)
self.assertListEqual(
sorted(task.task_id for task in task_manager.passed),
sorted(task.task_id for task in task_mock_factory.passed))
self.assertListEqual(
sorted(task.task_id for task in task_manager.failed),
sorted(task.task_id for task in task_mock_factory.failed))
self.assertEqual(task_manager.global_exit_code, expected_exit_code)
def test_passing_task_succeeds(self):
self.execute_tasks(tasks=[self.passing_task],
retries=0,
expected_exit_code=0)
def test_failing_task_fails(self):
self.execute_tasks(tasks=[self.failing_task],
retries=0,
expected_exit_code=1)
def test_failing_task_fails_even_with_retries(self):
self.execute_tasks(tasks=[self.failing_task],
retries=2,
expected_exit_code=1)
def test_executing_passing_and_failing_fails(self):
# Executing both a faling test and a passing one should make gtest-parallel
# fail, no matter if the failing task is run first or last.
self.execute_tasks(tasks=[self.failing_task, self.passing_task],
retries=2,
expected_exit_code=1)
self.execute_tasks(tasks=[self.passing_task, self.failing_task],
retries=2,
expected_exit_code=1)
def test_task_succeeds_with_one_retry(self):
# Executes test and retries once. The first run should fail and the second
# succeed, so gtest-parallel should succeed.
self.execute_tasks(tasks=[self.fails_once_then_succeeds],
retries=1,
expected_exit_code=0)
def test_task_fails_with_one_retry(self):
# Executes test and retries once, not enough for the test to start passing,
# so gtest-parallel should return an error.
self.execute_tasks(tasks=[self.fails_twice_then_succeeds],
retries=1,
expected_exit_code=1)
def test_runner_succeeds_when_all_tasks_eventually_succeeds(self):
# Executes the test and retries twice. One test should pass in the first
# attempt, another should take two runs, and the last one should take three
# runs. All tests should succeed, so gtest-parallel should succeed too.
self.execute_tasks(tasks=[
self.passing_task, self.fails_once_then_succeeds,
self.fails_twice_then_succeeds
],
retries=2,
expected_exit_code=0)
class TestSaveFilePath(unittest.TestCase):
class StreamMock(object):
def write(*args):
# Suppress any output.
pass
def test_get_save_file_path_unix(self):
with guard_temp_dir() as temp_dir, \
guard_patch_module('os.path.expanduser', lambda p: temp_dir), \
guard_patch_module('sys.stderr', TestSaveFilePath.StreamMock()), \
guard_patch_module('sys.platform', 'darwin'):
with guard_patch_module('os.environ', {}), \
guard_temp_subdir(temp_dir, '.cache'):
self.assertEqual(os.path.join(temp_dir, '.cache', 'gtest-parallel'),
gtest_parallel.get_save_file_path())
with guard_patch_module('os.environ', {'XDG_CACHE_HOME': temp_dir}):
self.assertEqual(os.path.join(temp_dir, 'gtest-parallel'),
gtest_parallel.get_save_file_path())
with guard_patch_module('os.environ',
{'XDG_CACHE_HOME': os.path.realpath(__file__)}):
self.assertEqual(os.path.join(temp_dir, '.gtest-parallel-times'),
gtest_parallel.get_save_file_path())
def test_get_save_file_path_win32(self):
with guard_temp_dir() as temp_dir, \
guard_patch_module('os.path.expanduser', lambda p: temp_dir), \
guard_patch_module('sys.stderr', TestSaveFilePath.StreamMock()), \
guard_patch_module('sys.platform', 'win32'):
with guard_patch_module('os.environ', {}), \
guard_temp_subdir(temp_dir, 'AppData', 'Local'):
self.assertEqual(
os.path.join(temp_dir, 'AppData', 'Local', 'gtest-parallel'),
gtest_parallel.get_save_file_path())
with guard_patch_module('os.environ', {'LOCALAPPDATA': temp_dir}):
self.assertEqual(os.path.join(temp_dir, 'gtest-parallel'),
gtest_parallel.get_save_file_path())
with guard_patch_module('os.environ',
{'LOCALAPPDATA': os.path.realpath(__file__)}):
self.assertEqual(os.path.join(temp_dir, '.gtest-parallel-times'),
gtest_parallel.get_save_file_path())
class TestSerializeTestCases(unittest.TestCase):
def _execute_tasks(self, max_number_of_test_cases,
max_number_of_tests_per_test_case, max_number_of_repeats,
max_number_of_workers, timeout_per_test, serialize_test_cases):
tasks = []
for test_case in range(max_number_of_test_cases):
for test_name in range(max_number_of_tests_per_test_case):
# All arguments for gtest_parallel.Task except for test_name are fake.
test_name = 'TestCase{}.test{}'.format(test_case, test_name)
for execution_number in range(random.randint(1, max_number_of_repeats)):
tasks.append(
gtest_parallel.Task('path/to/binary', test_name,
['path/to/binary', '--gtest_filter=*'],
execution_number + 1, None, 'path/to/output'))
expected_tasks_number = len(tasks)
task_manager = TaskManagerMock()
gtest_parallel.execute_tasks(tasks, max_number_of_workers, task_manager,
None, timeout_per_test, serialize_test_cases)
self.assertEqual(serialize_test_cases,
not task_manager.had_running_parallel_groups)
self.assertEqual(expected_tasks_number, task_manager.total_tasks_run)
def test_running_parallel_test_cases_without_repeats(self):
self._execute_tasks(max_number_of_test_cases=4,
max_number_of_tests_per_test_case=32,
max_number_of_repeats=1,
max_number_of_workers=16,
timeout_per_test=30,
serialize_test_cases=True)
def test_running_parallel_test_cases_with_repeats(self):
self._execute_tasks(max_number_of_test_cases=4,
max_number_of_tests_per_test_case=32,
max_number_of_repeats=4,
max_number_of_workers=16,
timeout_per_test=30,
serialize_test_cases=True)
def test_running_parallel_tests(self):
self._execute_tasks(max_number_of_test_cases=4,
max_number_of_tests_per_test_case=128,
max_number_of_repeats=1,
max_number_of_workers=16,
timeout_per_test=30,
serialize_test_cases=False)
class TestTestTimes(unittest.TestCase):
def test_race_in_test_times_load_save(self):
max_number_of_workers = 8
max_number_of_read_write_cycles = 64
test_times_file_name = 'test_times.pickle'
def start_worker(save_file):
def test_times_worker():
thread_id = threading.current_thread().ident
path_to_binary = 'path/to/binary' + hex(thread_id)
for cnt in range(max_number_of_read_write_cycles):
times = gtest_parallel.TestTimes(save_file)
threads_test_times = [
binary for (binary, _) in times._TestTimes__times.keys()
if binary.startswith(path_to_binary)
]
self.assertEqual(cnt, len(threads_test_times))
times.record_test_time('{}-{}'.format(path_to_binary, cnt),
'TestFoo.testBar', 1000)
times.write_to_file(save_file)
self.assertEqual(
1000,
times.get_test_time('{}-{}'.format(path_to_binary, cnt),
'TestFoo.testBar'))
self.assertIsNone(
times.get_test_time('{}-{}'.format(path_to_binary, cnt), 'baz'))
t = threading.Thread(target=test_times_worker)
t.start()
return t
with guard_temp_dir() as temp_dir:
try:
workers = [
start_worker(os.path.join(temp_dir, test_times_file_name))
for _ in range(max_number_of_workers)
]
finally:
for worker in workers:
worker.join()
class TestTimeoutTestCases(unittest.TestCase):
def test_task_timeout(self):
timeout = 1
pool_size = 1
timeout_per_test = 30
task = gtest_parallel.Task('test_binary', 'test_name', ['test_command'], 1,
None, 'output_dir')
tasks = [task]
task_manager = TaskManagerMock()
gtest_parallel.execute_tasks(tasks, pool_size, task_manager, timeout, timeout_per_test, True)
self.assertEqual(1, task_manager.total_tasks_run)
self.assertEqual(None, task.exit_code)
self.assertEqual(1000, task.runtime_ms)
class TestTask(unittest.TestCase):
def test_log_file_names(self):
def root():
return 'C:\\' if sys.platform == 'win32' else '/'
self.assertEqual(os.path.join('.', 'bin-Test_case-100.log'),
gtest_parallel.Task._logname('.', 'bin', 'Test.case', 100))
self.assertEqual(
os.path.join('..', 'a', 'b', 'bin-Test_case_2-1.log'),
gtest_parallel.Task._logname(os.path.join('..', 'a', 'b'),
os.path.join('..', 'bin'), 'Test.case/2',
1))
self.assertEqual(
os.path.join('..', 'a', 'b', 'bin-Test_case_2-5.log'),
gtest_parallel.Task._logname(os.path.join('..', 'a', 'b'),
os.path.join(root(), 'c', 'd', 'bin'),
'Test.case/2', 5))
self.assertEqual(
os.path.join(root(), 'a', 'b', 'bin-Instantiation_Test_case_2-3.log'),
gtest_parallel.Task._logname(os.path.join(root(), 'a', 'b'),
os.path.join('..', 'c', 'bin'),
'Instantiation/Test.case/2', 3))
self.assertEqual(
os.path.join(root(), 'a', 'b', 'bin-Test_case-1.log'),
gtest_parallel.Task._logname(os.path.join(root(), 'a', 'b'),
os.path.join(root(), 'c', 'd', 'bin'),
'Test.case', 1))
def test_logs_to_temporary_files_without_output_dir(self):
log_file = gtest_parallel.Task._logname(None, None, None, None)
self.assertEqual(tempfile.gettempdir(), os.path.dirname(log_file))
os.remove(log_file)
def _execute_run_test(self, run_test_body, interrupt_test):
def popen_mock(*_args, **_kwargs):
return None
class SigHandlerMock(object):
class ProcessWasInterrupted(Exception):
pass
def wait(*_args):
if interrupt_test:
raise SigHandlerMock.ProcessWasInterrupted()
return 42
with guard_temp_dir() as temp_dir, \
guard_patch_module('subprocess.Popen', popen_mock), \
guard_patch_module('sigint_handler', SigHandlerMock()), \
guard_patch_module('thread.exit', lambda: None):
run_test_body(temp_dir)
def test_run_normal_task(self):
def run_test(temp_dir):
task = gtest_parallel.Task('fake/binary', 'test', ['fake/binary'], 1,
None, temp_dir)
self.assertFalse(os.path.isfile(task.log_file))
task.run(timeout_per_test=30)
self.assertTrue(os.path.isfile(task.log_file))
self.assertEqual(42, task.exit_code)
self._execute_run_test(run_test, False)
def test_run_interrupted_task_with_transient_log(self):
def run_test(_):
task = gtest_parallel.Task('fake/binary', 'test', ['fake/binary'], 1,
None, None)
self.assertTrue(os.path.isfile(task.log_file))
task.run(timeout_per_test=30)
self.assertTrue(os.path.isfile(task.log_file))
self.assertIsNone(task.exit_code)
self._execute_run_test(run_test, True)
class TestFilterFormat(unittest.TestCase):
def _execute_test(self, test_body, drop_output):
class StdoutMock(object):
def isatty(*_args):
return False
def write(*args):
pass
def flush(*args):
pass
with guard_temp_dir() as temp_dir, \
guard_patch_module('sys.stdout', StdoutMock()):
logger = gtest_parallel.FilterFormat(None if drop_output else temp_dir)
logger.log_tasks(42)
test_body(logger)
logger.flush()
def test_no_output_dir(self):
def run_test(logger):
passed = [
TaskMock(
('fake/binary', 'FakeTest'), 0, {
'runtime_ms': [10],
'exit_code': [0],
'last_execution_time': [10],
'log_file': [os.path.join(tempfile.gettempdir(), 'fake.log')]
})
]
open(passed[0].log_file, 'w').close()
self.assertTrue(os.path.isfile(passed[0].log_file))
logger.log_exit(passed[0])
self.assertFalse(os.path.isfile(passed[0].log_file))
logger.print_tests('', passed, print_try_number=True, print_test_command=True)
logger.move_to(None, passed)
logger.summarize(passed, [], [])
self._execute_test(run_test, True)
def test_with_output_dir(self):
def run_test(logger):
failed = [
TaskMock(
('fake/binary', 'FakeTest'), 0, {
'runtime_ms': [10],
'exit_code': [1],
'last_execution_time': [10],
'log_file': [os.path.join(logger.output_dir, 'fake.log')]
})
]
open(failed[0].log_file, 'w').close()
self.assertTrue(os.path.isfile(failed[0].log_file))
logger.log_exit(failed[0])
self.assertTrue(os.path.isfile(failed[0].log_file))
logger.print_tests('', failed, print_try_number=True, print_test_command=True)
logger.move_to('failed', failed)
self.assertFalse(os.path.isfile(failed[0].log_file))
self.assertTrue(
os.path.isfile(os.path.join(logger.output_dir, 'failed', 'fake.log')))
logger.summarize([], failed, [])
self._execute_test(run_test, False)
class TestFindTests(unittest.TestCase):
ONE_DISABLED_ONE_ENABLED_TEST = {
"fake_unittests": {
"FakeTest": {
"Test1": None,
"DISABLED_Test2": None,
}
}
}
ONE_FAILED_ONE_PASSED_TEST = {
"fake_unittests": {
"FakeTest": {
# Failed (and new) tests have no recorded runtime.
"FailedTest": None,
"Test": 1,
}
}
}
ONE_TEST = {
"fake_unittests": {
"FakeTest": {
"TestSomething": None,
}
}
}
MULTIPLE_BINARIES_MULTIPLE_TESTS_ONE_FAILURE = {
"fake_unittests": {
"FakeTest": {
"TestSomething": None,
"TestSomethingElse": 2,
},
"SomeOtherTest": {
"YetAnotherTest": 3,
},
},
"fake_tests": {
"Foo": {
"Bar": 4,
"Baz": 4,
}
}
}
def _process_options(self, options):
parser = gtest_parallel.default_options_parser()
options, binaries = parser.parse_args(options)
self.assertEqual(len(binaries), 0)
return options
def _call_find_tests(self, test_data, options=None):
subprocess_mock = SubprocessMock(test_data)
options = self._process_options(options or [])
with guard_patch_module('subprocess.check_output', subprocess_mock):
tasks = gtest_parallel.find_tests(test_data.keys(), [], options,
TestTimesMock(self, test_data))
# Clean transient tasks' log files created because
# by default now output_dir is None.
for task in tasks:
if os.path.isfile(task.log_file):
os.remove(task.log_file)
return tasks, subprocess_mock
def test_tasks_are_sorted(self):
tasks, _ = self._call_find_tests(
self.MULTIPLE_BINARIES_MULTIPLE_TESTS_ONE_FAILURE)
self.assertEqual([task.last_execution_time for task in tasks],
[None, 4, 4, 3, 2])
def test_does_not_run_disabled_tests_by_default(self):
tasks, subprocess_mock = self._call_find_tests(
self.ONE_DISABLED_ONE_ENABLED_TEST)
self.assertEqual(len(tasks), 1)
self.assertFalse("DISABLED_" in tasks[0].test_name)
self.assertNotIn("--gtest_also_run_disabled_tests",
subprocess_mock.last_invocation)
def test_runs_disabled_tests_when_asked(self):
tasks, subprocess_mock = self._call_find_tests(
self.ONE_DISABLED_ONE_ENABLED_TEST, ['--gtest_also_run_disabled_tests'])
self.assertEqual(len(tasks), 2)
self.assertEqual(sorted([task.test_name for task in tasks]),
["FakeTest.DISABLED_Test2", "FakeTest.Test1"])
self.assertIn("--gtest_also_run_disabled_tests",
subprocess_mock.last_invocation)
def test_runs_failed_tests_by_default(self):
tasks, _ = self._call_find_tests(self.ONE_FAILED_ONE_PASSED_TEST)
self.assertEqual(len(tasks), 2)
self.assertEqual(sorted([task.test_name for task in tasks]),
["FakeTest.FailedTest", "FakeTest.Test"])
self.assertEqual({task.last_execution_time for task in tasks}, {None, 1})
def test_runs_only_failed_tests_when_asked(self):
tasks, _ = self._call_find_tests(self.ONE_FAILED_ONE_PASSED_TEST,
['--failed'])
self.assertEqual(len(tasks), 1)
self.assertEqual(tasks[0].test_binary, "fake_unittests")
self.assertEqual(tasks[0].test_name, "FakeTest.FailedTest")
self.assertIsNone(tasks[0].last_execution_time)
def test_does_not_apply_gtest_filter_by_default(self):
_, subprocess_mock = self._call_find_tests(self.ONE_TEST)
self.assertFalse(
any(
arg.startswith('--gtest_filter=SomeFilter')
for arg in subprocess_mock.last_invocation))
def test_applies_gtest_filter(self):
_, subprocess_mock = self._call_find_tests(self.ONE_TEST,
['--gtest_filter=SomeFilter'])
self.assertIn('--gtest_filter=SomeFilter', subprocess_mock.last_invocation)
def test_applies_gtest_color_by_default(self):
tasks, _ = self._call_find_tests(self.ONE_TEST)
self.assertEqual(len(tasks), 1)
self.assertIn('--gtest_color=yes', tasks[0].test_command)
def test_applies_gtest_color(self):
tasks, _ = self._call_find_tests(self.ONE_TEST, ['--gtest_color=Lemur'])
self.assertEqual(len(tasks), 1)
self.assertIn('--gtest_color=Lemur', tasks[0].test_command)
def test_repeats_tasks_once_by_default(self):
tasks, _ = self._call_find_tests(self.ONE_TEST)
self.assertEqual(len(tasks), 1)
def test_repeats_tasks_multiple_times(self):
tasks, _ = self._call_find_tests(self.ONE_TEST, ['--repeat=3'])
self.assertEqual(len(tasks), 3)
# Test all tasks have the same test_name, test_binary and test_command
all_tasks_set = set(
(task.test_name, task.test_binary, tuple(task.test_command))
for task in tasks)
self.assertEqual(len(all_tasks_set), 1)
# Test tasks have consecutive execution_numbers starting from 1
self.assertEqual(sorted(task.execution_number for task in tasks), [1, 2, 3])
def test_gtest_list_tests_fails(self):
def exit_mock(*args):
raise AssertionError('Foo')
options = self._process_options([])
with guard_patch_module('sys.exit', exit_mock):
self.assertRaises(AssertionError, gtest_parallel.find_tests,
[sys.executable], [], options, None)
if __name__ == '__main__':
unittest.main()
+6 -6
View File
@@ -1,13 +1,13 @@
#ifndef HDR_MALLOC_H__
#define HDR_MALLOC_H__
void *kv_malloc(size_t size);
void *valkey_malloc(size_t size);
void *zcalloc_num(size_t num, size_t size);
void *kv_realloc(void *ptr, size_t size);
void kv_free(void *ptr);
void *valkey_realloc(void *ptr, size_t size);
void valkey_free(void *ptr);
#define hdr_malloc kv_malloc
#define hdr_malloc valkey_malloc
#define hdr_calloc zcalloc_num
#define hdr_realloc kv_realloc
#define hdr_free kv_free
#define hdr_realloc valkey_realloc
#define hdr_free valkey_free
#endif
+1 -1
View File
@@ -18,7 +18,7 @@ if (NOT EXISTS ${JEMALLOC_INSTALL_DIR}/lib/libjemalloc.a)
message(FATAL_ERROR "Jemalloc configure failed")
endif ()
execute_process(COMMAND make -j${KV_PROCESSOR_COUNT} lib/libjemalloc.a install
execute_process(COMMAND make -j${VALKEY_PROCESSOR_COUNT} lib/libjemalloc.a install
WORKING_DIRECTORY "${JEMALLOC_SRC_DIR}" RESULTS_VARIABLE MAKE_RESULT)
if (NOT ${MAKE_RESULT} EQUAL 0)
+1 -1
View File
@@ -8,7 +8,7 @@ cat <<EOF
/* A macro that is used to indicate that this the jemalloc vendored with the project
* and has been tested with active defragmentation. */
#define KV_VENDORED_JEMALLOC 1
#define VALKEY_VENDORED_JEMALLOC 1
#ifdef __cplusplus
extern "C" {
+1
View File
@@ -9,6 +9,7 @@ extend-exclude = [
[type.c.extend-identifiers]
clen = "clen"
[type.c.extend-words]
cpy = "cpy"
fo = "fo" # Used in sds.c testcase.
# Header files (sv = *.h)
+5 -5
View File
@@ -52,8 +52,8 @@ KiB
libc
libev
libevent
libkv
Libkv
libvalkey
Libvalkey
localhost
Lua
michael
@@ -99,7 +99,7 @@ txt
unparsed
UNSPEC
URI
kv
KV
KV
valkey
Valkey
VALKEY
variadic
+75 -15
View File
@@ -8,12 +8,12 @@ jobs:
runs-on: ubuntu-latest
container: almalinux:8
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install dependencies
run: |
dnf -y install epel-release
dnf -y install gcc make cmake3 openssl openssl-devel libevent-devel procps-ng kv
dnf -y install gcc make cmake3 openssl openssl-devel libevent-devel procps-ng valkey
- name: Build using cmake
env:
@@ -35,24 +35,21 @@ jobs:
runs-on: ubuntu-latest
container: rockylinux:8
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install dependencies
run: |
dnf -y upgrade --refresh
dnf -y install https://rpms.remirepo.net/enterprise/remi-release-8.rpm
dnf -y module install redis:remi-6.0
dnf -y group install "Development Tools"
dnf -y install openssl-devel cmake libevent-devel kv
- name: Build using cmake
dnf -y install openssl-devel cmake libevent-devel valkey
- name: Build using CMake
env:
EXTRA_CMAKE_OPTS: -DENABLE_EXAMPLES:BOOL=ON -DENABLE_TLS:BOOL=ON
run: mkdir build && cd build && cmake .. && make
run: |
mkdir build && cd build && cmake .. && make
cpack -G RPM
- name: Build using Makefile
run: USE_TLS=1 TEST_ASYNC=1 make
- name: Run tests
working-directory: tests
env:
@@ -64,16 +61,79 @@ jobs:
runs-on: ubuntu-latest
name: FreeBSD
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Build in FreeBSD
uses: vmactions/freebsd-vm@05856381fab64eeee9b038a0818f6cec649ca17a # v1.2.3
uses: vmactions/freebsd-vm@c9f815bc7aa0d34c9fdd0619b034a32d6ca7b57e # v1.4.2
with:
prepare: pkg install -y gmake cmake
run: |
gmake
mkdir build && cd build && cmake .. && gmake
solaris:
runs-on: ubuntu-latest
name: Solaris
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Build on Solaris
uses: vmactions/solaris-vm@69d382b4a775b25ea5955e6c1730e9d05047ca0d # v1.3.1
with:
prepare: pkgutil -y -i gmake gcc5core openssl_utils
run: USE_TLS=1 gmake
solaris-developer-studio:
name: Oracle DeveloperStudio
runs-on: ubuntu-24.04
# Secrets containing certificates are not available in forks,
# or for PRs created by dependabot or externally.
if: |
github.repository == 'valkey-io/libvalkey' &&
github.actor != 'dependabot[bot]' &&
(github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository)
env:
PKG_ORACLE_CERT: ${{ secrets.PKG_ORACLE_CERT }}
PKG_ORACLE_KEY: ${{ secrets.PKG_ORACLE_KEY }}
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Transfer Oracle Studio certificates
run: |
set -e
printf '%s\n' "$PKG_ORACLE_CERT" \
> pkg.oracle.com.certificate.pem
printf '%s\n' "$PKG_ORACLE_KEY" \
> pkg.oracle.com.key.pem
- name: Build on Solaris
uses: vmactions/solaris-vm@69d382b4a775b25ea5955e6c1730e9d05047ca0d # v1.3.1
with:
usesh: true
prepare: |
set -e
cp "$GITHUB_WORKSPACE/pkg.oracle.com.key.pem" \
/root/pkg.oracle.com.key.pem
cp "$GITHUB_WORKSPACE/pkg.oracle.com.certificate.pem" \
/root/pkg.oracle.com.certificate.pem
sudo pkg set-publisher \
-k /root/pkg.oracle.com.key.pem \
-c /root/pkg.oracle.com.certificate.pem \
-G "*" \
-g https://pkg.oracle.com/solarisstudio/release \
solarisstudio
sudo pkg install --accept developerstudio-126/cc
run: |
set -e
PATH=/opt/developerstudio12.6/bin:"$PATH"
export PATH
gmake USE_THREADS=1 USE_TLS=1 -j"$(psrinfo -p)"
build-cross:
name: Cross-compile ${{ matrix.config.target }}
runs-on: ubuntu-22.04
@@ -94,9 +154,9 @@ jobs:
- {target: mipsel, host: mipsel-linux-gnu, qemu: mipsel, gccver: 10 }
- {target: mips64el, host: mips64el-linux-gnuabi64, qemu: mips64el, gccver: 10 }
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: gcc-${{ matrix.config.gccver }}-${{ matrix.config.host }}
version: ${{ matrix.config.target }}-1.0
+43 -36
View File
@@ -9,12 +9,19 @@ jobs:
checkers:
name: Run static checkers
runs-on: ubuntu-latest
env:
LLVM_VERSION: '18'
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install clang-format
run: |
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh ${{ env.LLVM_VERSION }}
sudo apt install -y clang-format-${{ env.LLVM_VERSION }}
- name: Run clang-format style check (.c and .h)
uses: jidicula/clang-format-action@4726374d1aa3c6aecf132e5197e498979588ebc8 # v4.15.0
with:
clang-format-version: '18'
run: |
find . -regex '.*\.\(c\|h\)' -exec clang-format-${{ env.LLVM_VERSION }} -style=file --dry-run --Werror {} +
ubuntu-cmake:
name: Build with CMake ${{ matrix.sanitizer && format('and {0}-sanitizer', matrix.sanitizer ) }} [${{ matrix.compiler }}, cmake-${{ matrix.cmake-version }}, ${{ matrix.cmake-build-type }}]
@@ -43,18 +50,18 @@ jobs:
cmake-build-type: [RelWithDebInfo]
sanitizer: [thread, undefined, leak, address]
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: libevent-dev libuv1-dev libev-dev libglib2.0-dev kv-server
packages: libevent-dev libuv1-dev libev-dev libglib2.0-dev valkey-server
version: 1.0
- name: Setup compiler
uses: aminya/setup-cpp@a276e6e3d1db9160db5edc458e99a30d3b109949 # v1.7.1
uses: aminya/setup-cpp@1f17f92d6a52bfcb1a25348e2c526c2e5cbb1134 # v1.8.0
with:
compiler: ${{ matrix.compiler }}
- name: Setup CMake
uses: jwlawson/actions-setup-cmake@802fa1a2c4e212495c05bf94dba2704a92a472be # v2.0.2
uses: jwlawson/actions-setup-cmake@3a6cbe35ba64df7ca70c51365c4aff65db9a9037 # v2.1.1
with:
cmake-version: ${{ matrix.cmake-version }}
- name: Generate makefiles
@@ -84,14 +91,14 @@ jobs:
name: Build with make
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: libevent-dev valgrind kv-server
packages: libevent-dev valgrind valkey-server
version: 1.0
- name: Build
run: USE_TLS=1 TEST_ASYNC=1 make
run: USE_TLS=1 TEST_ASYNC=1 make -j$(nproc)
- name: Run tests
working-directory: tests
env:
@@ -110,11 +117,11 @@ jobs:
name: Build for 32-bit
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: gcc-multilib kv-server
packages: gcc-multilib valkey-server
version: 1.0
- name: Build
run: |
@@ -128,9 +135,9 @@ jobs:
name: Installation tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: libevent-dev libuv1-dev libev-dev libglib2.0-dev
version: 1.0
@@ -147,16 +154,16 @@ jobs:
mkdir build-static && cd build-static
cmake -DDISABLE_TESTS=ON -DENABLE_TLS=ON -DBUILD_SHARED_LIBS=OFF ..
make DESTDIR=${{ github.workspace }}/static-cmake-install install
- name: Build examples with Makefile using a Makefile-installed libkv
- name: Build examples with Makefile using a Makefile-installed libvalkey
run: |
make STLIBNAME="${{ github.workspace }}/make-install/usr/local/lib/libkv.a" \
TLS_STLIBNAME="${{ github.workspace }}/make-install/usr/local/lib/libkv_tls.a" \
make STLIBNAME="${{ github.workspace }}/make-install/usr/local/lib/libvalkey.a" \
TLS_STLIBNAME="${{ github.workspace }}/make-install/usr/local/lib/libvalkey_tls.a" \
INCLUDE_DIR="${{ github.workspace }}/make-install/usr/local/include" \
USE_TLS=1 -C examples
- name: Build examples with Makefile using a CMake-installed libkv
- name: Build examples with Makefile using a CMake-installed libvalkey
run: |
make STLIBNAME="${{ github.workspace }}/static-cmake-install/usr/local/lib/libkv.a" \
TLS_STLIBNAME="${{ github.workspace }}/static-cmake-install/usr/local/lib/libkv_tls.a" \
make STLIBNAME="${{ github.workspace }}/static-cmake-install/usr/local/lib/libvalkey.a" \
TLS_STLIBNAME="${{ github.workspace }}/static-cmake-install/usr/local/lib/libvalkey_tls.a" \
INCLUDE_DIR="${{ github.workspace }}/static-cmake-install/usr/local/include" \
USE_TLS=1 -C examples
- name: Build examples with CMake using CMake-installed dynamic libraries
@@ -174,9 +181,9 @@ jobs:
name: RDMA support enabled
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: librdmacm-dev libibverbs-dev
version: 1.0
@@ -198,9 +205,9 @@ jobs:
name: CMake 3.7.0 (min. required)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Setup CMake
uses: jwlawson/actions-setup-cmake@802fa1a2c4e212495c05bf94dba2704a92a472be # v2.0.2
uses: jwlawson/actions-setup-cmake@3a6cbe35ba64df7ca70c51365c4aff65db9a9037 # v2.1.1
with:
cmake-version: '3.7.0'
- name: Generate makefiles
@@ -213,15 +220,15 @@ jobs:
name: macOS
runs-on: macos-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install dependencies
run: |
brew update
brew install kv
brew install valkey
- name: Build and install using CMake
run: |
mkdir build && cd build
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_TLS=ON
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_TLS=ON -DENABLE_EXAMPLES=ON
sudo ninja -v install
- name: Build using Makefile
run: USE_TLS=1 make
@@ -235,7 +242,7 @@ jobs:
name: Windows
runs-on: windows-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: ilammy/msvc-dev-cmd@0b201ec74fa43914dc39ae48a89fd1d8cb592756 # v1.13.0
- name: Remove installed OpenSSL 1.1.1 (has reached End of Life)
shell: bash
@@ -266,9 +273,9 @@ jobs:
run: |
git config --global core.autocrlf input
choco install -y memurai-developer
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install Cygwin
uses: cygwin/cygwin-install-action@f2009323764960f80959895c7bc3bb30210afe4d # v6
uses: cygwin/cygwin-install-action@711d29f3da23c9f4a1798e369a6f01198c13b11a # v6
with:
packages: make gcc-core cmake libssl-devel
- name: Build with CMake using Cygwin
@@ -284,9 +291,9 @@ jobs:
name: Windows (MinGW64)
runs-on: windows-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up MinGW
uses: msys2/setup-msys2@40677d36a502eb2cf0fb808cc9dec31bf6152638 # v2.28.0
uses: msys2/setup-msys2@4f806de0a5a7294ffabaff804b38a9b435a73bda # v2.30.0
with:
msystem: mingw64
install: |
+4 -4
View File
@@ -9,12 +9,12 @@ permissions:
jobs:
coverity:
if: github.repository == 'kv-io/libkv'
if: github.repository == 'valkey-io/libvalkey'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install dependencies
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: libevent-dev
version: 1.0
@@ -25,7 +25,7 @@ jobs:
- name: Build, scan and report
uses: vapier/coverity-scan-action@2068473c7bdf8c2fb984a6a40ae76ee7facd7a85 # v1.8.0
with:
project: libkv
project: libvalkey
token: ${{ secrets.COVERITY_TOKEN }}
email: bjorn.a.svensson@est.tech
working-directory: build
+13 -13
View File
@@ -6,33 +6,33 @@ permissions:
contents: read
jobs:
kv:
name: KV ${{ matrix.kv-version }}
valkey:
name: Valkey ${{ matrix.valkey-version }}
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix:
include:
- kv-version: '8.1.0'
- kv-version: '8.0.2'
- kv-version: '7.2.8'
- valkey-version: '8.1.0'
- valkey-version: '8.0.2'
- valkey-version: '7.2.8'
steps:
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: libevent-dev
version: 1.0
- name: Install KV for non-cluster tests
- name: Install Valkey for non-cluster tests
run: |
git clone --depth 1 --branch ${{ matrix.kv-version }} https://github.com/kv-io/kv.git
cd kv && BUILD_TLS=yes make install
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
git clone --depth 1 --branch ${{ matrix.valkey-version }} https://github.com/valkey-io/valkey.git
cd valkey && BUILD_TLS=yes make install
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Create build folder
run: cmake -E make_directory build
- name: Generate makefiles
shell: bash
working-directory: build
run: cmake $GITHUB_WORKSPACE -DCMAKE_BUILD_TYPE=Release -DENABLE_IPV6_TESTS=ON -DTEST_WITH_KV_VERSION=${{ matrix.kv-version }} ..
run: cmake $GITHUB_WORKSPACE -DCMAKE_BUILD_TYPE=Release -DENABLE_IPV6_TESTS=ON -DTEST_WITH_VALKEY_VERSION=${{ matrix.valkey-version }} ..
- name: Build
shell: bash
working-directory: build
@@ -66,7 +66,7 @@ jobs:
- redis-version: '6.2.14'
steps:
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: libevent-dev
version: 1.0
@@ -74,7 +74,7 @@ jobs:
run: |
git clone --depth 1 --branch ${{ matrix.redis-version }} https://github.com/redis/redis.git
cd redis && BUILD_TLS=yes make install
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Create build folder
run: cmake -E make_directory build
+1 -1
View File
@@ -11,7 +11,7 @@ jobs:
runs-on: ubuntu-latest
steps:
# Drafts your next Release notes as Pull Requests are merged into "master"
- uses: release-drafter/release-drafter@b1476f6e6eb133afa41ed8589daba6dc69b4d3f5 # v6.1.0
- uses: release-drafter/release-drafter@6db134d15f3909ccc9eefd369f02bd1e9cffdf97 # v6.2.0
with:
# (Optional) specify config name to use, relative to .github/. Default: release-drafter.yml
config-name: release-drafter-config.yml
+4 -4
View File
@@ -10,18 +10,18 @@ jobs:
spellcheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Run spellcheck
uses: rojopolis/spellcheck-github-actions@35a02bae020e6999c5c37fabaf447f2eb8822ca7 # 0.51.0
uses: rojopolis/spellcheck-github-actions@0bf4b2f91efa259b52c202b09b0c3845c524ff36 # 0.58.0
with:
config_path: .github/spellcheck-settings.yml
task_name: Markdown
typos:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install typos
uses: taiki-e/install-action@ad95d4e02e061d4390c4b66ef5ed56c7fee3d2ce # v2.58.17
uses: taiki-e/install-action@288875dd3d64326724fa6d9593062d9f8ba0b131 # v2.67.30
with:
tool: typos
- name: Run typos
+167 -131
View File
@@ -1,8 +1,8 @@
cmake_minimum_required(VERSION 3.7.0)
macro(getDefinedVersion name)
set(VERSION_REGEX "^#define LIBKV_${name} (.+)$")
file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/include/kv/kv.h"
set(VERSION_REGEX "^#define LIBVALKEY_${name} (.+)$")
file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/include/valkey/valkey.h"
MATCHED_LINE REGEX ${VERSION_REGEX})
string(REGEX REPLACE ${VERSION_REGEX} "\\1" ${name} "${MATCHED_LINE}")
endmacro()
@@ -11,19 +11,21 @@ getDefinedVersion(VERSION_MAJOR)
getDefinedVersion(VERSION_MINOR)
getDefinedVersion(VERSION_PATCH)
set(VERSION "${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}")
MESSAGE("Detected version: ${VERSION}")
message("Detected libvalkey version: ${VERSION}")
PROJECT(kv LANGUAGES "C" VERSION "${VERSION}")
project(libvalkey LANGUAGES "C" VERSION "${VERSION}")
INCLUDE(GNUInstallDirs)
option(ENABLE_THREADS "Enable thread-safe initialization" ON)
OPTION(BUILD_SHARED_LIBS "Build shared libraries" ON)
OPTION(ENABLE_TLS "Build kv_tls for TLS support" OFF)
OPTION(ENABLE_TLS "Build valkey_tls for TLS support" OFF)
OPTION(DISABLE_TESTS "If tests should be compiled or not" OFF)
OPTION(ENABLE_EXAMPLES "Enable building kv examples" OFF)
OPTION(ENABLE_EXAMPLES "Enable building valkey examples" OFF)
option(ENABLE_IPV6_TESTS "Enable IPv6 tests requiring special prerequisites" OFF)
OPTION(ENABLE_RDMA "Build kv_rdma for RDMA support" OFF)
OPTION(ENABLE_RDMA "Build valkey_rdma for RDMA support" OFF)
OPTION(ENABLE_DLOPEN_RDMA "Build valkey_rdma with dynamic loading" OFF)
# Libkv requires C99 (-std=c99)
# Libvalkey requires C99 (-std=c99)
SET(CMAKE_C_STANDARD 99)
set(CMAKE_C_EXTENSIONS OFF)
SET(CMAKE_DEBUG_POSTFIX d)
@@ -36,7 +38,7 @@ else()
add_definitions(-D_CRT_SECURE_NO_WARNINGS -DWIN32_LEAN_AND_MEAN)
endif()
set(kv_sources
set(valkey_sources
src/adlist.c
src/alloc.c
src/async.c
@@ -47,55 +49,74 @@ set(kv_sources
src/net.c
src/read.c
src/sockcompat.c
src/kv.c
src/valkey.c
src/vkutil.c)
# Allow the libkv provided sds and dict types to be replaced by
# compatible implementations (like KV's).
# Allow the libvalkey provided sds and dict types to be replaced by
# compatible implementations (like Valkey's).
# A replaced type is not included in a built archive or shared library.
if(NOT DICT_INCLUDE_DIR)
set(kv_sources ${kv_sources} src/dict.c)
set(valkey_sources ${valkey_sources} src/dict.c)
set(DICT_INCLUDE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/src)
endif()
if(NOT SDS_INCLUDE_DIR)
set(kv_sources ${kv_sources} src/sds.c)
set(valkey_sources ${valkey_sources} src/sds.c)
set(SDS_INCLUDE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/src)
endif()
ADD_LIBRARY(kv ${kv_sources})
ADD_LIBRARY(kv::kv ALIAS kv)
set_target_properties(kv PROPERTIES
ADD_LIBRARY(valkey ${valkey_sources})
ADD_LIBRARY(valkey::valkey ALIAS valkey)
set_target_properties(valkey PROPERTIES
C_VISIBILITY_PRESET hidden
WINDOWS_EXPORT_ALL_SYMBOLS TRUE
SOVERSION "${VERSION_MAJOR}"
VERSION "${VERSION}")
IF(MSVC)
SET_TARGET_PROPERTIES(kv
PROPERTIES COMPILE_FLAGS /Z7)
ENDIF()
if(MSVC)
# Produce object files that contain debug info.
target_compile_options(valkey PRIVATE /Z7)
endif()
set(valkey_link_libraries)
set(valkey_compile_definitions)
IF(WIN32)
TARGET_LINK_LIBRARIES(kv PUBLIC ws2_32 crypt32)
list(APPEND valkey_link_libraries ws2_32 crypt32)
ELSEIF(CMAKE_SYSTEM_NAME MATCHES "FreeBSD")
TARGET_LINK_LIBRARIES(kv PUBLIC m)
list(APPEND valkey_link_libraries m)
ELSEIF(CMAKE_SYSTEM_NAME MATCHES "SunOS")
TARGET_LINK_LIBRARIES(kv PUBLIC socket)
list(APPEND valkey_link_libraries socket)
ENDIF()
TARGET_INCLUDE_DIRECTORIES(kv
if(ENABLE_THREADS)
find_package(Threads REQUIRED)
list(APPEND valkey_link_libraries Threads::Threads)
list(APPEND valkey_compile_definitions VALKEY_USE_THREADS)
endif()
if(valkey_link_libraries)
target_link_libraries(valkey PUBLIC ${valkey_link_libraries})
endif()
if(valkey_compile_definitions)
target_compile_definitions(valkey PRIVATE ${valkey_compile_definitions})
endif()
TARGET_INCLUDE_DIRECTORIES(valkey
PUBLIC
$<INSTALL_INTERFACE:include>
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include/kv>
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include/valkey>
PRIVATE
$<BUILD_INTERFACE:${DICT_INCLUDE_DIR}>
$<BUILD_INTERFACE:${SDS_INCLUDE_DIR}>
)
CONFIGURE_FILE(kv.pc.in kv.pc @ONLY)
CONFIGURE_FILE(valkey.pc.in valkey.pc @ONLY)
set(CPACK_PACKAGE_VENDOR "KV")
set(CPACK_PACKAGE_VENDOR "Valkey")
set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "Minimalistic C client library for Valkey")
set(CPACK_PACKAGE_DESCRIPTION "\
Libkv is a minimalistic C client library for the KV, KeyDB, and Redis databases
Libvalkey is a minimalistic C client library for the Valkey, KeyDB, and Redis databases.
It is minimalistic because it just adds minimal support for the protocol, \
but at the same time it uses a high level printf-alike API in order to make \
@@ -107,63 +128,71 @@ reply parser that is decoupled from the I/O layer. It is a stream parser designe
for easy reusability, which can for instance be used in higher level language bindings \
for efficient reply parsing.
kv only supports the binary-safe RESP protocol, so you can use it with any Redis \
Libvalkey only supports the binary-safe RESP protocol, so you can use it with any Redis \
compatible server >= 1.2.0.
The library comes with multiple APIs. There is the synchronous API, the asynchronous API \
and the reply parsing API.")
set(CPACK_PACKAGE_HOMEPAGE_URL "https://github.com/hanzoai/kv")
set(CPACK_PACKAGE_HOMEPAGE_URL "https://github.com/valkey-io/libvalkey")
set(CPACK_PACKAGE_CONTACT "michael dot grunder at gmail dot com")
set(CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON)
set(CPACK_RPM_PACKAGE_AUTOREQPROV ON)
set(CPACK_RPM_PACKAGE_DESCRIPTION "${CPACK_PACKAGE_DESCRIPTION}")
set(CPACK_RPM_PACKAGE_GROUP "Productivity/Databases/Clients")
set(CPACK_RPM_PACKAGE_LICENSE "BSD-3-Clause")
include(CPack)
INSTALL(TARGETS kv
EXPORT kv-targets
INSTALL(TARGETS valkey
EXPORT valkey-targets
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR})
if (MSVC AND BUILD_SHARED_LIBS)
INSTALL(FILES $<TARGET_PDB_FILE:kv>
DESTINATION ${CMAKE_INSTALL_BINDIR}
CONFIGURATIONS Debug RelWithDebInfo)
if(MSVC AND BUILD_SHARED_LIBS)
# Install linker generated program database file.
install(FILES $<TARGET_PDB_FILE:valkey>
DESTINATION ${CMAKE_INSTALL_BINDIR}
CONFIGURATIONS Debug RelWithDebInfo)
endif()
# Install public headers
install(DIRECTORY include/kv
install(DIRECTORY include/valkey
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}
PATTERN "tls.h" EXCLUDE
PATTERN "rdma.h" EXCLUDE)
PATTERN "rdma.h" EXCLUDE
PATTERN "adapters/macosx.h" EXCLUDE)
if(APPLE)
install(FILES include/valkey/adapters/macosx.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/valkey/adapters)
endif()
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/kv.pc
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/valkey.pc
DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
export(EXPORT kv-targets
FILE "${CMAKE_CURRENT_BINARY_DIR}/kv-targets.cmake"
NAMESPACE kv::)
export(EXPORT valkey-targets
FILE "${CMAKE_CURRENT_BINARY_DIR}/valkey-targets.cmake"
NAMESPACE valkey::)
if(WIN32)
SET(CMAKE_CONF_INSTALL_DIR share/kv)
SET(CMAKE_CONF_INSTALL_DIR share/valkey)
else()
SET(CMAKE_CONF_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR}/cmake/kv)
SET(CMAKE_CONF_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR}/cmake/valkey)
endif()
SET(INCLUDE_INSTALL_DIR include)
include(CMakePackageConfigHelpers)
write_basic_package_version_file("${CMAKE_CURRENT_BINARY_DIR}/kv-config-version.cmake"
write_basic_package_version_file("${CMAKE_CURRENT_BINARY_DIR}/valkey-config-version.cmake"
COMPATIBILITY SameMajorVersion)
configure_package_config_file(kv-config.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/kv-config.cmake
configure_package_config_file(valkey-config.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/valkey-config.cmake
INSTALL_DESTINATION ${CMAKE_CONF_INSTALL_DIR}
PATH_VARS INCLUDE_INSTALL_DIR)
INSTALL(EXPORT kv-targets
FILE kv-targets.cmake
NAMESPACE kv::
INSTALL(EXPORT valkey-targets
FILE valkey-targets.cmake
NAMESPACE valkey::
DESTINATION ${CMAKE_CONF_INSTALL_DIR})
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/kv-config.cmake
${CMAKE_CURRENT_BINARY_DIR}/kv-config-version.cmake
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/valkey-config.cmake
${CMAKE_CURRENT_BINARY_DIR}/valkey-config-version.cmake
DESTINATION ${CMAKE_CONF_INSTALL_DIR})
@@ -174,155 +203,162 @@ IF(ENABLE_TLS)
ENDIF()
ENDIF()
FIND_PACKAGE(OpenSSL REQUIRED)
SET(kv_tls_sources
SET(valkey_tls_sources
src/tls.c)
ADD_LIBRARY(kv_tls ${kv_tls_sources})
ADD_LIBRARY(kv::kv_tls ALIAS kv_tls)
ADD_LIBRARY(valkey_tls ${valkey_tls_sources})
ADD_LIBRARY(valkey::valkey_tls ALIAS valkey_tls)
TARGET_INCLUDE_DIRECTORIES(kv_tls
TARGET_INCLUDE_DIRECTORIES(valkey_tls
PRIVATE
$<INSTALL_INTERFACE:include>
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include/kv>
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include/valkey>
$<BUILD_INTERFACE:${DICT_INCLUDE_DIR}>
$<BUILD_INTERFACE:${SDS_INCLUDE_DIR}>
)
IF (APPLE AND BUILD_SHARED_LIBS)
SET_PROPERTY(TARGET kv_tls PROPERTY LINK_FLAGS "-Wl,-undefined -Wl,dynamic_lookup")
ENDIF()
set_target_properties(kv_tls PROPERTIES
set_target_properties(valkey_tls PROPERTIES
C_VISIBILITY_PRESET hidden
WINDOWS_EXPORT_ALL_SYMBOLS TRUE
SOVERSION "${VERSION_MAJOR}"
VERSION "${VERSION}")
IF(MSVC)
SET_TARGET_PROPERTIES(kv_tls
PROPERTIES COMPILE_FLAGS /Z7)
ENDIF()
TARGET_LINK_LIBRARIES(kv_tls PRIVATE OpenSSL::SSL)
if(WIN32 OR CYGWIN)
target_link_libraries(kv_tls PRIVATE kv)
if(MSVC)
# Produce object files that contain debug info.
target_compile_options(valkey_tls PRIVATE /Z7)
endif()
CONFIGURE_FILE(kv_tls.pc.in kv_tls.pc @ONLY)
target_link_libraries(valkey_tls PRIVATE valkey::valkey OpenSSL::SSL)
CONFIGURE_FILE(valkey_tls.pc.in valkey_tls.pc @ONLY)
INSTALL(TARGETS kv_tls
EXPORT kv_tls-targets
INSTALL(TARGETS valkey_tls
EXPORT valkey_tls-targets
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR})
# Install public header
install(FILES include/kv/tls.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/kv)
install(FILES include/valkey/tls.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/valkey)
if (MSVC AND BUILD_SHARED_LIBS)
INSTALL(FILES $<TARGET_PDB_FILE:kv_tls>
DESTINATION ${CMAKE_INSTALL_BINDIR}
CONFIGURATIONS Debug RelWithDebInfo)
if(MSVC AND BUILD_SHARED_LIBS)
# Install linker generated program database file.
install(FILES $<TARGET_PDB_FILE:valkey_tls>
DESTINATION ${CMAKE_INSTALL_BINDIR}
CONFIGURATIONS Debug RelWithDebInfo)
endif()
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/kv_tls.pc
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/valkey_tls.pc
DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
export(EXPORT kv_tls-targets
FILE "${CMAKE_CURRENT_BINARY_DIR}/kv_tls-targets.cmake"
NAMESPACE kv::)
export(EXPORT valkey_tls-targets
FILE "${CMAKE_CURRENT_BINARY_DIR}/valkey_tls-targets.cmake"
NAMESPACE valkey::)
if(WIN32)
SET(CMAKE_CONF_INSTALL_DIR share/kv_tls)
SET(CMAKE_CONF_INSTALL_DIR share/valkey_tls)
else()
SET(CMAKE_CONF_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR}/cmake/kv_tls)
SET(CMAKE_CONF_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR}/cmake/valkey_tls)
endif()
configure_package_config_file(kv_tls-config.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/kv_tls-config.cmake
configure_package_config_file(valkey_tls-config.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/valkey_tls-config.cmake
INSTALL_DESTINATION ${CMAKE_CONF_INSTALL_DIR}
PATH_VARS INCLUDE_INSTALL_DIR)
INSTALL(EXPORT kv_tls-targets
FILE kv_tls-targets.cmake
NAMESPACE kv::
INSTALL(EXPORT valkey_tls-targets
FILE valkey_tls-targets.cmake
NAMESPACE valkey::
DESTINATION ${CMAKE_CONF_INSTALL_DIR})
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/kv_tls-config.cmake
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/valkey_tls-config.cmake
DESTINATION ${CMAKE_CONF_INSTALL_DIR})
ENDIF()
if(ENABLE_RDMA)
find_library(RDMACM_LIBRARIES rdmacm REQUIRED)
find_library(IBVERBS_LIBRARIES ibverbs REQUIRED)
set(kv_rdma_sources src/rdma.c)
add_library(kv_rdma ${kv_rdma_sources})
add_library(kv::kv_rdma ALIAS kv_rdma)
set(valkey_rdma_sources src/rdma.c)
add_library(valkey_rdma ${valkey_rdma_sources})
add_library(valkey::valkey_rdma ALIAS valkey_rdma)
if (ENABLE_DLOPEN_RDMA)
message(STATUS "libvalkey: Building RDMA with dynamic loading (dlopen)")
target_compile_definitions(valkey_rdma PRIVATE DLOPEN_RDMA)
else()
message(STATUS "libvalkey: Building RDMA with static/hard linking")
find_library(RDMACM_LIBRARIES rdmacm REQUIRED)
find_library(IBVERBS_LIBRARIES ibverbs REQUIRED)
target_link_libraries(valkey_rdma LINK_PRIVATE ${RDMACM_LIBRARIES} ${IBVERBS_LIBRARIES})
endif()
target_link_libraries(kv_rdma LINK_PRIVATE ${RDMACM_LIBRARIES} ${IBVERBS_LIBRARIES})
target_include_directories(kv_rdma
target_include_directories(valkey_rdma
PRIVATE
$<INSTALL_INTERFACE:include>
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include/kv>
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include/valkey>
$<BUILD_INTERFACE:${DICT_INCLUDE_DIR}>
$<BUILD_INTERFACE:${SDS_INCLUDE_DIR}>
)
set_target_properties(kv_rdma PROPERTIES
set_target_properties(valkey_rdma PROPERTIES
C_VISIBILITY_PRESET hidden
WINDOWS_EXPORT_ALL_SYMBOLS TRUE
SOVERSION "${VERSION_MAJOR}"
VERSION "${VERSION}")
configure_file(kv_rdma.pc.in kv_rdma.pc @ONLY)
configure_file(valkey_rdma.pc.in valkey_rdma.pc @ONLY)
install(TARGETS kv_rdma
EXPORT kv_rdma-targets
install(TARGETS valkey_rdma
EXPORT valkey_rdma-targets
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR})
# Install public header
install(FILES include/kv/rdma.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/kv)
install(FILES include/valkey/rdma.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/valkey)
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/kv_rdma.pc
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/valkey_rdma.pc
DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
export(EXPORT kv_rdma-targets
FILE "${CMAKE_CURRENT_BINARY_DIR}/kv_rdma-targets.cmake"
NAMESPACE kv::)
export(EXPORT valkey_rdma-targets
FILE "${CMAKE_CURRENT_BINARY_DIR}/valkey_rdma-targets.cmake"
NAMESPACE valkey::)
set(CMAKE_CONF_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR}/cmake/kv_rdma)
configure_package_config_file(kv_rdma-config.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/kv_rdma-config.cmake
set(CMAKE_CONF_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR}/cmake/valkey_rdma)
configure_package_config_file(valkey_rdma-config.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/valkey_rdma-config.cmake
INSTALL_DESTINATION ${CMAKE_CONF_INSTALL_DIR}
PATH_VARS INCLUDE_INSTALL_DIR)
install(EXPORT kv_rdma-targets
FILE kv_rdma-targets.cmake
NAMESPACE kv::
install(EXPORT valkey_rdma-targets
FILE valkey_rdma-targets.cmake
NAMESPACE valkey::
DESTINATION ${CMAKE_CONF_INSTALL_DIR})
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/kv_rdma-config.cmake
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/valkey_rdma-config.cmake
DESTINATION ${CMAKE_CONF_INSTALL_DIR})
endif()
# Add tests
if(NOT DISABLE_TESTS)
if(BUILD_SHARED_LIBS)
# Test using a static library since symbols are not hidden then.
# Use same source, include dirs and dependencies as the shared library.
add_library(kv_unittest STATIC ${kv_sources})
get_target_property(include_directories kv::kv INCLUDE_DIRECTORIES)
target_include_directories(kv_unittest PUBLIC ${include_directories})
get_target_property(link_libraries kv::kv LINK_LIBRARIES)
if(link_libraries)
target_link_libraries(kv_unittest PUBLIC ${link_libraries})
endif()
# Create libkv_unittest.a in the tests directory.
set_target_properties(kv_unittest PROPERTIES
ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/tests")
else()
# Target is an alias for the static library.
add_library(kv_unittest ALIAS kv)
# Unit tests uses a static library to ensure all symbols are visible.
# This single library also bundles TLS and RDMA when enabled.
add_library(valkey_unittest STATIC ${valkey_sources} ${valkey_tls_sources} ${valkey_rdma_sources})
# Mirror the include directories.
get_target_property(include_directories valkey::valkey INCLUDE_DIRECTORIES)
target_include_directories(valkey_unittest PUBLIC ${include_directories})
if(valkey_compile_definitions)
target_compile_definitions(valkey_unittest PRIVATE ${valkey_compile_definitions})
endif()
if(valkey_link_libraries)
target_link_libraries(valkey_unittest PUBLIC ${valkey_link_libraries})
endif()
if(ENABLE_TLS)
target_link_libraries(valkey_unittest PRIVATE OpenSSL::SSL)
endif()
if(ENABLE_RDMA)
target_link_libraries(valkey_unittest PRIVATE ${RDMACM_LIBRARIES} ${IBVERBS_LIBRARIES})
endif()
# Create libvalkey_unittest.a in the tests directory.
set_target_properties(valkey_unittest PROPERTIES
ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/tests")
# Make sure ctest prints the output when a test fails.
# Must be set before including CTest.
set(CMAKE_CTEST_ARGUMENTS "--output-on-failure")
include(CTest)
add_subdirectory(tests)
+5 -5
View File
@@ -43,7 +43,7 @@ By making a contribution to this project, I certify that:
involved.
```
We require that every contribution to libkv to be signed with a DCO. We require the
We require that every contribution to libvalkey to be signed with a DCO. We require the
usage of known identity (such as a real or preferred name). We do not accept anonymous
contributors nor those utilizing pseudonyms. A DCO signed commit will contain a line like:
@@ -56,7 +56,7 @@ user.name and user.email are set in your git configs, you can use `git commit` w
or `--signoff` to add the `Signed-off-by` line to the end of the commit message. We also
require revert commits to include a DCO.
If you're contributing code to the libkv project in any other form, including
If you're contributing code to the libvalkey project in any other form, including
sending a code fragment or patch via private email or public discussion groups,
you need to ensure that the contribution is in accordance with the DCO.
@@ -69,17 +69,17 @@ Adhere to the existing coding style and make sure to mimic best possible.
### Code formatting
When making a change, please use `git clang-format` or [format-files.sh](./scripts/format-files.sh) to format your changes properly.
This repository is currently using `clang-format` 18.1.3 to format the code, which can be installed using `pip install clang-format==18.1.3` or other preferred method.
This repository is currently using `clang-format` 18.1.8 to format the code, which can be installed using `pip install clang-format==18.1.8` or other preferred method.
## Running cluster tests
Prerequisites:
* Build `libkv` using CMake.
* Build `libvalkey` using CMake.
* Perl with [JSON module](https://metacpan.org/pod/JSON). Can be installed using `sudo cpan JSON`.
* [Docker](https://docs.docker.com/engine/install/)
Some tests needs a KV Cluster which can be setup using build targets.
Some tests needs a Valkey Cluster which can be setup using build targets.
The clusters will be setup using Docker and it may take a while for them to be ready and accepting requests.
Run `make start` to start the clusters and then wait a few seconds before running `make test`.
To stop the running cluster containers run `make stop`.
+114 -66
View File
@@ -1,4 +1,4 @@
# Libkv Makefile
# Libvalkey Makefile
# Copyright (C) 2010-2011 Salvatore Sanfilippo <antirez at gmail dot com>
# Copyright (C) 2010-2011 Pieter Noordhuis <pcnoordhuis at gmail dot com>
# This file is released under the BSD license, see the COPYING file
@@ -10,7 +10,7 @@ OBJ_DIR = obj
LIB_DIR = lib
TEST_DIR = tests
INCLUDE_DIR = include/kv
INCLUDE_DIR = include/valkey
TEST_SRCS = $(TEST_DIR)/client_test.c $(TEST_DIR)/ut_parse_cmd.c $(TEST_DIR)/ut_slotmap_update.c
TEST_BINS = $(patsubst $(TEST_DIR)/%.c,$(TEST_DIR)/%,$(TEST_SRCS))
@@ -18,8 +18,8 @@ TEST_BINS = $(patsubst $(TEST_DIR)/%.c,$(TEST_DIR)/%,$(TEST_SRCS))
SOURCES = $(filter-out $(SRC_DIR)/tls.c $(SRC_DIR)/rdma.c, $(wildcard $(SRC_DIR)/*.c))
HEADERS = $(filter-out $(INCLUDE_DIR)/tls.h $(INCLUDE_DIR)/rdma.h, $(wildcard $(INCLUDE_DIR)/*.h))
# Allow the libkv provided sds and dict types to be replaced by
# compatible implementations (like KV's).
# Allow the libvalkey provided sds and dict types to be replaced by
# compatible implementations (like Valkey's).
# A replaced type is not included in a built archive or shared library.
SDS_INCLUDE_DIR ?= $(SRC_DIR)
DICT_INCLUDE_DIR ?= $(SRC_DIR)
@@ -32,39 +32,39 @@ endif
OBJS = $(patsubst $(SRC_DIR)/%.c,$(OBJ_DIR)/%.o,$(SOURCES))
LIBNAME=libkv
PKGCONFNAME=$(LIB_DIR)/kv.pc
LIBNAME=libvalkey
PKGCONFNAME=$(LIB_DIR)/valkey.pc
PKGCONF_TEMPLATE = kv.pc.in
TLS_PKGCONF_TEMPLATE = kv_tls.pc.in
RDMA_PKGCONF_TEMPLATE = kv_rdma.pc.in
PKGCONF_TEMPLATE = valkey.pc.in
TLS_PKGCONF_TEMPLATE = valkey_tls.pc.in
RDMA_PKGCONF_TEMPLATE = valkey_rdma.pc.in
LIBKV_HEADER=$(INCLUDE_DIR)/kv.h
LIBKV_VERSION=$(shell awk '/LIBKV_VERSION_(MAJOR|MINOR|PATCH)/{gsub(/"/, "", $$3); print $$3}' $(LIBKV_HEADER))
LIBKV_MAJOR=$(word 1,$(LIBKV_VERSION))
LIBKV_MINOR=$(word 2,$(LIBKV_VERSION))
LIBKV_PATCH=$(word 3,$(LIBKV_VERSION))
LIBVALKEY_HEADER=$(INCLUDE_DIR)/valkey.h
LIBVALKEY_VERSION=$(shell awk '/LIBVALKEY_VERSION_(MAJOR|MINOR|PATCH)/{gsub(/"/, "", $$3); print $$3}' $(LIBVALKEY_HEADER))
LIBVALKEY_MAJOR=$(word 1,$(LIBVALKEY_VERSION))
LIBVALKEY_MINOR=$(word 2,$(LIBVALKEY_VERSION))
LIBVALKEY_PATCH=$(word 3,$(LIBVALKEY_VERSION))
# Installation related variables and target
PREFIX?=/usr/local
INCLUDE_PATH?=include/kv
INCLUDE_PATH?=include/valkey
LIBRARY_PATH?=lib
PKGCONF_PATH?=pkgconfig
INSTALL_INCLUDE_PATH= $(DESTDIR)$(PREFIX)/$(INCLUDE_PATH)
INSTALL_LIBRARY_PATH= $(DESTDIR)$(PREFIX)/$(LIBRARY_PATH)
INSTALL_PKGCONF_PATH= $(INSTALL_LIBRARY_PATH)/$(PKGCONF_PATH)
# kv-server configuration used for testing
KV_PORT=56379
KV_SERVER=kv-server
define KV_TEST_CONFIG
# valkey-server configuration used for testing
VALKEY_PORT=56379
VALKEY_SERVER=valkey-server
define VALKEY_TEST_CONFIG
daemonize yes
pidfile /tmp/kv-test-kv.pid
port $(KV_PORT)
pidfile /tmp/valkey-test-valkey.pid
port $(VALKEY_PORT)
bind 127.0.0.1
unixsocket /tmp/kv-test-kv.sock
unixsocket /tmp/valkey-test-valkey.sock
endef
export KV_TEST_CONFIG
export VALKEY_TEST_CONFIG
# Fallback to gcc when $CC is not in $PATH.
CC := $(if $(shell command -v $(firstword $(CC)) >/dev/null 2>&1 && echo OK),$(CC),gcc)
@@ -81,25 +81,25 @@ REAL_LDFLAGS=$(LDFLAGS)
DYLIBSUFFIX=so
STLIBSUFFIX=a
DYLIB_PATCH_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(LIBKV_MAJOR).$(LIBKV_MINOR).$(LIBKV_PATCH)
DYLIB_MAJOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(LIBKV_MAJOR)
DYLIB_PATCH_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(LIBVALKEY_MAJOR).$(LIBVALKEY_MINOR).$(LIBVALKEY_PATCH)
DYLIB_MAJOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(LIBVALKEY_MAJOR)
DYLIB_ROOT_NAME=$(LIBNAME).$(DYLIBSUFFIX)
DYLIBNAME=$(LIB_DIR)/$(DYLIB_ROOT_NAME)
DYLIB_MAKE_CMD=$(CC) $(OPTIMIZATION) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(DYLIB_MAJOR_NAME)
DYLIB_MAKE_CMD=$(CC) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(DYLIB_MAJOR_NAME)
STLIBNAME=$(LIB_DIR)/$(LIBNAME).$(STLIBSUFFIX)
STLIB_MAKE_CMD=$(AR) rcs
#################### TLS variables start ####################
TLS_LIBNAME=libkv_tls
TLS_PKGCONFNAME=$(LIB_DIR)/kv_tls.pc
TLS_LIBNAME=libvalkey_tls
TLS_PKGCONFNAME=$(LIB_DIR)/valkey_tls.pc
TLS_INSTALLNAME=install-tls
TLS_DYLIB_PATCH_NAME=$(TLS_LIBNAME).$(DYLIBSUFFIX).$(LIBKV_MAJOR).$(LIBKV_MINOR).$(LIBKV_PATCH)
TLS_DYLIB_MAJOR_NAME=$(TLS_LIBNAME).$(DYLIBSUFFIX).$(LIBKV_MAJOR)
TLS_DYLIB_PATCH_NAME=$(TLS_LIBNAME).$(DYLIBSUFFIX).$(LIBVALKEY_MAJOR).$(LIBVALKEY_MINOR).$(LIBVALKEY_PATCH)
TLS_DYLIB_MAJOR_NAME=$(TLS_LIBNAME).$(DYLIBSUFFIX).$(LIBVALKEY_MAJOR)
TLS_DYLIB_ROOT_NAME=$(TLS_LIBNAME).$(DYLIBSUFFIX)
TLS_DYLIBNAME=$(LIB_DIR)/$(TLS_LIBNAME).$(DYLIBSUFFIX)
TLS_STLIBNAME=$(LIB_DIR)/$(TLS_LIBNAME).$(STLIBSUFFIX)
TLS_DYLIB_MAKE_CMD=$(CC) $(OPTIMIZATION) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(TLS_DYLIB_MAJOR_NAME)
TLS_DYLIB_MAKE_CMD=$(CC) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(TLS_DYLIB_MAJOR_NAME)
USE_TLS?=0
@@ -108,7 +108,7 @@ ifeq ($(USE_TLS),1)
TLS_OBJS = $(patsubst $(SRC_DIR)/%.c,$(OBJ_DIR)/%.o,$(TLS_SOURCES))
# This is required for test.c only
CFLAGS+=-DKV_TEST_TLS
CFLAGS+=-DVALKEY_TEST_TLS
TLS_STLIB=$(TLS_STLIBNAME)
TLS_DYLIB=$(TLS_DYLIBNAME)
TLS_PKGCONF=$(TLS_PKGCONFNAME)
@@ -122,24 +122,30 @@ endif
##################### TLS variables end #####################
#################### RDMA variables start ####################
RDMA_LIBNAME=libkv_rdma
RDMA_PKGCONFNAME=$(LIB_DIR)/kv_rdma.pc
RDMA_LIBNAME=libvalkey_rdma
RDMA_PKGCONFNAME=$(LIB_DIR)/valkey_rdma.pc
RDMA_INSTALLNAME=install-rdma
RDMA_DYLIB_PATCH_NAME=$(RDMA_LIBNAME).$(DYLIBSUFFIX).$(LIBKV_MAJOR).$(LIBKV_MINOR).$(LIBKV_PATCH)
RDMA_DYLIB_MAJOR_NAME=$(RDMA_LIBNAME).$(DYLIBSUFFIX).$(LIBKV_MAJOR)
RDMA_DYLIB_PATCH_NAME=$(RDMA_LIBNAME).$(DYLIBSUFFIX).$(LIBVALKEY_MAJOR).$(LIBVALKEY_MINOR).$(LIBVALKEY_PATCH)
RDMA_DYLIB_MAJOR_NAME=$(RDMA_LIBNAME).$(DYLIBSUFFIX).$(LIBVALKEY_MAJOR)
RDMA_DYLIB_ROOT_NAME=$(RDMA_LIBNAME).$(DYLIBSUFFIX)
RDMA_DYLIBNAME=$(LIB_DIR)/$(RDMA_LIBNAME).$(DYLIBSUFFIX)
RDMA_STLIBNAME=$(LIB_DIR)/$(RDMA_LIBNAME).$(STLIBSUFFIX)
RDMA_DYLIB_MAKE_CMD=$(CC) $(OPTIMIZATION) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(RDMA_DYLIB_MAJOR_NAME)
RDMA_DYLIB_MAKE_CMD=$(CC) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(RDMA_DYLIB_MAJOR_NAME)
USE_RDMA?=0
USE_DLOPEN_RDMA?=0
ifeq ($(USE_RDMA),1)
RDMA_SOURCES=$(SRC_DIR)/rdma.c
RDMA_OBJS=$(patsubst $(SRC_DIR)/%.c,$(OBJ_DIR)/%.o,$(RDMA_SOURCES))
RDMA_LDFLAGS=-lrdmacm -libverbs
ifeq ($(USE_DLOPEN_RDMA),1)
CFLAGS += -DDLOPEN_RDMA
RDMA_LDFLAGS=
else
RDMA_LDFLAGS=-lrdmacm -libverbs
endif
# This is required for test.c only
CFLAGS+=-DKV_TEST_RDMA
CFLAGS+=-DVALKEY_TEST_RDMA
RDMA_STLIB=$(RDMA_STLIBNAME)
RDMA_DYLIB=$(RDMA_DYLIBNAME)
RDMA_PKGCONF=$(RDMA_PKGCONFNAME)
@@ -157,7 +163,7 @@ uname_S := $(shell uname -s 2>/dev/null || echo not)
# This is required for test.c only
ifeq ($(TEST_ASYNC),1)
export CFLAGS+=-DKV_TEST_ASYNC
export CFLAGS+=-DVALKEY_TEST_ASYNC
endif
ifeq ($(USE_TLS),1)
@@ -182,28 +188,63 @@ ifeq ($(USE_TLS),1)
TLS_LDFLAGS+=-lssl -lcrypto
endif
ifeq ($(uname_S),FreeBSD)
LDFLAGS += -lm
else ifeq ($(UNAME_S),SunOS)
ifeq ($(shell $(CC) -V 2>&1 | grep -iq 'sun\|studio' && echo true),true)
USE_THREADS?=1
ifeq ($(USE_THREADS),1)
CFLAGS+=-DVALKEY_USE_THREADS
endif
PTHREAD_FLAGS :=
ifeq ($(uname_S),Linux)
ifeq ($(USE_THREADS),1)
PTHREAD_FLAGS += -pthread
endif
else ifeq ($(uname_S),FreeBSD)
REAL_LDFLAGS += -lm
ifeq ($(USE_THREADS),1)
PTHREAD_FLAGS += -pthread
endif
else ifeq ($(uname_S),SunOS)
# Solaris' default grep doesn't have -E so we need two checks
CC_VERSION := $(shell $(CC) -V 2>&1 || echo unknown)
ifneq (,$(findstring Sun C,$(CC_VERSION)))
HAVE_SUN_CC := 1
else ifneq (,$(findstring Studio,$(CC_VERSION)))
HAVE_SUN_CC := 1
endif
ifeq ($(HAVE_SUN_CC),1)
SUN_SHARED_FLAG = -G
REAL_CFLAGS += -mt
ifeq ($(USE_THREADS),1)
PTHREAD_FLAGS += -mt
endif
else
SUN_SHARED_FLAG = -shared
ifeq ($(USE_THREADS),1)
PTHREAD_FLAGS += -pthread
endif
endif
REAL_LDFLAGS += -ldl -lnsl -lsocket
DYLIB_MAKE_CMD = $(CC) $(OPTIMIZATION) $(SUN_SHARED_FLAG) -o $(DYLIBNAME) -h $(DYLIB_PATCH_NAME) $(LDFLAGS)
TLS_DYLIB_MAKE_CMD = $(CC) $(SUN_SHARED_FLAG) -o $(TLS_DYLIBNAME) -h $(TLS_DYLIB_PATCH_NAME) $(LDFLAGS) $(TLS_LDFLAGS)
DYLIB_MAKE_CMD = $(CC) $(SUN_SHARED_FLAG) -h $(DYLIB_PATCH_NAME)
TLS_DYLIB_MAKE_CMD = $(CC) $(SUN_SHARED_FLAG) -h $(TLS_DYLIB_PATCH_NAME)
else ifeq ($(uname_S),Darwin)
ifeq ($(USE_THREADS),1)
PTHREAD_FLAGS += -pthread
endif
DYLIBSUFFIX=dylib
DYLIB_PATCH_NAME=$(LIBNAME).$(LIBKV_MAJOR).$(LIBKV_MINOR).$(LIBKV_PATCH).$(DYLIBSUFFIX)
DYLIB_MAJOR_NAME=$(LIBNAME).$(LIBKV_MAJOR).$(DYLIBSUFFIX)
DYLIB_MAKE_CMD=$(CC) -dynamiclib -Wl,-install_name,$(PREFIX)/$(LIBRARY_PATH)/$(DYLIB_PATCH_NAME) -o $(DYLIBNAME) $(LDFLAGS)
TLS_DYLIB_PATCH_NAME=$(TLS_LIBNAME).$(LIBKV_MAJOR).$(LIBKV_MINOR).$(LIBKV_PATCH).$(DYLIBSUFFIX)
TLS_DYLIB_MAJOR_NAME=$(TLS_LIBNAME).$(LIBKV_MAJOR).$(DYLIBSUFFIX)
TLS_DYLIB_MAKE_CMD=$(CC) -dynamiclib -Wl,-install_name,$(PREFIX)/$(LIBRARY_PATH)/$(TLS_DYLIB_PATCH_NAME) -o $(TLS_DYLIBNAME) $(LDFLAGS) $(TLS_LDFLAGS)
DYLIB_PLUGIN=-Wl,-undefined -Wl,dynamic_lookup
DYLIB_PATCH_NAME=$(LIBNAME).$(LIBVALKEY_MAJOR).$(LIBVALKEY_MINOR).$(LIBVALKEY_PATCH).$(DYLIBSUFFIX)
DYLIB_MAJOR_NAME=$(LIBNAME).$(LIBVALKEY_MAJOR).$(DYLIBSUFFIX)
DYLIB_MAKE_CMD=$(CC) -dynamiclib \
-Wl,-install_name,$(PREFIX)/$(LIBRARY_PATH)/$(DYLIB_PATCH_NAME)
TLS_DYLIB_PATCH_NAME=$(TLS_LIBNAME).$(LIBVALKEY_MAJOR).$(LIBVALKEY_MINOR).$(LIBVALKEY_PATCH).$(DYLIBSUFFIX)
TLS_DYLIB_MAJOR_NAME=$(TLS_LIBNAME).$(LIBVALKEY_MAJOR).$(DYLIBSUFFIX)
TLS_DYLIB_MAKE_CMD=$(CC) -dynamiclib \
-Wl,-install_name,$(PREFIX)/$(LIBRARY_PATH)/$(TLS_DYLIB_PATCH_NAME)
endif
REAL_LDFLAGS += $(PTHREAD_FLAGS)
all: dynamic static pkgconfig tests
$(DYLIBNAME): $(OBJS) | $(LIB_DIR)
@@ -212,14 +253,16 @@ $(DYLIBNAME): $(OBJS) | $(LIB_DIR)
$(STLIBNAME): $(OBJS) | $(LIB_DIR)
$(STLIB_MAKE_CMD) $(STLIBNAME) $(OBJS)
$(TLS_DYLIBNAME): $(TLS_OBJS)
$(TLS_DYLIB_MAKE_CMD) $(DYLIB_PLUGIN) -o $(TLS_DYLIBNAME) $(TLS_OBJS) $(REAL_LDFLAGS) $(LDFLAGS) $(TLS_LDFLAGS)
$(TLS_DYLIBNAME): $(TLS_OBJS) $(DYLIBNAME) | $(LIB_DIR)
$(TLS_DYLIB_MAKE_CMD) -o $(TLS_DYLIBNAME) \
$(TLS_OBJS) $(REAL_LDFLAGS) $(DYLIBNAME) $(TLS_LDFLAGS)
$(TLS_STLIBNAME): $(TLS_OBJS)
$(STLIB_MAKE_CMD) $(TLS_STLIBNAME) $(TLS_OBJS)
$(RDMA_DYLIBNAME): $(RDMA_OBJS)
$(RDMA_DYLIB_MAKE_CMD) $(DYLIB_PLUGIN) -o $(RDMA_DYLIBNAME) $(RDMA_OBJS) $(REAL_LDFLAGS) $(LDFLAGS) $(RDMA_LDFLAGS)
$(RDMA_DYLIBNAME): $(RDMA_OBJS) $(DYLIBNAME) | $(LIB_DIR)
$(RDMA_DYLIB_MAKE_CMD) -o $(RDMA_DYLIBNAME) \
$(RDMA_OBJS) $(REAL_LDFLAGS) $(DYLIBNAME) $(RDMA_LDFLAGS)
$(RDMA_STLIBNAME): $(RDMA_OBJS)
$(STLIB_MAKE_CMD) $(RDMA_STLIBNAME) $(RDMA_OBJS)
@@ -230,8 +273,8 @@ $(OBJ_DIR)/%.o: $(SRC_DIR)/%.c | $(OBJ_DIR)
$(OBJ_DIR)/%.o: $(TEST_DIR)/%.c | $(OBJ_DIR)
$(CC) -std=c99 -pedantic $(REAL_CFLAGS) -I$(INCLUDE_DIR) -I$(SDS_INCLUDE_DIR) -I$(DICT_INCLUDE_DIR) -I$(SRC_DIR) -MMD -MP -c $< -o $@
$(TEST_DIR)/%: $(OBJ_DIR)/%.o $(STLIBNAME)
$(CC) -o $@ $< $(RDMA_STLIB) $(STLIBNAME) $(TLS_STLIB) $(LDFLAGS) $(TEST_LDFLAGS)
$(TEST_DIR)/%: $(OBJ_DIR)/%.o $(STLIBNAME) $(TLS_STLIB)
$(CC) -o $@ $< $(RDMA_STLIB) $(STLIBNAME) $(TLS_STLIB) $(REAL_LDFLAGS) $(TEST_LDFLAGS)
$(OBJ_DIR):
mkdir -p $(OBJ_DIR)
@@ -249,7 +292,12 @@ pkgconfig: $(PKGCONFNAME) $(TLS_PKGCONF) $(RDMA_PKGCONF)
TEST_LDFLAGS = $(TLS_LDFLAGS) $(RDMA_LDFLAGS)
ifeq ($(USE_TLS),1)
TEST_LDFLAGS += -pthread
# Tests need pthreads if TLS is enabled, but only add it once
ifeq ($(HAVE_SUN_CC),1)
TEST_LDFLAGS += -mt
else ifeq (,$(findstring -pthread,$(REAL_LDFLAGS) $(TEST_LDFLAGS)))
TEST_LDFLAGS += -pthread
endif
endif
ifeq ($(TEST_ASYNC),1)
TEST_LDFLAGS += -levent
@@ -271,7 +319,7 @@ $(PKGCONFNAME): $(PKGCONF_TEMPLATE)
sed \
-e 's|@CMAKE_INSTALL_PREFIX@|$(PREFIX)|g' \
-e 's|@CMAKE_INSTALL_LIBDIR@|$(INSTALL_LIBRARY_PATH)|g' \
-e 's|@PROJECT_VERSION@|$(LIBKV_SONAME)|g' \
-e 's|@PROJECT_VERSION@|$(LIBVALKEY_SONAME)|g' \
$< > $@
$(TLS_PKGCONFNAME): $(TLS_PKGCONF_TEMPLATE)
@@ -279,7 +327,7 @@ $(TLS_PKGCONFNAME): $(TLS_PKGCONF_TEMPLATE)
sed \
-e 's|@CMAKE_INSTALL_PREFIX@|$(PREFIX)|g' \
-e 's|@CMAKE_INSTALL_LIBDIR@|$(INSTALL_LIBRARY_PATH)|g' \
-e 's|@PROJECT_VERSION@|$(LIBKV_SONAME)|g' \
-e 's|@PROJECT_VERSION@|$(LIBVALKEY_SONAME)|g' \
$< > $@
$(RDMA_PKGCONFNAME): $(RDMA_PKGCONF_TEMPLATE)
@@ -287,7 +335,7 @@ $(RDMA_PKGCONFNAME): $(RDMA_PKGCONF_TEMPLATE)
sed \
-e 's|@CMAKE_INSTALL_PREFIX@|$(PREFIX)|g' \
-e 's|@CMAKE_INSTALL_LIBDIR@|$(INSTALL_LIBRARY_PATH)|g' \
-e 's|@PROJECT_VERSION@|$(LIBKV_SONAME)|g' \
-e 's|@PROJECT_VERSION@|$(LIBVALKEY_SONAME)|g' \
$< > $@
install: $(DYLIBNAME) $(STLIBNAME) $(PKGCONFNAME) $(TLS_INSTALL) $(RDMA_INSTALL)
@@ -340,9 +388,9 @@ gcov:
coverage: gcov
make check
mkdir -p tmp/lcov
lcov -d . -c --exclude '/usr*' -o tmp/lcov/kv.info
lcov -q -l tmp/lcov/kv.info
genhtml --legend -q -o tmp/lcov/report tmp/lcov/kv.info
lcov -d . -c --exclude '/usr*' -o tmp/lcov/valkey.info
lcov -q -l tmp/lcov/valkey.info
genhtml --legend -q -o tmp/lcov/report tmp/lcov/valkey.info
debug:
$(MAKE) OPTIMIZATION="-O0"
+4 -4
View File
@@ -1,6 +1,6 @@
# Libkv
# Libvalkey
Libkv is the official C client for the [KV](https://hanzo.ai) database. It also supports any server that uses the `RESP` protocol (version 2 or 3). This project supports both standalone and cluster modes.
Libvalkey is the official C client for the [Valkey](https://valkey.io) database. It also supports any server that uses the `RESP` protocol (version 2 or 3). This project supports both standalone and cluster modes.
## Table of Contents
@@ -31,7 +31,7 @@ This library supports and is tested against `Linux`, `FreeBSD`, `macOS`, and `Wi
## Building
Libkv is written in C targeting C99. Unfortunately we have no plans on supporting C89 or earlier. The project does use a few widely supported compiler extensions, specifically for the bundled `sds` string library, although we have plans to remove this from the library.
Libvalkey is written in C targeting C99. Unfortunately we have no plans on supporting C89 or earlier. The project does use a few widely supported compiler extensions, specifically for the bundled `sds` string library, although we have plans to remove this from the library.
We support plain GNU make and CMake. Following is information on how to build the library.
@@ -66,4 +66,4 @@ sudo make install
## Contributing
Please see [`CONTRIBUTING.md`](https://github.com/hanzoai/kv/blob/main/CONTRIBUTING.md).
Please see [`CONTRIBUTING.md`](https://github.com/valkey-io/libvalkey/blob/main/CONTRIBUTING.md).
+100 -100
View File
@@ -1,6 +1,6 @@
# Cluster API documentation
This document describes using `libkv` in cluster mode, including an overview of the synchronous and asynchronous APIs.
This document describes using `libvalkey` in cluster mode, including an overview of the synchronous and asynchronous APIs.
It is not intended as a complete reference. For that it's always best to refer to the source code.
## Table of Contents
@@ -34,22 +34,22 @@ There are a few alternative ways to setup and connect to a cluster.
The basic alternatives lacks most options, but can be enough for some use cases.
```c
kvClusterContext *kvClusterConnect(const char *addrs);
kvClusterContext *kvClusterConnectWithTimeout(const char *addrs,
valkeyClusterContext *valkeyClusterConnect(const char *addrs);
valkeyClusterContext *valkeyClusterConnectWithTimeout(const char *addrs,
const struct timeval tv);
```
There is also a convenience struct to specify various options.
```c
kvClusterContext *kvClusterConnectWithOptions(const kvClusterOptions *options);
valkeyClusterContext *valkeyClusterConnectWithOptions(const valkeyClusterOptions *options);
```
When connecting to a cluster, `NULL` is returned when the context can't be allocated, or `err` and `errstr` are set in the returned allocated context when there are issues.
So when connecting it's simple to handle error states.
```c
kvClusterContext *cc = kvClusterConnect("127.0.0.1:6379,127.0.0.1:6380");
valkeyClusterContext *cc = valkeyClusterConnect("127.0.0.1:6379,127.0.0.1:6380");
if (cc == NULL || cc->err) {
fprintf(stderr, "Error: %s\n", cc ? cc->errstr : "OOM");
}
@@ -57,33 +57,33 @@ if (cc == NULL || cc->err) {
### Connection options
There are a variety of options you can specify using the `kvClusterOptions` struct when connecting to a cluster.
There are a variety of options you can specify using the `valkeyClusterOptions` struct when connecting to a cluster.
This includes information about how to connect to the cluster and defining optional callbacks and other options.
See [include/kv/cluster.h](../include/kv/cluster.h) for more details.
See [include/valkey/cluster.h](../include/valkey/cluster.h) for more details.
```c
kvClusterOptions opt = {
valkeyClusterOptions opt = {
.initial_nodes = "127.0.0.1:6379,127.0.0.1:6380"; // Addresses to initially connect to.
.options = KV_OPT_USE_CLUSTER_NODES; // See available option flags below.
.options = VALKEY_OPT_USE_CLUSTER_NODES; // See available option flags below.
.password = "password"; // Authenticate connections using the `AUTH` command.
};
kvClusterContext *cc = kvClusterConnectWithOptions(&opt);
valkeyClusterContext *cc = valkeyClusterConnectWithOptions(&opt);
if (cc == NULL || cc->err) {
fprintf(stderr, "Error: %s\n", cc ? cc->errstr : "OOM");
}
```
There are also several flags you can specify in `kvClusterOptions.options`. It's a bitwise OR of the following flags:
There are also several flags you can specify in `valkeyClusterOptions.options`. It's a bitwise OR of the following flags:
| Flag | Description |
| --- | --- |
| `KV_OPT_USE_CLUSTER_NODES` | Tells libkv to use the command `CLUSTER NODES` when updating its slot map (cluster topology).<br>Libkv uses `CLUSTER SLOTS` by default. |
| `KV_OPT_USE_REPLICAS` | Tells libkv to keep parsed information of replica nodes. |
| `KV_OPT_BLOCKING_INITIAL_UPDATE` | **ASYNC**: Tells libkv to perform the initial slot map update in a blocking fashion. The function call will wait for a slot map update before returning so that the returned context is immediately ready to accept commands. |
| `KV_OPT_REUSEADDR` | Tells libkv to set the [SO_REUSEADDR](https://man7.org/linux/man-pages/man7/socket.7.html) socket option |
| `KV_OPT_PREFER_IPV4`<br>`KV_OPT_PREFER_IPV6`<br>`KV_OPT_PREFER_IP_UNSPEC` | Informs libkv to either prefer IPv4 or IPv6 when invoking [getaddrinfo](https://man7.org/linux/man-pages/man3/gai_strerror.3.html). `KV_OPT_PREFER_IP_UNSPEC` will cause libkv to specify `AF_UNSPEC` in the getaddrinfo call, which means both IPv4 and IPv6 addresses will be searched simultaneously.<br>Libkv prefers IPv4 by default. |
| `KV_OPT_MPTCP` | Tells libkv to use multipath TCP (MPTCP). Note that only when both the server and client are using MPTCP do they establish an MPTCP connection between them; otherwise, they use a regular TCP connection instead. |
| `VALKEY_OPT_USE_CLUSTER_NODES` | Tells libvalkey to use the command `CLUSTER NODES` when updating its slot map (cluster topology).<br>Libvalkey uses `CLUSTER SLOTS` by default. |
| `VALKEY_OPT_USE_REPLICAS` | Tells libvalkey to keep parsed information of replica nodes. |
| `VALKEY_OPT_BLOCKING_INITIAL_UPDATE` | **ASYNC**: Tells libvalkey to perform the initial slot map update in a blocking fashion. The function call will wait for a slot map update before returning so that the returned context is immediately ready to accept commands. |
| `VALKEY_OPT_REUSEADDR` | Tells libvalkey to set the [SO_REUSEADDR](https://man7.org/linux/man-pages/man7/socket.7.html) socket option |
| `VALKEY_OPT_PREFER_IPV4`<br>`VALKEY_OPT_PREFER_IPV6`<br>`VALKEY_OPT_PREFER_IP_UNSPEC` | Informs libvalkey to either prefer IPv4 or IPv6 when invoking [getaddrinfo](https://man7.org/linux/man-pages/man3/gai_strerror.3.html). `VALKEY_OPT_PREFER_IP_UNSPEC` will cause libvalkey to specify `AF_UNSPEC` in the getaddrinfo call, which means both IPv4 and IPv6 addresses will be searched simultaneously.<br>Libvalkey prefers IPv4 by default. |
| `VALKEY_OPT_MPTCP` | Tells libvalkey to use multipath TCP (MPTCP). Note that only when both the server and client are using MPTCP do they establish an MPTCP connection between them; otherwise, they use a regular TCP connection instead. |
### Executing commands
@@ -91,11 +91,11 @@ The primary command interface is a `printf`-like function that takes a format st
This will construct a `RESP` command and deliver it to the correct node in the cluster.
```c
kvReply *reply = kvClusterCommand(cc, "SET %s %d", "counter", 42);
valkeyReply *reply = valkeyClusterCommand(cc, "SET %s %d", "counter", 42);
if (reply == NULL) {
fprintf(stderr, "Communication error: %s\n", cc->err ? cc->errstr : "Unknown error");
} else if (reply->type == KV_REPLY_ERROR) {
} else if (reply->type == VALKEY_REPLY_ERROR) {
fprintf(stderr, "Error response from node: %s\n", reply->str);
} else {
// Handle reply..
@@ -104,11 +104,11 @@ freeReplyObject(reply);
```
Commands will be sent to the cluster node that the client perceives handling the given key.
If the cluster topology has changed the KV node might respond with a redirection error which the client will handle, update its slot map and resend the command to correct node.
If the cluster topology has changed the Valkey node might respond with a redirection error which the client will handle, update its slot map and resend the command to correct node.
The reply will in this case arrive from the correct node.
If a node is unreachable, for example if the command times out or if the connect times out, it can indicate that there has been a failover and the node is no longer part of the cluster.
In this case, `kvClusterCommand` returns NULL and sets `err` and `errstr` on the cluster context, but additionally, libkv schedules a slot map update to be performed when the next command is sent.
In this case, `valkeyClusterCommand` returns NULL and sets `err` and `errstr` on the cluster context, but additionally, libvalkey schedules a slot map update to be performed when the next command is sent.
That means that if you try the same command again, there is a good chance the command will be sent to another node and the command may succeed.
### Executing commands on a specific node
@@ -116,20 +116,20 @@ That means that if you try the same command again, there is a good chance the co
When there is a need to send commands to a specific node, the following low-level API can be used.
```c
kvReply *reply = kvClusterCommandToNode(cc, node, "DBSIZE");
valkeyReply *reply = valkeyClusterCommandToNode(cc, node, "DBSIZE");
```
This function handles `printf`-like arguments similar to `kvClusterCommand`, but will only attempt to send the command to the given node and will not perform redirects or retries.
This function handles `printf`-like arguments similar to `valkeyClusterCommand`, but will only attempt to send the command to the given node and will not perform redirects or retries.
If the command times out or the connection to the node fails, a slot map update is scheduled to be performed when the next command is sent.
`kvClusterCommandToNode` also performs a slot map update if it has previously been scheduled.
`valkeyClusterCommandToNode` also performs a slot map update if it has previously been scheduled.
### Disconnecting/cleanup
To disconnect and free the context the following function can be used:
```c
kvClusterFree(cc);
valkeyClusterFree(cc);
```
This function closes the sockets and deallocates the context.
@@ -138,31 +138,31 @@ This function closes the sockets and deallocates the context.
For pipelined commands, every command is simply appended to the output buffer but not delivered to the server, until you attempt to read the first response, at which point the entire buffer will be delivered.
The `kvClusterAppendCommand` function can be used to append a command, which is identical to the `kvClusterCommand` family, apart from not returning a reply.
After calling an append function `kvClusterGetReply` can be used to receive the subsequent replies.
The `valkeyClusterAppendCommand` function can be used to append a command, which is identical to the `valkeyClusterCommand` family, apart from not returning a reply.
After calling an append function `valkeyClusterGetReply` can be used to receive the subsequent replies.
The following example shows a simple cluster pipeline.
```c
if (kvClusterAppendCommand(cc, "SET foo bar") != KV_OK) {
if (valkeyClusterAppendCommand(cc, "SET foo bar") != VALKEY_OK) {
fprintf(stderr, "Error appending command: %s\n", cc->errstr);
exit(1);
}
if (kvClusterAppendCommand(cc, "GET foo") != KV_OK) {
if (valkeyClusterAppendCommand(cc, "GET foo") != VALKEY_OK) {
fprintf(stderr, "Error appending command: %s\n", cc->errstr);
exit(1);
}
kvReply *reply;
if (kvClusterGetReply(cc,&reply) != KV_OK) {
valkeyReply *reply;
if (valkeyClusterGetReply(cc,&reply) != VALKEY_OK) {
fprintf(stderr, "Error reading reply %zu: %s\n", i, c->errstr);
exit(1);
}
// Handle the reply for SET here.
freeReplyObject(reply);
if (kvClusterGetReply(cc,&reply) != KV_OK) {
if (valkeyClusterGetReply(cc,&reply) != VALKEY_OK) {
fprintf(stderr, "Error reading reply %zu: %s\n", i, c->errstr);
exit(1);
}
@@ -178,28 +178,28 @@ There is a hook to get notified when certain events occur.
```c
/* Function to be called when events occur. */
void event_cb(const kvClusterContext *cc, int event, void *privdata) {
void event_cb(const valkeyClusterContext *cc, int event, void *privdata) {
switch (event) {
// Handle event
}
}
kvClusterOptions opt = {
valkeyClusterOptions opt = {
.event_callback = event_cb;
.event_privdata = my_privdata; // User defined data can be provided to the callback.
};
kvClusterContext *cc = kvClusterConnectWithOptions(&opt);
valkeyClusterContext *cc = valkeyClusterConnectWithOptions(&opt);
```
The callback is called with `event` set to one of the following values:
* `KVCLUSTER_EVENT_SLOTMAP_UPDATED` when the slot mapping has been updated;
* `KVCLUSTER_EVENT_READY` when the slot mapping has been fetched for the first
* `VALKEYCLUSTER_EVENT_SLOTMAP_UPDATED` when the slot mapping has been updated;
* `VALKEYCLUSTER_EVENT_READY` when the slot mapping has been fetched for the first
time and the client is ready to accept commands, useful when initiating the
client using `kvClusterAsyncConnectWithOptions` without enabling the option
`KV_OPT_BLOCKING_INITIAL_UPDATE` where a client is not immediately ready
client using `valkeyClusterAsyncConnectWithOptions` without enabling the option
`VALKEY_OPT_BLOCKING_INITIAL_UPDATE` where a client is not immediately ready
after a successful call;
* `KVCLUSTER_EVENT_FREE_CONTEXT` when the cluster context is being freed, so
* `VALKEYCLUSTER_EVENT_FREE_CONTEXT` when the cluster context is being freed, so
that the user can free the event `privdata`.
#### Events per connection
@@ -209,61 +209,61 @@ This is useful for applying socket options or access endpoint information for a
The callback is registered using an option.
```c
void connect_cb(const kvContext *c, int status) {
void connect_cb(const valkeyContext *c, int status) {
// Perform desired action
}
kvClusterOptions opt = {
valkeyClusterOptions opt = {
.connect_callback = connect_cb;
};
kvClusterContext *cc = kvClusterConnectWithOptions(&opt);
valkeyClusterContext *cc = valkeyClusterConnectWithOptions(&opt);
```
The callback is called just after connect, before TLS handshake and authentication.
On successful connection, `status` is set to `KV_OK` and the `kvContext` can be used, for example,
On successful connection, `status` is set to `VALKEY_OK` and the `valkeyContext` can be used, for example,
to see which IP and port it's connected to or to set socket options directly on the file descriptor which can be accessed as `c->fd`.
On failed connection attempt, this callback is called with `status` set to `KV_ERR`.
The `err` field in the `kvContext` can be used to find out the cause of the error.
On failed connection attempt, this callback is called with `status` set to `VALKEY_ERR`.
The `err` field in the `valkeyContext` can be used to find out the cause of the error.
## Asynchronous API
The asynchronous API supports a wide range of event libraries and uses [adapters](../include/kv/adapters/) to attach to a specific event library.
The asynchronous API supports a wide range of event libraries and uses [adapters](../include/valkey/adapters/) to attach to a specific event library.
Each adapter provide a convenience function that configures which event loop instance the created context will be attached to.
### Connecting
To asynchronously connect to a cluster a `kvClusterOptions` should first be initiated with initial nodes and more,
but it's also important to configure which event library to use before calling `kvClusterAsyncConnectWithOptions`.
To asynchronously connect to a cluster a `valkeyClusterOptions` should first be initiated with initial nodes and more,
but it's also important to configure which event library to use before calling `valkeyClusterAsyncConnectWithOptions`.
```c
kvClusterOptions options = {
valkeyClusterOptions options = {
.initial_nodes = "127.0.0.1:7000";
};
// Use convenience function to set which event library to use.
kvClusterOptionsUseLibev(&options, EV_DEFAULT);
valkeyClusterOptionsUseLibev(&options, EV_DEFAULT);
// Initiate the context and start connecting to the initial nodes.
kvClusterAsyncContext *acc = kvClusterAsyncConnectWithOptions(&options);
valkeyClusterAsyncContext *acc = valkeyClusterAsyncConnectWithOptions(&options);
```
Since an initial slot map update is performed asynchronously any command sent directly after `kvClusterAsyncConnectWithOptions` may fail
Since an initial slot map update is performed asynchronously any command sent directly after `valkeyClusterAsyncConnectWithOptions` may fail
because the initial slot map has not yet been retrieved and the client doesn't know which cluster node to send the command to.
You may use the [`eventCallback`](#events-per-cluster-context-1) to be notified when the slot map is updated and the client is ready to accept commands.
A crude example of using the `eventCallback` can be found in [this test case](../tests/ct_async.c).
Another option is to enable blocking initial slot map updates using the option `KV_OPT_BLOCKING_INITIAL_UPDATE`.
When enabled `kvClusterAsyncConnectWithOptions` will initially connect to the cluster in a blocking fashion and wait for the slot map before returning.
Another option is to enable blocking initial slot map updates using the option `VALKEY_OPT_BLOCKING_INITIAL_UPDATE`.
When enabled `valkeyClusterAsyncConnectWithOptions` will initially connect to the cluster in a blocking fashion and wait for the slot map before returning.
Any command sent by the user thereafter will create a new non-blocking connection, unless a non-blocking connection already exists to the destination.
The function returns a pointer to a newly created `kvClusterAsyncContext` struct and its `err` field should be checked to make sure the initial slot map update was successful.
The function returns a pointer to a newly created `valkeyClusterAsyncContext` struct and its `err` field should be checked to make sure the initial slot map update was successful.
### Connection options
There is a variety of options you can specify using the `kvClusterOptions` struct when connecting to a cluster.
There is a variety of options you can specify using the `valkeyClusterOptions` struct when connecting to a cluster.
One asynchronous API specific option is `KV_OPT_BLOCKING_INITIAL_UPDATE` which enables the initial slot map update to be performed in a blocking fashion.
One asynchronous API specific option is `VALKEY_OPT_BLOCKING_INITIAL_UPDATE` which enables the initial slot map update to be performed in a blocking fashion.
The connect function will wait for a slot map update before returning so that the returned context is immediately ready to accept commands.
See previous [Connection options](#connection-options) section for common options.
@@ -273,21 +273,21 @@ See previous [Connection options](#connection-options) section for common option
Executing commands in an asynchronous context work similarly to the synchronous context, except that you can pass a callback that will be invoked when the reply is received.
```c
int status = kvClusterAsyncCommand(cc, commandCallback, privdata,
int status = valkeyClusterAsyncCommand(cc, commandCallback, privdata,
"SET %s %s", "key", "value");
```
The return value is `KV_OK` when the command was successfully added to the output buffer and `KV_ERR` otherwise.
When the connection is being disconnected per user-request, no new commands may be added to the output buffer and `KV_ERR` is returned.
The return value is `VALKEY_OK` when the command was successfully added to the output buffer and `VALKEY_ERR` otherwise.
When the connection is being disconnected per user-request, no new commands may be added to the output buffer and `VALKEY_ERR` is returned.
Commands will be sent to the cluster node that the client perceives handling the given key.
If the cluster topology has changed the KV node might respond with a redirection error which the client will handle, update its slot map and resend the command to correct node.
If the cluster topology has changed the Valkey node might respond with a redirection error which the client will handle, update its slot map and resend the command to correct node.
The reply will in this case arrive from the correct node.
The reply callback, that is called when the reply is received, should have the following prototype:
```c
void(kvClusterAsyncContext *acc, void *reply, void *privdata);
void(valkeyClusterAsyncContext *acc, void *reply, void *privdata);
```
The `privdata` argument can be used to carry arbitrary data to the callback.
@@ -299,7 +299,7 @@ All pending callbacks are called with a `NULL` reply when the context encountere
When there is a need to send commands to a specific node, the following low-level API can be used.
```c
status = kvClusterAsyncCommandToNode(acc, node, commandCallback, privdata, "DBSIZE");
status = valkeyClusterAsyncCommandToNode(acc, node, commandCallback, privdata, "DBSIZE");
```
This functions will only attempt to send the command to a specific node and will not perform redirects or retries, but communication errors will trigger a slot map update just like the commonly used API.
@@ -310,25 +310,25 @@ This functions will only attempt to send the command to a specific node and will
Asynchronous cluster connections can be terminated using:
```c
kvClusterAsyncDisconnect(acc);
valkeyClusterAsyncDisconnect(acc);
```
When this function is called, connections are **not** immediately terminated.
Instead, new commands are no longer accepted and connections are only terminated when all pending commands have been written to a socket, their respective replies have been read and their respective callbacks have been executed.
After this, the disconnection callback is executed with the `KV_OK` status and the context object is freed.
After this, the disconnection callback is executed with the `VALKEY_OK` status and the context object is freed.
### Events
#### Events per cluster context
Use [`event_callback` in `kvClusterOptions`](#events-per-cluster-context) to get notified when certain events occur.
Use [`event_callback` in `valkeyClusterOptions`](#events-per-cluster-context) to get notified when certain events occur.
When the callback function requires the current `kvClusterAsyncContext`, it can typecast the given `kvClusterContext` to a `kvClusterAsyncContext`.
The `kvClusterAsyncContext` struct is an extension of the `kvClusterContext` struct.
When the callback function requires the current `valkeyClusterAsyncContext`, it can typecast the given `valkeyClusterContext` to a `valkeyClusterAsyncContext`.
The `valkeyClusterAsyncContext` struct is an extension of the `valkeyClusterContext` struct.
```c
void eventCallback(const kvClusterContext *cc, int event, void *privdata) {
kvClusterAsyncContext *acc = (kvClusterAsyncContext *)cc;
void eventCallback(const valkeyClusterContext *cc, int event, void *privdata) {
valkeyClusterAsyncContext *acc = (valkeyClusterAsyncContext *)cc;
}
```
@@ -337,31 +337,31 @@ void eventCallback(const kvClusterContext *cc, int event, void *privdata) {
Because the connections that will be created are non-blocking, the kernel is not able to instantly return if the specified host and port is able to accept a connection.
Instead, use a connect callback to be notified when a connection is established or failed.
Similarly, a disconnect callback can be used to be notified about a disconnected connection (either because of an error or per user request).
The callbacks can be enabled using the following options when calling `kvClusterAsyncConnectWithOptions`:
The callbacks can be enabled using the following options when calling `valkeyClusterAsyncConnectWithOptions`:
```c
kvClusterOptions opt = {
valkeyClusterOptions opt = {
.async_connect_callback = connect_cb;
.async_disconnect_callback = disconnect_cb;
}
```
The connect callback function should have the following prototype, aliased to `kvConnectCallback`:
The connect callback function should have the following prototype, aliased to `valkeyConnectCallback`:
```c
void(kvAsyncContext *ac, int status);
void(valkeyAsyncContext *ac, int status);
```
On a connection attempt, the `status` argument is set to `KV_OK` when the connection was successful.
The file description of the connection socket can be retrieved from a `kvAsyncContext` as `ac->c->fd`.
On a connection attempt, the `status` argument is set to `VALKEY_OK` when the connection was successful.
The file description of the connection socket can be retrieved from a `valkeyAsyncContext` as `ac->c->fd`.
The disconnect callback function should have the following prototype, aliased to `kvDisconnectCallback`:
The disconnect callback function should have the following prototype, aliased to `valkeyDisconnectCallback`:
```c
void(const kvAsyncContext *ac, int status);
void(const valkeyAsyncContext *ac, int status);
```
On a disconnection the `status` argument is set to `KV_OK` if it was initiated by the user, or to `KV_ERR` when it was caused by an error.
On a disconnection the `status` argument is set to `VALKEY_OK` if it was initiated by the user, or to `VALKEY_ERR` when it was caused by an error.
When caused by an error the `err` field in the context can be accessed to get the error cause.
You don't need to reconnect in the disconnect callback since libkv will reconnect by itself when the next command is handled.
You don't need to reconnect in the disconnect callback since libvalkey will reconnect by itself when the next command is handled.
## Miscellaneous
@@ -369,22 +369,22 @@ You don't need to reconnect in the disconnect callback since libkv will reconnec
TLS support is not enabled by default and requires an explicit build flag as described in [`README.md`](../README.md#building).
When support is enabled, TLS can be enabled on a cluster context using a prepared `kvTLSContext` and the options `kvClusterOptions.tls` and `kvClusterOptions.tls_init_fn`.
When support is enabled, TLS can be enabled on a cluster context using a prepared `valkeyTLSContext` and the options `valkeyClusterOptions.tls` and `valkeyClusterOptions.tls_init_fn`.
```c
// Initialize the OpenSSL library.
kvInitOpenSSL();
valkeyInitOpenSSL();
// Initialize a kvTLSContext, which holds an OpenSSL context.
kvTLSContext *tls = kvCreateTLSContext("ca.crt", NULL, "client.crt",
// Initialize a valkeyTLSContext, which holds an OpenSSL context.
valkeyTLSContext *tls = valkeyCreateTLSContext("ca.crt", NULL, "client.crt",
"client.key", NULL, NULL);
// Set options to enable TLS on context.
kvClusterOptions opt = {
valkeyClusterOptions opt = {
.tls = tls;
.tls_init_fn = &kvInitiateTLSWithContext;
.tls_init_fn = &valkeyInitiateTLSWithContext;
};
kvClusterContext *cc = kvClusterConnectWithOptions(&opt);
valkeyClusterContext *cc = valkeyClusterConnectWithOptions(&opt);
if (cc == NULL || cc->err) {
fprintf(stderr, "Error: %s\n", cc ? cc->errstr : "OOM");
}
@@ -392,16 +392,16 @@ if (cc == NULL || cc->err) {
### Cluster node iterator
A `kvClusterNodeIterator` can be used to iterate on all known master nodes in a cluster context.
First it needs to be initiated using `kvClusterInitNodeIterator` and then you can repeatedly call `kvClusterNodeNext` to get the next node from the iterator.
A `valkeyClusterNodeIterator` can be used to iterate on all known master nodes in a cluster context.
First it needs to be initiated using `valkeyClusterInitNodeIterator` and then you can repeatedly call `valkeyClusterNodeNext` to get the next node from the iterator.
```c
kvClusterNodeIterator ni;
kvClusterInitNodeIterator(&ni, cc);
valkeyClusterNodeIterator ni;
valkeyClusterInitNodeIterator(&ni, cc);
kvClusterNode *node;
while ((node = kvClusterNodeNext(&ni)) != NULL) {
kvReply *reply = kvClusterCommandToNode(cc, node, "DBSIZE");
valkeyClusterNode *node;
while ((node = valkeyClusterNodeNext(&ni)) != NULL) {
valkeyReply *reply = valkeyClusterCommandToNode(cc, node, "DBSIZE");
// Handle reply..
}
```
@@ -409,19 +409,19 @@ while ((node = kvClusterNodeNext(&ni)) != NULL) {
The iterator will handle changes due to slot map updates by restarting the iteration, but on the new set of master nodes.
There is no bookkeeping for already iterated nodes when a restart is triggered, which means that a node can be iterated over more than once depending on when the slot map update happened and the change of cluster nodes.
Note that when `kvClusterCommandToNode` is called, a slot map update can happen if it has been scheduled by the previous command, for example if the previous call to `kvClusterCommandToNode` timed out or the node wasn't reachable.
Note that when `valkeyClusterCommandToNode` is called, a slot map update can happen if it has been scheduled by the previous command, for example if the previous call to `valkeyClusterCommandToNode` timed out or the node wasn't reachable.
To detect when the slot map has been updated, you can check if the slot map version (`iter.route_version`) is equal to the current cluster context's slot map version (`cc->route_version`).
If it isn't, it means that the slot map has been updated and the iterator will restart itself at the next call to `kvClusterNodeNext`.
If it isn't, it means that the slot map has been updated and the iterator will restart itself at the next call to `valkeyClusterNodeNext`.
Another way to detect that the slot map has been updated is to [register an event callback](#events-per-cluster-context) and look for the event `KVCLUSTER_EVENT_SLOTMAP_UPDATED`.
Another way to detect that the slot map has been updated is to [register an event callback](#events-per-cluster-context) and look for the event `VALKEYCLUSTER_EVENT_SLOTMAP_UPDATED`.
### Extend the list of supported commands
The list of commands and the position of the first key in the command line is defined in [`src/cmddef.h`](../src/cmddef.h) which is included in this repository.
It has been generated using the `JSON` files describing the syntax of each command in the KV repository, which makes sure we support all commands in KV, at least in terms of cluster routing.
It has been generated using the `JSON` files describing the syntax of each command in the Valkey repository, which makes sure we support all commands in Valkey, at least in terms of cluster routing.
To add support for custom commands defined in modules, you can regenerate `cmddef.h` using the script [`gencommands.py`](../script/gencommands.py).
Use the `JSON` files from KV and any additional files on the same format as arguments to the script.
Use the `JSON` files from Valkey and any additional files on the same format as arguments to the script.
For details, see the comments inside `gencommands.py`.
### Random number generator
+39 -39
View File
@@ -1,14 +1,14 @@
# Migration guide
Libkv can replace both libraries `hiredis` and `hiredis-cluster`.
This guide highlights which APIs that have changed and what you need to do when migrating to libkv.
Libvalkey can replace both libraries `hiredis` and `hiredis-cluster`.
This guide highlights which APIs that have changed and what you need to do when migrating to libvalkey.
The general actions needed are:
* Replace the prefix `redis` with `kv` in API usages.
* Replace the prefix `redis` with `valkey` in API usages.
* Replace the term `SSL` with `TLS` in API usages for secure communication.
* Update include paths depending on your previous installation.
All `libkv` headers are now found under `include/kv/`.
All `libvalkey` headers are now found under `include/valkey/`.
* Update used build options, e.g. `USE_TLS` replaces `USE_SSL`.
## Migrating from `hiredis` v1.2.0
@@ -17,19 +17,19 @@ The type `sds` is removed from the public API.
### Renamed API functions
* `redisAsyncSetConnectCallbackNC` is renamed to `kvAsyncSetConnectCallback`.
* `redisAsyncSetConnectCallbackNC` is renamed to `valkeyAsyncSetConnectCallback`.
### Removed API functions
* `redisFormatSdsCommandArgv` removed from API. Can be replaced with `kvFormatCommandArgv`.
* `redisFormatSdsCommandArgv` removed from API. Can be replaced with `valkeyFormatCommandArgv`.
* `redisFreeSdsCommand` removed since the `sds` type is for internal use only.
* `redisAsyncSetConnectCallback` is removed, but can be replaced with `kvAsyncSetConnectCallback` which accepts the non-const callback function prototype.
* `redisAsyncSetConnectCallback` is removed, but can be replaced with `valkeyAsyncSetConnectCallback` which accepts the non-const callback function prototype.
### Renamed API defines
* `HIREDIS_MAJOR` is renamed to `LIBKV_VERSION_MAJOR`.
* `HIREDIS_MINOR` is renamed to `LIBKV_VERSION_MINOR`.
* `HIREDIS_PATCH` is renamed to `LIBKV_VERSION_PATCH`.
* `HIREDIS_MAJOR` is renamed to `LIBVALKEY_VERSION_MAJOR`.
* `HIREDIS_MINOR` is renamed to `LIBVALKEY_VERSION_MINOR`.
* `HIREDIS_PATCH` is renamed to `LIBVALKEY_VERSION_PATCH`.
### Removed API defines
@@ -37,7 +37,7 @@ The type `sds` is removed from the public API.
## Migrating from `hiredis-cluster` 0.14.0
* The cluster client initiation procedure is changed and `kvClusterOptions`
* The cluster client initiation procedure is changed and `valkeyClusterOptions`
should be used to specify options when creating a context.
See documentation for configuration examples when using the
[Synchronous API](cluster.md#synchronous-api) or the
@@ -45,51 +45,51 @@ The type `sds` is removed from the public API.
The [examples](../examples/) directory also contains some common client
initiation examples that might be helpful.
* The default command to update the internal slot map is changed to `CLUSTER SLOTS`.
`CLUSTER NODES` can be re-enabled through options using `KV_OPT_USE_CLUSTER_NODES`.
* A `kvClusterAsyncContext` now embeds a `kvClusterContext` instead of
`CLUSTER NODES` can be re-enabled through options using `VALKEY_OPT_USE_CLUSTER_NODES`.
* A `valkeyClusterAsyncContext` now embeds a `valkeyClusterContext` instead of
holding a pointer to it. Replace any use of `acc->cc` with `&acc->cc` or similar.
### Renamed API functions
* `ctx_get_by_node` is renamed to `kvClusterGetKVContext`.
* `actx_get_by_node` is renamed to `kvClusterGetKVAsyncContext`.
* `ctx_get_by_node` is renamed to `valkeyClusterGetValkeyContext`.
* `actx_get_by_node` is renamed to `valkeyClusterGetValkeyAsyncContext`.
### Renamed API defines
* `REDIS_ROLE_NULL` is renamed to `KV_ROLE_UNKNOWN`.
* `REDIS_ROLE_MASTER` is renamed to `KV_ROLE_PRIMARY`.
* `REDIS_ROLE_SLAVE` is renamed to `KV_ROLE_REPLICA`.
* `REDIS_ROLE_NULL` is renamed to `VALKEY_ROLE_UNKNOWN`.
* `REDIS_ROLE_MASTER` is renamed to `VALKEY_ROLE_PRIMARY`.
* `REDIS_ROLE_SLAVE` is renamed to `VALKEY_ROLE_REPLICA`.
### Removed API functions
* `redisClusterConnect2` removed, use `kvClusterConnectWithOptions`.
* `redisClusterContextInit` removed, use `kvClusterConnectWithOptions`.
* `redisClusterSetConnectCallback` removed, use `kvClusterOptions.connect_callback`.
* `redisClusterSetEventCallback` removed, use `kvClusterOptions.event_callback`.
* `redisClusterSetMaxRedirect` removed, use `kvClusterOptions.max_retry`.
* `redisClusterSetOptionAddNode` removed, use `kvClusterOptions.initial_nodes`.
* `redisClusterSetOptionAddNodes` removed, use `kvClusterOptions.initial_nodes`.
* `redisClusterConnect2` removed, use `valkeyClusterConnectWithOptions`.
* `redisClusterContextInit` removed, use `valkeyClusterConnectWithOptions`.
* `redisClusterSetConnectCallback` removed, use `valkeyClusterOptions.connect_callback`.
* `redisClusterSetEventCallback` removed, use `valkeyClusterOptions.event_callback`.
* `redisClusterSetMaxRedirect` removed, use `valkeyClusterOptions.max_retry`.
* `redisClusterSetOptionAddNode` removed, use `valkeyClusterOptions.initial_nodes`.
* `redisClusterSetOptionAddNodes` removed, use `valkeyClusterOptions.initial_nodes`.
* `redisClusterSetOptionConnectBlock` removed since it was deprecated.
* `redisClusterSetOptionConnectNonBlock` removed since it was deprecated.
* `redisClusterSetOptionConnectTimeout` removed, use `kvClusterOptions.connect_timeout`.
* `redisClusterSetOptionMaxRetry` removed, use `kvClusterOptions.max_retry`.
* `redisClusterSetOptionParseSlaves` removed, use `kvClusterOptions.options` and `KV_OPT_USE_REPLICAS`.
* `redisClusterSetOptionPassword` removed, use `kvClusterOptions.password`.
* `redisClusterSetOptionConnectTimeout` removed, use `valkeyClusterOptions.connect_timeout`.
* `redisClusterSetOptionMaxRetry` removed, use `valkeyClusterOptions.max_retry`.
* `redisClusterSetOptionParseSlaves` removed, use `valkeyClusterOptions.options` and `VALKEY_OPT_USE_REPLICAS`.
* `redisClusterSetOptionPassword` removed, use `valkeyClusterOptions.password`.
* `redisClusterSetOptionRouteUseSlots` removed, `CLUSTER SLOTS` is used by default.
* `redisClusterSetOptionUsername` removed, use `kvClusterOptions.username`.
* `redisClusterAsyncConnect` removed, use `kvClusterAsyncConnectWithOptions` with options flag `KV_OPT_BLOCKING_INITIAL_UPDATE`.
* `redisClusterAsyncConnect2` removed, use `kvClusterAsyncConnectWithOptions`.
* `redisClusterAsyncContextInit` removed, `kvClusterAsyncConnectWithOptions` will initiate the context.
* `redisClusterAsyncSetConnectCallback` removed, but `kvClusterOptions.async_connect_callback` can be used which accepts a non-const callback function prototype.
* `redisClusterAsyncSetConnectCallbackNC` removed, use `kvClusterOptions.async_connect_callback`.
* `redisClusterAsyncSetDisconnectCallback` removed, use `kvClusterOptions.async_disconnect_callback`.
* `redisClusterSetOptionUsername` removed, use `valkeyClusterOptions.username`.
* `redisClusterAsyncConnect` removed, use `valkeyClusterAsyncConnectWithOptions` with options flag `VALKEY_OPT_BLOCKING_INITIAL_UPDATE`.
* `redisClusterAsyncConnect2` removed, use `valkeyClusterAsyncConnectWithOptions`.
* `redisClusterAsyncContextInit` removed, `valkeyClusterAsyncConnectWithOptions` will initiate the context.
* `redisClusterAsyncSetConnectCallback` removed, but `valkeyClusterOptions.async_connect_callback` can be used which accepts a non-const callback function prototype.
* `redisClusterAsyncSetConnectCallbackNC` removed, use `valkeyClusterOptions.async_connect_callback`.
* `redisClusterAsyncSetDisconnectCallback` removed, use `valkeyClusterOptions.async_disconnect_callback`.
* `parse_cluster_nodes` removed from API, for internal use only.
* `parse_cluster_slots` removed from API, for internal use only.
### Removed API defines
* `HIRCLUSTER_FLAG_NULL` removed.
* `HIRCLUSTER_FLAG_ADD_SLAVE` removed, flag can be replaced with an option, see `KV_OPT_USE_REPLICAS`.
* `HIRCLUSTER_FLAG_ADD_SLAVE` removed, flag can be replaced with an option, see `VALKEY_OPT_USE_REPLICAS`.
* `HIRCLUSTER_FLAG_ROUTE_USE_SLOTS` removed, the use of `CLUSTER SLOTS` is enabled by default.
### Removed support for splitting multi-key commands per slot
@@ -101,5 +101,5 @@ Commands affected are `DEL`, `EXISTS`, `MGET` and `MSET`.
_Proposed action:_
Partition the keys by slot using `kvClusterGetSlotByKey` before sending affected commands.
Construct new commands when needed and send them using multiple calls to `kvClusterCommand` or equivalent.
Partition the keys by slot using `valkeyClusterGetSlotByKey` before sending affected commands.
Construct new commands when needed and send them using multiple calls to `valkeyClusterCommand` or equivalent.
+98 -98
View File
@@ -1,6 +1,6 @@
# Standalone API documentation
This document describes using `libkv` in standalone (non-cluster) mode, including an overview of the synchronous and asynchronous APIs. It is not intended as a complete reference. For that it's always best to refer to the source code.
This document describes using `libvalkey` in standalone (non-cluster) mode, including an overview of the synchronous and asynchronous APIs. It is not intended as a complete reference. For that it's always best to refer to the source code.
## Table of Contents
@@ -31,20 +31,20 @@ The synchronous API has a pretty small surface area, with only a few commands to
### Connecting
There are several convenience functions to connect in various ways (e.g. host and port, Unix socket, etc). See [include/kv/kv.h](../include/kv/kv.h) for more details.
There are several convenience functions to connect in various ways (e.g. host and port, Unix socket, etc). See [include/valkey/valkey.h](../include/valkey/valkey.h) for more details.
```c
kvContext *kvConnect(const char *host, int port);
kvContext *kvConnectUnix(const char *path);
valkeyContext *valkeyConnect(const char *host, int port);
valkeyContext *valkeyConnectUnix(const char *path);
// There is also a convenience struct to specify various options.
kvContext *kvConnectWithOptions(kvOptions *opt);
valkeyContext *valkeyConnectWithOptions(valkeyOptions *opt);
```
When connecting to a server, libkv will return `NULL` in the event that we can't allocate the context, and set the `err` member if we can connect but there are issues. So when connecting it's simple to handle error states.
When connecting to a server, libvalkey will return `NULL` in the event that we can't allocate the context, and set the `err` member if we can connect but there are issues. So when connecting it's simple to handle error states.
```c
kvContext *ctx = kvConnect("localhost", 6379);
valkeyContext *ctx = valkeyConnect("localhost", 6379);
if (ctx == NULL || ctx->err) {
fprintf(stderr, "Error connecting: %s\n", ctx ? ctx->errstr : "OOM");
}
@@ -52,46 +52,46 @@ if (ctx == NULL || ctx->err) {
### Connection options
There are a variety of options you can specify when connecting to the server, which are delivered via the `kvOptions` helper struct. This includes information to connect to the server as well as other flags.
There are a variety of options you can specify when connecting to the server, which are delivered via the `valkeyOptions` helper struct. This includes information to connect to the server as well as other flags.
```c
kvOptions opt = {0};
valkeyOptions opt = {0};
// You can set primary connection info
if (tcp) {
KV_OPTIONS_SET_TCP(&opt, "localhost", 6379);
VALKEY_OPTIONS_SET_TCP(&opt, "localhost", 6379);
} else {
KV_OPTIONS_SET_UNIX(&opt, "/tmp/kv.sock");
VALKEY_OPTIONS_SET_UNIX(&opt, "/tmp/valkey.sock");
}
// You may attach any arbitrary data to the context
KV_OPTIONS_SET_PRIVDATA(&opt, my_data);
VALKEY_OPTIONS_SET_PRIVDATA(&opt, my_data);
```
There are also several flags you can specify when using the `kvOptions` helper struct.
There are also several flags you can specify when using the `valkeyOptions` helper struct.
| Flag | Description |
| --- | --- |
| `KV_OPT_NONBLOCK` | Tells libkv to make a non-blocking connection. |
| `KV_OPT_REUSEADDR` | Tells libkv to set the [SO_REUSEADDR](https://man7.org/linux/man-pages/man7/socket.7.html) socket option |
| `KV_OPT_PREFER_IPV4`<br>`KV_OPT_PREFER_IPV6`<br>`KV_OPT_PREFER_IP_UNSPEC` | Informs libkv to either prefer IPv4 or IPv6 when invoking [getaddrinfo](https://man7.org/linux/man-pages/man3/gai_strerror.3.html). `KV_OPT_PREFER_IP_UNSPEC` will cause libkv to specify `AF_UNSPEC` in the getaddrinfo call, which means both IPv4 and IPv6 addresses will be searched simultaneously.<br>Libkv prefers IPv4 by default. |
| `KV_OPT_NO_PUSH_AUTOFREE` | Tells libkv to not install the default RESP3 PUSH handler (which just intercepts and frees the replies). This is useful in situations where you want to process these messages in-band. |
| `KV_OPT_NOAUTOFREEREPLIES` | **ASYNC**: tells libkv not to automatically invoke `freeReplyObject` after executing the reply callback. |
| `KV_OPT_NOAUTOFREE` | **ASYNC**: Tells libkv not to automatically free the `kvAsyncContext` on connection/communication failure, but only if the user makes an explicit call to `kvAsyncDisconnect` or `kvAsyncFree` |
| `KV_OPT_MPTCP` | Tells libkv to use multipath TCP (MPTCP). Note that only when both the server and client are using MPTCP do they establish an MPTCP connection between them; otherwise, they use a regular TCP connection instead. |
| `VALKEY_OPT_NONBLOCK` | Tells libvalkey to make a non-blocking connection. |
| `VALKEY_OPT_REUSEADDR` | Tells libvalkey to set the [SO_REUSEADDR](https://man7.org/linux/man-pages/man7/socket.7.html) socket option |
| `VALKEY_OPT_PREFER_IPV4`<br>`VALKEY_OPT_PREFER_IPV6`<br>`VALKEY_OPT_PREFER_IP_UNSPEC` | Informs libvalkey to either prefer IPv4 or IPv6 when invoking [getaddrinfo](https://man7.org/linux/man-pages/man3/gai_strerror.3.html). `VALKEY_OPT_PREFER_IP_UNSPEC` will cause libvalkey to specify `AF_UNSPEC` in the getaddrinfo call, which means both IPv4 and IPv6 addresses will be searched simultaneously.<br>Libvalkey prefers IPv4 by default. |
| `VALKEY_OPT_NO_PUSH_AUTOFREE` | Tells libvalkey to not install the default RESP3 PUSH handler (which just intercepts and frees the replies). This is useful in situations where you want to process these messages in-band. |
| `VALKEY_OPT_NOAUTOFREEREPLIES` | **ASYNC**: tells libvalkey not to automatically invoke `freeReplyObject` after executing the reply callback. |
| `VALKEY_OPT_NOAUTOFREE` | **ASYNC**: Tells libvalkey not to automatically free the `valkeyAsyncContext` on connection/communication failure, but only if the user makes an explicit call to `valkeyAsyncDisconnect` or `valkeyAsyncFree` |
| `VALKEY_OPT_MPTCP` | Tells libvalkey to use multipath TCP (MPTCP). Note that only when both the server and client are using MPTCP do they establish an MPTCP connection between them; otherwise, they use a regular TCP connection instead. |
### Executing commands
The primary command interface is a `printf`-like function that takes a format string along with a variable number of arguments. This will construct a `RESP` command and deliver it to the server.
```c
kvReply *reply = kvCommand(ctx, "INCRBY %s %d", "counter", 42);
valkeyReply *reply = valkeyCommand(ctx, "INCRBY %s %d", "counter", 42);
if (reply == NULL) {
fprintf(stderr, "Communication error: %s\n", c->err ? c->errstr : "Unknown error");
} else if (reply->type == KV_REPLY_ERROR) {
} else if (reply->type == VALKEY_REPLY_ERROR) {
fprintf(stderr, "Error response from server: %s\n", reply->str);
} else if (reply->type != KV_REPLY_INTEGER) {
} else if (reply->type != VALKEY_REPLY_INTEGER) {
// Very unlikely but should be checked.
fprintf(stderr, "Error: Non-integer reply to INCRBY?\n");
}
@@ -104,78 +104,78 @@ If you need to deliver binary safe strings to the server, you can use the `%b` f
```c
struct binary { int x; int y; } = {0xdeadbeef, 0xcafebabe};
kvReply *reply = kvCommand(ctx, "SET %s %b", "some-key", &binary, sizeof(binary));
valkeyReply *reply = valkeyCommand(ctx, "SET %s %b", "some-key", &binary, sizeof(binary));
```
Commands may also be constructed by sending an array of arguments along with an optional array of their lengths. If lengths are not provided, libkv will execute `strlen` on each argument.
Commands may also be constructed by sending an array of arguments along with an optional array of their lengths. If lengths are not provided, libvalkey will execute `strlen` on each argument.
```c
const char *argv[] = {"SET", "captain", "James Kirk"};
sonst size_t argvlens[] = {3, 7, 10};
kvReply *reply = kvCommandArgv(ctx, 3, argv, argvlens);
// Handle error conditions similarly to `kvCommand`
valkeyReply *reply = valkeyCommandArgv(ctx, 3, argv, argvlens);
// Handle error conditions similarly to `valkeyCommand`
```
### Using replies
The `kvCommand` and `kvCommandArgv` functions return a `kvReply` on success and `NULL` in the event of a severe error (e.g. a communication failure with the server, out of memory condition, etc).
The `valkeyCommand` and `valkeyCommandArgv` functions return a `valkeyReply` on success and `NULL` in the event of a severe error (e.g. a communication failure with the server, out of memory condition, etc).
If the reply is `NULL` you can inspect the nature of the error by querying `kvContext->err` for the error code and `kvContext->errstr` for a human readable error string.
If the reply is `NULL` you can inspect the nature of the error by querying `valkeyContext->err` for the error code and `valkeyContext->errstr` for a human readable error string.
When a `kvReply` is returned, you should test the `kvReply->type` field to determine which kind of reply was received from the server. If for example there was an error in the command, this reply can be `KV_REPLY_ERROR` and the specific error string will be in the `reply->str` member.
When a `valkeyReply` is returned, you should test the `valkeyReply->type` field to determine which kind of reply was received from the server. If for example there was an error in the command, this reply can be `VALKEY_REPLY_ERROR` and the specific error string will be in the `reply->str` member.
### Reply types
- `KV_REPLY_ERROR` - An error reply. The error string is in `reply->str`.
- `KV_REPLY_STATUS` - A status reply which will be in `reply->str`.
- `KV_REPLY_INTEGER` - An integer reply, which will be in `reply->integer`.
- `KV_REPLY_DOUBLE` - A double reply which will be in `reply->dval` as well as `reply->str`.
- `KV_REPLY_NIL` - a nil reply.
- `KV_REPLY_BOOL` - A boolean reply which will be in `reply->integer`.
- `KV_REPLY_BIGNUM` - As of yet unused, but the string would be in `reply->str`.
- `KV_REPLY_STRING` - A string reply which will be in `reply->str`.
- `KV_REPLY_VERB` - A verbatim string reply which will be in `reply->str` and who's type will be in `reply->vtype`.
- `KV_REPLY_ARRAY` - An array reply where each element is in `reply->element` with the number of elements in `reply->elements`.
- `KV_REPLY_MAP` - A map reply, which structurally looks just like `KV_REPLY_ARRAY` only is meant to represent keys and values. As with an array reply you can access the elements with `reply->element` and `reply->elements`.
- `KV_REPLY_SET` - Another array-like reply representing a set (e.g. a reply from `SMEMBERS`). Access via `reply->element` and `reply->elements`.
- `KV_REPLY_ATTR` - An attribute reply. As of yet unused by kv-server.
- `KV_REPLY_PUSH` - An out of band push reply. This is also array-like in nature.
- `VALKEY_REPLY_ERROR` - An error reply. The error string is in `reply->str`.
- `VALKEY_REPLY_STATUS` - A status reply which will be in `reply->str`.
- `VALKEY_REPLY_INTEGER` - An integer reply, which will be in `reply->integer`.
- `VALKEY_REPLY_DOUBLE` - A double reply which will be in `reply->dval` as well as `reply->str`.
- `VALKEY_REPLY_NIL` - a nil reply.
- `VALKEY_REPLY_BOOL` - A boolean reply which will be in `reply->integer`.
- `VALKEY_REPLY_BIGNUM` - As of yet unused, but the string would be in `reply->str`.
- `VALKEY_REPLY_STRING` - A string reply which will be in `reply->str`.
- `VALKEY_REPLY_VERB` - A verbatim string reply which will be in `reply->str` and who's type will be in `reply->vtype`.
- `VALKEY_REPLY_ARRAY` - An array reply where each element is in `reply->element` with the number of elements in `reply->elements`.
- `VALKEY_REPLY_MAP` - A map reply, which structurally looks just like `VALKEY_REPLY_ARRAY` only is meant to represent keys and values. As with an array reply you can access the elements with `reply->element` and `reply->elements`.
- `VALKEY_REPLY_SET` - Another array-like reply representing a set (e.g. a reply from `SMEMBERS`). Access via `reply->element` and `reply->elements`.
- `VALKEY_REPLY_ATTR` - An attribute reply. As of yet unused by valkey-server.
- `VALKEY_REPLY_PUSH` - An out of band push reply. This is also array-like in nature.
### Disconnecting/cleanup
When libkv returns non-null `kvReply` struts you are responsible for freeing them with `freeReplyObject`. In order to disconnect and free the context simply call `kvFree`.
When libvalkey returns non-null `valkeyReply` struts you are responsible for freeing them with `freeReplyObject`. In order to disconnect and free the context simply call `valkeyFree`.
```c
kvReply *reply = kvCommand(ctx, "set %s %s", "foo", "bar");
valkeyReply *reply = valkeyCommand(ctx, "set %s %s", "foo", "bar");
// Error handling ...
freeReplyObject(reply);
// Disconnect and free context
kvFree(ctx);
valkeyFree(ctx);
```
### Pipelining
`kvCommand` and `kvCommandArgv` each make a round-trip to the server, by sending the command and then waiting for a reply. Alternatively commands may be pipelined with the `kvAppendCommand` and `kvAppendCommandArgv` functions.
`valkeyCommand` and `valkeyCommandArgv` each make a round-trip to the server, by sending the command and then waiting for a reply. Alternatively commands may be pipelined with the `valkeyAppendCommand` and `valkeyAppendCommandArgv` functions.
When you use `kvAppendCommand` the command is simply appended to the output buffer of `kvContext` but not delivered to the server, until you attempt to read the first response, at which point the entire buffer will be delivered.
When you use `valkeyAppendCommand` the command is simply appended to the output buffer of `valkeyContext` but not delivered to the server, until you attempt to read the first response, at which point the entire buffer will be delivered.
```c
// No data will be delivered to the server while these commands are being appended.
for (size_t i = 0; i < 100000; i++) {
if (kvAppendCommand(c, "INCRBY key:%zu %zu", i, i) != KV_OK) {
if (valkeyAppendCommand(c, "INCRBY key:%zu %zu", i, i) != VALKEY_OK) {
fprintf(stderr, "Error appending command: %s\n", c->errstr);
exit(1);
}
}
// The entire output buffer will be delivered on the first call to `kvGetReply`.
// The entire output buffer will be delivered on the first call to `valkeyGetReply`.
for (size_t i = 0; i < 100000; i++) {
if (kvGetReply(c, (void**)&reply) != KV_OK) {
if (valkeyGetReply(c, (void**)&reply) != VALKEY_OK) {
fprintf(stderr, "Error reading reply %zu: %s\n", i, c->errstr);
exit(1);
} else if (reply->type != KV_REPLY_INTEGER) {
} else if (reply->type != VALKEY_REPLY_INTEGER) {
fprintf(stderr, "Error: Non-integer reply to INCRBY?\n");
exit(1);
}
@@ -185,13 +185,13 @@ for (size_t i = 0; i < 100000; i++) {
}
```
`kvGetReply` can also be used in other contexts than pipeline, for example when you want to continuously block for commands for example in a subscribe context.
`valkeyGetReply` can also be used in other contexts than pipeline, for example when you want to continuously block for commands for example in a subscribe context.
```c
kvReply *reply = kvCommand(c, "SUBSCRIBE channel");
valkeyReply *reply = valkeyCommand(c, "SUBSCRIBE channel");
assert(reply != NULL && !c->err);
while (kvGetReply(c, (void**)&reply) == KV_OK) {
while (valkeyGetReply(c, (void**)&reply) == VALKEY_OK) {
// Do something with the message...
freeReplyObject(reply);
}
@@ -199,26 +199,26 @@ while (kvGetReply(c, (void**)&reply) == KV_OK) {
### Errors
As previously mentioned, when there is a communication error libkv will return `NULL` and set the `err` and `errstr` members with the nature of the problem. The specific error types are as follows.
As previously mentioned, when there is a communication error libvalkey will return `NULL` and set the `err` and `errstr` members with the nature of the problem. The specific error types are as follows.
- `KV_ERR_IO` - A problem with the connection.
- `KV_ERR_EOF` - The server closed the connection.
- `KV_ERR_PROTOCOL` - There was an error parsing the reply.
- `KV_ERR_TIMEOUT` - A connect, read, or write timeout.
- `KV_ERR_OOM` - Out of memory.
- `KV_ERR_OTHER` - Some other error (check `c->errstr` for details).
- `VALKEY_ERR_IO` - A problem with the connection.
- `VALKEY_ERR_EOF` - The server closed the connection.
- `VALKEY_ERR_PROTOCOL` - There was an error parsing the reply.
- `VALKEY_ERR_TIMEOUT` - A connect, read, or write timeout.
- `VALKEY_ERR_OOM` - Out of memory.
- `VALKEY_ERR_OTHER` - Some other error (check `c->errstr` for details).
### Thread safety
Libkv context structs are **not** thread safe. You should not attempt to share them between threads, unless you really know what you're doing.
Libvalkey context structs are **not** thread safe. You should not attempt to share them between threads, unless you really know what you're doing.
### Reader configuration
Libkv contexts have a few more mechanisms you can customize to your needs.
Libvalkey contexts have a few more mechanisms you can customize to your needs.
#### Maximum input buffer size
Libkv uses a buffer to hold incoming bytes, which is typically restored to the configurable max buffer size (`16KB`) when it is empty. To avoid continually reallocating this buffer you can set the value higher, or to zero which means "no limit".
Libvalkey uses a buffer to hold incoming bytes, which is typically restored to the configurable max buffer size (`16KB`) when it is empty. To avoid continually reallocating this buffer you can set the value higher, or to zero which means "no limit".
```c
context->reader->maxbuf = 0;
@@ -226,7 +226,7 @@ context->reader->maxbuf = 0;
#### Maximum array elements
By default, libkv will refuse to parse array-like replies if they have more than 2^32-1 or 4,294,967,295 elements. This value can be set to any arbitrary 64-bit value or zero which just means "no limit".
By default, libvalkey will refuse to parse array-like replies if they have more than 2^32-1 or 4,294,967,295 elements. This value can be set to any arbitrary 64-bit value or zero which just means "no limit".
```c
context->reader->maxelements = 0;
@@ -234,11 +234,11 @@ context->reader->maxelements = 0;
#### RESP3 Push Replies
The `RESP` protocol introduced out-of-band "push" replies in the third version of the specification. These replies may come at any point in the data stream. By default, libkv will simply process these messages and discard them.
The `RESP` protocol introduced out-of-band "push" replies in the third version of the specification. These replies may come at any point in the data stream. By default, libvalkey will simply process these messages and discard them.
If your application needs to perform specific actions on PUSH messages you can install your own handler which will be called as they are received. It is also possible to set the push handler to NULL, in which case the messages will be delivered "in-band". This can be useful for example in a blocking subscribe loop.
**NOTE**: You may also specify a push handler in the `kvOptions` struct and set it on initialization .
**NOTE**: You may also specify a push handler in the `valkeyOptions` struct and set it on initialization .
#### Synchronous context
@@ -248,29 +248,29 @@ void my_push_handler(void *privdata, void *reply) {
}
// Initialization, etc.
kvSetPushCallback(c, my_push_handler);
valkeySetPushCallback(c, my_push_handler);
```
#### Asynchronous context
```c
void my_async_push_handler(kvAsyncContext *ac, void *reply) {
// As with other async replies, libkv will free it for you, unless you have
// configured the context with `KV_OPT_NOAUTOFREE`.
void my_async_push_handler(valkeyAsyncContext *ac, void *reply) {
// As with other async replies, libvalkey will free it for you, unless you have
// configured the context with `VALKEY_OPT_NOAUTOFREE`.
}
// Initialization, etc
kvAsyncSetPushCallback(ac, my_async_push_handler);
valkeyAsyncSetPushCallback(ac, my_async_push_handler);
```
#### Allocator injection
Internally libkv uses a layer of indirection from the standard allocation functions, by keeping a global structure with function pointers to the allocators we are going to use. By default they are just set to `malloc`, `calloc`, `realloc`, etc.
Internally libvalkey uses a layer of indirection from the standard allocation functions, by keeping a global structure with function pointers to the allocators we are going to use. By default they are just set to `malloc`, `calloc`, `realloc`, etc.
These can be overridden like so
```c
kvAllocFuncs my_allocators = {
valkeyAllocFuncs my_allocators = {
.mallocFn = my_malloc,
.callocFn = my_calloc,
.reallocFn = my_realloc,
@@ -278,30 +278,30 @@ kvAllocFuncs my_allocators = {
.freeFn = my_free,
};
// libkv will return the previously set allocators.
kvAllocFuncs old = kvSetAllocators(&my_allocators);
// libvalkey will return the previously set allocators.
valkeyAllocFuncs old = valkeySetAllocators(&my_allocators);
```
They can also be reset to the glibc or musl defaults
```c
kvResetAllocators();
valkeyResetAllocators();
```
**NOTE**: The `vk_calloc` function handles the case where `nmemb` * `size` would overflow a `size_t` and returns `NULL` in that case.
## Asynchronous API
Libkv also has an asynchronous API which supports a great many different event libraries. See the [examples](../examples) directory for specific information about each individual event library.
Libvalkey also has an asynchronous API which supports a great many different event libraries. See the [examples](../examples) directory for specific information about each individual event library.
### Connecting
Libkv provides an `kvAsyncContext` to manage asynchronous connections which works similarly to the synchronous context.
Libvalkey provides an `valkeyAsyncContext` to manage asynchronous connections which works similarly to the synchronous context.
```c
kvAsyncContext *ac = kvAsyncConnect("localhost", 6379);
valkeyAsyncContext *ac = valkeyAsyncConnect("localhost", 6379);
if (ac == NULL) {
fprintf(stderr, "Error: Out of memory trying to allocate kvAsyncContext\n");
fprintf(stderr, "Error: Out of memory trying to allocate valkeyAsyncContext\n");
exit(1);
} else if (ac->err) {
fprintf(stderr, "Error: %s (%d)\n", ac->errstr, ac->err);
@@ -309,10 +309,10 @@ if (ac == NULL) {
}
// If we're using libev
kvLibevAttach(EV_DEFAULT_ ac);
valkeyLibevAttach(EV_DEFAULT_ ac);
kvSetConnectCallback(ac, my_connect_callback);
kvSetDisconnectCallback(ac, my_disconnect_callback);
valkeySetConnectCallback(ac, my_connect_callback);
valkeySetDisconnectCallback(ac, my_disconnect_callback);
ev_run(EV_DEFAULT_ 0);
```
@@ -332,45 +332,45 @@ struct my_app_data {
size_t get_replies;
};
void my_incrby_callback(kvAsyncContext *ac, void *r, void *privdata) {
void my_incrby_callback(valkeyAsyncContext *ac, void *r, void *privdata) {
struct my_app_data *data = privdata;
kvReply *reply = r;
valkeyReply *reply = r;
assert(reply != NULL && reply->type == KV_REPLY_INTEGER);
assert(reply != NULL && reply->type == VALKEY_REPLY_INTEGER);
printf("Incremented value: %lld\n", reply->integer);
data->incrby_replies++;
}
void my_get_callback(kvAsyncContext *ac, void *r, void *privdata) {
void my_get_callback(valkeyAsyncContext *ac, void *r, void *privdata) {
struct my_app_data *data = privdata;
kvReply *reply = r;
valkeyReply *reply = r;
assert(reply != NULL && reply->type == KV_REPLY_STRING);
assert(reply != NULL && reply->type == VALKEY_REPLY_STRING);
printf("Key value: %s\n", reply->str);
data->get_replies++;
}
int exec_some_commands(struct my_app_data *data) {
kvAsyncCommand(ac, my_incrby_callback, data, "INCRBY mykey %d", 42);
kvAsyncCommand(ac, my_get_callback, data, "GET %s", "mykey");
valkeyAsyncCommand(ac, my_incrby_callback, data, "INCRBY mykey %d", 42);
valkeyAsyncCommand(ac, my_get_callback, data, "GET %s", "mykey");
}
```
### Disconnecting/cleanup
For a graceful disconnect use `kvAsyncDisconnect` which will block new commands from being issued.
For a graceful disconnect use `valkeyAsyncDisconnect` which will block new commands from being issued.
The connection is only terminated when all pending commands have been sent, their respective replies have been read, and their respective callbacks have been executed.
After this, the disconnection callback is called with the status, and the context object is freed.
To terminate the connection forcefully use `kvAsyncFree` which also will block new commands from being issued.
To terminate the connection forcefully use `valkeyAsyncFree` which also will block new commands from being issued.
There will be no more data sent on the socket and all pending callbacks will be called with a `NULL` reply.
After this, the disconnection callback is called with the `KV_OK` status, and the context object is freed.
After this, the disconnection callback is called with the `VALKEY_OK` status, and the context object is freed.
## TLS support
TLS support is not enabled by default and requires an explicit build flag as described in [`README.md`](../README.md#building).
Libkv implements TLS on top of its `kvContext` and `kvAsyncContext`, so you will need to establish a connection first and then initiate a TLS handshake.
Libvalkey implements TLS on top of its `valkeyContext` and `valkeyAsyncContext`, so you will need to establish a connection first and then initiate a TLS handshake.
See the [examples](../examples) directory for how to create the TLS context and initiate the handshake.
+35 -24
View File
@@ -1,32 +1,38 @@
cmake_minimum_required(VERSION 3.7.0)
project(examples LANGUAGES C)
option(ENABLE_THREADS "Enable thread-safe initialization" ON)
if(ENABLE_THREADS)
find_package(Threads REQUIRED)
endif()
if(CMAKE_SOURCE_DIR STREQUAL PROJECT_SOURCE_DIR)
# This CMakeLists.txt is used standalone to build the examples.
# Find required kv package to get include paths.
find_package(kv REQUIRED)
find_package(kv_tls QUIET)
# Find required valkey package to get include paths.
find_package(valkey REQUIRED)
find_package(valkey_tls QUIET)
else()
# This CMakeLists.txt is included, most likely from libkv's project
# CMakeLists.txt. Add path to enable <kv/x.h> includes.
# This CMakeLists.txt is included, most likely from libvalkey's project
# CMakeLists.txt. Add path to enable <valkey/x.h> includes.
include_directories(${CMAKE_SOURCE_DIR}/include)
endif()
add_executable(example-blocking blocking.c)
target_link_libraries(example-blocking kv::kv)
target_link_libraries(example-blocking valkey::valkey)
add_executable(example-blocking-push blocking-push.c)
target_link_libraries(example-blocking-push kv::kv)
target_link_libraries(example-blocking-push valkey::valkey)
add_executable(example-cluster-simple cluster-simple.c)
target_link_libraries(example-cluster-simple kv::kv)
target_link_libraries(example-cluster-simple valkey::valkey)
if(ENABLE_TLS)
add_executable(example-blocking-tls blocking-tls.c)
target_link_libraries(example-blocking-tls kv::kv kv::kv_tls)
target_link_libraries(example-blocking-tls valkey::valkey valkey::valkey_tls)
add_executable(example-cluster-tls cluster-tls.c)
target_link_libraries(example-cluster-tls kv::kv kv::kv_tls)
target_link_libraries(example-cluster-tls valkey::valkey valkey::valkey_tls)
endif()
# Examples using GLib
@@ -35,7 +41,7 @@ if(PkgConfig_FOUND)
pkg_check_modules(GLIB2 IMPORTED_TARGET glib-2.0)
if(GLIB2_FOUND)
add_executable(example-async-glib async-glib.c)
target_link_libraries(example-async-glib kv::kv PkgConfig::GLIB2)
target_link_libraries(example-async-glib valkey::valkey PkgConfig::GLIB2)
endif()
endif()
@@ -43,27 +49,30 @@ endif()
find_path(LIBEV ev.h)
if(LIBEV)
add_executable(example-async-libev async-libev.c)
target_link_libraries(example-async-libev kv::kv ev)
target_link_libraries(example-async-libev valkey::valkey ev)
endif()
# Examples using libevent
find_path(LIBEVENT event.h)
if(LIBEVENT)
find_path(LIBEVENT event2/event.h)
find_library(LIBEVENT_LIBRARY event)
if(LIBEVENT AND LIBEVENT_LIBRARY)
include_directories(${LIBEVENT})
add_executable(example-async-libevent async-libevent.c)
target_link_libraries(example-async-libevent kv::kv event)
target_link_libraries(example-async-libevent valkey::valkey ${LIBEVENT_LIBRARY})
add_executable(example-cluster-async cluster-async.c)
target_link_libraries(example-cluster-async kv::kv event)
target_link_libraries(example-cluster-async valkey::valkey ${LIBEVENT_LIBRARY})
add_executable(example-cluster-clientside-caching-async cluster-clientside-caching-async.c)
target_link_libraries(example-cluster-clientside-caching-async kv::kv event)
target_link_libraries(example-cluster-clientside-caching-async valkey::valkey ${LIBEVENT_LIBRARY})
if(ENABLE_TLS)
add_executable(example-async-libevent-tls async-libevent-tls.c)
target_link_libraries(example-async-libevent-tls kv::kv kv::kv_tls event)
target_link_libraries(example-async-libevent-tls valkey::valkey valkey::valkey_tls ${LIBEVENT_LIBRARY})
add_executable(example-cluster-async-tls cluster-async-tls.c)
target_link_libraries(example-cluster-async-tls kv::kv kv::kv_tls event)
target_link_libraries(example-cluster-async-tls valkey::valkey valkey::valkey_tls ${LIBEVENT_LIBRARY})
endif()
endif()
@@ -71,26 +80,28 @@ endif()
find_path(LIBHV hv/hv.h)
if(LIBHV)
add_executable(example-async-libhv async-libhv.c)
target_link_libraries(example-async-libhv kv::kv hv)
target_link_libraries(example-async-libhv valkey::valkey hv)
endif()
# Examples using libuv
find_path(LIBUV uv.h)
if(LIBUV)
find_library(LIBUV_LIBRARY uv)
if(LIBUV AND LIBUV_LIBRARY)
include_directories(${LIBUV})
add_executable(example-async-libuv async-libuv.c)
target_link_libraries(example-async-libuv kv::kv uv)
target_link_libraries(example-async-libuv valkey::valkey ${LIBUV_LIBRARY})
endif()
# Examples using libsystemd
find_path(LIBSDEVENT systemd/sd-event.h)
if(LIBSDEVENT)
add_executable(example-async-libsdevent async-libsdevent.c)
target_link_libraries(example-async-libsdevent kv::kv systemd)
target_link_libraries(example-async-libsdevent valkey::valkey systemd)
endif()
# Examples using the RunLoop in Apple's CoreFoundation
if(APPLE)
find_library(CF CoreFoundation)
add_executable(example-async-macosx async-macosx.c)
target_link_libraries(example-async-macosx kv::kv ${CF})
target_link_libraries(example-async-macosx valkey::valkey ${CF})
endif()
+4 -4
View File
@@ -1,8 +1,8 @@
CC?=gcc
CXX?=g++
STLIBNAME?=../lib/libkv.a
TLS_STLIBNAME?=../lib/libkv_tls.a
STLIBNAME?=../lib/libvalkey.a
TLS_STLIBNAME?=../lib/libvalkey_tls.a
INCLUDE_DIR?=../include
WARNINGS=-Wall -Wextra
@@ -73,7 +73,7 @@ example-async-poll: async-poll.c $(STLIBNAME)
ifndef AE_DIR
example-async-ae:
@echo "Please specify AE_DIR (e.g. <kv repository>/src)"
@echo "Please specify AE_DIR (e.g. <valkey repository>/src)"
@false
else
example-async-ae: async-ae.c $(STLIBNAME)
@@ -98,7 +98,7 @@ example-async-qt:
@false
else
example-async-qt: async-qt.cpp $(STLIBNAME)
$(QT_MOC) ../include/kv/adapters/qt.h | \
$(QT_MOC) ../include/valkey/adapters/qt.h | \
$(CXX) -x c++ -o qt-adapter-moc.o -c - $(CFLAGS) -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore
$(QT_MOC) async-qt.h | \
$(CXX) -x c++ -o qt-example-moc.o -c - $(CFLAGS) -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore
+16 -16
View File
@@ -1,7 +1,7 @@
#include <kv/async.h>
#include <kv/kv.h>
#include <valkey/async.h>
#include <valkey/valkey.h>
#include <kv/adapters/ae.h>
#include <valkey/adapters/ae.h>
#include <signal.h>
#include <stdio.h>
@@ -11,18 +11,18 @@
/* Put event loop in the global scope, so it can be explicitly stopped */
static aeEventLoop *loop;
void getCallback(kvAsyncContext *c, void *r, void *privdata) {
kvReply *reply = r;
void getCallback(valkeyAsyncContext *c, void *r, void *privdata) {
valkeyReply *reply = r;
if (reply == NULL)
return;
printf("argv[%s]: %s\n", (char *)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
kvAsyncDisconnect(c);
valkeyAsyncDisconnect(c);
}
void connectCallback(kvAsyncContext *c, int status) {
if (status != KV_OK) {
void connectCallback(valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
aeStop(loop);
return;
@@ -31,8 +31,8 @@ void connectCallback(kvAsyncContext *c, int status) {
printf("Connected...\n");
}
void disconnectCallback(const kvAsyncContext *c, int status) {
if (status != KV_OK) {
void disconnectCallback(const valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
aeStop(loop);
return;
@@ -45,7 +45,7 @@ void disconnectCallback(const kvAsyncContext *c, int status) {
int main(int argc, char **argv) {
signal(SIGPIPE, SIG_IGN);
kvAsyncContext *c = kvAsyncConnect("127.0.0.1", 6379);
valkeyAsyncContext *c = valkeyAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
@@ -53,12 +53,12 @@ int main(int argc, char **argv) {
}
loop = aeCreateEventLoop(64);
kvAeAttach(loop, c);
kvAsyncSetConnectCallback(c, connectCallback);
kvAsyncSetDisconnectCallback(c, disconnectCallback);
kvAsyncCommand(
valkeyAeAttach(loop, c);
valkeyAsyncSetConnectCallback(c, connectCallback);
valkeyAsyncSetDisconnectCallback(c, disconnectCallback);
valkeyAsyncCommand(
c, NULL, NULL, "SET key %b", argv[argc - 1], strlen(argv[argc - 1]));
kvAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
valkeyAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
aeMain(loop);
return 0;
}
+17 -17
View File
@@ -1,16 +1,16 @@
#include <kv/async.h>
#include <kv/kv.h>
#include <valkey/async.h>
#include <valkey/valkey.h>
#include <kv/adapters/glib.h>
#include <valkey/adapters/glib.h>
#include <stdlib.h>
static GMainLoop *mainloop;
static void
connect_cb(kvAsyncContext *ac G_GNUC_UNUSED,
connect_cb(valkeyAsyncContext *ac G_GNUC_UNUSED,
int status) {
if (status != KV_OK) {
if (status != VALKEY_OK) {
g_printerr("Failed to connect: %s\n", ac->errstr);
g_main_loop_quit(mainloop);
} else {
@@ -19,9 +19,9 @@ connect_cb(kvAsyncContext *ac G_GNUC_UNUSED,
}
static void
disconnect_cb(const kvAsyncContext *ac G_GNUC_UNUSED,
disconnect_cb(const valkeyAsyncContext *ac G_GNUC_UNUSED,
int status) {
if (status != KV_OK) {
if (status != VALKEY_OK) {
g_error("Failed to disconnect: %s", ac->errstr);
} else {
g_printerr("Disconnected...\n");
@@ -30,37 +30,37 @@ disconnect_cb(const kvAsyncContext *ac G_GNUC_UNUSED,
}
static void
command_cb(kvAsyncContext *ac,
command_cb(valkeyAsyncContext *ac,
gpointer r,
gpointer user_data G_GNUC_UNUSED) {
kvReply *reply = r;
valkeyReply *reply = r;
if (reply) {
g_print("REPLY: %s\n", reply->str);
}
kvAsyncDisconnect(ac);
valkeyAsyncDisconnect(ac);
}
gint main(gint argc G_GNUC_UNUSED, gchar *argv[] G_GNUC_UNUSED) {
kvAsyncContext *ac;
valkeyAsyncContext *ac;
GMainContext *context = NULL;
GSource *source;
ac = kvAsyncConnect("127.0.0.1", 6379);
ac = valkeyAsyncConnect("127.0.0.1", 6379);
if (ac->err) {
g_printerr("%s\n", ac->errstr);
exit(EXIT_FAILURE);
}
source = kv_source_new(ac);
source = valkey_source_new(ac);
mainloop = g_main_loop_new(context, FALSE);
g_source_attach(source, context);
kvAsyncSetConnectCallback(ac, connect_cb);
kvAsyncSetDisconnectCallback(ac, disconnect_cb);
kvAsyncCommand(ac, command_cb, NULL, "SET key 1234");
kvAsyncCommand(ac, command_cb, NULL, "GET key");
valkeyAsyncSetConnectCallback(ac, connect_cb);
valkeyAsyncSetDisconnectCallback(ac, disconnect_cb);
valkeyAsyncCommand(ac, command_cb, NULL, "SET key 1234");
valkeyAsyncCommand(ac, command_cb, NULL, "GET key");
g_main_loop_run(mainloop);
+16 -16
View File
@@ -1,33 +1,33 @@
#include <kv/async.h>
#include <kv/kv.h>
#include <valkey/async.h>
#include <valkey/valkey.h>
#include <kv/adapters/ivykis.h>
#include <valkey/adapters/ivykis.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
void getCallback(kvAsyncContext *c, void *r, void *privdata) {
kvReply *reply = r;
void getCallback(valkeyAsyncContext *c, void *r, void *privdata) {
valkeyReply *reply = r;
if (reply == NULL)
return;
printf("argv[%s]: %s\n", (char *)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
kvAsyncDisconnect(c);
valkeyAsyncDisconnect(c);
}
void connectCallback(kvAsyncContext *c, int status) {
if (status != KV_OK) {
void connectCallback(valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const kvAsyncContext *c, int status) {
if (status != KV_OK) {
void disconnectCallback(const valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
@@ -41,19 +41,19 @@ int main(int argc, char **argv) {
iv_init();
kvAsyncContext *c = kvAsyncConnect("127.0.0.1", 6379);
valkeyAsyncContext *c = valkeyAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
kvIvykisAttach(c);
kvAsyncSetConnectCallback(c, connectCallback);
kvAsyncSetDisconnectCallback(c, disconnectCallback);
kvAsyncCommand(
valkeyIvykisAttach(c);
valkeyAsyncSetConnectCallback(c, connectCallback);
valkeyAsyncSetDisconnectCallback(c, disconnectCallback);
valkeyAsyncCommand(
c, NULL, NULL, "SET key %b", argv[argc - 1], strlen(argv[argc - 1]));
kvAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
valkeyAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
iv_main();
+16 -16
View File
@@ -1,33 +1,33 @@
#include <kv/async.h>
#include <kv/kv.h>
#include <valkey/async.h>
#include <valkey/valkey.h>
#include <kv/adapters/libev.h>
#include <valkey/adapters/libev.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
void getCallback(kvAsyncContext *c, void *r, void *privdata) {
kvReply *reply = r;
void getCallback(valkeyAsyncContext *c, void *r, void *privdata) {
valkeyReply *reply = r;
if (reply == NULL)
return;
printf("argv[%s]: %s\n", (char *)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
kvAsyncDisconnect(c);
valkeyAsyncDisconnect(c);
}
void connectCallback(kvAsyncContext *c, int status) {
if (status != KV_OK) {
void connectCallback(valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const kvAsyncContext *c, int status) {
if (status != KV_OK) {
void disconnectCallback(const valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
@@ -39,19 +39,19 @@ int main(int argc, char **argv) {
signal(SIGPIPE, SIG_IGN);
#endif
kvAsyncContext *c = kvAsyncConnect("127.0.0.1", 6379);
valkeyAsyncContext *c = valkeyAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
kvLibevAttach(EV_DEFAULT_ c);
kvAsyncSetConnectCallback(c, connectCallback);
kvAsyncSetDisconnectCallback(c, disconnectCallback);
kvAsyncCommand(
valkeyLibevAttach(EV_DEFAULT_ c);
valkeyAsyncSetConnectCallback(c, connectCallback);
valkeyAsyncSetDisconnectCallback(c, disconnectCallback);
valkeyAsyncCommand(
c, NULL, NULL, "SET key %b", argv[argc - 1], strlen(argv[argc - 1]));
kvAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
valkeyAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
ev_loop(EV_DEFAULT_ 0);
return 0;
}
+24 -24
View File
@@ -1,34 +1,34 @@
#include <kv/async.h>
#include <kv/tls.h>
#include <kv/kv.h>
#include <valkey/async.h>
#include <valkey/tls.h>
#include <valkey/valkey.h>
#include <kv/adapters/libevent.h>
#include <valkey/adapters/libevent.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
void getCallback(kvAsyncContext *c, void *r, void *privdata) {
kvReply *reply = r;
void getCallback(valkeyAsyncContext *c, void *r, void *privdata) {
valkeyReply *reply = r;
if (reply == NULL)
return;
printf("argv[%s]: %s\n", (char *)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
kvAsyncDisconnect(c);
valkeyAsyncDisconnect(c);
}
void connectCallback(kvAsyncContext *c, int status) {
if (status != KV_OK) {
void connectCallback(valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const kvAsyncContext *c, int status) {
if (status != KV_OK) {
void disconnectCallback(const valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
@@ -57,36 +57,36 @@ int main(int argc, char **argv) {
const char *certKey = argv[5];
const char *caCert = argc > 5 ? argv[6] : NULL;
kvTLSContext *tls;
kvTLSContextError tls_error = KV_TLS_CTX_NONE;
valkeyTLSContext *tls;
valkeyTLSContextError tls_error = VALKEY_TLS_CTX_NONE;
kvInitOpenSSL();
valkeyInitOpenSSL();
tls = kvCreateTLSContext(caCert, NULL,
tls = valkeyCreateTLSContext(caCert, NULL,
cert, certKey, NULL, &tls_error);
if (!tls) {
printf("Error: %s\n", kvTLSContextGetError(tls_error));
printf("Error: %s\n", valkeyTLSContextGetError(tls_error));
return 1;
}
kvAsyncContext *c = kvAsyncConnect(hostname, port);
valkeyAsyncContext *c = valkeyAsyncConnect(hostname, port);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
if (kvInitiateTLSWithContext(&c->c, tls) != KV_OK) {
if (valkeyInitiateTLSWithContext(&c->c, tls) != VALKEY_OK) {
printf("TLS Error!\n");
exit(1);
}
kvLibeventAttach(c, base);
kvAsyncSetConnectCallback(c, connectCallback);
kvAsyncSetDisconnectCallback(c, disconnectCallback);
kvAsyncCommand(c, NULL, NULL, "SET key %b", value, nvalue);
kvAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
valkeyLibeventAttach(c, base);
valkeyAsyncSetConnectCallback(c, connectCallback);
valkeyAsyncSetDisconnectCallback(c, disconnectCallback);
valkeyAsyncCommand(c, NULL, NULL, "SET key %b", value, nvalue);
valkeyAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
event_base_dispatch(base);
kvFreeTLSContext(tls);
valkeyFreeTLSContext(tls);
return 0;
}
+18 -18
View File
@@ -1,15 +1,15 @@
#include <kv/async.h>
#include <kv/kv.h>
#include <valkey/async.h>
#include <valkey/valkey.h>
#include <kv/adapters/libevent.h>
#include <valkey/adapters/libevent.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
void getCallback(kvAsyncContext *c, void *r, void *privdata) {
kvReply *reply = r;
void getCallback(valkeyAsyncContext *c, void *r, void *privdata) {
valkeyReply *reply = r;
if (reply == NULL) {
if (c->errstr) {
printf("errstr: %s\n", c->errstr);
@@ -19,19 +19,19 @@ void getCallback(kvAsyncContext *c, void *r, void *privdata) {
printf("argv[%s]: %s\n", (char *)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
kvAsyncDisconnect(c);
valkeyAsyncDisconnect(c);
}
void connectCallback(kvAsyncContext *c, int status) {
if (status != KV_OK) {
void connectCallback(valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const kvAsyncContext *c, int status) {
if (status != KV_OK) {
void disconnectCallback(const valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
@@ -44,25 +44,25 @@ int main(int argc, char **argv) {
#endif
struct event_base *base = event_base_new();
kvOptions options = {0};
KV_OPTIONS_SET_TCP(&options, "127.0.0.1", 6379);
valkeyOptions options = {0};
VALKEY_OPTIONS_SET_TCP(&options, "127.0.0.1", 6379);
struct timeval tv = {0};
tv.tv_sec = 1;
options.connect_timeout = &tv;
kvAsyncContext *c = kvAsyncConnectWithOptions(&options);
valkeyAsyncContext *c = valkeyAsyncConnectWithOptions(&options);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
kvLibeventAttach(c, base);
kvAsyncSetConnectCallback(c, connectCallback);
kvAsyncSetDisconnectCallback(c, disconnectCallback);
kvAsyncCommand(
valkeyLibeventAttach(c, base);
valkeyAsyncSetConnectCallback(c, connectCallback);
valkeyAsyncSetDisconnectCallback(c, disconnectCallback);
valkeyAsyncCommand(
c, NULL, NULL, "SET key %b", argv[argc - 1], strlen(argv[argc - 1]));
kvAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
valkeyAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
event_base_dispatch(base);
return 0;
}
+21 -21
View File
@@ -1,45 +1,45 @@
#include <kv/async.h>
#include <kv/kv.h>
#include <valkey/async.h>
#include <valkey/valkey.h>
#include <kv/adapters/libhv.h>
#include <valkey/adapters/libhv.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
void getCallback(kvAsyncContext *c, void *r, void *privdata) {
kvReply *reply = r;
void getCallback(valkeyAsyncContext *c, void *r, void *privdata) {
valkeyReply *reply = r;
if (reply == NULL)
return;
printf("argv[%s]: %s\n", (char *)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
kvAsyncDisconnect(c);
valkeyAsyncDisconnect(c);
}
void debugCallback(kvAsyncContext *c, void *r, void *privdata) {
void debugCallback(valkeyAsyncContext *c, void *r, void *privdata) {
(void)privdata;
kvReply *reply = r;
valkeyReply *reply = r;
if (reply == NULL) {
printf("`DEBUG SLEEP` error: %s\n", c->errstr ? c->errstr : "unknown error");
return;
}
kvAsyncDisconnect(c);
valkeyAsyncDisconnect(c);
}
void connectCallback(kvAsyncContext *c, int status) {
if (status != KV_OK) {
void connectCallback(valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const kvAsyncContext *c, int status) {
if (status != KV_OK) {
void disconnectCallback(const valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
@@ -51,7 +51,7 @@ int main(int argc, char **argv) {
signal(SIGPIPE, SIG_IGN);
#endif
kvAsyncContext *c = kvAsyncConnect("127.0.0.1", 6379);
valkeyAsyncContext *c = valkeyAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
@@ -59,14 +59,14 @@ int main(int argc, char **argv) {
}
hloop_t *loop = hloop_new(HLOOP_FLAG_QUIT_WHEN_NO_ACTIVE_EVENTS);
kvLibhvAttach(c, loop);
kvAsyncSetTimeout(c, (struct timeval){.tv_sec = 0, .tv_usec = 500000});
kvAsyncSetConnectCallback(c, connectCallback);
kvAsyncSetDisconnectCallback(c, disconnectCallback);
kvAsyncCommand(
valkeyLibhvAttach(c, loop);
valkeyAsyncSetTimeout(c, (struct timeval){.tv_sec = 0, .tv_usec = 500000});
valkeyAsyncSetConnectCallback(c, connectCallback);
valkeyAsyncSetDisconnectCallback(c, disconnectCallback);
valkeyAsyncCommand(
c, NULL, NULL, "SET key %b", argv[argc - 1], strlen(argv[argc - 1]));
kvAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
kvAsyncCommand(c, debugCallback, NULL, "DEBUG SLEEP %d", 1);
valkeyAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
valkeyAsyncCommand(c, debugCallback, NULL, "DEBUG SLEEP %d", 1);
hloop_run(loop);
hloop_free(&loop);
return 0;
+21 -21
View File
@@ -1,28 +1,28 @@
#define _XOPEN_SOURCE 600 /* Required by libsdevent (CLOCK_MONOTONIC) */
#include <kv/async.h>
#include <kv/kv.h>
#include <valkey/async.h>
#include <valkey/valkey.h>
#include <kv/adapters/libsdevent.h>
#include <valkey/adapters/libsdevent.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
void debugCallback(kvAsyncContext *c, void *r, void *privdata) {
void debugCallback(valkeyAsyncContext *c, void *r, void *privdata) {
(void)privdata;
kvReply *reply = r;
valkeyReply *reply = r;
if (reply == NULL) {
/* The DEBUG SLEEP command will almost always fail, because we have set a 1 second timeout */
printf("`DEBUG SLEEP` error: %s\n", c->errstr ? c->errstr : "unknown error");
return;
}
/* Disconnect after receiving the reply of DEBUG SLEEP (which will not)*/
kvAsyncDisconnect(c);
valkeyAsyncDisconnect(c);
}
void getCallback(kvAsyncContext *c, void *r, void *privdata) {
kvReply *reply = r;
void getCallback(valkeyAsyncContext *c, void *r, void *privdata) {
valkeyReply *reply = r;
if (reply == NULL) {
printf("`GET key` error: %s\n", c->errstr ? c->errstr : "unknown error");
return;
@@ -30,19 +30,19 @@ void getCallback(kvAsyncContext *c, void *r, void *privdata) {
printf("`GET key` result: argv[%s]: %s\n", (char *)privdata, reply->str);
/* start another request that demonstrate timeout */
kvAsyncCommand(c, debugCallback, NULL, "DEBUG SLEEP %f", 1.5);
valkeyAsyncCommand(c, debugCallback, NULL, "DEBUG SLEEP %f", 1.5);
}
void connectCallback(kvAsyncContext *c, int status) {
if (status != KV_OK) {
void connectCallback(valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("connect error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const kvAsyncContext *c, int status) {
if (status != KV_OK) {
void disconnectCallback(const valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("disconnect because of error: %s\n", c->errstr);
return;
}
@@ -55,17 +55,17 @@ int main(int argc, char **argv) {
struct sd_event *event;
sd_event_default(&event);
kvAsyncContext *c = kvAsyncConnect("127.0.0.1", 6379);
valkeyAsyncContext *c = valkeyAsyncConnect("127.0.0.1", 6379);
if (c->err) {
printf("Error: %s\n", c->errstr);
kvAsyncFree(c);
valkeyAsyncFree(c);
return 1;
}
kvLibsdeventAttach(c, event);
kvAsyncSetConnectCallback(c, connectCallback);
kvAsyncSetDisconnectCallback(c, disconnectCallback);
kvAsyncSetTimeout(c, (struct timeval){.tv_sec = 1, .tv_usec = 0});
valkeyLibsdeventAttach(c, event);
valkeyAsyncSetConnectCallback(c, connectCallback);
valkeyAsyncSetDisconnectCallback(c, disconnectCallback);
valkeyAsyncSetTimeout(c, (struct timeval){.tv_sec = 1, .tv_usec = 0});
/*
In this demo, we first `set key`, then `get key` to demonstrate the basic usage of libsdevent adapter.
@@ -74,9 +74,9 @@ int main(int argc, char **argv) {
timeout error, which is shown in the `debugCallback`.
*/
kvAsyncCommand(
valkeyAsyncCommand(
c, NULL, NULL, "SET key %b", argv[argc - 1], strlen(argv[argc - 1]));
kvAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
valkeyAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
/* sd-event does not quit when there are no handlers registered. Manually exit after 1.5 seconds */
sd_event_source *s;
+20 -20
View File
@@ -1,28 +1,28 @@
#define _XOPEN_SOURCE 600 /* Required by libuv (pthread_rwlock_t) */
#include <kv/async.h>
#include <kv/kv.h>
#include <valkey/async.h>
#include <valkey/valkey.h>
#include <kv/adapters/libuv.h>
#include <valkey/adapters/libuv.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
void debugCallback(kvAsyncContext *c, void *r, void *privdata) {
void debugCallback(valkeyAsyncContext *c, void *r, void *privdata) {
(void)privdata; //unused
kvReply *reply = r;
valkeyReply *reply = r;
if (reply == NULL) {
/* The DEBUG SLEEP command will almost always fail, because we have set a 1 second timeout */
printf("`DEBUG SLEEP` error: %s\n", c->errstr ? c->errstr : "unknown error");
return;
}
/* Disconnect after receiving the reply of DEBUG SLEEP (which will not)*/
kvAsyncDisconnect(c);
valkeyAsyncDisconnect(c);
}
void getCallback(kvAsyncContext *c, void *r, void *privdata) {
kvReply *reply = r;
void getCallback(valkeyAsyncContext *c, void *r, void *privdata) {
valkeyReply *reply = r;
if (reply == NULL) {
printf("`GET key` error: %s\n", c->errstr ? c->errstr : "unknown error");
return;
@@ -30,19 +30,19 @@ void getCallback(kvAsyncContext *c, void *r, void *privdata) {
printf("`GET key` result: argv[%s]: %s\n", (char *)privdata, reply->str);
/* start another request that demonstrate timeout */
kvAsyncCommand(c, debugCallback, NULL, "DEBUG SLEEP %f", 1.5);
valkeyAsyncCommand(c, debugCallback, NULL, "DEBUG SLEEP %f", 1.5);
}
void connectCallback(kvAsyncContext *c, int status) {
if (status != KV_OK) {
void connectCallback(valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("connect error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const kvAsyncContext *c, int status) {
if (status != KV_OK) {
void disconnectCallback(const valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("disconnect because of error: %s\n", c->errstr);
return;
}
@@ -56,17 +56,17 @@ int main(int argc, char **argv) {
uv_loop_t *loop = uv_default_loop();
kvAsyncContext *c = kvAsyncConnect("127.0.0.1", 6379);
valkeyAsyncContext *c = valkeyAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
kvLibuvAttach(c, loop);
kvAsyncSetConnectCallback(c, connectCallback);
kvAsyncSetDisconnectCallback(c, disconnectCallback);
kvAsyncSetTimeout(c, (struct timeval){.tv_sec = 1, .tv_usec = 0});
valkeyLibuvAttach(c, loop);
valkeyAsyncSetConnectCallback(c, connectCallback);
valkeyAsyncSetDisconnectCallback(c, disconnectCallback);
valkeyAsyncSetTimeout(c, (struct timeval){.tv_sec = 1, .tv_usec = 0});
/*
In this demo, we first `set key`, then `get key` to demonstrate the basic usage of libuv adapter.
@@ -75,9 +75,9 @@ int main(int argc, char **argv) {
timeout error, which is shown in the `debugCallback`.
*/
kvAsyncCommand(
valkeyAsyncCommand(
c, NULL, NULL, "SET key %b", argv[argc - 1], strlen(argv[argc - 1]));
kvAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
valkeyAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
uv_run(loop, UV_RUN_DEFAULT);
return 0;
+16 -16
View File
@@ -31,33 +31,33 @@
* POSSIBILITY OF SUCH DAMAGE.
*/
#include <kv/async.h>
#include <kv/kv.h>
#include <valkey/async.h>
#include <valkey/valkey.h>
#include <kv/adapters/macosx.h>
#include <valkey/adapters/macosx.h>
#include <stdio.h>
void getCallback(kvAsyncContext *c, void *r, void *privdata) {
kvReply *reply = r;
void getCallback(valkeyAsyncContext *c, void *r, void *privdata) {
valkeyReply *reply = r;
if (reply == NULL)
return;
printf("argv[%s]: %s\n", (char *)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
kvAsyncDisconnect(c);
valkeyAsyncDisconnect(c);
}
void connectCallback(kvAsyncContext *c, int status) {
if (status != KV_OK) {
void connectCallback(valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const kvAsyncContext *c, int status) {
if (status != KV_OK) {
void disconnectCallback(const valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
@@ -74,21 +74,21 @@ int main(int argc, char **argv) {
return 1;
}
kvAsyncContext *c = kvAsyncConnect("127.0.0.1", 6379);
valkeyAsyncContext *c = valkeyAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
kvMacOSAttach(c, loop);
valkeyMacOSAttach(c, loop);
kvAsyncSetConnectCallback(c, connectCallback);
kvAsyncSetDisconnectCallback(c, disconnectCallback);
valkeyAsyncSetConnectCallback(c, connectCallback);
valkeyAsyncSetDisconnectCallback(c, disconnectCallback);
kvAsyncCommand(
valkeyAsyncCommand(
c, NULL, NULL, "SET key %b", argv[argc - 1], strlen(argv[argc - 1]));
kvAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
valkeyAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
CFRunLoopRun();
+16 -16
View File
@@ -1,6 +1,6 @@
#include <kv/async.h>
#include <valkey/async.h>
#include <kv/adapters/poll.h>
#include <valkey/adapters/poll.h>
#include <signal.h>
#include <stdio.h>
@@ -11,18 +11,18 @@
/* Put in the global scope, so that loop can be explicitly stopped */
static int exit_loop = 0;
void getCallback(kvAsyncContext *c, void *r, void *privdata) {
kvReply *reply = r;
void getCallback(valkeyAsyncContext *c, void *r, void *privdata) {
valkeyReply *reply = r;
if (reply == NULL)
return;
printf("argv[%s]: %s\n", (char *)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
kvAsyncDisconnect(c);
valkeyAsyncDisconnect(c);
}
void connectCallback(kvAsyncContext *c, int status) {
if (status != KV_OK) {
void connectCallback(valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
exit_loop = 1;
return;
@@ -31,9 +31,9 @@ void connectCallback(kvAsyncContext *c, int status) {
printf("Connected...\n");
}
void disconnectCallback(const kvAsyncContext *c, int status) {
void disconnectCallback(const valkeyAsyncContext *c, int status) {
exit_loop = 1;
if (status != KV_OK) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
@@ -44,21 +44,21 @@ void disconnectCallback(const kvAsyncContext *c, int status) {
int main(int argc, char **argv) {
signal(SIGPIPE, SIG_IGN);
kvAsyncContext *c = kvAsyncConnect("127.0.0.1", 6379);
valkeyAsyncContext *c = valkeyAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
kvPollAttach(c);
kvAsyncSetConnectCallback(c, connectCallback);
kvAsyncSetDisconnectCallback(c, disconnectCallback);
kvAsyncCommand(
valkeyPollAttach(c);
valkeyAsyncSetConnectCallback(c, connectCallback);
valkeyAsyncSetDisconnectCallback(c, disconnectCallback);
valkeyAsyncCommand(
c, NULL, NULL, "SET key %b", argv[argc - 1], strlen(argv[argc - 1]));
kvAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
valkeyAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
while (!exit_loop) {
kvPollTick(c, 0.1);
valkeyPollTick(c, 0.1);
}
return 0;
}
+7 -7
View File
@@ -1,7 +1,7 @@
#ifndef KV_EXAMPLE_QT_H
#define KV_EXAMPLE_QT_H
#ifndef VALKEY_EXAMPLE_QT_H
#define VALKEY_EXAMPLE_QT_H
#include <kv/adapters/qt.h>
#include <valkey/adapters/qt.h>
class ExampleQt : public QObject {
@@ -22,10 +22,10 @@ class ExampleQt : public QObject {
private:
const char *m_value;
kvAsyncContext *m_ctx;
KVQtAdapter m_adapter;
valkeyAsyncContext *m_ctx;
ValkeyQtAdapter m_adapter;
friend void getCallback(kvAsyncContext *, void *, void *);
friend void getCallback(valkeyAsyncContext *, void *, void *);
};
#endif /* KV_EXAMPLE_QT_H */
#endif /* VALKEY_EXAMPLE_QT_H */
+28 -28
View File
@@ -1,25 +1,25 @@
#include <adapters/kvmoduleapi.h>
#include <adapters/valkeymoduleapi.h>
#include <async.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <kv.h>
#include <valkey.h>
void debugCallback(kvAsyncContext *c, void *r, void *privdata) {
void debugCallback(valkeyAsyncContext *c, void *r, void *privdata) {
(void)privdata; //unused
kvReply *reply = r;
valkeyReply *reply = r;
if (reply == NULL) {
/* The DEBUG SLEEP command will almost always fail, because we have set a 1 second timeout */
printf("`DEBUG SLEEP` error: %s\n", c->errstr ? c->errstr : "unknown error");
return;
}
/* Disconnect after receiving the reply of DEBUG SLEEP (which will not)*/
kvAsyncDisconnect(c);
valkeyAsyncDisconnect(c);
}
void getCallback(kvAsyncContext *c, void *r, void *privdata) {
kvReply *reply = r;
void getCallback(valkeyAsyncContext *c, void *r, void *privdata) {
valkeyReply *reply = r;
if (reply == NULL) {
if (c->errstr) {
printf("errstr: %s\n", c->errstr);
@@ -29,19 +29,19 @@ void getCallback(kvAsyncContext *c, void *r, void *privdata) {
printf("argv[%s]: %s\n", (char *)privdata, reply->str);
/* start another request that demonstrate timeout */
kvAsyncCommand(c, debugCallback, NULL, "DEBUG SLEEP %f", 1.5);
valkeyAsyncCommand(c, debugCallback, NULL, "DEBUG SLEEP %f", 1.5);
}
void connectCallback(kvAsyncContext *c, int status) {
if (status != KV_OK) {
void connectCallback(valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const kvAsyncContext *c, int status) {
if (status != KV_OK) {
void disconnectCallback(const valkeyAsyncContext *c, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", c->errstr);
return;
}
@@ -49,21 +49,21 @@ void disconnectCallback(const kvAsyncContext *c, int status) {
}
/*
* 1- Compile this file as a shared library. Directory of "kvmodule.h" must
* 1- Compile this file as a shared library. Directory of "valkeymodule.h" must
* be in the include path.
* gcc -fPIC -shared -I../../kv/src/ -I.. example-kvmoduleapi.c -o example-kvmoduleapi.so
* gcc -fPIC -shared -I../../valkey/src/ -I.. example-valkeymoduleapi.c -o example-valkeymoduleapi.so
*
* 2- Load module:
* kv-server --loadmodule ./example-kvmoduleapi.so value
* valkey-server --loadmodule ./example-valkeymoduleapi.so value
*/
int KVModule_OnLoad(KVModuleCtx *ctx, KVModuleString **argv, int argc) {
int ret = KVModule_Init(ctx, "example-kvmoduleapi", 1, KVMODULE_APIVER_1);
if (ret != KVMODULE_OK) {
int ValkeyModule_OnLoad(ValkeyModuleCtx *ctx, ValkeyModuleString **argv, int argc) {
int ret = ValkeyModule_Init(ctx, "example-valkeymoduleapi", 1, VALKEYMODULE_APIVER_1);
if (ret != VALKEYMODULE_OK) {
printf("error module init \n");
return KVMODULE_ERR;
return VALKEYMODULE_ERR;
}
kvAsyncContext *c = kvAsyncConnect("127.0.0.1", 6379);
valkeyAsyncContext *c = valkeyAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
@@ -71,13 +71,13 @@ int KVModule_OnLoad(KVModuleCtx *ctx, KVModuleString **argv, int argc) {
}
size_t len;
const char *val = KVModule_StringPtrLen(argv[argc - 1], &len);
const char *val = ValkeyModule_StringPtrLen(argv[argc - 1], &len);
KVModuleCtx *module_ctx = KVModule_GetDetachedThreadSafeContext(ctx);
kvModuleAttach(c, module_ctx);
kvAsyncSetConnectCallback(c, connectCallback);
kvAsyncSetDisconnectCallback(c, disconnectCallback);
kvAsyncSetTimeout(c, (struct timeval){.tv_sec = 1, .tv_usec = 0});
ValkeyModuleCtx *module_ctx = ValkeyModule_GetDetachedThreadSafeContext(ctx);
valkeyModuleAttach(c, module_ctx);
valkeyAsyncSetConnectCallback(c, connectCallback);
valkeyAsyncSetDisconnectCallback(c, disconnectCallback);
valkeyAsyncSetTimeout(c, (struct timeval){.tv_sec = 1, .tv_usec = 0});
/*
In this demo, we first `set key`, then `get key` to demonstrate the basic usage of the adapter.
@@ -86,7 +86,7 @@ int KVModule_OnLoad(KVModuleCtx *ctx, KVModuleString **argv, int argc) {
timeout error, which is shown in the `debugCallback`.
*/
kvAsyncCommand(c, NULL, NULL, "SET key %b", val, len);
kvAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
valkeyAsyncCommand(c, NULL, NULL, "SET key %b", val, len);
valkeyAsyncCommand(c, getCallback, (char *)"end-1", "GET key");
return 0;
}
+29 -29
View File
@@ -27,7 +27,7 @@
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include <kv/kv.h>
#include <valkey/valkey.h>
#include <stdio.h>
#include <stdlib.h>
@@ -43,12 +43,12 @@
exit(-1); \
} while (0)
static void assertReplyAndFree(kvContext *context, kvReply *reply, int type) {
static void assertReplyAndFree(valkeyContext *context, valkeyReply *reply, int type) {
if (reply == NULL)
panicAbort("NULL reply from server (error: %s)", context->errstr);
if (reply->type != type) {
if (reply->type == KV_REPLY_ERROR)
if (reply->type == VALKEY_REPLY_ERROR)
fprintf(stderr, "Server Error: %s\n", reply->str);
panicAbort("Expected reply type %d but got type %d", type, reply->type);
@@ -58,34 +58,34 @@ static void assertReplyAndFree(kvContext *context, kvReply *reply, int type) {
}
/* Switch to the RESP3 protocol and enable client tracking */
static void enableClientTracking(kvContext *c) {
kvReply *reply = kvCommand(c, "HELLO 3");
static void enableClientTracking(valkeyContext *c) {
valkeyReply *reply = valkeyCommand(c, "HELLO 3");
if (reply == NULL || c->err) {
panicAbort("NULL reply or server error (error: %s)", c->errstr);
}
if (reply->type != KV_REPLY_MAP) {
if (reply->type != VALKEY_REPLY_MAP) {
fprintf(stderr, "Error: Can't send HELLO 3 command. Are you sure you're ");
fprintf(stderr, "connected to kv-server >= 6.0.0?\nServer error: %s\n",
reply->type == KV_REPLY_ERROR ? reply->str : "(unknown)");
fprintf(stderr, "connected to valkey-server >= 6.0.0?\nServer error: %s\n",
reply->type == VALKEY_REPLY_ERROR ? reply->str : "(unknown)");
exit(-1);
}
freeReplyObject(reply);
/* Enable client tracking */
reply = kvCommand(c, "CLIENT TRACKING ON");
assertReplyAndFree(c, reply, KV_REPLY_STATUS);
reply = valkeyCommand(c, "CLIENT TRACKING ON");
assertReplyAndFree(c, reply, VALKEY_REPLY_STATUS);
}
void pushReplyHandler(void *privdata, void *r) {
kvReply *reply = r;
valkeyReply *reply = r;
int *invalidations = privdata;
/* Sanity check on the invalidation reply */
if (reply->type != KV_REPLY_PUSH || reply->elements != 2 ||
reply->element[1]->type != KV_REPLY_ARRAY ||
reply->element[1]->element[0]->type != KV_REPLY_STRING) {
if (reply->type != VALKEY_REPLY_PUSH || reply->elements != 2 ||
reply->element[1]->type != VALKEY_REPLY_ARRAY ||
reply->element[1]->element[0]->type != VALKEY_REPLY_STRING) {
panicAbort("%s", "Can't parse PUSH message!");
}
@@ -99,7 +99,7 @@ void pushReplyHandler(void *privdata, void *r) {
}
/* We aren't actually freeing anything here, but it is included to show that we can
* have libkv call our data destructor when freeing the context */
* have libvalkey call our data destructor when freeing the context */
void privdata_dtor(void *privdata) {
unsigned int *icount = privdata;
printf("privdata_dtor(): In context privdata dtor (invalidations: %u)\n", *icount);
@@ -107,29 +107,29 @@ void privdata_dtor(void *privdata) {
int main(int argc, char **argv) {
unsigned int j, invalidations = 0;
kvContext *c;
kvReply *reply;
valkeyContext *c;
valkeyReply *reply;
const char *hostname = (argc > 1) ? argv[1] : "127.0.0.1";
int port = (argc > 2) ? atoi(argv[2]) : 6379;
kvOptions o = {0};
KV_OPTIONS_SET_TCP(&o, hostname, port);
valkeyOptions o = {0};
VALKEY_OPTIONS_SET_TCP(&o, hostname, port);
/* Set our context privdata to the address of our invalidation counter. Each
* time our PUSH handler is called, libkv will pass the privdata for context.
* time our PUSH handler is called, libvalkey will pass the privdata for context.
*
* This could also be done after we create the context like so:
*
* c->privdata = &invalidations;
* c->free_privdata = privdata_dtor;
*/
KV_OPTIONS_SET_PRIVDATA(&o, &invalidations, privdata_dtor);
VALKEY_OPTIONS_SET_PRIVDATA(&o, &invalidations, privdata_dtor);
/* Set our custom PUSH message handler */
o.push_cb = pushReplyHandler;
c = kvConnectWithOptions(&o);
c = valkeyConnectWithOptions(&o);
if (c == NULL || c->err)
panicAbort("Connection error: %s", c ? c->errstr : "OOM");
@@ -139,23 +139,23 @@ int main(int argc, char **argv) {
/* Set some keys and then read them back. Once we do that, Redis will deliver
* invalidation push messages whenever the key is modified */
for (j = 0; j < KEY_COUNT; j++) {
reply = kvCommand(c, "SET key:%d initial:%d", j, j);
assertReplyAndFree(c, reply, KV_REPLY_STATUS);
reply = valkeyCommand(c, "SET key:%d initial:%d", j, j);
assertReplyAndFree(c, reply, VALKEY_REPLY_STATUS);
reply = kvCommand(c, "GET key:%d", j);
assertReplyAndFree(c, reply, KV_REPLY_STRING);
reply = valkeyCommand(c, "GET key:%d", j);
assertReplyAndFree(c, reply, VALKEY_REPLY_STRING);
}
/* Trigger invalidation messages by updating keys we just read */
for (j = 0; j < KEY_COUNT; j++) {
printf(" main(): SET key:%d update:%d\n", j, j);
reply = kvCommand(c, "SET key:%d update:%d", j, j);
assertReplyAndFree(c, reply, KV_REPLY_STATUS);
reply = valkeyCommand(c, "SET key:%d update:%d", j, j);
assertReplyAndFree(c, reply, VALKEY_REPLY_STATUS);
printf(" main(): SET REPLY OK\n");
}
printf("\nTotal detected invalidations: %d, expected: %d\n", invalidations, KEY_COUNT);
/* PING server */
kvFree(c);
valkeyFree(c);
}
+29 -29
View File
@@ -1,5 +1,5 @@
#include <kv/tls.h>
#include <kv/kv.h>
#include <valkey/tls.h>
#include <valkey/valkey.h>
#include <stdio.h>
#include <stdlib.h>
@@ -11,10 +11,10 @@
int main(int argc, char **argv) {
unsigned int j;
kvTLSContext *tls;
kvTLSContextError tls_error = KV_TLS_CTX_NONE;
kvContext *c;
kvReply *reply;
valkeyTLSContext *tls;
valkeyTLSContextError tls_error = VALKEY_TLS_CTX_NONE;
valkeyContext *c;
valkeyReply *reply;
if (argc < 4) {
printf("Usage: %s <host> <port> <cert> <key> [ca]\n", argv[0]);
exit(1);
@@ -25,78 +25,78 @@ int main(int argc, char **argv) {
const char *key = argv[4];
const char *ca = argc > 4 ? argv[5] : NULL;
kvInitOpenSSL();
tls = kvCreateTLSContext(ca, NULL, cert, key, NULL, &tls_error);
if (!tls || tls_error != KV_TLS_CTX_NONE) {
printf("TLS Context error: %s\n", kvTLSContextGetError(tls_error));
valkeyInitOpenSSL();
tls = valkeyCreateTLSContext(ca, NULL, cert, key, NULL, &tls_error);
if (!tls || tls_error != VALKEY_TLS_CTX_NONE) {
printf("TLS Context error: %s\n", valkeyTLSContextGetError(tls_error));
exit(1);
}
struct timeval tv = {1, 500000}; // 1.5 seconds
kvOptions options = {0};
KV_OPTIONS_SET_TCP(&options, hostname, port);
valkeyOptions options = {0};
VALKEY_OPTIONS_SET_TCP(&options, hostname, port);
options.connect_timeout = &tv;
c = kvConnectWithOptions(&options);
c = valkeyConnectWithOptions(&options);
if (c == NULL || c->err) {
if (c) {
printf("Connection error: %s\n", c->errstr);
kvFree(c);
valkeyFree(c);
} else {
printf("Connection error: can't allocate kv context\n");
printf("Connection error: can't allocate valkey context\n");
}
exit(1);
}
if (kvInitiateTLSWithContext(c, tls) != KV_OK) {
if (valkeyInitiateTLSWithContext(c, tls) != VALKEY_OK) {
printf("Couldn't initialize TLS!\n");
printf("Error: %s\n", c->errstr);
kvFree(c);
valkeyFree(c);
exit(1);
}
/* PING server */
reply = kvCommand(c, "PING");
reply = valkeyCommand(c, "PING");
printf("PING: %s\n", reply->str);
freeReplyObject(reply);
/* Set a key */
reply = kvCommand(c, "SET %s %s", "foo", "hello world");
reply = valkeyCommand(c, "SET %s %s", "foo", "hello world");
printf("SET: %s\n", reply->str);
freeReplyObject(reply);
/* Set a key using binary safe API */
reply = kvCommand(c, "SET %b %b", "bar", (size_t)3, "hello", (size_t)5);
reply = valkeyCommand(c, "SET %b %b", "bar", (size_t)3, "hello", (size_t)5);
printf("SET (binary API): %s\n", reply->str);
freeReplyObject(reply);
/* Try a GET and two INCR */
reply = kvCommand(c, "GET foo");
reply = valkeyCommand(c, "GET foo");
printf("GET foo: %s\n", reply->str);
freeReplyObject(reply);
reply = kvCommand(c, "INCR counter");
reply = valkeyCommand(c, "INCR counter");
printf("INCR counter: %lld\n", reply->integer);
freeReplyObject(reply);
/* again ... */
reply = kvCommand(c, "INCR counter");
reply = valkeyCommand(c, "INCR counter");
printf("INCR counter: %lld\n", reply->integer);
freeReplyObject(reply);
/* Create a list of numbers, from 0 to 9 */
reply = kvCommand(c, "DEL mylist");
reply = valkeyCommand(c, "DEL mylist");
freeReplyObject(reply);
for (j = 0; j < 10; j++) {
char buf[64];
snprintf(buf, 64, "%u", j);
reply = kvCommand(c, "LPUSH mylist element-%s", buf);
reply = valkeyCommand(c, "LPUSH mylist element-%s", buf);
freeReplyObject(reply);
}
/* Let's check what we have inside the list */
reply = kvCommand(c, "LRANGE mylist 0 -1");
if (reply->type == KV_REPLY_ARRAY) {
reply = valkeyCommand(c, "LRANGE mylist 0 -1");
if (reply->type == VALKEY_REPLY_ARRAY) {
for (j = 0; j < reply->elements; j++) {
printf("%u) %s\n", j, reply->element[j]->str);
}
@@ -104,9 +104,9 @@ int main(int argc, char **argv) {
freeReplyObject(reply);
/* Disconnects and frees the context */
kvFree(c);
valkeyFree(c);
kvFreeTLSContext(tls);
valkeyFreeTLSContext(tls);
return 0;
}
+25 -25
View File
@@ -1,5 +1,5 @@
#define _XOPEN_SOURCE 600 /* For strdup() */
#include <kv/kv.h>
#include <valkey/valkey.h>
#include <stdio.h>
#include <stdlib.h>
@@ -9,10 +9,10 @@
#include <winsock2.h> /* For struct timeval */
#endif
static void example_argv_command(kvContext *c, size_t n) {
static void example_argv_command(valkeyContext *c, size_t n) {
char **argv, tmp[42];
size_t *argvlen;
kvReply *reply;
valkeyReply *reply;
/* We're allocating two additional elements for command and key */
argv = malloc(sizeof(*argv) * (2 + n));
@@ -32,18 +32,18 @@ static void example_argv_command(kvContext *c, size_t n) {
argv[i] = strdup(tmp);
}
/* Execute the command using kvCommandArgv. We're sending the arguments with
/* Execute the command using valkeyCommandArgv. We're sending the arguments with
* two explicit arrays. One for each argument's string, and the other for its
* length. */
reply = kvCommandArgv(
reply = valkeyCommandArgv(
c, n + 2, (const char **)argv, (const size_t *)argvlen);
if (reply == NULL || c->err) {
fprintf(stderr, "Error: Couldn't execute kvCommandArgv\n");
fprintf(stderr, "Error: Couldn't execute valkeyCommandArgv\n");
exit(1);
}
if (reply->type == KV_REPLY_INTEGER) {
if (reply->type == VALKEY_REPLY_INTEGER) {
printf("%s reply: %lld\n", argv[0], reply->integer);
}
@@ -60,8 +60,8 @@ static void example_argv_command(kvContext *c, size_t n) {
int main(int argc, char **argv) {
unsigned int j, isunix = 0;
kvContext *c;
kvReply *reply;
valkeyContext *c;
valkeyReply *reply;
const char *hostname = (argc > 1) ? argv[1] : "127.0.0.1";
if (argc > 2) {
@@ -76,73 +76,73 @@ int main(int argc, char **argv) {
struct timeval timeout = {1, 500000}; // 1.5 seconds
if (isunix) {
c = kvConnectUnixWithTimeout(hostname, timeout);
c = valkeyConnectUnixWithTimeout(hostname, timeout);
} else {
c = kvConnectWithTimeout(hostname, port, timeout);
c = valkeyConnectWithTimeout(hostname, port, timeout);
}
if (c == NULL || c->err) {
if (c) {
printf("Connection error: %s\n", c->errstr);
kvFree(c);
valkeyFree(c);
} else {
printf("Connection error: can't allocate kv context\n");
printf("Connection error: can't allocate valkey context\n");
}
exit(1);
}
/* PING server */
reply = kvCommand(c, "PING");
reply = valkeyCommand(c, "PING");
printf("PING: %s\n", reply->str);
freeReplyObject(reply);
/* Set a key */
reply = kvCommand(c, "SET %s %s", "foo", "hello world");
reply = valkeyCommand(c, "SET %s %s", "foo", "hello world");
printf("SET: %s\n", reply->str);
freeReplyObject(reply);
/* Set a key using binary safe API */
reply = kvCommand(c, "SET %b %b", "bar", (size_t)3, "hello", (size_t)5);
reply = valkeyCommand(c, "SET %b %b", "bar", (size_t)3, "hello", (size_t)5);
printf("SET (binary API): %s\n", reply->str);
freeReplyObject(reply);
/* Try a GET and two INCR */
reply = kvCommand(c, "GET foo");
reply = valkeyCommand(c, "GET foo");
printf("GET foo: %s\n", reply->str);
freeReplyObject(reply);
reply = kvCommand(c, "INCR counter");
reply = valkeyCommand(c, "INCR counter");
printf("INCR counter: %lld\n", reply->integer);
freeReplyObject(reply);
/* again ... */
reply = kvCommand(c, "INCR counter");
reply = valkeyCommand(c, "INCR counter");
printf("INCR counter: %lld\n", reply->integer);
freeReplyObject(reply);
/* Create a list of numbers, from 0 to 9 */
reply = kvCommand(c, "DEL mylist");
reply = valkeyCommand(c, "DEL mylist");
freeReplyObject(reply);
for (j = 0; j < 10; j++) {
char buf[64];
snprintf(buf, 64, "%u", j);
reply = kvCommand(c, "LPUSH mylist element-%s", buf);
reply = valkeyCommand(c, "LPUSH mylist element-%s", buf);
freeReplyObject(reply);
}
/* Let's check what we have inside the list */
reply = kvCommand(c, "LRANGE mylist 0 -1");
if (reply->type == KV_REPLY_ARRAY) {
reply = valkeyCommand(c, "LRANGE mylist 0 -1");
if (reply->type == VALKEY_REPLY_ARRAY) {
for (j = 0; j < reply->elements; j++) {
printf("%u) %s\n", j, reply->element[j]->str);
}
}
freeReplyObject(reply);
/* See function for an example of kvCommandArgv */
/* See function for an example of valkeyCommandArgv */
example_argv_command(c, 10);
/* Disconnects and frees the context */
kvFree(c);
valkeyFree(c);
return 0;
}
+27 -27
View File
@@ -1,7 +1,7 @@
#include <kv/cluster.h>
#include <kv/tls.h>
#include <valkey/cluster.h>
#include <valkey/tls.h>
#include <kv/adapters/libevent.h>
#include <valkey/adapters/libevent.h>
#include <assert.h>
#include <stdio.h>
@@ -9,8 +9,8 @@
#define CLUSTER_NODE_TLS "127.0.0.1:7300"
void getCallback(kvClusterAsyncContext *cc, void *r, void *privdata) {
kvReply *reply = (kvReply *)r;
void getCallback(valkeyClusterAsyncContext *cc, void *r, void *privdata) {
valkeyReply *reply = (valkeyReply *)r;
if (reply == NULL) {
if (cc->err) {
printf("errstr: %s\n", cc->errstr);
@@ -20,11 +20,11 @@ void getCallback(kvClusterAsyncContext *cc, void *r, void *privdata) {
printf("privdata: %s reply: %s\n", (char *)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
kvClusterAsyncDisconnect(cc);
valkeyClusterAsyncDisconnect(cc);
}
void setCallback(kvClusterAsyncContext *cc, void *r, void *privdata) {
kvReply *reply = (kvReply *)r;
void setCallback(valkeyClusterAsyncContext *cc, void *r, void *privdata) {
valkeyReply *reply = (valkeyReply *)r;
if (reply == NULL) {
if (cc->err) {
printf("errstr: %s\n", cc->errstr);
@@ -34,8 +34,8 @@ void setCallback(kvClusterAsyncContext *cc, void *r, void *privdata) {
printf("privdata: %s reply: %s\n", (char *)privdata, reply->str);
}
void connectCallback(kvAsyncContext *ac, int status) {
if (status != KV_OK) {
void connectCallback(valkeyAsyncContext *ac, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", ac->errstr);
return;
}
@@ -43,8 +43,8 @@ void connectCallback(kvAsyncContext *ac, int status) {
printf("Connected to %s:%d\n", ac->c.tcp.host, ac->c.tcp.port);
}
void disconnectCallback(const kvAsyncContext *ac, int status) {
if (status != KV_OK) {
void disconnectCallback(const valkeyAsyncContext *ac, int status) {
if (status != VALKEY_OK) {
printf("Error: %s\n", ac->errstr);
return;
}
@@ -52,43 +52,43 @@ void disconnectCallback(const kvAsyncContext *ac, int status) {
}
int main(void) {
kvTLSContext *tls;
kvTLSContextError tls_error;
valkeyTLSContext *tls;
valkeyTLSContextError tls_error;
kvInitOpenSSL();
tls = kvCreateTLSContext("ca.crt", NULL, "client.crt", "client.key",
valkeyInitOpenSSL();
tls = valkeyCreateTLSContext("ca.crt", NULL, "client.crt", "client.key",
NULL, &tls_error);
if (!tls) {
printf("TLS Context error: %s\n", kvTLSContextGetError(tls_error));
printf("TLS Context error: %s\n", valkeyTLSContextGetError(tls_error));
exit(1);
}
struct event_base *base = event_base_new();
kvClusterOptions options = {0};
valkeyClusterOptions options = {0};
options.initial_nodes = CLUSTER_NODE_TLS;
options.async_connect_callback = connectCallback;
options.async_disconnect_callback = disconnectCallback;
options.tls = tls;
options.tls_init_fn = &kvInitiateTLSWithContext;
kvClusterOptionsUseLibevent(&options, base);
options.tls_init_fn = &valkeyInitiateTLSWithContext;
valkeyClusterOptionsUseLibevent(&options, base);
kvClusterAsyncContext *acc = kvClusterAsyncConnectWithOptions(&options);
valkeyClusterAsyncContext *acc = valkeyClusterAsyncConnectWithOptions(&options);
if (acc == NULL || acc->err != 0) {
printf("Error: %s\n", acc ? acc->errstr : "OOM");
exit(-1);
}
int status;
status = kvClusterAsyncCommand(acc, setCallback, (char *)"THE_ID",
status = valkeyClusterAsyncCommand(acc, setCallback, (char *)"THE_ID",
"SET %s %s", "key", "value");
if (status != KV_OK) {
if (status != VALKEY_OK) {
printf("error: err=%d errstr=%s\n", acc->err, acc->errstr);
}
status = kvClusterAsyncCommand(acc, getCallback, (char *)"THE_ID",
status = valkeyClusterAsyncCommand(acc, getCallback, (char *)"THE_ID",
"GET %s", "key");
if (status != KV_OK) {
if (status != VALKEY_OK) {
printf("error: err=%d errstr=%s\n", acc->err, acc->errstr);
}
@@ -96,8 +96,8 @@ int main(void) {
event_base_dispatch(base);
printf("Done..\n");
kvClusterAsyncFree(acc);
kvFreeTLSContext(tls);
valkeyClusterAsyncFree(acc);
valkeyFreeTLSContext(tls);
event_base_free(base);
return 0;
}

Some files were not shown because too many files have changed in this diff Show More