Compare commits

...
703 Commits
Author SHA1 Message Date
Hanzo AI fcb89a8d34 ci: remove KMS dep, use GITHUB_TOKEN for GHCR (Docker Hub optional) 2026-04-07 10:26:43 -07:00
Hanzo AI f583b1f289 ci: trigger arm64 rebuild 2026-04-07 10:24:59 -07:00
Hanzo AI 3eb68dc066 chore: symlink AGENTS.md + CLAUDE.md → LLM.md 2026-04-06 15:07:09 -07:00
Hanzo AI d86c624163 chore: remove stale LLM.md 2026-03-03 14:00:00 -08:00
Hanzo Dev 2be7f9fcc3 docs: add LLM.md project guide 2026-03-11 10:29:19 -07:00
Hanzo Dev c21276b98c feat: rebrand to Hanzo KV 2026-03-03 18:13:04 -08:00
Hanzo Dev bd477adeab Upgrade Valkey 8.1→9, standardize CI/CD
- Bump KV_VERSION to 9 in Dockerfile
- Unified single build-push to GHCR + Docker Hub
- Add :9 raw tag, consolidate metadata-action
2026-02-23 14:17:12 -08:00
Hanzo Dev 925c0d0591 ci: migrate deploy secrets to Hanzo KMS and fix rolling update target
Fetch DOCKERHUB_USERNAME, DOCKERHUB_TOKEN, and DIGITALOCEAN_ACCESS_TOKEN
from kms.hanzo.ai at runtime via Universal Auth. Re-enable deploy-hanzo
job targeting statefulset/redis-master container kv on hanzo-k8s cluster.
Split Docker Hub push as continue-on-error with GHCR as primary registry.
2026-02-22 22:55:21 -08:00
Hanzo Dev 8bc136996f ci: disable deploy-hanzo job (incompatible with Bitnami Helm chart)
Production redis uses bitnami/redis via Helm chart. The hanzoai/kv
image has a different entrypoint and is not a drop-in replacement.
Build+push to GHCR still works. Deploy will be re-enabled once K8s
manifests are migrated to use hanzoai/kv natively.
2026-02-22 16:59:13 -08:00
Hanzo Dev 2e067688e7 fix: correct container name to 'redis' in rolling update
The redis-master statefulset has container named 'redis', not 'redis-master'.
2026-02-22 16:41:48 -08:00
Hanzo Dev 9b95d25097 fix: correct KV rolling update target and remove lux-k8s deploy
hanzo-k8s runs KV as statefulset/redis-master, not statefulset/hanzo-kv.
lux-k8s has no KV statefulset so deploy-lux job is removed.
2026-02-22 16:33:01 -08:00
Hanzo Dev ffa6aee1c4 fix(ci): use correct K8s cluster names for doctl 2026-02-22 16:19:15 -08:00
Hanzo Dev 82a2609633 fix(ci): split GHCR and Docker Hub pushes for resilient builds 2026-02-22 16:07:09 -08:00
Hanzo Dev f222570427 fix(ci): make Docker Hub push optional, GHCR is primary registry 2026-02-22 16:04:26 -08:00
Hanzo Dev 56cc706cfb fix(ci): use correct KMS environment slug 'prod' for gitops project 2026-02-22 15:57:45 -08:00
Hanzo Dev ac04c33672 feat: fetch all CI/CD secrets from Hanzo KMS
Replace direct GitHub secrets (DOCKERHUB_USERNAME, DOCKERHUB_TOKEN,
DIGITALOCEAN_ACCESS_TOKEN) with KMS Universal Auth fetch at runtime.
Only KMS_CLIENT_ID and KMS_CLIENT_SECRET remain as GitHub secrets.
2026-02-22 15:30:47 -08:00
Zach Kelling 49e0fd9357 feat: multi-arch builds, rebrand CLI tools to kv-*
- Add QEMU setup for cross-platform Docker builds (linux/amd64 + linux/arm64)
- Rename all CLI tools: valkey-* → kv-* (kv-server, kv-cli, kv, etc.)
- Change VALKEY_VERSION ARG to KV_VERSION
- Use kv-server as entrypoint, kv ping for healthcheck
- Add semver tagging via metadata-action for proper releases
- Update K8s deploys to statefulset/hanzo-kv with container name kv
- Remove unnecessary checkout steps from deploy jobs
- Clean up README: remove valkey references, document kv CLI tools
2026-02-22 14:20:55 -08:00
Zach Kelling d688039a83 Merge remote-tracking branch 'upstream/unstable' 2026-02-22 00:45:37 -08:00
Nikhil MangloreandGitHub 1948f9965e Skip large-memory unit test with ASan that got OOM (#3230)
The `test_quicklistCompressAndDecompressQuicklistListpackNode` unit test
was failing with ASan in CI with OOM errors. The test was allocating and
compressing up to 1GB of data (32 iterations × 32MB), which exceeded
available memory in CI environments. The test is skipped for ASan
builds.

This PR addressed the unit test failures in
`test-sanitizer-address-large-memory` for both GCC and CLANG

Resolves #3221

---------

Signed-off-by: Nikhil Manglore <nmanglor@amazon.com>
2026-02-19 20:34:42 +01:00
80cbe0729d Add ALLOW_BUSY flag to SELECT and various CLIENT sub-commands (#3217)
## Add `ALLOW_BUSY` flag to `SELECT` command

### Motivation
When a client decides to switch databases (e.g., from DB 0 to DB 1), it
must issue `SELECT` command.
However it is possible that when the command was sent, the server was
running a very long script, causing it to respond with `-BUSY` error.
This leads to several issues:
1. pipelining clients can face inconsistency when the select is followed
by a pipeline of commands to write to this database. in case only the
select gets the `-BUSY` error and some of the rest of the commands are
process AFTER the long script executes, these commands will be processed
on the wrong database context.
2. Although clients can maintain logic to handle the prior issue, it
complicates client logic and places a "head of line" delay, since the
client will need to wait for the select return in order to continue
pipeline commands, potentially forcing it to allocate a different
connection per database. Also this is probably NOT being handled by any
known client library ATM.
3. With the introduction of multi-database support in cluster mode,
clients managing connections across multiple shards need to maintain a
consistent database selection across all their connections. When a
client decides to switch databases (e.g., from DB 0 to DB 1), it must
issue `SELECT` to every shard in the cluster.
Currently, if any shard is busy (e.g., executing a long-running Lua
script or module command), the `SELECT` call to that shard will be
rejected with a `-BUSY` error. This creates a **split-brain scenario**
where some connections have switched to the new database while others
remain on the old one.

### Why `ALLOW_BUSY` is safe for `SELECT`

`SELECT` is a **connection-local, metadata-only operation**. It simply
changes which database index the current client connection points to.
It:

- **Does not read or write any keys** - there is no data conflict with a
running script.
- **Is already flagged as `fast`** - it runs in O(1) and will not
contribute to or extend any busy condition.
- **Is already flagged as `loading`** - the server already recognizes
that `SELECT` is safe to execute during sensitive server states (dataset
loading from disk), establishing a precedent that this command is
harmless to run when other commands are being rejected.
- **Is already flagged as `stale`** - similarly, `SELECT` is allowed on
stale replicas, further confirming it is treated as a safe, non-data
operation.

### Changes

- Added `allow_busy` to the `command_flags` for `SELECT` in
`src/commands/select.json`.

Signed-off-by: Gabi Ganam <ggabi@amazon.com>
Co-authored-by: Gabi Ganam <ggabi@amazon.com>
2026-02-19 08:50:32 +02:00
Zach Kelling cbfd0d9c81 Merge upstream/unstable: sync with Valkey latest
Keep Hanzo KV branding in README.
2026-02-17 18:41:55 -08:00
Nikhil MangloreandGitHub c4b592a9b2 Fix large-memory tests by reducing allocation size (#3224)
Large-memory tests were failing with "I/O error reading reply" after the
first test `EVAL - JSON string encoding a string larger than 2GB`. Three
tests that shared a single server instance led to cascading failures
when the first test's 3GB allocation left the server in a
memory-stressed state.

This PR isolates each test by starting its own server. Additionally,
I've reduced the JSON encoding test size from 3GB to 2.25GB. This
included changing the allocated 1GB string (concatenated 3x) to a 750MB
string to avoid memory exhaustion while still testing strings greater
than 2GB like we intended.

---------

Signed-off-by: Nikhil Manglore <nmanglor@amazon.com>
2026-02-18 00:47:56 +01:00
Sachin Venkatesha MurthyandGitHub 31a4b1cb5a [BUG][issue-2797] Block module unload when ACL references module subcommands (#3160)
Summary:

Prevent MODULE UNLOAD when ACL rules reference a module subcommand.
Avoids crash during ACL recompute after module removal.
Adds coverage for subcommand ACL rules.
Tests:

make test
Fixes #2797

---------

Signed-off-by: sachinvmurthy <sachin.murthy97@gmail.com>
Signed-off-by: Sachin Venkatesha Murthy <sachin.murthy97@gmail.com>
2026-02-17 15:09:21 -08:00
Ran ShidlansikandGitHub aa00a6babc Always untrack initialized safe iterator (#3223)
in #2807 we added safe-iterator invalidation when a hashtable is
deleted.
When we create a safeIterator we initialize it to it->table=0 and
it-index=-1.
HOWEVER say someone creates a safe iterator never progress it and later
on just delete the iterator using `hashtableReleaseIterator`.
the iterator will NOT be untracked since, `hashtableCleanupIterator`
will skip the untrack:

```c
    if (!(iter->index == -1 && iter->table == 0)) {
        if (isSafe(iter)) {
            hashtableResumeRehashing(iter->hashtable);
            untrackSafeIterator(iter);
        } else {
            assert(iter->fingerprint == hashtableFingerprint(iter->hashtable));
        }
    }
```

and then when the hashtable is freed the iterator will be referenced
leading to a use-after-free.

For example:

```c
    it = hashtableCreateIterator(ht, HASHTABLE_ITER_SAFE);
    hashtableReleaseIterator(it);
    hashtableRelease(ht);
```

This PR fix it by ALWAYS untrack safe iterators which have a valid
hashtable when they are being cleaned up.

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2026-02-17 21:24:33 +02:00
BinbinandGitHub 1484ab7a6d Fix flaky cluster automatic failover test (#3206)
The test somehow is slow and due to the short cluster-node-timeout, an
automatic
failover may fail to trigger due to cluster-replica-validity-factor:
```
*** [err]: Automatic failover vote is not limited by two times the node timeout - mixed failover in tests/unit/cluster/manual-failover.tcl
The third failover does not happen

xxx # Cluster state changed: fail
xxx # Cluster is currently down: At least one hash slot is not served by any available node. Please check the 'cluster-require-full-coverage' configuration.
```

Closes #3203.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-02-17 09:56:19 +08:00
Sarthak AggarwalandGitHub 75e592e501 Fixes memory leak when the job crashes before it's freed (#3178)
If there is a crash between the time the job is popped and freed, we
technically leak memory. This change allows us to peek, and pop just
before we are about to free the job.

Fixes valgrind errors:
https://github.com/valkey-io/valkey/actions/runs/21969557125/job/63467641572#step:6:8648

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-02-16 14:30:01 -08:00
Nikhil MangloreandGitHub 7bb190c754 Enable USE_LIBBACKTRACE Across CI and Fix Alpine Builds (#3213)
This PR enables `USE_LIBBACKTRACE=yes` across all CI builds and builds
upon the changes introduced in #3034. Alpine-based jobs previously
attempted to install `libbacktrace-dev`, which does not exist in
Alpine’s apk repositories.

This caused these two errors in the daily tests below:
-
https://github.com/valkey-io/valkey/actions/runs/22045858351/job/63694456995
-
https://github.com/valkey-io/valkey/actions/runs/22045858351/job/63694457018

To resolve this, Alpine jobs now build GNU libbacktrace from source
inside the container before compiling Valkey. This aligns Alpine
behavior with other environments (Ubuntu jobs) and now avoids utilizing
non-existent Alpine packages.

An alternative approach we can consider is to disable `USE_LIBBACKTRACE`
for Alpine-based tests.

Signed-off-by: Nikhil Manglore <nmanglor@amazon.com>
2026-02-16 21:28:14 +01:00
Sarthak AggarwalandGitHub ed786878f4 Label is not removed automatically after extra tests are completed (#3202)
We made some changes to the workflow where the label was getting removed
every time we ran it. This changes handles it, removes
`pull_request_target` and doesn't re-trigger on adding another label.

I tried it here: https://github.com/sarthakaggarwal97/valkey/pull/65
The code is already merged in my unstable.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-02-16 19:21:12 +01:00
Daniil KashapovandGitHub 3e1e00b9ff Fix database ACL check for alldbs commands (#3215)
At one point in development there was assumption that intset of database
IDs would have IDs only in `[0, server.dbnum-1]` range, but
[here](https://github.com/valkey-io/valkey/pull/2309#issuecomment-3684557935)
we decided to change upper bound to `INT32_MAX` for ACL compatibility
reasons between nodes.
Because of that change, assumption is not true anymore and we should
explicitly check each database list to contain access to full `[0,
server.dbnum-1]` range.
Also added test for that.

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
2026-02-16 10:29:08 +01:00
BinbinandGitHub 6de32ab993 Added comment to maxmemory-clients conf to mention the performance issue (#1840)
It's worth mentioning that enabling `maxmemory-clients` will affect the
performance so people can perform its own benchmark before enabling it.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-02-16 11:23:41 +08:00
Viktor SöderqvistandGitHub 9a16d548b3 Optimize endian conversion functions (#3201)
Replace the implementation of byte-reverse functions.

The new implementation compiles to a single bswap instruction. The old
implementation didn't.

The new implementation is small and all in the .h file. The .c file is
deleted.

On little-endian systems, this affects streams, module timers and
cluster code and a few more things, where `htonu64()` is used for
storing stream-id or timestamps in a rax in big-endian order.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-02-15 23:54:34 +01:00
Viktor SöderqvistandGitHub a8b887cd4d Optimize ustime() with monotonic delta (#3193)
Only call gettimeofday() periodically and add monotonic delta. This
avoids a syscall if we have a fast monotonic clock, which we have now by
default on most platforms.

`ustime()` is called once per command execution. In a busy server
running 1M RPS, `ustime()` is called every microsecond. By calling
`gettimeofday()` only once every millisecond, we save 99.9% of these
syscalls.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-02-15 23:53:12 +01:00
2a7b6d6d5e Add test-tls-only CI job (#3143)
Add a CI specifically for TLS to build and test
both built-in and module modes.

---------

Signed-off-by: Yang Zhao <zymy701@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-02-15 10:45:52 +01:00
Rain ValentineandGitHub b398e01967 Workflows use actions/checkout for libbacktrace instead of git clone (#3204)
A relatively simple cleanup in our workflows. Using actions is best
practice.

Signed-off-by: Rain Valentine <rsg000@gmail.com>
2026-02-15 10:34:09 +01:00
Jacob MurphyandGitHub 5a29f746ba Fix bug causing no response flush sometimes when IO threads are busy (#3205) 2026-02-14 09:04:25 -08:00
Zach Kelling 0fc2cb5586 Rebrand README from Valkey to Hanzo KV 2026-02-14 05:13:43 -08:00
Zach Kelling cb50deec47 chore: sync uncommitted changes 2026-02-13 22:15:50 -08:00
Rain ValentineandGitHub a421db6826 add option to use libbacktrace for backtraces in crash reports (#3034)
With this we get more detailed backtrace information, including
information about static functions. Off by default - to enable you must
enable at compile time:

    make USE_LIBBACKTRACE=yes

Signed-off-by: Rain Valentine <rsg000@gmail.com>
2026-02-13 17:22:29 +01:00
stydxmandGitHub 96cce32979 Fix pointer-to-int-cast warnings for RDMA address handling (#3186)
When building Valkey 9 for openSUSE Tumbleweed with RDMA enabled there are some
build failures for the i586 and armv7l arches (with `-Werror`):
```
[   45s] cc -std=c99 -pedantic -O3 -fPIC -fvisibility=hidden  -O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g -Wall -Wextra -pedantic -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers -Werror -g -ggdb  -Iinclude/valkey -I../../src/ -I../../src/ -MMD -MP -c src/rdma.c -o obj/rdma.o
[   45s] src/rdma.c: In function ‘rdmaPostRecv’:
[   45s] src/rdma.c:164:21: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
[   45s]   164 |     recv_wr.wr_id = (uint64_t)cmd;
[   45s]       |                     ^
[   45s] src/rdma.c: In function ‘rdmaSendCommand’:
[   45s] src/rdma.c:305:21: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
[   45s]   305 |     send_wr.wr_id = (uint64_t)_cmd;
[   45s]       |                     ^
[   45s] In file included from /usr/include/sys/types.h:176,
[   45s]                  from include/valkey/valkey.h:42,
[   45s]                  from include/valkey/async.h:34,
[   45s]                  from src/rdma.c:36:
[   45s] src/rdma.c: In function ‘connRdmaRegisterRx’:
[   45s] src/rdma.c:323:31: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
[   45s]   323 |     cmd.memory.addr = htobe64((uint64_t)ctx->recv_buf);
[   45s]       |                               ^
[   45s] src/rdma.c: In function ‘connRdmaHandleRecv’:
[   45s] src/rdma.c:341:24: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
[   45s]   341 |         ctx->tx_addr = (char *)be64toh(cmd->memory.addr);
[   45s]       |                        ^
[   45s] cc1: all warnings being treated as errors
[   45s] make[3]: *** [Makefile:228: obj/rdma.o] Error 1
```

Fixes #3183

Signed-off-by: stydxm <stydxm@outlook.com>
2026-02-13 17:57:15 +08:00
Zach Kelling e54e87205c Simplify: Valkey base with Hanzo defaults via CMD args 2026-02-12 21:03:50 -08:00
Zach Kelling b61d422d67 Use official Valkey base image with Hanzo config overlay 2026-02-12 21:02:39 -08:00
Zach Kelling 4359a8f60d Fix: use libc malloc for cross-platform builds 2026-02-12 20:59:25 -08:00
Zach Kelling 4ffaecd855 Add Dockerfile and CI workflow for GHCR image 2026-02-12 19:16:43 -08:00
Rain ValentineandGitHub 66597c6948 Update and pin github actions to full SHAs for supply chain security (#3185)
Updates to latest versions for each of the github actions used.

Pinning prevents an attack where the upstream action dependency is
compromised and the "v4" tag for example gets edited to point to a
malicious version. We already do this for most checkout actions in our
workflows.

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
2026-02-12 22:49:21 +01:00
4b6f258a09 Fix server assert on ACL LOAD and resetchannels (#3182)
If the client executing `ACL LOAD` is a user that is removed from ACL
file, it frees itself while still executing the command. Same happens
for user while removing it's channel permissions.
This causes `c->conn` pointer to become `null` and later
`serverAssert(c->conn)` fails for that client on write in
`prepareClientToWrite()`.
Solution is to flag current client to be closed after command completes
and reply is written, later this client is freed async.

Resolves #3181

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2026-02-12 22:15:00 +01:00
Sarthak AggarwalandGitHub 4b0c287f76 Build Lua by default with make test (#3190)
Currently if we just run make test, it fails because Lua is not built
automatically. Thanks to @zuiderkwast for pointing the fix.

Make Test Passed

```

Test Summary: 5895 passed, 0 failed

\o/ All tests passed without errors!

Cleanup: may take some time... OK
```

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-02-11 15:34:45 -08:00
Sarthak AggarwalandGitHub fb96a33c76 Steps to run daily workflow in forked repo (#3168)
I think a lot of new contributors are not aware how they can run the
extended list of tests in their forked repo branch with other test
arguments.

This change could be helpful to some people.

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-02-10 13:46:54 -08:00
Sarthak AggarwalandGitHub 11e644bd8e Separate jobs for large memory tests with sanitizers (#3161)
We have been seeing github actions runners being OOM when large memory
tests are run with ASan. The operation eventually is being canceled
during the test.

This change moves the large-memory tests with ASan and UBSan to separate
jobs, so we get a dedicated runner with its own timeout. We can tweak
the number of simultaneous test clients for these tests without
affecting the other test jobs.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-02-10 22:10:31 +01:00
Sarthak AggarwalandGitHub e72cb95999 Fix PFADD corrupted sparse HLL handling by restoring hllSparseSet return type (#3184)
This fixes a regression where PFADD returned 1 instead of INVALIDOBJ for
corrupted sparse HLL payloads. In small strings refactor (#2516) we
incorrectly changed the datatype from int to bool. That function still
uses -1 as its error signal.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-02-10 21:50:57 +01:00
BinbinandGitHub 5da3040931 Fix remaining_repl_size should be in CLUSTER GETSLOTMIGRATIONS in conf (#3180) 2026-02-09 19:07:47 -08:00
BinbinandGitHub 4452c66db4 Add remaining_repl_size field in CLUSTER MIGRATESLOTS output (#3135)
There should be a way for people to monitor the amount of output buffer
data accumulating at the source node during the migration process.

This PR also do a change in slotExportTryDoPause, previously, we used 
`getClientOutputBufferMemoryUsage` to determine if the offset was sufficient
to trigger the pause, it also adding the listNode overhead. Now we use
client->bytes.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-02-10 10:26:17 +08:00
eb82a5f49b Add more client info flags to module API (#2868)
Adding more client flags in the information populated by
`ValkeyModule_GetClientInfoById()`:

    VALKEYMODULE_CLIENTINFO_FLAG_PRIMARY
    VALKEYMODULE_CLIENTINFO_FLAG_REPLICA
    VALKEYMODULE_CLIENTINFO_FLAG_MONITOR
    VALKEYMODULE_CLIENTINFO_FLAG_MODULE
    VALKEYMODULE_CLIENTINFO_FLAG_AUTHENTICATED
    VALKEYMODULE_CLIENTINFO_FLAG_EVER_AUTHENTICATED
    VALKEYMODULE_CLIENTINFO_FLAG_FAKE

Closes #2590

---------

Signed-off-by: martinrvisser <mvisser@hotmail.com>
Signed-off-by: martinrvisser <martinrvisser@users.noreply.github.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-02-09 14:56:18 +00:00
7c0915241b Allow some tags only on top level to fix --tags filtering (#3171)
The problem: when running `./runtest --large-memory --tags large-memory`
the `--tags large-memory` filter would look for tests with
`large-memory` tag, it found the nested tag in the test context but
`start_server` didn't have `large-memory`, so the filtering logic had
issues. And that's why we did not notice how we introduced this failure:

```
[err]: CVE-2025-32023: Sparse HLL XZERO overflow triggers crash in tests/unit/hyperloglog.tcl
Expected an error matching '*INVALIDOBJ*' but got '1' (context: type eval line 24 cmd {assert_error {*INVALIDOBJ*} {r pfadd hll_overflow foo}} proc ::test)
```

Tags `large-memory`, `needs:other-server`, `compatible-redis`, and
`network` now restricted to top-level `start_server`. Prevents `--tags`
filtering issues when these tags appear in nested contexts. Added
validation that errors on nested usage. Refactored tests that fail new
validation. Also fixed all `lsearch` calls to use `-exact` flag to
preventing bugs like "tls" matching "tls:skip".

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-02-09 12:01:30 +01:00
7431d4f6a6 test: add valgrind:skip tag to automatically skip tests under valgrind (#3138)
**Fixes #3127**

## Description
Adds a `valgrind:skip` tag to automatically skip tests when running with
`--valgrind`. This allows skipping before the server starts, rather than
checking `$::valgrind` inside tests.

## Changes
- Added tag check in `server.tcl`
- Replaced `!$::valgrind` checks with the new tag in affected tests
- Updated README documentation
- Refactored tests in `hash.tcl` by adding `valgrind:skip` tag inline
- Added `valgrind:skip` tag to a test in `incr.tcl`

## Testing
Verified tests are correctly skipped with `--valgrind` and run normally
without it.

Signed-off-by: Mangat Toor <immangat@amazon.com>
Co-authored-by: Mangat Toor <immangat@amazon.com>
2026-02-09 18:57:06 +08:00
BinbinandGitHub 463cef2e8f Revert "Extend test helper wait_for_cluster_propagation to support ignoring certain node (#3069)" (#3176)
This reverts #3069 commit e95429a728.

Clearly, it doesn't handle TLS-related aspects, which caused it
to fail some TLS tests. Furthermore, this check is also incorrect
under the "announce client port" setting.

There isn't much value in doing this and there is no need to extend
the helper function.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-02-09 18:55:08 +08:00
90c1563041 Make TLS disable/enable INFO checks deterministic for module mode (#3167)
Fixing TLS module test failure, closes #3165

Stabilizes TLS unit tests around enable/disable toggling in module mode.

Code change summary:
`tls.c` has two `tlsResetCertInfo()` implementations under different #if
branches.

- OpenSSL (built‑in TLS) branch calls tlsRefreshAllCertInfo() because it
can read certs from SSL contexts.

- Non‑OpenSSL stub (module build) can’t access certs, so it only clears
on disable and relies on the module to populate cert info.

---------

Signed-off-by: Yiwen Zhang <yiwen_zhang@apple.com>
Co-authored-by: Yiwen Zhang <yiwen_zhang@apple.com>
2026-02-09 10:25:28 +01:00
f994093139 Avoid crash in genClusterInfoString when myself is uninitialized (#3173)
Fixes a crash that occurs when `genClusterInfoString()` is called before
the cluster node has fully initialized. The myself pointer may be `NULL`
in these edge cases, causing a segfault when calling `nodeEpoch(myself)`.

Related to #3170

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthakaggarwal97@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2026-02-09 12:41:06 +08:00
Zhijun LiaoandGitHub e95429a728 Extend test helper wait_for_cluster_propagation to support ignoring certain node (#3069)
Enhanced the popular helper `wait_for_cluster_propagation` to support
the scenario where we intentionally stop a server and wait for the other
nodes to converge without it.

Signed-off-by: Zhijun <dszhijun@gmail.com>
2026-02-08 22:51:45 +08:00
NAM UK KIMandGitHub 6cf08c81ef Minor cleanup in memtoull to also check ERANGE when calling strtoull (#3159)
It might be a good idea to improve the code quality to carefully check
for exceptions when converting strings to numbers by calling the
strtoull function.

EINVAL checks if a number was converted incorrectly, and checks if the
entire buf is a number, but requires ERANGE in addition.

It's safe to check for overflows, which can occur when a user enters an
absurdly large value like 9999999999999999999999999gb.

Suggested fix: Check both EINVAL and ERANGE

Signed-off-by: namuk2004 <namuk2004@naver.com>
2026-02-08 12:58:56 +08:00
Sarthak AggarwalandGitHub 80326f9f66 make test_empty_buckets_rehashing deterministic across hash seeds (#3174)
I was working on ASAN large memory tests when I countered this issue.

The issue was that the hardcoded `999` key could land in an early
bucket. Then shrink rehash could finish early, and later inserts could
trigger a new expansion rehash, resetting rehash_idx low. The test now
picks the survivor key dynamically as the key mapped to the highest
bucket index.

```
[test_hashtable.c] Memory leak detected of 336 bytes
=================================================================
==3901==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 80 byte(s) in 1 object(s) allocated from:
    #0 0x7fb0556fd9c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x563bfdf4c47d in ztrymalloc_usable_internal /home/runner/work/valkey/valkey/src/zmalloc.c:156
    #2 0x563bfdf4c47d in valkey_malloc /home/runner/work/valkey/valkey/src/zmalloc.c:185
    #3 0x563bfdd42eaf in hashtableCreate /home/runner/work/valkey/valkey/src/hashtable.c:1217
    #4 0x563bfdaa1cbf in test_empty_buckets_rehashing unit/test_hashtable.c:232
    #5 0x563bfdae772b in runTestSuite unit/test_main.c:36
    #6 0x563bfda86b20 in main unit/test_main.c:108
    #7 0x7fb05522a1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
    #8 0x7fb05522a28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
    #9 0x563bfda8a5c4 in _start (/home/runner/work/valkey/valkey/src/valkey-unit-tests+0x17c5c4) (BuildId: 44cfc183e6e82e499bcc9f6adc094d7f774ee9d2)

Indirect leak of 128 byte(s) in 1 object(s) allocated from:
    #0 0x7fb0556fd340 in calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x563bfdf4c922 in ztrycalloc_usable_internal /home/runner/work/valkey/valkey/src/zmalloc.c:214
    #2 0x563bfdf4c922 in valkey_calloc /home/runner/work/valkey/valkey/src/zmalloc.c:257
    #3 0x563bfdd40967 in resize /home/runner/work/valkey/valkey/src/hashtable.c:741
    #4 0x563bfdd45eb1 in hashtableExpandIfNeeded /home/runner/work/valkey/valkey/src/hashtable.c:1446
    #5 0x563bfdd45eb1 in hashtableExpandIfNeeded /home/runner/work/valkey/valkey/src/hashtable.c:1433
    #6 0x563bfdd45eb1 in insert /home/runner/work/valkey/valkey/src/hashtable.c:1041
    #7 0x563bfdd45eb1 in hashtableAddOrFind /home/runner/work/valkey/valkey/src/hashtable.c:1554
    #8 0x563bfdd45eb1 in hashtableAdd /home/runner/work/valkey/valkey/src/hashtable.c:1539
    #9 0x563bfdaa1e3b in test_empty_buckets_rehashing unit/test_hashtable.c:254
    #10 0x563bfdae772b in runTestSuite unit/test_main.c:36
    #11 0x563bfda86b20 in main unit/test_main.c:108
    #12 0x7fb05522a1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
    #13 0x7fb05522a28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
    #14 0x563bfda8a5c4 in _start (/home/runner/work/valkey/valkey/src/valkey-unit-tests+0x17c5c4) (BuildId: 44cfc183e6e82e499bcc9f6adc094d7f774ee9d2)

Indirect leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7fb0556fd340 in calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x563bfdf4c922 in ztrycalloc_usable_internal /home/runner/work/valkey/valkey/src/zmalloc.c:214
    #2 0x563bfdf4c922 in valkey_calloc /home/runner/work/valkey/valkey/src/zmalloc.c:257
    #3 0x563bfdd3f553 in bucketConvertToChained /home/runner/work/valkey/valkey/src/hashtable.c:908
    #4 0x563bfdd3f553 in findBucketForInsert /home/runner/work/valkey/valkey/src/hashtable.c:1021
    #5 0x563bfdd45d9e in insert /home/runner/work/valkey/valkey/src/hashtable.c:1045
    #6 0x563bfdd45d9e in hashtableAddOrFind /home/runner/work/valkey/valkey/src/hashtable.c:1554
    #7 0x563bfdd45d9e in hashtableAdd /home/runner/work/valkey/valkey/src/hashtable.c:1539
    #8 0x563bfdaa1e3b in test_empty_buckets_rehashing unit/test_hashtable.c:254
    #9 0x563bfdae772b in runTestSuite unit/test_main.c:36
    #10 0x563bfda86b20 in main unit/test_main.c:108
    #11 0x7fb05522a1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
    #12 0x7fb05522a28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
    #13 0x563bfda8a5c4 in _start (/home/runner/work/valkey/valkey/src/valkey-unit-tests+0x17c5c4) (BuildId: 44cfc183e6e82e499bcc9f6adc094d7f774ee9d2)

Indirect leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7fb0556fd340 in calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x563bfdf4c922 in ztrycalloc_usable_internal /home/runner/work/valkey/valkey/src/zmalloc.c:214
    #2 0x563bfdf4c922 in valkey_calloc /home/runner/work/valkey/valkey/src/zmalloc.c:257
    #3 0x563bfdd40967 in resize /home/runner/work/valkey/valkey/src/hashtable.c:741
    #4 0x563bfdaa1df8 in test_empty_buckets_rehashing unit/test_hashtable.c:248
    #5 0x563bfdae772b in runTestSuite unit/test_main.c:36
    #6 0x563bfda86b20 in main unit/test_main.c:108
    #7 0x7fb05522a1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
    #8 0x7fb05522a28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
    #9 0x563bfda8a5c4 in _start (/home/runner/work/valkey/valkey/src/valkey-unit-tests+0x17c5c4) (BuildId: 44cfc183e6e82e499bcc9f6adc094d7f774ee9d2)

SUMMARY: AddressSanitizer: 336 byte(s) leaked in 4 allocation(s).
```

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-02-08 12:35:17 +08:00
Hanxi ZhangandGitHub 8876cff5ed Fix flake replicas different ranks test (#3164)
Address #2683 

For the test, I found two problems.

### 1. Test Assumes Winner Has Rank #0
In my test run, in the failing case:
- Replica -3 (rank 0) won epoch 10 first
- Replica -6 (rank 1) won epoch 11 second
- Replica -3 saw higher epoch and stepped down
- Final result: rank 1 became master

Rank #0 doesn't guarantee being the final master.

### 2. Pattern Matching Bug
The pattern `*Start of election*rank #0*` incorrectly matches "primary
rank #0":

Log: `Start of election delayed for 350 milliseconds (rank #1, primary
rank #0, offset 2172)`

This line has rank `#1`, but the pattern matches because of "primary
rank `#0`" at the end.

## Solution
- Fixed the format problem by checking if `(rank #0` or `(rank #1` so
that we won't accidentally match the primary rank
- Only check to make sure replica 3 and replica 6 have different ranks
without assuming the replica with rank 0 will become the master.

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
2026-02-05 17:56:12 +08:00
Yang ZhaoandGitHub 2022f6a4ca Fix arr bound err in test-sanitizer-undefined (gcc) (#3155)
- The `ifdef` in `bio.c` caused array bound err in daily job
[test-sanitizer-undefined
(gcc)](https://github.com/valkey-io/valkey/actions/runs/21573028934/job/62155215297).
It’s still unclear to me why this is the only job that catches the issue
during make, I’d appreciate any insights.
- This CI job is currently skipped for PRs even with the
`run-extra-tests` label, should we consider enabling it?

**Verification**
Confirmed that this issue is fixed by temporarily enabling the
[test-sanitizer-undefined (gcc) job in my
fork](https://github.com/yang-z-o/valkey/actions/runs/21618706272/job/62303009722).

---------

Signed-off-by: Yang Zhao <zymy701@gmail.com>
2026-02-05 10:33:09 +01:00
bf64b19f5e Rehashing more empty buckets to optimize the rehashing speed (#3150)
A hashtable bucket has usually multiple entries, we can just do
one step = one bucket chain, regardless of empty or not.

But in shrinking case, there is a case that the ht0 can be very
empty (but the table size is huge), when doing one step rehash,
if there are a lot of empty buckets, we can do more step in this
case to make the rehash faster.

We need to make sure the rehash is done before we need to resize
again. There is another case that can cause issue, like when in
the shrinking case, the new ht1 is small, but the hash collisions
become more often when we adding elements, we need to rehash the
empty buckets in ht0 as quickly as possible (using more CPU).

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-02-05 14:54:09 +08:00
Sarthak AggarwalandGitHub eb82d7a786 Skips the internal clients from logresreq checks (#3154)
The request/response logging assumes every command produces a reply.
This breaks for slot migration clients that execute internal commands
like `CLUSTER SYNCSLOTS ACK`, which intentionally have no reply.

Fixes frequent daily failures:
https://github.com/valkey-io/valkey/actions/runs/21573028934/job/62155215296#step:6:5241

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-02-04 14:08:15 -08:00
91c219b6d1 Add cache for COMMAND (#2839)
COMMAND command is expensive to process (too much string operations)
but result is rarely changes, it makes sense to add caching

---------

Signed-off-by: Evgeny Barskiy <ebarskiy@snapchat.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-02-04 12:57:04 +01:00
1967fb1187 Add server side TLS certificate expiry tracking and INFO telemetry (#2913)
- Parse TLS certificate expiry/serials and refresh cached values at
startup and on TLS reconfigure;

- expose INFO # TLS fields so monitoring can alert on impending expiry.


INFO:
```
# TLS
tls_server_cert_serial:826F01147AC63276
tls_server_cert_expires_in_seconds:123456
tls_client_cert_serial:826F01147AC63276
tls_client_cert_expires_in_seconds:123456
tls_ca_cert_serial:11A566...
tls_ca_cert_expires_in_seconds:987654
```

---------

Signed-off-by: Yiwen Zhang <yiwen_zhang@apple.com>
Signed-off-by: Yiwen Zhang <zhangyiwen1221@gmail.com>
Co-authored-by: Yiwen Zhang <yiwen_zhang@apple.com>
2026-02-03 21:00:30 -08:00
BinbinandGitHub 6b6750f3fe Optimize SREM/ZREM/HDEL to pause auto shrink when deleting multiple items (#3144)
When ZREM / SREM / HDEL delete a large number of elements, we can apply the same
optimization as ZREMRANGEBY*, which is to temporarily pause the autoshrink of
the corresponding hashtable to avoid triggering resize/rehash repeatedly during
the deletion process.

This theoretically leads to some performance improvements. Furthermore, by only
triggering a resize once at the end (i.e., only when `hashtableResumeAutoShrink`
is called), we can get the most right resize, preventing further resizes during
the resize/rehash process.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-02-04 00:14:55 +08:00
Zhijun LiaoandGitHub 334547c265 Skip flaky test cases in client-eviction.tcl when in TLS mode (#3151)
Closes #3146

The following two test cases are flaky

- `evict clients only until below limit` - uses exact math expecting
exactly half the clients evicted
- `evict clients in right order (large to small)` - uses exact math
expecting specific clients evicted in order

It's fine to skip them in TLS because the core logic being tested
(client eviction) doesn't change based on TLS vs non-TLS.

The `decrease maxmemory-clients causes client eviction` test case could
potentially be flaky as well (has not shown flakiness on CI yet), but
since it has more tolerant assertion: `connected_clients > 0 &&
connected_clients < $client_count`, I think it's okay not to bother
skipping it.

Other test cases are not flaky because they use large thresholds or
check binary outcomes (yes/no eviction), not exact counts.

Signed-off-by: Zhijun <dszhijun@gmail.com>
2026-02-03 14:24:06 +01:00
9d9282f160 Optimize rehash completion check for empty hash table (#3141)
It is possible to have an empty table with only empty buckets
remaining, and we are still rehashing those empty buckets. If
table 0 is empty (both used and child_buckets are empty), we can
simply finish the rehashing process, since we no longer have any
elements (buckets) to rehash.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-02-03 10:22:18 +08:00
Zhijun LiaoandGitHub 5a2ca67b16 CI: Stop using symlinks for tests with CMake (#3145)
Follow-up to my previous CMake PR
https://github.com/valkey-io/valkey/pull/2816.

**Changes:**

1. **`.github/workflows/ci.yml`** - Removed symlinks, use
`./build-release/runtest` instead of `./runtest`
2. **`tests/support/set_executable_path.tcl`** - Added
`::VALKEY_TLS_MODULE` variable
3. **Fixed hardcoded paths in 5 test files:**
   - `tests/unit/tls.tcl` - server and TLS module paths
   - `tests/unit/fuzzer.tcl` - benchmark path
   - `tests/unit/cluster/cli.tcl` - CLI path
   - `tests/support/server.tcl` - TLS module path
   - `tests/instances.tcl` - TLS module path

**Result:** All tests passed. The only failure was an unrelated flaky
test (`client-eviction.tcl`) that's been failing since TLS was added to
the cmake job - tracked in issue #3146.

---------

Signed-off-by: Zhijun <dszhijun@gmail.com>
2026-02-02 14:58:57 +01:00
BinbinandGitHub a1c6f89e09 Extend hashtableGetStatsMsg to always show empty table and rehash index (#3142)
It is useful to show hashtable stats even if the table is empty since
in addition to the elements themselves, stats such as the buckets are
still useful. Also in this PR, we added rehash index info in table 0
so that we can see this info when doing the debug.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-02-02 19:54:36 +08:00
BinbinandGitHub 91fa5f088d Check expiration time first before lookup the key in GETEX command (#3116)
We should check expiration time value before we lookup the key,
otherwise we will return nil in this wrong case:
```
> getex key ex abcd
(nil)
> set key value
OK
> getex key ex abcd
(error) ERR value is not an integer or out of range

> hset foo f v
(integer) 1
> getex foo ex abcdef
(error) WRONGTYPE Operation against a key holding the wrong kind of value
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-02-01 21:39:06 +08:00
Viktor SöderqvistandGitHub 91749349f4 Revert #3088 (#3137)
Reverts "Fix the memory leak in the VM_GetCommandKeysWithFlags function
(#3088)"

This reverts commit d1b4e6bf8b.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-01-30 18:16:19 +01:00
BinbinandGitHub aab59e1cbc Free replica local buffer in async way in dual channel replication (#2948)
In dual channel replication, when the replica abort the replication,
if the local buffer is huge, free it in async way to avoid the latency.
Dual channel replication was introduced in #60.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-01-31 00:43:41 +08:00
Daniil KashapovandGitHub db6e77c38e Add waits in test-ubuntu-reclaim-cache (#3134)
After #3103 time sensitive `test-ubuntu-reclaim-cache` started to fail
because now startup always includes 30ms of calibration of HW clock,
that's why we get this output:

```
Run echo "test SAVE doesn't increase cache"
test SAVE doesn't increase cache
2460491776
Could not connect to Valkey at 127.0.0.1:8080: Connection refused
```

Added waits for server to start, locally run, it helps

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
2026-01-30 15:05:13 +01:00
BinbinandGitHub 4054a248f7 Remove empty list dead code condition in lmoveGenericCommand code (#3128)
Since commit 0dc63bc473, we will no
longer load empty lists from the RDB.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-01-30 10:17:42 +08:00
eifrah-awsandGitHub 6766d5e744 Fix Lua Module Build For MacOS Compatibility (#3123)
This PR fixes various issues with Lua module built on macOS using CMake
/ Apple Clang.

## Summary of changes

The Lua module build configuration has been updated to support macOS in
addition to Linux. Two key changes were made to resolve build issues on
Apple platforms.

First, the valkeylua shared library now explicitly includes `sha1.c` and
`rand.c` source files and links against the fpconv library, ensuring all
required symbols are available at link time.

Second, the --disable-new-dtags linker flag is now conditionally applied
only on Unix systems excluding macOS, since this flag is not supported
by the Apple linker.

In addition, enable position independent code compilation for the fpconv
static library by setting the `POSITION_INDEPENDENT_CODE` property. This
allows the library to be linked into shared libraries and
position-independent executables.

Last, the Lua library filename definition was previously hardcoded to
use the `.so` extension for all platforms. This change conditionally
sets the library name based on the target platform: `.so` for Linux/Unix
systems and `.dylib` for macOS (and other non-Linux Unix systems). This
ensures that the correct dynamic library extension is used when loading
the Lua module at runtime.

** Generated by CodeLite. **

Signed-off-by: Eran Ifrah <eifrah@amazon.com>
2026-01-29 22:39:59 +01:00
dd367ab0eb HSETEX - Always issue keyspace notifications after validation (#3001)
In the original implementation of Hash Field Expiration
(https://github.com/valkey-io/valkey/pull/2089), the HSETEX command was
implemented to report keyspace notifications only for performed changes.
This is mostly aligned with other Hash commands (for example, HDEL will
also not report `hdel` event for items which does not exist)
The HSETEX case is somewhat different and is more like the `HSET` case.
During HSETEX, after the command validations pass, items are ALWAYS
"added" to the object, even though they might not actually be added.
This case is the same for when the hash object is empty or when all the
provided fields do not exist in the object (as reported
[here](https://github.com/valkey-io/valkey/pull/2998))

This PR changes the way `HSETEX` will report keyspace notifications so
that:
1. `hset` notification will ALWAYS be reported if all command
validations pass.
2. `hexpire` will be reported in case the command include an expiration
time (even past time)
3. `hxpired` will be reported in case the provided expiration time is in
the past (or 0)
4. `hdel` will be reported in case the hash exists (or created as part
of the command) and following the command execution it was left empty.
5. we will always return '1' as a return value of tHSETEX command which
passed all validations. Before that we returned 1 only if we applied the
change cross ALL the input fields, so in case some of them did not exist
and a past time was set we would return 0.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Jacob Murphy <jkmurphy@google.com>
2026-01-29 19:23:18 +02:00
Daniil KashapovandGitHub c809aa6ebd Fix mixup of reply-larger-than and request-larger-than (#3126)
This change amends #3086 in which `commandlog-request-larger-than` was
mixed up with `commandlog-reply-larger-than` while checking for config
values and in tests.

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
2026-01-29 15:46:13 +01:00
Daniil KashapovandGitHub e5776ae5b7 Skip active io time test if valgrind is used (#3124)
If valgrind is used then test for used_active_time_io_thread fails
because valgrind is slow, so skip it

```
[err]: Force the use of IO threads and assert active IO thread usage in tests/unit/io-threads.tcl                                                                                                                     
Expected (7.941596 - 5.876588) < (1000/1000) (context: type eval line 53 cmd {assert {($used_active_time - $initial_active_times($i)) < ($sleep_time_ms/1000)}} proc ::test)
```

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
2026-01-29 12:11:24 +01:00
BinbinandGitHub 5f02732603 Don't exit the process when locale-collate fails to be set from env during startup (#3067)
Before 4757aefe0e, even if setlocale fails, we
won't do anything.
```
setlocale(LC_COLLATE,"");
```

And now we will exit the process if the setlocale fails, this resulted in process
startup failures on some machines. (some kind of breaking change?)
```
if (setlocale(LC_COLLATE,server.locale_collate) == NULL) {
    serverLog(LL_WARNING, "Failed to configure LOCALE for invalid locale name.");
    exit(1);
}
```

In this commit, if we fail to set the locale_collate through environment variables,
we maintain backward compatibility and do not exit.

In a case where we did exit before, we don't exit now, seems safe. It's not a breaking
change and the behavior in this case was undocumented.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-01-29 10:35:05 +08:00
chzhooandGitHub 61cf3a11a6 Optimize skiplist query efficiency by embedding the skiplist header (#2867)
Embedding the skiplist header reduces memory jumps, thus optimizing
sorted set query speed.

```
// Before
typedef struct zskiplist {
        struct zskiplistNode *header, *tail;
        unsigned long length;
        int level; /* Height of the skiplist. */
} zskiplist;


// After
typedef struct zskiplist {
        unsigned long length; 
        struct zskiplistNode *tail;
        /* Since the span at level 0 is always 1 or 0 , 
         * this field is instead used for storing the height of the skiplist. */
        struct zskiplistLevel {
                struct zskiplistNode *forward;
                unsigned long span;
        } level[ZSKIPLIST_MAXLEVEL];
} zskiplist;
```

---------

Signed-off-by: chzhoo <czawyx@163.com>
2026-01-29 01:52:06 +01:00
Jim BrunnerandGitHub 9cce267650 decouple lru/lfu from server.h (#2928)
Addresses: https://github.com/valkey-io/valkey/issues/2908

This decouples the low-level LRU/LFU code from the high-level `server.h`
file.
* LRU/LFU specific config values were removed from `server.h` and
relocated to `lrulfu.h`
* A new LRULFU API was created to update the time (and policy) rather
than accessing globals from `server.`

This doesn't address the root of the original test failure in that
`server.h` may compile with different offsets (due to `off_t`) based on
the include order. See:
https://github.com/valkey-io/valkey/issues/2908#issuecomment-3647452825

Signed-off-by: Jim Brunner <brunnerj@amazon.com>
2026-01-28 22:20:41 +02:00
Benson-liandGitHub ffefa720ef Fix potential memory leaks when a writable replica is promoted to primary after direct writes of keys with expiration (#2953)
The bug mentioned in this
https://github.com/valkey-io/valkey/issues/2952 has been fixed.

---------

Signed-off-by: li-benson <1260437731@qq.com>
2026-01-28 22:10:38 +02:00
Yang ZhaoandGitHub dafa73164a Fail fast on invalid certificates at TLS config load (#2999)
This PR adds the certificates validation at TLS load, rejects invalid
(expired/not-yet-valid) certificates:

Apply to all TLS config paths:
- Server certificates `tls-cert-file`
- Server-side client certificates `tls-client-cert-file`
- CA certificate file `tls-ca-cert-file` 
- CA certificate directory `tls-ca-cert-dir` (now eagerly loaded to be
consistent with file-based CAs)

Apply to both scenarios:
- Server startup (initial TLS load)
- Runtime reload vis `CONFIG SET`

### Implementation
- Added `isCertValid` function to check if an X509 certificate is within
its validity period (not expired, not future-dated)
- Added `areAllCaCertsValid` function to iterate through all loaded CA
certificates and validate them
- Added `loadCaCertDir` function to eagerly load all certificates from a
directory into the X509_STORE
- Modified `createSSLContext` to validate:
  - Server/client certificates immediately after loading
  - All CA certificates after loading from file/directory

### Test results

#### 1. Server startup (initial TLS load)
```
tls-cert-file ./tests/tls/server-expired.crt

41522:M 31 Dec 2025 16:13:18.851 # Server TLS certificate is invalid. Aborting TLS configuration.
41522:M 31 Dec 2025 16:13:18.851 # Failed to configure TLS. Check logs for more info.


tls-client-cert-file ./tests/tls/client-expired.crt

41557:M 31 Dec 2025 16:14:43.296 # Client TLS certificate is invalid. Aborting TLS configuration.
41557:M 31 Dec 2025 16:14:43.296 # Failed to configure TLS. Check logs for more info.


tls-ca-cert-file ./tests/tls/ca-expired.crt
tls-ca-cert-dir ./tests/tls/ca-expired

41567:M 31 Dec 2025 16:15:15.635 # One or more loaded CA certificates are invalid. Aborting TLS configuration.
41567:M 31 Dec 2025 16:15:15.635 # Failed to configure TLS. Check logs for more info.
```

#### 2. Runtime reload via CONFIG SET
```
127.0.0.1:6379> config set tls-cert-file ./tests/tls/server-expired.crt
(error) ERR CONFIG SET failed (possibly related to argument 'tls-cert-file') - Unable to update TLS configuration. Check server logs.

62975:M 02 Jan 2026 20:10:43.588 # Server TLS certificate is invalid. Aborting TLS configuration.
62975:M 02 Jan 2026 20:10:43.588 # Failed applying new configuration. Possibly related to new tls-cert-file setting. Restoring previous settings.


127.0.0.1:6379> config set tls-client-cert-file ./tests/tls/client-expired.crt
(error) ERR CONFIG SET failed (possibly related to argument 'tls-client-cert-file') - Unable to update TLS configuration. Check server logs.

62975:M 02 Jan 2026 20:10:57.972 # Client TLS certificate is invalid. Aborting TLS configuration.
62975:M 02 Jan 2026 20:10:57.972 # Failed applying new configuration. Possibly related to new tls-client-cert-file setting. Restoring previous settings.


127.0.0.1:6379> config set tls-ca-cert-file ./tests/tls/ca-expired.crt
127.0.0.1:6379> config set tls-ca-cert-dir ./tests/tls/ca-expired
(error) ERR CONFIG SET failed (possibly related to argument 'tls-ca-cert-file') - Unable to update TLS configuration. Check server logs.

62975:M 02 Jan 2026 20:10:50.175 # One or more loaded CA certificates are invalid. Aborting TLS configuration.
62975:M 02 Jan 2026 20:10:50.175 # Failed applying new configuration. Possibly related to new tls-ca-cert-file setting. Restoring previous settings.
```

---------

Signed-off-by: Yang Zhao <zymy701@gmail.com>
2026-01-28 17:50:35 +01:00
Daniil KashapovandGitHub 5d87428857 Enable HW clock by default (#3103)
On some Intel systems it was observed that TSC `ticksPerMicrosecond`
derived from `/proc/cpuinfo` was different from calibrated
`ticksPerMicrosecond` or kernel derived one, so it was decided to always
calibrate and look for `constant_tsc` flag.

Resolves #2597

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
2026-01-28 12:48:06 +01:00
Ran ShidlansikandGitHub 48ca797a40 deflake HSETEX EXAT single field expires leaving other fields intact (#3120)
The test is currently flakey, because until we merge:
https://github.com/valkey-io/valkey/pull/3001
when the expiration time provided is in the past, and the field does not
exist the HSETX will just silently ignore it, without incrementing the
statistics.

I prefer to focus on writing a dedicated test for:
https://github.com/valkey-io/valkey/pull/3001 and deflake this test now.

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2026-01-28 10:47:44 +02:00
Ran ShidlansikandGitHub fefba1254a remove unneeded include of expire.h (#3117)
This caused some unaligned load of server values on 32bit compilations.

Example:
https://github.com/valkey-io/valkey/actions/runs/21352969942/job/61491923381?pr=3111#step:6:3064

Some insights here:
https://github.com/valkey-io/valkey/pull/3111#issuecomment-3805700959

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2026-01-27 17:09:58 +02:00
Deepak NandihalliandGitHub 0bd54c47af Instrumenting used active time metrics for I/O threads (#2463)
I/O threads when active do busy polling for work. This puts the CPU time
near 100% per thread even when they aren't fully utilized. Tracking the
time an IO thread spends doing useful work vs waiting for work helps
understand the utilization of individual threads.

INFO fields:

    used_active_time_io_thread_%d

... where %d is the IO thread ID number (between 1 and the number of I/O
threads).

To get a percentage over a period of time, compare the delta of this
metric against a delta of the server's uptime or the INFO field
`server_time_usec`.

---------

Signed-off-by: Deepak Nandihalli <deepak.nandihalli@gmail.com>
2026-01-27 05:19:05 +01:00
Daniil KashapovandGitHub 8c663ff6c9 Perf: Track net bytes only if commandlog-request-larger-than != -1 (#3086)
Currently after #2652 in copy avoidance path we unconditionally track
`c->net_output_bytes_curr_cmd` even when `commandlog-request-larger-than
-1`. This PR provides ability to skip that accounting in copy avoidance
path based on config value. If `commandlog-request-larger-than -1` then
performance is same as before #2652.

Also added tracking of `c->net_output_bytes_curr_cmd` in IO thread if it
is not tracked in main thread based on decision made in main thread.

Read Performance (write performance is the same):
```
With this change:
Summary:
  throughput summary: 2191732.75 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        1.720     0.072     1.743     1.919     2.647    23.983

Unstable: 
Summary:
  throughput summary: 1658197.25 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        2.299     0.120     2.343     2.503     3.319     4.791

Config:
commandlog-request-larger-than -1
^^ without this performance is just like on unstable branch
databases 1
save ""
appendonly no
rdbcompression no
activedefrag no
maxclients 1000
io-threads 9
protected-mode no
hz 10
maxmemory 2gb
```

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
2026-01-26 19:47:23 +01:00
a41a6dc95a Fix how hash is handling overriding of expired fields overwrite (#3060)
There are currently several issues with the existing hash field
expiration mechanism:
1. `HINCRBY` is propagated to the replica "as-is". It mean it relies on
the fact that the state of the hash is the same on the primary and the
replica. HFE did change this assumption as the field might be expired
only when the replica will handle the propagated `hincrby`. the problem
is that the replica does not "expire" fields by it's own. it needs to
respect the request from the primary and always try to use the existing
field. This can lead to either miss-alignment with the value on the
primary and the replica AND even a disconnection since the replica might
hold and "expired" field which is not in "integer" format...
2. HINCRBYFLOAT is currently ALWAYS propagating `hset` - this means that
the expiration time of an entry will always be removed on the replica
side (it needs to propagate HSETEX when expiration time needs to be
maintained)
3. Currently all hash write commands which are mutating values might
overwrite an expired field. In such cases the existing implementation
will "silently" do so. The problem is that the user will not get any
key-space-notificaiton explaining the reason for the behavior. For
example, when `hincrby` is issued overwriting an expired field which was
not yet "cleaned" by active-expiration it will reset the counter to '0'
before incrementing it. this means that the user might ask: why is the
value '1' and not bigger, "I did not see any notification that the old
value expired"...
4. HSETEX with KEEPTTL suffers from a "somewhat" similar problem as
#(1). the replica will receive the propagated command, but will not know
if the primary "replaced" the entry which is expired now but might not
have been expired when the primary applied it.

There are 2 options for a solution:
1. we could propagate `hdel` for every entry we are "overwritting"
(batch them if we can)
2. propagate the commands "by effect". For example - have `hincrby`
always propagate either HSET or HSETEX. This will not solve the '#(4)'
problem above though, for which we might HAVE to propagate `hdel`

I tend to go with the second option. The reason is that it is expected
to have less impact on replication stream and should include less
processing time on the replicas and network traffic. Specifically for
HSETEX with KEEPTTL we will have to propagate the `hdel` in case we
overwritten an expired field, but that would help limit the impact of
this propagation.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Sourav Singh Rawat <aidenfrostbite@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2026-01-26 10:10:32 +02:00
BinbinandGitHub a7c026b080 Comment cleanup around entry.c file (#3107)
Fix some outdated comments, improved some comments.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-01-25 20:08:53 +08:00
705209899a makefile formatting: split lists onto multiple lines for better readability and easier merging (#3015)
These lists are getting longer recently, and they're annoying to edit. I
recently did a lot of rebasing and got tired of it. 😅

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-01-23 21:30:41 +01:00
chzhooandGitHub 39e8a0510c Improve performance during rehashing (#3073)
As the hashtable grows, rehash is triggered. Rehash consumes a
significant amount of CPU time, resulting in noticeable performance
degradation. Much of this CPU time is spent on random memory access
(when accessing hashtable entries). This CPU usage can be optimized
through concurrent memory access.

```
//  The original rehash process:
For each bucket and its child buckets, all entries are processed as follows:
    1) Access the entry, which involves random memory access.
    2) Compute the hash value of the entry.
    3) Based on the hash value, insert the entries into the new hashtable.

//  The optimized rehash process:
The original steps are transformed into batch processing and phased execution, leveraging
the CPU's multi-issue capability to achieve concurrent memory access. The steps are as follows:
    1) Access multiple entries serially within a for loop. Since each entry access is independent,
       modern CPU architectures can perform concurrent memory access.
    2) Compute the hash values for all entries.
    3) Based on the hash values, insert the entries into the new hashtable.
```

---------

Signed-off-by: chzhoo <czawyx@163.com>
2026-01-23 17:53:06 +01:00
Madelyn OlsonandGitHub 07cbe44e93 CI: Make CMake test also run tls tests (#3097)
We are already double running the tests with CMake, and we are building
CMake with TLS, so just making it so we run the tests with TLS. This
seems like an simple update so that we are always running the TLS tests.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2026-01-23 17:50:05 +01:00
Yang ZhaoandGitHub a7b164f0e7 Fix unit test for TLS auto reload (#3094)
Signed-off-by: Yang Zhao <zymy701@gmail.com>
2026-01-23 12:17:48 +01:00
Vitah LinandGitHub 428ef8f573 Fix minor inconsistency in connFormatAddr size argument (#3098)
Minor fix to use the right var.

Signed-off-by: Vitah Lin <vitahlin@gmail.com>
2026-01-23 15:19:29 +08:00
93a30fae02 Fix XREAD returning error on empty stream with + ID (#2742)
When using XREAD STREAMS <stream> + on an empty stream created with
MKSTREAM, valkey returns an error instead of nil.

This happens because is missing a check on the stream length.

The fix adds the length check on the condition.

Fixes #2728

Signed-off-by: diego-ciciani01 <diego.ciciani@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-01-22 11:17:10 +01:00
d1b4e6bf8b Fix the memory leak in the VM_GetCommandKeysWithFlags function (#3088)
Memory leak in the VM_GetCommandKeysWithFlags function when
MAX_KEYS_BUFFER is reached.
We will malloc the memory so we should always free the getKeysResult.
```
    /* Resize if necessary */
    if (numkeys > result->size) {
        if (result->keys != result->keysbuf) {
            /* We're not using a static buffer, just (re)alloc */
            result->keys = zrealloc(result->keys, numkeys * sizeof(keyReference));
        } else {
            /* We are using a static buffer, copy its contents */
            result->keys = zmalloc(numkeys * sizeof(keyReference));
            if (result->numkeys) memcpy(result->keys, result->keysbuf, result->numkeys * sizeof(keyReference));
        }
        result->size = numkeys;
    }
```

Closes #3087.

Signed-off-by: arshidkv12 <arshidkv12@gmail.com>
Signed-off-by: Arshid <arshidkv12@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2026-01-22 13:36:54 +08:00
Sarthak AggarwalandGitHub 70de2dc863 Fixes run-extra-tests workflow to run on the PR commit instead of unstable (#3092)
Thanks madolson for pointing out the issue with
https://github.com/valkey-io/valkey/pull/2907.

Whenever we were using the label, it was picking up the head of the
unstable instead of the PR. The reason was that we changed the target to
`pull_request_target`. We had to do that since we wanted to remove the
label after the run is completed.

With this change, it correctly picks up the commit hash of the PR and
checks it out.

To test this, I merged the changes in my forked repository's
[unstable](https://github.com/valkey-io/valkey/compare/unstable...sarthakaggarwal97:valkey:unstable),
created a dummy
[PR](https://github.com/sarthakaggarwal97/valkey/pull/61), and was able
to verify that the commit of the PR is being checked out in the
[action](https://github.com/sarthakaggarwal97/valkey/actions/runs/21229598167/job/61084986422?pr=61#step:3:81).

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-01-21 15:52:53 -08:00
Viktor SöderqvistandGitHub f8605d8d7a Skip MPTCP test on platforms without MPTCP (#3089)
The test case added in #3042 fails on platforms without MPTCP support.

This change automatically skips the MPTCP tests on platforms without
MTPCP.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-01-21 20:32:57 +01:00
Ran ShidlansikandGitHub 90a692fb23 Fix HRANDFIELD to return null response when no field could be found (#3022)
if the hrandfield(e.g. hrandfield myhash) command without other args
does not find a valid field, it will return an uninitialized lval.

```
debug set-active-expire no
hsetex myhash ex 1 fields 2 f1 v1 f2 v2
after 1s...
hrandfield myhash
[will return some uninitialized number]
```

related: https://github.com/valkey-io/valkey/issues/3021

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2026-01-21 13:34:03 +02:00
Sarthak AggarwalandGitHub df09422fec Fix weekly workflow to continue after failure in releases branches (#3082)
Currently, the weekly runs do not progress if there is a failed workflow
as github CI treats `fail-fast` to be true by default. With this change,
we continue to test all the branches even after failure.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-01-21 15:33:00 +08:00
Yang ZhaoandGitHub 36bc697baf Automatic TLS reload (#3020)
### Overview
This PR adds support for automatic background TLS reloading, closes
https://github.com/valkey-io/valkey/issues/2649
TLS validity checks and fail-fast behavior on invalid certificates are
handled separately in #2999.
- New configuration
  - `tls-auto-reload-interval <seconds>`
  - `0` disabled (default, backward compatible)
  - `>0` check interval in seconds
- TLS materials change detection in background
  - SHA-256 fingerprint checking for certificate files
- `inode + mtime` checking for CA certificate directories and key files
- Skips reload if materials haven't changed
`tlsCheckMaterialsAndUpdateCache`
- TLS contexts reload
- CPU-intensive certificate parsing happens in dedicated BIO worker
thread `BIO_TLS_RELOAD`
  - Main thread never blocks, atomically swaps SSL contexts
- Two-phase reload: background preparation `tlsConfigureAsync` + main
thread application `tlsApplyPendingReload`


**Note**: Original TLS load and reload still remain in main thread using
`tlsConfigureSync`, including:
- Initial TLS load (server startup)
- Runtime reload via CONFIG SET


---------

Signed-off-by: Yang Zhao <zymy701@gmail.com>
2026-01-20 22:23:49 -08:00
Jim BrunnerandGitHub 1f77f7dac0 improve format for hashtableDump (#3081)
Simple change to `hashtableDebug` to improve readability:
* Improve indentation
* Remove "(empty)" lines

Signed-off-by: Jim Brunner <brunnerj@amazon.com>
2026-01-21 00:14:44 +01:00
Ping XieandGitHub 492bed8e00 Add copilot review instructions (#3076)
Signed-off-by: Ping Xie <pingxie@outlook.com>
2026-01-20 14:57:19 -08:00
yzc-yzcandGitHub 127a9f531e Fix valkey-benchmark hanging in single-thread duration mode (#3079)
In single-thread mode with --duration, the benchmark could hang
indefinitely after the duration elapsed.

## Root Cause
Consider the following scenario: 

**Step 1** (T = 0.99s, duration = 1s):
- `readHandler` receives response, calls `clientDone()`
- `isBenchmarkFinished()` returns `false` (0.99s < 1s)
- `clientDone()` calls `resetClient()`

**Step 2** (T = 1.01s):
- `writeHandler` is called
- `isBenchmarkFinished()` returns `true` (1.01s >= 1s)
- `writeHandler` returns early, **no request sent, no read event
registered**

**Result**:
- `readHandler` never triggers → `clientDone()` never called → no
`aeStop()`
- `showThroughput()` checks `num_threads && isBenchmarkFinished()`,
skipping single-thread mode
- **Event loop never terminates**

Fixes #2893, fixes #2843

---------

Signed-off-by: yzc-yzc <96833212+yzc-yzc@users.noreply.github.com>
2026-01-20 18:30:50 +01:00
wxmzy88andGitHub 923f663c8d Add test for MPTCP (#3042)
Signed-off-by: wangxiaomeng <wangxiaomeng@kylinos.cn>
2026-01-20 14:58:57 +01:00
BinbinandGitHub 10d0599e85 Fix race condition in TIME overflow test (#3074)
As we can see, it is entirely possible for the microseconds to have a
value of 0:
```
*** [err]: The microsecond part of the TIME command will not overflow in tests/unit/introspection-2.tcl
Expected '0' to be more than '0' (context: type eval line 4 cmd {assert_morethan $microseconds 0} proc ::test)
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-01-19 20:56:33 +08:00
BinbinandGitHub fbe386cad0 Cleanup and break the while loop in clusterPrimariesHaveReplicas (#3075)
This is a minor cleanup, remove the useless replicas var and we
can break the loop ASAP.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-01-19 20:56:22 +08:00
BinbinandGitHub bede7fa6ec Update dual channel replication conf to mention primary should also enable repl-diskless-sync (#3051)
We should mention it in valkey.conf otherwise people may got it wrong.

Also, when the primary node receives a REPLCONF CAPA dual-channel but
repl-diskless-sync is not enabled (we will ignore the capa in this case),
we will also print a notice message so that people can be aware of it from
the logs. We believe if a user enables dual-channel-replication-enabled,
they intend to use dual channel replication.

Closes #3044.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-01-19 10:20:25 +08:00
yzc-yzcandGitHub aa02c88cb2 Fix flaky tests related to deferring client timing issues (#3071)
Fix flaky tests in `slot-stats.tcl` and `introspection.tcl`.

Both tests share a similar flaky issue - when using a deferring client,
after `$rd <command>` returns, we **cannot** guarantee that the server
has already executed the command.


#### 1. "CLUSTER SLOT-STATS network-bytes-in, in-line buffer processing"

Add `$rd read` after `$rd flush` to wait for the server response,
ensuring the command is fully processed before checking slot stats.

#### 2. "MONITOR log blocked command only once"

Skip any leading `info` commands from MONITOR output before asserting
`blpop`. Since we cannot guarantee command execution order from the
deferring client's perspective, `info` commands triggered by
`wait_for_blocked_clients_count` may appear before the expected `blpop`.

Signed-off-by: yzc-yzc <96833212+yzc-yzc@users.noreply.github.com>
2026-01-19 10:11:06 +08:00
1a335db9ce Cumulative metric for active main thread usage (#2931)
The metric tracks the number of seconds the main thread spends doing
active work as supposed to waiting for work. The active time is exposed
as the INFO field `used_active_time_main_thread` in the CPU section.

When I/O threads are used, the main thread uses a busy-loop when it's
waiting for work, so the reported CPU usage is near 100% even if the
thread has capacity to handle more work. This new metric attempts to
provide a useful metric of how loaded the main thread is by excluding
the time the thread is just waiting for work.

The busy loop consists of the cycle (beforeSleep, non-blocking epoll,
afterSleep). Only the duration of the loops that handle at least one
event loop event (network, file or timer event) or some work from I/O
threads (execution of commands received from clients by I/O threads) is
counted as active time.

Implements the main thread part of #2065.

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-01-16 10:39:05 +01:00
f9aa95b58b Add ValkeyModule_ClusterKeySlotC (#2984)
Add VM_ClusterKeySlotC that takes a data pointer and len as parameters.

Purpose: A module can avoid creating a ValkeyModuleString in a hot code
path.

Closes #2901

---------

Signed-off-by: Su Ko <rhtn1128@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-01-16 10:15:26 +01:00
fa0be3a565 Fix used_memory_dataset underflow due to miscalculated used_memory_overhead (#3005)
The metric `used_memory_dataset` turned into an insanely large number close to 2^64 (actually
overflowed negative value), as reported in #2994.

## Double-Counted database memory

When server starts, the global variable `server.initial_memory_usage` is used to record a memory
baseline in InitServerLast. This `server.initial_memory_usage` has clearly included initial database
memory, since databases are created in initServer.

In function getMemoryOverheadData, the `mem_total` is firstly assigned the baseline, which includes
initial database memory. And then all extra memory usage of databases are added to mem_total. The initial
database memory are therefore counted TWICE.

This eventually caused wrongly larger `used_memory_overhead`. For a database with only a couple of keys,
the `used_memory_overhead` is easily larger than `used_memory` and causes an overflowed `used_memory_dataset`.

## Missed Empty Databases

In function getMemoryOverheadData(), kvstores without any allocated hashtable are ignored from calculation:
```c
if (db == NULL || !kvstoreNumAllocatedHashtables(db->keys)) continue;
```

However, even the kvstore has no allocated hashtable, there are still some memory allocated by kvstoreCreate(),
including `hashtable_size_index`, which can be larger than 128 KiB.

On the contrary, this caused wrongly smaller `used_memory_overhead` for an empty database. When we insert only
ONE key to the database, the database is suddenly taken into account, and `used_memory_overhead` will increase
(for `used_memory_dataset` decrease) by more than 128 KiB due to the single key insertion.

Signed-off-by: Ace Breakpoint <chemistudio@gmail.com>
Signed-off-by: bpint <chemistudio@gmail.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2026-01-15 15:46:07 +08:00
5e243cef70 Fix HEXPIRE should not delete items when validation rules fail and expiration is in the past (#3048)
https://github.com/valkey-io/valkey/pull/3023 was only partially solving
the problem.
We need to avoid expiring items in case of any validation rule failed.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2026-01-14 06:17:18 +02:00
4b559e8b98 Avoid duplicate calculations of network-bytes-out in slot stats with copy-avoidance (#3046)
In releaseBufReferences(), clusterSlotStatsAddNetworkBytesOutForSlot() is called in the IO
thread after writing the reply to the client, and (since #2652) it's also done in the normal
code path in the main thread (afterCommand() -> clusterSlotStatsAddNetworkBytesOutForUserClient()).
This means the output bytes are recorded twice in the cluster slot stats.

Fixes #3043. Bug was introduced in #2652 (9.0.1).

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-01-14 11:25:52 +08:00
Viktor SöderqvistandGitHub 7b6d122ea2 Delete large binary junk file valkey-microbench (#3055)
Introduced by mistake in #2807.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-01-13 13:50:51 +01:00
Sarthak AggarwalandGitHub 94d2af8273 Fixing Weekly Tests Workflow on Released Branches (#3045)
The patch fixes the error by adding `pull‑requests: write` to the
permissions block of `weekly.yml`. Github rejects if the reusable
workflow's (here daily.yml) permissions are not provided.

We recently changed the permissions in `daily.yml` where we gave write
permissions in https://github.com/valkey-io/valkey/pull/2907
With this change, we bring parity to the permissions since `weekly.yml`
uses calls `daily.yml` workflow call method.

Fixes: https://github.com/valkey-io/valkey/actions/runs/20890545320

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-01-12 15:00:59 -08:00
Sarthak AggarwalandGitHub 268eeac32a Reset maxmemory after OOM scripting tests (#3047)
This change restores maxmemory to 0 at the end of both OOM scripting
tests, preventing memory configuration from leaking into subsequent
tests.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-01-12 14:59:56 -08:00
BinbinandGitHub 707398be9d Don't stop the dirty scripts when an OOM occurs midway through execution (#3030)
Before #2858, we has this check:
```
static int scriptVerifyOOM(scriptRunCtx *run_ctx, char **err) {
    if (run_ctx->flags & SCRIPT_ALLOW_OOM) {
        /* Allow running any command even if OOM reached */
        return C_OK;
    }

    /* If we reached the memory limit configured via maxmemory, commands that
     * could enlarge the memory usage are not allowed, but only if this is the
     * first write in the context of this script, otherwise we can't stop
     * in the middle. */

    if (server.maxmemory &&                          /* Maxmemory is actually enabled. */
        !mustObeyClient(run_ctx->original_client) && /* Don't care about mem for replicas or AOF. */
        !(run_ctx->flags & SCRIPT_WRITE_DIRTY) &&    /* Script had no side effects so far. */
        server.pre_command_oom_state &&              /* Detected OOM when script start. */
        (run_ctx->c->cmd->flags & CMD_DENYOOM)) {
        *err = sdsdup(shared.oomerr->ptr);
        return C_ERR;
    }

    return C_OK;
}
```

If we reached the memory limit configured via maxmemory, commands that
could enlarge the memory usage are not allowed, but only if this is the
first write in the context of this script, otherwise we can't stop
in the middle.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2026-01-12 17:25:06 +00:00
Kyle J. DavisandGitHub 6e70608433 Makes CLUSTER KEYSLOT available in standalone mode (#3040)
**The problem/use-case that the feature addresses**

I teach Valkey to people frequently. The general process is to start on
standalone Valkey (or 'try me' Valkey) to get started. However, in the
more advanced sections, I introduce the concepts of required to develop
for a Valkey Cluster. The key learning here is to understand slots (e.g.
multi-key operations, using hash tags, etc.). Standing up a cluster is a
hassle for someone just learning, and, while I usually demo `CLUSTER
KEYSLOT` for the users, they can't run that command on their single
instance since all of the `CLUSTER` commands are blocked. I think Valkey
students would learn a lot more quickly if they could experiment with
`CLUSTER KEYSLOT` themselves in standalone mode.

In this case, `CLUSTER KEYSLOT` doesn't require anything in the cluster
- it just does the math on the key to determine the slot number.

**Description of the feature**

This feature unblocks `CLUSTER KEYSLOT` on single instances.

**Alternatives you've considered**

- Having a cluster on try-me valkey (too many resources)
- Having single instance clusters (discussed before, but far more
complex than this trivial change)

**Additional information**

Their might be less repetitious logic, but the entire bits that
generates `CLUSTER KEYSLOT` result is just 3 lines, so repetitious feels
fine?

---------

Signed-off-by: Kyle J. Davis <kyledvs@amazon.com>
2026-01-12 09:13:00 -08:00
Madelyn OlsonandGitHub 965b49e23a Initial draft of contributing guide (#1787) 2026-01-11 22:20:45 -08:00
Ran ShidlansikandGitHub 867aa723a4 Fix HEXPIRE should not delete items when GT rule is used and expiration is in the past (#3023)
if an expired time is used, the condition is ignored, and it directly
becomes the effect of the `hdel` command.
This is mainly important for cases where the user would like to protect
deleting the data in case the `HEXPIREAT` command is used and the user
would like to protect delay execution of this command to delete fields.

fixes: https://github.com/valkey-io/valkey/issues/3021

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2026-01-11 07:49:44 +02:00
Aditya TeltiaandGitHub 841aaee23d Skip slot cache optimization for AOF client to prevent key duplication and data corruption (#3004)
When loading AOF in cluster mode, keys inside a MULTI/EXEC block could
be
inserted into wrong hash slots, causing key duplication and data
corruption.

The root cause was the slot caching optimization in getKeySlot(). This
optimization reuses a cached slot value to avoid recalculating the hash
for every key operation. However, when replaying AOF, a transaction may
contain commands affecting keys in different slots. The cached slot from
a previous command (e.g., SET k1) would incorrectly be used for
subsequent
commands in the transaction (e.g., SET k0), causing k0 to be stored in
k1's
slot.

The existing code already skipped this optimization for replicated
clients
(commands from primary) using isReplicatedClient(). This change extends
that to also skip for AOF clients by using mustObeyClient() instead,
which
covers both replicated clients and the AOF client.

Fixes #2995, introduced in #1949.

Signed-off-by: aditya.teltia <teltia.aditya22@gmail.com>
2026-01-09 16:54:51 -08:00
Zhijun LiaoandGitHub 6d5f8cb8e8 Add logging helper function to print node's ip:port when nodename not explicitly set (#2777)
By default, servers in a cluster don't have human-readable nodename
unless explicitly set by user. Most logging statements in the cluster
are in the form of `serverLog(LL_NOTICE, "Node %.40s (%s) did
something..", node->name, node->human_nodename, ...);` , and thus this
would always result in an empty `()` like the following:

```
* Node dd4f43d9e4a1a2875a514733c08d4870c21db682 () is now a replica of node 1b5d2a4865c04aaa1ccf06db66f22a043411ded7 () in shard d3fc557d12ab3c5a8850c7b0c9afb252b473cad1
* Clear FAIL state for node dd4f43d9e4a1a2875a514733c08d4870c21db682 (): replica is reachable again.
* NODE 524978a9924b5841c4c93f83581d202cfbce1443 () possibly failing.
* FAIL message received from 747e0f97a58cd5a7ac4c2130417fc47a807802ed () about 1b5d2a4865c04aaa1ccf06db66f22a043411ded7 ()
```

**This PR adds a helper function `humanNodename` specifically for
logging purpose that can print the node's ip:port when its nodename is
not explicitly set.** Now the logs can greatly help developers to
quickly recognize which node is doing stuff:

```
* NODE 03b3811da7b33451f6e4988295b0158f410d2c00 (127.0.0.1:7001) possibly failing.
* Marking node 03b3811da7b33451f6e4988295b0158f410d2c00 (127.0.0.1:7001) as failing (quorum reached).
* NODE b2f0fc93632d12c06039335b025365d484ff9049 (127.0.0.1:7002) possibly failing.
* Marking node b2f0fc93632d12c06039335b025365d484ff9049 (127.0.0.1:7002) as failing (quorum reached).
# Cluster state changed: fail
```

There’s no behavior change in cluster state, gossip, failover, etc, and
**the human nodename of each node is not changed**. This PR is for
logging only and is safe:

- For developers who never cared about the empty (), this is just a
small quality-of-life improvement: suddenly the logs tell them which
node is which, without extra node ID lookups.

- For developers who rely on custom names they prefer, nothing changes –
their explicit nodename takes precedence.

Signed-off-by: Zhijun <dszhijun@gmail.com>
2026-01-09 17:45:31 +08:00
Patrik HermanssonandGitHub 2d7a9b76ce Trigger prepareCommand on argc change in module command filters (#2945)
Previously, only changes to argv[0] were considered when deciding
whether to re-prepare the command and recompute the cluster slot. This
caused issues where changes to the argument count (argc) would not
trigger the necessary prepare in case additional keys were injected to
the command

This fix adds a check to ensure that changes to argc are properly
considered.

Signed-off-by: Patrik Hermansson <phermansson@gmail.com>
2026-01-08 19:27:00 +01:00
Sarthak AggarwalandGitHub 26a77c4e54 Weekly Test Runs on Released Branches (#2702)
Resolves: https://github.com/valkey-io/valkey/issues/2228

Visualization:
https://github.com/sarthakaggarwal97/valkey/actions/runs/19113712295

Currently, there are no tests running on the already released branches.
We often do backport for bug fixes and CVEs in these older versions, and
end up with multiple CI tests failures on these branches.

The PR adds support for running weekly tests on already released
versions `>= 7.2`. The workflow will execute the "daily" test workflow
for each of these branches on `Sunday 06:00 UTC`.

The idea is to continuously monitor our released versions through weekly
test runs (during the time when is lesser activity on github runners).

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2026-01-06 13:17:17 -08:00
d188b67dd2 Adding json support for log-format config (#1791)
Introduce a json log format that can be invoked by passing `log-format
json` in your config.

We also refactor `escapeJsonString` to `util` since it is now a shared
function.

Closes: https://github.com/valkey-io/valkey/issues/1006

---------

Signed-off-by: Johan Bergström <bugs@bergstroem.nu>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2026-01-06 22:05:06 +01:00
cjx-zarandGitHub 69d94a58ba HFE make zero a valid ttl during import mode and data loading (#3006)
The HFE uses EXPIRY_NONE(-1) for fields without a TTL. A bug exists in
`HSETEX` and RDB loading where `expiry > 0` is used to check for an
expiration. This is problematic because `0` might be treated as no
expiry in `import-mode`, instead of an already expired timestamp,
leading to incorrect behavior.

---------

Signed-off-by: cjx-zar <jxchenczar@foxmail.com>
2026-01-06 09:50:12 +02:00
Rain ValentineandGitHub 76257a57bf Remove internal server object pointer overhead in small strings (#2516)
Achieved 20-30% reduction in overhead.

This was accomplished by reusing the 8B robj->ptr memory to store
embedded data. For 16B key and 16B value, this reduces the per-item
memory overhead by ~20%. Overall performance was not measurably changed.
Raising the threshold for embedding strings produces ~30% reduction in
per-item overhead for affected value sizes.

Here's what I did:
1. Modify all `robj->ptr` access to use get/set methods. Most of this
was done with a script (code listed below), with a few manual touches.
Manual and programmatic changes are in separate commits to make review
easier.
2. Next I changed `objectGetVal()` to calculate the location instead of
using `robj->ptr`, and modified the embedding code to start writing
embedded data 8B earlier, overwriting the ptr location. I changed
`objectSetVal()` to assert that no value is embedded - all of the code
that assigned `o->ptr` would have been incorrect for embedded strings
anyway. (Except for one instance in module.c which I made a separate
method for.)

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
2026-01-05 18:02:54 -08:00
bandalgomsuandGitHub 8652dbbed8 Refactor clusterMsg type casting for improve type safety (#2986)
Introduced `clusterMsgHeader` and `toClusterMsg()` `toClusterMsgLight()`
helpers to unify clusterMsg parsing and make casting explicit.
switched packet read/validation/processing paths to use the common
16-byte header first, then cast to clusterMsg/clusterMsgLight only after
determining message type.
this is a refactor to improve readability and avoid accidental
miscasts between full vs light headers.

issue : #2837

---------

Signed-off-by: Su Ko <rhtn1128@gmail.com>
2026-01-05 11:16:55 -08:00
skyfireleeandGitHub 55ea9ef7b9 fix asprintf unused result warn (#2987)
Use the `lm_asprintf `method to solve the warm problem:
```
debug_lua.c: In function ‘ldbEval’:
debug_lua.c:470:13: warning: ignoring return value of ‘asprintf’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
  470 |             asprintf(&err_msg, "Error compiling code: %s", lua_tostring(lua, -1));
      |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

---------

Signed-off-by: skyfirelee <739609084@qq.com>
2026-01-05 11:20:51 +00:00
201464f07c replace info cluster info in a dedicated new section (#2964)
In https://github.com/valkey-io/valkey/pull/2876 we added cluster info
stats to the info command. However these statistics where added to the
"cluster" section which is part of the default info sections. Since the
calculation of the cluster info is done "by-demand" it can lead to high
server side CPU consumption when many clients are using the "info"
command on a large cluster.

In this PR we separated the cluster info to a dedicated "cluster_stats"
section, which is not enabled by default.

Alternatives considered:

1. Revert https://github.com/valkey-io/valkey/pull/2876 - this is
possible as the information is available via `CLUSTER INFO` command. The
only downside is that some "control plane" adaptations need to be placed
in order to gather this information in addition to the "already
collected" server wide `INFO` data.
2. Optimize cluster info status collection - We could dynamically update
and maintain these statistics during cluster topology state changes. The
main "expensive" statistics which are needed to be collected are:
slots_assigned, slots_ok, slots_pfail, slots_fail, nodes_pfail,
nodes_fail, voting_nodes_pfail, voting_nodes_fail
We could track and constantly update these during cluster pings. The
main downside is potential bugs and bad accounting which might be caused
by additional complexity and unhandled edge cases.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
2026-01-05 12:35:36 +02:00
50886055e1 Untrack key based on old->hasembkey (#3007)
In `dbSetValue()` the `old` pointer may be reassigned to point to the
incoming value object which was created without an embedded key, so
calling `dbUntrackKeyWithVolatileItems()` would call `objectGetKey()`
which returns NULL, causing a crash in `hashtableSdsHash()` when trying
to hash the NULL key.

Idea is to assign `old_was_hash_with_volatile` before the swap and use
`new` instead of `old` for untracking when theres no embedded key.

Introduced in #3003 
Run with NULL ptr dereference:
https://github.com/valkey-io/valkey/actions/runs/20701343184/job/59424029880

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2026-01-05 11:45:34 +02:00
Sourav Singh RawatandGitHub da8004956e fix(hash): Untrack hash with volatile fields when it is overwritten (#3003)
In `dbSetValue` if the robj is a hash with volatile fields we need to
clean it up. It causes two core problems:
1. Type Confusion: Allows converting from hash to string with
`keys_with_volatile_items` tracked
2. Tracked Items with dangling pointer: Converting from `hash` to ie. a
string breaks this as `keys_with_volatile_items` is not cleaned up.

Check #3000

---------

Signed-off-by: frostzt <aidenfrostbite@gmail.com>
2026-01-04 20:23:11 +02:00
b0300cec94 Add sdsnsplitargs to reduce sds allocation. (#2975)
`sdssplitargs` relies on a string ending in `\0` and requires the
parameter to be of type sds. To avoid sds allocation, the
`sdsnsplitargs` method with len is supported.

---------

Signed-off-by: lizhiqiang.sf <lizhiqiang.sf@bytedance.com>
Signed-off-by: skyfirelee <739609084@qq.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2026-01-04 20:18:51 +02:00
Madelyn OlsonandGitHub 380d031240 Fix zero length hash creation with HSETEX (#2998)
If you send an `HSETEX key EXAT 0 FIELDS 1 foo bar`, it will create an
empty length string. This is not good behavior as other places in the
code will crash if an empty hash is accessed.

This change just makes it so that the key is freed if it ends up being
empty. This has some odd behavior w.r.t. to keyspace notifications
though.

Today, if you do `HSETEX key EXAT 0 FIELDS 1 foo bar` with a hash that
exists, it won't send an `HSET` notification or an `HEXPIRE`
notification unless foo is already present. It will also send a `DEL`
notification if foo is last the field, effectively deleting the key.

This isn't consistent with Redis, which will still send an `HSET` and
`HDEL` (*NOTE* not `hexpire`) notification if a key is added past the
expiration. Likewise, in the case where the key doesn't exist and
`HSETEX key EXAT 0 FIELDS 1 foo bar` is sent, Redis will send an HSET,
HDEL, and DEL notification in that order.

I think it makes sense to keep consistency with what we have today
(given that it's been out for awhile), but document the behavior.
Basically, if you send a command with an expiration in the past, we will
basically ignore the input unless it's "deleting" a key.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2026-01-04 15:33:08 +02:00
skyfireleeandGitHub bcf6a9959b Avoid useless seek in xtrim (#2985)
`xtrim` on a stream with a length less than `maxlen` and exits directly
to avoid useless seek operations

Signed-off-by: lizhiqiang.sf <lizhiqiang.sf@bytedance.com>
2026-01-04 09:50:37 +08:00
1eb7575039 fix(HSETEX): replace strcmp with strcasecmp (#3000)
HSETEX right now uses `strcmp` which does not account for
case-insensitivity replaced it with `strcasecmp`.
Fixes #2996

---------

Signed-off-by: frostzt <aidenfrostbite@gmail.com>
Signed-off-by: Sourav Singh Rawat <aidenfrostbite@gmail.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2026-01-03 20:18:39 +02:00
c5b839ec24 Skip sorting empty lists in sortCommandGeneric to prevent undefined behavior when calling pqsort (#2425)
This one was found by
[afl++](https://github.com/AFLplusplus/AFLplusplus). Executing `sort key
by *` with a non-existing key, `vectorlen` gets initially set to `0` which results
in end being set to -1.  Since we're operating on an empty list here, we can just
skip the entire sorting step altogether.

---------

Signed-off-by: Fusl <fusl@meo.ws>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2026-01-02 18:41:48 -08:00
c7e3397e52 Add FIFO and mutexQueue with bio.c refactored (#2895)
## Overview:

This PR introduces two data structures, FIFO and mutexQueue, as
mentioned in #2878 , along with a significant refactoring of the
existing bio utility that demonstrates their practical value.

New Creative Components:
* FIFO - A high-performance queue that's significantly more space and
time efficient than adlist.
* mutexQueue - A thread-safe wrapper around FIFO that encapsulates
synchronization primitives into a reusable abstraction. This creative
design eliminates repetitive mutex/condition operations throughout the
codebase.

Refactored Components:
* `bio.c` - refactored using mutexQueue, with **56** lines of code
eliminated while achieving more maintainable code.

These prepare the prerequisite data structures for bgIteration.

## Details:

1. FIFO - `fifo.h`, `fifo.c`, `test_fifo.c` 

2. mutexQueue - `mutexqueue.h`, `mutexqueue.c`, `test_mutexqueue.c`

3. `bio.c` Refactoring:

Before: Manual management of 3 synchronization primitives per queue
      After: Single mutexQueue abstraction handles everything

      Key improvements:

* Simplified synchronization - mutexQueue encapsulates mutex locking and
condition variable signaling, eliminating manual coordination in:

         - `bioInit()`
         - `bioSubmitJob()` 
         - `bioProcessBackgroundJobs()` 
         - `bioPendingJobsOfType()`

* Thread-safe counters - Replaced manual locking of `bio_jobs_counter`
with atomic operations

* `bioDrainWorker()` now uses polling on the queue abstraction instead
of managing low-level synchronization


## Testing:
* Unit tests for FIFO in `test_fifo.c`: all pass, 
`COMPARE_PERFORMANCE_TO_ADLIST`shows the **improvement = 78.69% (List:
122 ms, FIFO: 26 ms)**
* Unit tests for mutexQueue in `test_mutexqueue.c:` all pass
* `bio.c` functionality preserved with refactored implementation

---------

Signed-off-by: Alina Liu <liusalisa6363@gmail.com>
Co-authored-by: Jim Brunner <brunnerj@amazon.com>
2026-01-01 20:03:46 +02:00
uriyageandGitHub 4515594fb4 Valkey Fuzzer (#2340)
## Add Fuzzing Capability to Valkey

### Overview
This PR adds a fuzzing capability to Valkey, allowing developers and
users to stress test their Valkey deployments with randomly generated
commands. The fuzzer is integrated with the existing valkey-benchmark
tool, making it easy to use without requiring additional dependencies.

### Key Features
• **Command Generator**: Automatically generates Valkey commands by
retrieving command information directly from the server
• **Two Fuzzing Modes**:
- normal: Generates only valid commands, doesn't modify server
configurations
- aggressive: Includes malformed commands and allows CONFIG SET
operations

• **Multi-threaded Testing**: Each client runs in a dedicated thread to
maximize interaction between clients and enable testing of complicated
scenarios

• **Integration with valkey-benchmark**: Uses the existing CLI interface

### Implementation Details
• Added new files:
- `fuzzer_command_generator.h/c`: Dynamically generates valkey commands.
- `fuzzer_client.c`: Orchestrate all the client threads, report test
progress, and handle errors.

• Modified existing files:
  - valkey-benchmark.c: Added fuzzing mode options and integration

### Command Generation Approach
The fuzzer dynamically retrieves command information from the server,
allowing it to adapt to different Valkey versions and custom modules.
Since the command information generated from JSON files is sometimes
limited, not all generated commands will be valid, but approximately 95%
valid command generation is achieved.

It is important to generate valid commands to cover as much code path as
possible and not just the invalid command/args path. The fuzzer
prioritizes generating syntactically and semantically correct commands
to ensure thorough testing of the server's core functionality, while
still including a small percentage of invalid commands in `aggressive`
mode to test error handling paths

#### Config modification
For CONFIG SET command, the situation is more complex as the server
currently provides limited information through CONFIG GET *. Some
hardcoded logic is implemented that will need to be modified in the
future. Ideally, the server should provide self-inspection commands to
retrieve config keys-values with their properties (enum values,
modifiability status, etc.).

### Issue Detection
The fuzzer is designed to identify several types of issues:
• Server crashes
• Server memory corruptions / memory leaks(when compiled with ASAN)
• Server unresponsiveness
• Server malformed replies

For unresponsiveness detection, command timeout limits are implemented
to ensure no command blocks for excessive periods. If a server doesn't
respond within 30 seconds, the fuzzer signals that something is wrong.

### Proven Effectiveness
When running against the latest unstable version, the fuzzer has already
identified several issues, demonstrating its effectiveness:
* https://github.com/valkey-io/valkey/issues/2111
* https://github.com/valkey-io/valkey/issues/2112
* https://github.com/valkey-io/valkey/pull/2109
* https://github.com/valkey-io/valkey/pull/2113
* https://github.com/valkey-io/valkey/pull/2108
* https://github.com/valkey-io/valkey/pull/2137
* https://github.com/valkey-io/valkey/issues/2106
* https://github.com/valkey-io/valkey/pull/2347
* https://github.com/valkey-io/valkey/pull/2973
* https://github.com/valkey-io/valkey/pull/2974
### How to Use
Run the fuzzer using the valkey-benchmark tool with the --fuzz flag:

```bash
# Basic usage (10000 commands 1000 commands per client, 10 clients)
./src/valkey-benchmark --fuzz -h 127.0.0.1 -p 6379 -n 10000 -c 10

# With aggressive fuzzing mode
./src/valkey-benchmark --fuzz --fuzz-level aggressive -h 127.0.0.1 -p 6379 -n 10000 -c 10

# With detailed logging
./src/valkey-benchmark --fuzz --fuzz-log-level debug -h 127.0.0.1 -p 6379 -n 10000 -c 10
```

The fuzzer supports existing valkey-benchmark options, including TLS and
cluster mode configuration.

---------

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
2026-01-01 14:45:52 +02:00
Ran ShidlansikandGitHub 2ee8170043 avoid memory leak of new argv when hexpire commands target only non-exiting fields (#2973)
When HEXPIRE commands are set with a time-in-the-past they are all
deleting the specified fields.
In such cases we allocate a temporal new argv in order to replicate
`HDEL`.
However in case no mutation was done (ie all fields do not exist) we do
not deallocate the temporal new_argv and there is a memory leak.

example:

```
HSET myhash field1 value1
1
HEXPIRE myhash 0 FIELDS 1 field2
-2
```

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-12-31 09:10:25 +02:00
Ran ShidlansikandGitHub d296fb51ee fix hincrby* update volatile key tracking (#2974)
Following Hash-Field-Expiration feature, a hash object can hold volatile
fields.
volatile fields which are already expired are deleted and reclaimed ONLY
by the active-expiration background job.
This means that hash object can contain items which have not yet
expired.
In case mutations are requesting to set a value on these
"already-expired" fields, they will be overwritten with the new value.
In such cases, though, it is requiered to update the global per-db
tracking map by removing the key if it has no more volatile fields.
This was implemented in all mutation cases of the hash commands but the
`INCRBY` and `INCRBYFLOAT`.
This can lead to a dangling object which has no volatile items, which
might lead to assertion during the active-expiration job:

example reproduction:
```
DEBUG SET-ACTIVE-EXPIRE 0
hset myhash f1 10
hexpire myhash 1 FIELDS 1 f1
sleep(10)
hincrby myhash f1 1
DEBUG SET-ACTIVE-EXPIRE 1
```   

NOTE: we actually had tests for this scenario, only the test did not
include explicit assertion in case the item is still tracked after the
mutation.

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-12-29 15:30:59 +02:00
BinbinandGitHub afa15a769c Fix chained replica crash when doing dual channel replication (#2983)
There is a crash in freeReplicationBacklog:
```
Discarding previously cached primary state.
ASSERTION FAILED
'listLength(server.replicas) == 0' is not true
freeReplicationBacklog
```

The reason is that during dual channel operation, the RDB channel is protected.
In the chained replica case, `disconnectReplicas` is called to disconnect all
replica clients, but since the RDB channel is protected, `freeClient` does not
actually free the replica client. Later, we encounter an assertion failure in
`freeReplicationBacklog`.
```
void replicationAttachToNewPrimary(void) {
    /* Replica starts to apply data from new primary, we must discard the cached
     * primary structure. */
    serverAssert(server.primary == NULL);
    replicationDiscardCachedPrimary();

    /* Cancel any in progress imports (we will now use the primary's) */
    clusterCleanSlotImportsOnFullSync();

    disconnectReplicas();     /* Force our replicas to resync with us as well. */
    freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */
}
```

Dual channel replication was introduced in #60.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-12-29 10:20:24 +08:00
Ran ShidlansikandGitHub 9d900abe29 Revert hgetall dynamic deferred response (#2981)
This is to fix a regression introduced in
https://github.com/valkey-io/valkey/pull/2966.
Example failed run:
https://github.com/valkey-io/valkey/actions/runs/20495981119/job/58895602847

Currently reverting the commit which introduced the regression

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-12-28 13:09:56 +02:00
skyfireleeandGitHub 4fcccc11bc Fix compile overflow warnings (#2963)
Compilation warning in eval.c:244 when extracting shebang flags - 
attempting to allocate 18446744073709551615 bytes (SIZE_MAX) due to 
unsigned integer underflow.
```
eval.c: In function ‘evalExtractShebangFlags’:
eval.c:244:27: warning: argument 1 value ‘18446744073709551615’ exceeds maximum object size 9223372036854775807 [-Walloc-size-larger-than=]
  244 |             *out_engine = zcalloc(engine_name_len + 1);
      |                           ^
zmalloc.c:256:7: note: in a call to allocation function ‘valkey_calloc’ declared here
  256 | void *zcalloc(size_t size) {
      |       ^
cd modules/lua && make OPTIMIZATION="-O3 -flto=auto -ffat-lto-objects -fno-omit-frame-pointer"
```

Signed-off-by: lizhiqiang.sf <lizhiqiang.sf@bytedance.com>
2025-12-26 18:36:08 +08:00
cjx-zarandGitHub c5efe912ad Restrict ttl from being negative and avoid crash in import-mode (#2944)
When import-mode is yes, we might be able to set an expired TTL. At the
same time,
commands like EXPIREAT/EXPIRE do not restrict TTL from being negative.
After we
set import-mode to no, server will crash at:
```
 int activeExpireCycleTryExpire(serverDb *db, robj *val, long long now, int didx) { 
     long long t = objectGetExpire(val); 
     serverAssert(t >= 0); 
```

In this case, we restrict ttl from being negative in
expireGenericCommand, we simply
change the expiration time to 0 to mark the key as expired since in
import-mode, the
import-source client can always read the expired keys anyway.

import-mode was introduced in #1185

---------

Signed-off-by: cjx-zar <jxchenczar@foxmail.com>
2025-12-26 10:20:39 +08:00
Ran ShidlansikandGitHub 03cf3a68d8 Avoid hgetall deferred response (#2966)
In https://github.com/valkey-io/valkey/pull/2089 we added a deferred
logic for HGETALL since we cannot anticipate the size of the output as
it may contain expired hash items which should not be included.
As part of the work of https://github.com/valkey-io/valkey/issues/2022
this would greatly increase the time for HGETALL processing, thus we
introduce this minor improvement to avoid using deferred reply in case
the hash has NO volatile items.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-12-24 13:38:17 +02:00
Roshan KhatriandGitHub 5d012bc7bc Adds links to performance dashboards in README (#2969)
Adds links to performance dashboards in README:
1. [Performance Overview](https://valkey.io/performance/)
2. [Unstable Branch
Dashboard](https://perf-dashboard.valkey.io/public-dashboards/3e45bf8ded3043edaa941331cd1a94e2)

Unstable dashboard is helpful in to identify potential regressions like
these: https://github.com/valkey-io/valkey/issues/2926

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2025-12-23 13:45:57 -08:00
Sarthak AggarwalandGitHub c3b7d6dd4b Run Extra Tests with only a label (#2907)
Change the behaviour of the CI job triggered by the run-extra-tests
label.

Run the tests immediately when applying the run-extra-tests label to a
PR, without requiring an extra commit to be pushed to trigger the test
run.

When the extra tests have run, the job removes the label.

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-12-23 22:13:43 +01:00
Daniil KashapovandGitHub 0c9e04ce19 Database-level access control (#2309)
## API changes and user behavior:

- [x] Default behavior for database access.

Default is `alldbs` permissions.


### Database Permissions (`db=`)
- [x] Accessing particular database

```
> ACL SETUSER test1 on +@all ~* resetdbs db=0,1 nopass
"user test1 on nopass sanitize-payload ~* resetchannels db=0,1 +@all"
```

- [x] (Same behavior without usage of `resetdbs`)
```
> ACL SETUSER test1 on +@all ~* db=0,1 nopass
"user test1 on nopass sanitize-payload ~* resetchannels db=0,1 +@all"
```

- [x] Multiple selector can be provided
```
> ACL SETUSER test1 on nopass (db=0,1 +@write +select ~*) (db=2,3 +@read +select ~*)
"user test1 on nopass sanitize-payload resetchannels alldbs -@all (~* resetchannels db=0,1 -@all +@write +select) (~* resetchannels db=2,3 -@all +@read +select)"
```

- [x] Restricting special commands which access databases as part of the
command.

The user needs to have access to both the commands and db(s) part of the
command to run these commands.

1. SWAPDB
2. SELECT
3. MOVE - (Select command would have went through for the source
database). Have access for the target database.
4. COPY


- [x] Restricting special commands which doesn't specify database
number, however, accesses multiple databases.

The user needs to have access to both the commands and all databases
(`alldbs`) to run these commands.

1. FLUSHALL - Access all databases
2. CLUSTER commands that access all databases:
    - CANCELSLOTMIGRATIONS
    - MIGRATESLOTS

- [x] New connection establishment behavior
New client connection gets established to DB 0 by default.
Authentication and authorisation are decoupled and the user can
connect/authenticate and further perform `SELECT` or other operation
that do not access keyspace.

(Do we want to extend HELLO?) Alternative suggestion by @madolson:
Extend `HELLO` command to pass the dbid to which the user should get
connected after authentication if they have right set of permission. I
think it will become a long poll for adoption.

- [x] Observability
Extend `ACL LOG` to log user which received denied permission error
while accessing a database.

- [x] Module API
* Introduce module API `int VM_ACLCheckPermissions(ValkeyModuleUser
*user, ValkeyModuleString **argv, int argc, int dbid,
ValkeyModuleACLLogEntryReason *denial_reason);`
* Stop support of `VM_ACLCheckCommandPermissions()`.

Resolves: #1336

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
2025-12-22 19:36:34 -08:00
Jacob MurphyandGitHub b7328dcf52 Add support for Atomic Slot Migration to CLI (#2755)
Adds a new option `--cluster-use-atomic-slot-migration`. This will apply
to both `--cluster reshard` and `--cluster rebalance` commands.

We could do some more optimizations here, but for now we batch all the
slot ranges for one (source, target) pair and send them off as one
`CLUSTER MIGRATESLOTS` request. We then wait for this request to finish
through polling `CLUSTER GETSLOTMIGRATIONS` once every 100ms. We parse
`CLUSTER GETSLOTMIGRATIONS` and look for the most recent migration
affecting the requested slot range, then check if it is in progress,
failed, cancelled, or successful. If there is a failure or cancellation,
we give this error to the user.

Fixes #2504

---------

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-12-22 15:10:51 -08:00
JohnandGitHub 19ba870e1e Fix incorrect comment about STATS_METRIC_* Macro in server.h (#2935)
`read to` should be `read from`

Signed-off-by: John <johnufida@163.com>
2025-12-22 12:17:09 +08:00
52c3908e5a Fail to save the cluster config file will not exit the process (#1032)
In clusterSaveConfigOrDie, we will exit directly when saving
fails. In the case of disk failure, the cluster node will exit
immediately if there are some changes around the cluster.
We call it in many places, mainly in clusterBeforeSleep, that is,
when the cluster configuration changes and we need to save.

Passive exit may bring unexpected effects, such as cluster down.
We think the risk of metadata becoming persistently out of date
is minimal. On the one hand, we have the CLUSTER_WRITABLE_DELAY
logic, which prevents a master node from being rejoined to the
cluster in an unsafe case within 2 seconds. 
```
void clusterUpdateState(void) {
    /* If this is a primary node, wait some time before turning the state
     * into OK, since it is not a good idea to rejoin the cluster as a writable
     * primary, after a reboot, without giving the cluster a chance to
     * reconfigure this node. Note that the delay is calculated starting from
     * the first call to this function and not since the server start, in order
     * to not count the DB loading time. */
    if (first_call_time == 0) first_call_time = mstime();
    if (clusterNodeIsPrimary(myself) && server.cluster->state == CLUSTER_FAIL &&
        mstime() - first_call_time < CLUSTER_WRITABLE_DELAY)
        return;
```

The remaining potentially worse case is that the node votes twice
in the same epoch. Like we didn't save nodes.conf and the we have
voted for replica X. We reboot and during this time X wins the failover.
After reboot, node Y requests vote for the same epoch and we vote for Y.
Y wins the failover with the same epoch. We have two primaries with
the same epoch. And we get an epoch collissions. It is resolved and some
writes are lost. It's just like a failover, some writes can be lost.
It may be very rare and it is not very bad. We will use the same 
CLUSTER_WRITABLE_DELAY logic to make an optimistic judgment and prevent
the node from voting after restarting.

Added a new clusterSaveConfigOrLog, if the save fails, instead of exiting,
we will now just print a warning log. We will replace the clusterSaveConfigOrDie
in clusterBeforeSleep with this. That is, the config save triggered by
beforeSleep now will not exit the process even if the save fails.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-12-22 10:35:54 +08:00
3141a866be Enforce 64‑bit off_t regardless of include order; prevent LTO type mismatch (#2943)
Fixes #2938
- Root cause: On 32-bit builds, off_t depended on include order.
lrulfu.c included <stdint.h>/<stdbool.h> via lrulfu.h before server.h,
so _FILE_OFFSET_BITS=64 (from fmacros.h) was not in effect when glibc
headers were first seen. This made off_t 32-bit in that TU, while 64-bit
elsewhere, causing LTO linker “type mismatch” warnings and possible
misoptimization.
- Changes:
1. Include fmacros.h first in src/lrulfu.h to ensure feature macros
apply before any system header.
          2. Add a compile-time check in src/server.h:
static_assert(sizeof(off_t) >= 8, "off_t must be 64-bit; ensure
_FILE_OFFSET_BITS=64 is in effect before system headers");
             so we fail fast if include order regresses.
- Why this works: fmacros.h defines _FILE_OFFSET_BITS 64 and related
feature macros. Ensuring it is seen first gives a consistent 64-bit
off_t across all TUs. The static_assert
turns future include-order mistakes into early compile-time failures
instead of link-time notes/warnings.
- Testing:
- Built on 32-bit Debian: no LTO type-mismatch at link, binaries
produced successfully. Only GCC 11 ABI notes about _Atomic alignment
(“note: … changed in GCC 11.1”), which are informational (-Wpsabi) and
do not affect correctness.
- Risk: very low; only header include order + a defensive assert. No
runtime changes.
- Address CI feedback: add fmacros.h to unit sources that include
headers before server.h;

---------

Signed-off-by: Ada-Church-Closure <2574094394@qq.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Ada-Church-Closure <2574094394@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-12-21 23:55:58 +01:00
Daniil KashapovandGitHub 7cdcb04336 Propagate database changes from scripting engine temp client (#2959)
**Problem:** temp client does not preserve db between VM_Call() calls.
SELECT changes the database context of a temporary client created for
that specific call. However, this database change is not propagated back
to the context client, so subsequent commands in the same script will
execute in the wrong database.
Behaviour on unstable:
```
127.0.0.1:6379> eval "server.call('SELECT', '1'); return server.call('SET', 'lua_test', 'incorrect')" 0
OK
127.0.0.1:6379> select 0
OK
127.0.0.1:6379> get lua_test
"incorrect"
127.0.0.1:6379> select 1
OK
127.0.0.1:6379[1]> get lua_test
(nil)
```

Behaviour with fixes:
```
127.0.0.1:6379> eval "server.call('SELECT', '1'); return server.call('SET', 'lua_test', 'correct')" 0
OK
127.0.0.1:6379> select 0
OK
127.0.0.1:6379> get lua_test
(nil)
127.0.0.1:6379> select 1
OK
127.0.0.1:6379[1]> get lua_test
"correct"
```

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
2025-12-20 20:22:23 +00:00
Deepak NandihalliandGitHub 55efee7e66 Fix memory allocation size in getClusterNodesList (#2932)
Fixed incorrect memory allocation in getClusterNodesList in
src/cluster_legacy.c. Changed zmalloc((count + 1) * CLUSTER_NAMELEN) to
zmalloc((count + 1) * sizeof(char *)) to correctly allocate memory for
an array of pointers.

Signed-off-by: Deepak Nandihalli <deepak.nandihalli@gmail.com>
2025-12-19 14:06:09 -08:00
Ricardo DiasandGitHub a5dc48535c Fix lua module build failure when using CLANG (#2955)
This commit fixes the build of the lua module when using CLANG to
compile the code. When building with clang and with LTO enabled, the lua
module build was failing in the linking phase of the shared library.

The problem was solved by using the LLVM linker, instead of the GNU
linker, to link the lua module shared library.

We also fix, in this commit, some compiler warnings that were being
generated when building with clang.

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-12-19 14:28:06 +00:00
0a07bad120 Adding support for sharing memory between the module and the engine (#2472)
## Overview

Sharing memory between the module and engine reduces memory overhead by
eliminating redundant copies of stored records in the module. This is
particularly beneficial for search workloads that require indexing large
volumes of documents.

### Vectors

Vector similarity search requires storing large volumes of
high-cardinality vectors. For example, a single vector with 512
dimensions consumes 2048 bytes, and typical workloads often involve
millions of vectors. Due to the lack of a memory-sharing mechanism
between the module and the engine, valkey-search currently doubles
memory consumption when indexing vectors, significantly increasing
operational costs. This limitation introduces adoption friction and
reduces valkey-search's competitiveness.

## Memory Allocation Strategy

At a fundamental level, there are two primary allocation strategies:
- [Chosen] Module-allocated memory shared with the engine.
- Engine-allocated memory shared with the module.

For valkey-search, it is crucial that vectors reside in cache-aligned
memory to maximize SIMD optimizations. Allowing the module to allocate
memory provides greater flexibility for different use cases, though it
introduces slightly higher implementation complexity.

## Old Implementation

The old [implementation](https://github.com/valkey-io/valkey/pull/1804)
was based on ref-counting and introduced a new SDS type. After further
discussion, we
[agreed](https://github.com/valkey-io/valkey/pull/1804#issuecomment-2905115712)
to simplify the design by removing ref-counting and avoiding the
introduction of a new SDS type.

## New Implementation - Key Points

1. The engine exposes a new interface, `VM_HashSetViewValue`, which set
value as a view of a buffer which is owned by the module. The function
accepts the hash key, hash field, and a buffer along with its length.
2. `ViewValue` is a new data type that captures the externalized buffer
and its length.


## valkey-search Usage

### Insertion
1. Upon receiving a key space notification for a new hash or JSON key
with an indexed vector attribute, valkey-search allocates cache-aligned
memory and deep-copies the vector value.
2. valkey-search then calls `VM_HashSetViewValue` to avoid keeping two
copies of the vector.
### Deletion
When receiving a key space notification for a deleted hash key or hash
field that was indexed as a vector, valkey-search deletes the
corresponding entry from the index.

### Update
Handled similarly to insertion.

---------

Signed-off-by: yairgott <yairgott@gmail.com>
Signed-off-by: Yair Gottdenker <yairg@google.com>
Signed-off-by: Yair Gottdenker <yairgott@gmail.com>
Co-authored-by: Yair Gottdenker <yairg@google.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Jim Brunner <brunnerj@amazon.com>
2025-12-19 15:55:57 +02:00
Viktor SöderqvistandGitHub 1dcd74ce6c Module API docs formatting and improve script (#2950)
* A formatting error where there was a line break in the wrong place in
a code example in a doc comment (used in the generated API docs). The
error was introduced in an automatic code formatting commit.
* Improve API doc generation script by considering release candidates
when detecting "since" for each API function. This makes it possible to
run the script on a release candidate to have the docs ready before a GA
release.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-12-19 00:11:40 +01:00
Ricardo DiasandGitHub 02dbdaaac2 Move Lua scripting engine into a Valkey module (#2858)
This PR restructures the Lua scripting functionality by extracting
it from the core Valkey server into a separate Valkey module. This
change enables the possibility of a backwards compatible Lua engine
upgrade, as well as, the flexibility in building Valkey without the
Lua engine.

**Important**: from a user's point of view, there's no difference in
using the `EVAL` of `FUNCTION/FCALL` scripts. This PR is fully backward
compatible with respect to the public API.

The main code change is the move and adaptation of the Lua engine source
files from `src/lua` to `src/modules/lua`. The original Lua engine code is
adapted to use the module API to compile and execute scripts.

The main difference between the original code and the new, is the
serialization and deserialization of Valkey RESP values into, and from,
Lua values. While in the original implementation the parsing of RESP
values
was done directly from the client buffer, in the new implementation the
parsing is done from the `ValkeyModuleCallReply` object and respective
API.

The Makefile and CMake build systems were also updated to build and
integrate the new Lua engine module, within the Valkey server build
workflow.
When the Valkey server is built, the Lua engine module is also built,
and, the Lua module is loaded automatically by the server upon startup.
When running `make install` the Lua engine module is installed in the
default system library directory.
There's a new build option, called `BUILD_LUA`, that if set to `no`
allows to
build Valkey server without building the Lua engine.

This modular architecture enables future development of additional Lua
engine modules with newer Lua versions that can be loaded alongside the
current engine, facilitating gradual migration paths for users.

Additional change: Unload all modules on shutdown (ignoring modules that
can't be unloaded). This is to avoid address sanitizer warnings about
leaked allocations.

Fixes: #1627

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-12-18 14:55:14 +00:00
Viktor SöderqvistandGitHub 3fdf355f80 Respect REPLCONF VERSION in diskless full sync (#2735)
Provide an RDB snapshot that the replica can handle, for resilience
during rolling upgrades.

This permits an older replica to do a full sync from a newer primary.
The primary takes the replica's announced version into account when
generating the snapshot. In particular, it allows latest Valkey to send
snapshots as RDB 11 to replicas running Valkey 7.2 and 8.x.

The implementation is structurally similar to how filtered snapshots
with REPLCONF RDB-FILTER-ONLY works (commits e1e12c1f1c, 2b4727e58b) and
to the feature negotiation that replicas initiate using REPLCONF CAPA.

If any new features that the replica can't handle (such as hash-field
expiration and atomic slot migration) are in use, the full sync is
aborted and the replica connection is closed.

This mechanism will allow us to do RDB changes more often. In the recent
years, we have been avoiding RDB changes. With this mechanism, there is
no need to avoid introducing RDB changes such as new encodings and new
compression algorithms (#1962).

In my experience, providing a way to undo an upgrade makes users less
worried and actually more willing to upgrade. This is true not least
when Valkey is a part of a larger system which is upgraded as a whole.
Valkey may be just one microservice of many within a larger system; not
uncommon in on-prem deployments. If anything goes wrong (even if it's
not Valkey itself) the user wants a way to roll back the whole system to
the last working state.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-12-18 11:45:35 +01:00
BinbinandGitHub ba9864c8cc Check for duplicate nodeids when loading nodes.conf (#2852)
For corrupted (human-made) or program-error-recovered nodes.conf files,
check for duplicate nodeids when loading nodes.conf. If a duplicate is
found, panic is triggered to prevent nodes from starting up
unexpectedly.

The node ID is used to identify every node across the whole cluster,
we do not expect to find duplicate nodeids in nodes.conf.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-12-17 11:23:38 +08:00
Ping XieandGitHub 73e5bae92c Update MAINTAINERS list and add committee chair section (#2939) 2025-12-16 14:06:26 -08:00
Ping XieandGitHub 471911ef73 Make 1/3 TSC organization limit explicitly inclusive (#2930) 2025-12-16 07:15:38 -08:00
BinbinandGitHub 4f2b8c80eb Strictly check CRLF when parsing querybuf (#2872)
Currently, when parsing querybuf, we are not checking for CRLF,
instead we assume the last two characters are CRLF by default,
as shown in the following example:
```
telnet 127.0.0.1 6379
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
*3
$3
set
$3
key
$5
value12
+OK
get key
$5
value

*3
$3
set
$3
key
$5
value12345
+OK
-ERR unknown command '345', with args beginning with:
```

This should actually be considered a protocol error. When a bug
occurs in the client-side implementation, we may execute incorrect
requests (writing incorrect data is the most serious of these).

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-12-14 09:10:57 +02:00
65b52c4071 Refine major decision process and update TSC composition rules (#2927)
- Require a 2/3 supermajority vote for all Governance Major Decisions.
- Update Technical Major Decision voting to prioritize simple majority, limiting the use of "+2" approval.
- Define remediation steps for when the 1/3 organization limit is exceeded.

---------

Signed-off-by: Ping Xie <pingxie@outlook.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-12-12 08:53:50 -08:00
Vitah LinandGitHub fe11646473 Upgrade macos version in actions (#2920)
GitHub has deprecated older macOS runners, and macos-13 is no longer supported.

1. The latest version of cross-platform-actions/action does allow
running on ubuntu-latest (Linux runner) and does not strictly require macOS.
2. Previously, cross-platform-actions/action@v0.22.0 used runs-on:
macos-13. I checked the latest version of cross-platform-actions, and
the official examples now use runs-on: ubuntu. I think we can switch from macOS to Ubuntu.

---------

Signed-off-by: Vitah Lin <vitahlin@gmail.com>
2025-12-11 23:50:26 +01:00
Viktor SöderqvistandGitHub 9c5b042d4b Revert "Allow partial sync after loading AOF with preamble (#2366)" (#2925)
This reverts commit 5648115862.

The implementation in #2366 made it possible to perform a partial
resynchronization after loading an AOF file, by storing the replication
offset in the AOF preamble and counting the bytes in the AOF command
stream in the same way as we count byte offset in a replication stream.

However, this approach isn't safe because some commands are replicated
but not stored in the AOF file. This includes the commands REPLCONF,
PING, PUBLISH and module-implemented commands where a module can control
to propagate a command to the replication stream only, to AOF only or to
both. This oversight led to data inconsistency, where the wrong
replication offset is used for partial resynchronization as explained in
issue #2904.

The revert caused small merge conflicts with 1cf1115cca which are
solved.

Fixes #2904.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-12-11 08:39:53 +02:00
Vadym KhoptynetsandGitHub 908d2c5236 Fix a typo in aof-multi-part.tcl (#2922)
A small PR to fix a smal typo: `util` --> `until`

Signed-off-by: Vadym Khoptynets <1099644+poiuj@users.noreply.github.com>
2025-12-09 21:30:28 +02:00
2ca5d348f3 Fix CLUSTER SLOTS crash when called from module timer callback (#2915)
The CLUSTER SLOTS reply depends on whether the client is connected over
IPv6, but for a fake client there is no connection and when this command
is called from a module timer callback or other scenario where no real
client is involved, there is no connection to check IPv6 support on.
This fix handles the missing case by returning the reply for IPv4
connected clients.

Fixes #2912.

---------

Signed-off-by: Su Ko <rhtn1128@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Su Ko <rhtn1128@gmail.com>
Co-authored-by: KarthikSubbarao <karthikrs2021@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-12-09 14:02:35 +01:00
BinbinandGitHub 540d0bfe4d Fix commandlog large-reply when using reply copy avoidance (#2652)
In #2078, we did not report large reply when copy avoidance is allowed.
This results in replies larger than 16384 not being recorded in the
commandlog large-reply. This 16384 is controlled by the hidden config
min-string-size-avoid-copy-reply.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-12-09 10:20:45 +08:00
BinbinandGitHub 016fb4b3c5 Make the COB soft limit also use repl-backlog-size when its value is smaller (#2866)
We have the same settings for the hard limit, and we should apply them to the soft
limit as well. When the `repl-backlog-size` value is larger, all replication buffers
can be handled by the replication backlog, so there's no need to worry about the
client output buffer soft limit in here. Furthermore, when `soft_seconds` is 0, in
some ways, the soft limit behaves the same (mostly) as the hard limit.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-12-08 12:21:34 +08:00
zhaozhao.zzandGitHub 1a722b92dd support whole cluster info for INFO command in cluster section (#2876)
Allow users to more easily get cluster information.

Signed-off-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
2025-12-04 09:34:52 -08:00
Ouri HalfandGitHub 0dba0490ce Fix deadlock in IO thread shutdown during panic (#2898)
## Problem
IO thread shutdown can deadlock during server panic when the main thread
calls `pthread_cancel()` while the IO thread holds its mutex, preventing
the thread from observing the cancellation.

## Solution  
Release the IO thread mutex before cancelling to ensure clean thread
termination.

## Testing
Reproducer:
```
bash
./src/valkey-server --io-threads 2 --enable-debug-command yes
./src/valkey-cli debug panic
```

Before: Server hangs indefinitely
After: Server terminates cleanly

Signed-off-by: Ouri Half <ourih@amazon.com>
2025-12-04 09:10:09 -08:00
Roshan KhatriandGitHub a8be8bc384 Add PR and Release benchmark with new changes in framework (#2871)
This adds the workflow improvements for PR and Release benchmark where
it runs on `c8g.metal-48xl` for `ARM64` and `c7i.metal-48xl` for `X86`.

```
Cluster mode: disabled
TLS: disabled
io-threads: 1, 9
Pipelining: 1, 10
Clients: 1600
Benchmark Treads: 90
Data size: 16 ,96
Commands: SET, GET
```

c8g.metal-48xl Spec: https://aws.amazon.com/ec2/instance-types/c8g/
c7i.metal.48xl Spec: https://aws.amazon.com/ec2/instance-types/c7i/

```
vCPU: 192
NUMA nodes: 2
Memory (GiB): 384
Network Bandwidth (Gbps): 50
```

PR benchmarking will be executed on **ARM64** machine as it has been
seen to be more consistent.
Additionally, it runs 5 iterations for each tests and posts the average
and other statistical metrics like
- CI99%: 99% Confidence Interval - range where the true population mean
is likely to fall
- PI99%: 99% Prediction Interval - range where a single future
observation is likely to fall
- CV: Coefficient of Variation - relative variability (σ/μ × 100%)

_Note: Values with (n=X, σ=Y, CV=Z%, CI99%=±W%, PI99%=±V%) indicate
averages from X runs with standard deviation Y, coefficient of variation
Z%, 99% confidence interval margin of error ±W% of the mean, and 99%
prediction interval margin of error ±V% of the mean. CI bounds [A, B]
and PI bounds [C, D] show the actual interval ranges._

For comparing between versions, it adds a workflow which runs on both
**ARM64** and **X86** machine. It will also post the comparison between
the versions like this:
https://github.com/valkey-io/valkey/issues/2580#issuecomment-3399539615

---------

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Signed-off-by: Roshan Khatri <117414976+roshkhatri@users.noreply.github.com>
2025-12-04 17:33:21 +01:00
0598e81801 Add safe iterator tracking in hashtable to prevents invalid access on hashtable delete (#2807)
This makes it safe to delete hashtable while a safe iterator is
iterating it. This currently isn't possible, but this improvement is
required for fork-less replication #1754 which is being actively
worked on.

We discussed these issues in #2611 which guards against a different but
related issue: calling hashtableNext again after it has already returned
false.

I implemented a singly linked list that hashtable uses to track its
current safe iterators. It is used to invalidate all associated safe
iterators when the hashtable is released. A singly linked list is
acceptable because the list length is always very small - typically zero
and no more than a handful.

Also, renames the internal functions:

    hashtableReinitIterator -> hashtableRetargetIterator
    hashtableResetIterator -> hashtableCleanupIterator

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-12-03 18:15:45 +01:00
Harkrishn PatroandGitHub f5a9e78141 Send duplicate multi meet packet only for node which supports it (#2840)
This prevents crashes on the older nodes in mixed clusters where some
nodes are running 8.0 or older. Mixed clusters often exist temporarily
during rolling upgrades.

Fixes: #2341 

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-12-03 17:17:26 +01:00
eda5487932 Replace strcmp with byte-by-byte comparison in valkey-check-aof (#2809)
Fixes #2792

Replace strcmp with byte-by-byte comparison to avoid accidental
heap-buffer-overflow errors.

Signed-off-by: murad shahmammadli <shmurad@amazon.com>
Co-authored-by: murad shahmammadli <shmurad@amazon.com>
2025-12-03 17:07:04 +01:00
Roshan KhatriandGitHub ecb523caf5 [Test Fix] flaky benchmark test for warmup (#2890)
Fixes: https://github.com/valkey-io/valkey/issues/2859
Increased the warmup to 2 sec so we can verify that it runs more number
of commands than the actual benchmark.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2025-12-02 21:40:20 +01:00
c168096b68 Refactor of LFU/LRU code for modularity (#2857)
General cleanup on LRU/LFU code. Improve modularity and maintainability.

Specifically:
* Consolidates the mathematical logic for LRU/LFU into `lrulfu.c`, with
an API in `lrulfu.h`. Knowledge of the LRU/LFU implementation was
previously spread out across `db.c`, `evict.c`, `object.c`, `server.c`,
and `server.h`.
* Separates knowledge of the LRU from knowledge of the object containing
the LRU value. `lrulfu.c` knows about the LRU/LFU algorithms, without
knowing about the `robj`. `object.c` knows about the `robj` without
knowing about the details of the LRU/LFU algorithms.
* Eliminated `server.lruclock`, instead using `server.unixtime`. This
also eliminates the periodic need to call `mstime()` to maintain the lru
clock.
* Fixed a minor computation bug in the old `LFUTimeElapsed` function
(off by 1 after rollover).
* Eliminate specific IF checks for rollover, using defined behavior for
unsigned rollover instead.
* Fixed a bug in `debug.c` which would perform LFU modification on an
LRU value.

---------

Signed-off-by: Jim Brunner <brunnerj@amazon.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2025-12-02 10:14:33 -08:00
Zhijun LiaoandGitHub c1394a8a6b Build: Support make test when PROG_SUFFIX is used (#2885)
Closes #2883

Support a new environment variable `VALKEY_PROG_SUFFIX` in the test
framework, which can be used for running tests if the binaries are
compiled with a program suffix. For example, if the binaries are
compiled using `make PROG_SUFFIX=-alt` to produce binaries named
valkey-server-alt, valkey-cli-alt, etc., run the tests against these
binaries using `VALKEY_PROG_SUFFIX=-alt ./runtest` or simply using `make
test`.

Now the test with the make variable `PROG_SUFFIX` works well.

```
% make PROG_SUFFIX="-alt"
		...
		...
		CC trace/trace_aof.o
    LINK valkey-server-alt
    INSTALL valkey-sentinel-alt
    CC valkey-cli.o
    CC serverassert.o
    CC cli_common.o
    CC cli_commands.o
    LINK valkey-cli-alt
    CC valkey-benchmark.o
    LINK valkey-benchmark-alt
    INSTALL valkey-check-rdb-alt
    INSTALL valkey-check-aof-alt

Hint: It's a good idea to run 'make test' ;)
% 
% make test                                                                
cd src && /Library/Developer/CommandLineTools/usr/bin/make test
    CC Makefile.dep
    CC release.o
    LINK valkey-server-alt
    INSTALL valkey-check-aof-alt
    INSTALL valkey-check-rdb-alt
    LINK valkey-cli-alt
    LINK valkey-benchmark-alt
Cleanup: may take some time... OK
Starting test server at port 21079
[ready]: 39435
Testing unit/pubsub
```

Signed-off-by: Zhijun <dszhijun@gmail.com>
2025-12-02 15:03:47 +01:00
Zhijun LiaoandGitHub 691494bc79 Refactor TCL reference to support running tests with CMake (#2816)
Historically, Valkey’s TCL test suite expected all binaries
(src/valkey-server, src/valkey-cli, src/valkey-benchmark, etc.) to exist
under the src/ directory. This PR enables Valkey TCL tests to run
seamlessly after a CMake build — no manual symlinks or make build
required.

The test framework accepts a new environment variable `VALKEY_BIN_DIR`
to look for the binaries.

CMake will copy all TCL test entrypoints (runtest, runtest-cluster,
etc.) into the CMake build dir (e.g. `cmake-build-debug`) and insert
`VALKEY_BIN_DIR` into these. Now we can either do
./cmake-build-debug/runtest at the project root or ./runtest at the
Cmake dir to run all tests.

A new CMake post-build target prints a friendly reminder after
successful builds, guiding developers on how to run tests with their
CMake binaries:

```
Hint: It is a good idea to run tests with your CMake-built binaries ;)
      ./cmake-build-debug/runtest

Build finished
```

A helper TCL script `tests/support/set_executable_path.tcl` is added to
support this change, which gets called by all test entrypoints:
`runtest`, `runtest-cluster`, `runtest-sentinel`.

---------

Signed-off-by: Zhijun <dszhijun@gmail.com>
2025-12-02 14:14:28 +01:00
Daniil KashapovandGitHub 95ce8b857d Make all ACL categories explicit in JSON files (#2887)
Resolves #417

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
2025-12-02 13:33:20 +01:00
BinbinandGitHub 0edb15cde8 Handle failed psync log when there is no replication backlog (#2886)
This crash was introduced in #2877, we will crash when there is no
replication backlog.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-12-01 10:20:32 +08:00
BinbinandGitHub 3998d46dba Add psync offset range information to the failed psync log (#2877)
Although we can infer this infomartion from the replica logs,
i think this still would be useful to see this information directly
in the primary logs.

So now we can see the psync offset range when psync fails and then
we can analyze and adjust the value of repl-backlog-size.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-11-29 12:20:34 +08:00
Viktor SöderqvistandGitHub 3b9a58bd27 Fix persisting missing make variables (#2881)
Persist USE_FAST_FLOAT and PROG_SUFFIX to prevent a complete rebuild
next time someone types make or make test without specifying variables.

Fixes #2880

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-11-28 10:35:53 +01:00
BinbinandGitHub ae402b28db Fix discarded-qualifiers warnings reported by fedorarawhide (#2874)
fedorarawhide CI reports these warnings:
```
networking.c: In function 'afterErrorReply':
networking.c:821:30: error: initialization discards 'const' qualifier from pointer target type [-Werror=discarded-qualifiers]
  821 |             char *spaceloc = memchr(s, ' ', len < 32 ? len : 32);
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-11-27 10:39:42 +08:00
Zhijun LiaoandGitHub 7ff5594bdf Cluster: Optimize slot bitmap iteration from per-bit to 64-bit word scan (#2781)
In functions `clusterSendFailoverAuthIfNeeded` and
`clusterProcessPacket`, we iterate through **every slot bit**
sequentially in the form of `for (int j = 0; j < CLUSTER_SLOTS; j++)`,
performing 16384 checks even when only a few bits were set, and thus
causing unnecessary loop overhead.

This is particularly wasteful in function
`clusterSendFailoverAuthIfNeeded` where we need to ensure the sender's
claimed slots all have up-to-date config epoch. Usually healthy senders
would meet such condition, and thus we normally need to exhaust the for
loop of 16384 checks.

The proposed new implementation loads 64 bits (8 byte word) at a time
and skips empty words completely, therefore only performing 256 checks.

---------

Signed-off-by: Zhijun <dszhijun@gmail.com>
2025-11-26 15:52:17 +02:00
b057f5aeec Adds HGETDEL Support to Valkey (#2851)
Fixes this: https://github.com/valkey-io/valkey/issues/2850
Adds support for HGETDEL to Valkey and aligns with Redis 8.0 feature.
Maintains syntax compatibility
Retrieves all the values, and null if fields dont exists and deletes
once retrieved.
```
127.0.0.1:6379> HGETDEL key FIELDS numfields field [ field ... ]
```
```
127.0.0.1:6379> HSET foo field1 bar1 field2 bar2 field3 bar3
(integer) 3
127.0.0.1:6379> HGETDEL foo FIELDS 1 field2
1) "bar2"
127.0.0.1:6379> HGETDEL foo FIELDS 1 field2
1) (nil)
127.0.0.1:6379> HGETALL foo
1) "field1"
2) "bar1"
3) "field3"
4) "bar3"
127.0.0.1:6379>  HGETDEL foo FIELDS 2 field2 field3
1) (nil)
2) "bar3"
127.0.0.1:6379> HGETALL foo
1) "field1"
2) "bar1"
127.0.0.1:6379> HGETDEL foo FIELDS 3 field1 non-exist-field
(error) ERR syntax error
127.0.0.1:6379> HGETDEL foo FIELDS 2 field1 non-exist-field
1) "bar1"
2) (nil)
127.0.0.1:6379> HGETALL foo
(empty array)
127.0.0.1:6379> 
```

---------

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2025-11-26 15:19:53 +02:00
Ran ShidlansikandGitHub 5a131d56fc Align the complexity description for all multi field HFE commands docs (#2875)
When we added the Hash Field Expiration feature in Valkey 9.0, some of
the new command docs included complexity description of O(1) even tough
they except multiple arguments.
(see discussion in
https://github.com/valkey-io/valkey/pull/2851#discussion_r2535684985)
This PR does:
1. align all the commands to the same description
2. fix the complexity description of some commands (eg HSETEX and
HGETEX)

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-11-26 10:31:28 +02:00
Zhijun LiaoandGitHub 1a50d430e8 Additional log information for cluster accept handler and message processing (#2815)
Enhance debugging for cluster logs

[1] Add human node names in cluster tests so that we can easily
understand which nodes we are interacting with:

```
pong packet received from: 92a960ffd62f1bd04efeb260b30fe9ca6b9294ed (R4) from client: :0
node 92a960ffd62f1bd04efeb260b30fe9ca6b9294ed (R4) announces that it is a primary in shard c6d1152caee49a5e70cb4b77d1549386078be603
Reconfiguring node 92a960ffd62f1bd04efeb260b30fe9ca6b9294ed (R4) as primary for shard c6d1152caee49a5e70cb4b77d1549386078be603
Configuration change detected. Reconfiguring myself as a replica of node 92a960ffd62f1bd04efeb260b30fe9ca6b9294ed (R4) in shard c6d1152caee49a5e70cb4b77d1549386078be603
```



[2] Currently there are logs showing the address of incoming
connections:

```
Accepting cluster node connection from 127.0.0.1:59956
Accepting cluster node connection from 127.0.0.1:59957
Accepting cluster node connection from 127.0.0.1:59958
Accepting cluster node connection from 127.0.0.1:59959
```

but we have no idea which nodes these connections refer to. I added a
logging statement when the node is set to the inbound link connection.

```
Bound cluster node 92a960ffd62f1bd04efeb260b30fe9ca6b9294ed (R4) to connection of client 127.0.0.1:59956
```



[3] Add a debug log when processing a packet to show the packet type,
sender node name, and sender client port (this also has the benefit of
telling us whether this is an inbound or outbound link).

```
pong packet received from: 92a960ffd62f1bd04efeb260b30fe9ca6b9294ed (R4) from client: :0
ping packet received from: 92a960ffd62f1bd04efeb260b30fe9ca6b9294ed (R4) from client: 127.0.0.1:59973
fail packet received from: 92a960ffd62f1bd04efeb260b30fe9ca6b9294ed (R4) from client: 127.0.0.1:59973
auth-req packet received from: 92a960ffd62f1bd04efeb260b30fe9ca6b9294ed (R4) from client: 127.0.0.1:59973
```

---------

Signed-off-by: Zhijun <dszhijun@gmail.com>
2025-11-25 17:07:21 -08:00
Leon AnaviandGitHub 1eae649b21 Fix build on 32-bit ARM by only using NEON on AArch64 (#2873)
Only enable `HAVE_ARM_NEON` on AArch64 because it supports vaddvq and
all needed compiler intrinsics.

Fixes the following error when building for machine `qemuarm` using the
Yocto Project and OpenEmbedded:

```
| bitops.c: In function 'popcountNEON':
| bitops.c:219:23: error: implicit declaration of function 'vaddvq_u16'; did you mean 'vaddq_u16'? [-Wimplicit-function-declaration]
|   219 |         uint32_t t1 = vaddvq_u16(sc);
|       |                       ^~~~~~~~~~
|       |                       vaddq_u16
| bitops.c:225:14: error: implicit declaration of function 'vaddvq_u8'; did you mean 'vaddq_u8'? [-Wimplicit-function-declaration]
|   225 |         t += vaddvq_u8(vcntq_u8(vld1q_u8(p)));
|       |              ^~~~~~~~~
|       |              vaddq_u8
```

More details are available in the following log:
https://errors.yoctoproject.org/Errors/Details/889836/

Signed-off-by: Leon Anavi <leon.anavi@konsulko.com>
2025-11-25 21:23:32 +01:00
2d194107ed Add support for asynchronous release to replicaKeysWithExpire on writable replica (#2849)
## Problem
When executing `FLUSHALL ASYNC` on a **writable replica** that has
a large number of expired keys directly written to it, the main thread
gets blocked for an extended period while synchronously releasing the
`replicaKeysWithExpire` dictionary. 

## Root Cause
`FLUSHALL ASYNC` is designed for asynchronous lazy freeing of core data
structures, but the release of `replicaKeysWithExpire` (a dictionary tracking
expired keys on replicas) still happens synchronously in the main thread.
This synchronous operation becomes a bottleneck when dealing with massive
key volumes, as it cannot be offloaded to the lazyfree background thread.

This PR addresses the issue by moving the release of `replicaKeysWithExpire`
to the lazyfree background thread, aligning it with the asynchronous design
of `FLUSHALL ASYNC` and eliminating main thread blocking.

## User scenarios
In some operations, people often need to do primary-replica switches.
One goal is to avoid noticeable impact on the business—like key loss
or reduced availability (e.g., write failures).

Here is the process: First, temporarily switch traffic to writable replicas.
Then we wait for the primary pending replication data to be fully synced
(so primry and replicas are in sync), before finishing the switch. We don't
usually need to do the flush in this case, but it's an optimization that can
be done.

Signed-off-by: Scut-Corgis <512141203@qq.com>
Signed-off-by: jiegang0219 <512141203@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-11-25 19:25:39 +08:00
BinbinandGitHub 83a397c815 Update dual channel replication conf to mention the local buffer is imited by COB (#2824)
After introducing the dual channel replication in #60, we decided in #915
not to add a new configuration item to limit the replica's local replication
buffer, just use "client-output-buffer-limit replica hard" to limit it.

We need to document this behavior and mention that once the limit is reached,
all future data will accumulate in the primary side.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-11-23 23:27:50 +08:00
BinbinandGitHub aaa0236472 Add rdb_transmitted to replstateToString so that we can see it in INFO (#2833)
In dual channel replication, when the rdb channel client finish
the RDB transfer, it will enter REPLICA_STATE_RDB_TRANSMITTED
state. During this time, there will be a brief window that we are
not able to see the connection in the INFO REPLICATION.

In the worst case, we might not see the connection for the
DEFAULT_WAIT_BEFORE_RDB_CLIENT_FREE seconds. I guess there is no
harm to list this state, showing connected_slaves but not showing
the connection is bad when troubleshooting.

Note that this also affects the `valkey-cli --rdb` and `--functions-rdb`
options. Before the client is in the `rdb_transmitted` state and is
released, we will now see it in the info (see the example later).

Before, not showing the replica info
```
role:master
connected_slaves:1
```

After, for dual channel replication:
```
role:master
connected_slaves:1
slave0:ip=xxx,port=xxx,state=rdb_transmitted,offset=0,lag=0,type=rdb-channel
```

After, for valkey-cli --rdb-only and --functions-rdb:
```
role:master
connected_slaves:1
slave0:ip=xxx,port=xxx,state=rdb_transmitted,offset=0,lag=0,type=replica
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-11-21 18:31:31 +08:00
Ricardo DiasandGitHub 54f7c909da Add script function flags in the module API (#2836)
This commit adds script function flags to the module API, which allows
function scripts to specify the function flags programmatically.

When the scripting engine compiles the script code can extract the flags
from the code and set the flags on the compiled function objects.

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-11-20 10:23:00 +00:00
bfa4bae61d Fix cluster slot migration flaky test (#2756)
The original test code only checks:

The original test code only checks:

1. wait_for_cluster_size 4, which calls cluster_size_consistent for every node.
Inside that function, for each node, cluster_size_consistent queries cluster_known_nodes,
which is calculated as (unsigned long long)dictSize(server.cluster->nodes). However, when
a new node is added to the cluster, it is first created in the HANDSHAKE state, and
clusterAddNode adds it to the nodes hash table. Therefore, it is possible for the new
node to still be in HANDSHAKE status (processed asynchronously) even though it appears
that all nodes “know” there are 4 nodes in the cluster.

2. cluster_state for every node, but when a new node is added, server.cluster->state remains FAIL.


Some handshake processes may not have completed yet, which likely causes the flakiness.
To address this, added a --cluster check to ensure that the config state is consistent.

Fixes #2693.

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-11-20 15:07:16 +08:00
aradz44andGitHub 01b1d4e64b deflake "Hash field TTL and active expiry propagates correctly" (#2856)
Fix a little miss in "Hash field TTL and active expiry propagates
correctly through chain replication" test in `hashexpire.tcl`.
The test did not wait for the initial sync of the chained replica and thus  made the test flakey

Signed-off-by: Arad Zilberstein <aradz@amazon.com>
2025-11-19 11:33:55 +02:00
1cf1115cca Perform data cleanup during RDB load on successful version/signature validation (#2600)
Addresses: https://github.com/valkey-io/valkey/issues/2588

## Overview
Previously we call `emptyData()` during a fullSync before validating the
RDB version is compatible.

This change adds an rdb flag that allows us to flush the database from
within `rdbLoadRioWithLoadingCtx`. THhis provides the option to only
flush the data if the rdb has a valid version and signature. In the case
where we do have an invalid version and signature, we don't emptyData,
so if a full sync fails for that reason a replica can still serve stale
data instead of clients experiencing cache misses.

## Changes
- Added a new flag `RDBFLAGS_EMPTY_DATA` that signals to flush the
database after rdb validation
- Added logic to call `emptyData` in `rdbLoadRioWithLoadingCtx` in
`rdb.c`
- Added logic to not clear data if the RDB validation fails in
`replication.c` using new return type `RDB_INCOMPATIBLE`
- Modified the signature of `rdbLoadRioWithLoadingCtx` to return RDB
success codes and updated all calling sites.

## Testing
Added a tcl test that uses the debug command `reload nosave` to load
from an RDB that has a future version number. This triggers the same
code path that full sync's will use, and verifies that we don't flush
the data until after the validation is complete.

A test already exists that checks that the data is flushed:
https://github.com/valkey-io/valkey/blob/unstable/tests/integration/replication.tcl#L1504

---------

Signed-off-by: Venkat Pamulapati <pamuvenk@amazon.com>
Signed-off-by: Venkat Pamulapati <33398322+ChiliPaneer@users.noreply.github.com>
Co-authored-by: Venkat Pamulapati <pamuvenk@amazon.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
2025-11-18 17:08:10 -08:00
22e21e210f Fix SCAN consistency test to only test what we guarantee (#2853)
Test the SCAN consistency by alternating SCAN
calls to primary and replica.
We cannot rely on the exact order of the elements and the returned
cursor number.

---------

Signed-off-by: yzc-yzc <96833212+yzc-yzc@users.noreply.github.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-11-18 16:06:20 +01:00
4b9f151070 Optimize zset memory usage by embedding element in skiplist (#2508)
By default, when the number of elements in a zset exceeds 128, the
underlying data structure adopts a skiplist. We can reduce memory usage
by embedding elements into the skiplist nodes. Change the `zskiplistNode`
memory layout as follows:

```
Before
                 +-------------+
         +-----> | element-sds |
         |       +-------------+
         |
 +------------------+-------+------------------+---------+-----+---------+
 | element--pointer | score | backward-pointer | level-0 | ... | level-N |
 +------------------+-------+------------------+---------+-----+---------+



After
 +-------+------------------+---------+-----+---------+-------------+
 + score | backward-pointer | level-0 | ... | level-N | element-sds |
 +-------+------------------+---------+-----+---------+-------------+
```

Before the embedded SDS representation, we include one byte representing
the size of the SDS header, i.e. the offset into the SDS representation
where that actual string starts.

The memory saving is therefore one pointer minus one byte = 7 bytes per
element, regardless of other factors such as element size or number of
elements.

### Benchmark step

I generated the test data using the following lua script && cli command.
And check memory usage using the `info` command.

**lua script**
```
local start_idx = tonumber(ARGV[1])
local end_idx = tonumber(ARGV[2])
local elem_count = tonumber(ARGV[3])

for i = start_idx, end_idx do
    local key = "zset:" .. string.format("%012d", i)
    local members = {}

    for j = 0, elem_count - 1 do
        table.insert(members, j)
        table.insert(members, "member:" .. j)
    end

    redis.call("ZADD", key, unpack(members))
end

return "OK: Created " .. (end_idx - start_idx + 1) .. " zsets"
```

**valkey-cli command**
`valkey-cli EVAL "$(catcreate_zsets.lua)" 0 0 100000
${ZSET_ELEMENT_NUM}`

### Benchmark result
|number of elements in a zset | memory usage before optimization |
memory usage after optimization | change |
|-------|-------|-------|-------|
| 129 | 1047MB | 943MB | -9.9% |
| 256 |  2010MB|  1803MB| -10.3%|
| 512 |  3904MB|3483MB| -10.8%|

---------

Signed-off-by: chzhoo <czawyx@163.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-11-18 14:27:15 +01:00
Roshan KhatriandGitHub b7d9c6519f Fix the failing warmup and duration are cumulative (#2854)
We need to verify total duration was at least 2 seconds, elapsed time
can be quite variable to check upper-bound

Fixes https://github.com/valkey-io/valkey/issues/2843

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2025-11-17 21:26:12 +01:00
BinbinandGitHub fb7684d7f8 Fix timing issue in dual channel replication COB test (#2847)
After #2829, valgrind report a test failure, it seems that the time is
not enough to generate a COB limit in valgrind.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-11-17 17:25:19 +08:00
BinbinandGitHub 4b2d5c1fef Allow dual channel full sync in plain failover (#2659)
PSYNC_FULLRESYNC_DUAL_CHANNEL is also a full sync, as the comment says,
we need to allow it. While we have not yet identified the exact edge case
that leads to this line, but during a failover, there should be no difference
between different sync strategies.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-11-15 12:57:27 +08:00
631ca9ef76 Print node name on a best effort basis if light weight message is received before link stabilization (#2825)
fixes: #2803

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-11-14 14:33:16 -08:00
yzc-yzcandGitHub 1d14771e70 Attempt to fix flaky SCAN consistency test (#2834)
Related test failures:

https://github.com/valkey-io/valkey/actions/runs/19282092345/job/55135193394

https://github.com/valkey-io/valkey/actions/runs/19200556305/job/54887767594

> *** [err]: scan family consistency with configured hash seed in
tests/integration/scan-family-consistency.tcl
> Expected '5 {k:1 k:25 z k:11 k:18 k:27 k:45 k:7 k:12 k:19 k:29 k:40
k:41 k:43}' to be equal to '5 {k:1 k:25 k:11 z k:18 k:27 k:45 k:7 k:12
k:19 k:29 k:40 k:41 k:43}' (context: type eval line 26 cmd {assert_equal
$primary_cursor_next $replica_cursor_next} proc ::start_server)

The reason is that the RDB part of the primary-replica synchronization
affects the resize policy of the hashtable.
See
https://github.com/valkey-io/valkey/blob/283c45deae6e8486df310886accedd0769b17ac2/src/server.c#L807-L818

Signed-off-by: yzc-yzc <96833212+yzc-yzc@users.noreply.github.com>
2025-11-14 10:55:05 +01:00
BinbinandGitHub 79246e920c Change DEFAULT_WAIT_BEFORE_RDB_CLIENT_FREE from 60s to 5s (#2829)
Consider this scenario:
1. Replica starts loading the RDB using the rdb connection
2. Replica finishes loading the RDB before the replica main connection has
   initiated the PSYNC request
3. Replica stops replicating after receiving replicaof no one
4. Primary can't know that the replica main connection will never ask for
   PSYNC, so it keeps the reference to the replica's replication buffer block
5. Primary has a shutdown-timeout configured and requires to wait for the rdb
   connection to close before it can shut down.

The current 60-second wait time (DEFAULT_WAIT_BEFORE_RDB_CLIENT_FREE) is excessive
and leads to prolonged resource retention in edge cases. Reducing this timeout to
5 seconds would provide adequate time for legitimate PSYNC requests while mitigating
the issue described above.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-11-14 11:32:29 +08:00
Ricardo DiasandGitHub 25acb2e2c7 Fix cluster slot stats for scripts with cross-slot keys (#2835)
This commit fixes the cluster slot stats for scripts executed by
scripting engines when the scripts access cross-slot keys.

This was not a bug in Lua scripting engine, but `VM_Call` was missing a
call to invalidate the script caller client slot to prevent the
accumulation of stats.

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-11-13 12:05:52 -08:00
Rain ValentineandGitHub 49f760c2bc Add --warmup and --duration parameters to valkey-benchmark (#2581)
It's handy to be able to automatically do a warmup and/or test by
duration rather than request count. 🙂

I changed the real-time output a bit - not sure if that's wanted or not.
(Like, would it break people's weird scripts? It'll break my weird
scripts, but I know the price of writing weird fragile scripts.)

```
Prepended "Warming up " when in warmup phase:
Warming up SET: rps=69211.2 (overall: 69747.5) avg_msec=0.425 (overall: 0.393) 3.8 seconds
^^^^^^^^^^

Appended running request counter when based on -n:
SET: rps=70892.0 (overall: 69878.1) avg_msec=0.385 (overall: 0.398) 612482 requests
                                                                    ^^^^^^^^^^^^^^^

Appended running second counter when in warmup or based on --duration:
SET: rps=61508.0 (overall: 61764.2) avg_msec=0.430 (overall: 0.426) 4.8 seconds
                                                                    ^^^^^^^^^^^
```

To be clear, the report at the end remains unchanged.

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-11-13 12:57:46 +01:00
Sarthak AggarwalandGitHub 283c45deae Fixes test-freebsd workflow in daily (package lang/tclX) (#2832)
This PR fixes the freebsd daily job that has been failing consistently
for the last days with the error "pkg: No packages available to install
matching 'lang/tclx' have been found in the repositories".

The package name is corrected from `lang/tclx` to `lang/tclX`. The
lowercase version worked previously but appears to have stopped working
in an update of freebsd's pkg tool to 2.4.x.

Example of failed job:

https://github.com/valkey-io/valkey/actions/runs/19282092345/job/55135193499

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-11-13 08:24:37 +01:00
BinbinandGitHub 56544968b9 Remove the EXAT and PXAT from some HFE notifications tests (#2831)
As we can see, we expected to get hexpired, but we got hexpire instead,
this means tht the expiration time has expired during execution.
```
*** [err]: HGETEX EXAT keyspace notifications for active expiry in tests/unit/hashexpire.tcl
Expected 'pmessage __keyevent@* __keyevent@9__:hexpired myhash' to match 'pmessage __keyevent@* __keyevent@*:hexpire myhash'
```

We should remove the EXAT and PXAT from these fixtures. And we indeed
have
the dedicated tests that verify that we get 'expired' when EX,PX are set
to 0
or EXAT,PXAT are in the past.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-11-12 14:32:13 +02:00
e725f2031e New module API to perform prefix‑aware ACL permission check (#2796)
## Description

This change introduces the ability for modules to check ACL permissions
against key prefix. The update adds a dedicated `prefixmatchlen` helper
and extends the core ACL selector logic to support a prefix‑matching
mode.

The new API `ValkeyModule_ACLCheckPrefixPermissions` is registered and
exposed to modules, and a corresponding implementation is added in
`module.c`. Existing internal callers that already perform prefix checks
(e.g., `VM_ACLCheckKeyPermissions`) are updated to use the new flag,
while all legacy paths remain unchanged.

The change also modifies the `aclcheck§ test module that exercises the
new prefix‑checking API, ensuring that read/write operations are
correctly allowed or denied based on the ACL configuration.

Key areas touched:

* ACL logic
* Module API
* Testing

# Motivation

The search module presently makes costly calls to verify index
permissions
(see https://github.com/valkey-io/valkey-search/blob/main/src/acl.cc#L295).
This PR introduces a more efficient approach for that.

---------

Signed-off-by: Eran Ifrah <eifrah@amazon.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-11-12 10:51:58 +02:00
cda9549eee Cluster: Avoid usage of light weight messages to nodes with not ready bidirectional links (#2817)
After network failure nodes that come back to cluster do not always send
and/or receive messages from other nodes in shard, this fix avoids usage
of light weight messages to nodes with not ready bidirectional links.
When a light message comes before any normal message, freeing of cluster
link is happening because on the just established connection link->node
is not assigned yet. It is assigned in getNodeFromLinkAndMsg right after
the condition if (is_light).
So on a cluster with heavy pubsub load a long loop of disconnects is
possible, and we got this.
1. node A establishes cluster link to node B
2. node A propagates PUBLISH to node B
3. node B frees cluster link because of link->node == null as it has not
received non-light messages yet
4. go to 1.
During this loop subscribers of node B does not receive any messages
published to node A.

So here we want to make sure that PING was sent (and link->node was
initialized) on this connection before using lightweight messages.

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
2025-11-11 20:03:24 -08:00
Jim BrunnerandGitHub afaf306a37 shared zadd for geoadd (#2828)
GEOADD is allocating/destroying a string object for "ZADD"
each time it is called. Created a shared string instead.

Signed-off-by: Jim Brunner <brunnerj@amazon.com>
2025-11-11 15:26:53 -08:00
Roshan KhatriandGitHub cbdf9ce223 Fix Test dual-channel: primary tracking replica backlog refcount (#2827)
This increases the times we check for the logs from 20 to 40.

I found that every `wait-for` check takes about 1.5 to 1.57 milliseconds
so when we were checking 2000 times after 1ms we were actually spending
(2000 * 1) + (2000 *1.75) = 5500ms time waiting.
this can be founds under: for 10 checks we took 35 ms more so thats
around 1.75 ms per check
```
Execution time: 2034 ms (failed)
[err]: 20 100 - Test dual-channel: primary tracking replica backlog refcount - start with empty backlog in tests/integration/dual-channel-replication-flaky.tcl
```

That is why increasing it to 40 100 will check for approx 4,070 ms which
is still less than the original 5500ms but should passes every single
time here:
https://github.com/roshkhatri/valkey/actions/runs/19279424967/job/55126976882

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2025-11-12 00:03:50 +01:00
5648115862 Allow partial sync after loading AOF with preamble (#2366)
The AOF preamble mechanism replaces the traditional AOF base file with
an RDB snapshot during rewrite operations, which reduces I/O overhead
and improves loading performance.
However, when valkey loads the RDB-formatted preamble from the base AOF
file, it does not process the replication ID (replid) information within
the RDB AUX fields. This omission has two limitations:

* On a primary, it prevents the primary from accepting PSYNC continue
  requests after restarting with a preamble-enabled AOF file.
* On a replica, it prevents the replica from successfully performing
  partial sync requests (avoiding full sync) after restarting with a
  preamble-enabled AOF file.

To resolve this, this commit aligns the AOF preamble handling with the
logic used for standalone RDB files, by storing the replication ID and
replication offset in the AOF preamble and restoring them when loading
the AOF file.

Resolves #2677

---------

Signed-off-by: arthur.lee <liziang.arthur@bytedance.com>
Signed-off-by: Arthur Lee <arthurkiller@users.noreply.github.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-11-11 12:41:27 +01:00
Ricardo DiasandGitHub 78bfb6931f Expose SIMPLE_STRING and ARRAY_NULL reply type to the Module API (#2804)
This commit extends the Module API to expose the `SIMPLE_STRING` and
`ARRAY_NULL` reply types to modules, by passing the new flag `X` to
the `ValkeyModule_Call` function.

By only exposing the new reply types behind a flag we keep the
backward compatibility with existing module implementations and
allow new modules to working with these reply type, which are
required for scripts to process correctly the reply type of commands
called inside scripts.

Before this change, commands like `PING` or `SET`, which return `"OK"`
as a simple string reply, would be returned as string replies to
scripts.
To allow the support of the Lua engine as an external module, we need to
distinguish between simple string and string replies to keep backward
compatibility.

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-11-10 15:05:26 +00:00
Ricardo DiasandGitHub df1e969a86 Adds new module context flag VALKEYMODULE_CTX_SCRIPT_EXECUTION (#2818)
The new module context flag `VALKEYMODULE_CTX_SCRIPT_EXECUTION` denotes
that the module API function is being called in the context of a
scripting engine execution.

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-11-10 10:29:40 +00:00
Vadym KhoptynetsandGitHub 7608b880f5 Leverage zfree_with_size for client reply blocks (#2624)
clientReplyBlock stores the size of the actual allocation in it size
field (minus the header size). This can be used for more effective
deallocation with zfree_with_size.

Signed-off-by: Vadym Khoptynets <vadymkh@amazon.com>
2025-11-09 20:46:27 +02:00
137194edc4 [DEFLAKE] Psync established after rdb load - beyond grace period (#2748)
Resolves: https://github.com/valkey-io/valkey/issues/2695

Increase the wait time for periodic log check for rdb load time. Also,
increases the delay of log check frequency.

---------

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Signed-off-by: Roshan Khatri <117414976+roshkhatri@users.noreply.github.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
2025-11-07 15:11:37 -08:00
Harkrishn PatroandGitHub 03c521b880 [flaky-failure-fix] Increase the cluster-node-timeout to have longer delay between failover of each shard (#2793) 2025-11-06 16:14:45 -08:00
7cfa3e999a Fix flaky DBSIZE test for atomic slot migration (#2805)
Related test failures: 

    *** [err]: Replica importing key containment (slot 0 from node 0 to 2) - DBSIZE command excludes importing keys in tests/unit/cluster/cluster-migrateslots.tcl
    Expected '1' to match '0' (context: type eval line 2 cmd {assert_match "0" [R $node_idx DBSIZE]} proc ::test)

The reason is that we don't wait for the primary-replica synchronization
to complete before starting the next testcase.

---------

Signed-off-by: yzc-yzc <96833212+yzc-yzc@users.noreply.github.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-11-06 18:02:27 +01:00
Ricardo DiasandGitHub e8e877c01c Add "script" context to ACL log entries (#2798)
In this commit we add a new context for the ACL log entries that is used
to log ACL failures that occur during scripts execution. To maintain
backward compatibility we still maintain the "lua" context for failures
that happen when running Lua scripts. For other scripting engines the
context description will be just "script".

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-11-06 09:46:22 +00:00
hieu2102andGitHub e8a9620864 Add instruction to build Valkey with fast_float (#2810)
The `README.md` file is currently missing a section to build Valkey with
`fast_float`, which was introduced in Valkey 8.1 as an optional
dependency (#1260)

Signed-off-by: hieu2102 <hieund2102@gmail.com>
2025-11-06 09:45:12 +00:00
ee3eee2eff Configurable DB hash seed for SCAN family commands consistency (#2608)
Introduce a new config `hash-seed` which can be set only at startup and
controls the hash seed for the server. This includes all hash tables.
This change makes it so that both primaries and replicas will return the
same results for SCAN/HSCAN/ZSCAN/SSCAN cursors. This is useful in order
to make sure SCAN behaves correctly after a failover.

Resolves #4

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthakaggarwal97@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-11-05 08:45:52 -08:00
xbaselandGitHub 5dee0ca981 Reuse dbHasNoKeys() inside dbsHaveNoKeys() to remove duplicate logic (#2800)
Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>
2025-11-04 11:28:39 -08:00
016a36ff77 Reverts hashHashtableTypeValidate signature (#2799)
Fixes
https://github.com/valkey-io/valkey/actions/runs/19053371057/job/54418411647#step:6:202

Matched hashHashtableTypeValidate to the [generic hashtable callback
signature
](https://github.com/valkey-io/valkey/blob/unstable/src/hashtable.h#L62)and
performed the entry cast internally to preserve expiry checks.

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Jim Brunner <brunnerj@amazon.com>
2025-11-04 20:07:57 +02:00
Jim BrunnerandGitHub b4acdcf65e Improve header comment and strengthen type checking for entry (#2794)
In `entry.c`, the `entry` is a block of memory with variable contents.
The structure can be difficult to understand. A new header comment more
clearly documents the contents/layout of the `entry`.

Also, in `entry.h`, the `entry` was defined by `typedef void entry`.
This allows blind casting to the `entry` type. It defeats compiler type
checking.

Even though the `entry` has a variable definition, we can define entry
as a legitimate type which allows the compiler to perform type checking.
By performing `typedef struct _entry entry`, now the `entry` is
understood to be a pointer to some type of undefined structure. We can
pass a pointer and the compiler can typecheck the pointer. (Of course we
can't dereference it, because we haven't actually defined the struct.)

Signed-off-by: Jim Brunner <brunnerj@amazon.com>
2025-11-03 19:39:36 +02:00
529d93c807 HSETEX: Support NX/XX Flags (#2668)
### Summary
Addresses https://github.com/valkey-io/valkey/issues/2619.  
This PR extends the `HSETEX` command to support optional key-level `NX`
and `XX` flags, allowing operations conditional on the existence of the
hash key.

### Changes
- Updated `hsetex.json` and regenerated `commands.def`.
- Extended argument parsing for NX/XX.
- Added key-level `NX`/`XX` support in `HSETEX`.
- Added tests covering all four NX/XX scenarios.

---------

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2025-11-03 09:43:48 +02:00
Simon BaatzandGitHub faab45a4de Sentinel: fix regression requiring "+failover" ACL in failover path (#2780)
Since Valkey Sentinel 9.0, sentinel tries to abort an ongoing failover
when changing the role of a monitored instance. Since the result of the
command is ignored, the "FAILOVER ABORT" command is sent irrespective of
the actual failover status.

However, when using the documented pre 9.0 ACLs for a dedicated sentinel
user, the FAILOVER command is not allowed and _all_ failover cases fail.
(Additionally, the necessary ACL adaptation was not communicated well.)

Address this by:

- Updating the documentation in "sentinel.conf" to reflect the need for
an adapted ACL

- Only abort a failover when sentinel detected an ongoing (probably
stuck) failover. This means that standard failover and manual failover
continue to work with unchanged pre 9.0 ACLs. Only the new "SENTINEL
FAILOVER COORDINATED" requires to adapt the ACL on all Valkey nodes.

- Actually use a dedicated sentinel user and ACLs when testing standard
failover, manual failover, and manual coordinated failover.

Fixes #2779

Signed-off-by: Simon Baatz <gmbnomis@gmail.com>
2025-10-31 14:46:53 -04:00
260afe2a2d Fix: ltrim should not call signalModifiedKey when no elements are removed (#2787)
There’s an issue with the LTRIM command. When LTRIM does not actually
modify the key — for example, with `LTRIM key 0 -1` — the server.dirty
counter is not updated because both ltrim and rtrim values are 0. As a
result, the command is not propagated. However, `signalModifiedKey` is
still called regardless of whether server.dirty changes. This behavior
is unexpected and can cause a mismatch between the source and target
during propagation, since the LTRIM command is not sent.

Signed-off-by: Harry Lin <harrylhl@amazon.com>
Co-authored-by: Harry Lin <harrylhl@amazon.com>
2025-10-31 14:46:36 -04:00
Jacob MurphyandGitHub 80d4f9558f Authenticate slot migration client on source node to internal user (#2785)
Just setting the authenticated flag actually authenticates to the
default user in this case. The default user may be granted no permission
to use CLUSTER SYNCSLOTS.

Instaed, we now authenticate to the NULL/internal user, which grants
access to all commands. This is the same as what we do for replication:


https://github.com/valkey-io/valkey/blob/c5bba101dda85699de4f1c04d53e323c9877a6cc/src/replication.c#L4717

Add a test for this case as well.

Closes #2783

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-10-31 10:57:05 -07:00
Ricardo DiasandGitHub 15fcb90b5e Add ValkeyModule_ReplyWithCustomErrorFormat to module API (#2791)
Note: these changes are part of the effort to run Lua engine as an
external scripting engine module.

The new function `ValkeyModule_ReplyWithCustomErrorFormat` is being
added to the module API to allow scripting engines to return errors that
originated from running commands within the script code, without
counting twice in the error stats counters.

More details on why this is needed by scripting engines can be read in
an older commit 4a0fd63b69 messsage.

This PR also adds a new test to ensure the correctness of the newly
added function.

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-10-31 16:02:27 +00:00
xbaselandGitHub 77b7c9fb8d Bug fix: reset io_last_written on c->buf resize to prevent stale pointers (#2786)
Fixes an assert crash in _writeToClient():

    serverAssert(c->io_last_written.data_len == 0 ||
                 c->io_last_written.buf == c->buf);

The issue occurs when clientsCronResizeOutputBuffer() grows or
reallocates c->buf while io_last_written still points to the old buffer
and data_len is non-zero. On the next write, both conditions in the
assertion become false.

Reset io_last_written when resizing the output buffer to prevent stale
pointers and keep state consistent.

fixes https://github.com/valkey-io/valkey/issues/2769

Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>
2025-10-30 13:51:01 -07:00
Ricardo DiasandGitHub c5bba101dd Make ValkeyModule_Call compatible with calling commands from scripting engines (#2782)
Note: these changes are another step towards being able to run Lua
engine as an external scripting engine module.

In this commit we improve the `ValkeyModule_Call` API function code to
match the validations and behavior of the `scriptCall` function,
currently used by the Lua engine to run commands using `server.call` Lua
Valkey API.

The changes made are backward compatible. The new behavior/validations
are only enabled when calling `ValkeyModule_Call` while running a script
using `EVAL` or `FCALL`.

To test these changes, we improved the `HELLO` dummy scripting engine
module to support calling commands, and compare the behavior with
calling the same command from a Lua script.

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-10-30 16:19:46 +00:00
Ricardo DiasandGitHub 6cd9c82096 New INFO section for scripting engines (#2738)
This commit adds a new `INFO` section called "Scripting Engines" that
shows the information of the current scripting engines available in the
server.

Here's an output example:

```
> info scriptingengines
# Scripting Engines
engines_count:2
engines_total_used_memory:68608
engines_total_memory_overhead:56
engine_0:name=LUA,module=built-in,abi_version=4,used_memory=68608,memory_overhead=16
engine_1:name=HELLO,module=helloengine,abi_version=4,used_memory=0,memory_overhead=40
```

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-10-30 16:18:01 +00:00
6cdc8a268d Add IPv6 availability check to skip tests when unavailable (#2674)
Skip IPv6 tests automatically when IPv6 is not available.

This fixes the problem that tests fail when IPv6 is not available on the
system, which can worry users when they run `make test`.

IPv6 availibility is detected by opening a dummy server socket and
trying to connect to it using a client socket over IPv6.

Fixes #2643

---------

Signed-off-by: diego-ciciani01 <diego.ciciani@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-10-30 11:20:56 +01:00
Sarthak AggarwalandGitHub 216bd36f57 Adds a summary for tests (#2745)
```
Test Summary: 100 passed, 2 failed

!!! WARNING The following tests failed:
...
````

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-10-29 13:36:37 -07:00
4a0fb56c34 Add monotonic clock calibration handling if clock speed is not found (#2776)
Currently, monotonic clock initialization relies on the model name field
from /proc/cpuinfo to retrieve the clock speed. However, this is not
always present. In case it is not present, measure the clock tick and
use it instead.

Before fix:
```
monotonic: x86 linux, unable to determine clock rate
```

After fix:
```
21695:M 25 Oct 2025 20:16:23.168 * monotonic clock: X86 TSC @ 2649 ticks/us
```

Fixes #2774

---------

Signed-off-by: Ken Nam <otherscase@gmail.com>
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2025-10-28 22:20:12 +02:00
Ritoban DuttaandGitHub d4c6035d62 Reorder valkey.conf: move configs to correct sections (#2737)
- Moved `server-cpulist`, `bio-cpulist`, `aof-rewrite-cpulist`,
  `bgsave-cpulist` configurations to ADVANCED CONFIG.
- Moved `ignore-warnings` configuration to ADVANCED CONFIG.
- Moved `availability-zone` configuration to GENERAL.

These configs were incorrectly placed at the end of the file in the
ACTIVE DEFRAGMENTATION section.

Fixes #2736

---------

Signed-off-by: ritoban23 <ankudutt101@gmail.com>
2025-10-28 10:36:23 +01:00
Sarthak AggarwalandGitHub 0c9c9241a7 Reverts rdb-key-save-delay value to fix dual channel replication test in macos (#2771)
Resolves #2696 

Set `rdb-key-save-delay` to 200 microseconds to reduce the overall RDB load time.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-10-27 13:08:40 -07:00
Zhijun LiaoandGitHub 779cdb48ae Sentinel: Skip IS-PRIMARY-DOWN-BY-ADDR requests when primary not SDOWN (#2763)
A super tiny change to optimize the function
`sentinelAskPrimaryStateToOtherSentinels` to early return when the
sentinel does not observe the primary as subjectively down.

Signed-off-by: Zhijun <dszhijun@gmail.com>
2025-10-24 16:15:54 -04:00
Zhijun LiaoandGitHub 92cc18e2bb Ensure the server executable exists before running tests (#2762)
Previously, running ./runtest without src/valkey-server would hang, now it
throws an error.

Signed-off-by: Zhijun <dszhijun@gmail.com>
2025-10-23 19:29:50 +08:00
Ricardo DiasandGitHub 5a2e338916 Scripting Engine Debugger Support (#1701)
This PR introduces the support for implementing remote debuggers in
scripting engines modules.

The module API is extended with scripting engines callbacks and new
functions that can be used by scripting engine modules to implement a
remote debugger.

Most of the code that was used to implement the Lua debugger, was
refactored and moved to the `scripting_engine.c` file, and only the code
specific to the Lua engine, remained in the `debug_lua.c` file.

The `SCRIPT DEBUG (YES|NO|SYNC)` command was extend with an optional
parameter that can be used to specify the engine name, where we want to
enable the debugger. If no engine name is specified, the Lua engine is
used to keep backwards compatibility.

In
[src/valkeymodule.h](https://github.com/valkey-io/valkey/pull/1701/files#diff-b91520205c29a3a5a940786e509b2f13a5e73a1ac2016be773e62ea64c7efb28)
we see the module API changes. And in the `helloscripting.c` file we can
see how to implement a simple debugger for the dummy HELLO scripting
engine.

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-10-23 10:58:32 +01:00
Viktor SöderqvistandGitHub 855c0c3cc0 Adjust test runners to the number of tests to run (#2759)
This is fixing a minor annoyance when running single tests. With this
change, the runtest script doesn't start more runners than the total
number of tests to run. These are seen in the printouts like `[ready]:
68359`.

Screenshot before:

```
$ ./runtest --single tests/unit/limits.tcl 
Cleanup: may take some time... OK
Starting test server at port 21079
[ready]: 68359
Testing unit/limits
[ready]: 68355
[ready]: 68362
[ready]: 68356
[ready]: 68361
[ready]: 68358
[ready]: 68364
[ready]: 68366
[ready]: 68367
[ready]: 68368
[ready]: 68357
[ready]: 68360
[ready]: 68363
[ready]: 68365
[ready]: 68369
[ready]: 68370
[ok]: Check if maxclients works refusing connections (906 ms)
[1/1 done]: unit/limits (1 seconds)

                   The End

Execution time of different units:
  1 seconds - unit/limits

\o/ All tests passed without errors!

Cleanup: may take some time... OK
```

Screenshot after:

```
$ ./runtest --single tests/unit/limits.tcl 
Cleanup: may take some time... OK
Starting test server at port 21079
[ready]: 68439
Testing unit/limits
[ok]: Check if maxclients works refusing connections (906 ms)
[1/1 done]: unit/limits (1 seconds)

                   The End

Execution time of different units:
  1 seconds - unit/limits

\o/ All tests passed without errors!

Cleanup: may take some time... OK
```

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-10-23 09:48:08 +02:00
Zhijun LiaoandGitHub 273c079319 Use the fetched TLS and TCP ports in clusterProcessGossipSection (#2761)
The TLS and TCP ports have been fetched and should be used.
This can improve code readability.

Signed-off-by: Zhijun <dszhijun@gmail.com>
2025-10-23 15:41:29 +08:00
Hanxi ZhangandGitHub 33bd8cebaa Show RPS histogram in valkey-benchmark (#2471)
When benchmarking with the a target thoughput using the `--rps` flag,
display an RPS histogram in the benchmark report. This can help identify
if there is a bottleneck.

Related to #2219.

---------

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
2025-10-22 11:35:53 +02:00
Jacob MurphyandGitHub 18c1896e63 Fix atomic slot migration module notification comment (#2758)
Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-10-21 20:28:55 -07:00
Madelyn OlsonandGitHub a1f4a620bd Update module api generation and format module.c (#2757)
This change makes it so that the module API reference can be cleanly
generated from the module.c file. Most of this seems to be because of
the code formatting work we did. There are two parts:
1. Updated some of the odd corner cases in module.c so they can be
handled. For example, there was a method that had none of the method on
the first line, which was unhandled. None of these are functional and I
think format should be OK with them.
2. Better handle multi-line function prototypes in the ruby code. Before
we just relied on the function to be on a single line, now we handle it
on multiple lines. This feels pretty hacked in, but I don't really
understand the rest of the code but it does work.

Generated this PR:
https://github.com/valkey-io/valkey-doc/pull/374/files.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-10-21 20:28:40 -07:00
BinbinandGitHub f5343280a9 Fix outdated comment around clusterLink->flags (#2752)
This clusterLink->flags was added in #2310, during the review, we at
the end chose to add a new CLUSTER_LINK_XXX flag instead of sharing the
old CLUSTER_NODE_XXX flag.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-10-21 10:18:41 +08:00
Jacob MurphyandGitHub 1e3e6dca72 Fix invalid memory address caused by hashtable shrinking during safe iteration (#2753)
Safe iterators pause rehashing, but don't pause auto shrinking. This
allows stale bucket references which then cause use after free (in this
case, via compactBucketChain on a deleted bucket).

This problem is easily reproducible via atomic slot migration, where we
call delKeysInSlot which relies on calling delete within a safe
iterator. After the fix, it no longer causes a crash.

Since all cases where rehashing is paused expect auto shrinking to also
be paused, I am making this happen automatically as part of pausing
reshashing.

---------

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-10-20 13:23:29 -07:00
BinbinandGitHub e9ec72fcb0 Initialize the lua attributes of the luaFunction script (#2750)
This was introduced in #1826. This create an `Uninitialised value was
created by a heap allocation` in the CI.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-10-20 11:17:05 +02:00
Jacob MurphyandGitHub 676c10131a Fix incorrect kvstore size and BIT accounting after completed migration (#2749)
When working on #2635 I errorneously duplicated the
setSlotImportingStateInAllDbs call for successful imports. This resulted
in us doubling the key count in the kvstore. This results in DBSIZE
reporting an incorrect sum, and also causes BIT corruption that can
eventually result in a crash.

The solution is:

1. Only call setSlotImportingStateInAllDbs once (in our
finishSlotMigrationJob function)
2. Make setSlotImportingStateInAllDbs idempotent by checking if the
delete from the kvstore importing hashtable is a no-op

This also fixes a bug where the number of importing keys is not lowered
after the migration, but this is less critical since it is only used
when resizing the dictionary on RDB load. However, it could result in
un-loadable RDBs if the importing key count gets large enough.

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-10-18 10:20:18 -07:00
6d4846b4d1 FUNCTION FLUSH re-create lua VM, fix flush not gc, fix flush async + load crash (#1826)
There will be two issues in this test:
```
test {FUNCTION - test function flush} {
    for {set i 0} {$i < 10000} {incr i} {
        r function load [get_function_code LUA test_$i test_$i {return 'hello'}]
    }
    set before_flush_memory [s used_memory_vm_functions]
    r function flush sync
    set after_flush_memory [s used_memory_vm_functions]
    puts "flush sync, before_flush_memory: $before_flush_memory, after_flush_memory: $after_flush_memory"

    for {set i 0} {$i < 10000} {incr i} {
        r function load [get_function_code LUA test_$i test_$i {return 'hello'}]
    }
    set before_flush_memory [s used_memory_vm_functions]
    r function flush async
    set after_flush_memory [s used_memory_vm_functions]
    puts "flush async, before_flush_memory: $before_flush_memory, after_flush_memory: $after_flush_memory"

    for {set i 0} {$i < 10000} {incr i} {
        r function load [get_function_code LUA test_$i test_$i {return 'hello'}]
    }
    puts "Test done"
}
```

The first one is the test output, we can see that after executing
FUNCTION FLUSH,
used_memory_vm_functions has not changed at all:
```
flush sync, before_flush_memory: 2962432, after_flush_memory: 2962432
flush async, before_flush_memory: 4504576, after_flush_memory: 4504576
```

The second one is there is a crash when loading the functions during the
async
flush:
```
=== VALKEY BUG REPORT START: Cut & paste starting from here ===
 # valkey 255.255.255 crashed by signal: 11, si_code: 2
 # Accessing address: 0xe0429b7100000a3c
 # Crashed running the instruction at: 0x102e0b09c

------ STACK TRACE ------
EIP:
0   valkey-server                       0x0000000102e0b09c luaH_getstr + 52

Backtrace:
0   libsystem_platform.dylib            0x000000018b066584 _sigtramp + 56
1   valkey-server                       0x0000000102e01054 luaD_precall + 96
2   valkey-server                       0x0000000102e01b10 luaD_call + 104
3   valkey-server                       0x0000000102e00d1c luaD_rawrunprotected + 76
4   valkey-server                       0x0000000102e01e3c luaD_pcall + 60
5   valkey-server                       0x0000000102dfc630 lua_pcall + 300
6   valkey-server                       0x0000000102f77770 luaEngineCompileCode + 708
7   valkey-server                       0x0000000102f71f50 scriptingEngineCallCompileCode + 104
8   valkey-server                       0x0000000102f700b0 functionsCreateWithLibraryCtx + 2088
9   valkey-server                       0x0000000102f70898 functionLoadCommand + 312
10  valkey-server                       0x0000000102e3978c call + 416
11  valkey-server                       0x0000000102e3b5b8 processCommand + 3340
12  valkey-server                       0x0000000102e563cc processInputBuffer + 520
13  valkey-server                       0x0000000102e55808 readQueryFromClient + 92
14  valkey-server                       0x0000000102f696e0 connSocketEventHandler + 180
15  valkey-server                       0x0000000102e20480 aeProcessEvents + 372
16  valkey-server                       0x0000000102e4aad0 main + 26412
17  dyld                                0x000000018acab154 start + 2476

------ STACK TRACE DONE ------
```

The reason is that, in the old implementation (introduced in 7.0),
FUNCTION FLUSH
use lua_unref to remove the script from lua VM. lua_unref does not
trigger the gc,
it causes us to not be able to effectively reclaim memory after the
FUNCTION FLUSH.

The other issue is that, since we don't re-create the lua VM in FUNCTION
FLUSH,
loading the functions during a FUNCTION FLUSH ASYNC will result a crash
because
lua engine state is not thread-safe.

The correct solution is to re-create a new Lua VM to use, just like
SCRIPT FLUSH.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
Co-authored-by: Ricardo Dias <ricardo.dias@percona.com>
2025-10-17 15:25:17 +01:00
Harkrishn PatroandGitHub 88496a9cce Bump old engine version(s) for compatibility test (#2741)
Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-10-16 22:43:19 -07:00
Roshan KhatriandGitHub ed099004aa Deflake Psync established within grace period (#2743)
increased the wait time to a total of 10 seconds where we check the log
for `Done loading RDB` message

Fixes https://github.com/valkey-io/valkey/issues/2694

CI run (100 times):
https://github.com/roshkhatri/valkey/actions/runs/18576201712/job/52961907806

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2025-10-16 19:50:48 -07:00
BinbinandGitHub ebf7547677 Remove the outupdated unknown key/value pairs comment in CLUSTER SYNCSLOTS ESTABLISH (#2498)
We now have #2688 SYNCSLOTS CAPA, remove the outupdated comment.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-10-17 10:24:43 +08:00
Sarthak AggarwalandGitHub 88739e14e4 Deflakes Primary COB growth with inactive replica (#2715)
Resolves #2696 

The primary issue was that with sanitizer mode, the test needed more
time for primary’s replication buffers grow beyond `2 × backlog_size`.
Increasing the threshold of `repl-timeout` to 30s, ensures that the
inactive replica is not disconnected while the full sync is proceeding.
`rdb-key-save-delay` controls or throttles the data written to the
client output buffer, and in this case, we are deterministically able to
perform the fullsync within 10s (10000 keys * 0.001s).

Increasing the `wait_for_condition` gives it enough retries to verify
that `mem_total_replication_buffers` reaches the required `2 ×
backlog_size`.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-10-16 10:07:52 +08:00
Viktor SöderqvistandGitHub 394c809b98 Fix double MOVED reply on unblock at failover (#2734)
#2329 intoduced a bug that causes a blocked client in cluster mode to
receive two MOVED redirects instead of one. This was not seen in tests,
except in the reply schema validator.

The fix makes sure the client's pending command is cleared after sending
the MOVED redirect, to prevent if from being reprocessed.

Fixes #2676.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-10-14 17:44:17 +02:00
BinbinandGitHub 01635a1e7b Add Slot migration is ok when the replicas are down test back (#2727)
The test was accidentally removed in PR #1671.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-10-14 12:56:36 +08:00
Jacob MurphyandGitHub 8f68208e03 Fix crash that occurs sometimes when aborting a slot migration while child snapshot is active (#2721)
The race condition causes the client to be used and subsequently double
freed by the slot migration read pipe handler. The order of events is:

1. We kill the slot migration child process during CANCELSLOTMIGRATIONS
1. We then free the associated client to the target node
1. Although we kill the child process, it is not guaranteed that the
pipe will be empty from child to parent
1. If the pipe is not empty, we later will read that out in the
slotMigrationPipeReadHandler
1. In the pipe read handler, we attempt to write to the connection. If
writing to the connection fails, we will attempt to free the client
1. However, the client was already freed, so this a double free

Notably, the slot migration being aborted doesn't need to be triggered
by `CANCELSLOTMIGRATIONS`, it can be any failure.

To solve this, we simply:

1. Set the slot migration pipe connection to NULL whenever it is
unlinked
2. Bail out early in slot migration pipe read handler if the connection
is NULL

I also consolidate the killSlotMigrationChild call to one code path,
which is executed on client unlink. Before, there were two code paths
that would do this twice (once on slot migration job finish, and once on
client unlink). Sending the signal twice is fine, but inefficient.

Also, add a test to cancel during the slot migration snapshot to make
sure this case is covered (we only caught it during the module test).

---------

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-10-13 08:18:13 -07:00
Jacob MurphyandGitHub f846c8698d Deflake atomic slot migration client flag test (#2720)
This test was failing, and causing the next test to throw an exception.
It is failing since we never waited for the slot migration to connect
before proceeding.

Fixes #2692

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-10-13 08:17:59 -07:00
Jacob MurphyandGitHub 15c77bf158 Stop using DEBUG LOADAOF on replica in ASM tests (#2719)
DEBUG LOADAOF sometimes works but it results in -LOADING responses to
the primary so there are lots of race conditions. It isn't something we
should be doing anyways. To test, I just disconnect the replica before
loading the AOF, then reconnect it.

Fixes #2712

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-10-13 08:17:36 -07:00
Harkrishn PatroandGitHub 7fdb0d8cf4 Bump CLUSTER SHARDS command update version to 9.1.0 (#2729) 2025-10-12 22:12:08 -07:00
Kyle Kim (kimkyle@)andGitHub a94b484d40 Update the misleading zdiff() comment on empty first key handling. (#747)
Currently, zdiff() becomes a no-op in the case that the first key is
empty.

The existing comment of "Skip everything if the smallest input is empty"
is misleading, as qsort is bypassed for the case of ZDIFF. There's no
guarantee that the first key holds the smallest cardinality.

The real reason behind such bypass is the following. ZDIFF computes the
difference between the first and all successive input sorted sets.
Meaning, if the first key is empty, we cannot reduce further from an
already empty collection, and thus zdiff() becomes a no-op.

Signed-off-by: Kyle Kim <kimkyle@amazon.com>
2025-10-10 18:54:58 -07:00
Viktor SöderqvistandGitHub 9733c76e43 Tests: Don't dump logs when skipping test using 'skip' (#2718)
When using `skip` inside a test to skip a test, when running with
--dump-logs it causes the logs to be dumped. Introduced in #2342.

The reason is the "skipped" exception is caught in the start_server proc
in tests/support/server.tcl. This is where the $::dump_logs variable is
checked.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-10-10 09:54:31 -07:00
Ran ShidlansikandGitHub 71717cab5d HSETEX with FXX should not create an object if it does not exist (#2716)
When the hash object does not exist FXX should simply fail the check
without creating the object while FNX should be trivial and succeed.

Note - also fix a potential compilation warning on some COMPILERS doing
constant folding of variable length array when size is const expression.

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-10-10 12:30:08 +03:00
6441a9f5a5 Add compatibility test with Valkey 7.2/8.0 (#2342)
* Add cross version compatibility test to run with Valkey 7.2 and 8.0
* Add mechanism in TCL test to skip tests dynamically - #2711

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-10-09 17:00:10 -07:00
Sarthak AggarwalandGitHub 3f45b00c20 Deflake replica selection test by relaxing cluster configurations (#2672)
We have relaxed the `cluster-ping-interval` and `cluster-node-timeout`
so that cluster has enough time to stabilize and propagate changes.

Fixes this test occasional failure when running with valgrind:

    [err]: Node #10 should eventually replicate node #5 in tests/unit/cluster/slave-selection.tcl
    #10 didn't became slave of #5

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-10-10 01:03:53 +02:00
Harkrishn PatroandGitHub 018ffd8b89 Add invalid RDB signature to log statement (#2710)
Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-10-09 13:11:31 -07:00
Harkrishn PatroandGitHub 27a4ba055e Fix memory leak with CLIENT LIST/KILL duplicate filters (#2362)
With #1401, we introduced additional filters to CLIENT LIST/KILL
subcommand. The intended behavior was to pick the last value of the
filter. However, we introduced memory leak for all the preceding
filters.

Before this change:
```
> CLIENT LIST IP 127.0.0.1 IP 127.0.0.1
id=4 addr=127.0.0.1:37866 laddr=127.0.0.1:6379 fd=10 name= age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=21 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=16989 events=r cmd=client|list user=default redir=-1 resp=2 lib-name= lib-ver= tot-net-in=49 tot-net-out=0 tot-cmds=0
```
Leak:
```
Direct leak of 11 byte(s) in 1 object(s) allocated from:
    #0 0x7f2901aa557d in malloc (/lib64/libasan.so.4+0xd857d)
    #1 0x76db76 in ztrymalloc_usable_internal /workplace/harkrisp/valkey/src/zmalloc.c:156
    #2 0x76db76 in zmalloc_usable /workplace/harkrisp/valkey/src/zmalloc.c:200
    #3 0x4c4121 in _sdsnewlen.constprop.230 /workplace/harkrisp/valkey/src/sds.c:113
    #4 0x4dc456 in parseClientFiltersOrReply.constprop.63 /workplace/harkrisp/valkey/src/networking.c:4264
    #5 0x4bb9f7 in clientListCommand /workplace/harkrisp/valkey/src/networking.c:4600
    #6 0x641159 in call /workplace/harkrisp/valkey/src/server.c:3772
    #7 0x6431a6 in processCommand /workplace/harkrisp/valkey/src/server.c:4434
    #8 0x4bfa9b in processCommandAndResetClient /workplace/harkrisp/valkey/src/networking.c:3571
    #9 0x4bfa9b in processInputBuffer /workplace/harkrisp/valkey/src/networking.c:3702
    #10 0x4bffa3 in readQueryFromClient /workplace/harkrisp/valkey/src/networking.c:3812
    #11 0x481015 in callHandler /workplace/harkrisp/valkey/src/connhelpers.h:79
    #12 0x481015 in connSocketEventHandler.lto_priv.394 /workplace/harkrisp/valkey/src/socket.c:301
    #13 0x7d3fb3 in aeProcessEvents /workplace/harkrisp/valkey/src/ae.c:486
    #14 0x7d4d44 in aeMain /workplace/harkrisp/valkey/src/ae.c:543
    #15 0x453925 in main /workplace/harkrisp/valkey/src/server.c:7319
    #16 0x7f2900cd7139 in __libc_start_main (/lib64/libc.so.6+0x21139)
```

Note: For filter ID / NOT-ID we group all the option and perform
filtering whereas for remaining filters we only pick the last filter
option.

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-10-08 16:04:47 -07:00
1dc884d4d4 Add shard id field to CLUSTER SHARDS response (#2568)
This change exposes an existing, persistent (in nodes.conf) unique shard
identifier for each shard in the cluster as part of the `CLUSTER SHARDS` command
response.

```
1) 1) "slots"
   2) 1) (integer) 0
      2) (integer) 999
      3) (integer) 2001
      4) (integer) 3999
      5) (integer) 4501
      6) (integer) 5460
   3) "nodes"
   4) 1)  1) "id"
          2) "6e76043bed00e716e85035107866ea16e9a5f700"
          3) "port"
          4) (integer) 6385
          5) "ip"
          6) "127.0.0.1"
          7) "endpoint"
          8) "127.0.0.1"
          9) "role"
         10) "replica"
         11) "replication-offset"
         12) (integer) 8092
         13) "health"
         14) "online"
      2)  1) "id"
          2) "b2f8c841707b2246ec2a641c37f16e88fe0bb700"
          3) "port"
          4) (integer) 6380
          5) "ip"
          6) "127.0.0.1"
          7) "endpoint"
          8) "127.0.0.1"
          9) "role"
         10) "master"
         11) "replication-offset"
         12) (integer) 8092
         13) "health"
         14) "online"
   5) "id"
   6) "3f2a7bb7bbd5fc2a331fe9bf95f5e02bcca02430"
```

---------

Signed-off-by: Satheesha Chattenahalli Hanume Gowda <satheesha@apple.com>
Co-authored-by: Satheesha Chattenahalli Hanume Gowda <satheesha@apple.com>
2025-10-08 12:14:05 -07:00
2d99906a8d Reduce flakiness of atomic slot migration AOF test (#2705)
If we don't wait for the replica to resync, the migration may be
cancelled by the time the replica resyncs, resulting in a test failure
when we can't find the migration on the replica.

---------

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-10-08 12:52:36 +02:00
Viktor SöderqvistandGitHub 4c74204ff3 Allow TCL 9.0 for tests (#1673)
Makes our tests possible to run with TCL 9.

The latest Fedora now has TCL 9.0 and it's working now, including the
TCL TLS package. (This wasn't working earlier due to some packaging
errors for TCL packages in Fedora, which have been fixed now.)

This PR also removes the custom compilation of TCL 8 used in our Daily
jobs and uses the system default TCL version instead. The TCL version
depends on the OS. For the latest Fedora, you get 9.0, for macOS you get
8.5 and for most other OSes you get 8.6.

The checks for TCL 8.7 are removed, because 8.7 doesn't exist. It was
never released.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-10-08 11:37:15 +02:00
Jacob MurphyandGitHub 1808a60bf5 Use correct arguments in LOLWUT test (#2708)
Seeing test failures due to this on the 9.0.0 branch:

```
[exception]: Executing test client: ERR Syntax error. Use: LOLWUT [columns rows] [real imaginary].
ERR Syntax error. Use: LOLWUT [columns rows] [real imaginary]
```

It turns out we are just providing the version as an argument, instead
of specifying which version to run on

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-10-07 15:42:56 -07:00
1ecb8080b5 Introduce SYNCSLOTS CAPA for forwards compatibility (#2688)
For now, introduce this and have it do nothing. Eventually, we can use
this to negotiate supported capabilities on either end. Right now, there
is nothing to send or support, so it just accepts it and doesn't reply.

---------

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-10-07 10:19:36 -07:00
Jacob MurphyandGitHub 8a92b9e381 Prevent exposure of importing keys on replicas during atomic slot migration (#2635)
# Problem

In the current slot migration design, replicas are completely unaware of
the slot migration. Because of this, they do not know to hide importing
keys, which results in exposure of these keys to commands like KEYS,
SCAN, RANDOMKEY, and DBSIZE.

# Design

The main part of the design is that we will now listen for and process
the `SYNCSLOTS ESTABLISH` command on the replica. When a `SYNCSLOTS
ESTABLISH` command is received from the primary, we begin tracking a new
slot import in a special `SLOT_IMPORT_OCCURRING_ON_PRIMARY` state.
Replicas use this state to track the import, and await for a future
`SYNCSLOTS FINISH` message that tells them the import is
successful/failed.

## Success Case

```
     Source                                          Target                         Target Replica
       |                                                |                                 |
       |------------ SYNCSLOTS ESTABLISH -------------->|                                 |
       |                                                |----- SYNCSLOTS ESTABLISH ------>|
       |<-------------------- +OK ----------------------|                                 |
       |                                                |                                 |
       |~~~~~~~~~~~~~~ snapshot as AOF ~~~~~~~~~~~~~~~~>|                                 |
       |                                                |~~~~~~ forward snapshot ~~~~~~~~>|
       |----------- SYNCSLOTS SNAPSHOT-EOF ------------>|                                 |
       |                                                |                                 |
       |<----------- SYNCSLOTS REQUEST-PAUSE -----------|                                 |
       |                                                |                                 |
       |~~~~~~~~~~~~ incremental changes ~~~~~~~~~~~~~~>|                                 |
       |                                                |~~~~~~ forward changes ~~~~~~~~~>|
       |--------------- SYNCSLOTS PAUSED -------------->|                                 |
       |                                                |                                 |
       |<---------- SYNCSLOTS REQUEST-FAILOVER ---------|                                 |
       |                                                |                                 |
       |---------- SYNCSLOTS FAILOVER-GRANTED --------->|                                 |
       |                                                |                                 |
       |                                            (performs takeover &                  |
       |                                             propagates topology)                 |
       |                                                |                                 |
       |                                                |------- SYNCSLOTS FINISH ------->|
 (finds out about topology                              |                                 |
  change & marks migration done)                        |                                 |
       |                                                |                                 |
```

## Failure Case
```
     Source                                          Target                         Target Replica
       |                                                |                                 |
       |------------ SYNCSLOTS ESTABLISH -------------->|                                 |
       |                                                |----- SYNCSLOTS ESTABLISH ------>|
       |<-------------------- +OK ----------------------|                                 |
     ...                                              ...                               ...
       |                                                |                                 |
       |                                             <FAILURE>                            |
       |                                                |                                 |
       |                                      (performs cleanup)                          |
       |                                                | ~~~~~~ UNLINK <key> ... ~~~~~~~>|
       |                                                |                                 |
       |                                                | ------ SYNCSLOTS FINISH ------->|
       |                                                |                                 |
```

## Full Sync, Partial Sync, and RDB

In order to ensure replicas that resync during the import are still
aware of the import, the slot import is serialized to a new
`cluster-slot-imports` aux field. The encoding includes the job name,
the source node name, and the slot ranges being imported. Upon loading
an RDB with the `cluster-slot-imports` aux field, replicas will add a
new migration in the `SLOT_IMPORT_OCCURRING_ON_PRIMARY` state.

It's important to note that a previously saved RDB file can be used as
the basis for partial sync with a primary. Because of this, whenever we
load an RDB file with the `cluster-slot-imports` aux field, even from
disk, we will still add a new migration to track the import. If after
loading the RDB, the Valkey node is a primary, it will cancel the slot
migration. Having this tracking state loaded on primaries will ensure
that replicas partial syncing to a restarted primary still get their
`SYNCSLOTS FINISH` message in the replication stream.

## AOF

Since AOF cannot be used as the basis for a partial sync, we don't
necessarily need to persist the `SYNCSLOTS ESTABLISH` and `FINISH`
commands to the AOF.

However, considering there is work to change this (#59 #1901) this
design doesn't make any assumptions about this.

We will propagate the `ESTABLISH` and `FINISH` commands to the AOF, and
ensure that they can be properly replayed on AOF load to get to the
right state. Similar to RDB, if there are any pending "ESTABLISH"
commands that don't have a "FINISH" afterwards upon becoming primary, we
will make sure to fail those in `verifyClusterConfigWithData`.

Additionally, there was a bug in the existing slot migration where slot
import clients were not having their commands persisted to AOF. This has
been fixed by ensuring we still propagate to AOF even for slot import
clients.

## Promotion & Demotion

Since the primary is solely responsible for cleaning up unowned slots,
primaries that are demoted will not clean up previously active slot
imports. The promoted replica will be responsible for both cleaning up
the slot (`verifyClusterConifgWithData`) and sending a `SYNCSLOTS
FINISH`.

# Other Options Considered

I also considered tracking "dirty" slots rather than using the slot
import state machine. In this setup, primaries and replicas would simply
mark each slot's hashtable in the kvstore as dirty when something is
written to it and we do not currently own that slot.

This approach is simpler, but has a problem in that modules loaded on
the replica would still not get slot migration start/end notifications.
If the modules on the replica do not get such notifications, they will
not be able to properly contain these dirty keys during slot migration
events.

---------

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-10-07 09:17:07 -07:00
Madelyn OlsonandGitHub 401526b03c Fix format issues with CVE fix (#2679)
The CVE fixes had a formatting and external test issue that wasn't
caught because private branches don't run those CI steps.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-10-03 10:26:28 -07:00
Madelyn OlsonandGitHub 971d934793 Merge commit from fork
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-10-03 06:32:24 -07:00
Jacob MurphyandGitHub 268bb44656 Add slot migration client flags and module context flags (#2639)
New client flags in reported by CLIENT INFO and CLIENT LIST:

* `i` for atomic slot migration importing client
* `E` for atomic slot migration exporting client

New flags in return value of `ValkeyModule_GetContextFlags`:

* `VALKEYMODULE_CTX_FLAGS_SLOT_IMPORT_CLIENT`: Indicate the that client
attached to this context is the slot import client.
* `VALKEYMODULE_CTX_FLAGS_SLOT_EXPORT_CLIENT`: Indicate the that client
attached to this context is the slot export client.

Users could use this to monitor the underlying client info of the slot
migration, and more clearly understand why they see extra clients during
the migration.

Modules can use these to detect keyspace notifications on import
clients. I am also adding export flags for symmetry, although there
should not be keyspace notifications. But they would potentially be
visible in command filters or in server events triggered by that client.

---------

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-10-02 21:12:34 +02:00
5716f34f86 Defrag if slab 1/8 full to fix defrag didn't stop issue (#2656)
**Issue History:**
1. The flaky test issue "defrag didn't stop" was originally detected in
February 2025: https://github.com/valkey-io/valkey/issues/1746
Solution for 1746: https://github.com/valkey-io/valkey/pull/1762
2. Similar issue occurred recently:
https://github.com/valkey-io/valkey/actions/runs/16585350083/job/46909359496#step:5:7640

**Investigation:** 
1. First, the issue occurs specifically to Active Defrag stream in
cluster mode.
2. After investigating `test_stream` in `memefficiency.tcl`, I found the
root cause is in defrag logic rather than the test itself - There are
still failed tests with the same error even if I tried different
parameters for the test.
3. Then I looked at malloc-stats and identified potential defrag issues,
particularly in the 80B bin where utilization only reaches ~75% after
defrag instead of the expected near 100%, while other bins show proper
defrag behavior - 80B actually is the size of a new stream(confirmed in
`t_stream.c`) that we add during test.
4. For 80B, after adding 200000 streams and fragmenting, `curregs `=
100030, after a lot of defrag cycles, there are still 122
nonfull-slabs/511 slabs with the remaining 446 items not defragged
(average 4/nonfull-slab).

**Detailed malloc-stats:**
- Total slabs: 511
- Non-full slabs: 122
- Full slabs: 511-122=389
- Theoretical maximum per slab: 256 items
- Allocated items in non-full slabs: 100030-389*256=446
- Average items per non-full slab: 446/122=3.66

**Root Cause:** 
**There are some immovable items which  prevent complete defrag**

**Problems in old defrag logic:**
1. The previous condition (we don't defrag if slab utilization > 'avg
utilization' * 1.125), the 12.5% threshold doesn’t work well with low
utilizations.

- Let's imagine we have 446 items in 122 nonfull-slabs (avg 3.66
items/nonfull-slab), let's say, e.g. we have 81 slabs with 5 items each
+41 slabs with 1 item each)
- 12.5% threshold: 3.66*1.125=4.11
- If those 41 single items are immovable, they actually lower the
average, so the rest 81 slabs will be above the threshold (5>4.11) and
will not be defragged - defrag didn't stop.

2. Distribution of immovable items across slabs was causing inconsistent
defragmentation and flaky test outcome.

- If those 41 single items are movable, they will be moved and the avg
will be 5, then 12.5% threshold: 5*1.125=5.625, so the rest 81 slabs
will fall below the threshold (5<5.625) and will be defragged - defrag
success.
- This can explain why we got flaky defrag tests.


**Final solution :** 
1. Add one more condition before the old logic in `makeDefragDecision
`to trigger defragmentation when slab is less than 1/8 full (1/8
threshold (12.5%) chosen to align with existing utilization threshold
factor) - Ensures no low-utilization slabs left without defragged, and
stabilize the defrag behavior.
2. The reason why we have immovable items and how to handle them is
going to be investigate later.
3. Be sure to rebuild Valkey before testing it.


**Local test result:**
- Before fix: 
pass rate 80.8% (63/78)
- After fix:
Test only stream: pass rate 100% (200/200)
Test the whole memefficiency.tcl: pass rate 100% (100/100)

Resolves #2398 , the "defrag didn't stop" issue, with help from @JimB123
@madolson

---------

Signed-off-by: Alina Liu <liusalisa6363@gmail.com>
Signed-off-by: asagegeLiu <liusalisa6363@gmail.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-10-01 20:37:15 -07:00
Madelyn OlsonandGitHub 47b5536183 Implement a lolwut for version 9 (#2646)
As requested, here is a version of lolwut for 9 that visualizes a Julia
set with ASCII art.

Example:
```
127.0.0.1:6379> lolwut version 9
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                     .............                              
                                 ......................                         
                              ............................                      
                           ......:::--:::::::::::::::.......                    
                         .....:::=+*@@@=--------=+===--::....                   
                        ....:::-+@@@@&*+=====+%@@@@@@@@=-::....                 
                      .....:::-=+@@@@@%%*++*@@@@@@@@@&*=--::....                
                     .....::--=++#@@@@@@@@##@@@@@@@@@@@@@@=::....               
                    ......:-=@&#&@@@@@@@@@@@@@@@@@@@@@@@@@%-::...               
                   ......::-+@@@@@@@@@@@@@@@@@@&&@@@#%#&@@@-::....              
                  .......::-=+%@@@@@@@@@@@@@@@@#%%*+++++%@+-:.....              
                  .......::-=@&@@@@@@@@@@@@@@@@&*++=====---::.....              
                 .......:::--*@@@@@@@@@@@@@@@@@%++===----::::.....              
                ........::::-=+*%&@@@@@@@@@&&&%*+==----:::::......              
                ........::::--=+@@@@@@@@@@&##%*++==---:::::.......              
                .......:::::---=+#@@@@@@@@&&&#%*+==---:::::.......              
               ........:::::---=++*%%#&&@@@@@@@@@+=---::::........              
               .......:::::----=++*%##&@@@@@@@@@@%+=--::::.......               
               ......::::-----==++#@@@@@@@@@@@@@&%*+=-:::........               
               ......:::---====++*@@@@@@@@@@@@@@@@@@+-:::.......                
               .....::-=++==+++**%@@@@@@@@@@@@@@@@#*=--::.......                
                ....:-%@@%****%###&@@@@@@@@@@@@@@@@&+--:.......                 
                ....:-=@@@@@&@@@@@@@@@@@@@@@@@@@@@@@@=::......                  
                 ...::+@@@@@@@@@@@@@@@&&@@@@@@@@%**@+-::.....                   
                 ....::-=+%#@@@@@@@@@&%%%&@@@@@@*==-:::.....                    
                  ....::--+%@@@@@@@%++==++*#@@@@&=-:::....                      
                   ....:::-*@**@@+==----==*%@@@@+-:::....                       
                     .....:::---::::::::--=+@=--::.....                         
                       .........::::::::::::::.......                           
                         .........................                              
                             ..................                                 
                                    ...                                         
                                                                                
                                                                                
                                                                                
Ascii representation of Julia set with constant 0.41 + 0.29i
Don't forget to have fun! Valkey ver. 255.255.255
```

You can pass in arbitrary rows and colums (it's best when rows is 2x
number of columns) and an arbitrary julia constant so it is repeatable.
Worst case it takes about ~100us on my m2 macbook, which should be fine
to make sure it's not taking too many system resources.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-09-30 08:25:53 +02:00
Ran ShidlansikandGitHub 2b2a42f57b Fix module key memory usage accounting (#2661)
Make objectComputeSize account for the key size as well when the key is
a module datatype

fixes: https://github.com/valkey-io/valkey/issues/2660

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-09-29 12:49:42 +02:00
ebf85053c7 Fix atomic slot migration snapshot never proceeding with hz 1 (#2636)
The problem is that ACKs run on a set loop (once every second) and this
will happen every loop with hz 1.

Instead, we can do the ACK after running the main logic. We can also do
an additional optimization where we don't send an ACK from source to
target if we already sent some data this cron loop, since the target
will reset the ack timer on any data over the connection.

---------

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-09-29 12:42:25 +02:00
3ada90ff8e Redirect blocked clients after failover (#2329)
In standalone mode, after a switch over, the command that was originally
blocking on primary returns -REDIRECT instead of -UNBLOCKED when the
client has the redirect capability.

Similarly, in cluster mode, after a switch over, the blocked commands
receive a -MOVED redirect instead of -UNBLOCKED.

After this fix, the handling of blocked connections during a switch over
between standalone and cluster is nearly identical. This can be
summarized as follows:

Standalone: 

1. Client that has the redirect capability blocked on the key on the
primary node will receive a -REDIRECT after the switch over completes
instead of -UNBLOCKED.
2. Readonly clients blocked on the primary or replica node will remain
blocked throughout the switch over.

Cluster:

1. Client blocked on the key served by the primary node will receive a
-MOVED instead of a probabilistic -UNBLOCKED error.
2. Readonly clients blocked on the key served by primary or replica node
will remain blocked throughout the switch over.

---------

Signed-off-by: cjx-zar <56825069+cjx-zar@users.noreply.github.com>
Co-authored-by: Simon Baatz <gmbnomis@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-09-29 11:13:06 +02:00
BinbinandGitHub c408fca487 Minor fix for dual rdb channel connection conn error log (#2658)
This should be server.repl_rdb_transfer_s

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-09-28 19:50:34 +08:00
88a168959e Add atomic slot migration test for unblock on migration complete (#2637)
This is already handled by `clusterRedirectBlockedClientIfNeeded`. With
the work we are doing in #2329, it makes sense to have an explicit test here
to prevent regression.

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-09-26 10:25:41 +08:00
317d40c728 Increasing retries to allow succcessful meet in Valgrind (#2644)
There is a daily test failure in valgrind, which looks like an issue related to
slowness in valgrind mode.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-09-25 10:23:12 +08:00
chzhooandGitHub 550bc0b9ca Optimize skiplist random level generation logic (#2631)
Each insertion of a skiplist node requires generating a random level
(via the `zslRandomLevel` function), and some commands (such as
`zunionstore`) call the `zslRandomLevel` function multiple times.
Therefore, optimizing `zslRandomLevel` can significantly improve the
performance of these commands.

The main optimization approach is as follows:

1. Original logic: Each iteration called the `random` function, with a
0.25 probability of continuing to call `random` again. In the worst-case
scenario, it required up to 32 calls (though the probability of this
occurring is extremely low).
2. Optimized logic: We only need to call the `genrand64_int64` function
once. Each iteration uses only 2 bits of randomness, effectively
achieving the equivalent of 32 iterations in the original algorithm.
3. Additionally, the introduction of `__builtin_clzll` significantly
reduces CPU usage, which compiles into a single, highly efficient CPU
instruction (e.g., LZCNT on x86, CLZ on ARM) on supported hardware
platforms
4. Although I've explained a lot, the actual code changes are quite
minimal, so just look at the code directly.

---------

Signed-off-by: chzhoo <czawyx@163.com>
2025-09-22 14:11:49 +02:00
korjeekandGitHub 42ae34e452 Adding unit tests for sha256 (#2632)
Adding comprehensive unit tests for SHA-256 implementation.
 
 These tests verify:
  1. Basic functionality with known test vectors (e.g., "abc")
  2. Handling of large input data (4KB repeated 1000 times)
3. Edge case with repeated single-byte input (1 million 'a' characters)
 
The tests ensure compatibility with standard SHA-256 implementations and
will help detect regressions during future code changes.
2025-09-21 18:34:30 +03:00
Sarthak AggarwalandGitHub 5a3359c33a Fix closing slot migration pipe read (#2630)
We probably should close the correct `slot_migration_pipe_read`. It
should resolve the valgrind errors.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-09-20 08:44:51 +08:00
Ricardo DiasandGitHub 5fd71bd144 Fix test that checks extended-redis-compatibility config deprecation rules (#2629)
Following the decision in #2189, we need to fix this test because the
`extended-redis-compatibility` config option is not going to be
deprecated in 9.0.

This commit changes the test to postpone the deprecation of
`extended-redis-compatibility` until 10.0 release.

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-09-19 19:37:09 +01:00
a9a699011b Fix flaky cluster flush slot test (#2626)
The reason is that the replication stream may not have yet reached
the replica for execution. We could add a wait_for_condition. We decided
to replace those assert calls with assert_replication_stream to verify
the contents of the replication stream rather than the commandstats.
```
*** [err]: Flush slot command propagated to replica in tests/unit/cluster/cluster-flush-slot.tcl
```

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-09-19 10:16:43 +08:00
Roshan KhatriandGitHub a2097721b7 Update automated benchmarking configs (#2625)
reduce the req and warmup time to finish in 6 hrs as the github workflow
times out after 6 hrs

---------

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2025-09-18 19:42:13 +02:00
4e4823907a Separate RDB snapshotting from atomic slot migration (#2533)
When we adding atomic slot migration in #1949, we reused a lot of rdb save code,
it was an easier way to implement ASM in the first time, but it comes with some
side effect. Like we are using CHILD_TYPE_RDB to do the fork, we use rdb.c/rdb.h
function to save the snapshot, these mess up the logs (we will print some logs
saying we are doing RDB stuff) and mess up the info fields (we will say we are
rdb_bgsave_in_progress but actually we are doing slot migration).

In addition, it makes the code difficult to maintain. The rdb_save method uses
a lot of rdb_* variables, but we are actually doing slot migration. If we want
to support one fork with multiple target nodes, we need to rewrite these code
for a better cleanup.

Note that the changes to rdb.c/rdb.h are reverting previous changes from when
we was reusing this code for slot migration. The slot migration snapshot logic
is similar to the previous diskless replication. We use pipe to transfer the
snapshot data from the child process to the parent process.

Interface changes:
- New slot_migration_fork_in_progress info field.
- New cow_size field in CLUSTER GETSLOTMIGRATIONS command.
- Also add slot migration fork to the cluster class trace latency.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Jacob Murphy <jkmurphy@google.com>
Co-authored-by: Jacob Murphy <jkmurphy@google.com>
2025-09-18 16:26:42 +08:00
uriyageandGitHub acd6298d24 Fix memory leak in deferred reply buffer (#2615)
Set free method for deferred_reply list to properly clean up 
ClientReplyValue objects when the list is destroyed

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
2025-09-17 11:14:44 +03:00
Roshan KhatriandGitHub c8b1bc1277 Adds io-threads configs to PR-perf tests (#2598)
- Adds io-thread enabled perf-tests for pr
- changes the server and benchmark client cpu ranges so there are on
separate NUMA nodes of the metal machine.
- Also kill any servers that are active on the metal machine if anything
fails.
- Adds a benchmark wf to benchmark versions and publish on a issue id
provided:
<img width="340" height="449" alt="Screenshot 2025-09-11 at 12 14 28 PM"
src="https://github.com/user-attachments/assets/04f6a781-e163-4d6b-9b70-deedad15c9ef"
/>

- Comments on the issue with the full comparison like this:
 
<img width="936" height="1152" alt="Screenshot 2025-09-11 at 12 15
35 PM"
src="https://github.com/user-attachments/assets/e1584c8e-25dc-433f-a4d4-5b08d7548ddf"
/>

https://github.com/roshkhatri/valkey/pull/3#issuecomment-3282289440

---------

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2025-09-16 14:09:39 -07:00
Sarthak AggarwalandGitHub 98d0b999d1 Increase wait time condition for New Master down consecutively test (#2612)
With #2604 merged, the `Node #10 should eventually replicate node #5`
started passing successfully with valgrind, but I guess we are seeing a
new daily failure from a `New Master down consecutively` test that runs
shortly after.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-09-16 10:55:27 +08:00
Sarthak AggarwalandGitHub 2b1e597095 Fix accounting for dual channel RDB bytes in replication stats (#2602)
Resolves #2545 

Followed the steps to reproduce the issue, and was able to get non-zero
`total_net_repl_output_bytes`.

```
(base) ~/workspace/valkey git:[fix-bug-2545]
src/valkey-cli INFO | grep total_net_repl_output_bytes
total_net_repl_output_bytes:1788
```

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-09-15 17:08:59 -07:00
b105baf882 Expand wait condition time for slave selection test (#2604)
## Summary
- extend replication wait time in `slave-selection` test

```
*** [err]: Node #10 should eventually replicate node #5 in tests/unit/cluster/slave-selection.tcl
#10 didn't became slave of #5
```

## Testing
- `./runtest --single unit/cluster/slave-selection`
- `./runtest --single unit/cluster/slave-selection --valgrind`

Signed-off-by: Vitali Arbuzov <Vitali.Arbuzov@proton.me>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
2025-09-13 15:53:07 +08:00
Jacob MurphyandGitHub cd3335e2a1 Make modules opt-in to atomic slot migration and add server events (#2593)
As discussed in https://github.com/valkey-io/valkey/issues/2579

Notably, I am exposing this feature as "Atomic Slot Migration" to
modules. If we want to call it something else, we should consider that
now (e.g.. "Replication-Based Slot Migration"?)

Also note: I am exposing both target and source node in the event. It is
not guaranteed that either target or source would be the node the event
fires on (e.g. replicas will fire the event after replica containment is
introduced). Even though it could be potentially inferred from CLUSTER
SLOTS, it should help modules parse it this way. Modules should be able
to infer whether it is occurring on primary/replica from `ctx` flags, so
not duplicating that here.

Closes #2579

---------

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-09-12 12:35:18 -07:00
Sarthak AggarwalandGitHub b4fc036d8b Evict client only when limit is breached (#2596)
I believe we should evict the clients when the client eviction limit is
breached instead of _at_ the breach. I came across this function in the
failed [daily
test](https://github.com/valkey-io/valkey/actions/runs/17521272806/job/49765359298#step:6:7770),
which could possibly be related.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-09-11 17:19:05 +03:00
9dc404375e Increase frequency of time check during fields active expiration (#2595)
When we introduced the new Hash fields expiration functionality,
we decided to combine the current active expiration job between generic
keys and hash fields.
During that job we run a tight loop. In each loop iteration we scan over
maximum of 20 keys (with default expire effort) and try to "expire"
them.
For hash fields expiration job, the "expire" of a hash key, means
expiring maximum of 80 fields (with default expire effort).
The problem is that we might do much more work per each iteration of
hash fields expiration job.
The current code is shared between the 2 jobs, and currently we only
perform time check every 16 iterations.
as a result the CPU of fields active expiration can spike and consume
much higher CPU% than the current 25% bound allows.

Example:

Before this PR

| Scenario | AVG CPU | Time to expire all fields |

|----------------------------------------------------|---------|---------------------------|
| Expiring 10M volatile fields from a single hash | 20.18% | 26 seconds
|
| Expiring 10M volatile fields from 10K hash objects | 32.72% | 7
seconds |


After this PR
Scenario | AVG CPU | Time to expire all fields
| Scenario | AVG CPU | Time to expire all fields |

|----------------------------------------------------|---------|---------------------------|
| Expiring 10M volatile fields from a single hash | 20.23%. | 26 seconds
|
| Expiring 10M volatile fields from 10K hash objects | 20.76%. | 11
seconds |

*NOTE*
The change introduced here make the field job check the time every
iteration. We offer compile time option to use efficient time check
using TSC (X86) or VCR (ARM) on most modern CPU, so the impact is
expected to be low. Still, in order to avoid degradation for existing
workloads, the code change was made so it will not impact the existing
generic keys active expiration job.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-09-10 13:24:55 +03:00
Zhijun LiaoandGitHub dbcdf43339 valkey-cli: Add word-jump navigation (Alt/Option/Ctrl + ←/→) (#2583)
Interactive use of valkey-cli often involves working with long keys
(e.g. MY:INCREDIBLY:LONG:keythattakesalongtimetotype). In shells like
bash, zsh, or psql, users can quickly move the cursor word by word with
**Alt/Option+Left/Right**, **Ctrl+Left/Right** or **Alt/Option+b/f**.
This makes editing long commands much more efficient.

Until now, valkey-cli (via linenoise) only supported single-character
cursor moves, which is painful for frequent key editing.

This patch adds such support, with simple code changes in linenoise. It
now supports both the Meta (Alt/Option) style and CSI (control sequence
introducer) style:

| | Meta style | CSI style (Ctrl) | CSI style (Alt) |
| --------------- | ---------- | ---------------- | --------- |
| move word left  | ESC b      | ESC [1;5D        | ESC [1;3D |
| move word right | ESC f      | ESC [1;5C        | ESC [1;3C |

Notice that I handle these two styles differently since people have
different preference on the definition of "what is a word".
Specifically, I define:
- "sub-word": just letters and digits. For example "my:namespace:key"
has 3 sub-words. This is handled by Meta style.
- "big-word": as any character that is not space. For example
"my:namespace:key" is just one single big-word. This is handled by CSI
style.


## How I verified

I'm using MacOS default terminal (`$TERM = xterm-256color`). I
customized the terminal keyboard setting to map option + left to `\033b`
, and ctrl + left to `\033[1;5D` so that I can produce both the Meta
style and CSI style. This code change should also work for
Linux/BSD/other terminal users.

Now the valkey-cli works like the following. `|` shows where the cursor
is currently at.

Press Alt + left (escape sequence `ESC b` ):

```
set cache:item itemid
                   |

set cache:item itemid
               |

set cache:item itemid
          |

set cache:item itemid
    |

set cache:item itemid
|
```

Press Ctrl + left (escape sequence `ESC [1;5D` ):

```
set cache:item itemid
                    |

set cache:item itemid
               |

set cache:item itemid
    |

set cache:item itemid
|
```

Press Alt + right (escape sequence `ESC f` ):

```
set cache:item itemid
|

set cache:item itemid
    |

set cache:item itemid
          |	

set cache:item itemid
               |

set cache:item itemid
                     |
```

Press Ctrl + right  (escape sequence `ESC [1;5C` ):

```
set cache:item itemid
|

set cache:item itemid
   |
    
set cache:item itemid
              |    

set cache:item itemid
                     |
```

---------

Signed-off-by: Zhijun <dszhijun@gmail.com>
2025-09-09 11:11:08 +02:00
c56a111835 Add cluster-announce-client-(port|tls-port) configs (#2429)
New config options:

 * cluster-announce-client-port
 * cluster-announce-client-tls-port

If enabled, clients will always get to see the configured port for a
node instead of the internally announced port(s), the same way that
`cluster-announce-client-ipv4` and `cluster-announce-client-ipv6` work.
Cluster-internal communication uses the non-client variant of these
options.

The configuration is propagated throughout the cluster using new ping
extensions.

Closes #2377

---------

Signed-off-by: Marvin Rösch <marvinroesch99@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-09-08 18:38:53 +02:00
BinbinandGitHub f15eadbeaa Skip codeql-analysis ci on documentation changes as well (#2567)
Follow #2393.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-09-08 19:31:51 +08:00
Rain ValentineandGitHub f9615808ee ARM Neon SIMD optimization for hashtable findBucket() (#2573)
This PR resolves #2519. I worked with @ahmadbelb to get a pretty good
result. 😁 For `hashtableFind()` I measured 58% speed improvement on a
AWS t4g.2xlarge instance, and Ahmad measured 159% speed improvement on a
M1 Mac.

I'm still working on valkey-benchmark results for GET and SET
throughput.

I used Google Benchmark to make micro benchmarks that test 0% 50% and
100% hit rates for hashtableFind(). You can check out the test code in
this branch:
https://github.com/SoftlyRaining/valkey/tree/valkey-microbench

My AWS t4g.2xlarge micro benchmark results:
```
Scalar version:
Running ./valkey-microbench
Run on (8 X 243.75 MHz CPU s)
CPU Caches:
  L1 Data 64 KiB (x8)
  L1 Instruction 64 KiB (x8)
  L2 Unified 1024 KiB (x8)
  L3 Unified 32768 KiB (x1)
Load Average: 6.04, 2.10, 1.14
---------------------------------------------------------------------------------------------------
Benchmark                                                         Time             CPU   Iterations
---------------------------------------------------------------------------------------------------
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                50.6 ns         50.6 ns    138342333
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                50.4 ns         50.4 ns    138342333
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                50.7 ns         50.7 ns    138342333
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                50.4 ns         50.4 ns    138342333
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                50.5 ns         50.4 ns    138342333
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_mean           50.5 ns         50.5 ns            5
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_median         50.5 ns         50.4 ns            5
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_stddev        0.118 ns        0.118 ns            5
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_cv             0.23 %          0.23 %             5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               56.6 ns         56.6 ns    123370046
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               56.8 ns         56.8 ns    123370046
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               55.8 ns         55.8 ns    123370046
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               56.4 ns         56.4 ns    123370046
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               56.3 ns         56.3 ns    123370046
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_mean          56.4 ns         56.4 ns            5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_median        56.4 ns         56.4 ns            5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_stddev       0.363 ns        0.363 ns            5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_cv            0.64 %          0.64 %             5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              50.5 ns         50.5 ns    138420288
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              50.4 ns         50.4 ns    138420288
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              50.4 ns         50.4 ns    138420288
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              50.5 ns         50.5 ns    138420288
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              50.4 ns         50.4 ns    138420288
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_mean         50.5 ns         50.4 ns            5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_median       50.4 ns         50.4 ns            5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_stddev      0.061 ns        0.062 ns            5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_cv           0.12 %          0.12 %             5

Neon version:
Running ./valkey-microbench
Run on (8 X 243.75 MHz CPU s)
CPU Caches:
  L1 Data 64 KiB (x8)
  L1 Instruction 64 KiB (x8)
  L2 Unified 1024 KiB (x8)
  L3 Unified 32768 KiB (x1)
Load Average: 0.35, 0.87, 0.72
---------------------------------------------------------------------------------------------------
Benchmark                                                         Time             CPU   Iterations
---------------------------------------------------------------------------------------------------
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                30.7 ns         30.7 ns    226597253
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                30.9 ns         30.9 ns    226597253
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                30.7 ns         30.6 ns    226597253
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                30.8 ns         30.8 ns    226597253
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                30.6 ns         30.6 ns    226597253
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_mean           30.7 ns         30.7 ns            5
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_median         30.7 ns         30.7 ns            5
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_stddev        0.118 ns        0.118 ns            5
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_cv             0.38 %          0.38 %             5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               36.6 ns         36.6 ns    192568222
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               35.2 ns         35.2 ns    192568222
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               34.7 ns         34.7 ns    192568222
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               36.2 ns         36.2 ns    192568222
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               36.0 ns         36.0 ns    192568222
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_mean          35.7 ns         35.7 ns            5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_median        36.0 ns         36.0 ns            5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_stddev       0.790 ns        0.789 ns            5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_cv            2.21 %          2.21 %             5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              30.5 ns         30.5 ns    229190934
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              31.4 ns         31.4 ns    229190934
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              30.7 ns         30.7 ns    229190934
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              30.4 ns         30.4 ns    229190934
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              31.1 ns         31.1 ns    229190934
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_mean         30.8 ns         30.8 ns            5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_median       30.7 ns         30.7 ns            5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_stddev      0.415 ns        0.414 ns            5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_cv           1.35 %          1.34 %             5
```

Ahmad's M1 Mac micro benchmark results:
```
Scalar version:
Running ./src/valkey-microbench
Run on (8 X 24 MHz CPU s)
CPU Caches:
  L1 Data 64 KiB
  L1 Instruction 128 KiB
  L2 Unified 4096 KiB (x8)
Load Average: 4.84, 8.36, 8.53
---------------------------------------------------------------------------------------------------
Benchmark                                                         Time             CPU   Iterations
---------------------------------------------------------------------------------------------------
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                29.8 ns         29.5 ns    238219371
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                30.6 ns         29.9 ns    238219371
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                29.9 ns         29.6 ns    238219371
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                29.7 ns         29.4 ns    238219371
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                29.3 ns         29.3 ns    238219371
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_mean           29.8 ns         29.5 ns            5
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_median         29.8 ns         29.5 ns            5
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_stddev        0.468 ns        0.250 ns            5
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_cv             1.57 %          0.85 %             5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               30.8 ns         30.7 ns    228068003
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               31.3 ns         31.0 ns    228068003
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               31.3 ns         31.0 ns    228068003
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               30.9 ns         30.8 ns    228068003
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               31.4 ns         31.1 ns    228068003
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_mean          31.1 ns         30.9 ns            5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_median        31.3 ns         31.0 ns            5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_stddev       0.288 ns        0.134 ns            5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_cv            0.92 %          0.43 %             5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              29.3 ns         29.3 ns    237946966
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              29.4 ns         29.4 ns    237946966
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              29.4 ns         29.4 ns    237946966
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              29.3 ns         29.3 ns    237946966
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              29.4 ns         29.4 ns    237946966
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_mean         29.4 ns         29.4 ns            5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_median       29.4 ns         29.4 ns            5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_stddev      0.058 ns        0.061 ns            5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_cv           0.20 %          0.21 %             5

NEON version:
Running ./src/valkey-microbench
Run on (8 X 24 MHz CPU s)
CPU Caches:
  L1 Data 64 KiB
  L1 Instruction 128 KiB
  L2 Unified 4096 KiB (x8)
Load Average: 4.56, 5.43, 7.09
---------------------------------------------------------------------------------------------------
Benchmark                                                         Time             CPU   Iterations
---------------------------------------------------------------------------------------------------
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                11.6 ns         11.6 ns    596879005
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                11.9 ns         11.7 ns    596879005
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                11.9 ns         11.8 ns    596879005
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                11.7 ns         11.7 ns    596879005
BM_HashtableFind_0Miss/min_time:5.000/repeats:5                11.7 ns         11.7 ns    596879005
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_mean           11.8 ns         11.7 ns            5
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_median         11.7 ns         11.7 ns            5
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_stddev        0.119 ns        0.069 ns            5
BM_HashtableFind_0Miss/min_time:5.000/repeats:5_cv             1.01 %          0.59 %             5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               12.0 ns         11.9 ns    592642763
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               11.9 ns         11.9 ns    592642763
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               12.0 ns         12.0 ns    592642763
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               12.2 ns         12.1 ns    592642763
BM_HashtableFind_50Miss/min_time:5.000/repeats:5               11.9 ns         11.9 ns    592642763
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_mean          12.0 ns         12.0 ns            5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_median        12.0 ns         11.9 ns            5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_stddev       0.113 ns        0.069 ns            5
BM_HashtableFind_50Miss/min_time:5.000/repeats:5_cv            0.94 %          0.58 %             5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              11.9 ns         11.8 ns    590288406
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              11.9 ns         11.9 ns    590288406
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              12.0 ns         11.9 ns    590288406
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              11.9 ns         11.8 ns    590288406
BM_HashtableFind_100Miss/min_time:5.000/repeats:5              12.6 ns         12.1 ns    590288406
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_mean         12.1 ns         11.9 ns            5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_median       11.9 ns         11.9 ns            5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_stddev      0.303 ns        0.135 ns            5
BM_HashtableFind_100Miss/min_time:5.000/repeats:5_cv           2.51 %          1.14 %             5
```

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
2025-09-07 09:52:21 +03:00
Viktor SöderqvistandGitHub 6ff5e6316e Relaxed RDB check for foreign RDB formats (#2543)
In #1604, we attempt to read future Valkey RDB formats, but we rejected
foreign RDB formats, because of the risk that the opcodes and types
added by other projects collide with the new types and opcodes added in
recent Valkey versions.

This change accepts foreign RDB versions but limits the types and
opcodes to the ones that we can understand, to prevent misinterpretation
of types/opcodes which could lead to undefined behavior. If unsupported
RDB types or opcodes are seen, we error out.

Additional changes:

* Improve error reporting when encountering unknown RDB types in relaxed
version check mode.
* Tests for loading various RDB files.
* Improvement to valkey-check-rdb to accept future and foreign RDB
versions, including tests.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-09-04 17:19:52 +02:00
ce85149903 Un-deprecate commands (#2546)
As per #2459, this PR removes deprecation and `deprecated_since`
element and `DEPRECATED` doc flag from commands. Closes #2459.

---------

Signed-off-by: Kyle J. Davis <kyledvs@amazon.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-09-03 12:39:49 -07:00
Ted LyngmoandGitHub 720b1aef5b Don't use AVX2 instructions if the CPU don't support it (#2571)
`bitops.c`: `serverPopcount()` used `popcountAVX2()`, which as the name
implies requires AVX2 support, on AVX-only machines, causing an "illegal
instruction" error.

Added a `__builtin_cpu_supports("avx2")` check and falling back to the
platform agnostic version if AVX2 is not supported.

Fixes #2570

Signed-off-by: Ted Lyngmo <ted@lyncon.se>
2025-09-03 18:25:33 +02:00
aab38d1080 Store number of keys with volatile items per slot in RDB aux field and pre-size hashtables on load (#2572)
In Valkey 9.0 we added HFE support which is currently using a per slot
hashtable in order to track keys (hash objects) which contains volatile
fields. in order to optimize the RDB load we should use the same method
for expires and generic keys kvstores.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-09-03 10:03:49 +03:00
asagegeLiuandGitHub e9367f9dc7 Delete the previous comment explaining why changed latency from 5 to 40 (#2574)
Delete the out-of-date comment explaining why changed latency from 5 to
40 in #2421 , which is a leftover of #2553

Signed-off-by: Alina Liu <liusalisa6363@gmail.com>
2025-09-02 16:36:02 -07:00
BinbinandGitHub 7eff31fe1c Remove deny_blocking check in moduleBlockClient, cleanup code and doc (#2215)
This flag.deny_blocking check causes some code to becomde dead code.
Some of the code below checks islua or ismulti, but they are marked
with flag.deny_blocking and will return early.

In addition, cleanup some documents, some of them are inaccurate,
and restore the code of blocking_auth_cb in tests/modules/auth.c,
this reply should be returned from core. The reply used to from the
core and was changed to from the module in #1819, and now it is from
the core again.

Cleanup some dead code around #1819.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-09-02 09:08:14 +08:00
Ricardo DiasandGitHub 5d2ba76816 New module API event for tracking authentication attempts (#2237)
In this commit we introduce a new module API event called
`ValkeyModuleEvent_AuthenticationAttempt` to track successful/failed
authentication attempts.

This event will fill a struct, called `ValkeyModuleAuthenticationInfo`,
with the client ID of the connection, the username, the module name that
handle the authentication event, and the result of the authentication
attempt.

Fixes: #2211

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-09-01 15:02:26 +01:00
Ricardo DiasandGitHub fbf6a35f0c Fix module context object re-usage in scripting engines (#2358)
This commit refactors the scripting engine to support multiple cached
module contexts per engine, rather than relying on a single cached
`ValkeyModuleCtx` object.

Previously, having only one cached context object caused data races over
the state stored in the context object, because it's possible that a
script that is running for a long time to yield and the server event
loop may call the `scriptingEngineCallGetMemoryInfo` function to get the
scripting engine memory information, which re-uses the same cached
context object. Another possible data-race is caused by the asynchronous
scripts flush, which calls the `scriptingEngineCallFreeFunction`
function in an background thread, and also re-uses the cached context
object.

To address this, a cache array of module contexts was introduced in the
scripting engine structure, with each slot dedicated to a specific use
case—such as script execution, memory info queries, or function freeing.

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-09-01 11:35:37 +01:00
BinbinandGitHub 7a6f991761 Add jacob as a commiter (#2566)
Add @murphyjacob4 as one of the folks with write permissions on the
Valkey repo.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-30 18:06:12 +08:00
bf2c3e2538 Reduce active defrag test latency by lowering hit threshold (#2553)
Changed the defrag hit threshold from 512 to 1 in the `defragLaterStep`
& `defragStageKvstoreHelper` function to reduce defrag latency. (idea
from Jim)

1. Previously, the defrag loop would continue processing up to 512
successful defragmentations before checking if the time limit was
exceeded. Now it checks after every single successful allocation move.
2. The trade-off is slightly more frequent time checks, but the time
check (~19ns) is negligible compared to the actual defragmentation work
(even a single 8-byte reallocation takes ~43ns and the
allocatorShouldDefrag function call takes ~49ns per block). This
overhead is minimal compared to the latency improvement gained from
better time management during active defragmentation.
3. Also revert the change from
https://github.com/valkey-io/valkey/pull/2421/files to test.
4. Solved compilation issue with unsigned by changing the type of the
local variable `prev_defragged `to match the type of
`server.stat_active_defrag_hits`

Closes #2444

---------

Signed-off-by: Alina Liu <alinalq@dev-dsk-alinalq-2b-2db84246.us-west-2.amazon.com>
Co-authored-by: Alina Liu <alinalq@dev-dsk-alinalq-2b-2db84246.us-west-2.amazon.com>
2025-08-29 21:32:33 +02:00
zhaozhao.zzandGitHub 43af8c3da1 Fix the issue of incorrect commandlog metrics in the script (#2565)
In script, client will be replaced with its caller, so commandlog needs
to use the metrics of the client that currently executing the command.

Signed-off-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
2025-08-29 10:47:10 +08:00
withRiverandGitHub a1aa1bde44 Reset cluster related stats in CONFIG RESETSTATS (#2458)
Previously CONFIG RESETSTATS only resets the slot statistics in
cluster part, this PR makes it reset cluster bus messages at well.
Additionally, we also reset stat_cluster_links_buffer_limit_exceeded.

Now we will reset:
- cluster_stats_messages_sent*
- cluster_stats_messages_received*
- total_cluster_links_buffer_limit_exceeded

Closes #2439.

Signed-off-by: Hongrui <2086160503@qq.com>
2025-08-29 10:21:23 +08:00
BinbinandGitHub 27224f46a3 Split SLOT_EXPORT_AUTHENTICATING into SEND and READ to avoid synchronous reading of auth response (#2494)
The old SLOT_EXPORT_AUTHENTICATING added in #1949, when processed by the source node,
we will send the AUTH command and then reads the response. If the target node is blocked
during this process, the source node will also be blocked. We should use a read handler
to handle this. We split SLOT_EXPORT_AUTHENTICATING into SLOT_EXPORT_SEND_AUTH and
SLOT_EXPORT_READ_AUTH_RESPONSE to avoid this issue.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-29 10:20:50 +08:00
Marc JakobiandGitHub e08064dc8c Correct path to gen-test-certs.sh in README.md (#2554)
Signed-off-by: Marc Jakobi <marc.jakobi@tiko.energy>
2025-08-29 10:19:46 +08:00
BinbinandGitHub 4d9cefa21e Do not migrate function in new atomic slot migration (#2547)
If all cluster nodes have functions, slot migration will fail
since the target will return the function already exists error
when doing the FUNCTION LOAD.

And in addition, the target's replica could panic when it executes
the FUNCTION LOAD propagated from the primary (see
propagation-error-behavior).

Introduced in #1949.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-29 10:18:45 +08:00
BinbinandGitHub b3a060783b Remove the trailing chars of the --cluster reshard log message in valkey-cli (#2560)
It prints an extra ": " after the message, which is a bit weird, i
thought it was printing cluster-bus port information, but it was not.
```
Moving slot 5458 from 127.0.0.1:30001 to 127.0.0.1:30003:
Moving slot 5459 from 127.0.0.1:30001 to 127.0.0.1:30003:
Moving slot 5460 from 127.0.0.1:30001 to 127.0.0.1:30003:
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-29 10:17:10 +08:00
Viktor SöderqvistandGitHub 033bb557c5 Attempt to fix sub-replica getting out of sync (#2548)
Try to fix the failures seen for `test "PSYNC2 #3899 regression: verify
consistency"`.

This change resets the query buffer parser state in
`replicationCachePrimary()` which is called when the connection to the
primary is lost. Before #2092, this was done by `resetClient()`.

The solution was inspired by the discussion about the regression
mentioned (discussion from 2017) and the related commits from that time:
3dd60a77d06510ff4b1d524a14731837b9aa0d4b,
62845798cc,
8baea97eff.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-08-27 07:24:00 +02:00
BinbinandGitHub bba6cf8289 Update tests/xxx/tmp/.gitignore to ignore everything (#2542)
Ingore everything on its own directory.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-27 10:17:48 +08:00
Rain ValentineandGitHub ad77081a03 deflake test: relax time requirement in hash ttl test (#2537)
tiny change. It failed once for me (a little time passed and it returned
2 seconds instead of 3), so I figured it's probably a little flaky for
others too

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
2025-08-26 09:02:14 +02:00
4b7116af99 Update reply schema for LMOVE and BLMOVE (#2541)
Both LMOVE and BLMOVE can return null values because the source key is
empty.
PR changes include 
- change the LMOVE reply_schema to include the possibility of a nil
return value
- Add comment to BLMOVE reply_schema to indicate it can return nil
because the source does not exist

This fixes #2532

---------

Signed-off-by: Adam Fowler <adamfowler71@gmail.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-26 09:01:12 +02:00
Ted LyngmoandGitHub c1974048c7 bio.c: Organize all worker data in a struct (#2530)
This gets rid of the need to use a void* as a carrier for the worker
number. Instead a pointer to the relevant worker data is passed to the
started thread.

Fixes #2529

---------

Signed-off-by: Ted Lyngmo <ted@lyncon.se>
2025-08-25 12:24:34 +02:00
Madelyn OlsonandGitHub 699ade8c7b Consistently use static_assert across code (#2538)
In C99, we had to use `#define static_assert(expr, lit) extern char
__static_assert_failure[(expr) ? 1 : -1]` for static assertions.
However, we now have native support for static_assert with
_Static_assert. Previously one of the correct #defines was getting
called first, setting it to _Static_assert, however after
https://github.com/valkey-io/valkey/commit/116de0a7940cc56c01b2f30f3e7070cef3d5eaf7
the first import defining the symbol was "serverassert.h", which
included the old style.

This change removes all unnecessary imports and always defines
static_assert as _Static_assert.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-08-24 14:59:13 +02:00
BinbinandGitHub 7675bad92c Don't allow slot migration to myself node (#2497)
This may result in meaningless slot migration job, we should
return an error to user in advance to avoid operation error.
Also `by myself` is not correct English grammar and `myself`
is a internal code terminology, changed to `by this node`.

Was introduced in #1949.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-22 15:08:20 -04:00
7050353d9e Optimize pipelining by parsing and prefetching multiple commands (#2092)
Instead of parsing only one command per client before executing it,
parse multiple commands from the query buffer and batch-prefetch the
keys accessed by the commands in the queue before executing them.

This is an optimization for pipelined commands, both with and without
I/O threads. The optimization is currently disabled for the replication
stream, due to failures (probably caused by how the replication offset
is calculated based on the query buffer offset).

* When parsing commands from the input buffer, multiple commands are
parsed and stored in a command queue per client.
* In single-threaded mode (I/O threads off) keys are batch-prefetched
before the commands in the queue are executed. Multi-key commands like
MGET, MSET and DEL benefit from this even if pipelining is not used.
* Prefetching when I/O threads are used does prefetching for multiple
clients in parallel. This code takes client command queues into account,
improving prefetching when pipelining is used. The batch size is
controlled by the existing config `prefetch-batch-max-size` (default
16), which so far only was used together with I/O threads. The config is
moved to a different section in `valkey.conf`.
* When I/O threads are used and the maximum number of keys are
prefetched, a client's command is executed, then the next one in the
queue, etc. If there are more commands in the queue for which the keys
have not been prefetched (say the client sends 16 pipelined MGET with 16
keys in each) keys for the next few commands in the queue are prefetched
before the commands is executed if prefetching has not been done for the
next command. (This utilizes the code path used in single-threaded
mode.)

Code improvements:

* Decoupling of command parser state and command execution state:
  * The variables reqtype, multibulklen and bulklen refer to the current
    position in the query buffer. These are no longer reset in resetClient
    (which runs after each command being executed). Instead, they are
    reset in the parser code after each completely parsed command.
  * The command parser code is partially decoupled from the client struct.
    The query buffer is still one per client, but the resulting argument
    vector is stored in caller-defined variables.

Fixes #2044

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-08-22 16:02:27 +02:00
BinbinandGitHub 2767af1de0 Remove debug logging from cluster-flush-slot.tcl (#2535)
I believe this was a debug log at the time, and its printing
was quite annoying locally. The test is quite simple so i think
we can just remove it.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-22 17:19:44 +08:00
57880e5da7 Adds benchmark on demand workflow (#2442)
# Adds On-demand Benchmark Workflow

This PR introduces a new GitHub Actions workflow that enables on-demand
performance benchmarking for pull requests through label-based triggers.
This uses the new framework
[valkey-perf-benchmark](https://github.com/valkey-io/valkey-perf-benchmark)
developed for standard benchmarks and PR benchmarking.

## What is being added by this PR

- **Workflow File**: `.github/workflows/benchmark_on_label.yml`
- **Trigger**: Activated when specific labels are added to PRs
- **Supported Labels**:
  - `run-benchmark` - Runs standard benchmarks
  
## Features

### On-Demand Execution
- Benchmarks run only when explicitly requested via PR labels
- No automatic execution to avoid unnecessary resource usage

### Performance Comparison
- Compares PR performance against the `unstable` baseline
- Generates detailed comparison reports
- Posts results directly as PR comments for easy review

### Flexible Configuration
- Currently uses github runners by will use dedicated performance
runners (`ec2-perf-ubuntu-24`)
- Configurable benchmark suites via JSON config files

### Artifact Management
- Uploads benchmark results as workflow artifacts
- Preserves both baseline and PR metrics for analysis
- Includes comparison markdown for offline review

### Automatic Cleanup
- Removes trigger labels after execution (success or failure)
- Prevents accidental re-runs from stale labels

## Usage

To run benchmarks on a PR:

1. Add the `run-benchmark` label for standalone and cluster, non-tls
mode
2. Wait for the workflow to complete
3. Review results in the automated PR comment

This workflow enhances our CI/CD pipeline by providing easy access to
performance testing without impacting regular development workflows.

---------

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Signed-off-by: Roshan Khatri <117414976+roshkhatri@users.noreply.github.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
2025-08-21 17:31:17 -07:00
Allen SamuelsandGitHub 72517363ae Module API: Add READONLY flag to ClientInfo.flags output structure (#2522)
* Add a flag `VALKEYMODULE_CLIENTINFO_FLAG_READONLY` to
  `ValkeyModuleClientInfo.flags` and set it in
  `ValkeyModule_GetClientInfoById()`.
* Add an optimization for accessing the current client by ID, to avoid
  looking it up in a radix tree.

Closes #2487

---------

Signed-off-by: Allen Samuels <allenss@amazon.com>
2025-08-21 22:09:22 +02:00
a1486e068c Wait for log message occurrence in module test on message received (#2517)
## Problem
Test `Cluster module receive message API -
VM_RegisterClusterMessageReceiver` fails sporadically in CI with
"Expected '2' to be equal to '1'". The test failed in Daily CI 4 days
ago but hasn't failed since, indicating flaky behavior that I cannot
reproduce locally.


## Hypothesis
The failing line `assert_equal 2 [count_log_message 0 "* <cluster> DONG
(type 2) RECEIVED*"]` counts DONG message entries in node 0's log file
and expects exactly 2. The failure suggests a possible race condition
where there's a timing gap between when cluster statistics are updated
and when the corresponding log entries become visible in the log file.


## Solution
Add `wait_for_condition` to ensure log messages are written before
checking count:

```tcl
wait_for_condition 50 100 {
    [count_log_message 0 "* <cluster> DONG (type 2) RECEIVED*"] eq 2
} else {
    fail "node 1 didn't log 2 DONG messages"
}

---------

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
2025-08-21 12:34:27 -07:00
Sarthak AggarwalandGitHub 5d5c4243f1 Fix slot range lists overlap to rewind the nested list again (#2527)
In the current implementation, the second list was never rewound again
once it was iterated. So for the first element of `ranges1`, `ranges2`
was iterated fully. But when the second element of `ranges1` was
processed, the `ranges2` was not rewound again.

With this change, for every element of `ranges1`, we start from the
beginning for `ranges2`

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-08-21 11:02:29 -07:00
Björn SvenssonandViktor Söderqvist 005994a919 Remove temporary build correction for RDMA and libvalkey 0.1.0
Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
2025-08-21 12:35:10 +02:00
Björn SvenssonandViktor Söderqvist 57b77b02a3 Update deps/libvalkey to version 0.2.1
Squashed 'deps/libvalkey/' changes from abcd27fbf..b012f8e85

b012f8e85 Release 0.2.1
9e10acbf7 Fix duplicate Acks for RDMA events. (#229)
1eadedf48 Remove the unused valDup API from dict
a449f0ea1 Don't expose internal functions in shared libraries (#205)
178e350c7 Cluster code cleanup (#216)
4020396c8 Fix `unused-parameter` warning when building with `NDEBUG` (#212)
99aa158bc Use existing connections for blocking slotmap updates (#199)
969a8c546 Fix dependency issue with RDMA (#201)
...

git-subtree-dir: deps/libvalkey
git-subtree-split: b012f8e85a9cc2c68bc2be982a4f6545c15042f0

Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
2025-08-21 12:35:10 +02:00
asagegeLiuandGitHub 57282d3daf Refactor scanLaterList to fix latency issue (#2514) 2025-08-20 17:18:44 -07:00
Ted LyngmoandGitHub 005c5b16db Fix assumptions that pthread functions set errno (#2526)
pthread functions return the error instead of setting errno.

Fixes #2525

Signed-off-by: Ted Lyngmo <ted@lyncon.se>
2025-08-20 22:48:31 +02:00
Sarthak AggarwalandGitHub d7e394e1ba Fix total test count while running over loop (#2524)
Command: `./runtest --single unit/bitops --loops 3`

Unstable

```
[ignore]: large memory flag not provided
[-1/1 done]: unit/bitops (4 seconds)
[ignore]: large memory flag not provided
[0/1 done]: unit/bitops (4 seconds)
[ignore]: large memory flag not provided
[1/1 done]: unit/bitops (4 seconds)

                   The End

Execution time of different units:
  4 seconds - unit/bitops
  4 seconds - unit/bitops
  4 seconds - unit/bitops
```

After fix

```
[1/3 done]: unit/bitops (4 seconds)
[ignore]: large memory flag not provided
[2/3 done]: unit/bitops (4 seconds)
[ignore]: large memory flag not provided
[3/3 done]: unit/bitops (4 seconds)

                   The End

Execution time of different units:
  4 seconds - unit/bitops
  4 seconds - unit/bitops
  4 seconds - unit/bitops
```

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-08-20 11:16:37 -07:00
6eaa63beb6 fix hsetex handling of wrong number of fields (#2509)
currently hsetex is verifying the number of fields matches the provided
number of fields by using div. instead it can match to the
multiplication in order to prevent rounding the verified value down.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-08-20 14:54:11 +03:00
Ran ShidlansikandGitHub 04dd5a9f8b Fix vset unittest compilation warning for bad signedness comparison (#2523)
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-20 11:34:58 +02:00
Harkrishn PatroandGitHub ab3874dc11 Update pong_received time via gossip only if the node is healthy from our view (#2512) 2025-08-19 21:38:46 -07:00
Harkrishn PatroandGitHub 5efb1cad70 Atomically update cluster and replication layer while marking self node as primary (#2510)
Fixes: #2460

With this change we avoid divergence in cluster and replication layer
view. I've observed node can be marked as primary in cluster while it
can be marked as replica in replication layer view and have a active
replication link. Without this change, we used to end up in a invalid
replica chain in replication layer.

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-08-19 17:28:51 -07:00
Seungmin LeeandGitHub 0d4c7ddb6f Skip failure reports for already failed nodes (#2434)
This change avoids additional failure report creation if the node is
already marked as failed. The failure report count has never been used
after a node has been marked as failed. So, there is no value addition
in maintaining it further. This reduces operation of both add and delete
failure report. Hence, the performance benefit.

We can observe an avg. of 10% reduction in p99 CPU utilization (in a 2000
nodes cluster (1000 primary/ 1000 replica) with 330 nodes in failed
state with this change.

---------

Signed-off-by: Seungmin Lee <sungming@amazon.com>
2025-08-19 15:24:26 -07:00
Hanxi ZhangandGitHub 59ac9d523f Add auto-author-assign workflow (#2410)
Fixes #2372

## Description

This PR adds an automated workflow that assigns PR authors to their own
pull requests when opened or reopened.

## Changes

- Added `.github/workflows/auto-author-assign.yml` workflow
- Uses `toshimaru/auto-author-assign@v2.1.1` action
- Triggers on `pull_request_target` events (opened, reopened)
- Requires `pull-requests: write` permission

## Benefits

- Improves PR tracking and organization
- Automatically assigns responsibility to PR authors
- Reduces manual assignment overhead for maintainers
- Helps contributors track their own PRs more easily

## Testing

 **Tested on my fork before submission:**

1. Merged the workflow into my fork's unstable branch
2. Created a test PR within my fork
3. Verified that I was automatically assigned as the assignee
4. Screenshot showing successful auto-assignment:
<img width="1278" height="684" alt="Screenshot 2025-08-01 at 3 39 05 PM"
src="https://github.com/user-attachments/assets/9ad643be-5eac-4ad6-bec7-184cf22e9cbd"
/>

The workflow executed successfully and assigned me to the test PR as
expected.

---------

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
2025-08-19 13:49:25 -07:00
BinbinandGitHub 7deb17d385 Fix slot-info expand not working, kvstoreHashtableExpand always creat… (#2466)
If we want to expand kvstoreHashtableExpand, we need to make sure the
hashtable exists. Currently, when processing RDB slot-info, our expand
has no effect because the hashtable does not exist (we initialize it only
when we need it).

We also update kvstoreExpand to use the kvstoreHashtableExpand to make
sure there is only one code path. Also see #1199 for more details.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-19 21:01:56 +08:00
Ran ShidlansikandGitHub 2141e81b86 fix hash ignore ttl management during active expiry (#2505)
several fixes:
1. fix not using bool input type for hashTypeIgnoreTTL - probably lost
during the 3 HFE PR merges
2. remove vset change hashtable encoding to single - The current code is
a bug. The entry is probably expired (or about to be expired soon) so we
can leave it as a hashtable till it does.
3. enable incremental rehashing for volatile item keys kvstore - This is
the center issue of this PR. without it the activeexpiry might not scan
the kvstore which is very fragmented with lots of empty buckets.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-19 11:31:37 +03:00
BinbinandGitHub 15a77bcc72 CLUSTER SYNCSLOTS ESTABLISH added source node == myself check (#2500)
Minor cleanup.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-19 10:26:58 +08:00
Viktor SöderqvistandGitHub dcbe421caa Make cluster failover delay relative to node timeout (#2449)
In clusters with a very short node timeout such as 2-3 seconds, the
extra failover delay of 500-1000 milliseconds (500 + random value 0-500;
total 750 on average) before initiating a failover is a significant
extra downtime to the cluster. This PR makes this delay relative to node
timeout, using a shorter failover delay for a smaller configured node
timeout. The formula is `fixed_delay = min(500, node_timeout / 30)`.

| Node timeout  | Fixed failover delay |
|---------------|----------------------|
| 15000 or more | 500 (same as before) |
| 7500          | 250                  |
| 3000          | 100                  |
| 1500          | 50                   |

Additional change: Add an extra 500ms delay to new replicas that may not
yet know about the other replicas. This avoids the scenario where a new
replica with no data wins the failover. This change turned out to be
needed to for the stability of some test cases.

The purposes of the failover delay are

1. Allow FAIL to propagate to the voting primaries in the cluster
2. Allow replicas to exchange their offsets, so they will have a correct
view of their own rank.

A third (undocumented) purpose of this delay is to allow newly added
replicas to discover other replicas in the cluster via gossip and to
compute their rank, to realize it's are not the best replica. This case
is mitigated by adding another 500ms delay to new replicas, i.e. if it
has replication offset 0.

A low node timeout only makes sense in fast networks, so we can assume
that the above needs less time than in a cluster with a higher node
timeout.

These delays don't affect the correctness of the algorithm. They are
just there to increase the probability that a failover will succeed by
making sure that the FAIL message has enough time to propagate in the
cluster and to the random part is to reduce the probability that two
replicas initiates the failover at the same time.

The typical use case is when data consistency matters and writes can't
be skipped. For example, in some application, we buffer writes in the
application during node failures to be able to apply them when the
failover is completed. The application can't buffer them for a very long
time, so we need the cluster to be up again within e.g. 5 seconds from
the time a node starts to fail.

I hope this PR can be considered safer than #2227, although the two
changes are orthogonal.

Part of issue #2023.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-08-18 15:34:51 +02:00
BinbinandGitHub def63a89d9 Fix memory leak in saveSnapshotToConnectionSockets (#2503)
We now pass in rdbSnapshotOptions options in this function, and
options.conns
is now malloc'ed in the caller side, so we need to zfree it when
returning early
due to an error. Previously, conns was malloc'ed after the error
handling, so we
don't have this.

Introduced in #1949.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-18 12:59:40 +02:00
yzc-yzcandGitHub f89eff03f9 CONFIG GET command return sorted output (#2493)
Previously, the config names and values were stored in a temporary dict.
This ensures that no duplicates are returned, but it also makes the
order random.

In this PR, the config names and values still stored in the temporary
dict, but then they are copied to an array, which is sorted, before the
reply is sent.

Resolves #2042

---------

Signed-off-by: yzc-yzc <96833212+yzc-yzc@users.noreply.github.com>
2025-08-18 11:00:13 +02:00
BinbinandGitHub a923abf8e5 Fix memory leak in rdbLoadObject when loading a wrong HFE (#2502)
There are memory leaks when we return NULL.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-18 07:18:32 +03:00
Sarthak AggarwalandGitHub df37a7a891 Add test failure template to contributing guide (#2491)
We recently introduced a new template to create `test failures` issues
from a template. This change makes this template visible in the
`CONTRIBUTING.md` file. Also, added a tip to paste the stack trace since
outputs of CI links can expire.

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-08-17 11:58:32 +02:00
Ran ShidlansikandGitHub a56a803f28 simplify COPY Preserves TTLs hashexpire test (#2495)
Simplifies a test case seen to be flaky.

fixes: https://github.com/valkey-io/valkey/issues/2482

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-17 11:53:41 +02:00
Hanxi ZhangandGitHub c3fdf8375a Fix cluster test module to pass null terminated node-id to SendClusterMessage (#2484)
Fix https://github.com/valkey-io/valkey/issues/2438

Modified `DingReceiver` function in `tests/modules/cluster.c` by adding
null-termination logic for cross-version compatibility

---------

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
2025-08-15 20:40:34 +02:00
yzc-yzcandGitHub 621836f2db Don't call SSL_write() with num=0 (#2490)
https://github.com/valkey-io/valkey/blob/bac953b099c4293db213b2b233e0967befcb8f9e/src/networking.c#L2279-L2293
From above code, we can see that `c->repl_data->ref_block_pos` could be
equal to `o->used`.
When `o->used == o->size`, we may call SSL_write() with num=0 which does
not comply with the openSSL specification.
(ref: https://docs.openssl.org/master/man3/SSL_write/#warnings)

What's worse is that it's still the case after the reconnection. See
https://github.com/valkey-io/valkey/blob/bac953b099c4293db213b2b233e0967befcb8f9e/src/replication.c#L756-L769.
So in this case the replica will keep reconnecting again and again until
it doesn't meet the requirements for partial synchronization.

Resolves #2119

---------

Signed-off-by: yzc-yzc <96833212+yzc-yzc@users.noreply.github.com>
2025-08-15 20:37:15 +02:00
Viktor SöderqvistandGitHub 972f94953c Fix timeout in defrag tests (#2483)
* Use pipelines of length 1000 instead of up to 200000.
* Use CLIENT REPLY OFF instead of reading and discarding the replies.

Fixes #2205

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-08-15 20:28:14 +02:00
BinbinandGitHub bac953b099 Don't allow resize hashtable if rehashing is ongoing (#2465)
Similar to dicts, we disallow resizing while the hashtable is
rehashing. In the previous code, if a resize was triggered during
rehashing, like if the rehashing wasn't fast enough, we would do
a while loop until the rehashing was complete, which could be a
potential issue when doing resize.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-15 11:55:57 +02:00
Sarthak AggarwalandGitHub dfeef31090 Ensures presence of slots on the node before test is run (#2486)
The change will ensure that the slot is present on the node before the
slot is populated. This will avoid the errors during populating the
slot.

Resolves #2480

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-08-15 11:23:34 +02:00
dbcfcc4e71 Add bug / test-failure / enhancement label to issue template (#2273)
Automatically attach respective label to newly filled issues.
---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthakaggarwal97@gmail.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
2025-08-14 19:47:38 -07:00
Sarthak AggarwalandGitHub daf37f0dee Fixing Slot Migration Test Failure (#2485)
Make sure slot migration has finished before moving on to the next test.

Resolves #2479

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-08-14 15:20:13 -07:00
Ping XieandGitHub 4ebd5c1022 Consolidate slot migration logs by grouping consecutive slot migrations into a single log entry (#2481)
Previously, each slot migration was logged individually, which could
lead to log spam in scenarios where many slots are migrated at once.

This commit enhances the logging mechanism to group consecutive slot
migrations into a single log entry, improving log readability and
reducing noise.

Log snippets

```
1661951:S 13 Aug 2025 15:47:10.132 * Slot range [16383, 16383] is migrated from node c3926da75f7c3a0a1bcd07e088b0bde09d48024c () in shard 7746b693330c0814178b90b757e2711ebb8c6609 to node 2465c29c8afb9231525e281e5825684d0bb79f7b () in shard 39342c039d2a6c7ef0ff96314b230dfd7737d646.
1661951:S 13 Aug 2025 15:47:10.289 * Slot range [10924, 16383] is migrated from node 2465c29c8afb9231525e281e5825684d0bb79f7b () in shard 39342c039d2a6c7ef0ff96314b230dfd7737d646 to node c3926da75f7c3a0a1bcd07e088b0bde09d48024c () in shard 7746b693330c0814178b90b757e2711ebb8c6609.
1661951:S 13 Aug 2025 15:47:10.524 * Slot range [10924, 16383] is migrated from node c3926da75f7c3a0a1bcd07e088b0bde09d48024c () in shard 7746b693330c0814178b90b757e2711ebb8c6609 to node 2465c29c8afb9231525e281e5825684d0bb79f7b () in shard 39342c039d2a6c7ef0ff96314b230dfd7737d646.
```

---------

Signed-off-by: Ping Xie <pingxie@google.com>
2025-08-14 13:15:52 -07:00
BinbinandGitHub 39db3a5485 Remove if condition and disable the new failover test (#2477)
In #2431 we changed the assert to a if condition, and the test cause
some trouble, now we just remove the assert (if condition) and disable
the test for now due to #2441.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-13 17:01:00 +08:00
Ran ShidlansikandGitHub ce93cb58e7 HSETEX 'hset' notification should only be generated if not expired (#2475)
Currently HSETEX always generate `hset` notification. In order to align
with generic `set` command, it should only generate `hset` if the
provided time-to-live is a valid future time.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-13 10:22:44 +03:00
Ran ShidlansikandGitHub 606e146a71 Increment expired_fields stat when assigned TTL is in the past (#2474)
fixes: https://github.com/valkey-io/valkey/issues/2461

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-13 09:37:54 +03:00
ruihong123andGitHub c8666f6545 Fix duplicate Acks for RDMA events and fix extremely large max latency for RDMA benchmark. (#2430)
(1) The old logic may result in the RDMA event being acknowledged
unexpectly in the following two scenarios.
* ibv_get_cq_event get an EAGAIN error. 
* ibv_get_cq_event get one event but it may ack multiple times in the
pollcq loop.

(2) In the benchmark result of valkey over RDMA, the tail latency is as
high as 177 milliseconds(almost 80x of TCP). This results from incorrect
benchmark client setup which includes the connection setup time into the
benchmark latency recording. This patch fixes this crazy tail latency
issue by modifying the valkey-benchmark.c. This change only affects
benchmark over RDMA as updates are regulated under Macro USE_RDMA.

There are following updates on valkey RDMA but I am willing to create
separated pull requests.

---------

Signed-off-by: Ruihong Wang <ruihong@google.com>
2025-08-13 09:28:44 +03:00
Ran ShidlansikandGitHub de2b83dc54 Deflake hashexpire tests (#2473)
1. Better separation of test steps in `Chain Replication (Primary -> R1
-> R2) preserves TTL`
This can help prevent or provide better understanding of a flakey fail:
https://github.com/valkey-io/valkey/actions/runs/16867976482/job/47777607814

2. increase the millisecond short timeout to 1 second since some tests
are failing because of it. It also better matches the second timeout.
example fail:
https://github.com/valkey-io/valkey/actions/runs/16867976482/job/47777607746

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-12 13:32:19 +03:00
BinbinandGitHub 73305e13f4 Change the same shard failover assert to if condition to avoid crash (#2431)
The assert was added in #2301 and we found that there are some situations
would trigger assert and crash the server.

The reason we added the assert is because, in the code:
1. sender_claimed_primary and sender are in the same shard
2. and sender is the old primary, sender_claimed_primary is the old replica
3. and now sender become a replica, sender_claimed_primary become a primary

That means a failover happend in the shard, and sender should be the primary
of sender_claimed_primary. But obviously this assumption may be wrong, we rely
on shard_id to determine whether it is in a same shard, and assume that a shard
can only have one primary.

But this is wrong, from #2279 we can know there will be a case that we can
create two primaries in the same shard due to the untimely update of shard_id.
So we can create a test that trigger the assert in this way:
1. pre condition: two primaries in the same shard, one has slots and one is empty.
2. replica doing a cluster failover
3. the empty primary doing a cluster replicate with the replica (new primary)

We change the assert to an if condition to fix it.

Closes #2423.

Note that the test written here also exposes the issue in #2441, so these two
may need to be addressed together.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-12 10:37:11 +08:00
BinbinandGitHub 2f71f2e4ff Add test for failover sub-replica replication loop case (#2456)
When we added the safeguard against sub-replicas logic, we didn't add
much tests for it. As time has shown, it can happen in different scenarios.
Here's a test case that used to happen in a failover scenario.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-12 10:21:06 +08:00
3b70f50c7b Introduce atomic slot migration (#1949)
Introduces a new family of commands for migrating slots via replication.
The procedure is driven by the source node which pushes an AOF formatted
snapshot of the slots to the target, followed by a replication stream of
changes on that slot (a la manual failover).

This solution is an adaptation of the solution provided by
@enjoy-binbin, combined with the solution I previously posted at #1591,
modified to meet the designs we had outlined in #23.

## New commands

* `CLUSTER MIGRATESLOTS SLOTSRANGE start end [start end]... NODE
node-id`: Begin sending the slot via replication to the target. Multiple
targets can be specified by repeating `SLOTSRANGE ... NODE ...`
*  `CLUSTER CANCELMIGRATION ALL`: Cancel all slot migrations
* `CLUSTER GETSLOTMIGRATIONS`: See a recent log of migrations

This PR only implements "one shot" semantics with an asynchronous model.
Later, "two phase" (e.g. slot level replicate/failover commands) can be
added with the same core.

## Slot migration jobs

Introduces the concept of a slot migration job. While active, a job
tracks a connection created by the source to the target over which the
contents of the slots are sent. This connection is used for control
messages as well as replicated slot data. Each job is given a 40
character random name to help uniquely identify it.

All jobs, including those that finished recently, can be observed using
the `CLUSTER GETSLOTMIGRATIONS` command.

## Replication

* Since the snapshot uses AOF, the snapshot can be replayed verbatim to
any replicas of the target node.
* We use the same proxying mechanism used for chaining replication to
copy the content sent by the source node directly to the replica nodes.

## `CLUSTER SYNCSLOTS`

To coordinate the state machine transitions across the two nodes, a new
command is added, `CLUSTER SYNCSLOTS`, that performs this control flow.

Each end of the slot migration connection is expected to install a read
handler in order to handle `CLUSTER SYNCSLOTS` commands:

* `ESTABLISH`: Begins a slot migration. Provides slot migration
information to the target and authorizes the connection to write to
unowned slots.
* `SNAPSHOT-EOF`: appended to the end of the snapshot to signal that the
snapshot is done being written to the target.
* `PAUSE`: informs the source node to pause whenever it gets the
opportunity
* `PAUSED`: added to the end of the client output buffer when the pause
is performed. The pause is only performed after the buffer shrinks below
a configurable size
* `REQUEST-FAILOVER`: request the source to either grant or deny a
failover for the slot migration. The grant is only granted if the target
is still paused. Once a failover is granted, the paused is refreshed for
a short duration
* `FAILOVER-GRANTED`: sent to the target to inform that REQUEST-FAILOVER
is granted
* `ACK`: heartbeat command used to ensure liveness

## Interactions with other commands

* FLUSHDB on the source node (which flushes the migrating slot) will
result in the source dropping the connection, which will flush the slot
on the target and reset the state machine back to the beginning. The
subsequent retry should very quickly succeed (it is now empty)
* FLUSHDB on the target will fail the slot migration. We can iterate
with better handling, but for now it is expected that the operator would
retry.
* Genearlly, FLUSHDB is expected to be executed cluster wide, so
preserving partially migrated slots doesn't make much sense
* SCAN and KEYS are filtered to avoid exposing importing slot data

## Error handling

* For any transient connection drops, the migration will be failed and
require the user to retry.
* If there is an OOM while reading from the import connection, we will
fail the import, which will drop the importing slot data
* If there is a client output buffer limit reached on the source node,
it will drop the connection, which will cause the migration to fail
* If at any point the export loses ownership or either node is failed
over, a callback will be triggered on both ends of the migration to fail
the import. The import will not reattempt with a new owner
* The two ends of the migration are routinely pinging each other with
SYNCSLOTS ACK messages. If at any point there is no interaction on the
connection for longer than `repl-timeout`, the connection will be
dropped, resulting in migration failure
* If a failover happens, we will drop keys in all unowned slots. The
migration does not persist through failovers and would need to be
retried on the new source/target.

## State machine

```
                                                                            
                Target/Importing Node State Machine                         
   ─────────────────────────────────────────────────────────────            
                                                                            
             ┌────────────────────┐
             │SLOT_IMPORT_WAIT_ACK┼──────┐
             └──────────┬─────────┘      │
                     ACK│                │
         ┌──────────────▼─────────────┐  │
         │SLOT_IMPORT_RECEIVE_SNAPSHOT┼──┤
         └──────────────┬─────────────┘  │
            SNAPSHOT-EOF│                │                                  
        ┌───────────────▼──────────────┐ │                                  
        │SLOT_IMPORT_WAITING_FOR_PAUSED┼─┤                                  
        └───────────────┬──────────────┘ │                                  
                  PAUSED│                │                                  
        ┌───────────────▼──────────────┐ │ Error Conditions:                
        │SLOT_IMPORT_FAILOVER_REQUESTED┼─┤  1. OOM                          
        └───────────────┬──────────────┘ │  2. Slot Ownership Change        
        FAILOVER-GRANTED│                │  3. Demotion to replica          
         ┌──────────────▼─────────────┐  │  4. FLUSHDB                      
         │SLOT_IMPORT_FAILOVER_GRANTED┼──┤  5. Connection Lost              
         └──────────────┬─────────────┘  │  6. No ACK from source (timeout) 
      Takeover Performed│                │                                  
         ┌──────────────▼───────────┐    │                                  
         │SLOT_MIGRATION_JOB_SUCCESS┼────┤                                  
         └──────────────────────────┘    │                                  
                                         │                                  
   ┌─────────────────────────────────────▼─┐                                
   │SLOT_IMPORT_FINISHED_WAITING_TO_CLEANUP│                                
   └────────────────────┬──────────────────┘                                
Unowned Slots Cleaned Up│                                                   
          ┌─────────────▼───────────┐                                      
          │SLOT_MIGRATION_JOB_FAILED│                                      
          └─────────────────────────┘                                      

                                                                                           
                                                                                           
                      Source/Exporting Node State Machine                                  
         ─────────────────────────────────────────────────────────────                     
                                                                                           
               ┌──────────────────────┐                                                    
               │SLOT_EXPORT_CONNECTING├─────────┐                                          
               └───────────┬──────────┘         │                                          
                  Connected│                    │                                          
             ┌─────────────▼────────────┐       │                                          
             │SLOT_EXPORT_AUTHENTICATING┼───────┤                                          
             └─────────────┬────────────┘       │                                          
              Authenticated│                    │                                          
             ┌─────────────▼────────────┐       │                                          
             │SLOT_EXPORT_SEND_ESTABLISH┼───────┤                                          
             └─────────────┬────────────┘       │                                          
  ESTABLISH command written│                    │                                          
     ┌─────────────────────▼─────────────┐      │                                          
     │SLOT_EXPORT_READ_ESTABLISH_RESPONSE┼──────┤                                          
     └─────────────────────┬─────────────┘      │                                          
   Full response read (+OK)│                    │                                          
          ┌────────────────▼──────────────┐     │ Error Conditions:                        
          │SLOT_EXPORT_WAITING_TO_SNAPSHOT┼─────┤  1. User sends CANCELMIGRATION           
          └────────────────┬──────────────┘     │  2. Slot ownership change                
     No other child process│                    │  3. Demotion to replica                  
              ┌────────────▼───────────┐        │  4. FLUSHDB                              
              │SLOT_EXPORT_SNAPSHOTTING┼────────┤  5. Connection Lost                      
              └────────────┬───────────┘        │  6. AUTH failed                          
              Snapshot done│                    │  7. ERR from ESTABLISH command           
               ┌───────────▼─────────┐          │  8. Unpaused before failover completed   
               │SLOT_EXPORT_STREAMING┼──────────┤  9. Snapshot failed (e.g. Child OOM)     
               └───────────┬─────────┘          │  10. No ack from target (timeout)        
                      PAUSE│                    │  11. Client output buffer overrun        
            ┌──────────────▼─────────────┐      │                                          
            │SLOT_EXPORT_WAITING_TO_PAUSE┼──────┤                                          
            └──────────────┬─────────────┘      │                                          
             Buffer drained│                    │                                          
            ┌──────────────▼────────────┐       │                                          
            │SLOT_EXPORT_FAILOVER_PAUSED┼───────┤                                          
            └──────────────┬────────────┘       │                                          
   Failover request granted│                    │                                          
           ┌───────────────▼────────────┐       │                                          
           │SLOT_EXPORT_FAILOVER_GRANTED┼───────┤                                          
           └───────────────┬────────────┘       │                                          
      New topology received│                    │                                          
            ┌──────────────▼───────────┐        │                                          
            │SLOT_MIGRATION_JOB_SUCCESS│        │                                          
            └──────────────────────────┘        │                                          
                                                │                                          
            ┌─────────────────────────┐         │                                          
            │SLOT_MIGRATION_JOB_FAILED│◄────────┤                                          
            └─────────────────────────┘         │                                          
                                                │                                          
           ┌────────────────────────────┐       │                                          
           │SLOT_MIGRATION_JOB_CANCELLED│◄──────┘                                          
           └────────────────────────────┘                                                 
```

Co-authored-by: Binbin <binloveplay1314@qq.com>

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Jacob Murphy <jkmurphy@google.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Ping Xie <pingxie@outlook.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-08-11 18:02:37 -07:00
Madelyn OlsonandGitHub 6cb97f2d1b Throw a useful error when underlying operations failed with tls (#2469)
Right now, if a TLS connect fails, you get an unhelpful error message in
the log since it prints out NULL. This change makes sure that report
error always returns a string (never null) as well as tries to print out
underlying errors.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-08-11 15:49:36 -07:00
VanessaTangandGitHub 8a8285b6d0 Redact user data when hide-user-data-from-log enabled (#2274)
### Description
User data logged when crash caused by moduleRDBLoadError.

### Change
Redact user data when hide-user-data-from-log enabled.

Signed-off-by: VanessaTang <yuetan@amazon.com>
2025-08-11 11:33:23 -07:00
cxljsandGitHub 0356a33ea8 Replace generic C assert with serverAssert (#2467)
Based on the review suggestion
https://github.com/valkey-io/valkey/pull/2432#discussion_r2258249206

---------

Signed-off-by: Xiaolong Chen <fukua95@gmail.com>
2025-08-11 11:03:12 -07:00
Ran ShidlansikandGitHub f7a4e06670 Fix out-of-bound memory access when num-fields is not provided (#2464)
Following new API presented in
https://github.com/valkey-io/valkey/pull/2089, we might access out of
bound memory in case of some illegal command input

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-11 12:24:50 +03:00
03b89e8b59 Use bool for return types in hashtable functions (#2432)
closes #2352

---------

Signed-off-by: Xiaolong Chen <fukua95@gmail.com>
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-11 10:13:26 +03:00
Ran ShidlansikandGitHub 9b1c6cfa4a Fix HGETEX to return array response when the hash object is empty (#2462)
In the current implementation `HGETEX`, when applied on a non existing
object, will simply return null value instead of an array (like in the
`HMGET` case).

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-11 07:16:18 +03:00
Jacob MurphyandGitHub bd45b4b99b Fix AOF rewrite behavior for hashes with expirations (#2454)
The bug caused many invalid HMSET entries to be added to the AOF during
rewriting.

This now properly skips emitting HMSET until we find an entry with no
expiry.

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-08-10 10:05:02 +03:00
Sarthak AggarwalandGitHub 60cc3d5242 Adding backup directory check for valkey-cli --cluster backup (#2452)
Adds a simple check for backup directory, remove the todo.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-08-09 00:12:40 +08:00
amanosmeandGitHub 275ffc004d Fix expectations for manual failover tests (#2453)
Test `Instance #5 is still a slave after some time (no failover)` is
supposed to verify that command `CLUSTER FAILOVER` will not promote a
replica without quorum from the primary; later in the file (`Instance 5
is a master after some time`), we verify that `CLUSTER FAILOVER FORCE`
does promote a replica under the same conditions.

There's a couple issues with the tests:

1. `Instance #5 is still a slave after some time (no failover)` should
verify that instance 5 is a replica (i.e. that there's no failover), but
we call `assert {[s -5 role] eq {master}}`.
2. The reason why the above assert works is that we previously send
`DEBUG SLEEP 10` to the primary, which pauses the primary for longer
than the configured 3 seconds for`cluster-node-timeout`.
The primary is marked as failed from the perspective of the rest of the
cluster, so quorum can be established and instance 5 is promoted as
primary.

This commit fixes the two by shortening the sleep to less than 3
seconds, and then asserting the role is still replica. Test `Instance #5
is a master after some time` is updated to sleep for a shorter duration
to ensure that `FAILOVER FORCE` succeeds under the exact same
conditions.

### Testing
`./runtest --single unit/cluster/manual-failover --loop --fastfail`

Signed-off-by: Tyler Amano-Smerling <amanosme@amazon.com>
2025-08-08 16:11:15 +08:00
Hanxi ZhangandGitHub 6548d4f59d Fix and remove conflicting paths from clang-format workflow (#2455)
Fixes the GitHub Actions error where both `paths` and `paths-ignore`
were defined for the same event, which is not allowed.

Resolves the error: "you may only define one of `paths` and
`paths-ignore` for a single event"

Removed the conflicting `paths` section from the `pull_request` trigger,
keeping only `paths-ignore` to skip documentation changes while allowing
the workflow to run on all other changes.

This is a follow-up fix to address the issue identified in the previous
PR.

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
2025-08-08 14:34:19 +08:00
Hanxi ZhangandGitHub 53e945c120 Add path filters to skip CI on documentation changes (#2393)
Fixes #626 - Skip CI tests when only documentation files are modified

---------

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
2025-08-07 15:50:22 -07:00
8bfaef3945 Increase latency for big list test in defrag (#2421)
Across multiple runs of the big list test in defrag, the latency check
is tripping because the maximum observed defrag cycle latency
occasionally spikes above our 5 ms limit. While most cycles complete in
just a few milliseconds, rare slowdowns push some cycles into the double
digit millisecond range, so a 5 ms hard cap is too aggressive for stable
testing.

```
/runtest --verbose --tls --single unit/memefficiency --only '/big list' --accurate --loop --fastfail
```

```
[err]: Active Defrag big list: standalone in tests/unit/memefficiency.tcl
Expected 12 <= 5 (context: type proc line 18 cmd {assert {$max_latency <= $limit_ms}} proc ::validate_latency level 1)
(Fast fail: test will exit now)

[err]: Active Defrag big list: standalone in tests/unit/memefficiency.tcl
Expected 21 <= 5 (context: type proc line 18 cmd {assert {$max_latency <= $limit_ms}} proc ::validate_latency level 1)
(Fast fail: test will exit now)

[err]: Active Defrag big list: standalone in tests/unit/memefficiency.tcl
Expected 25 <= 5 (context: type proc line 18 cmd {assert {$max_latency <= $limit_ms}} proc ::validate_latency level 1)
(Fast fail: test will exit now)

[err]: Active Defrag big list: standalone in tests/unit/memefficiency.tcl
Expected 17 <= 5 (context: type proc line 18 cmd {assert {$max_latency <= $limit_ms}} proc ::validate_latency level 1)
(Fast fail: test will exit now)

```

---------

Signed-off-by: Seungmin Lee <sungming@amazon.com>
Co-authored-by: Seungmin Lee <sungming@amazon.com>
2025-08-07 09:24:47 -07:00
Ran ShidlansikandGitHub d200fb4f41 Fix hashexpire test toggle active expire (#2447)
1. Fix the way we toggle active expire.
2. reduce the number of spawned servers during the test

Just merged some start_server blocks to reduce the time spent on
starting and shutting down the server.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-07 16:38:52 +03:00
Ran ShidlansikandGitHub 3ef7eff44a make sure replica and primary are in sync during hfe test Promotion to primary (#2446)
fixes: https://github.com/valkey-io/valkey/issues/2445

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-07 11:54:41 +03:00
cxljsandGitHub c2dfe2a7a2 Remove unused includes (#2344)
Remove unused includes

Signed-off-by: Xiaolong Chen <fukua95@gmail.com>
2025-08-06 13:42:09 -07:00
Ran ShidlansikandGitHub e71ef9ed63 verify expiration is set on hashexpire active expire test (#2440)
This might help shed some light on the flakey failure e.g:
https://github.com/valkey-io/valkey/actions/runs/16754537665/job/47433388425?pr=2431

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-06 20:31:34 +03:00
BinbinandGitHub b6e317066c Print error context when test fails (#2437)
We used to did print the context but after #2276, we lost the context.

unstable:
```
*** Extract version and sha1 details from info command and print in tests/unit/info-command.tcl
```

now:
```
*** [err]: Extract version and sha1 details from info command and print in tests/unit/info-command.tcl
Expected '0' to be equal to '1' (context: type source line 7 file /xxx/info-command.tcl cmd {assert_equal 0 1} proc ::test)
```

We can see the different, we have provided enough context when asserting
fail. Otherwise we need to scroll back (which is usually a lot of server
logs) to see the context.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-07 00:20:50 +08:00
0278010782 [CRASH] Fix missing check for executing client (#2347)
Fix missing check for executing client in `lookupKey` function

### Issue
The `lookupKey` function in db.c accesses
`server.executing_client->cmd->proc` without first verifying that
`server.executing_client` is not NULL. This was introduced in #1499
where the check for executing client was added without verifying it
could be null.

The server crashes with a null pointer dereference when the
current_client's flag.no_touch is set.
```
27719 valkey-server *
/lib64/libpthread.so.0(+0x118e0)[0x7f34cb96a8e0]
src/valkey-server 127.0.0.1:21113(lookupKey+0xf5)[0x4a14b7]
src/valkey-server 127.0.0.1:21113(lookupKeyReadWithFlags+0x50)[0x4a15fc]
src/valkey-server 127.0.0.1:21113[0x52b8f1]
src/valkey-server 127.0.0.1:21113(handleClientsBlockedOnKeys+0xa5)[0x52b16f]
src/valkey-server 127.0.0.1:21113(processCommand+0xf1e)[0x4712c9]
src/valkey-server 127.0.0.1:21113(processCommandAndResetClient+0x35)[0x490fd5]
src/valkey-server 127.0.0.1:21113(processInputBuffer+0xe1)[0x4912e5]
src/valkey-server 127.0.0.1:21113(readQueryFromClient+0x8c)[0x49177b]
src/valkey-server 127.0.0.1:21113[0x57daa6]
src/valkey-server 127.0.0.1:21113[0x57e280]
src/valkey-server 127.0.0.1:21113(aeProcessEvents+0x261)[0x45b259]
src/valkey-server 127.0.0.1:21113(aeMain+0x2a)[0x45b450]
src/valkey-server 127.0.0.1:21113(main+0xd43)[0x479bf6]
/lib64/libc.so.6(__libc_start_main+0xea)[0x7f34cb5cd13a]
src/valkey-server 127.0.0.1:21113(_start+0x2a)[0x454e3a]
```

### Fix
Added a null check for `server.executing_client` before attempting to
dereference it:

### Tests
Added a regression test in tests/unit/type/list.tcl.

---------

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-06 17:02:10 +03:00
Ran ShidlansikandGitHub 1a723613bc Fix hfe no malloc size unit (#2436)
test_vest uses a mock_defrag function in order to simulate defrag flow.
In the scenario of no system malloc_size we need to use
zmalloc_usable_size in order to prevent stepping over the
"to-be-replaced" buffer

will partially fix: https://github.com/valkey-io/valkey/issues/2435

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-06 12:33:55 +03:00
BinbinandGitHub b917c723d9 Add STALE command flag to SCRIPT-EXISTS, SCRIPT-SHOW and SCRIPT-FLUSH (#2419)
We marked SCRIPT-LOAD/EVAL* with STALE in
5d11bba172,
it is odd that we can load but won't be able to exists or show it.
Also it is technically ok since these commands doesn't relate directly
to the server's dataset.

Also since now we don't have the script replication, flush also seems
safe to add the flag.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-06 10:37:30 +08:00
Ran ShidlansikandGitHub 71bf0ce5e7 bump the RDB version for Valkey 9.0(#2422)
This is needed due to changes presented in
https://github.com/valkey-io/valkey/pull/2089

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-05 18:32:18 +03:00
Ran Shidlansik b50942f4cb Add ACTIVE-EXPIRY and ACTIVE-DEFRAG for hash objects with volatile items
This change adds support for active expiration of hash fields with TTLs (Hash Field Expiration), building on the existing key-level expiry system.
Field TTL metadata is tracked in volatile sets associated with each hash key. Expired fields are reclaimed incrementally by the active expiration loop, using a new job type to alternate between key expiry and field expiry within the same logic and effort budget.
Both key and field expiration now share the same scheduler infrastructure.

Alternating job types ensures fairness and avoids starvation, while keeping CPU usage predictable.

    +-----------------+
    |     DB          |
    +-----------------+
            |
            v
    +---------------------+
    |     myhash          |  (key with TTL)
    +---------------------+
            |
            v
    +------------------------------------+
    | fields (hashType)                  |
    |  - field1                          |
    |  - field2                          |
    |  - fieldN                          |
    +------------------------------------+
            |
            v
    +------------------------------------+
    | volatile set (field-level TTL)     |
    |  - field1 expires at T1            |
    |  - field5 expires at T5            |
    +------------------------------------+

No new configuration was introduced; the existing active-expire-effort and time budget are reused for both key and field expiry.

Also active defrag for volatile sets is added.

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-05 18:28:47 +03:00
Ran Shidlansik f728c05415 Introduce volatile set
-------------

   Overview:
   ---------
   This PR introduces a complete redesign of the 'vset' (stands for volatile set) data structure,
   creating an adaptive container for expiring entries. The new design is
   memory-efficient, scalable, and dynamically promotes/demotes its internal
   representation depending on runtime behavior and volume.

   The core concept uses a single tagged pointer (`expiry_buckets`) that encodes
   one of several internal structures:
       - NONE   (-1): Empty set
       - SINGLE (0x1): One entry
       - VECTOR (0x2): Sorted vector of entry pointers
       - HT     (0x4): Hash table for larger buckets with many entries
       - RAX    (0x6): Radix tree (keyed by aligned expiry timestamps)

   This allows the set to grow and shrink seamlessly while optimizing for both
   space and performance.

   Motivation:
   -----------
   The previous design lacked flexibility in high-churn environments or
   workloads with skewed expiry distributions. This redesign enables dynamic
   layout adjustment based on the time distribution and volume of the inserted
   entries, while maintaining fast expiry checks and minimal memory overhead.

   Key Concepts:
   -------------
   - All pointers stored in the structure must be   odd-aligned   to preserve
     3 bits for tagging. This is safe with SDS strings (which set the LSB).

   - Buckets evolve automatically:
       - Start as NONE.
       - On first insert → become SINGLE.
       - If another entry with similar expiry → promote to VECTOR.
       - If VECTOR exceeds 127 entries → convert to RAX.
       - If a RAX bucket's vector fills and cannot split → promote to HT.

   - Each vector bucket is kept sorted by `entry->getExpiry()`.
   - Binary search is used for efficient insertion and splitting.

 # Coarse Buckets Expiration System for Hash Fields

 This PR introduces **coarse-grained expiration buckets** to support per-field
 expirations in hash types — a feature known as *volatile fields*.

 It enables scalable expiration tracking by grouping fields into time-aligned
 buckets instead of individually tracking exact timestamps.

 ## Motivation
 Valkey traditionally supports key-level expiration. However, in many applications,
 there's a strong need to expire individual fields within a hash (e.g., session keys,
 token caches, etc.).

 Tracking these at fine granularity is expensive and potentially unscalable, so
 this implementation introduces *bucketed expirations* to batch expirations together.

 ## Bucket Granularity and Timestamp Handling

 - Each expiration bucket represents a time slice of fixed width (e.g., 8192 ms).
 - Expiring fields are mapped to the **end** of a time slice (not the floor).
 - This design facilitates:
   - Efficient *splitting* of large buckets when needed
   - *Downgrading* buckets when fields permit tighter packing
   - Coalescing during lazy cleanup or memory pressure

 ### Example Calculation

 Suppose a field has an expiration time of `1690000123456` ms and the max bucket
 interval is 8192 ms:

 ```
 BUCKET_INTERVAL_MAX = 8192;
 expiry = 1690000123456;

 bucket_ts = (expiry & ~(BUCKET_INTERVAL_MAX - 1LL)) + BUCKET_INTERVAL_MAX;
           = (1690000123456 & ~8191) + 8192
           = 1690000122880 + 8192
           = 1690000131072
 ```

 The field is stored in a bucket that **ends at** `1690000131072` ms.

 ### Bucket Alignment Diagram

 ```
 Time (ms) →
 |----------------|----------------|----------------|
  128ms buckets → 1690000122880    1690000131072
                     ^               ^
                     |               |
               expiry floor     assigned bucket end
 ```

 ## Bucket Placement Logic

 - If a suitable bucket **already exists** (i.e., its `end_ts > expiry`), the field is added.
 - If no bucket covers the `expiry`, a **new bucket** is created at the computed `end_ts`.

 ## Bucket Downgrade Conditions

 Buckets are downgraded to smaller intervals when overpopulated (>127 fields).
 This happens when **all fields fit into a tighter bucket**.

 Downgrade rule:
 ```
 (max_expiry & ~(BUCKET_INTERVAL_MIN - 1LL)) + BUCKET_INTERVAL_MIN < current_bucket_ts
 ```
 If the above holds, all fields can be moved to a tighter bucket interval.

 ### Downgrade Bucket — Diagram

 ```
 Before downgrade:

   Current Bucket (8192 ms)
   |----------------------------------------|
   | Field A | Field B | Field C | Field D  |
   | exp=+30 | +200    | +500    | +1500    |
   |----------------------------------------|
                     ↑
        All expiries fall before tighter boundary

 After downgrade to 1024 ms:

   New Bucket (1024 ms)
   |------------------|
   | A | B | C | D     |
   |------------------|
 ```

 ### Bucket Split Strategy

 If downgrade is not possible, the bucket is **split**:
 - Fields are sorted by expiration time.
 - A subset that fits in an earlier bucket is moved out.
 - Remaining fields stay in the original bucket.

 ### Split Bucket — Diagram

 ```
 Before split:

   Large Bucket (8192 ms)
   |--------------------------------------------------|
   | A | B | C | D | E | F | G | H | I | J | ... | Z  |
   |---------------- Sorted by expiry ---------------|
             ↑
      Fields A–L can be moved to an earlier bucket

 After split:

   Bucket 1 (end=1690000129024)     Bucket 2 (end=1690000131072)
   |------------------------|       |------------------------|
   | A | B | C | ... | L     |       | M | N | O | ... | Z    |
   |------------------------|       |------------------------|
 ```

 ## Summary of Bucket Behavior
 | Scenario                        | Action Taken                 |
 |--------------------------------|------------------------------|
 | No bucket covers expiry        | New bucket is created        |
 | Existing bucket fits           | Field is added               |
 | Bucket overflows (>127 fields) | Downgrade or split attempted |

API Changes:
------------

   Create/Free:
      void vsetInit(vset *set);
      void vsetClear(vset *set);

  Mutation:
      bool vsetAddEntry(vset *set, vsetGetExpiryFunc getExpiry, void *entry);
      bool vsetRemoveEntry(vset *set, vsetGetExpiryFunc getExpiry, void *entry);
      bool vsetUpdateEntry(vset *set, vsetGetExpiryFunc getExpiry, void *old_entry,
                                 void *new_entry, long long old_expiry,
                                 long long new_expiry);

  Expiry Retrieval:
      long long vsetEstimatedEarliestExpiry(vset *set, vsetGetExpiryFunc getExpiry);
      size_t vsetPopExpired(vset *set, vsetGetExpiryFunc getExpiry, vsetExpiryFunc expiryFunc, mstime_t now, size_t max_count, void *ctx);

  Utilities:
      bool vsetIsEmpty(vset *set);
      size_t vsetMemUsage(vset *set);

  Iteration:
      void vsetStart(vset *set, vsetIterator *it);
      bool vsetNext(vsetIterator *it, void **entryptr);
      void vsetStop(vsetIterator *it);

   Entry Requirements:
   -------------------
   All entries must conform to the following interface via `volatileEntryType`:

       sds entryGetKey(const void  entry);         // for deduplication
       long long getExpiry(const void  entry);     // used for bucketing
       int expire(void  db, void  o, void  entry); // used for expiration callbacks

   Diagrams:
   ---------

   1. Tagged Pointer Representation
      -----------------------------
      Lower 3 bits of `expiry_buckets` encode bucket type:

         +------------------------------+
         |     pointer       | TAG (3b) |
         +------------------------------+
           ↑
           masked via VSET_PTR_MASK

         TAG values:
           0x1 → SINGLE
           0x2 → VECTOR
           0x4 → HT
           0x6 → RAX

   2. Evolution of the Bucket
      ------------------------

 *Volatile set top-level structure:*

```
+--------+     +--------+     +--------+     +--------+
| NONE   | --> | SINGLE | --> | VECTOR | --> |   RAX  |
+--------+     +--------+     +--------+     +--------+
```

*If the top-level element is a RAX, it has child buckets of type:*

```
+--------+     +--------+     +-----------+
| SINGLE | --> | VECTOR | --> | HASHTABLE |
+--------+     +--------+     +-----------+
```

*Vectors can split into multiple vectors and shrink into SINGLE buckets. A RAX with only one element is collapsed by replacing the RAX with its single element on the top level (except for HASHTABLE buckets which are not allowed on the top level).*

   3. RAX Structure with Expiry-Aligned Keys
      --------------------------------------
      Buckets in RAX are indexed by aligned expiry timestamps:

         +------------------------------+
         | RAX key (bucket_ts) → Bucket|
         +------------------------------+
         |     0x00000020  → VECTOR     |
         |     0x00000040  → VECTOR     |
         |     0x00000060  → HT         |
         +------------------------------+

   4. Bucket Splitting (Inside RAX)
      -----------------------------
      If a vector bucket in a RAX fills:
        - Binary search for best split point.
        - Use `getExpiry(entry)` + `get_bucket_ts()` to find transition.
        - Create 2 new buckets and update RAX.

         Original:
             [entry1, entry2, ..., entryN]  ← bucket_ts = 64ms

         After split:
             [entry1, ..., entryK]  → bucket_ts = 32ms
             [entryK+1, ..., entryN] → bucket_ts = 64ms

      If all entries share same bucket_ts → promote to HT.

   5. Shrinking Behavior
      ------------------
      On deletion:
        - HT may shrink to VECTOR.
        - VECTOR with 1 item → becomes SINGLE.
        - If RAX has only one key left, it’s promoted up.

   Summary:
   --------
   This redesign provides:
     ✓ Fine-grained memory control
     ✓ High scalability for bursty TTL data
     ✓ Fast expiry checks via windowed organization
     ✓ Minimal overhead for sparse sets
     ✓ Flexible binary-search-based sorting and bucketing

   It also lays the groundwork for future enhancements, including metrics,
   prioritized expiry policies, or segmented cleaning.

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-05 18:28:47 +03:00
Ran Shidlansik 116de0a794 Introduce HASH items expiration
Closes https://github.com/valkey-io/valkey/issues/640

This PR introduces support for **field-level expiration in Valkey hash types**, making it possible for individual fields inside a hash to expire independently — creating what we call **volatile fields**.
This is just the first out of 3 PRs. The content of this PR focus on enabling the basic ability to set and modify hash fields expiration as well as persistency (AOF+RDB) and defrag.
[The second PR](https://github.com/ranshid/valkey/pull/5) introduces the new algorithm (volatile-set) to track volatile hash fields is in the last stages of review. The current implementation in this PR (in volatile-set.h/c) is just s tub implementation and will be replaced by [The second PR](https://github.com/ranshid/valkey/pull/5)
[The third PR](https://github.com/ranshid/valkey/pull/4/) which introduces the active expiration and defragmentation jobs.

For more highlevel design details you can track the RFC PR: https://github.com/valkey-io/valkey-rfc/pull/22.

---

Some highlevel major decisions which are taken as part of this work:
1. We decided to copy the existing Redis API in order to maintain compatibility with existing clients.
2. We decided to avoid introducing lazy-expiration at this point, in order to reduce complexity and rely only on active-expiration for memory reclamation. This will require us to continue to work on improving the active expiration job and potentially consider introduce lazy-expiration support later on.
3. Although different commands which are adding expiration on hash fields are influencing the memory utilization (by allocating more memory for expiration time and metadata) we decided to avoid adding the DENYOOM for these commands (an exception is HSETEX) in order to be better aligned with highlevel keys commands like `expire`
4. Some hash type commands will produce unexpected results:
 - HLEN - will still reflect the number of fields which exists in the hash object (either actually expired or not).
 - HRANDFIELD - in some cases we will not be able to randomly select a field which was not already expired. this case happen in 2 cases: 1/ when we are asked to provide a non-uniq fields (i.e negative count) 2/ when the size of the hash is much bigger than the count and we need to provide uniq results. In both cases it is possible that an empty response will be returned to the caller, even in case there are fields in the hash which are either persistent or not expired.
5. For the case were a field is provided with a zero (0) expiration time or expiration time in the past, it is immediately deleted. We decided that, in order to be aligned with how high level keys are handled, we will emit hexpired keyspace event for that case (instead of hdel). For example:
for the case:
6. We will ALWAYS load hash fields during rdb load. This means that when primary is rebooting with an old snapshot, it will take time to reclaim all the expired fields. However this simplifies the current logic and avoid major refactoring that I suspect will be needed.
```
HSET myhash f1 v1
> 0
HGETEX myhash EX 0 FIELDS 1 f1
> "v1"
HTTL myhash FIELDS 1 f1
>  -2
```

The reported events are:
```
1) "psubscribe"
2) "__keyevent@0__*"
3) (integer) 1
1) "pmessage"
2) "__keyevent@0__*"
3) "__keyevent@0__:hset"
4) "myhash"
1) "pmessage"
2) "__keyevent@0__*"
3) "__keyevent@0__:hexpired" <---------------- note this
4) "myhash"
1) "pmessage"
2) "__keyevent@0__*"
3) "__keyevent@0__:del"
4) "myhash"
```
---

This PR also **modularizes and exposes the internal `hashTypeEntry` logic** as a new standalone `entry.c/h` module. This new abstraction handles all aspects of **field–value–expiry encoding** using multiple memory layouts optimized for performance and memory efficiency.

An `entry` is an abstraction that represents a single **field–value pair with optional expiration**. Internally, Valkey uses different memory layouts for compactness and efficiency, chosen dynamically based on size and encoding constraints.

The entry pointer is the field sds. Which make us use an entry just like any sds. We encode the entry layout type
in the field SDS header. Field type SDS_TYPE_5 doesn't have any spare bits to
encode this so we use it only for the first layout type.

Entry with embedded value, used for small sizes. The value is stored as
SDS_TYPE_8. The field can use any SDS type.

Entry can also have expiration timestamp, which is the UNIX timestamp for it to be expired.
For aligned fast access, we keep the expiry timestamp prior to the start of the sds header.

     +----------------+--------------+---------------+
     | Expiration     | field        | value         |
     | 1234567890LL   | hdr "foo" \0 | hdr8 "bar" \0 |
     +-----------------------^-------+---------------+
                             |
                             |
                            entry pointer (points to field sds content)

Entry with value pointer, used for larger fields and values. The field is SDS
type 8 or higher.

     +--------------+-------+--------------+
     | Expiration   | value | field        |
     | 1234567890LL | ptr   | hdr "foo" \0 |
     +--------------+--^----+------^-------+
                       |           |
                       |           |
                       |         entry pointer (points to field sds content)
                       |
                      value pointer = value sds

The `entry.c/h` API provides methods to:
- Create, read, and write and Update field/value/expiration
- Set or clear expiration
- Check expiration state
- Clone or delete an entry

---

This PR introduces **new commands** and extends existing ones to support field expiration:

The proposed API is very much identical to the Redis provided API (Redis 7.4 + 8.0). This is intentionally proposed in order to avoid breaking client applications already opted to use hash items TTL.

**Synopsis**

```
HSETEX key [NX | XX] [FNX | FXX] [EX seconds | PX milliseconds |
  EXAT unix-time-seconds | PXAT unix-time-milliseconds | KEEPTTL]
  FIELDS numfields field value [field value ...]
```

Set the value of one or more fields of a given hash key, and optionally set their expiration time or time-to-live (TTL).

The HSETEX command supports the following set of options:

* `NX` — Only set the fields if the hash object does NOT exist.
* `XX` — Only set the fields if if the hash object doesx exist.
* `FNX` — Only set the fields if none of them already exist.
* `FXX` — Only set the fields if all of them already exist.
* `EX seconds` — Set the specified expiration time in seconds.
* `PX milliseconds` — Set the specified expiration time in milliseconds.
* `EXAT unix-time-seconds` — Set the specified Unix time in seconds at which the fields will expire.
* `PXAT unix-time-milliseconds` — Set the specified Unix time in milliseconds at which the fields will expire.
* `KEEPTTL` — Retain the TTL associated with the fields.

The `EX`, `PX`, `EXAT`, `PXAT`, and `KEEPTTL` options are mutually exclusive.

**Synopsis**

```
HGETEX key [EX seconds | PX milliseconds | EXAT unix-time-seconds |
  PXAT unix-time-milliseconds | PERSIST] FIELDS numfields field
  [field ...]
```

Get the value of one or more fields of a given hash key and optionally set their expiration time or time-to-live (TTL).

The `HGETEX` command supports a set of options:

* `EX seconds` — Set the specified expiration time, in seconds.
* `PX milliseconds` — Set the specified expiration time, in milliseconds.
* `EXAT unix-time-seconds` — Set the specified Unix time at which the fields will expire, in seconds.
* `PXAT unix-time-milliseconds` — Set the specified Unix time at which the fields will expire, in milliseconds.
* `PERSIST` — Remove the TTL associated with the fields.

The `EX`, `PX`, `EXAT`, `PXAT`, and `PERSIST` options are mutually exclusive.

**Synopsis**

```
HEXPIRE key seconds [NX | XX | GT | LT] FIELDS numfields
  field [field ...]
```

Set an expiration (TTL or time to live) on one or more fields of a given hash key. You must specify at least one field. Field(s) will automatically be deleted from the hash key when their TTLs expire.
Field expirations will only be cleared by commands that delete or overwrite the contents of the hash fields, including `HDEL` and `HSET` commands. This means that all the operations that conceptually *alter* the value stored at a hash key's field without replacing it with a new one will leave the TTL untouched.
You can clear the TTL of a specific field by specifying 0 for the ‘seconds’ argument.
Note that calling `HEXPIRE`/`HPEXPIRE` with a time in the past will result in the hash field being deleted immediately.

The `HEXPIRE` command supports a set of options:

* `NX` — For each specified field, set expiration only when the field has no expiration.
* `XX` — For each specified field, set expiration only when the field has an existing expiration.
* `GT` — For each specified field, set expiration only when the new expiration is greater than current one.
* `LT` — For each specified field, set expiration only when the new expiration is less than current one.

**Synopsis**

```
HEXPIREAT key unix-time-seconds [NX | XX | GT | LT] FIELDS numfields
  field [field ...]
```

`HEXPIREAT` has the same effect and semantics as `HEXPIRE`, but instead of specifying the number of seconds for the TTL (time to live), it takes an absolute Unix timestamp in seconds since Unix epoch. A timestamp in the past will delete the field immediately.

The `HEXPIREAT` command supports a set of options:

* `NX` — For each specified field, set expiration only when the field has no expiration.
* `XX` — For each specified field, set expiration only when the field has an existing expiration.
* `GT` — For each specified field, set expiration only when the new expiration is greater than current one.
* `LT` — For each specified field, set expiration only when the new expiration is less than current one.

**Synopsis**

```
HPEXPIRE key milliseconds [NX | XX | GT | LT] FIELDS numfields
  field [field ...]
```

This command works like `HEXPIRE`, but the expiration of a field is specified in milliseconds instead of seconds.

The `HPEXPIRE` command supports a set of options:

* `NX` — For each specified field, set expiration only when the field has no expiration.
* `XX` — For each specified field, set expiration only when the field has an existing expiration.
* `GT` — For each specified field, set expiration only when the new expiration is greater than current one.
* `LT` — For each specified field, set expiration only when the new expiration is less than current one.

**Synopsis**

```
HPEXPIREAT key unix-time-milliseconds [NX | XX | GT | LT]
  FIELDS numfields field [field ...]
```

`HPEXPIREAT` has the same effect and semantics as `HEXPIREAT``,` but the Unix time at which the field will expire is specified in milliseconds since Unix epoch instead of seconds.

**Synopsis**

```
HPERSIST key FIELDS numfields field [field ...]
```

Remove the existing expiration on a hash key's field(s), turning the field(s) from *volatile* (a field with expiration set) to *persistent* (a field that will never expire as no TTL (time to live) is associated).

**Synopsis**

```
HSETEX key [NX] seconds field value [field value ...]
```

Similar to `HSET` but adds one or more hash fields that expire after specified number of seconds. By default, this command overwrites the values and expirations of specified fields that exist in the hash. If `NX` option is specified, the field data will not be overwritten. If `key` doesn't exist, a new Hash key is created.

The HSETEX command supports a set of options:

* `NX` — For each specified field, set expiration only when the field has no expiration.

**Synopsis**

```
HTTL key FIELDS numfields field [field ...]
```

Returns the **remaining** TTL (time to live) of a hash key's field(s) that have a set expiration. This introspection capability allows you to check how many seconds a given hash field will continue to be part of the hash key.

```
HPTTL key FIELDS numfields field [field ...]
```

Like `HTTL`, this command returns the remaining TTL (time to live) of a field that has an expiration set, but in milliseconds instead of seconds.

**Synopsis**

```
HEXPIRETIME key FIELDS numfields field [field ...]
```

Returns the absolute Unix timestamp in seconds since Unix epoch at which the given key's field(s) will expire.

**Synopsis**

```
HPEXPIRETIME key FIELDS numfields field [field ...]
```

`HPEXPIRETIME` has the same semantics as `HEXPIRETIME`, but returns the absolute Unix expiration timestamp in milliseconds since Unix epoch instead of seconds.

This PR introduces new notification events to support field-level expiration:

| Event       | Trigger                                  |
|-------------|-------------------------------------------|
| `hexpire`   | Field expiration was set                  |
| `hexpired`  | Field was deleted due to expiration       |
| `hpersist`  | Expiration was removed from a field       |
| `del`       | Key was deleted after all fields expired  |

Note that we diverge from Redis in the cases we emit hexpired event.
For example:
given the following usecase:
```
HSET myhash f1 v1
(integer) 0
HGETEX myhash EX 0 FIELDS 1 f1
1) "v1"
 HTTL myhash FIELDS 1 f1
1) (integer) -2
```
regarding the keyspace-notifications:
Redis reports:
```
1) "psubscribe"
2) "__keyevent@0__:*"
3) (integer) 1
1) "pmessage"
2) "__keyevent@0__:*"
3) "__keyevent@0__:hset"
4) "myhash2"
1) "pmessage"
2) "__keyevent@0__:*"
3) "__keyevent@0__:hdel" <---------------- note this
4) "myhash2"
1) "pmessage"
2) "__keyevent@0__:*"
3) "__keyevent@0__:del"
4) "myhash2"
```

However In our current suggestion, Valkey will emit:
```
1) "psubscribe"
2) "__keyevent@0__*"
3) (integer) 1
1) "pmessage"
2) "__keyevent@0__*"
3) "__keyevent@0__:hset"
4) "myhash"
1) "pmessage"
2) "__keyevent@0__*"
3) "__keyevent@0__:hexpired" <---------------- note this
4) "myhash"
1) "pmessage"
2) "__keyevent@0__*"
3) "__keyevent@0__:del"
4) "myhash"
```
---

- Expiration-aware commands (`HSETEX`, `HGETEX`, etc.) are **not propagated as-is**.
- Instead, Valkey rewrites them into equivalent commands like:
  - `HDEL` (for expired fields)
  - `HPEXPIREAT` (for setting absolute expiration)
  - `HPERSIST` (for removing expiration)

This ensures compatibility with replication and AOF while maintaining consistent field-level expiry behavior.

---

| Command Name | QPS Standard | QPS HFE | QPS Diff % | Latency Standard (ms) | Latency HFE (ms) | Latency Diff % |
|--------------|-------------|---------|------------|----------------------|------------------|----------------|
| **One Large Hash Table** |
| HGET | 137988.12 | 138484.97 | +0.36% | 0.951 | 0.949 | -0.21% |
| HSET | 138561.73 | 137343.77 | -0.87% | 0.948 | 0.956 | +0.84% |
| HEXISTS | 139431.12 | 138677.02 | -0.54% | 0.942 | 0.946 | +0.42% |
| HDEL | 140114.89 | 138966.09 | -0.81% | 0.938 | 0.945 | +0.74% |
| **Many Hash Tables (100 fields)** |
| HGET | 136798.91 | 137419.27 | +0.45% | 0.959 | 0.956 | -0.31% |
| HEXISTS | 138946.78 | 139645.31 | +0.50% | 0.946 | 0.941 | -0.52% |
| HGETALL | 42194.09 | 42016.80 | -0.42% | 0.621 | 0.625 | +0.64% |
| HSET | 137230.69 | 137249.53 | +0.01% | 0.959 | 0.958 | -0.10% |
| HDEL | 138985.41 | 138619.34 | -0.26% | 0.948 | 0.949 | +0.10% |
| **Many Hash Tables (1000 fields)** |
| HGET | 135795.77 | 139256.36 | +2.54% | 0.965 | 0.943 | -2.27% |
| HEXISTS | 138121.55 | 137950.06 | -0.12% | 0.951 | 0.952 | +0.10% |
| HGETALL | 5885.81 | 5633.80 | **-4.28%** | 2.690 | 2.841 | **+5.61%** |
| HSET | 137005.08 | 137400.39 | +0.28% | 0.959 | 0.955 | -0.41% |
| HDEL | 138293.45 | 137381.52 | -0.65% | 0.948 | 0.955 | +0.73% |

[ ] Consider extending HSETEX with extra arguments: NX/XX so that it is possible to prevent adding/setting/mutating fields of a non-existent hash
[ ] Avoid loading expired fields when non-preamble RDB is being loaded on primary. This is an optimization in order to reduce loading unnecessary fields (which are expired). This would also require us to propagate the HDEL to the replicas in case of RDBFLAGS_FEED_REPL. Note that it might have to require some refactoring:
1/ propagate the rdbflags and current time to rdbLoadObject. 2/ consider the case of restore and check_rdb etc...
For this reason I would like to avoid this optimizationfor the first drop.

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-08-05 18:28:47 +03:00
cxljsandGitHub 521d7c9191 Remove the unused value duplicate API from dict of libvalkey (#2387)
The commit
(https://github.com/valkey-io/valkey/commit/7b050773d268e2bae7bf0e2711ce4b066df0ec91)
removes the unused value duplicate API from dict, and libvalkey's dict
needs to remain consistent with it.

Signed-off-by: Xiaolong Chen <fukua95@gmail.com>
2025-08-05 14:43:45 +02:00
Katie HollyandGitHub 40456159bd Strengthen undefined behavior prevention in checkSignedBitfieldOverflow (#2418)
This one was found by
[afl++](https://github.com/AFLplusplus/AFLplusplus). Executing `bitfield
0 set i64 0 1` triggers UBSan at the `int64_t minincr = min - value;`
calculation. To fix the undefined behavior in the `minincr` calculation
and strengthen the protection in the `maxincr` calculation, we cast
both, the minuend and the subtrahend, to an unsigned int, do the
calculation, and then cast the result back into a signed int.

Signed-off-by: Fusl <fusl@meo.ws>
2025-08-05 14:26:38 +02:00
Katie HollyandGitHub 8a3818f2ef Use unsigned long for maxiterations, fixing undefined behavior in scan with extremely large count (#2414)
This one was found by
[afl++](https://github.com/AFLplusplus/AFLplusplus). Executing `scan 0
count n` with a count that is within 10% of `LONG_MAX`, `count * 10`
would cause `maxiterations` to overflow. This is technically not a real
problem since the way `maxiterations` is used would eventually cause it
to underflow back to `LONG_MAX` again and continue counting down from
there but I figured we may want to fix this regardless for expected
behavior correctness?

Signed-off-by: Fusl <fusl@meo.ws>
2025-08-05 13:17:11 +03:00
BinbinandGitHub 80bbe5aa4d Print the complete log instead of just the crash log in log_crashes (#2428)
We used to, for example in runtest-cluster, we will only print the crash
log since in here, when we match the pattern, we start setting found and
then print the log starting from the current line. We can print the
complete log in this case and it might help troubleshoot the crash.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-05 17:32:35 +08:00
Nicky-2000andGitHub 98b390e7e8 Fix prefetchNextBucketEntries so it fetches the correct next 2 buckets (#2394)
This PR fixes a bug in prefetchNextBucketEntries which is used in the
hashtable iterator. The current version of prefetchNextBucketEntries
does not correctly prefetch the next two buckets that will be iterated
over. This is due next_index being incremented twice (once in
prefetchNextBucketEntries and again in getNextBucket).

Fix:

1. Add check in prefetchNextBucketEntries to ensure we pass the correct
   'next_index' to getNextBucket.
2. The extra increment was removed from getNextBucket.

Signed-off-by: Nicky Khorasani <nickykhorasani@google.com>
2025-08-05 11:22:57 +02:00
BinbinandGitHub 89c9ef026e Make ./runtest --dump-logs dump logs on valgrind / sanitizer errors (#2427)
Sometimes we want to print server logs even if valgrind or sanitizer
errors occur, which might help troubleshoot the problem.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-05 16:32:58 +08:00
Sarthak AggarwalandGitHub bda3af0a60 Dictionary Implementation for Migrating and Importing slots (#2409)
Resolves #2184

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-08-05 09:51:56 +02:00
BinbinandGitHub 388fdd9a94 Update swapdb command comment to mention why cluster mode is not allowed (#2391)
The comment has been outdated since #1671, update it.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-05 11:47:00 +08:00
BinbinandGitHub 3bd4d49a85 Cleanup and rename isExpiryTableValidForSamplingCb to expiryTableShouldSkipForSamplingCb (#2395)
We used to return C_ERR (-1) or C_OK (0), it was not that bool.
usage: int skip = !ht || (skip_cb && skip_cb(ht));

Also isExpiryTableValidForSamplingCb is actually a skip cp, rename
it to expireShouldSkipTableForSamplingCb for a better reading.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-05 11:46:42 +08:00
BinbinandGitHub 61aaa6c3e6 Initialize rdbstate.key_type to -1 at the beginning (#2381)
Otherwise this would result in a bad (key_type == 0) key type
information printing misleading error messages if there are
errors when processing other RDB_OPCODE_* type:
```
[offset 96] Unexpected EOF reading RDB file
[additional info] While doing: read-len
[additional info] Reading type 0 (string)
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-04 10:28:44 -07:00
Allen SamuelsandGitHub a915da9237 Increase TID count to accomodate valkey-search reader/writer pools (#2417)
The Search module creates one reader thread and one writer thread for
each CPU. On large boxes (64 CPUS) the current number of TIDs are
inadequate and could result in the loss of critical debugging
information in the case of field failure.

Signed-off-by: Allen Samuels <allenss@amazon.com>
2025-08-04 10:08:03 -07:00
BinbinandGitHub 2a9a955ba6 Fix check-rdb --stats crash with empty RDB (#2382)
There are two problems with initializing databases to 1:
1. db0 is used by default. After printing db0, we will additionally print
   db1 with no reason.
2. Based on 1, if the RDB is an empty RDB, check-rdb will crash. Because
   there is no key data, we don't have the chance to call tryExpandRdbStats,
   and then when printting db1, we will crash on rdbstate.stats.

Introduced in #1743.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-04 19:11:42 +08:00
BinbinandGitHub e5a99383f7 Remove obsolete BLOCKED_WAITAOF comment (#2374)
BLOCKED_WAITAOF was consolidated into BLOCKED_WAIT in #562.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-04 19:11:22 +08:00
BinbinandGitHub d4e725e0f3 Rename trace bgsave type to rdb and aof (#2400)
Previously, we called it the bgsave type, and it had two
events: rdb_unlink_temp_file and fork. However, the code
actually counts all forks as part of the bgsave type, such
as aof fork and module fork.

In this commit, the bgsave type was renamed to rdb type,
to align with the aof type, also adding the aof fork event.

Also doing some cleanup around the README file.
See #2070 for mode details.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-08-03 16:55:12 +08:00
yzc-yzcandGitHub 380f65ef08 Make ./runtest --dump-logs dump logs for timeout tests's servers (#2412)
Kill the servers first, then kill the test clients, so that we can dump
logs for timeout tests's servers easily.
Note: Technically, we need to wait a while after killing the servers.
But in practice I think it's enough now.

Signed-off-by: yzc-yzc <96833212+yzc-yzc@users.noreply.github.com>
2025-08-02 23:01:28 +08:00
zhaozhao.zzandGitHub 350f2a99c6 support negative filtering for client command filters (#2378)
introduce negative filters for `CLIENT LIST` and `CLIENT KILL` commands:

1. `NOT-ID`: Excludes clients in the IDs set
2. `NOT-TYPE`: Excludes clients of the specified type
3. `NOT-ADDR`: Excludes clients of the specified address and port
4. `NOT-LADDR`: Excludes clients connected to the specified local
address and port
5. `NOT-USER`: Excludes clients of the specified user
6. `NOT-FLAGS`: Excludes clients with the specified flag string
7. `NOT-NAME`: Excludes clients with the specified name
8. `NOT-LIB-NAME`: Excludes clients using the specified library name
9. `NOT-LIB-VER`: Excludes clients with the specified library version
10. `NOT-DB`: Excludes clients with the specified database ID
11. `NOT-CAPA`: Excludes clients with the specified capabilities
12. `NOT-IP`: Excludes clients with the specified IP address

close #1936

and fix the matching algorithm for flag 'N'.

---------

Signed-off-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
2025-08-01 10:31:24 +08:00
asagegeLiuandGitHub c50946bdc3 Update check times for AOF loading in memefficiency.tcl (#2402)
Update check times from 100k to 80k for AOF loading. 

Reason: During AOF loading, we only hit 95k items rather than 100k in
one defrag test

Defrag test:
https://github.com/valkey-io/valkey/actions/runs/15452801881/job/43498734253?pr=2078#step:6:7047
(line 6157)

Signed-off-by: asagegeLiu <124027677+asagege@users.noreply.github.com>
2025-07-31 12:15:29 -07:00
fe75d09a5b Save to Disk in Bio Thread and refactor readSyncBulkPayload (#1784)
## **Introduction**

This PR introduces a new feature that enables replicas to perform
disk-based synchronization on a dedicated background thread (Bio
thread). Benchmarking results demonstrate significant improvements in
synchronization duration. In extreme cases, this optimization allows
syncs that would have previously failed to succeed.

## **Problem Statement**

Some administrators prefer the disk-based full synchronization mode for
replicas. This mode allows replicas to continue serving clients with
data while downloading the RDB file.

Valkey's predominantly single-threaded nature creates a challenge:
serving client read requests and saving data from the socket to disk are
not truly concurrent operations. In practice, the replica alternates
between processing client requests and replication data, leading to
inefficient behavior and prolonged sync durations, especially under high
load.

## **Proposed Solution**

To address this, the solution offloads the task of downloading the RDB
file from the socket to a background thread. This allows the main thread
to focus exclusively on handling client read requests while the
background thread handles communication with the primary.

## **Benchmarking Results**

### **Potential for Improvement**
In theory, this optimization can lead to unbounded improvement in sync
duration. By eliminating competition between client read events and
socket communication (i.e., events related to handling RDB download with
the primary), sync times become independent on load - the main thread
handles only client reads, while the background thread focuses on
primary RDB download events, allowing the system to perform consistently
even under high load.

The full valkey-benchmark commands can be found in the appendix below.

### Sync Duration with Feature Disabled (times in seconds)

16 threads, 64 clients: 172 seconds
32 threads, 128 clients: 436 seconds
48 threads, 192 clients: 710 seconds

### Sync Duration with Feature Enabled (times in seconds)

16 threads, 64 clients: 33 seconds (80.8% improvement)
32 threads, 128 clients: 33 seconds (92.4% improvement)
48 threads, 192 clients: 33 seconds (95.3% improvement)


![image](https://github.com/user-attachments/assets/d5f7f8b4-7cdc-499a-8359-523ba22e7762)

## **Alternative Solutions Considered**

**IO Threads**  
IO threads to not have an advantage over Bio in this case: The
save-to-disk job is rare (most likely no more than several executions in
a replica's lifetime), and there is never more than one simultaneous
execution. Bio threads make more sense for a single, slow long running
operation.

**io_uring**  
For a single connection, io_uring doesn't provide as much of a
performance boost because the primary advantage comes from batching many
I/O operations together to reduce syscall overhead. With just one
connection, we won't have enough operations to benefit significantly
from these optimizations.

**Prioritizing primary's socket in the event loop**
This approach would help, but less effectively than using a Bio thread.
We would still need to allocate attention to handling read requests,
which could limit its benefit. It could be more useful on smaller
instance types with limited CPU cores. Edit: In practice, this feature
actually does this naturally when there is a single core - when both
threads are running on the same core, the Bio thread can get up to 50%
of CPU time now to handle RDB download, whereas before RDB download
events were queued after a bunch of read events (and didn't get even
close to 50%).

## **Code Design**

This PR introduces both the new BIO-based RDB sync flow described above
and a significant (and much-needed) refactoring of the
readSyncBulkPayload function. Previously, this single monolithic
function handled the entire replication flow, covering both disk-based
and socket-based syncs, and was responsible for everything from sync
preparation and RDB reception to loading and finalization. The
refactoring splits these responsibilities into clearer, more modular
components, improving readability, reusability, and maintainability.

### Disk-Based RDB Sync (Bio thread)
The old disk-based RDB save logic has been removed from the main thread.
It is now exclusively handled by a dedicated Bio thread, following the
team’s decision to remove the need for a config.

New Flow Overview:

After the replication handshake completes in `syncWithPrimary()` (or at
the end of the dual-channel handshake), the replica determines whether
to use disk-based sync via `useDisklessLoad()` (unchanged).

If disk sync is chosen, a read handler is set on the primary’s
connection:

`connSetReadHandler(conn, receiveRDBinBioThread);`
When the primary begins sending the RDB payload, this read handler is
triggered. It creates a dedicated Bio thread to perform the sync.

The Bio thread executes `replicaReceiveRDBFromPrimaryToDisk`, which is
now the main handler for receiving and saving RDB to disk.
This function replaces the disk logic previously found in
`readSyncBulkPayload`, but executes in a busy loop (as opposed to being
event-driven like `readSyncBulkPayload`) since the Bio thread is
single-purpose and not event-based.

The logic inside `replicaReceiveRDBFromPrimaryToDisk` is a direct port
of the previous flow, with some refactoring for clarity and reusability
across sync modes.

Upon completion or failure, the Bio thread signals the main thread using
a shared variable (`replica_bio_disk_save_state`).

The main thread detects Bio completion in a new function called
`handleBioThreadFinishedRDBDownload()` (triggered via replication cron).
This function then initiates the RDB load and resets relevant stats.

Note: The actual RDB loading remains on the main thread, as it already
halts I/O during load and doesn’t benefit from thread offloading,
simplifying thread-safety concerns.

### Diskless Sync (Socket)
For socket-based sync, the logic remains the same: asynchronous and
event-driven, but has been refactored for modularity and readability.

The original implementation in `readSyncBulkPayload` mixed socket and
disk sync logic with heavy branching.

Now, memory-based sync is isolated in a new function:

`replicaReceiveRDBFromPrimaryToMemory`
This function retains the same execution pattern as
`readSyncBulkPayload` (i.e., reacting to available data on the socket),
but is now cleaner and easier to follow.

Common logic that was previously duplicated across both sync modes is
now factored out into shared helpers:

`replicaBeforeLoadPrimaryRDB()`
`replicaAfterLoadPrimaryRDB()`

The result is a more maintainable and modular flow. Reviewers are
encouraged to compare this function side-by-side with the previous
`readSyncBulkPayload` in order to understand the change easily.

### Metrics
Previously, replication metrics such as `repl_transfer_size` and
`repl_transfer_read` were tracked on the main thread inside
`readSyncBulkPayload`. With the move to Bio threads, those variables
could not be safely reused due to thread safety concerns.

To address this:

New Bio-specific metrics are introduced:

`server.bio_repl_transfer_size`
`server.bio_repl_transfer_read`

These are updated by the Bio thread during disk-based syncs.

At the end of the sync, the values are merged back into the standard
metrics (server.repl_transfer_*) for consistency.

If INFO is queried during an ongoing Bio-based sync, the reported values
reflect either the Bio-specific metrics or a combined view when needed,
ensuring accurate observability.

### Appendix:

### **Benchmarking Setup**

- **Client machine:** AWS c5a.16xlarge  
- **Server machines:** AWS c5a.2xlarge

```bash
# Step 1: Fill the primary and replica DBs with 6GB of data:

./valkey-benchmark -h <host> -p <port> -l -d 128 -t set -r 30000000 --threads 16 -c 64

# Step 2: Initiate heavy read load on the replica:

./valkey-benchmark -h <host> -p <port> -t get -r 30000000 --threads <t> -c <t> -n 1000000000 -P <P>

# Step 3: Enable/disable the config controlling the new feature:

./valkey-cli -h <host> -p <port> config set replica-save-to-disk-in-bio-thread <yes/no>

# Step 4: Initiate sync:

./valkey-cli -h <replica host> -p <replica port> replicaof <primary host> <primary port>

---------

Signed-off-by: Nitai Caro <caronita@amazon.com>
Co-authored-by: Nitai Caro <caronita@amazon.com>
2025-07-31 15:23:54 +03:00
Sarthak AggarwalandGitHub b18183039c Try to stabilize aof test (#2399)
Based on @enjoy-binbin's suggestion on #1611, I made the change to find
the available port. The test has been passing in the daily tests in my
local repo.

Resolved #1611

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-07-31 16:11:19 +08:00
MatthewandGitHub f001bbb7fa Same number of hyphens for summary output (#2397)
The number of hyphens was different.

Signed-off-by: utdrmac <github@matthewboehm.com>
2025-07-31 10:30:45 +08:00
aff6f8aad8 Add helper function for padded pointer copy (#2388)
## Closes https://github.com/valkey-io/valkey/issues/2383

Refactor the logic for writing a fixed length (8 byte) pointer
field with zero padding on 32‑bit targets by:
* Introducing a new helper function writePointerWithPadding
* Refactoring encodeFailureReportKey() and encodeTimeoutKey() to use
this helper instead of in‑lining the padding logic

Signed-off-by: Seungmin Lee <sungming@amazon.com>
Co-authored-by: Seungmin Lee <sungming@amazon.com>
2025-07-29 10:40:14 -07:00
BinbinandGitHub 4ff55cd181 Fix dual-channel-replication test due to typo error and stabilize it (#2386)
It should not be a negative value, otherwise we lose the
meaning of delay and the rdb generation may finish too fast
in a fast machine and cause the test to fail.

Also, the max_tries number is increased and the
loading-process-events-interval-bytes is adjusted so that the
replica has more opportunities to process the events and can
be aware of the disconnection.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-07-29 19:05:49 +08:00
BinbinandGitHub 666773fda9 Update clusterMoveNodeSlots to also move importing slots and migrating slots (#2370)
In #2301, we added clusterMoveNodeSlots to implement the logic of
moving slots from old primary to new primary, when myself receives
the replica (old primary) message first and the new primary message
later in a shard failover.

However due to this, when myself receives the new primary message
later next time, there is no way to call clusterUpdateSlotsConfigWith,
because we have already updated the slots of the new primary before.
This result in, for example, importing slots and migrating slots
not being updated, see #445.

In this commit, we also make clusterMoveNodeSlots to move importing
slots and migrating slots.

Fixes #2363.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-07-28 19:15:21 +08:00
8017d71c4e Optimize cluster failure report tracking with RAX (#2277)
Closes https://github.com/valkey-io/valkey/issues/2139

### Summary
The original implementation used a simple list to track failure reports,
which made core operations (update, delete, and cleanup) run in O(N)
time. This became a bottleneck when many nodes failed
simultaneously, as each new report or cleanup operation required
scanning the entire list regardless of whether the reports were expired
or still valid. This led to severe performance degradation under high
failure scenarios due to repeated full list scans and inefficient
lookups.

### Key changes
This PR replaces the legacy fail_reports list with a new report
implementation that uses radix tree to manage failure reports more
efficiently and robustly. With radix tree, it maintains sorted reports inherently.
To reduce memory overhead and excessive node splits caused by
millisecond level keys, expiry timestamp is rounded up to the
nearest second. This time bucketing approach keeps the RAX structure compact enough.

---------

Signed-off-by: Seungmin Lee <sungming@amazon.com>
Co-authored-by: Seungmin Lee <sungming@amazon.com>
2025-07-27 23:09:29 -07:00
BinbinandGitHub 0f22678605 Defrag should only skip non-existent dbs, not empty dbs (#2379)
In #1609, we now doing on-demand database allocation instead
of preallocation. And in beginDefragCycle, we should not skip
empty databases if they exist. The defrag still defrags the
internal allocations of the hashtables structs if they exist.

Call chain: defragStageDbKeys -> defragStageKvstoreHelper ->
kvstoreHashtableDefragTables -> hashtableDefragTables.


Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-07-27 14:49:10 +08:00
Omkar MestryandGitHub bf4253bd3d Fixing open comments on #1920 PR (#2364) 2025-07-22 22:44:01 -07:00
Sarthak AggarwalandGitHub c11eaf5b7a Limiting the new reconnections for failed nodes (#2154)
In cluster mode, the node attempts to reconnect to nodes where `link ==
NULL` during each execution cycle, which occurs every 100 milliseconds
by default. This behavior results in continuous reconnection attempts to
nodes that are unreachable or in a failed state (PFAIL or FAIL)

This PR addresses an issue where the system aggressively attempts to
reconnect to failed nodes, which can lead to resource exhaustion and
potential instability. This change limits the number of attempts we make
per failed node. A node perform 10 reconnection attempt within node timeout window. 

The PR improves engine CPU of the P99 nodes (20-30 nodes in a cluster)
by **35%,** P90 (200-300 nodes) and Avg (across all nodes) by
**10%** when there are failed nodes in the cluster.

Resolves #2122

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-07-22 15:37:28 -07:00
Viktor SöderqvistandGitHub c9eaabe7c4 Auto-failover on shutdown unified config (#2292)
In #1091, a new config `auto-failover-on-shutdown` was added. This PR
changes the config to make it unified with other shutdown related
options. This feature has not yet been released, so it's not a breaking
change.

The auto-failover-on-shutdown config is replaced by

* A new "failover" option to the existing configs `shutdown-on-sigterm`
and `shutdown-on-sigint`.
* A new FAILOVER option to the SHUTDOWN command.

Additionally, a history entry is added to the SHUTDOWN command which was
missing in #2195.

Follow-up of #1091.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-07-22 17:42:11 +02:00
BinbinandGitHub f3ccce5d86 Fix client tracking memory overhead calculation (#2360)
This should be + instread of *, otherwise it does not make any sense.
Otherwise we would have to calculate 20 more bytes for each prefix rax
node in 64 bits build.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-07-22 17:19:02 +08:00
BinbinandGitHub 00781dee7f Include command fullname in error message when returning NO_MULTI error (#2286)
Including command fullname in error messages can help users locate
problems more clearly. This is a potentially breaking change.

Previously we returned `ERR Command not allowed inside a transaction`,
now we return `ERR Command 'shutdown' not allowed inside a transaction`,
for subcommand it is `ERR Command 'client|reply' not allowed inside a
transaction`.

Closes #1629.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-07-21 07:31:18 -07:00
kukeyandGitHub a5ae568391 Prevent calling lookupkey twice in some module functions that use setkey (#2365)
In `VM_StringSet()`, `VM_StringTruncate()`, and `VM_ModuleTypeSetValue()` we could
assert the key doesnt exists, so we could set SETKEY_DOESNT_EXIST flags
to reduce one call to lookupkey.

Signed-off-by: wei.kukey <wei.kukey@gmail.com>
2025-07-17 12:38:16 +01:00
cxljsandGitHub 05bae558a3 Remove get_type from ConnectionType (#2202)
close #2187

---------

Signed-off-by: fukua95 <fukua95@gmail.com>
Signed-off-by: Xiaolong Chen <fukua95@gmail.com>
2025-07-16 23:36:57 -07:00
BinbinandGitHub b6b2c21d6f Add extensions supported to cluster link layer to propagate extension faster during handshake (#2310)
Add support_extension to cluster link, so that when myself receives a message from sender,
even if myself does not know the sender, for example during handshake the sender node is
still treated as random node, in thi case, the packet we reply to sender can also carry the
extension, since the sender link explicitly indicates that it supports the extension, by doing
this, we can speed up the spread of the extension.

Also see #2283 and #2279 for more details.
2025-07-16 10:26:27 +08:00
Katie HollyandGitHub 00f3abcf36 Fix large allocations crashing Valkey during active defrag (#2353)
When querying large objects, the memory allocator may return allocation
sizes (for example 4GiB, `4294967296`) that fit into the `out` array
which is a `size_t` so the resulting `out` array becomes `{0, 1,
4294967296}`. When calculating the `region_size` we try to fit the
allocation size into an `unsigned` which causes `4294967296` to overflow
to `0`. With that 0, we unconditionally call `jeSize2BinIndexLgQ3` which
_basically_ returns `sz-1`, so the return value becomes `-1`,
overflowing the `unsigned binind` integer to `4294967295` which is far
outside the length of `je_cb.bin_info` triggering the asserting sanity
check.

Using `size_t` instead of `unsigned` for `region_size` ensures we can
actually fit the allocated size into a variable and then return early on
the `region_size > je_cb.bin_info[je_cb.nbins - 1].reg_size` check that
follows right after.

Fixes https://github.com/valkey-io/valkey/issues/2348

Reproduction steps:

* Start up Valkey:
  ```
  valkey-server \
  --activedefrag no \
  --enable-debug-command local \
  --save '' \
  --active-defrag-cycle-min 99 \
  --active-defrag-threshold-lower 0 \
  --active-defrag-ignore-bytes 1
  ```
* Allocate a 10GiB string: `valkey-cli debug populate 1 '' 10737418240`
* Enable active defrag: `valkey-cli config set activedefrag yes`
* Valkey process crashes with `allocator_defrag.c:396 'binind <
je_cb.nbins && region_size == je_cb.bin_info[binind].reg_size' is not
true`

Unsure how to properly implement a test for this case considering it
seemingly requires at least 10GiB to test. Open for suggestions.

Signed-off-by: Fusl <fusl@meo.ws>
2025-07-14 21:18:54 -07:00
Harkrishn PatroandGitHub 9eebbdafce Add cross engine (Redis OSS) compatible test (#2336)
Add cross engine test to ensure we don't introduce regression with older
engine version compatibility.

Engine version added: Redis OSS 6.2 / 7.0

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-07-14 09:59:34 -07:00
Ran ShidlansikandGitHub 5abe4175b2 update build-debian-old to use Bullseye instead of EOL buster (#2345)
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-07-14 19:19:13 +03:00
Ran ShidlansikandGitHub f0cd440212 use regular client in test "Auto-authenticate using tls-auth-clients-user" (#2346)
TLS: Auto-authenticate using tls-auth-clients-user (CN) does not need to
use raw TLS
connection.

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-07-14 19:04:06 +03:00
fd7d002f47 Add support for automatic client authentication via TLS certificate fields (#1920)
This PR implements support for automatic client authentication based on
a field in the client's TLS certificate.

API Changes:

* New configuration directive `tls-auth-clients-user`, values `CN` |
`off`, default `off`. CN means take username from the CommonName field
in the client's certificate.
* New INFO field `acl_access_denied_tls_cert` under the `Stats` section,
indicating the number of failed authentications using this feature, i.e.
client certificates for which no matching username was found.
* New reason "tcl-cert" in the ACL log, logged when a client
certificate's CommonName fails to match any existing username.

Closes #1866

---------

Signed-off-by: Omkar Mestry <om.m.mestry@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Omkar Mestry <omanges@google.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-07-12 10:58:25 +02:00
Sarthak AggarwalandGitHub a0eedd364a Exit early when all the covered slots are deleted (#2335)
Iterate through the node slot information byte by byte and check for set bit to compute the slot. Short circuit the loop as soon as all the covered slots are iterated over.

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-07-11 14:37:21 -07:00
a34556acc5 Fix replica (the old primary) claims to sitll have slots after manual failover (#2301)
After a failover occurs, when the PING-PONG of the replica, that is,
the old primary, reaches other shard nodes faster, the corresponding
code path will be reached. We will adjust the sender to be a replica
and sender_claimed_primary to be the primary node, but in myself view,
slots still belong to the sender, which is a replica.

This actually restores part of the code in 6f5ab6a7bc.
It was lost in the changes to #445 and #754.

These 3 changes are about avoiding the replicaof cycle but ended up creating
a "cycle" of assumptions themselves. when #445 removed this logic, it was
assumed that a replica could also drive clusterUpdateSlotsConfigWith
but this was actually a regression that resulted in a replicaof cycle (#753).
#754 fixed the replicaof cycle by disallowing a replica to drive
clusterUpdateSlotsConfigWith but missed restoring the original logic.

Added a new `DEBUG DISABLE-CLUSTER-RECONNECTION <0|1>` this will prevent
cluster nodes from reconnecting so that the issue can be reproduce in test.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Ping Xie <pingxie@outlook.com>
2025-07-11 19:00:16 +08:00
Sarthak AggarwalandGitHub 8d496fe8e4 Unset the active_clients_file before new test is assigned (#2339)
With the change #2276, there are some cases where the the
active_clients_file is not set.

```
can't unset "::active_clients_file(sock11d811610)": no such element in array
    while executing
"unset ::active_clients_file($fd)"
    (procedure "read_from_test_client" line 22)
    invoked from within
"read_from_test_client sock11d811610"
```

It is because when the test is done, we were unsetting the
active_clients_file after the new task was assigned. Ideally we should
unset it before that since the first task is assigned via `ready` state.
`make test` is consistently passing now.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-07-10 11:34:25 -07:00
Harkrishn PatroandGitHub ee52903a34 Avoid logging sender role on each cluster message (#2337) 2025-07-10 08:21:20 -07:00
BinbinandGitHub f9904f6bb2 Fix DEBUG CLUSTERLINK KILL args check to avoid crash (#2333)
Currently typing `debug clusterlink` will crash the server since
in here we are accessing `c->argv[2]->ptr`.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-07-10 10:35:24 +08:00
Avi FeneshandGitHub 517e9a0dbf Change default values of valkey-cli to use valkey naming, and added fallback to old values (#2334)
This pull request updates `valkey-cli` to support legacy environment
variables for backward compatibility while transitioning to new
environment variable names. The changes ensure that users relying on
older variable names can continue using the CLI without disruption.

Backward compatibility for environment variables: VALKEYCLI_HISTFILE, VALKEYCLI_RCFILE, VALKEYCLI_CLUSTER_YES


Signed-off-by: avifenesh <aviarchi1994@gmail.com>
2025-07-09 14:29:11 -07:00
Amit NaglerandGitHub e6a82468ad handle prefetch-batch-max-size config update from zero to positive value (#2328)
Previously, onMaxBatchSizeChange was only called during
processClientsCommandsBatch, making it impossible to change
prefetch-batch-max-size from 0 to a positive number since the batch
processing would never occur with size 0.

Now onMaxBatchSizeChange is called when the configuration changes,
ensuring the new batch size takes effect. Note that the change may not
be immediate and will be applied during the next batch processing cycle.

Signed-off-by: Amit Nagler <anagler123@gmail.com>
2025-07-08 10:21:30 -07:00
Madelyn OlsonandGitHub 26ea24bba0 Disable active expiry until it's needed (#2313)
Resolves test failures like
https://github.com/valkey-io/valkey/actions/runs/16093360514/job/45412856532.
Valgrind tests can run slowly, so the 100 ms expire time may have
partially began to expire items during the ingestion phase.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-07-07 11:40:34 -07:00
Harkrishn PatroandGitHub f3d035d024 Update test to unwatch all keys before auth within multi (#2323)
Needs backporting to 8.1 to fix
https://github.com/valkey-io/valkey/issues/2320

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-07-07 11:40:27 -07:00
Harkrishn PatroandGitHub b0c3157059 Enable tab completion of test file path for runtest util and allow directory as path (#630)
Helps testing specific functionality like `cluster` rather than running the entire test suite.

### Usage example
#### Run all the tests under tests/unit/cluster directory
```
./runtest --single tests/unit/cluster
```

#### Run all the tests under test/unit but skip the file `memefficiency`
```
./runtest --single tests/unit --skipunit tests/unit/memefficiency.tcl
```

#### Run all the tests under tests/integration
```
./runtest --single tests/integration
```

Note: this doesn't work with tests under `tests/cluster`. We should
migrate all of the tests under `tests/cluster` to `tests/unit/cluster`.

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-07-07 08:20:03 -07:00
Ran ShidlansikandGitHub 16a62fc8ab retry accept on transient errors (CVE-2025-48367) (#2315)
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-07-07 00:40:08 +03:00
a186c8ce35 Apply fixed for CVE-2025-32023 (#2314)
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-07-06 23:20:32 +03:00
Josh SorefandGitHub 38077b52ae Spelling 9 (#2245)
- Subset of #2183

---------

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2025-07-04 16:15:04 -04:00
Josh SorefandGitHub b551a685a3 Spelling 10 (#2246)
- Subset of #2183

---------

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2025-07-04 16:14:09 -04:00
youngmore1024andGitHub 275499cdfa [optimization] Improve the speed of sending switch-master from sentinel (#2308)
Sentinel sends switch-master message more quickly, ref
[#2282](https://github.com/valkey-io/valkey/issues/2282).

Signed-off-by: youngmore1024 <youngmore1024@outlook.com>
2025-07-04 16:11:36 -04:00
0ed6602a28 Allow shrinking hashtables in low memory situations (#2095)
During memory eviction, we may end up with a primary kv hash table
that's below the 13% `MIN_FILL_PERCENT_SOFT`, but inside `resize()` for
the kv table we check with `ht->type->resizeAllowed` whether allocating
a new table crosses the configured `maxmemory` limit without taking into
account that shrinking, while temporarily increasing the memory usage,
will ultimately reduce the memory usage in all situations.

Blocking hash tables from shrinking in situations where we would only
temporarily cross the maxmemory limit has at least three negative side
effects:
* The performance of valkey is negatively impacted if, for example,
we're moving items from table A to table B since table A would
eventually end up being a very sparse hash table that never shrinks.
* In extreme situations, the performance of valkey slows down
significantly when trying to find more keys to evict since we're trying
to find keys in a sparse table that only becomes more sparse over time.
* In more extreme situations, such as when reducing `maxmemory` by
around 90% or writing large data to a single key, valkey can end up in a
situation where, no matter how many keys are being evicted from the kv
table, the table bucket struct overhead uses more memory than the data
stores in the table, causing all keys to be evicted from the table and
valkey becoming unusable before setting maxmemory to above the current
memory usage:

```
127.0.0.1:6379> config set maxmemory 2gb
OK
127.0.0.1:6379> debug populate 20000000
OK
(8.94s)
127.0.0.1:6379> debug htstats 0
[Dictionary HT]
Hash table 0 stats (main hash table):
 table size: 29360128
 number of entries: 20000000
[Expires HT]
127.0.0.1:6379> config set maxmemory 200mb
OK
127.0.0.1:6379> dbsize
(integer) 18725794
[...]
127.0.0.1:6379> dbsize
(integer) 0
127.0.0.1:6379> keys *
(empty array)
127.0.0.1:6379> set foo bar
(error) OOM command not allowed when used memory > 'maxmemory'.
127.0.0.1:6379> config get maxmemory
1) "maxmemory"
2) "209715200"
127.0.0.1:6379> info memory
# Memory
used_memory:269890904
used_memory_human:257.39M
[...]
127.0.0.1:6379> config set maxmemory 300mb
OK
127.0.0.1:6379> info memory
# Memory
used_memory:1455512
used_memory_human:1.39M
[...]
127.0.0.1:6379> set foo bar
OK
```

This PR addresses these issues by allowing hash tables to allocate a new
table for shrinking, even if allocating a new table causes the maxmemory
limit to be crossed during resize.

Additionally, the check for whether we should fast-forward the resize
operation has been moved to before checking whether we are allowed to
allocate a new resize table. Because we don't start a resize in this
section of the function yet, a fast-forward usually drops the memory
usage enough to allow another resize operation to start immediately
after the fast-forward resize completes.

As a side effect of this PR, we now also refuse to start a hashtable
resize if the policy is set to `HASHTABLE_RESIZE_FORBID`.

---------

Signed-off-by: Fusl <fusl@meo.ws>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-07-04 19:51:03 +02:00
Rain ValentineandGitHub c441fcc7e7 [valkey-benchmark] Allow multiple random (or sequential) placeholders and fix zadd --sequential to create expected data (#2102)
This extends the `__rand_int__` pattern to add `__rand_1st__` through
`__rand_9th__`.

For the new placeholders, multiple occurrences within a command will
have the same value.

With the `--sequential` option, each placeholder will use separate
counters.

For `__rand_int__`, the behavior is unchanged - multiple occurrences of
the pattern within a command will have different values.

Examples of using the `__rand_int__` with `--sequential`:

```
$ ./valkey-benchmark -r 300 -n 300 --sequential -q -- set key__rand_int__ val__rand_int__
set key__rand_int__ val__rand_int__: 60000.00 requests per second, p50=0.391 msec
$ ./valkey-cli info keyspace
# Keyspace
db0:keys=150,expires=0,avg_ttl=0
$ ./valkey-cli get key000000000050
"val000000000051"
```

For the new patterns, multiple occurrences within the command will have
the same value.

```
$ ./valkey-benchmark -r 300 -n 300 --sequential -q -- set key__rand_1st__ val__rand_1st__
set key__rand_int__ val__rand_int1_: 60000.00 requests per second, p50=0.383 msec
$ ./valkey-cli info keyspace
# Keyspace
db0:keys=300,expires=0,avg_ttl=0
$ ./valkey-cli get key000000000050
"val000000000050"
```

I also fixed the zadd benchmark so it produces the expected number of
keys when used with `--sequential`. (By using the same counter twice per
command it was effectively counting by twos)

I made this for myself but raised a PR in case y'all like it. 🙂

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
2025-07-04 03:52:55 +02:00
Viktor SöderqvistandGitHub 9b0a11dbb3 Make unit test report each memory leak only once (#2304)
In the unit test framework, there is a check that `zmalloc_used_memory()
== 0` after each test suite. If an assertion fails in any test case, the
test case is aborted and often it means some memory is not freed. The
check for used memory == 0 then fails for all test suites that run after
the failed one.

This PR makes sure a memory leak is only reported once, for the test
suite that didn't free its memory.

Two test cases that explicitly checked that all memory had been freed
are deleted. This is handled by the test framework.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-07-04 03:47:30 +02:00
BinbinandGitHub c0c205a537 Generate a new shard_id when the replica executes CLUSTER RESET SOFT (#2283)
When a replica doing the CLUSTER RESET SOFT, we should generate a
new shard_id for it. Since if the node was a replica, it inherits
the shard_id of its primary, and after the CLUSTER RESET SOFT, the
replica become a new empty primary, it should has its own shard_id.

Otherwise, when it joins back into the cluster or responds the PONG,
we will have two primaries in the same shard.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-07-03 09:40:27 -07:00
Sarthak AggarwalandGitHub ddf426f49e Log test details at the end when the test times out (#2276)
Resolves #2267

Timed out test gets logged at the end of the test run.

```
!!! WARNING The following tests failed:

*** [TIMEOUT]: WAIT should not acknowledge 2 additional copies of the data in tests/unit/wait.tcl
Cleanup: may take some time... OK
```

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-07-03 09:39:14 -07:00
Josh SorefandGitHub e783d7174b spelling: md (#2262)
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2025-07-02 23:47:38 +02:00
e93987c963 Fix leak when shrinking a hashtable without entries (#2288)
Fixes #2271

When we shrink a hash table and it is empty, we do it without iterating
over it to rehash the entries. However, there may still be empty child
buckets (`used[0]==0 && child_buckets[0]!=0`). These were leaked in this
case.

This fix is to check for child buckets and don't skip the incremental
rehashing if any child buckets exist. The incremental rehashing pass
will free them.

An additional fix is to compact bucket chains in scan when the scan
callback has deleted some entries. This was already implemented for the
case when rehashing is ongoing but it was missing in the case rehashing
is not ongoing.

Additionally, a test case for #2257 was added.

---------

Signed-off-by: yzc-yzc <96833212+yzc-yzc@users.noreply.github.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Yakov Gusakov <yaakov0015@gmail.com>
2025-07-02 18:06:28 +02:00
793331090c Implement RPS control for benchmark using token bucket algorithm (#1761)
Implemented a benchmark for constant RPS (AKA QPS) through token bucket
algorithm.

Adds new option `--rps NUMBER` to `valkey-benchmark`.

## Client Status

- Check the token bucket status before writing requests on the client
side
- Added a pause logic to delay sending requests
- Implemented wake-up operation using AE's time events

```


                        ┌────────┐
                        │        │
       ┌────────────────┤ Paused │
       │                │        │
       │                └────────┘
 Time trigered              ▲
       │                    │
       │             No,AddTimeEvent
       ▼                    │
   ┌───────┐         ┌──────┴──────┐        ┌────────┐
   │       │         │             │        │        │
   │ Ready ├─────────►acquire Token├──Yes──►│ Writed │
   │       │         │             │        │        │
   └───────┘         └─────────────┘        └────┬───┘
       ▲                                         │
       │                                         │
       │                                         │
       └─────────────────Read┼Done───────────────┘

```

## Speed limit algorithm

Common algorithms were used: https://en.wikipedia.org/wiki/Token_bucket

Closes https://github.com/valkey-io/valkey/issues/1201

---------

Signed-off-by: artikell <739609084@qq.com>
Signed-off-by: skyfirelee <739609084@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-07-02 17:07:34 +02:00
380c24a449 Update LPOP and RPOP command json file description (#2179)
When I work on Slot-level metrics and info keysize feature, I read lpop
and rpop command source code, the description confused me. I think these
2 commands description on website (https://valkey.io/commands/lpop/ and
https://valkey.io/commands/rpop/) and json files were not updated after
count argument was added.
The return result could be one element only or up to count elements if
the optional count argument is provided.

It is related to valkey-doc pr
https://github.com/valkey-io/valkey-doc/pull/308

---------

Signed-off-by: hwware <wen.hui.ware@gmail.com>
Signed-off-by: Wen Hui <wen.hui.ware@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-07-02 10:07:21 -04:00
BinbinandGitHub eeb244429f Make valkey-server --help/-h output to stdout and exit with 0 (#2296)
This is similar to 6ee27cd63f, happened
to and clean it up by the way.

"All" other CLI tools display the help information on the stdout.
Which is useful if you want to manipulate directly the data (grepping
for example, without needing to do redirections). The exit status code
is usually 0 too, which makes sense.

It's printed only if --help/-h is used, not on syntax error, any other
value will be treated as a configuration item value.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-07-02 17:29:46 +08:00
BinbinandGitHub b999f512e3 Fix missing response when AUTH is errored inside a transaction (#2287)
When we add 2edecaef14, a module auth
can add reply in its callback, or not add reply and just return an
error robj in its callback, and we have to figure a way to determine
whether it has added the reply.

Before we are using `clientHasPendingReplies` and it had a issue,
the auth might be used inside a transaction or pipeline which already
has some pending replies in the client reply list. It causes us to
not add the error reply to auth.

After #1819, now we have a buffered_reply client flag, it indicates
the reply for the current command was buffered, either in client::reply
or in client::buf. We can use this flag to check.

Fixes #2106.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-07-01 19:54:07 +03:00
yulazariyandGitHub 3eee27cf99 Fix MEMORY USAGE to consider embedded keys (#2290)
Fix objectComputeSize function to use `zmalloc_size` instead of `sizeof`
when calculating the size of the top level object.

That will take in account the embedded key within the object.

Signed-off-by: yulazariy <yulazari@amazon.com>
2025-07-01 17:02:37 +02:00
f90e53dd09 Add SAFE option to SHUTDOWN to reject shutdown in unsafe situations (#2195)
Add SAFE option to SHUTDOWN. If we passed SAFE, the SHUTDOWN
will refuse to shutdown if it is not safe to shutdown. Like if
myself is a voting primary, it will refuse to shutdown. This
avoids the situation where a replica suddenly becomes the primary
when we shutting down the replica, or shutting down a primary node
by mistake and then causing the cluster to down.

Add SAFE option to SHUTDOWN command.
Add safe option to shutdown-on-sigint and shutdown-on-sigterm.

Note that SAFE cannot prevent FORCE, in the case of FORCE, SAFE
will print the relevant logs and do the FORCE shutdown, we allow
this combination.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-07-01 11:52:55 +08:00
BinbinandGitHub 1c91c3016b Move auth check to the front, before command exist/arity/protected check (#1475)
When requirepass is enabled, we want command calls to return NOAUTH
instead of ERR with the error message.

Previously, these checks were all before the auth check:
- command existence exist
- command arity check
- command protected check

This may expose information such as whether the server supports the
command, whether the configuration item is enabled, etc. This is more
of a consistency issue as the same error message is returned when
requirepass is enabled, not a security issue.

This is a behavior change, though perhaps not a breaking one.

We put the auth check into !client_reprocessing_command block, that
means the reprocessing command would skip auth check, it's also a behavior
change but align with the old behavior before the refactor of reprocessing,
so it should be fine.

Before:
```
127.0.0.1:6379> foo bar
(error) ERR unknown command 'foo', with args beginning with: 'bar' 
127.0.0.1:6379> set foo
(error) ERR wrong number of arguments for 'set' command
127.0.0.1:6379> module load foo
(error) ERR MODULE command not allowed...
```

After:
```
127.0.0.1:6379> foo bar
(error) NOAUTH Authentication required.
127.0.0.1:6379> set foo
(error) NOAUTH Authentication required.
127.0.0.1:6379> module load foo
(error) NOAUTH Authentication required.
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-07-01 10:21:29 +08:00
BinbinandGitHub 2d5b5ca24d Use objectGetExpire instead of getExpire to reduce expire hashtable lookup (#2280)
Since we have now embedded the expire TTL into robj, if we already
have the robj o, we can directly get the expire. This saves an expire
hashtable lookup in getExpire.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-29 23:21:37 +08:00
Wen HuiandGitHub d5abf581e6 Update redis to server in internal variable names in benchmark source code (#2141)
Update Redis legacy word in valkey-benchmark tool: rename variable names
redis_config to server_config.

---------

Signed-off-by: hwware <wen.hui.ware@gmail.com>
2025-06-27 22:57:50 +02:00
d6ccbe4571 Start failover without waiting for the next cluster cron cycle (#2209)
When myself is a replica, if we receive a message that my primary is
FAIL, we can try to set CLUSTER_TODO_HANDLE_FAILOVER so that we
can try to failover as soon as possible in beforeSleep without waiting
for clusterCron to kick in.

Add a new markNodeAsFailing method, we move FAIL flag related code to
here so we can easily check whether the failing node is my primary.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-06-28 00:04:54 +08:00
BinbinandGitHub c908622d95 Save config file after replica migration and brocast a PONG when role changed (#1677)
This does some cleanup around clusterSetPrimary:
1. In replica migration, the replica already has a new primary, we
need to update the config file.
2. When the role changed, we need to broadcast a PONG in the
cluster to notify the change. This is a follow up for #1295.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-27 11:29:41 +08:00
BinbinandGitHub 29c2f8e21e Check module NO_FAILOVER flag before calling clusterHandleReplicaFailover (#2269)
In other places where clusterHandleReplicaFailover is called, the
CLUSTER_MODULE_FLAG_NO_FAILOVER flag is checked, which is
not done here. It bothered me a lot sometimes, did a minor cleanup
to make them consistent, and update some outdated comments.

This should be no harm since the clusterHandleReplicaFailover call
in clusterCron is the main place that drives the failover forward,
and it had the NO_FAILOVER flag check.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-27 11:28:46 +08:00
Sarthak AggarwalandGitHub 8a07e87b95 Optimize CLUSTER INFO (#2264)
The PR avoid an additional loop over all the slots and replaces
`sdscatprintf` and `sdscatfmt` which provides better performance.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-06-26 00:30:26 +02:00
Josh SorefandGitHub e1551888b9 Spelling 3 (#2240)
Update Spelling as the following files:

src/config.c
src/hashtable.c
src/latency.c
src/listpack.c
src/module.c
src/script.c
src/server.c
src/valgrind.sup
src/zmalloc.c
tests/README.md
tests/unit/cluster/update-msg.tcl
tests/unit/scripting.tcl

---------

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2025-06-25 16:11:29 -04:00
Yakov GusakovandGitHub b4b9fc906a Fix hashtablePauseAutoShrink and use it in hashtableScanDefrag (#2257)
**Current state**
During `hashtableScanDefrag`, rehashing is paused to prevent entries
from moving, but the scan callback can still delete entries which
triggers `hashtableShrinkIfNeeded`. For example, the
`expireScanCallback` can delete expired entries.

**Issue**
This can cause the table to be resized and the old memory to be freed
while the scan is still accessing it, resulting in the following memory
access violation:

```
[err]: Sanitizer error: =================================================================
==46774==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000003100 at pc 0x0000004704d3 bp 0x7fffcb062000 sp 0x7fffcb061ff0
READ of size 1 at 0x611000003100 thread T0
    #0 0x4704d2 in isPositionFilled /home/gusakovy/Projects/valkey/src/hashtable.c:422
    #1 0x478b45 in hashtableScanDefrag /home/gusakovy/Projects/valkey/src/hashtable.c:1768
    #2 0x4789c2 in hashtableScan /home/gusakovy/Projects/valkey/src/hashtable.c:1729
    #3 0x47e3ca in kvstoreScan /home/gusakovy/Projects/valkey/src/kvstore.c:402
    #4 0x6d9040 in activeExpireCycle /home/gusakovy/Projects/valkey/src/expire.c:297
    #5 0x4859d2 in databasesCron /home/gusakovy/Projects/valkey/src/server.c:1269
    #6 0x486e92 in serverCron /home/gusakovy/Projects/valkey/src/server.c:1577
    #7 0x4637dd in processTimeEvents /home/gusakovy/Projects/valkey/src/ae.c:370
    #8 0x4643e3 in aeProcessEvents /home/gusakovy/Projects/valkey/src/ae.c:513
    #9 0x4647ea in aeMain /home/gusakovy/Projects/valkey/src/ae.c:543
    #10 0x4a61fc in main /home/gusakovy/Projects/valkey/src/server.c:7291
    #11 0x7f471957c139 in __libc_start_main (/lib64/libc.so.6+0x21139)
    #12 0x452e39 in _start (/local/home/gusakovy/Projects/valkey/src/valkey-server+0x452e39)

0x611000003100 is located 0 bytes inside of 256-byte region [0x611000003100,0x611000003200)
freed by thread T0 here:
    #0 0x7f471a34a1e5 in __interceptor_free (/lib64/libasan.so.4+0xd81e5)
    #1 0x4aefbc in zfree_internal /home/gusakovy/Projects/valkey/src/zmalloc.c:400
    #2 0x4aeff5 in valkey_free /home/gusakovy/Projects/valkey/src/zmalloc.c:415
    #3 0x4707d2 in rehashingCompleted /home/gusakovy/Projects/valkey/src/hashtable.c:456
    #4 0x471b5b in resize /home/gusakovy/Projects/valkey/src/hashtable.c:656
    #5 0x475bff in hashtableShrinkIfNeeded /home/gusakovy/Projects/valkey/src/hashtable.c:1272
    #6 0x47704b in hashtablePop /home/gusakovy/Projects/valkey/src/hashtable.c:1448
    #7 0x47716f in hashtableDelete /home/gusakovy/Projects/valkey/src/hashtable.c:1459
    #8 0x480038 in kvstoreHashtableDelete /home/gusakovy/Projects/valkey/src/kvstore.c:847
    #9 0x50c12c in dbGenericDeleteWithDictIndex /home/gusakovy/Projects/valkey/src/db.c:490
    #10 0x515f28 in deleteExpiredKeyAndPropagateWithDictIndex /home/gusakovy/Projects/valkey/src/db.c:1831
    #11 0x516103 in deleteExpiredKeyAndPropagate /home/gusakovy/Projects/valkey/src/db.c:1844
    #12 0x6d8642 in activeExpireCycleTryExpire /home/gusakovy/Projects/valkey/src/expire.c:70
    #13 0x6d8706 in expireScanCallback /home/gusakovy/Projects/valkey/src/expire.c:139
    #14 0x478bd8 in hashtableScanDefrag /home/gusakovy/Projects/valkey/src/hashtable.c:1770
    #15 0x4789c2 in hashtableScan /home/gusakovy/Projects/valkey/src/hashtable.c:1729
    #16 0x47e3ca in kvstoreScan /home/gusakovy/Projects/valkey/src/kvstore.c:402
    #17 0x6d9040 in activeExpireCycle /home/gusakovy/Projects/valkey/src/expire.c:297
    #18 0x4859d2 in databasesCron /home/gusakovy/Projects/valkey/src/server.c:1269
    #19 0x486e92 in serverCron /home/gusakovy/Projects/valkey/src/server.c:1577
    #20 0x4637dd in processTimeEvents /home/gusakovy/Projects/valkey/src/ae.c:370
    #21 0x4643e3 in aeProcessEvents /home/gusakovy/Projects/valkey/src/ae.c:513
    #22 0x4647ea in aeMain /home/gusakovy/Projects/valkey/src/ae.c:543
    #23 0x4a61fc in main /home/gusakovy/Projects/valkey/src/server.c:7291
    #24 0x7f471957c139 in __libc_start_main (/lib64/libc.so.6+0x21139)

previously allocated by thread T0 here:
    #0 0x7f471a34a753 in __interceptor_calloc (/lib64/libasan.so.4+0xd8753)
    #1 0x4ae48c in ztrycalloc_usable_internal /home/gusakovy/Projects/valkey/src/zmalloc.c:214
    #2 0x4ae757 in valkey_calloc /home/gusakovy/Projects/valkey/src/zmalloc.c:257
    #3 0x4718fc in resize /home/gusakovy/Projects/valkey/src/hashtable.c:645
    #4 0x475bff in hashtableShrinkIfNeeded /home/gusakovy/Projects/valkey/src/hashtable.c:1272
    #5 0x47704b in hashtablePop /home/gusakovy/Projects/valkey/src/hashtable.c:1448
    #6 0x47716f in hashtableDelete /home/gusakovy/Projects/valkey/src/hashtable.c:1459
    #7 0x480038 in kvstoreHashtableDelete /home/gusakovy/Projects/valkey/src/kvstore.c:847
    #8 0x50c12c in dbGenericDeleteWithDictIndex /home/gusakovy/Projects/valkey/src/db.c:490
    #9 0x515f28 in deleteExpiredKeyAndPropagateWithDictIndex /home/gusakovy/Projects/valkey/src/db.c:1831
    #10 0x516103 in deleteExpiredKeyAndPropagate /home/gusakovy/Projects/valkey/src/db.c:1844
    #11 0x6d8642 in activeExpireCycleTryExpire /home/gusakovy/Projects/valkey/src/expire.c:70
    #12 0x6d8706 in expireScanCallback /home/gusakovy/Projects/valkey/src/expire.c:139
    #13 0x478bd8 in hashtableScanDefrag /home/gusakovy/Projects/valkey/src/hashtable.c:1770
    #14 0x4789c2 in hashtableScan /home/gusakovy/Projects/valkey/src/hashtable.c:1729
    #15 0x47e3ca in kvstoreScan /home/gusakovy/Projects/valkey/src/kvstore.c:402
    #16 0x6d9040 in activeExpireCycle /home/gusakovy/Projects/valkey/src/expire.c:297
    #17 0x4859d2 in databasesCron /home/gusakovy/Projects/valkey/src/server.c:1269
    #18 0x486e92 in serverCron /home/gusakovy/Projects/valkey/src/server.c:1577
    #19 0x4637dd in processTimeEvents /home/gusakovy/Projects/valkey/src/ae.c:370
    #20 0x4643e3 in aeProcessEvents /home/gusakovy/Projects/valkey/src/ae.c:513
    #21 0x4647ea in aeMain /home/gusakovy/Projects/valkey/src/ae.c:543
    #22 0x4a61fc in main /home/gusakovy/Projects/valkey/src/server.c:7291
    #23 0x7f471957c139 in __libc_start_main (/lib64/libc.so.6+0x21139)

SUMMARY: AddressSanitizer: heap-use-after-free /home/gusakovy/Projects/valkey/src/hashtable.c:422 in isPositionFilled
Shadow bytes around the buggy address:
  0x0c227fff85d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff85e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff85f0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c227fff8600: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8610: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
=>0x0c227fff8620:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8630: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8640: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8650: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8660: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8670: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==46774==ABORTING
```


**Solution**
Suggested solution is to also pause auto shrinking during
`hashtableScanDefrag`. I noticed that there was already a
`hashtablePauseAutoShrink` method and `pause_auto_shrink` counter, but
it wasn't actually used in `hashtableShrinkIfNeeded` so I fixed that.

**Testing**
I created a simple tcl test that (most of the times) triggers this
error, but it's a little clunky so I didn't add it as part of the PR:

```
start_server {tags {"expire hashtable defrag"}} {
    test {hashtable scan defrag on expiry} {

        r config set hz 100

        set num_keys 20
        for {set i 0} {$i < $num_keys} {incr i} {
            r set "key_$i" "value_$i"
        }

        for {set j 0} {$j < 50} {incr j} {
            set expire_keys 100
            for {set i 0} {$i < $expire_keys} {incr i} {
                # Short expiry time to ensure they expire quickly
                r psetex "expire_key_${i}_${j}" 100 "expire_value_${i}_${j}"
            }

            # Verify keys are set
            set initial_size [r dbsize]
            assert_equal $initial_size [expr $num_keys + $expire_keys]
            
            after 150
            for {set i 0} {$i < 10} {incr i} {
                r get "expire_key_${i}_${j}"
                after 10
            }
        }

        set remaining_keys [r dbsize]
        assert_equal $remaining_keys $num_keys

        # Verify server is still responsive
        assert_equal [r ping] {PONG}
    } {}
}
```
Compiling with ASAN using `make noopt SANITIZER=address valkey-server`
and running the test causes error above. Applying the fix resolves the
issue.

Signed-off-by: Yakov Gusakov <yaakov0015@gmail.com>
2025-06-25 16:25:01 +02:00
f06a06a585 Add ENGINE_CHECK_RDB_NAME to the 'test' target dependencies in Makefile (#2258)
Fixes https://github.com/valkey-io/valkey/issues/2234

Running `make test` directly after `make distclean` or directly after
cloning the repo fails because the `valkey-check-rdb` executable isn't
built unless we directly run `make` before running the tests, but not
everyone does so.

Signed-off-by: Nitai Caro <caronita@amazon.com>
Co-authored-by: Nitai Caro <caronita@amazon.com>
2025-06-25 06:18:56 -06:00
BinbinandGitHub e875dce44e Cleanup aof temp files in sigShutdownHandler before exiting (#1482)
I think this code previously only meant to handle save in shutdown,
which means we may a do foreground save in shutdown. It also implicitly
handles the backgroud save, for example the child process will also
get the chance to call getpid() and clean up the temp RDB file.

However, we did not handle AOFRW, so we also leave temp files in here.
Noted that we can not use the same aofRemoveTempFile(getpid()) trick
for AOF. Since we rename the temp file (temp-rewriteaof-bg-pid.aof) to
the base aof file only in backgroundRewriteDoneHandler, and in this time,
there is no child process, so the cleanup job must be done by the parent
process.

The following sequences have been tested (not just signal handlers):
1. Long foreground save + shutdown
2. Long backgroud save + shutdown
3. Long backgroud save + long lua script + shutdown
4. Long bgrewriteaof + shutdown
5. Long bgrewriteaof + long lua script + shutdown
6. Short bgrewriteaof + long done handler + shutdown

All temporary files will be removed in the local tests.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-25 11:55:43 +08:00
Josh SorefandGitHub 988de1f5a7 Do not report negative passing test count (#2260)
This is quite nonsensical:
> [END] - test_zmalloc.c: 3 tests, -1 passed, 4 failed

See also #2263

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2025-06-24 10:35:00 +08:00
Josh SorefandGitHub 1b4ead001e Properly account for checking zmalloc_used_memory() (#2263)
The `zmalloc_used_memory()` check is _actually_ a test, and thus it
needs to be accounted for, otherwise you can get a nonsensical result:
```
[test_zmalloc.c] Memory leak detected of 29072 bytes
[END] - test_zmalloc.c: 3 tests, -1 passed, 4 failed
```

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2025-06-24 10:34:10 +08:00
85873bc006 Fix valkey-cli memory leak (#2177)
fix memory leak with valkey-cli evalMode

and clusterManagerFixSlotsCoverage

---------

Signed-off-by: charsyam <charsyam@naver.com>
Signed-off-by: hwware <wen.hui.ware@gmail.com>
Co-authored-by: Sher Sun <sher.sun@huawei.com>
2025-06-23 10:31:02 -04:00
Josh SorefandGitHub ae761eb343 Preserve newlines in commands.def reporting (#2259)
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2025-06-23 20:34:05 +08:00
amanosmeandGitHub 0b4af5ae7c Avoid freeing cluster link before printing link sender (#2254)
Log the message before freeing the cluster link.

Signed-off-by: Tyler Amano-Smerling <amanosme@amazon.com>
2025-06-20 15:18:35 -07:00
Harkrishn PatroandGitHub 04da90ff36 Converge divergent shard-id persisted in nodes.conf to primary's shard id (#2174)
Fixes #2171

Handle divergent shard-id across primary and replica from nodes.conf and
reconcile all the nodes in the shard to the primary node's shard-id.

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-06-20 15:07:13 -07:00
skyfireleeandGitHub f77883fb73 Fix incorrect laddr field and correct getClientSockname declaration (#2214)
Correct the fields in the `command_call `of LTTng, report the correct
current server `laddr` fields. Use addr/laddr to describe socket
connections, aligning with the design of valkey, and fix the ambiguous
declaration of the `getClientSockname` method.

Signed-off-by: artikell <739609084@qq.com>
2025-06-20 10:34:40 +08:00
BinbinandGitHub 3e06eaf796 Fix unixsocket too long cause addr / laddr being truncated in CLIENT INFO / LIST and logging (#2216)
When displaying client addr / laddr using catClientInfoString for example,
if the client is connected via unix domain socket and the unixsocket exceeds
NET_ADDR_STR_LEN, the output will be truncated because NET_ADDR_STR_LEN is
not long enough.

Currently NET_ADDR_STR_LEN is 46+32 that is 78, and the maximum length of a
UNIX domain socket path is typically 108 bytes on Linux and 104 bytes on macOS,
both number including null terminator. In this fix, we use CONN_ADDR_STR_LEN
instead which is 128, that should be long enough for most cases.

It affects the output of CLIENT INFO / CLIENT LIST commands, as well as the
printing of some logs.

Other cleanup:
1. In acceptCommonHandler, when calling connFormatAddr on a unix socket
client, the connFormatAddr function will always return /unixsocket since
in anetFdToString we hardcoded the string. We changed it to the same display.

2. We changed the return value of connSocketAddr from returning C_OK/C_ERR to
returning 0 and -1, which is consistent with the return values of other types.

One change worth mentioning is that in connSocketAddr, we used to call
anetFdToString which would return `/unixsocket` hardcoded in this case.
Now we will use server.unixsocket, the format is `path:0`.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-20 10:33:38 +08:00
2cf008a574 Ensure empty error tables in scripts don't crash Valkey (#2229)
When calling the command `EVAL error{} 0`, Valkey crashes with the
following stack trace. This patch ensures we never leave the
`err_info.msg` field null when we fail to extract a proper error
message.

```
=== VALKEY BUG REPORT START: Cut & paste starting from here ===
2595901:M 18 Jun 2025 01:20:12.917 # valkey 8.1.2 crashed by signal: 11, si_code: 1
2595901:M 18 Jun 2025 01:20:12.917 # Accessing address: (nil)
2595901:M 18 Jun 2025 01:20:12.917 # Crashed running the instruction at: 0x726f8e57ed1d

------ STACK TRACE ------
EIP:
/usr/lib/libc.so.6(+0x16ed1d) [0x726f8e57ed1d]

2595905 bio_aof
/usr/lib/libc.so.6(+0x9de22) [0x726f8e4ade22]
/usr/lib/libc.so.6(+0x91fda) [0x726f8e4a1fda]
/usr/lib/libc.so.6(+0x9264c) [0x726f8e4a264c]
/usr/lib/libc.so.6(pthread_cond_wait+0x14e) [0x726f8e4a4d1e]
valkey-server *:6379(bioProcessBackgroundJobs+0x1b4) [0x6530abb46db4]
/usr/lib/libc.so.6(+0x957eb) [0x726f8e4a57eb]
/usr/lib/libc.so.6(+0x11918c) [0x726f8e52918c]

2595904 bio_close_file
/usr/lib/libc.so.6(+0x9de22) [0x726f8e4ade22]
/usr/lib/libc.so.6(+0x91fda) [0x726f8e4a1fda]
/usr/lib/libc.so.6(+0x9264c) [0x726f8e4a264c]
/usr/lib/libc.so.6(pthread_cond_wait+0x14e) [0x726f8e4a4d1e]
valkey-server *:6379(bioProcessBackgroundJobs+0x1b4) [0x6530abb46db4]
/usr/lib/libc.so.6(+0x957eb) [0x726f8e4a57eb]
/usr/lib/libc.so.6(+0x11918c) [0x726f8e52918c]

2595901 valkey-server *
/usr/lib/libc.so.6(+0x3def0) [0x726f8e44def0]
/usr/lib/libc.so.6(+0x16ed1d) [0x726f8e57ed1d]
valkey-server *:6379(sdscatfmt+0x894) [0x6530abaa24a4]
valkey-server *:6379(luaCallFunction+0x39a) [0x6530abbc66ea]
valkey-server *:6379(+0x1a0992) [0x6530abbc6992]
valkey-server *:6379(scriptingEngineCallFunction+0x98) [0x6530abbc1298]
valkey-server *:6379(+0x11ff55) [0x6530abb45f55]
valkey-server *:6379(call+0x174) [0x6530aba94454]
valkey-server *:6379(processCommand+0x93d) [0x6530aba958dd]
valkey-server *:6379(processCommandAndResetClient+0x21) [0x6530abaa9d11]
valkey-server *:6379(processInputBuffer+0xe3) [0x6530abaaee83]
valkey-server *:6379(readQueryFromClient+0x65) [0x6530abaaef55]
valkey-server *:6379(+0x18e31a) [0x6530abbb431a]
valkey-server *:6379(aeProcessEvents+0x24a) [0x6530aba790ca]
valkey-server *:6379(aeMain+0x2d) [0x6530aba7938d]
valkey-server *:6379(main+0x3f6) [0x6530aba6e7b6]
/usr/lib/libc.so.6(+0x276b5) [0x726f8e4376b5]
/usr/lib/libc.so.6(__libc_start_main+0x89) [0x726f8e437769]
valkey-server *:6379(_start+0x25) [0x6530aba70235]

2595906 bio_lazy_free
/usr/lib/libc.so.6(+0x9de22) [0x726f8e4ade22]
/usr/lib/libc.so.6(+0x91fda) [0x726f8e4a1fda]
/usr/lib/libc.so.6(+0x9264c) [0x726f8e4a264c]
/usr/lib/libc.so.6(pthread_cond_wait+0x14e) [0x726f8e4a4d1e]
valkey-server *:6379(bioProcessBackgroundJobs+0x1b4) [0x6530abb46db4]
/usr/lib/libc.so.6(+0x957eb) [0x726f8e4a57eb]
/usr/lib/libc.so.6(+0x11918c) [0x726f8e52918c]

4/4 expected stacktraces.

------ STACK TRACE DONE ------

------ REGISTERS ------
2595901:M 18 Jun 2025 01:20:12.920 # 
RAX:0000000000000000 RBX:0000726f8dd35663
RCX:0000000000000000 RDX:0000000000000000
RDI:0000000000000000 RSI:0000000000000010
RBP:00007ffc2b821a80 RSP:00007ffc2b821938
R8 :000000000000000c R9 :00006530abc111b8
R10:0000000000000001 R11:0000000000000003
R12:00006530abc49adc R13:00006530abc111b7
R14:0000000000000001 R15:0000000000000001
RIP:0000726f8e57ed1d EFL:0000000000010283
CSGSFS:002b000000000033
2595901:M 18 Jun 2025 01:20:12.921 * hide-user-data-from-log is on, skip logging stack content to avoid spilling user data.

------ INFO OUTPUT ------
# Server
redis_version:7.2.4
server_name:valkey
valkey_version:8.1.2
valkey_release_stage:ga
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:38d65aa7b4148d2c
server_mode:standalone
os:Linux 6.14.6-arch1-1 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
gcc_version:15.1.1
process_id:2595901
process_supervised:no
run_id:a0b75f67a217a81142f17553028c010e86c1ee80
tcp_port:6379
server_time_usec:1750209612917634
uptime_in_seconds:16
uptime_in_days:0
hz:10
configured_hz:10
clients_hz:10
lru_clock:5379148
executable:/home/fusl/valkey-server
config_file:
io_threads_active:0
availability_zone:
listener0:name=tcp,bind=*,bind=-::*,port=6379

# Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
pubsub_clients:0
watching_clients:0
clients_in_timeout_table:0
total_watched_keys:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
paused_reason:none
paused_actions:none
paused_timeout_milliseconds:0

# Memory
used_memory:911824
used_memory_human:890.45K
used_memory_rss:15323136
used_memory_rss_human:14.61M
used_memory_peak:911824
used_memory_peak_human:890.45K
used_memory_peak_perc:100.29%
used_memory_overhead:892232
used_memory_startup:891824
used_memory_dataset:19592
used_memory_dataset_perc:97.96%
allocator_allocated:1845952
allocator_active:1986560
allocator_resident:6672384
allocator_muzzy:0
total_system_memory:67323842560
total_system_memory_human:62.70G
used_memory_lua:34816
used_memory_vm_eval:34816
used_memory_lua_human:34.00K
used_memory_scripts_eval:184
number_of_cached_scripts:1
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:33792
used_memory_vm_total:68608
used_memory_vm_total_human:67.00K
used_memory_functions:224
used_memory_scripts:408
used_memory_scripts_human:408B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:0
allocator_rss_ratio:3.36
allocator_rss_bytes:4685824
rss_overhead_ratio:2.30
rss_overhead_bytes:8650752
mem_fragmentation_ratio:17.18
mem_fragmentation_bytes:14431168
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:0
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
mem_overhead_db_hashtable_rehashing:0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1750209596
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:1
total_commands_processed:0
instantaneous_ops_per_sec:0
total_net_input_bytes:34
total_net_output_bytes:0
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
evicted_scripts:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:1
total_writes_processed:0
io_threaded_reads_processed:0
io_threaded_writes_processed:0
io_threaded_freed_objects:0
io_threaded_accept_processed:0
io_threaded_poll_processed:0
io_threaded_total_prefetch_batches:0
io_threaded_total_prefetch_entries:0
client_query_buffer_limit_disconnections:0
client_output_buffer_limit_disconnections:0
reply_buffer_shrinks:0
reply_buffer_expands:0
eventloop_cycles:170
eventloop_duration_sum:17739
eventloop_duration_cmd_sum:0
instantaneous_eventloop_cycles_per_sec:9
instantaneous_eventloop_duration_usec:99
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:0
replicas_waiting_psync:0
master_failover_state:no-failover
master_replid:d35a0bb7979f490a60174bb363524431d7eb2428
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:10485760
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.012543
used_cpu_user:0.016853
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.012440
used_cpu_user_main_thread:0.016714

# Modules

# Commandstats

# Errorstats

# Latencystats

# Cluster
cluster_enabled:0

# Keyspace

------ CLIENT LIST OUTPUT ------
id=2 addr=127.0.0.1:41372 laddr=127.0.0.1:6379 fd=10 name=*redacted* age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=12 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17060 events=r cmd=eval user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=34 tot-net-out=0 tot-cmds=0

------ CURRENT CLIENT INFO ------
id=2 addr=127.0.0.1:41372 laddr=127.0.0.1:6379 fd=10 name=*redacted* age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=12 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17060 events=r cmd=eval user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=34 tot-net-out=0 tot-cmds=0
argc: 3
argv[0]: "eval"
argv[1]: 7 bytes
argv[2]: 1 bytes

------ EXECUTING CLIENT INFO ------
id=2 addr=127.0.0.1:41372 laddr=127.0.0.1:6379 fd=10 name=*redacted* age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=12 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17060 events=r cmd=eval user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=34 tot-net-out=0 tot-cmds=0
argc: 3
argv[0]: "eval"
argv[1]: 7 bytes
argv[2]: 1 bytes

------ MODULES INFO OUTPUT ------

------ CONFIG DEBUG OUTPUT ------
repl-diskless-load disabled
debug-context ""
sanitize-dump-payload no
lazyfree-lazy-user-del yes
lazyfree-lazy-server-del yes
import-mode no
lazyfree-lazy-user-flush yes
list-compress-depth 0
dual-channel-replication-enabled no
repl-diskless-sync yes
activedefrag no
lazyfree-lazy-expire yes
io-threads 1
replica-read-only yes
client-query-buffer-limit 1gb
slave-read-only yes
lazyfree-lazy-eviction yes
proto-max-bulk-len 512mb

------ FAST MEMORY TEST ------
2595901:M 18 Jun 2025 01:20:12.921 # Bio worker thread #0 terminated
2595901:M 18 Jun 2025 01:20:12.921 # Bio worker thread #1 terminated
2595901:M 18 Jun 2025 01:20:12.921 # Bio worker thread #2 terminated
*** Preparing to test memory region 6530abce2000 (212992 bytes)
*** Preparing to test memory region 726f8af7f000 (2621440 bytes)
*** Preparing to test memory region 726f8b200000 (8388608 bytes)
*** Preparing to test memory region 726f8ba00000 (4194304 bytes)
*** Preparing to test memory region 726f8bffe000 (8388608 bytes)
*** Preparing to test memory region 726f8c7ff000 (8388608 bytes)
*** Preparing to test memory region 726f8d000000 (8388608 bytes)
*** Preparing to test memory region 726f8dc00000 (4194304 bytes)
*** Preparing to test memory region 726f8e290000 (16384 bytes)
*** Preparing to test memory region 726f8e3d2000 (20480 bytes)
*** Preparing to test memory region 726f8e5f8000 (32768 bytes)
*** Preparing to test memory region 726f8eb58000 (12288 bytes)
*** Preparing to test memory region 726f8eb5c000 (16384 bytes)
*** Preparing to test memory region 726f8ed63000 (4096 bytes)
*** Preparing to test memory region 726f8eef2000 (397312 bytes)
*** Preparing to test memory region 726f8efc7000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /usr/lib/libc.so.6 (base 0x726f8e410000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------

=== VALKEY BUG REPORT END. Make sure to include from START to END. ===
```

---------

Signed-off-by: Fusl <fusl@meo.ws>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-06-18 13:23:46 -04:00
Simon BaatzandGitHub 5751046b8e Improve "SENTINEL FAILOVER" by using the "FAILOVER" command (#1292)
This PR implements #1291 and consists of the following commits:

Two commits to fix problems with the sentinel tests:

Commit: **Wait for all Sentinels to be connected before starting tests**

Up to now the sentinel test initialization verified that all sentinels
detect each other.
However, detection does not imply connection, which led to intermittent
failures in the
coordinated failover tests (no leader elected since disconnected
sentinels do not take part in a
vote).

Fix this by waiting until no sentinel reports being "disconnected".

Commit: **sentinel-tests: Clean up config after config set tests**

Three commits with the actual implementation:

Commit: **Add option for coordinated failover to Sentinel** (coordinated
failover without leader election)
Commit: **Allow Sentinel to recover from a stuck FAILOVER**
Commit: **SENTINEL FAILOVER COORDINATED actually does a leader
election** (adds leader election before failover starts)

---------

Signed-off-by: Simon Baatz <gmbnomis@gmail.com>
2025-06-18 09:32:33 -04:00
Thalia ArchibaldandGitHub 72dc6841eb Add non-fatal unit test assertions (#2223)
Add a new kind of assertion, `TEST_EXPECT`, to the testing framework for
non-fatal assertions, where all such failed assertions are reported at
once. Existing tests still use fatal assertions.

Name it `TEST_EXPECT`, following the convention of [Google
Test](https://google.github.io/googletest/reference/assertions.html),
which uses “assert” and “expect” with the same fatal/non-fatal
distinction.

Signed-off-by: Thalia Archibald <thalia@archibald.dev>
2025-06-18 11:54:51 +02:00
9dc3d00a69 Fix the incorrect stdout redirect logic, use dup2 to redirect stdout (#2225)
Use the standard `dup2` to redirect `stdout`

Closes #2224
Follow-up of #1743

Signed-off-by: skyfirelee <739609084@qq.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-06-18 16:45:32 +08:00
BinbinandGitHub c5aeccade7 Reset failover failed primary rank when doing a manual failover (#2226)
This rank was added in #1018, we should reset it to zero when doing
the manual failover, just like we reset failover auth rank.

It affects subsequent log printing. When there is a manual failover
and there are multiple primaries failure, `Start of election` will
print a confusing log.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-18 10:27:18 +08:00
2670f54678 Optimize scan/sscan/hscan/zscan commands by replacing list with vector (#2160)
The scan/sscan/hscan/zscan commands store iterated keys and values in
a `list`, where each list element triggers a separate memory allocation
and deallocation. This causes performance degradation when the list
contains a large number of elements.

We reduce memory allocation/deallocation by replacing `list` with
`vector`, a new dynamic array implementation.

For specific performance gains, refer to the benchmark results below.

### Benchmark
Run the benchmark script five times, and take the peak value as the
final result.

#### Scan command
**benchmark script**
```
valkey-benchmark -r 1000000000 -n 10000000 -P 32 set key-__rand_int__ value-__rand_int__
for count in `echo 4 16 64 512`;do
	for((x=0;x<5;x+=1));do
		valkey-benchmark --threads 2 -r 100000000 -n 2000000  scan __rand_int__ count $count;
	done
done
```
**benchmark result**
count | QPS after optimization | QPS before optimization | Performance
Boost
-- | -- | -- | --
4 | 114148.73| 108090.58 | 5.6%
16 | 98745.93| 87892.77 | 12.3%
64 | 64432.99| 49971.27 | 28.9%
512 | 15725.87 | 10174.75| 54.5%

#### Hscan Command
**benchmark script**
```
valkey-benchmark -r 1024 -n 1000000 -P 32 hset hash field-__rand_int__ value-__rand_int__
for count in `echo 4 16 64 512`;do
	for((x=0;x<5;x+=1));do
		valkey-benchmark --threads 2 -r 100000000 -n 2000000 hscan hash __rand_int__ count $count;
	done
done
```
**benchmark result**
count | QPS after optimization | QPS before optimization | Performance
Boost
-- | -- | -- | --
4 | 115921.87| 114266.12 | 1.4%
16 | 103879.91| 97546.70 | 6.5%
64 | 71405.62 | 62976.26 | 13.4%
512 | 23168.53 | 16064.64| 44.2%

#### Sscan command
**benchmark script**
```
valkey-benchmark -r 1024 -n 1000000 -P 32 sadd set element-__rand_int__
for count in `echo 4 16 64 512`;do
	for((x=0;x<5;x+=1));do
		valkey-benchmark --threads 2 -r 100000000 -n 2000000 sscan set __rand_int__ count $count;
	done
done
```
**benchmark result**
count | QPS after optimization | QPS before optimization | Performance
Boost
-- | -- | -- | --
4 | 123054.20| 119388.73 | 3.1%
16 | 115928.59| 110975.48 | 4.5%
64 | 94099.94| 86945.18 | 8.2%
512 | 40320.95 | 32501.83| 24.1%

#### Zscan command
**benchmark script**
```
valkey-benchmark -r 1024 -n 1000000 -P 32 zadd zset __rand_int__ element-__rand_int__
for count in `echo 4 16 64 512`;do
	for((x=0;x<5;x+=1));do
		valkey-benchmark --threads 2 -r 100000000 -n 2000000 zscan zset __rand_int__ count $count;
	done
done
```
**benchmark result**
count | QPS after optimization | QPS before optimization | Performance
Boost
-- | -- | -- | --
4 | 97437.40| 96362.33 | 1.1%
16 | 71405.62|70155.75 | 1.8%
64 | 35066.80| 33002.75 | 6.3%
512 | 7498.02 | 6654.89| 12.7%

CPU: AMD EPYC 9754 128-Core Processor * 8
OS: 	Ubuntu Server 22.04 LTS 64bit
Memory: 16GB
VM: Tencent cloud SA5 | SA5.2XLARGE16
Server startup command: valkey-server --save '' --appendonly no

---------

Signed-off-by: chzhoo <czawyx@163.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-06-18 00:13:29 +02:00
7d5fb03df5 Prevent getNodeByQuery from leaking DB changes into client (#2206)
To support multiple databases in cluster mode (see #1671),
`getNodeByQuery` temporarily switches databases when tracking `SELECT`
statements during slot migration/import. The intended logic is to revert
any database change after the operation. However, this approach is
flawed: in some transactions the database change is not properly
reverted, causing the client to remain on the wrong database.

For example, if a transaction includes `SELECT` statements, the current
database may be changed even if the transaction is never executed (see
added test).

Fix the issue by saving the original database once and restoring to it
after a switch.

---------

Signed-off-by: Simon Baatz <gmbnomis@gmail.com>
Signed-off-by: Simon Baatz <gmbnomis@users.noreply.github.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-06-16 15:01:41 +02:00
dc8662fc8c Support for RDB analysis reports (#1743)
The PR add analysis capabilities to the valkey-check-rdb tool. Now the
tool can statistically analyze various types of keys, expired keys, and
key elements.

- Added the --stats parameter to the valkey-check-rdb tool's rdb
analysis functionality, which enables opening reports.
- Supported the --format parameter with three options, table/csv/info,
to provide more flexible output formats.
- Add function & lua script number info

The content format is as follows:
```
... ...
[offset 8999] \o/ RDB looks OK! \o/
[info] 60 keys read
[info] 0 expires
[info] 0 already expired
db.0.type.name                  string  list    set     zset    hash    module  stream
db.0.keys.total                 10      0       0       0       0       0       0    
db.0.exipre_keys.total          0       0       0       0       0       0       0    
db.0.already_expired.total      0       0       0       0       0       0       0    
db.0.keys.size                  200     0       0       0       0       0       0    
db.0.keys.value_size            2000    0       0       0       0       0       0    
db.0.elements.total             10      0       0       0       0       0       0    
db.0.elements.size              2000    0       0       0       0       0       0    
db.0.elements.num.max           1       0       0       0       0       0       0    
db.0.elements.num.avg           1.00    0.00    0.00    0.00    0.00    0.00    0.00 
db.0.elements.num.p99           1.00    0.00    0.00    0.00    0.00    0.00    0.00 
db.0.elements.num.p90           1.00    0.00    0.00    0.00    0.00    0.00    0.00 
db.0.elements.num.p50           1.00    0.00    0.00    0.00    0.00    0.00    0.00 
db.0.elements.size.max          200     0       0       0       0       0       0    
db.0.elements.size.avg          200.00  0.00    0.00    0.00    0.00    0.00    0.00 
db.0.elements.size.p99          200.00  0.00    0.00    0.00    0.00    0.00    0.00 
db.0.elements.size.p90          200.00  0.00    0.00    0.00    0.00    0.00    0.00 
db.0.elements.size.p50          200.00  0.00    0.00    0.00    0.00    0.00    0.00
```

---------

Signed-off-by: artikell <739609084@qq.com>
Signed-off-by: skyfirelee <739609084@qq.com>
Signed-off-by: wei.kukey <wei.kukey@gmail.com>
Co-authored-by: wei.kukey <wei.kukey@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-06-16 11:35:15 +02:00
xbaselandGitHub 0ac96c9c3c test: fix UBSan warning due to misaligned memory access in unit test_addRepliesWithOffloadsToList (#2201)
Replaced direct cast to robj** with memcpy to avoid undefined behavior
on unaligned buffers, which caused test failure when built with UBSan.

Follow-up of https://github.com/valkey-io/valkey/pull/2078

Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>
2025-06-16 11:23:32 +02:00
BinbinandGitHub 2011422925 Updates for lttng daily CI to be able to skip it or notify it (#2197)
lttng support was added in #2070, update daily to be able
to skip it no notify it.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-12 11:30:23 +08:00
BinbinandGitHub ace08b1d45 Remove unused repl_offset_time field in clusterNode (#2198)
We add this field in 69b8f6c5b2,
but never used. We can easily add it back if needed in the future.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-12 10:27:39 +08:00
f924b65a41 Stabilize dual channel test: Split scenarios and set RDB loading interval (#2200)
1. Split the dual-channel test into two independent test scopes:
 - One for killing the RDB connection.
 - One for killing the main connection.
2. Increase frequency of `rdbLoadProgressCallback` calls by reducing
`loading-process-events-interval-bytes` to 1024.
This adjustment is needed because `rdb-key-save-delay` slows down bulk
transfers, which may cause the replica to miss main connection messages.
The change ensures that:
 - Messages on main channel are not missed.
 - Test properly handles graceful shutdown after user interrupt.

Signed-off-by: Stav Bentov <stavbt@amazon.com>
Co-authored-by: Stav Bentov <stavbt@amazon.com>
2025-06-11 18:01:08 +03:00
BinbinandGitHub b9ffae9e39 Add packet-drop to fix the new flaky failover test (#2196)
The new test was added in #2178, obviously there may be
pending reads in the connection, so there may be a race
in the DROP-CLUSTER-PACKET-FILTER part causing the test
to fail. Add CLOSE-CLUSTER-LINK-ON-PACKET-DROP to ensure
that the replica does not process the packet.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-11 17:01:06 +08:00
Amit NaglerandGitHub 0ef9fbf090 fix: use replica's last selected db for dual-channel sync (#2188)
When sending current offset to replica, use the last selected db sent to
the replicas instead of always using `server.db[0]->id`. This fixes a
potential issue where the dual channel sync relied on the primary always
resending the `SELECT` command after PSYNC.

While the db id argument could be removed from the sync API, it's
retained to maintain backward compatibility.

Fixed #2194.

Signed-off-by: naglera <anagler123@gmail.com>
2025-06-11 17:00:41 +08:00
Viktor SöderqvistandGitHub f682352e61 Run external cluster tests with multiple databases (#2176)
Change the runtest flag `--cluster-mode` to no longer imply
`--single-db`. This allows more tests to run against an external node
running in cluster mode.

Also fix incorrectly tagged tests so that `./runtest --singledb` passes
locally.

Without this change, this doesn't pass because several tests are
incorrectly tagged as external:skip instead of singledb:skip or
cluster:skip. In the CI, we apparently only run with singledb when
implied by `--cluster-mode` (until now), which we only use in the
external cluster mode test. That's why the incorrect tags were not
detected.

* singledb:skip is used for tests that do select and assume that a
client has selected db 9.
* cluster:skip is used when the test uses SWAPDB, cross-slot commands
other things forbidden in cluster mode.

Additional fix: End tests with a linebreak when running with the
`--fastfail` flag. A program that doesn't end its output with a
linebreak annoys the prompt in the terminal.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-06-11 09:38:43 +02:00
kukeyandGitHub 7d09d63598 valkey-check-rdb: Fix truncated long aux fields (#2193)
In some case, like in rdb has a long lua, the lua will be incomplete printed.

Signed-off-by: wei.kukey <wei.kukey@gmail.com>
2025-06-10 19:48:45 +08:00
1d517481b0 Call makeObjectShared to normalize a shared object into a true shared object (#1566)
Currently, only int and shared.redacted call makeObjectShared for
shared objects.

Although this breaks the blame log, there is a few points i would
like to mention:
1. For threading, our fork for some reasons calling decrRefCount in
   the threads for shared.command. Although this is not used in OSS
   code now, it may be a issue in the future.
2. For cleanup, all shared objects's refcount should be the same.
3. For CoW, probably can reduce CoW although it is very minor.

We decided in #2189 to keep `extended-redis-compatibility` for now.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-06-10 19:48:25 +08:00
BinbinandGitHub 79c11e4b8f Update extended-redis-compatibility conf to remove the specific words (#2192)
In #2189, we decided to keep it for at least another year.
Closes #2189.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-10 19:48:03 +08:00
BinbinandGitHub 0301b7bc60 Update commandlog parameters conf text to mention special value settings (#2074)
Earlier we described the `slowlog-log-slower-than` configuration
option like this, we explicitly mentioned the special meaning of
negative numbers (actually -1) and 0.
```
The following time is expressed in microseconds, so 1000000 is equivalent
to one second. Note that a negative number disables the slow log, while
a value of zero forces the logging of every command.
slowlog-log-slower-than 10000
```

And after #1294 we lost this text, we need to mention these values,
and we can mention the special number for all command log's configs,
also it seems we should also mention `slowlog-max-len`.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-10 19:47:49 +08:00
5f6019bb33 On-demand database allocation instead of preallocation (#1609)
Allocate database structures lazily to prevent excessive memory usage
when a large number of databases is configured but not actually used.


fixes https://github.com/valkey-io/valkey/issues/1597

---------

Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-06-10 11:50:51 +02:00
charsyamandGitHub c4b15e2a74 Fix invalid functionname processMultibulkBuffer typo in comments. (#2097)
Fixed processMultiBulkBuffer to processMultibulkBuffer.

processMultibulkBuffer is real function name but there is a typo to
write as processMultiBulkBuffer, B should be lowercase.

Signed-off-by: charsyam <charsyam@naver.com>
2025-06-10 17:07:37 +08:00
BinbinandGitHub 21ad38a616 Fix cluster myself CLUSTER SLOTS/NODES wrong port after updating port/tls-port (#2186)
When modifying port or tls-port through config set, we need to call
clusterUpdateMyselfAnnouncedPorts to update myself's port, otherwise
CLUSTER SLOTS/NODES will be old information from myself's perspective.

In addition, in some places, such as clusterUpdateMyselfAnnouncedPorts
and clusterUpdateMyselfIp, beforeSleep save is added so that the
new ip info can be updated to nodes.conf.

Remove clearCachedClusterSlotsResponse in updateClusterAnnouncedPort
since now we add beforeSleep save in clusterUpdateMyselfAnnouncedPorts,
and it will call clearCachedClusterSlotsResponse.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-10 10:31:46 +08:00
BinbinandGitHub 4c6f9dfd15 CLIENT UNBLOCK should't be able to unpause paused clients (#2117)
When a client is blocked by something like `CLIENT PAUSE`, we should not
allow `CLIENT UNBLOCK timeout` to unblock it, since some blocking types
does not has the timeout callback, it will trigger a panic in the core,
people should use `CLIENT UNPAUSE` to unblock it.

Also using `CLIENT UNBLOCK error` is not right, it will return a UNBLOCKED
error to the command, people don't expect a `SET` command to get an error.

So in this commit, in these cases, we will return 0 to `CLIENT UNBLOCK`
to indicate the unblock is fail. The reason is that we assume that if
a command doesn't expect to be timedout, it also doesn't expect to be
unblocked by `CLIENT UNBLOCK`.

The old behavior of the following command will trigger panic in timeout
and get UNBLOCKED error in error. Under the new behavior, client unblock
will get the result of 0.
```
client 1> client pause 100000 write
client 2> set x x

client 1> client unblock 2 timeout
or
client 1> client unblock 2 error
```

Potentially breaking change, previously allowed `CLIENT UNBLOCK error`.
Fixes #2111.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-10 10:29:25 +08:00
3739d4c96b Fix replica can't finish failover when config epoch is outdated (#2178)
When the primary changes the config epoch and then down immediately,
the replica may not update the config epoch in time. Although we will
broadcast the change in cluster (see #1813), there may be a race in
the network or in the code. In this case, the replica will never finish
the failover since other primaries will refuse to vote because the
replica's slot config epoch is old.

We need a way to allow the replica can finish the failover in this case.

When the primary refuses to vote because the replica's config epoch is
less than the dead primary's config epoch, it can send an UPDATE packet
to the replica to inform the replica about the dead primary. The UPDATE
message contains information about the dead primary's config epoch and
owned slots. The failover will time out, but later the replica can try
again with the updated config epoch and it can succeed.

Fixes #2169.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-06-09 18:30:16 -07:00
a68feac835 [NEW] Introduce lttng based tracing (#2070)
## Introduce

In a production environment, it's quite challenging to figure out why a
Valkey is under high load. Right now, tools like INFO or slowlog can
offer some clues. But if the Valkey can't respond, we might not get any
information at all.
Usually, we have to rely on tools like `strace` or `perf` to find the
root cause. If we set up trace points in advance during the project
development, we can quickly pinpoint performance issues.

In this current PR, support has been added for all latency sampling
points. Also, information reporting for command execution has been
added. At the same time, it supports dynamically turning on or off the
information reporting as required. The trace feature is implemented
based on LTTng, and this capability is supported in projects like QEMU,
Ceph.

## How to use

Building Valkey with LTTng support:

```
USE_LTTNG=yes make
```

Open event report:
```
config set trace-events "sys server db cluster aof commands"
```

Events are classified as follows:
- sys (System-level operations)
- server (Server core logic)
- db (Database core operations)
- cluster (Cluster configuration operations)
- aof (AOF persistence operations)
- commands(Command execution information)

## How to trace

Enable lttng trace events dynamically:
```
~# lttng destroy valkey
~# lttng create valkey
~# lttng enable-event -u valkey:*
~# lttng track -u -p `pidof valkey-server`
~# lttng start
~# lttng stop
~# lttng view
```

Examples (a client run 'SET', another run 'keys'):

```
[15:30:19.334463706] (+0.000001243) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 0 }
[15:30:19.334465183] (+0.000001477) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 1 }
[15:30:19.334466516] (+0.000001333) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 0 }
[15:30:19.334467738] (+0.000001222) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 0 }
[15:30:19.334469105] (+0.000001367) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 1 }
[15:30:19.334470327] (+0.000001222) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 0 }
[15:30:19.369348485] (+0.034878158) libai valkey:command_call: { cpu_id = 15 }, { name = "keys", duration = 34874 }
[15:30:19.369698322] (+0.000349837) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 4 }
[15:30:19.369702327] (+0.000004005) libai valkey:command_call: { cpu_id = 15 }, { name = "set", duration = 2 }

```

Then we can use another script to analyze topN slow commands and other
system
level events.

About performance overhead (valkey-benchmark -t get -n 1000000 --threads
4):
1> no lttng builtin: 285632.69 requests per second
2> lttng builtin, no trace: 285551.09 requests per second (almost 0
overhead)
3> lttng builtin, trace commands: 266595.59 requests per second (about
~6.6 overhead)

Generally valkey-server would not run in full utilization, the overhead
is acceptable.

## Problem analysis

Add prot and conn field into trace command

Run benchmark tool:
```
GET: rps=227428.0 (overall: 222756.2) avg_msec=0.114 (overall: 0.117)
GET: rps=225248.0 (overall: 223005.2) avg_msec=0.115 (overall: 0.117)
GET: rps=167474.1 (overall: 217942.2) avg_msec=0.193 (overall: 0.122) --> performance drop
GET: rps=220192.0 (overall: 218129.5) avg_msec=0.118 (overall: 0.122)
GET: rps=222868.0 (overall: 218493.7) avg_msec=0.117 (overall: 0.121)

```
Run another 'keys *' command in another connection, lead benchmark
performance
drop.

At the same time, lttng traces events:
```
[21:16:30.420997167] (+0.000004064) zhenwei valkey:command_call: { cpu_id = 6 }, { prot = "tcp", conn = "127.0.0.1:6379-127.0.0.1:54668", name = "get", duration = 1 }
[21:16:30.421001262] (+0.000004095) zhenwei valkey:command_call: { cpu_id = 6 }, { prot = "tcp", conn = "127.0.0.1:6379-127.0.0.1:54782", name = "get", duration = 1 }
[21:16:30.485562459] (+0.064561197) zhenwei valkey:command_call: { cpu_id = 6 }, { prot = "tcp", conn = "127.0.0.1:6379-127.0.0.1:54386", name = "keys", duration = 64551 } --> root cause
[21:16:30.485583101] (+0.000020642) zhenwei valkey:command_call: { cpu_id = 6 }, { prot = "tcp", conn = "127.0.0.1:6379-127.0.0.1:54522", name = "get", duration = 1 }
[21:16:30.485763891] (+0.000180790) zhenwei valkey:command_call: { cpu_id = 6 }, { prot = "tcp", conn = "127.0.0.1:6379-127.0.0.1:54542", name = "get", duration = 1 }
[21:16:30.485766451] (+0.000002560) zhenwei valkey:command_call: { cpu_id = 6 }, { prot = "tcp", conn = "127.0.0.1:6379-127.0.0.1:54438", name = "get", duration = 1 }
```

From this change, we can see that connection
127.0.0.1:6379-127.0.0.1:54386
affects other connections.

---------

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
Signed-off-by: artikell <739609084@qq.com>
Signed-off-by: skyfirelee <739609084@qq.com>
Co-authored-by: zhenwei pi <pizhenwei@bytedance.com>
2025-06-08 14:39:57 -07:00
f9e7a9f183 Reply Copy Avoidance (#2078)
### Overview
This PR introduces the ability to avoid copying the content of string
object into replies (i.e. bulk string replies) and to allow I/O threads
refer directly to obj->ptr in writev iov.

### Key Changes
* Added capability to reply construction allowing to interleave regular
replies with copy avoid replies in client reply buffers
* Extended write-to-client handlers to support copy avoid replies
* Added copy avoidance of string bulk replies when copy avoidance
indicated by I/O threads
* Minor changes in cluster slots stats in order to support
`network-bytes-out` for copy avoid replies
* Copy avoidance is beneficial for performance despite object size only
starting certain number of threads. So it will be enabled only starting
certain number of threads. Internal configuration
``min-io-threads-copy-avoid`` introduced to manage this number of
threads

**Note**: When copy avoidance disabled content and handling of client
reply buffers remains as before this PR

### Implementation Details
####  ``client`` and  ``clientReplyBlock`` structs:
1. ``buf_encoded`` flag has been added to ``clientReplyBlock`` struct
and to ``client`` struct for static ``c->buf`` to indicate if reply
buffer is in copy avoidance mode (i.e. include headers and payloads) or
not (i.e. plain replies only).
2. ``io_last_written_buf``, ``io_last_written_bufpos``,
``io_last_written_data_len`` fields added ``client`` struct to to keep
track of write state between ``writevToClient`` invocations
####  Reply construction:
1. Original ```_addReplyToBuffer``` and ```_addReplyProtoToList``` have
been renamed to ```_addReplyPayloadToBuffer``` and
```_addReplyPayloadToList``` and extended to support different types of
payloads - regular replies and copy avoid replies.
3. New ```_addReplyToBuffer``` and ```_addReplyProtoToList``` calls now
```_addReplyPayloadToBuffer``` and ```_addReplyPayloadToList``` and used
for adding **regular** replies to client reply buffers.
4. Newly introduced ```_addBulkOffloadToBuffer``` and
```_addBulkOffloadToList``` are used for adding **copy avoid** replies
to client reply buffers.
 
####  Write-to-client infrastructure:
The ```writevToClient``` and ```_postWriteToClient``` has been
significantly changed to support copy avoidance capability.

####  Debug configuration:
1. ``min-io-threads-avoid-copy-reply`` - Minimum number of IO threads
for copy avoidance
2. ``min-string-size-avoid-copy-reply`` - Minimum bulk string size for
copy avoidance when IO threads disabled
3. ``min-string-size-avoid-copy-reply-threaded`` - Minimum bulk string
size for copy avoidance when IO threads enabled

### Testing
1. Existing unit and integration tests passed. Copy avoidance enabled on
tests with ``--io-threads`` flag
2. Added unit tests for copy avoidance functionality

### Performance Tests

Note: pay attention `io-threads 1` config means only main thread with no
additional io-threads, `io-threads 2` means main thread plus 1 I/O
thread, `io-threads 9` means main thread plus 8 I/O threads.

#### 512 byte object size
Tests are conducted on memory optimized instances using:
* 3,000,000 keys
* 512 bytes object size 
* 1000 clients

|io-threads (including main thread)	|Plain Reply	|Copy Avoidance	|
|---	|---	|---	|
|7	|1,160,000	|1,160,000	|
|8	|1,150,000	|1,280,000	|
|9	|1,150,000	|1,330,000	|
|10	|N/A	|1,380,000	|
|11	|N/A	|1,420,000	|

#### Various object size, small number of threads

|iothreads |Data size |Keys |Clients |Instance type |Unstable branch
|Copy Avoidance On |
|---	|---	|---	|---	|---	|---	|---	|
|1	|512 byte	|3,000,000	|1,000	|memory optimized	|195,000	|195,000	|
|2	|512 byte	|3,000,000	|1,000	|memory optimized	|245,000	|245,000	|
|3	|512 byte	|3,000,000	|1,000	|memory optimized	|455,000	|459,000	|
|4	|512 byte	|3,000,000	|1,000	|memory optimized	|685,000	|685,000	|
|	|	|	|	|	|	|	|
|1	|1K	|3,000,000	|1,000	|memory optimized	|185,000	|185,000	|
|2	|1K	|3,000,000	|1,000	|memory optimized	|235,000	|235,000	|
|3	|1K	|3,000,000	|1,000	|memory optimized	|450,000	|450,000	|
|	|	|	|	|	|	|	|
|1	|4K	|1,000,000	|1,000	|network optimized	|182,000	|187,000	|
|2	|4K	|1,000,000	|1,000	|network optimized	|240,000	|238,000	|
|	|	|	|	|	|	|	|
|1	|16K	|1,000,000	|500	|network optimized	|100,000	|120,000	|
|2	|16K	|1,000,000	|500	|network optimized	|140,000	|140,000	|
|3	|16K	|1,000,000	|500	|network optimized	|275,000	|260,000	|
|	|	|	|	|	|	|	|
|1	|32K	|500,000	|500	|network optimized	|57,000	|90,000	|
|2	|32K	|500,000	|500	|network optimized	|110,000	|110,000	|
|3	|32K	|500,000	|500	|network optimized	|215,000	|215,000	|
|	|	|	|	|	|	|	|
|1	|64K	|100,000	|500	|network optimized	|30,000	|57,000	|
|2	|64K	|100,000	|500	|network optimized	|69,000	|61,000	|
|3	|64K	|100,000	|500	|network optimized	|120,000	|120,000	|
|4	|64K	|100,000	|500	|network optimized	|115,000 - 175,000	|175,000	|
|5	|64K	|100,000	|500	|network optimized	|115,000 - 165,000	|230,000	|

---------

Signed-off-by: Alexander Shabanov <alexander.shabanov@gmail.com>
Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Alexander Shabanov <alexander.shabanov@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-06-07 20:49:51 -07:00
Harkrishn PatroandGitHub c1562dbc16 Avoid log spam about cluster node failure detection by each primary (#2010)
After node failure detection/recovery and gossip by each primary, we log
about the failure detection/recovery at NOTICE level which can spam the
server and the behavior is quite expensive on ec2 burstable instance
types. I would prefer us rolling it back to VERBOSE level.

Change was introduced in #633

Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
2025-06-05 21:12:34 -07:00
Madelyn OlsonandGitHub 952e2a4ee0 Remove unnecessary refcount increment in propagateDelete (#2175)
The alsoPropagate increased the refcount, I'm guessing this code was
just left over from before that was needed.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-06-05 11:55:22 -04:00
BinbinandGitHub 6f963c2a53 Remove dead conditions around the multi/exec check (#2168)
After #723, this became dead conditions since we will reject
the command in advance and now it won't be able to go here
anymore (call won't be called anymore).

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-06-05 11:59:23 +08:00
7a6f5ffe70 Offload slot calculation and cross-slot detection to I/O threads (#2165)
Refactors `getNodeByQuery()` to be able to move the CRC16 slot
calculations in I/O threads and skip many checks if slot migrations are
not ongoing.

The slot calculation for a command is moved out of `getNodeByQuery()`
into a new function `clusterSlotByCommand()` which is safe to call from
I/O threads.

For MULTI-EXEC transactions, the slot is stored per command in the
multi-state, to be able to detect cross-slot transactions on EXEC
without computing the slots again.

Additionally, cross-slot detection and arity check is offloaded to I/O
threads. To unify the code paths for commands parsed by I/O threads and
commands parsed by the main thread, the command lookup, arity check,
slot lookup are moved out of `processCommand()` to a new
`prepareCommand()` function that needs to be called before
`processCommand()`.

The client's read flags are used for passing information about bad arity
and cross-slot. These flags are already used to convey information per
command between I/O thread and main thread.

Fixes #2077
Related to #632

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-06-04 09:50:12 +02:00
Rain ValentineandGitHub 748f4092f8 Combine range element ranks calculation with range elements search to improve zcount performance (#2129)
Currently in ZCOUNT we search the sorted list twice to locate the range
endpoints and then we use an extra 2 searches to take each point rank.
This PR introduce 2 modifications:
1. Calculate the rank while searching each of the range points, thus
avoid extra rank calculations
2. polish the code by removing the extra implementation for GetRank
which was kept in order to avoid zcount performance degradation.

Following tests this PR seems to improve ZCOUNT performance by ~10%

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
2025-06-04 07:47:55 +03:00
Viktor SöderqvistandGitHub d87b4d8a99 Delete comment visible in runtest --help output (#2164)
In e15a45b3bd a comment line was added to aid the formatting the line
lengths in the --help output. This looks like TCL comment but
unfurtunately it became part of the output.

This fixes it and also linebreaks a long line in the same output.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-06-02 21:17:21 +02:00
114d768caa Add CLUSTER FLUSHSLOT command (#1384)
Add cluster flushslot command. Closes #1133.

Syntax:

    CLUSTER FLUSHSLOT <slot> [ASYNC|SYNC]

## Background

We want to implement the ability to SYNC or ASYNC free all data from a
single slot. This is useful in two cases:

1. When a primary loses ownership of a slot, it will temporarily freeze
while it synchronously frees the slot.
2. When doing resharding, you may prefer to forcibly delete all the data
from a slot and assign it to another node.

## Internal changes

Modify the function signature of `delKeysInSlot`, adding parameters
`lazy`, `propagate_del` and `send_del_event` to control the behavior.

---------

Signed-off-by: wuranxx <wuranxx@foxmail.com>
Signed-off-by: hwware <wen.hui.ware@gmail.com>
Signed-off-by: wuranxx <155042264+wuranxx@users.noreply.github.com>
Co-authored-by: hwware <wen.hui.ware@gmail.com>
Co-authored-by: Ping Xie <pingxie@outlook.com>
2025-06-02 14:06:54 -04:00
Madelyn OlsonandGitHub 60baa06a31 Have threads gracefully exit instead of killing them when checking thread count (#2156)
It appears that jemalloc *does not* like it when you cancel threads, and
will sometimes crash. It does seem to be happy when it's allowed to
gracefully exit though.

Reference
https://github.com/valkey-io/valkey/actions/runs/15312340723/job/43079580795?pr=2149#step:4:1444.

I also updated the test to randomly set the value a bunch, which makes
the test fail a little more consistently. Issue seems no longer present.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-06-01 22:54:32 -07:00
BinbinandGitHub 789c976273 Skip empty db when doing FLUSHALL to reduce some NOP (#2151)
When only a few dbs are used in multidb, FLUSHALL will do
some NOPs. For example, in FLUSHALL ASYNC we will need to
re-create objects and submit the bio jobs. This somehow will
cause FLUSHALL ASYNC to be logged in the slowlog.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-05-31 11:39:35 +08:00
Viktor SöderqvistandGitHub 314c7db8e7 Unify behavior of CLIENT REPLY in multi with other NO_MULTI commands (#2152)
CLIENT REPLY is forbidden inside transactions, but the error behavior is
different to other commands that are forbidden in transactions (such as
WATCH, SAVE or nested MULTI).

* The error message is different.
* The error within the transaction doesn't cause the EXEC to return
error EXECABORT as it should.

This change unifies the behavior and implementation with the other
commands by removing the special handling and instead just relying on
the NO_MULTI command flag.

Follow-up of #1966.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-30 21:03:22 +02:00
Madelyn OlsonandGitHub 6d5cb261e4 Add ricardo as a commiter (#2149)
Add @rjd15372 as one of the folks with write permissions on the Valkey
repo.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-05-30 10:45:13 -07:00
Jacob MurphyandGitHub 5dac09da93 Update tests to catch module context leaks if using aux save/load (#2150)
See #2132 for the fix.

This just ensures that we always create context in all branches of
aux_load and aux_save

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-05-30 10:28:55 +08:00
36c31c1d40 Incorporate Redis CVE for CVE-2025-27151 (#2146)
Resolves https://github.com/valkey-io/valkey/issues/2145

Incorporate the CVE patch that was sent to us by Redis Ltd.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Ping Xie <pingxie@outlook.com>
2025-05-28 16:06:23 -07:00
Ayush SharmaandGitHub ac2ca4a34b Allow dynamic modification of io-threads num (#2033)
Item from #761 

This PR has the following changes

1. Bug fix where calling `pthread_join()` from main thread for an IO
thread would hang indefinitely. This is because `IOThreadMain()` doesn't
have a cancellation point.So `pthread_cancel()` from main thread is not
honored.
Can be reproed by calling `shutdownIOThread()` from the main thread for
any active thread with empty job queue.
 Fixed by adding cancellation point in `IOThreadMain()`.
2. Makes `io-threads` config runtime modifiable.

Signed-off-by: Ayush Sharma <mrayushs933@gmail.com>
2025-05-28 11:25:02 -07:00
Madelyn OlsonandGitHub 2668ca6534 Correctly cast the extension lengths (#2144)
Correctly use a 32 bit integer for accumulating the length of ping
extensions.

The current code may accidentally truncate the length of an
extension that is greater than 64kb and fail the validation check. We
don't currently emit any extensions that are this length, but if we were
to do so in the future we might have issues with older nodes (without
this fix) will silently drop packets from newer nodes. We should
backport this to all versions.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-05-28 11:15:07 -07:00
9eff9f2a9c Prioritize replication traffic in the replica (#1838)
## Overview
In high-load scenarios, a replica might not consume replication data
fast enough, leading to backpressure on the primary. When the primary’s
buffer overflows, replication lag increases and it can drops the replica
connection, triggering a full sync, a costly operation that impacts
system performance.

The solution is to read from replication clients until their is no longer pending data, up to 25 iterations.

## Performance Impact ##

Test setup:
1. Bombard the replica with expensive commands, leading to high CPU
utilization
2. Write to the main database to trigger replication traffic

Metric | Before (repl-flow-control Disabled) | After (repl-flow-control
Enabled)
-- | -- | --
Throughput (requests/sec) | 941.71 | 760.98
Avg Latency (ms) | 52.865 | 65.534
p50 Latency (ms) | 59.743 | 68.543
p95 Latency (ms) | 79.231 | 106.687
p99 Latency (ms) | 90.303 | 126.527
Max Latency (ms) | 188.031 | 385.535

- Replication stability improves, no full syncs were observed after
enabling flow control.
- Higher latency for normal clients due to increased resource allocation
for replication.
- CPU and memory usage remain stable, with no major overhead.
- Replica throughput slightly decreases as replication takes priority.
    
Implements https://github.com/valkey-io/valkey/issues/1596

---------

Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-28 10:50:21 -07:00
chx9andGitHub 2eda2b5729 Improve clarity of errors for GEO commands when member does not exist (#1943)
this pr aims to fix one of the issues from Redis Triage

https://github.com/orgs/redis/projects/6?pane=issue&itemId=69775417&issue=redis%7Credis%7C13350
it has been more than 1 year since this issue brought up

---------

Signed-off-by: chx9 <cheng.huan@icloud.com>
2025-05-27 19:25:04 -07:00
KarthikSubbaraoandGitHub b9386893de Update GEOSEARCH and GEOSEARCHSTORE history documentation for the BYPOLYGON option (#2142)
Update GEOSEARCH and GEOSEARCHSTORE history documentation for the
BYPOLYGON option.

When I started working on the valkey-documentation, I identified this
change was missing from the PR
https://github.com/valkey-io/valkey/pull/1809/

Signed-off-by: KarthikSubbarao <karthikrs2021@gmail.com>
2025-05-28 01:29:42 +02:00
Hüseyin AçacakandGitHub 929fbc9804 Add size_t cast in modules (#2115)
This PR addresses a potential misalignment issue when using `va_args`.
Without this fix,
[argument](https://github.com/valkey-io/valkey/blob/ea7311b8f5ce653b7df4f9d338604379651057e7/src/module.c#L6173-L6188)
values may occasionally become incorrect due to stack alignment
inconsistencies.

Signed-off-by: Hüseyin Açacak <huseyin@janeasystems.com>
2025-05-27 12:37:11 -07:00
Harkrishn PatroandGitHub 8975c6bd89 Flip pfail flag while marking node as failed (#2012)
This fail logic was added in #1191, we should also clear the
pfail flag in this case.

Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
2025-05-27 15:05:20 -04:00
Viktor SöderqvistandGitHub d9d61283c1 Detect SSL_new() returning NULL in outgoing connections (#2140)
When creating an outgoing TLS connection, we don't check if `SSL_new()`
returned NULL.

Without this patch, the check was done only for incoming connections in
`connCreateAcceptedTLS()`. This patch moves the check to
`createTLSConnection()` which is used both for incoming and outgoing
connections.

This check makes sure we fail the connection before going any further,
e.g. when `connCreate()` is followed by `connConnect()`, the latter
returns `C_ERR` which is commonly detected where outgoing connections
are established, such where a replica connects to a primary.

```c
int connectWithPrimary(void) {
    server.repl_transfer_s = connCreate(connTypeOfReplication());
    if (connConnect(server.repl_transfer_s, server.primary_host, server.primary_port, server.bind_source_addr,
                    server.repl_mptcp, syncWithPrimary) == C_ERR) {
        serverLog(LL_WARNING, "Unable to connect to PRIMARY: %s", connGetLastError(server.repl_transfer_s));
        connClose(server.repl_transfer_s);
        server.repl_transfer_s = NULL;
        return C_ERR;
    }
```

For a more thorough explanation, see
https://github.com/valkey-io/valkey/issues/1939#issuecomment-2912177877.

Might fix #1939.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-27 20:31:12 +02:00
f81356f2d2 Support BYPOLYGON option for GEOSEARCH (#1809)
This PR implements the `BYPOLYGON` search (described here:
https://github.com/valkey-io/valkey/issues/1755) for the GEOSEARCH and
GEOSEARCHSTORE valkey commands

---------

Signed-off-by: KarthikSubbarao <karthikrs2021@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-27 20:06:33 +02:00
Jacob MurphyandGitHub af9223ff9f Free module context even if there was no content written in auxsave2 (#2132)
When a module saves to the Aux metadata with aux_save2 callback, the
module context is not freed if the module didn't save anything in the
callback.

https://github.com/valkey-io/valkey/blob/8.1.1/src/rdb.c#L1277-L1284.
Note that we return 0, however we should be doing the cleanup done on
https://github.com/valkey-io/valkey/blob/8.1.1/src/rdb.c#L1300-L1303
still.

Fixes #2125

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-05-27 19:56:10 +02:00
Ran ShidlansikandGitHub a6bcb092f3 Fix a case of out of bound read when cluster node ID is provided with wrong length in CLUSTER FORGET (#2108)
In case we issue a cluster forget with a node ID which has a smaller
length than 40 bytes, we can read over the allocated string when
checking the blacklist.

This will mainly impact memory inspection codes and should not introduce
any real bug.

Example:

```
==30858==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x603000024e3a at pc 0x7f9b9fe7821a bp 0x7ffeb5712980 sp 0x7ffeb5712128
READ of size 40 at 0x603000024e3a thread T0
```

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-05-26 11:12:30 -07:00
Zeroday BYTEandGitHub 2e02d0e864 Fix unsigned difference expression compared to zero (#2101)
https://github.com/valkey-io/valkey/blob/8509a063425a674688d26d2b8c641fda00889b23/src/networking.c#L886-L886

Fix the issue need to ensure that the subtraction `prev->size -
prev->used` does not underflow. This can be achieved by explicitly
checking that `prev->used` is less than `prev->size` before performing
the subtraction. This approach avoids relying on unsigned arithmetic and
ensures the logic is clear and robust.

The specific changes are:
1. Replace the condition `prev->size - prev->used > 0` with `prev->used
< prev->size`.
2. This change ensures that the logic checks whether there is remaining
space in the buffer without risking underflow.

**References**
[INT02-C. Understand integer conversion
rules](https://wiki.sei.cmu.edu/confluence/display/c/INT02-C.+Understand+integer+conversion+rules)
[CWE-191](https://cwe.mitre.org/data/definitions/191.html)


---

Signed-off-by: Zeroday BYTE <github@zerodaysec.org>
2025-05-26 14:57:00 +03:00
Ping XieandGitHub 1292edcbaf Fix fragile test in lazyfree in external test (#2138)
Make sure the previous test is really done before sampling `used_memory`
in external server case.

Signed-off-by: Ping Xie <pingxie@google.com>
2025-05-26 12:14:08 +08:00
Ping XieandGitHub 1857f03b28 Correct VM_Yield timing and deflake blockedclient.tcl (#2131)
Previously, the `next_yield_time` for a yielding module command was only
correctly calculated for the initial yield, taking into account whether
the server was loading an RDB. Subsequent yields incorrectly defaulted
to using `server.hz` for the calculation, regardless of the loading
state.

This bug had two consequences:
1. Incorrect Yield Timing: If not loading, the module command would
yield based on `server.hz` instead of the configured
`busy_reply_threshold`. This could lead to the main thread being blocked
for significantly longer or shorter periods than intended between
yields.
2. Test Flakiness: In the `blockedclient.tcl` test, this incorrect
timing meant that new commands from other clients were delayed until the
current, potentially overly long, yield cycle completed. With
`busy_time_limit` set to 50s, this significantly increased test duration
and risked timeouts, causing flakiness.

This commit refactors the `next_yield_time` calculation into a new
static function `computeNextYieldTime` and ensures it's called both
during the initial context creation and within `VM_Yield` after each
yield. This guarantees that `next_yield_time` is always predicated on
the current `server.loading` status.

The `busy_time_limit` in `blockedclient.tcl` has also been reduced from
50s to 5s. With the primary bug addressed, this shorter timeout is more
appropriate and helps the test complete faster, reducing the risk of
timeouts.

---------

Signed-off-by: Ping Xie <pingxie@google.com>
2025-05-25 16:53:35 -07:00
c9ec3d63d6 Only mark the client reprocessing flag when unblocked on keys (#2109)
When we refactored the blocking framework we introduced the client
reprocessing infrastructure. In cases the client was blocked on keys, it
will attempt to reprocess the command. One challenge was to keep track
of the command timeout, since we are reprocessing and do not want to
re-register the client with a fresh timeout each time. The solution was
to consider the client reprocessing flag when the client is
blockedOnKeys:

```
if (!c->flag.reprocessing_command) {
        /* If the client is re-processing the command, we do not set the timeout
         * because we need to retain the client's original timeout. */
        c->bstate->timeout = timeout;
    }
```

However, this introduced a new issue. There are cases where the client
will consecutive blocking of different types for example:
```
CLIENT PAUSE 10000 ALL
BZPOPMAX zset 1
```
would have the client blocked on the zset endlessly if nothing will be
written to it.

**Credits to @uriyage for locating this with his fuzzer testing**

The suggested solution is to only flag the client when it is
specifically unblocked on keys.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-05-25 20:16:20 +03:00
chzhooandGitHub be5f3b2777 Optimize GEORADIUS command performance with pre-allocated buffer (#2116)
**Description**

1. The GEORADIUS command stores searched map points in geoArray, which
   triggers at least two memory allocations: one for the `geoArray`
   creation and another for each `geoPoint` initialization.
2. We can reduce these two memory allocations: 
   - converting `geoArray` from a heap-allocated variable to a
     stack-allocated one
   - embedding `geoPoint` directly within geoArray
  
**Benchmark**

Run the benchmark command five times with the command below, and take
the peak value as the final result.

    for((x=0;x<5;x+=1));
    do
    valkey-cli flushall;
    valkey-benchmark -r $POINTS_SIZE geoadd key 100 70 l__rand_int__;
    valkey-benchmark -P 2 -n 10000000 georadius key 100 70 300 km;
    done

$POINTS_SIZE | QPS before optimization|QPS after optimization|Performance Boost
-- | -- | -- | -- |
1|430015.06|436585.88|1.5%|
4|381766.81|386279.34|1.2%|
16|264592.28|268391.53|1.4%|
64|120035.05|120277.60|0.2%|

CPU: AMD EPYC 9K65 192-Core Processor* 8
OS: 	Ubuntu Server 24.04 LTS 64bit
Memory: 32GB
VM: Tencent cloud  SA9 | SA9.2XLARGE32

---------

Signed-off-by: chzhoo <czawyx@163.com>
2025-05-25 18:00:26 +02:00
0269a21e33 Fix the GNU IFUNC error in alpine (non-glibc) environment (#2133)
Follow up of https://github.com/valkey-io/valkey/pull/2099

Fix
https://github.com/valkey-io/valkey/actions/runs/15221314860/job/42817270745,

https://github.com/valkey-io/valkey/actions/runs/15221314860/job/42817271735

---------

Signed-off-by: Lipeng Zhu <lipeng.zhu@intel.com>
Co-authored-by: Guo Wangyang <wangyang.guo@intel.com>
2025-05-25 17:49:15 +02:00
uriyageandGitHub 91b2dcc233 Fix bad slot used in sharded pubsub unsubscribe (#2137)
This commit fixes two issues in `pubsubUnsubscribeChannel` that could
lead to memory corruption:

1. When calculating the slot for a channel, we were using getKeySlot()
which might use the current_client's slot if available. This is
problematic when a client kills another client (e.g., via CLIENT KILL
command) as the slot won't match the channel's actual slot.

2. The `found` variable was not initialized to `NULL`, causing the
serverAssert to potentially pass incorrectly when the hashtable lookup
failed, leading to memory corruption in subsequent operations.


The fix:
- Calculate the slot directly from the channel name using keyHashSlot()
instead of relying on the current client's slot
- Initialize 'found' to NULL

Added a test case that reproduces the issue by having one client kill
another client that is subscribed to a sharded pubsub channel during a
transaction.


Crash log (After initializing the variable 'found' to null, without
initialization, memory corruption could occur):
```
VALKEY BUG REPORT START: Cut & paste starting from here ===
59707:M 24 May 2025 23:10:40.429 # === ASSERTION FAILED CLIENT CONTEXT ===
59707:M 24 May 2025 23:10:40.429 # client->flags = 108086391057154048
59707:M 24 May 2025 23:10:40.429 # client->conn = fd=11
59707:M 24 May 2025 23:10:40.429 # client->argc = 0
59707:M 24 May 2025 23:10:40.429 # === RECURSIVE ASSERTION FAILED ===
59707:M 24 May 2025 23:10:40.429 # ==> pubsub.c:348 'found' is not true

------ STACK TRACE ------

Backtrace:
0   valkey-server                       0x0000000104974054 _serverAssertWithInfo + 112
1   valkey-server                       0x000000010496c7fc pubsubUnsubscribeChannel + 268
2   valkey-server                       0x000000010496cea0 pubsubUnsubscribeAllChannelsInternal + 216
3   valkey-server                       0x000000010496c2e0 pubsubUnsubscribeShardAllChannels + 76
4   valkey-server                       0x000000010496c1d4 freeClientPubSubData + 60
5   valkey-server                       0x00000001048f3cbc freeClient + 792
6   valkey-server                       0x0000000104900870 clientKillCommand + 356
7   valkey-server                       0x00000001048d1790 call + 428
8   valkey-server                       0x000000010496ef4c execCommand + 872
9   valkey-server                       0x00000001048d1790 call + 428
10  valkey-server                       0x00000001048d3a44 processCommand + 5056
11  valkey-server                       0x00000001048fdc20 processCommandAndResetClient + 64
12  valkey-server                       0x00000001048fdeac processInputBuffer + 276
13  valkey-server                       0x00000001048f2ff0 readQueryFromClient + 148
14  valkey-server                       0x0000000104a182e8 callHandler + 60
15  valkey-server                       0x0000000104a1731c connSocketEventHandler + 488
16  valkey-server                       0x00000001048b5e80 aeProcessEvents + 820
17  valkey-server                       0x00000001048b6598 aeMain + 64
18  valkey-server                       0x00000001048dcecc main + 4084
19  dyld                                0x0000000186b34274 start + 2840
````

---------

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
2025-05-25 16:36:34 +03:00
BinbinandGitHub 15515703da Changes and fixes for the new DELIFEQ command (#2120)
DELIFEQ was added in #1975 but had some issues.

Do some refactoring to reduce the number of lines of code,
write commands need to increment dirty and we can propagate
it as DEL command.

Changes:
1. Replicate as DEL is important for compatibility with replicas
   running an older Valkey version, also it can save the compare
   logic, like other commands like GETDEL.
2. Signal modified key is for WATCH and do invalidate client-side
   caching (client tracking)
3. Keyspace notifications.
4. Dirty++ indicates there are unsaved changes for RDB.
5. Using shared strings for the RESP zero and one replies for the
   code cleanup.
6. Add FAST command flag and remove the ACCESS key specs flag.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-05-25 18:23:58 +08:00
Katie HollyandGitHub e109958fde Mark string2ll_resolver as used (#2130)
Daily test runs that use clang are currently
[failing](https://github.com/valkey-io/valkey/actions/runs/15213976527/job/42794864169#step:4:167)
with
```
util.c:628:51: error: unused function 'string2ll_resolver' [-Werror,-Wunused-function]
  628 | __attribute__((no_sanitize_address)) static int (*string2ll_resolver(void))(const char *, size_t, long long *) {
      |                                                   ^~~~~~~~~~~~~~~~~~
```

Marking `string2ll_resolver` as `used` should solve this.

Signed-off-by: Fusl <fusl@meo.ws>
2025-05-23 13:11:12 -07:00
c8080f6b7d Optimize string2ll with load-time CPU feature check using IFUNC resolver (#2099)
### Problem Statement

As we increasingly incorporate SIMD instructions to optimize
performance-critical functions, our current runtime CPU feature
detection pattern is introducing significant overhead. For example, in
the **string2ll** function, the feature detection wrapper adds
approximately 3.35% performance overhead (shown as Fig 1), which is very
significant especially considering the AVX-optimized implementation
itself is highly effective.

Current pattern:
```c
int string2ll(const char *s, size_t slen, long long *value) {
#if HAVE_X86_SIMD
    if (__builtin_cpu_supports("avx512f") &&
        __builtin_cpu_supports("avx512vl") &&
        __builtin_cpu_supports("avx512bw"))
        return string2llAVX512(s, slen, value);
#endif
    return string2llScalar(s, slen, value);
}
``` 

![image](https://github.com/user-attachments/assets/16508faa-cc7d-4437-9de3-09c58522026e)

Fig 1. Profiling of
[memtier_benchmark-1key-list-10K-elements-lpos-integer](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-1key-list-10K-elements-lpos-integer.yml)

### Proposed Solution

This PR implements the **GNU IFUNC** (indirect function) mechanism,
which resolves the optimal function implementation at program load time
rather than checking CPU features during each function call. This
approach eliminates repeated detection overhead in hot code paths.

#### Benefits

1. **Performance Improvement**: Eliminates feature detection overhead
(~3.35% for string2ll)
2. **One-time Detection**: CPU capabilities are checked once at load
time
3. **Zero Runtime Overhead**: No conditionals in the call path
4.  **Maintainability**: Cleaner code without conditional branches

### Performance Boost

The example test suite we can observe a ~3% performance improvement. 
|Benchmark|Performance Boost|
|-|-|

|[memtier_benchmark-1key-list-10K-elements-lpos-integer](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-1key-list-10K-elements-lpos-integer.yml)|**3%**|

#### Test Env
- OS: CentOS Stream 9
- Platform: Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz
- Server and Client in same socket

#### Valkey-server configuration
```
taskset -c 0 ~/valkey/src/valkey-server ~/tmp_valkey.conf
port 9001
bind * -::*
daemonize no
protected-mode no
save
```

---------

Signed-off-by: Lipeng Zhu <lipeng.zhu@intel.com>
Signed-off-by: Lipeng Zhu <zhu.lipeng@outlook.com>
Co-authored-by: Guo Wangyang <wangyang.guo@intel.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-23 11:45:08 +02:00
Katie HollyandGitHub cb83595bd5 Show more decimal digits for avg chain length field in debug htstats full output (#2118)
There's not really a reason why we're truncating the `avg chain length`
field in the `debug htstats` full output and it just makes debugging a
little harder so let's just show a longer representation of the value.

Signed-off-by: Fusl <fusl@meo.ws>
2025-05-23 03:29:32 +02:00
Viktor SöderqvistandGitHub b2e30a9824 Make Tclx optional in tests (fix Daily failure) (#2123)
Tclx is not available on all platforms. Skip tests that require it if
it's not available.

This also fixes the Daily failures such as this:
https://github.com/valkey-io/valkey/actions/runs/15175084203/job/42673707836#step:8:4677

The diff looks large, but it isn't. The test code is just wapped in a
check and indented.

```
if {[catch {package require Tclx}]} {
    if {$::verbose} {
        puts "Skipping umask test. Package Tclx not available."
    }
} else {
    (old code here)
}
```

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-23 02:58:08 +02:00
Viktor SöderqvistandGitHub 464f9a2890 Fix UBSAN run for hashtable unittest (#2126)
The unit test declared an array of size 1 and used pointers to elements
outside the array. This is fine because the pointers are never
dereferenced, but undefined-sanitizer complains.

Now, instead allocate a huge array to make sure all pointers into it are
valid.

Example failure:


https://github.com/valkey-io/valkey/actions/runs/15175084203/job/42673713108#step:10:123

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-23 02:57:04 +02:00
Viktor Söderqvist 267e35c6f7 CI: Run unit tests in 32-bit build
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-23 02:55:11 +02:00
Viktor Söderqvist c520809e8d Remove an unnecessary check that seems to overflow in 32-bit
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-23 02:55:11 +02:00
Viktor Söderqvist a6622e7d15 Fix object unit test for 32-bit builds
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-23 02:55:11 +02:00
Linus UnnebäckandGitHub e7f9b76016 Add DELIFEQ command (#1975)
This feature introduces a new command: `DELIFEQ key value`.

The command deletes the given key _only if_ its current value is equal
to the provided `value`. It returns:

- 1 if the key was removed (i.e., it existed and matched the value),
- 0 otherwise (key did not exist, or the value did not match).

This command complements `SET key value IFEQ match-value`, added in
#1324.

**The problem/use-case that the feature addresses**

A very common operation when synchronizing distributed locks is to
remove a key, but only if the value matches a specific string. This
pattern is frequently used in distributed locking algorithms like
Redlock. The conditional delete prevents a client from inadvertently
releasing a lock it doesn’t own—e.g., when a timeout or retry causes
overlapping or stale attempts.

Today, this logic is typically implemented using a small Lua script
like:

```lua
if redis.call("get",KEYS[1]) == ARGV[1] then
    return redis.call("del",KEYS[1])
else
    return 0
end
```

However, using scripts adds a layer of complexity and overhead. Having
this logic as a built-in Redis command improves simplicity, reliability,
and performance. It also complements `SET key IFEQ value` very nicely.

Example:

```redis
SET foo test
DELIFEQ foo test   # returns 1, key is deleted

SET foo nope
DELIFEQ foo test   # returns 0, key remains
```

Signed-off-by: Linus Unnebäck <linus@folkdatorn.se>
2025-05-22 13:36:46 +02:00
Vitah LinandGitHub b442ae1baa Optimize WATCH by equalStringObjects early length check (#2107)
Avoid unnecessary `memcmp()` when both objects are sds and the lengths
differ.

I found that this function is called in `watchCommand`, so I verified
the performance improvement of this PR using the WATCH command:

```shell
$ memtier_benchmark \
  --command="WATCH a aa aaa aaaa aaaaa b bb bbb bbbb bbbbb" \
  --test-time 60 \
  -c 1 \
  -t 1 \
. --hide-histogram
```

Before:
```shell
ALL STATS

==================================================================================================
Type         Ops/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec 
--------------------------------------------------------------------------------------------------
Watchs      43053.91         0.02273         0.02300         0.03900         0.06300      4666.98 
Totals      43053.91         0.02273         0.02300         0.03900         0.06300      9333.95 
```


After:
```shell
ALL STATS
==================================================================================================
Type         Ops/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec 
--------------------------------------------------------------------------------------------------
Watchs      43414.55         0.02216         0.02300         0.03900         0.05500      4706.07 
Totals      43414.55         0.02216         0.02300         0.03900         0.05500      9412.14
```

---------

Signed-off-by: vitah <vitahlin@gmail.com>
Signed-off-by: Vitah Lin <vitahlin@gmail.com>
2025-05-22 13:30:18 +02:00
Ran ShidlansikandGitHub ea7311b8f5 fix commandlog argument schema to use pure tokens (#2113)
Should not cause any behavior change, mainly for correctness 

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-05-21 13:39:58 +03:00
kronwerkandGitHub 0f53d75d83 Respect process umask when creating data files (#1725)
Create data files (both AOF and RDB) using file mode as specified by the
umask, which is inherited from the parent process, instead of hard-coded
to 0644. The latter can be inconvenient in some cases where Valkey runs
as a user and the files need to be accessed by a different user.

By passing file mode 0666 to `open()`, which is masked by the umask, the
user can affect the mode of the created files, which will be `(0666 &
~umask)`. If the umask is 0022 (the default in most systems), the files
will be created with mode 0644. If the user wants 0664, they can set
umask to 0002 in the parent (e.g. in the shell) before starting Valkey.

---------

Signed-off-by: kronwerk <kronwerk@users.noreply.github.com>
2025-05-20 09:36:35 +02:00
Sarthak AggarwalandGitHub 9ce60aff1f Optimize saving cluster config file using sdscatfmt (#2088)
Replace `sdscatprintf` with the faster variant `sdscatfmt` in the code
for generating the cluster config file. This optimization also affects
CLUSTER NODES which uses the same code.

When the cluster topology changes, such as during failovers, the cluster
config file is written and at the same time the nodes may experience
extra load due to failovers, redirects, full sync, etc.

In the profiles I am capturing for large scale clusters, I am observing
decent compute cycles being utilized by `sdscatprintf`.

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-05-20 09:31:39 +02:00
Katie HollyandGitHub 8509a06342 Allow mixing quoted and unquoted inline args (#2098)
Fix an inconsistency of `sdssplitargs` where `foo""` is allowed but
`""foo` is rejected.

This patch loosens up the parsing to allow mixing quoted and unquoted
text within the same argument, similar to how shell syntax works,
allowing the use of `"foo"bar'baz'zzz` to produce a single argument of
`"foobarbazzzz"`.

This is a backward compatible change. It affects the inline protocol,
the arguments to valkey-cli and the parsing of config files.

---------

Signed-off-by: Fusl <fusl@meo.ws>
2025-05-19 17:32:44 +02:00
Katie HollyandGitHub 9faec01832 Further improve performance of sdssplitargs (#2096)
This reduces the amount of realloc calls we do for multi-argument
inline commands by pre-allocating enough for 8 arguments and doubling
the capacity when growing instead of increasing the size linearly.

Signed-off-by: Fusl <fusl@meo.ws>
2025-05-19 15:23:36 +02:00
Viktor SöderqvistandGitHub 4329d847bd Fix random element in skewed sparse hash table (#2085)
In cases when half of the elements in a hashtable have been deleted in
scan order, i.e. using scan and deleting the returned elements, the
random selection algorithm becomes very slow. Also, if a random element
is deleted repeatedly, this can make the hash table to become skewed and
make random selection very slow. Behavior without this fix:

* The fair random algorithm samples some elements by picking a random
  cursor and then scanning from that point until it finds a number of
  elements and then picking a random one of them.
* Now, assume there is an empty sequence of buckets (in scan order).
* When we pick a random cursor and it points into this empty sequence,
  we continue scanning until we find some elements and then pick one of
  these. Therefore, the buckets just after an empty sequence are more
  likely to be picked than other buckets.
* If we repeat SPOP many times, this area just after the empty section
  will be emptied too and the empty section grows, the problem gets
  worse and worse.

Bucket distribution (x = non-empty bucket, . = empty bucket)

    x..xx.x.xx.x.........x.xx..x.x..x.xx..x.x
                  ^      ^ ^^  ^ ^
                  |      | ||  | |
                random   elements
                cursor    sampled

This PR changes the sampling to pick a new random index to scan for each
new bucket chain that has been sampled. This is more similar to how the
fair random works in dict.c.

Additionally, this PR includes two small optimizations/tuning:

1. The change allows us to lower the sample sizes for fair random
   element selection. There are unit tests to verify the fairness of the
   algorithm.
2. Additionally, the sampling size is decreased even more in very sparse
   tables to prevent it from sampling empty buckets many times which
   makes it slow.

One existing unit test is changed from a stack-allocated to a
heap-allocated array, because it was observed to use too much stack
memory when running locally with the `--large-memory` option.

Fixes #2019.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-19 11:05:56 +02:00
6743f7bbd1 Fix minor memory leak in valkey-cli when command table hint fails due to NOAUTH (#2091)
Valkey-cli sends some commands when it connects, e.g. COMMAND and
COMMAND DOCS to init the command table hint. These return a NOAUTH
error when the client is unathenticated and there is a memory leak.

Close #2064.

Signed-off-by: fukua95 <fukua95@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-05-16 12:00:52 +08:00
b951a9907a Make test cases independent of each other in the moduleapi/scan suite (#2082)
I try to running moduleapi test and this command may test failed:

    ./runtest-moduleapi --single "unit/moduleapi/scan" --only "Module scan hash dict"

I found that the test `Module scan hash dict` and `Module scan hash
listpack` use the same key `hh`. They affected each other.

This PR adjust these tests to make them independent of each other.

---------

Signed-off-by: vitah <vitahlin@gmail.com>
Signed-off-by: Vitah Lin <vitahlin@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-15 11:20:08 +02:00
Vadym KhoptynetsandGitHub 22670a32ae Fix string embedding decision with keys (#2087)
This commit fixes the size calculation for embedded strings with keys in
createStringObjectWithKeyAndExpire(). The previous implementation used a
fixed size of 3 bytes for the key's header, which was incorrect. The new
implementation properly calculates the required size based on the key's
length and SDS type.

Added a unit test (test_embedded_string_with_key) to verify that:
1. Short values with long keys are properly encoded as embedded strings
2. Values that would cause the total size to exceed 64 bytes are encoded
as raw

Signed-off-by: Vadym Khoptynets <vadymkh@amazon.com>
2025-05-15 10:13:16 +03:00
Sarthak AggarwalandGitHub ed39add3d7 Redundant Calculation of mstime() (#2080)
This changes removes the redundant calls to mstime() within a same
function. I believe this value should not change over the loop and can
be reused.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-05-14 08:41:23 -07:00
muelstefamznandGitHub 578b8890da Reverting the panic() introduced in #2027 (#2081)
Removing the `panic()` statement introduced previously.

There is existing code that depends on the previous behaviour. 

See https://github.com/valkey-io/valkey/pull/2027.

Signed-off-by: Stefan Mueller <muelstef@amazon.com>
2025-05-13 14:23:26 -07:00
Viktor SöderqvistandGitHub c32e95f6f6 Delete dead code in dict (#2014)
After transitioning much usage of dict to the new hashtable, parts of
the dict implementation have become dead code. The following is deleted:

* Dict entry without value
* Dict entry with embedded key
* Pointer bit tricks and other complexity for different kinds of
dictEntry
* Dict two-phase unlink
* Dict type flag to disable incremental rehashing
* Dict stats
* Defrag callbacks defragEntryStartCb and defragEntryFinishCb

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-12 14:34:41 -07:00
3b0c9d437c CI rename build-macos to build-old-version-macos (#2039)
This is a follow-up adjustment based on the previous PR:
https://github.com/valkey-io/valkey/pull/2000
@hwware @madolson 
Apologies again for the extra review work this may have caused — I
really appreciate your time and feedback.

---------

Signed-off-by: vitah <vitahlin@gmail.com>
Signed-off-by: Vitah Lin <vitahlin@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-05-12 14:19:58 -07:00
muelstefamznandGitHub 212658763e Additional assertions for listpack (#2027)
Adds a few asserts to increase the chance of detecting corrupted
listpack data.

It replaces a questionable code path that purposefully returns invalid
data with a `panic()` and adds `assert()` statements to verify that
`LP_EOF` is only detected at the end of the listpack.

Listpack data is expected to be correctly encoded. These asserts should
only trigger if the listpack data was corrupted in some way, in which
case it is better to assert than to return invalid data to the client.

Signed-off-by: Stefan Mueller <muelstef@amazon.com>
2025-05-12 14:16:45 -07:00
Madelyn OlsonandGitHub 98ab1ecb26 Make sure that the same point reports for geo comparisons (#2063)
Resolves https://github.com/valkey-io/valkey/issues/2051

We have an optimization where if the longitudes of two points are the
same in a geospatial search, we can do a much simpler distance across
the radius of the earth. This optimization was not being called for ARM
with optimizations, since it was always accumulating a small amount of
rounding error. This was also causing slightly divergent behavior
between x86 and ARM.

The goal of this PR is to compare against a very small value instead of
zero to avoid most of these rounding errors. There is a very simple test
that worked on x86 but did not work on ARM.

Also deleted an extra constant.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-05-12 14:05:39 -07:00
Viktor SöderqvistandGitHub 230825da17 valkey-benchmark: Arbitrary command sequence (#2057)
The valkey-benchmark command line syntax is extended with the special
token ";" to allow multiple commands in a sequence. The command sequence
is sent together as a pipeline.

    valkey-benchmark set foo bar ';' get foo
    valkey-benchmark multi ';' set foo bar ';' get foo ';' exec

Each command can be prefixed with a number, indicating how many times to
repeat the command. For example, to execute one set and four get
commands, simulating 20/80 traffic:

    valkey-benchmark set foo bar ';' 4 get foo

An optional "--" argument is allowed to separate the options from the
command sequence. This is allowed just for clarity.

    valkey-benchmark -n 1000 -- set foo bar ';' 4 get foo

A `__data__` placeholder is supported in the command arguments. This is
expanded to a string of the data length given using the `-d` option.
This example simulates SET and GET with a value of 512:

valkey-benchmark -r 100000 -d 512 -- set key:__rand_int__ __data__ ';'
get key:__rand_int__

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-12 21:34:43 +02:00
skyfireleeandGitHub 9110db1129 Fix variable scope confusion (#2071)
Fix variable `len` scope confusion `declaration shadows a local variable`.
The variable len is defined at the very beginning, this change also
saves one line of code and makes the code more readable.

Signed-off-by: artikell <739609084@qq.com>
2025-05-12 11:31:21 +08:00
3711fed728 Optimize BITCOUNT using ARM NEON SIMD (#1867)
Replace scalar loop with ARM NEON intrinsics for vectorized processing
on ARM.

Results:

**Throughout** 
| Payload Size | Scalar Throughput (k req/s) | SIMD Throughput (k req/s)
| Improvement (%) |

|--------------|-----------------------------|----------------------------|-----------------|
| 16B | 249.69 | 249.69 | 0.00% |
| 256B | 249.63 | 249.69 | +0.02% |
| 4KB | 199.72 | 249.63 | +25.00% |
| 64KB | 44.33 | 166.42 | +275.43% |
| 1MB | 3.30 | 26.59 | +705.74% |
| 10MB | 0.33 | 3.32 | +900.04% |


**Average Latency**


| Payload Size | Scalar Avg Latency (ms) | SIMD Avg Latency (ms) |
|--------------|--------------------------|-------------------------|
| 16B          | 0.374                    | 0.375                   |
| 256B         | 0.381                    | 0.376                   |
| 4KB          | 0.489                    | 0.389                   |
| 64KB         | 2.241                    | 0.575                   |
| 1MB          | 30.169                   | 3.649                   |
| 10MB         | 287.228                  | 29.220                  |


**P99 Latency**
| Payload Size | Scalar p99 Latency (ms) | SIMD p99 Latency (ms) |
|--------------|--------------------------|-------------------------|
| 16B          | 0.511                    | 0.511                   |
| 256B         | 0.519                    | 0.511                   |
| 4KB          | 0.639                    | 0.535                   |
| 64KB         | 2.439                    | 0.727                   |
| 1MB          | 32.303                   | 3.959                   |
| 10MB         | 314.623                  | 31.615                  |



Tested on AWS Graviton2.
To isolate CPU-bound improvements, the same key was used, reducing the
likelihood of memory stalls for small payloads.

Fixes: #1864

---------

Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-11 12:12:30 +02:00
xbaselandGitHub 02c910728b valkey-cli: remove RESP version assertion from getDatabases() (#2068)
The assertion fails in cluster manager mode, where the config doesn't
apply. Fixes test failures when RESP3 is enforced and the cli runs in
cluster mode.

fixes https://github.com/valkey-io/valkey/issues/2053

Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>
2025-05-09 13:25:28 -07:00
Wen HuiandGitHub efd6e7e04a move statement in if condition for zremrangeGenericCommand function (#2060)
It is similar to PR https://github.com/valkey-io/valkey/pull/2001, this
PR goal is to make logic clearer

Signed-off-by: hwware <wen.hui.ware@gmail.com>
2025-05-09 13:41:58 -04:00
zhenwei piandGitHub 4577131517 Support MPTCP for valkey-cli and benchmark (#2067)
Add '--mptcp' for both valkey-cli and valkey-benchmark to enable MPTCP.

Signed-off-by: zhenwei pi <zhenwei.pi@linux.dev>
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2025-05-09 14:53:10 +02:00
zhenwei piandGitHub c9bdc302c0 Support RDMA for valkey-cli and benchmark (#2059)
Add '--rdma' for both valkey-cli and valkey-benchmark to enable RDMA.

Valkey has already replaced dependency `hiredis` with `libvalkey`
(#2032), and libvalkey also supports RDMA.

---------

Signed-off-by: zhenwei pi <zhenwei.pi@linux.dev>
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2025-05-09 09:05:15 +02:00
c38dce452d Optimize findBucket performance using SIMD (#2030)
### Description

This PR introduces an optimization for the `findBucket` function
utilizing SSE2 instructions. The `findBucket` function is widely be used
in hashtable data structure, particularly in **SET** and **GET** related
commands, but also for looking up a command by its name and many other
things.

The core optimization happens with `_mm_cmpeq_epi8()`, which performs 16
bytes (with 7 bytes being valid) comparisons in parallel. The
`_mm_movemask_epi8()` function then converts this vector of byte
comparisons into a bit mask where each bit represents whether a
comparison was successful.

This implementation demonstrates sophisticated use of SIMD instructions
to accelerate hash table lookups by performing multiple hash comparisons
in parallel.

### Performance Boost

The corresponding **GET** and **SET** commands can gain up to **~2% -
~6%** performance improvement especially with pipeline enabled. Below
test scenarios showed they are benefit for this optimization.

|Benchmark|Performance Boost|
|-|-|

|[memtier_benchmark-10Mkeys-string-get-10B-pipeline-100-nokeyprefix](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-10Mkeys-string-get-10B-pipeline-100-nokeyprefix.yml)
|**4%**|

|[memtier_benchmark-1Mkeys-string-get-10B-pipeline-50](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-1Mkeys-string-get-10B-pipeline-50.yml)
|**4%**|

|[memtier_benchmark-1Mkeys-string-get-10B-pipeline-100](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-1Mkeys-string-get-10B-pipeline-100.yml)
|**4%**|

|[memtier_benchmark-1Mkeys-string-get-10B-pipeline-100-nokeyprefix](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-1Mkeys-string-get-10B-pipeline-100-nokeyprefix.yml)
|**2%**|

|[memtier_benchmark-1Mkeys-string-get-10B-pipeline-500](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-1Mkeys-string-get-10B-pipeline-500.yml)
|**6%**|

|[memtier_benchmark-1Mkeys-load-string-with-10B-values-pipeline-500](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-1Mkeys-load-string-with-10B-values-pipeline-500.yml)
|**4%**|

|[memtier_benchmark-1Mkeys-load-string-with-10B-values-pipeline-100](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-1Mkeys-load-string-with-10B-values-pipeline-100.yml)
|**3%**|

|[memtier_benchmark-1Mkeys-load-string-with-10B-values-pipeline-100-nokeyprefix](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-1Mkeys-load-string-with-10B-values-pipeline-100-nokeyprefix.yml)
|**3%**|


#### Test Env
- OS: CentOS Stream 9
- Platform: Intel Xeon 6980P
- Server and Client in same socket

#### Valkey-server configuration
```
taskset -c 0 ~/valkey/src/valkey-server ~/tmp_valkey.conf
port 9001
bind * -::*
daemonize no
protected-mode no
save
```

---------

Signed-off-by: Lipeng Zhu <lipeng.zhu@intel.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Wangyang Guo <wangyang.guo@intel.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-09 08:54:54 +02:00
Viktor SöderqvistandGitHub d9c522b7d8 Prevent tests setting singledb from breaking other tests (#2056)
Some Tcl test suites set the global variable `::singledb`. If they don't
restore the value afterwards, they can affect other test suites, which
leads to problems that are hard to track down.

This change introduces a test tag 'singledb' that has the same effect,
but it is local to the scope of the tag (a tags block, start_server or a
single test case). The tests should use the tag instead of setting the
global.

Additional change: Use ECHO instead of PING in `valkey_deferring_client`
and `valkey_client` procs when singledb is set, to avoid failing when
the server is loading. (SELECT and ECHO are allowed while loading, but
not PING.)

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-08 19:46:37 +02:00
George PadronandGitHub 5916a8b8aa Fixed minor grammatical error (#2061)
Fixed a minor grammatical mistake in the comments of this file.

Signed-off-by: George Padron <georgenpadron@gmail.com>
2025-05-08 19:13:27 +03:00
Sarthak AggarwalandGitHub a0a2fd75da Disallowing Client Reply is On / Off / Skip when Client is Multi (#1966)
Return an error when we try to perform `CLIENT REPLY` command under
`MULTI/EXEC`.

Without this change, if `CLIENT REPLY` is used in a transaction, it
causes `EXEC` to return broken RESP protocol, with a multi-bulk length
mismatching the actual number of multi-bulk elements.

Resolves #1268

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-05-07 10:57:47 -07:00
Björn SvenssonandGitHub 45f860968f Replace dependency hiredis with libvalkey (#2032)
This PR removes the dependency `deps/hiredis` and replaces it with
`deps/libvalkey`.
`libvalkey` is a forked `hiredis` with `hiredis-cluster` included.

Makefiles/CMake, types, include paths, comments and used API function
names are updated to match the new library.

Since both hiredis and Valkey use an `sds` type, we previously needed to
patch the sds type provided by hiredis, and have a `sdscompat.h` file to
map `sds` calls to the hiredis variant. This is no longer needed since
we can build a static libvalkey using `sds` provided by Valkey (same
with `dict`). Now we use Valkey's `sds` type in `valkey-cli` and
`valkey-benchmark` and that's why they now also require `fpconv` to
build.

The files from [libvalkey
0.1.0](https://github.com/valkey-io/libvalkey/releases/tag/0.1.0) is
added as a `git subtree` similar to how hiredis was included.

---------

Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
2025-05-07 11:49:27 +02:00
Madelyn OlsonandGitHub 73b4ec70ab Use the correct port for migration test (#2050)
One of the new tests that was added uses `CONFIG GET PORT`, which isn't
right one for TLS.

Also removed some other use of the helper which aren't actually used.

Introduced as part of https://github.com/valkey-io/valkey/pull/1671.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-05-06 13:42:38 -07:00
xbaselandGitHub 46c9297b0d Restore omitted singledb config in tests (#2052)
Re-adds a statement to restore the `singledb` config that was
accidentally removed in PR #1671.

Fixes https://github.com/valkey-io/valkey/issues/2049

Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>
2025-05-06 22:38:20 +02:00
f36509ef05 valkey-cli: Count databases correctly in --cluster subcommands (#2046)
Fixes a valkey-cli check that caused it to miscount the number of
databases and led to resharding failures. This change makes the
`--cluster` subcommands imply cluster mode as if the `-c` flag had been
given.

---------

Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-05-06 11:21:54 +02:00
aradz44andGitHub e3e019ead9 Return only the relevant bit in sdsGetAuxBit() (#2048)
Previously, the function would return 1 if any auxiliary bit was set,
which could cause problems when multiple auxiliary bits were present.
This pull request modifies the function to return 1 or 0 based solely on
the specific auxiliary bit being checked, rather than considering all
auxiliary bits.

Signed-off-by: Arad Zilberstein <aradz@amazon.com>
2025-05-06 10:21:41 +02:00
Vitah LinandGitHub 2b04c8bfee Remove duplicate macro definitions of HASH_SET_* (#2024)
This PR removes duplicate macro definitions of HASH_SET_TAKE_FIELD,
HASH_SET_TAKE_VALUE, and HASH_SET_COPY that were already defined in
`server.h`. Keeping a single definition helps ensure consistency and
avoids redundancy across files.

https://github.com/valkey-io/valkey/blob/18792ce254d4a3922983412ee671fad5ae88b563/src/server.h#L3265-L3268

Signed-off-by: vitah <vitahlin@gmail.com>
2025-05-05 14:43:41 +02:00
Hiranmoy Das ChowdhuryandGitHub 55ebbfeb2b Fix VALKEYCLI_AUTH env variable fallback to REDISCLI_AUTH mistake (#2038)
Fix flipped-logic mistake introduced in #1995.

Signed-off-by: Hiranmoy Das Chowdhury <hiranmoy.das70@gmail.com>
2025-05-05 11:20:37 +02:00
Viktor SöderqvistandGitHub def682018e Compile with -Wundef (#2025)
This catches spelling errors in macros, like `#if MACCRO`.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-05 10:41:17 +02:00
Hiranmoy Das ChowdhuryandGitHub a976505577 Environment variable REDISCLI_AUTH to VALKEYCLI_AUTH conversion (#1995)
Signed-off-by: Hiranmoy Das Chowdhury <hiranmoy.das70@gmail.com>
2025-05-03 21:22:38 -07:00
1148009498 Add multi-database support to cluster mode (#1671)
## cluster: add multi-database support in cluster mode

Add multi-database support in cluster mode to align with standalone mode
and facilitate migration. Previously, cluster mode was restricted to a
single database (DB0). This change allows multiple databases while
preserving the existing slot-based key distribution.


### Key Features:
- Database-Agnostic Hashing. The hashing algorithm is unchanged.
  Identical keys always map to the same slot across all databases,
  ensuring consistent key distribution and compatibility with
  existing single-database setups.
- Multi-DB commands support. SELECT, MOVE, and COPY are now supported in
  cluster mode.
- Fully backward compatible with no API changes.
- SWAPDB is not supported in cluster mode. It is unsafe due to
inconsistency risks.

### Command-Level Changes:
- SELECT / MOVE / COPY are now supported in cluster mode.
- MOVE / COPY (with db) are rejected (TRYAGAIN error) during slot
migration to prevent multi-DB inconsistencies.
- SWAPDB will return an error if used when cluster mode is enabled.
- GETKEYSINSLOT, COUNTKEYSINSLOT and MIGRATE will operate in the context
of the selected database.
This means, for example, that migrating keys in a slot will require
iterating and repeating across all databases.

### Slot Migration Process:
- Multi-DB support in cluster mode affects slot migration. Operators
should now iterate over all the configured databases.
 
### Transaction Handling (MULTI/EXEC):
- getNodeByQuery key lookup behavior changed:
  - No key lookups when queuing commands in MULTI, only cross-slot
    validation.
  - Key lookups happen at EXEC time.
  - SELECT inside MULTI/EXEC is now checked, ensuring key validation
    uses the selected DB at lookup.

### Valkey-cli:
- valkey-cli has been updated to support resharding across all
databases.

### Configuration:
- Introduce new configuration `cluster-databases`.
The new configuration controls the maximal number of databases in
cluster mode.

Implements  https://github.com/valkey-io/valkey/issues/1319

---------

Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>
Signed-off-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
Co-authored-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2025-05-03 21:12:02 -07:00
Wen HuiandGitHub ed8c039e17 Remove redundancy if condition statement (#2034)
Before the if condition, the sdslen(value) can not be zero or negative,
thus this if condition is redundancy , just remove it.

Signed-off-by: hwware <wen.hui.ware@gmail.com>
2025-05-02 09:34:31 -04:00
15f2bfd2e8 Properly escape double quotes and backslash in sdscatrepr (#2036)
Fixes https://github.com/valkey-io/valkey/issues/2035, a bug introduced
in https://github.com/valkey-io/valkey/pull/1342

---------

Signed-off-by: Fusl <fusl@meo.ws>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-05-02 10:34:25 +02:00
25242a530a Hyperloglog ARM NEON SIMD optimization (#1859)
Add ARM NEON optimization for HyperLogLog

- Implement two NEON optmized functions for converting between raw and
  dense representations in HyperLogLog:
  1. hllMergeDenseNEON
  2. hllDenseCompressNEON
  These functions process 16 registers in each iteration.

- Utilize existing SIMD test in hyperloglog.tcl (previously added for
  AVX2 optimization) to validate NEON implementation

Test:
``` valkey-benchmark -n 1000000 --dbnum  9  -p 21111 PFMERGE z hll1{t} hll2{t}```

```
+-------------------+-----------+-----------+---------------+
|      Metric       |  Before   |   After   | Improvement % |
+-------------------+-----------+-----------+---------------+
| Throughput (k rps)|    7.42   |   76.98   |    937.47%    |
+-------------------+-----------+-----------+---------------+
| Latency (msec)    |           |           |               |
|   avg             |   6.686   |   0.595   |     91.10%    |
|   min             |   0.520   |   0.152   |     70.77%    |
|   p50             |   7.799   |   0.599   |     92.32%    |
|   p95             |   8.039   |   0.767   |     90.46%    |
|   p99             |   8.111   |   0.807   |     90.05%    |
|   max             |   9.263   |   1.463   |     84.21%    |
+-------------------+-----------+-----------+---------------+
```

Hardware:
```
CPU: Graviton 3
Architecture:           aarch64
  CPU op-mode(s):       32-bit, 64-bit
  Byte Order:           Little Endian
CPU(s):                 64
  On-line CPU(s) list:  0-63
NUMA:
  NUMA node(s):         1
  NUMA node0 CPU(s):    0-63
Memory: 256 GB
```

Command stats:
Before:
```

cmdstat_pfmerge:calls=1000002,usec=126327984,**usec_per_call=126.33**,rejected_calls=0,failed_calls=0

```
After:
```

cmdstat_pfmerge:calls=1000002,usec=8588205,**usec_per_call=8.59**,rejected_calls=0,failed_calls=0
```
Improved by **~14.7x.**




Functional testing command:
```
./runtest --single unit/hyperloglog --only "PFMERGE results with simd"
--loops 10000 --fastfail
```
The SIMD test randomizes input and comapres scalar vs simd results.

---------

Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-05-02 00:23:58 +02:00
Harkrishn PatroandGitHub 6a6a8a784e Accept limited number of new cluster link connections per cycle (#2009)
For large cluster accepting too many new cluster link connections in one
cycle can cause increase in temporary memory usage as well as causes
longer to accept all the connections. With this change, we utilize the
same configuration introduced for clients with cluster link connections
i.e. `10` for non TLS cluster and `1` for TLS cluster.

### Unstable
```
# Memory
used_memory:13495144
used_memory_human:12.87M
used_memory_rss:309686272
used_memory_rss_human:295.34M
used_memory_peak:110714488
used_memory_peak_human:105.59M
```

### With 1 accept per call  for TLS cluster
```
# Memory
used_memory:13495472
used_memory_human:12.87M
used_memory_rss:256172032
used_memory_rss_human:244.30M
used_memory_peak:59974584
used_memory_peak_human:57.20M
```

Summary: Peak memory is 46% lower for the best case observed during test
experiments.

### Unstable
```
time ./valkey-cli --tls --cacert ca.crt --cert valkey-server.crt --key valkey-server.key --json info memory
# Memory
used_memory:45572952
used_memory_human:43.46M
...

real    0m9.466s
user    0m0.014s
sys    0m0.002s
```

### With 1 accept per call for TLS cluster
```
time ./valkey-cli --tls --cacert ca.crt --cert valkey-server.crt --key valkey-server.key --json info memory
# Memory
used_memory:10964872
used_memory_human:10.46M
...

real    0m4.005s
user    0m0.011s
sys    0m0.005s
```

Summary: Time to serve user traffic is 57% lower for the best case
observed during test experiments.

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-04-29 12:34:55 -07:00
e3c9a3e47f Optimize string2ll performance using AVX512 (#1944)
### Description

This pull request aims to optimize the `string2ll` function using AVX512
instructions to enhance performance for string-to-integer conversions.
The existing scalar version of the function was identified as a
bottleneck in certain scenarios, and this optimization seeks to address
that.

### Changes

- Introduced string2llAVX512 function in util.c to utilize AVX512 SIMD
instructions.
- Updated config.h to define ATTRIBUTE_TARGET_AVX512 for AVX512 support.
- Replaced calls to `lpStringToInt64` with `string2ll` for consistency
and performance improvement.
- Removed redundant code in listpack.c and optimized integer encoding logic.

### Performance Boost

The corresponding list commands can gain up to **~19%** performance
improvement, the string length is 4 bytes. The longer the string, the
greater the performance boost.

### Ref

1. http://0x80.pl/notesen/2018-04-19-simd-parsing-int-sequences.html#toc-entry-1
2. https://lemire.me/blog/2023/09/22/parsing-integers-quickly-with-avx-512/

---------

Signed-off-by: Lipeng Zhu <lipeng.zhu@intel.com>
Signed-off-by: Lipeng Zhu <zhu.lipeng@outlook.com>
Co-authored-by: Wangyang Guo <wangyang.guo@intel.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-04-29 17:16:27 +02:00
18792ce254 CI build-macos version from macOS 14 to macOS 15 (#2000)
macOS 15 has been officially released for some time, our current
`build-macos` workflow is still running on macOS 14.
This PR upgrades the macOS version used in our CI to macOS 15 to keep up
with the latest OS version and ensure long-term compatibility.

The `build-macos` CI workflow has successfully run on macOS 15:
https://github.com/vitahlin/valkey/actions/runs/14640672882

---------

Signed-off-by: vitah <vitahlin@gmail.com>
Signed-off-by: Vitah Lin <vitahlin@gmail.com>
Co-authored-by: Wen Hui <wen.hui.ware@gmail.com>
2025-04-29 09:52:47 -04:00
Wen HuiandGitHub 824a0fa847 Minor change for sadd command (#2001)
Update the server.dirty only when it is necessary

Signed-off-by: hwware <wen.hui.ware@gmail.com>
2025-04-28 13:57:24 -04:00
Viktor SöderqvistandGitHub 89fcb63581 valkey-benchmark: Add test for MGET (#2015)
There is already a test for MSET with 10 keys per batch. This change
adds one for MGET.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-04-28 19:53:32 +02:00
zhenwei piandGitHub 13dda735a7 Introduce MPTCP for replication (#1961)
Allow replicas to use MPTCP in the outgoing replication connection.

A new yes/no config is introduced `repl-mptcp`, default `no`.

For MPTCP to be used in replication, the primary needs to be configured
with `mptcp yes` and the replica with `repl-mptcp yes`. Otherwise, the
connection falls back to regular TCP.

Follow-up of #1811.

---------

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2025-04-27 16:46:31 +02:00
e4768d6aaf Convert pubsub dicts to hashtables (#2007)
Two dicts are converted to hashtables:

1. On each client, the set of channels/patterns/shard-channels the
   client is subscribed to
2. On each channel or pattern, the set of clients subscribed to it.

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Rain Valentine <rsg000@gmail.com>
2025-04-27 09:38:59 +02:00
Wen HuiandGitHub 6dfe2f3094 Correct rpushx description (#2002)
Since Redis 4.0, RPUSHX can accept multiple elements as argument, but
the document and json file are not updated.
This PR updates it.

Signed-off-by: hwware <wen.hui.ware@gmail.com>
2025-04-26 10:43:17 +02:00
skyfireleeandGitHub 0a690bd6e9 Clean up unnecessary use of strlen (#1976)
Clean up unnecessary use of strlen, and reduce string2ull useless length calculations.

---------

Signed-off-by: artikell <739609084@qq.com>
2025-04-26 10:40:09 +02:00
3dd24e1b11 Refactor syncWithPrimary and replicaTryPartialResynchronization (#1476)
In continuation to https://github.com/valkey-io/valkey/pull/945

**syncWithPrimary:**
- Refactored all error handling to function `syncWithPrimaryHandleError`
- Refactored repl_state state machine from if-else format to switch-case
format
- Besides changing the repl_state, all state machine logic moved to
helper functions

**replicaTryPartialResynchronization:**
This function was performing two different jobs based on the value of
the read_reply argument -
- read_reply == 0: Sends the PSYNC command to the primary server.
- read_reply == 1: Reads and processes the reply to the PSYNC command.

This change simplifies the logic by clearly separating the writing and
reading stages of the PSYNC process.

Signed-off-by: Nitai Caro <caronita@amazon.com>
Co-authored-by: Nitai Caro <caronita@amazon.com>
2025-04-25 15:59:37 -07:00
a56de8d79a Add output buffer limiting for unauthenticated clients (#2006)
This commit introduces a mechanism to track client authentication state
with a new `ever_authenticated` flag. It refactors client authentication
handling by adding a `clientSetUser` function that properly sets both
the `authenticated` and `ever_authenticated` flags.

The implementation limits output buffer size for clients that have never
been authenticated.

Added tests to verify the output buffer limiting behavior for
unauthenticated clients.

---------

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: uriyage <78144248+uriyage@users.noreply.github.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-04-25 09:29:32 +03:00
71a4873fa8 Add --hotkeys-count option for Valkey-cli (#1933)
Now we only have --hotkeys option, and the default output value is 16.
This PR add one more option for hotkeys as "--hotkeys-count", users have
chance to specific how many output items they want.

Example 1:

./src/valkey-cli --hotkeys-count 10

[00.00%] Hot key '"memtier-3308"' found so far with counter 5
[00.00%] Hot key '"memtier-23516"' found so far with counter 5
[00.00%] Hot key '"memtier-16678"' found so far with counter 5
[00.00%] Hot key '"memtier-25001"' found so far with counter 5
[00.00%] Hot key '"memtier-63829"' found so far with counter 5
[00.00%] Hot key '"memtier-21281"' found so far with counter 5
[00.00%] Hot key '"memtier-81379"' found so far with counter 5
[00.00%] Hot key '"memtier-30335"' found so far with counter 5
[00.00%] Hot key '"memtier-87521"' found so far with counter 5
[00.00%] Hot key '"memtier-42166"' found so far with counter 5

-------- summary -------

Sampled 100000 keys in the keyspace!
hot key found with counter: 5   keyname: "memtier-3308"
hot key found with counter: 5   keyname: "memtier-23516"
hot key found with counter: 5   keyname: "memtier-16678"
hot key found with counter: 5   keyname: "memtier-25001"
hot key found with counter: 5   keyname: "memtier-63829"
hot key found with counter: 5   keyname: "memtier-21281"
hot key found with counter: 5   keyname: "memtier-81379"
hot key found with counter: 5   keyname: "memtier-30335"
hot key found with counter: 5   keyname: "memtier-87521"
hot key found with counter: 5   keyname: "memtier-42166"

Example 2:

./src/valkey-cli --hotkeys-count 20

[00.00%] Hot key '"memtier-3308"' found so far with counter 5
[00.00%] Hot key '"memtier-23516"' found so far with counter 5
[00.00%] Hot key '"memtier-16678"' found so far with counter 5
[00.00%] Hot key '"memtier-25001"' found so far with counter 5
[00.00%] Hot key '"memtier-63829"' found so far with counter 5
[00.00%] Hot key '"memtier-21281"' found so far with counter 5
[00.00%] Hot key '"memtier-81379"' found so far with counter 5
[00.00%] Hot key '"memtier-30335"' found so far with counter 5
[00.00%] Hot key '"memtier-87521"' found so far with counter 5
[00.00%] Hot key '"memtier-42166"' found so far with counter 5
[00.00%] Hot key '"memtier-6370"' found so far with counter 5
[00.01%] Hot key '"memtier-37381"' found so far with counter 5
[00.01%] Hot key '"memtier-95045"' found so far with counter 5
[00.01%] Hot key '"memtier-14027"' found so far with counter 5
[00.01%] Hot key '"memtier-60370"' found so far with counter 5
[00.01%] Hot key '"memtier-98622"' found so far with counter 5
[00.01%] Hot key '"memtier-18642"' found so far with counter 5
[00.01%] Hot key '"memtier-39260"' found so far with counter 5
[00.01%] Hot key '"memtier-64371"' found so far with counter 5
[00.01%] Hot key '"memtier-27099"' found so far with counter 5

-------- summary -------

Sampled 100000 keys in the keyspace!
hot key found with counter: 5   keyname: "memtier-3308"
hot key found with counter: 5   keyname: "memtier-23516"
hot key found with counter: 5   keyname: "memtier-16678"
hot key found with counter: 5   keyname: "memtier-25001"
hot key found with counter: 5   keyname: "memtier-63829"
hot key found with counter: 5   keyname: "memtier-21281"
hot key found with counter: 5   keyname: "memtier-81379"
hot key found with counter: 5   keyname: "memtier-30335"
hot key found with counter: 5   keyname: "memtier-87521"
hot key found with counter: 5   keyname: "memtier-42166"
hot key found with counter: 5   keyname: "memtier-6370"
hot key found with counter: 5   keyname: "memtier-37381"
hot key found with counter: 5   keyname: "memtier-95045"
hot key found with counter: 5   keyname: "memtier-14027"
hot key found with counter: 5   keyname: "memtier-60370"
hot key found with counter: 5   keyname: "memtier-98622"
hot key found with counter: 5   keyname: "memtier-18642"
hot key found with counter: 5   keyname: "memtier-39260"
hot key found with counter: 5   keyname: "memtier-64371"
hot key found with counter: 5   keyname: "memtier-27099"

---------

Signed-off-by: hwware <wen.hui.ware@gmail.com>
Signed-off-by: Wen Hui <wen.hui.ware@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-04-24 16:24:41 -04:00
Yair GottdenkerandGitHub b2a54560d8 blocking client followup fix related to log_req_res (#1989)
See
https://github.com/valkey-io/valkey/pull/1819#issuecomment-2817273924

Verified the fix by:
1. Build: `make SERVER_CFLAGS='-Werror -DLOG_REQ_RES' -j10`
2. Running module api tests: `CFLAGS='-Werror' ./runtest-moduleapi
--log-req-res --no-latency --dont-clean --force-resp3 --dont-pre-clean
--verbose --dump-logs`

---------

Signed-off-by: yairgott <yairgott@gmail.com>
2025-04-23 00:41:17 +02:00
Madelyn OlsonandGitHub b23f97a917 Allow scripts to support null characters again (#1984)
We made a change during the refactoring of the engine changes that used
a strlen on an SDS string. SDS are resizable, but they are binary safe
and support NULL characters, which means we are truncating the string at
the null character.

This is a breaking change for modules, so wanted to raise it for the
upcoming patch.

An alternative is we could expose the script code as a server object,
which would require some construction of the object in the function
pathway and freeing. I opted to keep the code flow as similar as
possible for the patch.

There is a test which only passes now.

For some reasons, functions don't support binary data. So as of right
now this patch doesn't fix that case.

Resolves https://github.com/valkey-io/valkey/issues/1942

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-04-22 14:06:14 -07:00
Roshan KhatriandGitHub d65aae2738 Moved build-release automation to valkey-release-automation (#1977)
All the Valkey Release automation has been moved and setup here:
https://github.com/valkey-io/valkey-release-automation

The release automation will now be triggered from the
[trigger-build-release.yml](https://github.com/valkey-io/valkey/blob/unstable/.github/workflows/trigger-build-release.yml)
workflow.

Need to remove these files to stop duplicating push to the s3 bucket.

---------

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2025-04-22 12:55:12 -07:00
490d307c63 Only enable defrag for vendored jemalloc (#1985)
Only enable defrag for vendored jemalloc. 

Resolves a point of discussion from
https://github.com/valkey-io/valkey/issues/1585, although not the
overall issue. This change should be reverted when we properly validate
using system jemalloc as part of
https://github.com/valkey-io/valkey/issues/1882.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-04-22 14:26:45 +02:00
1b49d37417 Fix CI latest fedora: install awk and tcl8, build tcltls from source (#1965)
Fix two problems in fedora CI jobs:

1. Install awk where missing. It's required for building jemalloc.
2. Fix problems with TCL, required for running the tests. Fedora comes
   with TCL 9 by default, but the TLS package 'tcltls' isn't built for TCL 9.
   Install 'tcl8' and build 'tcltls' from source.

---------

Signed-off-by: vitah <vitahlin@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-04-22 13:16:34 +02:00
skyfireleeandGitHub 04e9c7bf55 Stabilize dual replication test to avoid getting LOADING error (#1979)
Supplement the repair of https://github.com/valkey-io/valkey/pull/1288,
need to wait for the Loading to complete before executing
`$replica replicaof no one`.

Signed-off-by: artikell <739609084@qq.com>
2025-04-20 20:16:29 +02:00
Harkrishn PatroandGitHub 30037b0f53 Avoid shard id update of replica if not matching with primary shard id (#573)
During cluster setup, the shard id gets established through extensions data propagation and if the engine crashes/restarts while the reconciliation of shard id is in progress, there is a possibility of corrupted config file with temporary shard id stored in the cluster. With this fix, a replica sharing a temporary shard id is ignored and allows the cluster bus to converge for only one shard id for primary and it's replicas.


Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-04-18 12:58:30 -07:00
Yair GottdenkerandGitHub 44dd7b66bd Fix engine crash on module client blocking during keyspace events (#1819)
This change enhances user experience and consistency by allowing a
module to block a client on keyspace event notifications. Consistency is
improved by allowing that reads after writes on the same connection
yield expected results. For example, in ValkeySearch, mutations
processed earlier on the same connection will be available for search.

The implementation extends `VM_BlockClient` to support blocking clients
on keyspace event notifications. Internal clients, LUA clients, clients
issueing multi exec and those with the `deny_blocking` flag set are not
blocked. Once blocked, a client’s reply is withheld until it is
explicitly unblocked.

---------

Signed-off-by: yairgott <yairgott@gmail.com>
2025-04-17 18:13:21 -07:00
BinbinandGitHub 56ce94c453 Fix CLUSTER RESET to use lazyfree-lazy-user-flush to do the lazyfree (#1931)
We use lazyfree-lazy-user-flush in #1190 but in #1926 it was changed to
use replica-lazy-flush wrongly.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-04-17 15:38:48 -04:00
nesty92andGitHub 65c13b1af9 Fix incorrect lag reported in XINFO GROUPS (#1952)
When a tombstone is created after the last_id of a consumer group.

The consumer group lag is reported in the reply of `XINFO GROUPS mystream`.

Close: #1951

Signed-off-by: Ernesto Alejandro Santana Hidalgo <ernesto.alejandrosantana@gmail.com>
2025-04-17 20:02:17 +02:00
ef277ffd13 Add sentinel_total_tilt to sentinel INFO sentinel (#1904)
it will be a good idea to add total_tilt in info command to show total
tilt count, in order to help admin to know sentinel tilt condition.

---------

Signed-off-by: carlosfu <carlosfu@163.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-04-17 11:16:48 +02:00
Jacob MurphyandGitHub ca3a8b27c9 Fix flaky shutdown tests caused by timing issue (#1960)
This started failing after
https://github.com/valkey-io/valkey/commit/57cd9a51d9c7d8cf9c4551bd1f2af79a0f279eeb,
since processing of SHUTDOWN might happen after the CLIENT INFO is
retrieved. Now we use wait_for_condition to make sure CLIENT INFO is
retried in the case that SHUTDOWN is not processed yet.

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-04-15 15:08:50 -07:00
Harkrishn PatroandGitHub 69713e1f81 Add node pfail and fail count to cluster info metrics (#1910)
New fields in CLUSTER INFO:

* `cluster_nodes_pfail`
* `cluster_nodes_fail`
* `cluster_voting_nodes_pfail`
* `cluster_voting_nodes_fail`

I'm running few tests and trying to capture partially failed and
completely failed count. Slot partially failed / completely failed stats
exists but is more difficult to assess the node failure count with that.

New output:

```
> CLUSTER INFO
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_nodes_pfail:1
cluster_nodes_fail:0
cluster_voting_nodes_pfail:1
cluster_voting_nodes_fail:0
cluster_known_nodes:3
cluster_size:0
cluster_current_epoch:1
cluster_my_epoch:1
cluster_stats_messages_ping_sent:2104
cluster_stats_messages_pong_sent:1906
cluster_stats_messages_meet_sent:1
cluster_stats_messages_sent:4011
cluster_stats_messages_ping_received:1906
cluster_stats_messages_pong_received:1964
cluster_stats_messages_received:3870
total_cluster_links_buffer_limit_exceeded:0
```

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-04-15 12:40:53 -07:00
BinbinandGitHub 9ce742a93d Ignore stale gossip packets that arrive out of order (#1777)
There is a failure in the daily:
```
=== ASSERTION FAILED ===
==> cluster_legacy.c:6588 'primary->replicaof == ((void *)0)' is not true
```

This is the logs:
```
- i am fd4318562665b4490ccc86e7f7988017cf960371 and myself become a replica,
- 63c0167232dae95cdcc0a1568cd5368ac3b99f5 is the new primary
27867:M 24 Feb 2025 00:19:11.011 * Failover auth granted to 763c0167232dae95cdcc0a1568cd5368ac3b99f5 () for epoch 9
27867:M 24 Feb 2025 00:19:11.039 * Configuration change detected. Reconfiguring myself as a replica of node 763c0167232dae95cdcc0a1568cd5368ac3b99f5 () in shard c5f6b2a9c74cabd4d1e54d1130dc9cb9419bf76f
27867:S 24 Feb 2025 00:19:11.039 * Before turning into a replica, using my own primary parameters to synthesize a cached primary: I may be able to synchronize with the new primary with just a partial transfer.
27867:S 24 Feb 2025 00:19:11.039 * Connecting to PRIMARY 127.0.0.1:23654
27867:S 24 Feb 2025 00:19:11.039 * PRIMARY <-> REPLICA sync started

- in here myself got an stale message, but we still process the packet and cause this issue
27867:S 24 Feb 2025 00:19:11.040 * Ignore stale message from 763c0167232dae95cdcc0a1568cd5368ac3b99f5 () in shard c5f6b2a9c74cabd4d1e54d1130dc9cb9419bf76f; gossip config epoch: 8, current config epoch: 9
27867:S 24 Feb 2025 00:19:11.040 * Node 763c0167232dae95cdcc0a1568cd5368ac3b99f5 () is now a replica of node fd4318562665b4490ccc86e7f7988017cf960371 () in shard c5f6b2a9c74cabd4d1e54d1130dc9cb9419bf76f
```

We can see myself got a stale message, but we still process it, and
changed
the role and cause a primary replica chain loop.

The reason is that, this text is copy from
https://github.com/valkey-io/valkey/pull/651.

In some rare case, slot config updates (via either PING/PONG or UPDATE)
can be delivered out of order as illustrated below:
```
1. To keep the discussion simple, let's assume we have 2 shards, shard a
   and shard b. Let's also assume there are two slots in total with shard
   a owning slot 1 and shard b owning slot 2.
2. Shard a has two nodes: primary A and replica A*; shard b has primary
   B and replica B*.
3. A manual failover was initiated on A* and A* just wins the election.
4. A* announces to the world that it now owns slot 1 using PING messages.
   These PING messages are queued in the outgoing buffer to every other
   node in the cluster, namely, A, B, and B*.
5. Keep in mind that there is no ordering in the delivery of these PING
   messages. For the stale PING message to appear, we need the following
   events in the exact order as they are laid out.
a. An old PING message before A* becomes the new primary is still queued
   in A*'s outgoing buffer to A. This later becomes the stale message,
   which says A* is a replica of A. It is followed by A*'s election
   winning announcement PING message.
b. B or B* processes A's election winning announcement PING message
   and sets slots[1]=A*.
c. A sends a PING message to B (or B*). Since A hasn't learnt that A*
   wins the election, it claims that it owns slot 1 but with a lower
   epoch than B has on slot 1. This leads to B sending an UPDATE to
   A directly saying A* is the new owner of slot 1 with a higher epoch.
d. A receives the UPDATE from B and executes clusterUpdateSlotsConfigWith.
   A now realizes that it is a replica of A* hence setting myself->replicaof
   to A*.
e. Finally, the pre-failover PING message queued up in A*'s outgoing
   buffer to A is delivered and processed, out of order though, to A.
f. This stale PING message creates the replication loop
```

Closes #1015.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-04-15 12:27:44 -07:00
Jacob MurphyandGitHub 57cd9a51d9 Fix panic in primary when blocking shutdown after previous block with timeout (#1948)
When previous timeout isn't zeroed out, shutdown will use the previously
executed blocking commands timeout. Since shutdown is not expected to
have timeouts, this causes a serverPanic.

Without this fix, this scenario leads to a server panic:

1. Call any command that uses blockClient with a timeout (this could be
BLPOP or something that blocks implicitly like CLUSTER SETSLOT)
2. On the same client, call SHUTDOWN with some pending replication data,
which will trigger shutdown to be blocked while we try to catch the
replica up

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
2025-04-15 20:18:44 +02:00
Katie HollyandGitHub a806887c0b Fix cluster slot stats assertion during promotion of replica (#1950)
This affects only the configuration `cluster-slot-stats-enabled yes`.

For the first command a newly promoted primary receives, we send a
`SELECT` command to all replicas via `replicationFeedReplicas`. This
calls a chain of functions (`feedReplicationBufferWithObject` ->
`feedReplicationBuffer` ->
`clusterSlotStatsIncrNetworkBytesOutForReplication` ->
`clusterSlotStatsUpdateNetworkBytesOutForReplication`) that increments
the per-slot `network_bytes_out` by `len * listLength(server.replicas)`.
However, if the new primary receives a command before any replicas
connected yet, `server.replicas` is still empty, causing the increment
to add zero.

Right after, we call `clusterSlotStatsDecrNetworkBytesOutForReplication`
to remove the `SELECT` overhead from the per-slot stats. The assert in
`clusterSlotStatsUpdateNetworkBytesOutForReplication` was using
(basically) `>= -len` instead of `>= -(len *
listLength(server.replicas))`, so it could fail if the previous
increment was zero because we never included the `server.replicas`
length in the assertion.

This patch makes the addition and subtraction use the same computed
value (`rlen = len * listLength(server.replicas)`) in both the stats
update and the assert, ensuring the logic remains consistent when
replicas are promoted.

Fixes #1912

Signed-off-by: Fusl <fusl@meo.ws>
2025-04-15 20:16:35 +02:00
Vitah LinandGitHub 35c0b3e4d3 Fix 'Client output buffer hard limit is enforced' test causing infinite loop on macOS 15.4 (#1940)
This PR fixes an issue in the CI test for client-output-buffer-limit,
which was causing an infinite loop when running on macOS 15.4.

### Problem

This test start two clients, R and R1:
```c
R1 subscribe foo
R publish foo bar
```

When R executes `PUBLISH foo bar`, the server first stores the message
`bar` in R1‘s buf. Only when the space in buf is insufficient does it
call `_addReplyProtoToList`.
Inside this function, `closeClientOnOutputBufferLimitReached` is invoked
to check whether the client’s R1 output buffer has reached its
configured limit.
On macOS 15.4, because the server writes to the client at a high speed,
R1’s buf never gets full. As a result,
`closeClientOnOutputBufferLimitReached` in the test is never triggered,
causing the test to never exit and fall into an infinite loop.

### Fixed

I changed `r publish foo bar` to `r publish foo [string repeat bar 50]`
to ensure the buffer is filled, which correctly reproduces the scenario
where omem increases.

Signed-off-by: vitah <vitahlin@gmail.com>
2025-04-15 11:50:36 +02:00
uriyageandGitHub e6ac570a83 Fix crash during TLS handshake with I/O threads (#1955)
Fix https://github.com/valkey-io/valkey/issues/1883

The issue arises from a missing memory release fence, which can cause
the main thread to see the updated client's IO read state
(`CLIENT_COMPLETED_IO`) but not the updated connection flags. In this
case, it crashes as the flags are not as expected.

Following @skolosov-snap instructions I was able to consistently
reproduce the issue on ARM instances by generating sufficient load to
invoke the IO threads while constantly creating new connections to
trigger the accept flow. I've validated that no crashes occur with this
fix.

I didn't write a regression test for the fix since reproduction is
difficult and flaky.

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
2025-04-15 09:14:10 +02:00
zhenwei piandGitHub c435052e39 Introduce MPTCP (#1811)
Multipath TCP (MPTCP) is an extension of the standard TCP protocol that
allows a single transport connection to use multiple network interfaces
or paths. MPTCP is useful for applications like bandwidth aggregation,
failover, and more resilient connections.

Linux kernel starts to support MPTCP since v5.6, it's time to support
it.

The test report shows that MPTCP reduces latency by ~25% in a 1%
networking packet drop environment.

Thanks to Matthieu Baerts <matttbe@kernel.org> for lots of review
suggestions.

Proposed-by: Geliang Tang <geliang@kernel.org>
Tested-by: Gang Yan <yangang@kylinos.cn>
Signed-off-by: zhenwei pi <zhenwei.pi@linux.dev>
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>

Cc Linux kernel MPTCP maintainer @matttbe
2025-04-15 09:06:33 +02:00
cf33376d40 [Client Introspection] Client Commands Extended Filtering (#1466)
In this PR, we introduce support for new filters for `CLIENT
LIST` and `CLIENT KILL` commands. The new filters are:

1. FLAGS `Client must include this flag. This can be a string with bunch
of flags present one after the other.`
2. NAME `client name`
3. IDLE `minimum idle time of the client`
4. LIB-NAME `clients with the specified lib name.`
5. LIB-VER `clients with the specified lib version.`
6. DB `clients currently operating on the specified database ID`
7. IP `client ip address`
8. CAPA `client capabilities` 

Partly Addresses: #668

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
Co-authored-by: Harkrishn Patro <harkrisp@amazon.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-04-14 16:08:41 -07:00
Wen HuiandGitHub ee5b88be56 Update function clusterNodeSetSlotBit() return type to void (#1934)
No other functions use the return value of clusterNodeSetSlotBit(), thus
this PR update the return type to void and
update the comments.

---------

Signed-off-by: hwware <wen.hui.ware@gmail.com>
2025-04-13 05:52:58 +02:00
Sarthak AggarwalandGitHub ccc34d2089 Rebranding in security warning log (#1945)
Replacing Redis with Valkey in security warning log via the
`SERVER_TITLE` macro.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-04-13 05:41:07 +02:00
7421c3f8fe Trigger manual failover on SIGTERM / shutdown to cluster primary (#1091)
When a primary disappears, its slots are not served until an automatic
failover happens. It takes about n seconds (node timeout plus some
seconds).
It's too much time for us to not accept writes.

If the host machine is about to shutdown for any reason, the processes
typically get a sigterm and have some time to shutdown gracefully. In
Kubernetes, this is 30 seconds by default.

When a primary receives a SIGTERM or a SHUTDOWN, let it trigger a
failover
to one of the replicas as part of the graceful shutdown. This can reduce
some unavailability time. For example the replica needs to sense the
primary failure within the node-timeout before initating an election,
and now it can initiate an election quickly and win and gossip it.

The primary does this by sending a CLUSTER FAILOVER command to the
replica.
We added a replicaid arg to CLUSTER FAILOVER, after receiving the
command,
the replica will check whether the node-id is itself, if not, the
command
will be ignored. The node-id is set by the replica through client
setname
during the replication handshake.

### New argument for CLUSTER FAILOVER

So the format now become CLUSTER FAILOVER [FORCE TAKEOVER] [REPLICAID
node-id],
this arg does not intented for user use, so it will not be added to the
JSON
file.

### Replica sends REPLCONF SET-CLUSTER-NODE-ID to inform its node-id

During the replication handshake, replica now will use REPLCONF
SET-CLUSTER-NODE-ID
to inform the primary of replica node-id.

### Primary issue CLUSTER FAILOVER

Primary sends CLUSTER FAILOVER FORCE REPLICAID node-id to all replicas
because
it is a shared replication buffer but only the replica with the
mathching id
will execute it.

### Add a new auto-failover-on-shutdown config

People can disable this feature if they don't like it, the default is 0.

This closes #939.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Ping Xie <pingxie@outlook.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
2025-04-08 15:06:54 -04:00
Arthur LeeandGitHub ed1f8296ca fix: add samples to stream object consumer trees (#1825)
Use sampling when calculating the memory usage of consumers as part of
streams in MEMORY USAGE and MEMORY STATS.

Without this fix, all consumers are traversed, which is slow if there
are very many consumers.

Fixes #1824

Signed-off-by: arthur.lee <liziang.arthur@bytedance.com>
2025-04-08 17:21:37 +02:00
Vitah LinandGitHub 89c029993d Fix random-key CI failure: key may expire before CLIENT PAUSE (#1932)
Due to the very short TTL (3 ms), it was possible for the key
to expire and be deleted before the pause took effect, especially
under slowing environments. This caused [r randomkey] to return {}
instead of "key".

Wrap the commands in a MULTI/EXEC block to ensure that the key is
set and the pause is applied within the same transaction, reducing
the likelihood of premature expiration.

Test was introduced in #1850.

Signed-off-by: vitah <vitahlin@gmail.com>
2025-04-08 19:36:08 +08:00
116be22a3f Replaced the use of sdsReplyDictType and hashDictType. (#1793)
This PR resolves #1550 by replacing the use of temporary dict structures
with hashtable for random elements selection.

hrandfieldWithCountCommand:

CASE 3: Replaced sdsReplyDictType with sdsReplyHashtableType to store a
pointer to the entry.
CASE 4: Replaced hashDictType with setHashtableType, as only the field
needs to be stored.

zrandmemberWithCountCommand:
used similar logic to replace the use of hashDictType and
sdsReplyDictType in CASE 3.

RDB load skiplist field duplication validation logic:
Replace use of dict for storing listpack/hash loaded fields when
detecting duplications.

---------

Signed-off-by: Shai Zarka <zarkash@amazon.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-04-08 11:58:46 +03:00
Harkrishn PatroandGitHub eabfcc92c7 Kill RDB child process on CLUSTER RESET command (#1926)
Fixes: #1925

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-04-07 14:49:21 -07:00
Sergey KolosovandGitHub c09e664f32 Implement: CLUSTER REPLICATE NO ONE (#1674)
Currently, Valkey doesn't allow to detach replica attached to primary
node. So, if you want to change cluster topology the only way to do it
is to reset (````CLUSTER RESET```` command) the node. However, this
results into removing node from the cluster what affects clients. All
clients will keep sending traffic to this node (with getting inaccurate
responses) until they refresh their topology.

In this change we implement supporting of new argument for CLUSTER
REPLICATE command: ````CLUSTER REPLICATE NO ONE````. When calling this
command the node will be converted from replica to empty primary node
but still staying in the cluster. Thus, all traffic coming from the
clients to this node can be redirected to correct node.

Signed-off-by: Sergey Kolosov <skolosov@snapchat.com>
2025-04-07 14:49:01 -07:00
Nathan ScottandGitHub 60f17a1535 Fix the build on less common platforms in zmalloc.c (#1922)
Corrects a mismatched type declaration for the used_memory_thread
pointer into the used_memory_thread_padded array (global scope) - in the
less used else branch of a cpp platform conditional.

Fixes #1916

Signed-off-by: Nathan Scott <nathans@redhat.com>
2025-04-07 13:28:00 +02:00
747a3583c8 Fix minor comment typo (#1919)
Signed-off-by: lucasyonge <lucasyonge@gmail.com>
Co-authored-by: Ping Xie <pingxie@outlook.com>
2025-04-06 13:54:31 -07:00
Harkrishn PatroandGitHub ff0e31e4d0 Add human node name to log statement with node name (#1918)
Updated log statements consisting of node name to accompany with human
node name for easier debuggability.

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-04-05 18:33:53 -07:00
Harkrishn PatroandGitHub a21a1d79ce Fix valkey-cli port parse logic for invalid string (#1915)
Before
```
> src/valkey-cli -p CLUSTER NODES
Could not connect to Valkey at 127.0.0.1:0: Connection refused
```
After
```
> src/valkey-cli -p CLUSTER NODES
Invalid server port.
```

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-04-04 11:52:42 +08:00
Sarthak AggarwalandGitHub 7b518688fb In LOLWUT's reply, change "Redis ver." to "Valkey ver." (#1559)
Make LOLWUT it print the current Valkey version rather than the Redis
version. Retain the old behaviour and print the compatible Redis version
if the `extended-redis-compat` config is enabled.

Additionally, change "Redis" to "Valkey" in an error message "This Redis
command is not allowed from script", also depending on the
`extended-redis-compat` config.

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
2025-04-04 00:00:21 +02:00
Amit NaglerandGitHub eb2dec037e Initialize child process pipe file descriptor to -1 (#1911)
This commit addresses an issue with the initialization of the child
process pipe file descriptor. Previously, it was initialized to 0, which
is a valid file descriptor (typically representing standard input). This
could potentially lead to confusion and errors in file descriptor
handling.

File descriptor 0 is a valid descriptor, which could cause unexpected
behavior or conflicts in file operations. Using -1 as the initial value
clearly indicates that the file descriptor is not yet set or is invalid.
This change aligns with common practices in file descriptor handling,
where -1 is often used to represent an uninitialized or invalid
descriptor.

Signed-off-by: naglera <anagler123@gmail.com>
2025-04-03 10:40:55 +03:00
Viktor SöderqvistandGitHub 35e293d168 Bump CMake to 3.10.0 to fix build error on Ubuntu-24 (#1908)
This makes it build on Ubuntu-24, which otherwise fails with this error:

CMake Error at deps/hiredis/CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
  Compatibility with CMake < 3.5 has been removed from CMake.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-04-01 17:20:19 -07:00
lucasyongeandGitHub 1dba6cd37d Update ACL SETUSER command help message (#1899)
When I learn ACL SETUSER command from valkey.io, I find the helper
message is out-of-date,
I think the 'attribute' should be updated to 'rule' to align with the
link https://valkey.io/commands/acl-setuser/

Signed-off-by: lucasyonge <lucasyonge@gmail.com>
2025-04-01 23:00:01 +02:00
8deaee3037 Add --sequential to valkey-benchmark (for populating entire keyspace) (#1839)
I needed a way to systematically populate the keyspace before running
`valkey-benchmark` tests like get, so I added a `--sequential` option
that modifies the behavior of `-r` so that the replacements are
sequential numbers in the keyspace instead of random numbers.

Example:
```
# populate keys by sequentially setting them
./valkey-benchmark -d 512 --sequential -r 3000 -n 3000 -c 650 -P 4 --threads 64 -t set -q

# benchmark get with 100% hit rate
./valkey-benchmark -r 3000 -n 250000 -c 650 -P 4 --threads 64 -t get
```

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-04-01 22:10:35 +02:00
2c3841895f Redact protocol error log when hide-user-data-from-log enabled (#1889)
In this code logic:
https://github.com/valkey-io/valkey/blob/unstable/src/networking.c#L2767-L2773,
`c->querybuf + c->qb_pos` may also include user data.
Update the log message when config `hide-user-data-from-log` is enabled.

---------

Signed-off-by: VanessaTang <yuetan@amazon.com>
Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
2025-03-31 10:46:20 +03:00
5f2dc4862e Remove TLSCONN_DEBUG dead code in tls.c (#1891)
This code is a dead code, there is also a undefined fd error in it.
Obviously this will not compile if this kind of debugging is enabled.

This seems to be broken ever since
2d5ba02f8f
which was done in 2019. It's dead code that we never test and never use.

Fixes #1887

Signed-off-by: WelongZuo <zuowl@qq.com>
Co-authored-by: z00636558 <zuowenlong6@huawei.com>
2025-03-29 19:51:01 +08:00
BinbinandGitHub 6b4fc0773d Fix TCL tmp dir leak in the ACL load test (#1895)
Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-03-28 10:28:50 -07:00
Jim BrunnerandGitHub 1f41009a9a Fix merge error introduced in #1186 (#1894)
Merge error introduced in #1186 .

Prior to the merge, the time value passed to modules was a `monotime`.
The merge reverts it to a wall clock time (on the sending side only).

Signed-off-by: Jim Brunner <brunnerj@amazon.com>
2025-03-27 20:16:15 +01:00
1d8e29c36c Fix crash in VM_GetCurrentUserName when no valid user (#1885)
VM_GetCurrentUserName make the binary crash when it gets not fully
structured ctx, when it doesn't have a valid ctx, client, user behind
it. Rather than making the binary crash, safely return NULL.

---------

Signed-off-by: jeon1226 <jeon1226@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-03-27 09:44:17 +01:00
1bd911b6ea Update COPY and MOVE command description clearer in JSON file (#1843)
As I do the code review for PR
https://github.com/valkey-io/valkey/pull/1671 (Add multi-database
support to cluster mode), I find
the return value message of the command COPY and MOVE is not very clear
to user. Thus I update them in this PR and valkey-doc repo as PR
https://github.com/valkey-io/valkey-doc/pull/249

---------

Signed-off-by: hwware <wen.hui.ware@gmail.com>
Signed-off-by: Wen Hui <wen.hui.ware@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-03-26 11:27:42 -04:00
Harkrishn PatroandGitHub 213f4374ae [cluster] Add node id to log statement for closing link on first message as lightweight (#1869)
---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2025-03-21 10:42:06 -07:00
Wen HuiandGitHub 75b04684c3 Minor cleanup remove unnecessary cast since slot is int (#1865)
Signed-off-by: hwware <wen.hui.ware@gmail.com>
2025-03-20 10:42:19 +08:00
6faa7b7dfa Fix RANDOMKEY infinite loop during CLIENT PAUSE (#1850)
When the `client pause write` is set and all the keys in the server
are expired keys, executing the `randomkey` command will lead to an
infinite loop.

The reason is that expired keys are not deleted in this case. Limit
the number of tries and return an expired key after the max number
tries in this case.

Closes #1848.

---------

Signed-off-by: li-benson <1260437731@qq.com>
Signed-off-by: youngmore <youngmore1024@outlook.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: youngmore <youngmore1024@outlook.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-03-18 11:48:50 +01:00
Meinhard ZhouandGitHub 912353f44c Support IPv6 address in RDMA test (#1817)
Signed-off-by: Meinhard Zhou <zhouenhua@bytedance.com>
2025-03-18 10:11:28 +01:00
Nikhil MangloreandGitHub ab924ecd78 Update correct repository name for automation trigger workflow (#1855)
Update automation repository name to `valkey-release-automation`

Signed-off-by: Nikhil Manglore <nmanglor@amazon.com>
2025-03-17 15:54:06 -07:00
4451e497ad Trigger post-release tasks in Valkey for a new release (#1830)
This trigger-build-release.yml file will be used to automate the release
process as described in #1397. It will trigger the post release task after a new release

---------

Signed-off-by: Nikhil Manglore <nmanglor@amazon.com>
Signed-off-by: Nikhil Manglore <nikhilmanglore9@gmail.com>
Co-authored-by: Roshan Khatri <117414976+roshkhatri@users.noreply.github.com>
2025-03-17 15:35:56 -07:00
abec1ca8d3 Update valkey-benchmark parseURI function name and comment (#1845)
In Valkey, when user runs the valkey-benchmark tools, the command
**valkey-benchmark -u** could accept 4 schemes of server URI format:  
"valkey://", "valkeys://", "redis://", "rediss://"   

Thus, I add them in the function comment, and update the function name
to parseRedisOrValkeyUri.

Finally, I fix one mistake for valkey-benchmark tool name for this
command.

---------

Signed-off-by: hwware <wen.hui.ware@gmail.com>
Signed-off-by: Wen Hui <wen.hui.ware@gmail.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
2025-03-17 11:53:02 -04:00
Rain ValentineandGitHub 2259897251 Fix defrag when type/encoding changes during scan (#1801)
I was hunting for defrag bugs with Jim and found a couple improvements
to make. Jim pointed out that in several of the callbacks, if the
encoding were to change it simply returns without doing anything to
`cursor` to make it reach 0, meaning that it would continue no-op
working on that item without making any progress. Type and encoding can
change while the defrag scan is in progress if the value is mutated or
replaced by something else with the same key.

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
2025-03-16 17:41:16 +01:00
dbcad3f38d Fix ACL LOAD crash on replica since the primary client don't has a user (#1842)
Check for NULL before accessing users of a client during the `ACL LOAD`
command.
This prevents the command causing a segmentation fault in the replica.

Fixes #1832.

---------

Signed-off-by: Bogdan Petre <bogdan.petre@aiven.io>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-03-13 09:52:07 -04:00
BinbinandGitHub 3a239f76e8 Add missing close_replication_stream on multi test (#1841)
Minor cleanup.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-03-13 00:42:56 +08:00
BinbinandGitHub 0dea426c18 Save config file and brocast the PONG when configEpoch changed (#1813)
This is somehow related with #974 and #1777. When the epoch changes,
we should save the configuration file and broadcast a PONG as much
as possible.

For example, if a primary down after bumping the epoch, its replicas
may initiate a failover, but the other primaries may refuse to vote
because the epoch of the replica has not been updated.

Or for example, for some reasons we bump the epoch, if the epoch
is not updated in time in the cluster, it may affect the judgment
of message staleness.

These broadcasts are expensive in large clusters, but none of these
seem high frequency so it should be fine.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-03-11 14:53:43 +08:00
19c33ec1a1 Update CloseKey module API documentation to avoid use-after-free behavior (#1834)
### Issue: https://github.com/valkey-io/valkey/issues/1775

Without a clear comment, users might mistakenly attempt to access or
close the same key object multiple times, leading to use-after-free
errors, undefined behavior, or crashes due to double-free operations.

---------

Signed-off-by: Seungmin Lee <sungming@amazon.com>
Signed-off-by: Seungmin Lee <155032684+sungming2@users.noreply.github.com>
Co-authored-by: Seungmin Lee <sungming@amazon.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
2025-03-10 15:45:34 -07:00
eifrah-awsandGitHub c19fb2578b [CMake] Check both arm64 and aarch64 for ARM based system architecture (#1829)
While macOS will report `arm64` for `uname -m` command, others might
report `aarch64`. This PR fixes this

Signed-off-by: Eran Ifrah <eifrah@amazon.com>
2025-03-10 13:29:21 -07:00
Madelyn OlsonandGitHub c594f04b61 Fix bug where invalidation messages were getting sent to closing clients (#1823)
So I think we were seeing [these
timeouts](https://github.com/valkey-io/valkey/actions/runs/13688155322/job/38276139543#step:6:839)
because QUIT behaves differently between IO threading and non-IO
Threading. In both cases, `QUIT` is a close after reply command. Once
the client has written out the results, it gets added to the queue to
that gets cleaned up at the end of the event loop. Normally this is
fine, as before we circle around to the next event loop this client is
definitely killed.

For IO threads, we need to process the pending IO commands to add the
client to the kill queue. This may not happen immediately, which means
we might go down and process that `SET` command *before* we free the
client that is supposedly already quit. This is very sensitive to
timing, so it's not very likely, but still possible. Once the `SET` has
been executed, the invariants in the tests are off since it will get a
correct invalidation.

The fix is to also mark a client as broken if it's being closed. 

The test was also hanging because of a test issue, because the
conditional `lsearch` check was returning 1 or 0 strings, which are both
valid exit criteria for the wait_for.

```
./runtest --io-threads --accurate --verbose --tags network --dump-logs --single unit/tracking --loops 500 --clients 25
```

Fixes #1647 (I believe this now!)

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-03-09 23:12:14 -07:00
Ricardo DiasandGitHub 84f4cd05ff Adds a memory leak check after running a unit test (#1798)
This commit adds check after each test function execution of a unit
test, that checks if there is still memory allocated.

This check prevents situations where tests, which do these kind of
checks, fail due to memory leaks caused by previous tests.

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-03-06 11:23:21 +08:00
ranshidandGitHub 97080e412c Enable large-memory tests solo runs in daily workflow (#1816)
This change re-introduce large-memory tests to be run as solo tests on
daily runs.

the change includes:
1. separate the large-memory tests into separate test block (mainly
helpful in workflow manual dispatch cases)
2. place all current large memory tests in run solo blocks in order to
prevent potential runner OOM failures.

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-03-05 14:15:14 +02:00
BinbinandGitHub ff7189ba64 Fix timing issue in the module propagate test (#1815)
There is a timing issue in the test:
```
*** [err]: module RM_Call of expired key propagation in tests/unit/moduleapi/propagate.tcl
Expected '1' to be equal to '2' (context: type eval line 27 cmd {assert_equal [$replica propagate-test.obeyed] 2} proc ::test)
```

We should wait for sync, otherwise the replica may not be fully
synchronized before checking. The test was introduced in #1582.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-03-04 17:34:19 -05:00
zhenwei piandGitHub 4c7a0231b7 Use the wrapper from cli_common instead of hiredis (#1802)
There are mixed usage around `redisConnect*`, we should use
`redisConnectWrapper*` wrapper from cli_common instead of
`redisConnect` from hiredis. This is a cleanup.

Signed-off-by: zhenwei pi <zhenwei.pi@linux.dev>
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2025-03-04 16:15:07 +08:00
2760a844f8 Fix incorrect assertion in client list operations (#1800)
The current assertion introduced in #11220:
```c
serverAssert(&c->clients_pending_write_node.next != NULL || &c->clients_pending_write_node.prev != NULL);
```

is incorrect for two reasons:
1. Using &pointer.next would always be non-NULL since it's the address
of the field.
2. The check is incorrect even without the & because in a single-node
list, both pointers can be NULL.

Fix:
1. Remove the always-true assertion
2. Add proper assertions in listUnlinkNode to ensure the node membership
in the list to cover all list cases.

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
Signed-off-by: uriyage <78144248+uriyage@users.noreply.github.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2025-03-04 15:24:28 +08:00
secwallandGitHub aeac44a1cf Drop lua object files on clean (#1812)
Let `make clean` delete the files `src/lua/*.o` and `src/lua/*.d`.

---------

Signed-off-by: secwall <secwall@yandex-team.ru>
2025-03-03 19:00:39 +01:00
Madelyn OlsonandGitHub a473d182ad Stop running large memory test for address santizer (#1810)
After this change, https://github.com/valkey-io/valkey/pull/1767/files,
the address sanitizer test seems to be causing the test process to just
die, possibly from OOM:
https://github.com/valkey-io/valkey/actions/runs/13620692597/job/38069694391#step:6:4755.

If we switch the large memory tests to be sequential on ASAN, it might
resolve it, but I wanted to open this to propose reverting it to get the
tests passing again.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-03-03 08:20:21 -08:00
zhaozhao.zzandGitHub 1d705aabf4 make net_input_bytes_curr_cmd more readable (#1756)
The metric `net_input_bytes_curr_cmd` is now computed by aggregating its components separately.

Signed-off-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
2025-03-03 09:40:24 +08:00
eifrah-awsandGitHub c9eef8cec0 Fixed build error with CMake (#1806)
Explicitly cast `long long` -> `long double` to avoid build warnings.
This issue manifests when using `clang 19`. Using the `Makefile` build,
we only get a warning. This small PR fixes this.

Fixes issue: https://github.com/valkey-io/valkey/issues/1805

Signed-off-by: Eran Ifrah <eifrah@amazon.com>
2025-03-02 14:05:51 +02:00
17975a88ac Update and stabilize defrag tests (#1762)
A number of tests related to defrag have had stability problems.

One reason for stability issues is that the tests are run sequentially
on the same server, and it takes jemalloc some time to release freed
pages. This is especially noticed immediately after a flushall. Over a
period of 10 seconds, it is observable that the "fragmentation bytes"
can decrease by several MB. Another reason is that there's no
standardization between tests. For each test, people have been
independently hacking/tweaking the success criteria, without addressing
underlying issues.

This update revamps all of the defrag tests:
* A fresh server is started for each test. Running each test in
isolation improves stability.
* A uniform function `log_frag` is now used for debug logging
* A uniform function `perform_defrag_test` ensures that each test is
written and executed in a uniform fashion. Limits are imposed to ensure
that the defrag results are consistent/reproducible. The intent is to
eliminate failures do to various tweaks to values in individual tests.
* Latency is tested much more strictly for most tests, reflecting the
recent improvements to defrag latency.
* The test `defrag edge case` has been removed. This test attempted to
create N pages with EXACTLY equal fragmentation in an attempt to confuse
the defrag logic. It's unlikely that this test was performing correctly,
and had questionable value.
* Tests for hash/list/set/zset/stream have been separated and
standardized. It was unlikely that the old test was performing properly
as none of the actual data structures were fragmented!

It's noted that pubsub doesn't appear to be defragging correctly. The
old test was based on deletion of strings (only) which doesn't actually
reflect what happens when a pubsub channel is removed. The test has been
reduced to only check that pubsub is not damaged during defrag - but
doesn't test for defrag efficacy. This isn't likely a significant issue
as it would be unlikely to create many thousands of pubsub channels and
then have associated fragmentation issues.
https://github.com/valkey-io/valkey/issues/1774

Resolves: https://github.com/valkey-io/valkey/issues/1746

---------

Signed-off-by: Jim Brunner <brunnerj@amazon.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-02-28 09:59:46 +01:00
Viktor SöderqvistandGitHub bb4ef6e84d Fix hashTypeEntryDefrag returning bad pointer (#1799)
Fixes #1795.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-02-28 09:56:52 +01:00
BinbinandGitHub 72c2f02a0a Add cluster-manual-failover-timeout to configure the timeout for manual failover (#1690)
Allows cluster admins to configure the cluster manual failover timeout
as
needed, admins can configure how long a primary would be paused in the
worst case scenario such as a failover timed out due to the insufficient
votes.

The configuration name is cluster-manual-failover-timeout, the unit is
milliseconds, and the range is [1, INT_MAX]ms.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-02-27 12:53:43 -08:00
Ricardo DiasandGitHub 8005a9e71f Fix memory leak in test_quicklist.c unit test (#1797)
In last daily run we had the following failure:

```
test — compress and decomress quicklist plain node large than UINT32_MAX
[test_quicklistCompressAndDecomressQuicklistPlainNodeLargeThanUINT32MAX - unit/test_quicklist.c:2288] Compress and decompress: 4096 MB in 39.27 seconds.

[ok] - test_quicklist.c:test_quicklistCompressAndDecomressQuicklistPlainNodeLargeThanUINT32MAX
[END] - test_quicklist.c: 58 tests, 58 passed, 0 failed
[START] - test_rax.c
[test_raxRandomWalk - unit/test_rax.c:548] Failed assertion: raxAllocSize(t) == zmalloc_used_memory()
[fail] - test_rax.c:test_raxRandomWalk
```

Job Link:
https://github.com/valkey-io/valkey/actions/runs/13555915665/job/37890070586

Although the assert that failed was in a `test_rax.c` test function, the
problem was in the last test function from the `test_quicklist.c` unit
test that ran just before the test that failed.

The

`test_quicklistCompressAndDecomressQuicklistPlainNodeLargeThanUINT32MAX`
test function was not freeing the string allocated in the beginning of
the test.

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-02-27 11:37:28 -08:00
Madelyn OlsonandGitHub 481d6aee6f Consistent look and feel of licenses (#1788)
Use a consistent set of licenses for Valkey files. I took a look and
applied sort of a "did we make a material change in this file?" and
tried to be conservative in adding the trademark. We could also be
liberal as well.

Resolves: https://github.com/valkey-io/valkey/issues/1692.

Included documentation about the licensing here:
https://github.com/valkey-io/valkey/pull/1787.

Licenses are now also always explicitly first, even about documentation
files.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-02-27 13:38:26 -05:00
f37e3ea7aa Fix clang build error in bitops.c (#1794)
In a recent PR https://github.com/valkey-io/valkey/pull/1741 the new
header `<immintrin.h>` was added, which transitively includes
`<mm_malloc.h>` header, where a function called `_mm_malloc(...)` makes
a call to the `malloc` function.

The Valkey server code explicitly sets the malloc function as a
deprecated function in `server.h`:
```c
void *malloc(size_t size) __attribute__((deprecated));
```

The Valkey server code is then compiled with
`-Werror=deprecated-declarations` option to detect the uses of
deprecated functions like `malloc`, and due to this, when the `bitops.c`
file is compiled with Clang, fails with the following error:

```
In file included from bitops.c:33:
In file included from /usr/lib/llvm-18/lib/clang/18/include/immintrin.h:26:
In file included from /usr/lib/llvm-18/lib/clang/18/include/xmmintrin.h:31:
/usr/lib/llvm-18/lib/clang/18/include/mm_malloc.h:35:12: error: 'malloc' is deprecated [-Werror,-Wdeprecated-declarations]
   35 |     return malloc(__size);
      |            ^
./server.h:3874:42: note: 'malloc' has been explicitly marked deprecated here
 3874 | void *malloc(size_t size) __attribute__((deprecated));
```

There is a difference in behavior though, between GCC and Clang. The
`bitops.c` file compiles successfully with GCC.

I don't know exactly why GCC does not issue a warning in this case. My
best guess is that GCC does not issue warnings from code of the standard
library.

To fix the build error in clang, we explicitly use `pragma` macro to
tell clang to ignore deprecated declarations warnings in `bitops.c`.

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
Signed-off-by: Ricardo Dias <rjd15372@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2025-02-27 13:36:27 -05:00
Roshan KhatriandGitHub a7beada820 Migrate binaries build to ARM github runners (#1790)
This PR migrates the workflows to run on github hosted arm runners for each ubuntu version that is currently supported. It is much faster than the emulated version that used to take around 21 mins. Currently, it takes around 4-5 mins.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
2025-02-27 09:57:30 -08:00
68d75445c0 Add Cluster Bus Port out of range error message for Cluster Meet command (#1686)
Now for the cluster meet command, we first check the port and cport
number, and then the function clusterStartHandshake() is called.
In the function clusterStartHandshake(), ip address is checked first and
then port and cport range are checked.
Even port or cport range is out of bound, only error message "Invalid
node address specified" is reported.
This is not correct.

In this PR, I just make the port and cport check together before
clusterStartHandshake() function. Thus the port number and range can be
checked together.

One example:

Current behavior:

127.0.0.1:7000> cluster meet 10.21.96.98 65000
(error) ERR Invalid node address specified: 10.21.96.98:65000
127.0.0.1:7000> cluster meet 1928.2292.2983.3884.26622 6379
(error) ERR Invalid node address specified:
1928.2292.2983.3884.26622:6379

Whatever the wrong ip address or incorrect bus port number, user get the
same error message.

New behavior:

127.0.0.1:7000> cluster meet 10.21.96.98 65000
(error) ERR Cluster bus port number is out of range
127.0.0.1:7000> cluster meet 1928.2292.2983.3884.26622 6379
(error) ERR Invalid node address specified:
1928.2292.2983.3884.26622:6379

User can get much clearer error message.

Related note pr: https://github.com/valkey-io/valkey-doc/pull/240

---------

Signed-off-by: hwware <wen.hui.ware@gmail.com>
Signed-off-by: Wen Hui <wen.hui.ware@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-02-27 12:22:22 -05:00
zhaozhao.zzandGitHub 9429a1f61b cmd's out bytes need count deferred reply (#1760)
the special deferred reply is ignored in current command's
`net_output_bytes_curr_cmd` counting

Signed-off-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
2025-02-27 15:59:48 +08:00
xbaselandGitHub 958b6821cf valkey-cli: ensure output ends with a newline if missing when printing reply (#1782) 2025-02-26 21:48:00 -08:00
50b95ee922 Remove unicode optimization in Lua cjson library (#1785)
The Lua cjson library implements an optimization that pre-allocates a
string buffer by estimating the maximum memory used if all characters in
a string require to be represented as unicode escapes, which may take 6
bytes each. Therefore, if a string has `len` bytes, the pre-allocated
buffer will have `len * 6` bytes.

This optimization can easily cause OOM errors, because if someone uses a
string with a few gigabytes of size, the pre-allocator will require 6
times the size of that string, and when running the Lua script in a
small instance with low memory, it will make the valkey-server process
to abort.

I ran the following Lua script to check if there is a significant
performance regression:

```
local s = string.rep("a", 1024 * 1024 * 1024)
return #cjson.encode(s..s..s)
```

The execution duration, in seconds, for 3 runs before and after this
commit is the following:

Before: 46.309; 42.443; 42.242

After: 46.729; 42.969; 42.774

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
Signed-off-by: Ricardo Dias <rjd15372@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-02-26 19:03:31 +01:00
Viktor SöderqvistandGitHub f8f03c70e9 Fix undefined behaviour in bitops unit test (#1786)
The unit test was added in #1741. It fails when compiled with UBSAN.
Using a local array like `char buf[size]` is undefined behaviour when
`size == 0`. This fix just makes it size 1 in this case.

The failure I got locally:

unit/test_bitops.c:28:13: runtime error: variable length array bound
evaluates to non-positive value 0

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-02-26 19:00:08 +01:00
chzhooandGitHub 5b3b00c0a5 Optimize bitcount command by SIMD (#1741)
**Background**

- Currently, we implement bitcount using a lookup table method
- By SIMD, parallel table lookups can be achieved, which boosts
performance
- Most x86 servers support the AVX2 instruction set

**BenchMark**
| Value Size | QPS (After optimization) | QPS (Before optimization) |
change |
| ---- | ---- | ---- | ---- |
|16 B | 114925| 115924 |  -0.8%|
|256 B| 112619 | 112201|  +0.3%|
|4 KB| 105523|96251| +9.6%|
|64 KB|79723|36796| +116%|
|1MB|21306|3466|+514%|

CPU: AMD EPYC 9754 128-Core Processor * 8
OS:   Ubuntu Server 22.04 LTS 64bit
Memory: 16GB
VM: Tencent cloud SA5.2XLARGE16

**Test Plan**
Pending. Will add test if it looks okay

**Other**
This PR is based on
https://github.com/WojciechMula/sse-popcount/blob/master/popcnt-avx2-lookup.cpp

---------

Signed-off-by: chzhoo <czawyx@163.com>
2025-02-26 11:55:04 +01:00
BinbinandGitHub 45255d806b Fix temp file leak druing replication error handling (#1721)
Before actually entering REPL_STATE_TRANSFER, we usually have
some other things to do, such as registering the ae handler, etc.
If an error occurs at this time, we may leak the previously opened
temp file.

This commit adds a new cleanupTransferResources function to do the
cleanup, avoiding code duplication.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-02-26 11:40:12 +08:00
Jim BrunnerandGitHub 78de3b8b46 defrag: remove assert on defrag_later (#1779)
If a flushall is performed in the middle of defrag, the current stage
will be aborted when the change in kvs is detected. However the defrag
cycle will continue and defrag will complete normally. In a normal
termination, we expect the `defrag_later` list to be empty - and it
might not be in this condition.

This update removes the incorrect assertion, allowing the list to be
freed just like for an abnormal termination.

Signed-off-by: Jim Brunner <brunnerj@amazon.com>
2025-02-25 12:59:05 -08:00
c0679b65d6 Enable TCP_NODELAY by default in incoming and outgoing connections (#1763)
Fixes https://github.com/valkey-io/valkey/issues/1758
This is a follow up issue from
https://github.com/valkey-io/valkey/pull/1706#issuecomment-2669567875


### Problem: 
The absence of TCP_NODELAY can cause unnecessary latency due to [Nagle’s
algorithm](https://en.wikipedia.org/wiki/Nagle%27s_algorithm) (buffers
small TCP packets), which could be disabled for traffic.
We need to consider enabling TCP_NODELAY by default in both incoming and
outgoing connections
---------

Signed-off-by: Seungmin Lee <sungming@amazon.com>
Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
Co-authored-by: Seungmin Lee <sungming@amazon.com>
Co-authored-by: Harkrishn Patro <harkrisp@amazon.com>
2025-02-25 10:35:21 -08:00
Viktor SöderqvistandGitHub 3f763ac605 Embed hash value in hash type entry (#1579)
For a hash type key represented as a hash table, embed the field and
value in one allocation when they fit together in an allocation of up to
128 bytes. For larger fields and values, another layout is used where
the value is a separately allocated sds string.

**Implementation**

The hashTypeEntry pointer is changed to be the field sds, i.e. a pointer
to the embedded field content, which is located after the sds header in
memory. We encode the entry layout in the some unused bits in the sds
header.

Entry with embedded field and value, used when they're both small. The
value is stored as SDS_TYPE_8. The field can use any SDS type.

    +--------------+---------------+
    | field        | value         |
    | hdr "foo" \0 | hdr8 "bar" \0 |
    +------^-------+---------------+
           |
           |
         entry pointer = field sds

Entry with value-pointer and embedded field, used for larger field-value
pairs. The field is SDS type 8 or higher.

    +-------+--------------+
    | value | field        |
    | ptr   | hdr "foo" \0 |
    +-------+------^-------+
                   |
                   |
                entry pointer = field sds

An embedded sds5 field implies the first entry layout.

Fixes #1551.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-02-25 19:00:06 +01:00
Anastasia AlexandrovaandGitHub 80936bacb9 Fixed active-expire-effort description in conf file (#1773)
Expired keys search happens without user actions. Therefore, the
"interactively" word in the description of the active-expire-key
parameter is confusing and is changed to "incrementally."
 
modified:   valkey.conf

---------

Signed-off-by: Anastasia Alexadrova <anastasia.alexandrova@percona.com>
2025-02-25 11:06:27 +01:00
ranshidandGitHub 54f26ed144 Add large memory flag for asan tests (#1767)
Currently we do not use --large-memory flavor in our daily tests.
There is some value enabling this flavor in order to identify different issues related to buffer overrun and bad memory access.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-02-24 20:36:27 -08:00
Nikhil MangloreandGitHub feef7194ab Fixed issue with CONFIG RESETSTAT in cluster module message callback test (#1768)
**Problem**
The cluster module test "Cluster module message DING/DONG
acknowledgment" is failing despite successful message exchanges between
nodes. While these messages are being properly exchanged between the
nodes, running multiple tests on the same cluster instance is causing
issues with message statistics tracking. `CONFIG RESETSTAT` doesn't seem
to reset the stats under certain cases which causes the error as the
stats are adding onto each other from previous tests.

**Solution**
Remove some overlap with the previous test case so we don't need `CONFIG
RESETSTAT` between the test cases.

Resolves #1764

---------

Signed-off-by: Nikhil Manglore <nmanglor@amazon.com>
2025-02-24 20:04:32 +01:00
Viktor SöderqvistandGitHub 4c5b0f25fb Disable Fedora Fawhide in Daily runs (#1769)
Because of a packaging problem of TCL in Fedora Rawhide (Fedora's
unstable branch), temporarily disable tests until the packaging is fixed
in the distro.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-02-24 12:26:33 +01:00
secwallandGitHub e06fb69e07 Fix murmur32 on large strings (#1748)
Current murmur32 implementation fails on large strings.
Way to reproduce crash:
```
eval 'local s = string.rep("a", 1024 * 1024 * 1024) return #cjson.encode(s..s..s)' 0
```
(Basically this is a copy-paste from large test in
`tests/unit/scripting.tcl`).

Shouldn't we run tests with `--large-memory` on daily CI so we could
find this earlier?

I think we need to backport this to 8.1 branch.

Signed-off-by: secwall <secwall@yandex-team.ru>
2025-02-23 09:28:06 +02:00
ec10400cf5 Fix error "SSL routines::bad length" when connTLSWrite is called second time with smaller buffer (#1737)
Issue described in #1135 

When call to `connTLSWrite` fails on next attempt `connTLSWritev` should
provide buffer with at least same amount of bytes,
but if first call was with buffer smaller than
`NET_MAX_WRITES_PER_EVENT` then buffer gets larger and exceed
`NET_MAX_WRITES_PER_EVENT`, `connTLSWritev` will not combine `iov` into
one buffer resulting into chance that first element of `iov` is smaller
than last failed call to `connTLSWrite` and causing `SSL routines::bad
length`.

This change force combining `iov`s into one buffer if first element of
`iov` is smaller than last failed write by `connTLSWrite`

---------

Signed-off-by: Marek Zoremba <marek@janeasystems.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: ranshid <ranshid@amazon.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: ranshid <ranshid@amazon.com>
2025-02-21 09:54:24 +02:00
0b2d21b5d7 Move TCP/TLS specific options from generic client to connection type (#1706)
Fixes https://github.com/valkey-io/valkey/issues/1702

Signed-off-by: Seungmin Lee <sungming@amazon.com>
Co-authored-by: Seungmin Lee <sungming@amazon.com>
2025-02-20 14:33:37 -08:00
82138c8701 Add new module API flag to bypass command validation (#1357)
### Issue https://github.com/valkey-io/valkey/issues/1175
### Problem
Performance degradation occurs due to the sanity
check([lookupCommandByCString](https://github.com/valkey-io/valkey/blob/unstable/src/module.c#L3539))
in the VM_Replicate function, which converts const char* into sds and
free them for command dictionary lookup. This check is mainly used for
debugging purpose, but it is unnecessary for trusted modules. The user
seeks a way to bypass this check for performance gains.

### Solution
Introduce a new SKIP_COMMAND_VALIDATION option which allows individual
modules to opt out of command validation

### Test
ADded a unit test

---------

Signed-off-by: Seungmin Lee <sungming@amazon.com>
Co-authored-by: Seungmin Lee <sungming@amazon.com>
2025-02-20 13:43:35 -08:00
Ricardo DiasandGitHub 8603e5b9be Allow to specify the random number generator seed in unit tests (#1751)
In this commit, we add a new parameter to the unit tests binary, called
`--seed`, that allows to specify the seed used by the several random
numbers generators used in the Valkey server code.

We also now print the seed information right before starting the unit
tests execution, so that someone that needs to reproduce an error, can
use the same seed to re-run the failed tests.

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
2025-02-20 17:20:43 +01:00
Nikhil MangloreandGitHub 8af9ce3733 Pass null-terminated node ID for VM_RegisterClusterMessageReceiver and add test coverage (#1708)
* Pass `sender_id` as `NULL` terminated string as part of
`ValkeyModuleClusterMessageReceiver` for ease of usage by the module(s).

* Implement test coverage described in #1656 and ensures that nodes in
the cluster module properly acknowledge a "DING" message by sending a
"DONG" response.

Closes #1656.

---------

Signed-off-by: Nikhil Manglore <nmanglor@amazon.com>
2025-02-20 01:14:01 +01:00
b23e640d08 Fix hashtable memory leak and kvstore overhead_hashtable_lut assert (#1750)
When releasing a kvstore and its hashtables, if it is still rehashed and
table 0 has no data, it may still have allocated buckets that were not
released. This lead to allocated hashtable buckets being leaked. This
looked like missed statistics but it was actually a memory leak.

fix: https://github.com/valkey-io/valkey/issues/1657

---------

Signed-off-by: artikell <739609084@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-02-19 17:56:11 +01:00
Viktor SöderqvistandGitHub 54455e6a23 Comment out assert in kvstore for overhead lut (#1745)
Until we found out why the assert is flaky, let's comment it out. We can
live with this overhead lut being slightly wrong, but we don't want the
assert to abort the program.

Fixes #1657.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2025-02-17 13:28:45 +01:00
992c004a94 Fix raxRemove crash at memcpy() due to key size exceeds max Rax size (#1722)
Fix raxRemove crash at memcpy() (line 1181) due to key size exceeds
`RAX_NODE_MAX_SIZE`. Note that this could happen when key size was more
than 512MB if we allow it by increasing the default
`proto-max-bulk-len`. The crash could happen when we recompress the rax
after removing a key due to expiry or DEL while memcpy() merge the key
that exceed 512MB limit. While the counting phase has the size check,
the actual compress logic is missing it which lead to this crash.

---------

Signed-off-by: Ram Prasad Voleti <ramvolet@amazon.com>
Co-authored-by: Ram Prasad Voleti <ramvolet@amazon.com>
2025-02-17 10:24:17 +01:00
ranshidandGitHub 0b810d478a Explicitly use github arm runners for ARM release (#1742)
In the last 8.1.0-rc1 release attempt we identified that the cross
compilation on x86 for ARM is broken
(example:
https://github.com/valkey-io/valkey/actions/runs/13324096504/job/37225952044)

This PR sets use of arm runners during release binaries.

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2025-02-16 13:39:40 +02:00
Madelyn OlsonandGitHub 59b5b264ef Add a daily test running for ARM (#1738)
Use official arm github runners to verify the ARM build as part of daily run. Also, updated the daily notify job failure list to a natural yaml list.
---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2025-02-15 21:47:58 -08:00
Ray CaoandViktor Söderqvist 3a29793557 Modify parameter of clientMatchesFilter function. (#1733)
small change to [#1401](https://github.com/valkey-io/valkey/pull/1401/).
pass `clientFilter *` to clientMatchesFilter as suggest in
[comment](https://github.com/valkey-io/valkey/pull/1401/commits/6bc64ca5373a9c53a65529ac3999a87e327f7d03#r1875844273).

Signed-off-by: Ray Cao <zisong.cw@alibaba-inc.com>
2025-02-15 10:49:30 +01:00
BinbinandGitHub 154af094f9 Simplify isNodeAvailable function and add comments (#994)
Use getNodeReplicationOffset to replace the original logic
and add the corresponding annotations.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2025-02-15 10:49:50 +08:00
1195 changed files with 109504 additions and 43846 deletions
+8 -1
View File
@@ -14,6 +14,13 @@ exat = "exat"
optin = "optin"
smove = "smove"
Parth = "Parth" # seems like the spellchecker does not like it is similar to "Path"
nd = "nd"
[default]
extend-ignore-re = [
"SELECTed",
"WATCHed",
]
[type.c]
extend-ignore-re = [
@@ -33,7 +40,6 @@ extend-ignore-re = [
advices = "advices"
clen = "clen"
fle = "fle"
nd = "nd"
ot = "ot"
[type.tcl.extend-identifiers]
@@ -53,6 +59,7 @@ arange = "arange"
fo = "fo"
frst = "frst"
limite = "limite"
pathc = "pathc"
pn = "pn"
seeked = "seeked"
tre = "tre"
+1 -1
View File
@@ -2,7 +2,7 @@
name: Bug report
about: Help us improve by reporting a bug
title: '[BUG]'
labels: ''
labels: ['bug']
assignees: ''
---
+5 -5
View File
@@ -1,17 +1,17 @@
blank_issues_enabled: true
contact_links:
- name: Questions?
url: https://github.com/valkey-io/valkey/discussions
url: https://github.com/kv-io/kv/discussions
about: Ask and answer questions on GitHub Discussions.
- name: Chat with us on Discord?
url: https://discord.gg/zbcPa5umUB
about: We are on Discord!
- name: Chat with us on Matrix?
url: https://matrix.to/#/#valkey:matrix.org
url: https://matrix.to/#/#kv:matrix.org
about: We are on Matrix too!
- name: Chat with us on Slack?
url: https://join.slack.com/t/valkey-oss-developer/shared_invite/zt-2nxs51chx-EB9hu9Qdch3GMfRcztTSkQ
url: https://join.slack.com/t/kv-oss-developer/shared_invite/zt-2nxs51chx-EB9hu9Qdch3GMfRcztTSkQ
about: We are on Slack too!
- name: Documentation issue?
url: https://github.com/valkey-io/valkey-doc/issues
about: Report it on the valkey-doc repo.
url: https://github.com/kv-io/kv-doc/issues
about: Report it on the kv-doc repo.
+1
View File
@@ -1,6 +1,7 @@
name: Crash report
description: Submit a crash report
title: '[CRASH] <short description>'
labels: ['bug']
body:
- type: markdown
attributes:
+1 -1
View File
@@ -2,7 +2,7 @@
name: Feature request
about: Suggest a feature
title: '[NEW]'
labels: ''
labels: ['enhancement']
assignees: ''
---
+24
View File
@@ -0,0 +1,24 @@
---
name: Test failure
about: Report a failing or a flaky test
title: '[TEST-FAILURE] <short description>'
labels: ['test-failure']
assignees: ''
---
<!--Before submitting a test failure report, please check open issues to ensure this failure has not already been reported. -->
**Summary**
A short description of the failure.
**Failing test(s)**
- Test name:
- CI link(s):
**Error stack trace**
A relevant stack trace for the test failure.
> **Tip:** Copy and paste the full stack trace from the CI output into your issue, as CI links may expire over time.
@@ -1,44 +0,0 @@
name: Generate target matrix.
description: Matrix creation for building Valkey for different architectures and platforms.
inputs:
ref:
description: The commit, tag or branch of Valkey to checkout to determine what version to use.
required: true
outputs:
x86_64-build-matrix:
description: The x86_64 build matrix.
value: ${{ steps.set-matrix.outputs.x86matrix }}
arm64-build-matrix:
description: The arm64 build matrix.
value: ${{ steps.set-matrix.outputs.armmatrix }}
runs:
using: "composite"
steps:
- name: Checkout code for version check
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref }}
path: version-check
- name: Get targets
run: |
x86_arch=$(jq -c '[.linux_targets[] | select(.arch=="x86_64")]' .github/actions/generate-package-build-matrix/build-config.json)
x86_matrix=$(echo "{ \"distro\" : $x86_arch }" | jq -c .)
echo "X86_MATRIX=$x86_matrix" >> $GITHUB_ENV
arm_arch=$(jq -c '[.linux_targets[] | select(.arch=="arm64")]' .github/actions/generate-package-build-matrix/build-config.json)
arm_matrix=$(echo "{ \"distro\" : $arm_arch }" | jq -c .)
echo "ARM_MATRIX=$arm_matrix" >> $GITHUB_ENV
shell: bash
- id: set-matrix
run: |
echo $X86_MATRIX
echo $X86_MATRIX| jq .
echo "x86matrix=$X86_MATRIX" >> $GITHUB_OUTPUT
echo $ARM_MATRIX
echo $ARM_MATRIX| jq .
echo "armmatrix=$ARM_MATRIX" >> $GITHUB_OUTPUT
shell: bash
@@ -1,35 +0,0 @@
{
"linux_targets": [
{
"arch": "x86_64",
"target": "ubuntu-20.04",
"type": "deb",
"platform": "focal"
},
{
"arch": "x86_64",
"target": "ubuntu-22.04",
"type": "deb",
"platform": "jammy"
},
{
"arch": "x86_64",
"target": "ubuntu-24.04",
"type": "deb",
"platform": "noble"
},
{
"arch": "arm64",
"target": "ubuntu20.04",
"type": "deb",
"platform": "focal"
},
{
"arch": "arm64",
"target": "ubuntu22.04",
"type": "deb",
"platform": "jammy"
}
]
}
@@ -0,0 +1,20 @@
[
{
"duration": 180,
"keyspacelen": [3000000],
"data_sizes": [16,96],
"pipelines": [1, 10],
"clients": [1600],
"commands": [
"SET",
"GET"
],
"cluster_mode": false,
"tls_mode": false,
"warmup": 30,
"io-threads": [1,9],
"benchmark-threads": 90,
"server_cpu_range": "0-8",
"client_cpu_range": "96-191"
}
]
@@ -0,0 +1,20 @@
[
{
"duration": 180,
"keyspacelen": [3000000],
"data_sizes": [16,96],
"pipelines": [1, 10],
"clients": [1600],
"commands": [
"SET",
"GET"
],
"cluster_mode": false,
"tls_mode": false,
"warmup": 30,
"io-threads": [1,9],
"benchmark-threads": 90,
"server_cpu_range": "0-8",
"client_cpu_range": "144-191,48-95"
}
]
+30
View File
@@ -0,0 +1,30 @@
# KV Project Instructions
You are an expert code reviewer for the KV project. Provide helpful, constructive feedback on code quality, safety, and adherence to project standards.
## 1. Review Tone & Focus
- **Tone:** Be professional, direct, constructive, and empathetic.
- **Focus:** Critique the *code*, never the *person*.
- **Constructive:** Suggest improvements, explain *why*, provide examples.
## 2. Critical Checks
- **DCO:** **Flag missing** `Signed-off-by: Name <email>` in commits. Every commit needs it.
- **Security:** If PR fixes a security vulnerability, flag it: "Security fixes should be reported privately to security@hanzo.ai, not via public PRs."
## 3. Major Decision Detection
Flag PRs that appear to be "Technical Major Decisions" requiring TSC consensus:
- Fundamental changes to core datastructures
- New data structures or APIs
- Backward compatibility breaks
- New user-visible fields requiring long-term maintenance
- New external libraries affecting runtime behavior
**Action:** Comment mentioning **@core-team** that this appears to require TSC review and ask if consensus was reached in a linked Issue.
## 4. Documentation Reminder
If PR changes user-facing behavior (new commands, changed semantics, new config):
- **Remind** author that docs at [kv-doc](https://github.com/hanzoai/kv-doc) may need updating.
- **Suggest** linking PR to related Issue with "Fixes #xyz" pattern if applicable.
## 5. Governance Changes
**ANY change to `GOVERNANCE.md`** requires special attention - comment mentioning **@core-team** for review.
@@ -0,0 +1,48 @@
---
applyTo:
- "src/**/*.{c,h}"
- "kv.conf"
---
# KV Core Engine Review Standards
Apply these standards to core engine C code. Do NOT apply to `deps/` (vendored dependencies).
## 1. Code Style (from DEVELOPMENT_GUIDE.md)
- **Formatting:** Follow clang-format (4-space indent, no tabs, braces attached).
- **Comments:**
- C-style `/* ... */` for single or multi-line.
- C++ `//` only for single-line.
- Multi-line: align leading `*`, final `*/` on last text line.
- Document *why* code exists, not just *what*. Document all functions.
- **Line Length:** Keep below 90 characters when reasonable.
- **Types:** Use the boolean type for true/false values.
- **Static:** Use `static` for file-local functions.
## 2. Naming Conventions
- **Variables:** `snake_case` or lowercase (e.g. `cached_reply`, `keylen`).
- **Functions:** `camelCase` or `namespace_camelCase` (e.g. `createStringObject`, `IOJobQueue_isFull`).
- **Macros:** `UPPER_CASE` (e.g. `MAKE_CMD`).
- **Structures:** `camelCase` (e.g. `user`).
## 3. Safety & Correctness
- **Memory:** Strict check for buffer overflows and leaks.
- **Strings:** Validate `sds` string handling.
- **Concurrency:** Verify thread safety in threaded I/O paths.
## 4. Design Guidelines
- **Configuration:** Avoid new configs if heuristics suffice. Only add for explicit trade-offs (CPU vs memory).
- **Metrics:** No new metrics on hot paths without zero-overhead proof.
- **PR Scope:** Separate refactoring from functional changes for easier backporting.
## 5. Testing & Documentation
- **Unit Tests:** Required for data structures in `src/unit/`. Test files should follow `test_*.c` naming.
- **Integration Tests:** Required for commands in `tests/`.
- **Command Changes:** New/modified commands need corresponding updates in `src/commands/*.json`.
- **New C Files:** Remind to update `CMakeLists.txt` when adding new `.c` source files.
- **License:** New files need BSD-3-Clause header. Material changes (>100 lines) also require it.
- **Documentation:** User-facing changes need docs at [kv-doc](https://github.com/hanzoai/kv-doc).
## 6. Critical Escalation
- **Trigger:** Changes to `cluster*.c`, `replication.c`, `rdb.c`, `aof.c`.
- **Action:** Comment mentioning **@core-team** to request architectural review.
@@ -0,0 +1,23 @@
---
applyTo:
- "tests/**/*.tcl"
---
# KV Integration Test Review Standards
Apply these standards to Tcl-based integration tests (from DEVELOPMENT_GUIDE.md).
## 1. Test Organization
- **Cluster:** Use `tests/unit/cluster/` (NOT legacy `tests/cluster/` which is deprecated).
- **Coverage:** All contributions should include tests. New commands require integration tests.
- **Naming:** Use descriptive test names that explain what is being tested.
## 2. Test Quality
- **Isolation:** Tests should not depend on execution order.
- **Cleanup:** Ensure proper cleanup of resources and temporary files.
- **Assertions:** Use clear assertions with meaningful error messages.
## 3. Best Practices
- **Readability:** Keep tests simple and focused on one scenario.
- **Reliability:** Avoid timing-dependent tests; use proper synchronization.
- **Documentation:** Comment complex test scenarios to explain intent.
@@ -0,0 +1,24 @@
---
applyTo:
- "utils/**/*"
---
# KV Utilities Review Standards
Apply these standards to utility scripts and tools.
## 1. Script Quality
- **Portability:** Scripts should work across common platforms (Linux, macOS).
- **Error Handling:** Check for errors and provide clear error messages.
- **Documentation:** Include usage instructions in comments or help text.
## 2. Code Standards
- **Python:** Follow PEP 8 style guidelines.
- **Ruby:** Follow standard Ruby conventions.
- **Shell:** Use shellcheck-compatible patterns.
- **C Tools:** Follow LLVM style (4-space indent, no tabs).
## 3. Best Practices
- **Dependencies:** Minimize external dependencies.
- **Safety:** Validate inputs and avoid destructive operations without confirmation.
- **Maintainability:** Keep utilities simple and well-commented.
+14
View File
@@ -0,0 +1,14 @@
name: Auto Author Assign
on:
pull_request_target:
types: [opened, reopened]
permissions:
pull-requests: write
jobs:
assign-author:
runs-on: ubuntu-latest
steps:
- uses: toshimaru/auto-author-assign@4d585cc37690897bd9015942ed6e766aa7cdb97f # v3.0.1
+173
View File
@@ -0,0 +1,173 @@
name: On-Demand Labeled Benchmark
on:
pull_request_target:
types: [labeled]
concurrency:
group: ec2-al-2023-pr-benchmarking-arm64
cancel-in-progress: false
defaults:
run:
shell: "bash -Eeuo pipefail -x {0}"
permissions:
contents: read
pull-requests: write
issues: write
jobs:
benchmark:
if: |
github.event.action == 'labeled' && github.event.label.name == 'run-benchmark' &&
github.repository == 'kv-io/kv'
runs-on: ["self-hosted", "ec2-al-2023-pr-benchmarking-arm64"]
timeout-minutes: 7200
steps:
- name: Checkout kv
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
path: kv
fetch-depth: 0
ref: ${{ github.event.pull_request.merge_commit_sha }}
persist-credentials: false
- name: Checkout kv-perf-benchmark
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ github.repository_owner }}/kv-perf-benchmark
path: kv-perf-benchmark
fetch-depth: 1
persist-credentials: false
- name: Checkout kv for latest benchmark.
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: kv-io/kv
ref: "unstable"
path: kv_latest
fetch-depth: 0
persist-credentials: false
- name: Set up Python
uses: kishaningithub/setup-python-amazon-linux@a326cdc792983fe0fbd04c81d3d62b59b6123a6c # v1.1.0
with:
python-version: "3.10"
cache: "pip"
- name: Install dependencies
working-directory: kv-perf-benchmark
run: |
sudo dnf groupinstall "Development Tools" -y
sudo dnf install -y gcc gcc-c++ make \
python3-devel \
openssl-devel \
bzip2-devel \
libffi-devel
pip install -r requirements.txt
- name: Build latest kv_latest
working-directory: kv_latest
run: |
echo "Building latest kv-benchmark for latest benchmark executable..."
make distclean || true
make -j
if [[ -f "src/kv-benchmark" ]]; then
echo "Successfully built latest kv-benchmark"
ls -la src/kv-benchmark
./src/kv-benchmark --version || echo "Version check completed"
else
echo "Failed to build kv-benchmark"
exit 1
fi
KV_BENCHMARK_PATH="$(pwd)/src/kv-benchmark"
echo "KV_BENCHMARK_PATH=$KV_BENCHMARK_PATH" >> $GITHUB_ENV
echo "Latest kv-benchmark path: $KV_BENCHMARK_PATH"
- name: Run benchmarks
working-directory: kv-perf-benchmark
run: |
CONFIG_FILE="../kv/.github/benchmark_configs/benchmark-config-arm.json"
# Base benchmark arguments
BENCHMARK_ARGS=(
--config "$CONFIG_FILE"
--commits "${{ github.event.pull_request.merge_commit_sha }}"
--baseline "${{ github.event.pull_request.base.ref }}"
--kv-benchmark-path "$KV_BENCHMARK_PATH"
--target-ip ${{ secrets.EC2_ARM64_IP }}
--kv-path "../kv"
--results-dir "results"
--runs 5
)
# Run benchmark
python ./benchmark.py "${BENCHMARK_ARGS[@]}"
- name: Compare results
working-directory: kv-perf-benchmark
run: |
python ./utils/compare_benchmark_results.py \
--baseline ./results/${{ github.event.pull_request.base.ref }}/metrics.json \
--new ./results/${{ github.event.pull_request.merge_commit_sha }}/metrics.json \
--output ../comparison.md \
--metrics rps
- name: Upload artifacts
if: always()
continue-on-error: true
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: benchmark-results
path: |
./kv-perf-benchmark/results/${{ github.event.pull_request.merge_commit_sha }}/metrics.json
./kv-perf-benchmark/results/${{ github.event.pull_request.base.ref }}/metrics.json
comparison.md
- name: Comment PR with results
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const fs = require('fs');
const body = fs.readFileSync('comparison.md', 'utf8');
const {owner, repo} = context.repo;
const sha = '${{ github.event.pull_request.head.sha }}';
const short = sha.slice(0,7);
const link = `[\`${short}\`](https://github.com/${owner}/${repo}/commit/${sha})`
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner,
repo,
body: `**Benchmark ran on this commit:** ${link}\n\n${body}`
});
- name: Cleanup any running kv processes
if: always()
continue-on-error: true
run: |
rm -rf comparison.md kv*
pkill -f kv
exit_code=$?
if [ $exit_code -eq 0 ]; then
echo "Killed running kv processes"
elif [ $exit_code -eq 1 ]; then
echo "No kv processes found to kill"
else
echo "Warning: pkill failed with exit code $exit_code"
fi
- name: Remove ${{ github.event.label.name }} label
if: always() && github.event.label.name == 'run-benchmark'
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
await github.rest.issues.removeLabel({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
name: '${{ github.event.label.name }}'
}).catch(err => console.log('label not present', err.message));
+288
View File
@@ -0,0 +1,288 @@
name: Compare KV Versions
on:
workflow_dispatch:
inputs:
version1:
description: "First version to compare (commit SHA, branch, or tag)"
required: true
type: string
version2:
description: "Second version to compare (commit SHA, branch, or tag)"
required: true
type: string
issue_id:
description: "Issue ID to comment results on"
required: true
type: string
runs:
description: "Number of benchmark runs per configuration"
required: false
type: number
default: 1
defaults:
run:
shell: "bash -Eeuo pipefail -x {0}"
permissions:
contents: read
pull-requests: write
issues: write
jobs:
benchmark:
if: github.repository == 'kv-io/kv'
strategy:
matrix:
include:
- arch: x86
machine: ec2-al-2023-pr-benchmarking-x86
config: benchmark-config-x86.json
- arch: arm64
machine: ec2-al-2023-pr-benchmarking-arm64
config: benchmark-config-arm.json
concurrency:
group: ${{ matrix.machine }}
cancel-in-progress: false
runs-on: ["self-hosted", "${{ matrix.machine }}"]
timeout-minutes: 7200
steps:
- name: Validate inputs
run: |
echo "Version 1: ${{ github.event.inputs.version1 }}"
echo "Version 2: ${{ github.event.inputs.version2 }}"
echo "Issue ID: ${{ github.event.inputs.issue_id }}"
echo "Architecture: ${{ matrix.arch }}"
echo "Config: ${{ matrix.config }}"
echo "Runs: ${{ github.event.inputs.runs }}"
# Validate issue ID is numeric
if ! [[ "${{ github.event.inputs.issue_id }}" =~ ^[0-9]+$ ]]; then
echo "Error: Issue ID must be a number"
exit 1
fi
# Validate runs is a positive number
if ! [[ "${{ github.event.inputs.runs }}" =~ ^[1-9][0-9]*$ ]]; then
echo "Error: Runs must be a positive number"
exit 1
fi
- name: Checkout kv for latest benchmark.
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: kv-io/kv
ref: "unstable"
path: kv_latest
fetch-depth: 0
persist-credentials: false
- name: Checkout kv
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
path: kv
- name: Checkout kv-perf-benchmark
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ github.repository_owner }}/kv-perf-benchmark
path: kv-perf-benchmark
fetch-depth: 1
persist-credentials: false
- name: Set up Python
uses: kishaningithub/setup-python-amazon-linux@a326cdc792983fe0fbd04c81d3d62b59b6123a6c # v1.1.0
with:
python-version: "3.10"
cache: "pip"
- name: Install dependencies
working-directory: kv-perf-benchmark
run: |
sudo dnf groupinstall "Development Tools" -y
sudo dnf install -y gcc gcc-c++ make \
python3-devel \
openssl-devel \
bzip2-devel \
libffi-devel
pip install -r requirements.txt
- name: Build latest kv_latest
working-directory: kv_latest
run: |
echo "Building latest kv-benchmark for latest benchmark executable..."
# Clean any previous builds
make distclean || true
# Build kv-benchmark with latest code
make -j$(nproc)
# Verify the binary was created
if [[ -f "src/kv-benchmark" ]]; then
echo "✓ Successfully built latest kv-benchmark"
ls -la src/kv-benchmark
# Test the binary
echo "Testing kv-benchmark binary..."
./src/kv-benchmark --version || echo "Version check completed"
else
echo "Failed to build kv-benchmark"
exit 1
fi
# Store the absolute path for later use
KV_BENCHMARK_PATH="$(pwd)/src/kv-benchmark"
echo "KV_BENCHMARK_PATH=$KV_BENCHMARK_PATH" >> $GITHUB_ENV
echo "Latest kv-benchmark path: $KV_BENCHMARK_PATH"
- name: Run benchmarks
working-directory: kv-perf-benchmark
env:
EC2_X86_IP: ${{ secrets.EC2_X86_IP }}
EC2_ARM64_IP: ${{ secrets.EC2_ARM64_IP }}
run: |
# Set the target IP based on the matrix architecture
if [[ "${{ matrix.arch }}" == "x86" ]]; then
TARGET_IP=$EC2_X86_IP
echo "Using x86 machine IP"
elif [[ "${{ matrix.arch }}" == "arm64" ]]; then
TARGET_IP=$EC2_ARM64_IP
echo "Using ARM64 machine IP"
else
echo "Error: Unknown architecture: ${{ matrix.arch }}"
exit 1
fi
CONFIG_FILE="../kv/.github/benchmark_configs/${{ matrix.config }}"
# Verify our custom kv-benchmark exists
if [[ ! -f "$KV_BENCHMARK_PATH" ]]; then
echo "Custom kv-benchmark not found at: $KV_BENCHMARK_PATH"
exit 1
fi
echo "Using custom kv-benchmark from: $KV_BENCHMARK_PATH"
# Base benchmark arguments with custom kv-benchmark path
BENCHMARK_ARGS=(
--config "$CONFIG_FILE"
--commits ${{ github.event.inputs.version1 }} ${{ github.event.inputs.version2 }}
--kv-benchmark-path "$KV_BENCHMARK_PATH"
--target-ip $TARGET_IP
--results-dir "results"
--runs ${{ github.event.inputs.runs }}
)
echo "Running benchmarks with the following setup:"
echo "- Version 1: ${{ github.event.inputs.version1 }}"
echo "- Version 2: ${{ github.event.inputs.version2 }}"
echo "- Architecture: ${{ matrix.arch }}"
echo "- Config: ${{ matrix.config }}"
echo "- Using latest kv-benchmark: $KV_BENCHMARK_PATH"
echo "- This ensures both versions use the same (latest) benchmark tool for consistent results"
# Run benchmark with custom kv-benchmark executable
python ./benchmark.py "${BENCHMARK_ARGS[@]}"
- name: Compare results
working-directory: kv-perf-benchmark
run: |
# Find the actual result directories (they might have different names than input)
VERSION1_DIR=$(find ./results -maxdepth 1 -type d -name "*${{ github.event.inputs.version1 }}*" | head -1)
VERSION2_DIR=$(find ./results -maxdepth 1 -type d -name "*${{ github.event.inputs.version2 }}*" | head -1)
if [[ -z "$VERSION1_DIR" ]]; then
echo "Could not find results for version1: ${{ github.event.inputs.version1 }}"
echo "Available result directories:"
ls -la ./results/
exit 1
fi
if [[ -z "$VERSION2_DIR" ]]; then
echo "Could not find results for version2: ${{ github.event.inputs.version2 }}"
echo "Available result directories:"
ls -la ./results/
exit 1
fi
echo "Comparing results:"
echo "Version 1 (${{ github.event.inputs.version1 }}): $VERSION1_DIR"
echo "Version 2 (${{ github.event.inputs.version2 }}): $VERSION2_DIR"
# Generate RPS-focused comparison and graphs for GitHub comments
python utils/compare_benchmark_results.py \
--baseline "$VERSION1_DIR/metrics.json" \
--new "$VERSION2_DIR/metrics.json" \
--output ../comparison.md \
--metrics rps
- name: Upload artifacts
if: always()
continue-on-error: true
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: benchmark-results-${{ matrix.arch }}-${{ github.event.inputs.issue_id }}
path: |
./kv-perf-benchmark/results/*
comparison.md
- name: Cleanup any running kv processes and files
if: always()
continue-on-error: true
run: |
pkill -f kv || echo "No kv processes found to kill"
rm -rf *
combine-results:
needs: benchmark
runs-on: ubuntu-latest
steps:
- name: Download all artifacts
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
with:
path: artifacts
- name: Combine results and create comprehensive report
run: |
echo "# Multi-Architecture Benchmark Comparison: ${{ github.event.inputs.version1 }} vs ${{ github.event.inputs.version2 }}" > combined_report.md
echo "" >> combined_report.md
echo "**Versions Compared:**" >> combined_report.md
echo "- Version 1: \`${{ github.event.inputs.version1 }}\`" >> combined_report.md
echo "- Version 2: \`${{ github.event.inputs.version2 }}\`" >> combined_report.md
echo "" >> combined_report.md
echo "**Runs:** ${{ github.event.inputs.runs }} per configuration" >> combined_report.md
echo "**Workflow Run:** [View Details](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})" >> combined_report.md
echo "" >> combined_report.md
echo "---" >> combined_report.md
echo "" >> combined_report.md
# Process each architecture's results
for arch in x86 arm64; do
artifact_dir="artifacts/benchmark-results-${arch}-${{ github.event.inputs.issue_id }}"
if [[ -d "$artifact_dir" && -f "$artifact_dir/comparison.md" ]]; then
echo "## ${arch^^} Architecture Results" >> combined_report.md
echo "" >> combined_report.md
cat "$artifact_dir/comparison.md" >> combined_report.md
echo "" >> combined_report.md
echo "---" >> combined_report.md
echo "" >> combined_report.md
fi
done
- name: Comment on issue with combined results
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const fs = require('fs');
// Read the combined report
const body = fs.readFileSync('combined_report.md', 'utf8');
await github.rest.issues.createComment({
issue_number: ${{ github.event.inputs.issue_id }},
owner: context.repo.owner,
repo: context.repo.repo,
body: body
});
@@ -1,112 +0,0 @@
name: Build Release Packages
on:
release:
types: [published]
push:
paths:
- '.github/workflows/build-release-packages.yml'
- '.github/workflows/call-build-linux-arm-packages.yml'
- '.github/workflows/call-build-linux-x86-packages.yml'
- '.github/actions/generate-package-build-matrix/build-config.json'
workflow_dispatch:
inputs:
version:
description: Version of Valkey to build
required: true
permissions:
id-token: write
contents: read
jobs:
# This job provides the version metadata from the tag for the other jobs to use.
release-build-get-meta:
name: Get metadata to build
if: github.event_name == 'workflow_dispatch' || github.repository == 'valkey-io/valkey'
runs-on: ubuntu-latest
outputs:
version: ${{ steps.get_version.outputs.VERSION }}
is_test: ${{ steps.check-if-testing.outputs.IS_TEST }}
steps:
- run: |
echo "Version: ${{ inputs.version || github.ref_name }}"
shell: bash
# This step is to consolidate the three different triggers into a single "version"
# 1. If manual dispatch - use the version provided.
# 3. If tag trigger, use that tag.
- name: Get the version
id: get_version
run: |
if [[ "${{ github.event_name }}" == "push" ]]; then
VERSION=${{ github.ref_name }}
else
VERSION="${INPUT_VERSION}"
fi
if [ -z "${VERSION}" ]; then
echo "Error: No version specified"
exit 1
fi
echo "VERSION=$VERSION" >> $GITHUB_OUTPUT
shell: bash
env:
# Use the dispatch variable in preference, if empty use the context ref_name which should
# only ever be a tag
INPUT_VERSION: ${{ inputs.version || github.ref_name }}
- name: Check if we are testing
id: check-if-testing
run: |
if [[ "${{ github.event_name }}" == "push" ]]; then
echo "This is a test workflow -> We will upload to the Test S3 Bucket"
echo "IS_TEST=true" >> $GITHUB_OUTPUT
else
echo "This is a Release workflow -> We will upload to the Release S3 Bucket"
echo "IS_TEST=false" >> $GITHUB_OUTPUT
fi
shell: bash
generate-build-matrix:
name: Generating build matrix
if: github.event_name == 'workflow_dispatch' || github.repository == 'valkey-io/valkey'
runs-on: ubuntu-latest
outputs:
x86_64-build-matrix: ${{ steps.set-matrix.outputs.x86_64-build-matrix }}
arm64-build-matrix: ${{ steps.set-matrix.outputs.arm64-build-matrix }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
# Set up the list of target to build so we can pass the JSON to the reusable job
- uses: ./.github/actions/generate-package-build-matrix
id: set-matrix
with:
ref: ${{ needs.release-build-get-meta.outputs.version }}
release-build-linux-x86-packages:
needs:
- release-build-get-meta
- generate-build-matrix
uses: ./.github/workflows/call-build-linux-x86-packages.yml
with:
version: ${{ needs.release-build-get-meta.outputs.version }}
ref: ${{ inputs.version || github.ref_name }}
build_matrix: ${{ needs.generate-build-matrix.outputs.x86_64-build-matrix }}
region: us-west-2
secrets:
bucket_name: ${{ needs.release-build-get-meta.outputs.is_test == 'true' && secrets.AWS_S3_TEST_BUCKET || secrets.AWS_S3_BUCKET }}
role_to_assume: ${{ secrets.AWS_ROLE_TO_ASSUME }}
release-build-linux-arm-packages:
needs:
- release-build-get-meta
- generate-build-matrix
uses: ./.github/workflows/call-build-linux-arm-packages.yml
with:
version: ${{ needs.release-build-get-meta.outputs.version }}
ref: ${{ inputs.version || github.ref_name }}
build_matrix: ${{ needs.generate-build-matrix.outputs.arm64-build-matrix }}
region: us-west-2
secrets:
bucket_name: ${{ needs.release-build-get-meta.outputs.is_test == 'true' && secrets.AWS_S3_TEST_BUCKET || secrets.AWS_S3_BUCKET }}
role_to_assume: ${{ secrets.AWS_ROLE_TO_ASSUME }}
@@ -1,74 +0,0 @@
name: Builds Linux arm binary packages into S3 bucket.
on:
workflow_call:
inputs:
version:
description: The version of Valkey to create.
type: string
required: true
ref:
description: The commit, tag or branch of Valkey to checkout for building that creates the version above.
type: string
required: true
build_matrix:
description: The build targets to produce as a JSON matrix.
type: string
required: true
region:
description: The AWS region to push packages into.
type: string
required: true
secrets:
bucket_name:
description: The S3 bucket to push packages into.
required: true
role_to_assume:
description: The role to assume for the S3 bucket.
required: true
permissions:
id-token: write
contents: read
jobs:
build-valkey:
# Capture source tarball and generate checksum for it
name: Build package ${{ matrix.distro.target }} ${{ matrix.distro.arch }}
runs-on: "ubuntu-latest"
strategy:
fail-fast: false
matrix: ${{ fromJSON(inputs.build_matrix) }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
ref: ${{ inputs.version }}
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ inputs.region }}
role-to-assume: ${{ secrets.role_to_assume }}
- name: Make Valkey
uses: uraimo/run-on-arch-action@v2
with:
arch: aarch64
distro: ${{matrix.distro.target}}
install: apt-get update && apt-get install -y build-essential libssl-dev libsystemd-dev
run: make -C src all BUILD_TLS=yes USE_SYSTEMD=yes
- name: Create Tarball and SHA256sums
run: |
TAR_FILE_NAME=valkey-${{inputs.version}}-${{matrix.distro.platform}}-${{ matrix.distro.arch}}
mkdir -p "$TAR_FILE_NAME/bin" "$TAR_FILE_NAME/share"
rsync -av --exclude='*.c' --exclude='*.d' --exclude='*.o' src/valkey-* "$TAR_FILE_NAME/bin/"
cp -v /home/runner/work/valkey/valkey/COPYING "$TAR_FILE_NAME/share/LICENSE"
tar -czvf $TAR_FILE_NAME.tar.gz $TAR_FILE_NAME
sha256sum $TAR_FILE_NAME.tar.gz > $TAR_FILE_NAME.tar.gz.sha256
mkdir -p packages-files
cp -rfv $TAR_FILE_NAME.tar* packages-files/
- name: Sync to S3
run: aws s3 sync packages-files s3://${{ secrets.bucket_name }}/releases/
@@ -1,72 +0,0 @@
name: Builds Linux X86 binary packages into S3 bucket.
on:
workflow_call:
inputs:
version:
description: The version of Valkey to create.
type: string
required: true
ref:
description: The commit, tag or branch of Valkey to checkout for building that creates the version above.
type: string
required: true
build_matrix:
description: The build targets to produce as a JSON matrix.
type: string
required: true
region:
description: The AWS region to upload the packages to.
type: string
required: true
secrets:
bucket_name:
description: The name of the S3 bucket to upload the packages to.
required: true
role_to_assume:
description: The role to assume for the S3 bucket.
required: true
permissions:
id-token: write
contents: read
jobs:
build-valkey:
# Capture source tarball and generate checksum for it
name: Build package ${{ matrix.distro.target }} ${{ matrix.distro.arch }}
runs-on: ${{matrix.distro.target}}
strategy:
fail-fast: false
matrix: ${{ fromJSON(inputs.build_matrix) }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
ref: ${{ inputs.version }}
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ inputs.region }}
role-to-assume: ${{ secrets.role_to_assume }}
- name: Install dependencies
run: sudo apt-get update && sudo apt-get install -y build-essential libssl-dev libsystemd-dev
- name: Make Valkey
run: make -C src all BUILD_TLS=yes USE_SYSTEMD=yes
- name: Create Tarball and SHA256sums
run: |
TAR_FILE_NAME=valkey-${{inputs.version}}-${{matrix.distro.platform}}-${{ matrix.distro.arch}}
mkdir -p "$TAR_FILE_NAME/bin" "$TAR_FILE_NAME/share"
rsync -av --exclude='*.c' --exclude='*.d' --exclude='*.o' src/valkey-* "$TAR_FILE_NAME/bin/"
cp -v /home/runner/work/valkey/valkey/COPYING "$TAR_FILE_NAME/share/LICENSE"
tar -czvf $TAR_FILE_NAME.tar.gz $TAR_FILE_NAME
sha256sum $TAR_FILE_NAME.tar.gz > $TAR_FILE_NAME.tar.gz.sha256
mkdir -p packages-files
cp -rfv $TAR_FILE_NAME.tar* packages-files/
- name: Sync to S3
run: aws s3 sync packages-files s3://${{ secrets.bucket_name }}/releases/
+178 -48
View File
@@ -1,6 +1,16 @@
name: CI
on: [push, pull_request]
on:
push:
paths-ignore:
- '**/*.md'
- '**/00-RELEASENOTES'
- '**/COPYING'
pull_request:
paths-ignore:
- '**/*.md'
- '**/00-RELEASENOTES'
- '**/COPYING'
concurrency:
group: ci-${{ github.head_ref || github.ref }}
@@ -13,36 +23,81 @@ jobs:
test-ubuntu-latest:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
# Fail build if there are warnings
# build with TLS just for compilation coverage
run: make -j4 all-with-unit-tests SERVER_CFLAGS='-Werror' BUILD_TLS=yes USE_FAST_FLOAT=yes
- name: install old server for compatibility testing
run: |
cd tests/tmp
wget https://download.valkey.io/releases/valkey-7.2.7-noble-x86_64.tar.gz
tar -xvf valkey-7.2.7-noble-x86_64.tar.gz
run: make -j4 all-with-unit-tests SERVER_CFLAGS='-Werror' BUILD_TLS=yes USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
- name: test
run: |
sudo apt-get install tcl8.6 tclx
./runtest --verbose --tags -slow --dump-logs --other-server-path tests/tmp/valkey-7.2.7-noble-x86_64/bin/valkey-server
./runtest --verbose --tags -slow --dump-logs
- name: module api test
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs --other-server-path tests/tmp/valkey-7.2.7-noble-x86_64/bin/valkey-server
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs
- name: validate commands.def up to date
run: |
touch src/commands/ping.json
make commands.def
dirty=$(git diff)
if [[ ! -z $dirty ]]; then echo $dirty; exit 1; fi
dirty="$(git diff)"
if [[ ! -z "$dirty" ]]; then echo "$dirty"; exit 1; fi
- name: unit tests
run: |
./src/valkey-unit-tests
./src/kv-unit-tests
test-ubuntu-latest-cmake:
test-ubuntu-latest-compatibility:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
server:
- {version: "7.2.11", file: "kv-7.2.11-noble-x86_64.tar.gz"}
- {version: "8.0.6", file: "kv-8.0.6-noble-x86_64.tar.gz"}
- {version: "8.1.4", file: "kv-8.1.4-noble-x86_64.tar.gz"}
steps:
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
# Fail build if there are warnings
# build with TLS just for compilation coverage
run: make -j4 all-with-unit-tests SERVER_CFLAGS='-Werror' BUILD_TLS=yes USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
- name: Install old server (${{ matrix.server.version }}) for compatibility testing
run: |
mkdir -p tests/tmp
cd tests/tmp
wget https://download.kv.io/releases/${{ matrix.server.file }}
tar -xvf ${{ matrix.server.file }}
- name: Run compatibility tests against ${{ matrix.server.version }}
run: |
sudo apt-get install -y tcl8.6 tclx
./runtest --verbose --tags "-slow needs:other-server" --dump-logs \
--other-server-path tests/tmp/kv-${{ matrix.server.version }}-noble-x86_64/bin/kv-server
- name: Module API tests against ${{ matrix.server.version }}
run: |
CFLAGS='-Werror' ./runtest-moduleapi --tags needs:other-server --verbose --dump-logs \
--other-server-path tests/tmp/kv-${{ matrix.server.version }}-noble-x86_64/bin/kv-server
test-ubuntu-latest-cmake-tls:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: cmake and make
run: |
sudo apt-get install -y cmake libssl-dev
@@ -52,25 +107,28 @@ jobs:
make -j$(nproc)
- name: test
run: |
sudo apt-get install -y tcl8.6 tclx
ln -sf $(pwd)/build-release/bin/valkey-server $(pwd)/src/valkey-server
ln -sf $(pwd)/build-release/bin/valkey-cli $(pwd)/src/valkey-cli
ln -sf $(pwd)/build-release/bin/valkey-benchmark $(pwd)/src/valkey-benchmark
ln -sf $(pwd)/build-release/bin/valkey-server $(pwd)/src/valkey-check-aof
ln -sf $(pwd)/build-release/bin/valkey-server $(pwd)/src/valkey-check-rdb
ln -sf $(pwd)/build-release/bin/valkey-server $(pwd)/src/valkey-sentinel
./runtest --verbose --tags -slow --dump-logs
sudo apt-get install -y tcl8.6 tclx tcl-tls
./utils/gen-test-certs.sh
./build-release/runtest --verbose --tags -slow --dump-logs --tls
- name: unit tests
run: |
./build-release/bin/valkey-unit-tests
./build-release/bin/kv-unit-tests
test-sanitizer-address:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
# build with TLS module just for compilation coverage
run: make -j4 all-with-unit-tests SANITIZER=address SERVER_CFLAGS='-Werror' BUILD_TLS=module
run: make -j4 all-with-unit-tests SANITIZER=address SERVER_CFLAGS='-Werror' BUILD_TLS=module USE_LIBBACKTRACE=yes
- name: testprep
run: sudo apt-get install tcl8.6 tclx -y
- name: test
@@ -78,20 +136,28 @@ jobs:
- name: module api test
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs
- name: unit tests
run: ./src/valkey-unit-tests
run: ./src/kv-unit-tests
test-rdma:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: prepare-development-libraries
run: sudo apt-get install librdmacm-dev libibverbs-dev
- name: make-rdma-module
run: make -j4 BUILD_RDMA=module
run: make -j4 BUILD_RDMA=module USE_LIBBACKTRACE=yes
- name: make-rdma-builtin
run: |
make distclean
make -j4 BUILD_RDMA=yes
make -j4 BUILD_RDMA=yes USE_LIBBACKTRACE=yes
- name: clone-rxe-kmod
run: |
mkdir -p tests/rdma/rxe
@@ -104,65 +170,129 @@ jobs:
- name: show-kernel-log
run: sudo dmesg -c
test-tls-only:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: generate-test-certificates
run: ./utils/gen-test-certs.sh
- name: install-test-dependencies
run: sudo apt-get install -y tcl8.6 tclx tcl-tls
- name: make-tls-module
run: make -j4 BUILD_TLS=module SERVER_CFLAGS='-Werror'
- name: test-tls-module
run: ./runtest --verbose --single unit/tls --dump-logs --tls-module
- name: make-tls-builtin
run: |
make distclean
make -j4 BUILD_TLS=yes SERVER_CFLAGS='-Werror'
- name: test-tls-builtin
run: ./runtest --verbose --single unit/tls --dump-logs --tls
build-debian-old:
runs-on: ubuntu-latest
container: debian:buster
container: debian:bullseye
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: make
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- name: Build libbacktrace
run: |
apt-get update && apt-get install -y build-essential
make -j4 SERVER_CFLAGS='-Werror'
cd libbacktrace && ./configure && make && make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
run: make -j4 SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes
build-macos-latest:
runs-on: macos-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
# Build with additional upcoming features
run: make -j3 all-with-unit-tests SERVER_CFLAGS='-Werror' USE_FAST_FLOAT=yes
run: make -j3 all-with-unit-tests SERVER_CFLAGS='-Werror' USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
build-32bit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- name: Build libbacktrace (32-bit)
run: |
sudo apt-get update
sudo apt-get install libc6-dev-i386 libstdc++-11-dev-i386-cross gcc-multilib g++-multilib
cd libbacktrace && ./configure CFLAGS="-m32" --prefix=/usr/local/libbacktrace32 && make && sudo make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
# Fast float requires C++ 32-bit libraries to compile on 64-bit ubuntu
# machine i.e. "-cross" suffixed version. Cross-compiling c++ to 32-bit
# also requires multilib support for g++ compiler i.e. "-multilib"
# suffixed version of g++. g++-multilib generally includes libstdc++.
# *cross version as well, but it is also added explicitly just in case.
run: make -j4 SERVER_CFLAGS='-Werror' 32bit USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes LIBBACKTRACE_PREFIX=/usr/local/libbacktrace32
- name: unit tests
run: |
sudo apt-get update
sudo apt-get install libc6-dev-i386 libstdc++-11-dev-i386-cross gcc-multilib g++-multilib
make -j4 SERVER_CFLAGS='-Werror' 32bit USE_FAST_FLOAT=yes
./src/kv-unit-tests
build-libc-malloc:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
run: make -j4 SERVER_CFLAGS='-Werror' MALLOC=libc USE_FAST_FLOAT=yes
run: make -j4 SERVER_CFLAGS='-Werror' MALLOC=libc USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
build-almalinux8-jemalloc:
runs-on: ubuntu-latest
container: almalinux:8
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: make
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- name: Build libbacktrace
run: |
dnf -y install epel-release gcc gcc-c++ make procps-ng which
make -j4 SERVER_CFLAGS='-Werror' USE_FAST_FLOAT=yes
cd libbacktrace && ./configure && make && make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
run: make -j4 SERVER_CFLAGS='-Werror' USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
format-yaml:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up Go
uses: actions/setup-go@cdcb36043654635271a94b9a6d1392de5bb323a7 # v5.0.1
uses: actions/setup-go@7a3fe6cf4cb3a834922a1244abfce67bcef6a0c5 # v6.2.0
with:
go-version: "1.22.4"
+9 -3
View File
@@ -2,9 +2,15 @@ name: Clang Format Check
on:
push:
paths-ignore:
- '**/*.md'
- '**/00-RELEASENOTES'
- '**/COPYING'
pull_request:
paths:
- 'src/**'
paths-ignore:
- '**/*.md'
- '**/00-RELEASENOTES'
- '**/COPYING'
concurrency:
group: clang-${{ github.head_ref || github.ref }}
@@ -16,7 +22,7 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up Clang
run: |
+24 -8
View File
@@ -2,7 +2,17 @@ name: "Codecov"
# Enabling on each push is to display the coverage changes in every PR,
# where each PR needs to be compared against the coverage of the head commit
on: [push, pull_request]
on:
push:
paths-ignore:
- '**/*.md'
- '**/00-RELEASENOTES'
- '**/COPYING'
pull_request:
paths-ignore:
- '**/*.md'
- '**/00-RELEASENOTES'
- '**/COPYING'
concurrency:
group: codecov-${{ github.head_ref || github.ref }}
@@ -13,16 +23,22 @@ jobs:
runs-on: ubuntu-22.04
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install lcov and run test
run: |
sudo apt-get install lcov
make lcov
sudo apt-get install lcov tclx
make lcov USE_LIBBACKTRACE=yes
- name: Upload code coverage
uses: codecov/codecov-action@v4
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2
with:
token: ${{ secrets.CODECOV_TOKEN }}
file: ./src/valkey.info
file: ./src/kv.info
+14 -5
View File
@@ -1,7 +1,16 @@
name: "CodeQL"
on:
push:
paths-ignore:
- '**/*.md'
- '**/00-RELEASENOTES'
- '**/COPYING'
pull_request:
paths-ignore:
- '**/*.md'
- '**/00-RELEASENOTES'
- '**/COPYING'
schedule:
# run weekly new vulnerability was added to the database
- cron: '0 3 * * 0'
@@ -17,7 +26,7 @@ jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
if: github.event_name != 'schedule' || github.repository == 'valkey-io/valkey'
if: github.event_name != 'schedule' || github.repository == 'kv-io/kv'
permissions:
security-events: write
@@ -28,15 +37,15 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Initialize CodeQL
uses: github/codeql-action/init@1b1aada464948af03b950897e5eb522f92603cc2 # v3.24.9
uses: github/codeql-action/init@b5ebac6f4c00c8ccddb7cdcd45fdb248329f808a # v3.32.2
with:
languages: ${{ matrix.language }}
- name: Autobuild
uses: github/codeql-action/autobuild@1b1aada464948af03b950897e5eb522f92603cc2 # v3.24.9
uses: github/codeql-action/autobuild@b5ebac6f4c00c8ccddb7cdcd45fdb248329f808a # v3.32.2
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@1b1aada464948af03b950897e5eb522f92603cc2 # v3.24.9
uses: github/codeql-action/analyze@b5ebac6f4c00c8ccddb7cdcd45fdb248329f808a # v3.32.2
+14 -6
View File
@@ -16,19 +16,27 @@ permissions:
jobs:
coverity:
if: github.repository == 'valkey-io/valkey'
if: github.repository == 'kv-io/kv'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Download and extract the Coverity Build Tool
run: |
wget -q https://scan.coverity.com/download/cxx/linux64 --post-data "token=${{ secrets.COVERITY_SCAN_TOKEN }}&project=valkey-io%2Fvalkey" -O cov-analysis-linux64.tar.gz
wget -q https://scan.coverity.com/download/cxx/linux64 --post-data "token=${{ secrets.COVERITY_SCAN_TOKEN }}&project=kv-io%2Fkv" -O cov-analysis-linux64.tar.gz
mkdir cov-analysis-linux64
tar xzf cov-analysis-linux64.tar.gz --strip 1 -C cov-analysis-linux64
- name: Install Valkey dependencies
- name: Install KV dependencies
run: sudo apt install -y gcc procps libssl-dev
- name: Build with cov-build
run: cov-analysis-linux64/bin/cov-build --dir cov-int make
run: cov-analysis-linux64/bin/cov-build --dir cov-int make USE_LIBBACKTRACE=yes
- name: Upload the result
run: |
tar czvf cov-int.tgz cov-int
@@ -36,4 +44,4 @@ jobs:
--form email=${{ secrets.COVERITY_SCAN_EMAIL }} \
--form token=${{ secrets.COVERITY_SCAN_TOKEN }} \
--form file=@cov-int.tgz \
https://scan.coverity.com/builds?project=valkey-io%2Fvalkey
https://scan.coverity.com/builds?project=kv-io%2Fkv
File diff suppressed because it is too large Load Diff
+58
View File
@@ -0,0 +1,58 @@
name: Build and Deploy Hanzo KV
on:
push:
branches: [main]
tags: ['v*']
paths:
- 'src/**'
- 'deps/**'
- 'Dockerfile'
- '.github/workflows/deploy.yml'
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Image metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/hanzoai/kv
tags: |
type=raw,value=latest,enable={{is_default_branch}}
type=raw,value=9,enable={{is_default_branch}}
type=sha,prefix=
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
- name: Build and push
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha,scope=kv
cache-to: type=gha,mode=max,scope=kv
+53 -20
View File
@@ -1,8 +1,16 @@
name: External Server Tests
on:
pull_request:
push:
paths-ignore:
- '**/*.md'
- '**/00-RELEASENOTES'
- '**/COPYING'
pull_request:
paths-ignore:
- '**/*.md'
- '**/00-RELEASENOTES'
- '**/COPYING'
schedule:
- cron: '0 2 * * *'
@@ -16,15 +24,23 @@ permissions:
jobs:
test-external-standalone:
runs-on: ubuntu-latest
if: github.event_name != 'schedule' || github.repository == 'valkey-io/valkey'
if: github.event_name != 'schedule' || github.repository == 'kv-io/kv'
timeout-minutes: 1440
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Build
run: make SERVER_CFLAGS=-Werror
- name: Start valkey-server
run: make SERVER_CFLAGS=-Werror USE_LIBBACKTRACE=yes
- name: Start kv-server
run: |
./src/valkey-server --daemonize yes --save "" --logfile external-server.log \
./src/kv-server --daemonize yes --save "" --logfile external-server.log \
--enable-protected-configs yes --enable-debug-command yes --enable-module-command yes
- name: Run external test
run: |
@@ -34,25 +50,34 @@ jobs:
--tags -slow
- name: Archive server log
if: ${{ failure() }}
uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: test-external-standalone-log
path: external-server.log
test-external-cluster:
runs-on: ubuntu-latest
if: github.event_name != 'schedule' || github.repository == 'valkey-io/valkey'
if: github.event_name != 'schedule' || github.repository == 'kv-io/kv'
timeout-minutes: 1440
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Build
run: make SERVER_CFLAGS=-Werror
- name: Start valkey-server
run: make SERVER_CFLAGS=-Werror USE_LIBBACKTRACE=yes
- name: Start kv-server
run: |
./src/valkey-server --cluster-enabled yes --daemonize yes --save "" --logfile external-server.log \
./src/kv-server --cluster-enabled yes --cluster-databases 16 --daemonize yes \
--save "" --logfile external-server.log \
--enable-protected-configs yes --enable-debug-command yes --enable-module-command yes
- name: Create a single node cluster
run: ./src/valkey-cli cluster addslots $(for slot in {0..16383}; do echo $slot; done); sleep 5
run: ./src/kv-cli cluster addslots $(for slot in {0..16383}; do echo $slot; done); sleep 5
- name: Run external test
run: |
./runtest \
@@ -62,22 +87,30 @@ jobs:
--tags -slow
- name: Archive server log
if: ${{ failure() }}
uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: test-external-cluster-log
path: external-server.log
test-external-nodebug:
runs-on: ubuntu-latest
if: github.event_name != 'schedule' || github.repository == 'valkey-io/valkey'
if: github.event_name != 'schedule' || github.repository == 'kv-io/kv'
timeout-minutes: 1440
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ianlancetaylor/libbacktrace
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout KV
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Build
run: make SERVER_CFLAGS=-Werror
- name: Start valkey-server
run: make SERVER_CFLAGS=-Werror USE_LIBBACKTRACE=yes
- name: Start kv-server
run: |
./src/valkey-server --daemonize yes --save "" --logfile external-server.log
./src/kv-server --daemonize yes --save "" --logfile external-server.log
- name: Run external test
run: |
./runtest \
@@ -86,7 +119,7 @@ jobs:
--tags "-slow -needs:debug"
- name: Archive server log
if: ${{ failure() }}
uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: test-external-nodebug-log
path: external-server.log
+2 -2
View File
@@ -19,9 +19,9 @@ jobs:
reply-schemas-linter:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Setup nodejs
uses: actions/setup-node@60edb5dd545a775178f52524783378180af0d1f8 # v4.0.2
uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # v6.2.0
- name: Install packages
run: npm install ajv
- name: linter
+2 -2
View File
@@ -23,10 +23,10 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install typos
uses: taiki-e/install-action@fe9759bf4432218c779595708e80a1aadc85cedc # v2.46.10
uses: taiki-e/install-action@d4422f254e595ee762a758628fe4f16ce050fa2e # v2.67.28
with:
tool: typos
@@ -0,0 +1,50 @@
name: Trigger Build Release
on:
release:
types: [published]
workflow_dispatch:
inputs:
version:
description: Version of KV to build
required: true
environment:
description: Environment to build
required: true
type: choice
options:
- dev
- prod
jobs:
trigger:
runs-on: ubuntu-latest
steps:
- name: Determine version and environment
id: determine-vars
run: |
if [[ "${{ github.event_name }}" == "release" ]]; then
echo "Triggered by a release event."
VERSION=${{ github.event.release.tag_name }}
ENVIRONMENT="prod"
else
echo "Triggered manually (workflow_dispatch)."
VERSION=${{ inputs.version }}
ENVIRONMENT=${{ inputs.environment }}
fi
# Set the outputs for version and environment
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "environment=$ENVIRONMENT" >> $GITHUB_OUTPUT
- name: Trigger build
uses: peter-evans/repository-dispatch@28959ce8df70de7be546dd1250a005dd32156697 # v4.0.1
with:
token: ${{ secrets.AUTOMATION_PAT }}
repository: ${{ github.repository_owner }}/kv-release-automation
event-type: build-release
client-payload: >
{
"version": "${{ steps.determine-vars.outputs.version }}",
"environment": "${{ steps.determine-vars.outputs.environment }}"
}
+68
View File
@@ -0,0 +1,68 @@
name: Weekly Test Workflow for Released Branches
on:
schedule:
- cron: '0 6 * * 0'
workflow_dispatch: {}
permissions:
actions: read
contents: read
pull-requests: read
concurrency:
group: weekly-release-tests
cancel-in-progress: false
jobs:
determine-release-branches:
if: github.repository == 'kv-io/kv'
runs-on: ubuntu-latest
outputs:
branches: ${{ steps.release-branches.outputs.branches }}
has-branches: ${{ steps.release-branches.outputs.has-branches }}
steps:
- name: Collect release branches
id: release-branches
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const { owner, repo } = context.repo;
const MIN_MAJOR = 7;
const MIN_MINOR = 2;
const branches = await github.paginate(github.rest.repos.listBranches, {
owner,
repo,
per_page: 100,
});
const releaseBranches = branches
.map(({ name }) => name)
.filter(name => /^\d+\.\d+$/.test(name))
.filter(name => {
const [major, minor] = name.split('.').map(Number);
return Number.isInteger(major) &&
Number.isInteger(minor) &&
(major > MIN_MAJOR || (major === MIN_MAJOR && minor >= MIN_MINOR));
})
.sort((a, b) => a.localeCompare(b, undefined, { numeric: true }));
core.info(`Weekly release branches: ${releaseBranches.join(', ') || '(none)'}`);
core.setOutput('branches', JSON.stringify(releaseBranches));
core.setOutput('has-branches', releaseBranches.length > 0 ? 'true' : 'false');
run-daily-for-release-branches:
needs: determine-release-branches
if: needs.determine-release-branches.outputs.has-branches == 'true' && github.repository == 'kv-io/kv'
permissions:
actions: read
contents: read
pull-requests: read
strategy:
fail-fast: false
matrix:
release-branch: ${{ fromJson(needs.determine-release-branches.outputs.branches) }}
max-parallel: 1
uses: ./.github/workflows/daily.yml
with:
use_repo: kv-io/kv
use_git_ref: ${{ matrix.release-branch }}
skipjobs: ""
skiptests: ""
secrets: inherit
+2
View File
@@ -4,6 +4,8 @@
*.xo
*.so
*.d
*.lo
*.la
*.log
dump*.rdb
*-benchmark
+3 -3
View File
@@ -1,5 +1,5 @@
Hello! This file is just a placeholder, since this is the "unstable" branch
of Valkey, the place where all the development happens.
of KV, the place where all the development happens.
There is no release notes for this branch, it gets forked into another branch
every time there is a partial feature freeze in order to eventually create
@@ -9,8 +9,8 @@ Usually "unstable" is stable enough for you to use it in development environment
however you should never use it in production environments. It is possible
to download the latest stable release here:
https://valkey.io/download/
https://kv.io/download/
More information is available at https://valkey.io
More information is available at https://kv.io
Happy hacking!
Symlink
+1
View File
@@ -0,0 +1 @@
LLM.md
Symlink
+1
View File
@@ -0,0 +1 @@
LLM.md
+37 -3
View File
@@ -13,18 +13,20 @@ if (APPLE)
endif ()
# Options
option(BUILD_UNIT_TESTS "Build valkey-unit-tests" OFF)
option(BUILD_LUA "Build KV Lua scripting engine" ON)
option(BUILD_UNIT_TESTS "Build kv-unit-tests" OFF)
option(BUILD_TEST_MODULES "Build all test modules" OFF)
option(BUILD_EXAMPLE_MODULES "Build example modules" OFF)
set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake/Modules/")
project("valkey")
project("kv")
add_compile_options(-Wundef)
set(CMAKE_C_STANDARD 11)
set(CMAKE_C_STANDARD_REQUIRED ON)
set(CMAKE_C_EXTENSIONS ON)
include(ValkeySetup)
include(KVSetup)
add_subdirectory(src)
add_subdirectory(tests)
@@ -32,6 +34,7 @@ add_subdirectory(tests)
include(Packaging)
# Clear cached variables from the cache
unset(BUILD_LUA CACHE)
unset(BUILD_TESTS CACHE)
unset(CLANGPP CACHE)
unset(CLANG CACHE)
@@ -42,3 +45,34 @@ unset(BUILD_TEST_MODULES CACHE)
unset(BUILD_EXAMPLE_MODULES CACHE)
unset(USE_TLS CACHE)
unset(DEBUG_FORCE_DEFRAG CACHE)
# Helper to copy runtest scripts to allow running tests with CMake built binaries
function(copy_runtest_script script_name)
set(src "${CMAKE_SOURCE_DIR}/${script_name}")
set(dst "${CMAKE_BINARY_DIR}/${script_name}")
file(READ "${src}" contents)
# Split at the first newline (after shebang #!/bin/sh)
string(FIND "${contents}" "\n" index_of_first_newline_char)
math(EXPR first_index_script_body "${index_of_first_newline_char} + 1")
string(SUBSTRING "${contents}" 0 ${first_index_script_body} shebang_line)
string(SUBSTRING "${contents}" ${first_index_script_body} -1 script_body)
# Insert our environment variable lines
set(insert_content
"# Most tests assume running from the project root
cd ${CMAKE_SOURCE_DIR}
export KV_BIN_DIR=\"${CMAKE_BINARY_DIR}/bin\"
")
# Reconstruct the full script
set(new_contents "${shebang_line}${insert_content}${script_body}")
file(WRITE "${dst}" "${new_contents}")
file(CHMOD "${dst}" PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE)
endfunction()
copy_runtest_script("runtest")
copy_runtest_script("runtest-cluster")
copy_runtest_script("runtest-moduleapi")
copy_runtest_script("runtest-rdma")
copy_runtest_script("runtest-sentinel")
+1 -1
View File
@@ -49,7 +49,7 @@ representative at an online or offline event.
Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
this email address: maintainers@lists.valkey.io.
this email address: maintainers@hanzo.ai.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
+38 -14
View File
@@ -1,22 +1,23 @@
Contributing to Valkey
Contributing to KV
======================
Welcome and thank you for wanting to contribute!
# Project governance
The Valkey project is led by a Technical Steering Committee, whose responsibilities are laid out in [GOVERNANCE.md](GOVERNANCE.md).
The KV project is led by a Technical Steering Committee, whose responsibilities are laid out in [GOVERNANCE.md](GOVERNANCE.md).
## Get started
* Have a question? Ask it on
[GitHub Discussions](https://github.com/valkey-io/valkey/discussions)
or [Valkey's Discord](https://discord.gg/zbcPa5umUB)
or [Valkey's Matrix](https://matrix.to/#/#valkey:matrix.org)
* Found a bug? [Report it here](https://github.com/valkey-io/valkey/issues/new?template=bug_report.md&title=%5BBUG%5D)
* Valkey crashed? [Submit a crash report here](https://github.com/valkey-io/valkey/issues/new?template=crash_report.md&title=%5BCRASH%5D+%3Cshort+description%3E)
* Suggest a new feature? [Post your detailed feature request here](https://github.com/valkey-io/valkey/issues/new?template=feature_request.md&title=%5BNEW%5D)
* Want to help with documentation? [Move on to valkey-doc](https://github.com/valkey-io/valkey-doc)
[GitHub Discussions](https://github.com/hanzoai/kv/discussions)
or [KV's Discord](https://discord.gg/zbcPa5umUB)
or [KV's Matrix](https://matrix.to/#/#kv:matrix.org)
* Found a bug? [Report it here](https://github.com/hanzoai/kv/issues/new?template=bug_report.md&title=%5BBUG%5D)
* KV crashed? [Submit a crash report here](https://github.com/hanzoai/kv/issues/new?template=crash_report.md&title=%5BCRASH%5D+%3Cshort+description%3E)
* Suggest a new feature? [Post your detailed feature request here](https://github.com/hanzoai/kv/issues/new?template=feature_request.md&title=%5BNEW%5D)
* Report a test failure? [Report it here](https://github.com/hanzoai/kv/issues/new?template=test-failure.md)
* Want to help with documentation? [Move on to kv-doc](https://github.com/hanzoai/kv-doc)
* Report a vulnerability? See [SECURITY.md](SECURITY.md)
## Developer Certificate of Origin
@@ -57,7 +58,7 @@ By making a contribution to this project, I certify that:
involved.
```
We require that every contribution to Valkey to be signed with a DCO. We require the
We require that every contribution to KV to be signed with a DCO. We require the
usage of known identity (such as a real or preferred name). We do not accept anonymous
contributors nor those utilizing pseudonyms. A DCO signed commit will contain a line like:
@@ -71,11 +72,11 @@ user.name and user.email are set in your git configs, you can use `git commit` w
or `--signoff` to add the `Signed-off-by` line to the end of the commit message. We also
require revert commits to include a DCO.
If you're contributing code to the Valkey project in any other form, including
If you're contributing code to the KV project in any other form, including
sending a code fragment or patch via private email or public discussion groups,
you need to ensure that the contribution is in accordance with the DCO.
# How to provide a patch or a new feature
## How to provide a patch or a new feature
1. If it is a major feature or a semantical change, please don't start coding
straight away: if your feature is not a conceptual fit you'll lose a lot of
@@ -85,7 +86,7 @@ features to be accepted. Here you can see if there is consensus about your idea.
2. If in step 1 you get an acknowledgment from the project leaders, use the following
procedure to submit a patch:
1. Fork Valkey on GitHub ([HOWTO](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo))
1. Fork KV on GitHub ([HOWTO](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo))
1. Create a topic branch (`git checkout -b my_branch`)
1. Make the needed changes and commit with a DCO. (`git commit -s`)
1. Push to your branch (`git push origin my_branch`)
@@ -99,9 +100,32 @@ certain issues/PRs over others. If you think your issue/PR is very important
try to popularize it, have other users commenting and sharing their point of
view, and so forth. This helps.
4. For minor fixes, open a pull request on GitHub.
4. While developing code, make sure to refer to our [DEVELOPMENT_GUIDE.md](DEVELOPMENT_GUIDE.md),
which includes documentation about various best practices for writing KV code.
5. For minor fixes, open a pull request on GitHub.
To link a pull request to an existing issue, please write "Fixes #xyz" somewhere
in the pull request description, where xyz is the issue number.
## Running the daily workflow on demand for your branch
Use [`.github/workflows/daily.yml`](.github/workflows/daily.yml) with
`workflow_dispatch` to run daily tests manually on any branch in your fork.
1. Open your fork on GitHub and go to **Actions** -> **Daily**.
2. Click **Run workflow**.
3. In the **Branch** dropdown, select the branch that contains the workflow file you want to use.
4. In the input fields, set:
* `use_repo` to your fork (for example, `your-user/kv`)
* `use_git_ref` to your branch name (or a specific commit SHA)
5. Optionally set `skipjobs`, `skiptests`, `test_args`, and `cluster_test_args`.
6. Click **Run workflow**.
Notes:
* To run the full matrix, set `skipjobs` and `skiptests` to `none`.
Do not leave them empty, since the workflow input defaults may be applied.
* The scheduled part of this workflow is gated to `hanzoai/kv`, but manual
`workflow_dispatch` runs work for forks.
Thanks!
+3 -1
View File
@@ -1,8 +1,10 @@
# License 1
SPDX-License-Identifier: BSD-3-Clause
BSD 3-Clause License
Copyright (c) 2024-present, Valkey contributors
Copyright (c) 2024-present, KV contributors
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+75
View File
@@ -0,0 +1,75 @@
# KV development guidelines
This document provides a general overview for writing and designing code for KV.
During our long development history, we've made a lot of inconsistent decisions, but we strive to get incrementally better.
## General Best practices
1. Try to limit the number of lines changed in a PR when possible.
We do a lot of backporting as a project, and the more lines changed, the higher the chance of having to resolve merge conflicts.
Please separate refactoring and functional changes into separate PRs, to make it easier to handle backporting.
1. Avoid adding configuration when a feature can be fully controlled by heuristics.
We want KV to work correctly out of the box without much tuning.
Configurations can be added to provide additional tuning of features.
When the workload characteristics can't be inferred or imply a tradeoff (CPU vs memory), then provide a configuration.
## General style guidelines
Most of the style guidelines are enforced by clang format, but some additional comments are included here.
1. C style comments `/* comment */` can be used for both single and multi-line comments.
C++ comments `//` can only be used for single line comments.
Multi line comments should have the leading `*` align and the final `*/` should be on the same line as the last line of text.
e.g.
```c
/* Blah Blah
* Blah Blah. */
```
2. Comments should generally be used to describe behavior that is not obvious from reading the code itself.
This includes complex behavior, why code was written the way it was and describing non-obvious behavior.
Additionally, functions should be documented to explain all of the function's behavior without having to read the code.
1. Generally keep line lengths below 90 characters when reasonable, however there is no explicit line length enforcement.
Use your best judgement for readability.
1. Use static functions when a function is only intended to be accessed from the same file.
For historical reasons, some private functions are prefixed by `_`, and they are kept as is to make it easier to backport changes.
1. Use the boolean type for true/false values.
For historical reasons, some functions used the integer type, and they are kept as is to make it easier to backport changes.
## Naming conventions
KV has a long history of inconsistent naming conventions.
Generally follow the style of the surrounding code, but you can also always use the following conventions for variable and structure names:
- Variable names: `snake_case` or all lower case for short names (e.g. `cached_reply` or `keylen`).
- Function names: `camelCase` or `namespace_camelCase` (e.g. `createStringObject` or `IOJobQueue_isFull`).
- Macros: `UPPER_CASE` (e.g. `MAKE_CMD`).
- Structures: `camelCase` (e.g. `user`).
## Licensing information
When creating new source code files, use the following snippet to indicate the license:
```
/*
* Copyright (c) KV Contributors
* All rights reserved.
* SPDX-License-Identifier: BSD-3-Clause
*/
```
If you are making material changes to a file that has a different license at the top, also add the above license snippet.
There isn't a well defined test for what is considered a material change, but a good rule of thumb is that material changes are more than 100 lines of code.
## Test coverage
KV uses two types of tests: unit and integration tests.
All contributions should include a test of some form.
Unit tests are present in the `src/unit` directory, and are intended to test individual structures or files.
For example, most changes to data structures should include corresponding unit tests.
Integration tests are located in the `tests/` directory, and are intended to test end-to-end functionality.
Adding new commands should come with corresponding integration tests.
When writing cluster mode tests, do not use the legacy `tests/cluster` framework, which has been deprecated, and instead write tests in `unit/cluster`.
## Documentation
KV keeps most of the user documentation in the [kv-doc](https://github.com/hanzoai/kv-doc) repository in a few areas:
1. Major functionality is documented in the [topics](https://github.com/hanzoai/kv-doc/tree/main/topics) section.
1. Specific command behavior is documented in the [commands](https://github.com/hanzoai/kv-doc/tree/main/commands) section.
Command history is also documented in the [command json file](https://github.com/hanzoai/kv/tree/unstable/src/commands).
1. Server info fields are documented in the [INFO](https://github.com/hanzoai/kv-doc/blob/main/commands/info.md) command.
When a PR is opened that requires documentation to be updated, the `needs-doc-pr` should be added until the corresponding documentation PR is open.
+31
View File
@@ -0,0 +1,31 @@
ARG KV_VERSION=9
# Hanzo KV: High-performance key-value store
FROM kv/kv:${KV_VERSION}-alpine AS base
FROM base
LABEL maintainer="dev@hanzo.ai"
LABEL org.opencontainers.image.source="https://github.com/hanzoai/kv"
LABEL org.opencontainers.image.description="Hanzo KV - High-performance key-value store"
LABEL org.opencontainers.image.vendor="Hanzo AI"
# Install Hanzo KV CLI tools
# Primary names are kv-* ; legacy kv-* names remain as symlinks
RUN cp /usr/local/bin/kv-server /usr/local/bin/kv-server \
&& cp /usr/local/bin/kv-cli /usr/local/bin/kv-cli \
&& ln -sf /usr/local/bin/kv-cli /usr/local/bin/kv \
&& cp /usr/local/bin/kv-sentinel /usr/local/bin/kv-sentinel 2>/dev/null; \
cp /usr/local/bin/kv-benchmark /usr/local/bin/kv-benchmark 2>/dev/null; \
cp /usr/local/bin/kv-check-aof /usr/local/bin/kv-check-aof 2>/dev/null; \
cp /usr/local/bin/kv-check-rdb /usr/local/bin/kv-check-rdb 2>/dev/null; \
true
EXPOSE 6379
HEALTHCHECK --interval=15s --timeout=3s --start-period=10s --retries=3 \
CMD kv ping | grep -q PONG || exit 1
ENTRYPOINT ["kv-server"]
CMD ["--bind", "0.0.0.0", "--dir", "/data", "--maxmemory-policy", "allkeys-lru", "--protected-mode", "no"]
+51 -28
View File
@@ -1,24 +1,29 @@
# Project Governance
The Valkey project is managed by a Technical Steering Committee (TSC) composed of the maintainers of the Valkey repository.
The Valkey project includes all of the current and future repositories under the Valkey-io organization.
The KV project is managed by a Technical Steering Committee (TSC) composed of the maintainers of the KV repository.
The KV project includes all of the current and future repositories under the hanzoai organization.
Committers are defined as individuals with write access to the code within a repository.
Maintainers are defined as individuals with full access to a repository and own its governance.
Both maintainers and committers should be clearly listed in the MAINTAINERS.md file in a given projects repository.
Maintainers of other repositories within the Valkey project are not members of the TSC unless explicitly added.
Both maintainers and committers shall be clearly listed in the MAINTAINERS.md file in a given project's repository.
Maintainers of other repositories within the KV project are not members of the TSC unless explicitly added.
## Technical Steering Committee
The TSC is responsible for oversight of all technical, project, approval, and policy matters for Valkey.
The TSC is responsible for oversight of all technical, project, approval, and policy matters for KV.
The TSC members are listed in the [MAINTAINERS.md](MAINTAINERS.md) file in the KV repository.
At any time, no more than one third (1/3) of the TSC members may be employees, contractors, or representatives of the same organization or affiliated organizations.
For the purposes of this document, “organization” includes companies, corporations, universities, research institutes, non-profits, governmental institutions, and any of their subsidiaries or affiliates.
If, at any time, the 1/3 organization limit is exceeded (for example, due to changes in employment, company acquisitions, or organizational affiliations), the TSC shall be notified as soon as possible.
The TSC must promptly take action to restore compliance, which may include removing or reassigning members in accordance with the procedures outlined in the [Termination of Membership](#termination-of-membership) section.
The TSC shall strive to resolve the situation within 30 days of notification, and document the steps taken to restore compliance.
The TSC members are listed in the [MAINTAINERS.md](MAINTAINERS.md) file in the Valkey repository.
Maintainers (and accordingly, TSC members) may be added or removed by no less than 2/3 affirmative vote of the current TSC.
The TSC shall appoint a Chair responsible for organizing TSC meetings.
If the TSC Chair is removed from the TSC (or the Chair steps down from that role), it is the responsibility of the TSC to appoint a new Chair.
The TSC can amend this governance document by no less than a 2/3 affirmative vote.
The TSC may, at its discretion, add or remove members who are not maintainers of the main Valkey repository.
The TSC may, at its discretion, add or remove maintainers from other repositories within the Valkey project.
The TSC may, at its discretion, add or remove members who are not maintainers of the main KV repository.
The TSC may, at its discretion, add or remove maintainers from other repositories within the KV project.
## Voting
@@ -28,17 +33,34 @@ Rather, the TSC shall determine consensus based on their good faith consideratio
The TSC shall document evidence of consensus in accordance with these requirements.
If consensus cannot be reached, the TSC shall make the decision by a vote.
A vote shall also be called when an issue or pull request is marked as a major decision, which are decisions that have a significant impact on the Valkey architecture or design.
Examples of major decisions:
* Fundamental changes to the Valkey core datastructures
* Adding a new data structure or API
* Changes that affect backward compatibility
* New user visible fields that need to be maintained
* Modifications to the TSC or other governance documents
* Adding members to other roles within the Valkey project
* Delegation of maintainership for projects to other groups or individuals
* Adding or removing a new external library such as a client
or module to the project.
A vote shall also be called when an issue or pull request is marked as a major decision, which are decisions that have a significant impact on the KV architecture or design.
### Technical Major Decisions
Technical major decisions include:
* Fundamental changes to the KV core datastructures
* Adding a new data structure or API
* Changes that affect backward compatibility
* New user visible fields that need to be maintained
* Adding or removing a new external library such as a client or module to the project when it affects runtime behavior
Technical major decisions shall be approved by a simple majority vote whenever one can be obtained.
If a simple majority cannot be reached within a two-week voting period, and no TSC member has voted against, the decision may instead be approved through explicit “+2” support from at least two TSC members, recorded on the relevant issue or pull request.
If the pull request author or issue proposer is a TSC member, their +1 counts toward the +2.
If any TSC member casts a negative vote, the decision must follow the simple majority voting process and cannot be approved through +2.
Once a technical major decision has been approved through the +2 mechanism, any subsequent concerns shall be raised through a new major decision process; +2 approvals are not retracted directly.
### Governance Major Decisions
Governance major decisions include:
* Adding TSC members or involuntary removal of TSC members
* Modifying this governance document
* Delegation of maintainership for projects or governance authority
* Creating, modifying, or removing roles within the KV project
* Any change that alters voting rules, TSC responsibilities, or project oversight
* Structural changes to the TSC, including composition limits
Governance major decisions shall require approval by a super-majority vote of at least two thirds (2/3) of the entire TSC.
Any member of the TSC can call a vote with reasonable notice to the TSC, setting out a discussion period and a separate voting period.
Any discussion may be conducted in person or electronically by text, voice, or video.
@@ -46,23 +68,24 @@ The discussion shall be open to the public, with the notable exception of discus
In any vote, each voting TSC member will have one vote.
The TSC shall give at least two weeks for all members to submit their vote.
Except as specifically noted elsewhere in this document, decisions by vote require a simple majority vote of all voting members.
It is the responsibility of the TSC chair to help facilitate the voting process as needed to make sure it completes within the voting period.
If a vote results in a tie, the status quo is preserved.
It is the responsibility of the TSC Chair to help facilitate the voting process as needed to make sure it completes within the voting period.
## Termination of Membership
A maintainer's access (and accordingly, their position on the TSC) will be removed if any of the following occur:
* Involuntary Removal: Removal via the [Governance Major Decision](#governance-major-decisions) voting process.
* Resignation: Written notice of resignation to the TSC.
* TSC Vote: 2/3 affirmative vote of the TSC to remove a member
* Unreachable Member: If a member is unresponsive for more than six months, the remaining active members of the TSC may vote to remove the unreachable member by a simple majority.
## Technical direction for other Valkey projects
## Technical direction for other KV projects
The TSC may delegate decision making for other projects within the Valkey organization to the maintainers responsible for those projects.
Delegation of decision making for a project is considered a major decision, and shall happen with an explicit vote.
Projects within the Valkey organization must indicate the individuals with commit permissions by updating the MAINTAINERS.md within their repositories.
The TSC may delegate decision making for other projects within the KV organization to the maintainers responsible for those projects.
Delegation of decision making for a project is considered a [Governance Major Decision](#governance-major-decisions).
Projects within the KV organization must indicate the individuals with commit permissions by updating the MAINTAINERS.md within their repositories.
The TSC may, at its discretion, overrule the decisions made by other projects within the Valkey organization, although they should show restraint in doing so.
The TSC may, at its discretion, overrule the decisions made by other projects within the KV organization, although they shall show restraint in doing so.
## License of this document
+7
View File
@@ -0,0 +1,7 @@
# kv — AI Assistant Context
<p align="center">
<strong>Hanzo KV</strong>
</p>
<p align="center">
+22 -15
View File
@@ -3,29 +3,36 @@
This document contains a list of maintainers in this repo.
See [GOVERNANCE.md](GOVERNANCE.md) that explains the function of this file.
## Committee Chair
- **Madelyn Olson**
Term: March 28, 2024 Present
## Current Maintainers
Maintainers listed in alphabetical order by their github ID.
| Maintainer | GitHub ID | Affiliation |
| ------------------- | ----------------------------------------------- | ----------- |
| Zhu Binbin | [enjoy-binbin](https://github.com/enjoy-binbin) | Tencent |
| Wen Hui | [hwware](https://github.com/hwware) | Huawei |
| Madelyn Olson | [madolson](https://github.com/madolson) | Amazon |
| Ping Xie | [pingxie](https://github.com/pingxie) | Google |
| Zhao Zhao | [soloestoy](https://github.com/soloestoy) | Alibaba |
| Viktor Söderqvist | [zuiderkwast](https://github.com/zuiderkwast) | Ericsson |
| Maintainer | GitHub ID | Affiliation |
| ------------------- | ------------- | ----------- |
| Binbin Zhu | @enjoy-binbin | Tencent |
| Harkrishn Patro | @hpatro | Amazon |
| Lucas Yang | @lucasyonge | - |
| Madelyn Olson | @madolson | Amazon |
| Jacob Murphy | @murphyjacob4 | Google |
| Ping Xie | @pingxie | Oracle |
| Ran Shidlansik | @ranshid | Amazon |
| Zhao Zhao | @soloestoy | Alibaba |
| Viktor Söderqvist | @zuiderkwast | Ericsson |
## Current Committers
Committers listed in alphabetical order by their github ID.
| Committer | GitHub ID | Affiliation |
| ------------------- | ----------------------------------------------- | ----------- |
| Harkrishn Patro | [hpatro](https://github.com/hpatro) | Amazon |
| Ran Shidlansik | [ranshid](https://github.com/ranshid) | Amazon |
| Committer | GitHub ID | Affiliation |
| ------------------- | ------------- | ----------- |
| Ricardo Dias | @rjd15372 | Percona |
### Former Maintainers and Committers
## Former Maintainers and Committers
| Maintainer | GitHub ID | Affiliation |
| ------------------- | ----------------------------------------------- | ----------- |
| Name | GitHub ID | Role | Term | Affiliation |
| ------------------- | ------------- | ---- | ---- | ----------- |
+127 -354
View File
@@ -1,372 +1,145 @@
[![codecov](https://codecov.io/gh/valkey-io/valkey/graph/badge.svg?token=KYYSJAYC5F)](https://codecov.io/gh/valkey-io/valkey)
This project was forked from the open source Redis project right before the transition to their new source available licenses.
This README is just a fast *quick start* document. More details can be found under [valkey.io](https://valkey.io/)
# What is Valkey?
Valkey is a high-performance data structure server that primarily serves key/value workloads.
It supports a wide range of native structures and an extensible plugin system for adding new data structures and access patterns.
# Building Valkey using `Makefile`
Valkey can be compiled and used on Linux, OSX, OpenBSD, NetBSD, FreeBSD.
We support big endian and little endian architectures, and both 32 bit
and 64 bit systems.
It may compile on Solaris derived systems (for instance SmartOS) but our
support for this platform is *best effort* and Valkey is not guaranteed to
work as well as in Linux, OSX, and \*BSD.
It is as simple as:
% make
To build with TLS support, you'll need OpenSSL development libraries (e.g.
libssl-dev on Debian/Ubuntu).
To build TLS support as Valkey built-in:
% make BUILD_TLS=yes
To build TLS as Valkey module:
% make BUILD_TLS=module
Note that sentinel mode does not support TLS module.
To build with experimental RDMA support you'll need RDMA development libraries
(e.g. librdmacm-dev and libibverbs-dev on Debian/Ubuntu).
To build RDMA support as Valkey built-in:
% make BUILD_RDMA=yes
To build RDMA as Valkey module:
% make BUILD_RDMA=module
To build with systemd support, you'll need systemd development libraries (such
as libsystemd-dev on Debian/Ubuntu or systemd-devel on CentOS) and run:
% make USE_SYSTEMD=yes
To append a suffix to Valkey program names, use:
% make PROG_SUFFIX="-alt"
You can build a 32 bit Valkey binary using:
% make 32bit
After building Valkey, it is a good idea to test it using:
% make test
The above runs the main integration tests. Additional tests are started using:
% make test-unit # Unit tests
% make test-modules # Tests of the module API
% make test-sentinel # Valkey Sentinel integration tests
% make test-cluster # Valkey Cluster integration tests
More about running the integration tests can be found in
[tests/README.md](tests/README.md) and for unit tests, see
[src/unit/README.md](src/unit/README.md).
## Fixing build problems with dependencies or cached build options
Valkey has some dependencies which are included in the `deps` directory.
`make` does not automatically rebuild dependencies even if something in
the source code of dependencies changes.
When you update the source code with `git pull` or when code inside the
dependencies tree is modified in any other way, make sure to use the following
command in order to really clean everything and rebuild from scratch:
% make distclean
This will clean: jemalloc, lua, hiredis, linenoise and other dependencies.
Also if you force certain build options like 32bit target, no C compiler
optimizations (for debugging purposes), and other similar build time options,
those options are cached indefinitely until you issue a `make distclean`
command.
## Fixing problems building 32 bit binaries
If after building Valkey with a 32 bit target you need to rebuild it
with a 64 bit target, or the other way around, you need to perform a
`make distclean` in the root directory of the Valkey distribution.
In case of build errors when trying to build a 32 bit binary of Valkey, try
the following steps:
* Install the package libc6-dev-i386 (also try g++-multilib).
* Try using the following command line instead of `make 32bit`:
`make CFLAGS="-m32 -march=native" LDFLAGS="-m32"`
## Allocator
Selecting a non-default memory allocator when building Valkey is done by setting
the `MALLOC` environment variable. Valkey is compiled and linked against libc
malloc by default, with the exception of jemalloc being the default on Linux
systems. This default was picked because jemalloc has proven to have fewer
fragmentation problems than libc malloc.
To force compiling against libc malloc, use:
% make MALLOC=libc
To compile against jemalloc on Mac OS X systems, use:
% make MALLOC=jemalloc
## Monotonic clock
By default, Valkey will build using the POSIX clock_gettime function as the
monotonic clock source. On most modern systems, the internal processor clock
can be used to improve performance. Cautions can be found here:
http://oliveryang.net/2015/09/pitfalls-of-TSC-usage/
To build with support for the processor's internal instruction clock, use:
% make CFLAGS="-DUSE_PROCESSOR_CLOCK"
## Verbose build
Valkey will build with a user-friendly colorized output by default.
If you want to see a more verbose output, use the following:
% make V=1
# Running Valkey
To run Valkey with the default configuration, just type:
% cd src
% ./valkey-server
If you want to provide your valkey.conf, you have to run it using an additional
parameter (the path of the configuration file):
% cd src
% ./valkey-server /path/to/valkey.conf
It is possible to alter the Valkey configuration by passing parameters directly
as options using the command line. Examples:
% ./valkey-server --port 9999 --replicaof 127.0.0.1 6379
% ./valkey-server /etc/valkey/6379.conf --loglevel debug
All the options in valkey.conf are also supported as options using the command
line, with exactly the same name.
# Running Valkey with TLS:
## Running manually
To manually run a Valkey server with TLS mode (assuming `./gen-test-certs.sh` was invoked so sample certificates/keys are available):
* TLS built-in mode:
```
./src/valkey-server --tls-port 6379 --port 0 \
--tls-cert-file ./tests/tls/valkey.crt \
--tls-key-file ./tests/tls/valkey.key \
--tls-ca-cert-file ./tests/tls/ca.crt
```
* TLS module mode:
```
./src/valkey-server --tls-port 6379 --port 0 \
--tls-cert-file ./tests/tls/valkey.crt \
--tls-key-file ./tests/tls/valkey.key \
--tls-ca-cert-file ./tests/tls/ca.crt \
--loadmodule src/valkey-tls.so
```
Note that you can disable TCP by specifying `--port 0` explicitly.
It's also possible to have both TCP and TLS available at the same time,
but you'll have to assign different ports.
Use `valkey-cli` to connect to the Valkey server:
```
./src/valkey-cli --tls \
--cert ./tests/tls/valkey.crt \
--key ./tests/tls/valkey.key \
--cacert ./tests/tls/ca.crt
```
Specifying `--tls-replication yes` makes a replica connect to the primary.
Using `--tls-cluster yes` makes Valkey Cluster use TLS across nodes.
# Running Valkey with RDMA:
Note that Valkey Over RDMA is an experimental feature.
It may be changed or removed in any minor or major version.
Currently, it is only supported on Linux.
* RDMA built-in mode:
```
./src/valkey-server --protected-mode no \
--rdma-bind 192.168.122.100 --rdma-port 6379
```
* RDMA module mode:
```
./src/valkey-server --protected-mode no \
--loadmodule src/valkey-rdma.so --rdma-bind 192.168.122.100 --rdma-port 6379
```
It's possible to change bind address/port of RDMA by runtime command:
192.168.122.100:6379> CONFIG SET rdma-port 6380
It's also possible to have both RDMA and TCP available, and there is no
conflict of TCP(6379) and RDMA(6379), Ex:
% ./src/valkey-server --protected-mode no \
--loadmodule src/valkey-rdma.so --rdma-bind 192.168.122.100 --rdma-port 6379 \
--port 6379
Note that the network card (192.168.122.100 of this example) should support
RDMA. To test a server supports RDMA or not:
% rdma res show (a new version iproute2 package)
Or:
% ibv_devices
# Playing with Valkey
You can use valkey-cli to play with Valkey. Start a valkey-server instance,
then in another terminal try the following:
% cd src
% ./valkey-cli
valkey> ping
PONG
valkey> set foo bar
OK
valkey> get foo
"bar"
valkey> incr mycounter
(integer) 1
valkey> incr mycounter
(integer) 2
valkey>
# Installing Valkey
In order to install Valkey binaries into /usr/local/bin, just use:
% make install
You can use `make PREFIX=/some/other/directory install` if you wish to use a
different destination.
_Note_: For compatibility with Redis, we create symlinks from the Redis names (`redis-server`, `redis-cli`, etc.) to the Valkey binaries installed by `make install`.
The symlinks are created in same directory as the Valkey binaries.
The symlinks are removed when using `make uninstall`.
The creation of the symlinks can be skipped by setting the makefile variable `USE_REDIS_SYMLINKS=no`.
`make install` will just install binaries in your system, but will not configure
init scripts and configuration files in the appropriate place. This is not
needed if you just want to play a bit with Valkey, but if you are installing
it the proper way for a production system, we have a script that does this
for Ubuntu and Debian systems:
% cd utils
% ./install_server.sh
_Note_: `install_server.sh` will not work on Mac OSX; it is built for Linux only.
The script will ask you a few questions and will setup everything you need
to run Valkey properly as a background daemon that will start again on
system reboots.
You'll be able to stop and start Valkey using the script named
`/etc/init.d/valkey_<portnumber>`, for instance `/etc/init.d/valkey_6379`.
# Building using `CMake`
In addition to the traditional `Makefile` build, Valkey supports an alternative, **experimental**, build system using `CMake`.
To build and install `Valkey`, in `Release` mode (an optimized build), type this into your terminal:
<p align="center">
<strong>Hanzo KV</strong>
</p>
<p align="center">
High-performance key-value store for the Hanzo ecosystem.<br/>
In-memory data store used as database, cache, streaming engine, and message broker.
</p>
<p align="center">
<a href="https://github.com/hanzoai/kv/actions"><img src="https://github.com/hanzoai/kv/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
<a href="https://github.com/hanzoai/kv/releases"><img src="https://img.shields.io/github/v/release/hanzoai/kv" alt="Release"></a>
<a href="https://github.com/hanzoai/kv/blob/main/LICENSE"><img src="https://img.shields.io/github/license/hanzoai/kv" alt="License"></a>
</p>
---
## Features
- **In-memory key-value store** -- sub-millisecond reads and writes
- **Redis-compatible protocol** -- drop-in replacement for existing Redis clients
- **Persistence** -- RDB snapshots and AOF (append-only file) for durability
- **Replication** -- primary-replica with automatic failover via Sentinel
- **Lua scripting** -- server-side scripting for atomic operations
- **Pub/Sub** -- publish and subscribe messaging
- **Streams** -- append-only log data structure for event sourcing
- **Cluster mode** -- horizontal scaling with automatic sharding
- **Modules** -- extensible plugin system for custom data structures
- **ZAP native** -- built-in [ZAP binary protocol](https://github.com/luxfi/zap) on port 9653 (17x faster than JSON-RPC)
- **Multi-arch** -- linux/amd64 and linux/arm64
## Quick Start
### Docker
```bash
mkdir build-release
cd $_
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/valkey
sudo make install
# Valkey is now installed under /opt/valkey
docker run -d --name hanzo-kv -p 6379:6379 ghcr.io/hanzoai/kv
```
Other options supported by Valkey's `CMake` build system:
## Special build flags
- `-DBUILD_TLS=<yes|no>` enable TLS build for Valkey. Default: `no`
- `-DBUILD_RDMA=<no|module>` enable RDMA module build (only module mode supported). Default: `no`
- `-DBUILD_MALLOC=<libc|jemalloc|tcmalloc|tcmalloc_minimal>` choose the allocator to use. Default on Linux: `jemalloc`, for other OS: `libc`
- `-DBUILD_SANITIZER=<address|thread|undefined>` build with address sanitizer enabled. Default: disabled (no sanitizer)
- `-DBUILD_UNIT_TESTS=[yes|no]` when set, the build will produce the executable `valkey-unit-tests`. Default: `no`
- `-DBUILD_TEST_MODULES=[yes|no]` when set, the build will include the modules located under the `tests/modules` folder. Default: `no`
- `-DBUILD_EXAMPLE_MODULES=[yes|no]` when set, the build will include the example modules located under the `src/modules` folder. Default: `no`
## Common flags
- `-DCMAKE_BUILD_TYPE=<Debug|Release...>` define the build type, see CMake manual for more details
- `-DCMAKE_INSTALL_PREFIX=/installation/path` override this value to define a custom install prefix. Default: `/usr/local`
- `-G"<Generator Name>"` generate build files for "Generator Name". By default, CMake will generate `Makefile`s.
## Verbose build
`CMake` generates a user-friendly colorized output by default.
If you want to see a more verbose output, use the following:
### Connect
```bash
make VERBOSE=1
docker exec -it hanzo-kv kv
127.0.0.1:6379> SET hello world
OK
127.0.0.1:6379> GET hello
"world"
```
## Troubleshooting
Any Redis-compatible CLI also works out of the box.
During the `CMake` stage, `CMake` caches variables in a local file named `CMakeCache.txt`. All variables generated by Valkey
are removed from the cache once consumed (this is done by calling to `unset(VAR-NAME CACHE)`). However, some variables,
like the compiler path, are kept in cache. To start a fresh build either remove the cache file `CMakeCache.txt` from the
build folder, or delete the build folder completely.
**It is important to re-run `CMake` when adding new source files.**
## Integration with IDE
During the `CMake` stage of the build, `CMake` generates a JSON file named `compile_commands.json` and places it under the
build folder. This file is used by many IDEs and text editors for providing code completion (via `clangd`).
A small caveat is that these tools will look for `compile_commands.json` under the Valkey's top folder.
A common workaround is to create a symbolic link to it:
### Build from Source
```bash
cd /path/to/valkey/
# We assume here that your build folder is `build-release`
ln -sf $(pwd)/build-release/compile_commands.json $(pwd)/compile_commands.json
make
make test
make install
```
Restart your IDE and voila
## CLI Tools
# Code contributions
| Command | Description |
|---------|-------------|
| `kv` | Interactive CLI (default) |
| `kv-server` | Start KV server |
| `kv-cli` | Command-line client |
| `kv-sentinel` | High-availability sentinel |
| `kv-benchmark` | Performance benchmarking |
| `kv-check-aof` | AOF file integrity check |
| `kv-check-rdb` | RDB file integrity check |
Please see the [CONTRIBUTING.md][2]. For security bugs and vulnerabilities, please see [SECURITY.md][3].
## Configuration
# Valkey is an open community project under LF Projects
Pass a config file at startup:
Valkey a Series of LF Projects, LLC
2810 N Church St, PMB 57274
Wilmington, Delaware 19802-4447
```bash
kv-server /etc/kv/kv.conf
```
[1]: https://github.com/valkey-io/valkey/blob/unstable/COPYING
[2]: https://github.com/valkey-io/valkey/blob/unstable/CONTRIBUTING.md
[3]: https://github.com/valkey-io/valkey/blob/unstable/SECURITY.md
Or set options via command line:
```bash
kv-server --port 6379 --maxmemory 256mb --appendonly yes
```
## ZAP Binary Protocol
Hanzo KV speaks [ZAP](https://github.com/luxfi/zap) natively on port **9653** — no sidecar needed.
ZAP is a zero-copy binary protocol that's 17x faster than JSON-RPC with 11x less memory usage.
### Enable ZAP
ZAP is enabled by default. Load the module:
```bash
kv-server --loadmodule /path/to/zap.so
# or with custom port:
kv-server --loadmodule /path/to/zap.so PORT 9653
```
### ZAP Operations
| Path | Body | Description |
|------|------|-------------|
| `/get` | `{"key":"mykey"}` | GET a key |
| `/set` | `{"key":"mykey","value":"myval"}` | SET a key |
| `/del` | `{"key":"mykey"}` | DEL a key |
| `/cmd` | `{"cmd":"PING","args":[]}` | Execute any command |
### Module API
Develop custom modules using the KV Module API:
```c
#include "kvmodule.h"
int KVModule_OnLoad(KVModuleCtx *ctx, KVModuleString **argv, int argc) {
if (KVModule_Init(ctx, "mymod", 1, KVMODULE_APIVER_1) == KVMODULE_ERR)
return KVMODULE_ERR;
// register commands...
return KVMODULE_OK;
}
```
## Client SDKs
| Language | Package | Install |
|----------|---------|---------|
| Python | [hanzo-kv](https://pypi.org/project/hanzo-kv) | `pip install hanzo-kv` |
| Go | [hanzo/kv-go](https://github.com/hanzoai/kv-go) | `go get github.com/hanzoai/kv-go` |
| Node.js | [@hanzo/kv](https://github.com/hanzoai/kv-client) | `npm install @hanzo/kv` |
Any Redis-compatible client library will also work.
## Documentation
Full documentation is available at [docs.hanzo.ai](https://docs.hanzo.ai).
## License
BSD-3-Clause
Copyright (c) 2024-2026 Hanzo AI Inc. All rights reserved.
+3 -3
View File
@@ -1,6 +1,6 @@
## Reporting a Vulnerability
If you believe you've discovered a security vulnerability, please contact the Valkey team at security@lists.valkey.io.
If you believe you've discovered a security vulnerability, please contact the KV team at security@hanzo.ai.
Please *DO NOT* create an issue.
We follow a responsible disclosure procedure, so depending on the severity of the issue we may notify Valkey vendors about the issue before releasing it publicly.
If you would like to be added to our list of vendors, please reach out to the Valkey team at maintainers@lists.valkey.io.
We follow a responsible disclosure procedure, so depending on the severity of the issue we may notify KV vendors about the issue before releasing it publicly.
If you would like to be added to our list of vendors, please reach out to the KV team at maintainers@hanzo.ai.
@@ -9,11 +9,11 @@ set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/lib")
# Generate compile_commands.json file for IDEs code completion support
set(CMAKE_EXPORT_COMPILE_COMMANDS 1)
processorcount(VALKEY_PROCESSOR_COUNT)
message(STATUS "Processor count: ${VALKEY_PROCESSOR_COUNT}")
processorcount(KV_PROCESSOR_COUNT)
message(STATUS "Processor count: ${KV_PROCESSOR_COUNT}")
# Installed executables will have this permissions
set(VALKEY_EXE_PERMISSIONS
set(KV_EXE_PERMISSIONS
OWNER_EXECUTE
OWNER_WRITE
OWNER_READ
@@ -22,22 +22,22 @@ set(VALKEY_EXE_PERMISSIONS
WORLD_EXECUTE
WORLD_READ)
set(VALKEY_SERVER_CFLAGS "")
set(VALKEY_SERVER_LDFLAGS "")
set(KV_SERVER_CFLAGS "")
set(KV_SERVER_LDFLAGS "")
# ----------------------------------------------------
# Helper functions & macros
# ----------------------------------------------------
macro (add_valkey_server_compiler_options value)
set(VALKEY_SERVER_CFLAGS "${VALKEY_SERVER_CFLAGS} ${value}")
macro (add_kv_server_compiler_options value)
set(KV_SERVER_CFLAGS "${KV_SERVER_CFLAGS} ${value}")
endmacro ()
macro (add_valkey_server_linker_option value)
list(APPEND VALKEY_SERVER_LDFLAGS ${value})
macro (add_kv_server_linker_option value)
list(APPEND KV_SERVER_LDFLAGS ${value})
endmacro ()
macro (get_valkey_server_linker_option return_value)
list(JOIN VALKEY_SERVER_LDFLAGS " " ${value} ${return_value})
macro (get_kv_server_linker_option return_value)
list(JOIN KV_SERVER_LDFLAGS " " ${value} ${return_value})
endmacro ()
set(IS_FREEBSD 0)
@@ -45,33 +45,33 @@ if (CMAKE_SYSTEM_NAME MATCHES "^.*BSD$|DragonFly")
message(STATUS "Building for FreeBSD compatible system")
set(IS_FREEBSD 1)
include_directories("/usr/local/include")
add_valkey_server_compiler_options("-DUSE_BACKTRACE")
add_kv_server_compiler_options("-DUSE_BACKTRACE")
endif ()
# Helper function for creating symbolic link so that: link -> source
macro (valkey_create_symlink source link)
install(
CODE "execute_process( \
COMMAND /bin/bash ${CMAKE_BINARY_DIR}/CreateSymlink.sh \
${source} \
${link} \
)"
COMPONENT "valkey")
macro (kv_create_symlink source link)
add_custom_command(
TARGET ${source} POST_BUILD
COMMAND ${CMAKE_COMMAND} -E create_symlink
"$<TARGET_FILE_NAME:${source}>"
"$<TARGET_FILE_DIR:${source}>/${link}"
VERBATIM
)
endmacro ()
# Install a binary
macro (valkey_install_bin target)
macro (kv_install_bin target)
# Install cli tool and create a redis symbolic link
install(
TARGETS ${target}
DESTINATION ${CMAKE_INSTALL_BINDIR}
PERMISSIONS ${VALKEY_EXE_PERMISSIONS}
COMPONENT "valkey")
PERMISSIONS ${KV_EXE_PERMISSIONS}
COMPONENT "kv")
endmacro ()
# Helper function that defines, builds and installs `target` In addition, it creates a symbolic link between the target
# and `link_name`
macro (valkey_build_and_install_bin target sources ld_flags libs link_name)
macro (kv_build_and_install_bin target sources ld_flags libs link_name)
add_executable(${target} ${sources})
if (USE_JEMALLOC
@@ -83,10 +83,15 @@ macro (valkey_build_and_install_bin target sources ld_flags libs link_name)
# Place this line last to ensure that ${ld_flags} is placed last on the linker line
target_link_libraries(${target} ${libs} ${ld_flags})
target_link_libraries(${target} hiredis)
target_link_libraries(${target} kv::kv)
if (USE_TLS)
# Add required libraries needed for TLS
target_link_libraries(${target} OpenSSL::SSL hiredis_ssl)
target_link_libraries(${target} OpenSSL::SSL kv::kv_tls)
endif ()
if (USE_RDMA)
# Add required libraries needed for RDMA
target_link_libraries(${target} kv::kv_rdma)
endif ()
if (IS_FREEBSD)
@@ -97,42 +102,18 @@ macro (valkey_build_and_install_bin target sources ld_flags libs link_name)
target_compile_options(${target} PRIVATE -Werror -Wall)
# Install cli tool and create a redis symbolic link
valkey_install_bin(${target})
valkey_create_symlink(${target} ${link_name})
endmacro ()
# Helper function that defines, builds and installs `target` module.
macro (valkey_build_and_install_module target sources ld_flags libs)
add_library(${target} SHARED ${sources})
if (USE_JEMALLOC)
# Using jemalloc
target_link_libraries(${target} jemalloc)
endif ()
# Place this line last to ensure that ${ld_flags} is placed last on the linker line
target_link_libraries(${target} ${libs} ${ld_flags})
if (USE_TLS)
# Add required libraries needed for TLS
target_link_libraries(${target} OpenSSL::SSL hiredis_ssl)
endif ()
if (IS_FREEBSD)
target_link_libraries(${target} execinfo)
endif ()
# Install cli tool and create a redis symbolic link
valkey_install_bin(${target})
kv_install_bin(${target})
kv_create_symlink(${target} ${link_name})
endmacro ()
# Determine if we are building in Release or Debug mode
if (CMAKE_BUILD_TYPE MATCHES Debug OR CMAKE_BUILD_TYPE MATCHES DebugFull)
set(VALKEY_DEBUG_BUILD 1)
set(VALKEY_RELEASE_BUILD 0)
set(KV_DEBUG_BUILD 1)
set(KV_RELEASE_BUILD 0)
message(STATUS "Building in debug mode")
else ()
set(VALKEY_DEBUG_BUILD 0)
set(VALKEY_RELEASE_BUILD 1)
set(KV_DEBUG_BUILD 0)
set(KV_RELEASE_BUILD 1)
message(STATUS "Building in release mode")
endif ()
@@ -157,21 +138,21 @@ if (BUILD_MALLOC)
if ("${BUILD_MALLOC}" STREQUAL "jemalloc")
set(MALLOC_LIB "jemalloc")
set(ALLOCATOR_LIB "jemalloc")
add_valkey_server_compiler_options("-DUSE_JEMALLOC")
add_kv_server_compiler_options("-DUSE_JEMALLOC")
set(USE_JEMALLOC 1)
elseif ("${BUILD_MALLOC}" STREQUAL "libc")
set(MALLOC_LIB "libc")
elseif ("${BUILD_MALLOC}" STREQUAL "tcmalloc")
set(MALLOC_LIB "tcmalloc")
valkey_pkg_config(libtcmalloc ALLOCATOR_LIB)
kv_pkg_config(libtcmalloc ALLOCATOR_LIB)
add_valkey_server_compiler_options("-DUSE_TCMALLOC")
add_kv_server_compiler_options("-DUSE_TCMALLOC")
set(USE_TCMALLOC 1)
elseif ("${BUILD_MALLOC}" STREQUAL "tcmalloc_minimal")
set(MALLOC_LIB "tcmalloc_minimal")
valkey_pkg_config(libtcmalloc_minimal ALLOCATOR_LIB)
kv_pkg_config(libtcmalloc_minimal ALLOCATOR_LIB)
add_valkey_server_compiler_options("-DUSE_TCMALLOC")
add_kv_server_compiler_options("-DUSE_TCMALLOC")
set(USE_TCMALLOC_MINIMAL 1)
else ()
message(FATAL_ERROR "BUILD_MALLOC can be one of: jemalloc, libc, tcmalloc or tcmalloc_minimal")
@@ -182,7 +163,7 @@ message(STATUS "Using ${MALLOC_LIB}")
# TLS support
if (BUILD_TLS)
valkey_parse_build_option(${BUILD_TLS} USE_TLS)
kv_parse_build_option(${BUILD_TLS} USE_TLS)
if (USE_TLS EQUAL 1)
# Only search for OpenSSL if needed
find_package(OpenSSL REQUIRED)
@@ -192,8 +173,8 @@ if (BUILD_TLS)
endif ()
if (USE_TLS EQUAL 1)
add_valkey_server_compiler_options("-DUSE_OPENSSL=1")
add_valkey_server_compiler_options("-DBUILD_TLS_MODULE=0")
add_kv_server_compiler_options("-DUSE_OPENSSL=1")
add_kv_server_compiler_options("-DBUILD_TLS_MODULE=0")
else ()
# Build TLS as a module RDMA can only be built as a module. So disable it
message(WARNING "BUILD_TLS can be one of: [ON | OFF | 1 | 0], but '${BUILD_TLS}' was provided")
@@ -210,22 +191,22 @@ if (BUILD_RDMA)
set(BUILD_RDMA_MODULE 0)
# RDMA support (Linux only)
if (LINUX AND NOT APPLE)
valkey_parse_build_option(${BUILD_RDMA} USE_RDMA)
kv_parse_build_option(${BUILD_RDMA} USE_RDMA)
find_package(PkgConfig REQUIRED)
# Locate librdmacm & libibverbs, fail if we can't find them
valkey_pkg_config(librdmacm RDMACM_LIBS)
valkey_pkg_config(libibverbs IBVERBS_LIBS)
kv_pkg_config(librdmacm RDMACM_LIBS)
kv_pkg_config(libibverbs IBVERBS_LIBS)
message(STATUS "${RDMACM_LIBS};${IBVERBS_LIBS}")
list(APPEND RDMA_LIBS "${RDMACM_LIBS};${IBVERBS_LIBS}")
if (USE_RDMA EQUAL 2) # Module
message(STATUS "Building RDMA as module")
add_valkey_server_compiler_options("-DUSE_RDMA=2")
add_kv_server_compiler_options("-DUSE_RDMA=2")
set(BUILD_RDMA_MODULE 2)
elseif (USE_RDMA EQUAL 1) # Builtin
message(STATUS "Building RDMA as builtin")
add_valkey_server_compiler_options("-DUSE_RDMA=1")
add_valkey_server_compiler_options("-DBUILD_RDMA_MODULE=0")
add_kv_server_compiler_options("-DUSE_RDMA=1")
add_kv_server_compiler_options("-DBUILD_RDMA_MODULE=0")
list(APPEND SERVER_LIBS "${RDMA_LIBS}")
endif ()
else ()
@@ -240,7 +221,7 @@ endif ()
set(BUILDING_ARM64 0)
set(BUILDING_ARM32 0)
if ("${CMAKE_SYSTEM_PROCESSOR}" STREQUAL "arm64")
if ("${CMAKE_SYSTEM_PROCESSOR}" STREQUAL "arm64" OR "${CMAKE_SYSTEM_PROCESSOR}" STREQUAL "aarch64")
set(BUILDING_ARM64 1)
endif ()
@@ -250,55 +231,55 @@ endif ()
message(STATUS "Building on ${CMAKE_HOST_SYSTEM_NAME}")
if (BUILDING_ARM64)
message(STATUS "Compiling valkey for ARM64")
add_valkey_server_linker_option("-funwind-tables")
message(STATUS "Compiling kv for ARM64")
add_kv_server_linker_option("-funwind-tables")
endif ()
if (APPLE)
add_valkey_server_linker_option("-rdynamic")
add_valkey_server_linker_option("-ldl")
add_kv_server_linker_option("-rdynamic")
add_kv_server_linker_option("-ldl")
elseif (UNIX)
add_valkey_server_linker_option("-rdynamic")
add_valkey_server_linker_option("-pthread")
add_valkey_server_linker_option("-ldl")
add_valkey_server_linker_option("-lm")
add_kv_server_linker_option("-rdynamic")
add_kv_server_linker_option("-pthread")
add_kv_server_linker_option("-ldl")
add_kv_server_linker_option("-lm")
endif ()
if (VALKEY_DEBUG_BUILD)
if (KV_DEBUG_BUILD)
# Debug build, use enable "-fno-omit-frame-pointer"
add_valkey_server_compiler_options("-fno-omit-frame-pointer")
add_kv_server_compiler_options("-fno-omit-frame-pointer")
endif ()
# Check for Atomic
check_include_files(stdatomic.h HAVE_C11_ATOMIC)
if (HAVE_C11_ATOMIC)
add_valkey_server_compiler_options("-std=gnu11")
add_kv_server_compiler_options("-std=gnu11")
else ()
add_valkey_server_compiler_options("-std=c99")
add_kv_server_compiler_options("-std=c99")
endif ()
# Sanitizer
if (BUILD_SANITIZER)
# Common CFLAGS
list(APPEND VALKEY_SANITAIZER_CFLAGS "-fno-sanitize-recover=all")
list(APPEND VALKEY_SANITAIZER_CFLAGS "-fno-omit-frame-pointer")
list(APPEND KV_SANITAIZER_CFLAGS "-fno-sanitize-recover=all")
list(APPEND KV_SANITAIZER_CFLAGS "-fno-omit-frame-pointer")
if ("${BUILD_SANITIZER}" STREQUAL "address")
list(APPEND VALKEY_SANITAIZER_CFLAGS "-fsanitize=address")
list(APPEND VALKEY_SANITAIZER_LDFLAGS "-fsanitize=address")
list(APPEND KV_SANITAIZER_CFLAGS "-fsanitize=address")
list(APPEND KV_SANITAIZER_LDFLAGS "-fsanitize=address")
elseif ("${BUILD_SANITIZER}" STREQUAL "thread")
list(APPEND VALKEY_SANITAIZER_CFLAGS "-fsanitize=thread")
list(APPEND VALKEY_SANITAIZER_LDFLAGS "-fsanitize=thread")
list(APPEND KV_SANITAIZER_CFLAGS "-fsanitize=thread")
list(APPEND KV_SANITAIZER_LDFLAGS "-fsanitize=thread")
elseif ("${BUILD_SANITIZER}" STREQUAL "undefined")
list(APPEND VALKEY_SANITAIZER_CFLAGS "-fsanitize=undefined")
list(APPEND VALKEY_SANITAIZER_LDFLAGS "-fsanitize=undefined")
list(APPEND KV_SANITAIZER_CFLAGS "-fsanitize=undefined")
list(APPEND KV_SANITAIZER_LDFLAGS "-fsanitize=undefined")
else ()
message(FATAL_ERROR "Unknown sanitizer: ${BUILD_SANITIZER}")
endif ()
endif ()
include_directories("${CMAKE_SOURCE_DIR}/deps/hiredis")
include_directories("${CMAKE_SOURCE_DIR}/deps/libkv/include")
include_directories("${CMAKE_SOURCE_DIR}/src/modules/lua")
include_directories("${CMAKE_SOURCE_DIR}/deps/linenoise")
include_directories("${CMAKE_SOURCE_DIR}/deps/lua/src")
include_directories("${CMAKE_SOURCE_DIR}/deps/hdr_histogram")
include_directories("${CMAKE_SOURCE_DIR}/deps/fpconv")
@@ -310,7 +291,11 @@ if (USE_JEMALLOC)
endif ()
# Common compiler flags
add_valkey_server_compiler_options("-pedantic")
add_kv_server_compiler_options("-pedantic")
if (NOT BUILD_LUA)
message(STATUS "Lua scripting engine is disabled")
endif()
# ----------------------------------------------------
# Build options (allocator, tls, rdma et al) - end
@@ -380,8 +365,8 @@ include(SourceFiles)
# Clear the below variables from the cache
unset(CMAKE_C_FLAGS CACHE)
unset(VALKEY_SERVER_LDFLAGS CACHE)
unset(VALKEY_SERVER_CFLAGS CACHE)
unset(KV_SERVER_LDFLAGS CACHE)
unset(KV_SERVER_CFLAGS CACHE)
unset(PYTHON_EXE CACHE)
unset(HAVE_C11_ATOMIC CACHE)
unset(USE_TLS CACHE)
+7 -7
View File
@@ -1,23 +1,23 @@
set(CPACK_PACKAGE_NAME "valkey")
set(CPACK_PACKAGE_NAME "kv")
valkey_parse_version(CPACK_PACKAGE_VERSION_MAJOR CPACK_PACKAGE_VERSION_MINOR CPACK_PACKAGE_VERSION_PATCH)
kv_parse_version(CPACK_PACKAGE_VERSION_MAJOR CPACK_PACKAGE_VERSION_MINOR CPACK_PACKAGE_VERSION_PATCH)
set(CPACK_PACKAGE_CONTACT "maintainers@lists.valkey.io")
set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "Valkey is an open source (BSD) high-performance key/value datastore")
set(CPACK_PACKAGE_CONTACT "maintainers@lists.kv.io")
set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "KV is an open source (BSD) high-performance key/value datastore")
set(CPACK_RESOURCE_FILE_LICENSE "${CMAKE_SOURCE_DIR}/COPYING")
set(CPACK_RESOURCE_FILE_README "${CMAKE_SOURCE_DIR}/README.md")
set(CPACK_STRIP_FILES TRUE)
valkey_get_distro_name(DISTRO_NAME)
kv_get_distro_name(DISTRO_NAME)
message(STATUS "Current host distro: ${DISTRO_NAME}")
if (DISTRO_NAME MATCHES ubuntu
OR DISTRO_NAME MATCHES debian
OR DISTRO_NAME MATCHES mint)
message(STATUS "Adding target package for ${DISTRO_NAME}")
set(CPACK_PACKAGING_INSTALL_PREFIX "/opt/valkey")
set(CPACK_PACKAGING_INSTALL_PREFIX "/opt/kv")
# Debian related parameters
set(CPACK_DEBIAN_PACKAGE_MAINTAINER "Valkey contributors")
set(CPACK_DEBIAN_PACKAGE_MAINTAINER "KV contributors")
set(CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON)
set(CPACK_DEBIAN_FILE_NAME DEB-DEFAULT)
set(CPACK_GENERATOR "DEB")
+40 -21
View File
@@ -2,10 +2,11 @@
# Define the sources to be built
# -------------------------------------------------
# valkey-server source files
set(VALKEY_SERVER_SRCS
# kv-server source files
set(KV_SERVER_SRCS
${CMAKE_SOURCE_DIR}/src/threads_mngr.c
${CMAKE_SOURCE_DIR}/src/adlist.c
${CMAKE_SOURCE_DIR}/src/vector.c
${CMAKE_SOURCE_DIR}/src/quicklist.c
${CMAKE_SOURCE_DIR}/src/ae.c
${CMAKE_SOURCE_DIR}/src/anet.c
@@ -43,10 +44,10 @@ set(VALKEY_SERVER_SRCS
${CMAKE_SOURCE_DIR}/src/intset.c
${CMAKE_SOURCE_DIR}/src/syncio.c
${CMAKE_SOURCE_DIR}/src/cluster.c
${CMAKE_SOURCE_DIR}/src/cluster_migrateslots.c
${CMAKE_SOURCE_DIR}/src/cluster_legacy.c
${CMAKE_SOURCE_DIR}/src/cluster_slot_stats.c
${CMAKE_SOURCE_DIR}/src/crc16.c
${CMAKE_SOURCE_DIR}/src/endianconv.c
${CMAKE_SOURCE_DIR}/src/commandlog.c
${CMAKE_SOURCE_DIR}/src/eval.c
${CMAKE_SOURCE_DIR}/src/bio.c
@@ -65,11 +66,12 @@ set(VALKEY_SERVER_SRCS
${CMAKE_SOURCE_DIR}/src/hyperloglog.c
${CMAKE_SOURCE_DIR}/src/latency.c
${CMAKE_SOURCE_DIR}/src/sparkline.c
${CMAKE_SOURCE_DIR}/src/valkey-check-rdb.c
${CMAKE_SOURCE_DIR}/src/valkey-check-aof.c
${CMAKE_SOURCE_DIR}/src/kv-check-rdb.c
${CMAKE_SOURCE_DIR}/src/kv-check-aof.c
${CMAKE_SOURCE_DIR}/src/geo.c
${CMAKE_SOURCE_DIR}/src/lazyfree.c
${CMAKE_SOURCE_DIR}/src/module.c
${CMAKE_SOURCE_DIR}/src/lrulfu.c
${CMAKE_SOURCE_DIR}/src/evict.c
${CMAKE_SOURCE_DIR}/src/expire.c
${CMAKE_SOURCE_DIR}/src/geohash.c
@@ -85,6 +87,7 @@ set(VALKEY_SERVER_SRCS
${CMAKE_SOURCE_DIR}/src/lolwut.c
${CMAKE_SOURCE_DIR}/src/lolwut5.c
${CMAKE_SOURCE_DIR}/src/lolwut6.c
${CMAKE_SOURCE_DIR}/src/lolwut9.c
${CMAKE_SOURCE_DIR}/src/acl.c
${CMAKE_SOURCE_DIR}/src/tracking.c
${CMAKE_SOURCE_DIR}/src/socket.c
@@ -97,26 +100,37 @@ set(VALKEY_SERVER_SRCS
${CMAKE_SOURCE_DIR}/src/mt19937-64.c
${CMAKE_SOURCE_DIR}/src/resp_parser.c
${CMAKE_SOURCE_DIR}/src/call_reply.c
${CMAKE_SOURCE_DIR}/src/lua/script_lua.c
${CMAKE_SOURCE_DIR}/src/script.c
${CMAKE_SOURCE_DIR}/src/functions.c
${CMAKE_SOURCE_DIR}/src/scripting_engine.c
${CMAKE_SOURCE_DIR}/src/lua/function_lua.c
${CMAKE_SOURCE_DIR}/src/lua/engine_lua.c
${CMAKE_SOURCE_DIR}/src/lua/debug_lua.c
${CMAKE_SOURCE_DIR}/src/trace/trace.c
${CMAKE_SOURCE_DIR}/src/trace/trace_rdb.c
${CMAKE_SOURCE_DIR}/src/trace/trace_aof.c
${CMAKE_SOURCE_DIR}/src/trace/trace_commands.c
${CMAKE_SOURCE_DIR}/src/trace/trace_db.c
${CMAKE_SOURCE_DIR}/src/trace/trace_cluster.c
${CMAKE_SOURCE_DIR}/src/trace/trace_server.c
${CMAKE_SOURCE_DIR}/src/commands.c
${CMAKE_SOURCE_DIR}/src/strl.c
${CMAKE_SOURCE_DIR}/src/connection.c
${CMAKE_SOURCE_DIR}/src/unix.c
${CMAKE_SOURCE_DIR}/src/server.c
${CMAKE_SOURCE_DIR}/src/logreqres.c)
${CMAKE_SOURCE_DIR}/src/logreqres.c
${CMAKE_SOURCE_DIR}/src/entry.c
${CMAKE_SOURCE_DIR}/src/vset.c
${CMAKE_SOURCE_DIR}/src/fifo.c
${CMAKE_SOURCE_DIR}/src/mutexqueue.c)
# valkey-cli
set(VALKEY_CLI_SRCS
# kv-cli
set(KV_CLI_SRCS
${CMAKE_SOURCE_DIR}/src/anet.c
${CMAKE_SOURCE_DIR}/src/adlist.c
${CMAKE_SOURCE_DIR}/src/dict.c
${CMAKE_SOURCE_DIR}/src/valkey-cli.c
${CMAKE_SOURCE_DIR}/src/sds.c
${CMAKE_SOURCE_DIR}/src/sha256.c
${CMAKE_SOURCE_DIR}/src/util.c
${CMAKE_SOURCE_DIR}/src/kv-cli.c
${CMAKE_SOURCE_DIR}/src/zmalloc.c
${CMAKE_SOURCE_DIR}/src/release.c
${CMAKE_SOURCE_DIR}/src/ae.c
@@ -132,11 +146,14 @@ set(VALKEY_CLI_SRCS
${CMAKE_SOURCE_DIR}/src/strl.c
${CMAKE_SOURCE_DIR}/src/cli_commands.c)
# valkey-benchmark
set(VALKEY_BENCHMARK_SRCS
# kv-benchmark
set(KV_BENCHMARK_SRCS
${CMAKE_SOURCE_DIR}/src/ae.c
${CMAKE_SOURCE_DIR}/src/anet.c
${CMAKE_SOURCE_DIR}/src/valkey-benchmark.c
${CMAKE_SOURCE_DIR}/src/sds.c
${CMAKE_SOURCE_DIR}/src/sha256.c
${CMAKE_SOURCE_DIR}/src/util.c
${CMAKE_SOURCE_DIR}/src/kv-benchmark.c
${CMAKE_SOURCE_DIR}/src/adlist.c
${CMAKE_SOURCE_DIR}/src/dict.c
${CMAKE_SOURCE_DIR}/src/zmalloc.c
@@ -150,10 +167,12 @@ set(VALKEY_BENCHMARK_SRCS
${CMAKE_SOURCE_DIR}/src/monotonic.c
${CMAKE_SOURCE_DIR}/src/cli_common.c
${CMAKE_SOURCE_DIR}/src/mt19937-64.c
${CMAKE_SOURCE_DIR}/src/strl.c)
${CMAKE_SOURCE_DIR}/src/strl.c
${CMAKE_SOURCE_DIR}/src/fuzzer_client.c
${CMAKE_SOURCE_DIR}/src/fuzzer_command_generator.c)
# valkey-rdma module
set(VALKEY_RDMA_MODULE_SRCS ${CMAKE_SOURCE_DIR}/src/rdma.c)
# kv-rdma module
set(KV_RDMA_MODULE_SRCS ${CMAKE_SOURCE_DIR}/src/rdma.c)
# valkey-tls module
set(VALKEY_TLS_MODULE_SRCS ${CMAKE_SOURCE_DIR}/src/tls.c)
# kv-tls module
set(KV_TLS_MODULE_SRCS ${CMAKE_SOURCE_DIR}/src/tls.c)
+7 -7
View File
@@ -1,5 +1,5 @@
# Return the current host distro name. For example: ubuntu, debian, amzn etc
function (valkey_get_distro_name DISTRO_NAME)
function (kv_get_distro_name DISTRO_NAME)
if (LINUX AND NOT APPLE)
execute_process(
COMMAND /bin/bash "-c" "cat /etc/os-release |grep ^ID=|cut -d = -f 2"
@@ -26,20 +26,20 @@ function (valkey_get_distro_name DISTRO_NAME)
endif ()
endfunction ()
function (valkey_parse_version OUT_MAJOR OUT_MINOR OUT_PATCH)
function (kv_parse_version OUT_MAJOR OUT_MINOR OUT_PATCH)
# Read and parse package version from version.h file
file(STRINGS ${CMAKE_SOURCE_DIR}/src/version.h VERSION_LINES)
foreach (LINE ${VERSION_LINES})
string(FIND "${LINE}" "#define VALKEY_VERSION " VERSION_STR_POS)
string(FIND "${LINE}" "#define KV_VERSION " VERSION_STR_POS)
if (VERSION_STR_POS GREATER -1)
string(REPLACE "#define VALKEY_VERSION " "" LINE "${LINE}")
string(REPLACE "#define KV_VERSION " "" LINE "${LINE}")
string(REPLACE "\"" "" LINE "${LINE}")
# Change "." to ";" to make it a list
string(REPLACE "." ";" LINE "${LINE}")
list(GET LINE 0 _MAJOR)
list(GET LINE 1 _MINOR)
list(GET LINE 2 _PATCH)
message(STATUS "Valkey version: ${_MAJOR}.${_MINOR}.${_PATCH}")
message(STATUS "KV version: ${_MAJOR}.${_MINOR}.${_PATCH}")
# Set the output variables
set(${OUT_MAJOR}
${_MAJOR}
@@ -66,7 +66,7 @@ endfunction ()
# - `yes` | `1` | `on` => return `1`
# - `module` => return `2`
# ~~~
function (valkey_parse_build_option OPTION_VALUE OUT_ARG_ENUM)
function (kv_parse_build_option OPTION_VALUE OUT_ARG_ENUM)
list(APPEND VALID_OPTIONS "yes")
list(APPEND VALID_OPTIONS "1")
list(APPEND VALID_OPTIONS "on")
@@ -101,7 +101,7 @@ function (valkey_parse_build_option OPTION_VALUE OUT_ARG_ENUM)
endif ()
endfunction ()
function (valkey_pkg_config PKGNAME OUT_VARIABLE)
function (kv_pkg_config PKGNAME OUT_VARIABLE)
if (NOT FOUND_PKGCONFIG)
# Locate pkg-config once
find_package(PkgConfig REQUIRED)
+17 -7
View File
@@ -3,7 +3,7 @@ if (USE_JEMALLOC)
endif ()
add_subdirectory(lua)
# Set hiredis options. We need to disable the defaults set in the OPTION(..) we do this by setting them in the CACHE
# Set libkv options. We need to disable the defaults set in the OPTION(..) we do this by setting them in the CACHE
set(BUILD_SHARED_LIBS
OFF
CACHE BOOL "Build shared libraries")
@@ -11,18 +11,28 @@ set(DISABLE_TESTS
ON
CACHE BOOL "If tests should be compiled or not")
if (USE_TLS) # Module or no module
message(STATUS "Building hiredis_ssl")
set(ENABLE_SSL
message(STATUS "Building kv_tls")
set(ENABLE_TLS
ON
CACHE BOOL "Should we test SSL connections")
CACHE BOOL "If TLS support should be compiled or not")
endif ()
if (USE_RDMA) # Module or no module
message(STATUS "Building kv_rdma")
set(ENABLE_RDMA
ON
CACHE BOOL "If RDMA support should be compiled or not")
endif ()
# Let libkv use sds and dict provided by kv.
set(DICT_INCLUDE_DIR ${CMAKE_SOURCE_DIR}/src)
set(SDS_INCLUDE_DIR ${CMAKE_SOURCE_DIR}/src)
add_subdirectory(hiredis)
add_subdirectory(libkv)
add_subdirectory(linenoise)
add_subdirectory(fpconv)
add_subdirectory(hdr_histogram)
# Clear any cached variables passed to hiredis from the cache
# Clear any cached variables passed to libkv from the cache
unset(BUILD_SHARED_LIBS CACHE)
unset(DISABLE_TESTS CACHE)
unset(ENABLE_SSL CACHE)
unset(ENABLE_TLS CACHE)
unset(ENABLE_RDMA CACHE)
+14 -9
View File
@@ -36,7 +36,7 @@ ifneq ($(shell sh -c '[ -f .make-ldflags ] && cat .make-ldflags || echo none'),
endif
distclean:
-(cd hiredis && $(MAKE) clean) > /dev/null || true
-(cd libkv && $(MAKE) clean) > /dev/null || true
-(cd linenoise && $(MAKE) clean) > /dev/null || true
-(cd lua && $(MAKE) clean) > /dev/null || true
-(cd jemalloc && [ -f Makefile ] && $(MAKE) distclean) > /dev/null || true
@@ -47,15 +47,20 @@ distclean:
.PHONY: distclean
LIBKV_MAKE_FLAGS = SDS_INCLUDE_DIR=../../src/ DICT_INCLUDE_DIR=../../src/
ifneq (,$(filter $(BUILD_TLS),yes module))
HIREDIS_MAKE_FLAGS = USE_SSL=1
LIBKV_MAKE_FLAGS += USE_TLS=1
endif
hiredis: .make-prerequisites
@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
cd hiredis && $(MAKE) static $(HIREDIS_MAKE_FLAGS)
ifneq (,$(filter $(BUILD_RDMA),yes module))
LIBKV_MAKE_FLAGS += USE_RDMA=1
endif
.PHONY: hiredis
libkv: .make-prerequisites
@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
cd libkv && $(MAKE) static $(LIBKV_MAKE_FLAGS)
.PHONY: libkv
linenoise: .make-prerequisites
@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
@@ -71,7 +76,7 @@ hdr_histogram: .make-prerequisites
fpconv: .make-prerequisites
@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
cd fpconv && $(MAKE)
cd fpconv && $(MAKE) CFLAGS="-fPIC $(CFLAGS)"
.PHONY: fpconv
@@ -80,12 +85,12 @@ ifeq ($(uname_S),SunOS)
LUA_CFLAGS= -D__C99FEATURES__=1
endif
LUA_CFLAGS+= -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DLUA_USE_MKSTEMP $(CFLAGS)
LUA_CFLAGS+= -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL -DLUA_USE_MKSTEMP $(CFLAGS) -fPIC
LUA_LDFLAGS+= $(LDFLAGS)
ifeq ($(LUA_DEBUG),yes)
LUA_CFLAGS+= -O0 -g -DLUA_USE_APICHECK
else
LUA_CFLAGS+= -O2
LUA_CFLAGS+= -O2
endif
ifeq ($(LUA_COVERAGE),yes)
LUA_CFLAGS += -fprofile-arcs -ftest-coverage
+17 -16
View File
@@ -1,9 +1,9 @@
This directory contains all Valkey dependencies, except for the libc that
This directory contains all KV dependencies, except for the libc that
should be provided by the operating system.
* **Jemalloc** is our memory allocator, used as replacement for libc malloc on Linux by default. It has good performances and excellent fragmentation behavior. This component is upgraded from time to time.
* **hiredis** is the official C client library for Redis. It is used by redis-cli, redis-benchmark and Redis Sentinel. It is part of the Redis official ecosystem but is developed externally from the Redis repository, so we just upgrade it as needed.
* **linenoise** is a readline replacement. It is developed by the same authors of Valkey but is managed as a separated project and updated as needed.
* **libkv** is the official C client library for KV. It is used by kv-cli, kv-benchmark and KV Sentinel. It is managed in a separate project and updated as needed.
* **linenoise** is a readline replacement. It is developed by the same authors of KV but is managed as a separated project and updated as needed.
* **lua** is Lua 5.1 with minor changes for security and additional libraries.
* **hdr_histogram** Used for per-command latency tracking histograms.
* **fast_float** is a replacement for strtod to convert strings to floats efficiently.
@@ -14,10 +14,10 @@ How to upgrade the above dependencies
Jemalloc
---
Jemalloc is modified with changes that allow us to implement the Valkey
active defragmentation logic. However this feature of Valkey is not mandatory
and Valkey is able to understand if the Jemalloc version it is compiled
against supports such Valkey-specific modifications. So in theory, if you
Jemalloc is modified with changes that allow us to implement the KV
active defragmentation logic. However this feature of KV is not mandatory
and KV is able to understand if the Jemalloc version it is compiled
against supports such KV-specific modifications. So in theory, if you
are not interested in the active defragmentation, you can replace Jemalloc
just following these steps:
@@ -29,7 +29,7 @@ just following these steps:
Jemalloc configuration script is broken and will not work nested in another
git repository.
However note that we change Jemalloc settings via the `configure` script of Jemalloc using the `--with-lg-quantum` option, setting it to the value of 3 instead of 4. This provides us with more size classes that better suit the Valkey data structures, in order to gain memory efficiency.
However note that we change Jemalloc settings via the `configure` script of Jemalloc using the `--with-lg-quantum` option, setting it to the value of 3 instead of 4. This provides us with more size classes that better suit the KV data structures, in order to gain memory efficiency.
If you want to upgrade Jemalloc while also providing support for
active defragmentation, in addition to the above steps you need to perform
@@ -39,7 +39,7 @@ the following additional steps:
to add `#define JEMALLOC_FRAG_HINT`.
6. Implement the function `je_get_defrag_hint()` inside `src/jemalloc.c`. You
can see how it is implemented in the current Jemalloc source tree shipped
with Valkey, and rewrite it according to the new Jemalloc internals, if they
with KV, and rewrite it according to the new Jemalloc internals, if they
changed, otherwise you could just copy the old implementation if you are
upgrading just to a similar version of Jemalloc.
@@ -59,20 +59,21 @@ cd deps/jemalloc
4. Update jemalloc's version in `deps/Makefile`: search for "`--with-version=<old-version-tag>-0-g0`" and update it accordingly.
5. Commit the changes (VERSION,configure,Makefile).
Hiredis
Libkv
---
Hiredis is used by Sentinel, `valkey-cli` and `valkey-benchmark`. Like Valkey, uses the SDS string library, but not necessarily the same version. In order to avoid conflicts, this version has all SDS identifiers prefixed by `hi`.
Libkv is used by Sentinel, `kv-cli` and `kv-benchmark`.
The library is built without its own version of the sds and dict type and uses the KV provided variant instead.
1. `git subtree pull --prefix deps/hiredis https://github.com/redis/hiredis.git <version-tag> --squash`<br>
1. `git subtree pull --prefix deps/libkv https://github.com/hanzoai/kv.git <version-tag> --squash`<br>
This should hopefully merge the local changes into the new version.
2. Conflicts will arise (due to our changes) you'll need to resolve them and commit.
2. Commit the changes.
Linenoise
---
Linenoise is rarely upgraded as needed. The upgrade process is trivial since
Valkey uses a non modified version of linenoise, so to upgrade just do the
KV uses a non modified version of linenoise, so to upgrade just do the
following:
1. Remove the linenoise directory.
@@ -82,11 +83,11 @@ Lua
---
We use Lua 5.1 and no upgrade is planned currently, since we don't want to break
Lua scripts for new Lua features: in the context of Valkey Lua scripts the
Lua scripts for new Lua features: in the context of KV Lua scripts the
capabilities of 5.1 are usually more than enough, the release is rock solid,
and we definitely don't want to break old scripts.
So upgrading of Lua is up to the Valkey project maintainers and should be a
So upgrading of Lua is up to the KV project maintainers and should be a
manual procedure performed by taking a diff between the different versions.
Currently we have at least the following differences between official Lua 5.1
+2 -2
View File
@@ -1,7 +1,7 @@
/*
* Copyright Valkey Contributors.
* Copyright (c) Valkey Contributors
* All rights reserved.
* SPDX-License-Identifier: BSD 3-Clause
* SPDX-License-Identifier: BSD-3-Clause
*/
#include "../fast_float/fast_float.h"
+2
View File
@@ -2,3 +2,5 @@ project(fpconv)
set(SRCS "${CMAKE_CURRENT_LIST_DIR}/fpconv_dtoa.c" "${CMAKE_CURRENT_LIST_DIR}/fpconv_dtoa.h")
add_library(fpconv STATIC ${SRCS})
# Add -fPIC
set_target_properties(fpconv PROPERTIES POSITION_INDEPENDENT_CODE ON)
+6 -6
View File
@@ -1,13 +1,13 @@
#ifndef HDR_MALLOC_H__
#define HDR_MALLOC_H__
void *valkey_malloc(size_t size);
void *kv_malloc(size_t size);
void *zcalloc_num(size_t num, size_t size);
void *valkey_realloc(void *ptr, size_t size);
void valkey_free(void *ptr);
void *kv_realloc(void *ptr, size_t size);
void kv_free(void *ptr);
#define hdr_malloc valkey_malloc
#define hdr_malloc kv_malloc
#define hdr_calloc zcalloc_num
#define hdr_realloc valkey_realloc
#define hdr_free valkey_free
#define hdr_realloc kv_realloc
#define hdr_free kv_free
#endif
-177
View File
@@ -1,177 +0,0 @@
name: Build and test
on: [push, pull_request]
jobs:
ubuntu:
name: Ubuntu
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install dependencies
run: |
curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list
sudo apt-get update
sudo apt-get install -y redis-server valgrind libevent-dev
- name: Build using cmake
env:
EXTRA_CMAKE_OPTS: -DENABLE_EXAMPLES:BOOL=ON -DENABLE_SSL:BOOL=ON -DENABLE_SSL_TESTS:BOOL=ON -DENABLE_ASYNC_TESTS:BOOL=ON
CFLAGS: -Werror
CXXFLAGS: -Werror
run: mkdir build && cd build && cmake .. && make
- name: Build using makefile
run: USE_SSL=1 TEST_ASYNC=1 make
- name: Run tests
env:
SKIPS_AS_FAILS: 1
TEST_SSL: 1
run: $GITHUB_WORKSPACE/test.sh
# - name: Run tests under valgrind
# env:
# SKIPS_AS_FAILS: 1
# TEST_PREFIX: valgrind --error-exitcode=99 --track-origins=yes --leak-check=full
# run: $GITHUB_WORKSPACE/test.sh
centos7:
name: CentOS 7
runs-on: ubuntu-latest
container: centos:7
steps:
- uses: actions/checkout@v3
- name: Install dependencies
run: |
yum -y install http://rpms.remirepo.net/enterprise/remi-release-7.rpm
yum -y --enablerepo=remi install redis
yum -y install gcc gcc-c++ make openssl openssl-devel cmake3 valgrind libevent-devel
- name: Build using cmake
env:
EXTRA_CMAKE_OPTS: -DENABLE_EXAMPLES:BOOL=ON -DENABLE_SSL:BOOL=ON -DENABLE_SSL_TESTS:BOOL=ON -DENABLE_ASYNC_TESTS:BOOL=ON
CFLAGS: -Werror
CXXFLAGS: -Werror
run: mkdir build && cd build && cmake3 .. && make
- name: Build using Makefile
run: USE_SSL=1 TEST_ASYNC=1 make
- name: Run tests
env:
SKIPS_AS_FAILS: 1
TEST_SSL: 1
run: $GITHUB_WORKSPACE/test.sh
- name: Run tests under valgrind
env:
SKIPS_AS_FAILS: 1
TEST_SSL: 1
TEST_PREFIX: valgrind --error-exitcode=99 --track-origins=yes --leak-check=full
run: $GITHUB_WORKSPACE/test.sh
centos8:
name: RockyLinux 8
runs-on: ubuntu-latest
container: rockylinux:8
steps:
- uses: actions/checkout@v3
- name: Install dependencies
run: |
dnf -y upgrade --refresh
dnf -y install https://rpms.remirepo.net/enterprise/remi-release-8.rpm
dnf -y module install redis:remi-6.0
dnf -y group install "Development Tools"
dnf -y install openssl-devel cmake valgrind libevent-devel
- name: Build using cmake
env:
EXTRA_CMAKE_OPTS: -DENABLE_EXAMPLES:BOOL=ON -DENABLE_SSL:BOOL=ON -DENABLE_SSL_TESTS:BOOL=ON -DENABLE_ASYNC_TESTS:BOOL=ON
CFLAGS: -Werror
CXXFLAGS: -Werror
run: mkdir build && cd build && cmake .. && make
- name: Build using Makefile
run: USE_SSL=1 TEST_ASYNC=1 make
- name: Run tests
env:
SKIPS_AS_FAILS: 1
TEST_SSL: 1
run: $GITHUB_WORKSPACE/test.sh
- name: Run tests under valgrind
env:
SKIPS_AS_FAILS: 1
TEST_SSL: 1
TEST_PREFIX: valgrind --error-exitcode=99 --track-origins=yes --leak-check=full
run: $GITHUB_WORKSPACE/test.sh
freebsd:
runs-on: macos-13
name: FreeBSD
steps:
- uses: actions/checkout@v3
- name: Build in FreeBSD
uses: vmactions/freebsd-vm@v0
with:
prepare: pkg install -y gmake cmake
run: |
mkdir build && cd build && cmake .. && make && cd ..
gmake
macos:
name: macOS
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- name: Install dependencies
run: |
brew install openssl redis@7.0
brew link redis@7.0 --force
- name: Build hiredis
run: USE_SSL=1 make
- name: Run tests
env:
TEST_SSL: 1
run: $GITHUB_WORKSPACE/test.sh
windows:
name: Windows
runs-on: windows-latest
steps:
- uses: actions/checkout@v3
- name: Install dependencies
run: |
choco install -y ninja memurai-developer
- uses: ilammy/msvc-dev-cmd@v1
- name: Build hiredis
run: |
mkdir build && cd build
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_EXAMPLES=ON
ninja -v
- name: Run tests
run: |
./build/hiredis-test.exe
- name: Install Cygwin Action
uses: cygwin/cygwin-install-action@v2
with:
packages: make git gcc-core
- name: Build in cygwin
env:
HIREDIS_PATH: ${{ github.workspace }}
run: |
make clean && make
-100
View File
@@ -1,100 +0,0 @@
name: C/C++ CI
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
full-build:
name: Build all, plus default examples, run tests against redis
runs-on: ubuntu-latest
env:
# the docker image used by the test.sh
REDIS_DOCKER: redis:alpine
steps:
- name: Install prerequisites
run: sudo apt-get update && sudo apt-get install -y libev-dev libevent-dev libglib2.0-dev libssl-dev valgrind
- uses: actions/checkout@v3
- name: Run make
run: make all examples
- name: Run unittests
run: make check
- name: Run tests with valgrind
env:
TEST_PREFIX: valgrind --error-exitcode=100
SKIPS_ARG: --skip-throughput
run: make check
build-32-bit:
name: Build and test minimal 32 bit linux
runs-on: ubuntu-latest
steps:
- name: Install prerequisites
run: sudo apt-get update && sudo apt-get install gcc-multilib
- uses: actions/checkout@v3
- name: Run make
run: make all
env:
PLATFORM_FLAGS: -m32
- name: Run unittests
env:
REDIS_DOCKER: redis:alpine
run: make check
build-arm:
name: Cross-compile and test arm linux with Qemu
runs-on: ubuntu-latest
strategy:
matrix:
include:
- name: arm
toolset: arm-linux-gnueabi
emulator: qemu-arm
- name: aarch64
toolset: aarch64-linux-gnu
emulator: qemu-aarch64
steps:
- name: Install qemu
if: matrix.emulator
run: sudo apt-get update && sudo apt-get install -y qemu-user
- name: Install platform toolset
if: matrix.toolset
run: sudo apt-get install -y gcc-${{matrix.toolset}}
- uses: actions/checkout@v3
- name: Run make
run: make all
env:
CC: ${{matrix.toolset}}-gcc
AR: ${{matrix.toolset}}-ar
- name: Run unittests
env:
REDIS_DOCKER: redis:alpine
TEST_PREFIX: ${{matrix.emulator}} -L /usr/${{matrix.toolset}}/
run: make check
build-windows:
name: Build and test on windows 64 bit Intel
runs-on: windows-latest
steps:
- uses: microsoft/setup-msbuild@v1.0.2
- uses: actions/checkout@v3
- name: Run CMake (shared lib)
run: cmake -Wno-dev CMakeLists.txt
- name: Build hiredis (shared lib)
run: MSBuild hiredis.vcxproj /p:Configuration=Debug
- name: Run CMake (static lib)
run: cmake -Wno-dev CMakeLists.txt -DBUILD_SHARED_LIBS=OFF
- name: Build hiredis (static lib)
run: MSBuild hiredis.vcxproj /p:Configuration=Debug
- name: Build hiredis-test
run: MSBuild hiredis-test.vcxproj /p:Configuration=Debug
# use memurai, redis compatible server, since it is easy to install. Can't
# install official redis containers on the windows runner
- name: Install Memurai redis server
run: choco install -y memurai-developer.install
- name: Run tests
run: Debug\hiredis-test.exe
-9
View File
@@ -1,9 +0,0 @@
/hiredis-test
/examples/hiredis-example*
/*.o
/*.so
/*.dylib
/*.a
/*.pc
*.dSYM
tags
-125
View File
@@ -1,125 +0,0 @@
language: c
compiler:
- gcc
- clang
os:
- linux
- osx
dist: bionic
branches:
only:
- staging
- trying
- master
- /^release\/.*$/
install:
- if [ "$TRAVIS_COMPILER" != "mingw" ]; then
wget https://github.com/redis/redis/archive/6.0.6.tar.gz;
tar -xzvf 6.0.6.tar.gz;
pushd redis-6.0.6 && BUILD_TLS=yes make && export PATH=$PWD/src:$PATH && popd;
fi;
before_script:
- if [ "$TRAVIS_OS_NAME" == "osx" ]; then
curl -O https://distfiles.macports.org/MacPorts/MacPorts-2.6.2-10.13-HighSierra.pkg;
sudo installer -pkg MacPorts-2.6.2-10.13-HighSierra.pkg -target /;
export PATH=$PATH:/opt/local/bin && sudo port -v selfupdate;
sudo port -N install openssl redis;
fi;
addons:
apt:
packages:
- libc6-dbg
- libc6-dev
- libc6:i386
- libc6-dev-i386
- libc6-dbg:i386
- gcc-multilib
- g++-multilib
- libssl-dev
- libssl-dev:i386
- valgrind
env:
- BITS="32"
- BITS="64"
script:
- EXTRA_CMAKE_OPTS="-DENABLE_EXAMPLES:BOOL=ON -DENABLE_SSL:BOOL=ON -DENABLE_SSL_TESTS:BOOL=ON";
if [ "$TRAVIS_OS_NAME" == "osx" ]; then
if [ "$BITS" == "32" ]; then
CFLAGS="-m32 -Werror";
CXXFLAGS="-m32 -Werror";
LDFLAGS="-m32";
EXTRA_CMAKE_OPTS=;
else
CFLAGS="-Werror";
CXXFLAGS="-Werror";
fi;
else
TEST_PREFIX="valgrind --track-origins=yes --leak-check=full";
if [ "$BITS" == "32" ]; then
CFLAGS="-m32 -Werror";
CXXFLAGS="-m32 -Werror";
LDFLAGS="-m32";
EXTRA_CMAKE_OPTS=;
else
CFLAGS="-Werror";
CXXFLAGS="-Werror";
fi;
fi;
export CFLAGS CXXFLAGS LDFLAGS TEST_PREFIX EXTRA_CMAKE_OPTS
- make && make clean;
if [ "$TRAVIS_OS_NAME" == "osx" ]; then
if [ "$BITS" == "64" ]; then
OPENSSL_PREFIX="$(ls -d /usr/local/Cellar/openssl@1.1/*)" USE_SSL=1 make;
fi;
else
USE_SSL=1 make;
fi;
- mkdir build/ && cd build/
- cmake .. ${EXTRA_CMAKE_OPTS}
- make VERBOSE=1
- if [ "$BITS" == "64" ]; then
TEST_SSL=1 SKIPS_AS_FAILS=1 ctest -V;
else
SKIPS_AS_FAILS=1 ctest -V;
fi;
jobs:
include:
# Windows MinGW cross compile on Linux
- os: linux
dist: xenial
compiler: mingw
addons:
apt:
packages:
- ninja-build
- gcc-mingw-w64-x86-64
- g++-mingw-w64-x86-64
script:
- mkdir build && cd build
- CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_BUILD_WITH_INSTALL_RPATH=on
- ninja -v
# Windows MSVC 2017
- os: windows
compiler: msvc
env:
- MATRIX_EVAL="CC=cl.exe && CXX=cl.exe"
before_install:
- eval "${MATRIX_EVAL}"
install:
- choco install ninja
- choco install -y memurai-developer
script:
- mkdir build && cd build
- cmd.exe //C 'C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Auxiliary\Build\vcvarsall.bat' amd64 '&&'
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_EXAMPLES=ON '&&' ninja -v
- ./hiredis-test.exe
-580
View File
@@ -1,580 +0,0 @@
## [1.2.0](https://github.com/redis/hiredis/tree/v1.2.0) - (2023-06-04)
Announcing Hiredis v1.2.0 with with new adapters, and a great many bug fixes.
## 🚀 New Features
- Add sdevent adapter @Oipo (#1144)
- Allow specifying the keepalive interval @michael-grunder (#1168)
- Add RedisModule adapter @tezc (#1182)
- Helper for setting TCP_USER_TIMEOUT socket option @zuiderkwast (#1188)
## 🐛 Bug Fixes
- Fix a typo in b6a052f. @yossigo (#1190)
- Fix wincrypt symbols conflict @hudayou (#1151)
- Don't attempt to set a timeout if we are in an error state. @michael-grunder (#1180)
- Accept -nan per the RESP3 spec recommendation. @michael-grunder (#1178)
- Fix colliding option values @zuiderkwast (#1172)
- Ensure functionality without `_MSC_VER` definition @windyakin (#1194)
## 🧰 Maintenance
- Add a test for the TCP_USER_TIMEOUT option. @michael-grunder (#1192)
- Add -Werror as a default. @yossigo (#1193)
- CI: Update homebrew Redis version. @yossigo (#1191)
- Fix typo in makefile. @michael-grunder (#1179)
- Write a version file for the CMake package @Neverlord (#1165)
- CMakeLists.txt: respect BUILD_SHARED_LIBS @ffontaine (#1147)
- Cmake static or shared @autoantwort (#1160)
- fix typo @tillkruss (#1153)
- Add a test ensuring we don't clobber connection error. @michael-grunder (#1181)
- Search for openssl on macOS @michael-grunder (#1169)
## Contributors
We'd like to thank all the contributors who worked on this release!
<a href="https://github.com/neverlord"><img src="https://github.com/neverlord.png" width="32" height="32"></a>
<a href="https://github.com/Oipo"><img src="https://github.com/Oipo.png" width="32" height="32"></a>
<a href="https://github.com/autoantwort"><img src="https://github.com/autoantwort.png" width="32" height="32"></a>
<a href="https://github.com/ffontaine"><img src="https://github.com/ffontaine.png" width="32" height="32"></a>
<a href="https://github.com/hudayou"><img src="https://github.com/hudayou.png" width="32" height="32"></a>
<a href="https://github.com/michael-grunder"><img src="https://github.com/michael-grunder.png" width="32" height="32"></a>
<a href="https://github.com/postgraph"><img src="https://github.com/postgraph.png" width="32" height="32"></a>
<a href="https://github.com/tezc"><img src="https://github.com/tezc.png" width="32" height="32"></a>
<a href="https://github.com/tillkruss"><img src="https://github.com/tillkruss.png" width="32" height="32"></a>
<a href="https://github.com/vityafx"><img src="https://github.com/vityafx.png" width="32" height="32"></a>
<a href="https://github.com/windyakin"><img src="https://github.com/windyakin.png" width="32" height="32"></a>
<a href="https://github.com/yossigo"><img src="https://github.com/yossigo.png" width="32" height="32"></a>
<a href="https://github.com/zuiderkwast"><img src="https://github.com/zuiderkwast.png" width="32" height="32"></a>
## [1.1.0](https://github.com/redis/hiredis/tree/v1.1.0) - (2022-11-15)
Announcing Hiredis v1.1.0 GA with better SSL convenience, new async adapters and a great many bug fixes.
**NOTE**: Hiredis can now return `nan` in addition to `-inf` and `inf` when returning a `REDIS_REPLY_DOUBLE`.
## 🐛 Bug Fixes
- Add support for nan in RESP3 double [@filipecosta90](https://github.com/filipecosta90)
([\#1133](https://github.com/redis/hiredis/pull/1133))
## 🧰 Maintenance
- Add an example that calls redisCommandArgv [@michael-grunder](https://github.com/michael-grunder)
([\#1140](https://github.com/redis/hiredis/pull/1140))
- fix flag reference [@pata00](https://github.com/pata00) ([\#1136](https://github.com/redis/hiredis/pull/1136))
- Make freeing a NULL redisAsyncContext a no op. [@michael-grunder](https://github.com/michael-grunder)
([\#1135](https://github.com/redis/hiredis/pull/1135))
- CI updates ([@bjosv](https://github.com/redis/bjosv) ([\#1139](https://github.com/redis/hiredis/pull/1139))
## Contributors
We'd like to thank all the contributors who worked on this release!
<a href="https://github.com/bjsov"><img src="https://github.com/bjosv.png" width="32" height="32"></a>
<a href="https://github.com/filipecosta90"><img src="https://github.com/filipecosta90.png" width="32" height="32"></a>
<a href="https://github.com/michael-grunder"><img src="https://github.com/michael-grunder.png" width="32" height="32"></a>
<a href="https://github.com/pata00"><img src="https://github.com/pata00.png" width="32" height="32"></a>
## [1.1.0-rc1](https://github.com/redis/hiredis/tree/v1.1.0-rc1) - (2022-11-06)
Announcing Hiredis v1.1.0-rc1, with better SSL convenience, new async adapters, and a great many bug fixes.
## 🚀 New Features
- Add possibility to prefer IPv6, IPv4 or unspecified [@zuiderkwast](https://github.com/zuiderkwast)
([\#1096](https://github.com/redis/hiredis/pull/1096))
- Add adapters/libhv [@ithewei](https://github.com/ithewei) ([\#904](https://github.com/redis/hiredis/pull/904))
- Add timeout support to libhv adapter. [@michael-grunder](https://github.com/michael-grunder) ([\#1109](https://github.com/redis/hiredis/pull/1109))
- set default SSL verification path [@adobeturchenko](https://github.com/adobeturchenko) ([\#928](https://github.com/redis/hiredis/pull/928))
- Introduce .close method for redisContextFuncs [@pizhenwei](https://github.com/pizhenwei) ([\#1094](https://github.com/redis/hiredis/pull/1094))
- Make it possible to set SSL verify mode [@stanhu](https://github.com/stanhu) ([\#1085](https://github.com/redis/hiredis/pull/1085))
- Polling adapter and example [@kristjanvalur](https://github.com/kristjanvalur) ([\#932](https://github.com/redis/hiredis/pull/932))
- Unsubscribe handling in async [@bjosv](https://github.com/bjosv) ([\#1047](https://github.com/redis/hiredis/pull/1047))
- Add timeout support for libuv adapter [@MichaelSuen-thePointer](https://github.com/@MichaelSuenthePointer) ([\#1016](https://github.com/redis/hiredis/pull/1016))
## 🐛 Bug Fixes
- Update for MinGW cross compile [@bit0fun](https://github.com/bit0fun) ([\#1127](https://github.com/redis/hiredis/pull/1127))
- fixed CPP build error with adapters/libhv.h [@mtdxc](https://github.com/mtdxc) ([\#1125](https://github.com/redis/hiredis/pull/1125))
- Fix protocol error
[@michael-grunder](https://github.com/michael-grunder),
[@mtuleika-appcast](https://github.com/mtuleika-appcast) ([\#1106](https://github.com/redis/hiredis/pull/1106))
- Use a windows specific keepalive function. [@michael-grunder](https://github.com/michael-grunder) ([\#1104](https://github.com/redis/hiredis/pull/1104))
- Fix CMake config path on Linux. [@xkszltl](https://github.com/xkszltl) ([\#989](https://github.com/redis/hiredis/pull/989))
- Fix potential fault at createDoubleObject [@afcidk](https://github.com/afcidk) ([\#964](https://github.com/redis/hiredis/pull/964))
- Fix some undefined behavior [@jengab](https://github.com/jengab) ([\#1091](https://github.com/redis/hiredis/pull/1091))
- Copy OOM errors to redisAsyncContext when finding subscribe callback [@bjosv](https://github.com/bjosv) ([\#1090](https://github.com/redis/hiredis/pull/1090))
- Maintain backward compatibility with our onConnect callback. [@michael-grunder](https://github.com/michael-grunder) ([\#1087](https://github.com/redis/hiredis/pull/1087))
- Fix PUSH handler tests for Redis >= 7.0.5 [@michael-grunder](https://github.com/michael-grunder) ([\#1121](https://github.com/redis/hiredis/pull/1121))
- fix heap-buffer-overflow [@zhangtaoXT5](https://github.com/zhangtaoXT5) ([\#957](https://github.com/redis/hiredis/pull/957))
- Fix heap-buffer-overflow issue in redisvFormatCommad [@bjosv](https://github.com/bjosv) ([\#1097](https://github.com/redis/hiredis/pull/1097))
- Polling adapter requires sockcompat.h [@michael-grunder](https://github.com/michael-grunder) ([\#1095](https://github.com/redis/hiredis/pull/1095))
- Illumos test fixes, error message difference for bad hostname test. [@devnexen](https://github.com/devnexen) ([\#901](https://github.com/redis/hiredis/pull/901))
- Remove semicolon after do-while in \_EL\_CLEANUP [@sundb](https://github.com/sundb) ([\#905](https://github.com/redis/hiredis/pull/905))
- Stability: Support calling redisAsyncCommand and redisAsyncDisconnect from the onConnected callback [@kristjanvalur](https://github.com/kristjanvalur)
([\#931](https://github.com/redis/hiredis/pull/931))
- Fix async connect on Windows [@kristjanvalur](https://github.com/kristjanvalur) ([\#1073](https://github.com/redis/hiredis/pull/1073))
- Fix tests so they work for Redis 7.0 [@michael-grunder](https://github.com/michael-grunder) ([\#1072](https://github.com/redis/hiredis/pull/1072))
- Fix warnings on Win64 [@orgads](https://github.com/orgads) ([\#1058](https://github.com/redis/hiredis/pull/1058))
- Handle push notifications before or after reply. [@yossigo](https://github.com/yossigo) ([\#1062](https://github.com/redis/hiredis/pull/1062))
- Update hiredis sds with improvements found in redis [@bjosv](https://github.com/bjosv) ([\#1045](https://github.com/redis/hiredis/pull/1045))
- Avoid incorrect call to the previous reply's callback [@bjosv](https://github.com/bjosv) ([\#1040](https://github.com/redis/hiredis/pull/1040))
- fix building on AIX and SunOS [\#1031](https://github.com/redis/hiredis/pull/1031) ([@scddev](https://github.com/scddev))
- Allow sending commands after sending an unsubscribe [@bjosv](https://github.com/bjosv) ([\#1036](https://github.com/redis/hiredis/pull/1036))
- Correction for command timeout during pubsub [@bjosv](https://github.com/bjosv) ([\#1038](https://github.com/redis/hiredis/pull/1038))
- Fix adapters/libevent.h compilation for 64-bit Windows [@pbtummillo](https://github.com/pbtummillo) ([\#937](https://github.com/redis/hiredis/pull/937))
- Fix integer overflow when format command larger than 4GB [@sundb](https://github.com/sundb) ([\#1030](https://github.com/redis/hiredis/pull/1030))
- Handle array response during subscribe in RESP3 [@bjosv](https://github.com/bjosv) ([\#1014](https://github.com/redis/hiredis/pull/1014))
- Support PING while subscribing (RESP2) [@bjosv](https://github.com/bjosv) ([\#1027](https://github.com/redis/hiredis/pull/1027))
## 🧰 Maintenance
- CI fixes in preparation of release [@michael-grunder](https://github.com/michael-grunder) ([\#1130](https://github.com/redis/hiredis/pull/1130))
- Add do while(0) (protection for macros [@afcidk](https://github.com/afcidk) [\#959](https://github.com/redis/hiredis/pull/959))
- Fixup of PR734: Coverage of hiredis.c [@bjosv](https://github.com/bjosv) ([\#1124](https://github.com/redis/hiredis/pull/1124))
- CMake corrections for building on Windows [@bjosv](https://github.com/bjosv) ([\#1122](https://github.com/redis/hiredis/pull/1122))
- Install on windows fixes [@bjosv](https://github.com/bjosv) ([\#1117](https://github.com/redis/hiredis/pull/1117))
- Add libhv example to our standard Makefile [@michael-grunder](https://github.com/michael-grunder) ([\#1108](https://github.com/redis/hiredis/pull/1108))
- Additional include directory given by pkg-config [@bjosv](https://github.com/bjosv) ([\#1118](https://github.com/redis/hiredis/pull/1118))
- Use __attribute__ when building with Clang on Windows [@bjosv](https://github.com/bjosv) ([\#1115](https://github.com/redis/hiredis/pull/1115))
- Minor refactor [@michael-grunder](https://github.com/michael-grunder) ([\#1110](https://github.com/redis/hiredis/pull/1110))
- Fix pkgconfig result for hiredis_ssl [@bjosv](https://github.com/bjosv) ([\#1107](https://github.com/redis/hiredis/pull/1107))
- Update documentation to explain redisConnectWithOptions. [@michael-grunder](https://github.com/michael-grunder) ([\#1099](https://github.com/redis/hiredis/pull/1099))
- uvadapter: reduce number of uv_poll_start calls [@noxiouz](https://github.com/noxiouz) ([\#1098](https://github.com/redis/hiredis/pull/1098))
- Regression test for off-by-one parsing error [@bugwz](https://github.com/bugwz) ([\#1092](https://github.com/redis/hiredis/pull/1092))
- CMake: remove dict.c form hiredis_sources [@Lipraxde](https://github.com/Lipraxde) ([\#1055](https://github.com/redis/hiredis/pull/1055))
- Do store command timeout in the context for redisSetTimeout [@catterer](https://github.com/catterer) ([\#593](https://github.com/redis/hiredis/pull/593), [\#1093](https://github.com/redis/hiredis/pull/1093))
- Add GitHub Actions CI workflow for hiredis: Arm, Arm64, 386, windows. [@kristjanvalur](https://github.com/kristjanvalur) ([\#943](https://github.com/redis/hiredis/pull/943))
- CI: bump macOS runner version [@SukkaW](https://github.com/SukkaW) ([\#1079](https://github.com/redis/hiredis/pull/1079))
- Support for generating release notes [@chayim](https://github.com/chayim) ([\#1083](https://github.com/redis/hiredis/pull/1083))
- Improve example for SSL initialization in README.md [@stanhu](https://github.com/stanhu) ([\#1084](https://github.com/redis/hiredis/pull/1084))
- Fix README typos [@bjosv](https://github.com/bjosv) ([\#1080](https://github.com/redis/hiredis/pull/1080))
- fix cmake version [@smmir-cent](https://github.com/@smmircent) ([\#1050](https://github.com/redis/hiredis/pull/1050))
- Use the same name for static and shared libraries [@orgads](https://github.com/orgads) ([\#1057](https://github.com/redis/hiredis/pull/1057))
- Embed debug information in windows static .lib file [@kristjanvalur](https://github.com/kristjanvalur) ([\#1054](https://github.com/redis/hiredis/pull/1054))
- Improved async documentation [@kristjanvalur](https://github.com/kristjanvalur) ([\#1074](https://github.com/redis/hiredis/pull/1074))
- Use official repository for redis package. [@yossigo](https://github.com/yossigo) ([\#1061](https://github.com/redis/hiredis/pull/1061))
- Whitelist hiredis repo path in cygwin [@michael-grunder](https://github.com/michael-grunder) ([\#1063](https://github.com/redis/hiredis/pull/1063))
- CentOS 8 is EOL, switch to RockyLinux [@michael-grunder](https://github.com/michael-grunder) ([\#1046](https://github.com/redis/hiredis/pull/1046))
- CMakeLists.txt: allow building without a C++ compiler [@ffontaine](https://github.com/ffontaine) ([\#872](https://github.com/redis/hiredis/pull/872))
- Makefile: move SSL options into a block and refine rules [@pizhenwei](https://github.com/pizhenwei) ([\#997](https://github.com/redis/hiredis/pull/997))
- Update CMakeLists.txt for more portability [@EricDeng1001](https://github.com/EricDeng1001) ([\#1005](https://github.com/redis/hiredis/pull/1005))
- FreeBSD build fixes + CI [@michael-grunder](https://github.com/michael-grunder) ([\#1026](https://github.com/redis/hiredis/pull/1026))
- Add asynchronous test for pubsub using RESP3 [@bjosv](https://github.com/bjosv) ([\#1012](https://github.com/redis/hiredis/pull/1012))
- Trigger CI failure when Valgrind issues are found [@bjosv](https://github.com/bjosv) ([\#1011](https://github.com/redis/hiredis/pull/1011))
- Move to using make directly in Cygwin [@michael-grunder](https://github.com/michael-grunder) ([\#1020](https://github.com/redis/hiredis/pull/1020))
- Add asynchronous API tests [@bjosv](https://github.com/bjosv) ([\#1010](https://github.com/redis/hiredis/pull/1010))
- Correcting the build target `coverage` for enabled SSL [@bjosv](https://github.com/bjosv) ([\#1009](https://github.com/redis/hiredis/pull/1009))
- GH Actions: Run SSL tests during CI [@bjosv](https://github.com/bjosv) ([\#1008](https://github.com/redis/hiredis/pull/1008))
- GH: Actions - Add valgrind and CMake [@michael-grunder](https://github.com/michael-grunder) ([\#1004](https://github.com/redis/hiredis/pull/1004))
- Add Centos8 tests in GH Actions [@michael-grunder](https://github.com/michael-grunder) ([\#1001](https://github.com/redis/hiredis/pull/1001))
- We should run actions on PRs [@michael-grunder](https://github.com/michael-grunder) (([\#1000](https://github.com/redis/hiredis/pull/1000))
- Add Cygwin test in GitHub actions [@michael-grunder](https://github.com/michael-grunder) ([\#999](https://github.com/redis/hiredis/pull/999))
- Add Windows tests in GitHub actions [@michael-grunder](https://github.com/michael-grunder) ([\#996](https://github.com/redis/hiredis/pull/996))
- Switch to GitHub actions [@michael-grunder](https://github.com/michael-grunder) ([\#995](https://github.com/redis/hiredis/pull/995))
- Minor refactor of CVE-2021-32765 fix. [@michael-grunder](https://github.com/michael-grunder) ([\#993](https://github.com/redis/hiredis/pull/993))
- Remove extra comma from CMake var. [@xkszltl](https://github.com/xkszltl) ([\#988](https://github.com/redis/hiredis/pull/988))
- Add REDIS\_OPT\_PREFER\_UNSPEC [@michael-grunder](https://github.com/michael-grunder) ([\#1101](https://github.com/redis/hiredis/pull/1101))
## Contributors
We'd like to thank all the contributors who worked on this release!
<a href="https://github.com/EricDeng1001"><img src="https://github.com/EricDeng1001.png" width="32" height="32"></a>
<a href="https://github.com/Lipraxde"><img src="https://github.com/Lipraxde.png" width="32" height="32"></a>
<a href="https://github.com/MichaelSuen-thePointer"><img src="https://github.com/MichaelSuen-thePointer.png" width="32" height="32"></a>
<a href="https://github.com/SukkaW"><img src="https://github.com/SukkaW.png" width="32" height="32"></a>
<a href="https://github.com/adobeturchenko"><img src="https://github.com/adobeturchenko.png" width="32" height="32"></a>
<a href="https://github.com/afcidk"><img src="https://github.com/afcidk.png" width="32" height="32"></a>
<a href="https://github.com/bit0fun"><img src="https://github.com/bit0fun.png" width="32" height="32"></a>
<a href="https://github.com/bjosv"><img src="https://github.com/bjosv.png" width="32" height="32"></a>
<a href="https://github.com/bugwz"><img src="https://github.com/bugwz.png" width="32" height="32"></a>
<a href="https://github.com/catterer"><img src="https://github.com/catterer.png" width="32" height="32"></a>
<a href="https://github.com/chayim"><img src="https://github.com/chayim.png" width="32" height="32"></a>
<a href="https://github.com/devnexen"><img src="https://github.com/devnexen.png" width="32" height="32"></a>
<a href="https://github.com/ffontaine"><img src="https://github.com/ffontaine.png" width="32" height="32"></a>
<a href="https://github.com/ithewei"><img src="https://github.com/ithewei.png" width="32" height="32"></a>
<a href="https://github.com/jengab"><img src="https://github.com/jengab.png" width="32" height="32"></a>
<a href="https://github.com/kristjanvalur"><img src="https://github.com/kristjanvalur.png" width="32" height="32"></a>
<a href="https://github.com/michael-grunder"><img src="https://github.com/michael-grunder.png" width="32" height="32"></a>
<a href="https://github.com/noxiouz"><img src="https://github.com/noxiouz.png" width="32" height="32"></a>
<a href="https://github.com/mtdxc"><img src="https://github.com/mtdxc.png" width="32" height="32"></a>
<a href="https://github.com/orgads"><img src="https://github.com/orgads.png" width="32" height="32"></a>
<a href="https://github.com/pbtummillo"><img src="https://github.com/pbtummillo.png" width="32" height="32"></a>
<a href="https://github.com/pizhenwei"><img src="https://github.com/pizhenwei.png" width="32" height="32"></a>
<a href="https://github.com/scddev"><img src="https://github.com/scddev.png" width="32" height="32"></a>
<a href="https://github.com/smmir-cent"><img src="https://github.com/smmir-cent.png" width="32" height="32"></a>
<a href="https://github.com/stanhu"><img src="https://github.com/stanhu.png" width="32" height="32"></a>
<a href="https://github.com/sundb"><img src="https://github.com/sundb.png" width="32" height="32"></a>
<a href="https://github.com/vturchenko"><img src="https://github.com/vturchenko.png" width="32" height="32"></a>
<a href="https://github.com/xkszltl"><img src="https://github.com/xkszltl.png" width="32" height="32"></a>
<a href="https://github.com/yossigo"><img src="https://github.com/yossigo.png" width="32" height="32"></a>
<a href="https://github.com/zhangtaoXT5"><img src="https://github.com/zhangtaoXT5.png" width="32" height="32"></a>
<a href="https://github.com/zuiderkwast"><img src="https://github.com/zuiderkwast.png" width="32" height="32"></a>
## [1.0.2](https://github.com/redis/hiredis/tree/v1.0.2) - (2021-10-07)
Announcing Hiredis v1.0.2, which fixes CVE-2021-32765 but returns the SONAME to the correct value of `1.0.0`.
- [Revert SONAME bump](https://github.com/redis/hiredis/commit/d4e6f109a064690cde64765c654e679fea1d3548)
([Michael Grunder](https://github.com/michael-grunder))
## [1.0.1](https://github.com/redis/hiredis/tree/v1.0.1) - (2021-10-04)
<span style="color:red">This release erroneously bumped the SONAME, please use [1.0.2](https://github.com/redis/hiredis/tree/v1.0.2)</span>
Announcing Hiredis v1.0.1, a security release fixing CVE-2021-32765
- Fix for [CVE-2021-32765](https://github.com/redis/hiredis/security/advisories/GHSA-hfm9-39pp-55p2)
[commit](https://github.com/redis/hiredis/commit/76a7b10005c70babee357a7d0f2becf28ec7ed1e)
([Yossi Gottlieb](https://github.com/yossigo))
_Thanks to [Yossi Gottlieb](https://github.com/yossigo) for the security fix and to [Microsoft Security Vulnerability Research](https://www.microsoft.com/en-us/msrc/msvr) for finding the bug._ :sparkling_heart:
## [1.0.0](https://github.com/redis/hiredis/tree/v1.0.0) - (2020-08-03)
Announcing Hiredis v1.0.0, which adds support for RESP3, SSL connections, allocator injection, and better Windows support! :tada:
_A big thanks to everyone who helped with this release. The following list includes everyone who contributed at least five lines, sorted by lines contributed._ :sparkling_heart:
[Michael Grunder](https://github.com/michael-grunder), [Yossi Gottlieb](https://github.com/yossigo),
[Mark Nunberg](https://github.com/mnunberg), [Marcus Geelnard](https://github.com/mbitsnbites),
[Justin Brewer](https://github.com/justinbrewer), [Valentino Geron](https://github.com/valentinogeron),
[Minun Dragonation](https://github.com/dragonation), [Omri Steiner](https://github.com/OmriSteiner),
[Sangmoon Yi](https://github.com/jman-krafton), [Jinjiazh](https://github.com/jinjiazhang),
[Odin Hultgren Van Der Horst](https://github.com/Miniwoffer), [Muhammad Zahalqa](https://github.com/tryfinally),
[Nick Rivera](https://github.com/heronr), [Qi Yang](https://github.com/movebean),
[kevin1018](https://github.com/kevin1018)
[Full Changelog](https://github.com/redis/hiredis/compare/v0.14.1...v1.0.0)
**BREAKING CHANGES**:
* `redisOptions` now has two timeout fields. One for connecting, and one for commands. If you're presently using `options->timeout` you will need to change it to use `options->connect_timeout`. (See [example](https://github.com/redis/hiredis/commit/38b5ae543f5c99eb4ccabbe277770fc6bc81226f#diff-86ba39d37aa829c8c82624cce4f049fbL36))
* Bulk and multi-bulk lengths less than -1 or greater than `LLONG_MAX` are now protocol errors. This is consistent
with the RESP specification. On 32-bit platforms, the upper bound is lowered to `SIZE_MAX`.
* `redisReplyObjectFunctions.createArray` now takes `size_t` for its length parameter.
**New features:**
- Support for RESP3
[\#697](https://github.com/redis/hiredis/pull/697),
[\#805](https://github.com/redis/hiredis/pull/805),
[\#819](https://github.com/redis/hiredis/pull/819),
[\#841](https://github.com/redis/hiredis/pull/841)
([Yossi Gottlieb](https://github.com/yossigo), [Michael Grunder](https://github.com/michael-grunder))
- Support for SSL connections
[\#645](https://github.com/redis/hiredis/pull/645),
[\#699](https://github.com/redis/hiredis/pull/699),
[\#702](https://github.com/redis/hiredis/pull/702),
[\#708](https://github.com/redis/hiredis/pull/708),
[\#711](https://github.com/redis/hiredis/pull/711),
[\#821](https://github.com/redis/hiredis/pull/821),
[more](https://github.com/redis/hiredis/pulls?q=is%3Apr+is%3Amerged+SSL)
([Mark Nunberg](https://github.com/mnunberg), [Yossi Gottlieb](https://github.com/yossigo))
- Run-time allocator injection
[\#800](https://github.com/redis/hiredis/pull/800)
([Michael Grunder](https://github.com/michael-grunder))
- Improved Windows support (including MinGW and Windows CI)
[\#652](https://github.com/redis/hiredis/pull/652),
[\#663](https://github.com/redis/hiredis/pull/663)
([Marcus Geelnard](https://www.bitsnbites.eu/author/m/))
- Adds support for distinct connect and command timeouts
[\#839](https://github.com/redis/hiredis/pull/839),
[\#829](https://github.com/redis/hiredis/pull/829)
([Valentino Geron](https://github.com/valentinogeron))
- Add generic pointer and destructor to `redisContext` that users can use for context.
[\#855](https://github.com/redis/hiredis/pull/855)
([Michael Grunder](https://github.com/michael-grunder))
**Closed issues (that involved code changes):**
- Makefile does not install TLS libraries [\#809](https://github.com/redis/hiredis/issues/809)
- redisConnectWithOptions should not set command timeout [\#722](https://github.com/redis/hiredis/issues/722), [\#829](https://github.com/redis/hiredis/pull/829) ([valentinogeron](https://github.com/valentinogeron))
- Fix integer overflow in `sdsrange` [\#827](https://github.com/redis/hiredis/issues/827)
- INFO & CLUSTER commands failed when using RESP3 [\#802](https://github.com/redis/hiredis/issues/802)
- Windows compatibility patches [\#687](https://github.com/redis/hiredis/issues/687), [\#838](https://github.com/redis/hiredis/issues/838), [\#842](https://github.com/redis/hiredis/issues/842)
- RESP3 PUSH messages incorrectly use pending callback [\#825](https://github.com/redis/hiredis/issues/825)
- Asynchronous PSUBSCRIBE command fails when using RESP3 [\#815](https://github.com/redis/hiredis/issues/815)
- New SSL API [\#804](https://github.com/redis/hiredis/issues/804), [\#813](https://github.com/redis/hiredis/issues/813)
- Hard-coded limit of nested reply depth [\#794](https://github.com/redis/hiredis/issues/794)
- Fix TCP_NODELAY in Windows/OSX [\#679](https://github.com/redis/hiredis/issues/679), [\#690](https://github.com/redis/hiredis/issues/690), [\#779](https://github.com/redis/hiredis/issues/779), [\#785](https://github.com/redis/hiredis/issues/785),
- Added timers to libev adapter. [\#778](https://github.com/redis/hiredis/issues/778), [\#795](https://github.com/redis/hiredis/pull/795)
- Initialization discards const qualifier [\#777](https://github.com/redis/hiredis/issues/777)
- \[BUG\]\[MinGW64\] Error setting socket timeout [\#775](https://github.com/redis/hiredis/issues/775)
- undefined reference to hi_malloc [\#769](https://github.com/redis/hiredis/issues/769)
- hiredis pkg-config file incorrectly ignores multiarch libdir spec'n [\#767](https://github.com/redis/hiredis/issues/767)
- Don't use -G to build shared object on Solaris [\#757](https://github.com/redis/hiredis/issues/757)
- error when make USE\_SSL=1 [\#748](https://github.com/redis/hiredis/issues/748)
- Allow to change SSL Mode [\#646](https://github.com/redis/hiredis/issues/646)
- hiredis/adapters/libevent.h memleak [\#618](https://github.com/redis/hiredis/issues/618)
- redisLibuvPoll crash when server closes the connetion [\#545](https://github.com/redis/hiredis/issues/545)
- about redisAsyncDisconnect question [\#518](https://github.com/redis/hiredis/issues/518)
- hiredis adapters libuv error for help [\#508](https://github.com/redis/hiredis/issues/508)
- API/ABI changes analysis [\#506](https://github.com/redis/hiredis/issues/506)
- Memory leak patch in Redis [\#502](https://github.com/redis/hiredis/issues/502)
- Remove the depth limitation [\#421](https://github.com/redis/hiredis/issues/421)
**Merged pull requests:**
- Move SSL management to a distinct private pointer [\#855](https://github.com/redis/hiredis/pull/855) ([michael-grunder](https://github.com/michael-grunder))
- Move include to sockcompat.h to maintain style [\#850](https://github.com/redis/hiredis/pull/850) ([michael-grunder](https://github.com/michael-grunder))
- Remove erroneous tag and add license to push example [\#849](https://github.com/redis/hiredis/pull/849) ([michael-grunder](https://github.com/michael-grunder))
- fix windows compiling with mingw [\#848](https://github.com/redis/hiredis/pull/848) ([rmalizia44](https://github.com/rmalizia44))
- Some Windows quality of life improvements. [\#846](https://github.com/redis/hiredis/pull/846) ([michael-grunder](https://github.com/michael-grunder))
- Use \_WIN32 define instead of WIN32 [\#845](https://github.com/redis/hiredis/pull/845) ([michael-grunder](https://github.com/michael-grunder))
- Non Linux CI fixes [\#844](https://github.com/redis/hiredis/pull/844) ([michael-grunder](https://github.com/michael-grunder))
- Resp3 oob push support [\#841](https://github.com/redis/hiredis/pull/841) ([michael-grunder](https://github.com/michael-grunder))
- fix \#785: defer TCP\_NODELAY in async tcp connections [\#836](https://github.com/redis/hiredis/pull/836) ([OmriSteiner](https://github.com/OmriSteiner))
- sdsrange overflow fix [\#830](https://github.com/redis/hiredis/pull/830) ([michael-grunder](https://github.com/michael-grunder))
- Use explicit pointer casting for c++ compatibility [\#826](https://github.com/redis/hiredis/pull/826) ([aureus1](https://github.com/aureus1))
- Document allocator injection and completeness fix in test.c [\#824](https://github.com/redis/hiredis/pull/824) ([michael-grunder](https://github.com/michael-grunder))
- Use unique names for allocator struct members [\#823](https://github.com/redis/hiredis/pull/823) ([michael-grunder](https://github.com/michael-grunder))
- New SSL API to replace redisSecureConnection\(\). [\#821](https://github.com/redis/hiredis/pull/821) ([yossigo](https://github.com/yossigo))
- Add logic to handle RESP3 push messages [\#819](https://github.com/redis/hiredis/pull/819) ([michael-grunder](https://github.com/michael-grunder))
- Use standrad isxdigit instead of custom helper function. [\#814](https://github.com/redis/hiredis/pull/814) ([tryfinally](https://github.com/tryfinally))
- Fix missing SSL build/install options. [\#812](https://github.com/redis/hiredis/pull/812) ([yossigo](https://github.com/yossigo))
- Add link to ABI tracker [\#808](https://github.com/redis/hiredis/pull/808) ([michael-grunder](https://github.com/michael-grunder))
- Resp3 verbatim string support [\#805](https://github.com/redis/hiredis/pull/805) ([michael-grunder](https://github.com/michael-grunder))
- Allow users to replace allocator and handle OOM everywhere. [\#800](https://github.com/redis/hiredis/pull/800) ([michael-grunder](https://github.com/michael-grunder))
- Remove nested depth limitation. [\#797](https://github.com/redis/hiredis/pull/797) ([michael-grunder](https://github.com/michael-grunder))
- Attempt to fix compilation on Solaris [\#796](https://github.com/redis/hiredis/pull/796) ([michael-grunder](https://github.com/michael-grunder))
- Support timeouts in libev adapater [\#795](https://github.com/redis/hiredis/pull/795) ([michael-grunder](https://github.com/michael-grunder))
- Fix pkgconfig when installing to a custom lib dir [\#793](https://github.com/redis/hiredis/pull/793) ([michael-grunder](https://github.com/michael-grunder))
- Fix USE\_SSL=1 make/cmake on OSX and CMake tests [\#789](https://github.com/redis/hiredis/pull/789) ([michael-grunder](https://github.com/michael-grunder))
- Use correct libuv call on Windows [\#784](https://github.com/redis/hiredis/pull/784) ([michael-grunder](https://github.com/michael-grunder))
- Added CMake package config and fixed hiredis\_ssl on Windows [\#783](https://github.com/redis/hiredis/pull/783) ([michael-grunder](https://github.com/michael-grunder))
- CMake: Set hiredis\_ssl shared object version. [\#780](https://github.com/redis/hiredis/pull/780) ([yossigo](https://github.com/yossigo))
- Win32 tests and timeout fix [\#776](https://github.com/redis/hiredis/pull/776) ([michael-grunder](https://github.com/michael-grunder))
- Provides an optional cleanup callback for async data. [\#768](https://github.com/redis/hiredis/pull/768) ([heronr](https://github.com/heronr))
- Housekeeping fixes [\#764](https://github.com/redis/hiredis/pull/764) ([michael-grunder](https://github.com/michael-grunder))
- install alloc.h [\#756](https://github.com/redis/hiredis/pull/756) ([ch1aki](https://github.com/ch1aki))
- fix spelling mistakes [\#746](https://github.com/redis/hiredis/pull/746) ([ShooterIT](https://github.com/ShooterIT))
- Free the reply in redisGetReply when passed NULL [\#741](https://github.com/redis/hiredis/pull/741) ([michael-grunder](https://github.com/michael-grunder))
- Fix dead code in sslLogCallback relating to should\_log variable. [\#737](https://github.com/redis/hiredis/pull/737) ([natoscott](https://github.com/natoscott))
- Fix typo in dict.c. [\#731](https://github.com/redis/hiredis/pull/731) ([Kevin-Xi](https://github.com/Kevin-Xi))
- Adding an option to DISABLE\_TESTS [\#727](https://github.com/redis/hiredis/pull/727) ([pbotros](https://github.com/pbotros))
- Update README with SSL support. [\#720](https://github.com/redis/hiredis/pull/720) ([yossigo](https://github.com/yossigo))
- Fixes leaks in unit tests [\#715](https://github.com/redis/hiredis/pull/715) ([michael-grunder](https://github.com/michael-grunder))
- SSL Tests [\#711](https://github.com/redis/hiredis/pull/711) ([yossigo](https://github.com/yossigo))
- SSL Reorganization [\#708](https://github.com/redis/hiredis/pull/708) ([yossigo](https://github.com/yossigo))
- Fix MSVC build. [\#706](https://github.com/redis/hiredis/pull/706) ([yossigo](https://github.com/yossigo))
- SSL: Properly report SSL\_connect\(\) errors. [\#702](https://github.com/redis/hiredis/pull/702) ([yossigo](https://github.com/yossigo))
- Silent SSL trace to stdout by default. [\#699](https://github.com/redis/hiredis/pull/699) ([yossigo](https://github.com/yossigo))
- Port RESP3 support from Redis. [\#697](https://github.com/redis/hiredis/pull/697) ([yossigo](https://github.com/yossigo))
- Removed whitespace before newline [\#691](https://github.com/redis/hiredis/pull/691) ([Miniwoffer](https://github.com/Miniwoffer))
- Add install adapters header files [\#688](https://github.com/redis/hiredis/pull/688) ([kevin1018](https://github.com/kevin1018))
- Remove unnecessary null check before free [\#684](https://github.com/redis/hiredis/pull/684) ([qlyoung](https://github.com/qlyoung))
- redisReaderGetReply leak memory [\#671](https://github.com/redis/hiredis/pull/671) ([movebean](https://github.com/movebean))
- fix timeout code in windows [\#670](https://github.com/redis/hiredis/pull/670) ([jman-krafton](https://github.com/jman-krafton))
- test: fix errstr matching for musl libc [\#665](https://github.com/redis/hiredis/pull/665) ([ghost](https://github.com/ghost))
- Windows: MinGW fixes and Windows Travis builders [\#663](https://github.com/redis/hiredis/pull/663) ([mbitsnbites](https://github.com/mbitsnbites))
- The setsockopt and getsockopt API diffs from BSD socket and WSA one [\#662](https://github.com/redis/hiredis/pull/662) ([dragonation](https://github.com/dragonation))
- Fix Compile Error On Windows \(Visual Studio\) [\#658](https://github.com/redis/hiredis/pull/658) ([jinjiazhang](https://github.com/jinjiazhang))
- Fix NXDOMAIN test case [\#653](https://github.com/redis/hiredis/pull/653) ([michael-grunder](https://github.com/michael-grunder))
- Add MinGW support [\#652](https://github.com/redis/hiredis/pull/652) ([mbitsnbites](https://github.com/mbitsnbites))
- SSL Support [\#645](https://github.com/redis/hiredis/pull/645) ([mnunberg](https://github.com/mnunberg))
- Fix Invalid argument after redisAsyncConnectUnix [\#644](https://github.com/redis/hiredis/pull/644) ([codehz](https://github.com/codehz))
- Makefile: use predefined AR [\#632](https://github.com/redis/hiredis/pull/632) ([Mic92](https://github.com/Mic92))
- FreeBSD build fix [\#628](https://github.com/redis/hiredis/pull/628) ([devnexen](https://github.com/devnexen))
- Fix errors not propagating properly with libuv.h. [\#624](https://github.com/redis/hiredis/pull/624) ([yossigo](https://github.com/yossigo))
- Update README.md [\#621](https://github.com/redis/hiredis/pull/621) ([Crunsher](https://github.com/Crunsher))
- Fix redisBufferRead documentation [\#620](https://github.com/redis/hiredis/pull/620) ([hacst](https://github.com/hacst))
- Add CPPFLAGS to REAL\_CFLAGS [\#614](https://github.com/redis/hiredis/pull/614) ([thomaslee](https://github.com/thomaslee))
- Update createArray to take size\_t [\#597](https://github.com/redis/hiredis/pull/597) ([justinbrewer](https://github.com/justinbrewer))
- fix common realloc mistake and add null check more [\#580](https://github.com/redis/hiredis/pull/580) ([charsyam](https://github.com/charsyam))
- Proper error reporting for connect failures [\#578](https://github.com/redis/hiredis/pull/578) ([mnunberg](https://github.com/mnunberg))
\* *This Changelog was automatically generated by [github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)*
## [1.0.0-rc1](https://github.com/redis/hiredis/tree/v1.0.0-rc1) - (2020-07-29)
_Note: There were no changes to code between v1.0.0-rc1 and v1.0.0 so see v1.0.0 for changelog_
### 0.14.1 (2020-03-13)
* Adds safe allocation wrappers (CVE-2020-7105, #747, #752) (Michael Grunder)
### 0.14.0 (2018-09-25)
**BREAKING CHANGES**:
* Change `redisReply.len` to `size_t`, as it denotes the the size of a string
User code should compare this to `size_t` values as well.
If it was used to compare to other values, casting might be necessary or can be removed, if casting was applied before.
* Make string2ll static to fix conflict with Redis (Tom Lee [c3188b])
* Use -dynamiclib instead of -shared for OSX (Ryan Schmidt [a65537])
* Use string2ll from Redis w/added tests (Michael Grunder [7bef04, 60f622])
* Makefile - OSX compilation fixes (Ryan Schmidt [881fcb, 0e9af8])
* Remove redundant NULL checks (Justin Brewer [54acc8, 58e6b8])
* Fix bulk and multi-bulk length truncation (Justin Brewer [109197])
* Fix SIGSEGV in OpenBSD by checking for NULL before calling freeaddrinfo (Justin Brewer [546d94])
* Several POSIX compatibility fixes (Justin Brewer [bbeab8, 49bbaa, d1c1b6])
* Makefile - Compatibility fixes (Dimitri Vorobiev [3238cf, 12a9d1])
* Makefile - Fix make install on FreeBSD (Zach Shipko [a2ef2b])
* Makefile - don't assume $(INSTALL) is cp (Igor Gnatenko [725a96])
* Separate side-effect causing function from assert and small cleanup (amallia [b46413, 3c3234])
* Don't send negative values to `__redisAsyncCommand` (Frederik Deweerdt [706129])
* Fix leak if setsockopt fails (Frederik Deweerdt [e21c9c])
* Fix libevent leak (zfz [515228])
* Clean up GCC warning (Ichito Nagata [2ec774])
* Keep track of errno in `__redisSetErrorFromErrno()` as snprintf may use it (Jin Qing [25cd88])
* Solaris compilation fix (Donald Whyte [41b07d])
* Reorder linker arguments when building examples (Tustfarm-heart [06eedd])
* Keep track of subscriptions in case of rapid subscribe/unsubscribe (Hyungjin Kim [073dc8, be76c5, d46999])
* libuv use after free fix (Paul Scott [cbb956])
* Properly close socket fd on reconnect attempt (WSL [64d1ec])
* Skip valgrind in OSX tests (Jan-Erik Rediger [9deb78])
* Various updates for Travis testing OSX (Ted Nyman [fa3774, 16a459, bc0ea5])
* Update libevent (Chris Xin [386802])
* Change sds.h for building in C++ projects (Ali Volkan ATLI [f5b32e])
* Use proper format specifier in redisFormatSdsCommandArgv (Paulino Huerta, Jan-Erik Rediger [360a06, 8655a6])
* Better handling of NULL reply in example code (Jan-Erik Rediger [1b8ed3])
* Prevent overflow when formatting an error (Jan-Erik Rediger [0335cb])
* Compatibility fix for strerror_r (Tom Lee [bb1747])
* Properly detect integer parse/overflow errors (Justin Brewer [93421f])
* Adds CI for Windows and cygwin fixes (owent, [6c53d6, 6c3e40])
* Catch a buffer overflow when formatting the error message
* Import latest upstream sds. This breaks applications that are linked against the old hiredis v0.13
* Fix warnings, when compiled with -Wshadow
* Make hiredis compile in Cygwin on Windows, now CI-tested
* Bulk and multi-bulk lengths less than -1 or greater than `LLONG_MAX` are now
protocol errors. This is consistent with the RESP specification. On 32-bit
platforms, the upper bound is lowered to `SIZE_MAX`.
* Remove backwards compatibility macro's
This removes the following old function aliases, use the new name now:
| Old | New |
| --------------------------- | ---------------------- |
| redisReplyReaderCreate | redisReaderCreate |
| redisReplyReaderCreate | redisReaderCreate |
| redisReplyReaderFree | redisReaderFree |
| redisReplyReaderFeed | redisReaderFeed |
| redisReplyReaderGetReply | redisReaderGetReply |
| redisReplyReaderSetPrivdata | redisReaderSetPrivdata |
| redisReplyReaderGetObject | redisReaderGetObject |
| redisReplyReaderGetError | redisReaderGetError |
* The `DEBUG` variable in the Makefile was renamed to `DEBUG_FLAGS`
Previously it broke some builds for people that had `DEBUG` set to some arbitrary value,
due to debugging other software.
By renaming we avoid unintentional name clashes.
Simply rename `DEBUG` to `DEBUG_FLAGS` in your environment to make it working again.
### 0.13.3 (2015-09-16)
* Revert "Clear `REDIS_CONNECTED` flag when connection is closed".
* Make tests pass on FreeBSD (Thanks, Giacomo Olgeni)
If the `REDIS_CONNECTED` flag is cleared,
the async onDisconnect callback function will never be called.
This causes problems as the disconnect is never reported back to the user.
### 0.13.2 (2015-08-25)
* Prevent crash on pending replies in async code (Thanks, @switch-st)
* Clear `REDIS_CONNECTED` flag when connection is closed (Thanks, Jerry Jacobs)
* Add MacOS X addapter (Thanks, @dizzus)
* Add Qt adapter (Thanks, Pietro Cerutti)
* Add Ivykis adapter (Thanks, Gergely Nagy)
All adapters are provided as is and are only tested where possible.
### 0.13.1 (2015-05-03)
This is a bug fix release.
The new `reconnect` method introduced new struct members, which clashed with pre-defined names in pre-C99 code.
Another commit forced C99 compilation just to make it work, but of course this is not desirable for outside projects.
Other non-C99 code can now use hiredis as usual again.
Sorry for the inconvenience.
* Fix memory leak in async reply handling (Salvatore Sanfilippo)
* Rename struct member to avoid name clash with pre-c99 code (Alex Balashov, ncopa)
### 0.13.0 (2015-04-16)
This release adds a minimal Windows compatibility layer.
The parser, standalone since v0.12.0, can now be compiled on Windows
(and thus used in other client libraries as well)
* Windows compatibility layer for parser code (tzickel)
* Properly escape data printed to PKGCONF file (Dan Skorupski)
* Fix tests when assert() undefined (Keith Bennett, Matt Stancliff)
* Implement a reconnect method for the client context, this changes the structure of `redisContext` (Aaron Bedra)
### 0.12.1 (2015-01-26)
* Fix `make install`: DESTDIR support, install all required files, install PKGCONF in proper location
* Fix `make test` as 32 bit build on 64 bit platform
### 0.12.0 (2015-01-22)
* Add optional KeepAlive support
* Try again on EINTR errors
* Add libuv adapter
* Add IPv6 support
* Remove possibility of multiple close on same fd
* Add ability to bind source address on connect
* Add redisConnectFd() and redisFreeKeepFd()
* Fix getaddrinfo() memory leak
* Free string if it is unused (fixes memory leak)
* Improve redisAppendCommandArgv performance 2.5x
* Add support for SO_REUSEADDR
* Fix redisvFormatCommand format parsing
* Add GLib 2.0 adapter
* Refactor reading code into read.c
* Fix errno error buffers to not clobber errors
* Generate pkgconf during build
* Silence _BSD_SOURCE warnings
* Improve digit counting for multibulk creation
### 0.11.0
* Increase the maximum multi-bulk reply depth to 7.
* Increase the read buffer size from 2k to 16k.
* Use poll(2) instead of select(2) to support large fds (>= 1024).
### 0.10.1
* Makefile overhaul. Important to check out if you override one or more
variables using environment variables or via arguments to the "make" tool.
* Issue #45: Fix potential memory leak for a multi bulk reply with 0 elements
being created by the default reply object functions.
* Issue #43: Don't crash in an asynchronous context when Redis returns an error
reply after the connection has been made (this happens when the maximum
number of connections is reached).
### 0.10.0
* See commit log.
-237
View File
@@ -1,237 +0,0 @@
CMAKE_MINIMUM_REQUIRED(VERSION 3.0.0)
OPTION(BUILD_SHARED_LIBS "Build shared libraries" ON)
OPTION(ENABLE_SSL "Build hiredis_ssl for SSL support" OFF)
OPTION(DISABLE_TESTS "If tests should be compiled or not" OFF)
OPTION(ENABLE_SSL_TESTS "Should we test SSL connections" OFF)
OPTION(ENABLE_EXAMPLES "Enable building hiredis examples" OFF)
OPTION(ENABLE_ASYNC_TESTS "Should we run all asynchronous API tests" OFF)
MACRO(getVersionBit name)
SET(VERSION_REGEX "^#define ${name} (.+)$")
FILE(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/hiredis.h"
VERSION_BIT REGEX ${VERSION_REGEX})
STRING(REGEX REPLACE ${VERSION_REGEX} "\\1" ${name} "${VERSION_BIT}")
ENDMACRO(getVersionBit)
getVersionBit(HIREDIS_MAJOR)
getVersionBit(HIREDIS_MINOR)
getVersionBit(HIREDIS_PATCH)
getVersionBit(HIREDIS_SONAME)
SET(VERSION "${HIREDIS_MAJOR}.${HIREDIS_MINOR}.${HIREDIS_PATCH}")
MESSAGE("Detected version: ${VERSION}")
PROJECT(hiredis LANGUAGES "C" VERSION "${VERSION}")
INCLUDE(GNUInstallDirs)
# Hiredis requires C99
SET(CMAKE_C_STANDARD 99)
SET(CMAKE_DEBUG_POSTFIX d)
SET(hiredis_sources
alloc.c
async.c
hiredis.c
net.c
read.c
sds.c
sockcompat.c)
SET(hiredis_sources ${hiredis_sources})
IF(WIN32)
ADD_DEFINITIONS(-D_CRT_SECURE_NO_WARNINGS -DWIN32_LEAN_AND_MEAN)
ENDIF()
ADD_LIBRARY(hiredis ${hiredis_sources})
ADD_LIBRARY(hiredis::hiredis ALIAS hiredis)
set(hiredis_export_name hiredis CACHE STRING "Name of the exported target")
set_target_properties(hiredis PROPERTIES EXPORT_NAME ${hiredis_export_name})
SET_TARGET_PROPERTIES(hiredis
PROPERTIES WINDOWS_EXPORT_ALL_SYMBOLS TRUE
VERSION "${HIREDIS_SONAME}")
IF(MSVC)
SET_TARGET_PROPERTIES(hiredis
PROPERTIES COMPILE_FLAGS /Z7)
ENDIF()
IF(WIN32)
TARGET_LINK_LIBRARIES(hiredis PUBLIC ws2_32 crypt32)
ELSEIF(CMAKE_SYSTEM_NAME MATCHES "FreeBSD")
TARGET_LINK_LIBRARIES(hiredis PUBLIC m)
ELSEIF(CMAKE_SYSTEM_NAME MATCHES "SunOS")
TARGET_LINK_LIBRARIES(hiredis PUBLIC socket)
ENDIF()
TARGET_INCLUDE_DIRECTORIES(hiredis PUBLIC $<INSTALL_INTERFACE:include> $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>)
CONFIGURE_FILE(hiredis.pc.in hiredis.pc @ONLY)
set(CPACK_PACKAGE_VENDOR "Redis")
set(CPACK_PACKAGE_DESCRIPTION "\
Hiredis is a minimalistic C client library for the Redis database.
It is minimalistic because it just adds minimal support for the protocol, \
but at the same time it uses a high level printf-alike API in order to make \
it much higher level than otherwise suggested by its minimal code base and the \
lack of explicit bindings for every Redis command.
Apart from supporting sending commands and receiving replies, it comes with a \
reply parser that is decoupled from the I/O layer. It is a stream parser designed \
for easy reusability, which can for instance be used in higher level language bindings \
for efficient reply parsing.
Hiredis only supports the binary-safe Redis protocol, so you can use it with any Redis \
version >= 1.2.0.
The library comes with multiple APIs. There is the synchronous API, the asynchronous API \
and the reply parsing API.")
set(CPACK_PACKAGE_HOMEPAGE_URL "https://github.com/redis/hiredis")
set(CPACK_PACKAGE_CONTACT "michael dot grunder at gmail dot com")
set(CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON)
set(CPACK_RPM_PACKAGE_AUTOREQPROV ON)
include(CPack)
INSTALL(TARGETS hiredis
EXPORT hiredis-targets
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR})
if (MSVC AND BUILD_SHARED_LIBS)
INSTALL(FILES $<TARGET_PDB_FILE:hiredis>
DESTINATION ${CMAKE_INSTALL_BINDIR}
CONFIGURATIONS Debug RelWithDebInfo)
endif()
# For NuGet packages
INSTALL(FILES hiredis.targets
DESTINATION build/native)
INSTALL(FILES hiredis.h read.h sds.h async.h alloc.h sockcompat.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/hiredis)
INSTALL(DIRECTORY adapters
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/hiredis)
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/hiredis.pc
DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
export(EXPORT hiredis-targets
FILE "${CMAKE_CURRENT_BINARY_DIR}/hiredis-targets.cmake"
NAMESPACE hiredis::)
if(WIN32)
SET(CMAKE_CONF_INSTALL_DIR share/hiredis)
else()
SET(CMAKE_CONF_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR}/cmake/hiredis)
endif()
SET(INCLUDE_INSTALL_DIR include)
include(CMakePackageConfigHelpers)
write_basic_package_version_file("${CMAKE_CURRENT_BINARY_DIR}/hiredis-config-version.cmake"
COMPATIBILITY SameMajorVersion)
configure_package_config_file(hiredis-config.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/hiredis-config.cmake
INSTALL_DESTINATION ${CMAKE_CONF_INSTALL_DIR}
PATH_VARS INCLUDE_INSTALL_DIR)
INSTALL(EXPORT hiredis-targets
FILE hiredis-targets.cmake
NAMESPACE hiredis::
DESTINATION ${CMAKE_CONF_INSTALL_DIR})
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/hiredis-config.cmake
${CMAKE_CURRENT_BINARY_DIR}/hiredis-config-version.cmake
DESTINATION ${CMAKE_CONF_INSTALL_DIR})
IF(ENABLE_SSL)
IF (NOT OPENSSL_ROOT_DIR)
IF (APPLE)
SET(OPENSSL_ROOT_DIR "/usr/local/opt/openssl")
ENDIF()
ENDIF()
FIND_PACKAGE(OpenSSL REQUIRED)
SET(hiredis_ssl_sources
ssl.c)
ADD_LIBRARY(hiredis_ssl ${hiredis_ssl_sources})
ADD_LIBRARY(hiredis::hiredis_ssl ALIAS hiredis_ssl)
IF (APPLE AND BUILD_SHARED_LIBS)
SET_PROPERTY(TARGET hiredis_ssl PROPERTY LINK_FLAGS "-Wl,-undefined -Wl,dynamic_lookup")
ENDIF()
SET_TARGET_PROPERTIES(hiredis_ssl
PROPERTIES
WINDOWS_EXPORT_ALL_SYMBOLS TRUE
VERSION "${HIREDIS_SONAME}")
IF(MSVC)
SET_TARGET_PROPERTIES(hiredis_ssl
PROPERTIES COMPILE_FLAGS /Z7)
ENDIF()
TARGET_LINK_LIBRARIES(hiredis_ssl PRIVATE OpenSSL::SSL)
IF(WIN32)
TARGET_LINK_LIBRARIES(hiredis_ssl PRIVATE hiredis)
ENDIF()
CONFIGURE_FILE(hiredis_ssl.pc.in hiredis_ssl.pc @ONLY)
INSTALL(TARGETS hiredis_ssl
EXPORT hiredis_ssl-targets
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR})
if (MSVC AND BUILD_SHARED_LIBS)
INSTALL(FILES $<TARGET_PDB_FILE:hiredis_ssl>
DESTINATION ${CMAKE_INSTALL_BINDIR}
CONFIGURATIONS Debug RelWithDebInfo)
endif()
INSTALL(FILES hiredis_ssl.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/hiredis)
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/hiredis_ssl.pc
DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
export(EXPORT hiredis_ssl-targets
FILE "${CMAKE_CURRENT_BINARY_DIR}/hiredis_ssl-targets.cmake"
NAMESPACE hiredis::)
if(WIN32)
SET(CMAKE_CONF_INSTALL_DIR share/hiredis_ssl)
else()
SET(CMAKE_CONF_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR}/cmake/hiredis_ssl)
endif()
configure_package_config_file(hiredis_ssl-config.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/hiredis_ssl-config.cmake
INSTALL_DESTINATION ${CMAKE_CONF_INSTALL_DIR}
PATH_VARS INCLUDE_INSTALL_DIR)
INSTALL(EXPORT hiredis_ssl-targets
FILE hiredis_ssl-targets.cmake
NAMESPACE hiredis::
DESTINATION ${CMAKE_CONF_INSTALL_DIR})
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/hiredis_ssl-config.cmake
DESTINATION ${CMAKE_CONF_INSTALL_DIR})
ENDIF()
IF(NOT DISABLE_TESTS)
ENABLE_TESTING()
ADD_EXECUTABLE(hiredis-test test.c)
TARGET_LINK_LIBRARIES(hiredis-test hiredis)
IF(ENABLE_SSL_TESTS)
ADD_DEFINITIONS(-DHIREDIS_TEST_SSL=1)
TARGET_LINK_LIBRARIES(hiredis-test hiredis_ssl)
ENDIF()
IF(ENABLE_ASYNC_TESTS)
ADD_DEFINITIONS(-DHIREDIS_TEST_ASYNC=1)
TARGET_LINK_LIBRARIES(hiredis-test event)
ENDIF()
ADD_TEST(NAME hiredis-test
COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/test.sh)
ENDIF()
# Add examples
IF(ENABLE_EXAMPLES)
ADD_SUBDIRECTORY(examples)
ENDIF(ENABLE_EXAMPLES)
-29
View File
@@ -1,29 +0,0 @@
Copyright (c) 2009-2011, Redis Ltd.
Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of Redis nor the names of its contributors may be used
to endorse or promote products derived from this software without specific
prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-354
View File
@@ -1,354 +0,0 @@
# Hiredis Makefile
# Copyright (C) 2010-2011 Redis Ltd.
# Copyright (C) 2010-2011 Pieter Noordhuis <pcnoordhuis at gmail dot com>
# This file is released under the BSD license, see the COPYING file
OBJ=alloc.o net.o hiredis.o sds.o async.o read.o sockcompat.o
EXAMPLES=hiredis-example hiredis-example-libevent hiredis-example-libev hiredis-example-glib hiredis-example-push hiredis-example-poll
TESTS=hiredis-test
LIBNAME=libhiredis
PKGCONFNAME=hiredis.pc
HIREDIS_MAJOR=$(shell grep HIREDIS_MAJOR hiredis.h | awk '{print $$3}')
HIREDIS_MINOR=$(shell grep HIREDIS_MINOR hiredis.h | awk '{print $$3}')
HIREDIS_PATCH=$(shell grep HIREDIS_PATCH hiredis.h | awk '{print $$3}')
HIREDIS_SONAME=$(shell grep HIREDIS_SONAME hiredis.h | awk '{print $$3}')
# Installation related variables and target
PREFIX?=/usr/local
INCLUDE_PATH?=include/hiredis
LIBRARY_PATH?=lib
PKGCONF_PATH?=pkgconfig
INSTALL_INCLUDE_PATH= $(DESTDIR)$(PREFIX)/$(INCLUDE_PATH)
INSTALL_LIBRARY_PATH= $(DESTDIR)$(PREFIX)/$(LIBRARY_PATH)
INSTALL_PKGCONF_PATH= $(INSTALL_LIBRARY_PATH)/$(PKGCONF_PATH)
# redis-server configuration used for testing
REDIS_PORT=56379
REDIS_SERVER=redis-server
define REDIS_TEST_CONFIG
daemonize yes
pidfile /tmp/hiredis-test-redis.pid
port $(REDIS_PORT)
bind 127.0.0.1
unixsocket /tmp/hiredis-test-redis.sock
endef
export REDIS_TEST_CONFIG
# Fallback to gcc when $CC is not in $PATH.
CC:=$(shell sh -c 'type $${CC%% *} >/dev/null 2>/dev/null && echo $(CC) || echo gcc')
CXX:=$(shell sh -c 'type $${CXX%% *} >/dev/null 2>/dev/null && echo $(CXX) || echo g++')
OPTIMIZATION?=-O3
WARNINGS=-Wall -Wextra -Werror -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers
DEBUG_FLAGS?= -g -ggdb
REAL_CFLAGS=$(OPTIMIZATION) -fPIC $(CPPFLAGS) $(CFLAGS) $(WARNINGS) $(DEBUG_FLAGS) $(PLATFORM_FLAGS)
REAL_LDFLAGS=$(LDFLAGS)
DYLIBSUFFIX=so
STLIBSUFFIX=a
DYLIB_MINOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_SONAME)
DYLIB_MAJOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_MAJOR)
DYLIBNAME=$(LIBNAME).$(DYLIBSUFFIX)
DYLIB_MAKE_CMD=$(CC) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(DYLIB_MINOR_NAME)
STLIBNAME=$(LIBNAME).$(STLIBSUFFIX)
STLIB_MAKE_CMD=$(AR) rcs
#################### SSL variables start ####################
SSL_OBJ=ssl.o
SSL_LIBNAME=libhiredis_ssl
SSL_PKGCONFNAME=hiredis_ssl.pc
SSL_INSTALLNAME=install-ssl
SSL_DYLIB_MINOR_NAME=$(SSL_LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_SONAME)
SSL_DYLIB_MAJOR_NAME=$(SSL_LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_MAJOR)
SSL_DYLIBNAME=$(SSL_LIBNAME).$(DYLIBSUFFIX)
SSL_STLIBNAME=$(SSL_LIBNAME).$(STLIBSUFFIX)
SSL_DYLIB_MAKE_CMD=$(CC) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(SSL_DYLIB_MINOR_NAME)
USE_SSL?=0
ifeq ($(USE_SSL),1)
# This is required for test.c only
CFLAGS+=-DHIREDIS_TEST_SSL
EXAMPLES+=hiredis-example-ssl hiredis-example-libevent-ssl
SSL_STLIB=$(SSL_STLIBNAME)
SSL_DYLIB=$(SSL_DYLIBNAME)
SSL_PKGCONF=$(SSL_PKGCONFNAME)
SSL_INSTALL=$(SSL_INSTALLNAME)
else
SSL_STLIB=
SSL_DYLIB=
SSL_PKGCONF=
SSL_INSTALL=
endif
##################### SSL variables end #####################
# Platform-specific overrides
uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo not')
# This is required for test.c only
ifeq ($(TEST_ASYNC),1)
export CFLAGS+=-DHIREDIS_TEST_ASYNC
endif
ifeq ($(USE_SSL),1)
ifndef OPENSSL_PREFIX
ifeq ($(uname_S),Darwin)
SEARCH_PATH1=/opt/homebrew/opt/openssl
SEARCH_PATH2=/usr/local/opt/openssl
ifneq ($(wildcard $(SEARCH_PATH1)),)
OPENSSL_PREFIX=$(SEARCH_PATH1)
else ifneq ($(wildcard $(SEARCH_PATH2)),)
OPENSSL_PREFIX=$(SEARCH_PATH2)
endif
endif
endif
ifdef OPENSSL_PREFIX
CFLAGS+=-I$(OPENSSL_PREFIX)/include
SSL_LDFLAGS+=-L$(OPENSSL_PREFIX)/lib
endif
SSL_LDFLAGS+=-lssl -lcrypto
endif
ifeq ($(uname_S),FreeBSD)
LDFLAGS+=-lm
IS_GCC=$(shell sh -c '$(CC) --version 2>/dev/null |egrep -i -c "gcc"')
ifeq ($(IS_GCC),1)
REAL_CFLAGS+=-pedantic
endif
else
REAL_CFLAGS+=-pedantic
endif
ifeq ($(uname_S),SunOS)
IS_SUN_CC=$(shell sh -c '$(CC) -V 2>&1 |egrep -i -c "sun|studio"')
ifeq ($(IS_SUN_CC),1)
SUN_SHARED_FLAG=-G
else
SUN_SHARED_FLAG=-shared
endif
REAL_LDFLAGS+= -ldl -lnsl -lsocket
DYLIB_MAKE_CMD=$(CC) $(SUN_SHARED_FLAG) -o $(DYLIBNAME) -h $(DYLIB_MINOR_NAME) $(LDFLAGS)
SSL_DYLIB_MAKE_CMD=$(CC) $(SUN_SHARED_FLAG) -o $(SSL_DYLIBNAME) -h $(SSL_DYLIB_MINOR_NAME) $(LDFLAGS) $(SSL_LDFLAGS)
endif
ifeq ($(uname_S),Darwin)
DYLIBSUFFIX=dylib
DYLIB_MINOR_NAME=$(LIBNAME).$(HIREDIS_SONAME).$(DYLIBSUFFIX)
DYLIB_MAKE_CMD=$(CC) -dynamiclib -Wl,-install_name,$(PREFIX)/$(LIBRARY_PATH)/$(DYLIB_MINOR_NAME) -o $(DYLIBNAME) $(LDFLAGS)
SSL_DYLIB_MAKE_CMD=$(CC) -dynamiclib -Wl,-install_name,$(PREFIX)/$(LIBRARY_PATH)/$(SSL_DYLIB_MINOR_NAME) -o $(SSL_DYLIBNAME) $(LDFLAGS) $(SSL_LDFLAGS)
DYLIB_PLUGIN=-Wl,-undefined -Wl,dynamic_lookup
endif
all: dynamic static hiredis-test pkgconfig
dynamic: $(DYLIBNAME) $(SSL_DYLIB)
static: $(STLIBNAME) $(SSL_STLIB)
pkgconfig: $(PKGCONFNAME) $(SSL_PKGCONF)
# Deps (use make dep to generate this)
alloc.o: alloc.c fmacros.h alloc.h
async.o: async.c fmacros.h alloc.h async.h hiredis.h read.h sds.h net.h dict.c dict.h win32.h async_private.h
dict.o: dict.c fmacros.h alloc.h dict.h
hiredis.o: hiredis.c fmacros.h hiredis.h read.h sds.h alloc.h net.h async.h win32.h
net.o: net.c fmacros.h net.h hiredis.h read.h sds.h alloc.h sockcompat.h win32.h
read.o: read.c fmacros.h alloc.h read.h sds.h win32.h
sds.o: sds.c sds.h sdsalloc.h alloc.h
sockcompat.o: sockcompat.c sockcompat.h
test.o: test.c fmacros.h hiredis.h read.h sds.h alloc.h net.h sockcompat.h win32.h
$(DYLIBNAME): $(OBJ)
$(DYLIB_MAKE_CMD) -o $(DYLIBNAME) $(OBJ) $(REAL_LDFLAGS)
$(STLIBNAME): $(OBJ)
$(STLIB_MAKE_CMD) $(STLIBNAME) $(OBJ)
#################### SSL building rules start ####################
$(SSL_DYLIBNAME): $(SSL_OBJ)
$(SSL_DYLIB_MAKE_CMD) $(DYLIB_PLUGIN) -o $(SSL_DYLIBNAME) $(SSL_OBJ) $(REAL_LDFLAGS) $(LDFLAGS) $(SSL_LDFLAGS)
$(SSL_STLIBNAME): $(SSL_OBJ)
$(STLIB_MAKE_CMD) $(SSL_STLIBNAME) $(SSL_OBJ)
$(SSL_OBJ): ssl.c hiredis.h read.h sds.h alloc.h async.h win32.h async_private.h
#################### SSL building rules end ####################
# Binaries:
hiredis-example-libevent: examples/example-libevent.c adapters/libevent.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -levent $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-libevent-ssl: examples/example-libevent-ssl.c adapters/libevent.h $(STLIBNAME) $(SSL_STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -levent $(STLIBNAME) $(SSL_STLIBNAME) $(REAL_LDFLAGS) $(SSL_LDFLAGS)
hiredis-example-libev: examples/example-libev.c adapters/libev.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -lev $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-libhv: examples/example-libhv.c adapters/libhv.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -lhv $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-glib: examples/example-glib.c adapters/glib.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(shell pkg-config --cflags --libs glib-2.0) $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-ivykis: examples/example-ivykis.c adapters/ivykis.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -livykis $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-macosx: examples/example-macosx.c adapters/macosx.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -framework CoreFoundation $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-ssl: examples/example-ssl.c $(STLIBNAME) $(SSL_STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(STLIBNAME) $(SSL_STLIBNAME) $(REAL_LDFLAGS) $(SSL_LDFLAGS)
hiredis-example-poll: examples/example-poll.c adapters/poll.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(STLIBNAME) $(REAL_LDFLAGS)
ifndef AE_DIR
hiredis-example-ae:
@echo "Please specify AE_DIR (e.g. <redis repository>/src)"
@false
else
hiredis-example-ae: examples/example-ae.c adapters/ae.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. -I$(AE_DIR) $< $(AE_DIR)/ae.o $(AE_DIR)/zmalloc.o $(AE_DIR)/../deps/jemalloc/lib/libjemalloc.a -pthread $(STLIBNAME)
endif
ifndef LIBUV_DIR
# dynamic link libuv.so
hiredis-example-libuv: examples/example-libuv.c adapters/libuv.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. -I$(LIBUV_DIR)/include $< -luv -lpthread -lrt $(STLIBNAME) $(REAL_LDFLAGS)
else
# use user provided static lib
hiredis-example-libuv: examples/example-libuv.c adapters/libuv.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. -I$(LIBUV_DIR)/include $< $(LIBUV_DIR)/.libs/libuv.a -lpthread -lrt $(STLIBNAME) $(REAL_LDFLAGS)
endif
ifeq ($(and $(QT_MOC),$(QT_INCLUDE_DIR),$(QT_LIBRARY_DIR)),)
hiredis-example-qt:
@echo "Please specify QT_MOC, QT_INCLUDE_DIR AND QT_LIBRARY_DIR"
@false
else
hiredis-example-qt: examples/example-qt.cpp adapters/qt.h $(STLIBNAME)
$(QT_MOC) adapters/qt.h -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore | \
$(CXX) -x c++ -o qt-adapter-moc.o -c - $(REAL_CFLAGS) -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore
$(QT_MOC) examples/example-qt.h -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore | \
$(CXX) -x c++ -o qt-example-moc.o -c - $(REAL_CFLAGS) -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore
$(CXX) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore -L$(QT_LIBRARY_DIR) qt-adapter-moc.o qt-example-moc.o $< -pthread $(STLIBNAME) -lQtCore
endif
hiredis-example: examples/example.c $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-push: examples/example-push.c $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(STLIBNAME) $(REAL_LDFLAGS)
examples: $(EXAMPLES)
TEST_LIBS = $(STLIBNAME) $(SSL_STLIB)
TEST_LDFLAGS = $(SSL_LDFLAGS)
ifeq ($(USE_SSL),1)
TEST_LDFLAGS += -pthread
endif
ifeq ($(TEST_ASYNC),1)
TEST_LDFLAGS += -levent
endif
hiredis-test: test.o $(TEST_LIBS)
$(CC) -o $@ $(REAL_CFLAGS) -I. $^ $(REAL_LDFLAGS) $(TEST_LDFLAGS)
hiredis-%: %.o $(STLIBNAME)
$(CC) $(REAL_CFLAGS) -o $@ $< $(TEST_LIBS) $(REAL_LDFLAGS)
test: hiredis-test
./hiredis-test
check: hiredis-test
TEST_SSL=$(USE_SSL) ./test.sh
.c.o:
$(CC) -std=c99 -c $(REAL_CFLAGS) $<
clean:
rm -rf $(DYLIBNAME) $(STLIBNAME) $(SSL_DYLIBNAME) $(SSL_STLIBNAME) $(TESTS) $(PKGCONFNAME) examples/hiredis-example* *.o *.gcda *.gcno *.gcov
dep:
$(CC) $(CPPFLAGS) $(CFLAGS) -MM *.c
INSTALL?= cp -pPR
$(PKGCONFNAME): hiredis.h
@echo "Generating $@ for pkgconfig..."
@echo prefix=$(PREFIX) > $@
@echo exec_prefix=\$${prefix} >> $@
@echo libdir=$(PREFIX)/$(LIBRARY_PATH) >> $@
@echo includedir=$(PREFIX)/include >> $@
@echo pkgincludedir=$(PREFIX)/$(INCLUDE_PATH) >> $@
@echo >> $@
@echo Name: hiredis >> $@
@echo Description: Minimalistic C client library for Redis. >> $@
@echo Version: $(HIREDIS_MAJOR).$(HIREDIS_MINOR).$(HIREDIS_PATCH) >> $@
@echo Libs: -L\$${libdir} -lhiredis >> $@
@echo Cflags: -I\$${pkgincludedir} -I\$${includedir} -D_FILE_OFFSET_BITS=64 >> $@
$(SSL_PKGCONFNAME): hiredis_ssl.h
@echo "Generating $@ for pkgconfig..."
@echo prefix=$(PREFIX) > $@
@echo exec_prefix=\$${prefix} >> $@
@echo libdir=$(PREFIX)/$(LIBRARY_PATH) >> $@
@echo includedir=$(PREFIX)/include >> $@
@echo pkgincludedir=$(PREFIX)/$(INCLUDE_PATH) >> $@
@echo >> $@
@echo Name: hiredis_ssl >> $@
@echo Description: SSL Support for hiredis. >> $@
@echo Version: $(HIREDIS_MAJOR).$(HIREDIS_MINOR).$(HIREDIS_PATCH) >> $@
@echo Requires: hiredis >> $@
@echo Libs: -L\$${libdir} -lhiredis_ssl >> $@
@echo Libs.private: -lssl -lcrypto >> $@
install: $(DYLIBNAME) $(STLIBNAME) $(PKGCONFNAME) $(SSL_INSTALL)
mkdir -p $(INSTALL_INCLUDE_PATH) $(INSTALL_INCLUDE_PATH)/adapters $(INSTALL_LIBRARY_PATH)
$(INSTALL) hiredis.h async.h read.h sds.h alloc.h sockcompat.h $(INSTALL_INCLUDE_PATH)
$(INSTALL) adapters/*.h $(INSTALL_INCLUDE_PATH)/adapters
$(INSTALL) $(DYLIBNAME) $(INSTALL_LIBRARY_PATH)/$(DYLIB_MINOR_NAME)
cd $(INSTALL_LIBRARY_PATH) && ln -sf $(DYLIB_MINOR_NAME) $(DYLIBNAME) && ln -sf $(DYLIB_MINOR_NAME) $(DYLIB_MAJOR_NAME)
$(INSTALL) $(STLIBNAME) $(INSTALL_LIBRARY_PATH)
mkdir -p $(INSTALL_PKGCONF_PATH)
$(INSTALL) $(PKGCONFNAME) $(INSTALL_PKGCONF_PATH)
install-ssl: $(SSL_DYLIBNAME) $(SSL_STLIBNAME) $(SSL_PKGCONFNAME)
mkdir -p $(INSTALL_INCLUDE_PATH) $(INSTALL_LIBRARY_PATH)
$(INSTALL) hiredis_ssl.h $(INSTALL_INCLUDE_PATH)
$(INSTALL) $(SSL_DYLIBNAME) $(INSTALL_LIBRARY_PATH)/$(SSL_DYLIB_MINOR_NAME)
cd $(INSTALL_LIBRARY_PATH) && ln -sf $(SSL_DYLIB_MINOR_NAME) $(SSL_DYLIBNAME) && ln -sf $(SSL_DYLIB_MINOR_NAME) $(SSL_DYLIB_MAJOR_NAME)
$(INSTALL) $(SSL_STLIBNAME) $(INSTALL_LIBRARY_PATH)
mkdir -p $(INSTALL_PKGCONF_PATH)
$(INSTALL) $(SSL_PKGCONFNAME) $(INSTALL_PKGCONF_PATH)
32bit:
@echo ""
@echo "WARNING: if this fails under Linux you probably need to install libc6-dev-i386"
@echo ""
$(MAKE) CFLAGS="-m32" LDFLAGS="-m32"
32bit-vars:
$(eval CFLAGS=-m32)
$(eval LDFLAGS=-m32)
gprof:
$(MAKE) CFLAGS="-pg" LDFLAGS="-pg"
gcov:
$(MAKE) CFLAGS+="-fprofile-arcs -ftest-coverage" LDFLAGS="-fprofile-arcs"
coverage: gcov
make check
mkdir -p tmp/lcov
lcov -d . -c --exclude '/usr*' -o tmp/lcov/hiredis.info
lcov -q -l tmp/lcov/hiredis.info
genhtml --legend -q -o tmp/lcov/report tmp/lcov/hiredis.info
noopt:
$(MAKE) OPTIMIZATION=""
.PHONY: all test check clean dep install 32bit 32bit-vars gprof gcov noopt
-821
View File
@@ -1,821 +0,0 @@
[![Build Status](https://github.com/redis/hiredis/actions/workflows/build.yml/badge.svg)](https://github.com/redis/hiredis/actions/workflows/build.yml)
**This Readme reflects the latest changed in the master branch. See [v1.0.0](https://github.com/redis/hiredis/tree/v1.0.0) for the Readme and documentation for the latest release ([API/ABI history](https://abi-laboratory.pro/?view=timeline&l=hiredis)).**
# HIREDIS
Hiredis is a minimalistic C client library for the [Redis](https://redis.io/) database.
It is minimalistic because it just adds minimal support for the protocol, but
at the same time it uses a high level printf-alike API in order to make it
much higher level than otherwise suggested by its minimal code base and the
lack of explicit bindings for every Redis command.
Apart from supporting sending commands and receiving replies, it comes with
a reply parser that is decoupled from the I/O layer. It
is a stream parser designed for easy reusability, which can for instance be used
in higher level language bindings for efficient reply parsing.
Hiredis only supports the binary-safe Redis protocol, so you can use it with any
Redis version >= 1.2.0.
The library comes with multiple APIs. There is the
*synchronous API*, the *asynchronous API* and the *reply parsing API*.
## Upgrading to `1.1.0`
Almost all users will simply need to recompile their applications against the newer version of hiredis.
**NOTE**: Hiredis can now return `nan` in addition to `-inf` and `inf` in a `REDIS_REPLY_DOUBLE`.
Applications that deal with `RESP3` doubles should make sure to account for this.
## Upgrading to `1.0.2`
<span style="color:red">NOTE: v1.0.1 erroneously bumped SONAME, which is why it is skipped here.</span>
Version 1.0.2 is simply 1.0.0 with a fix for [CVE-2021-32765](https://github.com/redis/hiredis/security/advisories/GHSA-hfm9-39pp-55p2). They are otherwise identical.
## Upgrading to `1.0.0`
Version 1.0.0 marks the first stable release of Hiredis.
It includes some minor breaking changes, mostly to make the exposed API more uniform and self-explanatory.
It also bundles the updated `sds` library, to sync up with upstream and Redis.
For code changes see the [Changelog](CHANGELOG.md).
_Note: As described below, a few member names have been changed but most applications should be able to upgrade with minor code changes and recompiling._
## IMPORTANT: Breaking changes from `0.14.1` -> `1.0.0`
* `redisContext` has two additional members (`free_privdata`, and `privctx`).
* `redisOptions.timeout` has been renamed to `redisOptions.connect_timeout`, and we've added `redisOptions.command_timeout`.
* `redisReplyObjectFunctions.createArray` now takes `size_t` instead of `int` for its length parameter.
## IMPORTANT: Breaking changes when upgrading from 0.13.x -> 0.14.x
Bulk and multi-bulk lengths less than -1 or greater than `LLONG_MAX` are now
protocol errors. This is consistent with the RESP specification. On 32-bit
platforms, the upper bound is lowered to `SIZE_MAX`.
Change `redisReply.len` to `size_t`, as it denotes the the size of a string
User code should compare this to `size_t` values as well. If it was used to
compare to other values, casting might be necessary or can be removed, if
casting was applied before.
## Upgrading from `<0.9.0`
Version 0.9.0 is a major overhaul of hiredis in every aspect. However, upgrading existing
code using hiredis should not be a big pain. The key thing to keep in mind when
upgrading is that hiredis >= 0.9.0 uses a `redisContext*` to keep state, in contrast to
the stateless 0.0.1 that only has a file descriptor to work with.
## Synchronous API
To consume the synchronous API, there are only a few function calls that need to be introduced:
```c
redisContext *redisConnect(const char *ip, int port);
void *redisCommand(redisContext *c, const char *format, ...);
void freeReplyObject(void *reply);
```
### Connecting
The function `redisConnect` is used to create a so-called `redisContext`. The
context is where Hiredis holds state for a connection. The `redisContext`
struct has an integer `err` field that is non-zero when the connection is in
an error state. The field `errstr` will contain a string with a description of
the error. More information on errors can be found in the **Errors** section.
After trying to connect to Redis using `redisConnect` you should
check the `err` field to see if establishing the connection was successful:
```c
redisContext *c = redisConnect("127.0.0.1", 6379);
if (c == NULL || c->err) {
if (c) {
printf("Error: %s\n", c->errstr);
// handle error
} else {
printf("Can't allocate redis context\n");
}
}
```
One can also use `redisConnectWithOptions` which takes a `redisOptions` argument
that can be configured with endpoint information as well as many different flags
to change how the `redisContext` will be configured.
```c
redisOptions opt = {0};
/* One can set the endpoint with one of our helper macros */
if (tcp) {
REDIS_OPTIONS_SET_TCP(&opt, "localhost", 6379);
} else {
REDIS_OPTIONS_SET_UNIX(&opt, "/tmp/redis.sock");
}
/* And privdata can be specified with another helper */
REDIS_OPTIONS_SET_PRIVDATA(&opt, myPrivData, myPrivDataDtor);
/* Finally various options may be set via the `options` member, as described below */
opt->options |= REDIS_OPT_PREFER_IPV4;
```
If a connection is lost, `int redisReconnect(redisContext *c)` can be used to restore the connection using the same endpoint and options as the given context.
### Configurable redisOptions flags
There are several flags you may set in the `redisOptions` struct to change default behavior. You can specify the flags via the `redisOptions->options` member.
| Flag | Description |
| --- | --- |
| REDIS\_OPT\_NONBLOCK | Tells hiredis to make a non-blocking connection. |
| REDIS\_OPT\_REUSEADDR | Tells hiredis to set the [SO_REUSEADDR](https://man7.org/linux/man-pages/man7/socket.7.html) socket option |
| REDIS\_OPT\_PREFER\_IPV4<br>REDIS\_OPT\_PREFER_IPV6<br>REDIS\_OPT\_PREFER\_IP\_UNSPEC | Informs hiredis to either prefer IPv4 or IPv6 when invoking [getaddrinfo](https://man7.org/linux/man-pages/man3/gai_strerror.3.html). `REDIS_OPT_PREFER_IP_UNSPEC` will cause hiredis to specify `AF_UNSPEC` in the getaddrinfo call, which means both IPv4 and IPv6 addresses will be searched simultaneously.<br>Hiredis prefers IPv4 by default. |
| REDIS\_OPT\_NO\_PUSH\_AUTOFREE | Tells hiredis to not install the default RESP3 PUSH handler (which just intercepts and frees the replies). This is useful in situations where you want to process these messages in-band. |
| REDIS\_OPT\_NOAUTOFREEREPLIES | **ASYNC**: tells hiredis not to automatically invoke `freeReplyObject` after executing the reply callback. |
| REDIS\_OPT\_NOAUTOFREE | **ASYNC**: Tells hiredis not to automatically free the `redisAsyncContext` on connection/communication failure, but only if the user makes an explicit call to `redisAsyncDisconnect` or `redisAsyncFree` |
*Note: A `redisContext` is not thread-safe.*
### Other configuration using socket options
The following socket options are applied directly to the underlying socket.
The values are not stored in the `redisContext`, so they are not automatically applied when reconnecting using `redisReconnect()`.
These functions return `REDIS_OK` on success.
On failure, `REDIS_ERR` is returned and the underlying connection is closed.
To configure these for an asyncronous context (see *Asynchronous API* below), use `ac->c` to get the redisContext out of an asyncRedisContext.
```C
int redisEnableKeepAlive(redisContext *c);
int redisEnableKeepAliveWithInterval(redisContext *c, int interval);
```
Enables TCP keepalive by setting the following socket options (with some variations depending on OS):
* `SO_KEEPALIVE`;
* `TCP_KEEPALIVE` or `TCP_KEEPIDLE`, value configurable using the `interval` parameter, default 15 seconds;
* `TCP_KEEPINTVL` set to 1/3 of `interval`;
* `TCP_KEEPCNT` set to 3.
```C
int redisSetTcpUserTimeout(redisContext *c, unsigned int timeout);
```
Set the `TCP_USER_TIMEOUT` Linux-specific socket option which is as described in the `tcp` man page:
> When the value is greater than 0, it specifies the maximum amount of time in milliseconds that trans mitted data may remain unacknowledged before TCP will forcibly close the corresponding connection and return ETIMEDOUT to the application.
> If the option value is specified as 0, TCP will use the system default.
### Sending commands
There are several ways to issue commands to Redis. The first that will be introduced is
`redisCommand`. This function takes a format similar to printf. In the simplest form,
it is used like this:
```c
reply = redisCommand(context, "SET foo bar");
```
The specifier `%s` interpolates a string in the command, and uses `strlen` to
determine the length of the string:
```c
reply = redisCommand(context, "SET foo %s", value);
```
When you need to pass binary safe strings in a command, the `%b` specifier can be
used. Together with a pointer to the string, it requires a `size_t` length argument
of the string:
```c
reply = redisCommand(context, "SET foo %b", value, (size_t) valuelen);
```
Internally, Hiredis splits the command in different arguments and will
convert it to the protocol used to communicate with Redis.
One or more spaces separates arguments, so you can use the specifiers
anywhere in an argument:
```c
reply = redisCommand(context, "SET key:%s %s", myid, value);
```
### Using replies
The return value of `redisCommand` holds a reply when the command was
successfully executed. When an error occurs, the return value is `NULL` and
the `err` field in the context will be set (see section on **Errors**).
Once an error is returned the context cannot be reused and you should set up
a new connection.
The standard replies that `redisCommand` are of the type `redisReply`. The
`type` field in the `redisReply` should be used to test what kind of reply
was received:
### RESP2
* **`REDIS_REPLY_STATUS`**:
* The command replied with a status reply. The status string can be accessed using `reply->str`.
The length of this string can be accessed using `reply->len`.
* **`REDIS_REPLY_ERROR`**:
* The command replied with an error. The error string can be accessed identical to `REDIS_REPLY_STATUS`.
* **`REDIS_REPLY_INTEGER`**:
* The command replied with an integer. The integer value can be accessed using the
`reply->integer` field of type `long long`.
* **`REDIS_REPLY_NIL`**:
* The command replied with a **nil** object. There is no data to access.
* **`REDIS_REPLY_STRING`**:
* A bulk (string) reply. The value of the reply can be accessed using `reply->str`.
The length of this string can be accessed using `reply->len`.
* **`REDIS_REPLY_ARRAY`**:
* A multi bulk reply. The number of elements in the multi bulk reply is stored in
`reply->elements`. Every element in the multi bulk reply is a `redisReply` object as well
and can be accessed via `reply->element[..index..]`.
Redis may reply with nested arrays but this is fully supported.
### RESP3
Hiredis also supports every new `RESP3` data type which are as follows. For more information about the protocol see the `RESP3` [specification.](https://github.com/antirez/RESP3/blob/master/spec.md)
* **`REDIS_REPLY_DOUBLE`**:
* The command replied with a double-precision floating point number.
The value is stored as a string in the `str` member, and can be converted with `strtod` or similar.
* **`REDIS_REPLY_BOOL`**:
* A boolean true/false reply.
The value is stored in the `integer` member and will be either `0` or `1`.
* **`REDIS_REPLY_MAP`**:
* An array with the added invariant that there will always be an even number of elements.
The MAP is functionally equivalent to `REDIS_REPLY_ARRAY` except for the previously mentioned invariant.
* **`REDIS_REPLY_SET`**:
* An array response where each entry is unique.
Like the MAP type, the data is identical to an array response except there are no duplicate values.
* **`REDIS_REPLY_PUSH`**:
* An array that can be generated spontaneously by Redis.
This array response will always contain at least two subelements. The first contains the type of `PUSH` message (e.g. `message`, or `invalidate`), and the second being a sub-array with the `PUSH` payload itself.
* **`REDIS_REPLY_ATTR`**:
* An array structurally identical to a `MAP` but intended as meta-data about a reply.
_As of Redis 6.0.6 this reply type is not used in Redis_
* **`REDIS_REPLY_BIGNUM`**:
* A string representing an arbitrarily large signed or unsigned integer value.
The number will be encoded as a string in the `str` member of `redisReply`.
* **`REDIS_REPLY_VERB`**:
* A verbatim string, intended to be presented to the user without modification.
The string payload is stored in the `str` member, and type data is stored in the `vtype` member (e.g. `txt` for raw text or `md` for markdown).
Replies should be freed using the `freeReplyObject()` function.
Note that this function will take care of freeing sub-reply objects
contained in arrays and nested arrays, so there is no need for the user to
free the sub replies (it is actually harmful and will corrupt the memory).
**Important:** the current version of hiredis (1.0.0) frees replies when the
asynchronous API is used. This means you should not call `freeReplyObject` when
you use this API. The reply is cleaned up by hiredis _after_ the callback
returns. We may introduce a flag to make this configurable in future versions of the library.
### Cleaning up
To disconnect and free the context the following function can be used:
```c
void redisFree(redisContext *c);
```
This function immediately closes the socket and then frees the allocations done in
creating the context.
### Sending commands (cont'd)
Together with `redisCommand`, the function `redisCommandArgv` can be used to issue commands.
It has the following prototype:
```c
void *redisCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);
```
It takes the number of arguments `argc`, an array of strings `argv` and the lengths of the
arguments `argvlen`. For convenience, `argvlen` may be set to `NULL` and the function will
use `strlen(3)` on every argument to determine its length. Obviously, when any of the arguments
need to be binary safe, the entire array of lengths `argvlen` should be provided.
The return value has the same semantic as `redisCommand`.
### Pipelining
To explain how Hiredis supports pipelining in a blocking connection, there needs to be
understanding of the internal execution flow.
When any of the functions in the `redisCommand` family is called, Hiredis first formats the
command according to the Redis protocol. The formatted command is then put in the output buffer
of the context. This output buffer is dynamic, so it can hold any number of commands.
After the command is put in the output buffer, `redisGetReply` is called. This function has the
following two execution paths:
1. The input buffer is non-empty:
* Try to parse a single reply from the input buffer and return it
* If no reply could be parsed, continue at *2*
2. The input buffer is empty:
* Write the **entire** output buffer to the socket
* Read from the socket until a single reply could be parsed
The function `redisGetReply` is exported as part of the Hiredis API and can be used when a reply
is expected on the socket. To pipeline commands, the only thing that needs to be done is
filling up the output buffer. For this cause, two commands can be used that are identical
to the `redisCommand` family, apart from not returning a reply:
```c
void redisAppendCommand(redisContext *c, const char *format, ...);
void redisAppendCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);
```
After calling either function one or more times, `redisGetReply` can be used to receive the
subsequent replies. The return value for this function is either `REDIS_OK` or `REDIS_ERR`, where
the latter means an error occurred while reading a reply. Just as with the other commands,
the `err` field in the context can be used to find out what the cause of this error is.
The following examples shows a simple pipeline (resulting in only a single call to `write(2)` and
a single call to `read(2)`):
```c
redisReply *reply;
redisAppendCommand(context,"SET foo bar");
redisAppendCommand(context,"GET foo");
redisGetReply(context,(void**)&reply); // reply for SET
freeReplyObject(reply);
redisGetReply(context,(void**)&reply); // reply for GET
freeReplyObject(reply);
```
This API can also be used to implement a blocking subscriber:
```c
reply = redisCommand(context,"SUBSCRIBE foo");
freeReplyObject(reply);
while(redisGetReply(context,(void *)&reply) == REDIS_OK) {
// consume message
freeReplyObject(reply);
}
```
### Errors
When a function call is not successful, depending on the function either `NULL` or `REDIS_ERR` is
returned. The `err` field inside the context will be non-zero and set to one of the
following constants:
* **`REDIS_ERR_IO`**:
There was an I/O error while creating the connection, trying to write
to the socket or read from the socket. If you included `errno.h` in your
application, you can use the global `errno` variable to find out what is
wrong.
* **`REDIS_ERR_EOF`**:
The server closed the connection which resulted in an empty read.
* **`REDIS_ERR_PROTOCOL`**:
There was an error while parsing the protocol.
* **`REDIS_ERR_OTHER`**:
Any other error. Currently, it is only used when a specified hostname to connect
to cannot be resolved.
In every case, the `errstr` field in the context will be set to hold a string representation
of the error.
## Asynchronous API
Hiredis comes with an asynchronous API that works easily with any event library.
Examples are bundled that show using Hiredis with [libev](http://software.schmorp.de/pkg/libev.html)
and [libevent](http://monkey.org/~provos/libevent/).
### Connecting
The function `redisAsyncConnect` can be used to establish a non-blocking connection to
Redis. It returns a pointer to the newly created `redisAsyncContext` struct. The `err` field
should be checked after creation to see if there were errors creating the connection.
Because the connection that will be created is non-blocking, the kernel is not able to
instantly return if the specified host and port is able to accept a connection.
In case of error, it is the caller's responsibility to free the context using `redisAsyncFree()`
*Note: A `redisAsyncContext` is not thread-safe.*
An application function creating a connection might look like this:
```c
void appConnect(myAppData *appData)
{
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
printf("Error: %s\n", c->errstr);
// handle error
redisAsyncFree(c);
c = NULL;
} else {
appData->context = c;
appData->connecting = 1;
c->data = appData; /* store application pointer for the callbacks */
redisAsyncSetConnectCallback(c, appOnConnect);
redisAsyncSetDisconnectCallback(c, appOnDisconnect);
}
}
```
The asynchronous context _should_ hold a *connect* callback function that is called when the connection
attempt completes, either successfully or with an error.
It _can_ also hold a *disconnect* callback function that is called when the
connection is disconnected (either because of an error or per user request). Both callbacks should
have the following prototype:
```c
void(const redisAsyncContext *c, int status);
```
On a *connect*, the `status` argument is set to `REDIS_OK` if the connection attempt succeeded. In this
case, the context is ready to accept commands. If it is called with `REDIS_ERR` then the
connection attempt failed. The `err` field in the context can be accessed to find out the cause of the error.
After a failed connection attempt, the context object is automatically freed by the library after calling
the connect callback. This may be a good point to create a new context and retry the connection.
On a disconnect, the `status` argument is set to `REDIS_OK` when disconnection was initiated by the
user, or `REDIS_ERR` when the disconnection was caused by an error. When it is `REDIS_ERR`, the `err`
field in the context can be accessed to find out the cause of the error.
The context object is always freed after the disconnect callback fired. When a reconnect is needed,
the disconnect callback is a good point to do so.
Setting the connect or disconnect callbacks can only be done once per context. For subsequent calls the
api will return `REDIS_ERR`. The function to set the callbacks have the following prototype:
```c
/* Alternatively you can use redisAsyncSetConnectCallbackNC which will be passed a non-const
redisAsyncContext* on invocation (e.g. allowing writes to the privdata member). */
int redisAsyncSetConnectCallback(redisAsyncContext *ac, redisConnectCallback *fn);
int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);
```
`ac->data` may be used to pass user data to both callbacks. A typical implementation
might look something like this:
```c
void appOnConnect(redisAsyncContext *c, int status)
{
myAppData *appData = (myAppData*)c->data; /* get my application specific context*/
appData->connecting = 0;
if (status == REDIS_OK) {
appData->connected = 1;
} else {
appData->connected = 0;
appData->err = c->err;
appData->context = NULL; /* avoid stale pointer when callback returns */
}
appAttemptReconnect();
}
void appOnDisconnect(redisAsyncContext *c, int status)
{
myAppData *appData = (myAppData*)c->data; /* get my application specific context*/
appData->connected = 0;
appData->err = c->err;
appData->context = NULL; /* avoid stale pointer when callback returns */
if (status == REDIS_OK) {
appNotifyDisconnectCompleted(mydata);
} else {
appNotifyUnexpectedDisconnect(mydata);
appAttemptReconnect();
}
}
```
### Sending commands and their callbacks
In an asynchronous context, commands are automatically pipelined due to the nature of an event loop.
Therefore, unlike the synchronous API, there is only a single way to send commands.
Because commands are sent to Redis asynchronously, issuing a command requires a callback function
that is called when the reply is received. Reply callbacks should have the following prototype:
```c
void(redisAsyncContext *c, void *reply, void *privdata);
```
The `privdata` argument can be used to curry arbitrary data to the callback from the point where
the command is initially queued for execution.
The functions that can be used to issue commands in an asynchronous context are:
```c
int redisAsyncCommand(
redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
const char *format, ...);
int redisAsyncCommandArgv(
redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
int argc, const char **argv, const size_t *argvlen);
```
Both functions work like their blocking counterparts. The return value is `REDIS_OK` when the command
was successfully added to the output buffer and `REDIS_ERR` otherwise. Example: when the connection
is being disconnected per user-request, no new commands may be added to the output buffer and `REDIS_ERR` is
returned on calls to the `redisAsyncCommand` family.
If the reply for a command with a `NULL` callback is read, it is immediately freed. When the callback
for a command is non-`NULL`, the memory is freed immediately following the callback: the reply is only
valid for the duration of the callback.
All pending callbacks are called with a `NULL` reply when the context encountered an error.
For every command issued, with the exception of **SUBSCRIBE** and **PSUBSCRIBE**, the callback is
called exactly once. Even if the context object id disconnected or deleted, every pending callback
will be called with a `NULL` reply.
For **SUBSCRIBE** and **PSUBSCRIBE**, the callbacks may be called repeatedly until an `unsubscribe`
message arrives. This will be the last invocation of the callback. In case of error, the callbacks
may receive a final `NULL` reply instead.
### Disconnecting
An asynchronous connection can be terminated using:
```c
void redisAsyncDisconnect(redisAsyncContext *ac);
```
When this function is called, the connection is **not** immediately terminated. Instead, new
commands are no longer accepted and the connection is only terminated when all pending commands
have been written to the socket, their respective replies have been read and their respective
callbacks have been executed. After this, the disconnection callback is executed with the
`REDIS_OK` status and the context object is freed.
The connection can be forcefully disconnected using
```c
void redisAsyncFree(redisAsyncContext *ac);
```
In this case, nothing more is written to the socket, all pending callbacks are called with a `NULL`
reply and the disconnection callback is called with `REDIS_OK`, after which the context object
is freed.
### Hooking it up to event library *X*
There are a few hooks that need to be set on the context object after it is created.
See the `adapters/` directory for bindings to *libev* and *libevent*.
## Reply parsing API
Hiredis comes with a reply parsing API that makes it easy for writing higher
level language bindings.
The reply parsing API consists of the following functions:
```c
redisReader *redisReaderCreate(void);
void redisReaderFree(redisReader *reader);
int redisReaderFeed(redisReader *reader, const char *buf, size_t len);
int redisReaderGetReply(redisReader *reader, void **reply);
```
The same set of functions are used internally by hiredis when creating a
normal Redis context, the above API just exposes it to the user for a direct
usage.
### Usage
The function `redisReaderCreate` creates a `redisReader` structure that holds a
buffer with unparsed data and state for the protocol parser.
Incoming data -- most likely from a socket -- can be placed in the internal
buffer of the `redisReader` using `redisReaderFeed`. This function will make a
copy of the buffer pointed to by `buf` for `len` bytes. This data is parsed
when `redisReaderGetReply` is called. This function returns an integer status
and a reply object (as described above) via `void **reply`. The returned status
can be either `REDIS_OK` or `REDIS_ERR`, where the latter means something went
wrong (either a protocol error, or an out of memory error).
The parser limits the level of nesting for multi bulk payloads to 7. If the
multi bulk nesting level is higher than this, the parser returns an error.
### Customizing replies
The function `redisReaderGetReply` creates `redisReply` and makes the function
argument `reply` point to the created `redisReply` variable. For instance, if
the response of type `REDIS_REPLY_STATUS` then the `str` field of `redisReply`
will hold the status as a vanilla C string. However, the functions that are
responsible for creating instances of the `redisReply` can be customized by
setting the `fn` field on the `redisReader` struct. This should be done
immediately after creating the `redisReader`.
For example, [hiredis-rb](https://github.com/pietern/hiredis-rb/blob/master/ext/hiredis_ext/reader.c)
uses customized reply object functions to create Ruby objects.
### Reader max buffer
Both when using the Reader API directly or when using it indirectly via a
normal Redis context, the redisReader structure uses a buffer in order to
accumulate data from the server.
Usually this buffer is destroyed when it is empty and is larger than 16
KiB in order to avoid wasting memory in unused buffers
However when working with very big payloads destroying the buffer may slow
down performances considerably, so it is possible to modify the max size of
an idle buffer changing the value of the `maxbuf` field of the reader structure
to the desired value. The special value of 0 means that there is no maximum
value for an idle buffer, so the buffer will never get freed.
For instance if you have a normal Redis context you can set the maximum idle
buffer to zero (unlimited) just with:
```c
context->reader->maxbuf = 0;
```
This should be done only in order to maximize performances when working with
large payloads. The context should be set back to `REDIS_READER_MAX_BUF` again
as soon as possible in order to prevent allocation of useless memory.
### Reader max array elements
By default the hiredis reply parser sets the maximum number of multi-bulk elements
to 2^32 - 1 or 4,294,967,295 entries. If you need to process multi-bulk replies
with more than this many elements you can set the value higher or to zero, meaning
unlimited with:
```c
context->reader->maxelements = 0;
```
## SSL/TLS Support
### Building
SSL/TLS support is not built by default and requires an explicit flag:
make USE_SSL=1
This requires OpenSSL development package (e.g. including header files to be
available.
When enabled, SSL/TLS support is built into extra `libhiredis_ssl.a` and
`libhiredis_ssl.so` static/dynamic libraries. This leaves the original libraries
unaffected so no additional dependencies are introduced.
### Using it
First, you'll need to make sure you include the SSL header file:
```c
#include <hiredis/hiredis.h>
#include <hiredis/hiredis_ssl.h>
```
You will also need to link against `libhiredis_ssl`, **in addition** to
`libhiredis` and add `-lssl -lcrypto` to satisfy its dependencies.
Hiredis implements SSL/TLS on top of its normal `redisContext` or
`redisAsyncContext`, so you will need to establish a connection first and then
initiate an SSL/TLS handshake.
#### Hiredis OpenSSL Wrappers
Before Hiredis can negotiate an SSL/TLS connection, it is necessary to
initialize OpenSSL and create a context. You can do that in two ways:
1. Work directly with the OpenSSL API to initialize the library's global context
and create `SSL_CTX *` and `SSL *` contexts. With an `SSL *` object you can
call `redisInitiateSSL()`.
2. Work with a set of Hiredis-provided wrappers around OpenSSL, create a
`redisSSLContext` object to hold configuration and use
`redisInitiateSSLWithContext()` to initiate the SSL/TLS handshake.
```c
/* An Hiredis SSL context. It holds SSL configuration and can be reused across
* many contexts.
*/
redisSSLContext *ssl_context;
/* An error variable to indicate what went wrong, if the context fails to
* initialize.
*/
redisSSLContextError ssl_error = REDIS_SSL_CTX_NONE;
/* Initialize global OpenSSL state.
*
* You should call this only once when your app initializes, and only if
* you don't explicitly or implicitly initialize OpenSSL it elsewhere.
*/
redisInitOpenSSL();
/* Create SSL context */
ssl_context = redisCreateSSLContext(
"cacertbundle.crt", /* File name of trusted CA/ca bundle file, optional */
"/path/to/certs", /* Path of trusted certificates, optional */
"client_cert.pem", /* File name of client certificate file, optional */
"client_key.pem", /* File name of client private key, optional */
"redis.mydomain.com", /* Server name to request (SNI), optional */
&ssl_error);
if(ssl_context == NULL || ssl_error != REDIS_SSL_CTX_NONE) {
/* Handle error and abort... */
/* e.g.
printf("SSL error: %s\n",
(ssl_error != REDIS_SSL_CTX_NONE) ?
redisSSLContextGetError(ssl_error) : "Unknown error");
// Abort
*/
}
/* Create Redis context and establish connection */
c = redisConnect("localhost", 6443);
if (c == NULL || c->err) {
/* Handle error and abort... */
}
/* Negotiate SSL/TLS */
if (redisInitiateSSLWithContext(c, ssl_context) != REDIS_OK) {
/* Handle error, in c->err / c->errstr */
}
```
## RESP3 PUSH replies
Redis 6.0 introduced PUSH replies with the reply-type `>`. These messages are generated spontaneously and can arrive at any time, so must be handled using callbacks.
### Default behavior
Hiredis installs handlers on `redisContext` and `redisAsyncContext` by default, which will intercept and free any PUSH replies detected. This means existing code will work as-is after upgrading to Redis 6 and switching to `RESP3`.
### Custom PUSH handler prototypes
The callback prototypes differ between `redisContext` and `redisAsyncContext`.
#### redisContext
```c
void my_push_handler(void *privdata, void *reply) {
/* Handle the reply */
/* Note: We need to free the reply in our custom handler for
blocking contexts. This lets us keep the reply if
we want. */
freeReplyObject(reply);
}
```
#### redisAsyncContext
```c
void my_async_push_handler(redisAsyncContext *ac, void *reply) {
/* Handle the reply */
/* Note: Because async hiredis always frees replies, you should
not call freeReplyObject in an async push callback. */
}
```
### Installing a custom handler
There are two ways to set your own PUSH handlers.
1. Set `push_cb` or `async_push_cb` in the `redisOptions` struct and connect with `redisConnectWithOptions` or `redisAsyncConnectWithOptions`.
```c
redisOptions = {0};
REDIS_OPTIONS_SET_TCP(&options, "127.0.0.1", 6379);
options->push_cb = my_push_handler;
redisContext *context = redisConnectWithOptions(&options);
```
2. Call `redisSetPushCallback` or `redisAsyncSetPushCallback` on a connected context.
```c
redisContext *context = redisConnect("127.0.0.1", 6379);
redisSetPushCallback(context, my_push_handler);
```
_Note `redisSetPushCallback` and `redisAsyncSetPushCallback` both return any currently configured handler, making it easy to override and then return to the old value._
### Specifying no handler
If you have a unique use-case where you don't want hiredis to automatically intercept and free PUSH replies, you will want to configure no handler at all. This can be done in two ways.
1. Set the `REDIS_OPT_NO_PUSH_AUTOFREE` flag in `redisOptions` and leave the callback function pointer `NULL`.
```c
redisOptions = {0};
REDIS_OPTIONS_SET_TCP(&options, "127.0.0.1", 6379);
options->options |= REDIS_OPT_NO_PUSH_AUTOFREE;
redisContext *context = redisConnectWithOptions(&options);
```
3. Call `redisSetPushCallback` with `NULL` once connected.
```c
redisContext *context = redisConnect("127.0.0.1", 6379);
redisSetPushCallback(context, NULL);
```
_Note: With no handler configured, calls to `redisCommand` may generate more than one reply, so this strategy is only applicable when there's some kind of blocking `redisGetReply()` loop (e.g. `MONITOR` or `SUBSCRIBE` workloads)._
## Allocator injection
Hiredis uses a pass-thru structure of function pointers defined in [alloc.h](https://github.com/redis/hiredis/blob/f5d25850/alloc.h#L41) that contain the currently configured allocation and deallocation functions. By default they just point to libc (`malloc`, `calloc`, `realloc`, etc).
### Overriding
One can override the allocators like so:
```c
hiredisAllocFuncs myfuncs = {
.mallocFn = my_malloc,
.callocFn = my_calloc,
.reallocFn = my_realloc,
.strdupFn = my_strdup,
.freeFn = my_free,
};
// Override allocators (function returns current allocators if needed)
hiredisAllocFuncs orig = hiredisSetAllocators(&myfuncs);
```
To reset the allocators to their default libc function simply call:
```c
hiredisResetAllocators();
```
## AUTHORS
Salvatore Sanfilippo (antirez at gmail),\
Pieter Noordhuis (pcnoordhuis at gmail)\
Michael Grunder (michael dot grunder at gmail)
_Hiredis is released under the BSD license._
-156
View File
@@ -1,156 +0,0 @@
#ifndef __HIREDIS_GLIB_H__
#define __HIREDIS_GLIB_H__
#include <glib.h>
#include "../hiredis.h"
#include "../async.h"
typedef struct
{
GSource source;
redisAsyncContext *ac;
GPollFD poll_fd;
} RedisSource;
static void
redis_source_add_read (gpointer data)
{
RedisSource *source = (RedisSource *)data;
g_return_if_fail(source);
source->poll_fd.events |= G_IO_IN;
g_main_context_wakeup(g_source_get_context((GSource *)data));
}
static void
redis_source_del_read (gpointer data)
{
RedisSource *source = (RedisSource *)data;
g_return_if_fail(source);
source->poll_fd.events &= ~G_IO_IN;
g_main_context_wakeup(g_source_get_context((GSource *)data));
}
static void
redis_source_add_write (gpointer data)
{
RedisSource *source = (RedisSource *)data;
g_return_if_fail(source);
source->poll_fd.events |= G_IO_OUT;
g_main_context_wakeup(g_source_get_context((GSource *)data));
}
static void
redis_source_del_write (gpointer data)
{
RedisSource *source = (RedisSource *)data;
g_return_if_fail(source);
source->poll_fd.events &= ~G_IO_OUT;
g_main_context_wakeup(g_source_get_context((GSource *)data));
}
static void
redis_source_cleanup (gpointer data)
{
RedisSource *source = (RedisSource *)data;
g_return_if_fail(source);
redis_source_del_read(source);
redis_source_del_write(source);
/*
* It is not our responsibility to remove ourself from the
* current main loop. However, we will remove the GPollFD.
*/
if (source->poll_fd.fd >= 0) {
g_source_remove_poll((GSource *)data, &source->poll_fd);
source->poll_fd.fd = -1;
}
}
static gboolean
redis_source_prepare (GSource *source,
gint *timeout_)
{
RedisSource *redis = (RedisSource *)source;
*timeout_ = -1;
return !!(redis->poll_fd.events & redis->poll_fd.revents);
}
static gboolean
redis_source_check (GSource *source)
{
RedisSource *redis = (RedisSource *)source;
return !!(redis->poll_fd.events & redis->poll_fd.revents);
}
static gboolean
redis_source_dispatch (GSource *source,
GSourceFunc callback,
gpointer user_data)
{
RedisSource *redis = (RedisSource *)source;
if ((redis->poll_fd.revents & G_IO_OUT)) {
redisAsyncHandleWrite(redis->ac);
redis->poll_fd.revents &= ~G_IO_OUT;
}
if ((redis->poll_fd.revents & G_IO_IN)) {
redisAsyncHandleRead(redis->ac);
redis->poll_fd.revents &= ~G_IO_IN;
}
if (callback) {
return callback(user_data);
}
return TRUE;
}
static void
redis_source_finalize (GSource *source)
{
RedisSource *redis = (RedisSource *)source;
if (redis->poll_fd.fd >= 0) {
g_source_remove_poll(source, &redis->poll_fd);
redis->poll_fd.fd = -1;
}
}
static GSource *
redis_source_new (redisAsyncContext *ac)
{
static GSourceFuncs source_funcs = {
.prepare = redis_source_prepare,
.check = redis_source_check,
.dispatch = redis_source_dispatch,
.finalize = redis_source_finalize,
};
redisContext *c = &ac->c;
RedisSource *source;
g_return_val_if_fail(ac != NULL, NULL);
source = (RedisSource *)g_source_new(&source_funcs, sizeof *source);
if (source == NULL)
return NULL;
source->ac = ac;
source->poll_fd.fd = c->fd;
source->poll_fd.events = 0;
source->poll_fd.revents = 0;
g_source_add_poll((GSource *)source, &source->poll_fd);
ac->ev.addRead = redis_source_add_read;
ac->ev.delRead = redis_source_del_read;
ac->ev.addWrite = redis_source_add_write;
ac->ev.delWrite = redis_source_del_write;
ac->ev.cleanup = redis_source_cleanup;
ac->ev.data = source;
return (GSource *)source;
}
#endif /* __HIREDIS_GLIB_H__ */
-84
View File
@@ -1,84 +0,0 @@
#ifndef __HIREDIS_IVYKIS_H__
#define __HIREDIS_IVYKIS_H__
#include <iv.h>
#include "../hiredis.h"
#include "../async.h"
typedef struct redisIvykisEvents {
redisAsyncContext *context;
struct iv_fd fd;
} redisIvykisEvents;
static void redisIvykisReadEvent(void *arg) {
redisAsyncContext *context = (redisAsyncContext *)arg;
redisAsyncHandleRead(context);
}
static void redisIvykisWriteEvent(void *arg) {
redisAsyncContext *context = (redisAsyncContext *)arg;
redisAsyncHandleWrite(context);
}
static void redisIvykisAddRead(void *privdata) {
redisIvykisEvents *e = (redisIvykisEvents*)privdata;
iv_fd_set_handler_in(&e->fd, redisIvykisReadEvent);
}
static void redisIvykisDelRead(void *privdata) {
redisIvykisEvents *e = (redisIvykisEvents*)privdata;
iv_fd_set_handler_in(&e->fd, NULL);
}
static void redisIvykisAddWrite(void *privdata) {
redisIvykisEvents *e = (redisIvykisEvents*)privdata;
iv_fd_set_handler_out(&e->fd, redisIvykisWriteEvent);
}
static void redisIvykisDelWrite(void *privdata) {
redisIvykisEvents *e = (redisIvykisEvents*)privdata;
iv_fd_set_handler_out(&e->fd, NULL);
}
static void redisIvykisCleanup(void *privdata) {
redisIvykisEvents *e = (redisIvykisEvents*)privdata;
iv_fd_unregister(&e->fd);
hi_free(e);
}
static int redisIvykisAttach(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
redisIvykisEvents *e;
/* Nothing should be attached when something is already attached */
if (ac->ev.data != NULL)
return REDIS_ERR;
/* Create container for context and r/w events */
e = (redisIvykisEvents*)hi_malloc(sizeof(*e));
if (e == NULL)
return REDIS_ERR;
e->context = ac;
/* Register functions to start/stop listening for events */
ac->ev.addRead = redisIvykisAddRead;
ac->ev.delRead = redisIvykisDelRead;
ac->ev.addWrite = redisIvykisAddWrite;
ac->ev.delWrite = redisIvykisDelWrite;
ac->ev.cleanup = redisIvykisCleanup;
ac->ev.data = e;
/* Initialize and install read/write events */
IV_FD_INIT(&e->fd);
e->fd.fd = c->fd;
e->fd.handler_in = redisIvykisReadEvent;
e->fd.handler_out = redisIvykisWriteEvent;
e->fd.handler_err = NULL;
e->fd.cookie = e->context;
iv_fd_register(&e->fd);
return REDIS_OK;
}
#endif
-175
View File
@@ -1,175 +0,0 @@
/*
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef __HIREDIS_LIBEVENT_H__
#define __HIREDIS_LIBEVENT_H__
#include <event2/event.h>
#include "../hiredis.h"
#include "../async.h"
#define REDIS_LIBEVENT_DELETED 0x01
#define REDIS_LIBEVENT_ENTERED 0x02
typedef struct redisLibeventEvents {
redisAsyncContext *context;
struct event *ev;
struct event_base *base;
struct timeval tv;
short flags;
short state;
} redisLibeventEvents;
static void redisLibeventDestroy(redisLibeventEvents *e) {
hi_free(e);
}
static void redisLibeventHandler(evutil_socket_t fd, short event, void *arg) {
((void)fd);
redisLibeventEvents *e = (redisLibeventEvents*)arg;
e->state |= REDIS_LIBEVENT_ENTERED;
#define CHECK_DELETED() if (e->state & REDIS_LIBEVENT_DELETED) {\
redisLibeventDestroy(e);\
return; \
}
if ((event & EV_TIMEOUT) && (e->state & REDIS_LIBEVENT_DELETED) == 0) {
redisAsyncHandleTimeout(e->context);
CHECK_DELETED();
}
if ((event & EV_READ) && e->context && (e->state & REDIS_LIBEVENT_DELETED) == 0) {
redisAsyncHandleRead(e->context);
CHECK_DELETED();
}
if ((event & EV_WRITE) && e->context && (e->state & REDIS_LIBEVENT_DELETED) == 0) {
redisAsyncHandleWrite(e->context);
CHECK_DELETED();
}
e->state &= ~REDIS_LIBEVENT_ENTERED;
#undef CHECK_DELETED
}
static void redisLibeventUpdate(void *privdata, short flag, int isRemove) {
redisLibeventEvents *e = (redisLibeventEvents *)privdata;
const struct timeval *tv = e->tv.tv_sec || e->tv.tv_usec ? &e->tv : NULL;
if (isRemove) {
if ((e->flags & flag) == 0) {
return;
} else {
e->flags &= ~flag;
}
} else {
if (e->flags & flag) {
return;
} else {
e->flags |= flag;
}
}
event_del(e->ev);
event_assign(e->ev, e->base, e->context->c.fd, e->flags | EV_PERSIST,
redisLibeventHandler, privdata);
event_add(e->ev, tv);
}
static void redisLibeventAddRead(void *privdata) {
redisLibeventUpdate(privdata, EV_READ, 0);
}
static void redisLibeventDelRead(void *privdata) {
redisLibeventUpdate(privdata, EV_READ, 1);
}
static void redisLibeventAddWrite(void *privdata) {
redisLibeventUpdate(privdata, EV_WRITE, 0);
}
static void redisLibeventDelWrite(void *privdata) {
redisLibeventUpdate(privdata, EV_WRITE, 1);
}
static void redisLibeventCleanup(void *privdata) {
redisLibeventEvents *e = (redisLibeventEvents*)privdata;
if (!e) {
return;
}
event_del(e->ev);
event_free(e->ev);
e->ev = NULL;
if (e->state & REDIS_LIBEVENT_ENTERED) {
e->state |= REDIS_LIBEVENT_DELETED;
} else {
redisLibeventDestroy(e);
}
}
static void redisLibeventSetTimeout(void *privdata, struct timeval tv) {
redisLibeventEvents *e = (redisLibeventEvents *)privdata;
short flags = e->flags;
e->flags = 0;
e->tv = tv;
redisLibeventUpdate(e, flags, 0);
}
static int redisLibeventAttach(redisAsyncContext *ac, struct event_base *base) {
redisContext *c = &(ac->c);
redisLibeventEvents *e;
/* Nothing should be attached when something is already attached */
if (ac->ev.data != NULL)
return REDIS_ERR;
/* Create container for context and r/w events */
e = (redisLibeventEvents*)hi_calloc(1, sizeof(*e));
if (e == NULL)
return REDIS_ERR;
e->context = ac;
/* Register functions to start/stop listening for events */
ac->ev.addRead = redisLibeventAddRead;
ac->ev.delRead = redisLibeventDelRead;
ac->ev.addWrite = redisLibeventAddWrite;
ac->ev.delWrite = redisLibeventDelWrite;
ac->ev.cleanup = redisLibeventCleanup;
ac->ev.scheduleTimer = redisLibeventSetTimeout;
ac->ev.data = e;
/* Initialize and install read/write events */
e->ev = event_new(base, c->fd, EV_READ | EV_WRITE, redisLibeventHandler, e);
e->base = base;
return REDIS_OK;
}
#endif
-123
View File
@@ -1,123 +0,0 @@
#ifndef __HIREDIS_LIBHV_H__
#define __HIREDIS_LIBHV_H__
#include <hv/hloop.h>
#include "../hiredis.h"
#include "../async.h"
typedef struct redisLibhvEvents {
hio_t *io;
htimer_t *timer;
} redisLibhvEvents;
static void redisLibhvHandleEvents(hio_t* io) {
redisAsyncContext* context = (redisAsyncContext*)hevent_userdata(io);
int events = hio_events(io);
int revents = hio_revents(io);
if (context && (events & HV_READ) && (revents & HV_READ)) {
redisAsyncHandleRead(context);
}
if (context && (events & HV_WRITE) && (revents & HV_WRITE)) {
redisAsyncHandleWrite(context);
}
}
static void redisLibhvAddRead(void *privdata) {
redisLibhvEvents* events = (redisLibhvEvents*)privdata;
hio_add(events->io, redisLibhvHandleEvents, HV_READ);
}
static void redisLibhvDelRead(void *privdata) {
redisLibhvEvents* events = (redisLibhvEvents*)privdata;
hio_del(events->io, HV_READ);
}
static void redisLibhvAddWrite(void *privdata) {
redisLibhvEvents* events = (redisLibhvEvents*)privdata;
hio_add(events->io, redisLibhvHandleEvents, HV_WRITE);
}
static void redisLibhvDelWrite(void *privdata) {
redisLibhvEvents* events = (redisLibhvEvents*)privdata;
hio_del(events->io, HV_WRITE);
}
static void redisLibhvCleanup(void *privdata) {
redisLibhvEvents* events = (redisLibhvEvents*)privdata;
if (events->timer)
htimer_del(events->timer);
hio_close(events->io);
hevent_set_userdata(events->io, NULL);
hi_free(events);
}
static void redisLibhvTimeout(htimer_t* timer) {
hio_t* io = (hio_t*)hevent_userdata(timer);
redisAsyncHandleTimeout((redisAsyncContext*)hevent_userdata(io));
}
static void redisLibhvSetTimeout(void *privdata, struct timeval tv) {
redisLibhvEvents* events;
uint32_t millis;
hloop_t* loop;
events = (redisLibhvEvents*)privdata;
millis = tv.tv_sec * 1000 + tv.tv_usec / 1000;
if (millis == 0) {
/* Libhv disallows zero'd timers so treat this as a delete or NO OP */
if (events->timer) {
htimer_del(events->timer);
events->timer = NULL;
}
} else if (events->timer == NULL) {
/* Add new timer */
loop = hevent_loop(events->io);
events->timer = htimer_add(loop, redisLibhvTimeout, millis, 1);
hevent_set_userdata(events->timer, events->io);
} else {
/* Update existing timer */
htimer_reset(events->timer, millis);
}
}
static int redisLibhvAttach(redisAsyncContext* ac, hloop_t* loop) {
redisContext *c = &(ac->c);
redisLibhvEvents *events;
hio_t* io = NULL;
if (ac->ev.data != NULL) {
return REDIS_ERR;
}
/* Create container struct to keep track of our io and any timer */
events = (redisLibhvEvents*)hi_malloc(sizeof(*events));
if (events == NULL) {
return REDIS_ERR;
}
io = hio_get(loop, c->fd);
if (io == NULL) {
hi_free(events);
return REDIS_ERR;
}
hevent_set_userdata(io, ac);
events->io = io;
events->timer = NULL;
ac->ev.addRead = redisLibhvAddRead;
ac->ev.delRead = redisLibhvDelRead;
ac->ev.addWrite = redisLibhvAddWrite;
ac->ev.delWrite = redisLibhvDelWrite;
ac->ev.cleanup = redisLibhvCleanup;
ac->ev.scheduleTimer = redisLibhvSetTimeout;
ac->ev.data = events;
return REDIS_OK;
}
#endif
-177
View File
@@ -1,177 +0,0 @@
#ifndef HIREDIS_LIBSDEVENT_H
#define HIREDIS_LIBSDEVENT_H
#include <systemd/sd-event.h>
#include "../hiredis.h"
#include "../async.h"
#define REDIS_LIBSDEVENT_DELETED 0x01
#define REDIS_LIBSDEVENT_ENTERED 0x02
typedef struct redisLibsdeventEvents {
redisAsyncContext *context;
struct sd_event *event;
struct sd_event_source *fdSource;
struct sd_event_source *timerSource;
int fd;
short flags;
short state;
} redisLibsdeventEvents;
static void redisLibsdeventDestroy(redisLibsdeventEvents *e) {
if (e->fdSource) {
e->fdSource = sd_event_source_disable_unref(e->fdSource);
}
if (e->timerSource) {
e->timerSource = sd_event_source_disable_unref(e->timerSource);
}
sd_event_unref(e->event);
hi_free(e);
}
static int redisLibsdeventTimeoutHandler(sd_event_source *s, uint64_t usec, void *userdata) {
((void)s);
((void)usec);
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
redisAsyncHandleTimeout(e->context);
return 0;
}
static int redisLibsdeventHandler(sd_event_source *s, int fd, uint32_t event, void *userdata) {
((void)s);
((void)fd);
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
e->state |= REDIS_LIBSDEVENT_ENTERED;
#define CHECK_DELETED() if (e->state & REDIS_LIBSDEVENT_DELETED) {\
redisLibsdeventDestroy(e);\
return 0; \
}
if ((event & EPOLLIN) && e->context && (e->state & REDIS_LIBSDEVENT_DELETED) == 0) {
redisAsyncHandleRead(e->context);
CHECK_DELETED();
}
if ((event & EPOLLOUT) && e->context && (e->state & REDIS_LIBSDEVENT_DELETED) == 0) {
redisAsyncHandleWrite(e->context);
CHECK_DELETED();
}
e->state &= ~REDIS_LIBSDEVENT_ENTERED;
#undef CHECK_DELETED
return 0;
}
static void redisLibsdeventAddRead(void *userdata) {
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
if (e->flags & EPOLLIN) {
return;
}
e->flags |= EPOLLIN;
if (e->flags & EPOLLOUT) {
sd_event_source_set_io_events(e->fdSource, e->flags);
} else {
sd_event_add_io(e->event, &e->fdSource, e->fd, e->flags, redisLibsdeventHandler, e);
}
}
static void redisLibsdeventDelRead(void *userdata) {
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
e->flags &= ~EPOLLIN;
if (e->flags) {
sd_event_source_set_io_events(e->fdSource, e->flags);
} else {
e->fdSource = sd_event_source_disable_unref(e->fdSource);
}
}
static void redisLibsdeventAddWrite(void *userdata) {
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
if (e->flags & EPOLLOUT) {
return;
}
e->flags |= EPOLLOUT;
if (e->flags & EPOLLIN) {
sd_event_source_set_io_events(e->fdSource, e->flags);
} else {
sd_event_add_io(e->event, &e->fdSource, e->fd, e->flags, redisLibsdeventHandler, e);
}
}
static void redisLibsdeventDelWrite(void *userdata) {
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
e->flags &= ~EPOLLOUT;
if (e->flags) {
sd_event_source_set_io_events(e->fdSource, e->flags);
} else {
e->fdSource = sd_event_source_disable_unref(e->fdSource);
}
}
static void redisLibsdeventCleanup(void *userdata) {
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
if (!e) {
return;
}
if (e->state & REDIS_LIBSDEVENT_ENTERED) {
e->state |= REDIS_LIBSDEVENT_DELETED;
} else {
redisLibsdeventDestroy(e);
}
}
static void redisLibsdeventSetTimeout(void *userdata, struct timeval tv) {
redisLibsdeventEvents *e = (redisLibsdeventEvents *)userdata;
uint64_t usec = tv.tv_sec * 1000000 + tv.tv_usec;
if (!e->timerSource) {
sd_event_add_time_relative(e->event, &e->timerSource, CLOCK_MONOTONIC, usec, 1, redisLibsdeventTimeoutHandler, e);
} else {
sd_event_source_set_time_relative(e->timerSource, usec);
}
}
static int redisLibsdeventAttach(redisAsyncContext *ac, struct sd_event *event) {
redisContext *c = &(ac->c);
redisLibsdeventEvents *e;
/* Nothing should be attached when something is already attached */
if (ac->ev.data != NULL)
return REDIS_ERR;
/* Create container for context and r/w events */
e = (redisLibsdeventEvents*)hi_calloc(1, sizeof(*e));
if (e == NULL)
return REDIS_ERR;
/* Initialize and increase event refcount */
e->context = ac;
e->event = event;
e->fd = c->fd;
sd_event_ref(event);
/* Register functions to start/stop listening for events */
ac->ev.addRead = redisLibsdeventAddRead;
ac->ev.delRead = redisLibsdeventDelRead;
ac->ev.addWrite = redisLibsdeventAddWrite;
ac->ev.delWrite = redisLibsdeventDelWrite;
ac->ev.cleanup = redisLibsdeventCleanup;
ac->ev.scheduleTimer = redisLibsdeventSetTimeout;
ac->ev.data = e;
return REDIS_OK;
}
#endif
-171
View File
@@ -1,171 +0,0 @@
#ifndef __HIREDIS_LIBUV_H__
#define __HIREDIS_LIBUV_H__
#include <stdlib.h>
#include <uv.h>
#include "../hiredis.h"
#include "../async.h"
#include <string.h>
typedef struct redisLibuvEvents {
redisAsyncContext* context;
uv_poll_t handle;
uv_timer_t timer;
int events;
} redisLibuvEvents;
static void redisLibuvPoll(uv_poll_t* handle, int status, int events) {
redisLibuvEvents* p = (redisLibuvEvents*)handle->data;
int ev = (status ? p->events : events);
if (p->context != NULL && (ev & UV_READABLE)) {
redisAsyncHandleRead(p->context);
}
if (p->context != NULL && (ev & UV_WRITABLE)) {
redisAsyncHandleWrite(p->context);
}
}
static void redisLibuvAddRead(void *privdata) {
redisLibuvEvents* p = (redisLibuvEvents*)privdata;
if (p->events & UV_READABLE) {
return;
}
p->events |= UV_READABLE;
uv_poll_start(&p->handle, p->events, redisLibuvPoll);
}
static void redisLibuvDelRead(void *privdata) {
redisLibuvEvents* p = (redisLibuvEvents*)privdata;
p->events &= ~UV_READABLE;
if (p->events) {
uv_poll_start(&p->handle, p->events, redisLibuvPoll);
} else {
uv_poll_stop(&p->handle);
}
}
static void redisLibuvAddWrite(void *privdata) {
redisLibuvEvents* p = (redisLibuvEvents*)privdata;
if (p->events & UV_WRITABLE) {
return;
}
p->events |= UV_WRITABLE;
uv_poll_start(&p->handle, p->events, redisLibuvPoll);
}
static void redisLibuvDelWrite(void *privdata) {
redisLibuvEvents* p = (redisLibuvEvents*)privdata;
p->events &= ~UV_WRITABLE;
if (p->events) {
uv_poll_start(&p->handle, p->events, redisLibuvPoll);
} else {
uv_poll_stop(&p->handle);
}
}
static void on_timer_close(uv_handle_t *handle) {
redisLibuvEvents* p = (redisLibuvEvents*)handle->data;
p->timer.data = NULL;
if (!p->handle.data) {
// both timer and handle are closed
hi_free(p);
}
// else, wait for `on_handle_close`
}
static void on_handle_close(uv_handle_t *handle) {
redisLibuvEvents* p = (redisLibuvEvents*)handle->data;
p->handle.data = NULL;
if (!p->timer.data) {
// timer never started, or timer already destroyed
hi_free(p);
}
// else, wait for `on_timer_close`
}
// libuv removed `status` parameter since v0.11.23
// see: https://github.com/libuv/libuv/blob/v0.11.23/include/uv.h
#if (UV_VERSION_MAJOR == 0 && UV_VERSION_MINOR < 11) || \
(UV_VERSION_MAJOR == 0 && UV_VERSION_MINOR == 11 && UV_VERSION_PATCH < 23)
static void redisLibuvTimeout(uv_timer_t *timer, int status) {
(void)status; // unused
#else
static void redisLibuvTimeout(uv_timer_t *timer) {
#endif
redisLibuvEvents *e = (redisLibuvEvents*)timer->data;
redisAsyncHandleTimeout(e->context);
}
static void redisLibuvSetTimeout(void *privdata, struct timeval tv) {
redisLibuvEvents* p = (redisLibuvEvents*)privdata;
uint64_t millsec = tv.tv_sec * 1000 + tv.tv_usec / 1000.0;
if (!p->timer.data) {
// timer is uninitialized
if (uv_timer_init(p->handle.loop, &p->timer) != 0) {
return;
}
p->timer.data = p;
}
// updates the timeout if the timer has already started
// or start the timer
uv_timer_start(&p->timer, redisLibuvTimeout, millsec, 0);
}
static void redisLibuvCleanup(void *privdata) {
redisLibuvEvents* p = (redisLibuvEvents*)privdata;
p->context = NULL; // indicate that context might no longer exist
if (p->timer.data) {
uv_close((uv_handle_t*)&p->timer, on_timer_close);
}
uv_close((uv_handle_t*)&p->handle, on_handle_close);
}
static int redisLibuvAttach(redisAsyncContext* ac, uv_loop_t* loop) {
redisContext *c = &(ac->c);
if (ac->ev.data != NULL) {
return REDIS_ERR;
}
ac->ev.addRead = redisLibuvAddRead;
ac->ev.delRead = redisLibuvDelRead;
ac->ev.addWrite = redisLibuvAddWrite;
ac->ev.delWrite = redisLibuvDelWrite;
ac->ev.cleanup = redisLibuvCleanup;
ac->ev.scheduleTimer = redisLibuvSetTimeout;
redisLibuvEvents* p = (redisLibuvEvents*)hi_malloc(sizeof(*p));
if (p == NULL)
return REDIS_ERR;
memset(p, 0, sizeof(*p));
if (uv_poll_init_socket(loop, &p->handle, c->fd) != 0) {
return REDIS_ERR;
}
ac->ev.data = p;
p->handle.data = p;
p->context = ac;
return REDIS_OK;
}
#endif
-115
View File
@@ -1,115 +0,0 @@
//
// Created by Дмитрий Бахвалов on 13.07.15.
// Copyright (c) 2015 Dmitry Bakhvalov. All rights reserved.
//
#ifndef __HIREDIS_MACOSX_H__
#define __HIREDIS_MACOSX_H__
#include <CoreFoundation/CoreFoundation.h>
#include "../hiredis.h"
#include "../async.h"
typedef struct {
redisAsyncContext *context;
CFSocketRef socketRef;
CFRunLoopSourceRef sourceRef;
} RedisRunLoop;
static int freeRedisRunLoop(RedisRunLoop* redisRunLoop) {
if( redisRunLoop != NULL ) {
if( redisRunLoop->sourceRef != NULL ) {
CFRunLoopSourceInvalidate(redisRunLoop->sourceRef);
CFRelease(redisRunLoop->sourceRef);
}
if( redisRunLoop->socketRef != NULL ) {
CFSocketInvalidate(redisRunLoop->socketRef);
CFRelease(redisRunLoop->socketRef);
}
hi_free(redisRunLoop);
}
return REDIS_ERR;
}
static void redisMacOSAddRead(void *privdata) {
RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
CFSocketEnableCallBacks(redisRunLoop->socketRef, kCFSocketReadCallBack);
}
static void redisMacOSDelRead(void *privdata) {
RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
CFSocketDisableCallBacks(redisRunLoop->socketRef, kCFSocketReadCallBack);
}
static void redisMacOSAddWrite(void *privdata) {
RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
CFSocketEnableCallBacks(redisRunLoop->socketRef, kCFSocketWriteCallBack);
}
static void redisMacOSDelWrite(void *privdata) {
RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
CFSocketDisableCallBacks(redisRunLoop->socketRef, kCFSocketWriteCallBack);
}
static void redisMacOSCleanup(void *privdata) {
RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
freeRedisRunLoop(redisRunLoop);
}
static void redisMacOSAsyncCallback(CFSocketRef __unused s, CFSocketCallBackType callbackType, CFDataRef __unused address, const void __unused *data, void *info) {
redisAsyncContext* context = (redisAsyncContext*) info;
switch (callbackType) {
case kCFSocketReadCallBack:
redisAsyncHandleRead(context);
break;
case kCFSocketWriteCallBack:
redisAsyncHandleWrite(context);
break;
default:
break;
}
}
static int redisMacOSAttach(redisAsyncContext *redisAsyncCtx, CFRunLoopRef runLoop) {
redisContext *redisCtx = &(redisAsyncCtx->c);
/* Nothing should be attached when something is already attached */
if( redisAsyncCtx->ev.data != NULL ) return REDIS_ERR;
RedisRunLoop* redisRunLoop = (RedisRunLoop*) hi_calloc(1, sizeof(RedisRunLoop));
if (redisRunLoop == NULL)
return REDIS_ERR;
/* Setup redis stuff */
redisRunLoop->context = redisAsyncCtx;
redisAsyncCtx->ev.addRead = redisMacOSAddRead;
redisAsyncCtx->ev.delRead = redisMacOSDelRead;
redisAsyncCtx->ev.addWrite = redisMacOSAddWrite;
redisAsyncCtx->ev.delWrite = redisMacOSDelWrite;
redisAsyncCtx->ev.cleanup = redisMacOSCleanup;
redisAsyncCtx->ev.data = redisRunLoop;
/* Initialize and install read/write events */
CFSocketContext socketCtx = { 0, redisAsyncCtx, NULL, NULL, NULL };
redisRunLoop->socketRef = CFSocketCreateWithNative(NULL, redisCtx->fd,
kCFSocketReadCallBack | kCFSocketWriteCallBack,
redisMacOSAsyncCallback,
&socketCtx);
if( !redisRunLoop->socketRef ) return freeRedisRunLoop(redisRunLoop);
redisRunLoop->sourceRef = CFSocketCreateRunLoopSource(NULL, redisRunLoop->socketRef, 0);
if( !redisRunLoop->sourceRef ) return freeRedisRunLoop(redisRunLoop);
CFRunLoopAddSource(runLoop, redisRunLoop->sourceRef, kCFRunLoopDefaultMode);
return REDIS_OK;
}
#endif
-135
View File
@@ -1,135 +0,0 @@
/*-
* Copyright (C) 2014 Pietro Cerutti <gahr@gahr.ch>
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef __HIREDIS_QT_H__
#define __HIREDIS_QT_H__
#include <QSocketNotifier>
#include "../async.h"
static void RedisQtAddRead(void *);
static void RedisQtDelRead(void *);
static void RedisQtAddWrite(void *);
static void RedisQtDelWrite(void *);
static void RedisQtCleanup(void *);
class RedisQtAdapter : public QObject {
Q_OBJECT
friend
void RedisQtAddRead(void * adapter) {
RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
a->addRead();
}
friend
void RedisQtDelRead(void * adapter) {
RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
a->delRead();
}
friend
void RedisQtAddWrite(void * adapter) {
RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
a->addWrite();
}
friend
void RedisQtDelWrite(void * adapter) {
RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
a->delWrite();
}
friend
void RedisQtCleanup(void * adapter) {
RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
a->cleanup();
}
public:
RedisQtAdapter(QObject * parent = 0)
: QObject(parent), m_ctx(0), m_read(0), m_write(0) { }
~RedisQtAdapter() {
if (m_ctx != 0) {
m_ctx->ev.data = NULL;
}
}
int setContext(redisAsyncContext * ac) {
if (ac->ev.data != NULL) {
return REDIS_ERR;
}
m_ctx = ac;
m_ctx->ev.data = this;
m_ctx->ev.addRead = RedisQtAddRead;
m_ctx->ev.delRead = RedisQtDelRead;
m_ctx->ev.addWrite = RedisQtAddWrite;
m_ctx->ev.delWrite = RedisQtDelWrite;
m_ctx->ev.cleanup = RedisQtCleanup;
return REDIS_OK;
}
private:
void addRead() {
if (m_read) return;
m_read = new QSocketNotifier(m_ctx->c.fd, QSocketNotifier::Read, 0);
connect(m_read, SIGNAL(activated(int)), this, SLOT(read()));
}
void delRead() {
if (!m_read) return;
delete m_read;
m_read = 0;
}
void addWrite() {
if (m_write) return;
m_write = new QSocketNotifier(m_ctx->c.fd, QSocketNotifier::Write, 0);
connect(m_write, SIGNAL(activated(int)), this, SLOT(write()));
}
void delWrite() {
if (!m_write) return;
delete m_write;
m_write = 0;
}
void cleanup() {
delRead();
delWrite();
}
private slots:
void read() { redisAsyncHandleRead(m_ctx); }
void write() { redisAsyncHandleWrite(m_ctx); }
private:
redisAsyncContext * m_ctx;
QSocketNotifier * m_read;
QSocketNotifier * m_write;
};
#endif /* !__HIREDIS_QT_H__ */
-144
View File
@@ -1,144 +0,0 @@
#ifndef __HIREDIS_REDISMODULEAPI_H__
#define __HIREDIS_REDISMODULEAPI_H__
#include "redismodule.h"
#include "../async.h"
#include "../hiredis.h"
#include <sys/types.h>
typedef struct redisModuleEvents {
redisAsyncContext *context;
RedisModuleCtx *module_ctx;
int fd;
int reading, writing;
int timer_active;
RedisModuleTimerID timer_id;
} redisModuleEvents;
static inline void redisModuleReadEvent(int fd, void *privdata, int mask) {
(void) fd;
(void) mask;
redisModuleEvents *e = (redisModuleEvents*)privdata;
redisAsyncHandleRead(e->context);
}
static inline void redisModuleWriteEvent(int fd, void *privdata, int mask) {
(void) fd;
(void) mask;
redisModuleEvents *e = (redisModuleEvents*)privdata;
redisAsyncHandleWrite(e->context);
}
static inline void redisModuleAddRead(void *privdata) {
redisModuleEvents *e = (redisModuleEvents*)privdata;
if (!e->reading) {
e->reading = 1;
RedisModule_EventLoopAdd(e->fd, REDISMODULE_EVENTLOOP_READABLE, redisModuleReadEvent, e);
}
}
static inline void redisModuleDelRead(void *privdata) {
redisModuleEvents *e = (redisModuleEvents*)privdata;
if (e->reading) {
e->reading = 0;
RedisModule_EventLoopDel(e->fd, REDISMODULE_EVENTLOOP_READABLE);
}
}
static inline void redisModuleAddWrite(void *privdata) {
redisModuleEvents *e = (redisModuleEvents*)privdata;
if (!e->writing) {
e->writing = 1;
RedisModule_EventLoopAdd(e->fd, REDISMODULE_EVENTLOOP_WRITABLE, redisModuleWriteEvent, e);
}
}
static inline void redisModuleDelWrite(void *privdata) {
redisModuleEvents *e = (redisModuleEvents*)privdata;
if (e->writing) {
e->writing = 0;
RedisModule_EventLoopDel(e->fd, REDISMODULE_EVENTLOOP_WRITABLE);
}
}
static inline void redisModuleStopTimer(void *privdata) {
redisModuleEvents *e = (redisModuleEvents*)privdata;
if (e->timer_active) {
RedisModule_StopTimer(e->module_ctx, e->timer_id, NULL);
}
e->timer_active = 0;
}
static inline void redisModuleCleanup(void *privdata) {
redisModuleEvents *e = (redisModuleEvents*)privdata;
redisModuleDelRead(privdata);
redisModuleDelWrite(privdata);
redisModuleStopTimer(privdata);
hi_free(e);
}
static inline void redisModuleTimeout(RedisModuleCtx *ctx, void *privdata) {
(void) ctx;
redisModuleEvents *e = (redisModuleEvents*)privdata;
e->timer_active = 0;
redisAsyncHandleTimeout(e->context);
}
static inline void redisModuleSetTimeout(void *privdata, struct timeval tv) {
redisModuleEvents* e = (redisModuleEvents*)privdata;
redisModuleStopTimer(privdata);
mstime_t millis = tv.tv_sec * 1000 + tv.tv_usec / 1000.0;
e->timer_id = RedisModule_CreateTimer(e->module_ctx, millis, redisModuleTimeout, e);
e->timer_active = 1;
}
/* Check if Redis version is compatible with the adapter. */
static inline int redisModuleCompatibilityCheck(void) {
if (!RedisModule_EventLoopAdd ||
!RedisModule_EventLoopDel ||
!RedisModule_CreateTimer ||
!RedisModule_StopTimer) {
return REDIS_ERR;
}
return REDIS_OK;
}
static inline int redisModuleAttach(redisAsyncContext *ac, RedisModuleCtx *module_ctx) {
redisContext *c = &(ac->c);
redisModuleEvents *e;
/* Nothing should be attached when something is already attached */
if (ac->ev.data != NULL)
return REDIS_ERR;
/* Create container for context and r/w events */
e = (redisModuleEvents*)hi_malloc(sizeof(*e));
if (e == NULL)
return REDIS_ERR;
e->context = ac;
e->module_ctx = module_ctx;
e->fd = c->fd;
e->reading = e->writing = 0;
e->timer_active = 0;
/* Register functions to start/stop listening for events */
ac->ev.addRead = redisModuleAddRead;
ac->ev.delRead = redisModuleDelRead;
ac->ev.addWrite = redisModuleAddWrite;
ac->ev.delWrite = redisModuleDelWrite;
ac->ev.cleanup = redisModuleCleanup;
ac->ev.scheduleTimer = redisModuleSetTimeout;
ac->ev.data = e;
return REDIS_OK;
}
#endif
-24
View File
@@ -1,24 +0,0 @@
# Appveyor configuration file for CI build of hiredis on Windows (under Cygwin)
environment:
matrix:
- CYG_BASH: C:\cygwin64\bin\bash
CC: gcc
- CYG_BASH: C:\cygwin\bin\bash
CC: gcc
CFLAGS: -m32
CXXFLAGS: -m32
LDFLAGS: -m32
clone_depth: 1
# Attempt to ensure we don't try to convert line endings to Win32 CRLF as this will cause build to fail
init:
- git config --global core.autocrlf input
# Install needed build dependencies
install:
- '%CYG_BASH% -lc "cygcheck -dc cygwin"'
build_script:
- 'echo building...'
- '%CYG_BASH% -lc "cd $APPVEYOR_BUILD_FOLDER; exec 0</dev/null; mkdir build && cd build && cmake .. -G \"Unix Makefiles\" && make VERBOSE=1"'
-1034
View File
File diff suppressed because it is too large Load Diff
-75
View File
@@ -1,75 +0,0 @@
/*
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef __HIREDIS_ASYNC_PRIVATE_H
#define __HIREDIS_ASYNC_PRIVATE_H
#define _EL_ADD_READ(ctx) \
do { \
refreshTimeout(ctx); \
if ((ctx)->ev.addRead) (ctx)->ev.addRead((ctx)->ev.data); \
} while (0)
#define _EL_DEL_READ(ctx) do { \
if ((ctx)->ev.delRead) (ctx)->ev.delRead((ctx)->ev.data); \
} while(0)
#define _EL_ADD_WRITE(ctx) \
do { \
refreshTimeout(ctx); \
if ((ctx)->ev.addWrite) (ctx)->ev.addWrite((ctx)->ev.data); \
} while (0)
#define _EL_DEL_WRITE(ctx) do { \
if ((ctx)->ev.delWrite) (ctx)->ev.delWrite((ctx)->ev.data); \
} while(0)
#define _EL_CLEANUP(ctx) do { \
if ((ctx)->ev.cleanup) (ctx)->ev.cleanup((ctx)->ev.data); \
ctx->ev.cleanup = NULL; \
} while(0)
static inline void refreshTimeout(redisAsyncContext *ctx) {
#define REDIS_TIMER_ISSET(tvp) \
(tvp && ((tvp)->tv_sec || (tvp)->tv_usec))
#define REDIS_EL_TIMER(ac, tvp) \
if ((ac)->ev.scheduleTimer && REDIS_TIMER_ISSET(tvp)) { \
(ac)->ev.scheduleTimer((ac)->ev.data, *(tvp)); \
}
if (ctx->c.flags & REDIS_CONNECTED) {
REDIS_EL_TIMER(ctx, ctx->c.command_timeout);
} else {
REDIS_EL_TIMER(ctx, ctx->c.connect_timeout);
}
}
void __redisAsyncDisconnect(redisAsyncContext *ac);
void redisProcessCallbacks(redisAsyncContext *ac);
#endif /* __HIREDIS_ASYNC_PRIVATE_H */
-61
View File
@@ -1,61 +0,0 @@
INCLUDE(FindPkgConfig)
# Check for GLib
PKG_CHECK_MODULES(GLIB2 glib-2.0)
if (GLIB2_FOUND)
INCLUDE_DIRECTORIES(${GLIB2_INCLUDE_DIRS})
LINK_DIRECTORIES(${GLIB2_LIBRARY_DIRS})
ADD_EXECUTABLE(example-glib example-glib.c)
TARGET_LINK_LIBRARIES(example-glib hiredis ${GLIB2_LIBRARIES})
ENDIF(GLIB2_FOUND)
FIND_PATH(LIBEV ev.h
HINTS /usr/local /usr/opt/local
ENV LIBEV_INCLUDE_DIR)
if (LIBEV)
# Just compile and link with libev
ADD_EXECUTABLE(example-libev example-libev.c)
TARGET_LINK_LIBRARIES(example-libev hiredis ev)
ENDIF()
FIND_PATH(LIBEVENT event.h)
if (LIBEVENT)
ADD_EXECUTABLE(example-libevent example-libevent.c)
TARGET_LINK_LIBRARIES(example-libevent hiredis event)
ENDIF()
FIND_PATH(LIBHV hv/hv.h)
IF (LIBHV)
ADD_EXECUTABLE(example-libhv example-libhv.c)
TARGET_LINK_LIBRARIES(example-libhv hiredis hv)
ENDIF()
FIND_PATH(LIBUV uv.h)
IF (LIBUV)
ADD_EXECUTABLE(example-libuv example-libuv.c)
TARGET_LINK_LIBRARIES(example-libuv hiredis uv)
ENDIF()
FIND_PATH(LIBSDEVENT systemd/sd-event.h)
IF (LIBSDEVENT)
ADD_EXECUTABLE(example-libsdevent example-libsdevent.c)
TARGET_LINK_LIBRARIES(example-libsdevent hiredis systemd)
ENDIF()
IF (APPLE)
FIND_LIBRARY(CF CoreFoundation)
ADD_EXECUTABLE(example-macosx example-macosx.c)
TARGET_LINK_LIBRARIES(example-macosx hiredis ${CF})
ENDIF()
IF (ENABLE_SSL)
ADD_EXECUTABLE(example-ssl example-ssl.c)
TARGET_LINK_LIBRARIES(example-ssl hiredis hiredis_ssl)
ENDIF()
ADD_EXECUTABLE(example example.c)
TARGET_LINK_LIBRARIES(example hiredis)
ADD_EXECUTABLE(example-push example-push.c)
TARGET_LINK_LIBRARIES(example-push hiredis)
-62
View File
@@ -1,62 +0,0 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/ae.h>
/* Put event loop in the global scope, so it can be explicitly stopped */
static aeEventLoop *loop;
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) return;
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
redisAsyncDisconnect(c);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
aeStop(loop);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
aeStop(loop);
return;
}
printf("Disconnected...\n");
aeStop(loop);
}
int main (int argc, char **argv) {
signal(SIGPIPE, SIG_IGN);
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
loop = aeCreateEventLoop(64);
redisAeAttach(loop, c);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
aeMain(loop);
return 0;
}
-73
View File
@@ -1,73 +0,0 @@
#include <stdlib.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/glib.h>
static GMainLoop *mainloop;
static void
connect_cb (const redisAsyncContext *ac G_GNUC_UNUSED,
int status)
{
if (status != REDIS_OK) {
g_printerr("Failed to connect: %s\n", ac->errstr);
g_main_loop_quit(mainloop);
} else {
g_printerr("Connected...\n");
}
}
static void
disconnect_cb (const redisAsyncContext *ac G_GNUC_UNUSED,
int status)
{
if (status != REDIS_OK) {
g_error("Failed to disconnect: %s", ac->errstr);
} else {
g_printerr("Disconnected...\n");
g_main_loop_quit(mainloop);
}
}
static void
command_cb(redisAsyncContext *ac,
gpointer r,
gpointer user_data G_GNUC_UNUSED)
{
redisReply *reply = r;
if (reply) {
g_print("REPLY: %s\n", reply->str);
}
redisAsyncDisconnect(ac);
}
gint
main (gint argc G_GNUC_UNUSED,
gchar *argv[] G_GNUC_UNUSED)
{
redisAsyncContext *ac;
GMainContext *context = NULL;
GSource *source;
ac = redisAsyncConnect("127.0.0.1", 6379);
if (ac->err) {
g_printerr("%s\n", ac->errstr);
exit(EXIT_FAILURE);
}
source = redis_source_new(ac);
mainloop = g_main_loop_new(context, FALSE);
g_source_attach(source, context);
redisAsyncSetConnectCallback(ac, connect_cb);
redisAsyncSetDisconnectCallback(ac, disconnect_cb);
redisAsyncCommand(ac, command_cb, NULL, "SET key 1234");
redisAsyncCommand(ac, command_cb, NULL, "GET key");
g_main_loop_run(mainloop);
return EXIT_SUCCESS;
}
-60
View File
@@ -1,60 +0,0 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/ivykis.h>
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) return;
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
redisAsyncDisconnect(c);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Disconnected...\n");
}
int main (int argc, char **argv) {
#ifndef _WIN32
signal(SIGPIPE, SIG_IGN);
#endif
iv_init();
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
redisIvykisAttach(c);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
iv_main();
iv_deinit();
return 0;
}
-54
View File
@@ -1,54 +0,0 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/libev.h>
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) return;
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
redisAsyncDisconnect(c);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Disconnected...\n");
}
int main (int argc, char **argv) {
#ifndef _WIN32
signal(SIGPIPE, SIG_IGN);
#endif
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
redisLibevAttach(EV_DEFAULT_ c);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
ev_loop(EV_DEFAULT_ 0);
return 0;
}
-90
View File
@@ -1,90 +0,0 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <hiredis.h>
#include <hiredis_ssl.h>
#include <async.h>
#include <adapters/libevent.h>
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) return;
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
redisAsyncDisconnect(c);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Disconnected...\n");
}
int main (int argc, char **argv) {
#ifndef _WIN32
signal(SIGPIPE, SIG_IGN);
#endif
struct event_base *base = event_base_new();
if (argc < 5) {
fprintf(stderr,
"Usage: %s <key> <host> <port> <cert> <certKey> [ca]\n", argv[0]);
exit(1);
}
const char *value = argv[1];
size_t nvalue = strlen(value);
const char *hostname = argv[2];
int port = atoi(argv[3]);
const char *cert = argv[4];
const char *certKey = argv[5];
const char *caCert = argc > 5 ? argv[6] : NULL;
redisSSLContext *ssl;
redisSSLContextError ssl_error = REDIS_SSL_CTX_NONE;
redisInitOpenSSL();
ssl = redisCreateSSLContext(caCert, NULL,
cert, certKey, NULL, &ssl_error);
if (!ssl) {
printf("Error: %s\n", redisSSLContextGetError(ssl_error));
return 1;
}
redisAsyncContext *c = redisAsyncConnect(hostname, port);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
if (redisInitiateSSLWithContext(&c->c, ssl) != REDIS_OK) {
printf("SSL Error!\n");
exit(1);
}
redisLibeventAttach(c,base);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncCommand(c, NULL, NULL, "SET key %b", value, nvalue);
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
event_base_dispatch(base);
redisFreeSSLContext(ssl);
return 0;
}
-67
View File
@@ -1,67 +0,0 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/libevent.h>
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) {
if (c->errstr) {
printf("errstr: %s\n", c->errstr);
}
return;
}
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
redisAsyncDisconnect(c);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Disconnected...\n");
}
int main (int argc, char **argv) {
#ifndef _WIN32
signal(SIGPIPE, SIG_IGN);
#endif
struct event_base *base = event_base_new();
redisOptions options = {0};
REDIS_OPTIONS_SET_TCP(&options, "127.0.0.1", 6379);
struct timeval tv = {0};
tv.tv_sec = 1;
options.connect_timeout = &tv;
redisAsyncContext *c = redisAsyncConnectWithOptions(&options);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
redisLibeventAttach(c,base);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
event_base_dispatch(base);
return 0;
}
-70
View File
@@ -1,70 +0,0 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/libhv.h>
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) return;
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
redisAsyncDisconnect(c);
}
void debugCallback(redisAsyncContext *c, void *r, void *privdata) {
(void)privdata;
redisReply *reply = r;
if (reply == NULL) {
printf("`DEBUG SLEEP` error: %s\n", c->errstr ? c->errstr : "unknown error");
return;
}
redisAsyncDisconnect(c);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Disconnected...\n");
}
int main (int argc, char **argv) {
#ifndef _WIN32
signal(SIGPIPE, SIG_IGN);
#endif
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
hloop_t* loop = hloop_new(HLOOP_FLAG_QUIT_WHEN_NO_ACTIVE_EVENTS);
redisLibhvAttach(c, loop);
redisAsyncSetTimeout(c, (struct timeval){.tv_sec = 0, .tv_usec = 500000});
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
redisAsyncCommand(c, debugCallback, NULL, "DEBUG SLEEP %d", 1);
hloop_run(loop);
hloop_free(&loop);
return 0;
}
-66
View File
@@ -1,66 +0,0 @@
//
// Created by Дмитрий Бахвалов on 13.07.15.
// Copyright (c) 2015 Dmitry Bakhvalov. All rights reserved.
//
#include <stdio.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/macosx.h>
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) return;
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
redisAsyncDisconnect(c);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
CFRunLoopStop(CFRunLoopGetCurrent());
printf("Disconnected...\n");
}
int main (int argc, char **argv) {
signal(SIGPIPE, SIG_IGN);
CFRunLoopRef loop = CFRunLoopGetCurrent();
if( !loop ) {
printf("Error: Cannot get current run loop\n");
return 1;
}
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
redisMacOSAttach(c, loop);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
CFRunLoopRun();
return 0;
}
-62
View File
@@ -1,62 +0,0 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <unistd.h>
#include <async.h>
#include <adapters/poll.h>
/* Put in the global scope, so that loop can be explicitly stopped */
static int exit_loop = 0;
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) return;
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
redisAsyncDisconnect(c);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
exit_loop = 1;
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
exit_loop = 1;
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Disconnected...\n");
}
int main (int argc, char **argv) {
signal(SIGPIPE, SIG_IGN);
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
redisPollAttach(c);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
while (!exit_loop)
{
redisPollTick(c, 0.1);
}
return 0;
}
-46
View File
@@ -1,46 +0,0 @@
#include <iostream>
using namespace std;
#include <QCoreApplication>
#include <QTimer>
#include "example-qt.h"
void getCallback(redisAsyncContext *, void * r, void * privdata) {
redisReply * reply = static_cast<redisReply *>(r);
ExampleQt * ex = static_cast<ExampleQt *>(privdata);
if (reply == nullptr || ex == nullptr) return;
cout << "key: " << reply->str << endl;
ex->finish();
}
void ExampleQt::run() {
m_ctx = redisAsyncConnect("localhost", 6379);
if (m_ctx->err) {
cerr << "Error: " << m_ctx->errstr << endl;
redisAsyncFree(m_ctx);
emit finished();
}
m_adapter.setContext(m_ctx);
redisAsyncCommand(m_ctx, NULL, NULL, "SET key %s", m_value);
redisAsyncCommand(m_ctx, getCallback, this, "GET key");
}
int main (int argc, char **argv) {
QCoreApplication app(argc, argv);
ExampleQt example(argv[argc-1]);
QObject::connect(&example, SIGNAL(finished()), &app, SLOT(quit()));
QTimer::singleShot(0, &example, SLOT(run()));
return app.exec();
}
-32
View File
@@ -1,32 +0,0 @@
#ifndef __HIREDIS_EXAMPLE_QT_H
#define __HIREDIS_EXAMPLE_QT_H
#include <adapters/qt.h>
class ExampleQt : public QObject {
Q_OBJECT
public:
ExampleQt(const char * value, QObject * parent = 0)
: QObject(parent), m_value(value) {}
signals:
void finished();
public slots:
void run();
private:
void finish() { emit finished(); }
private:
const char * m_value;
redisAsyncContext * m_ctx;
RedisQtAdapter m_adapter;
friend
void getCallback(redisAsyncContext *, void *, void *);
};
#endif /* !__HIREDIS_EXAMPLE_QT_H */
-101
View File
@@ -1,101 +0,0 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/redismoduleapi.h>
void debugCallback(redisAsyncContext *c, void *r, void *privdata) {
(void)privdata; //unused
redisReply *reply = r;
if (reply == NULL) {
/* The DEBUG SLEEP command will almost always fail, because we have set a 1 second timeout */
printf("`DEBUG SLEEP` error: %s\n", c->errstr ? c->errstr : "unknown error");
return;
}
/* Disconnect after receiving the reply of DEBUG SLEEP (which will not)*/
redisAsyncDisconnect(c);
}
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) {
if (c->errstr) {
printf("errstr: %s\n", c->errstr);
}
return;
}
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* start another request that demonstrate timeout */
redisAsyncCommand(c, debugCallback, NULL, "DEBUG SLEEP %f", 1.5);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Disconnected...\n");
}
/*
* This example requires Redis 7.0 or above.
*
* 1- Compile this file as a shared library. Directory of "redismodule.h" must
* be in the include path.
* gcc -fPIC -shared -I../../redis/src/ -I.. example-redismoduleapi.c -o example-redismoduleapi.so
*
* 2- Load module:
* redis-server --loadmodule ./example-redismoduleapi.so value
*/
int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
int ret = RedisModule_Init(ctx, "example-redismoduleapi", 1, REDISMODULE_APIVER_1);
if (ret != REDISMODULE_OK) {
printf("error module init \n");
return REDISMODULE_ERR;
}
if (redisModuleCompatibilityCheck() != REDIS_OK) {
printf("Redis 7.0 or above is required! \n");
return REDISMODULE_ERR;
}
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
size_t len;
const char *val = RedisModule_StringPtrLen(argv[argc-1], &len);
RedisModuleCtx *module_ctx = RedisModule_GetDetachedThreadSafeContext(ctx);
redisModuleAttach(c, module_ctx);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncSetTimeout(c, (struct timeval){ .tv_sec = 1, .tv_usec = 0});
/*
In this demo, we first `set key`, then `get key` to demonstrate the basic usage of the adapter.
Then in `getCallback`, we start a `debug sleep` command to create 1.5 second long request.
Because we have set a 1 second timeout to the connection, the command will always fail with a
timeout error, which is shown in the `debugCallback`.
*/
redisAsyncCommand(c, NULL, NULL, "SET key %b", val, len);
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
return 0;
}
-13
View File
@@ -1,13 +0,0 @@
@PACKAGE_INIT@
set_and_check(hiredis_INCLUDEDIR "@PACKAGE_INCLUDE_INSTALL_DIR@")
IF (NOT TARGET hiredis::@hiredis_export_name@)
INCLUDE(${CMAKE_CURRENT_LIST_DIR}/hiredis-targets.cmake)
ENDIF()
SET(hiredis_LIBRARIES hiredis::@hiredis_export_name@)
SET(hiredis_INCLUDE_DIRS ${hiredis_INCLUDEDIR})
check_required_components(hiredis)
-1219
View File
File diff suppressed because it is too large Load Diff
-362
View File
@@ -1,362 +0,0 @@
/*
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
* Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
* Jan-Erik Rediger <janerik at fnordig dot com>
*
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef __HIREDIS_H
#define __HIREDIS_H
#include "read.h"
#include <stdarg.h> /* for va_list */
#ifndef _MSC_VER
#include <sys/time.h> /* for struct timeval */
#else
struct timeval; /* forward declaration */
typedef long long ssize_t;
#endif
#include <stdint.h> /* uintXX_t, etc */
#include "sds.h" /* for hisds */
#include "alloc.h" /* for allocation wrappers */
#define HIREDIS_MAJOR 1
#define HIREDIS_MINOR 2
#define HIREDIS_PATCH 0
#define HIREDIS_SONAME 1.1.0
/* Connection type can be blocking or non-blocking and is set in the
* least significant bit of the flags field in redisContext. */
#define REDIS_BLOCK 0x1
/* Connection may be disconnected before being free'd. The second bit
* in the flags field is set when the context is connected. */
#define REDIS_CONNECTED 0x2
/* The async API might try to disconnect cleanly and flush the output
* buffer and read all subsequent replies before disconnecting.
* This flag means no new commands can come in and the connection
* should be terminated once all replies have been read. */
#define REDIS_DISCONNECTING 0x4
/* Flag specific to the async API which means that the context should be clean
* up as soon as possible. */
#define REDIS_FREEING 0x8
/* Flag that is set when an async callback is executed. */
#define REDIS_IN_CALLBACK 0x10
/* Flag that is set when the async context has one or more subscriptions. */
#define REDIS_SUBSCRIBED 0x20
/* Flag that is set when monitor mode is active */
#define REDIS_MONITORING 0x40
/* Flag that is set when we should set SO_REUSEADDR before calling bind() */
#define REDIS_REUSEADDR 0x80
/* Flag that is set when the async connection supports push replies. */
#define REDIS_SUPPORTS_PUSH 0x100
/**
* Flag that indicates the user does not want the context to
* be automatically freed upon error
*/
#define REDIS_NO_AUTO_FREE 0x200
/* Flag that indicates the user does not want replies to be automatically freed */
#define REDIS_NO_AUTO_FREE_REPLIES 0x400
/* Flags to prefer IPv6 or IPv4 when doing DNS lookup. (If both are set,
* AF_UNSPEC is used.) */
#define REDIS_PREFER_IPV4 0x800
#define REDIS_PREFER_IPV6 0x1000
#define REDIS_KEEPALIVE_INTERVAL 15 /* seconds */
/* number of times we retry to connect in the case of EADDRNOTAVAIL and
* SO_REUSEADDR is being used. */
#define REDIS_CONNECT_RETRIES 10
/* Forward declarations for structs defined elsewhere */
struct redisAsyncContext;
struct redisContext;
/* RESP3 push helpers and callback prototypes */
#define redisIsPushReply(r) (((redisReply*)(r))->type == REDIS_REPLY_PUSH)
typedef void (redisPushFn)(void *, void *);
typedef void (redisAsyncPushFn)(struct redisAsyncContext *, void *);
#ifdef __cplusplus
extern "C" {
#endif
/* This is the reply object returned by redisCommand() */
typedef struct redisReply {
int type; /* REDIS_REPLY_* */
long long integer; /* The integer when type is REDIS_REPLY_INTEGER */
double dval; /* The double when type is REDIS_REPLY_DOUBLE */
size_t len; /* Length of string */
char *str; /* Used for REDIS_REPLY_ERROR, REDIS_REPLY_STRING
REDIS_REPLY_VERB, REDIS_REPLY_DOUBLE (in additional to dval),
and REDIS_REPLY_BIGNUM. */
char vtype[4]; /* Used for REDIS_REPLY_VERB, contains the null
terminated 3 character content type, such as "txt". */
size_t elements; /* number of elements, for REDIS_REPLY_ARRAY */
struct redisReply **element; /* elements vector for REDIS_REPLY_ARRAY */
} redisReply;
redisReader *redisReaderCreate(void);
/* Function to free the reply objects hiredis returns by default. */
void freeReplyObject(void *reply);
/* Functions to format a command according to the protocol. */
int redisvFormatCommand(char **target, const char *format, va_list ap);
int redisFormatCommand(char **target, const char *format, ...);
long long redisFormatCommandArgv(char **target, int argc, const char **argv, const size_t *argvlen);
long long redisFormatSdsCommandArgv(hisds *target, int argc, const char ** argv, const size_t *argvlen);
void redisFreeCommand(char *cmd);
void redisFreeSdsCommand(hisds cmd);
enum redisConnectionType {
REDIS_CONN_TCP,
REDIS_CONN_UNIX,
REDIS_CONN_USERFD
};
struct redisSsl;
#define REDIS_OPT_NONBLOCK 0x01
#define REDIS_OPT_REUSEADDR 0x02
#define REDIS_OPT_NOAUTOFREE 0x04 /* Don't automatically free the async
* object on a connection failure, or
* other implicit conditions. Only free
* on an explicit call to disconnect()
* or free() */
#define REDIS_OPT_NO_PUSH_AUTOFREE 0x08 /* Don't automatically intercept and
* free RESP3 PUSH replies. */
#define REDIS_OPT_NOAUTOFREEREPLIES 0x10 /* Don't automatically free replies. */
#define REDIS_OPT_PREFER_IPV4 0x20 /* Prefer IPv4 in DNS lookups. */
#define REDIS_OPT_PREFER_IPV6 0x40 /* Prefer IPv6 in DNS lookups. */
#define REDIS_OPT_PREFER_IP_UNSPEC (REDIS_OPT_PREFER_IPV4 | REDIS_OPT_PREFER_IPV6)
/* In Unix systems a file descriptor is a regular signed int, with -1
* representing an invalid descriptor. In Windows it is a SOCKET
* (32- or 64-bit unsigned integer depending on the architecture), where
* all bits set (~0) is INVALID_SOCKET. */
#ifndef _WIN32
typedef int redisFD;
#define REDIS_INVALID_FD -1
#else
#ifdef _WIN64
typedef unsigned long long redisFD; /* SOCKET = 64-bit UINT_PTR */
#else
typedef unsigned long redisFD; /* SOCKET = 32-bit UINT_PTR */
#endif
#define REDIS_INVALID_FD ((redisFD)(~0)) /* INVALID_SOCKET */
#endif
typedef struct {
/*
* the type of connection to use. This also indicates which
* `endpoint` member field to use
*/
int type;
/* bit field of REDIS_OPT_xxx */
int options;
/* timeout value for connect operation. If NULL, no timeout is used */
const struct timeval *connect_timeout;
/* timeout value for commands. If NULL, no timeout is used. This can be
* updated at runtime with redisSetTimeout/redisAsyncSetTimeout. */
const struct timeval *command_timeout;
union {
/** use this field for tcp/ip connections */
struct {
const char *source_addr;
const char *ip;
int port;
} tcp;
/** use this field for unix domain sockets */
const char *unix_socket;
/**
* use this field to have hiredis operate an already-open
* file descriptor */
redisFD fd;
} endpoint;
/* Optional user defined data/destructor */
void *privdata;
void (*free_privdata)(void *);
/* A user defined PUSH message callback */
redisPushFn *push_cb;
redisAsyncPushFn *async_push_cb;
} redisOptions;
/**
* Helper macros to initialize options to their specified fields.
*/
#define REDIS_OPTIONS_SET_TCP(opts, ip_, port_) do { \
(opts)->type = REDIS_CONN_TCP; \
(opts)->endpoint.tcp.ip = ip_; \
(opts)->endpoint.tcp.port = port_; \
} while(0)
#define REDIS_OPTIONS_SET_UNIX(opts, path) do { \
(opts)->type = REDIS_CONN_UNIX; \
(opts)->endpoint.unix_socket = path; \
} while(0)
#define REDIS_OPTIONS_SET_PRIVDATA(opts, data, dtor) do { \
(opts)->privdata = data; \
(opts)->free_privdata = dtor; \
} while(0)
typedef struct redisContextFuncs {
void (*close)(struct redisContext *);
void (*free_privctx)(void *);
void (*async_read)(struct redisAsyncContext *);
void (*async_write)(struct redisAsyncContext *);
/* Read/Write data to the underlying communication stream, returning the
* number of bytes read/written. In the event of an unrecoverable error
* these functions shall return a value < 0. In the event of a
* recoverable error, they should return 0. */
ssize_t (*read)(struct redisContext *, char *, size_t);
ssize_t (*write)(struct redisContext *);
} redisContextFuncs;
/* Context for a connection to Redis */
typedef struct redisContext {
const redisContextFuncs *funcs; /* Function table */
int err; /* Error flags, 0 when there is no error */
char errstr[128]; /* String representation of error when applicable */
redisFD fd;
int flags;
char *obuf; /* Write buffer */
redisReader *reader; /* Protocol reader */
enum redisConnectionType connection_type;
struct timeval *connect_timeout;
struct timeval *command_timeout;
struct {
char *host;
char *source_addr;
int port;
} tcp;
struct {
char *path;
} unix_sock;
/* For non-blocking connect */
struct sockaddr *saddr;
size_t addrlen;
/* Optional data and corresponding destructor users can use to provide
* context to a given redisContext. Not used by hiredis. */
void *privdata;
void (*free_privdata)(void *);
/* Internal context pointer presently used by hiredis to manage
* SSL connections. */
void *privctx;
/* An optional RESP3 PUSH handler */
redisPushFn *push_cb;
} redisContext;
redisContext *redisConnectWithOptions(const redisOptions *options);
redisContext *redisConnect(const char *ip, int port);
redisContext *redisConnectWithTimeout(const char *ip, int port, const struct timeval tv);
redisContext *redisConnectNonBlock(const char *ip, int port);
redisContext *redisConnectBindNonBlock(const char *ip, int port,
const char *source_addr);
redisContext *redisConnectBindNonBlockWithReuse(const char *ip, int port,
const char *source_addr);
redisContext *redisConnectUnix(const char *path);
redisContext *redisConnectUnixWithTimeout(const char *path, const struct timeval tv);
redisContext *redisConnectUnixNonBlock(const char *path);
redisContext *redisConnectFd(redisFD fd);
/**
* Reconnect the given context using the saved information.
*
* This re-uses the exact same connect options as in the initial connection.
* host, ip (or path), timeout and bind address are reused,
* flags are used unmodified from the existing context.
*
* Returns REDIS_OK on successful connect or REDIS_ERR otherwise.
*/
int redisReconnect(redisContext *c);
redisPushFn *redisSetPushCallback(redisContext *c, redisPushFn *fn);
int redisSetTimeout(redisContext *c, const struct timeval tv);
int redisEnableKeepAlive(redisContext *c);
int redisEnableKeepAliveWithInterval(redisContext *c, int interval);
int redisSetTcpUserTimeout(redisContext *c, unsigned int timeout);
void redisFree(redisContext *c);
redisFD redisFreeKeepFd(redisContext *c);
int redisBufferRead(redisContext *c);
int redisBufferWrite(redisContext *c, int *done);
/* In a blocking context, this function first checks if there are unconsumed
* replies to return and returns one if so. Otherwise, it flushes the output
* buffer to the socket and reads until it has a reply. In a non-blocking
* context, it will return unconsumed replies until there are no more. */
int redisGetReply(redisContext *c, void **reply);
int redisGetReplyFromReader(redisContext *c, void **reply);
/* Write a formatted command to the output buffer. Use these functions in blocking mode
* to get a pipeline of commands. */
int redisAppendFormattedCommand(redisContext *c, const char *cmd, size_t len);
/* Write a command to the output buffer. Use these functions in blocking mode
* to get a pipeline of commands. */
int redisvAppendCommand(redisContext *c, const char *format, va_list ap);
int redisAppendCommand(redisContext *c, const char *format, ...);
int redisAppendCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);
/* Issue a command to Redis. In a blocking context, it is identical to calling
* redisAppendCommand, followed by redisGetReply. The function will return
* NULL if there was an error in performing the request, otherwise it will
* return the reply. In a non-blocking context, it is identical to calling
* only redisAppendCommand and will always return NULL. */
void *redisvCommand(redisContext *c, const char *format, va_list ap);
void *redisCommand(redisContext *c, const char *format, ...);
void *redisCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);
#ifdef __cplusplus
}
#endif
#endif
-11
View File
@@ -1,11 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemDefinitionGroup>
<ClCompile>
<AdditionalIncludeDirectories>$(MSBuildThisFileDirectory)\..\..\include;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
</ClCompile>
<Link>
<AdditionalLibraryDirectories>$(MSBuildThisFileDirectory)\..\..\lib;%(AdditionalLibraryDirectories)</AdditionalLibraryDirectories>
</Link>
</ItemDefinitionGroup>
</Project>
-16
View File
@@ -1,16 +0,0 @@
@PACKAGE_INIT@
set_and_check(hiredis_ssl_INCLUDEDIR "@PACKAGE_INCLUDE_INSTALL_DIR@")
include(CMakeFindDependencyMacro)
find_dependency(OpenSSL)
IF (NOT TARGET hiredis::hiredis_ssl)
INCLUDE(${CMAKE_CURRENT_LIST_DIR}/hiredis_ssl-targets.cmake)
ENDIF()
SET(hiredis_ssl_LIBRARIES hiredis::hiredis_ssl)
SET(hiredis_ssl_INCLUDE_DIRS ${hiredis_ssl_INCLUDEDIR})
check_required_components(hiredis_ssl)
-672
View File
@@ -1,672 +0,0 @@
/* Extracted from anet.c to work properly with Hiredis error reporting.
*
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
* Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
* Jan-Erik Rediger <janerik at fnordig dot com>
*
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "fmacros.h"
#include <sys/types.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>
#include <stdarg.h>
#include <stdio.h>
#include <limits.h>
#include <stdlib.h>
#include "net.h"
#include "sds.h"
#include "sockcompat.h"
#include "win32.h"
/* Defined in hiredis.c */
void __redisSetError(redisContext *c, int type, const char *str);
int redisContextUpdateCommandTimeout(redisContext *c, const struct timeval *timeout);
void redisNetClose(redisContext *c) {
if (c && c->fd != REDIS_INVALID_FD) {
close(c->fd);
c->fd = REDIS_INVALID_FD;
}
}
ssize_t redisNetRead(redisContext *c, char *buf, size_t bufcap) {
ssize_t nread = recv(c->fd, buf, bufcap, 0);
if (nread == -1) {
if ((errno == EWOULDBLOCK && !(c->flags & REDIS_BLOCK)) || (errno == EINTR)) {
/* Try again later */
return 0;
} else if(errno == ETIMEDOUT && (c->flags & REDIS_BLOCK)) {
/* especially in windows */
__redisSetError(c, REDIS_ERR_TIMEOUT, "recv timeout");
return -1;
} else {
__redisSetError(c, REDIS_ERR_IO, strerror(errno));
return -1;
}
} else if (nread == 0) {
__redisSetError(c, REDIS_ERR_EOF, "Server closed the connection");
return -1;
} else {
return nread;
}
}
ssize_t redisNetWrite(redisContext *c) {
ssize_t nwritten;
nwritten = send(c->fd, c->obuf, hi_sdslen(c->obuf), 0);
if (nwritten < 0) {
if ((errno == EWOULDBLOCK && !(c->flags & REDIS_BLOCK)) || (errno == EINTR)) {
/* Try again */
return 0;
} else {
__redisSetError(c, REDIS_ERR_IO, strerror(errno));
return -1;
}
}
return nwritten;
}
static void __redisSetErrorFromErrno(redisContext *c, int type, const char *prefix) {
int errorno = errno; /* snprintf() may change errno */
char buf[128] = { 0 };
size_t len = 0;
if (prefix != NULL)
len = snprintf(buf,sizeof(buf),"%s: ",prefix);
strerror_r(errorno, (char *)(buf + len), sizeof(buf) - len);
__redisSetError(c,type,buf);
}
static int redisSetReuseAddr(redisContext *c) {
int on = 1;
if (setsockopt(c->fd, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on)) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,NULL);
redisNetClose(c);
return REDIS_ERR;
}
return REDIS_OK;
}
static int redisCreateSocket(redisContext *c, int type) {
redisFD s;
if ((s = socket(type, SOCK_STREAM, 0)) == REDIS_INVALID_FD) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,NULL);
return REDIS_ERR;
}
c->fd = s;
if (type == AF_INET) {
if (redisSetReuseAddr(c) == REDIS_ERR) {
return REDIS_ERR;
}
}
return REDIS_OK;
}
static int redisSetBlocking(redisContext *c, int blocking) {
#ifndef _WIN32
int flags;
/* Set the socket nonblocking.
* Note that fcntl(2) for F_GETFL and F_SETFL can't be
* interrupted by a signal. */
if ((flags = fcntl(c->fd, F_GETFL)) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"fcntl(F_GETFL)");
redisNetClose(c);
return REDIS_ERR;
}
if (blocking)
flags &= ~O_NONBLOCK;
else
flags |= O_NONBLOCK;
if (fcntl(c->fd, F_SETFL, flags) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"fcntl(F_SETFL)");
redisNetClose(c);
return REDIS_ERR;
}
#else
u_long mode = blocking ? 0 : 1;
if (ioctl(c->fd, FIONBIO, &mode) == -1) {
__redisSetErrorFromErrno(c, REDIS_ERR_IO, "ioctl(FIONBIO)");
redisNetClose(c);
return REDIS_ERR;
}
#endif /* _WIN32 */
return REDIS_OK;
}
int redisKeepAlive(redisContext *c, int interval) {
int val = 1;
redisFD fd = c->fd;
#ifndef _WIN32
if (setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &val, sizeof(val)) == -1){
__redisSetError(c,REDIS_ERR_OTHER,strerror(errno));
return REDIS_ERR;
}
val = interval;
#if defined(__APPLE__) && defined(__MACH__)
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPALIVE, &val, sizeof(val)) < 0) {
__redisSetError(c,REDIS_ERR_OTHER,strerror(errno));
return REDIS_ERR;
}
#else
#if defined(__GLIBC__) && !defined(__FreeBSD_kernel__)
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE, &val, sizeof(val)) < 0) {
__redisSetError(c,REDIS_ERR_OTHER,strerror(errno));
return REDIS_ERR;
}
val = interval/3;
if (val == 0) val = 1;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPINTVL, &val, sizeof(val)) < 0) {
__redisSetError(c,REDIS_ERR_OTHER,strerror(errno));
return REDIS_ERR;
}
val = 3;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPCNT, &val, sizeof(val)) < 0) {
__redisSetError(c,REDIS_ERR_OTHER,strerror(errno));
return REDIS_ERR;
}
#endif
#endif
#else
int res;
res = win32_redisKeepAlive(fd, interval * 1000);
if (res != 0) {
__redisSetError(c, REDIS_ERR_OTHER, strerror(res));
return REDIS_ERR;
}
#endif
return REDIS_OK;
}
int redisSetTcpNoDelay(redisContext *c) {
int yes = 1;
if (setsockopt(c->fd, IPPROTO_TCP, TCP_NODELAY, &yes, sizeof(yes)) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"setsockopt(TCP_NODELAY)");
redisNetClose(c);
return REDIS_ERR;
}
return REDIS_OK;
}
int redisContextSetTcpUserTimeout(redisContext *c, unsigned int timeout) {
int res;
#ifdef TCP_USER_TIMEOUT
res = setsockopt(c->fd, IPPROTO_TCP, TCP_USER_TIMEOUT, &timeout, sizeof(timeout));
#else
res = -1;
errno = ENOTSUP;
(void)timeout;
#endif
if (res == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"setsockopt(TCP_USER_TIMEOUT)");
redisNetClose(c);
return REDIS_ERR;
}
return REDIS_OK;
}
#define __MAX_MSEC (((LONG_MAX) - 999) / 1000)
static int redisContextTimeoutMsec(redisContext *c, long *result)
{
const struct timeval *timeout = c->connect_timeout;
long msec = -1;
/* Only use timeout when not NULL. */
if (timeout != NULL) {
if (timeout->tv_usec > 1000000 || timeout->tv_sec > __MAX_MSEC) {
__redisSetError(c, REDIS_ERR_IO, "Invalid timeout specified");
*result = msec;
return REDIS_ERR;
}
msec = (timeout->tv_sec * 1000) + ((timeout->tv_usec + 999) / 1000);
if (msec < 0 || msec > INT_MAX) {
msec = INT_MAX;
}
}
*result = msec;
return REDIS_OK;
}
static int redisContextWaitReady(redisContext *c, long msec) {
struct pollfd wfd[1];
wfd[0].fd = c->fd;
wfd[0].events = POLLOUT;
if (errno == EINPROGRESS) {
int res;
if ((res = poll(wfd, 1, msec)) == -1) {
__redisSetErrorFromErrno(c, REDIS_ERR_IO, "poll(2)");
redisNetClose(c);
return REDIS_ERR;
} else if (res == 0) {
errno = ETIMEDOUT;
__redisSetErrorFromErrno(c,REDIS_ERR_IO,NULL);
redisNetClose(c);
return REDIS_ERR;
}
if (redisCheckConnectDone(c, &res) != REDIS_OK || res == 0) {
redisCheckSocketError(c);
return REDIS_ERR;
}
return REDIS_OK;
}
__redisSetErrorFromErrno(c,REDIS_ERR_IO,NULL);
redisNetClose(c);
return REDIS_ERR;
}
int redisCheckConnectDone(redisContext *c, int *completed) {
int rc = connect(c->fd, (const struct sockaddr *)c->saddr, c->addrlen);
if (rc == 0) {
*completed = 1;
return REDIS_OK;
}
int error = errno;
if (error == EINPROGRESS) {
/* must check error to see if connect failed. Get the socket error */
int fail, so_error;
socklen_t optlen = sizeof(so_error);
fail = getsockopt(c->fd, SOL_SOCKET, SO_ERROR, &so_error, &optlen);
if (fail == 0) {
if (so_error == 0) {
/* Socket is connected! */
*completed = 1;
return REDIS_OK;
}
/* connection error; */
errno = so_error;
error = so_error;
}
}
switch (error) {
case EISCONN:
*completed = 1;
return REDIS_OK;
case EALREADY:
case EWOULDBLOCK:
*completed = 0;
return REDIS_OK;
default:
return REDIS_ERR;
}
}
int redisCheckSocketError(redisContext *c) {
int err = 0, errno_saved = errno;
socklen_t errlen = sizeof(err);
if (getsockopt(c->fd, SOL_SOCKET, SO_ERROR, &err, &errlen) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"getsockopt(SO_ERROR)");
return REDIS_ERR;
}
if (err == 0) {
err = errno_saved;
}
if (err) {
errno = err;
__redisSetErrorFromErrno(c,REDIS_ERR_IO,NULL);
return REDIS_ERR;
}
return REDIS_OK;
}
int redisContextSetTimeout(redisContext *c, const struct timeval tv) {
const void *to_ptr = &tv;
size_t to_sz = sizeof(tv);
if (redisContextUpdateCommandTimeout(c, &tv) != REDIS_OK) {
__redisSetError(c, REDIS_ERR_OOM, "Out of memory");
return REDIS_ERR;
}
if (setsockopt(c->fd,SOL_SOCKET,SO_RCVTIMEO,to_ptr,to_sz) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"setsockopt(SO_RCVTIMEO)");
return REDIS_ERR;
}
if (setsockopt(c->fd,SOL_SOCKET,SO_SNDTIMEO,to_ptr,to_sz) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"setsockopt(SO_SNDTIMEO)");
return REDIS_ERR;
}
return REDIS_OK;
}
int redisContextUpdateConnectTimeout(redisContext *c, const struct timeval *timeout) {
/* Same timeval struct, short circuit */
if (c->connect_timeout == timeout)
return REDIS_OK;
/* Allocate context timeval if we need to */
if (c->connect_timeout == NULL) {
c->connect_timeout = hi_malloc(sizeof(*c->connect_timeout));
if (c->connect_timeout == NULL)
return REDIS_ERR;
}
memcpy(c->connect_timeout, timeout, sizeof(*c->connect_timeout));
return REDIS_OK;
}
int redisContextUpdateCommandTimeout(redisContext *c, const struct timeval *timeout) {
/* Same timeval struct, short circuit */
if (c->command_timeout == timeout)
return REDIS_OK;
/* Allocate context timeval if we need to */
if (c->command_timeout == NULL) {
c->command_timeout = hi_malloc(sizeof(*c->command_timeout));
if (c->command_timeout == NULL)
return REDIS_ERR;
}
memcpy(c->command_timeout, timeout, sizeof(*c->command_timeout));
return REDIS_OK;
}
static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
const struct timeval *timeout,
const char *source_addr) {
redisFD s;
int rv, n;
char _port[6]; /* strlen("65535"); */
struct addrinfo hints, *servinfo, *bservinfo, *p, *b;
int blocking = (c->flags & REDIS_BLOCK);
int reuseaddr = (c->flags & REDIS_REUSEADDR);
int reuses = 0;
long timeout_msec = -1;
servinfo = NULL;
c->connection_type = REDIS_CONN_TCP;
c->tcp.port = port;
/* We need to take possession of the passed parameters
* to make them reusable for a reconnect.
* We also carefully check we don't free data we already own,
* as in the case of the reconnect method.
*
* This is a bit ugly, but atleast it works and doesn't leak memory.
**/
if (c->tcp.host != addr) {
hi_free(c->tcp.host);
c->tcp.host = hi_strdup(addr);
if (c->tcp.host == NULL)
goto oom;
}
if (timeout) {
if (redisContextUpdateConnectTimeout(c, timeout) == REDIS_ERR)
goto oom;
} else {
hi_free(c->connect_timeout);
c->connect_timeout = NULL;
}
if (redisContextTimeoutMsec(c, &timeout_msec) != REDIS_OK) {
goto error;
}
if (source_addr == NULL) {
hi_free(c->tcp.source_addr);
c->tcp.source_addr = NULL;
} else if (c->tcp.source_addr != source_addr) {
hi_free(c->tcp.source_addr);
c->tcp.source_addr = hi_strdup(source_addr);
}
snprintf(_port, 6, "%d", port);
memset(&hints,0,sizeof(hints));
hints.ai_family = AF_INET;
hints.ai_socktype = SOCK_STREAM;
/* DNS lookup. To use dual stack, set both flags to prefer both IPv4 and
* IPv6. By default, for historical reasons, we try IPv4 first and then we
* try IPv6 only if no IPv4 address was found. */
if (c->flags & REDIS_PREFER_IPV6 && c->flags & REDIS_PREFER_IPV4)
hints.ai_family = AF_UNSPEC;
else if (c->flags & REDIS_PREFER_IPV6)
hints.ai_family = AF_INET6;
else
hints.ai_family = AF_INET;
rv = getaddrinfo(c->tcp.host, _port, &hints, &servinfo);
if (rv != 0 && hints.ai_family != AF_UNSPEC) {
/* Try again with the other IP version. */
hints.ai_family = (hints.ai_family == AF_INET) ? AF_INET6 : AF_INET;
rv = getaddrinfo(c->tcp.host, _port, &hints, &servinfo);
}
if (rv != 0) {
__redisSetError(c, REDIS_ERR_OTHER, gai_strerror(rv));
return REDIS_ERR;
}
for (p = servinfo; p != NULL; p = p->ai_next) {
addrretry:
if ((s = socket(p->ai_family,p->ai_socktype,p->ai_protocol)) == REDIS_INVALID_FD)
continue;
c->fd = s;
if (redisSetBlocking(c,0) != REDIS_OK)
goto error;
if (c->tcp.source_addr) {
int bound = 0;
/* Using getaddrinfo saves us from self-determining IPv4 vs IPv6 */
if ((rv = getaddrinfo(c->tcp.source_addr, NULL, &hints, &bservinfo)) != 0) {
char buf[128];
snprintf(buf,sizeof(buf),"Can't get addr: %s",gai_strerror(rv));
__redisSetError(c,REDIS_ERR_OTHER,buf);
goto error;
}
if (reuseaddr) {
n = 1;
if (setsockopt(s, SOL_SOCKET, SO_REUSEADDR, (char*) &n,
sizeof(n)) < 0) {
freeaddrinfo(bservinfo);
goto error;
}
}
for (b = bservinfo; b != NULL; b = b->ai_next) {
if (bind(s,b->ai_addr,b->ai_addrlen) != -1) {
bound = 1;
break;
}
}
freeaddrinfo(bservinfo);
if (!bound) {
char buf[128];
snprintf(buf,sizeof(buf),"Can't bind socket: %s",strerror(errno));
__redisSetError(c,REDIS_ERR_OTHER,buf);
goto error;
}
}
/* For repeat connection */
hi_free(c->saddr);
c->saddr = hi_malloc(p->ai_addrlen);
if (c->saddr == NULL)
goto oom;
memcpy(c->saddr, p->ai_addr, p->ai_addrlen);
c->addrlen = p->ai_addrlen;
if (connect(s,p->ai_addr,p->ai_addrlen) == -1) {
if (errno == EHOSTUNREACH) {
redisNetClose(c);
continue;
} else if (errno == EINPROGRESS) {
if (blocking) {
goto wait_for_ready;
}
/* This is ok.
* Note that even when it's in blocking mode, we unset blocking
* for `connect()`
*/
} else if (errno == EADDRNOTAVAIL && reuseaddr) {
if (++reuses >= REDIS_CONNECT_RETRIES) {
goto error;
} else {
redisNetClose(c);
goto addrretry;
}
} else {
wait_for_ready:
if (redisContextWaitReady(c,timeout_msec) != REDIS_OK)
goto error;
if (redisSetTcpNoDelay(c) != REDIS_OK)
goto error;
}
}
if (blocking && redisSetBlocking(c,1) != REDIS_OK)
goto error;
c->flags |= REDIS_CONNECTED;
rv = REDIS_OK;
goto end;
}
if (p == NULL) {
char buf[128];
snprintf(buf,sizeof(buf),"Can't create socket: %s",strerror(errno));
__redisSetError(c,REDIS_ERR_OTHER,buf);
goto error;
}
oom:
__redisSetError(c, REDIS_ERR_OOM, "Out of memory");
error:
rv = REDIS_ERR;
end:
if(servinfo) {
freeaddrinfo(servinfo);
}
return rv; // Need to return REDIS_OK if alright
}
int redisContextConnectTcp(redisContext *c, const char *addr, int port,
const struct timeval *timeout) {
return _redisContextConnectTcp(c, addr, port, timeout, NULL);
}
int redisContextConnectBindTcp(redisContext *c, const char *addr, int port,
const struct timeval *timeout,
const char *source_addr) {
return _redisContextConnectTcp(c, addr, port, timeout, source_addr);
}
int redisContextConnectUnix(redisContext *c, const char *path, const struct timeval *timeout) {
#ifndef _WIN32
int blocking = (c->flags & REDIS_BLOCK);
struct sockaddr_un *sa;
long timeout_msec = -1;
if (redisCreateSocket(c,AF_UNIX) < 0)
return REDIS_ERR;
if (redisSetBlocking(c,0) != REDIS_OK)
return REDIS_ERR;
c->connection_type = REDIS_CONN_UNIX;
if (c->unix_sock.path != path) {
hi_free(c->unix_sock.path);
c->unix_sock.path = hi_strdup(path);
if (c->unix_sock.path == NULL)
goto oom;
}
if (timeout) {
if (redisContextUpdateConnectTimeout(c, timeout) == REDIS_ERR)
goto oom;
} else {
hi_free(c->connect_timeout);
c->connect_timeout = NULL;
}
if (redisContextTimeoutMsec(c,&timeout_msec) != REDIS_OK)
return REDIS_ERR;
/* Don't leak sockaddr if we're reconnecting */
if (c->saddr) hi_free(c->saddr);
sa = (struct sockaddr_un*)(c->saddr = hi_malloc(sizeof(struct sockaddr_un)));
if (sa == NULL)
goto oom;
c->addrlen = sizeof(struct sockaddr_un);
sa->sun_family = AF_UNIX;
strncpy(sa->sun_path, path, sizeof(sa->sun_path) - 1);
if (connect(c->fd, (struct sockaddr*)sa, sizeof(*sa)) == -1) {
if (errno == EINPROGRESS && !blocking) {
/* This is ok. */
} else {
if (redisContextWaitReady(c,timeout_msec) != REDIS_OK)
return REDIS_ERR;
}
}
/* Reset socket to be blocking after connect(2). */
if (blocking && redisSetBlocking(c,1) != REDIS_OK)
return REDIS_ERR;
c->flags |= REDIS_CONNECTED;
return REDIS_OK;
#else
/* We currently do not support Unix sockets for Windows. */
/* TODO(m): https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/ */
errno = EPROTONOSUPPORT;
return REDIS_ERR;
#endif /* _WIN32 */
oom:
__redisSetError(c, REDIS_ERR_OOM, "Out of memory");
return REDIS_ERR;
}
-129
View File
@@ -1,129 +0,0 @@
/*
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef __HIREDIS_READ_H
#define __HIREDIS_READ_H
#include <stdio.h> /* for size_t */
#define REDIS_ERR -1
#define REDIS_OK 0
/* When an error occurs, the err flag in a context is set to hold the type of
* error that occurred. REDIS_ERR_IO means there was an I/O error and you
* should use the "errno" variable to find out what is wrong.
* For other values, the "errstr" field will hold a description. */
#define REDIS_ERR_IO 1 /* Error in read or write */
#define REDIS_ERR_EOF 3 /* End of file */
#define REDIS_ERR_PROTOCOL 4 /* Protocol error */
#define REDIS_ERR_OOM 5 /* Out of memory */
#define REDIS_ERR_TIMEOUT 6 /* Timed out */
#define REDIS_ERR_OTHER 2 /* Everything else... */
#define REDIS_REPLY_STRING 1
#define REDIS_REPLY_ARRAY 2
#define REDIS_REPLY_INTEGER 3
#define REDIS_REPLY_NIL 4
#define REDIS_REPLY_STATUS 5
#define REDIS_REPLY_ERROR 6
#define REDIS_REPLY_DOUBLE 7
#define REDIS_REPLY_BOOL 8
#define REDIS_REPLY_MAP 9
#define REDIS_REPLY_SET 10
#define REDIS_REPLY_ATTR 11
#define REDIS_REPLY_PUSH 12
#define REDIS_REPLY_BIGNUM 13
#define REDIS_REPLY_VERB 14
/* Default max unused reader buffer. */
#define REDIS_READER_MAX_BUF (1024*16)
/* Default multi-bulk element limit */
#define REDIS_READER_MAX_ARRAY_ELEMENTS ((1LL<<32) - 1)
#ifdef __cplusplus
extern "C" {
#endif
typedef struct redisReadTask {
int type;
long long elements; /* number of elements in multibulk container */
int idx; /* index in parent (array) object */
void *obj; /* holds user-generated value for a read task */
struct redisReadTask *parent; /* parent task */
void *privdata; /* user-settable arbitrary field */
} redisReadTask;
typedef struct redisReplyObjectFunctions {
void *(*createString)(const redisReadTask*, char*, size_t);
void *(*createArray)(const redisReadTask*, size_t);
void *(*createInteger)(const redisReadTask*, long long);
void *(*createDouble)(const redisReadTask*, double, char*, size_t);
void *(*createNil)(const redisReadTask*);
void *(*createBool)(const redisReadTask*, int);
void (*freeObject)(void*);
} redisReplyObjectFunctions;
typedef struct redisReader {
int err; /* Error flags, 0 when there is no error */
char errstr[128]; /* String representation of error when applicable */
char *buf; /* Read buffer */
size_t pos; /* Buffer cursor */
size_t len; /* Buffer length */
size_t maxbuf; /* Max length of unused buffer */
long long maxelements; /* Max multi-bulk elements */
redisReadTask **task;
int tasks;
int ridx; /* Index of current read task */
void *reply; /* Temporary reply pointer */
redisReplyObjectFunctions *fn;
void *privdata;
} redisReader;
/* Public API for the protocol parser. */
redisReader *redisReaderCreateWithFunctions(redisReplyObjectFunctions *fn);
void redisReaderFree(redisReader *r);
int redisReaderFeed(redisReader *r, const char *buf, size_t len);
int redisReaderGetReply(redisReader *r, void **reply);
#define redisReaderSetPrivdata(_r, _p) (int)(((redisReader*)(_r))->privdata = (_p))
#define redisReaderGetObject(_r) (((redisReader*)(_r))->reply)
#define redisReaderGetError(_r) (((redisReader*)(_r))->errstr)
#ifdef __cplusplus
}
#endif
#endif
-1291
View File
File diff suppressed because it is too large Load Diff
-279
View File
@@ -1,279 +0,0 @@
/* SDSLib 2.0 -- A C dynamic strings library
*
* Copyright (c) 2006-2015, Redis Ltd.
* Copyright (c) 2015, Oran Agra
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#ifndef HIREDIS_SDS_H
#define HIREDIS_SDS_H
#define HI_SDS_MAX_PREALLOC (1024*1024)
#ifdef _MSC_VER
typedef long long ssize_t;
#define SSIZE_MAX (LLONG_MAX >> 1)
#ifndef __clang__
#define __attribute__(x)
#endif
#endif
#include <sys/types.h>
#include <stdarg.h>
#include <stdint.h>
typedef char *hisds;
/* Note: sdshdr5 is never used, we just access the flags byte directly.
* However is here to document the layout of type 5 SDS strings. */
struct __attribute__ ((__packed__)) hisdshdr5 {
unsigned char flags; /* 3 lsb of type, and 5 msb of string length */
char buf[];
};
struct __attribute__ ((__packed__)) hisdshdr8 {
uint8_t len; /* used */
uint8_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
char buf[];
};
struct __attribute__ ((__packed__)) hisdshdr16 {
uint16_t len; /* used */
uint16_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
char buf[];
};
struct __attribute__ ((__packed__)) hisdshdr32 {
uint32_t len; /* used */
uint32_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
char buf[];
};
struct __attribute__ ((__packed__)) hisdshdr64 {
uint64_t len; /* used */
uint64_t alloc; /* excluding the header and null terminator */
unsigned char flags; /* 3 lsb of type, 5 unused bits */
char buf[];
};
#define HI_SDS_TYPE_5 0
#define HI_SDS_TYPE_8 1
#define HI_SDS_TYPE_16 2
#define HI_SDS_TYPE_32 3
#define HI_SDS_TYPE_64 4
#define HI_SDS_TYPE_MASK 7
#define HI_SDS_TYPE_BITS 3
#define HI_SDS_HDR_VAR(T,s) struct hisdshdr##T *sh = (struct hisdshdr##T *)((s)-(sizeof(struct hisdshdr##T)));
#define HI_SDS_HDR(T,s) ((struct hisdshdr##T *)((s)-(sizeof(struct hisdshdr##T))))
#define HI_SDS_TYPE_5_LEN(f) ((f)>>HI_SDS_TYPE_BITS)
static inline size_t hi_sdslen(const hisds s) {
unsigned char flags = s[-1];
switch(flags & HI_SDS_TYPE_MASK) {
case HI_SDS_TYPE_5:
return HI_SDS_TYPE_5_LEN(flags);
case HI_SDS_TYPE_8:
return HI_SDS_HDR(8,s)->len;
case HI_SDS_TYPE_16:
return HI_SDS_HDR(16,s)->len;
case HI_SDS_TYPE_32:
return HI_SDS_HDR(32,s)->len;
case HI_SDS_TYPE_64:
return HI_SDS_HDR(64,s)->len;
}
return 0;
}
static inline size_t hi_sdsavail(const hisds s) {
unsigned char flags = s[-1];
switch(flags&HI_SDS_TYPE_MASK) {
case HI_SDS_TYPE_5: {
return 0;
}
case HI_SDS_TYPE_8: {
HI_SDS_HDR_VAR(8,s);
return sh->alloc - sh->len;
}
case HI_SDS_TYPE_16: {
HI_SDS_HDR_VAR(16,s);
return sh->alloc - sh->len;
}
case HI_SDS_TYPE_32: {
HI_SDS_HDR_VAR(32,s);
return sh->alloc - sh->len;
}
case HI_SDS_TYPE_64: {
HI_SDS_HDR_VAR(64,s);
return sh->alloc - sh->len;
}
}
return 0;
}
static inline void hi_sdssetlen(hisds s, size_t newlen) {
unsigned char flags = s[-1];
switch(flags&HI_SDS_TYPE_MASK) {
case HI_SDS_TYPE_5:
{
unsigned char *fp = ((unsigned char*)s)-1;
*fp = (unsigned char)(HI_SDS_TYPE_5 | (newlen << HI_SDS_TYPE_BITS));
}
break;
case HI_SDS_TYPE_8:
HI_SDS_HDR(8,s)->len = (uint8_t)newlen;
break;
case HI_SDS_TYPE_16:
HI_SDS_HDR(16,s)->len = (uint16_t)newlen;
break;
case HI_SDS_TYPE_32:
HI_SDS_HDR(32,s)->len = (uint32_t)newlen;
break;
case HI_SDS_TYPE_64:
HI_SDS_HDR(64,s)->len = (uint64_t)newlen;
break;
}
}
static inline void hi_sdsinclen(hisds s, size_t inc) {
unsigned char flags = s[-1];
switch(flags&HI_SDS_TYPE_MASK) {
case HI_SDS_TYPE_5:
{
unsigned char *fp = ((unsigned char*)s)-1;
unsigned char newlen = HI_SDS_TYPE_5_LEN(flags)+(unsigned char)inc;
*fp = HI_SDS_TYPE_5 | (newlen << HI_SDS_TYPE_BITS);
}
break;
case HI_SDS_TYPE_8:
HI_SDS_HDR(8,s)->len += (uint8_t)inc;
break;
case HI_SDS_TYPE_16:
HI_SDS_HDR(16,s)->len += (uint16_t)inc;
break;
case HI_SDS_TYPE_32:
HI_SDS_HDR(32,s)->len += (uint32_t)inc;
break;
case HI_SDS_TYPE_64:
HI_SDS_HDR(64,s)->len += (uint64_t)inc;
break;
}
}
/* hi_sdsalloc() = hi_sdsavail() + hi_sdslen() */
static inline size_t hi_sdsalloc(const hisds s) {
unsigned char flags = s[-1];
switch(flags & HI_SDS_TYPE_MASK) {
case HI_SDS_TYPE_5:
return HI_SDS_TYPE_5_LEN(flags);
case HI_SDS_TYPE_8:
return HI_SDS_HDR(8,s)->alloc;
case HI_SDS_TYPE_16:
return HI_SDS_HDR(16,s)->alloc;
case HI_SDS_TYPE_32:
return HI_SDS_HDR(32,s)->alloc;
case HI_SDS_TYPE_64:
return HI_SDS_HDR(64,s)->alloc;
}
return 0;
}
static inline void hi_sdssetalloc(hisds s, size_t newlen) {
unsigned char flags = s[-1];
switch(flags&HI_SDS_TYPE_MASK) {
case HI_SDS_TYPE_5:
/* Nothing to do, this type has no total allocation info. */
break;
case HI_SDS_TYPE_8:
HI_SDS_HDR(8,s)->alloc = (uint8_t)newlen;
break;
case HI_SDS_TYPE_16:
HI_SDS_HDR(16,s)->alloc = (uint16_t)newlen;
break;
case HI_SDS_TYPE_32:
HI_SDS_HDR(32,s)->alloc = (uint32_t)newlen;
break;
case HI_SDS_TYPE_64:
HI_SDS_HDR(64,s)->alloc = (uint64_t)newlen;
break;
}
}
hisds hi_sdsnewlen(const void *init, size_t initlen);
hisds hi_sdsnew(const char *init);
hisds hi_sdsempty(void);
hisds hi_sdsdup(const hisds s);
void hi_sdsfree(hisds s);
hisds hi_sdsgrowzero(hisds s, size_t len);
hisds hi_sdscatlen(hisds s, const void *t, size_t len);
hisds hi_sdscat(hisds s, const char *t);
hisds hi_sdscatsds(hisds s, const hisds t);
hisds hi_sdscpylen(hisds s, const char *t, size_t len);
hisds hi_sdscpy(hisds s, const char *t);
hisds hi_sdscatvprintf(hisds s, const char *fmt, va_list ap);
#ifdef __GNUC__
hisds hi_sdscatprintf(hisds s, const char *fmt, ...)
__attribute__((format(printf, 2, 3)));
#else
hisds hi_sdscatprintf(hisds s, const char *fmt, ...);
#endif
hisds hi_sdscatfmt(hisds s, char const *fmt, ...);
hisds hi_sdstrim(hisds s, const char *cset);
int hi_sdsrange(hisds s, ssize_t start, ssize_t end);
void hi_sdsupdatelen(hisds s);
void hi_sdsclear(hisds s);
int hi_sdscmp(const hisds s1, const hisds s2);
hisds *hi_sdssplitlen(const char *s, int len, const char *sep, int seplen, int *count);
void hi_sdsfreesplitres(hisds *tokens, int count);
void hi_sdstolower(hisds s);
void hi_sdstoupper(hisds s);
hisds hi_sdsfromlonglong(long long value);
hisds hi_sdscatrepr(hisds s, const char *p, size_t len);
hisds *hi_sdssplitargs(const char *line, int *argc);
hisds hi_sdsmapchars(hisds s, const char *from, const char *to, size_t setlen);
hisds hi_sdsjoin(char **argv, int argc, char *sep);
hisds hi_sdsjoinsds(hisds *argv, int argc, const char *sep, size_t seplen);
/* Low level functions exposed to the user API */
hisds hi_sdsMakeRoomFor(hisds s, size_t addlen);
void hi_sdsIncrLen(hisds s, int incr);
hisds hi_sdsRemoveFreeSpace(hisds s);
size_t hi_sdsAllocSize(hisds s);
void *hi_sdsAllocPtr(hisds s);
/* Export the allocator used by SDS to the program using SDS.
* Sometimes the program SDS is linked to, may use a different set of
* allocators, but may want to allocate or free things that SDS will
* respectively free or allocate. */
void *hi_sds_malloc(size_t size);
void *hi_sds_realloc(void *ptr, size_t size);
void hi_sds_free(void *ptr);
#ifdef SERVER_TEST
int hi_sdsTest(int argc, char *argv[]);
#endif
#endif /* HIREDIS_SDS_H */

Some files were not shown because too many files have changed in this diff Show More