hanzo/kv
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: hanzo/kv:dev
Branches Tags
hanzo/kv:main hanzo/kv:chore/platform-native-ci hanzo/kv:rescue/head-e003ec2907 hanzo/kv:upstream-sync/2026-05-19-9.0.4 hanzo/kv:rebrand-topology-fix hanzo/kv:preserve/spark-local hanzo/kv:rescue/main-local hanzo/kv:ci/canonical-docker-build-1776995843 hanzo/kv:ci/canonical-docker-build hanzo/kv:dev hanzo/kv:test hanzo/kv:dependabot/github_actions/actions/checkout-6 hanzo/kv:dependabot/github_actions/docker/build-push-action-6 hanzo/kv:dependabot/github_actions/taiki-e/install-action-2.68.1 hanzo/kv:dependabot/github_actions/cross-platform-actions/action-492b0c80085400348c599edace11141a4ee73524 hanzo/kv:unstable
hanzo/kv:9.1 hanzo/kv:9.1.0 hanzo/kv:v1.0.2 hanzo/kv:v1.0.1 hanzo/kv:8.0.9 hanzo/kv:9.0.4 hanzo/kv:v9.0.4 hanzo/kv:8.1.7 hanzo/kv:7.2.13 hanzo/kv:8.0.8 hanzo/kv:9.1.0-rc2 hanzo/kv:9.1.0-rc1 hanzo/kv:9.0.3 hanzo/kv:8.1.6 hanzo/kv:8.0.7 hanzo/kv:7.2.12 hanzo/kv:v8.1.1 hanzo/kv:v8.1.0 hanzo/kv:v1.0.0 hanzo/kv:9.0.2 hanzo/kv:9.0.1 hanzo/kv:8.1.5 hanzo/kv:9.0.0 hanzo/kv:9.0.0-rc3 hanzo/kv:8.0.6 hanzo/kv:8.1.4 hanzo/kv:7.2.11 hanzo/kv:9.0.0-rc2 hanzo/kv:8.0.5 hanzo/kv:9.0.0-rc1 hanzo/kv:7.2.10 hanzo/kv:8.0.4 hanzo/kv:8.1.3 hanzo/kv:8.1.2 hanzo/kv:8.0.3 hanzo/kv:8.1.1 hanzo/kv:7.2.9 hanzo/kv:8.1.0 hanzo/kv:8.1.0-rc2 hanzo/kv:8.1.0-rc1 hanzo/kv:7.2.8 hanzo/kv:8.0.2 hanzo/kv:8.0.1 hanzo/kv:7.2.7 hanzo/kv:8.0.0 hanzo/kv:8.0.0-rc2 hanzo/kv:8.0.0-rc1 hanzo/kv:7.2.6 hanzo/kv:7.2.5 hanzo/kv:7.2.5-rc1 hanzo/kv:7.2.4-rc1 hanzo/kv:7.2.4
...
compare: hanzo/kv:9.1.0-rc2
Branches Tags
hanzo/kv:main hanzo/kv:chore/platform-native-ci hanzo/kv:rescue/head-e003ec2907 hanzo/kv:upstream-sync/2026-05-19-9.0.4 hanzo/kv:rebrand-topology-fix hanzo/kv:preserve/spark-local hanzo/kv:rescue/main-local hanzo/kv:ci/canonical-docker-build-1776995843 hanzo/kv:ci/canonical-docker-build hanzo/kv:dev hanzo/kv:test hanzo/kv:dependabot/github_actions/actions/checkout-6 hanzo/kv:dependabot/github_actions/docker/build-push-action-6 hanzo/kv:dependabot/github_actions/taiki-e/install-action-2.68.1 hanzo/kv:dependabot/github_actions/cross-platform-actions/action-492b0c80085400348c599edace11141a4ee73524 hanzo/kv:unstable
hanzo/kv:9.1 hanzo/kv:9.1.0 hanzo/kv:v1.0.2 hanzo/kv:v1.0.1 hanzo/kv:8.0.9 hanzo/kv:9.0.4 hanzo/kv:v9.0.4 hanzo/kv:8.1.7 hanzo/kv:7.2.13 hanzo/kv:8.0.8 hanzo/kv:9.1.0-rc2 hanzo/kv:9.1.0-rc1 hanzo/kv:9.0.3 hanzo/kv:8.1.6 hanzo/kv:8.0.7 hanzo/kv:7.2.12 hanzo/kv:v8.1.1 hanzo/kv:v8.1.0 hanzo/kv:v1.0.0 hanzo/kv:9.0.2 hanzo/kv:9.0.1 hanzo/kv:8.1.5 hanzo/kv:9.0.0 hanzo/kv:9.0.0-rc3 hanzo/kv:8.0.6 hanzo/kv:8.1.4 hanzo/kv:7.2.11 hanzo/kv:9.0.0-rc2 hanzo/kv:8.0.5 hanzo/kv:9.0.0-rc1 hanzo/kv:7.2.10 hanzo/kv:8.0.4 hanzo/kv:8.1.3 hanzo/kv:8.1.2 hanzo/kv:8.0.3 hanzo/kv:8.1.1 hanzo/kv:7.2.9 hanzo/kv:8.1.0 hanzo/kv:8.1.0-rc2 hanzo/kv:8.1.0-rc1 hanzo/kv:7.2.8 hanzo/kv:8.0.2 hanzo/kv:8.0.1 hanzo/kv:7.2.7 hanzo/kv:8.0.0 hanzo/kv:8.0.0-rc2 hanzo/kv:8.0.0-rc1 hanzo/kv:7.2.6 hanzo/kv:7.2.5 hanzo/kv:7.2.5-rc1 hanzo/kv:7.2.4-rc1 hanzo/kv:7.2.4
136 Commits
dev ... 9.1.0-rc2
Author SHA1 Message Date
Sarthak AggarwalGitHubMadelyn Olson
 		81bed3ec00 Update version to 9.1.0-rc2 and add release notes (#3521) 
Bump `VALKEY_RELEASE_STAGE` from `rc1` to `rc2` and add release notes
for changes backported from `unstable` in #3519

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthakaggarwal97@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
 2026-04-28 09:32:37 -07:00
Sarthak AggarwalandGitHub 80022b4c2c Revert "Fail fast on invalid certificates at TLS config load (#2999)" (#3572) 
This reverts commit dafa73164a.

As discussed in the TSC Core Meeting, this PR reverts #2999 from Valkey
9.1. We plan to do in next major version as it's a breaking change.

**Note**: Merge conflicts in `src/tls.c` includes were resolved due to
overlapping changes from PR #2913, which added
`tlsUpdateCertInfoFromDir()` with shared header dependencies.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
 2026-04-27 13:36:03 -07:00
eifrah-awsandMadelyn Olson a3e1fe0cd8 Fix remove cached eval scripts on engine unregister (#3503) 
Remove eval script cache entries that belong to a scripting engine when
that engine is unregistered. This prevents the eval cache from retaining
dangling engine pointers and keeps the tracked script memory in sync
after engine shutdown.

The scripting engine unregister path now invokes a new eval cleanup
helper, which scans the cached scripts, drops matching entries from the
LRU list and dictionary, and adjusts cache memory accounting accordingly.

* scripting engine
* eval cache

Signed-off-by: Eran Ifrah <eifrah@amazon.com>
 2026-04-27 13:35:11 -07:00
Jacob MurphyandMadelyn Olson 01ee9f276d Ensure client slot migration pointer is cleared during reset (#3554) 
If not cleared, the job may no longer be valid by the time the client
goes to cleanup. This dangling reference could cause a crash if you set
slot-migration-log-max-len to 0 and are very unlucky.

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
 2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 754e908e14 Fix lua-enable-insecure-api default value cannot be changed to yes (#3548) 
The default value of lua-enable-insecure-api cannot be safely changed
from no to yes due to two issues:

1. In createEngineContext(), lua_enable_insecure_api was hardcoded to 0
before initializing Lua states, so deprecated APIs (newproxy, setfenv,
   getfenv) were never registered in the global table regardless of the
   actual config value. Once the global table is locked, the config
   change has no effect.

2. lua_insecure_api_current was initialized to 0 (struct zero-init) and
   never synced with the final config value. If the default was changed
   to yes(1), a subsequent CONFIG SET no would see both values as 0 and
   skip the evalReset() call in updateLuaEnableInsecureApi().

Fix by reading the real config via isLuaInsecureAPIEnabled() in
createEngineContext() before Lua state initialization, and syncing
lua_insecure_api_current after all config sources (default, config file,
command-line args) are applied.

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 69bdda9fb8 Fix rdmaServer leaks when create listen cm id error (#3557) 
In here we should go to error to free the resources:
```
error:
    if (listen_cmid) rdma_destroy_id(listen_cmid);
    if (listen_channel) rdma_destroy_event_channel(listen_channel);
    ret = ANET_ERR;

end:
    freeaddrinfo(servinfo);
    return ret;
}
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-04-27 13:35:11 -07:00
charsyamandMadelyn Olson 7decd4f4b7 hashtable: fix dismissHashtable madvise size (#3533) 
The bug was in dismissHashtable(), which computes the size passed to
zmadvise_dontneed() for the top-level hashtable
  tables.

ht->tables[i] points to a contiguous array of bucket objects, but the
code used sizeof(bucket *) instead of
sizeof(bucket) when calculating the length. That means it treated the
allocation like an array of pointers rather than an
  array of buckets.

As a result, the advised range was much smaller than the actual table
allocation. On 64-bit builds, bucket is 64 bytes
while bucket * is 8 bytes, so only about one eighth of the table was
covered. This does not usually break correctness,
but it defeats the purpose of the function: after a fork, we want to
tell the kernel that the hashtable pages are no
longer needed so we reduce copy-on-write overhead. With the wrong size,
most of the table memory was never included in
  that hint.

The fix is to use sizeof(bucket) so the full top-level bucket array is
passed to zmadvise_dontneed().

Signed-off-by: DaeMyung Kang <charsyam@gmail.com>
 2026-04-27 13:35:11 -07:00
Hanxi ZhangandMadelyn Olson cb824a9c29 Strip LTO flags from static Lua module build (#3555) 
### Summary

The daily CI sanitizer jobs with clang are failing during the build
step.
When the static Lua module is built with `-flto`, the `.o` files contain
LLVM bitcode that gets archived into `libvalkeylua.a`. The system linker
cannot read this bitcode, causing build failures:

`/usr/bin/ld:
/home/runner/work/valkey/valkey/src/modules/lua/libvalkeylua.a: member
/home/runner/work/valkey/valkey/src/modules/lua/libvalkeylua.a(debug_lua.o)
in archive is not an object`

The previous fix (#3546) pinned clang to version 17, but this was
insufficient, the issue is not just a version mismatch but that the
system linker fundamentally cannot read LTO bitcode from `.a` archives.

Example failure:
https://github.com/valkey-io/valkey/actions/runs/24865821147/job/72801509768

### Fix

Strip LTO flags from OPTIMIZATION in the Lua module Makefile using
  `override`


Tested: https://github.com/hanxizh9910/valkey/actions/runs/24913834442

---------

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
 2026-04-27 13:35:11 -07:00
Sarthak AggarwalandMadelyn Olson 1aedfa455a Fix module commandresult event cleanup during unsubscribe and module unload (#3545) 
This follows up on the commandresult API work and fixes cleanup around
unsubscribe and module unload.

The main issue was that command-result event listeners could leave stale
state behind. On unload, we removed the listeners themselves but didn’t
fully update the fast-path listener counters. Separately, unsubscribing
with a NULL callback could behave badly if the listener wasn’t present
anymore. In practice, that meant later commands could still walk into
command-result event handling after the module was supposed to be
cleaned up.

Failed in Daily as well yesterday:
https://github.com/valkey-io/valkey/actions/runs/24753491944/job/72421581610#step:10:852
Related Failures:
https://github.com/valkey-io/valkey/pull/2936#issuecomment-4290490199

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
 2026-04-27 13:35:11 -07:00
Sarthak AggarwalandMadelyn Olson 98790669e5 Add zmalloc_aligned() and fix SPMC queue buffer alignment (#3504) 
The SPMC queue from #3324 needs each `spmcCell` to be cache-line
aligned, but plain `zmalloc()` does not guarantee that in all build
configurations.

This change introduces `zmalloc_cache_aligned()` and uses it for the
SPMC queue buffer allocation in `spmcInit()`.

Failing CI: https://github.com/valkey-io/valkey/actions/runs/24374139344

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
 2026-04-27 13:35:11 -07:00
charsyamandMadelyn Olson cc73cf1615 Optimize HGETDEL to pause auto shrink when deleting multiple items (#3535) 
Match HGETDEL with the existing batch-delete pattern used by HDEL.

HDEL already pauses hashtable auto-shrink while deleting multiple fields
so shrink evaluation is deferred until the batch completes. HGETDEL was
missing the same optimization even though it also deletes fields in a loop.

Pause auto-shrink for hashtable-encoded hashes before the HGETDEL delete
loop and resume it once afterwards. This preserves observable behavior
and reduces redundant shrink work for multi-field deletes.

Same as #3144.

Signed-off-by: DaeMyung Kang <charsyam@gmail.com>
 2026-04-27 13:35:11 -07:00
Madelyn Olson 8550b82ef2 Fix FD leak in connSocketBlockingConnect on timeout (#3541) 
## Summary
Fix a file descriptor leak in `connSocketBlockingConnect()` when
`aeWait()` times out.

## Bug
When `anetTcpNonBlockConnect()` succeeds but `aeWait()` times out (e.g.,
MIGRATE to an unreachable host), the fd is leaked because it was never
assigned to `conn->fd`. The caller's `connClose()` checks `conn->fd !=
-1` and skips cleanup.

## Fix
Assign `conn->fd = fd` immediately after `anetTcpNonBlockConnect()`
succeeds, before `aeWait()`. This way the caller's normal `connClose()`
cleanup path handles the fd on any error, which is consistent with how
the rest of the connection lifecycle works.

TLS connections also benefit since `connTLSBlockingConnect` delegates to
this function for the TCP layer.

## Reproducer
```
valkey-cli SET key hello
# Repeat against unreachable host:
for i in $(seq 1 30); do valkey-cli MIGRATE 192.0.2.1 6379 key 0 500; done
# Check: /proc/<pid>/fd shows 30 leaked socket fds
```

*This issue was generated by AI but verified, with love, by a human.*

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
 2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 1427741f3b Fix double free in stream consumer PEL loading with corrupt RDB data (#3498) 
There is a double free issue in the code. The error handling path called
both decrRefCount(o) and streamFreeNACK(nack), but the nack was obtained
from cgroup->pel via raxFind and is still referenced there. decrRefCount(o)
frees it through freeStream -> streamFreeCG -> raxFreeWithCallback(cg->pel, zfree),
so the explicit streamFreeNACK(nack) causes a double free.

Remove the redundant streamFreeNACK(nack) call and add a regression
test with a crafted corrupt payload that triggers the duplicate consumer
PEL entry path.

This was introduced in 6ec241d934.

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-04-27 13:35:11 -07:00
Roshan KhatriMadelyn Olsonsarthakaggarwal97
 		c883ca9a08 Deflake many-slot-migration under valgrind (#3462) 
## Problem
`Fix cluster` in `tests/unit/cluster/many-slot-migration.tcl` has been
timing out daily on valgrind jobs since April 3, 2026. The test runs 10
cluster nodes under valgrind, migrating 40,000 keys across 1,000 slots —
too much work for valgrind-instrumented builds.

The slowdown is caused by #3366 (dict→hashtable wrapper). Under `-O0`
(valgrind builds), the `static inline` wrappers become real function
calls that valgrind instruments, adding ~75% overhead to hot paths like
`dictSize`. This compounds across 10 valgrind processes over a 20-minute
migration test. No impact on production builds (`-O2` inlines everything).

## Fix
Scale the test workload down under valgrind: 10,000 keys / 250 slots
instead of 40,000 / 1,000. Normal runs are unchanged. Still exercises
the same cluster repair path.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Co-authored-by: sarthakaggarwal97 <sarthakaggarwal97@users.noreply.github.com>
 2026-04-27 13:35:11 -07:00
Deepak NandihalliandMadelyn Olson 8e360c7eb7 Fix race condition during async client freeing with IO threading enabled (#3458) 
When close_asap flag is set, set bytes read to 0
    
In the readToQueryBuf, the c->nread represents the number of bytes read.
When close_asap flag is set, there is a bug where the c->nread isn't
reset to 0 and this breaks the invariant. IOThreads then incorrectly
think there is data to read and results in a crash. This change fixes
this bug.

To elaborate on the race possible:

1. Let's say that a IO thread job for reading query from a client got
enqueued as part of a epoll -
https://github.com/valkey-io/valkey/blob/unstable/src/io_threads.c#L417.
2. Later the client gets freed async and is marked as close_asap -
https://github.com/valkey-io/valkey/blob/unstable/src/networking.c#L2175
3. While processing the io_thread job for the client, it invokes
iothreadReadQueryFromClient. Here,
[`readToQueryBuf`](https://github.com/valkey-io/valkey/blob/unstable/src/networking.c#L6497)
returns as a no-op since the client is marked close-asap. Also, the
c->nread is not reset to 0 and count contain the value from a previous
read.
4. Later parseInputBuffer [gets
invoked](https://github.com/valkey-io/valkey/blob/unstable/src/networking.c#L6514).
5. The parseInputBuffer then [accesses the
query_buf](https://github.com/valkey-io/valkey/blob/unstable/src/networking.c#L3864).
The query_buf here would be null in resetSharedQueryBuf as part of
beforeNextClient.

Signed-off-by: Deepak Nandihalli <deepak.nandihalli@gmail.com>
 2026-04-27 13:35:11 -07:00
Sarthak AggarwalMadelyn OlsonSarthak Aggarwal
 		3e2fd3c800 Stabilize diskless no-drop replication test (#3511) 
This deflakes all variants of `diskless replicas drop during rdb pipe`.

The main issue turned out to be that the test was too sensitive to
timing and log ordering under TLS, not that the core behavior was wrong.
This keeps the same five subcases (no, slow, fast, all, timeout) but
makes them much less CI-fragile.

CI passes 200 times:
https://github.com/sarthakaggarwal97/valkey/actions/runs/24547258515

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <25262500+sarthakaggarwal97@users.noreply.github.com>
Co-authored-by: Sarthak Aggarwal <25262500+sarthakaggarwal97@users.noreply.github.com>
 2026-04-27 13:35:11 -07:00
Yang ZhaoandMadelyn Olson 40b32ac4a8 Fix VLA warning in io_threads (#3518) 
https://github.com/valkey-io/valkey/pull/3324 introduced `BATCH_SIZE` as
a const int local variable and used it as an array bound. Clang 17
rejects this with:
```
io_threads.c:305:22: error: variable length array folded to constant array as an extension [-Werror,-Wgnu-folding-constant]
  305 |     void *batch_jobs[BATCH_SIZE];
      |                      ^~~~~~~~~~
1 error generated.
make[1]: *** [io_threads.o] Error 1
make: *** [all] Error 2

```
Old Clang versions do not emit this warning, maybe that is why the CI
passed. Fix by promoting `BATCH_SIZE` to a file-scope `#define`.

Signed-off-by: Yang Zhao <zymy701@gmail.com>
 2026-04-27 13:35:11 -07:00
martinrvisserMadelyn OlsonRicardo Dias
 		7464d45eaa Module command result callback addition (#2936) 
## Add Command Result Event Notifications for Modules

### Summary

1. Adds new server events `ValkeyModuleEvent_CommandResultSuccess` and
`ValkeyModuleEvent_CommandResultFailure` for that can notify subscribed
modules after command execution. This enables modules to implement audit
logging, error monitoring, performance tracking, and observability
without modifying core server code.
2. Adds new server event `ValkeyModuleEvent_CommandResultACLDenied` for
commands rejected by ACL. Together with PR #2237 this covers auditing of
authentication and authorisation.

### Motivation

There is currently no module API to observe command outcomes after
execution or to capture ACL denied commands. Modules that need audit
logging or error monitoring have no mechanism to be notified when
commands succeed or fail, what arguments were used, how long they took,
or how many keys were modified. This feature fills that gap using the
existing `ValkeyModule_SubscribeToServerEvent()` infrastructure.

### API

#### Events

| Event | Description |
|---|---|
| `ValkeyModuleEvent_CommandResultSuccess` | Fired after a command
completes successfully |
| `ValkeyModuleEvent_CommandResultFailure` | Fired after a command
returns an error |
| `ValkeyModuleEvent_CommandACLDenied` | Fired after a command is
rejected by ACL |

These are separate events (not sub-events), so modules can for example
only subscribe to failures without incurring any callback overhead for
successful commands.

#### Event Data: `ValkeyModuleCommandResultInfo`

The `data` pointer passed to the callback can be cast to
`ValkeyModuleCommandResultInfo`:

```c
typedef struct ValkeyModuleCommandResultInfo {
    uint64_t version;           /* Version of this structure for ABI compat. */
    const char *command_name;   /* Full command name (e.g., "SET", "CLIENT|LIST"). */
    long long duration_us;      /* Execution duration in microseconds. */
    long long dirty;            /* Number of keys modified. */
    uint64_t client_id;         /* Client ID that executed the command. */
    int is_module_client;       /* 1 if command was from RM_Call, 0 otherwise. */
    int argc;                   /* Number of command arguments. */
    ValkeyModuleString **argv;  /* Command arguments array (zero-copy, read-only). */
    int acl_deny_reason;        /* ACL_DENIED_CMD/KEY/CHANNEL/AUTH; 0 for non-ACL events */
    const char *acl_object;     /* Denied resource name (key/channel); NULL for CMD/AUTH */
} ValkeyModuleCommandResultInfoV1;
```

The struct is versioned (`VALKEYMODULE_COMMANDRESULTINFO_VERSION`) for
forward-compatible API evolution.

### Usage Example

```c
/* Callback receives events for whichever event(s) you subscribed to */
void OnCommandResult(ValkeyModuleCtx *ctx, ValkeyModuleEvent eid,
                     uint64_t subevent, void *data) {
    VALKEYMODULE_NOT_USED(ctx);
    VALKEYMODULE_NOT_USED(subevent);

    ValkeyModuleCommandResultInfo *info = (ValkeyModuleCommandResultInfo *)data;
    if (info->version != VALKEYMODULE_COMMANDRESULTINFO_VERSION) return;

    int failed = (eid.id == VALKEYMODULE_EVENT_COMMAND_RESULT_FAILURE);

    /* Access fields directly */
    printf("command=%s status=%s duration=%lldus dirty=%lld client=%llu\n",
           info->command_name,
           failed ? "FAIL" : "OK",
           info->duration_us,
           info->dirty,
           info->client_id);

    /* Access argv (read-only, zero-copy) */
    for (int i = 0; i < info->argc; i++) {
        size_t len;
        const char *arg = ValkeyModule_StringPtrLen(info->argv[i], &len);
        printf("  argv[%d] = %.*s\n", i, (int)len, arg);
    }
}

/* Subscribe in ValkeyModule_OnLoad or at runtime */

/* Option A: command failures only (recommended for audit logging) */
ValkeyModule_SubscribeToServerEvent(ctx,
    ValkeyModuleEvent_CommandResultFailure, OnCommandResult);

/* Option B: command successes only */
ValkeyModule_SubscribeToServerEvent(ctx,
    ValkeyModuleEvent_CommandResultSuccess, OnCommandResult);

/* Option C: both command outcomes*/
ValkeyModule_SubscribeToServerEvent(ctx,
    ValkeyModuleEvent_CommandResultSuccess, OnCommandResult);
ValkeyModule_SubscribeToServerEvent(ctx,
    ValkeyModuleEvent_CommandResultFailure, OnCommandResult);

/* Subscribe to ACL Denied */
ValkeyModule_SubscribeToServerEvent(ctx,
        ValkeyModuleEvent_CommandResultACLDenied, onCommandResult);

/* Unsubscribe pass NULL callback */
ValkeyModule_SubscribeToServerEvent(ctx,
    ValkeyModuleEvent_CommandResultFailure, NULL);
```

### Design Decisions

- **Separate events instead of sub-events**: Modules subscribing only to
failures have zero overhead for successful commands (~2ns listener-list
check vs ~30ns callback invocation per command). This is critical since
success events fire on the hot path of every command.
- **Stack-allocated info struct**: The `ValkeyModuleCommandResultInfoV1`
is built on the stack ΓÇö no heap allocation per event.
- **Zero-copy argv**: Arguments are passed directly from the client's
argv array. Any integer-encoded arguments (from `tryObjectEncoding()`
during command execution) are decoded to string-encoded objects before
being passed to the callback, ensuring compatibility with
`ValkeyModule_StringPtrLen()`.
- **Early exit**: If no modules are subscribed to any server events, the
event firing function returns immediately before building the info
struct.
- **Uses existing server event infrastructure**: Follows the
`ValkeyModule_SubscribeToServerEvent()` pattern used by all other server
events, rather than introducing a new callback mechanism.

### Files Changed

| File | Change |
|---|---|
| `src/valkeymodule.h` | Event IDs, event constants,
`ValkeyModuleCommandResultInfoV1` struct |
| `src/module.c` | `moduleFireCommandResultEvent()`, event
documentation, event version entries |
| `src/module.h` | Function declaration |
| `src/server.c` | Call `moduleFireCommandResultEvent()` from `call()`
after command execution |
| `src/server.c` | Call to `moduleFireCommandACLDeniedEvent` in
`processCommand` after ACL rejection |
| `tests/modules/commandresult.c` | Test module exercising the full API
|
| `tests/unit/moduleapi/commandresult.tcl` | Integration tests |

---------

Signed-off-by: martinrvisser <mvisser@hotmail.com>
Signed-off-by: martinrvisser <martinrvisser@users.noreply.github.com>
Co-authored-by: Ricardo Dias <rjd15372@gmail.com>
 2026-04-27 13:35:11 -07:00
eifrah-awsandMadelyn Olson 37242267c9 Add Static Module Support (#3392) 
Add a build option to compile the Lua scripting engine as a static
module and wire the server to load it directly at startup when enabled.
The module load path now resolves on-load and on-unload entry points
from the main binary, and the module lifecycle keeps those callbacks so
unload works without a shared library handle.

The Lua module build was updated to support both static and shared
variants, with the static path exporting visible wrapper symbols and
linking the server with the module archive. While touching the Lua code,
a few internal symbols were renamed for consistency and the monotonic
time helper was clarified.

Note that this PR addresses the LUA module, but it can be applied to
other "core" modules (like: Bloom, Json, Search and others). With this
change, it will be easier to ship Valkey bundle with modules.

Areas touched:

* CMake
* Makefile
* Lua scripting module
* Core module loading

**Generated by CodeLite**

---------

Signed-off-by: Eran Ifrah <eifrah@amazon.com>
 2026-04-27 13:35:11 -07:00
Daniil KashapovandMadelyn Olson 28f1cc9b29 Improve COB memory tracking with copy avoidance (#3306) 
This improves COB memory tracking when using copy avoidance for bulk
string replies. This fix addresses underestimation of client memory
usage that occurred when reply buffers stored pointers to shared `robj`
instead of copying data.
IO threads calculate actual reply sizes by calling `sdslen()` on strings
before writing, for that we need atomic `tracked_for_cob` flag in
payload headers to prevent race conditions and double accounting.

See #2396

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
 2026-04-27 13:35:11 -07:00
Madelyn Olson 60fbe16c63 Fix HPERSIST RESP protocol violation on wrong-type key (#3516) 
`hpersistCommand` calls `addReplyArrayLen` before `lookupKeyWrite` +
`checkType`. When HPERSIST targets a non-hash key, the server writes a
RESP array header followed by a WRONGTYPE error — a malformed response
that permanently desynchronizes the client connection.

This moves `lookupKeyWrite` + `checkType` before `addReplyArrayLen`,
matching the pattern used by every other HFE command (e.g.
`hgetdelCommand`, `hexpireGenericCommand`).

Added a test for HPERSIST on a wrong-type key.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
 2026-04-27 13:35:11 -07:00
Sarthak AggarwalMadelyn OlsonSarthak Aggarwal
 		a09e12e27b [Flaky Tests] Avoid re-triggering io-thread activation (#3509) 
The test was accidentally waking the IO threads while trying to check
that they had gone idle.

After the recent IO-thread refactor in #3324, the
[test](https://github.com/valkey-io/valkey/pull/3324/changes#diff-21314ec3a338f739eab1536f91f528d1efe7c6a93935a71b9c02f77a3858f121R112)
started forcing `io-threads-always-active`, and its repeated `INFO`
polling counted as fresh activity. So instead of just observing the
worker threads, the test kept reactivating them and then flaked.

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <25262500+sarthakaggarwal97@users.noreply.github.com>
Co-authored-by: Sarthak Aggarwal <25262500+sarthakaggarwal97@users.noreply.github.com>
 2026-04-27 13:35:11 -07:00
Roshan KhatriandMadelyn Olson 728595058a Fix use-after-unload crash in test auth module's blocking thread (#3464) 
## Problem

The test `Test module aof save on server start from empty` in
`tests/unit/moduleapi/hooks.tcl` sporadically crashes with `I/O error
reading reply`.

**Frequency:** 2 out of 15 days (March 26 on
`centosstream9-tls-module-no-tls`, April 8 on `fedorarawhide-jemalloc`).

**Example failing run:**
https://github.com/valkey-io/valkey/actions/runs/24110987718/job/70345236353

## Root Cause

The crash is a **use-after-unload** in the auth test module's blocking
authentication thread, NOT a timing issue in the AOF test.

The crash log from April 8 shows:
```
71112:M 00:42:59.710 * Module testacl unloaded
71112:M 00:42:59.711 # crashed by signal: 11, si_code: 1
71112:M 00:42:59.711 # Crashed running the instruction at: 0x7f9dc717384b
```

The sequence:
1. `blocking_auth_cb` spawns a background thread
(`AuthBlock_ThreadMain`) that sleeps 500ms
2. Thread wakes, calls `ValkeyModule_UnblockClient()` → main thread
processes unblock, decrements `module->blocked_clients`
3. Auth command completes, test calls `r module unload testacl`
4. `moduleUnloadInternal` checks `blocked_clients == 0` if true,
proceeds with `dlclose()`
5. **But the background thread is still executing cleanup code**
(freeing strings, returning from function)
6. Thread returns into unmapped memory → **SIGSEGV**

The `invalidFunctionWasCalled` in the stack trace is the crash handler's
safety stub, and the crashing address `0x7f9dc717384b` is in the
unmapped auth.so address space.

## Fix

Track the background thread ID and `pthread_join()` it in
`ValkeyModule_OnUnload` before the module is dlclose'd. This ensures the
thread has fully exited before the code is unmapped.

The key insight is that `ValkeyModule_UnblockClient()` signals "auth is
done" but not "thread is done" — the thread still has cleanup code to
execute after that call. `pthread_join()` is the correct synchronization
point because it only returns after the thread has fully exited.

No mutex is needed since both `blocking_auth_cb` (which creates the
thread) and `OnUnload` (which joins it) run on the main event loop
thread.

Changes to `tests/modules/auth.c`:
- Add global `blocking_auth_tid` and `blocking_auth_tid_valid` flag
- Set `blocking_auth_tid_valid = 1` after successful `pthread_create`
- In `OnUnload`, `pthread_join` the thread if one was created

## Testing

Ran `unit/moduleapi/hooks` 100 loops on rpm-distros and ubuntu runners —
**all passed**:
- **Workflow run:**
https://github.com/roshkhatri/valkey/actions/runs/24164276124
- **Config:** `--loops 100 --single unit/moduleapi/hooks` on
`almalinux8`, `almalinux9`, `fedoralatest`, `fedorarawhide`,
`centosstream9`, `ubuntu-jemalloc`, `ubuntu-arm`
- **Result:** 7/7 jobs ✅, zero failures across 700 total test iterations

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
 2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 6a6b6c8746 Rewrite the faster failover test case (#3495) 
The test introduced in #2227 is fragile because it unconditionally asserted
that "best ranked replica" would be logged for every shard's replica. This
assumption was incorrect for two reasons:

1. myselfIsBestRankedReplica() requires failover_failed_primary_rank == 0.
   When two primaries fail simultaneously, only the shard with the lowest
   shard_id gets failed_primary_rank == 0. The other shard's replicas can
   only trigger "Myself become the best ranked replica" after the first
   shard completes failover and the result propagates via gossip.

2. clusterAllReplicasThinkPrimaryIsFail() requires all sibling replicas
   to have their MY_PRIMARY_FAIL flag propagated via gossip. If the
   election delay is shorter than the gossip propagation time, the
   replica may start the election before receiving the flag.

It's also too strict and checks too many things. Like we can check that there
is no delay before starting the failover, but we shouldn't check that the rank
0 replica actually wins, because maybe it doesn't. Also we're not sure the other
replica does a psync afterwards, it can also do a full sync.

Restructure the test into three focused scenarios with retry logic:

- Test 0 (3 primaries + 1 replica): The sole replica is deterministically
  the best ranked replica. All conditions are trivially satisfied.

- Test 1 (3 primaries + 2 replicas): Single primary failure with two
  competing replicas. failed_primary_rank is deterministically 0, but
  MY_PRIMARY_FAIL gossip timing may prevent triggering in some runs.
  Uses internal retry with cluster recovery to ensure at least one
  successful trigger.

- Test 2 (5 primaries + 7 replicas): Two primaries failure. Verifies no
  failover timeout and uses retry to cover the "best ranked replica"
  path for at least one shard.

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 0d297bddb4 Change rdbSaveStreamConsumers return type from size_t to ssize_t (#3499) 
Minor cleanup, the function will return -1 on error.
It worked anyway before because it is implicitly converted:
-1 => SIZE_MAX => -1 again when it's implicitly cast back to
ssize_t.

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-04-27 13:35:11 -07:00
Akash KumarMadelyn OlsonUri YagelnikDan Touitou
 		267e3a54f9 Redesign IO threading communication model (#3324) 
#### Summary
This PR redesigns the IO threading communication model, replacing the
inefficient client-list polling approach with a high-performance,
lock-free queue architecture. This change improves throughput by
**8–17%** across various workloads and lays the groundwork for
offloading command execution to IO threads in following PRs.



### Performance Comparison: Unstable vs New IO Queues

| Type | Operation | Unstable Branch (M TPS) | New IO Queues (M TPS)|
Difference (%) |
| :--- | :--- | :--- | :--- | :--- |
| **CME**<sup>1</sup> | SET | 1.02 | 1.19 | **+16.67%** |
| **CME** | GET | 1.30 | 1.47 | **+13.08%** |
| **CMD**<sup>2</sup> | SET | 1.15 | 1.35 | **+17.39%** |
| **CMD** | GET | 1.52 | 1.64 | **+7.89%** |

<sup>1</sup> Amazon terminology for cluster mode
<sup>2</sup> Amazon termonology for standalone mode, i.e. config
`cluster-enabled no`

- Test Configuration: 8 IO threads • 400 clients • 512-byte values • 3M
keys

#### Motivation
The previous IO model had several limitations that created performance
bottlenecks:
* **Inefficient Polling:** The main thread lacks a direct notification
mechanism for completed work. Instead, it must constantly iterate
through a list of all pending clients to check their state, wasting
significant CPU cycles.
* **Manual Load Balancing:** Jobs are assigned to specific threads
upfront. This requires the main thread to predict which thread to use,
often leaving some threads idle while others are overloaded.
* **Static Scaling:** Thread activation relies on a fixed heuristic
(e.g., 1 thread per 2 events). This approach fails to adapt to varying
workloads, such as TLS connections or differing read/write sizes.

### The Solution
To address these inefficiencies, this PR replaces the single SPSC queue
used currently with three specialized queues to handle communication and
load balancing more effectively.

#### 1. Main > IO: Shared Queue (Single Producer Multi Consumer)
Single queue from the main-thread to IO threads.
* **Automatic Load Balancing:** All threads pull from the same source.
Busy threads take less work, and idle threads take more, so we don't
need to manually select a thread.
* **Adaptive Scaling:** We now use the queue depth to decide when to add
or remove threads. If the queue is full, we scale up; if it's empty, we
scale down.
* *Ignition:* To get things started before the queue fills up, we
monitor the main thread's CPU. If usage goes over 30%, we wake up the
first IO thread.
* **Implementation:** To prevent contention among consumers, each item
in the ring buffer is padded to reside in its own cache line. Sequence
numbers are utilized to indicate whether a cell is empty or populated,
allowing threads to safely claim work.

#### 2. IO > Main: The Response Channel (MPSC Queue)
We replaced the old polling loop with a response queue.
* ** Faster Completion:** IO threads push completed jobs into this
queue. The main thread detects new data simply by checking if the queue
is not empty, removing the need to scan pending clients.
* **Contention Management:** To avoid lock contention, each thread
reserves a slot by atomically incrementing the tail index. In the rare
event that the queue is full, pending jobs are buffered in a local
temporary list until space becomes available.

#### 3. MAIN > IO (Thread-Specific): Private Inbox (SPSC Queue)
We kept the existing Single-Producer Single-Consumer (SPSC) queues for
tasks that must happen on a specific thread (like freeing memory
allocated by that thread). IO threads always check their private inbox
before looking at the shared queue.

### Changes Required
* **Async client release**
The main thread no longer busy-waits for IO threads to finish with a
client. Since the client must be popped from the multi-producer queue
before it can be released, clients with pending IO are now marked for
asynchronous closure.
* **eviction clients logic**
Updated evictClients() to account for memory pending release (clients
marked close_asap). freeClient() now returns a status code (1 for freed,
0 for async-close) to ensure the eviction loop does not over-evict by
ignoring memory that is about to be reclaimed.
* **events-per-io-thread config**
Replaced the `events-per-io-thread` configuration with
`io-threads-always-active`. as we no longer track events, since this
config is use only for tests no backward compatibility issue arises.
* **packed job instead of handlers**
Jobs are now represented as tagged pointers (using lower 3 bits for job
type) instead of separate `{handler, data}` structs. This reduces memory
overhead and allows jobs to be passed through the queues as single
pointers.
* **head caching in spsc queue**
The SPSC queue now caches the `head` index on the producer side
(`head_cache`) to avoid frequent atomic loads. The producer only
refreshes from the atomic `head` when the cache indicates the queue
might be full, reducing cross-thread cache-line bouncing.

* **deferred commit in SPSC queue**.   
`spscEnqueue()` supports batching via a `commit` flag. Multiple jobs can
be enqueued with `commit=false`, then flushed with a single
`spscCommit()` call, reducing atomic operations and cache-line bouncing.
* **rollback on fullness check failure**
When `spmcEnqueue()` fails due to a full queue, the client state is
rolled back (e.g., `io_write_state` reset to `CLIENT_IDLE`). This
rollback approach removes the need to call an expensive `isFull` check
before every enqueue, we just attempt the enqueue and revert if it
fails.

* **epoll offloading via SPSC at high thread counts**.  
When `active_io_threads_num > 9`, poll jobs are sent to per-thread SPSC
queues (round-robin). Since threads check their private queue first,
this ensures poll jobs are processed promptly without waiting behind
jobs in the shared SPMC queue.
* **avoid offload write before read comes back**
Added a check `if (c->io_read_state == CLIENT_PENDING_IO) return C_OK`
in `trySendWriteToIOThreads()`. In the previous per-thread SPSC
implementation, we could send consecutive read and write jobs for the
same client knowing a single thread would handle them in order. With the
shared SPMC queue, different threads may pick up the jobs, so we must
wait for the read to complete before sending a write to avoid 2 threads
handling the same client.
* **removing pending_read_list_node from client and
clients_pending_io_read/write lists from server**
Removed `pending_read_list_node` from the `client` struct and
`clients_pending_io_read`/`clients_pending_io_write` lists from
`valkeyServer`. as the new mpsc eliminates the need for these tracking
structures.
* **added inst metrics for pending io jobs**
Added `instantaneous_io_pending_jobs` metric via `STATS_METRIC_IO_WAIT`
to track average queue depth over time.
* **added stat for current active threads number**
Added `active_io_threads_num` to the INFO stats output for better
visibility.
* **added internal inst metric for main-thread cpu (non apple
compliant)**
Added `STATS_METRIC_MAIN_THREAD_CPU_SYS` to track main thread CPU usage
via `getrusage(RUSAGE_THREAD)`. This powers the "ignition" policy, when
CPU exceeds 30%, the first IO thread is activated. `RUSAGE_THREAD` is
Linux-specific, so macOS falls back to event-count heuristics.
* **added stat for pending read and writes for io**
Added `io_threaded_reads_pending` and `io_threaded_writes_pending` stats
to track how many read/write jobs are currently in-flight to IO threads.
* **added volatile for crashed**
Changed `server.crashed` from `int` to `volatile int` to ensure the
crash flag is visible across threads immediately, allowing IO threads to
detect a crash and stop sending responses back to the main thread to
avoid deadlock on crash.

---------

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
Signed-off-by: akash kumar <akumdev@amazon.com>
Co-authored-by: Uri Yagelnik <uriy@amazon.com>
Co-authored-by: Dan Touitou <dan.touitou@gmail.com>
 2026-04-27 13:35:11 -07:00
Ricardo DiasandMadelyn Olson 2f351c1b2b Adds new (hidden) ValkeyModule_CallArgv API functions (#3122) 
This PR introduces `ValkeyModule_CallArgv`, a new low-level module API
for calling server commands. It is designed as a complement to the
existing `ValkeyModule_Call` covering two scenarios where
`ValkeyModule_Call` is suboptimal:

1. **The module already holds the arguments as an argv array.**
`ValkeyModule_Call` always allocates a new argv array, creates a new
string object for the command name, and calls
`incrRefCount`/`decrRefCount` on every argument. `ValkeyModule_CallArgv`
borrows the caller's argv array directly — no allocation, no refcount
manipulation. Ownership of the array stays with the caller.

2. **The module wants to forward the reply to its own client without
inspecting it.** `ValkeyModule_Call` always parses the raw RESP bytes
into an intermediate `ValkeyModuleCallReply` tree and then re-serializes
it back to RESP to send to the caller. `ValkeyModule_CallArgv` exposes
the raw RESP bytes directly through a callback, allowing a pass-through
module command to copy bytes from the inner command's output buffer
straight into the outer client's output buffer.

---

## New API

### `ValkeyModule_CallArgv`

```c
int ValkeyModule_CallArgv(ValkeyModuleCtx *ctx,
                          ValkeyModuleString **argv,
                          int argc,
                          int flags,
                          ValkeyModuleReplyHandlers *reply_handlers,
                          void *reply_ctx);
```

Calls a server command using an existing argv array. The caller retains
ownership of `argv` and its elements — they are never freed or
ref-counted by the function.

Returns `VALKEYMODULE_OK` on success or `VALKEYMODULE_ERR` on error,
with `errno` set to the same values as `ValkeyModule_Call`. The function
does **not** return a `ValkeyModuleCallReply`; the reply is delivered
through `reply_handlers` instead.

`flags` is a bitwise OR of one or more `VALKEYMODULE_CALL_ARGV_*`
constants, which correspond directly to the format-string specifiers of
`ValkeyModule_Call`:

| Flag | Equivalent format char | Meaning |
|---|---|---|
| `VALKEYMODULE_CALL_ARGV_REPLICATE` | `!` | Propagate to replicas and
AOF |
| `VALKEYMODULE_CALL_ARGV_NO_AOF` | `A` | Skip AOF propagation |
| `VALKEYMODULE_CALL_ARGV_NO_REPLICAS` | `R` | Skip replica propagation
|
| `VALKEYMODULE_CALL_ARGV_RESP_3` | `3` | Force RESP3 for the inner call
|
| `VALKEYMODULE_CALL_ARGV_RESP_AUTO` | `0` | Match the calling client's
protocol (RESP2 or RESP3) |
| `VALKEYMODULE_CALL_ARGV_RUN_AS_USER` | `C` | Apply ACL checks for the
context user |
| `VALKEYMODULE_CALL_ARGV_SCRIPT_MODE` | `S` | Mark as script execution
|
| `VALKEYMODULE_CALL_ARGV_NO_WRITES` | `W` | Disallow write commands |
| `VALKEYMODULE_CALL_ARGV_ERRORS_AS_REPLIES` | `E` | Return error
replies instead of raising errno |
| `VALKEYMODULE_CALL_ARGV_RESPECT_DENY_OOM` | `M` | Honour deny-oom
policy |
| `VALKEYMODULE_CALL_ARGV_DRY_RUN` | `D` | Dry-run mode (implies
`ERRORS_AS_REPLIES`) |
| `VALKEYMODULE_CALL_ARGV_ALLOW_BLOCK` | `K` | Allow the inner command
to block |
| `VALKEYMODULE_CALL_ARGV_REPLY_EXACT` | `X` | Disable reply type
coercion |

---

### `ValkeyModuleReplyHandlers`

```c
typedef struct ValkeyModuleReplyHandlers {
    /* Scalar types */
    void (*null)(void *ctx);
    void (*bulkString)(void *ctx, const char *str, size_t len);
    void (*simpleString)(void *ctx, const char *str, size_t len);
    void (*error)(void *ctx, const char *msg, size_t len);
    void (*integer)(void *ctx, long long val);
    void (*doubleVal)(void *ctx, double val);
    void (*boolVal)(void *ctx, int val);
    void (*bigNumber)(void *ctx, const char *str, size_t len);
    void (*verbatimString)(void *ctx, const char *str, size_t len, const char *fmt);
    /* Collection types — paired start/end callbacks */
    void (*arrayStart)(void *ctx, size_t len);
    void (*arrayEnd)(void *ctx);
    void (*mapStart)(void *ctx, size_t len);
    void (*mapEnd)(void *ctx);
    void (*setStart)(void *ctx, size_t len);
    void (*setEnd)(void *ctx);
    void (*attributeStart)(void *ctx, size_t len);
    void (*attributeEnd)(void *ctx);
    /* Invoked on parse error */
    void (*replyParsingError)(void *ctx);
    /* Raw-bytes intercept — called before per-type callbacks */
    int (*onRespAvailable)(void *ctx, ValkeyModuleCtx *module_ctx, const char *proto, size_t proto_len);
    /* Blocking command support */
    void (*onBlocked)(void *ctx, ValkeyModuleCtx *module_ctx, ValkeyModuleCallArgvBlockedHandle *handle);

    void *context;   /* Passed as first argument to every callback */
} ValkeyModuleReplyHandlers;
```

All fields are optional; set unused ones to `NULL`.

**`onRespAvailable`** is called first, before any per-type callback,
with the complete raw RESP reply in the `proto`/`proto_len` arguments.
Return `1` to continue into the per-type callbacks; return `0` to stop
after the raw bytes. Pass `NULL` to skip and go straight to per-type
dispatching.

**`onBlocked`** is only called when `VALKEYMODULE_CALL_ARGV_ALLOW_BLOCK`
is set and the inner command blocks. The `handle` can be passed to
`ValkeyModule_CallArgvAbort` to cancel the call. The handle is valid
until either `onRespAvailable` is invoked or
`ValkeyModule_CallArgvAbort` is called, whichever comes first.

---

### `ValkeyModule_CallArgvAbort`

```c
int ValkeyModule_CallArgvAbort(ValkeyModuleCallArgvBlockedHandle *handle);
```

Cancels a blocking call whose `onBlocked` callback has already fired.
Returns `VALKEYMODULE_OK` if the abort succeeded (neither
`onRespAvailable` nor any other reply callback will be invoked), or
`VALKEYMODULE_ERR` if the call has already completed.

---

### `ValkeyModule_ReplyRaw`

```c
int ValkeyModule_ReplyRaw(ValkeyModuleCtx *ctx, const char *proto, size_t proto_len);
```

Appends raw RESP bytes directly to the calling client's output buffer,
with no parsing or re-serialization. Intended for use inside an
`onRespAvailable` callback to implement zero-copy reply forwarding.

---

## Usage Example: `ValkeyModule_Call` vs `ValkeyModule_CallArgv`

The following two implementations of a generic pass-through proxy
command are functionally equivalent, but use different code paths
internally.

### With `ValkeyModule_Call` (existing API)

```c
int ProxyCommand(ValkeyModuleCtx *ctx, ValkeyModuleString **argv, int argc) {
    if (argc < 2) return ValkeyModule_WrongArity(ctx);

    /* VM_Call builds a new argv array internally, creates a string object for
     * the command name, and calls incrRefCount/decrRefCount on every argument.
     * The reply is parsed into a CallReply tree, then re-serialized to RESP. */
    ValkeyModuleCallReply *reply =
        ValkeyModule_Call(ctx, ValkeyModule_StringPtrLen(argv[1], NULL), "v",
                          argv + 2, (size_t)(argc - 2));

    return ValkeyModule_ReplyWithCallReply(ctx, reply);
}
```

### With `ValkeyModule_CallArgv` and `onRespAvailable` (new API —
pass-through)

```c
static int forwardRawReply(void *ctx, ValkeyModuleCtx *mctx,
                           const char *proto, size_t proto_len) {
    (void)ctx;
    /* Write the raw RESP bytes directly to the outer client — no parse, no
     * re-serialize, no intermediate allocation. */
    ValkeyModule_ReplyRaw(mctx, proto, proto_len);
    return 0; /* stop — do not invoke per-type callbacks */
}

static ValkeyModuleReplyHandlers passthrough_handlers = {
    .onRespAvailable = forwardRawReply,
};

int ProxyCommand(ValkeyModuleCtx *ctx, ValkeyModuleString **argv, int argc) {
    if (argc < 2) return ValkeyModule_WrongArity(ctx);

    /* VM_CallArgv borrows argv directly (no copy, no refcount changes).
     * The reply is written straight into the outer client's output buffer
     * through the onRespAvailable callback above. */
    if (ValkeyModule_CallArgv(ctx, argv + 1, argc - 1,
                              VALKEYMODULE_CALL_ARGV_RESP_AUTO,
                              &passthrough_handlers,
                              NULL) == VALKEYMODULE_ERR) {
        return ValkeyModule_ReplyWithError(ctx, "ERR command failed");
    }
    return VALKEYMODULE_OK;
}
```

Use `VALKEYMODULE_CALL_ARGV_RESP_AUTO` to ensure the inner command
replies in the same protocol version as the outer client (RESP2 or
RESP3), so the raw bytes can be forwarded without modification.

---

## Internal changes

### `argv_borrowed` client flag

A new `argv_borrowed` bit is added to the client flags struct. When
`VM_CallArgv` runs the inner command on a temporary client, it sets this
flag to signal that the temp client does not own the argv array.
`freeClientArgv` and `freeClientOriginalArgv` respect the flag and skip
freeing. `resetClient` clears it unconditionally after each command.
Several string commands (`SET`, `MSET`, `APPEND`, …) that previously
encoded or reused argv objects in-place are guarded so they do not
mutate argv elements they do not own.

A debug-only assertion (enabled when `enable-debug-assert yes` is set in
the config) verifies at runtime that no command modifies the argv array
or decrements any element's ref-count when `argv_borrowed` is set.

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
Signed-off-by: Ricardo Dias <rjd15372@gmail.com>
 2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 7661f53622 Add cluster-config-save-behavior option to control nodes.conf save behavior (#3372) 
This commit introduces a new configuration option `cluster-config-save-behavior`
that controls how the cluster handles nodes.conf file save failures.

The option supports two modes:
- `sync` (default): Synchronously save the config file. If the save fails,
  the process exits. This maintains backward compatibility with the old
  behavior (before 9.1).
- `best-effort`: Synchronously save the config file. If the save fails,
  only log a warning and continue running. This allows the node to survive
  disk failures (e.g., disk full, read-only filesystem) without exitting,
  giving administrators time to address the issue.

Note that this modifies the behavior of #1032, whereas #1032 was "best-effort",
we have now introduced a configuration option that defaults to "sync."
See #1032 discussion for more details.

Background:
When a disk becomes read-only or full, any cluster metadata change would
trigger a nodes.conf save attempt. With the old behavior, the node would
immediately exit via clusterSaveConfigOrDie(), potentially causing multiple
nodes on the same machine to crash simultaneously, leading to cluster
unavailability.

The new `best-effort` mode addresses this by allowing nodes to continue
operating even when disk writes fail. This is particularly useful in cloud
environments where disk failures are more common due to scale.

Note: Startup-time config saves (in clusterInit and verifyClusterConfigWithData)
still use clusterSaveConfigOrDie() since disk issues at startup should cause
immediate failure.

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-04-27 13:35:11 -07:00
Ricardo DiasandMadelyn Olson 3cbf0abb9d Defer argument redaction by storing indices instead of rewriting argv (#3471) 
Previously, `redactClientCommandArgument()` forced a full copy of
`c->argv` into `original_argv` immediately, then replaced the sensitive
slot with `shared.redacted`. This meant any call to redact an argument
triggered an eager argv backup even when no rewriting was needed.

The new approach stores the indices of arguments to be redacted in a
small dynamic array (`redact_args`) on the client struct. Redaction is
applied lazily when the argv is actually consumed (e.g. by the command
log), avoiding the unnecessary argv copy on the hot path.

This change is required for the implementation of
`ValkeyModule_CallArgv` module API function (#3122) , because CallArgv
owns the `argv` array, which can't be modified by the redact function.

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
 2026-04-27 13:35:11 -07:00
Roshan KhatriandMadelyn Olson 34ce284e3d Increase timeouts in faster-failover test for slow CI runners (#3463) 
Two changes to tests/unit/cluster/faster-failover.tcl:

1. FAIL detection timeout: `wait_for_condition 1000 10` → `1000 50`
   (10s → 50s)
2. psync_max_retries: 1200 → 2400 normal (120s → 240s),
   6000 → 12000 valgrind (600s → 1200s)

The test `The best replica can initiate an election immediately in an
automatic failover` in `tests/unit/cluster/faster-failover.tcl` has been
flaky since it was introduced on March 27, 2026 by #2227.

**Frequency:** 8 out of 15 days (Mar 27 – Apr 8), across valgrind,
sanitizer, and slow CI runners.

**Common errors:**
- `log message of "Successful partial resynchronization with primary"
  not found` (timeout waiting for psync)
- `expected pattern found in srv -N log file: *best ranked replica*`
  (timeout waiting for FAIL propagation)

The test spins up a 12-node cluster (5 primaries + 7 replicas), pauses
nodes, and waits for FAIL detection to propagate across all nodes before
failover + partial resync. 

A previous fix attempt #3424 increased the psync timeout from 50s to
120s (600s valgrind), which reduced frequency but did not eliminate it.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
 2026-04-27 13:35:11 -07:00
charsyamandMadelyn Olson 43f39b08d9 Fix trivial double-free issue in rdbLoadObject (#3453) 
In rdbLoadObject, when sdstrynewlen is OK and hashtableAdd is OK,
but lpSafeToAdd is FAIL, field will be double-freed.

But lpSafeToAdd will only fail when len + add > LISTPACK_MAX_SAFETY_SIZE(1GB),
so it can rarely happened and trivial.

Signed-off-by: charsyam <charsyam@naver.com>
 2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson f3b496193b Fix config rewrite producing negative values for unsigned memory configs (#3440) 
rewriteConfigFormatMemory() and rewriteConfigBytesOption() used long long
parameters to represent memory bytes, but configs like maxmemory are stored
as unsigned long long and allow values up to ULLONG_MAX.

When the value exceeds LLONG_MAX (e.g. 9223372036854775808), it overflows to
negative when passed as long long, causing config rewrite to produce incorrect
output like "maxmemory -8589934592gb".

Change both functions to use unsigned long long, matching the actual semantics
of memory byte values. This is consistent with how numericConfigGet() already
handles MEMORY_CONFIG using ull2string().

There are some MEMORY_CONFIG are signed and can be negative:
1. maxmemory-clients is a PERCENT_CONFIG
2. slot-migration-max-failover-repl-bytes is a SIGNED_MEMORY_CONFIG (after #3443) 

None of them were affected:
```
static void numericConfigRewrite(standardConfig *config, const char *name, struct rewriteConfigState *state) {
    ...
    if (config->data.numeric.flags & PERCENT_CONFIG && value < 0) {
        rewriteConfigPercentOption(state, name, -value, config->data.numeric.default_value);
    } else if (config->data.numeric.flags & SIGNED_MEMORY_CONFIG && value < 0) {
        rewriteConfigNumericalOption(state, name, value, config->data.numeric.default_value);
    } else if (config->data.numeric.flags & MEMORY_CONFIG) {
        rewriteConfigBytesOption(state, name, value, config->data.numeric.default_value);
    ...
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-04-27 13:35:11 -07:00
Viktor SöderqvistandMadelyn Olson 15d6dc347f Attempt to deflake 'diskless no replicas drop during rdb pipe' (#3461) 
Increase time to wait for bgsave to start. This wait has been seen
failing in these test cases.

The test case loops with 'no', 'slow', 'fast', 'all' and 'timeout'
replicas, in tests/integration/replication.tcl

Example:

*** [err]: diskless fast replicas drop during rdb pipe in
tests/integration/replication.tcl
    rdb child didn't terminate

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
 2026-04-27 13:35:11 -07:00
BinbinMadelyn OlsonViktor Söderqvist
 		110dc03968 Fix slot-migration-max-failover-repl-bytes unable to accept -1 (#3443) 
In valkey.conf, slot-migration-max-failover-repl-bytes allows setting
to -1 to disable the limit.
```
Setting this to -1 will disable this limit
```

But slot-migration-max-failover-repl-bytes is defined as MEMORY_CONFIG
and memtoull() rejects negative inputs, making it impossible to set the
value to -1 via config file or CONFIG SET.
```
>>> 'slot-migration-max-failover-repl-bytes "-1"'
argument must be a memory value
```

Introduce SIGNED_MEMORY_CONFIG flag for memory configs that also accept
plain negative number. When memtoull() fails and this flag is set, fall back to string2ll()
for parsing. Use ll2string() for CONFIG GET and rewriteConfigNumericalOption() for
CONFIG REWRITE when the value is negative.

Add a serverAssert in initConfigValues() to enforce that PERCENT_CONFIG
and SIGNED_MEMORY_CONFIG are never combined on the same config, since
both use negative values with different semantics.

This means we have had this issue since it was introduced in #1949.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
 2026-04-27 13:35:11 -07:00
RaghavandMadelyn Olson fe11fcb2a8 Skip faster-failover test under TLS (#3444) 
The faster-failover.tcl test starts 12 cluster nodes 5 times with TLS,
which is CPU intensive due to the volume of TLS handshakes. Running
alongside other tests on CI runners causes random I/O timeouts in
unrelated tests.

Tagging it as `tls:skip`

Signed-off-by: nmvk <r@nmvk.com>
 2026-04-27 13:35:11 -07:00
Viktor SöderqvistandMadelyn Olson 99333f9a69 Fix some flaky tests using CLIENT REPLY OFF (#3452) 
Some test cases write thoughsands of commands in a pipeline and
afterwards read the replies. This can lead to TCP ACK being dropped and
the connection broken. CLIENT REPLY OFF prevents this.

"Main db not affected when fail to diskless load" in
cluster/diskless-load-swapdb has been observed to be flaky.

The others are just defensive but they follow the same pattern.

Similar fixes in the past: #3430, #2483.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
 2026-04-27 13:35:11 -07:00
charsyamMadelyn OlsonHarkrishn Patro
 		27f23b4d0e Fix RDB expiry write length and leak when loading zipmap (#3422) 
Fix two things:

1. Incorrect RDB object size reported when writing hash fields with
expiration, because of wrong position of parentheses. Affects the
serialized size reported in DEBUG OBJECT.

2. Memory leak when loading a zipmap (only used in really old RDB
versions).

Signed-off-by: charsyam <charsyam@naver.com>
Co-authored-by: Harkrishn Patro <bunty.hari@gmail.com>
 2026-04-27 13:35:11 -07:00
Nikhil MangloreandMadelyn Olson 8a909a06e6 Increase embedded string threshold from 64 to 128 bytes (#3397) 
This PR increases the embedded string threshold to 128 bytes (2 cache
lines) in order to improve memory overhead. I also fixed the tests that
pertained to embedded values.

Closes #3025

---------

Signed-off-by: Nikhil Manglore <nmanglor@amazon.com>
 2026-04-27 13:35:11 -07:00
Lucas YangandMadelyn Olson 6c0bf7c539 Update maintainer affiliation (#3449) 
Update Lucas Affiliation to Percona

Signed-off-by: Lucas Yang <lucasyonge@gmail.com>
 2026-04-27 13:35:11 -07:00
Joseph HeckMadelyn OlsonSarthak AggarwalLucas Yang
 		24a0721daf Grammar, spelling, and punctuation fixes across the summary fields for Valkey commands. (#3309) 
I was fixing grammar issues in the valkey-swift client
(https://github.com/valkey-io/valkey-swift/pull/357), and would like to
apply the same fixes - missing words, full sentence punctuation, and
imperative verb forms - to the breadth of the commands.

---------

Signed-off-by: Joe Heck <j_heck@apple.com>
Signed-off-by: Joseph Heck <j_heck@apple.com>
Co-authored-by: Sarthak Aggarwal <sarthakaggarwal97@gmail.com>
Co-authored-by: Lucas Yang <lucasyonge@gmail.com>
 2026-04-27 13:35:11 -07:00
Ran ShidlansikandMadelyn Olson 04cf1f8550 Avoid having server.h being included by cli and benchmark (#3420) 
valkey-cli and valkey-benchmark link only a small subset of .o files and
do not include cluster_migrateslots.o. At -O3 this works because the
compiler either inlines or discards the unused static function —
the symbol reference never reaches the linker. At -O0 (no inlining, no
dead-code elimination), the compiler emits the full body of
getClientType into every .o that includes server.h, producing an
unresolved
reference to _isImportSlotMigrationJob at link time.

fix is to avoid including server.h as part of external application
compilation dependency

fixes: https://github.com/valkey-io/valkey/issues/3415

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
 2026-04-27 13:35:11 -07:00
Madelyn Olson f94c1a7edf Fix VLA warning in linenoise and enable -Werror (#3439) 
Replace 'const int seqBufferMaxLength' with a #define to avoid a
variable-length array warning (-Wgnu-folding-constant) in C.

Also add -Werror to the linenoise Makefile so future warnings are caught
at build time.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
 2026-04-27 13:35:11 -07:00
RaghavMadelyn OlsonBinbin
 		da8d047a9b Big endian bitmap byte order mismatch fix (#3401) 
Found while implementing `clusterNodeGetSlotRangeEnd` for `CLUSTERSCAN`
range bounded scanning, which uses the simlilar approach.

When tested on s390x via Docker and QEMU. Slot boundary came as 5440
instead of 5461.

```
127.0.0.1:6001> clusterscan 0-{06S}-0
1) "0-{4HD}-0"
2) 1) "{06S}key1"
127.0.0.1:6001> keyslot 0-{4HD}-0
(error) ERR unknown command 'keyslot', with args beginning with: '0-{4HD}-0'
127.0.0.1:6001> cluster keyslot 0-{4HD}-0
(integer) 5440
127.0.0.1:6001> cluster nodes
08e28d7e8dcfc731ac537d0518bfa32577da6ec7 127.0.0.1:6002@16002 master - 0 1774415944347 2 connected 5461-10922
53f6d4e61eee13ea98441eec05b3c4c95c6c83a4 127.0.0.1:6003@16003 master - 0 1774415943314 3 connected 10923-16383
46664ac624f001cc0708e8ddcfcc9e45e8f444a0 127.0.0.1:6001@16001 myself,master - 0 0 1 connected 0-5460
```

Updating here as it is same approach and does not follow `memrev64ifbe`
followed by `memcpy`

Signed-off-by: nmvk <r@nmvk.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
 2026-04-27 13:35:11 -07:00
Viktor SöderqvistandMadelyn Olson 1ddaefbb93 Fix some flaky tests (#3430) 
Fixing multiple flaky tests.

    slave buffer are counted correctly in tests/unit/maxmemory.tcl
    Memory efficiency with values in range * in tests/unit/memefficiency.tcl

These tests send large numbers of pipelined commands using deferring
clients without reading replies, causing the server's client output
buffer to grow. On slow CI runners, this leads to TCP backpressure and
I/O errors that crash the test runner. Fix: Use CLIENT REPLY OFF to
suppress reply generation, matching the pattern from commit 972f94953c.

---

    Sub-replica reports zero repl offset and rank, and fails to win election
    in tests/unit/cluster/replica-migration.tcl
    New non-empty replica reports zero repl offset and rank, and fails to
    win election in tests/unit/cluster/replica-migration.tcl

In the replica-migration tests, a MOVED errors results in an Tcl
exception. After failover, wait_for_condition blocks issue GET commands
to cluster nodes that may not have fully updated their slot routing. An
unhandled MOVED exception crashes the test runner. Fix: Wrap the
condition in catch so MOVED errors are retried. Also wrap debug prints
in the else clause. Fixes the following tests:

---

    Replica can update the config epoch when trigger the failover -
    automatic in tests/unit/cluster/failover2.tcl

Increase wait timeout for failover expiry. The test waits 10 seconds for
"Failover attempt expired", but the default cluster-node-timeout in
start_cluster is 3000ms, making auth_timeout 6 seconds plus ~3
seconds for failure detection — barely fitting in 10 seconds and failing
on slow CI runners. Fix: Increase wait from 1000×10ms to 1200×50ms
(60 seconds).

---

    dual-channel-replication lazyfree test

The test looks up the replica's main-channel connection id after writing
50MB of data. On slow CI runners, the replica connection may have been
disconnected by the output buffer soft limit (64MB/60s) before the
lookup, causing get_client_id_by_last_cmd to return empty. Two changes:

1. Move the connection id lookup before the write loop, while the sync
   is known to be in progress.
2. Reduce writes from 50 x 1MB to 10 x 1MB. The test only needs enough
   data to exceed the lazyfree threshold (64 blocks ~= 1MB). 10MB is
   sufficient and avoids approaching the output buffer limit.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
 2026-04-27 13:35:11 -07:00
Zhijun LiaoMadelyn OlsonBinbin
 		d7d85ea831 Enhance cluster stale packet detection to prevent sub-replica and empty primary (#2811) 
## Summary
This PR handles a network race condition that would cause a replica to read stale cluster
packet and then incorrectly promote itself to an empty primary within an existing shard.

## Issue
"Migrated replica reports zero repl offset and rank, and fails to win election - sigstop"
the test case in `replica-migration.tcl` is a known example where the network race condition
can occur.

Here's the timeline:
- T1: Slot migration — R7 and R3 both try to replicate from R0. Only R3 succeeds, triggering
  a BGSAVE on R0, while R7 blocks in `receiveSynchronousResponse()`.
- T2: While R7 is blocked, R0 is SIGSTOP'd by the test. R4 wins the election and becomes the
  new primary. R4 sends PINGs to R7 - These PINGs land in R7's kernel TCP receive buffer but
  are not read by R7 yet.
- T3 (5s after T1, due to receiveSynchronousResponse): R7 wakes up:
  - It reads from inbound links and finds out the remote nodes have already closed their end
    (they detected R7 as FAIL), getting "I/O error: connection closed" on each. R7 calls `accept()`
    for the new connections that were established during the block. These connections carry data
    that was sent seconds ago while R7 was dead.
  - R7's outbound links are still valid, so it sends PING to R4.
- T4: R7 receives and processes fresh PONG packet from R4 on outbound link, reconfiguring itself
  to follow R4.
- T5: R7 reads stale PING packet of R4 via inbound link. This packet was generated R4 was following
  R0, so R7 incorrectly believes R4 is still following R0 now. And Reconfiguring itself as a replica
  of R0 from R4.
- T6: R7 finds R0 is FAIL, so it starts an election and wins, and becomes an empty-primary.

## Analyze
So in T4, R4 is the new primary, and R7 is reconfiguring itself as a replica. So in R7's view,
R4 is the primary, and myself (R7) is a replica.

And in T5, there is a stale packet from sender (R4), the stale packet is saying: sender (R4) is
a replica and R0 is the primary.

We originally had a logic for stale packet, meaning we would try to ignore stale packet that
would cause exceptions. So in T5:
- sender_claims_to_be_primary is false since R4 is saying it is a replica.
- sender_last_reported_as_primary is true since in R7's view, R4 is a primary.
- sender_claimed_primary is R0, and sender (R4) and sender_claimed_primary (R0) is in the same shard.
- nodeEpoch(sender_claimed_primary) is R0's epoch. R0 is an old and dead (not yet) primary.
- sender_claimed_config_epoch is R0's epoch since R4 is a replica, and R0 is R4's primary.
- nodeEpoch(sender_claimed_primary) == sender_claimed_config_epoch, so the logic fail and we process
  a stale packet. So in this point, the packet is not a stale packet in R4 and R0's view.
- But it is a stale packet in myself (R7) view. In R7's local view, R4 is the new primary and it
  should have a bigger epoch, that is nodeEpoch(sender) should > sender_claimed_config_epoch.

## Fix
The PR fixes the issue by enhancing the existing guardrail logic against stale packet. Previously
that logic only detects `nodeEpoch(sender_claimed_primary) > sender_claimed_config_epoch` as stale
packet, now it also checks `nodeEpoch(sender) > sender_claimed_config_epoch` to make sure we have
up-to-date primary-replica chain.

Signed-off-by: Zhijun <dszhijun@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
 2026-04-27 13:35:11 -07:00
Viktor SöderqvistandMadelyn Olson 3c9b27b476 Increase timeout in flaky "failover immediately" test case (#3424) 
The test case "The best replica can initiate an election immediately
test" has been failing in CI jobs.

Increase the timeout to account for slow runners.

Old waiting time: 50 seconds. New waiting time: 120 seconds, with
valgrind: 600 seconds.

Intoduced in #2227.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
 2026-04-27 13:35:11 -07:00
Viktor SöderqvistandMadelyn Olson 5b25c93731 Reduce corrupt-dump-fuzzer tests from 10 to 1 minute (#3425) 
In Daily test runs with the `--accurate` flag, the corrupt-dump-fuzzer
test runs for 10 minutes (600 seconds) with "sanitize_dump: no" and then
another 10 minutes with "sanitize_dump: yes". This causes the runner to
time out the whole test job to be aborted with a sigterm from the
runner.

Example:

13697:signal-handler (1774917673) Received SIGTERM scheduling
shutdown...

This change reduces this hard-coded fuzzer run from 10 to 1 minute. We
have many tests jobs so the fuzzer gets plenty of time to run anyway.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
 2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson d598dbb991 Optimize WATCH duplicate key check from O(N) to O(1) using per-db hashtable (#3360) 
Previously, watchForKey() checked for duplicate watched keys by iterating
through the client's entire watched_keys list with O(N) complexity, where
N is the total number of keys watched by the client. So the time complexity
for the WATCH command could be quite poor and become a slow command.

This commit introduces a per-db hashtable (watched_keys_by_db) in the
client's multiState structure to enable O(1) duplicate key detection.
The hashtable is lazily allocated only when the client starts watching
keys, minimizing memory overhead for clients that don't use WATCH.

The per-db hashtable stores watchedKey* directly as the hashtable entry
since it already contains the key, so no custom destructors are needed.
Memory management remains centralized in the watched_keys list.

This optimization is especially beneficial when a client watches many
keys across different databases, as the check no longer scales with
the total watched key count.

This might be a minor scenario, but there's no harm in optimizing it.

There is a test in multi.tcl, before this patch, it took 15s, and after
this patch, it only took 50ms.
```
        set elements {}
        for {set i 0} {$i < 50000} {incr i} {
            lappend elements key-$i
        }
        r watch {*}$elements
        r watch {*}$elements
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 84240f51c6 Handle EAGAIN in clusterWriteHandler (#3421) 
To avoid freeing the cluster link when EAGAIN occurs,
so that we can try again and keeping the send messages.

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-04-27 13:35:11 -07:00
Ran ShidlansikandMadelyn Olson ab7498e020 fix test_entry to consider diffrerent allocator size classes (#3416) 
fixes: https://github.com/valkey-io/valkey/issues/3200

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
 2026-04-27 13:35:11 -07:00
KimHuiSuMadelyn OlsonViktor Söderqvist
 		76b4b4b47e Improve listpack threshold guidance in valkey.conf (#3419) 
Fixes #3299

Add brief guidance in valkey.conf explaining what the listpack
thresholds control and the memory/CPU tradeoff when tuning them.

---------

Signed-off-by: Tarte <emprimula@gmail.com>
Signed-off-by: KimHuiSu <101166683+Tarte12@users.noreply.github.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
 2026-04-27 13:35:11 -07:00
zhenwei piandMadelyn Olson 3b1ae0afd0 Show uname -a in RDMA CI job (#3418) 
The RXE project should keep the same version with the CI machine,
showing uname in RDMA CI job to find out the reason of kmod installing
failure.

Signed-off-by: zhenwei pi <zhenwei.pi@linux.dev>
 2026-04-27 13:35:11 -07:00
Nikhil MangloreandMadelyn Olson 2c5a748ba3 Fix race condition in diskless swapdb RedisModuleEvent_ReplAsyncLoad tests (#3404) 
These failures seem to be attributed to a race condition in the Aborted
test case.

`rdb-key-save-delay` 10000 was being set after `$master exec` triggered
the full resync. Since repl-diskless-sync-delay 0 was set, the master
would immediately start streaming the RDB to the replica once it
reconnected. On the ARM runner, when it's fast enough, the entire RDB
generation and transfer could complete in ~78ms, before the delay was
ever applied. This meant the replica would complete the swap and have
1010 keys instead of the expected 200 and there would be no
async_loading window to observe or abort which led to the failures.

We saw this in the daily test failure logs
```
92948:S * RDB memory usage when created 110.85 Mb
92948:S * Done loading RDB, keys loaded: 1010, keys expired: 0
```

The fix moves `rdb-key-save-delay 10000` to before `$master exec` to
guarantee the delay is in effect on the master before the RDB generation
begins.

Closes #3394, closes #3395.

Signed-off-by: Nikhil Manglore <nmanglor@amazon.com>
 2026-04-27 13:35:11 -07:00
BinbinMadelyn OlsonViktor Söderqvist
 		3d80bf6e6c Do the failover immediately if the replica is the best ranked replica (#2227) 
In #2023 (#2209, etc.), we are exploring ways to make failover faster,
that is, to minimize the delay.

When a node is marked as FAIL and before the failover starts, there is a
delay of 500-1000ms. The original purpose of this delay:

1. Allow FAIL to propagate to at least a majority of the primaries. This
   makes sure they will vote when a replica sends failover auth request.
2. Allow replicas to exchange their offsets, so they will have a correct
   view of their own rank.

We want to minimize this delay while ensuring safety. It is useful for
example in these cases:

1. If there is only one replica, then we don't need any delay, or
2. If there are more replicas, with a fast network, the replicas can
   exchange the offsets very quickly and start the failover within a few
   milliseconds instead of 500-1000ms.

In this PR, when we can be sure that the replica is the best ranked
replica, we let
it initiate a failover immediately and completely remove the delay.

### How to ensure safety?

1. To make sure this replica has the best rank, it only skips the delay
   if it is sure that it have the best rank and that all replicas in the
   same shard agree that the primary is failing. A new flag
   `CLUSTER_NODE_MY_PRIMARY_FAIL` is introduced to indicate that each
   replica has marked its primary as FAIL. If all replicas say that the
   primary is failing, we also know that the offset is not updated,
   because the offset is not incrementing when the primary is failing.
   We can skip the delay only if we have received a message from all
   replicas and they all have set this flag.

2. To make sure the primaries will vote even if they didn't receive the
   FAIL yet, we use the `CLUSTERMSG_FLAG0_FORCEACK` to make sure they
   will vote. This is equivalent to broadcasting a FAIL message to all
   primaries before we broadcast the failover auth request (but
   cheaper).

The race between FAIL (broadcast by A) and AUTH REQUEST (broadcast by R)
is illustrated in the following sequence diagram:

```
A      R        B      C
|      |        |      |
| FAIL |        |      |
|----->| AUTH R.|      |
|      |------->|      |
| FAIL |        |      |
|-------------->|      |
|      | AUTH R.|      |
|      |-------------->|
| FAIL |        |      |
|--------------------->|
```

### Details

This is the how the failover is initiated, with new steps marked with
**(new)**:

1. A majority of primaries have marked another primary as PFAIL.

2. Some nodes counts failure reports and marks the failing primary as
   FAIL. The node
   that detects FAIL broadcasts it to all nodes in the cluster.

3. When a replica receives FAIL (or detects FAIL itself by counting
   PFAIL reports) it schedules a failover:
   
   a. It sets a timeout (500ms + random 0-500ms).
   b. It broadcasts pong to the other replicas in the same shard.   
   c. **(new)** The pong (actually the clusterMsg header) has a new flag
      `CLUSTER_NODE_MY_PRIMARY_FAIL`. When the replicas broadcast pong
      to each other here, this flag is set.

4. **(new)** When the following conditions are met, skip the remaining
   delay and start the failover using AUTH REQUEST with the FORCE ACK
   flag set, that is if
   
   a. a PONG is received from every other replica in the same shard
      (broadcast within the shard) and
   b. all replicas have marked that its primary is FAIL in their last
      message (the new `CLUSTER_NODE_MY_PRIMARY_FAIL` flag is set) and
   c. this is the best replica (rank = 0) and
   d. my replication offset != 0.

5. When the delay has passed and no other replica has initiated
   failover, then initiate failover.

Notes:

* With 3(c), we don't need to wait for FAIL to propagate to all voting
  primaries. At this point, a FAIL has already been broadcast by some
  node, but there is a race so our auth request may arrive to some node
  before the FAIL. Using the FORCE ACK flag ensures the primaries will
  vote for us. (It is equivalent to broacasting another FAIL just before
  broadcasting auth request.)
  
* 4(b) ensures that we have received the replication offset from all
  other replicas and that it's up to date. If a replica says that it's
  primary is failing, it also means that the replication from the
  primary to that replica has stopped.

* 4(c) is to avoid a special bad case. It can happen that not all
  replicas know about each other. In this case, two replicas can think
  they are both the best replica and start the failover at the same
  time. This can already happen without this PR. When it happens, it
  usually means that a new replica has just joined and it has no data
  (offset = 0) and if it wins the election, there is a problem of
  dataloss (discussed and partially mitigated for the replica migration
  case in #885). To avoid this case, skip this fast failover path if the
  replica has offset = 0.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
 2026-04-27 13:35:11 -07:00
ahmadbelbMadelyn OlsonRan Shidlansik
 		75bb8bd3e4 ARM NEON SIMD optimization for pvFind() in vset.c (#3033) 
**Title:** ARM NEON SIMD optimization for pvFind() in vset.c

**Description:**

This PR resolves #2806. Thanks to @ranshid for guidance on testing
methodology and workload design.

### Summary

This PR adds ARM64 NEON SIMD optimization to the `pvFind()` function in
vset.c, which performs linear pointer search in pVector. The pVector is
used internally by vset to track expired fields in hash objects (HFE).
The optimization processes 4 pointers per iteration using 128-bit NEON
vector instructions.


### Implementation Details

- Added `pvFindSIMD_NEON64()` static inline helper function using NEON
intrinsics
- Modified `pvFind()` to use SIMD path when `len >= 8` on ARM64
- Added `#include <arm_neon.h>` guarded by `HAVE_ARM_NEON`
- No changes to function signatures or external behavior
- Scalar fallback remains for non-ARM64 platforms and small vectors

### Benchmark Results

#### Micro benchmark (Apple M4 Pro, 50M iterations)

Scalar version:
```
len16_mid              | len= 16 pos=  8 |    2.8 ns
len16_last             | len= 16 pos= 15 |    5.0 ns
len32_mid              | len= 32 pos= 16 |    5.1 ns
len64_last             | len= 64 pos= 63 |   15.7 ns
len127_last            | len=127 pos=126 |   34.0 ns
len127_notfound        | len=127 pos= -1 |   33.8 ns
```

NEON version:
```
len16_mid              | len= 16 pos=  8 |    2.0 ns | 1.42x
len16_last             | len= 16 pos= 15 |    2.7 ns | 1.85x
len32_mid              | len= 32 pos= 16 |    2.8 ns | 1.79x
len64_last             | len= 64 pos= 63 |    9.2 ns | 1.71x
len127_last            | len=127 pos=126 |   18.1 ns | 1.88x
len127_notfound        | len=127 pos= -1 |   17.8 ns | 1.90x
```

#### Production HFE benchmark (stress profile, 120s duration)

Workload profile:
- 5000 hashes with 64-127 fields each
- Short TTLs (2-10 seconds) to trigger expiration
- 30% deletes, 60% updates (both trigger pvFind)
- 4 concurrent threads

```
Platform        Mode      Avg Time    Throughput   Speedup
-----------     ------    --------    ----------   -------
Apple M4        scalar     94.3 ns   10.61 M/s     baseline
Apple M4        NEON       74.4 ns   13.44 M/s     1.27x (21% faster)
```

SIMD utilization (M4 NEON):
```
- 100% of calls used SIMD path
- 99.3% of elements scanned via SIMD
- 74.7% of matches found in SIMD section
- 25.3% found in scalar tail (last 0-3 elements)
```



### Platform Support

- **ARM64 (aarch64)**: SIMD enabled via `HAVE_ARM_NEON`
- **x86_64 / other**: Falls back to scalar implementation, no behavior
change

---------

Signed-off-by: Ahmad Belbeisi <ahmadbelb@gmail.com>
Signed-off-by: Ahmad Belbeisi <ahmad.belbeisi@tum.de>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
 2026-04-27 13:35:11 -07:00
Alina LiuandMadelyn Olson 178e9e99d2 Fix EntryTest.entryUpdate failure on macOS due to allocator differences (#3398) 
## Problem

The `EntryTest.entryUpdate` unit test fails on macOS with (mentioned in
#3200):

Expected: (entryMemUsage(e10)) < (current_embedded_allocation_size * 3 /
4)
      actual: 48 vs 48

## Root Cause

`entryMemUsage` for embedded entries reflects the actual zmalloc
allocation size, which depends on the platform allocator's bucket sizes.

Valkey's bundled jemalloc is configured with `LG_QUANTUM=3` (8-byte
granularity), giving size classes: 8, 16, 24, 32, 40, 48, 56, 64, ...
However, macOS libc uses 16-byte aligned buckets: 16, 32, 48, 64, 80,
...

The test's value10 (21 chars) produces an entryReqSize of 40 bytes.
Jemalloc has a 40-byte size class, so entryMemUsage returns 40. macOS
rounds up to 48, which equals 3/4 of e9's 64-byte allocation, causing
the strict less-than assertion to fail.

## Fix

Shrink value10 from 21 to 13 characters, reducing entryReqSize from 40
to 32 bytes. Both allocators have a 32-byte bucket, and 32 < 48 holds on
both platforms.

## Test

All tests pass on macOS.

Signed-off-by: Alina Liu <liusalisa6363@gmail.com>
 2026-04-27 13:35:11 -07:00
harrylin98andMadelyn Olson 1de84d9d4f ci: include gtests in code coverage report 
Signed-off-by: harrylin98 <harrylin980107@gmail.com>
 2026-04-27 13:35:11 -07:00
Yang ZhaoandMadelyn Olson 8d062cc469 Fix valkey-cli --cluster del-node for unreachable nodes (#3209) 
The `valkey-cli --cluster del-node` command fails when attempting to
delete unreachable or failed nodes, reporting `No such node ID` even
though the node exists in the cluster topology.
The root cause is the command only loads information about reachable
nodes, causing the lookup to fail. This PR added a new function for
loading all nodes information to solve this.


### Implementation

1. Loading all nodes from gossip:
- Added `clusterManagerLoadAllInfoFromNode()` that loads both reachable
and unreachable nodes from cluster gossip
- Extracts common logic into `clusterManagerLoadInfoCommon()` with an
`include_unreachable` flag
- Keeps the original `clusterManagerLoadInfoFromNode()` unchanged to
avoid affecting existing callers
2. Added success message to be consistent with other cluster commands:
`[OK] Node <id> removed from the cluster.`
3. Added test coverage for `del-node` which previously had none.
4. Load slot information from gossip for unreachable nodes in
`clusterManagerNodeLoadInfo()`
5. Skip unreachable primaries in `clusterManagerNodeWithLeastReplicas()`


### Testing
```
./runtest --single unit/cluster/cli

[ok]: del-node: Cannot delete node with slots (9 ms)
[ok]: del-node: Delete reachable node without slots (23 ms)
[ok]: del-node: Delete unreachable node without slots (1333 ms)
[ok]: del-node: Cannot delete unreachable primary with slots (3368 ms)
```
```
valkey-cli --cluster del-node 127.0.0.1:7000 eb837ea7c48908e5304eafd8b1b3ced57147c448

Could not connect to Valkey at 127.0.0.1:7002: Connection refused
>>> Removing node eb837ea7c48908e5304eafd8b1b3ced57147c448 from cluster 127.0.0.1:7000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> WARNING: Could not connect to node 127.0.0.1:7002, unable to send CLUSTER RESET.
[OK] Node eb837ea7c48908e5304eafd8b1b3ced57147c448 removed from the cluster.
```

### Behavior change

Before
```
$ valkey-cli --cluster del-node <entry-node-ip>:<entry-node-port> <failed-node-id>
Could not connect to Valkey at <target-node-ip>:<target-node-port>: Connection refused
>>> Removing node <id> from cluster <entry-node-ip>:<entry-node-port>
[ERR] No such node ID <id>
```
After
```
$ valkey-cli --cluster del-node <entry-node-ip>:<entry-node-port> <failed-node-id>
Could not connect to Valkey at <target-node-ip>:<target-node-port>: Connection refused
>>> Removing node <id> from cluster <entry-node-ip>:<entry-node-port>
>>> Sending CLUSTER FORGET messages to the cluster...
>>> WARNING: Could not connect to node <target-node-ip>:<target-node-port>, unable to send CLUSTER RESET.
[OK] Node <id> removed from the cluster.
```

### Related Issue
Fixes https://github.com/valkey-io/valkey/issues/3208

---------

Signed-off-by: Yang Zhao <zymy701@gmail.com>
 2026-04-27 13:35:11 -07:00
Roshan KhatriandMadelyn Olson b454c95776 Upload all benchmark artifacts including server logs (#3388) 
Upload the entire results directory instead of only metrics JSON files.
This includes server logs which are useful for debugging benchmark
failures.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
 2026-04-27 13:35:11 -07:00
Roshan KhatriandMadelyn Olson 2eadce4ff5 Pin workflow pip/go/npm dependencies for OpenSSF compliance (#3276) 
Pin package manager dependencies in CI workflows to improve the Pinned-Dependencies
score in OpenSSF Scorecard.

Changes:
- benchmark-on-label.yml, benchmark-release.yml: add `--require-hashes`
  to `pip install` adding on valkey-perf-benchmark repo:
  https://github.com/valkey-io/valkey-perf-benchmark/pull/44
- ci.yml: pin `yamlfmt` to `v0.21.0` instead of `@latest`
- reply-schemas-linter.yml: use npm ci with `package-lock.json` instead
  of unpinned npm install, package files in `utils/reply-schema-linter/`

Signed-off-by: Roshaan Khatri <rvkhatri@amazon.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
 2026-04-27 13:35:11 -07:00
Roshan KhatriandMadelyn Olson 218ad732a0 fix benchmark queue and reduce the total duration (#3387) 
Previously, our workflow used a global concurrency group, which
effectively limited execution to one running job and one pending job.
Any additional requests were automatically canceled, preventing a true
queue from forming.

We are now shifting to a model where we remove the concurrency
restriction and allow jobs to queue directly on the self-hosted runner.
This enables multiple workflow runs to be accepted and queued instead of
being dropped.

While GitHub can accept workflow triggers at a high rate (e.g., hundreds
per minute), the actual execution is still constrained by runner
capacity, in our case, a single runner processing one job at a time.

However, queued jobs are subject to GitHub’s 24-hour timeout policy.
This means any job that waits in the queue for more than 24 hours before
starting will be automatically canceled (timedout).

In practical terms, this approach improves reliability by eliminating
premature cancellations, but the effective queue size is still bounded
by how many jobs the runner can process within a 24-hour window. we
could increase the number of runners to run these in parallel.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
 2026-04-27 13:35:11 -07:00
RaghavandMadelyn Olson 258060ddbb CLUSTERSCAN MATCH pattern maps to a specific slot optimizations (#3380) 
When a MATCH pattern maps to a specific slot, `CLUSTERSCAN` can skip
directly to that slot
instead of walking through all slots one by one.

- On `cursor 0`, starts directly at the matching slot
- If cursor is behind the matching slot, jumps forward
- If cursor is ahead of the matching slot, we conclude the scan as we
cannot match keys
- If both SLOT and MATCH are provided but target different slots,
returns 0 immediately

Signed-off-by: nmvk <r@nmvk.com>
 2026-04-27 13:35:11 -07:00
Harry LinandMadelyn Olson 9142991068 Ensure the daily workflow uses gtest-parallel to run unit tests in isolation (#3375) 
The daily workflow was directly invoking the `valkey-unit-gtests` executable.
The intended invocation is to use `gtest-parallel` to ensure that the tests are executed in isolation.

Signed-off-by: harrylin98 <harrylin980107@gmail.com>
 2026-04-27 13:35:11 -07:00
Ran ShidlansikandMadelyn Olson 6c4bb03dc5 remove duplicated lline (#3379) 
Probably added by mistake during some merge of #1566

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
 2026-04-27 13:35:11 -07:00
Harkrishn PatroandMadelyn Olson ca82cdde9c Add AGENTS.md file for agentic coding assistant steering (#3371) 
Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
 2026-04-27 13:35:11 -07:00
Jacob MurphyandMadelyn Olson 0a33b19ea2 Add design-docs folder and README. (#3300) 
These are forked from the RFC instructions created by
@zuiderkwast and @hpatro in
https://github.com/valkey-io/valkey-rfc/pulls/1 and
https://github.com/valkey-io/valkey-rfc/pulls/6. It also includes the Atomic Slot Migration design to bootstrap the folder.

---------

Signed-off-by: Jacob Murphy <jkmurphy@google.com>
 2026-04-27 13:35:11 -07:00
secwallandMadelyn Olson 4147dd14f8 Make scripting debug test skippable (#3368) 
I'm developing a module to provide luajit as lua execution engine.
See #1229 for details.

Unfortunately said module doesn't support debugging yet. So in order to
test it with valkey tests scripting debug tests need to be skipped. This
patch makes an anonymous test skippable by name.

Signed-off-by: secwall <secwall@yandex-team.ru>
 2026-04-27 13:35:11 -07:00
Harkrishn PatroandMadelyn Olson c91cd8c926 Make macOS leaks check skippable (#3370) 2026-04-27 13:35:11 -07:00
BinbinandMadelyn Olson 347e1168b5 Fix incorrect memory overhead calculation for watched keys (#3359) 
The multiStateMemOverhead() function was incorrectly calculating the
memory overhead for watched keys. It used sizeof(c->mstate->watched_keys)
which is the size of the list structure itself, instead of sizeof(watchedKey)
which is the actual per-key overhead.

This was introduced in #1405.

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-04-27 13:35:11 -07:00
Madelyn OlsonandGitHub 3f9fc9ff69 Update version to 9.1.0-rc1 and add release notes (#3333) 
Add release notes for Valkey 9.1 for review and bumping the version so we can stamp it.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
 2026-03-16 16:39:41 -07:00
Sarthak AggarwalandGitHub 803fc6e7c6 Weekly tests branches are not honored on scheduled workflow (#3340) 
`weekly.yml` calls `daily.yml` with `use_git_ref` set to each release
branch (for example 7.2). But the checkout logic in `daily.yml` only
used `inputs.use_git_ref` when `github.event_name` was
`workflow_dispatch` or `workflow_call`. otherwise it fell back to
`github.ref`.

For reusable workflows, GitHub keeps the caller workflow’s github
context. That means when `weekly.yml` is triggered by schedule, the
called `daily.yml` still sees `github.event_name == 'schedule'` and
`github.ref` for the caller branch (unstable). As a result, jobs labeled
as release-branch runs could still check out unstable.

Added a guard for Gtest Unit Tests. It will skip the job if gtest is not
available / supported.

Run with CI Issue:
https://github.com/valkey-io/valkey/actions/runs/22815380713

---------

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
 2026-03-13 14:50:00 -07:00
nesty92andGitHub b3ae643246 Optimize streams range hot path (#3002) 
This PR improves stream performance in the range iteration and reply
generation paths, benefits xadd, xrange, xrevrange, xreadgroup.

- ull2string memcpy optimization
- streamID struct + streamCompareID
- streamID2string + reply path
- getClientType inline + cache locality

Inspired by the high-level description (not the code) of
https://github.com/redis/redis/pull/14480.

---------

Signed-off-by: Ernesto Alejandro Santana Hidalgo <ernesto.alejandrosantana@gmail.com>
 2026-03-13 16:33:49 +01:00
Rain ValentineandGitHub 5a2509b743 Fix OOM aborts in large-memory ASAN tests on GitHub runners (#3263) 
Carries on from where #3161 left off. The test-sanitizer-address-large-memory
jobs were being OOM-killed on GitHub-hosted runners (15.6GB RAM) due to
ASAN's 2-3x memory overhead.

Changes:
- Skip 4GB quicklist compression test under ASAN (requires ~16-24GB with
dual buffers + ASAN overhead)
- Reduce integration test sizes from 5GB to 4.1GB (preserves >4GB 32-bit
boundary coverage)
- Reduce XADD iterations from 10 to 3
- Add memory monitoring to track minimum free memory during CI runs

Signed-off-by: Rain Valentine <rsg000@gmail.com>
 2026-03-12 12:33:33 +08:00
Rain ValentineandGitHub dc31c5f91a [DEFLAKE] Deflake replica selection test by relaxing cluster configurations (#3261) 
The "New Master down consecutively" test was sometimes failing under
Valgrind by timing out. The new overrides match those used for the
cluster in the first part of the file - see #2672

Under Valgrind's 10-20x slowdown, a single failover requiring ~15
seconds of server time can exceed the test's 100-second wall-clock wait.

Error text:
```
*** [err]: New Master down consecutively in tests/unit/cluster/slave-selection.tcl
No failover detected when master 12 fails
```

Daily run failure:
https://github.com/valkey-io/valkey/actions/runs/22421982161/job/64921545936#logs

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
Signed-off-by: Rain Valentine <rainval@amazon.com>
 2026-03-11 17:52:02 -07:00
Ray CaoandGitHub d9b200a02e Reuse RDB file as AOF preamble on replica after disk-based full sync (#1901) 
When a replica finishes a **disk-based** full sync and both `appendonly
yes` and `aof-use-rdb-preamble yes` are enabled, this PR reuses the
received `dump.rdb` as the new AOF base file, instead of triggering
`BGREWRITEAOF`.

This avoids generating an almost identical base snapshot and removes
redundant CPU/IO work.

Before:

1. Replica receives RDB via disk-based full sync
2. Replica loads RDB into memory
3. Replica may delete synced RDB (`rdb-del-sync-files`)
4. `BGREWRITEAOF` runs and generates a new base snapshot
5. New snapshot becomes AOF base

After:

1. Replica receives RDB via disk-based full sync
2. Replica loads RDB into memory
3. Synced RDB is renamed to the new AOF base file
4. A new incremental AOF is created
5. No `BGREWRITEAOF` is needed

Tests:

- [x] Disk-based sync + preamble on: RDB reused as AOF base
- [x] New writes after sync: incr AOF works correctly
- [x] Replica restart: reused AOF loads correctly
- [x] Disk-based sync + preamble off: fallback to `BGREWRITEAOF`
- [x] Diskless sync + preamble on: fallback to `BGREWRITEAOF`
- [x] Diskless sync with stale local `dump.rdb`: stale file is not
       reused

---------

Signed-off-by: Ray Cao <zisong.cw@alibaba-inc.com>
 2026-03-11 18:28:26 +01:00
BinbinandGitHub df1738b4d7 Add availability-zone to the reply schema for CLUSTER SHARDS/SLOTS (#3352) 
`availability_zone` was added to the HELLO command in #1487, and
it was missing the reply schema and the test was wrongly marked
with `logreqres:skip`.

`availability-zone` was added to CLUSTER SHARDS and CLUSTER SLOTS
commands in #3156, and it was missing the reply schema.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-03-11 15:59:55 +01:00
Sarthak AggarwalandGitHub 8848a9ca9c Deflake io-threads test by generating enough work (#3354) 
Fixes the failing run:

```
[err]: Force the use of IO threads and assert active IO thread usage in tests/unit/io-threads.tcl
Expected '0.000000' to be more than '0' (context: type source line 49 file /Users/runner/work/valkey/valkey/tests/unit/io-threads.tcl cmd {assert_morethan $used_active_time 0} proc ::test)
```

The change deflakes the test by generating enough I/O-thread work.

The test previously created 16 deferred clients but only sent 15 total
`INCR` commands. On fast runners, that burst could be short enough that
`used_active_time_io_thread_*` was still reported as `0.000000`, causing
the test to fail even though I/O threads had run.

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
 2026-03-11 12:39:40 +01:00
skyfireleeGitHubMadelyn Olson
 		ff1c26159a Add unsigned 64-bit numeric config in module API (#1546) 
- Supports `VALKEYMODULE_CONFIG_UNSIGNED` and `UNSIGNED_CONFIG` for a
   wider range of configurations.
- Supported the API for `ValkeyModule_RegisterUnsignedNumericConfig`
- `string2ull` method supports the length parameter, making it more
   secure

---------

Signed-off-by: artikell <739609084@qq.com>
Signed-off-by: skyfirelee <739609084@qq.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
 2026-03-11 12:31:46 +01:00
Daniel LemireandGitHub eed79cb6cc Replace fast_float (C++) with ffc.h (#3329) 
There is now a port of fast_float in C. So instead of having an optional
fast_float dependency, we can just use ffc instead, unconditionally.

https://github.com/kolemannix/ffc.h

It is a high quality port. The performance should be the same or
improved.

Note : I am the maintainer and main author of fast_float.

---------

Signed-off-by: Daniel Lemire <daniel@lemire.me>
 2026-03-11 12:26:44 +01:00
Remi ColletandGitHub e00129eb87 Inherit LDFLAGS for TLS and RDMA modules (#3344) 
With current Makefile, `LDFLAGS` are not used for modules.

This results in security options not applied.

```
$ annocheck /usr/lib64/valkey/modules/rdma.so 
annocheck: Version 12.99.
Hardened: rdma.so: FAIL: bind-now test because not linked with -Wl,-z,now 
Hardened: Rerun annocheck with --verbose to see more information on the tests.
Hardened: rdma.so: Overall: FAIL.
```

With this patch

```
$ annocheck /usr/lib64/valkey/modules/rdma.so 
annocheck: Version 12.99.
Hardened: rdma.so: PASS.
```

Signed-off-by: Remi Collet <remi@remirepo.net>
 2026-03-11 11:40:15 +01:00
BinbinandGitHub 9239bf28db Optimize SET key value EX/PX/EXAT ttl to reduce calls of rewriteClientCommandVector (#3279) 
If the command is in the form of "SET key value EX/PX/EXAT ttl",
then we don't need to rewrite the entire command vector.

In addition to saving the rewriting of the command vector, we also
save one command table lookup in this case ince we don't need to
lookup SET command again.

We already have many relevant test coverages in expire.tcl. This
PR does not change anything around the propagate.

Somehow like 8c03ef7d06.

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-03-11 10:35:14 +08:00
BinbinandGitHub 837a48cef1 Fix timing issue in RDB abort test (#3343) 
In flushall we will call killRDBChild to kill the RDB child, but
we need to wait for checkChildrenDone and waitpid to kick in, so
that we can call resetChildState to reset the state. The previous
wait_for_condition 10 100 (1s) might not be enough, changed it to
wait_for_condition 50 100 (5s).
```
[err]: Test FLUSHALL aborts bgsave in tests/integration/rdb.tcl
bgsave not aborted
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-03-11 10:34:35 +08:00
Sarthak AggarwalGitHubViktor Söderqvist
 		d3240a9b8d Fix ZDIFF algorithm 2 memory leak on early exit (#3342) 
`zdiffAlgorithm2()` can break out early once the destination cardinality
reaches zero. In that path, a temporary SDS created by `zuiSdsFromValue()`
was left dirty and never released, because that cleanup normally happens on
the next iterator step.

This patch explicitly discards the dirty value before the early `break`.
```
==11724== 11 bytes in 1 blocks are definitely lost in loss record 36 of 1,442
==11724==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==11724==    by 0x2FA974: ztrymalloc_usable_internal (zmalloc.c:156)
==11724==    by 0x2FAAEA: zmalloc_usable (zmalloc.c:200)
==11724==    by 0x282D25: _sdsnewlen (sds.c:102)
==11724==    by 0x2830B2: sdsnewlen (sds.c:169)
==11724==    by 0x2DB537: zuiSdsFromValue (t_zset.c:2290)
==11724==    by 0x2DBE00: zdiffAlgorithm2 (t_zset.c:2502)
==11724==    by 0x2DC085: zdiff (t_zset.c:2568)
==11724==    by 0x2DD01F: zunionInterDiffGenericCommand (t_zset.c:2817)
==11724==    by 0x2DD574: zdiffCommand (t_zset.c:2898)
==11724==    by 0x29F5AE: call (server.c:3883)
==11724==    by 0x2A1598: processCommand (server.c:4569)
```

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
Signed-off-by: Sarthak Aggarwal <sarthakaggarwal97@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
 2026-03-11 10:34:15 +08:00
bandalgomsuandGitHub cd67bc9494 Add availability-zone field support to CLUSTER SHARDS / CLUSTER SLOTS (#3156) 
Implemented a way to propagate az through gossip and obtain az
information
through the CLUSTER SHARDS and CLUSTER SLOTS commands. It will only be
displayed if the node is configured with it.

Closes #3110

---------

Signed-off-by: Su Ko <rhtn1128@gmail.com>
 2026-03-10 19:56:05 +01:00
BinbinandGitHub 5d27f8335a New MSETEX command allows to set an expiration time with MSET (#3121) 
It's a combination of MSET and EXPIRE, which allows us to set an
expiration
time simultaneously with the MSET operation.

Syntax for the new MSETEX command:
```
/* MSETEX numkeys key value [key value ...] [NX | XX]
 *     [EX seconds | PX milliseconds |
 *      EXAT seconds-timestamp | PXAT milliseconds-timestamp | KEEPTTL] */
```

The MSETEX command supports a set of options (like SET command):
- EX seconds -- Set the specified expire time, in seconds (a positive
integer).
- PX milliseconds -- Set the specified expire time, in milliseconds (a
positive
  integer).
- EXAT timestamp-seconds -- Set the specified Unix time at which the
keys will
  expire, in seconds (a positive integer).
- PXAT timestamp-milliseconds -- Set the specified Unix time at which
the keys
  will expire, in milliseconds (a positive integer).
- KEEPTTL -- Retain the time to live associated with the keys.
- NX -- Only set the keys if all keys do not already exist.
- XX -- Only set the keys if all keys already exists.

Return Value (like MSETNX command):
- Integer reply: 0 if no key was set due to NX or XX options.
- Integer reply: 1 if all the keys were set.

Closes #2592

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-03-10 19:09:12 +02:00
cjx-zarandGitHub 8c0c6b2044 fix alignment different between c and c++ in gtest (#3331) 
I found that on 32-bit platforms, the gtest fails if I add a declaration
`int aof_rewrite_for_replication;` between
https://github.com/valkey-io/valkey/blob/971e5f5c30a27bf73d3ad59a8b24c7f3128db128/src/server.h#L1843
and
https://github.com/valkey-io/valkey/blob/971e5f5c30a27bf73d3ad59a8b24c7f3128db128/src/server.h#L2108
The crash is caused by
https://github.com/valkey-io/valkey/blob/971e5f5c30a27bf73d3ad59a8b24c7f3128db128/src/unit/wrappers.h#L40
stripping the _Atomic qualifier in C++ builds. This makes struct
valkeyServer have different alignment/field offsets in C-compiled code
(libvalkey.a) vs C++-compiled tests (gtest). As a result, fields after
these atomics are accessed at different offsets, leading to corrupted
reads/writes and a segfault.

I added VALKEY_ATOMIC, and it works as follows:
- C builds (production): VALKEY_ATOMIC(uint64_t) -> _Atomic uint64_t
(unchanged behavior)
- C++ builds (tests via wrappers.h): VALKEY_ATOMIC(uint64_t) ->
alignas(8) uint64_t (preserves 8-byte alignment)
- All existing atomic_load_explicit / atomic_store_explicit calls in C
code continue to work because _Atomic is present in C mode
- C++ test code only does direct field access (never uses atomic
operations), so alignas(N) type is sufficient

---------

Signed-off-by: cjx-zar <jxchenczar@foxmail.com>
 2026-03-10 07:02:24 -07:00
Harkrishn PatroandGitHub 1303dc2fbb Use dictGetSomeKeys instead of dictGetRandomKey for gossip node population (#3258) 
Observation around `dictGetSomeKeys(N)` performance is that it is around
10x faster than randomly picking `N` elements over N*3 attempts.
Further, this avoids chances of not picking N elements due to duplicate
element selection with individual random pick. Overall, it should help
picking gossip node(s) for cluster bus message faster.

---------

Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
 2026-03-10 01:48:40 -07:00
Sarthak AggarwalandGitHub 259921486b [flaky-tests] atomically snapshot dual-channel memory stats (#3336) 
This fixes a flaky dual-channel replication integration test:
https://github.com/valkey-io/valkey/actions/runs/22810251608/job/66165776198#step:8:7701

`INFO memory` field `used_memory_overhead` and `MEMORY STATS` field
`overhead.total` can change during dual-channel sync if replica's
pending replication buffer is still changing. This is probably more
visible in slower environments.

The test now collects `INFO` and `MEMORY STATS` in a single `MULTI/EXEC`
on both the primary and replica, so the compared values come from the
same snapshot.

Passing here:
https://github.com/sarthakaggarwal97/valkey/actions/runs/22864585326/job/66327772967

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
 2026-03-09 15:45:29 -07:00
Harkrishn PatroandGitHub fa232d72bb Make the number of healthy nodes in gossip section configurable (#2746) 
This PR introduces a new config `cluster-message-gossip-perc` which
allows an operator to modify the amount of gossip node information to be
sent per ping/pong/meet message. It can be modified dynamically (Related
to https://github.com/valkey-io/valkey/issues/2291). The default value
is 10% i.e. 10% of peer node information would be gossiped along with
each ping/pong/meet packet. Users can tune this configuration, setting
the value higher allows faster information dissemination whereas setting
it lower would lead to direct PING messages if no information was
received about a node with the `server.cluster_node_timeout/2` period.

Note: the behavior for partially failed gossip nodes still remains
intact where all the `pfail` nodes are part of the message for faster
propagation of information and faster transition of `PFAIL` to `FAIL`.

---------

Signed-off-by: Harkrishn Patro <bunty.hari@gmail.com>
 2026-03-09 14:57:58 -07:00
Roshan KhatriandGitHub c16c83bd34 Adds cluster benchmark support for benchmark-on-label (#3338) 
Now we will be able to add a `run-cluster-benchmark` label to run a
benchmark with cluster-mode enabled valkey-server

It will use the config
https://github.com/valkey-io/valkey/blob/unstable/.github/benchmark_configs/benchmark-config-arm.json
modified for for cluster mode with a single clustermode enabled instance of
valkey.

It uses the same single instance for the benchmark as for run-benchmark.
 
If both labels are used, they are sequential in the same concurrency group `group:
ec2-al-2023-pr-benchmarking-arm64`.

---------

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
 2026-03-09 22:48:29 +01:00
curious-george-rkGitHubcurious-george-rk
 		2b594a69f0 Replace long long timestamp references with more specific mstime_t/ustime_t (#3252) 
Addresses issue #2350

As noted in the issue, much of the expire code uses raw long long for
timestamps, which provides no semantic meaning about the unit or purpose
of the value. Valkey already defines mstime_t (milliseconds) and
ustime_t (microseconds) typedefs — this PR replaces bare long long
declarations with the appropriate typedef wherever the value represents
an expiration timestamp or time duration.

This PR only fixes a small subset in the codebase, but it is an
incremental step toward fully replacing the bare long long references.

---------

Signed-off-by: curious-george-rk <r.ebu@gmail.com>
Co-authored-by: curious-george-rk <r.ebu@gmail.com>
 2026-03-09 20:03:26 +02:00
eifrah-awsandGitHub ed85c734c2 Changed LUA module dependency (#3325) 
Before this change, if you built `valkey-server` (e.g. `make
valkey-server`) the LUA module was not built. With this change, the LUA
module is now a direct a dependency of `valkey-server`
- (unless `BUILD_LUA=no` is passed)

Added some colors to the Lua module & Lua lib `Makefile`s to they blend
nicely in the build output.

Signed-off-by: Eran Ifrah <eifrah@amazon.com>
 2026-03-09 20:01:29 +02:00
RaghavGitHubViktor SöderqvistMadelyn Olson
 		bb89628d19 CLUSTERSCAN Command (#2934) 
Implemented CLUSTERSCAN command for topology-aware scanning

Unlike `SCAN` which is local to a single node, `CLUSTERSCAN` provides a
mechanism that helps clients iterate across slot boundaries and handles
`MOVED` redirections.

**Key details**

* Global cluster iteration via `fingerprint-{hashtag}-cursor`
* Scan one slot at a time
* Start the CLUSTERSCAN with 0
* SLOT argument for parallel scanning of multiple slots
* Re-use scanGenericCommand for the response

**Cursor format:** `fingerprint-{hashtag}-localcursor`
 - Fingerprint is a hash of the node's DB seed that identifies the
   current memory layout. On mismatch, scan restarts from cursor 0
   rather than returning an error.
 - Fingerprint 0 indicates a cross slot cursor (e.g., initial cursor
   or slot transition) where validation is skipped.
 - Hashtag encodes the target slot
 - Local cursor tracks position within the slot

**Usage:**

```
CLUSTERSCAN <cursor> [MATCH pattern] [COUNT count] [TYPE type] [SLOT number]
```

```
  CLUSTERSCAN 0                    # Start scanning from slot 0
  CLUSTERSCAN <cursor>             # Continue from cursor
  CLUSTERSCAN 0 SLOT 1000          # Start scanning specific slot
  CLUSTERSCAN <cursor> MATCH user:* COUNT 100
```

---------

Signed-off-by: nmvk <r@nmvk.com>
Signed-off-by: Raghav <r@nmvk.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
 2026-03-09 07:41:36 -07:00
BinbinandGitHub 971e5f5c30 Update cluster config file when NOFAILOVER changes (#3280) 
Currently after changing cluster-replica-no-failover, we will
call clusterUpdateMyselfFlags to trigger CLUSTER_TODO_SAVE_CONFIG.
But in gossip when the sender's NOFAILOVER changes, we won't
trigger the save, this cause nodes.conf to not save the latest
nofailover flag.

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-03-08 00:01:55 +08:00
Daniil KashapovandGitHub 9568ee4f05 Fixed point arithmetic for x86 TSC (#3314) 
Replace integer mono_ticksPerMicrosecond with fixed-point arithmetic in
x86 TSC monotonic clock for better accuracy (and performance because of
multiplication + shift instead of division).
Calibration now uses double precision to compute multiplier.

**Comparison with TSC @ 2400.5 ticks/us**

**Old Method:**
Calibration: 2400 (truncated from 2400.5)
Converting 1 second of ticks: 2,400,500,000 / 2400 = 1,000,208 us
Error: +208 us per second

**New Method:**
Calibration: sample_ticks_per_us = 2400.5 (double stores exactly in this
case)
Multiplier: 2^24 / 2400.5 = 6989.942 -> stored as 6989 in uint64_t
Converting 1 second: (2,400,500,000 * 6989) >> 24 = 999,992 us
Error: -8 us per second

---------

Signed-off-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
 2026-03-06 17:57:03 +01:00
Yang ZhaoandGitHub 653e330af4 Recommend passwordless users for mTLS certificate-based authentication (#3311) 
Closes https://github.com/valkey-io/valkey/issues/3286

---------

Signed-off-by: Yang Zhao <zymy701@gmail.com>
 2026-03-06 14:39:05 +01:00
Björn SvenssonandGitHub f6ac10dcef Use ar archiver installed by brew in CI build-macos-latest (#3317) 
Since there is some mismatch between the already installed `ar` tool on
a macOS runner
and Clang 22, installed by brew; lets use the brew installed `llvm-ar`.

Expected to fix the issue in CI job `build-macos-latest`.

---------

Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
 2026-03-06 12:37:29 +01:00
BinbinandGitHub fe7e4392ef Fix timing issue in LATENCY GRAPH test (#3265) 
In #3260 we try to deflake it by using a assert_range, but the upper
limit of the range is low so the test is still flaky. Increase the upper
limit to 200ms more than the expected latency, e.g. from 550 to 700 when
the expected latency is 500.

```
*** [err]: LATENCY GRAPH can output the event graph in tests/unit/latency-monitor.tcl
Expected '625' to be between to '450' and '550' (context: type source line 143 file /Users/runner/work/valkey/valkey/tests/unit/latency-monitor.tcl cmd {assert_range $high 450 550} proc ::test)
```

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-03-05 20:03:32 +01:00
Alon ArenbergGitHubBinbin
 		ffaa9ad1d7 Let script continue if busy-reply-threshold is zero (#3307) 
Before we added LUA as a module we had a logic to NOT register the
luaHook when the value of `busy-reply-threshold` config is set to 0.

Now we ALWAYS register the hook and in order to keep aligned with
old behavior we will let the execution of the script continue from
the interrupt hook when `busy-reply-threshold` config is set == 0.

And in value.conf, we are saying the config can be negative, but in
fact the config is minimum at 0, fix the valkey.conf as well.
```
# The default is 5 seconds. It is possible to set it to 0 or a negative value
# to disable this mechanism (uninterrupted execution)
```

It was introduced in #2858.

Signed-off-by: Alon Arenberg <alonare@amazon.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
 2026-03-05 23:52:47 +08:00
Sarthak AggarwalandGitHub 255b827f35 fix Codecov v5 input from file to files (#3308) 
This PR fixes a Codecov workflow misconfiguration introduced when
upgrading codecov/codecov-action from v4 to v5 (in #3185).
In v5, the action expects files (plural), but the workflow still used
file.

The coverage shown is 0 right now:
https://app.codecov.io/gh/valkey-io/valkey


Documentation from -
https://github.com/codecov/codecov-action/tree/v5?tab=readme-ov-file#arguments

```
The following arguments have been changed

    file (this has been deprecated in favor of files)
    plugin (this has been deprecated in favor of plugins)

```

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
 2026-03-05 13:49:38 +01:00
Ran ShidlansikandGitHub aa06454d3f Fix zset to operate defrag scan lexicographically (#3316) 
as part of #2508 we introduce a defrag optimization to avoid matching
the replaced defrag entry.
Even though defrag replacements does not impact the skip list order,
when scores are equal, we MUST compare elements lexicographically to
maintain correct skip list ordering.
Otherwise we might miss locating the entry.

---------

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
 2026-03-05 13:51:47 +02:00
Roshan KhatriandGitHub 3550a651fd Fix compatibility for OpenSSL < 3.0 and Almalinux version mismatch for daily tests (#3303) 
`SSL_get0_peer_certificate()` was introduced in OpenSSL 3.0. The recent
commit 136852efc (Support TLS authentication using SAN URI) used it in
`tlsGetPeerUser()` without a version guard, breaking builds against
`OpenSSL 1.1.x.`

Use `SSL_get_peer_certificate()` on OpenSSL < 3.0 with the corresponding
`X509_free()` since the older API increments the reference count.

Fixes build failure: implicit declaration of function
`SSL_get0_peer_certificate [-Werror=implicit-function-declaration]`

Also fixes the version mismatch for almalinux 9 daily tests.
Closes #3304.

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
 2026-03-05 19:34:17 +08:00
Kurt McKeeandGitHub 276cbd7c43 Update the CodeQL action to resolve v3 deprecation warnings (#3310) 
The CodeQL workflow is currently throwing a deprecation warning
regarding use of v3.

> CodeQL Action v3 will be deprecated in December 2026. Please update
all occurrences of the CodeQL Action in your workflow files to v4.

This PR introduces the following changes:
* References to CodeQL v3 have been updated to the SHA of the latest
CodeQL release, [v4.32.5].

Signed-off-by: Kurt McKee <contactme@kurtmckee.org>
 2026-03-05 10:55:47 +08:00
Harry LinandGitHub ef1da0f81c Fix compatibility issue for death test with Valgrind (#3301) 2026-03-03 19:03:57 -08:00
Yang ZhaoandGitHub 136852efcc Support TLS authentication using SAN URI (#3078) 
Closes https://github.com/valkey-io/valkey/issues/3077

### Overview 
URI in SAN is used to represent client identities in modern mTLS
deployments where CN may be empty or deprecated. See the
https://github.com/valkey-io/valkey/issues/3077#issuecomment-3775615304
for more details.
When `tls-auth-clients-user URI` is configured, during the TLS
handshake, the server iterates through the URIs in the client
certificate and authenticates the client as the first enabled user whose
name matches one of those URIs.

### Implementation
- Introduced a new value `URI` for `tls-auth-clients-user`
- Added new function `getCertSanUri` that:
- Extracts URI entries from the certificate's SAN extension
- Checks each URI against existing Valkey users
- Returns the first URI that matches an enabled user
- Renamed `getCertFieldByName` → `getCertSubjectFieldByName` for clarity
- Modified `tlsGetPeerUsername` to support both CN and URI
authentication modes

### Example behavior
Common setup
``` 
# client certificate X509v3 Subject Alternative Name
URI:urn:valkey:user:first, URI:urn:valkey:user:second

# valkey.conf
tls-auth-clients-user URI
hide-user-data-from-log no
```
Use case 1: multiple enabled users
```
user urn:valkey:user:first on >clientpass allcommands allkeys
user urn:valkey:user:second on >clientpass allcommands allkeys

39762:M 26 Jan 2026 22:06:25.122 - TLS: Auto-authenticated client as urn:valkey:user:first
```
Use case 2: first URI disabled, second enabled
```
user urn:valkey:user:first off >clientpass allcommands allkeys
user urn:valkey:user:second on >clientpass allcommands allkeys

39792:M 26 Jan 2026 22:07:08.006 - TLS: Auto-authenticated client as urn:valkey:user:second
```
Use case 3: all matching users disabled or no matching user
```
user urn:valkey:user:first off >clientpass allcommands allkeys
user urn:valkey:user:second off >clientpass allcommands allkeys

39812:M 26 Jan 2026 22:07:34.174 * TLS: No matching user found in certificate SAN URI fields
127.0.0.1:6379> acl whoami
"default"
127.0.0.1:6379> acl log
1)  1) "count"
    2) (integer) 1
    3) "reason"
    4) "tls-cert"
    5) "context"
    6) "toplevel"
    7) "object"
    8) ""
    9) "username"
   10) "urn:valkey:user:second"
   11) "age-seconds"
   12) "17.381"
   13) "client-info"
   14) "id=3 addr=127.0.0.1:57236 laddr=127.0.0.1:6379 fd=8 name= age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=NULL user=default redir=-1 resp=2 lib-name= lib-ver= tot-net-in=0 tot-net-out=0 tot-cmds=0"
   15) "entry-id"
   16) (integer) 0
   17) "timestamp-created"
   18) (integer) 1771963041866
   19) "timestamp-last-updated"
   20) (integer) 1771963041866
127.0.0.1:6379>
```

---------

Signed-off-by: Yang Zhao <zymy701@gmail.com>
 2026-03-03 14:38:04 +01:00
Rain ValentineandGitHub 2622aa8e03 [DEFLAKE] Fix flaky block_keyspace_notification pipelining test (#3294) 
The test "Blocking keyspace notification with pipelining hset after
hget" was recently failing intermittently with two different errors:

1. `Expected [expr {114 * 10 < 1114}]` - timing assertion failed under Valgrind
2. `Timeout waiting for blocked clients` - race condition on normal runs

The test used wall-clock timing to verify that hget (non-blocking)
completed faster than hset (blocking). This is unreliable because:
- Valgrind slows execution 10-50x, making timing ratios meaningless
- Fast systems may complete both operations in <10ms, causing ratio failures

This fix replaces timing assertions with blocked client count checks,
which directly verify the blocking mechanism rather than inferring it
from timing. The test now confirms hget's response is available before
hset blocks, then waits for the blocked client count to transition
through the expected states.

Signed-off-by: Rain Valentine <rsg000@gmail.com>
 2026-03-03 15:56:28 +08:00
BaswanthGitHubBaswanth Vegunta
 		9a51cc390a Fix TSAN compatibility for module loading (#3257) 
Made few changes to support TSAN build in valkey-search repository.
Related: https://github.com/valkey-io/valkey-search/issues/703

Signed-off-by: Baswanth Vegunta <baswanth@amazon.com>
Co-authored-by: Baswanth Vegunta <baswanth@amazon.com>
 2026-03-03 15:54:37 +08:00
Rain ValentineandGitHub be048d4246 rename functions - prefix should be zset not zsl (#3278) 
Simple function renaming. The zsl prefix denotes the function is
specific to the zset skiplist, but zslFreeLexRange and zslParseLexRange
are not specific to zset encoding. `zset` is a more accurate prefix.

I am working on #3166 but this is worthwhile on its own.

Signed-off-by: Rain Valentine <rainval@amazon.com>
 2026-03-02 16:18:35 -08:00
Sarthak AggarwalandGitHub c1b6c248b3 Fix weekly release runs checking out unstable in daily workflow (#3295) 
Honors `workflow_call` inputs rather than checking out the
`GITHUB_HEAD_REF` always.

```
Determining the checkout info
  /usr/bin/git branch --list --remote origin/8.0
    origin/8.0
/usr/bin/git sparse-checkout disable
/usr/bin/git config --local --unset-all extensions.worktreeConfig
Checking out the ref
  /usr/bin/git checkout --progress --force -B 8.0 refs/remotes/origin/8.0
  Switched to a new branch '8.0'
  branch '8.0' set up to track 'origin/8.0'.

```


Now the workflow checks out the right branch. 
Link:
https://github.com/sarthakaggarwal97/valkey/actions/runs/22599943936/job/65479450708#step:3:83

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
 2026-03-02 16:13:26 -08:00
BinbinandGitHub 05fb04465a Show replica dual-channel replication buffer memory in INFO MEMORY and MEMORY STATS (#2924) 
After introducing dual channel replication in #60, we added two new info
fields `replicas_repl_buffer_size` and `replicas_repl_buffer_peak` to the
info replication section.

However, we did not display the actual memory usage of the replication
buffer in info memory section. Furthermore, since the replicas replication
buffer was not previously included in memory overhead section, this caused
the `used_memory_dataset` and `used_memory_overhead` fields in INFO MEMORY
to display incorrect values, and also caused the `overhead.total` field in
`MEMORY STATS` to display an incorrect value. Previously, on replica nodes,
the replicas replication buffer was included in dataset memory.

Changes:
- `mem_total_replication_buffers` should include all replication buffers, so
  now it will also contain `mem_replicas_repl_buffer`.
- Added a new `mem_replicas_repl_buffer` field in INFO MEMORY section. During
  the dual channel replication, it show total memory consumed by the replicas
  replication buffer on replica side.
- Added a `new replicas.repl.buffer` field in MEMORY STATS. It is the same
  metric as `mem_replicas_repl_buffer`.

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-03-02 10:44:45 +08:00
Ran ShidlansikandGitHub 4969ccc08a Deflake ttl persistence in aof test (#3285) 
Sometimes it might miss setting a field with short expiration

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
 2026-03-02 10:40:30 +08:00
Tony WoosterandGitHub 811abbf18f Fix use-after-free in generateSyncSlotsEstablishCommand (#3283) 
Noticed a use-after-free/double-free bug while load-testing manual slot
migrations. Was able to reliably trigger a process crash.

Original trace:

```
Backtrace:
0   libsystem_platform.dylib            0x00000001917496a4 _sigtramp + 56
1   libsystem_pthread.dylib             0x000000019170f848 pthread_kill + 296
2   libsystem_c.dylib                   0x00000001916189e4 abort + 124
3   libsystem_malloc.dylib              0x000000019151c174 malloc_vreport + 892
4   libsystem_malloc.dylib              0x000000019151fc90 malloc_report + 64
5   libsystem_malloc.dylib              0x000000019152421c ___BUG_IN_CLIENT_OF_LIBMALLOC_POINTER_BEING_FREED_WAS_NOT_ALLOCATED + 32
6   valkey-server                       0x0000000104fc2174 _sdsMakeRoomFor + 780
7   valkey-server                       0x0000000104fc30a0 sdscatfmt + 168
8   valkey-server                       0x00000001050318b8 generateSyncSlotsEstablishCommand + 176
9   valkey-server                       0x000000010503156c createSlotImportJob + 324
10  valkey-server                       0x0000000105034fd4 clusterCommandSyncSlots + 1572
11  valkey-server                       0x000000010502f370 clusterCommandSpecial + 2432
12  valkey-server                       0x0000000105021e80 clusterCommand + 2992
13  valkey-server                       0x0000000104fb59a8 call + 320
14  valkey-server                       0x0000000104fcc708 processCommandAndResetClient + 3088
15  valkey-server                       0x0000000104fc9c3c processInputBuffer + 440
16  valkey-server                       0x0000000104fc9198 readQueryFromClient + 108
17  valkey-server                       0x000000010508073c connSocketEventHandler + 136
18  valkey-server                       0x0000000104fa49b4 aeProcessEvents + 316
19  valkey-server                       0x0000000104fc081c main + 18408
20  dyld                                0x000000019136eb98 start + 6076
```

The bug seems to be that the result of `sdscatfmt` is being ignored.
Since `sdscatfmt` potentially reallocs a new buffer when it needs to,
and deallocs the old one, this can cause a use-after-free/double-free if
the loop triggers a buffer-size increase.

This PR fixes the bug.

Signed-off-by: Tony Wooster <twooster@gmail.coM>
 2026-03-02 10:40:01 +08:00
bandalgomsuandGitHub 6b2411da04 Fix use zfree in cli --eval path (#3281) 
In `evalMode()`, `argv2` is allocated with `zmalloc()` but freed with `free()`.
On builds using jemalloc/tcmalloc, this can cause `free(): invalid pointer`
Closes #3273

Signed-off-by: Su Ko <rhtn1128@gmail.com>
 2026-03-01 21:27:44 +08:00
Alina LiuandGitHub e97d127fb2 Suppress valgrind for BIO jobs (#3277) 
Suppress valgrind for BIO jobs valgrind errors:
https://github.com/valkey-io/valkey/actions/runs/21969557125/job/63467641572#step:6:8648

## Changes

- Add `allocBioJob()` with `__attribute__((noinline))` as a centralized
allocation function for all BIO jobs. The `noinline` attribute ensures
it appears as a distinct frame in valgrind stack traces.
- Replace all direct `zmalloc` calls in `bioCreate*Job` functions
with`allocBioJob()`.
- Add a valgrind suppression in `src/valgrind.sup` matching definite
leaks with `allocBioJob` in the call stack.
- Remove `current_job` introduced in #3256

---------

Signed-off-by: Alina Liu <liusalisa6363@gmail.com>
 2026-02-27 22:36:20 -08:00
Gagan H RandGitHub da206c609a Set read permission in workflow to improve OpenSSF score (#3267) 
This PR fixes #3264. Score is currently 6. It will get improved to
around 6.9 once this PR is merged.

Signed-off-by: Gagan H R <hrgagan4@gmail.com>
 2026-02-27 23:23:31 +01:00
Harry LinandGitHub ce291b3613 Add gtest dependencies to unit test workflows (#3270) 
Install gtest dependencies in daily.yml.

Signed-off-by: harrylin98 <harrylin980107@gmail.com>
 2026-02-27 13:39:51 -08:00
Madelyn OlsonandGitHub c84990bf56 Add Jim Brunner as a committer (#3272) 
---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
 2026-02-27 13:17:54 -08:00
Hanxi ZhangandGitHub 3110d910ef Deflake Restart target replica during migration (without save) causes success test (#3226) 
Fix #3222

- Replace assert_equal with wait_for_condition for replica DBSIZE checks
- Fix a duplicate assert_equal line that checked $not_owning_repl twice
instead of checking both $not_owning_prim and $not_owning_repl.

## Analysis

wait_for_countkeysinslot did work, the keys were there. The issue is
that COUNTKEYSINSLOT and DBSIZE read from different sources.

COUNTKEYSINSLOT reads the actual hashtable size:
```
// countKeysInSlotForDb (cluster.c, line 831)
kvstoreHashtableSize(db->keys, hashslot)
    → hashtableSize(ht)    // actual keys in the slot
```
DBSIZE reads a cached counter:
```
// dbsizeCommand (db.c)
kvstoreSize(c->db->keys) // This can be found in line 358, kvstore.c)
    → kvs->key_count       // cached total
```
When a slot is marked as importing, keys are excluded from key_count:
```
// cumulativeKeyCountAdd (kvstore.c, line 157)
if (kvstoreIsImporting(kvs, didx)) {
    kvs->importing_key_count += delta;  // hidden from DBSIZE
    return;                              // key_count NOT updated
}
kvs->key_count += delta;                // visible to DBSIZE
```

key_count will be updated when:
```
finishSlotMigrationJob (cluster_migrateslots.c, line 2302)
-> call setSlotImportingStateInAllDbs(slots, 0) (cluster_migrateslots.c, line 2340)
  -> call setSlotImportingStateInDb() (cluster_migrateslots.c, line 249)
    -> call kvstoreSetIsImporting() (cluster_migrates.c, line 238)
      -> call cmulativeKeyCountAdd() (kvstore.c, line 955)
        -> kvs->key_count += delta;
```

When slots are marked as importing, the keys exist in the hashtables but
are excluded from metrics like key_count. As documented in
kvstoreSetIsImporting (kvstore.c):

"Importing hashtables are not included in hashtable metrics and are
excluded from scanning and random key lookup."

After the replica restarts and resyncs, there is a brief window where
the migrated slots are still marked as importing. During this window,
COUNTKEYSINSLOT sees the keys (reads actual hashtable) but DBSIZE
returns 0 (reads key_count which excludes importing keys). Once the
importing flag is cleared, key_count is updated and DBSIZE returns the
correct value.

wait_for_condition handles this by retrying until DBSIZE reflects the
updated key_count.

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
 2026-02-27 10:42:50 +08:00
Harry LinGitHubHarry LinJim BrunnerMadelyn OlsonJacob MurphyAlina Liu
 		3107c4eb21 Introduce GoogleTest for Valkey unit testing (#3241) 
This PR adds GoogleTest (gtest) support to Valkey to enable 
writing modern unit tests，as mentioned in
https://github.com/valkey-io/valkey/issues/2878

**Motivation**: 
GoogleTest provides richer assertions, test fixtures, mocking
support, and improved diagnostics, helping improve test coverage
and maintainability over time.

For more details, see `src/gtest/README.md`.

**Changes**

This PR integrates the GoogleTest framework and migrates all
existing C unit tests to GoogleTest.

---------

Signed-off-by: Harry Lin <harrylhl@amazon.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: Alina Liu <liusalisa6363@gmail.com>
Signed-off-by: Jacob Murphy <jkmurphy@google.com>
Signed-off-by: Jim Brunner <brunnerj@amazon.com>
Co-authored-by: Harry Lin <harrylhl@amazon.com>
Co-authored-by: Jim Brunner <brunnerj@amazon.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Jacob Murphy <jkmurphy@google.com>
Co-authored-by: Alina Liu <liusalisa6363@gmail.com>
 2026-02-26 14:14:23 -08:00
Gagan H RandGitHub 1b52016873 Adds scorecard workflow to publish OpenSSF scores (#3163) 
Publish OpenSSF Scorecard results, which means users and downstream
consumers can easily discover the project’s security best-practice
signals via Scorecard API.

Publishing Scorecard results:

- Improves transparency for users and integrators
- Provides early visibility into missing or improvable security
practices

Fixes #3162

---------

Signed-off-by: Gagan H R <hrgagan4@gmail.com>
 2026-02-26 21:24:06 +01:00
Shinobu NunotabaandGitHub ea942753c4 Lazy loading of RDMA libs in CLI/Benchmark when building as module (#3072) 
This PR decouples the mandatory RDMA library dependency from
`valkey-cli` and `valkey-benchmark` by implementing a **runtime dynamic
loading (lazy loading)** mechanism.

When `BUILD_RDMA=module` is used, the CLI tools are no longer
hard-linked to `librdmacm` and `libibverbs`. Instead, these libraries
are only loaded via `dlopen` when the `--rdma` flag is explicitly
invoked by the user.

- **Dynamic Symbol Loader**: Implemented a symbol loader in libvalkey
using dlopen() and dlsym(), here:
https://github.com/valkey-io/libvalkey/pull/284. That was merged and
libvalkey was updated in valkey to include this change.
- **Build System Propagation**:
- **Makefile**: When building as a module, pass `USE_DLOPEN_RDMA` flag
to libvalkey's makefile.
- **CMake**: When building as a module, pass `ENABLE_DLOPEN_RDMA` flag
to libvalkey's CMake build.

Fixes #3070

Signed-off-by: Ada-Church-Closure <nunotabashinobu066@gmail.com>
 2026-02-26 13:57:16 +01:00
Yana MolodetskyandGitHub aa39375bdf Deflake test 'LATENCY GRAPH can output the event graph' (#3260) 
The test was using `assert_morethan_equal` with fixed minimum thresholds
(500ms for high, 300ms for low), which caused flaky failures when the
actual latency values were slightly below these thresholds due to timing
variations.

Example failure:

```
Expected '499' to be more than or equal to '500' (context: type eval line 12 cmd {assert_morethan_equal $high 500} proc ::test)
```

Changes:

- Replaced assert_morethan_equal $high 500 with assert_range $high 450
550
- Replaced assert_morethan_equal $low 300 with assert_range $low 250 350

The new ranges (450-550ms for high, 250-350ms for low) align with the
expected latency values from the debug sleep commands (0.5s and 0.3s
respectively) while providing tolerance for timing variations,
consistent with the approach used elsewhere in the test file.

Signed-off-by: Yana Molodetsky <yamolodu@amazon.com>
 2026-02-26 13:13:41 +02:00
Alina LiuandGitHub c3020e0a11 Replace mutexQueuePeek with current_job field for BIO memcheck fix (#3256) 
## Problem
The peek-then-pop pattern introduced in
https://github.com/valkey-io/valkey/pull/3178 solved the valgrind
false-positive leak report
(https://github.com/valkey-io/valkey/actions/runs/21969557125/job/63467641572#step:6:8648)
for in-flight BIO jobs, but `peek` is a problematic API on a mutexqueue:
multiple readers can peek the same item, and one reader can pop what
another peeked.

## Fix
Instead, store the in-flight job pointer in
`bio_worker_data.current_job` before processing and clear it after
freeing. Since `bio_workers[]` is a static array, valgrind can always
trace from global memory to the job allocation, even after the worker
thread is cancelled.

Removed `mutexQueuePeek` from mutexqueue.{c,h} and its test.

## Test
Manually run Daily workflow: All four valgrind jobs are green:
`test-valgrind-test` passed, `test-valgrind-misc` passed,
`test-valgrind-no-malloc-usable-size-test` passed,
`test-valgrind-no-malloc-usable-size-misc` passed.

Signed-off-by: Alina Liu <liusalisa6363@gmail.com>
 2026-02-25 15:59:16 -08:00
Hanxi ZhangandGitHub 617643f35e Fix type of now parameter in rdbLoadObject() to use mstime_t instead of time_t (#3255) 
Fix #3254 
### Problem
The now parameter in rdbLoadObject() was declared as time_t, which
truncates millisecond timestamps on 32-bit systems, causing hash field
expiry checks to fail during RDB load.
Reproduced:
https://github.com/hanxizh9910/valkey/actions/runs/22382739246/job/64786946078

### Fix

Changed time_t now → mstime_t now in the declaration and definition of
rdbLoadObject().

### Test run
https://github.com/hanxizh9910/valkey/actions/runs/22383363092

---------

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
 2026-02-25 10:56:20 +02:00
Hanxi ZhangandGitHub 63f37ee960 Fix flake dual-channel-replication primary gets cob overrun before established psync test (#3242) 
Fix #3231: Increase wait timeout for RDB load log detection in test

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
 2026-02-25 10:10:13 +08:00
Rain ValentineandGitHub 851ca54c66 Bugfix for GIL deadlock while unloading script engine, reenable memory test in crash report (#3029) 
The memory test was commented out in #2858 and should have been
reenabled. On further investigation I found that the server hangs during
shutdown inside the `bioDrainWorker(BIO_LAZY_FREE)` call. This causes
deadlock because the lock was acquired for shutdown but lazy free jobs
require the GIL too:

- main thread: `serverCron()` acquires GIL via `afterSleep()` then calls
`finishShutdown()`, which eventually calls our script module unload code
that calls `bioDrainWorker()`.
- bio threads: Pending lazy free jobs such as `lazyFreeEvalScripts()`
call `scriptingEngineCallFreeFunction()` which requires the GIL.

---------

Signed-off-by: Rain Valentine <rsg000@gmail.com>
 2026-02-24 19:50:31 +01:00
BinbinGitHubViktor SöderqvistDaniil Kashapov
 		b32c68a66f Abort and swap the tables if ht1 is very full during the hashtable shrink rehashing (#3175) 
During hashtable shrinking, all keys are inserted into ht1.
This PR adds a mechanism to the hashtable: when there are severe hash
collisions with the new ht1 during the shrink rehashing process, the
hashtable will stop shrinking by swapping ht0 and ht1 to avoid excessive
performance degradation of the new hashtable during this period.

In extreme cases, for example, if ht0 is very large and is reduced to
only one entry, and the new ht1 is very small after the resize, the ht0
rehash process is very slow since we have a lot of empty buckets. If a
large number of elements are added into ht1 at this time, it will lead
to severe hash collisions in ht1.

During the add operation during hashtable shrinking, we check the fill
percentage of ht1. If it exceeds MAX_FILL_PERCENT_HARD, we swap ht0 and
ht1, and abort the shrinking process, and then set rehash_idx back to 0
and restart the rehash.

This PR also added a new debug hashtable-can-abort-shrink subcommand to
control this behavior.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Daniil Kashapov <daniil.kashapov.ykt@gmail.com>
 2026-02-24 12:58:39 +01:00
Hanxi ZhangGitHubBinbin
 		6b81e36c9d Skip expired hash fields when loading non-preamble RDB on primary (#3091) 
## Summary

Fixes #2620

Skip loading expired hash fields when a non-preamble RDB is being loaded
on a primary server. Propagate `HDEL` to replicas when expired fields
are skipped.

## Changes

- Updated `rdbLoadObject` signature to accept `rdbflags` and `now`
parameters
- Added logic to skip expired hash fields during RDB load on primary
- Propagate `HDEL` to replicas when `RDBFLAGS_FEED_REPL` is set
- Updated all callers of `rdbLoadObject`
- Added unit test

---------

Signed-off-by: Hanxi Zhang <hanxizh@amazon.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
 2026-02-24 13:24:47 +02:00
BinbinandGitHub af7bac374b Logging fix or improvement around new shard ID generation (#3192) 
When a cluster reset is performed on a replica node, a new shard ID is generated
because the node is about to become an empty primary node, see #2283.

However, the log added in #2510 caused some confusions. In clusterSetNodeAsPrimary
we will print:
```
serverLog(LL_NOTICE, "Reconfiguring node %.40s (%s) as primary for shard %.40s", n->name, humanNodename(n), n->shard_id);
```

In clusterReset, we first call clusterSetNodeAsPrimary and then generate a new
shard ID, which causes us to print an error shard ID log first.

There is an exmaple, when a replica node performs a cluster reset, we will print:
```
xxx * Cluster reset (user request from 'xxx').
xxx * Reconfiguring node af76a3e0ffcd77bd14fa47ce4d07ab2bdc78702f (xxx) as primary for shard ea528667634af8beed83adac2b9af8360769a1b4
```

But the node shard id is actually:
```
xxx> cluster myshardid
"52ede26d1554dd203161ba09011af14574b2cc84"
```

Now after a new shard ID is generated we will print a log, and we also move the
call to clusterSetNodeAsPrimary after the new shard id, so that we can have the
right one. After this PR:
```
xxx * Cluster reset (user request from 'xxx').
xxx * Moving myself to a new shard bd31870ce73f5977084e6a46e337a4a1ad38fc66.
xxx * Reconfiguring node 1d54b904efd30cd9d7d1abbfd63c8fafbb62e1c8 (xxx) as primary for shard bd31870ce73f5977084e6a46e337a4a1ad38fc66
```

This is part of #2989, but i guess we won't merge the extension fix in a short
time, so i am gonna extracting it separately as a log fix (or improvement).

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-02-24 11:13:15 +08:00
Roshan KhatriandGitHub fd65d487eb Fix Tcl 9 compatibility in cluster packet test (#3251) 
Remove -encoding binary from fconfigure as it is no longer supported in
Tcl 9 (Fedora Rawhide and Daily / test-fedoralatest-jemalloc
(pull_request)). -translation binary alone is sufficient.
```
https://github.com/valkey-io/valkey/actions/runs/22324297258/job/64590589777?pr=3225
===== Start of server log (pid 17617) =====

### Starting server for test 
17617:M 23 Feb 2026 21:21:47.846 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
17617:M 23 Feb 2026 21:21:47.846 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
17617:M 23 Feb 2026 21:21:47.846 * Valkey version=9.0.3, bits=64, commit=00000000, modified=0, pid=17617, just started
17617:M 23 Feb 2026 21:21:47.846 * Configuration loaded
17617:M 23 Feb 2026 21:21:47.847 * monotonic clock: POSIX clock_gettime
                .+^+.                                                
            .+#########+.                                            
        .+########+########+.           Valkey 9.0.3 (00000000/0) 64 bit
    .+########+'     '+########+.                                    
 .########+'     .+.     '+########.    Running in cluster mode
 |####+'     .+#######+.     '+####|    Port: 25121
 |###|   .+###############+.   |###|    PID: 17617                     
 |###|   |#####*'' ''*#####|   |###|                                 
 |###|   |####'  .-.  '####|   |###|                                 
 |###|   |###(  (@@@)  )###|   |###|          https://valkey.io/      
 |###|   |####.  '-'  .####|   |###|                                 
 |###|   |#####*.   .*#####|   |###|                                 
 |###|   '+#####|   |#####+'   |###|                                 
 |####+.     +##|   |#+'     .+####|                                 
 '#######+   |##|        .+########'                                 
    '+###|   |##|    .+########+'                                    
        '|   |####+########+'                                        
             +#########+'                                            
                '+v+'                                                

17617:M 23 Feb 2026 21:21:47.849 * No cluster configuration found, I'm 5501916ccdf76dee6b652cab10402cab3a8f9152
17617:M 23 Feb 2026 21:21:47.852 * Server initialized
17617:M 23 Feb 2026 21:21:47.852 * Ready to accept connections tcp
17617:M 23 Feb 2026 21:21:47.852 * Ready to accept connections unix
17617:M 23 Feb 2026 21:21:47.963 - Accepted 127.0.0.1:37791
17617:M 23 Feb 2026 21:21:47.963 - Client closed connection id=2 addr=127.0.0.1:37791 laddr=127.0.0.1:25121 fd=12 name= age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37504 events=r cmd=ping user=default redir=-1 resp=2 lib-name= lib-ver= tot-net-in=7 tot-net-out=7 tot-cmds=1
17617:M 23 Feb 2026 21:21:47.969 - Accepted 127.0.0.1:39251
17617:M 23 Feb 2026 21:21:47.970 * configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH
17617:M 23 Feb 2026 21:21:47.975 # Missing implement of connection type tls
17617:M 23 Feb 2026 21:21:47.976 # DEBUG LOG: ========== I am primary 0 ==========
17617:M 23 Feb 2026 21:21:49.872 * Cluster state changed: ok
### Starting test Packet with missing gossip messages don't cause invalid read in tests/unit/cluster/packet.tcl
17617:M 23 Feb 2026 21:21:49.879 - Accepting cluster node connection from 127.0.0.1:33701
17617:M 23 Feb 2026 21:21:49.880 - Client closed connection id=3 addr=127.0.0.1:39251 laddr=127.0.0.1:25121 fd=12 name= age=2 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=426 obl=0 oll=0 omem=0 tot-mem=22144 events=r cmd=cluster|info user=default redir=-1 resp=2 lib-name= lib-ver= tot-net-in=1363 tot-net-out=17324 tot-cmds=47
17617:signal-handler (1771881709) Received SIGTERM scheduling shutdown...
17617:M 23 Feb 2026 21:21:49.973 * User requested shutdown...
17617:M 23 Feb 2026 21:21:49.973 * Removing the pid file.
17617:M 23 Feb 2026 21:21:49.973 * Saving the cluster configuration file before exiting.
17617:M 23 Feb 2026 21:21:49.978 * Removing the unix socket file.
17617:M 23 Feb 2026 21:21:49.978 # Valkey is now ready to exit, bye bye...
===== End of server log (pid 17617) =====


===== Start of server stderr log (pid 17617) =====


===== End of server stderr log (pid 17617) =====

[exception]: Executing test client: unknown encoding "binary": No longer supported.
	please use either "-translation binary" or "-encoding iso8859-1".
unknown encoding "binary": No longer supported.
	please use either "-translation binary" or "-encoding iso8859-1"
    while executing
"fconfigure $sock -translation binary -encoding binary -buffering none -blocking 1"
    ("uplevel" body line 18)
    invoked from within
"uplevel 1 $code"
    (procedure "test" line 62)
    invoked from within
"test "Packet with missing gossip messages don't cause invalid read" {
        set base_port [srv 0 port]
        set cluster_port [expr {$base_port + ..."
    ("uplevel" body line 2)
    invoked from within
"uplevel 1 $code"
    (procedure "cluster_setup" line 41)
    invoked from within
"cluster_setup 1 0 1 continuous_slot_allocation default_replica_allocation {
    test "Packet with missing gossip messages don't cause invalid read" {
..."
    ("uplevel" body line 1)
    invoked from within
"uplevel 1 $code "
    (procedure "start_server" line 2)
    invoked from within
"start_server {overrides {cluster-enabled yes cluster-ping-interval 100 cluster-node-timeout 3000 cluster-databases 16 cluster-slot-stats-enabled yes} ..."
    ("uplevel" body line 1)
    invoked from within
"uplevel 1 $code"
    (procedure "start_multiple_servers" line 5)
    invoked from within
"start_multiple_servers $node_count $options $code"
    (procedure "start_cluster" line 17)
    invoked from within
"start_cluster 1 0 {tags {external:skip cluster tls:skip}} {
    test "Packet with missing gossip messages don't cause invalid read" {
        set base..."
    (file "tests/unit/cluster/packet.tcl" line 84)
    invoked from within
"source $path"
    (procedure "execute_test_file" line 4)
    invoked from within
"execute_test_file $data"
    (procedure "test_client_main" line 10)
    invoked from within
"test_client_main $::test_server_port "
Killing still running Valkey server 10700
Killing still running Valkey server 11659
Killing still running Valkey server 11705
Killing still running Valkey server 12735
Killing still running Valkey server 12775
Killing still running Valkey server 12808
Killing still running Valkey server 12858
Killing still running Valkey server 12901
Killing still running Valkey server 12940
Killing still running Valkey server 12972
Killing still running Valkey server 13004
Killing still running Valkey server 13036
Killing still running Valkey server 13073
Killing still running Valkey server 14952
Killing still running Valkey server 16631
Killing still running Valkey server 16670
Killing still running Valkey server 16686
Killing still running Valkey server 16702
Killing still running Valkey server 16718
Killing still running Valkey server 16734
Killing still running Valkey server 16753
Killing still running Valkey server 16771
Killing still running Valkey server 17094
Killing still running Valkey server 17112
Killing still running Valkey server 17133
Killing still running Valkey server 17262
Killing still running Valkey server 17278
Killing still running Valkey server 17316
Killing still running Valkey server 17349
Killing still running Valkey server 17379
Killing still running Valkey server 17410
Killing still running Valkey server 17430
Killing still running Valkey server 17443
Killing still running Valkey server 17459
Killing still running Valkey server 17478
Killing still running Valkey server 17491
Killing still running Valkey server 17507
Killing still running Valkey server 17526
Killing still running Valkey server 17545
Killing still running Valkey server 17561
Killing still running Valkey server 17577
Killing still running Valkey server 17716
Killing still running Valkey server 17778
Killing still running Valkey server 17825
Killing still running Valkey server 17848
Killing still running Valkey server 17896
Killing still running Valkey server 17931
Killing still running Valkey server 17953
Killing still running Valkey server 17973
Killing still running Valkey server 17991
Killing still running Valkey server 18010
Killing still running Valkey server 18064
Killing still running Valkey server 18082
Killing still running Valkey server 18134
Killing still running Valkey server 18118
Killing still running Valkey server 18150
Killing still running Valkey server 18166
Killing still running Valkey server 18188
Killing still running Valkey server 18209
```

Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
 2026-02-23 15:19:25 -08:00
Madelyn Olson a737a11d66 Reset request type after handling empty requests 
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
 2026-02-23 11:26:53 -08:00
Roshan KhatriandMadelyn Olson 77ff9eb0e3 Fix for [CVE-2025-67733] RESP Protocol Injectiton via Lua error_reply 
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
 2026-02-23 09:56:06 -08:00
Roshan KhatriandMadelyn Olson 5a97f83b5b Fix for [CVE-2026-21863] Remote DoS with malformed Valkey Cluster bus message 
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
 2026-02-23 09:56:06 -08:00
BinbinandGitHub 274deca160 Fix getKeysUsingKeySpecs keynum case ignoring keystep when calc the last key (#3197) 
This issue was encountered while processing #3121. Currently in all our
commands with KSPEC_FK_KEYNUM, key step is 1. So this bug does not
currently
affect any core commands.

If we have commands with different key step values, calculting the last
key
in here will casue problems since we are not including step in the
calculation.

Signed-off-by: Binbin <binloveplay1314@qq.com>
 2026-02-23 09:07:22 -08:00
Joseph HeyburnandGitHub bdefaa54ff Dual-channel-replication announces itself at replica-announce-ip if configured (#2846) 
When dual-channel-replication is enabled, and replica-announce-ip is
set, the RDB/AOF channel does not announce itself at this endpoint. This
defaults to the IP address behind the NAT, or the Kubernetes Pod IP in
our case.

This means that if Sentinel is polling the primary for connected
replicas, it will first see the ephemeral pod IP, then revert to the
announce-ip - leaving behind the pod IP as a down replica.

This PR configures the RDB/AOF channel to also announce itself at the
announce-ip to prevent the stale replica.

## Testing

I evaluated writing unit tests for this, but I am not sure of a way we
can test an IP address different to localhost (127.0.0.1) that would
fail without the fix. I did test on Kubernetes against 9.0 tag and
verified the fix there too.

### Status quo

On 9.0 image tag:

```
$ kubectl get pods -n valkey-baseline -o custom-columns=NAME:.metadata.name,POD-IP:.status.podIP
NAME                              POD-IP
valkey-primary-5bd78c8566-llb6k   10.244.0.25
valkey-replica-0                  10.244.0.17
valkey-replica-1                  10.244.0.13

$ kubectl get services -n valkey-baseline -o custom-columns=NAME:.metadata.name,CLUSTER-IP:.spec.clusterIP
NAME               CLUSTER-IP
valkey-primary     10.96.147.28
valkey-replica-0   10.96.66.233
valkey-replica-1   10.96.57.230
```

Logs below show that pod IP for valkey-primary-5bd78c8566-llb6k
`10.244.0.25:6379` is being used for dual-channel replication. This
should be its cluster IP `10.96.147.28` as this is what is set in
replica-announce-ip.

```
1:M 14 Nov 2025 17:57:51.750 * Replica 10.96.147.28:6379 asks for synchronization
1:M 14 Nov 2025 17:57:51.751 * Replica 10.244.0.25:6379 asks for synchronization
1:M 14 Nov 2025 17:57:56.135 * Dual channel replication: Sending to replica 10.244.0.25:6379 RDB end offset 1763269 and client-id 35
1:M 14 Nov 2025 17:57:56.140 * Replica 10.96.147.28:6379 asks for synchronization
```

### This fix

```
$ kubectl get pods -n valkey-test -o custom-columns=NAME:.metadata.name,CLUSTER-IP:.status.podIP  
NAME                              POD-IP
valkey-primary-594c9597b5-qqvdk   10.244.0.26
valkey-replica-0                  10.244.0.10
valkey-replica-1                  10.244.0.18

$ kubectl get services -n valkey-test -o custom-columns=NAME:.metadata.name,CLUSTER-IP:.spec.clusterIP
NAME               CLUSTER-IP
valkey-primary     10.96.125.142
valkey-replica     None
valkey-replica-0   10.96.155.74
valkey-replica-1   10.96.64.111
valkey-sentinel    None
```

Logs show that the Cluster IP is now being used for dual-channel
replication.

```
1:M 14 Nov 2025 17:57:49.923 * Replica 10.96.125.142:6379 asks for synchronization
1:M 14 Nov 2025 17:57:49.924 * Replica 10.96.125.142:6379 asks for synchronization
1:M 14 Nov 2025 17:57:54.913 * Dual channel replication: Sending to replica 10.96.125.142:6379 RDB end offset 1771247 and client-id 36
1:M 14 Nov 2025 17:57:54.916 * Replica 10.96.125.142:6379 asks for synchronization
```

Fixes #2338

Signed-off-by: Joseph Heyburn <jdheyburn@gmail.com>
 2026-02-23 18:16:48 +02:00
Björn SvenssonandGitHub d3ae62cc9c Update deps/libvalkey to version 0.4.0 (#3216) 
Update deps/libvalkey to version 0.4.0

Squashed 'deps/libvalkey/' changes from b012f8e85..45c2ed15c

45c2ed15c Release 0.4.0 (#286)
40d6590d7 Implement runtime dynamic loading for RDMA libraries (#284)
62e757d17 Release 0.3.0 (#283)
a554f0942 Fix potential uint32_t underflow issue (#280)
8f9051ae0 Correcting command parser bug (#277)
29023eb36 Add valkey-json, valkey-bloom, valkey-search to cmddef.h
ae756bc89 Update cmddef.h to Valkey 9.0.0
21abd737e Replace problematic alloca() with fixed stack alloc
38191079c Fix compilation on Solaris with Sun/Solaris Studio
ef5de0312 Make libvalkey initialization thread-safe
ae341dea5 Support slotmap updates using CLUSTER NODES in RESP3 (#262)
36f6e2292 Fix the long-blocking read for Valkey RDMA. (#233)
c090c28be Use a uintptr_t hop for casting pointers to ints
daa7f11ac Avoid heap buffer overflow in valkeyAsyncFormattedCommand (#245)
15974930d Add option to select a logical database (#244)
983d67e4f Install the macosx adapter on Apple platforms only
...

git-subtree-dir: deps/libvalkey
git-subtree-split: 45c2ed15cab9fa0ea1a6cabc8460f5eea6240de5

Signed-off-by: Björn Svensson <bjorn.a.svensson@est.tech>
 2026-02-23 15:28:30 +01:00
322 changed files with 29440 additions and 17370 deletions
Expand all files Collapse all files
.config/typos.toml
+5 -1
View File
@@ -5,7 +5,7 @@ extend-exclude = [
".git/",
"deps/",
# crc16_slottable is primarily pre-generated random strings.
"src/crc16_slottable.h",
"src/crc16_slottable.c",
]
ignore-hidden = false
@@ -14,6 +14,7 @@ exat = "exat"
optin = "optin"
smove = "smove"
Parth = "Parth" # seems like the spellchecker does not like it is similar to "Path"
NAM = "NAM" # Name similar to Name
nd = "nd"
[default]
@@ -71,3 +72,6 @@ ake = "ake"
[type.tcl.extend-words]
fo = "fo"
tre = "tre"
[type.cpp.extend-words]
fo = "fo"
.github/instructions/core-engine.instructions.md
+1 -1
View File
@@ -36,7 +36,7 @@ Apply these standards to core engine C code. Do NOT apply to `deps/` (vendored d
- **PR Scope:** Separate refactoring from functional changes for easier backporting.
## 5. Testing & Documentation
- **Unit Tests:** Required for data structures in `src/unit/`. Test files should follow `test_*.c` naming.
- **Unit Tests:** Required for data structures in `src/unit/`. Test files should follow `test_*.cpp` naming.
- **Integration Tests:** Required for commands in `tests/`.
- **Command Changes:** New/modified commands need corresponding updates in `src/commands/*.json`.
- **New C Files:** Remind to update `CMakeLists.txt` when adding new `.c` source files.
.github/workflows/benchmark-on-label.yml
+21 -10
View File
@@ -5,8 +5,8 @@ on:
types: [labeled]
concurrency:
group: ec2-al-2023-pr-benchmarking-arm64
cancel-in-progress: false
group: benchmark-${{ github.event.pull_request.number }}-${{ github.event.label.name }}
cancel-in-progress: true
defaults:
run:
@@ -20,7 +20,8 @@ permissions:
jobs:
benchmark:
if: |
github.event.action == 'labeled' && github.event.label.name == 'run-benchmark' &&
github.event.action == 'labeled' &&
(github.event.label.name == 'run-benchmark' || github.event.label.name == 'run-cluster-benchmark') &&
github.repository == 'valkey-io/valkey'
runs-on: ["self-hosted", "ec2-al-2023-pr-benchmarking-arm64"]
@@ -66,7 +67,7 @@ jobs:
openssl-devel \
bzip2-devel \
libffi-devel
pip install -r requirements.txt
pip install --require-hashes -r requirements.txt
- name: Build latest valkey_latest
working-directory: valkey_latest
@@ -86,6 +87,15 @@ jobs:
echo "VALKEY_BENCHMARK_PATH=$VALKEY_BENCHMARK_PATH" >> $GITHUB_ENV
echo "Latest valkey-benchmark path: $VALKEY_BENCHMARK_PATH"
- name: Enable cluster mode in benchmark config
working-directory: valkey-perf-benchmark
if: github.event.label.name == 'run-cluster-benchmark'
run: |
CONFIG_FILE="../valkey/.github/benchmark_configs/benchmark-config-arm.json"
sed -i 's/"cluster_mode": false/"cluster_mode": true/g' "$CONFIG_FILE"
echo "Updated config for cluster mode:"
cat "$CONFIG_FILE"
- name: Run benchmarks
working-directory: valkey-perf-benchmark
run: |
@@ -100,7 +110,7 @@ jobs:
--target-ip ${{ secrets.EC2_ARM64_IP }}
--valkey-path "../valkey"
--results-dir "results"
--runs 5
--runs 3
)
# Run benchmark
@@ -120,10 +130,9 @@ jobs:
continue-on-error: true
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
with:
name: benchmark-results
name: ${{ github.event.label.name }}-results
path: |
./valkey-perf-benchmark/results/${{ github.event.pull_request.merge_commit_sha }}/metrics.json
./valkey-perf-benchmark/results/${{ github.event.pull_request.base.ref }}/metrics.json
./valkey-perf-benchmark/results/
comparison.md
- name: Comment PR with results
@@ -137,11 +146,13 @@ jobs:
const sha = '${{ github.event.pull_request.head.sha }}';
const short = sha.slice(0,7);
const link = `[\`${short}\`](https://github.com/${owner}/${repo}/commit/${sha})`
const label = '${{ github.event.label.name }}';
const prefix = label === 'run-cluster-benchmark' ? 'Cluster Benchmark' : 'Benchmark';
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner,
repo,
body: `**Benchmark ran on this commit:** ${link}\n\n${body}`
body: `**${prefix} ran on this commit:** ${link}\n\n${body}`
});
- name: Cleanup any running valkey processes
@@ -160,7 +171,7 @@ jobs:
fi
- name: Remove ${{ github.event.label.name }} label
if: always() && github.event.label.name == 'run-benchmark'
if: always() && (github.event.label.name == 'run-benchmark' || github.event.label.name == 'run-cluster-benchmark')
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
.github/workflows/benchmark-release.yml
+1 -1
View File
@@ -106,7 +106,7 @@ jobs:
openssl-devel \
bzip2-devel \
libffi-devel
pip install -r requirements.txt
pip install --require-hashes -r requirements.txt
- name: Build latest valkey_latest
working-directory: valkey_latest
.github/workflows/ci.yml
+51 -15
View File
@@ -35,7 +35,9 @@ jobs:
- name: make
# Fail build if there are warnings
# build with TLS just for compilation coverage
run: make -j4 all-with-unit-tests SERVER_CFLAGS='-Werror' BUILD_TLS=yes USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
run: |
sudo apt-get install pkg-config libgtest-dev libgmock-dev
make -j4 all-with-unit-tests SERVER_CFLAGS='-Werror' BUILD_TLS=yes USE_LIBBACKTRACE=yes
- name: test
run: |
sudo apt-get install tcl8.6 tclx
@@ -50,7 +52,7 @@ jobs:
if [[ ! -z "$dirty" ]]; then echo "$dirty"; exit 1; fi
- name: unit tests
run: |
./src/valkey-unit-tests
make test-unit
test-ubuntu-latest-compatibility:
runs-on: ubuntu-latest
@@ -62,6 +64,9 @@ jobs:
- {version: "8.0.6", file: "valkey-8.0.6-noble-x86_64.tar.gz"}
- {version: "8.1.4", file: "valkey-8.1.4-noble-x86_64.tar.gz"}
steps:
- name: Install gtest
run: |
sudo apt-get install pkg-config libgtest-dev libgmock-dev
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -74,7 +79,7 @@ jobs:
- name: make
# Fail build if there are warnings
# build with TLS just for compilation coverage
run: make -j4 all-with-unit-tests SERVER_CFLAGS='-Werror' BUILD_TLS=yes USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
run: make -j4 all-with-unit-tests SERVER_CFLAGS='-Werror' BUILD_TLS=yes USE_LIBBACKTRACE=yes
- name: Install old server (${{ matrix.server.version }}) for compatibility testing
run: |
@@ -103,7 +108,7 @@ jobs:
sudo apt-get install -y cmake libssl-dev
mkdir -p build-release
cd build-release
cmake -DCMAKE_BUILD_TYPE=Release .. -DBUILD_TLS=yes -DBUILD_UNIT_TESTS=yes
cmake -DCMAKE_BUILD_TYPE=Release .. -DBUILD_TLS=yes -DBUILD_UNIT_GTESTS=yes
make -j$(nproc)
- name: test
run: |
@@ -111,8 +116,7 @@ jobs:
./utils/gen-test-certs.sh
./build-release/runtest --verbose --tags -slow --dump-logs --tls
- name: unit tests
run: |
./build-release/bin/valkey-unit-tests
run: make -C build-release test-unit
test-sanitizer-address:
runs-on: ubuntu-latest
@@ -128,7 +132,9 @@ jobs:
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
# build with TLS module just for compilation coverage
run: make -j4 all-with-unit-tests SANITIZER=address SERVER_CFLAGS='-Werror' BUILD_TLS=module USE_LIBBACKTRACE=yes
run: |
sudo apt-get install pkg-config libgtest-dev libgmock-dev
make -j4 all-with-unit-tests SANITIZER=address SERVER_CFLAGS='-Werror' BUILD_TLS=module USE_LIBBACKTRACE=yes
- name: testprep
run: sudo apt-get install tcl8.6 tclx -y
- name: test
@@ -136,7 +142,8 @@ jobs:
- name: module api test
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs
- name: unit tests
run: ./src/valkey-unit-tests
run: |
make test-unit
test-rdma:
runs-on: ubuntu-latest
@@ -160,6 +167,7 @@ jobs:
make -j4 BUILD_RDMA=yes USE_LIBBACKTRACE=yes
- name: clone-rxe-kmod
run: |
uname -a
mkdir -p tests/rdma/rxe
git clone https://github.com/pizhenwei/rxe.git tests/rdma/rxe
make -C tests/rdma/rxe
@@ -220,9 +228,17 @@ jobs:
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install build dependencies
run: brew install llvm googletest
- name: make
# Build with additional upcoming features
run: make -j3 all-with-unit-tests SERVER_CFLAGS='-Werror' USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
run: |
export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
export CC=/opt/homebrew/opt/llvm/bin/clang
export CXX=/opt/homebrew/opt/llvm/bin/clang++
export AR=/opt/homebrew/opt/llvm/bin/llvm-ar
export RANLIB=/opt/homebrew/opt/llvm/bin/llvm-ranlib
make -j3 all-with-unit-tests SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes LIBBACKTRACE_PREFIX=/usr/local
build-32bit:
runs-on: ubuntu-latest
@@ -240,16 +256,36 @@ jobs:
cd libbacktrace && ./configure CFLAGS="-m32" --prefix=/usr/local/libbacktrace32 && make && sudo make install
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install gtest
run: |
sudo apt-get update
sudo apt-get install libgtest-dev
mkdir -p /tmp/gtest32
cd /tmp/gtest32
cmake -B build32 \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_FLAGS="-m32" \
-DCMAKE_CXX_FLAGS="-m32" \
-DCMAKE_EXE_LINKER_FLAGS="-m32" \
/usr/src/googletest
cmake --build build32 --parallel
sudo cp build32/lib/*.a /usr/lib32/
cd $GITHUB_WORKSPACE
- name: make
# Fast float requires C++ 32-bit libraries to compile on 64-bit ubuntu
# machine i.e. "-cross" suffixed version. Cross-compiling c++ to 32-bit
# also requires multilib support for g++ compiler i.e. "-multilib"
# suffixed version of g++. g++-multilib generally includes libstdc++.
# *cross version as well, but it is also added explicitly just in case.
run: make -j4 SERVER_CFLAGS='-Werror' 32bit USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes LIBBACKTRACE_PREFIX=/usr/local/libbacktrace32
run: |
make -j4 SERVER_CFLAGS='-Werror' 32bit USE_LIBBACKTRACE=yes LIBBACKTRACE_PREFIX=/usr/local/libbacktrace32 \
GTEST_CFLAGS="-I/usr/src/googletest/googletest/include -I/usr/src/googletest/googlemock/include" \
GTEST_LIBS="/usr/lib32/libgtest.a /usr/lib32/libgmock.a"
- name: unit tests
run: |
./src/valkey-unit-tests
make test-unit \
GTEST_CFLAGS="-I/usr/src/googletest/googletest/include -I/usr/src/googletest/googlemock/include" \
GTEST_LIBS="/usr/lib32/libgtest.a /usr/lib32/libgmock.a"
build-libc-malloc:
runs-on: ubuntu-latest
@@ -264,7 +300,7 @@ jobs:
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
run: make -j4 SERVER_CFLAGS='-Werror' MALLOC=libc USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
run: make -j4 SERVER_CFLAGS='-Werror' MALLOC=libc USE_LIBBACKTRACE=yes
build-almalinux8-jemalloc:
runs-on: ubuntu-latest
@@ -278,12 +314,12 @@ jobs:
path: libbacktrace
- name: Build libbacktrace
run: |
dnf -y install epel-release gcc gcc-c++ make procps-ng which
dnf -y install epel-release gcc gcc-c++ make procps-ng which git cmake
cd libbacktrace && ./configure && make && make install
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: make
run: make -j4 SERVER_CFLAGS='-Werror' USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes
run: make -j4 SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes
format-yaml:
runs-on: ubuntu-latest
@@ -298,7 +334,7 @@ jobs:
- name: Setup YAML formatter
run: |
go install github.com/google/yamlfmt/cmd/yamlfmt@latest
go install github.com/google/yamlfmt/cmd/yamlfmt@v0.21.0
- name: Run yamlfmt
id: yamlfmt
.github/workflows/clang-format.yml
+4 -1
View File
@@ -1,5 +1,8 @@
name: Clang Format Check
permissions:
contents: read
on:
push:
paths-ignore:
@@ -39,7 +42,7 @@ jobs:
# Run clang-format and capture the diff
cd src
shopt -s globstar
clang-format-18 -i **/*.c **/*.h
clang-format-18 -i **/*.c **/*.h **/*.cpp **/*.hpp
# Capture the diff output
DIFF=$(git diff)
if [ ! -z "$DIFF" ]; then
.github/workflows/codecov.yml
+10 -1
View File
@@ -1,5 +1,8 @@
name: "Codecov"
permissions:
contents: read
# Enabling on each push is to display the coverage changes in every PR,
# where each PR needs to be compared against the coverage of the head commit
on:
@@ -32,6 +35,12 @@ jobs:
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Checkout Valkey
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install gtest
run: |
git clone --depth 1 -b v1.17.x https://github.com/google/googletest.git
cmake -S googletest -B googletest/build
cmake --build googletest/build -- -j$(nproc)
sudo cmake --install googletest/build
- name: Install lcov and run test
run: |
sudo apt-get install lcov tclx
@@ -41,4 +50,4 @@ jobs:
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2
with:
token: ${{ secrets.CODECOV_TOKEN }}
file: ./src/valkey.info
files: ./src/valkey.info
.github/workflows/codeql-analysis.yml
+3 -3
View File
@@ -40,12 +40,12 @@ jobs:
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Initialize CodeQL
uses: github/codeql-action/init@b5ebac6f4c00c8ccddb7cdcd45fdb248329f808a # v3.32.2
uses: github/codeql-action/init@c793b717bc78562f491db7b0e93a3a178b099162 # v4.32.5
with:
languages: ${{ matrix.language }}
- name: Autobuild
uses: github/codeql-action/autobuild@b5ebac6f4c00c8ccddb7cdcd45fdb248329f808a # v3.32.2
uses: github/codeql-action/autobuild@c793b717bc78562f491db7b0e93a3a178b099162 # v4.32.5
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@b5ebac6f4c00c8ccddb7cdcd45fdb248329f808a # v3.32.2
uses: github/codeql-action/analyze@c793b717bc78562f491db7b0e93a3a178b099162 # v4.32.5
.github/workflows/daily.yml
+231 -88
View File
@@ -101,8 +101,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -110,6 +110,8 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Install gtest
run: sudo apt-get install pkg-config libgtest-dev libgmock-dev
- name: make
run: make all-with-unit-tests SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes
- name: testprep
@@ -128,7 +130,12 @@ jobs:
run: ./runtest-cluster ${{github.event.inputs.cluster_test_args}}
- name: unittest
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: ./src/valkey-unit-tests --accurate
run: |
if [ -f ./src/unit/valkey-unit-gtests ]; then
make test-unit accurate=1
elif [ -f ./src/valkey-unit-tests ]; then
./src/valkey-unit-tests --accurate
fi
- name: install Redis OSS 6.2 server for compatibility testing
run: |
cd tests/tmp
@@ -180,8 +187,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -189,6 +196,8 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Install gtest
run: sudo apt-get install pkg-config libgtest-dev libgmock-dev
- name: make
run: make all-with-unit-tests SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes
- name: testprep
@@ -207,7 +216,12 @@ jobs:
run: ./runtest-cluster ${{github.event.inputs.cluster_test_args}}
- name: unittest
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: ./src/valkey-unit-tests --accurate
run: |
if [ -f ./src/unit/valkey-unit-gtests ]; then
make test-unit accurate=1
elif [ -f ./src/valkey-unit-tests ]; then
./src/valkey-unit-tests --accurate
fi
test-ubuntu-jemalloc-fortify:
runs-on: ubuntu-latest
if: |
@@ -234,8 +248,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -244,9 +258,11 @@ jobs:
path: libbacktrace
- name: Build libbacktrace
run: |
apt-get update && apt-get install -y make gcc-13
apt-get update && apt-get install -y make gcc-13 git cmake g++ python3
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 100
cd libbacktrace && ./configure && make && make install
- name: Install gtest
run: apt-get install -y pkg-config libgtest-dev libgmock-dev
- name: make
run: make all-with-unit-tests CC=gcc OPT=-O3 SERVER_CFLAGS='-Werror -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3' USE_LIBBACKTRACE=yes
- name: testprep
@@ -265,7 +281,12 @@ jobs:
run: ./runtest-cluster ${{github.event.inputs.cluster_test_args}}
- name: unittest
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: ./src/valkey-unit-tests --accurate
run: |
if [ -f ./src/unit/valkey-unit-gtests ]; then
make test-unit accurate=1
elif [ -f ./src/valkey-unit-tests ]; then
./src/valkey-unit-tests --accurate
fi
test-ubuntu-libc-malloc:
runs-on: ubuntu-latest
if: |
@@ -291,8 +312,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -341,8 +362,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -391,8 +412,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -401,10 +422,28 @@ jobs:
path: libbacktrace
- name: Build libbacktrace
run: |
sudo apt-get update && sudo apt-get install libc6-dev-i386
sudo apt-get update && sudo apt-get install libc6-dev-i386 g++-multilib
cd libbacktrace && ./configure && make && sudo make install
- name: Install gtest
run: |
sudo apt-get update
sudo apt-get install libgtest-dev
mkdir -p /tmp/gtest32
cd /tmp/gtest32
cmake -B build32 \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_FLAGS="-m32" \
-DCMAKE_CXX_FLAGS="-m32" \
-DCMAKE_EXE_LINKER_FLAGS="-m32" \
/usr/src/googletest
cmake --build build32 --parallel
sudo cp build32/lib/*.a /usr/lib32/
cd $GITHUB_WORKSPACE
- name: make
run: make 32bit SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes
run: |
make 32bit SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes \
GTEST_CFLAGS="-I/usr/src/googletest/googletest/include -I/usr/src/googletest/googlemock/include" \
GTEST_LIBS="/usr/lib32/libgtest.a /usr/lib32/libgmock.a"
- name: testprep
run: sudo apt-get install tcl8.6 tclx
- name: test
@@ -423,7 +462,14 @@ jobs:
run: ./runtest-cluster ${{github.event.inputs.cluster_test_args}}
- name: unittest
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: ./src/valkey-unit-tests --accurate
run: |
if [ -f ./src/unit/valkey-unit-gtests ]; then
make test-unit accurate=1 \
GTEST_CFLAGS="-I/usr/src/googletest/googletest/include -I/usr/src/googletest/googlemock/include" \
GTEST_LIBS="/usr/lib32/libgtest.a /usr/lib32/libgmock.a"
elif [ -f ./src/valkey-unit-tests ]; then
./src/valkey-unit-tests --accurate
fi
test-ubuntu-tls:
runs-on: ubuntu-latest
if: |
@@ -449,8 +495,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -505,8 +551,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -515,7 +561,7 @@ jobs:
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: make
run: make BUILD_TLS=yes SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes USE_FAST_FLOAT=yes
run: make BUILD_TLS=yes SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes
- name: testprep
run: |
sudo apt-get install tcl8.6 tclx tcl-tls
@@ -561,8 +607,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -605,8 +651,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -653,8 +699,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -737,8 +783,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -775,8 +821,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -784,6 +830,8 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Install gtest
run: sudo apt-get install pkg-config libgtest-dev libgmock-dev
- name: make
run: make valgrind all-with-unit-tests SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes
- name: testprep
@@ -796,8 +844,13 @@ jobs:
- name: unittest
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: |
valgrind --track-origins=yes --suppressions=./src/valgrind.sup --show-reachable=no --show-possibly-lost=no --leak-check=full --log-file=err.txt ./src/valkey-unit-tests --valgrind
if grep -q 0x err.txt; then cat err.txt; exit 1; fi
if [ -f ./src/unit/valkey-unit-gtests ]; then
./deps/gtest-parallel/gtest-parallel valgrind -- --track-origins=yes --suppressions=./src/valgrind.sup --show-reachable=no --show-possibly-lost=no --leak-check=full --log-file=err.%p.txt ./src/unit/valkey-unit-gtests --valgrind
if grep -qlE '0x[0-9A-Fa-f]+:' err.*.txt 2>/dev/null; then grep -lE '0x[0-9A-Fa-f]+:' err.*.txt | xargs cat; exit 1; fi
elif [ -f ./src/valkey-unit-tests ]; then
valgrind --track-origins=yes --suppressions=./src/valgrind.sup --show-reachable=no --show-possibly-lost=no --leak-check=full --log-file=err.txt ./src/valkey-unit-tests --valgrind
if grep -q 0x err.txt; then cat err.txt; exit 1; fi
fi
test-valgrind-no-malloc-usable-size-test:
runs-on: ubuntu-latest
if: |
@@ -818,8 +871,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -856,8 +909,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -865,6 +918,8 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Install gtest
run: sudo apt-get install pkg-config libgtest-dev libgmock-dev
- name: make
run: make valgrind all-with-unit-tests CFLAGS="-DNO_MALLOC_USABLE_SIZE" SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes
- name: testprep
@@ -877,8 +932,13 @@ jobs:
- name: unittest
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: |
valgrind --track-origins=yes --suppressions=./src/valgrind.sup --show-reachable=no --show-possibly-lost=no --leak-check=full --log-file=err.txt ./src/valkey-unit-tests --valgrind
if grep -q 0x err.txt; then cat err.txt; exit 1; fi
if [ -f ./src/unit/valkey-unit-gtests ]; then
./deps/gtest-parallel/gtest-parallel valgrind -- --track-origins=yes --suppressions=./src/valgrind.sup --show-reachable=no --show-possibly-lost=no --leak-check=full --log-file=err.%p.txt ./src/unit/valkey-unit-gtests --valgrind
if grep -qlE '0x[0-9A-Fa-f]+:' err.*.txt 2>/dev/null; then grep -lE '0x[0-9A-Fa-f]+:' err.*.txt | xargs cat; exit 1; fi
elif [ -f ./src/valkey-unit-tests ]; then
valgrind --track-origins=yes --suppressions=./src/valgrind.sup --show-reachable=no --show-possibly-lost=no --leak-check=full --log-file=err.txt ./src/valkey-unit-tests --valgrind
if grep -q 0x err.txt; then cat err.txt; exit 1; fi
fi
test-sanitizer-address:
runs-on: ubuntu-latest
if: |
@@ -905,8 +965,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -914,6 +974,8 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Install gtest
run: sudo apt-get install pkg-config libgtest-dev libgmock-dev
- name: make
run: make all-with-unit-tests OPT=-O3 SANITIZER=address SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes
- name: testprep
@@ -934,7 +996,15 @@ jobs:
run: ./runtest-cluster ${{github.event.inputs.cluster_test_args}}
- name: unittest
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: ./src/valkey-unit-tests
run: |
if [ -f ./src/unit/valkey-unit-gtests ]; then
make test-unit
elif [ -f ./src/valkey-unit-tests ]; then
./src/valkey-unit-tests
fi
# Large-memory tests with sanitizers require 10-14GB RAM due to ASAN/UBSAN overhead.
# GitHub-hosted runners for public repos provide 16GB (ubuntu-latest).
# These tests are borderline - monitoring memory usage to determine if they can run reliably.
test-sanitizer-address-large-memory:
runs-on: ubuntu-latest
if: |
@@ -960,25 +1030,53 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- name: Log runner memory
run: free -h
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install gtest
run: sudo apt-get install pkg-config libgtest-dev libgmock-dev
- name: make
run: make all-with-unit-tests OPT=-O3 SANITIZER=address SERVER_CFLAGS='-Werror'
- name: testprep
run: |
sudo apt-get update
sudo apt-get install tcl8.6 tclx -y
- name: Start memory monitor
run: |
# Track minimum free memory to detect OOM risk
(while true; do
FREE=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
echo "$FREE" >> /tmp/memfree.log
sleep 5
done) &
echo $! > /tmp/memmon.pid
- name: unittest
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: ./src/valkey-unit-tests --large-memory
run: |
if [ -f ./src/unit/valkey-unit-gtests ]; then
make test-unit large_memory=1
elif [ -f ./src/valkey-unit-tests ]; then
./src/valkey-unit-tests --large-memory
fi
- name: large memory tests
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: ./runtest --accurate --verbose --dump-logs --clients 5 --large-memory --tags large-memory ${{github.event.inputs.test_args}}
run: ./runtest --accurate --verbose --dump-logs --clients 1 --large-memory --tags large-memory ${{github.event.inputs.test_args}}
- name: large memory module api tests
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs --clients 5 --large-memory --tags large-memory ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs --clients 1 --large-memory --tags large-memory ${{github.event.inputs.test_args}}
- name: Memory usage summary
if: always()
run: |
kill $(cat /tmp/memmon.pid) 2>/dev/null || true
echo "=== Memory Summary ==="
printf "Total RAM: %.1fGB\n" $(awk '/MemTotal/ {print $2/1024/1024}' /proc/meminfo)
if [ -f /tmp/memfree.log ]; then
MIN_FREE=$(sort -n /tmp/memfree.log | head -1)
printf "Minimum free memory: %.1fGB\n" $(echo "$MIN_FREE/1024/1024" | bc -l)
fi
test-sanitizer-undefined:
runs-on: ubuntu-latest
if: |
@@ -1005,8 +1103,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -1014,6 +1112,8 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Install gtest
run: sudo apt-get install pkg-config libgtest-dev libgmock-dev
- name: make
run: make all-with-unit-tests OPT=-O3 SANITIZER=undefined SERVER_CFLAGS='-Werror' LUA_DEBUG=yes USE_LIBBACKTRACE=yes # we (ab)use this flow to also check Lua C API violations
- name: testprep
@@ -1034,7 +1134,15 @@ jobs:
run: ./runtest-cluster ${{github.event.inputs.cluster_test_args}}
- name: unittest
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: ./src/valkey-unit-tests --accurate
run: |
if [ -f ./src/unit/valkey-unit-gtests ]; then
make test-unit accurate=1
elif [ -f ./src/valkey-unit-tests ]; then
./src/valkey-unit-tests --accurate
fi
# Large-memory tests with sanitizers require 10-14GB RAM due to ASAN/UBSAN overhead.
# GitHub-hosted runners for public repos provide 16GB (ubuntu-latest).
# These tests are borderline - monitoring memory usage to determine if they can run reliably.
test-sanitizer-undefined-large-memory:
runs-on: ubuntu-latest
if: |
@@ -1060,25 +1168,53 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- name: Log runner memory
run: free -h
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install gtest
run: sudo apt-get install pkg-config libgtest-dev libgmock-dev
- name: make
run: make all-with-unit-tests OPT=-O3 SANITIZER=undefined SERVER_CFLAGS='-Werror' LUA_DEBUG=yes
- name: testprep
run: |
sudo apt-get update
sudo apt-get install tcl8.6 tclx -y
- name: Start memory monitor
run: |
# Track minimum free memory to detect OOM risk
(while true; do
FREE=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
echo "$FREE" >> /tmp/memfree.log
sleep 5
done) &
echo $! > /tmp/memmon.pid
- name: unittest
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: ./src/valkey-unit-tests --accurate --large-memory
run: |
if [ -f ./src/unit/valkey-unit-gtests ]; then
make test-unit accurate=1 large_memory=1
elif [ -f ./src/valkey-unit-tests ]; then
./src/valkey-unit-tests --accurate --large-memory
fi
- name: large memory tests
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: ./runtest --accurate --verbose --dump-logs --clients 5 --large-memory --tags large-memory ${{github.event.inputs.test_args}}
run: ./runtest --accurate --verbose --dump-logs --clients 1 --large-memory --tags large-memory ${{github.event.inputs.test_args}}
- name: large memory module api tests
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs --clients 5 --large-memory --tags large-memory ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs --clients 1 --large-memory --tags large-memory ${{github.event.inputs.test_args}}
- name: Memory usage summary
if: always()
run: |
kill $(cat /tmp/memmon.pid) 2>/dev/null || true
echo "=== Memory Summary ==="
printf "Total RAM: %.1fGB\n" $(awk '/MemTotal/ {print $2/1024/1024}' /proc/meminfo)
if [ -f /tmp/memfree.log ]; then
MIN_FREE=$(sort -n /tmp/memfree.log | head -1)
printf "Minimum free memory: %.1fGB\n" $(echo "$MIN_FREE/1024/1024" | bc -l)
fi
test-sanitizer-force-defrag:
runs-on: ubuntu-latest
if: |
@@ -1101,8 +1237,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -1110,6 +1246,8 @@ jobs:
ref: b9e40069c0b47a722286b94eb5231f7f05c08713
path: libbacktrace
- run: cd libbacktrace && ./configure && make && sudo make install
- name: Install gtest
run: sudo apt-get install pkg-config libgtest-dev libgmock-dev
- name: make
run: make all-with-unit-tests OPT=-O3 SANITIZER=address DEBUG_FORCE_DEFRAG=yes USE_JEMALLOC=no SERVER_CFLAGS='-Werror' USE_LIBBACKTRACE=yes
- name: testprep
@@ -1130,7 +1268,12 @@ jobs:
run: ./runtest-cluster ${{github.event.inputs.cluster_test_args}}
- name: unittest
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: ./src/valkey-unit-tests
run: |
if [ -f ./src/unit/valkey-unit-gtests ]; then
make test-unit
elif [ -f ./src/valkey-unit-tests ]; then
./src/valkey-unit-tests
fi
test-ubuntu-lttng:
runs-on: ubuntu-latest
if: |
@@ -1156,8 +1299,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -1203,7 +1346,7 @@ jobs:
container: almalinux:8
install_epel: true
- name: test-almalinux9-jemalloc
container: almalinux:8
container: almalinux:9
install_epel: true
- name: test-centosstream9-jemalloc
container: quay.io/centos/centos:stream9
@@ -1228,8 +1371,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install EPEL
if: matrix.install_epel
run: dnf -y install epel-release
@@ -1278,7 +1421,7 @@ jobs:
container: almalinux:8
install_epel: true
- name: test-almalinux9-tls-module
container: almalinux:8
container: almalinux:9
install_epel: true
- name: test-centosstream9-tls-module
container: quay.io/centos/centos:stream9
@@ -1303,8 +1446,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install EPEL
if: matrix.install_epel
run: dnf -y install epel-release
@@ -1354,7 +1497,7 @@ jobs:
container: almalinux:8
install_epel: true
- name: test-almalinux9-tls-module-no-tls
container: almalinux:8
container: almalinux:9
install_epel: true
- name: test-centosstream9-tls-module-no-tls
container: quay.io/centos/centos:stream9
@@ -1379,8 +1522,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install EPEL
if: matrix.install_epel
run: dnf -y install epel-release
@@ -1441,8 +1584,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -1483,8 +1626,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -1522,8 +1665,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -1568,8 +1711,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -1600,8 +1743,8 @@ jobs:
echo "GITHUB_HEAD_REF=${{inputs.use_git_ref || github.event.inputs.use_git_ref}}" >> $GITHUB_ENV
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: test
uses: cross-platform-actions/action@39a2a80642eca0947594ad03e4355dc3d28c617a # v0.32.0
with:
@@ -1638,8 +1781,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -1691,8 +1834,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
@@ -1744,8 +1887,8 @@ jobs:
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_REPOSITORY) || github.repository }}
ref: ${{ ((github.event_name == 'workflow_dispatch' || github.event_name == 'workflow_call') && env.GITHUB_HEAD_REF) || github.ref }}
repository: ${{ inputs.use_repo || github.event.inputs.use_repo || github.repository }}
ref: ${{ inputs.use_git_ref || github.event.inputs.use_git_ref || github.ref }}
- name: Install libbacktrace
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
.github/workflows/reply-schemas-linter.yml
+3 -2
View File
@@ -23,7 +23,8 @@ jobs:
- name: Setup nodejs
uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # v6.2.0
- name: Install packages
run: npm install ajv
working-directory: utils/reply-schema-linter
run: npm ci --ignore-scripts
- name: linter
run: node ./utils/reply_schema_linter.js
run: NODE_PATH=utils/reply-schema-linter/node_modules node ./utils/reply_schema_linter.js
.github/workflows/scorecard.yml
+41
View File
@@ -0,0 +1,41 @@
name: OpenSSF Scorecard supply-chain security
on:
push:
branches: [unstable]
schedule:
- cron: '0 0 * * 1'
permissions: read-all
jobs:
analysis:
name: Scorecard analysis
runs-on: ubuntu-latest
permissions:
security-events: write
id-token: write
steps:
- name: "Checkout code"
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd #v6.0.2
with:
persist-credentials: false
- name: "Run analysis"
uses: ossf/scorecard-action@4eaacf0543bb3f2c246792bd56e8cdeffafb205a #v2.4.3
with:
results_file: results.sarif
results_format: sarif
publish_results: true
- name: "Upload artifact"
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f #v6.0.0
with:
name: SARIF file
path: results.sarif
retention-days: 5
- name: "Upload to code-scanning"
uses: github/codeql-action/upload-sarif@6bc82e05fd0ea64601dd4b465378bbcf57de0314 #v4.32.1
with:
sarif_file: results.sarif
.github/workflows/trigger-build-release.yml
+3
View File
@@ -1,5 +1,8 @@
name: Trigger Build Release
permissions:
contents: read
on:
release:
types: [published]
.gitignore
+3 -1
View File
@@ -15,7 +15,7 @@ dump*.rdb
*-cli
*-sentinel
*-server
*-unit-tests
*-unit-gtests
doc-tools
release
misc/*
@@ -56,3 +56,5 @@ build-debug/
build-release/
cmake-build-debug/
cmake-build-release/
__pycache__
src/unit/.flags
00-RELEASENOTES
+192 -11
View File
@@ -1,16 +1,197 @@
Hello! This file is just a placeholder, since this is the "unstable" branch
of Valkey, the place where all the development happens.
Valkey 9.1 release notes
========================
There is no release notes for this branch, it gets forked into another branch
every time there is a partial feature freeze in order to eventually create
a new stable release.
Upgrade urgency levels:
Usually "unstable" is stable enough for you to use it in development environments
however you should never use it in production environments. It is possible
to download the latest stable release here:
| Level | Meaning |
|----------|---------------------------------------------------------------------|
| LOW | No need to upgrade unless there are new features you want to use. |
| MODERATE | Program an upgrade of the server, but it's not urgent. |
| HIGH | There is a critical bug that may affect a subset of users. Upgrade! |
| CRITICAL | There is a critical bug affecting MOST USERS. Upgrade ASAP. |
| SECURITY | There are security fixes in the release. |
https://valkey.io/download/
Valkey 9.1.0-rc2 - Released Tue Apr 28 00:00:00 2026
-------------------------------------
More information is available at https://valkey.io
Upgrade urgency LOW: This is the second release candidate of Valkey 9.1.0.
Happy hacking!
### Behavior Changes
* Revert strict TLS certificate validation at config load as it is a breaking change, deferred to next major version (#3572)
### New Features and enhanced behavior
* Do the failover immediately if the replica is the best ranked replica by @enjoy-binbin (#2227)
* Add `cluster-config-save-behavior` option to control nodes.conf save behavior by @enjoy-binbin (#3372)
* Lua scripting engine is now statically linked by default instead of dynamically linked by @eifrah-aws (#3392)
* Module command result callback addition by @martinrvisser (#2936)
### Performance and Efficiency improvements
* Redesign IO threading communication model with lock-free queues (8-17% throughput gain) by @akashkgit (#3324)
* Increase embedded string threshold from 64 to 128 bytes (30% GET throughput gain) by @Nikhil-Manglore (#3397)
* ARM NEON SIMD optimization for pvFind() in vset.c (2-3x speedup) by @ahmadbelb (#3033)
* Optimize WATCH duplicate key check from O(N) to O(1) using per-db hashtable by @enjoy-binbin (#3360)
* Optimize `CLUSTERSCAN MATCH` so that it uses a specific slot if given by @nmvk (#3380)
* Improve COB memory tracking with copy avoidance by @dvkashapov (#3306)
### Bug Fixes
* Fix `valkey-cli --cluster del-node` for unreachable nodes by @yang-z-o (#3209)
* Enhance cluster stale packet detection to prevent sub-replica and empty primary by @zhijun42 (#2811)
* Big endian bitmap byte order mismatch fix by @nmvk (#3401)
* Fix slot-migration-max-failover-repl-bytes unable to accept -1 by @enjoy-binbin (#3443)
* Fix config rewrite producing negative values for unsigned memory configs by @enjoy-binbin (#3440)
* Fix HPERSIST RESP protocol violation on wrong-type key by @madolson (#3516)
* Fix lua-enable-insecure-api default value cannot be changed to yes by @enjoy-binbin (#3548)
Valkey 9.1.0-rc1 - Released Tue Mar 17 00:00:00 2026
-------------------------------------
Upgrade urgency LOW: This is the first release candidate of Valkey 9.1.0, with
new features, performance improvements, and bug fixes.
### New Features and enhanced behavior
* Database-level access control by @dvkashapov (#2309)
* Move Lua scripting engine into a Valkey module by @rjd15372 (#2858)
* Support cross node consistency for `SCAN` commands through configurable DB hash seed by @sarthakaggarwal97 (#2608)
* Support automatic TLS reload by @yang-z-o (#3020)
* Support TLS authentication using SAN URI by @yang-z-o (#3078)
* Prevent invalid TLS certificates from being loaded by @yang-z-o (#2999)
* Failing to save the cluster config file will no longer exit the process by @enjoy-binbin (#1032)
### Command and API updates
* Makes `CLUSTER KEYSLOT` available in standalone mode by @stockholmux (#3040)
* Update HSETEX so that it always issue keyspace notifications after validation by @ranshid (#3001)
* Adds HGETDEL command by @roshkhatri (#2851)
* Support NX/XX flag in HSETEX command by @hanxizh9910 (#2668)
* New `CLUSTERSCAN` command for cluster-wide key scanning across nodes by @nmvk (#2934)
* New `MSETEX` command to set multiple keys with a shared expiration by @enjoy-binbin (#3121)
* `CLUSTER SHARDS`/`CLUSTER SLOTS` now include an `availability-zone` field by @bandalgomsu (#3156)
### Performance and Efficiency improvements
* Optimize zset memory usage by embedding element in skiplist by @chzhoo (#2508)
* Remove internal server object pointer overhead in small strings by @rainsupreme (#2516)
* Optimize skiplist query efficiency by embedding the skiplist header by @chzhoo (#2867)
* Improve performance during rehashing by @chzhoo (#3073)
* Optimize SREM/ZREM/HDEL to pause auto shrink when deleting multiple items by @enjoy-binbin (#3144)
* Abort and swap the tables if ht1 is very full during the hashtable shrink rehashing by @enjoy-binbin (#3175)
* Improve performance of copy avoidance when command reply tracking is disabled by @dvkashapov (#3086)
* Enable hardware clock by default by @dvkashapov (#3103)
* Improve `COMMAND` performance by caching responses by @ebarskiy (#2839)
* Add support for asynchronous freeing of keys on writable replicas by @Scut-Corgis (#2849)
* Faster `XRANGE`/`XREVRANGE` via stream range hot-path optimization by @nesty92 (#3002)
* Replicas can reuse the RDB file as AOF preamble after disk-based full sync by @RayaCoo (#1901)
### Module API changes
* Add ValkeyModule_ClusterKeySlotC by @bandalgomsu (#2984)
* Add more client info flags to module API by @martinrvisser (#2868)
* Add prefix-aware ACL permission checks and new module API by @eifrah-aws (#2796)
* Support unsigned 64-bit numeric config values in module API by @artikell (#1546)
### Observability and logging
* Cumulative metrics for active I/O threads usage by @deepakrn (#2463)
* Cumulative metric for active main thread usage by @dvkashapov (#2931)
* Support whole cluster info for INFO command in cluster_info section by @soloestoy, @ranshid (#2876, #2964)
* Add remaining_repl_size field in `CLUSTER GETSLOTMIGRATIONS` output by @enjoy-binbin (#3135)
* Add logging helper function to print node's ip:port when nodename not explicitly set by @zhijun42 (#2777)
* Dual-channel-replication announces itself at replica-announce-ip if configured by @jdheyburn (#2846)
* Show replica dual-channel replication buffer memory in `INFO MEMORY` and `MEMORY STATS` by @enjoy-binbin (#2924)
* Add `rdb_transmitted` state to replica state in `INFO` by @enjoy-binbin (#2833)
* New INFO section for scripting engines by @rjd15372 (#2738)
* Adding json support for log-format config by @jbergstroem (#1791)
* Add server side TLS certificate expiry tracking and INFO telemetry by @YiwenZhang12 (#2913)
* Add option to use libbacktrace for backtraces in crash reports by @rainsupreme (#3034)
### Build and Tooling
* Show RPS histogram in valkey-benchmark by @hanxizh9910 (#2471)
* Add --warmup and --duration parameters to valkey-benchmark by @rainsupreme (#2581)
* Lazy loading of RDMA libs in CLI/Benchmark when building as module by @Ada-Church-Closure (#3072)
* Add support for atomic slot migration to valkey-cli by @murphyjacob4 (#2755)
* Replace C++ fast_float dependency with pure C implementation (ffc) by @lemire (#3329)
### Bug Fixes
* Strictly check CRLF when parsing querybuf by @enjoy-binbin (#2872)
### Contributors
* Adam Fowler @adam-fowler
* Aditya Teltia @AdityaTeltia
* Alina Liu @asagege
* Allen Samuels @allenss-amazon
* Alon Arenberg @alon-arenberg
* aradz44 @aradz44
* Arthur Lee @arthurkiller
* bandalgomsu @bandalgomsu
* Baswanth @baswanth09
* Benson-li @li-benson
* Binbin @enjoy-binbin
* Björn Svensson @bjosv
* bpint @bpint
* chzhoo @chzhoo
* cjx-zar @cjx-zar
* Daniil Kashapov @dvkashapov
* Deepak Nandihalli @deepakrn
* Diego Ciciani @diegociciani
* eifrah-aws @eifrah-aws
* Evgeny Barskiy @ebarskiy
* Gabi Ganam @gabiganam
* Gagan H R @gaganhr94
* Hanxi Zhang @hanxizh9910
* Harkrishn Patro @hpatro
* Harry Lin @harrylin98
* hieu2102 @hieu2102
* Jacob Murphy @murphyjacob4
* jiegang0219 @jiegang0219
* Jim Brunner @JimB123
* Johan Bergström @jbergstroem
* John @johnufida
* Joseph Heyburn @jdheyburn
* Katie Holly @Fusl
* Ken @otherscase
* korjeek @korjeek
* Kurt McKee @kurtmckee
* Kyle J. Davis @stockholmux
* Kyle Kim @kyle-yh-kim
* Leon Anavi @leon-anavi
* Madelyn Olson @madolson
* Mangat Singh Toor @immangat
* Marc Jakobi @mrcjkb
* martinrvisser @martinrvisser
* Marvin Rösch @marvinroesch
* Murad Shahmammadli @MuradSh
* NAM UK KIM @namuk2004
* Nikhil Manglore @Nikhil-Manglore
* Ouri Half @ouriamzn
* Patrik Hermansson @phermansson
* Ping Xie @PingXie
* Quanye Yang @Ada-Church-Closure
* Rain Valentine @rainsupreme
* Ran Shidlansik @ranshid
* Ricardo Dias @rjd15372
* Ritoban Dutta @ritoban23
* Roshan Khatri @roshkhatri
* ruihong123 @ruihong123
* Sachin Venkatesha Murthy @sachinvmurthy
* Sarthak Aggarwal @sarthakaggarwal97
* Satheesha CH Gowda @satheesha
* Seungmin Lee @sungming2
* Shinobu Nunotaba @Ada-Church-Closure
* Simon Baatz @gmbnomis
* skyfirelee @artikell
* Sourav Singh Rawat @frostzt
* stydxm @stydxm
* Ted Lyngmo @TedLyngmo
* Tony Wooster @twooster
* uriyage @uriyage
* Vadym Khoptynets @poiuj
* Venkat Pamulapati @ChiliPaneer
* Viktor Söderqvist @zuiderkwast
* Vitah Lin @vitahlin
* Vitali @VitalyAR
* withRiver @withRiver
* wxmzy88 @wxmzy88
* xbasel @xbasel
* Yair Gottdenker @yairgott
* Yana Molodetsky @yanamolo
* Yang Zhao @yang-z-o
* Yiwen Zhang @YiwenZhang12
* yzc-yzc @yzc-yzc
* zhaozhao.zz @soloestoy
* Zhijun Liao @zhijun42
AGENTS.md
+56
View File
@@ -0,0 +1,56 @@
# AGENTS.md
## Scope
- These instructions apply to the entire repository unless a deeper `AGENTS.md` overrides them.
## Repo overview
- This is the Valkey server codebase.
- The main implementation lives under `src/`.
- Unit tests live under `src/unit`
- Integration tests live under `tests/`.
- Top-level `Makefile` forwards most targets into `src/Makefile`.
## Working guidelines
- Keep changes minimal and easy to backport.
- Match the style of the surrounding code instead of introducing new patterns.
- Avoid unrelated refactors in the same change.
## Build
- Default build: `make`
- Clean rebuild when build settings or bundled deps change: `make distclean && make`
## Unit tests
- Unit tests live under `src/unit/` and use GoogleTest (gtest/gmock).
- Build and run all unit tests: `make -C src test-unit`
- Unit tests cover data-structure and low-level logic changes.
- A single test filter can be run with: `make -C src test-unit && ./src/unit/valkey-unit-gtests --gtest_filter='<TestSuite>.<TestName>'`
## Integration tests
- Integration tests live under `tests/` and are written in Tcl.
- Run the full integration suite: `make test` (from the repo root).
- Run a single test file: `./runtest --single <path/to/test.tcl>`
- Additional specialized suites:
- Cluster tests: `./runtest-cluster`
- Sentinel tests: `./runtest-sentinel`
- Module API tests: `./runtest-moduleapi`
- For targeted validation, run the smallest relevant test scope first before broader suites.
## Code style
- Follow the repository conventions described in `DEVELOPMENT_GUIDE.md`.
- Most formatting is enforced by `clang-format`.
- CI uses `clang-format-18` across `*.c`, `*.h`, `*.cpp`, and `*.hpp` files.
- When touching C/C++ sources or headers, run `clang-format-18 -i` on the modified files before finalizing when the tool is available.
- Use comments for non-obvious behavior and rationale, not for restating code.
## Tests
- Code changes should include relevant tests when the repo already has a matching test location.
- Data-structure and low-level logic changes usually belong in `src/unit/` (C++ gtest).
- End-to-end behavior changes usually belong in `tests/` (Tcl integration tests).
- If behavior or commands change, check whether related documentation also needs updating.
## Files to avoid touching unless required
- Do not commit local runtime artifacts such as `dump.rdb`, `nodes.conf`, `*.log`, or ad hoc cluster directories unless the task explicitly requires them.
- Treat vendored dependency code under `deps/` as special-case changes; modify it only when the task clearly requires it.
## Pull Requests
Always push to the user's fork. Never push to the upstream valkey-io/valkey repository. Never push directly to unstable. If a user fork does not exist, ask the contributor to create one.
CMakeLists.txt
+3 -3
View File
@@ -13,8 +13,8 @@ if (APPLE)
endif ()
# Options
option(BUILD_LUA "Build Valkey Lua scripting engine" ON)
option(BUILD_UNIT_TESTS "Build valkey-unit-tests" OFF)
set(BUILD_LUA "static" CACHE STRING "Build Valkey Lua scripting engine: static (default), module, no")
option(BUILD_UNIT_GTESTS "Build valkey-unit-gtests" OFF)
option(BUILD_TEST_MODULES "Build all test modules" OFF)
option(BUILD_EXAMPLE_MODULES "Build example modules" OFF)
@@ -40,7 +40,7 @@ unset(CLANGPP CACHE)
unset(CLANG CACHE)
unset(BUILD_RDMA_MODULE CACHE)
unset(BUILD_TLS_MODULE CACHE)
unset(BUILD_UNIT_TESTS CACHE)
unset(BUILD_UNIT_GTESTS CACHE)
unset(BUILD_TEST_MODULES CACHE)
unset(BUILD_EXAMPLE_MODULES CACHE)
unset(USE_TLS CACHE)
MAINTAINERS.md
+2 -1
View File
@@ -16,7 +16,7 @@ Maintainers listed in alphabetical order by their github ID.
| ------------------- | ------------- | ----------- |
| Binbin Zhu | @enjoy-binbin | Tencent |
| Harkrishn Patro | @hpatro | Amazon |
| Lucas Yang | @lucasyonge | - |
| Lucas Yang | @lucasyonge | Percona |
| Madelyn Olson | @madolson | Amazon |
| Jacob Murphy | @murphyjacob4 | Google |
| Ping Xie | @pingxie | Oracle |
@@ -30,6 +30,7 @@ Committers listed in alphabetical order by their github ID.
| Committer | GitHub ID | Affiliation |
| ------------------- | ------------- | ----------- |
| Jim Brunner | @JimB123 | Amazon |
| Ricardo Dias | @rjd15372 | Percona |
## Former Maintainers and Committers
README.md
+2 -8
View File
@@ -1,4 +1,5 @@
[![codecov](https://codecov.io/gh/valkey-io/valkey/graph/badge.svg?token=KYYSJAYC5F)](https://codecov.io/gh/valkey-io/valkey)
[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/valkey-io/valkey/badge)](https://securityscorecards.dev/viewer/?uri=github.com/valkey-io/valkey)
This project was forked from the open source Redis project right before the transition to their new source available licenses.
@@ -52,13 +53,6 @@ as libsystemd-dev on Debian/Ubuntu or systemd-devel on CentOS) and run:
% make USE_SYSTEMD=yes
Since Valkey version 8.1, `fast_float` has been introduced as an optional
dependency, which can speed up sorted sets and other commands that use
the double datatype. To build with `fast_float` support, you'll need a
C++ compiler and run:
% make USE_FAST_FLOAT=yes
To build with enhanced stack traces that include file names and line numbers
for all functions (including static functions), use libbacktrace:
@@ -341,7 +335,7 @@ Other options supported by Valkey's `CMake` build system:
- `-DBUILD_RDMA=<no|module>` enable RDMA module build (only module mode supported). Default: `no`
- `-DBUILD_MALLOC=<libc|jemalloc|tcmalloc|tcmalloc_minimal>` choose the allocator to use. Default on Linux: `jemalloc`, for other OS: `libc`
- `-DBUILD_SANITIZER=<address|thread|undefined>` build with address sanitizer enabled. Default: disabled (no sanitizer)
- `-DBUILD_UNIT_TESTS=[yes|no]` when set, the build will produce the executable `valkey-unit-tests`. Default: `no`
- `-DBUILD_UNIT_GTESTS=[yes|no]` when set, the build will produce unit tests executable `valkey-unit-gtests`. Default: `no`
- `-DBUILD_TEST_MODULES=[yes|no]` when set, the build will include the modules located under the `tests/modules` folder. Default: `no`
- `-DBUILD_EXAMPLE_MODULES=[yes|no]` when set, the build will include the example modules located under the `src/modules` folder. Default: `no`
cmake/Modules/SourceFiles.cmake
+7 -1
View File
@@ -48,6 +48,7 @@ set(VALKEY_SERVER_SRCS
${CMAKE_SOURCE_DIR}/src/cluster_legacy.c
${CMAKE_SOURCE_DIR}/src/cluster_slot_stats.c
${CMAKE_SOURCE_DIR}/src/crc16.c
${CMAKE_SOURCE_DIR}/src/crc16_slottable.c
${CMAKE_SOURCE_DIR}/src/commandlog.c
${CMAKE_SOURCE_DIR}/src/eval.c
${CMAKE_SOURCE_DIR}/src/bio.c
@@ -68,6 +69,7 @@ set(VALKEY_SERVER_SRCS
${CMAKE_SOURCE_DIR}/src/sparkline.c
${CMAKE_SOURCE_DIR}/src/valkey-check-rdb.c
${CMAKE_SOURCE_DIR}/src/valkey-check-aof.c
${CMAKE_SOURCE_DIR}/src/valkey_strtod.c
${CMAKE_SOURCE_DIR}/src/geo.c
${CMAKE_SOURCE_DIR}/src/lazyfree.c
${CMAKE_SOURCE_DIR}/src/module.c
@@ -119,7 +121,8 @@ set(VALKEY_SERVER_SRCS
${CMAKE_SOURCE_DIR}/src/entry.c
${CMAKE_SOURCE_DIR}/src/vset.c
${CMAKE_SOURCE_DIR}/src/fifo.c
${CMAKE_SOURCE_DIR}/src/mutexqueue.c)
${CMAKE_SOURCE_DIR}/src/mutexqueue.c
${CMAKE_SOURCE_DIR}/src/queues.c)
# valkey-cli
@@ -131,6 +134,7 @@ set(VALKEY_CLI_SRCS
${CMAKE_SOURCE_DIR}/src/sha256.c
${CMAKE_SOURCE_DIR}/src/util.c
${CMAKE_SOURCE_DIR}/src/valkey-cli.c
${CMAKE_SOURCE_DIR}/src/valkey_strtod.c
${CMAKE_SOURCE_DIR}/src/zmalloc.c
${CMAKE_SOURCE_DIR}/src/release.c
${CMAKE_SOURCE_DIR}/src/ae.c
@@ -154,6 +158,7 @@ set(VALKEY_BENCHMARK_SRCS
${CMAKE_SOURCE_DIR}/src/sha256.c
${CMAKE_SOURCE_DIR}/src/util.c
${CMAKE_SOURCE_DIR}/src/valkey-benchmark.c
${CMAKE_SOURCE_DIR}/src/valkey_strtod.c
${CMAKE_SOURCE_DIR}/src/adlist.c
${CMAKE_SOURCE_DIR}/src/dict.c
${CMAKE_SOURCE_DIR}/src/zmalloc.c
@@ -164,6 +169,7 @@ set(VALKEY_BENCHMARK_SRCS
${CMAKE_SOURCE_DIR}/src/crc64.c
${CMAKE_SOURCE_DIR}/src/siphash.c
${CMAKE_SOURCE_DIR}/src/crc16.c
${CMAKE_SOURCE_DIR}/src/crc16_slottable.c
${CMAKE_SOURCE_DIR}/src/monotonic.c
${CMAKE_SOURCE_DIR}/src/cli_common.c
${CMAKE_SOURCE_DIR}/src/mt19937-64.c
cmake/Modules/ValkeySetup.cmake
+1 -13
View File
@@ -236,7 +236,7 @@ if (BUILDING_ARM64)
endif ()
if (APPLE)
add_valkey_server_linker_option("-rdynamic")
add_valkey_server_linker_option("-Wl,-export_dynamic")
add_valkey_server_linker_option("-ldl")
elseif (UNIX)
add_valkey_server_linker_option("-rdynamic")
@@ -330,22 +330,10 @@ if (PYTHON_EXE)
COMMAND touch ${CMAKE_BINARY_DIR}/fmtargs_generated
WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}/src")
add_custom_target(generate_fmtargs_h DEPENDS ${CMAKE_BINARY_DIR}/fmtargs_generated)
# Rule for generating test_files.h
message(STATUS "Adding target generate_test_files_h")
file(GLOB UNIT_TEST_SRCS "${CMAKE_SOURCE_DIR}/src/unit/*.c")
add_custom_command(
OUTPUT ${CMAKE_BINARY_DIR}/test_files_generated
DEPENDS "${UNIT_TEST_SRCS};${CMAKE_SOURCE_DIR}/utils/generate-unit-test-header.py"
COMMAND ${PYTHON_EXE} ${CMAKE_SOURCE_DIR}/utils/generate-unit-test-header.py
COMMAND touch ${CMAKE_BINARY_DIR}/test_files_generated
WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}/src")
add_custom_target(generate_test_files_h DEPENDS ${CMAKE_BINARY_DIR}/test_files_generated)
else ()
# Fake targets
add_custom_target(generate_commands_def)
add_custom_target(generate_fmtargs_h)
add_custom_target(generate_test_files_h)
endif ()
# Generate release.h file (always)
deps/CMakeLists.txt
Vendored
+4
View File
@@ -21,6 +21,9 @@ if (USE_RDMA) # Module or no module
set(ENABLE_RDMA
ON
CACHE BOOL "If RDMA support should be compiled or not")
if (USE_RDMA EQUAL 2) # Module
set(ENABLE_DLOPEN_RDMA ON CACHE BOOL "Build valkey_rdma with dynamic loading")
endif()
endif ()
# Let libvalkey use sds and dict provided by valkey.
set(DICT_INCLUDE_DIR ${CMAKE_SOURCE_DIR}/src)
@@ -30,6 +33,7 @@ add_subdirectory(libvalkey)
add_subdirectory(linenoise)
add_subdirectory(fpconv)
add_subdirectory(hdr_histogram)
add_subdirectory(fast_float)
# Clear any cached variables passed to libvalkey from the cache
unset(BUILD_SHARED_LIBS CACHE)
deps/Makefile
Vendored
+9 -5
View File
@@ -42,7 +42,6 @@ distclean:
-(cd jemalloc && [ -f Makefile ] && $(MAKE) distclean) > /dev/null || true
-(cd hdr_histogram && $(MAKE) clean) > /dev/null || true
-(cd fpconv && $(MAKE) clean) > /dev/null || true
-(cd fast_float_c_interface && $(MAKE) clean) > /dev/null || true
-(rm -f .make-*)
.PHONY: distclean
@@ -54,6 +53,9 @@ endif
ifneq (,$(filter $(BUILD_RDMA),yes module))
LIBVALKEY_MAKE_FLAGS += USE_RDMA=1
ifneq (,$(filter $(BUILD_RDMA),module))
LIBVALKEY_MAKE_FLAGS += USE_DLOPEN_RDMA=1
endif
endif
libvalkey: .make-prerequisites
@@ -123,8 +125,10 @@ jemalloc: .make-prerequisites
.PHONY: jemalloc
fast_float_c_interface: .make-prerequisites
gtest-parallel: .make-prerequisites
@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
cd fast_float_c_interface && $(MAKE)
.PHONY: fast_float_c_interface
@if [ ! -f gtest-parallel/gtest_parallel.py ]; then \
echo "Downloading gtest-parallel..."; \
rm -rf gtest-parallel; \
git clone --depth 1 https://github.com/google/gtest-parallel.git gtest-parallel; \
fi
deps/README.md
Vendored
+43 -12
View File
@@ -6,7 +6,8 @@ should be provided by the operating system.
* **linenoise** is a readline replacement. It is developed by the same authors of Valkey but is managed as a separated project and updated as needed.
* **lua** is Lua 5.1 with minor changes for security and additional libraries.
* **hdr_histogram** Used for per-command latency tracking histograms.
* **fast_float** is a replacement for strtod to convert strings to floats efficiently.
* **ffc.h** is a C99 port of the fast_float library, used as a replacement for strtod to convert strings to floats efficiently.
* **gtest-parallel** is a script for running googletest tests in parallel.
How to upgrade the above dependencies
===
@@ -107,17 +108,47 @@ We use a customized version based on master branch commit e4448cf6d1cd08fff51981
2. Copy updated files from newer version onto files in /hdr_histogram.
3. Apply the changes from 1 above to the updated files.
fast_float
ffc.h
---
The fast_float library provides fast header-only implementations for the C++ from_chars functions for `float` and `double` types as well as integer types. These functions convert ASCII strings representing decimal values (e.g., `1.3e10`) into binary types. The functions are much faster than comparable number-parsing functions from existing C++ standard libraries.
Specifically, `fast_float` provides the following function to parse floating-point numbers with a C++17-like syntax (the library itself only requires C++11):
template <typename T, typename UC = char, typename = FASTFLOAT_ENABLE_IF(is_supported_float_type<T>())>
from_chars_result_t<UC> from_chars(UC const *first, UC const *last, T &value, chars_format fmt = chars_format::general);
ffc.h is a pure C99 port of the fast_float library, providing fast string-to-double
conversion without requiring a C++ compiler.
To upgrade the library,
1. Check out https://github.com/fastfloat/fast_float/tree/main
2. cd fast_float
3. Invoke "python3 ./script/amalgamate.py --output fast_float.h"
4. Copy fast_float.h file to "deps/fast_float/".
1. Download the latest ffc.h from https://github.com/kolemannix/ffc.h/releases
2. Copy ffc.h to "deps/fast_float/".
gtest-parallel
---
The `deps/gtest-parallel` directory is imported from the upstream
https://github.com/google/gtest-parallel repository as a subtree snapshot (not a real Git subtree).
Current upstream version:
- Upstream commit: `cd488bd` (from google/gtest-parallel)
Updating gtest-parallel
Run the following from the repository root.
1. Add the remote and fetch upstream:
```sh
git remote add gtest-parallel https://github.com/google/gtest-parallel.git
git fetch gtest-parallel master
```
2. Remove any previous import and commit (commit A):
```sh
rm -rf deps/gtest-parallel
```
3. Update the subtree from upstream:
```sh
git subtree add --prefix=deps/gtest-parallel gtest-parallel master --squash
```
4. Reset back to commit A with proper sign-off:
```sh
git reset --soft <commit-A-hash>
```
5. Commit the changes.
deps/fast_float/CMakeLists.txt
Vendored
+2
View File
@@ -0,0 +1,2 @@
add_library(ffc INTERFACE)
target_include_directories(ffc INTERFACE ${CMAKE_CURRENT_SOURCE_DIR})
deps/fast_float/fast_float.h
Vendored
-3912
View File
File diff suppressed because it is too large Load Diff
deps/fast_float/ffc.h
Vendored
+3235
View File
File diff suppressed because one or more lines are too long
deps/fast_float_c_interface/Makefile
Vendored
-37
View File
@@ -1,37 +0,0 @@
CCCOLOR:="\033[34m"
SRCCOLOR:="\033[33m"
ENDCOLOR:="\033[0m"
CXX?=c++
# we need = instead of := so that $@ in QUIET_CXX gets evaluated in the rule and is assigned appropriate value.
TEMP:=$(CXX)
QUIET_CXX=@printf ' %b %b\n' $(CCCOLOR)C++$(ENDCOLOR) $(SRCCOLOR)$@$(ENDCOLOR) 1>&2;
CXX=$(QUIET_CXX)$(TEMP)
WARN=-Wall -W -Wno-missing-field-initializers
STD=-pedantic -std=c++11
OPT?=-O3
CLANG := $(findstring clang,$(shell sh -c '$(CC) --version | head -1'))
ifeq ($(OPT),-O3)
ifeq (clang,$(CLANG))
OPT+=-flto
else
OPT+=-flto=auto -ffat-lto-objects
endif
endif
# 1) Today src/Makefile passes -m32 flag for explicit 32-bit build on 64-bit machine, via CFLAGS. For 32-bit build on
# 32-bit machine and 64-bit on 64-bit machine, CFLAGS are empty. No other flags are set that can conflict with C++,
# therefore let's use CFLAGS without changes for now.
# 2) FASTFLOAT_ALLOWS_LEADING_PLUS allows +inf to be parsed as inf, instead of error.
CXXFLAGS=$(STD) $(OPT) $(WARN) -static -fPIC -fno-exceptions $(CFLAGS) -D FASTFLOAT_ALLOWS_LEADING_PLUS
.PHONY: all clean
all: fast_float_strtod.o
clean:
rm -f *.o || true;
deps/fast_float_c_interface/fast_float_strtod.cpp
Vendored
-24
View File
@@ -1,24 +0,0 @@
/*
* Copyright (c) Valkey Contributors
* All rights reserved.
* SPDX-License-Identifier: BSD-3-Clause
*/
#include "../fast_float/fast_float.h"
#include <cerrno>
extern "C"
{
double fast_float_strtod(const char *str, const char** endptr)
{
double temp = 0;
auto answer = fast_float::from_chars(str, str + strlen(str), temp);
if (answer.ec != std::errc()) {
errno = (answer.ec == std::errc::result_out_of_range) ? ERANGE : EINVAL;
}
if (endptr) {
*endptr = answer.ptr;
}
return temp;
}
}
deps/gtest-parallel/.gitignore
+2
View File
@@ -0,0 +1,2 @@
.*.swp
*.py[co]
deps/gtest-parallel/.style.yapf
Vendored
+4
View File
@@ -0,0 +1,4 @@
[style]
based_on_style = pep8
indent_width = 2
column_limit = 80
deps/gtest-parallel/CONTRIBUTING.md
Vendored
+28
View File
@@ -0,0 +1,28 @@
# How to Contribute
We'd love to accept your patches and contributions to this project. There are
just a few small guidelines you need to follow.
## Contributor License Agreement
Contributions to this project must be accompanied by a Contributor License
Agreement. You (or your employer) retain the copyright to your contribution;
this simply gives us permission to use and redistribute your contributions as
part of the project. Head over to <https://cla.developers.google.com/> to see
your current agreements on file or to sign a new one.
You generally only need to submit a CLA once, so if you've already submitted one
(even if it was for a different project), you probably don't need to do it
again.
## Code reviews
All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose. Consult
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
information on using pull requests.
## Community Guidelines
This project follows [Google's Open Source Community
Guidelines](https://opensource.google.com/conduct/).
deps/gtest-parallel/LICENSE
Vendored
+202
View File
@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
deps/gtest-parallel/README.md
Vendored
+90
View File
@@ -0,0 +1,90 @@
# gtest-parallel
_This is not an official Google product._
`gtest-parallel` is a script that executes [Google
Test](https://github.com/google/googletest) binaries in parallel, providing good
speedup for single-threaded tests (on multi-core machines) and tests that do not
run at 100% CPU (on single- or multi-core machines).
The script works by listing the tests of each binary, and then executing them on
workers in separate processes. This works fine so long as the tests are self
contained and do not share resources (reading data is fine, writing to the same
log file is probably not).
## Basic Usage
_For a full list of options, see `--help`._
$ ./gtest-parallel path/to/binary...
This shards all enabled tests across a number of workers, defaulting to the
number of cores in the system. If your system uses Python 2, but you have no
python2 binary, run `python gtest-parallel` instead of `./gtest-parallel`.
To run only a select set of tests, run:
$ ./gtest-parallel path/to/binary... --gtest_filter=Foo.*:Bar.*
This filter takes the same parameters as Google Test, so -Foo.\* can be used for
test exclusion as well. This is especially useful for slow tests (that you're
not working on), or tests that may not be able to run in parallel.
## Flakiness
Flaky tests (tests that do not deterministically pass or fail) often cause a lot
of developer headache. A test that fails only 1% of the time can be very hard to
detect as flaky, and even harder to convince yourself of having fixed.
`gtest-parallel` supports repeating individual tests (`--repeat=`), which can be
very useful for flakiness testing. Some tests are also more flaky under high
loads (especially tests that use realtime clocks), so raising the number of
`--workers=` well above the number of available core can often cause contention
and be fruitful for detecting flaky tests as well.
$ ./gtest-parallel out/{binary1,binary2,binary3} --repeat=1000 --workers=128
The above command repeats all tests inside `binary1`, `binary2` and `binary3`
located in `out/`. The tests are run `1000` times each on `128` workers (this is
more than I have cores on my machine anyways). This can often be done and then
left overnight if you've no initial guess to which tests are flaky and which
ones aren't. When you've figured out which tests are flaky (and want to fix
them), repeat the above command with `--gtest_filter=` to only retry the flaky
tests that you are fixing.
Note that repeated tests do run concurrently with themselves for efficiency, and
as such they have problem writing to hard-coded files, even if they are only
used by that single test. `tmpfile()` and similar library functions are often
your friends here.
### Flakiness Summaries
Especially for disabled tests, you might wonder how stable a test seems before
trying to enable it. `gtest-parallel` prints summaries (number of passed/failed
tests) when `--repeat=` is used and at least one test fails. This can be used to
generate passed/failed statistics per test. If no statistics are generated then
all invocations tests are passing, congratulations!
For example, to try all disabled tests and see how stable they are:
$ ./gtest-parallel path/to/binary... -r1000 --gtest_filter=*.DISABLED_* --gtest_also_run_disabled_tests
Which will generate something like this at the end of the run:
SUMMARY:
path/to/binary... Foo.DISABLED_Bar passed 0 / 1000 times.
path/to/binary... FooBar.DISABLED_Baz passed 30 / 1000 times.
path/to/binary... Foo.DISABLED_Baz passed 1000 / 1000 times.
## Running Tests Within Test Cases Sequentially
Sometimes tests within a single test case use globally-shared resources
(hard-coded file paths, sockets, etc.) and cannot be run in parallel. Running
such tests in parallel will either fail or be flaky (if they happen to not
overlap during execution, they pass). So long as these resources are only shared
within the same test case `gtest-parallel` can still provide some parallelism.
For such binaries where test cases are independent, `gtest-parallel` provides
`--serialize_test_cases` that runs tests within the same test case sequentially.
While generally not providing as much speedup as fully parallel test execution,
this permits such binaries to partially benefit from parallel execution.
deps/gtest-parallel/gtest-parallel
Vendored Executable
+18
View File
@@ -0,0 +1,18 @@
#!/usr/bin/env python3
# Copyright 2017 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import gtest_parallel
import sys
sys.exit(gtest_parallel.main())
deps/gtest-parallel/gtest_parallel.py
Vendored Executable
+955
View File
@@ -0,0 +1,955 @@
# Copyright 2013 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import errno
from functools import total_ordering
import gzip
import io
import json
import multiprocessing
import optparse
import os
import re
import shutil
import signal
import subprocess
import sys
import tempfile
import threading
import time
if sys.version_info.major >= 3:
long = int
import _pickle as cPickle
import _thread as thread
else:
import cPickle
import thread
from pickle import HIGHEST_PROTOCOL as PICKLE_HIGHEST_PROTOCOL
if sys.platform == 'win32':
import msvcrt
else:
import fcntl
# An object that catches SIGINT sent to the Python process and notices
# if processes passed to wait() die by SIGINT (we need to look for
# both of those cases, because pressing Ctrl+C can result in either
# the main process or one of the subprocesses getting the signal).
#
# Before a SIGINT is seen, wait(p) will simply call p.wait() and
# return the result. Once a SIGINT has been seen (in the main process
# or a subprocess, including the one the current call is waiting for),
# wait(p) will call p.terminate() and raise ProcessWasInterrupted.
class SigintHandler(object):
class ProcessWasInterrupted(Exception):
pass
sigint_returncodes = {
-signal.SIGINT, # Unix
-1073741510, # Windows
}
def __init__(self):
self.__lock = threading.Lock()
self.__processes = set()
self.__got_sigint = False
signal.signal(signal.SIGINT, lambda signal_num, frame: self.interrupt())
def __on_sigint(self):
self.__got_sigint = True
while self.__processes:
try:
self.__processes.pop().terminate()
except OSError:
pass
def interrupt(self):
with self.__lock:
self.__on_sigint()
def got_sigint(self):
with self.__lock:
return self.__got_sigint
def wait(self, p, timeout_per_test):
with self.__lock:
if self.__got_sigint:
p.terminate()
self.__processes.add(p)
try:
code = p.wait(timeout_per_test)
except subprocess.TimeoutExpired :
p.terminate()
self.__processes.remove(p)
code = -errno.ETIME
with self.__lock:
self.__processes.discard(p)
if code in self.sigint_returncodes:
self.__on_sigint()
if self.__got_sigint:
raise self.ProcessWasInterrupted
return code
sigint_handler = SigintHandler()
# Return the width of the terminal, or None if it couldn't be
# determined (e.g. because we're not being run interactively).
def term_width(out):
if not out.isatty():
return None
try:
p = subprocess.Popen(["stty", "size"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
(out, err) = p.communicate()
if p.returncode != 0 or err:
return None
return int(out.split()[1])
except (IndexError, OSError, ValueError):
return None
# Output transient and permanent lines of text. If several transient
# lines are written in sequence, the new will overwrite the old. We
# use this to ensure that lots of unimportant info (tests passing)
# won't drown out important info (tests failing).
class Outputter(object):
def __init__(self, out_file):
self.__out_file = out_file
self.__previous_line_was_transient = False
self.__width = term_width(out_file) # Line width, or None if not a tty.
def transient_line(self, msg):
if self.__width is None:
self.__out_file.write(msg + "\n")
self.__out_file.flush()
else:
self.__out_file.write("\r" + msg[:self.__width].ljust(self.__width))
self.__previous_line_was_transient = True
def flush_transient_output(self):
if self.__previous_line_was_transient:
self.__out_file.write("\n")
self.__previous_line_was_transient = False
def permanent_line(self, msg):
self.flush_transient_output()
self.__out_file.write(msg + "\n")
if self.__width is None:
self.__out_file.flush()
def get_save_file_path():
"""Return path to file for saving transient data."""
if sys.platform == 'win32':
default_cache_path = os.path.join(os.path.expanduser('~'), 'AppData',
'Local')
cache_path = os.environ.get('LOCALAPPDATA', default_cache_path)
else:
# We don't use xdg module since it's not a standard.
default_cache_path = os.path.join(os.path.expanduser('~'), '.cache')
cache_path = os.environ.get('XDG_CACHE_HOME', default_cache_path)
if os.path.isdir(cache_path):
return os.path.join(cache_path, 'gtest-parallel')
else:
sys.stderr.write('Directory {} does not exist'.format(cache_path))
return os.path.join(os.path.expanduser('~'), '.gtest-parallel-times')
@total_ordering
class Task(object):
"""Stores information about a task (single execution of a test).
This class stores information about the test to be executed (gtest binary and
test name), and its result (log file, exit code and runtime).
Each task is uniquely identified by the gtest binary, the test name and an
execution number that increases each time the test is executed.
Additionaly we store the last execution time, so that next time the test is
executed, the slowest tests are run first.
"""
def __init__(self, test_binary, test_name, test_command, execution_number,
last_execution_time, output_dir):
self.test_name = test_name
self.output_dir = output_dir
self.test_binary = test_binary
self.test_command = test_command
self.execution_number = execution_number
self.last_execution_time = last_execution_time
self.exit_code = None
self.runtime_ms = None
self.test_id = (test_binary, test_name)
self.task_id = (test_binary, test_name, self.execution_number)
self.log_file = Task._logname(self.output_dir, self.test_binary, test_name,
self.execution_number)
def __sorting_key(self):
# Unseen or failing tests (both missing execution time) take precedence over
# execution time. Tests are greater (seen as slower) when missing times so
# that they are executed first.
return (1 if self.last_execution_time is None else 0,
self.last_execution_time)
def __eq__(self, other):
return self.__sorting_key() == other.__sorting_key()
def __ne__(self, other):
return not (self == other)
def __lt__(self, other):
return self.__sorting_key() < other.__sorting_key()
@staticmethod
def _normalize(string):
return re.sub('[^A-Za-z0-9]', '_', string)
@staticmethod
def _logname(output_dir, test_binary, test_name, execution_number):
# Store logs to temporary files if there is no output_dir.
if output_dir is None:
(log_handle, log_name) = tempfile.mkstemp(prefix='gtest_parallel_',
suffix=".log")
os.close(log_handle)
return log_name
log_name = '%s-%s-%d.log' % (Task._normalize(os.path.basename(test_binary)),
Task._normalize(test_name), execution_number)
return os.path.join(output_dir, log_name)
def run(self, timeout_per_test):
begin = time.time()
with open(self.log_file, 'w') as log:
task = subprocess.Popen(self.test_command, stdout=log, stderr=log)
try:
self.exit_code = sigint_handler.wait(task, timeout_per_test)
except sigint_handler.ProcessWasInterrupted:
thread.exit()
self.runtime_ms = int(1000 * (time.time() - begin))
self.last_execution_time = None if self.exit_code else self.runtime_ms
class TaskManager(object):
"""Executes the tasks and stores the passed, failed and interrupted tasks.
When a task is run, this class keeps track if it passed, failed or was
interrupted. After a task finishes it calls the relevant functions of the
Logger, TestResults and TestTimes classes, and in case of failure, retries the
test as specified by the --retry_failed flag.
"""
def __init__(self, times, logger, test_results, task_factory, times_to_retry,
initial_execution_number):
self.times = times
self.logger = logger
self.test_results = test_results
self.task_factory = task_factory
self.times_to_retry = times_to_retry
self.initial_execution_number = initial_execution_number
self.global_exit_code = 0
self.passed = []
self.failed = []
self.started = {}
self.timed_out = []
self.execution_number = {}
self.lock = threading.Lock()
def __get_next_execution_number(self, test_id):
with self.lock:
next_execution_number = self.execution_number.setdefault(
test_id, self.initial_execution_number)
self.execution_number[test_id] += 1
return next_execution_number
def __register_start(self, task):
with self.lock:
self.started[task.task_id] = task
def register_exit(self, task):
self.logger.log_exit(task)
self.times.record_test_time(task.test_binary, task.test_name,
task.last_execution_time)
if self.test_results:
self.test_results.log(task.test_name, task.runtime_ms / 1000.0,
task.exit_code)
with self.lock:
self.started.pop(task.task_id)
if task.exit_code == 0:
self.passed.append(task)
elif task.exit_code == -errno.ETIME:
self.timed_out.append(task)
else:
self.failed.append(task)
def run_task(self, task, timeout_per_test):
for try_number in range(self.times_to_retry + 1):
self.__register_start(task)
task.run(timeout_per_test)
self.register_exit(task)
if task.exit_code == 0:
break
if try_number < self.times_to_retry:
execution_number = self.__get_next_execution_number(task.test_id)
# We need create a new Task instance. Each task represents a single test
# execution, with its own runtime, exit code and log file.
task = self.task_factory(task.test_binary, task.test_name,
task.test_command, execution_number,
task.last_execution_time, task.output_dir)
with self.lock:
if task.exit_code != 0:
self.global_exit_code = task.exit_code
class FilterFormat(object):
def __init__(self, output_dir):
if sys.stdout.isatty():
# stdout needs to be unbuffered since the output is interactive.
if isinstance(sys.stdout, io.TextIOWrapper):
# workaround for https://bugs.python.org/issue17404
sys.stdout = io.TextIOWrapper(sys.stdout.detach(),
line_buffering=True,
write_through=True,
newline='\n')
else:
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
self.output_dir = output_dir
self.total_tasks = 0
self.finished_tasks = 0
self.out = Outputter(sys.stdout)
self.stdout_lock = threading.Lock()
def move_to(self, destination_dir, tasks):
if self.output_dir is None:
return
destination_dir = os.path.join(self.output_dir, destination_dir)
os.makedirs(destination_dir)
for task in tasks:
shutil.move(task.log_file, destination_dir)
def print_tests(self, message, tasks, print_try_number, print_test_command):
self.out.permanent_line("%s (%s/%s):" %
(message, len(tasks), self.total_tasks))
for task in sorted(tasks):
runtime_ms = 'Interrupted'
if task.runtime_ms is not None:
runtime_ms = '%d ms' % task.runtime_ms
if print_test_command:
try:
cmd_str = " ".join(task.test_command)
except TypeError:
cmd_str = task.test_command
self.out.permanent_line(
"%11s: %s%s" %
(runtime_ms, cmd_str,
(" (try #%d)" % task.execution_number) if print_try_number else ""))
else:
self.out.permanent_line(
"%11s: %s %s%s" %
(runtime_ms, task.test_binary, task.test_name,
(" (try #%d)" % task.execution_number) if print_try_number else ""))
def log_exit(self, task):
with self.stdout_lock:
self.finished_tasks += 1
self.out.transient_line("[%d/%d] %s (%d ms)" %
(self.finished_tasks, self.total_tasks,
task.test_name, task.runtime_ms))
if task.exit_code != 0:
signal_name = None
if task.exit_code < 0:
try:
signal_name = signal.Signals(-task.exit_code).name
except ValueError:
pass
with open(task.log_file) as f:
for line in f.readlines():
self.out.permanent_line(line.rstrip())
if task.exit_code is None:
self.out.permanent_line("[%d/%d] %s aborted after %d ms" %
(self.finished_tasks, self.total_tasks,
task.test_name, task.runtime_ms))
elif task.exit_code == -errno.ETIME:
self.out.permanent_line(
"\033[31m[ TIMEOUT ]\033[0m %s timed out after %d s"
% (task.test_name, task.runtime_ms/1000))
elif signal_name is not None:
self.out.permanent_line(
"[%d/%d] %s killed by signal %s (%d ms)" %
(self.finished_tasks, self.total_tasks, task.test_name,
signal_name, task.runtime_ms))
else:
self.out.permanent_line(
"[%d/%d] %s returned with exit code %d (%d ms)" %
(self.finished_tasks, self.total_tasks, task.test_name,
task.exit_code, task.runtime_ms))
if self.output_dir is None:
# Try to remove the file 100 times (sleeping for 0.1 second in between).
# This is a workaround for a process handle seemingly holding on to the
# file for too long inside os.subprocess. This workaround is in place
# until we figure out a minimal repro to report upstream (or a better
# suspect) to prevent os.remove exceptions.
num_tries = 100
for i in range(num_tries):
try:
os.remove(task.log_file)
except OSError as e:
if e.errno is not errno.ENOENT:
if i is num_tries - 1:
self.out.permanent_line('Could not remove temporary log file: ' +
str(e))
else:
time.sleep(0.1)
continue
break
def log_tasks(self, total_tasks):
self.total_tasks += total_tasks
self.out.transient_line("[0/%d] Running tests..." % self.total_tasks)
def summarize(self, passed_tasks, failed_tasks, interrupted_tasks):
stats = {}
def add_stats(stats, task, idx):
task_key = (task.test_binary, task.test_name)
if not task_key in stats:
# (passed, failed, interrupted) task_key is added as tie breaker to get
# alphabetic sorting on equally-stable tests
stats[task_key] = [0, 0, 0, task_key]
stats[task_key][idx] += 1
for task in passed_tasks:
add_stats(stats, task, 0)
for task in failed_tasks:
add_stats(stats, task, 1)
for task in interrupted_tasks:
add_stats(stats, task, 2)
self.out.permanent_line("SUMMARY:")
for task_key in sorted(stats, key=stats.__getitem__):
(num_passed, num_failed, num_interrupted, _) = stats[task_key]
(test_binary, task_name) = task_key
total_runs = num_passed + num_failed + num_interrupted
if num_passed == total_runs:
continue
self.out.permanent_line(" %s %s passed %d / %d times%s." %
(test_binary, task_name, num_passed, total_runs,
"" if num_interrupted == 0 else
(" (%d interrupted)" % num_interrupted)))
def flush(self):
self.out.flush_transient_output()
class CollectTestResults(object):
def __init__(self, json_dump_filepath):
self.test_results_lock = threading.Lock()
self.json_dump_file = open(json_dump_filepath, 'w')
self.test_results = {
"interrupted": False,
"path_delimiter": ".",
# Third version of the file format. See the link in the flag description
# for details.
"version": 3,
"seconds_since_epoch": int(time.time()),
"num_failures_by_type": {
"PASS": 0,
"FAIL": 0,
"TIMEOUT": 0,
},
"tests": {},
}
def log(self, test, runtime_seconds, exit_code):
if exit_code is None:
actual_result = "TIMEOUT"
elif exit_code == 0:
actual_result = "PASS"
else:
actual_result = "FAIL"
with self.test_results_lock:
self.test_results['num_failures_by_type'][actual_result] += 1
results = self.test_results['tests']
for name in test.split('.'):
results = results.setdefault(name, {})
if results:
results['actual'] += ' ' + actual_result
results['times'].append(runtime_seconds)
else: # This is the first invocation of the test
results['actual'] = actual_result
results['times'] = [runtime_seconds]
results['time'] = runtime_seconds
results['expected'] = 'PASS'
def dump_to_file_and_close(self):
json.dump(self.test_results, self.json_dump_file)
self.json_dump_file.close()
# Record of test runtimes. Has built-in locking.
class TestTimes(object):
class LockedFile(object):
def __init__(self, filename, mode):
self._filename = filename
self._mode = mode
self._fo = None
def __enter__(self):
self._fo = open(self._filename, self._mode)
# Regardless of opening mode we always seek to the beginning of file.
# This simplifies code working with LockedFile and also ensures that
# we lock (and unlock below) always the same region in file on win32.
self._fo.seek(0)
try:
if sys.platform == 'win32':
# We are locking here fixed location in file to use it as
# an exclusive lock on entire file.
msvcrt.locking(self._fo.fileno(), msvcrt.LK_LOCK, 1)
else:
fcntl.flock(self._fo.fileno(), fcntl.LOCK_EX)
except IOError:
self._fo.close()
raise
return self._fo
def __exit__(self, exc_type, exc_value, traceback):
# Flush any buffered data to disk. This is needed to prevent race
# condition which happens from the moment of releasing file lock
# till closing the file.
self._fo.flush()
try:
if sys.platform == 'win32':
self._fo.seek(0)
msvcrt.locking(self._fo.fileno(), msvcrt.LK_UNLCK, 1)
else:
fcntl.flock(self._fo.fileno(), fcntl.LOCK_UN)
finally:
self._fo.close()
return exc_value is None
def __init__(self, save_file):
"Create new object seeded with saved test times from the given file."
self.__times = {} # (test binary, test name) -> runtime in ms
# Protects calls to record_test_time(); other calls are not
# expected to be made concurrently.
self.__lock = threading.Lock()
try:
with TestTimes.LockedFile(save_file, 'rb') as fd:
times = TestTimes.__read_test_times_file(fd)
except IOError:
# We couldn't obtain the lock.
return
# Discard saved times if the format isn't right.
if type(times) is not dict:
return
for ((test_binary, test_name), runtime) in times.items():
if (type(test_binary) is not str or type(test_name) is not str
or type(runtime) not in {int, long, type(None)}):
return
self.__times = times
def get_test_time(self, binary, testname):
"""Return the last duration for the given test as an integer number of
milliseconds, or None if the test failed or if there's no record for it."""
return self.__times.get((binary, testname), None)
def record_test_time(self, binary, testname, runtime_ms):
"""Record that the given test ran in the specified number of
milliseconds. If the test failed, runtime_ms should be None."""
with self.__lock:
self.__times[(binary, testname)] = runtime_ms
def write_to_file(self, save_file):
"Write all the times to file."
try:
with TestTimes.LockedFile(save_file, 'a+b') as fd:
times = TestTimes.__read_test_times_file(fd)
if times is None:
times = self.__times
else:
times.update(self.__times)
# We erase data from file while still holding a lock to it. This
# way reading old test times and appending new ones are atomic
# for external viewer.
fd.seek(0)
fd.truncate()
with gzip.GzipFile(fileobj=fd, mode='wb') as gzf:
cPickle.dump(times, gzf, PICKLE_HIGHEST_PROTOCOL)
except IOError:
pass # ignore errors---saving the times isn't that important
@staticmethod
def __read_test_times_file(fd):
try:
with gzip.GzipFile(fileobj=fd, mode='rb') as gzf:
times = cPickle.load(gzf)
except Exception:
# File doesn't exist, isn't readable, is malformed---whatever.
# Just ignore it.
return None
else:
return times
def find_tests(binaries, additional_args, options, times):
test_count = 0
tasks = []
for test_binary in binaries:
command = [test_binary] + additional_args
if options.gtest_also_run_disabled_tests:
command += ['--gtest_also_run_disabled_tests']
list_command = command + ['--gtest_list_tests']
if options.gtest_filter != '':
list_command += ['--gtest_filter=' + options.gtest_filter]
try:
test_list = subprocess.check_output(list_command,
stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
sys.exit("%s: %s\n%s" % (test_binary, str(e), e.output))
try:
test_list = test_list.split('\n')
except TypeError:
# subprocess.check_output() returns bytes in python3
test_list = test_list.decode(sys.stdout.encoding).split('\n')
command += ['--gtest_color=' + options.gtest_color]
test_group = ''
for line in test_list:
if not line.strip():
continue
if line[0] != " ":
# Remove comments for typed tests and strip whitespace.
test_group = line.split('#')[0].strip()
continue
# Remove comments for parameterized tests and strip whitespace.
line = line.split('#')[0].strip()
if not line:
continue
test_name = test_group + line
if not options.gtest_also_run_disabled_tests and 'DISABLED_' in test_name:
continue
# Skip PRE_ tests which are used by Chromium.
if '.PRE_' in test_name:
continue
last_execution_time = times.get_test_time(test_binary, test_name)
if options.failed and last_execution_time is not None:
continue
test_command = command + ['--gtest_filter=' + test_name]
if (test_count - options.shard_index) % options.shard_count == 0:
for execution_number in range(options.repeat):
tasks.append(
Task(test_binary, test_name, test_command, execution_number + 1,
last_execution_time, options.output_dir))
test_count += 1
# Sort the tasks to run the slowest tests first, so that faster ones can be
# finished in parallel.
return sorted(tasks, reverse=True)
def execute_tasks(tasks, pool_size, task_manager, timeout_seconds,
timeout_per_test, serialize_test_cases):
class WorkerFn(object):
def __init__(self, tasks, running_groups, timeout_per_test):
self.tasks = tasks
self.running_groups = running_groups
self.timeout_per_test = timeout_per_test
self.task_lock = threading.Lock()
def __call__(self):
while True:
with self.task_lock:
for task_id in range(len(self.tasks)):
task = self.tasks[task_id]
if self.running_groups is not None:
test_group = task.test_name.split('.')[0]
if test_group in self.running_groups:
# Try to find other non-running test group.
continue
else:
self.running_groups.add(test_group)
del self.tasks[task_id]
break
else:
# Either there is no tasks left or number or remaining test
# cases (groups) is less than number or running threads.
return
task_manager.run_task(task, self.timeout_per_test)
if self.running_groups is not None:
with self.task_lock:
self.running_groups.remove(test_group)
def start_daemon(func):
t = threading.Thread(target=func)
t.daemon = True
t.start()
return t
timeout = None
try:
if timeout_seconds:
timeout = threading.Timer(timeout_seconds, sigint_handler.interrupt)
timeout.start()
running_groups = set() if serialize_test_cases else None
worker_fn = WorkerFn(tasks, running_groups, timeout_per_test)
workers = [start_daemon(worker_fn) for _ in range(pool_size)]
for worker in workers:
worker.join()
finally:
if timeout:
timeout.cancel()
for task in list(task_manager.started.values()):
task.runtime_ms = timeout_seconds * 1000
task_manager.register_exit(task)
def default_options_parser():
parser = optparse.OptionParser(
usage='usage: %prog [options] binary [binary ...] -- [additional args]')
parser.add_option('-d',
'--output_dir',
type='string',
default=None,
help='Output directory for test logs. Logs will be '
'available under gtest-parallel-logs/, so '
'--output_dir=/tmp will results in all logs being '
'available under /tmp/gtest-parallel-logs/.')
parser.add_option('-r',
'--repeat',
type='int',
default=1,
help='Number of times to execute all the tests.')
parser.add_option('--retry_failed',
type='int',
default=0,
help='Number of times to repeat failed tests.')
parser.add_option('--failed',
action='store_true',
default=False,
help='run only failed and new tests')
parser.add_option('-w',
'--workers',
type='int',
default=multiprocessing.cpu_count(),
help='number of workers to spawn')
parser.add_option('--gtest_color',
type='string',
default='yes',
help='color output')
parser.add_option('--gtest_filter',
type='string',
default='',
help='test filter')
parser.add_option('--gtest_also_run_disabled_tests',
action='store_true',
default=False,
help='run disabled tests too')
parser.add_option(
'--print_test_times',
action='store_true',
default=False,
help='list the run time of each test at the end of execution')
parser.add_option(
'--print_test_command',
action='store_true',
default=False,
help='Print full test command instead of name')
parser.add_option('--shard_count',
type='int',
default=1,
help='total number of shards (for sharding test execution '
'between multiple machines)')
parser.add_option('--shard_index',
type='int',
default=0,
help='zero-indexed number identifying this shard (for '
'sharding test execution between multiple machines)')
parser.add_option(
'--dump_json_test_results',
type='string',
default=None,
help='Saves the results of the tests as a JSON machine-'
'readable file. The format of the file is specified at '
'https://www.chromium.org/developers/the-json-test-results-format')
parser.add_option('--timeout',
type='int',
default=None,
help='Interrupt all remaining processes after the given '
'time (in seconds).')
parser.add_option('--timeout_per_test',
type='int',
default=None,
help='Interrupt single processes after the given '
'time (in seconds).')
parser.add_option('--serialize_test_cases',
action='store_true',
default=False,
help='Do not run tests from the same test '
'case in parallel.')
return parser
def main():
# Remove additional arguments (anything after --).
additional_args = []
for i in range(len(sys.argv)):
if sys.argv[i] == '--':
additional_args = sys.argv[i + 1:]
sys.argv = sys.argv[:i]
break
parser = default_options_parser()
(options, binaries) = parser.parse_args()
if (options.output_dir is not None and not os.path.isdir(options.output_dir)):
parser.error('--output_dir value must be an existing directory, '
'current value is "%s"' % options.output_dir)
# Append gtest-parallel-logs to log output, this is to avoid deleting user
# data if an user passes a directory where files are already present. If a
# user specifies --output_dir=Docs/, we'll create Docs/gtest-parallel-logs
# and clean that directory out on startup, instead of nuking Docs/.
if options.output_dir:
options.output_dir = os.path.join(options.output_dir, 'gtest-parallel-logs')
if binaries == []:
parser.print_usage()
sys.exit(1)
if options.shard_count < 1:
parser.error("Invalid number of shards: %d. Must be at least 1." %
options.shard_count)
if not (0 <= options.shard_index < options.shard_count):
parser.error("Invalid shard index: %d. Must be between 0 and %d "
"(less than the number of shards)." %
(options.shard_index, options.shard_count - 1))
# Check that all test binaries have an unique basename. That way we can ensure
# the logs are saved to unique files even when two different binaries have
# common tests.
unique_binaries = set(os.path.basename(binary) for binary in binaries)
assert len(unique_binaries) == len(binaries), (
"All test binaries must have an unique basename.")
if options.output_dir:
# Remove files from old test runs.
if os.path.isdir(options.output_dir):
shutil.rmtree(options.output_dir)
# Create directory for test log output.
try:
os.makedirs(options.output_dir)
except OSError as e:
# Ignore errors if this directory already exists.
if e.errno != errno.EEXIST or not os.path.isdir(options.output_dir):
raise e
test_results = None
if options.dump_json_test_results is not None:
test_results = CollectTestResults(options.dump_json_test_results)
save_file = get_save_file_path()
times = TestTimes(save_file)
logger = FilterFormat(options.output_dir)
task_manager = TaskManager(times, logger, test_results, Task,
options.retry_failed, options.repeat + 1)
tasks = find_tests(binaries, additional_args, options, times)
logger.log_tasks(len(tasks))
execute_tasks(tasks, options.workers, task_manager, options.timeout,
options.timeout_per_test, options.serialize_test_cases)
print_try_number = options.retry_failed > 0 or options.repeat > 1
if task_manager.passed:
logger.move_to('passed', task_manager.passed)
if options.print_test_times:
logger.print_tests('PASSED TESTS', task_manager.passed, print_try_number, options.print_test_command)
if task_manager.failed:
logger.print_tests('FAILED TESTS', task_manager.failed, print_try_number, options.print_test_command)
logger.move_to('failed', task_manager.failed)
if task_manager.timed_out:
logger.print_tests('TIMED OUT TESTS', task_manager.timed_out, print_try_number, options.print_test_command)
logger.move_to('timed_out', task_manager.timed_out)
if task_manager.started:
logger.print_tests('INTERRUPTED TESTS', task_manager.started.values(),
print_try_number, options.print_test_command)
logger.move_to('interrupted', task_manager.started.values())
if options.repeat > 1 and (task_manager.failed or task_manager.started):
logger.summarize(task_manager.passed, task_manager.failed,
task_manager.started.values())
logger.flush()
times.write_to_file(save_file)
if test_results:
test_results.dump_to_file_and_close()
if sigint_handler.got_sigint():
return -signal.SIGINT
return task_manager.global_exit_code
if __name__ == "__main__":
sys.exit(main())
deps/gtest-parallel/gtest_parallel_mocks.py
Vendored
+173
View File
@@ -0,0 +1,173 @@
# Copyright 2017 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import collections
import threading
import time
class LoggerMock(object):
def __init__(self, test_lib):
self.test_lib = test_lib
self.runtimes = collections.defaultdict(list)
self.exit_codes = collections.defaultdict(list)
self.last_execution_times = collections.defaultdict(list)
self.execution_numbers = collections.defaultdict(list)
def log_exit(self, task):
self.runtimes[task.test_id].append(task.runtime_ms)
self.exit_codes[task.test_id].append(task.exit_code)
self.last_execution_times[task.test_id].append(task.last_execution_time)
self.execution_numbers[task.test_id].append(task.execution_number)
def assertRecorded(self, test_id, expected, retries):
self.test_lib.assertIn(test_id, self.runtimes)
self.test_lib.assertListEqual(expected['runtime_ms'][:retries],
self.runtimes[test_id])
self.test_lib.assertListEqual(expected['exit_code'][:retries],
self.exit_codes[test_id])
self.test_lib.assertListEqual(expected['last_execution_time'][:retries],
self.last_execution_times[test_id])
self.test_lib.assertListEqual(expected['execution_number'][:retries],
self.execution_numbers[test_id])
class TestTimesMock(object):
def __init__(self, test_lib, test_data=None):
self.test_lib = test_lib
self.test_data = test_data or {}
self.last_execution_times = collections.defaultdict(list)
def record_test_time(self, test_binary, test_name, last_execution_time):
test_id = (test_binary, test_name)
self.last_execution_times[test_id].append(last_execution_time)
def get_test_time(self, test_binary, test_name):
test_group, test = test_name.split('.')
return self.test_data.get(test_binary, {}).get(test_group,
{}).get(test, None)
def assertRecorded(self, test_id, expected, retries):
self.test_lib.assertIn(test_id, self.last_execution_times)
self.test_lib.assertListEqual(expected['last_execution_time'][:retries],
self.last_execution_times[test_id])
class TestResultsMock(object):
def __init__(self, test_lib):
self.results = []
self.test_lib = test_lib
def log(self, test_name, runtime_ms, actual_result):
self.results.append((test_name, runtime_ms, actual_result))
def assertRecorded(self, test_id, expected, retries):
test_results = [
(test_id[1], runtime_ms / 1000.0, exit_code)
for runtime_ms, exit_code in zip(expected['runtime_ms'][:retries],
expected['exit_code'][:retries])
]
for test_result in test_results:
self.test_lib.assertIn(test_result, self.results)
class TaskManagerMock(object):
def __init__(self):
self.running_groups = []
self.check_lock = threading.Lock()
self.had_running_parallel_groups = False
self.total_tasks_run = 0
self.started = {}
def __register_start(self, task):
self.started[task.task_id] = task
def register_exit(self, task):
self.started.pop(task.task_id)
def run_task(self, task, timeout_per_test):
self.__register_start(task)
test_group = task.test_name.split('.')[0]
with self.check_lock:
self.total_tasks_run += 1
if test_group in self.running_groups:
self.had_running_parallel_groups = True
self.running_groups.append(test_group)
# Delay as if real test were run.
time.sleep(0.001)
with self.check_lock:
self.running_groups.remove(test_group)
class TaskMockFactory(object):
def __init__(self, test_data):
self.data = test_data
self.passed = []
self.failed = []
def get_task(self, test_id, execution_number=0):
task = TaskMock(test_id, execution_number, self.data[test_id])
if task.exit_code == 0:
self.passed.append(task)
else:
self.failed.append(task)
return task
def __call__(self, test_binary, test_name, test_command, execution_number,
last_execution_time, output_dir):
return self.get_task((test_binary, test_name), execution_number)
class TaskMock(object):
def __init__(self, test_id, execution_number, test_data):
self.test_id = test_id
self.execution_number = execution_number
self.runtime_ms = test_data['runtime_ms'][execution_number]
self.exit_code = test_data['exit_code'][execution_number]
self.last_execution_time = (
test_data['last_execution_time'][execution_number])
if 'log_file' in test_data:
self.log_file = test_data['log_file'][execution_number]
else:
self.log_file = None
self.test_command = None
self.output_dir = None
self.test_binary = test_id[0]
self.test_name = test_id[1]
self.task_id = (test_id[0], test_id[1], execution_number)
def run(self, timeout_per_test):
pass
class SubprocessMock(object):
def __init__(self, test_data=None):
self._test_data = test_data
self.last_invocation = None
def __call__(self, command, **kwargs):
self.last_invocation = command
binary = command[0]
test_list = []
tests_for_binary = sorted(self._test_data.get(binary, {}).items())
for test_group, tests in tests_for_binary:
test_list.append(test_group + ".")
for test in sorted(tests):
test_list.append(" " + test)
return '\n'.join(test_list)
deps/gtest-parallel/gtest_parallel_tests.py
Vendored Executable
+656
View File
@@ -0,0 +1,656 @@
#!/usr/bin/env python
# Copyright 2017 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import contextlib
import os.path
import random
import shutil
import sys
import tempfile
import threading
import unittest
import gtest_parallel
from gtest_parallel_mocks import LoggerMock
from gtest_parallel_mocks import SubprocessMock
from gtest_parallel_mocks import TestTimesMock
from gtest_parallel_mocks import TestResultsMock
from gtest_parallel_mocks import TaskManagerMock
from gtest_parallel_mocks import TaskMockFactory
from gtest_parallel_mocks import TaskMock
@contextlib.contextmanager
def guard_temp_dir():
try:
temp_dir = tempfile.mkdtemp()
yield temp_dir
finally:
shutil.rmtree(temp_dir)
@contextlib.contextmanager
def guard_temp_subdir(temp_dir, *path):
assert path, 'Path should not be empty'
try:
temp_subdir = os.path.join(temp_dir, *path)
os.makedirs(temp_subdir)
yield temp_subdir
finally:
shutil.rmtree(os.path.join(temp_dir, path[0]))
@contextlib.contextmanager
def guard_patch_module(import_name, new_val):
def patch(module, names, val):
if len(names) == 1:
old = getattr(module, names[0])
setattr(module, names[0], val)
return old
else:
return patch(getattr(module, names[0]), names[1:], val)
try:
old_val = patch(gtest_parallel, import_name.split('.'), new_val)
yield old_val
finally:
patch(gtest_parallel, import_name.split('.'), old_val)
class TestTaskManager(unittest.TestCase):
def setUp(self):
self.passing_task = (('fake_binary', 'Fake.PassingTest'), {
'runtime_ms': [10],
'exit_code': [0],
'last_execution_time': [10],
})
self.failing_task = (('fake_binary', 'Fake.FailingTest'), {
'runtime_ms': [20, 30, 40],
'exit_code': [1, 1, 1],
'last_execution_time': [None, None, None],
})
self.fails_once_then_succeeds = (('another_binary', 'Fake.Test.FailOnce'), {
'runtime_ms': [21, 22],
'exit_code': [1, 0],
'last_execution_time': [None, 22],
})
self.fails_twice_then_succeeds = (('yet_another_binary',
'Fake.Test.FailTwice'), {
'runtime_ms': [23, 25, 24],
'exit_code': [1, 1, 0],
'last_execution_time':
[None, None, 24],
})
def execute_tasks(self, tasks, retries, expected_exit_code):
repeat = 1
times = TestTimesMock(self)
logger = LoggerMock(self)
test_results = TestResultsMock(self)
task_mock_factory = TaskMockFactory(dict(tasks))
task_manager = gtest_parallel.TaskManager(times, logger, test_results,
task_mock_factory, retries,
repeat)
for test_id, expected in tasks:
task = task_mock_factory.get_task(test_id)
task_manager.run_task(task, timeout_per_test=30)
expected['execution_number'] = list(range(len(expected['exit_code'])))
logger.assertRecorded(test_id, expected, retries + 1)
times.assertRecorded(test_id, expected, retries + 1)
test_results.assertRecorded(test_id, expected, retries + 1)
self.assertEqual(len(task_manager.started), 0)
self.assertListEqual(
sorted(task.task_id for task in task_manager.passed),
sorted(task.task_id for task in task_mock_factory.passed))
self.assertListEqual(
sorted(task.task_id for task in task_manager.failed),
sorted(task.task_id for task in task_mock_factory.failed))
self.assertEqual(task_manager.global_exit_code, expected_exit_code)
def test_passing_task_succeeds(self):
self.execute_tasks(tasks=[self.passing_task],
retries=0,
expected_exit_code=0)
def test_failing_task_fails(self):
self.execute_tasks(tasks=[self.failing_task],
retries=0,
expected_exit_code=1)
def test_failing_task_fails_even_with_retries(self):
self.execute_tasks(tasks=[self.failing_task],
retries=2,
expected_exit_code=1)
def test_executing_passing_and_failing_fails(self):
# Executing both a faling test and a passing one should make gtest-parallel
# fail, no matter if the failing task is run first or last.
self.execute_tasks(tasks=[self.failing_task, self.passing_task],
retries=2,
expected_exit_code=1)
self.execute_tasks(tasks=[self.passing_task, self.failing_task],
retries=2,
expected_exit_code=1)
def test_task_succeeds_with_one_retry(self):
# Executes test and retries once. The first run should fail and the second
# succeed, so gtest-parallel should succeed.
self.execute_tasks(tasks=[self.fails_once_then_succeeds],
retries=1,
expected_exit_code=0)
def test_task_fails_with_one_retry(self):
# Executes test and retries once, not enough for the test to start passing,
# so gtest-parallel should return an error.
self.execute_tasks(tasks=[self.fails_twice_then_succeeds],
retries=1,
expected_exit_code=1)
def test_runner_succeeds_when_all_tasks_eventually_succeeds(self):
# Executes the test and retries twice. One test should pass in the first
# attempt, another should take two runs, and the last one should take three
# runs. All tests should succeed, so gtest-parallel should succeed too.
self.execute_tasks(tasks=[
self.passing_task, self.fails_once_then_succeeds,
self.fails_twice_then_succeeds
],
retries=2,
expected_exit_code=0)
class TestSaveFilePath(unittest.TestCase):
class StreamMock(object):
def write(*args):
# Suppress any output.
pass
def test_get_save_file_path_unix(self):
with guard_temp_dir() as temp_dir, \
guard_patch_module('os.path.expanduser', lambda p: temp_dir), \
guard_patch_module('sys.stderr', TestSaveFilePath.StreamMock()), \
guard_patch_module('sys.platform', 'darwin'):
with guard_patch_module('os.environ', {}), \
guard_temp_subdir(temp_dir, '.cache'):
self.assertEqual(os.path.join(temp_dir, '.cache', 'gtest-parallel'),
gtest_parallel.get_save_file_path())
with guard_patch_module('os.environ', {'XDG_CACHE_HOME': temp_dir}):
self.assertEqual(os.path.join(temp_dir, 'gtest-parallel'),
gtest_parallel.get_save_file_path())
with guard_patch_module('os.environ',
{'XDG_CACHE_HOME': os.path.realpath(__file__)}):
self.assertEqual(os.path.join(temp_dir, '.gtest-parallel-times'),
gtest_parallel.get_save_file_path())
def test_get_save_file_path_win32(self):
with guard_temp_dir() as temp_dir, \
guard_patch_module('os.path.expanduser', lambda p: temp_dir), \
guard_patch_module('sys.stderr', TestSaveFilePath.StreamMock()), \
guard_patch_module('sys.platform', 'win32'):
with guard_patch_module('os.environ', {}), \
guard_temp_subdir(temp_dir, 'AppData', 'Local'):
self.assertEqual(
os.path.join(temp_dir, 'AppData', 'Local', 'gtest-parallel'),
gtest_parallel.get_save_file_path())
with guard_patch_module('os.environ', {'LOCALAPPDATA': temp_dir}):
self.assertEqual(os.path.join(temp_dir, 'gtest-parallel'),
gtest_parallel.get_save_file_path())
with guard_patch_module('os.environ',
{'LOCALAPPDATA': os.path.realpath(__file__)}):
self.assertEqual(os.path.join(temp_dir, '.gtest-parallel-times'),
gtest_parallel.get_save_file_path())
class TestSerializeTestCases(unittest.TestCase):
def _execute_tasks(self, max_number_of_test_cases,
max_number_of_tests_per_test_case, max_number_of_repeats,
max_number_of_workers, timeout_per_test, serialize_test_cases):
tasks = []
for test_case in range(max_number_of_test_cases):
for test_name in range(max_number_of_tests_per_test_case):
# All arguments for gtest_parallel.Task except for test_name are fake.
test_name = 'TestCase{}.test{}'.format(test_case, test_name)
for execution_number in range(random.randint(1, max_number_of_repeats)):
tasks.append(
gtest_parallel.Task('path/to/binary', test_name,
['path/to/binary', '--gtest_filter=*'],
execution_number + 1, None, 'path/to/output'))
expected_tasks_number = len(tasks)
task_manager = TaskManagerMock()
gtest_parallel.execute_tasks(tasks, max_number_of_workers, task_manager,
None, timeout_per_test, serialize_test_cases)
self.assertEqual(serialize_test_cases,
not task_manager.had_running_parallel_groups)
self.assertEqual(expected_tasks_number, task_manager.total_tasks_run)
def test_running_parallel_test_cases_without_repeats(self):
self._execute_tasks(max_number_of_test_cases=4,
max_number_of_tests_per_test_case=32,
max_number_of_repeats=1,
max_number_of_workers=16,
timeout_per_test=30,
serialize_test_cases=True)
def test_running_parallel_test_cases_with_repeats(self):
self._execute_tasks(max_number_of_test_cases=4,
max_number_of_tests_per_test_case=32,
max_number_of_repeats=4,
max_number_of_workers=16,
timeout_per_test=30,
serialize_test_cases=True)
def test_running_parallel_tests(self):
self._execute_tasks(max_number_of_test_cases=4,
max_number_of_tests_per_test_case=128,
max_number_of_repeats=1,
max_number_of_workers=16,
timeout_per_test=30,
serialize_test_cases=False)
class TestTestTimes(unittest.TestCase):
def test_race_in_test_times_load_save(self):
max_number_of_workers = 8
max_number_of_read_write_cycles = 64
test_times_file_name = 'test_times.pickle'
def start_worker(save_file):
def test_times_worker():
thread_id = threading.current_thread().ident
path_to_binary = 'path/to/binary' + hex(thread_id)
for cnt in range(max_number_of_read_write_cycles):
times = gtest_parallel.TestTimes(save_file)
threads_test_times = [
binary for (binary, _) in times._TestTimes__times.keys()
if binary.startswith(path_to_binary)
]
self.assertEqual(cnt, len(threads_test_times))
times.record_test_time('{}-{}'.format(path_to_binary, cnt),
'TestFoo.testBar', 1000)
times.write_to_file(save_file)
self.assertEqual(
1000,
times.get_test_time('{}-{}'.format(path_to_binary, cnt),
'TestFoo.testBar'))
self.assertIsNone(
times.get_test_time('{}-{}'.format(path_to_binary, cnt), 'baz'))
t = threading.Thread(target=test_times_worker)
t.start()
return t
with guard_temp_dir() as temp_dir:
try:
workers = [
start_worker(os.path.join(temp_dir, test_times_file_name))
for _ in range(max_number_of_workers)
]
finally:
for worker in workers:
worker.join()
class TestTimeoutTestCases(unittest.TestCase):
def test_task_timeout(self):
timeout = 1
pool_size = 1
timeout_per_test = 30
task = gtest_parallel.Task('test_binary', 'test_name', ['test_command'], 1,
None, 'output_dir')
tasks = [task]
task_manager = TaskManagerMock()
gtest_parallel.execute_tasks(tasks, pool_size, task_manager, timeout, timeout_per_test, True)
self.assertEqual(1, task_manager.total_tasks_run)
self.assertEqual(None, task.exit_code)
self.assertEqual(1000, task.runtime_ms)
class TestTask(unittest.TestCase):
def test_log_file_names(self):
def root():
return 'C:\\' if sys.platform == 'win32' else '/'
self.assertEqual(os.path.join('.', 'bin-Test_case-100.log'),
gtest_parallel.Task._logname('.', 'bin', 'Test.case', 100))
self.assertEqual(
os.path.join('..', 'a', 'b', 'bin-Test_case_2-1.log'),
gtest_parallel.Task._logname(os.path.join('..', 'a', 'b'),
os.path.join('..', 'bin'), 'Test.case/2',
1))
self.assertEqual(
os.path.join('..', 'a', 'b', 'bin-Test_case_2-5.log'),
gtest_parallel.Task._logname(os.path.join('..', 'a', 'b'),
os.path.join(root(), 'c', 'd', 'bin'),
'Test.case/2', 5))
self.assertEqual(
os.path.join(root(), 'a', 'b', 'bin-Instantiation_Test_case_2-3.log'),
gtest_parallel.Task._logname(os.path.join(root(), 'a', 'b'),
os.path.join('..', 'c', 'bin'),
'Instantiation/Test.case/2', 3))
self.assertEqual(
os.path.join(root(), 'a', 'b', 'bin-Test_case-1.log'),
gtest_parallel.Task._logname(os.path.join(root(), 'a', 'b'),
os.path.join(root(), 'c', 'd', 'bin'),
'Test.case', 1))
def test_logs_to_temporary_files_without_output_dir(self):
log_file = gtest_parallel.Task._logname(None, None, None, None)
self.assertEqual(tempfile.gettempdir(), os.path.dirname(log_file))
os.remove(log_file)
def _execute_run_test(self, run_test_body, interrupt_test):
def popen_mock(*_args, **_kwargs):
return None
class SigHandlerMock(object):
class ProcessWasInterrupted(Exception):
pass
def wait(*_args):
if interrupt_test:
raise SigHandlerMock.ProcessWasInterrupted()
return 42
with guard_temp_dir() as temp_dir, \
guard_patch_module('subprocess.Popen', popen_mock), \
guard_patch_module('sigint_handler', SigHandlerMock()), \
guard_patch_module('thread.exit', lambda: None):
run_test_body(temp_dir)
def test_run_normal_task(self):
def run_test(temp_dir):
task = gtest_parallel.Task('fake/binary', 'test', ['fake/binary'], 1,
None, temp_dir)
self.assertFalse(os.path.isfile(task.log_file))
task.run(timeout_per_test=30)
self.assertTrue(os.path.isfile(task.log_file))
self.assertEqual(42, task.exit_code)
self._execute_run_test(run_test, False)
def test_run_interrupted_task_with_transient_log(self):
def run_test(_):
task = gtest_parallel.Task('fake/binary', 'test', ['fake/binary'], 1,
None, None)
self.assertTrue(os.path.isfile(task.log_file))
task.run(timeout_per_test=30)
self.assertTrue(os.path.isfile(task.log_file))
self.assertIsNone(task.exit_code)
self._execute_run_test(run_test, True)
class TestFilterFormat(unittest.TestCase):
def _execute_test(self, test_body, drop_output):
class StdoutMock(object):
def isatty(*_args):
return False
def write(*args):
pass
def flush(*args):
pass
with guard_temp_dir() as temp_dir, \
guard_patch_module('sys.stdout', StdoutMock()):
logger = gtest_parallel.FilterFormat(None if drop_output else temp_dir)
logger.log_tasks(42)
test_body(logger)
logger.flush()
def test_no_output_dir(self):
def run_test(logger):
passed = [
TaskMock(
('fake/binary', 'FakeTest'), 0, {
'runtime_ms': [10],
'exit_code': [0],
'last_execution_time': [10],
'log_file': [os.path.join(tempfile.gettempdir(), 'fake.log')]
})
]
open(passed[0].log_file, 'w').close()
self.assertTrue(os.path.isfile(passed[0].log_file))
logger.log_exit(passed[0])
self.assertFalse(os.path.isfile(passed[0].log_file))
logger.print_tests('', passed, print_try_number=True, print_test_command=True)
logger.move_to(None, passed)
logger.summarize(passed, [], [])
self._execute_test(run_test, True)
def test_with_output_dir(self):
def run_test(logger):
failed = [
TaskMock(
('fake/binary', 'FakeTest'), 0, {
'runtime_ms': [10],
'exit_code': [1],
'last_execution_time': [10],
'log_file': [os.path.join(logger.output_dir, 'fake.log')]
})
]
open(failed[0].log_file, 'w').close()
self.assertTrue(os.path.isfile(failed[0].log_file))
logger.log_exit(failed[0])
self.assertTrue(os.path.isfile(failed[0].log_file))
logger.print_tests('', failed, print_try_number=True, print_test_command=True)
logger.move_to('failed', failed)
self.assertFalse(os.path.isfile(failed[0].log_file))
self.assertTrue(
os.path.isfile(os.path.join(logger.output_dir, 'failed', 'fake.log')))
logger.summarize([], failed, [])
self._execute_test(run_test, False)
class TestFindTests(unittest.TestCase):
ONE_DISABLED_ONE_ENABLED_TEST = {
"fake_unittests": {
"FakeTest": {
"Test1": None,
"DISABLED_Test2": None,
}
}
}
ONE_FAILED_ONE_PASSED_TEST = {
"fake_unittests": {
"FakeTest": {
# Failed (and new) tests have no recorded runtime.
"FailedTest": None,
"Test": 1,
}
}
}
ONE_TEST = {
"fake_unittests": {
"FakeTest": {
"TestSomething": None,
}
}
}
MULTIPLE_BINARIES_MULTIPLE_TESTS_ONE_FAILURE = {
"fake_unittests": {
"FakeTest": {
"TestSomething": None,
"TestSomethingElse": 2,
},
"SomeOtherTest": {
"YetAnotherTest": 3,
},
},
"fake_tests": {
"Foo": {
"Bar": 4,
"Baz": 4,
}
}
}
def _process_options(self, options):
parser = gtest_parallel.default_options_parser()
options, binaries = parser.parse_args(options)
self.assertEqual(len(binaries), 0)
return options
def _call_find_tests(self, test_data, options=None):
subprocess_mock = SubprocessMock(test_data)
options = self._process_options(options or [])
with guard_patch_module('subprocess.check_output', subprocess_mock):
tasks = gtest_parallel.find_tests(test_data.keys(), [], options,
TestTimesMock(self, test_data))
# Clean transient tasks' log files created because
# by default now output_dir is None.
for task in tasks:
if os.path.isfile(task.log_file):
os.remove(task.log_file)
return tasks, subprocess_mock
def test_tasks_are_sorted(self):
tasks, _ = self._call_find_tests(
self.MULTIPLE_BINARIES_MULTIPLE_TESTS_ONE_FAILURE)
self.assertEqual([task.last_execution_time for task in tasks],
[None, 4, 4, 3, 2])
def test_does_not_run_disabled_tests_by_default(self):
tasks, subprocess_mock = self._call_find_tests(
self.ONE_DISABLED_ONE_ENABLED_TEST)
self.assertEqual(len(tasks), 1)
self.assertFalse("DISABLED_" in tasks[0].test_name)
self.assertNotIn("--gtest_also_run_disabled_tests",
subprocess_mock.last_invocation)
def test_runs_disabled_tests_when_asked(self):
tasks, subprocess_mock = self._call_find_tests(
self.ONE_DISABLED_ONE_ENABLED_TEST, ['--gtest_also_run_disabled_tests'])
self.assertEqual(len(tasks), 2)
self.assertEqual(sorted([task.test_name for task in tasks]),
["FakeTest.DISABLED_Test2", "FakeTest.Test1"])
self.assertIn("--gtest_also_run_disabled_tests",
subprocess_mock.last_invocation)
def test_runs_failed_tests_by_default(self):
tasks, _ = self._call_find_tests(self.ONE_FAILED_ONE_PASSED_TEST)
self.assertEqual(len(tasks), 2)
self.assertEqual(sorted([task.test_name for task in tasks]),
["FakeTest.FailedTest", "FakeTest.Test"])
self.assertEqual({task.last_execution_time for task in tasks}, {None, 1})
def test_runs_only_failed_tests_when_asked(self):
tasks, _ = self._call_find_tests(self.ONE_FAILED_ONE_PASSED_TEST,
['--failed'])
self.assertEqual(len(tasks), 1)
self.assertEqual(tasks[0].test_binary, "fake_unittests")
self.assertEqual(tasks[0].test_name, "FakeTest.FailedTest")
self.assertIsNone(tasks[0].last_execution_time)
def test_does_not_apply_gtest_filter_by_default(self):
_, subprocess_mock = self._call_find_tests(self.ONE_TEST)
self.assertFalse(
any(
arg.startswith('--gtest_filter=SomeFilter')
for arg in subprocess_mock.last_invocation))
def test_applies_gtest_filter(self):
_, subprocess_mock = self._call_find_tests(self.ONE_TEST,
['--gtest_filter=SomeFilter'])
self.assertIn('--gtest_filter=SomeFilter', subprocess_mock.last_invocation)
def test_applies_gtest_color_by_default(self):
tasks, _ = self._call_find_tests(self.ONE_TEST)
self.assertEqual(len(tasks), 1)
self.assertIn('--gtest_color=yes', tasks[0].test_command)
def test_applies_gtest_color(self):
tasks, _ = self._call_find_tests(self.ONE_TEST, ['--gtest_color=Lemur'])
self.assertEqual(len(tasks), 1)
self.assertIn('--gtest_color=Lemur', tasks[0].test_command)
def test_repeats_tasks_once_by_default(self):
tasks, _ = self._call_find_tests(self.ONE_TEST)
self.assertEqual(len(tasks), 1)
def test_repeats_tasks_multiple_times(self):
tasks, _ = self._call_find_tests(self.ONE_TEST, ['--repeat=3'])
self.assertEqual(len(tasks), 3)
# Test all tasks have the same test_name, test_binary and test_command
all_tasks_set = set(
(task.test_name, task.test_binary, tuple(task.test_command))
for task in tasks)
self.assertEqual(len(all_tasks_set), 1)
# Test tasks have consecutive execution_numbers starting from 1
self.assertEqual(sorted(task.execution_number for task in tasks), [1, 2, 3])
def test_gtest_list_tests_fails(self):
def exit_mock(*args):
raise AssertionError('Foo')
options = self._process_options([])
with guard_patch_module('sys.exit', exit_mock):
self.assertRaises(AssertionError, gtest_parallel.find_tests,
[sys.executable], [], options, None)
if __name__ == '__main__':
unittest.main()
deps/libvalkey/.github/typos.toml
Vendored
+1
View File
@@ -9,6 +9,7 @@ extend-exclude = [
[type.c.extend-identifiers]
clen = "clen"
[type.c.extend-words]
cpy = "cpy"
fo = "fo" # Used in sds.c testcase.
# Header files (sv = *.h)
deps/libvalkey/.github/workflows/build.yml
Vendored
+73 -13
View File
@@ -8,7 +8,7 @@ jobs:
runs-on: ubuntu-latest
container: almalinux:8
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install dependencies
run: |
@@ -35,24 +35,21 @@ jobs:
runs-on: ubuntu-latest
container: rockylinux:8
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install dependencies
run: |
dnf -y upgrade --refresh
dnf -y install https://rpms.remirepo.net/enterprise/remi-release-8.rpm
dnf -y module install redis:remi-6.0
dnf -y group install "Development Tools"
dnf -y install openssl-devel cmake libevent-devel valkey
- name: Build using cmake
- name: Build using CMake
env:
EXTRA_CMAKE_OPTS: -DENABLE_EXAMPLES:BOOL=ON -DENABLE_TLS:BOOL=ON
run: mkdir build && cd build && cmake .. && make
run: |
mkdir build && cd build && cmake .. && make
cpack -G RPM
- name: Build using Makefile
run: USE_TLS=1 TEST_ASYNC=1 make
- name: Run tests
working-directory: tests
env:
@@ -64,16 +61,79 @@ jobs:
runs-on: ubuntu-latest
name: FreeBSD
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Build in FreeBSD
uses: vmactions/freebsd-vm@05856381fab64eeee9b038a0818f6cec649ca17a # v1.2.3
uses: vmactions/freebsd-vm@c9f815bc7aa0d34c9fdd0619b034a32d6ca7b57e # v1.4.2
with:
prepare: pkg install -y gmake cmake
run: |
gmake
mkdir build && cd build && cmake .. && gmake
solaris:
runs-on: ubuntu-latest
name: Solaris
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Build on Solaris
uses: vmactions/solaris-vm@69d382b4a775b25ea5955e6c1730e9d05047ca0d # v1.3.1
with:
prepare: pkgutil -y -i gmake gcc5core openssl_utils
run: USE_TLS=1 gmake
solaris-developer-studio:
name: Oracle DeveloperStudio
runs-on: ubuntu-24.04
# Secrets containing certificates are not available in forks,
# or for PRs created by dependabot or externally.
if: |
github.repository == 'valkey-io/libvalkey' &&
github.actor != 'dependabot[bot]' &&
(github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository)
env:
PKG_ORACLE_CERT: ${{ secrets.PKG_ORACLE_CERT }}
PKG_ORACLE_KEY: ${{ secrets.PKG_ORACLE_KEY }}
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Transfer Oracle Studio certificates
run: |
set -e
printf '%s\n' "$PKG_ORACLE_CERT" \
> pkg.oracle.com.certificate.pem
printf '%s\n' "$PKG_ORACLE_KEY" \
> pkg.oracle.com.key.pem
- name: Build on Solaris
uses: vmactions/solaris-vm@69d382b4a775b25ea5955e6c1730e9d05047ca0d # v1.3.1
with:
usesh: true
prepare: |
set -e
cp "$GITHUB_WORKSPACE/pkg.oracle.com.key.pem" \
/root/pkg.oracle.com.key.pem
cp "$GITHUB_WORKSPACE/pkg.oracle.com.certificate.pem" \
/root/pkg.oracle.com.certificate.pem
sudo pkg set-publisher \
-k /root/pkg.oracle.com.key.pem \
-c /root/pkg.oracle.com.certificate.pem \
-G "*" \
-g https://pkg.oracle.com/solarisstudio/release \
solarisstudio
sudo pkg install --accept developerstudio-126/cc
run: |
set -e
PATH=/opt/developerstudio12.6/bin:"$PATH"
export PATH
gmake USE_THREADS=1 USE_TLS=1 -j"$(psrinfo -p)"
build-cross:
name: Cross-compile ${{ matrix.config.target }}
runs-on: ubuntu-22.04
@@ -94,9 +154,9 @@ jobs:
- {target: mipsel, host: mipsel-linux-gnu, qemu: mipsel, gccver: 10 }
- {target: mips64el, host: mips64el-linux-gnuabi64, qemu: mips64el, gccver: 10 }
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: gcc-${{ matrix.config.gccver }}-${{ matrix.config.host }}
version: ${{ matrix.config.target }}-1.0
deps/libvalkey/.github/workflows/ci.yml
Vendored
+33 -26
View File
@@ -9,12 +9,19 @@ jobs:
checkers:
name: Run static checkers
runs-on: ubuntu-latest
env:
LLVM_VERSION: '18'
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install clang-format
run: |
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh ${{ env.LLVM_VERSION }}
sudo apt install -y clang-format-${{ env.LLVM_VERSION }}
- name: Run clang-format style check (.c and .h)
uses: jidicula/clang-format-action@4726374d1aa3c6aecf132e5197e498979588ebc8 # v4.15.0
with:
clang-format-version: '18'
run: |
find . -regex '.*\.\(c\|h\)' -exec clang-format-${{ env.LLVM_VERSION }} -style=file --dry-run --Werror {} +
ubuntu-cmake:
name: Build with CMake ${{ matrix.sanitizer && format('and {0}-sanitizer', matrix.sanitizer ) }} [${{ matrix.compiler }}, cmake-${{ matrix.cmake-version }}, ${{ matrix.cmake-build-type }}]
@@ -43,18 +50,18 @@ jobs:
cmake-build-type: [RelWithDebInfo]
sanitizer: [thread, undefined, leak, address]
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: libevent-dev libuv1-dev libev-dev libglib2.0-dev valkey-server
version: 1.0
- name: Setup compiler
uses: aminya/setup-cpp@a276e6e3d1db9160db5edc458e99a30d3b109949 # v1.7.1
uses: aminya/setup-cpp@1f17f92d6a52bfcb1a25348e2c526c2e5cbb1134 # v1.8.0
with:
compiler: ${{ matrix.compiler }}
- name: Setup CMake
uses: jwlawson/actions-setup-cmake@802fa1a2c4e212495c05bf94dba2704a92a472be # v2.0.2
uses: jwlawson/actions-setup-cmake@3a6cbe35ba64df7ca70c51365c4aff65db9a9037 # v2.1.1
with:
cmake-version: ${{ matrix.cmake-version }}
- name: Generate makefiles
@@ -84,14 +91,14 @@ jobs:
name: Build with make
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: libevent-dev valgrind valkey-server
version: 1.0
- name: Build
run: USE_TLS=1 TEST_ASYNC=1 make
run: USE_TLS=1 TEST_ASYNC=1 make -j$(nproc)
- name: Run tests
working-directory: tests
env:
@@ -110,9 +117,9 @@ jobs:
name: Build for 32-bit
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: gcc-multilib valkey-server
version: 1.0
@@ -128,9 +135,9 @@ jobs:
name: Installation tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: libevent-dev libuv1-dev libev-dev libglib2.0-dev
version: 1.0
@@ -174,9 +181,9 @@ jobs:
name: RDMA support enabled
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: librdmacm-dev libibverbs-dev
version: 1.0
@@ -198,9 +205,9 @@ jobs:
name: CMake 3.7.0 (min. required)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Setup CMake
uses: jwlawson/actions-setup-cmake@802fa1a2c4e212495c05bf94dba2704a92a472be # v2.0.2
uses: jwlawson/actions-setup-cmake@3a6cbe35ba64df7ca70c51365c4aff65db9a9037 # v2.1.1
with:
cmake-version: '3.7.0'
- name: Generate makefiles
@@ -213,7 +220,7 @@ jobs:
name: macOS
runs-on: macos-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install dependencies
run: |
brew update
@@ -221,7 +228,7 @@ jobs:
- name: Build and install using CMake
run: |
mkdir build && cd build
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_TLS=ON
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release -DENABLE_TLS=ON -DENABLE_EXAMPLES=ON
sudo ninja -v install
- name: Build using Makefile
run: USE_TLS=1 make
@@ -235,7 +242,7 @@ jobs:
name: Windows
runs-on: windows-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: ilammy/msvc-dev-cmd@0b201ec74fa43914dc39ae48a89fd1d8cb592756 # v1.13.0
- name: Remove installed OpenSSL 1.1.1 (has reached End of Life)
shell: bash
@@ -266,9 +273,9 @@ jobs:
run: |
git config --global core.autocrlf input
choco install -y memurai-developer
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install Cygwin
uses: cygwin/cygwin-install-action@f2009323764960f80959895c7bc3bb30210afe4d # v6
uses: cygwin/cygwin-install-action@711d29f3da23c9f4a1798e369a6f01198c13b11a # v6
with:
packages: make gcc-core cmake libssl-devel
- name: Build with CMake using Cygwin
@@ -284,9 +291,9 @@ jobs:
name: Windows (MinGW64)
runs-on: windows-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up MinGW
uses: msys2/setup-msys2@40677d36a502eb2cf0fb808cc9dec31bf6152638 # v2.28.0
uses: msys2/setup-msys2@4f806de0a5a7294ffabaff804b38a9b435a73bda # v2.30.0
with:
msystem: mingw64
install: |
deps/libvalkey/.github/workflows/coverity.yml
Vendored
+2 -2
View File
@@ -12,9 +12,9 @@ jobs:
if: github.repository == 'valkey-io/libvalkey'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install dependencies
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: libevent-dev
version: 1.0
deps/libvalkey/.github/workflows/db-compatibility.yml
Vendored
+4 -4
View File
@@ -18,7 +18,7 @@ jobs:
- valkey-version: '7.2.8'
steps:
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: libevent-dev
version: 1.0
@@ -26,7 +26,7 @@ jobs:
run: |
git clone --depth 1 --branch ${{ matrix.valkey-version }} https://github.com/valkey-io/valkey.git
cd valkey && BUILD_TLS=yes make install
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Create build folder
run: cmake -E make_directory build
- name: Generate makefiles
@@ -66,7 +66,7 @@ jobs:
- redis-version: '6.2.14'
steps:
- name: Prepare
uses: awalsh128/cache-apt-pkgs-action@2c09a5e66da6c8016428a2172bd76e5e4f14bb17 # v1.5.3
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: libevent-dev
version: 1.0
@@ -74,7 +74,7 @@ jobs:
run: |
git clone --depth 1 --branch ${{ matrix.redis-version }} https://github.com/redis/redis.git
cd redis && BUILD_TLS=yes make install
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Create build folder
run: cmake -E make_directory build
deps/libvalkey/.github/workflows/release-drafter.yml
Vendored
+1 -1
View File
@@ -11,7 +11,7 @@ jobs:
runs-on: ubuntu-latest
steps:
# Drafts your next Release notes as Pull Requests are merged into "master"
- uses: release-drafter/release-drafter@b1476f6e6eb133afa41ed8589daba6dc69b4d3f5 # v6.1.0
- uses: release-drafter/release-drafter@6db134d15f3909ccc9eefd369f02bd1e9cffdf97 # v6.2.0
with:
# (Optional) specify config name to use, relative to .github/. Default: release-drafter.yml
config-name: release-drafter-config.yml
deps/libvalkey/.github/workflows/spellcheck.yml
Vendored
+4 -4
View File
@@ -10,18 +10,18 @@ jobs:
spellcheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Run spellcheck
uses: rojopolis/spellcheck-github-actions@35a02bae020e6999c5c37fabaf447f2eb8822ca7 # 0.51.0
uses: rojopolis/spellcheck-github-actions@0bf4b2f91efa259b52c202b09b0c3845c524ff36 # 0.58.0
with:
config_path: .github/spellcheck-settings.yml
task_name: Markdown
typos:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install typos
uses: taiki-e/install-action@ad95d4e02e061d4390c4b66ef5ed56c7fee3d2ce # v2.58.17
uses: taiki-e/install-action@288875dd3d64326724fa6d9593062d9f8ba0b131 # v2.67.30
with:
tool: typos
- name: Run typos
deps/libvalkey/CMakeLists.txt
Vendored
+88 -52
View File
@@ -11,17 +11,19 @@ getDefinedVersion(VERSION_MAJOR)
getDefinedVersion(VERSION_MINOR)
getDefinedVersion(VERSION_PATCH)
set(VERSION "${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}")
MESSAGE("Detected version: ${VERSION}")
message("Detected libvalkey version: ${VERSION}")
PROJECT(valkey LANGUAGES "C" VERSION "${VERSION}")
project(libvalkey LANGUAGES "C" VERSION "${VERSION}")
INCLUDE(GNUInstallDirs)
option(ENABLE_THREADS "Enable thread-safe initialization" ON)
OPTION(BUILD_SHARED_LIBS "Build shared libraries" ON)
OPTION(ENABLE_TLS "Build valkey_tls for TLS support" OFF)
OPTION(DISABLE_TESTS "If tests should be compiled or not" OFF)
OPTION(ENABLE_EXAMPLES "Enable building valkey examples" OFF)
option(ENABLE_IPV6_TESTS "Enable IPv6 tests requiring special prerequisites" OFF)
OPTION(ENABLE_RDMA "Build valkey_rdma for RDMA support" OFF)
OPTION(ENABLE_DLOPEN_RDMA "Build valkey_rdma with dynamic loading" OFF)
# Libvalkey requires C99 (-std=c99)
SET(CMAKE_C_STANDARD 99)
@@ -70,18 +72,36 @@ set_target_properties(valkey PROPERTIES
SOVERSION "${VERSION_MAJOR}"
VERSION "${VERSION}")
IF(MSVC)
SET_TARGET_PROPERTIES(valkey
PROPERTIES COMPILE_FLAGS /Z7)
ENDIF()
if(MSVC)
# Produce object files that contain debug info.
target_compile_options(valkey PRIVATE /Z7)
endif()
set(valkey_link_libraries)
set(valkey_compile_definitions)
IF(WIN32)
TARGET_LINK_LIBRARIES(valkey PUBLIC ws2_32 crypt32)
list(APPEND valkey_link_libraries ws2_32 crypt32)
ELSEIF(CMAKE_SYSTEM_NAME MATCHES "FreeBSD")
TARGET_LINK_LIBRARIES(valkey PUBLIC m)
list(APPEND valkey_link_libraries m)
ELSEIF(CMAKE_SYSTEM_NAME MATCHES "SunOS")
TARGET_LINK_LIBRARIES(valkey PUBLIC socket)
list(APPEND valkey_link_libraries socket)
ENDIF()
if(ENABLE_THREADS)
find_package(Threads REQUIRED)
list(APPEND valkey_link_libraries Threads::Threads)
list(APPEND valkey_compile_definitions VALKEY_USE_THREADS)
endif()
if(valkey_link_libraries)
target_link_libraries(valkey PUBLIC ${valkey_link_libraries})
endif()
if(valkey_compile_definitions)
target_compile_definitions(valkey PRIVATE ${valkey_compile_definitions})
endif()
TARGET_INCLUDE_DIRECTORIES(valkey
PUBLIC
$<INSTALL_INTERFACE:include>
@@ -94,8 +114,9 @@ TARGET_INCLUDE_DIRECTORIES(valkey
CONFIGURE_FILE(valkey.pc.in valkey.pc @ONLY)
set(CPACK_PACKAGE_VENDOR "Valkey")
set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "Minimalistic C client library for Valkey")
set(CPACK_PACKAGE_DESCRIPTION "\
Libvalkey is a minimalistic C client library for the Valkey, KeyDB, and Redis databases
Libvalkey is a minimalistic C client library for the Valkey, KeyDB, and Redis databases.
It is minimalistic because it just adds minimal support for the protocol, \
but at the same time it uses a high level printf-alike API in order to make \
@@ -107,7 +128,7 @@ reply parser that is decoupled from the I/O layer. It is a stream parser designe
for easy reusability, which can for instance be used in higher level language bindings \
for efficient reply parsing.
valkey only supports the binary-safe RESP protocol, so you can use it with any Redis \
Libvalkey only supports the binary-safe RESP protocol, so you can use it with any Redis \
compatible server >= 1.2.0.
The library comes with multiple APIs. There is the synchronous API, the asynchronous API \
@@ -116,7 +137,9 @@ set(CPACK_PACKAGE_HOMEPAGE_URL "https://github.com/valkey-io/libvalkey")
set(CPACK_PACKAGE_CONTACT "michael dot grunder at gmail dot com")
set(CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON)
set(CPACK_RPM_PACKAGE_AUTOREQPROV ON)
set(CPACK_RPM_PACKAGE_DESCRIPTION "${CPACK_PACKAGE_DESCRIPTION}")
set(CPACK_RPM_PACKAGE_GROUP "Productivity/Databases/Clients")
set(CPACK_RPM_PACKAGE_LICENSE "BSD-3-Clause")
include(CPack)
INSTALL(TARGETS valkey
@@ -125,17 +148,23 @@ INSTALL(TARGETS valkey
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR})
if (MSVC AND BUILD_SHARED_LIBS)
INSTALL(FILES $<TARGET_PDB_FILE:valkey>
DESTINATION ${CMAKE_INSTALL_BINDIR}
CONFIGURATIONS Debug RelWithDebInfo)
if(MSVC AND BUILD_SHARED_LIBS)
# Install linker generated program database file.
install(FILES $<TARGET_PDB_FILE:valkey>
DESTINATION ${CMAKE_INSTALL_BINDIR}
CONFIGURATIONS Debug RelWithDebInfo)
endif()
# Install public headers
install(DIRECTORY include/valkey
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}
PATTERN "tls.h" EXCLUDE
PATTERN "rdma.h" EXCLUDE)
PATTERN "rdma.h" EXCLUDE
PATTERN "adapters/macosx.h" EXCLUDE)
if(APPLE)
install(FILES include/valkey/adapters/macosx.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/valkey/adapters)
endif()
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/valkey.pc
DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig)
@@ -187,23 +216,16 @@ IF(ENABLE_TLS)
$<BUILD_INTERFACE:${SDS_INCLUDE_DIR}>
)
IF (APPLE AND BUILD_SHARED_LIBS)
SET_PROPERTY(TARGET valkey_tls PROPERTY LINK_FLAGS "-Wl,-undefined -Wl,dynamic_lookup")
ENDIF()
set_target_properties(valkey_tls PROPERTIES
C_VISIBILITY_PRESET hidden
WINDOWS_EXPORT_ALL_SYMBOLS TRUE
SOVERSION "${VERSION_MAJOR}"
VERSION "${VERSION}")
IF(MSVC)
SET_TARGET_PROPERTIES(valkey_tls
PROPERTIES COMPILE_FLAGS /Z7)
ENDIF()
TARGET_LINK_LIBRARIES(valkey_tls PRIVATE OpenSSL::SSL)
if(WIN32 OR CYGWIN)
target_link_libraries(valkey_tls PRIVATE valkey)
if(MSVC)
# Produce object files that contain debug info.
target_compile_options(valkey_tls PRIVATE /Z7)
endif()
target_link_libraries(valkey_tls PRIVATE valkey::valkey OpenSSL::SSL)
CONFIGURE_FILE(valkey_tls.pc.in valkey_tls.pc @ONLY)
INSTALL(TARGETS valkey_tls
@@ -216,10 +238,11 @@ IF(ENABLE_TLS)
install(FILES include/valkey/tls.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/valkey)
if (MSVC AND BUILD_SHARED_LIBS)
INSTALL(FILES $<TARGET_PDB_FILE:valkey_tls>
DESTINATION ${CMAKE_INSTALL_BINDIR}
CONFIGURATIONS Debug RelWithDebInfo)
if(MSVC AND BUILD_SHARED_LIBS)
# Install linker generated program database file.
install(FILES $<TARGET_PDB_FILE:valkey_tls>
DESTINATION ${CMAKE_INSTALL_BINDIR}
CONFIGURATIONS Debug RelWithDebInfo)
endif()
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/valkey_tls.pc
@@ -248,13 +271,20 @@ IF(ENABLE_TLS)
ENDIF()
if(ENABLE_RDMA)
find_library(RDMACM_LIBRARIES rdmacm REQUIRED)
find_library(IBVERBS_LIBRARIES ibverbs REQUIRED)
set(valkey_rdma_sources src/rdma.c)
add_library(valkey_rdma ${valkey_rdma_sources})
add_library(valkey::valkey_rdma ALIAS valkey_rdma)
if (ENABLE_DLOPEN_RDMA)
message(STATUS "libvalkey: Building RDMA with dynamic loading (dlopen)")
target_compile_definitions(valkey_rdma PRIVATE DLOPEN_RDMA)
else()
message(STATUS "libvalkey: Building RDMA with static/hard linking")
find_library(RDMACM_LIBRARIES rdmacm REQUIRED)
find_library(IBVERBS_LIBRARIES ibverbs REQUIRED)
target_link_libraries(valkey_rdma LINK_PRIVATE ${RDMACM_LIBRARIES} ${IBVERBS_LIBRARIES})
endif()
target_link_libraries(valkey_rdma LINK_PRIVATE ${RDMACM_LIBRARIES} ${IBVERBS_LIBRARIES})
target_include_directories(valkey_rdma
PRIVATE
$<INSTALL_INTERFACE:include>
@@ -303,26 +333,32 @@ endif()
# Add tests
if(NOT DISABLE_TESTS)
if(BUILD_SHARED_LIBS)
# Test using a static library since symbols are not hidden then.
# Use same source, include dirs and dependencies as the shared library.
add_library(valkey_unittest STATIC ${valkey_sources})
get_target_property(include_directories valkey::valkey INCLUDE_DIRECTORIES)
target_include_directories(valkey_unittest PUBLIC ${include_directories})
get_target_property(link_libraries valkey::valkey LINK_LIBRARIES)
if(link_libraries)
target_link_libraries(valkey_unittest PUBLIC ${link_libraries})
endif()
# Create libvalkey_unittest.a in the tests directory.
set_target_properties(valkey_unittest PROPERTIES
ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/tests")
else()
# Target is an alias for the static library.
add_library(valkey_unittest ALIAS valkey)
# Unit tests uses a static library to ensure all symbols are visible.
# This single library also bundles TLS and RDMA when enabled.
add_library(valkey_unittest STATIC ${valkey_sources} ${valkey_tls_sources} ${valkey_rdma_sources})
# Mirror the include directories.
get_target_property(include_directories valkey::valkey INCLUDE_DIRECTORIES)
target_include_directories(valkey_unittest PUBLIC ${include_directories})
if(valkey_compile_definitions)
target_compile_definitions(valkey_unittest PRIVATE ${valkey_compile_definitions})
endif()
if(valkey_link_libraries)
target_link_libraries(valkey_unittest PUBLIC ${valkey_link_libraries})
endif()
if(ENABLE_TLS)
target_link_libraries(valkey_unittest PRIVATE OpenSSL::SSL)
endif()
if(ENABLE_RDMA)
target_link_libraries(valkey_unittest PRIVATE ${RDMACM_LIBRARIES} ${IBVERBS_LIBRARIES})
endif()
# Create libvalkey_unittest.a in the tests directory.
set_target_properties(valkey_unittest PROPERTIES
ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/tests")
# Make sure ctest prints the output when a test fails.
# Must be set before including CTest.
set(CMAKE_CTEST_ARGUMENTS "--output-on-failure")
include(CTest)
add_subdirectory(tests)
deps/libvalkey/CONTRIBUTING.md
Vendored
+1 -1
View File
@@ -69,7 +69,7 @@ Adhere to the existing coding style and make sure to mimic best possible.
### Code formatting
When making a change, please use `git clang-format` or [format-files.sh](./scripts/format-files.sh) to format your changes properly.
This repository is currently using `clang-format` 18.1.3 to format the code, which can be installed using `pip install clang-format==18.1.3` or other preferred method.
This repository is currently using `clang-format` 18.1.8 to format the code, which can be installed using `pip install clang-format==18.1.8` or other preferred method.
## Running cluster tests
deps/libvalkey/Makefile
Vendored
+68 -20
View File
@@ -86,7 +86,7 @@ DYLIB_MAJOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(LIBVALKEY_MAJOR)
DYLIB_ROOT_NAME=$(LIBNAME).$(DYLIBSUFFIX)
DYLIBNAME=$(LIB_DIR)/$(DYLIB_ROOT_NAME)
DYLIB_MAKE_CMD=$(CC) $(OPTIMIZATION) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(DYLIB_MAJOR_NAME)
DYLIB_MAKE_CMD=$(CC) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(DYLIB_MAJOR_NAME)
STLIBNAME=$(LIB_DIR)/$(LIBNAME).$(STLIBSUFFIX)
STLIB_MAKE_CMD=$(AR) rcs
@@ -99,7 +99,7 @@ TLS_DYLIB_MAJOR_NAME=$(TLS_LIBNAME).$(DYLIBSUFFIX).$(LIBVALKEY_MAJOR)
TLS_DYLIB_ROOT_NAME=$(TLS_LIBNAME).$(DYLIBSUFFIX)
TLS_DYLIBNAME=$(LIB_DIR)/$(TLS_LIBNAME).$(DYLIBSUFFIX)
TLS_STLIBNAME=$(LIB_DIR)/$(TLS_LIBNAME).$(STLIBSUFFIX)
TLS_DYLIB_MAKE_CMD=$(CC) $(OPTIMIZATION) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(TLS_DYLIB_MAJOR_NAME)
TLS_DYLIB_MAKE_CMD=$(CC) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(TLS_DYLIB_MAJOR_NAME)
USE_TLS?=0
@@ -130,14 +130,20 @@ RDMA_DYLIB_MAJOR_NAME=$(RDMA_LIBNAME).$(DYLIBSUFFIX).$(LIBVALKEY_MAJOR)
RDMA_DYLIB_ROOT_NAME=$(RDMA_LIBNAME).$(DYLIBSUFFIX)
RDMA_DYLIBNAME=$(LIB_DIR)/$(RDMA_LIBNAME).$(DYLIBSUFFIX)
RDMA_STLIBNAME=$(LIB_DIR)/$(RDMA_LIBNAME).$(STLIBSUFFIX)
RDMA_DYLIB_MAKE_CMD=$(CC) $(OPTIMIZATION) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(RDMA_DYLIB_MAJOR_NAME)
RDMA_DYLIB_MAKE_CMD=$(CC) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(RDMA_DYLIB_MAJOR_NAME)
USE_RDMA?=0
USE_DLOPEN_RDMA?=0
ifeq ($(USE_RDMA),1)
RDMA_SOURCES=$(SRC_DIR)/rdma.c
RDMA_OBJS=$(patsubst $(SRC_DIR)/%.c,$(OBJ_DIR)/%.o,$(RDMA_SOURCES))
RDMA_LDFLAGS=-lrdmacm -libverbs
ifeq ($(USE_DLOPEN_RDMA),1)
CFLAGS += -DDLOPEN_RDMA
RDMA_LDFLAGS=
else
RDMA_LDFLAGS=-lrdmacm -libverbs
endif
# This is required for test.c only
CFLAGS+=-DVALKEY_TEST_RDMA
RDMA_STLIB=$(RDMA_STLIBNAME)
@@ -182,28 +188,63 @@ ifeq ($(USE_TLS),1)
TLS_LDFLAGS+=-lssl -lcrypto
endif
ifeq ($(uname_S),FreeBSD)
LDFLAGS += -lm
else ifeq ($(UNAME_S),SunOS)
ifeq ($(shell $(CC) -V 2>&1 | grep -iq 'sun\|studio' && echo true),true)
USE_THREADS?=1
ifeq ($(USE_THREADS),1)
CFLAGS+=-DVALKEY_USE_THREADS
endif
PTHREAD_FLAGS :=
ifeq ($(uname_S),Linux)
ifeq ($(USE_THREADS),1)
PTHREAD_FLAGS += -pthread
endif
else ifeq ($(uname_S),FreeBSD)
REAL_LDFLAGS += -lm
ifeq ($(USE_THREADS),1)
PTHREAD_FLAGS += -pthread
endif
else ifeq ($(uname_S),SunOS)
# Solaris' default grep doesn't have -E so we need two checks
CC_VERSION := $(shell $(CC) -V 2>&1 || echo unknown)
ifneq (,$(findstring Sun C,$(CC_VERSION)))
HAVE_SUN_CC := 1
else ifneq (,$(findstring Studio,$(CC_VERSION)))
HAVE_SUN_CC := 1
endif
ifeq ($(HAVE_SUN_CC),1)
SUN_SHARED_FLAG = -G
REAL_CFLAGS += -mt
ifeq ($(USE_THREADS),1)
PTHREAD_FLAGS += -mt
endif
else
SUN_SHARED_FLAG = -shared
ifeq ($(USE_THREADS),1)
PTHREAD_FLAGS += -pthread
endif
endif
REAL_LDFLAGS += -ldl -lnsl -lsocket
DYLIB_MAKE_CMD = $(CC) $(OPTIMIZATION) $(SUN_SHARED_FLAG) -o $(DYLIBNAME) -h $(DYLIB_PATCH_NAME) $(LDFLAGS)
TLS_DYLIB_MAKE_CMD = $(CC) $(SUN_SHARED_FLAG) -o $(TLS_DYLIBNAME) -h $(TLS_DYLIB_PATCH_NAME) $(LDFLAGS) $(TLS_LDFLAGS)
DYLIB_MAKE_CMD = $(CC) $(SUN_SHARED_FLAG) -h $(DYLIB_PATCH_NAME)
TLS_DYLIB_MAKE_CMD = $(CC) $(SUN_SHARED_FLAG) -h $(TLS_DYLIB_PATCH_NAME)
else ifeq ($(uname_S),Darwin)
ifeq ($(USE_THREADS),1)
PTHREAD_FLAGS += -pthread
endif
DYLIBSUFFIX=dylib
DYLIB_PATCH_NAME=$(LIBNAME).$(LIBVALKEY_MAJOR).$(LIBVALKEY_MINOR).$(LIBVALKEY_PATCH).$(DYLIBSUFFIX)
DYLIB_MAJOR_NAME=$(LIBNAME).$(LIBVALKEY_MAJOR).$(DYLIBSUFFIX)
DYLIB_MAKE_CMD=$(CC) -dynamiclib -Wl,-install_name,$(PREFIX)/$(LIBRARY_PATH)/$(DYLIB_PATCH_NAME) -o $(DYLIBNAME) $(LDFLAGS)
DYLIB_MAKE_CMD=$(CC) -dynamiclib \
-Wl,-install_name,$(PREFIX)/$(LIBRARY_PATH)/$(DYLIB_PATCH_NAME)
TLS_DYLIB_PATCH_NAME=$(TLS_LIBNAME).$(LIBVALKEY_MAJOR).$(LIBVALKEY_MINOR).$(LIBVALKEY_PATCH).$(DYLIBSUFFIX)
TLS_DYLIB_MAJOR_NAME=$(TLS_LIBNAME).$(LIBVALKEY_MAJOR).$(DYLIBSUFFIX)
TLS_DYLIB_MAKE_CMD=$(CC) -dynamiclib -Wl,-install_name,$(PREFIX)/$(LIBRARY_PATH)/$(TLS_DYLIB_PATCH_NAME) -o $(TLS_DYLIBNAME) $(LDFLAGS) $(TLS_LDFLAGS)
DYLIB_PLUGIN=-Wl,-undefined -Wl,dynamic_lookup
TLS_DYLIB_MAKE_CMD=$(CC) -dynamiclib \
-Wl,-install_name,$(PREFIX)/$(LIBRARY_PATH)/$(TLS_DYLIB_PATCH_NAME)
endif
REAL_LDFLAGS += $(PTHREAD_FLAGS)
all: dynamic static pkgconfig tests
$(DYLIBNAME): $(OBJS) | $(LIB_DIR)
@@ -212,14 +253,16 @@ $(DYLIBNAME): $(OBJS) | $(LIB_DIR)
$(STLIBNAME): $(OBJS) | $(LIB_DIR)
$(STLIB_MAKE_CMD) $(STLIBNAME) $(OBJS)
$(TLS_DYLIBNAME): $(TLS_OBJS)
$(TLS_DYLIB_MAKE_CMD) $(DYLIB_PLUGIN) -o $(TLS_DYLIBNAME) $(TLS_OBJS) $(REAL_LDFLAGS) $(LDFLAGS) $(TLS_LDFLAGS)
$(TLS_DYLIBNAME): $(TLS_OBJS) $(DYLIBNAME) | $(LIB_DIR)
$(TLS_DYLIB_MAKE_CMD) -o $(TLS_DYLIBNAME) \
$(TLS_OBJS) $(REAL_LDFLAGS) $(DYLIBNAME) $(TLS_LDFLAGS)
$(TLS_STLIBNAME): $(TLS_OBJS)
$(STLIB_MAKE_CMD) $(TLS_STLIBNAME) $(TLS_OBJS)
$(RDMA_DYLIBNAME): $(RDMA_OBJS)
$(RDMA_DYLIB_MAKE_CMD) $(DYLIB_PLUGIN) -o $(RDMA_DYLIBNAME) $(RDMA_OBJS) $(REAL_LDFLAGS) $(LDFLAGS) $(RDMA_LDFLAGS)
$(RDMA_DYLIBNAME): $(RDMA_OBJS) $(DYLIBNAME) | $(LIB_DIR)
$(RDMA_DYLIB_MAKE_CMD) -o $(RDMA_DYLIBNAME) \
$(RDMA_OBJS) $(REAL_LDFLAGS) $(DYLIBNAME) $(RDMA_LDFLAGS)
$(RDMA_STLIBNAME): $(RDMA_OBJS)
$(STLIB_MAKE_CMD) $(RDMA_STLIBNAME) $(RDMA_OBJS)
@@ -230,8 +273,8 @@ $(OBJ_DIR)/%.o: $(SRC_DIR)/%.c | $(OBJ_DIR)
$(OBJ_DIR)/%.o: $(TEST_DIR)/%.c | $(OBJ_DIR)
$(CC) -std=c99 -pedantic $(REAL_CFLAGS) -I$(INCLUDE_DIR) -I$(SDS_INCLUDE_DIR) -I$(DICT_INCLUDE_DIR) -I$(SRC_DIR) -MMD -MP -c $< -o $@
$(TEST_DIR)/%: $(OBJ_DIR)/%.o $(STLIBNAME)
$(CC) -o $@ $< $(RDMA_STLIB) $(STLIBNAME) $(TLS_STLIB) $(LDFLAGS) $(TEST_LDFLAGS)
$(TEST_DIR)/%: $(OBJ_DIR)/%.o $(STLIBNAME) $(TLS_STLIB)
$(CC) -o $@ $< $(RDMA_STLIB) $(STLIBNAME) $(TLS_STLIB) $(REAL_LDFLAGS) $(TEST_LDFLAGS)
$(OBJ_DIR):
mkdir -p $(OBJ_DIR)
@@ -249,7 +292,12 @@ pkgconfig: $(PKGCONFNAME) $(TLS_PKGCONF) $(RDMA_PKGCONF)
TEST_LDFLAGS = $(TLS_LDFLAGS) $(RDMA_LDFLAGS)
ifeq ($(USE_TLS),1)
TEST_LDFLAGS += -pthread
# Tests need pthreads if TLS is enabled, but only add it once
ifeq ($(HAVE_SUN_CC),1)
TEST_LDFLAGS += -mt
else ifeq (,$(findstring -pthread,$(REAL_LDFLAGS) $(TEST_LDFLAGS)))
TEST_LDFLAGS += -pthread
endif
endif
ifeq ($(TEST_ASYNC),1)
TEST_LDFLAGS += -levent
deps/libvalkey/examples/CMakeLists.txt
Vendored
+20 -9
View File
@@ -1,6 +1,12 @@
cmake_minimum_required(VERSION 3.7.0)
project(examples LANGUAGES C)
option(ENABLE_THREADS "Enable thread-safe initialization" ON)
if(ENABLE_THREADS)
find_package(Threads REQUIRED)
endif()
if(CMAKE_SOURCE_DIR STREQUAL PROJECT_SOURCE_DIR)
# This CMakeLists.txt is used standalone to build the examples.
# Find required valkey package to get include paths.
@@ -47,23 +53,26 @@ if(LIBEV)
endif()
# Examples using libevent
find_path(LIBEVENT event.h)
if(LIBEVENT)
find_path(LIBEVENT event2/event.h)
find_library(LIBEVENT_LIBRARY event)
if(LIBEVENT AND LIBEVENT_LIBRARY)
include_directories(${LIBEVENT})
add_executable(example-async-libevent async-libevent.c)
target_link_libraries(example-async-libevent valkey::valkey event)
target_link_libraries(example-async-libevent valkey::valkey ${LIBEVENT_LIBRARY})
add_executable(example-cluster-async cluster-async.c)
target_link_libraries(example-cluster-async valkey::valkey event)
target_link_libraries(example-cluster-async valkey::valkey ${LIBEVENT_LIBRARY})
add_executable(example-cluster-clientside-caching-async cluster-clientside-caching-async.c)
target_link_libraries(example-cluster-clientside-caching-async valkey::valkey event)
target_link_libraries(example-cluster-clientside-caching-async valkey::valkey ${LIBEVENT_LIBRARY})
if(ENABLE_TLS)
add_executable(example-async-libevent-tls async-libevent-tls.c)
target_link_libraries(example-async-libevent-tls valkey::valkey valkey::valkey_tls event)
target_link_libraries(example-async-libevent-tls valkey::valkey valkey::valkey_tls ${LIBEVENT_LIBRARY})
add_executable(example-cluster-async-tls cluster-async-tls.c)
target_link_libraries(example-cluster-async-tls valkey::valkey valkey::valkey_tls event)
target_link_libraries(example-cluster-async-tls valkey::valkey valkey::valkey_tls ${LIBEVENT_LIBRARY})
endif()
endif()
@@ -76,9 +85,11 @@ endif()
# Examples using libuv
find_path(LIBUV uv.h)
if(LIBUV)
find_library(LIBUV_LIBRARY uv)
if(LIBUV AND LIBUV_LIBRARY)
include_directories(${LIBUV})
add_executable(example-async-libuv async-libuv.c)
target_link_libraries(example-async-libuv valkey::valkey uv)
target_link_libraries(example-async-libuv valkey::valkey ${LIBUV_LIBRARY})
endif()
# Examples using libsystemd
deps/libvalkey/examples/cluster-clientside-caching-async.c
Vendored
+5
View File
@@ -26,6 +26,7 @@ void modifyKey(const char *key, const char *value);
* and sets the push callback in the libvalkey context. */
void connectCallback(valkeyAsyncContext *ac, int status) {
assert(status == VALKEY_OK);
(void)status; /* Suppress unused warning when NDEBUG is defined. */
valkeyAsyncSetPushCallback(ac, pushCallback);
valkeyAsyncCommand(ac, NULL, NULL, "HELLO 3");
valkeyAsyncCommand(ac, NULL, NULL, "CLIENT TRACKING ON");
@@ -48,6 +49,7 @@ void eventCallback(const valkeyClusterContext *cc, int event, void *privdata) {
int status =
valkeyClusterAsyncCommand(acc, setCallback, NULL, "SET %s 1", KEY);
assert(status == VALKEY_OK);
(void)status; /* Suppress unused warning when NDEBUG is defined. */
}
}
@@ -62,6 +64,7 @@ void setCallback(valkeyClusterAsyncContext *acc, void *r, void *privdata) {
int status =
valkeyClusterAsyncCommand(acc, getCallback1, NULL, "GET %s", KEY);
assert(status == VALKEY_OK);
(void)status; /* Suppress unused warning when NDEBUG is defined. */
}
/* Message callback for the first 'GET' command. Modifies the key to
@@ -82,6 +85,7 @@ void getCallback1(valkeyClusterAsyncContext *acc, void *r, void *privdata) {
int status =
valkeyClusterAsyncCommand(acc, getCallback2, NULL, "GET %s", KEY);
assert(status == VALKEY_OK);
(void)status; /* Suppress unused warning when NDEBUG is defined. */
}
/* Push message callback handling invalidation messages. */
@@ -121,6 +125,7 @@ void getCallback2(valkeyClusterAsyncContext *acc, void *r, void *privdata) {
/* A disconnect callback should invalidate all cached keys. */
void disconnectCallback(const valkeyAsyncContext *ac, int status) {
assert(status == VALKEY_OK);
(void)status; /* Suppress unused warning when NDEBUG is defined. */
printf("Disconnected from %s:%d\n", ac->c.tcp.host, ac->c.tcp.port);
printf("Invalidate all\n");
deps/libvalkey/include/valkey/cluster.h
Vendored
+5
View File
@@ -93,6 +93,7 @@ typedef struct valkeyClusterContext {
int max_retry_count; /* Allowed retry attempts */
char *username; /* Authenticate using user */
char *password; /* Authentication password */
int select_db;
struct dict *nodes; /* Known valkeyClusterNode's */
uint64_t route_version; /* Increased when the node lookup table changes */
@@ -163,6 +164,10 @@ typedef struct {
const char *password; /* Authentication password. */
int max_retry; /* Allowed retry attempts. */
/* Select a logical database after a successful connect.
* Default 0, i.e. the SELECT command is not sent. */
int select_db;
/* Common callbacks. */
/* A hook to get notified when certain events occur. The `event` is set to
deps/libvalkey/include/valkey/valkey.h
Vendored
+8 -2
View File
@@ -50,8 +50,8 @@ typedef SSIZE_T ssize_t;
#include <stdint.h> /* uintXX_t, etc */
#define LIBVALKEY_VERSION_MAJOR 0
#define LIBVALKEY_VERSION_MINOR 2
#define LIBVALKEY_VERSION_PATCH 1
#define LIBVALKEY_VERSION_MINOR 4
#define LIBVALKEY_VERSION_PATCH 0
/* Connection type can be blocking or non-blocking and is set in the
* least significant bit of the flags field in valkeyContext. */
@@ -270,6 +270,12 @@ typedef struct valkeyContextFuncs {
* these functions shall return a value < 0. In the event of a
* recoverable error, they should return 0. */
ssize_t (*read)(struct valkeyContext *, char *, size_t);
/* ZC means zero copy, it provides underlay transport layer buffer directly,
* so it has better performance than generic read. After consuming the read
* buffer, it's necessary to notify the underlay transport to advance the
* read buffer by read_zc_done. */
ssize_t (*read_zc)(struct valkeyContext *, char **);
ssize_t (*read_zc_done)(struct valkeyContext *);
ssize_t (*write)(struct valkeyContext *);
int (*set_timeout)(struct valkeyContext *, const struct timeval);
} valkeyContextFuncs;
deps/libvalkey/scripts/format-files.sh
Vendored
+1 -1
View File
@@ -1,5 +1,5 @@
#!/bin/sh
find examples include libvalkey src tests \
find examples include src tests \
\( -name '*.c' -or -name '*.cpp' -or -name '*.h' \) \
-exec clang-format -i {} + ;
deps/libvalkey/src/async.c
Vendored
+92 -34
View File
@@ -46,6 +46,7 @@
#include "async_private.h"
#include "net.h"
#include "valkey_private.h"
#include "vkutil.h"
#include <dict.h>
#include <sds.h>
@@ -61,6 +62,7 @@
typedef struct {
sds command;
size_t len;
valkeyCallbackFn *user_callback;
void *user_priv_data;
} ssubscribeCallbackData;
@@ -808,21 +810,78 @@ void valkeyAsyncHandleTimeout(valkeyAsyncContext *ac) {
valkeyAsyncDisconnectInternal(ac);
}
/* Sets a pointer to the first argument and its length starting at p. Returns
* the number of bytes to skip to get to the following argument. */
static const char *nextArgument(const char *start, const char **str, size_t *len) {
const char *p = start;
if (p[0] != '$') {
p = strchr(p, '$');
if (p == NULL)
static inline int vk_isdigit_ascii(char c) {
return (unsigned)(c - '0') < 10;
}
#define MAX_BULK_LEN (512ULL * 1024ULL * 1024ULL)
vk_static_assert(MAX_BULK_LEN < (UINT64_MAX - 9U) / 10);
static const char *parseBulkLen(const char *p, const char *end, uint64_t *len) {
uint64_t acc = 0;
assert(p != NULL && end != NULL && end - p >= 0 && len != NULL);
if (end == p || !vk_isdigit_ascii(*p))
return NULL;
while (p < end && vk_isdigit_ascii(*p)) {
unsigned d = *p - '0';
acc = acc * 10 + d;
if (acc > (uint64_t)MAX_BULK_LEN)
return NULL;
p++;
}
*len = (int)strtol(p + 1, NULL, 10);
p = strchr(p, '\r');
assert(p);
*len = acc;
return p;
}
/* Find the next argument in a command buffer, i.e. find the next bulkstring
* in an array of bulkstrings.
* Returns a pointer to the end of a found argument, which can be used when
* finding following arguments, or NULL when an argument is not found.
* The found string is returned by pointer via `str` and length in `strlen`. */
static const char *nextArgument(const char *buf, size_t buflen, const char **str, size_t *strlen) {
if (buf == NULL || buflen == 0)
goto error;
const char *p = buf;
/* Find a bulkstring identifier. */
if (p[0] != '$') {
if ((p = memchr(p, '$', buflen)) == NULL)
goto error;
}
p++; /* Skip found '$' */
uint64_t len;
p = parseBulkLen(p, buf + buflen, &len);
if (p == NULL)
goto error;
/* Calculate end pointer for \r\n<payload>\r\n */
const char *end = p + 2 + len + 2;
/* Validate the parsed length and field separators. */
if ((size_t)(end - buf) > buflen || p[0] != '\r' || p[len + 2] != '\r')
goto error;
/* Return pointer to the string, length, and pointer to next element. */
*str = p + 2;
return p + 2 + (*len) + 2;
*strlen = len;
if ((size_t)(end - buf) == buflen) /* No more data in buffer? */
return NULL;
return end;
error:
*str = NULL;
*strlen = 0;
return NULL;
}
void valkeySsubscribeCallback(struct valkeyAsyncContext *ac, void *reply, void *privdata) {
@@ -846,8 +905,8 @@ void valkeySsubscribeCallback(struct valkeyAsyncContext *ac, void *reply, void *
assert(r != NULL);
if (r->type == VALKEY_REPLY_ERROR) {
/*/ On CROSSSLOT, MOVED and other errors */
p = nextArgument(data->command, &cstr, &clen);
while ((p = nextArgument(p, &astr, &alen)) != NULL) {
p = nextArgument(data->command, data->len, &cstr, &clen);
while ((p = nextArgument(p, data->len - (p - data->command), &astr, &alen)) != NULL || astr != NULL) {
sname = sdsnewlen(astr, alen);
if (sname == NULL)
goto oom;
@@ -865,8 +924,8 @@ void valkeySsubscribeCallback(struct valkeyAsyncContext *ac, void *reply, void *
}
} else {
if ((r->type == VALKEY_REPLY_ARRAY || r->type == VALKEY_REPLY_PUSH) && strncasecmp(r->element[0]->str, "ssubscribe", 10) == 0) {
p = nextArgument(data->command, &cstr, &clen);
while ((p = nextArgument(p, &astr, &alen)) != NULL) {
p = nextArgument(data->command, data->len, &cstr, &clen);
while ((p = nextArgument(p, data->len - (p - data->command), &astr, &alen)) != NULL || astr != NULL) {
sname = sdsnewlen(astr, alen);
if (sname == NULL)
goto oom;
@@ -921,6 +980,18 @@ static int valkeyAsyncAppendCmdLen(valkeyAsyncContext *ac, valkeyCallbackFn *fn,
if (c->flags & (VALKEY_DISCONNECTING | VALKEY_FREEING))
return VALKEY_ERR;
/* Get the first string in the command, and don't accept empty commands. */
p = nextArgument(cmd, len, &cstr, &clen);
if (cstr == NULL)
return VALKEY_ERR;
hasnext = (p && (p[0] == '$'));
pvariant = (tolower(cstr[0]) == 'p') ? 1 : 0;
svariant = valkeyIsShardedVariant(cstr);
hasprefix = svariant || pvariant;
cstr += hasprefix;
clen -= hasprefix;
/* Setup callback */
cb.fn = fn;
cb.privdata = privdata;
@@ -928,22 +999,12 @@ static int valkeyAsyncAppendCmdLen(valkeyAsyncContext *ac, valkeyCallbackFn *fn,
cb.unsubscribe_sent = 0;
cb.subscribed = 0;
/* Find out which command will be appended. */
p = nextArgument(cmd, &cstr, &clen);
assert(p != NULL);
hasnext = (p[0] == '$');
pvariant = (tolower(cstr[0]) == 'p') ? 1 : 0;
svariant = valkeyIsShardedVariant(cstr);
hasprefix = svariant || pvariant;
cstr += hasprefix;
clen -= hasprefix;
if (hasnext && strncasecmp(cstr, "subscribe\r\n", 11) == 0) {
int was_subscribed = c->flags & VALKEY_SUBSCRIBED;
c->flags |= VALKEY_SUBSCRIBED;
/* Add every channel/pattern to the list of subscription callbacks. */
while ((p = nextArgument(p, &astr, &alen)) != NULL) {
while ((p = nextArgument(p, len - (p - cmd), &astr, &alen)) != NULL || astr != NULL) {
sname = sdsnewlen(astr, alen);
if (sname == NULL)
goto oom;
@@ -977,15 +1038,12 @@ static int valkeyAsyncAppendCmdLen(valkeyAsyncContext *ac, valkeyCallbackFn *fn,
if (ssubscribe_data == NULL)
goto oom;
/* copy command to iterate over all channels.
* actual length of cmd is actually len + 1 (see valkeyvFormatCommand).
* last byte important in nextArgument function.
*/
ssubscribe_data->command = vk_malloc(len + 1);
/* Copy command to iterate over all channels. */
ssubscribe_data->command = vk_malloc(len);
if (ssubscribe_data->command == NULL)
goto oom;
memcpy(ssubscribe_data->command, cmd, len + 1);
memcpy(ssubscribe_data->command, cmd, len);
ssubscribe_data->len = len;
ssubscribe_data->user_callback = fn;
ssubscribe_data->user_priv_data = privdata;
@@ -1015,7 +1073,7 @@ static int valkeyAsyncAppendCmdLen(valkeyAsyncContext *ac, valkeyCallbackFn *fn,
if (hasnext) {
/* Send an unsubscribe with specific channels/patterns.
* Bookkeeping the number of expected replies */
while ((p = nextArgument(p, &astr, &alen)) != NULL) {
while ((p = nextArgument(p, len - (p - cmd), &astr, &alen)) != NULL || astr != NULL) {
sname = sdsnewlen(astr, alen);
if (sname == NULL)
goto oom;
deps/libvalkey/src/cluster.c
Vendored
+52 -9
View File
@@ -358,6 +358,25 @@ error:
return VALKEY_ERR;
}
/* Select a logical database by sending the SELECT command. */
static int select_db(valkeyClusterContext *cc, valkeyContext *c) {
if (cc->select_db == 0)
return VALKEY_OK;
valkeyReply *reply = valkeyCommand(c, "SELECT %d", cc->select_db);
if (reply == NULL) {
valkeyClusterSetError(cc, VALKEY_ERR_OTHER, "Failed to select logical database");
return VALKEY_ERR;
}
if (reply->type == VALKEY_REPLY_ERROR) {
valkeyClusterSetError(cc, VALKEY_ERR_OTHER, reply->str);
freeReplyObject(reply);
return VALKEY_ERR;
}
freeReplyObject(reply);
return VALKEY_OK;
}
/**
* Return a new node with the "cluster slots" command reply.
*/
@@ -902,7 +921,7 @@ static dict *parse_cluster_nodes(valkeyClusterContext *cc, valkeyContext *c, val
int add_replicas = cc->flags & VALKEY_FLAG_PARSE_REPLICAS;
dict *replicas = NULL;
if (reply->type != VALKEY_REPLY_STRING) {
if (reply->type != VALKEY_REPLY_STRING && reply->type != VALKEY_REPLY_VERB) {
valkeyClusterSetError(cc, VALKEY_ERR_OTHER, "Unexpected reply type");
goto error;
}
@@ -1189,6 +1208,9 @@ static int valkeyClusterContextInit(valkeyClusterContext *cc,
} else {
cc->max_retry_count = CLUSTER_DEFAULT_MAX_RETRY_COUNT;
}
if (options->select_db > 0) {
cc->select_db = options->select_db;
}
if (options->initial_nodes != NULL &&
valkeyClusterSetOptionAddNodes(cc, options->initial_nodes) != VALKEY_OK) {
return VALKEY_ERR; /* err and errstr already set. */
@@ -1563,8 +1585,10 @@ valkeyContext *valkeyClusterGetValkeyContext(valkeyClusterContext *cc,
if (cc->tls && cc->tls_init_fn(c, cc->tls) != VALKEY_OK) {
valkeyClusterSetError(cc, c->err, c->errstr);
}
authenticate(cc, c); // err and errstr handled in function
/* Authenticate and select a logical database when configured.
* cc->err and cc->errstr are set when failing. */
authenticate(cc, c);
select_db(cc, c);
}
return c;
@@ -1606,6 +1630,10 @@ valkeyContext *valkeyClusterGetValkeyContext(valkeyClusterContext *cc,
valkeyFree(c);
return NULL;
}
if (select_db(cc, c) != VALKEY_OK) {
valkeyFree(c);
return NULL;
}
node->con = c;
@@ -1951,8 +1979,6 @@ ask_retry:
}
goto moved_retry;
break;
case CLUSTER_ERR_ASK:
node = getNodeFromRedirectReply(cc, c, reply, NULL);
if (node == NULL) {
@@ -1980,15 +2006,12 @@ ask_retry:
reply = NULL;
goto ask_retry;
break;
case CLUSTER_ERR_TRYAGAIN:
case CLUSTER_ERR_CLUSTERDOWN:
freeReplyObject(reply);
reply = NULL;
goto retry;
break;
default:
break;
@@ -2582,6 +2605,18 @@ static void unlinkAsyncContextAndNode(void *data) {
}
}
/* Reply callback function for SELECT */
void selectReplyCallback(valkeyAsyncContext *ac, void *r, void *privdata) {
valkeyReply *reply = (valkeyReply *)r;
valkeyClusterAsyncContext *acc = (valkeyClusterAsyncContext *)privdata;
if (reply == NULL || reply->type == VALKEY_REPLY_ERROR) {
valkeyClusterAsyncSetError(acc, VALKEY_ERR_OTHER,
"Failed to select logical database");
valkeyAsyncDisconnect(ac);
}
}
valkeyAsyncContext *
valkeyClusterGetValkeyAsyncContext(valkeyClusterAsyncContext *acc,
valkeyClusterNode *node) {
@@ -2658,6 +2693,15 @@ valkeyClusterGetValkeyAsyncContext(valkeyClusterAsyncContext *acc,
return NULL;
}
}
// Select logical database when needed
if (acc->cc.select_db > 0) {
ret = valkeyAsyncCommand(ac, selectReplyCallback, acc, "SELECT %d", acc->cc.select_db);
if (ret != VALKEY_OK) {
valkeyClusterAsyncSetError(acc, ac->c.err, ac->c.errstr);
valkeyAsyncFree(ac);
return NULL;
}
}
if (acc->attach_fn) {
ret = acc->attach_fn(ac, acc->attach_data);
@@ -2996,7 +3040,6 @@ static void valkeyClusterAsyncCallback(valkeyAsyncContext *ac, void *r,
default:
goto done;
break;
}
goto retry;
deps/libvalkey/src/cmddef.h
Vendored
+68 -2
View File
@@ -17,6 +17,15 @@ COMMAND(ACL_WHOAMI, "ACL", "WHOAMI", 2, NONE, 0)
COMMAND(APPEND, "APPEND", NULL, 3, INDEX, 1)
COMMAND(ASKING, "ASKING", NULL, 1, NONE, 0)
COMMAND(AUTH, "AUTH", NULL, -2, NONE, 0)
COMMAND(BF_ADD, "BF.ADD", NULL, 3, INDEX, 1)
COMMAND(BF_CARD, "BF.CARD", NULL, 2, NONE, 0)
COMMAND(BF_EXISTS, "BF.EXISTS", NULL, 3, INDEX, 1)
COMMAND(BF_INFO, "BF.INFO", NULL, -2, INDEX, 1)
COMMAND(BF_INSERT, "BF.INSERT", NULL, -2, INDEX, 1)
COMMAND(BF_LOAD, "BF.LOAD", NULL, 3, INDEX, 1)
COMMAND(BF_MADD, "BF.MADD", NULL, 3, INDEX, 1)
COMMAND(BF_MEXISTS, "BF.MEXISTS", NULL, 3, INDEX, 1)
COMMAND(BF_RESERVE, "BF.RESERVE", NULL, -4, INDEX, 1)
COMMAND(BGREWRITEAOF, "BGREWRITEAOF", NULL, 1, NONE, 0)
COMMAND(BGSAVE, "BGSAVE", NULL, -1, NONE, 0)
COMMAND(BITCOUNT, "BITCOUNT", NULL, -2, INDEX, 1)
@@ -33,10 +42,12 @@ COMMAND(BZMPOP, "BZMPOP", NULL, -5, KEYNUM, 2)
COMMAND(BZPOPMAX, "BZPOPMAX", NULL, -3, INDEX, 1)
COMMAND(BZPOPMIN, "BZPOPMIN", NULL, -3, INDEX, 1)
COMMAND(CLIENT_CACHING, "CLIENT", "CACHING", 3, NONE, 0)
COMMAND(CLIENT_CAPA, "CLIENT", "CAPA", -3, NONE, 0)
COMMAND(CLIENT_GETNAME, "CLIENT", "GETNAME", 2, NONE, 0)
COMMAND(CLIENT_GETREDIR, "CLIENT", "GETREDIR", 2, NONE, 0)
COMMAND(CLIENT_HELP, "CLIENT", "HELP", 2, NONE, 0)
COMMAND(CLIENT_ID, "CLIENT", "ID", 2, NONE, 0)
COMMAND(CLIENT_IMPORT_SOURCE, "CLIENT", "IMPORT-SOURCE", 3, NONE, 0)
COMMAND(CLIENT_INFO, "CLIENT", "INFO", 2, NONE, 0)
COMMAND(CLIENT_KILL, "CLIENT", "KILL", -3, NONE, 0)
COMMAND(CLIENT_LIST, "CLIENT", "LIST", -2, NONE, 0)
@@ -53,31 +64,41 @@ COMMAND(CLIENT_UNPAUSE, "CLIENT", "UNPAUSE", 2, NONE, 0)
COMMAND(CLUSTER_ADDSLOTS, "CLUSTER", "ADDSLOTS", -3, NONE, 0)
COMMAND(CLUSTER_ADDSLOTSRANGE, "CLUSTER", "ADDSLOTSRANGE", -4, NONE, 0)
COMMAND(CLUSTER_BUMPEPOCH, "CLUSTER", "BUMPEPOCH", 2, NONE, 0)
COMMAND(CLUSTER_CANCELSLOTMIGRATIONS, "CLUSTER", "CANCELSLOTMIGRATIONS", 2, NONE, 0)
COMMAND(CLUSTER_COUNT_FAILURE_REPORTS, "CLUSTER", "COUNT-FAILURE-REPORTS", 3, NONE, 0)
COMMAND(CLUSTER_COUNTKEYSINSLOT, "CLUSTER", "COUNTKEYSINSLOT", 3, NONE, 0)
COMMAND(CLUSTER_DELSLOTS, "CLUSTER", "DELSLOTS", -3, NONE, 0)
COMMAND(CLUSTER_DELSLOTSRANGE, "CLUSTER", "DELSLOTSRANGE", -4, NONE, 0)
COMMAND(CLUSTER_FAILOVER, "CLUSTER", "FAILOVER", -2, NONE, 0)
COMMAND(CLUSTER_FLUSHSLOT, "CLUSTER", "FLUSHSLOT", -3, NONE, 0)
COMMAND(CLUSTER_FLUSHSLOTS, "CLUSTER", "FLUSHSLOTS", 2, NONE, 0)
COMMAND(CLUSTER_FORGET, "CLUSTER", "FORGET", 3, NONE, 0)
COMMAND(CLUSTER_GETKEYSINSLOT, "CLUSTER", "GETKEYSINSLOT", 4, NONE, 0)
COMMAND(CLUSTER_GETSLOTMIGRATIONS, "CLUSTER", "GETSLOTMIGRATIONS", 2, NONE, 0)
COMMAND(CLUSTER_HELP, "CLUSTER", "HELP", 2, NONE, 0)
COMMAND(CLUSTER_INFO, "CLUSTER", "INFO", 2, NONE, 0)
COMMAND(CLUSTER_KEYSLOT, "CLUSTER", "KEYSLOT", 3, NONE, 0)
COMMAND(CLUSTER_LINKS, "CLUSTER", "LINKS", 2, NONE, 0)
COMMAND(CLUSTER_MEET, "CLUSTER", "MEET", -4, NONE, 0)
COMMAND(CLUSTER_MIGRATESLOTS, "CLUSTER", "MIGRATESLOTS", -4, NONE, 0)
COMMAND(CLUSTER_MYID, "CLUSTER", "MYID", 2, NONE, 0)
COMMAND(CLUSTER_MYSHARDID, "CLUSTER", "MYSHARDID", 2, NONE, 0)
COMMAND(CLUSTER_NODES, "CLUSTER", "NODES", 2, NONE, 0)
COMMAND(CLUSTER_REPLICAS, "CLUSTER", "REPLICAS", 3, NONE, 0)
COMMAND(CLUSTER_REPLICATE, "CLUSTER", "REPLICATE", 3, NONE, 0)
COMMAND(CLUSTER_REPLICATE, "CLUSTER", "REPLICATE", -3, NONE, 0)
COMMAND(CLUSTER_RESET, "CLUSTER", "RESET", -2, NONE, 0)
COMMAND(CLUSTER_SAVECONFIG, "CLUSTER", "SAVECONFIG", 2, NONE, 0)
COMMAND(CLUSTER_SET_CONFIG_EPOCH, "CLUSTER", "SET-CONFIG-EPOCH", 3, NONE, 0)
COMMAND(CLUSTER_SETSLOT, "CLUSTER", "SETSLOT", -4, NONE, 0)
COMMAND(CLUSTER_SHARDS, "CLUSTER", "SHARDS", 2, NONE, 0)
COMMAND(CLUSTER_SLAVES, "CLUSTER", "SLAVES", 3, NONE, 0)
COMMAND(CLUSTER_SLOT_STATS, "CLUSTER", "SLOT-STATS", -4, NONE, 0)
COMMAND(CLUSTER_SLOTS, "CLUSTER", "SLOTS", 2, NONE, 0)
COMMAND(CLUSTER_SYNCSLOTS, "CLUSTER", "SYNCSLOTS", -3, NONE, 0)
COMMAND(COMMANDLOG_GET, "COMMANDLOG", "GET", 4, NONE, 0)
COMMAND(COMMANDLOG_HELP, "COMMANDLOG", "HELP", 2, NONE, 0)
COMMAND(COMMANDLOG_LEN, "COMMANDLOG", "LEN", 3, NONE, 0)
COMMAND(COMMANDLOG_RESET, "COMMANDLOG", "RESET", 3, NONE, 0)
COMMAND(COMMAND_COUNT, "COMMAND", "COUNT", 2, NONE, 0)
COMMAND(COMMAND_DOCS, "COMMAND", "DOCS", -2, NONE, 0)
COMMAND(COMMAND_GETKEYS, "COMMAND", "GETKEYS", -3, NONE, 0)
@@ -96,6 +117,7 @@ COMMAND(DEBUG, "DEBUG", NULL, -2, NONE, 0)
COMMAND(DECR, "DECR", NULL, 2, INDEX, 1)
COMMAND(DECRBY, "DECRBY", NULL, 3, INDEX, 1)
COMMAND(DEL, "DEL", NULL, -2, INDEX, 1)
COMMAND(DELIFEQ, "DELIFEQ", NULL, 3, INDEX, 1)
COMMAND(DISCARD, "DISCARD", NULL, 1, NONE, 0)
COMMAND(DUMP, "DUMP", NULL, 2, INDEX, 1)
COMMAND(ECHO, "ECHO", NULL, 2, NONE, 0)
@@ -113,6 +135,11 @@ COMMAND(FCALL, "FCALL", NULL, -3, KEYNUM, 2)
COMMAND(FCALL_RO, "FCALL_RO", NULL, -3, KEYNUM, 2)
COMMAND(FLUSHALL, "FLUSHALL", NULL, -1, NONE, 0)
COMMAND(FLUSHDB, "FLUSHDB", NULL, -1, NONE, 0)
COMMAND(FT_CREATE, "FT.CREATE", NULL, -1, NONE, 0)
COMMAND(FT_DROPINDEX, "FT.DROPINDEX", NULL, 2, NONE, 0)
COMMAND(FT_INFO, "FT.INFO", NULL, 2, NONE, 0)
COMMAND(FT_SEARCH, "FT.SEARCH", NULL, -3, INDEX, 1)
COMMAND(FT__LIST, "FT._LIST", NULL, 1, NONE, 0)
COMMAND(FUNCTION_DELETE, "FUNCTION", "DELETE", 3, NONE, 0)
COMMAND(FUNCTION_DUMP, "FUNCTION", "DUMP", 2, NONE, 0)
COMMAND(FUNCTION_FLUSH, "FUNCTION", "FLUSH", -2, NONE, 0)
@@ -141,24 +168,58 @@ COMMAND(GETSET, "GETSET", NULL, 3, INDEX, 1)
COMMAND(HDEL, "HDEL", NULL, -3, INDEX, 1)
COMMAND(HELLO, "HELLO", NULL, -1, NONE, 0)
COMMAND(HEXISTS, "HEXISTS", NULL, 3, INDEX, 1)
COMMAND(HEXPIRE, "HEXPIRE", NULL, -6, INDEX, 1)
COMMAND(HEXPIREAT, "HEXPIREAT", NULL, -6, INDEX, 1)
COMMAND(HEXPIRETIME, "HEXPIRETIME", NULL, -5, INDEX, 1)
COMMAND(HGET, "HGET", NULL, 3, INDEX, 1)
COMMAND(HGETALL, "HGETALL", NULL, 2, INDEX, 1)
COMMAND(HGETEX, "HGETEX", NULL, -5, INDEX, 1)
COMMAND(HINCRBY, "HINCRBY", NULL, 4, INDEX, 1)
COMMAND(HINCRBYFLOAT, "HINCRBYFLOAT", NULL, 4, INDEX, 1)
COMMAND(HKEYS, "HKEYS", NULL, 2, INDEX, 1)
COMMAND(HLEN, "HLEN", NULL, 2, INDEX, 1)
COMMAND(HMGET, "HMGET", NULL, -3, INDEX, 1)
COMMAND(HMSET, "HMSET", NULL, -4, INDEX, 1)
COMMAND(HPERSIST, "HPERSIST", NULL, -5, INDEX, 1)
COMMAND(HPEXPIRE, "HPEXPIRE", NULL, -6, INDEX, 1)
COMMAND(HPEXPIREAT, "HPEXPIREAT", NULL, -6, INDEX, 1)
COMMAND(HPEXPIRETIME, "HPEXPIRETIME", NULL, -5, INDEX, 1)
COMMAND(HPTTL, "HPTTL", NULL, -5, INDEX, 1)
COMMAND(HRANDFIELD, "HRANDFIELD", NULL, -2, INDEX, 1)
COMMAND(HSCAN, "HSCAN", NULL, -3, INDEX, 1)
COMMAND(HSET, "HSET", NULL, -4, INDEX, 1)
COMMAND(HSETEX, "HSETEX", NULL, -6, INDEX, 1)
COMMAND(HSETNX, "HSETNX", NULL, 4, INDEX, 1)
COMMAND(HSTRLEN, "HSTRLEN", NULL, 3, INDEX, 1)
COMMAND(HTTL, "HTTL", NULL, -5, INDEX, 1)
COMMAND(HVALS, "HVALS", NULL, 2, INDEX, 1)
COMMAND(INCR, "INCR", NULL, 2, INDEX, 1)
COMMAND(INCRBY, "INCRBY", NULL, 3, INDEX, 1)
COMMAND(INCRBYFLOAT, "INCRBYFLOAT", NULL, 3, INDEX, 1)
COMMAND(INFO, "INFO", NULL, -1, NONE, 0)
COMMAND(JSON_ARRAPPEND, "JSON.ARRAPPEND", NULL, -4, INDEX, 1)
COMMAND(JSON_ARRINDEX, "JSON.ARRINDEX", NULL, 4, INDEX, 1)
COMMAND(JSON_ARRINSERT, "JSON.ARRINSERT", NULL, -5, INDEX, 1)
COMMAND(JSON_ARRLEN, "JSON.ARRLEN", NULL, 2, INDEX, 1)
COMMAND(JSON_ARRPOP, "JSON.ARRPOP", NULL, 2, INDEX, 1)
COMMAND(JSON_ARRTRIM, "JSON.ARRTRIM", NULL, 5, INDEX, 1)
COMMAND(JSON_CLEAR, "JSON.CLEAR", NULL, 2, INDEX, 1)
COMMAND(JSON_DEBUG, "JSON.DEBUG", NULL, 2, NONE, 0)
COMMAND(JSON_DEL, "JSON.DEL", NULL, 2, INDEX, 1)
COMMAND(JSON_FORGET, "JSON.FORGET", NULL, -1, NONE, 0)
COMMAND(JSON_GET, "JSON.GET", NULL, 2, INDEX, 1)
COMMAND(JSON_MGET, "JSON.MGET", NULL, -3, INDEX, 1)
COMMAND(JSON_MSET, "JSON.MSET", NULL, -4, NONE, 0)
COMMAND(JSON_NUMINCRBY, "JSON.NUMINCRBY", NULL, 4, INDEX, 1)
COMMAND(JSON_NUMMULTBY, "JSON.NUMMULTBY", NULL, 4, INDEX, 1)
COMMAND(JSON_OBJKEYS, "JSON.OBJKEYS", NULL, 2, INDEX, 1)
COMMAND(JSON_OBJLEN, "JSON.OBJLEN", NULL, 2, INDEX, 1)
COMMAND(JSON_RESP, "JSON.RESP", NULL, 2, INDEX, 1)
COMMAND(JSON_SET, "JSON.SET", NULL, 4, INDEX, 1)
COMMAND(JSON_STRAPPEND, "JSON.STRAPPEND", NULL, 3, INDEX, 1)
COMMAND(JSON_STRLEN, "JSON.STRLEN", NULL, 2, INDEX, 1)
COMMAND(JSON_TOGGLE, "JSON.TOGGLE", NULL, 2, INDEX, 1)
COMMAND(JSON_TYPE, "JSON.TYPE", NULL, 2, INDEX, 1)
COMMAND(KEYS, "KEYS", NULL, 2, NONE, 0)
COMMAND(LASTSAVE, "LASTSAVE", NULL, 1, NONE, 0)
COMMAND(LATENCY_DOCTOR, "LATENCY", "DOCTOR", 2, NONE, 0)
@@ -254,23 +315,28 @@ COMMAND(SCRIPT_FLUSH, "SCRIPT", "FLUSH", -2, NONE, 0)
COMMAND(SCRIPT_HELP, "SCRIPT", "HELP", 2, NONE, 0)
COMMAND(SCRIPT_KILL, "SCRIPT", "KILL", 2, NONE, 0)
COMMAND(SCRIPT_LOAD, "SCRIPT", "LOAD", 3, NONE, 0)
COMMAND(SCRIPT_SHOW, "SCRIPT", "SHOW", 3, NONE, 0)
COMMAND(SDIFF, "SDIFF", NULL, -2, INDEX, 1)
COMMAND(SDIFFSTORE, "SDIFFSTORE", NULL, -3, INDEX, 1)
COMMAND(SELECT, "SELECT", NULL, 2, NONE, 0)
COMMAND(SENTINEL_CKQUORUM, "SENTINEL", "CKQUORUM", 3, NONE, 0)
COMMAND(SENTINEL_CONFIG, "SENTINEL", "CONFIG", -4, NONE, 0)
COMMAND(SENTINEL_DEBUG, "SENTINEL", "DEBUG", -2, NONE, 0)
COMMAND(SENTINEL_FAILOVER, "SENTINEL", "FAILOVER", 3, NONE, 0)
COMMAND(SENTINEL_FAILOVER, "SENTINEL", "FAILOVER", -3, NONE, 0)
COMMAND(SENTINEL_FLUSHCONFIG, "SENTINEL", "FLUSHCONFIG", 2, NONE, 0)
COMMAND(SENTINEL_GET_MASTER_ADDR_BY_NAME, "SENTINEL", "GET-MASTER-ADDR-BY-NAME", 3, NONE, 0)
COMMAND(SENTINEL_GET_PRIMARY_ADDR_BY_NAME, "SENTINEL", "GET-PRIMARY-ADDR-BY-NAME", 3, NONE, 0)
COMMAND(SENTINEL_HELP, "SENTINEL", "HELP", 2, NONE, 0)
COMMAND(SENTINEL_INFO_CACHE, "SENTINEL", "INFO-CACHE", -3, NONE, 0)
COMMAND(SENTINEL_IS_MASTER_DOWN_BY_ADDR, "SENTINEL", "IS-MASTER-DOWN-BY-ADDR", 6, NONE, 0)
COMMAND(SENTINEL_IS_PRIMARY_DOWN_BY_ADDR, "SENTINEL", "IS-PRIMARY-DOWN-BY-ADDR", 6, NONE, 0)
COMMAND(SENTINEL_MASTER, "SENTINEL", "MASTER", 3, NONE, 0)
COMMAND(SENTINEL_MASTERS, "SENTINEL", "MASTERS", 2, NONE, 0)
COMMAND(SENTINEL_MONITOR, "SENTINEL", "MONITOR", 6, NONE, 0)
COMMAND(SENTINEL_MYID, "SENTINEL", "MYID", 2, NONE, 0)
COMMAND(SENTINEL_PENDING_SCRIPTS, "SENTINEL", "PENDING-SCRIPTS", 2, NONE, 0)
COMMAND(SENTINEL_PRIMARIES, "SENTINEL", "PRIMARIES", 2, NONE, 0)
COMMAND(SENTINEL_PRIMARY, "SENTINEL", "PRIMARY", 3, NONE, 0)
COMMAND(SENTINEL_REMOVE, "SENTINEL", "REMOVE", 3, NONE, 0)
COMMAND(SENTINEL_REPLICAS, "SENTINEL", "REPLICAS", 3, NONE, 0)
COMMAND(SENTINEL_RESET, "SENTINEL", "RESET", 3, NONE, 0)
deps/libvalkey/src/command.c
Vendored
+10 -12
View File
@@ -28,15 +28,12 @@
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#include "fmacros.h"
#include <assert.h>
#include <ctype.h>
#include <errno.h>
#ifndef _WIN32
#if !defined(__FreeBSD__)
#include <alloca.h>
#else
#include <stdlib.h>
#endif
#include <strings.h>
#else
#include <malloc.h>
@@ -52,6 +49,7 @@
#include <stdio.h>
#include <string.h>
#define MAX_COMMAND_LEN 64
#define LF (uint8_t)10
#define CR (uint8_t)13
@@ -99,11 +97,13 @@ static inline void to_upper(char *dst, const char *src, uint32_t len) {
* or NULL on failure. */
cmddef *valkey_lookup_cmd(const char *arg0, uint32_t arg0_len, const char *arg1,
uint32_t arg1_len) {
if (arg0_len > MAX_COMMAND_LEN)
return NULL;
char cmd[MAX_COMMAND_LEN];
char subcmd[MAX_COMMAND_LEN] = "";
int num_commands = sizeof(server_commands) / sizeof(cmddef);
/* Compare command name in uppercase. */
char *cmd = alloca(arg0_len);
to_upper(cmd, arg0, arg0_len);
char *subcmd = NULL; /* Alloca later on demand. */
/* Find the command using binary search. */
int left = 0, right = num_commands - 1;
while (left <= right) {
@@ -116,14 +116,12 @@ cmddef *valkey_lookup_cmd(const char *arg0, uint32_t arg0_len, const char *arg1,
/* If command name matches, compare subcommand if any */
if (cmp == 0 && c->subname != NULL) {
if (arg1 == NULL) {
/* Command has subcommands, but none given. */
if (arg1 == NULL || arg1_len == 0 || arg1_len > MAX_COMMAND_LEN) {
/* Command has subcommands, but none given or is too long. */
return NULL;
}
if (subcmd == NULL) {
subcmd = alloca(arg1_len);
if (subcmd[0] == '\0')
to_upper(subcmd, arg1, arg1_len);
}
cmp = strncmp(c->subname, subcmd, arg1_len);
if (cmp == 0 && strlen(c->subname) > arg1_len)
cmp = 1;
deps/libvalkey/src/conn.c
Vendored
+55 -6
View File
@@ -32,6 +32,53 @@
#include <assert.h>
#ifdef VALKEY_USE_THREADS
#if defined(__unix__) || defined(__APPLE__) || defined(__sun)
#define VALKEY_PTHREADS_ONCE 1
#elif defined(_WIN32)
#define VALKEY_WINDOWS_ONCE 1
#else
#error "No call-once implementation available on this platform"
#endif
#endif // VALKEY_USE_THREADS
static void vkRegisterFuncs(void) {
valkeyContextRegisterTcpFuncs();
valkeyContextRegisterUnixFuncs();
valkeyContextRegisterUserfdFuncs();
}
#if VALKEY_PTHREADS_ONCE
#include <pthread.h>
static void vkRegisterFuncsOnce(void) {
static pthread_once_t flag = PTHREAD_ONCE_INIT;
pthread_once(&flag, vkRegisterFuncs);
}
#elif VALKEY_WINDOWS_ONCE
#include <windows.h>
/* Windows uses a different signature for the init function */
static BOOL CALLBACK
vkRegisterFuncsWrapper(PINIT_ONCE once, PVOID param, PVOID *ctx) {
(void)once;
(void)param;
(void)ctx;
vkRegisterFuncs();
return TRUE;
}
static void vkRegisterFuncsOnce(void) {
static INIT_ONCE flag = INIT_ONCE_STATIC_INIT;
InitOnceExecuteOnce(&flag, vkRegisterFuncsWrapper, NULL, NULL);
}
#endif
static valkeyContextFuncs *valkeyContextFuncsArray[VALKEY_CONN_MAX];
int valkeyContextRegisterFuncs(valkeyContextFuncs *funcs, enum valkeyConnectionType type) {
@@ -43,14 +90,16 @@ int valkeyContextRegisterFuncs(valkeyContextFuncs *funcs, enum valkeyConnectionT
}
void valkeyContextSetFuncs(valkeyContext *c) {
static int initialized;
#ifdef VALKEY_USE_THREADS
vkRegisterFuncsOnce();
#else
static int registered = 0;
if (!initialized) {
initialized = 1;
valkeyContextRegisterTcpFuncs();
valkeyContextRegisterUnixFuncs();
valkeyContextRegisterUserfdFuncs();
if (!registered) {
registered = 1;
vkRegisterFuncs();
}
#endif
assert(c->connection_type < VALKEY_CONN_MAX);
assert(!c->funcs);
deps/libvalkey/src/fmacros.h
Vendored
+1 -1
View File
@@ -1,7 +1,7 @@
#ifndef VALKEY_FMACRO_H
#define VALKEY_FMACRO_H
#ifndef _AIX
#if !defined(_AIX) && !defined(__FreeBSD__)
#define _XOPEN_SOURCE 600
#define _POSIX_C_SOURCE 200112L
#endif
deps/libvalkey/src/rdma.c
Vendored
+182 -18
View File
@@ -47,10 +47,163 @@
#include <netdb.h>
#include <poll.h>
#include <rdma/rdma_cma.h>
#include <stdint.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <unistd.h>
#ifdef DLOPEN_RDMA
#include <dlfcn.h>
#include <stdio.h>
/* ====================================================================
* RDMA Dynamic Symbol Table
* ====================================================================
* * Why wrap these in a struct and use #define macros?
* * We must include <rdma/rdma_cma.h> and <infiniband/verbs.h> to access
* essential data structures. However, these headers also contain the
* native function prototypes. If we declare global function pointers
* with the exact same names, the compiler will throw a redeclaration
* error (symbol conflict).
* * By wrapping them in a struct and using #define macros *after* the
* includes, we cleanly intercept the function calls without violating
* C namespace rules, leaving all original call sites completely untouched.
* ==================================================================== */
static struct rdma_dyn_syms {
/* ====================================================================
* Symbols from: <rdma/rdma_cma.h>
* Compatible shared library: librdmacm.so.1
* ==================================================================== */
struct rdma_event_channel *(*rdma_create_event_channel)(void);
void (*rdma_destroy_event_channel)(struct rdma_event_channel *channel);
int (*rdma_create_id)(struct rdma_event_channel *channel, struct rdma_cm_id **id, void *context, enum rdma_port_space ps);
int (*rdma_create_qp)(struct rdma_cm_id *id, struct ibv_pd *pd, struct ibv_qp_init_attr *qp_init_attr);
int (*rdma_destroy_id)(struct rdma_cm_id *id);
int (*rdma_bind_addr)(struct rdma_cm_id *id, struct sockaddr *addr);
int (*rdma_resolve_addr)(struct rdma_cm_id *id, struct sockaddr *src_addr, struct sockaddr *dst_addr, int timeout_ms);
int (*rdma_resolve_route)(struct rdma_cm_id *id, int timeout_ms);
int (*rdma_connect)(struct rdma_cm_id *id, struct rdma_conn_param *conn_param);
int (*rdma_disconnect)(struct rdma_cm_id *id);
int (*rdma_get_cm_event)(struct rdma_event_channel *channel, struct rdma_cm_event **event);
int (*rdma_ack_cm_event)(struct rdma_cm_event *event);
int (*rdma_getaddrinfo)(const char *node, const char *service, struct rdma_addrinfo *hints, struct rdma_addrinfo **res);
void (*rdma_freeaddrinfo)(struct rdma_addrinfo *res);
const char *(*rdma_event_str)(enum rdma_cm_event_type event);
/* ====================================================================
* Symbols from: <infiniband/verbs.h>
* Compatible shared library: libibverbs.so.1
* ==================================================================== */
struct ibv_pd *(*ibv_alloc_pd)(struct ibv_context *context);
int (*ibv_dealloc_pd)(struct ibv_pd *pd);
struct ibv_comp_channel *(*ibv_create_comp_channel)(struct ibv_context *context);
int (*ibv_destroy_comp_channel)(struct ibv_comp_channel *channel);
struct ibv_cq *(*ibv_create_cq)(struct ibv_context *context, int cqe, void *cq_context, struct ibv_comp_channel *channel, int comp_vector);
int (*ibv_destroy_cq)(struct ibv_cq *cq);
struct ibv_mr *(*ibv_reg_mr)(struct ibv_pd *pd, void *addr, size_t length, int access);
int (*ibv_dereg_mr)(struct ibv_mr *mr);
int (*ibv_get_cq_event)(struct ibv_comp_channel *channel, struct ibv_cq **cq, void **cq_context);
void (*ibv_ack_cq_events)(struct ibv_cq *cq, unsigned int nevents);
int (*ibv_destroy_qp)(struct ibv_qp *qp);
} rdma_syms;
static void *rdmacm_handle = NULL;
static void *ibverbs_handle = NULL;
#define LOAD_SYM(handle, name) \
do { \
void *tmp_sym = dlsym(handle, #name); \
if (!tmp_sym) { \
fprintf(stderr, "RDMA Init Error: missing symbol %s\n", #name); \
return -1; \
} \
*(void **)(&rdma_syms.name) = tmp_sym; \
} while (0)
static int rdma_dyn_load_libs(void) {
if (rdmacm_handle && ibverbs_handle)
return 0;
const char *rdmacm_candidates[] = {"librdmacm.so.1", "librdmacm.so", NULL};
const char *verbs_candidates[] = {"libibverbs.so.1", "libibverbs.so", NULL};
for (int i = 0; !rdmacm_handle && rdmacm_candidates[i]; i++)
rdmacm_handle = dlopen(rdmacm_candidates[i], RTLD_NOW);
for (int i = 0; !ibverbs_handle && verbs_candidates[i]; i++)
ibverbs_handle = dlopen(verbs_candidates[i], RTLD_NOW);
if (!rdmacm_handle || !ibverbs_handle) {
fprintf(stderr, "Error: Required RDMA libraries (librdmacm/libibverbs) not found on this system.\n");
return -1;
}
LOAD_SYM(rdmacm_handle, rdma_create_event_channel);
LOAD_SYM(rdmacm_handle, rdma_destroy_event_channel);
LOAD_SYM(rdmacm_handle, rdma_create_id);
LOAD_SYM(rdmacm_handle, rdma_create_qp);
LOAD_SYM(rdmacm_handle, rdma_destroy_id);
LOAD_SYM(rdmacm_handle, rdma_bind_addr);
LOAD_SYM(rdmacm_handle, rdma_resolve_addr);
LOAD_SYM(rdmacm_handle, rdma_resolve_route);
LOAD_SYM(rdmacm_handle, rdma_connect);
LOAD_SYM(rdmacm_handle, rdma_disconnect);
LOAD_SYM(rdmacm_handle, rdma_get_cm_event);
LOAD_SYM(rdmacm_handle, rdma_ack_cm_event);
LOAD_SYM(rdmacm_handle, rdma_getaddrinfo);
LOAD_SYM(rdmacm_handle, rdma_freeaddrinfo);
LOAD_SYM(rdmacm_handle, rdma_event_str);
LOAD_SYM(ibverbs_handle, ibv_alloc_pd);
LOAD_SYM(ibverbs_handle, ibv_dealloc_pd);
LOAD_SYM(ibverbs_handle, ibv_create_comp_channel);
LOAD_SYM(ibverbs_handle, ibv_destroy_comp_channel);
LOAD_SYM(ibverbs_handle, ibv_create_cq);
LOAD_SYM(ibverbs_handle, ibv_destroy_cq);
LOAD_SYM(ibverbs_handle, ibv_reg_mr);
LOAD_SYM(ibverbs_handle, ibv_dereg_mr);
LOAD_SYM(ibverbs_handle, ibv_get_cq_event);
LOAD_SYM(ibverbs_handle, ibv_ack_cq_events);
LOAD_SYM(ibverbs_handle, ibv_destroy_qp);
return 0;
}
#define rdma_create_event_channel rdma_syms.rdma_create_event_channel
#define rdma_destroy_event_channel rdma_syms.rdma_destroy_event_channel
#define rdma_create_id rdma_syms.rdma_create_id
#define rdma_create_qp rdma_syms.rdma_create_qp
#define rdma_destroy_id rdma_syms.rdma_destroy_id
#define rdma_bind_addr rdma_syms.rdma_bind_addr
#define rdma_resolve_addr rdma_syms.rdma_resolve_addr
#define rdma_resolve_route rdma_syms.rdma_resolve_route
#define rdma_connect rdma_syms.rdma_connect
#define rdma_disconnect rdma_syms.rdma_disconnect
#define rdma_get_cm_event rdma_syms.rdma_get_cm_event
#define rdma_ack_cm_event rdma_syms.rdma_ack_cm_event
#define rdma_getaddrinfo rdma_syms.rdma_getaddrinfo
#define rdma_freeaddrinfo rdma_syms.rdma_freeaddrinfo
#define rdma_event_str rdma_syms.rdma_event_str
#ifdef ibv_reg_mr
#undef ibv_reg_mr
#endif
#define ibv_alloc_pd rdma_syms.ibv_alloc_pd
#define ibv_dealloc_pd rdma_syms.ibv_dealloc_pd
#define ibv_create_comp_channel rdma_syms.ibv_create_comp_channel
#define ibv_destroy_comp_channel rdma_syms.ibv_destroy_comp_channel
#define ibv_create_cq rdma_syms.ibv_create_cq
#define ibv_destroy_cq rdma_syms.ibv_destroy_cq
#define ibv_reg_mr rdma_syms.ibv_reg_mr
#define ibv_dereg_mr rdma_syms.ibv_dereg_mr
#define ibv_get_cq_event rdma_syms.ibv_get_cq_event
#define ibv_ack_cq_events rdma_syms.ibv_ack_cq_events
#define ibv_destroy_qp rdma_syms.ibv_destroy_qp
#endif /* DLOPEN_RDMA */
static valkeyContextFuncs valkeyContextRdmaFuncs;
typedef struct valkeyRdmaFeature {
@@ -129,6 +282,9 @@ typedef struct RdmaContext {
struct ibv_mr *cmd_mr;
} RdmaContext;
/* Apparently CHERI uintptr_t can be 128 bits */
vk_static_assert(sizeof(uintptr_t) <= sizeof(uint64_t));
static int valkeyRdmaCM(valkeyContext *c, long timeout);
static int valkeyRdmaSetFdBlocking(valkeyContext *c, int fd, int blocking) {
@@ -161,7 +317,7 @@ static int rdmaPostRecv(RdmaContext *ctx, struct rdma_cm_id *cm_id, valkeyRdmaCm
sge.length = length;
sge.lkey = ctx->cmd_mr->lkey;
recv_wr.wr_id = (uint64_t)cmd;
recv_wr.wr_id = (uint64_t)(uintptr_t)cmd;
recv_wr.sg_list = &sge;
recv_wr.num_sge = 1;
recv_wr.next = NULL;
@@ -302,7 +458,7 @@ static int rdmaSendCommand(valkeyContext *c, struct rdma_cm_id *cm_id, valkeyRdm
send_wr.sg_list = &sge;
send_wr.num_sge = 1;
send_wr.wr_id = (uint64_t)_cmd;
send_wr.wr_id = (uint64_t)(uintptr_t)_cmd;
send_wr.opcode = IBV_WR_SEND;
send_wr.send_flags = IBV_SEND_SIGNALED;
send_wr.next = NULL;
@@ -320,7 +476,7 @@ static int connRdmaRegisterRx(valkeyContext *c, struct rdma_cm_id *cm_id) {
valkeyRdmaCmd cmd = {0};
cmd.memory.opcode = htons(RegisterXferMemory);
cmd.memory.addr = htobe64((uint64_t)ctx->recv_buf);
cmd.memory.addr = htobe64((uint64_t)(uintptr_t)ctx->recv_buf);
cmd.memory.length = htonl(ctx->recv_length);
cmd.memory.key = htonl(ctx->recv_mr->rkey);
@@ -338,7 +494,7 @@ static int connRdmaHandleRecv(valkeyContext *c, RdmaContext *ctx, struct rdma_cm
switch (ntohs(cmd->keepalive.opcode)) {
case RegisterXferMemory:
ctx->tx_addr = (char *)be64toh(cmd->memory.addr);
ctx->tx_addr = (char *)(uintptr_t)be64toh(cmd->memory.addr);
ctx->tx_length = ntohl(cmd->memory.length);
ctx->tx_key = ntohl(cmd->memory.key);
ctx->tx_offset = 0;
@@ -501,11 +657,10 @@ static int valkeyRdmaPollCqCm(valkeyContext *c, long timed) {
return VALKEY_OK;
}
static ssize_t valkeyRdmaRead(valkeyContext *c, char *buf, size_t bufcap) {
static ssize_t valkeyRdmaReadZC(valkeyContext *c, char **buf) {
RdmaContext *ctx = c->privctx;
struct rdma_cm_id *cm_id = ctx->cm_id;
long timed, end;
uint32_t toread, remained;
uint32_t remained;
if (valkeyCommandTimeoutMsec(c, &timed)) {
return VALKEY_ERR;
@@ -521,16 +676,9 @@ pollcq:
if (ctx->recv_offset < ctx->rx_offset) {
remained = ctx->rx_offset - ctx->recv_offset;
toread = valkeyMin(remained, bufcap);
memcpy(buf, ctx->recv_buf + ctx->recv_offset, toread);
ctx->recv_offset += toread;
if (ctx->recv_offset == ctx->recv_length) {
connRdmaRegisterRx(c, cm_id);
}
return toread;
*buf = ctx->recv_buf + ctx->recv_offset;
ctx->recv_offset = ctx->rx_offset;
return remained;
}
if (valkeyRdmaPollCqCm(c, end) == VALKEY_OK) {
@@ -540,6 +688,16 @@ pollcq:
}
}
static ssize_t valkeyRdmaReadZCDone(valkeyContext *c) {
RdmaContext *ctx = c->privctx;
struct rdma_cm_id *cm_id = ctx->cm_id;
if (ctx->recv_offset == ctx->recv_length) {
return connRdmaRegisterRx(c, cm_id);
}
return VALKEY_OK;
}
static size_t connRdmaSend(RdmaContext *ctx, struct rdma_cm_id *cm_id, const void *data, size_t data_len) {
struct ibv_send_wr send_wr, *bad_wr;
struct ibv_sge sge;
@@ -990,11 +1148,17 @@ static valkeyContextFuncs valkeyContextRdmaFuncs = {
.free_privctx = valkeyRdmaFree,
.async_read = valkeyRdmaAsyncRead,
.async_write = valkeyRdmaAsyncWrite,
.read = valkeyRdmaRead,
.read_zc = valkeyRdmaReadZC,
.read_zc_done = valkeyRdmaReadZCDone,
.write = valkeyRdmaWrite,
.set_timeout = valkeyRdmaSetTimeout};
int valkeyInitiateRdma(void) {
#ifdef DLOPEN_RDMA
if (rdma_dyn_load_libs() != 0) {
return VALKEY_ERR;
}
#endif
valkeyContextRegisterFuncs(&valkeyContextRdmaFuncs, VALKEY_CONN_RDMA);
return VALKEY_OK;
deps/libvalkey/src/sds.c
Vendored
+21 -14
View File
@@ -453,7 +453,7 @@ static uint32_t digits10(uint64_t v) {
return 2;
if (v < 1000)
return 3;
if (v < 1000000000000UL) {
if (v < 1000000000000ULL) {
if (v < 100000000UL) {
if (v < 1000000) {
if (v < 10000)
@@ -462,12 +462,12 @@ static uint32_t digits10(uint64_t v) {
}
return 7 + (v >= 10000000UL);
}
if (v < 10000000000UL) {
if (v < 10000000000ULL) {
return 9 + (v >= 1000000000UL);
}
return 11 + (v >= 100000000000UL);
return 11 + (v >= 100000000000ULL);
}
return 12 + digits10(v / 1000000000000UL);
return 12 + digits10(v / 1000000000000ULL);
}
/* Convert a unsigned long long into a string. Returns the number of
@@ -487,13 +487,16 @@ static int ull2string(char *dst, size_t dstlen, unsigned long long value) {
/* Check length. */
uint32_t length = digits10(value);
if (length >= dstlen)
if (length == 0 || length >= dstlen)
goto err;
/* Null term. */
uint32_t next = length - 1;
int next = length - 1;
dst[next + 1] = '\0';
while (value >= 100) {
while (next >= 1) {
/* value should not be 0 while next >= 1 */
if (value == 0)
goto err;
int const i = (value % 100) * 2;
value /= 100;
dst[next] = digits[i + 1];
@@ -501,14 +504,18 @@ static int ull2string(char *dst, size_t dstlen, unsigned long long value) {
next -= 2;
}
/* Handle last 1-2 digits. */
if (value < 10) {
/* still have digits to process, but out of room */
if (value > 0 && next < 0)
goto err;
/* even number of digits should have been processed in the loop */
if (value >= 10)
goto err;
/* if next is 0, process last digit */
if (next == 0)
dst[next] = '0' + (uint32_t)value;
} else {
int i = (uint32_t)value * 2;
dst[next] = digits[i + 1];
dst[next - 1] = digits[i];
}
return length;
err:
/* force add Null termination */
deps/libvalkey/src/valkey.c
Vendored
+12
View File
@@ -1009,6 +1009,18 @@ int valkeyBufferRead(valkeyContext *c) {
if (c->err)
return VALKEY_ERR;
if (c->funcs->read_zc) {
char *zc_buf;
nread = c->funcs->read_zc(c, &zc_buf);
if (nread < 0) {
return VALKEY_ERR;
}
if (nread > 0 && valkeyReaderFeed(c->reader, zc_buf, nread) != VALKEY_OK) {
valkeySetError(c, c->reader->err, c->reader->errstr);
return VALKEY_ERR;
}
return c->funcs->read_zc_done(c);
}
nread = c->funcs->read(c, buf, sizeof(buf));
if (nread < 0) {
return VALKEY_ERR;
deps/libvalkey/src/valkey_private.h
Vendored
+2 -1
View File
@@ -124,7 +124,8 @@ static inline int valkeyContextUpdateCommandTimeout(valkeyContext *c,
return VALKEY_OK;
}
int valkeyContextRegisterFuncs(valkeyContextFuncs *funcs, enum valkeyConnectionType type);
/* Visible although private since required by libvalkey_rdma.so */
LIBVALKEY_API int valkeyContextRegisterFuncs(valkeyContextFuncs *funcs, enum valkeyConnectionType type);
void valkeyContextRegisterTcpFuncs(void);
void valkeyContextRegisterUnixFuncs(void);
void valkeyContextRegisterUserfdFuncs(void);
deps/libvalkey/src/win32.h
Vendored
-4
View File
@@ -46,10 +46,6 @@
#define strncasecmp _strnicmp
#endif
#ifndef alloca
#define alloca _alloca
#endif
#ifndef va_copy
#define va_copy(d, s) ((d) = (s))
#endif
deps/libvalkey/tests/CMakeLists.txt
Vendored
+7 -18
View File
@@ -63,39 +63,28 @@ else()
add_compile_options("-UNDEBUG")
endif()
# Make sure ctest gives the output when tests fail
list(APPEND CMAKE_CTEST_ARGUMENTS "--output-on-failure")
# Add non-cluster tests
# Add the non-cluster test binary.
# Some tests are unit tests which require us to include private headers.
add_executable(client_test client_test.c)
target_include_directories(client_test PRIVATE "${PROJECT_SOURCE_DIR}/src")
target_link_libraries(client_test valkey_unittest)
if(TLS_LIBRARY)
add_test(NAME client_test COMMAND "${CMAKE_CURRENT_SOURCE_DIR}/test.sh")
if(ENABLE_TLS)
target_compile_definitions(client_test PUBLIC VALKEY_TEST_TLS=1)
target_link_libraries(client_test ${TLS_LIBRARY})
set_property(TEST client_test APPEND PROPERTY ENVIRONMENT "TEST_TLS=1")
endif()
if(ENABLE_RDMA)
target_compile_definitions(client_test PUBLIC VALKEY_TEST_RDMA=1)
target_link_libraries(client_test valkey_rdma)
set_property(TEST client_test APPEND PROPERTY ENVIRONMENT "TEST_RDMA=1")
endif()
if(LIBEVENT_LIBRARY)
target_compile_definitions(client_test PUBLIC VALKEY_TEST_ASYNC=1)
target_link_libraries(client_test ${LIBEVENT_LIBRARY})
set_property(TEST client_test APPEND PROPERTY ENVIRONMENT "TEST_CLUSTER=1")
endif()
add_test(NAME client_test COMMAND "${CMAKE_CURRENT_SOURCE_DIR}/test.sh")
if(TEST_WITH_REDIS_VERSION)
set_property(TEST client_test APPEND PROPERTY ENVIRONMENT "VALKEY_SERVER=redis-server")
endif()
if(TLS_LIBRARY)
set_property(TEST client_test APPEND PROPERTY ENVIRONMENT "TEST_TLS=1")
endif()
if(ENABLE_RDMA)
set_property(TEST client_test APPEND PROPERTY ENVIRONMENT "TEST_RDMA=1")
endif()
if(LIBEVENT_LIBRARY)
set_property(TEST client_test APPEND PROPERTY ENVIRONMENT "TEST_CLUSTER=1")
endif()
# Unit tests
add_executable(ut_parse_cmd ut_parse_cmd.c test_utils.c)
deps/libvalkey/tests/client_test.c
Vendored
+48
View File
@@ -1884,6 +1884,53 @@ void null_cb(valkeyAsyncContext *ac, void *r, void *privdata) {
state->checkpoint++;
}
/* Test the command parsing, required for pub/sub in the async API. */
void test_async_command_parsing(struct config config) {
test("Async command parsing: ");
valkeyOptions options = get_server_tcp_options(config);
valkeyAsyncContext *ac = valkeyAsyncConnectWithOptions(&options);
assert(ac);
/* Null ptr. */
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, NULL, 45));
/* Empty command. */
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, "", 0));
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, " $", 2));
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*0\r\n", 4));
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*1\r\n$-1\r\n", 9));
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*1\r\n$-1\r\nUNSUBSCRIBE\r\n", 22));
/* Protocol error: erroneous bulkstring length and data. */
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*1\r\n$100000", 11));
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*1\r\n$100000\r", 12));
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*1\r\n$100000\r\n", 13));
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*1\r\n$10HELP\r\n\r\n", 15));
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*1\r\n$100000\r\nTO-SHORT\r\n", 23));
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*1\r\n$1\r\nTO-LONG\r\n", 17));
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*1\r\n$123456789\r\n", 11));
/* Faulty length given to function. */
for (int i = 0; i < 19; i++) {
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*2\r\n$9\r\nSUBSCRIBE\r\n$7\r\nCHANNEL\r\n", i));
}
for (int i = 0; i < 21; i++) {
assert(VALKEY_ERR == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*1\r\n$11\r\nUNSUBSCRIBE\r\n", i));
}
/* Complete command. */
assert(VALKEY_OK == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*2\r\n$9\r\nSUBSCRIBE\r\n$7\r\nCHANNEL\r\n", 32));
assert(VALKEY_OK == valkeyAsyncFormattedCommand(ac, NULL, NULL, "*1\r\n$11\r\nUNSUBSCRIBE\r\n", 22));
// Heap allocate command without NULL terminator.
const char ping[] = "*1\r\n$4\r\nPING\r\n";
size_t len = sizeof(ping) - 1;
char *buf = vk_malloc_safe(len);
memcpy(buf, ping, len);
assert(VALKEY_OK == valkeyAsyncFormattedCommand(ac, NULL, NULL, buf, len));
free(buf);
valkeyAsyncFree(ac);
}
static void test_pubsub_handling(struct config config) {
test("Subscribe, handle published message and unsubscribe: ");
/* Setup event dispatcher with a testcase timeout */
@@ -2817,6 +2864,7 @@ int main(int argc, char **argv) {
get_server_version(c, &major, NULL);
disconnect(c, 0);
test_async_command_parsing(cfg);
test_pubsub_handling(cfg);
test_pubsub_multiple_channels(cfg);
test_monitor(cfg);
deps/libvalkey/tests/clusterclient.c
Vendored
+26 -2
View File
@@ -61,18 +61,36 @@ void eventCallback(const valkeyClusterContext *cc, int event, void *privdata) {
printf("Event: %s\n", e);
}
void connectCallback(const valkeyContext *c, int status) {
const char *s = "";
if (status != VALKEY_OK)
s = "failed to ";
printf("Event: %sconnect to %s:%d\n", s, c->tcp.host, c->tcp.port);
}
int main(int argc, char **argv) {
int show_events = 0;
int use_cluster_nodes = 0;
int send_to_all = 0;
int show_connection_events = 0;
int select_db = 0;
int argindex;
for (argindex = 1; argindex < argc && argv[argindex][0] == '-';
argindex++) {
if (strcmp(argv[argindex], "--events") == 0) {
show_events = 1;
} else if (strcmp(argv[argindex], "--connection-events") == 0) {
show_connection_events = 1;
} else if (strcmp(argv[argindex], "--use-cluster-nodes") == 0) {
use_cluster_nodes = 1;
} else if (strcmp(argv[argindex], "--select-db") == 0) {
if (++argindex < argc) /* Need an additional argument */
select_db = atoi(argv[argindex]);
if (select_db == 0) {
fprintf(stderr, "Missing or faulty argument for --select-db\n");
exit(1);
}
} else {
fprintf(stderr, "Unknown argument: '%s'\n", argv[argindex]);
exit(1);
@@ -80,8 +98,8 @@ int main(int argc, char **argv) {
}
if (argindex >= argc) {
fprintf(stderr, "Usage: clusterclient [--events] [--use-cluster-nodes] "
"HOST:PORT\n");
fprintf(stderr, "Usage: clusterclient [--events] [--connection-events] "
"[--use-cluster-nodes] [--select-db NUM] HOST:PORT\n");
exit(1);
}
const char *initnode = argv[argindex];
@@ -97,6 +115,12 @@ int main(int argc, char **argv) {
if (show_events) {
options.event_callback = eventCallback;
}
if (show_connection_events) {
options.connect_callback = connectCallback;
}
if (select_db > 0) {
options.select_db = select_db;
}
valkeyClusterContext *cc = valkeyClusterConnectWithOptions(&options);
if (cc == NULL || cc->err) {
deps/libvalkey/tests/clusterclient_async.c
Vendored
+13 -1
View File
@@ -242,6 +242,7 @@ void disconnectCallback(const valkeyAsyncContext *ac, int status) {
int main(int argc, char **argv) {
int use_cluster_nodes = 0;
int show_connection_events = 0;
int select_db = 0;
int optind;
for (optind = 1; optind < argc && argv[optind][0] == '-'; optind++) {
@@ -253,6 +254,13 @@ int main(int argc, char **argv) {
show_connection_events = 1;
} else if (strcmp(argv[optind], "--blocking-initial-update") == 0) {
blocking_initial_update = 1;
} else if (strcmp(argv[optind], "--select-db") == 0) {
if (++optind < argc) /* Need an additional argument */
select_db = atoi(argv[optind]);
if (select_db == 0) {
fprintf(stderr, "Missing or faulty argument for --select-db\n");
exit(1);
}
} else {
fprintf(stderr, "Unknown argument: '%s'\n", argv[optind]);
}
@@ -260,7 +268,8 @@ int main(int argc, char **argv) {
if (optind >= argc) {
fprintf(stderr,
"Usage: clusterclient_async [--use-cluster-nodes] HOST:PORT\n");
"Usage: clusterclient_async [--events] [--connection-events] "
"[--use-cluster-nodes] [--select-db NUM] HOST:PORT\n");
exit(1);
}
const char *initnode = argv[optind];
@@ -283,6 +292,9 @@ int main(int argc, char **argv) {
options.async_connect_callback = connectCallback;
options.async_disconnect_callback = disconnectCallback;
}
if (select_db > 0) {
options.select_db = select_db;
}
valkeyClusterOptionsUseLibevent(&options, base);
valkeyClusterAsyncContext *acc = valkeyClusterAsyncConnectWithOptions(&options);
deps/libvalkey/tests/scripts/client-disconnect-test.sh
Vendored
+5 -1
View File
@@ -16,11 +16,15 @@ syncpid1=$!;
# Start simulated valkey node
timeout 5s ./simulated-valkey.pl -p 7401 -d --sigcont $syncpid1 <<'EOF' &
EXPECT CONNECT
EXPECT ["SELECT", "2"]
SEND +OK
EXPECT ["CLUSTER", "SLOTS"]
SEND [[0, 16383, ["127.0.0.1", 7401, "nodeid1"]]]
EXPECT CLOSE
EXPECT CONNECT
EXPECT ["SELECT", "2"]
SEND +OK
EXPECT ["SET", "foo", "initial"]
SEND +OK
@@ -35,7 +39,7 @@ server1=$!
wait $syncpid1;
# Run client
timeout 4s "$clientprog" --blocking-initial-update --connection-events 127.0.0.1:7401 > "$testname.out" <<'EOF'
timeout 4s "$clientprog" --blocking-initial-update --connection-events --select-db 2 127.0.0.1:7401 > "$testname.out" <<'EOF'
SET foo initial
# Send a command that is expected to be redirected just before
deps/libvalkey/tests/scripts/slots-not-served-test.sh
Vendored
+5 -2
View File
@@ -13,6 +13,8 @@ syncpid1=$!;
timeout 5s ./simulated-valkey.pl -p 7401 -d --sigcont $syncpid1 <<'EOF' &
# The initial slotmap is not covering all slots.
EXPECT CONNECT
EXPECT ["SELECT", "5"]
SEND +OK
EXPECT ["CLUSTER", "SLOTS"]
SEND [[0, 1, ["127.0.0.1", 7401, "nodeid7401"]]]
@@ -36,7 +38,7 @@ server1=$!
wait $syncpid1;
# Run client
timeout 3s "$clientprog" --events 127.0.0.1:7401 > "$testname.out" <<'EOF'
timeout 3s "$clientprog" --events --connection-events --select-db 5 127.0.0.1:7401 > "$testname.out" <<'EOF'
GET foo1
GET foo2
EOF
@@ -56,7 +58,8 @@ if [ $clientexit -ne 0 ]; then
fi
# Check the output from clusterclient
expected="Event: slotmap-updated
expected="Event: connect to 127.0.0.1:7401
Event: slotmap-updated
Event: ready
Event: slotmap-updated
error: slot not served by any node
deps/libvalkey/tests/test_utils.h
Vendored
+2 -2
View File
@@ -29,10 +29,10 @@
}
#define CHECK_REPLY_OK(_ctx, _reply) \
{ CHECK_REPLY_STATUS(_ctx, _reply, "OK") }
{CHECK_REPLY_STATUS(_ctx, _reply, "OK")}
#define CHECK_REPLY_QUEUED(_ctx, _reply) \
{ CHECK_REPLY_STATUS(_ctx, _reply, "QUEUED") }
{CHECK_REPLY_STATUS(_ctx, _reply, "QUEUED")}
#define CHECK_REPLY_INT(_ctx, _reply, _value) \
{ \
deps/libvalkey/tests/ut_parse_cmd.c
Vendored
+72
View File
@@ -47,6 +47,48 @@ void test_valkey_parse_error_nonresp(void) {
command_destroy(c);
}
/* Create a 65 bytes command string while limit is 64 bytes */
void test_valkey_parse_too_long_cmd(void) {
char str[66];
memset(str, 'A', 65);
str[65] = '\0';
struct cmd *c = command_get();
int len = valkeyFormatCommand(&c->cmd, str);
ASSERT_MSG(len >= 0, "Format command error");
c->clen = len;
valkey_parse_cmd(c);
ASSERT_MSG(c->result == CMD_PARSE_ERROR, "Unexpected parse success");
ASSERT_MSG(!strncmp(c->errstr, "Unknown command AAAA", 20), c->errstr);
command_destroy(c);
}
/* Parse a subcommand longer than the 64 char limit */
void test_valkey_parse_too_long_subcommand(void) {
char subcmd[66];
memset(subcmd, 'A', 65);
subcmd[65] = '\0';
struct cmd *c = command_get();
int len = valkeyFormatCommand(&c->cmd, "XGROUP %s", subcmd);
ASSERT_MSG(len >= 0, "Format command error");
c->clen = len;
valkey_parse_cmd(c);
ASSERT_MSG(c->result == CMD_PARSE_ERROR, "Unexpected parse success");
ASSERT_MSG(!strncmp(c->errstr, "Unknown command XGROUP AAAAAAA", 30), c->errstr);
command_destroy(c);
}
void test_valkey_parse_unknown_cmd(void) {
struct cmd *c = command_get();
int len = valkeyFormatCommand(&c->cmd, "OIOIOI");
ASSERT_MSG(len >= 0, "Format command error");
c->clen = len;
valkey_parse_cmd(c);
ASSERT_MSG(c->result == CMD_PARSE_ERROR, "Unexpected parse success");
ASSERT_MSG(!strcmp(c->errstr, "Unknown command OIOIOI"), c->errstr);
command_destroy(c);
}
void test_valkey_parse_cmd_get(void) {
struct cmd *c = command_get();
int len = valkeyFormatCommand(&c->cmd, "GET foo");
@@ -102,6 +144,17 @@ void test_valkey_parse_cmd_xgroup_no_subcommand(void) {
command_destroy(c);
}
void test_valkey_parse_cmd_xgroup_unknown_subcommand(void) {
struct cmd *c = command_get();
int len = valkeyFormatCommand(&c->cmd, "XGROUP OIOIOI");
ASSERT_MSG(len >= 0, "Format command error");
c->clen = len;
valkey_parse_cmd(c);
ASSERT_MSG(c->result == CMD_PARSE_ERROR, "Unexpected parse success");
ASSERT_MSG(!strcmp(c->errstr, "Unknown command XGROUP OIOIOI"), c->errstr);
command_destroy(c);
}
void test_valkey_parse_cmd_xgroup_destroy_no_key(void) {
struct cmd *c = command_get();
int len = valkeyFormatCommand(&c->cmd, "xgroup destroy");
@@ -186,13 +239,31 @@ void test_valkey_parse_cmd_georadius_ro_ok(void) {
command_destroy(c);
}
void test_valkey_parse_cmd_sadd_ok(void) {
char key[201];
memset(key, 'A', 200);
key[200] = '\0';
struct cmd *c = command_get();
int len = valkeyFormatCommand(&c->cmd, "SADD %s value", key);
ASSERT_MSG(len >= 0, "Format command error");
c->clen = len;
valkey_parse_cmd(c);
ASSERT_KEY(c, key);
command_destroy(c);
}
int main(void) {
test_valkey_parse_error_nonresp();
test_valkey_parse_too_long_cmd();
test_valkey_parse_too_long_subcommand();
test_valkey_parse_unknown_cmd();
test_valkey_parse_cmd_get();
test_valkey_parse_cmd_mset();
test_valkey_parse_cmd_eval_1();
test_valkey_parse_cmd_eval_0();
test_valkey_parse_cmd_xgroup_no_subcommand();
test_valkey_parse_cmd_xgroup_unknown_subcommand();
test_valkey_parse_cmd_xgroup_destroy_no_key();
test_valkey_parse_cmd_xgroup_destroy_ok();
test_valkey_parse_cmd_xreadgroup_ok();
@@ -200,5 +271,6 @@ int main(void) {
test_valkey_parse_cmd_restore_ok();
test_valkey_parse_cmd_restore_asking_ok();
test_valkey_parse_cmd_georadius_ro_ok();
test_valkey_parse_cmd_sadd_ok();
return 0;
}
deps/libvalkey/tests/ut_slotmap_update.c
Vendored
+47
View File
@@ -37,6 +37,15 @@ valkeyReply *create_cluster_nodes_reply(const char *str) {
return create_reply(buf, len);
}
/* Helper to create a valkeyReply that contains a verbatim string. */
valkeyReply *create_cluster_nodes_reply_resp3(const char *str) {
char buf[1024];
const char *encoding = "txt:";
int len = sprintf(buf, "=%zu\r\n%s%s\r\n", strlen(encoding) + strlen(str), encoding, str);
return create_reply(buf, len);
}
/* Helper to create a cluster slots response.
* Parses the string using a rudimentary JSON like format which accepts:
* - arrays example: [elem1, elem2]
@@ -526,6 +535,43 @@ void test_parse_cluster_nodes_with_legacy_format(void) {
valkeyClusterFree(cc);
}
/* Parse a cluster nodes reply when RESP3 is used,
* i.e. the reply is a verbatim string. */
void test_parse_cluster_nodes_with_resp3(void) {
valkeyClusterOptions options = {0};
valkeyClusterContext *cc = createClusterContext(&options);
valkeyContext *c = valkeyContextInit();
valkeyClusterNode *node;
dictIterator di;
valkeyReply *reply = create_cluster_nodes_reply_resp3(
"67ed2db8d677e59ec4a4cefb06858cf2a1a89fa1 127.0.0.1:30002@31002,hostname2 master - 0 1426238316232 2 connected 5461-10922\n"
"292f8b365bb7edb5e285caf0b7e6ddc7265d2f4f 127.0.0.1:30003@31003,hostname3 master - 0 1426238318243 3 connected 10923-16383\n"
"e7d1eecce10fd6bb5eb35b9f99a514335d9ba9ca 127.0.0.1:30001@31001,hostname1 myself,master - 0 0 1 connected 0-5460\n");
dict *nodes = parse_cluster_nodes(cc, c, reply);
freeReplyObject(reply);
assert(nodes);
assert(dictSize(nodes) == 3); /* 3 primaries */
dictInitIterator(&di, nodes);
/* Verify node 1 */
node = dictGetVal(dictNext(&di));
assert(strcmp(node->name, "e7d1eecce10fd6bb5eb35b9f99a514335d9ba9ca") == 0);
assert(strcmp(node->addr, "127.0.0.1:30001") == 0);
/* Verify node 2 */
node = dictGetVal(dictNext(&di));
assert(strcmp(node->name, "67ed2db8d677e59ec4a4cefb06858cf2a1a89fa1") == 0);
assert(strcmp(node->addr, "127.0.0.1:30002") == 0);
/* Verify node 3 */
node = dictGetVal(dictNext(&di));
assert(strcmp(node->name, "292f8b365bb7edb5e285caf0b7e6ddc7265d2f4f") == 0);
assert(strcmp(node->addr, "127.0.0.1:30003") == 0);
dictRelease(nodes);
valkeyFree(c);
valkeyClusterFree(cc);
}
/* Parse a cluster slots reply from a basic deployment. */
void test_parse_cluster_slots(bool parse_replicas) {
valkeyClusterOptions options = {0};
@@ -810,6 +856,7 @@ int main(void) {
test_parse_cluster_nodes_with_multiple_replicas();
test_parse_cluster_nodes_with_parse_error();
test_parse_cluster_nodes_with_legacy_format();
test_parse_cluster_nodes_with_resp3();
test_parse_cluster_slots(false /* replicas not parsed */);
test_parse_cluster_slots(true /* replicas parsed */);
deps/linenoise/Makefile
Vendored
+1 -1
View File
@@ -1,5 +1,5 @@
STD=
WARN= -Wall
WARN= -Wall -Werror
OPT= -Os
R_CFLAGS= $(STD) $(WARN) $(OPT) $(DEBUG) $(CFLAGS)
deps/linenoise/linenoise.c
Vendored
+5 -3
View File
@@ -839,6 +839,9 @@ void linenoiseEditDeletePrevWord(struct linenoiseState *l) {
* when ctrl+d is typed.
*
* The function returns the length of the current buffer. */
/* Max length for CSI escape sequence parameter buffer */
#define SEQ_BUFFER_MAX_LENGTH 8
static int linenoiseEdit(int stdin_fd, int stdout_fd, char *buf, size_t buflen, const char *prompt)
{
struct linenoiseState l;
@@ -964,14 +967,13 @@ static int linenoiseEdit(int stdin_fd, int stdout_fd, char *buf, size_t buflen,
if (seq[1] >= '0' && seq[1] <= '9') {
/* Extended escape, read additional bytes.
* Examples: ESC [1;5C ESC [3~ */
const int seqBufferMaxLength = 8;
char seqBuffer[seqBufferMaxLength];
char seqBuffer[SEQ_BUFFER_MAX_LENGTH];
int i = 0;
seqBuffer[i++] = seq[1];
/* If first param is digit or ';', read more until we see a final in @~ */
char additionalChar;
while (i < seqBufferMaxLength-1 && read(l.ifd, &additionalChar, 1) != -1) {
while (i < SEQ_BUFFER_MAX_LENGTH-1 && read(l.ifd, &additionalChar, 1) != -1) {
seqBuffer[i++] = additionalChar;
if (additionalChar >= '@' && additionalChar <= '~') { /* CSI final byte */
seqBuffer[i] = '\0';
deps/lua/src/Makefile
Vendored
+22 -4
View File
@@ -18,10 +18,28 @@ MYCFLAGS=
MYLDFLAGS=
MYLIBS=
# Pretty-printing setup
CC_COLOR="\033[32m"
LNK_COLOR="\033[36;1m"
SCOPE_COLOR="\033[38:5:250m"
SRC_COLOR="\033[33m"
BIN_COLOR="\033[35m"
RESET="\033[0m"
ifndef V
QUIET_CC = @printf ' %b %b(LUA Lib)%b %b\n' $(CC_COLOR)CC$(RESET) $(SCOPE_COLOR) $(RESET) $(SRC_COLOR)$<$(RESET) 1>&2;
QUIET_LINK = @printf ' %b %b\n' $(LNK_COLOR)LINK$(RESET) $(BIN_COLOR)$@$(RESET) 1>&2;
QUIET_AR = @printf ' %b %b\n' $(CC_COLOR)ARCHIVE$(RESET) $(BIN_COLOR)$@$(RESET) 1>&2;
endif
# == END OF USER SETTINGS. NO NEED TO CHANGE ANYTHING BELOW THIS LINE =========
PLATS= aix ansi bsd freebsd generic linux macosx mingw posix solaris
# Explicit pattern rule for building object files from C sources
%.o: %.c
$(QUIET_CC)$(CC) $(CFLAGS) -c $< -o $@
LUA_A= liblua.a
CORE_O= lapi.o lcode.o ldebug.o ldo.o ldump.o lfunc.o lgc.o llex.o lmem.o \
lobject.o lopcodes.o lparser.o lstate.o lstring.o ltable.o ltm.o \
@@ -49,14 +67,14 @@ o: $(ALL_O)
a: $(ALL_A)
$(LUA_A): $(CORE_O) $(LIB_O)
$(AR) $@ $(CORE_O) $(LIB_O) # DLL needs all object files
$(RANLIB) $@
$(QUIET_AR)$(AR) $@ $(CORE_O) $(LIB_O) # DLL needs all object files
$(QUIET_AR)$(RANLIB) $@
$(LUA_T): $(LUA_O) $(LUA_A)
$(CC) -o $@ $(MYLDFLAGS) $(LUA_O) $(LUA_A) $(LIBS)
$(QUIET_LINK)$(CC) -o $@ $(MYLDFLAGS) $(LUA_O) $(LUA_A) $(LIBS)
$(LUAC_T): $(LUAC_O) $(LUA_A)
$(CC) -o $@ $(MYLDFLAGS) $(LUAC_O) $(LUA_A) $(LIBS)
$(QUIET_LINK)$(CC) -o $@ $(MYLDFLAGS) $(LUAC_O) $(LUA_A) $(LIBS)
clean:
$(RM) $(ALL_T) $(ALL_O)
design-docs/README.md
+67
View File
@@ -0,0 +1,67 @@
# Valkey Design Documents
This folder collects designs for Valkey. These designs detail features and
changes that require more detail than just the text in a pull request or an
issue.
A markdown file describes each feature or larger topic.
## Workflow
**IMPORTANT: Before writing a design, start an issue for some early alignment.**
This is the first step to find interested parties and collect initial
requirements. This issue serves as the overall tracking for the feature.
1. After finishing initial discussion on the issue, determine the necessity of a
design document. Small features or changes do not require designs.
Wide-reaching changes or those requiring major alignment make good candidates
for a design.
2. To create a design document, create a new markdown file in this `design-docs`
directory.
3. The maintainers review and approve the design document once authors address
feedback. Submitting a design document does not necessarily bind to a certain
design.
4. The design document serves as living documentation. As developers build the
feature, the design document captures key design aspects. Commit changes to
the design document alongside the code changes as developers implement the
mid-level design details.
## What's useful to include?
**Design documents do not follow a strict format.** There is no template.
When writing a design, consider that many people, including those unfamiliar
with the feature, read it. It should be self-contained and easy to understand.
The following sections are optional but provide a high-level overview of
potential inclusions:
- **High-level overview**: A brief summary of the feature and its purpose.
- **Key design elements**: The following are generally useful to include:
- State machines
- Data structures
- Algorithms
- Interaction with other Valkey components (replication, persistence, cluster,
modules, etc.)
- **Links to key issues/PRs**: Link to relevant issues/PRs for further reading.
- **Links to relevant code**: Link to relevant code files for further reading.
## What not to include?
Overdocumentation often leads to stale design documents. A design document is
not the best place for the following:
- **API Details**: API details belong in the
[public Valkey documentation](https://valkey.io/commands/)
- **Low-level Implementation Details**: Document difficult or complex
implementation details in code comments, not design documents.
- **Edge Cases**: Prefer to link to test cases or code locations that cover edge
cases, rather than documenting them in the design document.
- **Alternatives/Rejected Designs**: Document decision making in the issue or PR
where the contributors made the decision.
- **Overly Verbose Explanations**: Aim to use Mermaid or ASCII diagrams to
explain complex concepts rather than prose.
- **Boilerplate**: Keep every document minimal and to the point. Avoid
unnecessary sections.
- **Future work**: File issues for future work items to track them, rather than
including them in the design document.
design-docs/atomic-slot-migration.md
+184
View File
@@ -0,0 +1,184 @@
# Design Document: Atomic Slot Migration
## 1. Overview
Atomic Slot Migration (ASM) provides a seamless, atomic method for migrating
hash slots between nodes in a Valkey cluster. This mechanism replaces
`CLUSTER SETSLOT IMPORTING/MIGRATING` and `MIGRATE` for migrating slots between
nodes.
## 2. Core Mechanics
Rather than migrating data key-by-key, ASM operates at the slot level by
adapting existing replication and failover primitives:
- **Slot-Based Replication:** Physical data migration borrows from
Primary-Replica replication mechanisms, but strictly scopes to the specific
slots being moved.
- **Atomic Ownership Transfer:** ASM executes the final handover of slot
ownership using a coordinated process similar to a Manual Failover, ensuring
an atomic transfer.
- **Traffic Handling:** Throughout the migration process, the source node
retains the data and continues to actively serve business requests. The system
cleanly cuts over traffic to the target node only after completing the atomic
transfer.
## 3. Implementation Details
### 3.1 High Level Overview
1. **Snapshot Transfer:** The source node transfers data to the target node. The
source node forks a child process to iterate and serialize the slot's keys.
The source transfers data in "AOF" (Append Only File) format to the target
node. This format consists of a stream of commands. Consequently, the target
primary and target replicas replay these commands to restore the slot's
state.
2. **Incremental Updates:** While transferring the initial snapshot, the source
node serves business requests. The source node records any changes to the
slot's keys during this time and sends them to the target node as incremental
updates after completing the snapshot transfer.
3. **Pause:** After sending the incremental updates, the source node pauses
writes to the migrating slots. Consequently, the source node rejects any
further business requests for those slots. This pause ensures the target node
maintains the same state as the source node.
4. **Failover:** After the source node pauses, the target node performs a
takeover and becomes the primary node for the slot.
5. **Clean Up:** After the target node becomes the primary node for the slot,
the source node receives this information via cluster topology updates. The
source node then unpauses and completes the slot migration. Failed migrations
on the target side are cleaned up by deleting keys that are no longer owned
by the node.
### 3.2 CLUSTER SYNCSLOTS
The source, target, and target replica use the `CLUSTER SYNCSLOTS` command to
coordinate the handover state:
```
Source Target Target Replica
| | |
|------------ SYNCSLOTS ESTABLISH -------------->| |
| |----- SYNCSLOTS ESTABLISH ------>|
|<-------------------- +OK ----------------------| |
| | |
|---------------- SYNCSLOTS ACK ---------------->| |
| | |
|~~~~~~~~~~~~~~ snapshot as AOF ~~~~~~~~~~~~~~~~>| |
| |~~~~~~ forward snapshot ~~~~~~~~>|
|----------- SYNCSLOTS SNAPSHOT-EOF ------------>| |
| | |
|<----------- SYNCSLOTS REQUEST-PAUSE -----------| |
| | |
|~~~~~~~~~~~~ incremental changes ~~~~~~~~~~~~~~>| |
| |~~~~~~ forward changes ~~~~~~~~~>|
|--------------- SYNCSLOTS PAUSED -------------->| |
| | |
|<---------- SYNCSLOTS REQUEST-FAILOVER ---------| |
| | |
|---------- SYNCSLOTS FAILOVER-GRANTED --------->| |
| | |
| (performs takeover & |
| propagates topology) |
| | |
| |------- SYNCSLOTS FINISH ------->|
(finds out about topology | |
change & marks migration done) | |
| | |
```
Throughout the migration, both the source and target nodes exchange periodic
`SYNCSLOTS ACK` messages to monitor the health and progress of the operation.
If a node fails to receive an acknowledgment within the replication timeout,
the migration is aborted.
See code comments in [cluster_migrateslots.c](../src/cluster_migrateslots.c) for
detailed state machines.
### 3.3 Automatic Rollback
Various scenarios cause slot migration failure:
1. Link between source and target disconnects
2. Source or target node crash, halt, or encounter a partition
3. A failover occurs on the source or target node
4. Out of memory error occurs on the target node
5. Client output buffer on the source node grows too large
6. An administrator executes `FLUSHDB` on the source or target node
In such cases, ASM automatically rolls back the migration.
```
Source Target Target Replica
| | |
|------------ SYNCSLOTS ESTABLISH -------------->| |
| |----- SYNCSLOTS ESTABLISH ------>|
|<-------------------- +OK ----------------------| |
... ... ...
| | |
| <FAILURE> |
| | |
| (performs cleanup) |
| | ~~~~~~ UNLINK <key> ... ~~~~~~~>|
| | |
| | ------ SYNCSLOTS FINISH ------->|
| | |
```
#### 3.3.1 Cleanup
The cluster automatically cleans up failed or cancelled slot migrations. The
primary is solely responsible for cleaning up unowned slots. Primaries demoted
during migration do not clean up previously active slot imports. The promoted
replica is responsible for both cleaning up the slot and sending a
`SYNCSLOTS FINISH`.
### 3.4 Key Containment
The system rejects any keyed command executed on a node that is not the primary
for that slot with `-MOVED` (e.g. `GET`, `SET`, `DEL`, `INCR`, etc).
Nodes filter unkeyed read commands, like `SCAN` and `KEYS`, to avoid exposing
importing slot data. Each node in the target shard tracks the slot migration job
state and hides writes to that slot from the end user until the migration
completes.
#### 3.4.1 Full Sync, Partial Sync, RDB
To ensure that replicas resyncing during an import remain aware of it, Valkey
serializes each in-progress slot import into an RDB section defined by a new
opcode. The encoding includes the job name and the slot ranges being imported.
Whenever the system loads an RDB file containing a slot import section, whether
from disk or during a primary sync, it adds a new migration to track the import.
If the Valkey node becomes a primary after loading the RDB, it cancels the slot
migration.
Failure to load the opcode results in consistency problems, so the opcode is
mandatory. If the opcode is not recognized, the RDB load will fail.
Loading this tracking state on primaries ensures that replicas partially syncing
to a restarted primary still get their `SYNCSLOTS FINISH` message in the
replication stream.
#### 3.4.2 AOF
Valkey propagates the `ESTABLISH` and `FINISH` commands to the AOF, ensuring
they replay properly on AOF load. Similar to RDB, if any pending `ESTABLISH`
commands lack a subsequent `FINISH` upon becoming primary, the system fails them
after loading.
## 4. External References
- **API & User Commands:**
- [CLUSTER MIGRATESLOTS](https://valkey.io/commands/cluster-migrateslots/)
- [CLUSTER GETSLOTMIGRATIONS](https://valkey.io/commands/cluster-getslotmigrations/)
- [CLUSTER CANCELSLOTMIGRATIONS](https://valkey.io/commands/cluster-cancelslotmigrations/)
- [CLUSTER SYNCSLOTS](https://valkey.io/commands/cluster-syncslots/)
- **Corner Cases:** Read the test cases in
[cluster-migrateslots.tcl](../tests/unit/cluster/cluster-migrateslots.tcl)
- **PR References:** For further reading, refer to the following Pull Requests:
- [PR #1949](https://github.com/valkey-io/valkey/pull/1949)
- [PR #2755](https://github.com/valkey-io/valkey/pull/2755)
- [PR #2593](https://github.com/valkey-io/valkey/pull/2593)
- [PR #2635](https://github.com/valkey-io/valkey/pull/2635)
src/.gitignore
+2
View File
@@ -3,3 +3,5 @@
*.gcov
valkey.info
lcov-html
generated_*
unit/wrapped/
src/CMakeLists.txt
+28 -16
View File
@@ -10,30 +10,41 @@ message(STATUS "CFLAGS: ${CMAKE_C_FLAGS}")
get_valkey_server_linker_option(VALKEY_SERVER_LDFLAGS)
list(APPEND SERVER_LIBS "fpconv")
list(APPEND SERVER_LIBS "hdr_histogram")
list(APPEND SERVER_LIBS "ffc")
valkey_build_and_install_bin(valkey-server "${VALKEY_SERVER_SRCS}" "${VALKEY_SERVER_LDFLAGS}" "${SERVER_LIBS}"
"redis-server")
add_dependencies(valkey-server generate_commands_def)
add_dependencies(valkey-server generate_fmtargs_h)
add_dependencies(valkey-server release_header)
if (BUILD_LUA)
if (NOT BUILD_LUA STREQUAL "no")
message(STATUS "Build Lua scripting engine module")
if (BUILD_LUA STREQUAL "static")
add_compile_definitions(STATIC_LUA=1)
message(STATUS "Building LUA as a STATIC module")
else ()
add_compile_definitions(STATIC_LUA=0)
message(STATUS "Building LUA as a DYNAMIC module")
endif ()
add_subdirectory(modules/lua)
add_dependencies(valkey-server valkeylua)
if (BUILD_LUA STREQUAL "static")
target_link_libraries(valkey-server $<LINK_LIBRARY:WHOLE_ARCHIVE,valkeylua>)
else ()
add_dependencies(valkey-server valkeylua)
endif ()
target_compile_definitions(valkey-server PRIVATE LUA_ENABLED)
if (UNIX AND NOT APPLE)
target_compile_definitions(valkey-server PRIVATE LUA_LIB=libvalkeylua.so)
target_link_options(valkey-server PRIVATE -Wl,--disable-new-dtags)
else ()
target_compile_definitions(valkey-server PRIVATE LUA_LIB=libvalkeylua.dylib)
endif ()
set(VALKEY_INSTALL_RPATH "")
set_target_properties(valkey-server PROPERTIES
INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR};${CMAKE_LIBRARY_OUTPUT_DIRECTORY}"
INSTALL_RPATH_USE_LINK_PATH TRUE
BUILD_WITH_INSTALL_RPATH TRUE
)
endif()
set_target_properties(
valkey-server
PROPERTIES INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR};${CMAKE_LIBRARY_OUTPUT_DIRECTORY}"
INSTALL_RPATH_USE_LINK_PATH TRUE
BUILD_WITH_INSTALL_RPATH TRUE)
endif ()
unset(BUILD_LUA CACHE)
if (VALKEY_RELEASE_BUILD)
@@ -48,9 +59,8 @@ if (DEBUG_FORCE_DEFRAG)
endif ()
if (BUILD_SANITIZER)
# 'BUILD_SANITIZER' is defined in ValkeySetup module (based on user input)
# If defined, the variables 'VALKEY_SANITAIZER_CFLAGS' and 'VALKEY_SANITAIZER_LDFLAGS'
# are set with the link & compile flags required
# 'BUILD_SANITIZER' is defined in ValkeySetup module (based on user input) If defined, the variables
# 'VALKEY_SANITAIZER_CFLAGS' and 'VALKEY_SANITAIZER_LDFLAGS' are set with the link & compile flags required
message(STATUS "Adding sanitizer flags for target valkey-server")
target_compile_options(valkey-server PRIVATE ${VALKEY_SANITAIZER_CFLAGS})
target_link_options(valkey-server PRIVATE ${VALKEY_SANITAIZER_LDFLAGS})
@@ -60,6 +70,7 @@ unset(BUILD_SANITIZER CACHE)
# Target: valkey-cli
list(APPEND CLI_LIBS "fpconv")
list(APPEND CLI_LIBS "linenoise")
list(APPEND CLI_LIBS "ffc")
valkey_build_and_install_bin(valkey-cli "${VALKEY_CLI_SRCS}" "${VALKEY_SERVER_LDFLAGS}" "${CLI_LIBS}" "redis-cli")
add_dependencies(valkey-cli generate_commands_def)
add_dependencies(valkey-cli generate_fmtargs_h)
@@ -67,6 +78,7 @@ add_dependencies(valkey-cli generate_fmtargs_h)
# Target: valkey-benchmark
list(APPEND BENCH_LIBS "fpconv")
list(APPEND BENCH_LIBS "hdr_histogram")
list(APPEND BENCH_LIBS "ffc")
valkey_build_and_install_bin(valkey-benchmark "${VALKEY_BENCHMARK_SRCS}" "${VALKEY_SERVER_LDFLAGS}" "${BENCH_LIBS}"
"redis-benchmark")
add_dependencies(valkey-benchmark generate_commands_def)
@@ -109,16 +121,16 @@ if (BUILD_EXAMPLE_MODULES)
add_subdirectory(modules)
endif ()
if (BUILD_UNIT_TESTS)
if (BUILD_UNIT_GTESTS)
add_subdirectory(unit)
endif ()
# Friendly hint like the Makefile one
file(RELATIVE_PATH _CMAKE_DIR_RELATIVE_PATH "${CMAKE_SOURCE_DIR}" "${CMAKE_BINARY_DIR}")
add_custom_target(hint ALL
add_custom_target(
hint ALL
DEPENDS valkey-server valkey-cli valkey-benchmark
COMMAND ${CMAKE_COMMAND} -E echo ""
COMMAND ${CMAKE_COMMAND} -E echo "Hint: It is a good idea to run tests with your CMake-built binaries \\;\\)"
COMMAND ${CMAKE_COMMAND} -E echo " ./${_CMAKE_DIR_RELATIVE_PATH}/runtest"
COMMAND ${CMAKE_COMMAND} -E echo ""
)
COMMAND ${CMAKE_COMMAND} -E echo "")
src/Makefile
+60 -49
View File
@@ -171,6 +171,7 @@ ifeq ($(uname_S),SunOS)
else
ifeq ($(uname_S),Darwin)
# Darwin
FINAL_LDFLAGS+= -Wl,-export_dynamic
FINAL_LIBS+= -ldl
# Homebrew's OpenSSL is not linked to /usr/local to avoid
# conflicts with the system's LibreSSL installation so it
@@ -209,18 +210,22 @@ ifeq ($(uname_S),NetBSD)
else
ifeq ($(uname_S),FreeBSD)
# FreeBSD
FINAL_LDFLAGS+= -rdynamic
FINAL_LIBS+= -lpthread -lexecinfo
else
ifeq ($(uname_S),DragonFly)
# DragonFly
FINAL_LDFLAGS+= -rdynamic
FINAL_LIBS+= -lpthread -lexecinfo
else
ifeq ($(uname_S),OpenBSD)
# OpenBSD
FINAL_LDFLAGS+= -rdynamic
FINAL_LIBS+= -lpthread -lexecinfo
else
ifeq ($(uname_S),NetBSD)
# NetBSD
FINAL_LDFLAGS+= -rdynamic
FINAL_LIBS+= -lpthread -lexecinfo
else
ifeq ($(uname_S),Haiku)
@@ -251,25 +256,40 @@ ifdef OPENSSL_PREFIX
endif
# Include paths to dependencies
FINAL_CFLAGS+= -I../deps/libvalkey/include -I../deps/linenoise -I../deps/hdr_histogram -I../deps/fpconv
FINAL_CFLAGS+= -I../deps/libvalkey/include -I../deps/linenoise -I../deps/hdr_histogram -I../deps/fpconv -I../deps/fast_float
# Lua scripting engine module
LUA_MODULE_NAME:=modules/lua/libvalkeylua.so
ifeq ($(BUILD_LUA),no)
LUA_MODULE_NAME=
LUA_MODULE=
LUA_MODULE_INSTALL=
else
FINAL_CFLAGS+=-DSTATIC_LUA=0
else ifeq ($(BUILD_LUA),module)
LUA_MODULE_NAME=modules/lua/libvalkeylua.so
LUA_MODULE=$(LUA_MODULE_NAME)
LUA_MODULE_INSTALL=install-lua-module
current_dir = $(shell pwd)
FINAL_CFLAGS+=-DLUA_ENABLED -DLUA_LIB=libvalkeylua.so
ifeq ($(uname_S),Darwin)
FINAL_LDFLAGS+= -Wl,-rpath,$(PREFIX)/lib
FINAL_LDFLAGS+= -Wl,-rpath,$(current_dir)/modules/lua
FINAL_CFLAGS+=-DLUA_ENABLED -DLUA_LIB=libvalkeylua.so -DSTATIC_LUA=0
ifeq ($(uname_S),Darwin)
FINAL_LDFLAGS+= -Wl,-rpath,$(PREFIX)/lib
FINAL_LDFLAGS+= -Wl,-rpath,$(current_dir)/modules/lua
else
FINAL_LDFLAGS+= -Wl,-rpath,$(PREFIX)/lib:$(current_dir)/modules/lua -Wl,--disable-new-dtags
endif
else
FINAL_LDFLAGS+= -Wl,-rpath,$(PREFIX)/lib:$(current_dir)/modules/lua -Wl,--disable-new-dtags
endif
# The default: building Lua as a static module.
LUA_MODULE_NAME=modules/lua/libvalkeylua.a
LUA_MODULE=$(LUA_MODULE_NAME)
current_dir = $(shell pwd)
FINAL_CFLAGS+=-DLUA_ENABLED -DSTATIC_LUA=1
ifeq ($(uname_S),Darwin)
LUA_LDFLAGS=-Wl,-export_dynamic -Wl,-force_load,$(current_dir)/modules/lua/libvalkeylua.a ../deps/lua/src/liblua.a
else ifeq ($(uname_S),FreeBSD)
LUA_LDFLAGS=-Wl,--export-dynamic -Wl,--whole-archive $(current_dir)/modules/lua/libvalkeylua.a -Wl,--no-whole-archive ../deps/lua/src/liblua.a
else
LUA_LDFLAGS=-Wl,--whole-archive $(current_dir)/modules/lua/libvalkeylua.a -Wl,--no-whole-archive ../deps/lua/src/liblua.a
endif
endif
# Determine systemd support and/or build preference (defaulting to auto-detection)
@@ -373,7 +393,7 @@ RDMA_MODULE_NAME:=valkey-rdma$(PROG_SUFFIX).so
RDMA_MODULE_CFLAGS:=$(FINAL_CFLAGS)
ifeq ($(BUILD_RDMA),module)
FINAL_CFLAGS+=-DUSE_RDMA=$(BUILD_MODULE)
RDMA_CLIENT_LIBS = ../deps/libvalkey/lib/libvalkey_rdma.a $(RDMA_LIBS)
RDMA_CLIENT_LIBS = ../deps/libvalkey/lib/libvalkey_rdma.a
RDMA_MODULE=$(RDMA_MODULE_NAME)
RDMA_MODULE_CFLAGS+=-DUSE_RDMA=$(BUILD_MODULE) -DBUILD_RDMA_MODULE=$(BUILD_MODULE) $(RDMA_LIBS)
endif
@@ -471,6 +491,7 @@ ENGINE_SERVER_OBJ = \
config.o \
connection.o \
crc16.o \
crc16_slottable.o \
crc64.o \
crccombine.o \
crcspeed.o \
@@ -555,11 +576,13 @@ ENGINE_SERVER_OBJ = \
util.o \
valkey-check-aof.o \
valkey-check-rdb.o \
valkey_strtod.o \
vector.o \
vset.o \
ziplist.o \
zipmap.o \
zmalloc.o
zmalloc.o \
queues.o
ENGINE_SERVER_OBJ+=$(ENGINE_TRACE_OBJ)
ENGINE_CLI_NAME=$(ENGINE_NAME)-cli$(PROG_SUFFIX)
ENGINE_CLI_OBJ = \
@@ -583,6 +606,7 @@ ENGINE_CLI_OBJ = \
strl.o \
util.o \
valkey-cli.o \
valkey_strtod.o \
zmalloc.o
ENGINE_BENCHMARK_NAME=$(ENGINE_NAME)-benchmark$(PROG_SUFFIX)
ENGINE_BENCHMARK_OBJ = \
@@ -591,6 +615,7 @@ ENGINE_BENCHMARK_OBJ = \
anet.o \
cli_common.o \
crc16.o \
crc16_slottable.o \
crc64.o \
crccombine.o \
crcspeed.o \
@@ -607,26 +632,14 @@ ENGINE_BENCHMARK_OBJ = \
strl.o \
util.o \
valkey-benchmark.o \
valkey_strtod.o \
zmalloc.o
ENGINE_CHECK_RDB_NAME=$(ENGINE_NAME)-check-rdb$(PROG_SUFFIX)
ENGINE_CHECK_AOF_NAME=$(ENGINE_NAME)-check-aof$(PROG_SUFFIX)
ENGINE_LIB_NAME=lib$(ENGINE_NAME).a
ENGINE_TEST_FILES:=$(wildcard unit/*.c)
ENGINE_TEST_OBJ:=$(sort $(patsubst unit/%.c,unit/%.o,$(ENGINE_TEST_FILES)))
ENGINE_UNIT_TESTS:=$(ENGINE_NAME)-unit-tests$(PROG_SUFFIX)
ENGINE_UNIT_GTESTS:=$(ENGINE_NAME)-unit-gtests$(PROG_SUFFIX)
ALL_SOURCES=$(sort $(patsubst %.o,%.c,$(ENGINE_SERVER_OBJ) $(ENGINE_CLI_OBJ) $(ENGINE_BENCHMARK_OBJ)))
USE_FAST_FLOAT?=no
ifeq ($(USE_FAST_FLOAT),yes)
# valkey_strtod.h uses this flag to switch valkey_strtod function to fast_float_strtod,
# therefore let's pass it to compiler for preprocessing.
FINAL_CFLAGS += -D USE_FAST_FLOAT
# next, let's build and add actual library containing fast_float_strtod function for linking.
DEPENDENCY_TARGETS += fast_float_c_interface
FAST_FLOAT_STRTOD_OBJECT := ../deps/fast_float_c_interface/fast_float_strtod.o
FINAL_LIBS += $(FAST_FLOAT_STRTOD_OBJECT)
endif
USE_LIBBACKTRACE?=no
ifeq ($(USE_LIBBACKTRACE),yes)
FINAL_CFLAGS += -DUSE_LIBBACKTRACE
@@ -637,7 +650,9 @@ ifeq ($(USE_LIBBACKTRACE),yes)
FINAL_LIBS += -lbacktrace
endif
ALL_BUILD_PREREQUISITES=$(SERVER_NAME) $(ENGINE_SENTINEL_NAME) $(ENGINE_CLI_NAME) $(ENGINE_BENCHMARK_NAME) $(ENGINE_CHECK_RDB_NAME) $(ENGINE_CHECK_AOF_NAME) $(TLS_MODULE) $(RDMA_MODULE) $(LUA_MODULE)
DEPENDENCY_TARGETS+= gtest-parallel
ALL_BUILD_PREREQUISITES=$(SERVER_NAME) $(ENGINE_SENTINEL_NAME) $(ENGINE_CLI_NAME) $(ENGINE_BENCHMARK_NAME) $(ENGINE_CHECK_RDB_NAME) $(ENGINE_CHECK_AOF_NAME) $(TLS_MODULE) $(RDMA_MODULE)
all: $(ALL_BUILD_PREREQUISITES)
@echo ""
@@ -653,7 +668,7 @@ endif
.PHONY: all
all-with-unit-tests: all $(ENGINE_UNIT_TESTS)
all-with-unit-tests: all $(ENGINE_UNIT_GTESTS)
.PHONY: all
persist-settings: distclean
@@ -666,14 +681,15 @@ persist-settings: distclean
echo BUILD_RDMA=$(BUILD_RDMA) >> .make-settings
echo USE_SYSTEMD=$(USE_SYSTEMD) >> .make-settings
echo BUILD_LUA=$(BUILD_LUA) >> .make-settings
echo USE_FAST_FLOAT=$(USE_FAST_FLOAT) >> .make-settings
echo USE_LIBBACKTRACE=$(USE_LIBBACKTRACE) >> .make-settings
echo LIBBACKTRACE_PREFIX=$(LIBBACKTRACE_PREFIX) >> .make-settings
echo CFLAGS=$(CFLAGS) >> .make-settings
echo LDFLAGS=$(LDFLAGS) >> .make-settings
echo SERVER_CFLAGS=$(SERVER_CFLAGS) >> .make-settings
echo SERVER_LDFLAGS=$(SERVER_LDFLAGS) >> .make-settings
echo PREV_FINAL_CFLAGS=$(FINAL_CFLAGS) >> .make-settings
echo PREV_FINAL_LDFLAGS=$(FINAL_LDFLAGS) >> .make-settings
echo ENGINE_SERVER_OBJ=$(ENGINE_SERVER_OBJ) >> .make-settings
-(cd ../deps && $(MAKE) $(DEPENDENCY_TARGETS))
.PHONY: persist-settings
@@ -692,17 +708,13 @@ ifneq ($(strip $(PREV_FINAL_LDFLAGS)), $(strip $(FINAL_LDFLAGS)))
endif
# valkey-server
$(SERVER_NAME): $(ENGINE_SERVER_OBJ)
$(SERVER_LD) -o $@ $^ ../deps/libvalkey/lib/libvalkey.a ../deps/hdr_histogram/libhdrhistogram.a ../deps/fpconv/libfpconv.a $(FINAL_LIBS)
$(SERVER_NAME): $(ENGINE_SERVER_OBJ) $(LUA_MODULE)
$(SERVER_LD) -o $@ $(ENGINE_SERVER_OBJ) ../deps/libvalkey/lib/libvalkey.a ../deps/hdr_histogram/libhdrhistogram.a ../deps/fpconv/libfpconv.a $(FINAL_LIBS) $(LUA_LDFLAGS)
# Valkey static library, used to compile against for unit testing
$(ENGINE_LIB_NAME): $(ENGINE_SERVER_OBJ)
$(SERVER_AR) rcs $@ $^
# valkey-unit-tests
$(ENGINE_UNIT_TESTS): $(ENGINE_TEST_OBJ) $(ENGINE_LIB_NAME)
$(SERVER_LD) -o $@ $^ ../deps/libvalkey/lib/libvalkey.a ../deps/hdr_histogram/libhdrhistogram.a ../deps/fpconv/libfpconv.a $(FINAL_LIBS)
# valkey-sentinel
$(ENGINE_SENTINEL_NAME): $(SERVER_NAME)
$(ENGINE_INSTALL) $(SERVER_NAME) $(ENGINE_SENTINEL_NAME)
@@ -717,15 +729,15 @@ $(ENGINE_CHECK_AOF_NAME): $(SERVER_NAME)
# valkey-tls.so
$(TLS_MODULE_NAME): $(SERVER_NAME)
$(QUIET_CC)$(CC) -o $@ tls.c -shared -fPIC $(TLS_MODULE_CFLAGS) $(TLS_CLIENT_LIBS)
$(QUIET_CC)$(CC) $(LDFLAGS) -o $@ tls.c -shared -fPIC $(TLS_MODULE_CFLAGS) $(TLS_CLIENT_LIBS)
# valkey-rdma.so
$(RDMA_MODULE_NAME): $(SERVER_NAME)
$(QUIET_CC)$(CC) -o $@ rdma.c -shared -fPIC $(RDMA_MODULE_CFLAGS)
$(QUIET_CC)$(CC) $(LDFLAGS) -o $@ rdma.c -shared -fPIC $(RDMA_MODULE_CFLAGS)
# engine_lua.so
$(LUA_MODULE_NAME): $(SERVER_NAME)
cd modules/lua && $(MAKE) OPTIMIZATION="$(OPTIMIZATION)"
$(LUA_MODULE_NAME): .make-prerequisites
$(MAKE) -C modules/lua OPTIMIZATION="$(OPTIMIZATION)" BUILD_LUA="$(BUILD_LUA)"
# valkey-cli
$(ENGINE_CLI_NAME): $(ENGINE_CLI_OBJ)
@@ -747,9 +759,6 @@ DEP = $(ENGINE_SERVER_OBJ:%.o=%.d) $(ENGINE_CLI_OBJ:%.o=%.d) $(ENGINE_BENCHMARK_
trace/%.o: trace/%.c .make-prerequisites
$(SERVER_CC) -Itrace -MMD -o $@ -c $<
unit/%.o: unit/%.c .make-prerequisites
$(SERVER_CC) -MMD -o $@ -c $<
# The following files are checked in and don't normally need to be rebuilt. They
# are built only if python is available and their prereqs are modified.
ifneq (,$(PYTHON))
@@ -760,17 +769,12 @@ fmtargs.h: ../utils/generate-fmtargs.py
$(QUITE_GEN)sed '/Everything below this line/,$$d' $@ > $@.tmp
$(QUITE_GEN)$(PYTHON) ../utils/generate-fmtargs.py >> $@.tmp
$(QUITE_GEN)mv $@.tmp $@
unit/test_files.h: unit/*.c ../utils/generate-unit-test-header.py
$(QUIET_GEN)$(PYTHON) ../utils/generate-unit-test-header.py
unit/test_main.o: unit/test_files.h
endif
commands.c: $(COMMANDS_DEF_FILENAME).def
clean:
rm -rf $(SERVER_NAME) $(ENGINE_SENTINEL_NAME) $(ENGINE_CLI_NAME) $(ENGINE_BENCHMARK_NAME) $(ENGINE_CHECK_RDB_NAME) $(ENGINE_CHECK_AOF_NAME) $(ENGINE_UNIT_TESTS) $(ENGINE_LIB_NAME) unit/*.o unit/*.d trace/*.o trace/*.d *.o *.gcda *.gcno *.gcov valkey.info lcov-html Makefile.dep *.so
rm -rf $(SERVER_NAME) $(ENGINE_SENTINEL_NAME) $(ENGINE_CLI_NAME) $(ENGINE_BENCHMARK_NAME) $(ENGINE_CHECK_RDB_NAME) $(ENGINE_CHECK_AOF_NAME) $(ENGINE_UNIT_GTESTS) $(ENGINE_LIB_NAME) trace/*.o trace/*.d *.o *.gcda *.gcno *.gcov valkey.info lcov-html Makefile.dep *.so
rm -f $(DEP)
-(cd modules/lua && $(MAKE) clean)
@@ -780,15 +784,21 @@ distclean: clean
-(cd ../deps && $(MAKE) distclean)
-(cd modules && $(MAKE) clean)
-(cd ../tests/modules && $(MAKE) clean)
-(cd unit && $(MAKE) clean)
-(rm -f .make-*)
-(find unit -type d -name __pycache__ -exec rm -rf {} +)
.PHONY: distclean
test: $(ALL_BUILD_PREREQUISITES)
@(cd ..; VALKEY_PROG_SUFFIX="$(PROG_SUFFIX)" ./runtest)
test-unit: $(ENGINE_UNIT_TESTS)
./$(ENGINE_UNIT_TESTS)
test-unit:
@(cd unit && $(MAKE) test-unit)
# valkey-unit-gtests
$(ENGINE_UNIT_GTESTS): $(ENGINE_LIB_NAME)
@(cd unit && $(MAKE) $(ENGINE_UNIT_GTESTS))
test-modules: $(SERVER_NAME)
@(cd ..; ./runtest-moduleapi)
@@ -805,6 +815,7 @@ lcov:
@lcov --version
$(MAKE) gcov
@(set -e; cd ..; ./runtest)
$(MAKE) test-unit COVERAGE=1 COVERAGE_CFLAGS="-fprofile-arcs -ftest-coverage"
@geninfo -o valkey.info .
@genhtml --legend -o lcov-html valkey.info
src/aof.c
+127 -1
View File
@@ -1005,6 +1005,131 @@ int startAppendOnly(void) {
return C_OK;
}
/* Try to restart AOF after replica full sync by adopting `server.rdb_filename`
* as the new BASE file (RDB preamble mode), avoiding a redundant AOFRW.
* Returns C_OK on success; on C_ERR caller should fallback to
* restartAOFAfterSYNC(). */
int restartAOFWithSyncRdb(void) {
serverAssert(server.aof_state == AOF_OFF);
int ret = C_ERR;
int newfd = -1, rdbfile_renamed = 0;
sds new_base_filename = NULL;
sds new_base_filepath = NULL;
sds new_incr_filename = NULL;
sds new_incr_filepath = NULL;
aofManifest *temp_am = NULL;
if (dirCreateIfMissing(server.aof_dirname) == -1) {
serverLog(LL_WARNING, "Can't open or create append-only dir %s: %s", server.aof_dirname, strerror(errno));
goto cleanup;
}
serverAssert(server.aof_manifest != NULL);
temp_am = aofManifestDup(server.aof_manifest);
new_base_filename = getNewBaseFileNameAndMarkPreAsHistory(temp_am, server.aof_use_rdb_preamble);
serverAssert(new_base_filename != NULL);
new_base_filepath = makePath(server.aof_dirname, new_base_filename);
if (rename(server.rdb_filename, new_base_filepath) == -1) {
serverLog(LL_WARNING, "Error trying to rename the RDB file %s into %s: %s", server.rdb_filename,
new_base_filepath, strerror(errno));
goto cleanup;
}
rdbfile_renamed = 1;
/* Mark existing incr AOF files as history BEFORE creating the new one,
* so the new incr entry is not inadvertently moved to history. */
markRewrittenIncrAofAsHistory(temp_am);
new_incr_filename = getNewIncrAofName(temp_am);
new_incr_filepath = makePath(server.aof_dirname, new_incr_filename);
newfd = open(new_incr_filepath, O_WRONLY | O_TRUNC | O_CREAT, 0666);
if (newfd == -1) {
serverLog(LL_WARNING, "Can't open the append-only file %s: %s", new_incr_filename, strerror(errno));
goto cleanup;
}
if (persistAofManifest(temp_am) == C_ERR) {
goto cleanup;
}
aofManifestFreeAndUpdate(temp_am);
temp_am = NULL;
aofDelHistoryFiles();
sdsfree(new_base_filepath);
new_base_filepath = NULL;
sdsfree(new_incr_filepath);
new_incr_filepath = NULL;
/* Drain pending fsync jobs from the previous AOF before publishing the new
* fsynced offset to avoid races on fsynced_reploff_pending. */
bioDrainWorker(BIO_AOF_FSYNC);
atomic_store_explicit(&server.fsynced_reploff_pending, server.primary_repl_offset, memory_order_relaxed);
server.fsynced_reploff =
atomic_load_explicit(&server.fsynced_reploff_pending, memory_order_relaxed);
int aof_bio_fsync_status = atomic_load_explicit(&server.aof_bio_fsync_status, memory_order_relaxed);
if (aof_bio_fsync_status == C_ERR) {
serverLog(LL_WARNING, "AOF reopen, just ignore the AOF fsync error in bio job");
atomic_store_explicit(&server.aof_bio_fsync_status, C_OK, memory_order_relaxed);
}
if (server.aof_last_write_status == C_ERR) {
serverLog(LL_WARNING, "AOF reopen, just ignore the last error.");
server.aof_last_write_status = C_OK;
}
server.aof_lastbgrewrite_status = C_OK;
server.stat_aofrw_consecutive_failures = 0;
server.aof_last_fsync = server.mstime;
server.aof_last_incr_size = 0;
server.aof_last_incr_fsync_offset = 0;
server.aof_rewrite_base_size = getAppendOnlyFileSize(new_base_filename, NULL);
server.aof_current_size = server.aof_rewrite_base_size;
server.aof_fd = newfd;
server.aof_state = AOF_ON;
serverLog(LL_NOTICE, "Reused RDB file from primary sync as AOF base file: %s", new_base_filename);
ret = C_OK;
return ret;
cleanup:
if (rdbfile_renamed) {
if (rename(new_base_filepath, server.rdb_filename) == -1) {
serverLog(LL_WARNING,
"Failed to rename AOF base back to RDB file %s: %s. "
"Orphan file may remain at %s",
server.rdb_filename, strerror(errno), new_base_filepath);
} else {
rdbfile_renamed = 0;
}
}
if (server.rdb_del_sync_files && allPersistenceDisabled()) {
serverLog(LL_NOTICE, "Removing the RDB file obtained from "
"the primary. This replica has persistence "
"disabled");
if (rdbfile_renamed) {
bg_unlink(new_base_filepath);
} else {
bg_unlink(server.rdb_filename);
}
}
if (newfd != -1) close(newfd);
if (new_incr_filepath) {
bg_unlink(new_incr_filepath);
sdsfree(new_incr_filepath);
}
if (temp_am) aofManifestFree(temp_am);
if (new_base_filepath) sdsfree(new_base_filepath);
return ret;
}
/* This is a wrapper to the write syscall in order to retry on short writes
* or if the syscall gets interrupted. It could look strange that we retry
* on short writes given that we are writing to a block device: normally if
@@ -1376,7 +1501,8 @@ struct client *createAOFClient(void) {
* background processing there is a chance that the
* command execution order will be violated.
*/
c->raw_flag = 0;
c->raw_flag1 = 0;
c->raw_flag2 = 0;
c->flag.deny_blocking = 1;
c->flag.fake = 1;
src/bio.c
+13 -11
View File
@@ -144,6 +144,12 @@ typedef union bio_job {
void *bioProcessBackgroundJobs(void *arg);
/* Allocate a bio_job. Marked noinline so that it appears as a distinct frame in valgrind
* stack traces, allowing a targeted Valgrind suppression for BIO job leaks */
__attribute__((noinline)) static bio_job *allocBioJob(size_t extra) {
return zmalloc(sizeof(bio_job) + extra);
}
/* Make sure we have enough stack to perform all the things we do in the
* main thread. */
#define VALKEY_THREAD_STACK_SIZE (1024 * 1024 * 4)
@@ -188,7 +194,7 @@ void bioCreateLazyFreeJob(lazy_free_fn free_fn, int arg_count, ...) {
va_list valist;
/* Allocate memory for the job structure and all required
* arguments */
bio_job *job = zmalloc(sizeof(*job) + sizeof(void *) * (arg_count));
bio_job *job = allocBioJob(sizeof(void *) * (arg_count));
job->free_args.free_fn = free_fn;
va_start(valist, arg_count);
@@ -200,7 +206,7 @@ void bioCreateLazyFreeJob(lazy_free_fn free_fn, int arg_count, ...) {
}
void bioCreateCloseJob(int fd, int need_fsync, int need_reclaim_cache) {
bio_job *job = zmalloc(sizeof(*job));
bio_job *job = allocBioJob(0);
job->fd_args.fd = fd;
job->fd_args.need_fsync = need_fsync;
job->fd_args.need_reclaim_cache = need_reclaim_cache;
@@ -209,7 +215,7 @@ void bioCreateCloseJob(int fd, int need_fsync, int need_reclaim_cache) {
}
void bioCreateCloseAofJob(int fd, long long offset, int need_reclaim_cache) {
bio_job *job = zmalloc(sizeof(*job));
bio_job *job = allocBioJob(0);
job->fd_args.fd = fd;
job->fd_args.offset = offset;
job->fd_args.need_fsync = 1;
@@ -219,7 +225,7 @@ void bioCreateCloseAofJob(int fd, long long offset, int need_reclaim_cache) {
}
void bioCreateFsyncJob(int fd, long long offset, int need_reclaim_cache) {
bio_job *job = zmalloc(sizeof(*job));
bio_job *job = allocBioJob(0);
job->fd_args.fd = fd;
job->fd_args.offset = offset;
job->fd_args.need_reclaim_cache = need_reclaim_cache;
@@ -228,14 +234,14 @@ void bioCreateFsyncJob(int fd, long long offset, int need_reclaim_cache) {
}
void bioCreateSaveRDBToDiskJob(connection *conn, int is_dual_channel) {
bio_job *job = zmalloc(sizeof(*job));
bio_job *job = allocBioJob(0);
job->save_to_disk_args.conn = conn;
job->save_to_disk_args.is_dual_channel = is_dual_channel;
bioSubmitJob(BIO_RDB_SAVE, job);
}
void bioCreateTlsReloadJob(void) {
bio_job *job = zmalloc(sizeof(*job));
bio_job *job = allocBioJob(0);
bioSubmitJob(BIO_TLS_RELOAD, job);
}
@@ -260,9 +266,7 @@ void *bioProcessBackgroundJobs(void *arg) {
bio_worker_num = bioWorkerNum(bwd);
while (1) {
/* Keep the job in the queue until it's fully processed so cancellation
* won't leave an untracked in-flight allocation. */
bio_job *job = mutexQueuePeek(bwd->bio_jobs, true);
bio_job *job = mutexQueuePop(bwd->bio_jobs, true);
/* Process the job accordingly to its type. */
int job_type = job->header.type;
@@ -313,8 +317,6 @@ void *bioProcessBackgroundJobs(void *arg) {
} else {
serverPanic("Wrong job type in bioProcessBackgroundJobs().");
}
void *removed_job = mutexQueuePop(bwd->bio_jobs, false);
serverAssert(removed_job == job);
zfree(job);
atomic_fetch_sub(&bio_jobs_counter[job_type], 1);
}
src/call_reply.c
+286 -2
View File
@@ -35,6 +35,44 @@
#define REPLY_FLAG_RESP3 (1 << 2)
#define REPLY_FLAG_EXACT_TYPE (1 << 3)
/* ============================================================
* Two-level callback architecture
* ============================================================
*
* There are two distinct sets of callbacks used in this file:
*
* Level 1 – ReplyParserCallbacks (resp_parser.h)
* The low-level, recursive-descent parser in resp_parser.c calls these
* during a single pass over a raw RESP buffer. Collection callbacks
* (array, set, map, attribute) receive a `ReplyParser *` so they can
* recurse into child elements by calling parseReply() themselves.
* Scalar callbacks receive the raw wire bytes (proto/proto_len) in
* addition to the parsed value. Two separate Level-1 callback tables
* are defined below:
*
* CallReplyParserCallbacks – each callback receives the target
* CallReply* directly as ctx; used by callReplyParse() to build a
* CallReply node tree from a captured RESP buffer. Passing
* CallReply* directly makes this path naturally reentrant: a
* module that calls VM_Call from inside a VM_CallArgv typed
* callback triggers a nested callReplyParse() that operates on an
* entirely independent CallReply tree.
*
* ReplyHandlersParserCallbacks – wires the replyHandlers* adapter
* functions; used by invokeReplyHandlers() to dispatch a live
* reply to the ValkeyModuleReplyHandlers provided by the module.
*
* Level 2 – ValkeyModuleReplyHandlers (valkeymodule.h)
* The public module API. These user-supplied callbacks receive only
* clean, parsed values (no raw RESP bytes, no ReplyParser pointer).
* Collection callbacks come in matched Start/End pairs so the module
* does not need to drive the parser itself. The replyHandlers*
* adapters bridge Level 1 → Level 2: they accept Level-1 arguments,
* strip the parser internals, and call the corresponding
* ValkeyModuleReplyHandlers callback with just the value arguments.
* ============================================================ */
/* --------------------------------------------------------
* An opaque struct used to parse a RESP protocol reply and
* represent it. Used when parsing replies such as in RM_Call
@@ -270,7 +308,13 @@ CallReply *callReplyCreatePromise(void *private_data) {
return res;
}
static const ReplyParserCallbacks DefaultParserCallbacks = {
/* Callback table used by callReplyParse() to build a CallReply tree from a
* captured RESP buffer. Each callback receives the target CallReply* directly
* as its opaque ctx argument. Because every callReplyParse() invocation is
* stack-local and carries no shared state, this path is naturally reentrant:
* a module that calls VM_Call from inside a VM_CallArgv typed callback will
* trigger a nested callReplyParse() with an independent CallReply tree. */
static const ReplyParserCallbacks CallReplyParserCallbacks = {
.null_callback = callReplyNull,
.bulk_string_callback = callReplyBulkString,
.null_bulk_string_callback = callReplyNullBulkString,
@@ -296,7 +340,7 @@ static void callReplyParse(CallReply *rep) {
return;
}
ReplyParser parser = {.curr_location = rep->proto, .callbacks = DefaultParserCallbacks};
ReplyParser parser = {.curr_location = rep->proto, .callbacks = CallReplyParserCallbacks};
parseReply(&parser, rep);
rep->flags |= REPLY_FLAG_PARSED;
@@ -582,3 +626,243 @@ void enableParseExactReplyTypeFlag(CallReply *rep) {
serverAssert(!(rep->flags & REPLY_FLAG_PARSED));
rep->flags |= REPLY_FLAG_EXACT_TYPE;
}
/* Internal pairing of a const callback table with its associated context.
* Created on the stack by callers so the public ValkeyModuleReplyHandlers
* struct remains immutable (no context field). The recursive-descent parser
* receives a pointer to one of these as its opaque `ctx` argument. */
typedef struct {
const ValkeyModuleReplyHandlers *handlers;
void *context;
} RespHandlersCtx;
static void replyHandlersNull(void *ctx, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (!w->handlers->null) return;
w->handlers->null(w->context);
}
static void replyHandlersBulkString(void *ctx, const char *str, size_t len, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (!w->handlers->bulkString) return;
w->handlers->bulkString(w->context, str, len);
}
static void replyHandlersNullBulkString(void *ctx, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (!w->handlers->nullBulkString) return;
w->handlers->nullBulkString(w->context);
}
static void replyHandlersNullArray(void *ctx, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (!w->handlers->nullArray) return;
w->handlers->nullArray(w->context);
}
static void replyHandlersError(void *ctx, const char *str, size_t len, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (!w->handlers->error) return;
w->handlers->error(w->context, str, len);
}
static void replyHandlersSimpleStr(void *ctx, const char *str, size_t len, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (!w->handlers->simpleString) return;
w->handlers->simpleString(w->context, str, len);
}
static void replyHandlersLong(void *ctx, long long val, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (!w->handlers->integer) return;
w->handlers->integer(w->context, val);
}
static void replyHandlersArray(ReplyParser *parser, void *ctx, size_t len, const char *proto) {
UNUSED(proto);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (w->handlers->arrayStart) w->handlers->arrayStart(w->context, len);
/* Always consume all child elements to keep the parser in sync,
* regardless of whether the start/end callbacks are set. */
for (size_t i = 0; i < len; i++) {
parseReply(parser, ctx);
}
if (w->handlers->arrayEnd) w->handlers->arrayEnd(w->context);
}
static void replyHandlersSet(ReplyParser *parser, void *ctx, size_t len, const char *proto) {
UNUSED(proto);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (w->handlers->setStart) w->handlers->setStart(w->context, len);
for (size_t i = 0; i < len; i++) {
parseReply(parser, ctx);
}
if (w->handlers->setEnd) w->handlers->setEnd(w->context);
}
static void replyHandlersMap(ReplyParser *parser, void *ctx, size_t len, const char *proto) {
UNUSED(proto);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (w->handlers->mapStart) w->handlers->mapStart(w->context, len);
for (size_t i = 0; i < len; i++) {
parseReply(parser, ctx);
parseReply(parser, ctx);
}
if (w->handlers->mapEnd) w->handlers->mapEnd(w->context);
}
static void replyHandlersDouble(void *ctx, double val, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (!w->handlers->doubleVal) return;
w->handlers->doubleVal(w->context, val);
}
static void replyHandlersBool(void *ctx, int val, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (!w->handlers->boolVal) return;
w->handlers->boolVal(w->context, val);
}
static void replyHandlersBigNumber(void *ctx, const char *str, size_t len, const char *proto, size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (!w->handlers->bigNumber) return;
w->handlers->bigNumber(w->context, str, len);
}
static void replyHandlersVerbatimString(void *ctx,
const char *format,
const char *str,
size_t len,
const char *proto,
size_t proto_len) {
UNUSED(proto);
UNUSED(proto_len);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (!w->handlers->verbatimString) return;
w->handlers->verbatimString(w->context, str, len, format);
}
static void replyHandlersAttribute(ReplyParser *parser, void *ctx, size_t len, const char *proto) {
UNUSED(proto);
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (w->handlers->attributeStart) w->handlers->attributeStart(w->context, len);
for (size_t i = 0; i < len; i++) {
parseReply(parser, ctx);
parseReply(parser, ctx);
}
if (w->handlers->attributeEnd) w->handlers->attributeEnd(w->context);
/* Always parse the element that follows the attribute section. */
parseReply(parser, ctx);
}
static void replyHandlersParseError(void *ctx) {
RespHandlersCtx *w = (RespHandlersCtx *)ctx;
if (!w->handlers->replyParsingError) return;
w->handlers->replyParsingError(w->context);
}
static const ReplyParserCallbacks ReplyHandlersParserCallbacks = {
.null_callback = replyHandlersNull,
.bulk_string_callback = replyHandlersBulkString,
.null_bulk_string_callback = replyHandlersNullBulkString,
.null_array_callback = replyHandlersNullArray,
.error_callback = replyHandlersError,
.simple_str_callback = replyHandlersSimpleStr,
.long_callback = replyHandlersLong,
.array_callback = replyHandlersArray,
.set_callback = replyHandlersSet,
.map_callback = replyHandlersMap,
.double_callback = replyHandlersDouble,
.bool_callback = replyHandlersBool,
.big_number_callback = replyHandlersBigNumber,
.verbatim_string_callback = replyHandlersVerbatimString,
.attribute_callback = replyHandlersAttribute,
.error = replyHandlersParseError,
};
/* Parse the RESP reply accumulated in client `c`'s output buffer and deliver
* it to the handler callbacks in `handlers`.
*
* If `handlers->onRespAvailable` is set it is called first with the raw
* RESP bytes. When it returns 0, per-type callbacks are skipped; when it
* returns 1 (or is NULL), the reply is walked recursively and each value is
* dispatched to the matching typed callback.
*
* The client's output buffer is consumed by this call (bufpos reset, reply
* list drained). */
void invokeReplyHandlers(ValkeyModuleCtx *ctx, client *c, const ValkeyModuleReplyHandlers *handlers, void *reply_ctx) {
char *buf = NULL;
size_t buf_len = 0;
int free_buffer = 0;
serverAssert(!c->flag.blocked);
if (listLength(c->reply) == 0 && (size_t)c->bufpos < c->buf_usable_size) {
/* This is a fast path for the common case of a reply inside the
* client static buffer. Don't create an SDS string but just use
* the client buffer directly. */
c->buf[c->bufpos] = '\0';
buf = c->buf;
buf_len = c->bufpos;
c->bufpos = 0;
} else {
listIter iter;
listRewind(c->reply, &iter);
listNode *node;
size_t lensum = c->bufpos;
while ((node = listNext(&iter))) {
clientReplyBlock *o = listNodeValue(node);
lensum += o->used;
}
buf = zmalloc_usable(lensum + 1, NULL);
char *ptr = buf;
memcpy(ptr, c->buf, c->bufpos);
ptr += c->bufpos;
c->bufpos = 0;
while (listLength(c->reply)) {
clientReplyBlock *o = listNodeValue(listFirst(c->reply));
memcpy(ptr, o->buf, o->used);
ptr += o->used;
listDelNode(c->reply, listFirst(c->reply));
}
ptr[0] = '\0';
buf_len = lensum;
free_buffer = 1;
}
int continue_parsing = 1;
if (handlers->onRespAvailable) {
continue_parsing = handlers->onRespAvailable(reply_ctx, ctx, buf, buf_len);
}
if (continue_parsing) {
RespHandlersCtx dispatch_ctx = {.handlers = handlers, .context = reply_ctx};
ReplyParser parser = {.curr_location = buf, .callbacks = ReplyHandlersParserCallbacks};
parseReply(&parser, &dispatch_ctx);
}
if (free_buffer) {
zfree(buf);
}
}
src/call_reply.h
+3
View File
@@ -31,6 +31,7 @@
#define SRC_CALL_REPLY_H_
#include "resp_parser.h"
#include "valkeymodule.h"
typedef struct CallReply CallReply;
typedef void (*ValkeyModuleOnUnblocked)(void *ctx, CallReply *reply, void *private_data);
@@ -58,4 +59,6 @@ void freeCallReply(CallReply *rep);
CallReply *callReplyCreatePromise(void *private_data);
void enableParseExactReplyTypeFlag(CallReply *rep);
void invokeReplyHandlers(ValkeyModuleCtx *ctx, client *c, const ValkeyModuleReplyHandlers *handlers, void *reply_ctx);
#endif /* SRC_CALL_REPLY_H_ */
src/cluster.c
+210 -1
View File
@@ -41,6 +41,7 @@
#include "cluster.h"
#include "cluster_slot_stats.h"
#include "module.h"
#include "crc16_slottable.h"
#include <ctype.h>
@@ -268,7 +269,7 @@ void restoreCommand(client *c) {
return;
}
obj = rdbLoadObject(type, &payload, objectGetVal(key), c->db->id, NULL);
obj = rdbLoadObject(type, &payload, objectGetVal(key), c->db->id, NULL, RDBFLAGS_NONE, 0);
if (obj == NULL) {
addReplyError(c, "Bad data format");
return;
@@ -1433,6 +1434,11 @@ void addNodeToNodeReply(client *c, clusterNode *node) {
hostname[0] != '\0') {
length++;
}
if (sdslen(node->availability_zone) != 0) {
length++;
}
addReplyMapLen(c, length);
if (server.cluster_preferred_endpoint_type != CLUSTER_ENDPOINT_TYPE_IP) {
@@ -1446,6 +1452,13 @@ void addNodeToNodeReply(client *c, clusterNode *node) {
addReplyBulkCString(c, hostname);
length--;
}
if (sdslen(node->availability_zone) != 0) {
addReplyBulkCString(c, "availability-zone");
addReplyBulkCString(c, node->availability_zone);
length--;
}
serverAssert(length == 0);
}
@@ -1630,3 +1643,199 @@ void clusterCommandFlushslot(client *c) {
delKeysInSlot(slot, lazy, false, true);
addReply(c, shared.ok);
}
/* -----------------------------------------------------------------------------
* CLUSTERSCAN Command
* -------------------------------------------------------------------------- */
/* Compute and cache encoded fingerprint for CLUSTERSCAN cursor validation.
*
* Fingerprint uniquely identifies the current memory layout by hashing
* relevant configuration bits. Currently this includes only the hash seed,
* but additional factors (e.g., hash table type) can be mixed in later.
*
* Fingerprint is encoded using consecutive ASCII values starting from '0'
* (48 to 111), each represenenting 6 bits. This encoding avoids '-', '{', '}'
* which are part of the cursor.
*
* Returns a non zero 32-bit fingerprint. 0 is reserved for cross-node cursor */
static const char *clusterscanFingerprint(void) {
static char cached_fp[7];
if (cached_fp[0]) return cached_fp;
uint64_t *seed = (uint64_t *)hashtableGetHashFunctionSeed();
uint64_t hash = wangHash64(seed[0] ^ seed[1]);
/* Truncating to 32 bit instead of 64 bit */
uint32_t fp = (uint32_t)hash;
/* Ensure fingerprint is never 0, zero is reserved for cross node
* cursor where fingerprint validation would be skipped, scenarios
* include initial and end a given slots scan */
if (fp == 0) fp = 1;
/* Convert 32-bit fingerprint to 6 char string using base64 like encoding. */
for (int i = 5; i >= 0; i--) {
cached_fp[i] = '0' + (fp & 0x3F);
fp >>= 6;
}
cached_fp[6] = '\0';
return cached_fp;
}
/* Parse the cursor for CLUSTERSCAN Command.
* The format is <fingerprint>-<hashtag>-<cursor>.
*
* Fingerprint identifies the node's memory layout. On mismatch, the scan
* restarts from cursor 0 rather than returning an error.
*
* Hashtag is used to route to the correct node.
*
* Cursor is the actual local scan cursor.
*/
static int parseClusterScanCursor(robj *o, int *slot, unsigned long long *cursor) {
char *p = objectGetVal(o);
char *end = p + sdslen(p);
char *token;
/* Handle fingerprint */
token = strchr(p, '-');
if (!token) return C_ERR;
size_t fp_len = token - p;
char *fp_start = p;
p = token + 1;
/* Handle hashtag */
token = strchr(p, '-');
if (!token) return C_ERR;
int hash_slot = keyHashSlot(p, token - p);
*slot = hash_slot;
p = token + 1;
/* Handle the local cursor */
if (!string2ull(p, end - p, cursor)) return C_ERR;
/* Fingerprint is 0 when beginning to scan a slot so ignore the cursor value
* in that case. If fingerprint doesn't match the current node's fingerprint,
* restart the scan from cursor 0. From encoding we know fingerprint length is 6.*/
const char *fp = clusterscanFingerprint();
if (fp_len != 6 || memcmp(fp_start, fp, 6) != 0) {
*cursor = 0;
}
return C_OK;
}
/* CLUSTERSCAN command - topology-aware scan across cluster slots.
* Cursor format: <fingerprint>-<hashtag>-<local_cursor>
* Supports SLOT, MATCH, COUNT, and TYPE options. */
void clusterscanCommand(client *c) {
if (!server.cluster_enabled) {
addReplyError(c, "This instance has cluster support disabled");
return;
}
int slot;
unsigned long long cursor;
int input_slot = -1;
int match_slot = -1;
int skip_scan = 0;
/* Parse all arguments together so that values of MATCH/COUNT/SLOT/TYPE are
* not mistaken for the option names.*/
for (int i = 2; i < c->argc; i++) {
int remaining = c->argc - i;
char *opt = objectGetVal(c->argv[i]);
if (!strcasecmp(opt, "slot") && remaining >= 2) {
if (input_slot != -1) {
addReplyError(c, "SLOT option can only be specified once");
return;
}
if ((input_slot = getSlotOrReply(c, c->argv[i + 1])) == -1) return;
i++;
} else if (!strcasecmp(opt, "match") && remaining >= 2) {
sds pat = objectGetVal(c->argv[i + 1]);
int patlen = sdslen(pat);
match_slot = (patlen == 1 && pat[0] == '*') ? -1 : patternHashSlot(pat, patlen);
i++;
} else if ((!strcasecmp(opt, "count") || !strcasecmp(opt, "type")) && remaining >= 2) {
i++; /* Let scanGenericCommand parse this */
} else {
addReplyErrorObject(c, shared.syntaxerr);
return;
}
}
/* SLOT and single slot MATCH target different slots hence conclude the scan */
skip_scan = input_slot != -1 && match_slot != -1 && input_slot != match_slot;
/* Handle cursor "0" case. If slot information is provided we return
* the updated cursor to scan input slot, else scan from slot 0. */
if (strcmp(objectGetVal(c->argv[1]), "0") == 0) {
if (input_slot != -1) {
slot = input_slot;
} else if (match_slot != -1) {
slot = match_slot; /* If match maps to a particular slot, start scan from there */
} else {
slot = 0;
}
addReplyArrayLen(c, 2);
if (skip_scan) {
addReplyBulkCString(c, "0");
} else {
sds new_cursor = sdscatfmt(sdsempty(), "0-{%s}-0", crc16_slot_table[slot]);
addReplyBulkSds(c, new_cursor);
}
addReplyArrayLen(c, 0);
return;
} else {
if (parseClusterScanCursor(c->argv[1], &slot, &cursor) == C_ERR) {
addReplyError(c, "Invalid cursor");
return;
}
if (input_slot != -1 && slot != input_slot) {
addReplyError(c, "Cursor slot mismatch with SLOT argument");
return;
}
if (match_slot != -1 && slot != match_slot) {
/* Advance cursor to the slot matched by MATCH if required but do not go back. */
addReplyArrayLen(c, 2);
if (!skip_scan && match_slot > slot) {
sds new_cursor = sdscatfmt(sdsempty(), "0-{%s}-0", crc16_slot_table[match_slot]);
addReplyBulkSds(c, new_cursor);
} else {
addReplyBulkCString(c, "0");
}
addReplyArrayLen(c, 0);
return;
}
}
/* Scan the slot using scanGenericCommand */
sds cursor_prefix = sdscatfmt(sdsempty(), "%s-{%s}-", clusterscanFingerprint(), crc16_slot_table[slot]);
sds finished_cursor_prefix = NULL;
/* If SLOT argument was provided or implied by MATCH, don't advance to next slot then return 0 cursor.
* Else, advance to next slot for full cluster scan */
if (input_slot != -1 || match_slot != -1) {
finished_cursor_prefix = sdsnew("");
} else {
int next_slot = slot + 1;
if (next_slot >= CLUSTER_SLOTS) {
finished_cursor_prefix = sdsnew("");
} else {
finished_cursor_prefix = sdscatfmt(sdsempty(), "0-{%s}-", crc16_slot_table[next_slot]);
}
}
scanGenericCommand(c, NULL, cursor, slot, cursor_prefix, finished_cursor_prefix);
sdsfree(cursor_prefix);
sdsfree(finished_cursor_prefix);
}
src/cluster.h
+1
View File
@@ -67,6 +67,7 @@ void clusterUpdateMyselfClientIpV6(void);
void clusterUpdateMyselfHostname(void);
void clusterUpdateMyselfAnnouncedPorts(void);
void clusterUpdateMyselfHumanNodename(void);
void clusterUpdateMyselfAvailabilityZone(void);
void clusterPropagatePublish(robj *channel, robj *message, int sharded);
void clusterBroadcastPong(int target);
src/cluster_legacy.c
+207 -27
View File
@@ -101,7 +101,6 @@ const char *clusterGetMessageTypeString(int type);
void removeChannelsInSlot(unsigned int slot);
unsigned int countChannelsInSlot(unsigned int hashslot);
void clusterAddNodeToShard(const char *shard_id, clusterNode *node);
list *clusterLookupNodeListByShardId(const char *shard_id);
void clusterRemoveNodeFromShard(clusterNode *node);
int auxShardIdSetter(clusterNode *n, void *value, size_t length);
sds auxShardIdGetter(clusterNode *n, sds s);
@@ -109,6 +108,9 @@ int auxShardIdPresent(clusterNode *n);
int auxHumanNodenameSetter(clusterNode *n, void *value, size_t length);
sds auxHumanNodenameGetter(clusterNode *n, sds s);
int auxHumanNodenamePresent(clusterNode *n);
int auxAvailabilityZoneSetter(clusterNode *n, void *value, size_t length);
sds auxAvailabilityZoneGetter(clusterNode *n, sds s);
int auxAvailabilityZonePresent(clusterNode *n);
int auxAnnounceClientIpV4Setter(clusterNode *n, void *value, size_t length);
sds auxAnnounceClientIpV4Getter(clusterNode *n, sds s);
int auxAnnounceClientIpV4Present(clusterNode *n);
@@ -136,6 +138,7 @@ sds clusterEncodeOpenSlotsAuxField(int rdbflags);
int clusterDecodeOpenSlotsAuxField(int rdbflags, sds s);
static int nodeExceedsHandshakeTimeout(clusterNode *node, mstime_t now);
void clusterCommandFlushslot(client *c);
int clusterAllReplicasThinkPrimaryIsFail(void);
static inline clusterMsg *toClusterMsg(void *buf) {
clusterMsgHeader *hdr = (clusterMsgHeader *)buf;
@@ -155,6 +158,16 @@ int clusterNodeIsVotingPrimary(clusterNode *n) {
return (n->flags & CLUSTER_NODE_PRIMARY) && n->numslots;
}
/* Returns if myself is the best ranked replica in an automatic failover process.
* To avoid newly added empty replica from affecting the ranking, we will skip it. */
static inline int myselfIsBestRankedReplica(void) {
return (server.cluster->mf_end == 0 &&
getNodeReplicationOffset(myself) != 0 &&
server.cluster->failover_auth_rank == 0 &&
server.cluster->failover_failed_primary_rank == 0 &&
clusterAllReplicasThinkPrimaryIsFail());
}
int getNodeDefaultClientPort(clusterNode *n) {
return server.tls_cluster ? n->tls_port : n->tcp_port;
}
@@ -421,6 +434,7 @@ typedef enum {
af_announce_client_ipv6,
af_announce_client_tcp_port,
af_announce_client_tls_port,
af_availability_zone,
af_count, /* must be the last field */
} auxFieldIndex;
@@ -437,6 +451,7 @@ auxFieldHandler auxFieldHandlers[] = {
{"client-ipv6", auxAnnounceClientIpV6Setter, auxAnnounceClientIpV6Getter, auxAnnounceClientIpV6Present},
{"client-tcp-port", auxAnnounceClientTcpPortSetter, auxAnnounceClientTcpPortGetter, auxAnnounceClientTcpPortPresent},
{"client-tls-port", auxAnnounceClientTlsPortSetter, auxAnnounceClientTlsPortGetter, auxAnnounceClientTlsPortPresent},
{"availability-zone", auxAvailabilityZoneSetter, auxAvailabilityZoneGetter, auxAvailabilityZonePresent},
};
int auxShardIdSetter(clusterNode *n, void *value, size_t length) {
@@ -486,6 +501,22 @@ int auxHumanNodenamePresent(clusterNode *n) {
return sdslen(n->human_nodename);
}
int auxAvailabilityZoneSetter(clusterNode *n, void *value, size_t length) {
if (sdslen(n->availability_zone) == length && !strncmp(value, n->availability_zone, length)) {
return C_OK;
}
n->availability_zone = sdscpylen(n->availability_zone, value, length);
return C_OK;
}
sds auxAvailabilityZoneGetter(clusterNode *n, sds s) {
return sdscat(s, n->availability_zone);
}
int auxAvailabilityZonePresent(clusterNode *n) {
return sdslen(n->availability_zone);
}
int auxAnnounceClientIpV4Setter(clusterNode *n, void *value, size_t length) {
if (sdslen(n->announce_client_ipv4) == length && !strncmp(value, n->announce_client_ipv4, length)) {
/* Unchanged value */
@@ -1303,6 +1334,10 @@ static void updateAnnouncedHumanNodename(clusterNode *node, char *value) {
updateSdsExtensionField(&node->human_nodename, value);
}
static void updateAvailabilityZone(clusterNode *node, char *value) {
updateSdsExtensionField(&node->availability_zone, value);
}
static void updateAnnouncedClientIpV4(clusterNode *node, char *value) {
updateSdsExtensionField(&node->announce_client_ipv4, value);
}
@@ -1388,6 +1423,11 @@ void clusterUpdateMyselfHumanNodename(void) {
updateAnnouncedHumanNodename(myself, server.cluster_announce_human_nodename);
}
void clusterUpdateMyselfAvailabilityZone(void) {
if (!myself) return;
updateAvailabilityZone(myself, server.availability_zone);
}
void clusterUpdateMyselfClientIpV4(void) {
if (!myself) return;
updateAnnouncedClientIpV4(myself, server.cluster_announce_client_ipv4);
@@ -1495,6 +1535,7 @@ void clusterInit(void) {
clusterUpdateMyselfClientIpV6();
clusterUpdateMyselfHostname();
clusterUpdateMyselfHumanNodename();
clusterUpdateMyselfAvailabilityZone();
resetClusterStats();
}
@@ -1613,18 +1654,13 @@ void clusterHandleServerShutdown(bool auto_failover) {
* 4) Only for hard reset: a new Node ID is generated.
* 5) Only for hard reset: currentEpoch and configEpoch are set to 0.
* 6) The new configuration is saved and the cluster state updated.
* 7) If the node was a replica, the whole data set is flushed away. */
* 7) If the node was a replica, the whole data set is flushed away.
* 8) If it is a hard reset or the node was a replica: a new Shard ID is generated. */
void clusterReset(int hard) {
dictIterator *di;
dictEntry *de;
int j, was_replica = 0;
/* Turn into primary. */
if (nodeIsReplica(myself)) {
was_replica = 1;
clusterSetNodeAsPrimary(myself);
flushAllDataAndResetRDB(server.lazyfree_lazy_user_flush ? EMPTYDB_ASYNC : EMPTYDB_NO_FLAGS);
}
int j;
bool new_shard = false;
/* Close slots, reset manual failover state. */
clusterCloseAllSlots();
@@ -1664,22 +1700,34 @@ void clusterReset(int hard) {
/* To change the Node ID we need to remove the old name from the
* nodes table, change the ID, and re-add back with new name. */
new_shard = true;
oldname = sdsnewlen(myself->name, CLUSTER_NAMELEN);
dictDelete(server.cluster->nodes, oldname);
sdsfree(oldname);
getRandomHexChars(myself->name, CLUSTER_NAMELEN);
getRandomHexChars(myself->shard_id, CLUSTER_NAMELEN);
clusterAddNode(myself);
serverLog(LL_NOTICE, "Node hard reset, now I'm %.40s", myself->name);
} else {
/* If we were a replica, this means our shard_id is the shard_id of
* the primary node, and since now we become a new empty primary, we
* need to have our own shard_id. */
if (was_replica) getRandomHexChars(myself->shard_id, CLUSTER_NAMELEN);
new_shard = true;
}
/* Re-populate shards */
clusterAddNodeToShard(myself->shard_id, myself);
if (new_shard) {
clusterRemoveNodeFromShard(myself);
getRandomHexChars(myself->shard_id, CLUSTER_NAMELEN);
clusterAddNodeToShard(myself->shard_id, myself);
serverLog(LL_NOTICE, "Moving myself to a new shard %.40s.", myself->shard_id);
}
/* Turn into primary. clusterSetNodeAsPrimary prints the shard ID to the
* server logs, so calling it here we can print the new correct shard ID. */
if (nodeIsReplica(myself)) {
clusterSetNodeAsPrimary(myself);
flushAllDataAndResetRDB(server.lazyfree_lazy_user_flush ? EMPTYDB_ASYNC : EMPTYDB_NO_FLAGS);
}
/* Make sure to persist the new config and update the state. */
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG | CLUSTER_TODO_UPDATE_STATE | CLUSTER_TODO_FSYNC_CONFIG);
@@ -1917,6 +1965,7 @@ clusterNode *createClusterNode(char *nodename, int flags) {
node->announce_client_ipv6 = sdsempty();
node->hostname = sdsempty();
node->human_nodename = sdsempty();
node->availability_zone = sdsempty();
node->tcp_port = 0;
node->cport = 0;
node->tls_port = 0;
@@ -2155,6 +2204,7 @@ void freeClusterNode(clusterNode *n) {
/* Free these members after links are freed, as freeClusterLink may access them. */
sdsfree(n->hostname);
sdsfree(n->human_nodename);
sdsfree(n->availability_zone);
sdsfree(n->announce_client_ipv4);
sdsfree(n->announce_client_ipv6);
raxFree(n->fail_reports);
@@ -2496,6 +2546,10 @@ void markNodeAsFailing(clusterNode *node) {
/* Immediately check if the failing node is our primary node. */
if (nodeIsReplica(myself) && myself->replicaof == node) {
/* Mark my primary is FAIL so that we can bring out flags during gossip,
* so that other nodes know that my primary node has failed, so that other
* nodes know that my offset will no longer be updated. */
myself->flags |= CLUSTER_NODE_MY_PRIMARY_FAIL;
/* We can start an automatic failover as soon as possible, setting a flag
* here so that we don't need to waiting for the cron to kick in. */
clusterDoBeforeSleep(CLUSTER_TODO_HANDLE_FAILOVER);
@@ -2564,6 +2618,7 @@ void clearNodeFailureIfNeeded(clusterNode *node) {
serverLog(LL_NOTICE, "Clear FAIL state for node %.40s (%s): %s is reachable again.", node->name,
humanNodename(node), nodeIsReplica(node) ? "replica" : "primary without slots");
node->flags &= ~CLUSTER_NODE_FAIL;
if (nodeIsReplica(myself) && myself->replicaof == node) node->flags &= ~CLUSTER_NODE_MY_PRIMARY_FAIL;
clusterDoBeforeSleep(CLUSTER_TODO_UPDATE_STATE | CLUSTER_TODO_SAVE_CONFIG);
}
@@ -2578,6 +2633,7 @@ void clearNodeFailureIfNeeded(clusterNode *node) {
"Clear FAIL state for node %.40s (%s): is reachable again and nobody is serving its slots after some time.",
node->name, humanNodename(node));
node->flags &= ~CLUSTER_NODE_FAIL;
if (nodeIsReplica(myself) && myself->replicaof == node) node->flags &= ~CLUSTER_NODE_MY_PRIMARY_FAIL;
clusterDoBeforeSleep(CLUSTER_TODO_UPDATE_STATE | CLUSTER_TODO_SAVE_CONFIG);
}
}
@@ -3381,6 +3437,9 @@ static uint32_t writePingExtensions(clusterMsg *hdr, int gossipcount) {
writePortPingExtIfNonzero(&totlen, &cursor, CLUSTERMSG_EXT_TYPE_CLIENT_PORT, myself->announce_client_tcp_port);
extensions +=
writePortPingExtIfNonzero(&totlen, &cursor, CLUSTERMSG_EXT_TYPE_CLIENT_TLS_PORT, myself->announce_client_tls_port);
extensions +=
writeSdsPingExtIfNonempty(&totlen, &cursor, CLUSTERMSG_EXT_TYPE_AVAILABILITY_ZONE, myself->availability_zone);
/* Gossip forgotten nodes */
if (dictSize(server.cluster->nodes_black_list) > 0) {
@@ -3430,6 +3489,7 @@ void clusterProcessPingExtensions(clusterMsg *hdr, clusterLink *link) {
clusterNode *sender = link->node ? link->node : clusterLookupNode(hdr->sender, CLUSTER_NAMELEN);
char *ext_hostname = NULL;
char *ext_humannodename = NULL;
char *ext_availability_zone = NULL;
char *ext_clientipv4 = NULL;
char *ext_clientipv6 = NULL;
int ext_clientport = 0;
@@ -3481,6 +3541,10 @@ void clusterProcessPingExtensions(clusterMsg *hdr, clusterLink *link) {
} else if (type == CLUSTERMSG_EXT_TYPE_SHARDID) {
clusterMsgPingExtShardId *shardid_ext = (clusterMsgPingExtShardId *)&(ext->ext[0].shard_id);
ext_shardid = shardid_ext->shard_id;
} else if (type == CLUSTERMSG_EXT_TYPE_AVAILABILITY_ZONE) {
clusterMsgPingExtAvailabilityZone *availability_zone_ext =
(clusterMsgPingExtAvailabilityZone *)&(ext->ext[0].availability_zone);
ext_availability_zone = availability_zone_ext->availability_zone;
} else {
/* Unknown type, we will ignore it but log what happened. */
serverLog(LL_WARNING, "Received unknown extension type %d", type);
@@ -3499,6 +3563,7 @@ void clusterProcessPingExtensions(clusterMsg *hdr, clusterLink *link) {
updateAnnouncedClientIpV6(sender, ext_clientipv6);
updateAnnouncedClientPort(sender, ext_clientport);
updateAnnouncedClientTlsPort(sender, ext_clienttlsport);
updateAvailabilityZone(sender, ext_availability_zone);
/* If the node did not send us a shard-id extension, it means the sender
* does not support it (old version), node->shard_id is randomly generated.
* A cluster-wide consensus for the node's shard_id is not necessary.
@@ -3629,17 +3694,38 @@ int clusterIsValidPacket(clusterLink *link) {
explen = sizeof(clusterMsg) - sizeof(union clusterMsgData);
explen += (sizeof(clusterMsgDataGossip) * count);
/* Make sure that the number of gossip messages fit in the remaining
* space in the message. */
if (totlen < explen) {
serverLog(LL_WARNING,
"Received invalid %s packet with gossip count %d that exceeds "
"total packet length (%lld)",
clusterGetMessageTypeString(type), count, (unsigned long long)totlen);
return 0;
}
/* If there is extension data, which doesn't have a fixed length,
* loop through them and validate the length of it now. */
if (msg->mflags[0] & CLUSTERMSG_FLAG0_EXT_DATA) {
clusterMsgPingExt *ext = getInitialPingExt(msg, count);
while (extensions--) {
/* Make sure there is at least enough memory for the extension information so
* we can parse it. */
if ((totlen - explen) < sizeof(clusterMsgPingExt)) {
serverLog(LL_WARNING,
"Received invalid %s packet with extension data that exceeds "
"total packet length (%lld)",
clusterGetMessageTypeString(type), (unsigned long long)totlen);
return 0;
}
uint32_t extlen = getPingExtLength(ext);
if (extlen % 8 != 0) {
serverLog(LL_WARNING, "Received a %s packet without proper padding (%d bytes)",
clusterGetMessageTypeString(type), (int)extlen);
return 0;
}
/* Similar check to earlier, but we want to make sure the extension length is valid
* this time. */
if ((totlen - explen) < extlen) {
serverLog(LL_WARNING,
"Received invalid %s packet with extension data that exceeds "
@@ -3794,6 +3880,13 @@ int clusterProcessPacket(clusterLink *link) {
} else {
sender->flags &= ~CLUSTER_NODE_MULTI_MEET_SUPPORTED;
}
/* Check if the sender has marked its primary node as FAIL. */
if (flags & CLUSTER_NODE_MY_PRIMARY_FAIL) {
sender->flags |= CLUSTER_NODE_MY_PRIMARY_FAIL;
} else {
sender->flags &= ~CLUSTER_NODE_MY_PRIMARY_FAIL;
}
}
/* Update the last time we saw any data from this node. We
@@ -4028,6 +4121,9 @@ int clusterProcessPacket(clusterLink *link) {
* what are the instances really competing. */
if (sender) {
int nofailover = flags & CLUSTER_NODE_NOFAILOVER;
if (nofailover != clusterNodeIsNoFailover(sender)) {
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG);
}
sender->flags &= ~CLUSTER_NODE_NOFAILOVER;
sender->flags |= nofailover;
}
@@ -4077,13 +4173,14 @@ int clusterProcessPacket(clusterLink *link) {
/* Primary turned into a replica! Reconfigure the node. */
if (sender_claimed_primary && areInSameShard(sender_claimed_primary, sender)) {
/* `sender` was a primary and was in the same shard as its new primary */
if (nodeEpoch(sender_claimed_primary) > sender_claimed_config_epoch) {
if (nodeEpoch(sender) > sender_claimed_config_epoch ||
nodeEpoch(sender_claimed_primary) > sender_claimed_config_epoch) {
serverLog(LL_NOTICE,
"Ignore stale message from %.40s (%s) in shard %.40s;"
" gossip config epoch: %llu, current config epoch: %llu",
sender->name, humanNodename(sender), sender->shard_id,
(unsigned long long)sender_claimed_config_epoch,
(unsigned long long)nodeEpoch(sender_claimed_primary));
(unsigned long long)max(nodeEpoch(sender), nodeEpoch(sender_claimed_primary)));
/* This packet is stale so we avoid processing it anymore. Otherwise
* this may cause a primary-replica chain issue. */
return 1;
@@ -4240,6 +4337,7 @@ int clusterProcessPacket(clusterLink *link) {
for (size_t w = 0; w < CLUSTER_SLOT_WORDS && !found_new_owner; w++) {
uint64_t word;
memcpy(&word, msg->myslots + SLOT_WORD_OFFSET(w), sizeof(word));
memrev64ifbe(&word);
while (word) {
const int slot = clusterExtractSlotFromWord(&word, w);
@@ -4381,6 +4479,7 @@ void clusterWriteHandler(connection *conn) {
nwritten = connWrite(conn, (char *)msg + msg_offset, msg_len - msg_offset);
if (nwritten <= 0) {
if (nwritten == -1 && connGetState(conn) == CONN_STATE_CONNECTED) return; /* equivalent to EAGAIN */
serverLog(LL_DEBUG, "I/O error writing to node link: %s",
(nwritten == -1) ? connGetLastError(conn) : "short write");
handleLinkIOError(link);
@@ -4737,8 +4836,9 @@ void clusterSendPing(clusterLink *link, int type) {
*
* Since we have non-voting replicas that lower the probability of an entry
* to feature our node, we set the number of entries per packet as
* 10% of the total nodes we have. */
wanted = floor(dictSize(server.cluster->nodes) / 10);
* a configurable percentage (default 10%) of the total nodes we have,
* bounded by the actual number of known nodes. */
wanted = (dictSize(server.cluster->nodes) * server.cluster_message_gossip_perc / 100);
if (wanted < 3) wanted = 3;
if (wanted > freshnodes) wanted = freshnodes;
@@ -4769,11 +4869,21 @@ void clusterSendPing(clusterLink *link, int type) {
link->node->meet_sent = mstime();
}
/* Populate the gossip fields */
int maxiterations = wanted * 3;
while (freshnodes > 0 && gossipcount < wanted && maxiterations--) {
dictEntry *de = dictGetRandomKey(server.cluster->nodes);
clusterNode *this = dictGetVal(de);
/* Populate the gossip fields.
* Use dictGetSomeKeys() to sample candidates in a single batch instead
* of calling dictGetRandomKey() in a retry loop. We over-allocate to
* have enough candidates after filtering out ineligible nodes.
* dictGetSomeKeys() picks a random starting point each call, so over
* many ping rounds all nodes get even coverage without needing an
* explicit shuffle. */
int candidates_wanted = wanted + 2; /* +2 for myself and link->node */
if (candidates_wanted > (int)dictSize(server.cluster->nodes))
candidates_wanted = dictSize(server.cluster->nodes);
dictEntry **candidates = zmalloc(sizeof(dictEntry *) * candidates_wanted);
unsigned int ncandidates = dictGetSomeKeys(server.cluster->nodes, candidates, candidates_wanted);
for (unsigned int i = 0; i < ncandidates && gossipcount < wanted; i++) {
clusterNode *this = dictGetVal(candidates[i]);
/* Don't include this node: the whole packet header is about us
* already, so we just gossip about other nodes.
@@ -4791,7 +4901,6 @@ void clusterSendPing(clusterLink *link, int type) {
*/
if (this->flags & (CLUSTER_NODE_HANDSHAKE | CLUSTER_NODE_NOADDR) ||
(this->link == NULL && this->numslots == 0)) {
freshnodes--; /* Technically not correct, but saves CPU. */
continue;
}
@@ -4801,9 +4910,9 @@ void clusterSendPing(clusterLink *link, int type) {
/* Add it */
clusterSetGossipEntry(hdr, gossipcount, this);
this->last_in_ping_gossip = cluster_pings_sent;
freshnodes--;
gossipcount++;
}
zfree(candidates);
/* If there are PFAIL nodes, add them at the end. */
if (pfail_wanted) {
@@ -5122,6 +5231,16 @@ void clusterRequestFailoverAuth(void) {
* in the header to communicate the nodes receiving the message that
* they should authorized the failover even if the primary is working. */
if (server.cluster->mf_end) msgblock->data[0].msg.mflags[0] |= CLUSTERMSG_FLAG0_FORCEACK;
/* If this is an automatic failover and if myself is the best ranked replica,
* set the CLUSTERMSG_FLAG0_FORCEACK bit in the header as well.
*
* In this case, we hope that other primary nodes will not refuse to vote because
* they did not receive the FAIL message in time. */
if (server.cluster->mf_end == 0 && myselfIsBestRankedReplica()) {
msgblock->data[0].msg.mflags[0] |= CLUSTERMSG_FLAG0_FORCEACK;
}
clusterBroadcastMessage(msgblock);
clusterMsgSendBlockDecrRefCount(msgblock);
}
@@ -5211,6 +5330,7 @@ void clusterSendFailoverAuthIfNeeded(clusterNode *node, clusterMsg *request) {
for (size_t w = 0; w < CLUSTER_SLOT_WORDS; w++) {
uint64_t word;
memcpy(&word, claimed_slots + SLOT_WORD_OFFSET(w), sizeof(word));
memrev64ifbe(&word);
while (word) {
slot = clusterExtractSlotFromWord(&word, w);
@@ -5326,6 +5446,29 @@ int clusterGetFailedPrimaryRank(void) {
return rank;
}
/* Returns 1 if all replicas under my primary think the primary is in FAIL state.
*
* This is useful in automatic failover. For example, from my perspective,
* if all other replicas, including myself, both mark the primary node as FAIL,
* which means that myself and other replicas have exchanged new gossip information
* after the primary node went down, and we know the latest replication offset of
* the replicas. If a replica finds that its ranking is optimal in all cases, then
* the replica can initiate an election immediately in automatic failover without
* waiting for the delay. */
int clusterAllReplicasThinkPrimaryIsFail(void) {
serverAssert(nodeIsReplica(myself));
serverAssert(myself->replicaof);
clusterNode *primary = myself->replicaof;
for (int i = 0; i < primary->num_replicas; i++) {
if (!nodePrimaryIsFail(primary->replicas[i])) {
return 0;
}
}
return 1;
}
/* This function is called by clusterHandleReplicaFailover() in order to
* let the replica log why it is not able to failover. Sometimes there are
* not the conditions, but since the failover function is called again and
@@ -5539,10 +5682,22 @@ void clusterHandleReplicaFailover(void) {
server.cluster->failover_auth_time = now;
server.cluster->failover_auth_rank = 0;
server.cluster->failover_failed_primary_rank = 0;
/* Reset auth_age since it is outdated now and we can bypass the auth_timeout
}
if (server.cluster->mf_end == 0 && myselfIsBestRankedReplica()) {
/* If we find that myself is the best ranked replica, we can initiate the
* failover immediately. */
server.cluster->failover_auth_time = now;
serverLog(LL_NOTICE, "This is the best ranked replica and can initiate the election immediately.");
}
if (server.cluster->failover_auth_time == now) {
/* If we happen to initiate a failover (automatic or manual) immediately.
* Reset auth_age since it is outdated now and we can bypass the auth_timeout
* check in the next state and start the election ASAP. */
auth_age = 0;
}
serverLog(LL_NOTICE,
"Start of election delayed for %lld milliseconds "
"(rank #%d, primary rank #%d, offset %lld).",
@@ -5566,8 +5721,13 @@ void clusterHandleReplicaFailover(void) {
* It is also possible that we received the message that telling a
* shard is up. Update the delay if our failed_primary_rank changed.
*
* It is also possible that we received more message and then we figure
* out myself is the best ranked replica, in this case, we can initiate
* the election immediately.
*
* Not performed if this is a manual failover. */
if (server.cluster->failover_auth_sent == 0 && server.cluster->mf_end == 0) {
if (server.cluster->failover_auth_sent == 0 && server.cluster->mf_end == 0 &&
server.cluster->failover_auth_time != now) {
int newrank = clusterGetReplicaRank();
if (newrank != server.cluster->failover_auth_rank) {
long long added_delay = (newrank - server.cluster->failover_auth_rank) * (delay * 2);
@@ -5585,6 +5745,13 @@ void clusterHandleReplicaFailover(void) {
serverLog(LL_NOTICE, "Failed primary rank updated to #%d, added %lld milliseconds of delay.",
new_failed_primary_rank, added_delay);
}
if (myselfIsBestRankedReplica()) {
/* If we find that myself is the best ranked replica, we can initiate the
* failover immediately. */
server.cluster->failover_auth_time = now;
serverLog(LL_NOTICE, "Myself become the best ranked replica, initiate the election immediately.");
}
}
/* Return ASAP if we can't still start the election. */
@@ -6186,7 +6353,13 @@ void clusterBeforeSleep(void) {
/* Save the config, possibly using fsync. */
if (flags & CLUSTER_TODO_SAVE_CONFIG) {
int fsync = flags & CLUSTER_TODO_FSYNC_CONFIG;
clusterSaveConfigOrLog(fsync);
if (server.cluster_configfile_save_behavior == CLUSTER_CONFIGFILE_SAVE_BEHAVIOR_SYNC) {
/* Sync mode: exit the process if saving fails. */
clusterSaveConfigOrDie(fsync);
} else if (server.cluster_configfile_save_behavior == CLUSTER_CONFIGFILE_SAVE_BEHAVIOR_BEST_EFFORT) {
/* Best-effort mode: log (don't exit) if saving fails and wait for the next retry. */
clusterSaveConfigOrLog(fsync);
}
}
if (flags & CLUSTER_TODO_BROADCAST_ALL) {
@@ -7071,6 +7244,12 @@ void addNodeDetailsToShardReply(client *c, clusterNode *node) {
addReplyBulkCString(c, health_msg);
reply_count++;
if (sdslen(node->availability_zone) != 0) {
addReplyBulkCString(c, "availability-zone");
addReplyBulkCBuffer(c, node->availability_zone, sdslen(node->availability_zone));
reply_count++;
}
setDeferredMapLen(c, node_replylen, reply_count);
}
@@ -7882,6 +8061,7 @@ int clusterCommandSpecial(client *c) {
char new_shard_id[CLUSTER_NAMELEN];
getRandomHexChars(new_shard_id, CLUSTER_NAMELEN);
updateShardId(myself, new_shard_id);
serverLog(LL_NOTICE, "Moving myself to a new shard %.40s.", myself->shard_id);
clusterDoBeforeSleep(CLUSTER_TODO_UPDATE_STATE | CLUSTER_TODO_SAVE_CONFIG | CLUSTER_TODO_BROADCAST_ALL);
addReply(c, shared.ok);
src/cluster_legacy.h
+13
View File
@@ -65,6 +65,11 @@ typedef struct clusterLink {
#define CLUSTER_NODE_MULTI_MEET_SUPPORTED CLUSTER_NODE_LIGHT_HDR_MODULE_SUPPORTED /* This node handles multi meet packet. \
Light hdr for module and multi meet were both introduced in 8.1, \
so we could reduce the same flag value. */
#define CLUSTER_NODE_MY_PRIMARY_FAIL (1 << 13) /* myself is a replica and my primary is FAIL in my view. \
* myself will gossip this flag to other replica in the \
* shard so that the replicas can make a better ranking \
* decisions to help with the failover. */
#define CLUSTER_NODE_NULL_NAME \
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" \
"\000\000\000\000\000\000\000\000\000\000\000\000"
@@ -80,6 +85,7 @@ typedef struct clusterLink {
#define nodeSupportsExtensions(n) ((n)->flags & CLUSTER_NODE_EXTENSIONS_SUPPORTED)
#define nodeSupportsMultiMeet(n) ((n)->flags & CLUSTER_NODE_MULTI_MEET_SUPPORTED)
#define nodeInNormalState(n) (!((n)->flags & (CLUSTER_NODE_HANDSHAKE | CLUSTER_NODE_MEET | CLUSTER_NODE_PFAIL | CLUSTER_NODE_FAIL)))
#define nodePrimaryIsFail(n) ((n)->flags & CLUSTER_NODE_MY_PRIMARY_FAIL)
/* Cluster messages header */
@@ -166,6 +172,7 @@ typedef enum {
CLUSTERMSG_EXT_TYPE_CLIENT_IPV6,
CLUSTERMSG_EXT_TYPE_CLIENT_PORT,
CLUSTERMSG_EXT_TYPE_CLIENT_TLS_PORT,
CLUSTERMSG_EXT_TYPE_AVAILABILITY_ZONE,
} clusterMsgPingtypes;
/* Helper function for making sure extensions are eight byte aligned. */
@@ -179,6 +186,10 @@ typedef struct {
char human_nodename[1]; /* The announced nodename, ends with \0. */
} clusterMsgPingExtHumanNodename;
typedef struct {
char availability_zone[1]; /* The availability zone, ends with \0. */
} clusterMsgPingExtAvailabilityZone;
typedef struct {
char name[CLUSTER_NAMELEN]; /* Node name. */
uint64_t ttl; /* Remaining time to blacklist the node, in seconds. */
@@ -219,6 +230,7 @@ typedef struct {
clusterMsgPingExtClientIpV6 announce_client_ipv6;
clusterMsgPingExtClientPort announce_client_port;
clusterMsgPingExtClientTlsPort announce_client_tls_port;
clusterMsgPingExtAvailabilityZone availability_zone;
} ext[]; /* Actual extension information, formatted so that the data is 8
* byte aligned, regardless of its content. */
} clusterMsgPingExt;
@@ -392,6 +404,7 @@ struct _clusterNode {
sds announce_client_ipv6; /* IPv6 for clients only. */
sds hostname; /* The known hostname for this node */
sds human_nodename; /* The known human readable nodename for this node */
sds availability_zone; /* The known availability zone for this node */
int tcp_port; /* Latest known clients TCP port. */
int tls_port; /* Latest known clients TLS port */
int cport; /* Latest known cluster port of this node. */
src/cluster_migrateslots.c
+6 -5
View File
@@ -1410,9 +1410,9 @@ sds generateSyncSlotsEstablishCommand(slotMigrationJob *job) {
listRewind(job->slot_ranges, &li);
while ((ln = listNext(&li))) {
slotRange *range = (slotRange *)ln->value;
sdscatfmt(result, "$%i\r\n%i\r\n$%i\r\n%i\r\n",
digits10(range->start_slot), range->start_slot,
digits10(range->end_slot), range->end_slot);
result = sdscatfmt(result, "$%i\r\n%i\r\n$%i\r\n%i\r\n",
digits10(range->start_slot), range->start_slot,
digits10(range->end_slot), range->end_slot);
}
return result;
}
@@ -1448,8 +1448,8 @@ int slotExportTryDoPause(slotMigrationJob *job) {
return C_ERR;
}
serverLog(LL_NOTICE,
"Pausing writes to allow slot migration %s to finalize failover.",
job->description);
"Pausing writes (remaining_repl_size is %lld) to allow slot migration %s to finalize failover.",
job->client->reply_bytes, job->description);
job->mf_end = mstime() + server.cluster_mf_timeout * CLUSTER_MF_PAUSE_MULT;
pauseActions(PAUSE_DURING_SLOT_MIGRATION, job->mf_end,
PAUSE_ACTIONS_CLIENT_WRITE_SET);
@@ -2145,6 +2145,7 @@ void resetSlotMigrationJob(slotMigrationJob *job) {
/* Only one of client or conn should be set. */
serverAssert(!job->client || !job->conn);
if (job->client) {
job->client->slot_migration_job = NULL;
freeClientAsync(job->client);
job->client = NULL;
} else if (job->conn) {
src/commandlog.c
+5 -3
View File
@@ -43,9 +43,11 @@ static commandlogEntry *commandlogCreateEntry(client *c, robj **argv, int argc,
ce->argv[j] =
createObject(OBJ_STRING, sdscatprintf(sdsempty(), "... (%d more arguments)", argc - ceargc + 1));
} else {
/* Trim too long strings as well... */
if (argv[j]->type == OBJ_STRING && sdsEncodedObject(argv[j]) &&
sdslen(objectGetVal(argv[j])) > COMMANDLOG_ENTRY_MAX_STRING) {
if (clientCommandArgShouldBeRedacted(c, j)) {
ce->argv[j] = shared.redacted;
/* Trim too long strings as well... */
} else if (argv[j]->type == OBJ_STRING && sdsEncodedObject(argv[j]) &&
sdslen(objectGetVal(argv[j])) > COMMANDLOG_ENTRY_MAX_STRING) {
sds s = sdsnewlen(objectGetVal(argv[j]), COMMANDLOG_ENTRY_MAX_STRING);
s = sdscatprintf(s, "... (%lu more bytes)",
src/commands.def
+115 -29
View File
@@ -1007,7 +1007,7 @@ struct COMMAND_ARG CLUSTER_SETSLOT_Args[] = {
#ifndef SKIP_CMD_HISTORY_TABLE
/* CLUSTER SHARDS history */
commandHistory CLUSTER_SHARDS_History[] = {
{"9.1.0","Added shard id field to CLUSTER SHARDS response"},
{"9.1.0","Added shard level `id` field and node level `availability-zone` field."},
};
#endif
@@ -1104,6 +1104,7 @@ struct COMMAND_ARG CLUSTER_SLOT_STATS_Args[] = {
commandHistory CLUSTER_SLOTS_History[] = {
{"4.0.0","Added node IDs."},
{"7.0.0","Added additional networking metadata field."},
{"9.1.0","Added `availability-zone` field inside the networking metadata."},
};
#endif
@@ -1147,11 +1148,11 @@ struct COMMAND_STRUCT CLUSTER_Subcommands[] = {
{MAKE_CMD("delslots","Sets hash slots as unbound for a node.","O(N) where N is the total number of hash slot arguments","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_DELSLOTS_History,0,CLUSTER_DELSLOTS_Tips,0,clusterCommand,-3,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_DELSLOTS_Keyspecs,0,NULL,1),.args=CLUSTER_DELSLOTS_Args},
{MAKE_CMD("delslotsrange","Sets hash slot ranges as unbound for a node.","O(N) where N is the total number of the slots between the start slot and end slot arguments.","7.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_DELSLOTSRANGE_History,0,CLUSTER_DELSLOTSRANGE_Tips,0,clusterCommand,-4,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_DELSLOTSRANGE_Keyspecs,0,NULL,1),.args=CLUSTER_DELSLOTSRANGE_Args},
{MAKE_CMD("failover","Forces a replica to perform a manual failover of its primary.","O(1)","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_FAILOVER_History,0,CLUSTER_FAILOVER_Tips,0,clusterCommand,-2,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_FAILOVER_Keyspecs,0,NULL,1),.args=CLUSTER_FAILOVER_Args},
{MAKE_CMD("flushslot","Remove all keys from the target slot.","O(N) where N is the number of keys in the target slot","9.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_FLUSHSLOT_History,0,CLUSTER_FLUSHSLOT_Tips,0,clusterCommand,-3,CMD_WRITE|CMD_NO_ASYNC_LOADING|CMD_ADMIN,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_KEYSPACE|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,CLUSTER_FLUSHSLOT_Keyspecs,0,NULL,2),.args=CLUSTER_FLUSHSLOT_Args},
{MAKE_CMD("flushslot","Removes all keys from the target slot.","O(N) where N is the number of keys in the target slot","9.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_FLUSHSLOT_History,0,CLUSTER_FLUSHSLOT_Tips,0,clusterCommand,-3,CMD_WRITE|CMD_NO_ASYNC_LOADING|CMD_ADMIN,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_KEYSPACE|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,CLUSTER_FLUSHSLOT_Keyspecs,0,NULL,2),.args=CLUSTER_FLUSHSLOT_Args},
{MAKE_CMD("flushslots","Deletes all slots information from a node.","O(1)","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_FLUSHSLOTS_History,0,CLUSTER_FLUSHSLOTS_Tips,0,clusterCommand,2,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_FLUSHSLOTS_Keyspecs,0,NULL,0)},
{MAKE_CMD("forget","Removes a node from the nodes table.","O(1)","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_FORGET_History,0,CLUSTER_FORGET_Tips,0,clusterCommand,3,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_FORGET_Keyspecs,0,NULL,1),.args=CLUSTER_FORGET_Args},
{MAKE_CMD("getkeysinslot","Returns the key names in a hash slot.","O(N) where N is the number of requested keys","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_GETKEYSINSLOT_History,0,CLUSTER_GETKEYSINSLOT_Tips,1,clusterCommand,4,CMD_STALE,ACL_CATEGORY_SLOW,NULL,CLUSTER_GETKEYSINSLOT_Keyspecs,0,NULL,2),.args=CLUSTER_GETKEYSINSLOT_Args},
{MAKE_CMD("getslotmigrations","Get the status of ongoing and recently finished slot import and export operations.","O(N), where N is the number of active slot import and export jobs.","9.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_GETSLOTMIGRATIONS_History,0,CLUSTER_GETSLOTMIGRATIONS_Tips,0,clusterCommand,2,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_GETSLOTMIGRATIONS_Keyspecs,0,NULL,0)},
{MAKE_CMD("getslotmigrations","Gets the status of ongoing and recently finished slot import and export operations.","O(N), where N is the number of active slot import and export jobs.","9.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_GETSLOTMIGRATIONS_History,0,CLUSTER_GETSLOTMIGRATIONS_Tips,0,clusterCommand,2,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_GETSLOTMIGRATIONS_Keyspecs,0,NULL,0)},
{MAKE_CMD("help","Returns helpful text about the different subcommands.","O(1)","5.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_HELP_History,0,CLUSTER_HELP_Tips,0,clusterCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SLOW,NULL,CLUSTER_HELP_Keyspecs,0,NULL,0)},
{MAKE_CMD("info","Returns information about the state of a node.","O(1)","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_INFO_History,0,CLUSTER_INFO_Tips,1,clusterCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SLOW,NULL,CLUSTER_INFO_Keyspecs,0,NULL,0)},
{MAKE_CMD("keyslot","Returns the hash slot for a key.","O(N) where N is the number of bytes in the key","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_KEYSLOT_History,0,CLUSTER_KEYSLOT_Tips,0,clusterKeySlotCommand,3,CMD_STALE,ACL_CATEGORY_SLOW,NULL,CLUSTER_KEYSLOT_Keyspecs,0,NULL,1),.args=CLUSTER_KEYSLOT_Args},
@@ -1162,7 +1163,7 @@ struct COMMAND_STRUCT CLUSTER_Subcommands[] = {
{MAKE_CMD("myshardid","Returns the shard ID of a node.","O(1)","7.2.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_MYSHARDID_History,0,CLUSTER_MYSHARDID_Tips,1,clusterCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SLOW,NULL,CLUSTER_MYSHARDID_Keyspecs,0,NULL,0)},
{MAKE_CMD("nodes","Returns the cluster configuration for a node.","O(N) where N is the total number of Cluster nodes","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_NODES_History,0,CLUSTER_NODES_Tips,1,clusterCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SLOW,NULL,CLUSTER_NODES_Keyspecs,0,NULL,0)},
{MAKE_CMD("replicas","Lists the replica nodes of a primary node.","O(N) where N is the number of replicas.","5.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_REPLICAS_History,0,CLUSTER_REPLICAS_Tips,1,clusterCommand,3,CMD_ADMIN|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_REPLICAS_Keyspecs,0,NULL,1),.args=CLUSTER_REPLICAS_Args},
{MAKE_CMD("replicate","Configure a node as replica of a primary node or detach a replica from its primary.","O(1)","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_REPLICATE_History,1,CLUSTER_REPLICATE_Tips,0,clusterCommand,-3,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_REPLICATE_Keyspecs,0,NULL,1),.args=CLUSTER_REPLICATE_Args},
{MAKE_CMD("replicate","Configures a node as replica of a primary node or detaches a replica from its primary.","O(1)","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_REPLICATE_History,1,CLUSTER_REPLICATE_Tips,0,clusterCommand,-3,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_REPLICATE_Keyspecs,0,NULL,1),.args=CLUSTER_REPLICATE_Args},
{MAKE_CMD("reset","Resets a node.","O(N) where N is the number of known nodes. The command may execute a FLUSHALL as a side effect.","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_RESET_History,0,CLUSTER_RESET_Tips,0,clusterCommand,-2,CMD_ADMIN|CMD_STALE|CMD_NOSCRIPT,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_RESET_Keyspecs,0,NULL,1),.args=CLUSTER_RESET_Args},
{MAKE_CMD("saveconfig","Forces a node to save the cluster configuration to disk.","O(1)","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_SAVECONFIG_History,0,CLUSTER_SAVECONFIG_Tips,0,clusterCommand,2,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_SAVECONFIG_Keyspecs,0,NULL,0)},
{MAKE_CMD("set-config-epoch","Sets the configuration epoch for a new node.","O(1)","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_SET_CONFIG_EPOCH_History,0,CLUSTER_SET_CONFIG_EPOCH_Tips,0,clusterCommand,3,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_SET_CONFIG_EPOCH_Keyspecs,0,NULL,1),.args=CLUSTER_SET_CONFIG_EPOCH_Args},
@@ -1170,7 +1171,7 @@ struct COMMAND_STRUCT CLUSTER_Subcommands[] = {
{MAKE_CMD("shards","Returns the mapping of cluster slots to shards.","O(N) where N is the total number of cluster nodes","7.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_SHARDS_History,1,CLUSTER_SHARDS_Tips,1,clusterCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SLOW,NULL,CLUSTER_SHARDS_Keyspecs,0,NULL,0)},
{MAKE_CMD("slaves","Lists the replica nodes of a primary node.","O(N) where N is the number of replicas.","3.0.0",CMD_DOC_DEPRECATED,"`CLUSTER REPLICAS`","5.0.0","cluster",COMMAND_GROUP_CLUSTER,CLUSTER_SLAVES_History,0,CLUSTER_SLAVES_Tips,1,clusterCommand,3,CMD_ADMIN|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_SLAVES_Keyspecs,0,NULL,1),.args=CLUSTER_SLAVES_Args},
{MAKE_CMD("slot-stats","Return an array of slot usage statistics for slots assigned to the current node.","O(N) where N is the total number of slots based on arguments. O(N*log(N)) with ORDERBY subcommand.","8.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_SLOT_STATS_History,0,CLUSTER_SLOT_STATS_Tips,2,clusterSlotStatsCommand,-4,CMD_STALE|CMD_LOADING,ACL_CATEGORY_SLOW,NULL,CLUSTER_SLOT_STATS_Keyspecs,0,NULL,1),.args=CLUSTER_SLOT_STATS_Args},
{MAKE_CMD("slots","Returns the mapping of cluster slots to nodes.","O(N) where N is the total number of Cluster nodes","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_SLOTS_History,2,CLUSTER_SLOTS_Tips,1,clusterCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SLOW,NULL,CLUSTER_SLOTS_Keyspecs,0,NULL,0)},
{MAKE_CMD("slots","Returns the mapping of cluster slots to nodes.","O(N) where N is the total number of Cluster nodes","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_SLOTS_History,3,CLUSTER_SLOTS_Tips,1,clusterCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SLOW,NULL,CLUSTER_SLOTS_Keyspecs,0,NULL,0)},
{MAKE_CMD("syncslots","A container for internal slot migration commands.","Depends on subcommand.","9.0.0",CMD_DOC_SYSCMD,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_SYNCSLOTS_History,0,CLUSTER_SYNCSLOTS_Tips,0,clusterCommand,-3,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_STALE|CMD_MAY_REPLICATE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLUSTER_SYNCSLOTS_Keyspecs,0,NULL,0)},
{0}
};
@@ -1192,6 +1193,38 @@ struct COMMAND_STRUCT CLUSTER_Subcommands[] = {
#define CLUSTER_Keyspecs NULL
#endif
/********** CLUSTERSCAN ********************/
#ifndef SKIP_CMD_HISTORY_TABLE
/* CLUSTERSCAN history */
#define CLUSTERSCAN_History NULL
#endif
#ifndef SKIP_CMD_TIPS_TABLE
/* CLUSTERSCAN tips */
const char *CLUSTERSCAN_Tips[] = {
"nondeterministic_output",
"request_policy:special",
"response_policy:special",
};
#endif
#ifndef SKIP_CMD_KEY_SPECS_TABLE
/* CLUSTERSCAN key specs */
keySpec CLUSTERSCAN_Keyspecs[1] = {
{NULL,CMD_KEY_NOT_KEY,KSPEC_BS_INDEX,.bs.index={1},KSPEC_FK_RANGE,.fk.range={0,1,0}}
};
#endif
/* CLUSTERSCAN argument table */
struct COMMAND_ARG CLUSTERSCAN_Args[] = {
{MAKE_ARG("cursor",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("match-pattern",ARG_TYPE_PATTERN,-1,"MATCH",NULL,NULL,CMD_ARG_OPTIONAL,0,NULL)},
{MAKE_ARG("count",ARG_TYPE_INTEGER,-1,"COUNT",NULL,NULL,CMD_ARG_OPTIONAL,0,NULL)},
{MAKE_ARG("type",ARG_TYPE_STRING,-1,"TYPE",NULL,NULL,CMD_ARG_OPTIONAL,0,NULL)},
{MAKE_ARG("slot",ARG_TYPE_INTEGER,-1,"SLOT",NULL,NULL,CMD_ARG_OPTIONAL,0,NULL)},
};
/********** READONLY ********************/
#ifndef SKIP_CMD_HISTORY_TABLE
@@ -1870,7 +1903,7 @@ struct COMMAND_STRUCT CLIENT_Subcommands[] = {
{MAKE_CMD("getredir","Returns the client ID to which the connection's tracking notifications are redirected.","O(1)","6.0.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_GETREDIR_History,0,CLIENT_GETREDIR_Tips,0,clientGetredirCommand,2,CMD_ALLOW_BUSY|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_CONNECTION|ACL_CATEGORY_SLOW,NULL,CLIENT_GETREDIR_Keyspecs,0,NULL,0)},
{MAKE_CMD("help","Returns helpful text about the different subcommands.","O(1)","5.0.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_HELP_History,0,CLIENT_HELP_Tips,0,clientHelpCommand,2,CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_CONNECTION|ACL_CATEGORY_SLOW,NULL,CLIENT_HELP_Keyspecs,0,NULL,0)},
{MAKE_CMD("id","Returns the unique client ID of the connection.","O(1)","5.0.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_ID_History,0,CLIENT_ID_Tips,0,clientIDCommand,2,CMD_ALLOW_BUSY|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_CONNECTION|ACL_CATEGORY_SLOW,NULL,CLIENT_ID_Keyspecs,0,NULL,0)},
{MAKE_CMD("import-source","Mark this client as an import source when server is in import mode.","O(1)","8.1.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_IMPORT_SOURCE_History,0,CLIENT_IMPORT_SOURCE_Tips,0,clientImportSourceCommand,3,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE,ACL_CATEGORY_CONNECTION|ACL_CATEGORY_SLOW,NULL,CLIENT_IMPORT_SOURCE_Keyspecs,0,NULL,1),.args=CLIENT_IMPORT_SOURCE_Args},
{MAKE_CMD("import-source","Marks this client as an import source when the server is in import mode.","O(1)","8.1.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_IMPORT_SOURCE_History,0,CLIENT_IMPORT_SOURCE_Tips,0,clientImportSourceCommand,3,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE,ACL_CATEGORY_CONNECTION|ACL_CATEGORY_SLOW,NULL,CLIENT_IMPORT_SOURCE_Keyspecs,0,NULL,1),.args=CLIENT_IMPORT_SOURCE_Args},
{MAKE_CMD("info","Returns information about the connection.","O(1)","6.2.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_INFO_History,0,CLIENT_INFO_Tips,1,clientInfoCommand,2,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_CONNECTION|ACL_CATEGORY_SLOW,NULL,CLIENT_INFO_Keyspecs,0,NULL,0)},
{MAKE_CMD("kill","Terminates open connections.","O(N) where N is the number of client connections","2.4.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_KILL_History,9,CLIENT_KILL_Tips,0,clientKillCommand,-3,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_ADMIN|ACL_CATEGORY_CONNECTION|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLIENT_KILL_Keyspecs,0,NULL,1),.args=CLIENT_KILL_Args},
{MAKE_CMD("list","Lists open connections.","O(N) where N is the number of client connections","2.4.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_LIST_History,9,CLIENT_LIST_Tips,1,clientListCommand,-2,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_ADMIN|ACL_CATEGORY_CONNECTION|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,CLIENT_LIST_Keyspecs,0,NULL,27),.args=CLIENT_LIST_Args},
@@ -7471,7 +7504,7 @@ struct COMMAND_ARG COMMANDLOG_RESET_Args[] = {
/* COMMANDLOG command table */
struct COMMAND_STRUCT COMMANDLOG_Subcommands[] = {
{MAKE_CMD("get","Returns the specified command log's entries.","O(N) where N is the number of entries returned","8.1.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,COMMANDLOG_GET_History,0,COMMANDLOG_GET_Tips,2,commandlogCommand,4,CMD_ADMIN|CMD_LOADING|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,COMMANDLOG_GET_Keyspecs,0,NULL,2),.args=COMMANDLOG_GET_Args},
{MAKE_CMD("help","Show helpful text about the different subcommands","O(1)","8.1.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,COMMANDLOG_HELP_History,0,COMMANDLOG_HELP_Tips,0,commandlogCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SLOW,NULL,COMMANDLOG_HELP_Keyspecs,0,NULL,0)},
{MAKE_CMD("help","Shows helpful text about the different subcommands.","O(1)","8.1.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,COMMANDLOG_HELP_History,0,COMMANDLOG_HELP_Tips,0,commandlogCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SLOW,NULL,COMMANDLOG_HELP_Keyspecs,0,NULL,0)},
{MAKE_CMD("len","Returns the number of entries in the specified type of command log.","O(1)","8.1.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,COMMANDLOG_LEN_History,0,COMMANDLOG_LEN_Tips,3,commandlogCommand,3,CMD_ADMIN|CMD_LOADING|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,COMMANDLOG_LEN_Keyspecs,0,NULL,1),.args=COMMANDLOG_LEN_Args},
{MAKE_CMD("reset","Clears all entries from the specified type of command log.","O(N) where N is the number of entries in the commandlog","8.1.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,COMMANDLOG_RESET_History,0,COMMANDLOG_RESET_Tips,2,commandlogCommand,3,CMD_ADMIN|CMD_LOADING|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,COMMANDLOG_RESET_Keyspecs,0,NULL,1),.args=COMMANDLOG_RESET_Args},
{0}
@@ -8661,7 +8694,7 @@ const char *SLOWLOG_RESET_Tips[] = {
/* SLOWLOG command table */
struct COMMAND_STRUCT SLOWLOG_Subcommands[] = {
{MAKE_CMD("get","Returns the slow log's entries.","O(N) where N is the number of entries returned","2.2.12",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,SLOWLOG_GET_History,1,SLOWLOG_GET_Tips,2,slowlogCommand,-2,CMD_ADMIN|CMD_LOADING|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,SLOWLOG_GET_Keyspecs,0,NULL,1),.args=SLOWLOG_GET_Args},
{MAKE_CMD("help","Show helpful text about the different subcommands","O(1)","6.2.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,SLOWLOG_HELP_History,0,SLOWLOG_HELP_Tips,0,slowlogCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SLOW,NULL,SLOWLOG_HELP_Keyspecs,0,NULL,0)},
{MAKE_CMD("help","Shows helpful text about the different subcommands.","O(1)","6.2.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,SLOWLOG_HELP_History,0,SLOWLOG_HELP_Tips,0,slowlogCommand,2,CMD_LOADING|CMD_STALE,ACL_CATEGORY_SLOW,NULL,SLOWLOG_HELP_Keyspecs,0,NULL,0)},
{MAKE_CMD("len","Returns the number of entries in the slow log.","O(1)","2.2.12",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,SLOWLOG_LEN_History,0,SLOWLOG_LEN_Tips,3,slowlogCommand,2,CMD_ADMIN|CMD_LOADING|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,SLOWLOG_LEN_Keyspecs,0,NULL,0)},
{MAKE_CMD("reset","Clears all entries from the slow log.","O(N) where N is the number of entries in the slowlog","2.2.12",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,SLOWLOG_RESET_History,0,SLOWLOG_RESET_Tips,2,slowlogCommand,2,CMD_ADMIN|CMD_LOADING|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,SLOWLOG_RESET_Keyspecs,0,NULL,0)},
{0}
@@ -11435,6 +11468,57 @@ struct COMMAND_ARG MSET_Args[] = {
{MAKE_ARG("data",ARG_TYPE_BLOCK,-1,NULL,NULL,NULL,CMD_ARG_MULTIPLE,2,NULL),.subargs=MSET_data_Subargs},
};
/********** MSETEX ********************/
#ifndef SKIP_CMD_HISTORY_TABLE
/* MSETEX history */
#define MSETEX_History NULL
#endif
#ifndef SKIP_CMD_TIPS_TABLE
/* MSETEX tips */
const char *MSETEX_Tips[] = {
"request_policy:multi_shard",
"response_policy:all_succeeded",
};
#endif
#ifndef SKIP_CMD_KEY_SPECS_TABLE
/* MSETEX key specs */
keySpec MSETEX_Keyspecs[1] = {
{NULL,CMD_KEY_OW|CMD_KEY_UPDATE,KSPEC_BS_INDEX,.bs.index={1},KSPEC_FK_KEYNUM,.fk.keynum={0,1,2}}
};
#endif
/* MSETEX data argument table */
struct COMMAND_ARG MSETEX_data_Subargs[] = {
{MAKE_ARG("key",ARG_TYPE_KEY,0,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("value",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
};
/* MSETEX condition argument table */
struct COMMAND_ARG MSETEX_condition_Subargs[] = {
{MAKE_ARG("nx",ARG_TYPE_PURE_TOKEN,-1,"NX",NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("xx",ARG_TYPE_PURE_TOKEN,-1,"XX",NULL,NULL,CMD_ARG_NONE,0,NULL)},
};
/* MSETEX expiration argument table */
struct COMMAND_ARG MSETEX_expiration_Subargs[] = {
{MAKE_ARG("seconds",ARG_TYPE_INTEGER,-1,"EX",NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("milliseconds",ARG_TYPE_INTEGER,-1,"PX",NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("unix-time-seconds",ARG_TYPE_UNIX_TIME,-1,"EXAT",NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("unix-time-milliseconds",ARG_TYPE_UNIX_TIME,-1,"PXAT",NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("keepttl",ARG_TYPE_PURE_TOKEN,-1,"KEEPTTL",NULL,NULL,CMD_ARG_NONE,0,NULL)},
};
/* MSETEX argument table */
struct COMMAND_ARG MSETEX_Args[] = {
{MAKE_ARG("numkeys",ARG_TYPE_INTEGER,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("data",ARG_TYPE_BLOCK,-1,NULL,NULL,NULL,CMD_ARG_MULTIPLE,2,NULL),.subargs=MSETEX_data_Subargs},
{MAKE_ARG("condition",ARG_TYPE_ONEOF,-1,NULL,NULL,NULL,CMD_ARG_OPTIONAL,2,NULL),.subargs=MSETEX_condition_Subargs},
{MAKE_ARG("expiration",ARG_TYPE_ONEOF,-1,NULL,NULL,NULL,CMD_ARG_OPTIONAL,5,NULL),.subargs=MSETEX_expiration_Subargs},
};
/********** MSETNX ********************/
#ifndef SKIP_CMD_HISTORY_TABLE
@@ -11773,6 +11857,7 @@ struct COMMAND_STRUCT serverCommandTable[] = {
/* cluster */
{MAKE_CMD("asking","Signals that a cluster client is following an -ASK redirect.","O(1)","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,ASKING_History,0,ASKING_Tips,0,askingCommand,1,CMD_FAST,ACL_CATEGORY_CONNECTION|ACL_CATEGORY_FAST,NULL,ASKING_Keyspecs,0,NULL,0)},
{MAKE_CMD("cluster","A container for Cluster commands.","Depends on subcommand.","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTER_History,0,CLUSTER_Tips,0,NULL,-2,0,ACL_CATEGORY_SLOW,NULL,CLUSTER_Keyspecs,0,NULL,0),.subcommands=CLUSTER_Subcommands},
{MAKE_CMD("clusterscan","Iterates over the keys in the cluster.","O(N) where N is the number of elements returned.","9.1.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,CLUSTERSCAN_History,0,CLUSTERSCAN_Tips,3,clusterscanCommand,-2,CMD_READONLY|CMD_TOUCHES_ARBITRARY_KEYS,ACL_CATEGORY_KEYSPACE|ACL_CATEGORY_READ|ACL_CATEGORY_SLOW,NULL,CLUSTERSCAN_Keyspecs,1,NULL,5),.args=CLUSTERSCAN_Args},
{MAKE_CMD("readonly","Enables read-only queries for a connection to a Valkey replica node.","O(1)","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,READONLY_History,0,READONLY_Tips,0,readonlyCommand,1,CMD_FAST|CMD_LOADING|CMD_STALE,ACL_CATEGORY_CONNECTION|ACL_CATEGORY_FAST,NULL,READONLY_Keyspecs,0,NULL,0)},
{MAKE_CMD("readwrite","Enables read-write queries for a connection to a Valkey replica node.","O(1)","3.0.0",CMD_DOC_NONE,NULL,NULL,"cluster",COMMAND_GROUP_CLUSTER,READWRITE_History,0,READWRITE_Tips,0,readwriteCommand,1,CMD_FAST|CMD_LOADING|CMD_STALE,ACL_CATEGORY_CONNECTION|ACL_CATEGORY_FAST,NULL,READWRITE_Keyspecs,0,NULL,0)},
/* connection */
@@ -11828,13 +11913,13 @@ struct COMMAND_STRUCT serverCommandTable[] = {
/* hash */
{MAKE_CMD("hdel","Deletes one or more fields and their values from a hash. Deletes the hash if no fields remain.","O(N) where N is the number of fields to be removed.","2.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HDEL_History,1,HDEL_Tips,0,hdelCommand,-3,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HDEL_Keyspecs,1,NULL,2),.args=HDEL_Args},
{MAKE_CMD("hexists","Determines whether a field exists in a hash.","O(1)","2.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HEXISTS_History,0,HEXISTS_Tips,0,hexistsCommand,3,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_READ,NULL,HEXISTS_Keyspecs,1,NULL,2),.args=HEXISTS_Args},
{MAKE_CMD("hexpire","Set expiry time on hash fields.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HEXPIRE_History,0,HEXPIRE_Tips,0,hexpireCommand,-6,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HEXPIRE_Keyspecs,1,NULL,4),.args=HEXPIRE_Args},
{MAKE_CMD("hexpireat","Set expiry time on hash fields.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HEXPIREAT_History,0,HEXPIREAT_Tips,0,hexpireatCommand,-6,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HEXPIREAT_Keyspecs,1,NULL,4),.args=HEXPIREAT_Args},
{MAKE_CMD("hexpiretime","Returns Unix timestamps in seconds since the epoch at which the given key's field(s) will expire","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HEXPIRETIME_History,0,HEXPIRETIME_Tips,0,hexpiretimeCommand,-5,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_READ,NULL,HEXPIRETIME_Keyspecs,1,NULL,2),.args=HEXPIRETIME_Args},
{MAKE_CMD("hexpire","Sets expiry time on hash fields.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HEXPIRE_History,0,HEXPIRE_Tips,0,hexpireCommand,-6,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HEXPIRE_Keyspecs,1,NULL,4),.args=HEXPIRE_Args},
{MAKE_CMD("hexpireat","Sets expiry time on hash fields.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HEXPIREAT_History,0,HEXPIREAT_Tips,0,hexpireatCommand,-6,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HEXPIREAT_Keyspecs,1,NULL,4),.args=HEXPIREAT_Args},
{MAKE_CMD("hexpiretime","Returns Unix timestamps in seconds since the epoch at which the given key's field(s) will expire.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HEXPIRETIME_History,0,HEXPIRETIME_Tips,0,hexpiretimeCommand,-5,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_READ,NULL,HEXPIRETIME_Keyspecs,1,NULL,2),.args=HEXPIRETIME_Args},
{MAKE_CMD("hget","Returns the value of a field in a hash.","O(1)","2.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HGET_History,0,HGET_Tips,0,hgetCommand,3,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_READ,NULL,HGET_Keyspecs,1,NULL,2),.args=HGET_Args},
{MAKE_CMD("hgetall","Returns all fields and values in a hash.","O(N) where N is the size of the hash.","2.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HGETALL_History,0,HGETALL_Tips,1,hgetallCommand,2,CMD_READONLY,ACL_CATEGORY_HASH|ACL_CATEGORY_READ|ACL_CATEGORY_SLOW,NULL,HGETALL_Keyspecs,1,NULL,1),.args=HGETALL_Args},
{MAKE_CMD("hgetdel","Returns the values of one or more fields and deletes them from a hash.","O(N) where N is the number of fields to be retrieved and deleted.","9.1.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HGETDEL_History,0,HGETDEL_Tips,0,hgetdelCommand,-5,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HGETDEL_Keyspecs,1,NULL,2),.args=HGETDEL_Args},
{MAKE_CMD("hgetex","Get the value of one or more fields of a given hash key, and optionally set their expiration time or time-to-live (TTL).","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HGETEX_History,0,HGETEX_Tips,0,hgetexCommand,-5,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HGETEX_Keyspecs,1,NULL,3),.args=HGETEX_Args},
{MAKE_CMD("hgetex","Gets the value of one or more fields of a given hash key, and optionally sets their expiration time or time-to-live (TTL).","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HGETEX_History,0,HGETEX_Tips,0,hgetexCommand,-5,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HGETEX_Keyspecs,1,NULL,3),.args=HGETEX_Args},
{MAKE_CMD("hincrby","Increments the integer value of a field in a hash by a number. Uses 0 as initial value if the field doesn't exist.","O(1)","2.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HINCRBY_History,0,HINCRBY_Tips,0,hincrbyCommand,4,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HINCRBY_Keyspecs,1,NULL,3),.args=HINCRBY_Args},
{MAKE_CMD("hincrbyfloat","Increments the floating point value of a field by a number. Uses 0 as initial value if the field doesn't exist.","O(1)","2.6.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HINCRBYFLOAT_History,0,HINCRBYFLOAT_Tips,0,hincrbyfloatCommand,4,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HINCRBYFLOAT_Keyspecs,1,NULL,3),.args=HINCRBYFLOAT_Args},
{MAKE_CMD("hkeys","Returns all fields in a hash.","O(N) where N is the size of the hash.","2.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HKEYS_History,0,HKEYS_Tips,1,hkeysCommand,2,CMD_READONLY,ACL_CATEGORY_HASH|ACL_CATEGORY_READ|ACL_CATEGORY_SLOW,NULL,HKEYS_Keyspecs,1,NULL,1),.args=HKEYS_Args},
@@ -11842,14 +11927,14 @@ struct COMMAND_STRUCT serverCommandTable[] = {
{MAKE_CMD("hmget","Returns the values of all fields in a hash.","O(N) where N is the number of fields being requested.","2.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HMGET_History,0,HMGET_Tips,0,hmgetCommand,-3,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_READ,NULL,HMGET_Keyspecs,1,NULL,2),.args=HMGET_Args},
{MAKE_CMD("hmset","Sets the values of multiple fields.","O(N) where N is the number of fields being set.","2.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HMSET_History,0,HMSET_Tips,0,hsetCommand,-4,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HMSET_Keyspecs,1,NULL,2),.args=HMSET_Args},
{MAKE_CMD("hpersist","Remove the existing expiration on a hash key's field(s).","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HPERSIST_History,0,HPERSIST_Tips,0,hpersistCommand,-5,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HPERSIST_Keyspecs,1,NULL,2),.args=HPERSIST_Args},
{MAKE_CMD("hpexpire","Set expiry time on hash object.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HPEXPIRE_History,0,HPEXPIRE_Tips,0,hpexpireCommand,-6,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HPEXPIRE_Keyspecs,1,NULL,4),.args=HPEXPIRE_Args},
{MAKE_CMD("hpexpireat","Set expiration time on hash field.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HPEXPIREAT_History,0,HPEXPIREAT_Tips,0,hpexpireatCommand,-6,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HPEXPIREAT_Keyspecs,1,NULL,4),.args=HPEXPIREAT_Args},
{MAKE_CMD("hpexpiretime","Returns the Unix timestamp in milliseconds since Unix epoch at which the given key's field(s) will expire","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HPEXPIRETIME_History,0,HPEXPIRETIME_Tips,0,hpexpiretimeCommand,-5,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_READ,NULL,HPEXPIRETIME_Keyspecs,1,NULL,2),.args=HPEXPIRETIME_Args},
{MAKE_CMD("hpexpire","Sets expiry time on hash object.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HPEXPIRE_History,0,HPEXPIRE_Tips,0,hpexpireCommand,-6,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HPEXPIRE_Keyspecs,1,NULL,4),.args=HPEXPIRE_Args},
{MAKE_CMD("hpexpireat","Sets expiration time on hash field.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HPEXPIREAT_History,0,HPEXPIREAT_Tips,0,hpexpireatCommand,-6,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HPEXPIREAT_Keyspecs,1,NULL,4),.args=HPEXPIREAT_Args},
{MAKE_CMD("hpexpiretime","Returns the Unix timestamp in milliseconds since Unix epoch at which the given key's field(s) will expire.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HPEXPIRETIME_History,0,HPEXPIRETIME_Tips,0,hpexpiretimeCommand,-5,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_READ,NULL,HPEXPIRETIME_Keyspecs,1,NULL,2),.args=HPEXPIRETIME_Args},
{MAKE_CMD("hpttl","Returns the remaining time to live in milliseconds of a hash key's field(s) that have an associated expiration.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HPTTL_History,0,HPTTL_Tips,0,hpttlCommand,-5,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_READ,NULL,HPTTL_Keyspecs,1,NULL,2),.args=HPTTL_Args},
{MAKE_CMD("hrandfield","Returns one or more random fields from a hash.","O(N) where N is the number of fields returned","6.2.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HRANDFIELD_History,0,HRANDFIELD_Tips,1,hrandfieldCommand,-2,CMD_READONLY,ACL_CATEGORY_HASH|ACL_CATEGORY_READ|ACL_CATEGORY_SLOW,NULL,HRANDFIELD_Keyspecs,1,NULL,2),.args=HRANDFIELD_Args},
{MAKE_CMD("hscan","Iterates over fields and values of a hash.","O(1) for every call. O(N) for a complete iteration, including enough command calls for the cursor to return back to 0. N is the number of elements inside the collection.","2.8.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HSCAN_History,0,HSCAN_Tips,1,hscanCommand,-3,CMD_READONLY,ACL_CATEGORY_HASH|ACL_CATEGORY_READ|ACL_CATEGORY_SLOW,NULL,HSCAN_Keyspecs,1,NULL,5),.args=HSCAN_Args},
{MAKE_CMD("hset","Creates or modifies the value of a field in a hash.","O(1) for each field/value pair added, so O(N) to add N field/value pairs when the command is called with multiple field/value pairs.","2.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HSET_History,1,HSET_Tips,0,hsetCommand,-4,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HSET_Keyspecs,1,NULL,2),.args=HSET_Args},
{MAKE_CMD("hsetex","Set the value of one or more fields of a given hash key, and optionally set their expiration time.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HSETEX_History,0,HSETEX_Tips,0,hsetexCommand,-6,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HSETEX_Keyspecs,1,NULL,5),.args=HSETEX_Args},
{MAKE_CMD("hsetex","Sets the value of one or more fields of a given hash key, and optionally sets their expiration time.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HSETEX_History,0,HSETEX_Tips,0,hsetexCommand,-6,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HSETEX_Keyspecs,1,NULL,5),.args=HSETEX_Args},
{MAKE_CMD("hsetnx","Sets the value of a field in a hash only when the field doesn't exist.","O(1)","2.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HSETNX_History,0,HSETNX_Tips,0,hsetnxCommand,4,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_WRITE,NULL,HSETNX_Keyspecs,1,NULL,3),.args=HSETNX_Args},
{MAKE_CMD("hstrlen","Returns the length of the value of a field.","O(1)","3.2.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HSTRLEN_History,0,HSTRLEN_Tips,0,hstrlenCommand,3,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_READ,NULL,HSTRLEN_Keyspecs,1,NULL,2),.args=HSTRLEN_Args},
{MAKE_CMD("httl","Returns the remaining time to live (in seconds) of a hash key's field(s) that have an associated expiration.","O(N) where N is the number of specified fields.","9.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HTTL_History,0,HTTL_Tips,0,httlCommand,-5,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_HASH|ACL_CATEGORY_READ,NULL,HTTL_Keyspecs,1,NULL,2),.args=HTTL_Args},
@@ -11865,7 +11950,7 @@ struct COMMAND_STRUCT serverCommandTable[] = {
{MAKE_CMD("blmpop","Pops the first element from one of multiple lists. Blocks until an element is available otherwise. Deletes the list if the last element was popped.","O(N+M) where N is the number of provided keys and M is the number of elements returned.","7.0.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,BLMPOP_History,0,BLMPOP_Tips,0,blmpopCommand,-5,CMD_WRITE|CMD_BLOCKING,ACL_CATEGORY_BLOCKING|ACL_CATEGORY_LIST|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,BLMPOP_Keyspecs,1,blmpopGetKeys,5),.args=BLMPOP_Args},
{MAKE_CMD("blpop","Removes and returns the first element in a list. Blocks until an element is available otherwise. Deletes the list if the last element was popped.","O(N) where N is the number of provided keys.","2.0.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,BLPOP_History,1,BLPOP_Tips,0,blpopCommand,-3,CMD_WRITE|CMD_BLOCKING,ACL_CATEGORY_BLOCKING|ACL_CATEGORY_LIST|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,BLPOP_Keyspecs,1,NULL,2),.args=BLPOP_Args},
{MAKE_CMD("brpop","Removes and returns the last element in a list. Blocks until an element is available otherwise. Deletes the list if the last element was popped.","O(N) where N is the number of provided keys.","2.0.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,BRPOP_History,1,BRPOP_Tips,0,brpopCommand,-3,CMD_WRITE|CMD_BLOCKING,ACL_CATEGORY_BLOCKING|ACL_CATEGORY_LIST|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,BRPOP_Keyspecs,1,NULL,2),.args=BRPOP_Args},
{MAKE_CMD("brpoplpush","Pops an element from a list, pushes it to another list and returns it. Block until an element is available otherwise. Deletes the list if the last element was popped.","O(1)","2.2.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,BRPOPLPUSH_History,1,BRPOPLPUSH_Tips,0,brpoplpushCommand,4,CMD_WRITE|CMD_DENYOOM|CMD_BLOCKING,ACL_CATEGORY_BLOCKING|ACL_CATEGORY_LIST|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,BRPOPLPUSH_Keyspecs,2,NULL,3),.args=BRPOPLPUSH_Args},
{MAKE_CMD("brpoplpush","Pops an element from a list, pushes it to another list and returns it. Blocks until an element is available otherwise. Deletes the list if the last element was popped.","O(1)","2.2.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,BRPOPLPUSH_History,1,BRPOPLPUSH_Tips,0,brpoplpushCommand,4,CMD_WRITE|CMD_DENYOOM|CMD_BLOCKING,ACL_CATEGORY_BLOCKING|ACL_CATEGORY_LIST|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,BRPOPLPUSH_Keyspecs,2,NULL,3),.args=BRPOPLPUSH_Args},
{MAKE_CMD("lindex","Returns an element from a list by its index.","O(N) where N is the number of elements to traverse to get to the element at index. This makes asking for the first or the last element of the list O(1).","1.0.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,LINDEX_History,0,LINDEX_Tips,0,lindexCommand,3,CMD_READONLY,ACL_CATEGORY_LIST|ACL_CATEGORY_READ|ACL_CATEGORY_SLOW,NULL,LINDEX_Keyspecs,1,NULL,2),.args=LINDEX_Args},
{MAKE_CMD("linsert","Inserts an element before or after another element in a list.","O(N) where N is the number of elements to traverse before seeing the value pivot. This means that inserting somewhere on the left end on the list (head) can be considered O(1) and inserting somewhere on the right end (tail) is O(N).","2.2.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,LINSERT_History,0,LINSERT_Tips,0,linsertCommand,5,CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_LIST|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,LINSERT_Keyspecs,1,NULL,4),.args=LINSERT_Args},
{MAKE_CMD("llen","Returns the length of a list.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,LLEN_History,0,LLEN_Tips,0,llenCommand,2,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_LIST|ACL_CATEGORY_READ,NULL,LLEN_Keyspecs,1,NULL,1),.args=LLEN_Args},
@@ -11878,7 +11963,7 @@ struct COMMAND_STRUCT serverCommandTable[] = {
{MAKE_CMD("lrange","Returns a range of elements from a list.","O(S+N) where S is the distance of start offset from HEAD for small lists, from nearest end (HEAD or TAIL) for large lists; and N is the number of elements in the specified range.","1.0.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,LRANGE_History,0,LRANGE_Tips,0,lrangeCommand,4,CMD_READONLY,ACL_CATEGORY_LIST|ACL_CATEGORY_READ|ACL_CATEGORY_SLOW,NULL,LRANGE_Keyspecs,1,NULL,3),.args=LRANGE_Args},
{MAKE_CMD("lrem","Removes elements from a list. Deletes the list if the last element was removed.","O(N+M) where N is the length of the list and M is the number of elements removed.","1.0.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,LREM_History,0,LREM_Tips,0,lremCommand,4,CMD_WRITE,ACL_CATEGORY_LIST|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,LREM_Keyspecs,1,NULL,3),.args=LREM_Args},
{MAKE_CMD("lset","Sets the value of an element in a list by its index.","O(N) where N is the length of the list. Setting either the first or the last element of the list is O(1).","1.0.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,LSET_History,0,LSET_Tips,0,lsetCommand,4,CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_LIST|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,LSET_Keyspecs,1,NULL,3),.args=LSET_Args},
{MAKE_CMD("ltrim","Removes elements from both ends a list. Deletes the list if all elements were trimmed.","O(N) where N is the number of elements to be removed by the operation.","1.0.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,LTRIM_History,0,LTRIM_Tips,0,ltrimCommand,4,CMD_WRITE,ACL_CATEGORY_LIST|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,LTRIM_Keyspecs,1,NULL,3),.args=LTRIM_Args},
{MAKE_CMD("ltrim","Removes elements from both ends of a list. Deletes the list if all elements were trimmed.","O(N) where N is the number of elements to be removed by the operation.","1.0.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,LTRIM_History,0,LTRIM_Tips,0,ltrimCommand,4,CMD_WRITE,ACL_CATEGORY_LIST|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,LTRIM_Keyspecs,1,NULL,3),.args=LTRIM_Args},
{MAKE_CMD("rpop","Returns and removes one or more elements from the end of a list. Deletes the list if the last element was popped.","O(N) where N is the number of elements returned","1.0.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,RPOP_History,1,RPOP_Tips,0,rpopCommand,-2,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_LIST|ACL_CATEGORY_WRITE,NULL,RPOP_Keyspecs,1,NULL,2),.args=RPOP_Args},
{MAKE_CMD("rpoplpush","Returns the last element of a list after removing and pushing it to another list. Deletes the list if the last element was popped.","O(1)","1.2.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,RPOPLPUSH_History,0,RPOPLPUSH_Tips,0,rpoplpushCommand,3,CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_LIST|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,RPOPLPUSH_Keyspecs,2,NULL,2),.args=RPOPLPUSH_Args},
{MAKE_CMD("rpush","Appends one or more elements to a list. Creates the key if it doesn't exist.","O(1) for each element added, so O(N) to add N elements when the command is called with multiple arguments.","1.0.0",CMD_DOC_NONE,NULL,NULL,"list",COMMAND_GROUP_LIST,RPUSH_History,1,RPUSH_Tips,0,rpushCommand,-3,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_LIST|ACL_CATEGORY_WRITE,NULL,RPUSH_Keyspecs,1,NULL,2),.args=RPUSH_Args},
@@ -11888,7 +11973,7 @@ struct COMMAND_STRUCT serverCommandTable[] = {
{MAKE_CMD("publish","Posts a message to a channel.","O(N+M) where N is the number of clients subscribed to the receiving channel and M is the total number of subscribed patterns (by any client).","2.0.0",CMD_DOC_NONE,NULL,NULL,"pubsub",COMMAND_GROUP_PUBSUB,PUBLISH_History,0,PUBLISH_Tips,0,publishCommand,3,CMD_PUBSUB|CMD_LOADING|CMD_STALE|CMD_FAST|CMD_MAY_REPLICATE|CMD_SENTINEL,ACL_CATEGORY_FAST|ACL_CATEGORY_PUBSUB,NULL,PUBLISH_Keyspecs,0,NULL,2),.args=PUBLISH_Args},
{MAKE_CMD("pubsub","A container for Pub/Sub commands.","Depends on subcommand.","2.8.0",CMD_DOC_NONE,NULL,NULL,"pubsub",COMMAND_GROUP_PUBSUB,PUBSUB_History,0,PUBSUB_Tips,0,NULL,-2,0,ACL_CATEGORY_SLOW,NULL,PUBSUB_Keyspecs,0,NULL,0),.subcommands=PUBSUB_Subcommands},
{MAKE_CMD("punsubscribe","Stops listening to messages published to channels that match one or more patterns.","O(N) where N is the number of patterns to unsubscribe.","2.0.0",CMD_DOC_NONE,NULL,NULL,"pubsub",COMMAND_GROUP_PUBSUB,PUNSUBSCRIBE_History,0,PUNSUBSCRIBE_Tips,0,punsubscribeCommand,-1,CMD_PUBSUB|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_PUBSUB|ACL_CATEGORY_SLOW,NULL,PUNSUBSCRIBE_Keyspecs,0,NULL,1),.args=PUNSUBSCRIBE_Args},
{MAKE_CMD("spublish","Post a message to a shard channel","O(N) where N is the number of clients subscribed to the receiving shard channel.","7.0.0",CMD_DOC_NONE,NULL,NULL,"pubsub",COMMAND_GROUP_PUBSUB,SPUBLISH_History,0,SPUBLISH_Tips,0,spublishCommand,3,CMD_PUBSUB|CMD_LOADING|CMD_STALE|CMD_FAST|CMD_MAY_REPLICATE,ACL_CATEGORY_FAST|ACL_CATEGORY_PUBSUB,NULL,SPUBLISH_Keyspecs,1,NULL,2),.args=SPUBLISH_Args},
{MAKE_CMD("spublish","Posts a message to a shard channel.","O(N) where N is the number of clients subscribed to the receiving shard channel.","7.0.0",CMD_DOC_NONE,NULL,NULL,"pubsub",COMMAND_GROUP_PUBSUB,SPUBLISH_History,0,SPUBLISH_Tips,0,spublishCommand,3,CMD_PUBSUB|CMD_LOADING|CMD_STALE|CMD_FAST|CMD_MAY_REPLICATE,ACL_CATEGORY_FAST|ACL_CATEGORY_PUBSUB,NULL,SPUBLISH_Keyspecs,1,NULL,2),.args=SPUBLISH_Args},
{MAKE_CMD("ssubscribe","Listens for messages published to shard channels.","O(N) where N is the number of shard channels to subscribe to.","7.0.0",CMD_DOC_NONE,NULL,NULL,"pubsub",COMMAND_GROUP_PUBSUB,SSUBSCRIBE_History,0,SSUBSCRIBE_Tips,0,ssubscribeCommand,-2,CMD_PUBSUB|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE,ACL_CATEGORY_PUBSUB|ACL_CATEGORY_SLOW,NULL,SSUBSCRIBE_Keyspecs,1,NULL,1),.args=SSUBSCRIBE_Args},
{MAKE_CMD("subscribe","Listens for messages published to channels.","O(N) where N is the number of channels to subscribe to.","2.0.0",CMD_DOC_NONE,NULL,NULL,"pubsub",COMMAND_GROUP_PUBSUB,SUBSCRIBE_History,0,SUBSCRIBE_Tips,0,subscribeCommand,-2,CMD_PUBSUB|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_PUBSUB|ACL_CATEGORY_SLOW,NULL,SUBSCRIBE_Keyspecs,0,NULL,1),.args=SUBSCRIBE_Args},
{MAKE_CMD("sunsubscribe","Stops listening to messages posted to shard channels.","O(N) where N is the number of shard channels to unsubscribe.","7.0.0",CMD_DOC_NONE,NULL,NULL,"pubsub",COMMAND_GROUP_PUBSUB,SUNSUBSCRIBE_History,0,SUNSUBSCRIBE_Tips,0,sunsubscribeCommand,-1,CMD_PUBSUB|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE,ACL_CATEGORY_PUBSUB|ACL_CATEGORY_SLOW,NULL,SUNSUBSCRIBE_Keyspecs,1,NULL,1),.args=SUNSUBSCRIBE_Args},
@@ -11915,11 +12000,11 @@ struct COMMAND_STRUCT serverCommandTable[] = {
{MAKE_CMD("debug","A container for debugging commands.","Depends on subcommand.","1.0.0",CMD_DOC_SYSCMD,NULL,NULL,"server",COMMAND_GROUP_SERVER,DEBUG_History,0,DEBUG_Tips,0,debugCommand,-2,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_PROTECTED,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,DEBUG_Keyspecs,0,NULL,0)},
{MAKE_CMD("failover","Starts a coordinated failover from a server to one of its replicas.","O(1)","6.2.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,FAILOVER_History,0,FAILOVER_Tips,0,failoverCommand,-1,CMD_ADMIN|CMD_NOSCRIPT|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,FAILOVER_Keyspecs,0,NULL,3),.args=FAILOVER_Args},
{MAKE_CMD("flushall","Removes all keys from all databases.","O(N) where N is the total number of keys in all databases","1.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,FLUSHALL_History,2,FLUSHALL_Tips,2,flushallCommand,-1,CMD_WRITE|CMD_ALL_DBS,ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_KEYSPACE|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,FLUSHALL_Keyspecs,0,NULL,1),.args=FLUSHALL_Args},
{MAKE_CMD("flushdb","Remove all keys from the current database.","O(N) where N is the number of keys in the selected database","1.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,FLUSHDB_History,2,FLUSHDB_Tips,2,flushdbCommand,-1,CMD_WRITE,ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_KEYSPACE|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,FLUSHDB_Keyspecs,0,NULL,1),.args=FLUSHDB_Args},
{MAKE_CMD("flushdb","Removes all keys from the current database.","O(N) where N is the number of keys in the selected database","1.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,FLUSHDB_History,2,FLUSHDB_Tips,2,flushdbCommand,-1,CMD_WRITE,ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_KEYSPACE|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,FLUSHDB_Keyspecs,0,NULL,1),.args=FLUSHDB_Args},
{MAKE_CMD("info","Returns information and statistics about the server.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,INFO_History,1,INFO_Tips,3,infoCommand,-1,CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,INFO_Keyspecs,0,NULL,1),.args=INFO_Args},
{MAKE_CMD("lastsave","Returns the Unix timestamp of the last successful save to disk.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,LASTSAVE_History,0,LASTSAVE_Tips,1,lastsaveCommand,1,CMD_LOADING|CMD_STALE|CMD_FAST,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_FAST,NULL,LASTSAVE_Keyspecs,0,NULL,0)},
{MAKE_CMD("latency","A container for latency diagnostics commands.","Depends on subcommand.","2.8.13",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,LATENCY_History,0,LATENCY_Tips,0,NULL,-2,0,ACL_CATEGORY_SLOW,NULL,LATENCY_Keyspecs,0,NULL,0),.subcommands=LATENCY_Subcommands},
{MAKE_CMD("lolwut","Displays computer art and the server version",NULL,"5.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,LOLWUT_History,0,LOLWUT_Tips,0,lolwutCommand,-1,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_READ,NULL,LOLWUT_Keyspecs,0,NULL,1),.args=LOLWUT_Args},
{MAKE_CMD("lolwut","Displays computer art and the server version.",NULL,"5.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,LOLWUT_History,0,LOLWUT_Tips,0,lolwutCommand,-1,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_READ,NULL,LOLWUT_Keyspecs,0,NULL,1),.args=LOLWUT_Args},
{MAKE_CMD("memory","A container for memory diagnostics commands.","Depends on subcommand.","4.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,MEMORY_History,0,MEMORY_Tips,0,NULL,-2,0,ACL_CATEGORY_SLOW,NULL,MEMORY_Keyspecs,0,NULL,0),.subcommands=MEMORY_Subcommands},
{MAKE_CMD("module","A container for module commands.","Depends on subcommand.","4.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,MODULE_History,0,MODULE_Tips,0,NULL,-2,0,ACL_CATEGORY_SLOW,NULL,MODULE_Keyspecs,0,NULL,0),.subcommands=MODULE_Subcommands},
{MAKE_CMD("monitor","Listens for all requests received by the server in real-time.",NULL,"1.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,MONITOR_History,0,MONITOR_Tips,0,monitorCommand,1,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS|ACL_CATEGORY_SLOW,NULL,MONITOR_Keyspecs,0,NULL,0)},
@@ -11948,14 +12033,14 @@ struct COMMAND_STRUCT serverCommandTable[] = {
{MAKE_CMD("smismember","Determines whether multiple members belong to a set.","O(N) where N is the number of elements being checked for membership","6.2.0",CMD_DOC_NONE,NULL,NULL,"set",COMMAND_GROUP_SET,SMISMEMBER_History,0,SMISMEMBER_Tips,0,smismemberCommand,-3,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_READ|ACL_CATEGORY_SET,NULL,SMISMEMBER_Keyspecs,1,NULL,2),.args=SMISMEMBER_Args},
{MAKE_CMD("smove","Moves a member from one set to another.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"set",COMMAND_GROUP_SET,SMOVE_History,0,SMOVE_Tips,0,smoveCommand,4,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_SET|ACL_CATEGORY_WRITE,NULL,SMOVE_Keyspecs,2,NULL,3),.args=SMOVE_Args},
{MAKE_CMD("spop","Returns one or more random members from a set after removing them. Deletes the set if the last member was popped.","Without the count argument O(1), otherwise O(N) where N is the value of the passed count.","1.0.0",CMD_DOC_NONE,NULL,NULL,"set",COMMAND_GROUP_SET,SPOP_History,1,SPOP_Tips,1,spopCommand,-2,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_SET|ACL_CATEGORY_WRITE,NULL,SPOP_Keyspecs,1,NULL,2),.args=SPOP_Args},
{MAKE_CMD("srandmember","Get one or multiple random members from a set","Without the count argument O(1), otherwise O(N) where N is the absolute value of the passed count.","1.0.0",CMD_DOC_NONE,NULL,NULL,"set",COMMAND_GROUP_SET,SRANDMEMBER_History,1,SRANDMEMBER_Tips,1,srandmemberCommand,-2,CMD_READONLY,ACL_CATEGORY_READ|ACL_CATEGORY_SET|ACL_CATEGORY_SLOW,NULL,SRANDMEMBER_Keyspecs,1,NULL,2),.args=SRANDMEMBER_Args},
{MAKE_CMD("srandmember","Gets one or multiple random members from a set.","Without the count argument O(1), otherwise O(N) where N is the absolute value of the passed count.","1.0.0",CMD_DOC_NONE,NULL,NULL,"set",COMMAND_GROUP_SET,SRANDMEMBER_History,1,SRANDMEMBER_Tips,1,srandmemberCommand,-2,CMD_READONLY,ACL_CATEGORY_READ|ACL_CATEGORY_SET|ACL_CATEGORY_SLOW,NULL,SRANDMEMBER_Keyspecs,1,NULL,2),.args=SRANDMEMBER_Args},
{MAKE_CMD("srem","Removes one or more members from a set. Deletes the set if the last member was removed.","O(N) where N is the number of members to be removed.","1.0.0",CMD_DOC_NONE,NULL,NULL,"set",COMMAND_GROUP_SET,SREM_History,1,SREM_Tips,0,sremCommand,-3,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_SET|ACL_CATEGORY_WRITE,NULL,SREM_Keyspecs,1,NULL,2),.args=SREM_Args},
{MAKE_CMD("sscan","Iterates over members of a set.","O(1) for every call. O(N) for a complete iteration, including enough command calls for the cursor to return back to 0. N is the number of elements inside the collection.","2.8.0",CMD_DOC_NONE,NULL,NULL,"set",COMMAND_GROUP_SET,SSCAN_History,0,SSCAN_Tips,1,sscanCommand,-3,CMD_READONLY,ACL_CATEGORY_READ|ACL_CATEGORY_SET|ACL_CATEGORY_SLOW,NULL,SSCAN_Keyspecs,1,NULL,4),.args=SSCAN_Args},
{MAKE_CMD("sunion","Returns the union of multiple sets.","O(N) where N is the total number of elements in all given sets.","1.0.0",CMD_DOC_NONE,NULL,NULL,"set",COMMAND_GROUP_SET,SUNION_History,0,SUNION_Tips,1,sunionCommand,-2,CMD_READONLY,ACL_CATEGORY_READ|ACL_CATEGORY_SET|ACL_CATEGORY_SLOW,NULL,SUNION_Keyspecs,1,NULL,1),.args=SUNION_Args},
{MAKE_CMD("sunionstore","Stores the union of multiple sets in a key.","O(N) where N is the total number of elements in all given sets.","1.0.0",CMD_DOC_NONE,NULL,NULL,"set",COMMAND_GROUP_SET,SUNIONSTORE_History,0,SUNIONSTORE_Tips,0,sunionstoreCommand,-3,CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_SET|ACL_CATEGORY_SLOW|ACL_CATEGORY_WRITE,NULL,SUNIONSTORE_Keyspecs,2,NULL,2),.args=SUNIONSTORE_Args},
/* sorted_set */
{MAKE_CMD("bzmpop","Removes and returns a member by score from one or more sorted sets. Blocks until a member is available otherwise. Deletes the sorted set if the last element was popped.","O(K) + O(M*log(N)) where K is the number of provided keys, N being the number of elements in the sorted set, and M being the number of elements popped.","7.0.0",CMD_DOC_NONE,NULL,NULL,"sorted_set",COMMAND_GROUP_SORTED_SET,BZMPOP_History,0,BZMPOP_Tips,0,bzmpopCommand,-5,CMD_WRITE|CMD_BLOCKING,ACL_CATEGORY_BLOCKING|ACL_CATEGORY_SLOW|ACL_CATEGORY_SORTEDSET|ACL_CATEGORY_WRITE,NULL,BZMPOP_Keyspecs,1,blmpopGetKeys,5),.args=BZMPOP_Args},
{MAKE_CMD("bzpopmax","Removes and returns the member with the highest score from one or more sorted sets. Blocks until a member available otherwise. Deletes the sorted set if the last element was popped.","O(log(N)) with N being the number of elements in the sorted set.","5.0.0",CMD_DOC_NONE,NULL,NULL,"sorted_set",COMMAND_GROUP_SORTED_SET,BZPOPMAX_History,1,BZPOPMAX_Tips,0,bzpopmaxCommand,-3,CMD_WRITE|CMD_FAST|CMD_BLOCKING,ACL_CATEGORY_BLOCKING|ACL_CATEGORY_FAST|ACL_CATEGORY_SORTEDSET|ACL_CATEGORY_WRITE,NULL,BZPOPMAX_Keyspecs,1,NULL,2),.args=BZPOPMAX_Args},
{MAKE_CMD("bzpopmax","Removes and returns the member with the highest score from one or more sorted sets. Blocks until a member is available otherwise. Deletes the sorted set if the last element was popped.","O(log(N)) with N being the number of elements in the sorted set.","5.0.0",CMD_DOC_NONE,NULL,NULL,"sorted_set",COMMAND_GROUP_SORTED_SET,BZPOPMAX_History,1,BZPOPMAX_Tips,0,bzpopmaxCommand,-3,CMD_WRITE|CMD_FAST|CMD_BLOCKING,ACL_CATEGORY_BLOCKING|ACL_CATEGORY_FAST|ACL_CATEGORY_SORTEDSET|ACL_CATEGORY_WRITE,NULL,BZPOPMAX_Keyspecs,1,NULL,2),.args=BZPOPMAX_Args},
{MAKE_CMD("bzpopmin","Removes and returns the member with the lowest score from one or more sorted sets. Blocks until a member is available otherwise. Deletes the sorted set if the last element was popped.","O(log(N)) with N being the number of elements in the sorted set.","5.0.0",CMD_DOC_NONE,NULL,NULL,"sorted_set",COMMAND_GROUP_SORTED_SET,BZPOPMIN_History,1,BZPOPMIN_Tips,0,bzpopminCommand,-3,CMD_WRITE|CMD_FAST|CMD_BLOCKING,ACL_CATEGORY_BLOCKING|ACL_CATEGORY_FAST|ACL_CATEGORY_SORTEDSET|ACL_CATEGORY_WRITE,NULL,BZPOPMIN_Keyspecs,1,NULL,2),.args=BZPOPMIN_Args},
{MAKE_CMD("zadd","Adds one or more members to a sorted set, or updates their scores. Creates the key if it doesn't exist.","O(log(N)) for each item added, where N is the number of elements in the sorted set.","1.2.0",CMD_DOC_NONE,NULL,NULL,"sorted_set",COMMAND_GROUP_SORTED_SET,ZADD_History,3,ZADD_Tips,0,zaddCommand,-4,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_SORTEDSET|ACL_CATEGORY_WRITE,NULL,ZADD_Keyspecs,1,NULL,6),.args=ZADD_Args},
{MAKE_CMD("zcard","Returns the number of members in a sorted set.","O(1)","1.2.0",CMD_DOC_NONE,NULL,NULL,"sorted_set",COMMAND_GROUP_SORTED_SET,ZCARD_History,0,ZCARD_Tips,0,zcardCommand,2,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_READ|ACL_CATEGORY_SORTEDSET,NULL,ZCARD_Keyspecs,1,NULL,1),.args=ZCARD_Args},
@@ -11992,12 +12077,12 @@ struct COMMAND_STRUCT serverCommandTable[] = {
/* stream */
{MAKE_CMD("xack","Returns the number of messages that were successfully acknowledged by the consumer group member of a stream.","O(1) for each message ID processed.","5.0.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XACK_History,0,XACK_Tips,0,xackCommand,-4,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STREAM|ACL_CATEGORY_WRITE,NULL,XACK_Keyspecs,1,NULL,3),.args=XACK_Args},
{MAKE_CMD("xadd","Appends a new message to a stream. Creates the key if it doesn't exist.","O(1) when adding a new entry, O(N) when trimming where N being the number of entries evicted.","5.0.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XADD_History,2,XADD_Tips,1,xaddCommand,-5,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STREAM|ACL_CATEGORY_WRITE,NULL,XADD_Keyspecs,1,NULL,5),.args=XADD_Args},
{MAKE_CMD("xautoclaim","Changes, or acquires, ownership of messages in a consumer group, as if the messages were delivered to as consumer group member.","O(1) if COUNT is small.","6.2.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XAUTOCLAIM_History,1,XAUTOCLAIM_Tips,1,xautoclaimCommand,-6,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STREAM|ACL_CATEGORY_WRITE,NULL,XAUTOCLAIM_Keyspecs,1,NULL,7),.args=XAUTOCLAIM_Args},
{MAKE_CMD("xclaim","Changes, or acquires, ownership of a message in a consumer group, as if the message was delivered a consumer group member.","O(log N) with N being the number of messages in the PEL of the consumer group.","5.0.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XCLAIM_History,0,XCLAIM_Tips,1,xclaimCommand,-6,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STREAM|ACL_CATEGORY_WRITE,NULL,XCLAIM_Keyspecs,1,NULL,11),.args=XCLAIM_Args},
{MAKE_CMD("xautoclaim","Changes, or acquires, ownership of messages in a consumer group, as if the messages were delivered to a consumer group member.","O(1) if COUNT is small.","6.2.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XAUTOCLAIM_History,1,XAUTOCLAIM_Tips,1,xautoclaimCommand,-6,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STREAM|ACL_CATEGORY_WRITE,NULL,XAUTOCLAIM_Keyspecs,1,NULL,7),.args=XAUTOCLAIM_Args},
{MAKE_CMD("xclaim","Changes, or acquires, ownership of a message in a consumer group, as if the message was delivered to a consumer group member.","O(log N) with N being the number of messages in the PEL of the consumer group.","5.0.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XCLAIM_History,0,XCLAIM_Tips,1,xclaimCommand,-6,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STREAM|ACL_CATEGORY_WRITE,NULL,XCLAIM_Keyspecs,1,NULL,11),.args=XCLAIM_Args},
{MAKE_CMD("xdel","Returns the number of messages after removing them from a stream.","O(1) for each single item to delete in the stream, regardless of the stream size.","5.0.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XDEL_History,0,XDEL_Tips,0,xdelCommand,-3,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STREAM|ACL_CATEGORY_WRITE,NULL,XDEL_Keyspecs,1,NULL,2),.args=XDEL_Args},
{MAKE_CMD("xgroup","A container for consumer groups commands.","Depends on subcommand.","5.0.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XGROUP_History,0,XGROUP_Tips,0,NULL,-2,0,ACL_CATEGORY_SLOW,NULL,XGROUP_Keyspecs,0,NULL,0),.subcommands=XGROUP_Subcommands},
{MAKE_CMD("xinfo","A container for stream introspection commands.","Depends on subcommand.","5.0.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XINFO_History,0,XINFO_Tips,0,NULL,-2,0,ACL_CATEGORY_SLOW,NULL,XINFO_Keyspecs,0,NULL,0),.subcommands=XINFO_Subcommands},
{MAKE_CMD("xlen","Return the number of messages in a stream.","O(1)","5.0.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XLEN_History,0,XLEN_Tips,0,xlenCommand,2,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_READ|ACL_CATEGORY_STREAM,NULL,XLEN_Keyspecs,1,NULL,1),.args=XLEN_Args},
{MAKE_CMD("xlen","Returns the number of messages in a stream.","O(1)","5.0.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XLEN_History,0,XLEN_Tips,0,xlenCommand,2,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_READ|ACL_CATEGORY_STREAM,NULL,XLEN_Keyspecs,1,NULL,1),.args=XLEN_Args},
{MAKE_CMD("xpending","Returns the information and entries from a stream consumer group's pending entries list.","O(N) with N being the number of elements returned, so asking for a small fixed number of entries per call is O(1). O(M), where M is the total number of entries scanned when used with the IDLE filter. When the command returns just the summary and the list of consumers is small, it runs in O(1) time; otherwise, an additional O(N) time for iterating every consumer.","5.0.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XPENDING_History,1,XPENDING_Tips,1,xpendingCommand,-3,CMD_READONLY,ACL_CATEGORY_READ|ACL_CATEGORY_SLOW|ACL_CATEGORY_STREAM,NULL,XPENDING_Keyspecs,1,NULL,3),.args=XPENDING_Args},
{MAKE_CMD("xrange","Returns the messages from a stream within a range of IDs.","O(N) with N being the number of elements being returned. If N is constant (e.g. always asking for the first 10 elements with COUNT), you can consider it O(1).","5.0.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XRANGE_History,1,XRANGE_Tips,0,xrangeCommand,-4,CMD_READONLY,ACL_CATEGORY_READ|ACL_CATEGORY_SLOW|ACL_CATEGORY_STREAM,NULL,XRANGE_Keyspecs,1,NULL,4),.args=XRANGE_Args},
{MAKE_CMD("xread","Returns messages from multiple streams with IDs greater than the ones requested. Blocks until a message is available otherwise.",NULL,"5.0.0",CMD_DOC_NONE,NULL,NULL,"stream",COMMAND_GROUP_STREAM,XREAD_History,0,XREAD_Tips,0,xreadCommand,-4,CMD_BLOCKING|CMD_READONLY,ACL_CATEGORY_BLOCKING|ACL_CATEGORY_READ|ACL_CATEGORY_SLOW|ACL_CATEGORY_STREAM,NULL,XREAD_Keyspecs,1,xreadGetKeys,3),.args=XREAD_Args},
@@ -12008,7 +12093,7 @@ struct COMMAND_STRUCT serverCommandTable[] = {
/* string */
{MAKE_CMD("append","Appends a string to the value of a key. Creates the key if it doesn't exist.","O(1). The amortized time complexity is O(1) assuming the appended value is small and the already present value is of any size, since the dynamic string library used by the server will double the free space available on every reallocation.","2.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,APPEND_History,0,APPEND_Tips,0,appendCommand,3,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,APPEND_Keyspecs,1,NULL,2),.args=APPEND_Args},
{MAKE_CMD("decr","Decrements the integer value of a key by one. Uses 0 as initial value if the key doesn't exist.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,DECR_History,0,DECR_Tips,0,decrCommand,2,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,DECR_Keyspecs,1,NULL,1),.args=DECR_Args},
{MAKE_CMD("decrby","Decrements a number from the integer value of a key. Uses 0 as initial value if the key doesn't exist.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,DECRBY_History,0,DECRBY_Tips,0,decrbyCommand,3,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,DECRBY_Keyspecs,1,NULL,2),.args=DECRBY_Args},
{MAKE_CMD("decrby","Decrements the integer value of a key by a number. Uses 0 as initial value if the key doesn't exist.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,DECRBY_History,0,DECRBY_Tips,0,decrbyCommand,3,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,DECRBY_Keyspecs,1,NULL,2),.args=DECRBY_Args},
{MAKE_CMD("delifeq","Delete key if value matches string.","O(1)","9.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,DELIFEQ_History,0,DELIFEQ_Tips,0,delifeqCommand,3,CMD_FAST|CMD_WRITE,ACL_CATEGORY_FAST|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,DELIFEQ_Keyspecs,1,NULL,2),.args=DELIFEQ_Args},
{MAKE_CMD("get","Returns the string value of a key.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,GET_History,0,GET_Tips,0,getCommand,2,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_READ|ACL_CATEGORY_STRING,NULL,GET_Keyspecs,1,NULL,1),.args=GET_Args},
{MAKE_CMD("getdel","Returns the string value of a key after deleting the key.","O(1)","6.2.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,GETDEL_History,0,GETDEL_Tips,0,getdelCommand,2,CMD_WRITE|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,GETDEL_Keyspecs,1,NULL,1),.args=GETDEL_Args},
@@ -12017,15 +12102,16 @@ struct COMMAND_STRUCT serverCommandTable[] = {
{MAKE_CMD("getset","Returns the previous string value of a key after setting it to a new value.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,GETSET_History,0,GETSET_Tips,0,getsetCommand,3,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,GETSET_Keyspecs,1,NULL,2),.args=GETSET_Args},
{MAKE_CMD("incr","Increments the integer value of a key by one. Uses 0 as initial value if the key doesn't exist.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,INCR_History,0,INCR_Tips,0,incrCommand,2,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,INCR_Keyspecs,1,NULL,1),.args=INCR_Args},
{MAKE_CMD("incrby","Increments the integer value of a key by a number. Uses 0 as initial value if the key doesn't exist.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,INCRBY_History,0,INCRBY_Tips,0,incrbyCommand,3,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,INCRBY_Keyspecs,1,NULL,2),.args=INCRBY_Args},
{MAKE_CMD("incrbyfloat","Increment the floating point value of a key by a number. Uses 0 as initial value if the key doesn't exist.","O(1)","2.6.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,INCRBYFLOAT_History,0,INCRBYFLOAT_Tips,0,incrbyfloatCommand,3,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,INCRBYFLOAT_Keyspecs,1,NULL,2),.args=INCRBYFLOAT_Args},
{MAKE_CMD("incrbyfloat","Increments the floating point value of a key by a number. Uses 0 as initial value if the key doesn't exist.","O(1)","2.6.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,INCRBYFLOAT_History,0,INCRBYFLOAT_Tips,0,incrbyfloatCommand,3,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,INCRBYFLOAT_Keyspecs,1,NULL,2),.args=INCRBYFLOAT_Args},
{MAKE_CMD("lcs","Finds the longest common substring.","O(N*M) where N and M are the lengths of s1 and s2, respectively","7.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,LCS_History,0,LCS_Tips,0,lcsCommand,-3,CMD_READONLY,ACL_CATEGORY_READ|ACL_CATEGORY_SLOW|ACL_CATEGORY_STRING,NULL,LCS_Keyspecs,1,NULL,6),.args=LCS_Args},
{MAKE_CMD("mget","Atomically returns the string values of one or more keys.","O(N) where N is the number of keys to retrieve.","1.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,MGET_History,0,MGET_Tips,1,mgetCommand,-2,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_READ|ACL_CATEGORY_STRING,NULL,MGET_Keyspecs,1,NULL,1),.args=MGET_Args},
{MAKE_CMD("mset","Atomically creates or modifies the string values of one or more keys.","O(N) where N is the number of keys to set.","1.0.1",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,MSET_History,0,MSET_Tips,2,msetCommand,-3,CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_SLOW|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,MSET_Keyspecs,1,NULL,1),.args=MSET_Args},
{MAKE_CMD("msetex","Atomically creates or modifies the string values of one or more keys, and optionally set their expiration.","O(N) where N is the number of keys to set.","9.1.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,MSETEX_History,0,MSETEX_Tips,2,msetexCommand,-4,CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE|ACL_CATEGORY_SLOW,NULL,MSETEX_Keyspecs,1,NULL,4),.args=MSETEX_Args},
{MAKE_CMD("msetnx","Atomically modifies the string values of one or more keys only when all keys don't exist.","O(N) where N is the number of keys to set.","1.0.1",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,MSETNX_History,0,MSETNX_Tips,0,msetnxCommand,-3,CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_SLOW|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,MSETNX_Keyspecs,1,NULL,1),.args=MSETNX_Args},
{MAKE_CMD("psetex","Sets both string value and expiration time in milliseconds of a key. The key is created if it doesn't exist.","O(1)","2.6.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,PSETEX_History,0,PSETEX_Tips,0,psetexCommand,4,CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_SLOW|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,PSETEX_Keyspecs,1,NULL,3),.args=PSETEX_Args},
{MAKE_CMD("set","Sets the string value of a key, ignoring its type. The key is created if it doesn't exist.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,SET_History,5,SET_Tips,0,setCommand,-3,CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_SLOW|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,SET_Keyspecs,1,setGetKeys,5),.args=SET_Args},
{MAKE_CMD("setex","Sets the string value and expiration time of a key. Creates the key if it doesn't exist.","O(1)","2.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,SETEX_History,0,SETEX_Tips,0,setexCommand,4,CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_SLOW|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,SETEX_Keyspecs,1,NULL,3),.args=SETEX_Args},
{MAKE_CMD("setnx","Set the string value of a key only when the key doesn't exist.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,SETNX_History,0,SETNX_Tips,0,setnxCommand,3,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,SETNX_Keyspecs,1,NULL,2),.args=SETNX_Args},
{MAKE_CMD("setnx","Sets the string value of a key only when the key doesn't exist.","O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,SETNX_History,0,SETNX_Tips,0,setnxCommand,3,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,SETNX_Keyspecs,1,NULL,2),.args=SETNX_Args},
{MAKE_CMD("setrange","Overwrites a part of a string value with another by an offset. Creates the key if it doesn't exist.","O(1), not counting the time taken to copy the new string in place. Usually, this string is very small so the amortized complexity is O(1). Otherwise, complexity is O(M) with M being the length of the value argument.","2.2.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,SETRANGE_History,0,SETRANGE_Tips,0,setrangeCommand,4,CMD_WRITE|CMD_DENYOOM,ACL_CATEGORY_SLOW|ACL_CATEGORY_STRING|ACL_CATEGORY_WRITE,NULL,SETRANGE_Keyspecs,1,NULL,3),.args=SETRANGE_Args},
{MAKE_CMD("strlen","Returns the length of a string value.","O(1)","2.2.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,STRLEN_History,0,STRLEN_Tips,0,strlenCommand,2,CMD_READONLY|CMD_FAST,ACL_CATEGORY_FAST|ACL_CATEGORY_READ|ACL_CATEGORY_STRING,NULL,STRLEN_Keyspecs,1,NULL,1),.args=STRLEN_Args},
{MAKE_CMD("substr","Returns a substring from a string value.","O(N) where N is the length of the returned string. The complexity is ultimately determined by the returned length, but because creating a substring from an existing string is very cheap, it can be considered O(1) for small strings.","1.0.0",CMD_DOC_NONE,NULL,NULL,"string",COMMAND_GROUP_STRING,SUBSTR_History,0,SUBSTR_Tips,0,getrangeCommand,4,CMD_READONLY,ACL_CATEGORY_READ|ACL_CATEGORY_SLOW|ACL_CATEGORY_STRING,NULL,SUBSTR_Keyspecs,1,NULL,3),.args=SUBSTR_Args},
src/commands.h
+41
View File
@@ -19,6 +19,47 @@ typedef enum {
#define CMD_ARG_MULTIPLE (1 << 1)
#define CMD_ARG_MULTIPLE_TOKEN (1 << 2)
#define COMMAND_GET 0
#define COMMAND_SET 1
#define COMMAND_HGET 2
#define COMMAND_HSET 3
#define COMMAND_MSET 4
/* Command flags. Please check the definition of struct serverCommand in this file
* for more information about the meaning of every flag. */
#define CMD_WRITE (1ULL << 0)
#define CMD_READONLY (1ULL << 1)
#define CMD_DENYOOM (1ULL << 2)
#define CMD_MODULE (1ULL << 3) /* Command exported by module. */
#define CMD_ADMIN (1ULL << 4)
#define CMD_PUBSUB (1ULL << 5)
#define CMD_NOSCRIPT (1ULL << 6)
#define CMD_BLOCKING (1ULL << 8) /* Has potential to block. */
#define CMD_LOADING (1ULL << 9)
#define CMD_STALE (1ULL << 10)
#define CMD_SKIP_MONITOR (1ULL << 11)
#define CMD_SKIP_COMMANDLOG (1ULL << 12)
#define CMD_ASKING (1ULL << 13)
#define CMD_FAST (1ULL << 14)
#define CMD_NO_AUTH (1ULL << 15)
#define CMD_MAY_REPLICATE (1ULL << 16)
#define CMD_SENTINEL (1ULL << 17)
#define CMD_ONLY_SENTINEL (1ULL << 18)
#define CMD_NO_MANDATORY_KEYS (1ULL << 19)
#define CMD_PROTECTED (1ULL << 20)
#define CMD_MODULE_GETKEYS (1ULL << 21) /* Use the modules getkeys interface. */
#define CMD_MODULE_NO_CLUSTER (1ULL << 22) /* Deny on Cluster. */
#define CMD_NO_ASYNC_LOADING (1ULL << 23)
#define CMD_NO_MULTI (1ULL << 24)
#define CMD_MOVABLE_KEYS (1ULL << 25) /* The legacy range spec doesn't cover all keys. \
* Populated by populateCommandLegacyRangeSpec. */
#define CMD_ALLOW_BUSY ((1ULL << 26))
#define CMD_MODULE_GETCHANNELS (1ULL << 27) /* Use the modules getchannels interface. */
#define CMD_TOUCHES_ARBITRARY_KEYS (1ULL << 28)
#define CMD_ALL_DBS (1ULL << 29)
/* Command flags. Please don't forget to add command flag documentation in struct
* serverCommand in server.h file. */
/* Must be compatible with RedisModuleCommandArg. See moduleCopyCommandArgs. */
typedef struct serverCommandArg {
const char *name;
src/commands/brpoplpush.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"BRPOPLPUSH": {
"summary": "Pops an element from a list, pushes it to another list and returns it. Block until an element is available otherwise. Deletes the list if the last element was popped.",
"summary": "Pops an element from a list, pushes it to another list and returns it. Blocks until an element is available otherwise. Deletes the list if the last element was popped.",
"complexity": "O(1)",
"group": "list",
"since": "2.2.0",
src/commands/bzpopmax.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"BZPOPMAX": {
"summary": "Removes and returns the member with the highest score from one or more sorted sets. Blocks until a member available otherwise. Deletes the sorted set if the last element was popped.",
"summary": "Removes and returns the member with the highest score from one or more sorted sets. Blocks until a member is available otherwise. Deletes the sorted set if the last element was popped.",
"complexity": "O(log(N)) with N being the number of elements in the sorted set.",
"group": "sorted_set",
"since": "5.0.0",
src/commands/client-import-source.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"IMPORT-SOURCE": {
"summary": "Mark this client as an import source when server is in import mode.",
"summary": "Marks this client as an import source when the server is in import mode.",
"complexity": "O(1)",
"group": "connection",
"since": "8.1.0",
src/commands/cluster-flushslot.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"FLUSHSLOT": {
"summary": "Remove all keys from the target slot.",
"summary": "Removes all keys from the target slot.",
"complexity": "O(N) where N is the number of keys in the target slot",
"group": "cluster",
"since": "9.0.0",
src/commands/cluster-getslotmigrations.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"GETSLOTMIGRATIONS": {
"summary": "Get the status of ongoing and recently finished slot import and export operations.",
"summary": "Gets the status of ongoing and recently finished slot import and export operations.",
"complexity": "O(N), where N is the number of active slot import and export jobs.",
"group": "cluster",
"since": "9.0.0",
src/commands/cluster-replicate.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"REPLICATE": {
"summary": "Configure a node as replica of a primary node or detach a replica from its primary.",
"summary": "Configures a node as replica of a primary node or detaches a replica from its primary.",
"complexity": "O(1)",
"group": "cluster",
"since": "3.0.0",
src/commands/cluster-shards.json
+4 -1
View File
@@ -10,7 +10,7 @@
"history": [
[
"9.1.0",
"Added shard id field to CLUSTER SHARDS response"
"Added shard level `id` field and node level `availability-zone` field."
]
],
"command_flags": [
@@ -84,6 +84,9 @@
"const": "online"
}
]
},
"availability-zone": {
"type": "string"
}
}
}
src/commands/cluster-slots.json
+12 -2
View File
@@ -15,6 +15,10 @@
[
"7.0.0",
"Added additional networking metadata field."
],
[
"9.1.0",
"Added `availability-zone` field inside the networking metadata."
]
],
"command_flags": [
@@ -68,7 +72,7 @@
"type": "string"
},
{
"description": "Array of node descriptions.",
"description": "Map of additional networking metadata fields.",
"type": "object",
"additionalProperties": false,
"properties": {
@@ -77,6 +81,9 @@
},
"ip": {
"type": "string"
},
"availability-zone": {
"type": "string"
}
}
}
@@ -111,7 +118,7 @@
"type": "string"
},
{
"description": "Array of node descriptions.",
"description": "Map of additional networking metadata fields.",
"type": "object",
"additionalProperties": false,
"properties": {
@@ -120,6 +127,9 @@
},
"ip": {
"type": "string"
},
"availability-zone": {
"type": "string"
}
}
}

Some files were not shown because too many files have changed in this diff Show More