Compare commits

...
89 Commits
Author SHA1 Message Date
YaacovHazanandGitHub 31d93c5928 Release 7.2.8 (#13965) 2025-04-23 14:54:38 +03:00
YaacovHazan 62b766aef6 Redis 7.2.8 2025-04-23 08:11:54 +00:00
YaacovHazan 42fb340ce4 Limiting output buffer for unauthenticated client (CVE-2025-21605)
For unauthenticated clients the output buffer is limited to prevent
them from abusing it by not reading the replies
2025-04-23 08:09:40 +00:00
guybe7anddebing.sun 21d5e64ace Module unblock on keys: updateStatsOnUnblock is called twice (#13405)
This commit reverts the deletion of the condition `!bc->blocked_on_keys`
that was accidentally introduced by
https://github.com/redis/redis/pull/12817.
In case a blocked-on-keys module client is unblocked both
`moduleUnblockClientOnKey` and `moduleHandleBlockedClients` are called
which resulted in `updateStatsOnUnblock` being called twice

Now, that `moduleHandleBlockedClients` doesn't call
`updateStatsOnUnblock` in case of unblocked module key-blocked clients,
in the unlikely event that the module decides to call `RM_UnblockClient`
on a key-blocked client, we need to call `updateStatsOnUnblock` from
within `moduleBlockedClientTimedOut`, but since
`moduleBlockedClientTimedOut` is not tread-safe we can't call it
directly from withing `RM_UnblockClient`.
Added a new flag `blocked_on_keys_explicit_unblock` for that specific
case, which will cause `moduleBlockedClientTimedOut` to be called from
`moduleHandleBlockedClients` (which is only called from the main thread)

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2025-04-22 20:56:52 +08:00
Vitah Linanddebing.sun 067a0dac61 Fix oldTC CI dk.archive.ubuntu.com could not connect (#13961) 2025-04-22 20:20:44 +08:00
YaacovHazan 9af9c4deff Avoid sanitizer warning for stable CI 2025-04-22 14:54:27 +03:00
JasonandYaacovHazan 779a20058b Ignore shardId updates from replica nodes (#13877)
Close https://github.com/redis/redis/issues/13868

This bug was introduced by https://github.com/redis/redis/pull/13468

To maintain compatibility with older versions that do not support
shardid, when a replica passes a shardid, we also update the master’s
shardid accordingly.

However, when both the master and replica support shardid, an issue
arises: in one moment, the master may pass a shardid, causing us to
update both the master and all its replicas to match the master’s
shardid. But if the replica later passes a different shardid, we would
then update the master’s shardid again, leading to continuous changes in
shardid.

Regardless of the situation, we always ensure that the replica’s shardid
remains consistent with the master’s shardid.
2025-04-22 13:49:12 +03:00
a26774cee1 Fix potential infinite loop of RANDOMKEY during client pause (#13863)
The bug mentioned in this
[#13862](https://github.com/redis/redis/issues/13862) has been fixed.

---------

Signed-off-by: li-benson <1260437731@qq.com>
Signed-off-by: youngmore1024 <youngmore1024@outlook.com>
Co-authored-by: youngmore1024 <youngmore1024@outlook.com>
2025-04-22 13:49:12 +03:00
Mingyi KangandYaacovHazan 9c5c2dc3a8 Bump actions/upload-artifact from 3 to 4 (#13780)
Update `upload-artifact` from v3 to v4 to avoid the failure of `External
Server Tests` (I encountered this error when opening
[#13779](https://github.com/redis/redis/pull/13779)):

> Error: This request has been automatically failed because it uses a
deprecated version of `actions/upload-artifact: v3`. Learn more:
https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/
2025-04-22 13:49:12 +03:00
6e819e188b Fix module loadex command crash due to invalid config (#13653)
Fix to https://github.com/redis/redis/issues/13650

providing an invalid config to a module with datatype crashes when redis
tries to unload the module due to the invalid config

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2025-04-22 13:49:12 +03:00
guybe7andYaacovHazan 8e72b42a9a Modules: defrag CB should take robj, not sds (#13627)
Added a log of the keyname in the test modules to reproduce the problem
(tests crash without the fix)
2025-04-22 13:49:12 +03:00
Ping XieandYaacovHazan bdc78175b5 Fix PONG message processing for primary-ship tracking during failovers (#13055)
This commit updates the processing of PONG gossip messages in the
cluster. When a node (B) becomes a replica due to a failover, its PONG
messages include its new primary node's (A) information and B's
configuration epoch is aligned with A's. This allows observer nodes to
identify changes in primary-ship, addressing issues of intermediate
states and enhancing cluster state consistency during topology changes.

Fix #13018
2025-04-22 13:49:12 +03:00
30fe743638 Fix race condition issues between the main thread and module threads (#12817)
Fix #12785 and other race condition issues.
See the following isolated comments.

The following report was obtained using SANITIZER thread.
```sh
make SANITIZER=thread
./runtest-moduleapi --config io-threads 4 --config io-threads-do-reads yes --accurate
```

1. Fixed thread-safe issue in RM_UnblockClient()
Related discussion:
https://github.com/redis/redis/pull/12817#issuecomment-1831181220
* When blocking a client in a module using `RM_BlockClientOnKeys()` or
`RM_BlockClientOnKeysWithFlags()`
with a timeout_callback, calling RM_UnblockClient() in module threads
can lead to race conditions
     in `updateStatsOnUnblock()`.

     - Introduced: 
        Version: 6.2
        PR: #7491

     - Touch:
`server.stat_numcommands`, `cmd->latency_histogram`, `server.slowlog`,
and `server.latency_events`
     
     - Harm Level: High
Potentially corrupts the memory data of `cmd->latency_histogram`,
`server.slowlog`, and `server.latency_events`

     - Solution:
Differentiate whether the call to moduleBlockedClientTimedOut() comes
from the module or the main thread.
Since we can't know if RM_UnblockClient() comes from module threads, we
always assume it does and
let `updateStatsOnUnblock()` asynchronously update the unblock status.
     
* When error reply is called in timeout_callback(), ctx is not
thread-safe, eventually lead to race conditions in `afterErrorReply`.

     - Introduced: 
        Version: 6.2
        PR: #8217

     - Touch
       `server.stat_total_error_replies`, `server.errors`, 

     - Harm Level: High
       Potentially corrupts the memory data of `server.errors`
   
      - Solution: 
Make the ctx in `timeout_callback()` with `REDISMODULE_CTX_THREAD_SAFE`,
and asynchronously reply errors to the client.

2. Made RM_Reply*() family API thread-safe
Related discussion:
https://github.com/redis/redis/pull/12817#discussion_r1408707239
Call chain: `RM_Reply*()` -> `_addReplyToBufferOrList()` -> touch
server.current_client

    - Introduced: 
       Version: 7.2.0
       PR: #12326

   - Harm Level: None
Since the module fake client won't have the `CLIENT_PUSHING` flag, even
if we touch server.current_client,
     we can still exit after `c->flags & CLIENT_PUSHING`.

   - Solution
      Checking `c->flags & CLIENT_PUSHING` earlier.

3. Made freeClient() thread-safe
    Fix #12785

    - Introduced: 
       Version: 4.0
Commit:
https://github.com/redis/redis/commit/3fcf959e609e850a114d4016843e4c991066ebac

    - Harm Level: Moderate
       * Trigger assertion
It happens when the module thread calls freeClient while the io-thread
is in progress,
which just triggers an assertion, and doesn't make any race condiaions.

* Touch `server.current_client`, `server.stat_clients_type_memory`, and
`clientMemUsageBucket->clients`.
It happens between the main thread and the module threads, may cause
data corruption.
1. Error reset `server.current_client` to NULL, but theoretically this
won't happen,
because the module has already reset `server.current_client` to old
value before entering freeClient.
2. corrupts `clientMemUsageBucket->clients` in
updateClientMemUsageAndBucket().
3. Causes server.stat_clients_type_memory memory statistics to be
inaccurate.
    
    - Solution:
* No longer counts memory usage on fake clients, to avoid updating
`server.stat_clients_type_memory` in freeClient.
* No longer resetting `server.current_client` in unlinkClient, because
the fake client won't be evicted or disconnected in the mid of the
process.
* Judgment assertion `io_threads_op == IO_THREADS_OP_IDLE` only if c is
not a fake client.

4. Fixed free client args without GIL
Related discussion:
https://github.com/redis/redis/pull/12817#discussion_r1408706695
When freeing retained strings in the module thread (refcount decr), or
using them in some way (refcount incr), we should do so while holding
the GIL,
otherwise, they might be simultaneously freed while the main thread is
processing the unblock client state.

    - Introduced: 
       Version: 6.2.0
       PR: #8141

   - Harm Level: Low
     Trigger assertion or double free or memory leak. 

   - Solution:
Documenting that module API users need to ensure any access to these
retained strings is done with the GIL locked

5. Fix adding fake client to server.clients_pending_write
    It will incorrectly log the memory usage for the fake client.
Related discussion:
https://github.com/redis/redis/pull/12817#issuecomment-1851899163

    - Introduced: 
       Version: 4.0
Commit:
https://github.com/redis/redis/commit/9b01b64430fbc1487429144d2e4e72a4a7fd9db2

    - Harm Level: None
      Only result in NOP

    - Solution:
       * Don't add fake client into server.clients_pending_write
* Add c->conn assertion for updateClientMemUsageAndBucket() and
updateClientMemoryUsage() to avoid same
         issue in the future.
So now it will be the responsibility of the caller of both of them to
avoid passing in fake client.

6. Fix calling RM_BlockedClientMeasureTimeStart() and
RM_BlockedClientMeasureTimeEnd() without GIL
    - Introduced: 
       Version: 6.2
       PR: #7491

   - Harm Level: Low
Causes inaccuracies in command latency histogram and slow logs, but does
not corrupt memory.

   - Solution:
Module API users, if know that non-thread-safe APIs will be used in
multi-threading, need to take responsibility for protecting them with
their own locks instead of the GIL, as using the GIL is too expensive.

### Other issue
1. RM_Yield is not thread-safe, fixed via #12905.

### Summarize
1. Fix thread-safe issues for `RM_UnblockClient()`, `freeClient()` and
`RM_Yield`, potentially preventing memory corruption, data disorder, or
assertion.
2. Updated docs and module test to clarify module API users'
responsibility for locking non-thread-safe APIs in multi-threading, such
as RM_BlockedClientMeasureTimeStart/End(), RM_FreeString(),
RM_RetainString(), and RM_HoldString().

### About backpot to 7.2
1. The implement of (1) is not too satisfying, would like to get more
eyes.
2. (2), (3) can be safely for backport
3. (4), (6) just modifying the module tests and updating the
documentation, no need for a backpot.
4. (5) is harmless, no need for a backpot.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2025-04-22 12:35:44 +03:00
7278a0c26a Make RM_Yield thread-safe (#12905)
## Issues and solutions from #12817
1. Touch ProcessingEventsWhileBlocked and calling moduleCount() without
GIL in afterSleep()
    - Introduced: 
       Version: 7.0.0
       PR: #9963

   - Harm Level: Very High
If the module thread calls `RM_Yield()` before the main thread enters
afterSleep(),
and modifies `ProcessingEventsWhileBlocked`(+1), it will cause the main
thread to not wait for GIL,
which can lead to all kinds of unforeseen problems, including memory
data corruption.

   - Initial / Abandoned Solution:
      * Added `__thread` specifier for ProcessingEventsWhileBlocked.
`ProcessingEventsWhileBlocked` is used to protect against nested event
processing, but event processing
in the main thread and module threads should be completely independent
and unaffected, so it is safer
         to use TLS.
* Adding a cached module count to keep track of the current number of
modules, to avoid having to use `dictSize()`.
    
    - Related Warnings:
```
WARNING: ThreadSanitizer: data race (pid=1136)
  Write of size 4 at 0x0001045990c0 by thread T4 (mutexes: write M0):
    #0 processEventsWhileBlocked networking.c:4135 (redis-server:arm64+0x10006d124)
    #1 RM_Yield module.c:2410 (redis-server:arm64+0x10018b66c)
    #2 bg_call_worker <null>:83232836 (blockedclient.so:arm64+0x16a8)

  Previous read of size 4 at 0x0001045990c0 by main thread:
    #0 afterSleep server.c:1861 (redis-server:arm64+0x100024f98)
    #1 aeProcessEvents ae.c:408 (redis-server:arm64+0x10000fd64)
    #2 aeMain ae.c:496 (redis-server:arm64+0x100010f0c)
    #3 main server.c:7220 (redis-server:arm64+0x10003f38c)
```

2. aeApiPoll() is not thread-safe
When using RM_Yield to handle events in a module thread, if the main
thread has not yet
entered `afterSleep()`, both the module thread and the main thread may
touch `server.el` at the same time.

    - Introduced: 
       Version: 7.0.0
       PR: #9963

   - Old / Abandoned Solution:
Adding a new mutex to protect timing between after beforeSleep() and
before afterSleep().
Defect: If the main thread enters the ae loop without any IO events, it
will wait until
the next timeout or until there is any event again, and the module
thread will
always hang until the main thread leaves the event loop.

    - Related Warnings:
```
SUMMARY: ThreadSanitizer: data race ae_kqueue.c:55 in addEventMask
==================
==================
WARNING: ThreadSanitizer: data race (pid=14682)
  Write of size 4 at 0x000100b54000 by thread T9 (mutexes: write M0):
    #0 aeApiPoll ae_kqueue.c:175 (redis-server:arm64+0x100010588)
    #1 aeProcessEvents ae.c:399 (redis-server:arm64+0x10000fb84)
    #2 processEventsWhileBlocked networking.c:4138 (redis-server:arm64+0x10006d3c4)
    #3 RM_Yield module.c:2410 (redis-server:arm64+0x10018b66c)
    #4 bg_call_worker <null>:16042052 (blockedclient.so:arm64+0x169c)

  Previous write of size 4 at 0x000100b54000 by main thread:
    #0 aeApiPoll ae_kqueue.c:175 (redis-server:arm64+0x100010588)
    #1 aeProcessEvents ae.c:399 (redis-server:arm64+0x10000fb84)
    #2 aeMain ae.c:496 (redis-server:arm64+0x100010da8)
    #3 main server.c:7238 (redis-server:arm64+0x10003f51c)
```

## The final fix as the comments:
https://github.com/redis/redis/pull/12817#discussion_r1436427232
Optimized solution based on the above comment:

First, we add `module_gil_acquring` to indicate whether the main thread
is currently in the acquiring GIL state.

When the module thread starts to yield, there are two possibilities(we
assume the caller keeps the GIL):
1. The main thread is in the mid of beforeSleep() and afterSleep(), that
is, `module_gil_acquring` is not 1 now.
At this point, the module thread will wake up the main thread through
the pipe and leave the yield,
waiting for the next yield when the main thread may already in the
acquiring GIL state.
    
2. The main thread is in the acquiring GIL state.
The module thread release the GIL, yielding CPU to give the main thread
an opportunity to start
event processing, and then acquire the GIL again until the main thread
releases it.
This is what
https://github.com/redis/redis/pull/12817#discussion_r1436427232
mentioned direction.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2025-04-22 12:35:36 +03:00
YaacovHazanandYaacovHazan ba18105722 Redis 7.2.7 2025-01-06 16:03:47 +02:00
ee4f0c5af7 Deprecate ubuntu lunar and macos-12 in workflows (#13669)
1. Ubuntu Lunar reached End of Life on January 25, 2024, so upgrade the
ubuntu version to plucky in action `test-ubuntu-jemalloc-fortify` to
pass the daily CI
2. The macOS-12 environment is deprecated so upgrade macos-12 to
macos-13 in daily CI

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2025-01-06 16:03:47 +02:00
YaacovHazanandYaacovHazan e344b2b587 Fix LUA garbage collector (CVE-2024-46981)
Reset GC state before closing the lua VM to prevent user data
to be wrongly freed while still might be used on destructor callbacks.
2025-01-06 16:03:47 +02:00
YaacovHazanandYaacovHazan 15e212bf69 Fix Read/Write key pattern selector (CVE-2024-51741)
The '%' rule must contain one or both of R/W
2025-01-06 16:03:47 +02:00
941ca6c07f Upgrade action/checkout version and add old-chain CI actions to test gcc4.8 (#13394)
https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

Due to GitHub removing support for CentOS 7 in GitHub Actions, all
actions utilizing CentOS 7 need to be upgraded, upgrade the centos
version from `contos:7` to `quay.io/centos/centos:stream9` which is the
official RedHat centos container.

Create some new actions named `old-chain` to verify support for gcc 4.8.

This PR also includes the upgrade of actions/checkout from version 3 to
version 4.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2025-01-06 16:03:47 +02:00
Yossi GottliebandYaacovHazan ffde2cf74d Use cross-platform-actions for FreeBSD support. (#12732)
This change overcomes many stability issues experienced with the
vmactions action.

We need to limit VMs to 8GB for better stability, as the 13GB default
seems to hang them occasionally.

Shell code has been simplified since this action seem to use `bash -e`
which will abort on non-zero exit codes anyway.
2025-01-06 16:03:47 +02:00
52b6c0a27b Avoid cluster.nodes load corruption due to shard-id generation (#13468)
PR #13428 doesn't fully resolve an issue where corruption errors can
still occur on loading of cluster.nodes file - seen on upgrade where
there were no shard_ids (from old Redis), 7.2.5 loading generated new
random ones, and persisted them to the file before gossip/handshake
could propagate the correct ones (or some other nodes unreachable).
This results in a primary/replica having differing shard_id in the
cluster.nodes and then the server cannot startup - reports corruption.

This PR builds on #13428 by simply ignoring the replica's shard_id in
cluster.nodes (if it exists), and uses the replica's primary's shard_id.
Additional handling was necessary to cover the case where the replica
appears before the primary in cluster.nodes, where it will first use a
generated shard_id for the primary, and then correct after it loads the
primary cluster.nodes entry.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2025-01-06 16:03:47 +02:00
debing.sunandYaacovHazan a904067f6f Fix incorrect lag due to trimming stream via XTRIM command (#13473)
## Describe
When using the `XTRIM` command to trim a stream, it does not update the
maximal tombstone (`max_deleted_entry_id`). This leads to an issue where
the lag calculation incorrectly assumes that there are no tombstones
after the consumer group's last_id, resulting in an inaccurate lag.

The reason XTRIM doesn't need to update the maximal tombstone is that it
always trims from the beginning of the stream. This means that it
consistently changes the position of the first entry, leading to the
following scenarios:

1) First entry trimmed after maximal tombstone:
If the first entry is trimmed to a position after the maximal tombstone,
all tombstones will be before the first entry, so they won't affect the
consumer group's lag.

2) First entry trimmed before maximal tombstone:
If the first entry is trimmed to a position before the maximal
tombstone, the maximal tombstone will not be updated.

## Solution
Therefore, this PR optimizes the lag calculation by ensuring that when
both the consumer group's last_id and the maximal tombstone are behind
the first entry, the consumer group's lag is always equal to the number
of remaining elements in the stream.

Supplement to PR https://github.com/redis/redis/pull/13338
2025-01-06 16:03:47 +02:00
3ce29e0b11 Pass extensions to node if extension processing is handled by it (#13465)
This PR is based on the commits from PR
https://github.com/valkey-io/valkey/pull/52.
Ref: https://github.com/redis/redis/pull/12760
Close https://github.com/redis/redis/issues/13401
This PR will replace https://github.com/redis/redis/pull/13449

Fixes compatibilty of Redis cluster (7.2 - extensions enabled by
default) with older Redis cluster (< 7.0 - extensions not handled) .

With some of the extensions enabled by default in 7.2 version, new nodes
running 7.2 and above start sending out larger clusterbus message
payload including the ping extensions. This caused an incompatibility
with node running engine versions < 7.0. Old nodes (< 7.0) would receive
the payload from new nodes (> 7.2) would observe a payload length
(totlen) > (estlen) and would perform an early exit and won't process
the message.

This fix does the following things:
1. Always set `CLUSTERMSG_FLAG0_EXT_DATA`, because during the meet
phase, we do not know whether the connected node supports ext data, we
need to make sure that it knows and send back its ext data if it has.
2. If another node does not support ext data, we will not send it ext
data to avoid the handshake failure due to the incorrect payload length.

Note: A successful `PING`/`PONG` is required as a sender for a given
node to be marked as `CLUSTERMSG_FLAG0_EXT_DATA` and then extensions
message
will be sent to it. This could cause a slight delay in receiving the
extensions message(s).

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
Co-authored-by: Harkrishn Patro <harkrisp@amazon.com>

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
Co-authored-by: Harkrishn Patro <harkrisp@amazon.com>
2025-01-06 16:03:47 +02:00
debing.sunandYaacovHazan 17be6d92e8 Ensure validity of myself as master or replica when loading cluster config (#13443)
First, we need to ensure that `curmaster` in
`clusterUpdateSlotsConfigWith()` is not NULL in the line
https://github.com/redis/redis/blob/82f00f5179720c8cee6cd650763d184ba943be92/src/cluster_legacy.c#L2320
otherwise, it will crash in the
https://github.com/redis/redis/blob/82f00f5179720c8cee6cd650763d184ba943be92/src/cluster_legacy.c#L2395

So when loading cluster node config, we need to ensure that the
following conditions are met:
1. A node must be at least one of the master or replica.
2. If a node is a replica, its master can't be NULL.
2025-01-06 16:03:47 +02:00
debing.sunandYaacovHazan f4a6721dfd Fix CLUSTER SHARDS command returns empty array (#13422)
Close https://github.com/redis/redis/issues/13414

When the cluster's master node fails and is switched to another node,
the first node in the shard node list (the old master) is no longer
valid.
Add a new method clusterGetMasterFromShard() to obtain the current
master.
2025-01-06 16:03:47 +02:00
debing.sunandYaacovHazan 0e06e67a36 Fix incorrect lag field in XINFO when tombstone is after the last_id of consume group (#13338)
Fix #13337

Ths PR fixes fixed two bugs that caused lag calculation errors.
1. When the latest tombstone is before the first entry, the tombstone
may stil be after the last id of consume group.
2. When a tombstone is after the last id of consume group, the group's
counter will be invalid, we should caculate the entries_read by using
estimates.
2025-01-06 16:03:47 +02:00
Oran AgraandYaacovHazan 045617e10a Fix possible crash due to OOM panic on invalid command (#13380)
getKeysUsingKeySpece had the range check AFTER the allocation, of the
keys buffer, which could lead to an OOM panic when invalid arguments are
provided, leading to an overflow.
The allocated memory is only used after the range check, so there's no
risk of buffer overrun.
The OOM panic can happen on 32bit builds, or 64 builds running on
systems with less than 4GB of RAM, and is reachable via the COMMAND
GETKEYSANDFLAGS, and ACL key name validation.
2025-01-06 16:03:47 +02:00
4a92d66ca2 Don't keep global replication buffer reference for replicas marked CLIENT_CLOSE_ASAP (#13363)
In certain situations, we might generate a large number of propagates
(e.g., multi/exec, Lua script, or a single command generating tons of
propagations) within an event loop.
During the process of propagating to a replica, if the replica is
disconnected(marked as CLIENT_CLOSE_ASAP) due to exceeding the output
buffer limit, we should remove its reference to the global replication
buffer to avoid the global replication buffer being unable to be
properly trimmed due to being referenced.

---------

Co-authored-by: oranagra <oran@redislabs.com>
2025-01-06 16:03:47 +02:00
2c8ae06b67 Fix crash due to unblock client during slot migration (#13311)
In #13224, we found a crash during cluster slot migration but don't know
why. So i check all the return C_OK in processCommand to see if we are
missing some duration reset and see this.

This fix is like #12247, when we reject the command, we should reset the
duration. I test it and verify it can fix #13224.

So the reason may because we are using stream block and then during the
slot migration, it got a redirect and then crash the server.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2025-01-06 16:03:47 +02:00
Ted LyngmoandYaacovHazan e2aaa9096f Log the real reason for why posix_fadvise failed (#13246)
`reclaimFilePageCache` did not set `errno` but `rdbSaveInternal` which
is logging the error assumed it did. This makes sure `errno` is set.

Fixes #13245

Signed-off-by: Ted Lyngmo <ted@lyncon.se>
2025-01-06 16:03:47 +02:00
9f823685d2 Have consistent behavior of SPUBLISH within multi/exec like regular command (#13276)
This PR is based on the commits from PR #12944.

Allow SPUBLISH command within multi/exec on replica

Behavior on unstable:

```
127.0.0.1:6380> CLUSTER NODES
39ce8aa20f1f0d91f1a88d976ee1926dfefcdf1a 127.0.0.1:6380@16380 myself,slave 8b0feb120b68aac489d6a5af9c77dc40d71bc792 0 0 0 connected
8b0feb120b68aac489d6a5af9c77dc40d71bc792 127.0.0.1:6379@16379 master - 0 1705091681202 0 connected 0-16383
127.0.0.1:6380> SPUBLISH hello world
(integer) 0
127.0.0.1:6380> MULTI
OK
127.0.0.1:6380(TX)> SPUBLISH hello world
QUEUED
127.0.0.1:6380(TX)> EXEC
(error) MOVED 866 127.0.0.1:6379
```

With this change:

```
127.0.0.1:6380> SPUBLISH hello world
(integer) 0
127.0.0.1:6380> MULTI
OK
127.0.0.1:6380(TX)> SPUBLISH hello world
QUEUED
127.0.0.1:6380(TX)> EXEC
1) (integer) 0
```

---------

Co-authored-by: Harkrishn Patro <harkrisp@amazon.com>
Co-authored-by: oranagra <oran@redislabs.com>
2025-01-06 16:03:47 +02:00
1936746e63 Fix oom-score-adj test due to no permission (#12887)
Fix #12792

On ubuntu 23(lunar), non-root users will not be allowed to change the
oom_score_adj of a process to a value that is too low.
Since terminal's default oom_score_adj is 200, if we run the test on
terminal, we won't be able to set the oom_score_adj of the redis process
to 9 or 22, which is too low.

Reproduction on ubuntu 23(lunar) terminal:
```sh
$ cat /proc/`pgrep redis-server`/oom_score_adj
200
$ echo 100 > /proc/`pgrep redis-server`/oom_score_adj
# success without error
$ echo 99 > /proc/`pgrep redis-server`/oom_score_adj
echo: write error: Permission denied
```

As from the output above, we can only set the minimum oom score of redis
processes to 100.
By modifying the test, make oom_score_adj only increase upwards and not
decrease.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2025-01-06 16:03:47 +02:00
Oran Agra ae6a2aa95c Release Redis 7.2.6 2024-10-02 22:13:33 +03:00
Oran Agra c8649f8e85 Prevent pattern matching abuse (CVE-2024-31228) 2024-10-02 22:13:33 +03:00
Oran Agra b351d5a321 Fix ACL SETUSER Read/Write key pattern selector (CVE-2024-31227)
The '%' rule must contain one or both of R/W
2024-10-02 22:13:33 +03:00
Oran Agra fe8de4313f Fix lua bit.tohex (CVE-2024-31449)
INT_MIN value must be explicitly checked, and cannot be negated.
2024-10-02 22:13:33 +03:00
debing.sunandGitHub 2ad2548747 Fixed crashes due to missed slotToKeyInit() and missed expires_cursor reset (#13315)
this PR fixes two crashes:

1. Fix missing slotToKeyInit() when using `flushdb async` under cluster
mode.
    https://github.com/redis/redis/issues/13205

2. Fix missing expires_cursor reset when stopping active defrag in the
middle of defragment.
    https://github.com/redis/redis/issues/13307
If we stop active defrag in the middle of defragging db->expires, if
`expires_cursor` is not reset to 0, the next time we enable active
defrag again, defragLaterStep(db, ...) will be entered. However, at this
time, `db` has been reset to NULL, which results in crash.

The affected code were removed by #11695 and #13058 in usntable, so we
just need backport this to 7.2.
2024-06-18 18:02:22 +08:00
YaacovHazanandYaacovHazan f60370ce28 Redis 7.2.5 2024-05-19 09:12:35 +03:00
Yanqi LvandYaacovHazan 464aad9ee7 fix wrong data type conversion in zrangeResultBeginStore (#13148)
In `beginResultEmission`, -1 means the result length is not known in
advance. But after #12185, if we pass -1 to `zrangeResultBeginStore`, it
will convert to SIZE_MAX in `zsetTypeCreate` and try to `dictExpand`.
Although `dictExpand` won't succeed because the size overflows, I think
we'd better to avoid this wrong conversion.

This bug can be triggered when the source of `zrangestore` doesn't exist
or we use `zrangestore` command with `byscore` or `bylex`.
The impact is that dst keys will be converted to use skiplist instead of
listpack.

(cherry picked from commit bad33f8738)
2024-05-19 09:12:35 +03:00
BinbinandYaacovHazan 439b8da475 Fix redis-check-aof incorrectly considering data in manifest format as MP-AOF (#12958)
The check in fileIsManifest misjudged the manifest file. For example,
if resp aof contains "file", it will be considered a manifest file and
the check will fail:
```
*3
$3
set
$4
file
$4
file
```

In #12951, if the preamble aof also contains it, it will also fail.
Fixes #12951.

the bug was happening if the the word "file" is mentioned
in the first 1024 lines of the AOF. and now as soon as it finds
a non-comment line it'll break (if it contains "file" or doesn't)

(cherry picked from commit da727ad445)
2024-05-19 09:12:35 +03:00
Matthew DouglassandYaacovHazan 1caaf581f1 Fix conversion of numbers in lua args to redis args (#13115)
Since lua_Number is not explicitly an integer or a double, we need to
make an effort
to convert it as an integer when that's possible, since the string could
later be used
in a context that doesn't support scientific notation (e.g. 1e9 instead
of 100000000).

Since fpconv_dtoa converts numbers with the equivalent of `%f` or `%e`,
which ever is shorter,
this would break if we try to pass a long integer number to a command
that takes integer.
we'll get an implicit conversion to string in Lua, and then the parsing
in getLongLongFromObjectOrReply will fail.

```
> eval "redis.call('hincrby', 'key', 'field', '1000000000')" 0
(nil)
> eval "redis.call('hincrby', 'key', 'field', tonumber('1000000000'))" 0
(error) ERR value is not an integer or out of range script: ac99c32e4daf7e300d593085b611de261954a946, on @user_script:1.
```

Switch to using ll2string if the number can be safely represented as a
long long.

The problem was introduced in #10587 (Redis 7.2).
closes #13113.

---------

Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 5fdaa53d20)
2024-05-19 09:12:35 +03:00
debing.sunandYaacovHazan e46ddde28d Check user's oom_score_adj write permission for oom-score-adj test (#13111)
`CONFIG SET oom-score-adj handles configuration failures` test failed in
some CI jobs today.
Failed CI: https://github.com/redis/redis/actions/runs/8152519326

Not sure why the github action's docker image perssions have changed,
but the issue is similar to #12887,
where we can't assume the range of oom_score_adj that a user can change.

## Solution:
Modify the way of determining whether the current user has no privileges
or not,
instead of relying on whether the user id is 0 or not.

(cherry picked from commit 9738ba9841)
2024-05-19 09:12:35 +03:00
LiiNenandYaacovHazan 8f70fcc65b Fix redis-cli --count (for --scan, --bigkeys, etc) was ignored unless --pattern was also used (#13092)
The --count option for redis-cli has been released in redis 7.2.
https://github.com/redis/redis/pull/12042
But I have found in code, that some logic was missing for using this
'count' option.

```
static redisReply *sendScan(unsigned long long *it) {
    redisReply *reply;

    if (config.pattern)
        reply = redisCommand(context, "SCAN %llu MATCH %b COUNT %d",
            *it, config.pattern, sdslen(config.pattern), config.count);
    else
        reply = redisCommand(context,"SCAN %llu",*it);
```

The intention was being able to using scan count.
But in this case, the --count will be only applied when 'pattern' is
declared.
So, I had fix it simply, to be worked properly - even if --pattern
option is not being used.

I tested it simply with time() command several times, and I could see it
works as intended with this commit.
The examples of test results are below:
```
# unstable build

time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan >/dev/null 2>/dev/null)

real    0m1.287s
user    0m0.011s
sys     0m0.022s

# count is not applied
time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan --count 1000 >/dev/null 2>/dev/null)

real    0m1.117s
user    0m0.011s
sys     0m0.020s

# count is applied with --pattern

time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan --count 1000 --pattern "hash:*" >/dev/null 2>/dev/null)

real    0m0.045s
user    0m0.002s
sys     0m0.002s
```

```
# fix-redis-cli-scan-count build
time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan >/dev/null 2>/dev/null)

real    0m1.084s
user    0m0.008s
sys     0m0.024s

# count is applied even if --pattern is not declared
time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan --count 1000 >/dev/null 2>/dev/null)

real    0m0.043s
user    0m0.000s
sys     0m0.004s

# of course this also applied
time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan --count 1000 --pattern "hash:*" >/dev/null 2>/dev/null)

real    0m0.031s
user    0m0.002s
sys     0m0.002s
```

Thanks a lot.

(cherry picked from commit 763827c981)
2024-05-19 09:12:35 +03:00
BinbinandYaacovHazan 8d3a1c978a Increase tolerance range to block reprocess tests to avoid timing issues (#13053)
These tests have all failed in daily CI:
```
*** [err]: Blocking XREADGROUP for stream key that has clients blocked on stream - reprocessing command in tests/unit/type/stream-cgroups.tcl
Expected '1101' to be between to '1000' and '1100' (context: type eval line 23 cmd {assert_range [expr $end-$start] 1000 1100} proc ::test)

*** [err]: BLPOP unblock but the key is expired and then block again - reprocessing command in tests/unit/type/list.tcl
Expected '1101' to be between to '1000' and '1100' (context: type eval line 23 cmd {assert_range [expr $end-$start] 1000 1100} proc ::test)

*** [err]: BZPOPMIN unblock but the key is expired and then block again - reprocessing command in tests/unit/type/zset.tcl
Expected '1103' to be between to '1000' and '1100' (context: type eval line 23 cmd {assert_range [expr $end-$start] 1000 1100} proc ::test)
```

Increase the range to avoid failures, and improve the comment to be
clearer.
tests was introduced in #13004.

(cherry picked from commit 32f44da510)
2024-05-19 09:12:35 +03:00
debing.sunandYaacovHazan c34e64848f Fix crash due to merge of quicklist node introduced by #12955 (#13040)
Fix two crash introducted by #12955

When a quicklist node can't be inserted and split, we eventually merge
the current node with its neighboring
nodes after inserting, and compress the current node and its siblings.

1. When the current node is merged with another node, the current node
may become invalid and can no longer be used.

   Solution: let `_quicklistMergeNodes()` return the merged nodes.

3. If the current node is a LZF quicklist node, its recompress will be
1. If the split node can be merged with a sibling node to become head or
tail, recompress may cause the head and tail to be compressed, which is
not allowed.

    Solution: always recompress to 0 after merging.

(cherry picked from commit 1e8dc1da0d)
2024-05-19 09:12:35 +03:00
debing.sunandYaacovHazan 1be66b98f5 Prevent LSET command from causing quicklist plain node size to exceed 4GB (#12955)
Fix #12864

The main reason for this crash is that when replacing a element of a
quicklist packed node with lpReplace() method,
if the final size is larger than 4GB, lpReplace() will fail and returns
NULL, causing `node->entry` to be incorrectly set to NULL.

Since the inserted data is not a large element, we can't just replace it
like a large element, first quicklistInsertAfter()
and then quicklistDelIndex(), because the current node may be merged and
invalidated in quicklistInsertAfter().

The solution of this PR:
When replacing a node fails (listpack exceeds 4GB), split the current
node, create a new node to put in the middle, and try to merge them.
This is the same as inserting a large element.
In the worst case, its size will not exceed 4GB.

(cherry picked from commit 1f00c951c2)
2024-05-19 09:12:35 +03:00
BinbinandYaacovHazan 423d1909c5 Fix timeout not being set in module blockClient case (#13011)
This was introduced in #13004, missing this assignment.
It causes timeout to be a random value (may be less than now),
and then in `Unblock by timer` test, the client is unblocked
and then it call timeout_callback, since the callback is NULL,
the server will crash.

The crash stack is:
```
beforesleep
handleBlockedClientsTimeout
checkBlockedClientTimeout
unblockClientOnTimeout
replyToBlockedClientTimedOut
moduleBlockedClientTimedOut
-- the timeout_callback is NULL, invalidFunctionWasCalled
bc->timeout_callback(&ctx,(void**)c->argv,c->argc);
```

(cherry picked from commit 45a35a79c7)
2024-05-19 09:12:35 +03:00
BinbinandYaacovHazan 1bda797dc0 Fix blocking commands timeout is reset due to re-processing command (#13004)
In #11012, we will reprocess command when client is unblocked on keys,
in some blocking commands, for example, in the XREADGROUP BLOCK
scenario,
because of the re-processing command, we will recalculate the block
timeout,
causing the blocking time to be reset.

This commit add a new CLIENT_REPROCESSING_COMMAND clent flag, explicitly
let the command know that it is being re-processed, later in
blockForKeys
we will not reset the timeout.

Affected BLOCK cases:
- list / zset / stream, added test cases for each.

Unaffected cases:
- module (never re-process the commands).
- WAIT / WAITAOF (never re-process the commands).

Fixes #12998.

(cherry picked from commit 492021db95)
2024-05-19 09:12:35 +03:00
Oran AgraandYaacovHazan d5bae505ba update redis-check-rdb types (#12969)
seems that we forgot to update the array in redis-check rdb.

(cherry picked from commit f9a0eb60f7)
2024-05-19 09:12:35 +03:00
bentottenandYaacovHazan 099a2f40ec When one shard, sole primary node marks potentially failed replica as FAIL instead of PFAIL (#12824)
Fixes issue where a single primary cannot mark a replica as failed in a
single-shard cluster.

(cherry picked from commit b3aaa0a136)
2024-05-19 09:12:35 +03:00
BinbinandYaacovHazan 4bd614c52d Add announced-endpoints test to all_tests and fix tls related tests (#12927)
The test was introduced in #10745, but we forgot to add it to the
test_helper.tcl, so our CI did not actually run it. This PR adds it
and ensures it passes CI tests.

(cherry picked from commit b351a04b1e)
2024-05-19 09:12:35 +03:00
Oran Agra d2c8a4b91e Redis 7.2.4 2024-01-09 13:51:49 +02:00
BinbinandOran Agra 85408b7391 Fix CLUSTER SHARDS crash in 7.0/7.2 mixed clusters where shard ids are not sync (#12832)
Crash reported in #12695. In the process of upgrading the cluster from
7.0 to 7.2, because the 7.0 nodes will not gossip shard id, in 7.2 we
will rely on shard id to build the server.cluster->shards dict.

In some cases, for example, the 7.0 master node and the 7.2 replica node.
From the view of 7.2 replica node, the cluster->shards dictionary does not
have its master node. In this case calling CLUSTER SHARDS on the 7.2 replica
node may crash.

We should fix the underlying assumption of updateShardId, which is that the
shard dict should be always in sync with the node's shard_id. The fix was
suggested by PingXie, see more details in #12695.

(cherry picked from commit 5b0c6a8255)
2024-01-09 13:51:49 +02:00
BinbinandOran Agra 5a2f4a1e94 Use shard-id of the master if the replica does not support shard-id (#12805)
If there are nodes in the cluster that do not support shard-id, they
will gossip shard-id. From the perspective of nodes that support shard-id,
their shard-id is meaningless (since shard-id is randomly generated when
we create a node.)

Nodes that support shard-id will save the shard-id information in nodes.conf.
If the node is restarted according to nodes.conf, the server will report a
corrupted cluster config file error. Because auxShardIdSetter will reject
configurations with inconsistent master-replica shard-ids.

A cluster-wide consensus for the node's shard_id is not necessary. The key
is maintaining consistency of the shard_id on each individual 7.2 node.
As the cluster progressively upgrades to version 7.2, we can expect the
shard_ids across all nodes to naturally converge and align.

In this PR, when processing the gossip, if sender is a replica and does not
support shard-id, set the shard_id to the shard_id of its master.

(cherry picked from commit 4cae66f5e8)
2024-01-09 13:51:49 +02:00
BinbinandOran Agra c4776cafcf Un-register notification and server event when RedisModule_OnLoad fails (#12809)
When we register notification or server event in RedisModule_OnLoad, but
RedisModule_OnLoad eventually fails, triggering notification or server
event
will cause the server to crash.

If the loading fails on a later stage of moduleLoad, we do call
moduleUnload
which handles all un-registration, but when it fails on the
RedisModule_OnLoad
call, we only un-register several specific things and these were
missing:

- moduleUnsubscribeNotifications
- moduleUnregisterFilters
- moduleUnsubscribeAllServerEvents

Refactored the code to reuse the code from moduleUnload.

Fixes #12808.

(cherry picked from commit d6f19539d2)
2024-01-09 13:51:49 +02:00
Meir Shpilraien (Spielrein)andOran Agra 4cbf903083 Before evicted and before expired server events are not executed inside an execution unit. (#12733)
Redis 7.2 (#9406) introduced a new modules event, `RedisModuleEvent_Key`.
This new event allows the module to read the key data just before it is removed
from the database (either deleted, expired, evicted, or overwritten).

When the key is removed from the database, either by active expire or eviction.
The new event was not called as part of an execution unit. This can cause an
issue if the module registers a post notification job inside the event. This job will
not be executed atomically with the expiration/eviction operation and will not
replicated inside a Multi/Exec. Moreover, the post notification job will be executed
right after the event where it is still not safe to perform any write operation, this will
violate the promise that post notification job will be called atomically with the
operation that triggered it and **only when it is safe to write**.

This PR fixes the issue by wrapping each expiration/eviction of a key with an execution
unit. This makes sure the entire operation will run atomically and all the post notification
jobs will be executed at the end where it is safe to write.

Tests were modified to verify the fix.

(cherry picked from commit 0ffb9d2ea9)
2024-01-09 13:51:49 +02:00
SankarandOran Agra a91b57eff7 Clear owner_not_claiming_slot bit for the slot in clusterDelSlot (#12564)
Clear owner_not_claiming_slot bit for the slot in clusterDelSlot to keep it
consistent with slot ownership information.

(cherry picked from commit 8cdeddc81c)
2024-01-09 13:51:49 +02:00
BinbinandOran Agra 8359ce266c Use CLZ in _dictNextExp to get the next power of two (#12815)
In the past, we did not call _dictNextExp frequently. It was only
called when the dictionary was expanded.

Later, dictTypeExpandAllowed was introduced in #7954, which is 6.2.
For the data dict and the expire dict, we can check maxmemory before
actually expanding the dict. This is a good optimization to avoid
maxmemory being exceeded due to the dict expansion.

And in #11692, we moved the dictTypeExpandAllowed check before the
threshold check, this caused a bit of performance degradation, every
time a key is added to the dict, dictTypeExpandAllowed is called to
check.

The main reason for degradation is that in a large dict, we need to
call _dictNextExp frequently, that is, every time we add a key, we
need to call _dictNextExp once. Then the threshold is checked to see
if the dict needs to be expanded. We can see that the order of checks
here can be optimized.

So we moved the dictTypeExpandAllowed check back to after the threshold
check in #12789. In this way, before the dict is actually expanded (that
is, before the threshold is reached), we will not do anything extra
compared to before, that is, we will not call _dictNextExp frequently.

But note we'll still hit the degradation when we over the thresholds.
When the threshold is reached, because #7954, we may delay the dict
expansion due to maxmemory limitations. In this case, we will call
_dictNextExp every time we add a key during this period.

This PR use CLZ in _dictNextExp to get the next power of two. CLZ (count
leading zeros) can easily give you the next power of two. It should be
noted that we have actually introduced the use of __builtin_clzl in
#8687,
which is 7.0. So i suppose all the platforms we use have it (even if the
CPU doesn't have an instruction).

We build 67108864 (2**26) keys through DEBUG POPULTE, which will use
approximately 5.49G memory (used_memory:5898522936). If expansion is
triggered, the additional hash table will consume approximately 1G
memory (2 ** 27 * 8). So we set maxmemory to 6871947673 (that is, 6.4G),
which will be less than 5.49G + 1G, so we will delay the dict rehash
while addint the keys.

After that, each time an element is added to the dict, an allow check
will be performed, that is, we can frequently call _dictNextExp to test
the comparison before and after the optimization. Using DEBUG HTSTATS 0
to
check and make sure that our dict expansion is dealyed.

Using `./src/redis-server redis.conf --save "" --maxmemory 6871947673`.
Using `./src/redis-benchmark -P 100 -r 1000000000 -t set -n 5000000`.
After ten rounds of testing:
```
unstable:           this PR:
769585.94           816860.00
771724.00           818196.69
775674.81           822368.44
781983.12           822503.69
783576.25           828088.75
784190.75           828637.75
791389.69           829875.50
794659.94           835660.69
798212.00           830013.25
801153.62           833934.56
```

We can see there is about 4-5% performance improvement in this case.

(cherry picked from commit 22cc9b5122)
2024-01-09 13:51:49 +02:00
BinbinandOran Agra 856c8a47d0 Optimize dict expand check, move allow check after the thresholds check (#12789)
dictExpandAllowed (for the main db dict and the expire dict) seems to
involve a few function calls and memory accesses, and we can do it only
after the thresholds checks and can get some performance improvements.

A simple benchmark test: there are 11032768 fixed keys in the database,
start a redis-server with `--maxmemory big_number --save ""`,
start a redis-benchmark with `-P 100 -r 1000000000 -t set -n 5000000`,
collect `throughput summary: n requests per second` result.

After five rounds of testing:
```
unstable     this PR
848032.56    897988.56
854408.69    913408.88
858663.94    914076.81
871839.56    916758.31
882612.56    920640.75
```

We can see a 5% performance improvement in general condition.
But note we'll still hit the degradation when we over the thresholds.

(cherry picked from commit 463476933c)
2024-01-09 13:51:49 +02:00
Oran Agra 5f5f298a4a Fix possible corruption in sdsResize (CVE-2023-41056)
#11766 introduced a bug in sdsResize where it could forget to update
the sds type in the sds header and then cause an overflow in sdsalloc.
it looks like the only implication of that is a possible assertion in HLL,
but it's hard to rule out possible heap corruption issues with clientsCronResizeQueryBuffer
2024-01-09 13:51:49 +02:00
Oran Agra 7f4bae8176 Redis 7.2.3 2023-11-01 14:38:13 +02:00
Viktor SöderqvistandOran Agra cee6d309d9 Don't crash when adding a forgotten node to blacklist twice (#12702)
Add a defensive checks to prevent double freeing a node from the cluster blacklist.

(cherry picked from commit 8d675950e6)
2023-11-01 14:38:13 +02:00
Oran Agra cf013b83df Fix fd leak causing deleted files to remain open and eat disk space (#12693)
This was introduced in v7.2 by #11248

(cherry picked from commit ba900f6cb8)
2023-11-01 14:38:13 +02:00
Oran Agra d424259a12 Redis 7.2.2 2023-10-18 10:44:10 +03:00
BinbinandOran Agra 7f6de086fe Fix crash when running rebalance command in a mixed cluster of 7.0 and 7.2 (#12604)
In #10536, we introduced the assert, some older versions of servers
(like 7.0) doesn't gossip shard_id, so we will not add the node to
cluster->shards, and node->shard_id is filled in randomly and may not
be found here.

It causes that if we add a 7.2 node to a 7.0 cluster and allocate slots
to the 7.2 node, the 7.2 node will crash when it hits this assert. Somehow
like #12538.

In this PR, we remove the assert and replace it with an unconditional removal.

(cherry picked from commit e5ef161374)
2023-10-18 10:44:10 +03:00
BinbinandOran Agra 8c6ebf84ae Fix redis-cli pubsub_mode and connect minor prompt / crash issue (#12571)
When entering pubsub mode and using the redis-cli only
connect command, we need to reset pubsub_mode because
we switch to a different connection.

This will affect the prompt when the connection is successful,
and redis-cli will crash when the connect fails:
```
127.0.0.1:6379> subscribe ch
1) "subscribe"
2) "ch"
3) (integer) 1
127.0.0.1:6379(subscribed mode)> connect 127.0.0.1 6380
127.0.0.1:6380(subscribed mode)> ping
PONG
127.0.0.1:6380(subscribed mode)> connect a b
Could not connect to Redis at a:0: Name or service not known
Segmentation fault
```

(cherry picked from commit 4de4fcf280)
2023-10-18 10:44:10 +03:00
guybe7andOran Agra ba113c8337 WAITAOF: Update fsynced_reploff_pending just before starting the initial AOFRW fork (#12620)
If we set `fsynced_reploff_pending` in `startAppendOnly`, and the fork doesn't start
immediately (e.g. there's another fork active at the time), any subsequent commands
will increment `server.master_repl_offset`, but will not cause a fsync (given they were
executed before the fork started, they just ended up in the RDB part of it)
Therefore, any WAITAOF will wait on the new master_repl_offset, but it will time out
because no fsync will be executed.

Release notes:
```
WAITAOF could timeout in the absence of write traffic in case a new AOF is created and
an AOFRW can't immediately start.
This can happen by the appendonly config is changed at runtime, but also after FLUSHALL,
and replica full sync.
```

(cherry picked from commit bfa3931a04)
2023-10-18 10:44:10 +03:00
Nir RattnerandOran Agra 348eda81e0 Fix overflow calculation for next timer event (#12474)
The `retval` variable is defined as an `int`, so with 4 bytes, it cannot properly represent
microsecond values greater than the equivalent of about 35 minutes.

This bug shouldn't impact standard Redis behavior because Redis doesn't have timer
events that are scheduled as far as 35 minutes out, but it may affect custom Redis modules
which interact with the event timers via the RM_CreateTimer API.

The impact is that `usUntilEarliestTimer` may return 0 for as long as `retval` is scaled to
an overflowing value. While `usUntilEarliestTimer` continues to return `0`, `aeApiPoll`
will have a zero timeout, and so Redis will use significantly more CPU iterating through
its event loop without pause. For timers scheduled far enough into the future, Redis will
cycle between ~35 minute periods of high CPU usage and ~35 minute periods of standard
CPU usage.

(cherry picked from commit 24187ed8e3)
2023-10-18 10:44:10 +03:00
Chen TianjieandOran Agra 3f6bd4fc25 Use server.current_client to decide whether cluster commands should return TLS info. (#12569)
Starting a change in #12233 (released in 7.2), CLUSTER commands use client's
connection to decide whether to return TLS port or non-TLS port, but commands
called by Lua script and module's RM_Call don't have a real client with connection,
and would currently be regarded as non-TLS connections.

We can use server.current_client instead when it is available. When it is not (module calls
commands without a real client), we may see this as an undefined behavior, and return null
or default port (currently in this PR it returns default port, judged by server.tls_cluster).

(cherry picked from commit 2aad03fa39)
2023-10-18 10:44:10 +03:00
BinbinandOran Agra d6601de73f Fix that slot return in CLUSTER SHARDS should be integer (#12561)
An unintentional change was introduced in #10536, we used
to use addReplyLongLong and now it is addReplyBulkLonglong,
revert it back the previous behavior.

(cherry picked from commit 4031a18732)
2023-10-18 10:44:10 +03:00
JachinandOran Agra 00023873a8 Fix compile on macOS 13 (#12611)
Use the __MAC_OS_X_VERSION_MIN_REQUIRED macro to detect the
macOS system version instead of using MAC_OS_X_VERSION_10_6.

From MacOSX14.0.sdk, the default definitions of MAC_OS_X_VERSION_xxx have
been removed in usr/include/AvailabilityMacros.h. It includes AvailabilityVersions.h,
where the following condition must be met:
`#if (!defined(_POSIX_C_SOURCE) && !defined(_XOPEN_SOURCE)) || defined(_DARWIN_C_SOURCE)`
Only then will MAC_OS_X_VERSION_xxx be defined.
However, in the project, _DARWIN_C_SOURCE is not defined, which leads to the
loss of the definition for MAC_OS_X_VERSION_10_6.

(cherry picked from commit a2b0701d2c)
2023-10-18 10:44:10 +03:00
BinbinandOran Agra 413c8be5ad Support NO ONE block in REPLICAOF command json (#12633)
The current commands.json doesn't mention the special NO ONE arguments.
This change is also applied to SLAVEOF

(cherry picked from commit 8d92f7f2b7)
2023-10-18 10:44:10 +03:00
Yossi GottliebandOran Agra 1119ecae6f Fix issue of listen before chmod on Unix sockets (CVE-2023-45145)
Before this commit, Unix socket setup performed chmod(2) on the socket
file after calling listen(2). Depending on what umask is used, this
could leave the file with the wrong permissions for a short period of
time. As a result, another process could exploit this race condition and
establish a connection that would otherwise not be possible.

We now make sure the socket permissions are set up prior to calling
listen(2).
2023-10-18 10:44:10 +03:00
Oran Agra cc244370a2 Redis 7.2.1 2023-09-06 20:56:15 +03:00
nihohitandOran Agra 3e7523aceb Update command tips on more admin / configuration commands (#12545)
Updated the command tips for ACL SAVE / SETUSER / DELUSER, CLIENT SETNAME / SETINFO, and LATENCY RESET.
The tips now match CONFIG SET, since there's a similar behavior for all of these commands - the
user expects to update the various configurations & states on all nodes, not only on a single, random node.
For LATENCY RESET the response tip is now agg_sum.

Co-authored-by: Shachar Langbeheim <shachlan@amazon.com>
(cherry picked from commit 90e9fc387c)
2023-09-06 20:56:15 +03:00
secwallandOran Agra 459acdeeb1 Check shard_id pointer validity in updateShardId (#12538)
When connecting between a 7.0 and 7.2 cluster, the 7.0 cluster will not populate the shard_id field, which is expect on the 7.2 cluster. This is not intended behavior, as the 7.2 cluster is supposed to use a temporary shard_id while the node is in the upgrading state, but it wasn't being correctly set in this case.

(cherry picked from commit a2046c1eb1)
2023-09-06 20:56:15 +03:00
BinbinandOran Agra 0b2cbf138e Update sort_ro reply_schema to mention the null reply (#12534)
Also added a test to cover this case, so this can
cover the reply schemas check.

(cherry picked from commit 9ce8c54d74)
2023-09-06 20:56:15 +03:00
bodong.ybdandOran Agra 9e505e6cd8 Fix sort_ro get-keys function return wrong key number (#12522)
Before:
```
127.0.0.1:6379> command getkeys sort_ro key
(empty array)
127.0.0.1:6379>
```
After:
```
127.0.0.1:6379> command getkeys sort_ro key
1) "key"
127.0.0.1:6379>
```

(cherry picked from commit b59f53efb3)
2023-09-06 20:56:15 +03:00
nihohitandOran Agra 68305c5b99 Align CONFIG RESETSTAT/REWRITE tips with SET. (#12530)
Since the three commands have similar behavior (change config, return
OK), the tips that govern how they should behave should be similar.

Co-authored-by: Shachar Langbeheim <shachlan@amazon.com>
(cherry picked from commit 4b281ce519)
2023-09-06 20:56:15 +03:00
Oran AgraandGitHub 29622276ec Release Redis 7.2.0 GA 2023-08-15 12:38:36 +03:00
Oran Agra 3d1d3e23ac Redis 7.2.0 2023-08-15 10:04:17 +03:00
Oran Agra 941cdc2759 Merge branch 'unstable' into 7.2 to prepare for 7.2.0 GA 2023-08-15 09:43:09 +03:00
Oran AgraandGitHub 1a58981868 Release Redis 7.2 RC3 2023-07-10 14:55:20 +03:00
Oran Agra c7b3ce90f1 Redis 7.2 RC3 2023-07-10 11:52:53 +03:00
Oran Agra 819d291bd7 Merge remote-tracking branch 'origin/unstable' into 7.2 2023-07-10 10:26:53 +03:00
Oran AgraandGitHub a51eb05b18 Release Redis 7.2 RC2 2023-05-15 13:08:15 +03:00
Oran Agra 986dbf716e Redis 7.2 RC2 2023-05-15 09:53:36 +03:00
Oran Agra d4439bd41c Merge remote-tracking branch 'origin/unstable' into 7.2 2023-05-15 09:50:01 +03:00
Oran Agra e26a769d96 Redis 7.2 RC1 2023-03-22 15:58:15 +02:00
95 changed files with 2620 additions and 450 deletions
+27 -10
View File
@@ -7,7 +7,7 @@ jobs:
test-ubuntu-latest:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
# Fail build if there are warnings
# build with TLS just for compilation coverage
@@ -28,7 +28,7 @@ jobs:
test-sanitizer-address:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
# build with TLS module just for compilation coverage
run: make SANITIZER=address REDIS_CFLAGS='-Werror' BUILD_TLS=module
@@ -43,7 +43,7 @@ jobs:
runs-on: ubuntu-latest
container: debian:buster
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
run: |
apt-get update && apt-get install -y build-essential
@@ -52,14 +52,14 @@ jobs:
build-macos-latest:
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
run: make REDIS_CFLAGS='-Werror'
build-32bit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
run: |
sudo apt-get update && sudo apt-get install libc6-dev-i386
@@ -68,17 +68,34 @@ jobs:
build-libc-malloc:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
run: make REDIS_CFLAGS='-Werror' MALLOC=libc
build-centos7-jemalloc:
build-centos-jemalloc:
runs-on: ubuntu-latest
container: centos:7
container: quay.io/centos/centos:stream9
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
run: |
yum -y install gcc make
dnf -y install which gcc make
make REDIS_CFLAGS='-Werror'
build-old-chain-jemalloc:
runs-on: ubuntu-latest
container: ubuntu:20.04
steps:
- uses: actions/checkout@v4
- name: make
run: |
apt-get update
apt-get install -y gnupg2
echo "deb http://archive.ubuntu.com/ubuntu/ xenial main" >> /etc/apt/sources.list
echo "deb http://archive.ubuntu.com/ubuntu/ xenial universe" >> /etc/apt/sources.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 40976EAF437D05B5
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3B4FE6ACC0B21F32
apt-get update
apt-get install -y make gcc-4.8
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 100
make CC=gcc REDIS_CFLAGS='-Werror'
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Initialize CodeQL
uses: github/codeql-action/init@v2
+236 -75
View File
@@ -11,7 +11,7 @@ on:
inputs:
skipjobs:
description: 'jobs to skip (delete the ones you wanna keep, do not leave empty)'
default: 'valgrind,sanitizer,tls,freebsd,macos,alpine,32bit,iothreads,ubuntu,centos,malloc,specific,fortify,reply-schema'
default: 'valgrind,sanitizer,tls,freebsd,macos,alpine,32bit,iothreads,ubuntu,centos,malloc,specific,fortify,reply-schema,oldTC'
skiptests:
description: 'tests to skip (delete the ones you wanna keep, do not leave empty)'
default: 'redis,modules,sentinel,cluster,unittest'
@@ -47,7 +47,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -76,7 +76,6 @@ jobs:
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'fortify')
container: ubuntu:lunar
timeout-minutes: 14400
steps:
- name: prep
@@ -88,17 +87,16 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
apt-get update && apt-get install -y make gcc-13
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 100
apt-get update && apt-get install -y make gcc
make CC=gcc REDIS_CFLAGS='-Werror -DREDIS_TEST -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3'
- name: testprep
run: apt-get install -y tcl8.6 tclx procps
run: sudo apt-get install -y tcl8.6 tclx procps
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
@@ -131,7 +129,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -168,7 +166,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -205,7 +203,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -249,7 +247,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -293,7 +291,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -337,7 +335,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -369,7 +367,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -447,7 +445,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -477,7 +475,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -512,7 +510,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -542,7 +540,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -582,7 +580,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -629,7 +627,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -655,12 +653,12 @@ jobs:
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: ./src/redis-server test all --accurate
test-centos7-jemalloc:
test-centos-jemalloc:
runs-on: ubuntu-latest
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'centos')
container: centos:7
container: quay.io/centos/centos:stream9
timeout-minutes: 14400
steps:
- name: prep
@@ -672,16 +670,18 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
yum -y install gcc make
dnf -y install which gcc make
make REDIS_CFLAGS='-Werror'
- name: testprep
run: yum -y install which tcl tclx
run: |
dnf -y install epel-release
dnf -y install tcl tcltls procps-ng /usr/bin/kill
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
@@ -695,12 +695,12 @@ jobs:
if: true && !contains(github.event.inputs.skiptests, 'cluster')
run: ./runtest-cluster ${{github.event.inputs.cluster_test_args}}
test-centos7-tls-module:
test-centos-tls-module:
runs-on: ubuntu-latest
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'tls')
container: centos:7
container: quay.io/centos/centos:stream9
timeout-minutes: 14400
steps:
- name: prep
@@ -712,18 +712,18 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
yum -y install centos-release-scl epel-release
yum -y install devtoolset-7 openssl-devel openssl
scl enable devtoolset-7 "make BUILD_TLS=module REDIS_CFLAGS='-Werror'"
dnf -y install which gcc make openssl-devel openssl
make BUILD_TLS=module REDIS_CFLAGS='-Werror'
- name: testprep
run: |
yum -y install tcl tcltls tclx
dnf -y install epel-release
dnf -y install tcl tcltls procps-ng /usr/bin/kill
./utils/gen-test-certs.sh
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
@@ -742,12 +742,12 @@ jobs:
run: |
./runtest-cluster --tls-module ${{github.event.inputs.cluster_test_args}}
test-centos7-tls-module-no-tls:
test-centos-tls-module-no-tls:
runs-on: ubuntu-latest
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'tls')
container: centos:7
container: quay.io/centos/centos:stream9
timeout-minutes: 14400
steps:
- name: prep
@@ -759,18 +759,18 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
yum -y install centos-release-scl epel-release
yum -y install devtoolset-7 openssl-devel openssl
scl enable devtoolset-7 "make BUILD_TLS=module REDIS_CFLAGS='-Werror'"
dnf -y install which gcc make openssl-devel openssl
make BUILD_TLS=module REDIS_CFLAGS='-Werror'
- name: testprep
run: |
yum -y install tcl tcltls tclx
dnf -y install epel-release
dnf -y install tcl tcltls procps-ng /usr/bin/kill
./utils/gen-test-certs.sh
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
@@ -805,7 +805,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -834,7 +834,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -860,7 +860,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -871,7 +871,7 @@ jobs:
run: ./runtest-cluster ${{github.event.inputs.cluster_test_args}}
test-freebsd:
runs-on: macos-12
runs-on: macos-13
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'freebsd') && !(contains(github.event.inputs.skiptests, 'redis') && contains(github.event.inputs.skiptests, 'modules'))
@@ -886,24 +886,28 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: test
uses: vmactions/freebsd-vm@v0.3.1
uses: cross-platform-actions/action@v0.21.1
env:
MAKE: gmake
with:
usesh: true
sync: rsync
copyback: false
prepare: pkg install -y bash gmake lang/tcl86 lang/tclx
run: >
gmake || exit 1 ;
if echo "${{github.event.inputs.skiptests}}" | grep -vq redis ; then ./runtest --verbose --timeout 2400 --no-latency --dump-logs ${{github.event.inputs.test_args}} || exit 1 ; fi ;
if echo "${{github.event.inputs.skiptests}}" | grep -vq modules ; then MAKE=gmake ./runtest-moduleapi --verbose --timeout 2400 --no-latency --dump-logs ${{github.event.inputs.test_args}} || exit 1 ; fi ;
operating_system: freebsd
environment_variables: MAKE
version: 13.2
memory: 8GB
shell: bash
run: |
sudo pkg install -y bash gmake lang/tcl86 lang/tclx
gmake
if echo "${{github.event.inputs.skiptests}}" | grep -vq redis ; then ./runtest --verbose --timeout 2400 --no-latency --dump-logs ${{github.event.inputs.test_args}} ; fi
if echo "${{github.event.inputs.skiptests}}" | grep -vq modules ; then ./runtest-moduleapi --verbose --timeout 2400 --no-latency --dump-logs ${{github.event.inputs.test_args}} ; fi
test-freebsd-sentinel:
runs-on: macos-12
runs-on: macos-13
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'freebsd') && !contains(github.event.inputs.skiptests, 'sentinel')
@@ -918,23 +922,24 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: test
uses: vmactions/freebsd-vm@v0.3.1
uses: cross-platform-actions/action@v0.21.1
with:
usesh: true
sync: rsync
copyback: false
prepare: pkg install -y bash gmake lang/tcl86 lang/tclx
run: >
gmake || exit 1 ;
if echo "${{github.event.inputs.skiptests}}" | grep -vq sentinel ; then ./runtest-sentinel ${{github.event.inputs.cluster_test_args}} || exit 1 ; fi ;
operating_system: freebsd
version: 13.2
memory: 8GB
shell: bash
run: |
sudo pkg install -y bash gmake lang/tcl86 lang/tclx
gmake
if echo "${{github.event.inputs.skiptests}}" | grep -vq sentinel ; then ./runtest-sentinel ${{github.event.inputs.cluster_test_args}} ; fi
test-freebsd-cluster:
runs-on: macos-12
runs-on: macos-13
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'freebsd') && !contains(github.event.inputs.skiptests, 'cluster')
@@ -954,15 +959,16 @@ jobs:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: test
uses: vmactions/freebsd-vm@v0.3.1
uses: cross-platform-actions/action@v0.21.1
with:
usesh: true
sync: rsync
copyback: false
prepare: pkg install -y bash gmake lang/tcl86 lang/tclx
run: >
gmake || exit 1 ;
if echo "${{github.event.inputs.skiptests}}" | grep -vq cluster ; then ./runtest-cluster ${{github.event.inputs.cluster_test_args}} || exit 1 ; fi ;
operating_system: freebsd
version: 13.2
memory: 8GB
shell: bash
run: |
sudo pkg install -y bash gmake lang/tcl86 lang/tclx
gmake
if echo "${{github.event.inputs.skiptests}}" | grep -vq cluster ; then ./runtest-cluster ${{github.event.inputs.cluster_test_args}} ; fi
test-alpine-jemalloc:
runs-on: ubuntu-latest
@@ -980,7 +986,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -1019,7 +1025,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -1058,7 +1064,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -1085,3 +1091,158 @@ jobs:
- name: validator
run: ./utils/req-res-log-validator.py --verbose --fail-missing-reply-schemas ${{ (!contains(github.event.inputs.skiptests, 'redis') && !contains(github.event.inputs.skiptests, 'module') && !contains(github.event.inputs.sentinel, 'redis') && !contains(github.event.inputs.skiptests, 'cluster')) && github.event.inputs.test_args == '' && github.event.inputs.cluster_test_args == '' && '--fail-commands-not-all-hit' || '' }}
test-old-chain-jemalloc:
runs-on: ubuntu-latest
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'oldTC')
container: ubuntu:20.04
timeout-minutes: 14400
steps:
- name: prep
if: github.event_name == 'workflow_dispatch'
run: |
echo "GITHUB_REPOSITORY=${{github.event.inputs.use_repo}}" >> $GITHUB_ENV
echo "GITHUB_HEAD_REF=${{github.event.inputs.use_git_ref}}" >> $GITHUB_ENV
echo "skipjobs: ${{github.event.inputs.skipjobs}}"
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
apt-get update
apt-get install -y gnupg2
echo "deb http://archive.ubuntu.com/ubuntu/ xenial main" >> /etc/apt/sources.list
echo "deb http://archive.ubuntu.com/ubuntu/ xenial universe" >> /etc/apt/sources.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 40976EAF437D05B5
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3B4FE6ACC0B21F32
apt-get update
apt-get install -y make gcc-4.8
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 100
make CC=gcc REDIS_CFLAGS='-Werror'
- name: testprep
run: apt-get install -y tcl tcltls tclx
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: ./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
- name: cluster tests
if: true && !contains(github.event.inputs.skiptests, 'cluster')
run: ./runtest-cluster ${{github.event.inputs.cluster_test_args}}
test-old-chain-tls-module:
runs-on: ubuntu-latest
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'tls') && !contains(github.event.inputs.skipjobs, 'oldTC')
container: ubuntu:20.04
timeout-minutes: 14400
steps:
- name: prep
if: github.event_name == 'workflow_dispatch'
run: |
echo "GITHUB_REPOSITORY=${{github.event.inputs.use_repo}}" >> $GITHUB_ENV
echo "GITHUB_HEAD_REF=${{github.event.inputs.use_git_ref}}" >> $GITHUB_ENV
echo "skipjobs: ${{github.event.inputs.skipjobs}}"
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
apt-get update
apt-get install -y gnupg2
echo "deb http://archive.ubuntu.com/ubuntu/ xenial main" >> /etc/apt/sources.list
echo "deb http://archive.ubuntu.com/ubuntu/ xenial universe" >> /etc/apt/sources.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 40976EAF437D05B5
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3B4FE6ACC0B21F32
apt-get update
apt-get install -y make gcc-4.8 openssl libssl-dev
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 100
make CC=gcc BUILD_TLS=module REDIS_CFLAGS='-Werror'
- name: testprep
run: |
apt-get install -y tcl tcltls tclx
./utils/gen-test-certs.sh
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
run: |
./runtest --accurate --verbose --dump-logs --tls-module --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: |
CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs --tls-module --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: |
./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
- name: cluster tests
if: true && !contains(github.event.inputs.skiptests, 'cluster')
run: |
./runtest-cluster --tls-module ${{github.event.inputs.cluster_test_args}}
test-old-chain-tls-module-no-tls:
runs-on: ubuntu-latest
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'tls') && !contains(github.event.inputs.skipjobs, 'oldTC')
container: ubuntu:20.04
timeout-minutes: 14400
steps:
- name: prep
if: github.event_name == 'workflow_dispatch'
run: |
echo "GITHUB_REPOSITORY=${{github.event.inputs.use_repo}}" >> $GITHUB_ENV
echo "GITHUB_HEAD_REF=${{github.event.inputs.use_git_ref}}" >> $GITHUB_ENV
echo "skipjobs: ${{github.event.inputs.skipjobs}}"
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
apt-get update
apt-get install -y gnupg2
echo "deb http://archive.ubuntu.com/ubuntu/ xenial main" >> /etc/apt/sources.list
echo "deb http://archive.ubuntu.com/ubuntu/ xenial universe" >> /etc/apt/sources.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 40976EAF437D05B5
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3B4FE6ACC0B21F32
apt-get update
apt-get install -y make gcc-4.8 openssl libssl-dev
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 100
make BUILD_TLS=module CC=gcc REDIS_CFLAGS='-Werror'
- name: testprep
run: |
apt-get install -y tcl tcltls tclx
./utils/gen-test-certs.sh
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
run: |
./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: |
CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: |
./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
- name: cluster tests
if: true && !contains(github.event.inputs.skiptests, 'cluster')
run: |
./runtest-cluster ${{github.event.inputs.cluster_test_args}}
+6 -6
View File
@@ -12,7 +12,7 @@ jobs:
if: github.event_name != 'schedule' || github.repository == 'redis/redis'
timeout-minutes: 14400
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Build
run: make REDIS_CFLAGS=-Werror
- name: Start redis-server
@@ -26,7 +26,7 @@ jobs:
--tags -slow
- name: Archive redis log
if: ${{ failure() }}
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: test-external-redis-log
path: external-redis.log
@@ -36,7 +36,7 @@ jobs:
if: github.event_name != 'schedule' || github.repository == 'redis/redis'
timeout-minutes: 14400
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Build
run: make REDIS_CFLAGS=-Werror
- name: Start redis-server
@@ -53,7 +53,7 @@ jobs:
--tags -slow
- name: Archive redis log
if: ${{ failure() }}
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: test-external-cluster-log
path: external-redis.log
@@ -63,7 +63,7 @@ jobs:
if: github.event_name != 'schedule' || github.repository == 'redis/redis'
timeout-minutes: 14400
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Build
run: make REDIS_CFLAGS=-Werror
- name: Start redis-server
@@ -76,7 +76,7 @@ jobs:
--tags "-slow -needs:debug"
- name: Archive redis log
if: ${{ failure() }}
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: test-external-redis-log
path: external-redis.log
+1 -1
View File
@@ -12,7 +12,7 @@ jobs:
reply-schemas-linter:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Setup nodejs
uses: actions/setup-node@v3
- name: Install packages
+1 -1
View File
@@ -16,7 +16,7 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: pip cache
uses: actions/cache@v3
+582 -11
View File
@@ -1,16 +1,587 @@
Hello! This file is just a placeholder, since this is the "unstable" branch
of Redis, the place where all the development happens.
Redis 7.2 release notes
=======================
There is no release notes for this branch, it gets forked into another branch
every time there is a partial feature freeze in order to eventually create
a new stable release.
--------------------------------------------------------------------------------
Upgrade urgency levels:
Usually "unstable" is stable enough for you to use it in development environments
however you should never use it in production environments. It is possible
to download the latest stable release here:
LOW: No need to upgrade unless there are new features you want to use.
MODERATE: Program an upgrade of the server, but it's not urgent.
HIGH: There is a critical bug that may affect a subset of users. Upgrade!
CRITICAL: There is a critical bug affecting MOST USERS. Upgrade ASAP.
SECURITY: There are security fixes in the release.
--------------------------------------------------------------------------------
https://download.redis.io/redis-stable.tar.gz
================================================================================
Redis 7.2.8 Released Wed 23 Apr 2025 12:00:00 IST
================================================================================
More information is available at https://redis.io
Update urgency: `SECURITY`: There are security fixes in the release.
### Security fixes
* (CVE-2025-21605) An unauthenticated client can cause an unlimited growth of output buffers
### Bug fixes
* #12817, #12905 Fix race condition issues between the main thread and module threads
* #13863 `RANDOMKEY` - infinite loop during client pause
* #13877 ShardID inconsistency when both primary and replica support it
================================================================================
Redis 7.2.7 Released Mon 6 Jan 2025 12:30:00 IDT
================================================================================
Upgrade urgency SECURITY: See security fixes below.
Security fixes
==============
* (CVE-2024-46981) Lua script commands may lead to remote code execution
* (CVE-2024-51741) Denial-of-service due to malformed ACL selectors
Bug fixes
=========
* #13380 Possible crash due to OOM panic on invalid command
* #13338 Streams: `XINFO` lag field is wrong when tombstone is after the `last_id` of the consume group
* #13473 Streams: `XTRIM` does not update the maximal tombstone, leading to an incorrect lag
* #13311 Cluster: crash due to unblocking client during slot migration
* #13443 Cluster: crash when loading cluster config
* #13422 Cluster: `CLUSTER SHARDS` returns empty array
* #13465 Cluster: incompatibility with older node versions
================================================================================
Redis 7.2.6 Released Wed 02 Oct 2024 20:17:04 IDT
================================================================================
Upgrade urgency SECURITY: See security fixes below.
Security fixes
==============
* (CVE-2024-31449) Lua library commands may lead to stack overflow and potential RCE.
* (CVE-2024-31227) Potential Denial-of-service due to malformed ACL selectors.
* (CVE-2024-31228) Potential Denial-of-service due to unbounded pattern matching.
Bug fixes
=========
* Fixed crashes in cluster mode (#13315)
================================================================================
Redis 7.2.5 Released Thu 16 May 2024 12:00:00 IST
================================================================================
Upgrade urgency MODERATE: Program an upgrade of the server, but it's not urgent.
Bug fixes
=========
* A single shard cluster leaves failed replicas in CLUSTER SLOTS instead of removing them (#12824)
* Crash in LSET command when replacing small items and exceeding 4GB (#12955)
* Blocking commands timeout is reset due to re-processing command (#13004)
* Conversion of numbers in Lua args to redis args can fail. Bug introduced in 7.2.0 (#13115)
Bug fixes in CLI tools
======================
* redis-cli: --count (for --scan, --bigkeys, etc) was ignored unless --pattern was also used (#13092)
* redis-check-aof: incorrectly considering data in manifest format as MP-AOF (#12958)
================================================================================
Redis 7.2.4 Released Tue 09 Jan 2024 10:45:52 IST
================================================================================
Upgrade urgency SECURITY: See security fixes below.
Security fixes
==============
* (CVE-2023-41056) In some cases, Redis may incorrectly handle resizing of memory
buffers which can result in incorrect accounting of buffer sizes and lead to
heap overflow and potential remote code execution.
Bug fixes
=========
* Fix crashes of cluster commands clusters with mixed versions of 7.0 and 7.2 (#12805, #12832)
* Fix slot ownership not being properly handled when deleting a slot from a node (#12564)
* Fix atomicity issues with the RedisModuleEvent_Key module API event (#12733)
================================================================================
Redis 7.2.3 Released Wed 01 Nov 2023 12:00:00 IST
================================================================================
Upgrade urgency: HIGH, Fixes critical bugs affecting most users.
Bug fixes
=========
* Fix file descriptor leak preventing deleted files from freeing disk space on
replicas (#12693)
* Fix a possible crash after cluster node removal (#12702)
================================================================================
Redis 7.2.2 Released Wed 18 Oct 2023 10:33:40 IDT
================================================================================
Upgrade urgency SECURITY: See security fixes below.
Security fixes
==============
* (CVE-2023-45145) The wrong order of listen(2) and chmod(2) calls creates a
race condition that can be used by another process to bypass desired Unix
socket permissions on startup.
Platform / toolchain support related changes
=================================================
* Fix compilation error on MacOS 13 (#12611)
Bug fixes
=========
* WAITAOF could timeout in the absence of write traffic in case a new AOF is
created and an AOF rewrite can't immediately start (#12620)
Redis cluster
=============
* Fix crash when running rebalance command in a mixed cluster of 7.0 and 7.2
nodes (#12604)
* Fix the return type of the slot number in cluster shards to integer, which
makes it consistent with past behavior (#12561)
* Fix CLUSTER commands are called from modules or scripts to return TLS info
appropriately (#12569)
Changes in CLI tools
====================
* redis-cli, fix crash on reconnect when in SUBSCRIBE mode (#12571)
Module API changes
==================
* Fix overflow calculation for next timer event (#12474)
================================================================================
Redis 7.2.1 Released Wed 06 Sep 2023 15:00:00 IDT
================================================================================
Upgrade urgency SECURITY: See security fixes below.
Security Fixes
==============
* (CVE-2023-41053) Redis does not correctly identify keys accessed by SORT_RO and,
as a result, may grant users executing this command access to keys that are not
explicitly authorized by the ACL configuration.
Bug Fixes
=========
* Fix crashes when joining a node to an existing 7.0 Redis Cluster (#12538)
* Correct request_policy and response_policy command tips on for some admin /
configuration commands (#12545, #12530)
================================================================================
Redis 7.2.0 GA Released Tue Aug 15 12:00:00 IDT 2023
================================================================================
Upgrade urgency LOW: This is the first stable Release for Redis 7.2.
Bug Fixes
=========
* redis-cli in cluster mode handles `unknown-endpoint` (#12273)
* Update request / response policy hints for a few commands (#12417)
* Ensure that the function load timeout is disabled during loading from RDB/AOF and on replicas. (#12451)
* Fix false success and a memory leak for ACL selector with bad parenthesis combination (#12452)
* Fix the assertion when script timeout occurs after it signaled a blocked client (#12459)
Fixes for issues in previous releases of Redis 7.2
--------------------------------------------------
* Update MONITOR client's memory correctly for INFO and client-eviction (#12420)
* The response of cluster nodes was unnecessarily adding an extra comma when no
hostname was present. (#12411)
================================================================================
Redis 7.2 RC3 Released Mon July 10 12:00:00 IDT 2023
================================================================================
Upgrade urgency LOW: This is the third Release Candidate for Redis 7.2.
Upgrade urgency SECURITY: If you're using a previous release candidate of 7.2.
Security Fixes:
* (CVE-2022-24834) A specially crafted Lua script executing in Redis can trigger
a heap overflow in the cjson and cmsgpack libraries, and result in heap
corruption and potentially remote code execution. The problem exists in all
versions of Redis with Lua scripting support, starting from 2.6, and affects
only authenticated and authorized users.
* (CVE-2023-36824) Extracting key names from a command and a list of arguments
may, in some cases, trigger a heap overflow and result in reading random heap
memory, heap corruption and potentially remote code execution. Specifically:
using COMMAND GETKEYS* and validation of key names in ACL rules.
New Features
============
New administrative and introspection commands and command arguments
-------------------------------------------------------------------
* Make SENTINEL CONFIG [SET|GET] variadic. (#10362)
Potentially Breaking / Behavior Changes
=======================================
* Cluster SHARD IDs are no longer visible in the cluster nodes output,
introduced in 7.2-RC1. (#10536, #12166)
* When calling PUBLISH with a RESP3 client that's also subscribed to the same channel,
the order is changed and the reply is sent before the published message (#12326)
New configuration options
=========================
* Add a new loglevel "nothing" to disable logging (#12133)
* Add cluster-announce-human-nodename - a unique identifier for a node that is
be used in logs for debugging (#9564)
Other General Improvements
==========================
* Allow CLUSTER SLOTS / SHARDS commands during loading (#12269)
* Support TLS service when "tls-cluster" is not enabled and persist both plain
and TLS port in nodes.conf (#12233)
* Update SPOP and RESTORE commands to replicate unlink commands to replicas
when the server is configured to use async server deletes (#12320)
* Try lazyfree the temporary zset in ZUNION / ZINTER / ZDIFF (#12229)
Performance and resource utilization improvements
=================================================
* Optimize PSUBSCRIBE and PUNSUBSCRIBE from O(N*M) to O(N) (#12298)
* Optimize SCAN, SSCAN, HSCAN, ZSCAN commands (#12209)
* Set Jemalloc --disable-cache-oblivious to reduce memory overhead (#12315)
* Optimize ZINTERCARD to avoid create a temporary zset (#12229)
* Optimize HRANDFIELD and ZRANDMEMBER listpack encoded (#12205)
* Numerous other optimizations (#12155, #12082, #11626, #11944, #12316, #12250,
#12177, #12185)
Changes in CLI tools
====================
* redis-cli: Handle RESP3 double responses that contain a NaN (#12254)
* redis-cli: Support URIs with IPv6 (#11834)
Module API changes
==================
* Align semantics of the new (v7.2 RC2) RM_ReplyWithErrorFormat with RM_ReplyWithError.
This is a breaking change that affects the generated error code. (#12321)
* Forbid RM_AddPostNotificationJob on loading and on read-only replicas (#12304)
* Add ability for module command filter to know which client is being handled (#12219)
Bug Fixes
=========
* Fix broken protocol when PUBLISH is used inside MULTI when the RESP3
publishing client is also subscribed for the channel (#12326)
* Fix WAIT to be effective after a blocked module command being unblocked (#12220)
* Re-enable downscale rehashing while there is a fork child (#12276)
* Fix possible hang in HRANDFIELD, SRANDMEMBER, ZRANDMEMBER when used with `<count>` (#12276)
* Improve fairness issue in RANDOMKEY, HRANDFIELD, SRANDMEMBER, ZRANDMEMBER, SPOP, and eviction (#12276)
* Cluster: fix a race condition where a slot migration may revert on a subsequent failover or node joining (#12344)
Fixes for issues in previous releases of Redis 7.2
--------------------------------------------------
* Fix XREADGROUP BLOCK with ">" from hanging (#12301)
* Fix assertion when a blocked command is rejected when re-processed. (#12247)
* Fix use after free on a blocking RM_Call. (#12342)
================================================================================
Redis 7.2 RC2 Released Mon May 15 12:00:00 IST 2023
================================================================================
Upgrade urgency LOW: This is the second Release Candidate for Redis 7.2.
INFO fields and introspection changes
=====================================
* Add a few low level event loop metrics to help diagnose latency (#11963)
Performance and resource utilization improvements
=================================================
* Minor performance improvement to SADD and HSET (#12019)
Platform / toolchain support related changes
=================================================
* Upgrade to Jemalloc 5.3.0, resolves a rare fork child hang (#12115)
* Fix a compiler fortification induced crash when used with link time optimizations (#11982)
* Fix local clients detection, 127.*.*.* instead of 127.0.0.1 (#11664)
* Report AOF failure status to systemd in shutdown (#12065)
Changes in CLI tools
====================
* redis-cli: Reimplement and improve help hints based on actual command arg docs (#10515)
* redis-cli: Add option --count for tuning SCAN based features (#12042)
* redis-benchmark: Add --seed option to seed the random number generator (#11945)
Module API changes
==================
* Add RM_RdbLoad and RM_RdbSave APIs (#11852)
* Add RM_ReplyWithErrorFormat that can support format string (#11923)
* Fix: Delete empty key when RM_ZsetAdd, RM_ZsetIncrby, RM_StreamAdd fail (#12129)
Bug Fixes
=========
* LPOS with RANK set to LONG_MIN returning wrong result (#12167)
* Avoid unnecessary full sync after master restart in a rare case (#12088)
* Iterate clients fairly when processing background chores (#12025)
* Avoid incorrect shrinking of query buffer when reading large data from clients (#12000)
* Sentinel: Fix config rewrite error when old known-slave is used (#11775)
* ACL: Disconnect pub-sub subscribers when revoking allchannels permission (#11992)
* Add a missing fsync of AOF file in rare cases (#11973)
Fixes for issues in previous releases of Redis 7.2
--------------------------------------------------
* Fix tracking of command duration metrics for MULTI, EVAL, WAIT and modules (#11970)
================================================================================
Redis 7.2 RC1 Released Wed Mar 22 12:00:00 IST 2023
================================================================================
Upgrade urgency LOW: This is the first Release Candidate for Redis 7.2.
Redis Release Candidate (RC) versions are early versions that are made available
for early adopters in the community to test them. We do not consider
them suitable for production environments.
Introduction to the Redis 7.2 release
=====================================
Redis 7.2 includes optimizations, several new commands, some improvements,
bug fixes, and several new module APIs.
In particular, users should be aware of the following changes:
1. Redis 7.2 uses a new format (version 11) for RDB files, which is incompatible
with older versions.
2. See section about breaking changes mentioned below.
3. If you use modules, see the module API breaking changes section below.
Here is a comprehensive list of changes in this release compared to 7.0.10.
Each one includes the PR number that added it so that you can get more details
at https://github.com/redis/redis/pull/<number>
New Features
============
* Introduce WAITAOF command, to block the client until a specified number
of Redises have synced all previous write commands to the AOF on disk,
see https://redis.io/commands/waitaof/
New user commands or command arguments
--------------------------------------
* WAITAOF blocks until writes have been synced to disk (#11713)
* Add WITHSCORE option to ZRANK and ZREVRANK (#11235)
New administrative and introspection commands and command arguments
-------------------------------------------------------------------
* CLIENT SETINFO lets client library report name and version Redis (#11758)
* CLIENT NO-TOUCH for clients to run commands without affecting LRU/LFU of keys (#11483)
* Introduce Shard IDs to logically group nodes in cluster mode based on
replication. Shard IDs are automatically assigned and visible via
`CLUSTER MYSHARDID`. (#10536)
Command replies that have been extended
---------------------------------------
* ACL LOG - Add entry id, timestamp created, and timestamp last updated time (#11477)
* COMMAND DOCS - Repurpose arg names as the unique ID (#11051)
* CLIENT LIST has `T` flag to indicate CLIENT NO-TOUCH (#11483)
* CLIENT LIST show lib-name, lib-ver (#11758)
Potentially Breaking / Behavior Changes
=======================================
* Client side tracking for scripts now tracks the keys that are read by the
script instead of the keys that are declared by the caller of EVAL / FCALL (#11770)
* Freeze time sampling during command execution and in scripts (#10300)
* When a blocked command is being unblocked, checks like ACL, OOM, etc are
re-evaluated (#11012)
* Unify ACL failure error message text and error codes (#11160)
* Blocked stream command that's released when key no longer exists carries a
different error code (#11012)
* Command stats are updated for blocked commands only when / if the command
actually executes (#11012)
* The way ACL users are stored internally no longer removes redundant command
and category rules, which may alter the way those rules are displayed as part
of `ACL SAVE`, `ACL GETUSER` and `ACL LIST` (#11224)
* Client connections created for TLS-based replication use SNI if possible (#11458)
* Stream consumers: Re-purpose seen-time, add active-time (#11099)
* XREADGROUP and X[AUTO]CLAIM create the consumer regardless of whether it was
able to perform some reading/claiming (#11099)
* ACL default newly created user set sanitize-payload flag in ACL LIST/GETUSER #11279
* Fix HELLO command not to affect the client state unless successful (#11659)
* Normalize `NAN` in replies to a single nan type, like we do with `inf` (#11597)
Deprecations
============
* Mark the QUIT command as deprecated (#11439)
* Delete RDB loading code for pre-release RDB formats (#11058)
Performance and resource utilization improvements
=================================================
* Significant memory optimization of small list type keys (#11303)
* Significant memory optimization for small set type keys (#11290)
* Significant memory optimization for large sets (#11595)
* Significant speed optimization in ZRANGE replies WITHSCORES in case of integer scores (#11779)
* Significant speed optimization in double replies, mainly sorted sets commands (#10587)
* Optimize the performance of commands with multiple keys in cluster mode (#11044)
* Incrementally reclaim OS page cache of RDB file (#11248)
* Improve memory management of cluster bus links when there is a large number of pending messages (#11343)
* Minor performance improvement for workloads that use commands without pipelining (#11220)
Changes in CLI tools
====================
* redis-cli accepts commands in subscribed mode (#11873)
Other General Improvements
==========================
* WAIT now no longer waits for the replication offset after your last command,
but rather the replication offset after your last write (#11713)
* Automatically propagate node deletion to other nodes in a cluster when
`CLUSTER FORGET` is called, allowing nodes to be deleted with a single call
in most cases (#10869)
* Blocking commands that were disallowed in scripts now behave in scripts the
same they did in MULTI (#11568)
Platform / toolchain support related changes
=================================================
* 32-bit builds compiled without HAVE_MALLOC_SIZE (not jemalloc or glibc)
will consume more memory (#11595)
* Use jemalloc by default also on ARM (#11407)
* Adds stack trace and register dump support in crash report for illumos/solaris (#11335)
New configuration options
=========================
* locale-collate runtime config to control setlocale affecting Lua and SORT (#11059)
* Add CONFIG SET and GET loglevel feature in Sentinel (#11214)
INFO fields and introspection changes
=====================================
* Added 4 new info fields for authentication errors and commands denied access
for keys, channels and commands (#11288)
* INFO SERVER includes a list of listeners (#9320)
Module API changes
==================
* Make it possible for module commands to be part of ACL categories (#11708)
* Add K flag to RM_Call to allow running blocking commands and set a callback to get the response (#11568)
* Add RM_AddPostNotificationJob to allow writes after keyspace notification hooks (#11199)
* RedisModule_Event_Key to notify about keys being unlinked together with reason and value (#9406)
* Add RM_BlockClient[Set|Get]PrivateData to associate a module data with the blocked client (#11568)
* APIs to allow modules to participate / handle AUTH validation (#11659)
* RM_GetContextFlags supports a new flag: REDISMODULE_CTX_FLAGS_SERVER_STARTUP (#9320)
* Add REDISMODULE_OPTIONS_ALLOW_NESTED_KEYSPACE_NOTIFICATIONS and RedisModule_GetModuleOptionsAll (#11199)
* RM_BlockClientOnKeysWithFlags allows module to request being unblocked when the key is deleted (#11310)
* Introduce aux_save2 makes it possible to skip saving that field in the RDB and
enable loading the file in the absence of the module (#11374)
* Add a dry run flag to RM_Call to do validations before actual execution (#11158)
* Add RM_Microseconds and RM_CachedMicroseconds (#11016)
* Add RM_ACLAddLogEntryByUserName API to be used without a user object (#11659)
* Make it possible to keep the RM_Call reply for longer than the context lifetime in case
auto memory was not used (#11568)
Potentially Breaking Changes in Module API
------------------------------------------
* RM_Call only enforces OOM on scripts if 'M' flag is set (#11425)
* Block some specific characters in module command names (#11434)
* Fix replication inconsistency on modules that uses keyspace notifications (#10969)
* Prevent command, configs, data types registration after the onload handler (#11708)
Bug Fixes
=========
* Introduce socket shutdown to properly disconnect a client while a fork is active (#11376)
* CLIENT RESET clears the CLIENT NO-EVICT flag (#11483)
* Reduce memory usage on strings loaded by a module from an RDB file (#11050)
* Fix a bug where nodes in a cluster may not replicate or handle internal events for
keys deleted when another node in the cluster claimed a slot (#11084)
* Fix HINCRBYFLOAT not to create a key if the new value is invalid (#11149)
* Make cluster config file saving atomic and fsync acl file saving (#10924)
* WAIT command would not block if used in RM_Call (#11713)
* Minor fixes to command metadata in COMMAND command (#11201, #10273)
Thanks to all the users and developers who made this release possible.
We'll follow up with more RC releases, until the code looks production ready
and we don't get reports of serious issues for a while.
A special thank you for the amount of work put into this release by:
- Meir Shpilraien
- Guy Benoish
- Viktor Söderqvist
- Zhu Binbin
- Oran Agra
- sundb
- Ran Shidlansik
- Zhenwei Pi
- Jason Elbaum
- Karthik Subbarao
- Madelyn Olson
- Huang Zhw
- Ping Xie
- Ozan Tezcan
- Chen Tianjie
- Deng Ju
- Wen Hui
- Brennan Cathcart
- Itamar Haber
- Shaya Potter
- Roshan Khatri
- Slava Koyfman
- Zhu Tian
- Moti Cohen
- Arad Zilberstein
- Basel Naamna
- Mingyi Kang
- Uri Yagelnik
- Filipe Oliveira
- Zhao Zhao
- Valentino Geron
- Yaacov Hazan
- Adi Pinsky
- David Carlier
- Li Changjun
Happy hacking!
+1
View File
@@ -132,6 +132,7 @@ static int bit_tohex(lua_State *L)
const char *hexdigits = "0123456789abcdef";
char buf[8];
int i;
if (n == INT32_MIN) n = INT32_MIN+1;
if (n < 0) { n = -n; hexdigits = "0123456789ABCDEF"; }
if (n > 8) n = 8;
for (i = (int)n; --i >= 0; ) { buf[i] = hexdigits[b & 15]; b >>= 4; }
+3
View File
@@ -101,6 +101,9 @@ else
ifeq ($(SANITIZER),undefined)
MALLOC=libc
CFLAGS+=-fsanitize=undefined -fno-sanitize-recover=all -fno-omit-frame-pointer
ifeq (clang,$(CLANG))
CFLAGS+=-fno-sanitize=function
endif
LDFLAGS+=-fsanitize=undefined
else
ifeq ($(SANITIZER),thread)
+7 -2
View File
@@ -1046,6 +1046,7 @@ int ACLSetSelector(aclSelector *selector, const char* op, size_t oplen) {
int flags = 0;
size_t offset = 1;
if (op[0] == '%') {
int perm_ok = 1;
for (; offset < oplen; offset++) {
if (toupper(op[offset]) == 'R' && !(flags & ACL_READ_PERMISSION)) {
flags |= ACL_READ_PERMISSION;
@@ -1055,10 +1056,14 @@ int ACLSetSelector(aclSelector *selector, const char* op, size_t oplen) {
offset++;
break;
} else {
errno = EINVAL;
return C_ERR;
perm_ok = 0;
break;
}
}
if (!flags || !perm_ok) {
errno = EINVAL;
return C_ERR;
}
} else {
flags = ACL_ALL_PERMISSION;
}
+1 -1
View File
@@ -333,7 +333,7 @@ static int processTimeEvents(aeEventLoop *eventLoop) {
processed++;
now = getMonotonicUs();
if (retval != AE_NOMORE) {
te->when = now + retval * 1000;
te->when = now + (monotime)retval * 1000;
} else {
te->id = AE_DELETED_EVENT_ID;
}
+6 -5
View File
@@ -417,13 +417,16 @@ int anetUnixGenericConnect(char *err, const char *path, int flags)
return s;
}
static int anetListen(char *err, int s, struct sockaddr *sa, socklen_t len, int backlog) {
static int anetListen(char *err, int s, struct sockaddr *sa, socklen_t len, int backlog, mode_t perm) {
if (bind(s,sa,len) == -1) {
anetSetError(err, "bind: %s", strerror(errno));
close(s);
return ANET_ERR;
}
if (sa->sa_family == AF_LOCAL && perm)
chmod(((struct sockaddr_un *) sa)->sun_path, perm);
if (listen(s, backlog) == -1) {
anetSetError(err, "listen: %s", strerror(errno));
close(s);
@@ -467,7 +470,7 @@ static int _anetTcpServer(char *err, int port, char *bindaddr, int af, int backl
if (af == AF_INET6 && anetV6Only(err,s) == ANET_ERR) goto error;
if (anetSetReuseAddr(err,s) == ANET_ERR) goto error;
if (anetListen(err,s,p->ai_addr,p->ai_addrlen,backlog) == ANET_ERR) s = ANET_ERR;
if (anetListen(err,s,p->ai_addr,p->ai_addrlen,backlog,0) == ANET_ERR) s = ANET_ERR;
goto end;
}
if (p == NULL) {
@@ -508,10 +511,8 @@ int anetUnixServer(char *err, char *path, mode_t perm, int backlog)
memset(&sa,0,sizeof(sa));
sa.sun_family = AF_LOCAL;
redis_strlcpy(sa.sun_path,path,sizeof(sa.sun_path));
if (anetListen(err,s,(struct sockaddr*)&sa,sizeof(sa),backlog) == ANET_ERR)
if (anetListen(err,s,(struct sockaddr*)&sa,sizeof(sa),backlog,perm) == ANET_ERR)
return ANET_ERR;
if (perm)
chmod(sa.sun_path, perm);
return s;
}
+19 -13
View File
@@ -117,7 +117,9 @@ aofInfo *aofInfoDup(aofInfo *orig) {
return ai;
}
/* Format aofInfo as a string and it will be a line in the manifest. */
/* Format aofInfo as a string and it will be a line in the manifest.
*
* When update this format, make sure to update redis-check-aof as well. */
sds aofInfoFormat(sds buf, aofInfo *ai) {
sds filename_repr = NULL;
@@ -976,18 +978,6 @@ void stopAppendOnly(void) {
int startAppendOnly(void) {
serverAssert(server.aof_state == AOF_OFF);
/* Wait for all bio jobs related to AOF to drain. This prevents a race
* between updates to `fsynced_reploff_pending` of the worker thread, belonging
* to the previous AOF, and the new one. This concern is specific for a full
* sync scenario where we don't wanna risk the ACKed replication offset
* jumping backwards or forward when switching to a different master. */
bioDrainWorker(BIO_AOF_FSYNC);
/* Set the initial repl_offset, which will be applied to fsynced_reploff
* when AOFRW finishes (after possibly being updated by a bio thread) */
atomicSet(server.fsynced_reploff_pending, server.master_repl_offset);
server.fsynced_reploff = 0;
server.aof_state = AOF_WAIT_REWRITE;
if (hasActiveChildProcess() && server.child_type != CHILD_TYPE_AOF) {
server.aof_rewrite_scheduled = 1;
@@ -2454,7 +2444,23 @@ int rewriteAppendOnlyFileBackground(void) {
server.aof_lastbgrewrite_status = C_ERR;
return C_ERR;
}
if (server.aof_state == AOF_WAIT_REWRITE) {
/* Wait for all bio jobs related to AOF to drain. This prevents a race
* between updates to `fsynced_reploff_pending` of the worker thread, belonging
* to the previous AOF, and the new one. This concern is specific for a full
* sync scenario where we don't wanna risk the ACKed replication offset
* jumping backwards or forward when switching to a different master. */
bioDrainWorker(BIO_AOF_FSYNC);
/* Set the initial repl_offset, which will be applied to fsynced_reploff
* when AOFRW finishes (after possibly being updated by a bio thread) */
atomicSet(server.fsynced_reploff_pending, server.master_repl_offset);
server.fsynced_reploff = 0;
}
server.stat_aof_rewrites++;
if ((childpid = redisFork(CHILD_TYPE_AOF)) == 0) {
char tmpfile[256];
+6 -2
View File
@@ -370,7 +370,12 @@ void blockForKeys(client *c, int btype, robj **keys, int numkeys, mstime_t timeo
list *l;
int j;
c->bstate.timeout = timeout;
if (!(c->flags & CLIENT_REPROCESSING_COMMAND)) {
/* If the client is re-processing the command, we do not set the timeout
* because we need to retain the client's original timeout. */
c->bstate.timeout = timeout;
}
for (j = 0; j < numkeys; j++) {
/* If the key already exists in the dictionary ignore it. */
if (!(client_blocked_entry = dictAddRaw(c->bstate.keys,keys[j],NULL))) {
@@ -392,7 +397,6 @@ void blockForKeys(client *c, int btype, robj **keys, int numkeys, mstime_t timeo
listAddNodeTail(l,c);
dictSetVal(c->bstate.keys,client_blocked_entry,listLast(l));
/* We need to add the key to blocking_keys_unblock_on_nokey, if the client
* wants to be awakened if key is deleted (like XREADGROUP) */
if (unblock_on_nokey) {
+205 -66
View File
@@ -61,6 +61,7 @@ list *clusterGetNodesInMyShard(clusterNode *node);
int clusterNodeAddSlave(clusterNode *master, clusterNode *slave);
int clusterAddSlot(clusterNode *n, int slot);
int clusterDelSlot(int slot);
int clusterMoveNodeSlots(clusterNode *from_node, clusterNode *to_node);
int clusterDelNodeSlots(clusterNode *node);
int clusterNodeSetSlotBit(clusterNode *n, int slot);
void clusterSetMaster(clusterNode *n);
@@ -102,6 +103,7 @@ int auxTlsPortSetter(clusterNode *n, void *value, int length);
sds auxTlsPortGetter(clusterNode *n, sds s);
int auxTlsPortPresent(clusterNode *n);
static void clusterBuildMessageHdr(clusterMsg *hdr, int type, size_t msglen);
static void updateShardId(clusterNode *node, const char *shard_id);
int getNodeDefaultClientPort(clusterNode *n) {
return server.tls_cluster ? n->tls_port : n->tcp_port;
@@ -119,6 +121,20 @@ static inline int defaultClientPort(void) {
return server.tls_cluster ? server.tls_port : server.port;
}
/* When a cluster command is called, we need to decide whether to return TLS info or
* non-TLS info by the client's connection type. However if the command is called by
* a Lua script or RM_call, there is no connection in the fake client, so we use
* server.current_client here to get the real client if available. And if it is not
* available (modules may call commands without a real client), we return the default
* info, which is determined by server.tls_cluster. */
static int shouldReturnTlsInfo(void) {
if (server.current_client && server.current_client->conn) {
return connIsTLS(server.current_client->conn);
} else {
return server.tls_cluster;
}
}
/* Links to the next and previous entries for keys in the same slot are stored
* in the dict entry metadata. See Slot to Key API below. */
#define dictEntryNextInSlot(de) \
@@ -237,12 +253,11 @@ int auxShardIdSetter(clusterNode *n, void *value, int length) {
return C_ERR;
}
memcpy(n->shard_id, value, CLUSTER_NAMELEN);
/* if n already has replicas, make sure they all agree
* on the shard id */
/* if n already has replicas, make sure they all use
* the primary shard id */
for (int i = 0; i < n->numslaves; i++) {
if (memcmp(n->slaves[i]->shard_id, n->shard_id, CLUSTER_NAMELEN) != 0) {
return C_ERR;
}
if (memcmp(n->slaves[i]->shard_id, n->shard_id, CLUSTER_NAMELEN) != 0)
updateShardId(n->slaves[i], n->shard_id);
}
clusterAddNodeToShard(value, n);
return C_OK;
@@ -580,18 +595,12 @@ int clusterLoadConfig(char *filename) {
clusterAddNode(master);
}
/* shard_id can be absent if we are loading a nodes.conf generated
* by an older version of Redis; we should follow the primary's
* shard_id in this case */
if (auxFieldHandlers[af_shard_id].isPresent(n) == 0) {
memcpy(n->shard_id, master->shard_id, CLUSTER_NAMELEN);
clusterAddNodeToShard(master->shard_id, n);
} else if (clusterGetNodesInMyShard(master) != NULL &&
memcmp(master->shard_id, n->shard_id, CLUSTER_NAMELEN) != 0)
{
/* If the primary has been added to a shard, make sure this
* node has the same persisted shard id as the primary. */
goto fmterr;
}
* by an older version of Redis;
* ignore replica's shard_id in the file, only use the primary's.
* If replica precedes primary in file, it will be corrected
* later by the auxShardIdSetter */
memcpy(n->shard_id, master->shard_id, CLUSTER_NAMELEN);
clusterAddNodeToShard(master->shard_id, n);
n->slaveof = master;
clusterNodeAddSlave(master,n);
} else if (auxFieldHandlers[af_shard_id].isPresent(n) == 0) {
@@ -669,6 +678,8 @@ int clusterLoadConfig(char *filename) {
}
/* Config sanity check */
if (server.cluster->myself == NULL) goto fmterr;
if (!(myself->flags & (CLUSTER_NODE_MASTER | CLUSTER_NODE_SLAVE))) goto fmterr;
if (nodeIsSlave(myself) && myself->slaveof == NULL) goto fmterr;
zfree(line);
fclose(fp);
@@ -936,22 +947,41 @@ static void updateAnnouncedHumanNodename(clusterNode *node, char *new) {
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG);
}
static void assignShardIdToNode(clusterNode *node, const char *shard_id, int flag) {
clusterRemoveNodeFromShard(node);
memcpy(node->shard_id, shard_id, CLUSTER_NAMELEN);
clusterAddNodeToShard(shard_id, node);
clusterDoBeforeSleep(flag);
}
int clusterNodeNumSlaves(clusterNode *node) {
return node->numslaves;
}
clusterNode *clusterNodeGetSlave(clusterNode *node, int slave_idx) {
return node->slaves[slave_idx];
}
static void updateShardId(clusterNode *node, const char *shard_id) {
if (memcmp(node->shard_id, shard_id, CLUSTER_NAMELEN) != 0) {
clusterRemoveNodeFromShard(node);
memcpy(node->shard_id, shard_id, CLUSTER_NAMELEN);
clusterAddNodeToShard(shard_id, node);
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG);
}
if (myself != node && myself->slaveof == node) {
if (memcmp(myself->shard_id, shard_id, CLUSTER_NAMELEN) != 0) {
/* shard-id can diverge right after a rolling upgrade
* from pre-7.2 releases */
clusterRemoveNodeFromShard(myself);
memcpy(myself->shard_id, shard_id, CLUSTER_NAMELEN);
clusterAddNodeToShard(shard_id, myself);
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG|CLUSTER_TODO_FSYNC_CONFIG);
if (shard_id && memcmp(node->shard_id, shard_id, CLUSTER_NAMELEN) != 0) {
/* We always make our best effort to keep the shard-id consistent
* between the master and its replicas:
*
* 1. When updating the master's shard-id, we simultaneously update the
* shard-id of all its replicas to ensure consistency.
* 2. When updating replica's shard-id, if it differs from its master's shard-id,
* we discard this replica's shard-id and continue using master's shard-id.
* This applies even if the master does not support shard-id, in which
* case we rely on the master's randomly generated shard-id. */
if (node->slaveof == NULL) {
assignShardIdToNode(node, shard_id, CLUSTER_TODO_SAVE_CONFIG);
for (int i = 0; i < clusterNodeNumSlaves(node); i++) {
clusterNode *slavenode = clusterNodeGetSlave(node, i);
if (memcmp(slavenode->shard_id, shard_id, CLUSTER_NAMELEN) != 0)
assignShardIdToNode(slavenode, shard_id, CLUSTER_TODO_SAVE_CONFIG|CLUSTER_TODO_FSYNC_CONFIG);
}
} else if (memcmp(node->slaveof->shard_id, shard_id, CLUSTER_NAMELEN) == 0) {
assignShardIdToNode(node, shard_id, CLUSTER_TODO_SAVE_CONFIG);
}
}
}
@@ -974,7 +1004,7 @@ void clusterInit(void) {
server.cluster->myself = NULL;
server.cluster->currentEpoch = 0;
server.cluster->state = CLUSTER_FAIL;
server.cluster->size = 1;
server.cluster->size = 0;
server.cluster->todo_before_sleep = 0;
server.cluster->nodes = dictCreate(&clusterNodesDictType);
server.cluster->shards = dictCreate(&clusterSdsToListType);
@@ -1120,6 +1150,9 @@ void clusterReset(int hard) {
}
dictReleaseIterator(di);
/* Empty the nodes blacklist. */
dictEmpty(server.cluster->nodes_black_list, NULL);
/* Hard reset only: set epochs to 0, change node ID. */
if (hard) {
sds oldname;
@@ -1670,6 +1703,7 @@ void clusterRenameNode(clusterNode *node, char *newname) {
serverAssert(retval == DICT_OK);
memcpy(node->name, newname, CLUSTER_NAMELEN);
clusterAddNode(node);
clusterAddNodeToShard(node->shard_id, node);
}
void clusterAddNodeToShard(const char *shard_id, clusterNode *node) {
@@ -1704,6 +1738,22 @@ void clusterRemoveNodeFromShard(clusterNode *node) {
sdsfree(s);
}
static clusterNode *clusterGetMasterFromShard(list *nodes) {
clusterNode *n = NULL;
listIter li;
listNode *ln;
listRewind(nodes,&li);
while ((ln = listNext(&li)) != NULL) {
clusterNode *node = listNodeValue(ln);
if (!nodeFailed(node)) {
n = node;
break;
}
}
if (!n) return NULL;
return clusterNodeGetMaster(n);
}
/* -----------------------------------------------------------------------------
* CLUSTER config epoch handling
* -------------------------------------------------------------------------- */
@@ -2217,6 +2267,7 @@ void clusterProcessGossipSection(clusterMsg *hdr, clusterLink *link) {
node->tls_port = msg_tls_port;
node->cport = ntohs(g->cport);
clusterAddNode(node);
clusterAddNodeToShard(node->shard_id, node);
}
}
@@ -2394,7 +2445,6 @@ void clusterUpdateSlotsConfigWith(clusterNode *sender, uint64_t senderConfigEpoc
}
clusterDelSlot(j);
clusterAddSlot(sender,j);
bitmapClearBit(server.cluster->owner_not_claiming_slot, j);
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG|
CLUSTER_TODO_UPDATE_STATE|
CLUSTER_TODO_FSYNC_CONFIG);
@@ -2609,9 +2659,6 @@ uint32_t writePingExt(clusterMsg *hdr, int gossipcount) {
extensions++;
if (hdr != NULL) {
if (extensions != 0) {
hdr->mflags[0] |= CLUSTERMSG_FLAG0_EXT_DATA;
}
hdr->extensions = htons(extensions);
}
@@ -2641,8 +2688,7 @@ void clusterProcessPingExtensions(clusterMsg *hdr, clusterLink *link) {
clusterNode *n = clusterLookupNode(forgotten_node_ext->name, CLUSTER_NAMELEN);
if (n && n != myself && !(nodeIsSlave(myself) && myself->slaveof == n)) {
sds id = sdsnewlen(forgotten_node_ext->name, CLUSTER_NAMELEN);
dictEntry *de = dictAddRaw(server.cluster->nodes_black_list, id, NULL);
serverAssert(de != NULL);
dictEntry *de = dictAddOrFind(server.cluster->nodes_black_list, id);
uint64_t expire = server.unixtime + ntohu64(forgotten_node_ext->ttl);
dictSetUnsignedIntegerVal(de, expire);
clusterDelNode(n);
@@ -2660,11 +2706,24 @@ void clusterProcessPingExtensions(clusterMsg *hdr, clusterLink *link) {
/* We know this will be valid since we validated it ahead of time */
ext = getNextPingExt(ext);
}
/* If the node did not send us a hostname extension, assume
* they don't have an announced hostname. Otherwise, we'll
* set it now. */
updateAnnouncedHostname(sender, ext_hostname);
updateAnnouncedHumanNodename(sender, ext_humannodename);
/* If the node did not send us a shard-id extension, it means the sender
* does not support it (old version), node->shard_id is randomly generated.
* A cluster-wide consensus for the node's shard_id is not necessary.
* The key is maintaining consistency of the shard_id on each individual 7.2 node.
* As the cluster progressively upgrades to version 7.2, we can expect the shard_ids
* across all nodes to naturally converge and align.
*
* If sender is a replica, set the shard_id to the shard_id of its master.
* Otherwise, we'll set it now. */
if (ext_shardid == NULL) ext_shardid = clusterNodeGetMaster(sender)->shard_id;
updateShardId(sender, ext_shardid);
}
@@ -2790,6 +2849,9 @@ int clusterProcessPacket(clusterLink *link) {
}
sender = getNodeFromLinkAndMsg(link, hdr);
if (sender && (hdr->mflags[0] & CLUSTERMSG_FLAG0_EXT_DATA)) {
sender->flags |= CLUSTER_NODE_EXTENSIONS_SUPPORTED;
}
/* Update the last time we saw any data from this node. We
* use this in order to avoid detecting a timeout from a node that
@@ -2991,7 +3053,50 @@ int clusterProcessPacket(clusterLink *link) {
if (nodeIsMaster(sender)) {
/* Master turned into a slave! Reconfigure the node. */
clusterDelNodeSlots(sender);
if (master && !memcmp(master->shard_id, sender->shard_id, CLUSTER_NAMELEN)) {
/* `sender` was a primary and was in the same shard as `master`, its new primary */
if (sender->configEpoch > senderConfigEpoch) {
serverLog(LL_NOTICE,
"Ignore stale message from %.40s (%s) in shard %.40s;"
" gossip config epoch: %llu, current config epoch: %llu",
sender->name,
sender->human_nodename,
sender->shard_id,
(unsigned long long)senderConfigEpoch,
(unsigned long long)sender->configEpoch);
} else {
/* A failover occurred in the shard where `sender` belongs to and `sender` is no longer
* a primary. Update slot assignment to `master`, which is the new primary in the shard */
int slots = clusterMoveNodeSlots(sender, master);
/* `master` is still a `slave` in this observer node's view; update its role and configEpoch */
clusterSetNodeAsMaster(master);
master->configEpoch = senderConfigEpoch;
serverLog(LL_NOTICE, "A failover occurred in shard %.40s; node %.40s (%s)"
" lost %d slot(s) to node %.40s (%s) with a config epoch of %llu",
sender->shard_id,
sender->name,
sender->human_nodename,
slots,
master->name,
master->human_nodename,
(unsigned long long) master->configEpoch);
}
} else {
/* `sender` was moved to another shard and has become a replica, remove its slot assignment */
int slots = clusterDelNodeSlots(sender);
serverLog(LL_NOTICE, "Node %.40s (%s) is no longer master of shard %.40s;"
" removed all %d slot(s) it used to own",
sender->name,
sender->human_nodename,
sender->shard_id,
slots);
if (master != NULL) {
serverLog(LL_NOTICE, "Node %.40s (%s) is now part of shard %.40s",
sender->name,
sender->human_nodename,
master->shard_id);
}
}
sender->flags &= ~(CLUSTER_NODE_MASTER|
CLUSTER_NODE_MIGRATE_TO);
sender->flags |= CLUSTER_NODE_SLAVE;
@@ -3008,6 +3113,10 @@ int clusterProcessPacket(clusterLink *link) {
clusterNodeAddSlave(master,sender);
sender->slaveof = master;
/* Update the shard_id when a replica is connected to its
* primary in the very first time. */
updateShardId(sender, master->shard_id);
/* Update config. */
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG);
}
@@ -3509,7 +3618,8 @@ static void clusterBuildMessageHdr(clusterMsg *hdr, int type, size_t msglen) {
/* Set the message flags. */
if (nodeIsMaster(myself) && server.cluster->mf_end)
hdr->mflags[0] |= CLUSTERMSG_FLAG0_PAUSED;
hdr->mflags[0] |= CLUSTERMSG_FLAG0_EXT_DATA; /* Always make other nodes know that
* this node supports extension data. */
hdr->totlen = htonl(msglen);
}
@@ -3587,7 +3697,9 @@ void clusterSendPing(clusterLink *link, int type) {
* to put inside the packet. */
estlen = sizeof(clusterMsg) - sizeof(union clusterMsgData);
estlen += (sizeof(clusterMsgDataGossip)*(wanted + pfail_wanted));
estlen += writePingExt(NULL, 0);
if (link->node && nodeSupportsExtensions(link->node)) {
estlen += writePingExt(NULL, 0);
}
/* Note: clusterBuildMessageHdr() expects the buffer to be always at least
* sizeof(clusterMsg) or more. */
if (estlen < (int)sizeof(clusterMsg)) estlen = sizeof(clusterMsg);
@@ -3655,7 +3767,9 @@ void clusterSendPing(clusterLink *link, int type) {
/* Compute the actual total length and send! */
uint32_t totlen = 0;
totlen += writePingExt(hdr, gossipcount);
if (link->node && nodeSupportsExtensions(link->node)) {
totlen += writePingExt(hdr, gossipcount);
}
totlen += sizeof(clusterMsg)-sizeof(union clusterMsgData);
totlen += (sizeof(clusterMsgDataGossip)*gossipcount);
serverAssert(gossipcount < USHRT_MAX);
@@ -4737,10 +4851,13 @@ void clusterCron(void) {
/* Timeout reached. Set the node as possibly failing if it is
* not already in this state. */
if (!(node->flags & (CLUSTER_NODE_PFAIL|CLUSTER_NODE_FAIL))) {
serverLog(LL_DEBUG,"*** NODE %.40s possibly failing",
node->name);
node->flags |= CLUSTER_NODE_PFAIL;
update_state = 1;
if (myself->flags & CLUSTER_NODE_MASTER && server.cluster->size == 1) {
markNodeAsFailingIfNeeded(node);
} else {
serverLog(LL_DEBUG,"*** NODE %.40s possibly failing", node->name);
}
}
}
}
@@ -4922,17 +5039,31 @@ int clusterDelSlot(int slot) {
if (!n) return C_ERR;
/* Cleanup the channels in master/replica as part of slot deletion. */
list *nodes_for_slot = clusterGetNodesInMyShard(n);
serverAssert(nodes_for_slot != NULL);
listNode *ln = listSearchKey(nodes_for_slot, myself);
if (ln != NULL) {
removeChannelsInSlot(slot);
}
removeChannelsInSlot(slot);
/* Clear the slot bit. */
serverAssert(clusterNodeClearSlotBit(n,slot) == 1);
server.cluster->slots[slot] = NULL;
/* Make owner_not_claiming_slot flag consistent with slot ownership information. */
bitmapClearBit(server.cluster->owner_not_claiming_slot, slot);
return C_OK;
}
/* Transfer slots from `from_node` to `to_node`.
* Iterates over all cluster slots, transferring each slot covered by `from_node` to `to_node`.
* Counts and returns the number of slots transferred. */
int clusterMoveNodeSlots(clusterNode *from_node, clusterNode *to_node) {
int processed = 0;
for (int j = 0; j < CLUSTER_SLOTS; j++) {
if (clusterNodeGetSlotBit(from_node, j)) {
clusterDelSlot(j);
clusterAddSlot(to_node, j);
processed++;
}
}
return processed;
}
/* Delete all the slots associated with the specified node.
* The number of deleted slots is returned. */
int clusterDelNodeSlots(clusterNode *node) {
@@ -5570,7 +5701,7 @@ void addNodeToNodeReply(client *c, clusterNode *node) {
}
/* Report TLS ports to TLS client, and report non-TLS port to non-TLS client. */
addReplyLongLong(c, getNodeClientPort(node, connIsTLS(c->conn)));
addReplyLongLong(c, getNodeClientPort(node, shouldReturnTlsInfo()));
addReplyBulkCBuffer(c, node->name, CLUSTER_NAMELEN);
/* Add the additional endpoint information, this is all the known networking information
@@ -5692,18 +5823,17 @@ void addNodeDetailsToShardReply(client *c, clusterNode *node) {
/* Add the shard reply of a single shard based off the given primary node. */
void addShardReplyForClusterShards(client *c, list *nodes) {
serverAssert(listLength(nodes) > 0);
clusterNode *n = listNodeValue(listFirst(nodes));
addReplyMapLen(c, 2);
addReplyBulkCString(c, "slots");
/* Use slot_info_pairs from the primary only */
while (n->slaveof != NULL) n = n->slaveof;
if (n->slot_info_pairs != NULL) {
clusterNode *n = clusterGetMasterFromShard(nodes);
if (n && n->slot_info_pairs != NULL) {
serverAssert((n->slot_info_pairs_count % 2) == 0);
addReplyArrayLen(c, n->slot_info_pairs_count);
for (int i = 0; i < n->slot_info_pairs_count; i++)
addReplyBulkLongLong(c, (unsigned long)n->slot_info_pairs[i]);
addReplyLongLong(c, (unsigned long)n->slot_info_pairs[i]);
} else {
/* If no slot info pair is provided, the node owns no slots */
addReplyArrayLen(c, 0);
@@ -5946,7 +6076,7 @@ NULL
} else if (!strcasecmp(c->argv[1]->ptr,"nodes") && c->argc == 2) {
/* CLUSTER NODES */
/* Report TLS ports to TLS client, and report non-TLS port to non-TLS client. */
sds nodes = clusterGenNodesDescription(c, 0, connIsTLS(c->conn));
sds nodes = clusterGenNodesDescription(c, 0, shouldReturnTlsInfo());
addReplyVerbatim(c,nodes,sdslen(nodes),"txt");
sdsfree(nodes);
} else if (!strcasecmp(c->argv[1]->ptr,"myid") && c->argc == 2) {
@@ -6312,7 +6442,7 @@ NULL
/* Report TLS ports to TLS client, and report non-TLS port to non-TLS client. */
addReplyArrayLen(c,n->numslaves);
for (j = 0; j < n->numslaves; j++) {
sds ni = clusterGenNodeDescription(c, n->slaves[j], connIsTLS(c->conn));
sds ni = clusterGenNodeDescription(c, n->slaves[j], shouldReturnTlsInfo());
addReplyBulkCString(c,ni);
sdsfree(ni);
}
@@ -7209,6 +7339,7 @@ clusterNode *getNodeByQuery(client *c, struct redisCommand *cmd, robj **argv, in
multiCmd mc;
int i, slot = 0, migrating_slot = 0, importing_slot = 0, missing_keys = 0,
existing_keys = 0;
int pubsubshard_included = 0; /* Flag to indicate if a pubsub shard cmd is included. */
/* Allow any key to be set if a module disabled cluster redirections. */
if (server.cluster_module_flags & CLUSTER_MODULE_FLAG_NO_REDIRECTION)
@@ -7240,10 +7371,6 @@ clusterNode *getNodeByQuery(client *c, struct redisCommand *cmd, robj **argv, in
mc.cmd = cmd;
}
int is_pubsubshard = cmd->proc == ssubscribeCommand ||
cmd->proc == sunsubscribeCommand ||
cmd->proc == spublishCommand;
/* Check that all the keys are in the same hash slot, and obtain this
* slot and the node associated. */
for (i = 0; i < ms->count; i++) {
@@ -7256,6 +7383,13 @@ clusterNode *getNodeByQuery(client *c, struct redisCommand *cmd, robj **argv, in
margc = ms->commands[i].argc;
margv = ms->commands[i].argv;
/* Only valid for sharded pubsub as regular pubsub can operate on any node and bypasses this layer. */
if (!pubsubshard_included &&
doesCommandHaveChannelsWithFlags(mcmd, CMD_CHANNEL_PUBLISH | CMD_CHANNEL_SUBSCRIBE))
{
pubsubshard_included = 1;
}
getKeysResult result = GETKEYS_RESULT_INIT;
numkeys = getKeysFromCommand(mcmd,margv,margc,&result);
keyindex = result.keys;
@@ -7319,7 +7453,7 @@ clusterNode *getNodeByQuery(client *c, struct redisCommand *cmd, robj **argv, in
* node until the migration completes with CLUSTER SETSLOT <slot>
* NODE <node-id>. */
int flags = LOOKUP_NOTOUCH | LOOKUP_NOSTATS | LOOKUP_NONOTIFY | LOOKUP_NOEXPIRE;
if ((migrating_slot || importing_slot) && !is_pubsubshard)
if ((migrating_slot || importing_slot) && !pubsubshard_included)
{
if (lookupKeyReadWithFlags(&server.db[0], thiskey, flags) == NULL) missing_keys++;
else existing_keys++;
@@ -7336,7 +7470,7 @@ clusterNode *getNodeByQuery(client *c, struct redisCommand *cmd, robj **argv, in
/* Cluster is globally down but we got keys? We only serve the request
* if it is a read command and when allow_reads_when_down is enabled. */
if (server.cluster->state != CLUSTER_OK) {
if (is_pubsubshard) {
if (pubsubshard_included) {
if (!server.cluster_allow_pubsubshard_when_down) {
if (error_code) *error_code = CLUSTER_REDIR_DOWN_STATE;
return NULL;
@@ -7399,7 +7533,7 @@ clusterNode *getNodeByQuery(client *c, struct redisCommand *cmd, robj **argv, in
* is serving, we can reply without redirection. */
int is_write_command = (cmd_flags & CMD_WRITE) ||
(c->cmd->proc == execCommand && (c->mstate.cmd_flags & CMD_WRITE));
if (((c->flags & CLIENT_READONLY) || is_pubsubshard) &&
if (((c->flags & CLIENT_READONLY) || pubsubshard_included) &&
!is_write_command &&
nodeIsSlave(myself) &&
myself->slaveof == n)
@@ -7438,7 +7572,7 @@ void clusterRedirectClient(client *c, clusterNode *n, int hashslot, int error_co
error_code == CLUSTER_REDIR_ASK)
{
/* Report TLS ports to TLS client, and report non-TLS port to non-TLS client. */
int port = getNodeClientPort(n, connIsTLS(c->conn));
int port = getNodeClientPort(n, shouldReturnTlsInfo());
addReplyErrorSds(c,sdscatprintf(sdsempty(),
"-%s %d %s:%d",
(error_code == CLUSTER_REDIR_ASK) ? "ASK" : "MOVED",
@@ -7631,6 +7765,11 @@ unsigned int countKeysInSlot(unsigned int hashslot) {
return (*server.db->slots_to_keys).by_slot[hashslot].count;
}
clusterNode *clusterNodeGetMaster(clusterNode *node) {
while (node->slaveof != NULL) node = node->slaveof;
return node;
}
/* -----------------------------------------------------------------------------
* Operation(s) on channel rax tree.
* -------------------------------------------------------------------------- */
+3
View File
@@ -56,6 +56,7 @@ typedef struct clusterLink {
#define CLUSTER_NODE_MEET 128 /* Send a MEET message to this node */
#define CLUSTER_NODE_MIGRATE_TO 256 /* Master eligible for replica migration. */
#define CLUSTER_NODE_NOFAILOVER 512 /* Slave will not try to failover. */
#define CLUSTER_NODE_EXTENSIONS_SUPPORTED 1024 /* This node supports extensions. */
#define CLUSTER_NODE_NULL_NAME "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
#define nodeIsMaster(n) ((n)->flags & CLUSTER_NODE_MASTER)
@@ -66,6 +67,7 @@ typedef struct clusterLink {
#define nodeTimedOut(n) ((n)->flags & CLUSTER_NODE_PFAIL)
#define nodeFailed(n) ((n)->flags & CLUSTER_NODE_FAIL)
#define nodeCantFailover(n) ((n)->flags & CLUSTER_NODE_NOFAILOVER)
#define nodeSupportsExtensions(n) ((n)->flags & CLUSTER_NODE_EXTENSIONS_SUPPORTED)
/* Reasons why a slave is not able to failover. */
#define CLUSTER_CANT_FAILOVER_NONE 0
@@ -413,6 +415,7 @@ void clusterInit(void);
void clusterInitListeners(void);
void clusterCron(void);
void clusterBeforeSleep(void);
clusterNode *clusterNodeGetMaster(clusterNode *node);
clusterNode *getNodeByQuery(client *c, struct redisCommand *cmd, robj **argv, int argc, int *hashslot, int *ask);
int verifyClusterNodeId(const char *name, int length);
clusterNode *clusterLookupNode(const char *name, int length);
+76 -21
View File
@@ -1391,7 +1391,10 @@ struct COMMAND_ARG CLIENT_REPLY_Args[] = {
#ifndef SKIP_CMD_TIPS_TABLE
/* CLIENT SETINFO tips */
#define CLIENT_SETINFO_Tips NULL
const char *CLIENT_SETINFO_Tips[] = {
"request_policy:all_nodes",
"response_policy:all_succeeded",
};
#endif
#ifndef SKIP_CMD_KEY_SPECS_TABLE
@@ -1419,7 +1422,10 @@ struct COMMAND_ARG CLIENT_SETINFO_Args[] = {
#ifndef SKIP_CMD_TIPS_TABLE
/* CLIENT SETNAME tips */
#define CLIENT_SETNAME_Tips NULL
const char *CLIENT_SETNAME_Tips[] = {
"request_policy:all_nodes",
"response_policy:all_succeeded",
};
#endif
#ifndef SKIP_CMD_KEY_SPECS_TABLE
@@ -1543,8 +1549,8 @@ struct COMMAND_STRUCT CLIENT_Subcommands[] = {
{MAKE_CMD("no-touch","Controls whether commands sent by the client affect the LRU/LFU of accessed keys.","O(1)","7.2.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_NO_TOUCH_History,0,CLIENT_NO_TOUCH_Tips,0,clientCommand,3,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE,ACL_CATEGORY_CONNECTION,CLIENT_NO_TOUCH_Keyspecs,0,NULL,1),.args=CLIENT_NO_TOUCH_Args},
{MAKE_CMD("pause","Suspends commands processing.","O(1)","3.0.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_PAUSE_History,1,CLIENT_PAUSE_Tips,0,clientCommand,-3,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_CONNECTION,CLIENT_PAUSE_Keyspecs,0,NULL,2),.args=CLIENT_PAUSE_Args},
{MAKE_CMD("reply","Instructs the server whether to reply to commands.","O(1)","3.2.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_REPLY_History,0,CLIENT_REPLY_Tips,0,clientCommand,3,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_CONNECTION,CLIENT_REPLY_Keyspecs,0,NULL,1),.args=CLIENT_REPLY_Args},
{MAKE_CMD("setinfo","Sets information specific to the client or connection.","O(1)","7.2.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_SETINFO_History,0,CLIENT_SETINFO_Tips,0,clientSetinfoCommand,4,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_CONNECTION,CLIENT_SETINFO_Keyspecs,0,NULL,1),.args=CLIENT_SETINFO_Args},
{MAKE_CMD("setname","Sets the connection name.","O(1)","2.6.9",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_SETNAME_History,0,CLIENT_SETNAME_Tips,0,clientCommand,3,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_CONNECTION,CLIENT_SETNAME_Keyspecs,0,NULL,1),.args=CLIENT_SETNAME_Args},
{MAKE_CMD("setinfo","Sets information specific to the client or connection.","O(1)","7.2.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_SETINFO_History,0,CLIENT_SETINFO_Tips,2,clientSetinfoCommand,4,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_CONNECTION,CLIENT_SETINFO_Keyspecs,0,NULL,1),.args=CLIENT_SETINFO_Args},
{MAKE_CMD("setname","Sets the connection name.","O(1)","2.6.9",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_SETNAME_History,0,CLIENT_SETNAME_Tips,2,clientCommand,3,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_CONNECTION,CLIENT_SETNAME_Keyspecs,0,NULL,1),.args=CLIENT_SETNAME_Args},
{MAKE_CMD("tracking","Controls server-assisted client-side caching for the connection.","O(1). Some options may introduce additional complexity.","6.0.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_TRACKING_History,0,CLIENT_TRACKING_Tips,0,clientCommand,-3,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_CONNECTION,CLIENT_TRACKING_Keyspecs,0,NULL,7),.args=CLIENT_TRACKING_Args},
{MAKE_CMD("trackinginfo","Returns information about server-assisted client-side caching for the connection.","O(1)","6.2.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_TRACKINGINFO_History,0,CLIENT_TRACKINGINFO_Tips,0,clientCommand,2,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_CONNECTION,CLIENT_TRACKINGINFO_Keyspecs,0,NULL,0)},
{MAKE_CMD("unblock","Unblocks a client blocked by a blocking command from a different connection.","O(log N) where N is the number of client connections","5.0.0",CMD_DOC_NONE,NULL,NULL,"connection",COMMAND_GROUP_CONNECTION,CLIENT_UNBLOCK_History,0,CLIENT_UNBLOCK_Tips,0,clientCommand,-3,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,ACL_CATEGORY_CONNECTION,CLIENT_UNBLOCK_Keyspecs,0,NULL,2),.args=CLIENT_UNBLOCK_Args},
@@ -5886,7 +5892,10 @@ struct COMMAND_ARG ACL_CAT_Args[] = {
#ifndef SKIP_CMD_TIPS_TABLE
/* ACL DELUSER tips */
#define ACL_DELUSER_Tips NULL
const char *ACL_DELUSER_Tips[] = {
"request_policy:all_nodes",
"response_policy:all_succeeded",
};
#endif
#ifndef SKIP_CMD_KEY_SPECS_TABLE
@@ -6060,7 +6069,10 @@ struct COMMAND_ARG ACL_LOG_Args[] = {
#ifndef SKIP_CMD_TIPS_TABLE
/* ACL SAVE tips */
#define ACL_SAVE_Tips NULL
const char *ACL_SAVE_Tips[] = {
"request_policy:all_nodes",
"response_policy:all_succeeded",
};
#endif
#ifndef SKIP_CMD_KEY_SPECS_TABLE
@@ -6080,7 +6092,10 @@ commandHistory ACL_SETUSER_History[] = {
#ifndef SKIP_CMD_TIPS_TABLE
/* ACL SETUSER tips */
#define ACL_SETUSER_Tips NULL
const char *ACL_SETUSER_Tips[] = {
"request_policy:all_nodes",
"response_policy:all_succeeded",
};
#endif
#ifndef SKIP_CMD_KEY_SPECS_TABLE
@@ -6131,7 +6146,7 @@ struct COMMAND_ARG ACL_SETUSER_Args[] = {
/* ACL command table */
struct COMMAND_STRUCT ACL_Subcommands[] = {
{MAKE_CMD("cat","Lists the ACL categories, or the commands inside a category.","O(1) since the categories and commands are a fixed set.","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_CAT_History,0,ACL_CAT_Tips,0,aclCommand,-2,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_CAT_Keyspecs,0,NULL,1),.args=ACL_CAT_Args},
{MAKE_CMD("deluser","Deletes ACL users, and terminates their connections.","O(1) amortized time considering the typical user.","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_DELUSER_History,0,ACL_DELUSER_Tips,0,aclCommand,-3,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_DELUSER_Keyspecs,0,NULL,1),.args=ACL_DELUSER_Args},
{MAKE_CMD("deluser","Deletes ACL users, and terminates their connections.","O(1) amortized time considering the typical user.","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_DELUSER_History,0,ACL_DELUSER_Tips,2,aclCommand,-3,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_DELUSER_Keyspecs,0,NULL,1),.args=ACL_DELUSER_Args},
{MAKE_CMD("dryrun","Simulates the execution of a command by a user, without executing the command.","O(1).","7.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_DRYRUN_History,0,ACL_DRYRUN_Tips,0,aclCommand,-4,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_DRYRUN_Keyspecs,0,NULL,3),.args=ACL_DRYRUN_Args},
{MAKE_CMD("genpass","Generates a pseudorandom, secure password that can be used to identify ACL users.","O(1)","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_GENPASS_History,0,ACL_GENPASS_Tips,0,aclCommand,-2,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_GENPASS_Keyspecs,0,NULL,1),.args=ACL_GENPASS_Args},
{MAKE_CMD("getuser","Lists the ACL rules of a user.","O(N). Where N is the number of password, command and pattern rules that the user has.","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_GETUSER_History,2,ACL_GETUSER_Tips,0,aclCommand,3,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_GETUSER_Keyspecs,0,NULL,1),.args=ACL_GETUSER_Args},
@@ -6139,8 +6154,8 @@ struct COMMAND_STRUCT ACL_Subcommands[] = {
{MAKE_CMD("list","Dumps the effective rules in ACL file format.","O(N). Where N is the number of configured users.","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_LIST_History,0,ACL_LIST_Tips,0,aclCommand,2,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_LIST_Keyspecs,0,NULL,0)},
{MAKE_CMD("load","Reloads the rules from the configured ACL file.","O(N). Where N is the number of configured users.","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_LOAD_History,0,ACL_LOAD_Tips,0,aclCommand,2,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_LOAD_Keyspecs,0,NULL,0)},
{MAKE_CMD("log","Lists recent security events generated due to ACL rules.","O(N) with N being the number of entries shown.","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_LOG_History,1,ACL_LOG_Tips,0,aclCommand,-2,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_LOG_Keyspecs,0,NULL,1),.args=ACL_LOG_Args},
{MAKE_CMD("save","Saves the effective ACL rules in the configured ACL file.","O(N). Where N is the number of configured users.","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_SAVE_History,0,ACL_SAVE_Tips,0,aclCommand,2,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_SAVE_Keyspecs,0,NULL,0)},
{MAKE_CMD("setuser","Creates and modifies an ACL user and its rules.","O(N). Where N is the number of rules provided.","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_SETUSER_History,2,ACL_SETUSER_Tips,0,aclCommand,-3,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_SETUSER_Keyspecs,0,NULL,2),.args=ACL_SETUSER_Args},
{MAKE_CMD("save","Saves the effective ACL rules in the configured ACL file.","O(N). Where N is the number of configured users.","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_SAVE_History,0,ACL_SAVE_Tips,2,aclCommand,2,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_SAVE_Keyspecs,0,NULL,0)},
{MAKE_CMD("setuser","Creates and modifies an ACL user and its rules.","O(N). Where N is the number of rules provided.","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_SETUSER_History,2,ACL_SETUSER_Tips,2,aclCommand,-3,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_SETUSER_Keyspecs,0,NULL,2),.args=ACL_SETUSER_Args},
{MAKE_CMD("users","Lists all ACL users.","O(N). Where N is the number of configured users.","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_USERS_History,0,ACL_USERS_Tips,0,aclCommand,2,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_USERS_Keyspecs,0,NULL,0)},
{MAKE_CMD("whoami","Returns the authenticated username of the current connection.","O(1)","6.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ACL_WHOAMI_History,0,ACL_WHOAMI_Tips,0,aclCommand,2,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_SENTINEL,0,ACL_WHOAMI_Keyspecs,0,NULL,0)},
{0}
@@ -6446,7 +6461,10 @@ struct COMMAND_ARG CONFIG_GET_Args[] = {
#ifndef SKIP_CMD_TIPS_TABLE
/* CONFIG RESETSTAT tips */
#define CONFIG_RESETSTAT_Tips NULL
const char *CONFIG_RESETSTAT_Tips[] = {
"request_policy:all_nodes",
"response_policy:all_succeeded",
};
#endif
#ifndef SKIP_CMD_KEY_SPECS_TABLE
@@ -6463,7 +6481,10 @@ struct COMMAND_ARG CONFIG_GET_Args[] = {
#ifndef SKIP_CMD_TIPS_TABLE
/* CONFIG REWRITE tips */
#define CONFIG_REWRITE_Tips NULL
const char *CONFIG_REWRITE_Tips[] = {
"request_policy:all_nodes",
"response_policy:all_succeeded",
};
#endif
#ifndef SKIP_CMD_KEY_SPECS_TABLE
@@ -6508,8 +6529,8 @@ struct COMMAND_ARG CONFIG_SET_Args[] = {
struct COMMAND_STRUCT CONFIG_Subcommands[] = {
{MAKE_CMD("get","Returns the effective values of configuration parameters.","O(N) when N is the number of configuration parameters provided","2.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,CONFIG_GET_History,1,CONFIG_GET_Tips,0,configGetCommand,-3,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE,0,CONFIG_GET_Keyspecs,0,NULL,1),.args=CONFIG_GET_Args},
{MAKE_CMD("help","Returns helpful text about the different subcommands.","O(1)","5.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,CONFIG_HELP_History,0,CONFIG_HELP_Tips,0,configHelpCommand,2,CMD_LOADING|CMD_STALE,0,CONFIG_HELP_Keyspecs,0,NULL,0)},
{MAKE_CMD("resetstat","Resets the server's statistics.","O(1)","2.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,CONFIG_RESETSTAT_History,0,CONFIG_RESETSTAT_Tips,0,configResetStatCommand,2,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE,0,CONFIG_RESETSTAT_Keyspecs,0,NULL,0)},
{MAKE_CMD("rewrite","Persists the effective configuration to file.","O(1)","2.8.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,CONFIG_REWRITE_History,0,CONFIG_REWRITE_Tips,0,configRewriteCommand,2,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE,0,CONFIG_REWRITE_Keyspecs,0,NULL,0)},
{MAKE_CMD("resetstat","Resets the server's statistics.","O(1)","2.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,CONFIG_RESETSTAT_History,0,CONFIG_RESETSTAT_Tips,2,configResetStatCommand,2,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE,0,CONFIG_RESETSTAT_Keyspecs,0,NULL,0)},
{MAKE_CMD("rewrite","Persists the effective configuration to file.","O(1)","2.8.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,CONFIG_REWRITE_History,0,CONFIG_REWRITE_Tips,2,configRewriteCommand,2,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE,0,CONFIG_REWRITE_Keyspecs,0,NULL,0)},
{MAKE_CMD("set","Sets configuration parameters in-flight.","O(N) when N is the number of configuration parameters provided","2.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,CONFIG_SET_History,1,CONFIG_SET_Tips,2,configSetCommand,-4,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE,0,CONFIG_SET_Keyspecs,0,NULL,1),.args=CONFIG_SET_Args},
{0}
};
@@ -6862,7 +6883,7 @@ const char *LATENCY_LATEST_Tips[] = {
/* LATENCY RESET tips */
const char *LATENCY_RESET_Tips[] = {
"request_policy:all_nodes",
"response_policy:all_succeeded",
"response_policy:agg_sum",
};
#endif
@@ -7292,12 +7313,29 @@ struct COMMAND_ARG PSYNC_Args[] = {
#define REPLICAOF_Keyspecs NULL
#endif
/* REPLICAOF argument table */
struct COMMAND_ARG REPLICAOF_Args[] = {
/* REPLICAOF args host_port argument table */
struct COMMAND_ARG REPLICAOF_args_host_port_Subargs[] = {
{MAKE_ARG("host",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("port",ARG_TYPE_INTEGER,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
};
/* REPLICAOF args no_one argument table */
struct COMMAND_ARG REPLICAOF_args_no_one_Subargs[] = {
{MAKE_ARG("no",ARG_TYPE_PURE_TOKEN,-1,"NO",NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("one",ARG_TYPE_PURE_TOKEN,-1,"ONE",NULL,NULL,CMD_ARG_NONE,0,NULL)},
};
/* REPLICAOF args argument table */
struct COMMAND_ARG REPLICAOF_args_Subargs[] = {
{MAKE_ARG("host-port",ARG_TYPE_BLOCK,-1,NULL,NULL,NULL,CMD_ARG_NONE,2,NULL),.subargs=REPLICAOF_args_host_port_Subargs},
{MAKE_ARG("no-one",ARG_TYPE_BLOCK,-1,NULL,NULL,NULL,CMD_ARG_NONE,2,NULL),.subargs=REPLICAOF_args_no_one_Subargs},
};
/* REPLICAOF argument table */
struct COMMAND_ARG REPLICAOF_Args[] = {
{MAKE_ARG("args",ARG_TYPE_ONEOF,-1,NULL,NULL,NULL,CMD_ARG_NONE,2,NULL),.subargs=REPLICAOF_args_Subargs},
};
/********** RESTORE_ASKING ********************/
#ifndef SKIP_CMD_HISTORY_TABLE
@@ -7416,12 +7454,29 @@ struct COMMAND_ARG SHUTDOWN_Args[] = {
#define SLAVEOF_Keyspecs NULL
#endif
/* SLAVEOF argument table */
struct COMMAND_ARG SLAVEOF_Args[] = {
/* SLAVEOF args host_port argument table */
struct COMMAND_ARG SLAVEOF_args_host_port_Subargs[] = {
{MAKE_ARG("host",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("port",ARG_TYPE_INTEGER,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
};
/* SLAVEOF args no_one argument table */
struct COMMAND_ARG SLAVEOF_args_no_one_Subargs[] = {
{MAKE_ARG("no",ARG_TYPE_PURE_TOKEN,-1,"NO",NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("one",ARG_TYPE_PURE_TOKEN,-1,"ONE",NULL,NULL,CMD_ARG_NONE,0,NULL)},
};
/* SLAVEOF args argument table */
struct COMMAND_ARG SLAVEOF_args_Subargs[] = {
{MAKE_ARG("host-port",ARG_TYPE_BLOCK,-1,NULL,NULL,NULL,CMD_ARG_NONE,2,NULL),.subargs=SLAVEOF_args_host_port_Subargs},
{MAKE_ARG("no-one",ARG_TYPE_BLOCK,-1,NULL,NULL,NULL,CMD_ARG_NONE,2,NULL),.subargs=SLAVEOF_args_no_one_Subargs},
};
/* SLAVEOF argument table */
struct COMMAND_ARG SLAVEOF_Args[] = {
{MAKE_ARG("args",ARG_TYPE_ONEOF,-1,NULL,NULL,NULL,CMD_ARG_NONE,2,NULL),.subargs=SLAVEOF_args_Subargs},
};
/********** SLOWLOG GET ********************/
#ifndef SKIP_CMD_HISTORY_TABLE
@@ -10731,12 +10786,12 @@ struct COMMAND_STRUCT redisCommandTable[] = {
{MAKE_CMD("monitor","Listens for all requests received by the server in real-time.",NULL,"1.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,MONITOR_History,0,MONITOR_Tips,0,monitorCommand,1,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE,0,MONITOR_Keyspecs,0,NULL,0)},
{MAKE_CMD("psync","An internal command used in replication.",NULL,"2.8.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,PSYNC_History,0,PSYNC_Tips,0,syncCommand,-3,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_NO_MULTI|CMD_NOSCRIPT,0,PSYNC_Keyspecs,0,NULL,2),.args=PSYNC_Args},
{MAKE_CMD("replconf","An internal command for configuring the replication stream.","O(1)","3.0.0",CMD_DOC_SYSCMD,NULL,NULL,"server",COMMAND_GROUP_SERVER,REPLCONF_History,0,REPLCONF_Tips,0,replconfCommand,-1,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_ALLOW_BUSY,0,REPLCONF_Keyspecs,0,NULL,0)},
{MAKE_CMD("replicaof","Configures a server as replica of another, or promotes it to a master.","O(1)","5.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,REPLICAOF_History,0,REPLICAOF_Tips,0,replicaofCommand,3,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_NOSCRIPT|CMD_STALE,0,REPLICAOF_Keyspecs,0,NULL,2),.args=REPLICAOF_Args},
{MAKE_CMD("replicaof","Configures a server as replica of another, or promotes it to a master.","O(1)","5.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,REPLICAOF_History,0,REPLICAOF_Tips,0,replicaofCommand,3,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_NOSCRIPT|CMD_STALE,0,REPLICAOF_Keyspecs,0,NULL,1),.args=REPLICAOF_Args},
{MAKE_CMD("restore-asking","An internal command for migrating keys in a cluster.","O(1) to create the new key and additional O(N*M) to reconstruct the serialized value, where N is the number of Redis objects composing the value and M their average size. For small string values the time complexity is thus O(1)+O(1*M) where M is small, so simply O(1). However for sorted set values the complexity is O(N*M*log(N)) because inserting values into sorted sets is O(log(N)).","3.0.0",CMD_DOC_SYSCMD,NULL,NULL,"server",COMMAND_GROUP_SERVER,RESTORE_ASKING_History,3,RESTORE_ASKING_Tips,0,restoreCommand,-4,CMD_WRITE|CMD_DENYOOM|CMD_ASKING,ACL_CATEGORY_KEYSPACE|ACL_CATEGORY_DANGEROUS,RESTORE_ASKING_Keyspecs,1,NULL,7),.args=RESTORE_ASKING_Args},
{MAKE_CMD("role","Returns the replication role.","O(1)","2.8.12",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,ROLE_History,0,ROLE_Tips,0,roleCommand,1,CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_FAST|CMD_SENTINEL,ACL_CATEGORY_ADMIN|ACL_CATEGORY_DANGEROUS,ROLE_Keyspecs,0,NULL,0)},
{MAKE_CMD("save","Synchronously saves the database(s) to disk.","O(N) where N is the total number of keys in all databases","1.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,SAVE_History,0,SAVE_Tips,0,saveCommand,1,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_NOSCRIPT|CMD_NO_MULTI,0,SAVE_Keyspecs,0,NULL,0)},
{MAKE_CMD("shutdown","Synchronously saves the database(s) to disk and shuts down the Redis server.","O(N) when saving, where N is the total number of keys in all databases when saving data, otherwise O(1)","1.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,SHUTDOWN_History,1,SHUTDOWN_Tips,0,shutdownCommand,-1,CMD_ADMIN|CMD_NOSCRIPT|CMD_LOADING|CMD_STALE|CMD_NO_MULTI|CMD_SENTINEL|CMD_ALLOW_BUSY,0,SHUTDOWN_Keyspecs,0,NULL,4),.args=SHUTDOWN_Args},
{MAKE_CMD("slaveof","Sets a Redis server as a replica of another, or promotes it to being a master.","O(1)","1.0.0",CMD_DOC_DEPRECATED,"`REPLICAOF`","5.0.0","server",COMMAND_GROUP_SERVER,SLAVEOF_History,0,SLAVEOF_Tips,0,replicaofCommand,3,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_NOSCRIPT|CMD_STALE,0,SLAVEOF_Keyspecs,0,NULL,2),.args=SLAVEOF_Args},
{MAKE_CMD("slaveof","Sets a Redis server as a replica of another, or promotes it to being a master.","O(1)","1.0.0",CMD_DOC_DEPRECATED,"`REPLICAOF`","5.0.0","server",COMMAND_GROUP_SERVER,SLAVEOF_History,0,SLAVEOF_Tips,0,replicaofCommand,3,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_NOSCRIPT|CMD_STALE,0,SLAVEOF_Keyspecs,0,NULL,1),.args=SLAVEOF_Args},
{MAKE_CMD("slowlog","A container for slow log commands.","Depends on subcommand.","2.2.12",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,SLOWLOG_History,0,SLOWLOG_Tips,0,NULL,-2,0,0,SLOWLOG_Keyspecs,0,NULL,0),.subcommands=SLOWLOG_Subcommands},
{MAKE_CMD("swapdb","Swaps two Redis databases.","O(N) where N is the count of clients watching or blocking on keys from both databases.","4.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,SWAPDB_History,0,SWAPDB_Tips,0,swapdbCommand,3,CMD_WRITE|CMD_FAST,ACL_CATEGORY_KEYSPACE|ACL_CATEGORY_DANGEROUS,SWAPDB_Keyspecs,0,NULL,2),.args=SWAPDB_Args},
{MAKE_CMD("sync","An internal command used in replication.",NULL,"1.0.0",CMD_DOC_NONE,NULL,NULL,"server",COMMAND_GROUP_SERVER,SYNC_History,0,SYNC_Tips,0,syncCommand,1,CMD_NO_ASYNC_LOADING|CMD_ADMIN|CMD_NO_MULTI|CMD_NOSCRIPT,0,SYNC_Keyspecs,0,NULL,0)},
+4
View File
@@ -14,6 +14,10 @@
"STALE",
"SENTINEL"
],
"command_tips": [
"REQUEST_POLICY:ALL_NODES",
"RESPONSE_POLICY:ALL_SUCCEEDED"
],
"reply_schema": {
"type": "integer",
"description": "The number of users that were deleted"
+4
View File
@@ -14,6 +14,10 @@
"STALE",
"SENTINEL"
],
"command_tips": [
"REQUEST_POLICY:ALL_NODES",
"RESPONSE_POLICY:ALL_SUCCEEDED"
],
"reply_schema": {
"const": "OK"
}
+4
View File
@@ -24,6 +24,10 @@
"STALE",
"SENTINEL"
],
"command_tips": [
"REQUEST_POLICY:ALL_NODES",
"RESPONSE_POLICY:ALL_SUCCEEDED"
],
"reply_schema": {
"const": "OK"
},
+4
View File
@@ -13,6 +13,10 @@
"STALE",
"SENTINEL"
],
"command_tips": [
"REQUEST_POLICY:ALL_NODES",
"RESPONSE_POLICY:ALL_SUCCEEDED"
],
"acl_categories": [
"CONNECTION"
],
+4
View File
@@ -13,6 +13,10 @@
"STALE",
"SENTINEL"
],
"command_tips": [
"REQUEST_POLICY:ALL_NODES",
"RESPONSE_POLICY:ALL_SUCCEEDED"
],
"acl_categories": [
"CONNECTION"
],
+1 -1
View File
@@ -26,7 +26,7 @@
"description": "an even number element array specifying the start and end slot numbers for slot ranges owned by this shard",
"type": "array",
"items": {
"type": "string"
"type": "integer"
}
},
"nodes": {
+4
View File
@@ -13,6 +13,10 @@
"LOADING",
"STALE"
],
"command_tips": [
"REQUEST_POLICY:ALL_NODES",
"RESPONSE_POLICY:ALL_SUCCEEDED"
],
"reply_schema": {
"const": "OK"
}
+4
View File
@@ -13,6 +13,10 @@
"LOADING",
"STALE"
],
"command_tips": [
"REQUEST_POLICY:ALL_NODES",
"RESPONSE_POLICY:ALL_SUCCEEDED"
],
"reply_schema": {
"const": "OK"
}
+1 -1
View File
@@ -15,7 +15,7 @@
],
"command_tips": [
"REQUEST_POLICY:ALL_NODES",
"RESPONSE_POLICY:ALL_SUCCEEDED"
"RESPONSE_POLICY:AGG_SUM"
],
"reply_schema": {
"type": "integer",
+34 -6
View File
@@ -14,12 +14,40 @@
],
"arguments": [
{
"name": "host",
"type": "string"
},
{
"name": "port",
"type": "integer"
"name": "args",
"type": "oneof",
"arguments": [
{
"name": "host-port",
"type": "block",
"arguments": [
{
"name": "host",
"type": "string"
},
{
"name": "port",
"type": "integer"
}
]
},
{
"name": "no-one",
"type": "block",
"arguments": [
{
"name": "no",
"type": "pure-token",
"token": "NO"
},
{
"name": "one",
"type": "pure-token",
"token": "ONE"
}
]
}
]
}
],
"reply_schema": {
+34 -6
View File
@@ -19,12 +19,40 @@
],
"arguments": [
{
"name": "host",
"type": "string"
},
{
"name": "port",
"type": "integer"
"name": "args",
"type": "oneof",
"arguments": [
{
"name": "host-port",
"type": "block",
"arguments": [
{
"name": "host",
"type": "string"
},
{
"name": "port",
"type": "integer"
}
]
},
{
"name": "no-one",
"type": "block",
"arguments": [
{
"name": "no",
"type": "pure-token",
"token": "NO"
},
{
"name": "one",
"type": "pure-token",
"token": "ONE"
}
]
}
]
}
],
"reply_schema": {
+1 -1
View File
@@ -150,7 +150,7 @@
"type": "string"
},
{
"description": "GET option is specified, but no object was found ",
"description": "GET option is specified, but no object was found",
"type": "null"
}
]
+9 -1
View File
@@ -117,7 +117,15 @@
"description": "a list of sorted elements",
"type": "array",
"items": {
"type": "string"
"oneOf": [
{
"type": "string"
},
{
"description": "GET option is specified, but no object was found",
"type": "null"
}
]
}
}
}
+7 -3
View File
@@ -40,8 +40,12 @@
#include <fcntl.h>
#endif
#if defined(__APPLE__) && defined(__MAC_OS_X_VERSION_MAX_ALLOWED) && __MAC_OS_X_VERSION_MAX_ALLOWED >= 1060
#define MAC_OS_10_6_DETECTED
#endif
/* Define redis_fstat to fstat or fstat64() */
#if defined(__APPLE__) && !defined(MAC_OS_X_VERSION_10_6)
#if defined(__APPLE__) && !defined(MAC_OS_10_6_DETECTED)
#define redis_fstat fstat64
#define redis_stat stat64
#else
@@ -96,7 +100,7 @@
#define HAVE_ACCEPT4 1
#endif
#if (defined(__APPLE__) && defined(MAC_OS_X_VERSION_10_6)) || defined(__FreeBSD__) || defined(__OpenBSD__) || defined (__NetBSD__)
#if (defined(__APPLE__) && defined(MAC_OS_10_6_DETECTED)) || defined(__FreeBSD__) || defined(__OpenBSD__) || defined (__NetBSD__)
#define HAVE_KQUEUE 1
#endif
@@ -293,7 +297,7 @@ void setproctitle(const char *fmt, ...);
#include <kernel/OS.h>
#define redis_set_thread_title(name) rename_thread(find_thread(0), name)
#else
#if (defined __APPLE__ && defined(MAC_OS_X_VERSION_10_7))
#if (defined __APPLE__ && defined(__MAC_OS_X_VERSION_MAX_ALLOWED) && __MAC_OS_X_VERSION_MAX_ALLOWED >= 1070)
int pthread_setname_np(const char *name);
#include <pthread.h>
#define redis_set_thread_title(name) pthread_setname_np(name)
+8 -7
View File
@@ -333,7 +333,7 @@ robj *dbRandomKey(redisDb *db) {
key = dictGetKey(de);
keyobj = createStringObject(key,sdslen(key));
if (dictFind(db->expires,key)) {
if (allvolatile && server.masterhost && --maxtries == 0) {
if (allvolatile && (server.masterhost || isPausedActions(PAUSE_ACTION_EXPIRE)) && --maxtries == 0) {
/* If the DB is composed only of keys with an expire set,
* it could happen that all the keys are already logically
* expired in the slave, so the function cannot stop because
@@ -1871,7 +1871,7 @@ int64_t getAllKeySpecsFlags(struct redisCommand *cmd, int inv) {
* found in other valid keyspecs.
*/
int getKeysUsingKeySpecs(struct redisCommand *cmd, robj **argv, int argc, int search_flags, getKeysResult *result) {
int j, i, last, first, step;
long j, i, last, first, step;
keyReference *keys;
serverAssert(result->numkeys == 0); /* caller should initialize or reset it */
@@ -1931,20 +1931,20 @@ int getKeysUsingKeySpecs(struct redisCommand *cmd, robj **argv, int argc, int se
}
first += spec->fk.keynum.firstkey;
last = first + (int)numkeys-1;
last = first + (long)numkeys-1;
} else {
/* unknown spec */
goto invalid_spec;
}
int count = ((last - first)+1);
keys = getKeysPrepareResult(result, result->numkeys + count);
/* First or last is out of bounds, which indicates a syntax error */
if (last >= argc || last < first || first >= argc) {
goto invalid_spec;
}
int count = ((last - first)+1);
keys = getKeysPrepareResult(result, result->numkeys + count);
for (i = first; i <= last; i += step) {
if (i >= argc || i < first) {
/* Modules commands, and standard commands with a not fixed number
@@ -2294,7 +2294,8 @@ int sortROGetKeys(struct redisCommand *cmd, robj **argv, int argc, getKeysResult
keys = getKeysPrepareResult(result, 1);
keys[0].pos = 1; /* <sort-key> is always present. */
keys[0].flags = CMD_KEY_RO | CMD_KEY_ACCESS;
return 1;
result->numkeys = 1;
return result->numkeys;
}
/* Helper function to extract keys from the SORT command.
+3 -3
View File
@@ -1190,7 +1190,7 @@ static void* getAndSetMcontextEip(ucontext_t *uc, void *eip) {
} \
return old_val; \
} while(0)
#if defined(__APPLE__) && !defined(MAC_OS_X_VERSION_10_6)
#if defined(__APPLE__) && !defined(MAC_OS_10_6_DETECTED)
/* OSX < 10.6 */
#if defined(__x86_64__)
GET_SET_RETURN(uc->uc_mcontext->__ss.__rip, eip);
@@ -1199,7 +1199,7 @@ static void* getAndSetMcontextEip(ucontext_t *uc, void *eip) {
#else
GET_SET_RETURN(uc->uc_mcontext->__ss.__srr0, eip);
#endif
#elif defined(__APPLE__) && defined(MAC_OS_X_VERSION_10_6)
#elif defined(__APPLE__) && defined(MAC_OS_10_6_DETECTED)
/* OSX >= 10.6 */
#if defined(_STRUCT_X86_THREAD_STATE64) && !defined(__i386__)
GET_SET_RETURN(uc->uc_mcontext->__ss.__rip, eip);
@@ -1290,7 +1290,7 @@ void logRegisters(ucontext_t *uc) {
} while(0)
/* OSX */
#if defined(__APPLE__) && defined(MAC_OS_X_VERSION_10_6)
#if defined(__APPLE__) && defined(MAC_OS_10_6_DETECTED)
/* OSX AMD64 */
#if defined(_STRUCT_X86_THREAD_STATE64) && !defined(__i386__)
serverLog(LL_WARNING,
+7 -3
View File
@@ -662,8 +662,9 @@ void defragStream(redisDb *db, dictEntry *kde) {
void defragModule(redisDb *db, dictEntry *kde) {
robj *obj = dictGetVal(kde);
serverAssert(obj->type == OBJ_MODULE);
if (!moduleDefragValue(dictGetKey(kde), obj, db->id))
robj keyobj;
initStaticStringObject(keyobj, dictGetKey(kde));
if (!moduleDefragValue(&keyobj, obj, db->id))
defragLater(db, kde);
}
@@ -806,7 +807,9 @@ int defragLaterItem(dictEntry *de, unsigned long *cursor, long long endtime, int
} else if (ob->type == OBJ_STREAM) {
return scanLaterStreamListpacks(ob, cursor, endtime);
} else if (ob->type == OBJ_MODULE) {
return moduleLateDefrag(dictGetKey(de), ob, cursor, endtime, dbid);
robj keyobj;
initStaticStringObject(keyobj, dictGetKey(de));
return moduleLateDefrag(&keyobj, ob, cursor, endtime, dbid);
} else {
*cursor = 0; /* object type may have changed since we schedule it for later */
}
@@ -941,6 +944,7 @@ void activeDefragCycle(void) {
defrag_later_cursor = 0;
current_db = -1;
cursor = 0;
expires_cursor = 0;
db = NULL;
goto update_metrics;
}
+5 -10
View File
@@ -1414,30 +1414,25 @@ static int _dictExpandIfNeeded(dict *d)
* table (global setting) or we should avoid it but the ratio between
* elements/buckets is over the "safe" threshold, we resize doubling
* the number of buckets. */
if (!dictTypeExpandAllowed(d))
return DICT_OK;
if ((dict_can_resize == DICT_RESIZE_ENABLE &&
d->ht_used[0] >= DICTHT_SIZE(d->ht_size_exp[0])) ||
(dict_can_resize != DICT_RESIZE_FORBID &&
d->ht_used[0] / DICTHT_SIZE(d->ht_size_exp[0]) > dict_force_resize_ratio))
{
if (!dictTypeExpandAllowed(d))
return DICT_OK;
return dictExpand(d, d->ht_used[0] + 1);
}
return DICT_OK;
}
/* TODO: clz optimization */
/* Our hash table capability is a power of two */
static signed char _dictNextExp(unsigned long size)
{
unsigned char e = DICT_HT_INITIAL_EXP;
if (size <= DICT_HT_INITIAL_SIZE) return DICT_HT_INITIAL_EXP;
if (size >= LONG_MAX) return (8*sizeof(long)-1);
while(1) {
if (((unsigned long)1<<e) >= size)
return e;
e++;
}
return 8*sizeof(long) - __builtin_clzl(size-1);
}
/* Finds and returns the position within the dict where the provided key should
+1
View File
@@ -272,6 +272,7 @@ void scriptingRelease(int async) {
else
dictRelease(lctx.lua_scripts);
lctx.lua_scripts_mem = 0;
lua_gc(lctx.lua, LUA_GCCOLLECT, 0);
lua_close(lctx.lua);
}
+2
View File
@@ -667,6 +667,7 @@ int performEvictions(void) {
*
* AOF and Output buffer memory will be freed eventually so
* we only care about memory used by the key space. */
enterExecutionUnit(1, 0);
delta = (long long) zmalloc_used_memory();
latencyStartMonitor(eviction_latency);
dbGenericDelete(db,keyobj,server.lazyfree_lazy_eviction,DB_FLAG_KEY_EVICTED);
@@ -679,6 +680,7 @@ int performEvictions(void) {
notifyKeyspaceEvent(NOTIFY_EVICTED, "evicted",
keyobj, db->id);
propagateDeletion(db,keyobj,server.lazyfree_lazy_eviction);
exitExecutionUnit();
postExecutionUnitOperations();
decrRefCount(keyobj);
keys_freed++;
+2
View File
@@ -54,10 +54,12 @@
int activeExpireCycleTryExpire(redisDb *db, dictEntry *de, long long now) {
long long t = dictGetSignedIntegerVal(de);
if (now > t) {
enterExecutionUnit(1, 0);
sds key = dictGetKey(de);
robj *keyobj = createStringObject(key,sdslen(key));
deleteExpiredKeyAndPropagate(db,keyobj);
decrRefCount(keyobj);
exitExecutionUnit();
return 1;
} else {
return 0;
+5
View File
@@ -2,6 +2,7 @@
#include "bio.h"
#include "atomicvar.h"
#include "functions.h"
#include "cluster.h"
static redisAtomic size_t lazyfree_objects = 0;
static redisAtomic size_t lazyfreed_objects = 0;
@@ -177,6 +178,10 @@ void emptyDbAsync(redisDb *db) {
dict *oldht1 = db->dict, *oldht2 = db->expires;
db->dict = dictCreate(&dbDictType);
db->expires = dictCreate(&dbExpiresDictType);
if (server.cluster_enabled) {
slotToKeyDestroy(db);
slotToKeyInit(db);
}
atomicIncr(lazyfree_objects,dictSize(oldht1));
bioCreateLazyFreeJob(lazyfreeFreeDatabase,2,oldht1,oldht2);
}
+98 -43
View File
@@ -282,6 +282,10 @@ typedef struct RedisModuleBlockedClient {
monotime background_timer; /* Timer tracking the start of background work */
uint64_t background_duration; /* Current command background time duration.
Used for measuring latency of blocking cmds */
int blocked_on_keys_explicit_unblock; /* Set to 1 only in the case of an explicit RM_Unblock on
* a client that is blocked on keys. In this case we will
* call the timeout call back from within
* moduleHandleBlockedClients which runs from the main thread */
} RedisModuleBlockedClient;
/* This is a list of Module Auth Contexts. Each time a Module registers a callback, a new ctx is
@@ -306,7 +310,6 @@ static size_t moduleTempClientMinCount = 0; /* Min client count in pool since
* allow thread safe contexts to execute commands at a safe moment. */
static pthread_mutex_t moduleGIL = PTHREAD_MUTEX_INITIALIZER;
/* Function pointer type for keyspace event notification subscriptions from modules. */
typedef int (*RedisModuleNotificationFunc) (RedisModuleCtx *ctx, int type, const char *event, RedisModuleString *key);
@@ -2294,7 +2297,10 @@ ustime_t RM_CachedMicroseconds(void) {
* Within the same command, you can call multiple times
* RM_BlockedClientMeasureTimeStart() and RM_BlockedClientMeasureTimeEnd()
* to accumulate independent time intervals to the background duration.
* This method always return REDISMODULE_OK. */
* This method always return REDISMODULE_OK.
*
* This function is not thread safe, If used in module thread and blocked callback (possibly main thread)
* simultaneously, it's recommended to protect them with lock owned by caller instead of GIL. */
int RM_BlockedClientMeasureTimeStart(RedisModuleBlockedClient *bc) {
elapsedStart(&(bc->background_timer));
return REDISMODULE_OK;
@@ -2304,7 +2310,10 @@ int RM_BlockedClientMeasureTimeStart(RedisModuleBlockedClient *bc) {
* to calculate the elapsed execution time.
* On success REDISMODULE_OK is returned.
* This method only returns REDISMODULE_ERR if no start time was
* previously defined ( meaning RM_BlockedClientMeasureTimeStart was not called ). */
* previously defined ( meaning RM_BlockedClientMeasureTimeStart was not called ).
*
* This function is not thread safe, If used in module thread and blocked callback (possibly main thread)
* simultaneously, it's recommended to protect them with lock owned by caller instead of GIL. */
int RM_BlockedClientMeasureTimeEnd(RedisModuleBlockedClient *bc) {
// If the counter is 0 then we haven't called RM_BlockedClientMeasureTimeStart
if (!bc->background_timer)
@@ -2363,7 +2372,33 @@ void RM_Yield(RedisModuleCtx *ctx, int flags, const char *busy_reply) {
server.busy_module_yield_flags |= BUSY_MODULE_YIELD_CLIENTS;
/* Let redis process events */
processEventsWhileBlocked();
if (!pthread_equal(server.main_thread_id, pthread_self())) {
/* If we are not in the main thread, we defer event loop processing to the main thread
* after the main thread enters acquiring GIL state in order to protect the event
* loop (ae.c) and avoid potential race conditions. */
int acquiring;
atomicGet(server.module_gil_acquring, acquiring);
if (!acquiring) {
/* If the main thread has not yet entered the acquiring GIL state,
* we attempt to wake it up and exit without waiting for it to
* acquire the GIL. This avoids blocking the caller, allowing them to
* continue with unfinished tasks before the next yield.
* We assume the caller keeps the GIL locked. */
if (write(server.module_pipe[1],"A",1) != 1) {
/* Ignore the error, this is best-effort. */
}
} else {
/* Release the GIL, yielding CPU to give the main thread an opportunity to start
* event processing, and then acquire the GIL again until the main thread releases it. */
moduleReleaseGIL();
sched_yield();
moduleAcquireGIL();
}
} else {
/* If we are in the main thread, we can safely process events. */
processEventsWhileBlocked();
}
server.busy_module_yield_reply = prev_busy_module_yield_reply;
/* Possibly restore the previous flags in case of two nested contexts
@@ -2647,7 +2682,10 @@ RedisModuleString *RM_CreateStringFromStreamID(RedisModuleCtx *ctx, const RedisM
* pass ctx as NULL when releasing the string (but passing a context will not
* create any issue). Strings created with a context should be freed also passing
* the context, so if you want to free a string out of context later, make sure
* to create it using a NULL context. */
* to create it using a NULL context.
*
* This API is not thread safe, access to these retained strings (if they originated
* from a client command arguments) must be done with GIL locked. */
void RM_FreeString(RedisModuleCtx *ctx, RedisModuleString *str) {
decrRefCount(str);
if (ctx != NULL) autoMemoryFreed(ctx,REDISMODULE_AM_STRING,str);
@@ -2684,7 +2722,10 @@ void RM_FreeString(RedisModuleCtx *ctx, RedisModuleString *str) {
*
* Threaded modules that reference retained strings from other threads *must*
* explicitly trim the allocation as soon as the string is retained. Not doing
* so may result with automatic trimming which is not thread safe. */
* so may result with automatic trimming which is not thread safe.
*
* This API is not thread safe, access to these retained strings (if they originated
* from a client command arguments) must be done with GIL locked. */
void RM_RetainString(RedisModuleCtx *ctx, RedisModuleString *str) {
if (ctx == NULL || !autoMemoryFreed(ctx,REDISMODULE_AM_STRING,str)) {
/* Increment the string reference counting only if we can't
@@ -2726,7 +2767,10 @@ void RM_RetainString(RedisModuleCtx *ctx, RedisModuleString *str) {
*
* Threaded modules that reference held strings from other threads *must*
* explicitly trim the allocation as soon as the string is held. Not doing
* so may result with automatic trimming which is not thread safe. */
* so may result with automatic trimming which is not thread safe.
*
* This API is not thread safe, access to these retained strings (if they originated
* from a client command arguments) must be done with GIL locked. */
RedisModuleString* RM_HoldString(RedisModuleCtx *ctx, RedisModuleString *str) {
if (str->refcount == OBJ_STATIC_REFCOUNT) {
return RM_CreateStringFromString(ctx, str);
@@ -7680,7 +7724,7 @@ RedisModuleBlockedClient *moduleBlockClient(RedisModuleCtx *ctx, RedisModuleCmdF
int islua = scriptIsRunning();
int ismulti = server.in_exec;
c->bstate.module_blocked_handle = zmalloc(sizeof(RedisModuleBlockedClient));
c->bstate.module_blocked_handle = zcalloc(sizeof(RedisModuleBlockedClient));
RedisModuleBlockedClient *bc = c->bstate.module_blocked_handle;
ctx->module->blocked_clients++;
@@ -7706,15 +7750,15 @@ RedisModuleBlockedClient *moduleBlockClient(RedisModuleCtx *ctx, RedisModuleCmdF
bc->background_timer = 0;
bc->background_duration = 0;
c->bstate.timeout = 0;
mstime_t timeout = 0;
if (timeout_ms) {
mstime_t now = mstime();
if (timeout_ms > LLONG_MAX - now) {
if (timeout_ms > LLONG_MAX - now) {
c->bstate.module_blocked_handle = NULL;
addReplyError(c, "timeout is out of range"); /* 'timeout_ms+now' would overflow */
return bc;
}
c->bstate.timeout = timeout_ms + now;
timeout = timeout_ms + now;
}
if (islua || ismulti) {
@@ -7730,8 +7774,9 @@ RedisModuleBlockedClient *moduleBlockClient(RedisModuleCtx *ctx, RedisModuleCmdF
addReplyError(c, "Clients undergoing module based authentication can only be blocked on auth");
} else {
if (keys) {
blockForKeys(c,BLOCKED_MODULE,keys,numkeys,c->bstate.timeout,flags&REDISMODULE_BLOCK_UNBLOCK_DELETED);
blockForKeys(c,BLOCKED_MODULE,keys,numkeys,timeout,flags&REDISMODULE_BLOCK_UNBLOCK_DELETED);
} else {
c->bstate.timeout = timeout;
blockClient(c,BLOCKED_MODULE);
}
}
@@ -8158,7 +8203,7 @@ int RM_UnblockClient(RedisModuleBlockedClient *bc, void *privdata) {
* argument, but better to be safe than sorry. */
if (bc->timeout_callback == NULL) return REDISMODULE_ERR;
if (bc->unblocked) return REDISMODULE_OK;
if (bc->client) moduleBlockedClientTimedOut(bc->client);
if (bc->client) bc->blocked_on_keys_explicit_unblock = 1;
}
moduleUnblockClientByHandle(bc,privdata);
return REDISMODULE_OK;
@@ -8236,6 +8281,10 @@ void moduleHandleBlockedClients(void) {
reply_us = elapsedUs(replyTimer);
moduleFreeContext(&ctx);
}
if (c && bc->blocked_on_keys_explicit_unblock) {
serverAssert(bc->blocked_on_keys);
moduleBlockedClientTimedOut(c);
}
/* Hold onto the blocked client if module auth is in progress. The reply callback is invoked
* when the client is reprocessed. */
if (c && clientHasModuleAuthInProgress(c)) {
@@ -8256,6 +8305,9 @@ void moduleHandleBlockedClients(void) {
/* Update stats now that we've finished the blocking operation.
* This needs to be out of the reply callback above given that a
* module might not define any callback and still do blocking ops.
*
* If the module is blocked on keys updateStatsOnUnblock will be
* called from moduleUnblockClientOnKey
*/
if (c && !clientHasModuleAuthInProgress(c) && !bc->blocked_on_keys) {
updateStatsOnUnblock(c, bc->background_duration, reply_us, server.stat_total_error_replies != prev_error_replies);
@@ -8276,7 +8328,7 @@ void moduleHandleBlockedClients(void) {
* if there are pending replies here. This is needed since
* during a non blocking command the client may receive output. */
if (!clientHasModuleAuthInProgress(c) && clientHasPendingReplies(c) &&
!(c->flags & CLIENT_PENDING_WRITE))
!(c->flags & CLIENT_PENDING_WRITE) && c->conn)
{
c->flags |= CLIENT_PENDING_WRITE;
listLinkNodeHead(server.clients_pending_write, &c->clients_pending_write_node);
@@ -8311,23 +8363,25 @@ int moduleBlockedClientMayTimeout(client *c) {
/* Called when our client timed out. After this function unblockClient()
* is called, and it will invalidate the blocked client. So this function
* does not need to do any cleanup. Eventually the module will call the
* API to unblock the client and the memory will be released. */
* API to unblock the client and the memory will be released.
*
* This function should only be called from the main thread, we must handle the unblocking
* of the client synchronously. This ensures that we can reply to the client before
* resetClient() is called. */
void moduleBlockedClientTimedOut(client *c) {
RedisModuleBlockedClient *bc = c->bstate.module_blocked_handle;
/* Protect against re-processing: don't serve clients that are already
* in the unblocking list for any reason (including RM_UnblockClient()
* explicit call). See #6798. */
if (bc->unblocked) return;
RedisModuleCtx ctx;
moduleCreateContext(&ctx, bc->module, REDISMODULE_CTX_BLOCKED_TIMEOUT);
ctx.client = bc->client;
ctx.blocked_client = bc;
ctx.blocked_privdata = bc->privdata;
long long prev_error_replies = server.stat_total_error_replies;
bc->timeout_callback(&ctx,(void**)c->argv,c->argc);
moduleFreeContext(&ctx);
updateStatsOnUnblock(c, bc->background_duration, 0, server.stat_total_error_replies != prev_error_replies);
/* For timeout events, we do not want to call the disconnect callback,
@@ -11848,6 +11902,7 @@ void moduleInitModulesSystem(void) {
moduleUnblockedClients = listCreate();
server.loadmodule_queue = listCreate();
server.module_configs_queue = dictCreate(&sdsKeyValueHashDictType);
server.module_gil_acquring = 0;
modules = dictCreate(&modulesDictType);
moduleAuthCallbacks = listCreate();
@@ -12115,6 +12170,19 @@ int parseLoadexArguments(RedisModuleString ***module_argv, int *module_argc) {
return REDISMODULE_OK;
}
/* Unregister module-related things, called when moduleLoad fails or moduleUnload. */
void moduleUnregisterCleanup(RedisModule *module) {
moduleFreeAuthenticatedClients(module);
moduleUnregisterCommands(module);
moduleUnsubscribeNotifications(module);
moduleUnregisterSharedAPI(module);
moduleUnregisterUsedAPI(module);
moduleUnregisterFilters(module);
moduleUnsubscribeAllServerEvents(module);
moduleRemoveConfigs(module);
moduleUnregisterAuthCBs(module);
}
/* Load a module and initialize it. On success C_OK is returned, otherwise
* C_ERR is returned. */
int moduleLoad(const char *path, void **module_argv, int module_argc, int is_loadex) {
@@ -12149,11 +12217,7 @@ int moduleLoad(const char *path, void **module_argv, int module_argc, int is_loa
serverLog(LL_WARNING,
"Module %s initialization failed. Module not loaded",path);
if (ctx.module) {
moduleUnregisterCommands(ctx.module);
moduleUnregisterSharedAPI(ctx.module);
moduleUnregisterUsedAPI(ctx.module);
moduleRemoveConfigs(ctx.module);
moduleUnregisterAuthCBs(ctx.module);
moduleUnregisterCleanup(ctx.module);
moduleFreeModuleStructure(ctx.module);
}
moduleFreeContext(&ctx);
@@ -12194,9 +12258,7 @@ int moduleLoad(const char *path, void **module_argv, int module_argc, int is_loa
}
if (post_load_err) {
/* Unregister module auth callbacks (if any exist) that this Module registered onload. */
moduleUnregisterAuthCBs(ctx.module);
moduleUnload(ctx.module->name, NULL);
serverAssert(moduleUnload(ctx.module->name, NULL, 1) == C_OK);
moduleFreeContext(&ctx);
return C_ERR;
}
@@ -12212,14 +12274,17 @@ int moduleLoad(const char *path, void **module_argv, int module_argc, int is_loa
/* Unload the module registered with the specified name. On success
* C_OK is returned, otherwise C_ERR is returned and errmsg is set
* with an appropriate message. */
int moduleUnload(sds name, const char **errmsg) {
* with an appropriate message.
* Only forcefully unload this module, passing forced_unload != 0,
* if it is certain that it has not yet been in use (e.g., immediate
* unload on failed load). */
int moduleUnload(sds name, const char **errmsg, int forced_unload) {
struct RedisModule *module = dictFetchValue(modules,name);
if (module == NULL) {
*errmsg = "no such module with that name";
return C_ERR;
} else if (listLength(module->types)) {
} else if (listLength(module->types) && !forced_unload) {
*errmsg = "the module exports one or more module-side data "
"types, can't unload";
return C_ERR;
@@ -12253,17 +12318,7 @@ int moduleUnload(sds name, const char **errmsg) {
}
}
moduleFreeAuthenticatedClients(module);
moduleUnregisterCommands(module);
moduleUnregisterSharedAPI(module);
moduleUnregisterUsedAPI(module);
moduleUnregisterFilters(module);
moduleUnregisterAuthCBs(module);
moduleRemoveConfigs(module);
/* Remove any notification subscribers this module might have */
moduleUnsubscribeNotifications(module);
moduleUnsubscribeAllServerEvents(module);
moduleUnregisterCleanup(module);
/* Unload the dynamic library. */
if (dlclose(module->handle) == -1) {
@@ -13022,7 +13077,7 @@ NULL
} else if (!strcasecmp(subcmd,"unload") && c->argc == 3) {
const char *errmsg = NULL;
if (moduleUnload(c->argv[2]->ptr, &errmsg) == C_OK)
if (moduleUnload(c->argv[2]->ptr, &errmsg, 0) == C_OK)
addReply(c,shared.ok);
else {
if (errmsg == NULL) errmsg = "operation not possible.";
+20 -9
View File
@@ -413,8 +413,9 @@ void _addReplyToBufferOrList(client *c, const char *s, size_t len) {
* to a channel which we are subscribed to, then we wanna postpone that message to be added
* after the command's reply (specifically important during multi-exec). the exception is
* the SUBSCRIBE command family, which (currently) have a push message instead of a proper reply.
* The check for executing_client also avoids affecting push messages that are part of eviction. */
if (c == server.current_client && (c->flags & CLIENT_PUSHING) &&
* The check for executing_client also avoids affecting push messages that are part of eviction.
* Check CLIENT_PUSHING first to avoid race conditions, as it's absent in module's fake client. */
if ((c->flags & CLIENT_PUSHING) && c == server.current_client &&
server.executing_client && !cmdHasPushAsReply(server.executing_client->cmd))
{
_addReplyProtoToList(c,server.pending_push_messages,s,len);
@@ -1433,7 +1434,7 @@ void unlinkClient(client *c) {
listNode *ln;
/* If this is marked as current client unset it. */
if (server.current_client == c) server.current_client = NULL;
if (c->conn && server.current_client == c) server.current_client = NULL;
/* Certain operations must be done only if the client has an active connection.
* If the client was already unlinked or if it's a "fake client" the
@@ -1477,7 +1478,7 @@ void unlinkClient(client *c) {
}
/* Remove from the list of pending reads if needed. */
serverAssert(io_threads_op == IO_THREADS_OP_IDLE);
serverAssert(!c->conn || io_threads_op == IO_THREADS_OP_IDLE);
if (c->pending_read_list_node != NULL) {
listDelNode(server.clients_pending_read,c->pending_read_list_node);
c->pending_read_list_node = NULL;
@@ -1630,6 +1631,12 @@ void freeClient(client *c) {
reqresReset(c, 1);
#endif
/* Remove the contribution that this client gave to our
* incrementally computed memory usage. */
if (c->conn)
server.stat_clients_type_memory[c->last_memory_type] -=
c->last_memory_usage;
/* Unlink the client: this will close the socket, remove the I/O
* handlers, and remove references of the client from different
* places where active clients may be referenced. */
@@ -1678,10 +1685,6 @@ void freeClient(client *c) {
* we lost the connection with the master. */
if (c->flags & CLIENT_MASTER) replicationHandleMasterDisconnection();
/* Remove the contribution that this client gave to our
* incrementally computed memory usage. */
server.stat_clients_type_memory[c->last_memory_type] -=
c->last_memory_usage;
/* Remove client from memory usage buckets */
if (c->mem_usage_bucket) {
c->mem_usage_bucket->mem_usage_sum -= c->last_memory_usage;
@@ -1712,6 +1715,9 @@ void freeClientAsync(client *c) {
* idle. */
if (c->flags & CLIENT_CLOSE_ASAP || c->flags & CLIENT_SCRIPT) return;
c->flags |= CLIENT_CLOSE_ASAP;
/* Replicas that was marked as CLIENT_CLOSE_ASAP should not keep the
* replication backlog from been trimmed. */
if (c->flags & CLIENT_SLAVE) freeReplicaReferencedReplBuffer(c);
if (server.io_threads_num == 1) {
/* no need to bother with locking if there's just one thread (the main thread) */
listAddNodeTail(server.clients_to_close,c);
@@ -2467,7 +2473,7 @@ int processCommandAndResetClient(client *c) {
commandProcessed(c);
/* Update the client's memory to include output buffer growth following the
* processed command. */
updateClientMemUsageAndBucket(c);
if (c->conn) updateClientMemUsageAndBucket(c);
}
if (server.current_client == NULL) deadclient = 1;
@@ -3852,6 +3858,11 @@ int checkClientOutputBufferLimits(client *c) {
int soft = 0, hard = 0, class;
unsigned long used_mem = getClientOutputBufferMemoryUsage(c);
/* For unauthenticated clients the output buffer is limited to prevent
* them from abusing it by not reading the replies */
if (used_mem > 1024 && authRequired(c))
return 1;
class = getClientType(c);
/* For the purpose of output buffer limiting, masters are handled
* like normal clients. */
+64 -18
View File
@@ -104,6 +104,9 @@ quicklistBookmark *_quicklistBookmarkFindByName(quicklist *ql, const char *name)
quicklistBookmark *_quicklistBookmarkFindByNode(quicklist *ql, quicklistNode *node);
void _quicklistBookmarkDelete(quicklist *ql, quicklistBookmark *bm);
quicklistNode *_quicklistSplitNode(quicklistNode *node, int offset, int after);
quicklistNode *_quicklistMergeNodes(quicklist *quicklist, quicklistNode *center);
/* Simple way to give quicklistEntry structs default values with one call. */
#define initEntry(e) \
do { \
@@ -378,6 +381,15 @@ REDIS_STATIC void __quicklistCompress(const quicklist *quicklist,
quicklistCompressNode(reverse);
}
/* This macro is used to compress a node.
*
* If the 'recompress' flag of the node is true, we compress it directly without
* checking whether it is within the range of compress depth.
* However, it's important to ensure that the 'recompress' flag of head and tail
* is always false, as we always assume that head and tail are not compressed.
*
* If the 'recompress' flag of the node is false, we check whether the node is
* within the range of compress depth before compressing it. */
#define quicklistCompress(_ql, _node) \
do { \
if ((_node)->recompress) \
@@ -529,19 +541,25 @@ REDIS_STATIC int _quicklistNodeAllowMerge(const quicklistNode *a,
(node)->sz = lpBytes((node)->entry); \
} while (0)
static quicklistNode* __quicklistCreatePlainNode(void *value, size_t sz) {
static quicklistNode* __quicklistCreateNode(int container, void *value, size_t sz) {
quicklistNode *new_node = quicklistCreateNode();
new_node->entry = zmalloc(sz);
new_node->container = QUICKLIST_NODE_CONTAINER_PLAIN;
memcpy(new_node->entry, value, sz);
new_node->container = container;
if (container == QUICKLIST_NODE_CONTAINER_PLAIN) {
new_node->entry = zmalloc(sz);
memcpy(new_node->entry, value, sz);
} else {
new_node->entry = lpPrepend(lpNew(0), value, sz);
}
new_node->sz = sz;
new_node->count++;
return new_node;
}
static void __quicklistInsertPlainNode(quicklist *quicklist, quicklistNode *old_node,
void *value, size_t sz, int after) {
__quicklistInsertNode(quicklist, old_node, __quicklistCreatePlainNode(value, sz), after);
void *value, size_t sz, int after)
{
quicklistNode *new_node = __quicklistCreateNode(QUICKLIST_NODE_CONTAINER_PLAIN, value, sz);
__quicklistInsertNode(quicklist, old_node, new_node, after);
quicklist->count++;
}
@@ -741,9 +759,13 @@ void quicklistReplaceEntry(quicklistIter *iter, quicklistEntry *entry,
void *data, size_t sz)
{
quicklist* quicklist = iter->quicklist;
quicklistNode *node = entry->node;
unsigned char *newentry;
if (likely(!QL_NODE_IS_PLAIN(entry->node) && !isLargeElement(sz))) {
entry->node->entry = lpReplace(entry->node->entry, &entry->zi, data, sz);
if (likely(!QL_NODE_IS_PLAIN(entry->node) && !isLargeElement(sz) &&
(newentry = lpReplace(entry->node->entry, &entry->zi, data, sz)) != NULL))
{
entry->node->entry = newentry;
quicklistNodeUpdateSz(entry->node);
/* quicklistNext() and quicklistGetIteratorEntryAtIdx() provide an uncompressed node */
quicklistCompress(quicklist, entry->node);
@@ -758,17 +780,37 @@ void quicklistReplaceEntry(quicklistIter *iter, quicklistEntry *entry,
quicklistInsertAfter(iter, entry, data, sz);
__quicklistDelNode(quicklist, entry->node);
}
} else {
entry->node->dont_compress = 1; /* Prevent compression in quicklistInsertAfter() */
quicklistInsertAfter(iter, entry, data, sz);
} else { /* The node is full or data is a large element */
quicklistNode *split_node = NULL, *new_node;
node->dont_compress = 1; /* Prevent compression in __quicklistInsertNode() */
/* If the entry is not at the tail, split the node at the entry's offset. */
if (entry->offset != node->count - 1 && entry->offset != -1)
split_node = _quicklistSplitNode(node, entry->offset, 1);
/* Create a new node and insert it after the original node.
* If the original node was split, insert the split node after the new node. */
new_node = __quicklistCreateNode(isLargeElement(sz) ?
QUICKLIST_NODE_CONTAINER_PLAIN : QUICKLIST_NODE_CONTAINER_PACKED, data, sz);
__quicklistInsertNode(quicklist, node, new_node, 1);
if (split_node) __quicklistInsertNode(quicklist, new_node, split_node, 1);
quicklist->count++;
/* Delete the replaced element. */
if (entry->node->count == 1) {
__quicklistDelNode(quicklist, entry->node);
} else {
unsigned char *p = lpSeek(entry->node->entry, -1);
quicklistDelIndex(quicklist, entry->node, &p);
entry->node->dont_compress = 0; /* Re-enable compression */
quicklistCompress(quicklist, entry->node);
quicklistCompress(quicklist, entry->node->next);
new_node = _quicklistMergeNodes(quicklist, new_node);
/* We can't know if the current node and its sibling nodes are correctly compressed,
* and we don't know if they are within the range of compress depth, so we need to
* use quicklistCompress() for compression, which checks if node is within compress
* depth before compressing. */
quicklistCompress(quicklist, new_node);
quicklistCompress(quicklist, new_node->prev);
if (new_node->next) quicklistCompress(quicklist, new_node->next);
}
}
@@ -826,6 +868,8 @@ REDIS_STATIC quicklistNode *_quicklistListpackMerge(quicklist *quicklist,
}
keep->count = lpLength(keep->entry);
quicklistNodeUpdateSz(keep);
keep->recompress = 0; /* Prevent 'keep' from being recompressed if
* it becomes head or tail after merging. */
nokeep->count = 0;
__quicklistDelNode(quicklist, nokeep);
@@ -844,9 +888,10 @@ REDIS_STATIC quicklistNode *_quicklistListpackMerge(quicklist *quicklist,
* - (center->next, center->next->next)
* - (center->prev, center)
* - (center, center->next)
*
* Returns the new 'center' after merging.
*/
REDIS_STATIC void _quicklistMergeNodes(quicklist *quicklist,
quicklistNode *center) {
REDIS_STATIC quicklistNode *_quicklistMergeNodes(quicklist *quicklist, quicklistNode *center) {
int fill = quicklist->fill;
quicklistNode *prev, *prev_prev, *next, *next_next, *target;
prev = prev_prev = next = next_next = target = NULL;
@@ -886,8 +931,9 @@ REDIS_STATIC void _quicklistMergeNodes(quicklist *quicklist,
/* Use result of center merge (or original) to merge with next node. */
if (_quicklistNodeAllowMerge(target, target->next, fill)) {
_quicklistListpackMerge(quicklist, target, target->next);
target = _quicklistListpackMerge(quicklist, target, target->next);
}
return target;
}
/* Split 'node' into two parts, parameterized by 'offset' and 'after'.
@@ -1002,7 +1048,7 @@ REDIS_STATIC void _quicklistInsert(quicklistIter *iter, quicklistEntry *entry,
} else {
quicklistDecompressNodeForUse(node);
new_node = _quicklistSplitNode(node, entry->offset, after);
quicklistNode *entry_node = __quicklistCreatePlainNode(value, sz);
quicklistNode *entry_node = __quicklistCreateNode(QUICKLIST_NODE_CONTAINER_PLAIN, value, sz);
__quicklistInsertNode(quicklist, node, entry_node, after);
__quicklistInsertNode(quicklist, entry_node, new_node, after);
quicklist->count++;
@@ -3224,7 +3270,7 @@ int quicklistTest(int argc, char *argv[], int flags) {
memcpy(s, "helloworld", 10);
memcpy(s + sz - 10, "1234567890", 10);
quicklistNode *node = __quicklistCreatePlainNode(s, sz);
quicklistNode *node = __quicklistCreateNode(QUICKLIST_NODE_CONTAINER_PLAIN, s, sz);
/* Just to avoid triggering the assertion in __quicklistCompressNode(),
* it disables the passing of quicklist head or tail node. */
+1 -4
View File
@@ -81,9 +81,6 @@
#define RDB_TYPE_MODULE_PRE_GA 6 /* Used in 4.0 release candidates */
#define RDB_TYPE_MODULE_2 7 /* Module value with annotations for parsing without
the generating module being loaded. */
/* NOTE: WHEN ADDING NEW RDB TYPE, UPDATE rdbIsObjectType() BELOW */
/* Object types for encoded objects. */
#define RDB_TYPE_HASH_ZIPMAP 9
#define RDB_TYPE_LIST_ZIPLIST 10
#define RDB_TYPE_SET_INTSET 11
@@ -97,7 +94,7 @@
#define RDB_TYPE_STREAM_LISTPACKS_2 19
#define RDB_TYPE_SET_LISTPACK 20
#define RDB_TYPE_STREAM_LISTPACKS_3 21
/* NOTE: WHEN ADDING NEW RDB TYPE, UPDATE rdbIsObjectType() BELOW */
/* NOTE: WHEN ADDING NEW RDB TYPE, UPDATE rdbIsObjectType(), and rdb_type_string[] */
/* Test if a type is an object type. */
#define rdbIsObjectType(t) (((t) >= 0 && (t) <= 7) || ((t) >= 9 && (t) <= 21))
+9 -1
View File
@@ -233,6 +233,7 @@ int checkSingleAof(char *aof_filename, char *aof_filepath, int last_file, int fi
struct redis_stat sb;
if (redis_fstat(fileno(fp),&sb) == -1) {
printf("Cannot stat file: %s, aborting...\n", aof_filename);
fclose(fp);
exit(1);
}
@@ -343,6 +344,7 @@ int fileIsRDB(char *filepath) {
struct redis_stat sb;
if (redis_fstat(fileno(fp), &sb) == -1) {
printf("Cannot stat file: %s\n", filepath);
fclose(fp);
exit(1);
}
@@ -379,6 +381,7 @@ int fileIsManifest(char *filepath) {
struct redis_stat sb;
if (redis_fstat(fileno(fp), &sb) == -1) {
printf("Cannot stat file: %s\n", filepath);
fclose(fp);
exit(1);
}
@@ -395,15 +398,20 @@ int fileIsManifest(char *filepath) {
break;
} else {
printf("Cannot read file: %s\n", filepath);
fclose(fp);
exit(1);
}
}
/* Skip comments lines */
/* We will skip comments lines.
* At present, the manifest format is fixed, see aofInfoFormat.
* We will break directly as long as it encounters other items. */
if (buf[0] == '#') {
continue;
} else if (!memcmp(buf, "file", strlen("file"))) {
is_manifest = 1;
} else {
break;
}
}
+2
View File
@@ -98,7 +98,9 @@ char *rdb_type_string[] = {
"hash-listpack",
"zset-listpack",
"quicklist-v2",
"stream-v2",
"set-listpack",
"stream-v3",
};
/* Show a few stats collected into 'rdbstate' */
+3 -1
View File
@@ -1657,6 +1657,7 @@ static int cliConnect(int flags) {
redisFree(context);
config.dbnum = 0;
config.in_multi = 0;
config.pubsub_mode = 0;
cliRefreshPrompt();
}
@@ -8838,7 +8839,8 @@ static redisReply *sendScan(unsigned long long *it) {
reply = redisCommand(context, "SCAN %llu MATCH %b COUNT %d",
*it, config.pattern, sdslen(config.pattern), config.count);
else
reply = redisCommand(context,"SCAN %llu",*it);
reply = redisCommand(context, "SCAN %llu COUNT %d",
*it, config.count);
/* Handle any error conditions */
if(reply == NULL) {
+4
View File
@@ -210,6 +210,9 @@ int canFeedReplicaReplBuffer(client *replica) {
/* Don't feed replicas that are still waiting for BGSAVE to start. */
if (replica->replstate == SLAVE_STATE_WAIT_BGSAVE_START) return 0;
/* Don't feed replicas that are going to be closed ASAP. */
if (replica->flags & CLIENT_CLOSE_ASAP) return 0;
return 1;
}
@@ -2250,6 +2253,7 @@ void readSyncBulkPayload(connection *conn) {
}
zfree(server.repl_transfer_tmpfile);
close(server.repl_transfer_fd);
server.repl_transfer_fd = -1;
server.repl_transfer_tmpfile = NULL;
}
+11 -2
View File
@@ -818,8 +818,17 @@ static robj **luaArgsToRedisArgv(lua_State *lua, int *argc, int *argv_len) {
/* We can't use lua_tolstring() for number -> string conversion
* since Lua uses a format specifier that loses precision. */
lua_Number num = lua_tonumber(lua,j+1);
obj_len = fpconv_dtoa((double)num, dbuf);
dbuf[obj_len] = '\0';
/* Integer printing function is much faster, check if we can safely use it.
* Since lua_Number is not explicitly an integer or a double, we need to make an effort
* to convert it as an integer when that's possible, since the string could later be used
* in a context that doesn't support scientific notation (e.g. 1e9 instead of 100000000). */
long long lvalue;
if (double2ll((double)num, &lvalue))
obj_len = ll2string(dbuf, sizeof(dbuf), lvalue);
else {
obj_len = fpconv_dtoa((double)num, dbuf);
dbuf[obj_len] = '\0';
}
obj_s = dbuf;
} else {
obj_s = (char*)lua_tolstring(lua,j+1,&obj_len);
+15 -13
View File
@@ -349,20 +349,22 @@ sds sdsResize(sds s, size_t size, int would_regrow) {
* type. */
int use_realloc = (oldtype==type || (type < oldtype && type > SDS_TYPE_8));
size_t newlen = use_realloc ? oldhdrlen+size+1 : hdrlen+size+1;
int alloc_already_optimal = 0;
#if defined(USE_JEMALLOC)
/* je_nallocx returns the expected allocation size for the newlen.
* We aim to avoid calling realloc() when using Jemalloc if there is no
* change in the allocation size, as it incurs a cost even if the
* allocation size stays the same. */
alloc_already_optimal = (je_nallocx(newlen, 0) == zmalloc_size(sh));
#endif
if (use_realloc && !alloc_already_optimal) {
newsh = s_realloc(sh, newlen);
if (newsh == NULL) return NULL;
s = (char*)newsh+oldhdrlen;
} else if (!alloc_already_optimal) {
if (use_realloc) {
int alloc_already_optimal = 0;
#if defined(USE_JEMALLOC)
/* je_nallocx returns the expected allocation size for the newlen.
* We aim to avoid calling realloc() when using Jemalloc if there is no
* change in the allocation size, as it incurs a cost even if the
* allocation size stays the same. */
alloc_already_optimal = (je_nallocx(newlen, 0) == zmalloc_size(sh));
#endif
if (!alloc_already_optimal) {
newsh = s_realloc(sh, newlen);
if (newsh == NULL) return NULL;
s = (char*)newsh+oldhdrlen;
}
} else {
newsh = s_malloc(newlen);
if (newsh == NULL) return NULL;
memcpy((char*)newsh+hdrlen, s, len);
+15 -3
View File
@@ -872,6 +872,7 @@ static inline clientMemUsageBucket *getMemUsageBucket(size_t mem) {
* usage bucket.
*/
void updateClientMemoryUsage(client *c) {
serverAssert(c->conn);
size_t mem = getClientMemoryUsage(c, NULL);
int type = getClientType(c);
/* Now that we have the memory used by the client, remove the old
@@ -884,7 +885,7 @@ void updateClientMemoryUsage(client *c) {
}
int clientEvictionAllowed(client *c) {
if (server.maxmemory_clients == 0 || c->flags & CLIENT_NO_EVICT) {
if (server.maxmemory_clients == 0 || c->flags & CLIENT_NO_EVICT || !c->conn) {
return 0;
}
int type = getClientType(c);
@@ -924,7 +925,7 @@ void removeClientFromMemUsageBucket(client *c, int allow_eviction) {
* returns 1 if client eviction for this client is allowed, 0 otherwise.
*/
int updateClientMemUsageAndBucket(client *c) {
serverAssert(io_threads_op == IO_THREADS_OP_IDLE);
serverAssert(io_threads_op == IO_THREADS_OP_IDLE && c->conn);
int allow_eviction = clientEvictionAllowed(c);
removeClientFromMemUsageBucket(c, allow_eviction);
@@ -1793,7 +1794,9 @@ void afterSleep(struct aeEventLoop *eventLoop) {
mstime_t latency;
latencyStartMonitor(latency);
atomicSet(server.module_gil_acquring, 1);
moduleAcquireGIL();
atomicSet(server.module_gil_acquring, 0);
moduleFireServerEvent(REDISMODULE_EVENT_EVENTLOOP,
REDISMODULE_SUBEVENT_EVENTLOOP_AFTER_SLEEP,
NULL);
@@ -3512,12 +3515,20 @@ void call(client *c, int flags) {
* re-processing and unblock the client.*/
c->flags |= CLIENT_EXECUTING_COMMAND;
/* Setting the CLIENT_REPROCESSING_COMMAND flag so that during the actual
* processing of the command proc, the client is aware that it is being
* re-processed. */
if (reprocessing_command) c->flags |= CLIENT_REPROCESSING_COMMAND;
monotime monotonic_start = 0;
if (monotonicGetType() == MONOTONIC_CLOCK_HW)
monotonic_start = getMonotonicUs();
c->cmd->proc(c);
/* Clear the CLIENT_REPROCESSING_COMMAND flag after the proc is executed. */
if (reprocessing_command) c->flags &= ~CLIENT_REPROCESSING_COMMAND;
exitExecutionUnit();
/* In case client is blocked after trying to execute the command,
@@ -3575,7 +3586,7 @@ void call(client *c, int flags) {
/* Send the command to clients in MONITOR mode if applicable,
* since some administrative commands are considered too dangerous to be shown.
* Other exceptions is a client which is unblocked and retring to process the command
* Other exceptions is a client which is unblocked and retrying to process the command
* or we are currently in the process of loading AOF. */
if (update_command_stats && !reprocessing_command &&
!(c->cmd->flags & (CMD_SKIP_MONITOR|CMD_ADMIN))) {
@@ -3953,6 +3964,7 @@ int processCommand(client *c) {
flagTransaction(c);
}
clusterRedirectClient(c,n,c->slot,error_code);
c->duration = 0;
c->cmd->rejected_calls++;
return C_OK;
}
+3 -1
View File
@@ -400,6 +400,7 @@ extern int configOOMScoreAdjValuesDefaults[CONFIG_OOM_COUNT];
auth had been authenticated from the Module. */
#define CLIENT_MODULE_PREVENT_AOF_PROP (1ULL<<48) /* Module client do not want to propagate to AOF */
#define CLIENT_MODULE_PREVENT_REPL_PROP (1ULL<<49) /* Module client do not want to propagate to replica */
#define CLIENT_REPROCESSING_COMMAND (1ULL<<50) /* The client is re-processing the command. */
/* Client block type (btype field in client structure)
* if CLIENT_BLOCKED flag is set. */
@@ -1580,6 +1581,7 @@ struct redisServer {
int module_pipe[2]; /* Pipe used to awake the event loop by module threads. */
pid_t child_pid; /* PID of current child */
int child_type; /* Type of current child */
redisAtomic int module_gil_acquring; /* Indicates whether the GIL is being acquiring by the main thread. */
/* Networking */
int port; /* TCP listening port */
int tls_port; /* TLS listening port */
@@ -2467,7 +2469,7 @@ void moduleInitModulesSystem(void);
void moduleInitModulesSystemLast(void);
void modulesCron(void);
int moduleLoad(const char *path, void **argv, int argc, int is_loadex);
int moduleUnload(sds name, const char **errmsg);
int moduleUnload(sds name, const char **errmsg, int forced_unload);
void moduleLoadFromQueue(void);
int moduleGetCommandKeysViaAPI(struct redisCommand *cmd, robj **argv, int argc, getKeysResult *result);
int moduleGetCommandChannelsViaAPI(struct redisCommand *cmd, robj **argv, int argc, getKeysResult *result);
+19 -9
View File
@@ -1424,11 +1424,6 @@ int streamRangeHasTombstones(stream *s, streamID *start, streamID *end) {
return 0;
}
if (streamCompareID(&s->first_id,&s->max_deleted_entry_id) > 0) {
/* The latest tombstone is before the first entry. */
return 0;
}
if (start) {
start_id = *start;
} else {
@@ -1465,6 +1460,17 @@ void streamReplyWithCGLag(client *c, stream *s, streamCG *cg) {
/* The lag of a newly-initialized stream is 0. */
lag = 0;
valid = 1;
} else if (!s->length) { /* All entries deleted, now empty. */
lag = 0;
valid = 1;
} else if (streamCompareID(&cg->last_id,&s->first_id) < 0 &&
streamCompareID(&s->max_deleted_entry_id,&s->first_id) < 0)
{
/* When both the consumer group's last_id and the maximum tombstone are behind
* the stream's first entry, the consumer group's lag will always be equal to
* the number of remainin entries in the stream. */
lag = s->length;
valid = 1;
} else if (cg->entries_read != SCG_INVALID_ENTRIES_READ && !streamRangeHasTombstones(s,&cg->last_id,NULL)) {
/* No fragmentation ahead means that the group's logical reads counter
* is valid for performing the lag calculation. */
@@ -1521,6 +1527,10 @@ long long streamEstimateDistanceFromFirstEverEntry(stream *s, streamID *id) {
return s->entries_added;
}
/* There are fragmentations between the `id` and the stream's last-generated-id. */
if (!streamIDEqZero(id) && streamCompareID(id,&s->max_deleted_entry_id) < 0)
return SCG_INVALID_ENTRIES_READ;
int cmp_last = streamCompareID(id,&s->last_id);
if (cmp_last == 0) {
/* Return the exact counter of the last entry in the stream. */
@@ -1701,10 +1711,10 @@ size_t streamReplyWithRange(client *c, stream *s, streamID *start, streamID *end
while(streamIteratorGetID(&si,&id,&numfields)) {
/* Update the group last_id if needed. */
if (group && streamCompareID(&id,&group->last_id) > 0) {
if (group->entries_read != SCG_INVALID_ENTRIES_READ && !streamRangeHasTombstones(s,&id,NULL)) {
/* A valid counter and no future tombstones mean we can
* increment the read counter to keep tracking the group's
* progress. */
if (group->entries_read != SCG_INVALID_ENTRIES_READ && !streamRangeHasTombstones(s,&group->last_id,NULL)) {
/* A valid counter and no tombstones between the group's last-delivered-id
* and the stream's last-generated-id mean we can increment the read counter
* to keep tracking the group's progress. */
group->entries_read++;
} else if (s->entries_added) {
/* The group's counter may be invalid, so we try to obtain it. */
+3 -2
View File
@@ -1172,7 +1172,8 @@ unsigned long zsetLength(const robj *zobj) {
* and the value len hint indicates the approximate individual size of the added elements,
* they are used to determine the initial representation.
*
* If the hints are not known, and underestimation or 0 is suitable. */
* If the hints are not known, and underestimation or 0 is suitable.
* We should never pass a negative value because it will convert to a very large unsigned number. */
robj *zsetTypeCreate(size_t size_hint, size_t val_len_hint) {
if (size_hint <= server.zset_max_listpack_entries &&
val_len_hint <= server.zset_max_listpack_value)
@@ -3001,7 +3002,7 @@ static void zrangeResultFinalizeClient(zrange_result_handler *handler,
/* Result handler methods for storing the ZRANGESTORE to a zset. */
static void zrangeResultBeginStore(zrange_result_handler *handler, long length)
{
handler->dstobj = zsetTypeCreate(length, 0);
handler->dstobj = zsetTypeCreate(length >= 0 ? length : 0, 0);
}
static void zrangeResultEmitCBufferForStore(zrange_result_handler *handler,
+10 -4
View File
@@ -54,8 +54,11 @@
/* Glob-style pattern matching. */
static int stringmatchlen_impl(const char *pattern, int patternLen,
const char *string, int stringLen, int nocase, int *skipLongerMatches)
const char *string, int stringLen, int nocase, int *skipLongerMatches, int nesting)
{
/* Protection against abusive patterns. */
if (nesting > 1000) return 0;
while(patternLen && stringLen) {
switch(pattern[0]) {
case '*':
@@ -67,7 +70,7 @@ static int stringmatchlen_impl(const char *pattern, int patternLen,
return 1; /* match */
while(stringLen) {
if (stringmatchlen_impl(pattern+1, patternLen-1,
string, stringLen, nocase, skipLongerMatches))
string, stringLen, nocase, skipLongerMatches, nesting+1))
return 1; /* match */
if (*skipLongerMatches)
return 0; /* no match */
@@ -189,7 +192,7 @@ static int stringmatchlen_impl(const char *pattern, int patternLen,
int stringmatchlen(const char *pattern, int patternLen,
const char *string, int stringLen, int nocase) {
int skipLongerMatches = 0;
return stringmatchlen_impl(pattern,patternLen,string,stringLen,nocase,&skipLongerMatches);
return stringmatchlen_impl(pattern,patternLen,string,stringLen,nocase,&skipLongerMatches,0);
}
int stringmatch(const char *pattern, const char *string, int nocase) {
@@ -1138,7 +1141,10 @@ int fsyncFileDir(const char *filename) {
int reclaimFilePageCache(int fd, size_t offset, size_t length) {
#ifdef HAVE_FADVISE
int ret = posix_fadvise(fd, offset, length, POSIX_FADV_DONTNEED);
if (ret) return -1;
if (ret) {
errno = ret;
return -1;
}
return 0;
#else
UNUSED(fd);
+2 -2
View File
@@ -1,2 +1,2 @@
#define REDIS_VERSION "255.255.255"
#define REDIS_VERSION_NUM 0x00ffffff
#define REDIS_VERSION "7.2.8"
#define REDIS_VERSION_NUM 0x00070208
+27
View File
@@ -60,6 +60,33 @@ test "slot migration is valid from primary to another primary" {
assert_equal {OK} [$nodeto(link) cluster setslot $slot node $nodeto(id)]
}
test "Client unblocks after slot migration from one primary to another" {
set cluster [redis_cluster 127.0.0.1:[get_instance_attrib redis 0 port]]
set key mystream
set slot [$cluster cluster keyslot $key]
array set nodefrom [$cluster masternode_for_slot $slot]
array set nodeto [$cluster masternode_notfor_slot $slot]
# Create a stream group on the source node
$nodefrom(link) XGROUP CREATE $key mygroup $ MKSTREAM
# block another client on xreadgroup
set rd [redis_deferring_client_by_addr $nodefrom(host) $nodefrom(port)]
$rd XREADGROUP GROUP mygroup Alice BLOCK 0 STREAMS $key ">"
wait_for_condition 1000 50 {
[getInfoProperty [$nodefrom(link) info clients] blocked_clients] eq {1}
} else {
fail "client wasn't blocked"
}
# Start slot migration from the source node to the target node.
# Because the `unblock_on_nokey` option of xreadgroup is set to 1, the client
# will be unblocked when the key `mystream` is migrated.
assert_equal {OK} [$nodefrom(link) CLUSTER SETSLOT $slot MIGRATING $nodeto(id)]
assert_equal {OK} [$nodeto(link) CLUSTER SETSLOT $slot IMPORTING $nodefrom(id)]
$nodefrom(link) MIGRATE $nodeto(host) $nodeto(port) $key 0 5000
}
test "slot migration is invalid from primary to replica" {
set cluster [redis_cluster 127.0.0.1:[get_instance_attrib redis 0 port]]
set key order1
@@ -124,6 +124,10 @@ test "Verify health as fail for killed node" {
}
}
test "Verify that other nodes can correctly output the new master's slots" {
assert_not_equal {} [dict get [get_node_info_from_shard [R 4 CLUSTER MYID] 8 "shard"] slots]
}
set primary_id 4
set replica_id 0
+25
View File
@@ -529,6 +529,18 @@ tags {"aof external:skip"} {
assert_match "*Start checking Old-Style AOF*is valid*" $result
}
test {Test redis-check-aof for old style resp AOF - has data in the same format as manifest} {
create_aof $aof_dirpath $aof_file {
append_to_aof [formatCommand set file file]
append_to_aof [formatCommand set "file appendonly.aof.2.base.rdb seq 2 type b" "file appendonly.aof.2.base.rdb seq 2 type b"]
}
catch {
exec src/redis-check-aof $aof_file
} result
assert_match "*Start checking Old-Style AOF*is valid*" $result
}
test {Test redis-check-aof for old style rdb-preamble AOF} {
catch {
exec src/redis-check-aof tests/assets/rdb-preamble.aof
@@ -577,6 +589,19 @@ tags {"aof external:skip"} {
assert_match "*Start checking Multi Part AOF*Start to check BASE AOF (RDB format)*DB preamble is OK, proceeding with AOF tail*BASE AOF*is valid*Start to check INCR files*INCR AOF*is valid*All AOF files and manifest are valid*" $result
}
test {Test redis-check-aof for Multi Part AOF contains a format error} {
create_aof_manifest $aof_dirpath $aof_manifest_file {
append_to_manifest "file appendonly.aof.1.base.aof seq 1 type b\n"
append_to_manifest "file appendonly.aof.1.incr.aof seq 1 type i\n"
append_to_manifest "!!!\n"
}
catch {
exec src/redis-check-aof $aof_manifest_file
} result
assert_match "*Invalid AOF manifest file format*" $result
}
test {Test redis-check-aof only truncates the last file for Multi Part AOF in fix mode} {
create_aof $aof_dirpath $aof_base_file {
append_to_aof [formatCommand set foo hello]
+8 -7
View File
@@ -102,6 +102,7 @@ typedef struct {
void *bg_call_worker(void *arg) {
bg_call_data *bg = arg;
RedisModuleBlockedClient *bc = bg->bc;
// Get Redis module context
RedisModuleCtx *ctx = RedisModule_GetThreadSafeContext(bg->bc);
@@ -135,6 +136,12 @@ void *bg_call_worker(void *arg) {
RedisModuleCallReply *rep = RedisModule_Call(ctx, cmd, format, bg->argv + cmd_pos + 1, bg->argc - cmd_pos - 1);
RedisModule_FreeString(NULL, format_redis_str);
/* Free the arguments within GIL to prevent simultaneous freeing in main thread. */
for (int i=0; i<bg->argc; i++)
RedisModule_FreeString(ctx, bg->argv[i]);
RedisModule_Free(bg->argv);
RedisModule_Free(bg);
// Release GIL
RedisModule_ThreadSafeContextUnlock(ctx);
@@ -147,13 +154,7 @@ void *bg_call_worker(void *arg) {
}
// Unblock client
RedisModule_UnblockClient(bg->bc, NULL);
/* Free the arguments */
for (int i=0; i<bg->argc; i++)
RedisModule_FreeString(ctx, bg->argv[i]);
RedisModule_Free(bg->argv);
RedisModule_Free(bg);
RedisModule_UnblockClient(bc, NULL);
// Free the Redis module context
RedisModule_FreeThreadSafeContext(ctx);
+51 -16
View File
@@ -7,12 +7,41 @@
#define UNUSED(x) (void)(x)
typedef struct {
/* Mutex for protecting RedisModule_BlockedClientMeasureTime*() API from race
* conditions due to timeout callback triggered in the main thread. */
pthread_mutex_t measuretime_mutex;
int measuretime_completed; /* Indicates that time measure has ended and will not continue further */
int myint; /* Used for replying */
} BlockPrivdata;
void blockClientPrivdataInit(RedisModuleBlockedClient *bc) {
BlockPrivdata *block_privdata = RedisModule_Calloc(1, sizeof(*block_privdata));
block_privdata->measuretime_mutex = (pthread_mutex_t)PTHREAD_MUTEX_INITIALIZER;
RedisModule_BlockClientSetPrivateData(bc, block_privdata);
}
void blockClientMeasureTimeStart(RedisModuleBlockedClient *bc, BlockPrivdata *block_privdata) {
pthread_mutex_lock(&block_privdata->measuretime_mutex);
RedisModule_BlockedClientMeasureTimeStart(bc);
pthread_mutex_unlock(&block_privdata->measuretime_mutex);
}
void blockClientMeasureTimeEnd(RedisModuleBlockedClient *bc, BlockPrivdata *block_privdata, int completed) {
pthread_mutex_lock(&block_privdata->measuretime_mutex);
if (!block_privdata->measuretime_completed) {
RedisModule_BlockedClientMeasureTimeEnd(bc);
if (completed) block_privdata->measuretime_completed = 1;
}
pthread_mutex_unlock(&block_privdata->measuretime_mutex);
}
/* Reply callback for blocking command BLOCK.DEBUG */
int HelloBlock_Reply(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
UNUSED(argv);
UNUSED(argc);
int *myint = RedisModule_GetBlockedClientPrivateData(ctx);
return RedisModule_ReplyWithLongLong(ctx,*myint);
BlockPrivdata *block_privdata = RedisModule_GetBlockedClientPrivateData(ctx);
return RedisModule_ReplyWithLongLong(ctx,block_privdata->myint);
}
/* Timeout callback for blocking command BLOCK.DEBUG */
@@ -20,13 +49,16 @@ int HelloBlock_Timeout(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
UNUSED(argv);
UNUSED(argc);
RedisModuleBlockedClient *bc = RedisModule_GetBlockedClientHandle(ctx);
RedisModule_BlockedClientMeasureTimeEnd(bc);
BlockPrivdata *block_privdata = RedisModule_GetBlockedClientPrivateData(ctx);
blockClientMeasureTimeEnd(bc, block_privdata, 1);
return RedisModule_ReplyWithSimpleString(ctx,"Request timedout");
}
/* Private data freeing callback for BLOCK.DEBUG command. */
void HelloBlock_FreeData(RedisModuleCtx *ctx, void *privdata) {
UNUSED(ctx);
BlockPrivdata *block_privdata = privdata;
pthread_mutex_destroy(&block_privdata->measuretime_mutex);
RedisModule_Free(privdata);
}
@@ -42,19 +74,20 @@ void *BlockDebug_ThreadMain(void *arg) {
RedisModuleBlockedClient *bc = targ[0];
long long delay = (unsigned long)targ[1];
long long enable_time_track = (unsigned long)targ[2];
BlockPrivdata *block_privdata = RedisModule_BlockClientGetPrivateData(bc);
if (enable_time_track)
RedisModule_BlockedClientMeasureTimeStart(bc);
blockClientMeasureTimeStart(bc, block_privdata);
RedisModule_Free(targ);
struct timespec ts;
ts.tv_sec = delay / 1000;
ts.tv_nsec = (delay % 1000) * 1000000;
nanosleep(&ts, NULL);
int *r = RedisModule_Alloc(sizeof(int));
*r = rand();
if (enable_time_track)
RedisModule_BlockedClientMeasureTimeEnd(bc);
RedisModule_UnblockClient(bc,r);
blockClientMeasureTimeEnd(bc, block_privdata, 0);
block_privdata->myint = rand();
RedisModule_UnblockClient(bc,block_privdata);
return NULL;
}
@@ -64,23 +97,22 @@ void *DoubleBlock_ThreadMain(void *arg) {
void **targ = arg;
RedisModuleBlockedClient *bc = targ[0];
long long delay = (unsigned long)targ[1];
RedisModule_BlockedClientMeasureTimeStart(bc);
BlockPrivdata *block_privdata = RedisModule_BlockClientGetPrivateData(bc);
blockClientMeasureTimeStart(bc, block_privdata);
RedisModule_Free(targ);
struct timespec ts;
ts.tv_sec = delay / 1000;
ts.tv_nsec = (delay % 1000) * 1000000;
nanosleep(&ts, NULL);
int *r = RedisModule_Alloc(sizeof(int));
*r = rand();
RedisModule_BlockedClientMeasureTimeEnd(bc);
blockClientMeasureTimeEnd(bc, block_privdata, 0);
/* call again RedisModule_BlockedClientMeasureTimeStart() and
* RedisModule_BlockedClientMeasureTimeEnd and ensure that the
* total execution time is 2x the delay. */
RedisModule_BlockedClientMeasureTimeStart(bc);
blockClientMeasureTimeStart(bc, block_privdata);
nanosleep(&ts, NULL);
RedisModule_BlockedClientMeasureTimeEnd(bc);
RedisModule_UnblockClient(bc,r);
blockClientMeasureTimeEnd(bc, block_privdata, 0);
block_privdata->myint = rand();
RedisModule_UnblockClient(bc,block_privdata);
return NULL;
}
@@ -107,6 +139,7 @@ int HelloBlock_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int a
pthread_t tid;
RedisModuleBlockedClient *bc = RedisModule_BlockClient(ctx,HelloBlock_Reply,HelloBlock_Timeout,HelloBlock_FreeData,timeout);
blockClientPrivdataInit(bc);
/* Here we set a disconnection handler, however since this module will
* block in sleep() in a thread, there is not much we can do in the
@@ -148,6 +181,7 @@ int HelloBlockNoTracking_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **a
pthread_t tid;
RedisModuleBlockedClient *bc = RedisModule_BlockClient(ctx,HelloBlock_Reply,HelloBlock_Timeout,HelloBlock_FreeData,timeout);
blockClientPrivdataInit(bc);
/* Here we set a disconnection handler, however since this module will
* block in sleep() in a thread, there is not much we can do in the
@@ -184,6 +218,7 @@ int HelloDoubleBlock_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv,
pthread_t tid;
RedisModuleBlockedClient *bc = RedisModule_BlockClient(ctx,HelloBlock_Reply,HelloBlock_Timeout,HelloBlock_FreeData,0);
blockClientPrivdataInit(bc);
/* Now that we setup a blocking client, we need to pass the control
* to the thread. However we need to pass arguments to the thread:
+14 -3
View File
@@ -1,6 +1,7 @@
#include "redismodule.h"
#include <string.h>
#include <strings.h>
static RedisModuleString *log_key_name;
@@ -92,7 +93,7 @@ int CommandFilter_LogCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int
return REDISMODULE_OK;
}
int CommandFilter_UnfilteredClientdId(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
int CommandFilter_UnfilteredClientId(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
{
if (argc < 2)
return RedisModule_WrongArity(ctx);
@@ -192,7 +193,7 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
if (RedisModule_Init(ctx,"commandfilter",1,REDISMODULE_APIVER_1)
== REDISMODULE_ERR) return REDISMODULE_ERR;
if (argc != 2) {
if (argc != 2 && argc != 3) {
RedisModule_Log(ctx, "warning", "Log key name not specified");
return REDISMODULE_ERR;
}
@@ -219,7 +220,7 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
return REDISMODULE_ERR;
if (RedisModule_CreateCommand(ctx, unfiltered_clientid_name,
CommandFilter_UnfilteredClientdId, "admin", 1,1,1) == REDISMODULE_ERR)
CommandFilter_UnfilteredClientId, "admin", 1,1,1) == REDISMODULE_ERR)
return REDISMODULE_ERR;
if ((filter = RedisModule_RegisterCommandFilter(ctx, CommandFilter_CommandFilter,
@@ -229,6 +230,16 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
if ((filter1 = RedisModule_RegisterCommandFilter(ctx, CommandFilter_BlmoveSwap, 0)) == NULL)
return REDISMODULE_ERR;
if (argc == 3) {
const char *ptr = RedisModule_StringPtrLen(argv[2], NULL);
if (!strcasecmp(ptr, "noload")) {
/* This is a hint that we return ERR at the last moment of OnLoad. */
RedisModule_FreeString(ctx, log_key_name);
if (retained) RedisModule_FreeString(NULL, retained);
return REDISMODULE_ERR;
}
}
return REDISMODULE_OK;
}
+9
View File
@@ -312,3 +312,12 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
return REDISMODULE_OK;
}
int RedisModule_OnUnload(RedisModuleCtx *ctx) {
REDISMODULE_NOT_USED(ctx);
if (datatype) {
RedisModule_Free(datatype);
datatype = NULL;
}
return REDISMODULE_OK;
}
+2 -1
View File
@@ -141,13 +141,14 @@ size_t FragFreeEffort(RedisModuleString *key, const void *value) {
}
int FragDefrag(RedisModuleDefragCtx *ctx, RedisModuleString *key, void **value) {
REDISMODULE_NOT_USED(key);
unsigned long i = 0;
int steps = 0;
int dbid = RedisModule_GetDbIdFromDefragCtx(ctx);
RedisModule_Assert(dbid != -1);
RedisModule_Log(NULL, "notice", "Defrag key: %s", RedisModule_StringPtrLen(key, NULL));
/* Attempt to get cursor, validate it's what we're exepcting */
if (RedisModule_DefragCursorGet(ctx, &i) == REDISMODULE_OK) {
if (i > 0) datatype_resumes++;
+13 -3
View File
@@ -33,6 +33,7 @@
#include "redismodule.h"
#include <stdio.h>
#include <string.h>
#include <strings.h>
#include <assert.h>
/* We need to store events to be able to test and see what we got, and we can't
@@ -407,9 +408,6 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
return REDISMODULE_ERR; \
}
REDISMODULE_NOT_USED(argv);
REDISMODULE_NOT_USED(argc);
if (RedisModule_Init(ctx,"testhook",1,REDISMODULE_APIVER_1)
== REDISMODULE_ERR) return REDISMODULE_ERR;
@@ -471,6 +469,18 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
if (RedisModule_CreateCommand(ctx,"hooks.pexpireat", cmdKeyExpiry,"",0,0,0) == REDISMODULE_ERR)
return REDISMODULE_ERR;
if (argc == 1) {
const char *ptr = RedisModule_StringPtrLen(argv[0], NULL);
if (!strcasecmp(ptr, "noload")) {
/* This is a hint that we return ERR at the last moment of OnLoad. */
RedisModule_FreeDict(ctx, event_log);
RedisModule_FreeDict(ctx, removed_event_log);
RedisModule_FreeDict(ctx, removed_subevent_type);
RedisModule_FreeDict(ctx, removed_expiry_log);
return REDISMODULE_ERR;
}
}
return REDISMODULE_OK;
}
+11 -3
View File
@@ -36,6 +36,7 @@
#include "redismodule.h"
#include <stdio.h>
#include <string.h>
#include <strings.h>
#include <unistd.h>
ustime_t cached_time = 0;
@@ -318,9 +319,6 @@ static int cmdGetDels(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
/* This function must be present on each Redis module. It is used in order to
* register the commands into the Redis server. */
int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
REDISMODULE_NOT_USED(argv);
REDISMODULE_NOT_USED(argc);
if (RedisModule_Init(ctx,"testkeyspace",1,REDISMODULE_APIVER_1) == REDISMODULE_ERR){
return REDISMODULE_ERR;
}
@@ -405,6 +403,16 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
return REDISMODULE_ERR;
}
if (argc == 1) {
const char *ptr = RedisModule_StringPtrLen(argv[0], NULL);
if (!strcasecmp(ptr, "noload")) {
/* This is a hint that we return ERR at the last moment of OnLoad. */
RedisModule_FreeDict(ctx, loaded_event_log);
RedisModule_FreeDict(ctx, module_event_log);
return REDISMODULE_ERR;
}
}
return REDISMODULE_OK;
}
+67
View File
@@ -90,6 +90,10 @@ static int KeySpace_NotificationEvicted(RedisModuleCtx *ctx, int type, const cha
return REDISMODULE_OK; /* do not count the evicted key */
}
if (strncmp(key_str, "before_evicted", 14) == 0) {
return REDISMODULE_OK; /* do not count the before_evicted key */
}
RedisModuleString *new_key = RedisModule_CreateString(NULL, "evicted", 7);
RedisModule_AddPostNotificationJob(ctx, KeySpace_PostNotificationString, new_key, KeySpace_PostNotificationStringFreePD);
return REDISMODULE_OK;
@@ -186,6 +190,55 @@ static int KeySpace_PostNotificationsAsyncSet(RedisModuleCtx *ctx, RedisModuleSt
return REDISMODULE_OK;
}
typedef struct KeySpace_EventPostNotificationCtx {
RedisModuleString *triggered_on;
RedisModuleString *new_key;
} KeySpace_EventPostNotificationCtx;
static void KeySpace_ServerEventPostNotificationFree(void *pd) {
KeySpace_EventPostNotificationCtx *pn_ctx = pd;
RedisModule_FreeString(NULL, pn_ctx->new_key);
RedisModule_FreeString(NULL, pn_ctx->triggered_on);
RedisModule_Free(pn_ctx);
}
static void KeySpace_ServerEventPostNotification(RedisModuleCtx *ctx, void *pd) {
REDISMODULE_NOT_USED(ctx);
KeySpace_EventPostNotificationCtx *pn_ctx = pd;
RedisModuleCallReply* rep = RedisModule_Call(ctx, "lpush", "!ss", pn_ctx->new_key, pn_ctx->triggered_on);
RedisModule_FreeCallReply(rep);
}
static void KeySpace_ServerEventCallback(RedisModuleCtx *ctx, RedisModuleEvent eid, uint64_t subevent, void *data) {
REDISMODULE_NOT_USED(eid);
REDISMODULE_NOT_USED(data);
if (subevent > 3) {
RedisModule_Log(ctx, "warning", "Got an unexpected subevent '%ld'", subevent);
return;
}
static const char* events[] = {
"before_deleted",
"before_expired",
"before_evicted",
"before_overwritten",
};
const RedisModuleString *key_name = RedisModule_GetKeyNameFromModuleKey(((RedisModuleKeyInfo*)data)->key);
const char *key_str = RedisModule_StringPtrLen(key_name, NULL);
for (int i = 0 ; i < 4 ; ++i) {
const char *event = events[i];
if (strncmp(key_str, event , strlen(event)) == 0) {
return; /* don't log any event on our tracking keys */
}
}
KeySpace_EventPostNotificationCtx *pn_ctx = RedisModule_Alloc(sizeof(*pn_ctx));
pn_ctx->triggered_on = RedisModule_HoldString(NULL, (RedisModuleString*)key_name);
pn_ctx->new_key = RedisModule_CreateString(NULL, events[subevent], strlen(events[subevent]));
RedisModule_AddPostNotificationJob(ctx, KeySpace_ServerEventPostNotification, pn_ctx, KeySpace_ServerEventPostNotificationFree);
}
/* This function must be present on each Redis module. It is used in order to
* register the commands into the Redis server. */
int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
@@ -200,6 +253,14 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
return REDISMODULE_ERR;
}
int with_key_events = 0;
if (argc >= 1) {
const char *arg = RedisModule_StringPtrLen(argv[0], 0);
if (strcmp(arg, "with_key_events") == 0) {
with_key_events = 1;
}
}
RedisModule_SetModuleOptions(ctx, REDISMODULE_OPTIONS_ALLOW_NESTED_KEYSPACE_NOTIFICATIONS);
if(RedisModule_SubscribeToKeyspaceEvents(ctx, REDISMODULE_NOTIFY_STRING, KeySpace_NotificationString) != REDISMODULE_OK){
@@ -222,6 +283,12 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
return REDISMODULE_ERR;
}
if (with_key_events) {
if(RedisModule_SubscribeToServerEvent(ctx, RedisModuleEvent_Key, KeySpace_ServerEventCallback) != REDISMODULE_OK){
return REDISMODULE_ERR;
}
}
if (RedisModule_CreateCommand(ctx, "postnotification.async_set", KeySpace_PostNotificationsAsyncSet,
"write", 0, 0, 0) == REDISMODULE_ERR){
return REDISMODULE_ERR;
+20
View File
@@ -323,6 +323,21 @@ int propagateTestIncr(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
return REDISMODULE_OK;
}
int propagateTestVerbatim(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
if (argc < 2){
RedisModule_WrongArity(ctx);
return REDISMODULE_OK;
}
long long replicate_num;
RedisModule_StringToLongLong(argv[1], &replicate_num);
/* Replicate the command verbatim for the specified number of times. */
for (long long i = 0; i < replicate_num; i++)
RedisModule_ReplicateVerbatim(ctx);
RedisModule_ReplyWithSimpleString(ctx,"OK");
return REDISMODULE_OK;
}
int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
REDISMODULE_NOT_USED(argv);
REDISMODULE_NOT_USED(argc);
@@ -389,6 +404,11 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
propagateTestIncr,
"write",1,1,1) == REDISMODULE_ERR)
return REDISMODULE_ERR;
if (RedisModule_CreateCommand(ctx,"propagate-test.verbatim",
propagateTestVerbatim,
"write",1,1,1) == REDISMODULE_ERR)
return REDISMODULE_ERR;
return REDISMODULE_OK;
}
+8 -7
View File
@@ -115,6 +115,7 @@ typedef struct {
void *bg_call_worker(void *arg) {
bg_call_data *bg = arg;
RedisModuleBlockedClient *bc = bg->bc;
// Get Redis module context
RedisModuleCtx *ctx = RedisModule_GetThreadSafeContext(bg->bc);
@@ -136,6 +137,12 @@ void *bg_call_worker(void *arg) {
RedisModuleCallReply *rep = RedisModule_Call(ctx, cmd, format, bg->argv + 3, bg->argc - 3);
RedisModule_FreeString(NULL, format_redis_str);
/* Free the arguments within GIL to prevent simultaneous freeing in main thread. */
for (int i=0; i<bg->argc; i++)
RedisModule_FreeString(ctx, bg->argv[i]);
RedisModule_Free(bg->argv);
RedisModule_Free(bg);
// Release GIL
RedisModule_ThreadSafeContextUnlock(ctx);
@@ -148,13 +155,7 @@ void *bg_call_worker(void *arg) {
}
// Unblock client
RedisModule_UnblockClient(bg->bc, NULL);
/* Free the arguments */
for (int i=0; i<bg->argc; i++)
RedisModule_FreeString(ctx, bg->argv[i]);
RedisModule_Free(bg->argv);
RedisModule_Free(bg);
RedisModule_UnblockClient(bc, NULL);
// Free the Redis module context
RedisModule_FreeThreadSafeContext(ctx);
+27
View File
@@ -199,3 +199,30 @@ proc are_hostnames_propagated {match_string} {
}
return 1
}
proc wait_node_marked_fail {ref_node_index instance_id_to_check} {
wait_for_condition 1000 50 {
[check_cluster_node_mark fail $ref_node_index $instance_id_to_check]
} else {
fail "Replica node never marked as FAIL ('fail')"
}
}
proc wait_node_marked_pfail {ref_node_index instance_id_to_check} {
wait_for_condition 1000 50 {
[check_cluster_node_mark fail\? $ref_node_index $instance_id_to_check]
} else {
fail "Replica node never marked as PFAIL ('fail?')"
}
}
proc check_cluster_node_mark {flag ref_node_index instance_id_to_check} {
set nodes [get_cluster_nodes $ref_node_index]
foreach n $nodes {
if {[dict get $n id] eq $instance_id_to_check} {
return [cluster_has_flag $n $flag]
}
}
fail "Unable to find instance id in cluster nodes. ID: $instance_id_to_check"
}
+3
View File
@@ -94,6 +94,7 @@ set ::all_tests {
unit/client-eviction
unit/violations
unit/replybufsize
unit/cluster/announced-endpoints
unit/cluster/misc
unit/cluster/cli
unit/cluster/scripting
@@ -103,6 +104,8 @@ set ::all_tests {
unit/cluster/slot-ownership
unit/cluster/links
unit/cluster/cluster-response-tls
unit/cluster/failure-marking
unit/cluster/sharded-pubsub
}
# Index to the next test to run in the ::all_tests list.
set ::next_test 0
+26
View File
@@ -116,6 +116,32 @@ start_server {tags {"acl external:skip"}} {
assert_match "*NOPERM*key*" $err
}
test {Validate read and write permissions format - empty permission} {
catch {r ACL SETUSER key-permission-RW %~} err
set err
} {ERR Error in ACL SETUSER modifier '%~': Syntax error}
test {Validate read and write permissions format - empty selector} {
catch {r ACL SETUSER key-permission-RW %} err
set err
} {ERR Error in ACL SETUSER modifier '%': Syntax error}
test {Validate read and write permissions format - empty pattern} {
# Empty pattern results with R/W access to no key
r ACL SETUSER key-permission-RW on nopass %RW~ +@all
$r2 auth key-permission-RW password
catch {$r2 SET x 5} err
set err
} {NOPERM No permissions to access a key}
test {Validate read and write permissions format - no pattern} {
# No pattern results with R/W access to no key (currently we accept this syntax error)
r ACL SETUSER key-permission-RW on nopass %RW +@all
$r2 auth key-permission-RW password
catch {$r2 SET x 5} err
set err
} {NOPERM No permissions to access a key}
test {Test separate read and write permissions on different selectors are not additive} {
r ACL SETUSER key-permission-RW-selector on nopass "(%R~read* +@all)" "(%W~write* +@all)"
$r2 auth key-permission-RW-selector password
+18
View File
@@ -45,6 +45,24 @@ start_server {tags {"auth external:skip"} overrides {requirepass foobar}} {
assert_match {*unauthenticated bulk length*} $e
$rr close
}
test {For unauthenticated clients output buffer is limited} {
set rr [redis [srv "host"] [srv "port"] 1 $::tls]
$rr SET x 5
catch {[$rr read]} e
assert_match {*NOAUTH Authentication required*} $e
# Fill the output buffer in a loop without reading it and make
# sure the client disconnected.
# Considering the socket eat some of the replies, we are testing
# that such client can't consume more than few MB's.
catch {
for {set j 0} {$j < 1000000} {incr j} {
$rr SET x 5
}
} e
assert_match {I/O error reading reply} $e
}
}
start_server {tags {"auth_binary_password external:skip"}} {
+13 -5
View File
@@ -1,8 +1,12 @@
start_cluster 2 2 {tags {external:skip cluster}} {
test "Test change cluster-announce-port and cluster-announce-tls-port at runtime" {
set baseport [lindex [R 0 config get port] 1]
set count [expr [llength $::servers] +1 ]
if {$::tls} {
set baseport [lindex [R 0 config get tls-port] 1]
} else {
set baseport [lindex [R 0 config get port] 1]
}
set count [expr [llength $::servers] + 1]
set used_port [find_available_port $baseport $count]
R 0 config set cluster-announce-tls-port $used_port
@@ -17,12 +21,16 @@ start_cluster 2 2 {tags {external:skip cluster}} {
R 0 config set cluster-announce-tls-port 0
R 0 config set cluster-announce-port 0
assert_match "*:$baseport@*" [R 0 CLUSTER NODES]
assert_match "*:$baseport@*" [R 0 CLUSTER NODES]
}
test "Test change cluster-announce-bus-port at runtime" {
set baseport [lindex [R 0 config get port] 1]
set count [expr [llength $::servers] +1 ]
if {$::tls} {
set baseport [lindex [R 0 config get tls-port] 1]
} else {
set baseport [lindex [R 0 config get port] 1]
}
set count [expr [llength $::servers] + 1]
set used_port [find_available_port $baseport $count]
# Verify config set cluster-announce-bus-port
+53
View File
@@ -0,0 +1,53 @@
# Test a single primary can mark replica as `fail`
start_cluster 1 1 {tags {external:skip cluster}} {
test "Verify that single primary marks replica as failed" {
set primary [srv -0 client]
set replica1 [srv -1 client]
set replica1_pid [srv -1 pid]
set replica1_instance_id [dict get [cluster_get_myself 1] id]
assert {[lindex [$primary role] 0] eq {master}}
assert {[lindex [$replica1 role] 0] eq {slave}}
wait_for_sync $replica1
pause_process $replica1_pid
wait_node_marked_fail 0 $replica1_instance_id
}
}
# Test multiple primaries wait for a quorum and then mark a replica as `fail`
start_cluster 2 1 {tags {external:skip cluster}} {
test "Verify that multiple primaries mark replica as failed" {
set primary1 [srv -0 client]
set primary2 [srv -1 client]
set primary2_pid [srv -1 pid]
set replica1 [srv -2 client]
set replica1_pid [srv -2 pid]
set replica1_instance_id [dict get [cluster_get_myself 2] id]
assert {[lindex [$primary1 role] 0] eq {master}}
assert {[lindex [$primary2 role] 0] eq {master}}
assert {[lindex [$replica1 role] 0] eq {slave}}
wait_for_sync $replica1
pause_process $replica1_pid
# Pause other primary to allow time for pfail flag to appear
pause_process $primary2_pid
wait_node_marked_pfail 0 $replica1_instance_id
# Resume other primary and wait for to show replica as failed
resume_process $primary2_pid
wait_node_marked_fail 0 $replica1_instance_id
}
}
+37 -11
View File
@@ -1,3 +1,16 @@
#
# Copyright (c) 2009-Present, Redis Ltd.
# All rights reserved.
#
# Copyright (c) 2024-present, Valkey contributors.
# All rights reserved.
#
# Licensed under your choice of the Redis Source Available License 2.0
# (RSALv2) or the Server Side Public License v1 (SSPLv1).
#
# Portions of this file are available under BSD3 terms; see REDISCONTRIBUTIONS for more information.
#
proc get_slot_field {slot_output shard_id node_id attrib_id} {
return [lindex [lindex [lindex $slot_output $shard_id] $node_id] $attrib_id]
}
@@ -116,10 +129,11 @@ test "Verify the nodes configured with prefer hostname only show hostname for ne
# Have everyone forget node 6 and isolate it from the cluster.
isolate_node 6
# Set hostnames for the masters, now that the node is isolated
R 0 config set cluster-announce-hostname "shard-1.com"
R 1 config set cluster-announce-hostname "shard-2.com"
R 2 config set cluster-announce-hostname "shard-3.com"
set primaries 3
for {set j 0} {$j < $primaries} {incr j} {
# Set hostnames for the masters, now that the node is isolated
R $j config set cluster-announce-hostname "shard-$j.com"
}
# Prevent Node 0 and Node 6 from properly meeting,
# they'll hang in the handshake phase. This allows us to
@@ -149,9 +163,17 @@ test "Verify the nodes configured with prefer hostname only show hostname for ne
} else {
fail "Node did not learn about the 2 shards it can talk to"
}
set slot_result [R 6 CLUSTER SLOTS]
assert_equal [lindex [get_slot_field $slot_result 0 2 3] 1] "shard-2.com"
assert_equal [lindex [get_slot_field $slot_result 1 2 3] 1] "shard-3.com"
wait_for_condition 50 100 {
[lindex [get_slot_field [R 6 CLUSTER SLOTS] 0 2 3] 1] eq "shard-1.com"
} else {
fail "hostname for shard-1 didn't reach node 6"
}
wait_for_condition 50 100 {
[lindex [get_slot_field [R 6 CLUSTER SLOTS] 1 2 3] 1] eq "shard-2.com"
} else {
fail "hostname for shard-2 didn't reach node 6"
}
# Also make sure we know about the isolated master, we
# just can't reach it.
@@ -170,10 +192,14 @@ test "Verify the nodes configured with prefer hostname only show hostname for ne
} else {
fail "Node did not learn about the 2 shards it can talk to"
}
set slot_result [R 6 CLUSTER SLOTS]
assert_equal [lindex [get_slot_field $slot_result 0 2 3] 1] "shard-1.com"
assert_equal [lindex [get_slot_field $slot_result 1 2 3] 1] "shard-2.com"
assert_equal [lindex [get_slot_field $slot_result 2 2 3] 1] "shard-3.com"
for {set j 0} {$j < $primaries} {incr j} {
wait_for_condition 50 100 {
[lindex [get_slot_field [R 6 CLUSTER SLOTS] $j 2 3] 1] eq "shard-$j.com"
} else {
fail "hostname information for shard-$j didn't reach node 6"
}
}
}
test "Test restart will keep hostname information" {
+66
View File
@@ -0,0 +1,66 @@
#
# Copyright (c) 2009-Present, Redis Ltd.
# All rights reserved.
#
# Licensed under your choice of the Redis Source Available License 2.0
# (RSALv2) or the Server Side Public License v1 (SSPLv1).
#
# Portions of this file are available under BSD3 terms; see REDISCONTRIBUTIONS for more information.
#
start_cluster 1 1 {tags {external:skip cluster}} {
set primary_id 0
set replica1_id 1
set primary [Rn $primary_id]
set replica [Rn $replica1_id]
test "Sharded pubsub publish behavior within multi/exec" {
foreach {node} {primary replica} {
set node [set $node]
$node MULTI
$node SPUBLISH ch1 "hello"
$node EXEC
}
}
test "Sharded pubsub within multi/exec with cross slot operation" {
$primary MULTI
$primary SPUBLISH ch1 "hello"
$primary GET foo
catch {[$primary EXEC]} err
assert_match {CROSSSLOT*} $err
}
test "Sharded pubsub publish behavior within multi/exec with read operation on primary" {
$primary MULTI
$primary SPUBLISH foo "hello"
$primary GET foo
$primary EXEC
} {0 {}}
test "Sharded pubsub publish behavior within multi/exec with read operation on replica" {
$replica MULTI
$replica SPUBLISH foo "hello"
catch {[$replica GET foo]} err
assert_match {MOVED*} $err
catch {[$replica EXEC]} err
assert_match {EXECABORT*} $err
}
test "Sharded pubsub publish behavior within multi/exec with write operation on primary" {
$primary MULTI
$primary SPUBLISH foo "hello"
$primary SET foo bar
$primary EXEC
} {0 OK}
test "Sharded pubsub publish behavior within multi/exec with write operation on replica" {
$replica MULTI
$replica SPUBLISH foo "hello"
catch {[$replica SET foo bar]} err
assert_match {MOVED*} $err
catch {[$replica EXEC]} err
assert_match {EXECABORT*} $err
}
}
+10 -4
View File
@@ -300,15 +300,21 @@ start_server {tags {"info" "external:skip"}} {
test {stats: instantaneous metrics} {
r config resetstat
after 1600 ;# hz is 10, wait for 16 cron tick so that sample array is fulfilled
set value [s instantaneous_eventloop_cycles_per_sec]
set retries 0
for {set retries 1} {$retries < 4} {incr retries} {
after 1600 ;# hz is 10, wait for 16 cron tick so that sample array is fulfilled
set value [s instantaneous_eventloop_cycles_per_sec]
if {$value > 0} break
}
assert_lessthan $retries 4
if {$::verbose} { puts "instantaneous metrics instantaneous_eventloop_cycles_per_sec: $value" }
assert_morethan $value 0
assert_lessthan $value 15 ;# default hz is 10
assert_lessthan $value [expr $retries*15] ;# default hz is 10
set value [s instantaneous_eventloop_duration_usec]
if {$::verbose} { puts "instantaneous metrics instantaneous_eventloop_duration_usec: $value" }
assert_morethan $value 0
assert_lessthan $value 22000 ;# default hz is 10, so duration < 1000 / 10, allow some tolerance
assert_lessthan $value [expr $retries*22000] ;# default hz is 10, so duration < 1000 / 10, allow some tolerance
}
test {stats: debug metrics} {
+4
View File
@@ -133,6 +133,10 @@ start_server {tags {"introspection"}} {
assert_equal {{k1 {RO access}} {k2 {OW update}}} [r command getkeysandflags sort k1 store k2]
}
test {COMMAND GETKEYSANDFLAGS invalid args} {
assert_error "ERR Invalid arguments*" {r command getkeysandflags ZINTERSTORE zz 1443677133621497600 asdf}
}
test {COMMAND GETKEYS MEMORY USAGE} {
assert_equal {key} [r command getkeys memory usage key]
}
+6
View File
@@ -499,4 +499,10 @@ foreach {type large} [array get largevalue] {
r SET aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 1
r KEYS "a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*b"
} {}
test {Regression for pattern matching very long nested loops} {
r flushdb
r SET [string repeat "a" 50000] 1
r KEYS [string repeat "*?" 50000]
} {}
}
+53
View File
@@ -577,4 +577,57 @@ start_server {tags {"defrag external:skip"} overrides {appendonly yes auto-aof-r
}
}
}
start_cluster 1 0 {tags {"defrag external:skip"} overrides {appendonly yes auto-aof-rewrite-percentage 0 save ""}} {
if {[string match {*jemalloc*} [s mem_allocator]] && [r debug mallctl arenas.page] <= 8192} {
test "Active defrag stability during async flush or mid stopping in cluster mode" {
# Verify that asynchronously empting db doesn't cause defragmentation crashes, see issue #13205.
r flushdb async
r config set hz 100
r config set activedefrag no
r config set active-defrag-threshold-lower 1
r config set active-defrag-cycle-min 65
r config set active-defrag-cycle-max 75
r config set active-defrag-ignore-bytes 100k
# create big keys with 10k items
set rd [redis_deferring_client]
for {set j 0} {$j < 100000} {incr j} {
$rd set $j a
$rd expire $j 99999
}
for {set j 0} {$j < 100000} {incr j} {
$rd read ; # Discard replies
$rd read ; # Discard replies
}
catch {r config set activedefrag yes} e
if {[r config get activedefrag] eq "activedefrag yes"} {
# It repeatedly enables and disables active defragmentation,
# and checks if it crashes, see issue #13307.
for {set i 0} {$i < 10} {incr i} {
r config set activedefrag no
# Wait for the active defrag to start working (decision once a second).
wait_for_condition 50 100 {
[s active_defrag_running] eq 0
} else {
after 120 ;# serverCron only updates the info once in 100ms
puts [r info memory]
puts [r memory malloc-stats]
fail "defrag didn't stop."
}
# Wait for the active defrag to stop working.
r config set activedefrag yes
wait_for_condition 50 100 {
[s active_defrag_running] ne 0
} else {
fail "defrag not started."
}
}
}
r ping
}
}
}
} ;# run_solo
+27
View File
@@ -314,6 +314,14 @@ start_server {tags {"modules"}} {
r lpush l a
assert_equal [$rd read] {OK}
# Explanation of the first multi exec block:
# {lpop l} - pop the value by our blocking command 'blpop_and_set_multiple_keys'
# {set string_foo 1} - the action of our blocking command 'blpop_and_set_multiple_keys'
# {set string_bar 2} - the action of our blocking command 'blpop_and_set_multiple_keys'
# {incr string_changed{string_foo}} - post notification job that was registered when 'string_foo' changed
# {incr string_changed{string_bar}} - post notification job that was registered when 'string_bar' changed
# {incr string_total} - post notification job that was registered when string_changed{string_foo} changed
# {incr string_total} - post notification job that was registered when string_changed{string_bar} changed
assert_replication_stream $repl {
{select *}
{lpush l a}
@@ -355,6 +363,25 @@ start_server {tags {"modules"}} {
r lpush l a
assert_equal [$rd read] {OK}
# Explanation of the first multi exec block:
# {lpop l} - pop the value by our blocking command 'blpop_and_set_multiple_keys'
# {set string_foo 1} - the action of our blocking command 'blpop_and_set_multiple_keys'
# {set string_bar 2} - the action of our blocking command 'blpop_and_set_multiple_keys'
# {incr string_changed{string_foo}} - post notification job that was registered when 'string_foo' changed
# {incr string_changed{string_bar}} - post notification job that was registered when 'string_bar' changed
# {incr string_total} - post notification job that was registered when string_changed{string_foo} changed
# {incr string_total} - post notification job that was registered when string_changed{string_bar} changed
#
# Explanation of the second multi exec block:
# {lpop l} - pop the value by our blocking command 'blpop_and_set_multiple_keys'
# {del string_foo} - lazy expiration of string_foo when 'blpop_and_set_multiple_keys' tries to write to it.
# {set string_foo 1} - the action of our blocking command 'blpop_and_set_multiple_keys'
# {set string_bar 2} - the action of our blocking command 'blpop_and_set_multiple_keys'
# {incr expired} - the post notification job, registered after string_foo got expired
# {incr string_changed{string_foo}} - post notification job triggered when we set string_foo
# {incr string_changed{string_bar}} - post notification job triggered when we set string_bar
# {incr string_total} - post notification job triggered when we incr 'string_changed{string_foo}'
# {incr string_total} - post notification job triggered when we incr 'string_changed{string_bar}'
assert_replication_stream $repl {
{select *}
{lpush l a}
+12 -2
View File
@@ -95,7 +95,7 @@ start_server {tags {"modules"}} {
test "Unload the module - commandfilter" {
assert_equal {OK} [r module unload commandfilter]
}
}
}
test {RM_CommandFilterArgInsert and script argv caching} {
# coverage for scripts calling commands that expand the argv array
@@ -162,4 +162,14 @@ test {Filtering based on client id} {
$rr close
}
}
}
start_server {} {
test {OnLoad failure will handle un-registration} {
catch {r module load $testmodule log-key 0 noload}
r set mykey @log
assert_equal [r lrange log-key 0 -1] {}
r rpush mylist elem1 @delme elem2
assert_equal [r lrange mylist 0 -1] {elem1 @delme elem2}
}
}
+5
View File
@@ -1,6 +1,11 @@
set testmodule [file normalize tests/modules/datatype.so]
start_server {tags {"modules"}} {
test {DataType: test loadex with invalid config} {
catch { r module loadex $testmodule CONFIG invalid_config 1 } e
assert_match {*ERR Error loading the extension*} $e
}
r module load $testmodule
test {DataType: Test module is sane, GET/SET work.} {
+8
View File
@@ -310,4 +310,12 @@ tags "modules" {
assert_equal [string match {*module-event-shutdown*} [exec tail -5 < $replica_stdout]] 1
}
}
start_server {} {
test {OnLoad failure will handle un-registration} {
catch {r module load $testmodule noload}
r flushall
r ping
}
}
}
+13
View File
@@ -102,4 +102,17 @@ tags "modules" {
assert_equal {OK} [r set x 1 EX 1]
}
}
start_server {} {
test {OnLoad failure will handle un-registration} {
catch {r module load $testmodule noload}
r set x 1
r hset y f v
r lpush z 1 2 3
r sadd p 1 2 3
r zadd t 1 f1 2 f2
r xadd s * f v
r ping
}
}
}
+21 -7
View File
@@ -1,7 +1,8 @@
set testmodule [file normalize tests/modules/postnotifications.so]
tags "modules" {
start_server [list overrides [list loadmodule "$testmodule"]] {
start_server {} {
r module load $testmodule with_key_events
test {Test write on post notification callback} {
set repl [attach_to_replication_stream]
@@ -9,11 +10,12 @@ tags "modules" {
r set string_x 1
assert_equal {1} [r get string_changed{string_x}]
assert_equal {1} [r get string_total]
r set string_x 2
assert_equal {2} [r get string_changed{string_x}]
assert_equal {2} [r get string_total]
# the {lpush before_overwritten string_x} is a post notification job registered when 'string_x' was overwritten
assert_replication_stream $repl {
{multi}
{select *}
@@ -23,6 +25,7 @@ tags "modules" {
{exec}
{multi}
{set string_x 2}
{lpush before_overwritten string_x}
{incr string_changed{string_x}}
{incr string_total}
{exec}
@@ -37,7 +40,7 @@ tags "modules" {
assert_equal {OK} [r postnotification.async_set]
assert_equal {1} [r get string_changed{string_x}]
assert_equal {1} [r get string_total]
assert_replication_stream $repl {
{multi}
{select *}
@@ -63,12 +66,14 @@ tags "modules" {
fail "Failed waiting for x to expired"
}
# the {lpush before_expired x} is a post notification job registered before 'x' got expired
assert_replication_stream $repl {
{select *}
{set x 1}
{pexpireat x *}
{multi}
{del x}
{lpush before_expired x}
{incr expired}
{exec}
}
@@ -85,12 +90,14 @@ tags "modules" {
after 10
assert_equal {} [r get x]
# the {lpush before_expired x} is a post notification job registered before 'x' got expired
assert_replication_stream $repl {
{select *}
{set x 1}
{pexpireat x *}
{multi}
{del x}
{lpush before_expired x}
{incr expired}
{exec}
}
@@ -108,6 +115,7 @@ tags "modules" {
after 10
assert_equal {OK} [r set read_x 1]
# the {lpush before_expired x} is a post notification job registered before 'x' got expired
assert_replication_stream $repl {
{select *}
{set x 1}
@@ -115,6 +123,7 @@ tags "modules" {
{multi}
{set read_x 1}
{del x}
{lpush before_expired x}
{incr expired}
{exec}
}
@@ -143,16 +152,18 @@ tags "modules" {
r flushall
set repl [attach_to_replication_stream]
r set x 1
r config set maxmemory-policy allkeys-random
r config set maxmemory-policy allkeys-random
r config set maxmemory 1
assert_error {OOM *} {r set y 1}
# the {lpush before_evicted x} is a post notification job registered before 'x' got evicted
assert_replication_stream $repl {
{select *}
{set x 1}
{multi}
{del x}
{lpush before_evicted x}
{incr evicted}
{exec}
}
@@ -164,7 +175,8 @@ tags "modules" {
set testmodule2 [file normalize tests/modules/keyspace_events.so]
tags "modules" {
start_server [list overrides [list loadmodule "$testmodule"]] {
start_server {} {
r module load $testmodule with_key_events
r module load $testmodule2
test {Test write on post notification callback} {
set repl [attach_to_replication_stream]
@@ -172,7 +184,7 @@ tags "modules" {
r set string_x 1
assert_equal {1} [r get string_changed{string_x}]
assert_equal {1} [r get string_total]
r set string_x 2
assert_equal {2} [r get string_changed{string_x}]
assert_equal {2} [r get string_total]
@@ -181,6 +193,7 @@ tags "modules" {
assert_equal {1} [r get string_changed{string1_x}]
assert_equal {3} [r get string_total]
# the {lpush before_overwritten string_x} is a post notification job registered before 'string_x' got overwritten
assert_replication_stream $repl {
{multi}
{select *}
@@ -190,6 +203,7 @@ tags "modules" {
{exec}
{multi}
{set string_x 2}
{lpush before_overwritten string_x}
{incr string_changed{string_x}}
{incr string_total}
{exec}
@@ -202,4 +216,4 @@ tags "modules" {
close_replication_stream $repl
}
}
}
}
+38
View File
@@ -761,3 +761,41 @@ tags "modules aof" {
}
}
}
# This test does not really test module functionality, but rather uses a module
# command to test Redis replication mechanisms.
test {Replicas that was marked as CLIENT_CLOSE_ASAP should not keep the replication backlog from been trimmed} {
start_server [list overrides [list loadmodule "$testmodule"]] {
set replica [srv 0 client]
start_server [list overrides [list loadmodule "$testmodule"]] {
set master [srv 0 client]
set master_host [srv 0 host]
set master_port [srv 0 port]
$master config set client-output-buffer-limit "replica 10mb 5mb 0"
# Start the replication process...
$replica replicaof $master_host $master_port
wait_for_sync $replica
test {module propagates from timer} {
# Replicate large commands to make the replica disconnected.
$master write [format_command propagate-test.verbatim 100000 [string repeat "a" 1000]] ;# almost 100mb
# Execute this command together with module commands within the same
# event loop to prevent periodic cleanup of replication backlog.
$master write [format_command info memory]
$master flush
$master read ;# propagate-test.verbatim
set res [$master read] ;# info memory
# Wait for the replica to be disconnected.
wait_for_log_messages 0 {"*flags=S*scheduled to be closed ASAP for overcoming of output buffer limits*"} 0 1500 10
# Due to the replica reaching the soft limit (5MB), memory peaks should not significantly
# exceed the replica soft limit. Furthermore, as the replica release its reference to
# replication backlog, it should be properly trimmed, the memory usage of replication
# backlog should not significantly exceed repl-backlog-size (default 1MB). */
assert_lessthan [getInfoProperty $res used_memory_peak] 10000000;# less than 10mb
assert_lessthan [getInfoProperty $res mem_replication_backlog] 2000000;# less than 2mb
}
}
}
}
+21 -8
View File
@@ -1,5 +1,4 @@
set system_name [string tolower [exec uname -s]]
set user_id [exec id -u]
if {$system_name eq {linux}} {
start_server {tags {"oom-score-adj external:skip"}} {
@@ -56,8 +55,15 @@ if {$system_name eq {linux}} {
}
}
# Determine whether the current user is unprivileged
set original_value [exec cat /proc/self/oom_score_adj]
catch {
set fd [open "/proc/self/oom_score_adj" "w"]
puts $fd -1000
close $fd
} e
# Failed oom-score-adj tests can only run unprivileged
if {$user_id != 0} {
if {[string match "*permission denied*" $e]} {
test {CONFIG SET oom-score-adj handles configuration failures} {
# Bad config
r config set oom-score-adj no
@@ -81,6 +87,11 @@ if {$system_name eq {linux}} {
# Make sure previous values remain
assert {[r config get oom-score-adj-values] == {oom-score-adj-values {0 100 100}}}
}
} else {
# Restore the original oom_score_adj value
set fd [open "/proc/self/oom_score_adj" "w"]
puts $fd $original_value
close $fd
}
test {CONFIG SET oom-score-adj-values doesn't touch proc when disabled} {
@@ -101,25 +112,27 @@ if {$system_name eq {linux}} {
test {CONFIG SET oom score restored on disable} {
r config set oom-score-adj no
set_oom_score_adj 22
assert_equal [get_oom_score_adj] 22
set custom_oom [expr [get_oom_score_adj] + 1]
set_oom_score_adj $custom_oom
assert_equal [get_oom_score_adj] $custom_oom
r config set oom-score-adj-values "9 9 9" oom-score-adj yes
assert_equal [get_oom_score_adj] [expr 9+22]
assert_equal [get_oom_score_adj] [expr 9+$custom_oom]
r config set oom-score-adj no
assert_equal [get_oom_score_adj] 22
assert_equal [get_oom_score_adj] $custom_oom
}
test {CONFIG SET oom score relative and absolute} {
set custom_oom 9
r config set oom-score-adj no
set base_oom [get_oom_score_adj]
set custom_oom 9
r config set oom-score-adj-values "$custom_oom $custom_oom $custom_oom" oom-score-adj relative
assert_equal [get_oom_score_adj] [expr $base_oom+$custom_oom]
r config set oom-score-adj absolute
set custom_oom [expr [get_oom_score_adj] + 1]
r config set oom-score-adj-values "$custom_oom $custom_oom $custom_oom" oom-score-adj absolute
assert_equal [get_oom_score_adj] $custom_oom
}
+20
View File
@@ -359,6 +359,26 @@ start_server {tags {"pause network"}} {
} {bar2}
}
test "Test the randomkey command will not cause the server to get into an infinite loop during the client pause write" {
r flushall
r multi
r set key value px 3
r client pause 10000 write
r exec
after 5
wait_for_condition 50 100 {
[r randomkey] == "key"
} else {
fail "execute randomkey failed, caused by the infinite loop"
}
r client unpause
assert_equal [r randomkey] {}
}
# Make sure we unpause at the end
r client unpause
}
+14
View File
@@ -146,6 +146,14 @@ start_server {tags {"scripting"}} {
} 1 x
} {number 1}
test {EVAL - Lua number -> Redis integer conversion} {
r del hash
run_script {
local foo = redis.pcall('hincrby','hash','field',200000000)
return {type(foo),foo}
} 0
} {number 200000000}
test {EVAL - Redis bulk -> Lua type conversion} {
r set mykey myval
run_script {
@@ -605,6 +613,12 @@ start_server {tags {"scripting"}} {
set e
} {ERR *Attempt to modify a readonly table*}
test {lua bit.tohex bug} {
set res [run_script {return bit.tohex(65535, -2147483648)} 0]
r ping
set res
} {0000FFFF}
test {Test an example script DECR_IF_GT} {
set decr_if_gt {
local current
+8 -2
View File
@@ -75,12 +75,14 @@ start_server {
assert_equal [lsort -integer $result] [r sort tosort GET #]
} {} {cluster:skip}
test "SORT GET <const>" {
foreach command {SORT SORT_RO} {
test "$command GET <const>" {
r del foo
set res [r sort tosort GET foo]
set res [r $command tosort GET foo]
assert_equal 16 [llength $res]
foreach item $res { assert_equal {} $item }
} {} {cluster:skip}
}
test "SORT GET (key and hash) with sanity check" {
set l1 [r sort tosort GET # GET weight_*]
@@ -109,6 +111,10 @@ start_server {
test "SORT extracts STORE correctly" {
r command getkeys sort abc store def
} {abc def}
test "SORT_RO get keys" {
r command getkeys sort_ro abc
} {abc}
test "SORT extracts multiple STORE correctly" {
r command getkeys sort abc store invalid store stillbad store def
+88 -1
View File
@@ -220,6 +220,7 @@ start_server [list overrides [list save ""] ] {
# checking LSET in case ziplist needs to be split
test {Test LSET with packed is split in the middle} {
set original_config [config_get_set list-max-listpack-size 4]
r flushdb
r debug quicklist-packed-threshold 5b
r RPUSH lst "aa"
@@ -227,6 +228,7 @@ start_server [list overrides [list save ""] ] {
r RPUSH lst "cc"
r RPUSH lst "dd"
r RPUSH lst "ee"
assert_encoding quicklist lst
r lset lst 2 [string repeat e 10]
assert_equal [r lpop lst] "aa"
assert_equal [r lpop lst] "bb"
@@ -234,6 +236,7 @@ start_server [list overrides [list save ""] ] {
assert_equal [r lpop lst] "dd"
assert_equal [r lpop lst] "ee"
r debug quicklist-packed-threshold 0
r config set list-max-listpack-size $original_config
} {OK} {needs:debug}
@@ -381,7 +384,63 @@ if {[lindex [r config get proto-max-bulk-len] 1] == 10000000000} {
assert_equal [read_big_bulk {r rpop lst}] $str_length
} {} {large-memory}
test {Test LMOVE on plain nodes over 4GB} {
test {Test LSET on plain nodes with large elements under packed_threshold over 4GB} {
r flushdb
r rpush lst a b c d e
for {set i 0} {$i < 5} {incr i} {
r write "*4\r\n\$4\r\nlset\r\n\$3\r\nlst\r\n\$1\r\n$i\r\n"
write_big_bulk 1000000000
}
r ping
} {PONG} {large-memory}
test {Test LSET splits a quicklist node, and then merge} {
# Test when a quicklist node can't be inserted and is split, the split
# node merges with the node before it and the `before` node is kept.
r flushdb
r rpush lst [string repeat "x" 4096]
r lpush lst a b c d e f g
r lpush lst [string repeat "y" 4096]
# now: [y...] [g f e d c b a x...]
# (node0) (node1)
# Keep inserting elements into node1 until node1 is split into two
# nodes([g] [...]), eventually node0 will merge with the [g] node.
# Since node0 is larger, after the merge node0 will be kept and
# the [g] node will be deleted.
for {set i 7} {$i >= 3} {incr i -1} {
r write "*4\r\n\$4\r\nlset\r\n\$3\r\nlst\r\n\$1\r\n$i\r\n"
write_big_bulk 1000000000
}
assert_equal "g" [r lindex lst 1]
r ping
} {PONG} {large-memory}
test {Test LSET splits a LZF compressed quicklist node, and then merge} {
# Test when a LZF compressed quicklist node can't be inserted and is split,
# the split node merges with the node before it and the split node is kept.
r flushdb
r config set list-compress-depth 1
r lpush lst [string repeat "x" 2000]
r rpush lst [string repeat "y" 7000]
r rpush lst a b c d e f g
r rpush lst [string repeat "z" 8000]
r lset lst 0 h
# now: [h] [y... a b c d e f g] [z...]
# node0 node1(LZF)
# Keep inserting elements into node1 until node1 is split into two
# nodes([y...] [...]), eventually node0 will merge with the [y...] node.
# Since [y...] node is larger, after the merge node0 will be deleted and
# the [y...] node will be kept.
for {set i 7} {$i >= 3} {incr i -1} {
r write "*4\r\n\$4\r\nlset\r\n\$3\r\nlst\r\n\$1\r\n$i\r\n"
write_big_bulk 1000000000
}
assert_equal "h" [r lindex lst 0]
r config set list-compress-depth 0
r ping
} {PONG} {large-memory}
test {Test LMOVE on plain nodes over 4GB} {
r flushdb
r RPUSH lst2{t} "aa"
r RPUSH lst2{t} "bb"
@@ -1186,6 +1245,34 @@ foreach {pop} {BLPOP BLMPOP_LEFT} {
r select 9
} {OK} {singledb:skip needs:debug}
test {BLPOP unblock but the key is expired and then block again - reprocessing command} {
r flushall
r debug set-active-expire 0
set rd [redis_deferring_client]
set start [clock milliseconds]
$rd blpop mylist 1
wait_for_blocked_clients_count 1
# The exec will try to awake the blocked client, but the key is expired,
# so the client will be blocked again during the command reprocessing.
r multi
r rpush mylist a
r pexpire mylist 100
r debug sleep 0.2
r exec
assert_equal {} [$rd read]
set end [clock milliseconds]
# Before the fix in #13004, this time would have been 1200+ (i.e. more than 1200ms),
# now it should be 1000, but in order to avoid timing issues, we increase the range a bit.
assert_range [expr $end-$start] 1000 1150
r debug set-active-expire 1
$rd close
} {0} {needs:debug}
foreach {pop} {BLPOP BLMPOP_LEFT} {
test "$pop when new key is moved into place" {
set rd [redis_deferring_client]
+97 -1
View File
@@ -475,7 +475,7 @@ start_server {
$rd close
}
test {Blocking XREADGROUP for stream key that has clients blocked on list - avoid endless loop} {
test {Blocking XREADGROUP for stream key that has clients blocked on stream - avoid endless loop} {
r DEL mystream
r XGROUP CREATE mystream mygroup $ MKSTREAM
@@ -498,6 +498,34 @@ start_server {
assert_equal [r ping] {PONG}
}
test {Blocking XREADGROUP for stream key that has clients blocked on stream - reprocessing command} {
r DEL mystream
r XGROUP CREATE mystream mygroup $ MKSTREAM
set rd1 [redis_deferring_client]
set rd2 [redis_deferring_client]
$rd1 xreadgroup GROUP mygroup myuser BLOCK 0 STREAMS mystream >
wait_for_blocked_clients_count 1
set start [clock milliseconds]
$rd2 xreadgroup GROUP mygroup myuser BLOCK 1000 STREAMS mystream >
wait_for_blocked_clients_count 2
# After a while call xadd and let rd2 re-process the command.
after 200
r xadd mystream * field value
assert_equal {} [$rd2 read]
set end [clock milliseconds]
# Before the fix in #13004, this time would have been 1200+ (i.e. more than 1200ms),
# now it should be 1000, but in order to avoid timing issues, we increase the range a bit.
assert_range [expr $end-$start] 1000 1150
$rd1 close
$rd2 close
}
test {XGROUP DESTROY should unblock XREADGROUP with -NOGROUP} {
r config resetstat
r del mystream
@@ -1149,6 +1177,74 @@ start_server {
assert_equal [dict get $group lag] 2
}
test {Consumer Group Lag with XDELs and tombstone after the last_id of consume group} {
r DEL x
r XGROUP CREATE x g1 $ MKSTREAM
r XADD x 1-0 data a
r XREADGROUP GROUP g1 alice STREAMS x > ;# Read one entry
r XADD x 2-0 data c
r XADD x 3-0 data d
r XDEL x 2-0
# Now the latest tombstone(2-0) is before the first entry(3-0), but there is still
# a tombstone(2-0) after the last_id(1-0) of the consume group.
set reply [r XINFO STREAM x FULL]
set group [lindex [dict get $reply groups] 0]
assert_equal [dict get $group entries-read] 1
assert_equal [dict get $group lag] {}
r XDEL x 1-0
# Although there is a tombstone(2-0) after the consumer group's last_id(1-0), all
# entries before the maximal tombstone have been deleted. This means that both the
# last_id and the largest tombstone are behind the first entry. Therefore, tombstones
# no longer affect the lag, which now reflects the remaining entries in the stream.
set reply [r XINFO STREAM x FULL]
set group [lindex [dict get $reply groups] 0]
assert_equal [dict get $group entries-read] 1
assert_equal [dict get $group lag] 1
# Now there is a tombstone(2-0) after the last_id of the consume group, so after consuming
# entry(3-0), the group's counter will be invalid.
r XREADGROUP GROUP g1 alice STREAMS x >
set reply [r XINFO STREAM x FULL]
set group [lindex [dict get $reply groups] 0]
assert_equal [dict get $group entries-read] 3
assert_equal [dict get $group lag] 0
}
test {Consumer group lag with XTRIM} {
r DEL x
r XGROUP CREATE x mygroup $ MKSTREAM
r XADD x 1-0 data a
r XADD x 2-0 data b
r XADD x 3-0 data c
r XADD x 4-0 data d
r XADD x 5-0 data e
r XREADGROUP GROUP mygroup alice COUNT 1 STREAMS x >
set reply [r XINFO STREAM x FULL]
set group [lindex [dict get $reply groups] 0]
assert_equal [dict get $group entries-read] 1
assert_equal [dict get $group lag] 4
# Although XTRIM doesn't update the `max-deleted-entry-id`, it always updates the
# position of the first entry. When trimming causes the first entry to be behind
# the consumer group's last_id, the consumer group's lag will always be equal to
# the number of remainin entries in the stream.
r XTRIM x MAXLEN 1
set reply [r XINFO STREAM x FULL]
set group [lindex [dict get $reply groups] 0]
assert_equal [dict get $reply max-deleted-entry-id] "0-0"
assert_equal [dict get $group entries-read] 1
assert_equal [dict get $group lag] 1
# When all the entries were deleted, the lag is always 0.
r XTRIM x MAXLEN 0
set reply [r XINFO STREAM x FULL]
set group [lindex [dict get $reply groups] 0]
assert_equal [dict get $group lag] 0
}
test {Loading from legacy (Redis <= v6.2.x, rdb_ver < 10) persistence} {
# The payload was DUMPed from a v5 instance after:
# XADD x 1-0 data a
+34
View File
@@ -1942,6 +1942,34 @@ start_server {tags {"zset"}} {
}
}
test {BZPOPMIN unblock but the key is expired and then block again - reprocessing command} {
r flushall
r debug set-active-expire 0
set rd [redis_deferring_client]
set start [clock milliseconds]
$rd bzpopmin zset{t} 1
wait_for_blocked_clients_count 1
# The exec will try to awake the blocked client, but the key is expired,
# so the client will be blocked again during the command reprocessing.
r multi
r zadd zset{t} 1 one
r pexpire zset{t} 100
r debug sleep 0.2
r exec
assert_equal {} [$rd read]
set end [clock milliseconds]
# Before the fix in #13004, this time would have been 1200+ (i.e. more than 1200ms),
# now it should be 1000, but in order to avoid timing issues, we increase the range a bit.
assert_range [expr $end-$start] 1000 1150
r debug set-active-expire 1
$rd close
} {0} {needs:debug}
test "BZPOPMIN with same key multiple times should work" {
set rd [redis_deferring_client]
r del z1{t} z2{t}
@@ -2211,12 +2239,18 @@ start_server {tags {"zset"}} {
} {b 2 c 3}
test {ZRANGESTORE BYLEX} {
set res [r zrangestore z3{t} z1{t} \[b \[c BYLEX]
assert_equal $res 2
assert_encoding listpack z3{t}
set res [r zrangestore z2{t} z1{t} \[b \[c BYLEX]
assert_equal $res 2
r zrange z2{t} 0 -1 withscores
} {b 2 c 3}
test {ZRANGESTORE BYSCORE} {
set res [r zrangestore z4{t} z1{t} 1 2 BYSCORE]
assert_equal $res 2
assert_encoding listpack z4{t}
set res [r zrangestore z2{t} z1{t} 1 2 BYSCORE]
assert_equal $res 2
r zrange z2{t} 0 -1 withscores
+31
View File
@@ -140,6 +140,37 @@ tags {"wait aof network external:skip"} {
assert_error {ERR WAITAOF cannot be used when numlocal is set but appendonly is disabled.} {$master waitaof 1 0 0}
}
test {WAITAOF local if AOFRW was postponed} {
r config set appendfsync everysec
# turn off AOF
r config set appendonly no
# create an RDB child that takes a lot of time to run
r set x y
r config set rdb-key-save-delay 100000000 ;# 100 seconds
r bgsave
assert_equal [s rdb_bgsave_in_progress] 1
# turn on AOF
r config set appendonly yes
assert_equal [s aof_rewrite_scheduled] 1
# create a write command (to increment master_repl_offset)
r set x y
# reset save_delay and kill RDB child
r config set rdb-key-save-delay 0
catch {exec kill -9 [get_child_pid 0]}
# wait for AOF (will unblock after AOFRW finishes)
assert_equal [r waitaof 1 0 10000] {1 0}
# make sure AOFRW finished
assert_equal [s aof_rewrite_in_progress] 0
assert_equal [s aof_rewrite_scheduled] 0
}
$master config set appendonly yes
waitForBgrewriteaof $master