Compare commits

...
Author SHA1 Message Date
YaacovHazan 095c131fbb Redis 8.0 M04 2025-03-16 10:12:40 +02:00
YaacovHazan 89eef40ca2 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-03-16 10:01:10 +02:00
kei-nanandGitHub 752576ce47 Use Search v7.99.5 (#13859) 2025-03-16 10:00:51 +02:00
YaacovHazanandGitHub 84471e238e Redis 8.0 RC1 (#13851)
- Merge latest unstable
- Update version and release notes
2025-03-11 20:16:27 +02:00
YaacovHazanandYaacovHazan d1df881ec5 Redis 8.0 RC1 2025-03-11 10:03:17 +02:00
YaacovHazan 53949521de Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-03-11 09:39:04 +02:00
Eran HadadandGitHub b704179f15 Update release of RedisJSON, RedisTS and RedisBloom 7.99.4 (#13850) 2025-03-11 09:36:28 +02:00
DvirDukhanandGitHub 557e0b1c07 Update Makefile with search 7.99.4 (#13848) 2025-03-09 13:55:27 +02:00
YaacovHazan a39ffc1fe9 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-03-09 10:08:03 +02:00
f364dcca2d Make RM_DefragRedisModuleDict API support incremental defragmentation for dict leaf (#13840)
After https://github.com/redis/redis/pull/13816, we make a new API to
defrag RedisModuleDict.
Currently, we only support incremental defragmentation of the dictionary
itself, but the defragmentation of values is still not incremental. If
the values are very large, it could lead to significant blocking.
Therefore, in this PR, we have added incremental defragmentation for the
values.

The main change is to the `RedisModuleDefragDictValueCallback`, we
modified the return value of this callback.
When the callback returns 1, we will save the `seekTo` as the key of the
current unfinished node, and the next time we enter, we will continue
defragmenting this node.
When the return value is 0, we will proceed to the next node.

## Test
Since each dictionary in the global dict originally contained only 10
strings, but now it has been changed to a nested dictionary, each
dictionary now has 10 sub-dictionaries, with each sub-dictionary
containing 10 strings, this has led to a corresponding reduction in the
defragmentation time obtained from other tests.
Therefore, the other tests have been modified to always wait for
defragmentation to be turned off before the test begins, then start it
after creating fragmentation, ensuring that they can always run for a
full defragmentation cycle.

---------

Co-authored-by: ephraimfeldblum <ephraim.feldblum@redis.com>
2025-03-04 17:19:41 +08:00
YaacovHazanandGitHub cb261828bd Merge unstable into 8.0 (#13835)
preparing for 8.0 RC1
2025-02-27 08:29:01 +02:00
YaacovHazan 9265234299 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-02-26 21:23:36 +02:00
7939ba031d Enable the callback to be NULL for RM_DefragRedisModuleDict() and reduce the system calls of RM_DefragShouldStop() (#13830)
1) Enable the callback to be NULL for RM_DefragRedisModuleDict()
    Because the dictionary may store only the key without the value.

2) Reduce the system calls of RM_DefragShouldStop()
The API checks the following thresholds before performing a time check:
over 512 defrag hits, or over 1024 defrag misses, and performs the time
judgment if any of these thresholds are reached.

3) Added defragmentation statistics for dictionary items to cover the
associated code for RM_DefragRedisModuleDict().

4) Removed `module_ctx` from `defragModuleCtx` struct, which can be
replaced by a temporary variable.

---------

Co-authored-by: oranagra <oran@redislabs.com>
2025-02-26 20:04:29 +08:00
Yuan WangandGitHub f1d6542b1a Stabilize tcl test cases (#13829)
Recently encountered some errors as bellow,

HGETEX/HSETEX with PXAT/EXAT options, after getting ttl, we calculate
current time by `[clock seconds]` that may have a delay that causes
results greater than expected.

Dismiss memory test error, now we introduced rdb-channel replication,
the full synchronization might finish before the child process exits. So
we may fail if calling `bgsave` immediately after full sync.
2025-02-25 16:31:53 +08:00
33f03f6fc8 Fix wrong behavior of XREAD + after last entry of stream have been removed (#13632)
Close #13628

This PR changes behavior of special `+` id of XREAD command. Now it uses
`streamLastValidID` to find last entry instead of `last_id` field of
stream object.
This PR adds test for the issue.

**Notes**

Initial idea to update `last_id` while executing XDEL seems to be wrong.
`last_id` is used to strore last generated id and not id of last entry.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: guybe7 <guy.benoish@redislabs.com>
2025-02-25 13:40:24 +08:00
985bf68f34 Reduce redundant key slot calculations on expiration checks (#13796)
On high-pipeline/fast commands use-cases, expireIfNeeded can take up to
3% cpu cycles. 

This PR introduces an optimization where key expiration checks leverage
key slots to improve efficiency.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: ShooterIT <wangyuancode@163.com>
2025-02-25 11:55:30 +08:00
Moti CohenandGitHub 0200e8ada6 Fix multiple issues with "INFO KEYSIZES" (#13825)
This commit addresses several issues related to the `INFO KEYSIZES` feature:
- HyperLogLog commands: `KEYSIZES` hooks were not properly set or tested.
- HFE lazy expiration: `KEYSIZES` hooks were not properly set or tested.
- Empty DB & SYNC flow: On `blocking_async=0` flow, global `keysizes`
  histogram were not reset (can reproduced using `DEBUG RELOAD`).
- Empty string handling: Fix histogram for strings of size 0. Not 
  relevant to other data-types.
2025-02-25 00:38:44 +02:00
1848809f66 Optimize dictFind by leveraging key length functions to avoid redundant computations. (#13792)
This PR enhances dictFind by introducing support for key length
functions, allowing the use of keyCompareWithLen when available. This
avoids redundant key length computations, improving efficiency,
especially when the dictionary is rehashing or there are a significant
number of hash collisions.

Additionally, it maintains backward compatibility and optimizes key
lookups without altering existing behavior.

Performance improvement on 100% GETs use-case

benchmark command used
```
taskset -c 1-11 memtier_benchmark --ratio 0:1 --key-maximum 1000000 --key-minimum 1 -c 1 -t 5 --pipeline 100 --key-pattern P:P --test-time 30 --hide-histogram -d 1024 -S /tmp/1.socket  -x 3
```

In unstable dictFindByHash takes 29% (and sdslen within it takes 8.9%)
of CPU cycles for a high-pipeline 100% gets use-case.
After this change dictFindByHash takes 27.8% (and sdslen within it takes
7.7%)

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: Yuan Wang <wangyuancode@163.com>
2025-02-24 22:27:06 +08:00
Filipe Oliveira (Redis)andGitHub d7a448f9ae Avoid redundant calls to sdslen(c->querybuf) in processMultibulkBuffer (#13787)
Optimize processMultibulkBuffer by avoiding redundant calls to
sdslen(c->querybuf).
The cached length is updated only when querybuf is modified.
2025-02-24 20:06:14 +08:00
debing.sunandGitHub 658424fc83 Revert "Update history for ban-list propagation (#13749)" (#13827)
As discussed in
https://github.com/redis/redis/pull/13749#issuecomment-2673612941.
After #10398 we should record only the arguments and output changes in
the command history, while placing all others in the redis-doc, so
revert #13749.
2025-02-24 17:40:25 +08:00
3f06ddfb7b Reuse lookupCommand data on consecutive same command calls on main thread (#13764)
We can see that on fast commands and fast pipeline use-cases,
lookupCommand() takes 1.9% to 3.4% of total cpu cyles (depending on
pipeline). In cases in which consecutives commands are the same we can
avoid the call to lookupCommand() completely without changing or adding
new fields to the client struct (we simply reuse the info already
avaiable in lastcmd). This change can represent an improvement of around
4.4% in QPS on the high pipeline use-cases.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2025-02-24 12:33:14 +08:00
YaacovHazanandGitHub 1cddd0d3ba Merge unstable into 8.0 (#13824) 2025-02-23 08:27:33 +02:00
ee933d9e2b Fixed passing incorrect endtime value for module context (#13822)
1) Fix a bug that passing an incorrect endtime to module.
   This bug was found by @ShooterIT.
After #13814, all endtime will be monotonic time, and we should no
longer convert it to ustime relative.
Add assertions to prevent endtime from being much larger thatn the
current time.

2) Fix a race in test `Reduce defrag CPU usage when module data can't be
defragged`

---------

Co-authored-by: ShooterIT <wangyuancode@163.com>
2025-02-23 12:58:48 +08:00
YaacovHazan 1d5e13e121 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-02-22 21:48:15 +02:00
032357ec0f Add RM_DefragRedisModuleDict module API (#13816)
After #13815, we introduced incremental defragmentation for global data
for module.
Now we added a new module API `RM_DefragRedisModuleDict` to incremental
defrag `RedisModuleDict`.

This PR adds a new APIs and a new defrag callback:
```c
RedisModuleDict *RM_DefragRedisModuleDict(RedisModuleDefragCtx *ctx, RedisModuleDict *dict, RedisModuleDefragDictValueCallback valueCB, RedisModuleString **seekTo);

typedef void *(*RedisModuleDefragDictValueCallback)(RedisModuleDefragCtx *ctx, void *data, unsigned char *key, size_t keylen);
```

Usage:
```c
RedisModuleString *seekTo = NULL;
RedisModuleDict *dict = = RedisModule_CreateDict(ctx);
... populate the dict code ...
/* Defragment a dictionary completely */
do {
    RedisModuleDict *new = RedisModule_DefragRedisModuleDict(ctx, dict, defragGlobalDictValueCB, &seekTo);
    if (new != NULL) {
        dict = new;
    }
} while (seekTo);
```

---------

Co-authored-by: ShooterIT <wangyuancode@163.com>
Co-authored-by: oranagra <oran@redislabs.com>
2025-02-20 21:09:29 +08:00
695126ccce Add support for incremental defragmentation of global module data (#13815)
## Description

Currently, when performing defragmentation on non-key data within the
module, we cannot process the defragmentation incrementally. This
limitation affects the efficiency and flexibility of defragmentation in
certain scenarios.
The primary goal of this PR is to introduce support for incremental
defragmentation of global module data.

## Interface Change
New module API `RegisterDefragFunc2`

This is a more advanced version of `RM_RegisterDefragFunc`, in that it
takes a new callbacks(`RegisterDefragFunc2`) that has a return value,
and can use RM_DefragShouldStop in and indicate that it should be called
again later, or is it done (returned 0).

## Note
The `RegisterDefragFunc` API remains available.

---------

Co-authored-by: ShooterIT <wangyuancode@163.com>
Co-authored-by: oranagra <oran@redislabs.com>
2025-02-20 00:28:16 +08:00
debing.sunGitHubMadelyn Olson madelyneolson@gmail.comShooterIT
725cd268e6 Refactor of ActiveDefrag to reduce latencies (#13814)
This PR is based on: https://github.com/valkey-io/valkey/pull/1462

## Issue/Problems

Duty Cycle: Active Defrag has configuration values which determine the
intended percentage of CPU to be used based on a gradient of the
fragmentation percentage. However, Active Defrag performs its work on
the 100ms serverCron timer. It then computes a duty cycle and performs a
single long cycle. For example, if the intended CPU is computed to be
10%, Active Defrag will perform 10ms of work on this 100ms timer cron.

* This type of cycle introduces large latencies on the client (up to
25ms with default configurations)
* This mechanism is subject to starvation when slow commands delay the
serverCron

Maintainability: The current Active Defrag code is difficult to read &
maintain. Refactoring of the high level control mechanisms and functions
will allow us to more seamlessly adapt to new defragmentation needs.
Specific examples include:

* A single function (activeDefragCycle) includes the logic to
start/stop/modify the defragmentation as well as performing one "step"
of the defragmentation. This should be separated out, so that the actual
defrag activity can be performed on an independent timer (see duty cycle
above).
* The code is focused on kvstores, with other actions just thrown in at
the end (defragOtherGlobals). There's no mechanism to break this up to
reduce latencies.
* For the main dictionary (only), there is a mechanism to set aside
large keys to be processed in a later step. However this code creates a
separate list in each kvstore (main dict or not), bleeding/exposing
internal defrag logic. We only need 1 list - inside defrag. This logic
should be more contained for the main key store.
* The structure is not well suited towards other non-main-dictionary
items. For example, pub-sub and pub-sub-shard was added, but it's added
in such a way that in CMD mode, with multiple DBs, we will defrag
pub-sub repeatedly after each DB.

## Description of the feature

Primarily, this feature will split activeDefragCycle into 2 functions.

1. One function will be called from serverCron to determine if a defrag
cycle (a complete scan) needs to be started. It will also determine if
the CPU expenditure needs to be adjusted.
2. The 2nd function will be a timer proc dedicated to performing defrag.
This will be invoked independently from serverCron.

Once the functions are split, there is more control over the latency
created by the defrag process. A new configuration will be used to
determine the running time for the defrag timer proc. The default for
this will be 500us (one-half of the current minimum time). Then the
timer will be adjusted to achieve the desired CPU. As an example, 5% of
CPU will run the defrag process for 500us every 10ms. This is much
better than running for 5ms every 100ms.

The timer function will also adjust to compensate for starvation. If a
slow command delays the timer, the process will run proportionately
longer to ensure that the configured CPU is achieved. Given the presence
of slow commands, the proportional extra time is insignificant to
latency. This also addresses the overload case. At 100% CPU, if the
event loop slows, defrag will run proportionately longer to achieve the
configured CPU utilization.

Optionally, in low CPU situations, there would be little impact in
utilizing more than the configured CPU. We could optionally allow the
timer to pop more often (even with a 0ms delay) and the (tail) latency
impact would not change.

And we add a time limit for the defrag duty cycle to prevent excessive
latency. When latency is already high (indicated by a long time between
calls), we don't want to make it worse by running defrag for too long.

Addressing maintainability:

* The basic code structure can more clearly be organized around a
"cycle".
* Have clear begin/end functions and a set of "stages" to be executed.
* Rather than stages being limited to "kvstore" type data, a cycle
should be more flexible, incorporating the ability to incrementally
perform arbitrary work. This will likely be necessary in the future for
certain module types. It can be used today to address oddballs like
defragOtherGlobals.
* We reduced some of the globals, and reduce some of the coupling.
defrag_later should be removed from serverDb.
* Each stage should begin on a fresh cycle. So if there are
non-time-bounded operations like kvstoreDictLUTDefrag, these would be
less likely to introduce additional latency.


Signed-off-by: Jim Brunner
[brunnerj@amazon.com](mailto:brunnerj@amazon.com)
Signed-off-by: Madelyn Olson
[madelyneolson@gmail.com](mailto:madelyneolson@gmail.com)
Co-authored-by: Madelyn Olson
[madelyneolson@gmail.com](mailto:madelyneolson@gmail.com)

---------

Signed-off-by: Jim Brunner brunnerj@amazon.com
Signed-off-by: Madelyn Olson madelyneolson@gmail.com
Co-authored-by: Madelyn Olson madelyneolson@gmail.com
Co-authored-by: ShooterIT <wangyuancode@163.com>
2025-02-20 00:05:24 +08:00
guybe7andGitHub 66df58f961 Do not send NL if replica client is already closed (#13813)
In case a replica connection was closed mid-RDB, we should not send a \n
to that replica, otherwise, it may reach the replica BEFORE it realizes
that the RDB transfer failed, causing it to treat the \n as if it was
read from the RDB stream
2025-02-19 15:04:28 +07:00
luozongle01andGitHub b045fe4e17 Fix overflow on 32-bit systems when calculating idle time for eviction (#13804)
the `dictGetSignedIntegerVal` function should be used here,
because in some cases (especially on 32-bit systems) long may
be 4 bytes, and the ttl time saved in expires is a unix timestamp
(millisecond value), which is more than 4 bytes. In this case, we may
not be able to get the correct idle time, which may cause eviction
disorder, in other words, keys that should be evicted later may be
evicted earlier.
2025-02-19 11:01:15 +08:00
Yunxiao DuandGitHub c5f91abaf7 Fix syntax issue in comments of src/module.c (#13802)
closes https://github.com/redis/redis/issues/13797, just fix syntax
issue in comments instead of real code.
2025-02-19 10:58:14 +08:00
Ozan TezcanandGitHub 6c202f495c Remove DENYOOM flag from hexpire command (#13800)
Remove DENYOOM flag from hexpire / hexpireat / hpexpire / hpexpireat
commands.

h(p)expire(at) commands may allocate some memory but it is not that big.
Similary, we don't have DENYOOM flag for EXPIRE command. This change
will align EXPIRE and HEXPIRE commands in this manner.
2025-02-16 20:07:29 +03:00
Ozan TezcanandGitHub e2608478b6 Add HGETDEL, HGETEX and HSETEX hash commands (#13798)
This PR adds three new hash commands: HGETDEL, HGETEX and HSETEX. These
commands enable user to do multiple operations in one step atomically
e.g. set a hash field and update its TTL with a single command.
Previously, it was only possible to do it by calling hset and hexpire
commands subsequently.

- **HGETDEL command**

  ```
  HGETDEL <key> FIELDS <numfields> field [field ...]
  ```
  
  **Description**  
  Get and delete the value of one or more fields of a given hash key
  
  **Reply**  
Array reply: list of the value associated with each field or nil if the
field doesn’t exist.

- **HGETEX command**

  ```
   HGETEX <key>  
[EX seconds | PX milliseconds | EXAT unix-time-seconds | PXAT
unix-time-milliseconds | PERSIST]
     FIELDS <numfields> field [field ...]
  ```

  **Description**
Get the value of one or more fields of a given hash key, and optionally
set their expiration

  **Options:**
  EX seconds: Set the specified expiration time, in seconds.
  PX milliseconds: Set the specified expiration time, in milliseconds.
EXAT timestamp-seconds: Set the specified Unix time at which the field
will expire, in seconds.
PXAT timestamp-milliseconds: Set the specified Unix time at which the
field will expire, in milliseconds.
  PERSIST: Remove the time to live associated with the field.

  **Reply** 
Array reply: list of the value associated with each field or nil if the
field doesn’t exist.

- **HSETEX command**

  ```
  HSETEX <key>
     [FNX | FXX]
[EX seconds | PX milliseconds | EXAT unix-time-seconds | PXAT
unix-time-milliseconds | KEEPTTL]
     FIELDS <numfields> field value [field value...]
  ```
  **Description**
Set the value of one or more fields of a given hash key, and optionally
set their expiration

  **Options:**
  FNX: Only set the fields if all do not already exist.
  FXX: Only set the fields if all already exist.

  EX seconds: Set the specified expiration time, in seconds.
  PX milliseconds: Set the specified expiration time, in milliseconds.
EXAT timestamp-seconds: Set the specified Unix time at which the field
will expire, in seconds.
PXAT timestamp-milliseconds: Set the specified Unix time at which the
field will expire, in milliseconds.
  KEEPTTL: Retain the time to live associated with the field.

  
Note: If no option is provided, any associated expiration time will be
discarded similar to how SET command behaves.

  **Reply**
  Integer reply: 0 if no fields were set
  Integer reply: 1 if all the fields were set
2025-02-14 17:13:35 +03:00
Ofir LuzonandGitHub 57807cd338 Memory Usage command LIST accuracy fix (#13783)
MEMORY USAGE on a List samples quicklist entries, but does not account
to how many elements are in each sampled node. This can skew the
calculation when the sampled nodes are not balanced.
The fix calculate the average element size in the sampled nodes instead
of the average node size.
2025-02-14 09:18:47 +08:00
Yuan WangandGitHub 7f5f588232 AOF offset info (#13773)
### Background 
AOF is often used as an effective data recovery method, but now if we
have two AOFs from different nodes, it is hard to learn which one has
latest data. Generally, we determine whose data is more up-to-date by
reading the latest modification time of the AOF file, but because of
replication delay, even if both master and replica write to the AOF at
the same time, the data in the master is more up-to-date (there are
commands that didn't arrive at the replica yet, or a large number of
commands have accumulated on replica side ), so we may make wrong
decision.

### Solution
The replication offset always increments when AOF is enabled even if
there is no replica, we think replication offset is better method to
determine which one has more up-to-date data, whoever has a larger
offset will have newer data, so we add the start replication offset info
for AOF, as bellow.
```
file appendonly.aof.2.base.rdb seq 2 type b
file appendonly.aof.2.incr.aof seq 2 type i startoffset 224
```
And if we close gracefully the AOF file, not a crash, such as
`shutdown`, `kill signal 15` or `config set appendonly no`, we will add
the end replication offset, as bellow.
```
file appendonly.aof.2.base.rdb seq 2 type b
file appendonly.aof.2.incr.aof seq 2 type i startoffset 224 endoffset 532
```

#### Things to pay attention to
- For BASE AOF, we do not add `startoffset` and `endoffset` info, since
we could not know the start replication replication of data, and it is
useless to help us to determine which one has more up-to-date data.
- For AOFs from old version, we also don't add `startoffset` and
`endoffset` info, since we also don't know start replication replication
of them. If we add the start offset from 0, we might make the judgment
even less accurate. For example, if the master has just rewritten the
AOF, its INCR AOF will inevitably be very small. However, if the replica
has not rewritten AOF for a long time, its INCR AOF might be much
larger. By applying the following method, we might make incorrect
decisions, so we still just check timestamp instead of adding offset
info
- If the last INCR AOF has `startoffset` or `endoffset`, we need to
restore `server.master_repl_offset` according to them to avoid the
rollback of the `startoffset` of next INCR AOF. If it has `endoffset`,
we just use this value as `server.master_repl_offset`, and a very
important thing is to remove this information from the manifest file to
avoid the next time we load the manifest file with wrong `endoffset`. If
it only has `startoffset`, we calculate `server.master_repl_offset` by
the `startoffset` plus the file size.

### How to determine which one has more up-to-date data
If AOF has a larger replication offset, it will have more up-to-date
data. The following is how to get AOF offset:

Read the AOF manifest file to obtain information about **the last INCR
AOF**
1. If the last INCR AOF has `endoffset` field, we can directly use the
`endoffset` to present the replication offset of AOF
2. If there is no `endoffset`(such as redis crashes abnormally), but
there is `startoffset` filed of the last INCR AOF, we can get the
replication offset of AOF by `startoffset` plus the file size
3. Finally, if the AOF doesn’t have both `startoffset` and `endoffset`,
maybe from old version, and new version redis has not rewritten AOF yet,
we still need to check the modification timestamp of the last INCR AOF

### TODO
Fix ping causing inconsistency between AOF size and replication
offset in the future PR. Because we increment the replication offset
when sending PING/REPLCONF to the replica but do not write data to the
AOF file, this might cause the starting offset of the AOF file plus its
size to be inconsistent with the actual replication offset.
2025-02-13 17:31:40 +08:00
Yuan WangandGitHub 662cb2fe75 Don't send unnecessary PING to replicas (#13790)
The reason why master sends PING is to keep the connection with replica
active, so master need not send PING to replicas if already sent
replication stream in the past `repl_ping_slave_period` time.

Now master only sends PINGs and increases `master_repl_offset` if there
is no traffic, so this PR also can reduce the impact of issue in
https://github.com/redis/redis/pull/13773, of course, does not resolve
it completely.
> Fix ping causing inconsistency between AOF size and replication offset
in the future PR. Because we increment the replication offset when
sending PING/REPLCONF to the replica but do not write data to the AOF
file, this might cause the starting offset of the AOF file plus its size
to be inconsistent with the actual replication offset.
2025-02-13 10:52:19 +08:00
Yuan WangandGitHub 87124a38b6 Fix wrongly updating fsynced_reploff_pending when appendfsync=everysecond (#13793)
```
if (server.aof_fsync == AOF_FSYNC_EVERYSEC &&
    server.aof_last_incr_fsync_offset != server.aof_last_incr_size &&
    server.mstime - server.aof_last_fsync >= 1000 &&
    !(sync_in_progress = aofFsyncInProgress())) {
    goto try_fsync;
```
In https://github.com/redis/redis/pull/12622, when when
appendfsync=everysecond, if redis has written some data to AOF but not
`fsync`, and less than 1 second has passed since the last `fsync `,
redis will won't fsync AOF, but we will update `
fsynced_reploff_pending`, so it cause the `WAITAOF` to return
prematurely.

this bug is introduced in https://github.com/redis/redis/pull/12622,
from 7.4

The bug fix
https://github.com/redis/redis/pull/13793/commits/1bd6688bcae4add51dc829d96776359dfa39b100
is just as follows:
```diff
diff --git a/src/aof.c b/src/aof.c
index 8ccd8d8f8..521b30449 100644
--- a/src/aof.c
+++ b/src/aof.c
@@ -1096,8 +1096,11 @@ void flushAppendOnlyFile(int force) {
              * in which case master_repl_offset will increase but fsynced_reploff_pending won't be updated
              * (because there's no reason, from the AOF POV, to call fsync) and then WAITAOF may wait on
              * the higher offset (which contains data that was only propagated to replicas, and not to AOF) */
-            if (!sync_in_progress && server.aof_fsync != AOF_FSYNC_NO)
+            if (server.aof_last_incr_fsync_offset == server.aof_last_incr_size &&
+                !(sync_in_progress = aofFsyncInProgress()))
+            {
                 atomicSet(server.fsynced_reploff_pending, server.master_repl_offset);
+            }
             return;
```
Additionally, we slightly refactored fsync AOF to make it simpler, as
https://github.com/redis/redis/pull/13793/commits/584f008d1c39b90ae1d4862a313e04e3b426b136
2025-02-13 10:48:29 +08:00
Yves LeBrasandGitHub 1583d60cd6 Missing --memkeys and --keystats for some options in redis-cli help text (#13794)
Help text modified for -i, --count, --pattern.
2025-02-13 08:42:38 +08:00
1cd622bdca Add an API to load default configuration values (#13788)
Currently we have RedisModule_LoadConfigs which the module is expected
to call during OnLoad which sets the configuration values from the
config queue or it sets the default value.

The problem is that the module might still want to support loading
values from the command line. If we want to give precedence to the
config file values then it means the module needs to set the values
before calling the Load Config function.

The problem is that then the API overrides the variables which were set
from the module command line with default values.

The new API should solve that in the following way.

1.Module registers its configuration parameters with redis 2.Module
calls RedisModule_LoadDefaultConfigs which loads the default values for
all the registered configuration parameters of the module 3.Module sets
the variables internally using the values it got from the command line
4.Module calls RedisModule_LoadConfigs which will set the values based
on the redis configuration file.

This allows for the default values to be set, for the module to override
them and for redis to override what the module wrote. In short it
determines a logical flow and ordering of where the values for the
parameters should come from.

The change done by all these previous commits:
d9134f8f9
7a40fd630
b9361ad5f
83c034855
f164012c1
98be450f1
0f6e3a827
de4e92ac3
49455c43a
c2694fb69
c88f9fe26
855ec46a6
f7353db7e
294492dbf
192799539
a8850a8d3
f35ad8231
a03477349
fd5c32588

Co-authored-by: YaacovHazan <yaacov.hazan@redislabs.com>
2025-02-06 13:38:07 +02:00
d9134f8f95 Update tests/modules/moduleconfigs.c
missing else clause

Co-authored-by: debing.sun <debing.sun@redis.com>
2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan 7a40fd630d * fix comments 2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan b9361ad5fe * only use new api if override-default was provided as an argument 2025-02-06 13:16:33 +02:00
83c0348553 Apply suggestions from code review
* apply comment suggestions

Co-authored-by: Oran Agra <oran@redislabs.com>
2025-02-06 13:16:33 +02:00
f164012c19 Update tests/unit/moduleapi/moduleconfigs.tcl
Co-authored-by: nafraf <nafraf@users.noreply.github.com>
2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan 98be450f1d * fix typo 2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan 0f6e3a8273 * improve function documentation 2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan de4e92ac39 * addressing code review comments 2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan 49455c43ae * change foo to goo so test will be correct and pass 2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan c2694fb696 * change config value in test to be different than overwritten value 2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan c88f9fe26f * update comment 2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan 855ec46a6a * rename MODULE_ONLOAD_CONFIG to MODULE_NON_DEFAULT_CONFIG 2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan f7353db7eb * fix test
* cleanup strval2 on if an error during the OnLoad was encountered.
2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan 294492dbf2 * fix tests
* add some logging to test module
2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan 192799539f * register LoadDefaultConfigs 2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan a8850a8d30 * remove unused variable 2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan f35ad82314 * add missing newline 2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan a034773497 * remove redundant module config variable 2025-02-06 13:16:33 +02:00
jonathan keinanandYaacovHazan fd5c325886 * initial commit 2025-02-06 13:16:33 +02:00
Alexander GorbulyaandGitHub 17eb33e0c3 Fix typo in repl-ping-replica-period comment in redis.conf (#13782)
The comment for the `repl-ping-replica-period` option in `redis.conf`
mistakenly refers to `repl_ping_replica_period` (with underscores).
This PR corrects it to use the proper format with dashes, as per the
actual configuration option.
2025-02-06 10:14:56 +08:00
YaacovHazanandYaacovHazan 0aeb86d78d Revert "Improve GETRANGE command behavior (#12272)"
Although the commit #6ceadfb58 improves GETRANGE command behavior,
we can't accept it as we should avoid breaking changes for non-critical bug fixes.

This reverts commit 6ceadfb580.
2025-02-05 20:49:42 +02:00
YaacovHazanandYaacovHazan 8afb72a326 Revert "improve performance for scan command when matching data type (#12395)"
Although the commit #7f0a7f0a6 improves the performance of the SCAN command,
we can't accept it as we should avoid breaking changes for non-critical bug fixes.

This reverts commit 7f0a7f0a69.
2025-02-05 20:49:42 +02:00
04589f90d7 Add internal connection and command mechanism (#13740)
# PR: Add Mechanism for Internal Commands and Connections in Redis

This PR introduces a mechanism to handle **internal commands and
connections** in Redis. It includes enhancements for command
registration, internal authentication, and observability.

## Key Features

1. **Internal Command Flag**:
   - Introduced a new **module command registration flag**: `internal`.
- Commands marked with `internal` can only be executed by **internal
connections**, AOF loading flows, and master-replica connections.
- For any other connection, these commands will appear as non-existent.

2. **Support for internal authentication added to `AUTH`**:
- Used by depicting the special username `internal connection` with the
right internal password, i.e.,: `AUTH "internal connection"
<internal_secret>`.
- No user-defined ACL username can have this name, since spaces are not
aloud in the ACL parser.
   - Allows connections to authenticate as **internal connections**.
- Authenticated internal connections can execute internal commands
successfully.

4. **Module API for Internal Secret**:
- Added the `RedisModule_GetInternalSecret()` API, that exposes the
internal secret that should be used as the password for the new `AUTH
"internal connection" <password>` command.
- This API enables the modules to authenticate against other shards as
local connections.

## Notes on Behavior

- **ACL validation**:
- Commands dispatched by internal connections bypass ACL validation, to
give the caller full access regardless of the user with which it is
connected.

- **Command Visibility**:
- Internal commands **do not appear** in `COMMAND <subcommand>` and
`MONITOR` for non-internal connections.
- Internal commands **are logged** in the slow log, latency report and
commands' statistics to maintain observability.

- **`RM_Call()` Updates**:
  - **Non-internal connections**:
- Cannot execute internal commands when the command is sent with the `C`
flag (otherwise can).
- Internal connections bypass ACL validations (i.e., run as the
unrestricted user).

- **Internal commands' success**:
- Internal commands succeed upon being sent from either an internal
connection (i.e., authenticated via the new `AUTH "internal connection"
<internal_secret>` API), an AOF loading process, or from a master via
the replication link.
Any other connections that attempt to execute an internal command fail
with the `unknown command` error message raised.

- **`CLIENT LIST` flags**:
  - Added the `I` flag, to indicate that the connection is internal.

- **Lua Scripts**:
   - Prevented internal commands from being executed via Lua scripts.

---------

Co-authored-by: Meir Shpilraien <meir@redis.com>
2025-02-05 11:48:08 +02:00
Ozan TezcanandGitHub 09f8a2f374 Start AOFRW before streaming repl buffer during fullsync (#13758)
During fullsync, before loading RDB on the replica, we stop aof child to
prevent copy-on-write disaster.
Once rdb is loaded, aof is started again and it will trigger aof
rewrite. With https://github.com/redis/redis/pull/13732 , for rdbchannel
replication, this behavior was changed. Currently, we start aof after
replication buffer is streamed to db. This PR changes it back to start
aof just after rdb is loaded (before repl buffer is streamed)

Both approaches may have pros and cons. If we start aof before streaming
repl buffers, we may still face with copy-on-write issues as repl
buffers potentially include large amount of changes. If we wait until
replication buffer drained, it means we are delaying starting aof
persistence.

Additional changes are introduced as part of this PR:

- Interface change:
Added `mem_replica_full_sync_buffer` field to the `INFO MEMORY` command
reply. During full sync, it shows total memory consumed by accumulated
replication stream buffer on replica. Added same metric to `MEMORY
STATS` command reply as `replica.fullsync.buffer` field.
  
  
- Fixes: 
- Count repl stream buffer size of replica as part of 'memory overhead'
calculation for fields in "INFO MEMORY" and "MEMORY STATS" outputs.
Before this PR, repl buffer was not counted as part of memory overhead
calculation, causing misreports for fields like `used_memory_overhead`
and `used_memory_dataset` in "INFO STATS" and for `overhead.total` field
in "MEMORY STATS" command reply.
- Dismiss replication stream buffers memory of replica in the fork to
reduce COW impact during a fork.
- Fixed a few time sensitive flaky tests, deleted a noop statement,
fixed some comments and fail messages in rdbchannel tests.
2025-02-04 21:40:18 +03:00
Meir Shpilraien (Spielrein)andGitHub 870b6bd487 Added a shared secret over Redis cluster. (#13763)
The PR introduces a new shared secret that is shared over all the nodes
on the Redis cluster. The main idea is to leverage the cluster bus to
share a secret between all the nodes such that later the nodes will be
able to authenticate using this secret and send internal commands to
each other (see #13740 for more information about internal commands).

The way the shared secret is chosen is the following:
1. Each node, when start, randomly generate its own internal secret.
2. Each node share its internal secret over the cluster ping messages.
3. If a node gets a ping message with secret smaller then his current
secret, it embrace it.
4. Eventually all nodes should embrace the minimal secret

The converges of the secret is as good as the topology converges.

To extend the ping messages to contain the secret, we leverage the
extension mechanism. Nodes that runs an older Redis version will just
ignore those extensions.

Specific tests were added to verify that eventually all nodes see the
secrets. In addition, a verification was added to the test infra to
verify the secret on `cluster_config_consistent` and to
`assert_cluster_state`.
2025-02-03 09:54:37 +02:00
Raz MonsonegoandGitHub c688537d49 Add flag for ability of a module context to execute debug commands (#13774)
This PR adds a flag to the `RM_GetContextFlags` module-API function that
depicts whether the context may execute debug commands, according to
redis's standards.
2025-02-03 09:52:41 +02:00
Mingyi KangandGitHub e3b9397dfe Bump actions/upload-artifact from 3 to 4 (#13780)
Update `upload-artifact` from v3 to v4 to avoid the failure of `External
Server Tests` (I encountered this error when opening
[#13779](https://github.com/redis/redis/pull/13779)):

> Error: This request has been automatically failed because it uses a
deprecated version of `actions/upload-artifact: v3`. Learn more:
https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/
2025-02-01 14:09:00 +08:00
Moti CohenandGitHub 2bfffe85e9 Fix memleak of SFLUSH experimental command (#13766)
On flushallSyncBgDone, if client doesn't exists, take care release
sflush struct.
2025-01-30 13:35:02 +02:00
f5e046a730 Update history for ban-list propagation (#13749)
Update CLUSTER FORGET docs for changes in
https://github.com/redis/redis/pull/10869

Docs PR:
https://github.com/redis/docs/pull/1057

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2025-01-27 21:05:37 +08:00
5dbcb3e4ab Add codecov for automated code coverage (#13393)
This PR introduces Codecov to automate code coverage tracking for our
project's tests.
For more information about the Codecov platform, please refer to
https://docs.codecov.com/docs/quick-start

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2025-01-27 21:04:11 +08:00
Dustin RodriguesandGitHub eb50eb20a5 Fix compile error with TLS_DEBUGGING (#13772) 2025-01-26 10:14:43 +08:00
YaacovHazanandGitHub c2d3e28540 Update RQE version 7.99.3 (#13767) (#13769)
Update RQE version 7.99.3
2025-01-24 09:20:23 +02:00
YaacovHazan 0c1a764074 Merge commit 'dcd0b3d02' into HEAD 2025-01-24 09:15:59 +02:00
f86575f210 Gradually reduce defrag CPU usage when defragmentation is ineffective (#13752)
This PR addresses an issue where if a module does not provide a
defragmentation callback, we cannot defragment the fragmentation it
generates. However, the defragmentation process still considers a large
amount of fragmentation to be present, leading to more aggressive
defragmentation efforts that ultimately have no effect.

To mitigate this, the PR introduces a mechanism to gradually reduce the
CPU consumption for defragmentation when the defragmentation
effectiveness is poor. This occurs when the fragmentation rate drops
below 2% and the hit ratio is less than 1%, or when the fragmentation
rate increases by no more than 2%. The CPU consumption will be gradually
decreased until it reaches the minimum threshold defined by
`active-defrag-cycle-min`.

---------

Co-authored-by: oranagra <oran@redislabs.com>
2025-01-24 11:35:32 +08:00
nafrafandGitHub dcd0b3d020 Update RQE version 7.99.3 (#13767)
Update RQE version 7.99.3
2025-01-23 22:46:50 +02:00
YaacovHazanandGitHub d88611d36f Redis 8.0 M03 (#13759) 2025-01-20 19:45:10 +02:00
YaacovHazan 4c5a9076d7 Redis 8.0 M03 2025-01-20 16:00:49 +02:00
YaacovHazan 8aab7ca84c Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-01-20 11:06:49 +02:00
781ccc1bee Update modules with latest version (#13755)
Update redisbloom, redisjson and redistimeseries versions to 7.99.2

Co-authored-by: YaacovHazan <yaacov.hazan@redislabs.com>
2025-01-20 10:08:19 +02:00
DvirDukhanandGitHub ee96a5a6f5 Update RQE version (#13750)
v7.99.2
2025-01-16 08:40:19 +02:00
YaacovHazan 9c81f8bd61 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-01-14 14:01:19 +02:00
debing.sunandGitHub 0f65806b5b Update info.tcl test to revert client output limits sooner (#13738)
This PR is based on: https://github.com/valkey-io/valkey/pull/1462

We set the client output buffer limits to 10 bytes, and then execute
info stats which produces more than 10 bytes of output, which can cause
that command to throw an error.

I'm not sure why it wasn't consistently erroring before, might have been
some change related to the ubuntu upgrade though.

failed CI:
https://github.com/redis/redis/actions/runs/12738281720/job/35500381299

------
Co-authored-by: Madelyn Olson
[madelyneolson@gmail.com](mailto:madelyneolson@gmail.com)
2025-01-14 17:30:18 +08:00
Yuan WangandGitHub 5b8b58e472 Fix incorrect parameter type reports (#13744)
After upgrading of ubuntu 24.04, clang18 can check runtime error: call
to function XXX through pointer to incorrect function type, our daily CI
reports the errors by UndefinedBehaviorSanitizer (UBSan):

https://github.com/redis/redis/actions/runs/12738281720/job/35500380251#step:6:346

now we add generic version of some existing `free` functions to support
to call function through (void*) pointer, actually, they just are the
wrapper functions that will cast the data type and call the
corresponding functions.
2025-01-14 15:51:05 +08:00
YaacovHazanandYaacovHazan 342ee426ad Fix LUA garbage collector (CVE-2024-46981)
Reset GC state before closing the lua VM to prevent user data
to be wrongly freed while still might be used on destructor callbacks.
2025-01-13 21:20:19 +02:00
YaacovHazanandYaacovHazan 4a95b3005a Fix Read/Write key pattern selector (CVE-2024-51741)
The '%' rule must contain one or both of R/W
2025-01-13 21:20:19 +02:00
+4 73a9b916c9 Rdb channel replication (#13732)
This PR is based on:

https://github.com/redis/redis/pull/12109
https://github.com/valkey-io/valkey/pull/60

Closes: https://github.com/redis/redis/issues/11678

**Motivation**

During a full sync, when master is delivering RDB to the replica,
incoming write commands are kept in a replication buffer in order to be
sent to the replica once RDB delivery is completed. If RDB delivery
takes a long time, it might create memory pressure on master. Also, once
a replica connection accumulates replication data which is larger than
output buffer limits, master will kill replica connection. This may
cause a replication failure.

The main benefit of the rdb channel replication is streaming incoming
commands in parallel to the RDB delivery. This approach shifts
replication stream buffering to the replica and reduces load on master.
We do this by opening another connection for RDB delivery. The main
channel on replica will be receiving replication stream while rdb
channel is receiving the RDB.

This feature also helps to reduce master's main process CPU load. By
opening a dedicated connection for the RDB transfer, the bgsave process
has access to the new connection and it will stream RDB directly to the
replicas. Before this change, due to TLS connection restriction, the
bgsave process was writing RDB bytes to a pipe and the main process was
forwarding
it to the replica. This is no longer necessary, the main process can
avoid these expensive socket read/write syscalls. It also means RDB
delivery to replica will be faster as it avoids this step.

In summary, replication will be faster and master's performance during
full syncs will improve.


**Implementation steps**

1. When replica connects to the master, it sends 'rdb-channel-repl' as
part of capability exchange to let master to know replica supports rdb
channel.
2. When replica lacks sufficient data for PSYNC, master sends
+RDBCHANNELSYNC reply with replica's client id. As the next step, the
replica opens a new connection (rdb-channel) and configures it against
the master with the appropriate capabilities and requirements. It also
sends given client id back to master over rdbchannel, so that master can
associate these channels. (initial replica connection will be referred
as main-channel) Then, replica requests fullsync using the RDB channel.
3. Prior to forking, master attaches the replica's main channel to the
replication backlog to deliver replication stream starting at the
snapshot end offset.
4. The master main process sends replication stream via the main
channel, while the bgsave process sends the RDB directly to the replica
via the rdb-channel. Replica accumulates replication stream in a local
buffer, while the RDB is being loaded into the memory.
5. Once the replica completes loading the rdb, it drops the rdb channel
and streams the accumulated replication stream into the db. Sync is
completed.

**Some details**
- Currently, rdbchannel replication is supported only if
`repl-diskless-sync` is enabled on master. Otherwise, replication will
happen over a single connection as in before.
- On replica, there is a limit to replication stream buffering. Replica
uses a new config `replica-full-sync-buffer-limit` to limit number of
bytes to accumulate. If it is not set, replica inherits
`client-output-buffer-limit <replica>` hard limit config. If we reach
this limit, replica stops accumulating. This is not a failure scenario
though. Further accumulation will happen on master side. Depending on
the configured limits on master, master may kill the replica connection.

**API changes in INFO output:**

1. New replica state: `send_bulk_and_stream`. Indicates full sync is
still in progress for this replica. It is receiving replication stream
and rdb in parallel.
```
slave0:ip=127.0.0.1,port=5002,state=send_bulk_and_stream,offset=0,lag=0
```
Replica state changes in steps:
- First, replica sends psync and receives +RDBCHANNELSYNC
:`state=wait_bgsave`
- After replica connects with rdbchannel and delivery starts:
`state=send_bulk_and_stream`
 - After full sync: `state=online`

2. On replica side, replication stream buffering metrics:
- replica_full_sync_buffer_size: Currently accumulated replication
stream data in bytes.
- replica_full_sync_buffer_peak: Peak number of bytes that this instance
accumulated in the lifetime of the process.

```
replica_full_sync_buffer_size:20485             
replica_full_sync_buffer_peak:1048560
```

**API changes in CLIENT LIST**

In `client list` output, rdbchannel clients will have 'C' flag in
addition to 'S' replica flag:
```
id=11 addr=127.0.0.1:39108 laddr=127.0.0.1:5001 fd=14 name= age=5 idle=5 flags=SC db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1920 events=r cmd=psync user=default redir=-1 resp=2 lib-name= lib-ver= io-thread=0
```

**Config changes:**
- `replica-full-sync-buffer-limit`: Controls how much replication data
replica can accumulate during rdbchannel replication. If it is not set,
a value of 0 means replica will inherit `client-output-buffer-limit
<replica>` hard limit config to limit accumulated data.
- `repl-rdb-channel` config is added as a hidden config. This is mostly
for testing as we need to support both rdbchannel replication and the
older single connection replication (to keep compatibility with older
versions and rdbchannel replication will not be enabled if
repl-diskless-sync is not enabled). it affects both the master (not to
respond to rdb channel requests), and the replica (not to declare
capability)

**Internal API changes:**
Changes that were introduced to Redis replication:
- New replication capability is added to replconf command: `capa
rdb-channel-repl`. Indicates replica is capable of rdb channel
replication. Replica sends it when it connects to master along with
other capabilities.
- If replica needs fullsync, master replies `+RDBCHANNELSYNC
<client-id>` to the replica's PSYNC request.
- When replica opens rdbchannel connection, as part of replconf command,
it sends `rdb-channel 1` to let master know this is rdb channel. Also,
it sends `main-ch-client-id <client-id>` as part of replconf command so
master can associate channels.
  
**Testing:**
As rdbchannel replication is enabled by default, we run whole test suite
with it. Though, as we need to support both rdbchannel and single
connection replication, we'll be running some tests twice with
`repl-rdb-channel yes/no` config.

**Replica state diagram**
```
* * Replica state machine *
 *
 * Main channel state
 * ┌───────────────────┐
 * │RECEIVE_PING_REPLY │
 * └────────┬──────────┘
 *          │ +PONG
 * ┌────────▼──────────┐
 * │SEND_HANDSHAKE     │                     RDB channel state
 * └────────┬──────────┘            ┌───────────────────────────────┐
 *          │+OK                ┌───► RDB_CH_SEND_HANDSHAKE         │
 * ┌────────▼──────────┐        │   └──────────────┬────────────────┘
 * │RECEIVE_AUTH_REPLY │        │    REPLCONF main-ch-client-id <clientid>
 * └────────┬──────────┘        │   ┌──────────────▼────────────────┐
 *          │+OK                │   │ RDB_CH_RECEIVE_AUTH_REPLY     │
 * ┌────────▼──────────┐        │   └──────────────┬────────────────┘
 * │RECEIVE_PORT_REPLY │        │                  │ +OK
 * └────────┬──────────┘        │   ┌──────────────▼────────────────┐
 *          │+OK                │   │  RDB_CH_RECEIVE_REPLCONF_REPLY│
 * ┌────────▼──────────┐        │   └──────────────┬────────────────┘
 * │RECEIVE_IP_REPLY   │        │                  │ +OK
 * └────────┬──────────┘        │   ┌──────────────▼────────────────┐
 *          │+OK                │   │ RDB_CH_RECEIVE_FULLRESYNC     │
 * ┌────────▼──────────┐        │   └──────────────┬────────────────┘
 * │RECEIVE_CAPA_REPLY │        │                  │+FULLRESYNC
 * └────────┬──────────┘        │                  │Rdb delivery
 *          │                   │   ┌──────────────▼────────────────┐
 * ┌────────▼──────────┐        │   │ RDB_CH_RDB_LOADING            │
 * │SEND_PSYNC         │        │   └──────────────┬────────────────┘
 * └─┬─────────────────┘        │                  │ Done loading
 *   │PSYNC (use cached-master) │                  │
 * ┌─▼─────────────────┐        │                  │
 * │RECEIVE_PSYNC_REPLY│        │    ┌────────────►│ Replica streams replication
 * └─┬─────────────────┘        │    │             │ buffer into memory
 *   │                          │    │             │
 *   │+RDBCHANNELSYNC client-id │    │             │
 *   ├──────┬───────────────────┘    │             │
 *   │      │ Main channel           │             │
 *   │      │ accumulates repl data  │             │
 *   │   ┌──▼────────────────┐       │     ┌───────▼───────────┐
 *   │   │ REPL_TRANSFER     ├───────┘     │    CONNECTED      │
 *   │   └───────────────────┘             └────▲───▲──────────┘
 *   │                                          │   │
 *   │                                          │   │
 *   │  +FULLRESYNC    ┌───────────────────┐    │   │
 *   ├────────────────► REPL_TRANSFER      ├────┘   │
 *   │                 └───────────────────┘        │
 *   │  +CONTINUE                                   │
 *   └──────────────────────────────────────────────┘
 */
 ```
 -----
 This PR also contains changes and ideas from: 
https://github.com/valkey-io/valkey/pull/837
https://github.com/valkey-io/valkey/pull/1173
https://github.com/valkey-io/valkey/pull/804
https://github.com/valkey-io/valkey/pull/945
https://github.com/valkey-io/valkey/pull/989
---------

Co-authored-by: Yuan Wang <wangyuancode@163.com>
Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: Moti Cohen <moticless@gmail.com>
Co-authored-by: naglera <anagler123@gmail.com>
Co-authored-by: Amit Nagler <58042354+naglera@users.noreply.github.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Ping Xie <pingxie@outlook.com>
Co-authored-by: Ran Shidlansik <ranshid@amazon.com>
Co-authored-by: ranshid <88133677+ranshid@users.noreply.github.com>
Co-authored-by: xbasel <103044017+xbasel@users.noreply.github.com>
2025-01-13 15:09:52 +03:00
dc0ee51cb1 Refactor Client Write Preparation and Handling (#13721)
This update refactors prepareClientToWrite by introducing
_prepareClientToWrite for inline checks within network.c file, and
separates replica and non-replica handling for pending replies and
writes (_clientHasPendingRepliesSlave/NonSlave and
_writeToClientSlave/NonSlave).

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: Yuan Wang <wangyuancode@163.com>
2025-01-13 15:40:36 +08:00
21aee83abd Fix issue with argv not being shrunk (#13698)
Found by @ShooterIT 

## Describe
If a client first creates a command with a very large number of
parameters, such as 10,000 parameters, the argv will be expanded to
accommodate 10,000. If the subsequent commands have fewer than 10,000
parameters, this argv will continue to be reused and will never be
shrunk.

## Solution
When determining whether it is necessary to rebuild argv, if the length
of the previous argv has already exceeded 1024, we will progressively
create argv regardless.

## Free argv in cron
Add a new condition to determine whether argv needs to be resized in
cron. When the number of parameters exceeds 128, we will resize it
regardless to avoid a single client consuming too much memory. It will
now occupy a maximum of (128 * 8 bytes).

---------

Co-authored-by: Yuan Wang <wangyuancode@163.com>
2025-01-08 16:12:52 +08:00
08d714d0e5 Fix crash due to cron argv release (#13725)
Introduced by https://github.com/redis/redis/issues/13521

If the client argv was released due to a timeout before sending the
complete command, `argv_len` will be reset to 0.
When argv is parsed again and resized, requesting a length of 0 may
result in argv being NULL, then leading to a crash.

And fix a bug that `argv_len` is not updated correctly in
`replaceClientCommandVector()`.

---------

Co-authored-by: ShooterIT <wangyuancode@163.com>
Co-authored-by: meiravgri <109056284+meiravgri@users.noreply.github.com>
2025-01-08 09:57:23 +08:00
RQfreeflyandGitHub 4a12291765 Fix typos in multiple Redis source files (#13716) 2025-01-07 15:35:47 +08:00
Yuan WangandGitHub 8e9f5146dd Add reads/writes metrics for IO threads (#13703)
The main job of the IO thread is read queries and write replies, so
reads/writes metrics can reflect the workload of IO threads, now we also
support this metrics `io_threaded_reads/writes_processed` in detail for
each IO thread.

Of course, to avoid break changes, `io_threaded_reads/writes_processed`
is still there. But before async io thread commit, we may sum the IO
done by the main thread if IO threads are active, but now we only sum
the IO done by IO threads.

Now threads section in `info` command output is as follows:
```
# Threads
io_thread_0:clients=0,reads=0,writes=0
io_thread_1:clients=54,reads=6546940,writes=6546919
io_thread_2:clients=54,reads=6513650,writes=6513625
io_thread_3:clients=54,reads=6396571,writes=6396525
io_thread_4:clients=53,reads=6511120,writes=6511097
io_thread_5:clients=53,reads=6539302,writes=6539280
io_thread_6:clients=53,reads=6502269,writes=6502248
```
2025-01-06 15:59:02 +08:00
04f63d4af7 Fix index error of CRLF when replying with integer-encoded strings (#13711)
close #13709

Fix the index error of CRLF character for integer-encoded strings
in addReplyBulk function

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-12-31 21:41:10 +08:00
dc57ee03b1 Do security attack check only when command not found to reduce the critical path (#13702)
This PR is based on the commits from PR
https://github.com/valkey-io/valkey/pull/1212.

When explored the cycles distribution for main thread with io-threads
enabled. We found this security attack check takes significant time in
main thread, **~3%** cycles were used to do the commands security check
in main thread.

This patch try to completely avoid doing it in the hot path. We can do
it only after we looked up the command and it wasn't found, just before
we call commandCheckExistence.

---------

Co-authored-by: Lipeng Zhu <lipeng.zhu@intel.com>
Co-authored-by: Wangyang Guo <wangyang.guo@intel.com>
2024-12-26 12:51:44 +08:00
Thalia ArchibaldandGitHub 8144019a13 Check length before reading in stringmatchlen (#13690)
Fixes four cases where `stringmatchlen` could overrun the pattern if it is not
terminated with NUL.

These commits are cherry-picked from my
[fork](https://github.com/thaliaarchi/antirez-stringmatch) which
extracts `stringmatch` as a library and compares it to other projects by
antirez which uses the same matcher.
2024-12-26 12:37:23 +08:00
Yuan WangandGitHub 7665bdc91a Offload lookupCommand into IO threads when threaded IO is enabled (#13696)
From flame graph, we could see `lookupCommand` in main thread costs much
CPU, so we can let IO threads to perform `lookupCommand`.

To avoid race condition among multiple IO threads, made the following
changes:
- Pause all IO threads when register or unregister commands
- Force a full rehashing of the command table dict when resizing
2024-12-25 16:03:22 +08:00
64a40b20d9 Async IO Threads (#13695)
## Introduction
Redis introduced IO Thread in 6.0, allowing IO threads to handle client
request reading, command parsing and reply writing, thereby improving
performance. The current IO thread implementation has a few drawbacks.
- The main thread is blocked during IO thread read/write operations and
must wait for all IO threads to complete their current tasks before it
can continue execution. In other words, the entire process is
synchronous. This prevents the efficient utilization of multi-core CPUs
for parallel processing.

- When the number of clients and requests increases moderately, it
causes all IO threads to reach full CPU utilization due to the busy wait
mechanism used by the IO threads. This makes it challenging for us to
determine which part of Redis has reached its bottleneck.

- When IO threads are enabled with TLS and io-threads-do-reads, a
disconnection of a connection with pending data may result in it being
assigned to multiple IO threads simultaneously. This can cause race
conditions and trigger assertion failures. Related issue:
redis#12540

Therefore, we designed an asynchronous IO threads solution. The IO
threads adopt an event-driven model, with the main thread dedicated to
command processing, meanwhile, the IO threads handle client read and
write operations in parallel.

## Implementation
### Overall
As before, we did not change the fact that all client commands must be
executed on the main thread, because Redis was originally designed to be
single-threaded, and processing commands in a multi-threaded manner
would inevitably introduce numerous race and synchronization issues. But
now each IO thread has independent event loop, therefore, IO threads can
use a multiplexing approach to handle client read and write operations,
eliminating the CPU overhead caused by busy-waiting.

the execution process can be briefly described as follows:
the main thread assigns clients to IO threads after accepting
connections, IO threads will notify the main thread when clients
finish reading and parsing queries, then the main thread processes
queries from IO threads and generates replies, IO threads handle
writing reply to clients after receiving clients list from main thread,
and then continue to handle client read and write events.

### Each IO thread has independent event loop
We now assign each IO thread its own event loop. This approach
eliminates the need for the main thread to perform the costly
`epoll_wait` operation for handling connections (except for specific
ones). Instead, the main thread processes requests from the IO threads
and hands them back once completed, fully offloading read and write
events to the IO threads.

Additionally, all TLS operations, including handling pending data, have
been moved entirely to the IO threads. This resolves the issue where
io-threads-do-reads could not be used with TLS.

### Event-notified client queue
To facilitate communication between the IO threads and the main thread,
we designed an event-notified client queue. Each IO thread and the main
thread have two such queues to store clients waiting to be processed.
These queues are also integrated with the event loop to enable handling.
We use pthread_mutex to ensure the safety of queue operations, as well
as data visibility and ordering, and race conditions are minimized, as
each IO thread and the main thread operate on independent queues,
avoiding thread suspension due to lock contention. And we implemented an
event notifier based on `eventfd` or `pipe` to support event-driven
handling.

### Thread safety
Since the main thread and IO threads can execute in parallel, we must
handle data race issues carefully.

**client->flags**
The primary tasks of IO threads are reading and writing, i.e.
`readQueryFromClient` and `writeToClient`. However, IO threads and the
main thread may concurrently modify or access `client->flags`, leading
to potential race conditions. To address this, we introduced an io-flags
variable to record operations performed by IO threads, thereby avoiding
race conditions on `client->flags`.

**Pause IO thread**
In the main thread, we may want to operate data of IO threads, maybe
uninstall event handler, access or operate query/output buffer or resize
event loop, we need a clean and safe context to do that. We pause IO
thread in `IOThreadBeforeSleep`, do some jobs and then resume it. To
avoid thread suspended, we use busy waiting to confirm the target
status. Besides we use atomic variable to make sure memory visibility
and ordering. We introduce these functions to pause/resume IO Threads as
below.
```
pauseIOThread, resumeIOThread
pauseAllIOThreads, resumeAllIOThreads
pauseIOThreadsRange, resumeIOThreadsRange
```
Testing has shown that `pauseIOThread` is highly efficient, allowing the
main thread to execute nearly 200,000 operations per second during
stress tests. Similarly, `pauseAllIOThreads` with 8 IO threads can
handle up to nearly 56,000 operations per second. But operations
performed between pausing and resuming IO threads must be quick;
otherwise, they could cause the IO threads to reach full CPU
utilization.

**freeClient and freeClientAsync**
The main thread may need to terminate a client currently running on an
IO thread, for example, due to ACL rule changes, reaching the output
buffer limit, or evicting a client. In such cases, we need to pause the
IO thread to safely operate on the client.

**maxclients and maxmemory-clients updating**
When adjusting `maxclients`, we need to resize the event loop for all IO
threads. Similarly, when modifying `maxmemory-clients`, we need to
traverse all clients to calculate their memory usage. To ensure safe
operations, we pause all IO threads during these adjustments.

**Client info reading**
The main thread may need to read a client’s fields to generate a
descriptive string, such as for the `CLIENT LIST` command or logging
purposes. In such cases, we need to pause the IO thread handling that
client. If information for all clients needs to be displayed, all IO
threads must be paused.

**Tracking redirect**
Redis supports the tracking feature and can even send invalidation
messages to a connection with a specified ID. But the target client may
be running on IO thread, directly manipulating the client’s output
buffer is not thread-safe, and the IO thread may not be aware that the
client requires a response. In such cases, we pause the IO thread
handling the client, modify the output buffer, and install a write event
handler to ensure proper handling.

**clientsCron**
In the `clientsCron` function, the main thread needs to traverse all
clients to perform operations such as timeout checks, verifying whether
they have reached the soft output buffer limit, resizing the
output/query buffer, or updating memory usage. To safely operate on a
client, the IO thread handling that client must be paused.
If we were to pause the IO thread for each client individually, the
efficiency would be very low. Conversely, pausing all IO threads
simultaneously would be costly, especially when there are many IO
threads, as clientsCron is invoked relatively frequently.
To address this, we adopted a batched approach for pausing IO threads.
At most, 8 IO threads are paused at a time. The operations mentioned
above are only performed on clients running in the paused IO threads,
significantly reducing overhead while maintaining safety.

### Observability
In the current design, the main thread always assigns clients to the IO
thread with the least clients. To clearly observe the number of clients
handled by each IO thread, we added the new section in INFO output. The
`INFO THREADS` section can show the client count for each IO thread.
```
# Threads
io_thread_0:clients=0
io_thread_1:clients=2
io_thread_2:clients=2
```

Additionally, in the `CLIENT LIST` output, we also added a field to
indicate the thread to which each client is assigned.

`id=244 addr=127.0.0.1:41870 laddr=127.0.0.1:6379 ... resp=2 lib-name=
lib-ver= io-thread=1`

## Trade-off
### Special Clients
For certain special types of clients, keeping them running on IO threads
would result in severe race issues that are difficult to resolve.
Therefore, we chose not to offload these clients to the IO threads.

For replica, monitor, subscribe, and tracking clients, main thread may
directly write them a reply when conditions are met. Race issues are
difficult to resolve, so we have them processed in the main thread. This
includes the Lua debug clients as well, since we may operate connection
directly.

For blocking client, after the IO thread reads and parses a command and
hands it over to the main thread, if the client is identified as a
blocking type, it will be remained in the main thread. Once the blocking
operation completes and the reply is generated, the client is
transferred back to the IO thread to send the reply and wait for event
triggers.

### Clients Eviction
To support client eviction, it is necessary to update each client’s
memory usage promptly during operations such as read, write, or command
execution. However, when a client operates on an IO thread, it is not
feasible to update the memory usage immediately due to the risk of data
races. As a result, memory usage can only be updated either in the main
thread while processing commands or in the `ClientsCron` periodically.
The downside of this approach is that updates might experience a delay
of up to one second, which could impact the precision of memory
management for eviction.

To avoid incorrectly evicting clients. We adopted a best-effort
compensation solution, when we decide to eviction a client, we update
its memory usage again before evicting, if the memory used by the client
does not decrease or memory usage bucket is not changed, then we will
evict it, otherwise, not evict it.

However, we have not completely solved this problem. Due to the delay in
memory usage updates, it may lead us to make incorrect decisions about
the need to evict clients.

### Defragment
In the majority of cases we do NOT use the data from argv directly in
the db.
1. key names
We store a copy that we allocate in the main thread, see `sdsdup()` in
`dbAdd()`.
2. hash key and value
We store key as hfield and store value as sds, see `hfieldNew()` and
`sdsdup()` in `hashTypeSet()`.
3. other datatypes
   They don't even use SDS, so there is no reference issues.

But in some cases client the data from argv may be retain by the main
thread.
As a result, during fragmentation cleanup, we need to move allocations
from the IO thread’s arena to the main thread’s arena. We always
allocate new memory in the main thread’s arena, but the memory released
by IO threads may not yet have been reclaimed. This ultimately causes
the fragmentation rate to be higher compared to creating and allocating
entirely within a single thread.
The following cases below will lead to memory allocated by the IO thread
being kept by the main thread.
1. string related command: `append`, `getset`, `mset` and `set`.
If `tryObjectEncoding()` does not change argv, we will keep it directly
in the main thread, see the code in `tryObjectEncoding()`(specifically
`trimStringObjectIfNeeded()`)
2. block related command.
    the key names will be kept in `c->db->blocking_keys`.
3. watch command
    the key names will be kept in `c->db->watched_keys`.
4. [s]subscribe command
    channel name will be kept in `serverPubSubChannels`.
5. script load command
    script will be kept in `server.lua_scripts`.
7. some module API: `RM_RetainString`, `RM_HoldString`

Those issues will be handled in other PRs.

## Testing
### Functional Testing
The commit with enabling IO Threads has passed all TCL tests, but we did
some changes:
**Client query buffer**: In the original code, when using a reusable
query buffer, ownership of the query buffer would be released after the
command was processed. However, with IO threads enabled, the client
transitions from an IO thread to the main thread for processing. This
causes the ownership release to occur earlier than the command
execution. As a result, when IO threads are enabled, the client's
information will never indicate that a shared query buffer is in use.
Therefore, we skip the corresponding query buffer tests in this case.
**Defragment**: Add a new defragmentation test to verify the effect of
io threads on defragmentation.
**Command delay**: For deferred clients in TCL tests, due to clients
being assigned to different threads for execution, delays may occur. To
address this, we introduced conditional waiting: the process proceeds to
the next step only when the `client list` contains the corresponding
commands.

### Sanitizer Testing
The commit passed all TCL tests and reported no errors when compiled
with the `fsanitizer=thread` and `fsanitizer=address` options enabled.
But we made the following modifications: we suppressed the sanitizer
warnings for clients with watched keys when updating `client->flags`, we
think IO threads read `client->flags`, but never modify it or read the
`CLIENT_DIRTY_CAS` bit, main thread just only modifies this bit, so
there is no actual data race.

## Others
### IO thread number
In the new multi-threaded design, the main thread is primarily focused
on command processing to improve performance. Typically, the main thread
does not handle regular client I/O operations but is responsible for
clients such as replication and tracking clients. To avoid breaking
changes, we still consider the main thread as the first IO thread.

When the io-threads configuration is set to a low value (e.g., 2),
performance does not show a significant improvement compared to a
single-threaded setup for simple commands (such as SET or GET), as the
main thread does not consume much CPU for these simple operations. This
results in underutilized multi-core capacity. However, for more complex
commands, having a low number of IO threads may still be beneficial.
Therefore, it’s important to adjust the `io-threads` based on your own
performance tests.

Additionally, you can clearly monitor the CPU utilization of the main
thread and IO threads using `top -H -p $redis_pid`. This allows you to
easily identify where the bottleneck is. If the IO thread is the
bottleneck, increasing the `io-threads` will improve performance. If the
main thread is the bottleneck, the overall performance can only be
scaled by increasing the number of shards or replicas.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: oranagra <oran@redislabs.com>
2024-12-23 14:16:40 +08:00
Moti CohenandGitHub 08c2b276fb Optimize dict no_value also for even addresses (#13683)
This pull request enhances the no_value flag option in the dict implementation,
which is used to store keys without associated values. Previously, when a key
had an odd memory address and was the only item in a table entry, it could be
directly stored as a pointer without requiring an intermediate dictEntry. With
this update, the optimization has been extended to also handle keys with even
memory addresses in the same manner.
2024-12-22 14:10:07 +02:00
1f09a55eba Avoid importing memory aligned malloc (#13693)
This PR is based on the commits from PR
https://github.com/valkey-io/valkey/pull/1442.

We deprecate the usage of classic malloc and free, but under certain
circumstances they might get imported from intrinsics. The original
thought is we should just override malloc and free to use zmalloc and
zfree, but I think we should continue to deprecate it to avoid
accidental imports of allocations.

---------

Co-authored-by: Madelyn Olson <matolson@amazon.com>
2024-12-20 09:39:14 +08:00
684077682e Fix bug in PFMERGE command (#13672)
The bug was introduced in #13558 . 

When merging dense hll structures, `hllDenseCompress` writes to wrong
location and the result will be zero. The unit tests didn't cover this
case.

This PR
+ fixes the bug
+ adds `PFDEBUG SIMD (ON|OFF)` for unit tests
+ adds a new TCL test to cover the cases

Synchronized from https://github.com/valkey-io/valkey/pull/1293

---------

Signed-off-by: Xuyang Wang <xuyangwang@link.cuhk.edu.cn>
Co-authored-by: debing.sun <debing.sun@redis.com>
2024-12-18 14:41:04 +08:00
f8942f93a6 Avoid unnecessary hfield Creation/Deletion on updates in hashTypeSet. HSET updates improvement of ~10% (#13655)
This PR eliminates unnecessary creation and destruction of hfield
objects, ensuring only required updates or insertions are performed.
This reduces overhead and improves performance by streamlining field
management in hash dictionaries, particularly in scenarios involving
frequent updates, like the benchmarks in:
-
[memtier_benchmark-100Kkeys-load-hash-50-fields-with-100B-values](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-100Kkeys-load-hash-50-fields-with-100B-values.yml)
-
[memtier_benchmark-10Mkeys-load-hash-5-fields-with-100B-values-pipeline-10](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-10Mkeys-load-hash-5-fields-with-100B-values-pipeline-10.yml)


To test it we can simply focus on the hfield related tests

```
tclsh tests/test_helper.tcl --single unit/type/hash-field-expire
tclsh tests/test_helper.tcl --single unit/type/hash
tclsh tests/test_helper.tcl --dump-logs --single unit/other
```

Extra check on full CI:
- [x] https://github.com/filipecosta90/redis/actions/runs/12225788759

## microbenchmark results 
16.7% improvement (drop in time) in dictAddNonExistingRaw vs dictAddRaw
```
make REDIS_CFLAGS="-g -fno-omit-frame-pointer -O3 -DREDIS_TEST" -j
$ ./src/redis-server test dict --accurate
(...)
Inserting via dictAddRaw() non existing: 5000000 items in 2592 ms
(...)
Inserting via dictAddNonExistingRaw() non existing: 5000000 items in 2160 ms
```

8% improvement (drop in time) in find (non existing) and adding via
`dictGetHash()+dictFindWithHash()+dictAddNonExistingRaw()` vs
`dictFind()+dictAddRaw()`
```
make REDIS_CFLAGS="-g -fno-omit-frame-pointer -O3 -DREDIS_TEST" -j
$ ./src/redis-server test dict --accurate
(...)
Find() and inserting via dictFind()+dictAddRaw() non existing: 5000000 items in 2983 ms
Find() and inserting via dictGetHash()+dictFindWithHash()+dictAddNonExistingRaw() non existing: 5000000 items in 2740 ms

```

## benchmark results 


To benchmark:

```
pip3 install redis-benchmarks-specification==0.1.250
taskset -c 0 ./src/redis-server --save '' --protected-mode no --daemonize yes
redis-benchmarks-spec-client-runner --tests-regexp ".*load-hash.*" --flushall_on_every_test_start --flushall_on_every_test_end  --cpuset_start_pos 2 --override-memtier-test-time 60
```

Improvements on achievable throughput in:

test | ops/sec unstable (59953d2df6) |
ops/sec this PR (24af7190fdc0ad3c6d5957c17853490990c35dcc) | % change
-- | -- | -- | --
memtier_benchmark-1key-load-hash-1K-fields-with-5B-values | 4097 | 5032
| 22.8%
memtier_benchmark-100Kkeys-load-hash-50-fields-with-100B-values | 37658
| 44688 | 18.7%
memtier_benchmark-100Kkeys-load-hash-50-fields-with-1000B-values | 14736
| 17350 | 17.7%

memtier_benchmark-1Mkeys-load-hash-5-fields-with-1000B-values-pipeline-10
| 131848 | 143485 | 8.8%
memtier_benchmark-1Mkeys-load-hash-hmset-5-fields-with-1000B-values |
82071 | 85681 | 4.4%
memtier_benchmark-1Mkeys-load-hash-5-fields-with-1000B-values | 82882 |
86336 | 4.2%

memtier_benchmark-10Mkeys-load-hash-5-fields-with-100B-values-pipeline-10
| 262502 | 273376 | 4.1%
memtier_benchmark-10Kkeys-load-hash-50-fields-with-10000B-values | 2821
| 2936 | 4.1%

---------

Co-authored-by: Moti Cohen <moticless@gmail.com>
2024-12-12 19:41:08 +02:00
c51c96656b modules API: Add test for ACL check of empty prefix (#13678)
- Add empty string test for the new API
`RedisModule_ACLCheckKeyPrefixPermissions`.
- Fix order of checks: `(pattern[patternLen - 1] != '*' || patternLen ==
0)`

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-12-10 09:16:30 +02:00
Moti CohenandGitHub 0dd057222b Modules API: new HashFieldMinExpire(). Add flag REDISMODULE_HASH_EXPIRE_TIME to HashGet(). (#13676)
This PR introduces API to query Expiration time of hash fields.

# New `RedisModule_HashFieldMinExpire()`
For a given hash, retrieves the minimum expiration time across all
fields. If no fields have expiration or if the key is not a hash then
return `REDISMODULE_NO_EXPIRE` (-1).
```
mstime_t RM_HashFieldMinExpire(RedisModuleKey *hash);
```

# Extension to `RedisModule_HashGet()`
Adds a new flag, `REDISMODULE_HASH_EXPIRE_TIME`, to retrieve the
expiration time of a specific hash field. If the field does not exist or
has no expiration, returns `REDISMODULE_NO_EXPIRE`. It is fully
backward-compatible (RM_HashGet retains its original behavior unless the
new flag is used).

Example:
```
mstime_t expiry1, expiry2;
RedisModule_HashGet(mykey, REDISMODULE_HASH_EXPIRE_TIME, "field1", &expiry1, NULL);
RedisModule_HashGet(mykey, REDISMODULE_HASH_EXPIRE_TIME, "field1", &expiry1, "field2", &expiry2, NULL);
```
2024-12-05 11:14:52 +02:00
59953d2df6 Improve listpack Handling and Decoding Efficiency: 16.3% improvement on LRANGE command (#13652)
This PR focused on refining listpack encoding/decoding functions and
optimizing reply handling mechanisms related to it.
Each commit has the measured improvement up until the last accumulated
improvement of 16.3% on
[memtier_benchmark-1key-list-100-elements-lrange-all-elements-pipeline-10](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-1key-list-100-elements-lrange-all-elements-pipeline-10.yml)
benchmark.

Connection mode | CE Baseline (Nov 14th)
701f06657d | CE PR #13652 | CE PR vs CE
Unstable
-- | -- | -- | --
TCP | 155696 | 178874 | 14.9%
Unix socket | 169743 | 197428 | 16.3%

To test it we can simply focus on the scan.tcl

```
tclsh tests/test_helper.tcl --single unit/replybufsize
```

### Commit details:
- 2e58d048fd9d1365c56292ebd69f2ed141bfeda1 +
29c6c86c6b96376f807656f203b61d0e7fcb65a4 : Eliminate an indirect memory
access on lpCurrentEncodedSizeBytes and completely avoid passing p*
fully to lpCurrentEncodedSizeBytes + Add lpNextWithBytes helper function
and optimize addListListpackRangeReply

**- Improvement of 3.1%, from 168969.88 ops/sec to 174239.75 ops/sec**
- af52aacff86e2809f3451e2561043d532b9ee223 Refactor lpDecodeBacklen for
loop-based decoding, improving readability and branch efficiency.
**- NO CHANGE. REVERTED in 09f6680ba0d0b5acabca537c651008f0c8ec061b**
    
- 048bfe4edaaf5671f7f06b06d1492f81e6af3e59 +
03e8ff3af70891cbd3d53ca865558976459bb38e : reducing condition checks in
_addReplyToBuffer, inlining it, and avoid entering it when there are
there already entries in the reply list
and check if the reply length exceeds available buffer space before
calling _addReplyToBuffer
**- accumulated Improvement of 12.4%, from 168969.88 ops/sec to
189726.81 ops/sec**

- 9a63d4d6a9fa946505e31ecce4c7796845fc022c: always update the buf_peak
on _addReplyToBufferOrList
**- accumulated Improvement of 14.2%, from 168969.88 ops/sec to 193887
ops/sec**

- b544ade67628a1feaf714d6cfd114930e0c7670b: Introduce
lpEncodeBacklenBytes to avoid any indirect memory access on previous
usage of lpEncodeBacklen(NULL,...). inline lpEncodeBacklenBytes().
**- accumulated Improvement of 16.3%, from 168969.88 ops/sec to
197427.70 ops/sec**

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-12-04 18:04:37 +08:00
ddafac4c6c Optimize dictFind with prefetching and branch prediction hints (#13646)
This pull request optimizes the `dictFind` function by adding software
prefetching and branch prediction hints to improve cache efficiency and
reduce memory latency.
It introduces 2 prefetch hints (read/write) that became no-ops in case
the compiler does not support it.

Baseline profiling with Intel VTune indicated that dictFind was
significantly back-end bound, with memory latency accounting for 59.6%
of clockticks, with frequent stalls from DRAM-bound operations due to
cache misses during hash table lookups.

![microarch](https://github.com/user-attachments/assets/9e3cf334-ae6b-4767-b568-713a4ac24e87)

---------

Co-authored-by: Yuan Wang <wangyuancode@163.com>
2024-12-04 17:16:14 +08:00
Ozan TezcanandGitHub 2af69a931a Do not call _dictClear()'s callback for the first 65k items (#13674)
In https://github.com/redis/redis/pull/13495, we introduced a feature to
reply -LOADING while flushing a large db on a replica.
While `_dictClear()` is in progress, it calls a callback for every 65k
items and we yield back to eventloop to reply -LOADING.

This change has made some tests unstable as those tests don't expect new
-LOADING reply.
One observation, inside `_dictClear()`, we call the callback even if db
has a few keys. Most tests run with small amount of keys. So, each
replication and cluster test has to handle potential -LOADING reply now.

This PR changes this behavior, skips calling callback when `i=0` to
stabilize replication tests.
Callback will be called after the first 65k items. Most tests use less
than 65k keys and they won't get -LOADING reply.
2024-12-03 09:26:19 +03:00
Moti CohenandGitHub 06b144aa09 Modules API: Add RedisModule_ACLCheckKeyPrefixPermissions (#13666)
This PR introduces a new API function to the Redis Module API:
```
int RedisModule_ACLCheckKeyPrefixPermissions(RedisModuleUser *user, RedisModuleString *prefix, int flags);
```
Purpose:
The function checks if a given user has access permissions to any key
that match a specific prefix. This validation is based on the user’s ACL
permissions and the specified flags.

Note, this prefix-based approach API may fail to detect prefixes that
are individually uncovered but collectively covered by the patterns. For
example the prefix `ID-*` is not fully included in pattern `ID-[0]*` and
is not fully included in pattern `ID-[^0]*` but it is fully included in
the set of patterns `{ID-[0]*, ID-[^0]*}`
2024-11-28 18:33:58 +02:00
db33b67d37 Deprecate ubuntu lunar and macos-12 in workflows (#13669)
1. Ubuntu Lunar reached End of Life on January 25, 2024, so upgrade the
ubuntu version to plucky in action `test-ubuntu-jemalloc-fortify` to
pass the daily CI
2. The macOS-12 environment is deprecated so upgrade macos-12 to
macos-13 in daily CI

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-11-28 21:59:43 +08:00
Filipe Oliveira (Redis)andGitHub a106198878 Optimize addReplyBulk on sds/int encoded strings: 2.2% to 4% reduction of CPU Time on GET high pipeline use-cases (#13644)
### Summary

By profing 1KiB 100% GET's use-case, on high pipeline use-cases, we can
see that addReplyBulk and it's inner calls takes 8.30% of the CPU
cycles. This PR reduces from 2.2% to 4% the CPU time spent on
addReplyBulk. Specifically for GET use-cases, we saw an improvement from
2.7% to 9.1% on the achievable ops/sec

### Improvement

By reducing the duplicate work we can improve by around 2.7% on sds
encoded strings, and around 9% on int encoded strings. This PR does the
following:
- Avoid duplicate sdslen on addReplyBulk() for sds enconded objects
- Avoid duplicate sdigits10() call on int incoded objects on
addReplyBulk()
- avoid final "\r\n" addReplyProto() in the OBJ_ENCODING_INT type on
addReplyBulk

Altogether this improvements results in the following improvement on the
achievable ops/sec :

Encoding | unstable (commit 9906daf5c9) |
this PR | % improvement
-- | -- | -- | --
1KiB Values string SDS encoded | 1478081.88 | 1517635.38 | 2.7%
Values string "1" OBJ_ENCODING_INT | 1521139.36 | 1658876.59 | 9.1%

### CPU Time: Total of addReplyBulk

Encoding | unstable (commit 9906daf5c9) |
this PR | reduction of CPU Time: Total
-- | -- | -- | --
1KiB Values string SDS encoded | 8.30% | 6.10% | 2.2%
Values string "1" OBJ_ENCODING_INT | 7.20% | 3.20% | 4.0%

### To reproduce

Run redis with unix socket enabled
```
taskset -c 0 /root/redis/src/redis-server  --unixsocket /tmp/1.socket --save '' --enable-debug-command local
```

#### 1KiB Values string SDS encoded

Load data
```
taskset -c 2-5 memtier_benchmark  --ratio 1:0 -n allkeys --key-pattern P:P --key-maximum 1000000  --hide-histogram  --pipeline 10 -S /tmp/1.socket

```

Benchmark
```
taskset -c 2-6 memtier_benchmark --ratio 0:1 -c 1 -t 5 --test-time 60 --hide-histogram -d 1000 --pipeline 500  -S /tmp/1.socket --key-maximum 1000000 --json-out-file results.json
```

#### Values string "1" OBJ_ENCODING_INT 

Load data
```
$ taskset -c 2-5 memtier_benchmark  --command "SET __key__ 1" -n allkeys --command-key-pattern P --key-maximum 1000000  --hide-histogram -c 1 -t 1  --pipeline 100 -S /tmp/1.socket

# confirm we have the expected reply and format 
$ redis-cli get memtier-1
"1"

$ redis-cli debug object memtier-1
Value at:0x7f14cec57570 refcount:2147483647 encoding:int serializedlength:2 lru:2861503 lru_seconds_idle:8

```

Benchmark
```
taskset -c 2-6 memtier_benchmark --ratio 0:1 -c 1 -t 5 --test-time 60 --hide-histogram -d 1000 --pipeline 500  -S /tmp/1.socket --key-maximum 1000000 --json-out-file results.json
```
2024-11-26 16:11:01 +08:00
AliandGitHub 05b99c8f4c Fix typo in redis.conf (#12634)
unnecessarily and repetitive "OR"
2024-11-22 20:29:17 +08:00
9ebf80a28c Fix memory leak of jemalloc tcache on function flush command (#13661)
Starting from https://github.com/redis/redis/pull/13133, we allocate a
jemalloc thread cache and use it for lua vm.
On certain cases, like `script flush` or `function flush` command, we
free the existing thread cache and create a new one.

Though, for `function flush`, we were not actually destroying the
existing thread cache itself. Each call creates a new thread cache on
jemalloc and we leak the previous thread cache instances. Jemalloc
allows maximum 4096 thread cache instances. If we reach this limit,
Redis prints "Failed creating the lua jemalloc tcache" log and abort.

There are other cases that can cause this memory leak, including
replication scenarios when emptyData() is called.

The implication is that it looks like redis `used_memory` is low, but
`allocator_allocated` and RSS remain high.

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-11-21 14:12:58 +03:00
Moti CohenandGitHub 155634502d modules API: Support register unprefixed config parameters (#13656)
PR #10285 introduced support for modules to register four types of
configurations — Bool, Numeric, String, and Enum. Accessible through the
Redis config file and the CONFIG command.

With this PR, it will be possible to register configuration parameters
without automatically prefixing the parameter names. This provides
greater flexibility in configuration naming, enabling, for instance,
both `bf-initial-size` or `initial-size` to be defined in the module
without automatically prefixing with `<MODULE-NAME>.`. In addition it
will also be possible to create a single additional alias via the same
API. This brings us another step closer to integrate modules into redis
core.

**Example:** Register a configuration parameter `bf-initial-size` with
an alias `initial-size` without the automatic module name prefix, set
with new `REDISMODULE_CONFIG_UNPREFIXED` flag:
```
RedisModule_RegisterBoolConfig(ctx, "bf-initial-size|initial-size", default_val, optflags | REDISMODULE_CONFIG_UNPREFIXED, getfn, setfn, applyfn, privdata);
```
# API changes
Related functions that now support unprefixed configuration flag
(`REDISMODULE_CONFIG_UNPREFIXED`) along with optional alias:
```
RedisModule_RegisterBoolConfig
RedisModule_RegisterEnumConfig
RedisModule_RegisterNumericConfig
RedisModule_RegisterStringConfig
```

# Implementation Details:
`config.c`: On load server configuration, at function
`loadServerConfigFromString()`, it collects all unknown configurations
into `module_configs_queue` dictionary. These may include valid module
configurations or invalid ones. They will be validated later by
`loadModuleConfigs()` against the configurations declared by the loaded
module(s).
`Module.c:` The `ModuleConfig` structure has been modified to store now:
(1) Full configuration name (2) Alias (3) Unprefixed flag status -
ensuring that configurations retain their original registration format
when triggered in notifications.

Added error printout:
This change introduces an error printout for unresolved configurations,
detailing each unresolved parameter detected during startup. The last
line in the output existed prior to this change and has been retained to
systems relies on it:
```
595011:M 18 Nov 2024 08:26:23.616 # Unresolved Configuration(s) Detected:
595011:M 18 Nov 2024 08:26:23.616 #  >>> 'bf-initiel-size 8'
595011:M 18 Nov 2024 08:26:23.616 #  >>> 'search-sizex 32'
595011:M 18 Nov 2024 08:26:23.616 # Module Configuration detected without loadmodule directive or no ApplyConfig call: aborting
```

# Backward Compatibility:
Existing modules will function without modification, as the new
functionality only applies if REDISMODULE_CONFIG_UNPREFIXED is
explicitly set.

# Module vs. Core API Conflict Behavior
The new API allows to modules loading duplication of same configuration
name or same configuration alias, just like redis core configuration
allows (i.e. the users sets two configs with a different value, but
these two configs are actually the same one). Unlike redis core, given a
name and its alias, it doesn't allow have both configuration on load. To
implement it, it is required to modify DS `module_configs_queue` to
reflect the order of their loading and later on, during
`loadModuleConfigs()`, resolve pairs of names and aliases and which one
is the last one to apply. "Relaxing" this limitation can be deferred to
a future update if necessary, but for now, we error in this case.
2024-11-21 09:55:02 +02:00
Oran AgraandGitHub 79fd255828 Add Lua VM memory to memory overhead, now that it's part of zmalloc (#13660)
To complement the work done in #13133.
it added the script VMs memory to be counted as part of zmalloc, but
that means they
should be also counted as part of the non-value overhead.

this commit contains some refactoring to make variable names and
function names less confusing.
it also adds a new field named `script.VMs` into the `MEMORY STATS`
command.

additionally, clear scripts and stats between tests in external mode
(which is related to how this issue was discovered)
2024-11-21 08:22:17 +02:00
5b84dc9678 Fix module loadex command crash due to invalid config (#13653)
Fix to https://github.com/redis/redis/issues/13650

providing an invalid config to a module with datatype crashes when redis
tries to unload the module due to the invalid config

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-11-21 14:14:14 +08:00
701f06657d Reuse c->argv after command execution to reduce memory allocation overhead (#13521)
inspred by https://github.com/redis/redis/pull/12730

Before this PR, we allocate new memory to store the user command
arguments, however, if the size of the current `c->argv` is larger than
the current command, we can reuse the previously allocated argv to avoid
allocating new memory for the current command.
And we will free `c->argv` in client cron when the client is idle for 2
seconds.

---------

Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
2024-11-14 20:35:31 +08:00
cf83803880 CRC64 perf improvements (#13638)
Improve the performance of crc64 for large batches by processing large
number
of bytes in parallel and combining the results.

---------
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Josiah Carlson <josiah.carlson@gmail.com>
2024-11-12 09:21:22 +02:00
Ozan TezcanandGitHub 54038811c0 Print command tokens on a crash when hide-user-data-from-log is enabled (#13639)
If `hide-user-data-from-log` config is enabled, we don't print client
argv in the crashlog to avoid leaking user info.
Though, debugging a crash becomes harder as we don't see the command
arguments causing the crash.

With this PR, we'll be printing command tokens to the log. As we have
command tokens defined in json schema for each command, using this data,
we can find tokens in the client argv.

e.g. 
`SET key value GET EX 10` ---> we'll print `SET * * GET EX *` in the
log.

Modules should introduce their command structure via
`RM_SetCommandInfo()`.
Then, on a crash we'll able to know module command tokens.
2024-11-11 09:34:18 +03:00
fdeb97629e Optimize PFCOUNT, PFMERGE command by SIMD acceleration (#13558)
This PR optimizes the performance of HyperLogLog commands (PFCOUNT,
PFMERGE) by adding AVX2 fast paths.

Two AVX2 functions are added for conversion between raw representation
and dense representation. They are 15 ~ 30 times faster than scalar
implementaion. Note that sparse representation is not accelerated.

AVX2 fast paths are enabled when the CPU supports AVX2 (checked at
runtime) and the hyperloglog configuration is default (HLL_REGISTERS ==
16384 && HLL_BITS == 6).

When merging 3 dense hll structures, the benchmark shows a 12x speedup
compared to the scalar version.

```
pfcount key1 key2 key3
pfmerge keyall key1 key2 key3
```

```
======================================================================================================
Type             Ops/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec 
------------------------------------------------------------------------------------------------------
PFCOUNT-scalar    5570.09        35.89060        32.51100        65.27900        69.11900       299.17
PFCOUNT-avx2     72604.92         2.82072         2.73500         5.50300         7.13500      3899.68
------------------------------------------------------------------------------------------------------
PFMERGE-scalar    7879.13        25.52156        24.19100        46.33500        48.38300       492.45
PFMERGE-avx2    126448.64         1.58120         1.53500         3.08700         4.89500      7903.04
------------------------------------------------------------------------------------------------------

scalar: redis:unstable   9906daf5c9
avx2:   Nugine:hll-simd  02e09f85ac07eace50ebdddd0fd70822f7b9152d 

CPU:    13th Gen Intel® Core™ i9-13900H × 20
Memory: 32.0 GiB
OS:     Ubuntu 22.04.5 LTS
```

Experiment repo: https://github.com/Nugine/redis-hyperloglog
Benchmark script:
https://github.com/Nugine/redis-hyperloglog/blob/main/scripts/memtier.sh
Algorithm:
https://github.com/Nugine/redis-hyperloglog/blob/main/cpp/bench.cpp

resolves #13551

---------

Co-authored-by: Yuan Wang <wangyuancode@163.com>
Co-authored-by: debing.sun <debing.sun@redis.com>
2024-11-08 15:19:38 +08:00
David DoughertyandGitHub 9906daf5c9 Update old links for modules-api-ref.md (#13479)
This PR replaces old .../topics/... links with current links,
specifically for the modules-api-ref.md file and the new automation that
Paolo Lazzari is working on. A few of the topics links have redirects,
but some don't. Best to use updated links.
2024-11-04 18:18:22 +02:00
guybe7andGitHub ded8d993b7 Modules: defrag CB should take robj, not sds (#13627)
Added a log of the keyname in the test modules to reproduce the problem
(tests crash without the fix)
2024-10-30 17:32:51 +08:00
Moti CohenandGitHub 6437d07b03 Fix memory leak on rdbload error (#13626)
On RDB load error, if an invalid `expireAt` value is read,
`dupSearchDict` is not released.
2024-10-30 10:03:31 +02:00
debing.sunandGitHub 4b29be3f36 Avoid redundant lpGet to boost quicklistCompare (#11533)
`lpCompare()` in `quicklistCompare()` will call `lpGet()` again, which
would be a waste.
The change will result in a boost for all commands that use
`quicklistCompre()`, including `linsert`, `lpos` and `lrem`.
2024-10-30 08:45:25 +08:00
Moti CohenandGitHub 2ec78d262d Add KEYSIZES section to INFO (#13592)
This PR adds a new section to the `INFO` command output, called
`keysizes`. This section provides detailed statistics on the
distribution of key sizes for each data type (strings, lists, sets,
hashes and zsets) within the dataset. The distribution is tracked using
a base-2 logarithmic histogram.

# Motivation
Currently, Redis lacks a built-in feature to track key sizes and item
sizes per data type at a granular level. Understanding the distribution
of key sizes is critical for monitoring memory usage and optimizing
performance, particularly in large datasets. This enhancement will allow
users to inspect the size distribution of keys directly from the `INFO`
command, assisting with performance analysis and capacity planning.

# Changes
New Section in `INFO` Command: A new section called `keysizes` has been
added to the `INFO` command output. This section reports a per-database,
per-type histogram of key sizes. It provides insights into how many keys
fall into specific size ranges (represented in powers of 2).

**Example output:**
```
127.0.0.1:6379> INFO keysizes
# Keysizes
db0_distrib_strings_sizes:1=19,2=655,512=100899,1K=31,2K=29,4K=23,8K=16,16K=3,32K=2
db0_distrib_lists_items:1=5784492,32=3558,64=1047,128=676,256=533,512=218,4K=1,8K=42
db0_distrib_sets_items:1=735564=50612,8=21462,64=1365,128=974,2K=292,4K=154,8K=89,
db0_distrib_hashes_items:2=1,4=544,32=141169,64=207329,128=4349,256=136226,1K=1
```
## Future Use Cases:
The key size distribution is collected per slot as well, laying the
groundwork for future enhancements related to Redis Cluster.
2024-10-29 13:07:26 +02:00
YaacovHazanandGitHub 0a24bfec13 Redis 8.0 M02 (#13624) 2024-10-28 19:09:18 +02:00
Shockingly GoodandGitHub 611c950293 Fix crash in RM_GetCurrentUserName() when the user isn't accessible (#13619)
The crash happens whenever the user isn't accessible, for example, it
isn't set for the context (when it is temporary) or in some other cases
like `notifyKeyspaceEvent`. To properly check for the ACL compliance, we
need to get the user name and the user to invoke other APIs. However, it
is not possible if it crashes, and it is impossible to work that around
in the code since we don't know (and **shouldn't know**!) when it is
available and when it is not.
2024-10-28 21:26:29 +08:00
YaacovHazan 14c48cc5ed Redis 8.0 M02 2024-10-28 14:01:22 +02:00
YaacovHazanandGitHub b08fd9f989 Merge unstable into 8.0 (#13622) 2024-10-27 17:23:40 +02:00
YaacovHazan dd20aa5160 Merge remote-tracking branch 'upstream/unstable' into HEAD 2024-10-27 13:50:58 +02:00
0a8e546957 Fix get # option in sort command (#13608)
From 7.4, Redis allows `GET` options in cluster mode when the pattern maps to
the same slot as the key, but GET # pattern that represents key itself is missed.
This commit resolves it, bug report #13607.

---------

Co-authored-by: Yuan Wang <yuan.wang@redis.com>
2024-10-22 09:55:00 +08:00
4f8cdc2a1e Fix compilation on compilers that do not support target attribute (#13609)
introduced by https://github.com/redis/redis/pull/13359
failure CI on ARM64:
https://github.com/redis/redis-extra-ci/actions/runs/11377893230/job/31652773710

---------

Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
Co-authored-by: ShooterIT <wangyuancode@163.com>
2024-10-18 09:11:23 +08:00
3788a055fe Optimize bitcount command by using popcnt (#13359)
Nowadays popcnt instruction is almost supported by X86 machine, which is
used to calculate "Hamming weight", it can bring much performance boost
in redis bitcount comand.

---------

Signed-off-by: hanhui365(hanhui@hygon.cn)
Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: oranagra <oran@redislabs.com>
Co-authored-by: Nugine <nugine@foxmail.com>
2024-10-17 09:13:19 +08:00
Yuan WangandGitHub b71a610f5c Clean up .rediscli_history_test temporary file (#13601)
After running test in local, there will be a file named
`.rediscli_history_test`, and it is not in `.gitignore` file, so this is
considered to have changed the code base. It is a little annoying, this
commit just clean up the temporary file.

We should delete `.rediscli_history_test` in the end since the second
server tests also write somethings into it, to make it corresponding, i
put `set ::env(REDISCLI_HISTFILE) ".rediscli_history_test"` at the
beginning.

Maybe we also can add this file into `.gitignore`
2024-10-17 09:12:11 +08:00
efcfffc528 Update modules with latest version (#13606)
Update redisbloom, redisjson and redistimeseries versions to 7.99.1

Co-authored-by: YaacovHazan <yaacov.hazan@redislabs.com>
2024-10-15 19:58:42 +03:00
paoloredisandGitHub 99d09c824c Only run redis_docs_sync.yaml on latest release (#13603)
We only want to trigger the workflow on the documentation repository for
the latest release
2024-10-15 16:02:11 +03:00
6c5e263d7b Temporarily hide the new SFLUSH command by marking it as experimental (#13600)
- Add a new 'EXPERIMENTAL' command flag, which causes the command
generator to skip over it and make the command to be unavailable for
execution
- Skip experimental tests by default
- Move the SFLUSH tests from the old framework to the new one

---------

Co-authored-by: YaacovHazan <yaacov.hazan@redislabs.com>
2024-10-15 11:02:51 +03:00
debing.sunandGitHub 3fc7ef8f81 Fix race in stream-cgroups test (#13593)
failed CI:
https://github.com/redis/redis/actions/runs/11171608362/job/31056659165
https://github.com/redis/redis/actions/runs/11226025974/job/31205787575
2024-10-12 09:23:19 +08:00
guybe7andGitHub a38c29b6c8 Cleanups related to expiry/eviction (#13591)
1. `dbRandomKey`: excessive call to `dbFindExpires` (will always return
1 if `allvolatile` + anyway called inside `expireIfNeeded`
2. Add `deleteKeyAndPropagate` that is used by both expiry/eviction
3. Change the order of calls in `expireIfNeeded` to save redundant calls
to `keyIsExpired`
4. `expireIfNeeded`: move `OBJ_STATIC_REFCOUNT` to
`deleteKeyAndPropagate`
5. `performEvictions` now uses `deleteEvictedKeyAndPropagate`
6. active-expire: moved `postExecutionUnitOperations` inside
`activeExpireCycleTryExpire`
7. `activeExpireCycleTryExpire`: less indentation + expire a key if `now
== t`
8. rename `lazy_expire_disabled` to `allow_access_expired`
2024-10-10 16:58:52 +08:00
Oran AgraandYaacovHazan 472d8a0df5 Prevent pattern matching abuse (CVE-2024-31228) 2024-10-08 20:55:44 +03:00
Oran AgraandYaacovHazan 8ec5da785b Fix ACL SETUSER Read/Write key pattern selector (CVE-2024-31227)
The '%' rule must contain one or both of R/W
2024-10-08 20:55:44 +03:00
Oran AgraandYaacovHazan 3a2669e8ae Fix lua bit.tohex (CVE-2024-31449)
INT_MIN value must be explicitly checked, and cannot be negated.
2024-10-08 20:55:44 +03:00
alonre24andGitHub f39e51178e Update target module in search (#13578)
Update search target path and version from M02
2024-10-08 13:58:28 +03:00
chx9andGitHub 5f7d7ce8b0 fix typo in test_helper.tcl (#13576)
fix typo in test_helper.tcl: even driven => event driven
2024-10-08 14:15:48 +08:00
Moti CohenandGitHub d092d64d7a Add new SFLUSH command to cluster for slot-based FLUSH (#13564)
This PR introduces a new `SFLUSH` command to cluster mode that allows
partial flushing of nodes based on specified slot ranges. Current
implementation is designed to flush all slots of a shard, but future
extensions could allow for more granular flushing.

**Command Usage:**
`SFLUSH <start-slot> <end-slot> [<start-slot> <end-slot>]* [SYNC|ASYNC]`

This command removes all data from the specified slots, either
synchronously or asynchronously depending on the optional SYNC/ASYNC
argument.

**Functionality:**
Current imp of `SFLUSH` command verifies that the provided slot ranges
are valid and cover all of the node's slots before proceeding. If slots
are partially or incorrectly specified, the command will fail and return
an error, ensuring that all slots of a node must be fully covered for
the flush to proceed.

The function supports both synchronous (default) and asynchronous
flushing. In addition, if possible, SFLUSH SYNC will be run as blocking
ASYNC as an optimization.
2024-09-29 09:13:21 +03:00
YaacovHazanandGitHub 30eb6c3210 Merge unstable into 8.0 (#13573) 2024-09-26 14:36:04 +03:00
YaacovHazan 80458d0490 Merge remote-tracking branch 'upstream/unstable' into HEAD 2024-09-26 10:50:12 +03:00
99c40ab53d Use hashtable as the default type of temp set object during sunion/sdiff (#13567)
This PR is based on https://github.com/valkey-io/valkey/pull/996


Currently, for operations like SUNION or SDIFF, temporary set object can
be intset or listpack. Search operation is costly for these encodings.
This patch tries to set the temporary set object as hash table by
default. It also tries to determine correct encoding for the temporary
set object to reduce the unnecessary conversation.

This change is supposed to give performance boost for tests like:
-
[memtier_benchmark-2keys-set-10-100-elements-sdiff](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-2keys-set-10-100-elements-sdiff.yml)
66.2% IMPROVEMENT
-
[memtier_benchmark-2keys-set-10-100-elements-sunion](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-2keys-set-10-100-elements-sunion.yml)
126.5% IMPROVEMENT

-------
Co-authored-by: Lipeng Zhu <lipeng.zhu@intel.com>
Co-authored-by: Wangyang Guo <wangyang.guo@intel.com>

Co-authored-by: Lipeng Zhu <lipeng.zhu@intel.com>
Co-authored-by: Wangyang Guo <wangyang.guo@intel.com>
2024-09-25 12:41:17 +03:00
Moti CohenGitHubRayacoo zisong.cw@alibaba-inc.com
26ef28467a Optimize ZUNION[STORE] by avoiding redundant temporary dict usage (#13566)
This PR is based on valkey-io/valkey#829

Previously, ZUNION and ZUNIONSTORE commands used a temporary accumulator dict
and at the end copied it as-is to dstzset->dict. This PR removes accumulator and directly
stores into dstzset->dict, eliminating the extra copy.

Co-authored-by: Rayacoo zisong.cw@alibaba-inc.com
2024-09-25 11:55:00 +03:00
Moti CohenandGitHub 5f28bd96db Fix race in HFE tests (#13563)
Test 1 - give more time for expiration
Test 2 - Evaluate expiration time boundaries [+1,+2] before setting expiration [+1]
Test 3 - Avoid race on test HFEs propagated to replica
2024-09-23 10:30:29 +03:00
debing.sunandGitHub 438cfed70a Replace wrongly free with zfree in redis-cli (#13560)
#13258 Incorrect use of free instead of zfree
2024-09-23 09:40:47 +08:00
Moti CohenandGitHub 3a3cacfefa Extend modules API to read also expired keys and subkeys (#13526)
The PR extends `RedisModule_OpenKey`'s flags to include
`REDISMODULE_OPEN_KEY_ACCESS_EXPIRED`, which allows to access expired
keys.

It also allows to access expired subkeys. Currently relevant only for
hash fields
and has its impact on `RM_HashGet` and `RM_Scan`.
2024-09-19 20:47:00 +03:00
debing.sunandGitHub 617909e943 Align the offset in ASCII logo (#13557)
Since `\\` is only one character, we need to add an extra space to the right.
2024-09-18 14:42:32 +08:00
adamiBsandGitHub e9cbfccec6 Support musl Rust Installation in Modules Makefile (#13549)
This PR introduces the installation of the `musl`-based version of Rust,
in order to support alpine-based runtime environments (Rust is used by
[RedisJSON](https://github.com/RedisJSON/RedisJSON)).
2024-09-15 20:23:05 +03:00
7b69183a8d Replace usage of _addReplyLongLongWithPrefix with specific bulk/mbulk functions to reduce condition checks in hotpath. (#13520)
Instead of adding runtime logic to decide which prefix/shared object to
use when doing the reply we can simply use an inline method to avoid runtime
overhead of condition checks, and also keep the code change small.
Preliminary data show improvements on commands that heavily rely on
bulk/mbulk replies (example of LRANGE).

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-09-15 21:40:09 +08:00
af7fca797a Using fast_float library for faster parsing of 64 decimal strings. (#11884)
Fixes #8825 

We're using the fast_float library[1] in our (compiled-in)
floating-point fast_float_strtod implementation for faster and more
portable parsing of 64 decimal strings.

The single file fast_float.h is an amalgamation of the entire library,
which can be (re)generated with the amalgamate.py script (from the
fast_float repository) via the command:

```
python3 ./script/amalgamate.py --license=MIT > $REDIS_SRC/deps/fast_float/fast_float.h
```

[1]: https://github.com/fastfloat/fast_float

The used commit from fast_float library was the one from
https://github.com/fastfloat/fast_float/releases/tag/v3.10.1

---------

Co-authored-by: fcostaoliveira <filipe@redis.com>
2024-09-15 21:37:29 +08:00
9146ac050b Optimize HSCAN/ZSCAN command in case of listpack encoding: avoid the usage of intermediate list (#13531)
Similar to #13530 , applied to HSCAN and ZSCAN in case of listpack
encoding.

**Preliminary benchmark results showcase an improvement of 108% on the
achievable ops/sec for ZSCAN and 65% for HSCAN**.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-09-13 20:36:19 +08:00
debing.sunandGitHub ef3a5f58a8 Fix missing initialization of EbucketsIterator->isRax (#13545)
in https://github.com/redis/redis/pull/13519, when `eb` is empty,
`isRax` is not correctly initialized to 0, which can lead to `ebStop()`
potentially entering the wrong rax branch.
2024-09-13 17:12:27 +08:00
Filipe Oliveira (Redis)andGitHub f2f85ba354 Optimize SSCAN command in case of listpack or intset encoding: avoid the usage of intermediate list. From 2N to N iterations (#13530)
On SSCAN, in case of listpack and intset encoding we actually reply the
entire set, and always reply with the cursor 0.

For those cases, we don't need to accumulate the replies in a list and
can completely avoid the overhead of list appending and then iterating
over the list again -- meaning we do N iterations instead of 2N
iterations over the SET and save intermediate memory as well.

Preliminary benchmarks, `SSCAN set:100 0`, showcased an improvement of
60% as visible bellow on a SET with 100 string elements (listpack
encoded).
2024-09-12 22:36:54 +08:00
Moti CohenandGitHub c115c5230e Add iterator capability to ebuckets DS (#13519)
Add basic iterator API for ebuckets of start, next, nextBucket and stop.
2024-09-12 15:02:32 +03:00
Moti CohenandGitHub 65a87cb773 Correct spelling error at t_hash.c comment (#13540)
spell check error : ./src/t_hash.c:1141: RESOTRE ==> RESTORE
2024-09-12 12:50:44 +03:00
Moti CohenandGitHub 9a89e32a95 HFE - Fix key ref by the hash on RENAME/MOVE/SWAPDB/RESTORE (#13539)
If the hash previously had HFEs (hash-fields with expiration) but later no longer
does, the key ref in the hash might become outdated after a MOVE, COPY,
RENAME or RESTORE operation. These commands maintain the key ref only
if HFEs are present. That is, we can only be sure that key ref is valid as long as the
hash has HFEs.
2024-09-12 12:40:12 +03:00
YaacovHazanandGitHub 5fe3e74a38 Redis 8.0 M01 (#13538)
### New Features in binary distributions

- 7 new data structures: JSON, Time series, Bloom filter, Cuckoo filter,
Count-min sketch, Top-k, t-digest
- Redis scalable query engine (including vector search)

### Potentially breaking changes

- #12272 `GETRANGE` returns an empty bulk when the negative end index is
out of range
- #12395 Optimize `SCAN` command when matching data type

### Bug fixes

- #13510 Fix `RM_RdbLoad` to enable AOF after RDB loading is completed
- #13489 `ACL CAT` - return module commands
- #13476 Fix a race condition in the `cache_memory` of `functionsLibCtx`
- #13473 Fix incorrect lag due to trimming stream via `XTRIM` command
- #13338 Fix incorrect lag field in `XINFO` when tombstone is after the
`last_id` of the consume group
- #13470 On `HDEL` of last field - update the global hash field
expiration data structure
- #13465 Cluster: Pass extensions to node if extension processing is
handled by it
- #13443 Cluster: Ensure validity of myself when loading cluster config
- #13422 Cluster: Fix `CLUSTER SHARDS` command returns empty array

### Modules API

- #13509 New API calls: `RM_DefragAllocRaw`, `RM_DefragFreeRaw`, and
`RM_RegisterDefragCallbacks` - defrag API to allocate and free raw
memory

### Performance and resource utilization improvements

- #13503 Avoid overhead of comparison function pointer calls in listpack
`lpFind`
- #13505 Optimize `STRING` datatype write commands
- #13499 Optimize `SMEMBERS` command
- #13494 Optimize `GEO*` commands reply
- #13490 Optimize `HELLO` command
- #13488 Optimize client query buffer
- #12395 Optimize `SCAN` command when matching data type
- #13529 Optimize `LREM`, `LPOS`, `LINSERT`, and `LINDEX` commands
- #13516 Optimize `LRANGE` and other commands that perform several
writes to client buffers per call
- #13431 Avoid `used_memory` contention when updating from multiple
threads

### Other general improvements

- #13495 Reply `-LOADING` on replica while flushing the db

### CLI tools

- #13411 redis-cli: Fix wrong `dbnum` showed after the client
reconnected

### Notes

- No backward compatibility for replication or persistence.
- Additional distributions, upgrade paths, features, and improvements
will be introduced in upcoming pre-releases.
- With the GA release of 8.0 we will deprecate Redis Stack.
2024-09-12 11:52:28 +03:00
610eb26c11 RED-129256, Fix TOUCH command from script in no-touch mode (#13512)
When a client in no-touch mode issues a TOUCH command on a key, the
key’s access time should be updated, but in scripts, and module's
RM_Call, it isn’t updated.
Command proc should be matched to the executing client, not the current
client.

Co-authored-by: Udi Ron <udi@speedb.io>
2024-09-12 11:33:26 +03:00
YaacovHazan 4955375ec7 Redis 8.0 M01 2024-09-12 10:23:36 +03:00
d265a61438 Avoid cluster.nodes load corruption due to shard-id generation (#13468)
PR #13428 doesn't fully resolve an issue where corruption errors can
still occur on loading of cluster.nodes file - seen on upgrade where
there were no shard_ids (from old Redis), 7.2.5 loading generated new
random ones, and persisted them to the file before gossip/handshake
could propagate the correct ones (or some other nodes unreachable).
This results in a primary/replica having differing shard_id in the
cluster.nodes and then the server cannot startup - reports corruption.

This PR builds on #13428 by simply ignoring the replica's shard_id in
cluster.nodes (if it exists), and uses the replica's primary's shard_id.
Additional handling was necessary to cover the case where the replica
appears before the primary in cluster.nodes, where it will first use a
generated shard_id for the primary, and then correct after it loads the
primary cluster.nodes entry.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-09-12 14:01:09 +08:00
YaacovHazan 406a365f44 Bundle modules for Redis 8.0 M01 2024-09-11 16:40:16 +03:00
2dd4cca363 Increment kvstore's non_empty_dicts only on first insert (#13528)
Found by @oranagra 

Currently, when the size of dict becomes 1, we do not check whether
`delta` is positive or negative.
As a result, `non_empty_dicts` is still incremented when the size of
dict changes from 2 to 1.
We should only increment `non_empty_dicts` when `delta` is positive, as
this indicates the first time an element is inserted into the dict.

---------

Co-authored-by: oranagra <oran@redislabs.com>
2024-09-11 09:36:01 +08:00
Filipe Oliveira (Redis)andGitHub bcae770819 Optimize LREM, LPOS, LINSERT, LINDEX: Avoid N-1 sdslen() calls on listTypeEqual (#13529)
This is a very easy optimization, that avoids duplicate computation of
the object length for LREM, LPOS, LINSERT na LINDEX.

We can see that sdslen takes 7.7% of the total CPU cycles of the
benchmarks.

Function Stack | CPU Time: Total | CPU Time: Self | Module | Function
(Full) | Source File | Start Address
-- | -- | -- | -- | -- | -- | --
listTypeEqual | 15.50% | 2.346s | redis-server | listTypeEqual |
t_list.c | 0x845dd
sdslen | 7.70% | 2.300s | redis-server | sdslen | sds.h | 0x845e4

Preliminary data showcases 4% improvement in the achievable ops/sec of
LPOS in string elements, and 2% in int elements.
2024-09-10 20:26:36 +08:00
bf802b0764 Add the option to build Redis with modules (#13524)
A new BUILD_WITH_MODULES flag was added to the Makefile to control
building the module directory.

The new module directory includes a general Makefile that iterates
over each module, fetch a specific version, and build it.

Co-authored-by: YaacovHazan <yaacov.hazan@redislabs.com>
2024-09-09 15:47:02 +03:00
Ozan TezcanandGitHub ac03e3721d Fix flaky replication tests (#13518)
#13495 introduced a change to reply -LOADING while flushing existing db on a replica. Some of our tests are
 sensitive to this change and do no expect -LOADING reply.

Fixing a couple of tests that fail time to time.
2024-09-08 12:54:01 +03:00
31227f4faf Optimize client type check on reply hot code paths (#13516)
## Proposed improvement

This PR introduces the static inlined function `clientTypeIsSlave` which
is doing only 1 condition check vs 3 checks of `getClientType`, and also
uses the `unlikely` to tell the compiler that the most common outcome is
for the client not to be a slave.
Preliminary data show 3% improvement on the achievable ops/sec on the
specific LRANGE benchmark. After running the entire suite we see up to
5% improvement in 2 tests.
https://github.com/redis/redis/pull/13516#issuecomment-2331326052

## Context

This optimization efforts comes from analyzing the profile info from the
[memtier_benchmark-1key-list-1K-elements-lrange-all-elements](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-1key-list-1K-elements-lrange-all-elements.yml)
benchmark.
 
By going over it, we can see that `getClientType` consumes 2% of the cpu
time, strictly to check if the client is a slave (
https://github.com/redis/redis/blob/unstable/src/networking.c#L397 , and
https://github.com/redis/redis/blob/unstable/src/networking.c#L1254 )


Function | CPU Time: Total | CPU Time: Self | Module | Function (Full)
-- | -- | -- | -- | --
_addReplyToBufferOrList->getClientType | 1.20% | 0.728s | redis-server |
getClientType
clientHasPendingReplies->getClientType | 0.80% | 0.482s | redis-server |
getClientType

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-09-06 10:24:30 +08:00
Max MalekzadehandGitHub f6f11f3ef1 Remove outdated "Try Redis" link in README.md (#13498) 2024-09-05 22:04:49 +08:00
Moti CohenandGitHub 569584d463 HFE - Simplify logic of HGETALL command (#13425) 2024-09-05 12:48:44 +03:00
ea3e8b79a1 Introduce reusable query buffer for client reads (#13488)
This PR is based on the commits from PR
https://github.com/valkey-io/valkey/pull/258,
https://github.com/valkey-io/valkey/pull/593,
https://github.com/valkey-io/valkey/pull/639

This PR optimizes client query buffer handling in Redis by introducing
a reusable query buffer that is used by default for client reads. This
reduces memory usage by ~20KB per client by avoiding allocations for
most clients using short (<16KB) complete commands. For larger or
partial commands, the client still gets its own private buffer.

The primary changes are:

* Adding a reusable query buffer `thread_shared_qb` that clients use by
default.
* Modifying client querybuf initialization and reset logic.
* Freeing idle client query buffers when empty to allow reuse of the
reusable query buffer.
* Master client query buffers are kept private as their contents need to
be preserved for replication stream.
* When nested commands is executed, only the first user uses the reuse
buffer, and subsequent users will still use the private buffer.

In addition to the memory savings, this change shows a 3% improvement in
latency and throughput when running with 1000 active clients.

The memory reduction may also help reduce the need to evict clients when
reaching max memory limit, as the query buffer is the main memory
consumer per client.

This PR is different from https://github.com/valkey-io/valkey/pull/258
1. When a client is in the mid of requiring a reused buffer and
returning it, regardless of whether the query buffer has changed
(expanded), we do not update the reused query buffer in the middle, but
return the reused query buffer (expanded or with data remaining) or
reset it at the end.
2. Adding a new thread variable `thread_shared_qb_used` to avoid
multiple clients requiring the reusable query buffer at the same time.

---------

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
Signed-off-by: Madelyn Olson <matolson@amazon.com>
Co-authored-by: Uri Yagelnik <uriy@amazon.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: oranagra <oran@redislabs.com>
2024-09-04 19:10:40 +08:00
debing.sunandGitHub 74609d44cd Fix set with invalid length causes smembers to hang (#13515)
After https://github.com/redis/redis/pull/13499, If the length set by
`addReplySetLen()` does not match the actual number of elements in the
reply, it will cause protocol broken and result in the client hanging.
2024-09-04 17:35:46 +08:00
Ozan TezcanandGitHub ea05c6ac47 Fix RM_RdbLoad() to enable AOF after loading is completed (#13510)
RM_RdbLoad() disables AOF temporarily while loading RDB.
Later, it does not enable it back as it checks AOF state (disabled by then) 
rather than AOF config parameter.

Added a change to restart AOF according to config parameter.
2024-09-04 11:11:04 +03:00
05aed4cab9 Optimize SET/INCR*/DECR*/SETRANGE/APPEND by reducing duplicate computation (#13505)
- Avoid addReplyLongLong (which converts back to string) the value we
already have as a robj, by using addReplyProto + addReply
- Avoid doing dbFind Twice for the same dictEntry on
INCR*/DECR*/SETRANGE/APPEND commands.
- Avoid multiple sdslen calls with the same input on setrangeCommand and
appendCommand
- Introduce setKeyWithDictEntry, which is like setKey(), but accepts an
optional dictEntry input: Avoids the second dictFind in SET command

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-09-04 14:51:21 +08:00
de7f2f87f7 Avoid overhead of comparision function pointer calls in lpFind() (#13503)
In #13279 (found by @filipecosta90), for custom lookups, we introduce a
comparison function for `lpFind()` to compare entry, but it also
introduces some overhead.

To avoid the overhead of function pointer calls:
1. Extract the lpFindCb() method into a lpFindCbInternal() method that
is easier to inline.
2. Use unlikely to annotate the comparison method, as can only success
once.

---------

Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
2024-09-03 22:53:13 +08:00
Filipe Oliveira (Redis)andGitHub fb8755a636 changed addReplyHumanLongDouble to addReplyDouble in georadiusGeneric and geoposCommand (#13494)
# Summary 

- Addresses https://github.com/redis/redis/issues/11565
- Measured improvements of 30% and 37% on the simple use-case (GEOSEARCH
and GEOPOS) (check
https://github.com/redis/redis/pull/13494#issuecomment-2313668934), and
of 66% on a dataset with >60M datapoints and pipeline 10 benchmark.
2024-09-03 20:54:20 +08:00
Meir Shpilraien (Spielrein)andGitHub d3d94ccf2e Added new defrag API to allocate and free raw memory. (#13509)
All the defrag allocations API expects to get a value and replace it, leaving the old value untouchable.
In some cases a value might be shared between multiple keys, in such cases we can not simply replace
it when the defrag callback is called.

To allow support such use cases, the PR adds two new API's to the defrag API:

1. `RM_DefragAllocRaw` - allocate memory base on a given size.
2. `RM_DefragFreeRaw` - Free the given pointer.

Those API's avoid using tcache so they operate just like `RM_DefragAlloc` but allows the user to split
the allocation and the memory free operations into two stages and control when those happen.

In addition the PR adds new API to allow the module to receive notifications when defrag start and end: `RM_RegisterDefragCallbacks`
Those callbacks are the same as `RM_RegisterDefragFunc` but promised to be called and the start
and the end of the defrag process.
2024-09-03 15:03:19 +03:00
00a8e72cfc Created specific SMEMBERS command logic which avoids sinterGenericCommand, and minimizes processing and memory overhead (#13499)
This PR introduces a dedicated implementation for the SMEMBERS command
that avoids using the more generalized sinterGenericCommand function.
By tailoring the logic specifically for SMEMBERS, we reduce unnecessary
processing and memory overheads that were previously incurred by
handling more complex cases like set intersections.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-09-03 18:32:43 +08:00
Filipe Oliveira (Redis)andGitHub a31b516e25 Optimize the HELLO command reply (#13490)
# Overall improvement

TBD ( current is approximately 6% on the achievable ops/sec), coming
from:

- In case of no module we can skip 1.3% CPU cycles on dict Iterator
creation/deletion
- Use addReplyBulkCBuffer instead of addReplyBulkCString to avoid
runtime strlen overhead within HELLO reply on string constants.

## Optimization 1: In case of no module we can skip 1.3% CPU cycles on
dict Iterator creation/deletion.

## Optimization 2: Use addReplyBulkCBuffer instead of
addReplyBulkCString to avoid runtime strlen overhead within HELLO reply
on string constants.
2024-09-03 17:27:35 +08:00
c77b8f45e9 Fixed variable parameter formatting issues in serverPanic function (#13504)
Currently aeApiPoll panic does not record error code information. Added
variable parameter formatting to _serverPanic to fix the issue

---------

Co-authored-by: yingyin.chen <15816602944@163.com>
2024-09-03 15:51:46 +08:00
Ozan TezcanandGitHub a7afd1d2b2 Reply LOADING on replica while flushing the db (#13495)
On a full sync, replica starts discarding existing db. If the existing 
db is huge and flush is happening synchronously, replica may become 
unresponsive. 

Adding a change to yield back to event loop while flushing db on 
a replica. Replica will reply -LOADING in this case. Note that while 
replica is loading the new rdb, it may get an error and start flushing
the partial db. This step may take a long time as well. Similarly, 
replica will reply -LOADING in this case. 

To call processEventsWhileBlocked() and reply -LOADING, we need to do:
- Set connSetReadHandler() null not to process further data from the master
- Set server.loading flag
- Call blockingOperationStarts()

rdbload() already does these steps and calls processEventsWhileBlocked()
while loading the rdb. Added a new call rdbLoadWithEmptyFunc() which 
accepts callback to flush db before loading rdb or when an error 
happens while loading. 

For diskless replication, doing something similar and calling emptyData()
after setting required flags.

Additional changes:
- Allow `appendonly` config change during loading. 
 Config can be changed while loading data on startup or on replication 
 when slave is loading RDB. We allow config change command to update 
 `server.aof_enabled` and then lazily apply config change after loading
 operation is completed.
 
 - Added a test for `replica-lazy-flush` config
2024-09-03 09:48:44 +03:00
Oran AgraandGitHub 3fcddfb61f testsuite --dump-logs works on servers started before the test (#13500)
so far ./runtest --dump-logs used work for servers started within the
test proc.
now it'll also work on servers started outside the test proc scope.
the downside is that these logs can be huge if they served many tests
and not just the failing one.
but for some rare failures, we rather have that than nothing.
this feature isn't enabled y default, but is used by our GH actions.
2024-08-29 07:27:23 +01:00
CoolThiandGitHub 3c9f5954b5 Remove variable expired in expireSlaveKeys() to prevent confusing the compiler (#13299)
This change prevents missed optimization for some compilers:
https://godbolt.org/z/W66h86E13 (the reduced intermediate form in
optimization).
2024-08-28 21:52:23 +08:00
Raz MonsonegoandGitHub 3b1b1d1486 MOD-7645: Return module commands in ACL CAT (#13489)
Currently, module commands are not returned for the `ACL CAT <category>`
command, but skipped instead. Since now modules can add ACL categories
they should no longer be skipped.
2024-08-26 19:25:22 +01:00
paoloredisandGitHub 60f22ca830 Add redis_docs_sync workflow (#13426)
This workflow executes on releases. It invokes the `redis_docs_sync`
workflow on `redis/docs`.
2024-08-21 09:41:14 +01:00
6ceadfb580 Improve GETRANGE command behavior (#12272)
Fixed the issue about GETRANGE and SUBSTR command
return unexpected result caused by the `start` and `end` out of
definition range of string.

---
## break change
Before this PR, when negative `end` was out of range (i.e., end <
-strlen), we would fix it to 0 to get the substring, which also resulted
in the first character still being returned for this kind of out of
range.
After this PR, we ensure that `GETRANGE` returns an empty bulk when the
negative end index is out of range.

Closes #11738

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-08-20 12:34:43 +08:00
judengandGitHub 7f0a7f0a69 improve performance for scan command when matching data type (#12395)
Move the TYPE filtering to the scan callback so that avoided the
`lookupKey` operation. This is the follow-up to #12209 . In this thread
we introduced two breaking changes:
1. we will not attempt to do lazy expire (delete) a key that was
filtered by not matching the TYPE (like we already do for MATCH
pattern).
2. when the specified key TYPE filter is an unknown type, server will
reply a error immediately instead of doing a full scan that comes back
empty handed.
2024-08-20 10:47:51 +08:00
Meir Shpilraien (Spielrein)andGitHub 3264deb24e Avoid used_memory contention when update from multiple threads. (#13431)
The PR attempt to avoid contention on the `used_memory` global variable
when allocate or free memory from multiple threads at the same time.

Each time a thread is allocating or releasing a memory, it needs to
update the `used_memory` global variable. This update might cause a
contention when done aggressively from multiple threads.

### The solution

Instead of having a single global variable that need to be updated from
multiple thread. We create an array of used_memory, each entry in the
array is updated by a single thread and the main thread summarizes all
the values to accumulate the memory usage.

This solution, though reduces the contention between threads on updating
the `used_memory` global variable, it adds work to the main thread that
need to summarize all the entries at the `used_memory` array. To avoid
increasing the work done by the main thread by too much, we limit the
size of the used memory array to 16. This means that up to 16 threads
can run without any contention between them. If there are more than 16
threads, we will reuse entries on the used_memory array, in this case we
might still have contention between threads, but it will be much less
significant.

Notice, that in order to really avoid contention, the entries in the
`used_memory` array must reside on different cache lines. To achieve
that we create a struct with padding such that its size will be exactly
cache_line size. In addition we make sure the address of the
`used_memory` array will be aligned to cache_line size.

### Benchmark

Some benchmark shows improvement (up to 15%):

| Test Case |Baseline unstable (median obs. +- std.dev)|Comparison
test_used_memory_per_thread_array (median obs. +- std.dev)|% change
(higher-better)| Note |

|-------------------------------------------------------------------------------|------------------------------------------|--------------------------------------------------------------------:|------------------------|------------------------------------|
|memtier_benchmark-1key-list-100-elements-lrange-all-elements | 92657 +-
2.0% (2 datapoints) | 101445|9.5% |IMPROVEMENT |
|memtier_benchmark-1key-list-1K-elements-lrange-all-elements | 14965 +-
1.3% (2 datapoints) | 16296|8.9% |IMPROVEMENT |
|memtier_benchmark-1key-set-10-elements-smembers-pipeline-10 | 431019 +-
5.2% (2 datapoints) | 461039|7.0% |waterline=5.2%. IMPROVEMENT |
|memtier_benchmark-1key-set-100-elements-smembers | 74367 +- 0.0% (2
datapoints) | 80190|7.8% |IMPROVEMENT |
|memtier_benchmark-1key-set-1K-elements-smembers | 11730 +- 0.4% (2
datapoints) | 13519|15.3% |IMPROVEMENT |


Full results:

| Test Case |Baseline unstable (median obs. +- std.dev)|Comparison
test_used_memory_per_thread_array (median obs. +- std.dev)|% change
(higher-better)| Note |

|-------------------------------------------------------------------------------|------------------------------------------|--------------------------------------------------------------------:|------------------------|------------------------------------|
|memtier_benchmark-10Mkeys-load-hash-5-fields-with-1000B-values | 88613
+- 1.0% (2 datapoints) | 88688|0.1% |No Change |

|memtier_benchmark-10Mkeys-load-hash-5-fields-with-1000B-values-pipeline-10
| 124786 +- 1.2% (2 datapoints) | 123671|-0.9% |No Change |
|memtier_benchmark-10Mkeys-load-hash-5-fields-with-100B-values | 122460
+- 1.4% (2 datapoints) | 122990|0.4% |No Change |

|memtier_benchmark-10Mkeys-load-hash-5-fields-with-100B-values-pipeline-10
| 333384 +- 5.1% (2 datapoints) | 319221|-4.2% |waterline=5.1%.
potential REGRESSION|
|memtier_benchmark-10Mkeys-load-hash-5-fields-with-10B-values | 137354
+- 0.3% (2 datapoints) | 138759|1.0% |No Change |

|memtier_benchmark-10Mkeys-load-hash-5-fields-with-10B-values-pipeline-10
| 401261 +- 4.3% (2 datapoints) | 398524|-0.7% |No Change |
|memtier_benchmark-1Mkeys-100B-expire-use-case | 179058 +- 0.4% (2
datapoints) | 180114|0.6% |No Change |
|memtier_benchmark-1Mkeys-10B-expire-use-case | 180390 +- 0.2% (2
datapoints) | 180401|0.0% |No Change |
|memtier_benchmark-1Mkeys-1KiB-expire-use-case | 175993 +- 0.7% (2
datapoints) | 175147|-0.5% |No Change |
|memtier_benchmark-1Mkeys-4KiB-expire-use-case | 165771 +- 0.0% (2
datapoints) | 164434|-0.8% |No Change |
|memtier_benchmark-1Mkeys-bitmap-getbit-pipeline-10 | 931339 +- 2.1% (2
datapoints) | 929487|-0.2% |No Change |
|memtier_benchmark-1Mkeys-generic-exists-pipeline-10 | 999462 +- 0.4% (2
datapoints) | 963226|-3.6% |potential REGRESSION |
|memtier_benchmark-1Mkeys-generic-expire-pipeline-10 | 905333 +- 1.4% (2
datapoints) | 896673|-1.0% |No Change |
|memtier_benchmark-1Mkeys-generic-expireat-pipeline-10 | 885015 +- 1.0%
(2 datapoints) | 865010|-2.3% |No Change |
|memtier_benchmark-1Mkeys-generic-pexpire-pipeline-10 | 897115 +- 1.2%
(2 datapoints) | 887544|-1.1% |No Change |
|memtier_benchmark-1Mkeys-generic-scan-pipeline-10 | 451103 +- 3.2% (2
datapoints) | 465571|3.2% |potential IMPROVEMENT |
|memtier_benchmark-1Mkeys-generic-touch-pipeline-10 | 996809 +- 0.6% (2
datapoints) | 984478|-1.2% |No Change |
|memtier_benchmark-1Mkeys-generic-ttl-pipeline-10 | 979570 +- 1.7% (2
datapoints) | 958752|-2.1% |No Change |

|memtier_benchmark-1Mkeys-hash-hget-hgetall-hkeys-hvals-with-100B-values
| 180888 +- 0.5% (2 datapoints) | 182295|0.8% |No Change |

|memtier_benchmark-1Mkeys-hash-hmget-5-fields-with-100B-values-pipeline-10
| 717881 +- 1.0% (2 datapoints) | 724814|1.0% |No Change |
|memtier_benchmark-1Mkeys-hash-transactions-multi-exec-pipeline-20 |
1055447 +- 0.4% (2 datapoints) | 1065836|1.0% |No Change |
|memtier_benchmark-1Mkeys-lhash-hexists | 164332 +- 0.1% (2 datapoints)
| 163636|-0.4% |No Change |
|memtier_benchmark-1Mkeys-lhash-hincbry | 171674 +- 0.3% (2 datapoints)
| 172737|0.6% |No Change |
|memtier_benchmark-1Mkeys-list-lpop-rpop-with-100B-values | 180904 +-
1.1% (2 datapoints) | 179467|-0.8% |No Change |
|memtier_benchmark-1Mkeys-list-lpop-rpop-with-10B-values | 181746 +-
0.8% (2 datapoints) | 182416|0.4% |No Change |
|memtier_benchmark-1Mkeys-list-lpop-rpop-with-1KiB-values | 182004 +-
0.7% (2 datapoints) | 180237|-1.0% |No Change |
|memtier_benchmark-1Mkeys-load-hash-5-fields-with-1000B-values | 105191
+- 0.9% (2 datapoints) | 105058|-0.1% |No Change |

|memtier_benchmark-1Mkeys-load-hash-5-fields-with-1000B-values-pipeline-10
| 150683 +- 0.9% (2 datapoints) | 153597|1.9% |No Change |
|memtier_benchmark-1Mkeys-load-hash-hmset-5-fields-with-1000B-values |
104122 +- 0.7% (2 datapoints) | 105236|1.1% |No Change |
|memtier_benchmark-1Mkeys-load-list-with-100B-values | 149770 +- 0.9% (2
datapoints) | 150510|0.5% |No Change |
|memtier_benchmark-1Mkeys-load-list-with-10B-values | 165537 +- 1.9% (2
datapoints) | 164329|-0.7% |No Change |
|memtier_benchmark-1Mkeys-load-list-with-1KiB-values | 113315 +- 0.5% (2
datapoints) | 114110|0.7% |No Change |
|memtier_benchmark-1Mkeys-load-stream-1-fields-with-100B-values | 131201
+- 0.7% (2 datapoints) | 129545|-1.3% |No Change |

|memtier_benchmark-1Mkeys-load-stream-1-fields-with-100B-values-pipeline-10
| 352891 +- 2.8% (2 datapoints) | 348338|-1.3% |No Change |
|memtier_benchmark-1Mkeys-load-stream-5-fields-with-100B-values | 104386
+- 0.7% (2 datapoints) | 105796|1.4% |No Change |

|memtier_benchmark-1Mkeys-load-stream-5-fields-with-100B-values-pipeline-10
| 227593 +- 5.5% (2 datapoints) | 218783|-3.9% |waterline=5.5%.
potential REGRESSION|
|memtier_benchmark-1Mkeys-load-string-with-100B-values | 167552 +- 0.2%
(2 datapoints) | 170282|1.6% |No Change |
|memtier_benchmark-1Mkeys-load-string-with-100B-values-pipeline-10 |
646888 +- 0.5% (2 datapoints) | 639680|-1.1% |No Change |
|memtier_benchmark-1Mkeys-load-string-with-10B-values | 174891 +- 0.7%
(2 datapoints) | 174382|-0.3% |No Change |
|memtier_benchmark-1Mkeys-load-string-with-10B-values-pipeline-10 |
749988 +- 5.1% (2 datapoints) | 769986|2.7% |waterline=5.1%. No Change |
|memtier_benchmark-1Mkeys-load-string-with-1KiB-values | 155929 +- 0.1%
(2 datapoints) | 156387|0.3% |No Change |
|memtier_benchmark-1Mkeys-load-zset-with-10-elements-double-score |
92241 +- 0.2% (2 datapoints) | 92189|-0.1% |No Change |
|memtier_benchmark-1Mkeys-load-zset-with-10-elements-int-score | 114328
+- 1.3% (2 datapoints) | 113154|-1.0% |No Change |
|memtier_benchmark-1Mkeys-string-get-100B | 180685 +- 0.2% (2
datapoints) | 180359|-0.2% |No Change |
|memtier_benchmark-1Mkeys-string-get-100B-pipeline-10 | 991291 +- 3.1%
(2 datapoints) | 1020086|2.9% |No Change |
|memtier_benchmark-1Mkeys-string-get-10B | 181183 +- 0.3% (2 datapoints)
| 177868|-1.8% |No Change |
|memtier_benchmark-1Mkeys-string-get-10B-pipeline-10 | 1032554 +- 0.8%
(2 datapoints) | 1023120|-0.9% |No Change |
|memtier_benchmark-1Mkeys-string-get-1KiB | 180479 +- 0.9% (2
datapoints) | 182215|1.0% |No Change |
|memtier_benchmark-1Mkeys-string-get-1KiB-pipeline-10 | 979286 +- 0.9%
(2 datapoints) | 989888|1.1% |No Change |
|memtier_benchmark-1Mkeys-string-mget-1KiB | 121950 +- 0.4% (2
datapoints) | 120996|-0.8% |No Change |
|memtier_benchmark-1key-geo-60M-elements-geodist | 179404 +- 1.0% (2
datapoints) | 181232|1.0% |No Change |
|memtier_benchmark-1key-geo-60M-elements-geodist-pipeline-10 | 1023797
+- 0.5% (2 datapoints) | 1014980|-0.9% |No Change |
|memtier_benchmark-1key-geo-60M-elements-geohash | 180808 +- 1.2% (2
datapoints) | 180606|-0.1% |No Change |
|memtier_benchmark-1key-geo-60M-elements-geohash-pipeline-10 | 1056458
+- 1.6% (2 datapoints) | 1040050|-1.6% |No Change |
|memtier_benchmark-1key-geo-60M-elements-geopos | 181808 +- 0.2% (2
datapoints) | 175945|-3.2% |potential REGRESSION |
|memtier_benchmark-1key-geo-60M-elements-geopos-pipeline-10 | 1038180 +-
3.4% (2 datapoints) | 1033005|-0.5% |No Change |
|memtier_benchmark-1key-geo-60M-elements-geosearch-fromlonlat | 142614
+- 0.3% (2 datapoints) | 144259|1.2% |No Change |
|memtier_benchmark-1key-geo-60M-elements-geosearch-fromlonlat-bybox |
141008 +- 0.4% (2 datapoints) | 139602|-1.0% |No Change |

|memtier_benchmark-1key-geo-60M-elements-geosearch-fromlonlat-pipeline-10
| 560698 +- 0.8% (2 datapoints) | 548806|-2.1% |No Change |
|memtier_benchmark-1key-list-10-elements-lrange-all-elements | 166132 +-
0.9% (2 datapoints) | 170259|2.5% |No Change |
|memtier_benchmark-1key-list-100-elements-lrange-all-elements | 92657 +-
2.0% (2 datapoints) | 101445|9.5% |IMPROVEMENT |
|memtier_benchmark-1key-list-1K-elements-lrange-all-elements | 14965 +-
1.3% (2 datapoints) | 16296|8.9% |IMPROVEMENT |
|memtier_benchmark-1key-pfadd-4KB-values-pipeline-10 | 264156 +- 0.2% (2
datapoints) | 262582|-0.6% |No Change |
|memtier_benchmark-1key-set-10-elements-smembers | 138916 +- 1.7% (2
datapoints) | 138016|-0.6% |No Change |
|memtier_benchmark-1key-set-10-elements-smembers-pipeline-10 | 431019 +-
5.2% (2 datapoints) | 461039|7.0% |waterline=5.2%. IMPROVEMENT |
|memtier_benchmark-1key-set-10-elements-smismember | 173545 +- 1.1% (2
datapoints) | 173488|-0.0% |No Change |
|memtier_benchmark-1key-set-100-elements-smembers | 74367 +- 0.0% (2
datapoints) | 80190|7.8% |IMPROVEMENT |
|memtier_benchmark-1key-set-100-elements-smismember | 155682 +- 1.6% (2
datapoints) | 151367|-2.8% |No Change |
|memtier_benchmark-1key-set-1K-elements-smembers | 11730 +- 0.4% (2
datapoints) | 13519|15.3% |IMPROVEMENT |
|memtier_benchmark-1key-set-200K-elements-sadd-constant | 181070 +- 1.1%
(2 datapoints) | 180214|-0.5% |No Change |
|memtier_benchmark-1key-set-2M-elements-sadd-increasing | 166364 +- 0.1%
(2 datapoints) | 166944|0.3% |No Change |
|memtier_benchmark-1key-zincrby-1M-elements-pipeline-1 | 46071 +- 0.6%
(2 datapoints) | 44979|-2.4% |No Change |
|memtier_benchmark-1key-zrank-1M-elements-pipeline-1 | 48429 +- 0.4% (2
datapoints) | 49265|1.7% |No Change |
|memtier_benchmark-1key-zrem-5M-elements-pipeline-1 | 48528 +- 0.4% (2
datapoints) | 48869|0.7% |No Change |
|memtier_benchmark-1key-zrevrangebyscore-256K-elements-pipeline-1 |
100580 +- 1.5% (2 datapoints) | 101782|1.2% |No Change |
|memtier_benchmark-1key-zrevrank-1M-elements-pipeline-1 | 48621 +- 2.0%
(2 datapoints) | 48473|-0.3% |No Change |
|memtier_benchmark-1key-zset-10-elements-zrange-all-elements | 83485 +-
0.6% (2 datapoints) | 83095|-0.5% |No Change |

|memtier_benchmark-1key-zset-10-elements-zrange-all-elements-long-scores
| 118673 +- 0.8% (2 datapoints) | 118006|-0.6% |No Change |
|memtier_benchmark-1key-zset-100-elements-zrange-all-elements | 19009 +-
1.1% (2 datapoints) | 19293|1.5% |No Change |
|memtier_benchmark-1key-zset-100-elements-zrangebyscore-all-elements |
18957 +- 0.5% (2 datapoints) | 19419|2.4% |No Change |

|memtier_benchmark-1key-zset-100-elements-zrangebyscore-all-elements-long-scores|
171693 +- 0.5% (2 datapoints) | 172432|0.4% |No Change |
|memtier_benchmark-1key-zset-1K-elements-zrange-all-elements | 3566 +-
0.6% (2 datapoints) | 3672|3.0% |No Change |
|memtier_benchmark-1key-zset-1M-elements-zcard-pipeline-10 | 1067713 +-
0.4% (2 datapoints) | 1071550|0.4% |No Change |
|memtier_benchmark-1key-zset-1M-elements-zrevrange-5-elements | 169195
+- 0.7% (2 datapoints) | 169620|0.3% |No Change |
|memtier_benchmark-1key-zset-1M-elements-zscore-pipeline-10 | 914338 +-
0.2% (2 datapoints) | 905540|-1.0% |No Change |
|memtier_benchmark-2keys-lua-eval-hset-expire | 88346 +- 1.7% (2
datapoints) | 87259|-1.2% |No Change |
|memtier_benchmark-2keys-lua-evalsha-hset-expire | 103273 +- 1.2% (2
datapoints) | 102393|-0.9% |No Change |
|memtier_benchmark-2keys-set-10-100-elements-sdiff | 15418 +- 10.9%
UNSTABLE (2 datapoints) | 14369|-6.8% |UNSTABLE (very high variance) |
|memtier_benchmark-2keys-set-10-100-elements-sinter | 83601 +- 3.6% (2
datapoints) | 82508|-1.3% |No Change |
|memtier_benchmark-2keys-set-10-100-elements-sunion | 14942 +- 11.2%
UNSTABLE (2 datapoints) | 14001|-6.3% |UNSTABLE (very high variance) |
|memtier_benchmark-2keys-stream-5-entries-xread-all-entries | 75938 +-
0.4% (2 datapoints) | 76565|0.8% |No Change |
|memtier_benchmark-2keys-stream-5-entries-xread-all-entries-pipeline-10
| 120781 +- 1.1% (2 datapoints) | 119142|-1.4% |No Change |
2024-08-19 13:22:16 +03:00
debing.sunandGitHub 6c6489280c Fix a race condition issue in the cache_memory of functionsLibCtx (#13476)
This is a missing of the PR https://github.com/redis/redis/pull/13383.
We will call `functionsLibCtxClear()` in bio, so we shouldn't touch
`curr_functions_lib_ctx` in it.
2024-08-19 10:11:45 +08:00
debing.sunandGitHub 2b88db90aa Fix incorrect lag due to trimming stream via XTRIM command (#13473)
## Describe
When using the `XTRIM` command to trim a stream, it does not update the
maximal tombstone (`max_deleted_entry_id`). This leads to an issue where
the lag calculation incorrectly assumes that there are no tombstones
after the consumer group's last_id, resulting in an inaccurate lag.

The reason XTRIM doesn't need to update the maximal tombstone is that it
always trims from the beginning of the stream. This means that it
consistently changes the position of the first entry, leading to the
following scenarios:

1) First entry trimmed after maximal tombstone:
If the first entry is trimmed to a position after the maximal tombstone,
all tombstones will be before the first entry, so they won't affect the
consumer group's lag.

2) First entry trimmed before maximal tombstone:
If the first entry is trimmed to a position before the maximal
tombstone, the maximal tombstone will not be updated.

## Solution
Therefore, this PR optimizes the lag calculation by ensuring that when
both the consumer group's last_id and the maximal tombstone are behind
the first entry, the consumer group's lag is always equal to the number
of remaining elements in the stream.

Supplement to PR https://github.com/redis/redis/pull/13338
2024-08-16 23:13:31 +08:00
debing.sunandGitHub b94b714f81 Fix error message for XREAD command with wrong parameter (#13474)
Fixed a missing from #13117.
When the number of streams is incorrect, the error message for `XREAD`
needs to include the '+' symbol.
2024-08-14 21:40:43 +08:00
Moti CohenandGitHub 806459f481 On HDEL last field with expiry, update global HFE DS (#13470)
Hash field expiration is optimized to avoid frequent update global HFE DS for
each field deletion. Eventually active-expiration will run and update or remove
the hash from global HFE DS gracefully. Nevertheless, statistic "subexpiry"
might reflect wrong number of hashes with HFE to the user if HDEL deletes
the last field with expiration in hash (yet there are more fields without expiration).

Following this change, if HDEL the last field with expiration in the hash then
take care to remove the hash from global HFE DS as well.
2024-08-11 16:39:03 +03:00
3a08819f51 Fix some memory leaks in redis-cli (#13258)
Fix memory leak related to variable slot_nodes  in the
clusterManagerFixSlotsCoverage() function.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-08-08 10:51:33 +08:00
6f0ddc9d92 Pass extensions to node if extension processing is handled by it (#13465)
This PR is based on the commits from PR
https://github.com/valkey-io/valkey/pull/52.
Ref: https://github.com/redis/redis/pull/12760
Close https://github.com/redis/redis/issues/13401
This PR will replace https://github.com/redis/redis/pull/13449

Fixes compatibilty of Redis cluster (7.2 - extensions enabled by
default) with older Redis cluster (< 7.0 - extensions not handled) .

With some of the extensions enabled by default in 7.2 version, new nodes
running 7.2 and above start sending out larger clusterbus message
payload including the ping extensions. This caused an incompatibility
with node running engine versions < 7.0. Old nodes (< 7.0) would receive
the payload from new nodes (> 7.2) would observe a payload length
(totlen) > (estlen) and would perform an early exit and won't process
the message.

This fix does the following things:
1. Always set `CLUSTERMSG_FLAG0_EXT_DATA`, because during the meet
phase, we do not know whether the connected node supports ext data, we
need to make sure that it knows and send back its ext data if it has.
2. If another node does not support ext data, we will not send it ext
data to avoid the handshake failure due to the incorrect payload length.

Note: A successful `PING`/`PONG` is required as a sender for a given
node to be marked as `CLUSTERMSG_FLAG0_EXT_DATA` and then extensions
message
will be sent to it. This could cause a slight delay in receiving the
extensions message(s).

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
Co-authored-by: Harkrishn Patro <harkrisp@amazon.com>

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
Co-authored-by: Harkrishn Patro <harkrisp@amazon.com>
2024-08-08 10:48:03 +08:00
731f2dc5c7 Make some commets more friendly (#13319)
Close  #13316

---------

Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: debing.sun <debing.sun@redis.com>
2024-08-07 01:20:06 +08:00
debing.sunandGitHub bf643a63c8 Ensure validity of myself as master or replica when loading cluster config (#13443)
First, we need to ensure that `curmaster` in
`clusterUpdateSlotsConfigWith()` is not NULL in the line
https://github.com/redis/redis/blob/82f00f5179720c8cee6cd650763d184ba943be92/src/cluster_legacy.c#L2320
otherwise, it will crash in the
https://github.com/redis/redis/blob/82f00f5179720c8cee6cd650763d184ba943be92/src/cluster_legacy.c#L2395

So when loading cluster node config, we need to ensure that the
following conditions are met:
1. A node must be at least one of the master or replica.
2. If a node is a replica, its master can't be NULL.
2024-08-06 20:40:46 +08:00
YaacovHazanandGitHub e4ddc34463 Keep cluster shards command implementation generic (#13440)
Make the clusterCommandShards function use only cluster API functions
instead of accessing cluster implementation details.
This way the cluster API implementation doesn't have to have intimate
knowledge of the command reply format, and doesn't need to interact with
the client directly (the addReply function family).
The PR has two commits, one moves the function from cluster_legacy.c to
cluster.c, and the other modifies it's implementation.


**better merge without squashing.**
2024-08-05 11:08:48 +03:00
Josh Hershberg 6d5d754119 Make cluster shards cmd implementation generic
This and the previous commit make the cluster
shards command a generic implementation instead of a
specific implementation for each cluster API implementation.
This commit (a) adds functions to the cluster API
and (b) modifies the cluster shards cmd implementation
to use cluster API functions instead of directly
accessing the legacy clustering implementation.

Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2024-08-05 10:31:40 +03:00
Josh Hershberg e3e631f394 Prep to make cluster shards cmd generic
This and the next following commit makes the cluster
shards command a generic implementation instead of a
specific implementation for each cluster API implementation.
This commit simply moves the cluster shards implementation
from cluster_legacy.c to cluster.c without changing the
implementation at all. The reason for doing so was to
help with reviewing the changes in the diff.

Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2024-08-05 10:27:03 +03:00
6263823e54 Replace bit shift with __builtin_ctzll in HyperLogLog (#13218)
## Replace bit shift with `__builtin_ctzll` in HyperLogLog

Builtin function `__builtin_ctzll` is more effective than bit shift even
though "in the average case there are high probabilities to find a 1
after a few iterations" mentioned in the source file comment.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-08-05 10:51:43 +08:00
Moti CohenandGitHub 4dd8b1faa9 Fix HTTL/HPTTL to be NONDETERMINISTIC_OUTPUT (#13461)
H[P]TTL should be marked as NONDETERMINISTIC_OUTPUT just like [P]TTL.
2024-08-04 17:42:50 +03:00
8038eb3147 Fix wrong dbnum showed in redis-cli after client reconnected (#13411)
When the server restarts while the CLI is connecting, the reconnection
does not automatically select the previous db.
This may lead users to believe they are still in the previous db, in
fact, they are in db0.

This PR will automatically reset the current dbnum and `cliSelect()`
again when reconnecting.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-08-03 12:06:02 +08:00
c8efandGitHub 89742a95db Fix typo in hyperloglog.c (#13458) 2024-08-02 07:49:52 +08:00
debing.sunandGitHub 60e9e630bd Fix CLUSTER SHARDS command returns empty array (#13422)
Close https://github.com/redis/redis/issues/13414

When the cluster's master node fails and is switched to another node,
the first node in the shard node list (the old master) is no longer
valid.
Add a new method clusterGetMasterFromShard() to obtain the current
master.
2024-08-02 07:22:13 +08:00
e750c619b2 Fix some test failures caused by key being deleted due to premature expiration (#13453)
1. Fix fuzzer test failure when the key was deleted due to expiration
before sending random traffic for the key.

After HFE, when all fields in a hash are expired, the hash might be
deleted due to expiration.

If the key was expired in the mid of `RESTORE` command and sending rand
trafic, `fuzzer` test will fail in the following code because the 'TYPE
key' will return `none` and then throw an exception because it cannot be
found in `$commands`

https://github.com/redis/redis/blob/94b9072e44af0bae8cfe2de5cfa4af7b8e399759/tests/support/util.tcl#L712-L713

This PR adds a `None` check for the reply of `KEY TYPE` command and adds
a print of `err` to avoid false positives in the future.

failed CI:
https://github.com/redis/redis/actions/runs/10127334121/job/28004985388

2. Fix the issue where key was deleted due to expiration before the
`scan.scan_key` command could execute, caused by premature enabling of
`set-active-expire`.

failed CI:
https://github.com/redis/redis/actions/runs/10153722715/job/28077610552

---------

Co-authored-by: oranagra <oran@redislabs.com>
2024-07-31 08:15:39 +08:00
debing.sunandGitHub 93fb83b4cb Fix incorrect lag field in XINFO when tombstone is after the last_id of consume group (#13338)
Fix #13337

Ths PR fixes fixed two bugs that caused lag calculation errors.
1. When the latest tombstone is before the first entry, the tombstone
may stil be after the last id of consume group.
2. When a tombstone is after the last id of consume group, the group's
counter will be invalid, we should caculate the entries_read by using
estimates.
2024-07-30 22:31:31 +08:00
Lior KoganandGitHub 94b9072e44 Rename to "Redis Community Edition" (#13448) 2024-07-28 20:54:28 +03:00
Oran AgraandGitHub e74550dd10 solve races in replication lpop tests (#13445)
* some tests didn't wait for replication offset sync
* tests that used deferring client, didn't wait for it to get blocked.
an in some cases, the replication offset sync ended before the deferring
client finished, so the digest match failed.
* some tests used deferring clients excessively
* the tests didn't read the client response
* the tests didn't close the client (fd leak)
2024-07-25 14:06:40 +03:00
Moti CohenandGitHub d0c64d78d4 On active expire, factor maxToExpire based on Hertz (#13439) 2024-07-25 13:22:02 +03:00
Moti CohenandGitHub 82f00f5179 Optimize RDB_TYPE_HASH_METADATA to keep relative expiration time (#13438)
Modify RDB_TYPE_HASH_METADATA layout to store expiration times relative
to the minimum expiration time, which is written at the start as absolute time.
2024-07-24 08:39:10 +03:00
Oran AgraandGitHub 447ce11a64 solve race conditions in tests (#13433)
[exception]: Executing test client: ERR FAILOVER target replica is not
online.. ERR FAILOVER target replica is not online.
    while executing
"$node_0 failover to $node_1_host $node_1_port"
    ("uplevel" body line 16)
    invoked from within
"uplevel 1 $code"
    (procedure "test" line 58)
    invoked from within
"test {failover command to specific replica works} {

[err]: client evicted due to percentage of maxmemory in
tests/unit/client-eviction.tcl
Expected 33622 >= 220200 && 33622 < 440401 (context: type eval line 17
cmd {assert {$tot_mem >= $n && $tot_mem < $maxmemory_clients_actual}}
proc ::test)
2024-07-22 10:11:56 +03:00
Oran AgraandGitHub 13d227fa46 Different fix for the race in #13361 (#13434)
Recently in #13361, i attempted to fix a race between FLUSHALL and
BGSAVE, where despite calling killRDBChild, the
backgroundSaveDoneHandler will terminate with success.
Turns out that even if the child didn't yet exit, there's a chance it'll
still miss our signal and exit with success.
in that case, we will still mess up the dirty counter (deducting
dirty_before_bgsave) which is reset by FLUSHALL, and override the
synchronous rdb file we saved.

instead, we'll set a flag to treat the next done handler as a failed
one.
2024-07-22 10:11:30 +03:00
Oran AgraandGitHub a331978583 Fix external test hang in redis-cli test when run in a certain order (#13423)
When the tests are run against an external server in this order:
`--single unit/introspection --single unit/moduleapi/blockonbackground
--single integration/redis-cli`
the test would hang when the "ASK redirect test" test attempts to create
a listening socket (it fails, and then redis-cli itself hangs waiting
for a non-responsive socket created by the introspection test).

the reasons are:
1. the blockedbackground test includes util.tcl and resets the
`::last_port_attempted` variable
2. the test in introspection didn't close the listening server, so it's
still alive.
3. find_available_port doesn't properly detect the busy port, and it
thinks that the port is free even though it's busy.

fixing all 3 of these problems, even though fixing just one would be
enough to let the test pass.
2024-07-17 15:42:28 +03:00
Oran AgraandGitHub fa46aa4d85 Test infra adjustments for external CI runs (#13421)
- when uploading server logs, make sure they don't overwrite each other.
- sort the test units to get consistent order between them (following
#13220)
- backup and restore the entire server configuration, to protect one
unit from config changes another unit performs
2024-07-16 11:38:20 +03:00
88af96c7a2 Trigger Lua GC after script loading (#13407)
Nowdays we do not trigger LUA GC after loading lua script. This means
that when a large number of scripts are loaded, such as when functions
are propagating from the master to the replica, if the LUA scripts are
never touched on the replica, the garbage might remain there
indefinitely.

Before this PR, we would share a gc_count between scripts and functions.
This means that, under certain circumstances, the GC trigger for scripts
and functions was not fair.
For example, loading a large number of scripts followed by a small
number of functions could result in the functions triggering GC.
In this PR, we assign a unique `gc_count` to each of them, so the GC
triggers between them will no longer affect each other.

on the other hand, this PR will to bring regession for script loading
commands(`FUNCTION LOAD` and `SCRIPT LOAD`), but they are not hot path,
we can ignore it, and it will be replaced
https://github.com/redis/redis/pull/13375 in the future.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2024-07-16 09:28:47 +08:00
76415fa2cf Prevent deleting RDB read event after restarting RDB saving for other diskless replicas (#13410)
When we terminate the diskless RDB saving child process and, at the same
time, we start a new BGSAVE for new replicas, we should not delete the
RDB read event. Otherwise, these replicas will never receive a response.
this is a result of the recent change in
https://github.com/redis/redis/pull/13361

---------

Co-authored-by: oranagra <oran@redislabs.com>
2024-07-16 09:22:43 +08:00
7d3545cb16 Reduce redundant call of prepareClientToWrite when call addReply* continuously (#13412)
This PR is based on the commits from PR
https://github.com/valkey-io/valkey/pull/670.

## Description

While exploring hotspots with profiling some benchmark workloads, we
noticed the high cycles ratio of `prepareClientToWrite`, taking about 9%
of the CPU of `smembers`, `lrange` commands. After deep dive the code
logic, we thought we can gain the performance by reducing the redundant
call of `prepareClientToWrite` when call addReply* continuously.

For example: In

https://github.com/valkey-io/valkey/blob/unstable/src/networking.c#L1080-L1082,
`prepareClientToWrite` is called three times in a row.

---------

Signed-off-by: Lipeng Zhu <lipeng.zhu@intel.com>
Co-authored-by: Lipeng Zhu <lipeng.zhu@intel.com>
Co-authored-by: Wangyang Guo <wangyang.guo@intel.com>
2024-07-15 23:19:19 +08:00
Oran AgraandGitHub 880e147d52 solve redis-cli test failures due to local history file (#13419)
test failure:
```
[err]: Interactive CLI: should find second search result if user presses ctrl+s in tests/integration/redis-cli.tcl
Expected '1' to be equal to '0' (context: type eval line 10 cmd {assert_equal 1 [regexp {\(i-search\): \x1B\[0mk\x1B\[1mey\x1B\[0ms one} $result]} proc ::test)
```

this test (introduced in #12543) depends on the local history file, so
it can fail if there's some match there.
the fix is to use a different history file, and delete it before each
run.
2024-07-15 15:24:43 +03:00
guybe7andGitHub b10e19e3d6 Crash report: Use more chars for argv (#13413)
128 is not enough chars when we're talking about commands like RESTORE.
Of course, it's impossible to find the perfect number, but 1024 is
better than 128, and it's not obscenely large.
2024-07-14 11:37:44 +08:00
d39548c854 Avoid starting defrag after config resetstat for defrag test (#13399)
If `config resetstat` is executed and a defrag is started after it, the
`total_active_defrag_time` will not be 0.
When we start the defrag again, we will skip the following steps:
1. waiting for the defrag to start. (s total_active_defrag_time is equal
0)
2. waiting for the test to complete. (active_defrag_running is euqal 0)
which result in the test failed.

---------

Co-authored-by: oranagra <oran@redislabs.com>
2024-07-12 10:07:09 +08:00
debing.sunandGitHub ad9277238f Disable aof for Scripts do not block on waitaof test (#13409)
The`Scripts do not block on waitaof` test keeps failing in the external
CI:
https://github.com/redis/redis/actions/runs/9875670156/job/27272955900?pr=13407

The main reason is that if AOF happens to be enabled, we get an acklocal
of 1.
2024-07-11 20:25:57 +08:00
guybe7andGitHub 915a9e4b93 KSN: HEXPIRE-like commands should emit hdel if expire-time is in the past (#13408)
To be more similar to EXPIRE-like commands, which emit a "del"
notification if the expire-time is in the past
2024-07-11 16:24:58 +08:00
81440a333d Module unblock on keys: updateStatsOnUnblock is called twice (#13405)
This commit reverts the deletion of the condition `!bc->blocked_on_keys`
that was accidentally introduced by
https://github.com/redis/redis/pull/12817.
In case a blocked-on-keys module client is unblocked both
`moduleUnblockClientOnKey` and `moduleHandleBlockedClients` are called
which resulted in `updateStatsOnUnblock` being called twice

Now, that `moduleHandleBlockedClients` doesn't call
`updateStatsOnUnblock` in case of unblocked module key-blocked clients,
in the unlikely event that the module decides to call `RM_UnblockClient`
on a key-blocked client, we need to call `updateStatsOnUnblock` from
within `moduleBlockedClientTimedOut`, but since
`moduleBlockedClientTimedOut` is not tread-safe we can't call it
directly from withing `RM_UnblockClient`.
Added a new flag `blocked_on_keys_explicit_unblock` for that specific
case, which will cause `moduleBlockedClientTimedOut` to be called from
`moduleHandleBlockedClients` (which is only called from the main thread)

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-07-11 16:13:38 +08:00
ffff7fea7c Rebuild function engines for function flush command (#13383)
### Issue
The current implementation of `FUNCTION FLUSH` command uses
`lua_unref()` to unreference script closures in Lua vm. However,
invoking `lua_unref()` during lazy free (`ASYNC` argument) is risky
since it is not thread-safe.

Another issue is that using `lua_unref()` to unreference references does
not trigger GC, This can result in the Lua VM leaves a significant
amount of garbage, which may never be cleaned up if not properly GC.

### Solution
The proposed solution is to completely rebuild the engines, resulting in
a brand new Lua VM.

---------

Co-authored-by: meir <meir@redis.com>
2024-07-10 17:35:36 +08:00
69b480cb7a Hide user data from log (#13400)
This PR is based on the commits from PR #11747.

In the event of an assertion failure, hide command arguments from the
operator.

In some cases, private client information can be voluntarily exposed
when a redis instance crashes due to an assertion failure.
This commit prevent וnintentional client info exposure.
Operators can still access the hidden data, but they must actively
request it.
Any of the client info commands remains the unchanged.

### Config
Add a new config `hide-user-data-from-log` to turn this feature on and
off, default off.

---------

Co-authored-by: naglera <anagler123@gmail.com>
Co-authored-by: naglera <58042354+naglera@users.noreply.github.com>
2024-07-09 18:54:18 +08:00
b699e8bfe0 Upgrade action/checkout version and add old-chain CI actions to test gcc4.8 (#13394)
https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

Due to GitHub removing support for CentOS 7 in GitHub Actions, all
actions utilizing CentOS 7 need to be upgraded, upgrade the centos
version from `contos:7` to `quay.io/centos/centos:stream9` which is the
official RedHat centos container.

Create some new actions named `old-chain` to verify support for gcc 4.8.

This PR also includes the upgrade of actions/checkout from version 3 to
version 4.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-07-08 21:34:14 +08:00
Moti CohenandGitHub dcf0229811 HFE - RDB serialize also hash min expiration (For ROF flow) (#13391)
* Following this feature, Redis (ROF) may implement flow that allows hashes to be
  dumped directly from RDB to FLUSH without parsing. In this scenario, it is still
  essential to determine when to update hashes due to expired fields. By writing
  and reading the next minimum hash-field expiration before serializing objects
  to and from RDB, we can effectively track and expire hash fields without the need
  to parse the hash during loading.

    Before:
    #define RDB_TYPE_HASH_METADATA 22
    #define RDB_TYPE_HASH_LISTPACK_EX 23
    
    After:
    /* Hash with HFEs. Doesn't attach min TTL at start */
    #define RDB_TYPE_HASH_METADATA_PRE_GA 22      
    /* Hash LP with HFEs. Doesn't attach min TTL at start */
    #define RDB_TYPE_HASH_LISTPACK_EX_PRE_GA 23   
    /* Hash with HFEs. Attach min TTL at start */
    #define RDB_TYPE_HASH_METADATA 24             
    /* Hash LP with HFEs. Attach min TTL at start */
    #define RDB_TYPE_HASH_LISTPACK_EX 25          


* Manually test loading RDB file before the change and verify hash and its HFEs 
  is as expected.
* Added `subexpires` counter to `redis-check-rdb`
2024-07-07 17:00:19 +03:00
debing.sunandGitHub 58edea7385 Add help for DEBUG SCRIPT command (#13385)
1. Add help for `DEBUG SCRIPT` command.
2. Remove a duplicate `getLuaScripts()` which is same as
`evalScriptsDict()`.
2024-07-04 14:44:42 +08:00
26a2dcb936 Reduce getNodeByQuery overhead (#13221)
The following PR does the following changes based upon on CPU profile
info. The `getNodeByQuery` function represents 8.2% of an overhead of
12.3% when comparing single shard cluster with standalone.
Proposed changes:
- inlinging keyHashSlot to reduce overhead of that function call
- Reduce duplicate calls to getCommandFlags within getNodeByQuery

The above changes represent an improvement of approximately 5% on the achievable ops/sec.

Co-authored-by: filipecosta90 <filipecosta.90@gmail.com>
2024-07-03 18:23:14 +08:00
Moti CohenandGitHub a84cc20aef HFE - Fix statistic to count also lazy expired and rename INFO params (#13372)
* INFO command : rename `hashes_with_expiry_fields` to `subexpiry`
* INFO command : rename `expired_hash_fields` to `expired_subkeys`
* Fix statistic of `expired_subkeys` to count also lazy expired
* Remove TODOs comments leftover in TCL
* Fix potential flaky test of rdb load of hash-field-expiration
2024-07-02 18:22:10 +03:00
debing.sunandGitHub 76d179c64f Update CentOS7 repo URL for EOL and macOS11 GitHub Action for deprecation (#13365)
1. Update macOS version in GitHub Actions due to deprecation of macOS 11
Because the macOS 11 runner image will be removed by 06/28/2024.

2. CentOS has arrived at EOL, we need to update the yum repo to new url.
2024-07-02 21:53:32 +08:00
David DoughertyandGitHub c6ceea9a23 DOC-3932: remove frontmatter attributes from module API spec generator (#13377)
This PR removes the Hugo frontmatter from this utility.
2024-07-02 08:25:42 +03:00
Oran AgraandGitHub 799c5e5f51 Solve a race between BGSAVE and FLUSHALL messing up the dirty counter (#13361)
If we run FLUSHALL when the 'save' config is set, and there's a fork
child ding BGSAVE, there's a chance the child is already finished, and
the parent process is unaware of it. in that case the child will not get
the kill signal and will finish successfully, but the parent process
thinks it killed it and will reset the dirty counter to 0, then the
backgroundSaveDoneHandlerDisk method can set the dirty counter to a
negative value.
2024-07-01 09:04:52 +03:00
Oran AgraandGitHub 69b7137d32 Fix possible crash due to OOM panic on invalid command (#13380)
getKeysUsingKeySpece had the range check AFTER the allocation, of the
keys buffer, which could lead to an OOM panic when invalid arguments are
provided, leading to an overflow.
The allocated memory is only used after the range check, so there's no
risk of buffer overrun.
The OOM panic can happen on 32bit builds, or 64 builds running on
systems with less than 4GB of RAM, and is reachable via the COMMAND
GETKEYSANDFLAGS, and ACL key name validation.
2024-07-01 09:04:20 +03:00
Moti CohenandGitHub a9267137ee HFE - count in command must match actual number of fields (#13369)
There was wrong preliminary assumption that we can optionally provide
vector of arguments more than count.
This is error-prone approach that leaded to actual error in that case.
This PR enforce that vector of argument match count.

Also fixed flaky HRANDFIELD test.
2024-06-26 14:12:06 +03:00
52e12d8bac Don't keep global replication buffer reference for replicas marked CLIENT_CLOSE_ASAP (#13363)
In certain situations, we might generate a large number of propagates
(e.g., multi/exec, Lua script, or a single command generating tons of
propagations) within an event loop.
During the process of propagating to a replica, if the replica is
disconnected(marked as CLIENT_CLOSE_ASAP) due to exceeding the output
buffer limit, we should remove its reference to the global replication
buffer to avoid the global replication buffer being unable to be
properly trimmed due to being referenced.

---------

Co-authored-by: oranagra <oran@redislabs.com>
2024-06-26 08:26:23 +08:00
Moti CohenandGitHub 5eac99c312 Fix H(P)EXPIREAT command to propagate HDEL as well (#13364)
H(P)EXPIREAT command might delete fields in case the absolute time is in the 
past. Those HDELs need to be propagated as well.
 
In general, as we need to propagate H(P)EXPIRE(AT) command to the replica, each 
field that is mentioned in the command should be categorized into one of the four
options:
1. Managed to update field’s expiration time - propagate it to replica as part 
   of the HPEXPIREAT command.
2. Deleted the field because the time is in the past - propagate also HDEL command
   to delete the field and remove the field from the propagated HPEXPIREAT.
3. Condition not met for the field - Remove the field from the propagated
   HPEXPIREAT command.
4. Field does not exists - Remove the field from the propagated HPEXPIREAT command.

If none of the provided fields match option number 1, then avoid also propagating 
the HPEXPIREAT command to the replica.

This approach is aligned with the EXPIRE command:
If a given key has already expired, then DEL will be propagated instead of
EXPIRE command. If condition not met, then command will be rejected. Otherwise, 
EXPIRE command will be propagated for given key.
2024-06-25 13:15:09 +03:00
Moti CohenandGitHub e26ea35cd4 Adapt HRANDFIELD to HFE feature (#13348)
Considerations for the selected imp of HRANDFIELD & HFE feature:

HRANDFIELD might access any of the fields in the hash as some of them
might be expired. And so the Implementation of HRANDFIELD along with HFEs
might be one of the two options:
1. Expire hash-fields before diving into handling HRANDFIELD.
2. Refine HRANDFIELD cases to deal with expired fields.

Regarding the first option, as reference, the command RANDOMKEY also
declareson O(1) complexity, yet might be stuck on a very long (but not infinite)
loop trying to find non-expired keys. Furthermore RANDOMKEY also evicts expired 
keys along the way even though it is categorized as a read-only command. Note
that the case of HRANDFIELD is more lightweight versus RANDOMKEY since 
HFEs have much more effective and aggressive active-expiration for fields behind.

The second option introduces additional implementation complexity to HRANDFIELD.
We could further refine HRANDFIELD cases to differentiate between scenarios
with many expired fields versus few expired fields, and adjust based on the
percentage of expired fields. However, this approach could still lead to long
loops or necessitate expiring fields before selecting them. For the “lightweight”
cases it is also expected to have a lightweight expiration.

Considering the pros and cons, and the fact that HRANDFIELD is an infrequent
command (particularly with HFEs) and the fact we have effective active-expiration
behind for hash-fields, it is better to keep it simple and choose option number 1.

Other changes:
* Don't mark command dirty by internal hashTypeExpire(). It causes to read 
  only command of HRANDFIELD to be accidently propagated (This flag
  should be indicated at higher level, by the command functions).
* Align `hashTypeExpireIfNeeded()` and `hashTypeGetValue()` to be more
  aligned with `expireIfNeeded()` logic of keyspace.
2024-06-24 18:11:53 +03:00
AcherTTandGitHub 811c5d7aeb Add debug script command (#13289)
Add two new debug commands for outputing script.
1. `DEBUG SCRIPT LIST`
   Output all scripts.
2. `DEBUG SCRIPT <sha1>`
    Output a specific script.
    
Close #3846
2024-06-21 18:21:25 +08:00
debing.sunandGitHub a03b6e29a9 Sort out mess document for RM_Replicate and RM_ReplicateVerbatim (#13323)
Related to #12647

1. Make clear that `RM_Replicate` and `RM_ReplicateVerbatim` are
non-thread safe.
2. Make clear that `RM_Replicate` and `RM_ReplicateVerbatim` are alwarys
wrapped into MULTI in any case.
2024-06-21 17:59:49 +08:00
Moti CohenandGitHub e18a173a81 Fix rdbLoadObject() empty hash (#13347)
As part of HFE feature, the logic of rdbLoadObject() was wrongly
modified to indicate of loaded empty hash from RDB as hash that all its
fields got expired. Rollback to `emptykey` logic. This function should
load blindly all fields, expired or not. Manually verified.

Few more minor fixes:
- remove hash double check of emptyKey
- Fix from `sds` to `hfield` in rdbLoadObject() (not really a bug. Both
are of type char*)
- Revert code rdbLoadObject() to get dbid instead of db
2024-06-20 13:53:47 +03:00
24c85cc368 reduce getNodeByQuery CPU time by using less cache lines (from 2064 Bytes struct to 64 Bytes): reduces LLC misses and Memory Loads (#13296)
The following PR goes from 33 cacheline on getKeysResult struct (by
default has 256 static buffer)

```
root@hpe10:~/redis# pahole -p   ./src/server.o -C getKeysResult
typedef struct {
	keyReference               keysbuf[256];         /*     0  2048 */
	/* --- cacheline 32 boundary (2048 bytes) --- */
	/* typedef keyReference */ struct {
		int                pos;
		int                flags;
	} *keys; /*  2048     8 */
	int                        numkeys;              /*  2056     4 */
	int                        size;                 /*  2060     4 */

	/* size: 2064, cachelines: 33, members: 4 */
	/* last cacheline: 16 bytes */
} getKeysResult;
```


to 1 cacheline with a static buffer of 6 keys per command):
```
root@hpe10:~/redis# pahole -p   ./src/server.o -C getKeysResult
typedef struct {
	int                        numkeys;              /*     0     4 */
	int                        size;                 /*     4     4 */
	keyReference               keysbuf[6];           /*     8    48 */
	/* typedef keyReference */ struct {
		int                pos;
		int                flags;
	} *keys; /*    56     8 */

	/* size: 64, cachelines: 1, members: 4 */
} getKeysResult; 
```

we get around 1.5% higher ops/sec, and a confirmation of around 15% less
LLC loads on getNodeByQuery and 37% less Stores.

Function / Call Stack | CPU Time: Difference | CPU Time:
9462436fa444e746716845b1d807c74d8945831b | CPU Time: this PR | Loads:
Difference | Loads: 9462436fa444e746716845b1d807c74d8945831b | Loads:
this PR | Stores: Difference | Stores:
9462436fa444e746716845b1d807c74d8945831b | Stores: This PR
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
getNodeByQuery | 0.753767 | 1.57118 | 0.817416 | 144297829 (15% less
loads) | 920575969 | 776278140 | 367607824 (37% less stores) | 991642384
| 624034560

## results on client side

### baseline 
```
taskset -c 2,3 memtier_benchmark -s 192.168.1.200 --port 6379 --authenticate perf --cluster-mode --pipeline 10 --data-size 100 --ratio 1:0 --key-pattern P:P --key-minimum=1 --key-maximum 1000000 --test-time 180 -c 25 -t 2 --hide-histogram 
Writing results to stdout
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%, 180 secs]  0 threads:   110333450 ops,  604992 (avg:  612942) ops/sec, 84.75MB/sec (avg: 85.86MB/sec),  0.82 (avg:  0.81) msec latency

2         Threads
25        Connections per thread
180       Seconds


ALL STATS
======================================================================================================================================================
Type         Ops/sec     Hits/sec   Misses/sec    MOVED/sec      ASK/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec 
------------------------------------------------------------------------------------------------------------------------------------------------------
Sets       612942.14          ---          ---         0.00         0.00         0.81332         0.80700         1.26300         2.92700     87924.12 
Gets            0.00         0.00         0.00         0.00         0.00             ---             ---             ---             ---         0.00 
Waits           0.00          ---          ---          ---          ---             ---             ---             ---             ---          --- 
Totals     612942.14         0.00         0.00         0.00         0.00         0.81332         0.80700         1.26300         2.92700     87924.12 
```

### comparison 
```
taskset -c 2,3 memtier_benchmark -s 192.168.1.200 --port 6379 --authenticate perf --cluster-mode --pipeline 10 --data-size 100 --ratio 1:0 --key-pattern P:P --key-minimum=1 --key-maximum 1000000 --test-time 180 -c 25 -t 2 --hide-histogram 
Writing results to stdout
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%, 180 secs]  0 threads:   111731310 ops,  610195 (avg:  620707) ops/sec, 85.48MB/sec (avg: 86.95MB/sec),  0.82 (avg:  0.80) msec latency

2         Threads
25        Connections per thread
180       Seconds


ALL STATS
======================================================================================================================================================
Type         Ops/sec     Hits/sec   Misses/sec    MOVED/sec      ASK/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec 
------------------------------------------------------------------------------------------------------------------------------------------------------
Sets       620707.72          ---          ---         0.00         0.00         0.80312         0.79900         1.23900         2.87900     89037.78 
Gets            0.00         0.00         0.00         0.00         0.00             ---             ---             ---             ---         0.00 
Waits           0.00          ---          ---          ---          ---             ---             ---             ---             ---          --- 
Totals     620707.72         0.00         0.00         0.00         0.00         0.80312         0.79900         1.23900         2.87900     89037.78
```

Co-authored-by: filipecosta90 <filipecosta.90@gmail.com>
2024-06-18 18:00:47 +08:00
Ozan TezcanandGitHub 4aa25d042c Reply with array of return codes if the key does not exist for HFE commands (#13343)
Currently, HFE commands reply with empty array if the key does not
exist. Though, non-existing key and empty key is the same thing. 
It means fields given in the command do not exist in the empty key. 
So, replying with an array of 'no field' error codes (-2) suits better 
to Redis logic. e.g. Similarly, `hmget` returns array of nulls if the 
key does not exist.

After this PR:
```
127.0.0.1:6379> hpersist missingkey fields 2 a b
1) (integer) -2
2) (integer) -2
```
2024-06-14 09:35:05 +03:00
JoandGitHub 871c985919 Update FIELDS argument to block type for HFE commands schema (#13339)
I reviewed `XREAD` command syntax:
```
XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] id [id ...]
```

Here’s the structure for `XREAD`:
```json
"arguments": [
            {
                "token": "COUNT",
                "name": "count",
                "type": "integer",
                "optional": true
            },
            {
                "token": "BLOCK",
                "name": "milliseconds",
                "type": "integer",
                "optional": true
            },
            {
                "name": "streams",
                "token": "STREAMS",
                "type": "block",
                "arguments": [
                    {
                        "name": "key",
                        "type": "key",
                        "key_spec_index": 0,
                        "multiple": true
                    },
                    {
                        "name": "ID",
                        "type": "string",
                        "multiple": true
                    }
                ]
            }
]
```

Now, consider the `HEXPIRE` syntax:
```
HEXPIRE key seconds [NX | XX | GT | LT] FIELDS numfields field [field ...]
```

Since the `FIELDS` token functions similarly to `STREAMS`, and given that `STREAMS` is defined as a block, I believe the `FIELDS` in `hepxire` should also be defined as a block.
2024-06-14 13:51:49 +08:00
ed10f737b8 Add new hexpired notification for HFE (#13329)
When the hash field expired, we will send a new `hexpired` notification.
It mainly includes the following three cases:
1. When field expired by active expiration.
2. When field expired by lazy expiration.
3. When the user uses the `h(p)expire(at)` command, the user will also
get a `hexpired` notification if the field expires during the command.

## Improvement
1. Now if more than one field expires in the hmget command, we will only
send a `hexpired` notification.
2. When a field with TTL is deleted by commands like hdel without
updating the global DS, active expire will not send a notification.

---------

Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
Co-authored-by: Moti Cohen <moti.cohen@redis.com>
2024-06-11 21:42:34 +08:00
Moti CohenandGitHub f01fdc3960 Reserve 2 bits out of EB_EXPIRE_TIME_MAX for possible future use (#13331)
Reserve 2 bits out of hash-field expiration time (`EB_EXPIRE_TIME_MAX`)
for possible future lightweight indexing/categorizing of fields. It can
be achieved by hacking HFE as follows:
```
HPEXPIREAT key [ 2^47 + USER_INDEX ] FIELDS numfields field [field …]
```

Redis will also need to expose kind of `HEXPIRESCAN` and `HEXPIRECOUNT`
for this idea. Yet to be better defined.

`HFE_MAX_ABS_TIME_MSEC` constraint must be enforced only at API level.
Internally, the expiration time can be up to `EB_EXPIRE_TIME_MAX` for
future readiness.
2024-06-10 16:57:26 +03:00
Moti CohenandGitHub ce121b9261 HFE - Avoid lazy expire if called by modules + cleanup (#13326)
Need to be carefull if called by modules since modules API allow to open
and close key handler. We don't want to invalidate the handler
underneath.

* hashTypeExists(), hashTypeGetValueObject() - will return the logical
state of the field. A flag will indicate noExpire.
* RM_HashGet() - Will get NULL if the field expired. Fields won’t be
deleted.
* RM_ScanKey() - might return 0 items if all fields got expired. Fields
won’t be deleted.
* RM_HashSet() - If set, then override expired field. If delete, we can
either delete or leave it to active-expiration. XX/NX - logically
correct (Verify with tests).

Nice to have (not implemented):
* RedisModule_CloseKey() - We can local active-expire up-to 100 items. 

Note:
Length will be wrong to modules just like redis (Count expired fields).
2024-06-10 11:24:26 +03:00
debing.sunandGitHub 690ef36330 Fix HFE notification test timing caused by field expiration in hexpire command (#13325)
In the old test, we give the `hexpire` a very short expire time, which
caused the filed to be deleted by the time `hpersist` command was
executed. As a result, the `hpersist` command won't be able to give a
`hpersist` notification, leading to test stuck.

fail CI:
https://github.com/redis/redis/actions/runs/9342175887/job/25709886471
2024-06-05 14:49:45 +08:00
9a2c6ba4e7 Prevent negative expire parameter in HEXPIRE/HEXPIREAT/HPEXPIRE/HPEXPIREAT commands (#13310)
1. Don't allow HEXPIRE/HEXPIREAT/HPEXPIRE/HPEXPIREAT command expire
parameters is negative

2. Remove a dead code reported from Coverity.
when `unit` is not `UNIT_SECONDS`, the second `if (expire > (long long)
EB_EXPIRE_TIME_MAX)` will be dead code.
```c
# t_hash.c
2988    /* Check expire overflow */
      	cond_at_most: Condition expire > 281474976710655LL, taking false branch. Now the value of expire is at most 281474976710655.
2989    if (expire > (long long) EB_EXPIRE_TIME_MAX) {
2990        addReplyErrorExpireTime(c);
2991        return;
2992    }

2994    if (unit == UNIT_SECONDS) {
2995        if (expire > (long long) EB_EXPIRE_TIME_MAX / 1000) {
2996            addReplyErrorExpireTime(c);
2997            return;
2998        }
2999        expire *= 1000;
3000    } else {
      	at_most: At condition expire > 281474976710655LL, the value of expire must be at most 281474976710655.
      	dead_error_condition: The condition expire > 281474976710655LL cannot be true.
3001        if (expire > (long long) EB_EXPIRE_TIME_MAX) {
      	
CID 494223: (#1 of 1): Logically dead code (DEADCODE)
dead_error_begin: Execution cannot reach this statement: addReplyErrorExpireTime(c);.
3002            addReplyErrorExpireTime(c);
3003            return;
3004        }
3005    }
```

---------

Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
2024-06-04 20:35:26 +08:00
f36b5a8586 Fix crash due to unblock client during slot migration (#13311)
In #13224, we found a crash during cluster slot migration but don't know
why. So i check all the return C_OK in processCommand to see if we are
missing some duration reset and see this.

This fix is like #12247, when we reject the command, we should reset the
duration. I test it and verify it can fix #13224.

So the reason may because we are using stream block and then during the
slot migration, it got a redirect and then crash the server.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-06-04 20:16:36 +08:00
Ozan TezcanandGitHub 293a68afdf Use lookupKeyWrite() for hpersist command (#13321)
As hpersist is a write command, we should use lookupKeyWrite() instead
of lookupKeyRead() to fetch the key.
2024-06-04 12:04:37 +03:00
Ozan TezcanandGitHub 44352beefa Fix crash in RM_ScanKey() when used with hexpire (#13320)
RM_ScanKey() was overlooked while introducing hash field expiration. 
An assert is triggered when it is called on a hash key with
OBJ_ENCODING_LISTPACK_EX encoding.

I've changed to code to handle listpackex encoding properly.
2024-06-04 11:42:02 +03:00
Moti CohenandGitHub 56169112e3 Fix returned value nextExpireTime by ebExpire() (#13313)
At `ebuckets` structure, On `ebExpire()`, if the callback indicated to update 
the item expiration time and return it back to ebuckets (`ACT_UPDATE_EXP_ITEM`), 
then returned value `nextExpireTime` should be updated, if needed. 
Invalid value of `nextExpireTime` was modified from 0 to `EB_EXPIRE_TIME_INVALID`.
2024-06-03 13:54:44 +03:00
Valentino GeronandGitHub 50569a906c Free current client asynchronously after user permissions changes (#13274)
The crash happens when the user that triggers the permission changes
should be affected (and should be disconnected eventually).

To handle such a scenario, we should use the
`CLIENT_CLOSE_AFTER_COMMAND` flag.

This commit encapsulates all the places that should be handled in the
same way in `deauthenticateAndCloseClient`

Also:
* bugfix: during the ACL LOAD we ignore clients that are marked as
`CLIENT MASTER`
2024-05-30 22:09:30 +08:00
5a3534f9b5 dynamically list test files (#13220)
**Related issue**
https://github.com/redis/redis/issues/13219

**Motivation**
Currently we have to manually update the all_tests variable when
introducing new test files.

**Modification**
I have modified it to list test files dynamically, but instead of
modifying it to add all test files, I have modified it to only add only
test files from the following 4 paths

- unit
- unit/type
- unit/cluster
- integration

so that it doesn't deviate too much from what we already do

**Result**
- dynamically list test files to all_tests variable
- close issue https://github.com/redis/redis/issues/13219

**Additional information**
- removed `list-common.tcl` file and added
`generate_largevalue_test_array` proc in `util.tcl`. because
`list-common.tcl` is not a test file
- There is an order dependency. So I added a code to the "Is a ziplist
encoded Hash promoted on big payload?" test that resets
hash-max-listpack-value to the default (64).

---------

Signed-off-by: jonghoonpark <dev@jonghoonpark.com>
Co-authored-by: debing.sun <debing.sun@redis.com>
2024-05-30 19:09:26 +08:00
debing.sunandGitHub 7b9e960690 Hash Field Expiration (#13303)
## Background

This PR introduces support for field-level expiration in Redis hashes. Previously, Redis supported expiration only at the key level, but this enhancement allows setting expiration times for individual fields within a hash.

## New commands
* HEXPIRE
* HEXPIREAT
* HEXPIRETIME
* HPERSIST
* HPEXPIRE
* HPEXPIREAT
* HPEXPIRETIME
* HPTTL
* HTTL

## Short example
from @moticless
```sh
127.0.0.1:6379>  hset myhash f1 v1 f2 v2 f3 v3                                                   
(integer) 3
127.0.0.1:6379>  hpexpire myhash 10000 NX fields 2 f2 f3                                         
1) (integer) 1
2) (integer) 1
127.0.0.1:6379>  hpttl myhash fields 3 f1 f2 f3                                                                                                                                                                         
1) (integer) -1
2) (integer) 9997
3) (integer) 9997
127.0.0.1:6379>  hgetall myhash  
1) "f3"
2) "v3"
3) "f2"
4) "v2"
5) "f1"
6) "v1"

... after 10 seconds ...

127.0.0.1:6379>  hgetall myhash  
1) "f1"
2) "v1"
127.0.0.1:6379>
```

## Expiration strategy
1. Integrate active
    Redis periodically performs active expiration and deletion of hash keys that contain expired fields, with a maximum attempt limit.
3. Lazy expiration
    When a client touches fields within a hash, Redis checks if the fields are expired. If a field is expired, it will be deleted. However, we do not delete expired fields during a traversal, we implicitly skip over them.

## RDB changes
Add two new rdb type s`RDB_TYPE_HASH_METADATA` and `RDB_TYPE_HASH_LISTPACK_EX`.

## Notification
1. Add `hpersist` notification for `HPERSIST` command.
5. Add `hexpire` notification for `HEXPIRE`, `HEXPIREAT`, `HPEXPIRE` and `HPEXPIREAT` commands.

## Internal
1. Add new data structure `ebuckets`, which is used to store TTL and keys, enabling quick retrieval of keys based on TTL.
2. Add new data structure `mstr` like sds, which is used to store a string with TTL.

This work was done by @moticless, @tezc, @ronen-kalish, @sundb, I just release it.
2024-05-30 15:26:19 +08:00
Ozan TezcanandGitHub f0389f2823 Fix position of numfields in H(P)EXPIRE json files (#13301)
Fix position of numfields in H(P)EXPIRE json files
2024-05-29 16:35:47 +03:00
33fc0fbfae HFE to support AOF and replicas (#13285)
* For replica sake, rewrite commands `H*EXPIRE*` , `HSETF`, `HGETF` to
have absolute unix time in msec.
* On active-expiration of field, propagate HDEL to replica
(`propagateHashFieldDeletion()`)
* On lazy-expiration, propagate HDEL to replica (`hashTypeGetValue()`
now calls `hashTypeDelete()`. It also takes care to call
`propagateHashFieldDeletion()`).
* Fix `H*EXPIRE*` command such that if it gets flag `LT` and it doesn’t
have any expiration on the field then it will considered as valid
condition.

Note, replicas doesn’t make any active expiration, and should avoid lazy
expiration. On `hashTypeGetValue()` it doesn't check expiration (As long
as the master didn’t request to delete the field, it is valid)

TODO: 
* Attach `dbid` to HASH metadata. See
[here](https://github.com/redis/redis/pull/13209#discussion_r1593385850)

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-05-29 19:47:48 +08:00
Ozan TezcanandGitHub 6a11d458be Fix hscan return value (#13297)
In the last step of hscan, while replying to client, we assume all items
in the result list are keys which are mstr instances. Though, there 
might be values which are sds instances. 

Added a check to avoid calling mstrlen() for value objects.

To reproduce:
```
127.0.0.1:6379> hset myhash1 a 11111111111111111111111111111111111111111111111111111111111111111
(integer) 0
127.0.0.1:6379> hscan myhash1 0
1) "0"
2) 1) "a"
   2) "11111111111111111111111111111111111111111111111111111111111111111\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
```
2024-05-28 11:59:57 +03:00
Ozan TezcanandGitHub e2918705c8 Fix hfe reply schemas (#13295)
In https://github.com/redis/redis/pull/13291, we've changed that hfe
commands to return empty array if the key does not exist.
Forgot to update json schemas.
2024-05-27 16:07:01 +03:00
debing.sunandGitHub 2d1bb42cba Update version references from 8.0 to 7.4 for upcoming release (#13294) 2024-05-27 16:47:23 +08:00
2f34f6f0b9 Delete hsetf and hgetf (#13291)
Changes:
- Delete hsetf and hgetf commands
- Hfe commands will return empty array instead of nil.
---------

Co-authored-by: Moti Cohen <moticless@gmail.com>
2024-05-26 13:30:45 +03:00
Moti CohenandGitHub 60e1582ddb Fix statistics test (#13293) 2024-05-26 03:34:15 +03:00
6801a3cec8 redis-cli --keystats and --keystats-samples with --top and --cursor (#12826)
Added `--keystats` and `--keystats-samples <n>` commands.

The new commands combines memkeys and bigkeys with additional
distribution data.
We often run memkeys and bigkeys one after the other. It will be
convenient to just have one command.
Distribution and top 10 key sizes are useful when we have multiple keys
taking a lot of memory.

Like for memkeys and bigkeys, we can use `-i <n>` and Ctrl-C to
interrupt the scan loop. It will still show the statistics on the keys
sampled so far.
We can use two new optional parameters:
- `--cursor <n>` to continue from the last cursor after an interruption
of the scan.
- `--top <n>` to change the number of top key sizes (10 by default).

Implemented a fix for the `--memkeys-samples 0` issue in order to use
`--keystats-samples 0`.

Used hdr_histogram for the key size distribution.

For key length, hdr_histogram seemed overkill and preset ranges were
used.

The memory used by keystats with hdr_histogram is around 7MB (compared
to 3MB for memkeys or bigkeys).

Execution time is somewhat equivalent to adding memkeys and bigkeys
time. Each scan loop is having more commands (key type, key size, key
length/cardinality).

We can redirect the output to a file. In that case, no color or text
refresh will happen.

Limitation:
- As the information printed during the loop is refreshed (moving cursor
up), stderr and information not fitting in the terminal window (width or
height) might create some refresh issues.

Comments:
- config.top_sizes_limit could be used globally like config.count, but
it is passed as parameter to be consistent with config.memkeys_samples.
- Not sure if we should move some utility functions to cli-common.c.

Got some tips and help from @ofirluzon.

---------

Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: debing.sun <debing.sun@redis.com>
2024-05-25 15:59:27 +08:00
Moti CohenandGitHub f34f2ade85 Add Statistics hashes_with_expiry_fields to INFO (#13275)
Added hashes_with_expiry_fields.
Optimially it would better to have statistic of that counts all fields
with expiry. But it requires careful logic and computation to follow and
deep dive listpacks and hashes. This statistics is trivial to achieve
and reflected by global HFE DS that has builtin enumeration of all the
hashes that are registered in it.
2024-05-23 17:29:45 +03:00
Moti CohenandGitHub ae6df30eb1 Fix ebuckets stop indication during ebExpire() (#13287)
on active-expire of buckets, at function `ebExpire()` ->
`ebSegExpire()`, if callback `onExpireItem()` indicated to stop, Yet
iterator (iter) was wrongly already got updated to point to next item.
In turn the segment will be updated without current item.
2024-05-23 10:46:50 +03:00
a25b15392c Improve performance of hfe listpack (#13279)
This PR contains a few optimizations for hfe listpack.
- Hfe fields are ordered by TTL in the listpack. There are two cases
that we want to search listpack according to TTLs:
- As part of active-expiry, we need to find the fields that are expired.
e.g. find fields that have smaller TTLs than given timestamp.
- When we want to add a new field, we need to find the correct position
to maintain the order by TTL. e.g. find the field that has a higher TTL
than the one we want to insert.
  
Iterating with lpNext() to compare TTLs has a performance cost as
lpNext() calls lpValidateIntegrity() for each entry. Instead, this PR
adds `lpFindCb()` to the listpack which accepts a comparator callback.
It preserves same validation logic of lpFind() which is faster than
search with lpNext().
  
- We have field name, value, ttl for a single hfe field. Inserting these
items one by one to listpack is costly. Especially, as we place fields
according to TTL, most additions will end up in the middle of the
listpack. Each insert causes realloc + memmove. This PR introduces
`lpBatchInsert()` to add multiple items in one go.

- For hsetf, if we are going to update value and TTL at the same time,
currently, we update the value first and later update the TTL (two
distinct listpack operation). This PR improves it by doing it with a
single update operation.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2024-05-22 23:05:32 +03:00
95cbe879e5 sanitize dump payload for HFE (#13278)
Add the following validations:
1. Get TTL using the lpGetIntegerValue() method instead of lpGetValue(),
Ref https://github.com/redis/redis/pull/13209#discussion_r1602569422
2. The TTL of listpackex is a number in the valid range
(0~EB_EXPIRE_TIME_MAX) and ordered.
3. The TTL fields of listpackex are ordered. 
4. The TTL of hashtable is within the valid range
(0~EB_EXPIRE_TIME_MAX).

Other:
Fix the missing of handling OBJ_ENCODING_LISTPACK_EX in
dismissHashObject().

---------

Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
2024-05-22 10:53:30 +08:00
Ted LyngmoandGitHub e92363e248 Log the real reason for why posix_fadvise failed (#13246)
`reclaimFilePageCache` did not set `errno` but `rdbSaveInternal` which
is logging the error assumed it did. This makes sure `errno` is set.

Fixes #13245

Signed-off-by: Ted Lyngmo <ted@lyncon.se>
2024-05-21 10:33:14 +08:00
9ffc35c98e Have consistent behavior of SPUBLISH within multi/exec like regular command (#13276)
This PR is based on the commits from PR #12944.

Allow SPUBLISH command within multi/exec on replica

Behavior on unstable:

```
127.0.0.1:6380> CLUSTER NODES
39ce8aa20f1f0d91f1a88d976ee1926dfefcdf1a 127.0.0.1:6380@16380 myself,slave 8b0feb120b68aac489d6a5af9c77dc40d71bc792 0 0 0 connected
8b0feb120b68aac489d6a5af9c77dc40d71bc792 127.0.0.1:6379@16379 master - 0 1705091681202 0 connected 0-16383
127.0.0.1:6380> SPUBLISH hello world
(integer) 0
127.0.0.1:6380> MULTI
OK
127.0.0.1:6380(TX)> SPUBLISH hello world
QUEUED
127.0.0.1:6380(TX)> EXEC
(error) MOVED 866 127.0.0.1:6379
```

With this change:

```
127.0.0.1:6380> SPUBLISH hello world
(integer) 0
127.0.0.1:6380> MULTI
OK
127.0.0.1:6380(TX)> SPUBLISH hello world
QUEUED
127.0.0.1:6380(TX)> EXEC
1) (integer) 0
```

---------

Co-authored-by: Harkrishn Patro <harkrisp@amazon.com>
Co-authored-by: oranagra <oran@redislabs.com>
2024-05-21 09:25:13 +08:00
Ozan TezcanandGitHub 36c3cec6d1 Fix hfe RDB tests by adding FIELDS keyword to hexpire commands (#13277)
FIELDS keyword was added as part of
[#13270](https://github.com/redis/redis/pull/13270). 

It was missing in
[#13243](https://github.com/redis/redis/pull/13243)
2024-05-18 11:34:13 +03:00
Ronen KalishandGitHub 323be4d699 Hfe serialization listpack (#13243)
Add RDB de/serialization for HFE

This PR adds two new RDB types: `RDB_TYPE_HASH_METADATA` and
`RDB_TYPE_HASH_LISTPACK_TTL` to save HFE data.
When the hash RAM encoding is dict, it will be saved in the former, and
when it is listpack it will be saved in the latter.
Both formats just add the TTL value for each field after the data that
was previously saved, i.e HASH_METADATA will save the number of entries
and, for each entry, key, value and TTL, whereas listpack is saved as a
blob.
On read, the usual dict <--> listpack conversion takes place if
required.
In addition, when reading a hash that was saved as a dict fields are
actively expired if expiry is due. Currently this slao holds for
listpack encoding, but it is supposed to be removed.

TODO:
Remove active expiry on load when loading from listpack format (unless
we'll decide to keep it)
2024-05-17 18:27:02 +08:00
Moti CohenandGitHub 71676513dd Fix commands H*EXPIRE* and H*TTL to include FIELDS constant (#13270)
The same goes to: HPEXPIRE, HEXPIREAT, HPEXPIREAT, HEXPIRETIME,
HPEXPIRETIME, HPTTL, HTTL, HPERSIST
2024-05-16 19:35:58 +03:00
debing.sunandGitHub f1b0212917 Revert "Change mmap rand bits as a temporary mitigation to resolve asan bug (#13150)" (#13266)
The kernel config `vm.mmap_rnd_bits` had been revert in
https://github.com/actions/runner-images/issues/9491, so we can revert
the changes from #13150.

CI only with ASAN passed:
https://github.com/sundb/redis/actions/runs/9058263634
2024-05-15 14:12:22 +08:00
ffbdf2f6f3 Fix test failure due to differing reply format of XREADGROUP under RESP3 in MULTI (#13255)
This test was introducted by #13251.
Normally we auto transform the reply format of XREADGROUP to array under
RESP3 (see trasformer_funcs).
But when we execute XREADGROUP command in multi it can't work, which
cause the new test failed.
The solution is to verity the reply of XREADGROUP in advance rather than
in MULTI.

Failed validate schema CI:
https://github.com/redis/redis/actions/runs/9025128323/job/24800285684

---------

Co-authored-by: guybe7 <guy.benoish@redislabs.com>
2024-05-14 20:08:32 +08:00
80be2cc291 Add defragment support for HFE (#13229)
## Background
1. All hash objects that contain HFE are referenced by db->hexpires.
2. All fields in a dict hash object with HFE are referenced by an
ebucket.

So when we defrag the hash object or the field in a dict with HFE, we
also need to update the references in them.

## Interface
1. Add a new interface `ebDefragItem`, which can accept a defrag
callback to defrag items in ebuckets, and simultaneously update their
references in the ebucket.

## Mainly changes
1. The key type of dict of hash object is no longer sds, so add new
`activeDefragHfieldDict()` to defrag the dict instead of
`activeDefragSdsDict()`.
2. When we defrag the dict of hash object by using `dictScanDefrag()`,
we always set the defrag callback `defragKey` of `dictDefragFunctions`
to NULL, because we can't reallocate a field with out updating it's
reference in ebuckets.
Instead, we will defrag the field of the dict and update its reference
in the callback `dictScanDefrag` of dictScanFunction().
3. When we defrag the hash robj with HFE, we will use `ebDefragItem` to
defrag the robj and update the reference in db->hexpires.

## TODO:
Defrag ebucket structure incremently, which will be handler in a future
PR.

---------

Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
Co-authored-by: Moti Cohen <moti.cohen@redis.com>
2024-05-14 17:32:33 +08:00
Ozan TezcanandGitHub 5066e6e9cd Fix hgetf/hsetf reply type by returning string (#13263)
If encoding is listpack, hgetf and hsetf commands reply field value type
as integer.
This PR fixes it by returning string.

Problematic cases:
```
127.0.0.1:6379> hset hash one 1
(integer) 1
127.0.0.1:6379> hgetf hash fields 1 one
1) (integer) 1
127.0.0.1:6379> hsetf hash GETOLD fvs 1 one 2
1) (integer) 1
127.0.0.1:6379> hsetf hash DOF GETNEW fvs 1 one 2
1) (integer) 2
```

Additional fixes:
- hgetf/hsetf command description text

Fixes #13261, #13262
2024-05-13 11:09:49 +03:00
8a05f0092b Add reverse history search in redis-cli (linenoise) (#12543)
added reverse history search to redis-cli, use it with the following:

* CTRL+R : enable search backward mode, and search next one when
pressing CTRL+R again until reach index 0.
```
127.0.0.1:6379> keys one
127.0.0.1:6379> keys two
(reverse-i-search):                   # press CTRL+R
(reverse-i-search): keys two          # input `keys`
(reverse-i-search): keys one          # press CTRL+R again
(reverse-i-search): keys one          # press CTRL+R again, still `keys one` due to reaching index 0
(i-search): keys two                  # press CTRL+S, enable search forward
(i-search): keys two                  # press CTRL+S, still `keys one` due to reaching index 1
```

* CTRL+S : enable search forward mode, and search next one when pressing
CTRL+S again until reach index 0.
```
127.0.0.1:6379> keys one
127.0.0.1:6379> keys two
(i-search):                       # press CTRL+S
(i-search): keys one              # input `keys`
(i-search): keys two              # press CTRL+S again
(i-search): keys two              # press CTRL+R again, still `keys two` due to reaching index 0
(reverse-i-search): keys one      # press CTRL+R, enable search backward
(reverse-i-search): keys one      # press CTRL+S, still `keys one` due to reaching index 1
```

* CTRL+G : disable
```
127.0.0.1:6379> keys one
127.0.0.1:6379> keys two
(reverse-i-search):                   # press CTRL+R
(reverse-i-search): keys two          # input `keys`
127.0.0.1:6379>                       # press CTRL+G
```

* CTRL+C : disable
```
127.0.0.1:6379> keys one
127.0.0.1:6379> keys two
(reverse-i-search):                   # press CTRL+R
(reverse-i-search): keys two          # input `keys`
127.0.0.1:6379>                       # press CTRL+G
```

* TAB : use the current search result and exit search mode
```
127.0.0.1:6379> keys one
127.0.0.1:6379> keys two
(reverse-i-search):                # press CTRL+R
(reverse-i-search): keys two       # input `keys`
127.0.0.1:6379> keys two           # press TAB
```

* ENTER : use the current search result and execute the command
```
127.0.0.1:6379> keys one
127.0.0.1:6379> keys two
(reverse-i-search):                 # press CTRL+R
(reverse-i-search): keys two        # input `keys`
127.0.0.1:6379> keys two            # press ENTER
(empty array)
127.0.0.1:6379>
```

* any arrow key will disable reverse search

your result will have the search match bolded, you can press enter to
execute the full result

note: I have _only added this for multi-line mode_, as it seems to be
forced that way when `repl` is called

Closes: https://github.com/redis/redis/issues/8277

---------

Co-authored-by: Clayton Northey <clayton@knowbl.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: Bjorn Svensson <bjorn.a.svensson@est.tech>
Co-authored-by: Viktor Söderqvist <viktor@zuiderkwast.se>
2024-05-10 11:10:14 +08:00
debing.sunandGitHub 7010f41c96 Add notification support for HFE (#13237)
1. Add `hpersist` notification for `hpersist` command.
2. Add `pexpire` notification for `hexpire`, `hexpireat` and `hpexpire`.
2024-05-09 22:23:00 +08:00
Ozan TezcanandGitHub ca4ed48db6 Add listpack support, hgetf and hsetf commands (#13209)
**Changes:**
- Adds listpack support to hash field expiration 
- Implements hgetf/hsetf commands

**Listpack support for hash field expiration**

We keep field name and value pairs in listpack for the hash type. With
this PR, if one of hash field expiration command is called on the key
for the first time, it converts listpack layout to triplets to hold
field name, value and ttl per field. If a field does not have a TTL, we
store zero as the ttl value. Zero is encoded as two bytes in the
listpack. So, once we convert listpack to hold triplets, for the fields
that don't have a TTL, it will be consuming those extra 2 bytes per
item. Fields are ordered by ttl in the listpack to find the field with
minimum expiry time efficiently.

**New command implementations as part of this PR:** 

- HGETF command

For each specified field get its value and optionally set the field's
expiration time in sec/msec /unix-sec/unix-msec:
  ```
  HGETF key 
    [NX | XX | GT | LT]
[EX seconds | PX milliseconds | EXAT unix-time-seconds | PXAT
unix-time-milliseconds | PERSIST]
    <FIELDS count field [field ...]>
  ```

- HSETF command

For each specified field value pair: set field to value and optionally
set the field's expiration time in sec/msec /unix-sec/unix-msec:
  ```
  HSETF key 
    [DC] 
    [DCF | DOF] 
    [NX | XX | GT | LT] 
    [GETNEW | GETOLD] 
[EX seconds | PX milliseconds | EXAT unix-time-seconds | PXAT
unix-time-milliseconds | KEEPTTL]
    <FVS count field value [field value …]>
  ```

Todo:
- Performance improvement.
- rdb load/save
- aof
- defrag
2024-05-08 23:11:32 +03:00
Moti CohenandGitHub 13401f8bc1 ebuckets: Add test for ACT_UPDATE_EXP_ITEM (#13249)
- On ebExpire() verify the logic of update expired value to a new time
rather than remove it.
- Refine ebuckets benchmark
2024-05-08 18:38:45 +03:00
guybe7andGitHub 0e1de78fca XREADGROUP from PEL should not affect server.dirty (#13251)
Because it does not cause any propagation (arguably it should, see the
comment in the tcl file)

The motivation for this fix is that in 6.2 if dirty changed without
propagation inside MULTI/EXEC it would cause propagation of EXEC only,
which would result in the replica sending errors to its master
2024-05-06 16:55:42 +08:00
debing.sunandGitHub 03cd525ffa Fix reply schema for hfe related commands (#13238) 2024-05-03 11:11:41 +08:00
Moti CohenandGitHub c33c91dbce Support HSET+expire in one command, at infra level (#13230)
Unify infra of `HSETF`, `HEXPIRE`, `HSET` and provide API for RDB load
as well. Whereas setting plain fields is rather straightforward, setting
expiration time to fields might be time-consuming and complex since each 
update of expiration time, not only updates `ebuckets` of corresponding hash, 
but also might update `ebuckets` of global HFE DS. It is required to opt 
sequence of field updates with expirartion for a given hash, such that only once
done, the global HFE DS will get updated.

To do so, follow the scheme:
1. Call `hashTypeSetExInit()` to initialize the HashTypeSetEx struct.
2. Call `hashTypeSetEx()` one time or more, for each field/expiration update.
3. Call `hashTypeSetExDone()` for notification and update of global HFE.

If expiration is not required, then avoid this API and use instead hashTypeSet().
2024-04-25 18:29:03 +03:00
debing.sunandGitHub f95031c473 Fix CI failure caused by PR #13231 (#13233)
For my mistake, in the last revert commit in #13231, I originally wanted
to revert the last one, but reverted the penultimate fix.
Now that we have fix another potential memory read issue in [`743f1dd`
(#13231)](https://github.com/redis/redis/pull/13231/commits/743f1dde79b433fdb8ea13de4fd73457d4fe25eb),
now it just seems to avoid confusion, i will verify in the future
whether it will have any impact, if so we will add this PR to backport.

Failed CI: https://github.com/sundb/redis/actions/runs/8826731960
2024-04-25 14:11:45 +08:00
debing.sunandGitHub 772564fc9e Fix forget to update the dict's node in the kvstore's rehashing list after defragment (#13231)
Introducted by #13013

After defragmenting the dictionary in the kvstore, if the dict is
reallocated, the value of its node in the kvstore rehashing list must be
updated.
2024-04-24 16:15:42 +08:00
Moti CohenandGitHub c18ff05665 Hash Field Expiration - Basic support
- Add ebuckets & mstr data structures
- Integrate active & lazy expiration
- Add most of the commands 
- Add support for dict (listpack is missing)
TODOs:  RDB, notification, listpack, HSET, HGETF, defrag, aof
2024-04-18 16:06:30 +03:00
804110a487 Allocate Lua VM code with jemalloc instead of libc, and count it used memory (#13133)
## Background
1. Currently Lua memory control does not pass through Redis's zmalloc.c.
Redis maxmemory cannot limit memory problems caused by users abusing lua
since these lua VM memory is not part of used_memory.

2. Since jemalloc is much better (fragmentation and speed), and also we
know it and trust it. we are
going to use jemalloc instead of libc to allocate the Lua VM code and
count it used memory.

## Process:
In this PR, we will use jemalloc in lua. 
1. Create an arena for all lua vm (script and function), which is
shared, in order to avoid blocking defragger.
2. Create a bound tcache for the lua VM, since the lua VM and the main
thread are by default in the same tcache, and if there is no isolated
tcache, lua may request memory from the tcache which has just been freed
by main thread, and vice versa
On the other hand, since lua vm might be release in bio thread, but
tcache is not thread-safe, we need to recreate
    the tcache every time we recreate the lua vm.
3. Remove lua memory statistics from memory fragmentation statistics to
avoid the effects of lua memory fragmentation

## Other
Add the following new fields to `INFO DEBUG` (we may promote them to
INFO MEMORY some day)
1. allocator_allocated_lua: total number of bytes allocated of lua arena
2. allocator_active_lua: total number of bytes in active pages allocated
in lua arena
3. allocator_resident_lua: maximum number of bytes in physically
resident data pages mapped in lua arena
4. allocator_frag_bytes_lua: fragment bytes in lua arena

This is oranagra's idea, and i got some help from sundb.

This solves the third point in #13102.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2024-04-16 12:43:33 +03:00
Yves LeBrasandGitHub e3550f01dd redis-cli - sendReadOnly() to work with Redis Cloud (#13195)
When using Redis Cloud, sendReadOnly() exit with `Error: ERR unknown
command 'READONLY'`.
It is impacting `--memkeys`, `--bigkeys`, `--hotkeys`, and will impact
`--keystats`.
Added one line to ignore this error.

issue introduced in #12735 (not yet released).
2024-04-08 11:12:57 +03:00
debing.sunandGitHub f4481e657f Use usleep() instead of sched_yield() to yield cpu (#13183)
when the main thread and the module thread are in the same thread,
sched_yield() can work well.
when they are both bind to different cpus, sched_yield() will look for
the thread with the highest priority, and if the module thread is always
the highest priority on a cpu, it will take a long time to let the main
thread to reacquire the GIL.

ref https://man7.org/linux/man-pages/man2/sched_yield.2.html
```
If the calling thread is the only thread in the highest priority
list at that time, it will continue to run after a call to
sched_yield().
```
2024-04-07 20:59:36 +08:00
debing.sunandGitHub 4581d43230 Fix daylight race condition and some thread leaks (#13191)
fix some issues that come from sanitizer thread report.

1. when the main thread is updating daylight_active, other threads (bio,
module thread) may be writing logs at the same time.
```
WARNING: ThreadSanitizer: data race (pid=661064)
  Read of size 4 at 0x55c9a4d11c70 by thread T2:
    #0 serverLogRaw /home/sundb/data/redis_fork/src/server.c:116 (redis-server+0x8d797) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0)
    #1 _serverLog.constprop.2 /home/sundb/data/redis_fork/src/server.c:146 (redis-server+0x2a3b14) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0)
    #2 bioProcessBackgroundJobs /home/sundb/data/redis_fork/src/bio.c:329 (redis-server+0x1c24ca) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0)

  Previous write of size 4 at 0x55c9a4d11c70 by main thread (mutexes: write M0, write M1, write M2, write M3):
    #0 updateCachedTimeWithUs /home/sundb/data/redis_fork/src/server.c:1102 (redis-server+0x925e7) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0)
    #1 updateCachedTimeWithUs /home/sundb/data/redis_fork/src/server.c:1087 (redis-server+0x925e7)
    #2 updateCachedTime /home/sundb/data/redis_fork/src/server.c:1118 (redis-server+0x925e7)
    #3 afterSleep /home/sundb/data/redis_fork/src/server.c:1811 (redis-server+0x925e7)
    #4 aeProcessEvents /home/sundb/data/redis_fork/src/ae.c:389 (redis-server+0x85ae0) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0)
    #5 aeProcessEvents /home/sundb/data/redis_fork/src/ae.c:342 (redis-server+0x85ae0)
    #6 aeMain /home/sundb/data/redis_fork/src/ae.c:477 (redis-server+0x85ae0)
    #7 main /home/sundb/data/redis_fork/src/server.c:7211 (redis-server+0x7168c) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0)
```

2. thread leaks in module tests
```
WARNING: ThreadSanitizer: thread leak (pid=668683)
  Thread T13 (tid=670041, finished) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1036 (libtsan.so.2+0x3d179) (BuildId: 28a9f70061dbb2dfa2cef661d3b23aff4ea13536)
    #1 HelloBlockNoTracking_RedisCommand /home/sundb/data/redis_fork/tests/modules/blockonbackground.c:200 (blockonbackground.so+0x97fd) (BuildId: 9cd187906c57e88cdf896d121d1d96448b37a136)
    #2 HelloBlockNoTracking_RedisCommand /home/sundb/data/redis_fork/tests/modules/blockonbackground.c:169 (blockonbackground.so+0x97fd)
    #3 call /home/sundb/data/redis_fork/src/server.c:3546 (redis-server+0x9b7fb) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0)
    #4 processCommand /home/sundb/data/redis_fork/src/server.c:4176 (redis-server+0xa091c) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0)
    #5 processCommandAndResetClient /home/sundb/data/redis_fork/src/networking.c:2468 (redis-server+0xd2b8e) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0)
    #6 processInputBuffer /home/sundb/data/redis_fork/src/networking.c:2576 (redis-server+0xd2b8e)
    #7 readQueryFromClient /home/sundb/data/redis_fork/src/networking.c:2722 (redis-server+0xd358f) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0)
    #8 callHandler /home/sundb/data/redis_fork/src/connhelpers.h:58 (redis-server+0x288a7b) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0)
    #9 connSocketEventHandler /home/sundb/data/redis_fork/src/socket.c:277 (redis-server+0x288a7b)
    #10 aeProcessEvents /home/sundb/data/redis_fork/src/ae.c:417 (redis-server+0x85b45) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0)
    #11 aeProcessEvents /home/sundb/data/redis_fork/src/ae.c:342 (redis-server+0x85b45)
    #12 aeMain /home/sundb/data/redis_fork/src/ae.c:477 (redis-server+0x85b45)
    #13 main /home/sundb/data/redis_fork/src/server.c:7211 (redis-server+0x7168c) (BuildId: dca0b1945ba30010e36129bdb296e488dd2b32d0)
```
2024-04-04 13:49:51 +03:00
Moti CohenandGitHub 4df037962d Change FLUSHALL/FLUSHDB SYNC to run as blocking ASYNC (#13167)
# Overview
Users utilize the `FLUSHDB SYNC` and `FLUSHALL SYNC` commands for a variety of 
reasons. The main issue with this command is that if the database becomes 
substantial in size, the server will be unresponsive for an extended period. 
Other than freezing application traffic, this may also lead some clients making 
incorrect judgments about the server's availability. For instance, a watchdog may 
erroneously decide to terminate the process, resulting in potential adverse 
outcomes. While a `FLUSH* ASYNC` can address these issues, it might not be used 
for two reasons: firstly, it's not the default, and secondly, in some cases, the 
client issuing the flush wants to wait for its completion before repopulating the 
database.

Between the option of triggering FLUSH* asynchronously in the background without 
indication for completion versus running it synchronously in the foreground by 
the main thread, there is another more appealing option. We can block the
client that requested the flush, execute the flush command in the background, and 
once done, unblock the client and return notification for completion. This approach 
ensures the server remains responsive to other clients, and the blocked client 
receives the expected response only after the flush operation has been successfully 
carried out.

# Implementation details
Instead of defining yet another flavor to the flush command, we can modify
`FLUSHALL SYNC` and `FLUSHDB SYNC` always run in this new mode.

## Extending BIO Threads capabilities
Today jobs that are carried out by BIO threads don't have the capability to 
indicate completion to the main thread. We can add this infrastructure by having
an additional dummy job, coined as completion-job, that eventually will be written 
by BIO threads to a response-queue. The main thread will take care to consume items
from the response-queue and call the provided callback function of each 
completion-job.

## FLUSH* SYNC to run as blocking ASYNC
Command `FLUSH* SYNC` will be modified to create one or more async jobs to flush
DB(s) and afterward will push additional completion-job request. By sending the
completion job request only at the end, the main thread will be called back only
after all the preceding jobs completed their task in the background. During that
time, the client of the command is suspended and marked as `BLOCKED_LAZYFREE`
whereas any other client will be able to communicate with the server without any
issue.
2024-04-02 15:09:52 +03:00
Moti CohenandGitHub ce47834309 kvstoreIteratorNext() wrongly reset iterator twice (#13178)
It calls kvstoreIteratorNextDict() which eventually calls dictResumeRehashing()
And then, on return, it calls dictResetIterator(iter) which calls dictResumeRehashing().
We end up with pauserehash value decremented twice instead of once.
2024-04-01 18:08:55 +03:00
Pieter CailliauandGitHub 0b34396924 Change license from BSD-3 to dual RSALv2+SSPLv1 (#13157)
[Read more about the license change
here](https://redis.com/blog/redis-adopts-dual-source-available-licensing/)
Live long and prosper 🖖
2024-03-20 22:38:24 +00:00
e64d91c371 Fix dict use-after-free problem in kvs->rehashing (#13154)
In ASAN CI, we find server may crash because of NULL ptr in `kvstoreIncrementallyRehash`.
the reason is that we use two phase unlink in `dbGenericDelete`. After `kvstoreDictTwoPhaseUnlinkFind`,
the dict may be in rehashing and only have one element in ht[0] of `db->keys`.

When we delete the last element in `db->keys` meanwhile `db->keys` is in rehashing, we may free the
dict in `kvstoreDictTwoPhaseUnlinkFree` without deleting the node in `kvs->rehashing`. Then we may
use this freed ptr in `kvstoreIncrementallyRehash` in the `serverCron` and cause the crash.
This is indeed a use-after-free problem.

The fix is to call rehashingCompleted in dictRelease and dictEmpty, so that every call for
rehashingStarted is always matched with a rehashingCompleted.

Adding a test in the unit test to catch it consistently

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: debing.sun <debing.sun@redis.com>
2024-03-20 22:44:28 +02:00
Yanqi LvandGitHub bad33f8738 fix wrong data type conversion in zrangeResultBeginStore (#13148)
In `beginResultEmission`, -1 means the result length is not known in
advance. But after #12185, if we pass -1 to `zrangeResultBeginStore`, it
will convert to SIZE_MAX in `zsetTypeCreate` and try to `dictExpand`.
Although `dictExpand` won't succeed because the size overflows, I think
we'd better to avoid this wrong conversion.

This bug can be triggered when the source of `zrangestore` doesn't exist
or we use `zrangestore` command with `byscore` or `bylex`.
The impact is that dst keys will be converted to use skiplist instead of
listpack.
2024-03-19 08:52:55 +02:00
BinbinandGitHub e04d41d78d Prevent lua error_reply abuse from causing errorstats to become larger (#13141)
Users who abuse lua error_reply will generate a new error object on each
error call, which can make server.errors get bigger and bigger. This
will
cause the server to block when calling INFO (we also return errorstats
by
default).

To prevent the damage it can cause, when a misuse is detected, we will
print a warning log and disable the errorstats to avoid adding more new
errors. It can be re-enabled via CONFIG RESETSTAT.

Because server.errors may be very large (it may be better now since we
have the limit), config resetstat may block for a while. So in
resetErrorTableStats, we will try to lazyfree server.errors.

See the related discussion at the end of #8217.
2024-03-19 08:18:22 +02:00
Chen TianjieandGitHub aeada20140 Avoid unnecessary dict shrink in zremrangeGenericCommand (#13143)
If the skiplist is emptied, there is no need to shrink the dict in
skiplist, it can be deleted directly.
2024-03-19 10:14:19 +08:00
BinbinandGitHub 7b070423b8 Fix dictionary use-after-free in active expire and make kvstore iter to respect EMPTY flag (#13135)
After #13072, there is an use-after-free error. In expireScanCallback, we
will delete the dict, and then in dictScan we will continue to use the dict,
like we will doing `dictResumeRehashing(d)` in the end, this casued an error.

In this PR, in freeDictIfNeeded, if the dict's pauserehash is set, don't
delete the dict yet, and then when scan returns try to delete it again.

At the same time, we noticed that there will be similar problems in iterator.
We may also delete elements during the iteration process, causing the dict
to be deleted, so the part related to iter in the PR has also been modified.
dictResetIterator was also missing from the previous kvstoreIteratorNextDict,
we currently have no scenario that elements will be deleted in kvstoreIterator
process, deal with it together to avoid future problems. Added some simple
tests to verify the changes.

In addition, the modification in #13072 omitted initTempDb and emptyDbAsync,
and they were also added. This PR also remove the slow flag from the expire
test (consumes 1.3s) so that problems can be found in CI in the future.
2024-03-18 17:41:54 +02:00
Alexander MahoneandGitHub 98a6e55d4e Add missing REDIS_STATIC in quicklist (#13147)
Compiler complained when I tried to compile only quicklist.c.
Since static keyword is needed when a static function declaration is
placed before its implementation.

```
#ifndef REDIS_STATIC
#define REDIS_STATIC static
#endif
```

[How to solve static declaration follows non-static declaration in GCC C
code?](https://stackoverflow.com/questions/3148244/how-to-solve-static-declaration-follows-non-static-declaration-in-gcc-c-code)
2024-03-18 08:22:19 +02:00
Madelyn OlsonandGitHub 3f725b8619 Change mmap rand bits as a temporary mitigation to resolve asan bug (#13150)
There is a new change in linux kernel 6.6.6 that uses randomization of
address space to harden security, but it interferes with the way ASAN
works. Folks are working on a fix, but this is a temporary mitigation
for us to get our CI to be green again.

See https://github.com/google/sanitizers/issues/1716 for more
information

See
https://github.com/redis/redis/actions/runs/8305126288/job/22731614306?pr=13149
for a recent failure
2024-03-17 09:06:51 +02:00
Viktor SöderqvistandGitHub 1d77a8e2c5 Makefile respect user's REDIS_CFLAGS and OPT (#13073)
This change to the Makefile makes it possible to opt out of
`-fno-omit-frame-pointer` added in #12973 and `-flto` (#11350). Those
features were implemented by conditionally modifying the `REDIS_CFLAGS`
and `REDIS_LDFLAGS` variables. Historically, those variables provided a
way for users to pass options to the compiler and linker unchanged.

Instead of conditionally appending optimization flags to REDIS_CFLAGS
and REDIS_LDFLAGS, I want to append them to the OPTIMIZATION variable.

Later in the Makefile, we have `OPT=$(OPTIMIZATION)` (meaning
OPTIMIZATION is only a default for OPT, but OPT can be overridden by the
user), and later the flags are combined like this:

FINAL_CFLAGS=$(STD) $(WARN) $(OPT) $(DEBUG) $(CFLAGS) $(REDIS_CFLAGS)
    FINAL_LDFLAGS=$(LDFLAGS) $(OPT) $(REDIS_LDFLAGS) $(DEBUG)

This makes it possible for the the user to override all optimization
flags with e.g. `make OPT=-O1` or just `make OPT=`.

For some reason `-O3` was also already added to REDIS_LDFLAGS by default
in #12339, so I added OPT to FINAL_LDFLAGS to avoid more complex logic
(such as introducing a separate LD_OPT variable).
2024-03-13 17:02:00 +02:00
3b3d16f748 Add KVSTORE_FREE_EMPTY_DICTS to cluster mode keys / expires kvstore (#13072)
Currently (following #11695, and #12822), keys kvstore and expires
kvstore both flag with ON_DEMAND, it means that a cluster node will
only allocate a dict when the slot is assigned to it and populated,
but on the other hand, when the slot is unassigned, the dict will
remain allocated.

We considered releasing the dict when the slot is unassigned, but it
causes complications on replicas. On the other hand, from benchmarks
we conducted, it looks like the performance impact of releasing the
dict when it becomes empty and re-allocate it when a key is added
again, isn't huge.

This PR add KVSTORE_FREE_EMPTY_DICTS to cluster mode keys / expires
kvstore.

The impact is about about 2% performance drop, for this hopefully
uncommon scenario.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2024-03-13 08:30:20 +02:00
BinbinandGitHub ad28d222ed Lua eval scripts first in first out LRU eviction (#13108)
In some cases, users will abuse lua eval. Each EVAL call generates
a new lua script, which is added to the lua interpreter and cached
to redis-server, consuming a large amount of memory over time.

Since EVAL is mostly the one that abuses the lua cache, and these
won't have pipeline issues (i.e. the script won't disappear
unexpectedly,
and cause errors like it would with SCRIPT LOAD and EVALSHA),
we implement a plain FIFO LRU eviction only for these (not for
scripts loaded with SCRIPT LOAD).

### Implementation notes:
When not abused we'll probably have less than 100 scripts, and when
abused we'll have many thousands. So we use a hard coded value of 500
scripts. And considering that we don't have many scripts, then unlike
keys, we don't need to worry about the memory usage of keeping a true
sorted LRU linked list. We compute the SHA of each script anyway,
and put the script in a dict, we can store a listNode there, and use
it for quick removal and re-insertion into an LRU list each time the
script is used.

### New interfaces:
At the same time, a new `evicted_scripts` field is added to
INFO, which represents the number of evicted eval scripts. Users
can check it to see if they are abusing EVAL.

### benchmark:
`./src/redis-benchmark -P 10 -n 1000000 -r 10000000000 eval "return
__rand_int__" 0`

The simple abuse of eval benchmark test that will create 1 million EVAL
scripts. The performance has been improved by 50%, and the max latency
has dropped from 500ms to 13ms (this may be caused by table expansion
inside Lua when the number of scripts is large). And in the INFO memory,
it used to consume 120MB (server cache) + 310MB (lua engine), but now
it only consumes 70KB (server cache) + 210KB (lua_engine) because of
the scripts eviction.

For non-abusive case of about 100 EVAL scripts, there's no noticeable
change in performance or memory usage.

### unlikely potentially breaking change:
in theory, a user can maybe load a
script with EVAL and then use EVALSHA to call it (by calculating the
SHA1 value on the client side), it could be that if we read the docs
carefully we'll realized it's a valid scenario, but we suppose it's
extremely rare. So it may happen that EVALSHA acts on a script created
by EVAL, and the script is evicted and EVALSHA returns a NOSCRIPT error.
that is if you have more than 500 scripts being used in the same
transaction / pipeline.

This solves the second point in #13102.
2024-03-13 08:27:41 +02:00
a8e745117f Xread last entry in stream (#7388) (#13117)
Allow using `+` as a special ID for last item in stream on XREAD
command.

This would allow to iterate on a stream with XREAD starting with the
last available message instead of the next one which `$` is used for.
I.e. the caller can use `BLOCK` and `+` on the first call, and change to
`$` on the next call.

Closes #7388

---------

Co-authored-by: Felipe Machado <462154+felipou@users.noreply.github.com>
2024-03-13 08:23:32 +02:00
Viktor SöderqvistandGitHub 9efc6ad6a6 Add API RedisModule_ClusterKeySlot and RedisModule_ClusterCanonicalKeyNameInSlot (#13069)
Sometimes it's useful to compute a key's cluster slot in a module.

This API function is just like the command CLUSTER KEYSLOT (but faster).

A "reverse" API is also added:
`RedisModule_ClusterCanonicalKeyNameInSlot`. Given a slot, it returns a
short string that we can call a canonical key for the slot.
2024-03-12 09:26:12 -07:00
Andy PanandGitHub 9c065c417d Enable accept4() on *BSD (#13104)
Redis enabled `accept4` on Linux after #9177, reducing extra system
calls for sockets.

`accept4` system call is also widely supported on *BSD and Solaris in
addition to Linux. This PR enables `accept4` on all platforms that
support it.

### References
- [accept4 on
FreeBSD](https://man.freebsd.org/cgi/man.cgi?query=accept4&sektion=2&n=1)
- [accept4 on
DragonFly](https://man.dragonflybsd.org/?command=accept&section=2)
- [accept4 on NetBSD](https://man.netbsd.org/accept.2)
- [accept4 on OpenBSD](https://man.openbsd.org/accept4.2)
- [accept4 on
Solaris](https://docs.oracle.com/cd/E88353_01/html/E37843/accept4-3c.html)
2024-03-12 16:35:52 +02:00
BinbinandGitHub da727ad445 Fix redis-check-aof incorrectly considering data in manifest format as MP-AOF (#12958)
The check in fileIsManifest misjudged the manifest file. For example,
if resp aof contains "file", it will be considered a manifest file and
the check will fail:
```
*3
$3
set
$4
file
$4
file
```

In #12951, if the preamble aof also contains it, it will also fail.
Fixes #12951.

the bug was happening if the the word "file" is mentioned
in the first 1024 lines of the AOF. and now as soon as it finds
a non-comment line it'll break (if it contains "file" or doesn't)
2024-03-12 08:47:43 +02:00
Harkrishn PatroandGitHub 3c8d15f8c3 Pick random slot for a node to distribute operation across slots in redis-benchmark (#12986)
Distribute operations via `redis-benchmark` on different slots owned by
node.

`current_slot_index` is never updated hence the value is always `0` and
the tag used is always the first slot owned by the node. Hence any
read/write operation via `redis-benchmark` in cluster mode always
happens on a particular slot.

This is inconvenient to load data uniformly via `redis-benchmark`.
2024-03-11 11:19:30 -07:00
5fdaa53d20 Fix conversion of numbers in lua args to redis args (#13115)
Since lua_Number is not explicitly an integer or a double, we need to
make an effort
to convert it as an integer when that's possible, since the string could
later be used
in a context that doesn't support scientific notation (e.g. 1e9 instead
of 100000000).

Since fpconv_dtoa converts numbers with the equivalent of `%f` or `%e`,
which ever is shorter,
this would break if we try to pass a long integer number to a command
that takes integer.
we'll get an implicit conversion to string in Lua, and then the parsing
in getLongLongFromObjectOrReply will fail.

```
> eval "redis.call('hincrby', 'key', 'field', '1000000000')" 0
(nil)
> eval "redis.call('hincrby', 'key', 'field', tonumber('1000000000'))" 0
(error) ERR value is not an integer or out of range script: ac99c32e4daf7e300d593085b611de261954a946, on @user_script:1.
```

Switch to using ll2string if the number can be safely represented as a
long long.

The problem was introduced in #10587 (Redis 7.2).
closes #13113.

---------

Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2024-03-10 08:46:49 +02:00
Madelyn OlsonandGitHub 4979cf02ff Change crc16 slot table to be fixed size character array instead of pointer to strings (#13112)
Update the crc16 hash lookup table to use fixed size character arrays instead of pointers 
to static string addresses. Since the actual values are so short, we can just store them
in a uniform array instead. This saves about 128kb of memory and should improve the 
performance as well since we should have much better memory locality.
2024-03-08 15:50:36 -08:00
debing.sunandGitHub 9738ba9841 Check user's oom_score_adj write permission for oom-score-adj test (#13111)
`CONFIG SET oom-score-adj handles configuration failures` test failed in
some CI jobs today.
Failed CI: https://github.com/redis/redis/actions/runs/8152519326

Not sure why the github action's docker image perssions have changed,
but the issue is similar to #12887,
where we can't assume the range of oom_score_adj that a user can change.

## Solution:
Modify the way of determining whether the current user has no privileges
or not,
instead of relying on whether the user id is 0 or not.
2024-03-05 14:42:28 +02:00
Ping XieandGitHub 28976a9003 Fix PONG message processing for primary-ship tracking during failovers (#13055)
This commit updates the processing of PONG gossip messages in the
cluster. When a node (B) becomes a replica due to a failover, its PONG
messages include its new primary node's (A) information and B's
configuration epoch is aligned with A's. This allows observer nodes to
identify changes in primary-ship, addressing issues of intermediate
states and enhancing cluster state consistency during topology changes.

Fix #13018
2024-03-04 17:32:25 -08:00
ad12730333 Implement defragmentation for pubsub kvstore (#13058)
After #13013

### This PR make effort to defrag the pubsub kvstore in the following
ways:

1. Till now server.pubsub(shard)_channels only share channel name obj
with the first subscribed client, now change it so that the clients and
the pubsub kvstore share the channel name robj.
This would save a lot of memory when there are many subscribers to the
same channel.
It also means that we only need to defrag the channel name robj in the
pubsub kvstore, and then update
all client references for the current channel, avoiding the need to
iterate through all the clients to do the same things.
    
2. Refactor the code to defragment pubsub(shard) in the same way as
defragment of keys and EXPIRES, with the exception that we only
defragment pubsub(without shard) when slot is zero.


### Other
Fix an overlook in #11695, if defragment doesn't reach the end time, we
should wait for the current
db's keys and expires, pubsub and pubsubshard to finish before leaving,
now it's possible to exit
early when the keys are defragmented.

---------

Co-authored-by: oranagra <oran@redislabs.com>
2024-03-04 16:56:50 +02:00
BinbinandGitHub 33ea432585 Call finalizerProc when free the aeTimeEvent in ae (#13101)
Supplement to #6189, we also need to call finalizerProc.
This is a minor cleanup, no one currently uses this finalizerProc
feature.
2024-03-03 09:20:18 +02:00
BinbinandGitHub df75153d79 Fix reply schemas validator build issue due to new regular expression (#13103)
The new regular expression break the validator:
```
In file included from commands.c:10:
commands_with_reply_schema.def:14528:72: error: stray ‘\’ in program
14528 | struct jsonObjectElement MEMORY_STATS_ReplySchema_patternProperties__db\_\d+__properties_overhead_hashtable_main_elements[] = {
```

The reason is that special characters are not added to to_c_name,
causes special characters to appear in the structure name, causing
c file compilation to fail.

Broken by #12913
2024-03-02 21:26:05 +02:00
a50bbcb656 redis-cli fixes around help hints version filtering (#13097)
- In removeUnsupportedArgs, trying to access the next item after the
last one and causing an out of bounds read.
- In versionIsSupported, when the 'version' is equal to 'since', the
return value is 0 (not supported).
Also, change the function to return `not supported` in case they have
different numbers of digits

Both issues were found by `Non-interactive non-TTY CLI: Test
command-line hinting - old server` under `test-sanitizer-address` (When
changing the `src/version.h` locally to `8.0.0`)

The new `MAXAGE` argument inside `client-kill` triggered the issue (new
argument at the end of the list)

---------

Co-authored-by: YaacovHazan <yaacov.hazan@redislabs.com>
2024-03-02 11:48:36 +02:00
4cae99e785 Add overhead of all DBs and rehashing dict count to info. (#12913)
Sometimes we need to make fast judgement about why Redis is suddenly
taking more memory. One of the reasons is main DB's dicts doing
rehashing.

We may use `MEMORY STATS` to monitor the overhead memory of each DB, but
there still lacks a total sum to show an overall trend. So this PR adds
the total overhead of all DBs to `INFO MEMORY` section, together with
the total count of rehashing DB dicts, providing some intuitive metrics
about main dicts rehashing.

This PR adds the following metrics to INFO MEMORY
* `mem_overhead_db_hashtable_rehashing` - only size of ht[0] in
dictionaries we're rehashing (i.e. the memory that's gonna get released
soon)

and a similar ones to MEMORY STATS:
* `overhead.db.hashtable.lut` (complements the existing
`overhead.hashtable.main` and `overhead.hashtable.expires` which also
counts the `dictEntry` structs too)
* `overhead.db.hashtable.rehashing` - temporary rehashing overhead.
* `db.dict.rehashing.count` - number of top level dictionaries being
rehashed.

---------

Co-authored-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2024-03-01 13:41:24 +08:00
BinbinandGitHub f17381a38d Fix propagation of entries_read by calling streamPropagateGroupID unconditionally (#12898)
In XREADGROUP ACK, because streamPropagateXCLAIM does not propagate
entries-read, entries-read will be inconsistent between master and
replicas.
I.e. if no entries were claimed, it would have propagated correctly, but
if some
were claimed, then the entries-read field would be inconsistent on the
replica.

The fix was suggested by guybe7, call streamPropagateGroupID
unconditionally,
so that we will normalize entries_read on the replicas. In the past, we
would
only set propagate_last_id when NOACK was specified. And in #9127,
XCLAIM did
not propagate entries_read in ACK, which would cause entries_read to be
inconsistent between master and replicas.

Another approach is add another arg to XCLAIM and let it propagate
entries_read,
but we decided not to use it. Because we want minimal damage in case
there's an
old target and new source (in the worst case scenario, the new source
doesn't
recognize XGROUP SETID ... ENTRIES READ and the lag is lost. If we
change XCLAIM,
the damage is much more severe).

In this patch, now if the user uses XREADGROUP .. COUNT 1 there will be
an additional
overhead of MULTI, EXEC and XGROUPSETID. We assume the extra command in
case of
COUNT 1 (4x factor, changing from one XCLAIM to
MULTI+XCLAIM+XSETID+EXEC), is probably
ok since reading just one entry is in any case very inefficient (a
client round trip
per record), so we're hoping it's not a common case.

Issue was introduced in #9127.
2024-02-29 09:48:20 +02:00
zhaozhao.zzandGitHub cc9fbd270e freeDictIfNeeded when kvstoreEmpty (#13098)
just like `kvstoreDictDelete`, we need check `freeDictIfNeeded` when
`kvstoreEmpty`.
2024-02-29 08:16:41 +02:00
BinbinandGitHub a7abc2f067 SCRIPT FLUSH run truly async, close lua interpreter in bio (#13087)
Even if we have SCRIPT FLUSH ASYNC now, when there are a lot of
lua scripts, SCRIPT FLUSH ASYNC will still block the main thread.
This is because lua_close is executed in the main thread, and lua
heap needs to release a lot of memory.

In this PR, we take the current lua instance on lctx.lua and call
lua_close on it in a background thread, to close it in async way.
This is MeirShpilraien's idea.
2024-02-28 17:57:29 +02:00
LiiNenandGitHub 763827c981 Fix redis-cli --count (for --scan, --bigkeys, etc) was ignored unless --pattern was also used (#13092)
The --count option for redis-cli has been released in redis 7.2.
https://github.com/redis/redis/pull/12042
But I have found in code, that some logic was missing for using this
'count' option.

```
static redisReply *sendScan(unsigned long long *it) {
    redisReply *reply;

    if (config.pattern)
        reply = redisCommand(context, "SCAN %llu MATCH %b COUNT %d",
            *it, config.pattern, sdslen(config.pattern), config.count);
    else
        reply = redisCommand(context,"SCAN %llu",*it);
```

The intention was being able to using scan count.
But in this case, the --count will be only applied when 'pattern' is
declared.
So, I had fix it simply, to be worked properly - even if --pattern
option is not being used.

I tested it simply with time() command several times, and I could see it
works as intended with this commit.
The examples of test results are below:
```
# unstable build

time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan >/dev/null 2>/dev/null)

real    0m1.287s
user    0m0.011s
sys     0m0.022s

# count is not applied
time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan --count 1000 >/dev/null 2>/dev/null)

real    0m1.117s
user    0m0.011s
sys     0m0.020s

# count is applied with --pattern

time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan --count 1000 --pattern "hash:*" >/dev/null 2>/dev/null)

real    0m0.045s
user    0m0.002s
sys     0m0.002s
```

```
# fix-redis-cli-scan-count build
time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan >/dev/null 2>/dev/null)

real    0m1.084s
user    0m0.008s
sys     0m0.024s

# count is applied even if --pattern is not declared
time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan --count 1000 >/dev/null 2>/dev/null)

real    0m0.043s
user    0m0.000s
sys     0m0.004s

# of course this also applied
time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan --count 1000 --pattern "hash:*" >/dev/null 2>/dev/null)

real    0m0.031s
user    0m0.002s
sys     0m0.002s
```



Thanks a lot.
2024-02-28 09:44:30 +02:00
0a12f380e8 Optimize DEL on expired keys (#13080)
If we call `DEL` on expired keys, keys may be deleted in
`expireIfNeeded` and we don't need to call `dbSyncDelete` or
`dbAsyncDelete` after, which repeat the deletion process(i.e. find keys
in main db).

In this PR, I refine the return values of `expireIfNeeded` to indicate
whether we have deleted the expired key to avoid the potential redundant
deletion logic in `delGenericCommand`. Besides, because both KEY_EXPIRED
and KEY_DELETED are non-zero, this PR won't affect other functions
calling `expireIfNeeded`.

I also make a performance test. I first close active expiration by
`debug set-active-expire 0` and write 1 million keys with 1ms TTL. Then
I repeatedly delete 100 expired keys in one `DEL`. The results are as
follow, which shows that this PR can improve performance by about 10% in
this situation.
**unstable**
```
Summary:
  throughput summary: 10080.65 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        0.953     0.136     0.959     1.215     1.335     2.247
```

**This PR**
```
Summary:			
  throughput summary: 11074.20 requests per second			
  latency summary (msec):			
          avg       min       p50       p95       p99       max			
        0.865     0.128     0.879     1.055     1.175     2.159			
```

---------

Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Oran Agra <oran@redislabs.com>
2024-02-26 12:50:04 +02:00
BinbinandGitHub 104b207602 Fix size stat in malloc(0) and cleanups around zmalloc file (#13068)
In #8554, we added a MALLOC_MIN_SIZE to use a minimum allocation
size when using malloc(0). However, we did not update the size,
when malloc_size is missing.

When malloc_size exists, we record the size that was allocated
instead of the size that was requested. This would work with both
jemalloc, and libc malloc (the change in #8554, doesn't break this).

When malloc_size is missing, we allocate extra size_t bytes and
store the requested size in it. In that case, the requested size
is probably different than the allocated size anyway (the change
in #8554 doesn't conceptually change that).

So we have room for improvement since in this case we are aware
of the extra bytes we asked for. Same as we're also aware of the
extra size_t bytes we asked for.

In addition, some cleaning was done:
1. fixes some outupdated comments.
2. test cleanups
2024-02-26 12:07:06 +02:00
BinbinandGitHub bfcaa7db0a Fix minor memory leak in rewriteSetObject (#13086)
It seems to be a leak caused by code refactoring in #11290.
it's a small leak, that only happens if there's an IO error.
2024-02-22 14:46:56 +02:00
4a265554ae Expose lua os.clock() api (#12971)
Implement #12699

This PR exposing Lua os.clock() api for getting the elapsed time of Lua
code execution.

Using:
```lua
local start = os.clock()
...
do something
...
local elpased = os.clock() - start
```

---------

Co-authored-by: Meir Shpilraien (Spielrein) <meir@redis.com>
Co-authored-by: Madelyn Olson <34459052+madolson@users.noreply.github.com>
2024-02-22 11:29:52 +02:00
debing.sunandGitHub 165afc5f2a Determine the large limit of the quicklist node based on fill (#12659)
Following #12568

In issue #9357, when inserting an element larger than 1GB, we currently
store it in a plain node instead of a listpack.
Presently, when we insert an element that exceeds the maximum size of a
packed node, it cannot be accommodated in any other nodes, thus ending
up isolated like a large element.
I.e. it's a node with only one element, but it's listpack encoded rather
than a plain buffer.

This PR lowers the threshold for considering an element as 'large' from
1GB to the maximum size of a node.
While this change doesn't completely resolve the bug mentioned in the
previous PR, it does mitigate its potential impact.

As a result of this change, we can now only use LSET to replace an
element with another element that falls below the maximum size
threshold.
In the worst-case scenario, with a fill of -5, the largest packed node
we can create is 2GB (32k * 64k):
* 32k: The smallest element in a listpack is 2 bytes, which allows us to
store up to 32k elements.
* 64k: This is the maximum size for a single quicklist node.

## Others
To fully fix #9357, we need more work, as discussed in #12568, when we
insert an element into a quicklistNode, it may be created in a new node,
put into another node, or merged, and we can't correctly delete the node
that was supposed to be deleted.
I'm not sure it's worth it, since it involves a lot of modifications.
2024-02-22 10:02:38 +02:00
guybe7andGitHub 820a4e45f1 Edit the history field of xinfo-consumers (#13078)
Now it matches the information in xinfo-stream.json
2024-02-22 09:44:29 +02:00
BinbinandGitHub 5b9fc46523 Add new allocator.muzzy field to memory-stats reply schema (#13076)
This field was added in #12996 but forgot to add it in json file.
This also causes reply-schemas-validator to fail.
2024-02-21 08:35:10 +02:00
f6785df663 Defragger improvements around large bins (#12996)
Implement #12963

## Changes
1. large bins don't have external fragmentation or are at least
non-defraggable, so we should ignore the effect of
large bins when measuring fragmentation, and only measure fragmentation
of small bins. this affects both the allocator_frag* metrics and also
the active-defrag trigger
2. Adding INFO metrics for `muzzy` memory, which is memory returned to
the OS but still shows as RSS until the OS reclaims it.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2024-02-20 18:11:09 +02:00
ca5cac998e xinfo-stream add minimum to seen-time, skip logreqres in fuzzer (#13056)
Recently I saw in CI that reply-schemas-validator fails here:
```
Failed validating 'minimum' in schema[1]['properties']['groups']['items']['properties']['consumers']['items']['properties']['active-time']:
    {'description': 'Last time this consumer was active (successful '
                    'reading/claiming).',
     'minimum': 0,
     'type': 'integer'}

On instance['groups'][0]['consumers'][0]['active-time']:
    -1729380548878722639
```

The reason is that in fuzzer, we may restore corrupted active-time,
which will cause the reply schema CI to fail.

The fuzzer can cause corrupt the state in many places, which will
bugs that mess up the reply, so we decided to skip logreqres.

Also, seen-time is the same type as active-time, adding the minimum.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2024-02-20 12:21:10 +02:00
BinbinandGitHub 3c2ea1ea95 Fix wathced client test timing issue caused by late close (#13062)
There is a timing issue in the test, close may arrive late, or in
freeClientAsync we will free the client in async way, which will
lead to errors in watching_clients statistics, since we will only
unwatch all keys when we truly freeClient.

Add a wait here to avoid this problem. Also fixed some outdated
comments i saw. The test was introduced in #12966.
2024-02-20 11:12:19 +02:00
BinbinandGitHub 4e3be944fc Fix timing issue in blockedclient test (#13071)
We can see that the past time here happens to be busy_time_limit,
causing the test to fail:
```
[err]: RM_Call from blocked client in tests/unit/moduleapi/blockedclient.tcl
Expected '50' to be more than '50' (context: type eval line 26 cmd {assert_morethan [expr [clock clicks -milliseconds]-$start] $busy_time_limit} proc ::test)
```

It is reasonable for them to be equal, so equal is added here.
It should be noted that in the previous `Busy module command` test,
we also used assert_morethan_equal, so this should have been missed
at the time.
2024-02-20 08:43:13 +02:00
judengandGitHub fc3a68d8fb add -fno-omit-frame-pointer to default complication flags (#12973)
Currently redis uses O3 level optimization would remove the frame pointer
in the target bin.

In the very old past, when gcc optimized at O1 and above levels, the
frame pointer is deleted by default to improve performance. This saves
the RBP registers and reduces the pop/push instructions. But it makes it
difficult for us to observe the running status of the program. For
example, the perf tool cannot be used effectively, especially the modern
eBPF tools such as bcc/memleak.
2024-02-19 11:47:02 -08:00
6df42df291 Adds a README to the command JSON files (#13066)
Add readme about the command json folder, what it does, and who should
(not) use it.
see discussion
https://github.com/redis/redis/issues/9359#issuecomment-1936420698

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2024-02-19 18:49:31 +02:00
zhaozhao.zzandGitHub 8876d264ac Calculate the incremental rehash time more precisely (#13063)
In the `databasesCron()`, the time consumed by
`kvstoreIncrementallyRehash()` is used to calculate the exit condition.
However, within `kvstoreIncrementallyRehash()`, the loop first checks
for timeout before performing rehashing. Therefore, the time for the
last rehash isn't accounted for, making the consumed time inaccurate. We
need to precisely calculate all the time spent on rehashing.
Additionally, the time allocated to `kvstoreIncrementallyRehash()`
should be the remaining time, which is
`INCREMENTAL_REHASHING_THRESHOLD_US` minus the already consumed
`elapsed_us`.
2024-02-19 14:29:54 +02:00
9103ccc398 AOF_FSYNC_EVERYSEC higher resolution, change aof_last_fsync and aof_flush_postponed_start to use mstime (#13041)
Currently aof_last_fsync is using a low resolution unixtime is really
bad,
it checks if the absolute number of (full) seconds changed by one.
depending on which side of the second barrier it falls, we can get very
different results.

This PR change the resolution to use milliseconds instead of complete
seconds.

In cases where the event loop cycle duration is short and their rapid
(e.g. running
many fast commands with short pipeline, or a high `hz` config), this
change will not
make much difference, since in anyway, we'll be quick to detect that
we're on a "new
second", and it's likely that these fsync will always be executed close
to the second
switch barrier.

But in cases of rare or slow event loops cycles (e.g. either slow
commands, or very
low rate of traffic to redis, and low `hz`), it could easily be that
with the old code,
in some cases we'll have over 1.5 seconds between fsyncs, and in others
less than 0.5.

see discussion in #8612

This PR also handle aof_flush_postponed_start as well, the damage there
is smaller
since the threshold is 2 seconds, and not 1.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2024-02-18 12:08:29 +02:00
BinbinandGitHub dd92dd8fb5 redis-cli - fix sscanf incorrect return-value check warnings (#13059)
From CodeQL: The result of scanf is only checked against 0, but
it can also return EOF.

Reported in https://github.com/redis/redis/security/code-scanning/38.
Reported in https://github.com/redis/redis/security/code-scanning/39.
2024-02-18 10:55:11 +02:00
zhaozhao.zzandGitHub 50d6fe8c4b Add metrics for WATCH (#12966)
Redis has some special commands that mark the client's state, such as
`subscribe` and `blpop`, which mark the client as `CLIENT_PUBSUB` or
`CLIENT_BLOCKED`, and we have metrics for the special use cases.

However, there are also other special commands, like `WATCH`, which
although do not have a specific flags, and should also be considered
stateful client types. For stateful clients, in many scenarios, the
connections cannot be shared in "connection pool", meaning connection
pool cannot be used. For example, whenever the `WATCH` command is
executed, a new connection is required to put the client into the "watch
state" because the watched keys are stored in the client.

If different business logic requires watching different keys, separate
connections must be used; otherwise, there will be contamination. This
also means that if a user's business heavily relies on the `WATCH`
command, a large number of connections will be required.

Recently we have encountered this situation in our platform, where some
users consume a significant number of connections when using Redis
because of `WATCH`.

I hope we can have a way to observe these special use cases and special
client connections. Here I add a few monitoring metrics:

1. `watching_clients` in `INFO` reply: The number of clients currently
in the "watching" state.
2. `total_watched_keys` in `INFO` reply: The total number of keys being
watched.
3. `watch` in `CLIENT LIST` reply: The number of keys each client is
currently watching.
2024-02-18 10:36:41 +02:00
BinbinandGitHub c854873746 Minor optimization in kvstoreDictAddRaw when dict exists (#13054)
Usually, the probability that a dict exists is much greater than the
probability that it does not exist. In kvstoreDictAddRaw, we will call
kvstoreGetDict multiple times. Based on this assumption, we change
createDictIfNeeded to something like get or create function:
```
before:
dict exist: 2 kvstoreGetDict
dict non-exist: 2 kvstoreGetDict

after:
dict exist: 1 kvstoreGetDict
dict non-exist: 3 kvstoreGetDict
```

A possible 3% performance improvement was observed:

In addition, some typos/comments i saw have been cleaned up.
2024-02-15 18:07:24 +02:00
BinbinandGitHub 063de675e0 zunionInterDiffGenericCommand use ztrycalloc to avoid OOM panic (#13052)
In low memory situations, sending a big number of arguments (sets)
may cause OOM panic. Use ztrycalloc, like we do on LCS and XAUTOCLAIM,
and fail gracefully.

This change affects the following commands: ZUNION, ZINTER, ZDIFF,
ZUNIONSTORE, ZINTERSTORE, ZDIFFSTORE, ZINTERCARD.
2024-02-15 10:49:10 +02:00
BinbinandGitHub 32f44da510 Increase tolerance range to block reprocess tests to avoid timing issues (#13053)
These tests have all failed in daily CI:
```
*** [err]: Blocking XREADGROUP for stream key that has clients blocked on stream - reprocessing command in tests/unit/type/stream-cgroups.tcl
Expected '1101' to be between to '1000' and '1100' (context: type eval line 23 cmd {assert_range [expr $end-$start] 1000 1100} proc ::test)

*** [err]: BLPOP unblock but the key is expired and then block again - reprocessing command in tests/unit/type/list.tcl
Expected '1101' to be between to '1000' and '1100' (context: type eval line 23 cmd {assert_range [expr $end-$start] 1000 1100} proc ::test)

*** [err]: BZPOPMIN unblock but the key is expired and then block again - reprocessing command in tests/unit/type/zset.tcl
Expected '1103' to be between to '1000' and '1100' (context: type eval line 23 cmd {assert_range [expr $end-$start] 1000 1100} proc ::test)
```

Increase the range to avoid failures, and improve the comment to be
clearer.
tests was introduced in #13004.
2024-02-15 10:44:49 +02:00
SankarandGitHub c1d2ac2a73 Do not include gossip about receiver in cluster messages (#13046)
The receiver does not update any of its cluster state based on gossip
about itself. This commit explicitly avoids sending or processing gossip
about the receiver.

Currently cluster bus gossips include 10% of nodes in the cluster with a
minimum of 3 nodes. For up to 30 node clusters, this commit makes sure
that 1/3 of the gossip (1 out of 3 gossips) is never discarded. This
should help with relatively faster convergence of cluster state in
general.
2024-02-13 16:38:37 -08:00
e9c795e777 Fix loading rdb opcode RDB_OPCODE_RESIZEDB (#13050)
Following the changes introduced by 8cd62f82c, the dbExpandExpires used
the db_size instead of expires_size.

Co-authored-by: YaacovHazan <yaacov.hazan@redislabs.com>
2024-02-12 21:55:37 +02:00
7ca0b84af6 Fix loading rdb opcode RDB_OPCODE_SLOT_INFO (#13049)
Following the changes introduced by 8cd62f82c, the kvstoreDictExpand for
the expires kvstore used the slot_size instead of expires_slot_size.

Co-authored-by: YaacovHazan <yaacov.hazan@redislabs.com>
2024-02-12 21:46:06 +02:00
BinbinandGitHub 8eeece4ab3 Fix CLIENAT KILL MAXAGE test timing issue (#13047)
This test fails occasionally:
```
*** [err]: CLIENT KILL maxAGE will kill old clients in tests/unit/introspection.tcl
Expected 2 == 1 (context: type eval line 14 cmd {assert {$res == 1}} proc ::test)
```

This test is very likely to do a false positive if the execute time
takes longer than the max age, for example, if the execution time
between sleep and kill exceeds 1s, rd2 will also be killed due to
the max age.

The test can adjust the order of execution statements to increase
the probability of passing, but this is still will be a timing issue
in some slow machines, so decided give it a few more chances.

The test was introduced in #12299.
2024-02-12 08:11:33 +02:00
676f27acb0 Fix the failure of defrag test under 32-bit (#13013)
Fail CI:
https://github.com/redis/redis/actions/runs/7837608438/job/21387609715

## Why defragment tests only failed under 32-bit

First of all, under 32-bit jemalloc will allocate more small bins and
less large bins, which will also lead to more external fragmentation,
therefore, the fragmentation ratio is higher in 32-bit than in 64-bit,
so the defragment tests(`Active defrag eval scripts: cluster` and
`Active defrag big keys: cluster`) always fails in 32-bit.

## Why defragment tests only failed with cluster
The fowllowing is the result of `Active defrag eval scripts: cluster`
test.

1) Before #11695, the fragmentation ratio is 3.11%.

2) After #11695, the fragmentation ratio grew to 4.58%.
Since we are using per-slot dictionary to manage slots, we will only
defragment the contents of these dictionaries (keys, values), but not
the dictionaries' struct and ht_table, which means that frequent
shrinking and expanding of the dictionaries, will make more fragments.

3) After #12850 and #12948, In cluster mode, a large number of cluster
slot dicts will be shrunk, creating additional fragmention, and the
dictionary will not be defragged.

## Solution
* Add defragmentation of the per-slot dictionary's own structures, dict
struct and ht_table.

## Other change
* Increase floating point print precision of `frags` and `rss` in debug
logs for defrag

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2024-02-11 15:12:42 +02:00
BinbinandGitHub 493e31e3ad Add new DEBUG dict-resizing command to disable the dict resize (#13043)
The test fails here and there:
```
*** [err]: expire scan should skip dictionaries with lot's of empty buckets in tests/unit/expire.tcl
scan didn't handle slot skipping logic.
```

There are two case:
1. In the case of passing the test, we use child process to avoid the
dict resize, but it can not completely limit it, since in the dictDelete
we still have chance to trigger the resize (hit the force radio). The
reason why our test passed before is because the expire dict is still
in the rehashing process, so the dictDelete, the dictShrinkIfNeeded can
not trigger the resize.

2. In the case of failing the test, the expire dict finished the
rehashing,
so the last dictDelete, the dictShrinkIfNeeded trigger the dict resize
since it hit the force radio, so the skipping logic fail.

This PR add a new DEBUG command to disbale the dict resize.
2024-02-08 16:39:58 +02:00
BinbinandGitHub 813327b231 Fix SORT STORE quicklist with the right options (#13042)
We forgot to call quicklistSetOptions after createQuicklistObject,
in the sort store scenario, we will create a quicklist with default
fill or compress options.

This PR adds fill and depth parameters to createQuicklistObject to
specify that options need to be set after creating a quicklist.

This closes #12871.

release notes:
> Fix lists created by SORT STORE to respect list compression and
packing configs.
2024-02-08 14:36:11 +02:00
debing.sunandGitHub 1e8dc1da0d Fix crash due to merge of quicklist node introduced by #12955 (#13040)
Fix two crash introducted by #12955

When a quicklist node can't be inserted and split, we eventually merge
the current node with its neighboring
nodes after inserting, and compress the current node and its siblings.

1. When the current node is merged with another node, the current node
may become invalid and can no longer be used.

   Solution: let `_quicklistMergeNodes()` return the merged nodes.

3. If the current node is a LZF quicklist node, its recompress will be
1. If the split node can be merged with a sibling node to become head or
tail, recompress may cause the head and tail to be compressed, which is
not allowed.

    Solution: always recompress to 0 after merging.
2024-02-08 14:29:16 +02:00
81666a6510 Fix heap-use-after-free when pubsubshard_channels became NULL (#13038)
After fix for #13033, address sanitizer reports this heap-use-after-free
error. When the pubsubshard_channels dict becomes empty, we will delete
the dict, and the dictReleaseIterator will call dictResetIterator, it
will use the dict so we will trigger the error.

This PR introduced a new struct kvstoreDictIterator to wrap
dictIterator.
Replace the original dict iterator with the new kvstore dict iterator.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: guybe7 <guy.benoish@redislabs.com>
2024-02-07 14:53:50 +02:00
BinbinandGitHub 886b117031 Fix dict don't rehash when there is child test (#13035)
The reason is the same as #13016. The reason is that in #12819,
in cron, in addition to trying to shrink, we will also tyring
to expand. The dict was expanded by cron before we trigger the
bgsave since we do have the enough keys (4096) to hit the radio.

Before the bgsave, we only add 4095 keys to avoid this issue.
2024-02-07 09:19:18 +02:00
debing.sunandGitHub 1f00c951c2 Prevent LSET command from causing quicklist plain node size to exceed 4GB (#12955)
Fix #12864

The main reason for this crash is that when replacing a element of a
quicklist packed node with lpReplace() method,
if the final size is larger than 4GB, lpReplace() will fail and returns
NULL, causing `node->entry` to be incorrectly set to NULL.

Since the inserted data is not a large element, we can't just replace it
like a large element, first quicklistInsertAfter()
and then quicklistDelIndex(), because the current node may be merged and
invalidated in quicklistInsertAfter().

The solution of this PR:
When replacing a node fails (listpack exceeds 4GB), split the current
node, create a new node to put in the middle, and try to merge them.
This is the same as inserting a large element.
In the worst case, its size will not exceed 4GB.
2024-02-06 18:21:28 +02:00
GannandGitHub 0777dc7896 Improve error handling in connSocketBlockingConnect for various connction failures (#13008)
This commit addresses a problem in connSocketBlockingConnect where
different types of connection failures, including timeouts and other
errors, were not consistently handled. Previously, the function did not
return C_ERR immediately after detecting a connection failure, which
could lead to inconsistent states and misinterpretation of the
connection status.

With this update, connSocketBlockingConnect now correctly returns C_ERR
upon encountering any connection error, ensuring that all types of
connection failures are handled consistently and the behavior of the
function aligns with expected outcomes in case of connection issues.

Closes #12900
2024-02-06 14:31:08 +02:00
BinbinandGitHub 8096515432 Fix invalid dictNext usage when pubsubshard_channels became empty (#13033)
After #12822, when pubsubshard_channels became empty, kvstoreDictDelete
will delete the dict (which is the only one currently deleting dicts
that
become empty) and in the next loop, we will make an invalid call to
dictNext.

After the dict becomes empty, we break out of the loop without calling
dictNext.
2024-02-06 13:41:02 +02:00
BinbinandGitHub 13bd3643c2 Re-compute active_defrag_running after adjusting defrag configurations (#13020)
Currently, once active defrag starts, we can not adjust
active_defrag_running
downwards. This is because active_defrag_running will be dynamically
compute
based on the fragmentation, we think we should not lower the effort when
the
fragmentation drops.

However, we need to note that active_defrag_running will also be
dynamically
computed based on configurations. In this case, we are not respecting
cycle-min
or cycle-max. Some people may realize halfway through that defrag
consumes a
lot and want to adjust it.

Previously we could only turn off activedefrag and then turn it on again
to
adjust active_defrag_running downwards. So in this PR, when a active
defrag
configuration change is made, we will re-compute it.

These configuration items are:
- active-defrag-cycle-min
- active-defrag-cycle-max
- active-defrag-threshold-upper
2024-02-06 13:39:07 +02:00
BinbinandGitHub 87eaf119cd Minor optimization for expire dict in defragKey (#13027)
Since now a DB in cluster mode is divided into 16384 dicts, here
we directly check kvstoreDictSize instead of kvstoreSize, which
may have a higher probability that we can save the lookup.

The other change is a cleanup, obviously kvstoreGetHash should be
applied to the db->expires dicts.
2024-02-06 12:19:44 +02:00
84fd745d65 Fix kvstore unable to push resize_cursor for resize when dict is NULL (#13031)
When the dict is NULL, we also need to push resize_cursor, otherwise it
will keep doing useless continue here, and there is no way to resize the
other dict behind it.

Introduced in #12822.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2024-02-06 09:41:14 +02:00
guybe7andGitHub 8cd62f82ca Refactor the per-slot dict-array db.c into a new kvstore data structure (#12822)
# Description
Gather most of the scattered `redisDb`-related code from the per-slot
dict PR (#11695) and turn it to a new data structure, `kvstore`. i.e.
it's a class that represents an array of dictionaries.

# Motivation
The main motivation is code cleanliness, the idea of using an array of
dictionaries is very well-suited to becoming a self-contained data
structure.
This allowed cleaning some ugly code, among others: loops that run twice
on the main dict and expires dict, and duplicate code for allocating and
releasing this data structure.

# Notes
1. This PR reverts the part of https://github.com/redis/redis/pull/12848
where the `rehashing` list is global (handling rehashing `dict`s is
under the responsibility of `kvstore`, and should not be managed by the
server)
2. This PR also replaces the type of `server.pubsubshard_channels` from
`dict**` to `kvstore` (original PR:
https://github.com/redis/redis/pull/12804). After that was done,
server.pubsub_channels was also chosen to be a `kvstore` (with only one
`dict`, which seems odd) just to make the code cleaner by making it the
same type as `server.pubsubshard_channels`, see
`pubsubtype.serverPubSubChannels`
3. the keys and expires kvstores are currenlty configured to allocate
the individual dicts only when the first key is added (unlike before, in
which they allocated them in advance), but they won't release them when
the last key is deleted.

Worth mentioning that due to the recent change the reply of DEBUG
HTSTATS changed, in case no keys were ever added to the db.

before:
```
127.0.0.1:6379> DEBUG htstats 9
[Dictionary HT]
Hash table 0 stats (main hash table):
No stats available for empty dictionaries
[Expires HT]
Hash table 0 stats (main hash table):
No stats available for empty dictionaries
```

after:
```
127.0.0.1:6379> DEBUG htstats 9
[Dictionary HT]
[Expires HT]
```
2024-02-05 17:21:35 +02:00
BinbinandGitHub f20774eced Fix active expire timeout when db done the scanning (#13030)
When db->expires_cursor==0, it means the DB is done the scanning,
we should exit the loop to avoid the useless scanning.

It is easy to see the active expire timeout in the modified test,
for example, let's assume that there is only 1 expired key in the
DB, and the size / buckets ratio is less than 1%, which means that
we will skip it in isExpiryDictValidForSamplingCb, and the return
value of expires_cursor is 0.

Because `data.sampled == 0` is always true, so `repeat` is also
always true, we will keep scanning the DB, but every time it is
skipped by the previous judgment (expires_cursor = 0), until the
timelimit is finally exhausted.
2024-02-05 16:56:46 +02:00
02a87885e6 Add missing structural API changes to JSON file (#12434)
The JSON file lacks the following structural API changes:

- GEORADIUSBYMEMBER: add the ANY option for COUNT since 6.2.0.
- GEORADIUSBYMEMBER_RO: add the ANY option for COUNT since 6.2.0.
- GEORADIUS_RO: Added support for uppercase unit names since 7.0.0.
- GEORADIUSBYMEMBER_RO: Added support for uppercase unit names since
7.0.0.

---------

Signed-off-by: daz-3ux <daz-3ux@proton.me>
Co-authored-by: bodong.ybd <bodong.ybd@alibaba-inc.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: yangpengda.333 <yangpengda.333@bytedance.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2024-02-04 08:42:15 +02:00
c1041c2c0d Make db->avg_ttl more precise (#12949)
Currently, We compute `db->avg_ttl` after each short `dbScan` sweep (a
few buckets without checking the time limit). But after each `dbScan`
sweep, we don't have much data and this makes the db->avg_ttl less
precise. For example, even if we scan the whole db, we can't get the
exact avg_ttl because we separate the data.
i.e. because of the running average, if we issue 16 calls to scan, we'll
give lower weight to the first one, and higher weight to the last one.
I think we should calculate `db->avg_ttl` until completing more of the
db iteration (judgement of time limit or the beginning of iterating next
db) because we have more sample data in this db and can get more
accurate result. In the best case, if we scan the whole db, we can get
the exact avg_ttl.

In this PR, we postpone the avg_ttl calculation until the judgement of
time limit or iteration of next db, so we can accumulate more data to
get more precise avg_ttl.
Note that we still need to make sure to decay the old TTLs at the same
speed as before, which is why we want to run the decay mechanism several
times, or use the Pow formula, see the comment in the code.

In my experiment, this PR can improve 89% or 52% accuracy in different
workload.

Co-authored-by: Oran Agra <oran@redislabs.com>
2024-02-04 08:34:26 +02:00
Yanqi LvandGitHub 62153b3b2f Refine the purpose of rdb saving with accurate flags (#12925)
In Redis, rdb is produced in three scenarios mainly.

- backup, such as `bgsave` and `save` command
- full sync in replication
- aof rewrite if `aof-use-rdb-preamble` is yes

We also have some RDB flags to identify the purpose of rdb saving.
```C
/* flags on the purpose of rdb save or load */
#define RDBFLAGS_NONE 0                 /* No special RDB loading. */
#define RDBFLAGS_AOF_PREAMBLE (1<<0)    /* Load/save the RDB as AOF preamble. */
#define RDBFLAGS_REPLICATION (1<<1)     /* Load/save for SYNC. */
```

But currently, it seems that these flags and purposes of rdb saving
don't exactly match. I find it in `rdbSaveRioWithEOFMark` which calls
`startSaving` with `RDBFLAGS_REPLICATION` but `rdbSaveRio` with
`RDBFLAGS_NONE`.
```C
int rdbSaveRioWithEOFMark(int req, rio *rdb, int *error, rdbSaveInfo *rsi) {
    char eofmark[RDB_EOF_MARK_SIZE];

    startSaving(RDBFLAGS_REPLICATION);
    getRandomHexChars(eofmark,RDB_EOF_MARK_SIZE);
    if (error) *error = 0;
    if (rioWrite(rdb,"$EOF:",5) == 0) goto werr;
    if (rioWrite(rdb,eofmark,RDB_EOF_MARK_SIZE) == 0) goto werr;
    if (rioWrite(rdb,"\r\n",2) == 0) goto werr;
    if (rdbSaveRio(req,rdb,error,RDBFLAGS_NONE,rsi) == C_ERR) goto werr;
    if (rioWrite(rdb,eofmark,RDB_EOF_MARK_SIZE) == 0) goto werr;
    stopSaving(1);
    return C_OK;

werr: /* Write error. */
    /* Set 'error' only if not already set by rdbSaveRio() call. */
    if (error && *error == 0) *error = errno;
    stopSaving(0);
    return C_ERR;
}
```

In this PR, I refine the purpose of rdb saving with accurate flags.
2024-02-01 13:41:02 +02:00
BinbinandGitHub 9a7d311855 Fix dict resize allow test (#13016)
Ci report this failure:
```
*** [err]: Don't rehash if used memory exceeds maxmemory after rehash in tests/unit/maxmemory.tcl
Expected '4098' to equal or match '4002'

WARNING: the new maxmemory value set via CONFIG SET (1176088) is smaller than the current memory usage (1231083)
```

It can be seen from the log that used_memory changed before we set
maxmemory.
The reason is that in #12819, in cron, in addition to trying to shrink,
we will
also tyring to expand. The dict was expanded by cron before we set
maxmemory,
causing the test to fail.

Before setting maxmemory, we only add 4095 keys to avoid triggering
resize.
2024-01-31 13:11:52 +02:00
BinbinandGitHub 6016973ac0 Fix module assertion crash when timer and timeout are unlocked in the same event loop (#13015)
When we use a timer to unblock a client in module, if the timer
period and the block timeout are very close, they will unblock the
client in the same event loop, and it will trigger the assertion.
The reason is that in moduleBlockedClientTimedOut we will protect
against re-processing, so we don't actually call updateStatsOnUnblock
(see #12817), so we are not able to reset the c->duration. 

The reason is unblockClientOnTimeout() didn't realize that bc had
been unblocked. We add a function to the module to determine if bc
is blocked, and then use it in unblockClientOnTimeout() to exit.

There is the stack:
```
beforeSleep
blockedBeforeSleep
handleBlockedClientsTimeout
checkBlockedClientTimeout
unblockClientOnTimeout
unblockClient
resetClient
-- assertion, crash the server
'c->duration == 0' is not true
```
2024-01-31 13:10:19 +02:00
BinbinandGitHub 74a6e48a3d Fix module unblock crash due to no timeout_callback (#13017)
The block timeout is passed in the test case, but we do not pass
in the timeout_callback, and it will crash when unlocking. In this
case, in moduleBlockedClientTimedOut we will check timeout_callback.
There is the stack:
```
beforeSleep
blockedBeforeSleep
handleBlockedClientsTimeout
checkBlockedClientTimeout
unblockClientOnTimeout
replyToBlockedClientTimedOut
moduleBlockedClientTimedOut
-- timeout_callback is NULL, invalidFunctionWasCalled
bc->timeout_callback(&ctx,(void**)c->argv,c->argc);
```
2024-01-31 09:28:50 +02:00
f469dd8ca6 Add novalues option to command HSCAN. (#12765)
Add a way to HSCAN a hash key, and get only the filed names.
Command syntax is now:
```
HSCAN key cursor [MATCH pattern] [COUNT count] [NOVALUES]
```
when `NOVALUES` is on, the command will only return keys in the hash.

---------

Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2024-01-30 20:32:58 +02:00
24f6d08b3f Implement CLIENT KILL MAXAGE <maxage> (#12299)
Adds an ability to kill clients older than a specified age.

Also, fixed the age calculation in `catClientInfoString` to use
`commandTimeSnapshot`
instead of the old `server.unixtime`, and added missing documentation
for
`CLIENT KILL ID` to output of `CLIENT help`.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2024-01-30 20:24:36 +02:00
Oran AgraandGitHub 7c9f41b52b fix dict rehash tests introduced by #12802 broken by #12819 (#13009)
tests consistently fail on timeout (sleep that's too short).
it now takes more time because in #12819 we iterate on all dicts, not
just non-empty ones.
it passed the PR's CI because it skips the `slow` tag, which might have
been misplaced, but now it is probably required.
with the fix, the tests take quite a lot of time:
```
[ok]: Redis can trigger resizing (1860 ms)
[ok]: Redis can rewind and trigger smaller slot resizing (744 ms)
```
before #12819:
```
[ok]: Redis can trigger resizing (309 ms)
[ok]: Redis can rewind and trigger smaller slot resizing (295 ms)
```

failure:
https://github.com/redis/redis/actions/runs/7704158180/job/20995931735
```
*** [err]: expire scan should skip dictionaries with lot's of empty buckets in tests/unit/expire.tcl
scan didn't handle slot skipping logic.
*** [err]: Redis can trigger resizing in tests/unit/other.tcl
Expected '[Dictionary HT]
Hash table 0 stats (main hash table):
 table size: 128
 number of elements: 5
[Expires HT]
Hash table 0 stats (main hash table):
No stats available for empty dictionaries
' to match '*table size: 8*' (context: type eval line 29 cmd {assert_match "*table size: 8*" [r debug HTSTATS 0]} proc ::test) 
*** [err]: Redis can rewind and trigger smaller slot resizing in tests/unit/other.tcl
Expected '[Dictionary HT]
Hash table 0 stats (main hash table):
 table size: 256
 number of elements: 10
[Expires HT]
Hash table 0 stats (main hash table):
No stats available for empty dictionaries
' to match '*table size: 16*' (context: type eval line 27 cmd {assert_match "*table size: 16*" [r debug HTSTATS 0]} proc ::test) 
```
2024-01-30 14:32:38 +02:00
BinbinandGitHub 45a35a79c7 Fix timeout not being set in module blockClient case (#13011)
This was introduced in #13004, missing this assignment.
It causes timeout to be a random value (may be less than now),
and then in `Unblock by timer` test, the client is unblocked
and then it call timeout_callback, since the callback is NULL,
the server will crash.

The crash stack is:
```
beforesleep
handleBlockedClientsTimeout
checkBlockedClientTimeout
unblockClientOnTimeout
replyToBlockedClientTimedOut
moduleBlockedClientTimedOut
-- the timeout_callback is NULL, invalidFunctionWasCalled
bc->timeout_callback(&ctx,(void**)c->argv,c->argc);
```
2024-01-30 14:32:17 +02:00
76adbf6ff0 Adds connection timeout option to redis-cli (#10609)
This allows specifying the timeout value for opening the TCP
connection to a server. The timeout, default 0 means no limit,
depending on the OS. It can be specified using the new `-t` switch.

revive #3764, fixes #3763

---------

Co-authored-by: Itamar Haber <itamar@redislabs.com>
Co-authored-by: yoav-steinberg <yoav@redislabs.com>
2024-01-30 13:43:39 +02:00
BinbinandGitHub 492021db95 Fix blocking commands timeout is reset due to re-processing command (#13004)
In #11012, we will reprocess command when client is unblocked on keys,
in some blocking commands, for example, in the XREADGROUP BLOCK
scenario,
because of the re-processing command, we will recalculate the block
timeout,
causing the blocking time to be reset.

This commit add a new CLIENT_REPROCESSING_COMMAND clent flag, explicitly
let the command know that it is being re-processed, later in
blockForKeys
we will not reset the timeout.

Affected BLOCK cases: 
- list / zset / stream, added test cases for each.

Unaffected cases:
- module (never re-process the commands).
- WAIT / WAITAOF (never re-process the commands).

Fixes #12998.
2024-01-30 11:32:59 +02:00
Chen TianjieandGitHub af7ceeb765 Optimize resizing hash table to resize not only non-empty dicts. (#12819)
The function `tryResizeHashTables` only attempts to shrink the dicts
that has keys (change from #11695), this was a serious problem until the
change in #12850 since it meant if all keys are deleted, we won't shrink
the dick.
But still, both dictShrink and dictExpand may be blocked by a fork child
process, therefore, the cron job needs to perform both dictShrink and
dictExpand, for not just non-empty dicts, but all dicts in DBs.

What this PR does:

1. Try to resize all dicts in DBs (not just non-empty ones, as it was
since #12850)
2. handle both shrink and expand (not just shrink, as it was since
forever)
3. Refactor some APIs about dict resizing (get rid of `htNeedsShrink`
`htNeedsShrink` `dictShrinkToFit`, and expose `dictShrinkIfNeeded`
`dictExpandIfNeeded` which already contains all the code of those
functions we get rid of, to make APIs more neat)
4. In the `Don't rehash if redis has child process` test, now that cron
would do resizing, we no longer need to write to DB after the child
process got killed, and can wait for the cron to expand the hash table.
2024-01-29 21:02:07 +02:00
Ozan TezcanandGitHub c5273cae18 Add RM_TryCalloc() and RM_TryRealloc() (#12985)
Modules may want to handle allocation failures gracefully. Adding
RM_TryCalloc() and RM_TryRealloc() for it.
RM_TryAlloc() was added before:
https://github.com/redis/redis/pull/10541
2024-01-29 20:56:03 +02:00
BinbinandGitHub acd9605223 Fix maxmemory-samples stack overflow crash in evictionPoolPopulate, limit its value to [1,64] (#13000)
We have not limited the value of maxmemory-samples in the past, it can
be set very large. If it is set very large, we will have stack overflow
in evictionPoolPopulate when we trigger the key eviction.

There is no reason for this config to be set too high, so just limit its
range to [1,64].
2024-01-29 10:38:52 +02:00
5358bd7cdd Reduce performance impact of dict rehashing and make it shorter. (#12899)
#### Problem Statement:
For any read/update operation during rehashing, we're doing ~10+ random
DRAM lookups to do the rehashing, as we are using the `rehashidx` to
rehash 10 buckets, whose dict entries most likely aren't cached in the
CPU or near the bucket we are operating on. If these random bucket are
empty, the rehashing process during that command execution is skipped.

#### Implementation:
For reducing the performance recession while dict is rehashing, we
determine the index at which the key would be stored in the 0th HT, we
check if that index has already been rehashed, if not we will rehash the
bucket containing the key and the bucket will be moved from 0th HT to
the 1st HT.

If the key has already been rehashed, we perform the random access
bucket rehash (using `rehashidx`) and we again verify if rehashing is
still ongoing and look up the key in the respective HT.

This ensures rehashing is not skipped in any command call and that we
rehash a particular bucket or random bucket in each call.

#### Changes in this PR:
- Added a new method `dictBucketRehash` to perform rehash on a single
bucket.
- Helper function `moveKeysInBucketOldtoNew` for `dictRehash` and
`dictBucketRehash` to move all the keys in a bucket from the old to the
new hash HT.
- Helper function `verifyMoreRehashRequired` for `dictRehash` and
`dictBucketRehash` to check if we have already rehashed the whole table
and if more rehashing is required.

### Benchmark:
- This PR still shows **~13%** improvement in the latency during
rehashing.

- Rehashing is now **~2%** faster for this PR when compared to unstable.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Madelyn Olson <34459052+madolson@users.noreply.github.com>
2024-01-27 11:11:53 +02:00
judengandGitHub 98881f7558 fix the wrong path in mkreleasehdr.sh (#12993)
The question is introduced in #12799 , the script cannot find the
correct src and deps directories, so it always returns dirty as 0.
2024-01-26 15:01:54 -08:00
BinbinandGitHub 4cb5ad85a5 Fix unauthenticated client query buffer 1MB limit (#12989)
Code incorrectly set the limit value to 1024MB.
Introduced in #12961.
2024-01-25 14:56:21 +02:00
zhaozhao.zzandGitHub 85a834bfa2 Revert multi OOM limit and add multi buffer limit (#12961)
Fix #9926 , and introduce an alternative method to prevent abuse of
transactions:

1. revert #5454 (which was blocking read-only transactions in OOM
state), and break the tie of MULTI state memory usage and the server OOM
state. Meaning that we'll limit the total memory a single client can
queue, and do that unconditionally regardless of the server being OOM or
not.
2. to prevent abuse of transactions, we use the
`client-query-buffer-limit` to restrict the size of the transaction.
Because the commands cached in the MULTI/EXEC queue have not been
executed yet, so they are also considered a part of the "query buffer"
in a broader sense. In other words, the commands in the MULTI queue and
the `querybuf` of the client together constitute the "query buffer".
When they exceed the limit, the connection will be disconnected.

The reasoning is that it's sensible to sends a single command with a
huge (1GB) argument, and it's sensible to sends a transaction with many
small commands, but it's probably not common to sends a long transaction
with many huge arguments (will consume a lot of memory before even being
executed).

If anyone runs into that, they can simply increase the
`client-query-buffer-limit` config.

P.S. To prevent DDoS attacks, unauthenticated clients have a separate
hard limit. Their query buffer should not exceed a maximum of 1MB. In
other words, if the query buffer of an unauthenticated client exceeds
1MB or the `client-query-buffer-limit` (if it is set to a value smaller
than 1MB,), the connection will be disconnected.
2024-01-25 11:17:39 +02:00
BinbinandGitHub 07b292af5e Add sender NULL check in clusterProcessGossipSection invalid_ids case (#12980)
In the following case sender may be unknown, so we need to set up a
NULL check for sender:
```
/* If this is a MEET packet from an unknown node, we still process
 * the gossip section here since we have to trust the sender because
 * of the message type. */
if (!sender && type == CLUSTERMSG_TYPE_MEET)
    clusterProcessGossipSection(hdr,link);
```
2024-01-23 09:45:02 -08:00
Wen HuiandGitHub 685409139b Add INCR type command against wrong argument test cases. (#12836)
We have test cases for incr related commands with no key exist and
spaces in key and wrong type of key. However, we dont have test cases
covered for INCRBY INCRBYFLOAT DECRBY INCR DECR HINCRBY HINCRBYFLOAT
ZINCRBY with valid key and invalid value as argument, and float value to
incrby and decrby. So added test cases for the scenarios in incr.tcl.

Thank you!
2024-01-23 15:39:38 +02:00
BinbinandGitHub 85c31e0cff Allow running WAITAOF in scripts, remove NOSCRIPT flag (#12977)
In #11568 we removed the NOSCRIPT flag from commands, e.g. removing
NOSCRIPT flag from WAIT. Aiming to allow them in scripts and let them
implicitly behave in the non-blocking way.

This PR remove NOSCRIPT flag from WAITAOF just like WAIT (to be
symmetrical)).
And this PR also add BLOCKING flag for WAIT and WAITAOF.
2024-01-23 15:19:41 +02:00
BinbinandGitHub 628c0dea1b Some cleanups around function (#12940)
This PR did some cleanups around function:
- drop the comment about Libraries Ctx, since we do have comment
  in functionsLibCtx, no need to maintain multiple copies.
- remove outdated comment about the dropped Library description.
- remove unused desc and code vars in functionExtractLibMetaData.
- fix engines_nemory typo, changed it to engines_memory.
- remove outdated comment about FUNCTION CREATE and FUNCTION INFO,
  FUNCTION CREATE was renamed to FUNCTION LOAD.
- Check in initServer whether the return of functionsInit is OK.
2024-01-23 14:26:33 +02:00
Oran AgraandGitHub f9a0eb60f7 update redis-check-rdb types (#12969)
seems that we forgot to update the array in redis-check rdb.
2024-01-23 11:48:02 +02:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
12fd752443 Bump actions/cache from 3 to 4 (#12978)
Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/cache/releases">actions/cache's
releases</a>.</em></p>
<blockquote>
<h2>v4.0.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Update action to node20 by <a
href="https://github.com/takost"><code>@​takost</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1284">actions/cache#1284</a></li>
<li>feat: save-always flag by <a
href="https://github.com/to-s"><code>@​to-s</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1242">actions/cache#1242</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/takost"><code>@​takost</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1284">actions/cache#1284</a></li>
<li><a href="https://github.com/to-s"><code>@​to-s</code></a> made their
first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1242">actions/cache#1242</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/cache/compare/v3...v4.0.0">https://github.com/actions/cache/compare/v3...v4.0.0</a></p>
<h2>v3.3.3</h2>
<h2>What's Changed</h2>
<ul>
<li>Cache v3.3.3 by <a
href="https://github.com/robherley"><code>@​robherley</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1302">actions/cache#1302</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/robherley"><code>@​robherley</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1302">actions/cache#1302</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/cache/compare/v3...v3.3.3">https://github.com/actions/cache/compare/v3...v3.3.3</a></p>
<h2>v3.3.2</h2>
<h2>What's Changed</h2>
<ul>
<li>Fixed readme with new segment timeout values by <a
href="https://github.com/kotewar"><code>@​kotewar</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1133">actions/cache#1133</a></li>
<li>Readme fixes by <a
href="https://github.com/kotewar"><code>@​kotewar</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1134">actions/cache#1134</a></li>
<li>Updated description of the lookup-only input for main action by <a
href="https://github.com/kotewar"><code>@​kotewar</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1130">actions/cache#1130</a></li>
<li>Change two new actions mention as quoted text by <a
href="https://github.com/bishal-pdMSFT"><code>@​bishal-pdMSFT</code></a>
in <a
href="https://redirect.github.com/actions/cache/pull/1131">actions/cache#1131</a></li>
<li>Update Cross-OS Caching tips by <a
href="https://github.com/pdotl"><code>@​pdotl</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1122">actions/cache#1122</a></li>
<li>Bazel example (Take <a
href="https://redirect.github.com/actions/cache/issues/2">#2</a>️⃣) by
<a href="https://github.com/vorburger"><code>@​vorburger</code></a> in
<a
href="https://redirect.github.com/actions/cache/pull/1132">actions/cache#1132</a></li>
<li>Remove actions to add new PRs and issues to a project board by <a
href="https://github.com/jorendorff"><code>@​jorendorff</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1187">actions/cache#1187</a></li>
<li>Consume latest toolkit and fix dangling promise bug by <a
href="https://github.com/chkimes"><code>@​chkimes</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1217">actions/cache#1217</a></li>
<li>Bump action version to 3.3.2 by <a
href="https://github.com/bethanyj28"><code>@​bethanyj28</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1236">actions/cache#1236</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/vorburger"><code>@​vorburger</code></a>
made their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1132">actions/cache#1132</a></li>
<li><a
href="https://github.com/jorendorff"><code>@​jorendorff</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1187">actions/cache#1187</a></li>
<li><a href="https://github.com/chkimes"><code>@​chkimes</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1217">actions/cache#1217</a></li>
<li><a
href="https://github.com/bethanyj28"><code>@​bethanyj28</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/cache/pull/1236">actions/cache#1236</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/cache/compare/v3...v3.3.2">https://github.com/actions/cache/compare/v3...v3.3.2</a></p>
<h2>v3.3.1</h2>
<h2>What's Changed</h2>
<ul>
<li>Reduced download segment size to 128 MB and timeout to 10 minutes by
<a href="https://github.com/kotewar"><code>@​kotewar</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1129">actions/cache#1129</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/cache/compare/v3...v3.3.1">https://github.com/actions/cache/compare/v3...v3.3.1</a></p>
<h2>v3.3.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Bug: Permission is missing in cache delete example by <a
href="https://github.com/kotokaze"><code>@​kotokaze</code></a> in <a
href="https://redirect.github.com/actions/cache/pull/1123">actions/cache#1123</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/actions/cache/blob/main/RELEASES.md">actions/cache's
changelog</a>.</em></p>
<blockquote>
<h1>Releases</h1>
<h3>3.0.0</h3>
<ul>
<li>Updated minimum runner version support from node 12 -&gt; node
16</li>
</ul>
<h3>3.0.1</h3>
<ul>
<li>Added support for caching from GHES 3.5.</li>
<li>Fixed download issue for files &gt; 2GB during restore.</li>
</ul>
<h3>3.0.2</h3>
<ul>
<li>Added support for dynamic cache size cap on GHES.</li>
</ul>
<h3>3.0.3</h3>
<ul>
<li>Fixed avoiding empty cache save when no files are available for
caching. (<a
href="https://redirect.github.com/actions/cache/issues/624">issue</a>)</li>
</ul>
<h3>3.0.4</h3>
<ul>
<li>Fixed tar creation error while trying to create tar with path as
<code>~/</code> home folder on <code>ubuntu-latest</code>. (<a
href="https://redirect.github.com/actions/cache/issues/689">issue</a>)</li>
</ul>
<h3>3.0.5</h3>
<ul>
<li>Removed error handling by consuming actions/cache 3.0 toolkit, Now
cache server error handling will be done by toolkit. (<a
href="https://redirect.github.com/actions/cache/pull/834">PR</a>)</li>
</ul>
<h3>3.0.6</h3>
<ul>
<li>Fixed <a
href="https://redirect.github.com/actions/cache/issues/809">#809</a> -
zstd -d: no such file or directory error</li>
<li>Fixed <a
href="https://redirect.github.com/actions/cache/issues/833">#833</a> -
cache doesn't work with github workspace directory</li>
</ul>
<h3>3.0.7</h3>
<ul>
<li>Fixed <a
href="https://redirect.github.com/actions/cache/issues/810">#810</a> -
download stuck issue. A new timeout is introduced in the download
process to abort the download if it gets stuck and doesn't finish within
an hour.</li>
</ul>
<h3>3.0.8</h3>
<ul>
<li>Fix zstd not working for windows on gnu tar in issues <a
href="https://redirect.github.com/actions/cache/issues/888">#888</a> and
<a
href="https://redirect.github.com/actions/cache/issues/891">#891</a>.</li>
<li>Allowing users to provide a custom timeout as input for aborting
download of a cache segment using an environment variable
<code>SEGMENT_DOWNLOAD_TIMEOUT_MINS</code>. Default is 60 minutes.</li>
</ul>
<h3>3.0.9</h3>
<ul>
<li>Enhanced the warning message for cache unavailablity in case of
GHES.</li>
</ul>
<h3>3.0.10</h3>
<ul>
<li>Fix a bug with sorting inputs.</li>
<li>Update definition for restore-keys in README.md</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/actions/cache/commit/13aacd865c20de90d75de3b17ebe84f7a17d57d2"><code>13aacd8</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/cache/issues/1242">#1242</a>
from to-s/main</li>
<li><a
href="https://github.com/actions/cache/commit/53b35c543921fe2e8b288765ff817de9de8d906f"><code>53b35c5</code></a>
Merge branch 'main' into main</li>
<li><a
href="https://github.com/actions/cache/commit/65b8989fab3bb394817bdb845a453dff480c2b51"><code>65b8989</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/cache/issues/1284">#1284</a>
from takost/update-to-node-20</li>
<li><a
href="https://github.com/actions/cache/commit/d0be34d54485f31ca2ccbe66e6ea3d96544a807b"><code>d0be34d</code></a>
Fix dist</li>
<li><a
href="https://github.com/actions/cache/commit/66cf064d47313d2cccf392d01bd10925da2bd072"><code>66cf064</code></a>
Merge branch 'main' into update-to-node-20</li>
<li><a
href="https://github.com/actions/cache/commit/1326563738ddb735c5f2ce85cba8c79f33b728cd"><code>1326563</code></a>
Merge branch 'main' into main</li>
<li><a
href="https://github.com/actions/cache/commit/e71876755e268d6cc25a5d3e3c46ae447e35290a"><code>e718767</code></a>
Fix format</li>
<li><a
href="https://github.com/actions/cache/commit/01229828ffa049a8dee4db27bcb23ed33f2b451f"><code>0122982</code></a>
Apply workaround for earlyExit</li>
<li><a
href="https://github.com/actions/cache/commit/3185ecfd6135856ca6d904ae032cff4f39b8b365"><code>3185ecf</code></a>
Update &quot;only-&quot; actions to node20</li>
<li><a
href="https://github.com/actions/cache/commit/25618a0a675e8447e5ffc8ed9b7ddb2aaf927f65"><code>25618a0</code></a>
Bump version</li>
<li>Additional commits viewable in <a
href="https://github.com/actions/cache/compare/v3...v4">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/cache&package-manager=github_actions&previous-version=3&new-version=4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-23 11:09:49 +02:00
Harkrishn PatroandGitHub 2bce71b5ff Exit early if slowlog/acllog max len set to zero (#12965)
Currently slowlog gets disabled if slowlog-log-slower-than is set to less than zero. I think we should also disable it if slowlog-max-len is set to zero. We apply the same logic to acllog-max-len.
2024-01-22 16:01:04 -08:00
e12f2decc1 Prevent nodes with invalid IDs from being propagated through gossip (#12921)
There have been occasional instances of memory corruption (though code bugs or bit flips) leading to invalid node information being gossiped around. To prevent this invalid information spreading, we verify the node IDs in received gossip are in an acceptable format, and disregard any gossiped nodes with invalid IDs. This PR uses the existing verifyClusterNodeId function to check the validity of the gossiped node IDs and if an invalid one is encountered, logs raw byte information to help debug the corruption.

---------

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2024-01-22 11:25:43 -08:00
zhaozhao.zzandGitHub 8d0156eb18 Set the correct id for tempDb (#12947)
background: some modules need to know the `dbid` information, such as
the function used during RDB loading:

```
robj *rdbLoadObject(int rdbtype, rio *rdb, sds key, int dbid, int *error) {
....
        moduleInitIOContext(io,mt,rdb,&keyobj,dbid);
```

However, during replication, the "tempDb" created for diskless RDB
loading is not correctly set with the dbid. This leads to passing the
wrong dbid to the `rdbLoadObject` function (as tempDb uses zcalloc, all
ids are 0).

```
disklessLoadInitTempDb()->rdbLoadRioWithLoadingCtx()->
        /* Read value */
        val = rdbLoadObject(type,rdb,key,db->id,&error);
```

To fix it, set the correct ID (relative index) for the tempdb.
2024-01-22 11:47:51 +08:00
Yanqi LvandGitHub 85a239b363 Change dictGetSafeIterator to dictGetIterator in pubsub (#12931)
In #12838, we misuse the safe iterator of the client dict, so we can't
catch the synchronous release of the client if there is a bug.

Since we realize that clients (even subscribers) are released with async
free, we change the safe iterators of the client dict into unsafe
iterators in `pubsub.c`. And I also remove redundant code.
2024-01-19 17:03:20 +02:00
Yanqi LvandGitHub b07174afc2 Change the threshold of dict expand, shrink and rehash (#12948)
Before this change (most recently modified in
https://github.com/redis/redis/pull/12850#discussion_r1421406393), The
trigger for normal expand threshold was 100% utilization and the trigger
for normal shrink threshold was 10% (HASHTABLE_MIN_FILL).
While during fork (DICT_RESIZE_AVOID), when we want to avoid rehash, the
trigger thresholds were multiplied by 5 (`dict_force_resize_ratio`),
meaning 500% for expand and 2% (100/10/5) for shrink.

However, in `dictRehash` (the incremental rehashing), the rehashing
threshold for shrinking during fork (DICT_RESIZE_AVOID) was 20% by
mistake.
This meant that if a shrinking is triggered when `dict_can_resize` is
`DICT_RESIZE_ENABLE` which the threshold is 10%, the rehashing can
continue when `dict_can_resize` is `DICT_RESIZE_AVOID`.
This would cause unwanted CopyOnWrite damage.

It'll make sense to change the thresholds of the rehash trigger and the
thresholds of the incremental rehashing the same, however, in one we
compare the size of the hash table to the number of records, and in the
other we compare the size of ht[0] to the size of ht[1], so the formula
is not exactly the same.

to make things easier we change all the thresholds to powers of 2, so
the normal shrinking threshold is changed from 100/10 (i.e. 10%) to
100/8 (i.e. 12.5%), and we change the threshold during forks from 5 to
4, i.e. from 500% to 400% for expand, and from 2% (100/10/5) to 3.125%
(100/8/4)
2024-01-19 17:00:43 +02:00
d0640029dc Fix race condition issues between the main thread and module threads (#12817)
Fix #12785 and other race condition issues.
See the following isolated comments.

The following report was obtained using SANITIZER thread.
```sh
make SANITIZER=thread
./runtest-moduleapi --config io-threads 4 --config io-threads-do-reads yes --accurate
```

1. Fixed thread-safe issue in RM_UnblockClient()
Related discussion:
https://github.com/redis/redis/pull/12817#issuecomment-1831181220
* When blocking a client in a module using `RM_BlockClientOnKeys()` or
`RM_BlockClientOnKeysWithFlags()`
with a timeout_callback, calling RM_UnblockClient() in module threads
can lead to race conditions
     in `updateStatsOnUnblock()`.

     - Introduced: 
        Version: 6.2
        PR: #7491

     - Touch:
`server.stat_numcommands`, `cmd->latency_histogram`, `server.slowlog`,
and `server.latency_events`
     
     - Harm Level: High
Potentially corrupts the memory data of `cmd->latency_histogram`,
`server.slowlog`, and `server.latency_events`

     - Solution:
Differentiate whether the call to moduleBlockedClientTimedOut() comes
from the module or the main thread.
Since we can't know if RM_UnblockClient() comes from module threads, we
always assume it does and
let `updateStatsOnUnblock()` asynchronously update the unblock status.
     
* When error reply is called in timeout_callback(), ctx is not
thread-safe, eventually lead to race conditions in `afterErrorReply`.

     - Introduced: 
        Version: 6.2
        PR: #8217

     - Touch
       `server.stat_total_error_replies`, `server.errors`, 

     - Harm Level: High
       Potentially corrupts the memory data of `server.errors`
   
      - Solution: 
Make the ctx in `timeout_callback()` with `REDISMODULE_CTX_THREAD_SAFE`,
and asynchronously reply errors to the client.

2. Made RM_Reply*() family API thread-safe
Related discussion:
https://github.com/redis/redis/pull/12817#discussion_r1408707239
Call chain: `RM_Reply*()` -> `_addReplyToBufferOrList()` -> touch
server.current_client

    - Introduced: 
       Version: 7.2.0
       PR: #12326

   - Harm Level: None
Since the module fake client won't have the `CLIENT_PUSHING` flag, even
if we touch server.current_client,
     we can still exit after `c->flags & CLIENT_PUSHING`.

   - Solution
      Checking `c->flags & CLIENT_PUSHING` earlier.

3. Made freeClient() thread-safe
    Fix #12785

    - Introduced: 
       Version: 4.0
Commit:
https://github.com/redis/redis/commit/3fcf959e609e850a114d4016843e4c991066ebac

    - Harm Level: Moderate
       * Trigger assertion
It happens when the module thread calls freeClient while the io-thread
is in progress,
which just triggers an assertion, and doesn't make any race condiaions.

* Touch `server.current_client`, `server.stat_clients_type_memory`, and
`clientMemUsageBucket->clients`.
It happens between the main thread and the module threads, may cause
data corruption.
1. Error reset `server.current_client` to NULL, but theoretically this
won't happen,
because the module has already reset `server.current_client` to old
value before entering freeClient.
2. corrupts `clientMemUsageBucket->clients` in
updateClientMemUsageAndBucket().
3. Causes server.stat_clients_type_memory memory statistics to be
inaccurate.
    
    - Solution:
* No longer counts memory usage on fake clients, to avoid updating
`server.stat_clients_type_memory` in freeClient.
* No longer resetting `server.current_client` in unlinkClient, because
the fake client won't be evicted or disconnected in the mid of the
process.
* Judgment assertion `io_threads_op == IO_THREADS_OP_IDLE` only if c is
not a fake client.

4. Fixed free client args without GIL
Related discussion:
https://github.com/redis/redis/pull/12817#discussion_r1408706695
When freeing retained strings in the module thread (refcount decr), or
using them in some way (refcount incr), we should do so while holding
the GIL,
otherwise, they might be simultaneously freed while the main thread is
processing the unblock client state.

    - Introduced: 
       Version: 6.2.0
       PR: #8141

   - Harm Level: Low
     Trigger assertion or double free or memory leak. 

   - Solution:
Documenting that module API users need to ensure any access to these
retained strings is done with the GIL locked

5. Fix adding fake client to server.clients_pending_write
    It will incorrectly log the memory usage for the fake client.
Related discussion:
https://github.com/redis/redis/pull/12817#issuecomment-1851899163

    - Introduced: 
       Version: 4.0
Commit:
https://github.com/redis/redis/commit/9b01b64430fbc1487429144d2e4e72a4a7fd9db2

    - Harm Level: None
      Only result in NOP

    - Solution:
       * Don't add fake client into server.clients_pending_write
* Add c->conn assertion for updateClientMemUsageAndBucket() and
updateClientMemoryUsage() to avoid same
         issue in the future.
So now it will be the responsibility of the caller of both of them to
avoid passing in fake client.

6. Fix calling RM_BlockedClientMeasureTimeStart() and
RM_BlockedClientMeasureTimeEnd() without GIL
    - Introduced: 
       Version: 6.2
       PR: #7491

   - Harm Level: Low
Causes inaccuracies in command latency histogram and slow logs, but does
not corrupt memory.

   - Solution:
Module API users, if know that non-thread-safe APIs will be used in
multi-threading, need to take responsibility for protecting them with
their own locks instead of the GIL, as using the GIL is too expensive.

### Other issue
1. RM_Yield is not thread-safe, fixed via #12905.

### Summarize
1. Fix thread-safe issues for `RM_UnblockClient()`, `freeClient()` and
`RM_Yield`, potentially preventing memory corruption, data disorder, or
assertion.
2. Updated docs and module test to clarify module API users'
responsibility for locking non-thread-safe APIs in multi-threading, such
as RM_BlockedClientMeasureTimeStart/End(), RM_FreeString(),
RM_RetainString(), and RM_HoldString().

### About backpot to 7.2
1. The implement of (1) is not too satisfying, would like to get more
eyes.
2. (2), (3) can be safely for backport
3. (4), (6) just modifying the module tests and updating the
documentation, no need for a backpot.
4. (5) is harmless, no need for a backpot.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2024-01-19 15:12:49 +02:00
Chen TianjieandGitHub f81c3fd89e Optimize dictTypeResizeAllowed to avoid mistaken OOM judgement. (#12950)
When doing dict resizing, dictTypeResizeAllowed is used to judge whether
the new allocated memory for rehashing would cause OOM.

However when shrinking, we alloc `_dictNextExp(d->ht_used[0])` bytes of
memory, while in `dictTypeResizeAllowed` we still use
`_dictNextExp(d->ht_used[0]+1)` as the new allocated memory size. This
will overestimate the memory used by shrinking at special conditions,
causing a false OOM judgement.
2024-01-18 16:35:12 +02:00
BinbinandGitHub 1c7eb0ad37 Fix minor memory leaks in dictTest (#12962)
Introduced in #12952, reported by valgrind.
2024-01-18 16:32:04 +02:00
BinbinandGitHub 0e5a4a27ea Call emptyData when disk-based sync rdbLoad fails (#12510)
We doing this in diskless on-empty-db mode, when diskless
loading fails, we will call emptyData to remove the half-loaded
data in case we started with an empty replica.

Now when a disk-based sync rdbLoad fails, we will call emptyData
too in case it loads partially incomplete data.

when the replica attempts another re-sync, it'll empty the dataset
again anyway, so this affects two things:
1. memory consumption in the time gap until the next rdb loading begins
2. if the unsynced replica is for some reason promoted, it would have kept
  the partial dataset instead of being empty.
2024-01-18 16:28:52 +02:00
BinbinandGitHub 29e6245a05 Fix unexpected resize causing test failure (#12960)
Before #12850, we will only try to shrink the dict in serverCron,
which we can control by using a child process, but now every time
we delete a key, the shrink check will be called.

In these test (added in #12802), we meant to disable the resizing,
but druing the delete, the dict will meet the force shrink, like
2 / 128 = 0.015 < 0.2, the delete will trigger a force resize and
will cause the test to fail.

In this commit, we try to keep the load factor at 3 / 128 = 0.023,
that is, do not meet the force shrink.
2024-01-18 11:19:29 +02:00
BinbinandGitHub 14b1edfd99 Fix dict resize ratio checks, avoid precision loss from integer division (#12952)
In the past we used integers to compare ratios, let us assume that
we have the following data in expanding:
```
used / size > 5
`80 / 16 > 5` is false
`81 / 16 > 5` is false
`95 / 16 > 5` is false
`96 / 16 > 5` is true
```

Because the integer result is rounded, our resize breaks the ratio
constraint, this has existed since the beginning, which resulted in
us not strictly following the ratio (shrink also has the same issue).

This PR change it to multiplication to avoid floating point
calculations.
2024-01-18 11:16:50 +02:00
BinbinandGitHub 131d95f203 Fix race in slot dict resize test (#12942)
The test have a race:
```
*** [err]: Redis can rewind and trigger smaller slot resizing in tests/unit/other.tcl
Expected '[Dictionary HT]
Hash table 0 stats (main hash table):
 table size: 12
 number of elements: 2
[Expires HT]
Hash table 0 stats (main hash table):
No stats available for empty dictionaries
' to match '*table size: 8*' (context: type eval line 12 cmd {assert_match "*table size: 8*" [r debug HTSTATS 0]} proc ::test)
```

When `r del "{alice}$j"` is executed in the loop, when the key is
deleted to [9, 12], the load factor has meet HASHTABLE_MIN_FILL,
if serverCron happens to trigger slot dict resize, then the test
will fail. Because there is not way to meet HASHTABLE_MIN_FILL in
the subsequent dels.

The solution is to avoid triggering the resize in advance. We can
use multi to delete them at once, or we can disable the resize.
Since we disabled resize in the previous test, the fix also uses
the method of disabling resize.

The test is introduced in #12802.
2024-01-17 08:46:09 +02:00
BinbinandGitHub ecc31bc697 Updated comments on dictResizeEnable for new dict shrink (#12946)
The new shrink was added in #12850.
Also updated outdated comments, see #11692.
2024-01-15 10:28:24 +02:00
e2b7932b34 Shrink dict when deleting dictEntry (#12850)
When we insert entries into dict, it may autonomously expand if needed.
However, when we delete entries from dict, it doesn't shrink to the
proper size. If there are few entries in a very large dict, it may cause
huge waste of memory and inefficiency when iterating.

The main keyspace dicts (keys and expires), are shrinked by cron
(`tryResizeHashTables` calls `htNeedsResize` and `dictResize`),
And some data structures such as zset and hash also do that (call
`htNeedsResize`) right after a loop of calls to `dictDelete`,
But many other dicts are completely missing that call (they can only
expand).

In this PR, we provide the ability to automatically shrink the dict when
deleting. The conditions triggering the shrinking is the same as
`htNeedsResize` used to have. i.e. we expand when we're over 100%
utilization, and shrink when we're below 10% utilization.

Additionally:
* Add `dictPauseAutoResize` so that flows that do mass deletions, will
only trigger shrinkage at the end.
* Rename `dictResize` to `dictShrinkToFit` (same logic as it used to
have, but better name describing it)
* Rename `_dictExpand` to `_dictResize` (same logic as it used to have,
but better name describing it)
 
related to discussion
https://github.com/redis/redis/pull/12819#discussion_r1409293878

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
2024-01-15 08:20:53 +02:00
zhaozhao.zzandGitHub bb2b6e2927 fix scripts access wrong slot if they disagree with pre-declared keys (#12906)
Regarding how to obtain the hash slot of a key, there is an optimization
in `getKeySlot()`, it is used to avoid redundant hash calculations for
keys: when the current client is in the process of executing a command,
it can directly use the slot of the current client because the slot to
access has already been calculated in advance in `processCommand()`.

However, scripts are a special case where, in default mode or with
`allow-cross-slot-keys` enabled, they are allowed to access keys beyond
the pre-declared range. This means that the keys they operate on may not
belong to the slot of the pre-declared keys. Currently, when the
commands in a script are executed, the slot of the original client
(i.e., the current client) is not correctly updated, leading to
subsequent access to the wrong slot.

This PR fixes the above issue. When checking the cluster constraints in
a script, the slot to be accessed by the current command is set for the
original client (i.e., the current client). This ensures that
`getKeySlot()` gets the correct slot cache.

Additionally, the following modifications are made:

1. The 'sort' and 'sort_ro' commands use `getKeySlot()` instead of
`c->slot` because the client could be an engine client in a script and
can lead to potential bug.
2. `getKeySlot()` is also used in pubsub to obtain the slot for the
channel, standardizing the way slots are retrieved.
2024-01-15 09:57:12 +08:00
BinbinandGitHub 284ef21ea0 Fix fd check in memtest_test_linux_anonymous_maps (#12943)
The open function returns a fd on success or -1 on failure,
here we should check fd != -1, otherwise -1 will be judged
as success.

This closes #12938.
2024-01-14 11:18:17 +02:00
Chen TianjieandGitHub 87786342a5 Correct bytes_per_key computing. (#12897)
Change the calculation method of bytes_per_key to make it closer to
the true average key size. The calculation method is as follows:

mh->bytes_per_key = mh->total_keys ? (mh->dataset / mh->total_keys) : 0;
2024-01-12 11:58:53 +08:00
Harkrishn PatroandGitHub 964f4a4576 Avoid double free of cluster link (#12930)
Avoid crash while performing `DEBUG CLUSTERLINK KILL` mutliple times
(cluster link might not be created/valid).
2024-01-11 15:59:22 -08:00
bentottenandGitHub b3aaa0a136 When one shard, sole primary node marks potentially failed replica as FAIL instead of PFAIL (#12824)
Fixes issue where a single primary cannot mark a replica as failed in a
single-shard cluster.
2024-01-11 15:48:19 -08:00
BinbinandGitHub b351a04b1e Add announced-endpoints test to all_tests and fix tls related tests (#12927)
The test was introduced in #10745, but we forgot to add it to the
test_helper.tcl, so our CI did not actually run it. This PR adds it
and ensures it passes CI tests.
2024-01-09 18:18:59 -08:00
Oran AgraandGitHub f7b1d0287d Fix possible corruption in sdsResize (CVE-2023-41056) (#12924)
#11766 introduced a bug in sdsResize where it could forget to update the
sds type in the sds header and then cause an overflow in sdsalloc. it
looks like the only implication of that is a possible assertion in HLL,
but it's hard to rule out possible heap corruption issues with
clientsCronResizeQueryBuffer
2024-01-09 13:51:56 +02:00
Madelyn OlsonandGitHub 8bb9a2895e Address some failures with new tests for improving debug report (#12915)
Fix a daily test failure because alpine doesn't support stack traces and
add in an extra assertion related to making sure the stack trace was
printed twice.
2024-01-08 17:56:06 -08:00
BinbinandGitHub 14e4a9835a Fix minor fd leak in rdbSaveToSlavesSockets (#12919)
We should close server.rdb_child_exit_pipe when redisFork fails,
otherwise the pipe fd will be leaked.

Just a cleanup.
2024-01-08 17:36:34 +02:00
Andy PanandGitHub 50b8b99763 Re-indent code and reduce code being complied on Solaris for anetKeepAlive (#12914)
This is a follow-up PR for #12782, in which we introduced nested
preprocessor directives for TCP keep-alive on Solaris and added
redundant indentation for code. Besides, it could result in unreachable
code due to the lack of `#else` on the latest Solaris 11.4 where
`TCP_KEEPIDLE`, `TCP_KEEPINTVL`, and `TCP_KEEPCNT` are available. As a
result, this PR does three main things:

- To eliminate the redundant indention for C code in nested preprocessor
directives
- To add `#else` directives and move `TCP_KEEPALIVE_THRESHOLD` +
`TCP_KEEPALIVE_ABORT_THRESHOLD` settings under it, avoid unreachable
code and compiler warnings when `#if defined(TCP_KEEPIDLE) &&
defined(TCP_KEEPINTVL) && defined(TCP_KEEPCNT)` is met on Solaris 11.4
- To remove a few trailing whitespace in comments
2024-01-08 11:12:24 +02:00
Yanqi LvandGitHub c452e414a8 Optimize performance when many clients [p|s]unsubscribe simultaneously (#12838)
I'm testing the performance of Pub/Sub command recently. I find if many
clients unsubscribe or are killed simultaneously, Redis needs a long
time to deal with it.

In my experiment, I set 5000 clients and each client subscribes 100
channels. Then I call `client kill type pubsub` to simulate the
situation where clients unsubscribe all channels at the same time and
calculate the execution time. The result shows that it takes about 23s.
I use the _perf_ and find that `listSearchKey` in
`pubsubUnsubscribeChannel` costs more than 90% cpu time. I think we can
optimize this situation.

In this PR, I replace list with dict to track the clients subscribing
the channel more efficiently. It changes O(N) to O(1) in the search
phase. Then I repeat the experiment as above. The results are as
follows.

|              | Execution Time(s) |used_memory(MB) |
| :---------------- | :------: | :----: |
| unstable(1bd0b54)        |   23.734   | 65.41 |
| optimize-pubsub           |   0.288   | 67.66 |

Thanks for #11595 , I use a no-value dict and the results shows that the
performance improves significantly but the memory usage only increases
slightly.

Notice:

- This PR will cause the performance degradation about 20% in
`[p|s]subscribe` command but won't freeze Redis.
2024-01-08 10:32:31 +02:00
debing.sunandGitHub 4730563e93 Change destination key's key-spec flag from RW to OW for SINTERSTORE command (#12917)
In #10122, we set the destination key's flag of SINTERSTORE to `RW`, 
however, this command doesn't actually read or modify the destination
key, just overwrites it.
Therefore, we change it to `OW` similarly to all other *STORE commands.
2024-01-08 10:17:13 +02:00
BinbinandGitHub 5b0c6a8255 Fix CLUSTER SHARDS crash in 7.0/7.2 mixed clusters where shard ids are not sync (#12832)
Crash reported in #12695. In the process of upgrading the cluster from
7.0 to 7.2, because the 7.0 nodes will not gossip shard id, in 7.2 we
will rely on shard id to build the server.cluster->shards dict.

In some cases, for example, the 7.0 master node and the 7.2 replica node.
From the view of 7.2 replica node, the cluster->shards dictionary does not
have its master node. In this case calling CLUSTER SHARDS on the 7.2 replica
node may crash.

We should fix the underlying assumption of updateShardId, which is that the
shard dict should be always in sync with the node's shard_id. The fix was
suggested by PingXie, see more details in #12695.
2024-01-07 20:54:41 -08:00
ca1f67af80 Make RM_Yield thread-safe (#12905)
## Issues and solutions from #12817
1. Touch ProcessingEventsWhileBlocked and calling moduleCount() without
GIL in afterSleep()
    - Introduced: 
       Version: 7.0.0
       PR: #9963

   - Harm Level: Very High
If the module thread calls `RM_Yield()` before the main thread enters
afterSleep(),
and modifies `ProcessingEventsWhileBlocked`(+1), it will cause the main
thread to not wait for GIL,
which can lead to all kinds of unforeseen problems, including memory
data corruption.

   - Initial / Abandoned Solution:
      * Added `__thread` specifier for ProcessingEventsWhileBlocked.
`ProcessingEventsWhileBlocked` is used to protect against nested event
processing, but event processing
in the main thread and module threads should be completely independent
and unaffected, so it is safer
         to use TLS.
* Adding a cached module count to keep track of the current number of
modules, to avoid having to use `dictSize()`.
    
    - Related Warnings:
```
WARNING: ThreadSanitizer: data race (pid=1136)
  Write of size 4 at 0x0001045990c0 by thread T4 (mutexes: write M0):
    #0 processEventsWhileBlocked networking.c:4135 (redis-server:arm64+0x10006d124)
    #1 RM_Yield module.c:2410 (redis-server:arm64+0x10018b66c)
    #2 bg_call_worker <null>:83232836 (blockedclient.so:arm64+0x16a8)

  Previous read of size 4 at 0x0001045990c0 by main thread:
    #0 afterSleep server.c:1861 (redis-server:arm64+0x100024f98)
    #1 aeProcessEvents ae.c:408 (redis-server:arm64+0x10000fd64)
    #2 aeMain ae.c:496 (redis-server:arm64+0x100010f0c)
    #3 main server.c:7220 (redis-server:arm64+0x10003f38c)
```

2. aeApiPoll() is not thread-safe
When using RM_Yield to handle events in a module thread, if the main
thread has not yet
entered `afterSleep()`, both the module thread and the main thread may
touch `server.el` at the same time.

    - Introduced: 
       Version: 7.0.0
       PR: #9963

   - Old / Abandoned Solution:
Adding a new mutex to protect timing between after beforeSleep() and
before afterSleep().
Defect: If the main thread enters the ae loop without any IO events, it
will wait until
the next timeout or until there is any event again, and the module
thread will
always hang until the main thread leaves the event loop.

    - Related Warnings:
```
SUMMARY: ThreadSanitizer: data race ae_kqueue.c:55 in addEventMask
==================
==================
WARNING: ThreadSanitizer: data race (pid=14682)
  Write of size 4 at 0x000100b54000 by thread T9 (mutexes: write M0):
    #0 aeApiPoll ae_kqueue.c:175 (redis-server:arm64+0x100010588)
    #1 aeProcessEvents ae.c:399 (redis-server:arm64+0x10000fb84)
    #2 processEventsWhileBlocked networking.c:4138 (redis-server:arm64+0x10006d3c4)
    #3 RM_Yield module.c:2410 (redis-server:arm64+0x10018b66c)
    #4 bg_call_worker <null>:16042052 (blockedclient.so:arm64+0x169c)

  Previous write of size 4 at 0x000100b54000 by main thread:
    #0 aeApiPoll ae_kqueue.c:175 (redis-server:arm64+0x100010588)
    #1 aeProcessEvents ae.c:399 (redis-server:arm64+0x10000fb84)
    #2 aeMain ae.c:496 (redis-server:arm64+0x100010da8)
    #3 main server.c:7238 (redis-server:arm64+0x10003f51c)
```

## The final fix as the comments:
https://github.com/redis/redis/pull/12817#discussion_r1436427232
Optimized solution based on the above comment:

First, we add `module_gil_acquring` to indicate whether the main thread
is currently in the acquiring GIL state.

When the module thread starts to yield, there are two possibilities(we
assume the caller keeps the GIL):
1. The main thread is in the mid of beforeSleep() and afterSleep(), that
is, `module_gil_acquring` is not 1 now.
At this point, the module thread will wake up the main thread through
the pipe and leave the yield,
waiting for the next yield when the main thread may already in the
acquiring GIL state.
    
2. The main thread is in the acquiring GIL state.
The module thread release the GIL, yielding CPU to give the main thread
an opportunity to start
event processing, and then acquire the GIL again until the main thread
releases it.
This is what
https://github.com/redis/redis/pull/12817#discussion_r1436427232
mentioned direction.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2024-01-07 12:10:29 +02:00
BinbinandGitHub 4cae66f5e8 Use shard-id of the master if the replica does not support shard-id (#12805)
If there are nodes in the cluster that do not support shard-id, they
will gossip shard-id. From the perspective of nodes that support shard-id,
their shard-id is meaningless (since shard-id is randomly generated when
we create a node.)

Nodes that support shard-id will save the shard-id information in nodes.conf.
If the node is restarted according to nodes.conf, the server will report a
corrupted cluster config file error. Because auxShardIdSetter will reject
configurations with inconsistent master-replica shard-ids.

A cluster-wide consensus for the node's shard_id is not necessary. The key
is maintaining consistency of the shard_id on each individual 7.2 node.
As the cluster progressively upgrades to version 7.2, we can expect the
shard_ids across all nodes to naturally converge and align.

In this PR, when processing the gossip, if sender is a replica and does not
support shard-id, set the shard_id to the shard_id of its master.
2024-01-06 20:24:41 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
38f0234946 Bump cross-platform-actions/action from 0.21.1 to 0.22.0 (#12904)
Bumps
[cross-platform-actions/action](https://github.com/cross-platform-actions/action)
from 0.21.1 to 0.22.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/cross-platform-actions/action/releases">cross-platform-actions/action's
releases</a>.</em></p>
<blockquote>
<h2>Cross Platform Action 0.22.0</h2>
<h3>Added</h3>
<ul>
<li>
<p>Added support for using the action in multiple steps in the same job
(<a
href="https://redirect.github.com/cross-platform-actions/action/issues/26">#26</a>).
All the inputs need to be the same for all steps, except for the
following
inputs: <code>sync_files</code>, <code>shutdown_vm</code> and
<code>run</code>.</p>
</li>
<li>
<p>Added support for specifying that the VM should not shutdown after
the action
has run. This adds a new input parameter: <code>shutdown_vm</code>. When
set to <code>false</code>,
this will hopefully mitigate very frequent freezing of VM during
teardown (<a
href="https://redirect.github.com/cross-platform-actions/action/issues/61">#61</a>,
<a
href="https://redirect.github.com/cross-platform-actions/action/issues/72">#72</a>).</p>
</li>
</ul>
<h3>Changed</h3>
<ul>
<li>
<p>Always terminate VM instead of shutting down. This is more efficient
and this
will hopefully mitigate very frequent freezing of VM during teardown
(<a
href="https://redirect.github.com/cross-platform-actions/action/issues/61">#61</a>,
<a
href="https://redirect.github.com/cross-platform-actions/action/issues/72">#72</a>).</p>
</li>
<li>
<p>Use <code>unsafe</code> as the cache mode for QEMU disks. This should
improve performance (<a
href="https://redirect.github.com/cross-platform-actions/action/issues/67">#67</a>).</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/cross-platform-actions/action/blob/master/changelog.md">cross-platform-actions/action's
changelog</a>.</em></p>
<blockquote>
<h2>[0.22.0] - 2023-12-27</h2>
<h3>Added</h3>
<ul>
<li>
<p>Added support for using the action in multiple steps in the same job
(<a
href="https://redirect.github.com/cross-platform-actions/action/issues/26">#26</a>).
All the inputs need to be the same for all steps, except for the
following
inputs: <code>sync_files</code>, <code>shutdown_vm</code> and
<code>run</code>.</p>
</li>
<li>
<p>Added support for specifying that the VM should not shutdown after
the action
has run. This adds a new input parameter: <code>shutdown_vm</code>. When
set to <code>false</code>,
this will hopefully mitigate very frequent freezing of VM during
teardown (<a
href="https://redirect.github.com/cross-platform-actions/action/issues/61">#61</a>,
<a
href="https://redirect.github.com/cross-platform-actions/action/issues/72">#72</a>).</p>
</li>
</ul>
<h3>Changed</h3>
<ul>
<li>
<p>Always terminate VM instead of shutting down. This is more efficient
and this
will hopefully mitigate very frequent freezing of VM during teardown
(<a
href="https://redirect.github.com/cross-platform-actions/action/issues/61">#61</a>,
<a
href="https://redirect.github.com/cross-platform-actions/action/issues/72">#72</a>).</p>
</li>
<li>
<p>Use <code>unsafe</code> as the cache mode for QEMU disks. This should
improve performance (<a
href="https://redirect.github.com/cross-platform-actions/action/issues/67">#67</a>).</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/cross-platform-actions/action/commit/5800fa0060a22edf69992a779adac3d2bb3a6f8a"><code>5800fa0</code></a>
Release 0.22.0</li>
<li><a
href="https://github.com/cross-platform-actions/action/commit/20ad4b2cebd9f7e4d0b9a17e71a811178b95e698"><code>20ad4b2</code></a>
Fix <a
href="https://redirect.github.com/cross-platform-actions/action/issues/67">#67</a>:
Use <code>unsafe</code> as the cache mode disks</li>
<li><a
href="https://github.com/cross-platform-actions/action/commit/d9184930c333266f6819290a62817f3dc884ffd5"><code>d918493</code></a>
Always terminate VM instead of shutting down.</li>
<li><a
href="https://github.com/cross-platform-actions/action/commit/626f1d6c958e17e60619b6a2936726ed4c59a36f"><code>626f1d6</code></a>
Fix error when terminating the VM</li>
<li><a
href="https://github.com/cross-platform-actions/action/commit/d59f08dc5c932d21a2663daac30c3a4942a177ba"><code>d59f08d</code></a>
Print stack trace for uncaught exceptions</li>
<li><a
href="https://github.com/cross-platform-actions/action/commit/7f2fab9c5601e3f6f057283dbac16f581ded8a8b"><code>7f2fab9</code></a>
Revert &quot;Run SSH in verbose mode when debug mode is
enabled&quot;</li>
<li><a
href="https://github.com/cross-platform-actions/action/commit/0f566c356e98c968c75937a2c38a5de468ffc0bb"><code>0f566c3</code></a>
[no ci] Update the changelog</li>
<li><a
href="https://github.com/cross-platform-actions/action/commit/b7f77446bb2277f3e4c16ddd30056c8bc1b706f7"><code>b7f7744</code></a>
[no ci] Fix spelling</li>
<li><a
href="https://github.com/cross-platform-actions/action/commit/9894a9b11876c7fc7872617db592776585446d8f"><code>9894a9b</code></a>
Wrap <code>host</code> module in namespace</li>
<li><a
href="https://github.com/cross-platform-actions/action/commit/87fdd346a2f9d849e6cd66745590059eb3e52fe2"><code>87fdd34</code></a>
Fix broken test-vm-shutdown tests</li>
<li>Additional commits viewable in <a
href="https://github.com/cross-platform-actions/action/compare/v0.21.1...v0.22.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=cross-platform-actions/action&package-manager=github_actions&previous-version=0.21.1&new-version=0.22.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-04 22:38:33 +02:00
Lior KoganandGitHub 5189838350 Update CONTRIBUTING.md (#12907)
- Referring to Redis Discord channel instead of the mailing list.
- Referring to the licensing instead of repeating it.
2024-01-03 17:21:19 +02:00
Madelyn OlsonandGitHub 068051e378 Handle recursive serverAsserts and provide more information for recursive segfaults (#12857)
This change is trying to make two failure modes a bit easier to deep dive:
1. If a serverPanic or serverAssert occurs during the info (or module)
printing, it will recursively panic, which is a lot of fun as it will
just keep recursively printing. It will eventually stack overflow, but
will generate a lot of text in the process.
2. When a segfault happens during the segfault handler, no information
is communicated other than it happened. This can be problematic because
`info` may help diagnose the real issue, but without fixing the
recursive crash it might be hard to get at that info.
2024-01-02 18:20:22 -08:00
c3f8b542ee Manage number of new connections per cycle (#12178)
There are situations (especially in TLS) in which the engine gets too occupied managing a large number of new connections. Existing connections may time-out while the server is processing the new connections initial TLS handshakes, which may cause cause new connections to be established, perpetuating the problem. To better manage the tradeoff between new connection rate and other workloads, this change adds a new config to manage maximum number of new connections per event loop cycle, instead of using a predetermined number (currently 1000).

This change introduces two new configurations, max-new-connections-per-cycle and max-new-tls-connections-per-cycle. The default value of the tcp connections is 10 per cycle and the default value of tls connections per cycle is 1.
---------

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2024-01-02 15:15:03 -08:00
Chen TianjieandGitHub 9d0158bf89 Reorder signalModifiedKey in xaddCommand. (#12895)
This PR is a supplement to #11144, moving `signalModifiedKey` in
`xaddCommand` after the trimming, to ensure the state of key eventual
consistency. Currently there is no problem with Redis, but it is better
to avoid issues in future development on Redis.
2023-12-28 13:29:27 +02:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2c5b51ad26 Bump github/codeql-action from 2 to 3 (#12869)
Bumps [github/codeql-action](https://github.com/github/codeql-action)
from 2 to 3.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/releases">github/codeql-action's
releases</a>.</em></p>
<blockquote>
<h2>CodeQL Bundle v2.15.4</h2>
<p>Bundles CodeQL CLI v2.15.4</p>
<ul>
<li>(<a
href="https://github.com/github/codeql-cli-binaries/blob/HEAD/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql-cli-binaries/releases/tag/v2.15.4">release</a>)</li>
</ul>
<p>Includes the following CodeQL language packs from <a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4"><code>github/codeql@codeql-cli/v2.15.4</code></a>:</p>
<ul>
<li><code>codeql/cpp-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/cpp/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/cpp/ql/src">source</a>)</li>
<li><code>codeql/cpp-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/cpp/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/cpp/ql/lib">source</a>)</li>
<li><code>codeql/csharp-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/csharp/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/csharp/ql/src">source</a>)</li>
<li><code>codeql/csharp-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/csharp/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/csharp/ql/lib">source</a>)</li>
<li><code>codeql/go-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/go/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/go/ql/src">source</a>)</li>
<li><code>codeql/go-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/go/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/go/ql/lib">source</a>)</li>
<li><code>codeql/java-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/java/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/java/ql/src">source</a>)</li>
<li><code>codeql/java-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/java/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/java/ql/lib">source</a>)</li>
<li><code>codeql/javascript-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/javascript/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/javascript/ql/src">source</a>)</li>
<li><code>codeql/javascript-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/javascript/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/javascript/ql/lib">source</a>)</li>
<li><code>codeql/python-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/python/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/python/ql/src">source</a>)</li>
<li><code>codeql/python-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/python/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/python/ql/lib">source</a>)</li>
<li><code>codeql/ruby-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/ruby/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/ruby/ql/src">source</a>)</li>
<li><code>codeql/ruby-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/ruby/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/ruby/ql/lib">source</a>)</li>
<li><code>codeql/swift-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/swift/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/swift/ql/src">source</a>)</li>
<li><code>codeql/swift-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/swift/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.4/swift/ql/lib">source</a>)</li>
</ul>
<h2>CodeQL Bundle</h2>
<p>Bundles CodeQL CLI v2.15.3</p>
<ul>
<li>(<a
href="https://github.com/github/codeql-cli-binaries/blob/HEAD/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql-cli-binaries/releases/tag/v2.15.3">release</a>)</li>
</ul>
<p>Includes the following CodeQL language packs from <a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3"><code>github/codeql@codeql-cli/v2.15.3</code></a>:</p>
<ul>
<li><code>codeql/cpp-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/cpp/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/cpp/ql/src">source</a>)</li>
<li><code>codeql/cpp-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/cpp/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/cpp/ql/lib">source</a>)</li>
<li><code>codeql/csharp-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/csharp/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/csharp/ql/src">source</a>)</li>
<li><code>codeql/csharp-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/csharp/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/csharp/ql/lib">source</a>)</li>
<li><code>codeql/go-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/go/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/go/ql/src">source</a>)</li>
<li><code>codeql/go-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/go/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/go/ql/lib">source</a>)</li>
<li><code>codeql/java-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/java/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/java/ql/src">source</a>)</li>
<li><code>codeql/java-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/java/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/java/ql/lib">source</a>)</li>
<li><code>codeql/javascript-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/javascript/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/javascript/ql/src">source</a>)</li>
<li><code>codeql/javascript-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/javascript/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/javascript/ql/lib">source</a>)</li>
<li><code>codeql/python-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/python/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/python/ql/src">source</a>)</li>
<li><code>codeql/python-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/python/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/python/ql/lib">source</a>)</li>
<li><code>codeql/ruby-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/ruby/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/ruby/ql/src">source</a>)</li>
<li><code>codeql/ruby-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/ruby/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/ruby/ql/lib">source</a>)</li>
<li><code>codeql/swift-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/swift/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/swift/ql/src">source</a>)</li>
<li><code>codeql/swift-all</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/swift/ql/lib/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.3/swift/ql/lib">source</a>)</li>
</ul>
<h2>CodeQL Bundle</h2>
<p>Bundles CodeQL CLI v2.15.2</p>
<ul>
<li>(<a
href="https://github.com/github/codeql-cli-binaries/blob/HEAD/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql-cli-binaries/releases/tag/v2.15.2">release</a>)</li>
</ul>
<p>Includes the following CodeQL language packs from <a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.2"><code>github/codeql@codeql-cli/v2.15.2</code></a>:</p>
<ul>
<li><code>codeql/cpp-queries</code> (<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.2/cpp/ql/src/CHANGELOG.md">changelog</a>,
<a
href="https://github.com/github/codeql/tree/codeql-cli/v2.15.2/cpp/ql/src">source</a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action's
changelog</a>.</em></p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/github/codeql-action/commit/3a9f6a89e06fb331ce22481637d19617501a5b7f"><code>3a9f6a8</code></a>
update javascript files</li>
<li><a
href="https://github.com/github/codeql-action/commit/cc4fead714532424ab15b501a01d18c7f34d17e2"><code>cc4fead</code></a>
update version in various hardcoded locations</li>
<li><a
href="https://github.com/github/codeql-action/commit/183559cea87c163bb01e99e0a15fd5500b23307f"><code>183559c</code></a>
Merge branch 'main' into update-bundle/codeql-bundle-v2.15.4</li>
<li><a
href="https://github.com/github/codeql-action/commit/5b52b36d41451ee775917788e3bf225d7c1ceab3"><code>5b52b36</code></a>
reintroduce PR check that confirm action can be still be compiled on
node16</li>
<li><a
href="https://github.com/github/codeql-action/commit/5b19bef41e08b00f3d8f48219f46b1866a4f940d"><code>5b19bef</code></a>
change to node20 for all actions</li>
<li><a
href="https://github.com/github/codeql-action/commit/f2d0c2e7ae58b70f81b81bf4c8b5c7a81fb2a5d6"><code>f2d0c2e</code></a>
upgrade node type definitions</li>
<li><a
href="https://github.com/github/codeql-action/commit/d651fbc494fab13836ab76ffcfc8ce08c7c98d78"><code>d651fbc</code></a>
change to node20 for all actions</li>
<li><a
href="https://github.com/github/codeql-action/commit/382a50a0284c0de445104889a9d6003acb4b3c1d"><code>382a50a</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/2021">#2021</a>
from github/mergeback/v2.22.9-to-main-c0d1daa7</li>
<li><a
href="https://github.com/github/codeql-action/commit/458b4226ad8e38f90ff6a4ad1e18ab2593e7e3dc"><code>458b422</code></a>
Update checked-in dependencies</li>
<li><a
href="https://github.com/github/codeql-action/commit/5e0f9dbc48f564b68392e465dcdacd74eab63e25"><code>5e0f9db</code></a>
Update changelog and version after v2.22.9</li>
<li>See full diff in <a
href="https://github.com/github/codeql-action/compare/v2...v3">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github/codeql-action&package-manager=github_actions&previous-version=2&new-version=3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-28 11:32:23 +02:00
guybe7andGitHub 12b611b374 WAITAOF: Try to wake blocked clients ASAP in the next beforeSleep (#12627)
In case server.fsynced_reploff changed (e.g. flushAppendOnly set it to
server.master_repl_offset in case there was nothing to fsync) we want to
avoid sleeping before the next beforeSleep so we can call
blockedBeforeSleep ASAP.
without that, in case there's no incoming traffic, we could be waiting
for the next cron timer event to wake us up.
2023-12-28 11:27:58 +02:00
BinbinandGitHub 99c468c38c Fix crash caused by pubsubShardUnsubscribeAllChannelsInSlot not deleting the client (#12896)
The code does not delete the corresponding node when traversing clients,
resulting in a loop, causing the dictDelete() == DICT_OK assertion to
fail.

In addition, did a cleanup, in the dictCreate scenario, we can avoid a
dictFind call since the dict is empty.

Issue was introduced in #12804.
2023-12-28 08:32:51 +02:00
BinbinandGitHub 5b1fe925f2 Adjust redis-cli --cluster create arity from -2 to -1 (#12892)
When arity is -2, it allows us to input two nodes, but returns:
```
*** ERROR: Invalid configuration for cluster creation.
*** Redis Cluster requires at least 3 master nodes.
*** This is not possible with 2 nodes and 0 replicas per node.
*** At least 3 nodes are required.
```

When we input one node, it return:
```
[ERR] Wrong number of arguments for specified --cluster sub command
```

Strictly speaking -2 should also be rejected, because redis-cli
requires at least three nodes. However, the error message was not
very friendly, so decided to change it arity -1.

This closes #12891.
2023-12-28 08:26:23 +02:00
8527959598 Replace slots_to_channels radix tree with slot specific dictionaries for shard channels. (#12804)
We have achieved replacing `slots_to_keys` radix tree with key->slot
linked list (#9356), and then replacing the list with slot specific
dictionaries for keys (#11695).

Shard channels behave just like keys in many ways, and we also need a
slots->channels mapping. Currently this is still done by using a radix
tree. So we should split `server.pubsubshard_channels` into 16384 dicts
and drop the radix tree, just like what we did to DBs.

Some benefits (basically the benefits of what we've done to DBs):
1. Optimize counting channels in a slot. This is currently used only in
removing channels in a slot. But this is potentially more useful:
sometimes we need to know how many channels there are in a specific slot
when doing slot migration. Counting is now implemented by traversing the
radix tree, and with this PR it will be as simple as calling `dictSize`,
from O(n) to O(1).
2. The radix tree in the cluster has been removed. The shard channel
names no longer require additional storage, which can save memory.
3. Potentially useful in slot migration, as shard channels are logically
split by slots, thus making it easier to migrate, remove or add as a
whole.
4. Avoid rehashing a big dict when there is a large number of channels.

Drawbacks:
1. Takes more memory than using radix tree when there are relatively few
shard channels.

What this PR does:
1. in cluster mode, split `server.pubsubshard_channels` into 16384
dicts, in standalone mode, still use only one dict.
2. drop the `slots_to_channels` radix tree.
3. to save memory (to solve the drawback above), all 16384 dicts are
created lazily, which means only when a channel is about to be inserted
to the dict will the dict be initialized, and when all channels are
deleted, the dict would delete itself.
5. use `server.shard_channel_count` to keep track of the number of all
shard channels.

---------

Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2023-12-27 17:40:45 +08:00
Moshe KaplanandGitHub fa751f9bef config.c: Avoid leaking file handle if file is 0 bytes (#12828)
If fopen() is successful and redis_fstat determines that the file is 0
bytes, the file handle stored in fp will leak. This change closes the
filehandle stored in fp if the file is 0 bytes.

Second attempt at fixing Coverity 390029

This is a follow-up to #12796
2023-12-27 08:53:56 +02:00
bef5715374 Fix oom-score-adj test due to no permission (#12887)
Fix #12792

On ubuntu 23(lunar), non-root users will not be allowed to change the
oom_score_adj of a process to a value that is too low.
Since terminal's default oom_score_adj is 200, if we run the test on
terminal, we won't be able to set the oom_score_adj of the redis process
to 9 or 22, which is too low.

Reproduction on ubuntu 23(lunar) terminal:
```sh
$ cat /proc/`pgrep redis-server`/oom_score_adj
200
$ echo 100 > /proc/`pgrep redis-server`/oom_score_adj
# success without error
$ echo 99 > /proc/`pgrep redis-server`/oom_score_adj
echo: write error: Permission denied
```

As from the output above, we can only set the minimum oom score of redis
processes to 100.
By modifying the test, make oom_score_adj only increase upwards and not
decrease.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2023-12-27 08:42:46 +02:00
1aa633d61b Implement TCP Keep-Alives across most Unix-like systems (#12782)
## TCP Keep-Alives

[TCP
Keep-Alives](https://datatracker.ietf.org/doc/html/rfc9293#name-tcp-keep-alives)
provides a way to detect whether a TCP connection is alive or dead,
which can be useful for reducing system resources by cleaning up dead
connections.

There is full support of TCP Keep-Alives on Linux and partial support on
macOS in `redis` at present.

This PR intends to complete the rest.

## Unix-like OS's support

`TCP_KEEPIDLE`, `TCP_KEEPINTVL`, and `TCP_KEEPCNT` are not included in
the POSIX standard for `setsockopts`, while these three socket options
are widely available on most Unix-like systems and Windows.

### References

- [AIX](https://www.ibm.com/support/pages/ibm-aix-tcp-keepalive-probes)
- [DragonflyBSD](https://man.dragonflybsd.org/?command=tcp&section=4)
- [FreeBSD](https://www.freebsd.org/cgi/man.cgi?query=tcp)
-
[HP-UX](https://docstore.mik.ua/manuals/hp-ux/en/B2355-60130/TCP.7P.html)
- [illumos](https://illumos.org/man/4P/tcp)
- [Linux](https://man7.org/linux/man-pages/man7/tcp.7.html)
- [NetBSD](https://man.netbsd.org/NetBSD-8.0/tcp.4)
-
[Windows](https://learn.microsoft.com/en-us/windows/win32/winsock/ipproto-tcp-socket-options)

### Mac OS

In earlier versions, macOS only supported setting `TCP_KEEPALIVE` (the
equivalent of `TCP_KEEPIDLE` on other platforms), but since macOS 10.8
it has supported `TCP_KEEPINTVL` and `TCP_KEEPCNT`.

Check out [this mailing
list](https://lists.apple.com/archives/macnetworkprog/2012/Jul/msg00005.html)
and [the source
code](https://github.com/apple/darwin-xnu/blob/main/bsd/netinet/tcp.h#L215-L230)
for more details.

### Solaris

Solaris claimed it supported the TCP-Alives mechanism, but
`TCP_KEEPIDLE`, `TCP_KEEPINTVL`, and `TCP_KEEPCNT` were not available on
Solaris until the latest version 11.4. Therefore, we need to simulate
the TCP-Alives mechanism on other platforms via
`TCP_KEEPALIVE_THRESHOLD` + `TCP_KEEPALIVE_ABORT_THRESHOLD`.

- [Solaris
11.3](https://docs.oracle.com/cd/E86824_01/html/E54777/tcp-7p.html)
- [Solaris
11.4](https://docs.oracle.com/cd/E88353_01/html/E37851/tcp-4p.html)

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-12-26 18:44:18 +02:00
Jeff LiuandGitHub 27a8e3b04e fix missing comments (#12878)
add a missing comment for `dont_compress` and fix the bits calculation
2023-12-25 19:30:05 -08:00
zaljandGitHub baf5699d77 fix comment of aeProcessEvents (#12884)
The implementation of aeProcessEvents seems have different behavior from
the top comment.
The implementation process file events first, then process time events.
2023-12-25 18:58:14 -08:00
BinbinandGitHub 71f31da66f Add restart option to create-cluster script (#12885)
When testing and debugging the cluster code before, you need
to stop the cluster after making changes, and then start the
cluster again. Add a restart option for ease of use.
2023-12-25 18:36:44 -08:00
20214b26a4 Don't disconnect all clients in ACL LOAD (#12171)
Previous implementation would disconnect _all_ clients when running `ACL
LOAD`, which wasn't very useful.

This change brings the behavior in line with that of `ACL SETUSER`, `ACL
DELUSER`, in that only clients whose user is deleted or clients
subscribed to channels which they no longer have access to will be
disconnected.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Madelyn Olson <34459052+madolson@users.noreply.github.com>
2023-12-24 11:56:44 +02:00
09e0d338f5 redis-cli adds -4 / -6 options to determine IPV4 / IPV6 priority in DNS lookup (#11315)
This PR, we added -4 and -6 options to redis-cli to determine
IPV4 / IPV6 priority in DNS lookup.
This was mentioned in
https://github.com/redis/redis/pull/11151#issuecomment-1231570651

For now it's only used in CLUSTER MEET.

The options also made it possible to reliably test dns lookup in CI,
using this option, we can add some localhost tests for #11151.

The commit was cherry-picked from #11151, back then we decided to split
the PR.

Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2023-12-24 10:40:34 +02:00
BinbinandGitHub 23e980e77a Move cliVersion to cli_common and add --version support for redis-check-aof (#10856)
Let us see which version of redis this tool is part of.
Similarly to redis-cli, redis-benchmark and redis-check-rdb

redis-rdb-check and redis-aof-check are actually symlinks to redis,
so they will directly use getVersion in server, the format became:
```
{title} v={redis_version} sha={sha}:{dirty} malloc={malloc} bits={bits} build={build}
```

Move cliVersion into cli_common, redis-cli and redis-benchmark will
use it, and the format is not change:
```
{title} {redis_version} (git:{sha})
```
2023-12-21 13:51:46 +02:00
Wen HuiandGitHub 5dc631d880 Add missing test cases for hash commands (#12851)
We dont have test for hgetall against key doesnot exist so added the
test in test suite and along with this, added wrong type cases for other
missing commands.
2023-12-17 14:02:53 +02:00
BinbinandGitHub adbb534f03 Always keep an in-memory history of all commands in redis-cli (#12862)
redis-cli avoids saving sensitive commands in it's history (doesn't
persist them to the history file).
this means that if you had a typo and you wanna re-run the command, you
can't easily do that.
This PR changes that to keep an in-memory history of all the redacted
commands, and just
not persist them to disk. This way we would be able to press the up
arrow and
re-try the command freely, and it'll just not survive a redis-cli
restart.
2023-12-15 17:22:02 +02:00
zhaozhao.zzandGitHub d8a21c5767 Unified db rehash method for both standalone and cluster (#12848)
After #11695, we added two functions `rehashingStarted` and
`rehashingCompleted` to the dict structure. We also registered two
handlers for the main database's dict and expire structures. This allows
the main database to record the dict in `rehashing` list when rehashing
starts. Later, in `serverCron`, the `incrementallyRehash` function is
continuously called to perform the rehashing operation. However,
currently, when rehashing is completed, `rehashingCompleted` does not
remove the dict from the `rehashing` list. This results in the
`rehashing` list containing many invalid dicts. Although subsequent cron
checks and removes dicts that don't require rehashing, it is still
inefficient.

This PR implements the functionality to remove the dict from the
`rehashing` list in `rehashingCompleted`. This is achieved by adding
`metadata` to the dict structure, which keeps track of its position in
the `rehashing` list, allowing for quick removal. This approach avoids
storing duplicate dicts in the `rehashing` list.

Additionally, there are other modifications:

1. Whether in standalone or cluster mode, the dict in database is
inserted into the rehashing linked list when rehashing starts. This
eliminates the need to distinguish between standalone and cluster mode
in `incrementallyRehash`. The function only needs to focus on the dicts
in the `rehashing` list that require rehashing.
2. `rehashing` list is moved from per-database to Redis server level.
This decouples `incrementallyRehash` from the database ID, and in
standalone mode, there is no need to iterate over all databases,
avoiding unnecessary access to databases that do not require rehashing.
In the future, even if unsharded-cluster mode supports multiple
databases, there will be no risk involved.
3. The insertion and removal operations of dict structures in the
`rehashing` list are decoupled from `activerehashing` config.
`activerehashing` only controls whether `incrementallyRehash` is
executed in serverCron. There is no need for additional steps when
modifying the `activerehashing` switch, as in #12705.
2023-12-15 10:42:53 +08:00
Guillaume KoenigandGitHub 967fb3c6e8 Extend rax usage by allowing any long long value (#12837)
The raxFind implementation uses a special pointer value (the address of
a static string) as the "not found" value. It works as long as actual
pointers were used. However we've seen usages where long long,
non-pointer values have been used. It creates a risk that one of the
long long value precisely is the address of the special "not found"
value. This commit changes raxFind to return 1 or 0 to indicate
elementhood, and take in a new void **value to optionally return the
associated value.

By extension, this also allow the RedisModule_DictSet/Replace operations
to also safely insert integers instead of just pointers.
2023-12-14 14:50:18 -08:00
Chen TianjieandGitHub e95a5d4831 Support by/get options for sort(_ro) in cluster mode when pattern implies slot. (#12728)
The by/get options of sort/sort_ro command used to be forbidden in
cluster mode, since we are not sure which slot the pattern may be in.

As the optimization done in #12536, patterns now can be mapped to slots,
we should allow by/get options in cluster mode when the pattern maps to
the same slot as the key.
2023-12-13 21:16:36 +02:00
BinbinandGitHub 3c0fd25201 Redact ACL username information and mark *-key-file-pass configs as sensitive (#12860)
In #11489, we consider acl username to be sensitive information,
and consider the ACL GETUSER a sensitive command and remove it
from redis-cli historyfile.

This PR redact username information in ACL GETUSER and ACL DELUSER
from SLOWLOG, and also remove ACL DELUSER from redis-cli historyfile.

This PR also mark tls-key-file-pass and tls-client-key-file-pass
as sensitive config, will redact it from SLOWLOG and also
remove them from redis-cli historyfile.
2023-12-13 15:28:13 +02:00
Chen TianjieandGitHub f9cc25c1dd Add metric to INFO CLIENTS: pubsub_clients. (#12849)
In INFO CLIENTS section, we already have blocked_clients and
tracking_clients. We should add a new metric showing the number of
pubsub connections, which helps performance monitoring and trouble
shooting.
2023-12-13 13:44:13 +08:00
BinbinandGitHub c85a9b7896 Fix delKeysInSlot server events are not executed inside an execution unit (#12745)
This is a follow-up fix to #12733. We need to apply the same changes to
delKeysInSlot. Refer to #12733 for more details.

This PR contains some other minor cleanups / improvements to the test
suite and docs.
It uses the postnotifications test module in a cluster mode test which
revealed a leak in the test module (fixed).
2023-12-11 20:15:19 +02:00
BinbinandGitHub 62419c01db Handle missing fields in dbSwapDatabases and swapMainDbWithTempDb (#12763)
The change in dbSwapDatabases seems harmless. Because in non-clustered
mode, dbBuckets calculations are strictly accurate and in cluster mode,
we only have one DB. Modify it for uniformity (just like resize_cursor).

The change in swapMainDbWithTempDb is needed in case we swap with the
temp db, otherwise the overhead memory usage of db can be miscalculated.

In addition we will swap all fields (including rehashing list), just for
completeness (and reduce the chance of surprises in the future).

Introduced in #12697.
2023-12-10 17:30:20 +02:00
BinbinandGitHub e6423b7a7e Fix rehashingStarted miscalculating bucket_count in dict initialization (#12846)
In the old dictRehashingInfo implementation, for the initialization
scenario,
it mistakenly directly set to_size to DICTHT_SIZE(DICT_HT_INITIAL_EXP),
which
is 4 in our code by default.

In scenarios where dictExpand directly passes the target size as
initialization,
the code will calculate bucket_count incorrectly. For example, in DEBUG
POPULATE
or RDB load scenarios, it will cause the final bucket_count to be
initialized to
65536 (16384 * 4), see:
```
before:
DB 0: 10000000 keys (0 volatile) in 65536 slots HT.

it should be:
DB 0: 10000000 keys (0 volatile) in 16777216 slots HT.
```

In PR, new ht will also be initialized before calling rehashingStarted
in
_dictExpand, so that the calls in dictRehashingInfo can be unified.

Bug was introduced in #12697.
2023-12-10 10:55:30 +02:00
BinbinandGitHub a3ae2ed37b Remove dead code around should_expand_db (#12767)
when dbExpand is called from rdb.c with try_expand set to 0, it will
either panic panic on OOM, or be non-fatal (should not fail RDB loading)

At the same time, the log text has been slightly adjusted to make it
more unified.
2023-12-10 10:40:15 +02:00
BinbinandGitHub 7410d985bc Remove overhead.hashtable.slot-to-keys from memory-stats reply_schema (#12784)
overhead.hashtable.slot-to-keys was added in 7.0 in #10017, then removed
in #11695. Now remove it from reply_schema.
2023-12-10 09:46:21 +02:00
Yanqi LvandGitHub 15b993f1ef Optimize dictExpand of empty dict (#12847)
If a dict is empty before `dictExpand`, we don't need to do rehashing
actually. We can just create a new dict and set it as ht_table[0] to
skip incremental rehashing and free the memory quickly.
2023-12-08 17:48:52 +08:00
bentottenandGitHub 826b39e016 Align server.lastsave and server.rdb_save_time_last by removing multiple calls to time(NULL) (#12823)
This makes sure the various times (server.lastsave and server.rdb_save_time_last) are aligned by using the result of the same time call.
2023-12-07 17:03:51 -08:00
Chen TianjieandGitHub f2d59c4f91 Avoid unnecessary slot computing in KEYS command. (#12843)
If not in cluster mode, there is no need to compute slot.

A bit optimization for #12754
2023-12-07 19:48:15 +08:00
zhaozhao.zzandGitHub 8e11f84ded Fix replica node cannot expand dicts when loading legacy RDB (#12839)
When loading RDB on cluster nodes, it is necessary to consider the
scenario where a node is a replica.

For example, during a rolling upgrade, new version instances are often
mounted as replicas on old version instances. In this case, the full
synchronization legacy RDB does not contain slot information, and the
new version instance, acting as a replica, should be able to handle the
legacy RDB correctly for `dbExpand`.

Additionally, renaming `getMyClusterSlotCount` to `getMyShardSlotCount`
would be appropriate.

Introduced in #11695
2023-12-07 14:30:48 +08:00
Moshe KaplanandGitHub e2a3f3091f coverity.yml: Upload should go to project redis-unstable (#12841)
Coverity project name was changed from redis to redis-unstable. Fix the
upload destination to also go to redis-unstable.

Continuation of #12807
2023-12-06 20:51:58 +02:00
BinbinandGitHub 2f6d4dabaa Fix outdated LFU comments to eliminate confusion (#12244)
The decrement time was replaced by access time in
583c314725.

The halved and doubled LFU_INIT_VAL logic has been changed in
06ca9d6839.
Now we just decrement the counter by num_periods. This has been
previously fixed in redis.conf, #11108.
2023-12-06 20:46:57 +02:00
77e69d8884 GH Workflows: Create CI job for Coverity scan (#12807)
I've noticed that https://scan.coverity.com/projects/redis already
exists, but appears to be only updated on an ad-hoc basis. creating
[redis-unstable](https://scan.coverity.com/projects/redis-unstable?tab=project_settings)
project in coverity for this CI.

This PR adds a GitHub Action-based CI job to create a new Coverity build
once daily, so that there is always a recent scan available.

This is within the limit, as Redis is ~150K LOC and per
https://scan.coverity.com/faq#frequency :

> Up to 21 builds per week, with a maximum of 3 builds per day, for
projects with 100K to 500K lines of code

Before this is merged in, two new secrets will need to be created:

COVERITY_SCAN_EMAIL with the email address used for accessing Coverity
COVERITY_SCAN_TOKEN with the Project token from
https://scan.coverity.com/projects/redis-unstable?tab=project_settings

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-12-06 14:50:00 +02:00
zhaozhao.zzandGitHub b730404c2f Fix multi dbs donot dbExpand when loading RDB (#12840)
Currently, during RDB loading, once a `dbExpand` is performed, the
`should_expand_db` flag is set to 0. This causes the remaining DBs
unable to do `dbExpand` when there are multiple DBs.

To fix this issue, we need to set `should_expand_db` back to 1 whenever
we encounter `RDB_OPCODE_RESIZEDB`. This ensures that each DB can
perform `dbExpand` correctly.

Additionally, the initial value of `should_expand_db` should also be set
to 0 to prevent invalid `dbExpand` in older versions of RDB where
`RDB_OPCODE_RESIZEDB` is not present.

problem introduced in #11695
2023-12-06 10:59:56 +02:00
9ee1cc33a3 Make the sampling logic in eviction clearer (#12781)
Additional optimizations for the eviction logic in #11695:

To make the eviction logic clearer and decouple the number of sampled
keys from the running mode (cluster or standalone).
* When sampling in each database, we only care about the number of keys
in the current database (not the dicts we sampled from).
* If there are a insufficient number of keys in the current database
(e.g. 10 times the value of `maxmemory_samples`), we can break out
sooner (to avoid looping on a sparse database).
* We'll never try to sample the db dicts more times than the number of
non-empty dicts in the db (max 1 in non-cluster mode).

And it also ensures that each database has a sufficient amount of
sampled keys, so even if unsharded-cluster supports multiple databases,
there won't be any issues.

other changes:
1. keep track of the number of non-empty dicts in each database.
2. move key_count tracking into cumulativeKeyCountAdd rather than all
it's callers

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-12-06 09:37:24 +08:00
Chen TianjieandGitHub 991aff1c0f Optimize KEYS when pattern includes hashtag and implies a single slot. (#12754)
in #12536 we made a similar optimization for SCAN, now that hashtags in
patterns. When we can make sure all keys matching the pettern will be in
the same slot, we can limit the iteration to run only one one.
2023-12-05 16:21:50 +02:00
Madelyn OlsonandGitHub 35c8d616cf Only rebuild server when src or deps are modified (#12799)
`make test` will unnecessary rebuild `redis-server` if tests are modified because mkheader will touch releae.c.
This changes scopes down when we re-generate release.c to only when actual source files are modified.
2023-12-04 19:24:24 -08:00
BinbinandGitHub 764838d66f Check whether the client is NULL in luaCreateFunction (#12829)
It was first added to load lua from RDB, see 28dfdca7. After #9812,
we no longer save lua in RDB. luaCreateFunction will only be called
in script load and eval*, both of which are available in the client.

It could be that that some day we'll still want to load scripts from
somewhere that's not a client. This fix is in dead code.
2023-12-04 20:12:48 +02:00
BinbinandGitHub 8a4ccb01b3 Fix clusterLoadConfig aux_argv minor memory leak (#12726)
We forgot to call sdsfreesplitres. This is just a cleanup since it will
only be leaked in the error paths, and we will exit on the error paths.
2023-12-03 11:00:53 +02:00
91309f7981 Fix compilation warning in KeySpace_ServerEventCallback and add CFLAGS=-Werror flag for module CI (#12786)
Warning:
```
postnotifications.c:216:77: warning: format specifies type 'long' but the argument has type 'uint64_t' (aka 'unsigned long long') [-Wformat]
        RedisModule_Log(ctx, "warning", "Got an unexpected subevent '%ld'", subevent);
                                                                     ~~~    ^~~~~~~~
                                                                     %llu
```

CI:
https://github.com/redis/redis/actions/runs/6937308713/job/18871124342#step:6:115

## Other
Add `CFLAGS=-Werror` flag for module CI.

---------

Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2023-11-30 17:41:00 +02:00
BinbinandGitHub e216c83909 Change addReplyErrorFormat to addReplyError when there is no format (#12641)
This is just a cleanup, although they are both correct, the change
is normatively better, and addReplyError is also much faster.
Although not important, speed is not important for these error cases.
2023-11-30 12:36:17 +02:00
lyq2333andGitHub 423565f784 Optimize CPU cache efficiency on dict while rehashing in dictTwoPhaseUnlinkFind (#12818)
In #5692, we optimize CPU cache efficiency on dict while rehashing but
missed the modification in dictTwoPhaseUnlinkFind.

This PR supplements it.
2023-11-30 13:50:09 +08:00
BinbinandGitHub 22cc9b5122 Use CLZ in _dictNextExp to get the next power of two (#12815)
In the past, we did not call _dictNextExp frequently. It was only
called when the dictionary was expanded.

Later, dictTypeExpandAllowed was introduced in #7954, which is 6.2.
For the data dict and the expire dict, we can check maxmemory before
actually expanding the dict. This is a good optimization to avoid
maxmemory being exceeded due to the dict expansion.

And in #11692, we moved the dictTypeExpandAllowed check before the
threshold check, this caused a bit of performance degradation, every
time a key is added to the dict, dictTypeExpandAllowed is called to
check.

The main reason for degradation is that in a large dict, we need to
call _dictNextExp frequently, that is, every time we add a key, we
need to call _dictNextExp once. Then the threshold is checked to see
if the dict needs to be expanded. We can see that the order of checks
here can be optimized.

So we moved the dictTypeExpandAllowed check back to after the threshold
check in #12789. In this way, before the dict is actually expanded (that
is, before the threshold is reached), we will not do anything extra
compared to before, that is, we will not call _dictNextExp frequently.

But note we'll still hit the degradation when we over the thresholds.
When the threshold is reached, because #7954, we may delay the dict
expansion due to maxmemory limitations. In this case, we will call
_dictNextExp every time we add a key during this period.

This PR use CLZ in _dictNextExp to get the next power of two. CLZ (count
leading zeros) can easily give you the next power of two. It should be
noted that we have actually introduced the use of __builtin_clzl in
#8687,
which is 7.0. So i suppose all the platforms we use have it (even if the
CPU doesn't have an instruction).

We build 67108864 (2**26) keys through DEBUG POPULTE, which will use
approximately 5.49G memory (used_memory:5898522936). If expansion is
triggered, the additional hash table will consume approximately 1G
memory (2 ** 27 * 8). So we set maxmemory to 6871947673 (that is, 6.4G),
which will be less than 5.49G + 1G, so we will delay the dict rehash
while addint the keys.

After that, each time an element is added to the dict, an allow check
will be performed, that is, we can frequently call _dictNextExp to test
the comparison before and after the optimization. Using DEBUG HTSTATS 0
to
check and make sure that our dict expansion is dealyed.

Using `./src/redis-server redis.conf --save "" --maxmemory 6871947673`.
Using `./src/redis-benchmark -P 100 -r 1000000000 -t set -n 5000000`.
After ten rounds of testing:
```
unstable:           this PR:
769585.94           816860.00
771724.00           818196.69
775674.81           822368.44
781983.12           822503.69
783576.25           828088.75
784190.75           828637.75
791389.69           829875.50
794659.94           835660.69
798212.00           830013.25
801153.62           833934.56
```

We can see there is about 4-5% performance improvement in this case.
2023-11-29 14:42:22 +02:00
BinbinandGitHub bdceaf50e4 Fix the check for new RDB_OPCODE_SLOT_INFO in redis-check-rdb (#12768)
We did not read expires_slot_size, causing its check to fail.
An overlook in #11695.
2023-11-29 14:25:17 +02:00
zhaozhao.zzandGitHub 3431b1f156 format cpu config as redis style (#7351)
The following four configurations are renamed to align with Redis style:

1. server_cpulist renamed to server-cpulist
2. bio_cpulist renamed to bio-cpulist
3. aof_rewrite_cpulist renamed to aof-rewrite-cpulist
4. bgsave_cpulist renamed to bgsave-cpulist

The original names are retained as aliases to ensure compatibility with
old configuration files. We recommend users to gradually transition to
using the new configuration names to maintain consistency in style.
2023-11-29 13:40:06 +08:00
zhaozhao.zzandGitHub a1c5171c1d Fix resize hash tables stuck on the last non-empty slot (#12802)
Introduced in #11695 .

The tryResizeHashTables function gets stuck on the last non-empty slot
while iterating through dictionaries. It does not restart from the
beginning. The reason for this issue is a problem with the usage of
dbIteratorNextDict:

/* Returns next dictionary from the iterator, or NULL if iteration is complete. */
dict *dbIteratorNextDict(dbIterator *dbit) {
    if (dbit->next_slot == -1) return NULL;
    dbit->slot = dbit->next_slot;
    dbit->next_slot = dbGetNextNonEmptySlot(dbit->db, dbit->slot, dbit->keyType);
    return dbGetDictFromIterator(dbit);
}

When iterating to the last non-empty slot, next_slot is set to -1,
causing it to loop indefinitely on that slot. We need to modify the code
to ensure that after iterating to the last non-empty slot, it returns to
the first non-empty slot.

BTW, function tryResizeHashTables is actually iterating over slots
that have keys. However, in its implementation, it leverages the
dbIterator (which is a key iterator) to obtain slot and dictionary
information. While this approach works fine, but it is not very
intuitive. This PR also improves readability by changing the iteration
to directly iterate over slots, thereby enhancing clarity.
2023-11-28 18:50:16 +08:00
zhaozhao.zzandGitHub 095d05786f clarify the comment of findSlotByKeyIndex function (#12811)
The current comment for `findSlotByKeyIndex` is a bit ambiguous and can
be misleading, as it may be misunderstood as getting the next slot
corresponding to target.
2023-11-27 18:45:40 -08:00
BinbinandGitHub d6f19539d2 Un-register notification and server event when RedisModule_OnLoad fails (#12809)
When we register notification or server event in RedisModule_OnLoad, but
RedisModule_OnLoad eventually fails, triggering notification or server
event
will cause the server to crash.

If the loading fails on a later stage of moduleLoad, we do call
moduleUnload
which handles all un-registration, but when it fails on the
RedisModule_OnLoad
call, we only un-register several specific things and these were
missing:

- moduleUnsubscribeNotifications
- moduleUnregisterFilters
- moduleUnsubscribeAllServerEvents

Refactored the code to reuse the code from moduleUnload.

Fixes #12808.
2023-11-27 17:26:33 +02:00
zhaozhao.zzandGitHub 1bd0b54957 Optimize the efficiency of active expiration when databases exceeds 16. (#12738)
Currently, when the number of databases exceeds 16,
the efficiency of cleaning expired keys is relatively low.

The reason is that by default only 16 databases are scanned when
attempting to clean expired keys (CRON_DBS_PER_CALL is 16). But users
may set databases higher than 16, such as 256, but it does not
necessarily mean that all 256 databases have expiration time set. If
only one database has expiration time set, this database needs 16
activeExpireCycle rounds in order to be scanned once, and 15 of those
rounds are meaningless.

To optimize the efficiency of expiration in such scenarios, we use dbs_per_call
to control the number of databases with expired keys being scanned.

Additionally, add a condition to limit the maximum number of rounds
to server.dbnum to prevent excessive spinning. This ensures that even if
only one database has expired keys, it can be triggered within a single cron job.
2023-11-27 12:12:12 +08:00
56ec1ff1ce Call signalModifiedKey after the key modification is completed (#11144)
Fix `signalModifiedKey()` order, call it after key modification
completed, to ensure the state of key eventual consistency.

When a key is modified, Redis calls `signalModifiedKey` to notify other
systems, such as the watch system of transactions and the tracking
system of client side caching. However, in some commands, the
`signalModifiedKey` call happens during the key modification process
instead of after the key modification is completed. This can potentially
cause issues, as systems relying on `signalModifiedKey` may receive the
"write in flight" status of the key rather than its final state.

These commands include:
1. PFADD
2. LSET, LMOVE, LREM
3. ZPOPMIN, ZPOPMAX, BZPOPMIN, BZPOPMAX, ZMPOP, BZMPOP

Currently there is no problem with Redis, but it is better to adjust the
order of `signalModifiedKey()`, to avoid issues in future development on
Redis.

---------

Co-authored-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
2023-11-27 11:16:41 +08:00
2e854bccc6 Fix async safety in signal handlers (#12658)
see discussion from after https://github.com/redis/redis/pull/12453 was
merged
----
This PR replaces signals that are not considered async-signal-safe
(AS-safe) with safe calls.

#### **1. serverLog() and serverLogFromHandler()**
`serverLog` uses unsafe calls. It was decided that we will **avoid**
`serverLog` calls by the signal handlers when:
* The signal is not fatal, such as SIGALRM. In these cases, we prefer
using `serverLogFromHandler` which is the safe version of `serverLog`.
Note they have different prompts:
`serverLog`: `62220:M 26 Oct 2023 14:39:04.526 # <msg>`
`serverLogFromHandler`: `62220:signal-handler (1698331136) <msg>`
* The code was added recently. Calls to `serverLog` by the signal
handler have been there ever since Redis exists and it hasn't caused
problems so far. To avoid regression, from now we should use
`serverLogFromHandler`

#### **2. `snprintf` `fgets` and `strtoul`(base = 16) -------->
`_safe_snprintf`, `fgets_async_signal_safe`, `string_to_hex`**
The safe version of `snprintf` was taken from
[here](https://github.com/twitter/twemcache/blob/8cfc4ca5e76ed936bd3786c8cc43ed47e7778c08/src/mc_util.c#L754)

#### **3. fopen(), fgets(), fclose() --------> open(), read(), close()**

#### **4. opendir(), readdir(), closedir() --------> open(),
syscall(SYS_getdents64), close()**

#### **5. Threads_mngr sync mechanisms**
* waiting for the thread to generate stack trace: semaphore -------->
busy-wait
* `globals_rw_lock` was removed: as we are not using malloc and the
semaphore anymore we don't need to protect `ThreadsManager_cleanups`.

#### **6. Stacktraces buffer**
The initial problem was that we were not able to safely call malloc
within the signal handler.
To solve that we created a buffer on the stack of `writeStacktraces` and
saved it in a global pointer, assuming that under normal circumstances,
the function `writeStacktraces` would complete before any thread
attempted to write to it. However, **if threads lag behind, they might
access this global pointer after it no longer belongs to the
`writeStacktraces` stack, potentially corrupting memory.**
To address this, various solutions were discussed
[here](https://github.com/redis/redis/pull/12658#discussion_r1390442896)
Eventually, we decided to **create a pipe** at server startup that will
remain valid as long as the process is alive.
We chose this solution due to its minimal memory usage, and since
`write()` and `read()` are atomic operations. It ensures that stack
traces from different threads won't mix.

**The stacktraces collection process is now as  follows:**
* Cleaning the pipe to eliminate writes of late threads from previous
runs.
* Each thread writes to the pipe its stacktrace
* Waiting for all the threads to mark completion or until a timeout (2
sec) is reached
* Reading from the pipe to print the stacktraces.

#### **7. Changes that were considered and eventually were dropped**
* replace watchdog timer with a POSIX timer: 
according to [settimer man](https://linux.die.net/man/2/setitimer)

> POSIX.1-2008 marks getitimer() and setitimer() obsolete, recommending
the use of the POSIX timers API
([timer_gettime](https://linux.die.net/man/2/timer_gettime)(2),
[timer_settime](https://linux.die.net/man/2/timer_settime)(2), etc.)
instead.

However, although it is supposed to conform to POSIX std, POSIX timers
API is not supported on Mac.
You can take a look here at the Linux implementation:

[here](https://github.com/redis/redis/commit/c7562ee13546e504977372fdf40d33c3f86775a5)
To avoid messing up the code, and uncertainty regarding compatibility,
it was decided to drop it for now.

* avoid using sds (uses malloc) in logConfigDebugInfo
It was considered to print config info instead of using sds, however
apparently, `logConfigDebugInfo` does more than just print the sds, so
it was decided this fix is out of this issue scope.

#### **8. fix Signal mask check**
The check `signum & sig_mask` intended to indicate whether the signal is
blocked by the thread was incorrect. Actually, the bit position in the
signal mask corresponds to the signal number. We fixed this by changing
the condition to: `sig_mask & (1L << (sig_num - 1))`

#### **9. Unrelated changes**
both `fork.tcl `and `util.tcl` implemented a function called
`count_log_message` expecting different parameters. This caused
confusion when trying to run daily tests with additional test parameters
to run a specific test.
The `count_log_message` in `fork.tcl` was removed and the calls were
replaced with calls to `count_log_message` located in `util.tcl`

---------

Co-authored-by: Ozan Tezcan <ozantezcan@gmail.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2023-11-23 13:22:20 +02:00
BinbinandGitHub 5e403099bd Fix misleading error message in redis.log when loglevel is invalid (#12636)
We don't have any debug level, change it to log level.
2023-11-23 10:23:30 +02:00
Moshe KaplanandGitHub c9aa586b6b rdb.c: Avoid potential file handle leak (Coverity 404720) (#12795)
`open()` can return any non-negative integer on success, including zero.
This change modifies the check from open()'s return value to also
include checking for a return value of zero (e.g., if stdin were closed
and then `open()` was called).

Fixes Coverity 404720

Can't happen in Redis. just a cleanup.
2023-11-23 10:14:17 +02:00
Moshe KaplanandGitHub ae09d4d3ef redis-check-aof.c: Avoid leaking file handle if file is zero bytes (#12797)
If fopen() is successful, but redis_fstat() determines the file is zero
bytes, the file handle stored in fp will leak. This change closes the
filehandle stored in fp if the file is zero bytes.

An FD leak on a tool like redis-check-aof isn't an issue (it'll exit soon anyway).
This is just a cleanup.
2023-11-23 10:08:49 +02:00
Moshe KaplanandGitHub 1c48d3dab2 config.c: Avoid leaking file handle if redis_fstat() fails (#12796)
If fopen() is successful, but redis_fstat() fails, the file handle
stored in fp will leak. This change closes the filehandle stored in fp
if redis_fstat() fails.

Fixes Coverity 390029
2023-11-23 10:06:13 +02:00
Moshe KaplanandGitHub 157e5d47b5 util.c: Don't leak directory handle if recursive call to dirRemove fails (#12800)
If a recursive call to dirRemove() returns -1, dirRemove() the directory
handle stored in dir will leak. This change closes the directory handle
stored in dir even if the recursive call to dirRemove() returns -1.

Fixed Coverity 371073
2023-11-23 10:04:02 +02:00
BinbinandGitHub 463476933c Optimize dict expand check, move allow check after the thresholds check (#12789)
dictExpandAllowed (for the main db dict and the expire dict) seems to
involve a few function calls and memory accesses, and we can do it only
after the thresholds checks and can get some performance improvements.

A simple benchmark test: there are 11032768 fixed keys in the database,
start a redis-server with `--maxmemory big_number --save ""`,
start a redis-benchmark with `-P 100 -r 1000000000 -t set -n 5000000`,
collect `throughput summary: n requests per second` result.

After five rounds of testing:
```
unstable     this PR
848032.56    897988.56
854408.69    913408.88
858663.94    914076.81
871839.56    916758.31
882612.56    920640.75
```

We can see a 5% performance improvement in general condition.
But note we'll still hit the degradation when we over the thresholds.
2023-11-22 11:33:26 +02:00
ef1ca6c882 add some file level comments and copyright (#12793)
A followup PR for #12742
Add some brief comments explaining the purpose of the file to the head
of cluster_legacy.c and cluster.c.
Add copyright notice to cluster.c

Signed-off-by: Josh Hershberg <yehoshua@redis.com>
Co-authored-by: Josh Hershberg <yehoshua@redis.com>
2023-11-22 11:32:23 +02:00
BinbinandGitHub dedbf99a80 Fix dbExpand not dividing by slots, resulting in consuming slots times the dictExpand (#12773)
We meant to divide it by the number of slots, otherwise it will do slots
times
dictExpand, bug was introduced in #11695.
2023-11-22 11:16:06 +02:00
Oran AgraandGitHub 58cb302526 Cluster refactor, common API for different implementations (#12742)
This PR reworks the clustering code to support multiple clustering
implementations, specifically, the current "legacy" clustering
implementation or, although not part of this PR, flotilla (see #10875).
Which implementation is used could be a compile-time flag (will be added
later). Legacy clustering functionality remains unchanged.

The basic idea is as follows. The header cluster.h now contains function
declarations that define the "Cluster API." These are the contract and
interface between any clustering implementation and the rest of the
Redis source code. Some of the function definitions are shared between
all clustering implementations. These functions are in cluster.c. The
functions and data structures specific to legacy clustering are in
cluster-legacy.c/h. One consequence of this is that the structs
clusterNode and clusterState which were previously "public" to the rest
of Redis are now hidden behind the Cluster API.

The PR is divided up into commits, each with a commit message explaining
the changes. some are just mass rename or moving code between files (may
not require close inspection / review), others are manual changes.

One other, related change is:
- The "failover" command is now plugged into the Cluster API so that the
clustering implementation can (a) enable/disable the command to begin
with and if enabled (b) perform the actual failover. The "failover"
command remains disabled for legacy clustering.
2023-11-22 09:31:19 +02:00
Josh Hershberg eebb025826 Cluster refactor: Some code convention fixes
Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-22 05:54:48 +02:00
Josh Hershberg 290f376429 Cluster refactor: fn renames + small compilation issue on ubuntu
Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-22 05:54:06 +02:00
Josh Hershberg 13b754853c Cluster refactor: cluster.h - reorder functions into logical groups
Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-22 05:54:06 +02:00
Josh Hershberg 2e5181ef28 Cluster refactor: Add failover cmd support to cluster api
The failover command is up until now not supported
in cluster mode. This commit allows a cluster
implementation to support the command. The legacy
clustering implementation still does not support
this command.

Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-22 05:54:06 +02:00
Josh Hershberg c6157b3510 Cluster refactor: Make clustering functions common
Move primary functions used to implement datapath
clustering into cluster.c, making them shared. This
required adding "accessor" and other functions to
abstract access to node details and cluster state.

Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-22 05:54:06 +02:00
Josh Hershberg 4afc54ad9b Cluster refactor: break up clusterCommand
Divide up clusterCommand into clusterCommand for shared
sub-commands and clusterCommandSpecial for implementation
specific sub-commands. So to, the cluster command help
sub-command has been divided into two implementations,
clusterCommandHelp and clusterCommandHelpSpecial. Some
common sub-subcommand implementations have been extracted
and their implemenations either made shared or else
implementation specific.

Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-22 05:54:06 +02:00
Josh Hershberg 33ef6a3003 Cluster refactor: s/clusterNodeGetSlotBit/clusterNodeCoversSlot/
Simple rename, "GetSlotBit" is implementation specific

Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-22 05:54:06 +02:00
Josh Hershberg ac1513221b Cluster refactor: Move items from cluster_legacy.c to cluster.c
Move (but do not change) some items from cluster_legacy.c
back info cluster.c. These items are shared code that all
clustering implementations will use.

Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-22 05:54:06 +02:00
Josh Hershberg 040cb6a4aa Cluster refactor: verifyClusterNodeId need not be 'public'
Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-22 05:54:06 +02:00
Josh Hershberg 4944eda696 Cluster refactor: Move more stuff from cluster.h to cluster_legacy.h
More declerations can be moved into cluster_legacy.h
as they are not requied for the cluster api. The code
was simply moved, not changed in any way.

Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-22 05:54:03 +02:00
Josh Hershberg d9a0478599 Cluster refactor: Make clusterNode private
Move clusterNode into cluster_legacy.h.
In order to achieve this some accessor methods
were added and also a refactor of how debugCommand
handles cluster related subcommands.

Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-22 05:50:46 +02:00
Josh Hershberg 98a6c44b75 Cluster refactor: Make clusterState private
Move clusterState into cluster_legacy.h. In order to achieve
this some "accessor" methods needed to be added to the
cluster API and some other minor refactors.

Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-22 05:44:10 +02:00
BinbinandGitHub 2c41b13505 Block DEBUG POPULATE in loading and async-loading (#12790)
When we are loading data, it is not safe to generate data
through DEBUG POPULATE. POPULATE may generate keys, causing
panic when loading data with duplicate keys.
2023-11-21 17:00:27 +02:00
Josh Hershberg 5292adb985 Cluster refactor: Move trivial stuff into cluster_legacy.h
Move some declerations from cluster.h to cluster_legacy.h.
The items moved are specific to the legacy clustering
implementation and DO NOT require any other refactoring
other than moving them from one file to another.

Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-21 12:49:14 +02:00
Josh Hershberg 6a6ae6ffe8 Cluster refactor: Create new cluster.c and include of cluster_legacy.h
create new cluster.c

Signed-off-by: Josh Hershberg <yehoshua@redis.com>

forgot to #include cluster_legacy.h

Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-21 12:49:14 +02:00
Josh Hershberg 86915775f1 Cluster refactor: rename cluster.c -> cluster_legacy.c
Signed-off-by: Josh Hershberg <yehoshua@redis.com>
2023-11-21 12:49:14 +02:00
4278ed8de5 redis-cli --bigkeys ,--hotkeys and --memkeys to replica in cluster mode (#12735)
Make redis-cli --bigkeys and --memkeys usable on a replicas in cluster
mode, by sending the READONLY command. This is only if -c is also given.

We don't detect if a node is a master or a replica so we send READONLY
in both cases. The READONLY has no effect on masters.

    Release notes:
    Make redis-cli --bigkeys and --memkeys usable on cluster replicas

---------

Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Oran Agra <oran@redislabs.com>
2023-11-20 09:33:28 +02:00
Hwang Si YeonandGitHub a1f91ffa18 Add an explanation for URI with -u in redis-cli --help (#12751)
Add documentation of the URI format in the `--help` output of
`redis-cli` and `redis-benchmark`.

In particular, it's good for users to know that they need to specify
"default" as the username when authenticating without a username. Other
details of the URI format are described too, like scheme and dbnum.

It used to be possible to connect to Redis using an URL with an empty
username, like `redis-cli -u redis://:PASSWORD@localhost:6379/0`. This
was broken in 6.2 (#8048), and there was a discussion about it #9186.
Now, users need to specify "default" as the username and it's better to
document it.

Refer to #12746 for more details.
2023-11-19 15:09:14 +02:00
Wen HuiandGitHub 5a1f4b9aec Adding missing SWAPDB related test cases. (#12769)
We have some test cases of swapdb with watchkey but missing seperate
basic swapdb test cases, unhappy path and flushdb after swapdb. So added
the test cases in keyspace.tcl.
2023-11-19 12:44:48 +02:00
BinbinandGitHub 3d9c427f8c Fix timing issue in CLUSTER SLAVE / REPLICAS consistent test (#12774)
CI reports that this test failed, the reason is because during
the command processing, the node processed PING/PONG, resulting
in ping_sent or pong_received mismatch.

Change to use MULTI to avoid timing issue. The test was introduced
in #12224.
2023-11-19 11:09:33 +02:00
zhaozhao.zzandGitHub eb392c0a6f replace calculateKeySlot with known slot in evictionPoolPopulate (#12777) 2023-11-17 19:01:06 +08:00
zhaozhao.zzandGitHub 6013122c8e check dbSize when rewrite AOF, in case unnecessary SELECT (#12775)
Introduced by #11695, should skip empty db in case unnecessary SELECT.
2023-11-17 19:00:26 +08:00
BinbinandGitHub 9b6dded421 Fix empty rehashing list in swapdb mode (#12770)
In swapdb mode, the temp db does not init the rehashing list,
the change added in #12764 caused cluster ci to fail.
2023-11-16 11:18:25 +02:00
BinbinandGitHub 4366bbaa61 Empty rehashing list in emptyDbStructure (#12764)
This is currently harmless, since we have already cleared the dict
before, it will reset the rehashidx to -1, and in incrementallyRehash
we will call dictIsRehashing to check.

It would be nice to empty the list to avoid meaningless attempts, and
the code is also unified to reduce misunderstandings.
2023-11-15 07:55:34 +02:00
BinbinandGitHub fe36306340 Fix DB iterator not resetting pauserehash causing dict being unable to rehash (#12757)
When using DB iterator, it will use dictInitSafeIterator to init a old safe
dict iterator. When dbIteratorNext is used, it will jump to the next slot db
dict when we are done a dict. During this process, we do not have any calls to
dictResumeRehashing, which causes the dict's pauserehash to always be > 0.

And at last, it will be returned directly in dictRehashMilliseconds, which causes
us to have slot dict in a state where rehash cannot be completed.

In the "expire scan should skip dictionaries with lot's of empty buckets" test,
adding a `keys *` can reproduce the problem stably. `keys *` will call dbIteratorNext
to trigger a traversal of all slot dicts.

Added dbReleaseIterator and dbIteratorInitNextSafeIterator methods to call dictResetIterator.
Issue was introduced in #11695.
2023-11-14 14:28:46 +02:00
Yossi GottliebandGitHub a9e73c00bc Reduce FreeBSD daily scope. (#12758)
The full test is very flaky running on a VM inside GitHub worker, so we
have to settle for only building and running a small smoke test.
2023-11-13 17:22:09 +02:00
Roshan KhatriandGitHub 88e83e517b Add DEBUG_ASSERTIONS option to custom assert (#12667)
This PR introduces a new macro, serverAssertWithInfoDebug, to do complex assertions only for debugging. The main intention is to allow running complex operations during tests without impacting runtime performance. This assertion is enabled when setting DEBUG_ASSERTIONS.

The DEBUG_ASSERTIONS flag is set for the daily and CI variants of `test-sanitizer-address`.
2023-11-11 20:31:34 -08:00
Harkrishn PatroandGitHub 9ca8490315 Increase timeout for expiry cluster tests (#12752)
Test recently added fails on timeout in valgrind in GH actions.

Locally with valgrind the test finishes within 1.5 sec(s). Couldn't find
any issue due to lack of reproducibility. Increasing the timeout and
adding an additional log to the test to understand how many keys
were left at the end.
2023-11-11 12:01:04 +02:00
zhaozhao.zzandGitHub 6258edebf0 reset bucket_count when empty db (#12750)
Introduced in #12697 , should reset bucket_count when empty db, or the overhead memory usage of db can be miscalculated.
2023-11-10 15:52:57 +02:00
zhaozhao.zzandGitHub cf6ed3feeb fix the wrong judgement for activerehashing in standalone mode (#12741)
Introduced by #11695, the judgement should be dictIsRehashing.
2023-11-09 11:30:50 +02:00
BinbinandGitHub 53294e537c Fix genClusterDebugString minor sds leaks (#12739)
This function now will only be called in printCrashReport,
so this is just a cleanup.
2023-11-08 19:14:36 +02:00
Meir Shpilraien (Spielrein)andGitHub 0ffb9d2ea9 Before evicted and before expired server events are not executed inside an execution unit. (#12733)
Redis 7.2 (#9406) introduced a new modules event, `RedisModuleEvent_Key`.
This new event allows the module to read the key data just before it is removed
from the database (either deleted, expired, evicted, or overwritten).

When the key is removed from the database, either by active expire or eviction.
The new event was not called as part of an execution unit. This can cause an
issue if the module registers a post notification job inside the event. This job will
not be executed atomically with the expiration/eviction operation and will not
replicated inside a Multi/Exec. Moreover, the post notification job will be executed
right after the event where it is still not safe to perform any write operation, this will
violate the promise that post notification job will be called atomically with the
operation that triggered it and **only when it is safe to write**.

This PR fixes the issue by wrapping each expiration/eviction of a key with an execution
unit. This makes sure the entire operation will run atomically and all the post notification
jobs will be executed at the end where it is safe to write.

Tests were modified to verify the fix.
2023-11-08 09:28:22 +02:00
Yossi GottliebandGitHub 6223355cf3 Use cross-platform-actions for FreeBSD support. (#12732)
This change overcomes many stability issues experienced with the
vmactions action.

We need to limit VMs to 8GB for better stability, as the 13GB default
seems to hang them occasionally.

Shell code has been simplified since this action seem to use `bash -e`
which will abort on non-zero exit codes anyway.
2023-11-06 18:07:14 +02:00
dingruiandGitHub a888503b4f Remove unnecessary argument(tp) in gettimeofday() call for retrieving timezone (#12722)
changes the `gettimeofday` caller, by removing an unused optional output argument.

It would take 2 benefits:

- simplify code, discard unnecessary arg.
- possibly faster due to the implementation in kernel.
2023-11-06 15:10:09 +02:00
282b82e9d2 Handle all CLUSTER_REDIR_ error code when verifying script. (#12707)
Clarify the errors related to the cluster mode in the script, return the command that encountered an execution error along with the specific error message.

---------

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2023-11-06 17:48:58 +08:00
Wen HuiandGitHub 28b6155ba5 Fix the bug that write redis sensitive command information to redis_cli historyfile (#11489)
Currently, we do not write the following sensitive commands into the ~/.rediscli_history file:

ACL SETUSER username [rule [rule ...]]
AUTH [username] password
HELLO [AUTH username password] 
MIGRATE host port <key | ""> destination-db timeout [[AUTH password | AUTH2 username password]]
CONFIG SET masterauth master-password
CONFIG SET masteruser username
CONFIG SET requirepass foobared

However, we still write the following sensitive commands into the ~/.rediscli_history file:
ACL GETUSER username
Sentinel CONFIG set sentinel-pass password
Sentinel CONFIG set sentinel-user username
Sentinel set mastername auth-pass password
Sentinel set mastername auth-user username

This change adds the commands of the second list to be skipped from being written to the history file.
2023-11-05 14:20:15 +02:00
15a048d4f0 re-enable defrag tests in cluster mode. (#12710)
Reverts the skipping defrag tests in cluster mode (done in #12672.
instead it skips only some defrag tests that are relevant for cluster modes.
The test now run well after investigating and making the changes in #12674 and #12694.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-11-02 13:55:48 +02:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
0ce74872c4 Bump actions/setup-node from 3 to 4 (#12708)
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 3 to 4.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-02 12:22:50 +02:00
Viktor SöderqvistandGitHub 8878817d89 Optimize SCAN with MATCH when pattern implies cluster slot (#12536)
Optimize the performance of SCAN commands when a match pattern can only contain keys from a 
single slot in cluster mode. This can happen when the pattern contains a hash tag before any 
wildcard matchers or when the key contains no matchers.
2023-11-01 00:06:49 -07:00
Chen TianjieandGitHub e9f312e087 Change stat_client_qbuf_limit_disconnections to atomic. (#12711)
In #12476 server.stat_client_qbuf_limit_disconnections was added. It is written in readQueryFromClient, which may be called by multiple threads when io-threads and io-threads-do-reads are turned on. Somehow we missed to make it an atomic variable.
2023-11-01 10:57:24 +08:00
Viktor SöderqvistandGitHub 8d675950e6 Don't crash when adding a forgotten node to blacklist twice (#12702)
Add a defensive checks to prevent double freeing a node from the cluster blacklist.
2023-10-31 07:20:06 -07:00
4bbb2b0152 Optimize CPU cache efficiency on dict while it's being rehashed (#5692)
when find a key  ,if redis is rehashing, currently we should lookup both tables (ht[0] and ht[1]).
if we use the key's index comparing to the rehashidx,if index < rehashidx,then  we can conclude: 
1. it is rehashing(rehashidx is -1 if it is not rehashing) 
2. we can't find key in ht[0] so just continue to find key in ht[1]

The possible performance gain here, is not the looping over the linked list (which is empty),
but rather the lookup in the table (which could be a cache miss).
---------

Co-authored-by: zhangshihua003 <zhangshihua003@ke.com>
Co-authored-by: sundb <sundbcn@gmail.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: judeng <abc3844@126.com>
2023-10-31 09:57:26 +02:00
Roshan KhatriandGitHub f7fa481156 Optimize finding the slot for a given key count in a fenwick tree (#12704)
This PR optimizes the time complexity of findSlotByKeyIndex from O(log^2(N)) to O(log(N)) by using the 
tree structure of binary index tree to find a slot in one search of the index.
2023-10-27 17:15:19 -07:00
4145d628b4 Reduce dbBuckets operation time complexity from O(N) to O(1) (#12697)
As part of #11695 independent dictionaries were introduced per slot.
Time complexity to discover total no. of buckets across all dictionaries
increased to O(N) with straightforward implementation of iterating over
all dictionaries and adding dictBuckets of each.

To optimize the time complexity, we could maintain a global counter at
db level to keep track of the count of buckets and update it on the start
and end of rehashing.

---------

Co-authored-by: Roshan Khatri <rvkhatri@amazon.com>
2023-10-27 22:05:40 +03:00
Roshan KhatriandGitHub 7d68208a6e Reset later item flag after defrag later is done (#12694)
Fixing issues described in #12672, started after #11695
Related to #12674

Fixes the `defrag didn't stop' issue.

In some cases of how the keys were stored in memory
defrag_later_item_in_progress was not getting reset once we finish
defragging the later items and we move to the next slot. This stopped
the scan to happen in the later slots and did not get
2023-10-27 13:56:15 +03:00
Oran AgraandGitHub ba900f6cb8 Fix fd leak causing deleted files to remain open and eat disk space (#12693)
This was introduced in v7.2 by #11248
2023-10-25 20:54:02 +03:00
372ea21875 Update comment around propagateDeletion (#12687)
Fix some outdated comments and add comment for moduleNotifyKeyspaceEvent
we added in #11084 since it seems a bit implicit.

---------

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-10-24 13:10:03 +03:00
Harkrishn PatroandGitHub 3fac869f02 Fix test, disable expiration until empty buckets are formed (#12689)
Test failure on freebsd CI:
```
*** [err]: expire scan should skip dictionaries with lot's of empty buckets in tests/unit/expire.tcl
  scan didn't handle slot skipping logic.
```

Observation:

expiry of keys might happen before the empty buckets are formed and won't help with the expiry skip logic validation.

Solution:

Disable expiration until the empty buckets are formed.
2023-10-24 11:29:40 +03:00
Harkrishn PatroandGitHub 26eb4ce397 Fix defrag test (#12674)
Fixing issues started after #11695 when the defrag tests are being executed in cluster mode too.
For some reason, it looks like the defragmentation is over too quickly, before the test is able to
detect that it's running.
so now instead of waiting to see that it's active, we wait to see that it did some work
```
[err]: Active defrag big list: cluster in tests/unit/memefficiency.tcl
defrag not started.
[err]: Active defrag big keys: cluster in tests/unit/memefficiency.tcl
defrag didn't stop.
```
2023-10-22 11:56:45 +03:00
Harkrishn PatroandGitHub becd50d0da Disable flaky defrag tests affecting daily run (#12672)
Temporarily disabling few of the defrag tests in cluster mode to make the daily run stable:

Active defrag eval scripts
Active defrag big keys
Active defrag big list
Active defrag edge case
2023-10-19 21:12:58 +03:00
Harkrishn PatroandGitHub f3bf8485d8 Fix resize hash table dictionary iterator (#12660)
Dictionary iterator logic in the `tryResizeHashTables` method is picking the next
(incorrect) dictionary while the cursor is at a given slot. This could lead to some
dictionary/slot getting skipped from resizing.

Also stabilize the test.

problem introduced recently in #11695
2023-10-19 13:58:32 +03:00
03345ddc7f Fix issue of listen before chmod on Unix sockets (CVE-2023-45145) (#12671)
Before this commit, Unix socket setup performed chmod(2) on the socket
file after calling listen(2). Depending on what umask is used, this
could leave the file with the wrong permissions for a short period of
time. As a result, another process could exploit this race condition and
establish a connection that would otherwise not be possible.

We now make sure the socket permissions are set up prior to calling
listen(2).

(cherry picked from commit 1119ecae6f)

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
2023-10-18 14:00:00 +03:00
sundbandGitHub 3c734b8e9d Add new compilation CI for macos-11 and macos-13 (#12666)
As discussed in #12611
Add a build CI for macox 11 and 13 to avoid compatibility breakage introduced by future macos sdk versions.
2023-10-18 13:25:52 +03:00
meiravgriandGitHub d27c7413a9 remove heap allocations from signal handlers. (#12655)
Using heap allocation during signal handlers is unsafe.
This PR purpose is to replace all the heap allocations done within the signal
handlers raised upon server crash and assertions.
These were added in #12453.

writeStacktraces(): allocates the stacktraces output array on the calling thread's
stack and assigns the address to a global variable.
It calls `ThreadsManager_runOnThreads()` that invokes `collect_stacktrace_data()`
by each thread: each thread writes to a different location in the above array to allow
sync writes.

get_ready_to_signal_threads_tids(): instead of allocating the `tids` array, it receives it
as a fixed size array parameter, allocated on on the stack of the calling function, and
returns the number of valid threads. The array size is hard-coded to 50.

`ThreadsManager_runOnThreads():` To avoid the outputs array allocation, the
**callback signature** was changed. Now it should return void. This function return type
has also changed to int - returns 1 if successful, and 0 otherwise.

Other unsafe calls will be handled in following PRs
2023-10-16 17:21:49 +03:00
0270abda82 Replace cluster metadata with slot specific dictionaries (#11695)
This is an implementation of https://github.com/redis/redis/issues/10589 that eliminates 16 bytes per entry in cluster mode, that are currently used to create a linked list between entries in the same slot.  Main idea is splitting main dictionary into 16k smaller dictionaries (one per slot), so we can perform all slot specific operations, such as iteration, without any additional info in the `dictEntry`. For Redis cluster, the expectation is that there will be a larger number of keys, so the fixed overhead of 16k dictionaries will be The expire dictionary is also split up so that each slot is logically decoupled, so that in subsequent revisions we will be able to atomically flush a slot of data.

## Important changes
* Incremental rehashing - one big change here is that it's not one, but rather up to 16k dictionaries that can be rehashing at the same time, in order to keep track of them, we introduce a separate queue for dictionaries that are rehashing. Also instead of rehashing a single dictionary, cron job will now try to rehash as many as it can in 1ms.
* getRandomKey - now needs to not only select a random key, from the random bucket, but also needs to select a random dictionary. Fairness is a major concern here, as it's possible that keys can be unevenly distributed across the slots. In order to address this search we introduced binary index tree). With that data structure we are able to efficiently find a random slot using binary search in O(log^2(slot count)) time.
* Iteration efficiency - when iterating dictionary with a lot of empty slots, we want to skip them efficiently. We can do this using same binary index that is used for random key selection, this index allows us to find a slot for a specific key index. For example if there are 10 keys in the slot 0, then we can quickly find a slot that contains 11th key using binary search on top of the binary index tree.
* scan API - in order to perform a scan across the entire DB, the cursor now needs to not only save position within the dictionary but also the slot id. In this change we append slot id into LSB of the cursor so it can be passed around between client and the server. This has interesting side effect, now you'll be able to start scanning specific slot by simply providing slot id as a cursor value. The plan is to not document this as defined behavior, however. It's also worth nothing the SCAN API is now technically incompatible with previous versions, although practically we don't believe it's an issue.
* Checksum calculation optimizations - During command execution, we know that all of the keys are from the same slot (outside of a few notable exceptions such as cross slot scripts and modules). We don't want to compute the checksum multiple multiple times, hence we are relying on cached slot id in the client during the command executions. All operations that access random keys, either should pass in the known slot or recompute the slot. 
* Slot info in RDB - in order to resize individual dictionaries correctly, while loading RDB, it's not enough to know total number of keys (of course we could approximate number of keys per slot, but it won't be precise). To address this issue, we've added additional metadata into RDB that contains number of keys in each slot, which can be used as a hint during loading.
* DB size - besides `DBSIZE` API, we need to know size of the DB in many places want, in order to avoid scanning all dictionaries and summing up their sizes in a loop, we've introduced a new field into `redisDb` that keeps track of `key_count`. This way we can keep DBSIZE operation O(1). This is also kept for O(1) expires computation as well.

## Performance
This change improves SET performance in cluster mode by ~5%, most of the gains come from us not having to maintain linked lists for keys in slot, non-cluster mode has same performance. For workloads that rely on evictions, the performance is similar because of the extra overhead for finding keys to evict. 

RDB loading performance is slightly reduced, as the slot of each key needs to be computed during the load.

## Interface changes
* Removed `overhead.hashtable.slot-to-keys` to `MEMORY STATS`
* Scan API will now require 64 bits to store the cursor, even on 32 bit systems, as the slot information will be stored.
* New RDB version to support the new op code for SLOT information. 

---------

Co-authored-by: Vitaly Arbuzov <arvit@amazon.com>
Co-authored-by: Harkrishn Patro <harkrisp@amazon.com>
Co-authored-by: Roshan Khatri <rvkhatri@amazon.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2023-10-14 23:58:26 -07:00
Oran AgraandGitHub f0c1c730d4 test suite: clean server pids after server crashed (#12639)
when a server in the test suite crashes and is restarted by redstart_server, we didn't clean it's pid from the list.
we can see that when the corrupt-dump-fuzzer hangs, it has a long list of servers to lean, but in fact they're all already dead.
2023-10-13 16:28:52 +03:00
Harkrishn PatroandGitHub b784c5375e Unsubscribe all clients from replica for shard channel if the master ownership changes (#12577)
Unsubscribe all clients from replica for shard channel if the master ownership changes
2023-10-12 20:48:27 -07:00
Ye Lin AungandGitHub b705049a7a Replace emptyDb() with new emptyData() (#12646)
The function was renamed, but the comments were outdated.
2023-10-12 15:34:08 +03:00
zhaozhao.zzandGitHub 77a65e82b2 support XREAD[GROUP] with BLOCK option in scripts (#12596)
In #11568 we removed the NOSCRIPT flag from commands and keep the BLOCKING flag.
Aiming to allow them in scripts and let them implicitly behave in the non-blocking way.

In that sense, the old behavior was to allow LPOP and reject BLPOP, and the new behavior,
is to allow BLPOP too, and fail it only in case it ends up blocking.
So likewise, so far we allowed XREAD and rejected XREAD BLOCK, and we will now allow
that too, and only reject it if it ends up blocking.
2023-10-12 10:54:50 +03:00
BinbinandGitHub e5ef161374 Fix crash when running rebalance command in a mixed cluster of 7.0 and 7.2 (#12604)
In #10536, we introduced the assert, some older versions of servers
(like 7.0) doesn't gossip shard_id, so we will not add the node to
cluster->shards, and node->shard_id is filled in randomly and may not
be found here.

It causes that if we add a 7.2 node to a 7.0 cluster and allocate slots
to the 7.2 node, the 7.2 node will crash when it hits this assert. Somehow
like #12538.

In this PR, we remove the assert and replace it with an unconditional removal.
2023-10-11 22:15:25 -07:00
BinbinandGitHub 4de4fcf280 Fix redis-cli pubsub_mode and connect minor prompt / crash issue (#12571)
When entering pubsub mode and using the redis-cli only
connect command, we need to reset pubsub_mode because
we switch to a different connection.

This will affect the prompt when the connection is successful,
and redis-cli will crash when the connect fails:
```
127.0.0.1:6379> subscribe ch
1) "subscribe"
2) "ch"
3) (integer) 1
127.0.0.1:6379(subscribed mode)> connect 127.0.0.1 6380
127.0.0.1:6380(subscribed mode)> ping
PONG
127.0.0.1:6380(subscribed mode)> connect a b
Could not connect to Redis at a:0: Name or service not known
Segmentation fault
```
2023-10-11 10:45:38 +03:00
BinbinandGitHub 8d92f7f2b7 Support NO ONE block in REPLICAOF command json (#12633)
The current commands.json doesn't mention the special NO ONE arguments.
This change is also applied to SLAVEOF
2023-10-10 11:10:40 +03:00
Oran AgraandGitHub b810384c62 dump server longs on hang corrupt dump fuzzer test
recently there are some incidents of hanged tests in the CI
when we try to reproduce them, we get an assertion, not a hang.
maybe the server logs will reveal some info.
2023-10-08 16:19:31 +03:00
JachinandGitHub a2b0701d2c Fix compile on macOS 13 (#12611)
Use the __MAC_OS_X_VERSION_MIN_REQUIRED macro to detect the
macOS system version instead of using MAC_OS_X_VERSION_10_6.

From MacOSX14.0.sdk, the default definitions of MAC_OS_X_VERSION_xxx have
been removed in usr/include/AvailabilityMacros.h. It includes AvailabilityVersions.h,
where the following condition must be met:
`#if (!defined(_POSIX_C_SOURCE) && !defined(_XOPEN_SOURCE)) || defined(_DARWIN_C_SOURCE)`
Only then will MAC_OS_X_VERSION_xxx be defined.
However, in the project, _DARWIN_C_SOURCE is not defined, which leads to the
loss of the definition for MAC_OS_X_VERSION_10_6.
2023-10-08 11:12:50 +03:00
Oran AgraandGitHub fe37e4fc87 Cleanup nested module keyspace notifications (#12630)
Recently we added a way for the module to declare that it wishes to
receive nested KSN, by setting ALLOW_NESTED_KEYSPACE_NOTIFICATIONS.
but it looks like this flow has a bug, clearing the `active` member
when it was previously set. however, since nesting is permitted,
this bug has no implications, since regardless of the active member,
the notification is permitted.
2023-10-05 13:50:17 +03:00
YaacovHazanandGitHub 2cf50ddbad Fix 'load corrupted rdb with no CRC' test (#12629)
After the change in #12626 (2e0f6724e), the is_alive proc gets pid and not
server config.

This PR aligns it in 'load corrupted rdb with no CRC' test.
2023-10-03 11:09:25 +03:00
Madelyn OlsonandGitHub 31c3172d9b Better standardize around assertions (#12539)
We use the C standard assert() in various places in the codebase, which requires NDEBUG to be undefined. We introduced the redisassert.h file in order to allow low level files to access the assert that maps to serverPanic, but this was only applied tactically and is not available broadly.

This PR removes all usage of the standard library asserts and replaces them with an assert that maps to serverPanic. It makes us immune to accidentally setting the NDEBUG flag preventing assertions. I also marked marked the server asserts as "likely" to not execute. I spot checked various points in the code, and it didn't change the code layout on my x86 mac, but it is more consistent with redisassert.h and seems more correct overall.
2023-10-02 18:58:44 -07:00
Madelyn OlsonandGitHub 9d31768cbb Fix a couple of tabs that caused misindentation (#12541)
Fixed some usages of tabs which caused weird indentation in the code. Tried to find all of the places so their was one PR. I ignored all of the usages of tabs which don't really affect readability.
2023-10-02 16:44:09 -07:00
4ba9e18ef0 fix crash in crash-report and other improvements (#12623)
## Crash fix
### Current behavior
We might crash if we fail to collect some of the threads' output. If it exceeds timeout for example.

The threads mngr API guarantees that the output array length will be `tids_len`, however, some
indices can be NULL, in case it fails to collect some of the threads' outputs.

When we use the threads mngr to collect the threads' stacktraces, we rely on this and skip NULL
entries. Since the output array was allocated with malloc, instead of NULL, it contained garbage,
so we got a segmentation fault when trying to read this garbage. (in debug.c:writeStacktraces() )

### fix
Allocate the global output array with zcalloc.

### To reproduce the bug, you'll have to change the code:
**in threadsmngr:ThreadsManager_runOnThreads():**
make sure the g_output_array allocation is initialized with garbage and not 0s 
(add `memset(g_output_array, 2, sizeof(void*) * tids_len);` below the allocation).

Force one of the threads to write to the array:
add a global var: `static redisAtomic size_t return_now = 0;` 
add to `invoke_callback()` before writing to the output array:
```
    size_t i_return;
    atomicGetIncr(return_now, i_return, 1);
    if(i_return == 1) return;
```
compile, start the server with `--enable-debug-command local` and run `redis-cli debug assert`
The assertion triggers the the stacktrace collection. 
Expect to get 2 prints of the stack trace - since we get the segmentation fault after we return from
the threads mngr, it can be safely triggered again.

## Added global variables r/w lock in ThreadsManager
To avoid a situation where the main thread runs `ThreadsManager_cleanups` while threads are still
invoking the signal handler, we use a r/w lock.
For cleanups, we will acquire the write lock.
The threads will acquire the read lock to enable them to write simultaneously.
If we fail to acquire the read lock, it means cleanups are in progress and we return immediately.
After acquiring the lock we can safely check that the global output array wasn't nullified and proceed
to write to it.
This way we ensure the threads are not modifying the global variables/ trying to write to the output
array after they were zeroed/nullified/destroyed(the semaphore).

## other minor logging change
1. removed logging if the semaphore times out because the threads can still write to the output array
  after this check. Instead, we print the total number of printed stacktraces compared to the exacted
  number (len_tids).
2. use noinline attribute to make sure the uplevel number of ignored stack trace entries stays correct.
3. improve testing

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-10-02 20:02:02 +03:00
YaacovHazanandGitHub 2e0f6724e0 Stabilization and improvements around aof tests (#12626)
In some tests, the code manually searches for a log message, and it
uses tail -1 with a delay of 1 second, which can miss the expected line.

Also, because the aof tests use start_server_aof and not start_server,
the test name doesn't log into the server log.

To fix the above, I made the following changes:
- Change the start_server_aof to wrap the start_server.
  This will add the created aof server to the servers list, and make
  srv() and wait_for_log_messages() available for the tests.

- Introduce a new option for start_server.
  'wait_ready' - an option to let the caller start the test code without
  waiting for the server to be ready. useful for tests on a server that
  is expected to exit on startup.

- Create a new start_server_aof_ex.
  The new proc also accept options as argument and make use of the
  new 'short_life' option for tests that are expected to exit on startup
  because of some error in the aof file(s).

Because of the above, I had to change many lines and replace every
local srv variable (a server config) usage with the srv().
2023-10-02 08:20:53 +03:00
guybe7andGitHub c2a4b78491 WAITAOF: Update fsynced_reploff_pending even if there's nothing to fsync (#12622)
The problem is that WAITAOF could have hang in case commands were
propagated only to replicas.
This can happen if a module uses RM_Call with the REDISMODULE_ARGV_NO_AOF flag.
In that case, master_repl_offset would increase, but there would be nothing to fsync, so
in the absence of other traffic, fsynced_reploff_pending would stay the static, and WAITAOF can hang.

This commit updates fsynced_reploff_pending to the latest offset in flushAppendOnlyFile in case
there's nothing to fsync. i.e. in case it's behind because of the above mentions case it'll be refreshed
and release the WAITAOF.

Other changes:
Fix a race in wait.tcl (client getting blocked vs. the fsync thread)
2023-09-28 17:19:20 +03:00
guybe7andGitHub bfa3931a04 WAITAOF: Update fsynced_reploff_pending just before starting the initial AOFRW fork (#12620)
If we set `fsynced_reploff_pending` in `startAppendOnly`, and the fork doesn't start
immediately (e.g. there's another fork active at the time), any subsequent commands
will increment `server.master_repl_offset`, but will not cause a fsync (given they were
executed before the fork started, they just ended up in the RDB part of it)
Therefore, any WAITAOF will wait on the new master_repl_offset, but it will time out
because no fsync will be executed.

Release notes:
```
WAITAOF could timeout in the absence of write traffic in case a new AOF is created and
an AOFRW can't immediately start.
This can happen by the appendonly config is changed at runtime, but also after FLUSHALL,
and replica full sync.
```
2023-09-28 17:05:53 +03:00
f924bebd83 Rewrite huge printf calls to smaller ones for readability (#12257)
In a long printf call with many placeholders, it's hard to see which argument
belongs to which placeholder.

The long printf-like calls in the INFO and CLIENT commands are rewritten into
pairs of (format, argument). These pairs are then rewritten to a single call with
a long format string and a long list of arguments, using a macro called FMTARGS.

The file `fmtargs.h` is added to the repo.

Co-authored-by: Madelyn Olson <34459052+madolson@users.noreply.github.com>
2023-09-28 09:21:23 +03:00
BinbinandGitHub 9fe63bdc80 Dump server logs when corrupt fuzzer reports crash (#12612)
Recently we found some signal crashes, but unable to reproduce them.
It is a good idea to dump the server logs when a failure happens.
2023-09-27 09:08:18 +03:00
SankarandGitHub 8cdeddc81c Clear owner_not_claiming_slot bit for the slot in clusterDelSlot (#12564)
Clear owner_not_claiming_slot bit for the slot in clusterDelSlot to keep it
consistent with slot ownership information.
2023-09-26 14:03:27 -07:00
Nir RattnerandGitHub 24187ed8e3 Fix overflow calculation for next timer event (#12474)
The `retval` variable is defined as an `int`, so with 4 bytes, it cannot properly represent
microsecond values greater than the equivalent of about 35 minutes. 

This bug shouldn't impact standard Redis behavior because Redis doesn't have timer
events that are scheduled as far as 35 minutes out, but it may affect custom Redis modules
which interact with the event timers via the RM_CreateTimer API.

The impact is that `usUntilEarliestTimer` may return 0 for as long as `retval` is scaled to
an overflowing value. While `usUntilEarliestTimer` continues to return `0`, `aeApiPoll`
will have a zero timeout, and so Redis will use significantly more CPU iterating through
its event loop without pause. For timers scheduled far enough into the future, Redis will
cycle between ~35 minute periods of high CPU usage and ~35 minute periods of standard
CPU usage.
2023-09-24 13:31:12 +03:00
meiravgriandGitHub cc2be63997 Print stack trace from all threads in crash report (#12453)
In this PR we are adding the functionality to collect all the process's threads' backtraces.

## Changes made in this PR

### **introduce threads mngr API**
The **threads mngr API** which has 2 abilities:
* `ThreadsManager_init() `- register to SIGUSR2. called on the server start-up.
* ` ThreadsManager_runOnThreads()` - receives a list of a pid_t and a callback, tells every
  thread in the list to invoke the callback, and returns the output collected by each invocation.
**Elaborating atomicvar API**
* `atomicIncrGet(var,newvalue_var,count) `-- Increment and get the atomic counter new value
* `atomicFlagGetSet` -- Get and set the atomic counter value to 1

### **Always set SIGALRM handler**
SIGALRM handler prints the process's stacktrace to the log file. Up until now, it was set only if the
`server.watchdog_period` > 0. This can be also useful if debugging is needed. However, in situations
where the server can't get requests, (a deadlock, for example) we weren't able to change the signal handler.
To make it available at run time we set SIGALRM handler on server startup. The signal handler name was
changed to a more general `sigalrmSignalHandler`.

### **Print all the process' threads' stacktraces**

`logStackTrace()` now calls `writeStacktraces()`, instead of logging the current thread stacktrace.
`writeStacktraces()`:
* On Linux systems we use the threads manager API to collect the backtraces of all the process' threads.
  To get the `tids` list (threads ids) we read the `/proc/<redis-server-pid>/tasks` file which includes a list of directories.
  Each directory name corresponds to one tid (including the main thread). For each thread, we also need to check if it
  can get the signal from the threads manager (meaning it is not blocking/ignoring that signal). We send the threads
  manager this tids list and `collect_stacktrace_data()` callback, which collects the thread's backtrace addresses,
  its name, and tid.
* On other systems, the behavior remained as it was (writing only the current thread stacktrace to the log file).

## compatibility notes
1. **The threads mngr API is only supported in linux.** 
2. glibc earlier than 2.3 We use `syscall(SYS_gettid)` and `syscall(SYS_tgkill...)` because their dedicated
  alternatives (`gettid()` and `tgkill`) were added in glibc 2.3.

## Output example

Each thread backtrace will have the following format:
`<tid> <thread_name> [additional_info]`
* **tid**: as read from the `/proc/<redis-server-pid>/tasks` file
* **thread_name**: the tread name as it is registered in the os/
* **additional_info**: Sometimes we want to add specific information about one of the threads. currently.
  it is only used to mark the thread that handles the backtraces collection by adding "*".
  In case of crash - this also indicates which thread caused the crash. The handling thread in won't
  necessarily appear first.

```
------ STACK TRACE ------
EIP:
/lib/aarch64-linux-gnu/libc.so.6(epoll_pwait+0x9c)[0xffffb9295ebc]

67089 redis-server *
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffffb9437790]
/lib/aarch64-linux-gnu/libc.so.6(epoll_pwait+0x9c)[0xffffb9295ebc]
redis-server *:6379(+0x75e0c)[0xaaaac2fe5e0c]
redis-server *:6379(aeProcessEvents+0x18c)[0xaaaac2fe6c00]
redis-server *:6379(aeMain+0x24)[0xaaaac2fe7038]
redis-server *:6379(main+0xe0c)[0xaaaac3001afc]
/lib/aarch64-linux-gnu/libc.so.6(+0x273fc)[0xffffb91d73fc]
/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98)[0xffffb91d74cc]
redis-server *:6379(_start+0x30)[0xaaaac2fe0370]

67093 bio_lazy_free
/lib/aarch64-linux-gnu/libc.so.6(+0x79dfc)[0xffffb9229dfc]
/lib/aarch64-linux-gnu/libc.so.6(pthread_cond_wait+0x208)[0xffffb922c8fc]
redis-server *:6379(bioProcessBackgroundJobs+0x174)[0xaaaac30976e8]
/lib/aarch64-linux-gnu/libc.so.6(+0x7d5c8)[0xffffb922d5c8]
/lib/aarch64-linux-gnu/libc.so.6(+0xe5d1c)[0xffffb9295d1c]

67091 bio_close_file
/lib/aarch64-linux-gnu/libc.so.6(+0x79dfc)[0xffffb9229dfc]
/lib/aarch64-linux-gnu/libc.so.6(pthread_cond_wait+0x208)[0xffffb922c8fc]
redis-server *:6379(bioProcessBackgroundJobs+0x174)[0xaaaac30976e8]
/lib/aarch64-linux-gnu/libc.so.6(+0x7d5c8)[0xffffb922d5c8]
/lib/aarch64-linux-gnu/libc.so.6(+0xe5d1c)[0xffffb9295d1c]

67092 bio_aof
/lib/aarch64-linux-gnu/libc.so.6(+0x79dfc)[0xffffb9229dfc]
/lib/aarch64-linux-gnu/libc.so.6(pthread_cond_wait+0x208)[0xffffb922c8fc]
redis-server *:6379(bioProcessBackgroundJobs+0x174)[0xaaaac30976e8]
/lib/aarch64-linux-gnu/libc.so.6(+0x7d5c8)[0xffffb922d5c8]
/lib/aarch64-linux-gnu/libc.so.6(+0xe5d1c)[0xffffb9295d1c]
67089:signal-handler (1693824528) --------
```
2023-09-24 09:47:23 +03:00
Chen TianjieandGitHub 2aad03fa39 Use server.current_client to decide whether cluster commands should return TLS info. (#12569)
Starting a change in #12233 (released in 7.2), CLUSTER commands use client's
connection to decide whether to return TLS port or non-TLS port, but commands
called by Lua script and module's RM_Call don't have a real client with connection,
and would currently be regarded as non-TLS connections.

We can use server.current_client instead when it is available. When it is not (module calls
commands without a real client), we may see this as an undefined behavior, and return null
or default port (currently in this PR it returns default port, judged by server.tls_cluster).
2023-09-21 18:41:32 +03:00
BinbinandGitHub 4031a18732 Fix that slot return in CLUSTER SHARDS should be integer (#12561)
An unintentional change was introduced in #10536, we used
to use addReplyLongLong and now it is addReplyBulkLonglong, 
revert it back the previous behavior.
2023-09-09 23:33:00 -07:00
BinbinandGitHub 96e9dec419 Bump codespell from 2.2.4 to 2.2.5 (#12557)
and adjustments.
2023-09-08 16:10:17 +03:00
90e9fc387c Update command tips on more admin / configuration commands (#12545)
Updated the command tips for ACL SAVE / SETUSER / DELUSER, CLIENT SETNAME / SETINFO, and LATENCY RESET.
The tips now match CONFIG SET, since there's a similar behavior for all of these commands - the
user expects to update the various configurations & states on all nodes, not only on a single, random node.
For LATENCY RESET the response tip is now agg_sum.

Co-authored-by: Shachar Langbeheim <shachlan@amazon.com>
2023-09-04 21:30:42 +03:00
secwallandGitHub a2046c1eb1 Check shard_id pointer validity in updateShardId (#12538)
When connecting between a 7.0 and 7.2 cluster, the 7.0 cluster will not populate the shard_id field, which is expect on the 7.2 cluster. This is not intended behavior, as the 7.2 cluster is supposed to use a temporary shard_id while the node is in the upgrading state, but it wasn't being correctly set in this case.
2023-09-02 20:14:48 -07:00
044e29dd34 redis-benchmark - add the support for binary strings (#9414)
Recently, the option of sending an argument from stdin using `-x` flag
was added to redis-benchmark (this option is available in redis-cli as well).
However, using the `-x` option for sending a blobs that contains null-characters
doesn't work as expected - the argument is trimmed in the first occurrence of
`\X00` (unlike in redis-cli).  
This PR aims to fix this issue and add the support for every binary string input,
by sending arguments length to `redisFormatCommandArgv` when processing
redis-benchmark command, so we won't treat the arguments as C-strings.

Additionally, we add a simple test coverage for `-x` (without binary strings, and
also remove an excessive server started in tests, and make sure to select db 0
so that `r` and the benchmark work on the same db.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-09-02 15:37:04 +03:00
BinbinandGitHub 4ba144a4eb Add logreqres:skip flag to new INFO obuf limit test (#12537)
The new test added in #12476 causes reply-schemas-validator to fail.
When doing `catch {r get key}`, the req-res output is:
```
3
get
3
key
12
__argv_end__
$100000
aaaaaaaaaaaaaaaaaaaa...4
info
5
stats
12
__argv_end__
=1670
txt:# Stats
...
```

And we can see the link after `$100000`, there is a 4 in the last,
it break the req-res-log-validator script since the format is wrong.

The reason i guess is after the client reconnection (after the output
buf limit), we will not add newlines, but append args directly.
Since obuf-limits.tcl is doing the same thing, and it had the logreqres:skip
flag, so this PR is following it.
2023-09-01 14:15:11 +03:00
Roshan KhatriandGitHub 49f7d173b4 Remove unnecessary use of sds and mem copy in module.c (#12533)
Found that in moduleConfigValidityCheck and isModuleConfigNameRegistered, sds is not required. This also allowed to remove unnecessary memcopy from some of the config registering APIs.
2023-08-31 14:08:05 -07:00
icy17andGitHub 370d38016f Fix potential crash on failed OpenSSL init (#12447) 2023-08-31 22:45:36 +03:00
Chen TianjieandGitHub b26e8e3213 Optimize ZRANGE offset location from linear search to skiplist jump. (#12450)
ZRANGE BYSCORE/BYLEX with [LIMIT offset count] option was
using every level in skiplist to jump to the first/last node in range,
but only use level[0] in skiplist to locate the node at offset, resulting
in sub-optimal performance using LIMIT:
```
while (ln && offset--) {
    if (reverse) {
        ln = ln->backward;
    } else {
        ln = ln->level[0].forward;
    }
}
```
It could be slow when offset is very big. We can get the total rank of
the offset location and use skiplist to jump to it. It is an improvement
from O(offset) to O(log rank).

Below shows how this is implemented (if the offset is positve):

Use the skiplist to seach for the first element in the range, record its
rank `rank_0`, so we can have the rank of the target node `rank_t`.
Meanwhile we record the last node we visited which has zsl->level-1
levels and its rank `rank_1`. Then we start from the zsl->level-1 node,
use skiplist to go forward `rank_t-rank_1` nodes to reach the target node.

It is very similiar when the offset is reversed.

Note that if `rank_t` is very close to `rank_0`, we just start from the first
element in range and go node by node, this for the case when zsl->level-1
node is to far away and it is quicker to reach the target node by node.

Here is a test using a random generated zset including 10000 elements
(with different positive scores), doing a bench mark which compares how
fast the `ZRANGE` command is exucuted before and after the optimization. 

The start score is set to 0 and the count is set to 1 to make sure that
most of the time is spent on locating the offset.
```
memtier_benchmark -h 127.0.0.1 -p 6379 --command="zrange test 0 +inf byscore limit <offset> 1"
```
| offset | QPS(unstable) | QPS(optimized) |
|--------|--------|--------|
| 10 | 73386.02 | 74819.82 |
| 1000 | 48084.96 | 73177.73 |
| 2000 | 31156.79 | 72805.83 |
| 5000 | 10954.83 | 71218.21 |

With the result above, we can see that the original code is greatly
slowed down when offset gets bigger, and with the optimization the
speed is almost not affected.

Similiar results are generated when testing reversed offset:
```
memtier_benchmark -h 127.0.0.1 -p 6379 --command="zrange test +inf 0 byscore rev limit <offset> 1"
```
| offset | QPS(unstable) | QPS(optimized) |
|--------|--------|--------|
| 10 | 74505.14 | 71653.67 |
| 1000 | 46829.25 | 72842.75 |
| 2000 | 28985.48 | 73669.01 |
| 5000 | 11066.22 | 73963.45 | 

And the same conclusion is drawn from the tests of ZRANGE BYLEX.
2023-08-31 14:42:08 +03:00
BinbinandGitHub 9ce8c54d74 Update sort_ro reply_schema to mention the null reply (#12534)
Also added a test to cover this case, so this can
cover the reply schemas check.
2023-08-31 06:36:35 +03:00
7519960527 Allows modules to declare new ACL categories. (#12486)
This PR adds a new Module API int RM_AddACLCategory(RedisModuleCtx *ctx, const char *category_name) to add a new ACL command category.

Here, we initialize the ACLCommandCategories array by allocating space for 64 categories and duplicate the 21 default categories from the predefined array 'ACLDefaultCommandCategories' into the ACLCommandCategories array while ACL initialization. Valid ACL category names can only contain alphanumeric characters, underscores, and dashes.

The API when called, checks for the onload flag, category name validity, and for duplicate category name if present. If the conditions are satisfied, the API adds the new category to the trailing end of the ACLCommandCategories array and assigns the acl_categories flag bit according to the index at which the category is added.

If any error is encountered the errno is set accordingly by the API.

---------

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2023-08-30 13:01:24 -07:00
bodong.ybdandGitHub b59f53efb3 Fix sort_ro get-keys function return wrong key number (#12522)
Before:
```
127.0.0.1:6379> command getkeys sort_ro key
(empty array)
127.0.0.1:6379>
```
After:
```
127.0.0.1:6379> command getkeys sort_ro key
1) "key"
127.0.0.1:6379>
```
2023-08-30 22:00:02 +03:00
Chen TianjieandGitHub e3d4b30d09 Add two stats to count client input and output buffer oom. (#12476)
Add these INFO metrics:
* client_query_buffer_limit_disconnections
* client_output_buffer_limit_disconnections

Sometimes it is useful to monitor whether clients reaches size limit of
query buffer and output buffer, to decide whether we need to adjust the
buffer size limit or reduce client query payload.
2023-08-30 21:51:14 +03:00
4b281ce519 Align CONFIG RESETSTAT/REWRITE tips with SET. (#12530)
Since the three commands have similar behavior (change config, return
OK), the tips that govern how they should behave should be similar.

Co-authored-by: Shachar Langbeheim <shachlan@amazon.com>
2023-08-30 21:49:02 +03:00
BinbinandGitHub e792653753 Add printing for LATENCY related tests (#12514)
This test failed several times:
```
*** [err]: LATENCY GRAPH can output the event graph in tests/unit/latency-monitor.tcl
Expected '478' to be more than or equal to '500' (context: type eval
line 8 cmd {assert_morethan_equal $high 500} proc ::test)
```

Not sure why, adding some verbose printing that'll print the command
result on the next time.
2023-08-27 11:42:55 +03:00
a6eff389b5 redis.conf: Add data loss warning to "appendonly" (#12506)
warning against editing the config file and restarting the server.
which will attempt to load an AOF file and disregard the RDB.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-08-22 18:15:47 +03:00
BinbinandGitHub 1407ac1f3e BITCOUNT and BITPOS with non-existing key and illegal arguments should return error, not 0 (#11734)
BITCOUNT and BITPOS with non-existing key will return 0 even the
arguments are error, before this commit:
```
> flushall
OK
> bitcount s 0
(integer) 0
> bitpos s 0 0 1 hello
(integer) 0

> set s 1
OK
> bitcount s 0
(error) ERR syntax error
> bitpos s 0 0 1 hello
(error) ERR syntax error
```

The reason is that we judged non-existing before parameter checking and
returned. This PR fixes it, and after this commit:
```
> flushall
OK
> bitcount s 0
(error) ERR syntax error
> bitpos s 0 0 1 hello
(error) ERR syntax error
```

Also BITPOS made the same fix as #12394, check for wrong argument, before
checking for key.
```
> lpush mylist a b c
(integer) 3                                                                                    
> bitpos mylist 1 a b
(error) WRONGTYPE Operation against a key holding the wrong kind of value
```
2023-08-21 19:48:30 +03:00
Wen HuiandGitHub 45d3310694 BITCOUNT: check for argument, before checking for key (#12394)
Generally, In any command we first check for  the argument and then check if key exist.

Some of the examples are

```
127.0.0.1:6379> getrange no-key invalid1 invalid2
(error) ERR value is not an integer or out of range
127.0.0.1:6379> setbit no-key 1 invalid
(error) ERR bit is not an integer or out of range
127.0.0.1:6379> xrange no-key invalid1 invalid2
(error) ERR Invalid stream ID specified as stream command argument
```

**Before change** 
```
bitcount no-key invalid1 invalid2
0
```

**After change**
```
bitcount no-key invalid1 invalid2
(error) ERR value is not an integer or out of range
```
2023-08-21 12:53:46 +03:00
BinbinandGitHub c98a28a848 Fix LREM count LONG_MIN overflow minor issue (#12465)
Limit the range of LREM count to -LONG_MAX ~ LONG_MAX.
Before the fix, passing -LONG_MAX would cause an overflow
and would effectively be the same as passing 0. (Because
this condition `toremove && removed == toremove `can never
be satisfied).

This is a minor fix as it shouldn't really affect users,
more like a cleanup.
2023-08-21 12:50:41 +03:00
Yves LeBrasandGitHub 16988208bd config.memkeys init for consistency (#12505)
Initializing `memkeys` to 0 for consistency and clarity.
the config struct is anyway zeroed, but other fields are explicitly initialized.
2023-08-21 08:17:07 +03:00
Wen HuiandGitHub e532c95dfc Added tests for Client commands (#10276)
In our test case, now we missed some test coverage for client sub-commands.
This pr goal is to add some test coverage cases of the following commands:

Client caching
Client kill
Client no-evict
Client pause
Client reply
Client tracking
Client setname

At the very least, this is useful to make sure there are no leaks and crashes in these code paths.
2023-08-20 19:17:51 +03:00
meiravgriandGitHub fe47c2027b Signal handler attributes (#12426)
This PR purpose is to make the crash report process thread safe.
main changes include:

1. `setupSigSegvHandler()` is introduced to initialize the signal handler.
This function first initializes the signal handler mutex (if not initialized yet)
and then registers the process to the signal handler. 

2. **sigsegvHandler** flags :
SA_NODEFER - don't add the signal to the process signal mask. We use this
flag because we want to be able to handle a second call to the signal manually.
removed SA_RESETHAND: this flag resets the signal handler function upon the first
entrance to the registered function. The reason to use this flag is to protect from
recursively entering the signal handler by the same thread. But, it also means
that if a second thread crashes while handling a signal, the process will be
terminated immediately and we won't get the crash report.
In this PR we discard this flag. The signal handler guard described below purpose
is to solve the above issues.

3. Add a **signal handler lock** with ERRORCHECK attributes. 
The lock's purpose is to ensure that only one thread generates a crash report.
Once a second thread enters the signal handler it will be blocked.
We use the ERRORCHECK lock in order to protect from possible deadlock in
case the thread handling the crash gets a signal. In the latest scenario, we log
what we have collected until the handler crashed.

At the end of the crash report we reset the signal handler SIG_DFL, with no flags, and
rethrow the signal to generate a core dump (if enabled) and exit the process.

During the work on this PR we wanted to understand the historical reasons for
how crash is handled.
With respect to the choice of the flag, we believe the **SA_RESETHAND** was not
added for any specific purpose.
**SA_ONSTACK** which is removed here from bugReportEnd(), was originally also
set in the initial registration to signal handler, but removed in 3ada43e73. In addition,
it was removed from another location in deee2c1ef with the following description,
which is also relevant to why it should be removed from bugReportEnd:

> it seems to be some valgrind bug with SA_ONSTACK.
> SA_ONSTACK seems unneeded since WD is not recursive (SA_NODEFER was removed),
> also, not sure if it's even valid without a call to sigaltstack()
2023-08-20 19:16:45 +03:00
BinbinandGitHub 44cc0fcb9d redis-cli --stat take dbnum value from CONFIG GET to output total keys (#12279)
In the past we hardcoded it to 20, causing it to not count keys
for more databases.
2023-08-16 10:54:37 +03:00
Tyler Bream (Event pipeline)andGitHub ac6bc5d1a8 redis-cli: Fix print of keys per cluster host when over int max (#11698)
When running cluster info, and the number of keys overflows the integer
value, the summary no longer makes sense. This fixes by using an appropriate
type to handle values over the max int value.
2023-08-16 10:48:49 +03:00
17904780ae skip the rehashed entries in dictNext (#12386)
If dict is rehashing, the  entries in the head of table[0] is moved to table[1]
and all entries in `table[0][0:rehashidx]` is NULL.

`dictNext` start looking for non-NULL entry from table 0 index 0, and the first call
of `dictNext` on a rehashing dict will Iterate many times to skip those NULL entries.
We can easily skip those entries by setting `iter->index` as `iter->d->rehashidx` when
dict is rehashing and it's the first call of dictNext (`iter->index == -1 && iter->table == 0`).

Co-authored-by: sundb <sundbcn@gmail.com>
2023-08-16 10:45:26 +03:00
Wen HuiandGitHub 965dc90b72 change return type to be consistant (#12479)
Currently rdbSaveMillisecondTime, rdbSaveDoubleValue api's return type is
int but they return the value directly from rdbWriteRaw function which has the
return type of ssize_t. As this may cause overflow to int so changed to ssize_t.
2023-08-16 10:38:59 +03:00
Oran AgraandGitHub 2b8cde71bb Update supported version list. (#12488)
Add 7.2, drop 6.0 as per https://redis.io/docs/about/releases/
Also replace a few concordances of the `’` char, with standard `'`
2023-08-16 08:36:40 +03:00
BinbinandGitHub f4549d1cf4 Fix CLUSTER REPLICAS time complexity, should be O(N) (#12477)
We iterate over all replicas to get the result, the time complexity
should be O(N), like CLUSTER NODES complexity is O(N).
2023-08-14 20:57:55 -07:00
Madelyn OlsonandGitHub 7c179f9bf4 Fixed a bug where sequential matching ACL rules weren't compressed (#12472)
When adding a new ACL rule was added, an attempt was made to remove
any "overlapping" rules. However, there when a match was found, the search
was not resumed at the right location, but instead after the original position of
the original command.

For example, if the current rules were `-config +config|get` and a rule `+config`
was added. It would identify that `-config` was matched, but it would skip over
`+config|get`, leaving the compacted rule `-config +config`. This would be evaluated
safely, but looks weird.

This bug can only be triggered with subcommands, since that is the only way to
have sequential matching rules. Resolves #12470. This is also only present in 7.2.
I think there was also a minor risk of removing another valid rule, since it would start
the search of the next command at an arbitrary point. I couldn't find a valid offset that
would have cause a match using any of the existing commands that have subcommands
with another command.
2023-08-10 09:58:53 +03:00
BinbinandGitHub 6abfda54c3 Fix flaky SENTINEL RESET test (#12437)
After SENTINEL RESET, sometimes the sentinel can
sense the master again, causing the test to fail.
Here we give it a few more chances.
2023-08-10 08:58:52 +03:00
zhaozhao.zzandGitHub 1b6bdff48d optimize the check of kill pubsub clients after modifying ACL rules (#12457)
if there are no subscribers, we can ignore the operation
2023-08-05 10:00:54 +03:00
8226f39fb2 do not call handleClientsBlockedOnKeys inside yielding command (#12459)
Fix the assertion when a busy script (timeout) signal ready keys (like LPUSH),
and then an arbitrary client's `allow-busy` command steps into `handleClientsBlockedOnKeys`
try wake up clients blocked on keys (like BLPOP).

Reproduction process:
1. start a redis with aof
    `./redis-server --appendonly yes`
2. exec blpop
    `127.0.0.1:6379> blpop a 0`
3. use another client call a busy script and this script push the blocked key
    `127.0.0.1:6379> eval "redis.call('lpush','a','b') while(1) do end" 0`
4. user a new client call an allow-busy command like auth
    `127.0.0.1:6379> auth a`

BTW, this issue also break the atomicity of script.

This bug has been around for many years, the old versions only have the
atomic problem, only 7.0/7.2 has the assertion problem.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-08-05 09:52:03 +03:00
sundbandGitHub da9c2804a5 Avoid mostly harmless integer overflow in cjson (#12456)
This PR mainly fixes a possible integer overflow in `json_append_string()`.
When we use `cjson.encoding()` to encode a string larger than 2GB, at specific
compilation flags, an integer overflow may occur leading to truncation, resulting
in the part of the string larger than 2GB not being encoded.
On the other hand, this overflow doesn't cause any read or write out-of-range or segment fault.

1) using -O0 for lua_cjson (`make LUA_DEBUG=yes`)
    In this case, `i` will overflow and leads to truncation.
    When `i` reaches `INT_MAX+1` and overflows to INT_MIN, when compared to
    len, `i` (1000000..00) is expanded to 64 bits signed integer (1111111.....000000) .
    At this point i will be greater than len and jump out of the loop, so `for (i = 0; i < len; i++)`
    will loop up to 2^31 times, and the part of larger than 2GB will be truncated.

```asm
`i` => -0x24(%rbp)
<+253>:   addl   $0x1,-0x24(%rbp)       ; overflow if i large than 2^31
<+257>:   mov    -0x24(%rbp),%eax
<+260>:   movslq %eax,%rdx	            ; move a 32-bit value with sign extension into a 64-bit signed
<+263>:   mov    -0x20(%rbp),%rax
<+267>:   cmp    %rax,%rdx              ; check `i < len`
<+270>:   jb     0x212600 <json_append_string+148>
```
   
2) using -O2/-O3 for lua_cjson (`make LUA_DEBUG=no`, **the default**)
    In this case, because singed integer overflow is an undefined behavior, `i` will not overflow.
   `i` will be optimized by the compiler and use 64-bit registers for all subsequent instructions.

```asm
<+180>:   add    $0x1,%rbx           ; Using 64-bit register `rbx` for i++
<+184>:   lea    0x1(%rdx),%rsi
<+188>:   mov    %rsi,0x10(%rbp)
<+192>:   mov    %al,(%rcx,%rdx,1)
<+195>:   cmp    %rbx,(%rsp)         ; check `i < len`
<+199>:   ja     0x20b63a <json_append_string+154>
```

3) using 32bit
    Because `strbuf_ensure_empty_length()` preallocates memory of length (len * 6 + 2),
    in 32-bit `cjson.encode()` can only handle strings smaller than ((2 ^ 32) - 3 ) / 6.
    So 32bit is not affected.

Also change `i` in `strbuf_append_string()` to `size_t`.
Since its second argument `str` is taken from the `char2escape` string array which is never
larger than 6, so `strbuf_append_string()` is not at risk of overflow (the bug was unreachable).
2023-08-05 07:57:06 +03:00
BinbinandGitHub 7af9f4b36e Fix GEOHASH / GEODIST / GEOPOS time complexity, should be O(1) (#12445)
GEOHASH / GEODIST / GEOPOS use zsetScore to get the score, in skiplist encoding,
we use dictFind to get the score, which is O(1), same as ZSCORE command.
It is not clear why these commands had O(Log(N)), and O(N) until now.
2023-08-05 07:29:24 +03:00
Meir Shpilraien (Spielrein)andGitHub 2ee1bbb53b Ensure that the function load timeout is disabled during loading from RDB/AOF and on replicas. (#12451)
When loading a function from either RDB/AOF or a replica, it is essential not to
fail on timeout errors. The loading time may vary due to various factors, such as
hardware specifications or the system's workload during the loading process.
Once a function has been successfully loaded, it should be allowed to load from
persistence or on replicas without encountering a timeout failure.

To maintain a clear separation between the engine and Redis internals, the
implementation refrains from directly checking the state of Redis within the
engine itself. Instead, the engine receives the desired timeout as part of the
library creation and duly respects this timeout value. If Redis wishes to disable
any timeout, it can simply send a value of 0.
2023-08-02 11:43:31 +03:00
90ab91f00b fix false success and a memory leak for ACL selector with bad parenthesis combination (#12452)
When doing merge selector, we should check whether the merge
has started (i.e., whether open_bracket_start is -1) every time.
Otherwise, encountering an illegal selector pattern could succeed
and also cause memory leaks, for example:

```
acl setuser test1 (+PING (+SELECT (+DEL )
```

The above would leak memory and succeed with only DEL being applied,
and would now error after the fix.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-08-02 10:46:06 +03:00
Diego Lopez RecasandGitHub b653c759cd Fix race condition in tests/unit/auth.tcl (#12444)
Changing the masterauth while turning into a replica is racy.

Turn into replica after changing the masterauth instead.
2023-08-01 18:03:33 +03:00
DarrenJiang13andGitHub 6abb3c4038 change log match to line match in tcl sanitizer_errors_from_file. (#12446)
In the tcl foreach loop, the function should compare line rather than the whole file.
2023-07-30 08:48:29 +03:00
Harkrishn PatroandGitHub 42985b00ea Test coverage for incr/decr operation on robj encoding type optimization (#12435)
Additional test coverage for incr/decr operation.

integer number could be present in raw encoding format due to operation like append. A incr/decr operation following it optimize the string to int encoding format.
2023-07-25 16:43:31 -07:00
zhaozhao.zzandGitHub 01eb939a06 update monitor client's memory and evict correctly (#12420)
A bug introduced in #11657 (7.2 RC1), causes client-eviction (#8687)
and INFO to have inaccurate memory usage metrics of MONITOR clients.

Because the type in `c->type` and the type in `getClientType()` are confusing
(in the later, `CLIENT_TYPE_NORMAL` not `CLIENT_TYPE_SLAVE`), the comment
we wrote in `updateClientMemUsageAndBucket` was wrong, and in fact that function
didn't skip monitor clients.
And since it doesn't skip monitor clients, it was wrong to delete the call for it from
`replicationFeedMonitors` (it wasn't a NOP).
That deletion could mean that the monitor client memory usage is not always up to
date (updated less frequently, but still a candidate for client eviction).
2023-07-25 16:10:38 +03:00
9f512017aa Update request/response policies. (#12417)
changing the response and request policy of a few commands,
see https://redis.io/docs/reference/command-tips

1. RANDOMKEY used to have no response policy, which means
  that when sent to multiple shards, the responses should be aggregated.
  this normally applies to commands that return arrays, but since RANDOMKEY
  replies with a simple string, it actually requires a SPECIAL response policy
  (for the client to select just one)
2. SCAN used to have no response policy, but although the key names part of
  the response can be aggregated, the cursor part certainly can't.
3. MSETNX had a request policy of MULTI_SHARD and response policy of AGG_MIN,
  but in fact the contract with MSETNX is that when one key exists, it returns 0
  and doesn't set any key, routing it to multiple shards would mean that if one failed
  and another succeeded, it's atomicity is broken and it's impossible to return a valid
  response to the caller.

Co-authored-by: Shachar Langbeheim <shachlan@amazon.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
2023-07-25 10:21:23 +03:00
Harkrishn PatroandGitHub 34b95f752c Add test case for APPEND command usage on integer value (#12429)
Add test coverage to validate object encoding update on APPEND command usage on a integer value
2023-07-24 18:25:50 -07:00
2495b90a64 redis-cli: use previous hostip when not provided by redis cluster server (#12273)
When the redis server cluster running on cluster-preferred-endpoint-type unknown-endpoint mode, and receive a request that should be redirected to another redis server node, it does not reply the hostip, but a empty host like MOVED 3999 :6381.

The redis-cli would try to connect to an address without a host, which cause the issue:
```
127.0.0.1:7002> set bar bar
-> Redirected to slot [5061] located at :7000
Could not connect to Redis at :7000: No address associated with hostname
Could not connect to Redis at :7000: No address associated with hostname
not connected> exit
```

In this case, the redis-cli should use the previous hostip when there's no host provided by the server.

---------

Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Madelyn Olson <madelynolson@gmail.com>
2023-07-20 15:31:06 -07:00
Oran AgraandGitHub 4b1fbb2338 Merge pull request #12415 from georgeanderson/issue_12378
Replaced comment with excessive warning.
2023-07-18 09:36:45 +03:00
George Guimares 9a3b99cbd1 Replaced comment with excessive warning.
The data structures in the comment are not in sync and don't need to be.
Referring to function that handles conversion.
2023-07-16 17:04:15 -05:00
Chen TianjieandGitHub 91011100ba Hide the comma after cport when there is no hostname. (#12411)
According to the format shown in https://redis.io/commands/cluster-nodes/
```
<ip:port@cport[,hostname[,auxiliary_field=value]*]>
```
when there is no hostname, and the auxiliary fields are hidden, the cluster topology should be
```
<ip:port@cport>
```
However in the code we always print the hostname even when it is an empty string, leaving an unnecessary comma tailing after cport, which is weird and conflicts with the doc.
```
94ca2f6cf85228a49fde7b738ee1209de7bee325 127.0.0.1:6379@16379, myself,master - 0 0 0 connected 0-16383
```
2023-07-15 20:31:42 -07:00
Yossi GottliebandGitHub 9b1d4f003d Merge pull request #12409 from chayim/ck-hired120
Upgrade to hiredis 1.2.0.
2023-07-13 17:45:38 +03:00
Chayim I. Kirshen 0175fe8219 Explaining hiredis upgrade 2023-07-13 09:26:12 +03:00
Chayim I. Kirshen 8e138ba44f merging hiredis v1.2.0 2023-07-13 09:25:22 +03:00
Chayim I. Kirshen 67ed3183b5 Squashed 'deps/hiredis/' changes from b6a052fe0..60e5075d4
60e5075d4 Version 1.2.0 (#1202)
adef139a7 Remove support in deprecated TLS versions 1.0 and 1.1 (#1205)
d543baba6 Install major version symlink of shared objects.
052f99ab2 Ensure functionality without _MSC_VER definition
ded32c7d1 Add a test for the TCP_USER_TIMEOUT option. (#1192)
5cbd1f296 Add -Werror as a default. (#1193)
af1445638 CI: Update homebrew Redis version. (#1191)
6de326e87 Fix a typo in b6a052f.

git-subtree-dir: deps/hiredis
git-subtree-split: 60e5075d4ac77424809f855ba3e398df7aacefe8
2023-07-13 09:23:10 +03:00
936cfa464f Lua cjson and cmsgpack integer overflow issues (CVE-2022-24834) (#12398)
* Fix integer overflows due to using wrong integer size.
* Add assertions / panic when overflow still happens.
* Deletion of dead code to avoid need to maintain it
* Some changes are not because of bugs, but rather paranoia.
* Improve cmsgpack and cjson test coverage.

Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
2023-07-10 10:26:09 +03:00
BinbinandGitHub 14f802b360 Initialize cluster owner_not_claiming_slot to avoid warning (#12391)
valgrind report a Uninitialised warning:
```
==25508==  Uninitialised value was created by a heap allocation
==25508==    at 0x4848899: malloc (in
/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==25508==    by 0x1A35A1: ztrymalloc_usable_internal (zmalloc.c:117)
==25508==    by 0x1A368D: zmalloc (zmalloc.c:145)
==25508==    by 0x21FDEA: clusterInit (cluster.c:973)
==25508==    by 0x19DC09: main (server.c:7306)
```

Introduced in #12344
2023-07-07 06:40:44 -07:00
zhaozhao.zzandGitHub aefdc57f1e add an assertion to make sure numkeys is 0 in getKeysUsingKeySpecs (#12389)
This is an addition to #12380, to prevent potential bugs when collecting
keys from multiple commands in the future.
Note that this function also resets numkeys in some cases.
2023-07-06 08:17:25 +03:00
SankarandGitHub 1190f25ca7 Process loss of slot ownership in cluster bus (#12344)
Process loss of slot ownership in cluster bus

When a node no longer owns a slot, it clears the bit corresponding
to the slot in the cluster bus messages. The receiving nodes
currently don't record the fact that the sender stopped claiming
a slot until some other node in the cluster starts claiming the slot.
This can cause a slot to go missing during slot migration when subjected
to inopportune race with addition of new shards or a failover.
This fix forces the receiving nodes to process the loss of ownership
to avoid spreading wrong information.
2023-07-05 17:46:23 -07:00
Job Henandez LaraandGitHub b35e20d1de redis-benchmark: add documentation for the amount of clients needed (#12372)
In cluster mode, we need more clients than the number of nodes.
2023-07-05 10:27:14 +03:00
BinbinandGitHub d56a7d9b05 Fix and increase tollerance of event loop test, add verbose logs (#12385)
The test fails on freebsd CI:
```
*** [err]: stats: eventloop metrics in tests/unit/info.tcl
Expected '31777' to be less than '16183' (context: type eval line 17 cmd
{assert_lessthan $el_sum2 [expr $el_sum1+10000] } proc ::test)
```

The test added in #11963, fails on freebsd CI which is slow,
increase tollerance and also add some verbose logs, now we can
see these logs in verbose mode (for better views):
```
eventloop metrics cycle1: 12, cycle2: 15
eventloop metrics el_sum1: 315, el_sum2: 411
eventloop metrics cmd_sum1: 126, cmd_sum2: 137
[ok]: stats: eventloop metrics (111 ms)
instantaneous metrics instantaneous_eventloop_cycles_per_sec: 8
instantaneous metrics instantaneous_eventloop_duration_usec: 55
[ok]: stats: instantaneous metrics (1603 ms)
[ok]: stats: debug metrics (112 ms)
```
2023-07-05 09:32:30 +03:00
b7559d9f3b Fix possible crash in command getkeys (#12380)
When getKeysUsingKeySpecs processes a command with more than one key-spec,
and called with a total of more than 256 keys, it'll call getKeysPrepareResult again,
but since numkeys isn't updated, getKeysPrepareResult will not bother to copy key
names from the old result (leaving these slots uninitialized). Furthermore, it did not
consider the keys it already found when allocating more space.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-07-03 12:45:18 +03:00
BinbinandGitHub 6b57241fa8 Revert zrangeGenericCommand negative offset check (#12377)
The negative offset check was added in #9052, we realized
that this is a non-mandatory breaking change and we would
like to add it only in 8.0.

This reverts PR #9052, will be re-introduced later in 8.0.
2023-07-02 20:38:30 +03:00
BinbinandGitHub 26174123ee Avoid DEBUG POPULATE crash at dictExpand OOM (#12363)
Change to use dictTryExpand, return error on OOM.
2023-07-01 07:35:35 -07:00
6bf9b144ef redis-cli: Support URIs with IPv6 (#11834)
Co-authored-by: hrliu <hrliu@alauda.io>
2023-06-29 19:32:01 +03:00
07ed0eafa9 improve performance for scan command when matching pattern or data type (#12209)
Optimized the performance of the SCAN command in a few ways:
1. Move the key filtering (by MATCH pattern) in the scan callback,
  so as to avoid collecting them for later filtering.
2. Reduce a many memory allocations and copying (use a reference
  to the original sds, instead of creating an robj, an excessive 2 mallocs
  and one string duplication)
3. Compare TYPE filter directly (as integers), instead of inefficient string
  compare per key.
4. fixed a small bug: when scan zset and hash types, maxiterations uses
  a more accurate number to avoid wrong double maxiterations.

Changes **postponed** for a later version (8.0):
1. Prepare to move the TYPE filtering to the scan callback as well. this was
  put on hold since it has side effects that can be considered a breaking
  change, which is that we will not attempt to do lazy expire (delete) a key
  that was filtered by not matching the TYPE (changing it would mean TYPE filter
  starts behaving the same as MATCH filter already does in that respect). 
2. when the specified key TYPE filter is an unknown type, server will reply a error
  immediately instead of doing a full scan that comes back empty handed. 

Benchmark result:
For different scenarios, we obtained about 30% or more performance improvement.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-06-27 16:43:46 +03:00
Maria MarkovaandGitHub b2cdf6bcc3 Adding of O3 on linking stage for OPTIMIZATION=-O3 cases (#12339)
Added missing O3 flag to linking stage in default option "-O3 -flto". 
Flags doesn't lead to significant changes in performance:
- +0.21% in geomean for all benchmarks on ICX bare-metal (256 cpus)
- +0.33% in geomean for all benchmarks on m6i.2xlarge (16 cpus) 
Checked on redis from Mar'30 (commit 1f76bb17dd ). Comparison file is attached.
2023-06-27 11:54:17 +03:00
michalbiesekandGitHub ef4bb4e374 Add RISC-V debug support (#12349)
- Add support for `getAndSetMcontextEip`
- Add support for `logRegisters`
2023-06-27 11:53:42 +03:00
BinbinandGitHub f58fd9e6d2 Set HIDDEN_CONFIG flag on aof-disable-auto-gc (#12355)
aof-disable-auto-gc was created for testing purposes,
to check if certain AOF files were actually generated
and if they were deletedcorrectly during testing.

So hiding it, see #12249 for more discussion.
2023-06-27 10:21:58 +03:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1634a0f271 Bump vmactions/freebsd-vm from 0.3.0 to 0.3.1 (#12352)
Bumps [vmactions/freebsd-vm](https://github.com/vmactions/freebsd-vm) from 0.3.0 to 0.3.1.
- [Release notes](https://github.com/vmactions/freebsd-vm/releases)
- [Commits](https://github.com/vmactions/freebsd-vm/compare/v0.3.0...v0.3.1)

---
updated-dependencies:
- dependency-name: vmactions/freebsd-vm
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-27 09:17:34 +03:00
Chen TianjieandGitHub 22a29935ff Support TLS service when "tls-cluster" is not enabled and persist both plain and TLS port in nodes.conf (#12233)
Originally, when "tls-cluster" is enabled, `port` is set to TLS port. In order to support non-TLS clients, `pport` is used to propagate TCP port across cluster nodes. However when "tls-cluster" is disabled, `port` is set to TCP port, and `pport` is not used, which means the cluster cannot provide TLS service unless "tls-cluster" is on.
```
typedef struct {
    // ...
    uint16_t port;  /* Latest known clients port (TLS or plain). */
    uint16_t pport; /* Latest known clients plaintext port. Only used if the main clients port is for TLS. */
    // ...
} clusterNode;
```
```
typedef struct {
    // ...
    uint16_t port;   /* TCP base port number. */
    uint16_t pport;  /* Sender TCP plaintext port, if base port is TLS */
    // ...
} clusterMsg;
```
This PR renames `port` and `pport` in `clusterNode` to `tcp_port` and `tls_port`, to record both ports no matter "tls-cluster" is enabled or disabled.

This allows to provide TLS service to clients when "tls-cluster" is disabled: when displaying cluster topology, or giving `MOVED` error, server can provide TLS or TCP port according to client's connection type, no matter what type of connection cluster bus is using.

For backwards compatibility, `port` and `pport` in `clusterMsg` are preserved, when "tls-cluster" is enabled, `port` is set to TLS port and `pport` is set to TCP port, when "tls-cluster" is disabled, `port` is set to TCP port and `pport` is set to TLS port (instead of 0).

Also, in the nodes.conf file, a new aux field displaying an extra port is added to complete the persisted info. We may have `tls_port=xxxxx` or `tcp_port=xxxxx` in the aux field, to complete the cluster topology, while the other port is stored in the normal `<ip>:<port>` field. The format is shown below.
```
<node-id> <ip>:<tcp_port>@<cport>,<hostname>,shard-id=...,tls-port=6379 myself,master - 0 0 0 connected 0-1000
```
Or we can switch the position of two ports, both can be correctly resolved.
```
<node-id> <ip>:<tls_port>@<cport>,<hostname>,shard-id=...,tcp-port=6379 myself,master - 0 0 0 connected 0-1000
```
2023-06-26 07:43:38 -07:00
BinbinandGitHub 9600553ef2 Move clusterBeforeSleep before blockedBeforeSleep (#12343)
The new blockedBeforeSleep was added in #12337, it breaks the order in 2ecb5ed.

This may be related to #2288, quoted from comment in #2288:
```
Moreover the clusterBeforeSleep() call was misplaced at the end of the chain of the
beforeSleep() call in redis.c. It should be at the top, before processing un blocking
clients. This is exactly the reason in the specific instance of the bug as reported,
why the state was not updated in time before clients served.
```
2023-06-25 14:21:03 +03:00
Meir Shpilraien (Spielrein)andGitHub 153f8f082e Fix use after free on blocking RM_Call. (#12342)
blocking RM_Call was introduced on: #11568, It allows a module to perform
blocking commands and get the reply asynchronously.If the command gets
block, a special promise CallReply is returned that allow to set the unblock
handler. The unblock handler will be called when the command invocation
finish and it gets, as input, the command real reply.

The issue was that the real CallReply was created using a stack allocated
RedisModuleCtx which is no longer available after the unblock handler finishes.
So if the module keeps the CallReply after the unblock handler finished, the
CallReply holds a pointer to invalid memory and will try to access it when the
CallReply will be released.

The solution is to create the CallReply with a NULL context to make it totally
detached and can be freed freely when the module wants.

Test was added to cover this case, running the test with valgrind before the
fix shows the use after free error. With the fix, there are no valgrind errors.

unrelated: adding a missing `$rd close` in many tests in that file.
2023-06-25 14:12:27 +03:00
guybe7andGitHub 3230199920 Modules: Unblock from within a timer coverage (#12337)
Apart from adding the missing coverage, this PR also adds `blockedBeforeSleep`
that gathers all block-related functions from `beforeSleep`

The order inside `blockedBeforeSleep` is different: now `handleClientsBlockedOnKeys`
(which may unblock clients) is called before `processUnblockedClients` (which handles
unblocked clients).
It makes sense to have this order.

There are no visible effects of the wrong ordering, except some cleanups of the now-unblocked
client would have  happen in the next `beforeSleep` (will now happen in the current one)

The reason we even got into it is because i triggers an assertion in logresreq.c (breaking
the assumption that `unblockClient` is called **before** actually flushing the reply to the socket):
`handleClientsBlockedOnKeys` is called, then it calls `moduleUnblockClientOnKey`, which calls
`moduleUnblockClient`, which adds the client to `moduleUnblockedClients` back to `beforeSleep`,
we call `handleClientsWithPendingWritesUsingThreads`, it writes the data of buf to the client, so
`client->bufpos` became 0
On the next `beforeSleep`, we call `moduleHandleBlockedClients`, which calls `unblockClient`,
which calls `reqresAppendResponse`, triggering the assert. (because the `bufpos` is 0) - see https://github.com/redis/redis/pull/12301#discussion_r1226386716
2023-06-22 23:15:16 +03:00
Gabi GanamandGitHub 9e5f45f230 Fix typos in comments (#12338) 2023-06-22 08:10:42 -07:00
BinbinandGitHub 47f32bc998 Print strerror when bio initialization fails (#12333)
Now we can see something like this:
```
Fatal: Can't initialize Background Jobs. Error message: Cannot allocate memory
```
2023-06-21 17:57:11 +03:00
guybe7andGitHub d46ef88671 Improve moduleBlockClient timeout overflow handling (#12174)
Continuation of https://github.com/redis/redis/pull/11338
avoid overflow adding input timeout to "now" in moduleBlockClient.
2023-06-21 12:48:13 +03:00
Madelyn OlsonandGitHub 73cf0243df Make nodename test more consistent (#12330)
To determine when everything was stable, we couldn't just query the nodename since they aren't API visible by design. Instead, we were using a proxy piece of information which was bumping the epoch and waiting for everyone to observe that. This works for making source Node 0 and Node 1 had pinged, and Node 0 and Node 2 had pinged, but did not guarantee that Node 1 and Node 2 had pinged. Although unlikely, this can cause this failure message. To fix it I hijacked hostnames and used its validation that it has been propagated, since we know that it is stable.

I also noticed while stress testing this sometimes the test took almost 4.5 seconds to finish, which is really close to the current 5 second limit of the log check, so I bumped that up as well just to make it a bit more consistent.
2023-06-20 18:00:55 -07:00
guybe7andGitHub 20fa156067 Align RM_ReplyWithErrorFormat with RM_ReplyWithError (#12321)
Introduced by https://github.com/redis/redis/pull/11923 (Redis 7.2 RC2)

It's very weird and counterintuitive that `RM_ReplyWithError` requires the error-code
**without** a hyphen while `RM_ReplyWithErrorFormat` requires either the error-code
**with** a hyphen or no error-code at all
```
RedisModule_ReplyWithError(ctx, "BLA bla bla");
```
vs.
```
RedisModule_ReplyWithErrorFormat(ctx, "-BLA %s", "bla bla");
```

This commit aligns RM_ReplyWithErrorFormat to behvae like RM_ReplyWithError.
it's a breaking changes but it's done before 7.2 goes GA.
2023-06-20 20:44:43 +03:00
Oran AgraandGitHub 8ad8f0f9d8 Fix broken protocol when PUBLISH emits local push inside MULTI (#12326)
When a connection that's subscribe to a channel emits PUBLISH inside MULTI-EXEC,
the push notification messes up the EXEC response.

e.g. MULTI, PING, PUSH foo bar, PING, EXEC
the EXEC's response will contain: PONG, {message foo bar}, 1. and the second PONG
will be delivered outside the EXEC's response.

Additionally, this PR changes the order of responses in case of a plain PUBLISH (when
the current client also subscribed to it), by delivering the push after the command's
response instead of before it.
This also affects modules calling RM_PublishMessage in a similar way, so that we don't
run the risk of getting that push mixed together with the module command's response.
2023-06-20 20:41:41 +03:00
93708c7f6a use embedded string object and more efficient ll2string for long long value convert to string (#12250)
A value of type long long is always less than 21 bytes when convert to a
string, so always meets the conditions for using embedded string object
which can always get memory reduction and performance gain (less calls
to the heap allocator).
Additionally, for the conversion of longlong type to sds, we also use a faster
algorithm (the one in util.c instead of the one that used to be in sds.c). 

For the DECR command on 32-bit Redis, we get about a 5.7% performance
improvement. There will also be some performance gains for some commands
that heavily use sdscatfmt to convert numbers, such as INFO.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-06-20 15:14:44 +03:00
BinbinandGitHub d9c2ef8a72 zrangeGenericCommand add check for negative offset (#9052)
Now we will check the offset in zrangeGenericCommand.
With a negative offset, we will throw an error and return.

This also resolve the issue of zeroing the destination key
in case of the "store" variant when we input a negative offset.
```
127.0.0.1:6379> set key value
OK
127.0.0.1:6379> zrangestore key myzset 0 10 byscore limit -1 10
(integer) 0
127.0.0.1:6379> exists key
(integer) 0
```

This change affects the following commands:
- ZRANGE / ZRANGESTORE / ZRANGEBYLEX / ZRANGEBYSCORE
- ZREVRANGE / ZREVRANGEBYSCORE / ZREVRANGEBYLEX
2023-06-20 14:33:17 +03:00
Wen HuiandGitHub 66ea178cb6 adding geo command edge cases tests (#12274)
For geosearch and georadius we have already test coverage for wrong type, but we dont have for geodist, geohash, geopos commands. So adding the wrong type test cases for geodist, geohash, geopos commands.

Existing code, we have verify_geo_edge_response_bymember function for wrong type test cases which has member as an option. But the function is being called in other test cases where the output is not inline with these commnds(geodist, geohash, geopos). So I could not include these commands(geodist, geohash, geopos) as part of existing function, hence implemented a new function verify_geo_edge_response_generic and called from the test case.
2023-06-20 12:50:03 +03:00
BinbinandGitHub d306d86146 Fix ZRANK/ZREVRANK reply_schema description (#12331)
The parameter name is WITHSCORE instead of WITHSCORES.
2023-06-20 11:15:40 +03:00
mstmdevandGitHub 13e17e94d8 Doc indentation fix for the --functions-rdb option (#12328) 2023-06-20 11:15:11 +03:00
813924b4c3 Sanitizer reported memory leak for '--invalid' option or port number is missed cases to redis-server. (#12322)
Observed that the sanitizer reported memory leak as clean up is not done
before the process termination in negative/following cases:

**- when we passed '--invalid' as option to redis-server.**

```
 -vm:~/mem-leak-issue/redis$ ./src/redis-server --invalid

*** FATAL CONFIG FILE ERROR (Redis 255.255.255) ***
Reading the configuration file, at line 2
>>> 'invalid'
Bad directive or wrong number of arguments

=================================================================
==865778==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x7f0985f65867 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x558ec86686ec in ztrymalloc_usable_internal /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:117
    #2 0x558ec86686ec in ztrymalloc_usable /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:135
    #3 0x558ec86686ec in ztryrealloc_usable_internal /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:276
    #4 0x558ec86686ec in zrealloc /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:327
    #5 0x558ec865dd7e in sdssplitargs /home/ubuntu/mem-leak-issue/redis/src/sds.c:1172
    #6 0x558ec87a1be7 in loadServerConfigFromString /home/ubuntu/mem-leak-issue/redis/src/config.c:472
    #7 0x558ec87a13b3 in loadServerConfig /home/ubuntu/mem-leak-issue/redis/src/config.c:718
    #8 0x558ec85e6f15 in main /home/ubuntu/mem-leak-issue/redis/src/server.c:7258
    #9 0x7f09856e5d8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

SUMMARY: AddressSanitizer: 8 byte(s) leaked in 1 allocation(s).

```

**- when we pass '--port' as option and missed to add port number to redis-server.**

```
vm:~/mem-leak-issue/redis$ ./src/redis-server --port

*** FATAL CONFIG FILE ERROR (Redis 255.255.255) ***
Reading the configuration file, at line 2
>>> 'port'
wrong number of arguments

=================================================================
==865846==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x7fdcdbb1f867 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x557e8b04f6ec in ztrymalloc_usable_internal /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:117
    #2 0x557e8b04f6ec in ztrymalloc_usable /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:135
    #3 0x557e8b04f6ec in ztryrealloc_usable_internal /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:276
    #4 0x557e8b04f6ec in zrealloc /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:327
    #5 0x557e8b044d7e in sdssplitargs /home/ubuntu/mem-leak-issue/redis/src/sds.c:1172
    #6 0x557e8b188be7 in loadServerConfigFromString /home/ubuntu/mem-leak-issue/redis/src/config.c:472
    #7 0x557e8b1883b3 in loadServerConfig /home/ubuntu/mem-leak-issue/redis/src/config.c:718
    #8 0x557e8afcdf15 in main /home/ubuntu/mem-leak-issue/redis/src/server.c:7258
    #9 0x7fdcdb29fd8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

Indirect leak of 10 byte(s) in 1 object(s) allocated from:
    #0 0x7fdcdbb1fc18 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:164
    #1 0x557e8b04f9aa in ztryrealloc_usable_internal /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:287
    #2 0x557e8b04f9aa in ztryrealloc_usable /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:317
    #3 0x557e8b04f9aa in zrealloc_usable /home/ubuntu/mem-leak-issue/redis/src/zmalloc.c:342
    #4 0x557e8b033f90 in _sdsMakeRoomFor /home/ubuntu/mem-leak-issue/redis/src/sds.c:271
    #5 0x557e8b033f90 in sdsMakeRoomFor /home/ubuntu/mem-leak-issue/redis/src/sds.c:295
    #6 0x557e8b033f90 in sdscatlen /home/ubuntu/mem-leak-issue/redis/src/sds.c:486
    #7 0x557e8b044e1f in sdssplitargs /home/ubuntu/mem-leak-issue/redis/src/sds.c:1165
    #8 0x557e8b188be7 in loadServerConfigFromString /home/ubuntu/mem-leak-issue/redis/src/config.c:472
    #9 0x557e8b1883b3 in loadServerConfig /home/ubuntu/mem-leak-issue/redis/src/config.c:718
    #10 0x557e8afcdf15 in main /home/ubuntu/mem-leak-issue/redis/src/server.c:7258
    #11 0x7fdcdb29fd8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

SUMMARY: AddressSanitizer: 18 byte(s) leaked in 2 allocation(s).

```

As part analysis found that the sdsfreesplitres is not called when this condition checks are being hit.

Output after the fix:


```
vm:~/mem-leak-issue/redis$ ./src/redis-server --invalid

*** FATAL CONFIG FILE ERROR (Redis 255.255.255) ***
Reading the configuration file, at line 2
>>> 'invalid'
Bad directive or wrong number of arguments
vm:~/mem-leak-issue/redis$

===========================================
vm:~/mem-leak-issue/redis$ ./src/redis-server --jdhg

*** FATAL CONFIG FILE ERROR (Redis 255.255.255) ***
Reading the configuration file, at line 2
>>> 'jdhg'
Bad directive or wrong number of arguments

---------------------------------------------------------------------------
vm:~/mem-leak-issue/redis$ ./src/redis-server --port

*** FATAL CONFIG FILE ERROR (Redis 255.255.255) ***
Reading the configuration file, at line 2
>>> 'port'
wrong number of arguments
```

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-06-20 10:07:29 +03:00
Shaya PotterandGitHub 07316f16f0 Add ability for modules to know which client is being cmd filtered (#12219)
Adds API
- RedisModule_CommandFilterGetClientId()

Includes addition to commandfilter test module to validate that it works
by performing the same command from 2 different clients
2023-06-20 09:03:52 +03:00
BinbinandGitHub cd4f3e20c1 Fix cluster human_nodename Getter data loss in nodes.conf (#12325)
auxHumanNodenameGetter limited to %.40s, since we did not limit the
length of config cluster-announce-human-nodename, %.40s will cause
nodename data loss (we will persist it in nodes.conf).

Additional modified auxHumanNodenamePresent to use sdslen.
2023-06-19 17:13:18 -07:00
BinbinandGitHub b510624978 Optimize PSUBSCRIBE and PUNSUBSCRIBE from O(N*M) to O(N) (#12298)
In the original implementation, the time complexity of the commands
is actually O(N*M), where N is the number of patterns the client is
already subscribed and M is the number of patterns to subscribe to.
The docs are all wrong about this.

Specifically, because the original client->pubsub_patterns is a list,
so we need to do listSearchKey which is O(N). In this PR, we change it
to a dict, so the search becomes O(1).

At the same time, both pubsub_channels and pubsubshard_channels are dicts.
Changing pubsub_patterns to a dictionary improves the readability and
maintainability of the code.
2023-06-19 16:31:18 +03:00
Oran AgraandGitHub 07c14672bf Set Jemalloc --disable-cache-oblivious to reduce memory overhead (#12315)
Apparently for large size classes Jemalloc allocate some extra
memory (can be up to 25% overhead for allocations of 16kb).
see https://github.com/jemalloc/jemalloc/issues/1098#issuecomment-1589870476

p.s. from Redis's perspective that looks like external fragmentation,
(i.e. allocated bytes will be low, and active pages bytes will be large)
which  can cause active-defrag to eat CPU cycles in vain.

Some details about this mechanism we disable:
---------------------------------------------------------------
Disabling this mechanism only affects large allocations (above 16kb)
Not only that it isn't expected to cause any performance regressions,
it's actually recommended, unless you have a specific workload pattern
and hardware that benefit from this feature -- by default it's enabled and
adds address randomization to all large buffers, by over allocating 1 page
per large size class, and offsetting into that page to make the starting
address of the user buffer randomized. Workloads such as scientific
computation often handle multiple big matrixes at the same time, and the
randomization makes sure that the cacheline level accesses don't suffer
bad conflicts (when they all start from page-aligned addresses).

However the downsize is also quite noticeable, like you observed that extra
page per large size can cause memory overhead, plus the extra TLB entry.
The other factor is, hardware in the last few years started doing the
randomization at the hardware level, i.e. the address to cacheline mapping isn't
a direct mapping anymore. So there's debate to disable the randomization by default,
but we are still hesitant because when it matters, it could matter a lot, and having
it enabled by default limits that worst case behavior, even though it means the
majority of workloads suffers a regression.

So in short, it's safe and offers better performance in most cases.
2023-06-18 10:30:23 +03:00
070453eef3 Cluster human readable nodename feature (#9564)
This PR adds a human readable name to a node in clusters that are visible as part of error logs. This is useful so that admins and operators of Redis cluster have better visibility into failures without having to cross-reference the generated ID with some logical identifier (such as pod-ID or EC2 instance ID). This is mentioned in #8948. Specific nodenames can be set by using the variable cluster-announce-human-nodename. The nodename is gossiped using the clusterbus extension in #9530.

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2023-06-17 21:16:51 -07:00
b00a235186 Use Reservoir Sampling for random sampling of dict, and fix hang during fork (#12276)
## Issue:
When a dict has a long chain or the length of the chain is longer than
the number of samples, we will never be able to sample the elements
at the end of the chain using dictGetSomeKeys().
This could mean that SRANDMEMBER can be hang in and endless loop.
The most severe case, is the pathological case of when someone uses SCAN+DEL
or SSCAN+SREM creating an unevenly distributed dict.
This was amplified by the recent change in #11692 which prevented a
down-sizing rehashing while there is a fork.

## Solution
1. Before, we will stop sampling when we reach the maximum number
  of samples, even if there is more data after the current chain.
  Now when we reach the maximum we use the Reservoir Sampling
  algorithm to fairly sample the end of the chain that cannot be sampled
2. Fix the rehashing code, so that the same as it allows rehashing for up-sizing
  during fork when the ratio is extreme, it will allow it for down-sizing as well.

Issue was introduced (or became more severe) by #11692

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-06-16 19:13:07 +03:00
BinbinandGitHub 439b0315c8 Fix SPOP/RESTORE propagation when doing lazy free (#12320)
In SPOP, when COUNT is greater than or equal to set's size,
we will remove the set. In dbDelete, we will do DEL or UNLINK
according to the lazy flag. This is also required for propagate.

In RESTORE, we won't store expired keys into the db, see #7472.
When used together with REPLACE, it should emit a DEL or UNLINK
according to the lazy flag.

This PR also adds tests to cover the propagation. The RESTORE
test will also cover #7472.
2023-06-16 08:14:11 -07:00
YaacovHazanandGitHub 5da9eecdb8 Removing duplicated tests (#12318)
In 4ba47d2d2 the following tests added in both tracking.tcl and introspection.tcl

- Coverage: Basic CLIENT CACHING
- Coverage: Basic CLIENT REPLY
- Coverage: Basic CLIENT TRACKINGINFO
- Coverage: Basic CLIENT GETREDIR
2023-06-16 15:55:24 +03:00
BinbinandGitHub 9dc6f93e9d Add command being unblocked cause another command to get unblocked execution order test (#12324)
* Add command being unblocked cause another command to get unblocked execution order test

In #12301, we observed that if the
`while(listLength(server.ready_keys) != 0)`
in handleClientsBlockedOnKeys is changed to
`if(listLength(server.ready_keys) != 0)`,
the order of command execution will change.

It is wrong to change that. It means that if a command
being unblocked causes another command to get unblocked
(like a BLMOVE would do), then the new unblocked command
will wait for later to get processed rather than right away.

It'll not have any real implication if we change that since
we do call handleClientsBlockedOnKeys in beforeSleep again,
and redis will still behave correctly, but we don't change that.

An example:
1. $rd1 blmove src{t} dst{t} left right 0
2. $rd2 blmove dst{t} src{t} right left 0
3. $rd3 set key1{t}, $rd3 lpush src{t}, $rd3 set key2{t} in a pipeline

The correct order would be:
1. set key1{t}
2. lpush src{t}
3. lmove src{t} dst{t} left right
4. lmove dst{t} src{t} right left
5. set key2{t}

The wrong order would be:
1. set key1{t}
2. lpush src{t}
3. lmove src{t} dst{t} left right
4. set key2{t}
5. lmove dst{t} src{t} right left

This PR adds corresponding test to cover it.

* Add comment near while(listLength(server.ready_keys) != 0)
2023-06-16 15:39:00 +03:00
Meir Shpilraien (Spielrein)andGitHub cefe4566e9 Prevent PEJ on loading and on readonly replica. (#12304)
While Redis loading data from disk (AOF or RDB), modules will get
key space notifications. In such stage the module should not register
any PEJ, the main reason this is forbidden is that PEJ purpose is to
perform a write operation as a reaction to the key space notification.
Write operations should not be performed while loading data and so
there is no reason to register a PEJ. 

Same argument also apply to readonly replica. module should not
perform any writes as a reaction to key space notifications and so it
should not register a PEJ.

If a module need to perform some other task which is not involve
writing, he can do it on the key space notification callback itself.
2023-06-15 10:10:23 +03:00
BinbinandGitHub 254475537a Optimize SET PXAT to reduce calls of rewriteClientCommandVector (#12316)
In PXAT case, there is no need to do the rewriteClientCommandVector,
a simply benchmark show we gain a improvement of 10%.
2023-06-15 10:07:47 +03:00
judengandGitHub 789c33bb3e improve performance for keys with expiration time (#12177)
This change only affects keys with expiry time.
For SETEX, the average improvement is 5%, and for GET with
expiation key, we gain a improvement of 13%.

When keys have expiration time, Redis has an assertion to look
up the main dict every time when it touches the expires.
This comes with a performance const, especially during rehash.
the damage will be double.

It looks like that assert was added some ten years old, maybe out
of paranoia, and there's probably no reason to keep it at that cost.
2023-06-14 11:17:39 +03:00
Wen HuiandGitHub 1946083410 Removing the duplicate test case (#12310)
Looks like the Zadd test case was copied to create Zincrby test case ,but missed to change the command.
2023-06-14 10:03:33 +03:00
Harkrishn PatroandGitHub a9e32767f7 Allow cluster slots/shards api to respond during loading (#12269)
It would be helpful for clients to get cluster slots/shards information during a node failover and is loading data.
2023-06-13 18:16:32 +03:00
e7129e43e0 Fix XREADGROUP BLOCK stuck in endless loop (#12301)
For the XREADGROUP BLOCK > scenario, there is an endless loop.
Due to #11012, it keep going, reprocess command -> blockForKeys -> reprocess command

The right fix is to avoid an endless loop in handleClientsBlockedOnKey and handleClientsBlockedOnKeys,
looks like there was some attempt in handleClientsBlockedOnKeys but maybe not sufficiently good,
and it looks like using a similar trick in handleClientsBlockedOnKey is complicated.
i.e. stashing the list on the stack and iterating on it after creating a fresh one for future use,
is problematic since the code keeps accessing the global list.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-06-13 13:27:05 +03:00
Oran AgraandGitHub f228ec1ea5 flushSlavesOutputBuffers should not write to replicas scheduled to drop (#12242)
This will increase the size of an already large COB (one already passed
the threshold for disconnection)

This could also mean that we'll attempt to write that data to the socket
and the replica will manage to read it, which will result in an
undesired partial sync (undesired for the test)
2023-06-12 14:05:34 +03:00
0bfb6d5582 Reset command duration for rejected command. (#12247)
In 7.2, After 971b177fa we make sure (assert) that
the duration has been recorded when resetting the client.

This is not true for rejected commands.
The use case I found is a blocking command that an ACL rule changed before
it was unblocked, and while reprocessing it, the command rejected and triggered the assert.

The PR reset the command duration inside rejectCommand / rejectCommandSds.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-06-11 23:12:05 +03:00
Chen TianjieandGitHub 2d7d3911da Allow bigger tolerance in eventloop duration test. (#12179)
In #11963, some new tests about eventloop duration were added, which includes time measurement in TCL scripts. This has caused some unexpected CI failures, such as #12169 and #12177, due to slow test servers or some performance jittering.
2023-06-11 09:02:41 +03:00
Wen HuiandGitHub d412269ff8 Adding missing test cases for Addslot Command (#12288)
Added missing test case coverage for below scenarios:

1. The command only works if all the specified slots are, from
  the point of view of the node receiving the command, currently
  not assigned. A node will refuse to take ownership for slots that
  already belong to some other node (including itself).
2. The command fails if the same slot is specified multiple times.
2023-06-11 08:36:26 +03:00
BinbinandGitHub 5fd9756d2e Fix ACLAppendUserForLoading memory leak when merge error (#12296)
This leak will only happen in loadServerConfigFromString,
that is, when we are loading a redis.conf, and the user is wrong.

Because it happens in loadServerConfigFromString, redis will
exit if there is an error, so this is actually just a cleanup.
2023-06-10 18:11:16 -07:00
BinbinandGitHub e4d183afd3 Add missing return on -UNKILLABLE sent by master case (#12277)
We now no longer propagate scripts (started from 7.0), so this is a
very rare issue that in nearly-dead-code.

This is an overlook in #9780
2023-06-08 15:13:53 +03:00
Yossi GottliebandGitHub e7a3d3d152 Merge pull request #12254 from yossigo/hiredis-refresh
Refresh deps/hiredis to latest (unreleased) version.
2023-06-06 06:40:43 +03:00
Yossi GottliebandGitHub 0bd1a3a4a8 Use std=gnu11 instead of std=c11. (#12253)
Adding this as it's required by the latest version of libmusl (but not
clear if it's a regression or an intentional change).
2023-06-05 12:11:30 +03:00
Yossi Gottlieb 032bb2a273 Update deps/README.md.
* We patch hiredis rather than rely on having a compatible sds version.
* We now have better test coverage for redis-cli and redis-benchmark.
2023-06-04 17:32:27 +03:00
Yossi Gottlieb 2eaf23601d Merge commit 'fad6c713c26d6e4f2dd867db4d341c7abe0fcbda' into hiredis-refresh 2023-05-30 23:28:07 +03:00
Yossi Gottlieb fad6c713c2 Squashed 'deps/hiredis/' changes from f8de9a4bd..b6a052fe0
b6a052fe0 Helper for setting TCP_USER_TIMEOUT socket option (#1188)
3fa9b6944 Add RedisModule adapter (#1182)
d13c091e9 Fix wincrypt symbols conflict
5d84c8cfd Add a test ensuring we don't clobber connection error.
3f95fcdae Don't attempt to set a timeout if we are in an error state.
aacb84b8d Fix typo in makefile.
563b062e3 Accept -nan per the RESP3 spec recommendation.
04c1b5b02 Fix colliding option values
4ca8e73f6 Rework searching for openssl
cd208812f Attempt to find the correct path for openssl.
011f7093c Allow specifying the keepalive interval
e9243d4f7 Cmake static or shared (#1160)
1cbd5bc76 Write a version file for the CMake package (#1165)
6f5bae8c6 fix typo
acd09461d CMakeLists.txt: respect BUILD_SHARED_LIBS
97fcf0fd1 Add sdevent adapter
ccff093bc Bump dev version for the next release cycle.
c14775b4e Prepare for v1.1.0 GA
f0bdf8405 Add support for nan in RESP3 double (#1133)
991b0b0b3 Add an example that calls redisCommandArgv (#1140)
a36686f84 CI updates (#1139)
8ad4985e9 fix flag reference
7583ebb1b Make freeing a NULL redisAsyncContext a no op.
2c53dea7f Update version in dev branch.
f063370ed Prepare for v1.1.0-rc1
2b069573a CI fixes in preparation of release
e1e9eb40d Add author information to release-drafter template.
afc29ee1a Update for mingw cross compile
ceb8a8815 fixed cpp build error with adapters/libhv.h
3b15a04b5 Fixup of PR734: Coverage of hiredis.c (#1124)
c245df9fb CMake corrections for building on Windows (#1122)
9c338a598 Fix PUSH handler tests for Redis >= 7.0.5
6d5c3ee74 Install on windows fixes (#1117)
68b29e1ad Add timeout support to libhv adapter. (#1109)
722e3409c Additional include directory given by pkg-config (#1118)
bd9ccb8c4 Use __attribute__ when building with clang on windows
5392adc26 set default SSL certificate directory
560e66486 Minor refactor
d756f68a5 Add libhv example to our standard Makefile
a66916719 Add adapters/libhv
855b48a81 Fix pkgconfig for hiredis_ssl
79ae5ffc6 Fix protocol error (#1106)
61b5b299f Use a windows specific keepalive function. (#1104)
fce8abc1c Introduce .close method for redisContextFuncs
cfb6ca881 Add REDIS_OPT_PREFER_UNSPEC (#1101)
cc7c35ce6 Update documentation to explain redisConnectWithOptions.
bc8d837b7 fix heap-buffer-overflow (#957)
ca4a0e850 uvadapter: reduce number of uv_poll_start calls
35d398c90 Fix cmake config path on Linux. CMake config files were installed to `/usr/local/share/hiredis`, which is not recognizable by `find_package()`. I'm not sure why it was set that way. Given the commit introducing it is for Windows, I keep that behavior consistent there, but fix the rest.
10c78c6e1 Add possibility to prefer IPv6, IPv4 or unspecified
1abe0c828 fuzzer: No alloc in redisFormatCommand() when fail
329eaf9ba Fix heap-buffer-overflow issue in redisvFormatCommad
eaae7321c Polling adapter requires sockcompat.h
0a5fa3dde Regression test for off-by-one parsing error
9e174e8f7 Add do while(0) protection for macros
4ad99c69a Rework asSleep to be a generic millisleep function.
75cb6c1ea Do store command timeout in the context for redisSetTimeout (#593)
c57cad658 CMake: remove dict.c form hiredis_sources
8491a65a9 Add Github Actions CI workflow for hiredis: Arm, Arm64, 386, windows. (#943)
77e4f09ea Merge pull request #964 from afcidk/fix-createDoubleObject
9219f7e7c Merge pull request #901 from devnexen/illumos_test_fix
810cc6104 Merge pull request #905 from sundb/master
df8b74d69 Merge pull request #1091 from redis/ssl-error-ub-fix
0ed6cdec3 Fix some undefined behaviour
507a6dcaa Merge pull request #1090 from Nordix/subscribe-oom-error
b044eaa6a Copy error to redisAsyncContext when finding subscribe cb
e0200b797 Merge pull request #1087 from redis/const-and-non-const-callback
6a3e96ad2 Maintain backward compatibiliy withour onConnect callback.
e7afd998f Merge pull request #1079 from SukkaW/drop-macos-10.15-runner
17c8fe079 Merge pull request #931 from kristjanvalur/pr2
b808c0c20 Merge pull request #1083 from chayim/ck-drafter
367a82bf0 Merge pull request #1085 from stanhu/ssl-improve-options-setting
71119a71d Make it possible to set SSL verify mode
dd7979ac1 Merge pull request #1084 from stanhu/sh-improve-ssl-docs
c71116178 Improve example for SSL initialization in README.md
5c9b6b571 Release drafter
a606ccf2a CI: use recommended `vmactions/freebsd-vm@v0`
0865c115b Merge pull request #1080 from Nordix/readme-corrections
f6cee7142 Fix README typos
06be7ff31 Merge pull request #1050 from smmir-cent/fix-cmake-version
7dd833d54 CI: bump macos runner version
f69fac769 Drop `const` on redisAsyncContext in redisConnectCallback Since the callback is now re-entrant, it can call apis such as redisAsyncDisconnect()
005d7edeb Support calling redisAsyncDisconnect from the onConnected callback, by deferring context deletion
6ed060920 Add async regression test for issue #931
eaa2a7ee7 Merge pull request #932 from kristjanvalur/pr3
2ccef30f3 Add regression test for issue #945
4b901d44a Initial async tests
31c91408e Polling adapter and example
8a15f4d65 Merge pull request #1057 from orgads/static-name
902dd047f Merge pull request #1054 from kristjanvalur/pr08
c78d0926b Merge pull request #1074 from michael-grunder/kristjanvalur-pr4
2b115d56c Whitespace
1343988ce Fix typos
47b57aa24 Add some documentation on connect/disconnect callbacks and command callbacks
a890d9ce2 Merge pull request #1073 from michael-grunder/kristjanvalur-pr1
f246ee433 Whitespace, style
94c1985bd Use correct type for getsockopt()
5e002bc21 Support failed async connects on windows.
5d68ad2f4 Merge pull request #1072 from michael-grunder/fix-redis7-unit-tests
f4b6ed289 Fix tests so they work for Redis 7.0
95a0c1283 Merge pull request #1058 from orgads/win64
eedb37a65 Fix warnings on Win64
47c3ecefc Merge pull request #1062 from yossigo/fix-push-notification-order
e23d91c97 Merge pull request #1061 from yossigo/update-redis-apt
34211ad54 Merge pull request #1063 from redis/fix-windows-tests
9957af7e3 Whitelist hiredis repo path in cygwin
b455b3381 Handle push notifications before or after reply.
aed9ce446 Use official repository for redis package.
d7683f35a Merge pull request #1047 from Nordix/unsubscribe-handling
7c44a9d7e Merge pull request #1045 from Nordix/sds-updates
dd4bf9783 Use the same name for static and shared libraries
ff57c18b9 Embed debug information in windows static lib, rather than create a .pdb file
8310ad4f5 fix cmake version
7123b87f6 Handle any pipelined unsubscribe in async
b6fb548fc Ignore pubsub replies without a channel/pattern
00b82683b Handle overflows as errors instead of asserting
64062a1d4 Catch size_t overflows in sds.c
066c6de79 Use size_t/long to avoid truncation
c6657ef65 Merge branch 'redis:master' into master
50cdcab49 Fix potential fault at createDoubleObject
fd033e983 Remove semicolon after do-while in _EL_CLEANUP
664c415e7 Illumos test fixes, error message difference fot bad hostname test.

git-subtree-dir: deps/hiredis
git-subtree-split: b6a052fe0959dae69e16b9d74449faeb1b70dbe1
2023-05-30 22:23:45 +03:00
Oran AgraandGitHub c2f1815bcb Avoid trying to trim string loaded from RDB file. (#12241)
This is a followup fix for #11817
2023-05-30 10:43:25 +03:00
BinbinandGitHub 0dfb0250e6 Fix GETEX db delete call to emit DB_FLAG_KEY_EXPIRED on expiration (#12243)
We should emit DB_FLAG_KEY_EXPIRED instead of DB_FLAG_KEY_DELETED.
This is an overlook in #9406.
2023-05-29 15:39:32 +03:00
BinbinandGitHub da8f7428fa Try to fix SENTINEL SIMULATE-FAILURE test by re-source init-tests before each test (#12194)
This test was introduced in #12079, it works well most of the time, but
occasionally fails:
```
00:34:45> SENTINEL SIMULATE-FAILURE crash-after-election works: OK
00:34:45> SENTINEL SIMULATE-FAILURE crash-after-promotion works: FAILED: Sentinel set crash-after-promotion but did not exit
```

Don't know the reason, it may be affected by the exit of the previous
crash-after-election test. Because it doesn't really make much sense to
go deeper into it now, we re-source init-tests to get a clean environment
before each test, to try to fix this.

After applying this change, we found a new error:
```
16:39:33> SENTINEL SIMULATE-FAILURE crash-after-election works: FAILED: caught an error in the test couldn't open socket: connection refused
couldn't open socket: connection refused
```

I am guessing the sentinel triggers failover and exits before SENTINEL FAILOVER,
added a new || condition in wait_for_condition to fix it.
2023-05-29 13:43:26 +03:00
BinbinandGitHub 32f45215c3 Try lazyfree temp zset in ZUNION / ZINTER / ZDIFF and optimize ZINTERCARD to avoid create temp zset (#12229)
We check lazyfree_lazy_server_del in sunionDiffGenericCommand
to see if we need to lazyfree the temp set. Now do the same in
zunionInterDiffGenericCommand to lazyfree the temp zset.

This is a minor change, follow #5903. Also improved the comments.

Additionally, avoid creating unused zset object in ZINTERCARD,
results in some 10% performance improvement.
2023-05-29 09:55:17 +03:00
Oran AgraandGitHub 2764dc3768 Optimize MSETNX to avoid double lookup (#11944)
This is a redo of #11594 which got reverted in #11940
It improves performance by avoiding double lookup of the the key.
2023-05-28 10:58:29 +03:00
Oran AgraandGitHub 6117f28822 Fix WAIT for clients being blocked in a module command (#12220)
So far clients being blocked and unblocked by a module command would
update the c->woff variable and so WAIT was ineffective and got released
without waiting for the command actions to propagate.

This seems to have existed since forever, but not for RM_BlockClientOnKeys.

It is problematic though to know if the module did or didn't propagate
anything in that command, so for now, instead of adding an API, we'll
just update the woff to the latest offset when unblocking, this will
cause the client to possibly wait excessively, but that's not that bad.
2023-05-28 10:10:52 +03:00
David CARLIERandGitHub da46e926ac enable malloc_usable_size on haiku and dragonflybsd. (#12237) 2023-05-28 08:44:28 +03:00
1a188e4ed6 [BUG] Incorrect error msg for XREAD command (#12238)
XREAD only supports a special ID of $ and XREADGROUP only supports ^.
make sure not to suggest the wrong one when rerunning an error about unbalanced ID arguments

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-05-28 08:37:32 +03:00
zhenwei piandGitHub cb78acb865 Support maxiov per connection type (#12234)
Rather than a fixed iovcnt for connWritev, support maxiov per connection type instead.
A minor change to reduce memory for struct connection.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2023-05-28 08:35:27 +03:00
BinbinandGitHub e775b34e81 Add new loglevel nothing to sentinel-config.json (#12235)
It was missing in #12223, and the reply-schemas daily
was failing:
```
jsonschema.exceptions.ValidationError: 'nothing' is not valid under any of the given schemas

Failed validating 'oneOf' in schema[0]['properties']['loglevel']:
    {'oneOf': [{'const': 'debug'},
               {'const': 'verbose'},
               {'const': 'notice'},
               {'const': 'warning'},
               {'const': 'unknown'}]}

On instance['loglevel']:
    'nothing'
```
2023-05-26 09:12:23 +03:00
mojh7andGitHub 627c610f49 Fix typo functoin -> function (#12218)
functoin -> function
2023-05-24 15:03:17 -07:00
Oran AgraandGitHub 3ca451c46f Make a light weight version (default) of DEBUG HTSTATS (#12212)
The light version only shows the table sizes, while the pre-existing
version that shows chain length stats is reachable with the `full` argument.

This should allow looking into rehashing state, even on huge dicts, on
which we're afraid to run the command for fear of causing a server freeze.

Also, fix a possible overflow in dictGetStats.
2023-05-24 16:27:44 +03:00
Oran AgraandGitHub c871db24c4 CI to validate commands.def is up to date (#12227)
and update recent SENTINEL CONFIG changes.
2023-05-24 16:21:18 +03:00
judengandGitHub d71478a889 postpone the initialization of oject's lru&lfu until it is added to the db as a value object (#11626)
This pr can get two performance benefits:
1. Stop redundant initialization when most robj objects are created
2. LRU_CLOCK will no longer be called in io threads, so we can avoid the `atomicGet`

Another code optimization:
deleted the redundant judgment in dbSetValue, no matter in LFU or LRU, the lru field inold
robj is always the freshest (it is always updated in lookupkey), so we don't need to judge if in LFU
2023-05-24 09:40:11 +03:00
Wen HuiandGitHub d664889992 Adding test case for hvals, hkeys, hexists against wrong type (#12198)
HVALS, HKEYS and HEXISTS commands wrong type test cases were not covered so added the test cases.
2023-05-24 09:34:13 +03:00
BinbinandGitHub d0994c5bca Sync the new loglevel nothing to sentinel (#12223)
We add a new loglevel 'nothing' to disable logging in #12133.
This PR syncs that config change to sentinel. Because in #11214
we support modifying loglevel in runtime.

Although I think sentinel doesn't need this nothing config,
it's better to be consistent.
2023-05-24 09:32:39 +03:00
BinbinandGitHub ec5721d6ca Add dummy CLUSTER SLAVES call tests to fix reply ci (#12224)
In #12166, we removed a call to CLUSTER SLAVES, which
then caused reply-schemas ci to fail:
```
WARNING! The following commands were not hit at all:
  cluster|slaves
  ERROR! at least one command was not hit by the tests
```

Because we already have command output that cover CLUSTER REPLICAS
elsewhere, here we simply add some dummy tests to fix the ci.
2023-05-24 09:28:38 +03:00
Ping XieandGitHub 4c74dd986f Exclude aux fields from "cluster nodes" and "cluster replicas" output (#12166)
This commit excludes aux fields from the output of the `cluster nodes` and `cluster replicas` command.
We may decide to re-introduce them in some form or another in the future, but not in v7.2.
2023-05-23 18:32:37 +03:00
zhaozhao.zzandGitHub 07ea220419 add a new loglevel 'nothing' to disable logging (#12133)
Users can record logs of different levels by setting the `loglevel`.
However, sometimes there are many logs even at the warning level,
which can affect the performance of Redis.

For example, when a user accesses the tls-port using a non-encrypted link,
Redis will log lots of "# Error accepting a client connection: ...".

We can provide the ability to disable logging so that users can temporarily turn
off logging and turn it back on after the problem is resolved.
2023-05-23 18:30:44 +03:00
71e6abe423 Fix memory leak when RM_Call's RUN_AS_USER fails (#12158)
previously the argv wasn't freed so would leak.  not a common case, but should be handled.

Solution: move RUN_AS_USER setup and error exit to the right place.
this way, when we do `goto cleanup` (instead of return) it'll automatically do the right thing (including autoMemoryAdd)
Removed the user argument from moduleAllocTempClient (reverted to the state before 6e993a5)

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-05-23 14:29:28 +03:00
BinbinandGitHub 006ab26c37 Optimize HRANDFIELD and ZRANDMEMBER case 3 when listpack encoded (#12205)
Optimized HRANDFIELD and ZRANDMEMBER commands as in #8444,
CASE 3 under listpack encoding. Boost optimization to CASE 2.5. 

CASE 2.5 listpack only. Sampling unique elements, in non-random order.
Listpack encoded hashes / zsets are meant to be relatively small, so
HRANDFIELD_SUB_STRATEGY_MUL / ZRANDMEMBER_SUB_STRATEGY_MUL
isn't necessary and we rather not make copies of the entries. Instead, we
emit them directly to the output buffer.

Simple benchmarks shows it provides some 400% improvement in HRANDFIELD
and ZRANGESTORE both in CASE 3.

Unrelated changes: remove useless setTypeRandomElements and fix a typo.
2023-05-22 15:48:32 +03:00
38e284f106 optimize spopwithcount propagation (#12082)
A single SPOP with command with count argument resulted in many SPOP
commands being propagated to the replica.
This is inefficient because the key name is repeated many times, and is also
being looked-up many times.
also it results in high QPS metrics on the replica.
To solve that, we flush batches of 1024 fields per SPOP command.

Co-authored-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
2023-05-22 10:27:14 +03:00
BinbinandGitHub 48757934ff Performance improvement to ZADD and ZRANGESTORE, convert to skiplist and expand dict in advance (#12185)
For zsets that will eventually be stored as the skiplist encoding (has a dict),
we can convert it to skiplist ahead of time. This change checks the number
of arguments in the ZADD command, and converts the data-structure
if the number of new entries exceeds the listpack-max-entries configuration.
This can cause us to over-allocate memory if there are duplicate entries in the
input, which is unexpected.

For ZRANGESTORE, we know the size of the zset, so we can expand
the dict in advance, to avoid the temporary dict from being rehashed
while it grows.

Simple benchmarks shows it provides some 4% improvement in ZADD and 20% in ZRANGESTORE
2023-05-18 15:24:46 +03:00
Hanna FadidaandGitHub 37cf1984b9 Add BITFIELD_RO basic tests for non-repl use cases (#12187)
Current tests for BITFIELD_RO command are skipped in the external mode,
and therefore reply-schemas-validator reports a coverage error.
This PR adds basic tests to increase coverage.
2023-05-18 12:16:46 +03:00
BrennanandGitHub 40e6131ba5 Prevent repetitive backlog trimming (#12155)
When `replicationFeedSlaves()` serializes a command, it repeatedly calls
`feedReplicationBuffer()` to feed it to the replication backlog piece by piece.
It is unnecessary to call `incrementalTrimReplicationBacklog()` for every small
amount of data added with `feedReplicationBuffer()` as the chance of the conditions
being met for trimming are very low and these frequent calls add up to a notable
performance cost. Instead, we will only attempt trimming when a new block is added
to the replication backlog.

Using redis-benchmark to saturate a local redis server indicated a performance
improvement of around 3-3.5% for 100 byte SET commands with this change.
2023-05-18 09:25:56 +03:00
YaacovHazanandGitHub 49845c24b1 fix CMD_CALL_FROM_MODULE value (#12195)
CMD_CALL_FROM_MODULE overlapped CMD_CALL_REPROCESSING,
changing its value to (1<<3)
2023-05-18 09:23:21 +03:00
df1890ef7f Allow SENTINEL CONFIG SET and SENTINEL CONFIG GET to handle multiple parameters. (#10362)
Extend SENTINEL CONFIG SET and SENTINEL CONFIG GET to be
compatible with variadic CONFIG SET and CONFIG GET and allow multiple
parameters to be modified in a single call atomically.

Co-authored-by: Oran Agra <oran@redislabs.com>
2023-05-17 10:26:02 +03:00
BinbinandGitHub fd566f4050 Fix for set max entries edge case in setTypeCreate / setTypeMaybeConvert (#12183)
In the judgment in setTypeCreate, we should judge size_hint <= max_entries.

This results in the following inconsistencies:
```
127.0.0.1:6379> config set set-max-intset-entries 5 set-max-listpack-entries 5
OK

127.0.0.1:6379> sadd intset_set1 1 2 3 4 5
(integer) 5
127.0.0.1:6379> object encoding intset_set1
"hashtable"
127.0.0.1:6379> sadd intset_set2 1 2 3 4
(integer) 4
127.0.0.1:6379> sadd intset_set2 5
(integer) 1
127.0.0.1:6379> object encoding intset_set2
"intset"

127.0.0.1:6379> sadd listpack_set1 a 1 2 3 4
(integer) 5
127.0.0.1:6379> object encoding listpack_set1
"hashtable"
127.0.0.1:6379> sadd listpack_set2 a 1 2 3
(integer) 4
127.0.0.1:6379> sadd listpack_set2 4
(integer) 1
127.0.0.1:6379> object encoding listpack_set2
"listpack"
```

This was introduced in #12019, added corresponding tests.
2023-05-16 11:32:21 -07:00
478 changed files with 56507 additions and 17230 deletions
+1 -1
View File
@@ -1 +1 @@
codespell==2.2.4
codespell==2.2.5
+38 -14
View File
@@ -7,7 +7,7 @@ jobs:
test-ubuntu-latest:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
# Fail build if there are warnings
# build with TLS just for compilation coverage
@@ -17,27 +17,33 @@ jobs:
sudo apt-get install tcl8.6 tclx
./runtest --verbose --tags -slow --dump-logs
- name: module api test
run: ./runtest-moduleapi --verbose --dump-logs
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs
- name: validate commands.def up to date
run: |
touch src/commands/ping.json
make commands.def
dirty=$(git diff)
if [[ ! -z $dirty ]]; then echo $dirty; exit 1; fi
test-sanitizer-address:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
# build with TLS module just for compilation coverage
run: make SANITIZER=address REDIS_CFLAGS='-Werror' BUILD_TLS=module
run: make SANITIZER=address REDIS_CFLAGS='-Werror -DDEBUG_ASSERTIONS' BUILD_TLS=module
- name: testprep
run: sudo apt-get install tcl8.6 tclx -y
- name: test
run: ./runtest --verbose --tags -slow --dump-logs
- name: module api test
run: ./runtest-moduleapi --verbose --dump-logs
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs
build-debian-old:
runs-on: ubuntu-latest
container: debian:buster
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
run: |
apt-get update && apt-get install -y build-essential
@@ -46,33 +52,51 @@ jobs:
build-macos-latest:
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
run: make REDIS_CFLAGS='-Werror'
build-32bit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
run: |
sudo apt-get update && sudo apt-get install libc6-dev-i386
sudo apt-get update && sudo apt-get install libc6-dev-i386 gcc-multilib g++-multilib
make REDIS_CFLAGS='-Werror' 32bit
build-libc-malloc:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
run: make REDIS_CFLAGS='-Werror' MALLOC=libc
build-centos7-jemalloc:
build-centos-jemalloc:
runs-on: ubuntu-latest
container: centos:7
container: quay.io/centos/centos:stream9
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: make
run: |
yum -y install gcc make
dnf -y install which gcc gcc-c++ make
make REDIS_CFLAGS='-Werror'
build-old-chain-jemalloc:
runs-on: ubuntu-latest
container: ubuntu:20.04
steps:
- uses: actions/checkout@v4
- name: make
run: |
apt-get update
apt-get install -y gnupg2
echo "deb http://dk.archive.ubuntu.com/ubuntu/ xenial main" >> /etc/apt/sources.list
echo "deb http://dk.archive.ubuntu.com/ubuntu/ xenial universe" >> /etc/apt/sources.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 40976EAF437D05B5
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3B4FE6ACC0B21F32
apt-get update
apt-get install -y make gcc-4.8 g++-4.8
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 100
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 100
make CC=gcc REDIS_CFLAGS='-Werror'
+24
View File
@@ -0,0 +1,24 @@
name: "Codecov"
# Enabling on each push is to display the coverage changes in every PR,
# where each PR needs to be compared against the coverage of the head commit
on: [push, pull_request]
jobs:
code-coverage:
runs-on: ubuntu-22.04
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Install lcov and run test
run: |
sudo apt-get install lcov
make lcov
- name: Upload coverage reports to Codecov
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
file: ./src/redis.info
+4 -4
View File
@@ -19,15 +19,15 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Initialize CodeQL
uses: github/codeql-action/init@v2
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
- name: Autobuild
uses: github/codeql-action/autobuild@v2
uses: github/codeql-action/autobuild@v3
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v2
uses: github/codeql-action/analyze@v3
+32
View File
@@ -0,0 +1,32 @@
# Creates and uploads a Coverity build on a schedule
name: Coverity Scan
on:
schedule:
# Run once daily, since below 500k LOC can have 21 builds per week, per https://scan.coverity.com/faq#frequency
- cron: '0 0 * * *'
# Support manual execution
workflow_dispatch:
jobs:
coverity:
if: github.repository == 'redis/redis'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@main
- name: Download and extract the Coverity Build Tool
run: |
wget -q https://scan.coverity.com/download/cxx/linux64 --post-data "token=${{ secrets.COVERITY_SCAN_TOKEN }}&project=redis-unstable" -O cov-analysis-linux64.tar.gz
mkdir cov-analysis-linux64
tar xzf cov-analysis-linux64.tar.gz --strip 1 -C cov-analysis-linux64
- name: Install Redis dependencies
run: sudo apt install -y gcc tcl8.6 tclx procps libssl-dev
- name: Build with cov-build
run: cov-analysis-linux64/bin/cov-build --dir cov-int make
- name: Upload the result
run: |
tar czvf cov-int.tgz cov-int
curl \
--form project=redis-unstable \
--form email=${{ secrets.COVERITY_SCAN_EMAIL }} \
--form token=${{ secrets.COVERITY_SCAN_TOKEN }} \
--form file=@cov-int.tgz \
https://scan.coverity.com/builds
+270 -146
View File
@@ -11,7 +11,7 @@ on:
inputs:
skipjobs:
description: 'jobs to skip (delete the ones you wanna keep, do not leave empty)'
default: 'valgrind,sanitizer,tls,freebsd,macos,alpine,32bit,iothreads,ubuntu,centos,malloc,specific,fortify,reply-schema'
default: 'valgrind,sanitizer,tls,freebsd,macos,alpine,32bit,iothreads,ubuntu,centos,malloc,specific,fortify,reply-schema,oldTC'
skiptests:
description: 'tests to skip (delete the ones you wanna keep, do not leave empty)'
default: 'redis,modules,sentinel,cluster,unittest'
@@ -47,7 +47,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -60,7 +60,7 @@ jobs:
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: ./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
@@ -76,7 +76,6 @@ jobs:
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'fortify')
container: ubuntu:lunar
timeout-minutes: 14400
steps:
- name: prep
@@ -88,23 +87,22 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
apt-get update && apt-get install -y make gcc-13
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 100
apt-get update && apt-get install -y make gcc g++
make CC=gcc REDIS_CFLAGS='-Werror -DREDIS_TEST -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3'
- name: testprep
run: apt-get install -y tcl8.6 tclx procps
run: sudo apt-get install -y tcl8.6 tclx procps
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: ./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
@@ -131,7 +129,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -144,7 +142,7 @@ jobs:
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: ./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
@@ -168,7 +166,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -181,7 +179,7 @@ jobs:
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: ./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
@@ -205,13 +203,13 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
sudo apt-get update && sudo apt-get install libc6-dev-i386
sudo apt-get update && sudo apt-get install libc6-dev-i386 g++ gcc-multilib g++-multilib
make 32bit REDIS_CFLAGS='-Werror -DREDIS_TEST'
- name: testprep
run: sudo apt-get install tcl8.6 tclx
@@ -222,7 +220,7 @@ jobs:
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: |
make -C tests/modules 32bit # the script below doesn't have an argument, we must build manually ahead of time
./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: ./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
@@ -249,7 +247,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -267,7 +265,7 @@ jobs:
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: |
./runtest-moduleapi --verbose --dump-logs --tls --dump-logs ${{github.event.inputs.test_args}}
CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs --tls --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: |
@@ -293,7 +291,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -311,7 +309,7 @@ jobs:
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: |
./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: |
@@ -337,7 +335,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -369,7 +367,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -447,7 +445,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -456,7 +454,7 @@ jobs:
- name: testprep
run: |
sudo apt-get update
sudo apt-get install tcl8.6 tclx valgrind -y
sudo apt-get install tcl8.6 tclx valgrind g++ -y
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
run: ./runtest --valgrind --no-latency --verbose --clients 1 --timeout 2400 --dump-logs ${{github.event.inputs.test_args}}
@@ -477,7 +475,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -489,7 +487,7 @@ jobs:
sudo apt-get install tcl8.6 tclx valgrind -y
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: ./runtest-moduleapi --valgrind --no-latency --verbose --clients 1 --timeout 2400 --dump-logs ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --valgrind --no-latency --verbose --clients 1 --timeout 2400 --dump-logs ${{github.event.inputs.test_args}}
- name: unittest
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: |
@@ -512,7 +510,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -521,7 +519,7 @@ jobs:
- name: testprep
run: |
sudo apt-get update
sudo apt-get install tcl8.6 tclx valgrind -y
sudo apt-get install tcl8.6 tclx valgrind g++ -y
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
run: ./runtest --valgrind --no-latency --verbose --clients 1 --timeout 2400 --dump-logs ${{github.event.inputs.test_args}}
@@ -542,7 +540,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -554,7 +552,7 @@ jobs:
sudo apt-get install tcl8.6 tclx valgrind -y
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: ./runtest-moduleapi --valgrind --no-latency --verbose --clients 1 --timeout 2400 --dump-logs ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --valgrind --no-latency --verbose --clients 1 --timeout 2400 --dump-logs ${{github.event.inputs.test_args}}
- name: unittest
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: |
@@ -582,12 +580,12 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: make SANITIZER=address REDIS_CFLAGS='-DREDIS_TEST -Werror'
run: make SANITIZER=address REDIS_CFLAGS='-DREDIS_TEST -Werror -DDEBUG_ASSERTIONS'
- name: testprep
run: |
sudo apt-get update
@@ -597,7 +595,7 @@ jobs:
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: ./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
@@ -629,7 +627,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -644,7 +642,7 @@ jobs:
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: ./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
@@ -655,12 +653,12 @@ jobs:
if: true && !contains(github.event.inputs.skiptests, 'unittest')
run: ./src/redis-server test all --accurate
test-centos7-jemalloc:
test-centos-jemalloc:
runs-on: ubuntu-latest
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'centos')
container: centos:7
container: quay.io/centos/centos:stream9
timeout-minutes: 14400
steps:
- name: prep
@@ -672,22 +670,24 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
yum -y install gcc make
dnf -y install which gcc make g++
make REDIS_CFLAGS='-Werror'
- name: testprep
run: yum -y install which tcl tclx
run: |
dnf -y install epel-release
dnf -y install tcl tcltls procps-ng /usr/bin/kill
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: ./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
@@ -695,12 +695,12 @@ jobs:
if: true && !contains(github.event.inputs.skiptests, 'cluster')
run: ./runtest-cluster ${{github.event.inputs.cluster_test_args}}
test-centos7-tls-module:
test-centos-tls-module:
runs-on: ubuntu-latest
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'tls')
container: centos:7
container: quay.io/centos/centos:stream9
timeout-minutes: 14400
steps:
- name: prep
@@ -712,18 +712,18 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
yum -y install centos-release-scl epel-release
yum -y install devtoolset-7 openssl-devel openssl
scl enable devtoolset-7 "make BUILD_TLS=module REDIS_CFLAGS='-Werror'"
dnf -y install which gcc make openssl-devel openssl g++
make BUILD_TLS=module REDIS_CFLAGS='-Werror'
- name: testprep
run: |
yum -y install tcl tcltls tclx
dnf -y install epel-release
dnf -y install tcl tcltls procps-ng /usr/bin/kill
./utils/gen-test-certs.sh
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
@@ -732,7 +732,7 @@ jobs:
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: |
./runtest-moduleapi --verbose --dump-logs --tls-module --dump-logs ${{github.event.inputs.test_args}}
CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs --tls-module --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: |
@@ -742,12 +742,12 @@ jobs:
run: |
./runtest-cluster --tls-module ${{github.event.inputs.cluster_test_args}}
test-centos7-tls-module-no-tls:
test-centos-tls-module-no-tls:
runs-on: ubuntu-latest
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'tls')
container: centos:7
container: quay.io/centos/centos:stream9
timeout-minutes: 14400
steps:
- name: prep
@@ -759,18 +759,18 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
yum -y install centos-release-scl epel-release
yum -y install devtoolset-7 openssl-devel openssl
scl enable devtoolset-7 "make BUILD_TLS=module REDIS_CFLAGS='-Werror'"
dnf -y install which gcc make openssl-devel openssl g++
make BUILD_TLS=module REDIS_CFLAGS='-Werror'
- name: testprep
run: |
yum -y install tcl tcltls tclx
dnf -y install epel-release
dnf -y install tcl tcltls procps-ng /usr/bin/kill
./utils/gen-test-certs.sh
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
@@ -779,7 +779,7 @@ jobs:
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: |
./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: |
@@ -805,7 +805,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -813,10 +813,10 @@ jobs:
run: make REDIS_CFLAGS='-Werror'
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
run: ./runtest --accurate --verbose --verbose --clients 1 --no-latency --dump-logs ${{github.event.inputs.test_args}}
run: ./runtest --accurate --verbose --clients 1 --no-latency --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: ./runtest-moduleapi --verbose --verbose --clients 1 --no-latency --dump-logs ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --clients 1 --no-latency --dump-logs ${{github.event.inputs.test_args}}
test-macos-latest-sentinel:
runs-on: macos-latest
@@ -834,7 +834,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -860,7 +860,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -870,99 +870,65 @@ jobs:
if: true && !contains(github.event.inputs.skiptests, 'cluster')
run: ./runtest-cluster ${{github.event.inputs.cluster_test_args}}
build-macos:
strategy:
matrix:
os: [macos-13, macos-15]
runs-on: ${{ matrix.os }}
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'macos')
timeout-minutes: 14400
steps:
- uses: maxim-lobanov/setup-xcode@v1
with:
xcode-version: latest
- name: prep
if: github.event_name == 'workflow_dispatch'
run: |
echo "GITHUB_REPOSITORY=${{github.event.inputs.use_repo}}" >> $GITHUB_ENV
echo "GITHUB_HEAD_REF=${{github.event.inputs.use_git_ref}}" >> $GITHUB_ENV
echo "skipjobs: ${{github.event.inputs.skipjobs}}"
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: make REDIS_CFLAGS='-Werror -DREDIS_TEST'
test-freebsd:
runs-on: macos-12
runs-on: macos-13
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'freebsd') && !(contains(github.event.inputs.skiptests, 'redis') && contains(github.event.inputs.skiptests, 'modules'))
!contains(github.event.inputs.skipjobs, 'freebsd')
timeout-minutes: 14400
env:
CC: clang
CXX: clang++
steps:
- name: prep
if: github.event_name == 'workflow_dispatch'
run: |
echo "GITHUB_REPOSITORY=${{github.event.inputs.use_repo}}" >> $GITHUB_ENV
echo "GITHUB_HEAD_REF=${{github.event.inputs.use_git_ref}}" >> $GITHUB_ENV
echo "skipjobs: ${{github.event.inputs.skipjobs}}"
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: test
uses: vmactions/freebsd-vm@v0.3.0
uses: cross-platform-actions/action@v0.22.0
with:
usesh: true
sync: rsync
copyback: false
prepare: pkg install -y bash gmake lang/tcl86 lang/tclx
run: >
gmake || exit 1 ;
if echo "${{github.event.inputs.skiptests}}" | grep -vq redis ; then ./runtest --verbose --timeout 2400 --no-latency --dump-logs ${{github.event.inputs.test_args}} || exit 1 ; fi ;
if echo "${{github.event.inputs.skiptests}}" | grep -vq modules ; then MAKE=gmake ./runtest-moduleapi --verbose --timeout 2400 --no-latency --dump-logs ${{github.event.inputs.test_args}} || exit 1 ; fi ;
test-freebsd-sentinel:
runs-on: macos-12
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'freebsd') && !contains(github.event.inputs.skiptests, 'sentinel')
timeout-minutes: 14400
steps:
- name: prep
if: github.event_name == 'workflow_dispatch'
run: |
echo "GITHUB_REPOSITORY=${{github.event.inputs.use_repo}}" >> $GITHUB_ENV
echo "GITHUB_HEAD_REF=${{github.event.inputs.use_git_ref}}" >> $GITHUB_ENV
echo "skipjobs: ${{github.event.inputs.skipjobs}}"
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: test
uses: vmactions/freebsd-vm@v0.3.0
with:
usesh: true
sync: rsync
copyback: false
prepare: pkg install -y bash gmake lang/tcl86 lang/tclx
run: >
gmake || exit 1 ;
if echo "${{github.event.inputs.skiptests}}" | grep -vq sentinel ; then ./runtest-sentinel ${{github.event.inputs.cluster_test_args}} || exit 1 ; fi ;
test-freebsd-cluster:
runs-on: macos-12
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'freebsd') && !contains(github.event.inputs.skiptests, 'cluster')
timeout-minutes: 14400
steps:
- name: prep
if: github.event_name == 'workflow_dispatch'
run: |
echo "GITHUB_REPOSITORY=${{github.event.inputs.use_repo}}" >> $GITHUB_ENV
echo "GITHUB_HEAD_REF=${{github.event.inputs.use_git_ref}}" >> $GITHUB_ENV
echo "skipjobs: ${{github.event.inputs.skipjobs}}"
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: test
uses: vmactions/freebsd-vm@v0.3.0
with:
usesh: true
sync: rsync
copyback: false
prepare: pkg install -y bash gmake lang/tcl86 lang/tclx
run: >
gmake || exit 1 ;
if echo "${{github.event.inputs.skiptests}}" | grep -vq cluster ; then ./runtest-cluster ${{github.event.inputs.cluster_test_args}} || exit 1 ; fi ;
operating_system: freebsd
environment_variables: MAKE
version: 13.2
shell: bash
run: |
sudo pkg install -y bash gmake lang/tcl86 lang/tclx gcc
gmake
./runtest --single unit/keyspace --single unit/auth --single unit/networking --single unit/protocol
test-alpine-jemalloc:
runs-on: ubuntu-latest
@@ -980,7 +946,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -995,7 +961,7 @@ jobs:
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: ./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
@@ -1019,7 +985,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -1034,7 +1000,7 @@ jobs:
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: ./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
@@ -1058,7 +1024,7 @@ jobs:
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
@@ -1071,7 +1037,7 @@ jobs:
run: ./runtest --log-req-res --no-latency --dont-clean --force-resp3 --tags -slow --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: ./runtest-moduleapi --log-req-res --no-latency --dont-clean --force-resp3 --dont-pre-clean --verbose --dump-logs ${{github.event.inputs.test_args}}
run: CFLAGS='-Werror' ./runtest-moduleapi --log-req-res --no-latency --dont-clean --force-resp3 --dont-pre-clean --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: ./runtest-sentinel --log-req-res --dont-clean --force-resp3 ${{github.event.inputs.cluster_test_args}}
@@ -1085,3 +1051,161 @@ jobs:
- name: validator
run: ./utils/req-res-log-validator.py --verbose --fail-missing-reply-schemas ${{ (!contains(github.event.inputs.skiptests, 'redis') && !contains(github.event.inputs.skiptests, 'module') && !contains(github.event.inputs.sentinel, 'redis') && !contains(github.event.inputs.skiptests, 'cluster')) && github.event.inputs.test_args == '' && github.event.inputs.cluster_test_args == '' && '--fail-commands-not-all-hit' || '' }}
test-old-chain-jemalloc:
runs-on: ubuntu-latest
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'oldTC')
container: ubuntu:20.04
timeout-minutes: 14400
steps:
- name: prep
if: github.event_name == 'workflow_dispatch'
run: |
echo "GITHUB_REPOSITORY=${{github.event.inputs.use_repo}}" >> $GITHUB_ENV
echo "GITHUB_HEAD_REF=${{github.event.inputs.use_git_ref}}" >> $GITHUB_ENV
echo "skipjobs: ${{github.event.inputs.skipjobs}}"
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
apt-get update
apt-get install -y gnupg2
echo "deb http://dk.archive.ubuntu.com/ubuntu/ xenial main" >> /etc/apt/sources.list
echo "deb http://dk.archive.ubuntu.com/ubuntu/ xenial universe" >> /etc/apt/sources.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 40976EAF437D05B5
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3B4FE6ACC0B21F32
apt-get update
apt-get install -y make gcc-4.8 g++-4.8
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 100
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 100
make CC=gcc REDIS_CFLAGS='-Werror'
- name: testprep
run: apt-get install -y tcl tcltls tclx
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: ./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
- name: cluster tests
if: true && !contains(github.event.inputs.skiptests, 'cluster')
run: ./runtest-cluster ${{github.event.inputs.cluster_test_args}}
test-old-chain-tls-module:
runs-on: ubuntu-latest
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'tls') && !contains(github.event.inputs.skipjobs, 'oldTC')
container: ubuntu:20.04
timeout-minutes: 14400
steps:
- name: prep
if: github.event_name == 'workflow_dispatch'
run: |
echo "GITHUB_REPOSITORY=${{github.event.inputs.use_repo}}" >> $GITHUB_ENV
echo "GITHUB_HEAD_REF=${{github.event.inputs.use_git_ref}}" >> $GITHUB_ENV
echo "skipjobs: ${{github.event.inputs.skipjobs}}"
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
apt-get update
apt-get install -y gnupg2
echo "deb http://dk.archive.ubuntu.com/ubuntu/ xenial main" >> /etc/apt/sources.list
echo "deb http://dk.archive.ubuntu.com/ubuntu/ xenial universe" >> /etc/apt/sources.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 40976EAF437D05B5
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3B4FE6ACC0B21F32
apt-get update
apt-get install -y make gcc-4.8 g++-4.8 openssl libssl-dev
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 100
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 100
make CC=gcc CXX=g++ BUILD_TLS=module REDIS_CFLAGS='-Werror'
- name: testprep
run: |
apt-get install -y tcl tcltls tclx
./utils/gen-test-certs.sh
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
run: |
./runtest --accurate --verbose --dump-logs --tls-module --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: |
CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs --tls-module --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: |
./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
- name: cluster tests
if: true && !contains(github.event.inputs.skiptests, 'cluster')
run: |
./runtest-cluster --tls-module ${{github.event.inputs.cluster_test_args}}
test-old-chain-tls-module-no-tls:
runs-on: ubuntu-latest
if: |
(github.event_name == 'workflow_dispatch' || (github.event_name != 'workflow_dispatch' && github.repository == 'redis/redis')) &&
!contains(github.event.inputs.skipjobs, 'tls') && !contains(github.event.inputs.skipjobs, 'oldTC')
container: ubuntu:20.04
timeout-minutes: 14400
steps:
- name: prep
if: github.event_name == 'workflow_dispatch'
run: |
echo "GITHUB_REPOSITORY=${{github.event.inputs.use_repo}}" >> $GITHUB_ENV
echo "GITHUB_HEAD_REF=${{github.event.inputs.use_git_ref}}" >> $GITHUB_ENV
echo "skipjobs: ${{github.event.inputs.skipjobs}}"
echo "skiptests: ${{github.event.inputs.skiptests}}"
echo "test_args: ${{github.event.inputs.test_args}}"
echo "cluster_test_args: ${{github.event.inputs.cluster_test_args}}"
- uses: actions/checkout@v4
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- name: make
run: |
apt-get update
apt-get install -y gnupg2
echo "deb http://dk.archive.ubuntu.com/ubuntu/ xenial main" >> /etc/apt/sources.list
echo "deb http://dk.archive.ubuntu.com/ubuntu/ xenial universe" >> /etc/apt/sources.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 40976EAF437D05B5
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3B4FE6ACC0B21F32
apt-get update
apt-get install -y make gcc-4.8 g++-4.8 openssl libssl-dev
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 100
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 100
make BUILD_TLS=module CC=gcc REDIS_CFLAGS='-Werror'
- name: testprep
run: |
apt-get install -y tcl tcltls tclx
./utils/gen-test-certs.sh
- name: test
if: true && !contains(github.event.inputs.skiptests, 'redis')
run: |
./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: |
CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: sentinel tests
if: true && !contains(github.event.inputs.skiptests, 'sentinel')
run: |
./runtest-sentinel ${{github.event.inputs.cluster_test_args}}
- name: cluster tests
if: true && !contains(github.event.inputs.skiptests, 'cluster')
run: |
./runtest-cluster ${{github.event.inputs.cluster_test_args}}
+14 -11
View File
@@ -12,7 +12,7 @@ jobs:
if: github.event_name != 'schedule' || github.repository == 'redis/redis'
timeout-minutes: 14400
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Build
run: make REDIS_CFLAGS=-Werror
- name: Start redis-server
@@ -23,10 +23,11 @@ jobs:
run: |
./runtest \
--host 127.0.0.1 --port 6379 \
--verbose \
--tags -slow
- name: Archive redis log
if: ${{ failure() }}
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: test-external-redis-log
path: external-redis.log
@@ -36,12 +37,12 @@ jobs:
if: github.event_name != 'schedule' || github.repository == 'redis/redis'
timeout-minutes: 14400
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Build
run: make REDIS_CFLAGS=-Werror
- name: Start redis-server
run: |
./src/redis-server --cluster-enabled yes --daemonize yes --save "" --logfile external-redis.log \
./src/redis-server --cluster-enabled yes --daemonize yes --save "" --logfile external-redis-cluster.log \
--enable-protected-configs yes --enable-debug-command yes --enable-module-command yes
- name: Create a single node cluster
run: ./src/redis-cli cluster addslots $(for slot in {0..16383}; do echo $slot; done); sleep 5
@@ -49,34 +50,36 @@ jobs:
run: |
./runtest \
--host 127.0.0.1 --port 6379 \
--verbose \
--cluster-mode \
--tags -slow
- name: Archive redis log
if: ${{ failure() }}
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: test-external-cluster-log
path: external-redis.log
path: external-redis-cluster.log
test-external-nodebug:
runs-on: ubuntu-latest
if: github.event_name != 'schedule' || github.repository == 'redis/redis'
timeout-minutes: 14400
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Build
run: make REDIS_CFLAGS=-Werror
- name: Start redis-server
run: |
./src/redis-server --daemonize yes --save "" --logfile external-redis.log
./src/redis-server --daemonize yes --save "" --logfile external-redis-nodebug.log
- name: Run external test
run: |
./runtest \
--host 127.0.0.1 --port 6379 \
--verbose \
--tags "-slow -needs:debug"
- name: Archive redis log
if: ${{ failure() }}
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: test-external-redis-log
path: external-redis.log
name: test-external-redis-nodebug-log
path: external-redis-nodebug.log
+35
View File
@@ -0,0 +1,35 @@
name: redis_docs_sync
on:
release:
types: [published]
jobs:
redis_docs_sync:
if: github.repository == 'redis/redis'
runs-on: ubuntu-latest
steps:
- name: Generate a token
id: generate-token
uses: actions/create-github-app-token@v1
with:
app-id: ${{ secrets.DOCS_APP_ID }}
private-key: ${{ secrets.DOCS_APP_PRIVATE_KEY }}
- name: Invoke workflow on redis/docs
env:
GH_TOKEN: ${{ steps.generate-token.outputs.token }}
RELEASE_NAME: ${{ github.event.release.tag_name }}
run: |
LATEST_RELEASE=$(
curl -Ls \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer ${GH_TOKEN}" \
-H "X-GitHub-Api-Version: 2022-11-28" \
https://api.github.com/repos/redis/redis/releases/latest \
| jq -r '.tag_name'
)
if [[ "${LATEST_RELEASE}" == "${RELEASE_NAME}" ]]; then
gh workflow run -R redis/docs redis_docs_sync.yaml -f release="${RELEASE_NAME}"
fi
+2 -2
View File
@@ -12,9 +12,9 @@ jobs:
reply-schemas-linter:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Setup nodejs
uses: actions/setup-node@v3
uses: actions/setup-node@v4
- name: Install packages
run: npm install ajv
- name: linter
+2 -2
View File
@@ -16,10 +16,10 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: pip cache
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
+1
View File
@@ -30,6 +30,7 @@ deps/lua/src/luac
deps/lua/src/liblua.a
deps/hdr_histogram/libhdrhistogram.a
deps/fpconv/libfpconv.a
deps/fast_float/libfast_float.a
tests/tls/*
.make-*
.prerequisites
+244 -11
View File
@@ -1,16 +1,249 @@
Hello! This file is just a placeholder, since this is the "unstable" branch
of Redis, the place where all the development happens.
Redis Community Edition 8.0 release notes
=========================================
There is no release notes for this branch, it gets forked into another branch
every time there is a partial feature freeze in order to eventually create
a new stable release.
============================================================
8.0-M04 (v7.9.227) Committed Sun 16 Mar 2025 11:00:00 IST
============================================================
Usually "unstable" is stable enough for you to use it in development environments
however you should never use it in production environments. It is possible
to download the latest stable release here:
This is the fourth Milestone of Redis Community Edition 8.0.
https://download.redis.io/redis-stable.tar.gz
Milestones are non-feature-complete pre-releases. Pre-releases are not suitable for production use.
Once we reach feature-completeness we will release RC1.
More information is available at https://redis.io
### Headlines
8.0-M04 includes 3 new hash commands, performance improvements, and memory defragmentation improvements.
### Distributions
- Alpine and Debian Docker images - https://hub.docker.com/_/redis
- Install using snap - see https://github.com/redis/redis-snap
- Install using brew - see https://github.com/redis/homebrew-redis
- Install using RPM and Debian APT - will be added on the GA release
### New Features
- #13798 Hash - new commands:
- `HGETDEL` Get and delete the value of one or more fields of a given hash key
- `HGETEX` Get the value of one or more fields of a given hash key, and optionally set their expiration
- `HSETEX` Set the value of one or more fields of a given hash key, and optionally set their expiration
- #13773 Add replication offset to AOF, allowing more robust way to determine which AOF has a more up-to-date data during recovery
- #13740, #13763 shared secret - new mechanism to allow sending internal commands between nodes
### Bug fixes
- #13804 Overflow on 32-bit systems when calculating idle time for eviction
- #13793 `WAITAOF` returns prematurely
- #13800 Remove `DENYOOM` from `HEXPIRE`, `HEXPIREAT`, `HPEXPIRE`, and `HPEXPIREAT`
- #13632 Streams - wrong behavior of `XREAD +` after last entry
### Modules API
- #13788 `RedisModule_LoadDefaultConfigs` - load module configuration values from redis.conf
- #13815 `RM_RegisterDefragFunc2` - support for incremental defragmentation of global module data
- #13816 `RM_DefragRedisModuleDict` - allow modules to defrag `RedisModuleDict`
- #13774 `RM_GetContextFlags` - add a `REDISMODULE_CTX_FLAGS_DEBUG_ENABLED` flag to execute debug commands
### Performance and resource utilization improvements
- #13752 Reduce defrag CPU usage when defragmentation is ineffective
- #13764 Reduce latency when a command is called consecutively
- #13787 Optimize parsing data from clients, specifically multi-bulk (array) data
- #13792 Optimize dictionary lookup by avoiding duplicate key length calculation during comparisons
- #13796 Optimize expiration checks
============================================================
8.0-M03 (v7.9.226) Committed Mon 20 Jan 2025 15:00:00 IST
============================================================
This is the third Milestone of Redis Community Edition 8.0.
Milestones are non-feature-complete pre-releases. Pre-releases are not suitable for production use.
Once we reach feature-completeness we will release RC1.
### Headlines:
8.0-M03 introduces an improved replication mechanism which is more performant and robust, a new I/O threading implementation which enables throughput increase on multi-core environments, and many additional performance improvements. Both Alpine and Debian Docker images are now available on [Docker Hub](https://hub.docker.com/_/redis). A snap and Homebrew distributions are available as well.
### Security fixes
- (CVE-2024-46981) Lua script may lead to remote code execution
- (CVE-2024-51741) Denial-of-service due to malformed ACL selectors
### New Features
- #13695 New I/O threading implementation
- #13732 New replication mechanism
### Bug fixes
- #13653 `MODULE LOADEX` - crash on nonexistent parameter name
- #13661 `FUNCTION FLUSH` - memory leak when using jemalloc
- #13626 Memory leak on failed RDB loading
### Other general improvements
- #13639 When `hide-user-data-from-log` is enabled - also print command tokens on crash
- #13660 Add the Lua VM memory to memory overhead
### New metrics
- #13592 `INFO` - new `KEYSIZES` section includes key size distributions for basic data types
- #13695 `INFO` - new `Threads` section includes I/O threading metrics
### Modules API
- #13666 `RedisModule_ACLCheckKeyPrefixPermissions` - check access permissions to any key matching a given prefix
- #13676 `RedisModule_HashFieldMinExpire` - query the minimum expiration time over all the hashs fields
- #13676 `RedisModule_HashGet` - new `REDISMODULE_HASH_EXPIRE_TIME` flag - query the field expiration time
- #13656 `RedisModule_RegisterXXXConfig` - allow registering unprefixed configuration parameters
### Configuration parameters
- `replica-full-sync-buffer-limit` - maximum size of accumulated replication stream data on the replica side
- `io-threads-do-reads` is no longer effective. The new I/O threading implementation always use threads for both reads and writes
### Performance and resource utilization improvements
- #13638 Optimize CRC64 performance
- #13521 Optimize commands with large argument count - reuse c->argv after command execution
- #13558 Optimize `PFCOUNT` and `PFMERGE` - SIMD acceleration
- #13644 Optimize `GET` on high pipeline use-cases
- #13646 Optimize `EXISTS` - prefetching and branch prediction hints
- #13652 Optimize `LRANGE` - improve listpack handling and decoding efficiency
- #13655 Optimize `HSET` - avoid unnecessary hash field creation or deletion
- #13721 Optimize `LRANGE` and `HGETALL` - refactor client write preparation and handling
============================================================
8.0-M02 (v7.9.225) Committed Mon 28 Oct 2024 14:00:00 IST
============================================================
This is the second Milestone of Redis Community Edition 8.0.
Milestones are non-feature-complete pre-releases. Pre-releases are not suitable for production use.
Once we reach feature-completeness we will release RC1.
### Headlines:
8.0-M02 introduces significant performance improvements. Both Alpine and Debian Docker images are now available on [Docker Hub](https://hub.docker.com/_/redis). Additional distributions will be introduced in upcoming pre-releases.
### Supported upgrade paths (by replication or persistence) to 8.0-M02
- From previous Redis versions, without modules
The following upgrade paths (by replication or persistence) to 8.0-M02 are not yet tested and will be introduced in upcoming pre-releases:
- From previous Redis versions with modules (RediSearch, RedisJSON, RedisTimeSeries, RedisBloom)
- From Redis Stack 7.2 or 7.4
### Security fixes
- (CVE-2024-31449) Lua library commands may lead to stack overflow and potential RCE.
- (CVE-2024-31227) Potential Denial-of-service due to malformed ACL selectors.
- (CVE-2024-31228) Potential Denial-of-service due to unbounded pattern matching.
### Bug fixes
- #13539 Hash: Fix key ref for a hash that no longer has fields with expiration on `RENAME`/`MOVE`/`SWAPDB`/`RESTORE`
- #13512 Fix `TOUCH` command from a script in no-touch mode
- #13468 Cluster: Fix cluster node config corruption caused by mixing shard-id and non-shard-id versions
- #13608 Cluster: Fix `GET #` option in `SORT` command
### Modules API
- #13526 Extend `RedisModule_OpenKey` to read also expired keys and subkeys
### Performance and resource utilization improvements
- #11884 Optimize `ZADD` and `ZRANGE*` commands
- #13530 Optimize `SSCAN` command in case of listpack or intset encoding
- #13531 Optimize `HSCAN`/`ZSCAN` command in case of listpack encoding
- #13520 Optimize commands that heavily rely on bulk/mbulk replies (example of `LRANGE`)
- #13566 Optimize `ZUNION[STORE]` by avoiding redundant temporary dict usage
- #13567 Optimize `SUNION`/`SDIFF` commands by avoiding redundant temporary dict usage
- #11533 Avoid redundant `lpGet` to boost `quicklistCompare`
- #13412 Reduce redundant call of `prepareClientToWrite` when call `addReply*` continuously
===========================================================
8.0-M01 (v7.9.224) Released Thu 12 Sep 2024 10:00:00 IST
===========================================================
This is the first Milestone of Redis Community Edition 8.0.
Milestones are non-feature-complete pre-releases. Pre-releases are not suitable for production use.
Once we reach feature-completeness we will release RC1.
### Headlines:
Redis 8.0 introduces new data structures: JSON, time series, and 5 probabilistic data structures (previously available as separate Redis modules) and incorporates the enhanced Redis Query Enginer (with vector search).
8.0-M01 is available as a Docker image and can be downloaded from [Docker Hub](https://hub.docker.com/_/redis). Additional distributions will be introduced in upcoming pre-releases.
### Supported upgrade paths (by replication or persistence) to 8.0-M01
- From previous Redis versions, without modules
The following upgrade paths (by replication or persistence) to 8.0-M01 are not yet tested and will be introduced in upcoming pre-releases:
- From previous Redis versions with modules (RediSearch, RedisJSON, RedisTimeSeries, RedisBloom)
- From Redis Stack 7.2 or 7.4
### New Features in binary distributions
- 7 new data structures: JSON, Time series, Bloom filter, Cuckoo filter, Count-min sketch, Top-k, t-digest
- The enhanced Redis Query Engine (with vector search)
### Potentially breaking changes
- #12272 `GETRANGE` returns an empty bulk when the negative end index is out of range
- #12395 Optimize `SCAN` command when matching data type
### Bug fixes
- #13510 Fix `RM_RdbLoad` to enable AOF after RDB loading is completed
- #13489 `ACL CAT` - return module commands
- #13476 Fix a race condition in the `cache_memory` of `functionsLibCtx`
- #13473 Fix incorrect lag due to trimming stream via `XTRIM` command
- #13338 Fix incorrect lag field in `XINFO` when tombstone is after the `last_id` of the consume group
- #13470 On `HDEL` of last field - update the global hash field expiration data structure
- #13465 Cluster: Pass extensions to node if extension processing is handled by it
- #13443 Cluster: Ensure validity of myself when loading cluster config
- #13422 Cluster: Fix `CLUSTER SHARDS` command returns empty array
### Modules API
- #13509 New API calls: `RM_DefragAllocRaw`, `RM_DefragFreeRaw`, and `RM_RegisterDefragCallbacks` - defrag API to allocate and free raw memory
### Performance and resource utilization improvements
- #13503 Avoid overhead of comparison function pointer calls in listpack `lpFind`
- #13505 Optimize `STRING` datatype write commands
- #13499 Optimize `SMEMBERS` command
- #13494 Optimize `GEO*` commands reply
- #13490 Optimize `HELLO` command
- #13488 Optimize client query buffer
- #12395 Optimize `SCAN` command when matching data type
- #13529 Optimize `LREM`, `LPOS`, `LINSERT`, and `LINDEX` commands
- #13516 Optimize `LRANGE` and other commands that perform several writes to client buffers per call
- #13431 Avoid `used_memory` contention when updating from multiple threads
### Other general improvements
- #13495 Reply `-LOADING` on replica while flushing the db
### CLI tools
- #13411 redis-cli: Fix wrong `dbnum` showed after the client reconnected
### Notes
- No backward compatibility for replication or persistence.
- Additional distributions, upgrade paths, features, and improvements will be introduced in upcoming pre-releases.
- With the GA release of 8.0 we will deprecate Redis Stack.
Happy hacking!
+2 -2
View File
@@ -26,7 +26,7 @@ Examples of unacceptable behavior include:
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others private information, such as a physical or email
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
@@ -89,7 +89,7 @@ Attribution
This Code of Conduct is adapted from the Contributor Covenant,
version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
Community Impact Guidelines were inspired by Mozillas code of conduct
Community Impact Guidelines were inspired by Mozilla's code of conduct
enforcement ladder.
For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
+80 -18
View File
@@ -1,20 +1,82 @@
Note: by contributing code to the Redis project in any form, including sending
a pull request via Github, a code fragment or patch via private email or
public discussion groups, you agree to release your code under the terms
of the BSD license that you can find in the COPYING file included in the Redis
source distribution. You will include BSD license in the COPYING file within
each source file that you contribute.
By contributing code to the Redis project in any form you agree to the Redis Software Grant and
Contributor License Agreement attached below. Only contributions made under the Redis Software Grant
and Contributor License Agreement may be accepted by Redis, and any contribution is subject to the
terms of the Redis dual-license under RSALv2/SSPLv1 as described in the LICENSE.txt file included in
the Redis source distribution.
# REDIS SOFTWARE GRANT AND CONTRIBUTOR LICENSE AGREEMENT
To specify the intellectual property license granted in any Contribution, Redis Ltd., ("**Redis**")
requires a Software Grant and Contributor License Agreement ("**Agreement**"). This Agreement is for
your protection as a contributor as well as the protection of Redis and its users; it does not
change your rights to use your own Contribution for any other purpose.
By making any Contribution, You accept and agree to the following terms and conditions for the
Contribution. Except for the license granted in this Agreement to Redis and the recipients of the
software distributed by Redis, You reserve all right, title, and interest in and to Your
Contribution.
1. **Definitions**
1.1. "**You**" (or "**Your**") means the copyright owner or legal entity authorized by the
copyright owner that is entering into this Agreement with Redis. For legal entities, the entity
making a Contribution and all other entities that Control, are Controlled by, or are under
common Control with that entity are considered to be a single contributor. For the purposes of
this definition, "**Control**" means (i) the power, direct or indirect, to cause the direction
or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty
percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
1.2. "**Contribution**" means the code, documentation, or any original work of authorship,
including any modifications or additions to an existing work described above.
2. "**Work**" means any software project stewarded by Redis.
3. **Grant of Copyright License**. Subject to the terms and conditions of this Agreement, You grant
to Redis and to the recipients of the software distributed by Redis a perpetual, worldwide,
non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare
derivative works of, publicly display, publicly perform, sublicense, and distribute Your
Contribution and such derivative works.
4. **Grant of Patent License**. Subject to the terms and conditions of this Agreement, You grant to
Redis and to the recipients of the software distributed by Redis a perpetual, worldwide,
non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent
license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable by You that are necessarily
infringed by Your Contribution alone or by a combination of Your Contribution with the Work to
which such Contribution was submitted. If any entity institutes patent litigation against You or
any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your
Contribution, or the Work to which you have contributed, constitutes a direct or contributory
patent infringement, then any patent licenses granted to the claimant entity under this Agreement
for that Contribution or Work terminate as of the date such litigation is filed.
5. **Representations and Warranties**. You represent and warrant that: (i) You are legally entitled
to grant the above licenses; and (ii) if You are an entity, each employee or agent designated by
You is authorized to submit the Contribution on behalf of You; and (iii) your Contribution is
Your original work, and that it will not infringe on any third party's intellectual property
right(s).
6. **Disclaimer**. You are not expected to provide support for Your Contribution, except to the
extent You desire to provide support. You may provide support for free, for a fee, or not at all.
Unless required by applicable law or agreed to in writing, You provide Your Contribution on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied,
including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT,
MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
7. **Enforceability**. Nothing in this Agreement will be construed as creating any joint venture,
employment relationship, or partnership between You and Redis. If any provision of this Agreement
is held to be unenforceable, the remaining provisions of this Agreement will not be affected.
This represents the entire agreement between You and Redis relating to the Contribution.
# IMPORTANT: HOW TO USE REDIS GITHUB ISSUES
Github issues SHOULD ONLY BE USED to report bugs, and for DETAILED feature
requests. Everything else belongs to the Redis Google Group:
GitHub issues SHOULD ONLY BE USED to report bugs and for DETAILED feature
requests. Everything else should be asked on Discord:
https://groups.google.com/forum/m/#!forum/Redis-db
https://discord.com/invite/redis
PLEASE DO NOT POST GENERAL QUESTIONS that are not about bugs or suspected
bugs in the Github issues system. We'll be very happy to help you and provide
all the support in the mailing list.
bugs in the GitHub issues system. We'll be delighted to help you and provide
all the support on Discord.
There is also an active community of Redis users at Stack Overflow:
@@ -33,24 +95,24 @@ straight away: if your feature is not a conceptual fit you'll lose a lot of
time writing the code without any reason. Start by posting in the mailing list
and creating an issue at Github with the description of, exactly, what you want
to accomplish and why. Use cases are important for features to be accepted.
Here you'll see if there is consensus about your idea.
Here you can see if there is consensus about your idea.
2. If in step 1 you get an acknowledgment from the project leaders, use the
following procedure to submit a patch:
a. Fork Redis on github ( https://docs.github.com/en/github/getting-started-with-github/fork-a-repo )
a. Fork Redis on GitHub ( https://docs.github.com/en/github/getting-started-with-github/fork-a-repo )
b. Create a topic branch (git checkout -b my_branch)
c. Push to your branch (git push origin my_branch)
d. Initiate a pull request on github ( https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request )
d. Initiate a pull request on GitHub ( https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request )
e. Done :)
3. Keep in mind that we are very overloaded, so issues and PRs sometimes wait
for a *very* long time. However this is not lack of interest, as the project
for a *very* long time. However this is not a lack of interest, as the project
gets more and more users, we find ourselves in a constant need to prioritize
certain issues/PRs over others. If you think your issue/PR is very important
try to popularize it, have other users commenting and sharing their point of
view and so forth. This helps.
view, and so forth. This helps.
4. For minor fixes just open a pull request on Github.
4. For minor fixes - open a pull request on GitHub.
Thanks!
Additional information on the RSALv2/SSPLv1 dual-license is also found in the LICENSE.txt file.
-10
View File
@@ -1,10 +0,0 @@
Copyright (c) 2006-2020, Salvatore Sanfilippo
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
* Neither the name of Redis nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+733
View File
@@ -0,0 +1,733 @@
Starting on March 20th, 2024, Redis follows a dual-licensing model with all Redis project code
contributions under version 7.4 and subsequent releases governed by the Redis Software Grant and
Contributor License Agreement. After this date, contributions are subject to the user's choice of
the Redis Source Available License v2 (RSALv2) or the Server Side Public License v1 (SSPLv1), as
follows:
1. Redis Source Available License 2.0 (RSALv2) Agreement
========================================================
Last Update: December 30, 2023
Acceptance
----------
This Agreement sets forth the terms and conditions on which the Licensor
makes available the Software. By installing, downloading, accessing,
Using, or distributing any of the Software, You agree to all of the
terms and conditions of this Agreement.
If You are receiving the Software on behalf of Your Company, You
represent and warrant that You have the authority to agree to this
Agreement on behalf of such entity.
The Licensor reserves the right to update this Agreement from time to
time.
The terms below have the meanings set forth below for purposes of this
Agreement:
Definitions
-----------
Agreement: this Redis Source Available License 2.0 Agreement.
Control: ownership, directly or indirectly, of substantially all the
assets of an entity, or the power to direct its management and policies
by vote, contract, or otherwise.
License: the License as described in the License paragraph below.
Licensor: the entity offering these terms, which includes Redis Ltd. on
behalf of itself and its subsidiaries and affiliates worldwide.
Modify, Modified, or Modification: copy from or adapt all or part of the
work in a fashion requiring copyright permission other than making an
exact copy. The resulting work is called a Modified version of the
earlier work.
Redis: the Redis software as described in redis.com redis.io.
Software: certain Software components designed to work with Redis and
provided to You under this Agreement.
Trademark: the trademarks, service marks, and any other similar rights.
Use: anything You do with the Software requiring one of Your Licenses.
You: the recipient of the Software, the individual or entity on whose
behalf You are agreeing to this Agreement.
Your Company: any legal entity, sole proprietorship, or other kind of
organization that You work for, plus all organizations that have control
over, are under the control of, or are under common control with that
organization.
Your Licenses: means all the Licenses granted to You for the Software
under this Agreement.
License
-------
The Licensor grants You a non-exclusive, royalty-free, worldwide,
non-sublicensable, non-transferable license to use, copy, distribute,
make available, and prepare derivative works of the Software, in each
case subject to the limitations and conditions below.
Limitations
-----------
You may not make the functionality of the Software or a Modified version
available to third parties as a service or distribute the Software or a
Modified version in a manner that makes the functionality of the
Software available to third parties.
Making the functionality of the Software or Modified version available
to third parties includes, without limitation, enabling third parties to
interact with the functionality of the Software or Modified version in
distributed form or remotely through a computer network, offering a
product or service, the value of which entirely or primarily derives
from the value of the Software or Modified version, or offering a
product or service that accomplishes for users the primary purpose of
the Software or Modified version.
You may not alter, remove, or obscure any licensing, copyright, or other
notices of the Licensor in the Software. Any use of the Licensor's
Trademarks is subject to applicable law.
Patents
-------
The Licensor grants You a License, under any patent claims the Licensor
can License, or becomes able to License, to make, have made, use, sell,
offer for sale, import and have imported the Software, in each case
subject to the limitations and conditions in this License. This License
does not cover any patent claims that You cause to be infringed by
Modifications or additions to the Software. If You or Your Company make
any written claim that the Software infringes or contributes to
infringement of any patent, your patent License for the Software granted
under this Agreement ends immediately. If Your Company makes such a
claim, your patent License ends immediately for work on behalf of Your
Company.
Notices
-------
You must ensure that anyone who gets a copy of any part of the Software
from You also gets a copy of the terms and conditions in this Agreement.
If You modify the Software, You must include in any Modified copies of
the Software prominent notices stating that You have Modified the
Software.
No Other Rights
---------------
The terms and conditions of this Agreement do not imply any Licenses
other than those expressly granted in this Agreement.
Termination
-----------
If You Use the Software in violation of this Agreement, such Use is not
Licensed, and Your Licenses will automatically terminate. If the
Licensor provides You with a notice of your violation, and You cease all
violations of this License no later than 30 days after You receive that
notice, Your Licenses will be reinstated retroactively. However, if You
violate this Agreement after such reinstatement, any additional
violation of this Agreement will cause your Licenses to terminate
automatically and permanently.
No Liability
------------
As far as the law allows, the Software comes as is, without any
warranty or condition, and the Licensor will not be liable to You for
any damages arising out of this Agreement or the Use or nature of the
Software, under any kind of legal claim.
Governing Law and Jurisdiction
------------------------------
If You are located in Asia, Pacific, Americas, or other jurisdictions
not listed below, the Agreement will be construed and enforced in all
respects in accordance with the laws of the State of California, U.S.A.,
without reference to its choice of law rules. The courts located in the
County of Santa Clara, California, have exclusive jurisdiction for all
purposes relating to this Agreement.
If You are located in Israel, the Agreement will be construed and
enforced in all respects in accordance with the laws of the State of
Israel without reference to its choice of law rules. The courts located
in the Central District of the State of Israel have exclusive
jurisdiction for all purposes relating to this Agreement.
If You are located in Europe, United Kingdom, Middle East or Africa, the
Agreement will be construed and enforced in all respects in accordance
with the laws of England and Wales without reference to its choice of
law rules. The competent courts located in London, England, have
exclusive jurisdiction for all purposes relating to this Agreement.
2. Server Side Public License (SSPL)
====================================
Server Side Public License
VERSION 1, OCTOBER 16, 2018
Copyright (c) 2018 MongoDB, Inc.
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to Server Side Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work in
a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based on
the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through a
computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices" to the
extent that it includes a convenient and prominently visible feature that
(1) displays an appropriate copyright notice, and (2) tells the user that
there is no warranty for the work (except to the extent that warranties
are provided), that licensees may convey the work under this License, and
how to view a copy of this License. If the interface presents a list of
user commands or options, such as a menu, a prominent item in the list
meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work for
making modifications to it. "Object code" means any non-source form of a
work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that is
widely used among developers working in that language. The "System
Libraries" of an executable work include anything, other than the work as
a whole, that (a) is included in the normal form of packaging a Major
Component, but which is not part of that Major Component, and (b) serves
only to enable use of the work with that Major Component, or to implement
a Standard Interface for which an implementation is available to the
public in source code form. A "Major Component", in this context, means a
major essential component (kernel, window system, and so on) of the
specific operating system (if any) on which the executable work runs, or
a compiler used to produce the work, or an object code interpreter used
to run it.
The "Corresponding Source" for a work in object code form means all the
source code needed to generate, install, and (for an executable work) run
the object code and to modify the work, including scripts to control
those activities. However, it does not include the work's System
Libraries, or general-purpose tools or generally available free programs
which are used unmodified in performing those activities but which are
not part of the work. For example, Corresponding Source includes
interface definition files associated with source files for the work, and
the source code for shared libraries and dynamically linked subprograms
that the work is specifically designed to require, such as by intimate
data communication or control flow between those subprograms and other
parts of the work.
The Corresponding Source need not include anything that users can
regenerate automatically from other parts of the Corresponding Source.
The Corresponding Source for a work in source code form is that same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program, subject to section 13. The
output from running a covered work is covered by this License only if the
output, given its content, constitutes a covered work. This License
acknowledges your rights of fair use or other equivalent, as provided by
copyright law. Subject to section 13, you may make, run and propagate
covered works that you do not convey, without conditions so long as your
license otherwise remains in force. You may convey covered works to
others for the sole purpose of having them make modifications exclusively
for you, or provide you with facilities for running those works, provided
that you comply with the terms of this License in conveying all
material for which you do not control copyright. Those thus making or
running the covered works for you must do so exclusively on your
behalf, under your direction and control, on terms that prohibit them
from making any copies of your copyrighted material outside their
relationship with you.
Conveying under any other circumstances is permitted solely under the
conditions stated below. Sublicensing is not allowed; section 10 makes it
unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article 11
of the WIPO copyright treaty adopted on 20 December 1996, or similar laws
prohibiting or restricting circumvention of such measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention is
effected by exercising rights under this License with respect to the
covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's users,
your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice; keep
intact all notices stating that this License and any non-permissive terms
added in accord with section 7 apply to the code; keep intact all notices
of the absence of any warranty; and give all recipients a copy of this
License along with the Program. You may charge any price or no price for
each copy that you convey, and you may offer support or warranty
protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the terms
of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified it,
and giving a relevant date.
b) The work must carry prominent notices stating that it is released
under this License and any conditions added under section 7. This
requirement modifies the requirement in section 4 to "keep intact all
notices".
c) You must license the entire work, as a whole, under this License to
anyone who comes into possession of a copy. This License will therefore
apply, along with any applicable section 7 additional terms, to the
whole of the work, and all its parts, regardless of how they are
packaged. This License gives no permission to license the work in any
other way, but it does not invalidate such permission if you have
separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your work
need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work, and
which are not combined with it such as to form a larger program, in or on
a volume of a storage or distribution medium, is called an "aggregate" if
the compilation and its resulting copyright are not used to limit the
access or legal rights of the compilation's users beyond what the
individual works permit. Inclusion of a covered work in an aggregate does
not cause this License to apply to the other parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms of
sections 4 and 5, provided that you also convey the machine-readable
Corresponding Source under the terms of this License, in one of these
ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium customarily
used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a written
offer, valid for at least three years and valid for as long as you
offer spare parts or customer support for that product model, to give
anyone who possesses the object code either (1) a copy of the
Corresponding Source for all the software in the product that is
covered by this License, on a durable physical medium customarily used
for software interchange, for a price no more than your reasonable cost
of physically performing this conveying of source, or (2) access to
copy the Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This alternative is
allowed only occasionally and noncommercially, and only if you received
the object code with such an offer, in accord with subsection 6b.
d) Convey the object code by offering access from a designated place
(gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to copy
the object code is a network server, the Corresponding Source may be on
a different server (operated by you or a third party) that supports
equivalent copying facilities, provided you maintain clear directions
next to the object code saying where to find the Corresponding Source.
Regardless of what server hosts the Corresponding Source, you remain
obligated to ensure that it is available for as long as needed to
satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided you
inform other peers where the object code and Corresponding Source of
the work are being offered to the general public at no charge under
subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be included
in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as part
of a transaction in which the right of possession and use of the User
Product is transferred to the recipient in perpetuity or for a fixed term
(regardless of how the transaction is characterized), the Corresponding
Source conveyed under this section must be accompanied by the
Installation Information. But this requirement does not apply if neither
you nor any third party retains the ability to install modified object
code on the User Product (for example, the work has been installed in
ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access
to a network may be denied when the modification itself materially
and adversely affects the operation of the network or violates the
rules and protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided, in
accord with this section must be in a format that is publicly documented
(and with an implementation available to the public in source code form),
and must require no special password or key for unpacking, reading or
copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall be
treated as though they were included in this License, to the extent that
they are valid under applicable law. If additional permissions apply only
to part of the Program, that part may be used separately under those
permissions, but the entire Program remains governed by this License
without regard to the additional permissions. When you convey a copy of
a covered work, you may at your option remove any additional permissions
from that copy, or from any part of it. (Additional permissions may be
written to require their own removal in certain cases when you modify the
work.) You may place additional permissions on material, added by you to
a covered work, for which you have or can give appropriate copyright
permission.
Notwithstanding any other provision of this License, for material you add
to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some trade
names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that material
by anyone who conveys the material (or modified versions of it) with
contractual assumptions of liability to the recipient, for any
liability that these contractual assumptions directly impose on those
licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further restriction,
you may remove that term. If a license document contains a further
restriction but permits relicensing or conveying under this License, you
may add to a covered work material governed by the terms of that license
document, provided that the further restriction does not survive such
relicensing or conveying.
If you add terms to a covered work in accord with this section, you must
place, in the relevant source files, a statement of the additional terms
that apply to those files, or a notice indicating where to find the
applicable terms. Additional terms, permissive or non-permissive, may be
stated in the form of a separately written license, or stated as
exceptions; the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or modify
it is void, and will automatically terminate your rights under this
License (including any patent licenses granted under the third paragraph
of section 11).
However, if you cease all violation of this License, then your license
from a particular copyright holder is reinstated (a) provisionally,
unless and until the copyright holder explicitly and finally terminates
your license, and (b) permanently, if the copyright holder fails to
notify you of the violation by some reasonable means prior to 60 days
after the cessation.
Moreover, your license from a particular copyright holder is reinstated
permanently if the copyright holder notifies you of the violation by some
reasonable means, this is the first time you have received notice of
violation of this License (for any work) from that copyright holder, and
you cure the violation prior to 30 days after your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or run a
copy of the Program. Ancillary propagation of a covered work occurring
solely as a consequence of using peer-to-peer transmission to receive a
copy likewise does not require acceptance. However, nothing other than
this License grants you permission to propagate or modify any covered
work. These actions infringe copyright if you do not accept this License.
Therefore, by modifying or propagating a covered work, you indicate your
acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically receives
a license from the original licensors, to run, modify and propagate that
work, subject to this License. You are not responsible for enforcing
compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered work
results from an entity transaction, each party to that transaction who
receives a copy of the work also receives whatever licenses to the work
the party's predecessor in interest had or could give under the previous
paragraph, plus a right to possession of the Corresponding Source of the
work from the predecessor in interest, if the predecessor has it or can
get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the rights
granted or affirmed under this License. For example, you may not impose a
license fee, royalty, or other charge for exercise of rights granted
under this License, and you may not initiate litigation (including a
cross-claim or counterclaim in a lawsuit) alleging that any patent claim
is infringed by making, using, selling, offering for sale, or importing
the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The work
thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims owned or
controlled by the contributor, whether already acquired or hereafter
acquired, that would be infringed by some manner, permitted by this
License, of making, using, or selling its contributor version, but do not
include claims that would be infringed only as a consequence of further
modification of the contributor version. For purposes of this definition,
"control" includes the right to grant patent sublicenses in a manner
consistent with the requirements of this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to make,
use, sell, offer for sale, import and otherwise run, modify and propagate
the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a party
means to make such an agreement or commitment not to enforce a patent
against the party.
If you convey a covered work, knowingly relying on a patent license, and
the Corresponding Source of the work is not available for anyone to copy,
free of charge and under the terms of this License, through a publicly
available network server or other readily accessible means, then you must
either (1) cause the Corresponding Source to be so available, or (2)
arrange to deprive yourself of the benefit of the patent license for this
particular work, or (3) arrange, in a manner consistent with the
requirements of this License, to extend the patent license to downstream
recipients. "Knowingly relying" means you have actual knowledge that, but
for the patent license, your conveying the covered work in a country, or
your recipient's use of the covered work in a country, would infringe
one or more identifiable patents in that country that you have reason
to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties receiving
the covered work authorizing them to use, propagate, modify or convey a
specific copy of the covered work, then the patent license you grant is
automatically extended to all recipients of the covered work and works
based on it.
A patent license is "discriminatory" if it does not include within the
scope of its coverage, prohibits the exercise of, or is conditioned on
the non-exercise of one or more of the rights that are specifically
granted under this License. You may not convey a covered work if you are
a party to an arrangement with a third party that is in the business of
distributing software, under which you make payment to the third party
based on the extent of your activity of conveying the work, and under
which the third party grants, to any of the parties who would receive the
covered work from you, a discriminatory patent license (a) in connection
with copies of the covered work conveyed by you (or copies made from
those copies), or (b) primarily for and in connection with specific
products or compilations that contain the covered work, unless you
entered into that arrangement, or that patent license was granted, prior
to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting any
implied license or other defenses to infringement that may otherwise be
available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot use,
propagate or convey a covered work so as to satisfy simultaneously your
obligations under this License and any other pertinent obligations, then
as a consequence you may not use, propagate or convey it at all. For
example, if you agree to terms that obligate you to collect a royalty for
further conveying from those to whom you convey the Program, the only way
you could satisfy both those terms and this License would be to refrain
entirely from conveying the Program.
13. Offering the Program as a Service.
If you make the functionality of the Program or a modified version
available to third parties as a service, you must make the Service Source
Code available via network download to everyone at no charge, under the
terms of this License. Making the functionality of the Program or
modified version available to third parties as a service includes,
without limitation, enabling third parties to interact with the
functionality of the Program or modified version remotely through a
computer network, offering a service the value of which entirely or
primarily derives from the value of the Program or modified version, or
offering a service that accomplishes for users the primary purpose of the
Program or modified version.
"Service Source Code" means the Corresponding Source for the Program or
the modified version, and the Corresponding Source for all programs that
you use to make the Program or modified version available as a service,
including, without limitation, management software, user interfaces,
application program interfaces, automation software, monitoring software,
backup software, storage software and hosting software, all such that a
user could run an instance of the service using the Service Source Code
you make available.
14. Revised Versions of this License.
MongoDB, Inc. may publish revised and/or new versions of the Server Side
Public License from time to time. Such new versions will be similar in
spirit to the present version, but may differ in detail to address new
problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies that a certain numbered version of the Server Side Public
License "or any later version" applies to it, you have the option of
following the terms and conditions either of that numbered version or of
any later version published by MongoDB, Inc. If the Program does not
specify a version number of the Server Side Public License, you may
choose any version ever published by MongoDB, Inc.
If the Program specifies that a proxy can decide which future versions of
the Server Side Public License can be used, that proxy's public statement
of acceptance of a version permanently authorizes you to choose that
version for the Program.
Later license versions may give you additional or different permissions.
However, no additional obligations are imposed on any author or copyright
holder as a result of your choosing to follow a later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING
ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF
THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO
LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU
OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided above
cannot be given local legal effect according to their terms, reviewing
courts shall apply local law that most closely approximates an absolute
waiver of all civil liability in connection with the Program, unless a
warranty or assumption of liability accompanies a copy of the Program in
return for a fee.
END OF TERMS AND CONDITIONS
+8 -3
View File
@@ -1,11 +1,16 @@
# Top level makefile, the real shit is at src/Makefile
# Top level makefile, the real stuff is at ./src/Makefile and in ./modules/Makefile
SUBDIRS = src
ifeq ($(BUILD_WITH_MODULES), yes)
SUBDIRS += modules
endif
default: all
.DEFAULT:
cd src && $(MAKE) $@
for dir in $(SUBDIRS); do $(MAKE) -C $$dir $@; done
install:
cd src && $(MAKE) $@
for dir in $(SUBDIRS); do $(MAKE) -C $$dir $@; done
.PHONY: install
+53 -31
View File
@@ -1,7 +1,9 @@
[![codecov](https://codecov.io/github/redis/redis/graph/badge.svg?token=6bVHb5fRuz)](https://codecov.io/github/redis/redis)
This README is just a fast *quick start* document. You can find more detailed documentation at [redis.io](https://redis.io).
What is Redis?
--------------
---
Redis is often referred to as a *data structures* server. What this means is that Redis provides access to mutable data structures via a set of commands, which are sent using a *server-client* model with TCP sockets and a simple protocol. So different processes can query and modify the same data structures in a shared way.
@@ -15,13 +17,23 @@ Another good example is to think of Redis as a more complex version of memcached
If you want to know more, this is a list of selected starting points:
* Introduction to Redis data types. https://redis.io/topics/data-types-intro
* Try Redis directly inside your browser. https://try.redis.io
* Introduction to Redis data types. https://redis.io/docs/latest/develop/data-types/
* The full list of Redis commands. https://redis.io/commands
* There is much more inside the official Redis documentation. https://redis.io/documentation
What is Redis Community Edition?
---
Redis OSS was renamed Redis Community Edition (CE) with the v7.4 release.
Redis Ltd. also offers [Redis Software](https://redis.io/enterprise/), a self-managed software with additional compliance, reliability, and resiliency for enterprise scaling,
and [Redis Cloud](https://redis.io/cloud/), a fully managed service integrated with Google Cloud, Azure, and AWS for production-ready apps.
Read more about the differences between Redis Community Edition and Redis [here](https://redis.io/comparisons/oss-vs-enterprise/).
Building Redis
--------------
---
Redis can be compiled and used on Linux, OSX, OpenBSD, NetBSD, FreeBSD.
We support big endian and little endian architectures, and both 32 bit
@@ -65,7 +77,7 @@ installed):
Fixing build problems with dependencies or cached build options
---------
---
Redis has some dependencies which are included in the `deps` directory.
`make` does not automatically rebuild dependencies even if something in
@@ -85,7 +97,7 @@ those options are cached indefinitely until you issue a `make distclean`
command.
Fixing problems building 32 bit binaries
---------
---
If after building Redis with a 32 bit target you need to rebuild it
with a 64 bit target, or the other way around, you need to perform a
@@ -99,7 +111,7 @@ the following steps:
`make CFLAGS="-m32 -march=native" LDFLAGS="-m32"`
Allocator
---------
---
Selecting a non-default memory allocator when building Redis is done by setting
the `MALLOC` environment variable. Redis is compiled and linked against libc
@@ -116,7 +128,7 @@ To compile against jemalloc on Mac OS X systems, use:
% make MALLOC=jemalloc
Monotonic clock
---------------
---
By default, Redis will build using the POSIX clock_gettime function as the
monotonic clock source. On most modern systems, the internal processor clock
@@ -128,7 +140,7 @@ To build with support for the processor's internal instruction clock, use:
% make CFLAGS="-DUSE_PROCESSOR_CLOCK"
Verbose build
-------------
---
Redis will build with a user-friendly colorized output by default.
If you want to see a more verbose output, use the following:
@@ -136,7 +148,7 @@ If you want to see a more verbose output, use the following:
% make V=1
Running Redis
-------------
---
To run Redis with the default configuration, just type:
@@ -158,14 +170,14 @@ as options using the command line. Examples:
All the options in redis.conf are also supported as options using the command
line, with exactly the same name.
Running Redis with TLS:
------------------
Running Redis with TLS
---
Please consult the [TLS.md](TLS.md) file for more information on
how to use Redis with TLS.
Playing with Redis
------------------
---
You can use redis-cli to play with Redis. Start a redis-server instance,
then in another terminal try the following:
@@ -187,7 +199,7 @@ then in another terminal try the following:
You can find the list of all the available commands at https://redis.io/commands.
Installing Redis
-----------------
---
In order to install Redis binaries into /usr/local/bin, just use:
@@ -215,25 +227,35 @@ You'll be able to stop and start Redis using the script named
`/etc/init.d/redis_<portnumber>`, for instance `/etc/init.d/redis_6379`.
Code contributions
-----------------
---
Note: By contributing code to the Redis project in any form, including sending
a pull request via Github, a code fragment or patch via private email or
public discussion groups, you agree to release your code under the terms
of the BSD license that you can find in the [COPYING][1] file included in the Redis
source distribution.
By contributing code to the Redis project in any form, including sending a pull request via GitHub,
a code fragment or patch via private email or public discussion groups, you agree to release your
code under the terms of the [Redis Software Grant and Contributor License Agreement][1]. Redis software
contains contributions to the original Redis core project, which are owned by their contributors and
licensed under the 3BSD license. Any copy of that license in this repository applies only to those
contributions. Redis releases all Redis Community Edition versions from 7.4.x and thereafter under the
RSALv2/SSPL dual-license as described in the [LICENSE.txt][2] file included in the Redis Community Edition source distribution.
Please see the [CONTRIBUTING.md][2] file in this source distribution for more
information. For security bugs and vulnerabilities, please see [SECURITY.md][3].
Please see the [CONTRIBUTING.md][1] file in this source distribution for more information. For
security bugs and vulnerabilities, please see [SECURITY.md][3].
[1]: https://github.com/redis/redis/blob/unstable/COPYING
[2]: https://github.com/redis/redis/blob/unstable/CONTRIBUTING.md
[1]: https://github.com/redis/redis/blob/unstable/CONTRIBUTING.md
[2]: https://github.com/redis/redis/blob/unstable/LICENSE.txt
[3]: https://github.com/redis/redis/blob/unstable/SECURITY.md
Redis Trademarks
---
The purpose of a trademark is to identify the goods and services of a person or company without
causing confusion. As the registered owner of its name and logo, Redis accepts certain limited uses
of its trademarks but it has requirements that must be followed as described in its Trademark
Guidelines available at: https://redis.com/legal/trademark-guidelines/.
Redis internals
===
If you are reading this README you are likely in front of a Github page
If you are reading this README you are likely in front of a GitHub page
or you just untarred the Redis distribution tar ball. In both the cases
you are basically one step away from the source code, so here we explain
the Redis source code layout, what is in each file as a general idea, the
@@ -249,7 +271,7 @@ Source code layout
The Redis root directory just contains this README, the Makefile which
calls the real Makefile inside the `src` directory and an example
configuration for Redis and Sentinel. You can find a few shell
configuration for Redis and Redis Sentinel. You can find a few shell
scripts that are used in order to execute the Redis, Redis Cluster and
Redis Sentinel unit tests, which are implemented inside the `tests`
directory.
@@ -420,7 +442,7 @@ implementations are the following:
* `lookupKeyRead()` and `lookupKeyWrite()` are used in order to get a pointer to the value associated to a given key, or `NULL` if the key does not exist.
* `dbAdd()` and its higher level counterpart `setKey()` create a new key in a Redis database.
* `dbDelete()` removes a key and its associated value.
* `emptyDb()` removes an entire single database or all the databases defined.
* `emptyData()` removes an entire single database or all the databases defined.
The rest of the file implements the generic commands exposed to the client.
@@ -458,9 +480,9 @@ Script
The script unit is composed of 3 units:
* `script.c` - integration of scripts with Redis (commands execution, set replication/resp, ...)
* `script_lua.c` - responsible to execute Lua code, uses script.c to interact with Redis from within the Lua code.
* `function_lua.c` - contains the Lua engine implementation, uses script_lua.c to execute the Lua code.
* `functions.c` - contains Redis Functions implementation (FUNCTION command), uses functions_lua.c if the function it wants to invoke needs the Lua engine.
* `script_lua.c` - responsible to execute Lua code, uses `script.c` to interact with Redis from within the Lua code.
* `function_lua.c` - contains the Lua engine implementation, uses `script_lua.c` to execute the Lua code.
* `functions.c` - contains Redis Functions implementation (`FUNCTION` command), uses `functions_lua.c` if the function it wants to invoke needs the Lua engine.
* `eval.c` - contains the `eval` implementation using `script_lua.c` to invoke the Lua code.
@@ -474,7 +496,7 @@ Other C files
* `dict.c` is an implementation of a non-blocking hash table which rehashes incrementally.
* `cluster.c` implements the Redis Cluster. Probably a good read only after being very familiar with the rest of the Redis code base. If you want to read `cluster.c` make sure to read the [Redis Cluster specification][4].
[4]: https://redis.io/topics/cluster-spec
[4]: https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/
Anatomy of a Redis command
---
+30
View File
@@ -0,0 +1,30 @@
Copyright (c) 2006-Present, Redis Ltd. and Contributors
All rights reserved.
Note: Continued Applicability of the BSD-3-Clause License
Despite the shift to the dual-licensing model with Redis Community Edition version 7.4 (RSALv2 or SSPLv1), portions of
Redis Community Edition remain available subject to the BSD-3-Clause License (BSD). See below for the full BSD
license:
Redistribution and use in source and binary forms, with or without modification, are permitted
provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions
and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions
and the following disclaimer in the documentation and/or other materials provided with the
distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse
or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+7 -11
View File
@@ -2,7 +2,7 @@
## Supported Versions
Redis is generally backwards compatible with very few exceptions, so we
Redis is generally backward compatible with very few exceptions, so we
recommend users to always use the latest version to experience stability,
performance and security.
@@ -11,17 +11,18 @@ unless this is not possible or feasible with a reasonable effort.
| Version | Supported |
| ------- | ------------------ |
| 7.0.x | :white_check_mark: |
| 7.4.x | :white_check_mark: |
| 7.2.x | :white_check_mark: |
| < 7.2.x | :x: |
| 6.2.x | :white_check_mark: |
| 6.0.x | :white_check_mark: |
| < 6.0 | :x: |
| < 6.2.x | :x: |
## Reporting a Vulnerability
If you believe youve discovered a serious vulnerability, please contact the
If you believe you've discovered a serious vulnerability, please contact the
Redis core team at redis@redis.io. We will evaluate your report and if
necessary issue a fix and an advisory. If the issue was previously undisclosed,
well also mention your name in the credits.
we'll also mention your name in the credits.
## Responsible Disclosure
@@ -34,10 +35,5 @@ This process involves providing an early notification about the vulnerability,
its impact and mitigations to a short list of vendors under a time-limited
embargo on public disclosure.
Vendors on the list are individuals or organizations that maintain Redis
distributions or provide Redis as a service, who have third party users who
will benefit from the vendors ability to prepare for a new version or deploy a
fix early.
If you believe you should be on the list, please contact us and we will
consider your request based on the above criteria.
+19
View File
@@ -0,0 +1,19 @@
coverage:
status:
patch:
default:
informational: true
project:
default:
informational: true
comment:
require_changes: false
require_head: false
require_base: false
layout: "condensed_header, diff, files"
hide_project_coverage: false
behavior: default
github_checks:
annotations: false
+14 -1
View File
@@ -3,6 +3,7 @@
uname_S:= $(shell sh -c 'uname -s 2>/dev/null || echo not')
LUA_DEBUG?=no
LUA_COVERAGE?=no
CCCOLOR="\033[34m"
LINKCOLOR="\033[34;1m"
@@ -41,6 +42,7 @@ distclean:
-(cd jemalloc && [ -f Makefile ] && $(MAKE) distclean) > /dev/null || true
-(cd hdr_histogram && $(MAKE) clean) > /dev/null || true
-(cd fpconv && $(MAKE) clean) > /dev/null || true
-(cd fast_float && $(MAKE) clean) > /dev/null || true
-(rm -f .make-*)
.PHONY: distclean
@@ -73,6 +75,12 @@ fpconv: .make-prerequisites
.PHONY: fpconv
fast_float: .make-prerequisites
@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
cd fast_float && $(MAKE) libfast_float
.PHONY: fast_float
ifeq ($(uname_S),SunOS)
# Make isinf() available
LUA_CFLAGS= -D__C99FEATURES__=1
@@ -85,6 +93,11 @@ ifeq ($(LUA_DEBUG),yes)
else
LUA_CFLAGS+= -O2
endif
ifeq ($(LUA_COVERAGE),yes)
LUA_CFLAGS += -fprofile-arcs -ftest-coverage
LUA_LDFLAGS += -fprofile-arcs -ftest-coverage
endif
# lua's Makefile defines AR="ar rcu", which is unusual, and makes it more
# challenging to cross-compile lua (and redis). These defines make it easier
# to fit redis into cross-compilation environments, which typically set AR.
@@ -106,7 +119,7 @@ endif
jemalloc: .make-prerequisites
@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
cd jemalloc && ./configure --disable-cxx --with-version=5.3.0-0-g0 --with-lg-quantum=3 --with-jemalloc-prefix=je_ CFLAGS="$(JEMALLOC_CFLAGS)" LDFLAGS="$(JEMALLOC_LDFLAGS)" $(JEMALLOC_CONFIGURE_OPTS)
cd jemalloc && ./configure --disable-cxx --with-version=5.3.0-0-g0 --with-lg-quantum=3 --disable-cache-oblivious --with-jemalloc-prefix=je_ CFLAGS="$(JEMALLOC_CFLAGS)" LDFLAGS="$(JEMALLOC_LDFLAGS)" $(JEMALLOC_CONFIGURE_OPTS)
cd jemalloc && $(MAKE) lib/libjemalloc.a
.PHONY: jemalloc
+4 -5
View File
@@ -61,12 +61,11 @@ cd deps/jemalloc
Hiredis
---
Hiredis uses the SDS string library, that must be the same version used inside Redis itself. Hiredis is also very critical for Sentinel. Historically Redis often used forked versions of hiredis in a way or the other. In order to upgrade it is advised to take a lot of care:
Hiredis is used by Sentinel, `redis-cli` and `redis-benchmark`. Like Redis, uses the SDS string library, but not necessarily the same version. In order to avoid conflicts, this version has all SDS identifiers prefixed by `hi`.
1. Check with diff if hiredis API changed and what impact it could have in Redis.
2. Make sure that the SDS library inside Hiredis and inside Redis are compatible.
3. After the upgrade, run the Redis Sentinel test.
4. Check manually that redis-cli and redis-benchmark behave as expected, since we have no tests for CLI utilities currently.
1. `git subtree pull --prefix deps/hiredis https://github.com/redis/hiredis.git <version-tag> --squash`<br>
This should hopefully merge the local changes into the new version.
2. Conflicts will arise (due to our changes) you'll need to resolve them and commit.
Linenoise
---
+24
View File
@@ -0,0 +1,24 @@
# Fallback to gcc/g++ when $CC or $CXX is not in $PATH.
CC ?= gcc
CXX ?= g++
CFLAGS=-Wall -O3
# This avoids loosing the fastfloat specific compile flags when we override the CFLAGS via the main project
FASTFLOAT_CFLAGS=-std=c++11 -DFASTFLOAT_ALLOWS_LEADING_PLUS
LDFLAGS=
libfast_float: fast_float_strtod.o
$(AR) -r libfast_float.a fast_float_strtod.o
32bit: CFLAGS += -m32
32bit: LDFLAGS += -m32
32bit: libfast_float
fast_float_strtod.o: fast_float_strtod.cpp
$(CXX) $(CFLAGS) $(FASTFLOAT_CFLAGS) -c fast_float_strtod.cpp $(LDFLAGS)
clean:
rm -f *.o
rm -f *.a
rm -f *.h.gch
rm -rf *.dSYM
+21
View File
@@ -0,0 +1,21 @@
README for fast_float v6.1.4
----------------------------------------------
We're using the fast_float library[1] in our (compiled-in)
floating-point fast_float_strtod implementation for faster and more
portable parsing of 64 decimal strings.
The single file fast_float.h is an amalgamation of the entire library,
which can be (re)generated with the amalgamate.py script (from the
fast_float repository) via the command
```
git clone https://github.com/fastfloat/fast_float
cd fast_float
git checkout v6.1.4
python3 ./script/amalgamate.py --license=MIT \
> $REDIS_SRC/deps/fast_float/fast_float.h
```
[1]: https://github.com/fastfloat/fast_float
+3838
View File
File diff suppressed because it is too large Load Diff
+32
View File
@@ -0,0 +1,32 @@
#include "fast_float.h"
#include <iostream>
#include <string>
#include <system_error>
#include <cerrno>
/* Convert NPTR to a double using the fast_float library.
*
* This function behaves similarly to the standard strtod function, converting
* the initial portion of the string pointed to by `nptr` to a `double` value,
* using the fast_float library for high performance. If the conversion fails,
* errno is set to EINVAL error code.
*
* @param nptr A pointer to the null-terminated byte string to be interpreted.
* @param endptr A pointer to a pointer to character. If `endptr` is not NULL,
* it will point to the character after the last character used
* in the conversion.
* @return The converted value as a double. If no valid conversion could
* be performed, returns 0.0.
* If ENDPTR is not NULL, a pointer to the character after the last one used
* in the number is put in *ENDPTR. */
extern "C" double fast_float_strtod(const char *nptr, char **endptr) {
double result = 0.0;
auto answer = fast_float::from_chars(nptr, nptr + strlen(nptr), result);
if (answer.ec != std::errc()) {
errno = EINVAL; // Fallback to for other errors
}
if (endptr != NULL) {
*endptr = (char *)answer.ptr;
}
return result;
}
+15
View File
@@ -0,0 +1,15 @@
#ifndef __FAST_FLOAT_STRTOD_H__
#define __FAST_FLOAT_STRTOD_H__
#if defined(__cplusplus)
extern "C"
{
#endif
double fast_float_strtod(const char *in, char **out);
#if defined(__cplusplus)
}
#endif
#endif /* __FAST_FLOAT_STRTOD_H__ */
+49
View File
@@ -0,0 +1,49 @@
name-template: '$NEXT_MAJOR_VERSION'
tag-template: 'v$NEXT_MAJOR_VERSION'
autolabeler:
- label: 'maintenance'
files:
- '*.md'
- '.github/*'
- label: 'bug'
branch:
- '/bug-.+'
- label: 'maintenance'
branch:
- '/maintenance-.+'
- label: 'feature'
branch:
- '/feature-.+'
categories:
- title: 'Breaking Changes'
labels:
- 'breakingchange'
- title: '🧪 Experimental Features'
labels:
- 'experimental'
- title: '🚀 New Features'
labels:
- 'feature'
- 'enhancement'
- title: '🐛 Bug Fixes'
labels:
- 'fix'
- 'bugfix'
- 'bug'
- 'BUG'
- title: '🧰 Maintenance'
label: 'maintenance'
change-template: '- $TITLE @$AUTHOR (#$NUMBER)'
exclude-labels:
- 'skip-changelog'
template: |
## Changes
$CHANGES
## Contributors
We'd like to thank all the contributors who worked on this release!
$CONTRIBUTORS
+23 -51
View File
@@ -6,24 +6,21 @@ jobs:
name: Ubuntu
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- uses: actions/checkout@v3
- name: Install dependencies
run: |
sudo add-apt-repository -y ppa:chris-lea/redis-server
curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list
sudo apt-get update
sudo apt-get install -y redis-server valgrind libevent-dev
- name: Build using cmake
env:
env:
EXTRA_CMAKE_OPTS: -DENABLE_EXAMPLES:BOOL=ON -DENABLE_SSL:BOOL=ON -DENABLE_SSL_TESTS:BOOL=ON -DENABLE_ASYNC_TESTS:BOOL=ON
CFLAGS: -Werror
CXXFLAGS: -Werror
run: mkdir build-ubuntu && cd build-ubuntu && cmake ..
run: mkdir build && cd build && cmake .. && make
- name: Build using makefile
run: USE_SSL=1 TEST_ASYNC=1 make
@@ -45,11 +42,7 @@ jobs:
runs-on: ubuntu-latest
container: centos:7
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- uses: actions/checkout@v3
- name: Install dependencies
run: |
@@ -58,11 +51,11 @@ jobs:
yum -y install gcc gcc-c++ make openssl openssl-devel cmake3 valgrind libevent-devel
- name: Build using cmake
env:
env:
EXTRA_CMAKE_OPTS: -DENABLE_EXAMPLES:BOOL=ON -DENABLE_SSL:BOOL=ON -DENABLE_SSL_TESTS:BOOL=ON -DENABLE_ASYNC_TESTS:BOOL=ON
CFLAGS: -Werror
CXXFLAGS: -Werror
run: mkdir build-centos7 && cd build-centos7 && cmake3 ..
run: mkdir build && cd build && cmake3 .. && make
- name: Build using Makefile
run: USE_SSL=1 TEST_ASYNC=1 make
@@ -85,25 +78,22 @@ jobs:
runs-on: ubuntu-latest
container: rockylinux:8
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- uses: actions/checkout@v3
- name: Install dependencies
run: |
dnf -y upgrade --refresh
dnf -y install https://rpms.remirepo.net/enterprise/remi-release-8.rpm
dnf -y module install redis:remi-6.0
dnf -y group install "Development Tools"
dnf -y install openssl-devel cmake valgrind libevent-devel
- name: Build using cmake
env:
env:
EXTRA_CMAKE_OPTS: -DENABLE_EXAMPLES:BOOL=ON -DENABLE_SSL:BOOL=ON -DENABLE_SSL_TESTS:BOOL=ON -DENABLE_ASYNC_TESTS:BOOL=ON
CFLAGS: -Werror
CXXFLAGS: -Werror
run: mkdir build-centos8 && cd build-centos8 && cmake ..
run: mkdir build && cd build && cmake .. && make
- name: Build using Makefile
run: USE_SSL=1 TEST_ASYNC=1 make
@@ -122,17 +112,13 @@ jobs:
run: $GITHUB_WORKSPACE/test.sh
freebsd:
runs-on: macos-10.15
runs-on: macos-12
name: FreeBSD
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- uses: actions/checkout@v3
- name: Build in FreeBSD
uses: vmactions/freebsd-vm@v0.1.5
uses: vmactions/freebsd-vm@v0
with:
prepare: pkg install -y gmake cmake
run: |
@@ -143,15 +129,12 @@ jobs:
name: macOS
runs-on: macos-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- uses: actions/checkout@v3
- name: Install dependencies
run: |
brew install openssl redis
brew install openssl redis@7.0
brew link redis@7.0 --force
- name: Build hiredis
run: USE_SSL=1 make
@@ -165,11 +148,7 @@ jobs:
name: Windows
runs-on: windows-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
with:
repository: ${{ env.GITHUB_REPOSITORY }}
ref: ${{ env.GITHUB_HEAD_REF }}
- uses: actions/checkout@v3
- name: Install dependencies
run: |
@@ -186,20 +165,13 @@ jobs:
run: |
./build/hiredis-test.exe
- name: Setup cygwin
uses: egor-tensin/setup-cygwin@v3
- name: Install Cygwin Action
uses: cygwin/cygwin-install-action@v2
with:
platform: x64
packages: make git gcc-core
- name: Build in cygwin
env:
HIREDIS_PATH: ${{ github.workspace }}
run: |
build_hiredis() {
cd $(cygpath -u $HIREDIS_PATH)
git clean -xfd
make
}
build_hiredis
shell: C:\tools\cygwin\bin\bash.exe --login --norc -eo pipefail -o igncr '{0}'
make clean && make
+19
View File
@@ -0,0 +1,19 @@
name: Release Drafter
on:
push:
# branches to consider in the event; optional, defaults to all
branches:
- master
jobs:
update_release_draft:
runs-on: ubuntu-latest
steps:
# Drafts your next Release notes as Pull Requests are merged into "master"
- uses: release-drafter/release-drafter@v5
with:
# (Optional) specify config name to use, relative to .github/. Default: release-drafter.yml
config-name: release-drafter-config.yml
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+100
View File
@@ -0,0 +1,100 @@
name: C/C++ CI
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
full-build:
name: Build all, plus default examples, run tests against redis
runs-on: ubuntu-latest
env:
# the docker image used by the test.sh
REDIS_DOCKER: redis:alpine
steps:
- name: Install prerequisites
run: sudo apt-get update && sudo apt-get install -y libev-dev libevent-dev libglib2.0-dev libssl-dev valgrind
- uses: actions/checkout@v3
- name: Run make
run: make all examples
- name: Run unittests
run: make check
- name: Run tests with valgrind
env:
TEST_PREFIX: valgrind --error-exitcode=100
SKIPS_ARG: --skip-throughput
run: make check
build-32-bit:
name: Build and test minimal 32 bit linux
runs-on: ubuntu-latest
steps:
- name: Install prerequisites
run: sudo apt-get update && sudo apt-get install gcc-multilib
- uses: actions/checkout@v3
- name: Run make
run: make all
env:
PLATFORM_FLAGS: -m32
- name: Run unittests
env:
REDIS_DOCKER: redis:alpine
run: make check
build-arm:
name: Cross-compile and test arm linux with Qemu
runs-on: ubuntu-latest
strategy:
matrix:
include:
- name: arm
toolset: arm-linux-gnueabi
emulator: qemu-arm
- name: aarch64
toolset: aarch64-linux-gnu
emulator: qemu-aarch64
steps:
- name: Install qemu
if: matrix.emulator
run: sudo apt-get update && sudo apt-get install -y qemu-user
- name: Install platform toolset
if: matrix.toolset
run: sudo apt-get install -y gcc-${{matrix.toolset}}
- uses: actions/checkout@v3
- name: Run make
run: make all
env:
CC: ${{matrix.toolset}}-gcc
AR: ${{matrix.toolset}}-ar
- name: Run unittests
env:
REDIS_DOCKER: redis:alpine
TEST_PREFIX: ${{matrix.emulator}} -L /usr/${{matrix.toolset}}/
run: make check
build-windows:
name: Build and test on windows 64 bit Intel
runs-on: windows-latest
steps:
- uses: microsoft/setup-msbuild@v1.0.2
- uses: actions/checkout@v3
- name: Run CMake (shared lib)
run: cmake -Wno-dev CMakeLists.txt
- name: Build hiredis (shared lib)
run: MSBuild hiredis.vcxproj /p:Configuration=Debug
- name: Run CMake (static lib)
run: cmake -Wno-dev CMakeLists.txt -DBUILD_SHARED_LIBS=OFF
- name: Build hiredis (static lib)
run: MSBuild hiredis.vcxproj /p:Configuration=Debug
- name: Build hiredis-test
run: MSBuild hiredis-test.vcxproj /p:Configuration=Debug
# use memurai, redis compatible server, since it is easy to install. Can't
# install official redis containers on the windows runner
- name: Install Memurai redis server
run: choco install -y memurai-developer.install
- name: Run tests
run: Debug\hiredis-test.exe
+216
View File
@@ -1,3 +1,219 @@
## [1.2.0](https://github.com/redis/hiredis/tree/v1.2.0) - (2023-06-04)
Announcing Hiredis v1.2.0 with with new adapters, and a great many bug fixes.
## 🚀 New Features
- Add sdevent adapter @Oipo (#1144)
- Allow specifying the keepalive interval @michael-grunder (#1168)
- Add RedisModule adapter @tezc (#1182)
- Helper for setting TCP_USER_TIMEOUT socket option @zuiderkwast (#1188)
## 🐛 Bug Fixes
- Fix a typo in b6a052f. @yossigo (#1190)
- Fix wincrypt symbols conflict @hudayou (#1151)
- Don't attempt to set a timeout if we are in an error state. @michael-grunder (#1180)
- Accept -nan per the RESP3 spec recommendation. @michael-grunder (#1178)
- Fix colliding option values @zuiderkwast (#1172)
- Ensure functionality without `_MSC_VER` definition @windyakin (#1194)
## 🧰 Maintenance
- Add a test for the TCP_USER_TIMEOUT option. @michael-grunder (#1192)
- Add -Werror as a default. @yossigo (#1193)
- CI: Update homebrew Redis version. @yossigo (#1191)
- Fix typo in makefile. @michael-grunder (#1179)
- Write a version file for the CMake package @Neverlord (#1165)
- CMakeLists.txt: respect BUILD_SHARED_LIBS @ffontaine (#1147)
- Cmake static or shared @autoantwort (#1160)
- fix typo @tillkruss (#1153)
- Add a test ensuring we don't clobber connection error. @michael-grunder (#1181)
- Search for openssl on macOS @michael-grunder (#1169)
## Contributors
We'd like to thank all the contributors who worked on this release!
<a href="https://github.com/neverlord"><img src="https://github.com/neverlord.png" width="32" height="32"></a>
<a href="https://github.com/Oipo"><img src="https://github.com/Oipo.png" width="32" height="32"></a>
<a href="https://github.com/autoantwort"><img src="https://github.com/autoantwort.png" width="32" height="32"></a>
<a href="https://github.com/ffontaine"><img src="https://github.com/ffontaine.png" width="32" height="32"></a>
<a href="https://github.com/hudayou"><img src="https://github.com/hudayou.png" width="32" height="32"></a>
<a href="https://github.com/michael-grunder"><img src="https://github.com/michael-grunder.png" width="32" height="32"></a>
<a href="https://github.com/postgraph"><img src="https://github.com/postgraph.png" width="32" height="32"></a>
<a href="https://github.com/tezc"><img src="https://github.com/tezc.png" width="32" height="32"></a>
<a href="https://github.com/tillkruss"><img src="https://github.com/tillkruss.png" width="32" height="32"></a>
<a href="https://github.com/vityafx"><img src="https://github.com/vityafx.png" width="32" height="32"></a>
<a href="https://github.com/windyakin"><img src="https://github.com/windyakin.png" width="32" height="32"></a>
<a href="https://github.com/yossigo"><img src="https://github.com/yossigo.png" width="32" height="32"></a>
<a href="https://github.com/zuiderkwast"><img src="https://github.com/zuiderkwast.png" width="32" height="32"></a>
## [1.1.0](https://github.com/redis/hiredis/tree/v1.1.0) - (2022-11-15)
Announcing Hiredis v1.1.0 GA with better SSL convenience, new async adapters and a great many bug fixes.
**NOTE**: Hiredis can now return `nan` in addition to `-inf` and `inf` when returning a `REDIS_REPLY_DOUBLE`.
## 🐛 Bug Fixes
- Add support for nan in RESP3 double [@filipecosta90](https://github.com/filipecosta90)
([\#1133](https://github.com/redis/hiredis/pull/1133))
## 🧰 Maintenance
- Add an example that calls redisCommandArgv [@michael-grunder](https://github.com/michael-grunder)
([\#1140](https://github.com/redis/hiredis/pull/1140))
- fix flag reference [@pata00](https://github.com/pata00) ([\#1136](https://github.com/redis/hiredis/pull/1136))
- Make freeing a NULL redisAsyncContext a no op. [@michael-grunder](https://github.com/michael-grunder)
([\#1135](https://github.com/redis/hiredis/pull/1135))
- CI updates ([@bjosv](https://github.com/redis/bjosv) ([\#1139](https://github.com/redis/hiredis/pull/1139))
## Contributors
We'd like to thank all the contributors who worked on this release!
<a href="https://github.com/bjsov"><img src="https://github.com/bjosv.png" width="32" height="32"></a>
<a href="https://github.com/filipecosta90"><img src="https://github.com/filipecosta90.png" width="32" height="32"></a>
<a href="https://github.com/michael-grunder"><img src="https://github.com/michael-grunder.png" width="32" height="32"></a>
<a href="https://github.com/pata00"><img src="https://github.com/pata00.png" width="32" height="32"></a>
## [1.1.0-rc1](https://github.com/redis/hiredis/tree/v1.1.0-rc1) - (2022-11-06)
Announcing Hiredis v1.1.0-rc1, with better SSL convenience, new async adapters, and a great many bug fixes.
## 🚀 New Features
- Add possibility to prefer IPv6, IPv4 or unspecified [@zuiderkwast](https://github.com/zuiderkwast)
([\#1096](https://github.com/redis/hiredis/pull/1096))
- Add adapters/libhv [@ithewei](https://github.com/ithewei) ([\#904](https://github.com/redis/hiredis/pull/904))
- Add timeout support to libhv adapter. [@michael-grunder](https://github.com/michael-grunder) ([\#1109](https://github.com/redis/hiredis/pull/1109))
- set default SSL verification path [@adobeturchenko](https://github.com/adobeturchenko) ([\#928](https://github.com/redis/hiredis/pull/928))
- Introduce .close method for redisContextFuncs [@pizhenwei](https://github.com/pizhenwei) ([\#1094](https://github.com/redis/hiredis/pull/1094))
- Make it possible to set SSL verify mode [@stanhu](https://github.com/stanhu) ([\#1085](https://github.com/redis/hiredis/pull/1085))
- Polling adapter and example [@kristjanvalur](https://github.com/kristjanvalur) ([\#932](https://github.com/redis/hiredis/pull/932))
- Unsubscribe handling in async [@bjosv](https://github.com/bjosv) ([\#1047](https://github.com/redis/hiredis/pull/1047))
- Add timeout support for libuv adapter [@MichaelSuen-thePointer](https://github.com/@MichaelSuenthePointer) ([\#1016](https://github.com/redis/hiredis/pull/1016))
## 🐛 Bug Fixes
- Update for MinGW cross compile [@bit0fun](https://github.com/bit0fun) ([\#1127](https://github.com/redis/hiredis/pull/1127))
- fixed CPP build error with adapters/libhv.h [@mtdxc](https://github.com/mtdxc) ([\#1125](https://github.com/redis/hiredis/pull/1125))
- Fix protocol error
[@michael-grunder](https://github.com/michael-grunder),
[@mtuleika-appcast](https://github.com/mtuleika-appcast) ([\#1106](https://github.com/redis/hiredis/pull/1106))
- Use a windows specific keepalive function. [@michael-grunder](https://github.com/michael-grunder) ([\#1104](https://github.com/redis/hiredis/pull/1104))
- Fix CMake config path on Linux. [@xkszltl](https://github.com/xkszltl) ([\#989](https://github.com/redis/hiredis/pull/989))
- Fix potential fault at createDoubleObject [@afcidk](https://github.com/afcidk) ([\#964](https://github.com/redis/hiredis/pull/964))
- Fix some undefined behavior [@jengab](https://github.com/jengab) ([\#1091](https://github.com/redis/hiredis/pull/1091))
- Copy OOM errors to redisAsyncContext when finding subscribe callback [@bjosv](https://github.com/bjosv) ([\#1090](https://github.com/redis/hiredis/pull/1090))
- Maintain backward compatibility with our onConnect callback. [@michael-grunder](https://github.com/michael-grunder) ([\#1087](https://github.com/redis/hiredis/pull/1087))
- Fix PUSH handler tests for Redis >= 7.0.5 [@michael-grunder](https://github.com/michael-grunder) ([\#1121](https://github.com/redis/hiredis/pull/1121))
- fix heap-buffer-overflow [@zhangtaoXT5](https://github.com/zhangtaoXT5) ([\#957](https://github.com/redis/hiredis/pull/957))
- Fix heap-buffer-overflow issue in redisvFormatCommad [@bjosv](https://github.com/bjosv) ([\#1097](https://github.com/redis/hiredis/pull/1097))
- Polling adapter requires sockcompat.h [@michael-grunder](https://github.com/michael-grunder) ([\#1095](https://github.com/redis/hiredis/pull/1095))
- Illumos test fixes, error message difference for bad hostname test. [@devnexen](https://github.com/devnexen) ([\#901](https://github.com/redis/hiredis/pull/901))
- Remove semicolon after do-while in \_EL\_CLEANUP [@sundb](https://github.com/sundb) ([\#905](https://github.com/redis/hiredis/pull/905))
- Stability: Support calling redisAsyncCommand and redisAsyncDisconnect from the onConnected callback [@kristjanvalur](https://github.com/kristjanvalur)
([\#931](https://github.com/redis/hiredis/pull/931))
- Fix async connect on Windows [@kristjanvalur](https://github.com/kristjanvalur) ([\#1073](https://github.com/redis/hiredis/pull/1073))
- Fix tests so they work for Redis 7.0 [@michael-grunder](https://github.com/michael-grunder) ([\#1072](https://github.com/redis/hiredis/pull/1072))
- Fix warnings on Win64 [@orgads](https://github.com/orgads) ([\#1058](https://github.com/redis/hiredis/pull/1058))
- Handle push notifications before or after reply. [@yossigo](https://github.com/yossigo) ([\#1062](https://github.com/redis/hiredis/pull/1062))
- Update hiredis sds with improvements found in redis [@bjosv](https://github.com/bjosv) ([\#1045](https://github.com/redis/hiredis/pull/1045))
- Avoid incorrect call to the previous reply's callback [@bjosv](https://github.com/bjosv) ([\#1040](https://github.com/redis/hiredis/pull/1040))
- fix building on AIX and SunOS [\#1031](https://github.com/redis/hiredis/pull/1031) ([@scddev](https://github.com/scddev))
- Allow sending commands after sending an unsubscribe [@bjosv](https://github.com/bjosv) ([\#1036](https://github.com/redis/hiredis/pull/1036))
- Correction for command timeout during pubsub [@bjosv](https://github.com/bjosv) ([\#1038](https://github.com/redis/hiredis/pull/1038))
- Fix adapters/libevent.h compilation for 64-bit Windows [@pbtummillo](https://github.com/pbtummillo) ([\#937](https://github.com/redis/hiredis/pull/937))
- Fix integer overflow when format command larger than 4GB [@sundb](https://github.com/sundb) ([\#1030](https://github.com/redis/hiredis/pull/1030))
- Handle array response during subscribe in RESP3 [@bjosv](https://github.com/bjosv) ([\#1014](https://github.com/redis/hiredis/pull/1014))
- Support PING while subscribing (RESP2) [@bjosv](https://github.com/bjosv) ([\#1027](https://github.com/redis/hiredis/pull/1027))
## 🧰 Maintenance
- CI fixes in preparation of release [@michael-grunder](https://github.com/michael-grunder) ([\#1130](https://github.com/redis/hiredis/pull/1130))
- Add do while(0) (protection for macros [@afcidk](https://github.com/afcidk) [\#959](https://github.com/redis/hiredis/pull/959))
- Fixup of PR734: Coverage of hiredis.c [@bjosv](https://github.com/bjosv) ([\#1124](https://github.com/redis/hiredis/pull/1124))
- CMake corrections for building on Windows [@bjosv](https://github.com/bjosv) ([\#1122](https://github.com/redis/hiredis/pull/1122))
- Install on windows fixes [@bjosv](https://github.com/bjosv) ([\#1117](https://github.com/redis/hiredis/pull/1117))
- Add libhv example to our standard Makefile [@michael-grunder](https://github.com/michael-grunder) ([\#1108](https://github.com/redis/hiredis/pull/1108))
- Additional include directory given by pkg-config [@bjosv](https://github.com/bjosv) ([\#1118](https://github.com/redis/hiredis/pull/1118))
- Use __attribute__ when building with Clang on Windows [@bjosv](https://github.com/bjosv) ([\#1115](https://github.com/redis/hiredis/pull/1115))
- Minor refactor [@michael-grunder](https://github.com/michael-grunder) ([\#1110](https://github.com/redis/hiredis/pull/1110))
- Fix pkgconfig result for hiredis_ssl [@bjosv](https://github.com/bjosv) ([\#1107](https://github.com/redis/hiredis/pull/1107))
- Update documentation to explain redisConnectWithOptions. [@michael-grunder](https://github.com/michael-grunder) ([\#1099](https://github.com/redis/hiredis/pull/1099))
- uvadapter: reduce number of uv_poll_start calls [@noxiouz](https://github.com/noxiouz) ([\#1098](https://github.com/redis/hiredis/pull/1098))
- Regression test for off-by-one parsing error [@bugwz](https://github.com/bugwz) ([\#1092](https://github.com/redis/hiredis/pull/1092))
- CMake: remove dict.c form hiredis_sources [@Lipraxde](https://github.com/Lipraxde) ([\#1055](https://github.com/redis/hiredis/pull/1055))
- Do store command timeout in the context for redisSetTimeout [@catterer](https://github.com/catterer) ([\#593](https://github.com/redis/hiredis/pull/593), [\#1093](https://github.com/redis/hiredis/pull/1093))
- Add GitHub Actions CI workflow for hiredis: Arm, Arm64, 386, windows. [@kristjanvalur](https://github.com/kristjanvalur) ([\#943](https://github.com/redis/hiredis/pull/943))
- CI: bump macOS runner version [@SukkaW](https://github.com/SukkaW) ([\#1079](https://github.com/redis/hiredis/pull/1079))
- Support for generating release notes [@chayim](https://github.com/chayim) ([\#1083](https://github.com/redis/hiredis/pull/1083))
- Improve example for SSL initialization in README.md [@stanhu](https://github.com/stanhu) ([\#1084](https://github.com/redis/hiredis/pull/1084))
- Fix README typos [@bjosv](https://github.com/bjosv) ([\#1080](https://github.com/redis/hiredis/pull/1080))
- fix cmake version [@smmir-cent](https://github.com/@smmircent) ([\#1050](https://github.com/redis/hiredis/pull/1050))
- Use the same name for static and shared libraries [@orgads](https://github.com/orgads) ([\#1057](https://github.com/redis/hiredis/pull/1057))
- Embed debug information in windows static .lib file [@kristjanvalur](https://github.com/kristjanvalur) ([\#1054](https://github.com/redis/hiredis/pull/1054))
- Improved async documentation [@kristjanvalur](https://github.com/kristjanvalur) ([\#1074](https://github.com/redis/hiredis/pull/1074))
- Use official repository for redis package. [@yossigo](https://github.com/yossigo) ([\#1061](https://github.com/redis/hiredis/pull/1061))
- Whitelist hiredis repo path in cygwin [@michael-grunder](https://github.com/michael-grunder) ([\#1063](https://github.com/redis/hiredis/pull/1063))
- CentOS 8 is EOL, switch to RockyLinux [@michael-grunder](https://github.com/michael-grunder) ([\#1046](https://github.com/redis/hiredis/pull/1046))
- CMakeLists.txt: allow building without a C++ compiler [@ffontaine](https://github.com/ffontaine) ([\#872](https://github.com/redis/hiredis/pull/872))
- Makefile: move SSL options into a block and refine rules [@pizhenwei](https://github.com/pizhenwei) ([\#997](https://github.com/redis/hiredis/pull/997))
- Update CMakeLists.txt for more portability [@EricDeng1001](https://github.com/EricDeng1001) ([\#1005](https://github.com/redis/hiredis/pull/1005))
- FreeBSD build fixes + CI [@michael-grunder](https://github.com/michael-grunder) ([\#1026](https://github.com/redis/hiredis/pull/1026))
- Add asynchronous test for pubsub using RESP3 [@bjosv](https://github.com/bjosv) ([\#1012](https://github.com/redis/hiredis/pull/1012))
- Trigger CI failure when Valgrind issues are found [@bjosv](https://github.com/bjosv) ([\#1011](https://github.com/redis/hiredis/pull/1011))
- Move to using make directly in Cygwin [@michael-grunder](https://github.com/michael-grunder) ([\#1020](https://github.com/redis/hiredis/pull/1020))
- Add asynchronous API tests [@bjosv](https://github.com/bjosv) ([\#1010](https://github.com/redis/hiredis/pull/1010))
- Correcting the build target `coverage` for enabled SSL [@bjosv](https://github.com/bjosv) ([\#1009](https://github.com/redis/hiredis/pull/1009))
- GH Actions: Run SSL tests during CI [@bjosv](https://github.com/bjosv) ([\#1008](https://github.com/redis/hiredis/pull/1008))
- GH: Actions - Add valgrind and CMake [@michael-grunder](https://github.com/michael-grunder) ([\#1004](https://github.com/redis/hiredis/pull/1004))
- Add Centos8 tests in GH Actions [@michael-grunder](https://github.com/michael-grunder) ([\#1001](https://github.com/redis/hiredis/pull/1001))
- We should run actions on PRs [@michael-grunder](https://github.com/michael-grunder) (([\#1000](https://github.com/redis/hiredis/pull/1000))
- Add Cygwin test in GitHub actions [@michael-grunder](https://github.com/michael-grunder) ([\#999](https://github.com/redis/hiredis/pull/999))
- Add Windows tests in GitHub actions [@michael-grunder](https://github.com/michael-grunder) ([\#996](https://github.com/redis/hiredis/pull/996))
- Switch to GitHub actions [@michael-grunder](https://github.com/michael-grunder) ([\#995](https://github.com/redis/hiredis/pull/995))
- Minor refactor of CVE-2021-32765 fix. [@michael-grunder](https://github.com/michael-grunder) ([\#993](https://github.com/redis/hiredis/pull/993))
- Remove extra comma from CMake var. [@xkszltl](https://github.com/xkszltl) ([\#988](https://github.com/redis/hiredis/pull/988))
- Add REDIS\_OPT\_PREFER\_UNSPEC [@michael-grunder](https://github.com/michael-grunder) ([\#1101](https://github.com/redis/hiredis/pull/1101))
## Contributors
We'd like to thank all the contributors who worked on this release!
<a href="https://github.com/EricDeng1001"><img src="https://github.com/EricDeng1001.png" width="32" height="32"></a>
<a href="https://github.com/Lipraxde"><img src="https://github.com/Lipraxde.png" width="32" height="32"></a>
<a href="https://github.com/MichaelSuen-thePointer"><img src="https://github.com/MichaelSuen-thePointer.png" width="32" height="32"></a>
<a href="https://github.com/SukkaW"><img src="https://github.com/SukkaW.png" width="32" height="32"></a>
<a href="https://github.com/adobeturchenko"><img src="https://github.com/adobeturchenko.png" width="32" height="32"></a>
<a href="https://github.com/afcidk"><img src="https://github.com/afcidk.png" width="32" height="32"></a>
<a href="https://github.com/bit0fun"><img src="https://github.com/bit0fun.png" width="32" height="32"></a>
<a href="https://github.com/bjosv"><img src="https://github.com/bjosv.png" width="32" height="32"></a>
<a href="https://github.com/bugwz"><img src="https://github.com/bugwz.png" width="32" height="32"></a>
<a href="https://github.com/catterer"><img src="https://github.com/catterer.png" width="32" height="32"></a>
<a href="https://github.com/chayim"><img src="https://github.com/chayim.png" width="32" height="32"></a>
<a href="https://github.com/devnexen"><img src="https://github.com/devnexen.png" width="32" height="32"></a>
<a href="https://github.com/ffontaine"><img src="https://github.com/ffontaine.png" width="32" height="32"></a>
<a href="https://github.com/ithewei"><img src="https://github.com/ithewei.png" width="32" height="32"></a>
<a href="https://github.com/jengab"><img src="https://github.com/jengab.png" width="32" height="32"></a>
<a href="https://github.com/kristjanvalur"><img src="https://github.com/kristjanvalur.png" width="32" height="32"></a>
<a href="https://github.com/michael-grunder"><img src="https://github.com/michael-grunder.png" width="32" height="32"></a>
<a href="https://github.com/noxiouz"><img src="https://github.com/noxiouz.png" width="32" height="32"></a>
<a href="https://github.com/mtdxc"><img src="https://github.com/mtdxc.png" width="32" height="32"></a>
<a href="https://github.com/orgads"><img src="https://github.com/orgads.png" width="32" height="32"></a>
<a href="https://github.com/pbtummillo"><img src="https://github.com/pbtummillo.png" width="32" height="32"></a>
<a href="https://github.com/pizhenwei"><img src="https://github.com/pizhenwei.png" width="32" height="32"></a>
<a href="https://github.com/scddev"><img src="https://github.com/scddev.png" width="32" height="32"></a>
<a href="https://github.com/smmir-cent"><img src="https://github.com/smmir-cent.png" width="32" height="32"></a>
<a href="https://github.com/stanhu"><img src="https://github.com/stanhu.png" width="32" height="32"></a>
<a href="https://github.com/sundb"><img src="https://github.com/sundb.png" width="32" height="32"></a>
<a href="https://github.com/vturchenko"><img src="https://github.com/vturchenko.png" width="32" height="32"></a>
<a href="https://github.com/xkszltl"><img src="https://github.com/xkszltl.png" width="32" height="32"></a>
<a href="https://github.com/yossigo"><img src="https://github.com/yossigo.png" width="32" height="32"></a>
<a href="https://github.com/zhangtaoXT5"><img src="https://github.com/zhangtaoXT5.png" width="32" height="32"></a>
<a href="https://github.com/zuiderkwast"><img src="https://github.com/zuiderkwast.png" width="32" height="32"></a>
## [1.0.2](https://github.com/redis/hiredis/tree/v1.0.2) - (2021-10-07)
Announcing Hiredis v1.0.2, which fixes CVE-2021-32765 but returns the SONAME to the correct value of `1.0.0`.
+40 -48
View File
@@ -1,8 +1,10 @@
CMAKE_MINIMUM_REQUIRED(VERSION 3.4.0)
CMAKE_MINIMUM_REQUIRED(VERSION 3.0.0)
OPTION(BUILD_SHARED_LIBS "Build shared libraries" ON)
OPTION(ENABLE_SSL "Build hiredis_ssl for SSL support" OFF)
OPTION(DISABLE_TESTS "If tests should be compiled or not" OFF)
OPTION(ENABLE_SSL_TESTS "Should we test SSL connections" OFF)
OPTION(ENABLE_EXAMPLES "Enable building hiredis examples" OFF)
OPTION(ENABLE_ASYNC_TESTS "Should we run all asynchronous API tests" OFF)
MACRO(getVersionBit name)
@@ -24,15 +26,11 @@ INCLUDE(GNUInstallDirs)
# Hiredis requires C99
SET(CMAKE_C_STANDARD 99)
SET(CMAKE_POSITION_INDEPENDENT_CODE ON)
SET(CMAKE_DEBUG_POSTFIX d)
SET(ENABLE_EXAMPLES OFF CACHE BOOL "Enable building hiredis examples")
SET(hiredis_sources
alloc.c
async.c
dict.c
hiredis.c
net.c
read.c
@@ -42,34 +40,30 @@ SET(hiredis_sources
SET(hiredis_sources ${hiredis_sources})
IF(WIN32)
ADD_COMPILE_DEFINITIONS(_CRT_SECURE_NO_WARNINGS WIN32_LEAN_AND_MEAN)
ADD_DEFINITIONS(-D_CRT_SECURE_NO_WARNINGS -DWIN32_LEAN_AND_MEAN)
ENDIF()
ADD_LIBRARY(hiredis SHARED ${hiredis_sources})
ADD_LIBRARY(hiredis_static STATIC ${hiredis_sources})
ADD_LIBRARY(hiredis ${hiredis_sources})
ADD_LIBRARY(hiredis::hiredis ALIAS hiredis)
ADD_LIBRARY(hiredis::hiredis_static ALIAS hiredis_static)
set(hiredis_export_name hiredis CACHE STRING "Name of the exported target")
set_target_properties(hiredis PROPERTIES EXPORT_NAME ${hiredis_export_name})
SET_TARGET_PROPERTIES(hiredis
PROPERTIES WINDOWS_EXPORT_ALL_SYMBOLS TRUE
VERSION "${HIREDIS_SONAME}")
SET_TARGET_PROPERTIES(hiredis_static
PROPERTIES COMPILE_PDB_NAME hiredis_static)
SET_TARGET_PROPERTIES(hiredis_static
PROPERTIES COMPILE_PDB_NAME_DEBUG hiredis_static${CMAKE_DEBUG_POSTFIX})
IF(WIN32 OR MINGW)
IF(MSVC)
SET_TARGET_PROPERTIES(hiredis
PROPERTIES COMPILE_FLAGS /Z7)
ENDIF()
IF(WIN32)
TARGET_LINK_LIBRARIES(hiredis PUBLIC ws2_32 crypt32)
TARGET_LINK_LIBRARIES(hiredis_static PUBLIC ws2_32 crypt32)
ELSEIF(CMAKE_SYSTEM_NAME MATCHES "FreeBSD")
TARGET_LINK_LIBRARIES(hiredis PUBLIC m)
TARGET_LINK_LIBRARIES(hiredis_static PUBLIC m)
ELSEIF(CMAKE_SYSTEM_NAME MATCHES "SunOS")
TARGET_LINK_LIBRARIES(hiredis PUBLIC socket)
TARGET_LINK_LIBRARIES(hiredis_static PUBLIC socket)
ENDIF()
TARGET_INCLUDE_DIRECTORIES(hiredis PUBLIC $<INSTALL_INTERFACE:include> $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>)
TARGET_INCLUDE_DIRECTORIES(hiredis_static PUBLIC $<INSTALL_INTERFACE:include> $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>)
CONFIGURE_FILE(hiredis.pc.in hiredis.pc @ONLY)
@@ -99,26 +93,23 @@ set(CPACK_RPM_PACKAGE_AUTOREQPROV ON)
include(CPack)
INSTALL(TARGETS hiredis hiredis_static
INSTALL(TARGETS hiredis
EXPORT hiredis-targets
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR})
if (MSVC)
if (MSVC AND BUILD_SHARED_LIBS)
INSTALL(FILES $<TARGET_PDB_FILE:hiredis>
DESTINATION ${CMAKE_INSTALL_BINDIR}
CONFIGURATIONS Debug RelWithDebInfo)
INSTALL(FILES $<TARGET_FILE_DIR:hiredis_static>/$<TARGET_FILE_BASE_NAME:hiredis_static>.pdb
DESTINATION ${CMAKE_INSTALL_LIBDIR}
CONFIGURATIONS Debug RelWithDebInfo)
endif()
# For NuGet packages
INSTALL(FILES hiredis.targets
DESTINATION build/native)
INSTALL(FILES hiredis.h read.h sds.h async.h alloc.h
INSTALL(FILES hiredis.h read.h sds.h async.h alloc.h sockcompat.h
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/hiredis)
INSTALL(DIRECTORY adapters
@@ -131,9 +122,15 @@ export(EXPORT hiredis-targets
FILE "${CMAKE_CURRENT_BINARY_DIR}/hiredis-targets.cmake"
NAMESPACE hiredis::)
SET(CMAKE_CONF_INSTALL_DIR share/hiredis)
if(WIN32)
SET(CMAKE_CONF_INSTALL_DIR share/hiredis)
else()
SET(CMAKE_CONF_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR}/cmake/hiredis)
endif()
SET(INCLUDE_INSTALL_DIR include)
include(CMakePackageConfigHelpers)
write_basic_package_version_file("${CMAKE_CURRENT_BINARY_DIR}/hiredis-config-version.cmake"
COMPATIBILITY SameMajorVersion)
configure_package_config_file(hiredis-config.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/hiredis-config.cmake
INSTALL_DESTINATION ${CMAKE_CONF_INSTALL_DIR}
PATH_VARS INCLUDE_INSTALL_DIR)
@@ -144,6 +141,7 @@ INSTALL(EXPORT hiredis-targets
DESTINATION ${CMAKE_CONF_INSTALL_DIR})
INSTALL(FILES ${CMAKE_CURRENT_BINARY_DIR}/hiredis-config.cmake
${CMAKE_CURRENT_BINARY_DIR}/hiredis-config-version.cmake
DESTINATION ${CMAKE_CONF_INSTALL_DIR})
@@ -156,12 +154,10 @@ IF(ENABLE_SSL)
FIND_PACKAGE(OpenSSL REQUIRED)
SET(hiredis_ssl_sources
ssl.c)
ADD_LIBRARY(hiredis_ssl SHARED
${hiredis_ssl_sources})
ADD_LIBRARY(hiredis_ssl_static STATIC
${hiredis_ssl_sources})
ADD_LIBRARY(hiredis_ssl ${hiredis_ssl_sources})
ADD_LIBRARY(hiredis::hiredis_ssl ALIAS hiredis_ssl)
IF (APPLE)
IF (APPLE AND BUILD_SHARED_LIBS)
SET_PROPERTY(TARGET hiredis_ssl PROPERTY LINK_FLAGS "-Wl,-undefined -Wl,dynamic_lookup")
ENDIF()
@@ -169,34 +165,26 @@ IF(ENABLE_SSL)
PROPERTIES
WINDOWS_EXPORT_ALL_SYMBOLS TRUE
VERSION "${HIREDIS_SONAME}")
SET_TARGET_PROPERTIES(hiredis_ssl_static
PROPERTIES COMPILE_PDB_NAME hiredis_ssl_static)
SET_TARGET_PROPERTIES(hiredis_ssl_static
PROPERTIES COMPILE_PDB_NAME_DEBUG hiredis_ssl_static${CMAKE_DEBUG_POSTFIX})
TARGET_INCLUDE_DIRECTORIES(hiredis_ssl PRIVATE "${OPENSSL_INCLUDE_DIR}")
TARGET_INCLUDE_DIRECTORIES(hiredis_ssl_static PRIVATE "${OPENSSL_INCLUDE_DIR}")
TARGET_LINK_LIBRARIES(hiredis_ssl PRIVATE ${OPENSSL_LIBRARIES})
IF (WIN32 OR MINGW)
IF(MSVC)
SET_TARGET_PROPERTIES(hiredis_ssl
PROPERTIES COMPILE_FLAGS /Z7)
ENDIF()
TARGET_LINK_LIBRARIES(hiredis_ssl PRIVATE OpenSSL::SSL)
IF(WIN32)
TARGET_LINK_LIBRARIES(hiredis_ssl PRIVATE hiredis)
TARGET_LINK_LIBRARIES(hiredis_ssl_static PUBLIC hiredis_static)
ENDIF()
CONFIGURE_FILE(hiredis_ssl.pc.in hiredis_ssl.pc @ONLY)
INSTALL(TARGETS hiredis_ssl hiredis_ssl_static
INSTALL(TARGETS hiredis_ssl
EXPORT hiredis_ssl-targets
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR})
if (MSVC)
if (MSVC AND BUILD_SHARED_LIBS)
INSTALL(FILES $<TARGET_PDB_FILE:hiredis_ssl>
DESTINATION ${CMAKE_INSTALL_BINDIR}
CONFIGURATIONS Debug RelWithDebInfo)
INSTALL(FILES $<TARGET_FILE_DIR:hiredis_ssl_static>/$<TARGET_FILE_BASE_NAME:hiredis_ssl_static>.pdb
DESTINATION ${CMAKE_INSTALL_LIBDIR}
CONFIGURATIONS Debug RelWithDebInfo)
endif()
INSTALL(FILES hiredis_ssl.h
@@ -209,7 +197,11 @@ IF(ENABLE_SSL)
FILE "${CMAKE_CURRENT_BINARY_DIR}/hiredis_ssl-targets.cmake"
NAMESPACE hiredis::)
SET(CMAKE_CONF_INSTALL_DIR share/hiredis_ssl)
if(WIN32)
SET(CMAKE_CONF_INSTALL_DIR share/hiredis_ssl)
else()
SET(CMAKE_CONF_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR}/cmake/hiredis_ssl)
endif()
configure_package_config_file(hiredis_ssl-config.cmake.in ${CMAKE_CURRENT_BINARY_DIR}/hiredis_ssl-config.cmake
INSTALL_DESTINATION ${CMAKE_CONF_INSTALL_DIR}
PATH_VARS INCLUDE_INSTALL_DIR)
@@ -241,5 +233,5 @@ ENDIF()
# Add examples
IF(ENABLE_EXAMPLES)
ADD_SUBDIRECTORY(examples)
ADD_SUBDIRECTORY(examples)
ENDIF(ENABLE_EXAMPLES)
+38 -22
View File
@@ -4,7 +4,7 @@
# This file is released under the BSD license, see the COPYING file
OBJ=alloc.o net.o hiredis.o sds.o async.o read.o sockcompat.o
EXAMPLES=hiredis-example hiredis-example-libevent hiredis-example-libev hiredis-example-glib hiredis-example-push
EXAMPLES=hiredis-example hiredis-example-libevent hiredis-example-libev hiredis-example-glib hiredis-example-push hiredis-example-poll
TESTS=hiredis-test
LIBNAME=libhiredis
PKGCONFNAME=hiredis.pc
@@ -39,9 +39,9 @@ export REDIS_TEST_CONFIG
CC:=$(shell sh -c 'type $${CC%% *} >/dev/null 2>/dev/null && echo $(CC) || echo gcc')
CXX:=$(shell sh -c 'type $${CXX%% *} >/dev/null 2>/dev/null && echo $(CXX) || echo g++')
OPTIMIZATION?=-O3
WARNINGS=-Wall -W -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers
WARNINGS=-Wall -Wextra -Werror -Wstrict-prototypes -Wwrite-strings -Wno-missing-field-initializers
DEBUG_FLAGS?= -g -ggdb
REAL_CFLAGS=$(OPTIMIZATION) -fPIC $(CPPFLAGS) $(CFLAGS) $(WARNINGS) $(DEBUG_FLAGS)
REAL_CFLAGS=$(OPTIMIZATION) -fPIC $(CPPFLAGS) $(CFLAGS) $(WARNINGS) $(DEBUG_FLAGS) $(PLATFORM_FLAGS)
REAL_LDFLAGS=$(LDFLAGS)
DYLIBSUFFIX=so
@@ -50,7 +50,7 @@ DYLIB_MINOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_SONAME)
DYLIB_MAJOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_MAJOR)
DYLIBNAME=$(LIBNAME).$(DYLIBSUFFIX)
DYLIB_MAKE_CMD=$(CC) -shared -Wl,-soname,$(DYLIB_MINOR_NAME)
DYLIB_MAKE_CMD=$(CC) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(DYLIB_MINOR_NAME)
STLIBNAME=$(LIBNAME).$(STLIBSUFFIX)
STLIB_MAKE_CMD=$(AR) rcs
@@ -63,7 +63,7 @@ SSL_DYLIB_MINOR_NAME=$(SSL_LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_SONAME)
SSL_DYLIB_MAJOR_NAME=$(SSL_LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_MAJOR)
SSL_DYLIBNAME=$(SSL_LIBNAME).$(DYLIBSUFFIX)
SSL_STLIBNAME=$(SSL_LIBNAME).$(STLIBSUFFIX)
SSL_DYLIB_MAKE_CMD=$(CC) -shared -Wl,-soname,$(SSL_DYLIB_MINOR_NAME)
SSL_DYLIB_MAKE_CMD=$(CC) $(PLATFORM_FLAGS) -shared -Wl,-soname,$(SSL_DYLIB_MINOR_NAME)
USE_SSL?=0
ifeq ($(USE_SSL),1)
@@ -92,18 +92,25 @@ ifeq ($(TEST_ASYNC),1)
endif
ifeq ($(USE_SSL),1)
ifeq ($(uname_S),Linux)
ifdef OPENSSL_PREFIX
CFLAGS+=-I$(OPENSSL_PREFIX)/include
SSL_LDFLAGS+=-L$(OPENSSL_PREFIX)/lib -lssl -lcrypto
else
SSL_LDFLAGS=-lssl -lcrypto
ifndef OPENSSL_PREFIX
ifeq ($(uname_S),Darwin)
SEARCH_PATH1=/opt/homebrew/opt/openssl
SEARCH_PATH2=/usr/local/opt/openssl
ifneq ($(wildcard $(SEARCH_PATH1)),)
OPENSSL_PREFIX=$(SEARCH_PATH1)
else ifneq ($(wildcard $(SEARCH_PATH2)),)
OPENSSL_PREFIX=$(SEARCH_PATH2)
endif
endif
else
OPENSSL_PREFIX?=/usr/local/opt/openssl
CFLAGS+=-I$(OPENSSL_PREFIX)/include
SSL_LDFLAGS+=-L$(OPENSSL_PREFIX)/lib -lssl -lcrypto
endif
ifdef OPENSSL_PREFIX
CFLAGS+=-I$(OPENSSL_PREFIX)/include
SSL_LDFLAGS+=-L$(OPENSSL_PREFIX)/lib
endif
SSL_LDFLAGS+=-lssl -lcrypto
endif
ifeq ($(uname_S),FreeBSD)
@@ -180,6 +187,9 @@ hiredis-example-libevent-ssl: examples/example-libevent-ssl.c adapters/libevent.
hiredis-example-libev: examples/example-libev.c adapters/libev.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -lev $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-libhv: examples/example-libhv.c adapters/libhv.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< -lhv $(STLIBNAME) $(REAL_LDFLAGS)
hiredis-example-glib: examples/example-glib.c adapters/glib.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(shell pkg-config --cflags --libs glib-2.0) $(STLIBNAME) $(REAL_LDFLAGS)
@@ -192,6 +202,9 @@ hiredis-example-macosx: examples/example-macosx.c adapters/macosx.h $(STLIBNAME)
hiredis-example-ssl: examples/example-ssl.c $(STLIBNAME) $(SSL_STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(STLIBNAME) $(SSL_STLIBNAME) $(REAL_LDFLAGS) $(SSL_LDFLAGS)
hiredis-example-poll: examples/example-poll.c adapters/poll.h $(STLIBNAME)
$(CC) -o examples/$@ $(REAL_CFLAGS) -I. $< $(STLIBNAME) $(REAL_LDFLAGS)
ifndef AE_DIR
hiredis-example-ae:
@echo "Please specify AE_DIR (e.g. <redis repository>/src)"
@@ -269,20 +282,22 @@ $(PKGCONFNAME): hiredis.h
@echo prefix=$(PREFIX) > $@
@echo exec_prefix=\$${prefix} >> $@
@echo libdir=$(PREFIX)/$(LIBRARY_PATH) >> $@
@echo includedir=$(PREFIX)/$(INCLUDE_PATH) >> $@
@echo includedir=$(PREFIX)/include >> $@
@echo pkgincludedir=$(PREFIX)/$(INCLUDE_PATH) >> $@
@echo >> $@
@echo Name: hiredis >> $@
@echo Description: Minimalistic C client library for Redis. >> $@
@echo Version: $(HIREDIS_MAJOR).$(HIREDIS_MINOR).$(HIREDIS_PATCH) >> $@
@echo Libs: -L\$${libdir} -lhiredis >> $@
@echo Cflags: -I\$${includedir} -D_FILE_OFFSET_BITS=64 >> $@
@echo Cflags: -I\$${pkgincludedir} -I\$${includedir} -D_FILE_OFFSET_BITS=64 >> $@
$(SSL_PKGCONFNAME): hiredis_ssl.h
@echo "Generating $@ for pkgconfig..."
@echo prefix=$(PREFIX) > $@
@echo exec_prefix=\$${prefix} >> $@
@echo libdir=$(PREFIX)/$(LIBRARY_PATH) >> $@
@echo includedir=$(PREFIX)/$(INCLUDE_PATH) >> $@
@echo includedir=$(PREFIX)/include >> $@
@echo pkgincludedir=$(PREFIX)/$(INCLUDE_PATH) >> $@
@echo >> $@
@echo Name: hiredis_ssl >> $@
@echo Description: SSL Support for hiredis. >> $@
@@ -293,10 +308,10 @@ $(SSL_PKGCONFNAME): hiredis_ssl.h
install: $(DYLIBNAME) $(STLIBNAME) $(PKGCONFNAME) $(SSL_INSTALL)
mkdir -p $(INSTALL_INCLUDE_PATH) $(INSTALL_INCLUDE_PATH)/adapters $(INSTALL_LIBRARY_PATH)
$(INSTALL) hiredis.h async.h read.h sds.h alloc.h $(INSTALL_INCLUDE_PATH)
$(INSTALL) hiredis.h async.h read.h sds.h alloc.h sockcompat.h $(INSTALL_INCLUDE_PATH)
$(INSTALL) adapters/*.h $(INSTALL_INCLUDE_PATH)/adapters
$(INSTALL) $(DYLIBNAME) $(INSTALL_LIBRARY_PATH)/$(DYLIB_MINOR_NAME)
cd $(INSTALL_LIBRARY_PATH) && ln -sf $(DYLIB_MINOR_NAME) $(DYLIBNAME)
cd $(INSTALL_LIBRARY_PATH) && ln -sf $(DYLIB_MINOR_NAME) $(DYLIBNAME) && ln -sf $(DYLIB_MINOR_NAME) $(DYLIB_MAJOR_NAME)
$(INSTALL) $(STLIBNAME) $(INSTALL_LIBRARY_PATH)
mkdir -p $(INSTALL_PKGCONF_PATH)
$(INSTALL) $(PKGCONFNAME) $(INSTALL_PKGCONF_PATH)
@@ -305,7 +320,7 @@ install-ssl: $(SSL_DYLIBNAME) $(SSL_STLIBNAME) $(SSL_PKGCONFNAME)
mkdir -p $(INSTALL_INCLUDE_PATH) $(INSTALL_LIBRARY_PATH)
$(INSTALL) hiredis_ssl.h $(INSTALL_INCLUDE_PATH)
$(INSTALL) $(SSL_DYLIBNAME) $(INSTALL_LIBRARY_PATH)/$(SSL_DYLIB_MINOR_NAME)
cd $(INSTALL_LIBRARY_PATH) && ln -sf $(SSL_DYLIB_MINOR_NAME) $(SSL_DYLIBNAME)
cd $(INSTALL_LIBRARY_PATH) && ln -sf $(SSL_DYLIB_MINOR_NAME) $(SSL_DYLIBNAME) && ln -sf $(SSL_DYLIB_MINOR_NAME) $(SSL_DYLIB_MAJOR_NAME)
$(INSTALL) $(SSL_STLIBNAME) $(INSTALL_LIBRARY_PATH)
mkdir -p $(INSTALL_PKGCONF_PATH)
$(INSTALL) $(SSL_PKGCONFNAME) $(INSTALL_PKGCONF_PATH)
@@ -330,7 +345,8 @@ coverage: gcov
make check
mkdir -p tmp/lcov
lcov -d . -c --exclude '/usr*' -o tmp/lcov/hiredis.info
genhtml --legend -o tmp/lcov/report tmp/lcov/hiredis.info
lcov -q -l tmp/lcov/hiredis.info
genhtml --legend -q -o tmp/lcov/report tmp/lcov/hiredis.info
noopt:
$(MAKE) OPTIMIZATION=""
+168 -18
View File
@@ -23,6 +23,13 @@ Redis version >= 1.2.0.
The library comes with multiple APIs. There is the
*synchronous API*, the *asynchronous API* and the *reply parsing API*.
## Upgrading to `1.1.0`
Almost all users will simply need to recompile their applications against the newer version of hiredis.
**NOTE**: Hiredis can now return `nan` in addition to `-inf` and `inf` in a `REDIS_REPLY_DOUBLE`.
Applications that deal with `RESP3` doubles should make sure to account for this.
## Upgrading to `1.0.2`
<span style="color:red">NOTE: v1.0.1 erroneously bumped SONAME, which is why it is skipped here.</span>
@@ -82,6 +89,7 @@ an error state. The field `errstr` will contain a string with a description of
the error. More information on errors can be found in the **Errors** section.
After trying to connect to Redis using `redisConnect` you should
check the `err` field to see if establishing the connection was successful:
```c
redisContext *c = redisConnect("127.0.0.1", 6379);
if (c == NULL || c->err) {
@@ -94,8 +102,74 @@ if (c == NULL || c->err) {
}
```
One can also use `redisConnectWithOptions` which takes a `redisOptions` argument
that can be configured with endpoint information as well as many different flags
to change how the `redisContext` will be configured.
```c
redisOptions opt = {0};
/* One can set the endpoint with one of our helper macros */
if (tcp) {
REDIS_OPTIONS_SET_TCP(&opt, "localhost", 6379);
} else {
REDIS_OPTIONS_SET_UNIX(&opt, "/tmp/redis.sock");
}
/* And privdata can be specified with another helper */
REDIS_OPTIONS_SET_PRIVDATA(&opt, myPrivData, myPrivDataDtor);
/* Finally various options may be set via the `options` member, as described below */
opt->options |= REDIS_OPT_PREFER_IPV4;
```
If a connection is lost, `int redisReconnect(redisContext *c)` can be used to restore the connection using the same endpoint and options as the given context.
### Configurable redisOptions flags
There are several flags you may set in the `redisOptions` struct to change default behavior. You can specify the flags via the `redisOptions->options` member.
| Flag | Description |
| --- | --- |
| REDIS\_OPT\_NONBLOCK | Tells hiredis to make a non-blocking connection. |
| REDIS\_OPT\_REUSEADDR | Tells hiredis to set the [SO_REUSEADDR](https://man7.org/linux/man-pages/man7/socket.7.html) socket option |
| REDIS\_OPT\_PREFER\_IPV4<br>REDIS\_OPT\_PREFER_IPV6<br>REDIS\_OPT\_PREFER\_IP\_UNSPEC | Informs hiredis to either prefer IPv4 or IPv6 when invoking [getaddrinfo](https://man7.org/linux/man-pages/man3/gai_strerror.3.html). `REDIS_OPT_PREFER_IP_UNSPEC` will cause hiredis to specify `AF_UNSPEC` in the getaddrinfo call, which means both IPv4 and IPv6 addresses will be searched simultaneously.<br>Hiredis prefers IPv4 by default. |
| REDIS\_OPT\_NO\_PUSH\_AUTOFREE | Tells hiredis to not install the default RESP3 PUSH handler (which just intercepts and frees the replies). This is useful in situations where you want to process these messages in-band. |
| REDIS\_OPT\_NOAUTOFREEREPLIES | **ASYNC**: tells hiredis not to automatically invoke `freeReplyObject` after executing the reply callback. |
| REDIS\_OPT\_NOAUTOFREE | **ASYNC**: Tells hiredis not to automatically free the `redisAsyncContext` on connection/communication failure, but only if the user makes an explicit call to `redisAsyncDisconnect` or `redisAsyncFree` |
*Note: A `redisContext` is not thread-safe.*
### Other configuration using socket options
The following socket options are applied directly to the underlying socket.
The values are not stored in the `redisContext`, so they are not automatically applied when reconnecting using `redisReconnect()`.
These functions return `REDIS_OK` on success.
On failure, `REDIS_ERR` is returned and the underlying connection is closed.
To configure these for an asyncronous context (see *Asynchronous API* below), use `ac->c` to get the redisContext out of an asyncRedisContext.
```C
int redisEnableKeepAlive(redisContext *c);
int redisEnableKeepAliveWithInterval(redisContext *c, int interval);
```
Enables TCP keepalive by setting the following socket options (with some variations depending on OS):
* `SO_KEEPALIVE`;
* `TCP_KEEPALIVE` or `TCP_KEEPIDLE`, value configurable using the `interval` parameter, default 15 seconds;
* `TCP_KEEPINTVL` set to 1/3 of `interval`;
* `TCP_KEEPCNT` set to 3.
```C
int redisSetTcpUserTimeout(redisContext *c, unsigned int timeout);
```
Set the `TCP_USER_TIMEOUT` Linux-specific socket option which is as described in the `tcp` man page:
> When the value is greater than 0, it specifies the maximum amount of time in milliseconds that trans mitted data may remain unacknowledged before TCP will forcibly close the corresponding connection and return ETIMEDOUT to the application.
> If the option value is specified as 0, TCP will use the system default.
### Sending commands
There are several ways to issue commands to Redis. The first that will be introduced is
@@ -250,7 +324,7 @@ following two execution paths:
* Read from the socket until a single reply could be parsed
The function `redisGetReply` is exported as part of the Hiredis API and can be used when a reply
is expected on the socket. To pipeline commands, the only things that needs to be done is
is expected on the socket. To pipeline commands, the only thing that needs to be done is
filling up the output buffer. For this cause, two commands can be used that are identical
to the `redisCommand` family, apart from not returning a reply:
```c
@@ -320,23 +394,48 @@ Redis. It returns a pointer to the newly created `redisAsyncContext` struct. The
should be checked after creation to see if there were errors creating the connection.
Because the connection that will be created is non-blocking, the kernel is not able to
instantly return if the specified host and port is able to accept a connection.
In case of error, it is the caller's responsibility to free the context using `redisAsyncFree()`
*Note: A `redisAsyncContext` is not thread-safe.*
An application function creating a connection might look like this:
```c
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
printf("Error: %s\n", c->errstr);
// handle error
void appConnect(myAppData *appData)
{
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
printf("Error: %s\n", c->errstr);
// handle error
redisAsyncFree(c);
c = NULL;
} else {
appData->context = c;
appData->connecting = 1;
c->data = appData; /* store application pointer for the callbacks */
redisAsyncSetConnectCallback(c, appOnConnect);
redisAsyncSetDisconnectCallback(c, appOnDisconnect);
}
}
```
The asynchronous context can hold a disconnect callback function that is called when the
connection is disconnected (either because of an error or per user request). This function should
The asynchronous context _should_ hold a *connect* callback function that is called when the connection
attempt completes, either successfully or with an error.
It _can_ also hold a *disconnect* callback function that is called when the
connection is disconnected (either because of an error or per user request). Both callbacks should
have the following prototype:
```c
void(const redisAsyncContext *c, int status);
```
On a *connect*, the `status` argument is set to `REDIS_OK` if the connection attempt succeeded. In this
case, the context is ready to accept commands. If it is called with `REDIS_ERR` then the
connection attempt failed. The `err` field in the context can be accessed to find out the cause of the error.
After a failed connection attempt, the context object is automatically freed by the library after calling
the connect callback. This may be a good point to create a new context and retry the connection.
On a disconnect, the `status` argument is set to `REDIS_OK` when disconnection was initiated by the
user, or `REDIS_ERR` when the disconnection was caused by an error. When it is `REDIS_ERR`, the `err`
field in the context can be accessed to find out the cause of the error.
@@ -344,12 +443,46 @@ field in the context can be accessed to find out the cause of the error.
The context object is always freed after the disconnect callback fired. When a reconnect is needed,
the disconnect callback is a good point to do so.
Setting the disconnect callback can only be done once per context. For subsequent calls it will
return `REDIS_ERR`. The function to set the disconnect callback has the following prototype:
Setting the connect or disconnect callbacks can only be done once per context. For subsequent calls the
api will return `REDIS_ERR`. The function to set the callbacks have the following prototype:
```c
/* Alternatively you can use redisAsyncSetConnectCallbackNC which will be passed a non-const
redisAsyncContext* on invocation (e.g. allowing writes to the privdata member). */
int redisAsyncSetConnectCallback(redisAsyncContext *ac, redisConnectCallback *fn);
int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);
```
`ac->data` may be used to pass user data to this callback, the same can be done for redisConnectCallback.
`ac->data` may be used to pass user data to both callbacks. A typical implementation
might look something like this:
```c
void appOnConnect(redisAsyncContext *c, int status)
{
myAppData *appData = (myAppData*)c->data; /* get my application specific context*/
appData->connecting = 0;
if (status == REDIS_OK) {
appData->connected = 1;
} else {
appData->connected = 0;
appData->err = c->err;
appData->context = NULL; /* avoid stale pointer when callback returns */
}
appAttemptReconnect();
}
void appOnDisconnect(redisAsyncContext *c, int status)
{
myAppData *appData = (myAppData*)c->data; /* get my application specific context*/
appData->connected = 0;
appData->err = c->err;
appData->context = NULL; /* avoid stale pointer when callback returns */
if (status == REDIS_OK) {
appNotifyDisconnectCompleted(mydata);
} else {
appNotifyUnexpectedDisconnect(mydata);
appAttemptReconnect();
}
}
```
### Sending commands and their callbacks
In an asynchronous context, commands are automatically pipelined due to the nature of an event loop.
@@ -382,6 +515,14 @@ valid for the duration of the callback.
All pending callbacks are called with a `NULL` reply when the context encountered an error.
For every command issued, with the exception of **SUBSCRIBE** and **PSUBSCRIBE**, the callback is
called exactly once. Even if the context object id disconnected or deleted, every pending callback
will be called with a `NULL` reply.
For **SUBSCRIBE** and **PSUBSCRIBE**, the callbacks may be called repeatedly until an `unsubscribe`
message arrives. This will be the last invocation of the callback. In case of error, the callbacks
may receive a final `NULL` reply instead.
### Disconnecting
An asynchronous connection can be terminated using:
@@ -394,6 +535,15 @@ have been written to the socket, their respective replies have been read and the
callbacks have been executed. After this, the disconnection callback is executed with the
`REDIS_OK` status and the context object is freed.
The connection can be forcefully disconnected using
```c
void redisAsyncFree(redisAsyncContext *ac);
```
In this case, nothing more is written to the socket, all pending callbacks are called with a `NULL`
reply and the disconnection callback is called with `REDIS_OK`, after which the context object
is freed.
### Hooking it up to event library *X*
There are a few hooks that need to be set on the context object after it is created.
@@ -497,8 +647,8 @@ unaffected so no additional dependencies are introduced.
First, you'll need to make sure you include the SSL header file:
```c
#include "hiredis.h"
#include "hiredis_ssl.h"
#include <hiredis/hiredis.h>
#include <hiredis/hiredis_ssl.h>
```
You will also need to link against `libhiredis_ssl`, **in addition** to
@@ -529,7 +679,7 @@ redisSSLContext *ssl_context;
/* An error variable to indicate what went wrong, if the context fails to
* initialize.
*/
redisSSLContextError ssl_error;
redisSSLContextError ssl_error = REDIS_SSL_CTX_NONE;
/* Initialize global OpenSSL state.
*
@@ -547,11 +697,11 @@ ssl_context = redisCreateSSLContext(
"redis.mydomain.com", /* Server name to request (SNI), optional */
&ssl_error);
if(ssl_context == NULL || ssl_error != 0) {
if(ssl_context == NULL || ssl_error != REDIS_SSL_CTX_NONE) {
/* Handle error and abort... */
/* e.g.
printf("SSL error: %s\n",
(ssl_error != 0) ?
/* e.g.
printf("SSL error: %s\n",
(ssl_error != REDIS_SSL_CTX_NONE) ?
redisSSLContextGetError(ssl_error) : "Unknown error");
// Abort
*/
@@ -633,7 +783,7 @@ If you have a unique use-case where you don't want hiredis to automatically inte
redisSetPushCallback(context, NULL);
```
_Note: With no handler configured, calls to `redisCommand` may generate more than one reply, so this strategy is only applicable when there's some kind of blocking`redisGetReply()` loop (e.g. `MONITOR` or `SUBSCRIBE` workloads)._
_Note: With no handler configured, calls to `redisCommand` may generate more than one reply, so this strategy is only applicable when there's some kind of blocking `redisGetReply()` loop (e.g. `MONITOR` or `SUBSCRIBE` workloads)._
## Allocator injection
+123
View File
@@ -0,0 +1,123 @@
#ifndef __HIREDIS_LIBHV_H__
#define __HIREDIS_LIBHV_H__
#include <hv/hloop.h>
#include "../hiredis.h"
#include "../async.h"
typedef struct redisLibhvEvents {
hio_t *io;
htimer_t *timer;
} redisLibhvEvents;
static void redisLibhvHandleEvents(hio_t* io) {
redisAsyncContext* context = (redisAsyncContext*)hevent_userdata(io);
int events = hio_events(io);
int revents = hio_revents(io);
if (context && (events & HV_READ) && (revents & HV_READ)) {
redisAsyncHandleRead(context);
}
if (context && (events & HV_WRITE) && (revents & HV_WRITE)) {
redisAsyncHandleWrite(context);
}
}
static void redisLibhvAddRead(void *privdata) {
redisLibhvEvents* events = (redisLibhvEvents*)privdata;
hio_add(events->io, redisLibhvHandleEvents, HV_READ);
}
static void redisLibhvDelRead(void *privdata) {
redisLibhvEvents* events = (redisLibhvEvents*)privdata;
hio_del(events->io, HV_READ);
}
static void redisLibhvAddWrite(void *privdata) {
redisLibhvEvents* events = (redisLibhvEvents*)privdata;
hio_add(events->io, redisLibhvHandleEvents, HV_WRITE);
}
static void redisLibhvDelWrite(void *privdata) {
redisLibhvEvents* events = (redisLibhvEvents*)privdata;
hio_del(events->io, HV_WRITE);
}
static void redisLibhvCleanup(void *privdata) {
redisLibhvEvents* events = (redisLibhvEvents*)privdata;
if (events->timer)
htimer_del(events->timer);
hio_close(events->io);
hevent_set_userdata(events->io, NULL);
hi_free(events);
}
static void redisLibhvTimeout(htimer_t* timer) {
hio_t* io = (hio_t*)hevent_userdata(timer);
redisAsyncHandleTimeout((redisAsyncContext*)hevent_userdata(io));
}
static void redisLibhvSetTimeout(void *privdata, struct timeval tv) {
redisLibhvEvents* events;
uint32_t millis;
hloop_t* loop;
events = (redisLibhvEvents*)privdata;
millis = tv.tv_sec * 1000 + tv.tv_usec / 1000;
if (millis == 0) {
/* Libhv disallows zero'd timers so treat this as a delete or NO OP */
if (events->timer) {
htimer_del(events->timer);
events->timer = NULL;
}
} else if (events->timer == NULL) {
/* Add new timer */
loop = hevent_loop(events->io);
events->timer = htimer_add(loop, redisLibhvTimeout, millis, 1);
hevent_set_userdata(events->timer, events->io);
} else {
/* Update existing timer */
htimer_reset(events->timer, millis);
}
}
static int redisLibhvAttach(redisAsyncContext* ac, hloop_t* loop) {
redisContext *c = &(ac->c);
redisLibhvEvents *events;
hio_t* io = NULL;
if (ac->ev.data != NULL) {
return REDIS_ERR;
}
/* Create container struct to keep track of our io and any timer */
events = (redisLibhvEvents*)hi_malloc(sizeof(*events));
if (events == NULL) {
return REDIS_ERR;
}
io = hio_get(loop, c->fd);
if (io == NULL) {
hi_free(events);
return REDIS_ERR;
}
hevent_set_userdata(io, ac);
events->io = io;
events->timer = NULL;
ac->ev.addRead = redisLibhvAddRead;
ac->ev.delRead = redisLibhvDelRead;
ac->ev.addWrite = redisLibhvAddWrite;
ac->ev.delWrite = redisLibhvDelWrite;
ac->ev.cleanup = redisLibhvCleanup;
ac->ev.scheduleTimer = redisLibhvSetTimeout;
ac->ev.data = events;
return REDIS_OK;
}
#endif
+177
View File
@@ -0,0 +1,177 @@
#ifndef HIREDIS_LIBSDEVENT_H
#define HIREDIS_LIBSDEVENT_H
#include <systemd/sd-event.h>
#include "../hiredis.h"
#include "../async.h"
#define REDIS_LIBSDEVENT_DELETED 0x01
#define REDIS_LIBSDEVENT_ENTERED 0x02
typedef struct redisLibsdeventEvents {
redisAsyncContext *context;
struct sd_event *event;
struct sd_event_source *fdSource;
struct sd_event_source *timerSource;
int fd;
short flags;
short state;
} redisLibsdeventEvents;
static void redisLibsdeventDestroy(redisLibsdeventEvents *e) {
if (e->fdSource) {
e->fdSource = sd_event_source_disable_unref(e->fdSource);
}
if (e->timerSource) {
e->timerSource = sd_event_source_disable_unref(e->timerSource);
}
sd_event_unref(e->event);
hi_free(e);
}
static int redisLibsdeventTimeoutHandler(sd_event_source *s, uint64_t usec, void *userdata) {
((void)s);
((void)usec);
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
redisAsyncHandleTimeout(e->context);
return 0;
}
static int redisLibsdeventHandler(sd_event_source *s, int fd, uint32_t event, void *userdata) {
((void)s);
((void)fd);
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
e->state |= REDIS_LIBSDEVENT_ENTERED;
#define CHECK_DELETED() if (e->state & REDIS_LIBSDEVENT_DELETED) {\
redisLibsdeventDestroy(e);\
return 0; \
}
if ((event & EPOLLIN) && e->context && (e->state & REDIS_LIBSDEVENT_DELETED) == 0) {
redisAsyncHandleRead(e->context);
CHECK_DELETED();
}
if ((event & EPOLLOUT) && e->context && (e->state & REDIS_LIBSDEVENT_DELETED) == 0) {
redisAsyncHandleWrite(e->context);
CHECK_DELETED();
}
e->state &= ~REDIS_LIBSDEVENT_ENTERED;
#undef CHECK_DELETED
return 0;
}
static void redisLibsdeventAddRead(void *userdata) {
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
if (e->flags & EPOLLIN) {
return;
}
e->flags |= EPOLLIN;
if (e->flags & EPOLLOUT) {
sd_event_source_set_io_events(e->fdSource, e->flags);
} else {
sd_event_add_io(e->event, &e->fdSource, e->fd, e->flags, redisLibsdeventHandler, e);
}
}
static void redisLibsdeventDelRead(void *userdata) {
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
e->flags &= ~EPOLLIN;
if (e->flags) {
sd_event_source_set_io_events(e->fdSource, e->flags);
} else {
e->fdSource = sd_event_source_disable_unref(e->fdSource);
}
}
static void redisLibsdeventAddWrite(void *userdata) {
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
if (e->flags & EPOLLOUT) {
return;
}
e->flags |= EPOLLOUT;
if (e->flags & EPOLLIN) {
sd_event_source_set_io_events(e->fdSource, e->flags);
} else {
sd_event_add_io(e->event, &e->fdSource, e->fd, e->flags, redisLibsdeventHandler, e);
}
}
static void redisLibsdeventDelWrite(void *userdata) {
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
e->flags &= ~EPOLLOUT;
if (e->flags) {
sd_event_source_set_io_events(e->fdSource, e->flags);
} else {
e->fdSource = sd_event_source_disable_unref(e->fdSource);
}
}
static void redisLibsdeventCleanup(void *userdata) {
redisLibsdeventEvents *e = (redisLibsdeventEvents*)userdata;
if (!e) {
return;
}
if (e->state & REDIS_LIBSDEVENT_ENTERED) {
e->state |= REDIS_LIBSDEVENT_DELETED;
} else {
redisLibsdeventDestroy(e);
}
}
static void redisLibsdeventSetTimeout(void *userdata, struct timeval tv) {
redisLibsdeventEvents *e = (redisLibsdeventEvents *)userdata;
uint64_t usec = tv.tv_sec * 1000000 + tv.tv_usec;
if (!e->timerSource) {
sd_event_add_time_relative(e->event, &e->timerSource, CLOCK_MONOTONIC, usec, 1, redisLibsdeventTimeoutHandler, e);
} else {
sd_event_source_set_time_relative(e->timerSource, usec);
}
}
static int redisLibsdeventAttach(redisAsyncContext *ac, struct sd_event *event) {
redisContext *c = &(ac->c);
redisLibsdeventEvents *e;
/* Nothing should be attached when something is already attached */
if (ac->ev.data != NULL)
return REDIS_ERR;
/* Create container for context and r/w events */
e = (redisLibsdeventEvents*)hi_calloc(1, sizeof(*e));
if (e == NULL)
return REDIS_ERR;
/* Initialize and increase event refcount */
e->context = ac;
e->event = event;
e->fd = c->fd;
sd_event_ref(event);
/* Register functions to start/stop listening for events */
ac->ev.addRead = redisLibsdeventAddRead;
ac->ev.delRead = redisLibsdeventDelRead;
ac->ev.addWrite = redisLibsdeventAddWrite;
ac->ev.delWrite = redisLibsdeventDelWrite;
ac->ev.cleanup = redisLibsdeventCleanup;
ac->ev.scheduleTimer = redisLibsdeventSetTimeout;
ac->ev.data = e;
return REDIS_OK;
}
#endif
+8
View File
@@ -30,6 +30,10 @@ static void redisLibuvPoll(uv_poll_t* handle, int status, int events) {
static void redisLibuvAddRead(void *privdata) {
redisLibuvEvents* p = (redisLibuvEvents*)privdata;
if (p->events & UV_READABLE) {
return;
}
p->events |= UV_READABLE;
uv_poll_start(&p->handle, p->events, redisLibuvPoll);
@@ -52,6 +56,10 @@ static void redisLibuvDelRead(void *privdata) {
static void redisLibuvAddWrite(void *privdata) {
redisLibuvEvents* p = (redisLibuvEvents*)privdata;
if (p->events & UV_WRITABLE) {
return;
}
p->events |= UV_WRITABLE;
uv_poll_start(&p->handle, p->events, redisLibuvPoll);
+197
View File
@@ -0,0 +1,197 @@
#ifndef HIREDIS_POLL_H
#define HIREDIS_POLL_H
#include "../async.h"
#include "../sockcompat.h"
#include <string.h> // for memset
#include <errno.h>
/* Values to return from redisPollTick */
#define REDIS_POLL_HANDLED_READ 1
#define REDIS_POLL_HANDLED_WRITE 2
#define REDIS_POLL_HANDLED_TIMEOUT 4
/* An adapter to allow manual polling of the async context by checking the state
* of the underlying file descriptor. Useful in cases where there is no formal
* IO event loop but regular ticking can be used, such as in game engines. */
typedef struct redisPollEvents {
redisAsyncContext *context;
redisFD fd;
char reading, writing;
char in_tick;
char deleted;
double deadline;
} redisPollEvents;
static double redisPollTimevalToDouble(struct timeval *tv) {
if (tv == NULL)
return 0.0;
return tv->tv_sec + tv->tv_usec / 1000000.00;
}
static double redisPollGetNow(void) {
#ifndef _MSC_VER
struct timeval tv;
gettimeofday(&tv,NULL);
return redisPollTimevalToDouble(&tv);
#else
FILETIME ft;
ULARGE_INTEGER li;
GetSystemTimeAsFileTime(&ft);
li.HighPart = ft.dwHighDateTime;
li.LowPart = ft.dwLowDateTime;
return (double)li.QuadPart * 1e-7;
#endif
}
/* Poll for io, handling any pending callbacks. The timeout argument can be
* positive to wait for a maximum given time for IO, zero to poll, or negative
* to wait forever */
static int redisPollTick(redisAsyncContext *ac, double timeout) {
int reading, writing;
struct pollfd pfd;
int handled;
int ns;
int itimeout;
redisPollEvents *e = (redisPollEvents*)ac->ev.data;
if (!e)
return 0;
/* local flags, won't get changed during callbacks */
reading = e->reading;
writing = e->writing;
if (!reading && !writing)
return 0;
pfd.fd = e->fd;
pfd.events = 0;
if (reading)
pfd.events = POLLIN;
if (writing)
pfd.events |= POLLOUT;
if (timeout >= 0.0) {
itimeout = (int)(timeout * 1000.0);
} else {
itimeout = -1;
}
ns = poll(&pfd, 1, itimeout);
if (ns < 0) {
/* ignore the EINTR error */
if (errno != EINTR)
return ns;
ns = 0;
}
handled = 0;
e->in_tick = 1;
if (ns) {
if (reading && (pfd.revents & POLLIN)) {
redisAsyncHandleRead(ac);
handled |= REDIS_POLL_HANDLED_READ;
}
/* on Windows, connection failure is indicated with the Exception fdset.
* handle it the same as writable. */
if (writing && (pfd.revents & (POLLOUT | POLLERR))) {
/* context Read callback may have caused context to be deleted, e.g.
by doing an redisAsyncDisconnect() */
if (!e->deleted) {
redisAsyncHandleWrite(ac);
handled |= REDIS_POLL_HANDLED_WRITE;
}
}
}
/* perform timeouts */
if (!e->deleted && e->deadline != 0.0) {
double now = redisPollGetNow();
if (now >= e->deadline) {
/* deadline has passed. disable timeout and perform callback */
e->deadline = 0.0;
redisAsyncHandleTimeout(ac);
handled |= REDIS_POLL_HANDLED_TIMEOUT;
}
}
/* do a delayed cleanup if required */
if (e->deleted)
hi_free(e);
else
e->in_tick = 0;
return handled;
}
static void redisPollAddRead(void *data) {
redisPollEvents *e = (redisPollEvents*)data;
e->reading = 1;
}
static void redisPollDelRead(void *data) {
redisPollEvents *e = (redisPollEvents*)data;
e->reading = 0;
}
static void redisPollAddWrite(void *data) {
redisPollEvents *e = (redisPollEvents*)data;
e->writing = 1;
}
static void redisPollDelWrite(void *data) {
redisPollEvents *e = (redisPollEvents*)data;
e->writing = 0;
}
static void redisPollCleanup(void *data) {
redisPollEvents *e = (redisPollEvents*)data;
/* if we are currently processing a tick, postpone deletion */
if (e->in_tick)
e->deleted = 1;
else
hi_free(e);
}
static void redisPollScheduleTimer(void *data, struct timeval tv)
{
redisPollEvents *e = (redisPollEvents*)data;
double now = redisPollGetNow();
e->deadline = now + redisPollTimevalToDouble(&tv);
}
static int redisPollAttach(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
redisPollEvents *e;
/* Nothing should be attached when something is already attached */
if (ac->ev.data != NULL)
return REDIS_ERR;
/* Create container for context and r/w events */
e = (redisPollEvents*)hi_malloc(sizeof(*e));
if (e == NULL)
return REDIS_ERR;
memset(e, 0, sizeof(*e));
e->context = ac;
e->fd = c->fd;
e->reading = e->writing = 0;
e->in_tick = e->deleted = 0;
e->deadline = 0.0;
/* Register functions to start/stop listening for events */
ac->ev.addRead = redisPollAddRead;
ac->ev.delRead = redisPollDelRead;
ac->ev.addWrite = redisPollAddWrite;
ac->ev.delWrite = redisPollDelWrite;
ac->ev.scheduleTimer = redisPollScheduleTimer;
ac->ev.cleanup = redisPollCleanup;
ac->ev.data = e;
return REDIS_OK;
}
#endif /* HIREDIS_POLL_H */
+144
View File
@@ -0,0 +1,144 @@
#ifndef __HIREDIS_REDISMODULEAPI_H__
#define __HIREDIS_REDISMODULEAPI_H__
#include "redismodule.h"
#include "../async.h"
#include "../hiredis.h"
#include <sys/types.h>
typedef struct redisModuleEvents {
redisAsyncContext *context;
RedisModuleCtx *module_ctx;
int fd;
int reading, writing;
int timer_active;
RedisModuleTimerID timer_id;
} redisModuleEvents;
static inline void redisModuleReadEvent(int fd, void *privdata, int mask) {
(void) fd;
(void) mask;
redisModuleEvents *e = (redisModuleEvents*)privdata;
redisAsyncHandleRead(e->context);
}
static inline void redisModuleWriteEvent(int fd, void *privdata, int mask) {
(void) fd;
(void) mask;
redisModuleEvents *e = (redisModuleEvents*)privdata;
redisAsyncHandleWrite(e->context);
}
static inline void redisModuleAddRead(void *privdata) {
redisModuleEvents *e = (redisModuleEvents*)privdata;
if (!e->reading) {
e->reading = 1;
RedisModule_EventLoopAdd(e->fd, REDISMODULE_EVENTLOOP_READABLE, redisModuleReadEvent, e);
}
}
static inline void redisModuleDelRead(void *privdata) {
redisModuleEvents *e = (redisModuleEvents*)privdata;
if (e->reading) {
e->reading = 0;
RedisModule_EventLoopDel(e->fd, REDISMODULE_EVENTLOOP_READABLE);
}
}
static inline void redisModuleAddWrite(void *privdata) {
redisModuleEvents *e = (redisModuleEvents*)privdata;
if (!e->writing) {
e->writing = 1;
RedisModule_EventLoopAdd(e->fd, REDISMODULE_EVENTLOOP_WRITABLE, redisModuleWriteEvent, e);
}
}
static inline void redisModuleDelWrite(void *privdata) {
redisModuleEvents *e = (redisModuleEvents*)privdata;
if (e->writing) {
e->writing = 0;
RedisModule_EventLoopDel(e->fd, REDISMODULE_EVENTLOOP_WRITABLE);
}
}
static inline void redisModuleStopTimer(void *privdata) {
redisModuleEvents *e = (redisModuleEvents*)privdata;
if (e->timer_active) {
RedisModule_StopTimer(e->module_ctx, e->timer_id, NULL);
}
e->timer_active = 0;
}
static inline void redisModuleCleanup(void *privdata) {
redisModuleEvents *e = (redisModuleEvents*)privdata;
redisModuleDelRead(privdata);
redisModuleDelWrite(privdata);
redisModuleStopTimer(privdata);
hi_free(e);
}
static inline void redisModuleTimeout(RedisModuleCtx *ctx, void *privdata) {
(void) ctx;
redisModuleEvents *e = (redisModuleEvents*)privdata;
e->timer_active = 0;
redisAsyncHandleTimeout(e->context);
}
static inline void redisModuleSetTimeout(void *privdata, struct timeval tv) {
redisModuleEvents* e = (redisModuleEvents*)privdata;
redisModuleStopTimer(privdata);
mstime_t millis = tv.tv_sec * 1000 + tv.tv_usec / 1000.0;
e->timer_id = RedisModule_CreateTimer(e->module_ctx, millis, redisModuleTimeout, e);
e->timer_active = 1;
}
/* Check if Redis version is compatible with the adapter. */
static inline int redisModuleCompatibilityCheck(void) {
if (!RedisModule_EventLoopAdd ||
!RedisModule_EventLoopDel ||
!RedisModule_CreateTimer ||
!RedisModule_StopTimer) {
return REDIS_ERR;
}
return REDIS_OK;
}
static inline int redisModuleAttach(redisAsyncContext *ac, RedisModuleCtx *module_ctx) {
redisContext *c = &(ac->c);
redisModuleEvents *e;
/* Nothing should be attached when something is already attached */
if (ac->ev.data != NULL)
return REDIS_ERR;
/* Create container for context and r/w events */
e = (redisModuleEvents*)hi_malloc(sizeof(*e));
if (e == NULL)
return REDIS_ERR;
e->context = ac;
e->module_ctx = module_ctx;
e->fd = c->fd;
e->reading = e->writing = 0;
e->timer_active = 0;
/* Register functions to start/stop listening for events */
ac->ev.addRead = redisModuleAddRead;
ac->ev.delRead = redisModuleDelRead;
ac->ev.addWrite = redisModuleAddWrite;
ac->ev.delWrite = redisModuleDelWrite;
ac->ev.cleanup = redisModuleCleanup;
ac->ev.scheduleTimer = redisModuleSetTimeout;
ac->ev.data = e;
return REDIS_OK;
}
#endif
+177 -52
View File
@@ -140,6 +140,7 @@ static redisAsyncContext *redisAsyncInitialize(redisContext *c) {
ac->ev.scheduleTimer = NULL;
ac->onConnect = NULL;
ac->onConnectNC = NULL;
ac->onDisconnect = NULL;
ac->replies.head = NULL;
@@ -148,6 +149,7 @@ static redisAsyncContext *redisAsyncInitialize(redisContext *c) {
ac->sub.replies.tail = NULL;
ac->sub.channels = channels;
ac->sub.patterns = patterns;
ac->sub.pending_unsubs = 0;
return ac;
oom:
@@ -225,17 +227,34 @@ redisAsyncContext *redisAsyncConnectUnix(const char *path) {
return redisAsyncConnectWithOptions(&options);
}
int redisAsyncSetConnectCallback(redisAsyncContext *ac, redisConnectCallback *fn) {
if (ac->onConnect == NULL) {
ac->onConnect = fn;
static int
redisAsyncSetConnectCallbackImpl(redisAsyncContext *ac, redisConnectCallback *fn,
redisConnectCallbackNC *fn_nc)
{
/* If either are already set, this is an error */
if (ac->onConnect || ac->onConnectNC)
return REDIS_ERR;
/* The common way to detect an established connection is to wait for
* the first write event to be fired. This assumes the related event
* library functions are already set. */
_EL_ADD_WRITE(ac);
return REDIS_OK;
if (fn) {
ac->onConnect = fn;
} else if (fn_nc) {
ac->onConnectNC = fn_nc;
}
return REDIS_ERR;
/* The common way to detect an established connection is to wait for
* the first write event to be fired. This assumes the related event
* library functions are already set. */
_EL_ADD_WRITE(ac);
return REDIS_OK;
}
int redisAsyncSetConnectCallback(redisAsyncContext *ac, redisConnectCallback *fn) {
return redisAsyncSetConnectCallbackImpl(ac, fn, NULL);
}
int redisAsyncSetConnectCallbackNC(redisAsyncContext *ac, redisConnectCallbackNC *fn) {
return redisAsyncSetConnectCallbackImpl(ac, NULL, fn);
}
int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn) {
@@ -302,6 +321,43 @@ static void __redisRunPushCallback(redisAsyncContext *ac, redisReply *reply) {
}
}
static void __redisRunConnectCallback(redisAsyncContext *ac, int status)
{
if (ac->onConnect == NULL && ac->onConnectNC == NULL)
return;
if (!(ac->c.flags & REDIS_IN_CALLBACK)) {
ac->c.flags |= REDIS_IN_CALLBACK;
if (ac->onConnect) {
ac->onConnect(ac, status);
} else {
ac->onConnectNC(ac, status);
}
ac->c.flags &= ~REDIS_IN_CALLBACK;
} else {
/* already in callback */
if (ac->onConnect) {
ac->onConnect(ac, status);
} else {
ac->onConnectNC(ac, status);
}
}
}
static void __redisRunDisconnectCallback(redisAsyncContext *ac, int status)
{
if (ac->onDisconnect) {
if (!(ac->c.flags & REDIS_IN_CALLBACK)) {
ac->c.flags |= REDIS_IN_CALLBACK;
ac->onDisconnect(ac, status);
ac->c.flags &= ~REDIS_IN_CALLBACK;
} else {
/* already in callback */
ac->onDisconnect(ac, status);
}
}
}
/* Helper function to free the context. */
static void __redisAsyncFree(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
@@ -337,12 +393,11 @@ static void __redisAsyncFree(redisAsyncContext *ac) {
/* Execute disconnect callback. When redisAsyncFree() initiated destroying
* this context, the status will always be REDIS_OK. */
if (ac->onDisconnect && (c->flags & REDIS_CONNECTED)) {
if (c->flags & REDIS_FREEING) {
ac->onDisconnect(ac,REDIS_OK);
} else {
ac->onDisconnect(ac,(ac->err == 0) ? REDIS_OK : REDIS_ERR);
}
if (c->flags & REDIS_CONNECTED) {
int status = ac->err == 0 ? REDIS_OK : REDIS_ERR;
if (c->flags & REDIS_FREEING)
status = REDIS_OK;
__redisRunDisconnectCallback(ac, status);
}
if (ac->dataCleanup) {
@@ -358,7 +413,11 @@ static void __redisAsyncFree(redisAsyncContext *ac) {
* free'ing. To do so, a flag is set on the context which is picked up by
* redisProcessCallbacks(). Otherwise, the context is immediately free'd. */
void redisAsyncFree(redisAsyncContext *ac) {
if (ac == NULL)
return;
redisContext *c = &(ac->c);
c->flags |= REDIS_FREEING;
if (!(c->flags & REDIS_IN_CALLBACK))
__redisAsyncFree(ac);
@@ -411,15 +470,15 @@ void redisAsyncDisconnect(redisAsyncContext *ac) {
static int __redisGetSubscribeCallback(redisAsyncContext *ac, redisReply *reply, redisCallback *dstcb) {
redisContext *c = &(ac->c);
dict *callbacks;
redisCallback *cb;
redisCallback *cb = NULL;
dictEntry *de;
int pvariant;
char *stype;
hisds sname;
hisds sname = NULL;
/* Match reply with the expected format of a pushed message.
* The type and number of elements (3 to 4) are specified at:
* https://redis.io/topics/pubsub#format-of-pushed-messages */
* https://redis.io/docs/latest/develop/interact/pubsub/#format-of-pushed-messages */
if ((reply->type == REDIS_REPLY_ARRAY && !(c->flags & REDIS_SUPPORTS_PUSH) && reply->elements >= 3) ||
reply->type == REDIS_REPLY_PUSH) {
assert(reply->element[0]->type == REDIS_REPLY_STRING);
@@ -432,42 +491,43 @@ static int __redisGetSubscribeCallback(redisAsyncContext *ac, redisReply *reply,
callbacks = ac->sub.channels;
/* Locate the right callback */
assert(reply->element[1]->type == REDIS_REPLY_STRING);
sname = hi_sdsnewlen(reply->element[1]->str,reply->element[1]->len);
if (sname == NULL)
goto oom;
if (reply->element[1]->type == REDIS_REPLY_STRING) {
sname = hi_sdsnewlen(reply->element[1]->str,reply->element[1]->len);
if (sname == NULL) goto oom;
de = dictFind(callbacks,sname);
if (de != NULL) {
cb = dictGetEntryVal(de);
/* If this is an subscribe reply decrease pending counter. */
if (strcasecmp(stype+pvariant,"subscribe") == 0) {
cb->pending_subs -= 1;
if ((de = dictFind(callbacks,sname)) != NULL) {
cb = dictGetEntryVal(de);
memcpy(dstcb,cb,sizeof(*dstcb));
}
}
memcpy(dstcb,cb,sizeof(*dstcb));
/* If this is an subscribe reply decrease pending counter. */
if (strcasecmp(stype+pvariant,"subscribe") == 0) {
assert(cb != NULL);
cb->pending_subs -= 1;
/* If this is an unsubscribe message, remove it. */
if (strcasecmp(stype+pvariant,"unsubscribe") == 0) {
if (cb->pending_subs == 0)
dictDelete(callbacks,sname);
} else if (strcasecmp(stype+pvariant,"unsubscribe") == 0) {
if (cb == NULL)
ac->sub.pending_unsubs -= 1;
else if (cb->pending_subs == 0)
dictDelete(callbacks,sname);
/* If this was the last unsubscribe message, revert to
* non-subscribe mode. */
assert(reply->element[2]->type == REDIS_REPLY_INTEGER);
/* If this was the last unsubscribe message, revert to
* non-subscribe mode. */
assert(reply->element[2]->type == REDIS_REPLY_INTEGER);
/* Unset subscribed flag only when no pipelined pending subscribe. */
if (reply->element[2]->integer == 0
&& dictSize(ac->sub.channels) == 0
&& dictSize(ac->sub.patterns) == 0) {
c->flags &= ~REDIS_SUBSCRIBED;
/* Unset subscribed flag only when no pipelined pending subscribe
* or pending unsubscribe replies. */
if (reply->element[2]->integer == 0
&& dictSize(ac->sub.channels) == 0
&& dictSize(ac->sub.patterns) == 0
&& ac->sub.pending_unsubs == 0) {
c->flags &= ~REDIS_SUBSCRIBED;
/* Move ongoing regular command callbacks. */
redisCallback cb;
while (__redisShiftCallback(&ac->sub.replies,&cb) == REDIS_OK) {
__redisPushCallback(&ac->replies,&cb);
}
/* Move ongoing regular command callbacks. */
redisCallback cb;
while (__redisShiftCallback(&ac->sub.replies,&cb) == REDIS_OK) {
__redisPushCallback(&ac->replies,&cb);
}
}
}
@@ -479,6 +539,7 @@ static int __redisGetSubscribeCallback(redisAsyncContext *ac, redisReply *reply,
return REDIS_OK;
oom:
__redisSetError(&(ac->c), REDIS_ERR_OOM, "Out of memory");
__redisAsyncCopyError(ac);
return REDIS_ERR;
}
@@ -540,7 +601,7 @@ void redisProcessCallbacks(redisAsyncContext *ac) {
/* Even if the context is subscribed, pending regular
* callbacks will get a reply before pub/sub messages arrive. */
redisCallback cb = {NULL, NULL, 0, NULL};
redisCallback cb = {NULL, NULL, 0, 0, NULL};
if (__redisShiftCallback(&ac->replies,&cb) != REDIS_OK) {
/*
* A spontaneous reply in a not-subscribed context can be the error
@@ -601,7 +662,7 @@ void redisProcessCallbacks(redisAsyncContext *ac) {
}
static void __redisAsyncHandleConnectFailure(redisAsyncContext *ac) {
if (ac->onConnect) ac->onConnect(ac, REDIS_ERR);
__redisRunConnectCallback(ac, REDIS_ERR);
__redisAsyncDisconnect(ac);
}
@@ -626,8 +687,19 @@ static int __redisAsyncHandleConnect(redisAsyncContext *ac) {
return REDIS_ERR;
}
if (ac->onConnect) ac->onConnect(ac, REDIS_OK);
/* flag us as fully connect, but allow the callback
* to disconnect. For that reason, permit the function
* to delete the context here after callback return.
*/
c->flags |= REDIS_CONNECTED;
__redisRunConnectCallback(ac, REDIS_OK);
if ((ac->c.flags & REDIS_DISCONNECTING)) {
redisAsyncDisconnect(ac);
return REDIS_ERR;
} else if ((ac->c.flags & REDIS_FREEING)) {
redisAsyncFree(ac);
return REDIS_ERR;
}
return REDIS_OK;
} else {
return REDIS_OK;
@@ -651,6 +723,8 @@ void redisAsyncRead(redisAsyncContext *ac) {
*/
void redisAsyncHandleRead(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
/* must not be called from a callback */
assert(!(c->flags & REDIS_IN_CALLBACK));
if (!(c->flags & REDIS_CONNECTED)) {
/* Abort connect was not successful. */
@@ -684,6 +758,8 @@ void redisAsyncWrite(redisAsyncContext *ac) {
void redisAsyncHandleWrite(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
/* must not be called from a callback */
assert(!(c->flags & REDIS_IN_CALLBACK));
if (!(c->flags & REDIS_CONNECTED)) {
/* Abort connect was not successful. */
@@ -700,6 +776,8 @@ void redisAsyncHandleWrite(redisAsyncContext *ac) {
void redisAsyncHandleTimeout(redisAsyncContext *ac) {
redisContext *c = &(ac->c);
redisCallback cb;
/* must not be called from a callback */
assert(!(c->flags & REDIS_IN_CALLBACK));
if ((c->flags & REDIS_CONNECTED)) {
if (ac->replies.head == NULL && ac->sub.replies.head == NULL) {
@@ -719,8 +797,8 @@ void redisAsyncHandleTimeout(redisAsyncContext *ac) {
__redisAsyncCopyError(ac);
}
if (!(c->flags & REDIS_CONNECTED) && ac->onConnect) {
ac->onConnect(ac, REDIS_ERR);
if (!(c->flags & REDIS_CONNECTED)) {
__redisRunConnectCallback(ac, REDIS_ERR);
}
while (__redisShiftCallback(&ac->replies, &cb) == REDIS_OK) {
@@ -757,6 +835,7 @@ static int __redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void
redisContext *c = &(ac->c);
redisCallback cb;
struct dict *cbdict;
dictIterator it;
dictEntry *de;
redisCallback *existcb;
int pvariant, hasnext;
@@ -773,6 +852,7 @@ static int __redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void
cb.fn = fn;
cb.privdata = privdata;
cb.pending_subs = 1;
cb.unsubscribe_sent = 0;
/* Find out which command will be appended. */
p = nextArgument(cmd,&cstr,&clen);
@@ -812,6 +892,51 @@ static int __redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void
* subscribed to one or more channels or patterns. */
if (!(c->flags & REDIS_SUBSCRIBED)) return REDIS_ERR;
if (pvariant)
cbdict = ac->sub.patterns;
else
cbdict = ac->sub.channels;
if (hasnext) {
/* Send an unsubscribe with specific channels/patterns.
* Bookkeeping the number of expected replies */
while ((p = nextArgument(p,&astr,&alen)) != NULL) {
sname = hi_sdsnewlen(astr,alen);
if (sname == NULL)
goto oom;
de = dictFind(cbdict,sname);
if (de != NULL) {
existcb = dictGetEntryVal(de);
if (existcb->unsubscribe_sent == 0)
existcb->unsubscribe_sent = 1;
else
/* Already sent, reply to be ignored */
ac->sub.pending_unsubs += 1;
} else {
/* Not subscribed to, reply to be ignored */
ac->sub.pending_unsubs += 1;
}
hi_sdsfree(sname);
}
} else {
/* Send an unsubscribe without specific channels/patterns.
* Bookkeeping the number of expected replies */
int no_subs = 1;
dictInitIterator(&it,cbdict);
while ((de = dictNext(&it)) != NULL) {
existcb = dictGetEntryVal(de);
if (existcb->unsubscribe_sent == 0) {
existcb->unsubscribe_sent = 1;
no_subs = 0;
}
}
/* Unsubscribing to all channels/patterns, where none is
* subscribed to, results in a single reply to be ignored. */
if (no_subs == 1)
ac->sub.pending_unsubs += 1;
}
/* (P)UNSUBSCRIBE does not have its own response: every channel or
* pattern that is unsubscribed will receive a message. This means we
* should not append a callback function for this command. */
+5
View File
@@ -46,6 +46,7 @@ typedef struct redisCallback {
struct redisCallback *next; /* simple singly linked list */
redisCallbackFn *fn;
int pending_subs;
int unsubscribe_sent;
void *privdata;
} redisCallback;
@@ -57,6 +58,7 @@ typedef struct redisCallbackList {
/* Connection callback prototypes */
typedef void (redisDisconnectCallback)(const struct redisAsyncContext*, int status);
typedef void (redisConnectCallback)(const struct redisAsyncContext*, int status);
typedef void (redisConnectCallbackNC)(struct redisAsyncContext *, int status);
typedef void(redisTimerCallback)(void *timer, void *privdata);
/* Context for an async connection to Redis */
@@ -92,6 +94,7 @@ typedef struct redisAsyncContext {
/* Called when the first write event was received. */
redisConnectCallback *onConnect;
redisConnectCallbackNC *onConnectNC;
/* Regular command callbacks */
redisCallbackList replies;
@@ -105,6 +108,7 @@ typedef struct redisAsyncContext {
redisCallbackList replies;
struct dict *channels;
struct dict *patterns;
int pending_unsubs;
} sub;
/* Any configured RESP3 PUSH handler */
@@ -119,6 +123,7 @@ redisAsyncContext *redisAsyncConnectBindWithReuse(const char *ip, int port,
const char *source_addr);
redisAsyncContext *redisAsyncConnectUnix(const char *path);
int redisAsyncSetConnectCallback(redisAsyncContext *ac, redisConnectCallback *fn);
int redisAsyncSetConnectCallbackNC(redisAsyncContext *ac, redisConnectCallbackNC *fn);
int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);
redisAsyncPushFn *redisAsyncSetPushCallback(redisAsyncContext *ac, redisAsyncPushFn *fn);
+12
View File
@@ -25,12 +25,24 @@ if (LIBEVENT)
TARGET_LINK_LIBRARIES(example-libevent hiredis event)
ENDIF()
FIND_PATH(LIBHV hv/hv.h)
IF (LIBHV)
ADD_EXECUTABLE(example-libhv example-libhv.c)
TARGET_LINK_LIBRARIES(example-libhv hiredis hv)
ENDIF()
FIND_PATH(LIBUV uv.h)
IF (LIBUV)
ADD_EXECUTABLE(example-libuv example-libuv.c)
TARGET_LINK_LIBRARIES(example-libuv hiredis uv)
ENDIF()
FIND_PATH(LIBSDEVENT systemd/sd-event.h)
IF (LIBSDEVENT)
ADD_EXECUTABLE(example-libsdevent example-libsdevent.c)
TARGET_LINK_LIBRARIES(example-libsdevent hiredis systemd)
ENDIF()
IF (APPLE)
FIND_LIBRARY(CF CoreFoundation)
ADD_EXECUTABLE(example-macosx example-macosx.c)
+1 -1
View File
@@ -56,7 +56,7 @@ int main (int argc, char **argv) {
const char *caCert = argc > 5 ? argv[6] : NULL;
redisSSLContext *ssl;
redisSSLContextError ssl_error;
redisSSLContextError ssl_error = REDIS_SSL_CTX_NONE;
redisInitOpenSSL();
+70
View File
@@ -0,0 +1,70 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/libhv.h>
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) return;
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
redisAsyncDisconnect(c);
}
void debugCallback(redisAsyncContext *c, void *r, void *privdata) {
(void)privdata;
redisReply *reply = r;
if (reply == NULL) {
printf("`DEBUG SLEEP` error: %s\n", c->errstr ? c->errstr : "unknown error");
return;
}
redisAsyncDisconnect(c);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Disconnected...\n");
}
int main (int argc, char **argv) {
#ifndef _WIN32
signal(SIGPIPE, SIG_IGN);
#endif
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
hloop_t* loop = hloop_new(HLOOP_FLAG_QUIT_WHEN_NO_ACTIVE_EVENTS);
redisLibhvAttach(c, loop);
redisAsyncSetTimeout(c, (struct timeval){.tv_sec = 0, .tv_usec = 500000});
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
redisAsyncCommand(c, debugCallback, NULL, "DEBUG SLEEP %d", 1);
hloop_run(loop);
hloop_free(&loop);
return 0;
}
+86
View File
@@ -0,0 +1,86 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/libsdevent.h>
void debugCallback(redisAsyncContext *c, void *r, void *privdata) {
(void)privdata;
redisReply *reply = r;
if (reply == NULL) {
/* The DEBUG SLEEP command will almost always fail, because we have set a 1 second timeout */
printf("`DEBUG SLEEP` error: %s\n", c->errstr ? c->errstr : "unknown error");
return;
}
/* Disconnect after receiving the reply of DEBUG SLEEP (which will not)*/
redisAsyncDisconnect(c);
}
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) {
printf("`GET key` error: %s\n", c->errstr ? c->errstr : "unknown error");
return;
}
printf("`GET key` result: argv[%s]: %s\n", (char*)privdata, reply->str);
/* start another request that demonstrate timeout */
redisAsyncCommand(c, debugCallback, NULL, "DEBUG SLEEP %f", 1.5);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("connect error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("disconnect because of error: %s\n", c->errstr);
return;
}
printf("Disconnected...\n");
}
int main (int argc, char **argv) {
signal(SIGPIPE, SIG_IGN);
struct sd_event *event;
sd_event_default(&event);
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
printf("Error: %s\n", c->errstr);
redisAsyncFree(c);
return 1;
}
redisLibsdeventAttach(c,event);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncSetTimeout(c, (struct timeval){ .tv_sec = 1, .tv_usec = 0});
/*
In this demo, we first `set key`, then `get key` to demonstrate the basic usage of libsdevent adapter.
Then in `getCallback`, we start a `debug sleep` command to create 1.5 second long request.
Because we have set a 1 second timeout to the connection, the command will always fail with a
timeout error, which is shown in the `debugCallback`.
*/
redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
/* sd-event does not quit when there are no handlers registered. Manually exit after 1.5 seconds */
sd_event_source *s;
sd_event_add_time_relative(event, &s, CLOCK_MONOTONIC, 1500000, 1, NULL, NULL);
sd_event_loop(event);
sd_event_source_disable_unref(s);
sd_event_unref(event);
return 0;
}
+62
View File
@@ -0,0 +1,62 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <unistd.h>
#include <async.h>
#include <adapters/poll.h>
/* Put in the global scope, so that loop can be explicitly stopped */
static int exit_loop = 0;
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) return;
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* Disconnect after receiving the reply to GET */
redisAsyncDisconnect(c);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
exit_loop = 1;
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
exit_loop = 1;
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Disconnected...\n");
}
int main (int argc, char **argv) {
signal(SIGPIPE, SIG_IGN);
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
redisPollAttach(c);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
while (!exit_loop)
{
redisPollTick(c, 0.1);
}
return 0;
}
+101
View File
@@ -0,0 +1,101 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <hiredis.h>
#include <async.h>
#include <adapters/redismoduleapi.h>
void debugCallback(redisAsyncContext *c, void *r, void *privdata) {
(void)privdata; //unused
redisReply *reply = r;
if (reply == NULL) {
/* The DEBUG SLEEP command will almost always fail, because we have set a 1 second timeout */
printf("`DEBUG SLEEP` error: %s\n", c->errstr ? c->errstr : "unknown error");
return;
}
/* Disconnect after receiving the reply of DEBUG SLEEP (which will not)*/
redisAsyncDisconnect(c);
}
void getCallback(redisAsyncContext *c, void *r, void *privdata) {
redisReply *reply = r;
if (reply == NULL) {
if (c->errstr) {
printf("errstr: %s\n", c->errstr);
}
return;
}
printf("argv[%s]: %s\n", (char*)privdata, reply->str);
/* start another request that demonstrate timeout */
redisAsyncCommand(c, debugCallback, NULL, "DEBUG SLEEP %f", 1.5);
}
void connectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Connected...\n");
}
void disconnectCallback(const redisAsyncContext *c, int status) {
if (status != REDIS_OK) {
printf("Error: %s\n", c->errstr);
return;
}
printf("Disconnected...\n");
}
/*
* This example requires Redis 7.0 or above.
*
* 1- Compile this file as a shared library. Directory of "redismodule.h" must
* be in the include path.
* gcc -fPIC -shared -I../../redis/src/ -I.. example-redismoduleapi.c -o example-redismoduleapi.so
*
* 2- Load module:
* redis-server --loadmodule ./example-redismoduleapi.so value
*/
int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
int ret = RedisModule_Init(ctx, "example-redismoduleapi", 1, REDISMODULE_APIVER_1);
if (ret != REDISMODULE_OK) {
printf("error module init \n");
return REDISMODULE_ERR;
}
if (redisModuleCompatibilityCheck() != REDIS_OK) {
printf("Redis 7.0 or above is required! \n");
return REDISMODULE_ERR;
}
redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
if (c->err) {
/* Let *c leak for now... */
printf("Error: %s\n", c->errstr);
return 1;
}
size_t len;
const char *val = RedisModule_StringPtrLen(argv[argc-1], &len);
RedisModuleCtx *module_ctx = RedisModule_GetDetachedThreadSafeContext(ctx);
redisModuleAttach(c, module_ctx);
redisAsyncSetConnectCallback(c,connectCallback);
redisAsyncSetDisconnectCallback(c,disconnectCallback);
redisAsyncSetTimeout(c, (struct timeval){ .tv_sec = 1, .tv_usec = 0});
/*
In this demo, we first `set key`, then `get key` to demonstrate the basic usage of the adapter.
Then in `getCallback`, we start a `debug sleep` command to create 1.5 second long request.
Because we have set a 1 second timeout to the connection, the command will always fail with a
timeout error, which is shown in the `debugCallback`.
*/
redisAsyncCommand(c, NULL, NULL, "SET key %b", val, len);
redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
return 0;
}
+3 -4
View File
@@ -12,7 +12,7 @@
int main(int argc, char **argv) {
unsigned int j;
redisSSLContext *ssl;
redisSSLContextError ssl_error;
redisSSLContextError ssl_error = REDIS_SSL_CTX_NONE;
redisContext *c;
redisReply *reply;
if (argc < 4) {
@@ -27,9 +27,8 @@ int main(int argc, char **argv) {
redisInitOpenSSL();
ssl = redisCreateSSLContext(ca, NULL, cert, key, NULL, &ssl_error);
if (!ssl) {
printf("SSL Context error: %s\n",
redisSSLContextGetError(ssl_error));
if (!ssl || ssl_error != REDIS_SSL_CTX_NONE) {
printf("SSL Context error: %s\n", redisSSLContextGetError(ssl_error));
exit(1);
}
+51
View File
@@ -7,6 +7,54 @@
#include <winsock2.h> /* For struct timeval */
#endif
static void example_argv_command(redisContext *c, size_t n) {
char **argv, tmp[42];
size_t *argvlen;
redisReply *reply;
/* We're allocating two additional elements for command and key */
argv = malloc(sizeof(*argv) * (2 + n));
argvlen = malloc(sizeof(*argvlen) * (2 + n));
/* First the command */
argv[0] = (char*)"RPUSH";
argvlen[0] = sizeof("RPUSH") - 1;
/* Now our key */
argv[1] = (char*)"argvlist";
argvlen[1] = sizeof("argvlist") - 1;
/* Now add the entries we wish to add to the list */
for (size_t i = 2; i < (n + 2); i++) {
argvlen[i] = snprintf(tmp, sizeof(tmp), "argv-element-%zu", i - 2);
argv[i] = strdup(tmp);
}
/* Execute the command using redisCommandArgv. We're sending the arguments with
* two explicit arrays. One for each argument's string, and the other for its
* length. */
reply = redisCommandArgv(c, n + 2, (const char **)argv, (const size_t*)argvlen);
if (reply == NULL || c->err) {
fprintf(stderr, "Error: Couldn't execute redisCommandArgv\n");
exit(1);
}
if (reply->type == REDIS_REPLY_INTEGER) {
printf("%s reply: %lld\n", argv[0], reply->integer);
}
freeReplyObject(reply);
/* Clean up */
for (size_t i = 2; i < (n + 2); i++) {
free(argv[i]);
}
free(argv);
free(argvlen);
}
int main(int argc, char **argv) {
unsigned int j, isunix = 0;
redisContext *c;
@@ -87,6 +135,9 @@ int main(int argc, char **argv) {
}
freeReplyObject(reply);
/* See function for an example of redisCommandArgv */
example_argv_command(c, 10);
/* Disconnects and frees the context */
redisFree(c);
+2 -3
View File
@@ -48,10 +48,9 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
memcpy(new_str, data, size);
new_str[size] = '\0';
redisFormatCommand(&cmd, new_str);
if (cmd != NULL)
if (redisFormatCommand(&cmd, new_str) != -1)
hi_free(cmd);
free(new_str);
return 0;
}
+2 -2
View File
@@ -2,11 +2,11 @@
set_and_check(hiredis_INCLUDEDIR "@PACKAGE_INCLUDE_INSTALL_DIR@")
IF (NOT TARGET hiredis::hiredis)
IF (NOT TARGET hiredis::@hiredis_export_name@)
INCLUDE(${CMAKE_CURRENT_LIST_DIR}/hiredis-targets.cmake)
ENDIF()
SET(hiredis_LIBRARIES hiredis::hiredis)
SET(hiredis_LIBRARIES hiredis::@hiredis_export_name@)
SET(hiredis_INCLUDE_DIRS ${hiredis_INCLUDEDIR})
check_required_components(hiredis)
+42 -11
View File
@@ -48,6 +48,7 @@ extern int redisContextUpdateConnectTimeout(redisContext *c, const struct timeva
extern int redisContextUpdateCommandTimeout(redisContext *c, const struct timeval *timeout);
static redisContextFuncs redisContextDefaultFuncs = {
.close = redisNetClose,
.free_privctx = NULL,
.async_read = redisAsyncRead,
.async_write = redisAsyncWrite,
@@ -221,6 +222,9 @@ static void *createIntegerObject(const redisReadTask *task, long long value) {
static void *createDoubleObject(const redisReadTask *task, double value, char *str, size_t len) {
redisReply *r, *parent;
if (len == SIZE_MAX) // Prevents hi_malloc(0) if len equals to SIZE_MAX
return NULL;
r = createReplyObject(REDIS_REPLY_DOUBLE);
if (r == NULL)
return NULL;
@@ -388,17 +392,22 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
while (*_p != '\0' && strchr(flags,*_p) != NULL) _p++;
/* Field width */
while (*_p != '\0' && isdigit(*_p)) _p++;
while (*_p != '\0' && isdigit((int) *_p)) _p++;
/* Precision */
if (*_p == '.') {
_p++;
while (*_p != '\0' && isdigit(*_p)) _p++;
while (*_p != '\0' && isdigit((int) *_p)) _p++;
}
/* Copy va_list before consuming with va_arg */
va_copy(_cpy,ap);
/* Make sure we have more characters otherwise strchr() accepts
* '\0' as an integer specifier. This is checked after above
* va_copy() to avoid UB in fmt_invalid's call to va_end(). */
if (*_p == '\0') goto fmt_invalid;
/* Integer conversion (without modifiers) */
if (strchr(intfmts,*_p) != NULL) {
va_arg(ap,int);
@@ -477,6 +486,8 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
touched = 1;
c++;
if (*c == '\0')
break;
}
c++;
}
@@ -719,7 +730,10 @@ static redisContext *redisContextInit(void) {
void redisFree(redisContext *c) {
if (c == NULL)
return;
redisNetClose(c);
if (c->funcs && c->funcs->close) {
c->funcs->close(c);
}
hi_sdsfree(c->obuf);
redisReaderFree(c->reader);
@@ -733,7 +747,7 @@ void redisFree(redisContext *c) {
if (c->privdata && c->free_privdata)
c->free_privdata(c->privdata);
if (c->funcs->free_privctx)
if (c->funcs && c->funcs->free_privctx)
c->funcs->free_privctx(c->privctx);
memset(c, 0xff, sizeof(*c));
@@ -756,7 +770,9 @@ int redisReconnect(redisContext *c) {
c->privctx = NULL;
}
redisNetClose(c);
if (c->funcs && c->funcs->close) {
c->funcs->close(c);
}
hi_sdsfree(c->obuf);
redisReaderFree(c->reader);
@@ -806,6 +822,12 @@ redisContext *redisConnectWithOptions(const redisOptions *options) {
if (options->options & REDIS_OPT_NOAUTOFREEREPLIES) {
c->flags |= REDIS_NO_AUTO_FREE_REPLIES;
}
if (options->options & REDIS_OPT_PREFER_IPV4) {
c->flags |= REDIS_PREFER_IPV4;
}
if (options->options & REDIS_OPT_PREFER_IPV6) {
c->flags |= REDIS_PREFER_IPV6;
}
/* Set any user supplied RESP3 PUSH handler or use freeReplyObject
* as a default unless specifically flagged that we don't want one. */
@@ -838,7 +860,9 @@ redisContext *redisConnectWithOptions(const redisOptions *options) {
return NULL;
}
if (options->command_timeout != NULL && (c->flags & REDIS_BLOCK) && c->fd != REDIS_INVALID_FD) {
if (c->err == 0 && c->fd != REDIS_INVALID_FD &&
options->command_timeout != NULL && (c->flags & REDIS_BLOCK))
{
redisContextSetTimeout(c, *options->command_timeout);
}
@@ -920,11 +944,18 @@ int redisSetTimeout(redisContext *c, const struct timeval tv) {
return REDIS_ERR;
}
int redisEnableKeepAliveWithInterval(redisContext *c, int interval) {
return redisKeepAlive(c, interval);
}
/* Enable connection KeepAlive. */
int redisEnableKeepAlive(redisContext *c) {
if (redisKeepAlive(c, REDIS_KEEPALIVE_INTERVAL) != REDIS_OK)
return REDIS_ERR;
return REDIS_OK;
return redisKeepAlive(c, REDIS_KEEPALIVE_INTERVAL);
}
/* Set the socket option TCP_USER_TIMEOUT. */
int redisSetTcpUserTimeout(redisContext *c, unsigned int timeout) {
return redisContextSetTcpUserTimeout(c, timeout);
}
/* Set a user provided RESP3 PUSH handler and return any old one set. */
@@ -964,8 +995,8 @@ int redisBufferRead(redisContext *c) {
* successfully written to the socket. When the buffer is empty after the
* write operation, "done" is set to 1 (if given).
*
* Returns REDIS_ERR if an error occurred trying to write and sets
* c->errstr to hold the appropriate error string.
* Returns REDIS_ERR if an unrecoverable error occurred in the underlying
* c->funcs->write function.
*/
int redisBufferWrite(redisContext *c, int *done) {
+41 -27
View File
@@ -46,9 +46,9 @@ typedef long long ssize_t;
#include "alloc.h" /* for allocation wrappers */
#define HIREDIS_MAJOR 1
#define HIREDIS_MINOR 0
#define HIREDIS_PATCH 3
#define HIREDIS_SONAME 1.0.3-dev
#define HIREDIS_MINOR 2
#define HIREDIS_PATCH 0
#define HIREDIS_SONAME 1.1.0
/* Connection type can be blocking or non-blocking and is set in the
* least significant bit of the flags field in redisContext. */
@@ -92,6 +92,11 @@ typedef long long ssize_t;
/* Flag that indicates the user does not want replies to be automatically freed */
#define REDIS_NO_AUTO_FREE_REPLIES 0x400
/* Flags to prefer IPv6 or IPv4 when doing DNS lookup. (If both are set,
* AF_UNSPEC is used.) */
#define REDIS_PREFER_IPV4 0x800
#define REDIS_PREFER_IPV6 0x1000
#define REDIS_KEEPALIVE_INTERVAL 15 /* seconds */
/* number of times we retry to connect in the case of EADDRNOTAVAIL and
@@ -149,20 +154,17 @@ struct redisSsl;
#define REDIS_OPT_NONBLOCK 0x01
#define REDIS_OPT_REUSEADDR 0x02
/**
* Don't automatically free the async object on a connection failure,
* or other implicit conditions. Only free on an explicit call to disconnect() or free()
*/
#define REDIS_OPT_NOAUTOFREE 0x04
/* Don't automatically intercept and free RESP3 PUSH replies. */
#define REDIS_OPT_NO_PUSH_AUTOFREE 0x08
/**
* Don't automatically free replies
*/
#define REDIS_OPT_NOAUTOFREEREPLIES 0x10
#define REDIS_OPT_NOAUTOFREE 0x04 /* Don't automatically free the async
* object on a connection failure, or
* other implicit conditions. Only free
* on an explicit call to disconnect()
* or free() */
#define REDIS_OPT_NO_PUSH_AUTOFREE 0x08 /* Don't automatically intercept and
* free RESP3 PUSH replies. */
#define REDIS_OPT_NOAUTOFREEREPLIES 0x10 /* Don't automatically free replies. */
#define REDIS_OPT_PREFER_IPV4 0x20 /* Prefer IPv4 in DNS lookups. */
#define REDIS_OPT_PREFER_IPV6 0x40 /* Prefer IPv6 in DNS lookups. */
#define REDIS_OPT_PREFER_IP_UNSPEC (REDIS_OPT_PREFER_IPV4 | REDIS_OPT_PREFER_IPV6)
/* In Unix systems a file descriptor is a regular signed int, with -1
* representing an invalid descriptor. In Windows it is a SOCKET
@@ -220,27 +222,37 @@ typedef struct {
/**
* Helper macros to initialize options to their specified fields.
*/
#define REDIS_OPTIONS_SET_TCP(opts, ip_, port_) \
(opts)->type = REDIS_CONN_TCP; \
(opts)->endpoint.tcp.ip = ip_; \
(opts)->endpoint.tcp.port = port_;
#define REDIS_OPTIONS_SET_TCP(opts, ip_, port_) do { \
(opts)->type = REDIS_CONN_TCP; \
(opts)->endpoint.tcp.ip = ip_; \
(opts)->endpoint.tcp.port = port_; \
} while(0)
#define REDIS_OPTIONS_SET_UNIX(opts, path) \
(opts)->type = REDIS_CONN_UNIX; \
(opts)->endpoint.unix_socket = path;
#define REDIS_OPTIONS_SET_UNIX(opts, path) do { \
(opts)->type = REDIS_CONN_UNIX; \
(opts)->endpoint.unix_socket = path; \
} while(0)
#define REDIS_OPTIONS_SET_PRIVDATA(opts, data, dtor) \
(opts)->privdata = data; \
(opts)->free_privdata = dtor; \
#define REDIS_OPTIONS_SET_PRIVDATA(opts, data, dtor) do { \
(opts)->privdata = data; \
(opts)->free_privdata = dtor; \
} while(0)
typedef struct redisContextFuncs {
void (*close)(struct redisContext *);
void (*free_privctx)(void *);
void (*async_read)(struct redisAsyncContext *);
void (*async_write)(struct redisAsyncContext *);
/* Read/Write data to the underlying communication stream, returning the
* number of bytes read/written. In the event of an unrecoverable error
* these functions shall return a value < 0. In the event of a
* recoverable error, they should return 0. */
ssize_t (*read)(struct redisContext *, char *, size_t);
ssize_t (*write)(struct redisContext *);
} redisContextFuncs;
/* Context for a connection to Redis */
typedef struct redisContext {
const redisContextFuncs *funcs; /* Function table */
@@ -310,6 +322,8 @@ int redisReconnect(redisContext *c);
redisPushFn *redisSetPushCallback(redisContext *c, redisPushFn *fn);
int redisSetTimeout(redisContext *c, const struct timeval tv);
int redisEnableKeepAlive(redisContext *c);
int redisEnableKeepAliveWithInterval(redisContext *c, int interval);
int redisSetTcpUserTimeout(redisContext *c, unsigned int timeout);
void redisFree(redisContext *c);
redisFD redisFreeKeepFd(redisContext *c);
int redisBufferRead(redisContext *c);
+1 -1
View File
@@ -9,4 +9,4 @@ Name: hiredis
Description: Minimalistic C client library for Redis.
Version: @PROJECT_VERSION@
Libs: -L${libdir} -lhiredis
Cflags: -I${pkgincludedir} -D_FILE_OFFSET_BITS=64
Cflags: -I${pkgincludedir} -I${includedir} -D_FILE_OFFSET_BITS=64
+3
View File
@@ -2,6 +2,9 @@
set_and_check(hiredis_ssl_INCLUDEDIR "@PACKAGE_INCLUDE_INSTALL_DIR@")
include(CMakeFindDependencyMacro)
find_dependency(OpenSSL)
IF (NOT TARGET hiredis::hiredis_ssl)
INCLUDE(${CMAKE_CURRENT_LIST_DIR}/hiredis_ssl-targets.cmake)
ENDIF()
+34
View File
@@ -56,11 +56,33 @@ typedef enum {
REDIS_SSL_CTX_CERT_KEY_REQUIRED, /* Client cert and key must both be specified or skipped */
REDIS_SSL_CTX_CA_CERT_LOAD_FAILED, /* Failed to load CA Certificate or CA Path */
REDIS_SSL_CTX_CLIENT_CERT_LOAD_FAILED, /* Failed to load client certificate */
REDIS_SSL_CTX_CLIENT_DEFAULT_CERT_FAILED, /* Failed to set client default certificate directory */
REDIS_SSL_CTX_PRIVATE_KEY_LOAD_FAILED, /* Failed to load private key */
REDIS_SSL_CTX_OS_CERTSTORE_OPEN_FAILED, /* Failed to open system certificate store */
REDIS_SSL_CTX_OS_CERT_ADD_FAILED /* Failed to add CA certificates obtained from system to the SSL context */
} redisSSLContextError;
/* Constants that mirror OpenSSL's verify modes. By default,
* REDIS_SSL_VERIFY_PEER is used with redisCreateSSLContext().
* Some Redis clients disable peer verification if there are no
* certificates specified.
*/
#define REDIS_SSL_VERIFY_NONE 0x00
#define REDIS_SSL_VERIFY_PEER 0x01
#define REDIS_SSL_VERIFY_FAIL_IF_NO_PEER_CERT 0x02
#define REDIS_SSL_VERIFY_CLIENT_ONCE 0x04
#define REDIS_SSL_VERIFY_POST_HANDSHAKE 0x08
/* Options to create an OpenSSL context. */
typedef struct {
const char *cacert_filename;
const char *capath;
const char *cert_filename;
const char *private_key_filename;
const char *server_name;
int verify_mode;
} redisSSLOptions;
/**
* Return the error message corresponding with the specified error code.
*/
@@ -101,6 +123,18 @@ redisSSLContext *redisCreateSSLContext(const char *cacert_filename, const char *
const char *cert_filename, const char *private_key_filename,
const char *server_name, redisSSLContextError *error);
/**
* Helper function to initialize an OpenSSL context that can be used
* to initiate SSL connections. This is a more extensible version of redisCreateSSLContext().
*
* options contains a structure of SSL options to use.
*
* If error is non-null, it will be populated in case the context creation fails
* (returning a NULL).
*/
redisSSLContext *redisCreateSSLContextWithOptions(redisSSLOptions *options,
redisSSLContextError *error);
/**
* Free a previously created OpenSSL context.
*/
+2 -1
View File
@@ -1,6 +1,7 @@
prefix=@CMAKE_INSTALL_PREFIX@
install_libdir=@CMAKE_INSTALL_LIBDIR@
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
libdir=${exec_prefix}/${install_libdir}
includedir=${prefix}/include
pkgincludedir=${includedir}/hiredis
+78 -18
View File
@@ -50,6 +50,8 @@
/* Defined in hiredis.c */
void __redisSetError(redisContext *c, int type, const char *str);
int redisContextUpdateCommandTimeout(redisContext *c, const struct timeval *timeout);
void redisNetClose(redisContext *c) {
if (c && c->fd != REDIS_INVALID_FD) {
close(c->fd);
@@ -68,7 +70,7 @@ ssize_t redisNetRead(redisContext *c, char *buf, size_t bufcap) {
__redisSetError(c, REDIS_ERR_TIMEOUT, "recv timeout");
return -1;
} else {
__redisSetError(c, REDIS_ERR_IO, NULL);
__redisSetError(c, REDIS_ERR_IO, strerror(errno));
return -1;
}
} else if (nread == 0) {
@@ -80,15 +82,19 @@ ssize_t redisNetRead(redisContext *c, char *buf, size_t bufcap) {
}
ssize_t redisNetWrite(redisContext *c) {
ssize_t nwritten = send(c->fd, c->obuf, hi_sdslen(c->obuf), 0);
ssize_t nwritten;
nwritten = send(c->fd, c->obuf, hi_sdslen(c->obuf), 0);
if (nwritten < 0) {
if ((errno == EWOULDBLOCK && !(c->flags & REDIS_BLOCK)) || (errno == EINTR)) {
/* Try again later */
/* Try again */
return 0;
} else {
__redisSetError(c, REDIS_ERR_IO, NULL);
__redisSetError(c, REDIS_ERR_IO, strerror(errno));
return -1;
}
}
return nwritten;
}
@@ -166,6 +172,7 @@ int redisKeepAlive(redisContext *c, int interval) {
int val = 1;
redisFD fd = c->fd;
#ifndef _WIN32
if (setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &val, sizeof(val)) == -1){
__redisSetError(c,REDIS_ERR_OTHER,strerror(errno));
return REDIS_ERR;
@@ -199,7 +206,15 @@ int redisKeepAlive(redisContext *c, int interval) {
}
#endif
#endif
#else
int res;
res = win32_redisKeepAlive(fd, interval * 1000);
if (res != 0) {
__redisSetError(c, REDIS_ERR_OTHER, strerror(res));
return REDIS_ERR;
}
#endif
return REDIS_OK;
}
@@ -213,6 +228,23 @@ int redisSetTcpNoDelay(redisContext *c) {
return REDIS_OK;
}
int redisContextSetTcpUserTimeout(redisContext *c, unsigned int timeout) {
int res;
#ifdef TCP_USER_TIMEOUT
res = setsockopt(c->fd, IPPROTO_TCP, TCP_USER_TIMEOUT, &timeout, sizeof(timeout));
#else
res = -1;
errno = ENOTSUP;
(void)timeout;
#endif
if (res == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"setsockopt(TCP_USER_TIMEOUT)");
redisNetClose(c);
return REDIS_ERR;
}
return REDIS_OK;
}
#define __MAX_MSEC (((LONG_MAX) - 999) / 1000)
static int redisContextTimeoutMsec(redisContext *c, long *result)
@@ -223,6 +255,7 @@ static int redisContextTimeoutMsec(redisContext *c, long *result)
/* Only use timeout when not NULL. */
if (timeout != NULL) {
if (timeout->tv_usec > 1000000 || timeout->tv_sec > __MAX_MSEC) {
__redisSetError(c, REDIS_ERR_IO, "Invalid timeout specified");
*result = msec;
return REDIS_ERR;
}
@@ -277,12 +310,28 @@ int redisCheckConnectDone(redisContext *c, int *completed) {
*completed = 1;
return REDIS_OK;
}
switch (errno) {
int error = errno;
if (error == EINPROGRESS) {
/* must check error to see if connect failed. Get the socket error */
int fail, so_error;
socklen_t optlen = sizeof(so_error);
fail = getsockopt(c->fd, SOL_SOCKET, SO_ERROR, &so_error, &optlen);
if (fail == 0) {
if (so_error == 0) {
/* Socket is connected! */
*completed = 1;
return REDIS_OK;
}
/* connection error; */
errno = so_error;
error = so_error;
}
}
switch (error) {
case EISCONN:
*completed = 1;
return REDIS_OK;
case EALREADY:
case EINPROGRESS:
case EWOULDBLOCK:
*completed = 0;
return REDIS_OK;
@@ -317,6 +366,10 @@ int redisContextSetTimeout(redisContext *c, const struct timeval tv) {
const void *to_ptr = &tv;
size_t to_sz = sizeof(tv);
if (redisContextUpdateCommandTimeout(c, &tv) != REDIS_OK) {
__redisSetError(c, REDIS_ERR_OOM, "Out of memory");
return REDIS_ERR;
}
if (setsockopt(c->fd,SOL_SOCKET,SO_RCVTIMEO,to_ptr,to_sz) == -1) {
__redisSetErrorFromErrno(c,REDIS_ERR_IO,"setsockopt(SO_RCVTIMEO)");
return REDIS_ERR;
@@ -400,7 +453,6 @@ static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
}
if (redisContextTimeoutMsec(c, &timeout_msec) != REDIS_OK) {
__redisSetError(c, REDIS_ERR_IO, "Invalid timeout specified");
goto error;
}
@@ -417,17 +469,25 @@ static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
hints.ai_family = AF_INET;
hints.ai_socktype = SOCK_STREAM;
/* Try with IPv6 if no IPv4 address was found. We do it in this order since
* in a Redis client you can't afford to test if you have IPv6 connectivity
* as this would add latency to every connect. Otherwise a more sensible
* route could be: Use IPv6 if both addresses are available and there is IPv6
* connectivity. */
if ((rv = getaddrinfo(c->tcp.host,_port,&hints,&servinfo)) != 0) {
hints.ai_family = AF_INET6;
if ((rv = getaddrinfo(addr,_port,&hints,&servinfo)) != 0) {
__redisSetError(c,REDIS_ERR_OTHER,gai_strerror(rv));
return REDIS_ERR;
}
/* DNS lookup. To use dual stack, set both flags to prefer both IPv4 and
* IPv6. By default, for historical reasons, we try IPv4 first and then we
* try IPv6 only if no IPv4 address was found. */
if (c->flags & REDIS_PREFER_IPV6 && c->flags & REDIS_PREFER_IPV4)
hints.ai_family = AF_UNSPEC;
else if (c->flags & REDIS_PREFER_IPV6)
hints.ai_family = AF_INET6;
else
hints.ai_family = AF_INET;
rv = getaddrinfo(c->tcp.host, _port, &hints, &servinfo);
if (rv != 0 && hints.ai_family != AF_UNSPEC) {
/* Try again with the other IP version. */
hints.ai_family = (hints.ai_family == AF_INET) ? AF_INET6 : AF_INET;
rv = getaddrinfo(c->tcp.host, _port, &hints, &servinfo);
}
if (rv != 0) {
__redisSetError(c, REDIS_ERR_OTHER, gai_strerror(rv));
return REDIS_ERR;
}
for (p = servinfo; p != NULL; p = p->ai_next) {
addrretry:
+1
View File
@@ -52,5 +52,6 @@ int redisKeepAlive(redisContext *c, int interval);
int redisCheckConnectDone(redisContext *c, int *completed);
int redisSetTcpNoDelay(redisContext *c);
int redisContextSetTcpUserTimeout(redisContext *c, unsigned int timeout);
#endif
+8 -5
View File
@@ -303,11 +303,14 @@ static int processLineItem(redisReader *r) {
d = INFINITY; /* Positive infinite. */
} else if (len == 4 && strcasecmp(buf,"-inf") == 0) {
d = -INFINITY; /* Negative infinite. */
} else if ((len == 3 && strcasecmp(buf,"nan") == 0) ||
(len == 4 && strcasecmp(buf, "-nan") == 0)) {
d = NAN; /* nan. */
} else {
d = strtod((char*)buf,&eptr);
/* RESP3 only allows "inf", "-inf", and finite values, while
* strtod() allows other variations on infinity, NaN,
* etc. We explicity handle our two allowed infinite cases
* strtod() allows other variations on infinity,
* etc. We explicity handle our two allowed infinite cases and NaN
* above, so strtod() should only result in finite values. */
if (buf[0] == '\0' || eptr != &buf[len] || !isfinite(d)) {
__redisReaderSetError(r,REDIS_ERR_PROTOCOL,
@@ -374,7 +377,7 @@ static int processLineItem(redisReader *r) {
if (r->fn && r->fn->createString)
obj = r->fn->createString(cur,p,len);
else
obj = (void*)(size_t)(cur->type);
obj = (void*)(uintptr_t)(cur->type);
}
if (obj == NULL) {
@@ -439,7 +442,7 @@ static int processBulkItem(redisReader *r) {
if (r->fn && r->fn->createString)
obj = r->fn->createString(cur,s+2,len);
else
obj = (void*)(long)cur->type;
obj = (void*)(uintptr_t)cur->type;
success = 1;
}
}
@@ -536,7 +539,7 @@ static int processAggregateItem(redisReader *r) {
if (r->fn && r->fn->createArray)
obj = r->fn->createArray(cur,elements);
else
obj = (void*)(long)cur->type;
obj = (void*)(uintptr_t)cur->type;
if (obj == NULL) {
__redisReaderSetErrorOOM(r);
+16 -13
View File
@@ -90,6 +90,7 @@ hisds hi_sdsnewlen(const void *init, size_t initlen) {
int hdrlen = hi_sdsHdrSize(type);
unsigned char *fp; /* flags pointer. */
if (hdrlen+initlen+1 <= initlen) return NULL; /* Catch size_t overflow */
sh = hi_s_malloc(hdrlen+initlen+1);
if (sh == NULL) return NULL;
if (!init)
@@ -174,7 +175,7 @@ void hi_sdsfree(hisds s) {
* the output will be "6" as the string was modified but the logical length
* remains 6 bytes. */
void hi_sdsupdatelen(hisds s) {
int reallen = strlen(s);
size_t reallen = strlen(s);
hi_sdssetlen(s, reallen);
}
@@ -196,7 +197,7 @@ void hi_sdsclear(hisds s) {
hisds hi_sdsMakeRoomFor(hisds s, size_t addlen) {
void *sh, *newsh;
size_t avail = hi_sdsavail(s);
size_t len, newlen;
size_t len, newlen, reqlen;
char type, oldtype = s[-1] & HI_SDS_TYPE_MASK;
int hdrlen;
@@ -205,7 +206,8 @@ hisds hi_sdsMakeRoomFor(hisds s, size_t addlen) {
len = hi_sdslen(s);
sh = (char*)s-hi_sdsHdrSize(oldtype);
newlen = (len+addlen);
reqlen = newlen = (len+addlen);
if (newlen <= len) return NULL; /* Catch size_t overflow */
if (newlen < HI_SDS_MAX_PREALLOC)
newlen *= 2;
else
@@ -219,6 +221,7 @@ hisds hi_sdsMakeRoomFor(hisds s, size_t addlen) {
if (type == HI_SDS_TYPE_5) type = HI_SDS_TYPE_8;
hdrlen = hi_sdsHdrSize(type);
if (hdrlen+newlen+1 <= reqlen) return NULL; /* Catch size_t overflow */
if (oldtype==type) {
newsh = hi_s_realloc(sh, hdrlen+newlen+1);
if (newsh == NULL) return NULL;
@@ -580,7 +583,7 @@ hisds hi_sdscatprintf(hisds s, const char *fmt, ...) {
*/
hisds hi_sdscatfmt(hisds s, char const *fmt, ...) {
const char *f = fmt;
int i;
long i;
va_list ap;
va_start(ap,fmt);
@@ -753,16 +756,16 @@ int hi_sdsrange(hisds s, ssize_t start, ssize_t end) {
return 0;
}
/* Apply tolower() to every character of the hisds string 's'. */
/* Apply tolower() to every character of the sds string 's'. */
void hi_sdstolower(hisds s) {
int len = hi_sdslen(s), j;
size_t len = hi_sdslen(s), j;
for (j = 0; j < len; j++) s[j] = tolower(s[j]);
}
/* Apply toupper() to every character of the hisds string 's'. */
/* Apply toupper() to every character of the sds string 's'. */
void hi_sdstoupper(hisds s) {
int len = hi_sdslen(s), j;
size_t len = hi_sdslen(s), j;
for (j = 0; j < len; j++) s[j] = toupper(s[j]);
}
@@ -945,7 +948,7 @@ hisds *hi_sdssplitargs(const char *line, int *argc) {
*argc = 0;
while(1) {
/* skip blanks */
while(*p && isspace(*p)) p++;
while(*p && isspace((int) *p)) p++;
if (*p) {
/* get a token */
int inq=0; /* set to 1 if we are in "quotes" */
@@ -956,8 +959,8 @@ hisds *hi_sdssplitargs(const char *line, int *argc) {
while(!done) {
if (inq) {
if (*p == '\\' && *(p+1) == 'x' &&
isxdigit(*(p+2)) &&
isxdigit(*(p+3)))
isxdigit((int) *(p+2)) &&
isxdigit((int) *(p+3)))
{
unsigned char byte;
@@ -981,7 +984,7 @@ hisds *hi_sdssplitargs(const char *line, int *argc) {
} else if (*p == '"') {
/* closing quote must be followed by a space or
* nothing at all. */
if (*(p+1) && !isspace(*(p+1))) goto err;
if (*(p+1) && !isspace((int) *(p+1))) goto err;
done=1;
} else if (!*p) {
/* unterminated quotes */
@@ -996,7 +999,7 @@ hisds *hi_sdssplitargs(const char *line, int *argc) {
} else if (*p == '\'') {
/* closing quote must be followed by a space or
* nothing at all. */
if (*(p+1) && !isspace(*(p+1))) goto err;
if (*(p+1) && !isspace((int) *(p+1))) goto err;
done=1;
} else if (!*p) {
/* unterminated quotes */
+3 -1
View File
@@ -35,9 +35,11 @@
#define HI_SDS_MAX_PREALLOC (1024*1024)
#ifdef _MSC_VER
#define __attribute__(x)
typedef long long ssize_t;
#define SSIZE_MAX (LLONG_MAX >> 1)
#ifndef __clang__
#define __attribute__(x)
#endif
#endif
#include <sys/types.h>
+34 -2
View File
@@ -180,10 +180,17 @@ int win32_connect(SOCKET sockfd, const struct sockaddr *addr, socklen_t addrlen)
/* For Winsock connect(), the WSAEWOULDBLOCK error means the same thing as
* EINPROGRESS for POSIX connect(), so we do that translation to keep POSIX
* logic consistent. */
if (errno == EWOULDBLOCK) {
* logic consistent.
* Additionally, WSAALREADY is can be reported as WSAEINVAL to and this is
* translated to EIO. Convert appropriately
*/
int err = errno;
if (err == EWOULDBLOCK) {
errno = EINPROGRESS;
}
else if (err == EIO) {
errno = EALREADY;
}
return ret != SOCKET_ERROR ? ret : -1;
}
@@ -205,6 +212,14 @@ int win32_getsockopt(SOCKET sockfd, int level, int optname, void *optval, sockle
} else {
ret = getsockopt(sockfd, level, optname, (char*)optval, optlen);
}
if (ret != SOCKET_ERROR && level == SOL_SOCKET && optname == SO_ERROR) {
/* translate SO_ERROR codes, if non-zero */
int err = *(int*)optval;
if (err != 0) {
err = _wsaErrorToErrno(err);
*(int*)optval = err;
}
}
_updateErrno(ret != SOCKET_ERROR);
return ret != SOCKET_ERROR ? ret : -1;
}
@@ -245,4 +260,21 @@ int win32_poll(struct pollfd *fds, nfds_t nfds, int timeout) {
_updateErrno(ret != SOCKET_ERROR);
return ret != SOCKET_ERROR ? ret : -1;
}
int win32_redisKeepAlive(SOCKET sockfd, int interval_ms) {
struct tcp_keepalive cfg;
DWORD bytes_in;
int res;
cfg.onoff = 1;
cfg.keepaliveinterval = interval_ms;
cfg.keepalivetime = interval_ms;
res = WSAIoctl(sockfd, SIO_KEEPALIVE_VALS, &cfg,
sizeof(struct tcp_keepalive), NULL, 0,
&bytes_in, NULL, NULL);
return res == 0 ? 0 : _wsaErrorToErrno(res);
}
#endif /* _WIN32 */
+3
View File
@@ -50,6 +50,7 @@
#include <ws2tcpip.h>
#include <stddef.h>
#include <errno.h>
#include <mstcpip.h>
#ifdef _MSC_VER
typedef long long ssize_t;
@@ -71,6 +72,8 @@ ssize_t win32_send(SOCKET sockfd, const void *buf, size_t len, int flags);
typedef ULONG nfds_t;
int win32_poll(struct pollfd *fds, nfds_t nfds, int timeout);
int win32_redisKeepAlive(SOCKET sockfd, int interval_ms);
#ifndef REDIS_SOCKCOMPAT_IMPLEMENTATION
#define getaddrinfo(node, service, hints, res) win32_getaddrinfo(node, service, hints, res)
#undef gai_strerror
+52 -4
View File
@@ -32,6 +32,7 @@
#include "hiredis.h"
#include "async.h"
#include "net.h"
#include <assert.h>
#include <errno.h>
@@ -39,6 +40,14 @@
#ifdef _WIN32
#include <windows.h>
#include <wincrypt.h>
#ifdef OPENSSL_IS_BORINGSSL
#undef X509_NAME
#undef X509_EXTENSIONS
#undef PKCS7_ISSUER_AND_SERIAL
#undef PKCS7_SIGNER_INFO
#undef OCSP_REQUEST
#undef OCSP_RESPONSE
#endif
#else
#include <pthread.h>
#endif
@@ -50,6 +59,8 @@
#include "async_private.h"
#include "hiredis_ssl.h"
#define OPENSSL_1_1_0 0x10100000L
void __redisSetError(redisContext *c, int type, const char *str);
struct redisSSLContext {
@@ -91,7 +102,7 @@ redisContextFuncs redisContextSSLFuncs;
* Note that this is only required for OpenSSL < 1.1.0.
*/
#if OPENSSL_VERSION_NUMBER < 0x10100000L
#if OPENSSL_VERSION_NUMBER < OPENSSL_1_1_0
#define HIREDIS_USE_CRYPTO_LOCKS
#endif
@@ -219,6 +230,25 @@ redisSSLContext *redisCreateSSLContext(const char *cacert_filename, const char *
const char *cert_filename, const char *private_key_filename,
const char *server_name, redisSSLContextError *error)
{
redisSSLOptions options = {
.cacert_filename = cacert_filename,
.capath = capath,
.cert_filename = cert_filename,
.private_key_filename = private_key_filename,
.server_name = server_name,
.verify_mode = REDIS_SSL_VERIFY_PEER,
};
return redisCreateSSLContextWithOptions(&options, error);
}
redisSSLContext *redisCreateSSLContextWithOptions(redisSSLOptions *options, redisSSLContextError *error) {
const char *cacert_filename = options->cacert_filename;
const char *capath = options->capath;
const char *cert_filename = options->cert_filename;
const char *private_key_filename = options->private_key_filename;
const char *server_name = options->server_name;
#ifdef _WIN32
HCERTSTORE win_store = NULL;
PCCERT_CONTEXT win_ctx = NULL;
@@ -228,14 +258,26 @@ redisSSLContext *redisCreateSSLContext(const char *cacert_filename, const char *
if (ctx == NULL)
goto error;
ctx->ssl_ctx = SSL_CTX_new(SSLv23_client_method());
const SSL_METHOD *ssl_method;
#if OPENSSL_VERSION_NUMBER >= OPENSSL_1_1_0
ssl_method = TLS_client_method();
#else
ssl_method = SSLv23_client_method();
#endif
ctx->ssl_ctx = SSL_CTX_new(ssl_method);
if (!ctx->ssl_ctx) {
if (error) *error = REDIS_SSL_CTX_CREATE_FAILED;
goto error;
}
SSL_CTX_set_options(ctx->ssl_ctx, SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3);
SSL_CTX_set_verify(ctx->ssl_ctx, SSL_VERIFY_PEER, NULL);
#if OPENSSL_VERSION_NUMBER >= OPENSSL_1_1_0
SSL_CTX_set_min_proto_version(ctx->ssl_ctx, TLS1_2_VERSION);
#else
SSL_CTX_set_options(ctx->ssl_ctx, SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3 | SSL_OP_NO_TLSv1 | SSL_OP_NO_TLSv1_1);
#endif
SSL_CTX_set_verify(ctx->ssl_ctx, options->verify_mode, NULL);
if ((cert_filename != NULL && private_key_filename == NULL) ||
(private_key_filename != NULL && cert_filename == NULL)) {
@@ -273,6 +315,11 @@ redisSSLContext *redisCreateSSLContext(const char *cacert_filename, const char *
if (error) *error = REDIS_SSL_CTX_CA_CERT_LOAD_FAILED;
goto error;
}
} else {
if (!SSL_CTX_set_default_verify_paths(ctx->ssl_ctx)) {
if (error) *error = REDIS_SSL_CTX_CLIENT_DEFAULT_CERT_FAILED;
goto error;
}
}
if (cert_filename) {
@@ -560,6 +607,7 @@ static void redisSSLAsyncWrite(redisAsyncContext *ac) {
}
redisContextFuncs redisContextSSLFuncs = {
.close = redisNetClose,
.free_privctx = redisSSLFree,
.async_read = redisSSLAsyncRead,
.async_write = redisSSLAsyncWrite,
+412 -18
View File
@@ -15,6 +15,7 @@
#include "hiredis.h"
#include "async.h"
#include "adapters/poll.h"
#ifdef HIREDIS_TEST_SSL
#include "hiredis_ssl.h"
#endif
@@ -34,11 +35,11 @@ enum connection_type {
struct config {
enum connection_type type;
struct timeval connect_timeout;
struct {
const char *host;
int port;
struct timeval timeout;
} tcp;
struct {
@@ -75,6 +76,15 @@ static int tests = 0, fails = 0, skips = 0;
#define test_cond(_c) if(_c) printf("\033[0;32mPASSED\033[0;0m\n"); else {printf("\033[0;31mFAILED\033[0;0m\n"); fails++;}
#define test_skipped() { printf("\033[01;33mSKIPPED\033[0;0m\n"); skips++; }
static void millisleep(int ms)
{
#ifdef _MSC_VER
Sleep(ms);
#else
usleep(ms*1000);
#endif
}
static long long usec(void) {
#ifndef _MSC_VER
struct timeval tv;
@@ -329,10 +339,14 @@ static void test_format_commands(void) {
FLOAT_WIDTH_TEST(float);
FLOAT_WIDTH_TEST(double);
test("Format command with invalid printf format: ");
test("Format command with unhandled printf format (specifier 'p' not supported): ");
len = redisFormatCommand(&cmd,"key:%08p %b",(void*)1234,"foo",(size_t)3);
test_cond(len == -1);
test("Format command with invalid printf format (specifier missing): ");
len = redisFormatCommand(&cmd,"%-");
test_cond(len == -1);
const char *argv[3];
argv[0] = "SET";
argv[1] = "foo\0xxx";
@@ -391,6 +405,25 @@ static void test_append_formatted_commands(struct config config) {
disconnect(c, 0);
}
static void test_tcp_options(struct config cfg) {
redisContext *c;
c = do_connect(cfg);
test("We can enable TCP_KEEPALIVE: ");
test_cond(redisEnableKeepAlive(c) == REDIS_OK);
#ifdef TCP_USER_TIMEOUT
test("We can set TCP_USER_TIMEOUT: ");
test_cond(redisSetTcpUserTimeout(c, 100) == REDIS_OK);
#else
test("Setting TCP_USER_TIMEOUT errors when unsupported: ");
test_cond(redisSetTcpUserTimeout(c, 100) == REDIS_ERR && c->err == REDIS_ERR_IO);
#endif
redisFree(c);
}
static void test_reply_reader(void) {
redisReader *reader;
void *reply, *root;
@@ -568,6 +601,19 @@ static void test_reply_reader(void) {
test_cond(ret == REDIS_ERR && reply == NULL);
redisReaderFree(reader);
test("Don't reset state after protocol error(not segfault): ");
reader = redisReaderCreate();
redisReaderFeed(reader,(char*)"*3\r\n$3\r\nSET\r\n$5\r\nhello\r\n$", 25);
ret = redisReaderGetReply(reader,&reply);
assert(ret == REDIS_OK);
redisReaderFeed(reader,(char*)"3\r\nval\r\n", 8);
ret = redisReaderGetReply(reader,&reply);
test_cond(ret == REDIS_OK &&
((redisReply*)reply)->type == REDIS_REPLY_ARRAY &&
((redisReply*)reply)->elements == 3);
freeReplyObject(reply);
redisReaderFree(reader);
/* Regression test for issue #45 on GitHub. */
test("Don't do empty allocation for empty multi bulk: ");
reader = redisReaderCreate();
@@ -637,12 +683,23 @@ static void test_reply_reader(void) {
freeReplyObject(reply);
redisReaderFree(reader);
test("Set error when RESP3 double is NaN: ");
test("Correctly parses RESP3 double NaN: ");
reader = redisReaderCreate();
redisReaderFeed(reader, ",nan\r\n",6);
ret = redisReaderGetReply(reader,&reply);
test_cond(ret == REDIS_ERR &&
strcasecmp(reader->errstr,"Bad double value") == 0);
test_cond(ret == REDIS_OK &&
((redisReply*)reply)->type == REDIS_REPLY_DOUBLE &&
isnan(((redisReply*)reply)->dval));
freeReplyObject(reply);
redisReaderFree(reader);
test("Correctly parses RESP3 double -Nan: ");
reader = redisReaderCreate();
redisReaderFeed(reader, ",-nan\r\n", 7);
ret = redisReaderGetReply(reader, &reply);
test_cond(ret == REDIS_OK &&
((redisReply*)reply)->type == REDIS_REPLY_DOUBLE &&
isnan(((redisReply*)reply)->dval));
freeReplyObject(reply);
redisReaderFree(reader);
@@ -745,6 +802,20 @@ static void test_reply_reader(void) {
!strcmp(((redisReply*)reply)->str,"3492890328409238509324850943850943825024385"));
freeReplyObject(reply);
redisReaderFree(reader);
test("Can parse RESP3 doubles in an array: ");
reader = redisReaderCreate();
redisReaderFeed(reader, "*1\r\n,3.14159265358979323846\r\n",31);
ret = redisReaderGetReply(reader,&reply);
test_cond(ret == REDIS_OK &&
((redisReply*)reply)->type == REDIS_REPLY_ARRAY &&
((redisReply*)reply)->elements == 1 &&
((redisReply*)reply)->element[0]->type == REDIS_REPLY_DOUBLE &&
fabs(((redisReply*)reply)->element[0]->dval - 3.14159265358979323846) < 0.00000001 &&
((redisReply*)reply)->element[0]->len == 22 &&
strcmp(((redisReply*)reply)->element[0]->str, "3.14159265358979323846") == 0);
freeReplyObject(reply);
redisReaderFree(reader);
}
static void test_free_null(void) {
@@ -819,9 +890,9 @@ static void test_allocator_injection(void) {
#define HIREDIS_BAD_DOMAIN "idontexist-noreally.com"
static void test_blocking_connection_errors(void) {
redisContext *c;
struct addrinfo hints = {.ai_family = AF_INET};
struct addrinfo *ai_tmp = NULL;
redisContext *c;
int rv = getaddrinfo(HIREDIS_BAD_DOMAIN, "6379", &hints, &ai_tmp);
if (rv != 0) {
@@ -835,6 +906,7 @@ static void test_blocking_connection_errors(void) {
strcmp(c->errstr, "Can't resolve: " HIREDIS_BAD_DOMAIN) == 0 ||
strcmp(c->errstr, "Name does not resolve") == 0 ||
strcmp(c->errstr, "nodename nor servname provided, or not known") == 0 ||
strcmp(c->errstr, "node name or service name not known") == 0 ||
strcmp(c->errstr, "No address associated with hostname") == 0 ||
strcmp(c->errstr, "Temporary failure in name resolution") == 0 ||
strcmp(c->errstr, "hostname nor servname provided, or not known") == 0 ||
@@ -847,12 +919,26 @@ static void test_blocking_connection_errors(void) {
}
#ifndef _WIN32
redisOptions opt = {0};
struct timeval tv;
test("Returns error when the port is not open: ");
c = redisConnect((char*)"localhost", 1);
test_cond(c->err == REDIS_ERR_IO &&
strcmp(c->errstr,"Connection refused") == 0);
redisFree(c);
/* Verify we don't regress from the fix in PR #1180 */
test("We don't clobber connection exception with setsockopt error: ");
tv = (struct timeval){.tv_sec = 0, .tv_usec = 500000};
opt.command_timeout = opt.connect_timeout = &tv;
REDIS_OPTIONS_SET_TCP(&opt, "localhost", 10337);
c = redisConnectWithOptions(&opt);
test_cond(c->err == REDIS_ERR_IO &&
strcmp(c->errstr, "Connection refused") == 0);
redisFree(c);
test("Returns error when the unix_sock socket path doesn't accept connections: ");
c = redisConnectUnix((char*)"/tmp/idontexist.sock");
test_cond(c->err == REDIS_ERR_IO); /* Don't care about the message... */
@@ -914,11 +1000,19 @@ static void test_resp3_push_handler(redisContext *c) {
old = redisSetPushCallback(c, push_handler);
test("We can set a custom RESP3 PUSH handler: ");
reply = redisCommand(c, "SET key:0 val:0");
/* We need another command because depending on the version of Redis, the
* notification may be delivered after the command's reply. */
assert(reply != NULL);
freeReplyObject(reply);
reply = redisCommand(c, "PING");
test_cond(reply != NULL && reply->type == REDIS_REPLY_STATUS && pc.str == 1);
freeReplyObject(reply);
test("We properly handle a NIL invalidation payload: ");
reply = redisCommand(c, "FLUSHDB");
assert(reply != NULL);
freeReplyObject(reply);
reply = redisCommand(c, "PING");
test_cond(reply != NULL && reply->type == REDIS_REPLY_STATUS && pc.nil == 1);
freeReplyObject(reply);
@@ -929,6 +1023,12 @@ static void test_resp3_push_handler(redisContext *c) {
assert((reply = redisCommand(c, "GET key:0")) != NULL);
freeReplyObject(reply);
assert((reply = redisCommand(c, "SET key:0 invalid")) != NULL);
/* Depending on Redis version, we may receive either push notification or
* status reply. Both cases are valid. */
if (reply->type == REDIS_REPLY_STATUS) {
freeReplyObject(reply);
reply = redisCommand(c, "PING");
}
test_cond(reply->type == REDIS_REPLY_PUSH);
freeReplyObject(reply);
@@ -1089,6 +1189,13 @@ static void test_blocking_connection(struct config config) {
strcasecmp(reply->element[1]->str,"pong") == 0);
freeReplyObject(reply);
test("Send command by passing argc/argv: ");
const char *argv[3] = {"SET", "foo", "bar"};
size_t argvlen[3] = {3, 3, 3};
reply = redisCommandArgv(c,3,argv,argvlen);
test_cond(reply->type == REDIS_REPLY_STATUS);
freeReplyObject(reply);
/* Make sure passing NULL to redisGetReply is safe */
test("Can pass NULL to redisGetReply: ");
assert(redisAppendCommand(c, "PING") == REDIS_OK);
@@ -1143,7 +1250,13 @@ static void test_blocking_connection_timeouts(struct config config) {
test("Does not return a reply when the command times out: ");
if (detect_debug_sleep(c)) {
redisAppendFormattedCommand(c, sleep_cmd, strlen(sleep_cmd));
// flush connection buffer without waiting for the reply
s = c->funcs->write(c);
assert(s == (ssize_t)hi_sdslen(c->obuf));
hi_sdsfree(c->obuf);
c->obuf = hi_sdsempty();
tv.tv_sec = 0;
tv.tv_usec = 10000;
redisSetTimeout(c, tv);
@@ -1156,6 +1269,9 @@ static void test_blocking_connection_timeouts(struct config config) {
strcmp(c->errstr, "recv timeout") == 0);
#endif
freeReplyObject(reply);
// wait for the DEBUG SLEEP to complete so that Redis server is unblocked for the following tests
millisleep(3000);
} else {
test_skipped();
}
@@ -1226,22 +1342,34 @@ static void test_blocking_io_errors(struct config config) {
static void test_invalid_timeout_errors(struct config config) {
redisContext *c;
test("Set error when an invalid timeout usec value is given to redisConnectWithTimeout: ");
test("Set error when an invalid timeout usec value is used during connect: ");
config.tcp.timeout.tv_sec = 0;
config.tcp.timeout.tv_usec = 10000001;
config.connect_timeout.tv_sec = 0;
config.connect_timeout.tv_usec = 10000001;
c = redisConnectWithTimeout(config.tcp.host, config.tcp.port, config.tcp.timeout);
if (config.type == CONN_TCP || config.type == CONN_SSL) {
c = redisConnectWithTimeout(config.tcp.host, config.tcp.port, config.connect_timeout);
} else if(config.type == CONN_UNIX) {
c = redisConnectUnixWithTimeout(config.unix_sock.path, config.connect_timeout);
} else {
assert(NULL);
}
test_cond(c->err == REDIS_ERR_IO && strcmp(c->errstr, "Invalid timeout specified") == 0);
redisFree(c);
test("Set error when an invalid timeout sec value is given to redisConnectWithTimeout: ");
test("Set error when an invalid timeout sec value is used during connect: ");
config.tcp.timeout.tv_sec = (((LONG_MAX) - 999) / 1000) + 1;
config.tcp.timeout.tv_usec = 0;
config.connect_timeout.tv_sec = (((LONG_MAX) - 999) / 1000) + 1;
config.connect_timeout.tv_usec = 0;
c = redisConnectWithTimeout(config.tcp.host, config.tcp.port, config.tcp.timeout);
if (config.type == CONN_TCP || config.type == CONN_SSL) {
c = redisConnectWithTimeout(config.tcp.host, config.tcp.port, config.connect_timeout);
} else if(config.type == CONN_UNIX) {
c = redisConnectUnixWithTimeout(config.unix_sock.path, config.connect_timeout);
} else {
assert(NULL);
}
test_cond(c->err == REDIS_ERR_IO && strcmp(c->errstr, "Invalid timeout specified") == 0);
redisFree(c);
@@ -1448,6 +1576,9 @@ static void test_throughput(struct config config) {
// }
#ifdef HIREDIS_TEST_ASYNC
#pragma GCC diagnostic ignored "-Woverlength-strings" /* required on gcc 4.8.x due to assert statements */
struct event_base *base;
typedef struct TestState {
@@ -1729,10 +1860,14 @@ void subscribe_channel_a_cb(redisAsyncContext *ac, void *r, void *privdata) {
strcmp(reply->element[2]->str,"Hello!") == 0);
state->checkpoint++;
/* Unsubscribe to channels, including a channel X which we don't subscribe to */
/* Unsubscribe to channels, including channel X & Z which we don't subscribe to */
redisAsyncCommand(ac,unexpected_cb,
(void*)"unsubscribe should not call unexpected_cb()",
"unsubscribe B X A");
"unsubscribe B X A A Z");
/* Unsubscribe to patterns, none which we subscribe to */
redisAsyncCommand(ac,unexpected_cb,
(void*)"punsubscribe should not call unexpected_cb()",
"punsubscribe");
/* Send a regular command after unsubscribing, then disconnect */
state->disconnect = 1;
redisAsyncCommand(ac,integer_cb,state,"LPUSH mylist foo");
@@ -1749,6 +1884,7 @@ void subscribe_channel_a_cb(redisAsyncContext *ac, void *r, void *privdata) {
void subscribe_channel_b_cb(redisAsyncContext *ac, void *r, void *privdata) {
redisReply *reply = r;
TestState *state = privdata;
(void)ac;
assert(reply != NULL && reply->type == REDIS_REPLY_ARRAY &&
reply->elements == 3);
@@ -1767,8 +1903,10 @@ void subscribe_channel_b_cb(redisAsyncContext *ac, void *r, void *privdata) {
/* Test handling of multiple channels
* - subscribe to channel A and B
* - a published message on A triggers an unsubscribe of channel B, X and A
* where channel X is not subscribed to.
* - a published message on A triggers an unsubscribe of channel B, X, A and Z
* where channel X and Z are not subscribed to.
* - the published message also triggers an unsubscribe to patterns. Since no
* pattern is subscribed to the responded pattern element type is NIL.
* - a command sent after unsubscribe triggers a disconnect */
static void test_pubsub_multiple_channels(struct config config) {
test("Subscribe to multiple channels: ");
@@ -1881,6 +2019,250 @@ static void test_monitor(struct config config) {
}
#endif /* HIREDIS_TEST_ASYNC */
/* tests for async api using polling adapter, requires no extra libraries*/
/* enum for the test cases, the callbacks have different logic based on them */
typedef enum astest_no
{
ASTEST_CONNECT=0,
ASTEST_CONN_TIMEOUT,
ASTEST_PINGPONG,
ASTEST_PINGPONG_TIMEOUT,
ASTEST_ISSUE_931,
ASTEST_ISSUE_931_PING
}astest_no;
/* a static context for the async tests */
struct _astest {
redisAsyncContext *ac;
astest_no testno;
int counter;
int connects;
int connect_status;
int disconnects;
int pongs;
int disconnect_status;
int connected;
int err;
char errstr[256];
};
static struct _astest astest;
/* async callbacks */
static void asCleanup(void* data)
{
struct _astest *t = (struct _astest *)data;
t->ac = NULL;
}
static void commandCallback(struct redisAsyncContext *ac, void* _reply, void* _privdata);
static void connectCallback(redisAsyncContext *c, int status) {
struct _astest *t = (struct _astest *)c->data;
assert(t == &astest);
assert(t->connects == 0);
t->err = c->err;
strcpy(t->errstr, c->errstr);
t->connects++;
t->connect_status = status;
t->connected = status == REDIS_OK ? 1 : -1;
if (t->testno == ASTEST_ISSUE_931) {
/* disconnect again */
redisAsyncDisconnect(c);
}
else if (t->testno == ASTEST_ISSUE_931_PING)
{
redisAsyncCommand(c, commandCallback, NULL, "PING");
}
}
static void disconnectCallback(const redisAsyncContext *c, int status) {
assert(c->data == (void*)&astest);
assert(astest.disconnects == 0);
astest.err = c->err;
strcpy(astest.errstr, c->errstr);
astest.disconnects++;
astest.disconnect_status = status;
astest.connected = 0;
}
static void commandCallback(struct redisAsyncContext *ac, void* _reply, void* _privdata)
{
redisReply *reply = (redisReply*)_reply;
struct _astest *t = (struct _astest *)ac->data;
assert(t == &astest);
(void)_privdata;
t->err = ac->err;
strcpy(t->errstr, ac->errstr);
t->counter++;
if (t->testno == ASTEST_PINGPONG ||t->testno == ASTEST_ISSUE_931_PING)
{
assert(reply != NULL && reply->type == REDIS_REPLY_STATUS && strcmp(reply->str, "PONG") == 0);
t->pongs++;
redisAsyncFree(ac);
}
if (t->testno == ASTEST_PINGPONG_TIMEOUT)
{
/* two ping pongs */
assert(reply != NULL && reply->type == REDIS_REPLY_STATUS && strcmp(reply->str, "PONG") == 0);
t->pongs++;
if (t->counter == 1) {
int status = redisAsyncCommand(ac, commandCallback, NULL, "PING");
assert(status == REDIS_OK);
} else {
redisAsyncFree(ac);
}
}
}
static redisAsyncContext *do_aconnect(struct config config, astest_no testno)
{
redisOptions options = {0};
memset(&astest, 0, sizeof(astest));
astest.testno = testno;
astest.connect_status = astest.disconnect_status = -2;
if (config.type == CONN_TCP) {
options.type = REDIS_CONN_TCP;
options.connect_timeout = &config.connect_timeout;
REDIS_OPTIONS_SET_TCP(&options, config.tcp.host, config.tcp.port);
} else if (config.type == CONN_SSL) {
options.type = REDIS_CONN_TCP;
options.connect_timeout = &config.connect_timeout;
REDIS_OPTIONS_SET_TCP(&options, config.ssl.host, config.ssl.port);
} else if (config.type == CONN_UNIX) {
options.type = REDIS_CONN_UNIX;
options.endpoint.unix_socket = config.unix_sock.path;
} else if (config.type == CONN_FD) {
options.type = REDIS_CONN_USERFD;
/* Create a dummy connection just to get an fd to inherit */
redisContext *dummy_ctx = redisConnectUnix(config.unix_sock.path);
if (dummy_ctx) {
redisFD fd = disconnect(dummy_ctx, 1);
printf("Connecting to inherited fd %d\n", (int)fd);
options.endpoint.fd = fd;
}
}
redisAsyncContext *c = redisAsyncConnectWithOptions(&options);
assert(c);
astest.ac = c;
c->data = &astest;
c->dataCleanup = asCleanup;
redisPollAttach(c);
redisAsyncSetConnectCallbackNC(c, connectCallback);
redisAsyncSetDisconnectCallback(c, disconnectCallback);
return c;
}
static void as_printerr(void) {
printf("Async err %d : %s\n", astest.err, astest.errstr);
}
#define ASASSERT(e) do { \
if (!(e)) \
as_printerr(); \
assert(e); \
} while (0);
static void test_async_polling(struct config config) {
int status;
redisAsyncContext *c;
struct config defaultconfig = config;
test("Async connect: ");
c = do_aconnect(config, ASTEST_CONNECT);
assert(c);
while(astest.connected == 0)
redisPollTick(c, 0.1);
assert(astest.connects == 1);
ASASSERT(astest.connect_status == REDIS_OK);
assert(astest.disconnects == 0);
test_cond(astest.connected == 1);
test("Async free after connect: ");
assert(astest.ac != NULL);
redisAsyncFree(c);
assert(astest.disconnects == 1);
assert(astest.ac == NULL);
test_cond(astest.disconnect_status == REDIS_OK);
if (config.type == CONN_TCP || config.type == CONN_SSL) {
/* timeout can only be simulated with network */
test("Async connect timeout: ");
config.tcp.host = "192.168.254.254"; /* blackhole ip */
config.connect_timeout.tv_usec = 100000;
c = do_aconnect(config, ASTEST_CONN_TIMEOUT);
assert(c);
assert(c->err == 0);
while(astest.connected == 0)
redisPollTick(c, 0.1);
assert(astest.connected == -1);
/*
* freeing should not be done, clearing should have happened.
*redisAsyncFree(c);
*/
assert(astest.ac == NULL);
test_cond(astest.connect_status == REDIS_ERR);
config = defaultconfig;
}
/* Test a ping/pong after connection */
test("Async PING/PONG: ");
c = do_aconnect(config, ASTEST_PINGPONG);
while(astest.connected == 0)
redisPollTick(c, 0.1);
status = redisAsyncCommand(c, commandCallback, NULL, "PING");
assert(status == REDIS_OK);
while(astest.ac)
redisPollTick(c, 0.1);
test_cond(astest.pongs == 1);
/* Test a ping/pong after connection that didn't time out.
* see https://github.com/redis/hiredis/issues/945
*/
if (config.type == CONN_TCP || config.type == CONN_SSL) {
test("Async PING/PONG after connect timeout: ");
config.connect_timeout.tv_usec = 10000; /* 10ms */
c = do_aconnect(config, ASTEST_PINGPONG_TIMEOUT);
while(astest.connected == 0)
redisPollTick(c, 0.1);
/* sleep 0.1 s, allowing old timeout to arrive */
millisleep(10);
status = redisAsyncCommand(c, commandCallback, NULL, "PING");
assert(status == REDIS_OK);
while(astest.ac)
redisPollTick(c, 0.1);
test_cond(astest.pongs == 2);
config = defaultconfig;
}
/* Test disconnect from an on_connect callback
* see https://github.com/redis/hiredis/issues/931
*/
test("Disconnect from onConnected callback (Issue #931): ");
c = do_aconnect(config, ASTEST_ISSUE_931);
while(astest.disconnects == 0)
redisPollTick(c, 0.1);
assert(astest.connected == 0);
assert(astest.connects == 1);
test_cond(astest.disconnects == 1);
/* Test ping/pong from an on_connect callback
* see https://github.com/redis/hiredis/issues/931
*/
test("Ping/Pong from onConnected callback (Issue #931): ");
c = do_aconnect(config, ASTEST_ISSUE_931_PING);
/* connect callback issues ping, reponse callback destroys context */
while(astest.ac)
redisPollTick(c, 0.1);
assert(astest.connected == 0);
assert(astest.connects == 1);
assert(astest.disconnects == 1);
test_cond(astest.pongs == 1);
}
/* End of Async polling_adapter driven tests */
int main(int argc, char **argv) {
struct config cfg = {
.tcp = {
@@ -1963,6 +2345,7 @@ int main(int argc, char **argv) {
test_blocking_io_errors(cfg);
test_invalid_timeout_errors(cfg);
test_append_formatted_commands(cfg);
test_tcp_options(cfg);
if (throughput) test_throughput(cfg);
printf("\nTesting against Unix socket connection (%s): ", cfg.unix_sock.path);
@@ -1972,6 +2355,7 @@ int main(int argc, char **argv) {
test_blocking_connection(cfg);
test_blocking_connection_timeouts(cfg);
test_blocking_io_errors(cfg);
test_invalid_timeout_errors(cfg);
if (throughput) test_throughput(cfg);
} else {
test_skipped();
@@ -2000,6 +2384,7 @@ int main(int argc, char **argv) {
#endif
#ifdef HIREDIS_TEST_ASYNC
cfg.type = CONN_TCP;
printf("\nTesting asynchronous API against TCP connection (%s:%d):\n", cfg.tcp.host, cfg.tcp.port);
cfg.type = CONN_TCP;
@@ -2017,6 +2402,15 @@ int main(int argc, char **argv) {
}
#endif /* HIREDIS_TEST_ASYNC */
cfg.type = CONN_TCP;
printf("\nTesting asynchronous API using polling_adapter TCP (%s:%d):\n", cfg.tcp.host, cfg.tcp.port);
test_async_polling(cfg);
if (test_unix_socket) {
cfg.type = CONN_UNIX;
printf("\nTesting asynchronous API using polling_adapter UNIX (%s):\n", cfg.unix_sock.path);
test_async_polling(cfg);
}
if (test_inherit_fd) {
printf("\nTesting against inherited fd (%s): ", cfg.unix_sock.path);
if (test_unix_socket) {
+43 -9
View File
@@ -4,9 +4,17 @@ REDIS_SERVER=${REDIS_SERVER:-redis-server}
REDIS_PORT=${REDIS_PORT:-56379}
REDIS_SSL_PORT=${REDIS_SSL_PORT:-56443}
TEST_SSL=${TEST_SSL:-0}
SKIPS_AS_FAILS=${SKIPS_AS_FAILS-:0}
SKIPS_AS_FAILS=${SKIPS_AS_FAILS:-0}
ENABLE_DEBUG_CMD=
SSL_TEST_ARGS=
SKIPS_ARG=
SKIPS_ARG=${SKIPS_ARG:-}
REDIS_DOCKER=${REDIS_DOCKER:-}
# We need to enable the DEBUG command for redis-server >= 7.0.0
REDIS_MAJOR_VERSION="$(redis-server --version|awk -F'[^0-9]+' '{ print $2 }')"
if [ "$REDIS_MAJOR_VERSION" -gt "6" ]; then
ENABLE_DEBUG_CMD="enable-debug-command local"
fi
tmpdir=$(mktemp -d)
PID_FILE=${tmpdir}/hiredis-test-redis.pid
@@ -43,20 +51,34 @@ if [ "$TEST_SSL" = "1" ]; then
fi
cleanup() {
set +e
kill $(cat ${PID_FILE})
if [ -n "${REDIS_DOCKER}" ] ; then
docker kill redis-test-server
else
set +e
kill $(cat ${PID_FILE})
fi
rm -rf ${tmpdir}
}
trap cleanup INT TERM EXIT
# base config
cat > ${tmpdir}/redis.conf <<EOF
daemonize yes
pidfile ${PID_FILE}
port ${REDIS_PORT}
bind 127.0.0.1
unixsocket ${SOCK_FILE}
unixsocketperm 777
EOF
# if not running in docker add these:
if [ ! -n "${REDIS_DOCKER}" ]; then
cat >> ${tmpdir}/redis.conf <<EOF
daemonize yes
${ENABLE_DEBUG_CMD}
bind 127.0.0.1
EOF
fi
# if doing ssl, add these
if [ "$TEST_SSL" = "1" ]; then
cat >> ${tmpdir}/redis.conf <<EOF
tls-port ${REDIS_SSL_PORT}
@@ -66,13 +88,25 @@ tls-key-file ${SSL_KEY}
EOF
fi
echo ${tmpdir}
cat ${tmpdir}/redis.conf
${REDIS_SERVER} ${tmpdir}/redis.conf
if [ -n "${REDIS_DOCKER}" ] ; then
chmod a+wx ${tmpdir}
chmod a+r ${tmpdir}/*
docker run -d --rm --name redis-test-server \
-p ${REDIS_PORT}:${REDIS_PORT} \
-p ${REDIS_SSL_PORT}:${REDIS_SSL_PORT} \
-v ${tmpdir}:${tmpdir} \
${REDIS_DOCKER} \
redis-server ${tmpdir}/redis.conf
else
${REDIS_SERVER} ${tmpdir}/redis.conf
fi
# Wait until we detect the unix socket
echo waiting for server
while [ ! -S "${SOCK_FILE}" ]; do sleep 1; done
# Treat skips as failures if directed
[ "$SKIPS_AS_FAILS" = 1 ] && SKIPS_ARG="--skips-as-fails"
[ "$SKIPS_AS_FAILS" = 1 ] && SKIPS_ARG="${SKIPS_ARG} --skips-as-fails"
${TEST_PREFIX:-} ./hiredis-test -h 127.0.0.1 -p ${REDIS_PORT} -s ${SOCK_FILE} ${SSL_TEST_ARGS} ${SKIPS_ARG}
+1 -1
View File
@@ -108,7 +108,7 @@ to search and re-edit already inserted lines of text.
The followings are the history API calls:
int linenoiseHistoryAdd(const char *line);
int linenoiseHistoryAdd(const char *line, int is_sensitive);
int linenoiseHistorySetMaxLen(int len);
int linenoiseHistorySave(const char *filename);
int linenoiseHistoryLoad(const char *filename);
+238 -11
View File
@@ -117,6 +117,7 @@
#include <sys/types.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <assert.h>
#include "linenoise.h"
#define LINENOISE_DEFAULT_HISTORY_MAX_LEN 100
@@ -134,6 +135,18 @@ static int atexit_registered = 0; /* Register atexit just 1 time. */
static int history_max_len = LINENOISE_DEFAULT_HISTORY_MAX_LEN;
static int history_len = 0;
static char **history = NULL;
static int *history_sensitive = NULL; /* An array records whether each line in
* history is sensitive. */
static int reverse_search_mode_enabled = 0;
static int reverse_search_direction = 0; /* 1 means forward, -1 means backward. */
static int cycle_to_next_search = 0; /* indicates whether to continue the search with CTRL+S or CTRL+R. */
static char search_result[LINENOISE_MAX_LINE];
static char search_result_friendly[LINENOISE_MAX_LINE];
static int search_result_history_index = 0;
static int search_result_start_offset = 0;
static int ignore_once_hint = 0; /* Flag to ignore hint once, preventing it from interfering
* with search results right after exiting search mode. */
/* The linenoiseState structure represents the state during line editing.
* We pass this state to functions implementing specific editing
@@ -143,6 +156,7 @@ struct linenoiseState {
int ofd; /* Terminal stdout file descriptor. */
char *buf; /* Edited line buffer. */
size_t buflen; /* Edited line buffer size. */
const char *origin_prompt; /* Original prompt, used to restore when exiting search mode. */
const char *prompt; /* Prompt to display. */
size_t plen; /* Prompt length. */
size_t pos; /* Current cursor position. */
@@ -153,6 +167,13 @@ struct linenoiseState {
int history_index; /* The history index we are currently editing. */
};
typedef struct {
int len; /* Length of the result string. */
char *result; /* Search result string. */
int search_term_index; /* Position of the search term in the history record. */
int search_term_len; /* Length of the search term. */
} linenoiseHistorySearchResult;
enum KEY_ACTION{
KEY_NULL = 0, /* NULL */
CTRL_A = 1, /* Ctrl+a */
@@ -161,6 +182,7 @@ enum KEY_ACTION{
CTRL_D = 4, /* Ctrl-d */
CTRL_E = 5, /* Ctrl-e */
CTRL_F = 6, /* Ctrl-f */
CTRL_G = 7, /* Ctrl-g */
CTRL_H = 8, /* Ctrl-h */
TAB = 9, /* Tab */
NL = 10, /* Enter typed before raw mode was enabled */
@@ -169,6 +191,8 @@ enum KEY_ACTION{
ENTER = 13, /* Enter */
CTRL_N = 14, /* Ctrl-n */
CTRL_P = 16, /* Ctrl-p */
CTRL_R = 18, /* Ctrl-r */
CTRL_S = 19, /* Ctrl-s */
CTRL_T = 20, /* Ctrl-t */
CTRL_U = 21, /* Ctrl+u */
CTRL_W = 23, /* Ctrl+w */
@@ -177,8 +201,14 @@ enum KEY_ACTION{
};
static void linenoiseAtExit(void);
int linenoiseHistoryAdd(const char *line);
int linenoiseHistoryAdd(const char *line, int is_sensitive);
static void refreshLine(struct linenoiseState *l);
static void refreshSearchResult(struct linenoiseState *ls);
static inline void resetSearchResult(void) {
memset(search_result, 0, sizeof(search_result));
memset(search_result_friendly, 0, sizeof(search_result_friendly));
}
/* Debugging macro. */
#if 0
@@ -219,6 +249,41 @@ void linenoiseSetMultiLine(int ml) {
mlmode = ml;
}
#define REVERSE_SEARCH_PROMPT(direction) ((direction) == -1 ? "(reverse-i-search): " : "(i-search): ")
/* Enables the reverse search mode and refreshes the prompt. */
static void enableReverseSearchMode(struct linenoiseState *l) {
assert(reverse_search_mode_enabled != 1);
reverse_search_mode_enabled = 1;
l->origin_prompt = l->prompt;
l->prompt = REVERSE_SEARCH_PROMPT(reverse_search_direction);
refreshLine(l);
}
/* This function disables the reverse search mode and returns the terminal to its original state.
* If the 'discard' parameter is true, it discards the user's input search keyword and search result.
* Otherwise, it copies the search result into 'buf', If there is no search result, it copies the
* input search keyword instead. */
static void disableReverseSearchMode(struct linenoiseState *l, char *buf, size_t buflen, int discard) {
if (discard) {
buf[0] = '\0';
l->pos = l->len = 0;
} else {
ignore_once_hint = 1;
if (strlen(search_result)) {
strncpy(buf, search_result, buflen);
buf[buflen-1] = '\0';
l->pos = l->len = strlen(buf);
}
}
/* Reset the state to non-search state. */
reverse_search_mode_enabled = 0;
l->prompt = l->origin_prompt;
resetSearchResult();
refreshLine(l);
}
/* Return true if the terminal name is in the list of terminals we know are
* not able to understand basic escape sequences. */
static int isUnsupportedTerm(void) {
@@ -231,8 +296,12 @@ static int isUnsupportedTerm(void) {
return 0;
}
/* Raw mode: 1960 magic shit. */
/* Raw mode: 1960's magic. */
static int enableRawMode(int fd) {
if (getenv("FAKETTY_WITH_PROMPT") != NULL) {
return 0;
}
struct termios raw;
if (!isatty(STDIN_FILENO)) goto fatal;
@@ -301,6 +370,9 @@ static int getCursorPosition(int ifd, int ofd) {
/* Try to get the number of columns in the current terminal, or assume 80
* if it fails. */
static int getColumns(int ifd, int ofd) {
if (getenv("FAKETTY_WITH_PROMPT") != NULL) {
goto failed;
}
struct winsize ws;
if (ioctl(1, TIOCGWINSZ, &ws) == -1 || ws.ws_col == 0) {
@@ -492,6 +564,13 @@ static void abFree(struct abuf *ab) {
* to the right of the prompt. */
void refreshShowHints(struct abuf *ab, struct linenoiseState *l, int plen) {
char seq[64];
/* Show hits when not in reverse search mode and not instructed to ignore once. */
if (reverse_search_mode_enabled || ignore_once_hint) {
ignore_once_hint = 0;
return;
}
if (hintsCallback && plen+l->len < l->cols) {
int color = -1, bold = 0;
char *hint = hintsCallback(l->buf,&color,&bold);
@@ -604,7 +683,12 @@ static void refreshMultiLine(struct linenoiseState *l) {
unsigned int i;
for (i = 0; i < l->len; i++) abAppend(&ab,"*",1);
} else {
abAppend(&ab,l->buf,l->len);
refreshSearchResult(l);
if (strlen(search_result) > 0) {
abAppend(&ab, search_result_friendly, strlen(search_result_friendly));
} else {
abAppend(&ab,l->buf,l->len);
}
}
/* Show hits if any. */
@@ -637,6 +721,9 @@ static void refreshMultiLine(struct linenoiseState *l) {
/* Set column. */
col = (plen+(int)l->pos) % (int)l->cols;
if (strlen(search_result) > 0) {
col += search_result_start_offset;
}
lndebug("set col %d", 1+col);
if (col)
snprintf(seq,64,"\r\x1b[%dC", col);
@@ -818,7 +905,7 @@ static int linenoiseEdit(int stdin_fd, int stdout_fd, char *buf, size_t buflen,
/* The latest history entry is always our current buffer, that
* initially is just an empty string. */
linenoiseHistoryAdd("");
linenoiseHistoryAdd("", 0);
if (write(l.ofd,prompt,l.plen) == -1) return -1;
while(1) {
@@ -832,7 +919,7 @@ static int linenoiseEdit(int stdin_fd, int stdout_fd, char *buf, size_t buflen,
/* Only autocomplete when the callback is set. It returns < 0 when
* there was an error reading from fd. Otherwise it will return the
* character that should be handled next. */
if (c == 9 && completionCallback != NULL) {
if (c == TAB && completionCallback != NULL && !reverse_search_mode_enabled) {
c = completeLine(&l);
/* Return on errors */
if (c < 0) return l.len;
@@ -843,6 +930,9 @@ static int linenoiseEdit(int stdin_fd, int stdout_fd, char *buf, size_t buflen,
switch(c) {
case NL: /* enter, typed before raw mode was enabled */
break;
case TAB:
if (reverse_search_mode_enabled) disableReverseSearchMode(&l, buf, buflen, 0);
break;
case ENTER: /* enter */
history_len--;
free(history[history_len]);
@@ -855,8 +945,14 @@ static int linenoiseEdit(int stdin_fd, int stdout_fd, char *buf, size_t buflen,
refreshLine(&l);
hintsCallback = hc;
}
if (reverse_search_mode_enabled) disableReverseSearchMode(&l, buf, buflen, 0);
return (int)l.len;
case CTRL_C: /* ctrl-c */
if (reverse_search_mode_enabled) {
disableReverseSearchMode(&l, buf, buflen, 1);
break;
}
errno = EAGAIN;
return -1;
case BACKSPACE: /* backspace */
@@ -891,6 +987,23 @@ static int linenoiseEdit(int stdin_fd, int stdout_fd, char *buf, size_t buflen,
case CTRL_P: /* ctrl-p */
linenoiseEditHistoryNext(&l, LINENOISE_HISTORY_PREV);
break;
case CTRL_R:
case CTRL_S:
reverse_search_direction = c == CTRL_R ? -1 : 1;
if (reverse_search_mode_enabled) {
/* cycle search results */
cycle_to_next_search = 1;
l.prompt = REVERSE_SEARCH_PROMPT(reverse_search_direction);
refreshLine(&l);
break;
}
buf[0] = '\0';
l.pos = l.len = 0;
enableReverseSearchMode(&l);
break;
case CTRL_G:
if (reverse_search_mode_enabled) disableReverseSearchMode(&l, buf, buflen, 1);
break;
case CTRL_N: /* ctrl-n */
linenoiseEditHistoryNext(&l, LINENOISE_HISTORY_NEXT);
break;
@@ -901,6 +1014,11 @@ static int linenoiseEdit(int stdin_fd, int stdout_fd, char *buf, size_t buflen,
if (read(l.ifd,seq,1) == -1) break;
if (read(l.ifd,seq+1,1) == -1) break;
if (reverse_search_mode_enabled) {
disableReverseSearchMode(&l, buf, buflen, 1);
break;
}
/* ESC [ sequences. */
if (seq[0] == '[') {
if (seq[1] >= '0' && seq[1] <= '9') {
@@ -1067,14 +1185,14 @@ static char *linenoiseNoTTY(void) {
* editing function or uses dummy fgets() so that you will be able to type
* something even in the most desperate of the conditions. */
char *linenoise(const char *prompt) {
char buf[LINENOISE_MAX_LINE];
char buf[LINENOISE_MAX_LINE] = {0};
int count;
if (!isatty(STDIN_FILENO)) {
if (getenv("FAKETTY_WITH_PROMPT") == NULL && !isatty(STDIN_FILENO)) {
/* Not a tty: read from file / pipe. In this mode we don't want any
* limit to the line size, so we call a function to handle that. */
return linenoiseNoTTY();
} else if (isUnsupportedTerm()) {
} else if (getenv("FAKETTY_WITH_PROMPT") == NULL && isUnsupportedTerm()) {
size_t len;
printf("%s",prompt);
@@ -1112,6 +1230,7 @@ static void freeHistory(void) {
for (j = 0; j < history_len; j++)
free(history[j]);
free(history);
free(history_sensitive);
}
}
@@ -1128,7 +1247,7 @@ static void linenoiseAtExit(void) {
* histories, but will work well for a few hundred of entries.
*
* Using a circular buffer is smarter, but a bit more complex to handle. */
int linenoiseHistoryAdd(const char *line) {
int linenoiseHistoryAdd(const char *line, int is_sensitive) {
char *linecopy;
if (history_max_len == 0) return 0;
@@ -1137,7 +1256,14 @@ int linenoiseHistoryAdd(const char *line) {
if (history == NULL) {
history = malloc(sizeof(char*)*history_max_len);
if (history == NULL) return 0;
history_sensitive = malloc(sizeof(int)*history_max_len);
if (history_sensitive == NULL) {
free(history);
history = NULL;
return 0;
}
memset(history,0,(sizeof(char*)*history_max_len));
memset(history_sensitive,0,(sizeof(int)*history_max_len));
}
/* Don't add duplicated lines. */
@@ -1150,9 +1276,11 @@ int linenoiseHistoryAdd(const char *line) {
if (history_len == history_max_len) {
free(history[0]);
memmove(history,history+1,sizeof(char*)*(history_max_len-1));
memmove(history_sensitive,history_sensitive+1,sizeof(int)*(history_max_len-1));
history_len--;
}
history[history_len] = linecopy;
history_sensitive[history_len] = is_sensitive;
history_len++;
return 1;
}
@@ -1163,6 +1291,7 @@ int linenoiseHistoryAdd(const char *line) {
* than the amount of items already inside the history. */
int linenoiseHistorySetMaxLen(int len) {
char **new;
int *new_sensitive;
if (len < 1) return 0;
if (history) {
@@ -1170,6 +1299,11 @@ int linenoiseHistorySetMaxLen(int len) {
new = malloc(sizeof(char*)*len);
if (new == NULL) return 0;
new_sensitive = malloc(sizeof(int)*len);
if (new_sensitive == NULL) {
free(new);
return 0;
}
/* If we can't copy everything, free the elements we'll not use. */
if (len < tocopy) {
@@ -1179,9 +1313,13 @@ int linenoiseHistorySetMaxLen(int len) {
tocopy = len;
}
memset(new,0,sizeof(char*)*len);
memset(new_sensitive,0,sizeof(int)*len);
memcpy(new,history+(history_len-tocopy), sizeof(char*)*tocopy);
memcpy(new_sensitive,history_sensitive+(history_len-tocopy), sizeof(int)*tocopy);
free(history);
free(history_sensitive);
history = new;
history_sensitive = new_sensitive;
}
history_max_len = len;
if (history_len > history_max_len)
@@ -1201,7 +1339,7 @@ int linenoiseHistorySave(const char *filename) {
if (fp == NULL) return -1;
fchmod(fileno(fp),S_IRUSR|S_IWUSR);
for (j = 0; j < history_len; j++)
fprintf(fp,"%s\n",history[j]);
if (!history_sensitive[j]) fprintf(fp,"%s\n",history[j]);
fclose(fp);
return 0;
}
@@ -1223,8 +1361,97 @@ int linenoiseHistoryLoad(const char *filename) {
p = strchr(buf,'\r');
if (!p) p = strchr(buf,'\n');
if (p) *p = '\0';
linenoiseHistoryAdd(buf);
linenoiseHistoryAdd(buf, 0);
}
fclose(fp);
return 0;
}
/* This function updates the search index based on the direction of the search.
* Returns 0 if the beginning or end of the history is reached, otherwise, returns 1. */
static int setNextSearchIndex(int *i) {
if (reverse_search_direction == 1) {
if (*i == history_len-1) return 0;
*i = *i + 1;
} else {
if (*i <= 0) return 0;
*i = *i - 1;
}
return 1;
}
linenoiseHistorySearchResult searchInHistory(char *search_term) {
linenoiseHistorySearchResult result = {0};
if (!history_len || !strlen(search_term)) return result;
int i = cycle_to_next_search ? search_result_history_index :
(reverse_search_direction == -1 ? history_len-1 : 0);
while (1) {
char *found = strstr(history[i], search_term);
/* check if we found the same string at another index when cycling, this would be annoying to cycle through
* as it might appear that cycling isn't working */
int strings_are_the_same = cycle_to_next_search && strcmp(history[i], history[search_result_history_index]) == 0;
if (found && !strings_are_the_same) {
int haystack_index = found - history[i];
result.result = history[i];
result.len = strlen(history[i]);
result.search_term_index = haystack_index;
result.search_term_len = strlen(search_term);
search_result_history_index = i;
break;
}
/* Exit if reached the end. */
if (!setNextSearchIndex(&i)) break;
}
return result;
}
static void refreshSearchResult(struct linenoiseState *ls) {
if (!reverse_search_mode_enabled) {
return;
}
linenoiseHistorySearchResult sr = searchInHistory(ls->buf);
int found = sr.result && sr.len;
/* If the search term has not changed and we are cycling to the next search result
* (using CTRL+R or CTRL+S), there is no need to reset the old search result. */
if (!cycle_to_next_search || found)
resetSearchResult();
cycle_to_next_search = 0;
if (found) {
char *bold = "\x1B[1m";
char *normal = "\x1B[0m";
int size_needed = sr.search_term_index + sr.search_term_len + sr.len -
(sr.search_term_index+sr.search_term_len) + sizeof(normal) + sizeof(bold) + sizeof(normal);
if (size_needed > sizeof(search_result_friendly) - 1) {
return;
}
/* Allocate memory for the prefix, match, and suffix strings, one extra byte for `\0`. */
char *prefix = calloc(sizeof(char), sr.search_term_index + 1);
char *match = calloc(sizeof(char), sr.search_term_len + 1);
char *suffix = calloc(sizeof(char), sr.len - (sr.search_term_index+sr.search_term_len) + 1);
memcpy(prefix, sr.result, sr.search_term_index);
memcpy(match, sr.result + sr.search_term_index, sr.search_term_len);
memcpy(suffix, sr.result + sr.search_term_index + sr.search_term_len,
sr.len - (sr.search_term_index+sr.search_term_len));
sprintf(search_result, "%s%s%s", prefix, match, suffix);
sprintf(search_result_friendly, "%s%s%s%s%s%s", normal, prefix, bold, match, normal, suffix);
free(prefix);
free(match);
free(suffix);
search_result_start_offset = sr.search_term_index;
}
}
+1 -1
View File
@@ -58,7 +58,7 @@ void linenoiseAddCompletion(linenoiseCompletions *, const char *);
char *linenoise(const char *prompt);
void linenoiseFree(void *ptr);
int linenoiseHistoryAdd(const char *line);
int linenoiseHistoryAdd(const char *line, int is_sensitive);
int linenoiseHistorySetMaxLen(int len);
int linenoiseHistorySave(const char *filename);
int linenoiseHistoryLoad(const char *filename);
+8 -1
View File
@@ -234,10 +234,17 @@ static const luaL_Reg syslib[] = {
/* }====================================================== */
#define UNUSED(V) ((void) V)
/* Only a subset is loaded currently, for sandboxing concerns. */
static const luaL_Reg sandbox_syslib[] = {
{"clock", os_clock},
{NULL, NULL}
};
LUALIB_API int luaopen_os (lua_State *L) {
luaL_register(L, LUA_OSLIBNAME, syslib);
UNUSED(syslib);
luaL_register(L, LUA_OSLIBNAME, sandbox_syslib);
return 1;
}
+1
View File
@@ -132,6 +132,7 @@ static int bit_tohex(lua_State *L)
const char *hexdigits = "0123456789abcdef";
char buf[8];
int i;
if (n == INT32_MIN) n = INT32_MIN+1;
if (n < 0) { n = -n; hexdigits = "0123456789ABCDEF"; }
if (n > 8) n = 8;
for (i = (int)n; --i >= 0; ) { buf[i] = hexdigits[b & 15]; b >>= 4; }
+7 -5
View File
@@ -39,6 +39,7 @@
#include <assert.h>
#include <string.h>
#include <math.h>
#include <stdint.h>
#include <limits.h>
#include "lua.h"
#include "lauxlib.h"
@@ -141,13 +142,13 @@ typedef struct {
typedef struct {
json_token_type_t type;
int index;
size_t index;
union {
const char *string;
double number;
int boolean;
} value;
int string_len;
size_t string_len;
} json_token_t;
static const char *char2escape[256] = {
@@ -463,9 +464,8 @@ static void json_encode_exception(lua_State *l, json_config_t *cfg, strbuf_t *js
static void json_append_string(lua_State *l, strbuf_t *json, int lindex)
{
const char *escstr;
int i;
const char *str;
size_t len;
size_t i, len;
str = lua_tolstring(l, lindex, &len);
@@ -473,6 +473,8 @@ static void json_append_string(lua_State *l, strbuf_t *json, int lindex)
* This buffer is reused constantly for small strings
* If there are any excess pages, they won't be hit anyway.
* This gains ~5% speedup. */
if (len > SIZE_MAX / 6 - 3)
abort(); /* Overflow check */
strbuf_ensure_empty_length(json, len * 6 + 2);
strbuf_append_char_unsafe(json, '\"');
@@ -706,7 +708,7 @@ static int json_encode(lua_State *l)
strbuf_t local_encode_buf;
strbuf_t *encode_buf;
char *json;
int len;
size_t len;
luaL_argcheck(l, lua_gettop(l) == 1, 1, "expected 1 argument");
+17 -14
View File
@@ -117,7 +117,9 @@ mp_buf *mp_buf_new(lua_State *L) {
void mp_buf_append(lua_State *L, mp_buf *buf, const unsigned char *s, size_t len) {
if (buf->free < len) {
size_t newsize = (buf->len+len)*2;
size_t newsize = buf->len+len;
if (newsize < buf->len || newsize >= SIZE_MAX/2) abort();
newsize *= 2;
buf->b = (unsigned char*)mp_realloc(L, buf->b, buf->len + buf->free, newsize);
buf->free = newsize - buf->len;
@@ -173,7 +175,7 @@ void mp_cur_init(mp_cur *cursor, const unsigned char *s, size_t len) {
void mp_encode_bytes(lua_State *L, mp_buf *buf, const unsigned char *s, size_t len) {
unsigned char hdr[5];
int hdrlen;
size_t hdrlen;
if (len < 32) {
hdr[0] = 0xa0 | (len&0xff); /* fix raw */
@@ -220,7 +222,7 @@ void mp_encode_double(lua_State *L, mp_buf *buf, double d) {
void mp_encode_int(lua_State *L, mp_buf *buf, int64_t n) {
unsigned char b[9];
int enclen;
size_t enclen;
if (n >= 0) {
if (n <= 127) {
@@ -290,9 +292,9 @@ void mp_encode_int(lua_State *L, mp_buf *buf, int64_t n) {
mp_buf_append(L,buf,b,enclen);
}
void mp_encode_array(lua_State *L, mp_buf *buf, int64_t n) {
void mp_encode_array(lua_State *L, mp_buf *buf, uint64_t n) {
unsigned char b[5];
int enclen;
size_t enclen;
if (n <= 15) {
b[0] = 0x90 | (n & 0xf); /* fix array */
@@ -313,7 +315,7 @@ void mp_encode_array(lua_State *L, mp_buf *buf, int64_t n) {
mp_buf_append(L,buf,b,enclen);
}
void mp_encode_map(lua_State *L, mp_buf *buf, int64_t n) {
void mp_encode_map(lua_State *L, mp_buf *buf, uint64_t n) {
unsigned char b[5];
int enclen;
@@ -791,7 +793,7 @@ void mp_decode_to_lua_type(lua_State *L, mp_cur *c) {
}
}
int mp_unpack_full(lua_State *L, int limit, int offset) {
int mp_unpack_full(lua_State *L, lua_Integer limit, lua_Integer offset) {
size_t len;
const char *s;
mp_cur c;
@@ -803,10 +805,10 @@ int mp_unpack_full(lua_State *L, int limit, int offset) {
if (offset < 0 || limit < 0) /* requesting negative off or lim is invalid */
return luaL_error(L,
"Invalid request to unpack with offset of %d and limit of %d.",
offset, len);
(int) offset, (int) len);
else if (offset > len)
return luaL_error(L,
"Start offset %d greater than input length %d.", offset, len);
"Start offset %d greater than input length %d.", (int) offset, (int) len);
if (decode_all) limit = INT_MAX;
@@ -828,12 +830,13 @@ int mp_unpack_full(lua_State *L, int limit, int offset) {
/* c->left is the remaining size of the input buffer.
* subtract the entire buffer size from the unprocessed size
* to get our next start offset */
int offset = len - c.left;
size_t new_offset = len - c.left;
if (new_offset > LONG_MAX) abort();
luaL_checkstack(L, 1, "in function mp_unpack_full");
/* Return offset -1 when we have have processed the entire buffer. */
lua_pushinteger(L, c.left == 0 ? -1 : offset);
lua_pushinteger(L, c.left == 0 ? -1 : (lua_Integer) new_offset);
/* Results are returned with the arg elements still
* in place. Lua takes care of only returning
* elements above the args for us.
@@ -852,15 +855,15 @@ int mp_unpack(lua_State *L) {
}
int mp_unpack_one(lua_State *L) {
int offset = luaL_optinteger(L, 2, 0);
lua_Integer offset = luaL_optinteger(L, 2, 0);
/* Variable pop because offset may not exist */
lua_pop(L, lua_gettop(L)-1);
return mp_unpack_full(L, 1, offset);
}
int mp_unpack_limit(lua_State *L) {
int limit = luaL_checkinteger(L, 2);
int offset = luaL_optinteger(L, 3, 0);
lua_Integer limit = luaL_checkinteger(L, 2);
lua_Integer offset = luaL_optinteger(L, 3, 0);
/* Variable pop because offset may not exist */
lua_pop(L, lua_gettop(L)-1);
+27 -81
View File
@@ -26,6 +26,7 @@
#include <stdlib.h>
#include <stdarg.h>
#include <string.h>
#include <stdint.h>
#include "strbuf.h"
@@ -38,22 +39,22 @@ static void die(const char *fmt, ...)
va_end(arg);
fprintf(stderr, "\n");
exit(-1);
abort();
}
void strbuf_init(strbuf_t *s, int len)
void strbuf_init(strbuf_t *s, size_t len)
{
int size;
size_t size;
if (len <= 0)
if (!len)
size = STRBUF_DEFAULT_SIZE;
else
size = len + 1; /* \0 terminator */
size = len + 1;
if (size < len)
die("Overflow, len: %zu", len);
s->buf = NULL;
s->size = size;
s->length = 0;
s->increment = STRBUF_DEFAULT_INCREMENT;
s->dynamic = 0;
s->reallocs = 0;
s->debug = 0;
@@ -65,7 +66,7 @@ void strbuf_init(strbuf_t *s, int len)
strbuf_ensure_null(s);
}
strbuf_t *strbuf_new(int len)
strbuf_t *strbuf_new(size_t len)
{
strbuf_t *s;
@@ -81,20 +82,10 @@ strbuf_t *strbuf_new(int len)
return s;
}
void strbuf_set_increment(strbuf_t *s, int increment)
{
/* Increment > 0: Linear buffer growth rate
* Increment < -1: Exponential buffer growth rate */
if (increment == 0 || increment == -1)
die("BUG: Invalid string increment");
s->increment = increment;
}
static inline void debug_stats(strbuf_t *s)
{
if (s->debug) {
fprintf(stderr, "strbuf(%lx) reallocs: %d, length: %d, size: %d\n",
fprintf(stderr, "strbuf(%lx) reallocs: %d, length: %zd, size: %zd\n",
(long)s, s->reallocs, s->length, s->size);
}
}
@@ -113,7 +104,7 @@ void strbuf_free(strbuf_t *s)
free(s);
}
char *strbuf_free_to_string(strbuf_t *s, int *len)
char *strbuf_free_to_string(strbuf_t *s, size_t *len)
{
char *buf;
@@ -131,57 +122,61 @@ char *strbuf_free_to_string(strbuf_t *s, int *len)
return buf;
}
static int calculate_new_size(strbuf_t *s, int len)
static size_t calculate_new_size(strbuf_t *s, size_t len)
{
int reqsize, newsize;
size_t reqsize, newsize;
if (len <= 0)
die("BUG: Invalid strbuf length requested");
/* Ensure there is room for optional NULL termination */
reqsize = len + 1;
if (reqsize < len)
die("Overflow, len: %zu", len);
/* If the user has requested to shrink the buffer, do it exactly */
if (s->size > reqsize)
return reqsize;
newsize = s->size;
if (s->increment < 0) {
if (reqsize >= SIZE_MAX / 2) {
newsize = reqsize;
} else {
/* Exponential sizing */
while (newsize < reqsize)
newsize *= -s->increment;
} else {
/* Linear sizing */
newsize = ((newsize + s->increment - 1) / s->increment) * s->increment;
newsize *= 2;
}
if (newsize < reqsize)
die("BUG: strbuf length would overflow, len: %zu", len);
return newsize;
}
/* Ensure strbuf can handle a string length bytes long (ignoring NULL
* optional termination). */
void strbuf_resize(strbuf_t *s, int len)
void strbuf_resize(strbuf_t *s, size_t len)
{
int newsize;
size_t newsize;
newsize = calculate_new_size(s, len);
if (s->debug > 1) {
fprintf(stderr, "strbuf(%lx) resize: %d => %d\n",
fprintf(stderr, "strbuf(%lx) resize: %zd => %zd\n",
(long)s, s->size, newsize);
}
s->size = newsize;
s->buf = realloc(s->buf, s->size);
if (!s->buf)
die("Out of memory");
die("Out of memory, len: %zu", len);
s->reallocs++;
}
void strbuf_append_string(strbuf_t *s, const char *str)
{
int space, i;
size_t i, space;
space = strbuf_empty_length(s);
@@ -197,55 +192,6 @@ void strbuf_append_string(strbuf_t *s, const char *str)
}
}
/* strbuf_append_fmt() should only be used when an upper bound
* is known for the output string. */
void strbuf_append_fmt(strbuf_t *s, int len, const char *fmt, ...)
{
va_list arg;
int fmt_len;
strbuf_ensure_empty_length(s, len);
va_start(arg, fmt);
fmt_len = vsnprintf(s->buf + s->length, len, fmt, arg);
va_end(arg);
if (fmt_len < 0)
die("BUG: Unable to convert number"); /* This should never happen.. */
s->length += fmt_len;
}
/* strbuf_append_fmt_retry() can be used when the there is no known
* upper bound for the output string. */
void strbuf_append_fmt_retry(strbuf_t *s, const char *fmt, ...)
{
va_list arg;
int fmt_len, try;
int empty_len;
/* If the first attempt to append fails, resize the buffer appropriately
* and try again */
for (try = 0; ; try++) {
va_start(arg, fmt);
/* Append the new formatted string */
/* fmt_len is the length of the string required, excluding the
* trailing NULL */
empty_len = strbuf_empty_length(s);
/* Add 1 since there is also space to store the terminating NULL. */
fmt_len = vsnprintf(s->buf + s->length, empty_len + 1, fmt, arg);
va_end(arg);
if (fmt_len <= empty_len)
break; /* SUCCESS */
if (try > 0)
die("BUG: length of formatted string changed");
strbuf_resize(s, s->length + fmt_len);
}
s->length += fmt_len;
}
/* vi:ai et sw=4 ts=4:
*/
+19 -27
View File
@@ -27,15 +27,13 @@
/* Size: Total bytes allocated to *buf
* Length: String length, excluding optional NULL terminator.
* Increment: Allocation increments when resizing the string buffer.
* Dynamic: True if created via strbuf_new()
*/
typedef struct {
char *buf;
int size;
int length;
int increment;
size_t size;
size_t length;
int dynamic;
int reallocs;
int debug;
@@ -44,32 +42,26 @@ typedef struct {
#ifndef STRBUF_DEFAULT_SIZE
#define STRBUF_DEFAULT_SIZE 1023
#endif
#ifndef STRBUF_DEFAULT_INCREMENT
#define STRBUF_DEFAULT_INCREMENT -2
#endif
/* Initialise */
extern strbuf_t *strbuf_new(int len);
extern void strbuf_init(strbuf_t *s, int len);
extern void strbuf_set_increment(strbuf_t *s, int increment);
extern strbuf_t *strbuf_new(size_t len);
extern void strbuf_init(strbuf_t *s, size_t len);
/* Release */
extern void strbuf_free(strbuf_t *s);
extern char *strbuf_free_to_string(strbuf_t *s, int *len);
extern char *strbuf_free_to_string(strbuf_t *s, size_t *len);
/* Management */
extern void strbuf_resize(strbuf_t *s, int len);
static int strbuf_empty_length(strbuf_t *s);
static int strbuf_length(strbuf_t *s);
static char *strbuf_string(strbuf_t *s, int *len);
static void strbuf_ensure_empty_length(strbuf_t *s, int len);
extern void strbuf_resize(strbuf_t *s, size_t len);
static size_t strbuf_empty_length(strbuf_t *s);
static size_t strbuf_length(strbuf_t *s);
static char *strbuf_string(strbuf_t *s, size_t *len);
static void strbuf_ensure_empty_length(strbuf_t *s, size_t len);
static char *strbuf_empty_ptr(strbuf_t *s);
static void strbuf_extend_length(strbuf_t *s, int len);
static void strbuf_extend_length(strbuf_t *s, size_t len);
/* Update */
extern void strbuf_append_fmt(strbuf_t *s, int len, const char *fmt, ...);
extern void strbuf_append_fmt_retry(strbuf_t *s, const char *format, ...);
static void strbuf_append_mem(strbuf_t *s, const char *c, int len);
static void strbuf_append_mem(strbuf_t *s, const char *c, size_t len);
extern void strbuf_append_string(strbuf_t *s, const char *str);
static void strbuf_append_char(strbuf_t *s, const char c);
static void strbuf_ensure_null(strbuf_t *s);
@@ -87,12 +79,12 @@ static inline int strbuf_allocated(strbuf_t *s)
/* Return bytes remaining in the string buffer
* Ensure there is space for a NULL terminator. */
static inline int strbuf_empty_length(strbuf_t *s)
static inline size_t strbuf_empty_length(strbuf_t *s)
{
return s->size - s->length - 1;
}
static inline void strbuf_ensure_empty_length(strbuf_t *s, int len)
static inline void strbuf_ensure_empty_length(strbuf_t *s, size_t len)
{
if (len > strbuf_empty_length(s))
strbuf_resize(s, s->length + len);
@@ -103,12 +95,12 @@ static inline char *strbuf_empty_ptr(strbuf_t *s)
return s->buf + s->length;
}
static inline void strbuf_extend_length(strbuf_t *s, int len)
static inline void strbuf_extend_length(strbuf_t *s, size_t len)
{
s->length += len;
}
static inline int strbuf_length(strbuf_t *s)
static inline size_t strbuf_length(strbuf_t *s)
{
return s->length;
}
@@ -124,14 +116,14 @@ static inline void strbuf_append_char_unsafe(strbuf_t *s, const char c)
s->buf[s->length++] = c;
}
static inline void strbuf_append_mem(strbuf_t *s, const char *c, int len)
static inline void strbuf_append_mem(strbuf_t *s, const char *c, size_t len)
{
strbuf_ensure_empty_length(s, len);
memcpy(s->buf + s->length, c, len);
s->length += len;
}
static inline void strbuf_append_mem_unsafe(strbuf_t *s, const char *c, int len)
static inline void strbuf_append_mem_unsafe(strbuf_t *s, const char *c, size_t len)
{
memcpy(s->buf + s->length, c, len);
s->length += len;
@@ -142,7 +134,7 @@ static inline void strbuf_ensure_null(strbuf_t *s)
s->buf[s->length] = 0;
}
static inline char *strbuf_string(strbuf_t *s, int *len)
static inline char *strbuf_string(strbuf_t *s, size_t *len)
{
if (len)
*len = s->length;
+90
View File
@@ -0,0 +1,90 @@
SUBDIRS = redisjson redistimeseries redisbloom redisearch
define submake
for dir in $(SUBDIRS); do $(MAKE) -C $$dir $(1); done
endef
all: prepare_source
$(call submake,$@)
get_source:
$(call submake,$@)
prepare_source: get_source handle-werrors setup_environment
clean:
$(call submake,$@)
distclean: clean_environment
$(call submake,$@)
pristine:
$(call submake,$@)
install:
$(call submake,$@)
setup_environment: install-rust handle-werrors
clean_environment: uninstall-rust
# Keep all of the Rust stuff in one place
install-rust:
ifeq ($(INSTALL_RUST_TOOLCHAIN),yes)
@RUST_VERSION=1.80.1; \
ARCH="$$(uname -m)"; \
if ldd --version 2>&1 | grep -q musl; then LIBC_TYPE="musl"; else LIBC_TYPE="gnu"; fi; \
echo "Detected architecture: $${ARCH} and libc: $${LIBC_TYPE}"; \
case "$${ARCH}" in \
'x86_64') \
if [ "$${LIBC_TYPE}" = "musl" ]; then \
RUST_INSTALLER="rust-$${RUST_VERSION}-x86_64-unknown-linux-musl"; \
RUST_SHA256="37bbec6a7b9f55fef79c451260766d281a7a5b9d2e65c348bbc241127cf34c8d"; \
else \
RUST_INSTALLER="rust-$${RUST_VERSION}-x86_64-unknown-linux-gnu"; \
RUST_SHA256="85e936d5d36970afb80756fa122edcc99bd72a88155f6bdd514f5d27e778e00a"; \
fi ;; \
'aarch64') \
if [ "$${LIBC_TYPE}" = "musl" ]; then \
RUST_INSTALLER="rust-$${RUST_VERSION}-aarch64-unknown-linux-musl"; \
RUST_SHA256="dd668c2d82f77c5458deb023932600fae633fff8d7f876330e01bc47e9976d17"; \
else \
RUST_INSTALLER="rust-$${RUST_VERSION}-aarch64-unknown-linux-gnu"; \
RUST_SHA256="2e89bad7857711a1c11d017ea28fbfeec54076317763901194f8f5decbac1850"; \
fi ;; \
*) echo >&2 "Unsupported architecture: '$${ARCH}'"; exit 1 ;; \
esac; \
echo "Downloading and installing Rust standalone installer: $${RUST_INSTALLER}"; \
wget --quiet -O $${RUST_INSTALLER}.tar.xz https://static.rust-lang.org/dist/$${RUST_INSTALLER}.tar.xz; \
echo "$${RUST_SHA256} $${RUST_INSTALLER}.tar.xz" | sha256sum -c --quiet || { echo "Rust standalone installer checksum failed!"; exit 1; }; \
tar -xf $${RUST_INSTALLER}.tar.xz; \
(cd $${RUST_INSTALLER} && ./install.sh); \
rm -rf $${RUST_INSTALLER}
endif
uninstall-rust:
ifeq ($(INSTALL_RUST_TOOLCHAIN),yes)
@if [ -x "/usr/local/lib/rustlib/uninstall.sh" ]; then \
echo "Uninstalling Rust using uninstall.sh script"; \
rm -rf ~/.cargo; \
/usr/local/lib/rustlib/uninstall.sh; \
else \
echo "WARNING: Rust toolchain not found or uninstall script is missing."; \
fi
endif
handle-werrors: get_source
ifeq ($(DISABLE_WERRORS),yes)
@echo "Disabling -Werror for all modules"
@for dir in $(SUBDIRS); do \
echo "Processing $$dir"; \
find $$dir/src -type f \
\( -name "Makefile" \
-o -name "*.mk" \
-o -name "CMakeLists.txt" \) \
-exec sed -i 's/-Werror//g' {} +; \
done
endif
.PHONY: all clean distclean install $(SUBDIRS) setup_environment clean_environment install-rust uninstall-rust handle-werrors
+49
View File
@@ -0,0 +1,49 @@
PREFIX ?= /usr/local
INSTALL_DIR ?= $(DESTDIR)$(PREFIX)/lib/redis/modules
INSTALL ?= install
# This logic *partially* follows the current module build system. It is a bit awkward and
# should be changed if/when the modules' build process is refactored.
ARCH_MAP_x86_64 := x64
ARCH_MAP_i386 := x86
ARCH_MAP_i686 := x86
ARCH_MAP_aarch64 := arm64v8
ARCH_MAP_arm64 := arm64v8
OS := $(shell uname -s | tr '[:upper:]' '[:lower:]')
ARCH := $(ARCH_MAP_$(shell uname -m))
ifeq ($(ARCH),)
$(error Unrecognized CPU architecture $(shell uname -m))
endif
FULL_VARIANT := $(OS)-$(ARCH)-release
# Common rules for all modules, based on per-module configuration
all: $(TARGET_MODULE)
$(TARGET_MODULE): get_source
$(MAKE) -C $(SRC_DIR)
get_source: $(SRC_DIR)/.prepared
$(SRC_DIR)/.prepared:
mkdir -p $(SRC_DIR)
git clone --recursive --depth 1 --branch $(MODULE_VERSION) $(MODULE_REPO) $(SRC_DIR)
touch $@
clean:
-$(MAKE) -C $(SRC_DIR) clean
distclean:
-$(MAKE) -C $(SRC_DIR) distclean
pristine:
-rm -rf $(SRC_DIR)
install: $(TARGET_MODULE)
mkdir -p $(INSTALL_DIR)
$(INSTALL) -m 0755 -D $(TARGET_MODULE) $(INSTALL_DIR)
.PHONY: all clean distclean pristine install
+6
View File
@@ -0,0 +1,6 @@
SRC_DIR = src
MODULE_VERSION = v7.99.4
MODULE_REPO = https://github.com/redisbloom/redisbloom
TARGET_MODULE = $(SRC_DIR)/bin/$(FULL_VARIANT)/redisbloom.so
include ../common.mk
+7
View File
@@ -0,0 +1,7 @@
SRC_DIR = src
MODULE_VERSION = v7.99.5
MODULE_REPO = https://github.com/redisearch/redisearch
TARGET_MODULE = $(SRC_DIR)/bin/$(FULL_VARIANT)/search-community/redisearch.so
include ../common.mk
+11
View File
@@ -0,0 +1,11 @@
SRC_DIR = src
MODULE_VERSION = v7.99.4
MODULE_REPO = https://github.com/redisjson/redisjson
TARGET_MODULE = $(SRC_DIR)/bin/$(FULL_VARIANT)/rejson.so
include ../common.mk
$(SRC_DIR)/.cargo_fetched:
cd $(SRC_DIR) && cargo fetch
get_source: $(SRC_DIR)/.cargo_fetched
+6
View File
@@ -0,0 +1,6 @@
SRC_DIR = src
MODULE_VERSION = v7.99.4
MODULE_REPO = https://github.com/redistimeseries/redistimeseries
TARGET_MODULE = $(SRC_DIR)/bin/$(FULL_VARIANT)/redistimeseries.so
include ../common.mk
+77 -33
View File
@@ -51,6 +51,7 @@
#
# loadmodule /path/to/my_module.so
# loadmodule /path/to/other_module.so
# loadmodule /path/to/args_module.so [arg [arg ...]]
################################## NETWORK #####################################
@@ -346,6 +347,7 @@ pidfile /var/run/redis_6379.pid
# verbose (many rarely useful info, but not a mess like the debug level)
# notice (moderately verbose, what you want in production probably)
# warning (only very important / critical messages are logged)
# nothing (nothing is logged)
loglevel notice
# Specify the log file name. Also the empty string can be used to force
@@ -387,6 +389,11 @@ databases 16
# ASCII art logo in startup logs by setting the following option to yes.
always-show-logo no
# To avoid logging personal identifiable information (PII) into server log file,
# uncomment the following:
#
# hide-user-data-from-log yes
# By default, Redis modifies the process title (as seen in 'top' and 'ps') to
# provide some runtime information. It is possible to disable this and leave
# the process name as executed by setting the following to no.
@@ -661,7 +668,7 @@ repl-diskless-sync-max-replicas 0
repl-diskless-load disabled
# Master send PINGs to its replicas in a predefined interval. It's possible to
# change this interval with the repl_ping_replica_period option. The default
# change this interval with the repl-ping-replica-period option. The default
# value is 10 seconds.
#
# repl-ping-replica-period 10
@@ -720,6 +727,24 @@ repl-disable-tcp-nodelay no
#
# repl-backlog-ttl 3600
# During a fullsync, the master may decide to send both the RDB file and the
# replication stream to the replica in parallel. This approach shifts the
# responsibility of buffering the replication stream to the replica during the
# fullsync process. The replica accumulates the replication stream data until
# the RDB file is fully loaded. Once the RDB delivery is completed and
# successfully loaded, the replica begins processing and applying the
# accumulated replication data to the db. The configuration below controls how
# much replication data the replica can accumulate during a fullsync.
#
# When the replica reaches this limit, it will stop accumulating further data.
# At this point, additional data accumulation may occur on the master side
# depending on the 'client-output-buffer-limit <replica>' config of master.
#
# A value of 0 means replica inherits hard limit of
# 'client-output-buffer-limit <replica>' config to limit accumulation size.
#
# replica-full-sync-buffer-limit 0
# The replica priority is an integer number published by Redis in the INFO
# output. It is used by Redis Sentinel in order to select a replica to promote
# into a master if the master is no longer working correctly.
@@ -831,7 +856,7 @@ replica-priority 100
# this is used in order to send invalidation messages to clients. Please
# check this page to understand more about the feature:
#
# https://redis.io/topics/client-side-caching
# https://redis.io/docs/latest/develop/use/client-side-caching/
#
# When tracking is enabled for a client, all the read only queries are assumed
# to be cached: this will force Redis to store information in the invalidation
@@ -1009,7 +1034,7 @@ replica-priority 100
# * stream - Data type: streams related.
#
# For more information about ACL configuration please refer to
# the Redis web site at https://redis.io/topics/acl
# the Redis web site at https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/
# ACL LOG
#
@@ -1161,7 +1186,8 @@ acllog-max-len 128
# configuration directive.
#
# The default of 5 produces good enough results. 10 Approximates very closely
# true LRU but costs more CPU. 3 is faster but not very accurate.
# true LRU but costs more CPU. 3 is faster but not very accurate. The maximum
# value that can be set is 64.
#
# maxmemory-samples 5
@@ -1283,38 +1309,27 @@ lazyfree-lazy-user-flush no
# in different I/O threads. Since especially writing is so slow, normally
# Redis users use pipelining in order to speed up the Redis performances per
# core, and spawn multiple instances in order to scale more. Using I/O
# threads it is possible to easily speedup two times Redis without resorting
# threads it is possible to easily speedup several times Redis without resorting
# to pipelining nor sharding of the instance.
#
# By default threading is disabled, we suggest enabling it only in machines
# that have at least 4 or more cores, leaving at least one spare core.
# Using more than 8 threads is unlikely to help much. We also recommend using
# threaded I/O only if you actually have performance problems, with Redis
# instances being able to use a quite big percentage of CPU time, otherwise
# there is no point in using this feature.
# We also recommend using threaded I/O only if you actually have performance
# problems, with Redis instances being able to use a quite big percentage of
# CPU time, otherwise there is no point in using this feature.
#
# So for instance if you have a four cores boxes, try to use 2 or 3 I/O
# threads, if you have a 8 cores, try to use 6 threads. In order to
# So for instance if you have a four cores boxes, try to use 3 I/O
# threads, if you have a 8 cores, try to use 7 threads. In order to
# enable I/O threads use the following configuration directive:
#
# io-threads 4
#
# Setting io-threads to 1 will just use the main thread as usual.
# When I/O threads are enabled, we only use threads for writes, that is
# to thread the write(2) syscall and transfer the client buffers to the
# socket. However it is also possible to enable threading of reads and
# protocol parsing using the following configuration directive, by setting
# it to yes:
# When I/O threads are enabled, we not only use threads for writes, that
# is to thread the write(2) syscall and transfer the client buffers to the
# socket, but also use threads for reads and protocol parsing.
#
# io-threads-do-reads no
#
# Usually threading reads doesn't help much.
#
# NOTE 1: This configuration directive cannot be changed at runtime via
# CONFIG SET. Also, this feature currently does not work when SSL is
# enabled.
#
# NOTE 2: If you want to test the Redis speedup using redis-benchmark, make
# NOTE: If you want to test the Redis speedup using redis-benchmark, make
# sure you also run the benchmark itself in threaded mode, using the
# --threads option to match the number of Redis threads, otherwise you'll not
# be able to notice the improvements.
@@ -1354,7 +1369,7 @@ oom-score-adj-values 0 200 800
#################### KERNEL transparent hugepage CONTROL ######################
# Usually the kernel Transparent Huge Pages control is set to "madvise" or
# or "never" by default (/sys/kernel/mm/transparent_hugepage/enabled), in which
# "never" by default (/sys/kernel/mm/transparent_hugepage/enabled), in which
# case this config has no effect. On systems in which it is set to "always",
# redis will attempt to disable it specifically for the redis process in order
# to avoid latency problems specifically with fork(2) and CoW.
@@ -1381,7 +1396,11 @@ disable-thp yes
# If the AOF is enabled on startup Redis will load the AOF, that is the file
# with the better durability guarantees.
#
# Please check https://redis.io/topics/persistence for more information.
# Note that changing this value in a config file of an existing database and
# restarting the server can lead to data loss. A conversion needs to be done
# by setting it via CONFIG command on a live server first.
#
# Please check https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/ for more information.
appendonly no
@@ -1742,6 +1761,11 @@ aof-timestamp-enabled no
#
# cluster-announce-hostname ""
# Clusters can configure an optional nodename to be used in addition to the node ID for
# debugging and admin information. This name is broadcasted between nodes, so will be used
# in addition to the node ID when reporting cross node events such as node failures.
# cluster-announce-human-nodename ""
# Clusters can advertise how clients should connect to them using either their IP address,
# a user defined hostname, or by declaring they have no endpoint. Which endpoint is
# shown as the preferred endpoint is set by using the cluster-preferred-endpoint-type
@@ -1863,7 +1887,7 @@ latency-monitor-threshold 0
############################# EVENT NOTIFICATION ##############################
# Redis can notify Pub/Sub clients about events happening in the key space.
# This feature is documented at https://redis.io/topics/notifications
# This feature is documented at https://redis.io/docs/latest/develop/use/keyspace-notifications/
#
# For instance if keyspace events notification is enabled, and a client
# performs a DEL operation on key "foo" stored in the Database 0, two
@@ -2064,7 +2088,7 @@ client-output-buffer-limit pubsub 32mb 8mb 60
# amount by default in order to avoid that a protocol desynchronization (for
# instance due to a bug in the client) will lead to unbound memory usage in
# the query buffer. However you can configure it here if you have very special
# needs, such us huge multi/exec requests or alike.
# needs, such as a command with huge argument, or huge multi/exec requests or alike.
#
# client-query-buffer-limit 1gb
@@ -2189,6 +2213,26 @@ rdb-save-incremental-fsync yes
# lfu-log-factor 10
# lfu-decay-time 1
# The maximum number of new client connections accepted per event-loop cycle. This configuration
# is set independently for TLS connections.
#
# By default, up to 10 new connection will be accepted per event-loop cycle for normal connections
# and up to 1 new connection per event-loop cycle for TLS connections.
#
# Adjusting this to a larger number can slightly improve efficiency for new connections
# at the risk of causing timeouts for regular commands on established connections. It is
# not advised to change this without ensuring that all clients have limited connection
# pools and exponential backoff in the case of command/connection timeouts.
#
# If your application is establishing a large number of new connections per second you should
# also consider tuning the value of tcp-backlog, which allows the kernel to buffer more
# pending connections before dropping or rejecting connections.
#
# max-new-connections-per-cycle 10
# max-new-tls-connections-per-cycle 1
########################### ACTIVE DEFRAGMENTATION #######################
#
# What is active defragmentation?
@@ -2270,16 +2314,16 @@ jemalloc-bg-thread yes
# the taskset command:
#
# Set redis server/io threads to cpu affinity 0,2,4,6:
# server_cpulist 0-7:2
# server-cpulist 0-7:2
#
# Set bio threads to cpu affinity 1,3:
# bio_cpulist 1,3
# bio-cpulist 1,3
#
# Set aof rewrite child process to cpu affinity 8,9,10,11:
# aof_rewrite_cpulist 8-11
# aof-rewrite-cpulist 8-11
#
# Set bgsave child process to cpu affinity 1,10,11
# bgsave_cpulist 1,10-11
# bgsave-cpulist 1,10-11
# In some cases redis will emit warnings and even refuse to start if it detects
# that the system is in bad state, it is possible to suppress these warnings
+2
View File
@@ -55,4 +55,6 @@ $TCLSH tests/test_helper.tcl \
--single unit/moduleapi/async_rm_call \
--single unit/moduleapi/moduleauth \
--single unit/moduleapi/rdbloadsave \
--single unit/moduleapi/crash \
--single unit/moduleapi/internalsecret \
"${@}"
+4 -3
View File
@@ -25,6 +25,7 @@ pidfile /var/run/redis-sentinel.pid
# verbose (many rarely useful info, but not a mess like the debug level)
# notice (moderately verbose, what you want in production probably)
# warning (only very important / critical messages are logged)
# nothing (nothing is logged)
loglevel notice
# Specify the log file name. Also the empty string can be used to force
@@ -132,7 +133,7 @@ sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 30000
# IMPORTANT NOTE: starting with Redis 6.2 ACL capability is supported for
# Sentinel mode, please refer to the Redis website https://redis.io/topics/acl
# Sentinel mode, please refer to the Redis website https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/
# for more details.
# Sentinel's ACL users are defined in the following format:
@@ -144,7 +145,7 @@ sentinel down-after-milliseconds mymaster 30000
# user worker +@admin +@connection ~* on >ffa9203c493aa99
#
# For more information about ACL configuration please refer to the Redis
# website at https://redis.io/topics/acl and redis server configuration
# website at https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl/ and redis server configuration
# template redis.conf.
# ACL LOG
@@ -173,7 +174,7 @@ acllog-max-len 128
# so Sentinel will try to authenticate with the same password to all the
# other Sentinels. So you need to configure all your Sentinels in a given
# group with the same "requirepass" password. Check the following documentation
# for more info: https://redis.io/topics/sentinel
# for more info: https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/
#
# IMPORTANT NOTE: starting with Redis 6.2 "requirepass" is a compatibility
# layer on top of the ACL system. The option effect will be just setting
+34 -19
View File
@@ -1,6 +1,9 @@
# Redis Makefile
# Copyright (C) 2009 Salvatore Sanfilippo <antirez at gmail dot com>
# This file is released under the BSD license, see the COPYING file
# Copyright (c) 2011-Present, Redis Ltd.
# All rights reserved.
#
# Licensed under your choice of the Redis Source Available License 2.0
# (RSALv2) or the Server Side Public License v1 (SSPLv1).
#
# The Makefile composes the final FINAL_CFLAGS and FINAL_LDFLAGS using
# what is needed for Redis plus the standard CFLAGS and LDFLAGS passed.
@@ -16,16 +19,22 @@ release_hdr := $(shell sh -c './mkreleasehdr.sh')
uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo not')
uname_M := $(shell sh -c 'uname -m 2>/dev/null || echo not')
CLANG := $(findstring clang,$(shell sh -c '$(CC) --version | head -1'))
# Optimization flags. To override, the OPTIMIZATION variable can be passed, but
# some automatic defaults are added to it. To specify optimization flags
# explicitly without any defaults added, pass the OPT variable instead.
OPTIMIZATION?=-O3
ifeq ($(OPTIMIZATION),-O3)
ifeq (clang,$(CLANG))
REDIS_CFLAGS+=-flto
OPTIMIZATION+=-flto
else
REDIS_CFLAGS+=-flto=auto
OPTIMIZATION+=-flto=auto
endif
REDIS_LDFLAGS+=-flto
endif
DEPENDENCY_TARGETS=hiredis linenoise lua hdr_histogram fpconv
ifneq ($(OPTIMIZATION),-O0)
OPTIMIZATION+=-fno-omit-frame-pointer
endif
DEPENDENCY_TARGETS=hiredis linenoise lua hdr_histogram fpconv fast_float
NODEPS:=clean distclean
# Default settings
@@ -47,10 +56,10 @@ OPT=$(OPTIMIZATION)
# NUMBER_SIGN_CHAR is a workaround to support both GNU Make 4.3 and older versions.
NUMBER_SIGN_CHAR := \#
C11_ATOMIC := $(shell sh -c 'echo "$(NUMBER_SIGN_CHAR)include <stdatomic.h>" > foo.c; \
$(CC) -std=c11 -c foo.c -o foo.o > /dev/null 2>&1; \
$(CC) -std=gnu11 -c foo.c -o foo.o > /dev/null 2>&1; \
if [ -f foo.o ]; then echo "yes"; rm foo.o; fi; rm foo.c')
ifeq ($(C11_ATOMIC),yes)
STD+=-std=c11
STD+=-std=gnu11
else
STD+=-std=c99
endif
@@ -117,8 +126,8 @@ endif
-include .make-settings
FINAL_CFLAGS=$(STD) $(WARN) $(OPT) $(DEBUG) $(CFLAGS) $(REDIS_CFLAGS)
FINAL_LDFLAGS=$(LDFLAGS) $(REDIS_LDFLAGS) $(DEBUG)
FINAL_LIBS=-lm
FINAL_LDFLAGS=$(LDFLAGS) $(OPT) $(REDIS_LDFLAGS) $(DEBUG)
FINAL_LIBS=-lm -lstdc++
DEBUG=-g -ggdb
# Linux ARM32 needs -latomic at linking time
@@ -226,7 +235,7 @@ ifdef OPENSSL_PREFIX
endif
# Include paths to dependencies
FINAL_CFLAGS+= -I../deps/hiredis -I../deps/linenoise -I../deps/lua/src -I../deps/hdr_histogram -I../deps/fpconv
FINAL_CFLAGS+= -I../deps/hiredis -I../deps/linenoise -I../deps/lua/src -I../deps/hdr_histogram -I../deps/fpconv -I../deps/fast_float
# Determine systemd support and/or build preference (defaulting to auto-detection)
BUILD_WITH_SYSTEMD=no
@@ -345,11 +354,11 @@ endif
REDIS_SERVER_NAME=redis-server$(PROG_SUFFIX)
REDIS_SENTINEL_NAME=redis-sentinel$(PROG_SUFFIX)
REDIS_SERVER_OBJ=adlist.o quicklist.o ae.o anet.o dict.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o eval.o bio.o rio.o rand.o memtest.o syscheck.o crcspeed.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o redis-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o lolwut.o lolwut5.o lolwut6.o acl.o tracking.o socket.o tls.o sha256.o timeout.o setcpuaffinity.o monotonic.o mt19937-64.o resp_parser.o call_reply.o script_lua.o script.o functions.o function_lua.o commands.o strl.o connection.o unix.o logreqres.o
REDIS_SERVER_OBJ=threads_mngr.o adlist.o quicklist.o ae.o anet.o dict.o ebuckets.o eventnotifier.o iothread.o mstr.o kvstore.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o cluster_legacy.o crc16.o endianconv.o slowlog.o eval.o bio.o rio.o rand.o memtest.o syscheck.o crcspeed.o crccombine.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o redis-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o lolwut.o lolwut5.o lolwut6.o acl.o tracking.o socket.o tls.o sha256.o timeout.o setcpuaffinity.o monotonic.o mt19937-64.o resp_parser.o call_reply.o script_lua.o script.o functions.o function_lua.o commands.o strl.o connection.o unix.o logreqres.o
REDIS_CLI_NAME=redis-cli$(PROG_SUFFIX)
REDIS_CLI_OBJ=anet.o adlist.o dict.o redis-cli.o zmalloc.o release.o ae.o redisassert.o crcspeed.o crc64.o siphash.o crc16.o monotonic.o cli_common.o mt19937-64.o strl.o cli_commands.o
REDIS_CLI_OBJ=anet.o adlist.o dict.o redis-cli.o zmalloc.o release.o ae.o redisassert.o crcspeed.o crccombine.o crc64.o siphash.o crc16.o monotonic.o cli_common.o mt19937-64.o strl.o cli_commands.o
REDIS_BENCHMARK_NAME=redis-benchmark$(PROG_SUFFIX)
REDIS_BENCHMARK_OBJ=ae.o anet.o redis-benchmark.o adlist.o dict.o zmalloc.o redisassert.o release.o crcspeed.o crc64.o siphash.o crc16.o monotonic.o cli_common.o mt19937-64.o strl.o
REDIS_BENCHMARK_OBJ=ae.o anet.o redis-benchmark.o adlist.o dict.o zmalloc.o redisassert.o release.o crcspeed.o crccombine.o crc64.o siphash.o crc16.o monotonic.o cli_common.o mt19937-64.o strl.o
REDIS_CHECK_RDB_NAME=redis-check-rdb$(PROG_SUFFIX)
REDIS_CHECK_AOF_NAME=redis-check-aof$(PROG_SUFFIX)
ALL_SOURCES=$(sort $(patsubst %.o,%.c,$(REDIS_SERVER_OBJ) $(REDIS_CLI_OBJ) $(REDIS_BENCHMARK_OBJ)))
@@ -400,7 +409,7 @@ endif
# redis-server
$(REDIS_SERVER_NAME): $(REDIS_SERVER_OBJ)
$(REDIS_LD) -o $@ $^ ../deps/hiredis/libhiredis.a ../deps/lua/src/liblua.a ../deps/hdr_histogram/libhdrhistogram.a ../deps/fpconv/libfpconv.a $(FINAL_LIBS)
$(REDIS_LD) -o $@ $^ ../deps/hiredis/libhiredis.a ../deps/lua/src/liblua.a ../deps/hdr_histogram/libhdrhistogram.a ../deps/fpconv/libfpconv.a ../deps/fast_float/libfast_float.a $(FINAL_LIBS)
# redis-sentinel
$(REDIS_SENTINEL_NAME): $(REDIS_SERVER_NAME)
@@ -420,7 +429,7 @@ $(TLS_MODULE_NAME): $(REDIS_SERVER_NAME)
# redis-cli
$(REDIS_CLI_NAME): $(REDIS_CLI_OBJ)
$(REDIS_LD) -o $@ $^ ../deps/hiredis/libhiredis.a ../deps/linenoise/linenoise.o $(FINAL_LIBS) $(TLS_CLIENT_LIBS)
$(REDIS_LD) -o $@ $^ ../deps/hiredis/libhiredis.a ../deps/linenoise/linenoise.o ../deps/hdr_histogram/libhdrhistogram.a $(FINAL_LIBS) $(TLS_CLIENT_LIBS)
# redis-benchmark
$(REDIS_BENCHMARK_NAME): $(REDIS_BENCHMARK_OBJ)
@@ -435,11 +444,16 @@ DEP = $(REDIS_SERVER_OBJ:%.o=%.d) $(REDIS_CLI_OBJ:%.o=%.d) $(REDIS_BENCHMARK_OBJ
%.o: %.c .make-prerequisites
$(REDIS_CC) -MMD -o $@ -c $<
# The file commands.def is checked in and doesn't normally need to be rebuilt. It
# is built only if python is available and its prereqs are modified.
# The following files are checked in and don't normally need to be rebuilt. They
# are built only if python is available and their prereqs are modified.
ifneq (,$(PYTHON))
$(COMMANDS_DEF_FILENAME).def: commands/*.json ../utils/generate-command-code.py
$(QUIET_GEN)$(PYTHON) ../utils/generate-command-code.py $(GEN_COMMANDS_FLAGS)
fmtargs.h: ../utils/generate-fmtargs.py
$(QUITE_GEN)sed '/Everything below this line/,$$d' $@ > $@.tmp
$(QUITE_GEN)$(PYTHON) ../utils/generate-fmtargs.py >> $@.tmp
$(QUITE_GEN)mv $@.tmp $@
endif
commands.c: $(COMMANDS_DEF_FILENAME).def
@@ -473,8 +487,9 @@ test-cluster: $(REDIS_SERVER_NAME) $(REDIS_CLI_NAME)
check: test
lcov:
@lcov --version
$(MAKE) gcov
@(set -e; cd ..; ./runtest --clients 1)
@(set -e; cd ..; ./runtest)
@geninfo -o redis.info .
@genhtml --legend -o lcov-html redis.info
+278 -124
View File
@@ -1,33 +1,13 @@
/*
* Copyright (c) 2018, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2018-Present, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
* Licensed under your choice of the Redis Source Available License 2.0
* (RSALv2) or the Server Side Public License v1 (SSPLv1).
*/
#include "server.h"
#include "cluster.h"
#include "sha256.h"
#include <fcntl.h>
#include <ctype.h>
@@ -59,10 +39,12 @@ static rax *commandId = NULL; /* Command name to id mapping */
static unsigned long nextid = 0; /* Next command id that has not been assigned */
#define ACL_MAX_CATEGORIES 64 /* Maximum number of command categories */
struct ACLCategoryItem {
const char *name;
char *name;
uint64_t flag;
} ACLCommandCategories[] = { /* See redis.conf for details on each category. */
} ACLDefaultCommandCategories[] = { /* See redis.conf for details on each category. */
{"keyspace", ACL_CATEGORY_KEYSPACE},
{"read", ACL_CATEGORY_READ},
{"write", ACL_CATEGORY_WRITE},
@@ -87,6 +69,54 @@ struct ACLCategoryItem {
{NULL,0} /* Terminator. */
};
static struct ACLCategoryItem *ACLCommandCategories = NULL;
static size_t nextCommandCategory = 0; /* Index of the next command category to be added */
/* Implements the ability to add to the list of ACL categories at runtime. Since each ACL category
* also requires a bit in the acl_categories flag, there is a limit to the number that can be added.
* The new ACL categories occupy the remaining bits of acl_categories flag, other than the bits
* occupied by the default ACL command categories.
*
* The optional `flag` argument allows the assignment of the `acl_categories` flag bit to the ACL category.
* When adding a new category, except for the default ACL command categories, this arguments should be `0`
* to allow the function to assign the next available `acl_categories` flag bit to the new ACL category.
*
* returns 1 -> Added, 0 -> Failed (out of space)
*
* This function is present here to gain access to the ACLCommandCategories array and add a new ACL category.
*/
int ACLAddCommandCategory(const char *name, uint64_t flag) {
if (nextCommandCategory >= ACL_MAX_CATEGORIES) return 0;
ACLCommandCategories[nextCommandCategory].name = zstrdup(name);
ACLCommandCategories[nextCommandCategory].flag = flag != 0 ? flag : (1ULL<<nextCommandCategory);
nextCommandCategory++;
return 1;
}
/* Initializes ACLCommandCategories with default ACL categories and allocates space for
* new ACL categories.
*/
void ACLInitCommandCategories(void) {
ACLCommandCategories = zcalloc(sizeof(struct ACLCategoryItem) * (ACL_MAX_CATEGORIES + 1));
for (int j = 0; ACLDefaultCommandCategories[j].flag; j++) {
serverAssert(ACLAddCommandCategory(ACLDefaultCommandCategories[j].name, ACLDefaultCommandCategories[j].flag));
}
}
/* This function removes the specified number of categories from the trailing end of
* the `ACLCommandCategories` array.
* The purpose of this is to remove the categories added by modules that fail
* during the onload function.
*/
void ACLCleanupCategoriesOnFailure(size_t num_acl_categories_added) {
for (size_t j = nextCommandCategory - num_acl_categories_added; j < nextCommandCategory; j++) {
zfree(ACLCommandCategories[j].name);
ACLCommandCategories[j].name = NULL;
ACLCommandCategories[j].flag = 0;
}
nextCommandCategory -= num_acl_categories_added;
}
struct ACLUserFlag {
const char *name;
uint64_t flag;
@@ -248,7 +278,7 @@ int ACLListMatchSds(void *a, void *b) {
/* Method to free list elements from ACL users password/patterns lists. */
void ACLListFreeSds(void *item) {
sdsfree(item);
sdsfreegeneric(item);
}
/* Method to duplicate list elements from ACL users password/patterns lists. */
@@ -259,7 +289,7 @@ void *ACLListDupSds(void *item) {
/* Structure used for handling key patterns with different key
* based permissions. */
typedef struct {
int flags; /* The CMD_KEYS_* flags for this key pattern */
int flags; /* The ACL key permission types for this key pattern */
sds pattern; /* The pattern to match keys against */
} keyPattern;
@@ -387,7 +417,7 @@ aclSelector *ACLUserGetRootSelector(user *u) {
*
* If the user with such name already exists NULL is returned. */
user *ACLCreateUser(const char *name, size_t namelen) {
if (raxFind(Users,(unsigned char*)name,namelen) != raxNotFound) return NULL;
if (raxFind(Users,(unsigned char*)name,namelen,NULL)) return NULL;
user *u = zmalloc(sizeof(*u));
u->name = sdsnewlen(name,namelen);
u->flags = USER_FLAG_DISABLED;
@@ -440,6 +470,11 @@ void ACLFreeUser(user *u) {
zfree(u);
}
/* Generic version of ACLFreeUser. */
void ACLFreeUserGeneric(void *u) {
ACLFreeUser((user *)u);
}
/* When a user is deleted we need to cycle the active
* connections in order to kill all the pending ones that
* are authenticated with such user. */
@@ -456,15 +491,7 @@ void ACLFreeUserAndKillClients(user *u) {
* this may result in some security hole: it's much
* more defensive to set the default user and put
* it in non authenticated mode. */
c->user = DefaultUser;
c->authenticated = 0;
/* We will write replies to this client later, so we can't
* close it directly even if async. */
if (c == server.current_client) {
c->flags |= CLIENT_CLOSE_AFTER_COMMAND;
} else {
freeClientAsync(c);
}
deauthenticateAndCloseClient(c);
}
}
ACLFreeUser(u);
@@ -489,12 +516,6 @@ void ACLCopyUser(user *dst, user *src) {
}
}
/* Free all the users registered in the radix tree 'users' and free the
* radix tree itself. */
void ACLFreeUsersSet(rax *users) {
raxFreeWithCallback(users,(void(*)(void*))ACLFreeUserAndKillClients);
}
/* Given a command ID, this function set by reference 'word' and 'bit'
* so that user->allowed_commands[word] will address the right word
* where the corresponding bit for the provided ID is stored, and
@@ -563,7 +584,7 @@ void ACLSelectorRemoveCommandRule(aclSelector *selector, sds new_rule) {
* as well if the command is removed. */
char *rule_end = strchr(existing_rule, ' ');
if (!rule_end) {
/* This is the last rule, so it it to the end of the string. */
/* This is the last rule, so move it to the end of the string. */
rule_end = existing_rule + strlen(existing_rule);
/* This approach can leave a trailing space if the last rule is removed,
@@ -580,6 +601,8 @@ void ACLSelectorRemoveCommandRule(aclSelector *selector, sds new_rule) {
/* Copy the remaining rules starting at the next rule to replace the rule to be
* deleted, including the terminating NULL character. */
memmove(copy_position, copy_end, strlen(copy_end) + 1);
existing_rule = copy_position;
continue;
}
}
existing_rule = copy_end;
@@ -911,7 +934,7 @@ void ACLResetFirstArgs(aclSelector *selector) {
selector->allowed_firstargs = NULL;
}
/* Add a first-arh to the list of subcommands for the user 'u' and
/* Add a first-arg to the list of subcommands for the user 'u' and
* the command id specified. */
void ACLAddAllowedFirstArg(aclSelector *selector, unsigned long id, const char *sub) {
/* If this is the first first-arg to be configured for
@@ -1044,6 +1067,7 @@ int ACLSetSelector(aclSelector *selector, const char* op, size_t oplen) {
int flags = 0;
size_t offset = 1;
if (op[0] == '%') {
int perm_ok = 1;
for (; offset < oplen; offset++) {
if (toupper(op[offset]) == 'R' && !(flags & ACL_READ_PERMISSION)) {
flags |= ACL_READ_PERMISSION;
@@ -1053,10 +1077,14 @@ int ACLSetSelector(aclSelector *selector, const char* op, size_t oplen) {
offset++;
break;
} else {
errno = EINVAL;
return C_ERR;
perm_ok = 0;
break;
}
}
if (!flags || !perm_ok) {
errno = EINVAL;
return C_ERR;
}
} else {
flags = ACL_ALL_PERMISSION;
}
@@ -1396,6 +1424,7 @@ user *ACLCreateDefaultUser(void) {
void ACLInit(void) {
Users = raxNew();
UsersToLoad = listCreate();
ACLInitCommandCategories();
listSetMatchMethod(UsersToLoad, ACLListMatchLoadedUser);
ACLLog = listCreate();
DefaultUser = ACLCreateDefaultUser();
@@ -1405,7 +1434,7 @@ void ACLInit(void) {
* otherwise C_ERR is returned and errno is set to:
*
* EINVAL: if the username-password do not match.
* ENONENT: if the specified user does not exist at all.
* ENOENT: if the specified user does not exist at all.
*/
int ACLCheckUserCredentials(robj *username, robj *password) {
user *u = ACLGetUserByName(username->ptr,sdslen(username->ptr));
@@ -1500,8 +1529,8 @@ unsigned long ACLGetCommandID(sds cmdname) {
sds lowername = sdsdup(cmdname);
sdstolower(lowername);
if (commandId == NULL) commandId = raxNew();
void *id = raxFind(commandId,(unsigned char*)lowername,sdslen(lowername));
if (id != raxNotFound) {
void *id;
if (raxFind(commandId,(unsigned char*)lowername,sdslen(lowername),&id)) {
sdsfree(lowername);
return (unsigned long)id;
}
@@ -1532,8 +1561,8 @@ void ACLClearCommandID(void) {
/* Return an username by its name, or NULL if the user does not exist. */
user *ACLGetUserByName(const char *name, size_t namelen) {
void *myuser = raxFind(Users,(unsigned char*)name,namelen);
if (myuser == raxNotFound) return NULL;
void *myuser = NULL;
raxFind(Users,(unsigned char*)name,namelen,&myuser);
return myuser;
}
@@ -1559,14 +1588,22 @@ static int ACLSelectorCheckKey(aclSelector *selector, const char *key, int keyle
if (keyspec_flags & CMD_KEY_DELETE) key_flags |= ACL_WRITE_PERMISSION;
if (keyspec_flags & CMD_KEY_UPDATE) key_flags |= ACL_WRITE_PERMISSION;
/* Is given key represent a prefix of a set of keys */
int prefix = keyspec_flags & CMD_KEY_PREFIX;
/* Test this key against every pattern. */
while((ln = listNext(&li))) {
keyPattern *pattern = listNodeValue(ln);
if ((pattern->flags & key_flags) != key_flags)
continue;
size_t plen = sdslen(pattern->pattern);
if (stringmatchlen(pattern->pattern,plen,key,keylen,0))
return ACL_OK;
if (prefix) {
if (prefixmatch(pattern->pattern,plen,key,keylen,0))
return ACL_OK;
} else {
if (stringmatchlen(pattern->pattern, plen, key, keylen, 0))
return ACL_OK;
}
}
return ACL_DENIED_KEY;
}
@@ -1856,23 +1893,20 @@ int ACLCheckAllPerm(client *c, int *idxptr) {
return ACLCheckAllUserCommandPerm(c->user, c->cmd, c->argv, c->argc, idxptr);
}
/* Check if the user's existing pub/sub clients violate the ACL pub/sub
* permissions specified via the upcoming argument, and kill them if so. */
void ACLKillPubsubClientsIfNeeded(user *new, user *original) {
/* If 'new' can access all channels 'original' could then return NULL;
Otherwise return a list of channels that the new user can access */
list *getUpcomingChannelList(user *new, user *original) {
listIter li, lpi;
listNode *ln, *lpn;
robj *o;
int kill = 0;
/* First optimization is we check if any selector has all channel
* permissions. */
/* Optimization: we check if any selector has all channel permissions. */
listRewind(new->selectors,&li);
while((ln = listNext(&li))) {
aclSelector *s = (aclSelector *) listNodeValue(ln);
if (s->flags & SELECTOR_FLAG_ALLCHANNELS) return;
if (s->flags & SELECTOR_FLAG_ALLCHANNELS) return NULL;
}
/* Second optimization is to check if the new list of channels
/* Next, check if the new list of channels
* is a strict superset of the original. This is done by
* created an "upcoming" list of all channels that are in
* the new user and checking each of the existing channels
@@ -1910,57 +1944,87 @@ void ACLKillPubsubClientsIfNeeded(user *new, user *original) {
if (match) {
/* All channels were matched, no need to kill clients. */
listRelease(upcoming);
return;
return NULL;
}
return upcoming;
}
/* Check if the client should be killed because it is subscribed to channels that were
* permitted in the past, are not in the `upcoming` channel list. */
int ACLShouldKillPubsubClient(client *c, list *upcoming) {
robj *o;
int kill = 0;
if (getClientType(c) == CLIENT_TYPE_PUBSUB) {
/* Check for pattern violations. */
dictIterator *di = dictGetIterator(c->pubsub_patterns);
dictEntry *de;
while (!kill && ((de = dictNext(di)) != NULL)) {
o = dictGetKey(de);
int res = ACLCheckChannelAgainstList(upcoming, o->ptr, sdslen(o->ptr), 1);
kill = (res == ACL_DENIED_CHANNEL);
}
dictReleaseIterator(di);
/* Check for channel violations. */
if (!kill) {
/* Check for global channels violation. */
di = dictGetIterator(c->pubsub_channels);
while (!kill && ((de = dictNext(di)) != NULL)) {
o = dictGetKey(de);
int res = ACLCheckChannelAgainstList(upcoming, o->ptr, sdslen(o->ptr), 0);
kill = (res == ACL_DENIED_CHANNEL);
}
dictReleaseIterator(di);
}
if (!kill) {
/* Check for shard channels violation. */
di = dictGetIterator(c->pubsubshard_channels);
while (!kill && ((de = dictNext(di)) != NULL)) {
o = dictGetKey(de);
int res = ACLCheckChannelAgainstList(upcoming, o->ptr, sdslen(o->ptr), 0);
kill = (res == ACL_DENIED_CHANNEL);
}
dictReleaseIterator(di);
}
if (kill) {
return 1;
}
}
return 0;
}
/* Check if the user's existing pub/sub clients violate the ACL pub/sub
* permissions specified via the upcoming argument, and kill them if so. */
void ACLKillPubsubClientsIfNeeded(user *new, user *original) {
/* Do nothing if there are no subscribers. */
if (pubsubTotalSubscriptions() == 0)
return;
list *channels = getUpcomingChannelList(new, original);
/* If the new user's pubsub permissions are a strict superset of the original, return early. */
if (!channels)
return;
listIter li;
listNode *ln;
/* Permissions have changed, so we need to iterate through all
* the clients and disconnect those that are no longer valid.
* Scan all connected clients to find the user's pub/subs. */
listRewind(server.clients,&li);
while ((ln = listNext(&li)) != NULL) {
client *c = listNodeValue(ln);
kill = 0;
if (c->user == original && getClientType(c) == CLIENT_TYPE_PUBSUB) {
/* Check for pattern violations. */
listRewind(c->pubsub_patterns,&lpi);
while (!kill && ((lpn = listNext(&lpi)) != NULL)) {
o = lpn->value;
int res = ACLCheckChannelAgainstList(upcoming, o->ptr, sdslen(o->ptr), 1);
kill = (res == ACL_DENIED_CHANNEL);
}
/* Check for channel violations. */
if (!kill) {
/* Check for global channels violation. */
dictIterator *di = dictGetIterator(c->pubsub_channels);
dictEntry *de;
while (!kill && ((de = dictNext(di)) != NULL)) {
o = dictGetKey(de);
int res = ACLCheckChannelAgainstList(upcoming, o->ptr, sdslen(o->ptr), 0);
kill = (res == ACL_DENIED_CHANNEL);
}
dictReleaseIterator(di);
/* Check for shard channels violation. */
di = dictGetIterator(c->pubsubshard_channels);
while (!kill && ((de = dictNext(di)) != NULL)) {
o = dictGetKey(de);
int res = ACLCheckChannelAgainstList(upcoming, o->ptr, sdslen(o->ptr), 0);
kill = (res == ACL_DENIED_CHANNEL);
}
dictReleaseIterator(di);
}
/* Kill it. */
if (kill) {
freeClient(c);
}
}
if (c->user != original)
continue;
if (ACLShouldKillPubsubClient(c, channels))
deauthenticateAndCloseClient(c);
}
listRelease(upcoming);
listRelease(channels);
}
/* =============================================================================
@@ -1989,7 +2053,8 @@ sds *ACLMergeSelectorArguments(sds *argv, int argc, int *merged_argc, int *inval
for (int j = 0; j < argc; j++) {
char *op = argv[j];
if (op[0] == '(' && op[sdslen(op) - 1] != ')') {
if (open_bracket_start == -1 &&
(op[0] == '(' && op[sdslen(op) - 1] != ')')) {
selector = sdsdup(argv[j]);
open_bracket_start = j;
continue;
@@ -2112,10 +2177,6 @@ int ACLAppendUserForLoading(sds *argv, int argc, int *argc_err) {
return C_ERR;
}
/* Try to apply the user rules in a fake user to see if they
* are actually valid. */
user *fakeuser = ACLCreateUnlinkedUser();
/* Merged selectors before trying to process */
int merged_argc;
sds *acl_args = ACLMergeSelectorArguments(argv + 2, argc - 2, &merged_argc, argc_err);
@@ -2124,6 +2185,10 @@ int ACLAppendUserForLoading(sds *argv, int argc, int *argc_err) {
return C_ERR;
}
/* Try to apply the user rules in a fake user to see if they
* are actually valid. */
user *fakeuser = ACLCreateUnlinkedUser();
for (int j = 0; j < merged_argc; j++) {
if (ACLSetUser(fakeuser,acl_args[j],sdslen(acl_args[j])) == C_ERR) {
if (errno != ENOENT) {
@@ -2366,11 +2431,46 @@ sds ACLLoadFromFile(const char *filename) {
ACLFreeUser(new_default);
raxInsert(Users,(unsigned char*)"default",7,DefaultUser,NULL);
raxRemove(old_users,(unsigned char*)"default",7,NULL);
ACLFreeUsersSet(old_users);
/* If there are some subscribers, we need to check if we need to drop some clients. */
rax *user_channels = NULL;
if (pubsubTotalSubscriptions() > 0) {
user_channels = raxNew();
}
listIter li;
listNode *ln;
listRewind(server.clients,&li);
while ((ln = listNext(&li)) != NULL) {
client *c = listNodeValue(ln);
/* a MASTER client can do everything (and user = NULL) so we can skip it */
if (c->flags & CLIENT_MASTER)
continue;
user *original = c->user;
list *channels = NULL;
user *new = ACLGetUserByName(c->user->name, sdslen(c->user->name));
if (new && user_channels) {
if (!raxFind(user_channels, (unsigned char*)(new->name), sdslen(new->name), (void**)&channels)) {
channels = getUpcomingChannelList(new, original);
raxInsert(user_channels, (unsigned char*)(new->name), sdslen(new->name), channels, NULL);
}
}
/* When the new channel list is NULL, it means the new user's channel list is a superset of the old user's list. */
if (!new || (channels && ACLShouldKillPubsubClient(c, channels))) {
deauthenticateAndCloseClient(c);
continue;
}
c->user = new;
}
if (user_channels)
raxFreeWithCallback(user_channels, listReleaseGeneric);
raxFreeWithCallback(old_users, ACLFreeUserGeneric);
sdsfree(errors);
return NULL;
} else {
ACLFreeUsersSet(Users);
raxFreeWithCallback(Users, ACLFreeUserGeneric);
Users = old_users;
errors = sdscat(errors,"WARNING: ACL errors detected, no change to the previously active ACL rules was performed");
return errors;
@@ -2548,6 +2648,15 @@ void ACLUpdateInfoMetrics(int reason){
}
}
static void trimACLLogEntriesToMaxLen(void) {
while(listLength(ACLLog) > server.acllog_max_len) {
listNode *ln = listLast(ACLLog);
ACLLogEntry *le = listNodeValue(ln);
ACLFreeLogEntry(le);
listDelNode(ACLLog,ln);
}
}
/* Adds a new entry in the ACL log, making sure to delete the old entry
* if we reach the maximum length allowed for the log. This function attempts
* to find similar entries in the current log in order to bump the counter of
@@ -2567,6 +2676,11 @@ void addACLLogEntry(client *c, int reason, int context, int argpos, sds username
/* Update ACL info metrics */
ACLUpdateInfoMetrics(reason);
if (server.acllog_max_len == 0) {
trimACLLogEntriesToMaxLen();
return;
}
/* Create a new entry. */
struct ACLLogEntry *le = zmalloc(sizeof(*le));
le->count = 1;
@@ -2629,12 +2743,7 @@ void addACLLogEntry(client *c, int reason, int context, int argpos, sds username
* to its maximum size. */
ACLLogEntryCount++; /* Incrementing the entry_id count to make each record in the log unique. */
listAddNodeHead(ACLLog, le);
while(listLength(ACLLog) > server.acllog_max_len) {
listNode *ln = listLast(ACLLog);
ACLLogEntry *le = listNodeValue(ln);
ACLFreeLogEntry(le);
listDelNode(ACLLog,ln);
}
trimACLLogEntriesToMaxLen();
}
}
@@ -2672,7 +2781,6 @@ void aclCatWithFlags(client *c, dict *commands, uint64_t cflag, int *arraylen) {
while ((de = dictNext(di)) != NULL) {
struct redisCommand *cmd = dictGetVal(de);
if (cmd->flags & CMD_MODULE) continue;
if (cmd->acl_categories & cflag) {
addReplyBulkCBuffer(c, cmd->fullname, sdslen(cmd->fullname));
(*arraylen)++;
@@ -2758,8 +2866,7 @@ void aclCommand(client *c) {
sds username = c->argv[2]->ptr;
/* Check username validity. */
if (ACLStringHasSpaces(username,sdslen(username))) {
addReplyErrorFormat(c,
"Usernames can't contain spaces or null characters");
addReplyError(c, "Usernames can't contain spaces or null characters");
return;
}
@@ -2777,6 +2884,10 @@ void aclCommand(client *c) {
}
return;
} else if (!strcasecmp(sub,"deluser") && c->argc >= 3) {
/* Initially redact all the arguments to not leak any information
* about the users. */
for (int j = 2; j < c->argc; j++) redactClientCommandArgument(c, j);
int deleted = 0;
for (int j = 2; j < c->argc; j++) {
sds username = c->argv[j]->ptr;
@@ -2799,6 +2910,9 @@ void aclCommand(client *c) {
}
addReplyLongLong(c,deleted);
} else if (!strcasecmp(sub,"getuser") && c->argc == 3) {
/* Redact the username to not leak any information about the user. */
redactClientCommandArgument(c, 2);
user *u = ACLGetUserByName(c->argv[2]->ptr,sdslen(c->argv[2]->ptr));
if (u == NULL) {
addReplyNull(c);
@@ -3081,6 +3195,38 @@ void addReplyCommandCategories(client *c, struct redisCommand *cmd) {
setDeferredSetLen(c, flaglen, flagcount);
}
/* When successful, initiates an internal connection, that is able to execute
* internal commands (see CMD_INTERNAL). */
static void internalAuth(client *c) {
if (server.cluster == NULL) {
addReplyError(c, "Cannot authenticate as an internal connection on non-cluster instances");
return;
}
sds password = c->argv[2]->ptr;
/* Get internal secret. */
size_t len = -1;
const char *internal_secret = clusterGetSecret(&len);
if (sdslen(password) != len) {
addReplyError(c, "-WRONGPASS invalid internal password");
return;
}
if (!time_independent_strcmp((char *)internal_secret, (char *)password, len)) {
c->flags |= CLIENT_INTERNAL;
/* No further authentication is needed. */
c->authenticated = 1;
/* Set the user to the unrestricted user, if it is not already set (default). */
if (c->user != NULL) {
c->user = NULL;
moduleNotifyUserChanged(c);
}
addReply(c, shared.ok);
} else {
addReplyError(c, "-WRONGPASS invalid internal password");
}
}
/* AUTH <password>
* AUTH <username> <password> (Redis >= 6.0 form)
*
@@ -3114,6 +3260,14 @@ void authCommand(client *c) {
username = c->argv[1];
password = c->argv[2];
redactClientCommandArgument(c, 2);
/* Handle internal authentication commands.
* Note: No user-defined ACL user can have this username (no spaces
* allowed), thus no conflicts with ACL possible. */
if (!strcmp(username->ptr, "internal connection")) {
internalAuth(c);
return;
}
}
robj *err = NULL;
+10 -24
View File
@@ -1,31 +1,10 @@
/* adlist.c - A generic doubly linked list implementation
*
* Copyright (c) 2006-2010, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-Present, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
* Licensed under your choice of the Redis Source Available License 2.0
* (RSALv2) or the Server Side Public License v1 (SSPLv1).
*/
@@ -76,10 +55,17 @@ void listEmpty(list *list)
* This function can't fail. */
void listRelease(list *list)
{
if (!list)
return;
listEmpty(list);
zfree(list);
}
/* Generic version of listRelease. */
void listReleaseGeneric(void *list) {
listRelease((struct list*)list);
}
/* Add a new node to the list, to head, containing the specified 'value'
* pointer as value.
*
+4 -24
View File
@@ -1,31 +1,10 @@
/* adlist.h - A generic doubly linked list implementation
*
* Copyright (c) 2006-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-Present, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
* Licensed under your choice of the Redis Source Available License 2.0
* (RSALv2) or the Server Side Public License v1 (SSPLv1).
*/
#ifndef __ADLIST_H__
@@ -72,6 +51,7 @@ typedef struct list {
/* Prototypes */
list *listCreate(void);
void listRelease(list *list);
void listReleaseGeneric(void *list);
void listEmpty(list *list);
list *listAddNodeHead(list *list, void *value);
list *listAddNodeTail(list *list, void *value);
+37 -39
View File
@@ -2,32 +2,11 @@
* for the Jim's event-loop (Jim is a Tcl interpreter) but later translated
* it in form of a library for easy reuse.
*
* Copyright (c) 2006-2010, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-Present, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
* Licensed under your choice of the Redis Source Available License 2.0
* (RSALv2) or the Server Side Public License v1 (SSPLv1).
*/
#include "ae.h"
@@ -63,7 +42,7 @@
#endif
#endif
#define INITIAL_EVENT 1024
aeEventLoop *aeCreateEventLoop(int setsize) {
aeEventLoop *eventLoop;
int i;
@@ -71,8 +50,9 @@ aeEventLoop *aeCreateEventLoop(int setsize) {
monotonicInit(); /* just in case the calling app didn't initialize */
if ((eventLoop = zmalloc(sizeof(*eventLoop))) == NULL) goto err;
eventLoop->events = zmalloc(sizeof(aeFileEvent)*setsize);
eventLoop->fired = zmalloc(sizeof(aeFiredEvent)*setsize);
eventLoop->nevents = setsize < INITIAL_EVENT ? setsize : INITIAL_EVENT;
eventLoop->events = zmalloc(sizeof(aeFileEvent)*eventLoop->nevents);
eventLoop->fired = zmalloc(sizeof(aeFiredEvent)*eventLoop->nevents);
if (eventLoop->events == NULL || eventLoop->fired == NULL) goto err;
eventLoop->setsize = setsize;
eventLoop->timeEventHead = NULL;
@@ -82,10 +62,11 @@ aeEventLoop *aeCreateEventLoop(int setsize) {
eventLoop->beforesleep = NULL;
eventLoop->aftersleep = NULL;
eventLoop->flags = 0;
memset(eventLoop->privdata, 0, sizeof(eventLoop->privdata));
if (aeApiCreate(eventLoop) == -1) goto err;
/* Events with mask == AE_NONE are not set. So let's initialize the
* vector with it. */
for (i = 0; i < setsize; i++)
for (i = 0; i < eventLoop->nevents; i++)
eventLoop->events[i].mask = AE_NONE;
return eventLoop;
@@ -123,20 +104,19 @@ void aeSetDontWait(aeEventLoop *eventLoop, int noWait) {
*
* Otherwise AE_OK is returned and the operation is successful. */
int aeResizeSetSize(aeEventLoop *eventLoop, int setsize) {
int i;
if (setsize == eventLoop->setsize) return AE_OK;
if (eventLoop->maxfd >= setsize) return AE_ERR;
if (aeApiResize(eventLoop,setsize) == -1) return AE_ERR;
eventLoop->events = zrealloc(eventLoop->events,sizeof(aeFileEvent)*setsize);
eventLoop->fired = zrealloc(eventLoop->fired,sizeof(aeFiredEvent)*setsize);
eventLoop->setsize = setsize;
/* Make sure that if we created new slots, they are initialized with
* an AE_NONE mask. */
for (i = eventLoop->maxfd+1; i < setsize; i++)
eventLoop->events[i].mask = AE_NONE;
/* If the current allocated space is larger than the requested size,
* we need to shrink it to the requested size. */
if (setsize < eventLoop->nevents) {
eventLoop->events = zrealloc(eventLoop->events,sizeof(aeFileEvent)*setsize);
eventLoop->fired = zrealloc(eventLoop->fired,sizeof(aeFiredEvent)*setsize);
eventLoop->nevents = setsize;
}
return AE_OK;
}
@@ -149,6 +129,8 @@ void aeDeleteEventLoop(aeEventLoop *eventLoop) {
aeTimeEvent *next_te, *te = eventLoop->timeEventHead;
while (te) {
next_te = te->next;
if (te->finalizerProc)
te->finalizerProc(eventLoop, te->clientData);
zfree(te);
te = next_te;
}
@@ -166,6 +148,22 @@ int aeCreateFileEvent(aeEventLoop *eventLoop, int fd, int mask,
errno = ERANGE;
return AE_ERR;
}
/* Resize the events and fired arrays if the file
* descriptor exceeds the current number of events. */
if (unlikely(fd >= eventLoop->nevents)) {
int newnevents = eventLoop->nevents;
newnevents = (newnevents * 2 > fd + 1) ? newnevents * 2 : fd + 1;
newnevents = (newnevents > eventLoop->setsize) ? eventLoop->setsize : newnevents;
eventLoop->events = zrealloc(eventLoop->events, sizeof(aeFileEvent) * newnevents);
eventLoop->fired = zrealloc(eventLoop->fired, sizeof(aeFiredEvent) * newnevents);
/* Initialize new slots with an AE_NONE mask */
for (int i = eventLoop->nevents; i < newnevents; i++)
eventLoop->events[i].mask = AE_NONE;
eventLoop->nevents = newnevents;
}
aeFileEvent *fe = &eventLoop->events[fd];
if (aeApiAddEvent(eventLoop, fd, mask) == -1)
@@ -333,7 +331,7 @@ static int processTimeEvents(aeEventLoop *eventLoop) {
processed++;
now = getMonotonicUs();
if (retval != AE_NOMORE) {
te->when = now + retval * 1000;
te->when = now + (monotime)retval * 1000;
} else {
te->id = AE_DELETED_EVENT_ID;
}
@@ -343,8 +341,8 @@ static int processTimeEvents(aeEventLoop *eventLoop) {
return processed;
}
/* Process every pending time event, then every pending file event
* (that may be registered by time event callbacks just processed).
/* Process every pending file event, then every pending time event
* (that may be registered by file event callbacks just processed).
* Without special flags the function sleeps until some file event
* fires, or when the next time event occurs (if any).
*
+5 -24
View File
@@ -2,32 +2,11 @@
* for the Jim's event-loop (Jim is a Tcl interpreter) but later translated
* it in form of a library for easy reuse.
*
* Copyright (c) 2006-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-Present, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
* Licensed under your choice of the Redis Source Available License 2.0
* (RSALv2) or the Server Side Public License v1 (SSPLv1).
*/
#ifndef __AE_H__
@@ -100,6 +79,7 @@ typedef struct aeEventLoop {
int maxfd; /* highest file descriptor currently registered */
int setsize; /* max number of file descriptors tracked */
long long timeEventNextId;
int nevents; /* Size of Registered events */
aeFileEvent *events; /* Registered events */
aeFiredEvent *fired; /* Fired events */
aeTimeEvent *timeEventHead;
@@ -108,6 +88,7 @@ typedef struct aeEventLoop {
aeBeforeSleepProc *beforesleep;
aeBeforeSleepProc *aftersleep;
int flags;
void *privdata[2];
} aeEventLoop;
/* Prototypes */
+3 -24
View File
@@ -1,31 +1,10 @@
/* Linux epoll(2) based ae.c module
*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-Present, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
* Licensed under your choice of the Redis Source Available License 2.0
* (RSALv2) or the Server Side Public License v1 (SSPLv1).
*/
+3 -24
View File
@@ -1,31 +1,10 @@
/* Select()-based ae.c module.
*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-Present, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
* Licensed under your choice of the Redis Source Available License 2.0
* (RSALv2) or the Server Side Public License v1 (SSPLv1).
*/
+146 -63
View File
@@ -1,31 +1,10 @@
/* anet.c -- Basic TCP socket stuff made a bit less boring
*
* Copyright (c) 2006-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-Present, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
* Licensed under your choice of the Redis Source Available License 2.0
* (RSALv2) or the Server Side Public License v1 (SSPLv1).
*/
#include "fmacros.h"
@@ -82,7 +61,7 @@ int anetSetBlock(char *err, int fd, int non_block) {
return ANET_ERR;
}
/* Check if this flag has been set or unset, if so,
/* Check if this flag has been set or unset, if so,
* then there is no need to call fcntl to set/unset it again. */
if (!!(flags & O_NONBLOCK) == !!non_block)
return ANET_OK;
@@ -107,8 +86,8 @@ int anetBlock(char *err, int fd) {
return anetSetBlock(err,fd,0);
}
/* Enable the FD_CLOEXEC on the given fd to avoid fd leaks.
* This function should be invoked for fd's on specific places
/* Enable the FD_CLOEXEC on the given fd to avoid fd leaks.
* This function should be invoked for fd's on specific places
* where fork + execve system calls are called. */
int anetCloexec(int fd) {
int r;
@@ -130,57 +109,145 @@ int anetCloexec(int fd) {
return r;
}
/* Set TCP keep alive option to detect dead peers. The interval option
* is only used for Linux as we are using Linux-specific APIs to set
* the probe send time, interval, and count. */
/* Enable TCP keep-alive mechanism to detect dead peers,
* TCP_KEEPIDLE, TCP_KEEPINTVL and TCP_KEEPCNT will be set accordingly. */
int anetKeepAlive(char *err, int fd, int interval)
{
int val = 1;
if (setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &val, sizeof(val)) == -1)
int enabled = 1;
if (setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &enabled, sizeof(enabled)))
{
anetSetError(err, "setsockopt SO_KEEPALIVE: %s", strerror(errno));
return ANET_ERR;
}
#ifdef __linux__
/* Default settings are more or less garbage, with the keepalive time
* set to 7200 by default on Linux. Modify settings to make the feature
* actually useful. */
int idle;
int intvl;
int cnt;
/* Send first probe after interval. */
val = interval;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE, &val, sizeof(val)) < 0) {
/* There are platforms that are expected to support the full mechanism of TCP keep-alive,
* we want the compiler to emit warnings of unused variables if the preprocessor directives
* somehow fail, and other than those platforms, just omit these warnings if they happen.
*/
#if !(defined(_AIX) || defined(__APPLE__) || defined(__DragonFly__) || \
defined(__FreeBSD__) || defined(__illumos__) || defined(__linux__) || \
defined(__NetBSD__) || defined(__sun))
UNUSED(interval);
UNUSED(idle);
UNUSED(intvl);
UNUSED(cnt);
#endif
#ifdef __sun
/* The implementation of TCP keep-alive on Solaris/SmartOS is a bit unusual
* compared to other Unix-like systems.
* Thus, we need to specialize it on Solaris.
*
* There are two keep-alive mechanisms on Solaris:
* - By default, the first keep-alive probe is sent out after a TCP connection is idle for two hours.
* If the peer does not respond to the probe within eight minutes, the TCP connection is aborted.
* You can alter the interval for sending out the first probe using the socket option TCP_KEEPALIVE_THRESHOLD
* in milliseconds or TCP_KEEPIDLE in seconds.
* The system default is controlled by the TCP ndd parameter tcp_keepalive_interval. The minimum value is ten seconds.
* The maximum is ten days, while the default is two hours. If you receive no response to the probe,
* you can use the TCP_KEEPALIVE_ABORT_THRESHOLD socket option to change the time threshold for aborting a TCP connection.
* The option value is an unsigned integer in milliseconds. The value zero indicates that TCP should never time out and
* abort the connection when probing. The system default is controlled by the TCP ndd parameter tcp_keepalive_abort_interval.
* The default is eight minutes.
*
* - The second implementation is activated if socket option TCP_KEEPINTVL and/or TCP_KEEPCNT are set.
* The time between each consequent probes is set by TCP_KEEPINTVL in seconds.
* The minimum value is ten seconds. The maximum is ten days, while the default is two hours.
* The TCP connection will be aborted after certain amount of probes, which is set by TCP_KEEPCNT, without receiving response.
*/
idle = interval;
if (idle < 10) idle = 10; // kernel expects at least 10 seconds
if (idle > 10*24*60*60) idle = 10*24*60*60; // kernel expects at most 10 days
/* `TCP_KEEPIDLE`, `TCP_KEEPINTVL`, and `TCP_KEEPCNT` were not available on Solaris
* until version 11.4, but let's take a chance here. */
#if defined(TCP_KEEPIDLE) && defined(TCP_KEEPINTVL) && defined(TCP_KEEPCNT)
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE, &idle, sizeof(idle))) {
anetSetError(err, "setsockopt TCP_KEEPIDLE: %s\n", strerror(errno));
return ANET_ERR;
}
/* Send next probes after the specified interval. Note that we set the
* delay as interval / 3, as we send three probes before detecting
* an error (see the next setsockopt call). */
val = interval/3;
if (val == 0) val = 1;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPINTVL, &val, sizeof(val)) < 0) {
intvl = idle/3;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPINTVL, &intvl, sizeof(intvl))) {
anetSetError(err, "setsockopt TCP_KEEPINTVL: %s\n", strerror(errno));
return ANET_ERR;
}
/* Consider the socket in error state after three we send three ACK
* probes without getting a reply. */
val = 3;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPCNT, &val, sizeof(val)) < 0) {
cnt = 3;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPCNT, &cnt, sizeof(cnt))) {
anetSetError(err, "setsockopt TCP_KEEPCNT: %s\n", strerror(errno));
return ANET_ERR;
}
#elif defined(__APPLE__)
/* Set idle time with interval */
val = interval;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPALIVE, &val, sizeof(val)) < 0) {
#else
/* Fall back to the first implementation of tcp-alive mechanism for older Solaris,
* simulate the tcp-alive mechanism on other platforms via `TCP_KEEPALIVE_THRESHOLD` + `TCP_KEEPALIVE_ABORT_THRESHOLD`.
*/
idle *= 1000; // kernel expects milliseconds
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPALIVE_THRESHOLD, &idle, sizeof(idle))) {
anetSetError(err, "setsockopt TCP_KEEPINTVL: %s\n", strerror(errno));
return ANET_ERR;
}
/* Note that the consequent probes will not be sent at equal intervals on Solaris,
* but will be sent using the exponential backoff algorithm. */
intvl = idle/3;
cnt = 3;
int time_to_abort = intvl * cnt;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPALIVE_ABORT_THRESHOLD, &time_to_abort, sizeof(time_to_abort))) {
anetSetError(err, "setsockopt TCP_KEEPCNT: %s\n", strerror(errno));
return ANET_ERR;
}
#endif
return ANET_OK;
#endif
#ifdef TCP_KEEPIDLE
/* Default settings are more or less garbage, with the keepalive time
* set to 7200 by default on Linux and other Unix-like systems.
* Modify settings to make the feature actually useful. */
/* Send first probe after interval. */
idle = interval;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE, &idle, sizeof(idle))) {
anetSetError(err, "setsockopt TCP_KEEPIDLE: %s\n", strerror(errno));
return ANET_ERR;
}
#elif defined(TCP_KEEPALIVE)
/* Darwin/macOS uses TCP_KEEPALIVE in place of TCP_KEEPIDLE. */
idle = interval;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPALIVE, &idle, sizeof(idle))) {
anetSetError(err, "setsockopt TCP_KEEPALIVE: %s\n", strerror(errno));
return ANET_ERR;
}
#else
((void) interval); /* Avoid unused var warning for non Linux systems. */
#endif
#ifdef TCP_KEEPINTVL
/* Send next probes after the specified interval. Note that we set the
* delay as interval / 3, as we send three probes before detecting
* an error (see the next setsockopt call). */
intvl = interval/3;
if (intvl == 0) intvl = 1;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPINTVL, &intvl, sizeof(intvl))) {
anetSetError(err, "setsockopt TCP_KEEPINTVL: %s\n", strerror(errno));
return ANET_ERR;
}
#endif
#ifdef TCP_KEEPCNT
/* Consider the socket in error state after three we send three ACK
* probes without getting a reply. */
cnt = 3;
if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPCNT, &cnt, sizeof(cnt))) {
anetSetError(err, "setsockopt TCP_KEEPCNT: %s\n", strerror(errno));
return ANET_ERR;
}
#endif
return ANET_OK;
@@ -239,7 +306,11 @@ int anetRecvTimeout(char *err, int fd, long long ms) {
*
* If flags is set to ANET_IP_ONLY the function only resolves hostnames
* that are actually already IPv4 or IPv6 addresses. This turns the function
* into a validating / normalizing function. */
* into a validating / normalizing function.
*
* If the flag ANET_PREFER_IPV4 is set, IPv4 is preferred over IPv6.
* If the flag ANET_PREFER_IPV6 is set, IPv6 is preferred over IPv4.
* */
int anetResolve(char *err, char *host, char *ipbuf, size_t ipbuf_len,
int flags)
{
@@ -249,9 +320,20 @@ int anetResolve(char *err, char *host, char *ipbuf, size_t ipbuf_len,
memset(&hints,0,sizeof(hints));
if (flags & ANET_IP_ONLY) hints.ai_flags = AI_NUMERICHOST;
hints.ai_family = AF_UNSPEC;
if (flags & ANET_PREFER_IPV4 && !(flags & ANET_PREFER_IPV6)) {
hints.ai_family = AF_INET;
} else if (flags & ANET_PREFER_IPV6 && !(flags & ANET_PREFER_IPV4)) {
hints.ai_family = AF_INET6;
}
hints.ai_socktype = SOCK_STREAM; /* specify socktype to avoid dups */
if ((rv = getaddrinfo(host, NULL, &hints, &info)) != 0) {
rv = getaddrinfo(host, NULL, &hints, &info);
if (rv != 0 && hints.ai_family != AF_UNSPEC) {
/* Try the other IP version. */
hints.ai_family = (hints.ai_family == AF_INET) ? AF_INET6 : AF_INET;
rv = getaddrinfo(host, NULL, &hints, &info);
}
if (rv != 0) {
anetSetError(err, "%s", gai_strerror(rv));
return ANET_ERR;
}
@@ -417,13 +499,16 @@ int anetUnixGenericConnect(char *err, const char *path, int flags)
return s;
}
static int anetListen(char *err, int s, struct sockaddr *sa, socklen_t len, int backlog) {
static int anetListen(char *err, int s, struct sockaddr *sa, socklen_t len, int backlog, mode_t perm) {
if (bind(s,sa,len) == -1) {
anetSetError(err, "bind: %s", strerror(errno));
close(s);
return ANET_ERR;
}
if (sa->sa_family == AF_LOCAL && perm)
chmod(((struct sockaddr_un *) sa)->sun_path, perm);
if (listen(s, backlog) == -1) {
anetSetError(err, "listen: %s", strerror(errno));
close(s);
@@ -467,7 +552,7 @@ static int _anetTcpServer(char *err, int port, char *bindaddr, int af, int backl
if (af == AF_INET6 && anetV6Only(err,s) == ANET_ERR) goto error;
if (anetSetReuseAddr(err,s) == ANET_ERR) goto error;
if (anetListen(err,s,p->ai_addr,p->ai_addrlen,backlog) == ANET_ERR) s = ANET_ERR;
if (anetListen(err,s,p->ai_addr,p->ai_addrlen,backlog,0) == ANET_ERR) s = ANET_ERR;
goto end;
}
if (p == NULL) {
@@ -508,10 +593,8 @@ int anetUnixServer(char *err, char *path, mode_t perm, int backlog)
memset(&sa,0,sizeof(sa));
sa.sun_family = AF_LOCAL;
redis_strlcpy(sa.sun_path,path,sizeof(sa.sun_path));
if (anetListen(err,s,(struct sockaddr*)&sa,sizeof(sa),backlog) == ANET_ERR)
if (anetListen(err,s,(struct sockaddr*)&sa,sizeof(sa),backlog,perm) == ANET_ERR)
return ANET_ERR;
if (perm)
chmod(sa.sun_path, perm);
return s;
}
+5 -24
View File
@@ -1,31 +1,10 @@
/* anet.c -- Basic TCP socket stuff made a bit less boring
*
* Copyright (c) 2006-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-Present, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
* Licensed under your choice of the Redis Source Available License 2.0
* (RSALv2) or the Server Side Public License v1 (SSPLv1).
*/
#ifndef ANET_H
@@ -40,6 +19,8 @@
/* Flags used with certain functions. */
#define ANET_NONE 0
#define ANET_IP_ONLY (1<<0)
#define ANET_PREFER_IPV4 (1<<1)
#define ANET_PREFER_IPV6 (1<<2)
#if defined(__sun) || defined(_AIX)
#define AF_LOCAL AF_UNIX
+218 -106
View File
@@ -1,30 +1,9 @@
/*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-Present, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
* Licensed under your choice of the Redis Source Available License 2.0
* (RSALv2) or the Server Side Public License v1 (SSPLv1).
*/
#include "server.h"
@@ -51,6 +30,13 @@ aofManifest *aofLoadManifestFromFile(sds am_filepath);
void aofManifestFreeAndUpdate(aofManifest *am);
void aof_background_fsync_and_close(int fd);
/* When we call 'startAppendOnly', we will create a temp INCR AOF, and rename
* it to the real INCR AOF name when the AOFRW is done, so if want to know the
* accurate start offset of the INCR AOF, we need to record it when we create
* the temp INCR AOF. This variable is used to record the start offset, and
* set the start offset of the real INCR AOF when the AOFRW is done. */
static long long tempIncAofStartReplOffset = 0;
/* ----------------------------------------------------------------------------
* AOF Manifest file implementation.
*
@@ -94,10 +80,15 @@ void aof_background_fsync_and_close(int fd);
#define AOF_MANIFEST_KEY_FILE_NAME "file"
#define AOF_MANIFEST_KEY_FILE_SEQ "seq"
#define AOF_MANIFEST_KEY_FILE_TYPE "type"
#define AOF_MANIFEST_KEY_FILE_STARTOFFSET "startoffset"
#define AOF_MANIFEST_KEY_FILE_ENDOFFSET "endoffset"
/* Create an empty aofInfo. */
aofInfo *aofInfoCreate(void) {
return zcalloc(sizeof(aofInfo));
aofInfo *ai = zcalloc(sizeof(aofInfo));
ai->start_offset = -1;
ai->end_offset = -1;
return ai;
}
/* Free the aofInfo structure (pointed to by ai) and its embedded file_name. */
@@ -114,20 +105,33 @@ aofInfo *aofInfoDup(aofInfo *orig) {
ai->file_name = sdsdup(orig->file_name);
ai->file_seq = orig->file_seq;
ai->file_type = orig->file_type;
ai->start_offset = orig->start_offset;
ai->end_offset = orig->end_offset;
return ai;
}
/* Format aofInfo as a string and it will be a line in the manifest. */
/* Format aofInfo as a string and it will be a line in the manifest.
*
* When update this format, make sure to update redis-check-aof as well. */
sds aofInfoFormat(sds buf, aofInfo *ai) {
sds filename_repr = NULL;
if (sdsneedsrepr(ai->file_name))
filename_repr = sdscatrepr(sdsempty(), ai->file_name, sdslen(ai->file_name));
sds ret = sdscatprintf(buf, "%s %s %s %lld %s %c\n",
sds ret = sdscatprintf(buf, "%s %s %s %lld %s %c",
AOF_MANIFEST_KEY_FILE_NAME, filename_repr ? filename_repr : ai->file_name,
AOF_MANIFEST_KEY_FILE_SEQ, ai->file_seq,
AOF_MANIFEST_KEY_FILE_TYPE, ai->file_type);
if (ai->start_offset != -1) {
ret = sdscatprintf(ret, " %s %lld", AOF_MANIFEST_KEY_FILE_STARTOFFSET, ai->start_offset);
if (ai->end_offset != -1) {
ret = sdscatprintf(ret, " %s %lld", AOF_MANIFEST_KEY_FILE_ENDOFFSET, ai->end_offset);
}
}
ret = sdscatlen(ret, "\n", 1);
sdsfree(filename_repr);
return ret;
@@ -323,6 +327,10 @@ aofManifest *aofLoadManifestFromFile(sds am_filepath) {
ai->file_seq = atoll(argv[i+1]);
} else if (!strcasecmp(argv[i], AOF_MANIFEST_KEY_FILE_TYPE)) {
ai->file_type = (argv[i+1])[0];
} else if (!strcasecmp(argv[i], AOF_MANIFEST_KEY_FILE_STARTOFFSET)) {
ai->start_offset = atoll(argv[i+1]);
} else if (!strcasecmp(argv[i], AOF_MANIFEST_KEY_FILE_ENDOFFSET)) {
ai->end_offset = atoll(argv[i+1]);
}
/* else if (!strcasecmp(argv[i], AOF_MANIFEST_KEY_OTHER)) {} */
}
@@ -452,12 +460,13 @@ sds getNewBaseFileNameAndMarkPreAsHistory(aofManifest *am) {
* for example:
* appendonly.aof.1.incr.aof
*/
sds getNewIncrAofName(aofManifest *am) {
sds getNewIncrAofName(aofManifest *am, long long start_reploff) {
aofInfo *ai = aofInfoCreate();
ai->file_type = AOF_FILE_TYPE_INCR;
ai->file_name = sdscatprintf(sdsempty(), "%s.%lld%s%s", server.aof_filename,
++am->curr_incr_file_seq, INCR_FILE_SUFFIX, AOF_FORMAT_SUFFIX);
ai->file_seq = am->curr_incr_file_seq;
ai->start_offset = start_reploff;
listAddNodeTail(am->incr_aof_list, ai);
am->dirty = 1;
return ai->file_name;
@@ -475,7 +484,7 @@ sds getLastIncrAofName(aofManifest *am) {
/* If 'incr_aof_list' is empty, just create a new one. */
if (!listLength(am->incr_aof_list)) {
return getNewIncrAofName(am);
return getNewIncrAofName(am, server.master_repl_offset);
}
/* Or return the last one. */
@@ -800,10 +809,11 @@ int openNewIncrAofForAppend(void) {
if (server.aof_state == AOF_WAIT_REWRITE) {
/* Use a temporary INCR AOF file to accumulate data during AOF_WAIT_REWRITE. */
new_aof_name = getTempIncrAofName();
tempIncAofStartReplOffset = server.master_repl_offset;
} else {
/* Dup a temp aof_manifest to modify. */
temp_am = aofManifestDup(server.aof_manifest);
new_aof_name = sdsdup(getNewIncrAofName(temp_am));
new_aof_name = sdsdup(getNewIncrAofName(temp_am, server.master_repl_offset));
}
sds new_aof_filepath = makePath(server.aof_dirname, new_aof_name);
newfd = open(new_aof_filepath, O_WRONLY|O_TRUNC|O_CREAT, 0644);
@@ -833,7 +843,7 @@ int openNewIncrAofForAppend(void) {
* is already synced at this point so fsync doesn't matter. */
if (server.aof_fd != -1) {
aof_background_fsync_and_close(server.aof_fd);
server.aof_last_fsync = server.unixtime;
server.aof_last_fsync = server.mstime;
}
server.aof_fd = newfd;
@@ -852,6 +862,50 @@ cleanup:
return C_ERR;
}
/* When we close gracefully the AOF file, we have the chance to persist the
* end replication offset of current INCR AOF. */
void updateCurIncrAofEndOffset(void) {
if (server.aof_state != AOF_ON) return;
serverAssert(server.aof_manifest != NULL);
if (listLength(server.aof_manifest->incr_aof_list) == 0) return;
aofInfo *ai = listNodeValue(listLast(server.aof_manifest->incr_aof_list));
ai->end_offset = server.master_repl_offset;
server.aof_manifest->dirty = 1;
/* It doesn't matter if the persistence fails since this information is not
* critical, we can get an approximate value by start offset plus file size. */
persistAofManifest(server.aof_manifest);
}
/* After loading AOF data, we need to update the `server.master_repl_offset`
* based on the information of the last INCR AOF, to avoid the rollback of
* the start offset of new INCR AOF. */
void updateReplOffsetAndResetEndOffset(void) {
if (server.aof_state != AOF_ON) return;
serverAssert(server.aof_manifest != NULL);
/* If the INCR file has an end offset, we directly use it, and clear it
* to avoid the next time we load the manifest file, we will use the same
* offset, but the real offset may have advanced. */
if (listLength(server.aof_manifest->incr_aof_list) == 0) return;
aofInfo *ai = listNodeValue(listLast(server.aof_manifest->incr_aof_list));
if (ai->end_offset != -1) {
server.master_repl_offset = ai->end_offset;
ai->end_offset = -1;
server.aof_manifest->dirty = 1;
/* We must update the end offset of INCR file correctly, otherwise we
* may keep wrong information in the manifest file, since we continue
* to append data to the same INCR file. */
if (persistAofManifest(server.aof_manifest) != AOF_OK)
exit(1);
} else {
/* If the INCR file doesn't have an end offset, we need to calculate
* the replication offset by the start offset plus the file size. */
server.master_repl_offset = (ai->start_offset == -1 ? 0 : ai->start_offset) +
getAppendOnlyFileSize(ai->file_name, NULL);
}
}
/* Whether to limit the execution of Background AOF rewrite.
*
* At present, if AOFRW fails, redis will automatically retry. If it continues
@@ -954,9 +1008,10 @@ void stopAppendOnly(void) {
if (redis_fsync(server.aof_fd) == -1) {
serverLog(LL_WARNING,"Fail to fsync the AOF file: %s",strerror(errno));
} else {
server.aof_last_fsync = server.unixtime;
server.aof_last_fsync = server.mstime;
}
close(server.aof_fd);
updateCurIncrAofEndOffset();
server.aof_fd = -1;
server.aof_selected_db = -1;
@@ -976,18 +1031,6 @@ void stopAppendOnly(void) {
int startAppendOnly(void) {
serverAssert(server.aof_state == AOF_OFF);
/* Wait for all bio jobs related to AOF to drain. This prevents a race
* between updates to `fsynced_reploff_pending` of the worker thread, belonging
* to the previous AOF, and the new one. This concern is specific for a full
* sync scenario where we don't wanna risk the ACKed replication offset
* jumping backwards or forward when switching to a different master. */
bioDrainWorker(BIO_AOF_FSYNC);
/* Set the initial repl_offset, which will be applied to fsynced_reploff
* when AOFRW finishes (after possibly being updated by a bio thread) */
atomicSet(server.fsynced_reploff_pending, server.master_repl_offset);
server.fsynced_reploff = 0;
server.aof_state = AOF_WAIT_REWRITE;
if (hasActiveChildProcess() && server.child_type != CHILD_TYPE_AOF) {
server.aof_rewrite_scheduled = 1;
@@ -1010,7 +1053,7 @@ int startAppendOnly(void) {
return C_ERR;
}
}
server.aof_last_fsync = server.unixtime;
server.aof_last_fsync = server.mstime;
/* If AOF fsync error in bio job, we just ignore it and log the event. */
int aof_bio_fsync_status;
atomicGet(server.aof_bio_fsync_status, aof_bio_fsync_status);
@@ -1028,6 +1071,29 @@ int startAppendOnly(void) {
return C_OK;
}
void startAppendOnlyWithRetry(void) {
unsigned int tries, max_tries = 10;
for (tries = 0; tries < max_tries; ++tries) {
if (startAppendOnly() == C_OK)
break;
serverLog(LL_WARNING, "Failed to enable AOF! Trying it again in one second.");
sleep(1);
}
if (tries == max_tries) {
serverLog(LL_WARNING, "FATAL: AOF can't be turned on. Exiting now.");
exit(1);
}
}
/* Called after "appendonly" config is changed. */
void applyAppendOnlyConfig(void) {
if (!server.aof_enabled && server.aof_state != AOF_OFF) {
stopAppendOnly();
} else if (server.aof_enabled && server.aof_state == AOF_OFF) {
startAppendOnlyWithRetry();
}
}
/* This is a wrapper to the write syscall in order to retry on short writes
* or if the syscall gets interrupted. It could look strange that we retry
* on short writes given that we are writing to a block device: normally if
@@ -1079,28 +1145,34 @@ void flushAppendOnlyFile(int force) {
mstime_t latency;
if (sdslen(server.aof_buf) == 0) {
/* Check if we need to do fsync even the aof buffer is empty,
* because previously in AOF_FSYNC_EVERYSEC mode, fsync is
* called only when aof buffer is not empty, so if users
* stop write commands before fsync called in one second,
* the data in page cache cannot be flushed in time. */
if (server.aof_fsync == AOF_FSYNC_EVERYSEC &&
server.aof_last_incr_fsync_offset != server.aof_last_incr_size &&
server.unixtime > server.aof_last_fsync &&
!(sync_in_progress = aofFsyncInProgress())) {
goto try_fsync;
/* Check if we need to do fsync even the aof buffer is empty,
* the reason is described in the previous AOF_FSYNC_EVERYSEC block,
* and AOF_FSYNC_ALWAYS is also checked here to handle a case where
* aof_fsync is changed from everysec to always. */
} else if (server.aof_fsync == AOF_FSYNC_ALWAYS &&
server.aof_last_incr_fsync_offset != server.aof_last_incr_size)
{
goto try_fsync;
if (server.aof_last_incr_fsync_offset == server.aof_last_incr_size) {
/* All data is fsync'd already: Update fsynced_reploff_pending just in case.
* This is needed to avoid a WAITAOF hang in case a module used RM_Call
* with the NO_AOF flag, in which case master_repl_offset will increase but
* fsynced_reploff_pending won't be updated (because there's no reason, from
* the AOF POV, to call fsync) and then WAITAOF may wait on the higher offset
* (which contains data that was only propagated to replicas, and not to AOF) */
if (!aofFsyncInProgress())
atomicSet(server.fsynced_reploff_pending, server.master_repl_offset);
} else {
return;
/* Check if we need to do fsync even the aof buffer is empty,
* because previously in AOF_FSYNC_EVERYSEC mode, fsync is
* called only when aof buffer is not empty, so if users
* stop write commands before fsync called in one second,
* the data in page cache cannot be flushed in time. */
if (server.aof_fsync == AOF_FSYNC_EVERYSEC &&
server.mstime - server.aof_last_fsync >= 1000 &&
!(sync_in_progress = aofFsyncInProgress()))
goto try_fsync;
/* Check if we need to do fsync even the aof buffer is empty,
* the reason is described in the previous AOF_FSYNC_EVERYSEC block,
* and AOF_FSYNC_ALWAYS is also checked here to handle a case where
* aof_fsync is changed from everysec to always. */
if (server.aof_fsync == AOF_FSYNC_ALWAYS)
goto try_fsync;
}
return;
}
if (server.aof_fsync == AOF_FSYNC_EVERYSEC)
@@ -1114,9 +1186,9 @@ void flushAppendOnlyFile(int force) {
if (server.aof_flush_postponed_start == 0) {
/* No previous write postponing, remember that we are
* postponing the flush and return. */
server.aof_flush_postponed_start = server.unixtime;
server.aof_flush_postponed_start = server.mstime;
return;
} else if (server.unixtime - server.aof_flush_postponed_start < 2) {
} else if (server.mstime - server.aof_flush_postponed_start < 2000) {
/* We were already waiting for fsync to finish, but for less
* than two seconds this is still ok. Postpone again. */
return;
@@ -1265,15 +1337,15 @@ try_fsync:
latencyEndMonitor(latency);
latencyAddSampleIfNeeded("aof-fsync-always",latency);
server.aof_last_incr_fsync_offset = server.aof_last_incr_size;
server.aof_last_fsync = server.unixtime;
server.aof_last_fsync = server.mstime;
atomicSet(server.fsynced_reploff_pending, server.master_repl_offset);
} else if (server.aof_fsync == AOF_FSYNC_EVERYSEC &&
server.unixtime > server.aof_last_fsync) {
server.mstime - server.aof_last_fsync >= 1000) {
if (!sync_in_progress) {
aof_background_fsync(server.aof_fd);
server.aof_last_incr_fsync_offset = server.aof_last_incr_size;
}
server.aof_last_fsync = server.unixtime;
server.aof_last_fsync = server.mstime;
}
}
@@ -1859,6 +1931,7 @@ int rewriteSetObject(rio *r, robj *key, robj *o) {
!rioWriteBulkString(r,"SADD",4) ||
!rioWriteBulkObject(r,key))
{
setTypeReleaseIterator(si);
return 0;
}
}
@@ -1962,19 +2035,21 @@ int rewriteSortedSetObject(rio *r, robj *key, robj *o) {
*
* The function returns 0 on error, non-zero on success. */
static int rioWriteHashIteratorCursor(rio *r, hashTypeIterator *hi, int what) {
if (hi->encoding == OBJ_ENCODING_LISTPACK) {
if ((hi->encoding == OBJ_ENCODING_LISTPACK) || (hi->encoding == OBJ_ENCODING_LISTPACK_EX)) {
unsigned char *vstr = NULL;
unsigned int vlen = UINT_MAX;
long long vll = LLONG_MAX;
hashTypeCurrentFromListpack(hi, what, &vstr, &vlen, &vll);
hashTypeCurrentFromListpack(hi, what, &vstr, &vlen, &vll, NULL);
if (vstr)
return rioWriteBulkString(r, (char*)vstr, vlen);
else
return rioWriteBulkLongLong(r, vll);
} else if (hi->encoding == OBJ_ENCODING_HT) {
sds value = hashTypeCurrentFromHashTable(hi, what);
return rioWriteBulkString(r, value, sdslen(value));
char *str;
size_t len;
hashTypeCurrentFromHashTable(hi, what, &str, &len, NULL);
return rioWriteBulkString(r, str, len);
}
serverPanic("Unknown hash encoding");
@@ -1984,37 +2059,60 @@ static int rioWriteHashIteratorCursor(rio *r, hashTypeIterator *hi, int what) {
/* Emit the commands needed to rebuild a hash object.
* The function returns 0 on error, 1 on success. */
int rewriteHashObject(rio *r, robj *key, robj *o) {
int res = 0; /*fail*/
hashTypeIterator *hi;
long long count = 0, items = hashTypeLength(o);
long long count = 0, items = hashTypeLength(o, 0);
int isHFE = hashTypeGetMinExpire(o, 0) != EB_EXPIRE_TIME_INVALID;
hi = hashTypeInitIterator(o);
while (hashTypeNext(hi) != C_ERR) {
if (count == 0) {
int cmd_items = (items > AOF_REWRITE_ITEMS_PER_CMD) ?
AOF_REWRITE_ITEMS_PER_CMD : items;
if (!rioWriteBulkCount(r,'*',2+cmd_items*2) ||
!rioWriteBulkString(r,"HMSET",5) ||
!rioWriteBulkObject(r,key))
{
hashTypeReleaseIterator(hi);
return 0;
if (!isHFE) {
while (hashTypeNext(hi, 0) != C_ERR) {
if (count == 0) {
int cmd_items = (items > AOF_REWRITE_ITEMS_PER_CMD) ?
AOF_REWRITE_ITEMS_PER_CMD : items;
if (!rioWriteBulkCount(r, '*', 2 + cmd_items * 2) ||
!rioWriteBulkString(r, "HMSET", 5) ||
!rioWriteBulkObject(r, key))
goto reHashEnd;
}
if (!rioWriteHashIteratorCursor(r, hi, OBJ_HASH_KEY) ||
!rioWriteHashIteratorCursor(r, hi, OBJ_HASH_VALUE))
goto reHashEnd;
if (++count == AOF_REWRITE_ITEMS_PER_CMD) count = 0;
items--;
}
} else {
while (hashTypeNext(hi, 0) != C_ERR) {
char hmsetCmd[] = "*4\r\n$5\r\nHMSET\r\n";
if ( (!rioWrite(r, hmsetCmd, sizeof(hmsetCmd) - 1)) ||
(!rioWriteBulkObject(r, key)) ||
(!rioWriteHashIteratorCursor(r, hi, OBJ_HASH_KEY)) ||
(!rioWriteHashIteratorCursor(r, hi, OBJ_HASH_VALUE)) )
goto reHashEnd;
if (hi->expire_time != EB_EXPIRE_TIME_INVALID) {
char cmd[] = "*6\r\n$10\r\nHPEXPIREAT\r\n";
if ( (!rioWrite(r, cmd, sizeof(cmd) - 1)) ||
(!rioWriteBulkObject(r, key)) ||
(!rioWriteBulkLongLong(r, hi->expire_time)) ||
(!rioWriteBulkString(r, "FIELDS", 6)) ||
(!rioWriteBulkString(r, "1", 1)) ||
(!rioWriteHashIteratorCursor(r, hi, OBJ_HASH_KEY)) )
goto reHashEnd;
}
}
if (!rioWriteHashIteratorCursor(r, hi, OBJ_HASH_KEY) ||
!rioWriteHashIteratorCursor(r, hi, OBJ_HASH_VALUE))
{
hashTypeReleaseIterator(hi);
return 0;
}
if (++count == AOF_REWRITE_ITEMS_PER_CMD) count = 0;
items--;
}
hashTypeReleaseIterator(hi);
res = 1; /* success */
return 1;
reHashEnd:
hashTypeReleaseIterator(hi);
return res;
}
/* Helper for rewriteStreamObject() that generates a bulk string into the
@@ -2245,11 +2343,11 @@ werr:
}
int rewriteAppendOnlyFileRio(rio *aof) {
dictIterator *di = NULL;
dictEntry *de;
int j;
long key_count = 0;
long long updated_time = 0;
kvstoreIterator *kvs_it = NULL;
/* Record timestamp at the beginning of rewriting AOF. */
if (server.aof_timestamp_enabled) {
@@ -2262,17 +2360,16 @@ int rewriteAppendOnlyFileRio(rio *aof) {
for (j = 0; j < server.dbnum; j++) {
char selectcmd[] = "*2\r\n$6\r\nSELECT\r\n";
redisDb *db = server.db+j;
dict *d = db->dict;
if (dictSize(d) == 0) continue;
di = dictGetSafeIterator(d);
redisDb *db = server.db + j;
if (kvstoreSize(db->keys) == 0) continue;
/* SELECT the new DB */
if (rioWrite(aof,selectcmd,sizeof(selectcmd)-1) == 0) goto werr;
if (rioWriteBulkLongLong(aof,j) == 0) goto werr;
kvs_it = kvstoreIteratorInit(db->keys);
/* Iterate this DB writing every entry */
while((de = dictNext(di)) != NULL) {
while((de = kvstoreIteratorNext(kvs_it)) != NULL) {
sds keystr;
robj key, *o;
long long expiretime;
@@ -2337,13 +2434,12 @@ int rewriteAppendOnlyFileRio(rio *aof) {
if (server.rdb_key_save_delay)
debugDelay(server.rdb_key_save_delay);
}
dictReleaseIterator(di);
di = NULL;
kvstoreIteratorRelease(kvs_it);
}
return C_OK;
werr:
if (di) dictReleaseIterator(di);
if (kvs_it) kvstoreIteratorRelease(kvs_it);
return C_ERR;
}
@@ -2454,7 +2550,23 @@ int rewriteAppendOnlyFileBackground(void) {
server.aof_lastbgrewrite_status = C_ERR;
return C_ERR;
}
if (server.aof_state == AOF_WAIT_REWRITE) {
/* Wait for all bio jobs related to AOF to drain. This prevents a race
* between updates to `fsynced_reploff_pending` of the worker thread, belonging
* to the previous AOF, and the new one. This concern is specific for a full
* sync scenario where we don't wanna risk the ACKed replication offset
* jumping backwards or forward when switching to a different master. */
bioDrainWorker(BIO_AOF_FSYNC);
/* Set the initial repl_offset, which will be applied to fsynced_reploff
* when AOFRW finishes (after possibly being updated by a bio thread) */
atomicSet(server.fsynced_reploff_pending, server.master_repl_offset);
server.fsynced_reploff = 0;
}
server.stat_aof_rewrites++;
if ((childpid = redisFork(CHILD_TYPE_AOF)) == 0) {
char tmpfile[256];
@@ -2626,7 +2738,7 @@ void backgroundRewriteDoneHandler(int exitcode, int bysignal) {
sds temp_incr_aof_name = getTempIncrAofName();
sds temp_incr_filepath = makePath(server.aof_dirname, temp_incr_aof_name);
/* Get next new incr aof name. */
sds new_incr_filename = getNewIncrAofName(temp_am);
sds new_incr_filename = getNewIncrAofName(temp_am, tempIncAofStartReplOffset);
new_incr_filepath = makePath(server.aof_dirname, new_incr_filename);
latencyStartMonitor(latency);
if (rename(temp_incr_filepath, new_incr_filepath) == -1) {
+5 -26
View File
@@ -1,37 +1,16 @@
/*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-Present, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
* Licensed under your choice of the Redis Source Available License 2.0
* (RSALv2) or the Server Side Public License v1 (SSPLv1).
*/
const char *ascii_logo =
" _._ \n"
" _.-``__ ''-._ \n"
" _.-`` `. `_. ''-._ Redis %s (%s/%d) %s bit\n"
" .-`` .-```. ```\\/ _.,_ ''-._ \n"
" _.-`` `. `_. ''-._ Redis Community Edition \n"
" .-`` .-```. ```\\/ _.,_ ''-._ %s (%s/%d) %s bit\n"
" ( ' , .-` | `, ) Running in %s mode\n"
" |`-._`-...-` __...-.``-._|'` _.-'| Port: %d\n"
" | `-._ `._ / _.-' | PID: %ld\n"
+42 -26
View File
@@ -1,16 +1,41 @@
/* This file implements atomic counters using c11 _Atomic, __atomic or __sync
* macros if available, otherwise we will throw an error when compile.
*
* The exported interface is composed of three macros:
* The exported interface is composed of the following macros:
*
* atomicIncr(var,count) -- Increment the atomic counter
* atomicGetIncr(var,oldvalue_var,count) -- Get and increment the atomic counter
* atomicIncrGet(var,newvalue_var,count) -- Increment and get the atomic counter new value
* atomicDecr(var,count) -- Decrement the atomic counter
* atomicGet(var,dstvar) -- Fetch the atomic counter value
* atomicSet(var,value) -- Set the atomic counter value
* atomicGetWithSync(var,value) -- 'atomicGet' with inter-thread synchronization
* atomicSetWithSync(var,value) -- 'atomicSet' with inter-thread synchronization
*
*
* Atomic operations on flags.
* Flag type can be int, long, long long or their unsigned counterparts.
* The value of the flag can be 1 or 0.
*
* atomicFlagGetSet(var,oldvalue_var) -- Get and set the atomic counter value
*
* NOTE1: __atomic* and _Atomic implementations can be actually elaborated to support any value by changing the
* hardcoded new value passed to __atomic_exchange* from 1 to @param count
* i.e oldvalue_var = atomic_exchange_explicit(&var, count).
* However, in order to be compatible with the __sync functions family, we can use only 0 and 1.
* The only exchange alternative suggested by __sync is __sync_lock_test_and_set,
* But as described by the gnu manual for __sync_lock_test_and_set():
* https://gcc.gnu.org/onlinedocs/gcc/_005f_005fsync-Builtins.html
* "A target may support reduced functionality here by which the only valid value to store is the immediate constant 1. The exact value
* actually stored in *ptr is implementation defined."
* Hence, we can't rely on it for a any value other than 1.
* We eventually chose to implement this method with __sync_val_compare_and_swap since it satisfies functionality needed for atomicFlagGetSet
* (if the flag was 0 -> set to 1, if it's already 1 -> do nothing, but the final result is that the flag is set),
* and also it has a full barrier (__sync_lock_test_and_set has acquire barrier).
*
* NOTE2: Unlike other atomic type, which aren't guaranteed to be lock free, c11 atomic_flag does.
* To check whether a type is lock free, atomic_is_lock_free() can be used.
* It can be considered to limit the flag type to atomic_flag to improve performance.
*
* Never use return value from the macros, instead use the AtomicGetIncr()
* if you need to get the current value and increment it atomically, like
* in the following example:
@@ -21,32 +46,11 @@
*
* ----------------------------------------------------------------------------
*
* Copyright (c) 2015, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2015-Present, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
* Licensed under your choice of the Redis Source Available License 2.0
* (RSALv2) or the Server Side Public License v1 (SSPLv1).
*/
#include <pthread.h>
@@ -93,6 +97,8 @@
#define atomicGetIncr(var,oldvalue_var,count) do { \
oldvalue_var = atomic_fetch_add_explicit(&var,(count),memory_order_relaxed); \
} while(0)
#define atomicIncrGet(var, newvalue_var, count) \
newvalue_var = atomicIncr(var,count) + count
#define atomicDecr(var,count) atomic_fetch_sub_explicit(&var,(count),memory_order_relaxed)
#define atomicGet(var,dstvar) do { \
dstvar = atomic_load_explicit(&var,memory_order_relaxed); \
@@ -103,6 +109,8 @@
} while(0)
#define atomicSetWithSync(var,value) \
atomic_store_explicit(&var,value,memory_order_seq_cst)
#define atomicFlagGetSet(var,oldvalue_var) \
oldvalue_var = atomic_exchange_explicit(&var,1,memory_order_relaxed)
#define REDIS_ATOMIC_API "c11-builtin"
#elif !defined(__ATOMIC_VAR_FORCE_SYNC_MACROS) && \
@@ -111,6 +119,8 @@
/* Implementation using __atomic macros. */
#define atomicIncr(var,count) __atomic_add_fetch(&var,(count),__ATOMIC_RELAXED)
#define atomicIncrGet(var, newvalue_var, count) \
newvalue_var = __atomic_add_fetch(&var,(count),__ATOMIC_RELAXED)
#define atomicGetIncr(var,oldvalue_var,count) do { \
oldvalue_var = __atomic_fetch_add(&var,(count),__ATOMIC_RELAXED); \
} while(0)
@@ -124,12 +134,16 @@
} while(0)
#define atomicSetWithSync(var,value) \
__atomic_store_n(&var,value,__ATOMIC_SEQ_CST)
#define atomicFlagGetSet(var,oldvalue_var) \
oldvalue_var = __atomic_exchange_n(&var,1,__ATOMIC_RELAXED)
#define REDIS_ATOMIC_API "atomic-builtin"
#elif defined(HAVE_ATOMIC)
/* Implementation using __sync macros. */
#define atomicIncr(var,count) __sync_add_and_fetch(&var,(count))
#define atomicIncrGet(var, newvalue_var, count) \
newvalue_var = __sync_add_and_fetch(&var,(count))
#define atomicGetIncr(var,oldvalue_var,count) do { \
oldvalue_var = __sync_fetch_and_add(&var,(count)); \
} while(0)
@@ -149,6 +163,8 @@
ANNOTATE_HAPPENS_BEFORE(&var); \
while(!__sync_bool_compare_and_swap(&var,var,value,__sync_synchronize)); \
} while(0)
#define atomicFlagGetSet(var,oldvalue_var) \
oldvalue_var = __sync_val_compare_and_swap(&var,0,1)
#define REDIS_ATOMIC_API "sync-builtin"
#else

Some files were not shown because too many files have changed in this diff Show More