Compare commits

...
55 Commits
Author SHA1 Message Date
YaacovHazanandYaacovHazan b49d5c0cf4 Redis 8.0.3 2025-07-06 14:59:42 +03:00
Mincho PaskalevandYaacovHazan b3ed748997 Remove string cat usage in tcl tests in order to support tcl8.5 2025-07-06 14:59:42 +03:00
Ozan TezcanandYaacovHazan bde62951ac Retry accept() even if accepted connection reports an error (CVE-2025-48367)
In case of accept4() returns an error, we should check errno value and decide if we should retry accept4() without waiting next event loop iteration.
2025-07-06 14:59:42 +03:00
50188747cb Fix out of bounds write in hyperloglog commands (CVE-2025-32023)
Co-authored-by: oranagra <oran@redislabs.com>
2025-07-06 14:59:42 +03:00
0eb59d1c3b Avoid performing IO on coverage when child exits due to signal handler (#14072)
Compiled Redis with COVERAGE_TEST, while using the fork API encountered
the following issue:
- Forked process calls `RedisModule_ExitFromChild` - child process
starts to report its COW while performing IO operations
- Parent process terminates child process with
`RedisModule_KillForkChild`
- Child process signal handler gets called while an IO operation is
called
- exit() is called because COVERAGE_TEST was on during compilation.
- exit() tries to perform more IO operations in its exit handlers.
- process gets deadlocked

Backtrace snippet:
```
#0  futex_wait (private=0, expected=2, futex_word=0x7e1220000c50) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait_private (futex=0x7e1220000c50) at ./nptl/lowlevellock.c:34
#2  0x00007e1234696429 in __GI__IO_flush_all () at ./libio/genops.c:698
#3  0x00007e123469680d in _IO_cleanup () at ./libio/genops.c:843
#4  0x00007e1234647b74 in __run_exit_handlers (status=status@entry=255, listp=<optimized out>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:129
#5  0x00007e1234647bbe in __GI_exit (status=status@entry=255) at ./stdlib/exit.c:138
#6  0x00005ef753264e13 in exitFromChild (retcode=255) at /home/jonathan/CLionProjects/redis/src/server.c:263
#7  sigKillChildHandler (sig=<optimized out>) at /home/jonathan/CLionProjects/redis/src/server.c:6794
#8  <signal handler called>
#9  0x00007e1234685b94 in _IO_fgets (buf=buf@entry=0x7e122dafdd90 "KSM:", ' ' <repeats 19 times>, "0 kB\n", n=n@entry=1024, fp=fp@entry=0x7e1220000b70) at ./libio/iofgets.c:47
#10 0x00005ef75326c5e0 in fgets (__stream=<optimized out>, __n=<optimized out>, __s=<optimized out>, __s=<optimized out>, __n=<optimized out>, __stream=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:200
#11 zmalloc_get_smap_bytes_by_field (field=0x5ef7534c42fd "Private_Dirty:", pid=<optimized out>) at /home/jonathan/CLionProjects/redis/src/zmalloc.c:928
#12 0x00005ef75338ab1f in zmalloc_get_private_dirty (pid=-1) at /home/jonathan/CLionProjects/redis/src/zmalloc.c:978
#13 sendChildInfoGeneric (info_type=CHILD_INFO_TYPE_MODULE_COW_SIZE, keys=0, progress=-1, pname=0x5ef7534c95b2 "Module fork") at /home/jonathan/CLionProjects/redis/src/childinfo.c:71
#14 0x00005ef75337962c in sendChildCowInfo (pname=0x5ef7534c95b2 "Module fork", info_type=CHILD_INFO_TYPE_MODULE_COW_SIZE) at /home/jonathan/CLionProjects/redis/src/server.c:6895
#15 RM_ExitFromChild (retcode=0) at /home/jonathan/CLionProjects/redis/src/module.c:11468
```

Change is to make the exit() _exit() calls conditional based on a
parameter to exitFromChild function.
The signal handler should exit without io operations since it doesn't
know its history.(If we were in the middle of IO operations before it
was called)

---------

Co-authored-by: Yuan Wang <wangyuancode@163.com>
2025-07-06 14:59:42 +03:00
debing.sunandYaacovHazan 4df48593ad Fix crash due to incorrect event deletion of evport (#14162)
This PR fixes
https://github.com/redis/redis/issues/14056#issuecomment-3026114590

## Summary
Because evport uses `eventLoop->events[fd].mask` to determine whether to
remove the event, but in ae.c we call `aeApiDelEvent()` before updating
`eventLoop->events[fd].mask`, this causes evport to always see the old
value, and as a result, `port_dissociate()` is never called to remove
the fd.
This issue may not surface easily in a non-multithreaded, but since in
the multi-threaded case we frequently reassign fds to different threads,
it makes the crash much more likely to occur.
2025-07-06 14:59:42 +03:00
Oran AgraandYaacovHazan 475da081d7 Make Active defrag big list test much faster (#14157)
it aims to create listpacks of 500k, but did that with 5 insertions of
100k each, instead do that in one insertion, reducing the need for
listpack gradual growth, and reducing the number of commands we send.
apparently there are some stalls reading the replies of the commands,
specifically in GH actions, reducing the number of commands seems to
eliminate that.
2025-07-06 14:59:42 +03:00
a39dda5462 Vector Sets fixes against corrupted data in absence of checksum verification (#14102)
Vector Sets deserialization was not designed to resist corrupted data,
assuming that a good checksum would mean everything is fine. However
Redis allows the user to specify extra protection via a specific
configuration option.

This commit makes the implementation more resistant, at the cost of some
slowdown. This also fixes a serialization bug that is unrelated (and has
no memory corruption effects) about the lack of the worst index /
distance serialization, that could lower the quality of a graph after
links are replaced. I'll address the serialization issues in a new PR
that will focus on that aspect alone (already work in progress).

The net result is that loading vector sets is, when the serialization of
worst index/distance is missing (always, for now) 100% slower, that is 2
times the loading time we had before. Instead when the info will be
added it will be just 10/15% slower, that is, just making the new sanity
checks.

It may be worth to export to modules if advanced sanity check if needed
or not. Anyway most of the slowdown in this patch comes from having to
recompute the worst neighbor, since duplicated and non reciprocal links
detection was heavy optimized with probabilistic algorithms.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2025-07-06 14:59:42 +03:00
debing.sunandYaacovHazan 6b951ba34c Fix db->expires can't be defragged due to incorrect comparison in the expires stage (#14092)
This bug was introduced by https://github.com/redis/redis/issues/13814

When defragmenting `db->expires`, if the process exits early and
`db->expires` was modified in the meantime (e.g., FLUSHDB), we need to
check whether the previously defragmented expires is still the same as
the current one when resuming. If they differ, we should abort the
current defragmentation of expires.

However, in https://github.com/redis/redis/issues/13814, I made a
mistake by using `db->keys` and `db->expires`, as expires will never be
defragged.
2025-07-06 14:59:42 +03:00
Ozan TezcanandYaacovHazan 74b5b31a32 Fix short read issue that causes exit() on replica (#14085)
When `repl-diskless-load` is enabled on a replica, and it is in the
process of loading an RDB file, a broken connection detected by the main
channel may trigger a call to rioAbort(). This sets a flag to cause the
rdb channel to fail on the next rioRead() call, allowing it to perform
necessary cleanup.

However, there are specific scenarios where the error is checked using
rioGetReadError(), which does not account for the RIO_ABORT flag (see
[source](https://github.com/redis/redis/blob/79b37ff53548ee4d0fca287cb37a2eaf84c519a7/src/rdb.c#L3098)).
As a result, the error goes undetected. The code then proceeds to
validate a module type, fails to find a match, and calls
rdbReportCorruptRDB() which logs the following error and exits the
process:

```
The RDB file contains module data I can't load: no matching module type '_________'
```

To fix this issue, the RIO_ABORT flag has been removed. Now, rioAbort()
sets both read and write error flags, so that subsequent operations and
error checks properly detect the failure.

Additional keys were added to the short read test. It reproduces the
issue with this change. We hit that problematic line once per key. My
guess is that with many smaller keys, the likelihood of the connection
being killed at just the right moment increases.
2025-07-06 14:59:42 +03:00
Salvatore SanfilippoandYaacovHazan 795ec9118b Implement WITHATTRIBS for VSIM. (#14065)
Hi, as described, this implements WITHATTRIBS, a feature requested by a
few users, and indeed needed.
This was requested the first time by @rowantrollope but I was not sure
how to make it work with RESP2 and RESP3 in a clean way, hopefully
that's it.

The patch includes tests and documentation updates.
2025-07-06 14:59:42 +03:00
Ozan TezcanandYaacovHazan 5362410de7 Fix flaky replication test (#14034)
- Fix flaky replication test which checks memory usage on master
- Fix comments in another replication test
2025-07-06 14:59:42 +03:00
YaacovHazanandYaacovHazan 994bc96bb1 Redis 8.0.2 2025-05-27 15:39:18 +03:00
YaacovHazanandYaacovHazan 643b5db235 Check length of AOF file name in redis-check-aof (CVE-2025-27151)
Ensure that the length of the input file name does not exceed PATH_MAX
2025-05-27 15:39:18 +03:00
debing.sunandYaacovHazan 7f9212ec22 Fix incorrect server.cronloops update in defragWhileBlocked() causing timer to run twice as fast (#14081)
This bug was introduced in
[#13814](https://github.com/redis/redis/issues/13814), and was found by
@guybe7.
It incorrectly moved the update of `server.cronloops` from
`whileBlockedCron()` to `activeDefragTimeProc()`,
causing the cron-based timers to effectively run twice as fast when
active defrag is enabled.
As a result, memory statistics are not updated during blocked
operations.
The repair parts from https://github.com/redis/redis/pull/13995, because
it needs to be backport, so use a separate pr repair it.
2025-05-27 15:39:18 +03:00
Salvatore SanfilippoandYaacovHazan 00cfa6ebbc LOLWUT for Redis 8. (#14048)
# Add LOLWUT 8: TAPE MARK I - Computer Poetry Generation

This PR introduces LOLWUT 8, implementing Nanni Balestrini's
groundbreaking TAPE MARK I algorithm from 1962 - one of the first
experiments in computer-generated poetry.

## Background

TAPE MARK I, created by Italian poet Nanni Balestrini and published in
Almanacco Letterario Bompiani (1962), represents a [pioneering moment in
computational creativity](https://en.wikipedia.org/wiki/Digital_poetry).
Using an IBM 7090 mainframe, Balestrini developed an algorithm that
combines verses from three different literary sources:

1. **Diary of Hiroshima** by Michihito Hachiya
2. **The Mystery of the Elevator** by Paul Goldwin  
3. **Tao Te Ching** by Lao Tse

The algorithm selects and arranges verses based on metrical
compatibility rules and ensures alternation between different literary
sources, creating unique poetic combinations with each execution.

## Implementation

This LOLWUT command faithfully reproduces Balestrini's original
algorithm.
The main difference is that the default output is in English, and not in
Italian. However it should be noted that Balestrini used three poems
that were not in Italian anyway, so the translation process was already
part of it. In the English versions, sometimes I operated minimal
changes in order to preserve either the metric, or to make sure that the
sentence stands on its own (like adding "it" before expands rapidly).

## Cultural Significance

TAPE MARK I predates most computational art experiments and demonstrates
the early intersection of literature, technology, and algorithmic
creativity. This implementation honors that pioneering work while making
it accessible to a modern audience through Redis's LOLWUT tradition.

Each execution generates a unique poem, just as Balestrini intended.

Trivia: the original code, running on an IBM 7090, used six minutes to
generate each verse :D

**IMPORTANT** This commit should be back-ported to Redis 8.
2025-05-27 15:39:18 +03:00
Salvatore SanfilippoandYaacovHazan bc71e8fe2d [Vector sets] More rdb loading fixes (#14032)
Hi all, this PR fixes two things:

1. An assertion, that prevented the RDB loading from recovery if there
was a quantization type mismatch (with regression test).
2. Two code paths that just returned NULL without proper cleanup during
RDB loading.
2025-05-27 15:39:18 +03:00
Salvatore SanfilippoandYaacovHazan ad9575dd6d [Vector sets] RDB IO errors handling (#13978)
This PR adds support for REDISMODULE_OPTIONS_HANDLE_IO_ERRORS.
and tests for short read and corrupted RESTORE payload.

Please: note that I also removed the comment about async loading support
since we should be already covered. No manipulation of global data
structures in Vector Sets, if not for the unique ID used to create new
vector sets with different IDs.
2025-05-27 15:39:18 +03:00
YaacovHazanandGitHub 63b87554f0 Redis 8.0.1 (#14025) 2025-05-13 16:28:36 +03:00
YaacovHazan 7bc6ff3442 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-05-06 21:40:34 +03:00
YaacovHazanandGitHub e91a340e24 Redis 8.0 GA (#14003) 2025-05-02 14:15:06 +03:00
YaacovHazan 4ec6d54426 Redis 8.0 GA 2025-05-02 12:13:21 +03:00
YaacovHazan b319f21df1 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-05-02 12:00:20 +03:00
YaacovHazan 7cd656a0c3 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-04-28 12:56:36 +03:00
YaacovHazanandGitHub b0403f1fa2 Redis CE 8.0 RC1 (#13924) 2025-04-07 11:38:16 +03:00
YaacovHazan ce8c3993c5 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-04-06 21:52:28 +03:00
YaacovHazan 385823ea49 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-04-02 22:05:08 +03:00
YaacovHazan c6e5d1d5fe Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-03-31 21:26:40 +03:00
YaacovHazanandGitHub 5d887c58ae Merge unstable into 8.0 (#13901)
Merging unstable towards GA
2025-03-30 15:11:57 +03:00
YaacovHazan 452b5b8a3b Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-03-30 09:54:48 +03:00
YaacovHazan 095c131fbb Redis 8.0 M04 2025-03-16 10:12:40 +02:00
YaacovHazan 89eef40ca2 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-03-16 10:01:10 +02:00
YaacovHazanandGitHub 84471e238e Redis 8.0 RC1 (#13851)
- Merge latest unstable
- Update version and release notes
2025-03-11 20:16:27 +02:00
YaacovHazanandYaacovHazan d1df881ec5 Redis 8.0 RC1 2025-03-11 10:03:17 +02:00
YaacovHazan 53949521de Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-03-11 09:39:04 +02:00
YaacovHazan a39ffc1fe9 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-03-09 10:08:03 +02:00
YaacovHazanandGitHub cb261828bd Merge unstable into 8.0 (#13835)
preparing for 8.0 RC1
2025-02-27 08:29:01 +02:00
YaacovHazan 9265234299 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-02-26 21:23:36 +02:00
YaacovHazanandGitHub 1cddd0d3ba Merge unstable into 8.0 (#13824) 2025-02-23 08:27:33 +02:00
YaacovHazan 1d5e13e121 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-02-22 21:48:15 +02:00
YaacovHazanandGitHub c2d3e28540 Update RQE version 7.99.3 (#13767) (#13769)
Update RQE version 7.99.3
2025-01-24 09:20:23 +02:00
YaacovHazan 0c1a764074 Merge commit 'dcd0b3d02' into HEAD 2025-01-24 09:15:59 +02:00
YaacovHazanandGitHub d88611d36f Redis 8.0 M03 (#13759) 2025-01-20 19:45:10 +02:00
YaacovHazan 4c5a9076d7 Redis 8.0 M03 2025-01-20 16:00:49 +02:00
YaacovHazan 8aab7ca84c Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-01-20 11:06:49 +02:00
YaacovHazan 9c81f8bd61 Merge remote-tracking branch 'upstream/unstable' into HEAD 2025-01-14 14:01:19 +02:00
YaacovHazanandGitHub 0a24bfec13 Redis 8.0 M02 (#13624) 2024-10-28 19:09:18 +02:00
YaacovHazan 14c48cc5ed Redis 8.0 M02 2024-10-28 14:01:22 +02:00
YaacovHazanandGitHub b08fd9f989 Merge unstable into 8.0 (#13622) 2024-10-27 17:23:40 +02:00
YaacovHazan dd20aa5160 Merge remote-tracking branch 'upstream/unstable' into HEAD 2024-10-27 13:50:58 +02:00
YaacovHazanandGitHub 30eb6c3210 Merge unstable into 8.0 (#13573) 2024-09-26 14:36:04 +03:00
YaacovHazan 80458d0490 Merge remote-tracking branch 'upstream/unstable' into HEAD 2024-09-26 10:50:12 +03:00
YaacovHazanandGitHub 5fe3e74a38 Redis 8.0 M01 (#13538)
### New Features in binary distributions

- 7 new data structures: JSON, Time series, Bloom filter, Cuckoo filter,
Count-min sketch, Top-k, t-digest
- Redis scalable query engine (including vector search)

### Potentially breaking changes

- #12272 `GETRANGE` returns an empty bulk when the negative end index is
out of range
- #12395 Optimize `SCAN` command when matching data type

### Bug fixes

- #13510 Fix `RM_RdbLoad` to enable AOF after RDB loading is completed
- #13489 `ACL CAT` - return module commands
- #13476 Fix a race condition in the `cache_memory` of `functionsLibCtx`
- #13473 Fix incorrect lag due to trimming stream via `XTRIM` command
- #13338 Fix incorrect lag field in `XINFO` when tombstone is after the
`last_id` of the consume group
- #13470 On `HDEL` of last field - update the global hash field
expiration data structure
- #13465 Cluster: Pass extensions to node if extension processing is
handled by it
- #13443 Cluster: Ensure validity of myself when loading cluster config
- #13422 Cluster: Fix `CLUSTER SHARDS` command returns empty array

### Modules API

- #13509 New API calls: `RM_DefragAllocRaw`, `RM_DefragFreeRaw`, and
`RM_RegisterDefragCallbacks` - defrag API to allocate and free raw
memory

### Performance and resource utilization improvements

- #13503 Avoid overhead of comparison function pointer calls in listpack
`lpFind`
- #13505 Optimize `STRING` datatype write commands
- #13499 Optimize `SMEMBERS` command
- #13494 Optimize `GEO*` commands reply
- #13490 Optimize `HELLO` command
- #13488 Optimize client query buffer
- #12395 Optimize `SCAN` command when matching data type
- #13529 Optimize `LREM`, `LPOS`, `LINSERT`, and `LINDEX` commands
- #13516 Optimize `LRANGE` and other commands that perform several
writes to client buffers per call
- #13431 Avoid `used_memory` contention when updating from multiple
threads

### Other general improvements

- #13495 Reply `-LOADING` on replica while flushing the db

### CLI tools

- #13411 redis-cli: Fix wrong `dbnum` showed after the client
reconnected

### Notes

- No backward compatibility for replication or persistence.
- Additional distributions, upgrade paths, features, and improvements
will be introduced in upcoming pre-releases.
- With the GA release of 8.0 we will deprecate Redis Stack.
2024-09-12 11:52:28 +03:00
YaacovHazan 4955375ec7 Redis 8.0 M01 2024-09-12 10:23:36 +03:00
YaacovHazan 406a365f44 Bundle modules for Redis 8.0 M01 2024-09-11 16:40:16 +03:00
39 changed files with 1416 additions and 103 deletions
+444 -11
View File
@@ -1,16 +1,449 @@
Hello! This file is just a placeholder, since this is the "unstable" branch
of Redis, the place where all the development happens.
Redis Open Source 8.0 release notes
===================================
There is no release notes for this branch, it gets forked into another branch
every time there is a partial feature freeze in order to eventually create
a new stable release.
--------------------------------------------------------------------------------
Upgrade urgency levels:
Usually "unstable" is stable enough for you to use it in development environments
however you should never use it in production environments. It is possible
to download the latest stable release here:
LOW: No need to upgrade unless there are new features you want to use.
MODERATE: Program an upgrade of the server, but it's not urgent.
HIGH: There is a critical bug that may affect a subset of users. Upgrade!
CRITICAL: There is a critical bug affecting MOST USERS. Upgrade ASAP.
SECURITY: There are security fixes in the release.
--------------------------------------------------------------------------------
https://download.redis.io/redis-stable.tar.gz
Starting with v8.0.1, the release notes contain PRs from multiple repositories:
More information is available at https://redis.io
#n - Redis (https://github.com/redis/redis)
#QEn = Query Engine (https://github.com/RediSearch/RediSearch)
#JSn = JSON (https://github.com/RedisJSON/RedisJSON)
#TSn = Time Series (https://github.com/RedisTimeSeries/RedisTimeSeries)
#PRn = Probabilistic (https://github.com/RedisBloom/RedisBloom)
================================================================================
Redis 8.0.3 Released Sun 6 Jul 2025 12:00:00 IST
================================================================================
Update urgency: `SECURITY`: There are security fixes in the release.
### Security fixes
* (CVE-2025-32023) Fix out-of-bounds write in `HyperLogLog` commands
* (CVE-2025-48367) Retry accepting other connections even if the accepted connection reports an error
### New Features
- #14065 `VSIM`: Add new `WITHATTRIBS` to return the JSON attribute associated with an element
### Bug fixes
- #14085 A short read may lead to an exit() on a replica
- #14092 db->expires is not defragmented
================================================================================
Redis 8.0.2 Released Tue 27 May 2025 12:00:00 IST
================================================================================
Update urgency: `SECURITY`: There are security fixes in the release.
### Security fixes
- (CVE-2025-27151) redis-check-aof may lead to stack overflow and potential RCE
### Other general improvements
- #14048 `LOLWUT` for Redis 8
================================================================================
Redis 8.0.1 Released Sun 13 May 2025 16:00:00 IST
================================================================================
Update urgency: `MODERATE`: No need to upgrade unless there are new features you want to use.
### Performance and resource utilization improvements
- #13959 Vector set - faster `VSIM` `FILTER` parsing
### Bug fixes
- #QE6083 Query Engine - revert default policy `search-on-timeout` to `RETURN`
- #QE6050 Query Engine - `@__key` on `FT.AGGREGATE` used as reserved field name preventing access to Redis keyspace
- #QE6077 Query Engine - crash when calling `FT.CURSOR DEL` while retrieving from the CURSOR
### Notes
- Fixed wrong text in the license files
=======================================================
8.0 GA (v8.0.0) Released Fri 2 May 2025 12:00:00 IST
=======================================================
This is the General Availability release of Redis Open Source 8.0.
Redis 8.0 deprecates previous Redis and Redis Stack versions.
Stand alone RediSearch, RedisJSON, RedisTimeSeries, and RedisBloom are no longer needed as they are now part of Redis.
### Major changes compared to 7.4.2
- Name change: Redis Community Edition is now Redis Open Source
- License change: licensed under your choice of
- (a) the Redis Source Available License 2.0 (RSALv2); or
- (b) the Server Side Public License v1 (SSPLv1); or
- (c) the GNU Affero General Public License (AGPLv3)
- Redis Query engine and 8 new data structures are now an integral part of Redis 8
- (1) Redis Query Engine, which now supports both horizontal and vertical scaling for search, query and vector workloads
- (2) JSON - a queryable JSON document
- (3) Time series
- (4-8) Five probabilistic data structures: Bloom filter, Cuckoo filter, Count-min sketch, Top-k, and t-digest
- (9) Vector set [beta] - a data structure designed for Vector Similarity Search, inspired by Sorted set
- These nine components are included in all binary distributions
- See instructions in the README.md file on how to build from source with all these components
- New configuration file: redis-full.conf - loads Redis with all these components,
and contains new configuration parameters for Redis Query engine and the new data structures
- New ACL categories: @search, @json, @timeseries, @bloom, @cuckoo, @cms, @topk, @tdigest
- Commands are also included in the existing ACL categories (@read, @write, etc.)
- More than 30 performance and resource utilization improvements
- A new I/O threading implementation which enables throughput increase on multi-core environments
(set with `io-threads` configuration parameter)
- An improved replication mechanism which is more performant and robust
- New hash commands - `HGETDEL`, `HGETEX`, `HSETEX`
For more details, see the release notes of 8.0-M01, 8.0-M02, 8.0-M03,8.0-M04, and 8.0-RC1
### Binary distributions
- Alpine and Debian Docker images - https://hub.docker.com/_/redis
- Install using snap - see https://github.com/redis/redis-snap
- Install using brew - see https://github.com/redis/homebrew-redis
- Install using RPM - see https://github.com/redis/redis-rpm
- Install using Debian APT - see https://github.com/redis/redis-debian
### Operating systems we test Redis 8.0 on
- Ubuntu 20.04 (Focal Fossa), 22.04 (Jammy Jellyfish), 24.04 (Noble Numbat)
- Rocky Linux 8.10, 9.5
- AlmaLinux 8.10, 9.5
- Debian 11 (Bullseye), 12 (Bookworm)
- macOS 13 (Ventura), 14 (Sonoma), 15 (Sequoia)
### Supported upgrade paths (by replication or persistence)
- From previous Redis versions, without modules
- From previous Redis versions with modules (RediSearch, RedisJSON, RedisTimeSeries, RedisBloom)
- From Redis Stack 7.2 or 7.4
### Security fixes (compared to 8.0-RC1)
* (CVE-2025-21605) An unauthenticated client can cause an unlimited growth of output buffers
### Bug fixes (compared to 8.0-RC1)
- #13966, #13932 `CLUSTER SLOTS` - TLS port update not reflected in CLUSTER SLOTS
- #13958 `XTRIM`, `XADD` - incorrect lag due to trimming stream
- #13931 `HGETEX` - wrong order of keyspace notifications
- #JS1337 - JSON - `JSON.DEL` emits no `DEL` notification when removing the entire value (MOD-9117)
- #TS1742 - Time Series - `TS.INFO` - `duplicatePolicy` is `nil` when set to the default value (MOD-5423) (edited)
==========================================================
8.0-RC1 (v7.9.240) Released Mon 7 Apr 2025 10:00:00 IST
==========================================================
This is the first Release Candidate of Redis Community Edition 8.0.
Release Candidates are feature-complete pre-releases. Pre-releases are not suitable for production use.
### Headlines
8.0-RC1 includes a new beta data structure - vector set.
### Distributions
- Alpine and Debian Docker images - https://hub.docker.com/_/redis
- Install using snap - see https://github.com/redis/redis-snap
- Install using brew - see https://github.com/redis/homebrew-redis
- Install using RPM and Debian APT - will be added on the GA release
### New Features
- #13915 Vector set - a new data structure [beta]:
Vector set extends the concept of sorted sets to allow the storage and querying of
high-dimensional vector embeddings, enhancing Redis for AI use cases that involve
semantic search and recommendation systems. Vector sets complement the existing
vector search capability in the Redis Query Engine. The vector set data type is
available in beta. We may change, or even break, the features and the API in
future versions. We are open to your feedback as you try out this new data type.
- #13846 Allow detecting incompatibility risks before switching to cluster mode
### Bug fixes
- #13895 RDB Channel replication - replica is online after BGSAVE is done
- #13877 Inconsistency for ShardID in case both master and replica support it
- #13883 Defrag scan may return nothing when type/encoding changes during it
- #13863 `RANDOMKEY` - infinite loop during client pause
- #13853 `SLAVEOF` - crash when clients are blocked on lazy free
- #13632 `XREAD` returns nil while stream is not empty
### Metrics
- #13846 `INFO`: `cluster_incompatible_ops` - number of cluster-incompatible commands
### Configuration parameters
- #13846 `cluster-compatibility-sample-ratio` - sampling ratio (0-100) for checking command compatibility with cluster mode
============================================================
8.0-M04 (v7.9.227) Committed Sun 16 Mar 2025 11:00:00 IST
============================================================
This is the fourth Milestone of Redis Community Edition 8.0.
Milestones are non-feature-complete pre-releases. Pre-releases are not suitable for production use.
Once we reach feature-completeness we will release RC1.
### Headlines
8.0-M04 includes 3 new hash commands, performance improvements, and memory defragmentation improvements.
### Distributions
- Alpine and Debian Docker images - https://hub.docker.com/_/redis
- Install using snap - see https://github.com/redis/redis-snap
- Install using brew - see https://github.com/redis/homebrew-redis
- Install using RPM and Debian APT - will be added on the GA release
### New Features
- #13798 Hash - new commands:
- `HGETDEL` Get and delete the value of one or more fields of a given hash key
- `HGETEX` Get the value of one or more fields of a given hash key, and optionally set their expiration
- `HSETEX` Set the value of one or more fields of a given hash key, and optionally set their expiration
- #13773 Add replication offset to AOF, allowing more robust way to determine which AOF has a more up-to-date data during recovery
- #13740, #13763 shared secret - new mechanism to allow sending internal commands between nodes
### Bug fixes
- #13804 Overflow on 32-bit systems when calculating idle time for eviction
- #13793 `WAITAOF` returns prematurely
- #13800 Remove `DENYOOM` from `HEXPIRE`, `HEXPIREAT`, `HPEXPIRE`, and `HPEXPIREAT`
- #13632 Streams - wrong behavior of `XREAD +` after last entry
### Modules API
- #13788 `RedisModule_LoadDefaultConfigs` - load module configuration values from redis.conf
- #13815 `RM_RegisterDefragFunc2` - support for incremental defragmentation of global module data
- #13816 `RM_DefragRedisModuleDict` - allow modules to defrag `RedisModuleDict`
- #13774 `RM_GetContextFlags` - add a `REDISMODULE_CTX_FLAGS_DEBUG_ENABLED` flag to execute debug commands
### Performance and resource utilization improvements
- #13752 Reduce defrag CPU usage when defragmentation is ineffective
- #13764 Reduce latency when a command is called consecutively
- #13787 Optimize parsing data from clients, specifically multi-bulk (array) data
- #13792 Optimize dictionary lookup by avoiding duplicate key length calculation during comparisons
- #13796 Optimize expiration checks
============================================================
8.0-M03 (v7.9.226) Committed Mon 20 Jan 2025 15:00:00 IST
============================================================
This is the third Milestone of Redis Community Edition 8.0.
Milestones are non-feature-complete pre-releases. Pre-releases are not suitable for production use.
Once we reach feature-completeness we will release RC1.
### Headlines:
8.0-M03 introduces an improved replication mechanism which is more performant and robust, a new I/O threading implementation which enables throughput increase on multi-core environments, and many additional performance improvements. Both Alpine and Debian Docker images are now available on [Docker Hub](https://hub.docker.com/_/redis). A snap and Homebrew distributions are available as well.
### Security fixes
- (CVE-2024-46981) Lua script may lead to remote code execution
- (CVE-2024-51741) Denial-of-service due to malformed ACL selectors
### New Features
- #13695 New I/O threading implementation
- #13732 New replication mechanism
### Bug fixes
- #13653 `MODULE LOADEX` - crash on nonexistent parameter name
- #13661 `FUNCTION FLUSH` - memory leak when using jemalloc
- #13626 Memory leak on failed RDB loading
### Other general improvements
- #13639 When `hide-user-data-from-log` is enabled - also print command tokens on crash
- #13660 Add the Lua VM memory to memory overhead
### New metrics
- #13592 `INFO` - new `KEYSIZES` section includes key size distributions for basic data types
- #13695 `INFO` - new `Threads` section includes I/O threading metrics
### Modules API
- #13666 `RedisModule_ACLCheckKeyPrefixPermissions` - check access permissions to any key matching a given prefix
- #13676 `RedisModule_HashFieldMinExpire` - query the minimum expiration time over all the hashs fields
- #13676 `RedisModule_HashGet` - new `REDISMODULE_HASH_EXPIRE_TIME` flag - query the field expiration time
- #13656 `RedisModule_RegisterXXXConfig` - allow registering unprefixed configuration parameters
### Configuration parameters
- `replica-full-sync-buffer-limit` - maximum size of accumulated replication stream data on the replica side
- `io-threads-do-reads` is no longer effective. The new I/O threading implementation always use threads for both reads and writes
### Performance and resource utilization improvements
- #13638 Optimize CRC64 performance
- #13521 Optimize commands with large argument count - reuse c->argv after command execution
- #13558 Optimize `PFCOUNT` and `PFMERGE` - SIMD acceleration
- #13644 Optimize `GET` on high pipeline use-cases
- #13646 Optimize `EXISTS` - prefetching and branch prediction hints
- #13652 Optimize `LRANGE` - improve listpack handling and decoding efficiency
- #13655 Optimize `HSET` - avoid unnecessary hash field creation or deletion
- #13721 Optimize `LRANGE` and `HGETALL` - refactor client write preparation and handling
============================================================
8.0-M02 (v7.9.225) Committed Mon 28 Oct 2024 14:00:00 IST
============================================================
This is the second Milestone of Redis Community Edition 8.0.
Milestones are non-feature-complete pre-releases. Pre-releases are not suitable for production use.
Once we reach feature-completeness we will release RC1.
### Headlines:
8.0-M02 introduces significant performance improvements. Both Alpine and Debian Docker images are now available on [Docker Hub](https://hub.docker.com/_/redis). Additional distributions will be introduced in upcoming pre-releases.
### Supported upgrade paths (by replication or persistence) to 8.0-M02
- From previous Redis versions, without modules
The following upgrade paths (by replication or persistence) to 8.0-M02 are not yet tested and will be introduced in upcoming pre-releases:
- From previous Redis versions with modules (RediSearch, RedisJSON, RedisTimeSeries, RedisBloom)
- From Redis Stack 7.2 or 7.4
### Security fixes
- (CVE-2024-31449) Lua library commands may lead to stack overflow and potential RCE.
- (CVE-2024-31227) Potential Denial-of-service due to malformed ACL selectors.
- (CVE-2024-31228) Potential Denial-of-service due to unbounded pattern matching.
### Bug fixes
- #13539 Hash: Fix key ref for a hash that no longer has fields with expiration on `RENAME`/`MOVE`/`SWAPDB`/`RESTORE`
- #13512 Fix `TOUCH` command from a script in no-touch mode
- #13468 Cluster: Fix cluster node config corruption caused by mixing shard-id and non-shard-id versions
- #13608 Cluster: Fix `GET #` option in `SORT` command
### Modules API
- #13526 Extend `RedisModule_OpenKey` to read also expired keys and subkeys
### Performance and resource utilization improvements
- #11884 Optimize `ZADD` and `ZRANGE*` commands
- #13530 Optimize `SSCAN` command in case of listpack or intset encoding
- #13531 Optimize `HSCAN`/`ZSCAN` command in case of listpack encoding
- #13520 Optimize commands that heavily rely on bulk/mbulk replies (example of `LRANGE`)
- #13566 Optimize `ZUNION[STORE]` by avoiding redundant temporary dict usage
- #13567 Optimize `SUNION`/`SDIFF` commands by avoiding redundant temporary dict usage
- #11533 Avoid redundant `lpGet` to boost `quicklistCompare`
- #13412 Reduce redundant call of `prepareClientToWrite` when call `addReply*` continuously
===========================================================
8.0-M01 (v7.9.224) Released Thu 12 Sep 2024 10:00:00 IST
===========================================================
This is the first Milestone of Redis Community Edition 8.0.
Milestones are non-feature-complete pre-releases. Pre-releases are not suitable for production use.
Once we reach feature-completeness we will release RC1.
### Headlines:
Redis 8.0 introduces new data structures: JSON, time series, and 5 probabilistic data structures (previously available as separate Redis modules) and incorporates the enhanced Redis Query Enginer (with vector search).
8.0-M01 is available as a Docker image and can be downloaded from [Docker Hub](https://hub.docker.com/_/redis). Additional distributions will be introduced in upcoming pre-releases.
### Supported upgrade paths (by replication or persistence) to 8.0-M01
- From previous Redis versions, without modules
The following upgrade paths (by replication or persistence) to 8.0-M01 are not yet tested and will be introduced in upcoming pre-releases:
- From previous Redis versions with modules (RediSearch, RedisJSON, RedisTimeSeries, RedisBloom)
- From Redis Stack 7.2 or 7.4
### New Features in binary distributions
- 7 new data structures: JSON, Time series, Bloom filter, Cuckoo filter, Count-min sketch, Top-k, t-digest
- The enhanced Redis Query Engine (with vector search)
### Potentially breaking changes
- #12272 `GETRANGE` returns an empty bulk when the negative end index is out of range
- #12395 Optimize `SCAN` command when matching data type
### Bug fixes
- #13510 Fix `RM_RdbLoad` to enable AOF after RDB loading is completed
- #13489 `ACL CAT` - return module commands
- #13476 Fix a race condition in the `cache_memory` of `functionsLibCtx`
- #13473 Fix incorrect lag due to trimming stream via `XTRIM` command
- #13338 Fix incorrect lag field in `XINFO` when tombstone is after the `last_id` of the consume group
- #13470 On `HDEL` of last field - update the global hash field expiration data structure
- #13465 Cluster: Pass extensions to node if extension processing is handled by it
- #13443 Cluster: Ensure validity of myself when loading cluster config
- #13422 Cluster: Fix `CLUSTER SHARDS` command returns empty array
### Modules API
- #13509 New API calls: `RM_DefragAllocRaw`, `RM_DefragFreeRaw`, and `RM_RegisterDefragCallbacks` - defrag API to allocate and free raw memory
### Performance and resource utilization improvements
- #13503 Avoid overhead of comparison function pointer calls in listpack `lpFind`
- #13505 Optimize `STRING` datatype write commands
- #13499 Optimize `SMEMBERS` command
- #13494 Optimize `GEO*` commands reply
- #13490 Optimize `HELLO` command
- #13488 Optimize client query buffer
- #12395 Optimize `SCAN` command when matching data type
- #13529 Optimize `LREM`, `LPOS`, `LINSERT`, and `LINDEX` commands
- #13516 Optimize `LRANGE` and other commands that perform several writes to client buffers per call
- #13431 Avoid `used_memory` contention when updating from multiple threads
### Other general improvements
- #13495 Reply `-LOADING` on replica while flushing the db
### CLI tools
- #13411 redis-cli: Fix wrong `dbnum` showed after the client reconnected
### Notes
- No backward compatibility for replication or persistence.
- Additional distributions, upgrade paths, features, and improvements will be introduced in upcoming pre-releases.
- With the GA release of 8.0 we will deprecate Redis Stack.
Happy hacking!
+7 -1
View File
@@ -66,7 +66,7 @@ performed in the background, while the command is executed in the main thread.
**VSIM: return elements by vector similarity**
VSIM key [ELE|FP32|VALUES] <vector or element> [WITHSCORES] [COUNT num] [EF search-exploration-factor] [FILTER expression] [FILTER-EF max-filtering-effort] [TRUTH] [NOTHREAD]
VSIM key [ELE|FP32|VALUES] <vector or element> [WITHSCORES] [WITHATTRIBS] [COUNT num] [EF search-exploration-factor] [FILTER expression] [FILTER-EF max-filtering-effort] [TRUTH] [NOTHREAD]
The command returns similar vectors, for simplicity (and verbosity) in the following example, instead of providing a vector using FP32 or VALUES (like in `VADD`), we will ask for elements having a vector similar to a given element already in the sorted set:
@@ -98,8 +98,14 @@ The `TRUTH` option forces the command to perform a linear scan of all the entrie
The `NOTHREAD` option forces the command to execute the search on the data structure in the main thread. Normally `VSIM` spawns a thread instead. This may be useful for benchmarking purposes, or when we work with extremely small vector sets and don't want to pay the cost of spawning a thread. It is possible that in the future this option will be automatically used by Redis when we detect small vector sets. Note that this option blocks the server for all the time needed to complete the command, so it is a source of potential latency issues: if you are in doubt, never use it.
The `WITHSCORES` option returns, for each returned element, a floating point number representing how near the element is from the query, as a similarity between 0 and 1, where 0 means the vectors are opposite, and 1 means they are pointing exactly in the same direction (maximum similarity).
The `WITHATTRIBS` option returns, for each element, the JSON attribute associated with the element, or NULL for the elements missing an attribute.
For `FILTER` and `FILTER-EF` options, please check the filtered search section of this documentation.
Note that when `WITHSCORES` and `WITHATTRIBS` are provided at the same time, the RESP2 reply guarantees that the returned elements are always in the sequence *ele*,*score*,*attribs*, while RESP3 replies will be in the form *ele > score|attrib* when just one is provided, or *ele -> [score,attrib]* when both are provided, that is, when both options are used and RESP3 is used the score and attribute will be a two-items array associated to the element key.
**VDIM: return the dimension of the vectors inside the vector set**
VDIM keyname
+6
View File
@@ -291,6 +291,12 @@
"name": "withscores",
"type": "pure-token",
"optional": true
},
{
"token": "WITHATTRIBS",
"name": "withattribs",
"type": "pure-token",
"optional": true
}
]
},
+99 -5
View File
@@ -45,6 +45,7 @@
#include <float.h> /* for INFINITY if not in math.h */
#include <assert.h>
#include "hnsw.h"
#include "mixer.h"
#if 0
#define debugmsg printf
@@ -2127,6 +2128,7 @@ void hnsw_free_serialized_node(hnswSerNode *sn) {
* The function returns NULL both on out of memory and if the remaining
* parameters length does not match the number of links or other items
* to load. */
#define HNSW_SER_WORSTLINK_MISSING UINT32_MAX
hnswNode *hnsw_insert_serialized(HNSW *index, void *vector, uint64_t *params, uint32_t params_len, void *value)
{
if (params_len < 2) return NULL;
@@ -2158,6 +2160,13 @@ hnswNode *hnsw_insert_serialized(HNSW *index, void *vector, uint64_t *params, ui
uint32_t num_links = params[param_idx++];
uint32_t max_links = params[param_idx++];
/* Sanity check: links should be less than max links and
* in general a reasonable amount. */
if (num_links > max_links || max_links > HNSW_MAX_M*4) {
hnsw_node_free(node);
return NULL;
}
/* If max_links is larger than current allocation, reallocate.
* It could happen in select_neighbors() that we over-allocate the
* node under very unlikely to happen conditions. */
@@ -2185,6 +2194,10 @@ hnswNode *hnsw_insert_serialized(HNSW *index, void *vector, uint64_t *params, ui
* fit more than 2^32 nodes in a 32 bit system. */
for (uint32_t j = 0; j < num_links; j++)
node->layers[i].links[j] = (hnswNode*)params[param_idx++];
/* XXX: fix me, we need to store the worst link info in a
* backward compatible way. */
node->layers[i].worst_idx = HNSW_SER_WORSTLINK_MISSING;
}
/* Get l2 and quantization range. */
@@ -2221,13 +2234,28 @@ uint64_t hnsw_hash_node_id(uint64_t id) {
return id;
}
/* Helper for duplicated link detection in hnsw_deserialize_index(). */
static int qsort_compare_pointers(const void *aptr, const void *bptr) {
uintptr_t a = *((uintptr_t*)aptr);
uintptr_t b = *((uintptr_t*)bptr);
if (a > b) return 1;
if (a < b) return -1;
return 0;
}
/* Fix pointers of neighbors nodes: after loading the serialized nodes, the
* neighbors links are just IDs (casted to pointers), instead of the actual
* pointers. We need to resolve IDs into pointers.
*
* The two integers salt0 and salt1 are used to make the internal state
* of the function unguessable to an external attacker, in order to protect
* from corruptions. Show be two random numbers from /dev/urandom if possible
* otherwise can be just 0,0 if the application is not security critical and
* never processes untrusted inputs.
*
* Return 0 on error (out of memory or some ID that can't be resolved), 1 on
* success. */
int hnsw_deserialize_index(HNSW *index) {
int hnsw_deserialize_index(HNSW *index, uint64_t salt0, uint64_t salt1) {
/* We will use simple linear probing, so over-allocating is a good
* idea: anyway this flat array of pointers will consume a fraction
* of the memory of the loaded index. */
@@ -2253,12 +2281,60 @@ int hnsw_deserialize_index(HNSW *index) {
node = node->next;
}
/* Second pass: fix pointers of all the neighbors links. */
/* Second pass: fix pointers of all the neighbors links.
* As we scan and fix the links, we also compute the accumulator
* register "reciprocal", that is used in order to guarantee that all
* the links are reciprocal.
*
* This is how it works, we hash (using a strong hash function) the
* following key for each link that we see from A to B (or vice versa):
*
* hash(salt || A || B || link-level)
*
* We always sort A and B, so the same link from A to B and from B to A
* will hash the same. The we xor the result into the 128 bit accumulator.
* If each link has its own backlink, the accumulator is guaranteed to
* be zero at the end.
*
* Collisions are extremely unlikely to happen, and an external attacker
* can't easily control the hash function output, since the salt is
* unknown, and also there would be to control the pointers.
*
* This algorithm is O(1) for each node so it is basically free for
* us, as we scan the list of nodes, and runs on constant and very
* small memory. */
uint64_t accumulator[2] = {0,0};
node = index->head; // Rewind.
while(node) {
uint64_t this_node_id = node->id;
for (uint32_t i = 0; i <= node->level; i++) {
// Check if there are duplicated links: those are
// also corruptions of the on-disk serialization format.
if (node->layers[i].num_links > 0) {
qsort(node->layers[i].links, node->layers[i].num_links,
sizeof(void*), qsort_compare_pointers);
for (uint32_t j = 0; j < node->layers[i].num_links-1; j++) {
if (node->layers[i].links[j] == node->layers[i].links[j+1])
goto corrupted;
}
}
// Resolve pointers.
for (uint32_t j = 0; j < node->layers[i].num_links; j++) {
uint64_t linked_id = (uint64_t) node->layers[i].links[j];
// We can't link to our own node.
if (linked_id == this_node_id) goto corrupted;
// Compute accumulator for reciprocal links check.
uint64_t mixed_h1, mixed_h2;
secure_pair_mixer_128(salt0, salt1, this_node_id, linked_id, (uint64_t)i, &mixed_h1, &mixed_h2);
accumulator[0] ^= mixed_h1;
accumulator[1] ^= mixed_h2;
// Fix links.
uint64_t bucket = hnsw_hash_node_id(linked_id) & (table_size-1);
hnswNode *neighbor = NULL;
for (uint64_t k = 0; k < table_size; k++) {
@@ -2268,19 +2344,37 @@ int hnsw_deserialize_index(HNSW *index) {
}
bucket = (bucket+1) & (table_size-1);
}
if (neighbor == NULL) {
/* The neighbor must exist and also exist at the right
* level. */
if (neighbor == NULL || neighbor->level < i) {
/* Unresolved link! Either a bug in this code
* or broken serialization data. */
hfree(table);
return 0;
goto corrupted;
}
node->layers[i].links[j] = neighbor;
}
/* The worst link information was missing from older
* serialization formats. Compute it on the fly if needed. */
if (node->layers[i].worst_idx == HNSW_SER_WORSTLINK_MISSING) {
hnsw_update_worst_neighbor(index,node,i);
}
}
node = node->next;
}
/* Check that links are reciprocal, otherwise fail. */
if (accumulator[0] || accumulator[1]) goto corrupted;
/* Everything fine. Return success. */
hfree(table);
return 1;
corrupted:
/* Some corruption error detected. */
hfree(table);
return 0;
}
/* ================================ Iterator ================================ */
+1 -1
View File
@@ -158,7 +158,7 @@ void hnsw_free_insert_context(InsertContext *ctx);
hnswSerNode *hnsw_serialize_node(HNSW *index, hnswNode *node);
void hnsw_free_serialized_node(hnswSerNode *sn);
hnswNode *hnsw_insert_serialized(HNSW *index, void *vector, uint64_t *params, uint32_t params_len, void *value);
int hnsw_deserialize_index(HNSW *index);
int hnsw_deserialize_index(HNSW *index, uint64_t salt0, uint64_t salt1);
// Helper function in case the user wants to directly copy
// the vector bytes.
+106
View File
@@ -0,0 +1,106 @@
/* Redis implementation for vector sets. The data structure itself
* is implemented in hnsw.c.
*
* Copyright (c) 2009-Present, Redis Ltd.
* All rights reserved.
*
* Licensed under your choice of (a) the Redis Source Available License 2.0
* (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the
* GNU Affero General Public License v3 (AGPLv3).
* Originally authored by: Salvatore Sanfilippo.
*
* =============================================================================
*
* Mixing function for HNSW link integrity verification
* Designed to resist collision attacks when salts are unknown.
*/
#include <stdint.h>
#include <string.h>
static inline uint64_t ROTL64(uint64_t x, int r) {
return (x << r) | (x >> (64 - r));
}
// Use more rounds and stronger constants
#define MIX_PRIME_1 0xFF51AFD7ED558CCDULL
#define MIX_PRIME_2 0xC4CEB9FE1A85EC53ULL
#define MIX_PRIME_3 0x9E3779B97F4A7C15ULL
#define MIX_PRIME_4 0xBF58476D1CE4E5B9ULL
#define MIX_PRIME_5 0x94D049BB133111EBULL
#define MIX_PRIME_6 0x2B7E151628AED2A7ULL
/* Mixer design goals:
* 1. Thorough mixing of the level parameter.
* 2. Enough rounds of mixing.
* 3. Cross-influence between h1 and h2.
* 4. Domain separation to prevent related-key attacks.
*/
void secure_pair_mixer_128(uint64_t salt0, uint64_t salt1,
uint64_t id1_in, uint64_t id2_in, uint64_t level,
uint64_t* out_h1, uint64_t* out_h2) {
// Order independence (A -> B links should hash as B -> A links).
uint64_t id_a = (id1_in < id2_in) ? id1_in : id2_in;
uint64_t id_b = (id1_in < id2_in) ? id2_in : id1_in;
// Domain separation: mix salts with a constant to prevent
// related-key attacks.
uint64_t h1 = salt0 ^ 0xDEADBEEFDEADBEEFULL;
uint64_t h2 = salt1 ^ 0xCAFEBABECAFEBABEULL;
// First, thoroughly mix the level into both accumulators
// This prevents predictable level values from being a weakness
uint64_t level_mix = level;
level_mix *= MIX_PRIME_5;
level_mix ^= level_mix >> 32;
level_mix *= MIX_PRIME_6;
h1 ^= level_mix;
h2 ^= ROTL64(level_mix, 31);
// Mix in id_a with strong diffusion.
h1 ^= id_a;
h1 *= MIX_PRIME_1;
h1 = ROTL64(h1, 23);
h1 *= MIX_PRIME_2;
// Mix in id_b.
h2 ^= id_b;
h2 *= MIX_PRIME_3;
h2 = ROTL64(h2, 29);
h2 *= MIX_PRIME_4;
// Three rounds of cross-mixing for better security.
for (int i = 0; i < 3; i++) {
// Cross-influence.
uint64_t tmp = h1;
h1 += h2;
h2 += tmp;
// Mix h1.
h1 ^= ROTL64(h1, 31);
h1 *= MIX_PRIME_1;
h1 ^= salt0;
// Mix h2.
h2 ^= ROTL64(h2, 37);
h2 *= MIX_PRIME_2;
h2 ^= salt1;
}
// Finalization with avalanche rounds.
h1 ^= h1 >> 33;
h1 *= MIX_PRIME_3;
h1 ^= h1 >> 29;
h1 *= MIX_PRIME_4;
h1 ^= h1 >> 32;
h2 ^= h2 >> 33;
h2 *= MIX_PRIME_5;
h2 ^= h2 >> 29;
h2 *= MIX_PRIME_6;
h2 ^= h2 >> 32;
*out_h1 = h1;
*out_h2 = h2;
}
+2
View File
@@ -94,6 +94,7 @@ class TestCase:
self.test_key = f"test:{self.__class__.__name__.lower()}"
# Primary Redis instance
self.redis = redis.Redis(port=primary_port)
self.redis3 = redis.Redis(port=primary_port,protocol=3)
# Replica Redis instance
self.replica = redis.Redis(port=replica_port)
# Replication status
@@ -184,6 +185,7 @@ def find_test_classes(primary_port, replica_port):
# Create test instance with specified ports
test_instance = obj()
test_instance.redis = redis.Redis(port=primary_port)
test_instance.redis3 = redis.Redis(port=primary_port,protocol=3)
test_instance.replica = redis.Redis(port=replica_port)
test_instance.primary_port = primary_port
test_instance.replica_port = replica_port
+214
View File
@@ -0,0 +1,214 @@
from test import TestCase, generate_random_vector
import struct
import json
import random
class VSIMWithAttribs(TestCase):
def getname(self):
return "VSIM WITHATTRIBS/WITHSCORES functionality testing"
def setup(self):
super().setup()
self.dim = 8
self.count = 20
# Create vectors with attributes
for i in range(self.count):
vec = generate_random_vector(self.dim)
vec_bytes = struct.pack(f'{self.dim}f', *vec)
# Item name
name = f"{self.test_key}:item:{i}"
# Add to Redis
self.redis.execute_command('VADD', self.test_key, 'FP32', vec_bytes, name)
# Create and add attribute
if i % 5 == 0:
# Every 5th item has no attribute (for testing NULL responses)
continue
category = random.choice(["electronics", "furniture", "clothing"])
price = random.randint(50, 1000)
attrs = {"category": category, "price": price, "id": i}
self.redis.execute_command('VSETATTR', self.test_key, name, json.dumps(attrs))
def is_numeric(self, value):
"""Check if a value can be converted to float"""
try:
if isinstance(value, (int, float)):
return True
if isinstance(value, bytes):
float(value.decode('utf-8'))
return True
if isinstance(value, str):
float(value)
return True
return False
except (ValueError, TypeError):
return False
def test(self):
# Create query vector
query_vec = generate_random_vector(self.dim)
# Test 1: VSIM with no additional options (should be same for RESP2 and RESP3)
cmd_args = ['VSIM', self.test_key, 'VALUES', self.dim]
cmd_args.extend([str(x) for x in query_vec])
cmd_args.extend(['COUNT', 5])
results_resp2 = self.redis.execute_command(*cmd_args)
results_resp3 = self.redis3.execute_command(*cmd_args)
# Both should return simple arrays of item names
assert len(results_resp2) == 5, f"RESP2: Expected 5 results, got {len(results_resp2)}"
assert len(results_resp3) == 5, f"RESP3: Expected 5 results, got {len(results_resp3)}"
assert all(isinstance(item, bytes) for item in results_resp2), "RESP2: Results should be byte strings"
assert all(isinstance(item, bytes) for item in results_resp3), "RESP3: Results should be byte strings"
# Test 2: VSIM with WITHSCORES only
cmd_args = ['VSIM', self.test_key, 'VALUES', self.dim]
cmd_args.extend([str(x) for x in query_vec])
cmd_args.extend(['COUNT', 5, 'WITHSCORES'])
results_resp2 = self.redis.execute_command(*cmd_args)
results_resp3 = self.redis3.execute_command(*cmd_args)
# RESP2: Should be a flat array alternating item, score
assert len(results_resp2) == 10, f"RESP2: Expected 10 elements (5 items × 2), got {len(results_resp2)}"
for i in range(0, len(results_resp2), 2):
assert isinstance(results_resp2[i], bytes), f"RESP2: Item at {i} should be bytes"
assert self.is_numeric(results_resp2[i+1]), f"RESP2: Score at {i+1} should be numeric"
score = float(results_resp2[i+1]) if isinstance(results_resp2[i+1], bytes) else results_resp2[i+1]
assert 0 <= score <= 1, f"RESP2: Score {score} should be between 0 and 1"
# RESP3: Should be a dict/map with items as keys and scores as DIRECT values (not arrays)
assert isinstance(results_resp3, dict), f"RESP3: Expected dict, got {type(results_resp3)}"
assert len(results_resp3) == 5, f"RESP3: Expected 5 entries, got {len(results_resp3)}"
for item, score in results_resp3.items():
assert isinstance(item, bytes), f"RESP3: Key should be bytes"
# Score should be a direct value, NOT an array
assert not isinstance(score, list), f"RESP3: With single WITH option, value should not be array"
assert self.is_numeric(score), f"RESP3: Score should be numeric, got {type(score)}"
score_val = float(score) if isinstance(score, bytes) else score
assert 0 <= score_val <= 1, f"RESP3: Score {score_val} should be between 0 and 1"
# Test 3: VSIM with WITHATTRIBS only
cmd_args = ['VSIM', self.test_key, 'VALUES', self.dim]
cmd_args.extend([str(x) for x in query_vec])
cmd_args.extend(['COUNT', 5, 'WITHATTRIBS'])
results_resp2 = self.redis.execute_command(*cmd_args)
results_resp3 = self.redis3.execute_command(*cmd_args)
# RESP2: Should be a flat array alternating item, attribute
assert len(results_resp2) == 10, f"RESP2: Expected 10 elements (5 items × 2), got {len(results_resp2)}"
for i in range(0, len(results_resp2), 2):
assert isinstance(results_resp2[i], bytes), f"RESP2: Item at {i} should be bytes"
attr = results_resp2[i+1]
assert attr is None or isinstance(attr, bytes), f"RESP2: Attribute at {i+1} should be None or bytes"
if attr is not None:
# Verify it's valid JSON
json.loads(attr)
# RESP3: Should be a dict/map with items as keys and attributes as DIRECT values (not arrays)
assert isinstance(results_resp3, dict), f"RESP3: Expected dict, got {type(results_resp3)}"
assert len(results_resp3) == 5, f"RESP3: Expected 5 entries, got {len(results_resp3)}"
for item, attr in results_resp3.items():
assert isinstance(item, bytes), f"RESP3: Key should be bytes"
# Attribute should be a direct value, NOT an array
assert not isinstance(attr, list), f"RESP3: With single WITH option, value should not be array"
assert attr is None or isinstance(attr, bytes), f"RESP3: Attribute should be None or bytes"
if attr is not None:
# Verify it's valid JSON
json.loads(attr)
# Test 4: VSIM with both WITHSCORES and WITHATTRIBS
cmd_args = ['VSIM', self.test_key, 'VALUES', self.dim]
cmd_args.extend([str(x) for x in query_vec])
cmd_args.extend(['COUNT', 5, 'WITHSCORES', 'WITHATTRIBS'])
results_resp2 = self.redis.execute_command(*cmd_args)
results_resp3 = self.redis3.execute_command(*cmd_args)
# RESP2: Should be a flat array with pattern: item, score, attribute
assert len(results_resp2) == 15, f"RESP2: Expected 15 elements (5 items × 3), got {len(results_resp2)}"
for i in range(0, len(results_resp2), 3):
assert isinstance(results_resp2[i], bytes), f"RESP2: Item at {i} should be bytes"
assert self.is_numeric(results_resp2[i+1]), f"RESP2: Score at {i+1} should be numeric"
score = float(results_resp2[i+1]) if isinstance(results_resp2[i+1], bytes) else results_resp2[i+1]
assert 0 <= score <= 1, f"RESP2: Score {score} should be between 0 and 1"
attr = results_resp2[i+2]
assert attr is None or isinstance(attr, bytes), f"RESP2: Attribute at {i+2} should be None or bytes"
# RESP3: Should be a dict where each value is a 2-element array [score, attribute]
assert isinstance(results_resp3, dict), f"RESP3: Expected dict, got {type(results_resp3)}"
assert len(results_resp3) == 5, f"RESP3: Expected 5 entries, got {len(results_resp3)}"
for item, value in results_resp3.items():
assert isinstance(item, bytes), f"RESP3: Key should be bytes"
# With BOTH options, value MUST be an array
assert isinstance(value, list), f"RESP3: With both WITH options, value should be a list, got {type(value)}"
assert len(value) == 2, f"RESP3: Value should have 2 elements [score, attr], got {len(value)}"
score, attr = value
assert self.is_numeric(score), f"RESP3: Score should be numeric"
score_val = float(score) if isinstance(score, bytes) else score
assert 0 <= score_val <= 1, f"RESP3: Score {score_val} should be between 0 and 1"
assert attr is None or isinstance(attr, bytes), f"RESP3: Attribute should be None or bytes"
# Test 5: Verify consistency - same items returned in same order
cmd_args = ['VSIM', self.test_key, 'VALUES', self.dim]
cmd_args.extend([str(x) for x in query_vec])
cmd_args.extend(['COUNT', 5, 'WITHSCORES', 'WITHATTRIBS'])
results_resp2 = self.redis.execute_command(*cmd_args)
results_resp3 = self.redis3.execute_command(*cmd_args)
# Extract items from RESP2 (every 3rd element starting from 0)
items_resp2 = [results_resp2[i] for i in range(0, len(results_resp2), 3)]
# Extract items from RESP3 (keys of the dict)
items_resp3 = list(results_resp3.keys())
# Verify same items returned
assert set(items_resp2) == set(items_resp3), "RESP2 and RESP3 should return the same items"
# Build a mapping from items to scores and attributes for comparison
data_resp2 = {}
for i in range(0, len(results_resp2), 3):
item = results_resp2[i]
score = float(results_resp2[i+1]) if isinstance(results_resp2[i+1], bytes) else results_resp2[i+1]
attr = results_resp2[i+2]
data_resp2[item] = (score, attr)
data_resp3 = {}
for item, value in results_resp3.items():
score = float(value[0]) if isinstance(value[0], bytes) else value[0]
attr = value[1]
data_resp3[item] = (score, attr)
# Verify scores and attributes match for each item
for item in data_resp2:
score_resp2, attr_resp2 = data_resp2[item]
score_resp3, attr_resp3 = data_resp3[item]
assert abs(score_resp2 - score_resp3) < 0.0001, \
f"Scores for {item} don't match: RESP2={score_resp2}, RESP3={score_resp3}"
assert attr_resp2 == attr_resp3, \
f"Attributes for {item} don't match: RESP2={attr_resp2}, RESP3={attr_resp3}"
# Test 6: Test ordering of WITHSCORES and WITHATTRIBS doesn't matter
cmd_args1 = ['VSIM', self.test_key, 'VALUES', self.dim]
cmd_args1.extend([str(x) for x in query_vec])
cmd_args1.extend(['COUNT', 3, 'WITHSCORES', 'WITHATTRIBS'])
cmd_args2 = ['VSIM', self.test_key, 'VALUES', self.dim]
cmd_args2.extend([str(x) for x in query_vec])
cmd_args2.extend(['COUNT', 3, 'WITHATTRIBS', 'WITHSCORES']) # Reversed order
results1_resp3 = self.redis3.execute_command(*cmd_args1)
results2_resp3 = self.redis3.execute_command(*cmd_args2)
# Both should return the same structure
assert results1_resp3 == results2_resp3, "Order of WITH options shouldn't matter"
+108 -32
View File
@@ -801,8 +801,8 @@ int vectorSetFilterCallback(void *value, void *privdata) {
* handles the HNSW locking explicitly. */
void VSIM_execute(RedisModuleCtx *ctx, struct vsetObject *vset,
float *vec, unsigned long count, float epsilon, unsigned long withscores,
unsigned long ef, exprstate *filter_expr, unsigned long filter_ef,
int ground_truth)
unsigned long withattribs, unsigned long ef, exprstate *filter_expr,
unsigned long filter_ef, int ground_truth)
{
/* In our scan, we can't just collect 'count' elements as
* if count is small we would explore the graph in an insufficient
@@ -837,28 +837,52 @@ void VSIM_execute(RedisModuleCtx *ctx, struct vsetObject *vset,
}
/* Return results */
if (withscores)
int resp3 = RedisModule_GetContextFlags(ctx) & REDISMODULE_CTX_FLAGS_RESP3;
int reply_with_map = resp3 && (withscores || withattribs);
if (reply_with_map)
RedisModule_ReplyWithMap(ctx, REDISMODULE_POSTPONED_LEN);
else
RedisModule_ReplyWithArray(ctx, REDISMODULE_POSTPONED_LEN);
long long arraylen = 0;
long long arraylen = 0;
for (unsigned int i = 0; i < found && i < count; i++) {
if (distances[i] > epsilon) break;
struct vsetNodeVal *nv = neighbors[i]->value;
RedisModule_ReplyWithString(ctx, nv->item);
arraylen++;
/* If the user asked for multiple properties at the same time using
* the RESP3 protocol, we wrap the value of the map into an N-items
* array. Two for now, since we have just two properties that can be
* requested.
*
* So in the case of RESP2 we will just have the flat reply:
* item, score, attribute. For RESP3 instead item -> [score, attribute]
*/
if (resp3 && withscores && withattribs)
RedisModule_ReplyWithArray(ctx,2);
if (withscores) {
/* The similarity score is provided in a 0-1 range. */
RedisModule_ReplyWithDouble(ctx, 1.0 - distances[i]/2.0);
}
if (withattribs) {
/* Return the attributes as well, if any. */
if (nv->attrib)
RedisModule_ReplyWithString(ctx, nv->attrib);
else
RedisModule_ReplyWithNull(ctx);
}
}
hnsw_release_read_slot(vset->hnsw,slot);
if (withscores)
if (reply_with_map) {
RedisModule_ReplySetMapLength(ctx, arraylen);
else
RedisModule_ReplySetArrayLength(ctx, arraylen);
} else {
int items_per_ele = 1+withattribs+withscores;
RedisModule_ReplySetArrayLength(ctx, arraylen * items_per_ele);
}
RedisModule_Free(vec);
RedisModule_Free(neighbors);
@@ -878,10 +902,11 @@ void *VSIM_thread(void *arg) {
unsigned long count = (unsigned long)targ[3];
float epsilon = *((float*)targ[4]);
unsigned long withscores = (unsigned long)targ[5];
unsigned long ef = (unsigned long)targ[6];
exprstate *filter_expr = targ[7];
unsigned long filter_ef = (unsigned long)targ[8];
unsigned long ground_truth = (unsigned long)targ[9];
unsigned long withattribs = (unsigned long)targ[6];
unsigned long ef = (unsigned long)targ[7];
exprstate *filter_expr = targ[8];
unsigned long filter_ef = (unsigned long)targ[9];
unsigned long ground_truth = (unsigned long)targ[10];
RedisModule_Free(targ[4]);
RedisModule_Free(targ);
@@ -894,7 +919,7 @@ void *VSIM_thread(void *arg) {
RedisModuleCtx *ctx = RedisModule_GetThreadSafeContext(bc);
// Run the query.
VSIM_execute(ctx, vset, vec, count, epsilon, withscores, ef, filter_expr, filter_ef, ground_truth);
VSIM_execute(ctx, vset, vec, count, epsilon, withscores, withattribs, ef, filter_expr, filter_ef, ground_truth);
pthread_rwlock_unlock(&vset->in_use_lock);
// Cleanup.
@@ -904,7 +929,7 @@ void *VSIM_thread(void *arg) {
return NULL;
}
/* VSIM key [ELE|FP32|VALUES] <vector or ele> [WITHSCORES] [COUNT num] [EPSILON eps] [EF exploration-factor] [FILTER expression] [FILTER-EF exploration-factor] */
/* VSIM key [ELE|FP32|VALUES] <vector or ele> [WITHSCORES] [WITHATTRIBS] [COUNT num] [EPSILON eps] [EF exploration-factor] [FILTER expression] [FILTER-EF exploration-factor] */
int VSIM_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
RedisModule_AutoMemory(ctx);
@@ -914,6 +939,7 @@ int VSIM_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
/* Defaults */
int withscores = 0;
int withattribs = 0;
long long count = VSET_DEFAULT_COUNT; /* New default value */
long long ef = 0; /* Exploration factor (see HNSW paper) */
double epsilon = 2.0; /* Max cosine distance */
@@ -1017,6 +1043,9 @@ int VSIM_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
if (!strcasecmp(opt, "WITHSCORES")) {
withscores = 1;
j++;
} else if (!strcasecmp(opt, "WITHATTRIBS")) {
withattribs = 1;
j++;
} else if (!strcasecmp(opt, "TRUTH")) {
ground_truth = 1;
j++;
@@ -1097,7 +1126,7 @@ int VSIM_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
* free slot if all the HNSW_MAX_THREADS slots are used. */
RedisModuleBlockedClient *bc = RedisModule_BlockClient(ctx,NULL,NULL,NULL,0);
pthread_t tid;
void **targ = RedisModule_Alloc(sizeof(void*)*10);
void **targ = RedisModule_Alloc(sizeof(void*)*11);
targ[0] = bc;
targ[1] = vset;
targ[2] = vec;
@@ -1105,10 +1134,11 @@ int VSIM_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
targ[4] = RedisModule_Alloc(sizeof(float));
*((float*)targ[4]) = epsilon;
targ[5] = (void*)(unsigned long)withscores;
targ[6] = (void*)(unsigned long)ef;
targ[7] = (void*)filter_expr;
targ[8] = (void*)(unsigned long)filter_ef;
targ[9] = (void*)(unsigned long)ground_truth;
targ[6] = (void*)(unsigned long)withattribs;
targ[7] = (void*)(unsigned long)ef;
targ[8] = (void*)filter_expr;
targ[9] = (void*)(unsigned long)filter_ef;
targ[10] = (void*)(unsigned long)ground_truth;
RedisModule_BlockedClientMeasureTimeStart(bc);
vset->thread_creation_pending++;
if (pthread_create(&tid,NULL,VSIM_thread,targ) != 0) {
@@ -1116,10 +1146,10 @@ int VSIM_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
RedisModule_AbortBlock(bc);
RedisModule_Free(targ[4]);
RedisModule_Free(targ);
VSIM_execute(ctx, vset, vec, count, epsilon, withscores, ef, filter_expr, filter_ef, ground_truth);
VSIM_execute(ctx, vset, vec, count, epsilon, withscores, withattribs, ef, filter_expr, filter_ef, ground_truth);
}
} else {
VSIM_execute(ctx, vset, vec, count, epsilon, withscores, ef, filter_expr, filter_ef, ground_truth);
VSIM_execute(ctx, vset, vec, count, epsilon, withscores, withattribs, ef, filter_expr, filter_ef, ground_truth);
}
return REDISMODULE_OK;
@@ -1734,18 +1764,24 @@ void VectorSetRdbSave(RedisModuleIO *rdb, void *value) {
}
}
/* Load object from RDB. Please note that we don't do any cleanup
* on errors, and just return NULL, as Redis will abort completely
* not just the module but the server itself in this case. */
/* Load object from RDB. Recover from recoverable errors (read errors)
* by performing cleanup. */
void *VectorSetRdbLoad(RedisModuleIO *rdb, int encver) {
if (encver != 0) return NULL; // Invalid version
uint32_t dim = RedisModule_LoadUnsigned(rdb);
uint64_t elements = RedisModule_LoadUnsigned(rdb);
uint32_t hnsw_config = RedisModule_LoadUnsigned(rdb);
if (RedisModule_IsIOError(rdb)) return NULL;
uint32_t quant_type = hnsw_config & 0xff;
uint32_t hnsw_m = (hnsw_config >> 8) & 0xffff;
/* Check that the quantization type is correct. Otherwise
* return ASAP signaling the error. */
if (quant_type != HNSW_QUANT_NONE &&
quant_type != HNSW_QUANT_Q8 &&
quant_type != HNSW_QUANT_BIN) return NULL;
if (hnsw_m == 0) hnsw_m = 16; // Default, useful for RDB files predating
// this configuration parameter: it was fixed
// to 16.
@@ -1754,22 +1790,21 @@ void *VectorSetRdbLoad(RedisModuleIO *rdb, int encver) {
/* Load projection matrix if present */
uint32_t save_flags = RedisModule_LoadUnsigned(rdb);
if (RedisModule_IsIOError(rdb)) goto ioerr;
int has_projection = save_flags & SAVE_FLAG_HAS_PROJMATRIX;
int has_attribs = save_flags & SAVE_FLAG_HAS_ATTRIBS;
if (has_projection) {
uint32_t input_dim = RedisModule_LoadUnsigned(rdb);
if (RedisModule_IsIOError(rdb)) goto ioerr;
uint32_t output_dim = dim;
size_t matrix_size = sizeof(float) * input_dim * output_dim;
vset->proj_matrix = RedisModule_Alloc(matrix_size);
if (!vset->proj_matrix) {
vectorSetReleaseObject(vset);
return NULL;
}
vset->proj_input_size = input_dim;
// Load projection matrix as a binary blob
char *matrix_blob = RedisModule_LoadStringBuffer(rdb, NULL);
if (matrix_blob == NULL) goto ioerr;
memcpy(vset->proj_matrix, matrix_blob, matrix_size);
RedisModule_Free(matrix_blob);
}
@@ -1777,9 +1812,14 @@ void *VectorSetRdbLoad(RedisModuleIO *rdb, int encver) {
while(elements--) {
// Load associated string element.
RedisModuleString *ele = RedisModule_LoadString(rdb);
if (RedisModule_IsIOError(rdb)) goto ioerr;
RedisModuleString *attrib = NULL;
if (has_attribs) {
attrib = RedisModule_LoadString(rdb);
if (RedisModule_IsIOError(rdb)) {
RedisModule_FreeString(NULL,ele);
goto ioerr;
}
size_t attrlen;
RedisModule_StringPtrLen(attrib,&attrlen);
if (attrlen == 0) {
@@ -1789,18 +1829,42 @@ void *VectorSetRdbLoad(RedisModuleIO *rdb, int encver) {
}
size_t vector_len;
void *vector = RedisModule_LoadStringBuffer(rdb, &vector_len);
if (RedisModule_IsIOError(rdb)) {
RedisModule_FreeString(NULL,ele);
if (attrib) RedisModule_FreeString(NULL,attrib);
goto ioerr;
}
uint32_t vector_bytes = hnsw_quants_bytes(vset->hnsw);
if (vector_len != vector_bytes) {
RedisModule_LogIOError(rdb,"warning",
"Mismatching vector dimension");
return NULL; // Loading error.
RedisModule_FreeString(NULL,ele);
if (attrib) RedisModule_FreeString(NULL,attrib);
RedisModule_Free(vector);
goto ioerr;
}
// Load node parameters back.
uint32_t params_count = RedisModule_LoadUnsigned(rdb);
if (RedisModule_IsIOError(rdb)) {
RedisModule_FreeString(NULL,ele);
if (attrib) RedisModule_FreeString(NULL,attrib);
RedisModule_Free(vector);
goto ioerr;
}
uint64_t *params = RedisModule_Alloc(params_count*sizeof(uint64_t));
for (uint32_t j = 0; j < params_count; j++)
for (uint32_t j = 0; j < params_count; j++) {
// Ignore loading errors here: handled at the end of the loop.
params[j] = RedisModule_LoadUnsigned(rdb);
}
if (RedisModule_IsIOError(rdb)) {
RedisModule_FreeString(NULL,ele);
if (attrib) RedisModule_FreeString(NULL,attrib);
RedisModule_Free(vector);
RedisModule_Free(params);
goto ioerr;
}
struct vsetNodeVal *nv = RedisModule_Alloc(sizeof(*nv));
nv->item = ele;
@@ -1809,15 +1873,28 @@ void *VectorSetRdbLoad(RedisModuleIO *rdb, int encver) {
if (node == NULL) {
RedisModule_LogIOError(rdb,"warning",
"Vector set node index loading error");
return NULL; // Loading error.
vectorSetReleaseNodeValue(nv);
RedisModule_Free(vector);
RedisModule_Free(params);
goto ioerr;
}
if (nv->attrib) vset->numattribs++;
RedisModule_DictSet(vset->dict,ele,node);
RedisModule_Free(vector);
RedisModule_Free(params);
}
hnsw_deserialize_index(vset->hnsw);
uint64_t salt[2];
RedisModule_GetRandomBytes((unsigned char*)salt,sizeof(salt));
if (!hnsw_deserialize_index(vset->hnsw, salt[0], salt[1])) goto ioerr;
return vset;
ioerr:
/* We want to recover from I/O errors and free the partially allocated
* data structure to support diskless replication. */
vectorSetReleaseObject(vset);
return NULL;
}
/* Calculate memory usage */
@@ -1944,7 +2021,6 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
if (RedisModule_Init(ctx,"vectorset",1,REDISMODULE_APIVER_1)
== REDISMODULE_ERR) return REDISMODULE_ERR;
/* TODO: Added to pass CI, need to make changes in order to support these options */
RedisModule_SetModuleOptions(ctx, REDISMODULE_OPTIONS_HANDLE_IO_ERRORS|REDISMODULE_OPTIONS_HANDLE_REPL_ASYNC_LOAD);
RedisModuleTypeMethods tm = {
+1 -1
View File
@@ -363,7 +363,7 @@ endif
REDIS_SERVER_NAME=redis-server$(PROG_SUFFIX)
REDIS_SENTINEL_NAME=redis-sentinel$(PROG_SUFFIX)
REDIS_SERVER_OBJ=threads_mngr.o adlist.o quicklist.o ae.o anet.o dict.o ebuckets.o eventnotifier.o iothread.o mstr.o kvstore.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o cluster_legacy.o crc16.o endianconv.o slowlog.o eval.o bio.o rio.o rand.o memtest.o syscheck.o crcspeed.o crccombine.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o redis-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o lolwut.o lolwut5.o lolwut6.o acl.o tracking.o socket.o tls.o sha256.o timeout.o setcpuaffinity.o monotonic.o mt19937-64.o resp_parser.o call_reply.o script_lua.o script.o functions.o function_lua.o commands.o strl.o connection.o unix.o logreqres.o
REDIS_SERVER_OBJ=threads_mngr.o adlist.o quicklist.o ae.o anet.o dict.o ebuckets.o eventnotifier.o iothread.o mstr.o kvstore.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o cluster_legacy.o crc16.o endianconv.o slowlog.o eval.o bio.o rio.o rand.o memtest.o syscheck.o crcspeed.o crccombine.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o redis-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o lolwut.o lolwut5.o lolwut6.o lolwut8.o acl.o tracking.o socket.o tls.o sha256.o timeout.o setcpuaffinity.o monotonic.o mt19937-64.o resp_parser.o call_reply.o script_lua.o script.o functions.o function_lua.o commands.o strl.o connection.o unix.o logreqres.o
REDIS_CLI_NAME=redis-cli$(PROG_SUFFIX)
REDIS_CLI_OBJ=anet.o adlist.o dict.o redis-cli.o zmalloc.o release.o ae.o redisassert.o crcspeed.o crccombine.o crc64.o siphash.o crc16.o monotonic.o cli_common.o mt19937-64.o strl.o cli_commands.o
REDIS_BENCHMARK_NAME=redis-benchmark$(PROG_SUFFIX)
+3 -1
View File
@@ -216,7 +216,9 @@ static void aeApiDelEvent(aeEventLoop *eventLoop, int fd, int mask) {
* the fact that our caller has already updated the mask in the eventLoop.
*/
fullmask = eventLoop->events[fd].mask;
/* We always remove the specified events from the current mask,
* regardless of whether eventLoop->events[fd].mask has been updated yet. */
fullmask = eventLoop->events[fd].mask & ~mask;
if (fullmask == AE_NONE) {
/*
* We're removing *all* events, so use port_dissociate to remove the
+24
View File
@@ -787,3 +787,27 @@ int anetIsFifo(char *filepath) {
if (stat(filepath, &sb) == -1) return 0;
return S_ISFIFO(sb.st_mode);
}
/* This function must be called after accept4() fails. It returns 1 if 'err'
* indicates accepted connection faced an error, and it's okay to continue
* accepting next connection by calling accept4() again. Other errors either
* indicate programming errors, e.g. calling accept() on a closed fd or indicate
* a resource limit has been reached, e.g. -EMFILE, open fd limit has been
* reached. In the latter case, caller might wait until resources are available.
* See accept4() documentation for details. */
int anetAcceptFailureNeedsRetry(int err) {
if (err == ECONNABORTED)
return 1;
#if defined(__linux__)
/* For details, see 'Error Handling' section on
* https://man7.org/linux/man-pages/man2/accept.2.html */
if (err == ENETDOWN || err == EPROTO || err == ENOPROTOOPT ||
err == EHOSTDOWN || err == ENONET || err == EHOSTUNREACH ||
err == EOPNOTSUPP || err == ENETUNREACH)
{
return 1;
}
#endif
return 0;
}
+1
View File
@@ -53,5 +53,6 @@ int anetPipe(int fds[2], int read_flags, int write_flags);
int anetSetSockMarkId(char *err, int fd, uint32_t id);
int anetGetError(int fd);
int anetIsFifo(char *filepath);
int anetAcceptFailureNeedsRetry(int err);
#endif
+2 -2
View File
@@ -2579,9 +2579,9 @@ int rewriteAppendOnlyFileBackground(void) {
serverLog(LL_NOTICE,
"Successfully created the temporary AOF base file %s", tmpfile);
sendChildCowInfo(CHILD_INFO_TYPE_AOF_COW_SIZE, "AOF rewrite");
exitFromChild(0);
exitFromChild(0, 0);
} else {
exitFromChild(1);
exitFromChild(1, 0);
}
} else {
/* Parent */
+1 -1
View File
@@ -94,7 +94,7 @@ void sendChildInfoGeneric(childInfoType info_type, size_t keys, double progress,
if (write(server.child_info_pipe[1], &data, wlen) != wlen) {
/* Failed writing to parent, it could have been killed, exit. */
serverLog(LL_WARNING,"Child failed reporting info to parent, exiting. %s", strerror(errno));
exitFromChild(1);
exitFromChild(1, 0);
}
}
+2
View File
@@ -1253,6 +1253,8 @@ void clusterAcceptHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
while(max--) {
cfd = anetTcpAccept(server.neterr, fd, cip, sizeof(cip), &cport);
if (cfd == ANET_ERR) {
if (anetAcceptFailureNeedsRetry(errno))
continue;
if (errno != EWOULDBLOCK)
serverLog(LL_VERBOSE,
"Error accepting cluster node: %s", server.neterr);
+1 -7
View File
@@ -1187,7 +1187,7 @@ static doneStatus defragStageDbKeys(void *ctx, monotime endtime) {
static doneStatus defragStageExpiresKvstore(void *ctx, monotime endtime) {
defragKeysCtx *defrag_keys_ctx = ctx;
redisDb *db = &server.db[defrag_keys_ctx->dbid];
if (db->keys != defrag_keys_ctx->kvstate.kvs) {
if (db->expires != defrag_keys_ctx->kvstate.kvs) {
/* There has been a change of the kvs (flushdb, swapdb, etc.). Just complete the stage. */
return DEFRAG_DONE;
}
@@ -1445,12 +1445,6 @@ static int activeDefragTimeProc(struct aeEventLoop *eventLoop, long long id, voi
monotime endtime = starttime + dutyCycleUs;
int haveMoreWork = 1;
/* Increment server.cronloops so that run_with_period works. */
long hz_ms = 1000 / server.hz;
int cronloops = (server.mstime - server.blocked_last_cron + (hz_ms - 1)) / hz_ms; /* rounding up */
server.blocked_last_cron += cronloops * hz_ms;
server.cronloops += cronloops;
mstime_t latency;
latencyStartMonitor(latency);
+1 -1
View File
@@ -920,7 +920,7 @@ void ldbEndSession(client *c) {
if (ldb.forked) {
writeToClient(c,0);
serverLog(LL_NOTICE,"Lua debugging session child exiting");
exitFromChild(0);
exitFromChild(0, 0);
} else {
serverLog(LL_NOTICE,
"Redis synchronous debugging eval session ended");
+42 -5
View File
@@ -579,6 +579,7 @@ int hllSparseToDense(robj *o) {
struct hllhdr *hdr, *oldhdr = (struct hllhdr*)sparse;
int idx = 0, runlen, regval;
uint8_t *p = (uint8_t*)sparse, *end = p+sdslen(sparse);
int valid = 1;
/* If the representation is already the right one return ASAP. */
hdr = (struct hllhdr*) sparse;
@@ -598,16 +599,27 @@ int hllSparseToDense(robj *o) {
while(p < end) {
if (HLL_SPARSE_IS_ZERO(p)) {
runlen = HLL_SPARSE_ZERO_LEN(p);
if ((runlen + idx) > HLL_REGISTERS) { /* Overflow. */
valid = 0;
break;
}
idx += runlen;
p++;
} else if (HLL_SPARSE_IS_XZERO(p)) {
runlen = HLL_SPARSE_XZERO_LEN(p);
if ((runlen + idx) > HLL_REGISTERS) { /* Overflow. */
valid = 0;
break;
}
idx += runlen;
p += 2;
} else {
runlen = HLL_SPARSE_VAL_LEN(p);
regval = HLL_SPARSE_VAL_VALUE(p);
if ((runlen + idx) > HLL_REGISTERS) break; /* Overflow. */
if ((runlen + idx) > HLL_REGISTERS) { /* Overflow. */
valid = 0;
break;
}
while(runlen--) {
HLL_DENSE_SET_REGISTER(hdr->registers,idx,regval);
idx++;
@@ -618,7 +630,7 @@ int hllSparseToDense(robj *o) {
/* If the sparse representation was valid, we expect to find idx
* set to HLL_REGISTERS. */
if (idx != HLL_REGISTERS) {
if (!valid || idx != HLL_REGISTERS) {
sdsfree(dense);
return C_ERR;
}
@@ -915,27 +927,40 @@ int hllSparseAdd(robj *o, unsigned char *ele, size_t elesize) {
void hllSparseRegHisto(uint8_t *sparse, int sparselen, int *invalid, int* reghisto) {
int idx = 0, runlen, regval;
uint8_t *end = sparse+sparselen, *p = sparse;
int valid = 1;
while(p < end) {
if (HLL_SPARSE_IS_ZERO(p)) {
runlen = HLL_SPARSE_ZERO_LEN(p);
if ((runlen + idx) > HLL_REGISTERS) { /* Overflow. */
valid = 0;
break;
}
idx += runlen;
reghisto[0] += runlen;
p++;
} else if (HLL_SPARSE_IS_XZERO(p)) {
runlen = HLL_SPARSE_XZERO_LEN(p);
if ((runlen + idx) > HLL_REGISTERS) { /* Overflow. */
valid = 0;
break;
}
idx += runlen;
reghisto[0] += runlen;
p += 2;
} else {
runlen = HLL_SPARSE_VAL_LEN(p);
regval = HLL_SPARSE_VAL_VALUE(p);
if ((runlen + idx) > HLL_REGISTERS) { /* Overflow. */
valid = 0;
break;
}
idx += runlen;
reghisto[regval] += runlen;
p++;
}
}
if (idx != HLL_REGISTERS && invalid) *invalid = 1;
if ((!valid || idx != HLL_REGISTERS) && invalid) *invalid = 1;
}
/* ========================= HyperLogLog Count ==============================
@@ -1204,22 +1229,34 @@ int hllMerge(uint8_t *max, robj *hll) {
} else {
uint8_t *p = hll->ptr, *end = p + sdslen(hll->ptr);
long runlen, regval;
int valid = 1;
p += HLL_HDR_SIZE;
i = 0;
while(p < end) {
if (HLL_SPARSE_IS_ZERO(p)) {
runlen = HLL_SPARSE_ZERO_LEN(p);
if ((runlen + i) > HLL_REGISTERS) { /* Overflow. */
valid = 0;
break;
}
i += runlen;
p++;
} else if (HLL_SPARSE_IS_XZERO(p)) {
runlen = HLL_SPARSE_XZERO_LEN(p);
if ((runlen + i) > HLL_REGISTERS) { /* Overflow. */
valid = 0;
break;
}
i += runlen;
p += 2;
} else {
runlen = HLL_SPARSE_VAL_LEN(p);
regval = HLL_SPARSE_VAL_VALUE(p);
if ((runlen + i) > HLL_REGISTERS) break; /* Overflow. */
if ((runlen + i) > HLL_REGISTERS) { /* Overflow. */
valid = 0;
break;
}
while(runlen--) {
if (regval > max[i]) max[i] = regval;
i++;
@@ -1227,7 +1264,7 @@ int hllMerge(uint8_t *max, robj *hll) {
p++;
}
}
if (i != HLL_REGISTERS) return C_ERR;
if (!valid || i != HLL_REGISTERS) return C_ERR;
}
return C_OK;
}
+4
View File
@@ -19,6 +19,7 @@
void lolwut5Command(client *c);
void lolwut6Command(client *c);
void lolwut8Command(client *c);
/* The default target for LOLWUT if no matching version was found.
* This is what unstable versions of Redis will display. */
@@ -54,6 +55,9 @@ void lolwutCommand(client *c) {
else if ((v[0] == '6' && v[1] == '.' && v[2] != '9') ||
(v[0] == '5' && v[1] == '.' && v[2] == '9'))
lolwut6Command(c);
else if ((v[0] == '8' && v[1] == '.' && v[2] != '9') ||
(v[0] == '7' && v[1] == '.' && v[2] == '9'))
lolwut8Command(c);
else
lolwutUnstableCommand(c);
+177
View File
@@ -0,0 +1,177 @@
/*
* Copyright (c) 2025-Present, Redis Ltd.
* All rights reserved.
*
* Licensed under your choice of (a) the Redis Source Available License 2.0
* (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the
* GNU Affero General Public License v3 (AGPLv3).
*
* Originally authored by: Salvatore Sanfilippo.
* Algorithm based on the Almanacco Bompiani description and the Python
* code written by Emiliano Russo.
*/
#include "server.h"
#include <ctype.h>
/* The LOLWUT 8 command:
*
* LOLWUT [EN|IT]
*
* By default the command produces verses in English language, in order for
* the output to be more universally accessible. However, passing IT as argument
* it is possible to reproduce the original output, exactly like done by
* Nanni Balestrini in TAPE MARK I, and described in the Almanacco Letterario
* Bompiani, 1962.
*/
// Structure to represent a verse with its metrical characteristics.
typedef struct {
char text_en[100]; // English verse text.
char text_it[100]; // Italian verse text.
char fraction1[5]; // First fraction (rhythm/meter indicator).
char fraction2[5]; // Second fraction (rhythm/meter indicator).
char group[2]; // Group number (1-3 representing different
// literary sources).
} Verse;
// Fisher-Yates shuffle algorithm to randomize verse order.
static void shuffle(Verse *array, int size) {
for (int i = size - 1; i > 0; i--) {
int j = rand() % (i + 1);
Verse temp = array[j];
array[j] = array[i];
array[i] = temp;
}
}
void lolwut8Command(client *c) {
int en_lang = 1; // Default to English.
/* Parse the optional arguments if any. */
if (c->argc > 1 && !strcasecmp(c->argv[1]->ptr,"IT"))
en_lang = 0;
// Define verses from three literary sources with their metrical fractions:
// Group 1: Diary of Hiroshima by Michihito Hachiya.
// Group 2: The Mystery of the Elevator by Paul Goldwin.
// Group 3: Tao Te Ching by Lao Tse.
Verse verses[] = {
// Group 1: Hiroshima verses.
{" The blinding / globe / of fire ",
" l accecante / globo / di fuoco ", "1/4", "2/3", "1"},
{" It expands / rapidly ",
" si espande / rapidamente ", "1/2", "3/4", "1"},
{" Thirty times / brighter / than the sun ",
" trenta volte / piu luminoso / del sole ", "2/3", "2/4", "1"},
{" When it reaches / the stratosphere ",
" quando raggiunge / la stratosfera ", "3/4", "1/2", "1"},
{" The summit / of the cloud ",
" la sommita / della nuvola ", "1/3", "2/3", "1"},
{" Assumes / the well-known shape / of a mushroom ",
" assume / la ben nota forma / di fungo ", "2/4", "3/4", "1"},
// Group 2: Elevator mystery verses.
{" The head / pressed / upon the shoulder ",
" la testa / premuta / sulla spalla ", "1/4", "2/4", "2"},
{" The hair / between the lips ",
" i capelli / tra le labbra ", "1/4", "2/4", "2"},
{" They lay / motionless / without speaking ",
" giacquero / immobili / senza parlare ", "2/3", "2/3", "2"},
{" Till he moved / his fingers / slowly ",
" finche non mosse / le dita / lentamente ", "3/4", "1/3", "2"},
{" Trying / to grasp ",
" cercando / di afferrare ", "3/4", "1/2", "2"},
// Group 3: Tao Te Ching verses.
{" While the multitude / of things / comes into being ",
" mentre la moltitudine / delle cose / accade ", "1/2", "1/2", "3"},
{" I envisage / their return ",
" io contemplo / il loro ritorno ", "2/3", "3/4", "3"},
{" Although / things / flourish ",
" malgrado / che le cose / fioriscano ", "1/2", "2/3", "3"},
{" They all return / to / their roots ",
" esse tornano / tutte / alla loro radice ", "2/3", "1/4", "3"}
};
// Calculate the total number of verses.
int num_verses = sizeof(verses) / sizeof(verses[0]);
// Create a working copy of verses for manipulation.
Verse *working_verses = zmalloc(num_verses * sizeof(Verse));
memcpy(working_verses, verses, num_verses * sizeof(Verse));
// Step 1: Shuffle the verses randomly.
shuffle(working_verses, num_verses);
// Step 2: Build stanza by finding compatible verses
// Each subsequent verse must:
// - Have compatible metrical fractions (connecting criteria).
// - Belong to a different group than the previous verse.
Verse stanza[10];
int j; // At the end, it will contain the number of added stanzas.
for (j = 0; j < 10; j++) {
int i = 0;
int found = 0;
// Search for compatible verse among remaining verses.
while (i < num_verses) {
// Metrical compatibility check: this is used to select verses
// that go somewhat well together, if their fractions match.
// The algorithm checks if current verse's first fraction matches
// with previous verse's second fraction in various ways, and
// force successive verses to be of different groups.
if (j == 0 || // First stanza is always accepted.
((working_verses[i].fraction1[0] == stanza[j-1].fraction2[0] ||
working_verses[i].fraction1[2] == stanza[j-1].fraction2[0] ||
working_verses[i].fraction1[2] == stanza[j-1].fraction2[2]) &&
strcmp(working_verses[i].group, stanza[j-1].group) != 0))
{
// Add compatible verse to stanza.
stanza[j] = working_verses[i];
// Remove selected verse from working set, to avoid reuse.
for (int k = i; k < num_verses - 1; k++)
working_verses[k] = working_verses[k + 1];
num_verses--;
found = 1;
break;
}
i++;
}
// Exit if there are no longer matching verses.
if (!found) break;
}
zfree(working_verses);
// Step 3: Combine all stanza verses into single SDS string.
sds combined = sdsempty();
for (int i = 0; i < j; i++) {
if (en_lang) {
combined = sdscat(combined, stanza[i].text_en);
} else {
combined = sdscat(combined, stanza[i].text_it);
}
combined = sdscat(combined, "\n");
}
// Step 4: Make uppercase, and strip the "/".
for (size_t j = 0; j < sdslen(combined); j++) {
combined[j] = toupper(combined[j]);
if (combined[j] == '/') combined[j] = ' ';
}
// Step 5: Add background info about what the user just saw.
combined = sdscat(combined,
"\nIn 1961, Nanni Balestrini created one of the first computer-generated poems, TAPE MARK I, using an IBM 7090 mainframe. Each execution combined verses from three literary sources following algorithmic rules based on metrical compatibility and group constraints. This LOLWUT command reproduces Balestrini's original algorithm, generating new stanzas through the same computational poetry process described in Almanacco Letterario Bompiani, 1962.\n\n"
"https://en.wikipedia.org/wiki/Digital_poetry\n"
"https://www.youtube.com/watch?v=8i7uFCK7G0o (English subs)\n\n"
"Use: LOLWUT IT for the original Italian output.\n\n");
addReplyVerbatim(c,combined,sdslen(combined),"txt");
sdsfree(combined);
}
+1 -1
View File
@@ -11445,7 +11445,7 @@ void RM_SendChildHeartbeat(double progress) {
*/
int RM_ExitFromChild(int retcode) {
sendChildCowInfo(CHILD_INFO_TYPE_MODULE_COW_SIZE, "Module fork");
exitFromChild(retcode);
exitFromChild(retcode, 0);
return REDISMODULE_OK;
}
+2 -2
View File
@@ -1658,7 +1658,7 @@ int rdbSaveBackground(int req, char *filename, rdbSaveInfo *rsi, int rdbflags) {
if (retval == C_OK) {
sendChildCowInfo(CHILD_INFO_TYPE_RDB_COW_SIZE, "RDB");
}
exitFromChild((retval == C_OK) ? 0 : 1);
exitFromChild((retval == C_OK) ? 0 : 1, 0);
} else {
/* Parent */
if (childpid == -1) {
@@ -3978,7 +3978,7 @@ int rdbSaveToSlavesSockets(int req, rdbSaveInfo *rsi) {
UNUSED(dummy);
}
zfree(conns);
exitFromChild((retval == C_OK) ? 0 : 1);
exitFromChild((retval == C_OK) ? 0 : 1, 0);
} else {
/* Parent */
if (childpid == -1) {
+6
View File
@@ -556,6 +556,12 @@ int redis_check_aof_main(int argc, char **argv) {
goto invalid_args;
}
/* Check if filepath is longer than PATH_MAX */
if (strlen(filepath) > PATH_MAX) {
printf("Error: filepath is too long (exceeds PATH_MAX)\n");
goto invalid_args;
}
/* In the glibc implementation dirname may modify their argument. */
memcpy(temp_filepath, filepath, strlen(filepath) + 1);
dirpath = dirname(temp_filepath);
+6 -5
View File
@@ -23,7 +23,6 @@
#define RIO_FLAG_READ_ERROR (1<<0)
#define RIO_FLAG_WRITE_ERROR (1<<1)
#define RIO_FLAG_ABORT (1<<2)
#define RIO_TYPE_FILE (1<<0)
#define RIO_TYPE_BUFFER (1<<1)
@@ -103,7 +102,7 @@ typedef struct _rio rio;
* if needed. */
static inline size_t rioWrite(rio *r, const void *buf, size_t len) {
if (r->flags & (RIO_FLAG_WRITE_ERROR | RIO_FLAG_ABORT)) return 0;
if (r->flags & (RIO_FLAG_WRITE_ERROR)) return 0;
while (len) {
size_t bytes_to_write = (r->max_processing_chunk && r->max_processing_chunk < len) ? r->max_processing_chunk : len;
if (r->update_cksum) r->update_cksum(r,buf,bytes_to_write);
@@ -119,7 +118,7 @@ static inline size_t rioWrite(rio *r, const void *buf, size_t len) {
}
static inline size_t rioRead(rio *r, void *buf, size_t len) {
if (r->flags & (RIO_FLAG_READ_ERROR | RIO_FLAG_ABORT)) return 0;
if (r->flags & (RIO_FLAG_READ_ERROR)) return 0;
while (len) {
size_t bytes_to_read = (r->max_processing_chunk && r->max_processing_chunk < len) ? r->max_processing_chunk : len;
if (r->read(r,buf,bytes_to_read) == 0) {
@@ -142,8 +141,10 @@ static inline int rioFlush(rio *r) {
return r->flush(r);
}
/* Abort RIO asynchronously by setting read and write error flags. Subsequent
* rioRead()/rioWrite() calls will fail, letting the caller terminate safely. */
static inline void rioAbort(rio *r) {
r->flags |= RIO_FLAG_ABORT;
r->flags |= (RIO_FLAG_READ_ERROR | RIO_FLAG_WRITE_ERROR);
}
/* This function allows to know if there was a read error in any past
@@ -159,7 +160,7 @@ static inline int rioGetWriteError(rio *r) {
}
static inline void rioClearErrors(rio *r) {
r->flags &= ~(RIO_FLAG_READ_ERROR|RIO_FLAG_WRITE_ERROR|RIO_FLAG_ABORT);
r->flags &= ~(RIO_FLAG_READ_ERROR|RIO_FLAG_WRITE_ERROR);
}
void rioInitWithFile(rio *r, FILE *fp);
+21 -4
View File
@@ -257,11 +257,19 @@ mstime_t commandTimeSnapshot(void) {
/* After an RDB dump or AOF rewrite we exit from children using _exit() instead of
* exit(), because the latter may interact with the same file objects used by
* the parent process. However if we are testing the coverage normal exit() is
* used in order to obtain the right coverage information. */
void exitFromChild(int retcode) {
* used in order to obtain the right coverage information.
* There is a caveat for when we exit due to a signal.
* In this case we want the function to be async signal safe, so we can't use exit()
*/
void exitFromChild(int retcode, int from_signal) {
#ifdef COVERAGE_TEST
exit(retcode);
if (!from_signal) {
exit(retcode);
} else {
_exit(retcode);
}
#else
UNUSED(from_signal);
_exit(retcode);
#endif
}
@@ -1663,6 +1671,12 @@ void whileBlockedCron(void) {
if (server.blocked_last_cron >= server.mstime)
return;
/* Increment server.cronloops so that run_with_period works. */
long hz_ms = 1000 / server.hz;
int cronloops = (server.mstime - server.blocked_last_cron + (hz_ms - 1)) / hz_ms; /* rounding up */
server.blocked_last_cron += cronloops * hz_ms;
server.cronloops += cronloops;
mstime_t latency;
latencyStartMonitor(latency);
@@ -6721,7 +6735,10 @@ static void sigKillChildHandler(int sig) {
UNUSED(sig);
int level = server.in_fork_child == CHILD_TYPE_MODULE? LL_VERBOSE: LL_WARNING;
serverLogRawFromHandler(level, "Received SIGUSR1 in child, exiting now.");
exitFromChild(SERVER_CHILD_NOERROR_RETVAL);
/* We don't want to perform any IO in the child when the parent is terminating us.
* We don't know what our stack trace is, it is possible that we were called during an IO operation
* If we were to do another IO operation, we might end up in a deadlock */
exitFromChild(SERVER_CHILD_NOERROR_RETVAL, 1);
}
void setupChildSignalHandlers(void) {
+1 -1
View File
@@ -2731,7 +2731,7 @@ mstime_t commandTimeSnapshot(void);
void getRandomHexChars(char *p, size_t len);
void getRandomBytes(unsigned char *p, size_t len);
uint64_t crc64(uint64_t crc, const unsigned char *s, uint64_t l);
void exitFromChild(int retcode);
void exitFromChild(int retcode, int from_signal);
long long redisPopcount(void *s, long count);
int redisSetProcTitle(char *title);
int validateProcTitleTemplate(const char *template);
+2
View File
@@ -308,6 +308,8 @@ static void connSocketAcceptHandler(aeEventLoop *el, int fd, void *privdata, int
while(max--) {
cfd = anetTcpAccept(server.neterr, fd, cip, sizeof(cip), &cport);
if (cfd == ANET_ERR) {
if (anetAcceptFailureNeedsRetry(errno))
continue;
if (errno != EWOULDBLOCK)
serverLog(LL_WARNING,
"Accepting client connection: %s", server.neterr);
+2
View File
@@ -771,6 +771,8 @@ static void tlsAcceptHandler(aeEventLoop *el, int fd, void *privdata, int mask)
while(max--) {
cfd = anetTcpAccept(server.neterr, fd, cip, sizeof(cip), &cport);
if (cfd == ANET_ERR) {
if (anetAcceptFailureNeedsRetry(errno))
continue;
if (errno != EWOULDBLOCK)
serverLog(LL_WARNING,
"Accepting client connection: %s", server.neterr);
+2
View File
@@ -102,6 +102,8 @@ static void connUnixAcceptHandler(aeEventLoop *el, int fd, void *privdata, int m
while(max--) {
cfd = anetUnixAccept(server.neterr, fd);
if (cfd == ANET_ERR) {
if (anetAcceptFailureNeedsRetry(errno))
continue;
if (errno != EWOULDBLOCK)
serverLog(LL_WARNING,
"Accepting client connection: %s", server.neterr);
+2 -2
View File
@@ -1,2 +1,2 @@
#define REDIS_VERSION "255.255.255"
#define REDIS_VERSION_NUM 0x00ffffff
#define REDIS_VERSION "8.0.3"
#define REDIS_VERSION_NUM 0x00080003
+8 -1
View File
@@ -15,6 +15,10 @@ if { ! [ catch {
proc generate_collections {suffix elements} {
set rd [redis_deferring_client]
set numcmd 7
set has_vsets [server_has_command vadd]
if {$has_vsets} {incr numcmd}
for {set j 0} {$j < $elements} {incr j} {
# add both string values and integers
if {$j % 2 == 0} {set val $j} else {set val "_$j"}
@@ -25,8 +29,11 @@ proc generate_collections {suffix elements} {
$rd zadd zset$suffix $j $val
$rd sadd set$suffix $val
$rd xadd stream$suffix * item 1 value $val
if {$has_vsets} {
$rd vadd vset$suffix VALUES 3 1 1 1 $j
}
}
for {set j 0} {$j < $elements * 7} {incr j} {
for {set j 0} {$j < $elements * $numcmd} {incr j} {
$rd read ; # Discard replies
}
$rd close
+11
View File
@@ -931,5 +931,16 @@ test {corrupt payload: hash listpack encoded with invalid length causes hscan to
}
}
test {corrupt payload: fuzzer findings - vector sets with wrong encoding} {
start_server [list overrides [list loglevel verbose use-exit-on-panic yes crash-memcheck-enabled no] ] {
r config set sanitize-dump-payload yes
r debug set-skip-checksum-validation 1
catch {r restore _key 0 "\x07\x81\xBD\xE7\x2D\xA2\xBB\x1E\xB4\x00\x02\x03\x02\x03\x02\x50\x8F\x02\x00\x05\xC0\x02\x05\x03\x7F\x7F\x7F\x02\x07\x02\x03\x02\x00\x02\x02\x02\x20\x02\x01\x02\x02\x02\x81\x3F\x13\xCD\x3A\x3F\xDD\xB3\xD7\x05\xC0\x01\x05\x03\x7F\x7F\x7F\x02\x0B\x02\x02\x02\x02\x02\x02\x02\x20\x02\x01\x02\x03\x02\x06\x02\x10\x02\x00\x02\x10\x02\x81\x3F\x13\xCD\x3A\x3F\xDD\xB3\xD7\x05\xC0\x00\x05\x03\x7F\x7F\x7F\x02\x07\x02\x01\x02\x00\x02\x02\x02\x20\x02\x02\x02\x03\x02\x81\x3F\x13\xCD\x3A\x3F\xDD\xB3\xD7\x00\x0C\x00\xC6\xA3\x70\x40\x02\x26\xE8\x9B"} err
assert_match "*Bad data format*" $err
r ping
}
}
} ;# tags
+6 -8
View File
@@ -248,9 +248,8 @@ start_server {tags {"repl external:skip"}} {
populate 10000 master 10000 ;# 10k keys of 10k, means 100mb
$replica config set loading-process-events-interval-bytes 262144 ;# process events every 256kb of rdb or command stream
# Start write traffic writing at most 5mbps
set load_handle [start_write_load $master_host $master_port 100 "key1" 10000 2]
# Start write traffic
set load_handle [start_write_load $master_host $master_port 100 "key1" 5000 4]
set prev_used [s 0 used_memory]
@@ -300,8 +299,8 @@ start_server {tags {"repl external:skip"}} {
assert_lessthan [expr $peak_master_used_mem - $prev_used - $backlog_size] 1000000
assert_lessthan $peak_master_rpl_buf [expr {$backlog_size + 1000000}]
assert_lessthan $peak_master_slave_buf_size 1000000
# buffers in the replica are more than 10mb
assert_morethan $peak_replica_buf_size 10000000
# buffers in the replica are more than 5mb
assert_morethan $peak_replica_buf_size 5000000
stop_write_load $load_handle
}
@@ -434,11 +433,10 @@ start_server {tags {"repl external:skip"}} {
fail "Replica did not start loading"
}
# Generate some traffic for backlog ~2mb
# Generate replication traffic of ~20mb to disconnect the slave on obuf limit
populate 20 master 1000000 -1
set res [wait_for_log_messages -1 {"*Client * closed * for overcoming of output buffer limits.*"} $loglines 1000 10]
set loglines [lindex $res 1]
wait_for_log_messages -1 {"*Client * closed * for overcoming of output buffer limits.*"} $loglines 1000 10
$replica config set key-load-delay 0
# Wait until replica loads RDB
+7
View File
@@ -756,6 +756,8 @@ test {diskless loading short read} {
redis.register_function('test', function() return 'hello1' end)
}
set has_vector_sets [server_has_command vadd]
for {set k 0} {$k < 3} {incr k} {
for {set i 0} {$i < 10} {incr i} {
r set "$k int_$i" [expr {int(rand()*10000)}]
@@ -769,6 +771,11 @@ test {diskless loading short read} {
r zadd "$k zset_large" [expr {rand()}] [string repeat A [expr {int(rand()*1000000)}]]
r lpush "$k list_small" [string repeat A [expr {int(rand()*10)}]]
r lpush "$k list_large" [string repeat A [expr {int(rand()*1000000)}]]
if {$has_vector_sets} {
r vadd "$k vector_set" VALUES 3 [expr {rand()}] [expr {rand()}] [expr {rand()}] [string repeat A [expr {int(rand()*1000)}]]
}
for {set j 0} {$j < 10} {incr j} {
r xadd "$k stream" * foo "asdf" bar "1234"
}
+23 -1
View File
@@ -738,7 +738,8 @@ proc generate_fuzzy_traffic_on_key {key type duration} {
set list_commands {LINDEX LINSERT LLEN LPOP LPOS LPUSH LPUSHX LRANGE LREM LSET LTRIM RPOP RPOPLPUSH RPUSH RPUSHX}
set set_commands {SADD SCARD SDIFF SDIFFSTORE SINTER SINTERSTORE SISMEMBER SMEMBERS SMOVE SPOP SRANDMEMBER SREM SSCAN SUNION SUNIONSTORE}
set stream_commands {XACK XADD XCLAIM XDEL XGROUP XINFO XLEN XPENDING XRANGE XREAD XREADGROUP XREVRANGE XTRIM}
set commands [dict create string $string_commands hash $hash_commands zset $zset_commands list $list_commands set $set_commands stream $stream_commands]
set vset_commands {VADD VREM}
set commands [dict create string $string_commands hash $hash_commands zset $zset_commands list $list_commands set $set_commands stream $stream_commands vectorset $vset_commands]
set cmds [dict get $commands $type]
set start_time [clock seconds]
@@ -788,6 +789,18 @@ proc generate_fuzzy_traffic_on_key {key type duration} {
lappend cmd [randomValue]
incr i 4
}
if {$cmd == "VADD"} {
lappend cmd $key
lappend cmd VALUES 3 1 1 1
lappend cmd [randomValue]
incr i 7
}
if {$cmd == "VREM"} {
lappend cmd $key
lappend cmd [randomValue]
incr i 2
}
for {} {$i < $arity} {incr i} {
if {$i == $firstkey || $i == $lastkey} {
lappend cmd $key
@@ -1144,6 +1157,15 @@ proc memory_usage {key} {
return $usage
}
# Test if the server supports the specified command.
proc server_has_command {cmd_wanted} {
set lowercase_commands {}
foreach cmd [r command list] {
lappend lowercase_commands [string tolower $cmd]
}
expr {[lsearch $lowercase_commands [string tolower $cmd_wanted]] != -1}
}
# forward compatibility, lmap missing in TCL 8.5
proc lmap args {
set body [lindex $args end]
+55
View File
@@ -137,6 +137,61 @@ start_server {tags {"hll"}} {
set e
} {*WRONGTYPE*}
test {Corrupted sparse HyperLogLogs doesn't cause overflow and out-of-bounds with XZERO opcode} {
r del hll
# Create a sparse-encoded HyperLogLog header
set header "HYLL"
set payload [binary format c12 {1 0 0 0 0 0 0 0 0 0 0 0}]
set pl [binary format a4a12 $header $payload]
# Create an XZERO opcode with the maximum run length of 16384(2^14)
set runlen [expr 16384 - 1]
set chunk [binary format cc [expr {0b01000000 | ($runlen >> 8)}] [expr {$runlen & 0xff}]]
# Fill the HLL with more than 131072(2^17) XZERO opcodes to make the total
# run length exceed 4GB, will cause an integer overflow.
set repeat [expr 131072 + 1000]
for {set i 0} {$i < $repeat} {incr i} {
append pl $chunk
}
# Create a VAL opcode with a value that will cause out-of-bounds.
append pl [binary format c 0b11111111]
r set hll $pl
# This should not overflow and out-of-bounds.
assert_error {*INVALIDOBJ*} {r pfcount hll hll}
assert_error {*INVALIDOBJ*} {r pfdebug getreg hll}
r ping
}
test {Corrupted sparse HyperLogLogs doesn't cause overflow and out-of-bounds with ZERO opcode} {
r del hll
# Create a sparse-encoded HyperLogLog header
set header "HYLL"
set payload [binary format c12 {1 0 0 0 0 0 0 0 0 0 0 0}]
set pl [binary format a4a12 $header $payload]
# # Create an ZERO opcode with the maximum run length of 64(2^6)
set chunk [binary format c [expr {0b00000000 | 0x3f}]]
# Fill the HLL with more than 33554432(2^17) ZERO opcodes to make the total
# run length exceed 4GB, will cause an integer overflow.
set repeat [expr 33554432 + 1000]
for {set i 0} {$i < $repeat} {incr i} {
append pl $chunk
}
# Create a VAL opcode with a value that will cause out-of-bounds.
append pl [binary format c 0b11111111]
r set hll $pl
# This should not overflow and out-of-bounds.
assert_error {*INVALIDOBJ*} {r pfcount hll hll}
assert_error {*INVALIDOBJ*} {r pfdebug getreg hll}
r ping
}
test {Corrupted dense HyperLogLogs are detected: Wrong length} {
r del hll
r pfadd hll a b c
+10 -10
View File
@@ -520,10 +520,10 @@ run_solo {defrag} {
r config resetstat
# TODO: Lower the threshold after defraging the ebuckets.
# Now just to ensure that the reference is updated correctly.
r config set active-defrag-threshold-lower 12
r config set active-defrag-threshold-lower 10
r config set active-defrag-cycle-min 65
r config set active-defrag-cycle-max 75
r config set active-defrag-ignore-bytes 1500kb
r config set active-defrag-ignore-bytes 1000kb
r config set maxmemory 0
r config set hash-max-listpack-value 512
r config set hash-max-listpack-entries 10
@@ -558,7 +558,7 @@ run_solo {defrag} {
puts "frag [s allocator_frag_ratio]"
puts "frag_bytes [s allocator_frag_bytes]"
}
assert_lessthan [s allocator_frag_ratio] 1.05
assert_lessthan [s allocator_frag_ratio] 1.1
# Delete all the keys to create fragmentation
for {set i 0} {$i < $n} {incr i} {
@@ -591,7 +591,7 @@ run_solo {defrag} {
}
# wait for the active defrag to stop working
wait_for_defrag_stop 500 100 1.5
wait_for_defrag_stop 500 100 1.1
# test the fragmentation is lower
after 120 ;# serverCron only updates the info once in 100ms
@@ -732,15 +732,15 @@ run_solo {defrag} {
r config set active-defrag-cycle-max 75
r config set active-defrag-ignore-bytes 2mb
r config set maxmemory 0
r config set list-max-ziplist-size 5 ;# list of 500k items will have 100k quicklist nodes
r config set list-max-ziplist-size 1 ;# list of 100k items will have 100k quicklist nodes
# create big keys with 10k items
set rd [redis_deferring_client]
set expected_frag 1.5
# add a mass of list nodes to two lists (allocations are interlaced)
set val [string repeat A 100] ;# 5 items of 100 bytes puts us in the 640 bytes bin, which has 32 regs, so high potential for fragmentation
set elements 500000
set val [string repeat A 500] ;# 1 item of 500 bytes puts us in the 640 bytes bin, which has 32 regs, so high potential for fragmentation
set elements 100000
for {set j 0} {$j < $elements} {incr j} {
$rd lpush biglist1 $val
$rd lpush biglist2 $val
@@ -811,9 +811,9 @@ run_solo {defrag} {
assert {$max_latency <= 30}
}
# in extreme cases of stagnation, we see over 20m misses before the tests aborts with "defrag didn't stop",
# in normal cases we only see 100k misses out of 500k elements
assert {$misses < $elements}
# in extreme cases of stagnation, we see over 5m misses before the tests aborts with "defrag didn't stop",
# in normal cases we only see 100k misses out of 100k elements
assert {$misses < $elements * 2}
}
# verify the data isn't corrupted or changed
set newdigest [debug_digest]
+5
View File
@@ -116,6 +116,11 @@ tags "modules" {
$master config set dynamic-hz no
$replica config set dynamic-hz no
set start [clock clicks -milliseconds]
# Generate small keys
for {set k 0} {$k < 20000} {incr k} {
r testrdb.set.key keysmall$k [string repeat A [expr {int(rand()*100)}]]
}
# Generate larger keys
for {set k 0} {$k < 30} {incr k} {
r testrdb.set.key key$k [string repeat A [expr {int(rand()*1000000)}]]
}