Compare commits

...
124 Commits
Author SHA1 Message Date
Madelyn OlsonandGitHub ba0a3b0137 Apply security fixes for CVEs (#1114)
Apply the security fixes for the release.

(CVE-2024-31449) Lua library commands may lead to stack overflow and
potential RCE.
(CVE-2024-31227) Potential Denial-of-service due to malformed ACL
selectors.
(CVE-2024-31228) Potential Denial-of-service due to unbounded pattern
matching.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2024-10-02 14:09:21 -07:00
Melroy van den BergandViktor Söderqvist c7952082d0 Build binary releases with systemd support (#1107)
- Add systemd support to the build artifact tarballs, so people can use
it under systemd compatible distros. As discussed here:
https://github.com/orgs/valkey-io/discussions/1103#discussioncomment-10815549.
Adding `libsystemd-dev` to install and add `USE_SYSTEMD=yes` to the
build.
- Cleanup & bring the arm & x86 workflow files in-sync. It was a bit of
a mess ;) (removing `jq wget awscli` from the 'Tarball' step)

Signed-off-by: Melroy van den Berg <melroy@melroy.org>
2024-10-02 20:10:55 +02:00
Melroy van den BergandViktor Söderqvist cc84b38f55 Avoid .c, .d and .o files from being copied to the binary tar.gz releases (#1106)
As discussed here:
https://github.com/orgs/valkey-io/discussions/1103#discussioncomment-10814006

`cp` can't be used anymore, `rsync` is more powerful and allow to
exclude files.

Alternatively:

1. Remove the c, d and o files. Which isn't ideal either.
2. Improve the build. Eg. by building inside a `build` directory instead
of in the src folder.

Ps. I know these workflows aren't trigger in this PR. Only via "Build
Release Packages" workflow action:
https://github.com/valkey-io/valkey/actions/workflows/build-release-packages.yml..
So I can't fully test in this PR. But it should work ^^

Ps. ps. I did test `rsync -av --exclude='*.c' --exclude='*.d'
--exclude='*.o' src/valkey-*` command in isolation and that works as
expected!

---------

Signed-off-by: Melroy van den Berg <melroy@melroy.org>
2024-10-02 20:10:55 +02:00
BinbinandMadelyn Olson 6e3a0482f2 Fix timing issue in the new tot-net-out replica test (#1060)
Apparently there is a timing issue when using wait_for_ofs_sync:
```
[exception]: Executing test client: can't read "out_before": no such variable.
can't read "out_before": no such variable
```

The reason is that if the connection between the primary
and the replica is not established yet, the master_repl_offset
of the primary and replica in wait_for_ofs_sync is 0, and
the check fails, resulting in no replica client in the
client list below.

In this case, we need to make sure the replica is online
before proceeding.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-30 17:11:51 -07:00
Madelyn Olson 5fbdfb0898 Initial staging for 8.0.1 security release
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2024-09-30 17:06:18 -07:00
zhenwei piandMadelyn Olson a206ec10f4 Fix RDMA build dependence (#1074)
RDMA module has dependence on '$(SERVER_NAME)' rather than the old style
'$(REDIS_SERVER_NAME)'.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2024-09-30 16:32:12 -07:00
Viktor SöderqvistandMadelyn Olson 9b45ca22a1 Fix bug for CLUSTER SLOTS from EVAL over TLS (#1072)
For fake clients like the ones used for Lua and modules, we don't
determine TLS in the right way, causing CLUSTER SLOTS from EVAL over TLS
to fail a debug-assert.

This error was introduced when the caching of CLUSTER SLOTS was
introduced, i.e. in 8.0.0.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2024-09-30 16:31:54 -07:00
8e975e2251 Print an empty primary log when primary lost its last slot (#1064)
The one in CLUSTER SETSLOT help us keep track of state better,
of course it also can make the test case happy.

The one in gossip process fixes a problem that a replica can
print a log saying it is an empty primary.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Ping Xie <pingxie@outlook.com>
2024-09-30 16:31:39 -07:00
BinbinandMadelyn Olson d5b4425f74 Fix module / script call CLUSTER SLOTS / SHARDS fake client check crash (#1063)
The reason is VM_Call will use a fake client without connection,
so we also need to check if c->conn is NULL.

This also affects scripts. If they are called in the script, the
server will crash. Injecting commands into AOF will also cause
startup failure.

Fixes #1054.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-30 16:31:16 -07:00
BinbinandMadelyn Olson e337cdd763 Use _Thread_local to solve threads.h build issue (#1053)
Apparently this will fail to compile in some masOS version.
And internet claims _Thread_local is portable.

Fixes #1051.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-30 16:28:47 -07:00
ranshidandMadelyn Olson 61243b9b22 Fix memory allocation for server databases (#1046)
Fix a bug in the way we allocate memory for the server databases
Introduced in #156.

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
2024-09-30 16:28:32 -07:00
BinbinandMadelyn Olson 8eefb05609 Fix default value of primary-reboot-down-after-period in sentinel.conf (#1040)
Since in here the monitor value is mymaster, we need to make sure the
primary name is the same, otherwise the default configuration cannot start
sentinel.
```
sentinel monitor mymaster 127.0.0.1 6379 2
```

The following error occurs when the default configuration is started:
```
*** FATAL CONFIG FILE ERROR (Version 255.255.255) ***
Reading the configuration file, at line 358
>>> 'SENTINEL primary-reboot-down-after-period myprimary 0'
No such master with specified name.
```

Introduced in #647.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-30 16:28:15 -07:00
Ping XieandGitHub 1549a4bc27 Fix a typo in the 8.0 release notes (#1036)
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 13:08:04 -07:00
dbcf7394b3 Add Valkey 8.0 GA Release Notes (#1034)
Based on #1031

---------

Signed-off-by: Ping Xie <pingxie@google.com>
Signed-off-by: Ping Xie <pingxie@outlook.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2024-09-15 12:53:12 -07:00
Ping XieandPing Xie 27ffb82e3b Revert "Update version number to 8.0"
This reverts commit 3179f2528db86582fb7fbf26d6d0e59555cd6b18.

Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
Ping XieandPing Xie 28245745f7 Update version number to 8.0
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
ShivshankarandPing Xie 4522953882 Update valkey-benchmark log output to reference 'server' instead of 'Redis' (#1029)
Replaced "Could not connect to Redis" with "Could not connect to server" in the log
output for connection errors in `getRedisContext` and `createClient`.

Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
BinbinandPing Xie 0204089bf5 Replica flush the old data after RDB file is ok in disk-based replication (#926)
Call emptyData right before rdbLoad to prevent errors in the middle
and we drop the replication stream and leaving an empty database.
The real changes is in disk-based part, the rest is just code movement.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
Ping XieandPing Xie bfd615d485 Improve code readability in dict.c (#943)
This pull request improves code readability, as a follow up of #749.

- Internal Naming Conventions: Removed the use of underscores (_) for
internal static structures/functions.

- Descriptive Function Names: Updated function names to be more
descriptive, making their purpose clearer. For instance, `_dictExpand`
is renamed to `dictExpandIfAutoResizeAllowed`.

---------

Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
BinbinandPing Xie c4c71b0ecc Fix replica unable trigger migration when it received CLUSTER SETSLOT in advance (#981)
Fix timing issue in evaluating `cluster-allow-replica-migration` for replicas

There is a timing bug where the primary and replica have different
`cluster-allow-replica-migration` settings. In issue #970, we found that if
the replica receives `CLUSTER SETSLOT` before the gossip update, it remains
in the original shard. This happens because we only process the
`cluster-allow-replica-migration` flag for primaries during `CLUSTER SETSLOT`.

This commit fixes the issue by also evaluating this flag for replicas in the
`CLUSTER SETSLOT` path, ensuring correct replica migration behavior.

Closes #970
---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Ping Xie <pingxie@outlook.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
BinbinandPing Xie 3c226143f2 Avoid false positive in election tests (#984)
The node may not be able to initiate an election in time due to
problems with cluster communication. If an election is initiated,
make sure its offset is 0.

Closes #967.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
BinbinandPing Xie be65f2e57d Trigger a save of the cluster configuration file before shutting down (#822)
The cluster configuration file is the metadata "database" for the
cluster. It is best to trigger a save when shutdown the server, to
avoid inconsistent content that is not refreshed.

We save the nodes.conf whenever something that affects the nodes.conf
has changed. But we are saving nodes.conf in clusterBeforeSleep, and
some events may save it without a fsync, there is a time gap.

And shutdown has its own save seems good to me, it doesn't need to
care about the others.

At the same time, a comment is added in unlock nodes.conf to explain
why we actively unlock when shutdown.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
Ping XieandPing Xie 1810c6f2df Re-enable empty-shard slot migration tests (#1024)
Related to #734 and #858

Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
xu0o0andPing Xie 6d683fca9f Make clang-format insert a newline at end of file if missing (#1023)
clang generates warning if there is no newline at the end of the source
file.

Update .clang-format to handle the missing newline at eof.

Signed-off-by: haoqixu <hq.xu0o0@gmail.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
04e00f7db5 Fix wrong count for replica's tot-net-out (#1013)
Fix duplicate calculation of replica's `net_output_bytes`

- Remove redundant calculation leftover from previous refactor
- Add test to prevent regression

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
Madelyn OlsonandPing Xie 359425d0bb Optimize the per slot dictionary by checking for cluster mode earlier (#995)
While doing some profiling, I noticed that getKeySlot() was a fairly
large part (~0.7%) of samples doing perf with high pipeline during
standalone. I think this is because we do a very late check for
server.cluster_mode, we first call getKeySlot() and then call
calculateKeySlot(). (calculateKeySlot was surprisingly not automatically
inlined, we were doing a jump into it and then immediately returning
zero). We then also do useless work in the form of caching zero in
client->slot, which will further mess with cache lines.

So, this PR tries to accomplish a few things things.
1) The usage of the `slot` name made a lot more sense before the
introduction of the kvstore. Now with kvstore, we call this the database
index, so all the references to slot in standalone are no longer really
accurate.
2) Pull the cluster mode check all the way out of getKeySlot(), so
hopefully a bit more performant.
3) Remove calculateKeySlot() as independent from getKeySlot().
calculateKeySlot used to have 3 call sites outside of db.c, which
warranted it's own function. It's now only called in two places,
pubsub.c and networking.c.

I ran some profiling, and saw about ~0.3% improvement, but don't really
trust it because you'll see a much higher (~2%) variance in test runs
just by how the branch predictions will get changed with a new memory
layout. Running perf again showed no samples in getKeySlot() and a
reduction in samples in lookupKey(), so maybe this will help a little
bit.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
Madelyn OlsonandPing Xie 586af0fe01 Improve stability of hostnames test (#1016)
Maybe partially resolves https://github.com/valkey-io/valkey/issues/952.

The hostnames test relies on an assumption that node zero and node six
don't communicate with each other to test a bunch of behavior in the
handshake stake. This was done by previously dropping all meet packets,
however it seems like there was some case where node zero was sending a
single pong message to node 6, which was partially initializing the
state.

I couldn't track down why this happened, but I adjusted the test to
simply pause node zero which also correctly emulates the state we want
to be in since we're just testing state on node 6, and removes the
chance of errant messages. The test was failing about 5% of the time
locally, and I wasn't able to reproduce a failure with this new
configuration.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
Mikhail KoviazinandPing Xie 4119e3b359 Added .git-blame-ignore-revs (#1010)
This file enables developers to ignore the certain revisions in
git-blame. This is quite handy considering there was a commit that
reformatted the large amount of code in valkey.

As a downside, one has to do a manual step for each clone of valkey to
enable this feature. The instructions are available in the file itself.

---------

Signed-off-by: Mikhail Koviazin <mikhail.koviazin@aiven.io>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
BinbinandPing Xie b88e03bbf7 Fix module RdbLoad wrongly disable the AOF (#1001)
In RdbLoad, we disable AOF before emptyData and rdbLoad to prevent copy-on-write issues. After rdbLoad completes, AOF should be re-enabled, but the code incorrectly checks server.aof_state, which has been reset to AOF_OFF in stopAppendOnly. This leads to AOF not being re-enabled after being disabled.
---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
Amit NaglerandPing Xie d2e98b14dc Dual Channel Replication - Verify Replica Local Buffer Limit Configuration (#989)
Prior to comparing the replica buffer against the configured limit, we
need to ensure that the limit configuration is enabled. If the limit is
set to zero, it indicates that there is no limit, and we should skip the
buffer limit check.

---------

Signed-off-by: naglera <anagler123@gmail.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
90a22d63fd Use hashtable as the default type of temp set object during sunion/sdiff (#996)
This patch try to set the temp set object as default hash table type.
And did a simple predication of the temp set object encoding when
initialize `dstset` to reduce the unnecessary conversation.

According to existing code logic, when did operation like `sunion` and
`sdiff` , the temp set object could be `intset`, `listpack` and
`hashtable`, for the `listpack`, the efficiency is low when did
operation like `find` and `compare` , need to traverse all elements.
When we exploring the hotspots, found the `lpFind` and `memcmp` has been
the bottleneck when running workloads like below:

-
[memtier_benchmark-2keys-set-10-100-elements-sunion.yml](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-2keys-set-10-100-elements-sunion.yml)
-
[memtier_benchmark-2keys-set-10-100-elements-sdiff.yml](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-2keys-set-10-100-elements-sdiff.yml)

![image](https://github.com/user-attachments/assets/71dfc70b-2ad5-4832-a338-712deefca20e)

This patch try to set the temp set object as default hash table type.
And did a simple predication of the temp set object encoding when
initialize `dstset` to reduce the unnecessary conversation.

- OPERATING SYSTEM: Ubuntu 22.04.4 LTS
- Kernel: 5.15.0-116-generic
- PROCESSOR: Intel Xeon Platinum 8380
- Server and Client in same socket.

```
taskset -c 0-3 ~/valkey/src/valkey-server /tmp/valkey.conf

port 9001
bind * -::*
daemonize no
protected-mode no
save ""
```

| Test Name| Perf Boost|
|-|-|

|[memtier_benchmark-2keys-set-10-100-elements-sunion.yml](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-2keys-set-10-100-elements-sunion.yml)
|41%|

|[memtier_benchmark-2keys-set-10-100-elements-sdiff.yml](https://github.com/redis/redis-benchmarks-specification/blob/main/redis_benchmarks_specification/test-suites/memtier_benchmark-2keys-set-10-100-elements-sdiff.yml)
|27%|

With above test set which have total 110 elements in the 2 given sets.
We also did some benchmarking by adjusting the total number of elements
in all given sets. We can still observe the performance boost.

![image](https://github.com/user-attachments/assets/b2ab420c-43e5-45de-9715-7d943df229cb)

---------

Signed-off-by: Lipeng Zhu <lipeng.zhu@intel.com>
Co-authored-by: Wangyang Guo <wangyang.guo@intel.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
uriyageandPing Xie 47b14ef377 Fix crash in async IO threads with TLS (#1011)
Fix for https://github.com/valkey-io/valkey/issues/997

Root Cause Analysis:
1. Two different jobs (READ and WRITE) may be sent to the same IO
thread.
2. When processing the read job in `processIOThreadsReadDone`, the IO
thread may find that the write job has also been completed.
3. In this case, the IO thread calls `processClientIOWriteDone` to first
process the completed write job and free the COBs
https://github.com/valkey-io/valkey/blob/89a4fa7695108cd2a1ac584f84e820325e5a3ad9/src/networking.c#L4666
4. If there are pending writes (resulting from pipeline commands), a new
async IO write job is sent before processing the completed read job
https://github.com/valkey-io/valkey/blob/89a4fa7695108cd2a1ac584f84e820325e5a3ad9/src/networking.c#L2417
When sending the write job, the `TLS_CONN_FLAG_POSTPONE_UPDATE_STATE`
flag is set to prevent the IO thread from updating the event loop, which
is not thread-safe.
5. Upon resuming the read job processing, the flag is cleared,
https://github.com/valkey-io/valkey/blob/89a4fa7695108cd2a1ac584f84e820325e5a3ad9/src/networking.c#L4685
causing the IO thread to update the event loop.

Fix:
Prevent sending async write job for pending writes when a read job is
about to be processed.

Testing:
The issue could not be reproduced due to its rare occurrence, which
requires multiple specific conditions to align simultaneously.

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
cd79981faf For MEETs, save the extensions support flag immediately during MEET processing (#778)
For backwards compatibility reasons, a node will wait until it receives
a cluster message with the extensions flag before sending its own
extensions. This leads to a delay in shard ID propagation that can
corrupt nodes.conf with inaccurate shard IDs if a node is restarted
before this can stabilize.

This fixes much of that delay by immediately triggering the
extensions-supported flag during the MEET processing and attaching the
node to the link, allowing the PONG reply to contain OSS extensions.

Partially fixes #774

---------

Signed-off-by: Ben Totten <btotten@amazon.com>
Co-authored-by: Ben Totten <btotten@amazon.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
BinbinandPing Xie 77decba670 Add missing moduleapi getchannels test and fix tests (#1002)
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
zhaozhao.zzandPing Xie 6413dade34 add assertion for kvstore's dictType (#1004)
Signed-off-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
373f51e9fe Migrate dict.c unit tests to new framework (#946)
This PR migrates the tests related to dict into new test framework as
part of #428.

Signed-off-by: haoqixu <hq.xu0o0@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
a888fabc81 Migrate listpack.c unit tests to new framework (#949)
This PR migrates the tests related to listpack into new test framework
as part of #428.

Signed-off-by: haoqixu <hq.xu0o0@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
BinbinandPing Xie 933b03140b Add client info to SHUTDOWN / CLUSTER FAILOVER logs (#875)
Print the full client info by using catClientInfoString, the
info is useful when we want to identify the source of request.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
BinbinandPing Xie 31473a4d58 Fix aof base suffix when modifying aof-use-rdb-preamble during rewrite (#886)
If we modify aof-use-rdb-preamble in the middle of rewrite,
we may get a wrong aof base suffix. This is because the suffix
is concatenated by the main process afterwards, and it may be
different from the beginning.

We cache this value when we start the rewrite.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
BinbinandPing Xie a35225204b Fix missing replication link re-connection when primary's IP/port is updated in clusterProcessGossipSection (#965)
`clusterProcessGossipSection` currently doesn't trigger a check and call `replicationSetPrimary` when `myself`'s primary node’s IP/port is updated. This fix ensures that after every node address update, `replicationSetPrimary` is called if the updated node is `myself`'s primary. This prevents missed updates and ensures that replicas reconnect properly to maintain their replication link with the primary.

Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
BinbinandPing Xie d752c6fa20 Add newline to argv in crash report when doing redact (#993)
Minor cleanup, introduced in #877.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
Kyle Kim (kimkyle@)andPing Xie 945673489f Add SLOT-STATS under CLUSTER HELP string. (#988)
Add help wording for cluster SLOT-STATS.

Signed-off-by: Kyle Kim <kimkyle@amazon.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
Viktor SöderqvistandPing Xie 692e12479a Rewrite lazyfree docs in valkey.conf to reflect that lazy is now default (#983)
Before this doc update, the comments in valkey.conf said that DEL is a
blocking command, and even refered to other synchronous freeing as "in a
blocking way, like if DEL was called". This has now become confusing and
incorrect, since DEL is now non-blocking by default.

The comments also mentioned too much about the "old default" and only
later explain that the "new default" is non-blocking.

This doc update focuses on the current default and expresses it like
"Starting from Valkey 8.0, lazy freeing is enabled by default", rather
than using words like old and new.

This is a follow-up to #913.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
NAM UK KIMandPing Xie ae911e898d Fix typo in valkey-cli.c (#979)
Change from replicsa to replicas in valkey-cli.c

Signed-off-by: NAM UK KIM <namuk2004@naver.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
a3e1acd944 Improve type safety and refactor dict entry handling (#749)
This pull request introduces several changes to improve the type safety
of Valkey's dictionary implementation:

- Getter/Setter Macros: Implemented macros `DICT_SET_VALUE` and
`DICT_GET_VALUE` to centralize type casting within these macros. This
change emulates the behavior of C++ templates in C, limiting type
casting to specific low-level operations and preventing it from being
spread across the codebase.

- Reduced Assert Overhead: Removed unnecessary asserts from critical hot
paths in the dictionary implementation.

- Consistent Naming: Standardized the naming of dictionary entry types.
For example, all dictionary entry types start their names with
`dictEntry`.

Fix #737

---------

Signed-off-by: Ping Xie <pingxie@google.com>
Signed-off-by: Ping Xie <pingxie@outlook.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-15 11:49:49 -07:00
Madelyn Olson d8c86482e4 Added release notes for RC2
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2024-09-03 09:00:45 -07:00
Madelyn Olson 6fb955d941 Initialize all the fields for the test kvstore (#982)
Follow up to https://github.com/valkey-io/valkey/pull/966, which didn't
update the kvstore tests. I'm not actually entirely clear why it fixes
it, but the consistency prevents the crash very reliably so will merge
it now and maybe see if Zhao has a better explanation.

---------

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2024-09-03 09:00:45 -07:00
Amit NaglerandMadelyn Olson f55e7c5ffa Add configuration hide-user-data-from-log to hide user data from server logs (#877)
Implement data masking for user data in server logs and diagnostic output. This change prevents potential exposure of confidential information, such as PII, and enhances privacy protection. It masks all command arguments, client names, and client usernames.

Added a new hide-user-data-from-log configuration item, default yes.

---------

Signed-off-by: Amit Nagler <anagler123@gmail.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 88e3151743 Fix set expire test due to the new lazyfree configs changes (#980)
Test failed because these two PRs #865 and #913.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
zhaozhao.zzandMadelyn Olson e9adffc3dd Use metadata to handle the reference relationship between kvstore and dict (#966)
Feature `one-dict-per-slot` refactors the database, and part of it
involved splitting the rehashing list from the global level back to the
database level, or more specifically, the kvstore level. This change is
fine, and it also simplifies the process of swapping databases, which is
good. And it should not have a major impact on the efficiency of
incremental rehashing.

To implement the kvstore-level rehashing list, each `dict` under the
`kvstore` needs to know which `kvstore` it belongs. However, kvstore did
not insert the reference relationship into the `dict` itself, instead,
it placed it in the `dictType`. In my view, this is a somewhat odd way.
Theoretically, `dictType` is just a collection of function handles, a
kind of virtual type that can be referenced globally, not an entity. But
now the `dictType` is instantiated, with each `kvstore` owning an actual
`dictType`, which in turn holds a reverse reference to the `kvstore`'s
resource pointer. This design is somewhat uncomfortable for me.

I think the `dictType` should not be instantiated. The references
between actual resources (`kvstore` and `dict`) should occur between
specific objects, rather than force materializing the `dictType`, which
is supposed to be virtual.

---------

Signed-off-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 8ab95da167 Change all the lazyfree configurations to yes by default (#913)
## Set replica-lazy-flush and lazyfree-lazy-user-flush to yes by
default.
There are many problems with running flush synchronously. Even in
single CPU environments, the thread managers should balance between
the freeing and serving incoming requests.

## Set lazy eviction, expire, server-del, user-del to yes by default
We now have a del and a lazyfree del, we also have these configuration
items to control: lazyfree-lazy-eviction, lazyfree-lazy-expire,
lazyfree-lazy-server-del, lazyfree-lazy-user-del. In most cases lazyfree
is better since it reduces the risk of blocking the main thread, and
because we have lazyfreeGetFreeEffort, on those with high effor
(currently
64) will use lazyfree.

Part of #653.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
Madelyn Olson f3134538f5 Fix zipmap test null pointer (#975)
The previous test does a strncmp on a NULL, which is not valid. It
should be using an empty length string instead. Addresses
https://github.com/valkey-io/valkey/actions/runs/10649272046/job/29519233939.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 8be025b198 Fast path in SET if the expiration time is expired (#865)
If the expiration time passed in SET is expired, for example, it
has expired due to the machine time (DTS) or the expiration time
passed in (wrong arg). In this case, we don't need to set the key
and wait for the active expire scan before deleting the key.

Compared with previous changes:
1. If the key does not exist, previously we would set the key and wait
for the active expire to delete it, so it is a set + del from the
perspective
of propaganda. Now we will no set the key and return, so it a NOP.

2. If the key exists, previously we woule set the key and wait
for the active expire to delete it, so it is a set + del From the
perspective
of propaganda. Now we will delete it and return, so it is a del.

Adding a new deleteExpiredKeyFromOverwriteAndPropagate function
to reduce the duplicate code.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2024-09-03 09:00:45 -07:00
Viktor SöderqvistandMadelyn Olson b5b2b7e187 Delete unused parts of zipmap (#973)
Deletes zipmapSet, zipmapGet, etc. Only keep iterator and validate
integrity, what we use when loading an old RDB file.

Adjust unit tests to not use zipmapSet, etc.

Solves a build failure where when compiling with fortify source.

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 57a86925f9 Fix timing issue in replica migration test (#968)
The reason is the server 3 still have the server 7 as its replica
due to a short wait, the wait is not enough, we should wait for
server loss its replica.
```
*** [err]: valkey-cli make source node ignores NOREPLICAS error when doing the last CLUSTER SETSLOT
Expected '{127.0.0.1 21497 267}' to be equal to '' (context: type eval line 34 cmd {assert_equal [lindex [R 3 role] 2] {}} proc ::test)
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
zhaozhao.zzandMadelyn Olson 98035ded96 standalone -REDIRECT handles special case of MULTI context (#895)
In standalone mode, when a `-REDIRECT` error occurs, special handling is
required if the client is in the `MULTI` context.

We have adopted the same handling method as the cluster mode:

1. If a command in the transaction encounters a `REDIRECT` at the time
of queuing, the execution of `EXEC` will return an `EXECABORT` error (we
expect the client to redirect and discard the transaction upon receiving
a `REDIRECT`). That is:

    ```
    MULTI    ==>  +OK
    SET x y  ==>  -REDIRECT
    EXEC     ==>  -EXECABORT
    ```
2. If all commands are successfully queued (i.e., `QUEUED` results are
received) but a redirect is detected during `EXEC` execution (such as a
primary-replica switch), a `REDIRECT` is returned to instruct the client
to perform a redirect. That is:

    ```
    MULTI    ==>  +OK
    SET x y  ==>  +QUEUED
    failover
    EXEC     ==>  -REDIRECT
    ```

---------

Signed-off-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
2024-09-03 09:00:45 -07:00
5629882f32 Migrate zipmap unit test to new framework (#474)
Migrate zipmap unit test to new unit test framework, parent ticket #428
.

---------

Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
Signed-off-by: hwware <wen.hui.ware@gmail.com>
Co-authored-by: hwware <wen.hui.ware@gmail.com>
2024-09-03 09:00:45 -07:00
cb09d23fc6 Fix reconfiguring sub-replica causing data loss when myself change shard_id (#944)
When reconfiguring sub-replica, there may a case that the sub-replica will
use the old offset and win the election and cause the data loss if the old
primary went down.

In this case, sender is myself's primary, when executing updateShardId,
not only the sender's shard_id is updated, but also the shard_id of
myself is updated, casuing the subsequent areInSameShard check, that is,
the full_sync_required check to fail.

As part of the recent fix of #885, the sub-replica needs to decide whether
a full sync is required or not when switching shards. This shard membership
check is supposed to be done against sub-replica's current shard_id, which
however was lost in this code path. This then leads to sub-replica joining
the other shard with a completely different and incorrect replication history.

This is the only place where replicaof state can be updated on this path
so the most natural fix would be to pull the chain replication reduction
logic into this code block and before the updateShardId call.

This one follow #885 and closes #942.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Ping Xie <pingxie@outlook.com>
2024-09-03 09:00:45 -07:00
zhaozhao.zzandMadelyn Olson e1c78a4b71 free client's multi state when it becomes dirty (#961)
Release the client's MULTI state when the transaction becomes dirty to
save memory.

---------

Signed-off-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
2024-09-03 09:00:45 -07:00
Ping XieandMadelyn Olson 9028417de2 Exclude '.' and ':' from isValidAuxChar's banned charset (#963)
Fix a bug in isValidAuxChar where valid characters '.' and ':' were
incorrectly included in the banned charset. This issue affected the
validation of auxiliary fields in the nodes.conf file used by Valkey in
cluster mode, particularly when handling IPv4 and IPv6 addresses. The
code now correctly allows '.' and ':' as valid characters, ensuring
proper handling of these fields. Comments were added to clarify the use
of the banned charset.
 
Related to #736

---------

Signed-off-by: Ping Xie <pingxie@google.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 4795df6189 Revert make KEYS to be an exact match if there is no pattern (#964)
In #792, the time complexity became ambiguous, fluctuating between
O(1) and O(n), which is a significant difference. And we agree uncertainty
can potentially bring disaster to the business, the right thing to do is
to persuade users to use EXISTS instead of KEYS in this case, to do the
right thing the right way, rather than accommodating this incorrect usage.

This reverts commit fcea7be0bd.
This reverts #792.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
Viktor SöderqvistandMadelyn Olson 9a1723d665 Delete TLS.md and update README.md about tests (#960)
Most of the content of TLS.md has already been copied to README.md in
#927.

The description of how to run tests with TLS is moved to
tests/README.md.

Descriptions of the additional scripts runtest-cluster, runtest-sentinel
and runtest-module are added in tests/README.md.

Links to tests/README.md and src/unit/README.md are added in the
top-level README.md along with a brief overview of the `make test-*`
commands.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2024-09-03 09:00:45 -07:00
Viktor SöderqvistandMadelyn Olson 97fea764ff Delete files MANIFESTO, BUGS and INSTALL (#958)
The MANIFESTO is not Valkey's manifesto and it doesn't even mention open
source. Let's write another one later, or some other document about our
project principles.

The other two files are one-line files with no relevant info. They're
polluting the file listing at root level. It's the first thing you see
when you start exploring the repo for the first time.

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2024-09-03 09:00:45 -07:00
326b5ed0ce Migrate the contents of TLS.md into README.md (#927)
Migrate the contents in TLS.md into TLS sections including building,
running and detail supports. TODO list in the TLS.md is almost done
except the implementation of benchmark support is still not the best
approach which should migrate to hiredis async mode.

Closes #888

---------

Signed-off-by: I Hsin Cheng <richard120310@gmail.com>
Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2024-09-03 09:00:45 -07:00
Ping XieandMadelyn Olson 96cecf709a Add comment explaining log file reopening for rotation support (#956) 2024-09-03 09:00:45 -07:00
d49faad4f7 Using intrinsics to optimize counting HyperLogLog trailing bits (#846)
Godbolt link: https://godbolt.org/z/3YPvxsr5s

__builtin_ctz would generate shorter code than hand-written loop.

---------

Signed-off-by: mwish <maplewish117@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 3ccbc84f09 Add pause path coverage to replica migration tests (#937)
In #885, we only add a shutdown path, there is another path
is that the server might got hang by slowlog. This PR added
the pause path coverage to cover it.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
Lipeng ZhuandMadelyn Olson 9872dcaa6b Move prepareClientToWrite out of loop for lrange command to reduce the redundant call. (#860)
## Description 
When I explore the cycles distributions for `lrange` test (
`valkey-benchmark -p 9001 -t lrange -d 100 -r 1000000 -n 1000000 -c 50
--threads 4`). I found the `prepareClientToWrite` and
`clientHasPendingReplies` could be reduced to single call outside
instead of called in a loop, ideally we can gain 3% performance. The
corresponding `LRANG_100`, `LRANG_300`, `LRANGE_500`, `LRANGE_600` have
~2% - 3% performance boost, the benchmark test prove it helps.

This patch try to move the `prepareClientToWrite` and its child
`clientHasPendingReplies` out of the loop to reduce the function
overhead.

---------

Signed-off-by: Lipeng Zhu <lipeng.zhu@intel.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 6838d170e2 Wait for the role change and fix the timing issue in the new test (#947)
The test might be fast enough and then there is no change in the role
causing the test to fail. Adding a wait to avoid the timing issue:
```
*** [err]: valkey-cli make source node ignores NOREPLICAS error when doing the last CLUSTER SETSLOT
Expected '{127.0.0.1 23154 267}' to be equal to '' (context: type eval line 24 cmd {assert_equal [lindex [R 3 role] 2] {}} proc ::test)
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
Vadym KhoptynetsandMadelyn Olson 0b4a0fb95f Use sdsAllocSize instead of sdsZmallocSize (#923)
sdsAllocSize returns the correct size without consulting the
allocator. Which is much faster than consulting the allocator.
The only exception is SDS_TYPE_5, for which it has to
consult the allocator.

This PR also sets alloc field correctly for embedded string objects.
It assumes that no allocator would allocate a buffer larger
than `259 + sizeof(robj)` for embedded string. We use embedded strings
for strings up to 44 bytes. If this assumption is wrong, the whole
function would require a rewrite. In general case sds type adjustment
might be needed. Such logic should go to sds.c.

---------

Signed-off-by: Vadym Khoptynets <vadymkh@amazon.com>
2024-09-03 09:00:45 -07:00
Amit NaglerandMadelyn Olson 4e1d985fe4 Remove dual-channel-replication Feature Flag's Protection (#908)
Currently, the `dual-channel-replication` feature flag is immutable if
`enable-protected-configs` is enabled, which is the default behavior.
This PR proposes to make the `dual-channel-replication` flag mutable,
allowing it to be changed dynamically without restarting the cluster.

**Motivation:**
The ability to change the `dual-channel-replication` flag dynamically is
essential for testing and validating the feature on real clusters
running in production environments. By making the flag mutable, we can
enable or disable the feature without disrupting the cluster's
operations, facilitating easier testing and experimentation.
Additionally, this change would provide more flexibility for users to
enable or disable the feature based on their specific requirements or
operational needs without requiring a cluster restart.

---------

Signed-off-by: naglera <anagler123@gmail.com>
2024-09-03 09:00:45 -07:00
Viktor SöderqvistandMadelyn Olson f0171def80 Connection minor fixes (#953)
1. Remove redundant connIncrRefs/connDecrRefs

    In socket.c, the reference counter is incremented before calling
callHandler, but the same reference counter is also incremented inside
callHandler before calling the actual callback.

        static inline int callHandler(connection *conn, ConnectionCallbackFunc handler) {
            connIncrRefs(conn);
            if (handler) handler(conn);
            connDecrRefs(conn);
            ...
        }

    This commit removes the redundant incr/decr calls in socket.c

2. Correct return value of connRead for TLS when peer closed

    According to comments in connection.h, connRead returns 0 when the peer
has closed the connection. This patch corrects the return value for TLS
connections. (Without this patch, it returns -1 which means error.)

    There is an observable difference in what is logged in the verbose
level: "Client closed connection" vs "Reading from client: (null)".

---------

Signed-off-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
2024-09-03 09:00:45 -07:00
uriyageandMadelyn Olson 7c927ceb85 Improve multithreaded performance with memory prefetching (#861)
This PR utilizes the IO threads to execute commands in batches, allowing
us to prefetch the dictionary data in advance.

After making the IO threads asynchronous and offloading more work to
them in the first 2 PRs, the `lookupKey` function becomes a main
bottle-neck and it takes about 50% of the main-thread time (Tested with
SET command). This is because the Valkey dictionary is a straightforward
but inefficient chained hash implementation. While traversing the hash
linked lists, every access to either a dictEntry structure, pointer to
key, or a value object requires, with high probability, an expensive
external memory access.

### Memory Access Amortization

Memory Access Amortization (MAA) is a technique designed to optimize the
performance of dynamic data structures by reducing the impact of memory
access latency. It is applicable when multiple operations need to be
executed concurrently. The principle behind it is that for certain
dynamic data structures, executing operations in a batch is more
efficient than executing each one separately.

Rather than executing operations sequentially, this approach interleaves
the execution of all operations. This is done in such a way that
whenever a memory access is required during an operation, the program
prefetches the necessary memory and transitions to another operation.
This ensures that when one operation is blocked awaiting memory access,
other memory accesses are executed in parallel, thereby reducing the
average access latency.

We applied this method in the development of `dictPrefetch`, which takes
as parameters a vector of keys and dictionaries. It ensures that all
memory addresses required to execute dictionary operations for these
keys are loaded into the L1-L3 caches when executing commands.
Essentially, `dictPrefetch` is an interleaved execution of dictFind for
all the keys.


**Implementation details**

When the main thread iterates over the `clients-pending-io-read`, for
clients with ready-to-execute commands (i.e., clients for which the IO
thread has parsed the commands), a batch of up to 16 commands is
created. Initially, the command's argv, which were allocated by the IO
thread, is prefetched to the main thread's L1 cache. Subsequently, all
the dict entries and values required for the commands are prefetched
from the dictionary before the command execution. Only then will the
commands be executed.

---------

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 17c906109a Drop the outdated script replication example comments (#951)
This example was for script replication which we have
completely removed in 7.0, so this example is outdated now.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson d203361731 Make KEYS to be an exact match if there is no pattern (#792)
Although KEYS is a dangerous command and we recommend people
to avoid using it, some people who are not familiar with it
still using it, and even use KEYS with no pattern at all.

Once KEYS is using with no pattern, we can convert it to an
exact match to avoid iterating over all data.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
xu0o0andMadelyn Olson f736254b72 Fix invalid escape sequence in utils, minor cleanup in python script (#948)
According to the Python document[1], any invalid escape sequences in
string literals now generate a DeprecationWarning (SyntaxWarning as of
3.12) and in the future this will become a SyntaxError.

This Change uses Python’s raw string notation for regular expression
patterns to avoid it.

[1]: https://docs.python.org/3.10/library/re.html

Signed-off-by: haoqixu <hq.xu0o0@gmail.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson e97a2d2e3d Add explicit assert to ensure thread_shared_qb won't expand (#938)
Although this won't happen now, adding this statement explicitly.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 84e0d835ad Add epoch information to failover auth denied logs (#816)
When failover deny to vote, sometimes due to network or
some blocking operations, the time of FAILOVER_AUTH_REQUEST
packet arrival is very uncertain. Since there is no epoch
information in these logs, it is hard to associate the log
with other logs.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
NAM UK KIMandMadelyn Olson d21b292d6c Update "Total" message and used_memory_human log information in serverCron() function (#594)
At the VERBOSE/DEBUG log level, which is output once every 5 seconds,
added to show the "Total" message of all clients and to show memory
usage (used_memory) with used_memory_human.
Also, it seems clearer to show "total" number of keys and the number of
volatile in entire keys.

---------

Signed-off-by: NAM UK KIM <namuk2004@naver.com>
2024-09-03 09:00:45 -07:00
Ayush SharmaandMadelyn Olson a59a0b3265 Add support for setting the group on a unix domain socket (#901)
Add new optional, immutable string config called `unixsocketgroup`. 
Change the group of the unix socket to `unixsocketgroup` after `bind()`
if specified.

Adds tests to validate the behavior.

Fixes #873.

Signed-off-by: Ayush Sharma <mrayushs933@gmail.com>
2024-09-03 09:00:45 -07:00
Madelyn Olson 0e64577ae8 Remove accurate from extra test tag (#935)
Today if we attached the "run-extra-tests" tag it adds at least 20
minutes because the dump-fuzzer test runs with full accuracy. This
fuzzer is useful, but probably only really needed for the daily, so
removing it from the PRs. We still run the fuzzers, just not for as
long.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson e8774bf911 valkey-cli make source node ignores NOREPLICAS when doing the last CLUSTER SETSLOT (#928)
This fixes #899. In that issue, the primary is cluster-allow-replica-migration no
and its replica is cluster-allow-replica-migration yes.

And during the slot migration:
1. Primary calling blockClientForReplicaAck, waiting its replica.
2. Its replica reconfiguring itself as a replica of other shards due to
replica migration and disconnect from the old primary.
3. The old primary never got the chance to receive the ack, so it got a
timeout and got a NOREPLICAS error.

In this case, the replicas might automatically migrate to another primary,
resulting in the client being unblocked with the NOREPLICAS error. In this
case, since the configuration will eventually propagate itself, we can safely
ignore this error on the source node.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
37548047de Fix CLUSTER SETSLOT block and unblock error when all replicas are down (#879)
In CLUSTER SETSLOT propagation logic, if the replicas are down, the
client will get block during command processing and then unblock
with `NOREPLICAS Not enough good replicas to write`.

The reason is that all replicas are down (or some are down), but
myself->num_replicas is including all replicas, so the client will
get block and always get timeout.

We should only wait for those online replicas, otherwise the waiting
propagation will always timeout since there are not enough replicas.
The admin can easily check if there are replicas that are down for an
extended period of time. If they decide to move forward anyways, we
should not block it. If a replica  failed right before the replication and
was not included in the replication, it would also unlikely win the election.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Ping Xie <pingxie@google.com>
2024-09-03 09:00:45 -07:00
Yunxiao DuandMadelyn Olson b1d3b057f9 Delete redundant declaration clusterNodeCoversSlot and countKeysInSlot (#930)
Delete redundant declaration, clusterNodeCoversSlot and countKeysInSlot
has been declared in cluster.h

Signed-off-by: Yunxiao Du <me@jackdu.cn>
2024-09-03 09:00:45 -07:00
Madelyn Olson cd898efeae Revert repl backlog size back to 1mb for dual channel tests (#934)
There is a test that assumes that the backlog will get overrun, but
because of the recent changes to the default it no longer fails. It
seems like it is a bit flakey now though, so resetting the value in the
test back to 1mb. (This relates to the CoB of 1100k. So it should
consistently work with a 1mb limit).

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2024-09-03 09:00:45 -07:00
Wen HuiandMadelyn Olson 11c301d02e Decline unsubscribe related command in non-subscribed mode (#759)
Now, when clients run the unsubscribe, sunsubscribe and punsubscribe
commands in the non-subscribed mode, it returns 0.
Indeed this is a bug, we should not allow client run these kind of
commands here.

Thus, this PR fixes this bug, but it is a break change for existing
clients

---------

Signed-off-by: hwware <wen.hui.ware@gmail.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson fe7716366e Make runtest-cluster support --io-threads option (#933)
In #764, we add a --io-threads mode in test, but forgot
to handle runtest-cluster, they are different framework.

Currently runtest-cluster does not support tags, and we
don't have plan to support it. And currently cluster tests
does not have any io-threads tests, so this PR just align
--io-threads option with #764.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
ce0d3812f9 Avoid to re-establish replication if node is already myself primary in CLUSTER REPLICATE (#884)
If n is already myself primary, there is no need to re-establish the
replication connection.

In the past we allow a replica node to reconnect with its primary via
this CLUSTER REPLICATE command, it will use psync. But since #885, we
will assume that a full sync is needed in this case, so if we don't do
this, the replica will always use full sync.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Ping Xie <pingxie@google.com>
2024-09-03 09:00:45 -07:00
uriyageandMadelyn Olson 410df42ca6 Skip tracking clients OOM test when I/O threads are enabled (#764)
Fix feedback loop in key eviction with tracking clients when using I/O
threads.

Current issue:
Evicting keys while tracking clients or key space-notification exist
creates a feedback loop when using I/O threads:

While evicting keys we send tracking async writes to I/O threads,
preventing immediate release of tracking clients' COB memory
consumption.

Before the I/O thread finishes its write, we recheck used_memory, which
now includes the tracking clients' COB and thus continue to evict more
keys.

**Fix:**
We will skip the test for now while IO threads are active. We may
consider avoiding sending writes in `processPendingWrites` to I/O
threads for tracking clients when we are out of memory.

---------

Signed-off-by: Uri Yagelnik <uriy@amazon.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2024-09-03 09:00:45 -07:00
Harkrishn PatroandMadelyn Olson 62a6989c4f Update README.md file to reference valkey.io (#931)
Update README.md since the project is no longer under construction, and
can reference the main website.

---------

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 5616624244 Set repl-backlog-size from 1mb to 10mb by default (#911)
The repl-backlog-size 1mb is too small in most cases, now network
transmission and bandwidth performance have improved rapidly in more
than ten years.

The bigger the replication backlog, the longer the replica can endure
the disconnect and later be able to perform a partial resynchronization.

Part of #653.

---------

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
Wen HuiandMadelyn Olson 6d235257f6 Fix Error in Daily CI -- reply-schemas-validator (#922)
Just add one more test for command "sentinel IS-PRIMARY-DOWN-BY-ADDR" to
make the reply-schemas-validator
run successfully.

Note: test result here
https://github.com/hwware/valkey/actions/runs/10457516111

Signed-off-by: hwware <wen.hui.ware@gmail.com>
2024-09-03 09:00:45 -07:00
zhenwei piandMadelyn Olson c179b82482 RDMA: Support user keepalive command (#916)
If the client side crashes by any issue or exits normally, the kernel
will try to disconnect RDMA QPs. Then the kernel of server side receives
CM packets, valkey-server handles CM disconnected event and close
connection.

However, there is a lack of keepalive mechanism from RDMA transport
layer. Once the kernel of client side crashes, the server side will not
be notified. To avoid this issue, valkey server sents Keepaliv command
periodically to detect any dead QPs.

An example of mlx-cx5:

```
 # RDMA: CQ handle error status: transport retry counter exceeded[0xc], opcode : 0x0
 # RDMA: CQ handle error status: transport retry counter exceeded[0xc], opcode : 0x0
 # RDMA: CQ handle error status: Work Request Flushed Error[0x5], opcode : 0x0
 # RDMA: CQ handle error status: Work Request Flushed Error[0x5], opcode : 0x0
 # RDMA: CQ handle error status: Work Request Flushed Error[0x5], opcode : 0x0
 # RDMA: CQ handle error status: Work Request Flushed Error[0x5], opcode : 0x0
```

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 764e74273a Fix data loss when replica do a failover with a old history repl offset (#885)
Our current replica can initiate a failover without restriction when
it detects that the primary node is offline. This is generally not a
problem. However, consider the following scenarios:

1. In slot migration, a primary loses its last slot and then becomes
a replica. When it is fully synchronized with the new primary, the new
primary downs.

2. In CLUSTER REPLICATE command, a replica becomes a replica of another
primary. When it is fully synchronized with the new primary, the new
primary downs.

In the above scenario, case 1 may cause the empty primary to be elected
as the new primary, resulting in primary data loss. Case 2 may cause the
non-empty replica to be elected as the new primary, resulting in data
loss and confusion.

The reason is that we have cached primary logic, which is used for psync.
In the above scenario, when clusterSetPrimary is called, myself will cache
server.primary in server.cached_primary for psync. In replicationGetReplicaOffset,
we get server.cached_primary->reploff for offset, gossip it and rank it,
which causes the replica to use the old historical offset to initiate
failover, and it get a good rank, initiates election first, and then is
elected as the new primary.

The main problem here is that when the replica has not completed full
sync, it may get the historical offset in replicationGetReplicaOffset.

The fix is to clear cached_primary in these places where full sync is
obviously needed, and let the replica use offset == 0 to participate
in the election. In this way, this unhealthy replica has a worse rank
and is not easy to be elected.

Of course, it is possible that it will be elected with offset == 0.
In the future, we may need to prohibit the replica with offset == 0
from having the right to initiate elections.

Another point worth mentioning, in above cases:
1. In the ROLE command, the replica status will be handshake, and the
offset will be -1.
2. Before this PR, in the CLUSTER SHARD command, the replica status will
be online, and the offset will be the old cached value (which is wrong).
3. After this PR, in the CLUSTER SHARD, the replica status will be loading,
and the offset will be 0.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 92ba3d3443 Correct RDB_EOF_MARK_SIZE usage where EOF mark is relevant (#925)
In these places we should use RDB_EOF_MARK_SIZE, but we mixed
it with CONFIG_RUN_ID_SIZE. This is not an issue since they are
all 40, just a cleanup.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
9b12bdc374 Make use of a single listNode pointer for blocking utility lists (#919)
Saves some memory (one pointer) in the client struct.

Since a client cannot be blocked multiple times, we can assume
it will be held in only one extra utility list, so it is ok to maintain
a union of these listNode references. 

Signed-off-by: Ran Shidlansik <ranshid@amazon.com>
Signed-off-by: ranshid <88133677+ranshid@users.noreply.github.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
gmbnomisandMadelyn Olson 6fa83d8905 Fix valgrind timing issue failure in replica-redirect test (#917)
Wait for the replica to become online before starting the actual test.

Signed-off-by: Simon Baatz <gmbnomis@gmail.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 4ab8949b26 Optimize linear search of WAIT and WAITAOF when unblocking the client (#787)
Currently, if the client enters a blocked state, it will be
added to the server.clients_waiting_acks list. When the client
is unblocked, that is, when unblockClient is called, we will
need to linearly traverse server.clients_waiting_acks to delete
the client, and this search is O(N).

When WAIT (or WAITAOF) is used extensively in some cases, this
O(N) search may be time-consuming. We can remember the list node
and store it in the blockingState struct and it can avoid the
linear search in unblockClientWaitingReplicas.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
Wen HuiandMadelyn Olson 5a2ebc9112 Add 4 commands for sentinel and update most test cases and json files (#789)
Add 4 new commands for Sentinel (reference
https://github.com/valkey-io/valkey/issues/36)

Sentinel GET-PRIMARY-ADDR-BY-NAME
Sentinel PRIMARY
Sentinel PRIMARIES
Sentinel IS-PRIMARY-DOWN-BY-ADDR

and deprecate 4 old commands:

Sentinel GET-MASTER-ADDR-BY-NAME
Sentinel MASTER
Sentinel MASTERS
Sentinel IS-MASTER-DOWN-BY-ADDR

and all sentinel tests pass here
https://github.com/hwware/valkey/actions/runs/9962102363/job/27525124583

Note: 

1. runtest-sentinel pass all test cases
2. I finished a sentinel rolling upgrade test: 1 primary 2 replicas 3
sentinel
   there are 4 steps in this test scenario: 
step 1: all 3 sentinel nodes run old sentinel, shutdown primary, and
then new primary can be voted successfully.
step 2: replace sentinel 1 with new sentinel bin file, and then shutdown
primary, and then another new primary can be voted successfully
step 3: replace sentinel 2 with new sentinel bin file, and then shutdown
primary, and then another new primary can be voted successfully
step 4: replace sentinel 3 with new sentinel bin file, and then shutdown
primary, and then another new primary can be voted successfully
   
We can see, even mixed version sentinel running, whole system still
works.

---------

Signed-off-by: hwware <wen.hui.ware@gmail.com>
2024-09-03 09:00:45 -07:00
f4a72e7482 Add lfu support for DEBUG OBJECT command, added lfu_freq and lfu_access_time_minutes fields (#479)
For `debug object` command, we use `val->lru` but ignore the `lfu` mode.  
So in `lfu` mode, `debug object` would return meaningless `lru` descriptions. 

Added two new fields lfu_freq and lfu_access_time_minutes.

Signed-off-by: jiangyujie.jyj <yjjiang1996@163.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 4288a08782 Make a light weight version of DEBUG OBJECT, add FAST option (#881)
Adding FAST option to DEBUG OBJECT command.

The light version only shows the light weight infomation,
which mostly O(1). The pre-existing version that show more
stats such as serializedlength sometimes is time consuming.

This should allow looking into debug stats (the key expired
but not deleted), even on huge object, on which we're afraid
to run the command for fear of causing a server freeze.

Somehow like 98ee400b98.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 608dc468de Skip IPv6 tests when TCLSH version is < 8.6 (#910)
In #786, we did skip it in the daily, but not for the others.
When running ./runtest on MacOS, we will get the failure.
```
couldn't open socket: host is unreachable (nodename nor servname provided, or not known)
```

The reason is that TCL 8.5 doesn't support ipv6, so we skip tests
tagged with ipv6. This also revert #786.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
secwallandMadelyn Olson d4d3ac84d1 Log unexpected $ENDOFF responses in dual channel replication (#839)
I've tried to test a dual channel replication but forgot to add +sync
for my replication user. As a result replica entered silent cycle like
this:
```
* Connecting to PRIMARY 127.0.0.1:6379
* PRIMARY <-> REPLICA sync started
* Non blocking connect for SYNC fired the event.
* Primary replied to PING, replication can continue...
* Trying a partial resynchronization (request ...)
* PSYNC is not possible, initialize RDB channel.
* Aborting dual channel sync
```

And primary got endless cycle like this:
```
* Replica 127.0.0.1:6380 asks for synchronization
* Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '...', my replication IDs are '...' and '...')
* Replica 127.0.0.1:6380 is capable of dual channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated RDB channel.
```

There was no way to understand that replication user is missing +sync
acl on notice log level. With this one-line change we get a warning
message in our replica log.

---------

Signed-off-by: secwall <secwall@yandex-team.ru>
2024-09-03 09:00:45 -07:00
Pieter CailliauandMadelyn Olson fd10fdc8f5 Copyright update to reflect IP transfer from salvatore to Redis (#740)
Update references of copyright being assigned to Salvatore when it was
transferred to Redis Ltd. as per
https://github.com/valkey-io/valkey/issues/544.

---------

Signed-off-by: Pieter Cailliau <pieter@redis.com>
2024-09-03 09:00:45 -07:00
Salvatore MesoracaandMadelyn Olson 080069dabd Prevent later accesses to unallocated memory (#907)
A pointer to dtype is stored in the dict forever.
dtype is stack-allocated while the dict created is global.
The dict (and the pointer to dtype in it) will live past the lifetime of
dtype.
clusterManagerLinkDictType is a global object that has the same values
as dtype.

Signed-off-by: Salvatore Mesoraca <salvatore.mesoraca@aiven.io>
2024-09-03 09:00:45 -07:00
zhaozhao.zzandMadelyn Olson 0ef32c8c40 To avoid bouncing -REDIRECT during FAILOVER (#871)
Fix #821

During the `FAILOVER` process, when conditions are met (such as when the
force time is reached or the primary and replica offsets are
consistent), the primary actively becomes the replica and transitions to
the `FAILOVER_IN_PROGRESS` state. After the primary becomes the replica,
and after handshaking and other operations, it will eventually send the
`PSYNC FAILOVER` command to the replica, after which the replica will
become the primary. This means that the upgrade of the replica to the
primary is an asynchronous operation, which implies that during the
`FAILOVER_IN_PROGRESS` state, there may be a period of time where both
nodes are replicas. In this scenario, if a `-REDIRECT` is returned, the
request will be redirected to the replica and then redirected back,
causing back and forth redirection. To avoid this situation, during the
`FAILOVER_IN_PROGRESS state`, we temporarily suspend the clients that
need to be redirected until the replica truly becomes the primary, and
then resume the execution.

---------

Signed-off-by: zhaozhao.zz <zhaozhao.zz@alibaba-inc.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 2123819a71 Change server.daylight_active to an atomic variable (#876)
We are updating this variable in the main thread, and the
child threads can printing the logs at the same time. This
generating a warning in SANITIZER=thread:
```
WARNING: ThreadSanitizer: data race (pid=74208)
  Read of size 4 at 0x000102875c10 by thread T3:
    #0 serverLogRaw <null>:52173615 (valkey-server:x86_64+0x10003c556)
    #1 _serverLog <null>:52173615 (valkey-server:x86_64+0x10003ca89)
    #2 bioProcessBackgroundJobs <null>:52173615 (valkey-server:x86_64+0x1001402c9)

  Previous write of size 4 at 0x000102875c10 by main thread (mutexes: write M0):
    #0 afterSleep <null>:52173615 (valkey-server:x86_64+0x10004989b)
    #1 aeProcessEvents <null>:52173615 (valkey-server:x86_64+0x100031e52)
    #2 main <null>:52173615 (valkey-server:x86_64+0x100064a3c)
    #3 start <null>:52173615 (dyld:x86_64+0xfffffffffff5c365)
    #4 start <null>:52173615 (dyld:x86_64+0xfffffffffff5c365)
```

The refresh of daylight_active is not real time, we update
it in aftersleep, so we don't need a strong synchronization,
so using memory_order_relaxed. But also noted we are doing
load/store operations only for daylight_active, which is an
aligned 32-bit integer, so using memory_order_relaxed will
not provide more consistency than what we have today.

So this is just a cleanup that to clear the warning.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
Amit NaglerandMadelyn Olson ebee48c4a2 Fix dual-channel replication test under valgrind (#904)
Test dual-channel-replication primary gets cob overrun during replica
rdb load` fails during the Valgrind run. This is due to the load
handlers disconnecting before the tests complete, resulting in a low
primary COB. Increasing the handlers' timeout should resolve this issue.

Failure:
https://github.com/valkey-io/valkey/actions/runs/10361286333/job/28681321393

Server logs reveals that the load handler clients were disconnected
before the test started

Also the two previus test took about 20 seconds which is the handler
timeout.

---------

Signed-off-by: naglera <anagler123@gmail.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson c1da6e3993 Better messages when valkey-cli cluster --fix meet value check failed (#867)
The clusterManagerCompareKeysValues is introduced in
8f01adfb15, which calls
DEBUG DIGEST-VALUE to check whether the value of the
source node and the target node are consistent.

However, the DEBUG DIGEST-VALUE command is not supported
in older version, such as version 4, and the node will return
unknown subcommand. Or the DEBUG command can be disabled
in version 7, and the node will return DEBUG not allowed.

In these cases, we need to output friendly message to
allow users to proceed to the next step, instead of just
outputing `Value check failed!`.

Unknown subcommand example:
```
*** Target key exists
*** Checking key values on both nodes...
Node 127.0.0.1:30001 replied with error:
ERR unknown subcommand or wrong number of arguments for 'DIGEST-VALUE'. Try DEBUG HELP.
Node 127.0.0.1:30003 replied with error:
ERR unknown subcommand or wrong number of arguments for 'DIGEST-VALUE'. Try DEBUG HELP.
*** Value check failed!
DEBUG DIGEST-VALUE command is not supported.
You can relaunch the command with --cluster-replace option to force key overriding.
```

DEBUG not allowed example:
```
*** Target key exists
*** Checking key values on both nodes...
Node 127.0.0.1:30001 replied with error:
ERR DEBUG command not allowed. If the enable-debug-command option is ...
Node 127.0.0.1:30003 replied with error:
ERR DEBUG command not allowed. If the enable-debug-command option is ...
*** Value check failed!
DEBUG command is not allowed.
You can turn on the enable-debug-command option.
Or you can relaunch the command with --cluster-replace option to force key overriding.
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
RayacooandMadelyn Olson a651e8e683 Optimize ZUNION[STORE] command by removing unnecessary accumulator dict (#829)
In the past implementation of `ZUNION` and `ZUNIONSTORE` commands, we
first create a temporary dict called `accumulator`. After adding all
member-score mappings to `accumulator`, we still need to convert
`accumulator` back to the final dict `dstzset->dict`. However, we can
directly use `dstzset->dict` to avoid the additional copy operation.

This PR removes the `accumulator` dict and directly uses` dstzset->dict
`to store the member-score mappings.

- **Test**
First, I added 300 unique elements to two sorted sets called
'zunion_test1' and 'zunion_test2'. Then, I tested `zunion` and
`zunionstore` on these two sorted sets. The test results shown below
indicate that the performance of both zunion and zunionstore improved
about 31%.

### ZUNION
#### unstable
```
./valkey-benchmark -P 10 -n 100000 zunion 2 zunion_test1 zunion_test2

Summary:
  throughput summary: 2713.41 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
      146.252     3.464   153.343   182.015   184.959   192.895
```
#### This PR
```
./valkey-benchmark -P 10 -n 100000 zunion 2 zunion_test1 zunion_test2

Summary:
  throughput summary: 3543.84 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
      108.259     2.984   114.239   141.695   145.151   160.255
```
### ZUNIONSTORE
#### unstable
```
./valkey-benchmark -P 10 -n 100000 zunionstore out 2 zunion_test1 zunion_test2

Summary:
  throughput summary: 3168.07 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
      157.511     3.368   183.167   189.311   193.535   231.679
```
#### This PR
```
./valkey-benchmark -P 10 -n 100000 zunionstore out 2 zunion_test1 zunion_test2

Summary:
  throughput summary: 4144.73 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
      120.374     2.648   141.823   149.119   153.855   183.167
```

---------

Signed-off-by: RayCao <zisong.cw@alibaba-inc.com>
Signed-off-by: zisong.cw <zisong.cw@alibaba-inc.com>
2024-09-03 09:00:45 -07:00
511868bb07 Add debug-context config (#874)
A configuration option with zero impact on server operation but is
printed out on server crash and can be accessed by gdb for debugging. It
can be used by the user/operator to store any free-form string. This
string will persist as long as the server is running and will be
accessible in the following ways:

And printed in crash reports:
```
------ CONFIG DEBUG OUTPUT ------
lazyfree-lazy-eviction no
...
io-threads-do-reads yes
debug-context "test2"
proto-max-bulk-len 512mb
```

---------

Signed-off-by: Eran Liberty <eranl@amazon.com>
Co-authored-by: Eran Liberty <eranl@amazon.com>
2024-09-03 09:00:45 -07:00
2b33b547d3 Fix dual-channel-replication related issues (#837)
- Fix TLS bug where connection were shutdown by primary's main process
while the child process was still writing- causing main process to be
blocked.
- TLS connection fix -file descriptors are set to blocking mode in the
main thread, followed by a blocking write. This sets the file
descriptors to non-blocking if TLS is used (see `connTLSSyncWrite()`)
(@xbasel).
- Improve the reliability of dual-channel tests. Modify the pause
mechanism to verify process status directly, rather than relying on log.
- Ensure that `server.repl_offset` and `server.replid` are updated
correctly when dual channel synchronization completes successfully.
Thist led to failures in replication tests that validate replication IDs
or compare replication offsets.

---------

Signed-off-by: naglera <anagler123@gmail.com>
Signed-off-by: naglera <58042354+naglera@users.noreply.github.com>
Signed-off-by: xbasel <103044017+xbasel@users.noreply.github.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Signed-off-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: ranshid <88133677+ranshid@users.noreply.github.com>
Co-authored-by: xbasel <103044017+xbasel@users.noreply.github.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
nagleraandMadelyn Olson 143a32d683 Add debug assert on duplicate freeClientAsync (#896)
When debug assert mode enabled, verify that we don't insert the same
client twice into server.clients_to_close.

Signed-off-by: naglera <anagler123@gmail.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson f5577e7419 Dual channel replication should not update lastbgsave_status when transfer error (#811)
Currently lastbgsave_status is used in bgsave or disk-replication,
and the target is the disk. In #60, we update it when transfer error,
i think it is mainly used in tests, so we can use log to replace it.

It changes lastbgsave_status to err in this case, but it is strange
that it does not set ok or err in the above if and the following else.
Also noted this will affect stop-writes-on-bgsave-error.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson 585e60edd9 Correctly recode client infomation to the slowlog when runing script (#805)
Currently when we are runing a script, we will passing a fake client.
So if the command executed in the script is recoded in the slowlog,
the client's ip:port and name information will be empty.

before:
```
127.0.0.1:6379> client setname myclient
OK
127.0.0.1:6379> config set slowlog-log-slower-than 0
OK
127.0.0.1:6379> eval "redis.call('ping')" 0
(nil)
127.0.0.1:6379> slowlog get 2
1) 1) (integer) 2
   2) (integer) 1721314289
   3) (integer) 96
   4) 1) "eval"
      2) "redis.call('ping')"
      3) "0"
   5) "127.0.0.1:61106"
   6) "myclient"
2) 1) (integer) 1
   2) (integer) 1721314289
   3) (integer) 4
   4) 1) "ping"
   5) ""
   6) ""
```

after:
```
127.0.0.1:6379> client setname myclient
OK
127.0.0.1:6379> config set slowlog-log-slower-than 0
OK
127.0.0.1:6379> eval "redis.call('ping')" 0
(nil)
127.0.0.1:6379> slowlog get 2
1) 1) (integer) 2
   2) (integer) 1721314371
   3) (integer) 53
   4) 1) "eval"
      2) "redis.call('ping')"
      3) "0"
   5) "127.0.0.1:51983"
   6) "myclient"
2) 1) (integer) 1
   2) (integer) 1721314371
   3) (integer) 1
   4) 1) "ping"
   5) "127.0.0.1:51983"
   6) "myclient"
```

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
Harkrishn PatroandMadelyn Olson 81965923be Check if the server is currently running the feature before cron run (#838)
I think we should first check if the server is currently enabled in
cluster mode or if it has modules loaded prior to the throttled cron run
(`run_with_period`) condition.

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2024-09-03 09:00:45 -07:00
Harkrishn PatroandMadelyn Olson af3eb77b74 Assert network bytes out for replication slot stat computation is only allowed on primary (#847)
Added an assertion to avoid incorrect usage of the network bytes out for
replication code flow in slot stats computation.

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson f0754d70f1 Improve cluster cant failover log conditions (#780)
This PR adjusts the logging conditions of clusterLogCantFailover
in this two ways.

1. For the same cant_failover_reason, we will print the log once
in CLUSTER_CANT_FAILOVER_RELOG_PERIOD, but its value is 10s, which
is a bit long, shorten it to 1s, so we can better track its state.
We get to see the system making progress by watching the message.
Using 1s also covers pretty much all cases as i don't see a reason
for using a <1s node timeout, test or prod.

2. We will not print logs before the nolog_fail_time, its value
is cluster-node-timeout+5000. This may casue us to lose some logs,
for example, if cluster-node-timeout is small, auth_timeout will
be 2000, and auth_retry_time will be 4000. In this case, we will
lose all the reasons during the election if the failover is timedout.
So remove the nolog_fail_time logic, since we still do have the
CLUSTER_CANT_FAILOVER_RELOG_PERIOD logic, we won't print too many
logs.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
Yury-FridlyandandMadelyn Olson c4d6b45f50 Fix CI concurrency (#849)
Few CI improvements witch will reduce occupation CI queue and eliminate
stale runs.

1. Kill CI jobs on PRs once PR branch gets a new push. This will prevent
situation happened today - a huge job triggered twice in less than an
hour and occupied all **org** (for all repositories) runners queue for
the rest of the day (see pic). This completely blocked valkey-glide
team.
2. Distribute nightly croned jobs on time to prevent them running
together. Keep in mind, cron's TZ is UTC, so midnight tasks incur
developers located in other timezones.

This must be backported to all release branches (`valkey-x.y` and `x.y`)

![image](https://github.com/user-attachments/assets/923d8237-3cb7-42f5-80c8-5322b3f5187d)

---------

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
2024-09-03 09:00:45 -07:00
Harkrishn PatroandMadelyn Olson 8130caf97f Update sentinel conf access string to allow hello channel access (#854)
This example of a minimal user account in your Valkey server
for Sentinel is incorrect. If you add this ACL as-is to your
valkey users.acl, valkey will add resetchannels -@all before
the +client which prevents sentinel from publishing messages
to the __sentinel__:hello pubsub for sentinel discovery.

Fix #744.

Signed-off-by: Harkrishn Patro <harkrisp@amazon.com>
2024-09-03 09:00:45 -07:00
Wen HuiandMadelyn Olson 08ce534557 Update redis.conf to valkey.conf in log message (#855)
Update redis.conf to valkey.conf

Signed-off-by: hwware <wen.hui.ware@gmail.com>
2024-09-03 09:00:45 -07:00
BinbinandMadelyn Olson cb7edc4abc Fix outdated comment of migrate in valkey-cli --cluster (#864)
After 906dac85c2 the comment is outdated.

Signed-off-by: Binbin <binloveplay1314@qq.com>
2024-09-03 09:00:45 -07:00
Madelyn Olson 5d9ac5948f Disable empty shard slot migration test until test is de-flaked (#859)
We have a number of test failures in the empty shard migration which
seem to be related to race conditions in the failover, but could be more
pervasive. For now disable the tests to prevent so many false negative
test failures.

Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
2024-09-03 09:00:45 -07:00
52940f0899 Valkey 8.0 RC1 release notes (#850)
Signed-off-by: Ping Xie <pingxie@google.com>
Signed-off-by: Ping Xie <pingxie@outlook.com>
Signed-off-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
2024-08-01 08:59:43 -07:00
280 changed files with 6118 additions and 2996 deletions
+2
View File
@@ -5,6 +5,8 @@ extend-exclude = [
"deps/",
# crc16_slottable is primarily pre-generated random strings.
"src/crc16_slottable.h",
# 00-RELEASENOTES includes non-English names that might be flagged as typos in other contexts.
"00-RELEASENOTES",
]
[default.extend-words]
+14
View File
@@ -0,0 +1,14 @@
# This is a file that can be used by git-blame to ignore some revisions.
# (git 2.23+, released in August 2019)
#
# Can be configured as follow:
#
# $ git config blame.ignoreRevsFile .git-blame-ignore-revs
#
# For more information you can look at git-blame(1) man page.
# Applied clang-format (#323)
c41dd77a3e93e02be3c4bc75d8c76b7b4169a4ce
# Removed terms `master` and `slave` from the source code (#591)
54c97479356ecf41b4b63733494a1be2ab919e17
@@ -36,7 +36,7 @@ jobs:
build-valkey:
# Capture source tarball and generate checksum for it
name: Build package ${{ matrix.distro.target }} ${{ matrix.distro.arch }}
runs-on: 'ubuntu-latest'
runs-on: "ubuntu-latest"
strategy:
fail-fast: false
matrix: ${{ fromJSON(inputs.build_matrix) }}
@@ -51,15 +51,15 @@ jobs:
with:
arch: aarch64
distro: ${{matrix.distro.target}}
install: apt-get update && apt-get install -y build-essential libssl-dev
run: make -C src all BUILD_TLS=yes
install: apt-get update && apt-get install -y build-essential libssl-dev libsystemd-dev
run: make -C src all BUILD_TLS=yes USE_SYSTEMD=yes
- name: Create Tarball and SHA256sums
run: |
TAR_FILE_NAME=valkey-${{inputs.version}}-${{matrix.distro.platform}}-${{ matrix.distro.arch}}
mkdir -p $TAR_FILE_NAME/bin $TAR_FILE_NAME/share
cp -rfv src/valkey-* $TAR_FILE_NAME/bin
cp -v /home/runner/work/valkey/valkey/COPYING $TAR_FILE_NAME/share/LICENSE
mkdir -p "$TAR_FILE_NAME/bin" "$TAR_FILE_NAME/share"
rsync -av --exclude='*.c' --exclude='*.d' --exclude='*.o' src/valkey-* "$TAR_FILE_NAME/bin/"
cp -v /home/runner/work/valkey/valkey/COPYING "$TAR_FILE_NAME/share/LICENSE"
tar -czvf $TAR_FILE_NAME.tar.gz $TAR_FILE_NAME
sha256sum $TAR_FILE_NAME.tar.gz > $TAR_FILE_NAME.tar.gz.sha256
mkdir -p packages-files
@@ -36,7 +36,7 @@ jobs:
build-valkey:
# Capture source tarball and generate checksum for it
name: Build package ${{ matrix.distro.target }} ${{ matrix.distro.arch }}
runs-on: 'ubuntu-latest'
runs-on: "ubuntu-latest"
strategy:
fail-fast: false
matrix: ${{ fromJSON(inputs.build_matrix) }}
@@ -47,22 +47,26 @@ jobs:
ref: ${{ inputs.version }}
- name: Install dependencies
run: sudo apt-get update && sudo apt-get install -y build-essential libssl-dev jq wget awscli
run: sudo apt-get update && sudo apt-get install -y build-essential libssl-dev libsystemd-dev
- name: Make Valkey
run: make -C src all BUILD_TLS=yes
run: make -C src all BUILD_TLS=yes USE_SYSTEMD=yes
- name: Create Tarball and SHA256sums
run: |
TAR_FILE_NAME=valkey-${{inputs.version}}-${{matrix.distro.platform}}-${{ matrix.distro.arch}}
mkdir -p $TAR_FILE_NAME/bin $TAR_FILE_NAME/share
cp -rfv src/valkey-* $TAR_FILE_NAME/bin
cp -v /home/runner/work/valkey/valkey/COPYING $TAR_FILE_NAME/share/LICENSE
mkdir -p "$TAR_FILE_NAME/bin" "$TAR_FILE_NAME/share"
rsync -av --exclude='*.c' --exclude='*.d' --exclude='*.o' src/valkey-* "$TAR_FILE_NAME/bin/"
cp -v /home/runner/work/valkey/valkey/COPYING "$TAR_FILE_NAME/share/LICENSE"
tar -czvf $TAR_FILE_NAME.tar.gz $TAR_FILE_NAME
sha256sum $TAR_FILE_NAME.tar.gz > $TAR_FILE_NAME.tar.gz.sha256
mkdir -p packages-files
cp -rfv $TAR_FILE_NAME.tar* packages-files/
- name: Install AWS cli.
run: |
sudo apt-get install -y awscli
- name: Configure AWS credentials
run: |
aws configure set region us-west-2
+4
View File
@@ -2,6 +2,10 @@ name: CI
on: [push, pull_request]
concurrency:
group: ci-${{ github.head_ref || github.ref }}
cancel-in-progress: true
permissions:
contents: read
+4
View File
@@ -5,6 +5,10 @@ on:
paths:
- 'src/**'
concurrency:
group: clang-${{ github.head_ref || github.ref }}
cancel-in-progress: true
jobs:
clang-format-check:
runs-on: ubuntu-latest
+4
View File
@@ -4,6 +4,10 @@ name: "Codecov"
# where each PR needs to be compared against the coverage of the head commit
on: [push, pull_request]
concurrency:
group: codecov-${{ github.head_ref || github.ref }}
cancel-in-progress: true
jobs:
code-coverage:
runs-on: ubuntu-latest
+5 -1
View File
@@ -4,7 +4,11 @@ on:
pull_request:
schedule:
# run weekly new vulnerability was added to the database
- cron: '0 0 * * 0'
- cron: '0 3 * * 0'
concurrency:
group: codeql-${{ github.head_ref || github.ref }}
cancel-in-progress: true
permissions:
contents: read
+7 -1
View File
@@ -3,11 +3,17 @@ name: Coverity Scan
on:
schedule:
# Run once daily, since below 500k LOC can have 21 builds per week, per https://scan.coverity.com/faq#frequency
- cron: '0 0 * * *'
- cron: '0 1 * * *'
# Support manual execution
workflow_dispatch:
concurrency:
group: coverity-${{ github.head_ref || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
coverity:
if: github.repository == 'valkey-io/valkey'
+18 -14
View File
@@ -29,6 +29,10 @@ on:
description: "git branch or sha to use"
default: "unstable"
concurrency:
group: daily-${{ github.head_ref || github.ref }}
cancel-in-progress: true
permissions:
contents: read
@@ -61,7 +65,7 @@ jobs:
run: sudo apt-get install tcl8.6 tclx
- name: test
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
run: ./runtest ${{ github.event_name != 'pull_request' && '--accurate' || '' }} --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
@@ -110,7 +114,7 @@ jobs:
run: apt-get install -y tcl8.6 tclx procps
- name: test
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
run: ./runtest ${{ github.event_name != 'pull_request' && '--accurate' || '' }} --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
@@ -155,7 +159,7 @@ jobs:
run: sudo apt-get install tcl8.6 tclx
- name: test
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
run: ./runtest ${{ github.event_name != 'pull_request' && '--accurate' || '' }} --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
@@ -194,7 +198,7 @@ jobs:
run: sudo apt-get install tcl8.6 tclx
- name: test
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
run: ./runtest ${{ github.event_name != 'pull_request' && '--accurate' || '' }} --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
@@ -235,7 +239,7 @@ jobs:
run: sudo apt-get install tcl8.6 tclx
- name: test
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
run: ./runtest ${{ github.event_name != 'pull_request' && '--accurate' || '' }} --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: |
@@ -286,7 +290,7 @@ jobs:
- name: test
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: |
./runtest --accurate --verbose --dump-logs --tls --dump-logs ${{github.event.inputs.test_args}}
./runtest ${{ github.event_name != 'pull_request' && '--accurate' || '' }} --verbose --dump-logs --tls --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: |
@@ -332,7 +336,7 @@ jobs:
- name: test
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: |
./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
./runtest ${{ github.event_name != 'pull_request' && '--accurate' || '' }} --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: |
@@ -375,10 +379,10 @@ jobs:
run: sudo apt-get install tcl8.6 tclx
- name: test
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: ./runtest --config io-threads 2 --config events-per-io-thread 0 --accurate --verbose --tags network --dump-logs ${{github.event.inputs.test_args}}
run: ./runtest --io-threads --accurate --verbose --tags network --dump-logs ${{github.event.inputs.test_args}}
- name: cluster tests
if: true && !contains(github.event.inputs.skiptests, 'cluster')
run: ./runtest-cluster --config io-threads 2 --config events-per-io-thread 0 ${{github.event.inputs.cluster_test_args}}
run: ./runtest-cluster --io-threads ${{github.event.inputs.cluster_test_args}}
test-ubuntu-reclaim-cache:
runs-on: ubuntu-latest
@@ -753,7 +757,7 @@ jobs:
run: dnf -y install tcl tcltls
- name: test
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
run: ./runtest ${{ github.event_name != 'pull_request' && '--accurate' || '' }} --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
@@ -822,7 +826,7 @@ jobs:
- name: test
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: |
./runtest --accurate --verbose --dump-logs --tls-module --dump-logs ${{github.event.inputs.test_args}}
./runtest ${{ github.event_name != 'pull_request' && '--accurate' || '' }} --verbose --dump-logs --tls-module --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: |
@@ -934,7 +938,7 @@ jobs:
run: make SERVER_CFLAGS='-Werror'
- name: test
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: ./runtest --accurate --verbose --tags -ipv6 --clients 1 --no-latency --dump-logs ${{github.event.inputs.test_args}}
run: ./runtest ${{ github.event_name != 'pull_request' && '--accurate' || '' }} --verbose --clients 1 --no-latency --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --clients 1 --no-latency --dump-logs ${{github.event.inputs.test_args}}
@@ -1086,7 +1090,7 @@ jobs:
run: apk add tcl procps tclx
- name: test
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
run: ./runtest ${{ github.event_name != 'pull_request' && '--accurate' || '' }} --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
@@ -1127,7 +1131,7 @@ jobs:
run: apk add tcl procps tclx
- name: test
if: true && !contains(github.event.inputs.skiptests, 'valkey')
run: ./runtest --accurate --verbose --dump-logs ${{github.event.inputs.test_args}}
run: ./runtest ${{ github.event_name != 'pull_request' && '--accurate' || '' }} --verbose --dump-logs ${{github.event.inputs.test_args}}
- name: module api test
if: true && !contains(github.event.inputs.skiptests, 'modules')
run: CFLAGS='-Werror' ./runtest-moduleapi --verbose --dump-logs ${{github.event.inputs.test_args}}
+5 -1
View File
@@ -4,7 +4,11 @@ on:
pull_request:
push:
schedule:
- cron: '0 0 * * *'
- cron: '0 2 * * *'
concurrency:
group: external-${{ github.head_ref || github.ref }}
cancel-in-progress: true
permissions:
contents: read
@@ -8,6 +8,10 @@ on:
paths:
- 'src/commands/*.json'
concurrency:
group: reply-schemas-linter-${{ github.head_ref || github.ref }}
cancel-in-progress: true
permissions:
contents: read
+4
View File
@@ -9,6 +9,10 @@ on:
push:
pull_request:
concurrency:
group: spellcheck-${{ github.head_ref || github.ref }}
cancel-in-progress: true
permissions:
contents: read
+408 -11
View File
@@ -1,16 +1,413 @@
Hello! This file is just a placeholder, since this is the "unstable" branch
of Valkey, the place where all the development happens.
Valkey 8.0 release notes
========================
--------------------------------------------------------------------------------
Upgrade urgency levels:
LOW: No need to upgrade unless there are new features you want to use.
MODERATE: Program an upgrade of the server, but it's not urgent.
HIGH: There is a critical bug that may affect a subset of users. Upgrade!
CRITICAL: There is a critical bug affecting MOST USERS. Upgrade ASAP.
SECURITY: There are security fixes in the release.
--------------------------------------------------------------------------------
There is no release notes for this branch, it gets forked into another branch
every time there is a partial feature freeze in order to eventually create
a new stable release.
================================================================================
Valkey 8.0.1 - Released Tue 1 Oct 2024
================================================================================
Usually "unstable" is stable enough for you to use it in development environments
however you should never use it in production environments. It is possible
to download the latest stable release here:
Upgrade urgency SECURITY: This release includes security fixes we recommend you
apply as soon as possible.
https://valkey.io/download/
Bug fixes
=========
* Fix a build issue with RDMA when using additional make parameters. (#1074)
* Fix an issue where `CLUSTER SLOTS` might return the wrong tcp or tls port when called
from inside a script or from a module. (#1072)
* Fix a crash when `CLUSTER SLOTS` or `CLUSTER SHARDS` is called from inside
a script or from a module. (#1063)
* Fix a build issue on systems where `<threads.h>` is unavailable. (#1053)
* Fix an issue with the default `sentinel.conf` being invalid. (#1040)
More information is available at https://valkey.io
Security fixes
==============
Happy hacking!
* (CVE-2024-31449) Lua library commands may lead to stack overflow and potential RCE.
* (CVE-2024-31227) Potential Denial-of-service due to malformed ACL selectors.
* (CVE-2024-31228) Potential Denial-of-service due to unbounded pattern matching.
================================================================================
Valkey 8.0.0 GA - Released Sun 15 Sep 2024
================================================================================
Upgrade urgency LOW: This is the first release of Valkey 8.0, which
includes stability and performance improvements over the second release
candidate. This release is fully compatible with Redis OSS 7.2.4.
Logging and Tooling Improvements
================================
* Added full client info to SHUTDOWN and CLUSTER FAILOVER logs for better traceability
of requests. (#875)
Bug fixes
=========
* Resolved issues in replicationSetPrimary where the primary node's IP/port updates were
not correctly handled in the cluster gossip section. (#965)
* Fixed AOF base suffix during rewrites when modifying the aof-use-rdb-preamble setting,
ensuring correct suffix caching to prevent inconsistencies. (#886)
* Addressed rare crashes in async IO threads with TLS by preventing concurrent read and
write job overlap. (#1011)
* Prevented AOF from being incorrectly disabled after loading RDB data, ensuring proper
re-enabling of AOF. (#1001)
* Triggered a save of the cluster configuration file before shutdown to prevent
inconsistencies caused by unsaved node configuration changes. (#822)
* Fixed timing issue in CLUSTER SETSLOT to ensure replicas handle migration correctly
when receiving the command before the gossip update. (#981)
Performance
===========
* Optimized the handling of temporary set objects in SUNION and SDIFF commands, resulting
in a 41% performance improvement for SUNION and 27% for SDIFF. (#996)
Behavior Changes
================
* Replicas now flush old data after checking RDB file is valid during disk-based replication,
preventing partial data loss and ensuring a clean data load. (#926)
================================================================================
Valkey 8.0.0 RC2 - Released Tue 2 Sep 2024
================================================================================
Upgrade urgency LOW: This is the second release candidate of Valkey 8.0, which
generally includes stability and performance improvements over the first release
candidate. This release is fully compatible with Redis OSS 7.2.4.
New Features
============
* Add support for setting the group on a unix domain socket. (#901)
* Experimental - RDMA: Support user keepalive command. (#916)
Logging and Tooling Improvements
================================
* Improve logging when a cluster is unable to failover. (#780)
* Add epoch information to failover auth denied logs. (#816)
* Improve messaging when valkey-cli cluster --fix meet check failed. (#867)
* Log unexpected $ENDOFF responses in dual channel replication. (#839)
* Skip IPv6 tests when it is unsupported on TCLSH. (#910)
* Add a light weight version of DEBUG OBJECT. (#881)
* Add lfu support for DEBUG OBJECT command, added lfu_freq and
lfu_access_time_minutes fields. (#479)
Bug fixes
=========
* Correctly expose client infomation in the slowlog when running commands in scripts. (#805)
* Fix a bug where lastbgsave_status was unecessarily being updated when dual
channel replication failed. (#811)
* Fix a bug which caused dual channel replication to get stuck because of a TLS
issue. (#837)
* Fix a bug which caused bouncing `-REDIRECT` messages during a FAILOVER (#871)
* Avoid re-establishing replication to a node that is already the primary in `CLUSTER REPLICATE`. (#884)
* Fix `CLUSTER SETSLOT` with `BLOCK` to only wait for replicas that are online. (#879)
* Fix `valkey-cli` to make source node ignores NOREPLICAS when doing the last CLUSTER SETSLOT (#928)
* Exclude '.' and ':' from `isValidAuxChar`'s banned charset to better support
IPV6 addresses. (#963)
* Better handle `-REDIRECT` messages in the MULTI context. (#895)
Performance
===========
* Improve multithreaded performance with memory prefetching. (#861)
* Optimize ZUNION[STORE] command by removing unnecessary accumulator dict. (#829)
* Optimize linear search of WAIT and WAITAOF when unblocking the client. (#787)
* Move prepareClientToWrite out of loop for lrange command to remove redundant calls. (#860)
* Optimize various commands by using sdsAllocSize instead of sdsZmallocSize. (#923)
* Using intrinsics to optimize counting HyperLogLog trailing bits. (#846)
* Free client's MULTI state when it becomes dirty to reduce memory usage. (#961)
* Optimize the fast path of SET if the expiration time has already expired. (#865)
Compatibility Improvements
==========================
* Add 4 sentinel command `GET-PRIMARY-ADDR-BY-NAME`, `PRIMARY`, `PRIMARIES`, and
`IS-PRIMARY-DOWN-BY-ADDR` to allow clients to use inclusive language. (#789)
Behavior Changes
================
* Block unsubscribe related commands for clients that are not in subscribed mode. (#759)
Configuration Changes
=====================
* Change repl-backlog-size from 1MB to 10MB by default. (#911)
* Remove the protected flag from `dual-channel-replication`. (#908)
================================================================================
Valkey 8.0.0 RC1 - Released Thu 1 Aug 2024
================================================================================
Upgrade urgency LOW: This is the first release candidate of Valkey 8.0, with
performance, reliability, and observability improvements. It includes asynchronous
I/O threading, better cluster scaling reliability, dual primary-replica channel
for faster full synchronization, per-slot metrics for resource management, and
experimental RDMA support for increased throughput and reduced latency. This
release is fully compatible with Redis OSS 7.2.4.
Un-deprecated Commands - Cluster
================================
* Un-deprecate the `CLUSTER SLOTS` command. (#536)
New/Modified Commands - Core
============================
* Add `SCRIPT SHOW` sub-command to dump scripts via SHA1. (#617)
* Add `NOSCORES` option to `ZSCAN` command. (#324)
* Add `NOVALUES` option to `HSCAN` command. (Redis#12765)
* Expose Lua `os.clock()` API to allow scripts to determine how long the
script has been executing. (Redis#12971)
* Implement `CLIENT KILL MAXAGE <MAXAGE>`. (Redis#12299)
* Allow running `WAITAOF` in scripts, remove `NOSCRIPT` flag. (Redis#12977)
* Support `XREAD[GROUP]` with `BLOCK` option in scripts. (Redis#12596)
* Introduce `+` as a special ID for the last item in stream on `XREAD`
Command. (Redis#7388, Redis#13117)
New/Modified Commands - Cluster
===============================
* Introduce `CLUSTER SLOT-STATS` command which allows tracking of per slot
metrics for key count, CPU utilization, network bytes in, and network
bytes out. (#20, #351)
* Add `TIMEOUT` option to `CLUSTER SETSLOT` command. (#556, #445)
New/Modified Metrics - Core
===========================
* Introduce per-client metrics for network traffic and command execution
in `CLIENT LIST` and `CLIENT INFO`. (#327)
* Add metrics for DB memory overhead and rehashing status to `INFO MEMORY`
and `MEMORY STATS`. (Redis#12913)
* Add `pubsub_clients` metric to `INFO CLIENTS`. (Redis#12849)
* Add metrics for client buffer limit disconnections to `INFO`. (Redis#12476)
* Add metrics for monitoring clients using `WATCH` command and watched keys.
(Redis#12966)
* Added allocator muzzy memory metrics to `INFO MEMORY` and `MEMORY STATS`.
(Redis#12996)
New Features - Core
===================
* Support replica redirect for read/write operations to primary in standalone
mode. (#325)
* Add server config for cluster blacklist TTL. (#738)
* Add availability zone server config. (#700)
New Features - Cluster
======================
* Support IPv4 and IPv6 dual stack and client-specific IPs in clusters. (#736)
* Support `BY/GET` options for `SORT/SORT_RO` in cluster mode when pattern
implies a single slot. (Redis#12728)
Compatibility Improvements - Core
=================================
* Derive RDB and module child process names based on server start name for
compatibility. (#454)
* Update server identity in `serverPanic` output based on `extended-redis-compatibility`
config. (#415)
Compatibility Improvements - Sentinel
=====================================
* Accept `redis-sentinel` to start Valkey in sentinel mode. (#731)
Performance/Efficiency Improvements - Core
==========================================
* Introduce dual channel for more efficient full sync replication. (#60)
* Introduce async IO threading for improved multi-threaded performance.
(#763, #758)
* Embed key directly in main dictionary entry for improved memory efficiency.
(#541)
* Use thread-local storage to reduce atomic contention in updating memory
metrics. (#674)
* Reduce redundant calls to `prepareClientToWrite` for continuous `addReply*`.
(#670)
* Optimize the logic for checking conversion to skip list during `ZADD` operations.
(#806)
* Optimize `sdsfree` with `zfree_with_size` to avoid redundant size calculation.
(#453)
* Combine events to eliminate redundant `kevent(2)` calls. (#638)
* Introduce shared query buffer for client reads to reduce memory usage. (#258)
* Optimize CRC64 performance for large batches by processing bytes in parallel.
(#350)
* Use `SOCK_NONBLOCK` to reduce system calls for outgoing connections. (#293)
* Enable `accept4()` detection on specific versions of various platforms. (#294)
* Convert CRC16 slot table to fixed-size array for improved memory efficiency.
(Redis#13112)
* Run `SCRIPT FLUSH` truly asynchronously and close Lua interpreter in a
background thread. (Redis#13087)
* Optimize `DEL` command to avoid redundant deletions for expired keys. (Redis#13080)
* Improve defragmentation for large bins to enhance memory efficiency. (Redis#12996)
* Optimize hash table resizing to include empty dictionaries. (Redis#12819)
* Reduce performance impact of dictionary rehashing by optimizing bucket processing.
(Redis#12899)
* Optimize performance for simultaneous client `[P|S]UNSUBSCRIBE`. (Redis#12838)
* Optimize CPU cache efficiency during dictionary rehashing. (Redis#5692)
* Optimize `ZRANGE` offset location from linear search to skip list jump. (Redis#12450)
* Fix `aeSetDontWait` timing to avoid unnecessary waits in `aeProcessEvent`. (Redis#12068)
Performance/Efficiency Improvements - Cluster
=============================================
* Add lightweight cluster message header for Pub/Sub messages. (#654)
* Minor performance improvement in Valkey cluster by avoid initializing key
buffer in `getKeysResult`. (#631)
* Cache `CLUSTER SLOTS` response to improve throughput and reduce latency. (#53)
* Replace slots_to_channels radix tree with slot-specific dictionaries for
shard channels. (Redis#12804)
* Optimize `KEYS` command when pattern includes hashtag and implies a single
slot. (Redis#12754)
* Optimize `SCAN` command with `MATCH` when pattern implies a single slot.
(Redis#12536)
* Replace cluster metadata with slot specific dictionaries to reduce memory
usage when using Valkey cluster. (Redis#11695, Redis#12704)
Reliability Improvements - Core
===============================
* Limit tracking custom errors (e.g. from Lua) while allowing normal errors
to be tracked (#500, Redis#13141)
* Manage maximum number of new connections per cycle to prevent connection
storms. (Redis#12178)
Reliability Improvements - Cluster
==================================
* Reduce fail-over time in Valkey cluster when multiple sequential fail-overs
occurred by resetting `failover_auth_time` when the new primary node goes
down. (#782)
* Restrict node failure marking to primaries with assigned slots. (#634)
* Enhance cluster meet reliability under link failures. (#461)
* Improve reliability of slot migration in Valkey clusters. (#445)
Usability Improvements - Core
=============================
* Re-brand and refine latency report messages. (#644)
* Optimize `ACL LOAD` to avoid disconnecting clients whose users are unchanged.
(Redis#12171)
Usability Improvements - Cluster
================================
* Adjust log levels for various cluster-related logs to improve clarity. (#633)
* Maintain deterministic ordering of replica(s) in `CLUSTER SLOTS` response. (#265)
Usability Improvements - CLI
================================
* Add prompt message when Ctrl-C is pressed in `valkey-cli`. (#702)
* Keep an in-memory history of all commands in `valkey-cli` so that sensitive
commands can be shown within the same session. (Redis#12862)
Module Improvements - Core
==========================
* Add `ValkeyModule_TryCalloc()` and `ValkeyModule_TryRealloc()` to handle
allocation failures gracefully. (Redis#12985)
* Make `ValkeyModule_Yield` thread-safe by handling events in the main thread.
(Redis#12905)
* Allow modules to declare new ACL categories. (Redis#12486)
Module Improvements - Cluster
=============================
* Add API `ValkeyModule_ClusterKeySlot` and `ValkeyModule_ClusterCanonicalKeyNameInSlot`.
(Redis#13069)
Behavior Changes - Core
=======================
* Re-brand the Lua debugger. (#603)
* Change default pidfile from `redis.pid` to `valkey.pid`. (#378)
* Abort transactions on nested `MULTI` or `WATCH` commands. (#723)
* Ensure keys that match the `SCAN` filter are not lazily expired and return
an error for invalid types. (#501)
* Rename `redis` in AOF logs and proc title to `valkey-aof-rewrite`. (#393)
* Change default syslog-ident from `redis` to `valkey`. (#390)
* Update `Redis` to `Valkey` in `serverLog` messages in server.c file. (#231)
* Remove `Redis` from various error reply messages. See GitHub PR for more
details. (#206)
* Reject empty strings for configs `dir`, `dbfilename`, and `cluster-config-file`.
(#636)
* Change key-spec flag from `RW` to `OW` for `SINTERSTORE` command. (Redis#12917)
* Return more precise error messages for some cases verifying keys during script
execution. (Redis#12707)
* Return errors for `BITCOUNT` and `BITPOS` with non-existing keys or invalid
arguments instead of zero. (Redis#11734)
* Validate `BITCOUNT` arguments before key existence check. (Redis#12394)
* Redact ACL username information and mark `*-key-file-pass` configs as
sensitive. (Redis#12860)
* Allow `MULTI/EXEC` to use a small amount of additional memory beyond the
used-memory limit. (Redis#12961)
Behavior Changes - Cluster
==========================
* Allow `CLUSTER NODES/INFO/MYID/MYSHARDID` during loading state. (#596)
* Make cluster replicas return `ASK` and `TRYAGAIN` during slot migration. (#495)
Behavior Changes - Sentinel
===========================
* Replace `master-reboot-down-after-period` with `primary-reboot-down-after-period`
in `sentinel.conf`. (#647)
Bug Fixes - Core
================
* Fix a bug that caused LRU/LFU inconsistencies for some integer objects. (#250)
* Fix a bug where Valkey may use a sub-optimal encoding for some data types.
(Redis#13148)
* Fix propagation of `entries_read` by calling `streamPropagateGroupID`
unconditionally. (Redis#12898)
* Fix race condition issues between the main thread and module threads.
(Redis#12817)
* Wake blocked clients ASAP in next `beforeSleep` for `WAITAOF`. (Redis#12627)
* Fix crash in crash-report and improve thread management with RW locks.
(Redis#12623)
Bug Fixes - Cluster
===================
* Fix a bug where a shard returns the incorrect slot slot information in
`CLUSTER SHARDS` command on primary failure. (#790)
* Allow module authentication to succeed when the cluster is down. (#693)
* Fix `PONG` message processing for primary-ship tracking during fail-overs.
(Redis#13055)
* Prevent double freeing of cluster link with `DEBUG CLUSTERLINK KILL`.
(Redis#12930)
* Unsubscribe all clients from replica for shard channel if the primary
ownership changes. (Redis#12577)
Bug Fixes - Tooling
===============
* Fix `valkey-check-aof` misidentifying data in manifest format as MP-AOF.
(Redis#12958)
* Fix `valkey-cli` to respect the `--count` option without requiring
`--pattern`. (Redis#13092)
* Fix `valkey-benchmark` to distribute operations across all slots owned by
a node in cluster mode. (Redis#12986)
Internal Codebase Improvements
==============================
* Enable debug asserts for cluster and sentinel tests. (#588)
* Introduce a minimal debugger for Tcl integration test suite. (#683)
* Set up clang-format GitHub action for automated code formatting checks. (#538)
* Replace custom atomic logic with C11 _Atomics. (#490)
* Add fast fail option for Tcl test cases. (#482)
* Introduce a simple unit test framework. (#460)
* An initial simple unit test framework. (#344)
* Introduce Codecov for automated code coverage tracking. (#316)
* Remove deprecated `redis-trib` CLI program. (#281)
* Add `-fno-omit-frame-pointer` to default compilation flags to improve
debuggability. (Redis#12973)
* Refactor the per-slot dict-array db.c into a new kvstore data structure.
(Redis#12822)
* Unified database rehash method for both standalone and cluster modes.
(Redis#12848)
* Clarify and decouple the sampling logic in eviction to improve readability.
(Redis#12781)
* Rewrite large printf calls to smaller ones for readability. (Redis#12257)
Experimental
============
* Introduce Valkey Over RDMA transport (experimental). (#477)
We appreciate the efforts of all who contributed code to this release!
lan Slang, Binbin, Brennan, Chen Tianjie, Cui Fliter, Daniel House, Darren Jiang,
David Carlier, Debing Sun, Dingrui, Dmitry Polyakovsky, Eran Liberty, Gabi Ganam,
George Guimares, Guillaume Koenig, Guybe, Harkrishn Patro, Hassaan Khan, Hwang Si Yeon,
ICHINOSE Shogo, icy17, Ikko Eltociear Ashimine, iKun, Itamar Haber, Jachin, Jacob Murphy,
Jason Elbaum, Jeff Liu, John Sully, John Vandenberg, Jonathan Wright, Jonghoonpark, Joe Hu,
Josiah Carlson, Juho Kim, judeng, Jun Luo, K.G. Wang, Karthik Subbarao, Karthick Ariyaratnam,
kell0gg, Kyle Kim, Leibale Eidelman, LiiNen, Lipeng Zhu, Lior Kogan, Lior Lahav, Madelyn Olson,
Makdon, Maria Markova, Mason Hall, Matthew Douglass, meiravgri, michalbiesek, Mike Dolan,
Mikel Olasagasti Uranga, Moshe Kaplan, mwish, naglera, NAM UK KIM, Neal Gompa, nitaicaro,
Nir Rattner, Oran Agra, Ouri Half, Ozan Tezcan, Parth, PatrickJS, Pengfei Han, Pierre, Ping Xie,
poiuj, pshankinclarke, ranshid, Ronen Kalish, Roshan Khatri, Samuel Adetunji, Sankar, secwall,
Sergey Fedorov, Sher_Sun, Shivshankar, skyfirelee, Slava Koyfman, Subhi Al Hasan, sundb,
Ted Lyngmo, Thomas Fline, tison, Tom Morris, Tyler Bream, uriyage, Viktor Söderqvist, Vitaly,
Vitah Lin, VoletiRam, w. ian douglas, WangYu, Wen Hui, Wenwen Chen, Yaacov Hazan, Yanqi Lv,
Yehoshua Hershberg, Yves LeBras, zalj, Zhao Zhao, zhenwei pi, zisong.cw
-1
View File
@@ -1 +0,0 @@
Please check https://github.com/valkey-io/valkey/issues
+1 -1
View File
@@ -17,7 +17,7 @@ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
BSD 3-Clause License
Copyright (c) 2006-2020, Salvatore Sanfilippo
Copyright (c) 2006-2020, Redis Ltd.
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
-1
View File
@@ -1 +0,0 @@
See README
-106
View File
@@ -1,106 +0,0 @@
[Note: This was the manifesto of Redis. It does not represent the ideals of Valkey, but is
kept in remembrance for the ideals that Salvatore had for the project.]
Redis Manifesto
===============
1 - A DSL for Abstract Data Types. Redis is a DSL (Domain Specific Language)
that manipulates abstract data types and implemented as a TCP daemon.
Commands manipulate a key space where keys are binary-safe strings and
values are different kinds of abstract data types. Every data type
represents an abstract version of a fundamental data structure. For instance
Redis Lists are an abstract representation of linked lists. In Redis, the
essence of a data type isn't just the kind of operations that the data types
support, but also the space and time complexity of the data type and the
operations performed upon it.
2 - Memory storage is #1. The Redis data set, composed of defined key-value
pairs, is primarily stored in the computer's memory. The amount of memory in
all kinds of computers, including entry-level servers, is increasing
significantly each year. Memory is fast, and allows Redis to have very
predictable performance. Datasets composed of 10k or 40 millions keys will
perform similarly. Complex data types like Redis Sorted Sets are easy to
implement and manipulate in memory with good performance, making Redis very
simple. Redis will continue to explore alternative options (where data can
be optionally stored on disk, say) but the main goal of the project remains
the development of an in-memory database.
3 - Fundamental data structures for a fundamental API. The Redis API is a direct
consequence of fundamental data structures. APIs can often be arbitrary but
not an API that resembles the nature of fundamental data structures. If we
ever meet intelligent life forms from another part of the universe, they'll
likely know, understand and recognize the same basic data structures we have
in our computer science books. Redis will avoid intermediate layers in API,
so that the complexity is obvious and more complex operations can be
performed as the sum of the basic operations.
4 - We believe in code efficiency. Computers get faster and faster, yet we
believe that abusing computing capabilities is not wise: the amount of
operations you can do for a given amount of energy remains anyway a
significant parameter: it allows to do more with less computers and, at
the same time, having a smaller environmental impact. Similarly Redis is
able to "scale down" to smaller devices. It is perfectly usable in a
Raspberry Pi and other small ARM based computers. Faster code having
just the layers of abstractions that are really needed will also result,
often, in more predictable performances. We think likewise about memory
usage, one of the fundamental goals of the Redis project is to
incrementally build more and more memory efficient data structures, so that
problems that were not approachable in RAM in the past will be perfectly
fine to handle in the future.
5 - Code is like a poem; it's not just something we write to reach some
practical result. Sometimes people that are far from the Redis philosophy
suggest using other code written by other authors (frequently in other
languages) in order to implement something Redis currently lacks. But to us
this is like if Shakespeare decided to end Enrico IV using the Paradiso from
the Divina Commedia. Is using any external code a bad idea? Not at all. Like
in "One Thousand and One Nights" smaller self contained stories are embedded
in a bigger story, we'll be happy to use beautiful self contained libraries
when needed. At the same time, when writing the Redis story we're trying to
write smaller stories that will fit in to other code.
6 - We're against complexity. We believe designing systems is a fight against
complexity. We'll accept to fight the complexity when it's worthwhile but
we'll try hard to recognize when a small feature is not worth 1000s of lines
of code. Most of the time the best way to fight complexity is by not
creating it at all. Complexity is also a form of lock-in: code that is
very hard to understand cannot be modified by users in an independent way
regardless of the license. One of the main Redis goals is to remain
understandable, enough for a single programmer to have a clear idea of how
it works in detail just reading the source code for a couple of weeks.
7 - Threading is not a silver bullet. Instead of making Redis threaded we
believe on the idea of an efficient (mostly) single threaded Redis core.
Multiple of such cores, that may run in the same computer or may run
in multiple computers, are abstracted away as a single big system by
higher order protocols and features: Redis Cluster and the upcoming
Redis Proxy are our main goals. A shared nothing approach is not just
much simpler (see the previous point in this document), is also optimal
in NUMA systems. In the specific case of Redis it allows for each instance
to have a more limited amount of data, making the Redis persist-by-fork
approach more sounding. In the future we may explore parallelism only for
I/O, which is the low hanging fruit: minimal complexity could provide an
improved single process experience.
8 - Two levels of API. The Redis API has two levels: 1) a subset of the API fits
naturally into a distributed version of Redis and 2) a more complex API that
supports multi-key operations. Both are useful if used judiciously but
there's no way to make the more complex multi-keys API distributed in an
opaque way without violating our other principles. We don't want to provide
the illusion of something that will work magically when actually it can't in
all cases. Instead we'll provide commands to quickly migrate keys from one
instance to another to perform multi-key operations and expose the
trade-offs to the user.
9 - We optimize for joy. We believe writing code is a lot of hard work, and the
only way it can be worth is by enjoying it. When there is no longer joy in
writing code, the best thing to do is stop. To prevent this, we'll avoid
taking paths that will make Redis less of a joy to develop.
10 - All the above points are put together in what we call opportunistic
programming: trying to get the most for the user with minimal increases
in complexity (hanging fruits). Solve 95% of the problem with 5% of the
code when it is acceptable. Avoid a fixed schedule but follow the flow of
user requests, inspiration, Redis internal readiness for certain features
(sometimes many past changes reach a critical point making a previously
complex feature very easy to obtain).
+53 -10
View File
@@ -1,10 +1,8 @@
[![codecov](https://codecov.io/gh/valkey-io/valkey/graph/badge.svg?token=KYYSJAYC5F)](https://codecov.io/gh/valkey-io/valkey)
This README is under construction as we work to build a new community driven high performance key-value store.
This project was forked from the open source Redis project right before the transition to their new source available licenses.
This README is just a fast *quick start* document. We are currently working on a more permanent documentation page.
This README is just a fast *quick start* document. More details can be found under [valkey.io](https://valkey.io/)
What is Valkey?
--------------
@@ -27,10 +25,18 @@ It is as simple as:
% make
To build with TLS support, you'll need OpenSSL development libraries (e.g.
libssl-dev on Debian/Ubuntu) and run:
libssl-dev on Debian/Ubuntu).
To build TLS support as Valkey built-in:
% make BUILD_TLS=yes
TO build TLS as Valkey module:
% make BUILD_TLS=module
Note that sentinel mode does not support TLS module.
To build with experimental RDMA support you'll need RDMA development libraries
(e.g. librdmacm-dev and libibverbs-dev on Debian/Ubuntu). For now, Valkey only
supports RDMA as connection module mode. Run:
@@ -54,12 +60,16 @@ After building Valkey, it is a good idea to test it using:
% make test
If TLS is built, running the tests with TLS enabled (you will need `tcl-tls`
installed):
The above runs the main integration tests. Additional tests are started using:
% ./utils/gen-test-certs.sh
% ./runtest --tls
% make test-unit # Unit tests
% make test-modules # Tests of the module API
% make test-sentinel # Valkey Sentinel integration tests
% make test-cluster # Valkey Cluster integration tests
More about running the integration tests can be found in
[tests/README.md](tests/README.md) and for unit tests, see
[src/unit/README.md](src/unit/README.md).
Fixing build problems with dependencies or cached build options
---------
@@ -158,8 +168,41 @@ line, with exactly the same name.
Running Valkey with TLS:
------------------
Please consult the [TLS.md](TLS.md) file for more information on
how to use Valkey with TLS.
### Running manually
To manually run a Valkey server with TLS mode (assuming `./gen-test-certs.sh` was invoked so sample certificates/keys are available):
* TLS built-in mode:
```
./src/valkey-server --tls-port 6379 --port 0 \
--tls-cert-file ./tests/tls/valkey.crt \
--tls-key-file ./tests/tls/valkey.key \
--tls-ca-cert-file ./tests/tls/ca.crt
```
* TLS module mode:
```
./src/valkey-server --tls-port 6379 --port 0 \
--tls-cert-file ./tests/tls/valkey.crt \
--tls-key-file ./tests/tls/valkey.key \
--tls-ca-cert-file ./tests/tls/ca.crt \
--loadmodule src/valkey-tls.so
```
Note that you can disable TCP by specifying `--port 0` explicitly.
It's also possible to have both TCP and TLS available at the same time,
but you'll have to assign different ports.
Use `valkey-cli` to connect to the Valkey server:
```
./src/valkey-cli --tls \
--cert ./tests/tls/valkey.crt \
--key ./tests/tls/valkey.key \
--cacert ./tests/tls/ca.crt
```
Specifying `--tls-replication yes` makes a replica connect to the primary.
Using `--tls-cluster yes` makes Valkey Cluster use TLS across nodes.
Running Valkey with RDMA:
------------------
-106
View File
@@ -1,106 +0,0 @@
TLS Support
===========
Getting Started
---------------
### Building
To build with TLS support you'll need OpenSSL development libraries (e.g.
libssl-dev on Debian/Ubuntu).
To build TLS support as Valkey built-in:
Run `make BUILD_TLS=yes`.
Or to build TLS as Valkey module:
Run `make BUILD_TLS=module`.
Note that sentinel mode does not support TLS module.
### Tests
To run Valkey test suite with TLS, you'll need TLS support for TCL (i.e.
`tcl-tls` package on Debian/Ubuntu).
1. Run `./utils/gen-test-certs.sh` to generate a root CA and a server
certificate.
2. Run `./runtest --tls` or `./runtest-cluster --tls` to run Valkey and Valkey
Cluster tests in TLS mode.
3. Run `./runtest --tls-module` or `./runtest-cluster --tls-module` to
run Valkey and Valkey cluster tests in TLS mode with Valkey module.
### Running manually
To manually run a Valkey server with TLS mode (assuming `gen-test-certs.sh` was
invoked so sample certificates/keys are available):
For TLS built-in mode:
./src/valkey-server --tls-port 6379 --port 0 \
--tls-cert-file ./tests/tls/valkey.crt \
--tls-key-file ./tests/tls/valkey.key \
--tls-ca-cert-file ./tests/tls/ca.crt
For TLS module mode:
./src/valkey-server --tls-port 6379 --port 0 \
--tls-cert-file ./tests/tls/valkey.crt \
--tls-key-file ./tests/tls/valkey.key \
--tls-ca-cert-file ./tests/tls/ca.crt \
--loadmodule src/valkey-tls.so
To connect to this Valkey server with `valkey-cli`:
./src/valkey-cli --tls \
--cert ./tests/tls/valkey.crt \
--key ./tests/tls/valkey.key \
--cacert ./tests/tls/ca.crt
Specifying `port 0` will disable TCP. It's also possible to have
both TCP and TLS available, but you'll need to assign different ports.
To make a Replica connect to the master using TLS, use `--tls-replication yes`,
and to make Valkey Cluster use TLS across nodes use `--tls-cluster yes`.
Connections
-----------
All socket operations now go through a connection abstraction layer that hides
I/O and read/write event handling from the caller.
**Multi-threading I/O is not currently supported for TLS**, as a TLS connection
needs to do its own manipulation of AE events which is not thread safe. The
solution is probably to manage independent AE loops for I/O threads and longer
term association of connections with threads. This may potentially improve
overall performance as well.
Sync IO for TLS is currently implemented in a hackish way, i.e. making the
socket blocking and configuring socket-level timeout. This means the timeout
value may not be so accurate, and there would be a lot of syscall overhead.
However I believe that getting rid of syncio completely in favor of pure async
work is probably a better move than trying to fix that. For replication it would
probably not be so hard. For cluster keys migration it might be more difficult,
but there are probably other good reasons to improve that part anyway.
To-Do List
----------
- [ ] valkey-benchmark support. The current implementation is a mix of using
hiredis for parsing and basic networking (establishing connections), but
directly manipulating sockets for most actions. This will need to be cleaned
up for proper TLS support. The best approach is probably to migrate to hiredis
async mode.
- [ ] valkey-cli `--slave` and `--rdb` support.
Multi-port
----------
Consider the implications of allowing TLS to be configured on a separate port,
making Valkey listening on multiple ports:
1. Startup banner port notification
2. Proctitle
3. How slaves announce themselves
4. Cluster bus port calculation
+1 -1
View File
@@ -6,7 +6,7 @@
* [1] https://www.cs.tufts.edu/~nr/cs257/archive/florian-loitsch/printf.pdf
* ----------------------------------------------------------------------------
*
* Copyright (c) 2021, Redis Labs
* Copyright (c) 2021, Redis Ltd.
* Copyright (c) 2013-2019, night-shift <as.smljk at gmail dot com>
* Copyright (c) 2009, Florian Loitsch < florian.loitsch at inria dot fr >
* All rights reserved.
+1 -1
View File
@@ -1,4 +1,4 @@
Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
Copyright (c) 2009-2011, Redis Ltd.
Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
All rights reserved.
+1 -1
View File
@@ -1,5 +1,5 @@
# Hiredis Makefile
# Copyright (C) 2010-2011 Salvatore Sanfilippo <antirez at gmail dot com>
# Copyright (C) 2010-2011 Redis Ltd.
# Copyright (C) 2010-2011 Pieter Noordhuis <pcnoordhuis at gmail dot com>
# This file is released under the BSD license, see the COPYING file
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
+1 -1
View File
@@ -5,7 +5,7 @@
* tables of power of two in size are used, collisions are handled by
* chaining. See the source code for more information... :)
*
* Copyright (c) 2006-2010, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-2010, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -5,7 +5,7 @@
* tables of power of two in size are used, collisions are handled by
* chaining. See the source code for more information... :)
*
* Copyright (c) 2006-2010, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-2010, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2020, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2020, Redis Ltd.
* Copyright (c) 2020, Pieter Noordhuis <pcnoordhuis at gmail dot com>
* Copyright (c) 2020, Matt Stancliff <matt at genges dot com>,
* Jan-Erik Rediger <janerik at fnordig dot com>
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
* Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
* Jan-Erik Rediger <janerik at fnordig dot com>
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
* Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
* Jan-Erik Rediger <janerik at fnordig dot com>
+1 -1
View File
@@ -1,6 +1,6 @@
/*
* Copyright (c) 2019, Redis Labs
* Copyright (c) 2019, Redis Ltd.
*
* All rights reserved.
*
+1 -1
View File
@@ -1,6 +1,6 @@
/* Extracted from anet.c to work properly with Hiredis error reporting.
*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
* Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
* Jan-Erik Rediger <janerik at fnordig dot com>
+1 -1
View File
@@ -1,6 +1,6 @@
/* Extracted from anet.c to work properly with Hiredis error reporting.
*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
* Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
* Jan-Erik Rediger <janerik at fnordig dot com>
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
+1 -2
View File
@@ -1,8 +1,7 @@
/* SDSLib 2.0 -- A C dynamic strings library
*
* Copyright (c) 2006-2015, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-2015, Redis Ltd.
* Copyright (c) 2015, Oran Agra
* Copyright (c) 2015, Redis Labs, Inc
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -2
View File
@@ -1,8 +1,7 @@
/* SDSLib 2.0 -- A C dynamic strings library
*
* Copyright (c) 2006-2015, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-2015, Redis Ltd.
* Copyright (c) 2015, Oran Agra
* Copyright (c) 2015, Redis Labs, Inc
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -2
View File
@@ -1,8 +1,7 @@
/* SDSLib 2.0 -- A C dynamic strings library
*
* Copyright (c) 2006-2015, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-2015, Redis Ltd.
* Copyright (c) 2015, Oran Agra
* Copyright (c) 2015, Redis Labs, Inc
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+2 -2
View File
@@ -1,7 +1,7 @@
/*
* Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2011, Redis Ltd.
* Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
* Copyright (c) 2019, Redis Labs
* Copyright (c) 2019, Redis Ltd.
*
* All rights reserved.
*
+1 -1
View File
@@ -10,7 +10,7 @@
*
* ------------------------------------------------------------------------
*
* Copyright (c) 2010-2016, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2010-2016, Redis Ltd.
* Copyright (c) 2010-2013, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
+1 -1
View File
@@ -7,7 +7,7 @@
*
* ------------------------------------------------------------------------
*
* Copyright (c) 2010-2014, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2010-2014, Redis Ltd.
* Copyright (c) 2010-2013, Pieter Noordhuis <pcnoordhuis at gmail dot com>
*
* All rights reserved.
+1
View File
@@ -132,6 +132,7 @@ static int bit_tohex(lua_State *L)
const char *hexdigits = "0123456789abcdef";
char buf[8];
int i;
if (n == INT32_MIN) n = INT32_MIN+1;
if (n < 0) { n = -n; hexdigits = "0123456789ABCDEF"; }
if (n > 8) n = 8;
for (i = (int)n; --i >= 0; ) { buf[i] = hexdigits[b & 15]; b >>= 4; }
+3 -3
View File
@@ -10,7 +10,7 @@
#define LUACMSGPACK_NAME "cmsgpack"
#define LUACMSGPACK_SAFE_NAME "cmsgpack_safe"
#define LUACMSGPACK_VERSION "lua-cmsgpack 0.4.0"
#define LUACMSGPACK_COPYRIGHT "Copyright (C) 2012, Salvatore Sanfilippo"
#define LUACMSGPACK_COPYRIGHT "Copyright (C) 2012, Redis Ltd."
#define LUACMSGPACK_DESCRIPTION "MessagePack C implementation for Lua"
/* Allows a preprocessor directive to override MAX_NESTING */
@@ -39,7 +39,7 @@
/* =============================================================================
* MessagePack implementation and bindings for Lua 5.1/5.2.
* Copyright(C) 2012 Salvatore Sanfilippo <antirez@gmail.com>
* Copyright(C) 2012 Redis Ltd.
*
* http://github.com/antirez/lua-cmsgpack
*
@@ -958,7 +958,7 @@ LUALIB_API int luaopen_cmsgpack_safe(lua_State *L) {
}
/******************************************************************************
* Copyright (C) 2012 Salvatore Sanfilippo. All rights reserved.
* Copyright (C) 2012 Redis Ltd. All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining
* a copy of this software and associated documentation files (the
+1
View File
@@ -56,4 +56,5 @@ $TCLSH tests/test_helper.tcl \
--single unit/moduleapi/moduleauth \
--single unit/moduleapi/rdbloadsave \
--single unit/moduleapi/crash \
--single unit/moduleapi/getchannels \
"${@}"
+2 -2
View File
@@ -120,7 +120,7 @@ sentinel monitor mymaster 127.0.0.1 6379 2
# Sentinel instances, should be configured along the following lines:
#
# user sentinel-user >somepassword +client +subscribe +publish \
# +ping +info +multi +slaveof +config +client +exec on
# +ping +info +multi +slaveof +config +client +exec &__sentinel__:hello on
# sentinel down-after-milliseconds <master-name> <milliseconds>
#
@@ -355,4 +355,4 @@ SENTINEL announce-hostnames no
# accept a -LOADING response after a primary has been rebooted, before failing
# over.
SENTINEL primary-reboot-down-after-period myprimary 0
SENTINEL primary-reboot-down-after-period mymaster 0
+1
View File
@@ -30,3 +30,4 @@ SortIncludes: false
AllowAllParametersOfDeclarationOnNextLine: false
BinPackParameters: false
AlignAfterOpenBracket: Align
InsertNewlineAtEOF: true
+3 -3
View File
@@ -1,5 +1,5 @@
# Valkey Makefile
# Copyright (C) 2009 Salvatore Sanfilippo <antirez at gmail dot com>
# Copyright (C) 2009 Redis Ltd.
# This file is released under the BSD license, see the COPYING file
#
# The Makefile composes the final FINAL_CFLAGS and FINAL_LDFLAGS using
@@ -423,7 +423,7 @@ endif
ENGINE_NAME=valkey
SERVER_NAME=$(ENGINE_NAME)-server$(PROG_SUFFIX)
ENGINE_SENTINEL_NAME=$(ENGINE_NAME)-sentinel$(PROG_SUFFIX)
ENGINE_SERVER_OBJ=threads_mngr.o adlist.o quicklist.o ae.o anet.o dict.o kvstore.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o io_threads.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o cluster_legacy.o cluster_slot_stats.o crc16.o endianconv.o slowlog.o eval.o bio.o rio.o rand.o memtest.o syscheck.o crcspeed.o crccombine.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o valkey-check-rdb.o valkey-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o lolwut.o lolwut5.o lolwut6.o acl.o tracking.o socket.o tls.o sha256.o timeout.o setcpuaffinity.o monotonic.o mt19937-64.o resp_parser.o call_reply.o script_lua.o script.o functions.o function_lua.o commands.o strl.o connection.o unix.o logreqres.o
ENGINE_SERVER_OBJ=threads_mngr.o adlist.o quicklist.o ae.o anet.o dict.o kvstore.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o memory_prefetch.o io_threads.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o cluster_legacy.o cluster_slot_stats.o crc16.o endianconv.o slowlog.o eval.o bio.o rio.o rand.o memtest.o syscheck.o crcspeed.o crccombine.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o valkey-check-rdb.o valkey-check-aof.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o t_stream.o listpack.o localtime.o lolwut.o lolwut5.o lolwut6.o acl.o tracking.o socket.o tls.o sha256.o timeout.o setcpuaffinity.o monotonic.o mt19937-64.o resp_parser.o call_reply.o script_lua.o script.o functions.o function_lua.o commands.o strl.o connection.o unix.o logreqres.o
ENGINE_CLI_NAME=$(ENGINE_NAME)-cli$(PROG_SUFFIX)
ENGINE_CLI_OBJ=anet.o adlist.o dict.o valkey-cli.o zmalloc.o release.o ae.o serverassert.o crcspeed.o crccombine.o crc64.o siphash.o crc16.o monotonic.o cli_common.o mt19937-64.o strl.o cli_commands.o
ENGINE_BENCHMARK_NAME=$(ENGINE_NAME)-benchmark$(PROG_SUFFIX)
@@ -513,7 +513,7 @@ $(TLS_MODULE_NAME): $(SERVER_NAME)
$(QUIET_CC)$(CC) -o $@ tls.c -shared -fPIC $(TLS_MODULE_CFLAGS) $(TLS_CLIENT_LIBS)
# valkey-rdma.so
$(RDMA_MODULE_NAME): $(REDIS_SERVER_NAME)
$(RDMA_MODULE_NAME): $(SERVER_NAME)
$(QUIET_CC)$(CC) -o $@ rdma.c -shared -fPIC $(RDMA_MODULE_CFLAGS)
# valkey-cli
+3 -3
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2018, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2018, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -1079,7 +1079,7 @@ int ACLSetSelector(aclSelector *selector, const char *op, size_t oplen) {
flags |= ACL_READ_PERMISSION;
} else if (toupper(op[offset]) == 'W' && !(flags & ACL_WRITE_PERMISSION)) {
flags |= ACL_WRITE_PERMISSION;
} else if (op[offset] == '~') {
} else if (op[offset] == '~' && flags) {
offset++;
break;
} else {
@@ -2678,7 +2678,7 @@ void addACLLogEntry(client *c, int reason, int context, int argpos, sds username
/* if we have a real client from the network, use it (could be missing on module timers) */
client *realclient = server.current_client ? server.current_client : c;
le->cinfo = catClientInfoString(sdsempty(), realclient);
le->cinfo = catClientInfoString(sdsempty(), realclient, 0);
le->context = context;
/* Try to match this entry with past ones, to see if we can just
+1 -1
View File
@@ -1,6 +1,6 @@
/* adlist.c - A generic doubly linked list implementation
*
* Copyright (c) 2006-2010, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-2010, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -1,6 +1,6 @@
/* adlist.h - A generic doubly linked list implementation
*
* Copyright (c) 2006-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -2,7 +2,7 @@
* for the Jim's event-loop (Jim is a Tcl interpreter) but later translated
* it in form of a library for easy reuse.
*
* Copyright (c) 2006-2010, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-2010, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -2,7 +2,7 @@
* for the Jim's event-loop (Jim is a Tcl interpreter) but later translated
* it in form of a library for easy reuse.
*
* Copyright (c) 2006-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -1,6 +1,6 @@
/* Linux epoll(2) based ae.c module
*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -1,6 +1,6 @@
/* Select()-based ae.c module.
*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+22 -5
View File
@@ -1,6 +1,6 @@
/* anet.c -- Basic TCP socket stuff made a bit less boring
*
* Copyright (c) 2006-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -45,6 +45,7 @@
#include <errno.h>
#include <stdarg.h>
#include <stdio.h>
#include <grp.h>
#include "anet.h"
#include "config.h"
@@ -505,7 +506,7 @@ int anetTcpNonBlockBestEffortBindConnect(char *err, const char *addr, int port,
return anetTcpGenericConnect(err, addr, port, source_addr, ANET_CONNECT_NONBLOCK | ANET_CONNECT_BE_BINDING);
}
static int anetListen(char *err, int s, struct sockaddr *sa, socklen_t len, int backlog, mode_t perm) {
static int anetListen(char *err, int s, struct sockaddr *sa, socklen_t len, int backlog, mode_t perm, char *group) {
if (bind(s, sa, len) == -1) {
anetSetError(err, "bind: %s", strerror(errno));
close(s);
@@ -514,6 +515,22 @@ static int anetListen(char *err, int s, struct sockaddr *sa, socklen_t len, int
if (sa->sa_family == AF_LOCAL && perm) chmod(((struct sockaddr_un *)sa)->sun_path, perm);
if (sa->sa_family == AF_LOCAL && group != NULL) {
struct group *grp;
if ((grp = getgrnam(group)) == NULL) {
anetSetError(err, "getgrnam error for group '%s': %s", group, strerror(errno));
close(s);
return ANET_ERR;
}
/* Owner of the socket remains same. */
if (chown(((struct sockaddr_un *)sa)->sun_path, -1, grp->gr_gid) == -1) {
anetSetError(err, "chown error for group '%s': %s", group, strerror(errno));
close(s);
return ANET_ERR;
}
}
if (listen(s, backlog) == -1) {
anetSetError(err, "listen: %s", strerror(errno));
close(s);
@@ -553,7 +570,7 @@ static int _anetTcpServer(char *err, int port, char *bindaddr, int af, int backl
if (af == AF_INET6 && anetV6Only(err, s) == ANET_ERR) goto error;
if (anetSetReuseAddr(err, s) == ANET_ERR) goto error;
if (anetListen(err, s, p->ai_addr, p->ai_addrlen, backlog, 0) == ANET_ERR) s = ANET_ERR;
if (anetListen(err, s, p->ai_addr, p->ai_addrlen, backlog, 0, NULL) == ANET_ERR) s = ANET_ERR;
goto end;
}
if (p == NULL) {
@@ -577,7 +594,7 @@ int anetTcp6Server(char *err, int port, char *bindaddr, int backlog) {
return _anetTcpServer(err, port, bindaddr, AF_INET6, backlog);
}
int anetUnixServer(char *err, char *path, mode_t perm, int backlog) {
int anetUnixServer(char *err, char *path, mode_t perm, int backlog, char *group) {
int s;
struct sockaddr_un sa;
@@ -593,7 +610,7 @@ int anetUnixServer(char *err, char *path, mode_t perm, int backlog) {
memset(&sa, 0, sizeof(sa));
sa.sun_family = AF_LOCAL;
valkey_strlcpy(sa.sun_path, path, sizeof(sa.sun_path));
if (anetListen(err, s, (struct sockaddr *)&sa, sizeof(sa), backlog, perm) == ANET_ERR) return ANET_ERR;
if (anetListen(err, s, (struct sockaddr *)&sa, sizeof(sa), backlog, perm, group) == ANET_ERR) return ANET_ERR;
return s;
}
+2 -2
View File
@@ -1,6 +1,6 @@
/* anet.c -- Basic TCP socket stuff made a bit less boring
*
* Copyright (c) 2006-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -56,7 +56,7 @@ int anetTcpNonBlockBestEffortBindConnect(char *err, const char *addr, int port,
int anetResolve(char *err, char *host, char *ipbuf, size_t ipbuf_len, int flags);
int anetTcpServer(char *err, int port, char *bindaddr, int backlog);
int anetTcp6Server(char *err, int port, char *bindaddr, int backlog);
int anetUnixServer(char *err, char *path, mode_t perm, int backlog);
int anetUnixServer(char *err, char *path, mode_t perm, int backlog, char *group);
int anetTcpAccept(char *err, int serversock, char *ip, size_t ip_len, int *port);
int anetUnixAccept(char *err, int serversock);
int anetNonBlock(char *err, int fd);
+7 -5
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -423,7 +423,7 @@ void aofManifestFreeAndUpdate(aofManifest *am) {
* appendonly.aof.1.base.aof (server.aof_use_rdb_preamble is no)
* appendonly.aof.1.base.rdb (server.aof_use_rdb_preamble is yes)
*/
sds getNewBaseFileNameAndMarkPreAsHistory(aofManifest *am) {
sds getNewBaseFileNameAndMarkPreAsHistory(aofManifest *am, int aof_use_rdb_preamble) {
serverAssert(am != NULL);
if (am->base_aof_info) {
serverAssert(am->base_aof_info->file_type == AOF_FILE_TYPE_BASE);
@@ -431,7 +431,7 @@ sds getNewBaseFileNameAndMarkPreAsHistory(aofManifest *am) {
listAddNodeHead(am->history_aof_list, am->base_aof_info);
}
char *format_suffix = server.aof_use_rdb_preamble ? RDB_FORMAT_SUFFIX : AOF_FORMAT_SUFFIX;
char *format_suffix = aof_use_rdb_preamble ? RDB_FORMAT_SUFFIX : AOF_FORMAT_SUFFIX;
aofInfo *ai = aofInfoCreate();
ai->file_name = sdscatprintf(sdsempty(), "%s.%lld%s%s", server.aof_filename, ++am->curr_base_file_seq,
@@ -712,7 +712,7 @@ void aofOpenIfNeededOnServerStart(void) {
/* If we start with an empty dataset, we will force create a BASE file. */
size_t incr_aof_len = listLength(server.aof_manifest->incr_aof_list);
if (!server.aof_manifest->base_aof_info && !incr_aof_len) {
sds base_name = getNewBaseFileNameAndMarkPreAsHistory(server.aof_manifest);
sds base_name = getNewBaseFileNameAndMarkPreAsHistory(server.aof_manifest, server.aof_use_rdb_preamble);
sds base_filepath = makePath(server.aof_dirname, base_name);
if (rewriteAppendOnlyFile(base_filepath) != C_OK) {
exit(1);
@@ -1371,6 +1371,7 @@ struct client *createAOFClient(void) {
*/
c->raw_flag = 0;
c->flag.deny_blocking = 1;
c->flag.fake = 1;
/* We set the fake client as a replica waiting for the synchronization
* so that the server will not try to send replies to this client. */
@@ -2445,6 +2446,7 @@ int rewriteAppendOnlyFileBackground(void) {
serverLog(LL_NOTICE, "Background append only file rewriting started by pid %ld", (long)childpid);
server.aof_rewrite_scheduled = 0;
server.aof_rewrite_time_start = time(NULL);
server.aof_rewrite_use_rdb_preamble = server.aof_use_rdb_preamble;
return C_OK;
}
return C_OK; /* unreached */
@@ -2557,7 +2559,7 @@ void backgroundRewriteDoneHandler(int exitcode, int bysignal) {
/* Get a new BASE file name and mark the previous (if we have)
* as the HISTORY type. */
sds new_base_filename = getNewBaseFileNameAndMarkPreAsHistory(temp_am);
sds new_base_filename = getNewBaseFileNameAndMarkPreAsHistory(temp_am, server.aof_rewrite_use_rdb_preamble);
serverAssert(new_base_filename != NULL);
new_base_filepath = makePath(server.aof_dirname, new_base_filename);
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* Copyright (c) 2024, Valkey contributors
* All rights reserved.
*
+1 -1
View File
@@ -45,7 +45,7 @@
*
* ----------------------------------------------------------------------------
*
* Copyright (c) 2015, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2015, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -31,7 +31,7 @@
*
* ----------------------------------------------------------------------------
*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -1,6 +1,6 @@
/* Bit operations.
*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+15 -5
View File
@@ -1,6 +1,6 @@
/* blocked.c - generic support for blocking operations like BLPOP & WAIT.
*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -76,10 +76,13 @@ static void releaseBlockedEntry(client *c, dictEntry *de, int remove_key);
void initClientBlockingState(client *c) {
c->bstate.btype = BLOCKED_NONE;
c->bstate.timeout = 0;
c->bstate.unblock_on_nokey = 0;
c->bstate.keys = dictCreate(&objectKeyHeapPointerValueDictType);
c->bstate.numreplicas = 0;
c->bstate.numlocal = 0;
c->bstate.reploffset = 0;
c->bstate.unblock_on_nokey = 0;
c->bstate.generic_blocked_list_node = NULL;
c->bstate.module_blocked_handle = NULL;
c->bstate.async_rm_call_handle = NULL;
}
@@ -191,8 +194,9 @@ void unblockClient(client *c, int queue_for_reprocessing) {
if (moduleClientIsBlockedOnKeys(c)) unblockClientWaitingData(c);
unblockClientFromModule(c);
} else if (c->bstate.btype == BLOCKED_POSTPONE) {
listDelNode(server.postponed_clients, c->postponed_list_node);
c->postponed_list_node = NULL;
serverAssert(c->bstate.postponed_list_node);
listDelNode(server.postponed_clients, c->bstate.postponed_list_node);
c->bstate.postponed_list_node = NULL;
} else if (c->bstate.btype == BLOCKED_SHUTDOWN) {
/* No special cleanup. */
} else {
@@ -595,6 +599,11 @@ void blockClientForReplicaAck(client *c, mstime_t timeout, long long offset, lon
c->bstate.numreplicas = numreplicas;
c->bstate.numlocal = numlocal;
listAddNodeHead(server.clients_waiting_acks, c);
/* Note that we remember the linked list node where the client is stored,
* this way removing the client in unblockClientWaitingReplicas() will not
* require a linear scan, but just a constant time operation. */
serverAssert(c->bstate.client_waiting_acks_list_node == NULL);
c->bstate.client_waiting_acks_list_node = listFirst(server.clients_waiting_acks);
blockClient(c, BLOCKED_WAIT);
}
@@ -605,7 +614,8 @@ void blockPostponeClient(client *c) {
c->bstate.timeout = 0;
blockClient(c, BLOCKED_POSTPONE);
listAddNodeTail(server.postponed_clients, c);
c->postponed_list_node = listLast(server.postponed_clients);
serverAssert(c->bstate.postponed_list_node == NULL);
c->bstate.postponed_list_node = listLast(server.postponed_clients);
/* Mark this client to execute its command */
c->flag.pending_command = 1;
}
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2021, Redis Labs Ltd.
* Copyright (c) 2009-2021, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2021, Redis Labs Ltd.
* Copyright (c) 2009-2021, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2016, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2016, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -1,6 +1,6 @@
/* CLI (command line interface) common methods
*
* Copyright (c) 2020, Redis Labs
* Copyright (c) 2020, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+16 -3
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -262,6 +262,8 @@ void restoreCommand(client *c) {
if (ttl && !absttl) ttl += commandTimeSnapshot();
if (ttl && checkAlreadyExpired(ttl)) {
if (deleted) {
/* Here we don't use deleteExpiredKeyFromOverwriteAndPropagate because
* strictly speaking, the `delete` is triggered by the `replace`. */
robj *aux = server.lazyfree_lazy_server_del ? shared.unlink : shared.del;
rewriteClientCommandVector(c, 2, aux, key);
signalModifiedKey(c, c->db, key);
@@ -748,7 +750,16 @@ int verifyClusterNodeId(const char *name, int length) {
}
int isValidAuxChar(int c) {
return isalnum(c) || (strchr("!#$%&()*+.:;<>?@[]^{|}~", c) == NULL);
/* Return true if the character is alphanumeric */
if (isalnum(c)) {
return 1;
}
/* List of invalid characters */
static const char *invalid_charset = "!#$%&()*+;<>?@[]^{|}~";
/* Return true if the character is NOT in the invalid charset */
return strchr(invalid_charset, c) == NULL;
}
int isValidAuxString(char *s, unsigned int length) {
@@ -820,6 +831,8 @@ void clusterCommandHelp(client *c) {
"SLOTS",
" Return information about slots range mappings. Each range is made of:",
" start, end, primary and replicas IP addresses, ports and ids",
"SLOT-STATS",
" Return an array of slot usage statistics for slots assigned to the current node.",
"SHARDS",
" Return information about slot range mappings and the nodes associated with them.",
NULL};
@@ -1415,7 +1428,7 @@ void clusterCommandSlots(client *c) {
* ... continued until done
*/
int conn_type = 0;
if (connIsTLS(c->conn)) conn_type |= CACHE_CONN_TYPE_TLS;
if (shouldReturnTlsInfo()) conn_type |= CACHE_CONN_TYPE_TLS;
if (isClientConnIpV6(c)) conn_type |= CACHE_CONN_TYPE_IPv6;
if (c->resp == 3) conn_type |= CACHE_CONN_TYPE_RESP3;
+1 -1
View File
@@ -39,6 +39,7 @@ void clusterInitLast(void);
void clusterCron(void);
void clusterBeforeSleep(void);
int verifyClusterConfigWithData(void);
void clusterHandleServerShutdown(void);
int clusterSendModuleMessageToTarget(const char *target,
uint64_t module_id,
@@ -83,7 +84,6 @@ int getNodeDefaultClientPort(clusterNode *n);
clusterNode *getMyClusterNode(void);
int getClusterSize(void);
int getMyShardSlotCount(void);
int handleDebugClusterCommand(client *c);
int clusterNodePending(clusterNode *node);
int clusterNodeIsPrimary(clusterNode *n);
char **getClusterNodesList(size_t *numnodes);
+168 -64
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -62,14 +62,13 @@ void clusterSendPing(clusterLink *link, int type);
void clusterSendFail(char *nodename);
void clusterSendFailoverAuthIfNeeded(clusterNode *node, clusterMsg *request);
void clusterUpdateState(void);
int clusterNodeCoversSlot(clusterNode *n, int slot);
list *clusterGetNodesInMyShard(clusterNode *node);
int clusterNodeAddReplica(clusterNode *primary, clusterNode *replica);
int clusterAddSlot(clusterNode *n, int slot);
int clusterDelSlot(int slot);
int clusterDelNodeSlots(clusterNode *node);
int clusterNodeSetSlotBit(clusterNode *n, int slot);
void clusterSetPrimary(clusterNode *n, int closeSlots);
static void clusterSetPrimary(clusterNode *n, int closeSlots, int full_sync_required);
void clusterHandleReplicaFailover(void);
void clusterHandleReplicaMigration(int max_replicas);
int bitmapTestBit(unsigned char *bitmap, int pos);
@@ -93,7 +92,6 @@ void moduleCallClusterReceivers(const char *sender_id,
uint32_t len);
const char *clusterGetMessageTypeString(int type);
void removeChannelsInSlot(unsigned int slot);
unsigned int countKeysInSlot(unsigned int hashslot);
unsigned int countChannelsInSlot(unsigned int hashslot);
unsigned int delKeysInSlot(unsigned int hashslot);
void clusterAddNodeToShard(const char *shard_id, clusterNode *node);
@@ -1193,6 +1191,28 @@ void clusterInitLast(void) {
}
}
/* Called when a cluster node receives SHUTDOWN. */
void clusterHandleServerShutdown(void) {
/* The error logs have been logged in the save function if the save fails. */
serverLog(LL_NOTICE, "Saving the cluster configuration file before exiting.");
clusterSaveConfig(1);
#if !defined(__sun)
/* Unlock the cluster config file before shutdown, see clusterLockConfig.
*
* This is needed if you shutdown a very large server process, it will take
* a while for the OS to release resources and unlock the cluster configuration
* file. Therefore, if we immediately try to restart the server process, it
* may not be able to acquire the lock on the cluster configuration file and
* fail to start. We explicitly releases the lock on the cluster configuration
* file on shutdown, rather than relying on the OS to release the lock, which
* is a cleaner and safer way to release acquired resources. */
if (server.cluster_config_file_lock_fd != -1) {
flock(server.cluster_config_file_lock_fd, LOCK_UN | LOCK_NB);
}
#endif /* __sun */
}
/* Reset a node performing a soft or hard reset:
*
* 1) All other nodes are forgotten.
@@ -2277,6 +2297,23 @@ void clusterProcessGossipSection(clusterMsg *hdr, clusterLink *link) {
node->tls_port = msg_tls_port;
node->cport = ntohs(g->cport);
node->flags &= ~CLUSTER_NODE_NOADDR;
serverLog(LL_NOTICE, "Address updated for node %.40s (%s), now %s:%d", node->name, node->human_nodename,
node->ip, getNodeDefaultClientPort(node));
/* Check if this is our primary and we have to change the
* replication target as well.
*
* This is needed in case the check in nodeUpdateAddressIfNeeded
* failed due to a race condition. For example, if the replica just
* received a packet from another node that contains new address
* about the primary, we will update primary node address in here,
* when the replica receive the packet from the primary, the check
* in nodeUpdateAddressIfNeeded will fail since the address has been
* updated correctly, and we will not have the opportunity to call
* replicationSetPrimary and update the primary host. */
if (nodeIsReplica(myself) && myself->replicaof == node)
replicationSetPrimary(node->ip, getNodeDefaultReplicationPort(node), 0);
}
} else if (!node) {
/* If it's not in NOADDR state and we don't have it, we
@@ -2370,7 +2407,7 @@ int nodeUpdateAddressIfNeeded(clusterNode *node, clusterLink *link, clusterMsg *
/* Check if this is our primary and we have to change the
* replication target as well. */
if (nodeIsReplica(myself) && myself->replicaof == node)
replicationSetPrimary(node->ip, getNodeDefaultReplicationPort(node));
replicationSetPrimary(node->ip, getNodeDefaultReplicationPort(node), 0);
return 1;
}
@@ -2432,6 +2469,9 @@ void clusterUpdateSlotsConfigWith(clusterNode *sender, uint64_t senderConfigEpoc
return;
}
/* Sender and myself in the same shard? */
int are_in_same_shard = areInSameShard(sender, myself);
for (j = 0; j < CLUSTER_SLOTS; j++) {
if (bitmapTestBit(slots, j)) {
sender_slots++;
@@ -2474,7 +2514,7 @@ void clusterUpdateSlotsConfigWith(clusterNode *sender, uint64_t senderConfigEpoc
* the same shard and we should retain the migrating_slots_to state
* for the slot in question */
if (server.cluster->migrating_slots_to[j] != NULL) {
if (!areInSameShard(sender, myself)) {
if (!are_in_same_shard) {
serverLog(LL_NOTICE, "Slot %d is no longer being migrated to node %.40s (%s) in shard %.40s.",
j, server.cluster->migrating_slots_to[j]->name,
server.cluster->migrating_slots_to[j]->human_nodename,
@@ -2595,7 +2635,7 @@ void clusterUpdateSlotsConfigWith(clusterNode *sender, uint64_t senderConfigEpoc
* the new primary if my current config epoch is lower than the
* sender's. */
if (!new_primary && myself->replicaof != sender && sender_slots == 0 && myself->numslots == 0 &&
nodeEpoch(myself) < senderConfigEpoch && areInSameShard(sender, myself)) {
nodeEpoch(myself) < senderConfigEpoch && are_in_same_shard) {
new_primary = sender;
}
@@ -2619,16 +2659,18 @@ void clusterUpdateSlotsConfigWith(clusterNode *sender, uint64_t senderConfigEpoc
* sender. In this case we don't reconfigure ourselves as a replica
* of the sender. */
if (new_primary && cur_primary->numslots == 0) {
if (server.cluster_allow_replica_migration || areInSameShard(sender, myself)) {
if (server.cluster_allow_replica_migration || are_in_same_shard) {
serverLog(LL_NOTICE,
"Configuration change detected. Reconfiguring myself "
"as a replica of node %.40s (%s) in shard %.40s",
sender->name, sender->human_nodename, sender->shard_id);
/* Don't clear the migrating/importing states if this is a replica that
* just gets promoted to the new primary in the shard. */
clusterSetPrimary(sender, !areInSameShard(sender, myself));
* just gets promoted to the new primary in the shard.
*
* If the sender and myself are in the same shard, try psync. */
clusterSetPrimary(sender, !are_in_same_shard, !are_in_same_shard);
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG | CLUSTER_TODO_UPDATE_STATE | CLUSTER_TODO_FSYNC_CONFIG);
} else if ((sender_slots >= migrated_our_slots) && !areInSameShard(sender, myself)) {
} else if (nodeIsPrimary(myself) && (sender_slots >= migrated_our_slots) && !are_in_same_shard) {
/* When all our slots are lost to the sender and the sender belongs to
* a different shard, this is likely due to a client triggered slot
* migration. Don't reconfigure this node to migrate to the new shard
@@ -3136,7 +3178,10 @@ int clusterProcessPacket(clusterLink *link) {
/* Add this node if it is new for us and the msg type is MEET.
* In this stage we don't try to add the node with the right
* flags, replicaof pointer, and so forth, as this details will be
* resolved when we'll receive PONGs from the node. */
* resolved when we'll receive PONGs from the node. The exception
* to this is the flag that indicates extensions are supported, as
* we want to send extensions right away in the return PONG in order
* to reduce the amount of time needed to stabilize the shard ID. */
if (!sender && type == CLUSTERMSG_TYPE_MEET) {
clusterNode *node;
@@ -3144,6 +3189,10 @@ int clusterProcessPacket(clusterLink *link) {
serverAssert(nodeIp2String(node->ip, link, hdr->myip) == C_OK);
getClientPortFromClusterMsg(hdr, &node->tls_port, &node->tcp_port);
node->cport = ntohs(hdr->cport);
if (hdr->mflags[0] & CLUSTERMSG_FLAG0_EXT_DATA) {
node->flags |= CLUSTER_NODE_EXTENSIONS_SUPPORTED;
}
setClusterNodeToInboundClusterLink(node, link);
clusterAddNode(node);
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG);
}
@@ -3309,10 +3358,43 @@ int clusterProcessPacket(clusterLink *link) {
if (sender->replicaof) clusterNodeRemoveReplica(sender->replicaof, sender);
serverLog(LL_NOTICE, "Node %.40s (%s) is now a replica of node %.40s (%s) in shard %.40s",
sender->name, sender->human_nodename, sender_claimed_primary->name,
sender_claimed_primary->human_nodename, sender->shard_id);
sender_claimed_primary->human_nodename, sender_claimed_primary->shard_id);
clusterNodeAddReplica(sender_claimed_primary, sender);
sender->replicaof = sender_claimed_primary;
/* The chain reduction logic requires correctly establishing the replication relationship.
* A key decision when designating a new primary for 'myself' is determining whether
* 'myself' and the new primary belong to the same shard, which would imply shared
* replication history and allow a safe partial synchronization (psync).
*
* This decision hinges on the shard_id, a per-node property that helps verify if the
* two nodes share the same replication history. It's critical not to update myself's
* shard_id prematurely during this process. Doing so could incorrectly associate
* 'myself' with the sender's shard_id, leading the subsequent clusterSetPrimary call
* to falsely assume that 'myself' and the new primary have been in the same shard.
* This mistake could result in data loss by incorrectly permitting a psync.
*
* Therefore, it's essential to delay any shard_id updates until after the replication
* relationship has been properly established and verified. */
if (myself->replicaof && myself->replicaof->replicaof && myself->replicaof->replicaof != myself) {
/* Safeguard against sub-replicas.
*
* A replica's primary can turn itself into a replica if its last slot
* is removed. If no other node takes over the slot, there is nothing
* else to trigger replica migration. In this case, they are not in the
* same shard, so a full sync is required.
*
* Or a replica's primary can turn itself into a replica of its other
* replica during a failover. In this case, they are in the same shard,
* so we can try a psync. */
serverLog(LL_NOTICE, "I'm a sub-replica! Reconfiguring myself as a replica of %.40s from %.40s",
myself->replicaof->replicaof->name, myself->replicaof->name);
clusterSetPrimary(myself->replicaof->replicaof, 1,
!areInSameShard(myself->replicaof->replicaof, myself));
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG | CLUSTER_TODO_UPDATE_STATE |
CLUSTER_TODO_FSYNC_CONFIG);
}
/* Update the shard_id when a replica is connected to its
* primary in the very first time. */
updateShardId(sender, sender_claimed_primary->shard_id);
@@ -3380,18 +3462,6 @@ int clusterProcessPacket(clusterLink *link) {
}
}
/* Explicitly check for a replication loop before attempting the replication
* chain folding logic. */
if (myself->replicaof && myself->replicaof->replicaof && myself->replicaof->replicaof != myself) {
/* Safeguard against sub-replicas. A replica's primary can turn itself
* into a replica if its last slot is removed. If no other node takes
* over the slot, there is nothing else to trigger replica migration. */
serverLog(LL_NOTICE, "I'm a sub-replica! Reconfiguring myself as a replica of %.40s from %.40s",
myself->replicaof->replicaof->name, myself->replicaof->name);
clusterSetPrimary(myself->replicaof->replicaof, 1);
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG | CLUSTER_TODO_UPDATE_STATE | CLUSTER_TODO_FSYNC_CONFIG);
}
/* If our config epoch collides with the sender's try to fix
* the problem. */
if (sender && nodeIsPrimary(myself) && nodeIsPrimary(sender) &&
@@ -4250,14 +4320,14 @@ void clusterSendFailoverAuthIfNeeded(clusterNode *node, clusterMsg *request) {
* with CLUSTERMSG_FLAG0_FORCEACK (manual failover). */
if (clusterNodeIsPrimary(node) || primary == NULL || (!nodeFailed(primary) && !force_ack)) {
if (clusterNodeIsPrimary(node)) {
serverLog(LL_WARNING, "Failover auth denied to %.40s (%s): it is a primary node", node->name,
node->human_nodename);
serverLog(LL_WARNING, "Failover auth denied to %.40s (%s) for epoch %llu: it is a primary node", node->name,
node->human_nodename, (unsigned long long)requestCurrentEpoch);
} else if (primary == NULL) {
serverLog(LL_WARNING, "Failover auth denied to %.40s (%s): I don't know its primary", node->name,
node->human_nodename);
serverLog(LL_WARNING, "Failover auth denied to %.40s (%s) for epoch %llu: I don't know its primary",
node->name, node->human_nodename, (unsigned long long)requestCurrentEpoch);
} else if (!nodeFailed(primary)) {
serverLog(LL_WARNING, "Failover auth denied to %.40s (%s): its primary is up", node->name,
node->human_nodename);
serverLog(LL_WARNING, "Failover auth denied to %.40s (%s) for epoch %llu: its primary is up", node->name,
node->human_nodename, (unsigned long long)requestCurrentEpoch);
}
return;
}
@@ -4356,9 +4426,6 @@ int clusterGetReplicaRank(void) {
* 2) Also, the log is emitted again if the primary is still down and
* the reason for not failing over is still the same, but more than
* CLUSTER_CANT_FAILOVER_RELOG_PERIOD seconds elapsed.
* 3) Finally, the function only logs if the replica is down for more than
* five seconds + NODE_TIMEOUT. This way nothing is logged when a
* failover starts in a reasonable time.
*
* The function is called with the reason why the replica can't failover
* which is one of the integer macros CLUSTER_CANT_FAILOVER_*.
@@ -4367,7 +4434,6 @@ int clusterGetReplicaRank(void) {
void clusterLogCantFailover(int reason) {
char *msg;
static time_t lastlog_time = 0;
mstime_t nolog_fail_time = server.cluster_node_timeout + 5000;
/* Don't log if we have the same reason for some time. */
if (reason == server.cluster->cant_failover_reason &&
@@ -4376,13 +4442,6 @@ void clusterLogCantFailover(int reason) {
server.cluster->cant_failover_reason = reason;
/* We also don't emit any log if the primary failed no long ago, the
* goal of this function is to log replicas in a stalled condition for
* a long time. */
if (myself->replicaof && nodeFailed(myself->replicaof) &&
(mstime() - myself->replicaof->fail_time) < nolog_fail_time)
return;
switch (reason) {
case CLUSTER_CANT_FAILOVER_DATA_AGE:
msg = "Disconnected from primary for longer than allowed. "
@@ -4703,7 +4762,9 @@ void clusterHandleReplicaMigration(int max_replicas) {
!(server.cluster_module_flags & CLUSTER_MODULE_FLAG_NO_FAILOVER)) {
serverLog(LL_NOTICE, "Migrating to orphaned primary %.40s (%s) in shard %.40s", target->name,
target->human_nodename, target->shard_id);
clusterSetPrimary(target, 1);
/* We are migrating to a different shard that has a completely different
* replication history, so a full sync is required. */
clusterSetPrimary(target, 1, 1);
}
}
@@ -5016,7 +5077,7 @@ void clusterCron(void) {
* enable it if we know the address of our primary and it appears to
* be up. */
if (nodeIsReplica(myself) && server.primary_host == NULL && myself->replicaof && nodeHasAddr(myself->replicaof)) {
replicationSetPrimary(myself->replicaof->ip, getNodeDefaultReplicationPort(myself->replicaof));
replicationSetPrimary(myself->replicaof->ip, getNodeDefaultReplicationPort(myself->replicaof), 0);
}
/* Abort a manual failover if the timeout is reached. */
@@ -5409,7 +5470,7 @@ static inline void removeAllNotOwnedShardChannelSubscriptions(void) {
/* Set the specified node 'n' as primary for this node.
* If this node is currently a primary, it is turned into a replica. */
void clusterSetPrimary(clusterNode *n, int closeSlots) {
static void clusterSetPrimary(clusterNode *n, int closeSlots, int full_sync_required) {
serverAssert(n != myself);
serverAssert(myself->numslots == 0);
@@ -5423,7 +5484,7 @@ void clusterSetPrimary(clusterNode *n, int closeSlots) {
myself->replicaof = n;
updateShardId(myself, n->shard_id);
clusterNodeAddReplica(n, myself);
replicationSetPrimary(n->ip, getNodeDefaultReplicationPort(n));
replicationSetPrimary(n->ip, getNodeDefaultReplicationPort(n), full_sync_required);
removeAllNotOwnedShardChannelSubscriptions();
resetManualFailover();
@@ -6163,6 +6224,9 @@ int clusterParseSetSlotCommand(client *c, int *slot_out, clusterNode **node_out,
return 0;
}
/* If 'myself' is a replica, 'c' must be the primary client. */
serverAssert(!nodeIsReplica(myself) || c == server.primary);
if ((slot = getSlotOrReply(c, c->argv[2])) == -1) return 0;
if (!strcasecmp(c->argv[3]->ptr, "migrating") && c->argc >= 5) {
@@ -6270,7 +6334,7 @@ void clusterCommandSetSlot(client *c) {
* To mitigate this issue, the following order needs to be enforced for slot
* migration finalization such that the replicas finalize the slot ownership
* before the primary:
. *
*
* 1. Client C issues SETSLOT n NODE B against node B.
* 2. Primary B replicates `SETSLOT n NODE B` to all of its replicas (e.g., B', B'').
* 3. Upon replication completion, primary B executes `SETSLOT n NODE B` and
@@ -6290,17 +6354,25 @@ void clusterCommandSetSlot(client *c) {
* non-replicated behavior.*/
listIter li;
listNode *ln;
int legacy_replica_found = 0;
int num_eligible_replicas = 0;
listRewind(server.replicas, &li);
while ((ln = listNext(&li))) {
client *r = ln->value;
if (r->replica_version < 0x702ff /* 7.2.255 */) {
legacy_replica_found++;
break;
/* We think that when the command comes in, the primary only needs to
* wait for the online replicas. The admin can easily check if there
* are replicas that are down for an extended period of time. If they
* decide to move forward anyways, we should not block it. If a replica
* failed right before the replication and was not included in the
* replication, it would also unlikely win the election.
*
* And 0x702ff is 7.2.255, we only support new versions in this case. */
if (r->repl_state == REPLICA_STATE_ONLINE && r->replica_version > 0x702ff) {
num_eligible_replicas++;
}
}
if (!legacy_replica_found) {
if (num_eligible_replicas != 0) {
forceCommandPropagation(c, PROPAGATE_REPL);
/* We are a primary and this is the first time we see this `SETSLOT`
* command. Force-replicate the command to all of our replicas
@@ -6310,7 +6382,7 @@ void clusterCommandSetSlot(client *c) {
* 2. The repl offset target is set to the primary's current repl offset + 1.
* There is no concern of partial replication because replicas always
* ack the repl offset at the command boundary. */
blockClientForReplicaAck(c, timeout_ms, server.primary_repl_offset + 1, myself->num_replicas, 0);
blockClientForReplicaAck(c, timeout_ms, server.primary_repl_offset + 1, num_eligible_replicas, 0);
/* Mark client as pending command for execution after replication to replicas. */
c->flag.pending_command = 1;
replicationRequestAckFromReplicas();
@@ -6319,7 +6391,7 @@ void clusterCommandSetSlot(client *c) {
}
/* Slot states have been updated on the compatible replicas (if any).
* Now exuecte the command on the primary. */
* Now execute the command on the primary. */
if (!strcasecmp(c->argv[3]->ptr, "migrating")) {
serverLog(LL_NOTICE, "Migrating slot %d to node %.40s (%s)", slot, n->name, n->human_nodename);
server.cluster->migrating_slots_to[slot] = n;
@@ -6343,21 +6415,41 @@ void clusterCommandSetSlot(client *c) {
server.cluster->migrating_slots_to[slot] = NULL;
}
int slot_was_mine = server.cluster->slots[slot] == myself;
clusterNode *my_primary = clusterNodeGetPrimary(myself);
int slot_was_mine = server.cluster->slots[slot] == my_primary;
clusterDelSlot(slot);
clusterAddSlot(n, slot);
int shard_is_empty = my_primary->numslots == 0;
/* If we are a primary left without slots, we should turn into a
* replica of the new primary. */
if (slot_was_mine && n != myself && myself->numslots == 0 && server.cluster_allow_replica_migration) {
/* If replica migration is allowed, check if the primary of this shard
* loses its last slot and the shard becomes empty. In this case, we
* should turn into a replica of the new primary. */
if (server.cluster_allow_replica_migration && slot_was_mine && shard_is_empty) {
serverAssert(n != my_primary);
serverLog(LL_NOTICE,
"Lost my last slot during slot migration. Reconfiguring myself "
"as a replica of %.40s (%s) in shard %.40s",
n->name, n->human_nodename, n->shard_id);
clusterSetPrimary(n, 1);
/* `c` is the primary client if `myself` is a replica, prevent it
* from being freed by clusterSetPrimary. */
if (nodeIsReplica(myself)) protectClient(c);
/* We are migrating to a different shard that has a completely different
* replication history, so a full sync is required. */
clusterSetPrimary(n, 1, 1);
if (nodeIsReplica(myself)) unprotectClient(c);
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG | CLUSTER_TODO_UPDATE_STATE | CLUSTER_TODO_FSYNC_CONFIG);
}
/* If replica migration is not allowed, check if the primary of this shard
* loses its last slot and the shard becomes empty. In this case, we will
* print the exact same log as during the gossip process. */
if (!server.cluster_allow_replica_migration && nodeIsPrimary(myself) && slot_was_mine && shard_is_empty) {
serverAssert(n != my_primary);
serverLog(LL_NOTICE,
"My last slot was migrated to node %.40s (%s) in shard %.40s. I am now an empty primary.",
n->name, n->human_nodename, n->shard_id);
}
/* If this node or this node's primary was importing this slot,
* assigning the slot to itself also clears the importing status. */
if ((n == myself || n == myself->replicaof) && server.cluster->importing_slots_from[slot]) {
@@ -6559,8 +6651,18 @@ int clusterCommandSpecial(client *c) {
return 1;
}
/* Set the primary. */
clusterSetPrimary(n, 1);
/* If `n` is already my primary, there is no need to re-establish the
* replication connection. */
if (myself->replicaof == n) {
addReply(c, shared.ok);
return 1;
}
/* Set the primary.
* If the instance is a primary, it is an empty primary.
* If the instance is a replica, it had a totally different replication history.
* In these both cases, myself as a replica has to do a full sync. */
clusterSetPrimary(n, 1, 1);
clusterBroadcastPong(CLUSTER_BROADCAST_ALL);
clusterDoBeforeSleep(CLUSTER_TODO_UPDATE_STATE | CLUSTER_TODO_SAVE_CONFIG);
addReply(c, shared.ok);
@@ -6604,25 +6706,27 @@ int clusterCommandSpecial(client *c) {
}
resetManualFailover();
server.cluster->mf_end = mstime() + CLUSTER_MF_TIMEOUT;
sds client = catClientInfoString(sdsempty(), c, server.hide_user_data_from_log);
if (takeover) {
/* A takeover does not perform any initial check. It just
* generates a new configuration epoch for this node without
* consensus, claims the primary's slots, and broadcast the new
* configuration. */
serverLog(LL_NOTICE, "Taking over the primary (user request).");
serverLog(LL_NOTICE, "Taking over the primary (user request from '%s').", client);
clusterBumpConfigEpochWithoutConsensus();
clusterFailoverReplaceYourPrimary();
} else if (force) {
/* If this is a forced failover, we don't need to talk with our
* primary to agree about the offset. We just failover taking over
* it without coordination. */
serverLog(LL_NOTICE, "Forced failover user request accepted.");
serverLog(LL_NOTICE, "Forced failover user request accepted (user request from '%s').", client);
server.cluster->mf_can_start = 1;
} else {
serverLog(LL_NOTICE, "Manual failover user request accepted.");
serverLog(LL_NOTICE, "Manual failover user request accepted (user request from '%s').", client);
clusterSendMFStart(myself->replicaof);
}
sdsfree(client);
addReply(c, shared.ok);
} else if (!strcasecmp(c->argv[1]->ptr, "set-config-epoch") && c->argc == 3) {
/* CLUSTER SET-CONFIG-EPOCH <epoch>
+1 -1
View File
@@ -17,7 +17,7 @@
#define CLUSTER_CANT_FAILOVER_WAITING_DELAY 2
#define CLUSTER_CANT_FAILOVER_EXPIRED 3
#define CLUSTER_CANT_FAILOVER_WAITING_VOTES 4
#define CLUSTER_CANT_FAILOVER_RELOG_PERIOD (10) /* seconds. */
#define CLUSTER_CANT_FAILOVER_RELOG_PERIOD 1 /* seconds. */
/* clusterState todo_before_sleep flags. */
#define CLUSTER_TODO_HANDLE_FAILOVER (1 << 0)
+1
View File
@@ -143,6 +143,7 @@ static void clusterSlotStatsUpdateNetworkBytesOutForReplication(long long len) {
if (c == NULL || !canAddNetworkBytesOut(c)) return;
serverAssert(c->slot >= 0 && c->slot < CLUSTER_SLOTS);
serverAssert(nodeIsPrimary(server.cluster->myself));
if (len < 0) serverAssert(server.cluster->slot_stats[c->slot].network_bytes_out >= (uint64_t)llabs(len));
server.cluster->slot_stats[c->slot].network_bytes_out += (len * listLength(server.replicas));
}
+94 -4
View File
@@ -5605,6 +5605,28 @@ struct COMMAND_ARG SENTINEL_GET_MASTER_ADDR_BY_NAME_Args[] = {
{MAKE_ARG("primary-name",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
};
/********** SENTINEL GET_PRIMARY_ADDR_BY_NAME ********************/
#ifndef SKIP_CMD_HISTORY_TABLE
/* SENTINEL GET_PRIMARY_ADDR_BY_NAME history */
#define SENTINEL_GET_PRIMARY_ADDR_BY_NAME_History NULL
#endif
#ifndef SKIP_CMD_TIPS_TABLE
/* SENTINEL GET_PRIMARY_ADDR_BY_NAME tips */
#define SENTINEL_GET_PRIMARY_ADDR_BY_NAME_Tips NULL
#endif
#ifndef SKIP_CMD_KEY_SPECS_TABLE
/* SENTINEL GET_PRIMARY_ADDR_BY_NAME key specs */
#define SENTINEL_GET_PRIMARY_ADDR_BY_NAME_Keyspecs NULL
#endif
/* SENTINEL GET_PRIMARY_ADDR_BY_NAME argument table */
struct COMMAND_ARG SENTINEL_GET_PRIMARY_ADDR_BY_NAME_Args[] = {
{MAKE_ARG("primary-name",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
};
/********** SENTINEL HELP ********************/
#ifndef SKIP_CMD_HISTORY_TABLE
@@ -5669,6 +5691,31 @@ struct COMMAND_ARG SENTINEL_IS_MASTER_DOWN_BY_ADDR_Args[] = {
{MAKE_ARG("runid",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
};
/********** SENTINEL IS_PRIMARY_DOWN_BY_ADDR ********************/
#ifndef SKIP_CMD_HISTORY_TABLE
/* SENTINEL IS_PRIMARY_DOWN_BY_ADDR history */
#define SENTINEL_IS_PRIMARY_DOWN_BY_ADDR_History NULL
#endif
#ifndef SKIP_CMD_TIPS_TABLE
/* SENTINEL IS_PRIMARY_DOWN_BY_ADDR tips */
#define SENTINEL_IS_PRIMARY_DOWN_BY_ADDR_Tips NULL
#endif
#ifndef SKIP_CMD_KEY_SPECS_TABLE
/* SENTINEL IS_PRIMARY_DOWN_BY_ADDR key specs */
#define SENTINEL_IS_PRIMARY_DOWN_BY_ADDR_Keyspecs NULL
#endif
/* SENTINEL IS_PRIMARY_DOWN_BY_ADDR argument table */
struct COMMAND_ARG SENTINEL_IS_PRIMARY_DOWN_BY_ADDR_Args[] = {
{MAKE_ARG("ip",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("port",ARG_TYPE_INTEGER,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("current-epoch",ARG_TYPE_INTEGER,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
{MAKE_ARG("runid",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
};
/********** SENTINEL MASTER ********************/
#ifndef SKIP_CMD_HISTORY_TABLE
@@ -5767,6 +5814,45 @@ struct COMMAND_ARG SENTINEL_MONITOR_Args[] = {
#define SENTINEL_PENDING_SCRIPTS_Keyspecs NULL
#endif
/********** SENTINEL PRIMARIES ********************/
#ifndef SKIP_CMD_HISTORY_TABLE
/* SENTINEL PRIMARIES history */
#define SENTINEL_PRIMARIES_History NULL
#endif
#ifndef SKIP_CMD_TIPS_TABLE
/* SENTINEL PRIMARIES tips */
#define SENTINEL_PRIMARIES_Tips NULL
#endif
#ifndef SKIP_CMD_KEY_SPECS_TABLE
/* SENTINEL PRIMARIES key specs */
#define SENTINEL_PRIMARIES_Keyspecs NULL
#endif
/********** SENTINEL PRIMARY ********************/
#ifndef SKIP_CMD_HISTORY_TABLE
/* SENTINEL PRIMARY history */
#define SENTINEL_PRIMARY_History NULL
#endif
#ifndef SKIP_CMD_TIPS_TABLE
/* SENTINEL PRIMARY tips */
#define SENTINEL_PRIMARY_Tips NULL
#endif
#ifndef SKIP_CMD_KEY_SPECS_TABLE
/* SENTINEL PRIMARY key specs */
#define SENTINEL_PRIMARY_Keyspecs NULL
#endif
/* SENTINEL PRIMARY argument table */
struct COMMAND_ARG SENTINEL_PRIMARY_Args[] = {
{MAKE_ARG("primary-name",ARG_TYPE_STRING,-1,NULL,NULL,NULL,CMD_ARG_NONE,0,NULL)},
};
/********** SENTINEL REMOVE ********************/
#ifndef SKIP_CMD_HISTORY_TABLE
@@ -5942,15 +6028,19 @@ struct COMMAND_STRUCT SENTINEL_Subcommands[] = {
{MAKE_CMD("debug","Lists or updates the current configurable parameters of Sentinel.","O(N) where N is the number of configurable parameters","7.0.0",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_DEBUG_History,0,SENTINEL_DEBUG_Tips,0,sentinelCommand,-2,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_DEBUG_Keyspecs,0,NULL,1),.args=SENTINEL_DEBUG_Args},
{MAKE_CMD("failover","Forces a Sentinel failover.",NULL,"2.8.4",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_FAILOVER_History,0,SENTINEL_FAILOVER_Tips,0,sentinelCommand,3,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_FAILOVER_Keyspecs,0,NULL,1),.args=SENTINEL_FAILOVER_Args},
{MAKE_CMD("flushconfig","Rewrites the Sentinel configuration file.","O(1)","2.8.4",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_FLUSHCONFIG_History,0,SENTINEL_FLUSHCONFIG_Tips,0,sentinelCommand,2,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_FLUSHCONFIG_Keyspecs,0,NULL,0)},
{MAKE_CMD("get-master-addr-by-name","Returns the port and address of a primary instance.","O(1)","2.8.4",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_GET_MASTER_ADDR_BY_NAME_History,0,SENTINEL_GET_MASTER_ADDR_BY_NAME_Tips,0,sentinelCommand,3,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_GET_MASTER_ADDR_BY_NAME_Keyspecs,0,NULL,1),.args=SENTINEL_GET_MASTER_ADDR_BY_NAME_Args},
{MAKE_CMD("get-master-addr-by-name","Returns the port and address of a primary instance.","O(1)","2.8.4",CMD_DOC_DEPRECATED,"`SENTINEL GET-PRIMARY-ADDR-BY-NAME`","8.0.0","sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_GET_MASTER_ADDR_BY_NAME_History,0,SENTINEL_GET_MASTER_ADDR_BY_NAME_Tips,0,sentinelCommand,3,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_GET_MASTER_ADDR_BY_NAME_Keyspecs,0,NULL,1),.args=SENTINEL_GET_MASTER_ADDR_BY_NAME_Args},
{MAKE_CMD("get-primary-addr-by-name","Returns the port and address of a primary instance.","O(1)","8.0.0",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_GET_PRIMARY_ADDR_BY_NAME_History,0,SENTINEL_GET_PRIMARY_ADDR_BY_NAME_Tips,0,sentinelCommand,3,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_GET_PRIMARY_ADDR_BY_NAME_Keyspecs,0,NULL,1),.args=SENTINEL_GET_PRIMARY_ADDR_BY_NAME_Args},
{MAKE_CMD("help","Returns helpful text about the different subcommands.","O(1)","6.2.0",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_HELP_History,0,SENTINEL_HELP_Tips,0,sentinelCommand,2,CMD_LOADING|CMD_STALE|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_HELP_Keyspecs,0,NULL,0)},
{MAKE_CMD("info-cache","Returns the cached `INFO` replies from the deployment's instances.","O(N) where N is the number of instances","3.2.0",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_INFO_CACHE_History,0,SENTINEL_INFO_CACHE_Tips,0,sentinelCommand,-3,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_INFO_CACHE_Keyspecs,0,NULL,1),.args=SENTINEL_INFO_CACHE_Args},
{MAKE_CMD("is-master-down-by-addr","Determines whether a primary instance is down.","O(1)","2.8.4",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_IS_MASTER_DOWN_BY_ADDR_History,0,SENTINEL_IS_MASTER_DOWN_BY_ADDR_Tips,0,sentinelCommand,6,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_IS_MASTER_DOWN_BY_ADDR_Keyspecs,0,NULL,4),.args=SENTINEL_IS_MASTER_DOWN_BY_ADDR_Args},
{MAKE_CMD("master","Returns the state of a primary instance.","O(1)","2.8.4",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_MASTER_History,0,SENTINEL_MASTER_Tips,0,sentinelCommand,3,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_MASTER_Keyspecs,0,NULL,1),.args=SENTINEL_MASTER_Args},
{MAKE_CMD("masters","Returns a list of monitored primaries.","O(N) where N is the number of primaries","2.8.4",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_MASTERS_History,0,SENTINEL_MASTERS_Tips,0,sentinelCommand,2,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_MASTERS_Keyspecs,0,NULL,0)},
{MAKE_CMD("is-master-down-by-addr","Determines whether a primary instance is down.","O(1)","2.8.4",CMD_DOC_DEPRECATED,"`SENTINEL IS-PRIMARY-DOWN-BY-ADDR`","8.0.0","sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_IS_MASTER_DOWN_BY_ADDR_History,0,SENTINEL_IS_MASTER_DOWN_BY_ADDR_Tips,0,sentinelCommand,6,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_IS_MASTER_DOWN_BY_ADDR_Keyspecs,0,NULL,4),.args=SENTINEL_IS_MASTER_DOWN_BY_ADDR_Args},
{MAKE_CMD("is-primary-down-by-addr","Determines whether a primary instance is down.","O(1)","8.0.0",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_IS_PRIMARY_DOWN_BY_ADDR_History,0,SENTINEL_IS_PRIMARY_DOWN_BY_ADDR_Tips,0,sentinelCommand,6,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_IS_PRIMARY_DOWN_BY_ADDR_Keyspecs,0,NULL,4),.args=SENTINEL_IS_PRIMARY_DOWN_BY_ADDR_Args},
{MAKE_CMD("master","Returns the state of a primary instance.","O(1)","2.8.4",CMD_DOC_DEPRECATED,"`SENTINEL PRIMARY`","8.0.0","sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_MASTER_History,0,SENTINEL_MASTER_Tips,0,sentinelCommand,3,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_MASTER_Keyspecs,0,NULL,1),.args=SENTINEL_MASTER_Args},
{MAKE_CMD("masters","Returns a list of monitored primaries.","O(N) where N is the number of primaries","2.8.4",CMD_DOC_DEPRECATED,"`SENTINEL PRIMARIES`","8.0.0","sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_MASTERS_History,0,SENTINEL_MASTERS_Tips,0,sentinelCommand,2,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_MASTERS_Keyspecs,0,NULL,0)},
{MAKE_CMD("monitor","Starts monitoring.","O(1)","2.8.4",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_MONITOR_History,0,SENTINEL_MONITOR_Tips,0,sentinelCommand,6,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_MONITOR_Keyspecs,0,NULL,4),.args=SENTINEL_MONITOR_Args},
{MAKE_CMD("myid","Returns the Sentinel instance ID.","O(1)","6.2.0",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_MYID_History,0,SENTINEL_MYID_Tips,0,sentinelCommand,2,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_MYID_Keyspecs,0,NULL,0)},
{MAKE_CMD("pending-scripts","Returns information about pending scripts for Sentinel.",NULL,"2.8.4",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_PENDING_SCRIPTS_History,0,SENTINEL_PENDING_SCRIPTS_Tips,0,sentinelCommand,2,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_PENDING_SCRIPTS_Keyspecs,0,NULL,0)},
{MAKE_CMD("primaries","Returns a list of monitored primaries.","O(N) where N is the number of primaries","8.0.0",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_PRIMARIES_History,0,SENTINEL_PRIMARIES_Tips,0,sentinelCommand,2,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_PRIMARIES_Keyspecs,0,NULL,0)},
{MAKE_CMD("primary","Returns the state of a primary instance.","O(1)","8.0.0",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_PRIMARY_History,0,SENTINEL_PRIMARY_Tips,0,sentinelCommand,3,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_PRIMARY_Keyspecs,0,NULL,1),.args=SENTINEL_PRIMARY_Args},
{MAKE_CMD("remove","Stops monitoring.","O(1)","2.8.4",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_REMOVE_History,0,SENTINEL_REMOVE_Tips,0,sentinelCommand,3,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_REMOVE_Keyspecs,0,NULL,1),.args=SENTINEL_REMOVE_Args},
{MAKE_CMD("replicas","Returns a list of the monitored replicas.","O(N) where N is the number of replicas","5.0.0",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_REPLICAS_History,0,SENTINEL_REPLICAS_Tips,0,sentinelCommand,3,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_REPLICAS_Keyspecs,0,NULL,1),.args=SENTINEL_REPLICAS_Args},
{MAKE_CMD("reset","Resets primaries by name matching a pattern.","O(N) where N is the number of monitored primaries","2.8.4",CMD_DOC_NONE,NULL,NULL,"sentinel",COMMAND_GROUP_SENTINEL,SENTINEL_RESET_History,0,SENTINEL_RESET_Tips,0,sentinelCommand,3,CMD_ADMIN|CMD_SENTINEL|CMD_ONLY_SENTINEL,0,SENTINEL_RESET_Keyspecs,0,NULL,1),.args=SENTINEL_RESET_Args},
@@ -7,6 +7,12 @@
"arity": 3,
"container": "SENTINEL",
"function": "sentinelCommand",
"deprecated_since": "8.0.0",
"replaced_by": "`SENTINEL GET-PRIMARY-ADDR-BY-NAME`",
"doc_flags": [
"DEPRECATED"
],
"command_flags": [
"ADMIN",
"SENTINEL",
@@ -0,0 +1,38 @@
{
"GET-PRIMARY-ADDR-BY-NAME": {
"summary": "Returns the port and address of a primary instance.",
"complexity": "O(1)",
"group": "sentinel",
"since": "8.0.0",
"arity": 3,
"container": "SENTINEL",
"function": "sentinelCommand",
"command_flags": [
"ADMIN",
"SENTINEL",
"ONLY_SENTINEL"
],
"reply_schema": {
"type": "array",
"minItems": 2,
"maxItems": 2,
"items": [
{
"type": "string",
"description": "IP addr or hostname."
},
{
"type": "string",
"description": "Port.",
"pattern": "[0-9]+"
}
]
},
"arguments": [
{
"name": "primary-name",
"type": "string"
}
]
}
}
@@ -7,6 +7,11 @@
"arity": 6,
"container": "SENTINEL",
"function": "sentinelCommand",
"deprecated_since": "8.0.0",
"replaced_by": "`SENTINEL IS-PRIMARY-DOWN-BY-ADDR`",
"doc_flags": [
"DEPRECATED"
],
"command_flags": [
"ADMIN",
"SENTINEL",
@@ -0,0 +1,61 @@
{
"IS-PRIMARY-DOWN-BY-ADDR": {
"summary": "Determines whether a primary instance is down.",
"complexity": "O(1)",
"group": "sentinel",
"since": "8.0.0",
"arity": 6,
"container": "SENTINEL",
"function": "sentinelCommand",
"command_flags": [
"ADMIN",
"SENTINEL",
"ONLY_SENTINEL"
],
"reply_schema": {
"type": "array",
"minItems": 3,
"maxItems": 3,
"items": [
{
"oneOf": [
{
"const": 0,
"description": "Primary is up."
},
{
"const": 1,
"description": "Primary is down."
}
]
},
{
"type": "string",
"description": "Sentinel address."
},
{
"type": "integer",
"description": "Port."
}
]
},
"arguments": [
{
"name": "ip",
"type": "string"
},
{
"name": "port",
"type": "integer"
},
{
"name": "current-epoch",
"type": "integer"
},
{
"name": "runid",
"type": "string"
}
]
}
}
+5
View File
@@ -7,6 +7,11 @@
"arity": 3,
"container": "SENTINEL",
"function": "sentinelCommand",
"deprecated_since": "8.0.0",
"replaced_by": "`SENTINEL PRIMARY`",
"doc_flags": [
"DEPRECATED"
],
"command_flags": [
"ADMIN",
"SENTINEL",
+6 -1
View File
@@ -7,6 +7,11 @@
"arity": 2,
"container": "SENTINEL",
"function": "sentinelCommand",
"deprecated_since": "8.0.0",
"replaced_by": "`SENTINEL PRIMARIES`",
"doc_flags": [
"DEPRECATED"
],
"command_flags": [
"ADMIN",
"SENTINEL",
@@ -14,7 +19,7 @@
],
"reply_schema": {
"type": "array",
"description": "List of monitored primaries, and their state.",
"description": "List of monitored primaries, and their states.",
"items": {
"type": "object",
"additionalProperties": {
+26
View File
@@ -0,0 +1,26 @@
{
"PRIMARIES": {
"summary": "Returns a list of monitored primaries.",
"complexity": "O(N) where N is the number of primaries",
"group": "sentinel",
"since": "8.0.0",
"arity": 2,
"container": "SENTINEL",
"function": "sentinelCommand",
"command_flags": [
"ADMIN",
"SENTINEL",
"ONLY_SENTINEL"
],
"reply_schema": {
"type": "array",
"description": "List of monitored primaries, and their states.",
"items": {
"type": "object",
"additionalProperties": {
"type": "string"
}
}
}
}
}
+29
View File
@@ -0,0 +1,29 @@
{
"PRIMARY": {
"summary": "Returns the state of a primary instance.",
"complexity": "O(1)",
"group": "sentinel",
"since": "8.0.0",
"arity": 3,
"container": "SENTINEL",
"function": "sentinelCommand",
"command_flags": [
"ADMIN",
"SENTINEL",
"ONLY_SENTINEL"
],
"reply_schema": {
"type": "object",
"description": "The state and info of the specified primary.",
"additionalProperties": {
"type": "string"
}
},
"arguments": [
{
"name": "primary-name",
"type": "string"
}
]
}
}
+13 -9
View File
@@ -1,6 +1,6 @@
/* Configuration file parsing and CONFIG GET/SET commands implementation.
*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -3067,14 +3067,14 @@ standardConfig static_configs[] = {
createBoolConfig("stop-writes-on-bgsave-error", NULL, MODIFIABLE_CONFIG, server.stop_writes_on_bgsave_err, 1, NULL, NULL),
createBoolConfig("set-proc-title", NULL, IMMUTABLE_CONFIG, server.set_proc_title, 1, NULL, NULL), /* Should setproctitle be used? */
createBoolConfig("dynamic-hz", NULL, MODIFIABLE_CONFIG, server.dynamic_hz, 1, NULL, NULL), /* Adapt hz to # of clients.*/
createBoolConfig("lazyfree-lazy-eviction", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG, server.lazyfree_lazy_eviction, 0, NULL, NULL),
createBoolConfig("lazyfree-lazy-expire", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG, server.lazyfree_lazy_expire, 0, NULL, NULL),
createBoolConfig("lazyfree-lazy-server-del", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG, server.lazyfree_lazy_server_del, 0, NULL, NULL),
createBoolConfig("lazyfree-lazy-user-del", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG, server.lazyfree_lazy_user_del, 0, NULL, NULL),
createBoolConfig("lazyfree-lazy-user-flush", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG, server.lazyfree_lazy_user_flush, 0, NULL, NULL),
createBoolConfig("lazyfree-lazy-eviction", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG, server.lazyfree_lazy_eviction, 1, NULL, NULL),
createBoolConfig("lazyfree-lazy-expire", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG, server.lazyfree_lazy_expire, 1, NULL, NULL),
createBoolConfig("lazyfree-lazy-server-del", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG, server.lazyfree_lazy_server_del, 1, NULL, NULL),
createBoolConfig("lazyfree-lazy-user-del", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG, server.lazyfree_lazy_user_del, 1, NULL, NULL),
createBoolConfig("lazyfree-lazy-user-flush", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG, server.lazyfree_lazy_user_flush, 1, NULL, NULL),
createBoolConfig("repl-disable-tcp-nodelay", NULL, MODIFIABLE_CONFIG, server.repl_disable_tcp_nodelay, 0, NULL, NULL),
createBoolConfig("repl-diskless-sync", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG, server.repl_diskless_sync, 1, NULL, NULL),
createBoolConfig("dual-channel-replication-enabled", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG | PROTECTED_CONFIG, server.dual_channel_replication, 0, NULL, NULL),
createBoolConfig("dual-channel-replication-enabled", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG, server.dual_channel_replication, 0, NULL, NULL),
createBoolConfig("aof-rewrite-incremental-fsync", NULL, MODIFIABLE_CONFIG, server.aof_rewrite_incremental_fsync, 1, NULL, NULL),
createBoolConfig("no-appendfsync-on-rewrite", NULL, MODIFIABLE_CONFIG, server.aof_no_fsync_on_rewrite, 0, NULL, NULL),
createBoolConfig("cluster-require-full-coverage", NULL, MODIFIABLE_CONFIG, server.cluster_require_full_coverage, 1, NULL, NULL),
@@ -3083,7 +3083,7 @@ standardConfig static_configs[] = {
createBoolConfig("aof-use-rdb-preamble", NULL, MODIFIABLE_CONFIG, server.aof_use_rdb_preamble, 1, NULL, NULL),
createBoolConfig("aof-timestamp-enabled", NULL, MODIFIABLE_CONFIG, server.aof_timestamp_enabled, 0, NULL, NULL),
createBoolConfig("cluster-replica-no-failover", "cluster-slave-no-failover", MODIFIABLE_CONFIG, server.cluster_replica_no_failover, 0, NULL, updateClusterFlags), /* Failover by default. */
createBoolConfig("replica-lazy-flush", "slave-lazy-flush", MODIFIABLE_CONFIG, server.repl_replica_lazy_flush, 0, NULL, NULL),
createBoolConfig("replica-lazy-flush", "slave-lazy-flush", MODIFIABLE_CONFIG, server.repl_replica_lazy_flush, 1, NULL, NULL),
createBoolConfig("replica-serve-stale-data", "slave-serve-stale-data", MODIFIABLE_CONFIG, server.repl_serve_stale_data, 1, NULL, NULL),
createBoolConfig("replica-read-only", "slave-read-only", DEBUG_CONFIG | MODIFIABLE_CONFIG, server.repl_replica_ro, 1, NULL, NULL),
createBoolConfig("replica-ignore-maxmemory", "slave-ignore-maxmemory", MODIFIABLE_CONFIG, server.repl_replica_ignore_maxmemory, 1, NULL, NULL),
@@ -3106,10 +3106,12 @@ standardConfig static_configs[] = {
createBoolConfig("extended-redis-compatibility", NULL, MODIFIABLE_CONFIG, server.extended_redis_compat, 0, NULL, updateExtendedRedisCompat),
createBoolConfig("enable-debug-assert", NULL, IMMUTABLE_CONFIG | HIDDEN_CONFIG, server.enable_debug_assert, 0, NULL, NULL),
createBoolConfig("cluster-slot-stats-enabled", NULL, MODIFIABLE_CONFIG, server.cluster_slot_stats_enabled, 0, NULL, NULL),
createBoolConfig("hide-user-data-from-log", NULL, MODIFIABLE_CONFIG, server.hide_user_data_from_log, 1, NULL, NULL),
/* String Configs */
createStringConfig("aclfile", NULL, IMMUTABLE_CONFIG, ALLOW_EMPTY_STRING, server.acl_filename, "", NULL, NULL),
createStringConfig("unixsocket", NULL, IMMUTABLE_CONFIG, EMPTY_STRING_IS_NULL, server.unixsocket, NULL, NULL, NULL),
createStringConfig("unixsocketgroup", NULL, IMMUTABLE_CONFIG, EMPTY_STRING_IS_NULL, server.unixsocketgroup, NULL, NULL, NULL),
createStringConfig("pidfile", NULL, IMMUTABLE_CONFIG, EMPTY_STRING_IS_NULL, server.pidfile, NULL, NULL, NULL),
createStringConfig("replica-announce-ip", "slave-announce-ip", MODIFIABLE_CONFIG, EMPTY_STRING_IS_NULL, server.replica_announce_ip, NULL, NULL, NULL),
createStringConfig("primaryuser", "masteruser", MODIFIABLE_CONFIG | SENSITIVE_CONFIG, EMPTY_STRING_IS_NULL, server.primary_user, NULL, NULL, NULL),
@@ -3135,6 +3137,7 @@ standardConfig static_configs[] = {
createStringConfig("req-res-logfile", NULL, IMMUTABLE_CONFIG | HIDDEN_CONFIG, EMPTY_STRING_IS_NULL, server.req_res_logfile, NULL, NULL, NULL),
#endif
createStringConfig("locale-collate", NULL, MODIFIABLE_CONFIG, ALLOW_EMPTY_STRING, server.locale_collate, "", NULL, updateLocaleCollate),
createStringConfig("debug-context", NULL, MODIFIABLE_CONFIG | DEBUG_CONFIG | HIDDEN_CONFIG, ALLOW_EMPTY_STRING, server.debug_context, "", NULL, NULL),
/* SDS Configs */
createSDSConfig("primaryauth", "masterauth", MODIFIABLE_CONFIG | SENSITIVE_CONFIG, EMPTY_STRING_IS_NULL, server.primary_auth, NULL, NULL, NULL),
@@ -3164,6 +3167,7 @@ standardConfig static_configs[] = {
createIntConfig("port", NULL, MODIFIABLE_CONFIG, 0, 65535, server.port, 6379, INTEGER_CONFIG, NULL, updatePort), /* TCP port. */
createIntConfig("io-threads", NULL, DEBUG_CONFIG | IMMUTABLE_CONFIG, 1, 128, server.io_threads_num, 1, INTEGER_CONFIG, NULL, NULL), /* Single threaded by default */
createIntConfig("events-per-io-thread", NULL, MODIFIABLE_CONFIG, 0, INT_MAX, server.events_per_io_thread, 2, INTEGER_CONFIG, NULL, NULL),
createIntConfig("prefetch-batch-max-size", NULL, MODIFIABLE_CONFIG, 0, 128, server.prefetch_batch_max_size, 16, INTEGER_CONFIG, NULL, NULL),
createIntConfig("auto-aof-rewrite-percentage", NULL, MODIFIABLE_CONFIG, 0, INT_MAX, server.aof_rewrite_perc, 100, INTEGER_CONFIG, NULL, NULL),
createIntConfig("cluster-replica-validity-factor", "cluster-slave-validity-factor", MODIFIABLE_CONFIG, 0, INT_MAX, server.cluster_replica_validity_factor, 10, INTEGER_CONFIG, NULL, NULL), /* replica max data age factor. */
createIntConfig("list-max-listpack-size", "list-max-ziplist-size", MODIFIABLE_CONFIG, INT_MIN, INT_MAX, server.list_max_listpack_size, -2, INTEGER_CONFIG, NULL, NULL),
@@ -3223,7 +3227,7 @@ standardConfig static_configs[] = {
createLongLongConfig("latency-monitor-threshold", NULL, MODIFIABLE_CONFIG, 0, LLONG_MAX, server.latency_monitor_threshold, 0, INTEGER_CONFIG, NULL, NULL),
createLongLongConfig("proto-max-bulk-len", NULL, DEBUG_CONFIG | MODIFIABLE_CONFIG, 1024 * 1024, LONG_MAX, server.proto_max_bulk_len, 512ll * 1024 * 1024, MEMORY_CONFIG, NULL, NULL), /* Bulk request max size */
createLongLongConfig("stream-node-max-entries", NULL, MODIFIABLE_CONFIG, 0, LLONG_MAX, server.stream_node_max_entries, 100, INTEGER_CONFIG, NULL, NULL),
createLongLongConfig("repl-backlog-size", NULL, MODIFIABLE_CONFIG, 1, LLONG_MAX, server.repl_backlog_size, 1024 * 1024, MEMORY_CONFIG, NULL, updateReplBacklogSize), /* Default: 1mb */
createLongLongConfig("repl-backlog-size", NULL, MODIFIABLE_CONFIG, 1, LLONG_MAX, server.repl_backlog_size, 10 * 1024 * 1024, MEMORY_CONFIG, NULL, updateReplBacklogSize), /* Default: 10mb */
/* Unsigned Long Long configs */
createULongLongConfig("maxmemory", NULL, MODIFIABLE_CONFIG, 0, ULLONG_MAX, server.maxmemory, 0, MEMORY_CONFIG, NULL, updateMaxmemory),
+17 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -348,4 +348,20 @@ void setcpuaffinity(const char *cpulist);
#endif
#endif
/* Check for GCC version >= 4.9 */
#if defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 9))
#define HAS_BUILTIN_PREFETCH 1
/* Check for Clang version >= 3.6 */
#elif defined(__clang__) && (__clang_major__ > 3 || (__clang_major__ == 3 && __clang_minor__ >= 6))
#define HAS_BUILTIN_PREFETCH 1
#else
#define HAS_BUILTIN_PREFETCH 0
#endif
#if HAS_BUILTIN_PREFETCH
#define valkey_prefetch(addr) __builtin_prefetch(addr)
#else
#define valkey_prefetch(addr) ((void)(addr))
#endif
#endif
+3 -2
View File
@@ -1,6 +1,6 @@
/*
* Copyright (c) 2019, Redis Labs
* Copyright (c) 2019, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -144,7 +144,8 @@ struct connListener {
int bindaddr_count;
int port;
ConnectionType *ct;
void *priv; /* used by connection type specified data */
void *priv1; /* used by connection type specified data */
void *priv2; /* used by connection type specified data */
};
/* The connection module does not deal with listening and accepting sockets,
+1 -1
View File
@@ -1,6 +1,6 @@
/*
* Copyright (c) 2019, Redis Labs
* Copyright (c) 2019, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -2,7 +2,7 @@
/*
* Copyright 2001-2010 Georges Menie (www.menie.org)
* Copyright 2010-2012 Salvatore Sanfilippo (adapted to Redis coding style)
* Copyright 2010-2012 Redis Ltd. (adapted to Redis coding style)
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+49 -34
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -55,6 +55,7 @@ typedef enum {
keyStatus expireIfNeeded(serverDb *db, robj *key, int flags);
int keyIsExpired(serverDb *db, robj *key);
static void dbSetValue(serverDb *db, robj *key, robj *val, int overwrite, dictEntry *de);
static int getKVStoreIndexForKey(sds key);
/* Update LFU when an object is accessed.
* Firstly, decrement the counter if the decrement time is reached.
@@ -125,7 +126,7 @@ robj *lookupKey(serverDb *db, robj *key, int flags) {
if (!hasActiveChildProcess() && !(flags & LOOKUP_NOTOUCH)) {
if (!canUseSharedObject() && val->refcount == OBJ_SHARED_REFCOUNT) {
val = dupStringObject(val);
kvstoreDictSetVal(db->keys, getKeySlot(key->ptr), de, val);
kvstoreDictSetVal(db->keys, getKVStoreIndexForKey(key->ptr), de, val);
}
if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) {
updateLFU(val);
@@ -202,15 +203,15 @@ robj *lookupKeyWriteOrReply(client *c, robj *key, robj *reply) {
* if the key already exists, otherwise, it can fall back to dbOverwrite. */
static void dbAddInternal(serverDb *db, robj *key, robj *val, int update_if_existing) {
dictEntry *existing;
int slot = getKeySlot(key->ptr);
dictEntry *de = kvstoreDictAddRaw(db->keys, slot, key->ptr, &existing);
int dict_index = getKVStoreIndexForKey(key->ptr);
dictEntry *de = kvstoreDictAddRaw(db->keys, dict_index, key->ptr, &existing);
if (update_if_existing && existing) {
dbSetValue(db, key, val, 1, existing);
return;
}
serverAssertWithInfo(NULL, key, de != NULL);
initObjectLRUOrLFU(val);
kvstoreDictSetVal(db->keys, slot, de, val);
kvstoreDictSetVal(db->keys, dict_index, de, val);
signalKeyAsReady(db, key, val->type);
notifyKeyspaceEvent(NOTIFY_NEW, "new", key, db->id);
}
@@ -219,32 +220,33 @@ void dbAdd(serverDb *db, robj *key, robj *val) {
dbAddInternal(db, key, val, 0);
}
/* Returns key's hash slot when cluster mode is enabled, or 0 when disabled.
* The only difference between this function and getKeySlot, is that it's not using cached key slot from the
* current_client and always calculates CRC hash. This is useful when slot needs to be calculated for a key that user
* didn't request for, such as in case of eviction. */
int calculateKeySlot(sds key) {
return server.cluster_enabled ? keyHashSlot(key, (int)sdslen(key)) : 0;
/* Returns which dict index should be used with kvstore for a given key. */
static int getKVStoreIndexForKey(sds key) {
return server.cluster_enabled ? getKeySlot(key) : 0;
}
/* Return slot-specific dictionary for key based on key's hash slot when cluster mode is enabled, else 0.*/
/* Returns the cluster hash slot for a given key, trying to use the cached slot that
* stored on the server.current_client first. If there is no cached value, it will compute the hash slot
* and then cache the value.*/
int getKeySlot(sds key) {
serverAssert(server.cluster_enabled);
/* This is performance optimization that uses pre-set slot id from the current command,
* in order to avoid calculation of the key hash.
*
* This optimization is only used when current_client flag `CLIENT_EXECUTING_COMMAND` is set.
* It only gets set during the execution of command under `call` method. Other flows requesting
* the key slot would fallback to calculateKeySlot.
* the key slot would fallback to keyHashSlot.
*
* Modules and scripts executed on the primary may get replicated as multi-execs that operate on multiple slots,
* so we must always recompute the slot for commands coming from the primary.
*/
if (server.current_client && server.current_client->slot >= 0 && server.current_client->flag.executing_command &&
!server.current_client->flag.primary) {
debugServerAssertWithInfo(server.current_client, NULL, calculateKeySlot(key) == server.current_client->slot);
debugServerAssertWithInfo(server.current_client, NULL,
(int)keyHashSlot(key, (int)sdslen(key)) == server.current_client->slot);
return server.current_client->slot;
}
int slot = calculateKeySlot(key);
int slot = keyHashSlot(key, (int)sdslen(key));
/* For the case of replicated commands from primary, getNodeByQuery() never gets called,
* and thus c->slot never gets populated. That said, if this command ends up accessing a key,
* we are able to backfill c->slot here, where the key's hash calculation is made. */
@@ -267,11 +269,11 @@ int getKeySlot(sds key) {
* In this case a copy of `key` is copied in kvstore, the caller must ensure the `key` is properly freed.
*/
int dbAddRDBLoad(serverDb *db, sds key, robj *val) {
int slot = getKeySlot(key);
dictEntry *de = kvstoreDictAddRaw(db->keys, slot, key, NULL);
int dict_index = server.cluster_enabled ? getKeySlot(key) : 0;
dictEntry *de = kvstoreDictAddRaw(db->keys, dict_index, key, NULL);
if (de == NULL) return 0;
initObjectLRUOrLFU(val);
kvstoreDictSetVal(db->keys, slot, de, val);
kvstoreDictSetVal(db->keys, dict_index, de, val);
return 1;
}
@@ -288,8 +290,8 @@ int dbAddRDBLoad(serverDb *db, sds key, robj *val) {
*
* The program is aborted if the key was not already present. */
static void dbSetValue(serverDb *db, robj *key, robj *val, int overwrite, dictEntry *de) {
int slot = getKeySlot(key->ptr);
if (!de) de = kvstoreDictFind(db->keys, slot, key->ptr);
int dict_index = getKVStoreIndexForKey(key->ptr);
if (!de) de = kvstoreDictFind(db->keys, dict_index, key->ptr);
serverAssertWithInfo(NULL, key, de != NULL);
robj *old = dictGetVal(de);
@@ -309,7 +311,7 @@ static void dbSetValue(serverDb *db, robj *key, robj *val, int overwrite, dictEn
/* Because of RM_StringDMA, old may be changed, so we need get old again */
old = dictGetVal(de);
}
kvstoreDictSetVal(db->keys, slot, de, val);
kvstoreDictSetVal(db->keys, dict_index, de, val);
/* For efficiency, let the I/O thread that allocated an object also deallocate it. */
if (tryOffloadFreeObjToIOThreads(old) == C_OK) {
/* OK */
@@ -404,8 +406,8 @@ robj *dbRandomKey(serverDb *db) {
int dbGenericDelete(serverDb *db, robj *key, int async, int flags) {
dictEntry **plink;
int table;
int slot = getKeySlot(key->ptr);
dictEntry *de = kvstoreDictTwoPhaseUnlinkFind(db->keys, slot, key->ptr, &plink, &table);
int dict_index = getKVStoreIndexForKey(key->ptr);
dictEntry *de = kvstoreDictTwoPhaseUnlinkFind(db->keys, dict_index, key->ptr, &plink, &table);
if (de) {
robj *val = dictGetVal(de);
/* RM_StringDMA may call dbUnshareStringValue which may free val, so we
@@ -421,13 +423,13 @@ int dbGenericDelete(serverDb *db, robj *key, int async, int flags) {
if (async) {
/* Because of dbUnshareStringValue, the val in de may change. */
freeObjAsync(key, dictGetVal(de), db->id);
kvstoreDictSetVal(db->keys, slot, de, NULL);
kvstoreDictSetVal(db->keys, dict_index, de, NULL);
}
/* Deleting an entry from the expires dict will not free the sds of
* the key, because it is shared with the main dictionary. */
kvstoreDictDelete(db->expires, slot, key->ptr);
kvstoreDictDelete(db->expires, dict_index, key->ptr);
kvstoreDictTwoPhaseUnlinkFree(db->keys, slot, de, plink, table);
kvstoreDictTwoPhaseUnlinkFree(db->keys, dict_index, de, plink, table);
return 1;
} else {
return 0;
@@ -584,8 +586,8 @@ serverDb *initTempDb(void) {
serverDb *tempDb = zcalloc(sizeof(serverDb) * server.dbnum);
for (int i = 0; i < server.dbnum; i++) {
tempDb[i].id = i;
tempDb[i].keys = kvstoreCreate(&dbDictType, slot_count_bits, flags);
tempDb[i].expires = kvstoreCreate(&dbExpiresDictType, slot_count_bits, flags);
tempDb[i].keys = kvstoreCreate(&kvstoreKeysDictType, slot_count_bits, flags);
tempDb[i].expires = kvstoreCreate(&kvstoreExpiresDictType, slot_count_bits, flags);
}
return tempDb;
@@ -1278,7 +1280,7 @@ void shutdownCommand(client *c) {
}
blockClientShutdown(c);
if (prepareForShutdown(flags) == C_OK) exit(0);
if (prepareForShutdown(c, flags) == C_OK) exit(0);
/* If we're here, then shutdown is ongoing (the client is still blocked) or
* failed (the client has received an error). */
}
@@ -1664,7 +1666,7 @@ void swapdbCommand(client *c) {
*----------------------------------------------------------------------------*/
int removeExpire(serverDb *db, robj *key) {
return kvstoreDictDelete(db->expires, getKeySlot(key->ptr), key->ptr) == DICT_OK;
return kvstoreDictDelete(db->expires, getKVStoreIndexForKey(key->ptr), key->ptr) == DICT_OK;
}
/* Set an expire to the specified key. If the expire is set in the context
@@ -1675,10 +1677,10 @@ void setExpire(client *c, serverDb *db, robj *key, long long when) {
dictEntry *kde, *de, *existing;
/* Reuse the sds from the main dict in the expire dict */
int slot = getKeySlot(key->ptr);
kde = kvstoreDictFind(db->keys, slot, key->ptr);
int dict_index = getKVStoreIndexForKey(key->ptr);
kde = kvstoreDictFind(db->keys, dict_index, key->ptr);
serverAssertWithInfo(NULL, key, kde != NULL);
de = kvstoreDictAddRaw(db->expires, slot, dictGetKey(kde), &existing);
de = kvstoreDictAddRaw(db->expires, dict_index, dictGetKey(kde), &existing);
if (existing) {
dictSetSignedIntegerVal(existing, when);
} else {
@@ -1712,6 +1714,19 @@ void deleteExpiredKeyAndPropagate(serverDb *db, robj *keyobj) {
server.stat_expiredkeys++;
}
/* Delete the specified expired key from overwriting and propagate the DEL or UNLINK. */
void deleteExpiredKeyFromOverwriteAndPropagate(client *c, robj *keyobj) {
int deleted = dbGenericDelete(c->db, keyobj, server.lazyfree_lazy_expire, DB_FLAG_KEY_EXPIRED);
serverAssertWithInfo(c, keyobj, deleted);
server.dirty++;
/* Replicate/AOF this as an explicit DEL or UNLINK. */
robj *aux = server.lazyfree_lazy_expire ? shared.unlink : shared.del;
rewriteClientCommandVector(c, 2, aux, keyobj);
signalModifiedKey(c, c->db, keyobj);
notifyKeyspaceEvent(NOTIFY_GENERIC, "del", keyobj, c->db->id);
}
/* Propagate an implicit key deletion into replicas and the AOF file.
* When a key was deleted in the primary by eviction, expiration or a similar
* mechanism a DEL/UNLINK operation for this key is sent
@@ -1883,7 +1898,7 @@ int dbExpandExpires(serverDb *db, uint64_t db_size, int try_expand) {
}
static dictEntry *dbFindGeneric(kvstore *kvs, void *key) {
return kvstoreDictFind(kvs, getKeySlot(key), key);
return kvstoreDictFind(kvs, server.cluster_enabled ? getKeySlot(key) : 0, key);
}
dictEntry *dbFind(serverDb *db, void *key) {
+53 -21
View File
@@ -1,6 +1,5 @@
/*
* Copyright (c) 2009-2020, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2020, Redis Labs, Inc
* Copyright (c) 2009-2020, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -38,6 +37,7 @@
#include "cluster.h"
#include "threads_mngr.h"
#include "io_threads.h"
#include "sds.h"
#include <arpa/inet.h>
#include <signal.h>
@@ -426,8 +426,10 @@ void debugCommand(client *c) {
"MALLCTL-STR <key> [<val>]",
" Get or set a malloc tuning string.",
#endif
"OBJECT <key>",
"OBJECT <key> [fast]",
" Show low level info about `key` and associated value.",
" Some fields of the default behavior may be time consuming to fetch,",
" and `fast` can be passed to avoid fetching them.",
"DROP-CLUSTER-PACKET-FILTER <packet-type>",
" Drop all packets that match the filtered type. Set to -1 allow all packets.",
"CLOSE-CLUSTER-LINK-ON-PACKET-DROP <0|1>",
@@ -520,7 +522,7 @@ void debugCommand(client *c) {
int flags = !strcasecmp(c->argv[1]->ptr, "restart")
? (RESTART_SERVER_GRACEFULLY | RESTART_SERVER_CONFIG_REWRITE)
: RESTART_SERVER_NONE;
restartServer(flags, delay);
restartServer(c, flags, delay);
addReplyError(c, "failed to restart the server. Check server logs.");
} else if (!strcasecmp(c->argv[1]->ptr, "oom")) {
void *ptr = zmalloc(SIZE_MAX / 2); /* Should trigger an out of memory. */
@@ -604,11 +606,14 @@ void debugCommand(client *c) {
} else if (!strcasecmp(c->argv[1]->ptr, "close-cluster-link-on-packet-drop") && c->argc == 3) {
server.debug_cluster_close_link_on_packet_drop = atoi(c->argv[2]->ptr);
addReply(c, shared.ok);
} else if (!strcasecmp(c->argv[1]->ptr, "object") && c->argc == 3) {
} else if (!strcasecmp(c->argv[1]->ptr, "object") && (c->argc == 3 || c->argc == 4)) {
dictEntry *de;
robj *val;
char *strenc;
int fast = 0;
if (c->argc == 4 && !strcasecmp(c->argv[3]->ptr, "fast")) fast = 1;
if ((de = dbFind(c->db, c->argv[2]->ptr)) == NULL) {
addReplyErrorObject(c, shared.nokeyerr);
return;
@@ -639,22 +644,27 @@ void debugCommand(client *c) {
used = snprintf(nextra, remaining, " ql_compressed:%d", compressed);
nextra += used;
remaining -= used;
/* Add total uncompressed size */
unsigned long sz = 0;
for (quicklistNode *node = ql->head; node; node = node->next) {
sz += node->sz;
if (!fast) {
/* Add total uncompressed size */
unsigned long sz = 0;
for (quicklistNode *node = ql->head; node; node = node->next) {
sz += node->sz;
}
used = snprintf(nextra, remaining, " ql_uncompressed_size:%lu", sz);
nextra += used;
remaining -= used;
}
used = snprintf(nextra, remaining, " ql_uncompressed_size:%lu", sz);
nextra += used;
remaining -= used;
}
addReplyStatusFormat(c,
"Value at:%p refcount:%d "
"encoding:%s serializedlength:%zu "
"lru:%d lru_seconds_idle:%llu%s",
(void *)val, val->refcount, strenc, rdbSavedObjectLen(val, c->argv[2], c->db->id),
val->lru, estimateObjectIdleTime(val) / 1000, extra);
sds s = sdsempty();
s = sdscatprintf(s, "Value at:%p refcount:%d encoding:%s", (void *)val, val->refcount, strenc);
if (!fast) s = sdscatprintf(s, " serializedlength:%zu", rdbSavedObjectLen(val, c->argv[2], c->db->id));
/* Either lru or lfu field could work correctly which depends on server.maxmemory_policy. */
s = sdscatprintf(s, " lru:%d lru_seconds_idle:%llu", val->lru, estimateObjectIdleTime(val) / 1000);
s = sdscatprintf(s, " lfu_freq:%lu lfu_access_time_minutes:%u", LFUDecrAndReturn(val), val->lru >> 8);
s = sdscatprintf(s, "%s", extra);
addReplyStatusLength(c, s, sdslen(s));
sdsfree(s);
} else if (!strcasecmp(c->argv[1]->ptr, "sdslen") && c->argc == 3) {
dictEntry *de;
robj *val;
@@ -673,7 +683,7 @@ void debugCommand(client *c) {
addReplyStatusFormat(c,
"key_sds_len:%lld, key_sds_avail:%lld, key_zmalloc: %lld, "
"val_sds_len:%lld, val_sds_avail:%lld, val_zmalloc: %lld",
(long long)sdslen(key), (long long)sdsavail(key), (long long)sdsZmallocSize(key),
(long long)sdslen(key), (long long)sdsavail(key), (long long)sdsAllocSize(key),
(long long)sdslen(val->ptr), (long long)sdsavail(val->ptr),
(long long)getStringObjectSdsUsedMemory(val));
}
@@ -1031,6 +1041,16 @@ __attribute__((noinline)) void _serverAssert(const char *estr, const char *file,
bugReportEnd(0, 0);
}
/* Checks if the argument at the given index should be redacted from logs. */
int shouldRedactArg(const client *c, int idx) {
serverAssert(idx < c->argc);
/* Don't redact if the config is disabled */
if (!server.hide_user_data_from_log) return 0;
/* first_sensitive_arg_idx value should be changed based on the command type. */
int first_sensitive_arg_idx = 1;
return idx >= first_sensitive_arg_idx;
}
void _serverAssertPrintClientInfo(const client *c) {
int j;
char conninfo[CONN_INFO_LEN];
@@ -1041,6 +1061,10 @@ void _serverAssertPrintClientInfo(const client *c) {
serverLog(LL_WARNING, "client->conn = %s", connGetInfo(c->conn, conninfo, sizeof(conninfo)));
serverLog(LL_WARNING, "client->argc = %d", c->argc);
for (j = 0; j < c->argc; j++) {
if (shouldRedactArg(c, j)) {
serverLog(LL_WARNING, "client->argv[%d]: %zu bytes", j, sdslen((sds)c->argv[j]->ptr));
continue;
}
char buf[128];
char *arg;
@@ -1241,6 +1265,10 @@ static void *getAndSetMcontextEip(ucontext_t *uc, void *eip) {
VALKEY_NO_SANITIZE("address")
void logStackContent(void **sp) {
if (server.hide_user_data_from_log) {
serverLog(LL_NOTICE, "hide-user-data-from-log is on, skip logging stack content to avoid spilling user data.");
return;
}
int i;
for (i = 15; i >= 0; i--) {
unsigned long addr = (unsigned long)sp + i;
@@ -1816,7 +1844,7 @@ void logServerInfo(void) {
}
serverLogRaw(LL_WARNING | LL_RAW, infostring);
serverLogRaw(LL_WARNING | LL_RAW, "\n------ CLIENT LIST OUTPUT ------\n");
clients = getAllClientsInfoString(-1);
clients = getAllClientsInfoString(-1, server.hide_user_data_from_log);
serverLogRaw(LL_WARNING | LL_RAW, clients);
sdsfree(infostring);
sdsfree(clients);
@@ -1851,11 +1879,15 @@ void logCurrentClient(client *cc, const char *title) {
int j;
serverLog(LL_WARNING | LL_RAW, "\n------ %s CLIENT INFO ------\n", title);
client = catClientInfoString(sdsempty(), cc);
client = catClientInfoString(sdsempty(), cc, server.hide_user_data_from_log);
serverLog(LL_WARNING | LL_RAW, "%s\n", client);
sdsfree(client);
serverLog(LL_WARNING | LL_RAW, "argc: '%d'\n", cc->argc);
for (j = 0; j < cc->argc; j++) {
if (shouldRedactArg(cc, j)) {
serverLog(LL_WARNING | LL_RAW, "argv[%d]: %zu bytes\n", j, sdslen((sds)cc->argv[j]->ptr));
continue;
}
robj *decoded;
decoded = getDecodedObject(cc->argv[j]);
sds repr = sdscatrepr(sdsempty(), decoded->ptr, min(sdslen(decoded->ptr), 128));
+1 -1
View File
@@ -2,7 +2,7 @@
*
* -----------------------------------------------------------------------------
*
* Copyright (c) 2016, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2016, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -5,7 +5,7 @@
* We do that by scanning the keyspace and for each pointer we have, we can try to
* ask the allocator if moving it to a new address will help reduce fragmentation.
*
* Copyright (c) 2020, Redis Labs, Inc
* Copyright (c) 2020, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+194 -450
View File
File diff suppressed because it is too large Load Diff
+2 -9
View File
@@ -5,7 +5,7 @@
* tables of power of two in size are used, collisions are handled by
* chaining. See the source code for more information... :)
*
* Copyright (c) 2006-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2006-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -70,10 +70,6 @@ typedef struct dictType {
* computing the length of the key + header when buf is NULL. */
size_t (*embedKey)(unsigned char *buf, size_t buf_len, const void *key, unsigned char *header_size);
/* Data */
void *userdata;
/* Flags */
/* The 'no_value' flag, if set, indicates that values are not used, i.e. the
* dict is a set. When this flag is set, it's not possible to access the
@@ -229,6 +225,7 @@ void dictInitIterator(dictIterator *iter, dict *d);
void dictInitSafeIterator(dictIterator *iter, dict *d);
void dictResetIterator(dictIterator *iter);
dictEntry *dictNext(dictIterator *iter);
dictEntry *dictGetNext(const dictEntry *de);
void dictReleaseIterator(dictIterator *iter);
dictEntry *dictGetRandomKey(dict *d);
dictEntry *dictGetFairRandomKey(dict *d);
@@ -253,8 +250,4 @@ dictStats *dictGetStatsHt(dict *d, int htidx, int full);
void dictCombineStats(dictStats *from, dictStats *into);
void dictFreeStats(dictStats *stats);
#ifdef SERVER_TEST
int dictTest(int argc, char *argv[], int flags);
#endif
#endif /* __DICT_H */
+1 -1
View File
@@ -13,7 +13,7 @@
*
* ----------------------------------------------------------------------------
*
* Copyright (c) 2011-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2011-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+1 -1
View File
@@ -2,7 +2,7 @@
*
* ----------------------------------------------------------------------------
*
* Copyright (c) 2011-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2011-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+5 -3
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -42,6 +42,7 @@
#include "monotonic.h"
#include "resp_parser.h"
#include "script_lua.h"
#include "sds.h"
#include <lua.h>
#include <lauxlib.h>
@@ -259,6 +260,7 @@ void scriptingInit(int setup) {
if (lctx.lua_client == NULL) {
lctx.lua_client = createClient(NULL);
lctx.lua_client->flag.script = 1;
lctx.lua_client->flag.fake = 1;
/* We do not want to allow blocking commands inside Lua */
lctx.lua_client->flag.deny_blocking = 1;
@@ -490,7 +492,7 @@ sds luaCreateFunction(client *c, robj *body, int evalsha) {
l->node = luaScriptsLRUAdd(c, sha, evalsha);
int retval = dictAdd(lctx.lua_scripts, sha, l);
serverAssertWithInfo(c ? c : lctx.lua_client, NULL, retval == DICT_OK);
lctx.lua_scripts_mem += sdsZmallocSize(sha) + getStringObjectSdsUsedMemory(body);
lctx.lua_scripts_mem += sdsAllocSize(sha) + getStringObjectSdsUsedMemory(body);
incrRefCount(body);
return sha;
}
@@ -516,7 +518,7 @@ void luaDeleteFunction(client *c, sds sha) {
/* We only delete `EVAL` scripts, which must exist in the LRU list. */
serverAssert(l->node);
listDelNode(lctx.lua_scripts_lru_list, l->node);
lctx.lua_scripts_mem -= sdsZmallocSize(sha) + getStringObjectSdsUsedMemory(l->body);
lctx.lua_scripts_mem -= sdsAllocSize(sha) + getStringObjectSdsUsedMemory(l->body);
dictFreeUnlinkedEntry(lctx.lua_scripts, de);
}
+1 -1
View File
@@ -2,7 +2,7 @@
*
* ----------------------------------------------------------------------------
*
* Copyright (c) 2009-2016, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2016, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+2 -12
View File
@@ -2,7 +2,7 @@
*
* ----------------------------------------------------------------------------
*
* Copyright (c) 2009-2016, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2016, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@@ -668,17 +668,7 @@ void expireGenericCommand(client *c, long long basetime, int unit) {
}
if (checkAlreadyExpired(when)) {
robj *aux;
int deleted = dbGenericDelete(c->db, key, server.lazyfree_lazy_expire, DB_FLAG_KEY_EXPIRED);
serverAssertWithInfo(c, key, deleted);
server.dirty++;
/* Replicate/AOF this as an explicit DEL or UNLINK. */
aux = server.lazyfree_lazy_expire ? shared.unlink : shared.del;
rewriteClientCommandVector(c, 2, aux, key);
signalModifiedKey(c, c->db, key);
notifyKeyspaceEvent(NOTIFY_GENERIC, "del", key, c->db->id);
deleteExpiredKeyFromOverwriteAndPropagate(c, key);
addReply(c, shared.cone);
return;
} else {
+1 -1
View File
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* Copyright (c) 2009-2012, Redis Ltd.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
+82 -2
View File
@@ -44,9 +44,9 @@
/* Everything below this line is automatically generated by
* generate-fmtargs.py. Do not manually edit. */
#define ARG_N(_1, _2, _3, _4, _5, _6, _7, _8, _9, _10, _11, _12, _13, _14, _15, _16, _17, _18, _19, _20, _21, _22, _23, _24, _25, _26, _27, _28, _29, _30, _31, _32, _33, _34, _35, _36, _37, _38, _39, _40, _41, _42, _43, _44, _45, _46, _47, _48, _49, _50, _51, _52, _53, _54, _55, _56, _57, _58, _59, _60, _61, _62, _63, _64, _65, _66, _67, _68, _69, _70, _71, _72, _73, _74, _75, _76, _77, _78, _79, _80, _81, _82, _83, _84, _85, _86, _87, _88, _89, _90, _91, _92, _93, _94, _95, _96, _97, _98, _99, _100, _101, _102, _103, _104, _105, _106, _107, _108, _109, _110, _111, _112, _113, _114, _115, _116, _117, _118, _119, _120, N, ...) N
#define ARG_N(_1, _2, _3, _4, _5, _6, _7, _8, _9, _10, _11, _12, _13, _14, _15, _16, _17, _18, _19, _20, _21, _22, _23, _24, _25, _26, _27, _28, _29, _30, _31, _32, _33, _34, _35, _36, _37, _38, _39, _40, _41, _42, _43, _44, _45, _46, _47, _48, _49, _50, _51, _52, _53, _54, _55, _56, _57, _58, _59, _60, _61, _62, _63, _64, _65, _66, _67, _68, _69, _70, _71, _72, _73, _74, _75, _76, _77, _78, _79, _80, _81, _82, _83, _84, _85, _86, _87, _88, _89, _90, _91, _92, _93, _94, _95, _96, _97, _98, _99, _100, _101, _102, _103, _104, _105, _106, _107, _108, _109, _110, _111, _112, _113, _114, _115, _116, _117, _118, _119, _120, _121, _122, _123, _124, _125, _126, _127, _128, _129, _130, _131, _132, _133, _134, _135, _136, _137, _138, _139, _140, _141, _142, _143, _144, _145, _146, _147, _148, _149, _150, _151, _152, _153, _154, _155, _156, _157, _158, _159, _160, _161, _162, _163, _164, _165, _166, _167, _168, _169, _170, _171, _172, _173, _174, _175, _176, _177, _178, _179, _180, _181, _182, _183, _184, _185, _186, _187, _188, _189, _190, _191, _192, _193, _194, _195, _196, _197, _198, _199, _200, N, ...) N
#define RSEQ_N() 120, 119, 118, 117, 116, 115, 114, 113, 112, 111, 110, 109, 108, 107, 106, 105, 104, 103, 102, 101, 100, 99, 98, 97, 96, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 79, 78, 77, 76, 75, 74, 73, 72, 71, 70, 69, 68, 67, 66, 65, 64, 63, 62, 61, 60, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49, 48, 47, 46, 45, 44, 43, 42, 41, 40, 39, 38, 37, 36, 35, 34, 33, 32, 31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0
#define RSEQ_N() 200, 199, 198, 197, 196, 195, 194, 193, 192, 191, 190, 189, 188, 187, 186, 185, 184, 183, 182, 181, 180, 179, 178, 177, 176, 175, 174, 173, 172, 171, 170, 169, 168, 167, 166, 165, 164, 163, 162, 161, 160, 159, 158, 157, 156, 155, 154, 153, 152, 151, 150, 149, 148, 147, 146, 145, 144, 143, 142, 141, 140, 139, 138, 137, 136, 135, 134, 133, 132, 131, 130, 129, 128, 127, 126, 125, 124, 123, 122, 121, 120, 119, 118, 117, 116, 115, 114, 113, 112, 111, 110, 109, 108, 107, 106, 105, 104, 103, 102, 101, 100, 99, 98, 97, 96, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 79, 78, 77, 76, 75, 74, 73, 72, 71, 70, 69, 68, 67, 66, 65, 64, 63, 62, 61, 60, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49, 48, 47, 46, 45, 44, 43, 42, 41, 40, 39, 38, 37, 36, 35, 34, 33, 32, 31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0
#define COMPACT_FMT_2(fmt, value) fmt
#define COMPACT_FMT_4(fmt, value, ...) fmt COMPACT_FMT_2(__VA_ARGS__)
@@ -108,6 +108,46 @@
#define COMPACT_FMT_116(fmt, value, ...) fmt COMPACT_FMT_114(__VA_ARGS__)
#define COMPACT_FMT_118(fmt, value, ...) fmt COMPACT_FMT_116(__VA_ARGS__)
#define COMPACT_FMT_120(fmt, value, ...) fmt COMPACT_FMT_118(__VA_ARGS__)
#define COMPACT_FMT_122(fmt, value, ...) fmt COMPACT_FMT_120(__VA_ARGS__)
#define COMPACT_FMT_124(fmt, value, ...) fmt COMPACT_FMT_122(__VA_ARGS__)
#define COMPACT_FMT_126(fmt, value, ...) fmt COMPACT_FMT_124(__VA_ARGS__)
#define COMPACT_FMT_128(fmt, value, ...) fmt COMPACT_FMT_126(__VA_ARGS__)
#define COMPACT_FMT_130(fmt, value, ...) fmt COMPACT_FMT_128(__VA_ARGS__)
#define COMPACT_FMT_132(fmt, value, ...) fmt COMPACT_FMT_130(__VA_ARGS__)
#define COMPACT_FMT_134(fmt, value, ...) fmt COMPACT_FMT_132(__VA_ARGS__)
#define COMPACT_FMT_136(fmt, value, ...) fmt COMPACT_FMT_134(__VA_ARGS__)
#define COMPACT_FMT_138(fmt, value, ...) fmt COMPACT_FMT_136(__VA_ARGS__)
#define COMPACT_FMT_140(fmt, value, ...) fmt COMPACT_FMT_138(__VA_ARGS__)
#define COMPACT_FMT_142(fmt, value, ...) fmt COMPACT_FMT_140(__VA_ARGS__)
#define COMPACT_FMT_144(fmt, value, ...) fmt COMPACT_FMT_142(__VA_ARGS__)
#define COMPACT_FMT_146(fmt, value, ...) fmt COMPACT_FMT_144(__VA_ARGS__)
#define COMPACT_FMT_148(fmt, value, ...) fmt COMPACT_FMT_146(__VA_ARGS__)
#define COMPACT_FMT_150(fmt, value, ...) fmt COMPACT_FMT_148(__VA_ARGS__)
#define COMPACT_FMT_152(fmt, value, ...) fmt COMPACT_FMT_150(__VA_ARGS__)
#define COMPACT_FMT_154(fmt, value, ...) fmt COMPACT_FMT_152(__VA_ARGS__)
#define COMPACT_FMT_156(fmt, value, ...) fmt COMPACT_FMT_154(__VA_ARGS__)
#define COMPACT_FMT_158(fmt, value, ...) fmt COMPACT_FMT_156(__VA_ARGS__)
#define COMPACT_FMT_160(fmt, value, ...) fmt COMPACT_FMT_158(__VA_ARGS__)
#define COMPACT_FMT_162(fmt, value, ...) fmt COMPACT_FMT_160(__VA_ARGS__)
#define COMPACT_FMT_164(fmt, value, ...) fmt COMPACT_FMT_162(__VA_ARGS__)
#define COMPACT_FMT_166(fmt, value, ...) fmt COMPACT_FMT_164(__VA_ARGS__)
#define COMPACT_FMT_168(fmt, value, ...) fmt COMPACT_FMT_166(__VA_ARGS__)
#define COMPACT_FMT_170(fmt, value, ...) fmt COMPACT_FMT_168(__VA_ARGS__)
#define COMPACT_FMT_172(fmt, value, ...) fmt COMPACT_FMT_170(__VA_ARGS__)
#define COMPACT_FMT_174(fmt, value, ...) fmt COMPACT_FMT_172(__VA_ARGS__)
#define COMPACT_FMT_176(fmt, value, ...) fmt COMPACT_FMT_174(__VA_ARGS__)
#define COMPACT_FMT_178(fmt, value, ...) fmt COMPACT_FMT_176(__VA_ARGS__)
#define COMPACT_FMT_180(fmt, value, ...) fmt COMPACT_FMT_178(__VA_ARGS__)
#define COMPACT_FMT_182(fmt, value, ...) fmt COMPACT_FMT_180(__VA_ARGS__)
#define COMPACT_FMT_184(fmt, value, ...) fmt COMPACT_FMT_182(__VA_ARGS__)
#define COMPACT_FMT_186(fmt, value, ...) fmt COMPACT_FMT_184(__VA_ARGS__)
#define COMPACT_FMT_188(fmt, value, ...) fmt COMPACT_FMT_186(__VA_ARGS__)
#define COMPACT_FMT_190(fmt, value, ...) fmt COMPACT_FMT_188(__VA_ARGS__)
#define COMPACT_FMT_192(fmt, value, ...) fmt COMPACT_FMT_190(__VA_ARGS__)
#define COMPACT_FMT_194(fmt, value, ...) fmt COMPACT_FMT_192(__VA_ARGS__)
#define COMPACT_FMT_196(fmt, value, ...) fmt COMPACT_FMT_194(__VA_ARGS__)
#define COMPACT_FMT_198(fmt, value, ...) fmt COMPACT_FMT_196(__VA_ARGS__)
#define COMPACT_FMT_200(fmt, value, ...) fmt COMPACT_FMT_198(__VA_ARGS__)
#define COMPACT_VALUES_2(fmt, value) value
#define COMPACT_VALUES_4(fmt, value, ...) value, COMPACT_VALUES_2(__VA_ARGS__)
@@ -169,5 +209,45 @@
#define COMPACT_VALUES_116(fmt, value, ...) value, COMPACT_VALUES_114(__VA_ARGS__)
#define COMPACT_VALUES_118(fmt, value, ...) value, COMPACT_VALUES_116(__VA_ARGS__)
#define COMPACT_VALUES_120(fmt, value, ...) value, COMPACT_VALUES_118(__VA_ARGS__)
#define COMPACT_VALUES_122(fmt, value, ...) value, COMPACT_VALUES_120(__VA_ARGS__)
#define COMPACT_VALUES_124(fmt, value, ...) value, COMPACT_VALUES_122(__VA_ARGS__)
#define COMPACT_VALUES_126(fmt, value, ...) value, COMPACT_VALUES_124(__VA_ARGS__)
#define COMPACT_VALUES_128(fmt, value, ...) value, COMPACT_VALUES_126(__VA_ARGS__)
#define COMPACT_VALUES_130(fmt, value, ...) value, COMPACT_VALUES_128(__VA_ARGS__)
#define COMPACT_VALUES_132(fmt, value, ...) value, COMPACT_VALUES_130(__VA_ARGS__)
#define COMPACT_VALUES_134(fmt, value, ...) value, COMPACT_VALUES_132(__VA_ARGS__)
#define COMPACT_VALUES_136(fmt, value, ...) value, COMPACT_VALUES_134(__VA_ARGS__)
#define COMPACT_VALUES_138(fmt, value, ...) value, COMPACT_VALUES_136(__VA_ARGS__)
#define COMPACT_VALUES_140(fmt, value, ...) value, COMPACT_VALUES_138(__VA_ARGS__)
#define COMPACT_VALUES_142(fmt, value, ...) value, COMPACT_VALUES_140(__VA_ARGS__)
#define COMPACT_VALUES_144(fmt, value, ...) value, COMPACT_VALUES_142(__VA_ARGS__)
#define COMPACT_VALUES_146(fmt, value, ...) value, COMPACT_VALUES_144(__VA_ARGS__)
#define COMPACT_VALUES_148(fmt, value, ...) value, COMPACT_VALUES_146(__VA_ARGS__)
#define COMPACT_VALUES_150(fmt, value, ...) value, COMPACT_VALUES_148(__VA_ARGS__)
#define COMPACT_VALUES_152(fmt, value, ...) value, COMPACT_VALUES_150(__VA_ARGS__)
#define COMPACT_VALUES_154(fmt, value, ...) value, COMPACT_VALUES_152(__VA_ARGS__)
#define COMPACT_VALUES_156(fmt, value, ...) value, COMPACT_VALUES_154(__VA_ARGS__)
#define COMPACT_VALUES_158(fmt, value, ...) value, COMPACT_VALUES_156(__VA_ARGS__)
#define COMPACT_VALUES_160(fmt, value, ...) value, COMPACT_VALUES_158(__VA_ARGS__)
#define COMPACT_VALUES_162(fmt, value, ...) value, COMPACT_VALUES_160(__VA_ARGS__)
#define COMPACT_VALUES_164(fmt, value, ...) value, COMPACT_VALUES_162(__VA_ARGS__)
#define COMPACT_VALUES_166(fmt, value, ...) value, COMPACT_VALUES_164(__VA_ARGS__)
#define COMPACT_VALUES_168(fmt, value, ...) value, COMPACT_VALUES_166(__VA_ARGS__)
#define COMPACT_VALUES_170(fmt, value, ...) value, COMPACT_VALUES_168(__VA_ARGS__)
#define COMPACT_VALUES_172(fmt, value, ...) value, COMPACT_VALUES_170(__VA_ARGS__)
#define COMPACT_VALUES_174(fmt, value, ...) value, COMPACT_VALUES_172(__VA_ARGS__)
#define COMPACT_VALUES_176(fmt, value, ...) value, COMPACT_VALUES_174(__VA_ARGS__)
#define COMPACT_VALUES_178(fmt, value, ...) value, COMPACT_VALUES_176(__VA_ARGS__)
#define COMPACT_VALUES_180(fmt, value, ...) value, COMPACT_VALUES_178(__VA_ARGS__)
#define COMPACT_VALUES_182(fmt, value, ...) value, COMPACT_VALUES_180(__VA_ARGS__)
#define COMPACT_VALUES_184(fmt, value, ...) value, COMPACT_VALUES_182(__VA_ARGS__)
#define COMPACT_VALUES_186(fmt, value, ...) value, COMPACT_VALUES_184(__VA_ARGS__)
#define COMPACT_VALUES_188(fmt, value, ...) value, COMPACT_VALUES_186(__VA_ARGS__)
#define COMPACT_VALUES_190(fmt, value, ...) value, COMPACT_VALUES_188(__VA_ARGS__)
#define COMPACT_VALUES_192(fmt, value, ...) value, COMPACT_VALUES_190(__VA_ARGS__)
#define COMPACT_VALUES_194(fmt, value, ...) value, COMPACT_VALUES_192(__VA_ARGS__)
#define COMPACT_VALUES_196(fmt, value, ...) value, COMPACT_VALUES_194(__VA_ARGS__)
#define COMPACT_VALUES_198(fmt, value, ...) value, COMPACT_VALUES_196(__VA_ARGS__)
#define COMPACT_VALUES_200(fmt, value, ...) value, COMPACT_VALUES_198(__VA_ARGS__)
#endif

Some files were not shown because too many files have changed in this diff Show More