Non-LZF aware object.c APIs fixed.

This will be a non-op most of the times since the object will be
unshared / decoded, however it is more technically correct to start this
way since the object may be decoded even in the read-only code path.

00-RELEASENOTES

@@ -1,85 +1,16 @@
-Redis 2.6 release notes
+Hello! This file is just a placeholder, since this is the "unstable" branch
+of Redis, the place where all the development happens.
 
-Migrating from 2.4 to 2.6
-=========================
+There is no release notes for this branch, it gets forked into another branch
+every time there is a partial feature freeze in order to eventually create
+a new stable release.
 
-Redis 2.4 is mostly a strict subset of 2.6. However there are a few things
-that you should be aware of:
+Usually "unstable" is stable enough for you to use it in development enviromnets
+however you should never use it in production environments. It is possible
+to download the latest stable release here:
 
-* You can't use .rdb and AOF files generated with 2.6 into a 2.4 instance.
-* 2.6 slaves can be attached to 2.4 masters, but not the contrary, and only
-  for the time needed to perform the version upgrade.
+    http://download.redis.io/releases/redis-stable.tar.gz
 
-There are also a few API differences, that are unlikely to cause problems,
-but it is better to keep them in mind:
+More information is available at http://redis.io
 
-* SORT now will refuse to sort in numerical mode elements that can't be parsed
-  as numbers.
-* EXPIREs now all have millisecond resolution (but this is very unlikely to
-  break code that was not conceived exploting the previous resolution error
-  in some way.)
-* INFO output is a bit different now, and contains empty lines and comments
-  starting with '#'. All the major clients should be already fixed to work
-  with the new INFO format.
-
-Also the following redis.conf and CONFIG GET / SET parameters changed name:
-
-    * hash-max-zipmap-entries, now replaced by hash-max-ziplist-entries
-    * hash-max-zipmap-value, now replaced by hash-max-ziplist-value
-    * glueoutputbuf was now completely removed as it does not make sense
-
----------
-CHANGELOG
----------
-
-What's new in Redis 2.6.0
-=========================
-
-UPGRADE URGENCY: We suggest new users to start with 2.6.0, and old users to
-                 upgrade after some testing of the application with the new
-                 Redis version.
-
-* Server side Lua scripting, see http://redis.io/commands/eval
-* Virtual Memory removed (was deprecated in 2.4)
-* Hardcoded limits about max number of clients removed.
-* AOF low level semantics is generally more sane, and especially when used
-  in slaves.
-* Milliseconds resolution expires, also added new commands with milliseconds
-  precision (PEXPIRE, PTTL, ...).
-* Clients max output buffer soft and hard limits. You can specifiy different
-  limits for different classes of clients (normal,pubsub,slave).
-* AOF is now able to rewrite aggregate data types using variadic commands,
-  often producing an AOF that is faster to save, load, and is smaller in size.
-* Every redis.conf directive is now accepted as a command line option for the
-  redis-server binary, with the same name and number of arguments.
-* Hash table seed randomization for protection against collisions attacks.
-* Performances improved when writing large objects to Redis.
-* Significant parts of the core refactored or rewritten. New internal APIs
-  and core changes allowed to develop Redis Cluster on top of the new code,
-  however for 2.6 all the cluster code was removed, and will be released with
-  Redis 3.0 when it is more complete and stable.
-* Redis ASCII art logo added at startup.
-* Crash report on memory violation or failed asserts improved significantly
-  to make debugging of hard to catch bugs simpler.
-* redis-benchmark improvements: ability to run selected tests,
-  CSV output, faster, better help.
-* redis-cli improvements: --eval for comfortable development of Lua scripts.
-* SHUTDOWN now supports two optional arguments: "SAVE" and "NOSAVE".
-* INFO output split into sections, the command is now able to just show 
-  pecific sections.
-* New statistics about how many time a command was called, and how much
-  execution time it used (INFO commandstats).
-* More predictable SORT behavior in edge cases.
-* INCRBYFLOAT and HINCRBYFLOAT commands.
-
---------------------------------------------------------------------------------
-
-Credits: Where not specified the implementation and design are done by
-Salvatore Sanfilippo and Pieter Noordhuis. Thanks to VMware for making all
-this possible. Also many thanks to all the other contributors and the amazing
-community we have.
-
-See commit messages for more credits.
-
-Cheers,
-Salvatore
+Happy hacking!

README

@@ -130,7 +130,7 @@ it the proper way for a production system, we have a script doing this
 for Ubuntu and Debian systems:
 
     % cd utils
-    % ./install_server
+    % ./install_server.sh
 
 The script will ask you a few questions and will setup everything you need
 to run Redis properly as a background daemon that will start again on

deps/linenoise/linenoise.h

@@ -3,39 +3,44 @@
  *
  * See linenoise.c for more information.
  *
+ * ------------------------------------------------------------------------
+ *
  * Copyright (c) 2010, Salvatore Sanfilippo <antirez at gmail dot com>
  * Copyright (c) 2010, Pieter Noordhuis <pcnoordhuis at gmail dot com>
  *
  * All rights reserved.
- *
+ * 
  * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are met:
+ * modification, are permitted provided that the following conditions are
+ * met:
+ * 
+ *  *  Redistributions of source code must retain the above copyright
+ *     notice, this list of conditions and the following disclaimer.
  *
- *   * Redistributions of source code must retain the above copyright notice,
- *     this list of conditions and the following disclaimer.
- *   * Redistributions in binary form must reproduce the above copyright
+ *  *  Redistributions in binary form must reproduce the above copyright
  *     notice, this list of conditions and the following disclaimer in the
  *     documentation and/or other materials provided with the distribution.
- *   * Neither the name of Redis nor the names of its contributors may be used
- *     to endorse or promote products derived from this software without
- *     specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
- * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
- * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
- * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
- * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
- * POSSIBILITY OF SUCH DAMAGE.
+ * 
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  */
 
 #ifndef __LINENOISE_H
 #define __LINENOISE_H
 
+#ifdef __cplusplus
+extern "C" {
+#endif
+
 typedef struct linenoiseCompletions {
   size_t len;
   char **cvec;
@@ -43,13 +48,19 @@ typedef struct linenoiseCompletions {
 
 typedef void(linenoiseCompletionCallback)(const char *, linenoiseCompletions *);
 void linenoiseSetCompletionCallback(linenoiseCompletionCallback *);
-void linenoiseAddCompletion(linenoiseCompletions *, char *);
+void linenoiseAddCompletion(linenoiseCompletions *, const char *);
 
 char *linenoise(const char *prompt);
 int linenoiseHistoryAdd(const char *line);
 int linenoiseHistorySetMaxLen(int len);
-int linenoiseHistorySave(char *filename);
-int linenoiseHistoryLoad(char *filename);
+int linenoiseHistorySave(const char *filename);
+int linenoiseHistoryLoad(const char *filename);
 void linenoiseClearScreen(void);
+void linenoiseSetMultiLine(int ml);
+void linenoisePrintKeyCodes(void);
+
+#ifdef __cplusplus
+}
+#endif
 
 #endif /* __LINENOISE_H */

redis.conf

@@ -416,15 +416,18 @@ slave-priority 100
 #
 # The default is:
 #
-# maxmemory-policy volatile-lru
+# maxmemory-policy noeviction
 
 # LRU and minimal TTL algorithms are not precise algorithms but approximated
-# algorithms (in order to save memory), so you can select as well the sample
-# size to check. For instance for default Redis will check three keys and
-# pick the one that was used less recently, you can change the sample size
-# using the following configuration directive.
+# algorithms (in order to save memory), so you can tune it for speed or
+# accuracy. For default Redis will check five keys and pick the one that was
+# used less recently, you can change the sample size using the following
+# configuration directive.
 #
-# maxmemory-samples 3
+# The default of 5 produces good enough results. 10 Approximates very closely
+# true LRU but costs a bit more CPU. 3 is very fast but not very accurate.
+#
+# maxmemory-samples 5
 
 ############################## APPEND ONLY MODE ###############################
 
@@ -759,3 +762,7 @@ hz 10
 # big latency spikes.
 aof-rewrite-incremental-fsync yes
 
+# Transparently compress string objects in memory using LZF.
+# For default this is set to no since it may affect performances, however
+# when caching things like HTML fragments this may result into a big win.
+memcompression no

runtest-sentinel

@@ -0,0 +1,14 @@
+#!/bin/sh
+TCL_VERSIONS="8.5 8.6"
+TCLSH=""
+
+for VERSION in $TCL_VERSIONS; do
+	TCL=`which tclsh$VERSION 2>/dev/null` && TCLSH=$TCL
+done
+
+if [ -z $TCLSH ]
+then
+    echo "You need tcl 8.5 or newer in order to run the Redis Sentinel test"
+    exit 1
+fi
+$TCLSH tests/sentinel.tcl $*

sentinel.conf

@@ -86,10 +86,10 @@ sentinel failover-timeout mymaster 180000
 # or to reconfigure clients after a failover. The scripts are executed
 # with the following rules for error handling:
 #
-# If script exists with "1" the execution is retried later (up to a maximum
+# If script exits with "1" the execution is retried later (up to a maximum
 # number of times currently set to 10).
 #
-# If script exists with "2" (or an higher value) the script execution is
+# If script exits with "2" (or an higher value) the script execution is
 # not retried.
 #
 # If script terminates because it receives a signal the behavior is the same

src/Makefile

@@ -107,7 +107,7 @@ endif
 
 REDIS_SERVER_NAME=redis-server
 REDIS_SENTINEL_NAME=redis-sentinel
-REDIS_SERVER_OBJ=adlist.o ae.o anet.o dict.o redis.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o
+REDIS_SERVER_OBJ=adlist.o ae.o anet.o dict.o redis.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o
 REDIS_CLI_NAME=redis-cli
 REDIS_CLI_OBJ=anet.o sds.o adlist.o redis-cli.o zmalloc.o release.o anet.o ae.o crc64.o
 REDIS_BENCHMARK_NAME=redis-benchmark
@@ -204,6 +204,9 @@ distclean: clean
 test: $(REDIS_SERVER_NAME) $(REDIS_CHECK_AOF_NAME)
 	@(cd ..; ./runtest)
 
+test-sentinel: $(REDIS_SENTINEL_NAME)
+	@(cd ..; ./runtest-sentinel)
+
 check: test
 
 lcov:

src/Makefile.dep

@@ -14,6 +14,9 @@ bio.o: bio.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
 bitops.o: bitops.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
  ../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
  ziplist.h intset.h version.h util.h rdb.h rio.h
+blocked.o: blocked.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
+ ../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
+ ziplist.h intset.h version.h util.h rdb.h rio.h
 cluster.o: cluster.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
  ../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
  ziplist.h intset.h version.h util.h rdb.h rio.h cluster.h endianconv.h
@@ -32,6 +35,10 @@ debug.o: debug.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
  ziplist.h intset.h version.h util.h rdb.h rio.h sha1.h crc64.h bio.h
 dict.o: dict.c fmacros.h dict.h zmalloc.h redisassert.h
 endianconv.o: endianconv.c
+hyperloglog.o: hyperloglog.c redis.h fmacros.h config.h \
+ ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
+ adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h rdb.h \
+ rio.h
 intset.o: intset.c intset.h zmalloc.h endianconv.h config.h
 lzf_c.o: lzf_c.c lzfP.h
 lzf_d.o: lzf_d.c lzfP.h
@@ -117,6 +124,3 @@ ziplist.o: ziplist.c zmalloc.h util.h sds.h ziplist.h endianconv.h \
  config.h redisassert.h
 zipmap.o: zipmap.c zmalloc.h endianconv.h config.h
 zmalloc.o: zmalloc.c config.h zmalloc.h
-blocked.o: blocked.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
- ../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
- ziplist.h intset.h version.h util.h rdb.h rio.h

src/anet.c

@@ -234,11 +234,12 @@ static int anetCreateSocket(char *err, int domain) {
 
 #define ANET_CONNECT_NONE 0
 #define ANET_CONNECT_NONBLOCK 1
-static int anetTcpGenericConnect(char *err, char *addr, int port, int flags)
+static int anetTcpGenericConnect(char *err, char *addr, int port,
+                                 char *source_addr, int flags)
 {
     int s = ANET_ERR, rv;
     char portstr[6];  /* strlen("65535") + 1; */
-    struct addrinfo hints, *servinfo, *p;
+    struct addrinfo hints, *servinfo, *bservinfo, *p, *b;
 
     snprintf(portstr,sizeof(portstr),"%d",port);
     memset(&hints,0,sizeof(hints));
@@ -258,6 +259,24 @@ static int anetTcpGenericConnect(char *err, char *addr, int port, int flags)
         if (anetSetReuseAddr(err,s) == ANET_ERR) goto error;
         if (flags & ANET_CONNECT_NONBLOCK && anetNonBlock(err,s) != ANET_OK)
             goto error;
+        if (source_addr) {
+            int bound = 0;
+            /* Using getaddrinfo saves us from self-determining IPv4 vs IPv6 */
+            if ((rv = getaddrinfo(source_addr, NULL, &hints, &bservinfo)) != 0) {
+                anetSetError(err, "%s", gai_strerror(rv));
+                goto end;
+            }
+            for (b = bservinfo; b != NULL; b = b->ai_next) {
+                if (bind(s,b->ai_addr,b->ai_addrlen) != -1) {
+                    bound = 1;
+                    break;
+                }
+            }
+            if (!bound) {
+                anetSetError(err, "bind: %s", strerror(errno));
+                goto end;
+            }
+        }
         if (connect(s,p->ai_addr,p->ai_addrlen) == -1) {
             /* If the socket is non-blocking, it is ok for connect() to
              * return an EINPROGRESS error here. */
@@ -287,12 +306,17 @@ end:
 
 int anetTcpConnect(char *err, char *addr, int port)
 {
-    return anetTcpGenericConnect(err,addr,port,ANET_CONNECT_NONE);
+    return anetTcpGenericConnect(err,addr,port,NULL,ANET_CONNECT_NONE);
 }
 
 int anetTcpNonBlockConnect(char *err, char *addr, int port)
 {
-    return anetTcpGenericConnect(err,addr,port,ANET_CONNECT_NONBLOCK);
+    return anetTcpGenericConnect(err,addr,port,NULL,ANET_CONNECT_NONBLOCK);
+}
+
+int anetTcpNonBlockBindConnect(char *err, char *addr, int port, char *source_addr)
+{
+    return anetTcpGenericConnect(err,addr,port,source_addr,ANET_CONNECT_NONBLOCK);
 }
 
 int anetUnixGenericConnect(char *err, char *path, int flags)

src/anet.h

@@ -45,6 +45,7 @@
 
 int anetTcpConnect(char *err, char *addr, int port);
 int anetTcpNonBlockConnect(char *err, char *addr, int port);
+int anetTcpNonBlockBindConnect(char *err, char *addr, int port, char *source_addr);
 int anetUnixConnect(char *err, char *path);
 int anetUnixNonBlockConnect(char *err, char *path);
 int anetRead(int fd, char *buf, int count);

src/aof.c

@@ -226,6 +226,7 @@ int startAppendOnly(void) {
  *
  * However if force is set to 1 we'll write regardless of the background
  * fsync. */
+#define AOF_WRITE_LOG_ERROR_RATE 30 /* Seconds between errors logging. */
 void flushAppendOnlyFile(int force) {
     ssize_t nwritten;
     int sync_in_progress = 0;
@@ -267,27 +268,76 @@ void flushAppendOnlyFile(int force) {
      * or alike */
     nwritten = write(server.aof_fd,server.aof_buf,sdslen(server.aof_buf));
     if (nwritten != (signed)sdslen(server.aof_buf)) {
-        /* Ooops, we are in troubles. The best thing to do for now is
-         * aborting instead of giving the illusion that everything is
-         * working as expected. */
+        static time_t last_write_error_log = 0;
+        int can_log = 0;
+
+        /* Limit logging rate to 1 line per AOF_WRITE_LOG_ERROR_RATE seconds. */
+        if ((server.unixtime - last_write_error_log) > AOF_WRITE_LOG_ERROR_RATE) {
+            can_log = 1;
+            last_write_error_log = server.unixtime;
+        }
+
+        /* Lof the AOF write error and record the error code. */
         if (nwritten == -1) {
-            redisLog(REDIS_WARNING,"Exiting on error writing to the append-only file: %s",strerror(errno));
+            if (can_log) {
+                redisLog(REDIS_WARNING,"Error writing to the AOF file: %s",
+                    strerror(errno));
+                server.aof_last_write_errno = errno;
+            }
         } else {
-            redisLog(REDIS_WARNING,"Exiting on short write while writing to "
-                                   "the append-only file: %s (nwritten=%ld, "
-                                   "expected=%ld)",
-                                   strerror(errno),
-                                   (long)nwritten,
-                                   (long)sdslen(server.aof_buf));
+            if (can_log) {
+                redisLog(REDIS_WARNING,"Short write while writing to "
+                                       "the AOF file: (nwritten=%lld, "
+                                       "expected=%lld)",
+                                       (long long)nwritten,
+                                       (long long)sdslen(server.aof_buf));
+            }
 
             if (ftruncate(server.aof_fd, server.aof_current_size) == -1) {
-                redisLog(REDIS_WARNING, "Could not remove short write "
-                         "from the append-only file.  Redis may refuse "
-                         "to load the AOF the next time it starts.  "
-                         "ftruncate: %s", strerror(errno));
+                if (can_log) {
+                    redisLog(REDIS_WARNING, "Could not remove short write "
+                             "from the append-only file.  Redis may refuse "
+                             "to load the AOF the next time it starts.  "
+                             "ftruncate: %s", strerror(errno));
+                }
+            } else {
+                /* If the ftrunacate() succeeded we can set nwritten to
+                 * -1 since there is no longer partial data into the AOF. */
+                nwritten = -1;
             }
+            server.aof_last_write_errno = ENOSPC;
+        }
+
+        /* Handle the AOF write error. */
+        if (server.aof_fsync == AOF_FSYNC_ALWAYS) {
+            /* We can't recover when the fsync policy is ALWAYS since the
+             * reply for the client is already in the output buffers, and we
+             * have the contract with the user that on acknowledged write data
+             * is synched on disk. */
+            redisLog(REDIS_WARNING,"Can't recover from AOF write error when the AOF fsync policy is 'always'. Exiting...");
+            exit(1);
+        } else {
+            /* Recover from failed write leaving data into the buffer. However
+             * set an error to stop accepting writes as long as the error
+             * condition is not cleared. */
+            server.aof_last_write_status = REDIS_ERR;
+
+            /* Trim the sds buffer if there was a partial write, and there
+             * was no way to undo it with ftruncate(2). */
+            if (nwritten > 0) {
+                server.aof_current_size += nwritten;
+                sdsrange(server.aof_buf,nwritten,-1);
+            }
+            return; /* We'll try again on the next call... */
+        }
+    } else {
+        /* Successful write(2). If AOF was in error state, restore the
+         * OK state and log the event. */
+        if (server.aof_last_write_status == REDIS_ERR) {
+            redisLog(REDIS_WARNING,
+                "AOF write error looks solved, Redis can write again.");
+            server.aof_last_write_status = REDIS_OK;
         }
-        exit(1);
     }
     server.aof_current_size += nwritten;
 
@@ -919,9 +969,9 @@ int rewriteAppendOnlyFile(char *filename) {
     }
 
     /* Make sure data will not remain on the OS's output buffers */
-    fflush(fp);
-    aof_fsync(fileno(fp));
-    fclose(fp);
+    if (fflush(fp) == EOF) goto werr;
+    if (aof_fsync(fileno(fp)) == -1) goto werr;
+    if (fclose(fp) == EOF) goto werr;
 
     /* Use RENAME to make sure the DB file is changed atomically only
      * if the generate DB file is ok. */

src/bitops.c

@@ -60,11 +60,18 @@ static int getBitOffsetFromArgument(redisClient *c, robj *o, size_t *offset) {
  * work with a input string length up to 512 MB. */
 size_t redisPopcount(void *s, long count) {
     size_t bits = 0;
-    unsigned char *p;
-    uint32_t *p4 = s;
+    unsigned char *p = s;
+    uint32_t *p4;
     static const unsigned char bitsinbyte[256] = {0,1,1,2,1,2,2,3,1,2,2,3,2,3,3,4,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,4,5,5,6,5,6,6,7,5,6,6,7,6,7,7,8};
 
+    /* Count initial bytes not aligned to 32 bit. */
+    while((unsigned long)p & 3 && count) {
+        bits += bitsinbyte[*p++];
+        count--;
+    }
+
     /* Count bits 16 bytes at a time */
+    p4 = (uint32_t*)p;
     while(count>=16) {
         uint32_t aux1, aux2, aux3, aux4;
 
@@ -87,12 +94,99 @@ size_t redisPopcount(void *s, long count) {
                 ((((aux3 + (aux3 >> 4)) & 0x0F0F0F0F) * 0x01010101) >> 24) +
                 ((((aux4 + (aux4 >> 4)) & 0x0F0F0F0F) * 0x01010101) >> 24);
     }
-    /* Count the remaining bytes */
+    /* Count the remaining bytes. */
     p = (unsigned char*)p4;
     while(count--) bits += bitsinbyte[*p++];
     return bits;
 }
 
+/* Return the position of the first bit set to one (if 'bit' is 1) or
+ * zero (if 'bit' is 0) in the bitmap starting at 's' and long 'count' bytes.
+ *
+ * The function is guaranteed to return a value >= 0 if 'bit' is 0 since if
+ * no zero bit is found, it returns count*8 assuming the string is zero
+ * padded on the right. However if 'bit' is 1 it is possible that there is
+ * not a single set bit in the bitmap. In this special case -1 is returned. */
+long redisBitpos(void *s, long count, int bit) {
+    unsigned long *l;
+    unsigned char *c;
+    unsigned long skipval, word = 0, one;
+    long pos = 0; /* Position of bit, to return to the caller. */
+    int j;
+
+    /* Process whole words first, seeking for first word that is not
+     * all ones or all zeros respectively if we are lookig for zeros
+     * or ones. This is much faster with large strings having contiguous
+     * blocks of 1 or 0 bits compared to the vanilla bit per bit processing.
+     *
+     * Note that if we start from an address that is not aligned
+     * to sizeof(unsigned long) we consume it byte by byte until it is
+     * aligned. */
+
+    /* Skip initial bits not aligned to sizeof(unsigned long) byte by byte. */
+    skipval = bit ? 0 : UCHAR_MAX;
+    c = (unsigned char*) s;
+    while((unsigned long)c & (sizeof(*l)-1) && count) {
+        if (*c != skipval) break;
+        c++;
+        count--;
+        pos += 8;
+    }
+
+    /* Skip bits with full word step. */
+    skipval = bit ? 0 : ULONG_MAX;
+    l = (unsigned long*) c;
+    while (count >= sizeof(*l)) {
+        if (*l != skipval) break;
+        l++;
+        count -= sizeof(*l);
+        pos += sizeof(*l)*8;
+    }
+
+    /* Load bytes into "word" considering the first byte as the most significant
+     * (we basically consider it as written in big endian, since we consider the
+     * string as a set of bits from left to right, with the first bit at position
+     * zero.
+     *
+     * Note that the loading is designed to work even when the bytes left
+     * (count) are less than a full word. We pad it with zero on the right. */
+    c = (unsigned char*)l;
+    for (j = 0; j < sizeof(*l); j++) {
+        word <<= 8;
+        if (count) {
+            word |= *c;
+            c++;
+            count--;
+        }
+    }
+
+    /* Special case:
+     * If bits in the string are all zero and we are looking for one,
+     * return -1 to signal that there is not a single "1" in the whole
+     * string. This can't happen when we are looking for "0" as we assume
+     * that the right of the string is zero padded. */
+    if (bit == 1 && word == 0) return -1;
+
+    /* Last word left, scan bit by bit. The first thing we need is to
+     * have a single "1" set in the most significant position in an
+     * unsigned long. We don't know the size of the long so we use a
+     * simple trick. */
+    one = ULONG_MAX; /* All bits set to 1.*/
+    one >>= 1;       /* All bits set to 1 but the MSB. */
+    one = ~one;      /* All bits set to 0 but the MSB. */
+
+    while(one) {
+        if (((one & word) != 0) == bit) return pos;
+        pos++;
+        one >>= 1;
+    }
+
+    /* If we reached this point, there is a bug in the algorithm, since
+     * the case of no match is handled as a special case before. */
+    redisPanic("End of redisBitpos() reached.");
+    return 0; /* Just to avoid warnings. */
+}
+
 /* -----------------------------------------------------------------------------
  * Bits related string commands: GETBIT, SETBIT, BITCOUNT, BITOP.
  * -------------------------------------------------------------------------- */
@@ -129,14 +223,7 @@ void setbitCommand(redisClient *c) {
         dbAdd(c->db,c->argv[1],o);
     } else {
         if (checkType(c,o,REDIS_STRING)) return;
-
-        /* Create a copy when the object is shared or encoded. */
-        if (o->refcount != 1 || o->encoding != REDIS_ENCODING_RAW) {
-            robj *decoded = getDecodedObject(o);
-            o = createRawStringObject(decoded->ptr, sdslen(decoded->ptr));
-            decrRefCount(decoded);
-            dbOverwrite(c->db,c->argv[1],o);
-        }
+        o = dbUnshareStringValue(c->db,c->argv[1],o);
     }
 
     /* Grow sds value to the right length if necessary */
@@ -174,6 +261,7 @@ void getbitCommand(redisClient *c) {
 
     byte = bitoffset >> 3;
     bit = 7 - (bitoffset & 0x7);
+    if (lzfEncodedObject(o)) o = dbUnshareStringValue(c->db,c->argv[1],o);
     if (sdsEncodedObject(o)) {
         if (byte < sdslen(o->ptr))
             bitval = ((uint8_t*)o->ptr)[byte] & (1 << bit);
@@ -370,6 +458,7 @@ void bitcountCommand(redisClient *c) {
 
     /* Set the 'p' pointer to the string, that can be just a stack allocated
      * array if our string was integer encoded. */
+    if (lzfEncodedObject(o)) o = dbUnshareStringValue(c->db,c->argv[1],o);
     if (o->encoding == REDIS_ENCODING_INT) {
         p = (unsigned char*) llbuf;
         strlen = ll2string(llbuf,sizeof(llbuf),(long)o->ptr);
@@ -410,3 +499,91 @@ void bitcountCommand(redisClient *c) {
         addReplyLongLong(c,redisPopcount(p+start,bytes));
     }
 }
+
+/* BITPOS key bit [start [end]] */
+void bitposCommand(redisClient *c) {
+    robj *o;
+    long bit, start, end, strlen;
+    unsigned char *p;
+    char llbuf[32];
+    int end_given = 0;
+
+    /* Parse the bit argument to understand what we are looking for, set
+     * or clear bits. */
+    if (getLongFromObjectOrReply(c,c->argv[2],&bit,NULL) != REDIS_OK)
+        return;
+    if (bit != 0 && bit != 1) {
+        addReplyError(c, "The bit argument must be 1 or 0.");
+        return;
+    }
+
+    /* If the key does not exist, from our point of view it is an infinite
+     * array of 0 bits. If the user is looking for the fist clear bit return 0,
+     * If the user is looking for the first set bit, return -1. */
+    if ((o = lookupKeyRead(c->db,c->argv[1])) == NULL) {
+        addReplyLongLong(c, bit ? -1 : 0);
+        return;
+    }
+    if (checkType(c,o,REDIS_STRING)) return;
+
+    /* Set the 'p' pointer to the string, that can be just a stack allocated
+     * array if our string was integer encoded. */
+    if (lzfEncodedObject(o)) o = dbUnshareStringValue(c->db,c->argv[1],o);
+    if (o->encoding == REDIS_ENCODING_INT) {
+        p = (unsigned char*) llbuf;
+        strlen = ll2string(llbuf,sizeof(llbuf),(long)o->ptr);
+    } else {
+        p = (unsigned char*) o->ptr;
+        strlen = sdslen(o->ptr);
+    }
+
+    /* Parse start/end range if any. */
+    if (c->argc == 4 || c->argc == 5) {
+        if (getLongFromObjectOrReply(c,c->argv[3],&start,NULL) != REDIS_OK)
+            return;
+        if (c->argc == 5) {
+            if (getLongFromObjectOrReply(c,c->argv[4],&end,NULL) != REDIS_OK)
+                return;
+            end_given = 1;
+        } else {
+            end = strlen-1;
+        }
+        /* Convert negative indexes */
+        if (start < 0) start = strlen+start;
+        if (end < 0) end = strlen+end;
+        if (start < 0) start = 0;
+        if (end < 0) end = 0;
+        if (end >= strlen) end = strlen-1;
+    } else if (c->argc == 3) {
+        /* The whole string. */
+        start = 0;
+        end = strlen-1;
+    } else {
+        /* Syntax error. */
+        addReply(c,shared.syntaxerr);
+        return;
+    }
+
+    /* For empty ranges (start > end) we return -1 as an empty range does
+     * not contain a 0 nor a 1. */
+    if (start > end) {
+        addReplyLongLong(c, -1);
+    } else {
+        long bytes = end-start+1;
+        long pos = redisBitpos(p+start,bytes,bit);
+
+        /* If we are looking for clear bits, and the user specified an exact
+         * range with start-end, we can't consider the right of the range as
+         * zero padded (as we do when no explicit end is given).
+         *
+         * So if redisBitpos() returns the first bit outside the range,
+         * we return -1 to the caller, to mean, in the specified range there
+         * is not a single "0" bit. */
+        if (end_given && bit == 0 && pos == bytes*8) {
+            addReplyLongLong(c,-1);
+            return;
+        }
+        if (pos != -1) pos += start*8; /* Adjust for the bytes we skipped. */
+        addReplyLongLong(c,pos);
+    }
+}

src/cluster.h

@@ -10,7 +10,6 @@
 #define REDIS_CLUSTER_FAIL 1        /* The cluster can't work */
 #define REDIS_CLUSTER_NAMELEN 40    /* sha1 hex length */
 #define REDIS_CLUSTER_PORT_INCR 10000 /* Cluster port = baseport + PORT_INCR */
-#define REDIS_CLUSTER_IPLEN INET6_ADDRSTRLEN /* IPv6 address string length */
 
 /* The following defines are amunt of time, sometimes expressed as
  * multiplicators of the node timeout value (when ending with MULT). */
@@ -19,12 +18,18 @@
 #define REDIS_CLUSTER_FAIL_UNDO_TIME_MULT 2 /* Undo fail if master is back. */
 #define REDIS_CLUSTER_FAIL_UNDO_TIME_ADD 10 /* Some additional time. */
 #define REDIS_CLUSTER_SLAVE_VALIDITY_MULT 10 /* Slave data validity. */
-#define REDIS_CLUSTER_FAILOVER_AUTH_RETRY_MULT 4 /* Auth request retry time. */
 #define REDIS_CLUSTER_FAILOVER_DELAY 5 /* Seconds */
 #define REDIS_CLUSTER_DEFAULT_MIGRATION_BARRIER 1
 #define REDIS_CLUSTER_MF_TIMEOUT 5000 /* Milliseconds to do a manual failover. */
 #define REDIS_CLUSTER_MF_PAUSE_MULT 2 /* Master pause manual failover mult. */
 
+/* Redirection errors returned by getNodeByQuery(). */
+#define REDIS_CLUSTER_REDIR_NONE 0          /* Node can serve the request. */
+#define REDIS_CLUSTER_REDIR_CROSS_SLOT 1    /* Keys in different slots. */
+#define REDIS_CLUSTER_REDIR_UNSTABLE 2      /* Keys in slot resharding. */
+#define REDIS_CLUSTER_REDIR_ASK 3           /* -ASK redirection required. */
+#define REDIS_CLUSTER_REDIR_MOVED 4         /* -MOVED redirection required. */
+
 struct clusterNode;
 
 /* clusterLink encapsulates everything needed to talk with a remote node. */
@@ -113,7 +118,7 @@ typedef struct clusterState {
     int mf_can_start;           /* If non-zero signal that the manual failover
                                    can start requesting masters vote. */
     /* The followign fields are uesd by masters to take state on elections. */
-    uint64_t last_vote_epoch;   /* Epoch of the last vote granted. */
+    uint64_t lastVoteEpoch;     /* Epoch of the last vote granted. */
     int todo_before_sleep; /* Things to do in clusterBeforeSleep(). */
     long long stats_bus_messages_sent;  /* Num of msg sent via cluster bus. */
     long long stats_bus_messages_received; /* Num of msg rcvd via cluster bus.*/
@@ -148,7 +153,7 @@ typedef struct {
     char nodename[REDIS_CLUSTER_NAMELEN];
     uint32_t ping_sent;
     uint32_t pong_received;
-    char ip[16];    /* IP address last time it was seen */
+    char ip[REDIS_IP_STR_LEN];    /* IP address last time it was seen */
     uint16_t port;  /* port last time it was seen */
     uint16_t flags;
     uint32_t notused; /* for 64 bit alignment */
@@ -195,7 +200,10 @@ union clusterMsgData {
 
 
 typedef struct {
+    char sig[4];        /* Siganture "RCmb" (Redis Cluster message bus). */
     uint32_t totlen;    /* Total length of this message */
+    uint16_t ver;       /* Protocol version, currently set to 0. */
+    uint16_t notused0;  /* 2 bytes not used. */
     uint16_t type;      /* Message type */
     uint16_t count;     /* Only used for some kind of messages. */
     uint64_t currentEpoch;  /* The epoch accordingly to the sending node. */

src/config.c

@@ -297,6 +297,10 @@ void loadServerConfigFromString(char *config) {
             if ((server.rdb_compression = yesnotoi(argv[1])) == -1) {
                 err = "argument must be 'yes' or 'no'"; goto loaderr;
             }
+        } else if (!strcasecmp(argv[0],"memcompression") && argc == 2) {
+            if ((server.mem_compression = yesnotoi(argv[1])) == -1) {
+                err = "argument must be 'yes' or 'no'"; goto loaderr;
+            }
         } else if (!strcasecmp(argv[0],"rdbchecksum") && argc == 2) {
             if ((server.rdb_checksum = yesnotoi(argv[1])) == -1) {
                 err = "argument must be 'yes' or 'no'"; goto loaderr;
@@ -600,7 +604,7 @@ void configSetCommand(redisClient *c) {
     } else if (!strcasecmp(c->argv[2]->ptr,"maxclients")) {
         int orig_value = server.maxclients;
 
-        if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll < 0) goto badfmt;
+        if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll < 1) goto badfmt;
 
         /* Try to check if the OS is capable of supporting so many FDs. */
         server.maxclients = ll;
@@ -860,6 +864,11 @@ void configSetCommand(redisClient *c) {
 
         if (yn == -1) goto badfmt;
         server.rdb_compression = yn;
+    } else if (!strcasecmp(c->argv[2]->ptr,"memcompression")) {
+        int yn = yesnotoi(o->ptr);
+
+        if (yn == -1) goto badfmt;
+        server.mem_compression = yn;
     } else if (!strcasecmp(c->argv[2]->ptr,"notify-keyspace-events")) {
         int flags = keyspaceEventsStringToFlags(o->ptr);
 
@@ -1006,6 +1015,7 @@ void configGetCommand(redisClient *c) {
             server.stop_writes_on_bgsave_err);
     config_get_bool_field("daemonize", server.daemonize);
     config_get_bool_field("rdbcompression", server.rdb_compression);
+    config_get_bool_field("memcompression", server.mem_compression);
     config_get_bool_field("rdbchecksum", server.rdb_checksum);
     config_get_bool_field("activerehashing", server.activerehashing);
     config_get_bool_field("repl-disable-tcp-nodelay",
@@ -1461,7 +1471,7 @@ void rewriteConfigSaveOption(struct rewriteConfigState *state) {
      * resulting into no RDB persistence as expected. */
     for (j = 0; j < server.saveparamslen; j++) {
         line = sdscatprintf(sdsempty(),"save %ld %d",
-            server.saveparams[j].seconds, server.saveparams[j].changes);
+            (long) server.saveparams[j].seconds, server.saveparams[j].changes);
         rewriteConfigRewriteLine(state,"save",line,1);
     }
     /* Mark "save" as processed in case server.saveparamslen is zero. */
@@ -1721,6 +1731,7 @@ int rewriteConfig(char *path) {
     rewriteConfigNumericalOption(state,"databases",server.dbnum,REDIS_DEFAULT_DBNUM);
     rewriteConfigYesNoOption(state,"stop-writes-on-bgsave-error",server.stop_writes_on_bgsave_err,REDIS_DEFAULT_STOP_WRITES_ON_BGSAVE_ERROR);
     rewriteConfigYesNoOption(state,"rdbcompression",server.rdb_compression,REDIS_DEFAULT_RDB_COMPRESSION);
+    rewriteConfigYesNoOption(state,"memcompression",server.mem_compression,REDIS_DEFAULT_MEM_COMPRESSION);
     rewriteConfigYesNoOption(state,"rdbchecksum",server.rdb_checksum,REDIS_DEFAULT_RDB_CHECKSUM);
     rewriteConfigStringOption(state,"dbfilename",server.rdb_filename,REDIS_DEFAULT_RDB_FILENAME);
     rewriteConfigDirOption(state);
@@ -1807,14 +1818,7 @@ void configCommand(redisClient *c) {
         configGetCommand(c);
     } else if (!strcasecmp(c->argv[1]->ptr,"resetstat")) {
         if (c->argc != 2) goto badarity;
-        server.stat_keyspace_hits = 0;
-        server.stat_keyspace_misses = 0;
-        server.stat_numcommands = 0;
-        server.stat_numconnections = 0;
-        server.stat_expiredkeys = 0;
-        server.stat_rejected_conn = 0;
-        server.stat_fork_time = 0;
-        server.aof_delayed_fsync = 0;
+        resetServerStats();
         resetCommandTableStats();
         addReply(c,shared.ok);
     } else if (!strcasecmp(c->argv[1]->ptr,"rewrite")) {
@@ -1824,8 +1828,10 @@ void configCommand(redisClient *c) {
             return;
         }
         if (rewriteConfig(server.configfile) == -1) {
+            redisLog(REDIS_WARNING,"CONFIG REWRITE failed: %s", strerror(errno));
             addReplyErrorFormat(c,"Rewriting config file: %s", strerror(errno));
         } else {
+            redisLog(REDIS_WARNING,"CONFIG REWRITE executed with success.");
             addReply(c,shared.ok);
         }
     } else {

src/db.c

@@ -50,7 +50,7 @@ robj *lookupKey(redisDb *db, robj *key) {
          * Don't do it if we have a saving child, as this will trigger
          * a copy on write madness. */
         if (server.rdb_child_pid == -1 && server.aof_child_pid == -1)
-            val->lru = server.lruclock;
+            val->lru = LRU_CLOCK();
         return val;
     } else {
         return NULL;
@@ -104,7 +104,7 @@ void dbAdd(redisDb *db, robj *key, robj *val) {
  *
  * The program is aborted if the key was not already present. */
 void dbOverwrite(redisDb *db, robj *key, robj *val) {
-    struct dictEntry *de = dictFind(db->dict,key->ptr);
+    dictEntry *de = dictFind(db->dict,key->ptr);
     
     redisAssertWithInfo(NULL,key,de != NULL);
     dictReplace(db->dict, key->ptr, val);
@@ -136,7 +136,7 @@ int dbExists(redisDb *db, robj *key) {
  *
  * The function makes sure to return keys not already expired. */
 robj *dbRandomKey(redisDb *db) {
-    struct dictEntry *de;
+    dictEntry *de;
 
     while(1) {
         sds key;
@@ -170,6 +170,44 @@ int dbDelete(redisDb *db, robj *key) {
     }
 }
 
+/* Prepare the string object stored at 'key' to be modified destructively
+ * to implement commands like SETBIT or APPEND.
+ *
+ * An object is usually ready to be modified unless one of the two conditions
+ * are true:
+ *
+ * 1) The object 'o' is shared (refcount > 1), we don't want to affect
+ *    other users.
+ * 2) The object encoding is not "RAW".
+ *
+ * If the object is found in one of the above conditions (or both) by the
+ * function, an unshared / not-encoded copy of the string object is stored
+ * at 'key' in the specified 'db'. Otherwise the object 'o' itself is
+ * returned.
+ *
+ * USAGE:
+ *
+ * The object 'o' is what the caller already obtained by looking up 'key'
+ * in 'db', the usage pattern looks like this:
+ *
+ * o = lookupKeyWrite(db,key);
+ * if (checkType(c,o,REDIS_STRING)) return;
+ * o = dbUnshareStringValue(db,key,o);
+ *
+ * At this point the caller is ready to modify the object, for example
+ * using an sdscat() call to append some data, or anything else.
+ */
+robj *dbUnshareStringValue(redisDb *db, robj *key, robj *o) {
+    redisAssert(o->type == REDIS_STRING);
+    if (o->refcount != 1 || o->encoding != REDIS_ENCODING_RAW) {
+        robj *decoded = getDecodedObject(o);
+        o = createRawStringObject(decoded->ptr, sdslen(decoded->ptr));
+        decrRefCount(decoded);
+        dbOverwrite(db,key,o);
+    }
+    return o;
+}
+
 long long emptyDb(void(callback)(void*)) {
     int j;
     long long removed = 0;
@@ -930,6 +968,8 @@ void persistCommand(redisClient *c) {
  * API to get key arguments from commands
  * ---------------------------------------------------------------------------*/
 
+/* The base case is to use the keys position as given in the command table
+ * (firstkey, lastkey, step). */
 int *getKeysUsingCommandTable(struct redisCommand *cmd,robj **argv, int argc, int *numkeys) {
     int j, i = 0, last, *keys;
     REDIS_NOTUSED(argv);
@@ -949,42 +989,36 @@ int *getKeysUsingCommandTable(struct redisCommand *cmd,robj **argv, int argc, in
     return keys;
 }
 
-int *getKeysFromCommand(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags) {
+/* Return all the arguments that are keys in the command passed via argc / argv.
+ *
+ * The command returns the positions of all the key arguments inside the array,
+ * so the actual return value is an heap allocated array of integers. The
+ * length of the array is returned by reference into *numkeys.
+ *
+ * 'cmd' must be point to the corresponding entry into the redisCommand
+ * table, according to the command name in argv[0].
+ *
+ * This function uses the command table if a command-specific helper function
+ * is not required, otherwise it calls the command-specific function. */
+int *getKeysFromCommand(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
     if (cmd->getkeys_proc) {
-        return cmd->getkeys_proc(cmd,argv,argc,numkeys,flags);
+        return cmd->getkeys_proc(cmd,argv,argc,numkeys);
     } else {
         return getKeysUsingCommandTable(cmd,argv,argc,numkeys);
     }
 }
 
+/* Free the result of getKeysFromCommand. */
 void getKeysFreeResult(int *result) {
     zfree(result);
 }
 
-int *noPreloadGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags) {
-    if (flags & REDIS_GETKEYS_PRELOAD) {
-        *numkeys = 0;
-        return NULL;
-    } else {
-        return getKeysUsingCommandTable(cmd,argv,argc,numkeys);
-    }
-}
-
-int *renameGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags) {
-    if (flags & REDIS_GETKEYS_PRELOAD) {
-        int *keys = zmalloc(sizeof(int));
-        *numkeys = 1;
-        keys[0] = 1;
-        return keys;
-    } else {
-        return getKeysUsingCommandTable(cmd,argv,argc,numkeys);
-    }
-}
-
-int *zunionInterGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags) {
+/* Helper function to extract keys from following commands:
+ * ZUNIONSTORE <destkey> <num-keys> <key> <key> ... <key> <options>
+ * ZINTERSTORE <destkey> <num-keys> <key> <key> ... <key> <options> */
+int *zunionInterGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
     int i, num, *keys;
     REDIS_NOTUSED(cmd);
-    REDIS_NOTUSED(flags);
 
     num = atoi(argv[2]->ptr);
     /* Sanity check. Don't return any key if the command is going to
@@ -993,8 +1027,89 @@ int *zunionInterGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *num
         *numkeys = 0;
         return NULL;
     }
-    keys = zmalloc(sizeof(int)*num);
+
+    /* Keys in z{union,inter}store come from two places:
+     * argv[1] = storage key,
+     * argv[3...n] = keys to intersect */
+    keys = zmalloc(sizeof(int)*(num+1));
+
+    /* Add all key positions for argv[3...n] to keys[] */
     for (i = 0; i < num; i++) keys[i] = 3+i;
+
+    /* Finally add the argv[1] key position (the storage key target). */
+    keys[num] = 1;
+    *numkeys = num+1;  /* Total keys = {union,inter} keys + storage key */
+    return keys;
+}
+
+/* Helper function to extract keys from the following commands:
+ * EVAL <script> <num-keys> <key> <key> ... <key> [more stuff]
+ * EVALSHA <script> <num-keys> <key> <key> ... <key> [more stuff] */
+int *evalGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
+    int i, num, *keys;
+    REDIS_NOTUSED(cmd);
+
+    num = atoi(argv[2]->ptr);
+    /* Sanity check. Don't return any key if the command is going to
+     * reply with syntax error. */
+    if (num > (argc-3)) {
+        *numkeys = 0;
+        return NULL;
+    }
+
+    keys = zmalloc(sizeof(int)*num);
+    *numkeys = num;
+
+    /* Add all key positions for argv[3...n] to keys[] */
+    for (i = 0; i < num; i++) keys[i] = 3+i;
+
+    return keys;
+}
+
+/* Helper function to extract keys from the SORT command.
+ *
+ * SORT <sort-key> ... STORE <store-key> ...
+ *
+ * The first argument of SORT is always a key, however a list of options
+ * follow in SQL-alike style. Here we parse just the minimum in order to
+ * correctly identify keys in the "STORE" option. */
+int *sortGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
+    int i, j, num, *keys;
+    REDIS_NOTUSED(cmd);
+
+    num = 0;
+    keys = zmalloc(sizeof(int)*2); /* Alloc 2 places for the worst case. */
+
+    keys[num++] = 1; /* <sort-key> is always present. */
+
+    /* Search for STORE option. By default we consider options to don't
+     * have arguments, so if we find an unknown option name we scan the
+     * next. However there are options with 1 or 2 arguments, so we
+     * provide a list here in order to skip the right number of args. */
+    struct {
+        char *name;
+        int skip;
+    } skiplist[] = {
+        {"limit", 2},
+        {"get", 1},
+        {"by", 1},
+        {NULL, 0} /* End of elements. */
+    };
+
+    for (i = 2; i < argc; i++) {
+        for (j = 0; skiplist[j].name != NULL; j++) {
+            if (!strcasecmp(argv[i]->ptr,skiplist[j].name)) {
+                i += skiplist[j].skip;
+                break;
+            } else if (!strcasecmp(argv[i]->ptr,"store") && i+1 < argc) {
+                /* Note: we don't increment "num" here and continue the loop
+                 * to be sure to process the *last* "STORE" option if multiple
+                 * ones are provided. This is same behavior as SORT. */
+                keys[num] = i+1; /* <store-key> */
+                break;
+            }
+        }
+    }
     *numkeys = num;
     return keys;
 }

src/debug.c

@@ -292,7 +292,7 @@ void debugCommand(redisClient *c) {
         addReplyStatusFormat(c,
             "Value at:%p refcount:%d "
             "encoding:%s serializedlength:%lld "
-            "lru:%d lru_seconds_idle:%lu",
+            "lru:%d lru_seconds_idle:%llu",
             (void*)val, val->refcount,
             strenc, (long long) rdbSavedObjectLen(val),
             val->lru, estimateObjectIdleTime(val));
@@ -308,8 +308,10 @@ void debugCommand(redisClient *c) {
         val = dictGetVal(de);
         key = dictGetKey(de);
 
-        if (val->type != REDIS_STRING || !sdsEncodedObject(val)) {
-            addReplyError(c,"Not an sds encoded string.");
+        if (val->type != REDIS_STRING ||
+            (!sdsEncodedObject(val) && val->encoding != REDIS_ENCODING_LZF))
+        {
+            addReplyError(c,"Not an sds/lzf encoded string.");
         } else {
             addReplyStatusFormat(c,
                 "key_sds_len:%lld, key_sds_avail:%lld, "
@@ -363,6 +365,31 @@ void debugCommand(redisClient *c) {
     {
         server.active_expire_enabled = atoi(c->argv[2]->ptr);
         addReply(c,shared.ok);
+    } else if (!strcasecmp(c->argv[1]->ptr,"cmdkeys") && c->argc >= 3) {
+        struct redisCommand *cmd = lookupCommand(c->argv[2]->ptr);
+        int *keys, numkeys, j;
+
+        if (!cmd) {
+            addReplyError(c,"Invalid command specified");
+            return;
+        } else if ((cmd->arity > 0 && cmd->arity != c->argc-2) ||
+                   ((c->argc-2) < -cmd->arity))
+        {
+            addReplyError(c,"Invalid number of arguments specified for command");
+            return;
+        }
+
+        keys = getKeysFromCommand(cmd,c->argv+2,c->argc-2,&numkeys);
+        addReplyMultiBulkLen(c,numkeys);
+        for (j = 0; j < numkeys; j++) addReplyBulk(c,c->argv[keys[j]+2]);
+        getKeysFreeResult(keys);
+    } else if (!strcasecmp(c->argv[1]->ptr,"error") && c->argc == 3) {
+        sds errstr = sdsnewlen("-",1);
+
+        errstr = sdscatsds(errstr,c->argv[2]->ptr);
+        errstr = sdsmapchars(errstr,"\n\r","  ",2); /* no newlines in errors. */
+        errstr = sdscatlen(errstr,"\r\n",2);
+        addReplySds(c,errstr);
     } else {
         addReplyErrorFormat(c, "Unknown DEBUG subcommand or wrong number of arguments for '%s'",
             (char*)c->argv[1]->ptr);

src/dict.c

@@ -239,7 +239,7 @@ int dictExpand(dict *d, unsigned long size)
 /* Performs N steps of incremental rehashing. Returns 1 if there are still
  * keys to move from the old to the new hash table, otherwise 0 is returned.
  * Note that a rehashing step consists in moving a bucket (that may have more
- * thank one key as we use chaining) from the old to the new hash table. */
+ * than one key as we use chaining) from the old to the new hash table. */
 int dictRehash(dict *d, int n) {
     if (!dictIsRehashing(d)) return 0;
 
@@ -649,6 +649,58 @@ dictEntry *dictGetRandomKey(dict *d)
     return he;
 }
 
+/* This is a version of dictGetRandomKey() that is modified in order to
+ * return multiple entries by jumping at a random place of the hash table
+ * and scanning linearly for entries.
+ *
+ * Returned pointers to hash table entries are stored into 'des' that
+ * points to an array of dictEntry pointers. The array must have room for
+ * at least 'count' elements, that is the argument we pass to the function
+ * to tell how many random elements we need.
+ *
+ * The function returns the number of items stored into 'des', that may
+ * be less than 'count' if the hash table has less than 'count' elements
+ * inside.
+ *
+ * Note that this function is not suitable when you need a good distribution
+ * of the returned items, but only when you need to "sample" a given number
+ * of continuous elements to run some kind of algorithm or to produce
+ * statistics. However the function is much faster than dictGetRandomKey()
+ * at producing N elements, and the elements are guaranteed to be non
+ * repeating. */
+int dictGetRandomKeys(dict *d, dictEntry **des, int count) {
+    int j; /* internal hash table id, 0 or 1. */
+    int stored = 0;
+
+    if (dictSize(d) < count) count = dictSize(d);
+    while(stored < count) {
+        for (j = 0; j < 2; j++) {
+            /* Pick a random point inside the hash table 0 or 1. */
+            unsigned int i = random() & d->ht[j].sizemask;
+            int size = d->ht[j].size;
+
+            /* Make sure to visit every bucket by iterating 'size' times. */
+            while(size--) {
+                dictEntry *he = d->ht[j].table[i];
+                while (he) {
+                    /* Collect all the elements of the buckets found non
+                     * empty while iterating. */
+                    *des = he;
+                    des++;
+                    he = he->next;
+                    stored++;
+                    if (stored == count) return stored;
+                }
+                i = (i+1) & d->ht[j].sizemask;
+            }
+            /* If there is only one table and we iterated it all, we should
+             * already have 'count' elements. Assert this condition. */
+            assert(dictIsRehashing(d) != 0);
+        }
+    }
+    return stored; /* Never reached. */
+}
+
 /* Function to reverse bits. Algorithm from:
  * http://graphics.stanford.edu/~seander/bithacks.html#ReverseParallel */
 static unsigned long rev(unsigned long v) {
@@ -695,7 +747,7 @@ static unsigned long rev(unsigned long v) {
  * (where SIZE-1 is always the mask that is equivalent to taking the rest
  *  of the division between the Hash of the key and SIZE).
  *
- * For example if the current hash table size is 64, the mask is
+ * For example if the current hash table size is 16, the mask is
  * (in binary) 1111. The position of a key in the hash table will be always
  * the last four bits of the hash output, and so forth.
  *

src/dict.h

@@ -157,6 +157,7 @@ dictIterator *dictGetSafeIterator(dict *d);
 dictEntry *dictNext(dictIterator *iter);
 void dictReleaseIterator(dictIterator *iter);
 dictEntry *dictGetRandomKey(dict *d);
+int dictGetRandomKeys(dict *d, dictEntry **des, int count);
 void dictPrintStats(dict *d);
 unsigned int dictGenHashFunction(const void *key, int len);
 unsigned int dictGenCaseHashFunction(const unsigned char *buf, int len);

src/hyperloglog.c

@@ -0,0 +1,714 @@
+/* hyperloglog.c - Redis HyperLogLog probabilistic cardinality approximation.
+ * This file implements the algorithm and the exported Redis commands.
+ *
+ * Copyright (c) 2014, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "redis.h"
+
+#include <stdint.h>
+#include <math.h>
+
+/* The Redis HyperLogLog implementation is based on the following ideas:
+ *
+ * * The use of a 64 bit hash function as proposed in [1], in order to don't
+ *   limited to cardinalities up to 10^9, at the cost of just 1 additional
+ *   bit per register.
+ * * The use of 16384 6-bit registers for a great level of accuracy, using
+ *   a total of 12k per key.
+ * * The use of the Redis string data type. No new type is introduced.
+ * * No attempt is made to compress the data structure as in [1]. Also the
+ *   algorithm used is the original HyperLogLog Algorithm as in [2], with
+ *   the only difference that a 64 bit hash function is used, so no correction
+ *   is performed for values near 2^32 as in [1].
+ *
+ * [1] Heule, Nunkesser, Hall: HyperLogLog in Practice: Algorithmic
+ *     Engineering of a State of The Art Cardinality Estimation Algorithm.
+ *
+ * [2] P. Flajolet, Éric Fusy, O. Gandouet, and F. Meunier. Hyperloglog: The
+ *     analysis of a near-optimal cardinality estimation algorithm.
+ *
+ * The representation used by Redis is the following:
+ *
+ * +--------+--------+--------+------//      //--+----------+------+-----+
+ * |11000000|22221111|33333322|55444444 ....     | uint64_t | HYLL | Ver |
+ * +--------+--------+--------+------//      //--+----------+------+-----+
+ *
+ * The 6 bits counters are encoded one after the other starting from the
+ * LSB to the MSB, and using the next bytes as needed.
+ *
+ * At the end of the 16k counters, there is an additional 64 bit integer
+ * stored in little endian format with the latest cardinality computed that
+ * can be reused if the data structure was not modified since the last
+ * computation (this is useful because there are high probabilities that
+ * HLLADD operations don't modify the actual data structure and hence the
+ * approximated cardinality).
+ *
+ * After the cached cardinality there are 4 bytes of magic set to the
+ * string "HYLL", and a 4 bytes version field that is reserved for
+ * future uses and is currently set to 0.
+ *
+ * When the most significant bit in the most significant byte of the cached
+ * cardinality is set, it means that the data structure was modified and
+ * we can't reuse the cached value that must be recomputed. */
+
+#define REDIS_HLL_P 14 /* The greater is P, the smaller the error. */
+#define REDIS_HLL_REGISTERS (1<<REDIS_HLL_P) /* With P=14, 16384 registers. */
+#define REDIS_HLL_P_MASK (REDIS_HLL_REGISTERS-1) /* Mask to index register. */
+#define REDIS_HLL_BITS 6 /* Enough to count up to 63 leading zeroes. */
+#define REDIS_HLL_REGISTER_MAX ((1<<REDIS_HLL_BITS)-1)
+/* Note: REDIS_HLL_SIZE define has a final "+8" since we store a 64 bit
+ * integer at the end of the HyperLogLog structure to cache the cardinality. */
+#define REDIS_HLL_SIZE ((REDIS_HLL_REGISTERS*REDIS_HLL_BITS+7)/8)+8+8
+
+/* =========================== Low level bit macros ========================= */
+
+/* We need to get and set 6 bit counters in an array of 8 bit bytes.
+ * We use macros to make sure the code is inlined since speed is critical
+ * especially in order to compute the approximated cardinality in
+ * HLLCOUNT where we need to access all the registers at once.
+ * For the same reason we also want to avoid conditionals in this code path.
+ *
+ * +--------+--------+--------+------//
+ * |11000000|22221111|33333322|55444444
+ * +--------+--------+--------+------//
+ *
+ * Note: in the above representation the most significant bit (MSB)
+ * of every byte is on the left. We start using bits from the LSB to MSB,
+ * and so forth passing to the next byte.
+ *
+ * Example, we want to access to counter at pos = 1 ("111111" in the
+ * illustration above).
+ *
+ * The index of the first byte b0 containing our data is:
+ *
+ *  b0 = 6 * pos / 8 = 0
+ *
+ *   +--------+
+ *   |11000000|  <- Our byte at b0
+ *   +--------+
+ *
+ * The position of the first bit (counting from the LSB = 0) in the byte
+ * is given by:
+ *
+ *  fb = 6 * pos % 8 -> 6
+ *
+ * Right shift b0 of 'fb' bits.
+ *
+ *   +--------+
+ *   |11000000|  <- Initial value of b0
+ *   |00000011|  <- After right shift of 6 pos.
+ *   +--------+
+ *
+ * Left shift b1 of bits 8-fb bits (2 bits)
+ *
+ *   +--------+
+ *   |22221111|  <- Initial value of b1
+ *   |22111100|  <- After left shift of 2 bits.
+ *   +--------+
+ *
+ * OR the two bits, and finally AND with 111111 (63 in decimal) to
+ * clean the higher order bits we are not interested in:
+ *
+ *   +--------+
+ *   |00000011|  <- b0 right shifted
+ *   |22111100|  <- b1 left shifted
+ *   |22111111|  <- b0 OR b1
+ *   |  111111|  <- (b0 OR b1) AND 63, our value.
+ *   +--------+
+ *
+ * We can try with a different example, like pos = 0. In this case
+ * the 6-bit counter is actually contained in a single byte.
+ *
+ *  b0 = 6 * pos / 8 = 0
+ *
+ *   +--------+
+ *   |11000000|  <- Our byte at b0
+ *   +--------+
+ *
+ *  fb = 6 * pos % 8 = 0
+ *
+ *  So we right shift of 0 bits (no shift in practice) and
+ *  left shift the next byte of 8 bits, even if we don't use it,
+ *  but this has the effect of clearing the bits so the result
+ *  will not be affacted after the OR.
+ *
+ * -------------------------------------------------------------------------
+ *
+ * Setting the register is a bit more complex, let's assume that 'val'
+ * is the value we want to set, already in the right range.
+ *
+ * We need two steps, in one we need to clear the bits, and in the other
+ * we need to bitwise-OR the new bits.
+ *
+ * Let's try with 'pos' = 1, so our first byte at 'b' is 0,
+ *
+ * "fb" is 6 in this case.
+ *
+ *   +--------+
+ *   |11000000|  <- Our byte at b0
+ *   +--------+
+ *
+ * To create a AND-mask to clear the bits about this position, we just
+ * initialize the mask with the value 63, left shift it of "fs" bits,
+ * and finally invert the result.
+ *
+ *   +--------+
+ *   |00111111|  <- "mask" starts at 63
+ *   |11000000|  <- "mask" after left shift of "ls" bits.
+ *   |00111111|  <- "mask" after invert.
+ *   +--------+
+ *
+ * Now we can bitwise-AND the byte at "b" with the mask, and bitwise-OR
+ * it with "val" left-shifted of "ls" bits to set the new bits.
+ *
+ * Now let's focus on the next byte b1:
+ *
+ *   +--------+
+ *   |22221111|  <- Initial value of b1
+ *   +--------+
+ *
+ * To build the AND mask we start again with the 63 value, right shift
+ * it by 8-fb bits, and invert it.
+ *
+ *   +--------+
+ *   |00111111|  <- "mask" set at 2&6-1
+ *   |00001111|  <- "mask" after the right shift by 8-fb = 2 bits
+ *   |11110000|  <- "mask" after bitwise not.
+ *   +--------+
+ *
+ * Now we can mask it with b+1 to clear the old bits, and bitwise-OR
+ * with "val" left-shifted by "rs" bits to set the new value.
+ */
+
+/* Note: if we access the last counter, we will also access the b+1 byte
+ * that is out of the array, but sds strings always have an implicit null
+ * term, so the byte exists, and we can skip the conditional (or the need
+ * to allocate 1 byte more explicitly). */
+
+/* Store the value of the register at position 'regnum' into variable 'target'.
+ * 'p' is an array of unsigned bytes. */
+#define HLL_GET_REGISTER(target,p,regnum) do { \
+    uint8_t *_p = (uint8_t*) p; \
+    unsigned long _byte = regnum*REDIS_HLL_BITS/8; \
+    unsigned long _fb = regnum*REDIS_HLL_BITS&7; \
+    unsigned long _fb8 = 8 - _fb; \
+    unsigned long b0 = _p[_byte]; \
+    unsigned long b1 = _p[_byte+1]; \
+    target = ((b0 >> _fb) | (b1 << _fb8)) & REDIS_HLL_REGISTER_MAX; \
+} while(0)
+
+/* Set the value of the register at position 'regnum' to 'val'.
+ * 'p' is an array of unsigned bytes. */
+#define HLL_SET_REGISTER(p,regnum,val) do { \
+    uint8_t *_p = (uint8_t*) p; \
+    unsigned long _byte = regnum*REDIS_HLL_BITS/8; \
+    unsigned long _fb = regnum*REDIS_HLL_BITS&7; \
+    unsigned long _fb8 = 8 - _fb; \
+    unsigned long _v = val; \
+    _p[_byte] &= ~(REDIS_HLL_REGISTER_MAX << _fb); \
+    _p[_byte] |= _v << _fb; \
+    _p[_byte+1] &= ~(REDIS_HLL_REGISTER_MAX >> _fb8); \
+    _p[_byte+1] |= _v >> _fb8; \
+} while(0)
+
+/* ========================= HyperLogLog algorithm  ========================= */
+
+/* Our hash function is MurmurHash2, 64 bit version.
+ * It was modified for Redis in order to provide the same result in
+ * big and little endian archs (endian neutral). */
+uint64_t MurmurHash64A (const void * key, int len, unsigned int seed) {
+    const uint64_t m = 0xc6a4a7935bd1e995;
+    const int r = 47;
+    uint64_t h = seed ^ (len * m);
+    const uint8_t *data = (const uint8_t *)key;
+    const uint8_t *end = data + (len-(len&7));
+
+    while(data != end) {
+        uint64_t k;
+
+#if (BYTE_ORDER == LITTLE_ENDIAN)
+        k = *((uint64_t*)data);
+#else
+        k = (uint64_t) data[0];
+        k |= (uint64_t) data[1] << 8;
+        k |= (uint64_t) data[2] << 16;
+        k |= (uint64_t) data[3] << 24;
+        k |= (uint64_t) data[4] << 32;
+        k |= (uint64_t) data[5] << 40;
+        k |= (uint64_t) data[6] << 48;
+        k |= (uint64_t) data[7] << 56;
+#endif
+
+        k *= m;
+        k ^= k >> r;
+        k *= m;
+        h ^= k;
+        h *= m;
+        data += 8;
+    }
+
+    switch(len & 7) {
+    case 7: h ^= (uint64_t)data[6] << 48;
+    case 6: h ^= (uint64_t)data[5] << 40;
+    case 5: h ^= (uint64_t)data[4] << 32;
+    case 4: h ^= (uint64_t)data[3] << 24;
+    case 3: h ^= (uint64_t)data[2] << 16;
+    case 2: h ^= (uint64_t)data[1] << 8;
+    case 1: h ^= (uint64_t)data[0];
+            h *= m;
+    };
+
+    h ^= h >> r;
+    h *= m;
+    h ^= h >> r;
+    return h;
+}
+
+/* "Add" the element in the hyperloglog data structure.
+ * Actually nothing is added, but the max 0 pattern counter of the subset
+ * the element belongs to is incremented if needed.
+ *
+ * 'registers' is expected to have room for REDIS_HLL_REGISTERS plus an
+ * additional byte on the right. This requirement is met by sds strings
+ * automatically since they are implicitly null terminated.
+ *
+ * The function always succeed, however if as a result of the operation
+ * the approximated cardinality changed, 1 is returned. Otherwise 0
+ * is returned. */
+int hllAdd(uint8_t *registers, unsigned char *ele, size_t elesize) {
+    uint64_t hash, bit, index;
+    uint8_t oldcount, count;
+
+    /* Count the number of zeroes starting from bit REDIS_HLL_REGISTERS
+     * (that is a power of two corresponding to the first bit we don't use
+     * as index). The max run can be 64-P+1 bits.
+     *
+     * Note that the final "1" ending the sequence of zeroes must be
+     * included in the count, so if we find "001" the count is 3, and
+     * the smallest count possible is no zeroes at all, just a 1 bit
+     * at the first position, that is a count of 1.
+     *
+     * This may sound like inefficient, but actually in the average case
+     * there are high probabilities to find a 1 after a few iterations. */
+    hash = MurmurHash64A(ele,elesize,0xadc83b19ULL);
+    hash |= ((uint64_t)1<<63); /* Make sure the loop terminates. */
+    bit = REDIS_HLL_REGISTERS; /* First bit not used to address the register. */
+    count = 1; /* Initialized to 1 since we count the "00000...1" pattern. */
+    while((hash & bit) == 0) {
+        count++;
+        bit <<= 1;
+    }
+
+    /* Update the register if this element produced a longer run of zeroes. */
+    index = hash & REDIS_HLL_P_MASK; /* Index a register inside registers. */
+    HLL_GET_REGISTER(oldcount,registers,index);
+    if (count > oldcount) {
+        HLL_SET_REGISTER(registers,index,count);
+        return 1;
+    } else {
+        return 0;
+    }
+}
+
+/* Return the approximated cardinality of the set based on the armonic
+ * mean of the registers values. */
+uint64_t hllCount(uint8_t *registers) {
+    double m = REDIS_HLL_REGISTERS;
+    double alpha = 0.7213/(1+1.079/m);
+    double E = 0;
+    int ez = 0; /* Number of registers equal to 0. */
+    int j;
+
+    /* We precompute 2^(-reg[j]) in a small table in order to
+     * speedup the computation of SUM(2^-register[0..i]). */
+    static int initialized = 0;
+    static double PE[64];
+    if (!initialized) {
+        PE[0] = 1; /* 2^(-reg[j]) is 1 when m is 0. */
+        for (j = 1; j < 64; j++) {
+            /* 2^(-reg[j]) is the same as 1/2^reg[j]. */
+            PE[j] = 1.0/(1ULL << j);
+        }
+        initialized = 1;
+    }
+
+    /* Compute SUM(2^-register[0..i]).
+     * Redis default is to use 16384 registers 6 bits each. The code works
+     * with other values by modifying the defines, but for our target value
+     * we take a faster path with unrolled loops. */
+    if (REDIS_HLL_REGISTERS == 16384 && REDIS_HLL_BITS == 6) {
+        uint8_t *r = registers;
+        unsigned long r0, r1, r2, r3, r4, r5, r6, r7, r8, r9,
+                      r10, r11, r12, r13, r14, r15;
+        for (j = 0; j < 1024; j++) {
+            /* Handle 16 registers per iteration. */
+            r0 = r[0] & 63; if (r0 == 0) ez++;
+            r1 = (r[0] >> 6 | r[1] << 2) & 63; if (r1 == 0) ez++;
+            r2 = (r[1] >> 4 | r[2] << 4) & 63; if (r2 == 0) ez++;
+            r3 = (r[2] >> 2) & 63; if (r3 == 0) ez++;
+            r4 = r[3] & 63; if (r4 == 0) ez++;
+            r5 = (r[3] >> 6 | r[4] << 2) & 63; if (r5 == 0) ez++;
+            r6 = (r[4] >> 4 | r[5] << 4) & 63; if (r6 == 0) ez++;
+            r7 = (r[5] >> 2) & 63; if (r7 == 0) ez++;
+            r8 = r[6] & 63; if (r8 == 0) ez++;
+            r9 = (r[6] >> 6 | r[7] << 2) & 63; if (r9 == 0) ez++;
+            r10 = (r[7] >> 4 | r[8] << 4) & 63; if (r10 == 0) ez++;
+            r11 = (r[8] >> 2) & 63; if (r11 == 0) ez++;
+            r12 = r[9] & 63; if (r12 == 0) ez++;
+            r13 = (r[9] >> 6 | r[10] << 2) & 63; if (r13 == 0) ez++;
+            r14 = (r[10] >> 4 | r[11] << 4) & 63; if (r14 == 0) ez++;
+            r15 = (r[11] >> 2) & 63; if (r15 == 0) ez++;
+
+            /* Additional parens will allow the compiler to optimize the
+             * code more with a loss of precision that is not very relevant
+             * here (floating point math is not commutative!). */
+            E += (PE[r0] + PE[r1]) + (PE[r2] + PE[r3]) + (PE[r4] + PE[r5]) +
+                 (PE[r6] + PE[r7]) + (PE[r8] + PE[r9]) + (PE[r10] + PE[r11]) +
+                 (PE[r12] + PE[r13]) + (PE[r14] + PE[r15]);
+            r += 12;
+        }
+    } else {
+        for (j = 0; j < REDIS_HLL_REGISTERS; j++) {
+            unsigned long reg;
+
+            HLL_GET_REGISTER(reg,registers,j);
+            if (reg == 0) {
+                ez++;
+                E += 1; /* 2^(-reg[j]) is 1 when m is 0. */
+            } else {
+                E += PE[reg]; /* Precomputed 2^(-reg[j]). */
+            }
+        }
+    }
+    /* Muliply the inverse of E for alpha_m * m^2 to have the raw estimate. */
+    E = (1/E)*alpha*m*m;
+
+    /* Use the LINEARCOUNTING algorithm for small cardinalities.
+     * For larger values but up to 72000 HyperLogLog raw approximation is
+     * used since linear counting error starts to increase. However HyperLogLog
+     * shows a strong bias in the range 2.5*16384 - 72000, so we try to
+     * compensate for it. */
+    if (E < m*2.5 && ez != 0) {
+        E = m*log(m/ez); /* LINEARCOUNTING() */
+    } else if (m == 16384 && E < 72000) {
+        /* We did polynomial regression of the bias for this range, this
+         * way we can compute the bias for a given cardinality and correct
+         * according to it. Only apply the correction for P=14 that's what
+         * we use and the value the correction was verified with. */
+        double bias = 5.9119*1.0e-18*(E*E*E*E)
+                      -1.4253*1.0e-12*(E*E*E)+
+                      1.2940*1.0e-7*(E*E)
+                      -5.2921*1.0e-3*E+
+                      83.3216;
+        E -= E*(bias/100);
+    }
+    /* We don't apply the correction for E > 1/30 of 2^32 since we use
+     * a 64 bit function and 6 bit counters. To apply the correction for
+     * 1/30 of 2^64 is not needed since it would require a huge set
+     * to approach such a value. */
+    return (uint64_t) E;
+}
+
+/* ========================== HyperLogLog commands ========================== */
+
+/* An HyperLogLog object is a string with space for 16k 6-bit integers,
+ * a cached 64 bit cardinality value, and a 4 byte "magic" and additional
+ * 4 bytes for version reserved for future use. */
+robj *createHLLObject(void) {
+    robj *o;
+    char *p;
+
+    /* Create a string of the right size filled with zero bytes.
+     * Note that the cached cardinality is set to 0 as a side effect
+     * that is exactly the cardinality of an empty HLL. */
+    o = createObject(REDIS_STRING,sdsnewlen(NULL,REDIS_HLL_SIZE));
+    p = o->ptr;
+    memcpy(p+REDIS_HLL_SIZE-8,"HYLL",4);
+    return o;
+}
+
+/* Check if the object is a String of REDIS_HLL_SIZE bytes.
+ * Return REDIS_OK if this is true, otherwise reply to the client
+ * with an error and return REDIS_ERR. */
+int isHLLObjectOrReply(redisClient *c, robj *o) {
+    /* Key exists, check type */
+    if (checkType(c,o,REDIS_STRING))
+        return REDIS_ERR; /* Error already sent. */
+
+    /* If this is a string representing an HLL, the size should match
+     * exactly. */
+    if (stringObjectLen(o) != REDIS_HLL_SIZE) {
+        addReplySds(c,
+            sdsnew("-WRONGTYPE Key is not a valid "
+                   "HyperLogLog string value.\r\n"));
+        return REDIS_ERR;
+    }
+    return REDIS_OK;
+}
+
+/* PFADD var ele ele ele ... ele => :0 or :1 */
+void pfaddCommand(redisClient *c) {
+    robj *o = lookupKeyWrite(c->db,c->argv[1]);
+    uint8_t *registers;
+    int updated = 0, j;
+
+    if (o == NULL) {
+        /* Create the key with a string value of the exact length to
+         * hold our HLL data structure. sdsnewlen() when NULL is passed
+         * is guaranteed to return bytes initialized to zero. */
+        o = createHLLObject();
+        dbAdd(c->db,c->argv[1],o);
+        updated++;
+    } else {
+        if (isHLLObjectOrReply(c,o) != REDIS_OK) return;
+        o = dbUnshareStringValue(c->db,c->argv[1],o);
+    }
+    /* Perform the low level ADD operation for every element. */
+    registers = o->ptr;
+    for (j = 2; j < c->argc; j++) {
+        if (hllAdd(registers, (unsigned char*)c->argv[j]->ptr,
+                sdslen(c->argv[j]->ptr)))
+        {
+            updated++;
+        }
+    }
+    if (updated) {
+        signalModifiedKey(c->db,c->argv[1]);
+        notifyKeyspaceEvent(REDIS_NOTIFY_STRING,"pfadd",c->argv[1],c->db->id);
+        server.dirty++;
+        /* Invalidate the cached cardinality. */
+        registers[REDIS_HLL_SIZE-9] |= (1<<7);
+    }
+    addReply(c, updated ? shared.cone : shared.czero);
+}
+
+/* PFCOUNT var -> approximated cardinality of set. */
+void pfcountCommand(redisClient *c) {
+    robj *o = lookupKeyRead(c->db,c->argv[1]);
+    uint8_t *registers;
+    uint64_t card;
+
+    if (o == NULL) {
+        /* No key? Cardinality is zero since no element was added, otherwise
+         * we would have a key as HLLADD creates it as a side effect. */
+        addReply(c,shared.czero);
+    } else {
+        if (isHLLObjectOrReply(c,o) != REDIS_OK) return;
+        o = dbUnshareStringValue(c->db,c->argv[1],o);
+
+        /* Check if the cached cardinality is valid. */
+        registers = o->ptr;
+        if ((registers[REDIS_HLL_SIZE-9] & (1<<7)) == 0) {
+            /* Just return the cached value. */
+            card = (uint64_t)registers[REDIS_HLL_SIZE-16];
+            card |= (uint64_t)registers[REDIS_HLL_SIZE-15] << 8;
+            card |= (uint64_t)registers[REDIS_HLL_SIZE-14] << 16;
+            card |= (uint64_t)registers[REDIS_HLL_SIZE-13] << 24;
+            card |= (uint64_t)registers[REDIS_HLL_SIZE-12] << 32;
+            card |= (uint64_t)registers[REDIS_HLL_SIZE-11] << 40;
+            card |= (uint64_t)registers[REDIS_HLL_SIZE-10] << 48;
+            card |= (uint64_t)registers[REDIS_HLL_SIZE-9] << 56;
+        } else {
+            /* Recompute it and update the cached value. */
+            card = hllCount(registers);
+            registers[REDIS_HLL_SIZE-16] = card & 0xff;
+            registers[REDIS_HLL_SIZE-15] = (card >> 8) & 0xff;
+            registers[REDIS_HLL_SIZE-14] = (card >> 16) & 0xff;
+            registers[REDIS_HLL_SIZE-13] = (card >> 24) & 0xff;
+            registers[REDIS_HLL_SIZE-12] = (card >> 32) & 0xff;
+            registers[REDIS_HLL_SIZE-11] = (card >> 40) & 0xff;
+            registers[REDIS_HLL_SIZE-10] = (card >> 48) & 0xff;
+            registers[REDIS_HLL_SIZE-9] = (card >> 56) & 0xff;
+            /* This is not considered a read-only command even if the
+             * data structure is not modified, since the cached value
+             * may be modified and given that the HLL is a Redis string
+             * we need to propagate the change. */
+            signalModifiedKey(c->db,c->argv[1]);
+            server.dirty++;
+        }
+        addReplyLongLong(c,card);
+    }
+}
+
+/* PFMERGE dest src1 src2 src3 ... srcN => OK */
+void pfmergeCommand(redisClient *c) {
+    uint8_t max[REDIS_HLL_REGISTERS];
+    uint8_t *registers;
+    int j, i;
+
+    /* Compute an HLL with M[i] = MAX(M[i]_j).
+     * We we the maximum into the max array of registers. We'll write
+     * it to the target variable later. */
+    memset(max,0,sizeof(max));
+    for (j = 1; j < c->argc; j++) {
+        uint8_t val;
+
+        /* Check type and size. */
+        robj *o = lookupKeyRead(c->db,c->argv[j]);
+        if (o == NULL) continue; /* Assume empty HLL for non existing var. */
+        if (isHLLObjectOrReply(c,o) != REDIS_OK) return;
+
+        /* Merge with this HLL with our 'max' HHL by setting max[i]
+         * to MAX(max[i],hll[i]). */
+        registers = o->ptr;
+        for (i = 0; i < REDIS_HLL_REGISTERS; i++) {
+            HLL_GET_REGISTER(val,registers,i);
+            if (val > max[i]) max[i] = val;
+        }
+    }
+
+    /* Create / unshare the destination key's value if needed. */
+    robj *o = lookupKeyRead(c->db,c->argv[1]);
+    if (o == NULL) {
+        /* Create the key with a string value of the exact length to
+         * hold our HLL data structure. sdsnewlen() when NULL is passed
+         * is guaranteed to return bytes initialized to zero. */
+        o = createHLLObject();
+        dbAdd(c->db,c->argv[1],o);
+    } else {
+        /* If key exists we are sure it's of the right type/size
+         * since we checked when merging the different HLLs, so we
+         * don't check again. */
+        o = dbUnshareStringValue(c->db,c->argv[1],o);
+    }
+
+    /* Write the resulting HLL to the destination HLL registers and
+     * invalidate the cached value. */
+    registers = o->ptr;
+    for (j = 0; j < REDIS_HLL_REGISTERS; j++) {
+        HLL_SET_REGISTER(registers,j,max[j]);
+    }
+    registers[REDIS_HLL_SIZE-9] |= (1<<7);
+
+    signalModifiedKey(c->db,c->argv[1]);
+    /* We generate an HLLADD event for HLLMERGE for semantical simplicity
+     * since in theory this is a mass-add of elements. */
+    notifyKeyspaceEvent(REDIS_NOTIFY_STRING,"pfadd",c->argv[1],c->db->id);
+    server.dirty++;
+    addReply(c,shared.ok);
+}
+
+/* ========================== Testing / Debugging  ========================== */
+
+/* PFSELFTEST
+ * This command performs a self-test of the HLL registers implementation.
+ * Something that is not easy to test from within the outside. */
+#define REDIS_HLL_TEST_CYCLES 1000
+void pfselftestCommand(redisClient *c) {
+    int j, i;
+    sds bitcounters = sdsnewlen(NULL,REDIS_HLL_SIZE);
+    uint8_t bytecounters[REDIS_HLL_REGISTERS];
+
+    /* Test 1: access registers.
+     * The test is conceived to test that the different counters of our data
+     * structure are accessible and that setting their values both result in
+     * the correct value to be retained and not affect adjacent values. */
+    for (j = 0; j < REDIS_HLL_TEST_CYCLES; j++) {
+        /* Set the HLL counters and an array of unsigned byes of the
+         * same size to the same set of random values. */
+        for (i = 0; i < REDIS_HLL_REGISTERS; i++) {
+            unsigned int r = rand() & REDIS_HLL_REGISTER_MAX;
+
+            bytecounters[i] = r;
+            HLL_SET_REGISTER(bitcounters,i,r);
+        }
+        /* Check that we are able to retrieve the same values. */
+        for (i = 0; i < REDIS_HLL_REGISTERS; i++) {
+            unsigned int val;
+
+            HLL_GET_REGISTER(val,bitcounters,i);
+            if (val != bytecounters[i]) {
+                addReplyErrorFormat(c,
+                    "TESTFAILED Register %d should be %d but is %d",
+                    i, (int) bytecounters[i], (int) val);
+                goto cleanup;
+            }
+        }
+    }
+
+    /* Test 2: approximation error.
+     * The test is adds unique elements and check that the estimated value
+     * is always reasonable bounds.
+     * 
+     * We check that the error is smaller than 4 times than the expected
+     * standard error, to make it very unlikely for the test to fail because
+     * of a "bad" run. */
+    memset(bitcounters,0,REDIS_HLL_SIZE);
+    double relerr = 1.04/sqrt(REDIS_HLL_REGISTERS);
+    int64_t checkpoint = 1000;
+    uint64_t seed = (uint64_t)rand() | (uint64_t)rand() << 32;
+    uint64_t ele;
+    for (j = 1; j <= 10000000; j++) {
+        ele = j ^ seed;
+        hllAdd((uint8_t*)bitcounters,(unsigned char*)&ele,sizeof(ele));
+        if (j == checkpoint) {
+            int64_t abserr = checkpoint-
+                             (int64_t)hllCount((uint8_t*)bitcounters);
+            if (abserr < 0) abserr = -abserr;
+            if (abserr > (uint64_t)(relerr*4*checkpoint)) {
+                addReplyErrorFormat(c,
+                    "TESTFAILED Too big error. card:%llu abserr:%llu",
+                    (unsigned long long) checkpoint,
+                    (unsigned long long) abserr);
+                goto cleanup;
+            }
+            checkpoint *= 10;
+        }
+    }
+
+    /* Success! */
+    addReply(c,shared.ok);
+
+cleanup:
+    sdsfree(bitcounters);
+}
+
+/* PFGETREG
+ * Return the registers values of the specified HLL. */
+void pfgetregCommand(redisClient *c) {
+    robj *o = lookupKeyRead(c->db,c->argv[1]);
+    uint8_t *registers;
+    int j;
+
+    if (o == NULL) {
+        addReplyError(c,"The specified key does not exist");
+        return;
+    } else {
+        if (isHLLObjectOrReply(c,o) != REDIS_OK) return;
+
+        registers = o->ptr;
+        addReplyMultiBulkLen(c,REDIS_HLL_REGISTERS);
+        for (j = 0; j < REDIS_HLL_REGISTERS; j++) {
+            uint8_t val;
+
+            HLL_GET_REGISTER(val,registers,j);
+            addReplyLongLong(c,val);
+        }
+    }
+}

src/networking.c

@@ -319,6 +319,11 @@ void addReply(redisClient *c, robj *obj) {
         if (_addReplyToBuffer(c,obj->ptr,sdslen(obj->ptr)) != REDIS_OK)
             _addReplyObjectToList(c,obj);
         decrRefCount(obj);
+    } else if (obj->encoding == REDIS_ENCODING_LZF) {
+        obj = getDecodedObject(obj);
+        if (_addReplyToBuffer(c,obj->ptr,sdslen(obj->ptr)) != REDIS_OK)
+            _addReplyObjectToList(c,obj);
+        decrRefCount(obj);
     } else {
         redisPanic("Wrong obj->encoding in addReply()");
     }
@@ -488,7 +493,7 @@ void addReplyBulkLen(redisClient *c, robj *obj) {
 
     if (sdsEncodedObject(obj)) {
         len = sdslen(obj->ptr);
-    } else {
+    } else if (obj->encoding == REDIS_ENCODING_INT) {
         long n = (long)obj->ptr;
 
         /* Compute how many bytes will take this integer as a radix 10 string */
@@ -500,6 +505,9 @@ void addReplyBulkLen(redisClient *c, robj *obj) {
         while((n = n/10) != 0) {
             len++;
         }
+    } else {
+        /* LZF and others not handled explicitly. */
+        len = stringObjectLen(obj);
     }
 
     if (len < REDIS_SHARED_BULKHDR_LEN)
@@ -587,7 +595,7 @@ void acceptTcpHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
     REDIS_NOTUSED(privdata);
 
     cfd = anetTcpAccept(server.neterr, fd, cip, sizeof(cip), &cport);
-    if (cfd == AE_ERR) {
+    if (cfd == ANET_ERR) {
         redisLog(REDIS_WARNING,"Accepting client connection: %s", server.neterr);
         return;
     }

src/object.c

@@ -29,6 +29,7 @@
  */
 
 #include "redis.h"
+#include "lzf.h"    /* LZF compression library */
 #include <math.h>
 #include <ctype.h>
 
@@ -40,7 +41,7 @@ robj *createObject(int type, void *ptr) {
     o->refcount = 1;
 
     /* Set the LRU to the current lruclock (minutes resolution). */
-    o->lru = server.lruclock;
+    o->lru = LRU_CLOCK();
     return o;
 }
 
@@ -61,7 +62,7 @@ robj *createEmbeddedStringObject(char *ptr, size_t len) {
     o->encoding = REDIS_ENCODING_EMBSTR;
     o->ptr = sh+1;
     o->refcount = 1;
-    o->lru = server.lruclock;
+    o->lru = LRU_CLOCK();
 
     sh->len = len;
     sh->free = 0;
@@ -149,6 +150,10 @@ robj *dupStringObject(robj *o) {
         d->encoding = REDIS_ENCODING_INT;
         d->ptr = o->ptr;
         return d;
+    case REDIS_ENCODING_LZF:
+        d = createObject(REDIS_STRING,sdsdup(o->ptr));
+        d->encoding = REDIS_ENCODING_LZF;
+        return d;
     default:
         redisPanic("Wrong encoding.");
         break;
@@ -210,7 +215,8 @@ robj *createZsetZiplistObject(void) {
 }
 
 void freeStringObject(robj *o) {
-    if (o->encoding == REDIS_ENCODING_RAW) {
+    if (o->encoding == REDIS_ENCODING_RAW ||
+        o->encoding == REDIS_ENCODING_LZF) {
         sdsfree(o->ptr);
     }
 }
@@ -327,78 +333,138 @@ int checkType(redisClient *c, robj *o, int type) {
 
 int isObjectRepresentableAsLongLong(robj *o, long long *llval) {
     redisAssertWithInfo(NULL,o,o->type == REDIS_STRING);
-    if (o->encoding == REDIS_ENCODING_INT) {
+    if (intEncodedObject(o)) {
         if (llval) *llval = (long) o->ptr;
         return REDIS_OK;
-    } else {
+    } else if (sdsEncodedObject(o)) {
         return string2ll(o->ptr,sdslen(o->ptr),llval) ? REDIS_OK : REDIS_ERR;
+    } else if (lzfEncodedObject(o)) {
+        int retval;
+        robj *decoded = getDecodedObject(o);
+        retval = string2ll(o->ptr,sdslen(o->ptr),llval) ? REDIS_OK : REDIS_ERR;
+        decrRefCount(decoded);
+        return retval;
+    } else {
+        redisPanic("Unknown encoding.");
     }
 }
 
-/* Try to encode a string object in order to save space */
+/* Try to encode a string object in order to save space. */
+#define LZF_COMPR_STATIC_BUF (1024*32)
 robj *tryObjectEncoding(robj *o) {
     long value;
     sds s = o->ptr;
     size_t len;
 
-    if (o->encoding == REDIS_ENCODING_INT)
-        return o; /* Already encoded */
+    /* Make sure this is a string object, the only type we encode
+     * in this function. Other types use encoded memory efficient
+     * representations but are handled by the commands implementing
+     * the type. */
+    redisAssertWithInfo(NULL,o,o->type == REDIS_STRING);
+
+    /* We try some specialized encoding only for objects that are
+     * RAW or EMBSTR encoded, in other words objects that are still
+     * in represented by an actually array of chars. */
+    if (!sdsEncodedObject(o)) return o;
 
     /* It's not safe to encode shared objects: shared objects can be shared
-     * everywhere in the "object space" of Redis. Encoded objects can only
-     * appear as "values" (and not, for instance, as keys) */
+     * everywhere in the "object space" of Redis and may end in places where
+     * they are not handled. We handle them only as values in the keyspace. */
      if (o->refcount > 1) return o;
 
-    /* Currently we try to encode only strings */
-    redisAssertWithInfo(NULL,o,o->type == REDIS_STRING);
-
     /* Check if we can represent this string as a long integer.
      * Note that we are sure that a string larger than 21 chars is not
-     * representable as a 64 bit integer. */
+     * representable as a 32 nor 64 bit integer. */
     len = sdslen(s);
-    if (len > 21 || !string2l(s,len,&value)) {
-        /* Integer encoding not possible. Check if we can use EMBSTR. */
-        if (sdslen(s) <= REDIS_ENCODING_EMBSTR_SIZE_LIMIT) {
-            robj *emb = createEmbeddedStringObject(s,sdslen(s));
+    if (len <= 21 && string2l(s,len,&value)) {
+        /* This object is encodable as a long. Try to use a shared object.
+         * Note that we avoid using shared integers when maxmemory is used
+         * because every object needs to have a private LRU field for the LRU
+         * algorithm to work well. */
+        if (server.maxmemory == 0 &&
+            value >= 0 &&
+            value < REDIS_SHARED_INTEGERS)
+        {
             decrRefCount(o);
-            return emb;
+            incrRefCount(shared.integers[value]);
+            return shared.integers[value];
         } else {
-            /* We can't encode the object...
-             *
-             * Do the last try, and at least optimize the SDS string inside
-             * the string object to require little space, in case there
-             * is more than 10% of free space at the end of the SDS string.
-             *
-             * We do that only for relatively large strings as this branch
-             * is only entered if the length of the string is greater than
-             * REDIS_ENCODING_EMBSTR_SIZE_LIMIT. */
-            if (o->encoding == REDIS_ENCODING_RAW &&
-                sdsavail(s) > len/10)
-            {
-                o->ptr = sdsRemoveFreeSpace(o->ptr);
-            }
-            /* Return the original object. */
+            if (o->encoding == REDIS_ENCODING_RAW) sdsfree(o->ptr);
+            o->encoding = REDIS_ENCODING_INT;
+            o->ptr = (void*) value;
             return o;
         }
     }
 
-    /* Ok, this object can be encoded...
-     *
-     * Can I use a shared object? Only if the object is inside a given range
-     *
-     * Note that we also avoid using shared integers when maxmemory is used
-     * because every object needs to have a private LRU field for the LRU
-     * algorithm to work well. */
-    if (server.maxmemory == 0 && value >= 0 && value < REDIS_SHARED_INTEGERS) {
+    /* If the string is small and is still RAW encoded,
+     * try the EMBSTR encoding which is more efficient.
+     * In this representation the object and the SDS string are allocated
+     * in the same chunk of memory to save space and cache misses. */
+    if (len <= REDIS_ENCODING_EMBSTR_SIZE_LIMIT) {
+        robj *emb;
+
+        if (o->encoding == REDIS_ENCODING_EMBSTR) return o;
+        emb = createEmbeddedStringObject(s,sdslen(s));
         decrRefCount(o);
-        incrRefCount(shared.integers[value]);
-        return shared.integers[value];
-    } else {
-        if (o->encoding == REDIS_ENCODING_RAW) sdsfree(o->ptr);
-        o->encoding = REDIS_ENCODING_INT;
-        o->ptr = (void*) value;
-        return o;
+        return emb;
     }
+
+    /* Try LZF compression for objects up to server.mem_compression_max_size
+     * and greater than REDIS_ENCODING_EMBSTR_SIZE_LIMIT. */
+    if (server.mem_compression && len <= server.mem_compression_max_size) {
+        /* Allocate four more bytes in our buffer since we need to store
+         * the size of the compressed string as header. */
+        unsigned char compr_static[LZF_COMPR_STATIC_BUF];
+        unsigned char *compr = compr_static;
+        size_t comprlen, outlen;
+
+        /* Save want to save at least 25% of memory for this to make sense. */
+        outlen = len-4-(len/4);
+
+        /* Use an heap allocated buffer if the output is too big for our
+         * static buffer. We use the trick to directly allocating an empty
+         * SDS string so we can use it directly to create the object later. */
+        if (outlen+4 > LZF_COMPR_STATIC_BUF)
+            compr = (unsigned char*) sdsnewlen(NULL,outlen+4);
+        comprlen = lzf_compress(s,len,compr+4,outlen);
+        if (comprlen != 0) {
+            /* Object successfully compressed within the required space. */
+            compr[0] = len & 0xff;
+            compr[1] = (len >> 8) & 0xff;
+            compr[2] = (len >> 16) & 0xff;
+            compr[3] = (len >> 24) & 0xff;
+            if (o->encoding == REDIS_ENCODING_RAW) sdsfree(o->ptr);
+            o->encoding = REDIS_ENCODING_LZF;
+            if (compr == compr_static) {
+                /* When compressing to the static buffer we have to
+                 * generate the SDS string here. */
+                o->ptr = sdsnewlen(compr,comprlen+4);
+            } else {
+                /* Already an SDS, use it. */
+                o->ptr = compr;
+            }
+            return o;
+        }
+        if (compr != compr_static) sdsfree((char*)compr);
+    }
+
+    /* We can't encode the object...
+     *
+     * Do the last try, and at least optimize the SDS string inside
+     * the string object to require little space, in case there
+     * is more than 10% of free space at the end of the SDS string.
+     *
+     * We do that only for relatively large strings as this branch
+     * is only entered if the length of the string is greater than
+     * REDIS_ENCODING_EMBSTR_SIZE_LIMIT. */
+    if (o->encoding == REDIS_ENCODING_RAW &&
+        sdsavail(s) > len/10)
+    {
+        o->ptr = sdsRemoveFreeSpace(o->ptr);
+    }
+
+    /* Return the original object. */
+    return o;
 }
 
 /* Get a decoded version of an encoded object (returned as a new object).
@@ -410,12 +476,20 @@ robj *getDecodedObject(robj *o) {
         incrRefCount(o);
         return o;
     }
-    if (o->type == REDIS_STRING && o->encoding == REDIS_ENCODING_INT) {
+    if (o->type == REDIS_STRING && intEncodedObject(o)) {
         char buf[32];
 
         ll2string(buf,32,(long)o->ptr);
         dec = createStringObject(buf,strlen(buf));
         return dec;
+    } else if (o->type == REDIS_STRING && lzfEncodedObject(o)) {
+        int origlen = stringObjectLen(o);
+        sds orig = sdsnewlen(NULL,origlen);
+        unsigned char *p = o->ptr;
+
+        if (lzf_decompress(p+4,sdslen(o->ptr)-4,orig,origlen) == 0)
+            redisPanic("LZF error during object decoding.");
+        return createObject(REDIS_STRING,orig);
     } else {
         redisPanic("Unknown encoding type");
     }
@@ -436,8 +510,15 @@ int compareStringObjectsWithFlags(robj *a, robj *b, int flags) {
     redisAssertWithInfo(NULL,a,a->type == REDIS_STRING && b->type == REDIS_STRING);
     char bufa[128], bufb[128], *astr, *bstr;
     size_t alen, blen, minlen;
+    int decr = 0;
 
     if (a == b) return 0;
+
+    if (lzfEncodedObject(a) || lzfEncodedObject(b)) {
+        a = getDecodedObject(a);
+        b = getDecodedObject(b);
+        decr = 1;
+    }
     if (sdsEncodedObject(a)) {
         astr = a->ptr;
         alen = sdslen(astr);
@@ -462,6 +543,10 @@ int compareStringObjectsWithFlags(robj *a, robj *b, int flags) {
         if (cmp == 0) return alen-blen;
         return cmp;
     }
+    if (decr) {
+        decrRefCount(a);
+        decrRefCount(b);
+    }
 }
 
 /* Wrapper for compareStringObjectsWithFlags() using binary comparison. */
@@ -479,8 +564,7 @@ int collateStringObjects(robj *a, robj *b) {
  * this function is faster then checking for (compareStringObject(a,b) == 0)
  * because it can perform some more optimization. */
 int equalStringObjects(robj *a, robj *b) {
-    if (a->encoding == REDIS_ENCODING_INT &&
-        b->encoding == REDIS_ENCODING_INT){
+    if (intEncodedObject(a) && intEncodedObject(b)) {
         /* If both strings are integer encoded just check if the stored
          * long is the same. */
         return a->ptr == b->ptr;
@@ -489,13 +573,21 @@ int equalStringObjects(robj *a, robj *b) {
     }
 }
 
+/* Returns the original (uncompressed) size of an LZF encoded object.
+ * Only called by stringObjectLen() that should be the main interface. */
+size_t stringObjectUncompressedLen(robj *o) {
+    unsigned char *p = o->ptr;
+    return p[0] | (p[1]<<8) | (p[2]<<16) | (p[3]<<24);
+}
+
 size_t stringObjectLen(robj *o) {
     redisAssertWithInfo(NULL,o,o->type == REDIS_STRING);
     if (sdsEncodedObject(o)) {
         return sdslen(o->ptr);
+    } else if (o->encoding == REDIS_ENCODING_LZF) {
+        return stringObjectUncompressedLen(o);
     } else {
         char buf[32];
-
         return ll2string(buf,32,(long)o->ptr);
     }
 }
@@ -518,8 +610,14 @@ int getDoubleFromObject(robj *o, double *target) {
                 errno == EINVAL ||
                 isnan(value))
                 return REDIS_ERR;
-        } else if (o->encoding == REDIS_ENCODING_INT) {
+        } else if (intEncodedObject(o)) {
             value = (long)o->ptr;
+        } else if (lzfEncodedObject(o)) {
+            int retval;
+            o = getDecodedObject(o);
+            retval = getDoubleFromObject(o,target);
+            decrRefCount(o);
+            return retval;
         } else {
             redisPanic("Unknown string encoding");
         }
@@ -556,8 +654,14 @@ int getLongDoubleFromObject(robj *o, long double *target) {
             if (isspace(((char*)o->ptr)[0]) || eptr[0] != '\0' ||
                 errno == ERANGE || isnan(value))
                 return REDIS_ERR;
-        } else if (o->encoding == REDIS_ENCODING_INT) {
+        } else if (intEncodedObject(o)) {
             value = (long)o->ptr;
+        } else if (lzfEncodedObject(o)) {
+            int retval;
+            o = getDecodedObject(o);
+            retval = getLongDoubleFromObject(o,target);
+            decrRefCount(o);
+            return retval;
         } else {
             redisPanic("Unknown string encoding");
         }
@@ -594,8 +698,14 @@ int getLongLongFromObject(robj *o, long long *target) {
             if (isspace(((char*)o->ptr)[0]) || eptr[0] != '\0' ||
                 errno == ERANGE)
                 return REDIS_ERR;
-        } else if (o->encoding == REDIS_ENCODING_INT) {
+        } else if (intEncodedObject(o)) {
             value = (long)o->ptr;
+        } else if (lzfEncodedObject(o)) {
+            int retval;
+            o = getDecodedObject(o);
+            retval = getLongLongFromObject(o,target);
+            decrRefCount(o);
+            return retval;
         } else {
             redisPanic("Unknown string encoding");
         }
@@ -644,22 +754,24 @@ char *strEncoding(int encoding) {
     case REDIS_ENCODING_INTSET: return "intset";
     case REDIS_ENCODING_SKIPLIST: return "skiplist";
     case REDIS_ENCODING_EMBSTR: return "embstr";
+    case REDIS_ENCODING_LZF: return "lzf";
     default: return "unknown";
     }
 }
 
-/* Given an object returns the min number of seconds the object was never
+/* Given an object returns the min number of milliseconds the object was never
  * requested, using an approximated LRU algorithm. */
-unsigned long estimateObjectIdleTime(robj *o) {
-    if (server.lruclock >= o->lru) {
-        return (server.lruclock - o->lru) * REDIS_LRU_CLOCK_RESOLUTION;
+unsigned long long estimateObjectIdleTime(robj *o) {
+    unsigned long long lruclock = LRU_CLOCK();
+    if (lruclock >= o->lru) {
+        return (lruclock - o->lru) * REDIS_LRU_CLOCK_RESOLUTION;
     } else {
-        return ((REDIS_LRU_CLOCK_MAX - o->lru) + server.lruclock) *
+        return (lruclock + (REDIS_LRU_CLOCK_MAX - o->lru)) *
                     REDIS_LRU_CLOCK_RESOLUTION;
     }
 }
 
-/* This is a helper function for the DEBUG command. We need to lookup keys
+/* This is a helper function for the OBJECT command. We need to lookup keys
  * without any modification of LRU or other parameters. */
 robj *objectCommandLookup(redisClient *c, robj *key) {
     dictEntry *de;
@@ -691,7 +803,7 @@ void objectCommand(redisClient *c) {
     } else if (!strcasecmp(c->argv[1]->ptr,"idletime") && c->argc == 3) {
         if ((o = objectCommandLookupOrReply(c,c->argv[2],shared.nullbulk))
                 == NULL) return;
-        addReplyLongLong(c,estimateObjectIdleTime(o));
+        addReplyLongLong(c,estimateObjectIdleTime(o)/1000);
     } else {
         addReplyError(c,"Syntax error. Try OBJECT (refcount|encoding|idletime)");
     }

src/pubsub.c

@@ -50,7 +50,7 @@ int listMatchPubsubPattern(void *a, void *b) {
 /* Subscribe a client to a channel. Returns 1 if the operation succeeded, or
  * 0 if the client was already subscribed to that channel. */
 int pubsubSubscribeChannel(redisClient *c, robj *channel) {
-    struct dictEntry *de;
+    dictEntry *de;
     list *clients = NULL;
     int retval = 0;
 
@@ -80,7 +80,7 @@ int pubsubSubscribeChannel(redisClient *c, robj *channel) {
 /* Unsubscribe a client from a channel. Returns 1 if the operation succeeded, or
  * 0 if the client was not subscribed to the specified channel. */
 int pubsubUnsubscribeChannel(redisClient *c, robj *channel, int notify) {
-    struct dictEntry *de;
+    dictEntry *de;
     list *clients;
     listNode *ln;
     int retval = 0;
@@ -218,7 +218,7 @@ int pubsubUnsubscribeAllPatterns(redisClient *c, int notify) {
 /* Publish a message */
 int pubsubPublishMessage(robj *channel, robj *message) {
     int receivers = 0;
-    struct dictEntry *de;
+    dictEntry *de;
     listNode *ln;
     listIter li;
 
@@ -306,8 +306,10 @@ void punsubscribeCommand(redisClient *c) {
 
 void publishCommand(redisClient *c) {
     int receivers = pubsubPublishMessage(c->argv[1],c->argv[2]);
-    if (server.cluster_enabled) clusterPropagatePublish(c->argv[1],c->argv[2]);
-    forceCommandPropagation(c,REDIS_PROPAGATE_REPL);
+    if (server.cluster_enabled)
+        clusterPropagatePublish(c->argv[1],c->argv[2]);
+    else
+        forceCommandPropagation(c,REDIS_PROPAGATE_REPL);
     addReplyLongLong(c,receivers);
 }
 

src/rdb.c

@@ -209,11 +209,41 @@ int rdbTryIntegerEncoding(char *s, size_t len, unsigned char *enc) {
     return rdbEncodeInteger(value,enc);
 }
 
-int rdbSaveLzfStringObject(rio *rdb, unsigned char *s, size_t len) {
-    size_t comprlen, outlen;
+/* Save an already compressed object in LZF encoding.
+ *
+ * On success the length of the strored object is returned, otherwise
+ * 0 is returned. */
+int rdbSaveLzfStringObject(rio *rdb, unsigned char *out, size_t len, size_t comprlen) {
     unsigned char byte;
     int n, nwritten = 0;
+
+    /* Data compressed! Let's save it on disk */
+    byte = (REDIS_RDB_ENCVAL<<6)|REDIS_RDB_ENC_LZF;
+    if ((n = rdbWriteRaw(rdb,&byte,1)) == -1) goto writeerr;
+    nwritten += n;
+    if ((n = rdbSaveLen(rdb,comprlen)) == -1) goto writeerr;
+    nwritten += n;
+    if ((n = rdbSaveLen(rdb,len)) == -1) goto writeerr;
+    nwritten += n;
+    if ((n = rdbWriteRaw(rdb,out,comprlen)) == -1) goto writeerr;
+    nwritten += n;
+    return nwritten;
+
+writeerr:
+    zfree(out);
+    return -1;
+}
+
+/* Try to compress the string at 's' for 'len' bytes using LZF.
+ * If successful save the object with LZF encoding, otherwise
+ * returns 0 if the string can't be compressed, or -1 if the
+ * compressed string can't be saved.
+ *
+ * On success the number of bytes used is returned. */
+int rdbTrySaveLzfStringObject(rio *rdb, unsigned char *s, size_t len) {
+    size_t comprlen, outlen;
     void *out;
+    int retval;
 
     /* We require at least four bytes compression for this to be worth it */
     if (len <= 4) return 0;
@@ -224,26 +254,9 @@ int rdbSaveLzfStringObject(rio *rdb, unsigned char *s, size_t len) {
         zfree(out);
         return 0;
     }
-    /* Data compressed! Let's save it on disk */
-    byte = (REDIS_RDB_ENCVAL<<6)|REDIS_RDB_ENC_LZF;
-    if ((n = rdbWriteRaw(rdb,&byte,1)) == -1) goto writeerr;
-    nwritten += n;
-
-    if ((n = rdbSaveLen(rdb,comprlen)) == -1) goto writeerr;
-    nwritten += n;
-
-    if ((n = rdbSaveLen(rdb,len)) == -1) goto writeerr;
-    nwritten += n;
-
-    if ((n = rdbWriteRaw(rdb,out,comprlen)) == -1) goto writeerr;
-    nwritten += n;
-
+    retval = rdbSaveLzfStringObject(rdb,out,len,comprlen);
     zfree(out);
-    return nwritten;
-
-writeerr:
-    zfree(out);
-    return -1;
+    return retval;
 }
 
 robj *rdbLoadLzfStringObject(rio *rdb) {
@@ -283,7 +296,7 @@ int rdbSaveRawString(rio *rdb, unsigned char *s, size_t len) {
     /* Try LZF compression - under 20 bytes it's unable to compress even
      * aaaaaaaaaaaaaaaaaa so skip it */
     if (server.rdb_compression && len > 20) {
-        n = rdbSaveLzfStringObject(rdb,s,len);
+        n = rdbTrySaveLzfStringObject(rdb,s,len);
         if (n == -1) return -1;
         if (n > 0) return n;
         /* Return value of 0 means data can't be compressed, save the old way */
@@ -324,6 +337,11 @@ int rdbSaveStringObject(rio *rdb, robj *obj) {
      * object is already integer encoded. */
     if (obj->encoding == REDIS_ENCODING_INT) {
         return rdbSaveLongLongAsStringObject(rdb,(long)obj->ptr);
+    } else if (obj->encoding == REDIS_ENCODING_LZF) {
+        /* Data is already compressed, save it with LZF encoding. */
+        int len = stringObjectLen(obj);
+        unsigned char *p = obj->ptr;
+        return rdbSaveLzfStringObject(rdb,p+4,len,sdslen(obj->ptr)-4);
     } else {
         redisAssertWithInfo(NULL,obj,sdsEncodedObject(obj));
         return rdbSaveRawString(rdb,obj->ptr,sdslen(obj->ptr));
@@ -690,9 +708,9 @@ int rdbSave(char *filename) {
     rioWrite(&rdb,&cksum,8);
 
     /* Make sure data will not remain on the OS's output buffers */
-    fflush(fp);
-    fsync(fileno(fp));
-    fclose(fp);
+    if (fflush(fp) == EOF) goto werr;
+    if (fsync(fileno(fp)) == -1) goto werr;
+    if (fclose(fp) == EOF) goto werr;
 
     /* Use RENAME to make sure the DB file is changed atomically only
      * if the generate DB file is ok. */
@@ -1065,6 +1083,10 @@ void rdbLoadProgressCallback(rio *r, const void *buf, size_t len) {
     if (server.loading_process_events_interval_bytes &&
         (r->processed_bytes + len)/server.loading_process_events_interval_bytes > r->processed_bytes/server.loading_process_events_interval_bytes)
     {
+        /* The DB can take some non trivial amount of time to load. Update
+         * our cached time since it is used to create and update the last
+         * interaction time with clients and for other important things. */
+        updateCachedTime();
         if (server.masterhost && server.repl_state == REDIS_REPL_TRANSFER)
             replicationSendNewlineToMaster();
         loadingProgress(r->processed_bytes);

src/redis-cli.c

@@ -58,7 +58,7 @@
 #define OUTPUT_RAW 1
 #define OUTPUT_CSV 2
 #define REDIS_CLI_KEEPALIVE_INTERVAL 15 /* seconds */
-#define REDIS_DEFAULT_PIPE_TIMEOUT 30 /* seconds */
+#define REDIS_CLI_DEFAULT_PIPE_TIMEOUT 30 /* seconds */
 
 static redisContext *context;
 static struct config {
@@ -82,6 +82,8 @@ static struct config {
     int getrdb_mode;
     int stat_mode;
     int scan_mode;
+    int intrinsic_latency_mode;
+    int intrinsic_latency_duration;
     char *pattern;
     char *rdb_filename;
     int bigkeys;
@@ -94,6 +96,7 @@ static struct config {
 } config;
 
 static void usage();
+static void slaveMode(void);
 char *redisGitSHA1(void);
 char *redisGitDirty(void);
 
@@ -101,14 +104,18 @@ char *redisGitDirty(void);
  * Utility functions
  *--------------------------------------------------------------------------- */
 
-static long long mstime(void) {
+static long long ustime(void) {
     struct timeval tv;
-    long long mst;
+    long long ust;
 
     gettimeofday(&tv, NULL);
-    mst = ((long long)tv.tv_sec)*1000;
-    mst += tv.tv_usec/1000;
-    return mst;
+    ust = ((long long)tv.tv_sec)*1000000;
+    ust += tv.tv_usec;
+    return ust;
+}
+
+static long long mstime(void) {
+    return ustime()/1000;
 }
 
 static void cliRefreshPrompt(void) {
@@ -600,6 +607,8 @@ static int cliSendCommand(int argc, char **argv, int repeat) {
     if (!strcasecmp(command,"monitor")) config.monitor_mode = 1;
     if (!strcasecmp(command,"subscribe") ||
         !strcasecmp(command,"psubscribe")) config.pubsub_mode = 1;
+    if (!strcasecmp(command,"sync") ||
+        !strcasecmp(command,"psync")) config.slave_mode = 1;
 
     /* Setup argument length */
     argvlen = malloc(argc*sizeof(size_t));
@@ -621,6 +630,13 @@ static int cliSendCommand(int argc, char **argv, int repeat) {
             }
         }
 
+        if (config.slave_mode) {
+            printf("Entering slave output mode...  (press Ctrl-C to quit)\n");
+            slaveMode();
+            config.slave_mode = 0;
+            return REDIS_ERR;  /* Error = slaveMode lost connection to master */
+        }
+
         if (cliReadReply(output_raw) != REDIS_OK) {
             free(argvlen);
             return REDIS_ERR;
@@ -716,8 +732,11 @@ static int parseOptions(int argc, char **argv) {
             config.stat_mode = 1;
         } else if (!strcmp(argv[i],"--scan")) {
             config.scan_mode = 1;
-        } else if (!strcmp(argv[i],"--pattern")) {
+        } else if (!strcmp(argv[i],"--pattern") && !lastarg) {
             config.pattern = argv[++i];
+        } else if (!strcmp(argv[i],"--intrinsic-latency") && !lastarg) {
+            config.intrinsic_latency_mode = 1;
+            config.intrinsic_latency_duration = atoi(argv[++i]);
         } else if (!strcmp(argv[i],"--rdb") && !lastarg) {
             config.getrdb_mode = 1;
             config.rdb_filename = argv[++i];
@@ -803,6 +822,8 @@ static void usage() {
 "  --bigkeys          Sample Redis keys looking for big keys.\n"
 "  --scan             List all keys using the SCAN command.\n"
 "  --pattern <pat>    Useful with --scan to specify a SCAN pattern.\n"
+"  --intrinsic-latency <sec> Run a test to measure intrinsic system latency.\n"
+"                     The test will run for the specified amount of seconds.\n"
 "  --eval <file>      Send an EVAL command using the Lua script at <file>.\n"
 "  --help             Output this help and exit.\n"
 "  --version          Output version and exit.\n"
@@ -820,7 +841,7 @@ static void usage() {
 "When no command is given, redis-cli starts in interactive mode.\n"
 "Type \"help\" in interactive mode for information on available commands.\n"
 "\n",
-        version, REDIS_DEFAULT_PIPE_TIMEOUT);
+        version, REDIS_CLI_DEFAULT_PIPE_TIMEOUT);
     sdsfree(version);
     exit(1);
 }
@@ -845,6 +866,7 @@ static void repl() {
     sds *argv;
 
     config.interactive = 1;
+    linenoiseSetMultiLine(1);
     linenoiseSetCompletionCallback(completionCallback);
 
     /* Only use history when stdin is a tty. */
@@ -940,6 +962,10 @@ static int noninteractive(int argc, char **argv) {
     return retval;
 }
 
+/*------------------------------------------------------------------------------
+ * Eval mode
+ *--------------------------------------------------------------------------- */
+
 static int evalMode(int argc, char **argv) {
     sds script = sdsempty();
     FILE *fp;
@@ -978,6 +1004,10 @@ static int evalMode(int argc, char **argv) {
     return cliSendCommand(argc+3-got_comma, argv2, config.repeat);
 }
 
+/*------------------------------------------------------------------------------
+ * Latency and latency history modes
+ *--------------------------------------------------------------------------- */
+
 #define LATENCY_SAMPLE_RATE 10 /* milliseconds. */
 #define LATENCY_HISTORY_DEFAULT_INTERVAL 15000 /* milliseconds. */
 static void latencyMode(void) {
@@ -1022,6 +1052,10 @@ static void latencyMode(void) {
     }
 }
 
+/*------------------------------------------------------------------------------
+ * Slave mode
+ *--------------------------------------------------------------------------- */
+
 /* Sends SYNC and reads the number of bytes in the payload. Used both by
  * slaveMode() and getRDB(). */
 unsigned long long sendSync(int fd) {
@@ -1061,6 +1095,7 @@ static void slaveMode(void) {
     int fd = context->fd;
     unsigned long long payload = sendSync(fd);
     char buf[1024];
+    int original_output = config.output;
 
     fprintf(stderr,"SYNC with master, discarding %llu "
                    "bytes of bulk transfer...\n", payload);
@@ -1081,8 +1116,13 @@ static void slaveMode(void) {
     /* Now we can use hiredis to read the incoming protocol. */
     config.output = OUTPUT_CSV;
     while (cliReadReply(0) == REDIS_OK);
+    config.output = original_output;
 }
 
+/*------------------------------------------------------------------------------
+ * RDB transfer mode
+ *--------------------------------------------------------------------------- */
+
 /* This function implements --rdb, so it uses the replication protocol in order
  * to fetch the RDB file from a remote server. */
 static void getRDB(void) {
@@ -1128,6 +1168,10 @@ static void getRDB(void) {
     exit(0);
 }
 
+/*------------------------------------------------------------------------------
+ * Bulk import (pipe) mode
+ *--------------------------------------------------------------------------- */
+
 static void pipeMode(void) {
     int fd = context->fd;
     long long errors = 0, replies = 0, obuf_len = 0, obuf_pos = 0;
@@ -1279,92 +1323,287 @@ static void pipeMode(void) {
         exit(0);
 }
 
+/*------------------------------------------------------------------------------
+ * Find big keys
+ *--------------------------------------------------------------------------- */
+
 #define TYPE_STRING 0
 #define TYPE_LIST   1
 #define TYPE_SET    2
 #define TYPE_HASH   3
 #define TYPE_ZSET   4
+#define TYPE_NONE   5
+
+static redisReply *sendScan(unsigned long long *it) {
+    redisReply *reply = redisCommand(context, "SCAN %llu", *it);
+
+    /* Handle any error conditions */
+    if(reply == NULL) {
+        fprintf(stderr, "\nI/O error\n");
+        exit(1);
+    } else if(reply->type == REDIS_REPLY_ERROR) {
+        fprintf(stderr, "SCAN error: %s\n", reply->str);
+        exit(1);
+    } else if(reply->type != REDIS_REPLY_ARRAY) {
+        fprintf(stderr, "Non ARRAY response from SCAN!\n");
+        exit(1);
+    } else if(reply->elements != 2) {
+        fprintf(stderr, "Invalid element count from SCAN!\n");
+        exit(1);
+    }
+
+    /* Validate our types are correct */
+    assert(reply->element[0]->type == REDIS_REPLY_STRING);
+    assert(reply->element[1]->type == REDIS_REPLY_ARRAY);
+    
+    /* Update iterator */
+    *it = atoi(reply->element[0]->str);
+
+    return reply;
+}
+
+static int getDbSize(void) {
+    redisReply *reply;
+    int size;
+
+    reply = redisCommand(context, "DBSIZE");
+    
+    if(reply == NULL || reply->type != REDIS_REPLY_INTEGER) {
+        fprintf(stderr, "Couldn't determine DBSIZE!\n");
+        exit(1);
+    }
+
+    /* Grab the number of keys and free our reply */
+    size = reply->integer;
+    freeReplyObject(reply);
+
+    return size;
+}
+
+static int toIntType(char *key, char *type) {
+    if(!strcmp(type, "string")) {
+        return TYPE_STRING;
+    } else if(!strcmp(type, "list")) {
+        return TYPE_LIST;
+    } else if(!strcmp(type, "set")) {
+        return TYPE_SET;
+    } else if(!strcmp(type, "hash")) {
+        return TYPE_HASH;
+    } else if(!strcmp(type, "zset")) {
+        return TYPE_ZSET;
+    } else if(!strcmp(type, "none")) {
+        return TYPE_NONE;
+    } else {
+        fprintf(stderr, "Unknown type '%s' for key '%s'\n", type, key);
+        exit(1);
+    }
+}
+
+static void getKeyTypes(redisReply *keys, int *types) {
+    redisReply *reply;
+    int i;
+
+    /* Pipeline TYPE commands */
+    for(i=0;i<keys->elements;i++) {
+        redisAppendCommand(context, "TYPE %s", keys->element[i]->str);
+    }
+
+    /* Retrieve types */
+    for(i=0;i<keys->elements;i++) {
+        if(redisGetReply(context, (void**)&reply)!=REDIS_OK) {
+            fprintf(stderr, "Error getting type for key '%s' (%d: %s)\n",
+                keys->element[i]->str, context->err, context->errstr);
+            exit(1);
+        } else if(reply->type != REDIS_REPLY_STATUS) {
+            fprintf(stderr, "Invalid reply type (%d) for TYPE on key '%s'!\n",
+                reply->type, keys->element[i]->str);
+            exit(1);
+        }
+
+        types[i] = toIntType(keys->element[i]->str, reply->str); 
+        freeReplyObject(reply);
+    }
+}
+
+static void getKeySizes(redisReply *keys, int *types, 
+                        unsigned long long *sizes) 
+{
+    redisReply *reply;
+    char *sizecmds[] = {"STRLEN","LLEN","SCARD","HLEN","ZCARD"};
+    int i;
+
+    /* Pipeline size commands */
+    for(i=0;i<keys->elements;i++) {
+        /* Skip keys that were deleted */
+        if(types[i]==TYPE_NONE) 
+            continue;
+
+        redisAppendCommand(context, "%s %s", sizecmds[types[i]], 
+            keys->element[i]->str);
+    }
+
+    /* Retreive sizes */
+    for(i=0;i<keys->elements;i++) {
+        /* Skip keys that dissapeared between SCAN and TYPE */
+        if(types[i] == TYPE_NONE) {
+            sizes[i] = 0;
+            continue;
+        }
+
+        /* Retreive size */
+        if(redisGetReply(context, (void**)&reply)!=REDIS_OK) {
+            fprintf(stderr, "Error getting size for key '%s' (%d: %s)\n",
+                keys->element[i]->str, context->err, context->errstr);
+            exit(1);
+        } else if(reply->type != REDIS_REPLY_INTEGER) {
+            /* Theoretically the key could have been removed and
+             * added as a different type between TYPE and SIZE */
+            fprintf(stderr, 
+                "Warning:  %s on '%s' failed (may have changed type)\n",
+                 sizecmds[types[i]], keys->element[i]->str);
+            sizes[i] = 0;
+        } else {
+            sizes[i] = reply->integer;
+        }
+            
+        freeReplyObject(reply);
+    }
+}
 
 static void findBigKeys(void) {
-    unsigned long long biggest[5] = {0,0,0,0,0};
-    unsigned long long samples = 0;
-    redisReply *reply1, *reply2, *reply3 = NULL;
-    char *sizecmd, *typename[] = {"string","list","set","hash","zset"};
+    unsigned long long biggest[5] = {0}, counts[5] = {0}, totalsize[5] = {0};
+    unsigned long long sampled = 0, total_keys, totlen=0, *sizes=NULL, it=0;
+    sds maxkeys[5] = {0};
+    char *typename[] = {"string","list","set","hash","zset"};
     char *typeunit[] = {"bytes","items","members","fields","members"};
-    int type;
+    redisReply *reply, *keys;
+    int type, *types=NULL, arrsize=0, i;
+    double pct;
 
-    printf("\n# Press ctrl+c when you have had enough of it... :)\n");
-    printf("# You can use -i 0.1 to sleep 0.1 sec every 100 sampled keys\n");
-    printf("# in order to reduce server load (usually not needed).\n\n");
-    while(1) {
-        /* Sample with RANDOMKEY */
-        reply1 = redisCommand(context,"RANDOMKEY");
-        if (reply1 == NULL) {
-            fprintf(stderr,"\nI/O error\n");
-            exit(1);
-        } else if (reply1->type == REDIS_REPLY_ERROR) {
-            fprintf(stderr, "RANDOMKEY error: %s\n",
-                reply1->str);
-            exit(1);
-        } else if (reply1->type == REDIS_REPLY_NIL) {
-            fprintf(stderr, "It looks like the database is empty!\n");
+    /* Total keys pre scanning */
+    total_keys = getDbSize();
+
+    /* Status message */
+    printf("\n# Scanning the entire keyspace to find biggest keys as well as\n");
+    printf("# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec\n");
+    printf("# per 100 SCAN commands (not usually needed).\n\n");
+
+    /* New up sds strings to keep track of overall biggest per type */
+    for(i=0;i<TYPE_NONE; i++) {
+        maxkeys[i] = sdsempty();
+        if(!maxkeys[i]) {
+            fprintf(stderr, "Failed to allocate memory for largest key names!");
             exit(1);
         }
+    }
 
-        /* Get the key type */
-        reply2 = redisCommand(context,"TYPE %s",reply1->str);
-        assert(reply2 && reply2->type == REDIS_REPLY_STATUS);
-        samples++;
+    /* SCAN loop */
+    do {
+        /* Calculate approximate percentage completion */
+        pct = 100 * (double)sampled/total_keys;
 
-        /* Get the key "size" */
-        if (!strcmp(reply2->str,"string")) {
-            sizecmd = "STRLEN";
-            type = TYPE_STRING;
-        } else if (!strcmp(reply2->str,"list")) {
-            sizecmd = "LLEN";
-            type = TYPE_LIST;
-        } else if (!strcmp(reply2->str,"set")) {
-            sizecmd = "SCARD";
-            type = TYPE_SET;
-        } else if (!strcmp(reply2->str,"hash")) {
-            sizecmd = "HLEN";
-            type = TYPE_HASH;
-        } else if (!strcmp(reply2->str,"zset")) {
-            sizecmd = "ZCARD";
-            type = TYPE_ZSET;
-        } else if (!strcmp(reply2->str,"none")) {
-            freeReplyObject(reply1);
-            freeReplyObject(reply2);
-            continue;
-        } else {
-            fprintf(stderr, "Unknown key type '%s' for key '%s'\n",
-                reply2->str, reply1->str);
-            exit(1);
+        /* Grab some keys and point to the keys array */
+        reply = sendScan(&it);
+        keys  = reply->element[1];
+
+        /* Reallocate our type and size array if we need to */
+        if(keys->elements > arrsize) {
+            types = zrealloc(types, sizeof(int)*keys->elements);
+            sizes = zrealloc(sizes, sizeof(unsigned long long)*keys->elements);
+
+            if(!types || !sizes) {
+                fprintf(stderr, "Failed to allocate storage for keys!\n");
+                exit(1);
+            }
+
+            arrsize = keys->elements;
         }
 
-        reply3 = redisCommand(context,"%s %s", sizecmd, reply1->str);
-        if (reply3 && reply3->type == REDIS_REPLY_INTEGER) {
-            if (biggest[type] < reply3->integer) {
-                printf("Biggest %-6s found so far '%s' with %llu %s.\n",
-                    typename[type], reply1->str,
-                    (unsigned long long) reply3->integer,
-                    typeunit[type]);
-                biggest[type] = reply3->integer;
+        /* Retreive types and then sizes */
+        getKeyTypes(keys, types);
+        getKeySizes(keys, types, sizes);
+        
+        /* Now update our stats */
+        for(i=0;i<keys->elements;i++) {
+            if((type = types[i]) == TYPE_NONE)
+                continue;
+            
+            totalsize[type] += sizes[i];
+            counts[type]++;
+            totlen += keys->element[i]->len;
+            sampled++;
+
+            if(biggest[type]<sizes[i]) {
+                printf(
+                   "[%05.2f%%] Biggest %-6s found so far '%s' with %llu %s\n",
+                   pct, typename[type], keys->element[i]->str, sizes[i],
+                   typeunit[type]);
+
+                /* Keep track of biggest key name for this type */
+                maxkeys[type] = sdscpy(maxkeys[type], keys->element[i]->str);
+                if(!maxkeys[type]) {
+                    fprintf(stderr, "Failed to allocate memory for key!\n");
+                    exit(1);
+                }
+
+                /* Keep track of the biggest size for this type */
+                biggest[type] = sizes[i];                
+            }
+
+            /* Update overall progress */
+            if(sampled % 1000000 == 0) {
+                printf("[%05.2f%%] Sampled %llu keys so far\n", pct, sampled);
             }
         }
 
-        if ((samples % 1000000) == 0)
-            printf("(%llu keys sampled)\n", samples);
-
-        if ((samples % 100) == 0 && config.interval)
+        /* Sleep if we've been directed to do so */
+        if(sampled && (sampled %100) == 0 && config.interval) {
             usleep(config.interval);
+        }
+        
+        freeReplyObject(reply);
+    } while(it != 0);
 
-        freeReplyObject(reply1);
-        freeReplyObject(reply2);
-        if (reply3) freeReplyObject(reply3);
+    if(types) zfree(types);
+    if(sizes) zfree(sizes);
+
+    /* We're done */
+    printf("\n-------- summary -------\n\n");
+
+    printf("Sampled %llu keys in the keyspace!\n", sampled);
+    printf("Total key length in bytes is %llu (avg len %.2f)\n\n",
+       totlen, totlen ? (double)totlen/sampled : 0);
+
+    /* Output the biggest keys we found, for types we did find */
+    for(i=0;i<TYPE_NONE;i++) {
+        if(sdslen(maxkeys[i])>0) {
+            printf("Biggest %6s found '%s' has %llu %s\n", typename[i], maxkeys[i],
+               biggest[i], typeunit[i]);
+        }
     }
+
+    printf("\n");
+
+    for(i=0;i<TYPE_NONE;i++) {
+        printf("%llu %ss with %llu %s (%05.2f%% of keys, avg size %.2f)\n",
+           counts[i], typename[i], totalsize[i], typeunit[i],
+           sampled ? 100 * (double)counts[i]/sampled : 0,
+           counts[i] ? (double)totalsize[i]/counts[i] : 0);
+    }
+
+    /* Free sds strings containing max keys */
+    for(i=0;i<TYPE_NONE;i++) {
+        sdsfree(maxkeys[i]);
+    }
+
+    /* Success! */
+    exit(0);
 }
 
+/*------------------------------------------------------------------------------
+ * Stats mode
+ *--------------------------------------------------------------------------- */
+
 /* Return the specified INFO field from the INFO command output "info".
  * A new buffer is allocated for the result, that needs to be free'd.
  * If the field is not found NULL is returned. */
@@ -1504,6 +1743,10 @@ static void statMode() {
     }
 }
 
+/*------------------------------------------------------------------------------
+ * Scan mode
+ *--------------------------------------------------------------------------- */
+
 static void scanMode() {
     redisReply *reply;
     unsigned long long cur = 0;
@@ -1533,6 +1776,73 @@ static void scanMode() {
     exit(0);
 }
 
+/*------------------------------------------------------------------------------
+ * Intrisic latency mode.
+ *
+ * Measure max latency of a running process that does not result from
+ * syscalls. Basically this software should provide an hint about how much
+ * time the kernel leaves the process without a chance to run.
+ *--------------------------------------------------------------------------- */
+
+/* This is just some computation the compiler can't optimize out.
+ * Should run in less than 100-200 microseconds even using very
+ * slow hardware. Runs in less than 10 microseconds in modern HW. */
+unsigned long compute_something_fast(void) {
+    unsigned char s[256], i, j, t;
+    int count = 1000, k;
+    unsigned long output = 0;
+
+    for (k = 0; k < 256; k++) s[k] = k;
+
+    i = 0;
+    j = 0;
+    while(count--) {
+        i++;
+        j = j + s[i];
+        t = s[i];
+        s[i] = s[j];
+        s[j] = t;
+        output += s[(s[i]+s[j])&255];
+    }
+    return output;
+}
+
+static void intrinsicLatencyMode(void) {
+    long long test_end, run_time, max_latency = 0, runs = 0;
+
+    run_time = config.intrinsic_latency_duration*1000000;
+    test_end = ustime() + run_time;
+
+    while(1) {
+        long long start, end, latency;
+
+        start = ustime();
+        compute_something_fast();
+        end = ustime();
+        latency = end-start;
+        runs++;
+        if (latency <= 0) continue;
+
+        /* Reporting */
+        if (latency > max_latency) {
+            max_latency = latency;
+            printf("Max latency so far: %lld microseconds.\n", max_latency);
+        }
+
+        if (end > test_end) {
+            printf("\n%lld total runs (avg %lld microseconds per run).\n",
+                runs, run_time/runs);
+            printf("Worst run took %.02fx times the avarege.\n",
+                (double) max_latency / (run_time/runs));
+            exit(0);
+        }
+    }
+}
+
+/*------------------------------------------------------------------------------
+ * Program main()
+ *--------------------------------------------------------------------------- */
+
 int main(int argc, char **argv) {
     int firstarg;
 
@@ -1553,10 +1863,11 @@ int main(int argc, char **argv) {
     config.getrdb_mode = 0;
     config.stat_mode = 0;
     config.scan_mode = 0;
+    config.intrinsic_latency_mode = 0;
     config.pattern = NULL;
     config.rdb_filename = NULL;
     config.pipe_mode = 0;
-    config.pipe_timeout = REDIS_DEFAULT_PIPE_TIMEOUT;
+    config.pipe_timeout = REDIS_CLI_DEFAULT_PIPE_TIMEOUT;
     config.bigkeys = 0;
     config.stdinarg = 0;
     config.auth = NULL;
@@ -1615,6 +1926,9 @@ int main(int argc, char **argv) {
         scanMode();
     }
 
+    /* Intrinsic latency mode */
+    if (config.intrinsic_latency_mode) intrinsicLatencyMode();
+
     /* Start interactive mode when no command is provided */
     if (argc == 0 && !config.eval) {
         /* Note that in repl mode we don't abort on connection error.

src/redis-trib.rb

@@ -359,11 +359,11 @@ class RedisTrib
         @nodes.each{|n|
             if n.info[:migrating].size > 0
                 cluster_error \
-                    "[WARNING] Node #{n} has slots in migrating state."
+                    "[WARNING] Node #{n} has slots in migrating state (#{n.info[:migrating].keys.join(",")})."
                 open_slots += n.info[:migrating].keys
             elsif n.info[:importing].size > 0
                 cluster_error \
-                    "[WARNING] Node #{n} has slots in importing state."
+                    "[WARNING] Node #{n} has slots in importing state (#{n.info[:importing].keys.join(",")})."
                 open_slots += n.info[:importing].keys
             end
         }
@@ -469,6 +469,12 @@ class RedisTrib
         #         importing state in 1 slot. That's trivial to address.
         if migrating.length == 1 && importing.length == 1
             move_slot(migrating[0],importing[0],slot,:verbose=>true)
+        elsif migrating.length == 1 && importing.length == 0
+            xputs ">>> Setting #{slot} as STABLE"
+            migrating[0].r.cluster("setslot",slot,"stable")
+        elsif migrating.length == 0 && importing.length == 1
+            xputs ">>> Setting #{slot} as STABLE"
+            importing[0].r.cluster("setslot",slot,"stable")
         else
             xputs "[ERR] Sorry, Redis-trib can't fix this slot yet (work in progress)"
         end
@@ -504,7 +510,6 @@ class RedisTrib
     def alloc_slots
         nodes_count = @nodes.length
         masters_count = @nodes.length / (@replicas+1)
-        slots_per_node = ClusterHashSlots / masters_count
         masters = []
         slaves = []
 
@@ -535,34 +540,60 @@ class RedisTrib
         end
 
         # Alloc slots on masters
-        i = 0
+        slots_per_node = ClusterHashSlots.to_f / masters_count
+        first = 0
+        cursor = 0.0
         masters.each_with_index{|n,masternum|
-            first = i*slots_per_node
-            last = first+slots_per_node-1
-            last = ClusterHashSlots-1 if masternum == masters.length-1
+            last = (cursor+slots_per_node-1).round
+            if last > ClusterHashSlots || masternum == masters.length-1
+                last = ClusterHashSlots-1
+            end
+            last = first if last < first # Min step is 1.
             n.add_slots first..last
-            i += 1
+            first = last+1
+            cursor += slots_per_node
         }
 
         # Select N replicas for every master.
         # We try to split the replicas among all the IPs with spare nodes
         # trying to avoid the host where the master is running, if possible.
-        masters.each{|m|
-            i = 0
-            while i < @replicas
-                ips.each{|ip,nodes_list|
-                    next if nodes_list.length == 0
-                    # Skip instances with the same IP as the master if we
-                    # have some more IPs available.
-                    next if ip == m.info[:host] && nodes_count > nodes_list.length
-                    slave = nodes_list.shift
-                    slave.set_as_replica(m.info[:name])
-                    nodes_count -= 1
-                    i += 1
-                    puts "#{m} replica ##{i} is #{slave}"
-                    break if masters.length == masters_count
-                }
-            end
+        #
+        # Note we loop two times.  The first loop assigns the requested
+        # number of replicas to each master.  The second loop assigns any
+        # remaining instances as extra replicas to masters.  Some masters
+        # may end up with more than their requested number of replicas, but
+        # all nodes will be used.
+        assignment_verbose = false
+
+        [:requested,:unused].each{|assign|
+            masters.each{|m|
+                assigned_replicas = 0
+                while assigned_replicas < @replicas
+                    break if nodes_count == 0
+                    if assignment_verbose
+                        if assign == :requested
+                            puts "Requesting total of #{@replicas} replicas " \
+                                 "(#{assigned_replicas} replicas assigned " \
+                                 "so far with #{nodes_count} total remaining)."
+                        elsif assign == :unused
+                            puts "Assigning extra instance to replication " \
+                                 "role too (#{nodes_count} remaining)."
+                        end
+                    end
+                    ips.each{|ip,nodes_list|
+                        next if nodes_list.length == 0
+                        # Skip instances with the same IP as the master if we
+                        # have some more IPs available.
+                        next if ip == m.info[:host] && nodes_count > nodes_list.length
+                        slave = nodes_list.shift
+                        slave.set_as_replica(m.info[:name])
+                        nodes_count -= 1
+                        assigned_replicas += 1
+                        puts "Adding replica #{slave} to #{m}"
+                        break
+                    }
+                end
+            }
         }
     end
 
@@ -693,7 +724,7 @@ class RedisTrib
             keys = source.r.cluster("getkeysinslot",slot,10)
             break if keys.length == 0
             keys.each{|key|
-                source.r.migrate(target.info[:host],target.info[:port],key,0,1000)
+                source.r.client.call(["migrate",target.info[:host],target.info[:port],key,0,1000])
                 print "." if o[:verbose]
                 STDOUT.flush
             }
@@ -898,10 +929,10 @@ class RedisTrib
         xputs ">>> Sending CLUSTER FORGET messages to the cluster..."
         @nodes.each{|n|
             next if n == node
-            if n.info[:replicate] && n.info[:replicate].downcase == node_id
+            if n.info[:replicate] && n.info[:replicate].downcase == id
                 # Reconfigure the slave to replicate with some other node
-                xputs ">>> #{n} as replica of #{master}"
                 master = get_master_with_least_replicas
+                xputs ">>> #{n} as replica of #{master}"
                 n.r.cluster("replicate",master.info[:name])
             end
             n.r.cluster("forget",argv[1])
@@ -940,6 +971,23 @@ class RedisTrib
         xputs ">>> New node timeout set. #{ok_count} OK, #{err_count} ERR."
     end
 
+    def call_cluster_cmd(argv,opt)
+        cmd = argv[1..-1]
+        cmd[0] = cmd[0].upcase
+
+        # Load cluster information
+        load_cluster_info_from_node(argv[0])
+        xputs ">>> Calling #{cmd.join(" ")}"
+        @nodes.each{|n|
+            begin
+                res = n.r.send(*cmd)
+                puts "#{n}: #{res}"
+            rescue => e
+                puts "#{n}: #{e}"
+            end
+        }
+    end
+
     def help_cluster_cmd(argv,opt)
         show_help
         exit 0
@@ -983,6 +1031,7 @@ COMMANDS={
     "add-node" => ["addnode_cluster_cmd", 3, "new_host:new_port existing_host:existing_port"],
     "del-node" => ["delnode_cluster_cmd", 3, "host:port node_id"],
     "set-timeout" => ["set_timeout_cluster_cmd", 3, "host:port milliseconds"],
+    "call" =>    ["call_cluster_cmd", -3, "host:port command arg arg .. arg"],
     "help"    => ["help_cluster_cmd", 1, "(show this help)"]
 }
 

src/redis.c

@@ -118,13 +118,13 @@ struct redisCommand *commandTable;
  */
 struct redisCommand redisCommandTable[] = {
     {"get",getCommand,2,"r",0,NULL,1,1,1,0,0},
-    {"set",setCommand,-3,"wm",0,noPreloadGetKeys,1,1,1,0,0},
-    {"setnx",setnxCommand,3,"wm",0,noPreloadGetKeys,1,1,1,0,0},
-    {"setex",setexCommand,4,"wm",0,noPreloadGetKeys,1,1,1,0,0},
-    {"psetex",psetexCommand,4,"wm",0,noPreloadGetKeys,1,1,1,0,0},
+    {"set",setCommand,-3,"wm",0,NULL,1,1,1,0,0},
+    {"setnx",setnxCommand,3,"wm",0,NULL,1,1,1,0,0},
+    {"setex",setexCommand,4,"wm",0,NULL,1,1,1,0,0},
+    {"psetex",psetexCommand,4,"wm",0,NULL,1,1,1,0,0},
     {"append",appendCommand,3,"wm",0,NULL,1,1,1,0,0},
     {"strlen",strlenCommand,2,"r",0,NULL,1,1,1,0,0},
-    {"del",delCommand,-2,"w",0,noPreloadGetKeys,1,-1,1,0,0},
+    {"del",delCommand,-2,"w",0,NULL,1,-1,1,0,0},
     {"exists",existsCommand,2,"r",0,NULL,1,1,1,0,0},
     {"setbit",setbitCommand,4,"wm",0,NULL,1,1,1,0,0},
     {"getbit",getbitCommand,3,"r",0,NULL,1,1,1,0,0},
@@ -206,8 +206,8 @@ struct redisCommand redisCommandTable[] = {
     {"randomkey",randomkeyCommand,1,"rR",0,NULL,0,0,0,0,0},
     {"select",selectCommand,2,"rl",0,NULL,0,0,0,0,0},
     {"move",moveCommand,3,"w",0,NULL,1,1,1,0,0},
-    {"rename",renameCommand,3,"w",0,renameGetKeys,1,2,1,0,0},
-    {"renamenx",renamenxCommand,3,"w",0,renameGetKeys,1,2,1,0,0},
+    {"rename",renameCommand,3,"w",0,NULL,1,2,1,0,0},
+    {"renamenx",renamenxCommand,3,"w",0,NULL,1,2,1,0,0},
     {"expire",expireCommand,3,"w",0,NULL,1,1,1,0,0},
     {"expireat",expireatCommand,3,"w",0,NULL,1,1,1,0,0},
     {"pexpire",pexpireCommand,3,"w",0,NULL,1,1,1,0,0},
@@ -232,7 +232,7 @@ struct redisCommand redisCommandTable[] = {
     {"replconf",replconfCommand,-1,"arslt",0,NULL,0,0,0,0,0},
     {"flushdb",flushdbCommand,1,"w",0,NULL,0,0,0,0,0},
     {"flushall",flushallCommand,1,"w",0,NULL,0,0,0,0,0},
-    {"sort",sortCommand,-2,"wm",0,NULL,1,1,1,0,0},
+    {"sort",sortCommand,-2,"wm",0,sortGetKeys,1,1,1,0,0},
     {"info",infoCommand,-1,"rlt",0,NULL,0,0,0,0,0},
     {"monitor",monitorCommand,1,"ars",0,NULL,0,0,0,0,0},
     {"ttl",ttlCommand,2,"r",0,NULL,1,1,1,0,0},
@@ -247,7 +247,7 @@ struct redisCommand redisCommandTable[] = {
     {"punsubscribe",punsubscribeCommand,-1,"rpslt",0,NULL,0,0,0,0,0},
     {"publish",publishCommand,3,"pltr",0,NULL,0,0,0,0,0},
     {"pubsub",pubsubCommand,-2,"pltrR",0,NULL,0,0,0,0,0},
-    {"watch",watchCommand,-2,"rs",0,noPreloadGetKeys,1,-1,1,0,0},
+    {"watch",watchCommand,-2,"rs",0,NULL,1,-1,1,0,0},
     {"unwatch",unwatchCommand,1,"rs",0,NULL,0,0,0,0,0},
     {"cluster",clusterCommand,-2,"ar",0,NULL,0,0,0,0,0},
     {"restore",restoreCommand,-4,"awm",0,NULL,1,1,1,0,0},
@@ -259,16 +259,24 @@ struct redisCommand redisCommandTable[] = {
     {"dump",dumpCommand,2,"ar",0,NULL,1,1,1,0,0},
     {"object",objectCommand,-2,"r",0,NULL,2,2,2,0,0},
     {"client",clientCommand,-2,"ar",0,NULL,0,0,0,0,0},
-    {"eval",evalCommand,-3,"s",0,zunionInterGetKeys,0,0,0,0,0},
-    {"evalsha",evalShaCommand,-3,"s",0,zunionInterGetKeys,0,0,0,0,0},
+    {"eval",evalCommand,-3,"s",0,evalGetKeys,0,0,0,0,0},
+    {"evalsha",evalShaCommand,-3,"s",0,evalGetKeys,0,0,0,0,0},
     {"slowlog",slowlogCommand,-2,"r",0,NULL,0,0,0,0,0},
     {"script",scriptCommand,-2,"ras",0,NULL,0,0,0,0,0},
     {"time",timeCommand,1,"rR",0,NULL,0,0,0,0,0},
     {"bitop",bitopCommand,-4,"wm",0,NULL,2,-1,1,0,0},
     {"bitcount",bitcountCommand,-2,"r",0,NULL,1,1,1,0,0},
-    {"wait",waitCommand,3,"rs",0,NULL,0,0,0,0,0}
+    {"bitpos",bitposCommand,-3,"r",0,NULL,1,1,1,0,0},
+    {"wait",waitCommand,3,"rs",0,NULL,0,0,0,0,0},
+    {"pfselftest",pfselftestCommand,1,"r",0,NULL,0,0,0,0,0},
+    {"pfadd",pfaddCommand,-2,"wm",0,NULL,1,1,1,0,0},
+    {"pfcount",pfcountCommand,2,"w",0,NULL,1,1,1,0,0},
+    {"pfmerge",pfmergeCommand,-2,"wm",0,NULL,1,-1,1,0,0},
+    {"pfgetreg",pfgetregCommand,2,"r",0,NULL,0,0,0,0,0}
 };
 
+struct evictionPoolEntry *evictionPoolAlloc(void);
+
 /*============================ Utility functions ============================ */
 
 /* Low level logging. To use only for very big messages, otherwise
@@ -685,7 +693,7 @@ void updateDictResizePolicy(void) {
  *
  * The parameter 'now' is the current time in milliseconds as is passed
  * to the function to avoid too many gettimeofday() syscalls. */
-int activeExpireCycleTryExpire(redisDb *db, struct dictEntry *de, long long now) {
+int activeExpireCycleTryExpire(redisDb *db, dictEntry *de, long long now) {
     long long t = dictGetSignedIntegerVal(de);
     if (now > t) {
         sds key = dictGetKey(de);
@@ -842,12 +850,10 @@ void activeExpireCycle(int type) {
     }
 }
 
-void updateLRUClock(void) {
-    server.lruclock = (server.unixtime/REDIS_LRU_CLOCK_RESOLUTION) &
-                                                REDIS_LRU_CLOCK_MAX;
+unsigned int getLRUClock(void) {
+    return (mstime()/REDIS_LRU_CLOCK_RESOLUTION) & REDIS_LRU_CLOCK_MAX;
 }
 
-
 /* Add a sample to the operations per second array of samples. */
 void trackOperationsPerSecond(void) {
     long long t = mstime() - server.ops_sec_last_sample_time;
@@ -1001,6 +1007,15 @@ void databasesCron(void) {
     }
 }
 
+/* We take a cached value of the unix time in the global state because with
+ * virtual memory and aging there is to store the current time in objects at
+ * every object access, and accuracy is not needed. To access a global var is
+ * a lot faster than calling time(NULL) */
+void updateCachedTime(void) {
+    server.unixtime = time(NULL);
+    server.mstime = mstime();
+}
+
 /* This is our timer interrupt, called server.hz times per second.
  * Here is where we do a number of things that need to be done asynchronously.
  * For instance:
@@ -1030,33 +1045,31 @@ int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
      * handler if we don't return here fast enough. */
     if (server.watchdog_period) watchdogScheduleSignal(server.watchdog_period);
 
-    /* We take a cached value of the unix time in the global state because
-     * with virtual memory and aging there is to store the current time
-     * in objects at every object access, and accuracy is not needed.
-     * To access a global var is faster than calling time(NULL) */
-    server.unixtime = time(NULL);
-    server.mstime = mstime();
+    /* Update the time cache. */
+    updateCachedTime();
 
     run_with_period(100) trackOperationsPerSecond();
 
-    /* We have just 22 bits per object for LRU information.
-     * So we use an (eventually wrapping) LRU clock with 10 seconds resolution.
-     * 2^22 bits with 10 seconds resolution is more or less 1.5 years.
+    /* We have just REDIS_LRU_BITS bits per object for LRU information.
+     * So we use an (eventually wrapping) LRU clock.
      *
-     * Note that even if this will wrap after 1.5 years it's not a problem,
-     * everything will still work but just some object will appear younger
-     * to Redis. But for this to happen a given object should never be touched
-     * for 1.5 years.
+     * Note that even if the counter wraps it's not a big problem,
+     * everything will still work but some object will appear younger
+     * to Redis. However for this to happen a given object should never be
+     * touched for all the time needed to the counter to wrap, which is
+     * not likely.
      *
      * Note that you can change the resolution altering the
-     * REDIS_LRU_CLOCK_RESOLUTION define.
-     */
-    updateLRUClock();
+     * REDIS_LRU_CLOCK_RESOLUTION define. */
+    server.lruclock = getLRUClock();
 
     /* Record the max memory used since the server was started. */
     if (zmalloc_used_memory() > server.stat_peak_memory)
         server.stat_peak_memory = zmalloc_used_memory();
 
+    /* Sample the RSS here since this is a relatively slow call. */
+    server.resident_set_size = zmalloc_get_rss();
+
     /* We received a SIGTERM, shutting down here in a safe way, as it is
      * not ok doing so inside the signal handler. */
     if (server.shutdown_asap) {
@@ -1167,10 +1180,19 @@ int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
     }
 
 
-    /* If we postponed an AOF buffer flush, let's try to do it every time the
-     * cron function is called. */
+    /* AOF postponed flush: Try at every cron cycle if the slow fsync
+     * completed. */
     if (server.aof_flush_postponed_start) flushAppendOnlyFile(0);
 
+    /* AOF write errors: in this case we have a buffer to flush as well and
+     * clear the AOF error in case of success to make the DB writable again,
+     * however to try every second is enough in case of 'hz' is set to
+     * an higher frequency. */
+    run_with_period(1000) {
+        if (server.aof_last_write_status == REDIS_ERR)
+            flushAppendOnlyFile(0);
+    }
+
     /* Close clients that need to be closed asynchronous */
     freeClientsInAsyncFreeQueue();
 
@@ -1375,6 +1397,8 @@ void initServerConfig() {
     server.aof_filename = zstrdup(REDIS_DEFAULT_AOF_FILENAME);
     server.requirepass = NULL;
     server.rdb_compression = REDIS_DEFAULT_RDB_COMPRESSION;
+    server.mem_compression = REDIS_DEFAULT_MEM_COMPRESSION;
+    server.mem_compression_max_size = REDIS_DEFAULT_MEM_COMPRESSION_MAX_SIZE;
     server.rdb_checksum = REDIS_DEFAULT_RDB_CHECKSUM;
     server.stop_writes_on_bgsave_err = REDIS_DEFAULT_STOP_WRITES_ON_BGSAVE_ERROR;
     server.activerehashing = REDIS_DEFAULT_ACTIVE_REHASHING;
@@ -1407,7 +1431,7 @@ void initServerConfig() {
     server.migrate_cached_sockets = dictCreate(&migrateCacheDictType,NULL);
     server.loading_process_events_interval_bytes = (1024*1024*2);
 
-    updateLRUClock();
+    server.lruclock = getLRUClock();
     resetServerSaveParams();
 
     appendServerSaveParams(60*60,1);  /* save after 1 hour and 1 change */
@@ -1473,21 +1497,21 @@ void initServerConfig() {
 }
 
 /* This function will try to raise the max number of open files accordingly to
- * the configured max number of clients. It will also account for 32 additional
- * file descriptors as we need a few more for persistence, listening
- * sockets, log files and so forth.
+ * the configured max number of clients. It also reserves a number of file
+ * descriptors (REDIS_MIN_RESERVED_FDS) for extra operations of
+ * persistence, listening sockets, log files and so forth.
  *
  * If it will not be possible to set the limit accordingly to the configured
  * max number of clients, the function will do the reverse setting
  * server.maxclients to the value that we can actually handle. */
 void adjustOpenFilesLimit(void) {
-    rlim_t maxfiles = server.maxclients+32;
+    rlim_t maxfiles = server.maxclients+REDIS_MIN_RESERVED_FDS;
     struct rlimit limit;
 
     if (getrlimit(RLIMIT_NOFILE,&limit) == -1) {
         redisLog(REDIS_WARNING,"Unable to obtain the current NOFILE limit (%s), assuming 1024 and setting the max clients configuration accordingly.",
             strerror(errno));
-        server.maxclients = 1024-32;
+        server.maxclients = 1024-REDIS_MIN_RESERVED_FDS;
     } else {
         rlim_t oldlimit = limit.rlim_cur;
 
@@ -1495,22 +1519,54 @@ void adjustOpenFilesLimit(void) {
          * for our needs. */
         if (oldlimit < maxfiles) {
             rlim_t f;
-            
+            int setrlimit_error = 0;
+
+            /* Try to set the file limit to match 'maxfiles' or at least
+             * to the higher value supported less than maxfiles. */
             f = maxfiles;
             while(f > oldlimit) {
+                int decr_step = 16;
+
                 limit.rlim_cur = f;
                 limit.rlim_max = f;
                 if (setrlimit(RLIMIT_NOFILE,&limit) != -1) break;
-                f -= 128;
+                setrlimit_error = errno;
+
+                /* We failed to set file limit to 'f'. Try with a
+                 * smaller limit decrementing by a few FDs per iteration. */
+                if (f < decr_step) break;
+                f -= decr_step;
             }
+
+            /* Assume that the limit we get initially is still valid if
+             * our last try was even lower. */
             if (f < oldlimit) f = oldlimit;
+
             if (f != maxfiles) {
-                server.maxclients = f-32;
-                redisLog(REDIS_WARNING,"Unable to set the max number of files limit to %d (%s), setting the max clients configuration to %d.",
-                    (int) maxfiles, strerror(errno), (int) server.maxclients);
+                int old_maxclients = server.maxclients;
+                server.maxclients = f-REDIS_MIN_RESERVED_FDS;
+                if (server.maxclients < 1) {
+                    redisLog(REDIS_WARNING,"Your current 'ulimit -n' "
+                        "of %llu is not enough for Redis to start. "
+                        "Please increase your open file limit to at least "
+                        "%llu. Exiting.", oldlimit, maxfiles);
+                    exit(1);
+                }
+                redisLog(REDIS_WARNING,"You requested maxclients of %d "
+                    "requiring at least %llu max file descriptors.",
+                    old_maxclients, maxfiles);
+                redisLog(REDIS_WARNING,"Redis can't set maximum open files "
+                    "to %llu because of OS error: %s.",
+                    maxfiles, strerror(setrlimit_error));
+                redisLog(REDIS_WARNING,"Current maximum open files is %llu. "
+                    "maxclients has been reduced to %d to compensate for "
+                    "low ulimit. "
+                    "If you need higher maxclients increase 'ulimit -n'.",
+                    oldlimit, server.maxclients);
             } else {
-                redisLog(REDIS_NOTICE,"Max number of open files set to %d",
-                    (int) maxfiles);
+                redisLog(REDIS_NOTICE,"Increased maximum number of open files "
+                    "to %llu (it was originally set to %llu).",
+                    maxfiles, oldlimit);
             }
         }
     }
@@ -1567,7 +1623,7 @@ int listenToPort(int port, int *fds, int *count) {
             redisLog(REDIS_WARNING,
                 "Creating Server TCP listening socket %s:%d: %s",
                 server.bindaddr[j] ? server.bindaddr[j] : "*",
-                server.port, server.neterr);
+                port, server.neterr);
             return REDIS_ERR;
         }
         (*count)++;
@@ -1575,6 +1631,27 @@ int listenToPort(int port, int *fds, int *count) {
     return REDIS_OK;
 }
 
+/* Resets the stats that we expose via INFO or other means that we want
+ * to reset via CONFIG RESETSTAT. The function is also used in order to
+ * initialize these fields in initServer() at server startup. */
+void resetServerStats(void) {
+    server.stat_numcommands = 0;
+    server.stat_numconnections = 0;
+    server.stat_expiredkeys = 0;
+    server.stat_evictedkeys = 0;
+    server.stat_keyspace_misses = 0;
+    server.stat_keyspace_hits = 0;
+    server.stat_fork_time = 0;
+    server.stat_rejected_conn = 0;
+    server.stat_sync_full = 0;
+    server.stat_sync_partial_ok = 0;
+    server.stat_sync_partial_err = 0;
+    memset(server.ops_sec_samples,0,sizeof(server.ops_sec_samples));
+    server.ops_sec_idx = 0;
+    server.ops_sec_last_sample_time = mstime();
+    server.ops_sec_last_sample_ops = 0;
+}
+
 void initServer() {
     int j;
 
@@ -1633,6 +1710,7 @@ void initServer() {
         server.db[j].blocking_keys = dictCreate(&keylistDictType,NULL);
         server.db[j].ready_keys = dictCreate(&setDictType,NULL);
         server.db[j].watched_keys = dictCreate(&keylistDictType,NULL);
+        server.db[j].eviction_pool = evictionPoolAlloc();
         server.db[j].id = j;
         server.db[j].avg_ttl = 0;
     }
@@ -1650,27 +1728,16 @@ void initServer() {
     server.rdb_save_time_last = -1;
     server.rdb_save_time_start = -1;
     server.dirty = 0;
-    server.stat_numcommands = 0;
-    server.stat_numconnections = 0;
-    server.stat_expiredkeys = 0;
-    server.stat_evictedkeys = 0;
+    resetServerStats();
+    /* A few stats we don't want to reset: server startup time, and peak mem. */
     server.stat_starttime = time(NULL);
-    server.stat_keyspace_misses = 0;
-    server.stat_keyspace_hits = 0;
     server.stat_peak_memory = 0;
-    server.stat_fork_time = 0;
-    server.stat_rejected_conn = 0;
-    server.stat_sync_full = 0;
-    server.stat_sync_partial_ok = 0;
-    server.stat_sync_partial_err = 0;
-    memset(server.ops_sec_samples,0,sizeof(server.ops_sec_samples));
-    server.ops_sec_idx = 0;
-    server.ops_sec_last_sample_time = mstime();
-    server.ops_sec_last_sample_ops = 0;
-    server.unixtime = time(NULL);
-    server.mstime = mstime();
+    server.resident_set_size = 0;
     server.lastbgsave_status = REDIS_OK;
+    server.aof_last_write_status = REDIS_OK;
+    server.aof_last_write_errno = 0;
     server.repl_good_slaves_count = 0;
+    updateCachedTime();
 
     /* Create the serverCron() time event, that's our main way to process
      * background operations. */
@@ -2002,17 +2069,30 @@ int processCommand(redisClient *c) {
         int hashslot;
 
         if (server.cluster->state != REDIS_CLUSTER_OK) {
+            flagTransaction(c);
             addReplySds(c,sdsnew("-CLUSTERDOWN The cluster is down. Use CLUSTER INFO for more information\r\n"));
             return REDIS_OK;
         } else {
-            int ask;
-            clusterNode *n = getNodeByQuery(c,c->cmd,c->argv,c->argc,&hashslot,&ask);
+            int error_code;
+            clusterNode *n = getNodeByQuery(c,c->cmd,c->argv,c->argc,&hashslot,&error_code);
             if (n == NULL) {
-                addReplyError(c,"Multi keys request invalid in cluster");
+                flagTransaction(c);
+                if (error_code == REDIS_CLUSTER_REDIR_CROSS_SLOT) {
+                    addReplySds(c,sdsnew("-CROSSSLOT Keys in request don't hash to the same slot\r\n"));
+                } else if (error_code == REDIS_CLUSTER_REDIR_UNSTABLE) {
+                    /* The request spawns mutliple keys in the same slot,
+                     * but the slot is not "stable" currently as there is
+                     * a migration or import in progress. */
+                    addReplySds(c,sdsnew("-TRYAGAIN Multiple keys request during rehashing of slot\r\n"));
+                } else {
+                    redisPanic("getNodeByQuery() unknown error.");
+                }
                 return REDIS_OK;
             } else if (n != server.cluster->myself) {
+                flagTransaction(c);
                 addReplySds(c,sdscatprintf(sdsempty(),
-                    "-%s %d %s:%d\r\n", ask ? "ASK" : "MOVED",
+                    "-%s %d %s:%d\r\n",
+                    (error_code == REDIS_CLUSTER_REDIR_ASK) ? "ASK" : "MOVED",
                     hashslot,n->ip,n->port));
                 return REDIS_OK;
             }
@@ -2035,15 +2115,22 @@ int processCommand(redisClient *c) {
 
     /* Don't accept write commands if there are problems persisting on disk
      * and if this is a master instance. */
-    if (server.stop_writes_on_bgsave_err &&
-        server.saveparamslen > 0
-        && server.lastbgsave_status == REDIS_ERR &&
+    if (((server.stop_writes_on_bgsave_err &&
+          server.saveparamslen > 0 &&
+          server.lastbgsave_status == REDIS_ERR) ||
+          server.aof_last_write_status == REDIS_ERR) &&
         server.masterhost == NULL &&
         (c->cmd->flags & REDIS_CMD_WRITE ||
          c->cmd->proc == pingCommand))
     {
         flagTransaction(c);
-        addReply(c, shared.bgsaveerr);
+        if (server.aof_last_write_status == REDIS_OK)
+            addReply(c, shared.bgsaveerr);
+        else
+            addReplySds(c,
+                sdscatprintf(sdsempty(),
+                "-MISCONF Errors writing to the AOF file: %s\r\n",
+                strerror(server.aof_last_write_errno)));
         return REDIS_OK;
     }
 
@@ -2315,7 +2402,8 @@ sds genRedisInfoString(char *section) {
 
     /* Server */
     if (allsections || defsections || !strcasecmp(section,"server")) {
-        struct utsname name;
+        static int call_uname = 1;
+        static struct utsname name;
         char *mode;
 
         if (server.cluster_enabled) mode = "cluster";
@@ -2323,7 +2411,13 @@ sds genRedisInfoString(char *section) {
         else mode = "standalone";
     
         if (sections++) info = sdscat(info,"\r\n");
-        uname(&name);
+
+        if (call_uname) {
+            /* Uname can be slow and is always the same output. Cache it. */
+            uname(&name);
+            call_uname = 0;
+        }
+
         info = sdscatprintf(info,
             "# Server\r\n"
             "redis_version:%s\r\n"
@@ -2384,8 +2478,16 @@ sds genRedisInfoString(char *section) {
     if (allsections || defsections || !strcasecmp(section,"memory")) {
         char hmem[64];
         char peak_hmem[64];
+        size_t zmalloc_used = zmalloc_used_memory();
 
-        bytesToHuman(hmem,zmalloc_used_memory());
+        /* Peak memory is updated from time to time by serverCron() so it
+         * may happen that the instantaneous value is slightly bigger than
+         * the peak value. This may confuse users, so we update the peak
+         * if found smaller than the current memory usage. */
+        if (zmalloc_used > server.stat_peak_memory)
+            server.stat_peak_memory = zmalloc_used;
+
+        bytesToHuman(hmem,zmalloc_used);
         bytesToHuman(peak_hmem,server.stat_peak_memory);
         if (sections++) info = sdscat(info,"\r\n");
         info = sdscatprintf(info,
@@ -2398,13 +2500,13 @@ sds genRedisInfoString(char *section) {
             "used_memory_lua:%lld\r\n"
             "mem_fragmentation_ratio:%.2f\r\n"
             "mem_allocator:%s\r\n",
-            zmalloc_used_memory(),
+            zmalloc_used,
             hmem,
-            zmalloc_get_rss(),
+            server.resident_set_size,
             server.stat_peak_memory,
             peak_hmem,
             ((long long)lua_gc(server.lua,LUA_GCCOUNT,0))*1024LL,
-            zmalloc_get_fragmentation_ratio(),
+            zmalloc_get_fragmentation_ratio(server.resident_set_size),
             ZMALLOC_LIB
             );
     }
@@ -2426,7 +2528,8 @@ sds genRedisInfoString(char *section) {
             "aof_rewrite_scheduled:%d\r\n"
             "aof_last_rewrite_time_sec:%jd\r\n"
             "aof_current_rewrite_time_sec:%jd\r\n"
-            "aof_last_bgrewrite_status:%s\r\n",
+            "aof_last_bgrewrite_status:%s\r\n"
+            "aof_last_write_status:%s\r\n",
             server.loading,
             server.dirty,
             server.rdb_child_pid != -1,
@@ -2441,7 +2544,8 @@ sds genRedisInfoString(char *section) {
             (intmax_t)server.aof_rewrite_time_last,
             (intmax_t)((server.aof_child_pid == -1) ?
                 -1 : time(NULL)-server.aof_rewrite_time_start),
-            (server.aof_lastbgrewrite_status == REDIS_OK) ? "ok" : "err");
+            (server.aof_lastbgrewrite_status == REDIS_OK) ? "ok" : "err",
+            (server.aof_last_write_status == REDIS_OK) ? "ok" : "err");
 
         if (server.aof_state != REDIS_AOF_OFF) {
             info = sdscatprintf(info,
@@ -2732,8 +2836,9 @@ void monitorCommand(redisClient *c) {
 
 /* ============================ Maxmemory directive  ======================== */
 
-/* This function gets called when 'maxmemory' is set on the config file to limit
- * the max memory used by the server, before processing a command.
+/* freeMemoryIfNeeded() gets called when 'maxmemory' is set on the config
+ * file to limit the max memory used by the server, before processing a
+ * command.
  *
  * The goal of the function is to free enough memory to keep Redis under the
  * configured memory limit.
@@ -2746,7 +2851,123 @@ void monitorCommand(redisClient *c) {
  * function returns REDIS_OK, otherwise REDIS_ERR is returned, and the caller
  * should block the execution of commands that will result in more memory
  * used by the server.
- */
+ *
+ * ------------------------------------------------------------------------
+ *
+ * LRU approximation algorithm
+ *
+ * Redis uses an approximation of the LRU algorithm that runs in constant
+ * memory. Every time there is a key to expire, we sample N keys (with
+ * N very small, usually in around 5) to populate a pool of best keys to
+ * evict of M keys (the pool size is defined by REDIS_EVICTION_POOL_SIZE).
+ *
+ * The N keys sampled are added in the pool of good keys to expire (the one
+ * with an old access time) if they are better than one of the current keys
+ * in the pool.
+ *
+ * After the pool is populated, the best key we have in the pool is expired.
+ * However note that we don't remove keys from the pool when they are deleted
+ * so the pool may contain keys that no longer exist.
+ *
+ * When we try to evict a key, and all the entries in the pool don't exist
+ * we populate it again. This time we'll be sure that the pool has at least
+ * one key that can be evicted, if there is at least one key that can be
+ * evicted in the whole database. */
+
+/* Create a new eviction pool. */
+struct evictionPoolEntry *evictionPoolAlloc(void) {
+    struct evictionPoolEntry *ep;
+    int j;
+
+    ep = zmalloc(sizeof(*ep)*REDIS_EVICTION_POOL_SIZE);
+    for (j = 0; j < REDIS_EVICTION_POOL_SIZE; j++) {
+        ep[j].idle = 0;
+        ep[j].key = NULL;
+    }
+    return ep;
+}
+
+/* This is an helper function for freeMemoryIfNeeded(), it is used in order
+ * to populate the evictionPool with a few entries every time we want to
+ * expire a key. Keys with idle time smaller than one of the current
+ * keys are added. Keys are always added if there are free entries.
+ *
+ * We insert keys on place in ascending order, so keys with the smaller
+ * idle time are on the left, and keys with the higher idle time on the
+ * right. */
+
+#define EVICTION_SAMPLES_ARRAY_SIZE 16
+void evictionPoolPopulate(dict *sampledict, dict *keydict, struct evictionPoolEntry *pool) {
+    int j, k, count;
+    dictEntry *_samples[EVICTION_SAMPLES_ARRAY_SIZE];
+    dictEntry **samples;
+
+    /* Try to use a static buffer: this function is a big hit...
+     * Note: it was actually measured that this helps. */
+    if (server.maxmemory_samples <= EVICTION_SAMPLES_ARRAY_SIZE) {
+        samples = _samples;
+    } else {
+        samples = zmalloc(sizeof(samples[0])*server.maxmemory_samples);
+    }
+
+#if 1 /* Use bulk get by default. */
+    count = dictGetRandomKeys(sampledict,samples,server.maxmemory_samples);
+#else
+    count = server.maxmemory_samples;
+    for (j = 0; j < count; j++) samples[j] = dictGetRandomKey(sampledict);
+#endif
+
+    for (j = 0; j < count; j++) {
+        unsigned long long idle;
+        sds key;
+        robj *o;
+        dictEntry *de;
+
+        de = samples[j];
+        key = dictGetKey(de);
+        /* If the dictionary we are sampling from is not the main
+         * dictionary (but the expires one) we need to lookup the key
+         * again in the key dictionary to obtain the value object. */
+        if (sampledict != keydict) de = dictFind(keydict, key);
+        o = dictGetVal(de);
+        idle = estimateObjectIdleTime(o);
+
+        /* Insert the element inside the pool.
+         * First, find the first empty bucket or the first populated
+         * bucket that has an idle time smaller than our idle time. */
+        k = 0;
+        while (k < REDIS_EVICTION_POOL_SIZE &&
+               pool[k].key &&
+               pool[k].idle < idle) k++;
+        if (k == 0 && pool[REDIS_EVICTION_POOL_SIZE-1].key != NULL) {
+            /* Can't insert if the element is < the worst element we have
+             * and there are no empty buckets. */
+            continue;
+        } else if (k < REDIS_EVICTION_POOL_SIZE && pool[k].key == NULL) {
+            /* Inserting into empty position. No setup needed before insert. */
+        } else {
+            /* Inserting in the middle. Now k points to the first element
+             * greater than the element to insert.  */
+            if (pool[REDIS_EVICTION_POOL_SIZE-1].key == NULL) {
+                /* Free space on the right? Insert at k shifting
+                 * all the elements from k to end to the right. */
+                memmove(pool+k+1,pool+k,
+                    sizeof(pool[0])*(REDIS_EVICTION_POOL_SIZE-k-1));
+            } else {
+                /* No free space on right? Insert at k-1 */
+                k--;
+                /* Shift all elements on the left of k (included) to the
+                 * left, so we discard the element with smaller idle time. */
+                sdsfree(pool[0].key);
+                memmove(pool,pool+1,sizeof(pool[0])*k);
+            }
+        }
+        pool[k].key = sdsdup(key);
+        pool[k].idle = idle;
+    }
+    if (samples != _samples) zfree(samples);
+}
+
 int freeMemoryIfNeeded(void) {
     size_t mem_used, mem_tofree, mem_freed;
     int slaves = listLength(server.slaves);
@@ -2788,7 +3009,7 @@ int freeMemoryIfNeeded(void) {
         for (j = 0; j < server.dbnum; j++) {
             long bestval = 0; /* just to prevent warning */
             sds bestkey = NULL;
-            struct dictEntry *de;
+            dictEntry *de;
             redisDb *db = server.db+j;
             dict *dict;
 
@@ -2813,24 +3034,34 @@ int freeMemoryIfNeeded(void) {
             else if (server.maxmemory_policy == REDIS_MAXMEMORY_ALLKEYS_LRU ||
                 server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_LRU)
             {
-                for (k = 0; k < server.maxmemory_samples; k++) {
-                    sds thiskey;
-                    long thisval;
-                    robj *o;
+                struct evictionPoolEntry *pool = db->eviction_pool;
 
-                    de = dictGetRandomKey(dict);
-                    thiskey = dictGetKey(de);
-                    /* When policy is volatile-lru we need an additional lookup
-                     * to locate the real key, as dict is set to db->expires. */
-                    if (server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_LRU)
-                        de = dictFind(db->dict, thiskey);
-                    o = dictGetVal(de);
-                    thisval = estimateObjectIdleTime(o);
+                while(bestkey == NULL) {
+                    evictionPoolPopulate(dict, db->dict, db->eviction_pool);
+                    /* Go backward from best to worst element to evict. */
+                    for (k = REDIS_EVICTION_POOL_SIZE-1; k >= 0; k--) {
+                        if (pool[k].key == NULL) continue;
+                        de = dictFind(dict,pool[k].key);
 
-                    /* Higher idle time is better candidate for deletion */
-                    if (bestkey == NULL || thisval > bestval) {
-                        bestkey = thiskey;
-                        bestval = thisval;
+                        /* Remove the entry from the pool. */
+                        sdsfree(pool[k].key);
+                        /* Shift all elements on its right to left. */
+                        memmove(pool+k,pool+k+1,
+                            sizeof(pool[0])*(REDIS_EVICTION_POOL_SIZE-k-1));
+                        /* Clear the element on the right which is empty
+                         * since we shifted one position to the left.  */
+                        pool[REDIS_EVICTION_POOL_SIZE-1].key = NULL;
+                        pool[REDIS_EVICTION_POOL_SIZE-1].idle = 0;
+
+                        /* If the key exists, is our pick. Otherwise it is
+                         * a ghost and we need to try the next element. */
+                        if (de) {
+                            bestkey = dictGetKey(de);
+                            break;
+                        } else {
+                            /* Ghost... */
+                            continue;
+                        }
                     }
                 }
             }
@@ -3055,10 +3286,15 @@ void redisOutOfMemoryHandler(size_t allocation_size) {
 
 void redisSetProcTitle(char *title) {
 #ifdef USE_SETPROCTITLE
-    setproctitle("%s %s:%d",
+    char *server_mode = "";
+    if (server.cluster_enabled) server_mode = " [cluster]";
+    else if (server.sentinel_mode) server_mode = " [sentinel]";
+
+    setproctitle("%s %s:%d%s",
         title,
         server.bindaddr_count ? server.bindaddr[0] : "*",
-        server.port);
+        server.port,
+        server_mode);
 #else
     REDIS_NOTUSED(title);
 #endif
@@ -3129,10 +3365,10 @@ int main(int argc, char **argv) {
             }
             j++;
         }
+        if (configfile) server.configfile = getAbsolutePath(configfile);
         resetServerSaveParams();
         loadServerConfig(configfile,options);
         sdsfree(options);
-        if (configfile) server.configfile = getAbsolutePath(configfile);
     } else {
         redisLog(REDIS_WARNING, "Warning: no config file specified, using the default config. In order to specify a config file use %s /path/to/%s.conf", argv[0], server.sentinel_mode ? "sentinel" : "redis");
     }

src/redis.h

@@ -107,13 +107,15 @@
 #define REDIS_DEFAULT_SYSLOG_ENABLED 0
 #define REDIS_DEFAULT_STOP_WRITES_ON_BGSAVE_ERROR 1
 #define REDIS_DEFAULT_RDB_COMPRESSION 1
+#define REDIS_DEFAULT_MEM_COMPRESSION 0
+#define REDIS_DEFAULT_MEM_COMPRESSION_MAX_SIZE (1024*64)
 #define REDIS_DEFAULT_RDB_CHECKSUM 1
 #define REDIS_DEFAULT_RDB_FILENAME "dump.rdb"
 #define REDIS_DEFAULT_SLAVE_SERVE_STALE_DATA 1
 #define REDIS_DEFAULT_SLAVE_READ_ONLY 1
 #define REDIS_DEFAULT_REPL_DISABLE_TCP_NODELAY 0
 #define REDIS_DEFAULT_MAXMEMORY 0
-#define REDIS_DEFAULT_MAXMEMORY_SAMPLES 3
+#define REDIS_DEFAULT_MAXMEMORY_SAMPLES 5
 #define REDIS_DEFAULT_AOF_FILENAME "appendonly.aof"
 #define REDIS_DEFAULT_AOF_NO_FSYNC_ON_REWRITE 0
 #define REDIS_DEFAULT_ACTIVE_REHASHING 1
@@ -123,6 +125,7 @@
 #define REDIS_IP_STR_LEN INET6_ADDRSTRLEN
 #define REDIS_PEER_ID_LEN (REDIS_IP_STR_LEN+32) /* Must be enough for ip:port */
 #define REDIS_BINDADDR_MAX 16
+#define REDIS_MIN_RESERVED_FDS 32
 
 #define ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP 20 /* Loopkups per loop. */
 #define ACTIVE_EXPIRE_CYCLE_FAST_DURATION 1000 /* Microseconds */
@@ -139,9 +142,9 @@
 #define REDIS_LONGSTR_SIZE      21          /* Bytes needed for long -> str */
 #define REDIS_AOF_AUTOSYNC_BYTES (1024*1024*32) /* fdatasync every 32MB */
 /* When configuring the Redis eventloop, we setup it so that the total number
- * of file descriptors we can handle are server.maxclients + FDSET_INCR
+ * of file descriptors we can handle are server.maxclients + RESERVED_FDS + FDSET_INCR
  * that is our safety margin. */
-#define REDIS_EVENTLOOP_FDSET_INCR 128
+#define REDIS_EVENTLOOP_FDSET_INCR (REDIS_MIN_RESERVED_FDS+96)
 
 /* Hash table parameters */
 #define REDIS_HT_MINFILL        10      /* Minimal hash table fill 10% */
@@ -171,7 +174,7 @@
 
 /* Objects encoding. Some kind of objects like Strings and Hashes can be
  * internally represented in multiple ways. The 'encoding' field of the object
- * is set to one of this fields for this object. */
+ * is set to one of this values. */
 #define REDIS_ENCODING_RAW 0     /* Raw representation */
 #define REDIS_ENCODING_INT 1     /* Encoded as integer */
 #define REDIS_ENCODING_HT 2      /* Encoded as hash table */
@@ -181,6 +184,7 @@
 #define REDIS_ENCODING_INTSET 6  /* Encoded as intset */
 #define REDIS_ENCODING_SKIPLIST 7  /* Encoded as skiplist */
 #define REDIS_ENCODING_EMBSTR 8  /* Embedded sds string encoding */
+#define REDIS_ENCODING_LZF 9     /* LZF compressed string. */
 
 /* Defines related to the dump file format. To store 32 bits lengths for short
  * keys requires a lot of space, so we check the most significant 2 bits of
@@ -323,7 +327,7 @@
 #define REDIS_MAXMEMORY_ALLKEYS_LRU 3
 #define REDIS_MAXMEMORY_ALLKEYS_RANDOM 4
 #define REDIS_MAXMEMORY_NO_EVICTION 5
-#define REDIS_DEFAULT_MAXMEMORY_POLICY REDIS_MAXMEMORY_VOLATILE_LRU
+#define REDIS_DEFAULT_MAXMEMORY_POLICY REDIS_MAXMEMORY_NO_EVICTION
 
 /* Scripting */
 #define REDIS_LUA_TIME_LIMIT 5000 /* milliseconds */
@@ -382,17 +386,23 @@ typedef long long mstime_t; /* millisecond time type. */
 /* A redis object, that is a type able to hold a string / list / set */
 
 /* The actual Redis Object */
-#define REDIS_LRU_CLOCK_MAX ((1<<21)-1) /* Max value of obj->lru */
-#define REDIS_LRU_CLOCK_RESOLUTION 10 /* LRU clock resolution in seconds */
+#define REDIS_LRU_BITS 24
+#define REDIS_LRU_CLOCK_MAX ((1<<REDIS_LRU_BITS)-1) /* Max value of obj->lru */
+#define REDIS_LRU_CLOCK_RESOLUTION 1000 /* LRU clock resolution in ms */
 typedef struct redisObject {
     unsigned type:4;
-    unsigned notused:2;     /* Not used */
     unsigned encoding:4;
-    unsigned lru:22;        /* lru time (relative to server.lruclock) */
+    unsigned lru:REDIS_LRU_BITS; /* lru time (relative to server.lruclock) */
     int refcount;
     void *ptr;
 } robj;
 
+/* Macro used to obtain the current LRU clock.
+ * If the current resolution is lower than the frequency we refresh the
+ * LRU clock (as it should be in production servers) we return the
+ * precomputed value, otherwise we need to resort to a function call. */
+#define LRU_CLOCK() ((1000/server.hz <= REDIS_LRU_CLOCK_RESOLUTION) ? server.lruclock : getLRUClock())
+
 /* Macro used to initialize a Redis object allocated on the stack.
  * Note that this macro is taken near the structure definition to make sure
  * we'll update it when the structure is changed, to avoid bugs like
@@ -404,13 +414,30 @@ typedef struct redisObject {
     _var.ptr = _ptr; \
 } while(0);
 
+/* To improve the quality of the LRU approximation we take a set of keys
+ * that are good candidate for eviction across freeMemoryIfNeeded() calls.
+ *
+ * Entries inside the eviciton pool are taken ordered by idle time, putting
+ * greater idle times to the right (ascending order).
+ *
+ * Empty entries have the key pointer set to NULL. */
+#define REDIS_EVICTION_POOL_SIZE 16
+struct evictionPoolEntry {
+    unsigned long long idle;    /* Object idle time. */
+    sds key;                    /* Key name. */
+};
+
+/* Redis database representation. There are multiple databases identified
+ * by integers from 0 (the default database) up to the max configured
+ * database. The database number is the 'id' field in the structure. */
 typedef struct redisDb {
     dict *dict;                 /* The keyspace for this DB */
     dict *expires;              /* Timeout of keys with a timeout set */
     dict *blocking_keys;        /* Keys with clients waiting for data (BLPOP) */
     dict *ready_keys;           /* Blocked keys that received a PUSH */
     dict *watched_keys;         /* WATCHED keys for MULTI/EXEC CAS */
-    int id;
+    struct evictionPoolEntry *eviction_pool;    /* Eviction pool of keys */
+    int id;                     /* Database ID */
     long long avg_ttl;          /* Average TTL, just for stats */
 } redisDb;
 
@@ -597,8 +624,7 @@ struct redisServer {
     dict *commands;             /* Command table */
     dict *orig_commands;        /* Command table before command renaming. */
     aeEventLoop *el;
-    unsigned lruclock:22;       /* Clock incrementing every minute, for LRU */
-    unsigned lruclock_padding:10;
+    unsigned lruclock:REDIS_LRU_BITS; /* Clock for LRU eviction */
     int shutdown_asap;          /* SHUTDOWN needed ASAP */
     int activerehashing;        /* Incremental rehash in serverCron() */
     char *requirepass;          /* Pass for AUTH command, or NULL */
@@ -654,6 +680,7 @@ struct redisServer {
     long long slowlog_entry_id;     /* SLOWLOG current entry ID */
     long long slowlog_log_slower_than; /* SLOWLOG time limit (to get logged) */
     unsigned long slowlog_max_len;     /* SLOWLOG max number of items logged */
+    size_t resident_set_size;       /* RSS sampled in serverCron(). */
     /* The following two are used to track instantaneous "load" in terms
      * of operations per second. */
     long long ops_sec_last_sample_time; /* Timestamp of last sample (in ms) */
@@ -691,6 +718,8 @@ struct redisServer {
     int aof_lastbgrewrite_status;   /* REDIS_OK or REDIS_ERR */
     unsigned long aof_delayed_fsync;  /* delayed AOF fsync() counter */
     int aof_rewrite_incremental_fsync;/* fsync incrementally while rewriting? */
+    int aof_last_write_status;      /* REDIS_OK or REDIS_ERR */
+    int aof_last_write_errno;       /* Valid if aof_last_write_status is ERR */
     /* RDB persistence */
     long long dirty;                /* Changes to DB from the last save */
     long long dirty_before_bgsave;  /* Used to restore dirty on failed BGSAVE */
@@ -761,7 +790,7 @@ struct redisServer {
     list *clients_waiting_acks;         /* Clients waiting in WAIT command. */
     int get_ack_from_slaves;            /* If true we send REPLCONF GETACK. */
     /* Limits */
-    unsigned int maxclients;        /* Max number of simultaneous clients */
+    int maxclients;                 /* Max number of simultaneous clients */
     unsigned long long maxmemory;   /* Max number of memory bytes to use */
     int maxmemory_policy;           /* Policy for key eviction */
     int maxmemory_samples;          /* Pricision of random sampling */
@@ -783,8 +812,10 @@ struct redisServer {
     size_t set_max_intset_entries;
     size_t zset_max_ziplist_entries;
     size_t zset_max_ziplist_value;
+    int mem_compression;    /* In memory LZF compression. */
+    size_t mem_compression_max_size; /* Try to compress up to this size. */
     time_t unixtime;        /* Unix time sampled every cron cycle. */
-    long long mstime;       /* Like 'unixtime' but with milliseconds resolution. */
+    long long mstime;       /* Like unixtime but in milliseconds. */
     /* Pubsub */
     dict *pubsub_channels;  /* Map channels to list of subscribed clients */
     list *pubsub_patterns;  /* A list of pubsub_patterns */
@@ -824,7 +855,7 @@ typedef struct pubsubPattern {
 } pubsubPattern;
 
 typedef void redisCommandProc(redisClient *c);
-typedef int *redisGetKeysProc(struct redisCommand *cmd, robj **argv, int argc, int *numkeys, int flags);
+typedef int *redisGetKeysProc(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
 struct redisCommand {
     char *name;
     redisCommandProc *proc;
@@ -1053,7 +1084,10 @@ char *strEncoding(int encoding);
 int compareStringObjects(robj *a, robj *b);
 int collateStringObjects(robj *a, robj *b);
 int equalStringObjects(robj *a, robj *b);
-unsigned long estimateObjectIdleTime(robj *o);
+unsigned long long estimateObjectIdleTime(robj *o);
+#define rawEncodedObject(objptr) (objptr->encoding == REDIS_ENCODING_RAW)
+#define lzfEncodedObject(objptr) (objptr->encoding == REDIS_ENCODING_LZF)
+#define intEncodedObject(objptr) (objptr->encoding == REDIS_ENCODING_INT)
 #define sdsEncodedObject(objptr) (objptr->encoding == REDIS_ENCODING_RAW || objptr->encoding == REDIS_ENCODING_EMBSTR)
 
 /* Synchronous I/O with timeout */
@@ -1152,6 +1186,9 @@ void populateCommandTable(void);
 void resetCommandTableStats(void);
 void adjustOpenFilesLimit(void);
 void closeListeningSockets(int unlink_unix_socket);
+void updateCachedTime(void);
+void resetServerStats(void);
+unsigned int getLRUClock(void);
 
 /* Set data type */
 robj *setTypeCreate(robj *value);
@@ -1223,6 +1260,7 @@ void setKey(redisDb *db, robj *key, robj *val);
 int dbExists(redisDb *db, robj *key);
 robj *dbRandomKey(redisDb *db);
 int dbDelete(redisDb *db, robj *key);
+robj *dbUnshareStringValue(redisDb *db, robj *key, robj *o);
 long long emptyDb(void(callback)(void*));
 int selectDb(redisClient *c, int id);
 void signalModifiedKey(redisDb *db, robj *key);
@@ -1234,13 +1272,11 @@ void scanGenericCommand(redisClient *c, robj *o, unsigned long cursor);
 int parseScanCursorOrReply(redisClient *c, robj *o, unsigned long *cursor);
 
 /* API to get key arguments from commands */
-#define REDIS_GETKEYS_ALL 0
-#define REDIS_GETKEYS_PRELOAD 1
-int *getKeysFromCommand(struct redisCommand *cmd, robj **argv, int argc, int *numkeys, int flags);
+int *getKeysFromCommand(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
 void getKeysFreeResult(int *result);
-int *noPreloadGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags);
-int *renameGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags);
-int *zunionInterGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags);
+int *zunionInterGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys);
+int *evalGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
+int *sortGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
 
 /* Cluster */
 void clusterInit(void);
@@ -1417,8 +1453,14 @@ void scriptCommand(redisClient *c);
 void timeCommand(redisClient *c);
 void bitopCommand(redisClient *c);
 void bitcountCommand(redisClient *c);
+void bitposCommand(redisClient *c);
 void replconfCommand(redisClient *c);
 void waitCommand(redisClient *c);
+void pfselftestCommand(redisClient *c);
+void pfaddCommand(redisClient *c);
+void pfcountCommand(redisClient *c);
+void pfmergeCommand(redisClient *c);
+void pfgetregCommand(redisClient *c);
 
 #if defined(__GNUC__)
 void *calloc(size_t count, size_t size) __attribute__ ((deprecated));

src/scripting.c

@@ -958,6 +958,7 @@ void evalGenericCommand(redisClient *c, int evalsha) {
             rewriteClientCommandArgument(c,0,
                 resetRefCount(createStringObject("EVAL",4)));
             rewriteClientCommandArgument(c,1,script);
+            forceCommandPropagation(c,REDIS_PROPAGATE_REPL|REDIS_PROPAGATE_AOF);
         }
     }
 }

src/sds.c

@@ -292,24 +292,37 @@ sds sdscpy(sds s, const char *t) {
 /* Like sdscatpritf() but gets va_list instead of being variadic. */
 sds sdscatvprintf(sds s, const char *fmt, va_list ap) {
     va_list cpy;
-    char *buf, *t;
-    size_t buflen = 16;
+    char staticbuf[1024], *buf = staticbuf, *t;
+    size_t buflen = strlen(fmt)*2;
 
-    while(1) {
+    /* We try to start using a static buffer for speed.
+     * If not possible we revert to heap allocation. */
+    if (buflen > sizeof(staticbuf)) {
         buf = zmalloc(buflen);
         if (buf == NULL) return NULL;
+    } else {
+        buflen = sizeof(staticbuf);
+    }
+
+    /* Try with buffers two times bigger every time we fail to
+     * fit the string in the current buffer size. */
+    while(1) {
         buf[buflen-2] = '\0';
         va_copy(cpy,ap);
         vsnprintf(buf, buflen, fmt, cpy);
         if (buf[buflen-2] != '\0') {
-            zfree(buf);
+            if (buf != staticbuf) zfree(buf);
             buflen *= 2;
+            buf = zmalloc(buflen);
+            if (buf == NULL) return NULL;
             continue;
         }
         break;
     }
+
+    /* Finally concat the obtained string to the SDS string and return it. */
     t = sdscat(s, buf);
-    zfree(buf);
+    if (buf != staticbuf) zfree(buf);
     return t;
 }
 

src/sentinel.c

@@ -78,12 +78,13 @@ typedef struct sentinelAddr {
 #define SENTINEL_TILT_TRIGGER 2000
 #define SENTINEL_TILT_PERIOD (SENTINEL_PING_PERIOD*30)
 #define SENTINEL_DEFAULT_SLAVE_PRIORITY 100
-#define SENTINEL_SLAVE_RECONF_RETRY_PERIOD 10000
+#define SENTINEL_SLAVE_RECONF_TIMEOUT 10000
 #define SENTINEL_DEFAULT_PARALLEL_SYNCS 1
 #define SENTINEL_MIN_LINK_RECONNECT_PERIOD 15000
 #define SENTINEL_DEFAULT_FAILOVER_TIMEOUT (60*3*1000)
 #define SENTINEL_MAX_PENDING_COMMANDS 100
 #define SENTINEL_ELECTION_TIMEOUT 10000
+#define SENTINEL_MAX_DESYNC 1000
 
 /* Failover machine different states. */
 #define SENTINEL_FAILOVER_STATE_NONE 0  /* No failover in progress. */
@@ -128,6 +129,10 @@ typedef struct sentinelRedisInstance {
     mstime_t pc_last_activity; /* Last time we received any message. */
     mstime_t last_avail_time; /* Last time the instance replied to ping with
                                  a reply we consider valid. */
+    mstime_t last_ping_time;  /* Last time a pending ping was sent in the
+                                 context of the current command connection
+                                 with the instance. 0 if still not sent or
+                                 if pong already received. */
     mstime_t last_pong_time;  /* Last time the instance replied to ping,
                                  whatever the reply was. That's used to check
                                  if the link is idle and must be reconnected. */
@@ -327,6 +332,8 @@ void sentinelDiscardReplyCallback(redisAsyncContext *c, void *reply, void *privd
 int sentinelSendSlaveOf(sentinelRedisInstance *ri, char *host, int port);
 char *sentinelVoteLeader(sentinelRedisInstance *master, uint64_t req_epoch, char *req_runid, uint64_t *leader_epoch);
 void sentinelFlushConfig(void);
+void sentinelGenerateInitialMonitorEvents(void);
+int sentinelSendPing(sentinelRedisInstance *ri);
 
 /* ========================= Dictionary types =============================== */
 
@@ -369,6 +376,7 @@ dictType leaderVotesDictType = {
 void sentinelCommand(redisClient *c);
 void sentinelInfoCommand(redisClient *c);
 void sentinelSetCommand(redisClient *c);
+void sentinelPublishCommand(redisClient *c);
 
 struct redisCommand sentinelcmds[] = {
     {"ping",pingCommand,1,"",0,NULL,0,0,0,0,0},
@@ -377,6 +385,7 @@ struct redisCommand sentinelcmds[] = {
     {"unsubscribe",unsubscribeCommand,-1,"",0,NULL,0,0,0,0,0},
     {"psubscribe",psubscribeCommand,-2,"",0,NULL,0,0,0,0,0},
     {"punsubscribe",punsubscribeCommand,-1,"",0,NULL,0,0,0,0,0},
+    {"publish",sentinelPublishCommand,3,"",0,NULL,0,0,0,0,0},
     {"info",sentinelInfoCommand,-1,"",0,NULL,0,0,0,0,0},
     {"shutdown",shutdownCommand,-1,"",0,NULL,0,0,0,0,0}
 };
@@ -417,10 +426,20 @@ void initSentinel(void) {
 void sentinelIsRunning(void) {
     redisLog(REDIS_WARNING,"Sentinel runid is %s", server.runid);
 
-    if (server.configfile == NULL || access(server.configfile,W_OK) == -1) {
-        redisLog(REDIS_WARNING,"Sentinel started without a config file, or config file not writable. Exiting...");
+    if (server.configfile == NULL) {
+        redisLog(REDIS_WARNING,
+            "Sentinel started without a config file. Exiting...");
+        exit(1);
+    } else if (access(server.configfile,W_OK) == -1) {
+        redisLog(REDIS_WARNING,
+            "Sentinel config file %s is not writable: %s. Exiting...",
+            server.configfile,strerror(errno));
         exit(1);
     }
+
+    /* We want to generate a +monitor event for every configured master
+     * at startup. */
+    sentinelGenerateInitialMonitorEvents();
 }
 
 /* ============================== sentinelAddr ============================== */
@@ -552,6 +571,22 @@ void sentinelEvent(int level, char *type, sentinelRedisInstance *ri,
     }
 }
 
+/* This function is called only at startup and is used to generate a
+ * +monitor event for every configured master. The same events are also
+ * generated when a master to monitor is added at runtime via the
+ * SENTINEL MONITOR command. */
+void sentinelGenerateInitialMonitorEvents(void) {
+    dictIterator *di;
+    dictEntry *de;
+
+    di = dictGetIterator(sentinel.masters);
+    while((de = dictNext(di)) != NULL) {
+        sentinelRedisInstance *ri = dictGetVal(de);
+        sentinelEvent(REDIS_WARNING,"+monitor",ri,"%@ quorum %d",ri->quorum);
+    }
+    dictReleaseIterator(di);
+}
+
 /* ============================ script execution ============================ */
 
 /* Release a script job structure and all the associated data. */
@@ -895,6 +930,11 @@ sentinelRedisInstance *createSentinelRedisInstance(char *name, int flags, char *
     ri->cc_conn_time = 0;
     ri->pc_conn_time = 0;
     ri->pc_last_activity = 0;
+    /* We set the last_ping_time to "now" even if we actually don't have yet
+     * a connection with the node, nor we sent a ping.
+     * This is useful to detect a timeout in case we'll not be able to connect
+     * with the node at all. */
+    ri->last_ping_time = mstime();
     ri->last_avail_time = mstime();
     ri->last_pong_time = mstime();
     ri->last_pub_time = mstime();
@@ -1131,6 +1171,7 @@ void sentinelResetMaster(sentinelRedisInstance *ri, int flags) {
     sdsfree(ri->slave_master_host);
     ri->runid = NULL;
     ri->slave_master_host = NULL;
+    ri->last_ping_time = mstime();
     ri->last_avail_time = mstime();
     ri->last_pong_time = mstime();
     ri->role_reported_time = mstime();
@@ -1257,6 +1298,24 @@ sentinelAddr *sentinelGetCurrentMasterAddress(sentinelRedisInstance *master) {
     }
 }
 
+/* This function sets the down_after_period field value in 'master' to all
+ * the slaves and sentinel instances connected to this master. */
+void sentinelPropagateDownAfterPeriod(sentinelRedisInstance *master) {
+    dictIterator *di;
+    dictEntry *de;
+    int j;
+    dict *d[] = {master->slaves, master->sentinels, NULL};
+
+    for (j = 0; d[j]; j++) {
+        di = dictGetIterator(d[j]);
+        while((de = dictNext(di)) != NULL) {
+            sentinelRedisInstance *ri = dictGetVal(de);
+            ri->down_after_period = master->down_after_period;
+        }
+        dictReleaseIterator(di);
+    }
+}
+
 /* ============================ Config handling ============================= */
 char *sentinelHandleConfiguration(char **argv, int argc) {
     sentinelRedisInstance *ri;
@@ -1282,6 +1341,7 @@ char *sentinelHandleConfiguration(char **argv, int argc) {
         ri->down_after_period = atoi(argv[2]);
         if (ri->down_after_period <= 0)
             return "negative or zero time parameter.";
+        sentinelPropagateDownAfterPeriod(ri);
     } else if (!strcasecmp(argv[0],"failover-timeout") && argc == 3) {
         /* failover-timeout <name> <milliseconds> */
         ri = sentinelGetMasterByName(argv[1]);
@@ -1314,13 +1374,26 @@ char *sentinelHandleConfiguration(char **argv, int argc) {
         ri = sentinelGetMasterByName(argv[1]);
         if (!ri) return "No such master with specified name.";
         ri->auth_pass = sdsnew(argv[2]);
+    } else if (!strcasecmp(argv[0],"current-epoch") && argc == 2) {
+        /* current-epoch <epoch> */
+        unsigned long long current_epoch = strtoull(argv[1],NULL,10);
+        if (current_epoch > sentinel.current_epoch)
+            sentinel.current_epoch = current_epoch;
     } else if (!strcasecmp(argv[0],"config-epoch") && argc == 3) {
         /* config-epoch <name> <epoch> */
         ri = sentinelGetMasterByName(argv[1]);
         if (!ri) return "No such master with specified name.";
         ri->config_epoch = strtoull(argv[2],NULL,10);
+        /* The following update of current_epoch is not really useful as
+         * now the current epoch is persisted on the config file, but
+         * we leave this check here for redundancy. */
         if (ri->config_epoch > sentinel.current_epoch)
             sentinel.current_epoch = ri->config_epoch;
+    } else if (!strcasecmp(argv[0],"leader-epoch") && argc == 3) {
+        /* leader-epoch <name> <epoch> */
+        ri = sentinelGetMasterByName(argv[1]);
+        if (!ri) return "No such master with specified name.";
+        ri->leader_epoch = strtoull(argv[2],NULL,10);
     } else if (!strcasecmp(argv[0],"known-slave") && argc == 4) {
         sentinelRedisInstance *slave;
 
@@ -1359,13 +1432,13 @@ char *sentinelHandleConfiguration(char **argv, int argc) {
 void rewriteConfigSentinelOption(struct rewriteConfigState *state) {
     dictIterator *di, *di2;
     dictEntry *de;
+    sds line;
 
     /* For every master emit a "sentinel monitor" config entry. */
     di = dictGetIterator(sentinel.masters);
     while((de = dictNext(di)) != NULL) {
         sentinelRedisInstance *master, *ri;
         sentinelAddr *master_addr;
-        sds line;
 
         /* sentinel monitor */
         master = dictGetVal(de);
@@ -1429,6 +1502,12 @@ void rewriteConfigSentinelOption(struct rewriteConfigState *state) {
             master->name, (unsigned long long) master->config_epoch);
         rewriteConfigRewriteLine(state,"sentinel",line,1);
 
+        /* sentinel leader-epoch */
+        line = sdscatprintf(sdsempty(),
+            "sentinel leader-epoch %s %llu",
+            master->name, (unsigned long long) master->leader_epoch);
+        rewriteConfigRewriteLine(state,"sentinel",line,1);
+
         /* sentinel known-slave */
         di2 = dictGetIterator(master->slaves);
         while((de = dictNext(di2)) != NULL) {
@@ -1464,6 +1543,12 @@ void rewriteConfigSentinelOption(struct rewriteConfigState *state) {
         }
         dictReleaseIterator(di2);
     }
+
+    /* sentinel current-epoch is a global state valid for all the masters. */
+    line = sdscatprintf(sdsempty(),
+        "sentinel current-epoch %llu", (unsigned long long) sentinel.current_epoch);
+    rewriteConfigRewriteLine(state,"sentinel",line,1);
+
     dictReleaseIterator(di);
 }
 
@@ -1475,21 +1560,23 @@ void rewriteConfigSentinelOption(struct rewriteConfigState *state) {
  *
  * On failure the function logs a warning on the Redis log. */
 void sentinelFlushConfig(void) {
-    int fd;
+    int fd = -1;
     int saved_hz = server.hz;
+    int rewrite_status;
 
     server.hz = REDIS_DEFAULT_HZ;
-    if (rewriteConfig(server.configfile) != -1) {
-        /* Rewrite succeded, fsync it. */
-        if ((fd = open(server.configfile,O_RDONLY)) != -1) {
-            fsync(fd);
-            close(fd);
-        }
-    } else {
-        redisLog(REDIS_WARNING,"WARNING: Senitnel was not able to save the new configuration on disk!!!: %s", strerror(errno));
-    }
+    rewrite_status = rewriteConfig(server.configfile);
     server.hz = saved_hz;
+
+    if (rewrite_status == -1) goto werr;
+    if ((fd = open(server.configfile,O_RDONLY)) == -1) goto werr;
+    if (fsync(fd) == -1) goto werr;
+    if (close(fd) == EOF) goto werr;
     return;
+
+werr:
+    if (fd != -1) close(fd);
+    redisLog(REDIS_WARNING,"WARNING: Sentinel was not able to save the new configuration on disk!!!: %s", strerror(errno));
 }
 
 /* ====================== hiredis connection handling ======================= */
@@ -1560,6 +1647,23 @@ void sentinelSendAuthIfNeeded(sentinelRedisInstance *ri, redisAsyncContext *c) {
     }
 }
 
+/* Use CLIENT SETNAME to name the connection in the Redis instance as
+ * sentinel-<first_8_chars_of_runid>-<connection_type>
+ * The connection type is "cmd" or "pubsub" as specified by 'type'.
+ *
+ * This makes it possible to list all the sentinel instances connected
+ * to a Redis servewr with CLIENT LIST, grepping for a specific name format. */
+void sentinelSetClientName(sentinelRedisInstance *ri, redisAsyncContext *c, char *type) {
+    char name[64];
+
+    snprintf(name,sizeof(name),"sentinel-%.8s-%s",server.runid,type);
+    if (redisAsyncCommand(c, sentinelDiscardReplyCallback, NULL,
+        "CLIENT SETNAME %s", name) == REDIS_OK)
+    {
+        ri->pending_commands++;
+    }
+}
+
 /* Create the async connections for the specified instance if the instance
  * is disconnected. Note that the SRI_DISCONNECTED flag is set even if just
  * one of the two links (commands and pub/sub) is missing. */
@@ -1582,6 +1686,10 @@ void sentinelReconnectInstance(sentinelRedisInstance *ri) {
             redisAsyncSetDisconnectCallback(ri->cc,
                                             sentinelDisconnectCallback);
             sentinelSendAuthIfNeeded(ri,ri->cc);
+            sentinelSetClientName(ri,ri->cc,"cmd");
+
+            /* Send a PING ASAP when reconnecting. */
+            sentinelSendPing(ri);
         }
     }
     /* Pub / Sub */
@@ -1602,6 +1710,7 @@ void sentinelReconnectInstance(sentinelRedisInstance *ri) {
             redisAsyncSetDisconnectCallback(ri->pc,
                                             sentinelDisconnectCallback);
             sentinelSendAuthIfNeeded(ri,ri->pc);
+            sentinelSetClientName(ri,ri->pc,"pubsub");
             /* Now we subscribe to the Sentinels "Hello" channel. */
             retval = redisAsyncCommand(ri->pc,
                 sentinelReceiveHelloMessages, NULL, "SUBSCRIBE %s",
@@ -1765,8 +1874,20 @@ void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) {
         ri->role_reported_time = mstime();
         ri->role_reported = role;
         if (role == SRI_SLAVE) ri->slave_conf_change_time = mstime();
+        /* Log the event with +role-change if the new role is coherent or
+         * with -role-change if there is a mismatch with the current config. */
+        sentinelEvent(REDIS_VERBOSE,
+            ((ri->flags & (SRI_MASTER|SRI_SLAVE)) == role) ?
+            "+role-change" : "-role-change",
+            ri, "%@ new reported role is %s",
+            role == SRI_MASTER ? "master" : "slave",
+            ri->flags & SRI_MASTER ? "master" : "slave");
     }
 
+    /* None of the following conditions are processed when in tilt mode, so
+     * return asap. */
+    if (sentinel.tilt) return;
+
     /* Handle master -> slave role switch. */
     if ((ri->flags & SRI_MASTER) && role == SRI_SLAVE) {
         /* Nothing to do, but masters claiming to be slaves are
@@ -1778,8 +1899,7 @@ void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) {
     if ((ri->flags & SRI_SLAVE) && role == SRI_MASTER) {
         /* If this is a promoted slave we can change state to the
          * failover state machine. */
-        if (!sentinel.tilt &&
-            (ri->master->flags & SRI_FAILOVER_IN_PROGRESS) &&
+        if ((ri->master->flags & SRI_FAILOVER_IN_PROGRESS) &&
             (ri->master->failover_state ==
                 SENTINEL_FAILOVER_STATE_WAIT_PROMOTION))
         {
@@ -1797,7 +1917,7 @@ void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) {
                 ri->master,"%@");
             sentinelCallClientReconfScript(ri->master,SENTINEL_LEADER,
                 "start",ri->master->addr,ri->addr);
-        } else if (!sentinel.tilt) {
+        } else {
             /* A slave turned into a master. We want to force our view and
              * reconfigure as slave. Wait some time after the change before
              * going forward, to receive new configs if any. */
@@ -1817,7 +1937,7 @@ void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) {
     }
 
     /* Handle slaves replicating to a different master address. */
-    if ((ri->flags & SRI_SLAVE) && !sentinel.tilt &&
+    if ((ri->flags & SRI_SLAVE) &&
         role == SRI_SLAVE &&
         (ri->slave_master_port != ri->master->addr->port ||
          strcasecmp(ri->slave_master_host,ri->master->addr->ip)))
@@ -1838,10 +1958,6 @@ void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) {
         }
     }
 
-    /* None of the following conditions are processed when in tilt mode, so
-     * return asap. */
-    if (sentinel.tilt) return;
-
     /* Detect if the slave that is in the process of being reconfigured
      * changed state. */
     if ((ri->flags & SRI_SLAVE) && role == SRI_SLAVE &&
@@ -1908,6 +2024,7 @@ void sentinelPingReplyCallback(redisAsyncContext *c, void *reply, void *privdata
             strncmp(r->str,"MASTERDOWN",10) == 0)
         {
             ri->last_avail_time = mstime();
+            ri->last_ping_time = 0; /* Flag the pong as received. */
         } else {
             /* Send a SCRIPT KILL command if the instance appears to be
              * down because of a busy script. */
@@ -1942,17 +2059,107 @@ void sentinelPublishReplyCallback(redisAsyncContext *c, void *reply, void *privd
         ri->last_pub_time = mstime();
 }
 
+/* Process an hello message received via Pub/Sub in master or slave instance,
+ * or sent directly to this sentinel via the (fake) PUBLISH command of Sentinel.
+ *
+ * If the master name specified in the message is not known, the message is
+ * discareded. */
+void sentinelProcessHelloMessage(char *hello, int hello_len) {
+    /* Format is composed of 8 tokens:
+     * 0=ip,1=port,2=runid,3=current_epoch,4=master_name,
+     * 5=master_ip,6=master_port,7=master_config_epoch. */
+    int numtokens, port, removed, master_port;
+    uint64_t current_epoch, master_config_epoch;
+    char **token = sdssplitlen(hello, hello_len, ",", 1, &numtokens);
+    sentinelRedisInstance *si, *master;
+
+    if (numtokens == 8) {
+        /* Obtain a reference to the master this hello message is about */
+        master = sentinelGetMasterByName(token[4]);
+        if (!master) goto cleanup; /* Unknown master, skip the message. */
+
+        /* First, try to see if we already have this sentinel. */
+        port = atoi(token[1]);
+        master_port = atoi(token[6]);
+        si = getSentinelRedisInstanceByAddrAndRunID(
+                        master->sentinels,token[0],port,token[2]);
+        current_epoch = strtoull(token[3],NULL,10);
+        master_config_epoch = strtoull(token[7],NULL,10);
+
+        if (!si) {
+            /* If not, remove all the sentinels that have the same runid
+             * OR the same ip/port, because it's either a restart or a
+             * network topology change. */
+            removed = removeMatchingSentinelsFromMaster(master,token[0],port,
+                            token[2]);
+            if (removed) {
+                sentinelEvent(REDIS_NOTICE,"-dup-sentinel",master,
+                    "%@ #duplicate of %s:%d or %s",
+                    token[0],port,token[2]);
+            }
+
+            /* Add the new sentinel. */
+            si = createSentinelRedisInstance(NULL,SRI_SENTINEL,
+                            token[0],port,master->quorum,master);
+            if (si) {
+                sentinelEvent(REDIS_NOTICE,"+sentinel",si,"%@");
+                /* The runid is NULL after a new instance creation and
+                 * for Sentinels we don't have a later chance to fill it,
+                 * so do it now. */
+                si->runid = sdsnew(token[2]);
+                sentinelFlushConfig();
+            }
+        }
+
+        /* Update local current_epoch if received current_epoch is greater.*/
+        if (current_epoch > sentinel.current_epoch) {
+            sentinel.current_epoch = current_epoch;
+            sentinelFlushConfig();
+            sentinelEvent(REDIS_WARNING,"+new-epoch",master,"%llu",
+                (unsigned long long) sentinel.current_epoch);
+        }
+
+        /* Update master info if received configuration is newer. */
+        if (master->config_epoch < master_config_epoch) {
+            master->config_epoch = master_config_epoch;
+            if (master_port != master->addr->port ||
+                strcmp(master->addr->ip, token[5]))
+            {
+                sentinelAddr *old_addr;
+
+                sentinelEvent(REDIS_WARNING,"+switch-master",
+                    master,"%s %s %d %s %d",
+                    master->name,
+                    master->addr->ip, master->addr->port,
+                    token[5], master_port);
+
+                old_addr = dupSentinelAddr(master->addr);
+                sentinelResetMasterAndChangeAddress(master, token[5], master_port);
+                sentinelCallClientReconfScript(master,
+                    SENTINEL_OBSERVER,"start",
+                    old_addr,master->addr);
+                releaseSentinelAddr(old_addr);
+            }
+        }
+
+        /* Update the state of the Sentinel. */
+        if (si) si->last_hello_time = mstime();
+    }
+
+cleanup:
+    sdsfreesplitres(token,numtokens);
+}
+
+
 /* This is our Pub/Sub callback for the Hello channel. It's useful in order
  * to discover other sentinels attached at the same master. */
 void sentinelReceiveHelloMessages(redisAsyncContext *c, void *reply, void *privdata) {
-    sentinelRedisInstance *ri = c->data, *master;
+    sentinelRedisInstance *ri = c->data;
     redisReply *r;
 
     if (!reply || !ri) return;
     r = reply;
 
-    master = (ri->flags & SRI_MASTER) ? ri : ri->master;
-
     /* Update the last activity in the pubsub channel. Note that since we
      * receive our messages as well this timestamp can be used to detect
      * if the link is probably disconnected even if it seems otherwise. */
@@ -1970,90 +2177,7 @@ void sentinelReceiveHelloMessages(redisAsyncContext *c, void *reply, void *privd
     /* We are not interested in meeting ourselves */
     if (strstr(r->element[2]->str,server.runid) != NULL) return;
 
-    {
-        /* Format is composed of 8 tokens:
-         * 0=ip,1=port,2=runid,3=current_epoch,4=master_name,
-         * 5=master_ip,6=master_port,7=master_config_epoch. */
-        int numtokens, port, removed, master_port;
-        uint64_t current_epoch, master_config_epoch;
-        char **token = sdssplitlen(r->element[2]->str,
-                                   r->element[2]->len,
-                                   ",",1,&numtokens);
-        sentinelRedisInstance *si;
-
-        if (numtokens == 8) {
-            /* First, try to see if we already have this sentinel. */
-            port = atoi(token[1]);
-            master_port = atoi(token[6]);
-            si = getSentinelRedisInstanceByAddrAndRunID(
-                            master->sentinels,token[0],port,token[2]);
-            current_epoch = strtoull(token[3],NULL,10);
-            master_config_epoch = strtoull(token[7],NULL,10);
-            sentinelRedisInstance *msgmaster;
-
-            if (!si) {
-                /* If not, remove all the sentinels that have the same runid
-                 * OR the same ip/port, because it's either a restart or a
-                 * network topology change. */
-                removed = removeMatchingSentinelsFromMaster(master,token[0],port,
-                                token[2]);
-                if (removed) {
-                    sentinelEvent(REDIS_NOTICE,"-dup-sentinel",master,
-                        "%@ #duplicate of %s:%d or %s",
-                        token[0],port,token[2]);
-                }
-
-                /* Add the new sentinel. */
-                si = createSentinelRedisInstance(NULL,SRI_SENTINEL,
-                                token[0],port,master->quorum,master);
-                if (si) {
-                    sentinelEvent(REDIS_NOTICE,"+sentinel",si,"%@");
-                    /* The runid is NULL after a new instance creation and
-                     * for Sentinels we don't have a later chance to fill it,
-                     * so do it now. */
-                    si->runid = sdsnew(token[2]);
-                    sentinelFlushConfig();
-                }
-            }
-
-            /* Update local current_epoch if received current_epoch is greater.*/
-            if (current_epoch > sentinel.current_epoch) {
-                sentinel.current_epoch = current_epoch;
-                sentinelEvent(REDIS_WARNING,"+new-epoch",ri,"%llu",
-                    (unsigned long long) sentinel.current_epoch);
-            }
-
-            /* Update master info if received configuration is newer. */
-            if ((msgmaster = sentinelGetMasterByName(token[4])) != NULL) {
-                if (msgmaster->config_epoch < master_config_epoch) {
-                    msgmaster->config_epoch = master_config_epoch;
-                    if (master_port != msgmaster->addr->port ||
-                        strcmp(msgmaster->addr->ip, token[5]))
-                    {
-                        sentinelAddr *old_addr;
-
-                        sentinelEvent(REDIS_WARNING,"+switch-master",
-                            msgmaster,"%s %s %d %s %d",
-                            msgmaster->name,
-                            msgmaster->addr->ip, msgmaster->addr->port,
-                            token[5], master_port);
-
-                        old_addr = dupSentinelAddr(msgmaster->addr);
-                        sentinelResetMasterAndChangeAddress(msgmaster,
-                                                    token[5], master_port);
-                        sentinelCallClientReconfScript(msgmaster,
-                            SENTINEL_OBSERVER,"start",
-                            old_addr,msgmaster->addr);
-                        releaseSentinelAddr(old_addr);
-                    }
-                }
-            }
-
-            /* Update the state of the Sentinel. */
-            if (si) si->last_hello_time = mstime();
-        }
-        sdsfreesplitres(token,numtokens);
-    }
+    sentinelProcessHelloMessage(r->element[2]->str, r->element[2]->len);
 }
 
 /* Send an "Hello" message via Pub/Sub to the specified 'ri' Redis
@@ -2095,11 +2219,31 @@ int sentinelSendHello(sentinelRedisInstance *ri) {
     return REDIS_OK;
 }
 
+/* Send a PING to the specified instance and refresh the last_ping_time
+ * if it is zero (that is, if we received a pong for the previous ping).
+ *
+ * On error zero is returned, and we can't consider the PING command
+ * queued in the connection. */
+int sentinelSendPing(sentinelRedisInstance *ri) {
+    int retval = redisAsyncCommand(ri->cc,
+        sentinelPingReplyCallback, NULL, "PING");
+    if (retval == REDIS_OK) {
+        ri->pending_commands++;
+        /* We update the ping time only if we received the pong for
+         * the previous ping, otherwise we are technically waiting
+         * since the first ping that did not received a reply. */
+        if (ri->last_ping_time == 0) ri->last_ping_time = mstime();
+        return 1;
+    } else {
+        return 0;
+    }
+}
+
 /* Send periodic PING, INFO, and PUBLISH to the Hello channel to
  * the specified master or slave instance. */
-void sentinelPingInstance(sentinelRedisInstance *ri) {
+void sentinelSendPeriodicCommands(sentinelRedisInstance *ri) {
     mstime_t now = mstime();
-    mstime_t info_period;
+    mstime_t info_period, ping_period;
     int retval;
 
     /* Return ASAP if we have already a PING or INFO already pending, or
@@ -2125,6 +2269,12 @@ void sentinelPingInstance(sentinelRedisInstance *ri) {
         info_period = SENTINEL_INFO_PERIOD;
     }
 
+    /* We ping instances every time the last received pong is older than
+     * the configured 'down-after-milliseconds' time, but every second
+     * anyway if 'down-after-milliseconds' is greater than 1 second. */
+    ping_period = ri->down_after_period;
+    if (ping_period > SENTINEL_PING_PERIOD) ping_period = SENTINEL_PING_PERIOD;
+
     if ((ri->flags & SRI_SENTINEL) == 0 &&
         (ri->info_refresh == 0 ||
         (now - ri->info_refresh) > info_period))
@@ -2132,18 +2282,12 @@ void sentinelPingInstance(sentinelRedisInstance *ri) {
         /* Send INFO to masters and slaves, not sentinels. */
         retval = redisAsyncCommand(ri->cc,
             sentinelInfoReplyCallback, NULL, "INFO");
-        if (retval != REDIS_OK) return;
-        ri->pending_commands++;
-    } else if ((now - ri->last_pong_time) > SENTINEL_PING_PERIOD) {
+        if (retval == REDIS_OK) ri->pending_commands++;
+    } else if ((now - ri->last_pong_time) > ping_period) {
         /* Send PING to all the three kinds of instances. */
-        retval = redisAsyncCommand(ri->cc,
-            sentinelPingReplyCallback, NULL, "PING");
-        if (retval != REDIS_OK) return;
-        ri->pending_commands++;
-    } else if ((ri->flags & SRI_SENTINEL) == 0 &&
-               (now - ri->last_pub_time) > SENTINEL_PUBLISH_PERIOD)
-    {
-        /* PUBLISH hello messages to masters and slaves. */
+        sentinelSendPing(ri);
+    } else if ((now - ri->last_pub_time) > SENTINEL_PUBLISH_PERIOD) {
+        /* PUBLISH hello messages to all the three kinds of instances. */
         sentinelSendHello(ri);
     }
 }
@@ -2217,6 +2361,11 @@ void addReplySentinelRedisInstance(redisClient *c, sentinelRedisInstance *ri) {
         fields++;
     }
 
+    addReplyBulkCString(c,"last-ping-sent");
+    addReplyBulkLongLong(c,
+        ri->last_ping_time ? (mstime() - ri->last_ping_time) : 0);
+    fields++;
+
     addReplyBulkCString(c,"last-ok-ping-reply");
     addReplyBulkLongLong(c,mstime() - ri->last_avail_time);
     fields++;
@@ -2237,6 +2386,10 @@ void addReplySentinelRedisInstance(redisClient *c, sentinelRedisInstance *ri) {
         fields++;
     }
 
+    addReplyBulkCString(c,"down-after-milliseconds");
+    addReplyBulkLongLong(c,ri->down_after_period);
+    fields++;
+
     /* Masters and Slaves */
     if (ri->flags & (SRI_MASTER|SRI_SLAVE)) {
         addReplyBulkCString(c,"info-refresh");
@@ -2271,10 +2424,6 @@ void addReplySentinelRedisInstance(redisClient *c, sentinelRedisInstance *ri) {
         addReplyBulkLongLong(c,ri->quorum);
         fields++;
 
-        addReplyBulkCString(c,"down-after-milliseconds");
-        addReplyBulkLongLong(c,ri->down_after_period);
-        fields++;
-
         addReplyBulkCString(c,"failover-timeout");
         addReplyBulkLongLong(c,ri->failover_timeout);
         fields++;
@@ -2455,8 +2604,6 @@ void sentinelCommand(redisClient *c) {
         ri = sentinelGetMasterByName(c->argv[2]->ptr);
         if (ri == NULL) {
             addReply(c,shared.nullmultibulk);
-        } else if (ri->info_refresh == 0) {
-            addReplySds(c,sdsnew("-IDONTKNOW I have not enough information to reply. Please ask another Sentinel.\r\n"));
         } else {
             sentinelAddr *addr = sentinelGetCurrentMasterAddress(ri);
 
@@ -2491,6 +2638,7 @@ void sentinelCommand(redisClient *c) {
         sentinelPendingScriptsCommand(c);
     } else if (!strcasecmp(c->argv[1]->ptr,"monitor")) {
         /* SENTINEL MONITOR <name> <ip> <port> <quorum> */
+        sentinelRedisInstance *ri;
         long quorum, port;
         char buf[32];
 
@@ -2506,9 +2654,11 @@ void sentinelCommand(redisClient *c) {
             addReplyError(c,"Invalid IP address specified");
             return;
         }
-        if (createSentinelRedisInstance(c->argv[2]->ptr,SRI_MASTER,
-                c->argv[3]->ptr,port,quorum,NULL) == NULL)
-        {
+
+        /* Parameters are valid. Try to create the master instance. */
+        ri = createSentinelRedisInstance(c->argv[2]->ptr,SRI_MASTER,
+                c->argv[3]->ptr,port,quorum,NULL);
+        if (ri == NULL) {
             switch(errno) {
             case EBUSY:
                 addReplyError(c,"Duplicated master name");
@@ -2522,6 +2672,7 @@ void sentinelCommand(redisClient *c) {
             }
         } else {
             sentinelFlushConfig();
+            sentinelEvent(REDIS_WARNING,"+monitor",ri,"%@ quorum %d",ri->quorum);
             addReply(c,shared.ok);
         }
     } else if (!strcasecmp(c->argv[1]->ptr,"remove")) {
@@ -2530,6 +2681,7 @@ void sentinelCommand(redisClient *c) {
 
         if ((ri = sentinelGetMasterByNameOrReplyError(c,c->argv[2]))
             == NULL) return;
+        sentinelEvent(REDIS_WARNING,"-monitor",ri,"%@");
         dictDelete(sentinel.masters,c->argv[2]->ptr);
         sentinelFlushConfig();
         addReply(c,shared.ok);
@@ -2628,6 +2780,7 @@ void sentinelSetCommand(redisClient *c) {
             if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll <= 0)
                 goto badfmt;
             ri->down_after_period = ll;
+            sentinelPropagateDownAfterPeriod(ri);
             changes++;
         } else if (!strcasecmp(option,"failover-timeout")) {
             /* failover-timeout <milliseconds> */
@@ -2681,6 +2834,7 @@ void sentinelSetCommand(redisClient *c) {
             if (changes) sentinelFlushConfig();
             return;
         }
+        sentinelEvent(REDIS_WARNING,"+set",ri,"%@ %s %s",option,value);
     }
 
     if (changes) sentinelFlushConfig();
@@ -2693,20 +2847,42 @@ badfmt: /* Bad format errors */
             value, option);
 }
 
+/* Our fake PUBLISH command: it is actually useful only to receive hello messages
+ * from the other sentinel instances, and publishing to a channel other than
+ * SENTINEL_HELLO_CHANNEL is forbidden.
+ *
+ * Because we have a Sentinel PUBLISH, the code to send hello messages is the same
+ * for all the three kind of instances: masters, slaves, sentinels. */
+void sentinelPublishCommand(redisClient *c) {
+    if (strcmp(c->argv[1]->ptr,SENTINEL_HELLO_CHANNEL)) {
+        addReplyError(c, "Only HELLO messages are accepted by Sentinel instances.");
+        return;
+    }
+    sentinelProcessHelloMessage(c->argv[2]->ptr,sdslen(c->argv[2]->ptr));
+    addReplyLongLong(c,1);
+}
+
 /* ===================== SENTINEL availability checks ======================= */
 
 /* Is this instance down from our point of view? */
 void sentinelCheckSubjectivelyDown(sentinelRedisInstance *ri) {
-    mstime_t elapsed = mstime() - ri->last_avail_time;
+    mstime_t elapsed = 0;
+
+    if (ri->last_ping_time)
+        elapsed = mstime() - ri->last_ping_time;
 
     /* Check if we are in need for a reconnection of one of the 
      * links, because we are detecting low activity.
      *
      * 1) Check if the command link seems connected, was connected not less
-     *    than SENTINEL_MIN_LINK_RECONNECT_PERIOD, but still we have an
-     *    idle time that is greater than down_after_period / 2 seconds. */
+     *    than SENTINEL_MIN_LINK_RECONNECT_PERIOD, but still we have a
+     *    pending ping for more than half the timeout. */
     if (ri->cc &&
         (mstime() - ri->cc_conn_time) > SENTINEL_MIN_LINK_RECONNECT_PERIOD &&
+        ri->last_ping_time != 0 && /* Ther is a pending ping... */
+        /* The pending ping is delayed, and we did not received
+         * error replies as well. */
+        (mstime() - ri->last_ping_time) > (ri->down_after_period/2) &&
         (mstime() - ri->last_pong_time) > (ri->down_after_period/2))
     {
         sentinelKillLink(ri,ri->cc);
@@ -2889,6 +3065,7 @@ void sentinelAskMasterStateToOtherSentinels(sentinelRedisInstance *master, int f
 char *sentinelVoteLeader(sentinelRedisInstance *master, uint64_t req_epoch, char *req_runid, uint64_t *leader_epoch) {
     if (req_epoch > sentinel.current_epoch) {
         sentinel.current_epoch = req_epoch;
+        sentinelFlushConfig();
         sentinelEvent(REDIS_WARNING,"+new-epoch",master,"%llu",
             (unsigned long long) sentinel.current_epoch);
     }
@@ -2898,13 +3075,14 @@ char *sentinelVoteLeader(sentinelRedisInstance *master, uint64_t req_epoch, char
         sdsfree(master->leader);
         master->leader = sdsnew(req_runid);
         master->leader_epoch = sentinel.current_epoch;
+        sentinelFlushConfig();
         sentinelEvent(REDIS_WARNING,"+vote-for-leader",master,"%s %llu",
             master->leader, (unsigned long long) master->leader_epoch);
         /* If we did not voted for ourselves, set the master failover start
          * time to now, in order to force a delay before we can start a
          * failover for the same master. */
         if (strcasecmp(master->leader,server.runid))
-            master->failover_start_time = mstime();
+            master->failover_start_time = mstime()+rand()%SENTINEL_MAX_DESYNC;
     }
 
     *leader_epoch = master->leader_epoch;
@@ -3049,7 +3227,7 @@ void sentinelStartFailover(sentinelRedisInstance *master) {
     sentinelEvent(REDIS_WARNING,"+new-epoch",master,"%llu",
         (unsigned long long) sentinel.current_epoch);
     sentinelEvent(REDIS_WARNING,"+try-failover",master,"%@");
-    master->failover_start_time = mstime();
+    master->failover_start_time = mstime()+rand()%SENTINEL_MAX_DESYNC;
     master->failover_state_change_time = mstime();
 }
 
@@ -3359,14 +3537,17 @@ void sentinelFailoverReconfNextSlave(sentinelRedisInstance *master) {
         /* Skip the promoted slave, and already configured slaves. */
         if (slave->flags & (SRI_PROMOTED|SRI_RECONF_DONE)) continue;
 
-        /* Clear the SRI_RECONF_SENT flag if too much time elapsed without
-         * the slave moving forward to the next state. */
+        /* If too much time elapsed without the slave moving forward to
+         * the next state, consider it reconfigured even if it is not.
+         * Sentinels will detect the slave as misconfigured and fix its
+         * configuration later. */
         if ((slave->flags & SRI_RECONF_SENT) &&
             (mstime() - slave->slave_reconf_sent_time) >
-            SENTINEL_SLAVE_RECONF_RETRY_PERIOD)
+            SENTINEL_SLAVE_RECONF_TIMEOUT)
         {
             sentinelEvent(REDIS_NOTICE,"-slave-reconf-sent-timeout",slave,"%@");
             slave->flags &= ~SRI_RECONF_SENT;
+            slave->flags |= SRI_RECONF_DONE;
         }
 
         /* Nothing to do for instances that are disconnected or already
@@ -3457,7 +3638,7 @@ void sentinelHandleRedisInstance(sentinelRedisInstance *ri) {
     /* ========== MONITORING HALF ============ */
     /* Every kind of instance */
     sentinelReconnectInstance(ri);
-    sentinelPingInstance(ri);
+    sentinelSendPeriodicCommands(ri);
 
     /* ============== ACTING HALF ============= */
     /* We don't proceed with the acting half if we are in TILT mode.

src/slowlog.c

@@ -63,15 +63,17 @@ slowlogEntry *slowlogCreateEntry(robj **argv, int argc, long long duration) {
         } else {
             /* Trim too long strings as well... */
             if (argv[j]->type == REDIS_STRING &&
-                sdsEncodedObject(argv[j]) &&
-                sdslen(argv[j]->ptr) > SLOWLOG_ENTRY_MAX_STRING)
+                (sdsEncodedObject(argv[j]) || lzfEncodedObject(argv[j])) &&
+                stringObjectLen(argv[j]) > SLOWLOG_ENTRY_MAX_STRING)
             {
-                sds s = sdsnewlen(argv[j]->ptr, SLOWLOG_ENTRY_MAX_STRING);
+                robj *o = getDecodedObject(argv[j]);
+                sds s = sdsnewlen(o->ptr, SLOWLOG_ENTRY_MAX_STRING);
 
                 s = sdscatprintf(s,"... (%lu more bytes)",
                     (unsigned long)
-                    sdslen(argv[j]->ptr) - SLOWLOG_ENTRY_MAX_STRING);
+                    stringObjectLen(argv[j]) - SLOWLOG_ENTRY_MAX_STRING);
                 se->argv[j] = createObject(REDIS_STRING,s);
+                decrRefCount(o);
             } else {
                 se->argv[j] = argv[j];
                 incrRefCount(argv[j]);

src/sort.c

@@ -194,6 +194,7 @@ void sortCommand(redisClient *c) {
     int j, dontsort = 0, vectorlen;
     int getop = 0; /* GET operation counter */
     int int_convertion_error = 0;
+    int syntax_error = 0;
     robj *sortval, *sortby = NULL, *storekey = NULL;
     redisSortObject *vector; /* Resulting vector to sort */
 
@@ -231,8 +232,14 @@ void sortCommand(redisClient *c) {
         } else if (!strcasecmp(c->argv[j]->ptr,"alpha")) {
             alpha = 1;
         } else if (!strcasecmp(c->argv[j]->ptr,"limit") && leftargs >= 2) {
-            if ((getLongFromObjectOrReply(c, c->argv[j+1], &limit_start, NULL) != REDIS_OK) ||
-                (getLongFromObjectOrReply(c, c->argv[j+2], &limit_count, NULL) != REDIS_OK)) return;
+            if ((getLongFromObjectOrReply(c, c->argv[j+1], &limit_start, NULL)
+                 != REDIS_OK) ||
+                (getLongFromObjectOrReply(c, c->argv[j+2], &limit_count, NULL)
+                 != REDIS_OK))
+            {
+                syntax_error++;
+                break;
+            }
             j+=2;
         } else if (!strcasecmp(c->argv[j]->ptr,"store") && leftargs >= 1) {
             storekey = c->argv[j+1];
@@ -241,22 +248,43 @@ void sortCommand(redisClient *c) {
             sortby = c->argv[j+1];
             /* If the BY pattern does not contain '*', i.e. it is constant,
              * we don't need to sort nor to lookup the weight keys. */
-            if (strchr(c->argv[j+1]->ptr,'*') == NULL) dontsort = 1;
+            if (strchr(c->argv[j+1]->ptr,'*') == NULL) {
+                dontsort = 1;
+            } else {
+                /* If BY is specified with a real patter, we can't accept
+                 * it in cluster mode. */
+                if (server.cluster_enabled) {
+                    addReplyError(c,"BY option of SORT denied in Cluster mode.");
+                    syntax_error++;
+                    break;
+                }
+            }
             j++;
         } else if (!strcasecmp(c->argv[j]->ptr,"get") && leftargs >= 1) {
+            if (server.cluster_enabled) {
+                addReplyError(c,"GET option of SORT denied in Cluster mode.");
+                syntax_error++;
+                break;
+            }
             listAddNodeTail(operations,createSortOperation(
                 REDIS_SORT_GET,c->argv[j+1]));
             getop++;
             j++;
         } else {
-            decrRefCount(sortval);
-            listRelease(operations);
             addReply(c,shared.syntaxerr);
-            return;
+            syntax_error++;
+            break;
         }
         j++;
     }
 
+    /* Handle syntax errors set during options parsing. */
+    if (syntax_error) {
+        decrRefCount(sortval);
+        listRelease(operations);
+        return;
+    }
+
     /* For the STORE option, or when SORT is called from a Lua script,
      * we want to force a specific ordering even when no explicit ordering
      * was asked (SORT BY nosort). This guarantees that replication / AOF

src/t_string.c

@@ -215,12 +215,7 @@ void setrangeCommand(redisClient *c) {
             return;
 
         /* Create a copy when the object is shared or encoded. */
-        if (o->refcount != 1 || o->encoding != REDIS_ENCODING_RAW) {
-            robj *decoded = getDecodedObject(o);
-            o = createRawStringObject(decoded->ptr, sdslen(decoded->ptr));
-            decrRefCount(decoded);
-            dbOverwrite(c->db,c->argv[1],o);
-        }
+        o = dbUnshareStringValue(c->db,c->argv[1],o);
     }
 
     if (sdslen(value) > 0) {
@@ -247,6 +242,7 @@ void getrangeCommand(redisClient *c) {
     if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.emptybulk)) == NULL ||
         checkType(c,o,REDIS_STRING)) return;
 
+    if (lzfEncodedObject(o)) o = dbUnshareStringValue(c->db,c->argv[1],o);
     if (o->encoding == REDIS_ENCODING_INT) {
         str = llbuf;
         strlen = ll2string(llbuf,sizeof(llbuf),(long)o->ptr);
@@ -433,15 +429,8 @@ void appendCommand(redisClient *c) {
         if (checkStringLength(c,totlen) != REDIS_OK)
             return;
 
-        /* If the object is shared or encoded, we have to make a copy */
-        if (o->refcount != 1 || o->encoding != REDIS_ENCODING_RAW) {
-            robj *decoded = getDecodedObject(o);
-            o = createRawStringObject(decoded->ptr, sdslen(decoded->ptr));
-            decrRefCount(decoded);
-            dbOverwrite(c->db,c->argv[1],o);
-        }
-
         /* Append the value */
+        o = dbUnshareStringValue(c->db,c->argv[1],o);
         o->ptr = sdscatlen(o->ptr,append->ptr,sdslen(append->ptr));
         totlen = sdslen(o->ptr);
     }

src/zmalloc.c

@@ -321,8 +321,8 @@ size_t zmalloc_get_rss(void) {
 #endif
 
 /* Fragmentation = RSS / allocated-bytes */
-float zmalloc_get_fragmentation_ratio(void) {
-    return (float)zmalloc_get_rss()/zmalloc_used_memory();
+float zmalloc_get_fragmentation_ratio(size_t rss) {
+    return (float)rss/zmalloc_used_memory();
 }
 
 #if defined(HAVE_PROC_SMAPS)

src/zmalloc.h

@@ -73,7 +73,7 @@ char *zstrdup(const char *s);
 size_t zmalloc_used_memory(void);
 void zmalloc_enable_thread_safeness(void);
 void zmalloc_set_oom_handler(void (*oom_handler)(size_t));
-float zmalloc_get_fragmentation_ratio(void);
+float zmalloc_get_fragmentation_ratio(size_t rss);
 size_t zmalloc_get_rss(void);
 size_t zmalloc_get_private_dirty(void);
 void zlibc_free(void *ptr);

tests/sentinel-tests/00-base.tcl

@@ -0,0 +1,126 @@
+# Check the basic monitoring and failover capabilities.
+
+source "../sentinel-tests/includes/init-tests.tcl"
+
+if {$::simulate_error} {
+    test "This test will fail" {
+        fail "Simulated error"
+    }
+}
+
+test "Basic failover works if the master is down" {
+    set old_port [RI $master_id tcp_port]
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    assert {[lindex $addr 1] == $old_port}
+    kill_instance redis $master_id
+    foreach_sentinel_id id {
+        wait_for_condition 1000 50 {
+            [lindex [S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port
+        } else {
+            fail "At least one Sentinel did not received failover info"
+        }
+    }
+    restart_instance redis $master_id
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    set master_id [get_instance_id_by_port redis [lindex $addr 1]]
+}
+
+test "New master [join $addr {:}] role matches" {
+    assert {[RI $master_id role] eq {master}}
+}
+
+test "All the other slaves now point to the new master" {
+    foreach_redis_id id {
+        if {$id != $master_id && $id != 0} {
+            wait_for_condition 1000 50 {
+                [RI $id master_port] == [lindex $addr 1]
+            } else {
+                fail "Redis ID $id not configured to replicate with new master"
+            }
+        }
+    }
+}
+
+test "The old master eventually gets reconfigured as a slave" {
+    wait_for_condition 1000 50 {
+        [RI 0 master_port] == [lindex $addr 1]
+    } else {
+        fail "Old master not reconfigured as slave of new master"
+    }
+}
+
+test "ODOWN is not possible without N (quorum) Sentinels reports" {
+    foreach_sentinel_id id {
+        S $id SENTINEL SET mymaster quorum [expr $sentinels+1]
+    }
+    set old_port [RI $master_id tcp_port]
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    assert {[lindex $addr 1] == $old_port}
+    kill_instance redis $master_id
+
+    # Make sure failover did not happened.
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    assert {[lindex $addr 1] == $old_port}
+    restart_instance redis $master_id
+}
+
+test "Failover is not possible without majority agreement" {
+    foreach_sentinel_id id {
+        S $id SENTINEL SET mymaster quorum $quorum
+    }
+
+    # Crash majority of sentinels
+    for {set id 0} {$id < $quorum} {incr id} {
+        kill_instance sentinel $id
+    }
+
+    # Kill the current master
+    kill_instance redis $master_id
+
+    # Make sure failover did not happened.
+    set addr [S $quorum SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    assert {[lindex $addr 1] == $old_port}
+    restart_instance redis $master_id
+
+    # Cleanup: restart Sentinels to monitor the master.
+    for {set id 0} {$id < $quorum} {incr id} {
+        restart_instance sentinel $id
+    }
+}
+
+test "Failover works if we configure for absolute agreement" {
+    foreach_sentinel_id id {
+        S $id SENTINEL SET mymaster quorum $sentinels
+    }
+
+    # Wait for Sentinels to monitor the master again
+    foreach_sentinel_id id {
+        wait_for_condition 1000 50 {
+            [dict get [S $id SENTINEL MASTER mymaster] info-refresh] < 100000
+        } else {
+            fail "At least one Sentinel is not monitoring the master"
+        }
+    }
+
+    kill_instance redis $master_id
+
+    foreach_sentinel_id id {
+        wait_for_condition 1000 50 {
+            [lindex [S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port
+        } else {
+            fail "At least one Sentinel did not received failover info"
+        }
+    }
+    restart_instance redis $master_id
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    set master_id [get_instance_id_by_port redis [lindex $addr 1]]
+
+    # Set the min ODOWN agreement back to strict majority.
+    foreach_sentinel_id id {
+        S $id SENTINEL SET mymaster quorum $quorum
+    }
+}
+
+test "New master [join $addr {:}] role matches" {
+    assert {[RI $master_id role] eq {master}}
+}

tests/sentinel-tests/01-conf-update.tcl

@@ -0,0 +1,39 @@
+# Test Sentinel configuration consistency after partitions heal.
+
+source "../sentinel-tests/includes/init-tests.tcl"
+
+test "We can failover with Sentinel 1 crashed" {
+    set old_port [RI $master_id tcp_port]
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    assert {[lindex $addr 1] == $old_port}
+
+    # Crash Sentinel 1
+    kill_instance sentinel 1
+
+    kill_instance redis $master_id
+    foreach_sentinel_id id {
+        if {$id != 1} {
+            wait_for_condition 1000 50 {
+                [lindex [S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port
+            } else {
+                fail "Sentinel $id did not received failover info"
+            }
+        }
+    }
+    restart_instance redis $master_id
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    set master_id [get_instance_id_by_port redis [lindex $addr 1]]
+}
+
+test "After Sentinel 1 is restarted, its config gets updated" {
+    restart_instance sentinel 1
+    wait_for_condition 1000 50 {
+        [lindex [S 1 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port
+    } else {
+        fail "Restarted Sentinel did not received failover info"
+    }
+}
+
+test "New master [join $addr {:}] role matches" {
+    assert {[RI $master_id role] eq {master}}
+}

tests/sentinel-tests/02-slaves-reconf.tcl

@@ -0,0 +1,83 @@
+# Check that slaves are reconfigured at a latter time if they are partitioned.
+#
+# Here we should test:
+# 1) That slaves point to the new master after failover.
+# 2) That partitioned slaves point to new master when they are partitioned
+#    away during failover and return at a latter time.
+
+source "../sentinel-tests/includes/init-tests.tcl"
+
+proc 03_test_slaves_replication {} {
+    uplevel 1 {
+        test "Check that slaves replicate from current master" {
+            set master_port [RI $master_id tcp_port]
+            foreach_redis_id id {
+                if {$id == $master_id} continue
+                if {[instance_is_killed redis $id]} continue
+                wait_for_condition 1000 50 {
+                    [RI $id master_port] == $master_port
+                } else {
+                    fail "Redis slave $id is replicating from wrong master"
+                }
+            }
+        }
+    }
+}
+
+proc 03_crash_and_failover {} {
+    uplevel 1 {
+        test "Crash the master and force a failover" {
+            set old_port [RI $master_id tcp_port]
+            set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+            assert {[lindex $addr 1] == $old_port}
+            kill_instance redis $master_id
+            foreach_sentinel_id id {
+                wait_for_condition 1000 50 {
+                    [lindex [S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port
+                } else {
+                    fail "At least one Sentinel did not received failover info"
+                }
+            }
+            restart_instance redis $master_id
+            set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+            set master_id [get_instance_id_by_port redis [lindex $addr 1]]
+        }
+    }
+}
+
+03_test_slaves_replication
+03_crash_and_failover
+03_test_slaves_replication
+
+test "Kill a slave instance" {
+    foreach_redis_id id {
+        if {$id == $master_id} continue
+        set killed_slave_id $id
+        kill_instance redis $id
+        break
+    }
+}
+
+03_crash_and_failover
+03_test_slaves_replication
+
+test "Wait for failover to end" {
+    set inprogress 1
+    while {$inprogress} {
+        set inprogress 0
+        foreach_sentinel_id id {
+            if {[dict exists [S $id SENTINEL MASTER mymaster] failover-state]} {
+                incr inprogress
+            }
+        }
+        if {$inprogress} {after 100}
+    }
+}
+
+test "Restart killed slave and test replication of slaves again..." {
+    restart_instance redis $killed_slave_id
+}
+
+# Now we check if the slave rejoining the partition is reconfigured even
+# if the failover finished.
+03_test_slaves_replication

tests/sentinel-tests/03-runtime-reconf.tcl

@@ -0,0 +1 @@
+# Test runtime reconfiguration command SENTINEL SET.

tests/sentinel-tests/04-slave-selection.tcl

@@ -0,0 +1,5 @@
+# Test slave selection algorithm.
+#
+# This unit should test:
+# 1) That when there are no suitable slaves no failover is performed.
+# 2) That among the available slaves, the one with better offset is picked.

tests/sentinel-tests/includes/init-tests.tcl

@@ -0,0 +1,67 @@
+# Initialization tests -- most units will start including this.
+
+test "(init) Restart killed instances" {
+    foreach type {redis sentinel} {
+        foreach_${type}_id id {
+            if {[get_instance_attrib $type $id pid] == -1} {
+                puts -nonewline "$type/$id "
+                flush stdout
+                restart_instance $type $id
+            }
+        }
+    }
+}
+
+set redis_slaves 4
+test "(init) Create a master-slaves cluster of [expr $redis_slaves+1] instances" {
+    create_redis_master_slave_cluster [expr {$redis_slaves+1}]
+}
+set master_id 0
+
+test "(init) Sentinels can start monitoring a master" {
+    set sentinels [llength $::sentinel_instances]
+    set quorum [expr {$sentinels/2+1}]
+    foreach_sentinel_id id {
+        catch {S $id SENTINEL REMOVE mymaster}
+        S $id SENTINEL MONITOR mymaster \
+              [get_instance_attrib redis $master_id host] \
+              [get_instance_attrib redis $master_id port] $quorum
+    }
+    foreach_sentinel_id id {
+        assert {[S $id sentinel master mymaster] ne {}}
+        S $id SENTINEL SET mymaster down-after-milliseconds 2000
+        S $id SENTINEL SET mymaster failover-timeout 20000
+        S $id SENTINEL SET mymaster parallel-syncs 10
+    }
+}
+
+test "(init) Sentinels can talk with the master" {
+    foreach_sentinel_id id {
+        wait_for_condition 100 50 {
+            [catch {S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster}] == 0
+        } else {
+            fail "Sentinel $id can't talk with the master."
+        }
+    }
+}
+
+test "(init) Sentinels are able to auto-discover other sentinels" {
+    set sentinels [llength $::sentinel_instances]
+    foreach_sentinel_id id {
+        wait_for_condition 100 50 {
+            [dict get [S $id SENTINEL MASTER mymaster] num-other-sentinels] == ($sentinels-1)
+        } else {
+            fail "At least some sentinel can't detect some other sentinel"
+        }
+    }
+}
+
+test "(init) Sentinels are able to auto-discover slaves" {
+    foreach_sentinel_id id {
+        wait_for_condition 100 50 {
+            [dict get [S $id SENTINEL MASTER mymaster] num-slaves] == $redis_slaves
+        } else {
+            fail "At least some sentinel can't detect some slave"
+        }
+    }
+}

tests/sentinel-tmp/.gitignore

@@ -0,0 +1,2 @@
+redis_*
+sentinel_*

tests/sentinel.tcl

@@ -0,0 +1,396 @@
+# Sentinel test suite. Copyright (C) 2014 Salvatore Sanfilippo antirez@gmail.com
+# This softare is released under the BSD License. See the COPYING file for
+# more information.
+
+package require Tcl 8.5
+
+set tcl_precision 17
+source tests/support/redis.tcl
+source tests/support/util.tcl
+source tests/support/server.tcl
+source tests/support/test.tcl
+
+set ::verbose 0
+set ::pause_on_error 0
+set ::simulate_error 0
+set ::sentinel_instances {}
+set ::redis_instances {}
+set ::sentinel_base_port 20000
+set ::redis_base_port 30000
+set ::instances_count 5 ; # How many Sentinels / Instances we use at max
+set ::pids {} ; # We kill everything at exit
+set ::dirs {} ; # We remove all the temp dirs at exit
+set ::run_matching {} ; # If non empty, only tests matching pattern are run.
+
+if {[catch {cd tests/sentinel-tmp}]} {
+    puts "tests/sentinel-tmp directory not found."
+    puts "Please run this test from the Redis source root."
+    exit 1
+}
+
+# Spawn a redis or sentinel instance, depending on 'type'.
+proc spawn_instance {type base_port count} {
+    for {set j 0} {$j < $count} {incr j} {
+        set port [find_available_port $base_port]
+        incr base_port
+        puts "Starting $type #$j at port $port"
+
+        # Create a directory for this Sentinel.
+        set dirname "${type}_${j}"
+        lappend ::dirs $dirname
+        catch {exec rm -rf $dirname}
+        file mkdir $dirname
+
+        # Write the Sentinel config file.
+        set cfgfile [file join $dirname $type.conf]
+        set cfg [open $cfgfile w]
+        puts $cfg "port $port"
+        puts $cfg "dir ./$dirname"
+        puts $cfg "logfile log.txt"
+        close $cfg
+
+        # Finally exec it and remember the pid for later cleanup.
+        if {$type eq "redis"} {
+            set prgname redis-server
+        } else {
+            set prgname redis-sentinel
+        }
+        set pid [exec ../../src/${prgname} $cfgfile &]
+        lappend ::pids $pid
+
+        # Check availability
+        if {[server_is_up 127.0.0.1 $port 100] == 0} {
+            abort_sentinel_test "Problems starting $type #$j: ping timeout"
+        }
+
+        # Push the instance into the right list
+        lappend ::${type}_instances [list \
+            pid $pid \
+            host 127.0.0.1 \
+            port $port \
+            link [redis 127.0.0.1 $port] \
+        ]
+    }
+}
+
+proc cleanup {} {
+    puts "Cleaning up..."
+    foreach pid $::pids {
+        catch {exec kill -9 $pid}
+    }
+    foreach dir $::dirs {
+        catch {exec rm -rf $dir}
+    }
+}
+
+proc abort_sentinel_test msg {
+    puts "WARNING: Aborting the test."
+    puts ">>>>>>>> $msg"
+    cleanup
+    exit 1
+}
+
+proc parse_options {} {
+    for {set j 0} {$j < [llength $::argv]} {incr j} {
+        set opt [lindex $::argv $j]
+        set val [lindex $::argv [expr $j+1]]
+        if {$opt eq "--single"} {
+            incr j
+            set ::run_matching "*${val}*"
+        } elseif {$opt eq "--pause-on-error"} {
+            set ::pause_on_error 1
+        } elseif {$opt eq "--fail"} {
+            set ::simulate_error 1
+        } elseif {$opt eq "--help"} {
+            puts "Hello, I'm sentinel.tcl and I run Sentinel unit tests."
+            puts "\nOptions:"
+            puts "--single <pattern>      Only runs tests specified by pattern."
+            puts "--pause-on-error        Pause for manual inspection on error."
+            puts "--fail                  Simulate a test failure."
+            puts "--help                  Shows this help."
+            exit 0
+        } else {
+            puts "Unknown option $opt"
+            exit 1
+        }
+    }
+}
+
+proc main {} {
+    parse_options
+    spawn_instance sentinel $::sentinel_base_port $::instances_count
+    spawn_instance redis $::redis_base_port $::instances_count
+    run_tests
+    cleanup
+}
+
+# If --pause-on-error option was passed at startup this function is called
+# on error in order to give the developer a chance to understand more about
+# the error condition while the instances are still running.
+proc pause_on_error {} {
+    puts ""
+    puts [colorstr yellow "*** Please inspect the error now ***"]
+    puts "\nType \"continue\" to resume the test, \"help\" for help screen.\n"
+    while 1 {
+        puts -nonewline "> "
+        flush stdout
+        set line [gets stdin]
+        set argv [split $line " "]
+        set cmd [lindex $argv 0]
+        if {$cmd eq {continue}} {
+            break
+        } elseif {$cmd eq {show-sentinel-logs}} {
+            set count 10
+            if {[lindex $argv 1] ne {}} {set count [lindex $argv 1]}
+            foreach_sentinel_id id {
+                puts "=== SENTINEL $id ===="
+                puts [exec tail -$count sentinel_$id/log.txt]
+                puts "---------------------\n"
+            }
+        } elseif {$cmd eq {ls}} {
+            foreach_redis_id id {
+                puts -nonewline "Redis $id"
+                set errcode [catch {
+                    set str {}
+                    append str "@[RI $id tcp_port]: "
+                    append str "[RI $id role] "
+                    if {[RI $id role] eq {slave}} {
+                        append str "[RI $id master_host]:[RI $id master_port]"
+                    }
+                    set str
+                } retval]
+                if {$errcode} {
+                    puts " -- $retval"
+                } else {
+                    puts $retval
+                }
+            }
+            foreach_sentinel_id id {
+                puts -nonewline "Sentinel $id"
+                set errcode [catch {
+                    set str {}
+                    append str "@[SI $id tcp_port]: "
+                    append str "[join [S $id sentinel get-master-addr-by-name mymaster]]"
+                    set str
+                } retval]
+                if {$errcode} {
+                    puts " -- $retval"
+                } else {
+                    puts $retval
+                }
+            }
+        } elseif {$cmd eq {help}} {
+            puts "ls                     List Sentinel and Redis instances."
+            puts "show-sentinel-logs \[N\] Show latest N lines of logs."
+            puts "S <id> cmd ... arg     Call command in Sentinel <id>."
+            puts "R <id> cmd ... arg     Call command in Redis <id>."
+            puts "SI <id> <field>        Show Sentinel <id> INFO <field>."
+            puts "RI <id> <field>        Show Sentinel <id> INFO <field>."
+            puts "continue               Resume test."
+        } else {
+            set errcode [catch {eval $line} retval]
+            if {$retval ne {}} {puts "$retval"}
+        }
+    }
+}
+
+# We redefine 'test' as for Sentinel we don't use the server-client
+# architecture for the test, everything is sequential.
+proc test {descr code} {
+    set ts [clock format [clock seconds] -format %H:%M:%S]
+    puts -nonewline "$ts> $descr: "
+    flush stdout
+
+    if {[catch {set retval [uplevel 1 $code]} error]} {
+        if {[string match "assertion:*" $error]} {
+            set msg [string range $error 10 end]
+            puts [colorstr red $msg]
+            if {$::pause_on_error} pause_on_error
+            puts "(Jumping to next unit after error)"
+            return -code continue
+        } else {
+            # Re-raise, let handler up the stack take care of this.
+            error $error $::errorInfo
+        }
+    } else {
+        puts [colorstr green OK]
+    }
+}
+
+proc run_tests {} {
+    set tests [lsort [glob ../sentinel-tests/*]]
+    foreach test $tests {
+        if {$::run_matching ne {} && [string match $::run_matching $test] == 0} {
+            continue
+        }
+        if {[file isdirectory $test]} continue
+        puts [colorstr yellow "Testing unit: [lindex [file split $test] end]"]
+        source $test
+    }
+}
+
+# The "S" command is used to interact with the N-th Sentinel.
+# The general form is:
+#
+# S <sentinel-id> command arg arg arg ...
+#
+# Example to ping the Sentinel 0 (first instance): S 0 PING
+proc S {n args} {
+    set s [lindex $::sentinel_instances $n]
+    [dict get $s link] {*}$args
+}
+
+# Like R but to chat with Redis instances.
+proc R {n args} {
+    set r [lindex $::redis_instances $n]
+    [dict get $r link] {*}$args
+}
+
+proc get_info_field {info field} {
+    set fl [string length $field]
+    append field :
+    foreach line [split $info "\n"] {
+        set line [string trim $line "\r\n "]
+        if {[string range $line 0 $fl] eq $field} {
+            return [string range $line [expr {$fl+1}] end]
+        }
+    }
+    return {}
+}
+
+proc SI {n field} {
+    get_info_field [S $n info] $field
+}
+
+proc RI {n field} {
+    get_info_field [R $n info] $field
+}
+
+# Iterate over IDs of sentinel or redis instances.
+proc foreach_instance_id {instances idvar code} {
+    upvar 1 $idvar id
+    for {set id 0} {$id < [llength $instances]} {incr id} {
+        set errcode [catch {uplevel 1 $code} result]
+        if {$errcode == 1} {
+            error $result $::errorInfo $::errorCode
+        } elseif {$errcode == 4} {
+            continue
+        } elseif {$errcode == 3} {
+            break
+        } elseif {$errcode != 0} {
+            return -code $errcode $result
+        }
+    }
+}
+
+proc foreach_sentinel_id {idvar code} {
+    set errcode [catch {uplevel 1 [list foreach_instance_id $::sentinel_instances $idvar $code]} result]
+    return -code $errcode $result
+}
+
+proc foreach_redis_id {idvar code} {
+    set errcode [catch {uplevel 1 [list foreach_instance_id $::redis_instances $idvar $code]} result]
+    return -code $errcode $result
+}
+
+# Get the specific attribute of the specified instance type, id.
+proc get_instance_attrib {type id attrib} {
+    dict get [lindex [set ::${type}_instances] $id] $attrib
+}
+
+# Set the specific attribute of the specified instance type, id.
+proc set_instance_attrib {type id attrib newval} {
+    set d [lindex [set ::${type}_instances] $id]
+    dict set d $attrib $newval
+    lset ::${type}_instances $id $d
+}
+
+# Create a master-slave cluster of the given number of total instances.
+# The first instance "0" is the master, all others are configured as
+# slaves.
+proc create_redis_master_slave_cluster n {
+    foreach_redis_id id {
+        if {$id == 0} {
+            # Our master.
+            R $id slaveof no one
+            R $id flushall
+        } elseif {$id < $n} {
+            R $id slaveof [get_instance_attrib redis 0 host] \
+                          [get_instance_attrib redis 0 port]
+        } else {
+            # Instances not part of the cluster.
+            R $id slaveof no one
+        }
+    }
+    # Wait for all the slaves to sync.
+    wait_for_condition 1000 50 {
+        [RI 0 connected_slaves] == ($n-1)
+    } else {
+        fail "Unable to create a master-slaves cluster."
+    }
+}
+
+proc get_instance_id_by_port {type port} {
+    foreach_${type}_id id {
+        if {[get_instance_attrib $type $id port] == $port} {
+            return $id
+        }
+    }
+    fail "Instance $type port $port not found."
+}
+
+# Kill an instance of the specified type/id with SIGKILL.
+# This function will mark the instance PID as -1 to remember that this instance
+# is no longer running and will remove its PID from the list of pids that
+# we kill at cleanup.
+#
+# The instance can be restarted with restart-instance.
+proc kill_instance {type id} {
+    set pid [get_instance_attrib $type $id pid]
+    if {$pid == -1} {
+        error "You tried to kill $type $id twice."
+    }
+    exec kill -9 $pid
+    set_instance_attrib $type $id pid -1
+    set_instance_attrib $type $id link you_tried_to_talk_with_killed_instance
+
+    # Remove the PID from the list of pids to kill at exit.
+    set ::pids [lsearch -all -inline -not -exact $::pids $pid]
+}
+
+# Return true of the instance of the specified type/id is killed.
+proc instance_is_killed {type id} {
+    set pid [get_instance_attrib $type $id pid]
+    return $pid == -1
+}
+
+# Restart an instance previously killed by kill_instance
+proc restart_instance {type id} {
+    set dirname "${type}_${id}"
+    set cfgfile [file join $dirname $type.conf]
+    set port [get_instance_attrib $type $id port]
+
+    # Execute the instance with its old setup and append the new pid
+    # file for cleanup.
+    if {$type eq "redis"} {
+        set prgname redis-server
+    } else {
+        set prgname redis-sentinel
+    }
+    set pid [exec ../../src/${prgname} $cfgfile &]
+    set_instance_attrib $type $id pid $pid
+    lappend ::pids $pid
+
+    # Check that the instance is running
+    if {[server_is_up 127.0.0.1 $port 100] == 0} {
+        abort_sentinel_test "Problems starting $type #$j: ping timeout"
+    }
+
+    # Connect with it with a fresh link
+    set_instance_attrib $type $id link [redis 127.0.0.1 $port]
+}
+
+if {[catch main e]} {
+    puts $::errorInfo
+    cleanup
+}

tests/support/server.tcl

@@ -40,6 +40,10 @@ proc kill_server config {
                     test "Check for memory leaks (pid $pid)" {
                         set output {0 leaks}
                         catch {exec leaks $pid} output
+                        if {[string match {*process does not exist*} $output]} {
+                            # In a few tests we kill the server process.
+                            set output "0 leaks"
+                        }
                         set output
                     } {*0 leaks*}
                 }
@@ -79,7 +83,7 @@ proc is_alive config {
 proc ping_server {host port} {
     set retval 0
     if {[catch {
-        set fd [socket $::host $::port]
+        set fd [socket $host $port]
         fconfigure $fd -translation binary
         puts $fd "PING\r\n"
         flush $fd
@@ -101,6 +105,22 @@ proc ping_server {host port} {
     return $retval
 }
 
+# Return 1 if the server at the specified addr is reachable by PING, otherwise
+# returns 0. Performs a try every 50 milliseconds for the specified number
+# of retries.
+proc server_is_up {host port retrynum} {
+    after 10 ;# Use a small delay to make likely a first-try success.
+    set retval 0
+    while {[incr retrynum -1]} {
+        if {[catch {ping_server $host $port} ping]} {
+            set ping 0
+        }
+        if {$ping} {return 1}
+        after 50
+    }
+    return 0
+}
+
 # doesn't really belong here, but highly coupled to code in start_server
 proc tags {tags code} {
     set ::tags [concat $::tags $tags]
@@ -191,23 +211,13 @@ proc start_server {options {code undefined}} {
     # check that the server actually started
     # ugly but tries to be as fast as possible...
     if {$::valgrind} {set retrynum 1000} else {set retrynum 100}
-    set serverisup 0
 
     if {$::verbose} {
         puts -nonewline "=== ($tags) Starting server ${::host}:${::port} "
     }
 
-    after 10
     if {$code ne "undefined"} {
-        while {[incr retrynum -1]} {
-            catch {
-                if {[ping_server $::host $::port]} {
-                    set serverisup 1
-                }
-            }
-            if {$serverisup} break
-            after 50
-        }
+        set serverisup [server_is_up $::host $::port $retrynum]
     } else {
         set serverisup 1
     }

tests/support/test.tcl

@@ -53,41 +53,17 @@ proc assert_type {type key} {
 # executed.
 proc wait_for_condition {maxtries delay e _else_ elsescript} {
     while {[incr maxtries -1] >= 0} {
-        if {[uplevel 1 [list expr $e]]} break
+        set errcode [catch {uplevel 1 [list expr $e]} result]
+        if {$errcode == 0} {
+            if {$result} break
+        } else {
+            return -code $errcode $result
+        }
         after $delay
     }
     if {$maxtries == -1} {
-        uplevel 1 $elsescript
-    }
-}
-
-# Test if TERM looks like to support colors
-proc color_term {} {
-    expr {[info exists ::env(TERM)] && [string match *xterm* $::env(TERM)]}
-}
-
-proc colorstr {color str} {
-    if {[color_term]} {
-        set b 0
-        if {[string range $color 0 4] eq {bold-}} {
-            set b 1
-            set color [string range $color 5 end]
-        }
-        switch $color {
-            red {set colorcode {31}}
-            green {set colorcode {32}}
-            yellow {set colorcode {33}}
-            blue {set colorcode {34}}
-            magenta {set colorcode {35}}
-            cyan {set colorcode {36}}
-            white {set colorcode {37}}
-            default {set colorcode {37}}
-        }
-        if {$colorcode ne {}} {
-            return "\033\[$b;${colorcode};40m$str\033\[0m"
-        }
-    } else {
-        return $str
+        set errcode [catch [uplevel 1 $elsescript] result]
+        return -code $errcode $result
     }
 }
 

tests/support/util.tcl

@@ -312,3 +312,48 @@ proc csvstring s {
 proc roundFloat f {
     format "%.10g" $f
 }
+
+proc find_available_port start {
+    for {set j $start} {$j < $start+1024} {incr j} {
+        if {[catch {
+            set fd [socket 127.0.0.1 $j]
+        }]} {
+            return $j
+        } else {
+            close $fd
+        }
+    }
+    if {$j == $start+1024} {
+        error "Can't find a non busy port in the $start-[expr {$start+1023}] range."
+    }
+}
+
+# Test if TERM looks like to support colors
+proc color_term {} {
+    expr {[info exists ::env(TERM)] && [string match *xterm* $::env(TERM)]}
+}
+
+proc colorstr {color str} {
+    if {[color_term]} {
+        set b 0
+        if {[string range $color 0 4] eq {bold-}} {
+            set b 1
+            set color [string range $color 5 end]
+        }
+        switch $color {
+            red {set colorcode {31}}
+            green {set colorcode {32}}
+            yellow {set colorcode {33}}
+            blue {set colorcode {34}}
+            magenta {set colorcode {35}}
+            cyan {set colorcode {36}}
+            white {set colorcode {37}}
+            default {set colorcode {37}}
+        }
+        if {$colorcode ne {}} {
+            return "\033\[$b;${colorcode};40m$str\033\[0m"
+        }
+    } else {
+        return $str
+    }
+}

tests/test_helper.tcl

@@ -47,6 +47,7 @@ set ::all_tests {
     unit/obuf-limits
     unit/bitops
     unit/memefficiency
+    unit/hyperloglog
 }
 # Index to the next test to run in the ::all_tests list.
 set ::next_test 0
@@ -164,21 +165,6 @@ proc cleanup {} {
     if {!$::quiet} {puts "OK"}
 }
 
-proc find_available_port start {
-    for {set j $start} {$j < $start+1024} {incr j} {
-        if {[catch {
-            set fd [socket 127.0.0.1 $j]
-        }]} {
-            return $j
-        } else {
-            close $fd
-        }
-    }
-    if {$j == $start+1024} {
-        error "Can't find a non busy port in the $start-[expr {$start+1023}] range."
-    }
-}
-
 proc test_server_main {} {
     cleanup
     set tclsh [info nameofexecutable]

tests/unit/bitops.tcl

@@ -52,7 +52,7 @@ start_server {tags {"bitops"}} {
         }
     }
 
-    test {BITCOUNT fuzzing} {
+    test {BITCOUNT fuzzing without start/end} {
         for {set j 0} {$j < 100} {incr j} {
             set str [randstring 0 3000]
             r set str $str
@@ -60,6 +60,20 @@ start_server {tags {"bitops"}} {
         }
     }
 
+    test {BITCOUNT fuzzing with start/end} {
+        for {set j 0} {$j < 100} {incr j} {
+            set str [randstring 0 3000]
+            r set str $str
+            set l [string length $str]
+            set start [randomInt $l]
+            set end [randomInt $l]
+            if {$start > $end} {
+                lassign [list $end $start] start end
+            }
+            assert {[r bitcount str $start $end] == [count_bits [string range $str $start $end]]}
+        }
+    }
+
     test {BITCOUNT with start, end} {
         r set s "foobar"
         assert_equal [r bitcount s 0 -1] [count_bits "foobar"]
@@ -84,6 +98,18 @@ start_server {tags {"bitops"}} {
         }
     } {1}
 
+    test {BITCOUNT misaligned prefix} {
+        r del str
+        r set str ab
+        r bitcount str 1 -1
+    } {3}
+
+    test {BITCOUNT misaligned prefix + full words + remainder} {
+        r del str
+        r set str __PPxxxxxxxxxxxxxxxxRR__
+        r bitcount str 2 -3
+    } {74}
+
     test {BITOP NOT (empty string)} {
         r set s ""
         r bitop not dest s
@@ -177,4 +203,139 @@ start_server {tags {"bitops"}} {
         r set a "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
         r bitop or x a b
     } {32}
+
+    test {BITPOS bit=0 with empty key returns 0} {
+        r del str
+        r bitpos str 0
+    } {0}
+
+    test {BITPOS bit=1 with empty key returns -1} {
+        r del str
+        r bitpos str 1
+    } {-1}
+
+    test {BITPOS bit=0 with string less than 1 word works} {
+        r set str "\xff\xf0\x00"
+        r bitpos str 0
+    } {12}
+
+    test {BITPOS bit=1 with string less than 1 word works} {
+        r set str "\x00\x0f\x00"
+        r bitpos str 1
+    } {12}
+
+    test {BITPOS bit=0 starting at unaligned address} {
+        r set str "\xff\xf0\x00"
+        r bitpos str 0 1
+    } {12}
+
+    test {BITPOS bit=1 starting at unaligned address} {
+        r set str "\x00\x0f\xff"
+        r bitpos str 1 1
+    } {12}
+
+    test {BITPOS bit=0 unaligned+full word+reminder} {
+        r del str
+        r set str "\xff\xff\xff" ; # Prefix
+        # Followed by two (or four in 32 bit systems) full words
+        r append str "\xff\xff\xff\xff\xff\xff\xff\xff"
+        r append str "\xff\xff\xff\xff\xff\xff\xff\xff"
+        r append str "\xff\xff\xff\xff\xff\xff\xff\xff"
+        # First zero bit.
+        r append str "\x0f"
+        assert {[r bitpos str 0] == 216}
+        assert {[r bitpos str 0 1] == 216}
+        assert {[r bitpos str 0 2] == 216}
+        assert {[r bitpos str 0 3] == 216}
+        assert {[r bitpos str 0 4] == 216}
+        assert {[r bitpos str 0 5] == 216}
+        assert {[r bitpos str 0 6] == 216}
+        assert {[r bitpos str 0 7] == 216}
+        assert {[r bitpos str 0 8] == 216}
+    }
+
+    test {BITPOS bit=1 unaligned+full word+reminder} {
+        r del str
+        r set str "\x00\x00\x00" ; # Prefix
+        # Followed by two (or four in 32 bit systems) full words
+        r append str "\x00\x00\x00\x00\x00\x00\x00\x00"
+        r append str "\x00\x00\x00\x00\x00\x00\x00\x00"
+        r append str "\x00\x00\x00\x00\x00\x00\x00\x00"
+        # First zero bit.
+        r append str "\xf0"
+        assert {[r bitpos str 1] == 216}
+        assert {[r bitpos str 1 1] == 216}
+        assert {[r bitpos str 1 2] == 216}
+        assert {[r bitpos str 1 3] == 216}
+        assert {[r bitpos str 1 4] == 216}
+        assert {[r bitpos str 1 5] == 216}
+        assert {[r bitpos str 1 6] == 216}
+        assert {[r bitpos str 1 7] == 216}
+        assert {[r bitpos str 1 8] == 216}
+    }
+
+    test {BITPOS bit=1 returns -1 if string is all 0 bits} {
+        r set str ""
+        for {set j 0} {$j < 20} {incr j} {
+            assert {[r bitpos str 1] == -1}
+            r append str "\x00"
+        }
+    }
+
+    test {BITPOS bit=0 works with intervals} {
+        r set str "\x00\xff\x00"
+        assert {[r bitpos str 0 0 -1] == 0}
+        assert {[r bitpos str 0 1 -1] == 16}
+        assert {[r bitpos str 0 2 -1] == 16}
+        assert {[r bitpos str 0 2 200] == 16}
+        assert {[r bitpos str 0 1 1] == -1}
+    }
+
+    test {BITPOS bit=1 works with intervals} {
+        r set str "\x00\xff\x00"
+        assert {[r bitpos str 1 0 -1] == 8}
+        assert {[r bitpos str 1 1 -1] == 8}
+        assert {[r bitpos str 1 2 -1] == -1}
+        assert {[r bitpos str 1 2 200] == -1}
+        assert {[r bitpos str 1 1 1] == 8}
+    }
+
+    test {BITPOS bit=0 changes behavior if end is given} {
+        r set str "\xff\xff\xff"
+        assert {[r bitpos str 0] == 24}
+        assert {[r bitpos str 0 0] == 24}
+        assert {[r bitpos str 0 0 -1] == -1}
+    }
+
+    test {BITPOS bit=1 fuzzy testing using SETBIT} {
+        r del str
+        set max 524288; # 64k
+        set first_one_pos -1
+        for {set j 0} {$j < 1000} {incr j} {
+            assert {[r bitpos str 1] == $first_one_pos}
+            set pos [randomInt $max]
+            r setbit str $pos 1
+            if {$first_one_pos == -1 || $first_one_pos > $pos} {
+                # Update the position of the first 1 bit in the array
+                # if the bit we set is on the left of the previous one.
+                set first_one_pos $pos
+            }
+        }
+    }
+
+    test {BITPOS bit=0 fuzzy testing using SETBIT} {
+        set max 524288; # 64k
+        set first_zero_pos $max
+        r set str [string repeat "\xff" [expr $max/8]]
+        for {set j 0} {$j < 1000} {incr j} {
+            assert {[r bitpos str 0] == $first_zero_pos}
+            set pos [randomInt $max]
+            r setbit str $pos 0
+            if {$first_zero_pos > $pos} {
+                # Update the position of the first 0 bit in the array
+                # if the bit we clear is on the left of the previous one.
+                set first_zero_pos $pos
+            }
+        }
+    }
 }

tests/unit/hyperloglog.tcl

@@ -0,0 +1,68 @@
+start_server {tags {"hll"}} {
+    test {HyperLogLog self test passes} {
+        catch {r pfselftest} e
+        set e
+    } {OK}
+
+    test {PFADD without arguments creates an HLL value} {
+        r pfadd hll
+        r exists hll
+    } {1}
+
+    test {Approximated cardinality after creation is zero} {
+        r pfcount hll
+    } {0}
+
+    test {PFADD returns 1 when at least 1 reg was modified} {
+        r pfadd hll a b c
+    } {1}
+
+    test {PFADD returns 0 when no reg was modified} {
+        r pfadd hll a b c
+    } {0}
+
+    test {PFADD works with empty string (regression)} {
+        r pfadd hll ""
+    }
+
+    # Note that the self test stresses much better the
+    # cardinality estimation error. We are testing just the
+    # command implementation itself here.
+    test {PFCOUNT returns approximated cardinality of set} {
+        r del hll
+        set res {}
+        r pfadd hll 1 2 3 4 5
+        lappend res [r pfcount hll]
+        # Call it again to test cached value invalidation.
+        r pfadd hll 6 7 8 8 9 10
+        lappend res [r pfcount hll]
+        set res
+    } {5 10}
+
+    test {PFADD, PFCOUNT, PFMERGE type checking works} {
+        r set foo bar
+        catch {r pfadd foo 1} e
+        assert_match {*WRONGTYPE*} $e
+        catch {r pfcount foo} e
+        assert_match {*WRONGTYPE*} $e
+        catch {r pfmerge bar foo} e
+        assert_match {*WRONGTYPE*} $e
+        catch {r pfmerge foo bar} e
+        assert_match {*WRONGTYPE*} $e
+    }
+
+    test {PFMERGE results on the cardinality of union of sets} {
+        r del hll hll1 hll2 hll3
+        r pfadd hll1 a b c
+        r pfadd hll2 b c d
+        r pfadd hll3 c d e
+        r pfmerge hll hll1 hll2 hll3
+        r pfcount hll
+    } {5}
+
+    test {PFGETREG returns the HyperLogLog raw registers} {
+        r del hll
+        r pfadd hll 1 2 3
+        llength [r pfgetreg hll]
+    } {16384}
+}

tests/unit/scripting.tcl

@@ -417,5 +417,17 @@ start_server {tags {"scripting repl"}} {
             }
             set res
         } {a 1}
+
+        test {EVALSHA replication when first call is readonly} {
+            r del x
+            r eval {if tonumber(KEYS[1]) > 0 then redis.call('incr', 'x') end} 1 0
+            r evalsha 38fe3ddf5284a1d48f37f824b4c4e826879f3cb9 1 0
+            r evalsha 38fe3ddf5284a1d48f37f824b4c4e826879f3cb9 1 1
+            wait_for_condition 50 100 {
+                [r -1 get x] eq {1}
+            } else {
+                fail "Expected 1 in x, but value is '[r -1 get x]'"
+            }
+        }
     }
 }

utils/hyperloglog/.gitignore

@@ -0,0 +1 @@
+*.txt

utils/hyperloglog/hll-err.rb

@@ -0,0 +1,27 @@
+# hll-err.rb - Copyright (C) 2014 Salvatore Sanfilippo
+# BSD license, See the COPYING file for more information.
+#
+# Check error of HyperLogLog Redis implementation for different set sizes.
+
+require 'rubygems'
+require 'redis'
+require 'digest/sha1'
+
+r = Redis.new
+r.del('hll')
+i = 0
+while true do
+    100.times {
+        elements = []
+        1000.times {
+            ele = Digest::SHA1.hexdigest(i.to_s)
+            elements << ele
+            i += 1
+        }
+        r.pfadd('hll',*elements)
+    }
+    approx = r.pfcount('hll')
+    abs_err = (approx-i).abs
+    rel_err = 100.to_f*abs_err/i
+    puts "#{i} vs #{approx}: #{rel_err}%"
+end

utils/hyperloglog/hll-gnuplot-graph.rb

@@ -0,0 +1,88 @@
+# hll-err.rb - Copyright (C) 2014 Salvatore Sanfilippo
+# BSD license, See the COPYING file for more information.
+#
+# This program is suited to output average and maximum errors of
+# the Redis HyperLogLog implementation in a format suitable to print
+# graphs using gnuplot.
+
+require 'rubygems'
+require 'redis'
+require 'digest/sha1'
+
+# Generate an array of [cardinality,relative_error] pairs
+# in the 0 - max range, with the specified step.
+#
+# 'r' is the Redis object used to perform the queries.
+# 'seed' must be different every time you want a test performed
+# with a different set. The function guarantees that if 'seed' is the
+# same, exactly the same dataset is used, and when it is different,
+# a totally unrelated different data set is used (without any common
+# element in practice).
+def run_experiment(r,seed,max,step)
+    r.del('hll')
+    i = 0
+    samples = []
+    step = 1000 if step > 1000
+    while i < max do
+        elements = []
+        step.times {
+            ele = Digest::SHA1.hexdigest(i.to_s+seed.to_s)
+            elements << ele
+            i += 1
+        }
+        r.pfadd('hll',*elements)
+        approx = r.pfcount('hll')
+        err = approx-i
+        rel_err = 100.to_f*err/i
+        samples << [i,rel_err]
+    end
+    samples
+end
+
+def filter_samples(numsets,max,step,filter)
+    r = Redis.new
+    dataset = {}
+    (0...numsets).each{|i|
+        dataset[i] = run_experiment(r,i,max,step)
+        STDERR.puts "Set #{i}"
+    }
+    dataset[0].each_with_index{|ele,index|
+        if filter == :max
+            card=ele[0]
+            err=ele[1].abs
+            (1...numsets).each{|i|
+                err = dataset[i][index][1] if err < dataset[i][index][1]
+            }
+            puts "#{card} #{err}"
+        elsif filter == :avg
+            card=ele[0]
+            err = 0
+            (0...numsets).each{|i|
+                err += dataset[i][index][1]
+            }
+            err /= numsets
+            puts "#{card} #{err}"
+        elsif filter == :absavg
+            card=ele[0]
+            err = 0
+            (0...numsets).each{|i|
+                err += dataset[i][index][1].abs
+            }
+            err /= numsets
+            puts "#{card} #{err}"
+        elsif filter == :all
+            (0...numsets).each{|i|
+                card,err = dataset[i][index]
+                puts "#{card} #{err}"
+            }
+        else
+            raise "Unknown filter #{filter}"
+        end
+    }
+end
+
+if ARGV.length != 4
+    puts "Usage: hll-gnuplot-graph <samples> <max> <step> (max|avg|absavg|all)"
+    exit 1
+end
+filter_samples(ARGV[0].to_i,ARGV[1].to_i,ARGV[2].to_i,ARGV[3].to_sym)

utils/install_server.sh

@@ -30,46 +30,48 @@
 # this scripts should be run as root
 
 die () {
-	echo "ERROR: $1. Aborting!" 
+	echo "ERROR: $1. Aborting!"
 	exit 1
 }
 
+
+#Absolute path to this script
+SCRIPT=$(readlink -f $0)
+#Absolute path this script is in
+SCRIPTPATH=$(dirname $SCRIPT)
+
 #Initial defaults
 _REDIS_PORT=6379
 
 echo "Welcome to the redis service installer"
-echo "This script will help you easily set up a running redis server
+echo "This script will help you easily set up a running redis server"
+echo
 
-"
-
-#check for root user TODO: replace this with a call to "id"
-if [ `whoami` != "root" ] ; then
+#check for root user
+if [ "$(id -u)" -ne 0 ] ; then
 	echo "You must run this script as root. Sorry!"
 	exit 1
 fi
 
-
 #Read the redis port
-read  -p "Please select the redis port for this instance: [$_REDIS_PORT] " REDIS_PORT 
-if [ ! `echo $REDIS_PORT | egrep "^[0-9]+\$"`  ] ; then
+read  -p "Please select the redis port for this instance: [$_REDIS_PORT] " REDIS_PORT
+if ! echo $REDIS_PORT | egrep -q '^[0-9]+$' ; then
 	echo "Selecting default: $_REDIS_PORT"
-	REDIS_PORT=$_REDIS_PORT 
+	REDIS_PORT=$_REDIS_PORT
 fi
 
 #read the redis config file
 _REDIS_CONFIG_FILE="/etc/redis/$REDIS_PORT.conf"
 read -p "Please select the redis config file name [$_REDIS_CONFIG_FILE] " REDIS_CONFIG_FILE
-if [ !"$REDIS_CONFIG_FILE" ] ; then
+if [ -z "$REDIS_CONFIG_FILE" ] ; then
 	REDIS_CONFIG_FILE=$_REDIS_CONFIG_FILE
 	echo "Selected default - $REDIS_CONFIG_FILE"
 fi
-#try and create it
-mkdir -p `dirname "$REDIS_CONFIG_FILE"` || die "Could not create redis config directory"
 
 #read the redis log file path
 _REDIS_LOG_FILE="/var/log/redis_$REDIS_PORT.log"
 read -p "Please select the redis log file name [$_REDIS_LOG_FILE] " REDIS_LOG_FILE
-if [ !"$REDIS_LOG_FILE" ] ; then
+if [ -z "$REDIS_LOG_FILE" ] ; then
 	REDIS_LOG_FILE=$_REDIS_LOG_FILE
 	echo "Selected default - $REDIS_LOG_FILE"
 fi
@@ -78,55 +80,71 @@ fi
 #get the redis data directory
 _REDIS_DATA_DIR="/var/lib/redis/$REDIS_PORT"
 read -p "Please select the data directory for this instance [$_REDIS_DATA_DIR] " REDIS_DATA_DIR
-if [ !"$REDIS_DATA_DIR" ] ; then
+if [ -z "$REDIS_DATA_DIR" ] ; then
 	REDIS_DATA_DIR=$_REDIS_DATA_DIR
 	echo "Selected default - $REDIS_DATA_DIR"
 fi
-mkdir -p $REDIS_DATA_DIR || die "Could not create redis data directory"
 
 #get the redis executable path
-_REDIS_EXECUTABLE=`which redis-server`
+_REDIS_EXECUTABLE=`command -v redis-server`
 read -p "Please select the redis executable path [$_REDIS_EXECUTABLE] " REDIS_EXECUTABLE
-if [ ! -f "$REDIS_EXECUTABLE" ] ; then
+if [ ! -x "$REDIS_EXECUTABLE" ] ; then
 	REDIS_EXECUTABLE=$_REDIS_EXECUTABLE
-	
-	if [ ! -f "$REDIS_EXECUTABLE" ] ; then
+
+	if [ ! -x "$REDIS_EXECUTABLE" ] ; then
 		echo "Mmmmm...  it seems like you don't have a redis executable. Did you run make install yet?"
 		exit 1
 	fi
-	
 fi
 
-
-#render the tmplates
-TMP_FILE="/tmp/$REDIS_PORT.conf"
-DEFAULT_CONFIG="../redis.conf"
-INIT_TPL_FILE="./redis_init_script.tpl"
-INIT_SCRIPT_DEST="/etc/init.d/redis_$REDIS_PORT"
-PIDFILE="/var/run/redis_$REDIS_PORT.pid"
-
-
-
 #check the default for redis cli
-CLI_EXEC=`which redis-cli`
-if [ ! "$CLI_EXEC" ] ; then 
+CLI_EXEC=`command -v redis-cli`
+if [ -z "$CLI_EXEC" ] ; then
 	CLI_EXEC=`dirname $REDIS_EXECUTABLE`"/redis-cli"
 fi
 
+echo "Selected config:"
+
+echo "Port           : $REDIS_PORT"
+echo "Config file    : $REDIS_CONFIG_FILE"
+echo "Log file       : $REDIS_LOG_FILE"
+echo "Data dir       : $REDIS_DATA_DIR"
+echo "Executable     : $REDIS_EXECUTABLE"
+echo "Cli Executable : $CLI_EXEC"
+
+read -p "Is this ok? Then press ENTER to go on or Ctrl-C to abort." _UNUSED_
+
+mkdir -p `dirname "$REDIS_CONFIG_FILE"` || die "Could not create redis config directory"
+mkdir -p `dirname "$REDIS_LOG_FILE"` || die "Could not create redis log dir"
+mkdir -p "$REDIS_DATA_DIR" || die "Could not create redis data directory"
+
+#render the templates
+TMP_FILE="/tmp/${REDIS_PORT}.conf"
+DEFAULT_CONFIG="${SCRIPTPATH}/../redis.conf"
+INIT_TPL_FILE="${SCRIPTPATH}/redis_init_script.tpl"
+INIT_SCRIPT_DEST="/etc/init.d/redis_${REDIS_PORT}"
+PIDFILE="/var/run/redis_${REDIS_PORT}.pid"
+
+if [ ! -f "$DEFAULT_CONFIG" ]; then
+	echo "Mmmmm... the default config is missing. Did you switch to the utils directory?"
+	exit 1
+fi
+
 #Generate config file from the default config file as template
 #changing only the stuff we're controlling from this script
 echo "## Generated by install_server.sh ##" > $TMP_FILE
 
-SED_EXPR="s#^port [0-9]{4}\$#port ${REDIS_PORT}#;\
-s#^logfile .+\$#logfile ${REDIS_LOG_FILE}#;\
-s#^dir .+\$#dir ${REDIS_DATA_DIR}#;\
-s#^pidfile .+\$#pidfile ${PIDFILE}#;\
-s#^daemonize no\$#daemonize yes#;" 
-echo $SED_EXPR
+read -r SED_EXPR <<-EOF
+s#^port [0-9]{4}\$#port ${REDIS_PORT}#; \
+s#^logfile .+\$#logfile ${REDIS_LOG_FILE}#; \
+s#^dir .+\$#dir ${REDIS_DATA_DIR}#; \
+s#^pidfile .+\$#pidfile ${PIDFILE}#; \
+s#^daemonize no\$#daemonize yes#;
+EOF
 sed -r "$SED_EXPR" $DEFAULT_CONFIG  >> $TMP_FILE
 
 #cat $TPL_FILE | while read line; do eval "echo \"$line\"" >> $TMP_FILE; done
-cp -f $TMP_FILE $REDIS_CONFIG_FILE || exit 1
+cp $TMP_FILE $REDIS_CONFIG_FILE || die "Could not write redis config file $REDIS_CONFIG_FILE"
 
 #Generate sample script from template file
 rm -f $TMP_FILE
@@ -138,7 +156,7 @@ REDIS_INIT_HEADER=\
 #Configurations injected by install_server below....\n\n
 EXEC=$REDIS_EXECUTABLE\n
 CLIEXEC=$CLI_EXEC\n
-PIDFILE=$PIDFILE\n
+PIDFILE=\"$PIDFILE\"\n
 CONF=\"$REDIS_CONFIG_FILE\"\n\n
 REDISPORT=\"$REDIS_PORT\"\n\n
 ###############\n\n"
@@ -146,45 +164,82 @@ REDISPORT=\"$REDIS_PORT\"\n\n
 REDIS_CHKCONFIG_INFO=\
 "# REDHAT chkconfig header\n\n
 # chkconfig: - 58 74\n
-# description: redis_6379 is the redis daemon.\n
+# description: redis_${REDIS_PORT} is the redis daemon.\n
 ### BEGIN INIT INFO\n
 # Provides: redis_6379\n
-# Required-Start: $network $local_fs $remote_fs\n
-# Required-Stop: $network $local_fs $remote_fs\n
+# Required-Start: \$network \$local_fs \$remote_fs\n
+# Required-Stop: \$network \$local_fs \$remote_fs\n
 # Default-Start: 2 3 4 5\n
 # Default-Stop: 0 1 6\n
-# Should-Start: $syslog $named\n
-# Should-Stop: $syslog $named\n
-# Short-Description: start and stop redis_6379\n
+# Should-Start: \$syslog \$named\n
+# Should-Stop: \$syslog \$named\n
+# Short-Description: start and stop redis_${REDIS_PORT}\n
 # Description: Redis daemon\n
 ### END INIT INFO\n\n"
 
-if [ !`which chkconfig` ] ; then 
-	#combine the header and the template (which is actually a static footer)
-	echo $REDIS_INIT_HEADER > $TMP_FILE && cat $INIT_TPL_FILE >> $TMP_FILE || die "Could not write init script to $TMP_FILE"
-else
+if command -v chkconfig >/dev/null; then
 	#if we're a box with chkconfig on it we want to include info for chkconfig
-	echo -e $REDIS_INIT_HEADER $REDIS_CHKCONFIG_INFO > $TMP_FILE && cat $INIT_TPL_FILE >> $TMP_FILE || die "Could not write init script to $TMP_FILE"
+	echo "$REDIS_INIT_HEADER" "$REDIS_CHKCONFIG_INFO" > $TMP_FILE && cat $INIT_TPL_FILE >> $TMP_FILE || die "Could not write init script to $TMP_FILE"
+else
+	#combine the header and the template (which is actually a static footer)
+	echo "$REDIS_INIT_HEADER" > $TMP_FILE && cat $INIT_TPL_FILE >> $TMP_FILE || die "Could not write init script to $TMP_FILE"
 fi
 
+###
+# Generate sample script from template file
+# - No need to check which system we are on. The init info are comments and
+#   do not interfere with update_rc.d systems. Additionally:
+#     Ubuntu/debian by default does not come with chkconfig, but does issue a
+#     warning if init info is not available.
+
+cat > ${TMP_FILE} <<EOT
+#/bin/sh
+#Configurations injected by install_server below....
+
+EXEC=$REDIS_EXECUTABLE
+CLIEXEC=$CLI_EXEC
+PIDFILE=$PIDFILE
+CONF="$REDIS_CONFIG_FILE"
+REDISPORT="$REDIS_PORT"
+###############
+# SysV Init Information
+# chkconfig: - 58 74
+# description: redis_${REDIS_PORT} is the redis daemon.
+### BEGIN INIT INFO
+# Provides: redis_${REDIS_PORT}
+# Required-Start: \$network \$local_fs \$remote_fs
+# Required-Stop: \$network \$local_fs \$remote_fs
+# Default-Start: 2 3 4 5
+# Default-Stop: 0 1 6
+# Should-Start: \$syslog \$named
+# Should-Stop: \$syslog \$named
+# Short-Description: start and stop redis_${REDIS_PORT}
+# Description: Redis daemon
+### END INIT INFO
+
+EOT
+cat ${INIT_TPL_FILE} >> ${TMP_FILE}
+
 #copy to /etc/init.d
-cp -f $TMP_FILE $INIT_SCRIPT_DEST && chmod +x $INIT_SCRIPT_DEST || die "Could not copy redis init script to  $INIT_SCRIPT_DEST"
+cp $TMP_FILE $INIT_SCRIPT_DEST && \
+	chmod +x $INIT_SCRIPT_DEST || die "Could not copy redis init script to  $INIT_SCRIPT_DEST"
 echo "Copied $TMP_FILE => $INIT_SCRIPT_DEST"
 
 #Install the service
 echo "Installing service..."
-if [ !`which chkconfig` ] ; then 
-	#if we're not a chkconfig box assume we're able to use update-rc.d
-	update-rc.d redis_$REDIS_PORT defaults && echo "Success!"
-else
+if command -v chkconfig >/dev/null 2>&1; then
 	# we're chkconfig, so lets add to chkconfig and put in runlevel 345
-	chkconfig --add redis_$REDIS_PORT && echo "Successfully added to chkconfig!"
-	chkconfig --level 345 redis_$REDIS_PORT on && echo "Successfully added to runlevels 345!"
+	chkconfig --add redis_${REDIS_PORT} && echo "Successfully added to chkconfig!"
+	chkconfig --level 345 redis_${REDIS_PORT} on && echo "Successfully added to runlevels 345!"
+elif command -v update-rc.d >/dev/null 2>&1; then
+	#if we're not a chkconfig box assume we're able to use update-rc.d
+	update-rc.d redis_${REDIS_PORT} defaults && echo "Success!"
+else
+	echo "No supported init tool found."
 fi
-	
+
 /etc/init.d/redis_$REDIS_PORT start || die "Failed starting service..."
 
 #tada
 echo "Installation successful!"
 exit 0
-

utils/lru/README

@@ -0,0 +1,13 @@
+The test-lru.rb program can be used in order to check the behavior of the
+Redis approximated LRU algorithm against the theoretical output of true
+LRU algorithm.
+
+In order to use the program you need to recompile Redis setting the define
+REDIS_LRU_CLOCK_RESOLUTION to 1, by editing redis.h.
+This allows to execute the program in a fast way since the 1 ms resolution
+is enough for all the objects to have a different enough time stamp during
+the test.
+
+The program is executed like this:
+
+    ruby test-lru.rb > /tmp/lru.html

utils/lru/test-lru.rb

@@ -0,0 +1,112 @@
+require 'rubygems'
+require 'redis'
+
+r = Redis.new
+r.config("SET","maxmemory","2000000")
+r.config("SET","maxmemory-policy","allkeys-lru")
+r.config("SET","maxmemory-samples",5)
+r.config("RESETSTAT")
+r.flushall
+
+puts <<EOF
+<html>
+<body>
+<style>
+.box {
+    width:5px;
+    height:5px;
+    float:left;
+    margin: 1px;
+}
+
+.old {
+    border: 1px black solid;
+}
+
+.new {
+    border: 1px green solid;
+}
+
+.ex {
+    background-color: #666;
+}
+</style>
+<pre>
+EOF
+
+# Fill
+oldsize = r.dbsize
+id = 0
+while true
+    id += 1
+    r.set(id,"foo")
+    newsize = r.dbsize
+    break if newsize == oldsize
+    oldsize = newsize
+end
+
+inserted = r.dbsize
+first_set_max_id = id
+puts "#{r.dbsize} keys inserted"
+
+# Access keys sequencially
+
+puts "Access keys sequencially"
+(1..first_set_max_id).each{|id|
+    r.get(id)
+#    sleep 0.001
+}
+
+# Insert more 50% keys. We expect that the new keys
+half = inserted/2
+puts "Insert enough keys to evict half the keys we inserted"
+add = 0
+while true
+    add += 1
+    id += 1
+    r.set(id,"foo")
+    break if r.info['evicted_keys'].to_i >= half
+end
+
+puts "#{add} additional keys added."
+puts "#{r.dbsize} keys in DB"
+
+# Check if evicted keys respect LRU
+# We consider errors from 1 to N progressively more serious as they violate
+# more the access pattern.
+
+errors = 0
+e = 1
+edecr = 1.0/(first_set_max_id/2)
+(1..(first_set_max_id/2)).each{|id|
+    e -= edecr if e > 0
+    e = 0 if e < 0
+    if r.exists(id)
+        errors += e
+    end
+}
+
+puts "#{errors} errors!"
+puts "</pre>"
+
+# Generate the graphical representation
+(1..id).each{|id|
+    # Mark first set and added items in a different way.
+    c = "box"
+    if id <= first_set_max_id
+        c << " old"
+    else
+        c << " new"
+    end
+
+    # Add class if exists
+    c << " ex" if r.exists(id)
+    puts "<div class=\"#{c}\"></div>"
+}
+
+# Close HTML page
+
+puts <<EOF
+</body>
+</html>
+EOF

utils/redis_init_script.tpl

@@ -3,29 +3,41 @@ case "$1" in
     start)
         if [ -f $PIDFILE ]
         then
-                echo "$PIDFILE exists, process is already running or crashed"
+            echo "$PIDFILE exists, process is already running or crashed"
         else
-                echo "Starting Redis server..."
-                $EXEC $CONF
+            echo "Starting Redis server..."
+            $EXEC $CONF
         fi
         ;;
     stop)
         if [ ! -f $PIDFILE ]
         then
-                echo "$PIDFILE does not exist, process is not running"
+            echo "$PIDFILE does not exist, process is not running"
         else
-                PID=$(cat $PIDFILE)
-                echo "Stopping ..."
-                $CLIEXEC -p $REDISPORT shutdown
-                while [ -x /proc/${PID} ]
-                do
-                    echo "Waiting for Redis to shutdown ..."
-                    sleep 1
-                done
-                echo "Redis stopped"
+            PID=$(cat $PIDFILE)
+            echo "Stopping ..."
+            $CLIEXEC -p $REDISPORT shutdown
+            while [ -x /proc/${PID} ]
+            do
+                echo "Waiting for Redis to shutdown ..."
+                sleep 1
+            done
+            echo "Redis stopped"
         fi
         ;;
+    status)
+        if [ ! -f $PIDFILE ]
+        then
+            echo 'Redis is not running'
+        else
+            echo "Redis is running ($(<$PIDFILE))"
+        fi
+        ;;
+    restart)
+        $0 stop
+        $0 start
+        ;;
     *)
-        echo "Please use start or stop as first argument"
+        echo "Please use start, stop, restart or status as first argument"
         ;;
 esac