Redis 2.9.51 (Redis 3.0.0 beta-2).

Example:

./redis-trib.rb call 192.168.1.11:7000 config get cluster-node-timeout

00-RELEASENOTES

@@ -1,83 +1,138 @@
-Redis 2.6 release notes
+Redis 3.0 release notes
+=======================
 
-Migrating from 2.4 to 2.6
+WARNING: Redis 3.0 is currently a BETA not suitable for production environments.
+
+--------------------------------------------------------------------------------
+Upgrade urgency levels:
+
+LOW:      No need to upgrade unless there are new features you want to use.
+MODERATE: Program an upgrade of the server, but it's not urgent.
+HIGH:     There is a critical bug that may affect a subset of users. Upgrade!
+CRITICAL: There is a critical bug affecting MOST USERS. Upgrade ASAP.
+--------------------------------------------------------------------------------
+
+--[ Redis 3.0.0 Beta 2 (version 2.9.51) ] Release date: 11 mar 2014
+
+This is the second beta of Redis 3.0.0.
+
+>> General
+
+* [FIX] Sometimes the absolute config file path was obtained in a wrong way.
+        This happened when there was a "dir" directive inside the config file
+        and at the same time the configuration file was given as a relative
+        path to redis-server or redis-sentinel executables.
+* [FIX] redis-cli: Automatically enter --slave mode when SYNC or PSYNC are
+        called during an interactive session.
+* [FIX] BITCOUNT: fixed unaligned access causing issues in sparc and other
+        archs not capable of dealing with unaligned accesses. This also makes
+        the code faster in archs where unaligned accesses are allowed.
+* [FIX] Force INFO used_memory_peak to match peak memory. This generated some
+        confusion among users even if it was not an actual bug.
+* [FIX] Fixed an critical EVALSHA script cache bug: scripts executed may not
+        propagate to AOF / Slaves correctly under certain conditions.
+        See issue #1549 at Github for more information.
+* [FIX] Fixed multiple bugs resulting into closing the link with master or slave
+        during replication without good reasons. This will result in useless
+        resynchronizations, or infinite loops where the replication link can't
+        be established.
+* [FIX] Don't count the time needed to populate the buffers of clients waiting
+        in MONITOR mode when populating the Slow Log entries.
+
+* [NEW] Redis-cli updated to use SCAN instead of random sampling via
+        RANDOMKEY in order to implement --bigkeys feature. Moreover the
+        implementation now supports pipelining and reports more information
+        at the end of the scan. Much faster, much better. A special thank
+        you to Michael Grunder for this improvement.
+* [NEW] redis-cli now supports a new --intrinsic-latency mode that is able
+        to meter the latency of a system due to kernel / hypervisor.
+        How to use it is explained at http://redis.io/topics/latency.
+* [NEW] New command BITPOS: find first bit set or clear in a bitmap.
+* [NEW] CONFIG REWRITE calls are now logged.
+* [NEW] AOF write errors (like no space on device) no longer abort Redis if the
+        fsync policy is none or every second. The database enters a read-only
+        mode where every write is refused with an error. Normal operations are
+        restored as soon as Redis is able to append again data to the AOF file.
+* [NEW] Sentinel now accepts SHUTDOWN command.
+
+
+>> Cluster
+
+* [FIX] Bind the first interface listed in the "bind" configuration directive
+        if any, in order to perform outgoing connections. This fixes Cluster
+        usage when an address is bound but there are multiple interfaces that
+        may be used to connect with other nodes.
+* [FIX] When an "Importing" slot is closed via CLUSTER SETSLOT NODE ...
+        increment the configEpoch in the special case it is zero.
+* [FIX] Current transaction is invalidated on redirection errors.
+* [FIX] Abort if port does not allow for a valid cluster bus port that is
+        always at fixed +10000 offset.
+* [FIX] Keys extraction algorithm fixed for ZUNIONSTORE/ZINTERSTORE and SORT.
+* [FIX] Better failover timeout and retry times: failover should now work
+        reliabily when node-timeout is very small (a few milliseconds).
+* [FIX] Don't allow SORT GET/BY options in Cluster mode.
+* [FIX] Clear importing/migrating state when turning from master to slave role.
+* [FIX] Set slot error if we receive an update for a busy slot.
+* [FIX] Update node configEpoch on UPDATE messages.
+
+* [NEW] Support multi-key operations as long as keys resolve to the same
+        hash slot, and the slot is not migrating, or it is migrating but all
+        the mentioned keys are available.
+* [NEW] New DEBUG command CMDKEYS available to debug / test keys identification
+        in Redis commands.
+* [NEW] redis-trib: create subcommand is now able to assign spare slaves.
+* [NEW] redis-trib: new subcommand 'call'. Exec command in all nodes.
+
+
+>> Sentinel
+
+* [FIX] Sentinel "IDONTKNOW" error removed as it does not made sense with the
+        new Sentinel design. This error was actually a fix for a design error
+        in the first implementation of Sentinel.
+* [FIX] Sentinel: added a missing exit() call to abort after config file
+        checks at startup. This error was introduced with an improvement in
+        a previous 2.8 release.
+* [FIX] Sentinel: better nodes fail over start time desynchronization to avoid
+        split-brain during the voting process needed to get authorization to
+        fail over. This means the system is less likely to need to retry
+        and will fail over faster. No changes in behavior / correctness.
+
+* [NEW] Sentinel unit tests and framework. More tests needed and units must
+        be improved in order to have less false positives, but it is a start
+        and features a debugging console that is useful to fix tests or to
+        inspect bugs causing tests failures.
+* [NEW] New Sentinel events: +/-monitor and +set used to monitor when an
+        instance to monitor is added or removed, or when a configuration
+        is modified via SENTINEL SET.
+
+--[ Redis 3.0.0 Beta 1 (version 2.9.50) ] Release date: 11 Feb 2014
+
+This is the first beta of Redis 3.0.0.
+
+The following is a list of improvements in Redis 3.0, compared to Redis 2.8.
+
+* [NEW] Redis Cluster: a distributed implementation of a subset of Redis.
+* [NEW] New "embedded string" object encoding resulting in less cache
+        misses. Big speed gain under certain work loads.
+* [NEW] WAIT command to block waiting for a write to be transmitted to
+        the specified number of slaves.
+* [NEW] MIGRATE connection caching. Much faster keys migraitons.
+* [NEW] MIGARTE new options COPY and REPLACE.
+* [NEW] CLIENT PAUSE command: stop processing client requests for a
+        specified amount of time.
+
+Migrating from 2.8 to 3.0
 =========================
 
-Redis 2.4 is mostly a strict subset of 2.6. However there are a few things
-that you should be aware of:
-
-* You can't use .rdb and AOF files generated with 2.6 into a 2.4 instance.
-* 2.6 slaves can be attached to 2.4 masters, but not the contrary, and only
-  for the time needed to perform the version upgrade.
-
-There are also a few API differences, that are unlikely to cause problems,
-but it is better to keep them in mind:
-
-* SORT now will refuse to sort in numerical mode elements that can't be parsed
-  as numbers.
-* EXPIREs now all have millisecond resolution (but this is very unlikely to
-  break code that was not conceived exploting the previous resolution error
-  in some way.)
-* INFO output is a bit different now, and contains empty lines and comments
-  starting with '#'. All the major clients should be already fixed to work
-  with the new INFO format.
-
-Also the following redis.conf and CONFIG GET / SET parameters changed name:
-
-    * hash-max-zipmap-entries, now replaced by hash-max-ziplist-entries
-    * hash-max-zipmap-value, now replaced by hash-max-ziplist-value
-    * glueoutputbuf was now completely removed as it does not make sense
-
----------
-CHANGELOG
----------
-
-What's new in Redis 2.6.0
-=========================
-
-UPGRADE URGENCY: We suggest new users to start with 2.6.0, and old users to
-                 upgrade after some testing of the application with the new
-                 Redis version.
-
-* Server side Lua scripting, see http://redis.io/commands/eval
-* Virtual Memory removed (was deprecated in 2.4)
-* Hardcoded limits about max number of clients removed.
-* AOF low level semantics is generally more sane, and especially when used
-  in slaves.
-* Milliseconds resolution expires, also added new commands with milliseconds
-  precision (PEXPIRE, PTTL, ...).
-* Clients max output buffer soft and hard limits. You can specifiy different
-  limits for different classes of clients (normal,pubsub,slave).
-* AOF is now able to rewrite aggregate data types using variadic commands,
-  often producing an AOF that is faster to save, load, and is smaller in size.
-* Every redis.conf directive is now accepted as a command line option for the
-  redis-server binary, with the same name and number of arguments.
-* Hash table seed randomization for protection against collisions attacks.
-* Performances improved when writing large objects to Redis.
-* Significant parts of the core refactored or rewritten. New internal APIs
-  and core changes allowed to develop Redis Cluster on top of the new code,
-  however for 2.6 all the cluster code was removed, and will be released with
-  Redis 3.0 when it is more complete and stable.
-* Redis ASCII art logo added at startup.
-* Crash report on memory violation or failed asserts improved significantly
-  to make debugging of hard to catch bugs simpler.
-* redis-benchmark improvements: ability to run selected tests,
-  CSV output, faster, better help.
-* redis-cli improvements: --eval for comfortable development of Lua scripts.
-* SHUTDOWN now supports two optional arguments: "SAVE" and "NOSAVE".
-* INFO output split into sections, the command is now able to just show 
-  pecific sections.
-* New statistics about how many time a command was called, and how much
-  execution time it used (INFO commandstats).
-* More predictable SORT behavior in edge cases.
-* INCRBYFLOAT and HINCRBYFLOAT commands.
+Redis 3.0 is mostly a strict subset of 2.8, you should not have any problem
+upgrading your application from 2.8 to 3.0.
 
 --------------------------------------------------------------------------------
 
-Credits: Where not specified the implementation and design are done by
-Salvatore Sanfilippo and Pieter Noordhuis. Thanks to VMware for making all
-this possible. Also many thanks to all the other contributors and the amazing
-community we have.
+Credits: Where not specified the implementation and design is done by
+Salvatore Sanfilippo. Thanks to Pivotal for making all this possible.
+Also many thanks to all the other contributors and the amazing community
+we have.
 
 See commit messages for more credits.
 

runtest-sentinel

@@ -0,0 +1,14 @@
+#!/bin/sh
+TCL_VERSIONS="8.5 8.6"
+TCLSH=""
+
+for VERSION in $TCL_VERSIONS; do
+	TCL=`which tclsh$VERSION 2>/dev/null` && TCLSH=$TCL
+done
+
+if [ -z $TCLSH ]
+then
+    echo "You need tcl 8.5 or newer in order to run the Redis Sentinel test"
+    exit 1
+fi
+$TCLSH tests/sentinel.tcl $*

sentinel.conf

@@ -86,10 +86,10 @@ sentinel failover-timeout mymaster 180000
 # or to reconfigure clients after a failover. The scripts are executed
 # with the following rules for error handling:
 #
-# If script exists with "1" the execution is retried later (up to a maximum
+# If script exits with "1" the execution is retried later (up to a maximum
 # number of times currently set to 10).
 #
-# If script exists with "2" (or an higher value) the script execution is
+# If script exits with "2" (or an higher value) the script execution is
 # not retried.
 #
 # If script terminates because it receives a signal the behavior is the same

src/Makefile

@@ -204,6 +204,9 @@ distclean: clean
 test: $(REDIS_SERVER_NAME) $(REDIS_CHECK_AOF_NAME)
 	@(cd ..; ./runtest)
 
+test-sentinel: $(REDIS_SENTINEL_NAME)
+	@(cd ..; ./runtest-sentinel)
+
 check: test
 
 lcov:

src/anet.c

@@ -234,11 +234,12 @@ static int anetCreateSocket(char *err, int domain) {
 
 #define ANET_CONNECT_NONE 0
 #define ANET_CONNECT_NONBLOCK 1
-static int anetTcpGenericConnect(char *err, char *addr, int port, int flags)
+static int anetTcpGenericConnect(char *err, char *addr, int port,
+                                 char *source_addr, int flags)
 {
     int s = ANET_ERR, rv;
     char portstr[6];  /* strlen("65535") + 1; */
-    struct addrinfo hints, *servinfo, *p;
+    struct addrinfo hints, *servinfo, *bservinfo, *p, *b;
 
     snprintf(portstr,sizeof(portstr),"%d",port);
     memset(&hints,0,sizeof(hints));
@@ -258,6 +259,24 @@ static int anetTcpGenericConnect(char *err, char *addr, int port, int flags)
         if (anetSetReuseAddr(err,s) == ANET_ERR) goto error;
         if (flags & ANET_CONNECT_NONBLOCK && anetNonBlock(err,s) != ANET_OK)
             goto error;
+        if (source_addr) {
+            int bound = 0;
+            /* Using getaddrinfo saves us from self-determining IPv4 vs IPv6 */
+            if ((rv = getaddrinfo(source_addr, NULL, &hints, &bservinfo)) != 0) {
+                anetSetError(err, "%s", gai_strerror(rv));
+                goto end;
+            }
+            for (b = bservinfo; b != NULL; b = b->ai_next) {
+                if (bind(s,b->ai_addr,b->ai_addrlen) != -1) {
+                    bound = 1;
+                    break;
+                }
+            }
+            if (!bound) {
+                anetSetError(err, "bind: %s", strerror(errno));
+                goto end;
+            }
+        }
         if (connect(s,p->ai_addr,p->ai_addrlen) == -1) {
             /* If the socket is non-blocking, it is ok for connect() to
              * return an EINPROGRESS error here. */
@@ -287,12 +306,17 @@ end:
 
 int anetTcpConnect(char *err, char *addr, int port)
 {
-    return anetTcpGenericConnect(err,addr,port,ANET_CONNECT_NONE);
+    return anetTcpGenericConnect(err,addr,port,NULL,ANET_CONNECT_NONE);
 }
 
 int anetTcpNonBlockConnect(char *err, char *addr, int port)
 {
-    return anetTcpGenericConnect(err,addr,port,ANET_CONNECT_NONBLOCK);
+    return anetTcpGenericConnect(err,addr,port,NULL,ANET_CONNECT_NONBLOCK);
+}
+
+int anetTcpNonBlockBindConnect(char *err, char *addr, int port, char *source_addr)
+{
+    return anetTcpGenericConnect(err,addr,port,source_addr,ANET_CONNECT_NONBLOCK);
 }
 
 int anetUnixGenericConnect(char *err, char *path, int flags)

src/anet.h

@@ -45,6 +45,7 @@
 
 int anetTcpConnect(char *err, char *addr, int port);
 int anetTcpNonBlockConnect(char *err, char *addr, int port);
+int anetTcpNonBlockBindConnect(char *err, char *addr, int port, char *source_addr);
 int anetUnixConnect(char *err, char *path);
 int anetUnixNonBlockConnect(char *err, char *path);
 int anetRead(int fd, char *buf, int count);

src/aof.c

@@ -226,6 +226,7 @@ int startAppendOnly(void) {
  *
  * However if force is set to 1 we'll write regardless of the background
  * fsync. */
+#define AOF_WRITE_LOG_ERROR_RATE 30 /* Seconds between errors logging. */
 void flushAppendOnlyFile(int force) {
     ssize_t nwritten;
     int sync_in_progress = 0;
@@ -267,27 +268,76 @@ void flushAppendOnlyFile(int force) {
      * or alike */
     nwritten = write(server.aof_fd,server.aof_buf,sdslen(server.aof_buf));
     if (nwritten != (signed)sdslen(server.aof_buf)) {
-        /* Ooops, we are in troubles. The best thing to do for now is
-         * aborting instead of giving the illusion that everything is
-         * working as expected. */
+        static time_t last_write_error_log = 0;
+        int can_log = 0;
+
+        /* Limit logging rate to 1 line per AOF_WRITE_LOG_ERROR_RATE seconds. */
+        if ((server.unixtime - last_write_error_log) > AOF_WRITE_LOG_ERROR_RATE) {
+            can_log = 1;
+            last_write_error_log = server.unixtime;
+        }
+
+        /* Lof the AOF write error and record the error code. */
         if (nwritten == -1) {
-            redisLog(REDIS_WARNING,"Exiting on error writing to the append-only file: %s",strerror(errno));
+            if (can_log) {
+                redisLog(REDIS_WARNING,"Error writing to the AOF file: %s",
+                    strerror(errno));
+                server.aof_last_write_errno = errno;
+            }
         } else {
-            redisLog(REDIS_WARNING,"Exiting on short write while writing to "
-                                   "the append-only file: %s (nwritten=%ld, "
-                                   "expected=%ld)",
-                                   strerror(errno),
-                                   (long)nwritten,
-                                   (long)sdslen(server.aof_buf));
+            if (can_log) {
+                redisLog(REDIS_WARNING,"Short write while writing to "
+                                       "the AOF file: (nwritten=%lld, "
+                                       "expected=%lld)",
+                                       (long long)nwritten,
+                                       (long long)sdslen(server.aof_buf));
+            }
 
             if (ftruncate(server.aof_fd, server.aof_current_size) == -1) {
-                redisLog(REDIS_WARNING, "Could not remove short write "
-                         "from the append-only file.  Redis may refuse "
-                         "to load the AOF the next time it starts.  "
-                         "ftruncate: %s", strerror(errno));
+                if (can_log) {
+                    redisLog(REDIS_WARNING, "Could not remove short write "
+                             "from the append-only file.  Redis may refuse "
+                             "to load the AOF the next time it starts.  "
+                             "ftruncate: %s", strerror(errno));
+                }
+            } else {
+                /* If the ftrunacate() succeeded we can set nwritten to
+                 * -1 since there is no longer partial data into the AOF. */
+                nwritten = -1;
             }
+            server.aof_last_write_errno = ENOSPC;
+        }
+
+        /* Handle the AOF write error. */
+        if (server.aof_fsync == AOF_FSYNC_ALWAYS) {
+            /* We can't recover when the fsync policy is ALWAYS since the
+             * reply for the client is already in the output buffers, and we
+             * have the contract with the user that on acknowledged write data
+             * is synched on disk. */
+            redisLog(REDIS_WARNING,"Can't recover from AOF write error when the AOF fsync policy is 'always'. Exiting...");
+            exit(1);
+        } else {
+            /* Recover from failed write leaving data into the buffer. However
+             * set an error to stop accepting writes as long as the error
+             * condition is not cleared. */
+            server.aof_last_write_status = REDIS_ERR;
+
+            /* Trim the sds buffer if there was a partial write, and there
+             * was no way to undo it with ftruncate(2). */
+            if (nwritten > 0) {
+                server.aof_current_size += nwritten;
+                sdsrange(server.aof_buf,nwritten,-1);
+            }
+            return; /* We'll try again on the next call... */
+        }
+    } else {
+        /* Successful write(2). If AOF was in error state, restore the
+         * OK state and log the event. */
+        if (server.aof_last_write_status == REDIS_ERR) {
+            redisLog(REDIS_WARNING,
+                "AOF write error looks solved, Redis can write again.");
+            server.aof_last_write_status = REDIS_OK;
         }
-        exit(1);
     }
     server.aof_current_size += nwritten;
 

src/bitops.c

@@ -60,11 +60,18 @@ static int getBitOffsetFromArgument(redisClient *c, robj *o, size_t *offset) {
  * work with a input string length up to 512 MB. */
 size_t redisPopcount(void *s, long count) {
     size_t bits = 0;
-    unsigned char *p;
-    uint32_t *p4 = s;
+    unsigned char *p = s;
+    uint32_t *p4;
     static const unsigned char bitsinbyte[256] = {0,1,1,2,1,2,2,3,1,2,2,3,2,3,3,4,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,4,5,5,6,5,6,6,7,5,6,6,7,6,7,7,8};
 
+    /* Count initial bytes not aligned to 32 bit. */
+    while((unsigned long)p & 3 && count) {
+        bits += bitsinbyte[*p++];
+        count--;
+    }
+
     /* Count bits 16 bytes at a time */
+    p4 = (uint32_t*)p;
     while(count>=16) {
         uint32_t aux1, aux2, aux3, aux4;
 
@@ -87,12 +94,99 @@ size_t redisPopcount(void *s, long count) {
                 ((((aux3 + (aux3 >> 4)) & 0x0F0F0F0F) * 0x01010101) >> 24) +
                 ((((aux4 + (aux4 >> 4)) & 0x0F0F0F0F) * 0x01010101) >> 24);
     }
-    /* Count the remaining bytes */
+    /* Count the remaining bytes. */
     p = (unsigned char*)p4;
     while(count--) bits += bitsinbyte[*p++];
     return bits;
 }
 
+/* Return the position of the first bit set to one (if 'bit' is 1) or
+ * zero (if 'bit' is 0) in the bitmap starting at 's' and long 'count' bytes.
+ *
+ * The function is guaranteed to return a value >= 0 if 'bit' is 0 since if
+ * no zero bit is found, it returns count*8 assuming the string is zero
+ * padded on the right. However if 'bit' is 1 it is possible that there is
+ * not a single set bit in the bitmap. In this special case -1 is returned. */
+long redisBitpos(void *s, long count, int bit) {
+    unsigned long *l;
+    unsigned char *c;
+    unsigned long skipval, word = 0, one;
+    long pos = 0; /* Position of bit, to return to the caller. */
+    int j;
+
+    /* Process whole words first, seeking for first word that is not
+     * all ones or all zeros respectively if we are lookig for zeros
+     * or ones. This is much faster with large strings having contiguous
+     * blocks of 1 or 0 bits compared to the vanilla bit per bit processing.
+     *
+     * Note that if we start from an address that is not aligned
+     * to sizeof(unsigned long) we consume it byte by byte until it is
+     * aligned. */
+
+    /* Skip initial bits not aligned to sizeof(unsigned long) byte by byte. */
+    skipval = bit ? 0 : UCHAR_MAX;
+    c = (unsigned char*) s;
+    while((unsigned long)c & (sizeof(*l)-1) && count) {
+        if (*c != skipval) break;
+        c++;
+        count--;
+        pos += 8;
+    }
+
+    /* Skip bits with full word step. */
+    skipval = bit ? 0 : ULONG_MAX;
+    l = (unsigned long*) c;
+    while (count >= sizeof(*l)) {
+        if (*l != skipval) break;
+        l++;
+        count -= sizeof(*l);
+        pos += sizeof(*l)*8;
+    }
+
+    /* Load bytes into "word" considering the first byte as the most significant
+     * (we basically consider it as written in big endian, since we consider the
+     * string as a set of bits from left to right, with the first bit at position
+     * zero.
+     *
+     * Note that the loading is designed to work even when the bytes left
+     * (count) are less than a full word. We pad it with zero on the right. */
+    c = (unsigned char*)l;
+    for (j = 0; j < sizeof(*l); j++) {
+        word <<= 8;
+        if (count) {
+            word |= *c;
+            c++;
+            count--;
+        }
+    }
+
+    /* Special case:
+     * If bits in the string are all zero and we are looking for one,
+     * return -1 to signal that there is not a single "1" in the whole
+     * string. This can't happen when we are looking for "0" as we assume
+     * that the right of the string is zero padded. */
+    if (bit == 1 && word == 0) return -1;
+
+    /* Last word left, scan bit by bit. The first thing we need is to
+     * have a single "1" set in the most significant position in an
+     * unsigned long. We don't know the size of the long so we use a
+     * simple trick. */
+    one = ULONG_MAX; /* All bits set to 1.*/
+    one >>= 1;       /* All bits set to 1 but the MSB. */
+    one = ~one;      /* All bits set to 0 but the MSB. */
+
+    while(one) {
+        if (((one & word) != 0) == bit) return pos;
+        pos++;
+        one >>= 1;
+    }
+
+    /* If we reached this point, there is a bug in the algorithm, since
+     * the case of no match is handled as a special case before. */
+    redisPanic("End of redisBitpos() reached.");
+    return 0; /* Just to avoid warnings. */
+}
+
 /* -----------------------------------------------------------------------------
  * Bits related string commands: GETBIT, SETBIT, BITCOUNT, BITOP.
  * -------------------------------------------------------------------------- */
@@ -410,3 +504,90 @@ void bitcountCommand(redisClient *c) {
         addReplyLongLong(c,redisPopcount(p+start,bytes));
     }
 }
+
+/* BITPOS key bit [start [end]] */
+void bitposCommand(redisClient *c) {
+    robj *o;
+    long bit, start, end, strlen;
+    unsigned char *p;
+    char llbuf[32];
+    int end_given = 0;
+
+    /* Parse the bit argument to understand what we are looking for, set
+     * or clear bits. */
+    if (getLongFromObjectOrReply(c,c->argv[2],&bit,NULL) != REDIS_OK)
+        return;
+    if (bit != 0 && bit != 1) {
+        addReplyError(c, "The bit argument must be 1 or 0.");
+        return;
+    }
+
+    /* If the key does not exist, from our point of view it is an infinite
+     * array of 0 bits. If the user is looking for the fist clear bit return 0,
+     * If the user is looking for the first set bit, return -1. */
+    if ((o = lookupKeyRead(c->db,c->argv[1])) == NULL) {
+        addReplyLongLong(c, bit ? -1 : 0);
+        return;
+    }
+    if (checkType(c,o,REDIS_STRING)) return;
+
+    /* Set the 'p' pointer to the string, that can be just a stack allocated
+     * array if our string was integer encoded. */
+    if (o->encoding == REDIS_ENCODING_INT) {
+        p = (unsigned char*) llbuf;
+        strlen = ll2string(llbuf,sizeof(llbuf),(long)o->ptr);
+    } else {
+        p = (unsigned char*) o->ptr;
+        strlen = sdslen(o->ptr);
+    }
+
+    /* Parse start/end range if any. */
+    if (c->argc == 4 || c->argc == 5) {
+        if (getLongFromObjectOrReply(c,c->argv[3],&start,NULL) != REDIS_OK)
+            return;
+        if (c->argc == 5) {
+            if (getLongFromObjectOrReply(c,c->argv[4],&end,NULL) != REDIS_OK)
+                return;
+            end_given = 1;
+        } else {
+            end = strlen-1;
+        }
+        /* Convert negative indexes */
+        if (start < 0) start = strlen+start;
+        if (end < 0) end = strlen+end;
+        if (start < 0) start = 0;
+        if (end < 0) end = 0;
+        if (end >= strlen) end = strlen-1;
+    } else if (c->argc == 3) {
+        /* The whole string. */
+        start = 0;
+        end = strlen-1;
+    } else {
+        /* Syntax error. */
+        addReply(c,shared.syntaxerr);
+        return;
+    }
+
+    /* For empty ranges (start > end) we return -1 as an empty range does
+     * not contain a 0 nor a 1. */
+    if (start > end) {
+        addReplyLongLong(c, -1);
+    } else {
+        long bytes = end-start+1;
+        long pos = redisBitpos(p+start,bytes,bit);
+
+        /* If we are looking for clear bits, and the user specified an exact
+         * range with start-end, we can't consider the right of the range as
+         * zero padded (as we do when no explicit end is given).
+         *
+         * So if redisBitpos() returns the first bit outside the range,
+         * we return -1 to the caller, to mean, in the specified range there
+         * is not a single "0" bit. */
+        if (end_given && bit == 0 && pos == bytes*8) {
+            addReplyLongLong(c,-1);
+            return;
+        }
+        if (pos != -1) pos += start*8; /* Adjust for the bytes we skipped. */
+        addReplyLongLong(c,pos);
+    }
+}

src/cluster.c

@@ -68,26 +68,25 @@ int bitmapTestBit(unsigned char *bitmap, int pos);
 void clusterDoBeforeSleep(int flags);
 void clusterSendUpdate(clusterLink *link, clusterNode *node);
 void resetManualFailover(void);
+void clusterCloseAllSlots(void);
 
 /* -----------------------------------------------------------------------------
  * Initialization
  * -------------------------------------------------------------------------- */
 
-/* This function is called at startup in order to set the currentEpoch
- * (which is not saved on permanent storage) to the greatest configEpoch found
- * in the loaded nodes (configEpoch is stored on permanent storage as soon as
- * it changes for some node). */
-void clusterSetStartupEpoch() {
+/* Return the greatest configEpoch found in the cluster. */
+uint64_t clusterGetMaxEpoch(void) {
+    uint64_t max = 0;
     dictIterator *di;
     dictEntry *de;
 
     di = dictGetSafeIterator(server.cluster->nodes);
     while((de = dictNext(di)) != NULL) {
         clusterNode *node = dictGetVal(de);
-        if (node->configEpoch > server.cluster->currentEpoch)
-            server.cluster->currentEpoch = node->configEpoch;
+        if (node->configEpoch > max) max = node->configEpoch;
     }
     dictReleaseIterator(di);
+    return max;
 }
 
 int clusterLoadConfig(char *filename) {
@@ -227,7 +226,10 @@ int clusterLoadConfig(char *filename) {
     /* Config sanity check */
     redisAssert(server.cluster->myself != NULL);
     redisLog(REDIS_NOTICE,"Node configuration loaded, I'm %.40s", myself->name);
-    clusterSetStartupEpoch();
+    /* Set the currentEpoch to the max epoch found in the master.
+     * FIXME: this should actually be part of the persistent state, as
+     * documented in the Github issue #1479. */
+    server.cluster->currentEpoch = clusterGetMaxEpoch();
     return REDIS_OK;
 
 fmterr:
@@ -308,12 +310,8 @@ void clusterInit(void) {
     server.cluster->last_vote_epoch = 0;
     server.cluster->stats_bus_messages_sent = 0;
     server.cluster->stats_bus_messages_received = 0;
-    memset(server.cluster->migrating_slots_to,0,
-        sizeof(server.cluster->migrating_slots_to));
-    memset(server.cluster->importing_slots_from,0,
-        sizeof(server.cluster->importing_slots_from));
-    memset(server.cluster->slots,0,
-        sizeof(server.cluster->slots));
+    memset(server.cluster->slots,0, sizeof(server.cluster->slots));
+    clusterCloseAllSlots();
     if (clusterLoadConfig(server.cluster_configfile) == REDIS_ERR) {
         /* No configuration found. We will just use the random name provided
          * by the createClusterNode() function. */
@@ -328,6 +326,19 @@ void clusterInit(void) {
 
     /* We need a listening TCP port for our cluster messaging needs. */
     server.cfd_count = 0;
+
+    /* Port sanity check II
+     * The other handshake port check is triggered too late to stop
+     * us from trying to use a too-high cluster port number. */
+    if (server.port > (65535-REDIS_CLUSTER_PORT_INCR)) {
+        redisLog(REDIS_WARNING, "Redis port number too high. "
+                   "Cluster communication port is 10,000 port "
+                   "numbers higher than your Redis port. "
+                   "Your Redis port number must be "
+                   "lower than 55535.");
+        exit(1);
+    }
+
     if (listenToPort(server.port+REDIS_CLUSTER_PORT_INCR,
         server.cfd,&server.cfd_count) == REDIS_ERR)
     {
@@ -387,7 +398,7 @@ void clusterAcceptHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
     REDIS_NOTUSED(privdata);
 
     cfd = anetTcpAccept(server.neterr, fd, cip, sizeof(cip), &cport);
-    if (cfd == AE_ERR) {
+    if (cfd == ANET_ERR) {
         redisLog(REDIS_VERBOSE,"Accepting cluster node: %s", server.neterr);
         return;
     }
@@ -663,7 +674,11 @@ void clusterDelNode(clusterNode *delnode) {
     }
     dictReleaseIterator(di);
 
-    /* 3) Free the node, unlinking it from the cluster. */
+    /* 3) Remove this node from its master's slaves if needed. */
+    if (nodeIsSlave(delnode) && delnode->slaveof)
+        clusterNodeRemoveSlave(delnode->slaveof,delnode);
+
+    /* 4) Free the node, unlinking it from the cluster. */
     freeClusterNode(delnode);
 }
 
@@ -830,10 +845,11 @@ void clearNodeFailureIfNeeded(clusterNode *node) {
 
     /* For slaves we always clear the FAIL flag if we can contact the
      * node again. */
-    if (nodeIsSlave(node)) {
+    if (nodeIsSlave(node) || node->numslots == 0) {
         redisLog(REDIS_NOTICE,
-            "Clear FAIL state for node %.40s: slave is reachable again.",
-                node->name);
+            "Clear FAIL state for node %.40s: %s is reachable again.",
+                node->name,
+                nodeIsSlave(node) ? "slave" : "master without slots");
         node->flags &= ~REDIS_NODE_FAIL;
         clusterDoBeforeSleep(CLUSTER_TODO_UPDATE_STATE|CLUSTER_TODO_SAVE_CONFIG);
     }
@@ -909,11 +925,11 @@ int clusterStartHandshake(char *ip, int port) {
     if (sa.ss_family == AF_INET)
         inet_ntop(AF_INET,
             (void*)&(((struct sockaddr_in *)&sa)->sin_addr),
-            norm_ip,REDIS_CLUSTER_IPLEN);
+            norm_ip,REDIS_IP_STR_LEN);
     else
         inet_ntop(AF_INET6,
             (void*)&(((struct sockaddr_in6 *)&sa)->sin6_addr),
-            norm_ip,REDIS_CLUSTER_IPLEN);
+            norm_ip,REDIS_IP_STR_LEN);
 
     if (clusterHandshakeInProgress(norm_ip,port)) {
         errno = EAGAIN;
@@ -1015,22 +1031,9 @@ void clusterProcessGossipSection(clusterMsg *hdr, clusterLink *link) {
 
 /* IP -> string conversion. 'buf' is supposed to at least be 46 bytes. */
 void nodeIp2String(char *buf, clusterLink *link) {
-    struct sockaddr_storage sa;
-    socklen_t salen = sizeof(sa);
-
-    if (getpeername(link->fd, (struct sockaddr*) &sa, &salen) == -1)
-        redisPanic("getpeername() failed.");
-
-    if (sa.ss_family == AF_INET) {
-        struct sockaddr_in *s = (struct sockaddr_in *)&sa;
-        inet_ntop(AF_INET,(void*)&(s->sin_addr),buf,REDIS_CLUSTER_IPLEN);
-    } else {
-        struct sockaddr_in6 *s = (struct sockaddr_in6 *)&sa;
-        inet_ntop(AF_INET6,(void*)&(s->sin6_addr),buf,REDIS_CLUSTER_IPLEN);
-    }
+    anetPeerToString(link->fd, buf, REDIS_IP_STR_LEN, NULL);
 }
 
-
 /* Update the node address to the IP address that can be extracted
  * from link->fd, and at the specified port.
  * Also disconnect the node link so that we'll connect again to the new
@@ -1096,9 +1099,7 @@ void clusterSetNodeAsMaster(clusterNode *n) {
  * The 'sender' is the node for which we received a configuration update.
  * Sometimes it is not actaully the "Sender" of the information, like in the case
  * we receive the info via an UPDATE packet. */
-void clusterUpdateSlotsConfigWith(clusterNode *sender, uint64_t senderConfigEpoch,
-                                  unsigned char *slots)
-{
+void clusterUpdateSlotsConfigWith(clusterNode *sender, uint64_t senderConfigEpoch, unsigned char *slots) {
     int j;
     clusterNode *curmaster, *newmaster = NULL;
 
@@ -1109,14 +1110,34 @@ void clusterUpdateSlotsConfigWith(clusterNode *sender, uint64_t senderConfigEpoc
 
     for (j = 0; j < REDIS_CLUSTER_SLOTS; j++) {
         if (bitmapTestBit(slots,j)) {
-            /* We rebind the slot to the new node claiming it if:
-             * 1) The slot was unassigned.
-             * 2) The new node claims it with a greater configEpoch. */
+            /* The slot is already bound to the sender of this message. */
             if (server.cluster->slots[j] == sender) continue;
+
+            /* The slot is in importing state, it should be modified only
+             * manually via redis-trib (example: a resharding is in progress
+             * and the migrating side slot was already closed and is advertising
+             * a new config. We still want the slot to be closed manually). */
+            if (server.cluster->importing_slots_from[j]) continue;
+
+            /* We rebind the slot to the new node claiming it if:
+             * 1) The slot was unassigned or the new node claims it with a
+             *    greater configEpoch.
+             * 2) We are not currently importing the slot. */
             if (server.cluster->slots[j] == NULL ||
-                server.cluster->slots[j]->configEpoch <
-                senderConfigEpoch)
+                server.cluster->slots[j]->configEpoch < senderConfigEpoch)
             {
+                /* Was this slot mine, and still contains keys? Something
+                 * odd happened, put the slot in importing state so that
+                 * redis-trib fix can detect the condition (and no further
+                 * updates will be processed before the slot gets fixed). */
+                if (server.cluster->slots[j] == myself &&
+                    countKeysInSlot(j) &&
+                    sender != myself)
+                {
+                    redisLog(REDIS_WARNING,"Slot update for a slot I still have keys received. Putting the slot in IMPORTING state. Please run the 'redis-trib fix' command.");
+                    server.cluster->importing_slots_from[j] = sender;
+                }
+
                 if (server.cluster->slots[j] == curmaster)
                     newmaster = sender;
                 clusterDelSlot(j);
@@ -1166,7 +1187,8 @@ int clusterProcessPacket(clusterLink *link) {
         type, (unsigned long) totlen);
 
     /* Perform sanity checks */
-    if (totlen < 8) return 1;
+    if (totlen < 16) return 1; /* At least signature, version, totlen, count. */
+    if (ntohs(hdr->ver) != 0) return 1; /* Can't handle versions other than 0. */
     if (totlen > sdslen(link->rcvbuf)) return 1;
     if (type == CLUSTERMSG_TYPE_PING || type == CLUSTERMSG_TYPE_PONG ||
         type == CLUSTERMSG_TYPE_MEET)
@@ -1360,7 +1382,7 @@ int clusterProcessPacket(clusterLink *link) {
                 }
 
                 /* Master node changed for this slave? */
-                if (sender->slaveof != master) {
+                if (master && sender->slaveof != master) {
                     if (sender->slaveof)
                         clusterNodeRemoveSlave(sender->slaveof,sender);
                     clusterNodeAddSlave(master,sender);
@@ -1426,7 +1448,7 @@ int clusterProcessPacket(clusterLink *link) {
                     if (server.cluster->slots[j]->configEpoch >
                         senderConfigEpoch)
                     {
-                        redisLog(REDIS_WARNING,
+                        redisLog(REDIS_VERBOSE,
                             "Node %.40s has old slots configuration, sending "
                             "an UPDATE message about %.40s",
                                 sender->name, server.cluster->slots[j]->name);
@@ -1524,6 +1546,10 @@ int clusterProcessPacket(clusterLink *link) {
         /* If in our current config the node is a slave, set it as a master. */
         if (nodeIsSlave(n)) clusterSetNodeAsMaster(n);
 
+        /* Update the node's configEpoch. */
+        n->configEpoch = reportedConfigEpoch;
+        clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG|CLUSTER_TODO_FSYNC_CONFIG);
+
         /* Check the bitmap of served slots and udpate our
          * config accordingly. */
         clusterUpdateSlotsConfigWith(n,reportedConfigEpoch,
@@ -1579,18 +1605,22 @@ void clusterReadHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
 
     while(1) { /* Read as long as there is data to read. */
         rcvbuflen = sdslen(link->rcvbuf);
-        if (rcvbuflen < 4) {
-            /* First, obtain the first four bytes to get the full message
+        if (rcvbuflen < 8) {
+            /* First, obtain the first 8 bytes to get the full message
              * length. */
-            readlen = 4 - rcvbuflen;
+            readlen = 8 - rcvbuflen;
         } else {
             /* Finally read the full message. */
             hdr = (clusterMsg*) link->rcvbuf;
-            if (rcvbuflen == 4) {
-                /* Perform some sanity check on the message length. */
-                if (ntohl(hdr->totlen) < CLUSTERMSG_MIN_LEN) {
+            if (rcvbuflen == 8) {
+                /* Perform some sanity check on the message signature
+                 * and length. */
+                if (memcmp(hdr->sig,"RCmb",4) != 0 ||
+                    ntohl(hdr->totlen) < CLUSTERMSG_MIN_LEN)
+                {
                     redisLog(REDIS_WARNING,
-                        "Bad message length received from Cluster bus.");
+                        "Bad message length or signature received "
+                        "from Cluster bus.");
                     handleLinkIOError(link);
                     return;
                 }
@@ -1616,7 +1646,7 @@ void clusterReadHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
         }
 
         /* Total length obtained? Process this packet. */
-        if (rcvbuflen >= 4 && rcvbuflen == ntohl(hdr->totlen)) {
+        if (rcvbuflen >= 8 && rcvbuflen == ntohl(hdr->totlen)) {
             if (clusterProcessPacket(link)) {
                 sdsfree(link->rcvbuf);
                 link->rcvbuf = sdsempty();
@@ -1677,6 +1707,10 @@ void clusterBuildMessageHdr(clusterMsg *hdr, int type) {
               myself->slaveof : myself;
 
     memset(hdr,0,sizeof(*hdr));
+    hdr->sig[0] = 'R';
+    hdr->sig[1] = 'C';
+    hdr->sig[2] = 'm';
+    hdr->sig[3] = 'b';
     hdr->type = htons(type);
     memcpy(hdr->sender,myself->name,REDIS_CLUSTER_NAMELEN);
 
@@ -2047,6 +2081,18 @@ void clusterHandleSlaveFailover(void) {
     int manual_failover = server.cluster->mf_end != 0 &&
                           server.cluster->mf_can_start;
     int j;
+    mstime_t auth_timeout, auth_retry_time;
+
+    /* Compute the failover timeout (the max time we have to send votes
+     * and wait for replies), and the failover retry time (the time to wait
+     * before waiting again.
+     *
+     * Timeout is MIN(NODE_TIMEOUT*2,2000) milliseconds.
+     * Retry is two times the Timeout.
+     */
+    auth_timeout = server.cluster_node_timeout*2;
+    if (auth_timeout < 2000) auth_timeout = 2000;
+    auth_retry_time = auth_timeout*2;
 
     /* Pre conditions to run the function:
      * 1) We are a slave.
@@ -2057,8 +2103,6 @@ void clusterHandleSlaveFailover(void) {
         (!nodeFailed(myself->slaveof) && !manual_failover) ||
         myself->slaveof->numslots == 0) return;
 
-    /* If this is a manual failover, are we ready to start? */
-
     /* Set data_age to the number of seconds we are disconnected from
      * the master. */
     if (server.repl_state == REDIS_REPL_CONNECTED) {
@@ -2081,10 +2125,9 @@ void clusterHandleSlaveFailover(void) {
         (server.cluster_node_timeout * REDIS_CLUSTER_SLAVE_VALIDITY_MULT))
         return;
 
-    /* Compute the time at which we can start an election. */
-    if (auth_age >
-        server.cluster_node_timeout * REDIS_CLUSTER_FAILOVER_AUTH_RETRY_MULT)
-    {
+    /* If the previous failover attempt timedout and the retry time has
+     * elapsed, we can setup a new one. */
+    if (auth_age > auth_retry_time) {
         server.cluster->failover_auth_time = mstime() +
             500 + /* Fixed delay of 500 milliseconds, let FAIL msg propagate. */
             random() % 500; /* Random delay between 0 and 500 milliseconds. */
@@ -2136,7 +2179,7 @@ void clusterHandleSlaveFailover(void) {
     if (mstime() < server.cluster->failover_auth_time) return;
 
     /* Return ASAP if the election is too old to be valid. */
-    if (auth_age > server.cluster_node_timeout) return;
+    if (auth_age > auth_timeout) return;
 
     /* Ask for votes if needed. */
     if (server.cluster->failover_auth_sent == 0) {
@@ -2255,7 +2298,7 @@ void clusterHandleSlaveMigration(int max_slaves) {
         if (nodeIsSlave(node) || nodeFailed(node)) continue;
         okslaves = clusterCountNonFailingSlaves(node);
 
-        if (okslaves == 0 && target == NULL) target = node;
+        if (okslaves == 0 && target == NULL && node->numslots > 0) target = node;
         if (okslaves == max_slaves) {
             for (j = 0; j < node->numslaves; j++) {
                 if (memcmp(node->slaves[j]->name,
@@ -2395,9 +2438,16 @@ void clusterCron(void) {
             mstime_t old_ping_sent;
             clusterLink *link;
 
-            fd = anetTcpNonBlockConnect(server.neterr, node->ip,
-                node->port+REDIS_CLUSTER_PORT_INCR);
-            if (fd == -1) continue;
+            fd = anetTcpNonBlockBindConnect(server.neterr, node->ip,
+                node->port+REDIS_CLUSTER_PORT_INCR,
+                    server.bindaddr_count ? server.bindaddr[0] : NULL);
+            if (fd == -1) {
+                redisLog(REDIS_DEBUG, "Unable to connect to "
+                    "Cluster Node [%s]:%d -> %s", node->ip,
+                    node->port+REDIS_CLUSTER_PORT_INCR,
+                    server.neterr);
+                continue;
+            }
             link = createClusterLink(node);
             link->fd = fd;
             node->link = link;
@@ -2478,7 +2528,7 @@ void clusterCron(void) {
         if (nodeIsSlave(myself) && nodeIsMaster(node) && !nodeFailed(node)) {
             int okslaves = clusterCountNonFailingSlaves(node);
 
-            if (okslaves == 0) orphaned_masters++;
+            if (okslaves == 0 && node->numslots > 0) orphaned_masters++;
             if (okslaves > max_slaves) max_slaves = okslaves;
             if (nodeIsSlave(myself) && myself->slaveof == node)
                 this_slaves = okslaves;
@@ -2684,6 +2734,15 @@ int clusterDelNodeSlots(clusterNode *node) {
     return deleted;
 }
 
+/* Clear the migrating / importing state for all the slots.
+ * This is useful at initialization and when turning a master into slave. */
+void clusterCloseAllSlots(void) {
+    memset(server.cluster->migrating_slots_to,0,
+        sizeof(server.cluster->migrating_slots_to));
+    memset(server.cluster->importing_slots_from,0,
+        sizeof(server.cluster->importing_slots_from));
+}
+
 /* -----------------------------------------------------------------------------
  * Cluster state evaluation function
  * -------------------------------------------------------------------------- */
@@ -2859,8 +2918,8 @@ int verifyClusterConfigWithData(void) {
  * SLAVE nodes handling
  * -------------------------------------------------------------------------- */
 
-/* Set the specified node 'n' as master. Setup the node as a slave if
- * needed. */
+/* Set the specified node 'n' as master for this node.
+ * If this node is currently a master, it is turned into a slave. */
 void clusterSetMaster(clusterNode *n) {
     redisAssert(n != myself);
     redisAssert(myself->numslots == 0);
@@ -2868,6 +2927,7 @@ void clusterSetMaster(clusterNode *n) {
     if (nodeIsMaster(myself)) {
         myself->flags &= ~REDIS_NODE_MASTER;
         myself->flags |= REDIS_NODE_SLAVE;
+        clusterCloseAllSlots();
     } else {
         if (myself->slaveof)
             clusterNodeRemoveSlave(myself->slaveof,myself);
@@ -3143,10 +3203,10 @@ void clusterCommand(redisClient *c) {
                     return;
                 }
             }
-            /* If this node was the slot owner and the slot was marked as
-             * migrating, assigning the slot to another node will clear
+            /* If this slot is in migrating status but we have no keys
+             * for it assigning the slot to another node will clear
              * the migratig status. */
-            if (server.cluster->slots[slot] == myself &&
+            if (countKeysInSlot(slot) == 0 &&
                 server.cluster->migrating_slots_to[slot])
                 server.cluster->migrating_slots_to[slot] = NULL;
 
@@ -3154,14 +3214,32 @@ void clusterCommand(redisClient *c) {
              * itself also clears the importing status. */
             if (n == myself &&
                 server.cluster->importing_slots_from[slot])
+            {
+                /* This slot was manually migrated, set this node configEpoch
+                 * to a new epoch so that the new version can be propagated
+                 * by the cluster.
+                 *
+                 * FIXME: the new version should be agreed otherwise a race
+                 * is possible if while a manual resharding is in progress
+                 * the master is failed over by a slave. */
+                uint64_t maxEpoch = clusterGetMaxEpoch();
+
+                if (myself->configEpoch == 0 ||
+                    myself->configEpoch != maxEpoch)
+                {
+                    server.cluster->currentEpoch++;
+                    myself->configEpoch = server.cluster->currentEpoch;
+                    clusterDoBeforeSleep(CLUSTER_TODO_FSYNC_CONFIG);
+                }
                 server.cluster->importing_slots_from[slot] = NULL;
+            }
             clusterDelSlot(slot);
             clusterAddSlot(n,slot);
         } else {
             addReplyError(c,"Invalid CLUSTER SETSLOT action or number of arguments");
             return;
         }
-        clusterDoBeforeSleep(CLUSTER_TODO_UPDATE_STATE|CLUSTER_TODO_SAVE_CONFIG);
+        clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG|CLUSTER_TODO_UPDATE_STATE);
         addReply(c,shared.ok);
     } else if (!strcasecmp(c->argv[1]->ptr,"info") && c->argc == 2) {
         /* CLUSTER INFO */
@@ -3783,21 +3861,39 @@ void readwriteCommand(redisClient *c) {
 }
 
 /* Return the pointer to the cluster node that is able to serve the command.
- * For the function to succeed the command should only target a single
- * key (or the same key multiple times).
+ * For the function to succeed the command should only target either:
  *
- * If the returned node should be used only for this request, the *ask
- * integer is set to '1', otherwise to '0'. This is used in order to
- * let the caller know if we should reply with -MOVED or with -ASK.
+ * 1) A single key (even multiple times like LPOPRPUSH mylist mylist).
+ * 2) Multiple keys in the same hash slot, while the slot is stable (no
+ *    resharding in progress).
  *
- * If the command contains multiple keys, and as a consequence it is not
- * possible to handle the request in Redis Cluster, NULL is returned. */
-clusterNode *getNodeByQuery(redisClient *c, struct redisCommand *cmd, robj **argv, int argc, int *hashslot, int *ask) {
+ * On success the function returns the node that is able to serve the request.
+ * If the node is not 'myself' a redirection must be perfomed. The kind of
+ * redirection is specified setting the integer passed by reference
+ * 'error_code', which will be set to REDIS_CLUSTER_REDIR_ASK or
+ * REDIS_CLUSTER_REDIR_MOVED.
+ *
+ * When the node is 'myself' 'error_code' is set to REDIS_CLUSTER_REDIR_NONE.
+ *
+ * If the command fails NULL is returned, and the reason of the failure is
+ * provided via 'error_code', which will be set to:
+ *
+ * REDIS_CLUSTER_REDIR_CROSS_SLOT if the request contains multiple keys that
+ * don't belong to the same hash slot.
+ *
+ * REDIS_CLUSTER_REDIR_UNSTABLE if the request contains mutliple keys
+ * belonging to the same slot, but the slot is not stable (in migration or
+ * importing state, likely because a resharding is in progress). */
+clusterNode *getNodeByQuery(redisClient *c, struct redisCommand *cmd, robj **argv, int argc, int *hashslot, int *error_code) {
     clusterNode *n = NULL;
     robj *firstkey = NULL;
+    int multiple_keys = 0;
     multiState *ms, _ms;
     multiCmd mc;
-    int i, slot = 0;
+    int i, slot = 0, migrating_slot = 0, importing_slot = 0, missing_keys = 0;
+
+    /* Set error code optimistically for the base case. */
+    if (error_code) *error_code = REDIS_CLUSTER_REDIR_NONE;
 
     /* We handle all the cases as if they were EXEC commands, so we have
      * a common code path for everything */
@@ -3818,8 +3914,8 @@ clusterNode *getNodeByQuery(redisClient *c, struct redisCommand *cmd, robj **arg
         mc.cmd = cmd;
     }
 
-    /* Check that all the keys are the same key, and get the slot and
-     * node for this key. */
+    /* Check that all the keys are in the same hash slot, and obtain this
+     * slot and the node associated. */
     for (i = 0; i < ms->count; i++) {
         struct redisCommand *mcmd;
         robj **margv;
@@ -3829,51 +3925,91 @@ clusterNode *getNodeByQuery(redisClient *c, struct redisCommand *cmd, robj **arg
         margc = ms->commands[i].argc;
         margv = ms->commands[i].argv;
 
-        keyindex = getKeysFromCommand(mcmd,margv,margc,&numkeys,
-                                      REDIS_GETKEYS_ALL);
+        keyindex = getKeysFromCommand(mcmd,margv,margc,&numkeys);
         for (j = 0; j < numkeys; j++) {
+            robj *thiskey = margv[keyindex[j]];
+            int thisslot = keyHashSlot((char*)thiskey->ptr,
+                                       sdslen(thiskey->ptr));
+
             if (firstkey == NULL) {
                 /* This is the first key we see. Check what is the slot
                  * and node. */
-                firstkey = margv[keyindex[j]];
-
-                slot = keyHashSlot((char*)firstkey->ptr, sdslen(firstkey->ptr));
+                firstkey = thiskey;
+                slot = thisslot;
                 n = server.cluster->slots[slot];
                 redisAssertWithInfo(c,firstkey,n != NULL);
+                /* If we are migrating or importing this slot, we need to check
+                 * if we have all the keys in the request (the only way we
+                 * can safely serve the request, otherwise we return a TRYAGAIN
+                 * error). To do so we set the importing/migrating state and
+                 * increment a counter for every missing key. */
+                if (n == myself &&
+                    server.cluster->migrating_slots_to[slot] != NULL)
+                {
+                    migrating_slot = 1;
+                } else if (server.cluster->importing_slots_from[slot] != NULL) {
+                    importing_slot = 1;
+                }
             } else {
                 /* If it is not the first key, make sure it is exactly
                  * the same key as the first we saw. */
-                if (!equalStringObjects(firstkey,margv[keyindex[j]])) {
-                    getKeysFreeResult(keyindex);
-                    return NULL;
+                if (!equalStringObjects(firstkey,thiskey)) {
+                    if (slot != thisslot) {
+                        /* Error: multiple keys from different slots. */
+                        getKeysFreeResult(keyindex);
+                        if (error_code)
+                            *error_code = REDIS_CLUSTER_REDIR_CROSS_SLOT;
+                        return NULL;
+                    } else {
+                        /* Flag this request as one with multiple different
+                         * keys. */
+                        multiple_keys = 1;
+                    }
                 }
             }
+
+            /* Migarting / Improrting slot? Count keys we don't have. */
+            if ((migrating_slot || importing_slot) &&
+                lookupKeyRead(&server.db[0],thiskey) == NULL)
+            {
+                missing_keys++;
+            }
         }
         getKeysFreeResult(keyindex);
     }
-    if (ask) *ask = 0; /* This is the default. Set to 1 if needed later. */
+
     /* No key at all in command? then we can serve the request
-     * without redirections. */
+     * without redirections or errors. */
     if (n == NULL) return myself;
+
+    /* Return the hashslot by reference. */
     if (hashslot) *hashslot = slot;
+
     /* This request is about a slot we are migrating into another instance?
-     * Then we need to check if we have the key. If we have it we can reply.
-     * If instead is a new key, we pass the request to the node that is
-     * receiving the slot. */
-    if (n == myself && server.cluster->migrating_slots_to[slot] != NULL) {
-        if (lookupKeyRead(&server.db[0],firstkey) == NULL) {
-            if (ask) *ask = 1;
-            return server.cluster->migrating_slots_to[slot];
+     * Then if we have all the keys. */
+
+    /* If we don't have all the keys and we are migrating the slot, send
+     * an ASK redirection. */
+    if (migrating_slot && missing_keys) {
+        if (error_code) *error_code = REDIS_CLUSTER_REDIR_ASK;
+        return server.cluster->migrating_slots_to[slot];
+    }
+
+    /* If we are receiving the slot, and the client correctly flagged the
+     * request as "ASKING", we can serve the request. However if the request
+     * involves multiple keys and we don't have them all, the only option is
+     * to send a TRYAGAIN error. */
+    if (importing_slot &&
+        (c->flags & REDIS_ASKING || cmd->flags & REDIS_CMD_ASKING))
+    {
+        if (multiple_keys && missing_keys) {
+            if (error_code) *error_code = REDIS_CLUSTER_REDIR_UNSTABLE;
+            return NULL;
+        } else {
+            return myself;
         }
     }
-    /* Handle the case in which we are receiving this hash slot from
-     * another instance, so we'll accept the query even if in the table
-     * it is assigned to a different node, but only if the client
-     * issued an ASKING command before. */
-    if (server.cluster->importing_slots_from[slot] != NULL &&
-        (c->flags & REDIS_ASKING || cmd->flags & REDIS_CMD_ASKING)) {
-        return myself;
-    }
+
     /* Handle the read-only client case reading from a slave: if this
      * node is a slave and the request is about an hash slot our master
      * is serving, we can reply without redirection. */
@@ -3884,6 +4020,9 @@ clusterNode *getNodeByQuery(redisClient *c, struct redisCommand *cmd, robj **arg
     {
         return myself;
     }
-    /* It's not a -ASK case. Base case: just return the right node. */
+
+    /* Base case: just return the right node. However if this node is not
+     * myself, set error_code to MOVED since we need to issue a rediretion. */
+    if (n != myself && error_code) *error_code = REDIS_CLUSTER_REDIR_MOVED;
     return n;
 }

src/cluster.h

@@ -10,7 +10,6 @@
 #define REDIS_CLUSTER_FAIL 1        /* The cluster can't work */
 #define REDIS_CLUSTER_NAMELEN 40    /* sha1 hex length */
 #define REDIS_CLUSTER_PORT_INCR 10000 /* Cluster port = baseport + PORT_INCR */
-#define REDIS_CLUSTER_IPLEN INET6_ADDRSTRLEN /* IPv6 address string length */
 
 /* The following defines are amunt of time, sometimes expressed as
  * multiplicators of the node timeout value (when ending with MULT). */
@@ -19,12 +18,18 @@
 #define REDIS_CLUSTER_FAIL_UNDO_TIME_MULT 2 /* Undo fail if master is back. */
 #define REDIS_CLUSTER_FAIL_UNDO_TIME_ADD 10 /* Some additional time. */
 #define REDIS_CLUSTER_SLAVE_VALIDITY_MULT 10 /* Slave data validity. */
-#define REDIS_CLUSTER_FAILOVER_AUTH_RETRY_MULT 4 /* Auth request retry time. */
 #define REDIS_CLUSTER_FAILOVER_DELAY 5 /* Seconds */
 #define REDIS_CLUSTER_DEFAULT_MIGRATION_BARRIER 1
 #define REDIS_CLUSTER_MF_TIMEOUT 5000 /* Milliseconds to do a manual failover. */
 #define REDIS_CLUSTER_MF_PAUSE_MULT 2 /* Master pause manual failover mult. */
 
+/* Redirection errors returned by getNodeByQuery(). */
+#define REDIS_CLUSTER_REDIR_NONE 0          /* Node can serve the request. */
+#define REDIS_CLUSTER_REDIR_CROSS_SLOT 1    /* Keys in different slots. */
+#define REDIS_CLUSTER_REDIR_UNSTABLE 2      /* Keys in slot resharding. */
+#define REDIS_CLUSTER_REDIR_ASK 3           /* -ASK redirection required. */
+#define REDIS_CLUSTER_REDIR_MOVED 4         /* -MOVED redirection required. */
+
 struct clusterNode;
 
 /* clusterLink encapsulates everything needed to talk with a remote node. */
@@ -148,7 +153,7 @@ typedef struct {
     char nodename[REDIS_CLUSTER_NAMELEN];
     uint32_t ping_sent;
     uint32_t pong_received;
-    char ip[16];    /* IP address last time it was seen */
+    char ip[REDIS_IP_STR_LEN];    /* IP address last time it was seen */
     uint16_t port;  /* port last time it was seen */
     uint16_t flags;
     uint32_t notused; /* for 64 bit alignment */
@@ -195,7 +200,10 @@ union clusterMsgData {
 
 
 typedef struct {
+    char sig[4];        /* Siganture "RCmb" (Redis Cluster message bus). */
     uint32_t totlen;    /* Total length of this message */
+    uint16_t ver;       /* Protocol version, currently set to 0. */
+    uint16_t notused0;  /* 2 bytes not used. */
     uint16_t type;      /* Message type */
     uint16_t count;     /* Only used for some kind of messages. */
     uint64_t currentEpoch;  /* The epoch accordingly to the sending node. */

src/config.c

@@ -1461,7 +1461,7 @@ void rewriteConfigSaveOption(struct rewriteConfigState *state) {
      * resulting into no RDB persistence as expected. */
     for (j = 0; j < server.saveparamslen; j++) {
         line = sdscatprintf(sdsempty(),"save %ld %d",
-            server.saveparams[j].seconds, server.saveparams[j].changes);
+            (long) server.saveparams[j].seconds, server.saveparams[j].changes);
         rewriteConfigRewriteLine(state,"save",line,1);
     }
     /* Mark "save" as processed in case server.saveparamslen is zero. */
@@ -1810,6 +1810,7 @@ void configCommand(redisClient *c) {
         server.stat_keyspace_hits = 0;
         server.stat_keyspace_misses = 0;
         server.stat_numcommands = 0;
+        server.ops_sec_last_sample_ops = 0;
         server.stat_numconnections = 0;
         server.stat_expiredkeys = 0;
         server.stat_rejected_conn = 0;
@@ -1824,8 +1825,10 @@ void configCommand(redisClient *c) {
             return;
         }
         if (rewriteConfig(server.configfile) == -1) {
+            redisLog(REDIS_WARNING,"CONFIG REWRITE failed: %s", strerror(errno));
             addReplyErrorFormat(c,"Rewriting config file: %s", strerror(errno));
         } else {
+            redisLog(REDIS_WARNING,"CONFIG REWRITE executed with success.");
             addReply(c,shared.ok);
         }
     } else {

src/db.c

@@ -930,6 +930,8 @@ void persistCommand(redisClient *c) {
  * API to get key arguments from commands
  * ---------------------------------------------------------------------------*/
 
+/* The base case is to use the keys position as given in the command table
+ * (firstkey, lastkey, step). */
 int *getKeysUsingCommandTable(struct redisCommand *cmd,robj **argv, int argc, int *numkeys) {
     int j, i = 0, last, *keys;
     REDIS_NOTUSED(argv);
@@ -949,42 +951,36 @@ int *getKeysUsingCommandTable(struct redisCommand *cmd,robj **argv, int argc, in
     return keys;
 }
 
-int *getKeysFromCommand(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags) {
+/* Return all the arguments that are keys in the command passed via argc / argv.
+ *
+ * The command returns the positions of all the key arguments inside the array,
+ * so the actual return value is an heap allocated array of integers. The
+ * length of the array is returned by reference into *numkeys.
+ *
+ * 'cmd' must be point to the corresponding entry into the redisCommand
+ * table, according to the command name in argv[0].
+ *
+ * This function uses the command table if a command-specific helper function
+ * is not required, otherwise it calls the command-specific function. */
+int *getKeysFromCommand(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
     if (cmd->getkeys_proc) {
-        return cmd->getkeys_proc(cmd,argv,argc,numkeys,flags);
+        return cmd->getkeys_proc(cmd,argv,argc,numkeys);
     } else {
         return getKeysUsingCommandTable(cmd,argv,argc,numkeys);
     }
 }
 
+/* Free the result of getKeysFromCommand. */
 void getKeysFreeResult(int *result) {
     zfree(result);
 }
 
-int *noPreloadGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags) {
-    if (flags & REDIS_GETKEYS_PRELOAD) {
-        *numkeys = 0;
-        return NULL;
-    } else {
-        return getKeysUsingCommandTable(cmd,argv,argc,numkeys);
-    }
-}
-
-int *renameGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags) {
-    if (flags & REDIS_GETKEYS_PRELOAD) {
-        int *keys = zmalloc(sizeof(int));
-        *numkeys = 1;
-        keys[0] = 1;
-        return keys;
-    } else {
-        return getKeysUsingCommandTable(cmd,argv,argc,numkeys);
-    }
-}
-
-int *zunionInterGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags) {
+/* Helper function to extract keys from following commands:
+ * ZUNIONSTORE <destkey> <num-keys> <key> <key> ... <key> <options>
+ * ZINTERSTORE <destkey> <num-keys> <key> <key> ... <key> <options> */
+int *zunionInterGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
     int i, num, *keys;
     REDIS_NOTUSED(cmd);
-    REDIS_NOTUSED(flags);
 
     num = atoi(argv[2]->ptr);
     /* Sanity check. Don't return any key if the command is going to
@@ -993,8 +989,89 @@ int *zunionInterGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *num
         *numkeys = 0;
         return NULL;
     }
-    keys = zmalloc(sizeof(int)*num);
+
+    /* Keys in z{union,inter}store come from two places:
+     * argv[1] = storage key,
+     * argv[3...n] = keys to intersect */
+    keys = zmalloc(sizeof(int)*(num+1));
+
+    /* Add all key positions for argv[3...n] to keys[] */
     for (i = 0; i < num; i++) keys[i] = 3+i;
+
+    /* Finally add the argv[1] key position (the storage key target). */
+    keys[num] = 1;
+    *numkeys = num+1;  /* Total keys = {union,inter} keys + storage key */
+    return keys;
+}
+
+/* Helper function to extract keys from the following commands:
+ * EVAL <script> <num-keys> <key> <key> ... <key> [more stuff]
+ * EVALSHA <script> <num-keys> <key> <key> ... <key> [more stuff] */
+int *evalGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
+    int i, num, *keys;
+    REDIS_NOTUSED(cmd);
+
+    num = atoi(argv[2]->ptr);
+    /* Sanity check. Don't return any key if the command is going to
+     * reply with syntax error. */
+    if (num > (argc-3)) {
+        *numkeys = 0;
+        return NULL;
+    }
+
+    keys = zmalloc(sizeof(int)*num);
+    *numkeys = num;
+
+    /* Add all key positions for argv[3...n] to keys[] */
+    for (i = 0; i < num; i++) keys[i] = 3+i;
+
+    return keys;
+}
+
+/* Helper function to extract keys from the SORT command.
+ *
+ * SORT <sort-key> ... STORE <store-key> ...
+ *
+ * The first argument of SORT is always a key, however a list of options
+ * follow in SQL-alike style. Here we parse just the minimum in order to
+ * correctly identify keys in the "STORE" option. */
+int *sortGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
+    int i, j, num, *keys;
+    REDIS_NOTUSED(cmd);
+
+    num = 0;
+    keys = zmalloc(sizeof(int)*2); /* Alloc 2 places for the worst case. */
+
+    keys[num++] = 1; /* <sort-key> is always present. */
+
+    /* Search for STORE option. By default we consider options to don't
+     * have arguments, so if we find an unknown option name we scan the
+     * next. However there are options with 1 or 2 arguments, so we
+     * provide a list here in order to skip the right number of args. */
+    struct {
+        char *name;
+        int skip;
+    } skiplist[] = {
+        {"limit", 2},
+        {"get", 1},
+        {"by", 1},
+        {NULL, 0} /* End of elements. */
+    };
+
+    for (i = 2; i < argc; i++) {
+        for (j = 0; skiplist[j].name != NULL; j++) {
+            if (!strcasecmp(argv[i]->ptr,skiplist[j].name)) {
+                i += skiplist[j].skip;
+                break;
+            } else if (!strcasecmp(argv[i]->ptr,"store") && i+1 < argc) {
+                /* Note: we don't increment "num" here and continue the loop
+                 * to be sure to process the *last* "STORE" option if multiple
+                 * ones are provided. This is same behavior as SORT. */
+                keys[num] = i+1; /* <store-key> */
+                break;
+            }
+        }
+    }
     *numkeys = num;
     return keys;
 }

src/debug.c

@@ -363,6 +363,31 @@ void debugCommand(redisClient *c) {
     {
         server.active_expire_enabled = atoi(c->argv[2]->ptr);
         addReply(c,shared.ok);
+    } else if (!strcasecmp(c->argv[1]->ptr,"cmdkeys") && c->argc >= 3) {
+        struct redisCommand *cmd = lookupCommand(c->argv[2]->ptr);
+        int *keys, numkeys, j;
+
+        if (!cmd) {
+            addReplyError(c,"Invalid command specified");
+            return;
+        } else if ((cmd->arity > 0 && cmd->arity != c->argc-2) ||
+                   ((c->argc-2) < -cmd->arity))
+        {
+            addReplyError(c,"Invalid number of arguments specified for command");
+            return;
+        }
+
+        keys = getKeysFromCommand(cmd,c->argv+2,c->argc-2,&numkeys);
+        addReplyMultiBulkLen(c,numkeys);
+        for (j = 0; j < numkeys; j++) addReplyBulk(c,c->argv[keys[j]+2]);
+        getKeysFreeResult(keys);
+    } else if (!strcasecmp(c->argv[1]->ptr,"error") && c->argc == 3) {
+        sds errstr = sdsnewlen("-",1);
+
+        errstr = sdscatsds(errstr,c->argv[2]->ptr);
+        errstr = sdsmapchars(errstr,"\n\r","  ",2); /* no newlines in errors. */
+        errstr = sdscatlen(errstr,"\r\n",2);
+        addReplySds(c,errstr);
     } else {
         addReplyErrorFormat(c, "Unknown DEBUG subcommand or wrong number of arguments for '%s'",
             (char*)c->argv[1]->ptr);

src/dict.c

@@ -239,7 +239,7 @@ int dictExpand(dict *d, unsigned long size)
 /* Performs N steps of incremental rehashing. Returns 1 if there are still
  * keys to move from the old to the new hash table, otherwise 0 is returned.
  * Note that a rehashing step consists in moving a bucket (that may have more
- * thank one key as we use chaining) from the old to the new hash table. */
+ * than one key as we use chaining) from the old to the new hash table. */
 int dictRehash(dict *d, int n) {
     if (!dictIsRehashing(d)) return 0;
 
@@ -695,7 +695,7 @@ static unsigned long rev(unsigned long v) {
  * (where SIZE-1 is always the mask that is equivalent to taking the rest
  *  of the division between the Hash of the key and SIZE).
  *
- * For example if the current hash table size is 64, the mask is
+ * For example if the current hash table size is 16, the mask is
  * (in binary) 1111. The position of a key in the hash table will be always
  * the last four bits of the hash output, and so forth.
  *

src/networking.c

@@ -587,7 +587,7 @@ void acceptTcpHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
     REDIS_NOTUSED(privdata);
 
     cfd = anetTcpAccept(server.neterr, fd, cip, sizeof(cip), &cport);
-    if (cfd == AE_ERR) {
+    if (cfd == ANET_ERR) {
         redisLog(REDIS_WARNING,"Accepting client connection: %s", server.neterr);
         return;
     }

src/pubsub.c

@@ -306,8 +306,10 @@ void punsubscribeCommand(redisClient *c) {
 
 void publishCommand(redisClient *c) {
     int receivers = pubsubPublishMessage(c->argv[1],c->argv[2]);
-    if (server.cluster_enabled) clusterPropagatePublish(c->argv[1],c->argv[2]);
-    forceCommandPropagation(c,REDIS_PROPAGATE_REPL);
+    if (server.cluster_enabled)
+        clusterPropagatePublish(c->argv[1],c->argv[2]);
+    else
+        forceCommandPropagation(c,REDIS_PROPAGATE_REPL);
     addReplyLongLong(c,receivers);
 }
 

src/rdb.c

@@ -1065,6 +1065,10 @@ void rdbLoadProgressCallback(rio *r, const void *buf, size_t len) {
     if (server.loading_process_events_interval_bytes &&
         (r->processed_bytes + len)/server.loading_process_events_interval_bytes > r->processed_bytes/server.loading_process_events_interval_bytes)
     {
+        /* The DB can take some non trivial amount of time to load. Update
+         * our cached time since it is used to create and update the last
+         * interaction time with clients and for other important things. */
+        updateCachedTime();
         if (server.masterhost && server.repl_state == REDIS_REPL_TRANSFER)
             replicationSendNewlineToMaster();
         loadingProgress(r->processed_bytes);

src/redis-cli.c

@@ -58,7 +58,7 @@
 #define OUTPUT_RAW 1
 #define OUTPUT_CSV 2
 #define REDIS_CLI_KEEPALIVE_INTERVAL 15 /* seconds */
-#define REDIS_DEFAULT_PIPE_TIMEOUT 30 /* seconds */
+#define REDIS_CLI_DEFAULT_PIPE_TIMEOUT 30 /* seconds */
 
 static redisContext *context;
 static struct config {
@@ -82,6 +82,8 @@ static struct config {
     int getrdb_mode;
     int stat_mode;
     int scan_mode;
+    int intrinsic_latency_mode;
+    int intrinsic_latency_duration;
     char *pattern;
     char *rdb_filename;
     int bigkeys;
@@ -94,6 +96,7 @@ static struct config {
 } config;
 
 static void usage();
+static void slaveMode(void);
 char *redisGitSHA1(void);
 char *redisGitDirty(void);
 
@@ -101,14 +104,18 @@ char *redisGitDirty(void);
  * Utility functions
  *--------------------------------------------------------------------------- */
 
-static long long mstime(void) {
+static long long ustime(void) {
     struct timeval tv;
-    long long mst;
+    long long ust;
 
     gettimeofday(&tv, NULL);
-    mst = ((long long)tv.tv_sec)*1000;
-    mst += tv.tv_usec/1000;
-    return mst;
+    ust = ((long long)tv.tv_sec)*1000000;
+    ust += tv.tv_usec;
+    return ust;
+}
+
+static long long mstime(void) {
+    return ustime()/1000;
 }
 
 static void cliRefreshPrompt(void) {
@@ -600,6 +607,8 @@ static int cliSendCommand(int argc, char **argv, int repeat) {
     if (!strcasecmp(command,"monitor")) config.monitor_mode = 1;
     if (!strcasecmp(command,"subscribe") ||
         !strcasecmp(command,"psubscribe")) config.pubsub_mode = 1;
+    if (!strcasecmp(command,"sync") ||
+        !strcasecmp(command,"psync")) config.slave_mode = 1;
 
     /* Setup argument length */
     argvlen = malloc(argc*sizeof(size_t));
@@ -621,6 +630,13 @@ static int cliSendCommand(int argc, char **argv, int repeat) {
             }
         }
 
+        if (config.slave_mode) {
+            printf("Entering slave output mode...  (press Ctrl-C to quit)\n");
+            slaveMode();
+            config.slave_mode = 0;
+            return REDIS_ERR;  /* Error = slaveMode lost connection to master */
+        }
+
         if (cliReadReply(output_raw) != REDIS_OK) {
             free(argvlen);
             return REDIS_ERR;
@@ -716,8 +732,11 @@ static int parseOptions(int argc, char **argv) {
             config.stat_mode = 1;
         } else if (!strcmp(argv[i],"--scan")) {
             config.scan_mode = 1;
-        } else if (!strcmp(argv[i],"--pattern")) {
+        } else if (!strcmp(argv[i],"--pattern") && !lastarg) {
             config.pattern = argv[++i];
+        } else if (!strcmp(argv[i],"--intrinsic-latency") && !lastarg) {
+            config.intrinsic_latency_mode = 1;
+            config.intrinsic_latency_duration = atoi(argv[++i]);
         } else if (!strcmp(argv[i],"--rdb") && !lastarg) {
             config.getrdb_mode = 1;
             config.rdb_filename = argv[++i];
@@ -803,6 +822,8 @@ static void usage() {
 "  --bigkeys          Sample Redis keys looking for big keys.\n"
 "  --scan             List all keys using the SCAN command.\n"
 "  --pattern <pat>    Useful with --scan to specify a SCAN pattern.\n"
+"  --intrinsic-latency <sec> Run a test to measure intrinsic system latency.\n"
+"                     The test will run for the specified amount of seconds.\n"
 "  --eval <file>      Send an EVAL command using the Lua script at <file>.\n"
 "  --help             Output this help and exit.\n"
 "  --version          Output version and exit.\n"
@@ -820,7 +841,7 @@ static void usage() {
 "When no command is given, redis-cli starts in interactive mode.\n"
 "Type \"help\" in interactive mode for information on available commands.\n"
 "\n",
-        version, REDIS_DEFAULT_PIPE_TIMEOUT);
+        version, REDIS_CLI_DEFAULT_PIPE_TIMEOUT);
     sdsfree(version);
     exit(1);
 }
@@ -940,6 +961,10 @@ static int noninteractive(int argc, char **argv) {
     return retval;
 }
 
+/*------------------------------------------------------------------------------
+ * Eval mode
+ *--------------------------------------------------------------------------- */
+
 static int evalMode(int argc, char **argv) {
     sds script = sdsempty();
     FILE *fp;
@@ -978,6 +1003,10 @@ static int evalMode(int argc, char **argv) {
     return cliSendCommand(argc+3-got_comma, argv2, config.repeat);
 }
 
+/*------------------------------------------------------------------------------
+ * Latency and latency history modes
+ *--------------------------------------------------------------------------- */
+
 #define LATENCY_SAMPLE_RATE 10 /* milliseconds. */
 #define LATENCY_HISTORY_DEFAULT_INTERVAL 15000 /* milliseconds. */
 static void latencyMode(void) {
@@ -1022,6 +1051,10 @@ static void latencyMode(void) {
     }
 }
 
+/*------------------------------------------------------------------------------
+ * Slave mode
+ *--------------------------------------------------------------------------- */
+
 /* Sends SYNC and reads the number of bytes in the payload. Used both by
  * slaveMode() and getRDB(). */
 unsigned long long sendSync(int fd) {
@@ -1061,6 +1094,7 @@ static void slaveMode(void) {
     int fd = context->fd;
     unsigned long long payload = sendSync(fd);
     char buf[1024];
+    int original_output = config.output;
 
     fprintf(stderr,"SYNC with master, discarding %llu "
                    "bytes of bulk transfer...\n", payload);
@@ -1081,8 +1115,13 @@ static void slaveMode(void) {
     /* Now we can use hiredis to read the incoming protocol. */
     config.output = OUTPUT_CSV;
     while (cliReadReply(0) == REDIS_OK);
+    config.output = original_output;
 }
 
+/*------------------------------------------------------------------------------
+ * RDB transfer mode
+ *--------------------------------------------------------------------------- */
+
 /* This function implements --rdb, so it uses the replication protocol in order
  * to fetch the RDB file from a remote server. */
 static void getRDB(void) {
@@ -1128,6 +1167,10 @@ static void getRDB(void) {
     exit(0);
 }
 
+/*------------------------------------------------------------------------------
+ * Bulk import (pipe) mode
+ *--------------------------------------------------------------------------- */
+
 static void pipeMode(void) {
     int fd = context->fd;
     long long errors = 0, replies = 0, obuf_len = 0, obuf_pos = 0;
@@ -1279,92 +1322,287 @@ static void pipeMode(void) {
         exit(0);
 }
 
+/*------------------------------------------------------------------------------
+ * Find big keys
+ *--------------------------------------------------------------------------- */
+
 #define TYPE_STRING 0
 #define TYPE_LIST   1
 #define TYPE_SET    2
 #define TYPE_HASH   3
 #define TYPE_ZSET   4
+#define TYPE_NONE   5
+
+static redisReply *sendScan(unsigned long long *it) {
+    redisReply *reply = redisCommand(context, "SCAN %llu", *it);
+
+    /* Handle any error conditions */
+    if(reply == NULL) {
+        fprintf(stderr, "\nI/O error\n");
+        exit(1);
+    } else if(reply->type == REDIS_REPLY_ERROR) {
+        fprintf(stderr, "SCAN error: %s\n", reply->str);
+        exit(1);
+    } else if(reply->type != REDIS_REPLY_ARRAY) {
+        fprintf(stderr, "Non ARRAY response from SCAN!\n");
+        exit(1);
+    } else if(reply->elements != 2) {
+        fprintf(stderr, "Invalid element count from SCAN!\n");
+        exit(1);
+    }
+
+    /* Validate our types are correct */
+    assert(reply->element[0]->type == REDIS_REPLY_STRING);
+    assert(reply->element[1]->type == REDIS_REPLY_ARRAY);
+    
+    /* Update iterator */
+    *it = atoi(reply->element[0]->str);
+
+    return reply;
+}
+
+static int getDbSize(void) {
+    redisReply *reply;
+    int size;
+
+    reply = redisCommand(context, "DBSIZE");
+    
+    if(reply == NULL || reply->type != REDIS_REPLY_INTEGER) {
+        fprintf(stderr, "Couldn't determine DBSIZE!\n");
+        exit(1);
+    }
+
+    /* Grab the number of keys and free our reply */
+    size = reply->integer;
+    freeReplyObject(reply);
+
+    return size;
+}
+
+static int toIntType(char *key, char *type) {
+    if(!strcmp(type, "string")) {
+        return TYPE_STRING;
+    } else if(!strcmp(type, "list")) {
+        return TYPE_LIST;
+    } else if(!strcmp(type, "set")) {
+        return TYPE_SET;
+    } else if(!strcmp(type, "hash")) {
+        return TYPE_HASH;
+    } else if(!strcmp(type, "zset")) {
+        return TYPE_ZSET;
+    } else if(!strcmp(type, "none")) {
+        return TYPE_NONE;
+    } else {
+        fprintf(stderr, "Unknown type '%s' for key '%s'\n", type, key);
+        exit(1);
+    }
+}
+
+static void getKeyTypes(redisReply *keys, int *types) {
+    redisReply *reply;
+    int i;
+
+    /* Pipeline TYPE commands */
+    for(i=0;i<keys->elements;i++) {
+        redisAppendCommand(context, "TYPE %s", keys->element[i]->str);
+    }
+
+    /* Retrieve types */
+    for(i=0;i<keys->elements;i++) {
+        if(redisGetReply(context, (void**)&reply)!=REDIS_OK) {
+            fprintf(stderr, "Error getting type for key '%s' (%d: %s)\n",
+                keys->element[i]->str, context->err, context->errstr);
+            exit(1);
+        } else if(reply->type != REDIS_REPLY_STATUS) {
+            fprintf(stderr, "Invalid reply type (%d) for TYPE on key '%s'!\n",
+                reply->type, keys->element[i]->str);
+            exit(1);
+        }
+
+        types[i] = toIntType(keys->element[i]->str, reply->str); 
+        freeReplyObject(reply);
+    }
+}
+
+static void getKeySizes(redisReply *keys, int *types, 
+                        unsigned long long *sizes) 
+{
+    redisReply *reply;
+    char *sizecmds[] = {"STRLEN","LLEN","SCARD","HLEN","ZCARD"};
+    int i;
+
+    /* Pipeline size commands */
+    for(i=0;i<keys->elements;i++) {
+        /* Skip keys that were deleted */
+        if(types[i]==TYPE_NONE) 
+            continue;
+
+        redisAppendCommand(context, "%s %s", sizecmds[types[i]], 
+            keys->element[i]->str);
+    }
+
+    /* Retreive sizes */
+    for(i=0;i<keys->elements;i++) {
+        /* Skip keys that dissapeared between SCAN and TYPE */
+        if(types[i] == TYPE_NONE) {
+            sizes[i] = 0;
+            continue;
+        }
+
+        /* Retreive size */
+        if(redisGetReply(context, (void**)&reply)!=REDIS_OK) {
+            fprintf(stderr, "Error getting size for key '%s' (%d: %s)\n",
+                keys->element[i]->str, context->err, context->errstr);
+            exit(1);
+        } else if(reply->type != REDIS_REPLY_INTEGER) {
+            /* Theoretically the key could have been removed and
+             * added as a different type between TYPE and SIZE */
+            fprintf(stderr, 
+                "Warning:  %s on '%s' failed (may have changed type)\n",
+                 sizecmds[types[i]], keys->element[i]->str);
+            sizes[i] = 0;
+        } else {
+            sizes[i] = reply->integer;
+        }
+            
+        freeReplyObject(reply);
+    }
+}
 
 static void findBigKeys(void) {
-    unsigned long long biggest[5] = {0,0,0,0,0};
-    unsigned long long samples = 0;
-    redisReply *reply1, *reply2, *reply3 = NULL;
-    char *sizecmd, *typename[] = {"string","list","set","hash","zset"};
+    unsigned long long biggest[5] = {0}, counts[5] = {0}, totalsize[5] = {0};
+    unsigned long long sampled = 0, total_keys, totlen=0, *sizes=NULL, it=0;
+    sds maxkeys[5] = {0};
+    char *typename[] = {"string","list","set","hash","zset"};
     char *typeunit[] = {"bytes","items","members","fields","members"};
-    int type;
+    redisReply *reply, *keys;
+    int type, *types=NULL, arrsize=0, i;
+    double pct;
 
-    printf("\n# Press ctrl+c when you have had enough of it... :)\n");
-    printf("# You can use -i 0.1 to sleep 0.1 sec every 100 sampled keys\n");
-    printf("# in order to reduce server load (usually not needed).\n\n");
-    while(1) {
-        /* Sample with RANDOMKEY */
-        reply1 = redisCommand(context,"RANDOMKEY");
-        if (reply1 == NULL) {
-            fprintf(stderr,"\nI/O error\n");
-            exit(1);
-        } else if (reply1->type == REDIS_REPLY_ERROR) {
-            fprintf(stderr, "RANDOMKEY error: %s\n",
-                reply1->str);
-            exit(1);
-        } else if (reply1->type == REDIS_REPLY_NIL) {
-            fprintf(stderr, "It looks like the database is empty!\n");
+    /* Total keys pre scanning */
+    total_keys = getDbSize();
+
+    /* Status message */
+    printf("\n# Scanning the entire keyspace to find biggest keys as well as\n");
+    printf("# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec\n");
+    printf("# per 100 SCAN commands (not usually needed).\n\n");
+
+    /* New up sds strings to keep track of overall biggest per type */
+    for(i=0;i<TYPE_NONE; i++) {
+        maxkeys[i] = sdsempty();
+        if(!maxkeys[i]) {
+            fprintf(stderr, "Failed to allocate memory for largest key names!");
             exit(1);
         }
+    }
 
-        /* Get the key type */
-        reply2 = redisCommand(context,"TYPE %s",reply1->str);
-        assert(reply2 && reply2->type == REDIS_REPLY_STATUS);
-        samples++;
+    /* SCAN loop */
+    do {
+        /* Calculate approximate percentage completion */
+        pct = 100 * (double)sampled/total_keys;
 
-        /* Get the key "size" */
-        if (!strcmp(reply2->str,"string")) {
-            sizecmd = "STRLEN";
-            type = TYPE_STRING;
-        } else if (!strcmp(reply2->str,"list")) {
-            sizecmd = "LLEN";
-            type = TYPE_LIST;
-        } else if (!strcmp(reply2->str,"set")) {
-            sizecmd = "SCARD";
-            type = TYPE_SET;
-        } else if (!strcmp(reply2->str,"hash")) {
-            sizecmd = "HLEN";
-            type = TYPE_HASH;
-        } else if (!strcmp(reply2->str,"zset")) {
-            sizecmd = "ZCARD";
-            type = TYPE_ZSET;
-        } else if (!strcmp(reply2->str,"none")) {
-            freeReplyObject(reply1);
-            freeReplyObject(reply2);
-            continue;
-        } else {
-            fprintf(stderr, "Unknown key type '%s' for key '%s'\n",
-                reply2->str, reply1->str);
-            exit(1);
+        /* Grab some keys and point to the keys array */
+        reply = sendScan(&it);
+        keys  = reply->element[1];
+
+        /* Reallocate our type and size array if we need to */
+        if(keys->elements > arrsize) {
+            types = zrealloc(types, sizeof(int)*keys->elements);
+            sizes = zrealloc(sizes, sizeof(unsigned long long)*keys->elements);
+
+            if(!types || !sizes) {
+                fprintf(stderr, "Failed to allocate storage for keys!\n");
+                exit(1);
+            }
+
+            arrsize = keys->elements;
         }
 
-        reply3 = redisCommand(context,"%s %s", sizecmd, reply1->str);
-        if (reply3 && reply3->type == REDIS_REPLY_INTEGER) {
-            if (biggest[type] < reply3->integer) {
-                printf("Biggest %-6s found so far '%s' with %llu %s.\n",
-                    typename[type], reply1->str,
-                    (unsigned long long) reply3->integer,
-                    typeunit[type]);
-                biggest[type] = reply3->integer;
+        /* Retreive types and then sizes */
+        getKeyTypes(keys, types);
+        getKeySizes(keys, types, sizes);
+        
+        /* Now update our stats */
+        for(i=0;i<keys->elements;i++) {
+            if((type = types[i]) == TYPE_NONE)
+                continue;
+            
+            totalsize[type] += sizes[i];
+            counts[type]++;
+            totlen += keys->element[i]->len;
+            sampled++;
+
+            if(biggest[type]<sizes[i]) {
+                printf(
+                   "[%05.2f%%] Biggest %-6s found so far '%s' with %llu %s\n",
+                   pct, typename[type], keys->element[i]->str, sizes[i],
+                   typeunit[type]);
+
+                /* Keep track of biggest key name for this type */
+                maxkeys[type] = sdscpy(maxkeys[type], keys->element[i]->str);
+                if(!maxkeys[type]) {
+                    fprintf(stderr, "Failed to allocate memory for key!\n");
+                    exit(1);
+                }
+
+                /* Keep track of the biggest size for this type */
+                biggest[type] = sizes[i];                
+            }
+
+            /* Update overall progress */
+            if(sampled % 1000000 == 0) {
+                printf("[%05.2f%%] Sampled %llu keys so far\n", pct, sampled);
             }
         }
 
-        if ((samples % 1000000) == 0)
-            printf("(%llu keys sampled)\n", samples);
-
-        if ((samples % 100) == 0 && config.interval)
+        /* Sleep if we've been directed to do so */
+        if(sampled && (sampled %100) == 0 && config.interval) {
             usleep(config.interval);
+        }
+        
+        freeReplyObject(reply);
+    } while(it != 0);
 
-        freeReplyObject(reply1);
-        freeReplyObject(reply2);
-        if (reply3) freeReplyObject(reply3);
+    if(types) zfree(types);
+    if(sizes) zfree(sizes);
+
+    /* We're done */
+    printf("\n-------- summary -------\n\n");
+
+    printf("Sampled %llu keys in the keyspace!\n", sampled);
+    printf("Total key length in bytes is %llu (avg len %.2f)\n\n",
+       totlen, totlen ? (double)totlen/sampled : 0);
+
+    /* Output the biggest keys we found, for types we did find */
+    for(i=0;i<TYPE_NONE;i++) {
+        if(sdslen(maxkeys[i])>0) {
+            printf("Biggest %6s found '%s' has %llu %s\n", typename[i], maxkeys[i],
+               biggest[i], typeunit[i]);
+        }
     }
+
+    printf("\n");
+
+    for(i=0;i<TYPE_NONE;i++) {
+        printf("%llu %ss with %llu %s (%05.2f%% of keys, avg size %.2f)\n",
+           counts[i], typename[i], totalsize[i], typeunit[i],
+           sampled ? 100 * (double)counts[i]/sampled : 0,
+           counts[i] ? (double)totalsize[i]/counts[i] : 0);
+    }
+
+    /* Free sds strings containing max keys */
+    for(i=0;i<TYPE_NONE;i++) {
+        sdsfree(maxkeys[i]);
+    }
+
+    /* Success! */
+    exit(0);
 }
 
+/*------------------------------------------------------------------------------
+ * Stats mode
+ *--------------------------------------------------------------------------- */
+
 /* Return the specified INFO field from the INFO command output "info".
  * A new buffer is allocated for the result, that needs to be free'd.
  * If the field is not found NULL is returned. */
@@ -1504,6 +1742,10 @@ static void statMode() {
     }
 }
 
+/*------------------------------------------------------------------------------
+ * Scan mode
+ *--------------------------------------------------------------------------- */
+
 static void scanMode() {
     redisReply *reply;
     unsigned long long cur = 0;
@@ -1533,6 +1775,73 @@ static void scanMode() {
     exit(0);
 }
 
+/*------------------------------------------------------------------------------
+ * Intrisic latency mode.
+ *
+ * Measure max latency of a running process that does not result from
+ * syscalls. Basically this software should provide an hint about how much
+ * time the kernel leaves the process without a chance to run.
+ *--------------------------------------------------------------------------- */
+
+/* This is just some computation the compiler can't optimize out.
+ * Should run in less than 100-200 microseconds even using very
+ * slow hardware. Runs in less than 10 microseconds in modern HW. */
+unsigned long compute_something_fast(void) {
+    unsigned char s[256], i, j, t;
+    int count = 1000, k;
+    unsigned long output = 0;
+
+    for (k = 0; k < 256; k++) s[k] = k;
+
+    i = 0;
+    j = 0;
+    while(count--) {
+        i++;
+        j = j + s[i];
+        t = s[i];
+        s[i] = s[j];
+        s[j] = t;
+        output += s[(s[i]+s[j])&255];
+    }
+    return output;
+}
+
+static void intrinsicLatencyMode(void) {
+    long long test_end, run_time, max_latency = 0, runs = 0;
+
+    run_time = config.intrinsic_latency_duration*1000000;
+    test_end = ustime() + run_time;
+
+    while(1) {
+        long long start, end, latency;
+
+        start = ustime();
+        compute_something_fast();
+        end = ustime();
+        latency = end-start;
+        runs++;
+        if (latency <= 0) continue;
+
+        /* Reporting */
+        if (latency > max_latency) {
+            max_latency = latency;
+            printf("Max latency so far: %lld microseconds.\n", max_latency);
+        }
+
+        if (end > test_end) {
+            printf("\n%lld total runs (avg %lld microseconds per run).\n",
+                runs, run_time/runs);
+            printf("Worst run took %.02fx times the avarege.\n",
+                (double) max_latency / (run_time/runs));
+            exit(0);
+        }
+    }
+}
+
+/*------------------------------------------------------------------------------
+ * Program main()
+ *--------------------------------------------------------------------------- */
+
 int main(int argc, char **argv) {
     int firstarg;
 
@@ -1553,10 +1862,11 @@ int main(int argc, char **argv) {
     config.getrdb_mode = 0;
     config.stat_mode = 0;
     config.scan_mode = 0;
+    config.intrinsic_latency_mode = 0;
     config.pattern = NULL;
     config.rdb_filename = NULL;
     config.pipe_mode = 0;
-    config.pipe_timeout = REDIS_DEFAULT_PIPE_TIMEOUT;
+    config.pipe_timeout = REDIS_CLI_DEFAULT_PIPE_TIMEOUT;
     config.bigkeys = 0;
     config.stdinarg = 0;
     config.auth = NULL;
@@ -1615,6 +1925,9 @@ int main(int argc, char **argv) {
         scanMode();
     }
 
+    /* Intrinsic latency mode */
+    if (config.intrinsic_latency_mode) intrinsicLatencyMode();
+
     /* Start interactive mode when no command is provided */
     if (argc == 0 && !config.eval) {
         /* Note that in repl mode we don't abort on connection error.

src/redis-trib.rb

@@ -359,11 +359,11 @@ class RedisTrib
         @nodes.each{|n|
             if n.info[:migrating].size > 0
                 cluster_error \
-                    "[WARNING] Node #{n} has slots in migrating state."
+                    "[WARNING] Node #{n} has slots in migrating state (#{n.info[:migrating].keys.join(",")})."
                 open_slots += n.info[:migrating].keys
             elsif n.info[:importing].size > 0
                 cluster_error \
-                    "[WARNING] Node #{n} has slots in importing state."
+                    "[WARNING] Node #{n} has slots in importing state (#{n.info[:importing].keys.join(",")})."
                 open_slots += n.info[:importing].keys
             end
         }
@@ -469,6 +469,12 @@ class RedisTrib
         #         importing state in 1 slot. That's trivial to address.
         if migrating.length == 1 && importing.length == 1
             move_slot(migrating[0],importing[0],slot,:verbose=>true)
+        elsif migrating.length == 1 && importing.length == 0
+            xputs ">>> Setting #{slot} as STABLE"
+            migrating[0].r.cluster("setslot",slot,"stable")
+        elsif migrating.length == 0 && importing.length == 1
+            xputs ">>> Setting #{slot} as STABLE"
+            importing[0].r.cluster("setslot",slot,"stable")
         else
             xputs "[ERR] Sorry, Redis-trib can't fix this slot yet (work in progress)"
         end
@@ -547,22 +553,28 @@ class RedisTrib
         # Select N replicas for every master.
         # We try to split the replicas among all the IPs with spare nodes
         # trying to avoid the host where the master is running, if possible.
-        masters.each{|m|
-            i = 0
-            while i < @replicas
-                ips.each{|ip,nodes_list|
-                    next if nodes_list.length == 0
-                    # Skip instances with the same IP as the master if we
-                    # have some more IPs available.
-                    next if ip == m.info[:host] && nodes_count > nodes_list.length
-                    slave = nodes_list.shift
-                    slave.set_as_replica(m.info[:name])
-                    nodes_count -= 1
-                    i += 1
-                    puts "#{m} replica ##{i} is #{slave}"
-                    break if masters.length == masters_count
-                }
-            end
+        #
+        # Note that we loop two times, the first with spare_loop set to false,
+        # the second with the var set to true. The second loop changes the
+        # while condition so to assign remaining slaves.
+        [false,true].each{|spare_loop|
+            masters.each{|m|
+                i = 0
+                while (!spare_loop && i < @replicas) || \
+                      (spare_loop && nodes_count > 0)
+                    ips.each{|ip,nodes_list|
+                        next if nodes_list.length == 0
+                        # Skip instances with the same IP as the master if we
+                        # have some more IPs available.
+                        next if ip == m.info[:host] && nodes_count > nodes_list.length
+                        slave = nodes_list.shift
+                        slave.set_as_replica(m.info[:name])
+                        nodes_count -= 1
+                        i += 1
+                        puts "Adding replica #{slave} to #{m}"
+                    }
+                end
+            }
         }
     end
 
@@ -898,10 +910,10 @@ class RedisTrib
         xputs ">>> Sending CLUSTER FORGET messages to the cluster..."
         @nodes.each{|n|
             next if n == node
-            if n.info[:replicate] && n.info[:replicate].downcase == node_id
+            if n.info[:replicate] && n.info[:replicate].downcase == id
                 # Reconfigure the slave to replicate with some other node
-                xputs ">>> #{n} as replica of #{master}"
                 master = get_master_with_least_replicas
+                xputs ">>> #{n} as replica of #{master}"
                 n.r.cluster("replicate",master.info[:name])
             end
             n.r.cluster("forget",argv[1])
@@ -940,6 +952,23 @@ class RedisTrib
         xputs ">>> New node timeout set. #{ok_count} OK, #{err_count} ERR."
     end
 
+    def call_cluster_cmd(argv,opt)
+        cmd = argv[1..-1]
+        cmd[0] = cmd[0].upcase
+
+        # Load cluster information
+        load_cluster_info_from_node(argv[0])
+        xputs ">>> Calling #{cmd.join(" ")}"
+        @nodes.each{|n|
+            begin
+                res = n.r.send(*cmd)
+                puts "#{n}: #{res}"
+            rescue => e
+                puts "#{n}: #{e}"
+            end
+        }
+    end
+
     def help_cluster_cmd(argv,opt)
         show_help
         exit 0
@@ -983,6 +1012,7 @@ COMMANDS={
     "add-node" => ["addnode_cluster_cmd", 3, "new_host:new_port existing_host:existing_port"],
     "del-node" => ["delnode_cluster_cmd", 3, "host:port node_id"],
     "set-timeout" => ["set_timeout_cluster_cmd", 3, "host:port milliseconds"],
+    "call" =>    ["call_cluster_cmd", -3, "host:port command arg arg .. arg"],
     "help"    => ["help_cluster_cmd", 1, "(show this help)"]
 }
 

src/redis.c

@@ -118,13 +118,13 @@ struct redisCommand *commandTable;
  */
 struct redisCommand redisCommandTable[] = {
     {"get",getCommand,2,"r",0,NULL,1,1,1,0,0},
-    {"set",setCommand,-3,"wm",0,noPreloadGetKeys,1,1,1,0,0},
-    {"setnx",setnxCommand,3,"wm",0,noPreloadGetKeys,1,1,1,0,0},
-    {"setex",setexCommand,4,"wm",0,noPreloadGetKeys,1,1,1,0,0},
-    {"psetex",psetexCommand,4,"wm",0,noPreloadGetKeys,1,1,1,0,0},
+    {"set",setCommand,-3,"wm",0,NULL,1,1,1,0,0},
+    {"setnx",setnxCommand,3,"wm",0,NULL,1,1,1,0,0},
+    {"setex",setexCommand,4,"wm",0,NULL,1,1,1,0,0},
+    {"psetex",psetexCommand,4,"wm",0,NULL,1,1,1,0,0},
     {"append",appendCommand,3,"wm",0,NULL,1,1,1,0,0},
     {"strlen",strlenCommand,2,"r",0,NULL,1,1,1,0,0},
-    {"del",delCommand,-2,"w",0,noPreloadGetKeys,1,-1,1,0,0},
+    {"del",delCommand,-2,"w",0,NULL,1,-1,1,0,0},
     {"exists",existsCommand,2,"r",0,NULL,1,1,1,0,0},
     {"setbit",setbitCommand,4,"wm",0,NULL,1,1,1,0,0},
     {"getbit",getbitCommand,3,"r",0,NULL,1,1,1,0,0},
@@ -206,8 +206,8 @@ struct redisCommand redisCommandTable[] = {
     {"randomkey",randomkeyCommand,1,"rR",0,NULL,0,0,0,0,0},
     {"select",selectCommand,2,"rl",0,NULL,0,0,0,0,0},
     {"move",moveCommand,3,"w",0,NULL,1,1,1,0,0},
-    {"rename",renameCommand,3,"w",0,renameGetKeys,1,2,1,0,0},
-    {"renamenx",renamenxCommand,3,"w",0,renameGetKeys,1,2,1,0,0},
+    {"rename",renameCommand,3,"w",0,NULL,1,2,1,0,0},
+    {"renamenx",renamenxCommand,3,"w",0,NULL,1,2,1,0,0},
     {"expire",expireCommand,3,"w",0,NULL,1,1,1,0,0},
     {"expireat",expireatCommand,3,"w",0,NULL,1,1,1,0,0},
     {"pexpire",pexpireCommand,3,"w",0,NULL,1,1,1,0,0},
@@ -232,7 +232,7 @@ struct redisCommand redisCommandTable[] = {
     {"replconf",replconfCommand,-1,"arslt",0,NULL,0,0,0,0,0},
     {"flushdb",flushdbCommand,1,"w",0,NULL,0,0,0,0,0},
     {"flushall",flushallCommand,1,"w",0,NULL,0,0,0,0,0},
-    {"sort",sortCommand,-2,"wm",0,NULL,1,1,1,0,0},
+    {"sort",sortCommand,-2,"wm",0,sortGetKeys,1,1,1,0,0},
     {"info",infoCommand,-1,"rlt",0,NULL,0,0,0,0,0},
     {"monitor",monitorCommand,1,"ars",0,NULL,0,0,0,0,0},
     {"ttl",ttlCommand,2,"r",0,NULL,1,1,1,0,0},
@@ -247,7 +247,7 @@ struct redisCommand redisCommandTable[] = {
     {"punsubscribe",punsubscribeCommand,-1,"rpslt",0,NULL,0,0,0,0,0},
     {"publish",publishCommand,3,"pltr",0,NULL,0,0,0,0,0},
     {"pubsub",pubsubCommand,-2,"pltrR",0,NULL,0,0,0,0,0},
-    {"watch",watchCommand,-2,"rs",0,noPreloadGetKeys,1,-1,1,0,0},
+    {"watch",watchCommand,-2,"rs",0,NULL,1,-1,1,0,0},
     {"unwatch",unwatchCommand,1,"rs",0,NULL,0,0,0,0,0},
     {"cluster",clusterCommand,-2,"ar",0,NULL,0,0,0,0,0},
     {"restore",restoreCommand,-4,"awm",0,NULL,1,1,1,0,0},
@@ -259,13 +259,14 @@ struct redisCommand redisCommandTable[] = {
     {"dump",dumpCommand,2,"ar",0,NULL,1,1,1,0,0},
     {"object",objectCommand,-2,"r",0,NULL,2,2,2,0,0},
     {"client",clientCommand,-2,"ar",0,NULL,0,0,0,0,0},
-    {"eval",evalCommand,-3,"s",0,zunionInterGetKeys,0,0,0,0,0},
-    {"evalsha",evalShaCommand,-3,"s",0,zunionInterGetKeys,0,0,0,0,0},
+    {"eval",evalCommand,-3,"s",0,evalGetKeys,0,0,0,0,0},
+    {"evalsha",evalShaCommand,-3,"s",0,evalGetKeys,0,0,0,0,0},
     {"slowlog",slowlogCommand,-2,"r",0,NULL,0,0,0,0,0},
     {"script",scriptCommand,-2,"ras",0,NULL,0,0,0,0,0},
     {"time",timeCommand,1,"rR",0,NULL,0,0,0,0,0},
     {"bitop",bitopCommand,-4,"wm",0,NULL,2,-1,1,0,0},
     {"bitcount",bitcountCommand,-2,"r",0,NULL,1,1,1,0,0},
+    {"bitpos",bitposCommand,-3,"r",0,NULL,1,1,1,0,0},
     {"wait",waitCommand,3,"rs",0,NULL,0,0,0,0,0}
 };
 
@@ -1001,6 +1002,15 @@ void databasesCron(void) {
     }
 }
 
+/* We take a cached value of the unix time in the global state because with
+ * virtual memory and aging there is to store the current time in objects at
+ * every object access, and accuracy is not needed. To access a global var is
+ * a lot faster than calling time(NULL) */
+void updateCachedTime(void) {
+    server.unixtime = time(NULL);
+    server.mstime = mstime();
+}
+
 /* This is our timer interrupt, called server.hz times per second.
  * Here is where we do a number of things that need to be done asynchronously.
  * For instance:
@@ -1030,12 +1040,8 @@ int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
      * handler if we don't return here fast enough. */
     if (server.watchdog_period) watchdogScheduleSignal(server.watchdog_period);
 
-    /* We take a cached value of the unix time in the global state because
-     * with virtual memory and aging there is to store the current time
-     * in objects at every object access, and accuracy is not needed.
-     * To access a global var is faster than calling time(NULL) */
-    server.unixtime = time(NULL);
-    server.mstime = mstime();
+    /* Update the time cache. */
+    updateCachedTime();
 
     run_with_period(100) trackOperationsPerSecond();
 
@@ -1167,10 +1173,19 @@ int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
     }
 
 
-    /* If we postponed an AOF buffer flush, let's try to do it every time the
-     * cron function is called. */
+    /* AOF postponed flush: Try at every cron cycle if the slow fsync
+     * completed. */
     if (server.aof_flush_postponed_start) flushAppendOnlyFile(0);
 
+    /* AOF write errors: in this case we have a buffer to flush as well and
+     * clear the AOF error in case of success to make the DB writable again,
+     * however to try every second is enough in case of 'hz' is set to
+     * an higher frequency. */
+    run_with_period(1000) {
+        if (server.aof_last_write_status == REDIS_ERR)
+            flushAppendOnlyFile(0);
+    }
+
     /* Close clients that need to be closed asynchronous */
     freeClientsInAsyncFreeQueue();
 
@@ -1567,7 +1582,7 @@ int listenToPort(int port, int *fds, int *count) {
             redisLog(REDIS_WARNING,
                 "Creating Server TCP listening socket %s:%d: %s",
                 server.bindaddr[j] ? server.bindaddr[j] : "*",
-                server.port, server.neterr);
+                port, server.neterr);
             return REDIS_ERR;
         }
         (*count)++;
@@ -1667,10 +1682,11 @@ void initServer() {
     server.ops_sec_idx = 0;
     server.ops_sec_last_sample_time = mstime();
     server.ops_sec_last_sample_ops = 0;
-    server.unixtime = time(NULL);
-    server.mstime = mstime();
     server.lastbgsave_status = REDIS_OK;
+    server.aof_last_write_status = REDIS_OK;
+    server.aof_last_write_errno = 0;
     server.repl_good_slaves_count = 0;
+    updateCachedTime();
 
     /* Create the serverCron() time event, that's our main way to process
      * background operations. */
@@ -2002,17 +2018,30 @@ int processCommand(redisClient *c) {
         int hashslot;
 
         if (server.cluster->state != REDIS_CLUSTER_OK) {
+            flagTransaction(c);
             addReplySds(c,sdsnew("-CLUSTERDOWN The cluster is down. Use CLUSTER INFO for more information\r\n"));
             return REDIS_OK;
         } else {
-            int ask;
-            clusterNode *n = getNodeByQuery(c,c->cmd,c->argv,c->argc,&hashslot,&ask);
+            int error_code;
+            clusterNode *n = getNodeByQuery(c,c->cmd,c->argv,c->argc,&hashslot,&error_code);
             if (n == NULL) {
-                addReplyError(c,"Multi keys request invalid in cluster");
+                flagTransaction(c);
+                if (error_code == REDIS_CLUSTER_REDIR_CROSS_SLOT) {
+                    addReplySds(c,sdsnew("-CROSSSLOT Keys in request don't hash to the same slot\r\n"));
+                } else if (error_code == REDIS_CLUSTER_REDIR_UNSTABLE) {
+                    /* The request spawns mutliple keys in the same slot,
+                     * but the slot is not "stable" currently as there is
+                     * a migration or import in progress. */
+                    addReplySds(c,sdsnew("-TRYAGAIN Multiple keys request during rehashing of slot\r\n"));
+                } else {
+                    redisPanic("getNodeByQuery() unknown error.");
+                }
                 return REDIS_OK;
             } else if (n != server.cluster->myself) {
+                flagTransaction(c);
                 addReplySds(c,sdscatprintf(sdsempty(),
-                    "-%s %d %s:%d\r\n", ask ? "ASK" : "MOVED",
+                    "-%s %d %s:%d\r\n",
+                    (error_code == REDIS_CLUSTER_REDIR_ASK) ? "ASK" : "MOVED",
                     hashslot,n->ip,n->port));
                 return REDIS_OK;
             }
@@ -2035,15 +2064,22 @@ int processCommand(redisClient *c) {
 
     /* Don't accept write commands if there are problems persisting on disk
      * and if this is a master instance. */
-    if (server.stop_writes_on_bgsave_err &&
-        server.saveparamslen > 0
-        && server.lastbgsave_status == REDIS_ERR &&
+    if (((server.stop_writes_on_bgsave_err &&
+          server.saveparamslen > 0 &&
+          server.lastbgsave_status == REDIS_ERR) ||
+          server.aof_last_write_status == REDIS_ERR) &&
         server.masterhost == NULL &&
         (c->cmd->flags & REDIS_CMD_WRITE ||
          c->cmd->proc == pingCommand))
     {
         flagTransaction(c);
-        addReply(c, shared.bgsaveerr);
+        if (server.aof_last_write_status == REDIS_OK)
+            addReply(c, shared.bgsaveerr);
+        else
+            addReplySds(c,
+                sdscatprintf(sdsempty(),
+                "-MISCONF Errors writing to the AOF file: %s\r\n",
+                strerror(server.aof_last_write_errno)));
         return REDIS_OK;
     }
 
@@ -2384,8 +2420,16 @@ sds genRedisInfoString(char *section) {
     if (allsections || defsections || !strcasecmp(section,"memory")) {
         char hmem[64];
         char peak_hmem[64];
+        size_t zmalloc_used = zmalloc_used_memory();
 
-        bytesToHuman(hmem,zmalloc_used_memory());
+        /* Peak memory is updated from time to time by serverCron() so it
+         * may happen that the instantaneous value is slightly bigger than
+         * the peak value. This may confuse users, so we update the peak
+         * if found smaller than the current memory usage. */
+        if (zmalloc_used > server.stat_peak_memory)
+            server.stat_peak_memory = zmalloc_used;
+
+        bytesToHuman(hmem,zmalloc_used);
         bytesToHuman(peak_hmem,server.stat_peak_memory);
         if (sections++) info = sdscat(info,"\r\n");
         info = sdscatprintf(info,
@@ -2398,7 +2442,7 @@ sds genRedisInfoString(char *section) {
             "used_memory_lua:%lld\r\n"
             "mem_fragmentation_ratio:%.2f\r\n"
             "mem_allocator:%s\r\n",
-            zmalloc_used_memory(),
+            zmalloc_used,
             hmem,
             zmalloc_get_rss(),
             server.stat_peak_memory,
@@ -2426,7 +2470,8 @@ sds genRedisInfoString(char *section) {
             "aof_rewrite_scheduled:%d\r\n"
             "aof_last_rewrite_time_sec:%jd\r\n"
             "aof_current_rewrite_time_sec:%jd\r\n"
-            "aof_last_bgrewrite_status:%s\r\n",
+            "aof_last_bgrewrite_status:%s\r\n"
+            "aof_last_write_status:%s\r\n",
             server.loading,
             server.dirty,
             server.rdb_child_pid != -1,
@@ -2441,7 +2486,8 @@ sds genRedisInfoString(char *section) {
             (intmax_t)server.aof_rewrite_time_last,
             (intmax_t)((server.aof_child_pid == -1) ?
                 -1 : time(NULL)-server.aof_rewrite_time_start),
-            (server.aof_lastbgrewrite_status == REDIS_OK) ? "ok" : "err");
+            (server.aof_lastbgrewrite_status == REDIS_OK) ? "ok" : "err",
+            (server.aof_last_write_status == REDIS_OK) ? "ok" : "err");
 
         if (server.aof_state != REDIS_AOF_OFF) {
             info = sdscatprintf(info,
@@ -3055,10 +3101,15 @@ void redisOutOfMemoryHandler(size_t allocation_size) {
 
 void redisSetProcTitle(char *title) {
 #ifdef USE_SETPROCTITLE
-    setproctitle("%s %s:%d",
+    char *server_mode = "";
+    if (server.cluster_enabled) server_mode = " [cluster]";
+    else if (server.sentinel_mode) server_mode = " [sentinel]";
+
+    setproctitle("%s %s:%d%s",
         title,
         server.bindaddr_count ? server.bindaddr[0] : "*",
-        server.port);
+        server.port,
+        server_mode);
 #else
     REDIS_NOTUSED(title);
 #endif
@@ -3129,10 +3180,10 @@ int main(int argc, char **argv) {
             }
             j++;
         }
+        if (configfile) server.configfile = getAbsolutePath(configfile);
         resetServerSaveParams();
         loadServerConfig(configfile,options);
         sdsfree(options);
-        if (configfile) server.configfile = getAbsolutePath(configfile);
     } else {
         redisLog(REDIS_WARNING, "Warning: no config file specified, using the default config. In order to specify a config file use %s /path/to/%s.conf", argv[0], server.sentinel_mode ? "sentinel" : "redis");
     }

src/redis.h

@@ -382,7 +382,7 @@ typedef long long mstime_t; /* millisecond time type. */
 /* A redis object, that is a type able to hold a string / list / set */
 
 /* The actual Redis Object */
-#define REDIS_LRU_CLOCK_MAX ((1<<21)-1) /* Max value of obj->lru */
+#define REDIS_LRU_CLOCK_MAX ((1<<22)-1) /* Max value of obj->lru */
 #define REDIS_LRU_CLOCK_RESOLUTION 10 /* LRU clock resolution in seconds */
 typedef struct redisObject {
     unsigned type:4;
@@ -691,6 +691,8 @@ struct redisServer {
     int aof_lastbgrewrite_status;   /* REDIS_OK or REDIS_ERR */
     unsigned long aof_delayed_fsync;  /* delayed AOF fsync() counter */
     int aof_rewrite_incremental_fsync;/* fsync incrementally while rewriting? */
+    int aof_last_write_status;      /* REDIS_OK or REDIS_ERR */
+    int aof_last_write_errno;       /* Valid if aof_last_write_status is ERR */
     /* RDB persistence */
     long long dirty;                /* Changes to DB from the last save */
     long long dirty_before_bgsave;  /* Used to restore dirty on failed BGSAVE */
@@ -824,7 +826,7 @@ typedef struct pubsubPattern {
 } pubsubPattern;
 
 typedef void redisCommandProc(redisClient *c);
-typedef int *redisGetKeysProc(struct redisCommand *cmd, robj **argv, int argc, int *numkeys, int flags);
+typedef int *redisGetKeysProc(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
 struct redisCommand {
     char *name;
     redisCommandProc *proc;
@@ -1152,6 +1154,7 @@ void populateCommandTable(void);
 void resetCommandTableStats(void);
 void adjustOpenFilesLimit(void);
 void closeListeningSockets(int unlink_unix_socket);
+void updateCachedTime();
 
 /* Set data type */
 robj *setTypeCreate(robj *value);
@@ -1234,13 +1237,11 @@ void scanGenericCommand(redisClient *c, robj *o, unsigned long cursor);
 int parseScanCursorOrReply(redisClient *c, robj *o, unsigned long *cursor);
 
 /* API to get key arguments from commands */
-#define REDIS_GETKEYS_ALL 0
-#define REDIS_GETKEYS_PRELOAD 1
-int *getKeysFromCommand(struct redisCommand *cmd, robj **argv, int argc, int *numkeys, int flags);
+int *getKeysFromCommand(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
 void getKeysFreeResult(int *result);
-int *noPreloadGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags);
-int *renameGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags);
-int *zunionInterGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags);
+int *zunionInterGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys);
+int *evalGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
+int *sortGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
 
 /* Cluster */
 void clusterInit(void);
@@ -1417,6 +1418,7 @@ void scriptCommand(redisClient *c);
 void timeCommand(redisClient *c);
 void bitopCommand(redisClient *c);
 void bitcountCommand(redisClient *c);
+void bitposCommand(redisClient *c);
 void replconfCommand(redisClient *c);
 void waitCommand(redisClient *c);
 

src/scripting.c

@@ -958,6 +958,7 @@ void evalGenericCommand(redisClient *c, int evalsha) {
             rewriteClientCommandArgument(c,0,
                 resetRefCount(createStringObject("EVAL",4)));
             rewriteClientCommandArgument(c,1,script);
+            forceCommandPropagation(c,REDIS_PROPAGATE_REPL|REDIS_PROPAGATE_AOF);
         }
     }
 }

src/sentinel.c

@@ -78,12 +78,13 @@ typedef struct sentinelAddr {
 #define SENTINEL_TILT_TRIGGER 2000
 #define SENTINEL_TILT_PERIOD (SENTINEL_PING_PERIOD*30)
 #define SENTINEL_DEFAULT_SLAVE_PRIORITY 100
-#define SENTINEL_SLAVE_RECONF_RETRY_PERIOD 10000
+#define SENTINEL_SLAVE_RECONF_TIMEOUT 10000
 #define SENTINEL_DEFAULT_PARALLEL_SYNCS 1
 #define SENTINEL_MIN_LINK_RECONNECT_PERIOD 15000
 #define SENTINEL_DEFAULT_FAILOVER_TIMEOUT (60*3*1000)
 #define SENTINEL_MAX_PENDING_COMMANDS 100
 #define SENTINEL_ELECTION_TIMEOUT 10000
+#define SENTINEL_MAX_DESYNC 1000
 
 /* Failover machine different states. */
 #define SENTINEL_FAILOVER_STATE_NONE 0  /* No failover in progress. */
@@ -327,6 +328,7 @@ void sentinelDiscardReplyCallback(redisAsyncContext *c, void *reply, void *privd
 int sentinelSendSlaveOf(sentinelRedisInstance *ri, char *host, int port);
 char *sentinelVoteLeader(sentinelRedisInstance *master, uint64_t req_epoch, char *req_runid, uint64_t *leader_epoch);
 void sentinelFlushConfig(void);
+void sentinelGenerateInitialMonitorEvents(void);
 
 /* ========================= Dictionary types =============================== */
 
@@ -417,10 +419,20 @@ void initSentinel(void) {
 void sentinelIsRunning(void) {
     redisLog(REDIS_WARNING,"Sentinel runid is %s", server.runid);
 
-    if (server.configfile == NULL || access(server.configfile,W_OK) == -1) {
-        redisLog(REDIS_WARNING,"Sentinel started without a config file, or config file not writable. Exiting...");
+    if (server.configfile == NULL) {
+        redisLog(REDIS_WARNING,
+            "Sentinel started without a config file. Exiting...");
+        exit(1);
+    } else if (access(server.configfile,W_OK) == -1) {
+        redisLog(REDIS_WARNING,
+            "Sentinel config file %s is not writable: %s. Exiting...",
+            server.configfile,strerror(errno));
         exit(1);
     }
+
+    /* We want to generate a +monitor event for every configured master
+     * at startup. */
+    sentinelGenerateInitialMonitorEvents();
 }
 
 /* ============================== sentinelAddr ============================== */
@@ -552,6 +564,22 @@ void sentinelEvent(int level, char *type, sentinelRedisInstance *ri,
     }
 }
 
+/* This function is called only at startup and is used to generate a
+ * +monitor event for every configured master. The same events are also
+ * generated when a master to monitor is added at runtime via the
+ * SENTINEL MONITOR command. */
+void sentinelGenerateInitialMonitorEvents(void) {
+    dictIterator *di;
+    dictEntry *de;
+
+    di = dictGetIterator(sentinel.masters);
+    while((de = dictNext(di)) != NULL) {
+        sentinelRedisInstance *ri = dictGetVal(de);
+        sentinelEvent(REDIS_WARNING,"+monitor",ri,"%@ quorum %d",ri->quorum);
+    }
+    dictReleaseIterator(di);
+}
+
 /* ============================ script execution ============================ */
 
 /* Release a script job structure and all the associated data. */
@@ -1486,7 +1514,7 @@ void sentinelFlushConfig(void) {
             close(fd);
         }
     } else {
-        redisLog(REDIS_WARNING,"WARNING: Senitnel was not able to save the new configuration on disk!!!: %s", strerror(errno));
+        redisLog(REDIS_WARNING,"WARNING: Sentinel was not able to save the new configuration on disk!!!: %s", strerror(errno));
     }
     server.hz = saved_hz;
     return;
@@ -1765,6 +1793,14 @@ void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) {
         ri->role_reported_time = mstime();
         ri->role_reported = role;
         if (role == SRI_SLAVE) ri->slave_conf_change_time = mstime();
+        /* Log the event with +role-change if the new role is coherent or
+         * with -role-change if there is a mismatch with the current config. */
+        sentinelEvent(REDIS_VERBOSE,
+            ((ri->flags & (SRI_MASTER|SRI_SLAVE)) == role) ?
+            "+role-change" : "-role-change",
+            ri, "%@ new reported role is %s",
+            role == SRI_MASTER ? "master" : "slave",
+            ri->flags & SRI_MASTER ? "master" : "slave");
     }
 
     /* Handle master -> slave role switch. */
@@ -2455,8 +2491,6 @@ void sentinelCommand(redisClient *c) {
         ri = sentinelGetMasterByName(c->argv[2]->ptr);
         if (ri == NULL) {
             addReply(c,shared.nullmultibulk);
-        } else if (ri->info_refresh == 0) {
-            addReplySds(c,sdsnew("-IDONTKNOW I have not enough information to reply. Please ask another Sentinel.\r\n"));
         } else {
             sentinelAddr *addr = sentinelGetCurrentMasterAddress(ri);
 
@@ -2491,6 +2525,7 @@ void sentinelCommand(redisClient *c) {
         sentinelPendingScriptsCommand(c);
     } else if (!strcasecmp(c->argv[1]->ptr,"monitor")) {
         /* SENTINEL MONITOR <name> <ip> <port> <quorum> */
+        sentinelRedisInstance *ri;
         long quorum, port;
         char buf[32];
 
@@ -2506,9 +2541,11 @@ void sentinelCommand(redisClient *c) {
             addReplyError(c,"Invalid IP address specified");
             return;
         }
-        if (createSentinelRedisInstance(c->argv[2]->ptr,SRI_MASTER,
-                c->argv[3]->ptr,port,quorum,NULL) == NULL)
-        {
+
+        /* Parameters are valid. Try to create the master instance. */
+        ri = createSentinelRedisInstance(c->argv[2]->ptr,SRI_MASTER,
+                c->argv[3]->ptr,port,quorum,NULL);
+        if (ri == NULL) {
             switch(errno) {
             case EBUSY:
                 addReplyError(c,"Duplicated master name");
@@ -2522,6 +2559,7 @@ void sentinelCommand(redisClient *c) {
             }
         } else {
             sentinelFlushConfig();
+            sentinelEvent(REDIS_WARNING,"+monitor",ri,"%@ quorum %d",ri->quorum);
             addReply(c,shared.ok);
         }
     } else if (!strcasecmp(c->argv[1]->ptr,"remove")) {
@@ -2530,6 +2568,7 @@ void sentinelCommand(redisClient *c) {
 
         if ((ri = sentinelGetMasterByNameOrReplyError(c,c->argv[2]))
             == NULL) return;
+        sentinelEvent(REDIS_WARNING,"-monitor",ri,"%@");
         dictDelete(sentinel.masters,c->argv[2]->ptr);
         sentinelFlushConfig();
         addReply(c,shared.ok);
@@ -2681,6 +2720,7 @@ void sentinelSetCommand(redisClient *c) {
             if (changes) sentinelFlushConfig();
             return;
         }
+        sentinelEvent(REDIS_WARNING,"+set",ri,"%@ %s %s",option,value);
     }
 
     if (changes) sentinelFlushConfig();
@@ -2904,7 +2944,7 @@ char *sentinelVoteLeader(sentinelRedisInstance *master, uint64_t req_epoch, char
          * time to now, in order to force a delay before we can start a
          * failover for the same master. */
         if (strcasecmp(master->leader,server.runid))
-            master->failover_start_time = mstime();
+            master->failover_start_time = mstime()+rand()%SENTINEL_MAX_DESYNC;
     }
 
     *leader_epoch = master->leader_epoch;
@@ -3049,7 +3089,7 @@ void sentinelStartFailover(sentinelRedisInstance *master) {
     sentinelEvent(REDIS_WARNING,"+new-epoch",master,"%llu",
         (unsigned long long) sentinel.current_epoch);
     sentinelEvent(REDIS_WARNING,"+try-failover",master,"%@");
-    master->failover_start_time = mstime();
+    master->failover_start_time = mstime()+rand()%SENTINEL_MAX_DESYNC;
     master->failover_state_change_time = mstime();
 }
 
@@ -3359,14 +3399,17 @@ void sentinelFailoverReconfNextSlave(sentinelRedisInstance *master) {
         /* Skip the promoted slave, and already configured slaves. */
         if (slave->flags & (SRI_PROMOTED|SRI_RECONF_DONE)) continue;
 
-        /* Clear the SRI_RECONF_SENT flag if too much time elapsed without
-         * the slave moving forward to the next state. */
+        /* If too much time elapsed without the slave moving forward to
+         * the next state, consider it reconfigured even if it is not.
+         * Sentinels will detect the slave as misconfigured and fix its
+         * configuration later. */
         if ((slave->flags & SRI_RECONF_SENT) &&
             (mstime() - slave->slave_reconf_sent_time) >
-            SENTINEL_SLAVE_RECONF_RETRY_PERIOD)
+            SENTINEL_SLAVE_RECONF_TIMEOUT)
         {
             sentinelEvent(REDIS_NOTICE,"-slave-reconf-sent-timeout",slave,"%@");
             slave->flags &= ~SRI_RECONF_SENT;
+            slave->flags |= SRI_RECONF_DONE;
         }
 
         /* Nothing to do for instances that are disconnected or already

src/sort.c

@@ -194,6 +194,7 @@ void sortCommand(redisClient *c) {
     int j, dontsort = 0, vectorlen;
     int getop = 0; /* GET operation counter */
     int int_convertion_error = 0;
+    int syntax_error = 0;
     robj *sortval, *sortby = NULL, *storekey = NULL;
     redisSortObject *vector; /* Resulting vector to sort */
 
@@ -231,8 +232,14 @@ void sortCommand(redisClient *c) {
         } else if (!strcasecmp(c->argv[j]->ptr,"alpha")) {
             alpha = 1;
         } else if (!strcasecmp(c->argv[j]->ptr,"limit") && leftargs >= 2) {
-            if ((getLongFromObjectOrReply(c, c->argv[j+1], &limit_start, NULL) != REDIS_OK) ||
-                (getLongFromObjectOrReply(c, c->argv[j+2], &limit_count, NULL) != REDIS_OK)) return;
+            if ((getLongFromObjectOrReply(c, c->argv[j+1], &limit_start, NULL)
+                 != REDIS_OK) ||
+                (getLongFromObjectOrReply(c, c->argv[j+2], &limit_count, NULL)
+                 != REDIS_OK))
+            {
+                syntax_error++;
+                break;
+            }
             j+=2;
         } else if (!strcasecmp(c->argv[j]->ptr,"store") && leftargs >= 1) {
             storekey = c->argv[j+1];
@@ -241,22 +248,43 @@ void sortCommand(redisClient *c) {
             sortby = c->argv[j+1];
             /* If the BY pattern does not contain '*', i.e. it is constant,
              * we don't need to sort nor to lookup the weight keys. */
-            if (strchr(c->argv[j+1]->ptr,'*') == NULL) dontsort = 1;
+            if (strchr(c->argv[j+1]->ptr,'*') == NULL) {
+                dontsort = 1;
+            } else {
+                /* If BY is specified with a real patter, we can't accept
+                 * it in cluster mode. */
+                if (server.cluster_enabled) {
+                    addReplyError(c,"BY option of SORT denied in Cluster mode.");
+                    syntax_error++;
+                    break;
+                }
+            }
             j++;
         } else if (!strcasecmp(c->argv[j]->ptr,"get") && leftargs >= 1) {
+            if (server.cluster_enabled) {
+                addReplyError(c,"GET option of SORT denied in Cluster mode.");
+                syntax_error++;
+                break;
+            }
             listAddNodeTail(operations,createSortOperation(
                 REDIS_SORT_GET,c->argv[j+1]));
             getop++;
             j++;
         } else {
-            decrRefCount(sortval);
-            listRelease(operations);
             addReply(c,shared.syntaxerr);
-            return;
+            syntax_error++;
+            break;
         }
         j++;
     }
 
+    /* Handle syntax errors set during options parsing. */
+    if (syntax_error) {
+        decrRefCount(sortval);
+        listRelease(operations);
+        return;
+    }
+
     /* For the STORE option, or when SORT is called from a Lua script,
      * we want to force a specific ordering even when no explicit ordering
      * was asked (SORT BY nosort). This guarantees that replication / AOF

src/version.h

@@ -1 +1 @@
-#define REDIS_VERSION "2.9.11"
+#define REDIS_VERSION "2.9.51"

tests/sentinel-tests/00-base.tcl

@@ -0,0 +1,126 @@
+# Check the basic monitoring and failover capabilities.
+
+source "../sentinel-tests/includes/init-tests.tcl"
+
+if {$::simulate_error} {
+    test "This test will fail" {
+        fail "Simulated error"
+    }
+}
+
+test "Basic failover works if the master is down" {
+    set old_port [RI $master_id tcp_port]
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    assert {[lindex $addr 1] == $old_port}
+    kill_instance redis $master_id
+    foreach_sentinel_id id {
+        wait_for_condition 1000 50 {
+            [lindex [S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port
+        } else {
+            fail "At least one Sentinel did not received failover info"
+        }
+    }
+    restart_instance redis $master_id
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    set master_id [get_instance_id_by_port redis [lindex $addr 1]]
+}
+
+test "New master [join $addr {:}] role matches" {
+    assert {[RI $master_id role] eq {master}}
+}
+
+test "All the other slaves now point to the new master" {
+    foreach_redis_id id {
+        if {$id != $master_id && $id != 0} {
+            wait_for_condition 1000 50 {
+                [RI $id master_port] == [lindex $addr 1]
+            } else {
+                fail "Redis ID $id not configured to replicate with new master"
+            }
+        }
+    }
+}
+
+test "The old master eventually gets reconfigured as a slave" {
+    wait_for_condition 1000 50 {
+        [RI 0 master_port] == [lindex $addr 1]
+    } else {
+        fail "Old master not reconfigured as slave of new master"
+    }
+}
+
+test "ODOWN is not possible without N (quorum) Sentinels reports" {
+    foreach_sentinel_id id {
+        S $id SENTINEL SET mymaster quorum [expr $sentinels+1]
+    }
+    set old_port [RI $master_id tcp_port]
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    assert {[lindex $addr 1] == $old_port}
+    kill_instance redis $master_id
+
+    # Make sure failover did not happened.
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    assert {[lindex $addr 1] == $old_port}
+    restart_instance redis $master_id
+}
+
+test "Failover is not possible without majority agreement" {
+    foreach_sentinel_id id {
+        S $id SENTINEL SET mymaster quorum $quorum
+    }
+
+    # Crash majority of sentinels
+    for {set id 0} {$id < $quorum} {incr id} {
+        kill_instance sentinel $id
+    }
+
+    # Kill the current master
+    kill_instance redis $master_id
+
+    # Make sure failover did not happened.
+    set addr [S $quorum SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    assert {[lindex $addr 1] == $old_port}
+    restart_instance redis $master_id
+
+    # Cleanup: restart Sentinels to monitor the master.
+    for {set id 0} {$id < $quorum} {incr id} {
+        restart_instance sentinel $id
+    }
+}
+
+test "Failover works if we configure for absolute agreement" {
+    foreach_sentinel_id id {
+        S $id SENTINEL SET mymaster quorum $sentinels
+    }
+
+    # Wait for Sentinels to monitor the master again
+    foreach_sentinel_id id {
+        wait_for_condition 1000 50 {
+            [dict get [S $id SENTINEL MASTER mymaster] info-refresh] < 100000
+        } else {
+            fail "At least one Sentinel is not monitoring the master"
+        }
+    }
+
+    kill_instance redis $master_id
+
+    foreach_sentinel_id id {
+        wait_for_condition 1000 50 {
+            [lindex [S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port
+        } else {
+            fail "At least one Sentinel did not received failover info"
+        }
+    }
+    restart_instance redis $master_id
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    set master_id [get_instance_id_by_port redis [lindex $addr 1]]
+
+    # Set the min ODOWN agreement back to strict majority.
+    foreach_sentinel_id id {
+        S $id SENTINEL SET mymaster quorum $quorum
+    }
+}
+
+test "New master [join $addr {:}] role matches" {
+    assert {[RI $master_id role] eq {master}}
+}

tests/sentinel-tests/01-conf-update.tcl

@@ -0,0 +1,39 @@
+# Test Sentinel configuration consistency after partitions heal.
+
+source "../sentinel-tests/includes/init-tests.tcl"
+
+test "We can failover with Sentinel 1 crashed" {
+    set old_port [RI $master_id tcp_port]
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    assert {[lindex $addr 1] == $old_port}
+
+    # Crash Sentinel 1
+    kill_instance sentinel 1
+
+    kill_instance redis $master_id
+    foreach_sentinel_id id {
+        if {$id != 1} {
+            wait_for_condition 1000 50 {
+                [lindex [S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port
+            } else {
+                fail "Sentinel $id did not received failover info"
+            }
+        }
+    }
+    restart_instance redis $master_id
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    set master_id [get_instance_id_by_port redis [lindex $addr 1]]
+}
+
+test "After Sentinel 1 is restarted, its config gets updated" {
+    restart_instance sentinel 1
+    wait_for_condition 1000 50 {
+        [lindex [S 1 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port
+    } else {
+        fail "Restarted Sentinel did not received failover info"
+    }
+}
+
+test "New master [join $addr {:}] role matches" {
+    assert {[RI $master_id role] eq {master}}
+}

tests/sentinel-tests/02-slaves-reconf.tcl

@@ -0,0 +1,45 @@
+# Check that slaves are reconfigured at a latter time if they are partitioned.
+#
+# Here we should test:
+# 1) That slaves point to the new master after failover.
+# 2) That partitioned slaves point to new master when they are partitioned
+#    away during failover and return at a latter time.
+
+source "../sentinel-tests/includes/init-tests.tcl"
+
+proc 03_test_slaves_replication {} {
+    uplevel 1 {
+        test "Check that slaves replicate from current master" {
+            set master_port [RI $master_id tcp_port]
+            foreach_redis_id id {
+                if {$id == $master_id} continue
+                wait_for_condition 1000 50 {
+                    [RI $id master_port] == $master_port
+                } else {
+                    fail "Redis slave $id is replicating from wrong master"
+                }
+            }
+        }
+    }
+}
+
+03_test_slaves_replication
+
+test "Crash the master and force a failover" {
+    set old_port [RI $master_id tcp_port]
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    assert {[lindex $addr 1] == $old_port}
+    kill_instance redis $master_id
+    foreach_sentinel_id id {
+        wait_for_condition 1000 50 {
+            [lindex [S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port
+        } else {
+            fail "At least one Sentinel did not received failover info"
+        }
+    }
+    restart_instance redis $master_id
+    set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
+    set master_id [get_instance_id_by_port redis [lindex $addr 1]]
+}
+
+03_test_slaves_replication

tests/sentinel-tests/03-runtime-reconf.tcl

@@ -0,0 +1 @@
+# Test runtime reconfiguration command SENTINEL SET.

tests/sentinel-tests/04-slave-selection.tcl

@@ -0,0 +1,5 @@
+# Test slave selection algorithm.
+#
+# This unit should test:
+# 1) That when there are no suitable slaves no failover is performed.
+# 2) That among the available slaves, the one with better offset is picked.

tests/sentinel-tests/includes/init-tests.tcl

@@ -0,0 +1,67 @@
+# Initialization tests -- most units will start including this.
+
+test "(init) Restart killed instances" {
+    foreach type {redis sentinel} {
+        foreach_${type}_id id {
+            if {[get_instance_attrib $type $id pid] == -1} {
+                puts -nonewline "$type/$id "
+                flush stdout
+                restart_instance $type $id
+            }
+        }
+    }
+}
+
+set redis_slaves 4
+test "(init) Create a master-slaves cluster of [expr $redis_slaves+1] instances" {
+    create_redis_master_slave_cluster [expr {$redis_slaves+1}]
+}
+set master_id 0
+
+test "(init) Sentinels can start monitoring a master" {
+    set sentinels [llength $::sentinel_instances]
+    set quorum [expr {$sentinels/2+1}]
+    foreach_sentinel_id id {
+        catch {S $id SENTINEL REMOVE mymaster}
+        S $id SENTINEL MONITOR mymaster \
+              [get_instance_attrib redis $master_id host] \
+              [get_instance_attrib redis $master_id port] $quorum
+    }
+    foreach_sentinel_id id {
+        assert {[S $id sentinel master mymaster] ne {}}
+        S $id SENTINEL SET mymaster down-after-milliseconds 2000
+        S $id SENTINEL SET mymaster failover-timeout 20000
+        S $id SENTINEL SET mymaster parallel-syncs 10
+    }
+}
+
+test "(init) Sentinels can talk with the master" {
+    foreach_sentinel_id id {
+        wait_for_condition 100 50 {
+            [catch {S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster}] == 0
+        } else {
+            fail "Sentinel $id can't talk with the master."
+        }
+    }
+}
+
+test "(init) Sentinels are able to auto-discover other sentinels" {
+    set sentinels [llength $::sentinel_instances]
+    foreach_sentinel_id id {
+        wait_for_condition 100 50 {
+            [dict get [S $id SENTINEL MASTER mymaster] num-other-sentinels] == ($sentinels-1)
+        } else {
+            fail "At least some sentinel can't detect some other sentinel"
+        }
+    }
+}
+
+test "(init) Sentinels are able to auto-discover slaves" {
+    foreach_sentinel_id id {
+        wait_for_condition 100 50 {
+            [dict get [S $id SENTINEL MASTER mymaster] num-slaves] == $redis_slaves
+        } else {
+            fail "At least some sentinel can't detect some slave"
+        }
+    }
+}

tests/sentinel-tmp/.gitignore

@@ -0,0 +1,2 @@
+redis_*
+sentinel_*

tests/sentinel.tcl

@@ -0,0 +1,388 @@
+# Sentinel test suite. Copyright (C) 2014 Salvatore Sanfilippo antirez@gmail.com
+# This softare is released under the BSD License. See the COPYING file for
+# more information.
+
+package require Tcl 8.5
+
+set tcl_precision 17
+source tests/support/redis.tcl
+source tests/support/util.tcl
+source tests/support/server.tcl
+source tests/support/test.tcl
+
+set ::verbose 0
+set ::pause_on_error 0
+set ::simulate_error 0
+set ::sentinel_instances {}
+set ::redis_instances {}
+set ::sentinel_base_port 20000
+set ::redis_base_port 30000
+set ::instances_count 5 ; # How many Sentinels / Instances we use at max
+set ::pids {} ; # We kill everything at exit
+set ::dirs {} ; # We remove all the temp dirs at exit
+set ::run_matching {} ; # If non empty, only tests matching pattern are run.
+
+if {[catch {cd tests/sentinel-tmp}]} {
+    puts "tests/sentinel-tmp directory not found."
+    puts "Please run this test from the Redis source root."
+    exit 1
+}
+
+# Spawn a redis or sentinel instance, depending on 'type'.
+proc spawn_instance {type base_port count} {
+    for {set j 0} {$j < $count} {incr j} {
+        set port [find_available_port $base_port]
+        incr base_port
+        puts "Starting $type #$j at port $port"
+
+        # Create a directory for this Sentinel.
+        set dirname "${type}_${j}"
+        lappend ::dirs $dirname
+        catch {exec rm -rf $dirname}
+        file mkdir $dirname
+
+        # Write the Sentinel config file.
+        set cfgfile [file join $dirname $type.conf]
+        set cfg [open $cfgfile w]
+        puts $cfg "port $port"
+        puts $cfg "dir ./$dirname"
+        puts $cfg "logfile log.txt"
+        close $cfg
+
+        # Finally exec it and remember the pid for later cleanup.
+        if {$type eq "redis"} {
+            set prgname redis-server
+        } else {
+            set prgname redis-sentinel
+        }
+        set pid [exec ../../src/${prgname} $cfgfile &]
+        lappend ::pids $pid
+
+        # Check availability
+        if {[server_is_up 127.0.0.1 $port 100] == 0} {
+            abort_sentinel_test "Problems starting $type #$j: ping timeout"
+        }
+
+        # Push the instance into the right list
+        lappend ::${type}_instances [list \
+            pid $pid \
+            host 127.0.0.1 \
+            port $port \
+            link [redis 127.0.0.1 $port] \
+        ]
+    }
+}
+
+proc cleanup {} {
+    puts "Cleaning up..."
+    foreach pid $::pids {
+        catch {exec kill -9 $pid}
+    }
+    foreach dir $::dirs {
+        catch {exec rm -rf $dir}
+    }
+}
+
+proc abort_sentinel_test msg {
+    puts "WARNING: Aborting the test."
+    puts ">>>>>>>> $msg"
+    cleanup
+    exit 1
+}
+
+proc parse_options {} {
+    for {set j 0} {$j < [llength $::argv]} {incr j} {
+        set opt [lindex $::argv $j]
+        set val [lindex $::argv [expr $j+1]]
+        if {$opt eq "--single"} {
+            incr j
+            set ::run_matching "*${val}*"
+        } elseif {$opt eq "--pause-on-error"} {
+            set ::pause_on_error 1
+        } elseif {$opt eq "--fail"} {
+            set ::simulate_error 1
+        } elseif {$opt eq "--help"} {
+            puts "Hello, I'm sentinel.tcl and I run Sentinel unit tests."
+            puts "\nOptions:"
+            puts "--single <pattern>      Only runs tests specified by pattern."
+            puts "--pause-on-error        Pause for manual inspection on error."
+            puts "--fail                  Simulate a test failure."
+            puts "--help                  Shows this help."
+            exit 0
+        } else {
+            puts "Unknown option $opt"
+            exit 1
+        }
+    }
+}
+
+proc main {} {
+    parse_options
+    spawn_instance sentinel $::sentinel_base_port $::instances_count
+    spawn_instance redis $::redis_base_port $::instances_count
+    run_tests
+    cleanup
+}
+
+# If --pause-on-error option was passed at startup this function is called
+# on error in order to give the developer a chance to understand more about
+# the error condition while the instances are still running.
+proc pause_on_error {} {
+    puts ""
+    puts [colorstr yellow "*** Please inspect the error now ***"]
+    puts "\nType \"continue\" to resume the test, \"help\" for help screen.\n"
+    while 1 {
+        puts -nonewline "> "
+        flush stdout
+        set line [gets stdin]
+        set argv [split $line " "]
+        set cmd [lindex $argv 0]
+        if {$cmd eq {continue}} {
+            break
+        } elseif {$cmd eq {show-sentinel-logs}} {
+            set count 10
+            if {[lindex $argv 1] ne {}} {set count [lindex $argv 1]}
+            foreach_sentinel_id id {
+                puts "=== SENTINEL $id ===="
+                puts [exec tail -$count sentinel_$id/log.txt]
+                puts "---------------------\n"
+            }
+        } elseif {$cmd eq {ls}} {
+            foreach_redis_id id {
+                puts -nonewline "Redis $id"
+                set errcode [catch {
+                    set str {}
+                    append str "@[RI $id tcp_port]: "
+                    append str "[RI $id role] "
+                    if {[RI $id role] eq {slave}} {
+                        append str "[RI $id master_host]:[RI $id master_port]"
+                    }
+                    set str
+                } retval]
+                if {$errcode} {
+                    puts " -- $retval"
+                } else {
+                    puts $retval
+                }
+            }
+            foreach_sentinel_id id {
+                puts -nonewline "Sentinel $id"
+                set errcode [catch {
+                    set str {}
+                    append str "@[SI $id tcp_port]: "
+                    append str "[join [S $id sentinel get-master-addr-by-name mymaster]]"
+                    set str
+                } retval]
+                if {$errcode} {
+                    puts " -- $retval"
+                } else {
+                    puts $retval
+                }
+            }
+        } elseif {$cmd eq {help}} {
+            puts "ls                     List Sentinel and Redis instances."
+            puts "show-sentinel-logs \[N\] Show latest N lines of logs."
+            puts "S <id> cmd ... arg     Call command in Sentinel <id>."
+            puts "R <id> cmd ... arg     Call command in Redis <id>."
+            puts "SI <id> <field>        Show Sentinel <id> INFO <field>."
+            puts "RI <id> <field>        Show Sentinel <id> INFO <field>."
+            puts "continue               Resume test."
+        } else {
+            set errcode [catch {eval $line} retval]
+            if {$retval ne {}} {puts "$retval"}
+        }
+    }
+}
+
+# We redefine 'test' as for Sentinel we don't use the server-client
+# architecture for the test, everything is sequential.
+proc test {descr code} {
+    set ts [clock format [clock seconds] -format %H:%M:%S]
+    puts -nonewline "$ts> $descr: "
+    flush stdout
+
+    if {[catch {set retval [uplevel 1 $code]} error]} {
+        if {[string match "assertion:*" $error]} {
+            set msg [string range $error 10 end]
+            puts [colorstr red $msg]
+            if {$::pause_on_error} pause_on_error
+            puts "(Jumping to next unit after error)"
+            return -code continue
+        } else {
+            # Re-raise, let handler up the stack take care of this.
+            error $error $::errorInfo
+        }
+    } else {
+        puts [colorstr green OK]
+    }
+}
+
+proc run_tests {} {
+    set tests [lsort [glob ../sentinel-tests/*]]
+    foreach test $tests {
+        if {$::run_matching ne {} && [string match $::run_matching $test] == 0} {
+            continue
+        }
+        if {[file isdirectory $test]} continue
+        puts [colorstr yellow "Testing unit: [lindex [file split $test] end]"]
+        source $test
+    }
+}
+
+# The "S" command is used to interact with the N-th Sentinel.
+# The general form is:
+#
+# S <sentinel-id> command arg arg arg ...
+#
+# Example to ping the Sentinel 0 (first instance): S 0 PING
+proc S {n args} {
+    set s [lindex $::sentinel_instances $n]
+    [dict get $s link] {*}$args
+}
+
+# Like R but to chat with Redis instances.
+proc R {n args} {
+    set r [lindex $::redis_instances $n]
+    [dict get $r link] {*}$args
+}
+
+proc get_info_field {info field} {
+    set fl [string length $field]
+    append field :
+    foreach line [split $info "\n"] {
+        set line [string trim $line "\r\n "]
+        if {[string range $line 0 $fl] eq $field} {
+            return [string range $line [expr {$fl+1}] end]
+        }
+    }
+    return {}
+}
+
+proc SI {n field} {
+    get_info_field [S $n info] $field
+}
+
+proc RI {n field} {
+    get_info_field [R $n info] $field
+}
+
+# Iterate over IDs of sentinel or redis instances.
+proc foreach_instance_id {instances idvar code} {
+    upvar 1 $idvar id
+    for {set id 0} {$id < [llength $instances]} {incr id} {
+        set errcode [catch {uplevel 1 $code} result]
+        if {$errcode == 1} {
+            error $result $::errorInfo $::errorCode
+        } elseif {$errcode == 4} {
+            continue
+        } elseif {$errcode != 0} {
+            return -code $errcode $result
+        }
+    }
+}
+
+proc foreach_sentinel_id {idvar code} {
+    set errcode [catch {uplevel 1 [list foreach_instance_id $::sentinel_instances $idvar $code]} result]
+    return -code $errcode $result
+}
+
+proc foreach_redis_id {idvar code} {
+    set errcode [catch {uplevel 1 [list foreach_instance_id $::redis_instances $idvar $code]} result]
+    return -code $errcode $result
+}
+
+# Get the specific attribute of the specified instance type, id.
+proc get_instance_attrib {type id attrib} {
+    dict get [lindex [set ::${type}_instances] $id] $attrib
+}
+
+# Set the specific attribute of the specified instance type, id.
+proc set_instance_attrib {type id attrib newval} {
+    set d [lindex [set ::${type}_instances] $id]
+    dict set d $attrib $newval
+    lset ::${type}_instances $id $d
+}
+
+# Create a master-slave cluster of the given number of total instances.
+# The first instance "0" is the master, all others are configured as
+# slaves.
+proc create_redis_master_slave_cluster n {
+    foreach_redis_id id {
+        if {$id == 0} {
+            # Our master.
+            R $id slaveof no one
+            R $id flushall
+        } elseif {$id < $n} {
+            R $id slaveof [get_instance_attrib redis 0 host] \
+                          [get_instance_attrib redis 0 port]
+        } else {
+            # Instances not part of the cluster.
+            R $id slaveof no one
+        }
+    }
+    # Wait for all the slaves to sync.
+    wait_for_condition 1000 50 {
+        [RI 0 connected_slaves] == ($n-1)
+    } else {
+        fail "Unable to create a master-slaves cluster."
+    }
+}
+
+proc get_instance_id_by_port {type port} {
+    foreach_${type}_id id {
+        if {[get_instance_attrib $type $id port] == $port} {
+            return $id
+        }
+    }
+    fail "Instance $type port $port not found."
+}
+
+# Kill an instance of the specified type/id with SIGKILL.
+# This function will mark the instance PID as -1 to remember that this instance
+# is no longer running and will remove its PID from the list of pids that
+# we kill at cleanup.
+#
+# The instance can be restarted with restart-instance.
+proc kill_instance {type id} {
+    set pid [get_instance_attrib $type $id pid]
+    if {$pid == -1} {
+        error "You tried to kill $type $id twice."
+    }
+    exec kill -9 $pid
+    set_instance_attrib $type $id pid -1
+    set_instance_attrib $type $id link you_tried_to_talk_with_killed_instance
+
+    # Remove the PID from the list of pids to kill at exit.
+    set ::pids [lsearch -all -inline -not -exact $::pids $pid]
+}
+
+# Restart an instance previously killed by kill_instance
+proc restart_instance {type id} {
+    set dirname "${type}_${id}"
+    set cfgfile [file join $dirname $type.conf]
+    set port [get_instance_attrib $type $id port]
+
+    # Execute the instance with its old setup and append the new pid
+    # file for cleanup.
+    if {$type eq "redis"} {
+        set prgname redis-server
+    } else {
+        set prgname redis-sentinel
+    }
+    set pid [exec ../../src/${prgname} $cfgfile &]
+    set_instance_attrib $type $id pid $pid
+    lappend ::pids $pid
+
+    # Check that the instance is running
+    if {[server_is_up 127.0.0.1 $port 100] == 0} {
+        abort_sentinel_test "Problems starting $type #$j: ping timeout"
+    }
+
+    # Connect with it with a fresh link
+    set_instance_attrib $type $id link [redis 127.0.0.1 $port]
+}
+
+if {[catch main e]} {
+    puts $::errorInfo
+    cleanup
+}

tests/support/server.tcl

@@ -79,7 +79,7 @@ proc is_alive config {
 proc ping_server {host port} {
     set retval 0
     if {[catch {
-        set fd [socket $::host $::port]
+        set fd [socket $host $port]
         fconfigure $fd -translation binary
         puts $fd "PING\r\n"
         flush $fd
@@ -101,6 +101,22 @@ proc ping_server {host port} {
     return $retval
 }
 
+# Return 1 if the server at the specified addr is reachable by PING, otherwise
+# returns 0. Performs a try every 50 milliseconds for the specified number
+# of retries.
+proc server_is_up {host port retrynum} {
+    after 10 ;# Use a small delay to make likely a first-try success.
+    set retval 0
+    while {[incr retrynum -1]} {
+        if {[catch {ping_server $host $port} ping]} {
+            set ping 0
+        }
+        if {$ping} {return 1}
+        after 50
+    }
+    return 0
+}
+
 # doesn't really belong here, but highly coupled to code in start_server
 proc tags {tags code} {
     set ::tags [concat $::tags $tags]
@@ -191,23 +207,13 @@ proc start_server {options {code undefined}} {
     # check that the server actually started
     # ugly but tries to be as fast as possible...
     if {$::valgrind} {set retrynum 1000} else {set retrynum 100}
-    set serverisup 0
 
     if {$::verbose} {
         puts -nonewline "=== ($tags) Starting server ${::host}:${::port} "
     }
 
-    after 10
     if {$code ne "undefined"} {
-        while {[incr retrynum -1]} {
-            catch {
-                if {[ping_server $::host $::port]} {
-                    set serverisup 1
-                }
-            }
-            if {$serverisup} break
-            after 50
-        }
+        set serverisup [server_is_up $::host $::port $retrynum]
     } else {
         set serverisup 1
     }

tests/support/test.tcl

@@ -53,41 +53,17 @@ proc assert_type {type key} {
 # executed.
 proc wait_for_condition {maxtries delay e _else_ elsescript} {
     while {[incr maxtries -1] >= 0} {
-        if {[uplevel 1 [list expr $e]]} break
+        set errcode [catch {uplevel 1 [list expr $e]} result]
+        if {$errcode == 0} {
+            if {$result} break
+        } else {
+            return -code $errcode $result
+        }
         after $delay
     }
     if {$maxtries == -1} {
-        uplevel 1 $elsescript
-    }
-}
-
-# Test if TERM looks like to support colors
-proc color_term {} {
-    expr {[info exists ::env(TERM)] && [string match *xterm* $::env(TERM)]}
-}
-
-proc colorstr {color str} {
-    if {[color_term]} {
-        set b 0
-        if {[string range $color 0 4] eq {bold-}} {
-            set b 1
-            set color [string range $color 5 end]
-        }
-        switch $color {
-            red {set colorcode {31}}
-            green {set colorcode {32}}
-            yellow {set colorcode {33}}
-            blue {set colorcode {34}}
-            magenta {set colorcode {35}}
-            cyan {set colorcode {36}}
-            white {set colorcode {37}}
-            default {set colorcode {37}}
-        }
-        if {$colorcode ne {}} {
-            return "\033\[$b;${colorcode};40m$str\033\[0m"
-        }
-    } else {
-        return $str
+        set errcode [catch [uplevel 1 $elsescript] result]
+        return -code $errcode $result
     }
 }
 

tests/support/util.tcl

@@ -312,3 +312,48 @@ proc csvstring s {
 proc roundFloat f {
     format "%.10g" $f
 }
+
+proc find_available_port start {
+    for {set j $start} {$j < $start+1024} {incr j} {
+        if {[catch {
+            set fd [socket 127.0.0.1 $j]
+        }]} {
+            return $j
+        } else {
+            close $fd
+        }
+    }
+    if {$j == $start+1024} {
+        error "Can't find a non busy port in the $start-[expr {$start+1023}] range."
+    }
+}
+
+# Test if TERM looks like to support colors
+proc color_term {} {
+    expr {[info exists ::env(TERM)] && [string match *xterm* $::env(TERM)]}
+}
+
+proc colorstr {color str} {
+    if {[color_term]} {
+        set b 0
+        if {[string range $color 0 4] eq {bold-}} {
+            set b 1
+            set color [string range $color 5 end]
+        }
+        switch $color {
+            red {set colorcode {31}}
+            green {set colorcode {32}}
+            yellow {set colorcode {33}}
+            blue {set colorcode {34}}
+            magenta {set colorcode {35}}
+            cyan {set colorcode {36}}
+            white {set colorcode {37}}
+            default {set colorcode {37}}
+        }
+        if {$colorcode ne {}} {
+            return "\033\[$b;${colorcode};40m$str\033\[0m"
+        }
+    } else {
+        return $str
+    }
+}

tests/test_helper.tcl

@@ -164,21 +164,6 @@ proc cleanup {} {
     if {!$::quiet} {puts "OK"}
 }
 
-proc find_available_port start {
-    for {set j $start} {$j < $start+1024} {incr j} {
-        if {[catch {
-            set fd [socket 127.0.0.1 $j]
-        }]} {
-            return $j
-        } else {
-            close $fd
-        }
-    }
-    if {$j == $start+1024} {
-        error "Can't find a non busy port in the $start-[expr {$start+1023}] range."
-    }
-}
-
 proc test_server_main {} {
     cleanup
     set tclsh [info nameofexecutable]

tests/unit/bitops.tcl

@@ -52,7 +52,7 @@ start_server {tags {"bitops"}} {
         }
     }
 
-    test {BITCOUNT fuzzing} {
+    test {BITCOUNT fuzzing without start/end} {
         for {set j 0} {$j < 100} {incr j} {
             set str [randstring 0 3000]
             r set str $str
@@ -60,6 +60,20 @@ start_server {tags {"bitops"}} {
         }
     }
 
+    test {BITCOUNT fuzzing with start/end} {
+        for {set j 0} {$j < 100} {incr j} {
+            set str [randstring 0 3000]
+            r set str $str
+            set l [string length $str]
+            set start [randomInt $l]
+            set end [randomInt $l]
+            if {$start > $end} {
+                lassign [list $end $start] start end
+            }
+            assert {[r bitcount str $start $end] == [count_bits [string range $str $start $end]]}
+        }
+    }
+
     test {BITCOUNT with start, end} {
         r set s "foobar"
         assert_equal [r bitcount s 0 -1] [count_bits "foobar"]
@@ -84,6 +98,18 @@ start_server {tags {"bitops"}} {
         }
     } {1}
 
+    test {BITCOUNT misaligned prefix} {
+        r del str
+        r set str ab
+        r bitcount str 1 -1
+    } {3}
+
+    test {BITCOUNT misaligned prefix + full words + remainder} {
+        r del str
+        r set str __PPxxxxxxxxxxxxxxxxRR__
+        r bitcount str 2 -3
+    } {74}
+
     test {BITOP NOT (empty string)} {
         r set s ""
         r bitop not dest s
@@ -177,4 +203,139 @@ start_server {tags {"bitops"}} {
         r set a "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
         r bitop or x a b
     } {32}
+
+    test {BITPOS bit=0 with empty key returns 0} {
+        r del str
+        r bitpos str 0
+    } {0}
+
+    test {BITPOS bit=1 with empty key returns -1} {
+        r del str
+        r bitpos str 1
+    } {-1}
+
+    test {BITPOS bit=0 with string less than 1 word works} {
+        r set str "\xff\xf0\x00"
+        r bitpos str 0
+    } {12}
+
+    test {BITPOS bit=1 with string less than 1 word works} {
+        r set str "\x00\x0f\x00"
+        r bitpos str 1
+    } {12}
+
+    test {BITPOS bit=0 starting at unaligned address} {
+        r set str "\xff\xf0\x00"
+        r bitpos str 0 1
+    } {12}
+
+    test {BITPOS bit=1 starting at unaligned address} {
+        r set str "\x00\x0f\xff"
+        r bitpos str 1 1
+    } {12}
+
+    test {BITPOS bit=0 unaligned+full word+reminder} {
+        r del str
+        r set str "\xff\xff\xff" ; # Prefix
+        # Followed by two (or four in 32 bit systems) full words
+        r append str "\xff\xff\xff\xff\xff\xff\xff\xff"
+        r append str "\xff\xff\xff\xff\xff\xff\xff\xff"
+        r append str "\xff\xff\xff\xff\xff\xff\xff\xff"
+        # First zero bit.
+        r append str "\x0f"
+        assert {[r bitpos str 0] == 216}
+        assert {[r bitpos str 0 1] == 216}
+        assert {[r bitpos str 0 2] == 216}
+        assert {[r bitpos str 0 3] == 216}
+        assert {[r bitpos str 0 4] == 216}
+        assert {[r bitpos str 0 5] == 216}
+        assert {[r bitpos str 0 6] == 216}
+        assert {[r bitpos str 0 7] == 216}
+        assert {[r bitpos str 0 8] == 216}
+    }
+
+    test {BITPOS bit=1 unaligned+full word+reminder} {
+        r del str
+        r set str "\x00\x00\x00" ; # Prefix
+        # Followed by two (or four in 32 bit systems) full words
+        r append str "\x00\x00\x00\x00\x00\x00\x00\x00"
+        r append str "\x00\x00\x00\x00\x00\x00\x00\x00"
+        r append str "\x00\x00\x00\x00\x00\x00\x00\x00"
+        # First zero bit.
+        r append str "\xf0"
+        assert {[r bitpos str 1] == 216}
+        assert {[r bitpos str 1 1] == 216}
+        assert {[r bitpos str 1 2] == 216}
+        assert {[r bitpos str 1 3] == 216}
+        assert {[r bitpos str 1 4] == 216}
+        assert {[r bitpos str 1 5] == 216}
+        assert {[r bitpos str 1 6] == 216}
+        assert {[r bitpos str 1 7] == 216}
+        assert {[r bitpos str 1 8] == 216}
+    }
+
+    test {BITPOS bit=1 returns -1 if string is all 0 bits} {
+        r set str ""
+        for {set j 0} {$j < 20} {incr j} {
+            assert {[r bitpos str 1] == -1}
+            r append str "\x00"
+        }
+    }
+
+    test {BITPOS bit=0 works with intervals} {
+        r set str "\x00\xff\x00"
+        assert {[r bitpos str 0 0 -1] == 0}
+        assert {[r bitpos str 0 1 -1] == 16}
+        assert {[r bitpos str 0 2 -1] == 16}
+        assert {[r bitpos str 0 2 200] == 16}
+        assert {[r bitpos str 0 1 1] == -1}
+    }
+
+    test {BITPOS bit=1 works with intervals} {
+        r set str "\x00\xff\x00"
+        assert {[r bitpos str 1 0 -1] == 8}
+        assert {[r bitpos str 1 1 -1] == 8}
+        assert {[r bitpos str 1 2 -1] == -1}
+        assert {[r bitpos str 1 2 200] == -1}
+        assert {[r bitpos str 1 1 1] == 8}
+    }
+
+    test {BITPOS bit=0 changes behavior if end is given} {
+        r set str "\xff\xff\xff"
+        assert {[r bitpos str 0] == 24}
+        assert {[r bitpos str 0 0] == 24}
+        assert {[r bitpos str 0 0 -1] == -1}
+    }
+
+    test {BITPOS bit=1 fuzzy testing using SETBIT} {
+        r del str
+        set max 524288; # 64k
+        set first_one_pos -1
+        for {set j 0} {$j < 1000} {incr j} {
+            assert {[r bitpos str 1] == $first_one_pos}
+            set pos [randomInt $max]
+            r setbit str $pos 1
+            if {$first_one_pos == -1 || $first_one_pos > $pos} {
+                # Update the position of the first 1 bit in the array
+                # if the bit we set is on the left of the previous one.
+                set first_one_pos $pos
+            }
+        }
+    }
+
+    test {BITPOS bit=0 fuzzy testing using SETBIT} {
+        set max 524288; # 64k
+        set first_zero_pos $max
+        r set str [string repeat "\xff" [expr $max/8]]
+        for {set j 0} {$j < 1000} {incr j} {
+            assert {[r bitpos str 0] == $first_zero_pos}
+            set pos [randomInt $max]
+            r setbit str $pos 0
+            if {$first_zero_pos > $pos} {
+                # Update the position of the first 0 bit in the array
+                # if the bit we clear is on the left of the previous one.
+                set first_zero_pos $pos
+            }
+        }
+    }
 }

tests/unit/scripting.tcl

@@ -417,5 +417,17 @@ start_server {tags {"scripting repl"}} {
             }
             set res
         } {a 1}
+
+        test {EVALSHA replication when first call is readonly} {
+            r del x
+            r eval {if tonumber(KEYS[1]) > 0 then redis.call('incr', 'x') end} 1 0
+            r evalsha 38fe3ddf5284a1d48f37f824b4c4e826879f3cb9 1 0
+            r evalsha 38fe3ddf5284a1d48f37f824b4c4e826879f3cb9 1 1
+            wait_for_condition 50 100 {
+                [r -1 get x] eq {1}
+            } else {
+                fail "Expected 1 in x, but value is '[r -1 get x]'"
+            }
+        }
     }
 }