Fix typos in GetContextFromIO API declaration.

It was noted by @dvirsky that it is not possible to use string functions
when writing the AOF file. This sometimes is critical since the command
rewriting may need to be built in the context of the AOF callback, and
without access to the context, and the limited types that the AOF
production functions will accept, this can be an issue.

Moreover there are other needs that we can't anticipate regarding the
ability to use Redis Modules APIs using the context in order to build
representations to emit AOF / RDB.

Because of this a new API was added that allows the user to get a
temporary context from the IO context. The context is auto released
if obtained when the RDB / AOF callback returns.

Calling multiple time the function to get the context, always returns
the same one, since it is invalid to have more than a single context.

.gitignore

@@ -27,3 +27,4 @@ deps/lua/src/liblua.a
 .make-*
 .prerequisites
 *.dSYM
+Makefile.dep

CONTRIBUTING

@@ -12,7 +12,7 @@ each source file that you contribute.
 
   PLEASE DO NOT POST GENERAL QUESTIONS that are not about bugs or suspected
   bugs in the Github issues system. We'll be very happy to help you and provide
-  all the support Reddit sub:
+  all the support at the Reddit sub:
 
       http://reddit.com/r/redis
 
@@ -24,7 +24,7 @@ each source file that you contribute.
 
 1. If it is a major feature or a semantical change, please post it as a new submission in r/redis on Reddit at http://reddit.com/r/redis. Try to be passionate about why the feature is needed, make users upvote your proposal to gain traction and so forth. Read feedbacks about the community. But in this first step **please don't write code yet**.
 
-2. If in step 1 you get an acknowledge from the project leaders, use the
+2. If in step 1 you get an acknowledgment from the project leaders, use the
    following procedure to submit a patch:
 
     a. Fork Redis on github ( http://help.github.com/fork-a-repo/ )

README.md

@@ -39,7 +39,7 @@ You can run a 32 bit Redis binary using:
 
     % make 32bit
 
-After building Redis is a good idea to test it, using:
+After building Redis, it is a good idea to test it using:
 
     % make test
 
@@ -47,8 +47,8 @@ Fixing build problems with dependencies or cached build options
 ---------
 
 Redis has some dependencies which are included into the `deps` directory.
-`make` does not rebuild dependencies automatically, even if something in the
-source code of dependencies is changes.
+`make` does not automatically rebuild dependencies even if something in
+the source code of dependencies changes.
 
 When you update the source code with `git pull` or when code inside the
 dependencies tree is modified in any other way, make sure to use the following
@@ -109,14 +109,14 @@ To run Redis with the default configuration just type:
 
     % cd src
     % ./redis-server
-    
+
 If you want to provide your redis.conf, you have to run it using an additional
 parameter (the path of the configuration file):
 
     % cd src
     % ./redis-server /path/to/redis.conf
 
-It is possible to alter the Redis configuration passing parameters directly
+It is possible to alter the Redis configuration by passing parameters directly
 as options using the command line. Examples:
 
     % ./redis-server --port 9999 --slaveof 127.0.0.1 6379
@@ -174,7 +174,7 @@ You'll be able to stop and start Redis using the script named
 `/etc/init.d/redis_<portnumber>`, for instance `/etc/init.d/redis_6379`.
 
 Code contributions
----
+-----------------
 
 Note: by contributing code to the Redis project in any form, including sending
 a pull request via Github, a code fragment or patch via private email or
@@ -196,8 +196,8 @@ or you just untarred the Redis distribution tar ball. In both the cases
 you are basically one step away from the source code, so here we explain
 the Redis source code layout, what is in each file as a general idea, the
 most important functions and structures inside the Redis server and so forth.
-We keep all the discussion at an high level without digging into the details
-since this document would be huge otherwise, and our code base changes
+We keep all the discussion at a high level without digging into the details
+since this document would be huge otherwise and our code base changes
 continuously, but a general idea should be a good starting point to
 understand more. Moreover most of the code is heavily commented and easy
 to follow.
@@ -206,17 +206,17 @@ Source code layout
 ---
 
 The Redis root directory just contains this README, the Makefile which
-actually calls the real Makefile inside the `src` directory, an example
-configuration for Redis and Sentinel. Finally you can find a few shell
+calls the real Makefile inside the `src` directory and an example
+configuration for Redis and Sentinel. You can find a few shell
 scripts that are used in order to execute the Redis, Redis Cluster and
 Redis Sentinel unit tests, which are implemented inside the `tests`
 directory.
 
-Inside the root directory the are the following important directories:
+Inside the root are the following important directories:
 
 * `src`: contains the Redis implementation, written in C.
 * `tests`: contains the unit tests, implemented in Tcl.
-* `deps`: contains libraries Redis uses. Everything needed to compile Redis is inside this directory, your system needs to provide just the `libc`, a POSIX compatible interface, and a C compiler. Notably `deps` contains a copy of `jemalloc`, which is the default allocator of Redis under Linux. Note that under `deps` there are also things which started with the Redis project, but for which the main repository is not `anitrez/redis`. an exception to this rule is `deps/geohash-int` which is the low level geocoding library used by Redis: it originated from a different project, but at this point it diverged so much that it is developed as a separated entity directly inside the Redis repository.
+* `deps`: contains libraries Redis uses. Everything needed to compile Redis is inside this directory; your system just needs to provide `libc`, a POSIX compatible interface and a C compiler. Notably `deps` contains a copy of `jemalloc`, which is the default allocator of Redis under Linux. Note that under `deps` there are also things which started with the Redis project, but for which the main repository is not `anitrez/redis`. An exception to this rule is `deps/geohash-int` which is the low level geocoding library used by Redis: it originated from a different project, but at this point it diverged so much that it is developed as a separated entity directly inside the Redis repository.
 
 There are a few more directories but they are not very important for our goals
 here. We'll focus mostly on `src`, where the Redis implementation is contained,
@@ -225,34 +225,34 @@ exposed is the logical one to follow in order to disclose different layers
 of complexity incrementally.
 
 Note: lately Redis was refactored quite a bit. Function names and file
-names changed, so you may find that this documentation reflects the
+names have been changed, so you may find that this documentation reflects the
 `unstable` branch more closely. For instance in Redis 3.0 the `server.c`
-and `server.h` files were renamed `redis.c` and `redis.h`. However the overall
+and `server.h` files were named to `redis.c` and `redis.h`. However the overall
 structure is the same. Keep in mind that all the new developments and pull
 requests should be performed against the `unstable` branch.
 
 server.h
 ---
 
-The simplest way to understand how a program works, is to understand the
+The simplest way to understand how a program works is to understand the
 data structures it uses. So we'll start from the main header file of
 Redis, which is `server.h`.
 
 All the server configuration and in general all the shared state is
 defined in a global structure called `server`, of type `struct redisServer`.
-A few important fields in this structure:
+A few important fields in this structure are:
 
 * `server.db` is an array of Redis databases, where data is stored.
 * `server.commands` is the command table.
 * `server.clients` is a linked list of clients connected to the server.
 * `server.master` is a special client, the master, if the instance is a slave.
 
-There are tons of other fields, most fields are commented directly inside
+There are tons of other fields. Most fields are commented directly inside
 the structure definition.
 
 Another important Redis data structure is the one defining a client.
 In the past it was called `redisClient`, now just `client`. The structure
-has many fields, here we'll show just the main ones:
+has many fields, here we'll just show the main ones:
 
     struct client {
         int fd;
@@ -270,7 +270,7 @@ The client structure defines a *connected client*:
 
 * The `fd` field is the client socket file descriptor.
 * `argc` and `argv` are populated with the command the client is executing, so that functions implementing a given Redis command can read the arguments.
-* `querybuf` accumulates the requests from the client, which are parsed by the Redis server according to the Redis protocol, and executed calling the implementations of the commands the client is executing.
+* `querybuf` accumulates the requests from the client, which are parsed by the Redis server according to the Redis protocol and executed by calling the implementations of the commands the client is executing.
 * `reply` and `buf` are dynamic and static buffers that accumulate the replies the server sends to the client. These buffers are incrementally written to the socket as soon as the file descriptor is writable.
 
 As you can see in the client structure above, arguments in a command
@@ -288,16 +288,16 @@ structure, which defines a *Redis object*:
 Basically this structure can represent all the basic Redis data types like
 strings, lists, sets, sorted sets and so forth. The interesting thing is that
 it has a `type` field, so that it is possible to know what type a given
-object is, and a `refcount`, so that the same object can be referenced
+object has, and a `refcount`, so that the same object can be referenced
 in multiple places without allocating it multiple times. Finally the `ptr`
-field points to the actual representation of the object, that may vary
+field points to the actual representation of the object, which might vary
 even for the same type, depending on the `encoding` used.
 
 Redis objects are used extensively in the Redis internals, however in order
 to avoid the overhead of indirect accesses, recently in many places
 we just use plain dynamic strings not wrapped inside a Redis object.
 
-sever.c
+server.c
 ---
 
 This is the entry point of the Redis server, where the `main()` function
@@ -306,7 +306,7 @@ the Redis server.
 
 * `initServerConfig()` setups the default values of the `server` structure.
 * `initServer()` allocates the data structures needed to operate, setup the listening socket, and so forth.
-* `aeMain()` enters the event loop listening for new connections.
+* `aeMain()` starts the event loop which listens for new connections.
 
 There are two special functions called periodically by the event loop:
 
@@ -328,7 +328,7 @@ This file defines all the I/O functions with clients, masters and slaves
 
 * `createClient()` allocates and initializes a new client.
 * the `addReply*()` family of functions are used by commands implementations in order to append data to the client structure, that will be transmitted to the client as a reply for a given command executed.
-* `writeToClient()` transmits the data pending in the output buffers to the client, and is called by the *writable event handler* `sendReplyToClient()`.
+* `writeToClient()` transmits the data pending in the output buffers to the client and is called by the *writable event handler* `sendReplyToClient()`.
 * `readQueryFromClient()` is the *readable event handler* and accumulates data from read from the client into the query buffer.
 * `processInputBuffer()` is the entry point in order to parse the client query buffer according to the Redis protocol. Once commands are ready to be processed, it calls `processCommand()` which is defined inside `server.c` in order to actually execute the command.
 * `freeClient()` deallocates, disconnects and removes a client.
@@ -439,9 +439,8 @@ There are tons of commands implementations inside th Redis source code
 that can serve as examples of actual commands implementations. To write
 a few toy commands can be a good exercise to familiarize with the code base.
 
-There are also many other files not described here,  but it is useless to
-cover everything, we want just to help you with the first steps,
-eventually you'll find your way inside the Redis code base :-)
+There are also many other files not described here, but it is useless to
+cover everything. We want to just help you with the first steps.
+Eventually you'll find your way inside the Redis code base :-)
 
 Enjoy!
-

deps/Makefile

@@ -36,7 +36,6 @@ distclean:
 	-(cd hiredis && $(MAKE) clean) > /dev/null || true
 	-(cd linenoise && $(MAKE) clean) > /dev/null || true
 	-(cd lua && $(MAKE) clean) > /dev/null || true
-	-(cd geohash-int && $(MAKE) clean) > /dev/null || true
 	-(cd jemalloc && [ -f Makefile ] && $(MAKE) distclean) > /dev/null || true
 	-(rm -f .make-*)
 
@@ -78,13 +77,7 @@ JEMALLOC_LDFLAGS= $(LDFLAGS)
 
 jemalloc: .make-prerequisites
 	@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
-	cd jemalloc && ./configure --with-jemalloc-prefix=je_ --enable-cc-silence CFLAGS="$(JEMALLOC_CFLAGS)" LDFLAGS="$(JEMALLOC_LDFLAGS)"
+	cd jemalloc && ./configure --with-lg-quantum=3 --with-jemalloc-prefix=je_ --enable-cc-silence CFLAGS="$(JEMALLOC_CFLAGS)" LDFLAGS="$(JEMALLOC_LDFLAGS)"
 	cd jemalloc && $(MAKE) CFLAGS="$(JEMALLOC_CFLAGS)" LDFLAGS="$(JEMALLOC_LDFLAGS)" lib/libjemalloc.a
 
 .PHONY: jemalloc
-
-geohash-int: .make-prerequisites
-	@printf '%b %b\n' $(MAKECOLOR)MAKE$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR)
-	cd geohash-int && $(MAKE)
-
-.PHONY: geohash-int

deps/geohash-int/Makefile

@@ -1,23 +0,0 @@
-STD=
-WARN= -Wall
-OPT= -O2
-
-R_CFLAGS= $(STD) $(WARN) $(OPT) $(DEBUG) $(CFLAGS)
-R_LDFLAGS= $(LDFLAGS)
-DEBUG= -g
-
-R_CC=$(CC) $(R_CFLAGS)
-R_LD=$(CC) $(R_LDFLAGS)
-
-all: geohash.o geohash_helper.o
-
-.PHONY: all
-
-geohash.o: geohash.h geohash.c
-geohash_helper.o: geohash.h geohash_helper.h geohash_helper.c
-
-.c.o:
-	$(R_CC) -c $<
-
-clean:
-	rm -f *.o

deps/linenoise/README.markdown

@@ -6,6 +6,7 @@ MongoDB, and Android.
 * Single and multi line editing mode with the usual key bindings implemented.
 * History handling.
 * Completion.
+* Hints (suggestions at the right of the prompt as you type).
 * About 1,100 lines of BSD license source code.
 * Only uses a subset of VT100 escapes (ANSI.SYS compatible).
 
@@ -20,7 +21,7 @@ So what usually happens is either:
  
 The result is a pollution of binaries without line editing support.
 
-So I spent more or less two hours doing a reality check resulting in this little library: is it *really* needed for a line editing library to be 20k lines of code? Apparently not, it is possibe to get a very small, zero configuration, trivial to embed library, that solves the problem. Smaller programs will just include this, supporing line editing out of the box. Larger programs may use this little library or just checking with configure if readline/libedit is available and resorting to linenoise if not.
+So I spent more or less two hours doing a reality check resulting in this little library: is it *really* needed for a line editing library to be 20k lines of code? Apparently not, it is possibe to get a very small, zero configuration, trivial to embed library, that solves the problem. Smaller programs will just include this, supporing line editing out of the box. Larger programs may use this little library or just checking with configure if readline/libedit is available and resorting to Linenoise if not.
 
 ## Terminals, in 2010.
 
@@ -41,12 +42,183 @@ The library is currently about 1100 lines of code. In order to use it in your pr
  * IBM AIX 6.1
  * FreeBSD xterm ($TERM = xterm)
  * ANSI.SYS
+ * Emacs comint mode ($TERM = dumb)
 
 Please test it everywhere you can and report back!
 
 ## Let's push this forward!
 
-Patches should be provided in the respect of linenoise sensibility for small
+Patches should be provided in the respect of Linenoise sensibility for small
 easy to understand code.
 
 Send feedbacks to antirez at gmail
+
+# The API
+
+Linenoise is very easy to use, and reading the example shipped with the
+library should get you up to speed ASAP. Here is a list of API calls
+and how to use them.
+
+    char *linenoise(const char *prompt);
+
+This is the main Linenoise call: it shows the user a prompt with line editing
+and history capabilities. The prompt you specify is used as a prompt, that is,
+it will be printed to the left of the cursor. The library returns a buffer
+with the line composed by the user, or NULL on end of file or when there
+is an out of memory condition.
+
+When a tty is detected (the user is actually typing into a terminal session)
+the maximum editable line length is `LINENOISE_MAX_LINE`. When instead the
+standard input is not a tty, which happens every time you redirect a file
+to a program, or use it in an Unix pipeline, there are no limits to the
+length of the line that can be returned.
+
+The returned line should be freed with the `free()` standard system call.
+However sometimes it could happen that your program uses a different dynamic
+allocation library, so you may also used `linenoiseFree` to make sure the
+line is freed with the same allocator it was created.
+
+The canonical loop used by a program using Linenoise will be something like
+this:
+
+    while((line = linenoise("hello> ")) != NULL) {
+        printf("You wrote: %s\n", line);
+        linenoiseFree(line); /* Or just free(line) if you use libc malloc. */
+    }
+
+## Single line VS multi line editing
+
+By default, Linenoise uses single line editing, that is, a single row on the
+screen will be used, and as the user types more, the text will scroll towards
+left to make room. This works if your program is one where the user is
+unlikely to write a lot of text, otherwise multi line editing, where multiple
+screens rows are used, can be a lot more comfortable.
+
+In order to enable multi line editing use the following API call:
+
+    linenoiseSetMultiLine(1);
+
+You can disable it using `0` as argument.
+
+## History
+
+Linenoise supporst history, so that the user does not have to retype
+again and again the same things, but can use the down and up arrows in order
+to search and re-edit already inserted lines of text.
+
+The followings are the history API calls:
+
+    int linenoiseHistoryAdd(const char *line);
+    int linenoiseHistorySetMaxLen(int len);
+    int linenoiseHistorySave(const char *filename);
+    int linenoiseHistoryLoad(const char *filename);
+
+Use `linenoiseHistoryAdd` every time you want to add a new element
+to the top of the history (it will be the first the user will see when
+using the up arrow).
+
+Note that for history to work, you have to set a length for the history
+(which is zero by default, so history will be disabled if you don't set
+a proper one). This is accomplished using the `linenoiseHistorySetMaxLen`
+function.
+
+Linenoise has direct support for persisting the history into an history
+file. The functions `linenoiseHistorySave` and `linenoiseHistoryLoad` do
+just that. Both functions return -1 on error and 0 on success.
+
+## Completion
+
+Linenoise supports completion, which is the ability to complete the user
+input when she or he presses the `<TAB>` key.
+
+In order to use completion, you need to register a completion callback, which
+is called every time the user presses `<TAB>`. Your callback will return a
+list of items that are completions for the current string.
+
+The following is an example of registering a completion callback:
+
+    linenoiseSetCompletionCallback(completion);
+
+The completion must be a function returning `void` and getting as input
+a `const char` pointer, which is the line the user has typed so far, and
+a `linenoiseCompletions` object pointer, which is used as argument of
+`linenoiseAddCompletion` in order to add completions inside the callback.
+An example will make it more clear:
+
+    void completion(const char *buf, linenoiseCompletions *lc) {
+        if (buf[0] == 'h') {
+            linenoiseAddCompletion(lc,"hello");
+            linenoiseAddCompletion(lc,"hello there");
+        }
+    }
+
+Basically in your completion callback, you inspect the input, and return
+a list of items that are good completions by using `linenoiseAddCompletion`.
+
+If you want to test the completion feature, compile the example program
+with `make`, run it, type `h` and press `<TAB>`.
+
+## Hints
+
+Linenoise has a feature called *hints* which is very useful when you
+use Linenoise in order to implement a REPL (Read Eval Print Loop) for
+a program that accepts commands and arguments, but may also be useful in
+other conditions.
+
+The feature shows, on the right of the cursor, as the user types, hints that
+may be useful. The hints can be displayed using a different color compared
+to the color the user is typing, and can also be bold.
+
+For example as the user starts to type `"git remote add"`, with hints it's
+possible to show on the right of the prompt a string `<name> <url>`.
+
+The feature works similarly to the history feature, using a callback.
+To register the callback we use:
+
+    linenoiseSetHintsCallback(hints);
+
+The callback itself is implemented like this:
+
+    char *hints(const char *buf, int *color, int *bold) {
+        if (!strcasecmp(buf,"git remote add")) {
+            *color = 35;
+            *bold = 0;
+            return " <name> <url>";
+        }
+        return NULL;
+    }
+
+The callback function returns the string that should be displayed or NULL
+if no hint is available for the text the user currently typed. The returned
+string will be trimmed as needed depending on the number of columns available
+on the screen.
+
+It is possible to return a string allocated in dynamic way, by also registering
+a function to deallocate the hint string once used:
+
+    void linenoiseSetFreeHintsCallback(linenoiseFreeHintsCallback *);
+
+The free hint callback will just receive the pointer and free the string
+as needed (depending on how the hits callback allocated it).
+
+As you can see in the example above, a `color` (in xterm color terminal codes)
+can be provided together with a `bold` attribute. If no color is set, the
+current terminal foreground color is used. If no bold attribute is set,
+non-bold text is printed.
+
+Color codes are:
+
+    red = 31
+    green = 32
+    yellow = 33
+    blue = 34
+    magenta = 35
+    cyan = 36
+    white = 37;
+
+## Screen handling
+
+Sometimes you may want to clear the screen as a result of something the
+user typed. You can do this by calling the following function:
+
+    void linenoiseClearScreen(void);

deps/linenoise/example.c

@@ -11,6 +11,15 @@ void completion(const char *buf, linenoiseCompletions *lc) {
     }
 }
 
+char *hints(const char *buf, int *color, int *bold) {
+    if (!strcasecmp(buf,"hello")) {
+        *color = 35;
+        *bold = 0;
+        return " World";
+    }
+    return NULL;
+}
+
 int main(int argc, char **argv) {
     char *line;
     char *prgname = argv[0];
@@ -34,6 +43,7 @@ int main(int argc, char **argv) {
     /* Set the completion callback. This will be called every time the
      * user uses the <tab> key. */
     linenoiseSetCompletionCallback(completion);
+    linenoiseSetHintsCallback(hints);
 
     /* Load history from file. The history file is just a plain text file
      * where entries are separated by newlines. */

deps/linenoise/linenoise.c

@@ -10,7 +10,7 @@
  *
  * ------------------------------------------------------------------------
  *
- * Copyright (c) 2010-2013, Salvatore Sanfilippo <antirez at gmail dot com>
+ * Copyright (c) 2010-2016, Salvatore Sanfilippo <antirez at gmail dot com>
  * Copyright (c) 2010-2013, Pieter Noordhuis <pcnoordhuis at gmail dot com>
  *
  * All rights reserved.
@@ -111,6 +111,7 @@
 #include <string.h>
 #include <stdlib.h>
 #include <ctype.h>
+#include <sys/stat.h>
 #include <sys/types.h>
 #include <sys/ioctl.h>
 #include <unistd.h>
@@ -120,6 +121,8 @@
 #define LINENOISE_MAX_LINE 4096
 static char *unsupported_term[] = {"dumb","cons25","emacs",NULL};
 static linenoiseCompletionCallback *completionCallback = NULL;
+static linenoiseHintsCallback *hintsCallback = NULL;
+static linenoiseFreeHintsCallback *freeHintsCallback = NULL;
 
 static struct termios orig_termios; /* In order to restore at exit.*/
 static int rawmode = 0; /* For atexit() function to check if restore is needed*/
@@ -407,6 +410,18 @@ void linenoiseSetCompletionCallback(linenoiseCompletionCallback *fn) {
     completionCallback = fn;
 }
 
+/* Register a hits function to be called to show hits to the user at the
+ * right of the prompt. */
+void linenoiseSetHintsCallback(linenoiseHintsCallback *fn) {
+    hintsCallback = fn;
+}
+
+/* Register a function to free the hints returned by the hints callback
+ * registered with linenoiseSetHintsCallback(). */
+void linenoiseSetFreeHintsCallback(linenoiseFreeHintsCallback *fn) {
+    freeHintsCallback = fn;
+}
+
 /* This function is used by the callback function registered by the user
  * in order to add completion options given the input string when the
  * user typed <tab>. See the example.c source code for a very easy to
@@ -456,6 +471,30 @@ static void abFree(struct abuf *ab) {
     free(ab->b);
 }
 
+/* Helper of refreshSingleLine() and refreshMultiLine() to show hints
+ * to the right of the prompt. */
+void refreshShowHints(struct abuf *ab, struct linenoiseState *l, int plen) {
+    char seq[64];
+    if (hintsCallback && plen+l->len < l->cols) {
+        int color = -1, bold = 0;
+        char *hint = hintsCallback(l->buf,&color,&bold);
+        if (hint) {
+            int hintlen = strlen(hint);
+            int hintmaxlen = l->cols-(plen+l->len);
+            if (hintlen > hintmaxlen) hintlen = hintmaxlen;
+            if (bold == 1 && color == -1) color = 37;
+            if (color != -1 || bold != 0)
+                snprintf(seq,64,"\033[%d;%d;49m",bold,color);
+            abAppend(ab,seq,strlen(seq));
+            abAppend(ab,hint,hintlen);
+            if (color != -1 || bold != 0)
+                abAppend(ab,"\033[0m",4);
+            /* Call the function to free the hint returned. */
+            if (freeHintsCallback) freeHintsCallback(hint);
+        }
+    }
+}
+
 /* Single line low level line refresh.
  *
  * Rewrite the currently edited line accordingly to the buffer content,
@@ -485,6 +524,8 @@ static void refreshSingleLine(struct linenoiseState *l) {
     /* Write the prompt and the current buffer content */
     abAppend(&ab,l->prompt,strlen(l->prompt));
     abAppend(&ab,buf,len);
+    /* Show hits if any. */
+    refreshShowHints(&ab,l,plen);
     /* Erase to right */
     snprintf(seq,64,"\x1b[0K");
     abAppend(&ab,seq,strlen(seq));
@@ -538,6 +579,9 @@ static void refreshMultiLine(struct linenoiseState *l) {
     abAppend(&ab,l->prompt,strlen(l->prompt));
     abAppend(&ab,l->buf,l->len);
 
+    /* Show hits if any. */
+    refreshShowHints(&ab,l,plen);
+
     /* If we are at the very end of the screen with our prompt, we need to
      * emit a newline and move the prompt to the first column. */
     if (l->pos &&
@@ -598,7 +642,7 @@ int linenoiseEditInsert(struct linenoiseState *l, char c) {
             l->pos++;
             l->len++;
             l->buf[l->len] = '\0';
-            if ((!mlmode && l->plen+l->len < l->cols) /* || mlmode */) {
+            if ((!mlmode && l->plen+l->len < l->cols && !hintsCallback)) {
                 /* Avoid a full update of the line in the
                  * trivial case. */
                 if (write(l->ofd,&c,1) == -1) return -1;
@@ -772,6 +816,14 @@ static int linenoiseEdit(int stdin_fd, int stdout_fd, char *buf, size_t buflen,
             history_len--;
             free(history[history_len]);
             if (mlmode) linenoiseEditMoveEnd(&l);
+            if (hintsCallback) {
+                /* Force a refresh without hints to leave the previous
+                 * line as the user typed it after a newline. */
+                linenoiseHintsCallback *hc = hintsCallback;
+                hintsCallback = NULL;
+                refreshLine(&l);
+                hintsCallback = hc;
+            }
             return (int)l.len;
         case CTRL_C:     /* ctrl-c */
             errno = EAGAIN;
@@ -934,24 +986,50 @@ static int linenoiseRaw(char *buf, size_t buflen, const char *prompt) {
         errno = EINVAL;
         return -1;
     }
-    if (!isatty(STDIN_FILENO)) {
-        /* Not a tty: read from file / pipe. */
-        if (fgets(buf, buflen, stdin) == NULL) return -1;
-        count = strlen(buf);
-        if (count && buf[count-1] == '\n') {
-            count--;
-            buf[count] = '\0';
-        }
-    } else {
-        /* Interactive editing. */
-        if (enableRawMode(STDIN_FILENO) == -1) return -1;
-        count = linenoiseEdit(STDIN_FILENO, STDOUT_FILENO, buf, buflen, prompt);
-        disableRawMode(STDIN_FILENO);
-        printf("\n");
-    }
+
+    if (enableRawMode(STDIN_FILENO) == -1) return -1;
+    count = linenoiseEdit(STDIN_FILENO, STDOUT_FILENO, buf, buflen, prompt);
+    disableRawMode(STDIN_FILENO);
+    printf("\n");
     return count;
 }
 
+/* This function is called when linenoise() is called with the standard
+ * input file descriptor not attached to a TTY. So for example when the
+ * program using linenoise is called in pipe or with a file redirected
+ * to its standard input. In this case, we want to be able to return the
+ * line regardless of its length (by default we are limited to 4k). */
+static char *linenoiseNoTTY(void) {
+    char *line = NULL;
+    size_t len = 0, maxlen = 0;
+
+    while(1) {
+        if (len == maxlen) {
+            if (maxlen == 0) maxlen = 16;
+            maxlen *= 2;
+            char *oldval = line;
+            line = realloc(line,maxlen);
+            if (line == NULL) {
+                if (oldval) free(oldval);
+                return NULL;
+            }
+        }
+        int c = fgetc(stdin);
+        if (c == EOF || c == '\n') {
+            if (c == EOF && len == 0) {
+                free(line);
+                return NULL;
+            } else {
+                line[len] = '\0';
+                return line;
+            }
+        } else {
+            line[len] = c;
+            len++;
+        }
+    }
+}
+
 /* The high level function that is the main API of the linenoise library.
  * This function checks if the terminal has basic capabilities, just checking
  * for a blacklist of stupid terminals, and later either calls the line
@@ -961,7 +1039,11 @@ char *linenoise(const char *prompt) {
     char buf[LINENOISE_MAX_LINE];
     int count;
 
-    if (isUnsupportedTerm()) {
+    if (!isatty(STDIN_FILENO)) {
+        /* Not a tty: read from file / pipe. In this mode we don't want any
+         * limit to the line size, so we call a function to handle that. */
+        return linenoiseNoTTY();
+    } else if (isUnsupportedTerm()) {
         size_t len;
 
         printf("%s",prompt);
@@ -980,6 +1062,14 @@ char *linenoise(const char *prompt) {
     }
 }
 
+/* This is just a wrapper the user may want to call in order to make sure
+ * the linenoise returned buffer is freed with the same allocator it was
+ * created with. Useful when the main program is using an alternative
+ * allocator. */
+void linenoiseFree(void *ptr) {
+    free(ptr);
+}
+
 /* ================================ History ================================= */
 
 /* Free the history, but does not reset it. Only used when we have to
@@ -1071,10 +1161,14 @@ int linenoiseHistorySetMaxLen(int len) {
 /* Save the history in the specified file. On success 0 is returned
  * otherwise -1 is returned. */
 int linenoiseHistorySave(const char *filename) {
-    FILE *fp = fopen(filename,"w");
+    mode_t old_umask = umask(S_IXUSR|S_IRWXG|S_IRWXO);
+    FILE *fp;
     int j;
 
+    fp = fopen(filename,"w");
+    umask(old_umask);
     if (fp == NULL) return -1;
+    chmod(filename,S_IRUSR|S_IWUSR);
     for (j = 0; j < history_len; j++)
         fprintf(fp,"%s\n",history[j]);
     fclose(fp);

deps/linenoise/linenoise.h

@@ -1,12 +1,14 @@
-/* linenoise.h -- guerrilla line editing library against the idea that a
- * line editing lib needs to be 20,000 lines of C code.
+/* linenoise.h -- VERSION 1.0
+ *
+ * Guerrilla line editing library against the idea that a line editing lib
+ * needs to be 20,000 lines of C code.
  *
  * See linenoise.c for more information.
  *
  * ------------------------------------------------------------------------
  *
- * Copyright (c) 2010, Salvatore Sanfilippo <antirez at gmail dot com>
- * Copyright (c) 2010, Pieter Noordhuis <pcnoordhuis at gmail dot com>
+ * Copyright (c) 2010-2014, Salvatore Sanfilippo <antirez at gmail dot com>
+ * Copyright (c) 2010-2013, Pieter Noordhuis <pcnoordhuis at gmail dot com>
  *
  * All rights reserved.
  *
@@ -47,10 +49,15 @@ typedef struct linenoiseCompletions {
 } linenoiseCompletions;
 
 typedef void(linenoiseCompletionCallback)(const char *, linenoiseCompletions *);
+typedef char*(linenoiseHintsCallback)(const char *, int *color, int *bold);
+typedef void(linenoiseFreeHintsCallback)(void *);
 void linenoiseSetCompletionCallback(linenoiseCompletionCallback *);
+void linenoiseSetHintsCallback(linenoiseHintsCallback *);
+void linenoiseSetFreeHintsCallback(linenoiseFreeHintsCallback *);
 void linenoiseAddCompletion(linenoiseCompletions *, const char *);
 
 char *linenoise(const char *prompt);
+void linenoiseFree(void *ptr);
 int linenoiseHistoryAdd(const char *line);
 int linenoiseHistorySetMaxLen(int len);
 int linenoiseHistorySave(const char *filename);

deps/lua/src/lua_cmsgpack.c

@@ -91,7 +91,6 @@ void memrevifle(void *ptr, size_t len) {
  * behavior.  */
 
 typedef struct mp_buf {
-    lua_State *L;
     unsigned char *b;
     size_t len, free;
 } mp_buf;
@@ -111,27 +110,26 @@ mp_buf *mp_buf_new(lua_State *L) {
     /* Old size = 0; new size = sizeof(*buf) */
     buf = (mp_buf*)mp_realloc(L, NULL, 0, sizeof(*buf));
 
-    buf->L = L;
     buf->b = NULL;
     buf->len = buf->free = 0;
     return buf;
 }
 
-void mp_buf_append(mp_buf *buf, const unsigned char *s, size_t len) {
+void mp_buf_append(lua_State *L, mp_buf *buf, const unsigned char *s, size_t len) {
     if (buf->free < len) {
-        size_t newlen = buf->len+len;
+        size_t newsize = (buf->len+len)*2;
 
-        buf->b = (unsigned char*)mp_realloc(buf->L, buf->b, buf->len, newlen*2);
-        buf->free = newlen;
+        buf->b = (unsigned char*)mp_realloc(L, buf->b, buf->len + buf->free, newsize);
+        buf->free = newsize - buf->len;
     }
     memcpy(buf->b+buf->len,s,len);
     buf->len += len;
     buf->free -= len;
 }
 
-void mp_buf_free(mp_buf *buf) {
-    mp_realloc(buf->L, buf->b, buf->len, 0); /* realloc to 0 = free */
-    mp_realloc(buf->L, buf, sizeof(*buf), 0);
+void mp_buf_free(lua_State *L, mp_buf *buf) {
+    mp_realloc(L, buf->b, buf->len + buf->free, 0); /* realloc to 0 = free */
+    mp_realloc(L, buf, sizeof(*buf), 0);
 }
 
 /* ---------------------------- String cursor ----------------------------------
@@ -173,7 +171,7 @@ void mp_cur_init(mp_cur *cursor, const unsigned char *s, size_t len) {
 
 /* ------------------------- Low level MP encoding -------------------------- */
 
-void mp_encode_bytes(mp_buf *buf, const unsigned char *s, size_t len) {
+void mp_encode_bytes(lua_State *L, mp_buf *buf, const unsigned char *s, size_t len) {
     unsigned char hdr[5];
     int hdrlen;
 
@@ -197,12 +195,12 @@ void mp_encode_bytes(mp_buf *buf, const unsigned char *s, size_t len) {
         hdr[4] = len&0xff;
         hdrlen = 5;
     }
-    mp_buf_append(buf,hdr,hdrlen);
-    mp_buf_append(buf,s,len);
+    mp_buf_append(L,buf,hdr,hdrlen);
+    mp_buf_append(L,buf,s,len);
 }
 
 /* we assume IEEE 754 internal format for single and double precision floats. */
-void mp_encode_double(mp_buf *buf, double d) {
+void mp_encode_double(lua_State *L, mp_buf *buf, double d) {
     unsigned char b[9];
     float f = d;
 
@@ -211,16 +209,16 @@ void mp_encode_double(mp_buf *buf, double d) {
         b[0] = 0xca;    /* float IEEE 754 */
         memcpy(b+1,&f,4);
         memrevifle(b+1,4);
-        mp_buf_append(buf,b,5);
+        mp_buf_append(L,buf,b,5);
     } else if (sizeof(d) == 8) {
         b[0] = 0xcb;    /* double IEEE 754 */
         memcpy(b+1,&d,8);
         memrevifle(b+1,8);
-        mp_buf_append(buf,b,9);
+        mp_buf_append(L,buf,b,9);
     }
 }
 
-void mp_encode_int(mp_buf *buf, int64_t n) {
+void mp_encode_int(lua_State *L, mp_buf *buf, int64_t n) {
     unsigned char b[9];
     int enclen;
 
@@ -289,10 +287,10 @@ void mp_encode_int(mp_buf *buf, int64_t n) {
             enclen = 9;
         }
     }
-    mp_buf_append(buf,b,enclen);
+    mp_buf_append(L,buf,b,enclen);
 }
 
-void mp_encode_array(mp_buf *buf, int64_t n) {
+void mp_encode_array(lua_State *L, mp_buf *buf, int64_t n) {
     unsigned char b[5];
     int enclen;
 
@@ -312,10 +310,10 @@ void mp_encode_array(mp_buf *buf, int64_t n) {
         b[4] = n & 0xff;
         enclen = 5;
     }
-    mp_buf_append(buf,b,enclen);
+    mp_buf_append(L,buf,b,enclen);
 }
 
-void mp_encode_map(mp_buf *buf, int64_t n) {
+void mp_encode_map(lua_State *L, mp_buf *buf, int64_t n) {
     unsigned char b[5];
     int enclen;
 
@@ -335,7 +333,7 @@ void mp_encode_map(mp_buf *buf, int64_t n) {
         b[4] = n & 0xff;
         enclen = 5;
     }
-    mp_buf_append(buf,b,enclen);
+    mp_buf_append(L,buf,b,enclen);
 }
 
 /* --------------------------- Lua types encoding --------------------------- */
@@ -345,12 +343,12 @@ void mp_encode_lua_string(lua_State *L, mp_buf *buf) {
     const char *s;
 
     s = lua_tolstring(L,-1,&len);
-    mp_encode_bytes(buf,(const unsigned char*)s,len);
+    mp_encode_bytes(L,buf,(const unsigned char*)s,len);
 }
 
 void mp_encode_lua_bool(lua_State *L, mp_buf *buf) {
     unsigned char b = lua_toboolean(L,-1) ? 0xc3 : 0xc2;
-    mp_buf_append(buf,&b,1);
+    mp_buf_append(L,buf,&b,1);
 }
 
 /* Lua 5.3 has a built in 64-bit integer type */
@@ -360,7 +358,7 @@ void mp_encode_lua_integer(lua_State *L, mp_buf *buf) {
 #else
     lua_Integer i = lua_tointeger(L,-1);
 #endif
-    mp_encode_int(buf, (int64_t)i);
+    mp_encode_int(L, buf, (int64_t)i);
 }
 
 /* Lua 5.2 and lower only has 64-bit doubles, so we need to
@@ -372,7 +370,7 @@ void mp_encode_lua_number(lua_State *L, mp_buf *buf) {
     if (IS_INT64_EQUIVALENT(n)) {
         mp_encode_lua_integer(L, buf);
     } else {
-        mp_encode_double(buf,(double)n);
+        mp_encode_double(L,buf,(double)n);
     }
 }
 
@@ -386,7 +384,7 @@ void mp_encode_lua_table_as_array(lua_State *L, mp_buf *buf, int level) {
     size_t len = lua_rawlen(L,-1), j;
 #endif
 
-    mp_encode_array(buf,len);
+    mp_encode_array(L,buf,len);
     for (j = 1; j <= len; j++) {
         lua_pushnumber(L,j);
         lua_gettable(L,-2);
@@ -409,7 +407,7 @@ void mp_encode_lua_table_as_map(lua_State *L, mp_buf *buf, int level) {
     }
 
     /* Step two: actually encoding of the map. */
-    mp_encode_map(buf,len);
+    mp_encode_map(L,buf,len);
     lua_pushnil(L);
     while(lua_next(L,-2)) {
         /* Stack: ... key value */
@@ -474,10 +472,9 @@ void mp_encode_lua_table(lua_State *L, mp_buf *buf, int level) {
 
 void mp_encode_lua_null(lua_State *L, mp_buf *buf) {
     unsigned char b[1];
-    (void)L;
 
     b[0] = 0xc0;
-    mp_buf_append(buf,b,1);
+    mp_buf_append(L,buf,b,1);
 }
 
 void mp_encode_lua_type(lua_State *L, mp_buf *buf, int level) {
@@ -534,7 +531,7 @@ int mp_pack(lua_State *L) {
         buf->free += buf->len;
         buf->len = 0;
     }
-    mp_buf_free(buf);
+    mp_buf_free(L, buf);
 
     /* Concatenate all nargs buffers together */
     lua_concat(L, nargs);

deps/lua/src/lua_struct.c

@@ -89,12 +89,14 @@ typedef struct Header {
 } Header;
 
 
-static int getnum (const char **fmt, int df) {
+static int getnum (lua_State *L, const char **fmt, int df) {
   if (!isdigit(**fmt))  /* no number? */
     return df;  /* return default value */
   else {
     int a = 0;
     do {
+      if (a > (INT_MAX / 10) || a * 10 > (INT_MAX - (**fmt - '0')))
+        luaL_error(L, "integral size overflow");
       a = a*10 + *((*fmt)++) - '0';
     } while (isdigit(**fmt));
     return a;
@@ -115,9 +117,9 @@ static size_t optsize (lua_State *L, char opt, const char **fmt) {
     case 'f':  return sizeof(float);
     case 'd':  return sizeof(double);
     case 'x': return 1;
-    case 'c': return getnum(fmt, 1);
+    case 'c': return getnum(L, fmt, 1);
     case 'i': case 'I': {
-      int sz = getnum(fmt, sizeof(int));
+      int sz = getnum(L, fmt, sizeof(int));
       if (sz > MAXINTSIZE)
         luaL_error(L, "integral size %d is larger than limit of %d",
                        sz, MAXINTSIZE);
@@ -150,7 +152,7 @@ static void controloptions (lua_State *L, int opt, const char **fmt,
     case '>': h->endian = BIG; return;
     case '<': h->endian = LITTLE; return;
     case '!': {
-      int a = getnum(fmt, MAXALIGN);
+      int a = getnum(L, fmt, MAXALIGN);
       if (!isp2(a))
         luaL_error(L, "alignment %d is not a power of 2", a);
       h->align = a;

redis.conf

@@ -35,6 +35,14 @@
 # include /path/to/local.conf
 # include /path/to/other.conf
 
+################################## MODULES #####################################
+
+# Load modules at startup. If the server is not able to load modules
+# it will abort. It is possible to use multiple loadmodule directives.
+#
+# loadmodule /path/to/my_module.so
+# loadmodule /path/to/other_module.so
+
 ################################## NETWORK #####################################
 
 # By default, if no "bind" configuration directive is specified, Redis listens
@@ -56,10 +64,29 @@
 # is running).
 #
 # IF YOU ARE SURE YOU WANT YOUR INSTANCE TO LISTEN TO ALL THE INTERFACES
-# JUST UNCOMMENT THE FOLLOWING LINE.
+# JUST COMMENT THE FOLLOWING LINE.
 # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 bind 127.0.0.1
 
+# Protected mode is a layer of security protection, in order to avoid that
+# Redis instances left open on the internet are accessed and exploited.
+#
+# When protected mode is on and if:
+#
+# 1) The server is not binding explicitly to a set of addresses using the
+#    "bind" directive.
+# 2) No password is configured.
+#
+# The server only accepts connections from clients connecting from the
+# IPv4 and IPv6 loopback addresses 127.0.0.1 and ::1, and from Unix domain
+# sockets.
+#
+# By default protected mode is enabled. You should disable it only if
+# you are sure you want clients from other hosts to connect to Redis
+# even if no authentication is configured, nor a specific set of interfaces
+# are explicitly listed using the "bind" directive.
+protected-mode yes
+
 # Accept connections on the specified port, default is 6379 (IANA #815344).
 # If port 0 is specified Redis will not listen on a TCP socket.
 port 6379
@@ -98,8 +125,9 @@ timeout 0
 # Note that to close the connection the double of the time is needed.
 # On other kernels the period depends on the kernel configuration.
 #
-# A reasonable value for this option is 60 seconds.
-tcp-keepalive 0
+# A reasonable value for this option is 300 seconds, which is the new
+# Redis default starting with Redis 3.2.1.
+tcp-keepalive 300
 
 ################################# GENERAL #####################################
 
@@ -127,7 +155,7 @@ supervised no
 #
 # Creating a pid file is best effort: if Redis is not able to create it
 # nothing bad happens, the server will start and run normally.
-pidfile /var/run/redis.pid
+pidfile /var/run/redis_6379.pid
 
 # Specify the server verbosity level.
 # This can be one of:
@@ -415,6 +443,35 @@ slave-priority 100
 # By default min-slaves-to-write is set to 0 (feature disabled) and
 # min-slaves-max-lag is set to 10.
 
+# A Redis master is able to list the address and port of the attached
+# slaves in different ways. For example the "INFO replication" section
+# offers this information, which is used, among other tools, by
+# Redis Sentinel in order to discover slave instances.
+# Another place where this info is available is in the output of the
+# "ROLE" command of a masteer.
+#
+# The listed IP and address normally reported by a slave is obtained
+# in the following way:
+#
+#   IP: The address is auto detected by checking the peer address
+#   of the socket used by the slave to connect with the master.
+#
+#   Port: The port is communicated by the slave during the replication
+#   handshake, and is normally the port that the slave is using to
+#   list for connections.
+#
+# However when port forwarding or Network Address Translation (NAT) is
+# used, the slave may be actually reachable via different IP and port
+# pairs. The following two options can be used by a slave in order to
+# report to its master a specific set of IP and port, so that both INFO
+# and ROLE will report those values.
+#
+# There is no need to use both the options if you need to override just
+# the port or the IP address.
+#
+# slave-announce-ip 5.5.5.5
+# slave-announce-port 1234
+
 ################################## SECURITY ###################################
 
 # Require clients to issue AUTH <PASSWORD> before processing any other
@@ -464,7 +521,7 @@ slave-priority 100
 
 ############################## MEMORY MANAGEMENT ################################
 
-# Don't use more memory than the specified amount of bytes.
+# Set a memory usage limit to the specified amount of bytes.
 # When the memory limit is reached Redis will try to remove keys
 # according to the eviction policy selected (see maxmemory-policy).
 #
@@ -473,8 +530,8 @@ slave-priority 100
 # that would use more memory, like SET, LPUSH, and so on, and will continue
 # to reply to read-only commands like GET.
 #
-# This option is usually useful when using Redis as an LRU cache, or to set
-# a hard memory limit for an instance (using the 'noeviction' policy).
+# This option is usually useful when using Redis as an LRU or LFU cache, or to
+# set a hard memory limit for an instance (using the 'noeviction' policy).
 #
 # WARNING: If you have slaves attached to an instance with maxmemory on,
 # the size of the output buffers needed to feed the slaves are subtracted
@@ -492,12 +549,20 @@ slave-priority 100
 # MAXMEMORY POLICY: how Redis will select what to remove when maxmemory
 # is reached. You can select among five behaviors:
 #
-# volatile-lru -> remove the key with an expire set using an LRU algorithm
-# allkeys-lru -> remove any key according to the LRU algorithm
-# volatile-random -> remove a random key with an expire set
-# allkeys-random -> remove a random key, any key
-# volatile-ttl -> remove the key with the nearest expire time (minor TTL)
-# noeviction -> don't expire at all, just return an error on write operations
+# volatile-lru -> Evict using approximated LRU among the keys with an expire set.
+# allkeys-lru -> Evict any key using approximated LRU.
+# volatile-lfu -> Evict using approximated LFU among the keys with an expire set.
+# allkeys-lfu -> Evict any key using approximated LFU.
+# volatile-random -> Remove a random key among the ones with an expire set.
+# allkeys-random -> Remove a random key, any key.
+# volatile-ttl -> Remove the key with the nearest expire time (minor TTL)
+# noeviction -> Don't evict anything, just return an error on write operations.
+#
+# LRU means Least Recently Used
+# LFU means Least Frequently Used
+#
+# Both LRU, LFU and volatile-ttl are implemented using approximated
+# randomized algorithms.
 #
 # Note: with any of the above policies, Redis will return an error on write
 #       operations, when there are no suitable keys for eviction.
@@ -512,14 +577,14 @@ slave-priority 100
 #
 # maxmemory-policy noeviction
 
-# LRU and minimal TTL algorithms are not precise algorithms but approximated
+# LRU, LFU and minimal TTL algorithms are not precise algorithms but approximated
 # algorithms (in order to save memory), so you can tune it for speed or
 # accuracy. For default Redis will check five keys and pick the one that was
 # used less recently, you can change the sample size using the following
 # configuration directive.
 #
 # The default of 5 produces good enough results. 10 Approximates very closely
-# true LRU but costs a bit more CPU. 3 is very fast but not very accurate.
+# true LRU but costs more CPU. 3 is faster but not very accurate.
 #
 # maxmemory-samples 5
 
@@ -690,6 +755,20 @@ auto-aof-rewrite-min-size 64mb
 # will be found.
 aof-load-truncated yes
 
+# When rewriting the AOF file, Redis is able to use an RDB preamble in the
+# AOF file for faster rewrites and recoveries. When this option is turned
+# on the rewritten AOF file is composed of two different stanzas:
+#
+#   [RDB file][AOF tail]
+#
+# When loading Redis recognizes that the AOF file starts with the "REDIS"
+# string and loads the prefixed RDB file, and continues loading the AOF
+# tail.
+#
+# This is currently turned off by default in order to avoid the surprise
+# of a format change, but will at some point be used as the default.
+aof-use-rdb-preamble no
+
 ################################ LUA SCRIPTING  ###############################
 
 # Max execution time of a Lua script in milliseconds.
@@ -816,6 +895,39 @@ lua-time-limit 5000
 # In order to setup your cluster make sure to read the documentation
 # available at http://redis.io web site.
 
+########################## CLUSTER DOCKER/NAT support  ########################
+
+# In certain deployments, Redis Cluster nodes address discovery fails, because
+# addresses are NAT-ted or because ports are forwarded (the typical case is
+# Docker and other containers).
+#
+# In order to make Redis Cluster working in such environments, a static
+# configuration where each node known its public address is needed. The
+# following two options are used for this scope, and are:
+#
+# * cluster-announce-ip
+# * cluster-announce-port
+# * cluster-announce-bus-port
+#
+# Each instruct the node about its address, client port, and cluster message
+# bus port. The information is then published in the header of the bus packets
+# so that other nodes will be able to correctly map the address of the node
+# publishing the information.
+#
+# If the above options are not used, the normal Redis Cluster auto-detection
+# will be used instead.
+#
+# Note that when remapped, the bus port may not be at the fixed offset of
+# clients port + 10000, so you can specify any port and bus-port depending
+# on how they get remapped. If the bus-port is not set, a fixed offset of
+# 10000 will be used as usually.
+#
+# Example:
+#
+# cluster-announce-ip 10.1.1.5
+# cluster-announce-port 6379
+# cluster-announce-bus-port 6380
+
 ################################## SLOW LOG ###################################
 
 # The Redis Slow Log is a system to log queries that exceeded a specified
@@ -1052,3 +1164,55 @@ hz 10
 # in order to commit the file to the disk more incrementally and avoid
 # big latency spikes.
 aof-rewrite-incremental-fsync yes
+
+# Redis LFU eviction (see maxmemory setting) can be tuned. However it is a good
+# idea to start with the default settings and only change them after investigating
+# how to improve the performances and how the keys LFU change over time, which
+# is possible to inspect via the OBJECT FREQ command.
+#
+# There are two tunable parameters in the Redis LFU implementation: the
+# counter logarithm factor and the counter decay time. It is important to
+# understand what the two parameters mean before changing them.
+#
+# The LFU counter is just 8 bits per key, it's maximum value is 255, so Redis
+# uses a probabilistic increment with logarithmic behavior. Given the value
+# of the old counter, when a key is accessed, the counter is incremented in
+# this way:
+#
+# 1. A random number R between 0 and 1 is extracted.
+# 2. A probability P is calculated as 1/(old_value*lfu_log_factor+1).
+# 3. The counter is incremented only if R < P.
+#
+# The default lfu-log-factor is 10. This is a table of how the frequency
+# counter changes with a different number of accesses with different
+# logarithmic factors:
+#
+# +--------+------------+------------+------------+------------+------------+
+# | factor | 100 hits   | 1000 hits  | 100K hits  | 1M hits    | 10M hits   |
+# +--------+------------+------------+------------+------------+------------+
+# | 0      | 104        | 255        | 255        | 255        | 255        |
+# +--------+------------+------------+------------+------------+------------+
+# | 1      | 18         | 49         | 255        | 255        | 255        |
+# +--------+------------+------------+------------+------------+------------+
+# | 10     | 10         | 18         | 142        | 255        | 255        |
+# +--------+------------+------------+------------+------------+------------+
+# | 100    | 8          | 11         | 49         | 143        | 255        |
+# +--------+------------+------------+------------+------------+------------+
+#
+# NOTE: The above table was obtained by running the following commands:
+#
+#   redis-benchmark -n 1000000 incr foo
+#   redis-cli object freq foo
+#
+# NOTE 2: The counter initial value is 5 in order to give new objects a chance
+# to accumulate hits.
+#
+# The counter decay time is the time, in minutes, that must elapse in order
+# for the key counter to be divided by two (or decremented if it has a value
+# less <= 10).
+#
+# The default value for the lfu-decay-time is 1. A Special value of 0 means to
+# decay the counter every time it happens to be scanned.
+#
+# lfu-log-factor 10
+# lfu-decay-time 1

sentinel.conf

@@ -1,5 +1,21 @@
 # Example sentinel.conf
 
+# *** IMPORTANT ***
+#
+# By default Sentinel will not be reachable from interfaces different than
+# localhost, either use the 'bind' directive to bind to a list of network
+# interfaces, or disable protected mode with "protected-mode no" by
+# adding it to this configuration file.
+#
+# Before doing that MAKE SURE the instance is protected from the outside
+# world via firewalling or other means.
+#
+# For example you may use one of the following:
+#
+# bind 127.0.0.1 192.168.1.1
+#
+# protected-mode no
+
 # port <sentinel-port>
 # The port that this sentinel instance will run on
 port 26379
@@ -178,4 +194,3 @@ sentinel failover-timeout mymaster 180000
 #
 # sentinel client-reconfig-script mymaster /var/redis/reconfig.sh
 
-

src/Makefile

@@ -15,11 +15,12 @@
 release_hdr := $(shell sh -c './mkreleasehdr.sh')
 uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo not')
 OPTIMIZATION?=-O2
-DEPENDENCY_TARGETS=hiredis linenoise lua geohash-int
+DEPENDENCY_TARGETS=hiredis linenoise lua
+NODEPS:=clean distclean
 
 # Default settings
 STD=-std=c99 -pedantic -DREDIS_STATIC=''
-WARN=-Wall -W
+WARN=-Wall -W -Wno-missing-field-initializers
 OPT=$(OPTIMIZATION)
 
 PREFIX?=/usr/local
@@ -53,7 +54,7 @@ endif
 # Override default settings if possible
 -include .make-settings
 
-FINAL_CFLAGS=$(STD) $(WARN) $(OPT) $(DEBUG) $(CFLAGS) $(REDIS_CFLAGS) -I../deps/geohash-int
+FINAL_CFLAGS=$(STD) $(WARN) $(OPT) $(DEBUG) $(CFLAGS) $(REDIS_CFLAGS)
 FINAL_LDFLAGS=$(LDFLAGS) $(REDIS_LDFLAGS) $(DEBUG)
 FINAL_LIBS=-lm
 DEBUG=-g -ggdb
@@ -65,17 +66,27 @@ ifeq ($(uname_S),SunOS)
 	FINAL_LIBS+= -ldl -lnsl -lsocket -lresolv -lpthread -lrt
 else
 ifeq ($(uname_S),Darwin)
-	# Darwin (nothing to do)
+	# Darwin
+	FINAL_LIBS+= -ldl
 else
 ifeq ($(uname_S),AIX)
         # AIX
         FINAL_LDFLAGS+= -Wl,-bexpall
-        FINAL_LIBS+= -pthread -lcrypt -lbsd
-
+        FINAL_LIBS+=-ldl -pthread -lcrypt -lbsd
+else
+ifeq ($(uname_S),OpenBSD)
+	# OpenBSD
+	FINAL_LIBS+= -lpthread
+else
+ifeq ($(uname_S),FreeBSD)
+	# FreeBSD
+	FINAL_LIBS+= -lpthread
 else
 	# All the other OSes (notably Linux)
 	FINAL_LDFLAGS+= -rdynamic
-	FINAL_LIBS+= -pthread
+	FINAL_LIBS+=-ldl -pthread
+endif
+endif
 endif
 endif
 endif
@@ -95,7 +106,7 @@ endif
 ifeq ($(MALLOC),jemalloc)
 	DEPENDENCY_TARGETS+= jemalloc
 	FINAL_CFLAGS+= -DUSE_JEMALLOC -I../deps/jemalloc/include
-	FINAL_LIBS+= ../deps/jemalloc/lib/libjemalloc.a -ldl
+	FINAL_LIBS+= ../deps/jemalloc/lib/libjemalloc.a
 endif
 
 REDIS_CC=$(QUIET_CC)$(CC) $(FINAL_CFLAGS)
@@ -117,8 +128,7 @@ endif
 
 REDIS_SERVER_NAME=redis-server
 REDIS_SENTINEL_NAME=redis-sentinel
-REDIS_SERVER_OBJ=adlist.o quicklist.o ae.o anet.o dict.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o geo.o lazyfree.o
-REDIS_GEOHASH_OBJ=../deps/geohash-int/geohash.o ../deps/geohash-int/geohash_helper.o
+REDIS_SERVER_OBJ=adlist.o quicklist.o ae.o anet.o dict.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o
 REDIS_CLI_NAME=redis-cli
 REDIS_CLI_OBJ=anet.o adlist.o redis-cli.o zmalloc.o release.o anet.o ae.o crc64.o
 REDIS_BENCHMARK_NAME=redis-benchmark
@@ -132,16 +142,15 @@ all: $(REDIS_SERVER_NAME) $(REDIS_SENTINEL_NAME) $(REDIS_CLI_NAME) $(REDIS_BENCH
 	@echo "Hint: It's a good idea to run 'make test' ;)"
 	@echo ""
 
+Makefile.dep:
+	-$(REDIS_CC) -MM *.c > Makefile.dep 2> /dev/null || true
+
+ifeq (0, $(words $(findstring $(MAKECMDGOALS), $(NODEPS))))
+-include Makefile.dep
+endif
+
 .PHONY: all
 
-# Deps (use make dep to generate this)
-include Makefile.dep
-
-dep:
-	$(REDIS_CC) -MM *.c > Makefile.dep
-
-.PHONY: dep
-
 persist-settings: distclean
 	echo STD=$(STD) >> .make-settings
 	echo WARN=$(WARN) >> .make-settings
@@ -172,7 +181,7 @@ endif
 
 # redis-server
 $(REDIS_SERVER_NAME): $(REDIS_SERVER_OBJ)
-	$(REDIS_LD) -o $@ $^ ../deps/hiredis/libhiredis.a ../deps/lua/src/liblua.a $(REDIS_GEOHASH_OBJ) $(FINAL_LIBS)
+	$(REDIS_LD) -o $@ $^ ../deps/hiredis/libhiredis.a ../deps/lua/src/liblua.a $(FINAL_LIBS)
 
 # redis-sentinel
 $(REDIS_SENTINEL_NAME): $(REDIS_SERVER_NAME)
@@ -194,6 +203,9 @@ $(REDIS_BENCHMARK_NAME): $(REDIS_BENCHMARK_OBJ)
 $(REDIS_CHECK_AOF_NAME): $(REDIS_CHECK_AOF_OBJ)
 	$(REDIS_LD) -o $@ $^ $(FINAL_LIBS)
 
+dict-benchmark: dict.c zmalloc.c sds.c
+	$(REDIS_CC) $(FINAL_CFLAGS) dict.c zmalloc.c sds.c -D DICT_BENCHMARK_MAIN -o dict-benchmark
+
 # Because the jemalloc.h header is generated as a part of the jemalloc build,
 # building it should complete before building any other object. Instead of
 # depending on a single artifact, build all dependencies first.
@@ -201,7 +213,7 @@ $(REDIS_CHECK_AOF_NAME): $(REDIS_CHECK_AOF_OBJ)
 	$(REDIS_CC) -c $<
 
 clean:
-	rm -rf $(REDIS_SERVER_NAME) $(REDIS_SENTINEL_NAME) $(REDIS_CLI_NAME) $(REDIS_BENCHMARK_NAME) $(REDIS_CHECK_RDB_NAME) $(REDIS_CHECK_AOF_NAME) *.o *.gcda *.gcno *.gcov redis.info lcov-html
+	rm -rf $(REDIS_SERVER_NAME) $(REDIS_SENTINEL_NAME) $(REDIS_CLI_NAME) $(REDIS_BENCHMARK_NAME) $(REDIS_CHECK_RDB_NAME) $(REDIS_CHECK_AOF_NAME) *.o *.gcda *.gcno *.gcov redis.info lcov-html Makefile.dep dict-benchmark
 
 .PHONY: clean
 

src/Makefile.dep

@@ -1,186 +0,0 @@
-adlist.o: adlist.c adlist.h zmalloc.h
-ae.o: ae.c ae.h zmalloc.h config.h ae_kqueue.c ae_epoll.c ae_select.c ae_evport.c
-ae_epoll.o: ae_epoll.c
-ae_evport.o: ae_evport.c
-ae_kqueue.o: ae_kqueue.c
-ae_select.o: ae_select.c
-anet.o: anet.c fmacros.h anet.h
-aof.o: aof.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- bio.h
-bio.o: bio.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- bio.h
-bitops.o: bitops.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-blocked.o: blocked.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-cluster.o: cluster.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- cluster.h
-config.o: config.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- cluster.h
-crc16.o: crc16.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-crc64.o: crc64.c
-db.o: db.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- cluster.h atomicvar.h
-debug.o: debug.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- bio.h
-dict.o: dict.c fmacros.h dict.h zmalloc.h redisassert.h
-endianconv.o: endianconv.c
-geo.o: geo.c geo.h server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- ../deps/geohash-int/geohash_helper.h ../deps/geohash-int/geohash.h
-hyperloglog.o: hyperloglog.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-intset.o: intset.c intset.h zmalloc.h endianconv.h config.h
-latency.o: latency.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-lazyfree.o: lazyfree.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- bio.h atomicvar.h cluster.h
-lzf_c.o: lzf_c.c lzfP.h
-lzf_d.o: lzf_d.c lzfP.h
-memtest.o: memtest.c config.h
-multi.o: multi.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-networking.o: networking.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-notify.o: notify.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-object.o: object.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-pqsort.o: pqsort.c
-pubsub.o: pubsub.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-quicklist.o: quicklist.c quicklist.h zmalloc.h ziplist.h util.h sds.h \
- lzf.h
-rand.o: rand.c
-rdb.o: rdb.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- lzf.h
-redis-benchmark.o: redis-benchmark.c fmacros.h ../deps/hiredis/sds.h ae.h \
- ../deps/hiredis/hiredis.h adlist.h zmalloc.h
-redis-check-aof.o: redis-check-aof.c fmacros.h config.h
-redis-check-rdb.o: redis-check-rdb.c server.h fmacros.h config.h \
- solarisfixes.h ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h \
- sds.h dict.h adlist.h zmalloc.h anet.h ziplist.h intset.h version.h \
- util.h latency.h sparkline.h quicklist.h zipmap.h sha1.h endianconv.h \
- crc64.h rdb.h rio.h lzf.h
-redis-cli.o: redis-cli.c fmacros.h version.h ../deps/hiredis/hiredis.h \
- ../deps/hiredis/sds.h zmalloc.h ../deps/linenoise/linenoise.h help.h \
- anet.h ae.h
-release.o: release.c release.h version.h crc64.h
-replication.o: replication.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-rio.o: rio.c fmacros.h rio.h sds.h util.h crc64.h config.h server.h \
- solarisfixes.h ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h \
- dict.h adlist.h zmalloc.h anet.h ziplist.h intset.h version.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h rdb.h
-scripting.o: scripting.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- rand.h cluster.h ../deps/lua/src/lauxlib.h ../deps/lua/src/lua.h \
- ../deps/lua/src/lualib.h
-sds.o: sds.c sds.h sdsalloc.h zmalloc.h
-sentinel.o: sentinel.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- ../deps/hiredis/hiredis.h ../deps/hiredis/async.h \
- ../deps/hiredis/hiredis.h
-server.o: server.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- cluster.h slowlog.h bio.h asciilogo.h
-setproctitle.o: setproctitle.c
-sha1.o: sha1.c solarisfixes.h sha1.h config.h
-slowlog.o: slowlog.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- slowlog.h
-sort.o: sort.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h \
- pqsort.h
-sparkline.o: sparkline.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-syncio.o: syncio.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-t_hash.o: t_hash.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-t_list.o: t_list.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-t_set.o: t_set.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-t_string.o: t_string.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-t_zset.o: t_zset.c server.h fmacros.h config.h solarisfixes.h \
- ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
- adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
- sparkline.h quicklist.h zipmap.h sha1.h endianconv.h crc64.h rdb.h rio.h
-util.o: util.c fmacros.h util.h sds.h sha1.h
-ziplist.o: ziplist.c zmalloc.h util.h sds.h ziplist.h endianconv.h \
- config.h redisassert.h
-zipmap.o: zipmap.c zmalloc.h endianconv.h config.h
-zmalloc.o: zmalloc.c config.h zmalloc.h atomicvar.h

src/adlist.c

@@ -242,7 +242,7 @@ listNode *listNext(listIter *iter)
 list *listDup(list *orig)
 {
     list *copy;
-    listIter *iter;
+    listIter iter;
     listNode *node;
 
     if ((copy = listCreate()) == NULL)
@@ -250,26 +250,23 @@ list *listDup(list *orig)
     copy->dup = orig->dup;
     copy->free = orig->free;
     copy->match = orig->match;
-    iter = listGetIterator(orig, AL_START_HEAD);
-    while((node = listNext(iter)) != NULL) {
+    listRewind(orig, &iter);
+    while((node = listNext(&iter)) != NULL) {
         void *value;
 
         if (copy->dup) {
             value = copy->dup(node->value);
             if (value == NULL) {
                 listRelease(copy);
-                listReleaseIterator(iter);
                 return NULL;
             }
         } else
             value = node->value;
         if (listAddNodeTail(copy, value) == NULL) {
             listRelease(copy);
-            listReleaseIterator(iter);
             return NULL;
         }
     }
-    listReleaseIterator(iter);
     return copy;
 }
 
@@ -284,24 +281,21 @@ list *listDup(list *orig)
  * NULL is returned. */
 listNode *listSearchKey(list *list, void *key)
 {
-    listIter *iter;
+    listIter iter;
     listNode *node;
 
-    iter = listGetIterator(list, AL_START_HEAD);
-    while((node = listNext(iter)) != NULL) {
+    listRewind(list, &iter);
+    while((node = listNext(&iter)) != NULL) {
         if (list->match) {
             if (list->match(node->value, key)) {
-                listReleaseIterator(iter);
                 return node;
             }
         } else {
             if (key == node->value) {
-                listReleaseIterator(iter);
                 return node;
             }
         }
     }
-    listReleaseIterator(iter);
     return NULL;
 }
 

src/ae.c

@@ -221,21 +221,12 @@ long long aeCreateTimeEvent(aeEventLoop *eventLoop, long long milliseconds,
 
 int aeDeleteTimeEvent(aeEventLoop *eventLoop, long long id)
 {
-    aeTimeEvent *te, *prev = NULL;
-
-    te = eventLoop->timeEventHead;
+    aeTimeEvent *te = eventLoop->timeEventHead;
     while(te) {
         if (te->id == id) {
-            if (prev == NULL)
-                eventLoop->timeEventHead = te->next;
-            else
-                prev->next = te->next;
-            if (te->finalizerProc)
-                te->finalizerProc(eventLoop, te->clientData);
-            zfree(te);
+            te->id = AE_DELETED_EVENT_ID;
             return AE_OK;
         }
-        prev = te;
         te = te->next;
     }
     return AE_ERR; /* NO event with the specified ID found */
@@ -270,7 +261,7 @@ static aeTimeEvent *aeSearchNearestTimer(aeEventLoop *eventLoop)
 /* Process time events */
 static int processTimeEvents(aeEventLoop *eventLoop) {
     int processed = 0;
-    aeTimeEvent *te;
+    aeTimeEvent *te, *prev;
     long long maxId;
     time_t now = time(NULL);
 
@@ -291,12 +282,32 @@ static int processTimeEvents(aeEventLoop *eventLoop) {
     }
     eventLoop->lastTime = now;
 
+    prev = NULL;
     te = eventLoop->timeEventHead;
     maxId = eventLoop->timeEventNextId-1;
     while(te) {
         long now_sec, now_ms;
         long long id;
 
+        /* Remove events scheduled for deletion. */
+        if (te->id == AE_DELETED_EVENT_ID) {
+            aeTimeEvent *next = te->next;
+            if (prev == NULL)
+                eventLoop->timeEventHead = te->next;
+            else
+                prev->next = te->next;
+            if (te->finalizerProc)
+                te->finalizerProc(eventLoop, te->clientData);
+            zfree(te);
+            te = next;
+            continue;
+        }
+
+        /* Make sure we don't process time events created by time events in
+         * this iteration. Note that this check is currently useless: we always
+         * add new timers on the head, however if we change the implementation
+         * detail, this check may be useful again: we keep it here for future
+         * defense. */
         if (te->id > maxId) {
             te = te->next;
             continue;
@@ -310,28 +321,14 @@ static int processTimeEvents(aeEventLoop *eventLoop) {
             id = te->id;
             retval = te->timeProc(eventLoop, id, te->clientData);
             processed++;
-            /* After an event is processed our time event list may
-             * no longer be the same, so we restart from head.
-             * Still we make sure to don't process events registered
-             * by event handlers itself in order to don't loop forever.
-             * To do so we saved the max ID we want to handle.
-             *
-             * FUTURE OPTIMIZATIONS:
-             * Note that this is NOT great algorithmically. Redis uses
-             * a single time event so it's not a problem but the right
-             * way to do this is to add the new elements on head, and
-             * to flag deleted elements in a special way for later
-             * deletion (putting references to the nodes to delete into
-             * another linked list). */
             if (retval != AE_NOMORE) {
                 aeAddMillisecondsToNow(retval,&te->when_sec,&te->when_ms);
             } else {
-                aeDeleteTimeEvent(eventLoop, id);
+                te->id = AE_DELETED_EVENT_ID;
             }
-            te = eventLoop->timeEventHead;
-        } else {
-            te = te->next;
         }
+        prev = te;
+        te = te->next;
     }
     return processed;
 }
@@ -371,19 +368,22 @@ int aeProcessEvents(aeEventLoop *eventLoop, int flags)
         if (shortest) {
             long now_sec, now_ms;
 
-            /* Calculate the time missing for the nearest
-             * timer to fire. */
             aeGetTime(&now_sec, &now_ms);
             tvp = &tv;
-            tvp->tv_sec = shortest->when_sec - now_sec;
-            if (shortest->when_ms < now_ms) {
-                tvp->tv_usec = ((shortest->when_ms+1000) - now_ms)*1000;
-                tvp->tv_sec --;
+
+            /* How many milliseconds we need to wait for the next
+             * time event to fire? */
+            long long ms =
+                (shortest->when_sec - now_sec)*1000 +
+                shortest->when_ms - now_ms;
+
+            if (ms > 0) {
+                tvp->tv_sec = ms/1000;
+                tvp->tv_usec = (ms % 1000)*1000;
             } else {
-                tvp->tv_usec = (shortest->when_ms - now_ms)*1000;
+                tvp->tv_sec = 0;
+                tvp->tv_usec = 0;
             }
-            if (tvp->tv_sec < 0) tvp->tv_sec = 0;
-            if (tvp->tv_usec < 0) tvp->tv_usec = 0;
         } else {
             /* If we have to check for events but need to return
              * ASAP because of AE_DONT_WAIT we need to set the timeout

src/ae.h

@@ -33,6 +33,8 @@
 #ifndef __AE_H__
 #define __AE_H__
 
+#include <time.h>
+
 #define AE_OK 0
 #define AE_ERR -1
 
@@ -46,6 +48,7 @@
 #define AE_DONT_WAIT 4
 
 #define AE_NOMORE -1
+#define AE_DELETED_EVENT_ID -1
 
 /* Macros */
 #define AE_NOTUSED(V) ((void) V)

src/ae_epoll.c

@@ -72,7 +72,7 @@ static void aeApiFree(aeEventLoop *eventLoop) {
 
 static int aeApiAddEvent(aeEventLoop *eventLoop, int fd, int mask) {
     aeApiState *state = eventLoop->apidata;
-    struct epoll_event ee;
+    struct epoll_event ee = {0}; /* avoid valgrind warning */
     /* If the fd was already monitored for some event, we need a MOD
      * operation. Otherwise we need an ADD operation. */
     int op = eventLoop->events[fd].mask == AE_NONE ?
@@ -82,7 +82,6 @@ static int aeApiAddEvent(aeEventLoop *eventLoop, int fd, int mask) {
     mask |= eventLoop->events[fd].mask; /* Merge old events */
     if (mask & AE_READABLE) ee.events |= EPOLLIN;
     if (mask & AE_WRITABLE) ee.events |= EPOLLOUT;
-    ee.data.u64 = 0; /* avoid valgrind warning */
     ee.data.fd = fd;
     if (epoll_ctl(state->epfd,op,fd,&ee) == -1) return -1;
     return 0;
@@ -90,13 +89,12 @@ static int aeApiAddEvent(aeEventLoop *eventLoop, int fd, int mask) {
 
 static void aeApiDelEvent(aeEventLoop *eventLoop, int fd, int delmask) {
     aeApiState *state = eventLoop->apidata;
-    struct epoll_event ee;
+    struct epoll_event ee = {0}; /* avoid valgrind warning */
     int mask = eventLoop->events[fd].mask & (~delmask);
 
     ee.events = 0;
     if (mask & AE_READABLE) ee.events |= EPOLLIN;
     if (mask & AE_WRITABLE) ee.events |= EPOLLOUT;
-    ee.data.u64 = 0; /* avoid valgrind warning */
     ee.data.fd = fd;
     if (mask != AE_NONE) {
         epoll_ctl(state->epfd,EPOLL_CTL_MOD,fd,&ee);

src/ae_select.c

@@ -29,6 +29,7 @@
  */
 
 
+#include <sys/select.h>
 #include <string.h>
 
 typedef struct aeApiState {

src/anet.c

@@ -486,7 +486,7 @@ static int _anetTcpServer(char *err, int port, char *bindaddr, int af, int backl
         goto end;
     }
     if (p == NULL) {
-        anetSetError(err, "unable to bind socket");
+        anetSetError(err, "unable to bind socket, errno: %d", errno);
         goto error;
     }
 

src/anet.h

@@ -31,6 +31,8 @@
 #ifndef ANET_H
 #define ANET_H
 
+#include <sys/types.h>
+
 #define ANET_OK 0
 #define ANET_ERR -1
 #define ANET_ERR_LEN 256

src/aof.c

@@ -38,6 +38,7 @@
 #include <sys/time.h>
 #include <sys/resource.h>
 #include <sys/wait.h>
+#include <sys/param.h>
 
 void aofUpdateCurrentSize(void);
 void aofClosePipes(void);
@@ -234,14 +235,26 @@ void stopAppendOnly(void) {
 /* Called when the user switches from "appendonly no" to "appendonly yes"
  * at runtime using the CONFIG command. */
 int startAppendOnly(void) {
+    char cwd[MAXPATHLEN]; /* Current working dir path for error messages. */
+
     server.aof_last_fsync = server.unixtime;
     server.aof_fd = open(server.aof_filename,O_WRONLY|O_APPEND|O_CREAT,0644);
     serverAssert(server.aof_state == AOF_OFF);
     if (server.aof_fd == -1) {
-        serverLog(LL_WARNING,"Redis needs to enable the AOF but can't open the append only file: %s",strerror(errno));
+        char *cwdp = getcwd(cwd,MAXPATHLEN);
+
+        serverLog(LL_WARNING,
+            "Redis needs to enable the AOF but can't open the "
+            "append only file %s (in server root dir %s): %s",
+            server.aof_filename,
+            cwdp ? cwdp : "unknown",
+            strerror(errno));
         return C_ERR;
     }
-    if (rewriteAppendOnlyFileBackground() == C_ERR) {
+    if (server.rdb_child_pid != -1) {
+        server.aof_rewrite_scheduled = 1;
+        serverLog(LL_WARNING,"AOF was enabled but there is already a child process saving an RDB file on disk. An AOF background was scheduled to start when possible.");
+    } else if (rewriteAppendOnlyFileBackground() == C_ERR) {
         close(server.aof_fd);
         serverLog(LL_WARNING,"Redis needs to enable the AOF but can't trigger a background AOF rewrite operation. Check the above logs for more info about the error.");
         return C_ERR;
@@ -603,19 +616,23 @@ int loadAppendOnlyFile(char *filename) {
     struct redis_stat sb;
     int old_aof_state = server.aof_state;
     long loops = 0;
-    off_t valid_up_to = 0; /* Offset of the latest well-formed command loaded. */
-
-    if (fp && redis_fstat(fileno(fp),&sb) != -1 && sb.st_size == 0) {
-        server.aof_current_size = 0;
-        fclose(fp);
-        return C_ERR;
-    }
+    off_t valid_up_to = 0; /* Offset of latest well-formed command loaded. */
 
     if (fp == NULL) {
         serverLog(LL_WARNING,"Fatal error: can't open the append log file for reading: %s",strerror(errno));
         exit(1);
     }
 
+    /* Handle a zero-length AOF file as a special case. An emtpy AOF file
+     * is a valid AOF because an empty server with AOF enabled will create
+     * a zero length file at startup, that will remain like that if no write
+     * operation is received. */
+    if (fp && redis_fstat(fileno(fp),&sb) != -1 && sb.st_size == 0) {
+        server.aof_current_size = 0;
+        fclose(fp);
+        return C_ERR;
+    }
+
     /* Temporarily disable AOF, to prevent EXEC from feeding a MULTI
      * to the same file we're about to read. */
     server.aof_state = AOF_OFF;
@@ -623,6 +640,28 @@ int loadAppendOnlyFile(char *filename) {
     fakeClient = createFakeClient();
     startLoading(fp);
 
+    /* Check if this AOF file has an RDB preamble. In that case we need to
+     * load the RDB file and later continue loading the AOF tail. */
+    char sig[5]; /* "REDIS" */
+    if (fread(sig,1,5,fp) != 5 || memcmp(sig,"REDIS",5) != 0) {
+        /* No RDB preamble, seek back at 0 offset. */
+        if (fseek(fp,0,SEEK_SET) == -1) goto readerr;
+    } else {
+        /* RDB preamble. Pass loading the RDB functions. */
+        rio rdb;
+
+        serverLog(LL_NOTICE,"Reading RDB preamble from AOF file...");
+        if (fseek(fp,0,SEEK_SET) == -1) goto readerr;
+        rioInitWithFile(&rdb,fp);
+        if (rdbLoadRio(&rdb) != C_OK) {
+            serverLog(LL_WARNING,"Error reading the RDB preamble of the AOF file, AOF loading aborted");
+            goto readerr;
+        } else {
+            serverLog(LL_NOTICE,"Reading the remaining AOF tail...");
+        }
+    }
+
+    /* Read the actual AOF file, in REPL format, command by command. */
     while(1) {
         int argc, j;
         unsigned long len;
@@ -683,6 +722,7 @@ int loadAppendOnlyFile(char *filename) {
         }
 
         /* Run the command in the context of a fake client */
+        fakeClient->cmd = cmd;
         cmd->proc(fakeClient);
 
         /* The fake client should not have a reply */
@@ -693,6 +733,7 @@ int loadAppendOnlyFile(char *filename) {
         /* Clean up. Command code may have changed argv/argc so we use the
          * argv/argc of the client instead of the local variables. */
         freeFakeClientArgv(fakeClient);
+        fakeClient->cmd = NULL;
         if (server.aof_load_truncated) valid_up_to = ftello(fp);
     }
 
@@ -711,6 +752,7 @@ loaded_ok: /* DB loaded, cleanup and return C_OK to the caller. */
 
 readerr: /* Read error. If feof(fp) is true, fall through to unexpected EOF. */
     if (!feof(fp)) {
+        if (fakeClient) freeFakeClient(fakeClient); /* avoid valgrind warning */
         serverLog(LL_WARNING,"Unrecoverable error reading the append only file: %s", strerror(errno));
         exit(1);
     }
@@ -740,10 +782,12 @@ uxeof: /* Unexpected AOF end of file. */
             }
         }
     }
+    if (fakeClient) freeFakeClient(fakeClient); /* avoid valgrind warning */
     serverLog(LL_WARNING,"Unexpected end of file reading the append only file. You can: 1) Make a backup of your AOF file, then use ./redis-check-aof --fix <filename>. 2) Alternatively you can set the 'aof-load-truncated' configuration option to yes and restart the server.");
     exit(1);
 
 fmterr: /* Format error. */
+    if (fakeClient) freeFakeClient(fakeClient); /* avoid valgrind warning */
     serverLog(LL_WARNING,"Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>");
     exit(1);
 }
@@ -753,7 +797,7 @@ fmterr: /* Format error. */
  * ------------------------------------------------------------------------- */
 
 /* Delegate writing an object to writing a bulk string or bulk long long.
- * This is not placed in rio.c since that adds the redis.h dependency. */
+ * This is not placed in rio.c since that adds the server.h dependency. */
 int rioWriteBulkObject(rio *r, robj *obj) {
     /* Avoid using getDecodedObject to help copy-on-write (we are often
      * in a child process when this function is called). */
@@ -970,6 +1014,22 @@ int rewriteHashObject(rio *r, robj *key, robj *o) {
     return 1;
 }
 
+/* Call the module type callback in order to rewrite a data type
+ * that is exported by a module and is not handled by Redis itself.
+ * The function returns 0 on error, 1 on success. */
+int rewriteModuleObject(rio *r, robj *key, robj *o) {
+    RedisModuleIO io;
+    moduleValue *mv = o->ptr;
+    moduleType *mt = mv->type;
+    moduleInitIOContext(io,mt,r);
+    mt->aof_rewrite(&io,key,mv->value);
+    if (io.ctx) {
+        moduleFreeContext(io.ctx);
+        zfree(io.ctx);
+    }
+    return io.error ? 0 : 1;
+}
+
 /* This function is called by the child rewriting the AOF file to read
  * the difference accumulated from the parent into a buffer, that is
  * concatenated at the end of the rewrite. */
@@ -985,51 +1045,23 @@ ssize_t aofReadDiffFromParent(void) {
     return total;
 }
 
-/* Write a sequence of commands able to fully rebuild the dataset into
- * "filename". Used both by REWRITEAOF and BGREWRITEAOF.
- *
- * In order to minimize the number of commands needed in the rewritten
- * log Redis uses variadic commands when possible, such as RPUSH, SADD
- * and ZADD. However at max AOF_REWRITE_ITEMS_PER_CMD items per time
- * are inserted using a single command. */
-int rewriteAppendOnlyFile(char *filename) {
+int rewriteAppendOnlyFileRio(rio *aof) {
     dictIterator *di = NULL;
     dictEntry *de;
-    rio aof;
-    FILE *fp;
-    char tmpfile[256];
-    int j;
-    long long now = mstime();
-    char byte;
     size_t processed = 0;
+    long long now = mstime();
+    int j;
 
-    /* Note that we have to use a different temp name here compared to the
-     * one used by rewriteAppendOnlyFileBackground() function. */
-    snprintf(tmpfile,256,"temp-rewriteaof-%d.aof", (int) getpid());
-    fp = fopen(tmpfile,"w");
-    if (!fp) {
-        serverLog(LL_WARNING, "Opening the temp file for AOF rewrite in rewriteAppendOnlyFile(): %s", strerror(errno));
-        return C_ERR;
-    }
-
-    server.aof_child_diff = sdsempty();
-    rioInitWithFile(&aof,fp);
-    if (server.aof_rewrite_incremental_fsync)
-        rioSetAutoSync(&aof,AOF_AUTOSYNC_BYTES);
     for (j = 0; j < server.dbnum; j++) {
         char selectcmd[] = "*2\r\n$6\r\nSELECT\r\n";
         redisDb *db = server.db+j;
         dict *d = db->dict;
         if (dictSize(d) == 0) continue;
         di = dictGetSafeIterator(d);
-        if (!di) {
-            fclose(fp);
-            return C_ERR;
-        }
 
         /* SELECT the new DB */
-        if (rioWrite(&aof,selectcmd,sizeof(selectcmd)-1) == 0) goto werr;
-        if (rioWriteBulkLongLong(&aof,j) == 0) goto werr;
+        if (rioWrite(aof,selectcmd,sizeof(selectcmd)-1) == 0) goto werr;
+        if (rioWriteBulkLongLong(aof,j) == 0) goto werr;
 
         /* Iterate this DB writing every entry */
         while((de = dictNext(di)) != NULL) {
@@ -1050,37 +1082,83 @@ int rewriteAppendOnlyFile(char *filename) {
             if (o->type == OBJ_STRING) {
                 /* Emit a SET command */
                 char cmd[]="*3\r\n$3\r\nSET\r\n";
-                if (rioWrite(&aof,cmd,sizeof(cmd)-1) == 0) goto werr;
+                if (rioWrite(aof,cmd,sizeof(cmd)-1) == 0) goto werr;
                 /* Key and value */
-                if (rioWriteBulkObject(&aof,&key) == 0) goto werr;
-                if (rioWriteBulkObject(&aof,o) == 0) goto werr;
+                if (rioWriteBulkObject(aof,&key) == 0) goto werr;
+                if (rioWriteBulkObject(aof,o) == 0) goto werr;
             } else if (o->type == OBJ_LIST) {
-                if (rewriteListObject(&aof,&key,o) == 0) goto werr;
+                if (rewriteListObject(aof,&key,o) == 0) goto werr;
             } else if (o->type == OBJ_SET) {
-                if (rewriteSetObject(&aof,&key,o) == 0) goto werr;
+                if (rewriteSetObject(aof,&key,o) == 0) goto werr;
             } else if (o->type == OBJ_ZSET) {
-                if (rewriteSortedSetObject(&aof,&key,o) == 0) goto werr;
+                if (rewriteSortedSetObject(aof,&key,o) == 0) goto werr;
             } else if (o->type == OBJ_HASH) {
-                if (rewriteHashObject(&aof,&key,o) == 0) goto werr;
+                if (rewriteHashObject(aof,&key,o) == 0) goto werr;
+            } else if (o->type == OBJ_MODULE) {
+                if (rewriteModuleObject(aof,&key,o) == 0) goto werr;
             } else {
                 serverPanic("Unknown object type");
             }
             /* Save the expire time */
             if (expiretime != -1) {
                 char cmd[]="*3\r\n$9\r\nPEXPIREAT\r\n";
-                if (rioWrite(&aof,cmd,sizeof(cmd)-1) == 0) goto werr;
-                if (rioWriteBulkObject(&aof,&key) == 0) goto werr;
-                if (rioWriteBulkLongLong(&aof,expiretime) == 0) goto werr;
+                if (rioWrite(aof,cmd,sizeof(cmd)-1) == 0) goto werr;
+                if (rioWriteBulkObject(aof,&key) == 0) goto werr;
+                if (rioWriteBulkLongLong(aof,expiretime) == 0) goto werr;
             }
             /* Read some diff from the parent process from time to time. */
-            if (aof.processed_bytes > processed+1024*10) {
-                processed = aof.processed_bytes;
+            if (aof->processed_bytes > processed+AOF_READ_DIFF_INTERVAL_BYTES) {
+                processed = aof->processed_bytes;
                 aofReadDiffFromParent();
             }
         }
         dictReleaseIterator(di);
         di = NULL;
     }
+    return C_OK;
+
+werr:
+    if (di) dictReleaseIterator(di);
+    return C_ERR;
+}
+
+/* Write a sequence of commands able to fully rebuild the dataset into
+ * "filename". Used both by REWRITEAOF and BGREWRITEAOF.
+ *
+ * In order to minimize the number of commands needed in the rewritten
+ * log Redis uses variadic commands when possible, such as RPUSH, SADD
+ * and ZADD. However at max AOF_REWRITE_ITEMS_PER_CMD items per time
+ * are inserted using a single command. */
+int rewriteAppendOnlyFile(char *filename) {
+    rio aof;
+    FILE *fp;
+    char tmpfile[256];
+    char byte;
+
+    /* Note that we have to use a different temp name here compared to the
+     * one used by rewriteAppendOnlyFileBackground() function. */
+    snprintf(tmpfile,256,"temp-rewriteaof-%d.aof", (int) getpid());
+    fp = fopen(tmpfile,"w");
+    if (!fp) {
+        serverLog(LL_WARNING, "Opening the temp file for AOF rewrite in rewriteAppendOnlyFile(): %s", strerror(errno));
+        return C_ERR;
+    }
+
+    server.aof_child_diff = sdsempty();
+    rioInitWithFile(&aof,fp);
+
+    if (server.aof_rewrite_incremental_fsync)
+        rioSetAutoSync(&aof,AOF_AUTOSYNC_BYTES);
+
+    if (server.aof_use_rdb_preamble) {
+        int error;
+        if (rdbSaveRio(&aof,&error,RDB_SAVE_AOF_PREAMBLE) == C_ERR) {
+            errno = error;
+            goto werr;
+        }
+    } else {
+        if (rewriteAppendOnlyFileRio(&aof) == C_ERR) goto werr;
+    }
 
     /* Do an initial slow fsync here while the parent is still sending
      * data, in order to make the next final fsync faster. */
@@ -1146,7 +1224,6 @@ werr:
     serverLog(LL_WARNING,"Write error writing append only file on disk: %s", strerror(errno));
     fclose(fp);
     unlink(tmpfile);
-    if (di) dictReleaseIterator(di);
     return C_ERR;
 }
 
@@ -1244,8 +1321,9 @@ int rewriteAppendOnlyFileBackground(void) {
     pid_t childpid;
     long long start;
 
-    if (server.aof_child_pid != -1) return C_ERR;
+    if (server.aof_child_pid != -1 || server.rdb_child_pid != -1) return C_ERR;
     if (aofCreatePipes() != C_OK) return C_ERR;
+    openChildInfoPipe();
     start = ustime();
     if ((childpid = fork()) == 0) {
         char tmpfile[256];
@@ -1255,13 +1333,16 @@ int rewriteAppendOnlyFileBackground(void) {
         redisSetProcTitle("redis-aof-rewrite");
         snprintf(tmpfile,256,"temp-rewriteaof-bg-%d.aof", (int) getpid());
         if (rewriteAppendOnlyFile(tmpfile) == C_OK) {
-            size_t private_dirty = zmalloc_get_private_dirty();
+            size_t private_dirty = zmalloc_get_private_dirty(-1);
 
             if (private_dirty) {
                 serverLog(LL_NOTICE,
                     "AOF rewrite: %zu MB of memory used by copy-on-write",
                     private_dirty/(1024*1024));
             }
+
+            server.child_info_data.cow_size = private_dirty;
+            sendChildInfo(CHILD_INFO_TYPE_AOF);
             exitFromChild(0);
         } else {
             exitFromChild(1);
@@ -1272,6 +1353,7 @@ int rewriteAppendOnlyFileBackground(void) {
         server.stat_fork_rate = (double) zmalloc_used_memory() * 1000000 / server.stat_fork_time / (1024*1024*1024); /* GB per second. */
         latencyAddSampleIfNeeded("fork",server.stat_fork_time/1000);
         if (childpid == -1) {
+            closeChildInfoPipe();
             serverLog(LL_WARNING,
                 "Can't rewrite append only file in background: fork: %s",
                 strerror(errno));
@@ -1413,7 +1495,10 @@ void backgroundRewriteDoneHandler(int exitcode, int bysignal) {
         latencyStartMonitor(latency);
         if (rename(tmpfile,server.aof_filename) == -1) {
             serverLog(LL_WARNING,
-                "Error trying to rename the temporary AOF file: %s", strerror(errno));
+                "Error trying to rename the temporary AOF file %s into %s: %s",
+                tmpfile,
+                server.aof_filename,
+                strerror(errno));
             close(newfd);
             if (oldfd != -1) close(oldfd);
             goto cleanup;

src/bitops.c

@@ -34,27 +34,6 @@
  * Helpers and low level bit functions.
  * -------------------------------------------------------------------------- */
 
-/* This helper function used by GETBIT / SETBIT parses the bit offset argument
- * making sure an error is returned if it is negative or if it overflows
- * Redis 512 MB limit for the string value. */
-static int getBitOffsetFromArgument(client *c, robj *o, size_t *offset) {
-    long long loffset;
-    char *err = "bit offset is not an integer or out of range";
-
-    if (getLongLongFromObjectOrReply(c,o,&loffset,err) != C_OK)
-        return C_ERR;
-
-    /* Limit offset to 512MB in bytes */
-    if ((loffset < 0) || ((unsigned long long)loffset >> 3) >= (512*1024*1024))
-    {
-        addReplyError(c,err);
-        return C_ERR;
-    }
-
-    *offset = (size_t)loffset;
-    return C_OK;
-}
-
 /* Count number of bits set in the binary array pointed by 's' and long
  * 'count' bytes. The implementation of this function is required to
  * work with a input string length up to 512 MB. */
@@ -199,6 +178,209 @@ long redisBitpos(void *s, unsigned long count, int bit) {
     return 0; /* Just to avoid warnings. */
 }
 
+/* The following set.*Bitfield and get.*Bitfield functions implement setting
+ * and getting arbitrary size (up to 64 bits) signed and unsigned integers
+ * at arbitrary positions into a bitmap.
+ *
+ * The representation considers the bitmap as having the bit number 0 to be
+ * the most significant bit of the first byte, and so forth, so for example
+ * setting a 5 bits unsigned integer to value 23 at offset 7 into a bitmap
+ * previously set to all zeroes, will produce the following representation:
+ *
+ * +--------+--------+
+ * |00000001|01110000|
+ * +--------+--------+
+ *
+ * When offsets and integer sizes are aligned to bytes boundaries, this is the
+ * same as big endian, however when such alignment does not exist, its important
+ * to also understand how the bits inside a byte are ordered.
+ *
+ * Note that this format follows the same convention as SETBIT and related
+ * commands.
+ */
+
+void setUnsignedBitfield(unsigned char *p, uint64_t offset, uint64_t bits, uint64_t value) {
+    uint64_t byte, bit, byteval, bitval, j;
+
+    for (j = 0; j < bits; j++) {
+        bitval = (value & ((uint64_t)1<<(bits-1-j))) != 0;
+        byte = offset >> 3;
+        bit = 7 - (offset & 0x7);
+        byteval = p[byte];
+        byteval &= ~(1 << bit);
+        byteval |= bitval << bit;
+        p[byte] = byteval & 0xff;
+        offset++;
+    }
+}
+
+void setSignedBitfield(unsigned char *p, uint64_t offset, uint64_t bits, int64_t value) {
+    uint64_t uv = value; /* Casting will add UINT64_MAX + 1 if v is negative. */
+    setUnsignedBitfield(p,offset,bits,uv);
+}
+
+uint64_t getUnsignedBitfield(unsigned char *p, uint64_t offset, uint64_t bits) {
+    uint64_t byte, bit, byteval, bitval, j, value = 0;
+
+    for (j = 0; j < bits; j++) {
+        byte = offset >> 3;
+        bit = 7 - (offset & 0x7);
+        byteval = p[byte];
+        bitval = (byteval >> bit) & 1;
+        value = (value<<1) | bitval;
+        offset++;
+    }
+    return value;
+}
+
+int64_t getSignedBitfield(unsigned char *p, uint64_t offset, uint64_t bits) {
+    int64_t value;
+    union {uint64_t u; int64_t i;} conv;
+
+    /* Converting from unsigned to signed is undefined when the value does
+     * not fit, however here we assume two's complement and the original value
+     * was obtained from signed -> unsigned conversion, so we'll find the
+     * most significant bit set if the original value was negative.
+     *
+     * Note that two's complement is mandatory for exact-width types
+     * according to the C99 standard. */
+    conv.u = getUnsignedBitfield(p,offset,bits);
+    value = conv.i;
+
+    /* If the top significant bit is 1, propagate it to all the
+     * higher bits for two's complement representation of signed
+     * integers. */
+    if (value & ((uint64_t)1 << (bits-1)))
+        value |= ((uint64_t)-1) << bits;
+    return value;
+}
+
+/* The following two functions detect overflow of a value in the context
+ * of storing it as an unsigned or signed integer with the specified
+ * number of bits. The functions both take the value and a possible increment.
+ * If no overflow could happen and the value+increment fit inside the limits,
+ * then zero is returned, otherwise in case of overflow, 1 is returned,
+ * otherwise in case of underflow, -1 is returned.
+ *
+ * When non-zero is returned (oferflow or underflow), if not NULL, *limit is
+ * set to the value the operation should result when an overflow happens,
+ * depending on the specified overflow semantics:
+ *
+ * For BFOVERFLOW_SAT if 1 is returned, *limit it is set maximum value that
+ * you can store in that integer. when -1 is returned, *limit is set to the
+ * minimum value that an integer of that size can represent.
+ *
+ * For BFOVERFLOW_WRAP *limit is set by performing the operation in order to
+ * "wrap" around towards zero for unsigned integers, or towards the most
+ * negative number that is possible to represent for signed integers. */
+
+#define BFOVERFLOW_WRAP 0
+#define BFOVERFLOW_SAT 1
+#define BFOVERFLOW_FAIL 2 /* Used by the BITFIELD command implementation. */
+
+int checkUnsignedBitfieldOverflow(uint64_t value, int64_t incr, uint64_t bits, int owtype, uint64_t *limit) {
+    uint64_t max = (bits == 64) ? UINT64_MAX : (((uint64_t)1<<bits)-1);
+    int64_t maxincr = max-value;
+    int64_t minincr = -value;
+
+    if (value > max || (incr > 0 && incr > maxincr)) {
+        if (limit) {
+            if (owtype == BFOVERFLOW_WRAP) {
+                goto handle_wrap;
+            } else if (owtype == BFOVERFLOW_SAT) {
+                *limit = max;
+            }
+        }
+        return 1;
+    } else if (incr < 0 && incr < minincr) {
+        if (limit) {
+            if (owtype == BFOVERFLOW_WRAP) {
+                goto handle_wrap;
+            } else if (owtype == BFOVERFLOW_SAT) {
+                *limit = 0;
+            }
+        }
+        return -1;
+    }
+    return 0;
+
+handle_wrap:
+    {
+        uint64_t mask = ((uint64_t)-1) << bits;
+        uint64_t res = value+incr;
+
+        res &= ~mask;
+        *limit = res;
+    }
+    return 1;
+}
+
+int checkSignedBitfieldOverflow(int64_t value, int64_t incr, uint64_t bits, int owtype, int64_t *limit) {
+    int64_t max = (bits == 64) ? INT64_MAX : (((int64_t)1<<(bits-1))-1);
+    int64_t min = (-max)-1;
+
+    /* Note that maxincr and minincr could overflow, but we use the values
+     * only after checking 'value' range, so when we use it no overflow
+     * happens. */
+    int64_t maxincr = max-value;
+    int64_t minincr = min-value;
+
+    if (value > max || (bits != 64 && incr > maxincr) || (value >= 0 && incr > 0 && incr > maxincr))
+    {
+        if (limit) {
+            if (owtype == BFOVERFLOW_WRAP) {
+                goto handle_wrap;
+            } else if (owtype == BFOVERFLOW_SAT) {
+                *limit = max;
+            }
+        }
+        return 1;
+    } else if (value < min || (bits != 64 && incr < minincr) || (value < 0 && incr < 0 && incr < minincr)) {
+        if (limit) {
+            if (owtype == BFOVERFLOW_WRAP) {
+                goto handle_wrap;
+            } else if (owtype == BFOVERFLOW_SAT) {
+                *limit = min;
+            }
+        }
+        return -1;
+    }
+    return 0;
+
+handle_wrap:
+    {
+        uint64_t mask = ((uint64_t)-1) << bits;
+        uint64_t msb = (uint64_t)1 << (bits-1);
+        uint64_t a = value, b = incr, c;
+        c = a+b; /* Perform addition as unsigned so that's defined. */
+
+        /* If the sign bit is set, propagate to all the higher order
+         * bits, to cap the negative value. If it's clear, mask to
+         * the positive integer limit. */
+        if (c & msb) {
+            c |= mask;
+        } else {
+            c &= ~mask;
+        }
+        *limit = c;
+    }
+    return 1;
+}
+
+/* Debugging function. Just show bits in the specified bitmap. Not used
+ * but here for not having to rewrite it when debugging is needed. */
+void printBits(unsigned char *p, unsigned long count) {
+    unsigned long j, i, byte;
+
+    for (j = 0; j < count; j++) {
+        byte = p[j];
+        for (i = 0x80; i > 0; i /= 2)
+            printf("%c", (byte & i) ? '1' : '0');
+        printf("|");
+    }
+    printf("\n");
+}
+
 /* -----------------------------------------------------------------------------
  * Bits related string commands: GETBIT, SETBIT, BITCOUNT, BITOP.
  * -------------------------------------------------------------------------- */
@@ -208,16 +390,140 @@ long redisBitpos(void *s, unsigned long count, int bit) {
 #define BITOP_XOR   2
 #define BITOP_NOT   3
 
+#define BITFIELDOP_GET 0
+#define BITFIELDOP_SET 1
+#define BITFIELDOP_INCRBY 2
+
+/* This helper function used by GETBIT / SETBIT parses the bit offset argument
+ * making sure an error is returned if it is negative or if it overflows
+ * Redis 512 MB limit for the string value.
+ *
+ * If the 'hash' argument is true, and 'bits is positive, then the command
+ * will also parse bit offsets prefixed by "#". In such a case the offset
+ * is multiplied by 'bits'. This is useful for the BITFIELD command. */
+int getBitOffsetFromArgument(client *c, robj *o, size_t *offset, int hash, int bits) {
+    long long loffset;
+    char *err = "bit offset is not an integer or out of range";
+    char *p = o->ptr;
+    size_t plen = sdslen(p);
+    int usehash = 0;
+
+    /* Handle #<offset> form. */
+    if (p[0] == '#' && hash && bits > 0) usehash = 1;
+
+    if (string2ll(p+usehash,plen-usehash,&loffset) == 0) {
+        addReplyError(c,err);
+        return C_ERR;
+    }
+
+    /* Adjust the offset by 'bits' for #<offset> form. */
+    if (usehash) loffset *= bits;
+
+    /* Limit offset to 512MB in bytes */
+    if ((loffset < 0) || ((unsigned long long)loffset >> 3) >= (512*1024*1024))
+    {
+        addReplyError(c,err);
+        return C_ERR;
+    }
+
+    *offset = (size_t)loffset;
+    return C_OK;
+}
+
+/* This helper function for BITFIELD parses a bitfield type in the form
+ * <sign><bits> where sign is 'u' or 'i' for unsigned and signed, and
+ * the bits is a value between 1 and 64. However 64 bits unsigned integers
+ * are reported as an error because of current limitations of Redis protocol
+ * to return unsigned integer values greater than INT64_MAX.
+ *
+ * On error C_ERR is returned and an error is sent to the client. */
+int getBitfieldTypeFromArgument(client *c, robj *o, int *sign, int *bits) {
+    char *p = o->ptr;
+    char *err = "Invalid bitfield type. Use something like i16 u8. Note that u64 is not supported but i64 is.";
+    long long llbits;
+
+    if (p[0] == 'i') {
+        *sign = 1;
+    } else if (p[0] == 'u') {
+        *sign = 0;
+    } else {
+        addReplyError(c,err);
+        return C_ERR;
+    }
+
+    if ((string2ll(p+1,strlen(p+1),&llbits)) == 0 ||
+        llbits < 1 ||
+        (*sign == 1 && llbits > 64) ||
+        (*sign == 0 && llbits > 63))
+    {
+        addReplyError(c,err);
+        return C_ERR;
+    }
+    *bits = llbits;
+    return C_OK;
+}
+
+/* This is an helper function for commands implementations that need to write
+ * bits to a string object. The command creates or pad with zeroes the string
+ * so that the 'maxbit' bit can be addressed. The object is finally
+ * returned. Otherwise if the key holds a wrong type NULL is returned and
+ * an error is sent to the client. */
+robj *lookupStringForBitCommand(client *c, size_t maxbit) {
+    size_t byte = maxbit >> 3;
+    robj *o = lookupKeyWrite(c->db,c->argv[1]);
+
+    if (o == NULL) {
+        o = createObject(OBJ_STRING,sdsnewlen(NULL, byte+1));
+        dbAdd(c->db,c->argv[1],o);
+    } else {
+        if (checkType(c,o,OBJ_STRING)) return NULL;
+        o = dbUnshareStringValue(c->db,c->argv[1],o);
+        o->ptr = sdsgrowzero(o->ptr,byte+1);
+    }
+    return o;
+}
+
+/* Return a pointer to the string object content, and stores its length
+ * in 'len'. The user is required to pass (likely stack allocated) buffer
+ * 'llbuf' of at least LONG_STR_SIZE bytes. Such a buffer is used in the case
+ * the object is integer encoded in order to provide the representation
+ * without usign heap allocation.
+ *
+ * The function returns the pointer to the object array of bytes representing
+ * the string it contains, that may be a pointer to 'llbuf' or to the
+ * internal object representation. As a side effect 'len' is filled with
+ * the length of such buffer.
+ *
+ * If the source object is NULL the function is guaranteed to return NULL
+ * and set 'len' to 0. */
+unsigned char *getObjectReadOnlyString(robj *o, long *len, char *llbuf) {
+    serverAssert(o->type == OBJ_STRING);
+    unsigned char *p = NULL;
+
+    /* Set the 'p' pointer to the string, that can be just a stack allocated
+     * array if our string was integer encoded. */
+    if (o && o->encoding == OBJ_ENCODING_INT) {
+        p = (unsigned char*) llbuf;
+        if (len) *len = ll2string(llbuf,LONG_STR_SIZE,(long)o->ptr);
+    } else if (o) {
+        p = (unsigned char*) o->ptr;
+        if (len) *len = sdslen(o->ptr);
+    } else {
+        if (len) *len = 0;
+    }
+    return p;
+}
+
 /* SETBIT key offset bitvalue */
 void setbitCommand(client *c) {
     robj *o;
     char *err = "bit is not an integer or out of range";
     size_t bitoffset;
-    int byte, bit;
+    ssize_t byte, bit;
     int byteval, bitval;
     long on;
 
-    if (getBitOffsetFromArgument(c,c->argv[2],&bitoffset) != C_OK)
+    if (getBitOffsetFromArgument(c,c->argv[2],&bitoffset,0,0) != C_OK)
         return;
 
     if (getLongFromObjectOrReply(c,c->argv[3],&on,err) != C_OK)
@@ -229,18 +535,10 @@ void setbitCommand(client *c) {
         return;
     }
 
-    byte = bitoffset >> 3;
-    o = lookupKeyWrite(c->db,c->argv[1]);
-    if (o == NULL) {
-        o = createObject(OBJ_STRING,sdsnewlen(NULL, byte+1));
-        dbAdd(c->db,c->argv[1],o);
-    } else {
-        if (checkType(c,o,OBJ_STRING)) return;
-        o = dbUnshareStringValue(c->db,c->argv[1],o);
-        o->ptr = sdsgrowzero(o->ptr,byte+1);
-    }
+    if ((o = lookupStringForBitCommand(c,bitoffset)) == NULL) return;
 
     /* Get current values */
+    byte = bitoffset >> 3;
     byteval = ((uint8_t*)o->ptr)[byte];
     bit = 7 - (bitoffset & 0x7);
     bitval = byteval & (1 << bit);
@@ -263,7 +561,7 @@ void getbitCommand(client *c) {
     size_t byte, bit;
     size_t bitval = 0;
 
-    if (getBitOffsetFromArgument(c,c->argv[2],&bitoffset) != C_OK)
+    if (getBitOffsetFromArgument(c,c->argv[2],&bitoffset,0,0) != C_OK)
         return;
 
     if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.czero)) == NULL ||
@@ -461,21 +759,12 @@ void bitcountCommand(client *c) {
     robj *o;
     long start, end, strlen;
     unsigned char *p;
-    char llbuf[32];
+    char llbuf[LONG_STR_SIZE];
 
     /* Lookup, check for type, and return 0 for non existing keys. */
     if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.czero)) == NULL ||
         checkType(c,o,OBJ_STRING)) return;
-
-    /* Set the 'p' pointer to the string, that can be just a stack allocated
-     * array if our string was integer encoded. */
-    if (o->encoding == OBJ_ENCODING_INT) {
-        p = (unsigned char*) llbuf;
-        strlen = ll2string(llbuf,sizeof(llbuf),(long)o->ptr);
-    } else {
-        p = (unsigned char*) o->ptr;
-        strlen = sdslen(o->ptr);
-    }
+    p = getObjectReadOnlyString(o,&strlen,llbuf);
 
     /* Parse start/end range if any. */
     if (c->argc == 4) {
@@ -484,6 +773,10 @@ void bitcountCommand(client *c) {
         if (getLongFromObjectOrReply(c,c->argv[3],&end,NULL) != C_OK)
             return;
         /* Convert negative indexes */
+        if (start < 0 && end < 0 && start > end) {
+            addReply(c,shared.czero);
+            return;
+        }
         if (start < 0) start = strlen+start;
         if (end < 0) end = strlen+end;
         if (start < 0) start = 0;
@@ -515,7 +808,7 @@ void bitposCommand(client *c) {
     robj *o;
     long bit, start, end, strlen;
     unsigned char *p;
-    char llbuf[32];
+    char llbuf[LONG_STR_SIZE];
     int end_given = 0;
 
     /* Parse the bit argument to understand what we are looking for, set
@@ -535,16 +828,7 @@ void bitposCommand(client *c) {
         return;
     }
     if (checkType(c,o,OBJ_STRING)) return;
-
-    /* Set the 'p' pointer to the string, that can be just a stack allocated
-     * array if our string was integer encoded. */
-    if (o->encoding == OBJ_ENCODING_INT) {
-        p = (unsigned char*) llbuf;
-        strlen = ll2string(llbuf,sizeof(llbuf),(long)o->ptr);
-    } else {
-        p = (unsigned char*) o->ptr;
-        strlen = sdslen(o->ptr);
-    }
+    p = getObjectReadOnlyString(o,&strlen,llbuf);
 
     /* Parse start/end range if any. */
     if (c->argc == 4 || c->argc == 5) {
@@ -596,3 +880,234 @@ void bitposCommand(client *c) {
         addReplyLongLong(c,pos);
     }
 }
+
+/* BITFIELD key subcommmand-1 arg ... subcommand-2 arg ... subcommand-N ...
+ *
+ * Supported subcommands:
+ *
+ * GET <type> <offset>
+ * SET <type> <offset> <value>
+ * INCRBY <type> <offset> <increment>
+ * OVERFLOW [WRAP|SAT|FAIL]
+ */
+
+struct bitfieldOp {
+    uint64_t offset;    /* Bitfield offset. */
+    int64_t i64;        /* Increment amount (INCRBY) or SET value */
+    int opcode;         /* Operation id. */
+    int owtype;         /* Overflow type to use. */
+    int bits;           /* Integer bitfield bits width. */
+    int sign;           /* True if signed, otherwise unsigned op. */
+};
+
+void bitfieldCommand(client *c) {
+    robj *o;
+    size_t bitoffset;
+    int j, numops = 0, changes = 0;
+    struct bitfieldOp *ops = NULL; /* Array of ops to execute at end. */
+    int owtype = BFOVERFLOW_WRAP; /* Overflow type. */
+    int readonly = 1;
+    long higest_write_offset = 0;
+
+    for (j = 2; j < c->argc; j++) {
+        int remargs = c->argc-j-1; /* Remaining args other than current. */
+        char *subcmd = c->argv[j]->ptr; /* Current command name. */
+        int opcode; /* Current operation code. */
+        long long i64 = 0;  /* Signed SET value. */
+        int sign = 0; /* Signed or unsigned type? */
+        int bits = 0; /* Bitfield width in bits. */
+
+        if (!strcasecmp(subcmd,"get") && remargs >= 2)
+            opcode = BITFIELDOP_GET;
+        else if (!strcasecmp(subcmd,"set") && remargs >= 3)
+            opcode = BITFIELDOP_SET;
+        else if (!strcasecmp(subcmd,"incrby") && remargs >= 3)
+            opcode = BITFIELDOP_INCRBY;
+        else if (!strcasecmp(subcmd,"overflow") && remargs >= 1) {
+            char *owtypename = c->argv[j+1]->ptr;
+            j++;
+            if (!strcasecmp(owtypename,"wrap"))
+                owtype = BFOVERFLOW_WRAP;
+            else if (!strcasecmp(owtypename,"sat"))
+                owtype = BFOVERFLOW_SAT;
+            else if (!strcasecmp(owtypename,"fail"))
+                owtype = BFOVERFLOW_FAIL;
+            else {
+                addReplyError(c,"Invalid OVERFLOW type specified");
+                zfree(ops);
+                return;
+            }
+            continue;
+        } else {
+            addReply(c,shared.syntaxerr);
+            zfree(ops);
+            return;
+        }
+
+        /* Get the type and offset arguments, common to all the ops. */
+        if (getBitfieldTypeFromArgument(c,c->argv[j+1],&sign,&bits) != C_OK) {
+            zfree(ops);
+            return;
+        }
+
+        if (getBitOffsetFromArgument(c,c->argv[j+2],&bitoffset,1,bits) != C_OK){
+            zfree(ops);
+            return;
+        }
+
+        if (opcode != BITFIELDOP_GET) {
+            readonly = 0;
+            higest_write_offset = bitoffset + bits - 1;
+            /* INCRBY and SET require another argument. */
+            if (getLongLongFromObjectOrReply(c,c->argv[j+3],&i64,NULL) != C_OK){
+                zfree(ops);
+                return;
+            }
+        }
+
+        /* Populate the array of operations we'll process. */
+        ops = zrealloc(ops,sizeof(*ops)*(numops+1));
+        ops[numops].offset = bitoffset;
+        ops[numops].i64 = i64;
+        ops[numops].opcode = opcode;
+        ops[numops].owtype = owtype;
+        ops[numops].bits = bits;
+        ops[numops].sign = sign;
+        numops++;
+
+        j += 3 - (opcode == BITFIELDOP_GET);
+    }
+
+    if (readonly) {
+        /* Lookup for read is ok if key doesn't exit, but errors
+         * if it's not a string. */
+        o = lookupKeyRead(c->db,c->argv[1]);
+        if (o != NULL && checkType(c,o,OBJ_STRING)) return;
+    } else {
+        /* Lookup by making room up to the farest bit reached by
+         * this operation. */
+        if ((o = lookupStringForBitCommand(c,
+            higest_write_offset)) == NULL) return;
+    }
+
+    addReplyMultiBulkLen(c,numops);
+
+    /* Actually process the operations. */
+    for (j = 0; j < numops; j++) {
+        struct bitfieldOp *thisop = ops+j;
+
+        /* Execute the operation. */
+        if (thisop->opcode == BITFIELDOP_SET ||
+            thisop->opcode == BITFIELDOP_INCRBY)
+        {
+            /* SET and INCRBY: We handle both with the same code path
+             * for simplicity. SET return value is the previous value so
+             * we need fetch & store as well. */
+
+            /* We need two different but very similar code paths for signed
+             * and unsigned operations, since the set of functions to get/set
+             * the integers and the used variables types are different. */
+            if (thisop->sign) {
+                int64_t oldval, newval, wrapped, retval;
+                int overflow;
+
+                oldval = getSignedBitfield(o->ptr,thisop->offset,
+                        thisop->bits);
+
+                if (thisop->opcode == BITFIELDOP_INCRBY) {
+                    newval = oldval + thisop->i64;
+                    overflow = checkSignedBitfieldOverflow(oldval,
+                            thisop->i64,thisop->bits,thisop->owtype,&wrapped);
+                    if (overflow) newval = wrapped;
+                    retval = newval;
+                } else {
+                    newval = thisop->i64;
+                    overflow = checkSignedBitfieldOverflow(newval,
+                            0,thisop->bits,thisop->owtype,&wrapped);
+                    if (overflow) newval = wrapped;
+                    retval = oldval;
+                }
+
+                /* On overflow of type is "FAIL", don't write and return
+                 * NULL to signal the condition. */
+                if (!(overflow && thisop->owtype == BFOVERFLOW_FAIL)) {
+                    addReplyLongLong(c,retval);
+                    setSignedBitfield(o->ptr,thisop->offset,
+                                      thisop->bits,newval);
+                } else {
+                    addReply(c,shared.nullbulk);
+                }
+            } else {
+                uint64_t oldval, newval, wrapped, retval;
+                int overflow;
+
+                oldval = getUnsignedBitfield(o->ptr,thisop->offset,
+                        thisop->bits);
+
+                if (thisop->opcode == BITFIELDOP_INCRBY) {
+                    newval = oldval + thisop->i64;
+                    overflow = checkUnsignedBitfieldOverflow(oldval,
+                            thisop->i64,thisop->bits,thisop->owtype,&wrapped);
+                    if (overflow) newval = wrapped;
+                    retval = newval;
+                } else {
+                    newval = thisop->i64;
+                    overflow = checkUnsignedBitfieldOverflow(newval,
+                            0,thisop->bits,thisop->owtype,&wrapped);
+                    if (overflow) newval = wrapped;
+                    retval = oldval;
+                }
+                /* On overflow of type is "FAIL", don't write and return
+                 * NULL to signal the condition. */
+                if (!(overflow && thisop->owtype == BFOVERFLOW_FAIL)) {
+                    addReplyLongLong(c,retval);
+                    setUnsignedBitfield(o->ptr,thisop->offset,
+                                        thisop->bits,newval);
+                } else {
+                    addReply(c,shared.nullbulk);
+                }
+            }
+            changes++;
+        } else {
+            /* GET */
+            unsigned char buf[9];
+            long strlen = 0;
+            unsigned char *src = NULL;
+            char llbuf[LONG_STR_SIZE];
+
+            if (o != NULL)
+                src = getObjectReadOnlyString(o,&strlen,llbuf);
+
+            /* For GET we use a trick: before executing the operation
+             * copy up to 9 bytes to a local buffer, so that we can easily
+             * execute up to 64 bit operations that are at actual string
+             * object boundaries. */
+            memset(buf,0,9);
+            int i;
+            size_t byte = thisop->offset >> 3;
+            for (i = 0; i < 9; i++) {
+                if (src == NULL || i+byte >= (size_t)strlen) break;
+                buf[i] = src[i+byte];
+            }
+
+            /* Now operate on the copied buffer which is guaranteed
+             * to be zero-padded. */
+            if (thisop->sign) {
+                int64_t val = getSignedBitfield(buf,thisop->offset-(byte*8),
+                                            thisop->bits);
+                addReplyLongLong(c,val);
+            } else {
+                uint64_t val = getUnsignedBitfield(buf,thisop->offset-(byte*8),
+                                            thisop->bits);
+                addReplyLongLong(c,val);
+            }
+        }
+    }
+
+    if (changes) {
+        signalModifiedKey(c->db,c->argv[1]);
+        notifyKeyspaceEvent(NOTIFY_STRING,"setbit",c->argv[1],c->db->id);
+        server.dirty += changes;
+    }
+    zfree(ops);
+}

src/childinfo.c

@@ -0,0 +1,85 @@
+/*
+ * Copyright (c) 2016, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "server.h"
+#include <unistd.h>
+
+/* Open a child-parent channel used in order to move information about the
+ * RDB / AOF saving process from the child to the parent (for instance
+ * the amount of copy on write memory used) */
+void openChildInfoPipe(void) {
+    if (pipe(server.child_info_pipe) == -1) {
+        /* On error our two file descriptors should be still set to -1,
+         * but we call anyway cloesChildInfoPipe() since can't hurt. */
+        closeChildInfoPipe();
+    } else if (anetNonBlock(NULL,server.child_info_pipe[0]) != ANET_OK) {
+        closeChildInfoPipe();
+    } else {
+        memset(&server.child_info_data,0,sizeof(server.child_info_data));
+    }
+}
+
+/* Close the pipes opened with openChildInfoPipe(). */
+void closeChildInfoPipe(void) {
+    if (server.child_info_pipe[0] != -1 ||
+        server.child_info_pipe[1] != -1)
+    {
+        close(server.child_info_pipe[0]);
+        close(server.child_info_pipe[1]);
+        server.child_info_pipe[0] = -1;
+        server.child_info_pipe[1] = -1;
+    }
+}
+
+/* Send COW data to parent. The child should call this function after populating
+ * the corresponding fields it want to sent (according to the process type). */
+void sendChildInfo(int ptype) {
+    if (server.child_info_pipe[1] == -1) return;
+    server.child_info_data.magic = CHILD_INFO_MAGIC;
+    server.child_info_data.process_type = ptype;
+    ssize_t wlen = sizeof(server.child_info_data);
+    if (write(server.child_info_pipe[1],&server.child_info_data,wlen) != wlen) {
+        /* Nothing to do on error, this will be detected by the other side. */
+    }
+}
+
+/* Receive COW data from parent. */
+void receiveChildInfo(void) {
+    if (server.child_info_pipe[0] == -1) return;
+    ssize_t wlen = sizeof(server.child_info_data);
+    if (read(server.child_info_pipe[0],&server.child_info_data,wlen) == wlen &&
+        server.child_info_data.magic == CHILD_INFO_MAGIC)
+    {
+        if (server.child_info_data.process_type == CHILD_INFO_TYPE_RDB) {
+            server.stat_rdb_cow_bytes = server.child_info_data.cow_size;
+        } else if (server.child_info_data.process_type == CHILD_INFO_TYPE_AOF) {
+            server.stat_aof_cow_bytes = server.child_info_data.cow_size;
+        }
+    }
+}

src/cluster.h

@@ -23,6 +23,7 @@
 #define CLUSTER_DEFAULT_MIGRATION_BARRIER 1
 #define CLUSTER_MF_TIMEOUT 5000 /* Milliseconds to do a manual failover. */
 #define CLUSTER_MF_PAUSE_MULT 2 /* Master pause manual failover mult. */
+#define CLUSTER_SLAVE_MIGRATION_DELAY 5000 /* Delay for slave migration. */
 
 /* Redirection errors returned by getNodeByQuery(). */
 #define CLUSTER_REDIR_NONE 0          /* Node can serve the request. */
@@ -53,7 +54,7 @@ typedef struct clusterLink {
 #define CLUSTER_NODE_HANDSHAKE 32 /* We have still to exchange the first ping */
 #define CLUSTER_NODE_NOADDR   64  /* We don't know the address of this node */
 #define CLUSTER_NODE_MEET 128     /* Send a MEET message to this node */
-#define CLUSTER_NODE_PROMOTED 256 /* Master was a slave promoted by failover */
+#define CLUSTER_NODE_MIGRATE_TO 256 /* Master elegible for replica migration. */
 #define CLUSTER_NODE_NULL_NAME "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
 
 #define nodeIsMaster(n) ((n)->flags & CLUSTER_NODE_MASTER)
@@ -87,15 +88,20 @@ typedef struct clusterNode {
     int numslots;   /* Number of slots handled by this node */
     int numslaves;  /* Number of slave nodes, if this is a master */
     struct clusterNode **slaves; /* pointers to slave nodes */
-    struct clusterNode *slaveof; /* pointer to the master node */
+    struct clusterNode *slaveof; /* pointer to the master node. Note that it
+                                    may be NULL even if the node is a slave
+                                    if we don't have the master node in our
+                                    tables. */
     mstime_t ping_sent;      /* Unix time we sent latest ping */
     mstime_t pong_received;  /* Unix time we received the pong */
     mstime_t fail_time;      /* Unix time when FAIL flag was set */
     mstime_t voted_time;     /* Last time we voted for a slave of this master */
     mstime_t repl_offset_time;  /* Unix time we received offset for this node */
+    mstime_t orphaned_time;     /* Starting time of orphaned master condition */
     long long repl_offset;      /* Last known repl offset for this node. */
     char ip[NET_IP_STR_LEN];  /* Latest known IP address of this node */
-    int port;                   /* Latest known port of this node */
+    int port;                   /* Latest known clients port of this node */
+    int cport;                  /* Latest known cluster port of this node. */
     clusterLink *link;          /* TCP/IP link with this node */
     list *fail_reports;         /* List of nodes signaling this as failing */
 } clusterNode;
@@ -166,10 +172,10 @@ typedef struct {
     uint32_t ping_sent;
     uint32_t pong_received;
     char ip[NET_IP_STR_LEN];  /* IP address last time it was seen */
-    uint16_t port;              /* port last time it was seen */
+    uint16_t port;              /* base port last time it was seen */
+    uint16_t cport;             /* cluster port last time it was seen */
     uint16_t flags;             /* node->flags copy */
-    uint16_t notused1;          /* Some room for future improvements. */
-    uint32_t notused2;
+    uint32_t notused1;
 } clusterMsgDataGossip;
 
 typedef struct {
@@ -214,13 +220,13 @@ union clusterMsgData {
     } update;
 };
 
-#define CLUSTER_PROTO_VER 0 /* Cluster bus protocol version. */
+#define CLUSTER_PROTO_VER 1 /* Cluster bus protocol version. */
 
 typedef struct {
     char sig[4];        /* Siganture "RCmb" (Redis Cluster message bus). */
     uint32_t totlen;    /* Total length of this message */
     uint16_t ver;       /* Protocol version, currently set to 0. */
-    uint16_t notused0;  /* 2 bytes not used. */
+    uint16_t port;      /* TCP base port number. */
     uint16_t type;      /* Message type */
     uint16_t count;     /* Only used for some kind of messages. */
     uint64_t currentEpoch;  /* The epoch accordingly to the sending node. */
@@ -232,9 +238,10 @@ typedef struct {
     char sender[CLUSTER_NAMELEN]; /* Name of the sender node */
     unsigned char myslots[CLUSTER_SLOTS/8];
     char slaveof[CLUSTER_NAMELEN];
-    char notused1[32];  /* 32 bytes reserved for future usage. */
-    uint16_t port;      /* Sender TCP base port */
-    uint16_t flags;     /* Sender node flags */
+    char myip[NET_IP_STR_LEN];    /* Sender IP, if not all zeroed. */
+    char notused1[34];  /* 34 bytes reserved for future usage. */
+    uint16_t cport;      /* Sender TCP cluster bus port */
+    uint16_t flags;      /* Sender node flags */
     unsigned char state; /* Cluster state from the POV of the sender */
     unsigned char mflags[3]; /* Message flags: CLUSTERMSG_FLAG[012]_... */
     union clusterMsgData data;

src/config.c

@@ -45,9 +45,11 @@ typedef struct configEnum {
 
 configEnum maxmemory_policy_enum[] = {
     {"volatile-lru", MAXMEMORY_VOLATILE_LRU},
+    {"volatile-lfu", MAXMEMORY_VOLATILE_LFU},
     {"volatile-random",MAXMEMORY_VOLATILE_RANDOM},
     {"volatile-ttl",MAXMEMORY_VOLATILE_TTL},
     {"allkeys-lru",MAXMEMORY_ALLKEYS_LRU},
+    {"allkeys-lfu",MAXMEMORY_ALLKEYS_LFU},
     {"allkeys-random",MAXMEMORY_ALLKEYS_RANDOM},
     {"noeviction",MAXMEMORY_NO_EVICTION},
     {NULL, 0}
@@ -126,8 +128,8 @@ const char *configEnumGetNameOrUnknown(configEnum *ce, int val) {
 }
 
 /* Used for INFO generation. */
-const char *maxmemoryToString(void) {
-    return configEnumGetNameOrUnknown(maxmemory_policy_enum,server.maxmemory);
+const char *evictPolicyToString(void) {
+    return configEnumGetNameOrUnknown(maxmemory_policy_enum,server.maxmemory_policy);
 }
 
 /*-----------------------------------------------------------------------------
@@ -153,6 +155,20 @@ void resetServerSaveParams(void) {
     server.saveparamslen = 0;
 }
 
+void queueLoadModule(sds path, sds *argv, int argc) {
+    int i;
+    struct moduleLoadQueueEntry *loadmod;
+
+    loadmod = zmalloc(sizeof(struct moduleLoadQueueEntry));
+    loadmod->argv = zmalloc(sizeof(robj*)*argc);
+    loadmod->path = sdsnew(path);
+    loadmod->argc = argc;
+    for (i = 0; i < argc; i++) {
+        loadmod->argv[i] = createRawStringObject(argv[i],sdslen(argv[i]));
+    }
+    listAddNodeTail(server.loadmodule_queue,loadmod);
+}
+
 void loadServerConfigFromString(char *config) {
     char *err = NULL;
     int linenum = 0, totlines, i;
@@ -196,6 +212,10 @@ void loadServerConfigFromString(char *config) {
             if (server.tcpkeepalive < 0) {
                 err = "Invalid tcp-keepalive value"; goto loaderr;
             }
+        } else if (!strcasecmp(argv[0],"protected-mode") && argc == 2) {
+            if ((server.protected_mode = yesnotoi(argv[1])) == -1) {
+                err = "argument must be 'yes' or 'no'"; goto loaderr;
+            }
         } else if (!strcasecmp(argv[0],"port") && argc == 2) {
             server.port = atoi(argv[1]);
             if (server.port < 0 || server.port > 65535) {
@@ -304,6 +324,18 @@ void loadServerConfigFromString(char *config) {
                 err = "maxmemory-samples must be 1 or greater";
                 goto loaderr;
             }
+        } else if (!strcasecmp(argv[0],"lfu-log-factor") && argc == 2) {
+            server.lfu_log_factor = atoi(argv[1]);
+            if (server.maxmemory_samples < 0) {
+                err = "lfu-log-factor must be 0 or greater";
+                goto loaderr;
+            }
+        } else if (!strcasecmp(argv[0],"lfu-decay-time") && argc == 2) {
+            server.lfu_decay_time = atoi(argv[1]);
+            if (server.maxmemory_samples < 1) {
+                err = "lfu-decay-time must be 0 or greater";
+                goto loaderr;
+            }
         } else if (!strcasecmp(argv[0],"slaveof") && argc == 3) {
             slaveof_linenum = linenum;
             server.masterhost = sdsnew(argv[1]);
@@ -349,6 +381,7 @@ void loadServerConfigFromString(char *config) {
                 goto loaderr;
             }
         } else if (!strcasecmp(argv[0],"masterauth") && argc == 2) {
+            zfree(server.masterauth);
             server.masterauth = zstrdup(argv[1]);
         } else if (!strcasecmp(argv[0],"slave-serve-stale-data") && argc == 2) {
             if ((server.repl_serve_stale_data = yesnotoi(argv[1])) == -1) {
@@ -442,6 +475,10 @@ void loadServerConfigFromString(char *config) {
             if ((server.aof_load_truncated = yesnotoi(argv[1])) == -1) {
                 err = "argument must be 'yes' or 'no'"; goto loaderr;
             }
+        } else if (!strcasecmp(argv[0],"aof-use-rdb-preamble") && argc == 2) {
+            if ((server.aof_use_rdb_preamble = yesnotoi(argv[1])) == -1) {
+                err = "argument must be 'yes' or 'no'"; goto loaderr;
+            }
         } else if (!strcasecmp(argv[0],"requirepass") && argc == 2) {
             if (strlen(argv[1]) > CONFIG_AUTHPASS_MAX_LEN) {
                 err = "Password is longer than CONFIG_AUTHPASS_MAX_LEN";
@@ -509,6 +546,25 @@ void loadServerConfigFromString(char *config) {
         } else if (!strcasecmp(argv[0],"cluster-config-file") && argc == 2) {
             zfree(server.cluster_configfile);
             server.cluster_configfile = zstrdup(argv[1]);
+        } else if (!strcasecmp(argv[0],"cluster-announce-ip") && argc == 2) {
+            zfree(server.cluster_announce_ip);
+            server.cluster_announce_ip = zstrdup(argv[1]);
+        } else if (!strcasecmp(argv[0],"cluster-announce-port") && argc == 2) {
+            server.cluster_announce_port = atoi(argv[1]);
+            if (server.cluster_announce_port < 0 ||
+                server.cluster_announce_port > 65535)
+            {
+                err = "Invalid port"; goto loaderr;
+            }
+        } else if (!strcasecmp(argv[0],"cluster-announce-bus-port") &&
+                   argc == 2)
+        {
+            server.cluster_announce_bus_port = atoi(argv[1]);
+            if (server.cluster_announce_bus_port < 0 ||
+                server.cluster_announce_bus_port > 65535)
+            {
+                err = "Invalid port"; goto loaderr;
+            }
         } else if (!strcasecmp(argv[0],"cluster-require-full-coverage") &&
                     argc == 2)
         {
@@ -560,8 +616,9 @@ void loadServerConfigFromString(char *config) {
             unsigned long long hard, soft;
             int soft_seconds;
 
-            if (class == -1) {
-                err = "Unrecognized client limit class";
+            if (class == -1 || class == CLIENT_TYPE_MASTER) {
+                err = "Unrecognized client limit class: the user specified "
+                "an invalid one, or 'master' which has no buffer limits.";
                 goto loaderr;
             }
             hard = memtoll(argv[2],NULL);
@@ -581,6 +638,16 @@ void loadServerConfigFromString(char *config) {
             }
         } else if (!strcasecmp(argv[0],"slave-priority") && argc == 2) {
             server.slave_priority = atoi(argv[1]);
+        } else if (!strcasecmp(argv[0],"slave-announce-ip") && argc == 2) {
+            zfree(server.slave_announce_ip);
+            server.slave_announce_ip = zstrdup(argv[1]);
+        } else if (!strcasecmp(argv[0],"slave-announce-port") && argc == 2) {
+            server.slave_announce_port = atoi(argv[1]);
+            if (server.slave_announce_port < 0 ||
+                server.slave_announce_port > 65535)
+            {
+                err = "Invalid port"; goto loaderr;
+            }
         } else if (!strcasecmp(argv[0],"min-slaves-to-write") && argc == 2) {
             server.repl_min_slaves_to_write = atoi(argv[1]);
             if (server.repl_min_slaves_to_write < 0) {
@@ -608,6 +675,8 @@ void loadServerConfigFromString(char *config) {
                     "Allowed values: 'upstart', 'systemd', 'auto', or 'no'";
                 goto loaderr;
             }
+        } else if (!strcasecmp(argv[0],"loadmodule") && argc >= 2) {
+            queueLoadModule(argv[1],&argv[2],argc-2);
         } else if (!strcasecmp(argv[0],"sentinel")) {
             /* argc == 1 is handled by main() as we need to enter the sentinel
              * mode ASAP. */
@@ -693,7 +762,7 @@ void loadServerConfig(char *filename, char *options) {
 
 #define config_set_numerical_field(_name,_var,min,max) \
     } else if (!strcasecmp(c->argv[2]->ptr,_name)) { \
-        if (getLongLongFromObject(o,&ll) == C_ERR || ll < 0) goto badfmt; \
+        if (getLongLongFromObject(o,&ll) == C_ERR) goto badfmt; \
         if (min != LLONG_MIN && ll < min) goto badfmt; \
         if (max != LLONG_MAX && ll > max) goto badfmt; \
         _var = ll;
@@ -740,6 +809,9 @@ void configSetCommand(client *c) {
     } config_set_special_field("masterauth") {
         zfree(server.masterauth);
         server.masterauth = ((char*)o->ptr)[0] ? zstrdup(o->ptr) : NULL;
+    } config_set_special_field("cluster-announce-ip") {
+        zfree(server.cluster_announce_ip);
+        server.cluster_announce_ip = ((char*)o->ptr)[0] ? zstrdup(o->ptr) : NULL;
     } config_set_special_field("maxclients") {
         int orig_value = server.maxclients;
 
@@ -835,7 +907,8 @@ void configSetCommand(client *c) {
             long val;
 
             if ((j % 4) == 0) {
-                if (getClientTypeByName(v[j]) == -1) {
+                int class = getClientTypeByName(v[j]);
+                if (class == -1 || class == CLIENT_TYPE_MASTER) {
                     sdsfreesplitres(v,vlen);
                     goto badfmt;
                 }
@@ -868,6 +941,9 @@ void configSetCommand(client *c) {
 
         if (flags == -1) goto badfmt;
         server.notify_keyspace_events = flags;
+    } config_set_special_field("slave-announce-ip") {
+        zfree(server.slave_announce_ip);
+        server.slave_announce_ip = ((char*)o->ptr)[0] ? zstrdup(o->ptr) : NULL;
 
     /* Boolean fields.
      * config_set_bool_field(name,var). */
@@ -883,12 +959,16 @@ void configSetCommand(client *c) {
       "aof-rewrite-incremental-fsync",server.aof_rewrite_incremental_fsync) {
     } config_set_bool_field(
       "aof-load-truncated",server.aof_load_truncated) {
+    } config_set_bool_field(
+      "aof-use-rdb-preamble",server.aof_use_rdb_preamble) {
     } config_set_bool_field(
       "slave-serve-stale-data",server.repl_serve_stale_data) {
     } config_set_bool_field(
       "slave-read-only",server.repl_slave_ro) {
     } config_set_bool_field(
       "activerehashing",server.activerehashing) {
+    } config_set_bool_field(
+      "protected-mode",server.protected_mode) {
     } config_set_bool_field(
       "stop-writes-on-bgsave-error",server.stop_writes_on_bgsave_err) {
     } config_set_bool_field(
@@ -899,6 +979,8 @@ void configSetCommand(client *c) {
       "lazyfree-lazy-server-del",server.lazyfree_lazy_server_del) {
     } config_set_bool_field(
       "slave-lazy-flush",server.repl_slave_lazy_flush) {
+    } config_set_bool_field(
+      "no-appendfsync-on-rewrite",server.aof_no_fsync_on_rewrite) {
 
     /* Numerical fields.
      * config_set_numerical_field(name,var,min,max) */
@@ -906,6 +988,10 @@ void configSetCommand(client *c) {
       "tcp-keepalive",server.tcpkeepalive,0,LLONG_MAX) {
     } config_set_numerical_field(
       "maxmemory-samples",server.maxmemory_samples,1,LLONG_MAX) {
+    } config_set_numerical_field(
+      "lfu-log-factor",server.lfu_log_factor,0,LLONG_MAX) {
+    } config_set_numerical_field(
+      "lfu-decay-time",server.lfu_decay_time,0,LLONG_MAX) {
     } config_set_numerical_field(
       "timeout",server.maxidletime,0,LONG_MAX) {
     } config_set_numerical_field(
@@ -917,9 +1003,9 @@ void configSetCommand(client *c) {
     } config_set_numerical_field(
       "hash-max-ziplist-value",server.hash_max_ziplist_value,0,LLONG_MAX) {
     } config_set_numerical_field(
-      "list-max-ziplist-size",server.list_max_ziplist_size,0,LLONG_MAX) {
+      "list-max-ziplist-size",server.list_max_ziplist_size,INT_MIN,INT_MAX) {
     } config_set_numerical_field(
-      "list-compress-depth",server.list_compress_depth,0,LLONG_MAX) {
+      "list-compress-depth",server.list_compress_depth,0,INT_MAX) {
     } config_set_numerical_field(
       "set-max-intset-entries",server.set_max_intset_entries,0,LLONG_MAX) {
     } config_set_numerical_field(
@@ -948,6 +1034,8 @@ void configSetCommand(client *c) {
       "repl-diskless-sync-delay",server.repl_diskless_sync_delay,0,LLONG_MAX) {
     } config_set_numerical_field(
       "slave-priority",server.slave_priority,0,LLONG_MAX) {
+    } config_set_numerical_field(
+      "slave-announce-port",server.slave_announce_port,0,65535) {
     } config_set_numerical_field(
       "min-slaves-to-write",server.repl_min_slaves_to_write,0,LLONG_MAX) {
         refreshGoodSlavesCount();
@@ -956,6 +1044,10 @@ void configSetCommand(client *c) {
         refreshGoodSlavesCount();
     } config_set_numerical_field(
       "cluster-node-timeout",server.cluster_node_timeout,0,LLONG_MAX) {
+    } config_set_numerical_field(
+      "cluster-announce-port",server.cluster_announce_port,0,65535) {
+    } config_set_numerical_field(
+      "cluster-announce-bus-port",server.cluster_announce_bus_port,0,65535) {
     } config_set_numerical_field(
       "cluster-migration-barrier",server.cluster_migration_barrier,0,LLONG_MAX){
     } config_set_numerical_field(
@@ -1016,7 +1108,7 @@ badfmt: /* Bad format errors */
  *----------------------------------------------------------------------------*/
 
 #define config_get_string_field(_name,_var) do { \
-    if (stringmatch(pattern,_name,0)) { \
+    if (stringmatch(pattern,_name,1)) { \
         addReplyBulkCString(c,_name); \
         addReplyBulkCString(c,_var ? _var : ""); \
         matches++; \
@@ -1024,7 +1116,7 @@ badfmt: /* Bad format errors */
 } while(0);
 
 #define config_get_bool_field(_name,_var) do { \
-    if (stringmatch(pattern,_name,0)) { \
+    if (stringmatch(pattern,_name,1)) { \
         addReplyBulkCString(c,_name); \
         addReplyBulkCString(c,_var ? "yes" : "no"); \
         matches++; \
@@ -1032,7 +1124,7 @@ badfmt: /* Bad format errors */
 } while(0);
 
 #define config_get_numerical_field(_name,_var) do { \
-    if (stringmatch(pattern,_name,0)) { \
+    if (stringmatch(pattern,_name,1)) { \
         ll2string(buf,sizeof(buf),_var); \
         addReplyBulkCString(c,_name); \
         addReplyBulkCString(c,buf); \
@@ -1041,7 +1133,7 @@ badfmt: /* Bad format errors */
 } while(0);
 
 #define config_get_enum_field(_name,_var,_enumvar) do { \
-    if (stringmatch(pattern,_name,0)) { \
+    if (stringmatch(pattern,_name,1)) { \
         addReplyBulkCString(c,_name); \
         addReplyBulkCString(c,configEnumGetNameOrUnknown(_enumvar,_var)); \
         matches++; \
@@ -1060,9 +1152,11 @@ void configGetCommand(client *c) {
     config_get_string_field("dbfilename",server.rdb_filename);
     config_get_string_field("requirepass",server.requirepass);
     config_get_string_field("masterauth",server.masterauth);
+    config_get_string_field("cluster-announce-ip",server.cluster_announce_ip);
     config_get_string_field("unixsocket",server.unixsocket);
     config_get_string_field("logfile",server.logfile);
     config_get_string_field("pidfile",server.pidfile);
+    config_get_string_field("slave-announce-ip",server.slave_announce_ip);
 
     /* Numerical values */
     config_get_numerical_field("maxmemory",server.maxmemory);
@@ -1096,6 +1190,8 @@ void configGetCommand(client *c) {
     config_get_numerical_field("slowlog-max-len",
             server.slowlog_max_len);
     config_get_numerical_field("port",server.port);
+    config_get_numerical_field("cluster-announce-port",server.cluster_announce_port);
+    config_get_numerical_field("cluster-announce-bus-port",server.cluster_announce_bus_port);
     config_get_numerical_field("tcp-backlog",server.tcp_backlog);
     config_get_numerical_field("databases",server.dbnum);
     config_get_numerical_field("repl-ping-slave-period",server.repl_ping_slave_period);
@@ -1105,6 +1201,7 @@ void configGetCommand(client *c) {
     config_get_numerical_field("maxclients",server.maxclients);
     config_get_numerical_field("watchdog-period",server.watchdog_period);
     config_get_numerical_field("slave-priority",server.slave_priority);
+    config_get_numerical_field("slave-announce-port",server.slave_announce_port);
     config_get_numerical_field("min-slaves-to-write",server.repl_min_slaves_to_write);
     config_get_numerical_field("min-slaves-max-lag",server.repl_min_slaves_max_lag);
     config_get_numerical_field("hz",server.hz);
@@ -1129,6 +1226,7 @@ void configGetCommand(client *c) {
     config_get_bool_field("rdbcompression", server.rdb_compression);
     config_get_bool_field("rdbchecksum", server.rdb_checksum);
     config_get_bool_field("activerehashing", server.activerehashing);
+    config_get_bool_field("protected-mode", server.protected_mode);
     config_get_bool_field("repl-disable-tcp-nodelay",
             server.repl_disable_tcp_nodelay);
     config_get_bool_field("repl-diskless-sync",
@@ -1137,6 +1235,8 @@ void configGetCommand(client *c) {
             server.aof_rewrite_incremental_fsync);
     config_get_bool_field("aof-load-truncated",
             server.aof_load_truncated);
+    config_get_bool_field("aof-use-rdb-preamble",
+            server.aof_use_rdb_preamble);
     config_get_bool_field("lazyfree-lazy-eviction",
             server.lazyfree_lazy_eviction);
     config_get_bool_field("lazyfree-lazy-expire",
@@ -1160,12 +1260,12 @@ void configGetCommand(client *c) {
 
     /* Everything we can't handle with macros follows. */
 
-    if (stringmatch(pattern,"appendonly",0)) {
+    if (stringmatch(pattern,"appendonly",1)) {
         addReplyBulkCString(c,"appendonly");
         addReplyBulkCString(c,server.aof_state == AOF_OFF ? "no" : "yes");
         matches++;
     }
-    if (stringmatch(pattern,"dir",0)) {
+    if (stringmatch(pattern,"dir",1)) {
         char buf[1024];
 
         if (getcwd(buf,sizeof(buf)) == NULL)
@@ -1175,7 +1275,7 @@ void configGetCommand(client *c) {
         addReplyBulkCString(c,buf);
         matches++;
     }
-    if (stringmatch(pattern,"save",0)) {
+    if (stringmatch(pattern,"save",1)) {
         sds buf = sdsempty();
         int j;
 
@@ -1191,7 +1291,7 @@ void configGetCommand(client *c) {
         sdsfree(buf);
         matches++;
     }
-    if (stringmatch(pattern,"client-output-buffer-limit",0)) {
+    if (stringmatch(pattern,"client-output-buffer-limit",1)) {
         sds buf = sdsempty();
         int j;
 
@@ -1209,14 +1309,14 @@ void configGetCommand(client *c) {
         sdsfree(buf);
         matches++;
     }
-    if (stringmatch(pattern,"unixsocketperm",0)) {
+    if (stringmatch(pattern,"unixsocketperm",1)) {
         char buf[32];
         snprintf(buf,sizeof(buf),"%o",server.unixsocketperm);
         addReplyBulkCString(c,"unixsocketperm");
         addReplyBulkCString(c,buf);
         matches++;
     }
-    if (stringmatch(pattern,"slaveof",0)) {
+    if (stringmatch(pattern,"slaveof",1)) {
         char buf[256];
 
         addReplyBulkCString(c,"slaveof");
@@ -1228,7 +1328,7 @@ void configGetCommand(client *c) {
         addReplyBulkCString(c,buf);
         matches++;
     }
-    if (stringmatch(pattern,"notify-keyspace-events",0)) {
+    if (stringmatch(pattern,"notify-keyspace-events",1)) {
         robj *flagsobj = createObject(OBJ_STRING,
             keyspaceEventsFlagsToString(server.notify_keyspace_events));
 
@@ -1237,7 +1337,7 @@ void configGetCommand(client *c) {
         decrRefCount(flagsobj);
         matches++;
     }
-    if (stringmatch(pattern,"bind",0)) {
+    if (stringmatch(pattern,"bind",1)) {
         sds aux = sdsjoin(server.bindaddr,server.bindaddr_count," ");
 
         addReplyBulkCString(c,"bind");
@@ -1784,12 +1884,15 @@ int rewriteConfig(char *path) {
     rewriteConfigYesNoOption(state,"daemonize",server.daemonize,0);
     rewriteConfigStringOption(state,"pidfile",server.pidfile,CONFIG_DEFAULT_PID_FILE);
     rewriteConfigNumericalOption(state,"port",server.port,CONFIG_DEFAULT_SERVER_PORT);
+    rewriteConfigNumericalOption(state,"cluster-announce-port",server.cluster_announce_port,CONFIG_DEFAULT_CLUSTER_ANNOUNCE_PORT);
+    rewriteConfigNumericalOption(state,"cluster-announce-bus-port",server.cluster_announce_bus_port,CONFIG_DEFAULT_CLUSTER_ANNOUNCE_BUS_PORT);
     rewriteConfigNumericalOption(state,"tcp-backlog",server.tcp_backlog,CONFIG_DEFAULT_TCP_BACKLOG);
     rewriteConfigBindOption(state);
     rewriteConfigStringOption(state,"unixsocket",server.unixsocket,NULL);
     rewriteConfigOctalOption(state,"unixsocketperm",server.unixsocketperm,CONFIG_DEFAULT_UNIX_SOCKET_PERM);
     rewriteConfigNumericalOption(state,"timeout",server.maxidletime,CONFIG_DEFAULT_CLIENT_TIMEOUT);
     rewriteConfigNumericalOption(state,"tcp-keepalive",server.tcpkeepalive,CONFIG_DEFAULT_TCP_KEEPALIVE);
+    rewriteConfigNumericalOption(state,"slave-announce-port",server.slave_announce_port,CONFIG_DEFAULT_SLAVE_ANNOUNCE_PORT);
     rewriteConfigEnumOption(state,"loglevel",server.verbosity,loglevel_enum,CONFIG_DEFAULT_VERBOSITY);
     rewriteConfigStringOption(state,"logfile",server.logfile,CONFIG_DEFAULT_LOGFILE);
     rewriteConfigYesNoOption(state,"syslog-enabled",server.syslog_enabled,CONFIG_DEFAULT_SYSLOG_ENABLED);
@@ -1803,7 +1906,9 @@ int rewriteConfig(char *path) {
     rewriteConfigStringOption(state,"dbfilename",server.rdb_filename,CONFIG_DEFAULT_RDB_FILENAME);
     rewriteConfigDirOption(state);
     rewriteConfigSlaveofOption(state);
+    rewriteConfigStringOption(state,"slave-announce-ip",server.slave_announce_ip,CONFIG_DEFAULT_SLAVE_ANNOUNCE_IP);
     rewriteConfigStringOption(state,"masterauth",server.masterauth,NULL);
+    rewriteConfigStringOption(state,"cluster-announce-ip",server.cluster_announce_ip,NULL);
     rewriteConfigYesNoOption(state,"slave-serve-stale-data",server.repl_serve_stale_data,CONFIG_DEFAULT_SLAVE_SERVE_STALE_DATA);
     rewriteConfigYesNoOption(state,"slave-read-only",server.repl_slave_ro,CONFIG_DEFAULT_SLAVE_READ_ONLY);
     rewriteConfigNumericalOption(state,"repl-ping-slave-period",server.repl_ping_slave_period,CONFIG_DEFAULT_REPL_PING_SLAVE_PERIOD);
@@ -1847,10 +1952,12 @@ int rewriteConfig(char *path) {
     rewriteConfigNumericalOption(state,"zset-max-ziplist-value",server.zset_max_ziplist_value,OBJ_ZSET_MAX_ZIPLIST_VALUE);
     rewriteConfigNumericalOption(state,"hll-sparse-max-bytes",server.hll_sparse_max_bytes,CONFIG_DEFAULT_HLL_SPARSE_MAX_BYTES);
     rewriteConfigYesNoOption(state,"activerehashing",server.activerehashing,CONFIG_DEFAULT_ACTIVE_REHASHING);
+    rewriteConfigYesNoOption(state,"protected-mode",server.protected_mode,CONFIG_DEFAULT_PROTECTED_MODE);
     rewriteConfigClientoutputbufferlimitOption(state);
     rewriteConfigNumericalOption(state,"hz",server.hz,CONFIG_DEFAULT_HZ);
     rewriteConfigYesNoOption(state,"aof-rewrite-incremental-fsync",server.aof_rewrite_incremental_fsync,CONFIG_DEFAULT_AOF_REWRITE_INCREMENTAL_FSYNC);
     rewriteConfigYesNoOption(state,"aof-load-truncated",server.aof_load_truncated,CONFIG_DEFAULT_AOF_LOAD_TRUNCATED);
+    rewriteConfigYesNoOption(state,"aof-use-rdb-preamble",server.aof_use_rdb_preamble,CONFIG_DEFAULT_AOF_USE_RDB_PREAMBLE);
     rewriteConfigEnumOption(state,"supervised",server.supervised_mode,supervised_mode_enum,SUPERVISED_NONE);
     rewriteConfigYesNoOption(state,"lazyfree-lazy-eviction",server.lazyfree_lazy_eviction,CONFIG_DEFAULT_LAZYFREE_LAZY_EVICTION);
     rewriteConfigYesNoOption(state,"lazyfree-lazy-expire",server.lazyfree_lazy_expire,CONFIG_DEFAULT_LAZYFREE_LAZY_EXPIRE);
@@ -1880,6 +1987,12 @@ int rewriteConfig(char *path) {
  *----------------------------------------------------------------------------*/
 
 void configCommand(client *c) {
+    /* Only allow CONFIG GET while loading. */
+    if (server.loading && strcasecmp(c->argv[1]->ptr,"get")) {
+        addReplyError(c,"Only CONFIG GET is allowed during loading");
+        return;
+    }
+
     if (!strcasecmp(c->argv[1]->ptr,"set")) {
         if (c->argc != 4) goto badarity;
         configSetCommand(c);

src/db.c

@@ -38,7 +38,10 @@
  * C-level DB API
  *----------------------------------------------------------------------------*/
 
-robj *lookupKey(redisDb *db, robj *key) {
+/* Low level key lookup API, not actually called directly from commands
+ * implementations that should instead rely on lookupKeyRead(),
+ * lookupKeyWrite() and lookupKeyReadWithFlags(). */
+robj *lookupKey(redisDb *db, robj *key, int flags) {
     dictEntry *de = dictFind(db->dict,key->ptr);
     if (de) {
         robj *val = dictGetVal(de);
@@ -46,15 +49,46 @@ robj *lookupKey(redisDb *db, robj *key) {
         /* Update the access time for the ageing algorithm.
          * Don't do it if we have a saving child, as this will trigger
          * a copy on write madness. */
-        if (server.rdb_child_pid == -1 && server.aof_child_pid == -1)
-            val->lru = LRU_CLOCK();
+        if (server.rdb_child_pid == -1 &&
+            server.aof_child_pid == -1 &&
+            !(flags & LOOKUP_NOTOUCH))
+        {
+            if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) {
+                unsigned long ldt = val->lru >> 8;
+                unsigned long counter = LFULogIncr(val->lru & 255);
+                val->lru = (ldt << 8) | counter;
+            } else {
+                val->lru = LRU_CLOCK();
+            }
+        }
         return val;
     } else {
         return NULL;
     }
 }
 
-robj *lookupKeyRead(redisDb *db, robj *key) {
+/* Lookup a key for read operations, or return NULL if the key is not found
+ * in the specified DB.
+ *
+ * As a side effect of calling this function:
+ * 1. A key gets expired if it reached it's TTL.
+ * 2. The key last access time is updated.
+ * 3. The global keys hits/misses stats are updated (reported in INFO).
+ *
+ * This API should not be used when we write to the key after obtaining
+ * the object linked to the key, but only for read only operations.
+ *
+ * Flags change the behavior of this command:
+ *
+ *  LOOKUP_NONE (or zero): no special flags are passed.
+ *  LOOKUP_NOTOUCH: don't alter the last access time of the key.
+ *
+ * Note: this function also returns NULL is the key is logically expired
+ * but still existing, in case this is a slave, since this API is called only
+ * for read operations. Even if the key expiry is master-driven, we can
+ * correctly report a key is expired on slaves even if the master is lagging
+ * expiring our key via DELs in the replication link. */
+robj *lookupKeyReadWithFlags(redisDb *db, robj *key, int flags) {
     robj *val;
 
     if (expireIfNeeded(db,key) == 1) {
@@ -83,7 +117,7 @@ robj *lookupKeyRead(redisDb *db, robj *key) {
             return NULL;
         }
     }
-    val = lookupKey(db,key);
+    val = lookupKey(db,key,flags);
     if (val == NULL)
         server.stat_keyspace_misses++;
     else
@@ -91,9 +125,20 @@ robj *lookupKeyRead(redisDb *db, robj *key) {
     return val;
 }
 
+/* Like lookupKeyReadWithFlags(), but does not use any flag, which is the
+ * common case. */
+robj *lookupKeyRead(redisDb *db, robj *key) {
+    return lookupKeyReadWithFlags(db,key,LOOKUP_NONE);
+}
+
+/* Lookup a key for write operations, and as a side effect, if needed, expires
+ * the key if its TTL is reached.
+ *
+ * Returns the linked value object if the key exists or NULL if the key
+ * does not exist in the specified DB. */
 robj *lookupKeyWrite(redisDb *db, robj *key) {
     expireIfNeeded(db,key);
-    return lookupKey(db,key);
+    return lookupKey(db,key,LOOKUP_NONE);
 }
 
 robj *lookupKeyReadOrReply(client *c, robj *key, robj *reply) {
@@ -116,7 +161,7 @@ void dbAdd(redisDb *db, robj *key, robj *val) {
     sds copy = sdsdup(key->ptr);
     int retval = dictAdd(db->dict, copy, val);
 
-    serverAssertWithInfo(NULL,key,retval == C_OK);
+    serverAssertWithInfo(NULL,key,retval == DICT_OK);
     if (val->type == OBJ_LIST) signalListAsReady(db, key);
     if (server.cluster_enabled) slotToKeyAdd(key);
  }
@@ -130,7 +175,14 @@ void dbOverwrite(redisDb *db, robj *key, robj *val) {
     dictEntry *de = dictFind(db->dict,key->ptr);
 
     serverAssertWithInfo(NULL,key,de != NULL);
-    dictReplace(db->dict, key->ptr, val);
+    if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) {
+        robj *old = dictGetVal(de);
+        int saved_lru = old->lru;
+        dictReplace(db->dict, key->ptr, val);
+        val->lru = saved_lru;
+    } else {
+        dictReplace(db->dict, key->ptr, val);
+    }
 }
 
 /* High level Set operation. This function can be used in order to set
@@ -721,7 +773,7 @@ void typeCommand(client *c) {
     robj *o;
     char *type;
 
-    o = lookupKeyRead(c->db,c->argv[1]);
+    o = lookupKeyReadWithFlags(c->db,c->argv[1],LOOKUP_NOTOUCH);
     if (o == NULL) {
         type = "none";
     } else {
@@ -731,6 +783,10 @@ void typeCommand(client *c) {
         case OBJ_SET: type = "set"; break;
         case OBJ_ZSET: type = "zset"; break;
         case OBJ_HASH: type = "hash"; break;
+        case OBJ_MODULE: {
+            moduleValue *mv = o->ptr;
+            type = mv->type->name;
+        }; break;
         default: type = "unknown"; break;
         }
     }
@@ -887,7 +943,7 @@ void setExpire(redisDb *db, robj *key, long long when) {
     /* Reuse the sds from the main dict in the expire dict */
     kde = dictFind(db->dict,key->ptr);
     serverAssertWithInfo(NULL,key,kde != NULL);
-    de = dictReplaceRaw(db->expires,dictGetKey(kde));
+    de = dictAddOrFind(db->expires,dictGetKey(kde));
     dictSetSignedIntegerVal(de,when);
 }
 
@@ -967,126 +1023,6 @@ int expireIfNeeded(redisDb *db, robj *key) {
                                          dbSyncDelete(db,key);
 }
 
-/*-----------------------------------------------------------------------------
- * Expires Commands
- *----------------------------------------------------------------------------*/
-
-/* This is the generic command implementation for EXPIRE, PEXPIRE, EXPIREAT
- * and PEXPIREAT. Because the commad second argument may be relative or absolute
- * the "basetime" argument is used to signal what the base time is (either 0
- * for *AT variants of the command, or the current time for relative expires).
- *
- * unit is either UNIT_SECONDS or UNIT_MILLISECONDS, and is only used for
- * the argv[2] parameter. The basetime is always specified in milliseconds. */
-void expireGenericCommand(client *c, long long basetime, int unit) {
-    robj *key = c->argv[1], *param = c->argv[2];
-    long long when; /* unix time in milliseconds when the key will expire. */
-
-    if (getLongLongFromObjectOrReply(c, param, &when, NULL) != C_OK)
-        return;
-
-    if (unit == UNIT_SECONDS) when *= 1000;
-    when += basetime;
-
-    /* No key, return zero. */
-    if (lookupKeyWrite(c->db,key) == NULL) {
-        addReply(c,shared.czero);
-        return;
-    }
-
-    /* EXPIRE with negative TTL, or EXPIREAT with a timestamp into the past
-     * should never be executed as a DEL when load the AOF or in the context
-     * of a slave instance.
-     *
-     * Instead we take the other branch of the IF statement setting an expire
-     * (possibly in the past) and wait for an explicit DEL from the master. */
-    if (when <= mstime() && !server.loading && !server.masterhost) {
-        robj *aux;
-
-        int deleted = server.lazyfree_lazy_expire ? dbAsyncDelete(c->db,key) :
-                                                    dbSyncDelete(c->db,key);
-        serverAssertWithInfo(c,key,deleted);
-        server.dirty++;
-
-        /* Replicate/AOF this as an explicit DEL or UNLINK. */
-        aux = server.lazyfree_lazy_expire ? shared.unlink : shared.del;
-        rewriteClientCommandVector(c,2,aux,key);
-        signalModifiedKey(c->db,key);
-        notifyKeyspaceEvent(NOTIFY_GENERIC,"del",key,c->db->id);
-        addReply(c, shared.cone);
-        return;
-    } else {
-        setExpire(c->db,key,when);
-        addReply(c,shared.cone);
-        signalModifiedKey(c->db,key);
-        notifyKeyspaceEvent(NOTIFY_GENERIC,"expire",key,c->db->id);
-        server.dirty++;
-        return;
-    }
-}
-
-void expireCommand(client *c) {
-    expireGenericCommand(c,mstime(),UNIT_SECONDS);
-}
-
-void expireatCommand(client *c) {
-    expireGenericCommand(c,0,UNIT_SECONDS);
-}
-
-void pexpireCommand(client *c) {
-    expireGenericCommand(c,mstime(),UNIT_MILLISECONDS);
-}
-
-void pexpireatCommand(client *c) {
-    expireGenericCommand(c,0,UNIT_MILLISECONDS);
-}
-
-void ttlGenericCommand(client *c, int output_ms) {
-    long long expire, ttl = -1;
-
-    /* If the key does not exist at all, return -2 */
-    if (lookupKeyRead(c->db,c->argv[1]) == NULL) {
-        addReplyLongLong(c,-2);
-        return;
-    }
-    /* The key exists. Return -1 if it has no expire, or the actual
-     * TTL value otherwise. */
-    expire = getExpire(c->db,c->argv[1]);
-    if (expire != -1) {
-        ttl = expire-mstime();
-        if (ttl < 0) ttl = 0;
-    }
-    if (ttl == -1) {
-        addReplyLongLong(c,-1);
-    } else {
-        addReplyLongLong(c,output_ms ? ttl : ((ttl+500)/1000));
-    }
-}
-
-void ttlCommand(client *c) {
-    ttlGenericCommand(c, 0);
-}
-
-void pttlCommand(client *c) {
-    ttlGenericCommand(c, 1);
-}
-
-void persistCommand(client *c) {
-    dictEntry *de;
-
-    de = dictFind(c->db->dict,c->argv[1]->ptr);
-    if (de == NULL) {
-        addReply(c,shared.czero);
-    } else {
-        if (removeExpire(c->db,c->argv[1])) {
-            addReply(c,shared.cone);
-            server.dirty++;
-        } else {
-            addReply(c,shared.czero);
-        }
-    }
-}
-
 /* -----------------------------------------------------------------------------
  * API to get key arguments from commands
  * ---------------------------------------------------------------------------*/
@@ -1124,7 +1060,9 @@ int *getKeysUsingCommandTable(struct redisCommand *cmd,robj **argv, int argc, in
  * This function uses the command table if a command-specific helper function
  * is not required, otherwise it calls the command-specific function. */
 int *getKeysFromCommand(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
-    if (cmd->getkeys_proc) {
+    if (cmd->flags & CMD_MODULE_GETKEYS) {
+        return moduleGetCommandKeysViaAPI(cmd,argv,argc,numkeys);
+    } else if (!(cmd->flags & CMD_MODULE) && cmd->getkeys_proc) {
         return cmd->getkeys_proc(cmd,argv,argc,numkeys);
     } else {
         return getKeysUsingCommandTable(cmd,argv,argc,numkeys);
@@ -1238,6 +1176,33 @@ int *sortGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys)
     return keys;
 }
 
+int *migrateGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
+    int i, num, first, *keys;
+    UNUSED(cmd);
+
+    /* Assume the obvious form. */
+    first = 3;
+    num = 1;
+
+    /* But check for the extended one with the KEYS option. */
+    if (argc > 6) {
+        for (i = 6; i < argc; i++) {
+            if (!strcasecmp(argv[i]->ptr,"keys") &&
+                sdslen(argv[3]->ptr) == 0)
+            {
+                first = i+1;
+                num = argc-first;
+                break;
+            }
+        }
+    }
+
+    keys = zmalloc(sizeof(int)*num);
+    for (i = 0; i < num; i++) keys[i] = first+i;
+    *numkeys = num;
+    return keys;
+}
+
 /* Slot to Key API. This is used by Redis Cluster in order to obtain in
  * a fast way a key that belongs to a specified hash slot. This is useful
  * while rehashing the cluster. */

src/debug.c

@@ -33,12 +33,14 @@
 
 #include <arpa/inet.h>
 #include <signal.h>
+#include <dlfcn.h>
 
 #ifdef HAVE_BACKTRACE
 #include <execinfo.h>
 #include <ucontext.h>
 #include <fcntl.h>
 #include "bio.h"
+#include <unistd.h>
 #endif /* HAVE_BACKTRACE */
 
 #ifdef __CYGWIN__
@@ -250,14 +252,51 @@ void computeDatasetDigest(unsigned char *final) {
     }
 }
 
-void inputCatSds(void *result, const char *str) {
-    /* result is actually a (sds *), so re-cast it here */
-    sds *info = (sds *)result;
-    *info = sdscat(*info, str);
-}
-
 void debugCommand(client *c) {
-    if (!strcasecmp(c->argv[1]->ptr,"segfault")) {
+    if (c->argc == 1) {
+        addReplyError(c,"You must specify a subcommand for DEBUG. Try DEBUG HELP for info.");
+        return;
+    }
+
+    if (!strcasecmp(c->argv[1]->ptr,"help")) {
+        void *blenp = addDeferredMultiBulkLength(c);
+        int blen = 0;
+        blen++; addReplyStatus(c,
+        "DEBUG <subcommand> arg arg ... arg. Subcommands:");
+        blen++; addReplyStatus(c,
+        "segfault -- Crash the server with sigsegv.");
+        blen++; addReplyStatus(c,
+        "restart  -- Graceful restart: save config, db, restart.");
+        blen++; addReplyStatus(c,
+        "crash-and-recovery <milliseconds> -- Hard crash and restart after <milliseconds> delay.");
+        blen++; addReplyStatus(c,
+        "assert   -- Crash by assertion failed.");
+        blen++; addReplyStatus(c,
+        "reload   -- Save the RDB on disk and reload it back in memory.");
+        blen++; addReplyStatus(c,
+        "loadaof  -- Flush the AOF buffers on disk and reload the AOF in memory.");
+        blen++; addReplyStatus(c,
+        "object <key> -- Show low level info about key and associated value.");
+        blen++; addReplyStatus(c,
+        "sdslen <key> -- Show low level SDS string info representing key and value.");
+        blen++; addReplyStatus(c,
+        "populate <count> [prefix] -- Create <count> string keys named key:<num>. If a prefix is specified is used instead of the 'key' prefix.");
+        blen++; addReplyStatus(c,
+        "digest   -- Outputs an hex signature representing the current DB content.");
+        blen++; addReplyStatus(c,
+        "sleep <seconds> -- Stop the server for <seconds>. Decimals allowed.");
+        blen++; addReplyStatus(c,
+        "set-active-expire (0|1) -- Setting it to 0 disables expiring keys in background when they are not accessed (otherwise the Redis behavior). Setting it to 1 reenables back the default.");
+        blen++; addReplyStatus(c,
+        "lua-always-replicate-commands (0|1) -- Setting it to 1 makes Lua replication defaulting to replicating single commands, without the script having to enable effects replication.");
+        blen++; addReplyStatus(c,
+        "error <string> -- Return a Redis protocol error with <string> as message. Useful for clients unit tests to simulate Redis errors.");
+        blen++; addReplyStatus(c,
+        "structsize -- Return the size of different Redis core C structures.");
+        blen++; addReplyStatus(c,
+        "htstats <dbid> -- Return hash table statistics of the specified Redis database.");
+        setDeferredMultiBulkLength(c,blenp,blen);
+    } else if (!strcasecmp(c->argv[1]->ptr,"segfault")) {
         *((char*)-1) = 'x';
     } else if (!strcasecmp(c->argv[1]->ptr,"restart") ||
                !strcasecmp(c->argv[1]->ptr,"crash-and-recover"))
@@ -370,12 +409,14 @@ void debugCommand(client *c) {
             addReplyError(c,"Not an sds encoded string.");
         } else {
             addReplyStatusFormat(c,
-                "key_sds_len:%lld, key_sds_avail:%lld, "
-                "val_sds_len:%lld, val_sds_avail:%lld",
+                "key_sds_len:%lld, key_sds_avail:%lld, key_zmalloc: %lld, "
+                "val_sds_len:%lld, val_sds_avail:%lld, val_zmalloc: %lld",
                 (long long) sdslen(key),
                 (long long) sdsavail(key),
+                (long long) sdsZmallocSize(key),
                 (long long) sdslen(val->ptr),
-                (long long) sdsavail(val->ptr));
+                (long long) sdsavail(val->ptr),
+                (long long) getStringObjectSdsUsedMemory(val));
         }
     } else if (!strcasecmp(c->argv[1]->ptr,"populate") &&
                (c->argc == 3 || c->argc == 4)) {
@@ -397,6 +438,7 @@ void debugCommand(client *c) {
             snprintf(buf,sizeof(buf),"value:%lu",j);
             val = createStringObject(buf,strlen(buf));
             dbAdd(c->db,key,val);
+            signalModifiedKey(c->db,key);
             decrRefCount(key);
         }
         addReply(c,shared.ok);
@@ -468,18 +510,6 @@ void debugCommand(client *c) {
         stats = sdscat(stats,buf);
 
         addReplyBulkSds(c,stats);
-    } else if (!strcasecmp(c->argv[1]->ptr,"jemalloc") && c->argc == 3) {
-#if defined(USE_JEMALLOC)
-        if (!strcasecmp(c->argv[2]->ptr, "info")) {
-            sds info = sdsempty();
-            je_malloc_stats_print(inputCatSds, &info, NULL);
-            addReplyBulkSds(c, info);
-        } else {
-            addReplyErrorFormat(c, "Valid jemalloc debug fields: info");
-        }
-#else
-        addReplyErrorFormat(c, "jemalloc support not available");
-#endif
     } else {
         addReplyErrorFormat(c, "Unknown DEBUG subcommand or wrong number of arguments for '%s'",
             (char*)c->argv[1]->ptr);
@@ -488,7 +518,7 @@ void debugCommand(client *c) {
 
 /* =========================== Crash handling  ============================== */
 
-void _serverAssert(char *estr, char *file, int line) {
+void _serverAssert(const char *estr, const char *file, int line) {
     bugReportStart();
     serverLog(LL_WARNING,"=== ASSERTION FAILED ===");
     serverLog(LL_WARNING,"==> %s:%d '%s' is not true",file,line,estr);
@@ -501,7 +531,7 @@ void _serverAssert(char *estr, char *file, int line) {
     *((char*)-1) = 'x';
 }
 
-void _serverAssertPrintClientInfo(client *c) {
+void _serverAssertPrintClientInfo(const client *c) {
     int j;
 
     bugReportStart();
@@ -516,7 +546,7 @@ void _serverAssertPrintClientInfo(client *c) {
         if (c->argv[j]->type == OBJ_STRING && sdsEncodedObject(c->argv[j])) {
             arg = (char*) c->argv[j]->ptr;
         } else {
-            snprintf(buf,sizeof(buf),"Object type: %d, encoding: %d",
+            snprintf(buf,sizeof(buf),"Object type: %u, encoding: %u",
                 c->argv[j]->type, c->argv[j]->encoding);
             arg = buf;
         }
@@ -525,7 +555,7 @@ void _serverAssertPrintClientInfo(client *c) {
     }
 }
 
-void serverLogObjectDebugInfo(robj *o) {
+void serverLogObjectDebugInfo(const robj *o) {
     serverLog(LL_WARNING,"Object type: %d", o->type);
     serverLog(LL_WARNING,"Object encoding: %d", o->encoding);
     serverLog(LL_WARNING,"Object refcount: %d", o->refcount);
@@ -545,23 +575,23 @@ void serverLogObjectDebugInfo(robj *o) {
     } else if (o->type == OBJ_ZSET) {
         serverLog(LL_WARNING,"Sorted set size: %d", (int) zsetLength(o));
         if (o->encoding == OBJ_ENCODING_SKIPLIST)
-            serverLog(LL_WARNING,"Skiplist level: %d", (int) ((zset*)o->ptr)->zsl->level);
+            serverLog(LL_WARNING,"Skiplist level: %d", (int) ((const zset*)o->ptr)->zsl->level);
     }
 }
 
-void _serverAssertPrintObject(robj *o) {
+void _serverAssertPrintObject(const robj *o) {
     bugReportStart();
     serverLog(LL_WARNING,"=== ASSERTION FAILED OBJECT CONTEXT ===");
     serverLogObjectDebugInfo(o);
 }
 
-void _serverAssertWithInfo(client *c, robj *o, char *estr, char *file, int line) {
+void _serverAssertWithInfo(const client *c, const robj *o, const char *estr, const char *file, int line) {
     if (c) _serverAssertPrintClientInfo(c);
     if (o) _serverAssertPrintObject(o);
     _serverAssert(estr,file,line);
 }
 
-void _serverPanic(char *msg, char *file, int line) {
+void _serverPanic(const char *msg, const char *file, int line) {
     bugReportStart();
     serverLog(LL_WARNING,"------------------------------------------------");
     serverLog(LL_WARNING,"!!! Software Failure. Press left mouse button to continue");
@@ -575,8 +605,8 @@ void _serverPanic(char *msg, char *file, int line) {
 
 void bugReportStart(void) {
     if (server.bug_report_start == 0) {
-        serverLog(LL_WARNING,
-            "\n\n=== REDIS BUG REPORT START: Cut & paste starting from here ===");
+        serverLogRaw(LL_WARNING|LL_RAW,
+        "\n\n=== REDIS BUG REPORT START: Cut & paste starting from here ===\n");
         server.bug_report_start = 1;
     }
 }
@@ -607,6 +637,8 @@ static void *getMcontextEip(ucontext_t *uc) {
     return (void*) uc->uc_mcontext.gregs[16]; /* Linux 64 */
     #elif defined(__ia64__) /* Linux IA64 */
     return (void*) uc->uc_mcontext.sc_ip;
+    #elif defined(__arm__) /* Linux ARM */
+    return (void*) uc->uc_mcontext.arm_pc;
     #endif
 #else
     return NULL;
@@ -627,7 +659,7 @@ void logStackContent(void **sp) {
 }
 
 void logRegisters(ucontext_t *uc) {
-    serverLog(LL_WARNING, "--- REGISTERS");
+    serverLog(LL_WARNING|LL_RAW, "\n------ REGISTERS ------\n");
 
 /* OSX */
 #if defined(__APPLE__) && defined(MAC_OS_X_VERSION_10_6)
@@ -755,31 +787,51 @@ void logRegisters(ucontext_t *uc) {
 #endif
 }
 
+/* Return a file descriptor to write directly to the Redis log with the
+ * write(2) syscall, that can be used in critical sections of the code
+ * where the rest of Redis can't be trusted (for example during the memory
+ * test) or when an API call requires a raw fd.
+ *
+ * Close it with closeDirectLogFiledes(). */
+int openDirectLogFiledes(void) {
+    int log_to_stdout = server.logfile[0] == '\0';
+    int fd = log_to_stdout ?
+        STDOUT_FILENO :
+        open(server.logfile, O_APPEND|O_CREAT|O_WRONLY, 0644);
+    return fd;
+}
+
+/* Used to close what closeDirectLogFiledes() returns. */
+void closeDirectLogFiledes(int fd) {
+    int log_to_stdout = server.logfile[0] == '\0';
+    if (!log_to_stdout) close(fd);
+}
+
 /* Logs the stack trace using the backtrace() call. This function is designed
  * to be called from signal handlers safely. */
 void logStackTrace(ucontext_t *uc) {
-    void *trace[100];
-    int trace_size = 0, fd;
-    int log_to_stdout = server.logfile[0] == '\0';
+    void *trace[101];
+    int trace_size = 0, fd = openDirectLogFiledes();
 
-    /* Open the log file in append mode. */
-    fd = log_to_stdout ?
-        STDOUT_FILENO :
-        open(server.logfile, O_APPEND|O_CREAT|O_WRONLY, 0644);
-    if (fd == -1) return;
+    if (fd == -1) return; /* If we can't log there is anything to do. */
 
     /* Generate the stack trace */
-    trace_size = backtrace(trace, 100);
+    trace_size = backtrace(trace+1, 100);
 
-    /* overwrite sigaction with caller's address */
-    if (getMcontextEip(uc) != NULL)
-        trace[1] = getMcontextEip(uc);
+    if (getMcontextEip(uc) != NULL) {
+        char *msg1 = "EIP:\n";
+        char *msg2 = "\nBacktrace:\n";
+        if (write(fd,msg1,strlen(msg1)) == -1) {/* Avoid warning. */};
+        trace[0] = getMcontextEip(uc);
+        backtrace_symbols_fd(trace, 1, fd);
+        if (write(fd,msg2,strlen(msg2)) == -1) {/* Avoid warning. */};
+    }
 
     /* Write symbols to log file */
-    backtrace_symbols_fd(trace, trace_size, fd);
+    backtrace_symbols_fd(trace+1, trace_size, fd);
 
     /* Cleanup */
-    if (!log_to_stdout) close(fd);
+    closeDirectLogFiledes(fd);
 }
 
 /* Log information about the "current" client, that is, the client that is
@@ -792,15 +844,16 @@ void logCurrentClient(void) {
     sds client;
     int j;
 
-    serverLog(LL_WARNING, "--- CURRENT CLIENT INFO");
+    serverLogRaw(LL_WARNING|LL_RAW, "\n------ CURRENT CLIENT INFO ------\n");
     client = catClientInfoString(sdsempty(),cc);
-    serverLog(LL_WARNING,"client: %s", client);
+    serverLog(LL_WARNING|LL_RAW,"%s\n", client);
     sdsfree(client);
     for (j = 0; j < cc->argc; j++) {
         robj *decoded;
 
         decoded = getDecodedObject(cc->argv[j]);
-        serverLog(LL_WARNING,"argv[%d]: '%s'", j, (char*)decoded->ptr);
+        serverLog(LL_WARNING|LL_RAW,"argv[%d]: '%s'\n", j,
+            (char*)decoded->ptr);
         decrRefCount(decoded);
     }
     /* Check if the first argument, usually a key, is found inside the
@@ -821,19 +874,24 @@ void logCurrentClient(void) {
 }
 
 #if defined(HAVE_PROC_MAPS)
-void memtest_non_destructive_invert(void *addr, size_t size);
-void memtest_non_destructive_swap(void *addr, size_t size);
+
 #define MEMTEST_MAX_REGIONS 128
 
+/* A non destructive memory test executed during segfauls. */
 int memtest_test_linux_anonymous_maps(void) {
-    FILE *fp = fopen("/proc/self/maps","r");
+    FILE *fp;
     char line[1024];
+    char logbuf[1024];
     size_t start_addr, end_addr, size;
     size_t start_vect[MEMTEST_MAX_REGIONS];
     size_t size_vect[MEMTEST_MAX_REGIONS];
     int regions = 0, j;
-    uint64_t crc1 = 0, crc2 = 0, crc3 = 0;
 
+    int fd = openDirectLogFiledes();
+    if (!fd) return 0;
+
+    fp = fopen("/proc/self/maps","r");
+    if (!fp) return 0;
     while(fgets(line,sizeof(line),fp) != NULL) {
         char *start, *end, *p = line;
 
@@ -857,78 +915,92 @@ int memtest_test_linux_anonymous_maps(void) {
 
         start_vect[regions] = start_addr;
         size_vect[regions] = size;
-        printf("Testing %lx %lu\n", (unsigned long) start_vect[regions],
-                                    (unsigned long) size_vect[regions]);
+        snprintf(logbuf,sizeof(logbuf),
+            "*** Preparing to test memory region %lx (%lu bytes)\n",
+                (unsigned long) start_vect[regions],
+                (unsigned long) size_vect[regions]);
+        if (write(fd,logbuf,strlen(logbuf)) == -1) { /* Nothing to do. */ }
         regions++;
     }
 
-    /* Test all the regions as an unique sequential region.
-     * 1) Take the CRC64 of the memory region. */
+    int errors = 0;
     for (j = 0; j < regions; j++) {
-        crc1 = crc64(crc1,(void*)start_vect[j],size_vect[j]);
+        if (write(fd,".",1) == -1) { /* Nothing to do. */ }
+        errors += memtest_preserving_test((void*)start_vect[j],size_vect[j],1);
+        if (write(fd, errors ? "E" : "O",1) == -1) { /* Nothing to do. */ }
     }
-
-    /* 2) Invert bits, swap adjacent words, swap again, invert bits.
-     * This is the error amplification step. */
-    for (j = 0; j < regions; j++)
-        memtest_non_destructive_invert((void*)start_vect[j],size_vect[j]);
-    for (j = 0; j < regions; j++)
-        memtest_non_destructive_swap((void*)start_vect[j],size_vect[j]);
-    for (j = 0; j < regions; j++)
-        memtest_non_destructive_swap((void*)start_vect[j],size_vect[j]);
-    for (j = 0; j < regions; j++)
-        memtest_non_destructive_invert((void*)start_vect[j],size_vect[j]);
-
-    /* 3) Take the CRC64 sum again. */
-    for (j = 0; j < regions; j++)
-        crc2 = crc64(crc2,(void*)start_vect[j],size_vect[j]);
-
-    /* 4) Swap + Swap again */
-    for (j = 0; j < regions; j++)
-        memtest_non_destructive_swap((void*)start_vect[j],size_vect[j]);
-    for (j = 0; j < regions; j++)
-        memtest_non_destructive_swap((void*)start_vect[j],size_vect[j]);
-
-    /* 5) Take the CRC64 sum again. */
-    for (j = 0; j < regions; j++)
-        crc3 = crc64(crc3,(void*)start_vect[j],size_vect[j]);
+    if (write(fd,"\n",1) == -1) { /* Nothing to do. */ }
 
     /* NOTE: It is very important to close the file descriptor only now
      * because closing it before may result into unmapping of some memory
      * region that we are testing. */
     fclose(fp);
-
-    /* If the two CRC are not the same, we trapped a memory error. */
-    return crc1 != crc2 || crc2 != crc3;
+    closeDirectLogFiledes(fd);
+    return errors;
 }
 #endif
 
+/* Scans the (assumed) x86 code starting at addr, for a max of `len`
+ * bytes, searching for E8 (callq) opcodes, and dumping the symbols
+ * and the call offset if they appear to be valid. */
+void dumpX86Calls(void *addr, size_t len) {
+    size_t j;
+    unsigned char *p = addr;
+    Dl_info info;
+    /* Hash table to best-effort avoid printing the same symbol
+     * multiple times. */
+    unsigned long ht[256] = {0};
+
+    if (len < 5) return;
+    for (j = 0; j < len-4; j++) {
+        if (p[j] != 0xE8) continue; /* Not an E8 CALL opcode. */
+        unsigned long target = (unsigned long)addr+j+5;
+        target += *((int32_t*)(p+j+1));
+        if (dladdr((void*)target, &info) != 0 && info.dli_sname != NULL) {
+            if (ht[target&0xff] != target) {
+                printf("Function at 0x%lx is %s\n",target,info.dli_sname);
+                ht[target&0xff] = target;
+            }
+            j += 4; /* Skip the 32 bit immediate. */
+        }
+    }
+}
+
 void sigsegvHandler(int sig, siginfo_t *info, void *secret) {
     ucontext_t *uc = (ucontext_t*) secret;
+    void *eip = getMcontextEip(uc);
     sds infostring, clients;
     struct sigaction act;
     UNUSED(info);
 
     bugReportStart();
     serverLog(LL_WARNING,
-        "    Redis %s crashed by signal: %d", REDIS_VERSION, sig);
+        "Redis %s crashed by signal: %d", REDIS_VERSION, sig);
+    if (eip != NULL) {
+        serverLog(LL_WARNING,
+        "Crashed running the instuction at: %p", eip);
+    }
+    if (sig == SIGSEGV || sig == SIGBUS) {
+        serverLog(LL_WARNING,
+        "Accessing address: %p", (void*)info->si_addr);
+    }
     serverLog(LL_WARNING,
-        "    Failed assertion: %s (%s:%d)", server.assert_failed,
+        "Failed assertion: %s (%s:%d)", server.assert_failed,
                         server.assert_file, server.assert_line);
 
     /* Log the stack trace */
-    serverLog(LL_WARNING, "--- STACK TRACE");
+    serverLogRaw(LL_WARNING|LL_RAW, "\n------ STACK TRACE ------\n");
     logStackTrace(uc);
 
     /* Log INFO and CLIENT LIST */
-    serverLog(LL_WARNING, "--- INFO OUTPUT");
+    serverLogRaw(LL_WARNING|LL_RAW, "\n------ INFO OUTPUT ------\n");
     infostring = genRedisInfoString("all");
     infostring = sdscatprintf(infostring, "hash_init_value: %u\n",
         dictGetHashFunctionSeed());
-    serverLogRaw(LL_WARNING, infostring);
-    serverLog(LL_WARNING, "--- CLIENT LIST OUTPUT");
+    serverLogRaw(LL_WARNING|LL_RAW, infostring);
+    serverLogRaw(LL_WARNING|LL_RAW, "\n------ CLIENT LIST OUTPUT ------\n");
     clients = getAllClientsInfoString();
-    serverLogRaw(LL_WARNING, clients);
+    serverLogRaw(LL_WARNING|LL_RAW, clients);
     sdsfree(infostring);
     sdsfree(clients);
 
@@ -940,23 +1012,53 @@ void sigsegvHandler(int sig, siginfo_t *info, void *secret) {
 
 #if defined(HAVE_PROC_MAPS)
     /* Test memory */
-    serverLog(LL_WARNING, "--- FAST MEMORY TEST");
+    serverLogRaw(LL_WARNING|LL_RAW, "\n------ FAST MEMORY TEST ------\n");
     bioKillThreads();
     if (memtest_test_linux_anonymous_maps()) {
-        serverLog(LL_WARNING,
-            "!!! MEMORY ERROR DETECTED! Check your memory ASAP !!!");
+        serverLogRaw(LL_WARNING|LL_RAW,
+            "!!! MEMORY ERROR DETECTED! Check your memory ASAP !!!\n");
     } else {
-        serverLog(LL_WARNING,
-            "Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.");
+        serverLogRaw(LL_WARNING|LL_RAW,
+            "Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.\n");
     }
 #endif
 
-    serverLog(LL_WARNING,
+    if (eip != NULL) {
+        Dl_info info;
+        if (dladdr(eip, &info) != 0) {
+            serverLog(LL_WARNING|LL_RAW,
+                "\n------ DUMPING CODE AROUND EIP ------\n"
+                "Symbol: %s (base: %p)\n"
+                "Module: %s (base %p)\n"
+                "$ xxd -r -p /tmp/dump.hex /tmp/dump.bin\n"
+                "$ objdump --adjust-vma=%p -D -b binary -m i386:x86-64 /tmp/dump.bin\n"
+                "------\n",
+                info.dli_sname, info.dli_saddr, info.dli_fname, info.dli_fbase,
+                info.dli_saddr);
+            size_t len = (long)eip - (long)info.dli_saddr;
+            unsigned long sz = sysconf(_SC_PAGESIZE);
+            if (len < 1<<13) { /* we don't have functions over 8k (verified) */
+                /* Find the address of the next page, which is our "safety"
+                 * limit when dumping. Then try to dump just 128 bytes more
+                 * than EIP if there is room, or stop sooner. */
+                unsigned long next = ((unsigned long)eip + sz) & ~(sz-1);
+                unsigned long end = (unsigned long)eip + 128;
+                if (end > next) end = next;
+                len = end - (unsigned long)info.dli_saddr;
+                serverLogHexDump(LL_WARNING, "dump of function",
+                    info.dli_saddr ,len);
+                dumpX86Calls(info.dli_saddr,len);
+            }
+        }
+    }
+
+    serverLogRaw(LL_WARNING|LL_RAW,
 "\n=== REDIS BUG REPORT END. Make sure to include from START to END. ===\n\n"
 "       Please report the crash by opening an issue on github:\n\n"
 "           http://github.com/antirez/redis/issues\n\n"
 "  Suspect RAM error? Use redis-server --test-memory to verify it.\n\n"
 );
+
     /* free(messages); Don't call free() with possibly corrupted memory. */
     if (server.daemonize && server.supervised == 0) unlink(server.pidfile);
 
@@ -977,7 +1079,7 @@ void serverLogHexDump(int level, char *descr, void *value, size_t len) {
     unsigned char *v = value;
     char charset[] = "0123456789abcdef";
 
-    serverLog(level,"%s (hexdump):", descr);
+    serverLog(level,"%s (hexdump of %zu bytes):", descr, len);
     b = buf;
     while(len) {
         b[0] = charset[(*v)>>4];

src/debugmacro.h

@@ -0,0 +1,41 @@
+/* This file contains debugging macros to be used when investigating issues.
+ *
+ * -----------------------------------------------------------------------------
+ *
+ * Copyright (c) 2016, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <stdio.h>
+#define D(...)                                                               \
+    do {                                                                     \
+        FILE *fp = fopen("/tmp/log.txt","a");                                \
+        fprintf(fp,"%s:%s:%d:\t", __FILE__, __func__, __LINE__);             \
+        fprintf(fp,__VA_ARGS__);                                             \
+        fprintf(fp,"\n");                                                    \
+        fclose(fp);                                                          \
+    } while (0);

src/dict.c

@@ -45,7 +45,11 @@
 
 #include "dict.h"
 #include "zmalloc.h"
+#ifndef DICT_BENCHMARK_MAIN
 #include "redisassert.h"
+#else
+#include <assert.h>
+#endif
 
 /* Using dictEnableResize() / dictDisableResize() we make possible to
  * enable/disable resizing of the hash table as needed. This is very important
@@ -62,23 +66,11 @@ static unsigned int dict_force_resize_ratio = 5;
 
 static int _dictExpandIfNeeded(dict *ht);
 static unsigned long _dictNextPower(unsigned long size);
-static int _dictKeyIndex(dict *ht, const void *key);
+static int _dictKeyIndex(dict *ht, const void *key, unsigned int hash, dictEntry **existing);
 static int _dictInit(dict *ht, dictType *type, void *privDataPtr);
 
 /* -------------------------- hash functions -------------------------------- */
 
-/* Thomas Wang's 32 bit Mix Function */
-unsigned int dictIntHashFunction(unsigned int key)
-{
-    key += ~(key << 15);
-    key ^=  (key >> 10);
-    key +=  (key << 3);
-    key ^=  (key >> 6);
-    key += ~(key << 11);
-    key ^=  (key >> 16);
-    return key;
-}
-
 static uint32_t dict_hash_function_seed = 5381;
 
 void dictSetHashFunctionSeed(uint32_t seed) {
@@ -321,29 +313,32 @@ static void _dictRehashStep(dict *d) {
 /* Add an element to the target hash table */
 int dictAdd(dict *d, void *key, void *val)
 {
-    dictEntry *entry = dictAddRaw(d,key);
+    dictEntry *entry = dictAddRaw(d,key,NULL);
 
     if (!entry) return DICT_ERR;
     dictSetVal(d, entry, val);
     return DICT_OK;
 }
 
-/* Low level add. This function adds the entry but instead of setting
- * a value returns the dictEntry structure to the user, that will make
- * sure to fill the value field as he wishes.
+/* Low level add or find:
+ * This function adds the entry but instead of setting a value returns the
+ * dictEntry structure to the user, that will make sure to fill the value
+ * field as he wishes.
  *
  * This function is also directly exposed to the user API to be called
  * mainly in order to store non-pointers inside the hash value, example:
  *
- * entry = dictAddRaw(dict,mykey);
+ * entry = dictAddRaw(dict,mykey,NULL);
  * if (entry != NULL) dictSetSignedIntegerVal(entry,1000);
  *
  * Return values:
  *
- * If key already exists NULL is returned.
+ * If key already exists NULL is returned, and "*existing" is populated
+ * with the existing entry if existing is not NULL.
+ *
  * If key was added, the hash entry is returned to be manipulated by the caller.
  */
-dictEntry *dictAddRaw(dict *d, void *key)
+dictEntry *dictAddRaw(dict *d, void *key, dictEntry **existing)
 {
     int index;
     dictEntry *entry;
@@ -353,7 +348,7 @@ dictEntry *dictAddRaw(dict *d, void *key)
 
     /* Get the index of the new element, or -1 if
      * the element already exists. */
-    if ((index = _dictKeyIndex(d, key)) == -1)
+    if ((index = _dictKeyIndex(d, key, dictHashKey(d,key), existing)) == -1)
         return NULL;
 
     /* Allocate the memory and store the new entry.
@@ -371,51 +366,57 @@ dictEntry *dictAddRaw(dict *d, void *key)
     return entry;
 }
 
-/* Add an element, discarding the old if the key already exists.
+/* Add or Overwrite:
+ * Add an element, discarding the old value if the key already exists.
  * Return 1 if the key was added from scratch, 0 if there was already an
  * element with such key and dictReplace() just performed a value update
  * operation. */
 int dictReplace(dict *d, void *key, void *val)
 {
-    dictEntry *entry, auxentry;
+    dictEntry *entry, *existing, auxentry;
 
     /* Try to add the element. If the key
      * does not exists dictAdd will suceed. */
-    if (dictAdd(d, key, val) == DICT_OK)
+    entry = dictAddRaw(d,key,&existing);
+    if (entry) {
+        dictSetVal(d, entry, val);
         return 1;
-    /* It already exists, get the entry */
-    entry = dictFind(d, key);
+    }
+
     /* Set the new value and free the old one. Note that it is important
      * to do that in this order, as the value may just be exactly the same
      * as the previous one. In this context, think to reference counting,
      * you want to increment (set), and then decrement (free), and not the
      * reverse. */
-    auxentry = *entry;
-    dictSetVal(d, entry, val);
+    auxentry = *existing;
+    dictSetVal(d, existing, val);
     dictFreeVal(d, &auxentry);
     return 0;
 }
 
-/* dictReplaceRaw() is simply a version of dictAddRaw() that always
+/* Add or Find:
+ * dictAddOrFind() is simply a version of dictAddRaw() that always
  * returns the hash entry of the specified key, even if the key already
  * exists and can't be added (in that case the entry of the already
  * existing key is returned.)
  *
  * See dictAddRaw() for more information. */
-dictEntry *dictReplaceRaw(dict *d, void *key) {
-    dictEntry *entry = dictFind(d,key);
-
-    return entry ? entry : dictAddRaw(d,key);
+dictEntry *dictAddOrFind(dict *d, void *key) {
+    dictEntry *entry, *existing;
+    entry = dictAddRaw(d,key,&existing);
+    return entry ? entry : existing;
 }
 
-/* Search and remove an element */
-static int dictGenericDelete(dict *d, const void *key, int nofree)
-{
+/* Search and remove an element. This is an helper function for
+ * dictDelete() and dictUnlink(), please check the top comment
+ * of those functions. */
+static dictEntry *dictGenericDelete(dict *d, const void *key, int nofree) {
     unsigned int h, idx;
     dictEntry *he, *prevHe;
     int table;
 
-    if (d->ht[0].size == 0) return DICT_ERR; /* d->ht[0].table is NULL */
+    if (d->ht[0].used == 0 && d->ht[1].used == 0) return NULL;
+
     if (dictIsRehashing(d)) _dictRehashStep(d);
     h = dictHashKey(d, key);
 
@@ -424,7 +425,7 @@ static int dictGenericDelete(dict *d, const void *key, int nofree)
         he = d->ht[table].table[idx];
         prevHe = NULL;
         while(he) {
-            if (dictCompareKeys(d, key, he->key)) {
+            if (key==he->key || dictCompareKeys(d, key, he->key)) {
                 /* Unlink the element from the list */
                 if (prevHe)
                     prevHe->next = he->next;
@@ -433,27 +434,59 @@ static int dictGenericDelete(dict *d, const void *key, int nofree)
                 if (!nofree) {
                     dictFreeKey(d, he);
                     dictFreeVal(d, he);
+                    zfree(he);
                 }
-                zfree(he);
                 d->ht[table].used--;
-                return DICT_OK;
+                return he;
             }
             prevHe = he;
             he = he->next;
         }
         if (!dictIsRehashing(d)) break;
     }
-    return DICT_ERR; /* not found */
+    return NULL; /* not found */
 }
 
+/* Remove an element, returning DICT_OK on success or DICT_ERR if the
+ * element was not found. */
 int dictDelete(dict *ht, const void *key) {
-    return dictGenericDelete(ht,key,0);
+    return dictGenericDelete(ht,key,0) ? DICT_OK : DICT_ERR;
 }
 
-int dictDeleteNoFree(dict *ht, const void *key) {
+/* Remove an element from the table, but without actually releasing
+ * the key, value and dictionary entry. The dictionary entry is returned
+ * if the element was found (and unlinked from the table), and the user
+ * should later call `dictFreeUnlinkedEntry()` with it in order to release it.
+ * Otherwise if the key is not found, NULL is returned.
+ *
+ * This function is useful when we want to remove something from the hash
+ * table but want to use its value before actually deleting the entry.
+ * Without this function the pattern would require two lookups:
+ *
+ *  entry = dictFind(...);
+ *  // Do something with entry
+ *  dictDelete(dictionary,entry);
+ *
+ * Thanks to this function it is possible to avoid this, and use
+ * instead:
+ *
+ * entry = dictUnlink(dictionary,entry);
+ * // Do something with entry
+ * dictFreeUnlinkedEntry(entry); // <- This does not need to lookup again.
+ */
+dictEntry *dictUnlink(dict *ht, const void *key) {
     return dictGenericDelete(ht,key,1);
 }
 
+/* You need to call this function to really free the entry after a call
+ * to dictUnlink(). It's safe to call this function with 'he' = NULL. */
+void dictFreeUnlinkedEntry(dict *d, dictEntry *he) {
+    if (he == NULL) return;
+    dictFreeKey(d, he);
+    dictFreeVal(d, he);
+    zfree(he);
+}
+
 /* Destroy an entire dictionary */
 int _dictClear(dict *d, dictht *ht, void(callback)(void *)) {
     unsigned long i;
@@ -494,14 +527,14 @@ dictEntry *dictFind(dict *d, const void *key)
     dictEntry *he;
     unsigned int h, idx, table;
 
-    if (d->ht[0].size == 0) return NULL; /* We don't have a table at all */
+    if (d->ht[0].used + d->ht[1].used == 0) return NULL; /* dict is empty */
     if (dictIsRehashing(d)) _dictRehashStep(d);
     h = dictHashKey(d, key);
     for (table = 0; table <= 1; table++) {
         idx = h & d->ht[table].sizemask;
         he = d->ht[table].table[idx];
         while(he) {
-            if (dictCompareKeys(d, key, he->key))
+            if (key==he->key || dictCompareKeys(d, key, he->key))
                 return he;
             he = he->next;
         }
@@ -962,27 +995,29 @@ static unsigned long _dictNextPower(unsigned long size)
 
 /* Returns the index of a free slot that can be populated with
  * a hash entry for the given 'key'.
- * If the key already exists, -1 is returned.
+ * If the key already exists, -1 is returned
+ * and the optional output parameter may be filled.
  *
  * Note that if we are in the process of rehashing the hash table, the
  * index is always returned in the context of the second (new) hash table. */
-static int _dictKeyIndex(dict *d, const void *key)
+static int _dictKeyIndex(dict *d, const void *key, unsigned int hash, dictEntry **existing)
 {
-    unsigned int h, idx, table;
+    unsigned int idx, table;
     dictEntry *he;
+    if (existing) *existing = NULL;
 
     /* Expand the hash table if needed */
     if (_dictExpandIfNeeded(d) == DICT_ERR)
         return -1;
-    /* Compute the key hash value */
-    h = dictHashKey(d, key);
     for (table = 0; table <= 1; table++) {
-        idx = h & d->ht[table].sizemask;
+        idx = hash & d->ht[table].sizemask;
         /* Search if this slot does not already contain the given key */
         he = d->ht[table].table[idx];
         while(he) {
-            if (dictCompareKeys(d, key, he->key))
+            if (key==he->key || dictCompareKeys(d, key, he->key)) {
+                if (existing) *existing = he;
                 return -1;
+            }
             he = he->next;
         }
         if (!dictIsRehashing(d)) break;
@@ -1083,3 +1118,120 @@ void dictGetStats(char *buf, size_t bufsize, dict *d) {
     /* Make sure there is a NULL term at the end. */
     if (orig_bufsize) orig_buf[orig_bufsize-1] = '\0';
 }
+
+/* ------------------------------- Benchmark ---------------------------------*/
+
+#ifdef DICT_BENCHMARK_MAIN
+
+#include "sds.h"
+
+unsigned int hashCallback(const void *key) {
+    return dictGenHashFunction((unsigned char*)key, sdslen((char*)key));
+}
+
+int compareCallback(void *privdata, const void *key1, const void *key2) {
+    int l1,l2;
+    DICT_NOTUSED(privdata);
+
+    l1 = sdslen((sds)key1);
+    l2 = sdslen((sds)key2);
+    if (l1 != l2) return 0;
+    return memcmp(key1, key2, l1) == 0;
+}
+
+void freeCallback(void *privdata, void *val) {
+    DICT_NOTUSED(privdata);
+
+    sdsfree(val);
+}
+
+dictType BenchmarkDictType = {
+    hashCallback,
+    NULL,
+    NULL,
+    compareCallback,
+    freeCallback,
+    NULL
+};
+
+#define start_benchmark() start = timeInMilliseconds()
+#define end_benchmark(msg) do { \
+    elapsed = timeInMilliseconds()-start; \
+    printf(msg ": %ld items in %lld ms\n", count, elapsed); \
+} while(0);
+
+/* dict-benchmark [count] */
+int main(int argc, char **argv) {
+    long j;
+    long long start, elapsed;
+    dict *dict = dictCreate(&BenchmarkDictType,NULL);
+    long count = 0;
+
+    if (argc == 2) {
+        count = strtol(argv[1],NULL,10);
+    } else {
+        count = 5000000;
+    }
+
+    start_benchmark();
+    for (j = 0; j < count; j++) {
+        int retval = dictAdd(dict,sdsfromlonglong(j),(void*)j);
+        assert(retval == DICT_OK);
+    }
+    end_benchmark("Inserting");
+    assert((long)dictSize(dict) == count);
+
+    /* Wait for rehashing. */
+    while (dictIsRehashing(dict)) {
+        dictRehashMilliseconds(dict,100);
+    }
+
+    start_benchmark();
+    for (j = 0; j < count; j++) {
+        sds key = sdsfromlonglong(j);
+        dictEntry *de = dictFind(dict,key);
+        assert(de != NULL);
+        sdsfree(key);
+    }
+    end_benchmark("Linear access of existing elements");
+
+    start_benchmark();
+    for (j = 0; j < count; j++) {
+        sds key = sdsfromlonglong(j);
+        dictEntry *de = dictFind(dict,key);
+        assert(de != NULL);
+        sdsfree(key);
+    }
+    end_benchmark("Linear access of existing elements (2nd round)");
+
+    start_benchmark();
+    for (j = 0; j < count; j++) {
+        sds key = sdsfromlonglong(rand() % count);
+        dictEntry *de = dictFind(dict,key);
+        assert(de != NULL);
+        sdsfree(key);
+    }
+    end_benchmark("Random access of existing elements");
+
+    start_benchmark();
+    for (j = 0; j < count; j++) {
+        sds key = sdsfromlonglong(rand() % count);
+        key[0] = 'X';
+        dictEntry *de = dictFind(dict,key);
+        assert(de == NULL);
+        sdsfree(key);
+    }
+    end_benchmark("Accessing missing");
+
+    start_benchmark();
+    for (j = 0; j < count; j++) {
+        sds key = sdsfromlonglong(j);
+        int retval = dictDelete(dict,key);
+        assert(retval == DICT_OK);
+        key[0] += 17; /* Change first number to letter. */
+        retval = dictAdd(dict,key,(void*)j);
+        assert(retval == DICT_OK);
+    }
+    end_benchmark("Removing and adding");
+}
+#endif

src/dict.h

@@ -106,19 +106,19 @@ typedef void (dictScanFunction)(void *privdata, const dictEntry *de);
 
 #define dictSetVal(d, entry, _val_) do { \
     if ((d)->type->valDup) \
-        entry->v.val = (d)->type->valDup((d)->privdata, _val_); \
+        (entry)->v.val = (d)->type->valDup((d)->privdata, _val_); \
     else \
-        entry->v.val = (_val_); \
+        (entry)->v.val = (_val_); \
 } while(0)
 
 #define dictSetSignedIntegerVal(entry, _val_) \
-    do { entry->v.s64 = _val_; } while(0)
+    do { (entry)->v.s64 = _val_; } while(0)
 
 #define dictSetUnsignedIntegerVal(entry, _val_) \
-    do { entry->v.u64 = _val_; } while(0)
+    do { (entry)->v.u64 = _val_; } while(0)
 
 #define dictSetDoubleVal(entry, _val_) \
-    do { entry->v.d = _val_; } while(0)
+    do { (entry)->v.d = _val_; } while(0)
 
 #define dictFreeKey(d, entry) \
     if ((d)->type->keyDestructor) \
@@ -126,9 +126,9 @@ typedef void (dictScanFunction)(void *privdata, const dictEntry *de);
 
 #define dictSetKey(d, entry, _key_) do { \
     if ((d)->type->keyDup) \
-        entry->key = (d)->type->keyDup((d)->privdata, _key_); \
+        (entry)->key = (d)->type->keyDup((d)->privdata, _key_); \
     else \
-        entry->key = (_key_); \
+        (entry)->key = (_key_); \
 } while(0)
 
 #define dictCompareKeys(d, key1, key2) \
@@ -150,11 +150,12 @@ typedef void (dictScanFunction)(void *privdata, const dictEntry *de);
 dict *dictCreate(dictType *type, void *privDataPtr);
 int dictExpand(dict *d, unsigned long size);
 int dictAdd(dict *d, void *key, void *val);
-dictEntry *dictAddRaw(dict *d, void *key);
+dictEntry *dictAddRaw(dict *d, void *key, dictEntry **existing);
+dictEntry *dictAddOrFind(dict *d, void *key);
 int dictReplace(dict *d, void *key, void *val);
-dictEntry *dictReplaceRaw(dict *d, void *key);
 int dictDelete(dict *d, const void *key);
-int dictDeleteNoFree(dict *d, const void *key);
+dictEntry *dictUnlink(dict *ht, const void *key);
+void dictFreeUnlinkedEntry(dict *d, dictEntry *he);
 void dictRelease(dict *d);
 dictEntry * dictFind(dict *d, const void *key);
 void *dictFetchValue(dict *d, const void *key);

src/evict.c

@@ -0,0 +1,524 @@
+/* Maxmemory directive handling (LRU eviction and other policies).
+ *
+ * ----------------------------------------------------------------------------
+ *
+ * Copyright (c) 2009-2016, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "server.h"
+#include "bio.h"
+
+/* ----------------------------------------------------------------------------
+ * Data structures
+ * --------------------------------------------------------------------------*/
+
+/* To improve the quality of the LRU approximation we take a set of keys
+ * that are good candidate for eviction across freeMemoryIfNeeded() calls.
+ *
+ * Entries inside the eviciton pool are taken ordered by idle time, putting
+ * greater idle times to the right (ascending order).
+ *
+ * When an LFU policy is used instead, a reverse frequency indication is used
+ * instead of the idle time, so that we still evict by larger value (larger
+ * inverse frequency means to evict keys with the least frequent accesses).
+ *
+ * Empty entries have the key pointer set to NULL. */
+#define EVPOOL_SIZE 16
+#define EVPOOL_CACHED_SDS_SIZE 255
+struct evictionPoolEntry {
+    unsigned long long idle;    /* Object idle time (inverse frequency for LFU) */
+    sds key;                    /* Key name. */
+    sds cached;                 /* Cached SDS object for key name. */
+    int dbid;                   /* Key DB number. */
+};
+
+static struct evictionPoolEntry *EvictionPoolLRU;
+
+unsigned long LFUDecrAndReturn(robj *o);
+
+/* ----------------------------------------------------------------------------
+ * Implementation of eviction, aging and LRU
+ * --------------------------------------------------------------------------*/
+
+/* Return the LRU clock, based on the clock resolution. This is a time
+ * in a reduced-bits format that can be used to set and check the
+ * object->lru field of redisObject structures. */
+unsigned int getLRUClock(void) {
+    return (mstime()/LRU_CLOCK_RESOLUTION) & LRU_CLOCK_MAX;
+}
+
+/* Given an object returns the min number of milliseconds the object was never
+ * requested, using an approximated LRU algorithm. */
+unsigned long long estimateObjectIdleTime(robj *o) {
+    unsigned long long lruclock = LRU_CLOCK();
+    if (lruclock >= o->lru) {
+        return (lruclock - o->lru) * LRU_CLOCK_RESOLUTION;
+    } else {
+        return (lruclock + (LRU_CLOCK_MAX - o->lru)) *
+                    LRU_CLOCK_RESOLUTION;
+    }
+}
+
+/* freeMemoryIfNeeded() gets called when 'maxmemory' is set on the config
+ * file to limit the max memory used by the server, before processing a
+ * command.
+ *
+ * The goal of the function is to free enough memory to keep Redis under the
+ * configured memory limit.
+ *
+ * The function starts calculating how many bytes should be freed to keep
+ * Redis under the limit, and enters a loop selecting the best keys to
+ * evict accordingly to the configured policy.
+ *
+ * If all the bytes needed to return back under the limit were freed the
+ * function returns C_OK, otherwise C_ERR is returned, and the caller
+ * should block the execution of commands that will result in more memory
+ * used by the server.
+ *
+ * ------------------------------------------------------------------------
+ *
+ * LRU approximation algorithm
+ *
+ * Redis uses an approximation of the LRU algorithm that runs in constant
+ * memory. Every time there is a key to expire, we sample N keys (with
+ * N very small, usually in around 5) to populate a pool of best keys to
+ * evict of M keys (the pool size is defined by EVPOOL_SIZE).
+ *
+ * The N keys sampled are added in the pool of good keys to expire (the one
+ * with an old access time) if they are better than one of the current keys
+ * in the pool.
+ *
+ * After the pool is populated, the best key we have in the pool is expired.
+ * However note that we don't remove keys from the pool when they are deleted
+ * so the pool may contain keys that no longer exist.
+ *
+ * When we try to evict a key, and all the entries in the pool don't exist
+ * we populate it again. This time we'll be sure that the pool has at least
+ * one key that can be evicted, if there is at least one key that can be
+ * evicted in the whole database. */
+
+/* Create a new eviction pool. */
+void evictionPoolAlloc(void) {
+    struct evictionPoolEntry *ep;
+    int j;
+
+    ep = zmalloc(sizeof(*ep)*EVPOOL_SIZE);
+    for (j = 0; j < EVPOOL_SIZE; j++) {
+        ep[j].idle = 0;
+        ep[j].key = NULL;
+        ep[j].cached = sdsnewlen(NULL,EVPOOL_CACHED_SDS_SIZE);
+        ep[j].dbid = 0;
+    }
+    EvictionPoolLRU = ep;
+}
+
+/* This is an helper function for freeMemoryIfNeeded(), it is used in order
+ * to populate the evictionPool with a few entries every time we want to
+ * expire a key. Keys with idle time smaller than one of the current
+ * keys are added. Keys are always added if there are free entries.
+ *
+ * We insert keys on place in ascending order, so keys with the smaller
+ * idle time are on the left, and keys with the higher idle time on the
+ * right. */
+
+void evictionPoolPopulate(int dbid, dict *sampledict, dict *keydict, struct evictionPoolEntry *pool) {
+    int j, k, count;
+    dictEntry *samples[server.maxmemory_samples];
+
+    count = dictGetSomeKeys(sampledict,samples,server.maxmemory_samples);
+    for (j = 0; j < count; j++) {
+        unsigned long long idle;
+        sds key;
+        robj *o;
+        dictEntry *de;
+
+        de = samples[j];
+        key = dictGetKey(de);
+
+        /* If the dictionary we are sampling from is not the main
+         * dictionary (but the expires one) we need to lookup the key
+         * again in the key dictionary to obtain the value object. */
+        if (server.maxmemory_policy != MAXMEMORY_VOLATILE_TTL) {
+            if (sampledict != keydict) de = dictFind(keydict, key);
+            o = dictGetVal(de);
+        }
+
+        /* Calculate the idle time according to the policy. This is called
+         * idle just because the code initially handled LRU, but is in fact
+         * just a score where an higher score means better candidate. */
+        if (server.maxmemory_policy & MAXMEMORY_FLAG_LRU) {
+            idle = estimateObjectIdleTime(o);
+        } else if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) {
+            /* When we use an LRU policy, we sort the keys by idle time
+             * so that we expire keys starting from greater idle time.
+             * However when the policy is an LFU one, we have a frequency
+             * estimation, and we want to evict keys with lower frequency
+             * first. So inside the pool we put objects using the inverted
+             * frequency subtracting the actual frequency to the maximum
+             * frequency of 255. */
+            idle = 255-LFUDecrAndReturn(o);
+        } else if (server.maxmemory_policy == MAXMEMORY_VOLATILE_TTL) {
+            /* In this case the sooner the expire the better. */
+            idle = ULLONG_MAX - (long)dictGetVal(de);
+        } else {
+            serverPanic("Unknown eviction policy in evictionPoolPopulate()");
+        }
+
+        /* Insert the element inside the pool.
+         * First, find the first empty bucket or the first populated
+         * bucket that has an idle time smaller than our idle time. */
+        k = 0;
+        while (k < EVPOOL_SIZE &&
+               pool[k].key &&
+               pool[k].idle < idle) k++;
+        if (k == 0 && pool[EVPOOL_SIZE-1].key != NULL) {
+            /* Can't insert if the element is < the worst element we have
+             * and there are no empty buckets. */
+            continue;
+        } else if (k < EVPOOL_SIZE && pool[k].key == NULL) {
+            /* Inserting into empty position. No setup needed before insert. */
+        } else {
+            /* Inserting in the middle. Now k points to the first element
+             * greater than the element to insert.  */
+            if (pool[EVPOOL_SIZE-1].key == NULL) {
+                /* Free space on the right? Insert at k shifting
+                 * all the elements from k to end to the right. */
+
+                /* Save SDS before overwriting. */
+                sds cached = pool[EVPOOL_SIZE-1].cached;
+                memmove(pool+k+1,pool+k,
+                    sizeof(pool[0])*(EVPOOL_SIZE-k-1));
+                pool[k].cached = cached;
+            } else {
+                /* No free space on right? Insert at k-1 */
+                k--;
+                /* Shift all elements on the left of k (included) to the
+                 * left, so we discard the element with smaller idle time. */
+                sds cached = pool[0].cached; /* Save SDS before overwriting. */
+                if (pool[0].key != pool[0].cached) sdsfree(pool[0].key);
+                memmove(pool,pool+1,sizeof(pool[0])*k);
+                pool[k].cached = cached;
+            }
+        }
+
+        /* Try to reuse the cached SDS string allocated in the pool entry,
+         * because allocating and deallocating this object is costly
+         * (according to the profiler, not my fantasy. Remember:
+         * premature optimizbla bla bla bla. */
+        int klen = sdslen(key);
+        if (klen > EVPOOL_CACHED_SDS_SIZE) {
+            pool[k].key = sdsdup(key);
+        } else {
+            memcpy(pool[k].cached,key,klen+1);
+            sdssetlen(pool[k].cached,klen);
+            pool[k].key = pool[k].cached;
+        }
+        pool[k].idle = idle;
+        pool[k].dbid = dbid;
+    }
+}
+
+/* ----------------------------------------------------------------------------
+ * LFU (Least Frequently Used) implementation.
+
+ * We have 24 total bits of space in each object in order to implement
+ * an LFU (Least Frequently Used) eviction policy, since we re-use the
+ * LRU field for this purpose.
+ *
+ * We split the 24 bits into two fields:
+ *
+ *          16 bits      8 bits
+ *     +----------------+--------+
+ *     + Last decr time | LOG_C  |
+ *     +----------------+--------+
+ *
+ * LOG_C is a logarithmic counter that provides an indication of the access
+ * frequency. However this field must also be decremented otherwise what used
+ * to be a frequently accessed key in the past, will remain ranked like that
+ * forever, while we want the algorithm to adapt to access pattern changes.
+ *
+ * So the remaining 16 bits are used in order to store the "decrement time",
+ * a reduced-precision Unix time (we take 16 bits of the time converted
+ * in minutes since we don't care about wrapping around) where the LOG_C
+ * counter is halved if it has an high value, or just decremented if it
+ * has a low value.
+ *
+ * New keys don't start at zero, in order to have the ability to collect
+ * some accesses before being trashed away, so they start at COUNTER_INIT_VAL.
+ * The logarithmic increment performed on LOG_C takes care of COUNTER_INIT_VAL
+ * when incrementing the key, so that keys starting at COUNTER_INIT_VAL
+ * (or having a smaller value) have a very high chance of being incremented
+ * on access.
+ *
+ * During decrement, the value of the logarithmic counter is halved if
+ * its current value is greater than two times the COUNTER_INIT_VAL, otherwise
+ * it is just decremented by one.
+ * --------------------------------------------------------------------------*/
+
+/* Return the current time in minutes, just taking the least significant
+ * 16 bits. The returned time is suitable to be stored as LDT (last decrement
+ * time) for the LFU implementation. */
+unsigned long LFUGetTimeInMinutes(void) {
+    return (server.unixtime/60) & 65535;
+}
+
+/* Given an object last decrement time, compute the minimum number of minutes
+ * that elapsed since the last decrement. Handle overflow (ldt greater than
+ * the current 16 bits minutes time) considering the time as wrapping
+ * exactly once. */
+unsigned long LFUTimeElapsed(unsigned long ldt) {
+    unsigned long now = LFUGetTimeInMinutes();
+    if (now >= ldt) return now-ldt;
+    return 65535-ldt+now;
+}
+
+/* Logarithmically increment a counter. The greater is the current counter value
+ * the less likely is that it gets really implemented. Saturate it at 255. */
+uint8_t LFULogIncr(uint8_t counter) {
+    if (counter == 255) return 255;
+    double r = (double)rand()/RAND_MAX;
+    double baseval = counter - LFU_INIT_VAL;
+    if (baseval < 0) baseval = 0;
+    double p = 1.0/(baseval*server.lfu_log_factor+1);
+    if (r < p) counter++;
+    return counter;
+}
+
+/* If the object decrement time is reached, decrement the LFU counter and
+ * update the decrement time field. Return the object frequency counter.
+ *
+ * This function is used in order to scan the dataset for the best object
+ * to fit: as we check for the candidate, we incrementally decrement the
+ * counter of the scanned objects if needed. */
+#define LFU_DECR_INTERVAL 1
+unsigned long LFUDecrAndReturn(robj *o) {
+    unsigned long ldt = o->lru >> 8;
+    unsigned long counter = o->lru & 255;
+    if (LFUTimeElapsed(ldt) >= server.lfu_decay_time && counter) {
+        if (counter > LFU_INIT_VAL*2) {
+            counter /= 2;
+            if (counter < LFU_INIT_VAL*2) counter = LFU_INIT_VAL*2;
+        } else {
+            counter--;
+        }
+        o->lru = (LFUGetTimeInMinutes()<<8) | counter;
+    }
+    return counter;
+}
+
+/* ----------------------------------------------------------------------------
+ * The external API for eviction: freeMemroyIfNeeded() is called by the
+ * server when there is data to add in order to make space if needed.
+ * --------------------------------------------------------------------------*/
+
+int freeMemoryIfNeeded(void) {
+    size_t mem_reported, mem_used, mem_tofree, mem_freed;
+    int slaves = listLength(server.slaves);
+    mstime_t latency, eviction_latency;
+    long long delta;
+
+    /* Check if we are over the memory usage limit. If we are not, no need
+     * to subtract the slaves output buffers. We can just return ASAP. */
+    mem_reported = zmalloc_used_memory();
+    if (mem_reported <= server.maxmemory) return C_OK;
+
+    /* Remove the size of slaves output buffers and AOF buffer from the
+     * count of used memory. */
+    mem_used = mem_reported;
+    if (slaves) {
+        listIter li;
+        listNode *ln;
+
+        listRewind(server.slaves,&li);
+        while((ln = listNext(&li))) {
+            client *slave = listNodeValue(ln);
+            unsigned long obuf_bytes = getClientOutputBufferMemoryUsage(slave);
+            if (obuf_bytes > mem_used)
+                mem_used = 0;
+            else
+                mem_used -= obuf_bytes;
+        }
+    }
+    if (server.aof_state != AOF_OFF) {
+        mem_used -= sdslen(server.aof_buf);
+        mem_used -= aofRewriteBufferSize();
+    }
+
+    /* Check if we are still over the memory limit. */
+    if (mem_used <= server.maxmemory) return C_OK;
+
+    /* Compute how much memory we need to free. */
+    mem_tofree = mem_used - server.maxmemory;
+    mem_freed = 0;
+
+    if (server.maxmemory_policy == MAXMEMORY_NO_EVICTION)
+        goto cant_free; /* We need to free memory, but policy forbids. */
+
+    latencyStartMonitor(latency);
+    while (mem_freed < mem_tofree) {
+        int j, k, i, keys_freed = 0;
+        static int next_db = 0;
+        sds bestkey = NULL;
+        int bestdbid;
+        redisDb *db;
+        dict *dict;
+        dictEntry *de;
+
+        if (server.maxmemory_policy & (MAXMEMORY_FLAG_LRU|MAXMEMORY_FLAG_LFU) ||
+            server.maxmemory_policy == MAXMEMORY_VOLATILE_TTL)
+        {
+            struct evictionPoolEntry *pool = EvictionPoolLRU;
+
+            while(bestkey == NULL) {
+                unsigned long total_keys = 0, keys;
+
+                /* We don't want to make local-db choices when expiring keys,
+                 * so to start populate the eviction pool sampling keys from
+                 * every DB. */
+                for (i = 0; i < server.dbnum; i++) {
+                    db = server.db+i;
+                    dict = (server.maxmemory_policy & MAXMEMORY_FLAG_ALLKEYS) ?
+                            db->dict : db->expires;
+                    if ((keys = dictSize(dict)) != 0) {
+                        evictionPoolPopulate(i, dict, db->dict, pool);
+                        total_keys += keys;
+                    }
+                }
+                if (!total_keys) break; /* No keys to evict. */
+
+                /* Go backward from best to worst element to evict. */
+                for (k = EVPOOL_SIZE-1; k >= 0; k--) {
+                    if (pool[k].key == NULL) continue;
+                    bestdbid = pool[k].dbid;
+
+                    if (server.maxmemory_policy & MAXMEMORY_FLAG_ALLKEYS) {
+                        de = dictFind(server.db[pool[k].dbid].dict,
+                            pool[k].key);
+                    } else {
+                        de = dictFind(server.db[pool[k].dbid].expires,
+                            pool[k].key);
+                    }
+
+                    /* Remove the entry from the pool. */
+                    if (pool[k].key != pool[k].cached)
+                        sdsfree(pool[k].key);
+                    pool[k].key = NULL;
+                    pool[k].idle = 0;
+
+                    /* If the key exists, is our pick. Otherwise it is
+                     * a ghost and we need to try the next element. */
+                    if (de) {
+                        bestkey = dictGetKey(de);
+                        break;
+                    } else {
+                        /* Ghost... Iterate again. */
+                    }
+                }
+            }
+        }
+
+        /* volatile-random and allkeys-random policy */
+        else if (server.maxmemory_policy == MAXMEMORY_ALLKEYS_RANDOM ||
+                 server.maxmemory_policy == MAXMEMORY_VOLATILE_RANDOM)
+        {
+            /* When evicting a random key, we try to evict a key for
+             * each DB, so we use the static 'next_db' variable to
+             * incrementally visit all DBs. */
+            for (i = 0; i < server.dbnum; i++) {
+                j = (++next_db) % server.dbnum;
+                db = server.db+j;
+                dict = (server.maxmemory_policy == MAXMEMORY_ALLKEYS_RANDOM) ?
+                        db->dict : db->expires;
+                if (dictSize(dict) != 0) {
+                    de = dictGetRandomKey(dict);
+                    bestkey = dictGetKey(de);
+                    bestdbid = j;
+                    break;
+                }
+            }
+        }
+
+        /* Finally remove the selected key. */
+        if (bestkey) {
+            db = server.db+bestdbid;
+            robj *keyobj = createStringObject(bestkey,sdslen(bestkey));
+            propagateExpire(db,keyobj,server.lazyfree_lazy_eviction);
+            /* We compute the amount of memory freed by db*Delete() alone.
+             * It is possible that actually the memory needed to propagate
+             * the DEL in AOF and replication link is greater than the one
+             * we are freeing removing the key, but we can't account for
+             * that otherwise we would never exit the loop.
+             *
+             * AOF and Output buffer memory will be freed eventually so
+             * we only care about memory used by the key space. */
+            delta = (long long) zmalloc_used_memory();
+            latencyStartMonitor(eviction_latency);
+            if (server.lazyfree_lazy_eviction)
+                dbAsyncDelete(db,keyobj);
+            else
+                dbSyncDelete(db,keyobj);
+            latencyEndMonitor(eviction_latency);
+            latencyAddSampleIfNeeded("eviction-del",eviction_latency);
+            latencyRemoveNestedEvent(latency,eviction_latency);
+            delta -= (long long) zmalloc_used_memory();
+            mem_freed += delta;
+            server.stat_evictedkeys++;
+            notifyKeyspaceEvent(NOTIFY_EVICTED, "evicted",
+                keyobj, db->id);
+            decrRefCount(keyobj);
+            keys_freed++;
+
+            /* When the memory to free starts to be big enough, we may
+             * start spending so much time here that is impossible to
+             * deliver data to the slaves fast enough, so we force the
+             * transmission here inside the loop. */
+            if (slaves) flushSlavesOutputBuffers();
+        }
+
+        if (!keys_freed) {
+            latencyEndMonitor(latency);
+            latencyAddSampleIfNeeded("eviction-cycle",latency);
+            goto cant_free; /* nothing to free... */
+        }
+    }
+    latencyEndMonitor(latency);
+    latencyAddSampleIfNeeded("eviction-cycle",latency);
+    return C_OK;
+
+cant_free:
+    /* We are here if we are not able to reclaim memory. There is only one
+     * last thing we can try: check if the lazyfree thread has jobs in queue
+     * and wait... */
+    while(bioPendingJobsOfType(BIO_LAZY_FREE)) {
+        if (((mem_reported - zmalloc_used_memory()) + mem_freed) >= mem_tofree)
+            break;
+        usleep(1000);
+    }
+    return C_ERR;
+}
+

src/expire.c

@@ -0,0 +1,354 @@
+/* Implementation of EXPIRE (keys with fixed time to live).
+ *
+ * ----------------------------------------------------------------------------
+ *
+ * Copyright (c) 2009-2016, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "server.h"
+
+/*-----------------------------------------------------------------------------
+ * Incremental collection of expired keys.
+ *
+ * When keys are accessed they are expired on-access. However we need a
+ * mechanism in order to ensure keys are eventually removed when expired even
+ * if no access is performed on them.
+ *----------------------------------------------------------------------------*/
+
+/* Helper function for the activeExpireCycle() function.
+ * This function will try to expire the key that is stored in the hash table
+ * entry 'de' of the 'expires' hash table of a Redis database.
+ *
+ * If the key is found to be expired, it is removed from the database and
+ * 1 is returned. Otherwise no operation is performed and 0 is returned.
+ *
+ * When a key is expired, server.stat_expiredkeys is incremented.
+ *
+ * The parameter 'now' is the current time in milliseconds as is passed
+ * to the function to avoid too many gettimeofday() syscalls. */
+int activeExpireCycleTryExpire(redisDb *db, dictEntry *de, long long now) {
+    long long t = dictGetSignedIntegerVal(de);
+    if (now > t) {
+        sds key = dictGetKey(de);
+        robj *keyobj = createStringObject(key,sdslen(key));
+
+        propagateExpire(db,keyobj,server.lazyfree_lazy_expire);
+        if (server.lazyfree_lazy_expire)
+            dbAsyncDelete(db,keyobj);
+        else
+            dbSyncDelete(db,keyobj);
+        notifyKeyspaceEvent(NOTIFY_EXPIRED,
+            "expired",keyobj,db->id);
+        decrRefCount(keyobj);
+        server.stat_expiredkeys++;
+        return 1;
+    } else {
+        return 0;
+    }
+}
+
+/* Try to expire a few timed out keys. The algorithm used is adaptive and
+ * will use few CPU cycles if there are few expiring keys, otherwise
+ * it will get more aggressive to avoid that too much memory is used by
+ * keys that can be removed from the keyspace.
+ *
+ * No more than CRON_DBS_PER_CALL databases are tested at every
+ * iteration.
+ *
+ * This kind of call is used when Redis detects that timelimit_exit is
+ * true, so there is more work to do, and we do it more incrementally from
+ * the beforeSleep() function of the event loop.
+ *
+ * Expire cycle type:
+ *
+ * If type is ACTIVE_EXPIRE_CYCLE_FAST the function will try to run a
+ * "fast" expire cycle that takes no longer than EXPIRE_FAST_CYCLE_DURATION
+ * microseconds, and is not repeated again before the same amount of time.
+ *
+ * If type is ACTIVE_EXPIRE_CYCLE_SLOW, that normal expire cycle is
+ * executed, where the time limit is a percentage of the REDIS_HZ period
+ * as specified by the REDIS_EXPIRELOOKUPS_TIME_PERC define. */
+
+void activeExpireCycle(int type) {
+    /* This function has some global state in order to continue the work
+     * incrementally across calls. */
+    static unsigned int current_db = 0; /* Last DB tested. */
+    static int timelimit_exit = 0;      /* Time limit hit in previous call? */
+    static long long last_fast_cycle = 0; /* When last fast cycle ran. */
+
+    int j, iteration = 0;
+    int dbs_per_call = CRON_DBS_PER_CALL;
+    long long start = ustime(), timelimit;
+
+    if (type == ACTIVE_EXPIRE_CYCLE_FAST) {
+        /* Don't start a fast cycle if the previous cycle did not exited
+         * for time limt. Also don't repeat a fast cycle for the same period
+         * as the fast cycle total duration itself. */
+        if (!timelimit_exit) return;
+        if (start < last_fast_cycle + ACTIVE_EXPIRE_CYCLE_FAST_DURATION*2) return;
+        last_fast_cycle = start;
+    }
+
+    /* We usually should test CRON_DBS_PER_CALL per iteration, with
+     * two exceptions:
+     *
+     * 1) Don't test more DBs than we have.
+     * 2) If last time we hit the time limit, we want to scan all DBs
+     * in this iteration, as there is work to do in some DB and we don't want
+     * expired keys to use memory for too much time. */
+    if (dbs_per_call > server.dbnum || timelimit_exit)
+        dbs_per_call = server.dbnum;
+
+    /* We can use at max ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC percentage of CPU time
+     * per iteration. Since this function gets called with a frequency of
+     * server.hz times per second, the following is the max amount of
+     * microseconds we can spend in this function. */
+    timelimit = 1000000*ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/server.hz/100;
+    timelimit_exit = 0;
+    if (timelimit <= 0) timelimit = 1;
+
+    if (type == ACTIVE_EXPIRE_CYCLE_FAST)
+        timelimit = ACTIVE_EXPIRE_CYCLE_FAST_DURATION; /* in microseconds. */
+
+    for (j = 0; j < dbs_per_call; j++) {
+        int expired;
+        redisDb *db = server.db+(current_db % server.dbnum);
+
+        /* Increment the DB now so we are sure if we run out of time
+         * in the current DB we'll restart from the next. This allows to
+         * distribute the time evenly across DBs. */
+        current_db++;
+
+        /* Continue to expire if at the end of the cycle more than 25%
+         * of the keys were expired. */
+        do {
+            unsigned long num, slots;
+            long long now, ttl_sum;
+            int ttl_samples;
+
+            /* If there is nothing to expire try next DB ASAP. */
+            if ((num = dictSize(db->expires)) == 0) {
+                db->avg_ttl = 0;
+                break;
+            }
+            slots = dictSlots(db->expires);
+            now = mstime();
+
+            /* When there are less than 1% filled slots getting random
+             * keys is expensive, so stop here waiting for better times...
+             * The dictionary will be resized asap. */
+            if (num && slots > DICT_HT_INITIAL_SIZE &&
+                (num*100/slots < 1)) break;
+
+            /* The main collection cycle. Sample random keys among keys
+             * with an expire set, checking for expired ones. */
+            expired = 0;
+            ttl_sum = 0;
+            ttl_samples = 0;
+
+            if (num > ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP)
+                num = ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP;
+
+            while (num--) {
+                dictEntry *de;
+                long long ttl;
+
+                if ((de = dictGetRandomKey(db->expires)) == NULL) break;
+                ttl = dictGetSignedIntegerVal(de)-now;
+                if (activeExpireCycleTryExpire(db,de,now)) expired++;
+                if (ttl > 0) {
+                    /* We want the average TTL of keys yet not expired. */
+                    ttl_sum += ttl;
+                    ttl_samples++;
+                }
+            }
+
+            /* Update the average TTL stats for this database. */
+            if (ttl_samples) {
+                long long avg_ttl = ttl_sum/ttl_samples;
+
+                /* Do a simple running average with a few samples.
+                 * We just use the current estimate with a weight of 2%
+                 * and the previous estimate with a weight of 98%. */
+                if (db->avg_ttl == 0) db->avg_ttl = avg_ttl;
+                db->avg_ttl = (db->avg_ttl/50)*49 + (avg_ttl/50);
+            }
+
+            /* We can't block forever here even if there are many keys to
+             * expire. So after a given amount of milliseconds return to the
+             * caller waiting for the other active expire cycle. */
+            iteration++;
+            if ((iteration & 0xf) == 0) { /* check once every 16 iterations. */
+                long long elapsed = ustime()-start;
+
+                latencyAddSampleIfNeeded("expire-cycle",elapsed/1000);
+                if (elapsed > timelimit) timelimit_exit = 1;
+            }
+            if (timelimit_exit) return;
+            /* We don't repeat the cycle if there are less than 25% of keys
+             * found expired in the current DB. */
+        } while (expired > ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP/4);
+    }
+}
+
+/*-----------------------------------------------------------------------------
+ * Expires Commands
+ *----------------------------------------------------------------------------*/
+
+/* This is the generic command implementation for EXPIRE, PEXPIRE, EXPIREAT
+ * and PEXPIREAT. Because the commad second argument may be relative or absolute
+ * the "basetime" argument is used to signal what the base time is (either 0
+ * for *AT variants of the command, or the current time for relative expires).
+ *
+ * unit is either UNIT_SECONDS or UNIT_MILLISECONDS, and is only used for
+ * the argv[2] parameter. The basetime is always specified in milliseconds. */
+void expireGenericCommand(client *c, long long basetime, int unit) {
+    robj *key = c->argv[1], *param = c->argv[2];
+    long long when; /* unix time in milliseconds when the key will expire. */
+
+    if (getLongLongFromObjectOrReply(c, param, &when, NULL) != C_OK)
+        return;
+
+    if (unit == UNIT_SECONDS) when *= 1000;
+    when += basetime;
+
+    /* No key, return zero. */
+    if (lookupKeyWrite(c->db,key) == NULL) {
+        addReply(c,shared.czero);
+        return;
+    }
+
+    /* EXPIRE with negative TTL, or EXPIREAT with a timestamp into the past
+     * should never be executed as a DEL when load the AOF or in the context
+     * of a slave instance.
+     *
+     * Instead we take the other branch of the IF statement setting an expire
+     * (possibly in the past) and wait for an explicit DEL from the master. */
+    if (when <= mstime() && !server.loading && !server.masterhost) {
+        robj *aux;
+
+        int deleted = server.lazyfree_lazy_expire ? dbAsyncDelete(c->db,key) :
+                                                    dbSyncDelete(c->db,key);
+        serverAssertWithInfo(c,key,deleted);
+        server.dirty++;
+
+        /* Replicate/AOF this as an explicit DEL or UNLINK. */
+        aux = server.lazyfree_lazy_expire ? shared.unlink : shared.del;
+        rewriteClientCommandVector(c,2,aux,key);
+        signalModifiedKey(c->db,key);
+        notifyKeyspaceEvent(NOTIFY_GENERIC,"del",key,c->db->id);
+        addReply(c, shared.cone);
+        return;
+    } else {
+        setExpire(c->db,key,when);
+        addReply(c,shared.cone);
+        signalModifiedKey(c->db,key);
+        notifyKeyspaceEvent(NOTIFY_GENERIC,"expire",key,c->db->id);
+        server.dirty++;
+        return;
+    }
+}
+
+/* EXPIRE key seconds */
+void expireCommand(client *c) {
+    expireGenericCommand(c,mstime(),UNIT_SECONDS);
+}
+
+/* EXPIREAT key time */
+void expireatCommand(client *c) {
+    expireGenericCommand(c,0,UNIT_SECONDS);
+}
+
+/* PEXPIRE key milliseconds */
+void pexpireCommand(client *c) {
+    expireGenericCommand(c,mstime(),UNIT_MILLISECONDS);
+}
+
+/* PEXPIREAT key ms_time */
+void pexpireatCommand(client *c) {
+    expireGenericCommand(c,0,UNIT_MILLISECONDS);
+}
+
+/* Implements TTL and PTTL */
+void ttlGenericCommand(client *c, int output_ms) {
+    long long expire, ttl = -1;
+
+    /* If the key does not exist at all, return -2 */
+    if (lookupKeyReadWithFlags(c->db,c->argv[1],LOOKUP_NOTOUCH) == NULL) {
+        addReplyLongLong(c,-2);
+        return;
+    }
+    /* The key exists. Return -1 if it has no expire, or the actual
+     * TTL value otherwise. */
+    expire = getExpire(c->db,c->argv[1]);
+    if (expire != -1) {
+        ttl = expire-mstime();
+        if (ttl < 0) ttl = 0;
+    }
+    if (ttl == -1) {
+        addReplyLongLong(c,-1);
+    } else {
+        addReplyLongLong(c,output_ms ? ttl : ((ttl+500)/1000));
+    }
+}
+
+/* TTL key */
+void ttlCommand(client *c) {
+    ttlGenericCommand(c, 0);
+}
+
+/* PTTL key */
+void pttlCommand(client *c) {
+    ttlGenericCommand(c, 1);
+}
+
+/* PERSIST key */
+void persistCommand(client *c) {
+    dictEntry *de;
+
+    de = dictFind(c->db->dict,c->argv[1]->ptr);
+    if (de == NULL) {
+        addReply(c,shared.czero);
+    } else {
+        if (removeExpire(c->db,c->argv[1])) {
+            addReply(c,shared.cone);
+            server.dirty++;
+        } else {
+            addReply(c,shared.czero);
+        }
+    }
+}
+
+/* TOUCH key1 [key2 key3 ... keyN] */
+void touchCommand(client *c) {
+    int touched = 0;
+    for (int j = 1; j < c->argc; j++)
+        if (lookupKeyRead(c->db,c->argv[j]) != NULL) touched++;
+    addReplyLongLong(c,touched);
+}
+

src/geo.c

@@ -1,6 +1,6 @@
 /*
  * Copyright (c) 2014, Matt Stancliff <matt@genges.com>.
- * Copyright (c) 2015, Salvatore Sanfilippo <antirez@gmail.com>.
+ * Copyright (c) 2015-2016, Salvatore Sanfilippo <antirez@gmail.com>.
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -30,6 +30,7 @@
 
 #include "geo.h"
 #include "geohash_helper.h"
+#include "debugmacro.h"
 
 /* Things exported from t_zset.c only for geo.c, since it is the only other
  * part of Redis that requires close zset introspection. */
@@ -156,14 +157,21 @@ double extractDistanceOrReply(client *c, robj **argv,
         return -1;
     }
 
+    if (distance < 0) {
+        addReplyError(c,"radius cannot be negative");
+        return -1;
+    }
+    
     double to_meters = extractUnitOrReply(c,argv[1]);
-    if (to_meters < 0) return -1;
+    if (to_meters < 0) {
+        return -1;
+    }
 
     if (conversion) *conversion = to_meters;
     return distance * to_meters;
 }
 
-/* The defailt addReplyDouble has too much accuracy.  We use this
+/* The default addReplyDouble has too much accuracy.  We use this
  * for returning location distances. "5.2145 meters away" is nicer
  * than "5.2144992818115 meters away." We provide 4 digits after the dot
  * so that the returned value is decently accurate even when the unit is
@@ -425,10 +433,12 @@ void geoaddCommand(client *c) {
 #define RADIUS_MEMBER 2
 
 /* GEORADIUS key x y radius unit [WITHDIST] [WITHHASH] [WITHCOORD] [ASC|DESC]
- *                               [COUNT count]
+ *                               [COUNT count] [STORE key] [STOREDIST key]
  * GEORADIUSBYMEMBER key member radius unit ... options ... */
 void georadiusGeneric(client *c, int type) {
     robj *key = c->argv[1];
+    robj *storekey = NULL;
+    int storedist = 0; /* 0 for STORE, 1 for STOREDIST. */
 
     /* Look up the requested zset */
     robj *zobj = NULL;
@@ -489,6 +499,14 @@ void georadiusGeneric(client *c, int type) {
                     return;
                 }
                 i++;
+            } else if (!strcasecmp(arg, "store") && (i+1) < remaining) {
+                storekey = c->argv[base_args+i+1];
+                storedist = 0;
+                i++;
+            } else if (!strcasecmp(arg, "storedist") && (i+1) < remaining) {
+                storekey = c->argv[base_args+i+1];
+                storedist = 1;
+                i++;
             } else {
                 addReply(c, shared.syntaxerr);
                 return;
@@ -496,6 +514,14 @@ void georadiusGeneric(client *c, int type) {
         }
     }
 
+    /* Trap options not compatible with STORE and STOREDIST. */
+    if (storekey && (withdist || withhash || withcoords)) {
+        addReplyError(c,
+            "STORE option in GEORADIUS is not compatible with "
+            "WITHDIST, WITHHASH and WITHCOORDS options");
+        return;
+    }
+
     /* COUNT without ordering does not make much sense, force ASC
      * ordering if COUNT was specified but no sorting was requested. */
     if (count != 0 && sort == SORT_NONE) sort = SORT_ASC;
@@ -509,33 +535,17 @@ void georadiusGeneric(client *c, int type) {
     membersOfAllNeighbors(zobj, georadius, xy[0], xy[1], radius_meters, ga);
 
     /* If no matching results, the user gets an empty reply. */
-    if (ga->used == 0) {
+    if (ga->used == 0 && storekey == NULL) {
         addReply(c, shared.emptymultibulk);
         geoArrayFree(ga);
         return;
     }
 
     long result_length = ga->used;
+    long returned_items = (count == 0 || result_length < count) ?
+                          result_length : count;
     long option_length = 0;
 
-    /* Our options are self-contained nested multibulk replies, so we
-     * only need to track how many of those nested replies we return. */
-    if (withdist)
-        option_length++;
-
-    if (withcoords)
-        option_length++;
-
-    if (withhash)
-        option_length++;
-
-    /* The multibulk len we send is exactly result_length. The result is either
-     * all strings of just zset members  *or* a nested multi-bulk reply
-     * containing the zset member string _and_ all the additional options the
-     * user enabled for this request. */
-    addReplyMultiBulkLen(c, (count == 0 || result_length < count) ?
-                            result_length : count);
-
     /* Process [optional] requested sorting */
     if (sort == SORT_ASC) {
         qsort(ga->array, result_length, sizeof(geoPoint), sort_gp_asc);
@@ -543,35 +553,91 @@ void georadiusGeneric(client *c, int type) {
         qsort(ga->array, result_length, sizeof(geoPoint), sort_gp_desc);
     }
 
-    /* Finally send results back to the caller */
-    int i;
-    for (i = 0; i < result_length; i++) {
-        geoPoint *gp = ga->array+i;
-        gp->dist /= conversion; /* Fix according to unit. */
-
-        /* If we have options in option_length, return each sub-result
-         * as a nested multi-bulk.  Add 1 to account for result value itself. */
-        if (option_length)
-            addReplyMultiBulkLen(c, option_length + 1);
-
-        addReplyBulkSds(c,gp->member);
-        gp->member = NULL;
+    if (storekey == NULL) {
+        /* No target key, return results to user. */
 
+        /* Our options are self-contained nested multibulk replies, so we
+         * only need to track how many of those nested replies we return. */
         if (withdist)
-            addReplyDoubleDistance(c, gp->dist);
+            option_length++;
+
+        if (withcoords)
+            option_length++;
 
         if (withhash)
-            addReplyLongLong(c, gp->score);
+            option_length++;
 
-        if (withcoords) {
-            addReplyMultiBulkLen(c, 2);
-            addReplyDouble(c, gp->longitude);
-            addReplyDouble(c, gp->latitude);
+        /* The multibulk len we send is exactly result_length. The result is
+         * either all strings of just zset members  *or* a nested multi-bulk
+         * reply containing the zset member string _and_ all the additional
+         * options the user enabled for this request. */
+        addReplyMultiBulkLen(c, returned_items);
+
+        /* Finally send results back to the caller */
+        int i;
+        for (i = 0; i < returned_items; i++) {
+            geoPoint *gp = ga->array+i;
+            gp->dist /= conversion; /* Fix according to unit. */
+
+            /* If we have options in option_length, return each sub-result
+             * as a nested multi-bulk.  Add 1 to account for result value
+             * itself. */
+            if (option_length)
+                addReplyMultiBulkLen(c, option_length + 1);
+
+            addReplyBulkSds(c,gp->member);
+            gp->member = NULL;
+
+            if (withdist)
+                addReplyDoubleDistance(c, gp->dist);
+
+            if (withhash)
+                addReplyLongLong(c, gp->score);
+
+            if (withcoords) {
+                addReplyMultiBulkLen(c, 2);
+                addReplyHumanLongDouble(c, gp->longitude);
+                addReplyHumanLongDouble(c, gp->latitude);
+            }
+        }
+    } else {
+        /* Target key, create a sorted set with the results. */
+        robj *zobj;
+        zset *zs;
+        int i;
+        size_t maxelelen = 0;
+
+        if (returned_items) {
+            zobj = createZsetObject();
+            zs = zobj->ptr;
         }
 
-        /* Stop if COUNT was specified and we already provided the
-         * specified number of elements. */
-        if (count != 0 && count == i+1) break;
+        for (i = 0; i < returned_items; i++) {
+            zskiplistNode *znode;
+            geoPoint *gp = ga->array+i;
+            gp->dist /= conversion; /* Fix according to unit. */
+            double score = storedist ? gp->dist : gp->score;
+            size_t elelen = sdslen(gp->member);
+
+            if (maxelelen < elelen) maxelelen = elelen;
+            znode = zslInsert(zs->zsl,score,gp->member);
+            serverAssert(dictAdd(zs->dict,gp->member,&znode->score) == DICT_OK);
+            gp->member = NULL;
+        }
+
+        if (returned_items) {
+            zsetConvertToZiplistIfNeeded(zobj,maxelelen);
+            setKey(c->db,storekey,zobj);
+            decrRefCount(zobj);
+            notifyKeyspaceEvent(NOTIFY_LIST,"georadiusstore",storekey,
+                                c->db->id);
+            server.dirty += returned_items;
+        } else if (dbDelete(c->db,storekey)) {
+            signalModifiedKey(c->db,storekey);
+            notifyKeyspaceEvent(NOTIFY_GENERIC,"del",storekey,c->db->id);
+            server.dirty++;
+        }
+        addReplyLongLong(c, returned_items);
     }
     geoArrayFree(ga);
 }
@@ -668,8 +734,8 @@ void geoposCommand(client *c) {
                 continue;
             }
             addReplyMultiBulkLen(c,2);
-            addReplyDouble(c,xy[0]);
-            addReplyDouble(c,xy[1]);
+            addReplyHumanLongDouble(c,xy[0]);
+            addReplyHumanLongDouble(c,xy[1]);
         }
     }
 }
@@ -709,6 +775,6 @@ void geodistCommand(client *c) {
     if (!decodeGeohash(score1,xyxy) || !decodeGeohash(score2,xyxy+2))
         addReply(c,shared.nullbulk);
     else
-        addReplyDouble(c,
+        addReplyDoubleDistance(c,
             geohashGetDistance(xyxy[0],xyxy[1],xyxy[2],xyxy[3]) / to_meter);
 }

src/geohash.c

@@ -1,7 +1,7 @@
 /*
  * Copyright (c) 2013-2014, yinqiwen <yinqiwen@gmail.com>
  * Copyright (c) 2014, Matt Stancliff <matt@genges.com>.
- * Copyright (c) 2015, Salvatore Sanfilippo <antirez@gmail.com>.
+ * Copyright (c) 2015-2016, Salvatore Sanfilippo <antirez@gmail.com>.
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -118,7 +118,7 @@ void geohashGetCoordRange(GeoHashRange *long_range, GeoHashRange *lat_range) {
     lat_range->min = GEO_LAT_MIN;
 }
 
-int geohashEncode(GeoHashRange *long_range, GeoHashRange *lat_range,
+int geohashEncode(const GeoHashRange *long_range, const GeoHashRange *lat_range,
                   double longitude, double latitude, uint8_t step,
                   GeoHashBits *hash) {
     /* Check basic arguments sanity. */
@@ -151,7 +151,7 @@ int geohashEncode(GeoHashRange *long_range, GeoHashRange *lat_range,
 }
 
 int geohashEncodeType(double longitude, double latitude, uint8_t step, GeoHashBits *hash) {
-    GeoHashRange r[2] = { { 0 } };
+    GeoHashRange r[2] = {{0}};
     geohashGetCoordRange(&r[0], &r[1]);
     return geohashEncode(&r[0], &r[1], longitude, latitude, step, hash);
 }
@@ -194,7 +194,7 @@ int geohashDecode(const GeoHashRange long_range, const GeoHashRange lat_range,
 }
 
 int geohashDecodeType(const GeoHashBits hash, GeoHashArea *area) {
-    GeoHashRange r[2] = { { 0 } };
+    GeoHashRange r[2] = {{0}};
     geohashGetCoordRange(&r[0], &r[1]);
     return geohashDecode(r[0], r[1], hash, area);
 }
@@ -211,7 +211,7 @@ int geohashDecodeAreaToLongLat(const GeoHashArea *area, double *xy) {
 }
 
 int geohashDecodeToLongLatType(const GeoHashBits hash, double *xy) {
-    GeoHashArea area = { { 0 } };
+    GeoHashArea area = {{0}};
     if (!xy || !geohashDecodeType(hash, &area))
         return 0;
     return geohashDecodeAreaToLongLat(&area, xy);

src/geohash.h

@@ -95,7 +95,7 @@ typedef struct {
  * -1:failed
  */
 void geohashGetCoordRange(GeoHashRange *long_range, GeoHashRange *lat_range);
-int geohashEncode(GeoHashRange *long_range, GeoHashRange *lat_range,
+int geohashEncode(const GeoHashRange *long_range, const GeoHashRange *lat_range,
                   double longitude, double latitude, uint8_t step,
                   GeoHashBits *hash);
 int geohashEncodeType(double longitude, double latitude,

src/geohash_helper.c

@@ -1,7 +1,7 @@
 /*
  * Copyright (c) 2013-2014, yinqiwen <yinqiwen@gmail.com>
  * Copyright (c) 2014, Matt Stancliff <matt@genges.com>.
- * Copyright (c) 2015, Salvatore Sanfilippo <antirez@gmail.com>.
+ * Copyright (c) 2015-2016, Salvatore Sanfilippo <antirez@gmail.com>.
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -34,7 +34,10 @@
  * https://github.com/yinqiwen/ardb/blob/d42503/src/geo/geohash_helper.cpp
  */
 
+#include "fmacros.h"
 #include "geohash_helper.h"
+#include "debugmacro.h"
+#include <math.h>
 
 #define D_R (M_PI / 180.0)
 #define R_MAJOR 6378137.0
@@ -54,8 +57,8 @@ const double MERCATOR_MIN = -20037726.37;
 static inline double deg_rad(double ang) { return ang * D_R; }
 static inline double rad_deg(double ang) { return ang / D_R; }
 
-/* You must *ONLY* estimate steps when you are encoding.
- * If you are decoding, always decode to GEO_STEP_MAX (26). */
+/* This function is used in order to estimate the step (bits precision)
+ * of the 9 search area boxes during radius queries. */
 uint8_t geohashEstimateStepsByRadius(double range_meters, double lat) {
     if (range_meters == 0) return 26;
     int step = 1;
@@ -63,24 +66,22 @@ uint8_t geohashEstimateStepsByRadius(double range_meters, double lat) {
         range_meters *= 2;
         step++;
     }
-    step -= 2; /* Make sure range is included in the worst case. */
+    step -= 2; /* Make sure range is included in most of the base cases. */
+
     /* Wider range torwards the poles... Note: it is possible to do better
      * than this approximation by computing the distance between meridians
      * at this latitude, but this does the trick for now. */
-    if (lat > 67 || lat < -67) step--;
-    if (lat > 80 || lat < -80) step--;
+    if (lat > 66 || lat < -66) {
+        step--;
+        if (lat > 80 || lat < -80) step--;
+    }
 
     /* Frame to valid range. */
     if (step < 1) step = 1;
-    if (step > 26) step = 25;
+    if (step > 26) step = 26;
     return step;
 }
 
-int geohashBitsComparator(const GeoHashBits *a, const GeoHashBits *b) {
-    /* If step not equal, compare on step.  Else, compare on bits. */
-    return a->step != b->step ? a->step - b->step : a->bits - b->bits;
-}
-
 int geohashBoundingBox(double longitude, double latitude, double radius_meters,
                        double *bounds) {
     if (!bounds) return 0;
@@ -89,6 +90,8 @@ int geohashBoundingBox(double longitude, double latitude, double radius_meters,
     lonr = deg_rad(longitude);
     latr = deg_rad(latitude);
 
+    if (radius_meters > EARTH_RADIUS_IN_METERS)
+        radius_meters = EARTH_RADIUS_IN_METERS;
     double distance = radius_meters / EARTH_RADIUS_IN_METERS;
     double min_latitude = latr - distance;
     double max_latitude = latr + distance;
@@ -106,12 +109,14 @@ int geohashBoundingBox(double longitude, double latitude, double radius_meters,
     return 1;
 }
 
+/* Return a set of areas (center + 8) that are able to cover a range query
+ * for the specified position and radius. */
 GeoHashRadius geohashGetAreasByRadius(double longitude, double latitude, double radius_meters) {
     GeoHashRange long_range, lat_range;
-    GeoHashRadius radius = { { 0 } };
-    GeoHashBits hash = { 0 };
-    GeoHashNeighbors neighbors = { { 0 } };
-    GeoHashArea area = { { 0 } };
+    GeoHashRadius radius;
+    GeoHashBits hash;
+    GeoHashNeighbors neighbors;
+    GeoHashArea area;
     double min_lon, max_lon, min_lat, max_lat;
     double bounds[4];
     int steps;
@@ -124,12 +129,65 @@ GeoHashRadius geohashGetAreasByRadius(double longitude, double latitude, double
 
     steps = geohashEstimateStepsByRadius(radius_meters,latitude);
 
-    geohashGetCoordRange(&long_range, &lat_range);
-    geohashEncode(&long_range, &lat_range, longitude, latitude, steps, &hash);
-    geohashNeighbors(&hash, &neighbors);
-    geohashGetCoordRange(&long_range, &lat_range);
-    geohashDecode(long_range, lat_range, hash, &area);
+    geohashGetCoordRange(&long_range,&lat_range);
+    geohashEncode(&long_range,&lat_range,longitude,latitude,steps,&hash);
+    geohashNeighbors(&hash,&neighbors);
+    geohashDecode(long_range,lat_range,hash,&area);
 
+    /* Check if the step is enough at the limits of the covered area.
+     * Sometimes when the search area is near an edge of the
+     * area, the estimated step is not small enough, since one of the
+     * north / south / west / east square is too near to the search area
+     * to cover everything. */
+    int decrease_step = 0;
+    {
+        GeoHashArea north, south, east, west;
+
+        geohashDecode(long_range, lat_range, neighbors.north, &north);
+        geohashDecode(long_range, lat_range, neighbors.south, &south);
+        geohashDecode(long_range, lat_range, neighbors.east, &east);
+        geohashDecode(long_range, lat_range, neighbors.west, &west);
+
+        if (geohashGetDistance(longitude,latitude,longitude,north.latitude.max)
+            < radius_meters) decrease_step = 1;
+        if (geohashGetDistance(longitude,latitude,longitude,south.latitude.min)
+            < radius_meters) decrease_step = 1;
+        if (geohashGetDistance(longitude,latitude,east.longitude.max,latitude)
+            < radius_meters) decrease_step = 1;
+        if (geohashGetDistance(longitude,latitude,west.longitude.min,latitude)
+            < radius_meters) decrease_step = 1;
+    }
+
+    if (decrease_step) {
+        steps--;
+        geohashEncode(&long_range,&lat_range,longitude,latitude,steps,&hash);
+        geohashNeighbors(&hash,&neighbors);
+        geohashDecode(long_range,lat_range,hash,&area);
+    }
+
+    /* Example debug info. This turns to be very useful every time there is
+     * to investigate radius search potential bugs. So better to leave it
+     * here. */
+    if (0) {
+        GeoHashArea myarea = {{0}};
+        geohashDecode(long_range, lat_range, neighbors.west, &myarea);
+
+        /* Dump West. */
+        D("Neighbors");
+        D("area.longitude.min: %f\n", myarea.longitude.min);
+        D("area.longitude.max: %f\n", myarea.longitude.max);
+        D("area.latitude.min: %f\n", myarea.latitude.min);
+        D("area.latitude.max: %f\n", myarea.latitude.max);
+
+        /* Dump center square. */
+        D("Area");
+        D("area.longitude.min: %f\n", area.longitude.min);
+        D("area.longitude.max: %f\n", area.longitude.max);
+        D("area.latitude.min: %f\n", area.latitude.min);
+        D("area.latitude.max: %f\n", area.latitude.max);
+    }
+
+    /* Exclude the search areas that are useless. */
     if (area.latitude.min < min_lat) {
         GZERO(neighbors.south);
         GZERO(neighbors.south_west);

src/geohash_helper.h

@@ -32,7 +32,6 @@
 #ifndef GEOHASH_HELPER_HPP_
 #define GEOHASH_HELPER_HPP_
 
-#include <math.h>
 #include "geohash.h"
 
 #define GZERO(s) s.bits = s.step = 0;

src/help.h

@@ -1,4 +1,4 @@
-/* Automatically generated by generate-command-help.rb, do not edit. */
+/* Automatically generated by utils/generate-command-help.rb, do not edit. */
 
 #ifndef __REDIS_HELP_H
 #define __REDIS_HELP_H
@@ -52,6 +52,11 @@ struct commandHelp {
     "Count set bits in a string",
     1,
     "2.6.0" },
+    { "BITFIELD",
+    "key [GET type offset] [SET type offset value] [INCRBY type offset increment] [OVERFLOW WRAP|SAT|FAIL]",
+    "Perform arbitrary bitfield integer operations on strings",
+    1,
+    "3.2.0" },
     { "BITOP",
     "operation destkey key [key ...]",
     "Perform bitwise operations between strings",
@@ -83,7 +88,7 @@ struct commandHelp {
     9,
     "2.6.9" },
     { "CLIENT KILL",
-    "[ip:port] [ID client-id] [TYPE normal|slave|pubsub] [ADDR ip:port] [SKIPME yes/no]",
+    "[ip:port] [ID client-id] [TYPE normal|master|slave|pubsub] [ADDR ip:port] [SKIPME yes/no]",
     "Kill the connection of a client",
     9,
     "2.4.0" },
@@ -97,6 +102,11 @@ struct commandHelp {
     "Stop processing commands from clients for some time",
     9,
     "2.9.50" },
+    { "CLIENT REPLY",
+    "ON|OFF|SKIP",
+    "Instruct the server whether to reply to commands",
+    9,
+    "3.2" },
     { "CLIENT SETNAME",
     "connection-name",
     "Set the current connection name",
@@ -179,7 +189,7 @@ struct commandHelp {
     "3.0.0" },
     { "CLUSTER SETSLOT",
     "slot IMPORTING|MIGRATING|STABLE|NODE [node-id]",
-    "Bind an hash slot to a specific node",
+    "Bind a hash slot to a specific node",
     12,
     "3.0.0" },
     { "CLUSTER SLAVES",
@@ -321,32 +331,32 @@ struct commandHelp {
     "key longitude latitude member [longitude latitude member ...]",
     "Add one or more geospatial items in the geospatial index represented using a sorted set",
     13,
-    "" },
+    "3.2.0" },
     { "GEODIST",
     "key member1 member2 [unit]",
     "Returns the distance between two members of a geospatial index",
     13,
-    "" },
+    "3.2.0" },
     { "GEOHASH",
     "key member [member ...]",
     "Returns members of a geospatial index as standard geohash strings",
     13,
-    "" },
+    "3.2.0" },
     { "GEOPOS",
     "key member [member ...]",
     "Returns longitude and latitude of members of a geospatial index",
     13,
-    "" },
+    "3.2.0" },
     { "GEORADIUS",
-    "key longitude latitude radius m|km|ft|mi [WITHCOORD] [WITHDIST] [WITHHASH] [COUNT count]",
+    "key longitude latitude radius m|km|ft|mi [WITHCOORD] [WITHDIST] [WITHHASH] [COUNT count] [ASC|DESC] [STORE key] [STOREDIST key]",
     "Query a sorted set representing a geospatial index to fetch members matching a given maximum distance from a point",
     13,
-    "" },
+    "3.2.0" },
     { "GEORADIUSBYMEMBER",
-    "key member radius m|km|ft|mi [WITHCOORD] [WITHDIST] [WITHHASH] [COUNT count]",
+    "key member radius m|km|ft|mi [WITHCOORD] [WITHDIST] [WITHHASH] [COUNT count] [ASC|DESC] [STORE key] [STOREDIST key]",
     "Query a sorted set representing a geospatial index to fetch members matching a given maximum distance from a member",
     13,
-    "" },
+    "3.2.0" },
     { "GET",
     "key",
     "Get the value of a key",
@@ -528,7 +538,7 @@ struct commandHelp {
     1,
     "1.0.0" },
     { "MIGRATE",
-    "host port key destination-db timeout [COPY] [REPLACE]",
+    "host port key|"" destination-db timeout [COPY] [REPLACE] [KEYS key]",
     "Atomically transfer a key from a Redis instance to another one.",
     0,
     "2.6.0" },
@@ -593,7 +603,7 @@ struct commandHelp {
     11,
     "2.8.9" },
     { "PING",
-    "-",
+    "[message]",
     "Ping the server",
     8,
     "1.0.0" },
@@ -707,6 +717,11 @@ struct commandHelp {
     "Get the number of members in a set",
     3,
     "1.0.0" },
+    { "SCRIPT DEBUG",
+    "YES|SYNC|NO",
+    "Set the debug mode for executed scripts.",
+    10,
+    "3.2.0" },
     { "SCRIPT EXISTS",
     "script [script ...]",
     "Check existence of scripts in the script cache.",
@@ -768,7 +783,7 @@ struct commandHelp {
     1,
     "2.2.0" },
     { "SHUTDOWN",
-    "[NOSAVE] [SAVE]",
+    "[NOSAVE|SAVE]",
     "Synchronously save the dataset to disk and then shut down the server",
     9,
     "1.0.0" },

src/intset.c

@@ -272,7 +272,7 @@ uint8_t intsetGet(intset *is, uint32_t pos, int64_t *value) {
 }
 
 /* Return intset length */
-uint32_t intsetLen(intset *is) {
+uint32_t intsetLen(const intset *is) {
     return intrev32ifbe(is->length);
 }
 

src/intset.h

@@ -44,7 +44,7 @@ intset *intsetRemove(intset *is, int64_t value, int *success);
 uint8_t intsetFind(intset *is, int64_t value);
 int64_t intsetRandom(intset *is);
 uint8_t intsetGet(intset *is, uint32_t pos, int64_t *value);
-uint32_t intsetLen(intset *is);
+uint32_t intsetLen(const intset *is);
 size_t intsetBlobLen(intset *is);
 
 #ifdef REDIS_TEST

src/latency.c

@@ -79,7 +79,7 @@ int THPIsEnabled(void) {
  * value of the function is non-zero, the process is being targeted by
  * THP support, and is likely to have memory usage / latency issues. */
 int THPGetAnonHugePagesSize(void) {
-    return zmalloc_get_smap_bytes_by_field("AnonHugePages:");
+    return zmalloc_get_smap_bytes_by_field("AnonHugePages:",-1);
 }
 
 /* ---------------------------- Latency API --------------------------------- */

src/lazyfree.c

@@ -57,7 +57,7 @@ int dbAsyncDelete(redisDb *db, robj *key) {
     /* If the value is composed of a few allocations, to free in a lazy way
      * is actually just slower... So under a certain limit we just free
      * the object synchronously. */
-    dictEntry *de = dictFind(db->dict,key->ptr);
+    dictEntry *de = dictUnlink(db->dict,key->ptr);
     if (de) {
         robj *val = dictGetVal(de);
         size_t free_effort = lazyfreeGetFreeEffort(val);
@@ -65,7 +65,7 @@ int dbAsyncDelete(redisDb *db, robj *key) {
         /* If releasing the object is too much work, let's put it into the
          * lazy free list. */
         if (free_effort > LAZYFREE_THRESHOLD) {
-            atomicIncr(lazyfree_objects,1,&lazyfree_objects_mutex);
+            atomicIncr(lazyfree_objects,1,lazyfree_objects_mutex);
             bioCreateBackgroundJob(BIO_LAZY_FREE,val,NULL,NULL);
             dictSetVal(db->dict,de,NULL);
         }
@@ -73,7 +73,8 @@ int dbAsyncDelete(redisDb *db, robj *key) {
 
     /* Release the key-val pair, or just the key if we set the val
      * field to NULL in order to lazy free it later. */
-    if (dictDelete(db->dict,key->ptr) == DICT_OK) {
+    if (de) {
+        dictFreeUnlinkedEntry(db->dict,de);
         if (server.cluster_enabled) slotToKeyDel(key);
         return 1;
     } else {
@@ -89,7 +90,7 @@ void emptyDbAsync(redisDb *db) {
     db->dict = dictCreate(&dbDictType,NULL);
     db->expires = dictCreate(&keyptrDictType,NULL);
     atomicIncr(lazyfree_objects,dictSize(oldht1),
-        &lazyfree_objects_mutex);
+        lazyfree_objects_mutex);
     bioCreateBackgroundJob(BIO_LAZY_FREE,NULL,oldht1,oldht2);
 }
 
@@ -99,7 +100,7 @@ void slotToKeyFlushAsync(void) {
     zskiplist *oldsl = server.cluster->slots_to_keys;
     server.cluster->slots_to_keys = zslCreate();
     atomicIncr(lazyfree_objects,oldsl->length,
-        &lazyfree_objects_mutex);
+        lazyfree_objects_mutex);
     bioCreateBackgroundJob(BIO_LAZY_FREE,NULL,NULL,oldsl);
 }
 
@@ -107,7 +108,7 @@ void slotToKeyFlushAsync(void) {
  * updating the count of objects to release. */
 void lazyfreeFreeObjectFromBioThread(robj *o) {
     decrRefCount(o);
-    atomicDecr(lazyfree_objects,1,&lazyfree_objects_mutex);
+    atomicDecr(lazyfree_objects,1,lazyfree_objects_mutex);
 }
 
 /* Release a database from the lazyfree thread. The 'db' pointer is the
@@ -119,7 +120,7 @@ void lazyfreeFreeDatabaseFromBioThread(dict *ht1, dict *ht2) {
     size_t numkeys = dictSize(ht1);
     dictRelease(ht1);
     dictRelease(ht2);
-    atomicDecr(lazyfree_objects,numkeys,&lazyfree_objects_mutex);
+    atomicDecr(lazyfree_objects,numkeys,lazyfree_objects_mutex);
 }
 
 /* Release the skiplist mapping Redis Cluster keys to slots in the
@@ -127,5 +128,5 @@ void lazyfreeFreeDatabaseFromBioThread(dict *ht1, dict *ht2) {
 void lazyfreeFreeSlotsMapFromBioThread(zskiplist *sl) {
     size_t len = sl->length;
     zslFree(sl);
-    atomicDecr(lazyfree_objects,len,&lazyfree_objects_mutex);
+    atomicDecr(lazyfree_objects,len,lazyfree_objects_mutex);
 }

src/memtest.c

@@ -26,7 +26,7 @@
  * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
  * POSSIBILITY OF SUCH DAMAGE.
  */
-
+#include <stdint.h>
 #include <stdlib.h>
 #include <stdio.h>
 #include <string.h>
@@ -90,7 +90,7 @@ void memtest_progress_step(size_t curr, size_t size, char c) {
 /* Test that addressing is fine. Every location is populated with its own
  * address, and finally verified. This test is very fast but may detect
  * ASAP big issues with the memory subsystem. */
-void memtest_addressing(unsigned long *l, size_t bytes) {
+int memtest_addressing(unsigned long *l, size_t bytes, int interactive) {
     unsigned long words = bytes/sizeof(unsigned long);
     unsigned long j, *p;
 
@@ -99,48 +99,60 @@ void memtest_addressing(unsigned long *l, size_t bytes) {
     for (j = 0; j < words; j++) {
         *p = (unsigned long)p;
         p++;
-        if ((j & 0xffff) == 0) memtest_progress_step(j,words*2,'A');
+        if ((j & 0xffff) == 0 && interactive)
+            memtest_progress_step(j,words*2,'A');
     }
     /* Test */
     p = l;
     for (j = 0; j < words; j++) {
         if (*p != (unsigned long)p) {
-            printf("\n*** MEMORY ADDRESSING ERROR: %p contains %lu\n",
-                (void*) p, *p);
-            exit(1);
+            if (interactive) {
+                printf("\n*** MEMORY ADDRESSING ERROR: %p contains %lu\n",
+                    (void*) p, *p);
+                exit(1);
+            }
+            return 1;
         }
         p++;
-        if ((j & 0xffff) == 0) memtest_progress_step(j+words,words*2,'A');
+        if ((j & 0xffff) == 0 && interactive)
+            memtest_progress_step(j+words,words*2,'A');
     }
+    return 0;
 }
 
 /* Fill words stepping a single page at every write, so we continue to
  * touch all the pages in the smallest amount of time reducing the
  * effectiveness of caches, and making it hard for the OS to transfer
- * pages on the swap. */
-void memtest_fill_random(unsigned long *l, size_t bytes) {
+ * pages on the swap.
+ *
+ * In this test we can't call rand() since the system may be completely
+ * unable to handle library calls, so we have to resort to our own
+ * PRNG that only uses local state. We use an xorshift* PRNG. */
+#define xorshift64star_next() do { \
+        rseed ^= rseed >> 12; \
+        rseed ^= rseed << 25; \
+        rseed ^= rseed >> 27; \
+        rout = rseed * UINT64_C(2685821657736338717); \
+} while(0)
+
+void memtest_fill_random(unsigned long *l, size_t bytes, int interactive) {
     unsigned long step = 4096/sizeof(unsigned long);
     unsigned long words = bytes/sizeof(unsigned long)/2;
     unsigned long iwords = words/step;  /* words per iteration */
     unsigned long off, w, *l1, *l2;
+    uint64_t rseed = UINT64_C(0xd13133de9afdb566); /* Just a random seed. */
+    uint64_t rout = 0;
 
     assert((bytes & 4095) == 0);
     for (off = 0; off < step; off++) {
         l1 = l+off;
         l2 = l1+words;
         for (w = 0; w < iwords; w++) {
-#ifdef MEMTEST_32BIT
-            *l1 = *l2 = ((unsigned long)     (rand()&0xffff)) |
-                        (((unsigned long)    (rand()&0xffff)) << 16);
-#else
-            *l1 = *l2 = ((unsigned long)     (rand()&0xffff)) |
-                        (((unsigned long)    (rand()&0xffff)) << 16) |
-                        (((unsigned long)    (rand()&0xffff)) << 32) |
-                        (((unsigned long)    (rand()&0xffff)) << 48);
-#endif
+            xorshift64star_next();
+            *l1 = *l2 = (unsigned long) rout;
             l1 += step;
             l2 += step;
-            if ((w & 0xffff) == 0)
+            if ((w & 0xffff) == 0 && interactive)
                 memtest_progress_step(w+iwords*off,words,'R');
         }
     }
@@ -149,7 +161,7 @@ void memtest_fill_random(unsigned long *l, size_t bytes) {
 /* Like memtest_fill_random() but uses the two specified values to fill
  * memory, in an alternated way (v1|v2|v1|v2|...) */
 void memtest_fill_value(unsigned long *l, size_t bytes, unsigned long v1,
-                        unsigned long v2, char sym)
+                        unsigned long v2, char sym, int interactive)
 {
     unsigned long step = 4096/sizeof(unsigned long);
     unsigned long words = bytes/sizeof(unsigned long)/2;
@@ -173,13 +185,13 @@ void memtest_fill_value(unsigned long *l, size_t bytes, unsigned long v1,
 #endif
             l1 += step;
             l2 += step;
-            if ((w & 0xffff) == 0)
+            if ((w & 0xffff) == 0 && interactive)
                 memtest_progress_step(w+iwords*off,words,sym);
         }
     }
 }
 
-void memtest_compare(unsigned long *l, size_t bytes) {
+int memtest_compare(unsigned long *l, size_t bytes, int interactive) {
     unsigned long words = bytes/sizeof(unsigned long)/2;
     unsigned long w, *l1, *l2;
 
@@ -188,93 +200,158 @@ void memtest_compare(unsigned long *l, size_t bytes) {
     l2 = l1+words;
     for (w = 0; w < words; w++) {
         if (*l1 != *l2) {
-            printf("\n*** MEMORY ERROR DETECTED: %p != %p (%lu vs %lu)\n",
-                (void*)l1, (void*)l2, *l1, *l2);
-            exit(1);
+            if (interactive) {
+                printf("\n*** MEMORY ERROR DETECTED: %p != %p (%lu vs %lu)\n",
+                    (void*)l1, (void*)l2, *l1, *l2);
+                exit(1);
+            }
+            return 1;
         }
         l1 ++;
         l2 ++;
-        if ((w & 0xffff) == 0) memtest_progress_step(w,words,'=');
+        if ((w & 0xffff) == 0 && interactive)
+            memtest_progress_step(w,words,'=');
     }
+    return 0;
 }
 
-void memtest_compare_times(unsigned long *m, size_t bytes, int pass, int times) {
+int memtest_compare_times(unsigned long *m, size_t bytes, int pass, int times,
+                          int interactive)
+{
     int j;
+    int errors = 0;
 
     for (j = 0; j < times; j++) {
-        memtest_progress_start("Compare",pass);
-        memtest_compare(m,bytes);
-        memtest_progress_end();
+        if (interactive) memtest_progress_start("Compare",pass);
+        errors += memtest_compare(m,bytes,interactive);
+        if (interactive) memtest_progress_end();
     }
+    return errors;
 }
 
-void memtest_test(size_t megabytes, int passes) {
+/* Test the specified memory. The number of bytes must be multiple of 4096.
+ * If interactive is true the program exists with an error and prints
+ * ASCII arts to show progresses. Instead when interactive is 0, it can
+ * be used as an API call, and returns 1 if memory errors were found or
+ * 0 if there were no errors detected. */
+int memtest_test(unsigned long *m, size_t bytes, int passes, int interactive) {
+    int pass = 0;
+    int errors = 0;
+
+    while (pass != passes) {
+        pass++;
+
+        if (interactive) memtest_progress_start("Addressing test",pass);
+        errors += memtest_addressing(m,bytes,interactive);
+        if (interactive) memtest_progress_end();
+
+        if (interactive) memtest_progress_start("Random fill",pass);
+        memtest_fill_random(m,bytes,interactive);
+        if (interactive) memtest_progress_end();
+        errors += memtest_compare_times(m,bytes,pass,4,interactive);
+
+        if (interactive) memtest_progress_start("Solid fill",pass);
+        memtest_fill_value(m,bytes,0,(unsigned long)-1,'S',interactive);
+        if (interactive) memtest_progress_end();
+        errors += memtest_compare_times(m,bytes,pass,4,interactive);
+
+        if (interactive) memtest_progress_start("Checkerboard fill",pass);
+        memtest_fill_value(m,bytes,ULONG_ONEZERO,ULONG_ZEROONE,'C',interactive);
+        if (interactive) memtest_progress_end();
+        errors += memtest_compare_times(m,bytes,pass,4,interactive);
+    }
+    return errors;
+}
+
+/* A version of memtest_test() that tests memory in small pieces
+ * in order to restore the memory content at exit.
+ *
+ * One problem we have with this approach, is that the cache can avoid
+ * real memory accesses, and we can't test big chunks of memory at the
+ * same time, because we need to backup them on the stack (the allocator
+ * may not be usable or we may be already in an out of memory condition).
+ * So what we do is to try to trash the cache with useless memory accesses
+ * between the fill and compare cycles. */
+#define MEMTEST_BACKUP_WORDS (1024*(1024/sizeof(long)))
+/* Random accesses of MEMTEST_DECACHE_SIZE are performed at the start and
+ * end of the region between fill and compare cycles in order to trash
+ * the cache. */
+#define MEMTEST_DECACHE_SIZE (1024*8)
+int memtest_preserving_test(unsigned long *m, size_t bytes, int passes) {
+    unsigned long backup[MEMTEST_BACKUP_WORDS];
+    unsigned long *p = m;
+    unsigned long *end = (unsigned long*) (((unsigned char*)m)+(bytes-MEMTEST_DECACHE_SIZE));
+    size_t left = bytes;
+    int errors = 0;
+
+    if (bytes & 4095) return 0; /* Can't test across 4k page boundaries. */
+    if (bytes < 4096*2) return 0; /* Can't test a single page. */
+
+    while(left) {
+        /* If we have to test a single final page, go back a single page
+         * so that we can test two pages, since the code can't test a single
+         * page but at least two. */
+        if (left == 4096) {
+            left += 4096;
+            p -= 4096/sizeof(unsigned long);
+        }
+
+        int pass = 0;
+        size_t len = (left > sizeof(backup)) ? sizeof(backup) : left;
+
+        /* Always test an even number of pages. */
+        if (len/4096 % 2) len -= 4096;
+
+        memcpy(backup,p,len); /* Backup. */
+        while(pass != passes) {
+            pass++;
+            errors += memtest_addressing(p,len,0);
+            memtest_fill_random(p,len,0);
+            if (bytes >= MEMTEST_DECACHE_SIZE) {
+                memtest_compare_times(m,MEMTEST_DECACHE_SIZE,pass,1,0);
+                memtest_compare_times(end,MEMTEST_DECACHE_SIZE,pass,1,0);
+            }
+            errors += memtest_compare_times(p,len,pass,4,0);
+            memtest_fill_value(p,len,0,(unsigned long)-1,'S',0);
+            if (bytes >= MEMTEST_DECACHE_SIZE) {
+                memtest_compare_times(m,MEMTEST_DECACHE_SIZE,pass,1,0);
+                memtest_compare_times(end,MEMTEST_DECACHE_SIZE,pass,1,0);
+            }
+            errors += memtest_compare_times(p,len,pass,4,0);
+            memtest_fill_value(p,len,ULONG_ONEZERO,ULONG_ZEROONE,'C',0);
+            if (bytes >= MEMTEST_DECACHE_SIZE) {
+                memtest_compare_times(m,MEMTEST_DECACHE_SIZE,pass,1,0);
+                memtest_compare_times(end,MEMTEST_DECACHE_SIZE,pass,1,0);
+            }
+            errors += memtest_compare_times(p,len,pass,4,0);
+        }
+        memcpy(p,backup,len); /* Restore. */
+        left -= len;
+        p += len/sizeof(unsigned long);
+    }
+    return errors;
+}
+
+/* Perform an interactive test allocating the specified number of megabytes. */
+void memtest_alloc_and_test(size_t megabytes, int passes) {
     size_t bytes = megabytes*1024*1024;
     unsigned long *m = malloc(bytes);
-    int pass = 0;
 
     if (m == NULL) {
         fprintf(stderr,"Unable to allocate %zu megabytes: %s",
             megabytes, strerror(errno));
         exit(1);
     }
-    while (pass != passes) {
-        pass++;
-
-        memtest_progress_start("Addressing test",pass);
-        memtest_addressing(m,bytes);
-        memtest_progress_end();
-
-        memtest_progress_start("Random fill",pass);
-        memtest_fill_random(m,bytes);
-        memtest_progress_end();
-        memtest_compare_times(m,bytes,pass,4);
-
-        memtest_progress_start("Solid fill",pass);
-        memtest_fill_value(m,bytes,0,(unsigned long)-1,'S');
-        memtest_progress_end();
-        memtest_compare_times(m,bytes,pass,4);
-
-        memtest_progress_start("Checkerboard fill",pass);
-        memtest_fill_value(m,bytes,ULONG_ONEZERO,ULONG_ZEROONE,'C');
-        memtest_progress_end();
-        memtest_compare_times(m,bytes,pass,4);
-    }
+    memtest_test(m,bytes,passes,1);
     free(m);
 }
 
-void memtest_non_destructive_invert(void *addr, size_t size) {
-    volatile unsigned long *p = addr;
-    size_t words = size / sizeof(unsigned long);
-    size_t j;
-
-    /* Invert */
-    for (j = 0; j < words; j++)
-        p[j] = ~p[j];
-}
-
-void memtest_non_destructive_swap(void *addr, size_t size) {
-    volatile unsigned long *p = addr;
-    size_t words = size / sizeof(unsigned long);
-    size_t j;
-
-    /* Swap */
-    for (j = 0; j < words; j += 2) {
-        unsigned long a, b;
-
-        a = p[j];
-        b = p[j+1];
-        p[j] = b;
-        p[j+1] = a;
-    }
-}
-
 void memtest(size_t megabytes, int passes) {
     if (ioctl(1, TIOCGWINSZ, &ws) == -1) {
         ws.ws_col = 80;
         ws.ws_row = 20;
     }
-    memtest_test(megabytes,passes);
+    memtest_alloc_and_test(megabytes,passes);
     printf("\nYour memory passed this test.\n");
     printf("Please if you are still in doubt use the following two tools:\n");
     printf("1) memtest86: http://www.memtest86.com/\n");

src/modules/.gitignore

@@ -0,0 +1,2 @@
+*.so
+*.xo

src/modules/INTRO.md

@@ -0,0 +1,856 @@
+Redis Modules: an introduction to the API
+===
+
+The modules documentation is composed of the following files:
+
+* `INTRO.md` (this file). An overview about Redis Modules system and API. It's a good idea to start your reading here.
+* `API.md` is generated from module.c top comments of RedisMoule functions. It is a good reference in order to understand how each function works.
+* `TYPES.md` covers the implementation of native data types into modules.
+
+Redis modules make possible to extend Redis functionality using external
+modules, implementing new Redis commands at a speed and with features
+similar to what can be done inside the core itself.
+
+Redis modules are dynamic libraries, that can be loaded into Redis at
+startup or using the `MODULE LOAD` command. Redis exports a C API, in the
+form of a single C header file called `redismodule.h`. Modules are meant
+to be written in C, however it will be possible to use C++ or other languages
+that have C binding functionalities.
+
+Modules are designed in order to be loaded into different versions of Redis,
+so a given module does not need to be designed, or recompiled, in order to
+run with a specific version of Redis. For this reason, the module will
+register to the Redis core using a specific API version. The current API
+version is "1".
+
+This document is about an alpha version of Redis modules. API, functionalities
+and other details may change in the future.
+
+# Loading modules
+
+In order to test the module you are developing, you can load the module
+using the following `redis.conf` configuration directive:
+
+    loadmodule /path/to/mymodule.so
+
+It is also possible to load a module at runtime using the following command:
+
+    MODULE LOAD /path/to/mymodule.so
+
+In order to list all loaded modules, use:
+
+    MODULE LIST
+
+Finally, you can unload (and later reload if you wish) a module using the
+following command:
+
+    MODULE UNLOAD mymodule
+
+Note that `mymodule` above is not the filename without the `.so` suffix, but
+instead, the name the module used to register itself into the Redis core.
+The name can be obtained using `MODULE LIST`. However it is good practice
+that the filename of the dynamic library is the same as the name the module
+uses to register itself into the Redis core.
+
+# The simplest module you can write
+
+In order to show the different parts of a module, here we'll show a very
+simple module that implements a command that outputs a random number.
+
+    #include "redismodule.h"
+    #include <stdlib.h>
+
+    int HelloworldRand_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+        RedisModule_ReplyWithLongLong(ctx,rand());
+        return REDISMODULE_OK;
+    }
+
+    int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+        if (RedisModule_Init(ctx,"helloworld",1,REDISMODULE_APIVER_1)
+            == REDISMODULE_ERR) return REDISMODULE_ERR;
+
+        if (RedisModule_CreateCommand(ctx,"helloworld.rand",
+            HelloworldRand_RedisCommand) == REDISMODULE_ERR)
+            return REDISMODULE_ERR;
+
+        return REDISMODULE_OK;
+    }
+
+The example module has two functions. One implements a command called
+HELLOWORLD.RAND. This function is specific of that module. However the
+other function called `RedisModule_OnLoad()` must be present in each
+Redis module. It is the entry point for the module to be initialized,
+register its commands, and potentially other private data structures
+it uses.
+
+Note that it is a good idea for modules to call commands with the
+name of the module followed by a dot, and finally the command name,
+like in the case of `HELLOWORLD.RAND`. This way it is less likely to
+have collisions.
+
+Note that if different modules have colliding commands, they'll not be
+able to work in Redis at the same time, since the function
+`RedisModule_CreateCommand` will fail in one of the modules, so the module
+loading will abort returning an error condition.
+
+# Module initialization
+
+The above example shows the usage of the function `RedisModule_Init()`.
+It should be the first function called by the module `OnLoad` function.
+The following is the function prototype:
+
+    int RedisModule_Init(RedisModuleCtx *ctx, const char *modulename,
+                         int module_version, int api_version);
+
+The `Init` function announces the Redis core that the module has a given
+name, its version (that is reported by `MODULE LIST`), and that is willing
+to use a specific version of the API.
+
+If the API version is wrong, the name is already taken, or there are other
+similar errors, the function will return `REDISMODULE_ERR`, and the module
+`OnLoad` function should return ASAP with an error.
+
+Before the `Init` function is called, no other API function can be called,
+otherwise the module will segfault and the Redis instance will crash.
+
+The second function called, `RedisModule_CreateCommand`, is used in order
+to register commands into the Redis core. The following is the prototype:
+
+    int RedisModule_CreateCommand(RedisModuleCtx *ctx, const char *cmdname,
+                                  RedisModuleCmdFunc cmdfunc);
+
+As you can see, most Redis modules API calls all take as first argument
+the `context` of the module, so that they have a reference to the module
+calling it, to the command and client executing a given command, and so forth.
+
+To create a new command, the above function needs the context, the command
+name, and the function pointer of the function implementing the command,
+which must have the following prototype:
+
+
+    int mycommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc);
+
+The command function arguments are just the context, that will be passed
+to all the other API calls, the command argument vector, and total number
+of arguments, as passed by the user.
+
+As you can see, the arguments are provided as pointers to a specific data
+type, the `RedisModuleString`. This is an opaque data type you have API
+functions to access and use, direct access to its fields is never needed.
+
+Zooming into the example command implementation, we can find another call:
+
+    int RedisModule_ReplyWithLongLong(RedisModuleCtx *ctx, long long integer);
+
+This function returns an integer to the client that invoked the command,
+exactly like other Redis commands do, like for example `INCR` or `SCARD`.
+
+# Setup and dependencies of a Redis module
+
+Redis modules don't depend on Redis or some other library, nor they
+need to be compiled with a specific `redismodule.h` file. In order
+to create a new module, just copy a recent version of `redismodule.h`
+in your source tree, link all the libraries you want, and create
+a dynamic library having the `RedisModule_OnLoad()` function symbol
+exported.
+
+The module will be able to load into different versions of Redis.
+
+# Passing configuration parameters to Redis modules
+
+When the module is loaded with the `MODULE LOAD` command, or using the
+`loadmodule` directive in the `redis.conf` file, the user is able to pass
+configuration parameters to the module by adding arguments after the module
+file name:
+
+    loadmodule mymodule.so foo bar 1234
+
+In the above example the strings `foo`, `bar` and `123` will be passed
+to the module `OnLoad()` function in the `argv` argument as an array
+of RedisModuleString pointers. The number of arguments passed is into `argc`.
+
+The way you can access those strings will be explained in the rest of this
+document. Normally the module will store the module configuration parameters
+in some `static` global variable that can be accessed module wide, so that
+the configuration can change the behavior of different commands.
+
+# Working with RedisModuleString objects
+
+The command argument vector `argv` passed to module commands, and the
+return value of other module APIs functions, are of type `RedisModuleString`.
+
+Usually you directly pass module strings to other API calls, however sometimes
+you may need to directly access the string object.
+
+There are a few functions in order to work with string objects:
+
+    const char *RedisModule_StringPtrLen(RedisModuleString *string, size_t *len);
+
+The above function accesses a string by returning its pointer and setting its
+length in `len`.
+You should never write to a string object pointer, as you can see from the
+`const` pointer qualifier.
+
+However, if you want, you can create new string objects using the following
+API:
+
+    RedisModuleString *RedisModule_CreateString(RedisModuleCtx *ctx, const char *ptr, size_t len);
+
+The string returned by the above command must be freed using a corresponding
+call to `RedisModule_FreeString()`:
+
+    void RedisModule_FreeString(RedisModuleString *str);
+
+However if you want to avoid having to free strings, the automatic memory
+management, covered later in this document, can be a good alternative, by
+doing it for you.
+
+Note that the strings provided via the argument vector `argv` never need
+to be freed. You only need to free new strings you create, or new strings
+returned by other APIs, where it is specified that the returned string must
+be freed.
+
+## Creating strings from numbers or parsing strings as numbers
+
+Creating a new string from an integer is a very common operation, so there
+is a function to do this:
+
+    RedisModuleString *mystr = RedisModule_CreateStringFromLongLong(ctx,10);
+
+Similarly in order to parse a string as a number:
+
+    long long myval;
+    if (RedisModule_StringToLongLong(ctx,argv[1],&myval) == REDISMODULE_OK) {
+        /* Do something with 'myval' */
+    }
+
+## Accessing Redis keys from modules
+
+Most Redis modules, in order to be useful, have to interact with the Redis
+data space (this is not always true, for example an ID generator may
+never touch Redis keys). Redis modules have two different APIs in order to
+access the Redis data space, one is a low level API that provides very
+fast access and a set of functions to manipulate Redis data structures.
+The other API is more high level, and allows to call Redis commands and
+fetch the result, similarly to how Lua scripts access Redis.
+
+The high level API is also useful in order to access Redis functionalities
+that are not available as APIs.
+
+In general modules developers should prefer the low level API, because commands
+implemented using the low level API run at a speed comparable to the speed
+of native Redis commands. However there are definitely use cases for the
+higher level API. For example often the bottleneck could be processing the
+data and not accessing it.
+
+Also note that sometimes using the low level API is not harder compared to
+the higher level one.
+
+# Calling Redis commands
+
+The high level API to access Redis is the sum of the `RedisModule_Call()`
+function, together with the functions needed in order to access the
+reply object returned by `Call()`.
+
+`RedisModule_Call` uses a special calling convention, with a format specifier
+that is used to specify what kind of objects you are passing as arguments
+to the function.
+
+Redis commands are invoked just using a command name and a list of arguments.
+However when calling commands, the arguments may originate from different
+kind of strings: null-terminated C strings, RedisModuleString objects as
+received from the `argv` parameter in the command implementation, binary
+safe C buffers with a pointer and a length, and so forth.
+
+For example if I want to call `INCRBY` using a first argument (the key)
+a string received in the argument vector `argv`, which is an array
+of RedisModuleString object pointers, and a C string representing the
+number "10" as second argument (the increment), I'll use the following
+function call:
+
+    RedisModuleCallReply *reply;
+    reply = RedisModule_Call(ctx,"INCR","sc",argv[1],"10");
+
+The first argument is the context, and the second is always a null terminated
+C string with the command name. The third argument is the format specifier
+where each character corresponds to the type of the arguments that will follow.
+In the above case `"sc"` means a RedisModuleString object, and a null
+terminated C string. The other arguments are just the two arguments as
+specified. In fact `argv[1]` is a RedisModuleString and `"10"` is a null
+terminated C string.
+
+This is the full list of format specifiers:
+
+* **c** -- Null terminated C string pointer.
+* **b** -- C buffer, two arguments needed: C string pointer and `size_t` length.
+* **s** -- RedisModuleString as received in `argv` or by other Redis module APIs returning a RedisModuleString object.
+* **l** -- Long long integer.
+* **v** -- Array of RedisModuleString objects.
+* **!** -- This modifier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of arguments parsing.
+
+The function returns a `RedisModuleCallReply` object on success, on
+error NULL is returned.
+
+NULL is returned when the command name is invalid, the format specifier uses
+characters that are not recognized, or when the command is called with the
+wrong number of arguments. In the above cases the `errno` var is set to `EINVAL`. NULL is also returned when, in an instance with Cluster enabled, the target
+keys are about non local hash slots. In this case `errno` is set to `EPERM`.
+
+## Working with RedisModuleCallReply objects.
+
+`RedisModuleCall` returns reply objects that can be accessed using the
+`RedisModule_CallReply*` family of functions.
+
+In order to obtain the type or reply (corresponding to one of the data types
+supported by the Redis protocol), the function `RedisModule_CallReplyType()`
+is used:
+
+    reply = RedisModule_Call(ctx,"INCR","sc",argv[1],"10");
+    if (RedisModule_CallReplyType(reply) == REDISMODULE_REPLY_INTEGER) {
+        long long myval = RedisModule_CallReplyInteger(reply);
+        /* Do something with myval. */
+    }
+
+Valid reply types are:
+
+* `REDISMODULE_REPLY_STRING` Bulk string or status replies.
+* `REDISMODULE_REPLY_ERROR` Errors.
+* `REDISMODULE_REPLY_INTEGER` Signed 64 bit integers.
+* `REDISMODULE_REPLY_ARRAY` Array of replies.
+* `REDISMODULE_REPLY_NULL` NULL reply.
+
+Strings, errors and arrays have an associated length. For strings and errors
+the length corresponds to the length of the string. For arrays the length
+is the number of elements. To obtain the reply length the following function
+is used:
+
+    size_t reply_len = RedisModule_CallReplyLength(reply);
+
+In order to obtain the value of an integer reply, the following function is used, as already shown in the example above:
+
+    long long reply_integer_val = RedisModule_CallReplyInteger(reply);
+
+Called with a reply object of the wrong type, the above function always
+returns `LLONG_MIN`.
+
+Sub elements of array replies are accessed this way:
+
+    RedisModuleCallReply *subreply;
+    subreply = RedisModule_CallReplyArrayElement(reply,idx);
+
+The above function returns NULL if you try to access out of range elements.
+
+Strings and errors (which are like strings but with a different type) can
+be accessed using in the following way, making sure to never write to
+the resulting pointer (that is returned as as `const` pointer so that
+misusing must be pretty explicit):
+
+    size_t len;
+    char *ptr = RedisModule_CallReplyStringPtr(reply,&len);
+
+If the reply type is not a string or an error, NULL is returned.
+
+RedisCallReply objects are not the same as module string objects
+(RedisModuleString types). However sometimes you may need to pass replies
+of type string or integer, to API functions expecting a module string.
+
+When this is the case, you may want to evaluate if using the low level
+API could be a simpler way to implement your command, or you can use
+the following function in order to create a new string object from a
+call reply of type string, error or integer:
+
+    RedisModuleString *mystr = RedisModule_CreateStringFromCallReply(myreply);
+
+If the reply is not of the right type, NULL is returned.
+The returned string object should be released with `RedisModule_FreeString()`
+as usually, or by enabling automatic memory management (see corresponding
+section).
+
+# Releasing call reply objects
+
+Reply objects must be freed using `RedisModule_FreeCallReply`. For arrays,
+you need to free only the top level reply, not the nested replies.
+Currently the module implementation provides a protection in order to avoid
+crashing if you free a nested reply object for error, however this feature
+is not guaranteed to be here forever, so should not be considered part
+of the API.
+
+If you use automatic memory management (explained later in this document)
+you don't need to free replies (but you still could if you wish to release
+memory ASAP).
+
+## Returning values from Redis commands
+
+Like normal Redis commands, new commands implemented via modules must be
+able to return values to the caller. The API exports a set of functions for
+this goal, in order to return the usual types of the Redis protocol, and
+arrays of such types as elemented. Also errors can be returned with any
+error string and code (the error code is the initial uppercase letters in
+the error message, like the "BUSY" string in the "BUSY the sever is busy" error
+message).
+
+All the functions to send a reply to the client are called
+`RedisModule_ReplyWith<something>`.
+
+To return an error, use:
+
+    RedisModule_ReplyWithError(RedisModuleCtx *ctx, const char *err);
+
+There is a predefined error string for key of wrong type errors:
+
+    REDISMODULE_ERRORMSG_WRONGTYPE
+
+Example usage:
+
+    RedisModule_ReplyWithError(ctx,"ERR invalid arguments");
+
+We already saw how to reply with a long long in the examples above:
+
+    RedisModule_ReplyWithLongLong(ctx,12345);
+
+To reply with a simple string, that can't contain binary values or newlines,
+(so it's suitable to send small words, like "OK") we use:
+
+    RedisModule_ReplyWithSimpleString(ctx,"OK");
+
+It's possible to reply with "bulk strings" that are binary safe, using
+two different functions:
+
+    int RedisModule_ReplyWithStringBuffer(RedisModuleCtx *ctx, const char *buf, size_t len);
+
+    int RedisModule_ReplyWithString(RedisModuleCtx *ctx, RedisModuleString *str);
+
+The first function gets a C pointer and length. The second a RedisMoudleString
+object. Use one or the other depending on the source type you have at hand.
+
+In order to reply with an array, you just need to use a function to emit the
+array length, followed by as many calls to the above functions as the number
+of elements of the array are:
+
+    RedisModule_ReplyWithArray(ctx,2);
+    RedisModule_ReplyWithStringBuffer(ctx,"age",3);
+    RedisModule_ReplyWithLongLong(ctx,22);
+
+To return nested arrays is easy, your nested array element just uses another
+call to `RedisModule_ReplyWithArray()` followed by the calls to emit the
+sub array elements.
+
+## Returning arrays with dynamic length
+
+Sometimes it is not possible to know beforehand the number of items of
+an array. As an example, think of a Redis module implementing a FACTOR
+command that given a number outputs the prime factors. Instead of
+factorializing the number, storing the prime factors into an array, and
+later produce the command reply, a better solution is to start an array
+reply where the length is not known, and set it later. This is accomplished
+with a special argument to `RedisModule_ReplyWithArray()`:
+
+    RedisModule_ReplyWithArray(ctx, REDISMODULE_POSTPONED_ARRAY_LEN);
+
+The above call starts an array reply so we can use other `ReplyWith` calls
+in order to produce the array items. Finally in order to set the length
+se use the following call:
+
+    RedisModule_ReplySetArrayLength(ctx, number_of_items);
+
+In the case of the FACTOR command, this translates to some code similar
+to this:
+
+    RedisModule_ReplyWithArray(ctx, REDISMODULE_POSTPONED_ARRAY_LEN);
+    number_of_factors = 0;
+    while(still_factors) {
+        RedisModule_ReplyWithLongLong(ctx, some_factor);
+        number_of_factors++;
+    }
+    RedisModule_ReplySetArrayLength(ctx, number_of_factors);
+
+Another common use case for this feature is iterating over the arrays of
+some collection and only returning the ones passing some kind of filtering.
+
+It is possible to have multiple nested arrays with postponed reply.
+Each call to `SetArray()` will set the length of the latest corresponding
+call to `ReplyWithArray()`:
+
+    RedisModule_ReplyWithArray(ctx, REDISMODULE_POSTPONED_ARRAY_LEN);
+    ... generate 100 elements ...
+    RedisModule_ReplyWithArray(ctx, REDISMODULE_POSTPONED_ARRAY_LEN);
+    ... generate 10 elements ...
+    RedisModule_ReplySetArrayLength(ctx, 10);
+    RedisModule_ReplySetArrayLength(ctx, 100);
+
+This creates a 100 items array having as last element a 10 items array.
+
+# Arity and type checks
+
+Often commands need to check that the number of arguments and type of the key
+is correct. In order to report a wrong arity, there is a specific function
+called `RedisModule_WrongArity()`. The usage is trivial:
+
+    if (argc != 2) return RedisModule_WrongArity(ctx);
+
+Checking for the wrong type involves opening the key and checking the type:
+
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+
+    int keytype = RedisModule_KeyType(key);
+    if (keytype != REDISMODULE_KEYTYPE_STRING &&
+        keytype != REDISMODULE_KEYTYPE_EMPTY)
+    {
+        RedisModule_CloseKey(key);
+        return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
+    }
+
+Note that you often want to proceed with a command both if the key
+is of the expected type, or if it's empty.
+
+## Low level access to keys
+
+Low level access to keys allow to perform operations on value objects associated
+to keys directly, with a speed similar to what Redis uses internally to
+implement the built-in commands.
+
+Once a key is opened, a key pointer is returned that will be used with all the
+other low level API calls in order to perform operations on the key or its
+associated value.
+
+Because the API is meant to be very fast, it cannot do too many run-time
+checks, so the user must be aware of certain rules to follow:
+
+* Opening the same key multiple times where at least one instance is opened for writing, is undefined and may lead to crashes.
+* While a key is open, it should only be accessed via the low level key API. For example opening a key, then calling DEL on the same key using the `RedisModule_Call()` API will result into a crash. However it is safe to open a key, perform some operation with the low level API, closing it, then using other APIs to manage the same key, and later opening it again to do some more work.
+
+In order to open a key the `RedisModule_OpenKey` function is used. It returns
+a key pointer, that we'll use with all the next calls to access and modify
+the value:
+
+    RedisModuleKey *key;
+    key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ);
+
+The second argument is the key name, that must be a `RedisModuleString` object.
+The third argument is the mode: `REDISMODULE_READ` or `REDISMODULE_WRITE`.
+It is possible to use `|` to bitwise OR the two modes to open the key in
+both modes. Currently a key opened for writing can also be accessed for reading
+but this is to be considered an implementation detail. The right mode should
+be used in sane modules.
+
+You can open non exisitng keys for writing, since the keys will be created
+when an attempt to write to the key is performed. However when opening keys
+just for reading, `RedisModule_OpenKey` will return NULL if the key does not
+exist.
+
+Once you are done using a key, you can close it with:
+
+    RedisModule_CloseKey(key);
+
+Note that if automatic memory management is enabled, you are not forced to
+close keys. When the module function returns, Redis will take care to close
+all the keys which are still open.
+
+## Getting the key type
+
+In order to obtain the value of a key, use the `RedisModule_KeyType()` function:
+
+    int keytype = RedisModule_KeyType(key);
+
+It returns one of the following values:
+
+    REDISMODULE_KEYTYPE_EMPTY
+    REDISMODULE_KEYTYPE_STRING
+    REDISMODULE_KEYTYPE_LIST
+    REDISMODULE_KEYTYPE_HASH
+    REDISMODULE_KEYTYPE_SET
+    REDISMODULE_KEYTYPE_ZSET
+
+The above are just the usual Redis key types, with the addition of an empty
+type, that signals the key pointer is associated with an empty key that
+does not yet exists.
+
+## Creating new keys
+
+To create a new key, open it for writing and then write to it using one
+of the key writing functions. Example:
+
+    RedisModuleKey *key;
+    key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ);
+    if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) {
+        RedisModule_StringSet(key,argv[2]);
+    }
+
+## Deleting keys
+
+Just use:
+
+    RedisModule_DeleteKey(key);
+
+The function returns `REDISMODULE_ERR` if the key is not open for writing.
+Note that after a key gets deleted, it is setup in order to be targeted
+by new key commands. For example `RedisModule_KeyType()` will return it is
+an empty key, and writing to it will create a new key, possibly of another
+type (depending on the API used).
+
+## Managing key expires (TTLs)
+
+To control key expires two functions are provided, that are able to set,
+modify, get, and unset the time to live associated with a key.
+
+One function is used in order to query the current expire of an open key:
+
+    mstime_t RedisModule_GetExpire(RedisModuleKey *key);
+
+The function returns the time to live of the key in milliseconds, or
+`REDISMODULE_NO_EXPIRE` as a special value to signal the key has no associated
+expire or does not exist at all (you can differentiate the two cases checking
+if the key type is `REDISMODULE_KEYTYPE_EMPTY`).
+
+In order to change the expire of a key the following function is used instead:
+
+    int RedisModule_SetExpire(RedisModuleKey *key, mstime_t expire);
+
+When called on a non existing key, `REDISMODULE_ERR` is returned, because
+the function can only associate expires to existing open keys (non existing
+open keys are only useful in order to create new values with data type
+specific write operations).
+
+Again the `expire` time is specified in milliseconds. If the key has currently
+no expire, a new expire is set. If the key already have an expire, it is
+replaced with the new value.
+
+If the key has an expire, and the special value `REDISMODULE_NO_EXPIRE` is
+used as a new expire, the expire is removed, similarly to the Redis
+`PERSIST` command. In case the key was already persistent, no operation is
+performed.
+
+## Obtaining the length of values
+
+There is a single function in order to retrieve the length of the value
+associated to an open key. The returned length is value-specific, and is
+the string length for strings, and the number of elements for the aggregated
+data types (how many elements there is in a list, set, sorted set, hash).
+
+    size_t len = RedisModule_ValueLength(key);
+
+If the key does not exist, 0 is returned by the function:
+
+## String type API
+
+Setting a new string value, like the Redis `SET` command does, is performed
+using:
+
+    int RedisModule_StringSet(RedisModuleKey *key, RedisModuleString *str);
+
+The function works exactly like the Redis `SET` command itself, that is, if
+there is a prior value (of any type) it will be deleted.
+
+Accessing existing string values is performed using DMA (direct memory
+access) for speed. The API will return a pointer and a length, so that's
+possible to access and, if needed, modify the string directly.
+
+    size_t len, j;
+    char *myptr = RedisModule_StringDMA(key,&len,REDISMODULE_WRITE);
+    for (j = 0; j < len; j++) myptr[j] = 'A';
+
+In the above example we write directly on the string. Note that if you want
+to write, you must be sure to ask for `WRITE` mode.
+
+DMA pointers are only valid if no other operations are performed with the key
+before using the pointer, after the DMA call.
+
+Sometimes when we want to manipulate strings directly, we need to change
+their size as well. For this scope, the `RedisModule_StringTruncate` function
+is used. Example:
+
+    RedisModule_StringTruncate(mykey,1024);
+
+The function truncates, or enlarges the string as needed, padding it with
+zero bytes if the previos length is smaller than the new length we request.
+If the string does not exist since `key` is associated to an open empty key,
+a string value is created and associated to the key.
+
+Note that every time `StringTruncate()` is called, we need to re-obtain
+the DMA pointer again, since the old may be invalid.
+
+## List type API
+
+It's possible to push and pop values from list values:
+
+    int RedisModule_ListPush(RedisModuleKey *key, int where, RedisModuleString *ele);
+    RedisModuleString *RedisModule_ListPop(RedisModuleKey *key, int where);
+
+In both the APIs the `where` argument specifies if to push or pop from tail
+or head, using the following macros:
+
+    REDISMODULE_LIST_HEAD
+    REDISMODULE_LIST_TAIL
+
+Elements returned by `RedisModule_ListPop()` are like strings craeted with
+`RedisModule_CreateString()`, they must be released with
+`RedisModule_FreeString()` or by enabling automatic memory management.
+
+## Set type API
+
+Work in progress.
+
+## Sorted set type API
+
+Documentation missing, please refer to the top comments inside `module.c`
+for the following functions:
+
+* `RedisModule_ZsetAdd`
+* `RedisModule_ZsetIncrby`
+* `RedisModule_ZsetScore`
+* `RedisModule_ZsetRem`
+
+And for the sorted set iterator:
+
+* `RedisModule_ZsetRangeStop`
+* `RedisModule_ZsetFirstInScoreRange`
+* `RedisModule_ZsetLastInScoreRange`
+* `RedisModule_ZsetFirstInLexRange`
+* `RedisModule_ZsetLastInLexRange`
+* `RedisModule_ZsetRangeCurrentElement`
+* `RedisModule_ZsetRangeNext`
+* `RedisModule_ZsetRangePrev`
+* `RedisModule_ZsetRangeEndReached`
+
+## Hash type API
+
+Documentation missing, please refer to the top comments inside `module.c`
+for the following functions:
+
+* `RedisModule_HashSet`
+* `RedisModule_HashGet`
+
+## Iterating aggregated values
+
+Work in progress.
+
+# Replicating commands
+
+If you want to use module commands exactly like normal Redis commands, in the
+context of replicated Redis instances, or using the AOF file for persistence,
+it is important for module commands to handle their replication in a consistent
+way.
+
+When using the higher level APIs to invoke commands, replication happens
+automatically if you use the "!" modifier in the format string of
+`RedisModule_Call()` as in the following example:
+
+    reply = RedisModule_Call(ctx,"INCR","!sc",argv[1],"10");
+
+As you can see the format specifier is `"!sc"`. The bang is not parsed as a
+format specifier, but it internally flags the command as "must replicate".
+
+If you use the above programming style, there are no problems.
+However sometimes things are more complex than that, and you use the low level
+API. In this case, if there are no side effects in the command execution, and
+it consistently always performs the same work, what is possible to do is to
+replicate the command verbatim as the user executed it. To do that, you just
+need to call the following function:
+
+    RedisModule_ReplicateVerbatim(ctx);
+
+When you use the above API, you should not use any other replication function
+since they are not guaranteed to mix well.
+
+However this is not the only option. It's also possible to exactly tell
+Redis what commands to replicate as the effect of the command execution, using
+an API similar to `RedisModule_Call()` but that instead of calling the command
+sends it to the AOF / slaves stream. Example:
+
+    RedisModule_Replicate(ctx,"INCRBY","cl","foo",my_increment);
+
+It's possible to call `RedisModule_Replicate` multiple times, and each
+will emit a command. All the sequence emitted is wrapped between a
+`MULTI/EXEC` transaction, so that the AOF and replication effects are the
+same as executing a single command.
+
+Note that `Call()` replication and `Replicate()` replication have a rule,
+in case you want to mix both forms of replication (not necessarily a good
+idea if there are simpler approaches). Commands replicated with `Call()`
+are always the first emitted in the final `MULTI/EXEC` block, while all
+the commands emitted with `Replicate()` will follow.
+
+# Automatic memory management
+
+Normally when writing programs in the C language, programmers need to manage
+memory manually. This is why the Redis modules API has functions to release
+strings, close open keys, free replies, and so forth.
+
+However given that commands are executed in a contained environment and
+with a set of strict APIs, Redis is able to provide automatic memory management
+to modules, at the cost of some performance (most of the time, a very low
+cost).
+
+When automatic memory management is enabled:
+
+1. You don't need to close open keys.
+2. You don't need to free replies.
+3. You don't need to free RedisModuleString objects.
+
+However you can still do it, if you want. For example, automatic memory
+management may be active, but inside a loop allocating a lot of strings,
+you may still want to free strings no longer used.
+
+In order to enable automatic memory management, just call the following
+function at the start of the command implementation:
+
+    RedisModule_AutoMemory(ctx);
+
+Automatic memory management is usually the way to go, however experienced
+C programmers may not use it in order to gain some speed and memory usage
+benefit.
+
+# Allocating memory into modules
+
+Normal C programs use `malloc()` and `free()` in order to allocate and
+release memory dynamically. While in Redis modules the use of malloc is
+not technically forbidden, it is a lot better to use the Redis Modules
+specific functions, that are exact replacements for `malloc`, `free`,
+`realloc` and `strdup`. These functions are:
+
+    void *RedisModule_Alloc(size_t bytes);
+    void* RedisModule_Realloc(void *ptr, size_t bytes);
+    void RedisModule_Free(void *ptr);
+    void RedisModule_Calloc(size_t nmemb, size_t size);
+    char *RedisModule_Strdup(const char *str);
+
+They work exactly like their `libc` equivalent calls, however they use
+the same allocator Redis uses, and the memory allocated using these
+functions is reported by the `INFO` command in the memory section, is
+accounted when enforcing the `maxmemory` policy, and in general is
+a first citizen of the Redis executable. On the contrar, the method
+allocated inside modules with libc `malloc()` is transparent to Redis.
+
+Another reason to use the modules functions in order to allocate memory
+is that, when creating native data types inside modules, the RDB loading
+functions can return deserialized strings (from the RDB file) directly
+as `RedisModule_Alloc()` allocations, so they can be used directly to
+populate data structures after loading, instead of having to copy them
+to the data structure.
+
+## Pool allocator
+
+Sometimes in commands implementations, it is required to perform many
+small allocations that will be not retained at the end of the command
+execution, but are just functional to execute the command itself.
+
+This work can be more easily accomplished using the Redis pool allocator:
+
+    void *RedisModule_PoolAlloc(RedisModuleCtx *ctx, size_t bytes);
+
+It works similarly to `malloc()`, and returns memory aligned to the
+next power of two of greater or equal to `bytes` (for a maximum alignment
+of 8 bytes). However it allocates memory in blocks, so it the overhead
+of the allocations is small, and more important, the memory allocated
+is automatically released when the command returns.
+
+So in general short living allocations are a good candidates for the pool
+allocator.
+
+# Writing commands compatible with Redis Cluster
+
+Documentation missing, please check the following functions inside `module.c`:
+
+    RedisModule_IsKeysPositionRequest(ctx);
+    RedisModule_KeyAtPos(ctx,pos);

src/modules/Makefile

@@ -0,0 +1,37 @@
+
+# find the OS
+uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo not')
+
+# Compile flags for linux / osx
+ifeq ($(uname_S),Linux)
+	SHOBJ_CFLAGS ?= -fno-common -g -ggdb -std=c99 -O2
+	SHOBJ_LDFLAGS ?= -shared
+else
+	SHOBJ_CFLAGS ?= -dynamic -fno-common -g -ggdb -std=c99 -O2
+	SHOBJ_LDFLAGS ?= -bundle -undefined dynamic_lookup
+endif
+
+.SUFFIXES: .c .so .xo .o
+
+all: helloworld.so hellotype.so testmodule.so
+
+.c.xo:
+	$(CC) -I. $(CFLAGS) $(SHOBJ_CFLAGS) -fPIC -c $< -o $@
+
+helloworld.xo: ../redismodule.h
+
+helloworld.so: helloworld.xo
+	$(LD) -o $@ $< $(SHOBJ_LDFLAGS) $(LIBS) -lc
+
+hellotype.xo: ../redismodule.h
+
+hellotype.so: hellotype.xo
+	$(LD) -o $@ $< $(SHOBJ_LDFLAGS) $(LIBS) -lc
+
+testmodule.xo: ../redismodule.h
+
+testmodule.so: testmodule.xo
+	$(LD) -o $@ $< $(SHOBJ_LDFLAGS) $(LIBS) -lc
+
+clean:
+	rm -rf *.xo *.so

src/modules/TYPES.md

@@ -0,0 +1,371 @@
+Native types in Redis modules
+===
+
+Redis modules can access Redis built-in data structures both at high level,
+by calling Redis commands, and at low level, by manipulating the data structures
+directly.
+
+By using these capabilities in order to build new abstractions on top of existing
+Redis data structures, or by using strings DMA in order to encode modules
+data structures into Redis strings, it is possible to create modules that
+*feel like* they are exporting new data types. However, for more complex
+problems, this is not enough, and the implementation of new data structures
+inside the module is needed.
+
+We call the ability of Redis modules to implement new data structures that
+feel like native Redis ones **native types support**. This document describes
+the API exported by the Redis modules system in order to create new data
+structures and handle the serialization in RDB files, the rewriting process
+in AOF, the type reporting via the `TYPE` command, and so forth.
+
+Overview of native types
+---
+
+A module exporting a native type is composed of the following main parts:
+
+* The implementation of some kind of new data structure and of commands operating on the new data structure.
+* A set of callbacks that handle: RDB saving, RDB loading, AOF rewriting, releasing of a value associated with a key, calculation of a value digest (hash) to be used with the `DEBUG DIGEST` command.
+* A 9 characters name that is unique to each module native data type.
+* An encoding version, used to persist into RDB files a module-specific data version, so that a module will be able to load older representations from RDB files.
+
+While to handle RDB loading, saving and AOF rewriting may look complex as a first glance, the modules API provide very high level function for handling all this, without requiring the user to handle read/write errors, so in practical terms, writing a new data structure for Redis is a simple task.
+
+A **very easy** to understand but complete example of native type implementation
+is available inside the Redis distribution in the `/modules/hellotype.c` file.
+The reader is encouraged to read the documentation by looking at this example
+implementation to see how things are applied in the practice.
+
+Registering a new data type
+===
+
+In order to register a new native type into the Redis core, the module needs
+to declare a global variable that will hold a reference to the data type.
+The API to register the data type will return a data type reference that will
+be stored in the global variable.
+
+    static RedisModuleType *MyType;
+    #define MYTYPE_ENCODING_VERSION 0
+
+    int RedisModule_OnLoad(RedisModuleCtx *ctx) {
+        MyType = RedisModule_CreateDataType("MyType-AZ", MYTYPE_ENCODING_VERSION,
+            MyTypeRDBLoad, MyTypeRDBSave, MyTypeAOFRewrite, MyTypeDigest,
+            MyTypeFree);
+        if (MyType == NULL) return REDISMODULE_ERR;
+    }
+
+As you can see from the example above, a single API call is needed in order to
+register the new type. However a number of function pointers are passed as
+arguments. The prototype of `RedisModule_CreateDataType` is the following:
+
+    moduleType *RedisModule_CreateDataType(RedisModuleCtx *ctx,
+               const char *name, int encver,
+               moduleTypeLoadFunc rdb_load,
+               moduleTypeSaveFunc rdb_save,
+               moduleTypeRewriteFunc aof_rewrite,
+               moduleTypeDigestFunc digest,
+               moduleTypeFreeFunc free);
+
+The `ctx` argument is the context that we receive in the `OnLoad` function.
+The type `name` is a 9 character name in the character set that includes
+from `A-Z`, `a-z`, `0-9`, plus the underscore `_` and minus `-` characters.
+
+Note that **this name must be unique** for each data type in the Redis
+ecosystem, so be creative, use both lower-case and upper case if it makes
+sense, and try to use the convention of mixing the type name with the name
+of the author of the module, to create a 9 character unique name.
+
+For example if I'm building a *b-tree* data structure and my name is *antirez*
+I'll call my type **btree1-az**. The name, converted to a 64 bit integer,
+is stored inside the RDB file when saving the type, and will be used when the
+RDB data is loaded in order to resolve what module can load the data. If Redis
+finds no matching module, the integer is converted back to a name in order to
+provide some clue to the user about what module is missing in order to load
+the data.
+
+The type name is also used as a reply for the `TYPE` command when called
+with a key holding the registered type.
+
+The `encver` argument is the encoding version used by the module to store data
+inside the RDB file. For example I can start with an encoding version of 0,
+but later when I release version 2.0 of my module, I can switch encoding to
+something better. The new module will register with an encoding version of 1,
+so when it saves new RDB files, the new version will be stored on disk. However
+when loading RDB files, the module `rdb_load` method will be called even if
+there is data found for a different encoding version (and the encoding version
+is passed as argument to `rdb_load`), so that the module can still load old
+RDB files.
+
+The remaining arguments `rdb_load`, `rdb_save`, `aof_rewrite`, `digest` and
+`free` are all callbacks with the following prototypes and uses:
+
+    typedef void *(*RedisModuleTypeLoadFunc)(RedisModuleIO *rdb, int encver);
+    typedef void (*RedisModuleTypeSaveFunc)(RedisModuleIO *rdb, void *value);
+    typedef void (*RedisModuleTypeRewriteFunc)(RedisModuleIO *aof, RedisModuleString *key, void *value);
+    typedef void (*RedisModuleTypeDigestFunc)(RedisModuleDigest *digest, void *value);
+    typedef void (*RedisModuleTypeFreeFunc)(void *value);
+
+* `rdb_load` is called when loading data from the RDB file. It loads data in the same format as `rdb_save` produces.
+* `rdb_save` is called when saving data to the RDB file.
+* `aof_rewrite` is called when the AOF is being rewritten, and the module needs to tell Redis what is the sequence of commands to recreate the content of a given key.
+* `digest` is called when `DEBUG DIGEST` is executed and a key holding this module type is found. Currently this is not yet implemented so the function ca be left empty.
+* `free` is called when a key with the module native type is deleted via `DEL` or in any other mean, in order to let the module reclaim the memory associated with such a value.
+
+Ok, but *why* modules types require a 9 characters name?
+---
+
+Oh, I understand you need to understand this, so here is a very specific
+explanation.
+
+When Redis persists to RDB files, modules specific data types require to
+be persisted as well. Now RDB files are sequences of key-value pairs
+like the following:
+
+    [1 byte type] [key] [a type specific value]
+
+The 1 byte type identifies strings, lists, sets, and so forth. In the case
+of modules data, it is set to a special value of `module data`, but of
+course this is not enough, we need the information needed to link a specific
+value with a specific module type that is able to load and handle it.
+
+So when we save a `type specific value` about a module, we prefix it with
+a 64 bit integer. 64 bits is large enough to store the informations needed
+in order to lookup the module that can handle that specific type, but is
+short enough that we can prefix each module value we store inside the RDB
+without making the final RDB file too big. At the same time, this solution
+of prefixing the value with a 64 bit *signature* does not require to do
+strange things like defining in the RDB header a list of modules specific
+types. Everything is pretty simple.
+
+So, what you can store in 64 bits in order to identify a given module in
+a reliable way? Well if you build a character set of 64 symbols, you can
+easily store 9 characters of 6 bits, and you are left with 10 bits, that
+are used in order to store the *encoding version* of the type, so that
+the same type can evolve in the future and provide a different and more
+efficient or updated serialization format for RDB files.
+
+So the 64 bit prefix stored before each module value is like the following:
+
+    6|6|6|6|6|6|6|6|6|10
+
+The first 9 elements are 6-bits characters, the final 10 bits is the
+encoding version.
+
+When the RDB file is loaded back, it reads the 64 bit value, masks the final
+10 bits, and searches for a matching module in the modules types cache.
+When a matching one is found, the method to load the RDB file value is called
+with the 10 bits encoding version as argument, so that the module knows
+what version of the data layout to load, if it can support multiple versions.
+
+Now the interesting thing about all this is that, if instead the module type
+cannot be resolved, since there is no loaded module having this signature,
+we can convert back the 64 bit value into a 9 characters name, and print
+an error to the user that includes the module type name! So that she or he
+immediately realizes what's wrong.
+
+Setting and getting keys
+---
+
+After registering our new data type in the `RedisModule_OnLoad()` function,
+we also need to be able to set Redis keys having as value our native type.
+
+This normally happens in the context of commands that write data to a key.
+The native types API allow to set and get keys to module native data types,
+and to test if a given key is already associated to a value of a specific data
+type.
+
+The API uses the normal modules `RedisModule_OpenKey()` low level key access
+interface in order to deal with this. This is an eaxmple of setting a
+native type private data structure to a Redis key:
+
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,keyname,REDISMODULE_WRITE);
+    struct some_private_struct *data = createMyDataStructure();
+    RedisModule_ModuleTypeSetValue(key,MyType,data);
+
+The function `RedisModule_ModuleTypeSetValue()` is used with a key handle open
+for writing, and gets three arguments: the key handle, the reference to the
+native type, as obtained during the type registration, and finally a `void*`
+pointer that contains the private data implementing the module native type.
+
+Note that Redis has no clues at all about what your data contains. It will
+just call the callbacks you provided during the method registration in order
+to perform operations on the type.
+
+Similarly we can retrieve the private data from a key using this function:
+
+    struct some_private_struct *data;
+    data = RedisModule_ModuleTypeGetValue(key);
+
+We can also test for a key to have our native type as value:
+
+    if (RedisModule_ModuleTypeGetType(key) == MyType) {
+        /* ... do something ... */
+    }
+
+However for the calls to do the right thing, we need to check if the key
+is empty, if it contains a value of the right kind, and so forth. So
+the idiomatic code to implement a command writing to our native type
+is along these lines:
+
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+    int type = RedisModule_KeyType(key);
+    if (type != REDISMODULE_KEYTYPE_EMPTY &&
+        RedisModule_ModuleTypeGetType(key) != MyType)
+    {
+        return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
+    }
+
+Then if we successfully verified the key is not of the wrong type, and
+we are going to write to it, we usually want to create a new data structure if
+the key is empty, or retrieve the reference to the value associated to the
+key if there is already one:
+
+    /* Create an empty value object if the key is currently empty. */
+    struct some_private_struct *data;
+    if (type == REDISMODULE_KEYTYPE_EMPTY) {
+        data = createMyDataStructure();
+        RedisModule_ModuleTypeSetValue(key,MyTyke,data);
+    } else {
+        data = RedisModule_ModuleTypeGetValue(key);
+    }
+    /* Do something with 'data'... */
+
+Free method
+---
+
+As already mentioned, when Redis needs to free a key holding a native type
+value, it needs help from the module in order to release the memory. This
+is the reason why we pass a `free` callback during the type registration:
+
+    typedef void (*RedisModuleTypeFreeFunc)(void *value);
+
+A trivial implementation of the free method can be something like this,
+assuming our data structure is composed of a single allocation:
+
+    void MyTypeFreeCallback(void *value) {
+        RedisModule_Free(value);
+    }
+
+However a more real world one will call some function that performs a more
+complex memory reclaiming, by casting the void pointer to some structure
+and freeing all the resources composing the value.
+
+RDB load and save methods
+---
+
+The RDB saving and loading callbacks need to create (and load back) a
+representation of the data type on disk. Redis offers an high level API
+that can automatically store inside the RDB file the following types:
+
+* Unsigned 64 bit integers.
+* Signed 64 bit integers.
+* Doubles.
+* Strings.
+
+It is up to the module to find a viable representation using the above base
+types. However note that while the integer and double values are stored
+and loaded in an architecture and *endianess* agnostic way, if you use
+the raw string saving API to, for example, save a structure on disk, you
+have to care those details yourself.
+
+This is the list of functions performing RDB saving and loading:
+
+    void RedisModule_SaveUnsigned(RedisModuleIO *io, uint64_t value);
+    uint64_t RedisModule_LoadUnsigned(RedisModuleIO *io);
+    void RedisModule_SaveSigned(RedisModuleIO *io, int64_t value);
+    int64_t RedisModule_LoadSigned(RedisModuleIO *io);
+    void RedisModule_SaveString(RedisModuleIO *io, RedisModuleString *s);
+    void RedisModule_SaveStringBuffer(RedisModuleIO *io, const char *str, size_t len);
+    RedisModuleString *RedisModule_LoadString(RedisModuleIO *io);
+    char *RedisModule_LoadStringBuffer(RedisModuleIO *io, size_t *lenptr);
+    void RedisModule_SaveDouble(RedisModuleIO *io, double value);
+    double RedisModule_LoadDouble(RedisModuleIO *io);
+
+The functions don't require any error checking from the module, that can
+always assume calls succeed.
+
+As an example, imagine I've a native type that implements an array of
+double values, with the following structure:
+
+    struct double_array {
+        size_t count;
+        double *values;
+    };
+
+My `rdb_save` method may look like the following:
+
+    void DoubleArrayRDBSave(RedisModuleIO *io, void *ptr) {
+        struct dobule_array *da = ptr;
+        RedisModule_SaveUnsigned(io,da->count);
+        for (size_t j = 0; j < da->count; j++)
+            RedisModule_SaveDouble(io,da->values[j]);
+    }
+
+What we did was to store the number of elements followed by each double
+value. So when later we'll have to load the structure in the `rdb_load`
+method we'll do something like this:
+
+    void *DoubleArrayRDBLoad(RedisModuleIO *io, int encver) {
+        if (encver != DOUBLE_ARRAY_ENC_VER) {
+            /* We should actually log an error here, or try to implement
+               the ability to load older versions of our data structure. */
+            return NULL;
+        }
+
+        struct double_array *da;
+        da = RedisModule_Alloc(sizeof(*da));
+        da->count = RedisModule_LoadUnsigned(io);
+        da->values = RedisModule_Alloc(da->count * sizeof(double));
+        for (size_t j = 0; j < da->count; j++)
+            da->values = RedisModule_LoadDouble(io);
+        return da;
+    }
+
+The load callback just reconstruct back the data structure from the data
+we stored in the RDB file.
+
+Note that while there is no error handling on the API that writes and reads
+from disk, still the load callback can return NULL on errors in case what
+it reads does not look correct. Redis will just panic in that case.
+
+AOF rewriting
+---
+
+    void RedisModule_EmitAOF(RedisModuleIO *io, const char *cmdname, const char *fmt, ...);
+
+Handling multiple encodings
+---
+
+    WORK IN PROGRESS
+
+Allocating memory
+---
+
+Modules data types should try to use `RedisModule_Alloc()` functions family
+in order to allocate, reallocate and release heap memory used to implement the native data structures (see the other Redis Modules documentation for detailed information).
+
+This is not just useful in order for Redis to be able to account for the memory used by the module, but there are also more advantages:
+
+* Redis uses the `jemalloc` allcator, that often prevents fragmentation problems that could be caused by using the libc allocator.
+* When loading strings from the RDB file, the native types API is able to return strings allocated directly with `RedisModule_Alloc()`, so that the module can directly link this memory into the data structure representation, avoiding an useless copy of the data.
+
+Even if you are using external libraries implementing your data structures, the
+allocation functions provided by the module API is exactly compatible with
+`malloc()`, `realloc()`, `free()` and `strdup()`, so converting the libraries
+in order to use these functions should be trivial.
+
+In case you have an external library that uses libc `malloc()`, and you want
+to avoid replacing manually all the calls with the Redis Modules API calls,
+an approach could be to use simple macros in order to replace the libc calls
+with the Redis API calls. Something like this could work:
+
+    #define malloc RedisModule_Alloc
+    #define realloc RedisModule_Realloc
+    #define free RedisModule_Free
+    #define strdup RedisModule_Strdup
+
+However take in mind that mixing libc calls with Redis API calls will result
+into troubles and crashes, so if you replace calls using macros, you need to
+make sure that all the calls are correctly replaced, and that the code with
+the substituted calls will never, for example, attempt to call
+`RedisModule_Free()` with a pointer allocated using libc `malloc()`.

src/modules/gendoc.rb

@@ -0,0 +1,43 @@
+# gendoc.rb -- Converts the top-comments inside module.c to modules API
+#              reference documentaiton in markdown format.
+
+# Convert the C comment to markdown
+def markdown(s)
+    s = s.gsub(/\*\/$/,"")
+    s = s.gsub(/^ \* {0,1}/,"")
+    s = s.gsub(/^\/\* /,"")
+    if s[0] != ' '
+        s = s.gsub(/RM_[A-z()]+/){|x| "`#{x}`"}
+        s = s.gsub(/RedisModule_[A-z()]+/){|x| "`#{x}`"}
+        s = s.gsub(/REDISMODULE_[A-z]+/){|x| "`#{x}`"}
+    end
+    s.chop! while s[-1] == "\n" || s[-1] == " "
+    return s
+end
+
+# Given the source code array and the index at which an exported symbol was
+# detected, extracts and outputs the documentation.
+def docufy(src,i)
+    m = /RM_[A-z0-9]+/.match(src[i])
+    proto = src[i].sub("{","").strip+";\n"
+    puts "## `#{m[0]}`\n\n"
+    puts "    #{proto}\n"
+    comment = ""
+    while true
+        i = i-1
+        comment = src[i]+comment
+        break if src[i] =~ /\/\*/
+    end
+    comment = markdown(comment)
+    puts comment+"\n\n"
+end
+
+puts "# Modules API reference\n\n"
+src = File.open("../module.c").to_a
+src.each_with_index{|line,i|
+    if line =~ /RM_/ && line[0] != ' ' && line[0] != '#' && line[0] != '/'
+        if src[i-1] =~ /\*\//
+            docufy(src,i)
+        end
+    end
+}

src/modules/hellotype.c

@@ -0,0 +1,259 @@
+/* This file implements a new module native data type called "HELLOTYPE".
+ * The data structure implemented is a very simple ordered linked list of
+ * 64 bit integers, in order to have something that is real world enough, but
+ * at the same time, extremely simple to understand, to show how the API
+ * works, how a new data type is created, and how to write basic methods
+ * for RDB loading, saving and AOF rewriting.
+ *
+ * -----------------------------------------------------------------------------
+ *
+ * Copyright (c) 2016, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "../redismodule.h"
+#include <stdio.h>
+#include <stdlib.h>
+#include <ctype.h>
+#include <string.h>
+#include <stdint.h>
+
+static RedisModuleType *HelloType;
+
+/* ========================== Internal data structure  =======================
+ * This is just a linked list of 64 bit integers where elements are inserted
+ * in-place, so it's ordered. There is no pop/push operation but just insert
+ * because it is enough to show the implementation of new data types without
+ * making things complex. */
+
+struct HelloTypeNode {
+    int64_t value;
+    struct HelloTypeNode *next;
+};
+
+struct HelloTypeObject {
+    struct HelloTypeNode *head;
+    size_t len; /* Number of elements added. */
+};
+
+struct HelloTypeObject *createHelloTypeObject(void) {
+    struct HelloTypeObject *o;
+    o = RedisModule_Alloc(sizeof(*o));
+    o->head = NULL;
+    o->len = 0;
+    return o;
+}
+
+void HelloTypeInsert(struct HelloTypeObject *o, int64_t ele) {
+    struct HelloTypeNode *next = o->head, *newnode, *prev = NULL;
+
+    while(next && next->value < ele) {
+        prev = next;
+        next = next->next;
+    }
+    newnode = RedisModule_Alloc(sizeof(*newnode));
+    newnode->value = ele;
+    newnode->next = next;
+    if (prev) {
+        prev->next = newnode;
+    } else {
+        o->head = newnode;
+    }
+    o->len++;
+}
+
+void HelloTypeReleaseObject(struct HelloTypeObject *o) {
+    struct HelloTypeNode *cur, *next;
+    cur = o->head;
+    while(cur) {
+        next = cur->next;
+        RedisModule_Free(cur);
+        cur = next;
+    }
+    RedisModule_Free(o);
+}
+
+/* ========================= "hellotype" type commands ======================= */
+
+/* HELLOTYPE.INSERT key value */
+int HelloTypeInsert_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    RedisModule_AutoMemory(ctx); /* Use automatic memory management. */
+
+    if (argc != 3) return RedisModule_WrongArity(ctx);
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+    int type = RedisModule_KeyType(key);
+    if (type != REDISMODULE_KEYTYPE_EMPTY &&
+        RedisModule_ModuleTypeGetType(key) != HelloType)
+    {
+        return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
+    }
+
+    long long value;
+    if ((RedisModule_StringToLongLong(argv[2],&value) != REDISMODULE_OK)) {
+        return RedisModule_ReplyWithError(ctx,"ERR invalid value: must be a signed 64 bit integer");
+    }
+
+    /* Create an empty value object if the key is currently empty. */
+    struct HelloTypeObject *hto;
+    if (type == REDISMODULE_KEYTYPE_EMPTY) {
+        hto = createHelloTypeObject();
+        RedisModule_ModuleTypeSetValue(key,HelloType,hto);
+    } else {
+        hto = RedisModule_ModuleTypeGetValue(key);
+    }
+
+    /* Insert the new element. */
+    HelloTypeInsert(hto,value);
+
+    RedisModule_ReplyWithLongLong(ctx,hto->len);
+    RedisModule_ReplicateVerbatim(ctx);
+    return REDISMODULE_OK;
+}
+
+/* HELLOTYPE.RANGE key first count */
+int HelloTypeRange_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    RedisModule_AutoMemory(ctx); /* Use automatic memory management. */
+
+    if (argc != 4) return RedisModule_WrongArity(ctx);
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+    int type = RedisModule_KeyType(key);
+    if (type != REDISMODULE_KEYTYPE_EMPTY &&
+        RedisModule_ModuleTypeGetType(key) != HelloType)
+    {
+        return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
+    }
+
+    long long first, count;
+    if (RedisModule_StringToLongLong(argv[2],&first) != REDISMODULE_OK ||
+        RedisModule_StringToLongLong(argv[3],&count) != REDISMODULE_OK ||
+        first < 0 || count < 0)
+    {
+        return RedisModule_ReplyWithError(ctx,
+            "ERR invalid first or count parameters");
+    }
+
+    struct HelloTypeObject *hto = RedisModule_ModuleTypeGetValue(key);
+    struct HelloTypeNode *node = hto ? hto->head : NULL;
+    RedisModule_ReplyWithArray(ctx,REDISMODULE_POSTPONED_ARRAY_LEN);
+    long long arraylen = 0;
+    while(node && count--) {
+        RedisModule_ReplyWithLongLong(ctx,node->value);
+        arraylen++;
+        node = node->next;
+    }
+    RedisModule_ReplySetArrayLength(ctx,arraylen);
+    return REDISMODULE_OK;
+}
+
+/* HELLOTYPE.LEN key */
+int HelloTypeLen_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    RedisModule_AutoMemory(ctx); /* Use automatic memory management. */
+
+    if (argc != 2) return RedisModule_WrongArity(ctx);
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+    int type = RedisModule_KeyType(key);
+    if (type != REDISMODULE_KEYTYPE_EMPTY &&
+        RedisModule_ModuleTypeGetType(key) != HelloType)
+    {
+        return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
+    }
+
+    struct HelloTypeObject *hto = RedisModule_ModuleTypeGetValue(key);
+    RedisModule_ReplyWithLongLong(ctx,hto ? hto->len : 0);
+    return REDISMODULE_OK;
+}
+
+
+/* ========================== "hellotype" type methods ======================= */
+
+void *HelloTypeRdbLoad(RedisModuleIO *rdb, int encver) {
+    if (encver != 0) {
+        /* RedisModule_Log("warning","Can't load data with version %d", encver);*/
+        return NULL;
+    }
+    uint64_t elements = RedisModule_LoadUnsigned(rdb);
+    struct HelloTypeObject *hto = createHelloTypeObject();
+    while(elements--) {
+        int64_t ele = RedisModule_LoadSigned(rdb);
+        HelloTypeInsert(hto,ele);
+    }
+    return hto;
+}
+
+void HelloTypeRdbSave(RedisModuleIO *rdb, void *value) {
+    struct HelloTypeObject *hto = value;
+    struct HelloTypeNode *node = hto->head;
+    RedisModule_SaveUnsigned(rdb,hto->len);
+    while(node) {
+        RedisModule_SaveSigned(rdb,node->value);
+        node = node->next;
+    }
+}
+
+void HelloTypeAofRewrite(RedisModuleIO *aof, RedisModuleString *key, void *value) {
+    struct HelloTypeObject *hto = value;
+    struct HelloTypeNode *node = hto->head;
+    while(node) {
+        RedisModule_EmitAOF(aof,"HELLOTYPE.INSERT","sl",key,node->value);
+        node = node->next;
+    }
+}
+
+void HelloTypeDigest(RedisModuleDigest *digest, void *value) {
+    /* TODO: The DIGEST module interface is yet not implemented. */
+}
+
+void HelloTypeFree(void *value) {
+    HelloTypeReleaseObject(value);
+}
+
+/* This function must be present on each Redis module. It is used in order to
+ * register the commands into the Redis server. */
+int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    if (RedisModule_Init(ctx,"hellotype",1,REDISMODULE_APIVER_1)
+        == REDISMODULE_ERR) return REDISMODULE_ERR;
+
+    HelloType = RedisModule_CreateDataType(ctx,"hellotype",0,HelloTypeRdbLoad,HelloTypeRdbSave,HelloTypeAofRewrite,HelloTypeDigest,HelloTypeFree);
+    if (HelloType == NULL) return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hellotype.insert",
+        HelloTypeInsert_RedisCommand,"write deny-oom",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hellotype.range",
+        HelloTypeRange_RedisCommand,"readonly",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hellotype.len",
+        HelloTypeLen_RedisCommand,"readonly",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    return REDISMODULE_OK;
+}

src/modules/helloworld.c

@@ -0,0 +1,618 @@
+/* Helloworld module -- A few examples of the Redis Modules API in the form
+ * of commands showing how to accomplish common tasks.
+ *
+ * This module does not do anything useful, if not for a few commands. The
+ * examples are designed in order to show the API.
+ *
+ * -----------------------------------------------------------------------------
+ *
+ * Copyright (c) 2016, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "../redismodule.h"
+#include <stdio.h>
+#include <stdlib.h>
+#include <ctype.h>
+#include <string.h>
+
+/* HELLO.SIMPLE is among the simplest commands you can implement.
+ * It just returns the currently selected DB id, a functionality which is
+ * missing in Redis. The command uses two important API calls: one to
+ * fetch the currently selected DB, the other in order to send the client
+ * an integer reply as response. */
+int HelloSimple_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    RedisModule_ReplyWithLongLong(ctx,RedisModule_GetSelectedDb(ctx));
+    return REDISMODULE_OK;
+}
+
+/* HELLO.PUSH.NATIVE re-implements RPUSH, and shows the low level modules API
+ * where you can "open" keys, make low level operations, create new keys by
+ * pushing elements into non-existing keys, and so forth.
+ *
+ * You'll find this command to be roughly as fast as the actual RPUSH
+ * command. */
+int HelloPushNative_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
+{
+    if (argc != 3) return RedisModule_WrongArity(ctx);
+
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+
+    RedisModule_ListPush(key,REDISMODULE_LIST_TAIL,argv[2]);
+    size_t newlen = RedisModule_ValueLength(key);
+    RedisModule_CloseKey(key);
+    RedisModule_ReplyWithLongLong(ctx,newlen);
+    return REDISMODULE_OK;
+}
+
+/* HELLO.PUSH.CALL implements RPUSH using an higher level approach, calling
+ * a Redis command instead of working with the key in a low level way. This
+ * approach is useful when you need to call Redis commands that are not
+ * available as low level APIs, or when you don't need the maximum speed
+ * possible but instead prefer implementation simplicity. */
+int HelloPushCall_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
+{
+    if (argc != 3) return RedisModule_WrongArity(ctx);
+
+    RedisModuleCallReply *reply;
+
+    reply = RedisModule_Call(ctx,"RPUSH","ss",argv[1],argv[2]);
+    long long len = RedisModule_CallReplyInteger(reply);
+    RedisModule_FreeCallReply(reply);
+    RedisModule_ReplyWithLongLong(ctx,len);
+    return REDISMODULE_OK;
+}
+
+/* HELLO.PUSH.CALL2
+ * This is exaxctly as HELLO.PUSH.CALL, but shows how we can reply to the
+ * client using directly a reply object that Call() returned. */
+int HelloPushCall2_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
+{
+    if (argc != 3) return RedisModule_WrongArity(ctx);
+
+    RedisModuleCallReply *reply;
+
+    reply = RedisModule_Call(ctx,"RPUSH","ss",argv[1],argv[2]);
+    RedisModule_ReplyWithCallReply(ctx,reply);
+    RedisModule_FreeCallReply(reply);
+    return REDISMODULE_OK;
+}
+
+/* HELLO.LIST.SUM.LEN returns the total length of all the items inside
+ * a Redis list, by using the high level Call() API.
+ * This command is an example of the array reply access. */
+int HelloListSumLen_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
+{
+    if (argc != 2) return RedisModule_WrongArity(ctx);
+
+    RedisModuleCallReply *reply;
+
+    reply = RedisModule_Call(ctx,"LRANGE","sll",argv[1],(long long)0,(long long)-1);
+    size_t strlen = 0;
+    size_t items = RedisModule_CallReplyLength(reply);
+    size_t j;
+    for (j = 0; j < items; j++) {
+        RedisModuleCallReply *ele = RedisModule_CallReplyArrayElement(reply,j);
+        strlen += RedisModule_CallReplyLength(ele);
+    }
+    RedisModule_FreeCallReply(reply);
+    RedisModule_ReplyWithLongLong(ctx,strlen);
+    return REDISMODULE_OK;
+}
+
+/* HELLO.LIST.SPLICE srclist dstlist count
+ * Moves 'count' elements from the tail of 'srclist' to the head of
+ * 'dstlist'. If less than count elements are available, it moves as much
+ * elements as possible. */
+int HelloListSplice_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    if (argc != 4) return RedisModule_WrongArity(ctx);
+
+    RedisModuleKey *srckey = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+    RedisModuleKey *dstkey = RedisModule_OpenKey(ctx,argv[2],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+
+    /* Src and dst key must be empty or lists. */
+    if ((RedisModule_KeyType(srckey) != REDISMODULE_KEYTYPE_LIST &&
+         RedisModule_KeyType(srckey) != REDISMODULE_KEYTYPE_EMPTY) ||
+        (RedisModule_KeyType(dstkey) != REDISMODULE_KEYTYPE_LIST &&
+         RedisModule_KeyType(dstkey) != REDISMODULE_KEYTYPE_EMPTY))
+    {
+        RedisModule_CloseKey(srckey);
+        RedisModule_CloseKey(dstkey);
+        return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
+    }
+
+    long long count;
+    if ((RedisModule_StringToLongLong(argv[3],&count) != REDISMODULE_OK) ||
+        (count < 0)) {
+        RedisModule_CloseKey(srckey);
+        RedisModule_CloseKey(dstkey);
+        return RedisModule_ReplyWithError(ctx,"ERR invalid count");
+    }
+
+    while(count-- > 0) {
+        RedisModuleString *ele;
+
+        ele = RedisModule_ListPop(srckey,REDISMODULE_LIST_TAIL);
+        if (ele == NULL) break;
+        RedisModule_ListPush(dstkey,REDISMODULE_LIST_HEAD,ele);
+        RedisModule_FreeString(ctx,ele);
+    }
+
+    size_t len = RedisModule_ValueLength(srckey);
+    RedisModule_CloseKey(srckey);
+    RedisModule_CloseKey(dstkey);
+    RedisModule_ReplyWithLongLong(ctx,len);
+    return REDISMODULE_OK;
+}
+
+/* Like the HELLO.LIST.SPLICE above, but uses automatic memory management
+ * in order to avoid freeing stuff. */
+int HelloListSpliceAuto_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    if (argc != 4) return RedisModule_WrongArity(ctx);
+
+    RedisModule_AutoMemory(ctx);
+
+    RedisModuleKey *srckey = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+    RedisModuleKey *dstkey = RedisModule_OpenKey(ctx,argv[2],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+
+    /* Src and dst key must be empty or lists. */
+    if ((RedisModule_KeyType(srckey) != REDISMODULE_KEYTYPE_LIST &&
+         RedisModule_KeyType(srckey) != REDISMODULE_KEYTYPE_EMPTY) ||
+        (RedisModule_KeyType(dstkey) != REDISMODULE_KEYTYPE_LIST &&
+         RedisModule_KeyType(dstkey) != REDISMODULE_KEYTYPE_EMPTY))
+    {
+        return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
+    }
+
+    long long count;
+    if ((RedisModule_StringToLongLong(argv[3],&count) != REDISMODULE_OK) ||
+        (count < 0))
+    {
+        return RedisModule_ReplyWithError(ctx,"ERR invalid count");
+    }
+
+    while(count-- > 0) {
+        RedisModuleString *ele;
+
+        ele = RedisModule_ListPop(srckey,REDISMODULE_LIST_TAIL);
+        if (ele == NULL) break;
+        RedisModule_ListPush(dstkey,REDISMODULE_LIST_HEAD,ele);
+    }
+
+    size_t len = RedisModule_ValueLength(srckey);
+    RedisModule_ReplyWithLongLong(ctx,len);
+    return REDISMODULE_OK;
+}
+
+/* HELLO.RAND.ARRAY <count>
+ * Shows how to generate arrays as commands replies.
+ * It just outputs <count> random numbers. */
+int HelloRandArray_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    if (argc != 2) return RedisModule_WrongArity(ctx);
+    long long count;
+    if (RedisModule_StringToLongLong(argv[1],&count) != REDISMODULE_OK ||
+        count < 0)
+        return RedisModule_ReplyWithError(ctx,"ERR invalid count");
+
+    /* To reply with an array, we call RedisModule_ReplyWithArray() followed
+     * by other "count" calls to other reply functions in order to generate
+     * the elements of the array. */
+    RedisModule_ReplyWithArray(ctx,count);
+    while(count--) RedisModule_ReplyWithLongLong(ctx,rand());
+    return REDISMODULE_OK;
+}
+
+/* This is a simple command to test replication. Because of the "!" modified
+ * in the RedisModule_Call() call, the two INCRs get replicated.
+ * Also note how the ECHO is replicated in an unexpected position (check
+ * comments the function implementation). */
+int HelloRepl1_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
+{
+    RedisModuleCallReply *reply;
+    RedisModule_AutoMemory(ctx);
+
+    /* This will be replicated *after* the two INCR statements, since
+     * the Call() replication has precedence, so the actual replication
+     * stream will be:
+     *
+     * MULTI
+     * INCR foo
+     * INCR bar
+     * ECHO c foo
+     * EXEC
+     */
+    RedisModule_Replicate(ctx,"ECHO","c","foo");
+
+    /* Using the "!" modifier we replicate the command if it
+     * modified the dataset in some way. */
+    reply = RedisModule_Call(ctx,"INCR","c!","foo");
+    reply = RedisModule_Call(ctx,"INCR","c!","bar");
+
+    RedisModule_ReplyWithLongLong(ctx,0);
+
+    return REDISMODULE_OK;
+}
+
+/* Another command to show replication. In this case, we call
+ * RedisModule_ReplicateVerbatim() to mean we want just the command to be
+ * propagated to slaves / AOF exactly as it was called by the user.
+ *
+ * This command also shows how to work with string objects.
+ * It takes a list, and increments all the elements (that must have
+ * a numerical value) by 1, returning the sum of all the elements
+ * as reply.
+ *
+ * Usage: HELLO.REPL2 <list-key> */
+int HelloRepl2_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    if (argc != 2) return RedisModule_WrongArity(ctx);
+
+    RedisModule_AutoMemory(ctx); /* Use automatic memory management. */
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+
+    if (RedisModule_KeyType(key) != REDISMODULE_KEYTYPE_LIST)
+        return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
+
+    size_t listlen = RedisModule_ValueLength(key);
+    long long sum = 0;
+
+    /* Rotate and increment. */
+    while(listlen--) {
+        RedisModuleString *ele = RedisModule_ListPop(key,REDISMODULE_LIST_TAIL);
+        long long val;
+        if (RedisModule_StringToLongLong(ele,&val) != REDISMODULE_OK) val = 0;
+        val++;
+        sum += val;
+        RedisModuleString *newele = RedisModule_CreateStringFromLongLong(ctx,val);
+        RedisModule_ListPush(key,REDISMODULE_LIST_HEAD,newele);
+    }
+    RedisModule_ReplyWithLongLong(ctx,sum);
+    RedisModule_ReplicateVerbatim(ctx);
+    return REDISMODULE_OK;
+}
+
+/* This is an example of strings DMA access. Given a key containing a string
+ * it toggles the case of each character from lower to upper case or the
+ * other way around.
+ *
+ * No automatic memory management is used in this example (for the sake
+ * of variety).
+ *
+ * HELLO.TOGGLE.CASE key */
+int HelloToggleCase_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    if (argc != 2) return RedisModule_WrongArity(ctx);
+
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+
+    int keytype = RedisModule_KeyType(key);
+    if (keytype != REDISMODULE_KEYTYPE_STRING &&
+        keytype != REDISMODULE_KEYTYPE_EMPTY)
+    {
+        RedisModule_CloseKey(key);
+        return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
+    }
+
+    if (keytype == REDISMODULE_KEYTYPE_STRING) {
+        size_t len, j;
+        char *s = RedisModule_StringDMA(key,&len,REDISMODULE_WRITE);
+        for (j = 0; j < len; j++) {
+            if (isupper(s[j])) {
+                s[j] = tolower(s[j]);
+            } else {
+                s[j] = toupper(s[j]);
+            }
+        }
+    }
+
+    RedisModule_CloseKey(key);
+    RedisModule_ReplyWithSimpleString(ctx,"OK");
+    RedisModule_ReplicateVerbatim(ctx);
+    return REDISMODULE_OK;
+}
+
+/* HELLO.MORE.EXPIRE key milliseconds.
+ *
+ * If they key has already an associated TTL, extends it by "milliseconds"
+ * milliseconds. Otherwise no operation is performed. */
+int HelloMoreExpire_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    RedisModule_AutoMemory(ctx); /* Use automatic memory management. */
+    if (argc != 3) return RedisModule_WrongArity(ctx);
+
+    mstime_t addms, expire;
+
+    if (RedisModule_StringToLongLong(argv[2],&addms) != REDISMODULE_OK)
+        return RedisModule_ReplyWithError(ctx,"ERR invalid expire time");
+
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+    expire = RedisModule_GetExpire(key);
+    if (expire != REDISMODULE_NO_EXPIRE) {
+        expire += addms;
+        RedisModule_SetExpire(key,expire);
+    }
+    return RedisModule_ReplyWithSimpleString(ctx,"OK");
+}
+
+/* HELLO.ZSUMRANGE key startscore endscore
+ * Return the sum of all the scores elements between startscore and endscore.
+ *
+ * The computation is performed two times, one time from start to end and
+ * another time backward. The two scores, returned as a two element array,
+ * should match.*/
+int HelloZsumRange_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    double score_start, score_end;
+    if (argc != 4) return RedisModule_WrongArity(ctx);
+
+    if (RedisModule_StringToDouble(argv[2],&score_start) != REDISMODULE_OK ||
+        RedisModule_StringToDouble(argv[3],&score_end) != REDISMODULE_OK)
+    {
+        return RedisModule_ReplyWithError(ctx,"ERR invalid range");
+    }
+
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+    if (RedisModule_KeyType(key) != REDISMODULE_KEYTYPE_ZSET) {
+        return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
+    }
+
+    double scoresum_a = 0;
+    double scoresum_b = 0;
+
+    RedisModule_ZsetFirstInScoreRange(key,score_start,score_end,0,0);
+    while(!RedisModule_ZsetRangeEndReached(key)) {
+        double score;
+        RedisModuleString *ele = RedisModule_ZsetRangeCurrentElement(key,&score);
+        RedisModule_FreeString(ctx,ele);
+        scoresum_a += score;
+        RedisModule_ZsetRangeNext(key);
+    }
+    RedisModule_ZsetRangeStop(key);
+
+    RedisModule_ZsetLastInScoreRange(key,score_start,score_end,0,0);
+    while(!RedisModule_ZsetRangeEndReached(key)) {
+        double score;
+        RedisModuleString *ele = RedisModule_ZsetRangeCurrentElement(key,&score);
+        RedisModule_FreeString(ctx,ele);
+        scoresum_b += score;
+        RedisModule_ZsetRangePrev(key);
+    }
+
+    RedisModule_ZsetRangeStop(key);
+
+    RedisModule_CloseKey(key);
+
+    RedisModule_ReplyWithArray(ctx,2);
+    RedisModule_ReplyWithDouble(ctx,scoresum_a);
+    RedisModule_ReplyWithDouble(ctx,scoresum_b);
+    return REDISMODULE_OK;
+}
+
+/* HELLO.LEXRANGE key min_lex max_lex min_age max_age
+ * This command expects a sorted set stored at key in the following form:
+ * - All the elements have score 0.
+ * - Elements are pairs of "<name>:<age>", for example "Anna:52".
+ * The command will return all the sorted set items that are lexicographically
+ * between the specified range (using the same format as ZRANGEBYLEX)
+ * and having an age between min_age and max_age. */
+int HelloLexRange_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    RedisModule_AutoMemory(ctx); /* Use automatic memory management. */
+
+    if (argc != 6) return RedisModule_WrongArity(ctx);
+
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+    if (RedisModule_KeyType(key) != REDISMODULE_KEYTYPE_ZSET) {
+        return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
+    }
+
+    if (RedisModule_ZsetFirstInLexRange(key,argv[2],argv[3]) != REDISMODULE_OK) {
+        return RedisModule_ReplyWithError(ctx,"invalid range");
+    }
+
+    int arraylen = 0;
+    RedisModule_ReplyWithArray(ctx,REDISMODULE_POSTPONED_ARRAY_LEN);
+    while(!RedisModule_ZsetRangeEndReached(key)) {
+        double score;
+        RedisModuleString *ele = RedisModule_ZsetRangeCurrentElement(key,&score);
+        RedisModule_ReplyWithString(ctx,ele);
+        RedisModule_FreeString(ctx,ele);
+        RedisModule_ZsetRangeNext(key);
+        arraylen++;
+    }
+    RedisModule_ZsetRangeStop(key);
+    RedisModule_ReplySetArrayLength(ctx,arraylen);
+    RedisModule_CloseKey(key);
+    return REDISMODULE_OK;
+}
+
+/* HELLO.HCOPY key srcfield dstfield
+ * This is just an example command that sets the hash field dstfield to the
+ * same value of srcfield. If srcfield does not exist no operation is
+ * performed.
+ *
+ * The command returns 1 if the copy is performed (srcfield exists) otherwise
+ * 0 is returned. */
+int HelloHCopy_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    RedisModule_AutoMemory(ctx); /* Use automatic memory management. */
+
+    if (argc != 4) return RedisModule_WrongArity(ctx);
+    RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
+        REDISMODULE_READ|REDISMODULE_WRITE);
+    int type = RedisModule_KeyType(key);
+    if (type != REDISMODULE_KEYTYPE_HASH &&
+        type != REDISMODULE_KEYTYPE_EMPTY)
+    {
+        return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
+    }
+
+    /* Get the old field value. */
+    RedisModuleString *oldval;
+    RedisModule_HashGet(key,REDISMODULE_HASH_NONE,argv[2],&oldval,NULL);
+    if (oldval) {
+        RedisModule_HashSet(key,REDISMODULE_HASH_NONE,argv[3],oldval,NULL);
+    }
+    RedisModule_ReplyWithLongLong(ctx,oldval != NULL);
+    return REDISMODULE_OK;
+}
+
+/* HELLO.LEFTPAD str len ch
+ * This is an implementation of the infamous LEFTPAD function, that
+ * was at the center of an issue with the npm modules system in March 2016.
+ *
+ * LEFTPAD is a good example of using a Redis Modules API called
+ * "pool allocator", that was a famous way to allocate memory in yet another
+ * open source project, the Apache web server.
+ *
+ * The concept is very simple: there is memory that is useful to allocate
+ * only in the context of serving a request, and must be freed anyway when
+ * the callback implementing the command returns. So in that case the module
+ * does not need to retain a reference to these allocations, it is just
+ * required to free the memory before returning. When this is the case the
+ * module can call RedisModule_PoolAlloc() instead, that works like malloc()
+ * but will automatically free the memory when the module callback returns.
+ *
+ * Note that PoolAlloc() does not necessarily require AutoMemory to be
+ * active. */
+int HelloLeftPad_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    RedisModule_AutoMemory(ctx); /* Use automatic memory management. */
+    long long padlen;
+
+    if (argc != 4) return RedisModule_WrongArity(ctx);
+
+    if ((RedisModule_StringToLongLong(argv[2],&padlen) != REDISMODULE_OK) ||
+        (padlen< 0)) {
+        return RedisModule_ReplyWithError(ctx,"ERR invalid padding length");
+    }
+    size_t strlen, chlen;
+    const char *str = RedisModule_StringPtrLen(argv[1], &strlen);
+    const char *ch = RedisModule_StringPtrLen(argv[3], &chlen);
+
+    /* If the string is already larger than the target len, just return
+     * the string itself. */
+    if (strlen >= padlen)
+        return RedisModule_ReplyWithString(ctx,argv[1]);
+
+    /* Padding must be a single character in this simple implementation. */
+    if (chlen != 1)
+        return RedisModule_ReplyWithError(ctx,
+            "ERR padding must be a single char");
+
+    /* Here we use our pool allocator, for our throw-away allocation. */
+    padlen -= strlen;
+    char *buf = RedisModule_PoolAlloc(ctx,padlen+strlen);
+    for (size_t j = 0; j < padlen; j++) buf[j] = *ch;
+    memcpy(buf+padlen,str,strlen);
+
+    RedisModule_ReplyWithStringBuffer(ctx,buf,padlen+strlen);
+    return REDISMODULE_OK;
+}
+
+/* This function must be present on each Redis module. It is used in order to
+ * register the commands into the Redis server. */
+int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    if (RedisModule_Init(ctx,"helloworld",1,REDISMODULE_APIVER_1)
+        == REDISMODULE_ERR) return REDISMODULE_ERR;
+
+    /* Log the list of parameters passing loading the module. */
+    for (int j = 0; j < argc; j++) {
+        const char *s = RedisModule_StringPtrLen(argv[j],NULL);
+        printf("Module loaded with ARGV[%d] = %s\n", j, s);
+    }
+
+    if (RedisModule_CreateCommand(ctx,"hello.simple",
+        HelloSimple_RedisCommand,"readonly",0,0,0) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.push.native",
+        HelloPushNative_RedisCommand,"write deny-oom",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.push.call",
+        HelloPushCall_RedisCommand,"write deny-oom",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.push.call2",
+        HelloPushCall2_RedisCommand,"write deny-oom",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.list.sum.len",
+        HelloListSumLen_RedisCommand,"readonly",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.list.splice",
+        HelloListSplice_RedisCommand,"write deny-oom",1,2,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.list.splice.auto",
+        HelloListSpliceAuto_RedisCommand,
+        "write deny-oom",1,2,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.rand.array",
+        HelloRandArray_RedisCommand,"readonly",0,0,0) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.repl1",
+        HelloRepl1_RedisCommand,"write",0,0,0) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.repl2",
+        HelloRepl2_RedisCommand,"write",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.toggle.case",
+        HelloToggleCase_RedisCommand,"write",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.more.expire",
+        HelloMoreExpire_RedisCommand,"write",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.zsumrange",
+        HelloZsumRange_RedisCommand,"readonly",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.lexrange",
+        HelloLexRange_RedisCommand,"readonly",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.hcopy",
+        HelloHCopy_RedisCommand,"write deny-oom",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"hello.leftpad",
+        HelloLeftPad_RedisCommand,"",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    return REDISMODULE_OK;
+}

src/modules/testmodule.c

@@ -0,0 +1,222 @@
+/* Module designed to test the Redis modules subsystem.
+ *
+ * -----------------------------------------------------------------------------
+ *
+ * Copyright (c) 2016, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "../redismodule.h"
+#include <string.h>
+
+/* --------------------------------- Helpers -------------------------------- */
+
+/* Return true if the reply and the C null term string matches. */
+int TestMatchReply(RedisModuleCallReply *reply, char *str) {
+    RedisModuleString *mystr;
+    mystr = RedisModule_CreateStringFromCallReply(reply);
+    if (!mystr) return 0;
+    const char *ptr = RedisModule_StringPtrLen(mystr,NULL);
+    return strcmp(ptr,str) == 0;
+}
+
+/* ------------------------------- Test units ------------------------------- */
+
+/* TEST.CALL -- Test Call() API. */
+int TestCall(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    RedisModule_AutoMemory(ctx);
+    RedisModuleCallReply *reply;
+
+    RedisModule_Call(ctx,"DEL","c","mylist");
+    RedisModuleString *mystr = RedisModule_CreateString(ctx,"foo",3);
+    RedisModule_Call(ctx,"RPUSH","csl","mylist",mystr,(long long)1234);
+    reply = RedisModule_Call(ctx,"LRANGE","ccc","mylist","0","-1");
+    long long items = RedisModule_CallReplyLength(reply);
+    if (items != 2) goto fail;
+
+    RedisModuleCallReply *item0, *item1;
+
+    item0 = RedisModule_CallReplyArrayElement(reply,0);
+    item1 = RedisModule_CallReplyArrayElement(reply,1);
+    if (!TestMatchReply(item0,"foo")) goto fail;
+    if (!TestMatchReply(item1,"1234")) goto fail;
+
+    RedisModule_ReplyWithSimpleString(ctx,"OK");
+    return REDISMODULE_OK;
+
+fail:
+    RedisModule_ReplyWithSimpleString(ctx,"ERR");
+    return REDISMODULE_OK;
+}
+
+/* TEST.STRING.APPEND -- Test appending to an existing string object. */
+int TestStringAppend(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    RedisModuleString *s = RedisModule_CreateString(ctx,"foo",3);
+    RedisModule_StringAppendBuffer(ctx,s,"bar",3);
+    RedisModule_ReplyWithString(ctx,s);
+    RedisModule_FreeString(ctx,s);
+    return REDISMODULE_OK;
+}
+
+/* TEST.STRING.APPEND.AM -- Test append with retain when auto memory is on. */
+int TestStringAppendAM(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    RedisModule_AutoMemory(ctx);
+    RedisModuleString *s = RedisModule_CreateString(ctx,"foo",3);
+    RedisModule_RetainString(ctx,s);
+    RedisModule_StringAppendBuffer(ctx,s,"bar",3);
+    RedisModule_ReplyWithString(ctx,s);
+    RedisModule_FreeString(ctx,s);
+    return REDISMODULE_OK;
+}
+
+/* TEST.STRING.PRINTF -- Test string formatting. */
+int TestStringPrintf(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    RedisModule_AutoMemory(ctx);
+    if (argc < 3) {
+        return RedisModule_WrongArity(ctx);
+    }
+    RedisModuleString *s = RedisModule_CreateStringPrintf(ctx, 
+        "Got %d args. argv[1]: %s, argv[2]: %s", 
+        argc, 
+        RedisModule_StringPtrLen(argv[1], NULL),
+        RedisModule_StringPtrLen(argv[2], NULL)
+    );
+
+    RedisModule_ReplyWithString(ctx,s);
+
+    return REDISMODULE_OK;
+}
+
+
+/* ----------------------------- Test framework ----------------------------- */
+
+/* Return 1 if the reply matches the specified string, otherwise log errors
+ * in the server log and return 0. */
+int TestAssertStringReply(RedisModuleCtx *ctx, RedisModuleCallReply *reply, char *str, size_t len) {
+    RedisModuleString *mystr, *expected;
+
+    if (RedisModule_CallReplyType(reply) != REDISMODULE_REPLY_STRING) {
+        RedisModule_Log(ctx,"warning","Unexpected reply type %d",
+            RedisModule_CallReplyType(reply));
+        return 0;
+    }
+    mystr = RedisModule_CreateStringFromCallReply(reply);
+    expected = RedisModule_CreateString(ctx,str,len);
+    if (RedisModule_StringCompare(mystr,expected) != 0) {
+        const char *mystr_ptr = RedisModule_StringPtrLen(mystr,NULL);
+        const char *expected_ptr = RedisModule_StringPtrLen(expected,NULL);
+        RedisModule_Log(ctx,"warning",
+            "Unexpected string reply '%s' (instead of '%s')",
+            mystr_ptr, expected_ptr);
+        return 0;
+    }
+    return 1;
+}
+
+/* Return 1 if the reply matches the specified integer, otherwise log errors
+ * in the server log and return 0. */
+int TestAssertIntegerReply(RedisModuleCtx *ctx, RedisModuleCallReply *reply, long long expected) {
+    if (RedisModule_CallReplyType(reply) != REDISMODULE_REPLY_INTEGER) {
+        RedisModule_Log(ctx,"warning","Unexpected reply type %d",
+            RedisModule_CallReplyType(reply));
+        return 0;
+    }
+    long long val = RedisModule_CallReplyInteger(reply);
+    if (val != expected) {
+        RedisModule_Log(ctx,"warning",
+            "Unexpected integer reply '%lld' (instead of '%lld')",
+            val, expected);
+        return 0;
+    }
+    return 1;
+}
+
+#define T(name,...) \
+    do { \
+        RedisModule_Log(ctx,"warning","Testing %s", name); \
+        reply = RedisModule_Call(ctx,name,__VA_ARGS__); \
+    } while (0);
+
+/* TEST.IT -- Run all the tests. */
+int TestIt(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    RedisModule_AutoMemory(ctx);
+    RedisModuleCallReply *reply;
+
+    /* Make sure the DB is empty before to proceed. */
+    T("dbsize","");
+    if (!TestAssertIntegerReply(ctx,reply,0)) goto fail;
+
+    T("ping","");
+    if (!TestAssertStringReply(ctx,reply,"PONG",4)) goto fail;
+
+    T("test.call","");
+    if (!TestAssertStringReply(ctx,reply,"OK",2)) goto fail;
+
+    T("test.string.append","");
+    if (!TestAssertStringReply(ctx,reply,"foobar",6)) goto fail;
+
+    T("test.string.append.am","");
+    if (!TestAssertStringReply(ctx,reply,"foobar",6)) goto fail;
+
+    T("test.string.printf", "cc", "foo", "bar");
+    if (!TestAssertStringReply(ctx,reply,"Got 3 args. argv[1]: foo, argv[2]: bar",38)) goto fail;
+
+    RedisModule_ReplyWithSimpleString(ctx,"ALL TESTS PASSED");
+    return REDISMODULE_OK;
+
+fail:
+    RedisModule_ReplyWithSimpleString(ctx,
+        "SOME TEST NOT PASSED! Check server logs");
+    return REDISMODULE_OK;
+}
+
+int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
+    if (RedisModule_Init(ctx,"test",1,REDISMODULE_APIVER_1)
+        == REDISMODULE_ERR) return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"test.call",
+        TestCall,"write deny-oom",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"test.string.append",
+        TestStringAppend,"write deny-oom",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"test.string.append.am",
+        TestStringAppendAM,"write deny-oom",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"test.string.printf",
+        TestStringPrintf,"write deny-oom",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    if (RedisModule_CreateCommand(ctx,"test.it",
+        TestIt,"readonly",1,1,1) == REDISMODULE_ERR)
+        return REDISMODULE_ERR;
+
+    return REDISMODULE_OK;
+}

src/networking.c

@@ -109,6 +109,7 @@ client *createClient(int fd) {
     c->repl_ack_off = 0;
     c->repl_ack_time = 0;
     c->slave_listening_port = 0;
+    c->slave_ip[0] = '\0';
     c->slave_capa = SLAVE_CAPA_NONE;
     c->reply = listCreate();
     c->reply_bytes = 0;
@@ -158,7 +159,7 @@ client *createClient(int fd) {
 int prepareClientToWrite(client *c) {
     /* If it's the Lua client we always return ok without installing any
      * handler since there is no socket at all. */
-    if (c->flags & CLIENT_LUA) return C_OK;
+    if (c->flags & (CLIENT_LUA|CLIENT_MODULE)) return C_OK;
 
     /* CLIENT REPLY OFF / SKIP handling: don't send replies. */
     if (c->flags & (CLIENT_REPLY_OFF|CLIENT_REPLY_SKIP)) return C_ERR;
@@ -171,7 +172,7 @@ int prepareClientToWrite(client *c) {
     if (c->fd <= 0) return C_ERR; /* Fake client for AOF loading. */
 
     /* Schedule the client to write the output buffers to the socket only
-     * if not already done (there were no pending writes alreday and the client
+     * if not already done (there were no pending writes already and the client
      * was yet not flagged), and, for slaves, if the slave can actually
      * receive writes at this stage. */
     if (!clientHasPendingReplies(c) &&
@@ -455,6 +456,15 @@ void addReplyDouble(client *c, double d) {
     }
 }
 
+/* Add a long double as a bulk reply, but uses a human readable formatting
+ * of the double instead of exposing the crude behavior of doubles to the
+ * dear user. */
+void addReplyHumanLongDouble(client *c, long double d) {
+    robj *o = createStringObjectFromLongDouble(d,1);
+    addReplyBulk(c,o);
+    decrRefCount(o);
+}
+
 /* Add a long long as integer reply or bulk len / multi bulk count.
  * Basically this is used to output <prefix><long long><crlf>. */
 void addReplyLongLongWithPrefix(client *c, long long ll, char prefix) {
@@ -579,7 +589,7 @@ int clientHasPendingReplies(client *c) {
 }
 
 #define MAX_ACCEPTS_PER_CALL 1000
-static void acceptCommonHandler(int fd, int flags) {
+static void acceptCommonHandler(int fd, int flags, char *ip) {
     client *c;
     if ((c = createClient(fd)) == NULL) {
         serverLog(LL_WARNING,
@@ -603,6 +613,48 @@ static void acceptCommonHandler(int fd, int flags) {
         freeClient(c);
         return;
     }
+
+    /* If the server is running in protected mode (the default) and there
+     * is no password set, nor a specific interface is bound, we don't accept
+     * requests from non loopback interfaces. Instead we try to explain the
+     * user what to do to fix it if needed. */
+    if (server.protected_mode &&
+        server.bindaddr_count == 0 &&
+        server.requirepass == NULL &&
+        !(flags & CLIENT_UNIX_SOCKET) &&
+        ip != NULL)
+    {
+        if (strcmp(ip,"127.0.0.1") && strcmp(ip,"::1")) {
+            char *err =
+                "-DENIED Redis is running in protected mode because protected "
+                "mode is enabled, no bind address was specified, no "
+                "authentication password is requested to clients. In this mode "
+                "connections are only accepted from the loopback interface. "
+                "If you want to connect from external computers to Redis you "
+                "may adopt one of the following solutions: "
+                "1) Just disable protected mode sending the command "
+                "'CONFIG SET protected-mode no' from the loopback interface "
+                "by connecting to Redis from the same host the server is "
+                "running, however MAKE SURE Redis is not publicly accessible "
+                "from internet if you do so. Use CONFIG REWRITE to make this "
+                "change permanent. "
+                "2) Alternatively you can just disable the protected mode by "
+                "editing the Redis configuration file, and setting the protected "
+                "mode option to 'no', and then restarting the server. "
+                "3) If you started the server manually just for testing, restart "
+                "it with the '--protected-mode no' option. "
+                "4) Setup a bind address or an authentication password. "
+                "NOTE: You only need to do one of the above things in order for "
+                "the server to start accepting connections from the outside.\r\n";
+            if (write(c->fd,err,strlen(err)) == -1) {
+                /* Nothing to do, Just to avoid the warning... */
+            }
+            server.stat_rejected_conn++;
+            freeClient(c);
+            return;
+        }
+    }
+
     server.stat_numconnections++;
     c->flags |= flags;
 }
@@ -623,7 +675,7 @@ void acceptTcpHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
             return;
         }
         serverLog(LL_VERBOSE,"Accepted %s:%d", cip, cport);
-        acceptCommonHandler(cfd,0);
+        acceptCommonHandler(cfd,0,cip);
     }
 }
 
@@ -642,7 +694,7 @@ void acceptUnixHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
             return;
         }
         serverLog(LL_VERBOSE,"Accepted connection to %s", server.unixsocket);
-        acceptCommonHandler(cfd,CLIENT_UNIX_SOCKET);
+        acceptCommonHandler(cfd,CLIENT_UNIX_SOCKET,NULL);
     }
 }
 
@@ -694,6 +746,7 @@ void unlinkClient(client *c) {
         ln = listSearchKey(server.clients_pending_write,c);
         serverAssert(ln != NULL);
         listDelNode(server.clients_pending_write,ln);
+        c->flags &= ~CLIENT_PENDING_WRITE;
     }
 
     /* When client was just unblocked because of a blocking operation,
@@ -702,6 +755,7 @@ void unlinkClient(client *c) {
         ln = listSearchKey(server.unblocked_clients,c);
         serverAssert(ln != NULL);
         listDelNode(server.unblocked_clients,ln);
+        c->flags &= ~CLIENT_UNBLOCKED;
     }
 }
 
@@ -1151,10 +1205,10 @@ int processMultibulkBuffer(client *c) {
             {
                 c->argv[c->argc++] = createObject(OBJ_STRING,c->querybuf);
                 sdsIncrLen(c->querybuf,-2); /* remove CRLF */
-                c->querybuf = sdsempty();
                 /* Assume that if we saw a fat argument we'll see another one
                  * likely... */
-                c->querybuf = sdsMakeRoomFor(c->querybuf,c->bulklen+2);
+                c->querybuf = sdsnewlen(NULL,c->bulklen+2);
+                sdsclear(c->querybuf);
                 pos = 0;
             } else {
                 c->argv[c->argc++] =
@@ -1188,8 +1242,10 @@ void processInputBuffer(client *c) {
 
         /* CLIENT_CLOSE_AFTER_REPLY closes the connection once the reply is
          * written to the client. Make sure to not let the reply grow after
-         * this flag has been set (i.e. don't process more commands). */
-        if (c->flags & CLIENT_CLOSE_AFTER_REPLY) break;
+         * this flag has been set (i.e. don't process more commands).
+         *
+         * The same applies for clients we want to terminate ASAP. */
+        if (c->flags & (CLIENT_CLOSE_AFTER_REPLY|CLIENT_CLOSE_ASAP)) break;
 
         /* Determine request type when unknown. */
         if (!c->reqtype) {
@@ -1215,6 +1271,9 @@ void processInputBuffer(client *c) {
             /* Only reset the client when the command was executed. */
             if (processCommand(c) == C_OK)
                 resetClient(c);
+            /* freeMemoryIfNeeded may flush slave output buffers. This may result
+             * into a slave, that may be the active client, to be freed. */
+            if (server.current_client == NULL) break;
         }
     }
     server.current_client = NULL;
@@ -1387,9 +1446,8 @@ sds getAllClientsInfoString(void) {
     listNode *ln;
     listIter li;
     client *client;
-    sds o = sdsempty();
-
-    o = sdsMakeRoomFor(o,200*listLength(server.clients));
+    sds o = sdsnewlen(NULL,200*listLength(server.clients));
+    sdsclear(o);
     listRewind(server.clients,&li);
     while ((ln = listNext(&li)) != NULL) {
         client = listNodeValue(ln);
@@ -1550,10 +1608,30 @@ void clientCommand(client *c) {
         pauseClients(duration);
         addReply(c,shared.ok);
     } else {
-        addReplyError(c, "Syntax error, try CLIENT (LIST | KILL ip:port | GETNAME | SETNAME connection-name)");
+        addReplyError(c, "Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)");
     }
 }
 
+/* This callback is bound to POST and "Host:" command names. Those are not
+ * really commands, but are used in security attacks in order to talk to
+ * Redis instances via HTTP, with a technique called "cross protocol scripting"
+ * which exploits the fact that services like Redis will discard invalid
+ * HTTP headers and will process what follows.
+ *
+ * As a protection against this attack, Redis will terminate the connection
+ * when a POST or "Host:" header is seen, and will log the event from
+ * time to time (to avoid creating a DOS as a result of too many logs). */
+void securityWarningCommand(client *c) {
+    static time_t logged_time;
+    time_t now = time(NULL);
+
+    if (labs(now-logged_time) > 60) {
+        serverLog(LL_WARNING,"Possible SECURITY ATTACK detected. It looks like somebody is sending POST or Host: commands to Redis. This is likely due to an attacker attempting to use Cross Protocol Scripting to compromise your Redis instance. Connection aborted.");
+        logged_time = now;
+    }
+    freeClientAsync(c);
+}
+
 /* Rewrite the command vector of the client. All the new objects ref count
  * is incremented. The old command vector is freed, and the old objects
  * ref count is decremented. */
@@ -1595,15 +1673,28 @@ void replaceClientCommandVector(client *c, int argc, robj **argv) {
 }
 
 /* Rewrite a single item in the command vector.
- * The new val ref count is incremented, and the old decremented. */
+ * The new val ref count is incremented, and the old decremented.
+ *
+ * It is possible to specify an argument over the current size of the
+ * argument vector: in this case the array of objects gets reallocated
+ * and c->argc set to the max value. However it's up to the caller to
+ *
+ * 1. Make sure there are no "holes" and all the arguments are set.
+ * 2. If the original argument vector was longer than the one we
+ *    want to end with, it's up to the caller to set c->argc and
+ *    free the no longer used objects on c->argv. */
 void rewriteClientCommandArgument(client *c, int i, robj *newval) {
     robj *oldval;
 
-    serverAssertWithInfo(c,NULL,i < c->argc);
+    if (i >= c->argc) {
+        c->argv = zrealloc(c->argv,sizeof(robj*)*(i+1));
+        c->argc = i+1;
+        c->argv[i] = NULL;
+    }
     oldval = c->argv[i];
     c->argv[i] = newval;
     incrRefCount(newval);
-    decrRefCount(oldval);
+    if (oldval) decrRefCount(oldval);
 
     /* If this is the command name make sure to fix c->cmd. */
     if (i == 0) {
@@ -1818,7 +1909,7 @@ int clientsArePaused(void) {
  * and so forth.
  *
  * It calls the event loop in order to process a few events. Specifically we
- * try to call the event loop for times as long as we receive acknowledge that
+ * try to call the event loop 4 times as long as we receive acknowledge that
  * some event was processed, in order to go forward with the accept, read,
  * write, close sequence needed to serve a client.
  *

src/object.c

@@ -36,6 +36,8 @@
 #define strtold(a,b) ((long double)strtod((a),(b)))
 #endif
 
+/* ===================== Creation and parsing of objects ==================== */
+
 robj *createObject(int type, void *ptr) {
     robj *o = zmalloc(sizeof(*o));
     o->type = type;
@@ -43,8 +45,13 @@ robj *createObject(int type, void *ptr) {
     o->ptr = ptr;
     o->refcount = 1;
 
-    /* Set the LRU to the current lruclock (minutes resolution). */
-    o->lru = LRU_CLOCK();
+    /* Set the LRU to the current lruclock (minutes resolution), or
+     * alternatively the LFU counter. */
+    if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) {
+        o->lru = (LFUGetTimeInMinutes()<<8) | LFU_INIT_VAL;
+    } else {
+        o->lru = LRU_CLOCK();
+    }
     return o;
 }
 
@@ -68,7 +75,7 @@ robj *makeObjectShared(robj *o) {
 /* Create a string object with encoding OBJ_ENCODING_RAW, that is a plain
  * string object where o->ptr points to a proper sds string. */
 robj *createRawStringObject(const char *ptr, size_t len) {
-    return createObject(OBJ_STRING,sdsnewlen(ptr,len));
+    return createObject(OBJ_STRING, sdsnewlen(ptr,len));
 }
 
 /* Create a string object with encoding OBJ_ENCODING_EMBSTR, that is
@@ -82,7 +89,11 @@ robj *createEmbeddedStringObject(const char *ptr, size_t len) {
     o->encoding = OBJ_ENCODING_EMBSTR;
     o->ptr = sh+1;
     o->refcount = 1;
-    o->lru = LRU_CLOCK();
+    if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) {
+        o->lru = (LFUGetTimeInMinutes()<<8) | LFU_INIT_VAL;
+    } else {
+        o->lru = LRU_CLOCK();
+    }
 
     sh->len = len;
     sh->alloc = len;
@@ -97,7 +108,7 @@ robj *createEmbeddedStringObject(const char *ptr, size_t len) {
 }
 
 /* Create a string object with EMBSTR encoding if it is smaller than
- * REIDS_ENCODING_EMBSTR_SIZE_LIMIT, otherwise the RAW encoding is
+ * OBJ_ENCODING_EMBSTR_SIZE_LIMIT, otherwise the RAW encoding is
  * used.
  *
  * The current limit of 39 is chosen so that the biggest string object
@@ -147,7 +158,7 @@ robj *createStringObjectFromLongDouble(long double value, int humanfriendly) {
  * will always result in a fresh object that is unshared (refcount == 1).
  *
  * The resulting object always has refcount set to 1. */
-robj *dupStringObject(robj *o) {
+robj *dupStringObject(const robj *o) {
     robj *d;
 
     serverAssert(o->type == OBJ_STRING);
@@ -221,6 +232,13 @@ robj *createZsetZiplistObject(void) {
     return o;
 }
 
+robj *createModuleObject(moduleType *mt, void *value) {
+    moduleValue *mv = zmalloc(sizeof(*mv));
+    mv->type = mt;
+    mv->value = value;
+    return createObject(OBJ_MODULE,mv);
+}
+
 void freeStringObject(robj *o) {
     if (o->encoding == OBJ_ENCODING_RAW) {
         sdsfree(o->ptr);
@@ -281,6 +299,12 @@ void freeHashObject(robj *o) {
     }
 }
 
+void freeModuleObject(robj *o) {
+    moduleValue *mv = o->ptr;
+    mv->type->free(mv->value);
+    zfree(mv);
+}
+
 void incrRefCount(robj *o) {
     if (o->refcount != OBJ_SHARED_REFCOUNT) o->refcount++;
 }
@@ -293,6 +317,7 @@ void decrRefCount(robj *o) {
         case OBJ_SET: freeSetObject(o); break;
         case OBJ_ZSET: freeZsetObject(o); break;
         case OBJ_HASH: freeHashObject(o); break;
+        case OBJ_MODULE: freeModuleObject(o); break;
         default: serverPanic("Unknown object type"); break;
         }
         zfree(o);
@@ -371,17 +396,16 @@ robj *tryObjectEncoding(robj *o) {
      if (o->refcount > 1) return o;
 
     /* Check if we can represent this string as a long integer.
-     * Note that we are sure that a string larger than 21 chars is not
+     * Note that we are sure that a string larger than 20 chars is not
      * representable as a 32 nor 64 bit integer. */
     len = sdslen(s);
-    if (len <= 21 && string2l(s,len,&value)) {
+    if (len <= 20 && string2l(s,len,&value)) {
         /* This object is encodable as a long. Try to use a shared object.
          * Note that we avoid using shared integers when maxmemory is used
          * because every object needs to have a private LRU field for the LRU
          * algorithm to work well. */
         if ((server.maxmemory == 0 ||
-             (server.maxmemory_policy != MAXMEMORY_VOLATILE_LRU &&
-              server.maxmemory_policy != MAXMEMORY_ALLKEYS_LRU)) &&
+            !(server.maxmemory_policy & MAXMEMORY_FLAG_NO_SHARED_INTEGERS)) &&
             value >= 0 &&
             value < OBJ_SHARED_INTEGERS)
         {
@@ -525,7 +549,7 @@ size_t stringObjectLen(robj *o) {
     }
 }
 
-int getDoubleFromObject(robj *o, double *target) {
+int getDoubleFromObject(const robj *o, double *target) {
     double value;
     char *eptr;
 
@@ -536,7 +560,7 @@ int getDoubleFromObject(robj *o, double *target) {
         if (sdsEncodedObject(o)) {
             errno = 0;
             value = strtod(o->ptr, &eptr);
-            if (isspace(((char*)o->ptr)[0]) ||
+            if (isspace(((const char*)o->ptr)[0]) ||
                 eptr[0] != '\0' ||
                 (errno == ERANGE &&
                     (value == HUGE_VAL || value == -HUGE_VAL || value == 0)) ||
@@ -605,20 +629,6 @@ int getLongDoubleFromObjectOrReply(client *c, robj *o, long double *target, cons
     return C_OK;
 }
 
-/* Helper function for getLongLongFromObject(). The function parses the string
- * as a long long value in a strict way (no spaces before/after). On success
- * C_OK is returned, otherwise C_ERR is returned. */
-int strict_strtoll(char *str, long long *vp) {
-    char *eptr;
-    long long value;
-
-    errno = 0;
-    value = strtoll(str, &eptr, 10);
-    if (isspace(str[0]) || eptr[0] != '\0' || errno == ERANGE) return C_ERR;
-    if (vp) *vp = value;
-    return C_OK;
-}
-
 int getLongLongFromObject(robj *o, long long *target) {
     long long value;
 
@@ -627,7 +637,7 @@ int getLongLongFromObject(robj *o, long long *target) {
     } else {
         serverAssertWithInfo(NULL,o,o->type == OBJ_STRING);
         if (sdsEncodedObject(o)) {
-            if (strict_strtoll(o->ptr,&value) == C_ERR) return C_ERR;
+            if (string2ll(o->ptr,sdslen(o->ptr),&value) == 0) return C_ERR;
         } else if (o->encoding == OBJ_ENCODING_INT) {
             value = (long)o->ptr;
         } else {
@@ -682,18 +692,299 @@ char *strEncoding(int encoding) {
     }
 }
 
-/* Given an object returns the min number of milliseconds the object was never
- * requested, using an approximated LRU algorithm. */
-unsigned long long estimateObjectIdleTime(robj *o) {
-    unsigned long long lruclock = LRU_CLOCK();
-    if (lruclock >= o->lru) {
-        return (lruclock - o->lru) * LRU_CLOCK_RESOLUTION;
+/* =========================== Memory introspection ========================== */
+
+/* Returns the size in bytes consumed by the key's value in RAM.
+ * Note that the returned value is just an approximation, especially in the
+ * case of aggregated data types where only "sample_size" elements
+ * are checked and averaged to estimate the total size. */
+#define OBJ_COMPUTE_SIZE_DEF_SAMPLES 5 /* Default sample size. */
+size_t objectComputeSize(robj *o, size_t sample_size) {
+    sds ele, ele2;
+    dict *d;
+    dictIterator *di;
+    struct dictEntry *de;
+    size_t asize = 0, elesize = 0, samples = 0;
+
+    if (o->type == OBJ_STRING) {
+        if(o->encoding == OBJ_ENCODING_INT) {
+            asize = sizeof(*o);
+        } else if(o->encoding == OBJ_ENCODING_RAW) {
+            asize = sdsAllocSize(o->ptr)+sizeof(*o);
+        } else if(o->encoding == OBJ_ENCODING_EMBSTR) {
+            asize = sdslen(o->ptr)+2+sizeof(*o);
+        } else {
+            serverPanic("Unknown string encoding");
+        }
+    } else if (o->type == OBJ_LIST) {
+        if (o->encoding == OBJ_ENCODING_QUICKLIST) {
+            quicklist *ql = o->ptr;
+            quicklistNode *node = ql->head;
+            asize = sizeof(*o)+sizeof(quicklist);
+            do {
+                elesize += sizeof(quicklistNode)+ziplistBlobLen(node->zl);
+                samples++;
+            } while ((node = node->next) && samples < sample_size);
+            asize += (double)elesize/samples*listTypeLength(o);
+        } else if (o->encoding == OBJ_ENCODING_ZIPLIST) {
+            asize = sizeof(*o)+ziplistBlobLen(o->ptr);
+        } else {
+            serverPanic("Unknown list encoding");
+        }
+    } else if (o->type == OBJ_SET) {
+        if (o->encoding == OBJ_ENCODING_HT) {
+            d = o->ptr;
+            di = dictGetIterator(d);
+            asize = sizeof(*o)+sizeof(dict)+(sizeof(struct dictEntry*)*dictSlots(d));
+            while((de = dictNext(di)) != NULL && samples < sample_size) {
+                ele = dictGetKey(de);
+                elesize += sizeof(struct dictEntry) + sdsAllocSize(ele);
+                samples++;
+            }
+            dictReleaseIterator(di);
+            if (samples) asize += (double)elesize/samples*dictSize(d);
+        } else if (o->encoding == OBJ_ENCODING_INTSET) {
+            intset *is = o->ptr;
+            asize = sizeof(*o)+sizeof(*is)+is->encoding*is->length;
+        } else {
+            serverPanic("Unknown set encoding");
+        }
+    } else if (o->type == OBJ_ZSET) {
+        if (o->encoding == OBJ_ENCODING_ZIPLIST) {
+            asize = sizeof(*o)+(ziplistBlobLen(o->ptr));
+        } else if (o->encoding == OBJ_ENCODING_SKIPLIST) {
+            d = ((zset*)o->ptr)->dict;
+            zskiplist *zsl = ((zset*)o->ptr)->zsl;
+            zskiplistNode *znode = zsl->header->level[0].forward;
+            asize = sizeof(*o)+sizeof(zset)+(sizeof(struct dictEntry*)*dictSlots(d));
+            while(znode != NULL && samples < sample_size) {
+                elesize += sdsAllocSize(znode->ele);
+                elesize += sizeof(struct dictEntry) + zmalloc_size(znode);
+                samples++;
+                znode = znode->level[0].forward;
+            }
+            if (samples) asize += (double)elesize/samples*dictSize(d);
+        } else {
+            serverPanic("Unknown sorted set encoding");
+        }
+    } else if (o->type == OBJ_HASH) {
+        if (o->encoding == OBJ_ENCODING_ZIPLIST) {
+            asize = sizeof(*o)+(ziplistBlobLen(o->ptr));
+        } else if (o->encoding == OBJ_ENCODING_HT) {
+            d = o->ptr;
+            di = dictGetIterator(d);
+            asize = sizeof(*o)+sizeof(dict)+(sizeof(struct dictEntry*)*dictSlots(d));
+            while((de = dictNext(di)) != NULL && samples < sample_size) {
+                ele = dictGetKey(de);
+                ele2 = dictGetVal(de);
+                elesize += sdsAllocSize(ele) + sdsAllocSize(ele2);
+                elesize += sizeof(struct dictEntry);
+                samples++;
+            }
+            dictReleaseIterator(di);
+            if (samples) asize += (double)elesize/samples*dictSize(d);
+        } else {
+            serverPanic("Unknown hash encoding");
+        }
     } else {
-        return (lruclock + (LRU_CLOCK_MAX - o->lru)) *
-                    LRU_CLOCK_RESOLUTION;
+        serverPanic("Unknown object type");
     }
+    return asize;
 }
 
+/* Release data obtained with getMemoryOverheadData(). */
+void freeMemoryOverheadData(struct redisMemOverhead *mh) {
+    zfree(mh->db);
+    zfree(mh);
+}
+
+/* Return a struct redisMemOverhead filled with memory overhead
+ * information used for the MEMORY OVERHEAD and INFO command. The returned
+ * structure pointer should be freed calling freeMemoryOverheadData(). */
+struct redisMemOverhead *getMemoryOverheadData(void) {
+    int j;
+    size_t mem_total = 0;
+    size_t mem = 0;
+    size_t zmalloc_used = zmalloc_used_memory();
+    struct redisMemOverhead *mh = zcalloc(sizeof(*mh));
+
+    mh->total_allocated = zmalloc_used;
+    mh->startup_allocated = server.initial_memory_usage;
+    mh->peak_allocated = server.stat_peak_memory;
+    mh->fragmentation =
+        zmalloc_get_fragmentation_ratio(server.resident_set_size);
+    mem_total += server.initial_memory_usage;
+
+    mem = 0;
+    if (server.repl_backlog)
+        mem += zmalloc_size(server.repl_backlog);
+    mh->repl_backlog = mem;
+    mem_total += mem;
+
+    mem = 0;
+    if (listLength(server.slaves)) {
+        listIter li;
+        listNode *ln;
+
+        listRewind(server.slaves,&li);
+        while((ln = listNext(&li))) {
+            client *client = listNodeValue(ln);
+            mem += getClientOutputBufferMemoryUsage(client);
+            mem += sdsAllocSize(client->querybuf);
+            mem += sizeof(client);
+        }
+    }
+    mh->clients_slaves = mem;
+    mem_total+=mem;
+
+    mem = 0;
+    if (listLength(server.clients)) {
+        listIter li;
+        listNode *ln;
+
+        listRewind(server.clients,&li);
+        while((ln = listNext(&li))) {
+            client *client = listNodeValue(ln);
+            if (client->flags & CLIENT_SLAVE)
+                continue;
+            mem += getClientOutputBufferMemoryUsage(client);
+            mem += sdsAllocSize(client->querybuf);
+            mem += sizeof(client);
+        }
+    }
+    mh->clients_normal = mem;
+    mem_total+=mem;
+
+    mem = 0;
+    if (server.aof_state != AOF_OFF) {
+        mem += sdslen(server.aof_buf);
+        mem += aofRewriteBufferSize();
+    }
+    mh->aof_buffer = mem;
+    mem_total+=mem;
+
+    for (j = 0; j < server.dbnum; j++) {
+        redisDb *db = server.db+j;
+        long long keyscount = dictSize(db->dict);
+        if (keyscount==0) continue;
+
+        mh->total_keys += keyscount;
+        mh->db = zrealloc(mh->db,sizeof(mh->db[0])*(mh->num_dbs+1));
+        mh->db[mh->num_dbs].dbid = j;
+
+        mem = dictSize(db->dict) * sizeof(dictEntry) +
+              dictSlots(db->dict) * sizeof(dictEntry*) +
+              dictSize(db->dict) * sizeof(robj);
+        mh->db[mh->num_dbs].overhead_ht_main = mem;
+        mem_total+=mem;
+
+        mem = dictSize(db->expires) * sizeof(dictEntry) +
+              dictSlots(db->expires) * sizeof(dictEntry*);
+        mh->db[mh->num_dbs].overhead_ht_expires = mem;
+        mem_total+=mem;
+
+        mh->num_dbs++;
+    }
+
+    mh->overhead_total = mem_total;
+    mh->dataset = zmalloc_used - mem_total;
+    mh->peak_perc = (float)zmalloc_used*100/mh->peak_allocated;
+
+    /* Metrics computed after subtracting the startup memory from
+     * the total memory. */
+    size_t net_usage = 1;
+    if (zmalloc_used > mh->startup_allocated)
+        net_usage = zmalloc_used - mh->startup_allocated;
+    mh->dataset_perc = (float)mh->dataset*100/net_usage;
+    mh->bytes_per_key = mh->total_keys ? (net_usage / mh->total_keys) : 0;
+
+    return mh;
+}
+
+/* Helper for "MEMORY allocator-stats", used as a callback for the jemalloc
+ * stats output. */
+void inputCatSds(void *result, const char *str) {
+    /* result is actually a (sds *), so re-cast it here */
+    sds *info = (sds *)result;
+    *info = sdscat(*info, str);
+}
+
+/* This implements MEMORY DOCTOR. An human readable analysis of the Redis
+ * memory condition. */
+sds getMemoryDoctorReport(void) {
+    int empty = 0;          /* Instance is empty or almost empty. */
+    int big_peak = 0;       /* Memory peak is much larger than used mem. */
+    int high_frag = 0;      /* High fragmentation. */
+    int big_slave_buf = 0;  /* Slave buffers are too big. */
+    int big_client_buf = 0; /* Client buffers are too big. */
+    int num_reports = 0;
+    struct redisMemOverhead *mh = getMemoryOverheadData();
+
+    if (mh->total_allocated < (1024*1024*5)) {
+        empty = 1;
+        num_reports++;
+    } else {
+        /* Peak is > 150% of current used memory? */
+        if (((float)mh->peak_allocated / mh->total_allocated) > 1.5) {
+            big_peak = 1;
+            num_reports++;
+        }
+
+        /* Fragmentation is higher than 1.4? */
+        if (mh->fragmentation > 1.4) {
+            high_frag = 1;
+            num_reports++;
+        }
+
+        /* Clients using more than 200k each average? */
+        long numslaves = listLength(server.slaves);
+        long numclients = listLength(server.clients)-numslaves;
+        if (mh->clients_normal / numclients > (1024*200)) {
+            big_client_buf = 1;
+            num_reports++;
+        }
+
+        /* Slaves using more than 10 MB each? */
+        if (mh->clients_slaves / numslaves > (1024*1024*10)) {
+            big_slave_buf = 1;
+            num_reports++;
+        }
+    }
+
+    sds s;
+    if (num_reports == 0) {
+        s = sdsnew(
+        "Hi Sam, I can't find any memory issue in your instance. "
+        "I can only account for what occurs on this base.");
+    } else if (empty == 1) {
+        s = sdsnew(
+        "Hi Sam, this instance is empty or is using very little memory, "
+        "my issues detector can't be used in these conditions. "
+        "Please, leave for your mission on Earth and fill it with some data. "
+        "The new Sam and I will be back to our programming as soon as I "
+        "finished rebooting.");
+    } else {
+        s = sdsnew("Sam, I detected a few issues in this Redis instance memory implants:\n\n");
+        if (big_peak) {
+            s = sdscat(s," * Peak memory: In the past this instance used more than 150% the memory that is currently using. The allocator is normally not able to release memory after a peak, so you can expect to see a big fragmentation ratio, however this is actually harmless and is only due to the memory peak, and if the Redis instance Resident Set Size (RSS) is currently bigger than expected, the memory will be used as soon as you fill the Redis instance with more data. If the memory peak was only occasional and you want to try to reclaim memory, please try the MEMORY PURGE command, otherwise the only other option is to shutdown and restart the instance.\n\n");
+        }
+        if (high_frag) {
+            s = sdscatprintf(s," * High fragmentation: This instance has a memory fragmentation greater than 1.4 (this means that the Resident Set Size of the Redis process is much larger than the sum of the logical allocations Redis performed). This problem is usually due either to a large peak memory (check if there is a peak memory entry above in the report) or may result from a workload that causes the allocator to fragment memory a lot. If the problem is a large peak memory, then there is no issue. Otherwise, make sure you are using the Jemalloc allocator and not the default libc malloc. Note: The currently used allocator is \"%s\".\n\n", ZMALLOC_LIB);
+        }
+        if (big_slave_buf) {
+            s = sdscat(s," * Big slave buffers: The slave output buffers in this instance are greater than 10MB for each slave (on average). This likely means that there is some slave instance that is struggling receiving data, either because it is too slow or because of networking issues. As a result, data piles on the master output buffers. Please try to identify what slave is not receiving data correctly and why. You can use the INFO output in order to check the slaves delays and the CLIENT LIST command to check the output buffers of each slave.\n\n");
+        }
+        if (big_client_buf) {
+            s = sdscat(s," * Big client buffers: The clients output buffers in this instance are greater than 200K per client (on average). This may result from different causes, like Pub/Sub clients subscribed to channels bot not receiving data fast enough, so that data piles on the Redis instance output buffer, or clients sending commands with large replies or very large sequences of commands in the same pipeline. Please use the CLIENT LIST command in order to investigate the issue if it causes problems in your instance, or to understand better why certain clients are using a big amount of memory.\n\n");
+        }
+        s = sdscat(s,"I'm here to keep you safe, Sam. I want to help you.\n");
+    }
+    freeMemoryOverheadData(mh);
+    return s;
+}
+
+/* ======================= The OBJECT and MEMORY commands =================== */
+
 /* This is a helper function for the OBJECT command. We need to lookup keys
  * without any modification of LRU or other parameters. */
 robj *objectCommandLookup(client *c, robj *key) {
@@ -726,9 +1017,156 @@ void objectCommand(client *c) {
     } else if (!strcasecmp(c->argv[1]->ptr,"idletime") && c->argc == 3) {
         if ((o = objectCommandLookupOrReply(c,c->argv[2],shared.nullbulk))
                 == NULL) return;
+        if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) {
+            addReplyError(c,"An LFU maxmemory policy is selected, idle time not tracked. Please note that when switching between policies at runtime LRU and LFU data will take some time to adjust.");
+            return;
+        }
         addReplyLongLong(c,estimateObjectIdleTime(o)/1000);
+    } else if (!strcasecmp(c->argv[1]->ptr,"freq") && c->argc == 3) {
+        if ((o = objectCommandLookupOrReply(c,c->argv[2],shared.nullbulk))
+                == NULL) return;
+        if (server.maxmemory_policy & MAXMEMORY_FLAG_LRU) {
+            addReplyError(c,"An LRU maxmemory policy is selected, access frequency not tracked. Please note that when switching between policies at runtime LRU and LFU data will take some time to adjust.");
+            return;
+        }
+        addReplyLongLong(c,o->lru&255);
     } else {
-        addReplyError(c,"Syntax error. Try OBJECT (refcount|encoding|idletime)");
+        addReplyError(c,"Syntax error. Try OBJECT (refcount|encoding|idletime|freq)");
     }
 }
 
+/* The memory command will eventually be a complete interface for the
+ * memory introspection capabilities of Redis.
+ *
+ * Usage: MEMORY usage <key> */
+void memoryCommand(client *c) {
+    robj *o;
+
+    if (!strcasecmp(c->argv[1]->ptr,"usage") && c->argc >= 3) {
+        long long samples = OBJ_COMPUTE_SIZE_DEF_SAMPLES;
+        for (int j = 3; j < c->argc; j++) {
+            if (!strcasecmp(c->argv[j]->ptr,"samples") &&
+                j+1 < c->argc)
+            {
+                if (getLongLongFromObjectOrReply(c,c->argv[j+1],&samples,NULL)
+                     == C_ERR) return;
+                if (samples < 0) {
+                    addReply(c,shared.syntaxerr);
+                    return;
+                }
+                if (samples == 0) samples = LLONG_MAX;;
+                j++; /* skip option argument. */
+            } else {
+                addReply(c,shared.syntaxerr);
+                return;
+            }
+        }
+        if ((o = objectCommandLookupOrReply(c,c->argv[2],shared.nullbulk))
+                == NULL) return;
+        size_t usage = objectComputeSize(o,samples);
+        usage += sdsAllocSize(c->argv[1]->ptr);
+        usage += sizeof(dictEntry);
+        addReplyLongLong(c,usage);
+    } else if (!strcasecmp(c->argv[1]->ptr,"stats") && c->argc == 2) {
+        struct redisMemOverhead *mh = getMemoryOverheadData();
+
+        addReplyMultiBulkLen(c,(14+mh->num_dbs)*2);
+
+        addReplyBulkCString(c,"peak.allocated");
+        addReplyLongLong(c,mh->peak_allocated);
+
+        addReplyBulkCString(c,"total.allocated");
+        addReplyLongLong(c,mh->total_allocated);
+
+        addReplyBulkCString(c,"startup.allocated");
+        addReplyLongLong(c,mh->startup_allocated);
+
+        addReplyBulkCString(c,"replication.backlog");
+        addReplyLongLong(c,mh->repl_backlog);
+
+        addReplyBulkCString(c,"clients.slaves");
+        addReplyLongLong(c,mh->clients_slaves);
+
+        addReplyBulkCString(c,"clients.normal");
+        addReplyLongLong(c,mh->clients_normal);
+
+        addReplyBulkCString(c,"aof.buffer");
+        addReplyLongLong(c,mh->aof_buffer);
+
+        for (size_t j = 0; j < mh->num_dbs; j++) {
+            char dbname[32];
+            snprintf(dbname,sizeof(dbname),"db.%zd",mh->db[j].dbid);
+            addReplyBulkCString(c,dbname);
+            addReplyMultiBulkLen(c,4);
+
+            addReplyBulkCString(c,"overhead.hashtable.main");
+            addReplyLongLong(c,mh->db[j].overhead_ht_main);
+
+            addReplyBulkCString(c,"overhead.hashtable.expires");
+            addReplyLongLong(c,mh->db[j].overhead_ht_expires);
+        }
+
+        addReplyBulkCString(c,"overhead.total");
+        addReplyLongLong(c,mh->overhead_total);
+
+        addReplyBulkCString(c,"keys.count");
+        addReplyLongLong(c,mh->total_keys);
+
+        addReplyBulkCString(c,"keys.bytes-per-key");
+        addReplyLongLong(c,mh->bytes_per_key);
+
+        addReplyBulkCString(c,"dataset.bytes");
+        addReplyLongLong(c,mh->dataset);
+
+        addReplyBulkCString(c,"dataset.percentage");
+        addReplyDouble(c,mh->dataset_perc);
+
+        addReplyBulkCString(c,"peak.percentage");
+        addReplyDouble(c,mh->peak_perc);
+
+        addReplyBulkCString(c,"fragmentation");
+        addReplyDouble(c,mh->fragmentation);
+
+        freeMemoryOverheadData(mh);
+    } else if (!strcasecmp(c->argv[1]->ptr,"malloc-stats") && c->argc == 2) {
+#if defined(USE_JEMALLOC)
+        sds info = sdsempty();
+        je_malloc_stats_print(inputCatSds, &info, NULL);
+        addReplyBulkSds(c, info);
+#else
+        addReplyBulkCString(c,"Stats not supported for the current allocator");
+#endif
+    } else if (!strcasecmp(c->argv[1]->ptr,"doctor") && c->argc == 2) {
+        sds report = getMemoryDoctorReport();
+        addReplyBulkSds(c,report);
+    } else if (!strcasecmp(c->argv[1]->ptr,"purge") && c->argc == 2) {
+#if defined(USE_JEMALLOC)
+        char tmp[32];
+        unsigned narenas = 0;
+        size_t sz = sizeof(unsigned);
+        if (!je_mallctl("arenas.narenas", &narenas, &sz, NULL, 0)) {
+            sprintf(tmp, "arena.%d.purge", narenas);
+            if (!je_mallctl(tmp, NULL, 0, NULL, 0)) {
+                addReply(c, shared.ok);
+                return;
+            }
+        }
+        addReplyError(c, "Error purging dirty pages");
+#else
+        addReply(c, shared.ok);
+        /* Nothing to do for other allocators. */
+#endif
+    } else if (!strcasecmp(c->argv[1]->ptr,"help") && c->argc == 2) {
+        addReplyMultiBulkLen(c,4);
+        addReplyBulkCString(c,
+"MEMORY USAGE <key> [SAMPLES <count>] - Estimate memory usage of key");
+        addReplyBulkCString(c,
+"MEMORY STATS                         - Show memory usage details");
+        addReplyBulkCString(c,
+"MEMORY PURGE                         - Ask the allocator to release memory");
+        addReplyBulkCString(c,
+"MEMORY MALLOC-STATS                  - Show allocator internal stats");
+    } else {
+        addReplyError(c,"Syntax error. Try MEMORY HELP");
+    }
+}

src/quicklist.c

@@ -149,7 +149,7 @@ REDIS_STATIC quicklistNode *quicklistCreateNode(void) {
 }
 
 /* Return cached quicklist count */
-unsigned int quicklistCount(quicklist *ql) { return ql->count; }
+unsigned int quicklistCount(const quicklist *ql) { return ql->count; }
 
 /* Free entire quicklist. */
 void quicklistRelease(quicklist *quicklist) {
@@ -671,6 +671,7 @@ int quicklistReplaceAtIndex(quicklist *quicklist, long index, void *data,
         /* quicklistIndex provides an uncompressed node */
         entry.node->zl = ziplistDelete(entry.node->zl, &entry.zi);
         entry.node->zl = ziplistInsert(entry.node->zl, entry.zi, data, sz);
+        quicklistNodeUpdateSz(entry.node);
         quicklistCompress(quicklist, entry.node);
         return 1;
     } else {

src/quicklist.h

@@ -92,8 +92,8 @@ typedef struct quicklistEntry {
     quicklistNode *node;
     unsigned char *zi;
     unsigned char *value;
-    unsigned int sz;
     long long longval;
+    unsigned int sz;
     int offset;
 } quicklistEntry;
 
@@ -154,7 +154,7 @@ int quicklistPopCustom(quicklist *quicklist, int where, unsigned char **data,
                        void *(*saver)(unsigned char *data, unsigned int sz));
 int quicklistPop(quicklist *quicklist, int where, unsigned char **data,
                  unsigned int *sz, long long *slong);
-unsigned int quicklistCount(quicklist *ql);
+unsigned int quicklistCount(const quicklist *ql);
 int quicklistCompare(unsigned char *p1, unsigned char *p2, int p2_len);
 size_t quicklistGetLzf(const quicklistNode *node, void **data);
 

src/rdb.c

@@ -39,19 +39,32 @@
 #include <sys/wait.h>
 #include <arpa/inet.h>
 #include <sys/stat.h>
+#include <sys/param.h>
 
-#define RDB_LOAD_NONE   0
-#define RDB_LOAD_ENC    (1<<0)
-#define RDB_LOAD_PLAIN  (1<<1)
-#define RDB_LOAD_SDS    (1<<2)
+#define rdbExitReportCorruptRDB(...) rdbCheckThenExit(__LINE__,__VA_ARGS__)
 
-#define rdbExitReportCorruptRDB(reason) rdbCheckThenExit(reason, __LINE__);
+extern int rdbCheckMode;
+void rdbCheckError(const char *fmt, ...);
+void rdbCheckSetError(const char *fmt, ...);
 
-void rdbCheckThenExit(char *reason, int where) {
-    serverLog(LL_WARNING, "Corrupt RDB detected at rdb.c:%d (%s). "
-        "Running 'redis-check-rdb %s'",
-        where, reason, server.rdb_filename);
-    redis_check_rdb(server.rdb_filename);
+void rdbCheckThenExit(int linenum, char *reason, ...) {
+    va_list ap;
+    char msg[1024];
+    int len;
+
+    len = snprintf(msg,sizeof(msg),
+        "Internal error in RDB reading function at rdb.c:%d -> ", linenum);
+    va_start(ap,reason);
+    vsnprintf(msg+len,sizeof(msg)-len,reason,ap);
+    va_end(ap);
+
+    if (!rdbCheckMode) {
+        serverLog(LL_WARNING, "%s", msg);
+        char *argv[2] = {"",server.rdb_filename};
+        redis_check_rdb_main(2,argv);
+    } else {
+        rdbCheckError("%s",msg);
+    }
     exit(1);
 }
 
@@ -94,7 +107,7 @@ long long rdbLoadMillisecondTime(rio *rdb) {
 /* Saves an encoded length. The first two bits in the first byte are used to
  * hold the encoding type. See the RDB_* definitions for more information
  * on the types of encoding. */
-int rdbSaveLen(rio *rdb, uint32_t len) {
+int rdbSaveLen(rio *rdb, uint64_t len) {
     unsigned char buf[2];
     size_t nwritten;
 
@@ -109,44 +122,79 @@ int rdbSaveLen(rio *rdb, uint32_t len) {
         buf[1] = len&0xFF;
         if (rdbWriteRaw(rdb,buf,2) == -1) return -1;
         nwritten = 2;
-    } else {
+    } else if (len <= UINT32_MAX) {
         /* Save a 32 bit len */
-        buf[0] = (RDB_32BITLEN<<6);
+        buf[0] = RDB_32BITLEN;
         if (rdbWriteRaw(rdb,buf,1) == -1) return -1;
-        len = htonl(len);
-        if (rdbWriteRaw(rdb,&len,4) == -1) return -1;
+        uint32_t len32 = htonl(len);
+        if (rdbWriteRaw(rdb,&len32,4) == -1) return -1;
         nwritten = 1+4;
+    } else {
+        /* Save a 64 bit len */
+        buf[0] = RDB_64BITLEN;
+        if (rdbWriteRaw(rdb,buf,1) == -1) return -1;
+        len = htonu64(len);
+        if (rdbWriteRaw(rdb,&len,8) == -1) return -1;
+        nwritten = 1+8;
     }
     return nwritten;
 }
 
-/* Load an encoded length. The "isencoded" argument is set to 1 if the length
- * is not actually a length but an "encoding type". See the RDB_ENC_*
- * definitions in rdb.h for more information. */
-uint32_t rdbLoadLen(rio *rdb, int *isencoded) {
+
+/* Load an encoded length. If the loaded length is a normal length as stored
+ * with rdbSaveLen(), the read length is set to '*lenptr'. If instead the
+ * loaded length describes a special encoding that follows, then '*isencoded'
+ * is set to 1 and the encoding format is stored at '*lenptr'.
+ *
+ * See the RDB_ENC_* definitions in rdb.h for more information on special
+ * encodings.
+ *
+ * The function returns -1 on error, 0 on success. */
+int rdbLoadLenByRef(rio *rdb, int *isencoded, uint64_t *lenptr) {
     unsigned char buf[2];
-    uint32_t len;
     int type;
 
     if (isencoded) *isencoded = 0;
-    if (rioRead(rdb,buf,1) == 0) return RDB_LENERR;
+    if (rioRead(rdb,buf,1) == 0) return -1;
     type = (buf[0]&0xC0)>>6;
     if (type == RDB_ENCVAL) {
         /* Read a 6 bit encoding type. */
         if (isencoded) *isencoded = 1;
-        return buf[0]&0x3F;
+        *lenptr = buf[0]&0x3F;
     } else if (type == RDB_6BITLEN) {
         /* Read a 6 bit len. */
-        return buf[0]&0x3F;
+        *lenptr = buf[0]&0x3F;
     } else if (type == RDB_14BITLEN) {
         /* Read a 14 bit len. */
-        if (rioRead(rdb,buf+1,1) == 0) return RDB_LENERR;
-        return ((buf[0]&0x3F)<<8)|buf[1];
-    } else {
+        if (rioRead(rdb,buf+1,1) == 0) return -1;
+        *lenptr = ((buf[0]&0x3F)<<8)|buf[1];
+    } else if (buf[0] == RDB_32BITLEN) {
         /* Read a 32 bit len. */
-        if (rioRead(rdb,&len,4) == 0) return RDB_LENERR;
-        return ntohl(len);
+        uint32_t len;
+        if (rioRead(rdb,&len,4) == 0) return -1;
+        *lenptr = ntohl(len);
+    } else if (buf[0] == RDB_64BITLEN) {
+        /* Read a 64 bit len. */
+        uint64_t len;
+        if (rioRead(rdb,&len,8) == 0) return -1;
+        *lenptr = ntohu64(len);
+    } else {
+        rdbExitReportCorruptRDB(
+            "Unknown length encoding %d in rdbLoadLen()",type);
+        return -1; /* Never reached. */
     }
+    return 0;
+}
+
+/* This is like rdbLoadLenByRef() but directly returns the value read
+ * from the RDB stream, signaling an error by returning RDB_LENERR
+ * (since it is a too large count to be applicable in any Redis data
+ * structure). */
+uint64_t rdbLoadLen(rio *rdb, int *isencoded) {
+    uint64_t len;
+
+    if (rdbLoadLenByRef(rdb,isencoded,&len) == -1) return RDB_LENERR;
+    return len;
 }
 
 /* Encodes the "value" argument as integer when it fits in the supported ranges
@@ -178,7 +226,7 @@ int rdbEncodeInteger(long long value, unsigned char *enc) {
 /* Loads an integer-encoded object with the specified encoding type "enctype".
  * The returned value changes according to the flags, see
  * rdbGenerincLoadStringObject() for more info. */
-void *rdbLoadIntegerObject(rio *rdb, int enctype, int flags) {
+void *rdbLoadIntegerObject(rio *rdb, int enctype, int flags, size_t *lenptr) {
     int plain = flags & RDB_LOAD_PLAIN;
     int sds = flags & RDB_LOAD_SDS;
     int encode = flags & RDB_LOAD_ENC;
@@ -200,11 +248,12 @@ void *rdbLoadIntegerObject(rio *rdb, int enctype, int flags) {
         val = (int32_t)v;
     } else {
         val = 0; /* anti-warning */
-        rdbExitReportCorruptRDB("Unknown RDB integer encoding type");
+        rdbExitReportCorruptRDB("Unknown RDB integer encoding type %d",enctype);
     }
     if (plain || sds) {
         char buf[LONG_STR_SIZE], *p;
         int len = ll2string(buf,sizeof(buf),val);
+        if (lenptr) *lenptr = len;
         p = plain ? zmalloc(len) : sdsnewlen(NULL,len);
         memcpy(p,buf,len);
         return p;
@@ -280,10 +329,10 @@ ssize_t rdbSaveLzfStringObject(rio *rdb, unsigned char *s, size_t len) {
 /* Load an LZF compressed string in RDB format. The returned value
  * changes according to 'flags'. For more info check the
  * rdbGenericLoadStringObject() function. */
-void *rdbLoadLzfStringObject(rio *rdb, int flags) {
+void *rdbLoadLzfStringObject(rio *rdb, int flags, size_t *lenptr) {
     int plain = flags & RDB_LOAD_PLAIN;
     int sds = flags & RDB_LOAD_SDS;
-    unsigned int len, clen;
+    uint64_t len, clen;
     unsigned char *c = NULL;
     char *val = NULL;
 
@@ -294,13 +343,17 @@ void *rdbLoadLzfStringObject(rio *rdb, int flags) {
     /* Allocate our target according to the uncompressed size. */
     if (plain) {
         val = zmalloc(len);
+        if (lenptr) *lenptr = len;
     } else {
         val = sdsnewlen(NULL,len);
     }
 
     /* Load the compressed representation and uncompress it to target. */
     if (rioRead(rdb,c,clen) == 0) goto err;
-    if (lzf_decompress(c,clen,val,len) == 0) goto err;
+    if (lzf_decompress(c,clen,val,len) == 0) {
+        if (rdbCheckMode) rdbCheckSetError("Invalid LZF compressed string");
+        goto err;
+    }
     zfree(c);
 
     if (plain || sds) {
@@ -390,15 +443,17 @@ int rdbSaveStringObject(rio *rdb, robj *obj) {
  *               efficient. When this flag is passed the function
  *               no longer guarantees that obj->ptr is an SDS string.
  * RDB_LOAD_PLAIN: Return a plain string allocated with zmalloc()
- *                 instead of a Redis object.
+ *                 instead of a Redis object with an sds in it.
  * RDB_LOAD_SDS: Return an SDS string instead of a Redis object.
+ *
+ * On I/O error NULL is returned.
  */
-void *rdbGenericLoadStringObject(rio *rdb, int flags) {
+void *rdbGenericLoadStringObject(rio *rdb, int flags, size_t *lenptr) {
     int encode = flags & RDB_LOAD_ENC;
     int plain = flags & RDB_LOAD_PLAIN;
     int sds = flags & RDB_LOAD_SDS;
     int isencoded;
-    uint32_t len;
+    uint64_t len;
 
     len = rdbLoadLen(rdb,&isencoded);
     if (isencoded) {
@@ -406,17 +461,18 @@ void *rdbGenericLoadStringObject(rio *rdb, int flags) {
         case RDB_ENC_INT8:
         case RDB_ENC_INT16:
         case RDB_ENC_INT32:
-            return rdbLoadIntegerObject(rdb,len,flags);
+            return rdbLoadIntegerObject(rdb,len,flags,lenptr);
         case RDB_ENC_LZF:
-            return rdbLoadLzfStringObject(rdb,flags);
+            return rdbLoadLzfStringObject(rdb,flags,lenptr);
         default:
-            rdbExitReportCorruptRDB("Unknown RDB encoding type");
+            rdbExitReportCorruptRDB("Unknown RDB string encoding type %d",len);
         }
     }
 
     if (len == RDB_LENERR) return NULL;
     if (plain || sds) {
         void *buf = plain ? zmalloc(len) : sdsnewlen(NULL,len);
+        if (lenptr) *lenptr = len;
         if (len && rioRead(rdb,buf,len) == 0) {
             if (plain)
                 zfree(buf);
@@ -437,11 +493,11 @@ void *rdbGenericLoadStringObject(rio *rdb, int flags) {
 }
 
 robj *rdbLoadStringObject(rio *rdb) {
-    return rdbGenericLoadStringObject(rdb,RDB_LOAD_NONE);
+    return rdbGenericLoadStringObject(rdb,RDB_LOAD_NONE,NULL);
 }
 
 robj *rdbLoadEncodedStringObject(rio *rdb) {
-    return rdbGenericLoadStringObject(rdb,RDB_LOAD_ENC);
+    return rdbGenericLoadStringObject(rdb,RDB_LOAD_ENC,NULL);
 }
 
 /* Save a double value. Doubles are saved as strings prefixed by an unsigned
@@ -504,6 +560,37 @@ int rdbLoadDoubleValue(rio *rdb, double *val) {
     }
 }
 
+/* Saves a double for RDB 8 or greater, where IE754 binary64 format is assumed.
+ * We just make sure the integer is always stored in little endian, otherwise
+ * the value is copied verbatim from memory to disk.
+ *
+ * Return -1 on error, the size of the serialized value on success. */
+int rdbSaveBinaryDoubleValue(rio *rdb, double val) {
+    memrev64ifbe(&val);
+    return rdbWriteRaw(rdb,&val,sizeof(val));
+}
+
+/* Loads a double from RDB 8 or greater. See rdbSaveBinaryDoubleValue() for
+ * more info. On error -1 is returned, otherwise 0. */
+int rdbLoadBinaryDoubleValue(rio *rdb, double *val) {
+    if (rioRead(rdb,val,sizeof(*val)) == 0) return -1;
+    memrev64ifbe(val);
+    return 0;
+}
+
+/* Like rdbSaveBinaryDoubleValue() but single precision. */
+int rdbSaveBinaryFloatValue(rio *rdb, float val) {
+    memrev32ifbe(&val);
+    return rdbWriteRaw(rdb,&val,sizeof(val));
+}
+
+/* Like rdbLoadBinaryDoubleValue() but single precision. */
+int rdbLoadBinaryFloatValue(rio *rdb, float *val) {
+    if (rioRead(rdb,val,sizeof(*val)) == 0) return -1;
+    memrev32ifbe(val);
+    return 0;
+}
+
 /* Save the object type of object "o". */
 int rdbSaveObjectType(rio *rdb, robj *o) {
     switch (o->type) {
@@ -525,7 +612,7 @@ int rdbSaveObjectType(rio *rdb, robj *o) {
         if (o->encoding == OBJ_ENCODING_ZIPLIST)
             return rdbSaveType(rdb,RDB_TYPE_ZSET_ZIPLIST);
         else if (o->encoding == OBJ_ENCODING_SKIPLIST)
-            return rdbSaveType(rdb,RDB_TYPE_ZSET);
+            return rdbSaveType(rdb,RDB_TYPE_ZSET_2);
         else
             serverPanic("Unknown sorted set encoding");
     case OBJ_HASH:
@@ -535,6 +622,8 @@ int rdbSaveObjectType(rio *rdb, robj *o) {
             return rdbSaveType(rdb,RDB_TYPE_HASH);
         else
             serverPanic("Unknown hash encoding");
+    case OBJ_MODULE:
+        return rdbSaveType(rdb,RDB_TYPE_MODULE);
     default:
         serverPanic("Unknown object type");
     }
@@ -628,7 +717,7 @@ ssize_t rdbSaveObject(rio *rdb, robj *o) {
                 if ((n = rdbSaveRawString(rdb,(unsigned char*)ele,sdslen(ele)))
                     == -1) return -1;
                 nwritten += n;
-                if ((n = rdbSaveDoubleValue(rdb,*score)) == -1) return -1;
+                if ((n = rdbSaveBinaryDoubleValue(rdb,*score)) == -1) return -1;
                 nwritten += n;
             }
             dictReleaseIterator(di);
@@ -666,6 +755,26 @@ ssize_t rdbSaveObject(rio *rdb, robj *o) {
             serverPanic("Unknown hash encoding");
         }
 
+    } else if (o->type == OBJ_MODULE) {
+        /* Save a module-specific value. */
+        RedisModuleIO io;
+        moduleValue *mv = o->ptr;
+        moduleType *mt = mv->type;
+        moduleInitIOContext(io,mt,rdb);
+
+        /* Write the "module" identifier as prefix, so that we'll be able
+         * to call the right module during loading. */
+        int retval = rdbSaveLen(rdb,mt->id);
+        if (retval == -1) return -1;
+        io.bytes += retval;
+
+        /* Then write the module-specific representation. */
+        mt->rdb_save(&io,mv->value);
+        if (io.ctx) {
+            moduleFreeContext(io.ctx);
+            zfree(io.ctx);
+        }
+        return io.error ? -1 : (ssize_t)io.bytes;
     } else {
         serverPanic("Unknown object type");
     }
@@ -726,14 +835,16 @@ int rdbSaveAuxFieldStrInt(rio *rdb, char *key, long long val) {
 }
 
 /* Save a few default AUX fields with information about the RDB generated. */
-int rdbSaveInfoAuxFields(rio *rdb) {
+int rdbSaveInfoAuxFields(rio *rdb, int flags) {
     int redis_bits = (sizeof(void*) == 8) ? 64 : 32;
+    int aof_preamble = (flags & RDB_SAVE_AOF_PREAMBLE) != 0;
 
     /* Add a few fields about the state when the RDB was created. */
     if (rdbSaveAuxFieldStrStr(rdb,"redis-ver",REDIS_VERSION) == -1) return -1;
     if (rdbSaveAuxFieldStrInt(rdb,"redis-bits",redis_bits) == -1) return -1;
     if (rdbSaveAuxFieldStrInt(rdb,"ctime",time(NULL)) == -1) return -1;
     if (rdbSaveAuxFieldStrInt(rdb,"used-mem",zmalloc_used_memory()) == -1) return -1;
+    if (rdbSaveAuxFieldStrInt(rdb,"aof-preamble",aof_preamble) == -1) return -1;
     return 1;
 }
 
@@ -745,19 +856,20 @@ int rdbSaveInfoAuxFields(rio *rdb) {
  * When the function returns C_ERR and if 'error' is not NULL, the
  * integer pointed by 'error' is set to the value of errno just after the I/O
  * error. */
-int rdbSaveRio(rio *rdb, int *error) {
+int rdbSaveRio(rio *rdb, int *error, int flags) {
     dictIterator *di = NULL;
     dictEntry *de;
     char magic[10];
     int j;
     long long now = mstime();
     uint64_t cksum;
+    size_t processed = 0;
 
     if (server.rdb_checksum)
         rdb->update_cksum = rioGenericUpdateChecksum;
     snprintf(magic,sizeof(magic),"REDIS%04d",RDB_VERSION);
     if (rdbWriteRaw(rdb,magic,9) == -1) goto werr;
-    if (rdbSaveInfoAuxFields(rdb) == -1) goto werr;
+    if (rdbSaveInfoAuxFields(rdb,flags) == -1) goto werr;
 
     for (j = 0; j < server.dbnum; j++) {
         redisDb *db = server.db+j;
@@ -778,7 +890,7 @@ int rdbSaveRio(rio *rdb, int *error) {
         db_size = (dictSize(db->dict) <= UINT32_MAX) ?
                                 dictSize(db->dict) :
                                 UINT32_MAX;
-        expires_size = (dictSize(db->dict) <= UINT32_MAX) ?
+        expires_size = (dictSize(db->expires) <= UINT32_MAX) ?
                                 dictSize(db->expires) :
                                 UINT32_MAX;
         if (rdbSaveType(rdb,RDB_OPCODE_RESIZEDB) == -1) goto werr;
@@ -794,6 +906,16 @@ int rdbSaveRio(rio *rdb, int *error) {
             initStaticStringObject(key,keystr);
             expire = getExpire(db,&key);
             if (rdbSaveKeyValuePair(rdb,&key,o,expire,now) == -1) goto werr;
+
+            /* When this RDB is produced as part of an AOF rewrite, move
+             * accumulated diff from parent to child while rewriting in
+             * order to have a smaller final write. */
+            if (flags & RDB_SAVE_AOF_PREAMBLE &&
+                rdb->processed_bytes > processed+AOF_READ_DIFF_INTERVAL_BYTES)
+            {
+                processed = rdb->processed_bytes;
+                aofReadDiffFromParent();
+            }
         }
         dictReleaseIterator(di);
     }
@@ -831,7 +953,7 @@ int rdbSaveRioWithEOFMark(rio *rdb, int *error) {
     if (rioWrite(rdb,"$EOF:",5) == 0) goto werr;
     if (rioWrite(rdb,eofmark,RDB_EOF_MARK_SIZE) == 0) goto werr;
     if (rioWrite(rdb,"\r\n",2) == 0) goto werr;
-    if (rdbSaveRio(rdb,error) == C_ERR) goto werr;
+    if (rdbSaveRio(rdb,error,RDB_SAVE_NONE) == C_ERR) goto werr;
     if (rioWrite(rdb,eofmark,RDB_EOF_MARK_SIZE) == 0) goto werr;
     return C_OK;
 
@@ -844,6 +966,7 @@ werr: /* Write error. */
 /* Save the DB on disk. Return C_ERR on error, C_OK on success. */
 int rdbSave(char *filename) {
     char tmpfile[256];
+    char cwd[MAXPATHLEN]; /* Current working dir path for error messages. */
     FILE *fp;
     rio rdb;
     int error = 0;
@@ -851,13 +974,18 @@ int rdbSave(char *filename) {
     snprintf(tmpfile,256,"temp-%d.rdb", (int) getpid());
     fp = fopen(tmpfile,"w");
     if (!fp) {
-        serverLog(LL_WARNING, "Failed opening .rdb for saving: %s",
+        char *cwdp = getcwd(cwd,MAXPATHLEN);
+        serverLog(LL_WARNING,
+            "Failed opening the RDB file %s (in server root dir %s) "
+            "for saving: %s",
+            filename,
+            cwdp ? cwdp : "unknown",
             strerror(errno));
         return C_ERR;
     }
 
     rioInitWithFile(&rdb,fp);
-    if (rdbSaveRio(&rdb,&error) == C_ERR) {
+    if (rdbSaveRio(&rdb,&error,RDB_SAVE_NONE) == C_ERR) {
         errno = error;
         goto werr;
     }
@@ -870,10 +998,18 @@ int rdbSave(char *filename) {
     /* Use RENAME to make sure the DB file is changed atomically only
      * if the generate DB file is ok. */
     if (rename(tmpfile,filename) == -1) {
-        serverLog(LL_WARNING,"Error moving temp DB file on the final destination: %s", strerror(errno));
+        char *cwdp = getcwd(cwd,MAXPATHLEN);
+        serverLog(LL_WARNING,
+            "Error moving temp DB file %s on the final "
+            "destination %s (in server root dir %s): %s",
+            tmpfile,
+            filename,
+            cwdp ? cwdp : "unknown",
+            strerror(errno));
         unlink(tmpfile);
         return C_ERR;
     }
+
     serverLog(LL_NOTICE,"DB saved on disk");
     server.dirty = 0;
     server.lastsave = time(NULL);
@@ -891,10 +1027,11 @@ int rdbSaveBackground(char *filename) {
     pid_t childpid;
     long long start;
 
-    if (server.rdb_child_pid != -1) return C_ERR;
+    if (server.aof_child_pid != -1 || server.rdb_child_pid != -1) return C_ERR;
 
     server.dirty_before_bgsave = server.dirty;
     server.lastbgsave_try = time(NULL);
+    openChildInfoPipe();
 
     start = ustime();
     if ((childpid = fork()) == 0) {
@@ -905,13 +1042,16 @@ int rdbSaveBackground(char *filename) {
         redisSetProcTitle("redis-rdb-bgsave");
         retval = rdbSave(filename);
         if (retval == C_OK) {
-            size_t private_dirty = zmalloc_get_private_dirty();
+            size_t private_dirty = zmalloc_get_private_dirty(-1);
 
             if (private_dirty) {
                 serverLog(LL_NOTICE,
                     "RDB: %zu MB of memory used by copy-on-write",
                     private_dirty/(1024*1024));
             }
+
+            server.child_info_data.cow_size = private_dirty;
+            sendChildInfo(CHILD_INFO_TYPE_RDB);
         }
         exitFromChild((retval == C_OK) ? 0 : 1);
     } else {
@@ -920,6 +1060,7 @@ int rdbSaveBackground(char *filename) {
         server.stat_fork_rate = (double) zmalloc_used_memory() * 1000000 / server.stat_fork_time / (1024*1024*1024); /* GB per second. */
         latencyAddSampleIfNeeded("fork",server.stat_fork_time/1000);
         if (childpid == -1) {
+            closeChildInfoPipe();
             server.lastbgsave_status = C_ERR;
             serverLog(LL_WARNING,"Can't save in background: fork: %s",
                 strerror(errno));
@@ -946,7 +1087,7 @@ void rdbRemoveTempFile(pid_t childpid) {
  * On success a newly allocated object is returned, otherwise NULL. */
 robj *rdbLoadObject(int rdbtype, rio *rdb) {
     robj *o = NULL, *ele, *dec;
-    size_t len;
+    uint64_t len;
     unsigned int i;
 
     if (rdbtype == RDB_TYPE_STRING) {
@@ -990,8 +1131,8 @@ robj *rdbLoadObject(int rdbtype, rio *rdb) {
             long long llval;
             sds sdsele;
 
-            if ((sdsele = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS)) == NULL)
-                return NULL;
+            if ((sdsele = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS,NULL))
+                == NULL) return NULL;
 
             if (o->encoding == OBJ_ENCODING_INTSET) {
                 /* Fetch integer value from element. */
@@ -1011,9 +1152,9 @@ robj *rdbLoadObject(int rdbtype, rio *rdb) {
                 sdsfree(sdsele);
             }
         }
-    } else if (rdbtype == RDB_TYPE_ZSET) {
+    } else if (rdbtype == RDB_TYPE_ZSET_2 || rdbtype == RDB_TYPE_ZSET) {
         /* Read list/set value. */
-        size_t zsetlen;
+        uint64_t zsetlen;
         size_t maxelelen = 0;
         zset *zs;
 
@@ -1027,9 +1168,14 @@ robj *rdbLoadObject(int rdbtype, rio *rdb) {
             double score;
             zskiplistNode *znode;
 
-            if ((sdsele = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS)) == NULL)
-                return NULL;
-            if (rdbLoadDoubleValue(rdb,&score) == -1) return NULL;
+            if ((sdsele = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS,NULL))
+                == NULL) return NULL;
+
+            if (rdbtype == RDB_TYPE_ZSET_2) {
+                if (rdbLoadBinaryDoubleValue(rdb,&score) == -1) return NULL;
+            } else {
+                if (rdbLoadDoubleValue(rdb,&score) == -1) return NULL;
+            }
 
             /* Don't care about integer-encoded strings. */
             if (sdslen(sdsele) > maxelelen) maxelelen = sdslen(sdsele);
@@ -1043,7 +1189,7 @@ robj *rdbLoadObject(int rdbtype, rio *rdb) {
             maxelelen <= server.zset_max_ziplist_value)
                 zsetConvert(o,OBJ_ENCODING_ZIPLIST);
     } else if (rdbtype == RDB_TYPE_HASH) {
-        size_t len;
+        uint64_t len;
         int ret;
         sds field, value;
 
@@ -1060,10 +1206,10 @@ robj *rdbLoadObject(int rdbtype, rio *rdb) {
         while (o->encoding == OBJ_ENCODING_ZIPLIST && len > 0) {
             len--;
             /* Load raw strings */
-            if ((field = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS)) == NULL)
-                return NULL;
-            if ((value = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS)) == NULL)
-                return NULL;
+            if ((field = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS,NULL))
+                == NULL) return NULL;
+            if ((value = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS,NULL))
+                == NULL) return NULL;
 
             /* Add pair to ziplist */
             o->ptr = ziplistPush(o->ptr, (unsigned char*)field,
@@ -1088,10 +1234,10 @@ robj *rdbLoadObject(int rdbtype, rio *rdb) {
         while (o->encoding == OBJ_ENCODING_HT && len > 0) {
             len--;
             /* Load encoded strings */
-            if ((field = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS)) == NULL)
-                return NULL;
-            if ((value = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS)) == NULL)
-                return NULL;
+            if ((field = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS,NULL))
+                == NULL) return NULL;
+            if ((value = rdbGenericLoadStringObject(rdb,RDB_LOAD_SDS,NULL))
+                == NULL) return NULL;
 
             /* Add pair to hash table */
             ret = dictAdd((dict*)o->ptr, field, value);
@@ -1109,7 +1255,8 @@ robj *rdbLoadObject(int rdbtype, rio *rdb) {
                             server.list_compress_depth);
 
         while (len--) {
-            unsigned char *zl = rdbGenericLoadStringObject(rdb,RDB_LOAD_PLAIN);
+            unsigned char *zl =
+                rdbGenericLoadStringObject(rdb,RDB_LOAD_PLAIN,NULL);
             if (zl == NULL) return NULL;
             quicklistAppendZiplist(o->ptr, zl);
         }
@@ -1119,7 +1266,8 @@ robj *rdbLoadObject(int rdbtype, rio *rdb) {
                rdbtype == RDB_TYPE_ZSET_ZIPLIST ||
                rdbtype == RDB_TYPE_HASH_ZIPLIST)
     {
-        unsigned char *encoded = rdbGenericLoadStringObject(rdb,RDB_LOAD_PLAIN);
+        unsigned char *encoded =
+            rdbGenericLoadStringObject(rdb,RDB_LOAD_PLAIN,NULL);
         if (encoded == NULL) return NULL;
         o = createObject(OBJ_STRING,encoded); /* Obj type fixed below. */
 
@@ -1183,11 +1331,32 @@ robj *rdbLoadObject(int rdbtype, rio *rdb) {
                     hashTypeConvert(o, OBJ_ENCODING_HT);
                 break;
             default:
-                rdbExitReportCorruptRDB("Unknown encoding");
+                rdbExitReportCorruptRDB("Unknown RDB encoding type %d",rdbtype);
                 break;
         }
+    } else if (rdbtype == RDB_TYPE_MODULE) {
+        uint64_t moduleid = rdbLoadLen(rdb,NULL);
+        moduleType *mt = moduleTypeLookupModuleByID(moduleid);
+        char name[10];
+
+        if (mt == NULL) {
+            moduleTypeNameByID(name,moduleid);
+            serverLog(LL_WARNING,"The RDB file contains module data I can't load: no matching module '%s'", name);
+            exit(1);
+        }
+        RedisModuleIO io;
+        moduleInitIOContext(io,mt,rdb);
+        /* Call the rdb_load method of the module providing the 10 bit
+         * encoding version in the lower 10 bits of the module ID. */
+        void *ptr = mt->rdb_load(&io,moduleid&1023);
+        if (ptr == NULL) {
+            moduleTypeNameByID(name,moduleid);
+            serverLog(LL_WARNING,"The RDB file contains module data for the module type '%s', that the responsible module is not able to load. Check for modules log above for additional clues.", name);
+            exit(1);
+        }
+        o = createModuleObject(mt,ptr);
     } else {
-        rdbExitReportCorruptRDB("Unknown object type");
+        rdbExitReportCorruptRDB("Unknown RDB encoding type %d",rdbtype);
     }
     return o;
 }
@@ -1239,67 +1408,61 @@ void rdbLoadProgressCallback(rio *r, const void *buf, size_t len) {
     }
 }
 
-int rdbLoad(char *filename) {
-    uint32_t dbid;
+/* Load an RDB file from the rio stream 'rdb'. On success C_OK is returned,
+ * otherwise C_ERR is returned and 'errno' is set accordingly. */
+int rdbLoadRio(rio *rdb) {
+    uint64_t dbid;
     int type, rdbver;
     redisDb *db = server.db+0;
     char buf[1024];
     long long expiretime, now = mstime();
-    FILE *fp;
-    rio rdb;
 
-    if ((fp = fopen(filename,"r")) == NULL) return C_ERR;
-
-    rioInitWithFile(&rdb,fp);
-    rdb.update_cksum = rdbLoadProgressCallback;
-    rdb.max_processing_chunk = server.loading_process_events_interval_bytes;
-    if (rioRead(&rdb,buf,9) == 0) goto eoferr;
+    rdb->update_cksum = rdbLoadProgressCallback;
+    rdb->max_processing_chunk = server.loading_process_events_interval_bytes;
+    if (rioRead(rdb,buf,9) == 0) goto eoferr;
     buf[9] = '\0';
     if (memcmp(buf,"REDIS",5) != 0) {
-        fclose(fp);
         serverLog(LL_WARNING,"Wrong signature trying to load DB from file");
         errno = EINVAL;
         return C_ERR;
     }
     rdbver = atoi(buf+5);
     if (rdbver < 1 || rdbver > RDB_VERSION) {
-        fclose(fp);
         serverLog(LL_WARNING,"Can't handle RDB format version %d",rdbver);
         errno = EINVAL;
         return C_ERR;
     }
 
-    startLoading(fp);
     while(1) {
         robj *key, *val;
         expiretime = -1;
 
         /* Read type. */
-        if ((type = rdbLoadType(&rdb)) == -1) goto eoferr;
+        if ((type = rdbLoadType(rdb)) == -1) goto eoferr;
 
         /* Handle special types. */
         if (type == RDB_OPCODE_EXPIRETIME) {
             /* EXPIRETIME: load an expire associated with the next key
              * to load. Note that after loading an expire we need to
              * load the actual type, and continue. */
-            if ((expiretime = rdbLoadTime(&rdb)) == -1) goto eoferr;
+            if ((expiretime = rdbLoadTime(rdb)) == -1) goto eoferr;
             /* We read the time so we need to read the object type again. */
-            if ((type = rdbLoadType(&rdb)) == -1) goto eoferr;
+            if ((type = rdbLoadType(rdb)) == -1) goto eoferr;
             /* the EXPIRETIME opcode specifies time in seconds, so convert
              * into milliseconds. */
             expiretime *= 1000;
         } else if (type == RDB_OPCODE_EXPIRETIME_MS) {
             /* EXPIRETIME_MS: milliseconds precision expire times introduced
              * with RDB v3. Like EXPIRETIME but no with more precision. */
-            if ((expiretime = rdbLoadMillisecondTime(&rdb)) == -1) goto eoferr;
+            if ((expiretime = rdbLoadMillisecondTime(rdb)) == -1) goto eoferr;
             /* We read the time so we need to read the object type again. */
-            if ((type = rdbLoadType(&rdb)) == -1) goto eoferr;
+            if ((type = rdbLoadType(rdb)) == -1) goto eoferr;
         } else if (type == RDB_OPCODE_EOF) {
             /* EOF: End of file, exit the main loop. */
             break;
         } else if (type == RDB_OPCODE_SELECTDB) {
             /* SELECTDB: Select the specified database. */
-            if ((dbid = rdbLoadLen(&rdb,NULL)) == RDB_LENERR)
+            if ((dbid = rdbLoadLen(rdb,NULL)) == RDB_LENERR)
                 goto eoferr;
             if (dbid >= (unsigned)server.dbnum) {
                 serverLog(LL_WARNING,
@@ -1313,10 +1476,10 @@ int rdbLoad(char *filename) {
         } else if (type == RDB_OPCODE_RESIZEDB) {
             /* RESIZEDB: Hint about the size of the keys in the currently
              * selected data base, in order to avoid useless rehashing. */
-            uint32_t db_size, expires_size;
-            if ((db_size = rdbLoadLen(&rdb,NULL)) == RDB_LENERR)
+            uint64_t db_size, expires_size;
+            if ((db_size = rdbLoadLen(rdb,NULL)) == RDB_LENERR)
                 goto eoferr;
-            if ((expires_size = rdbLoadLen(&rdb,NULL)) == RDB_LENERR)
+            if ((expires_size = rdbLoadLen(rdb,NULL)) == RDB_LENERR)
                 goto eoferr;
             dictExpand(db->dict,db_size);
             dictExpand(db->expires,expires_size);
@@ -1328,8 +1491,8 @@ int rdbLoad(char *filename) {
              *
              * An AUX field is composed of two strings: key and value. */
             robj *auxkey, *auxval;
-            if ((auxkey = rdbLoadStringObject(&rdb)) == NULL) goto eoferr;
-            if ((auxval = rdbLoadStringObject(&rdb)) == NULL) goto eoferr;
+            if ((auxkey = rdbLoadStringObject(rdb)) == NULL) goto eoferr;
+            if ((auxval = rdbLoadStringObject(rdb)) == NULL) goto eoferr;
 
             if (((char*)auxkey->ptr)[0] == '%') {
                 /* All the fields with a name staring with '%' are considered
@@ -1351,9 +1514,9 @@ int rdbLoad(char *filename) {
         }
 
         /* Read key */
-        if ((key = rdbLoadStringObject(&rdb)) == NULL) goto eoferr;
+        if ((key = rdbLoadStringObject(rdb)) == NULL) goto eoferr;
         /* Read value */
-        if ((val = rdbLoadObject(type,&rdb)) == NULL) goto eoferr;
+        if ((val = rdbLoadObject(type,rdb)) == NULL) goto eoferr;
         /* Check if the key already expired. This function is used when loading
          * an RDB file from disk, either at startup, or when an RDB was
          * received from the master. In the latter case, the master is
@@ -1374,9 +1537,9 @@ int rdbLoad(char *filename) {
     }
     /* Verify the checksum if RDB version is >= 5 */
     if (rdbver >= 5 && server.rdb_checksum) {
-        uint64_t cksum, expected = rdb.cksum;
+        uint64_t cksum, expected = rdb->cksum;
 
-        if (rioRead(&rdb,&cksum,8) == 0) goto eoferr;
+        if (rioRead(rdb,&cksum,8) == 0) goto eoferr;
         memrev64ifbe(&cksum);
         if (cksum == 0) {
             serverLog(LL_WARNING,"RDB file was saved with checksum disabled: no check performed.");
@@ -1385,9 +1548,6 @@ int rdbLoad(char *filename) {
             rdbExitReportCorruptRDB("RDB CRC error");
         }
     }
-
-    fclose(fp);
-    stopLoading();
     return C_OK;
 
 eoferr: /* unexpected end of file is handled here with a fatal exit */
@@ -1396,6 +1556,24 @@ eoferr: /* unexpected end of file is handled here with a fatal exit */
     return C_ERR; /* Just to avoid warning */
 }
 
+/* Like rdbLoadRio() but takes a filename instead of a rio stream. The
+ * filename is open for reading and a rio stream object created in order
+ * to do the actual loading. Moreover the ETA displayed in the INFO
+ * output is initialized and finalized. */
+int rdbLoad(char *filename) {
+    FILE *fp;
+    rio rdb;
+    int retval;
+
+    if ((fp = fopen(filename,"r")) == NULL) return C_ERR;
+    startLoading(fp);
+    rioInitWithFile(&rdb,fp);
+    retval = rdbLoadRio(&rdb);
+    fclose(fp);
+    stopLoading();
+    return retval;
+}
+
 /* A background saving child (BGSAVE) terminated its work. Handle this.
  * This function covers the case of actual BGSAVEs. */
 void backgroundSaveDoneHandlerDisk(int exitcode, int bysignal) {
@@ -1553,7 +1731,7 @@ int rdbSaveToSlavesSockets(void) {
     long long start;
     int pipefds[2];
 
-    if (server.rdb_child_pid != -1) return C_ERR;
+    if (server.aof_child_pid != -1 || server.rdb_child_pid != -1) return C_ERR;
 
     /* Before to fork, create a pipe that will be used in order to
      * send back to the parent the IDs of the slaves that successfully
@@ -1579,7 +1757,7 @@ int rdbSaveToSlavesSockets(void) {
             clientids[numfds] = slave->id;
             fds[numfds++] = slave->fd;
             replicationSetupSlaveForFullResync(slave,getPsyncInitialOffset());
-            /* Put the socket in non-blocking mode to simplify RDB transfer.
+            /* Put the socket in blocking mode to simplify RDB transfer.
              * We'll restore it when the children returns (since duped socket
              * will share the O_NONBLOCK attribute with the parent). */
             anetBlock(NULL,slave->fd);
@@ -1588,6 +1766,7 @@ int rdbSaveToSlavesSockets(void) {
     }
 
     /* Create the child process. */
+    openChildInfoPipe();
     start = ustime();
     if ((childpid = fork()) == 0) {
         /* Child */
@@ -1605,7 +1784,7 @@ int rdbSaveToSlavesSockets(void) {
             retval = C_ERR;
 
         if (retval == C_OK) {
-            size_t private_dirty = zmalloc_get_private_dirty();
+            size_t private_dirty = zmalloc_get_private_dirty(-1);
 
             if (private_dirty) {
                 serverLog(LL_NOTICE,
@@ -1613,6 +1792,9 @@ int rdbSaveToSlavesSockets(void) {
                     private_dirty/(1024*1024));
             }
 
+            server.child_info_data.cow_size = private_dirty;
+            sendChildInfo(CHILD_INFO_TYPE_RDB);
+
             /* If we are returning OK, at least one slave was served
              * with the RDB file as expected, so we need to send a report
              * to the parent via the pipe. The format of the message is:
@@ -1653,6 +1835,7 @@ int rdbSaveToSlavesSockets(void) {
             zfree(msg);
         }
         zfree(clientids);
+        rioFreeFdset(&slave_sockets);
         exitFromChild((retval == C_OK) ? 0 : 1);
     } else {
         /* Parent */
@@ -1680,6 +1863,7 @@ int rdbSaveToSlavesSockets(void) {
             }
             close(pipefds[0]);
             close(pipefds[1]);
+            closeChildInfoPipe();
         } else {
             serverLog(LL_NOTICE,"Background RDB transfer started by pid %d",
                 childpid);
@@ -1707,11 +1891,33 @@ void saveCommand(client *c) {
     }
 }
 
+/* BGSAVE [SCHEDULE] */
 void bgsaveCommand(client *c) {
+    int schedule = 0;
+
+    /* The SCHEDULE option changes the behavior of BGSAVE when an AOF rewrite
+     * is in progress. Instead of returning an error a BGSAVE gets scheduled. */
+    if (c->argc > 1) {
+        if (c->argc == 2 && !strcasecmp(c->argv[1]->ptr,"schedule")) {
+            schedule = 1;
+        } else {
+            addReply(c,shared.syntaxerr);
+            return;
+        }
+    }
+
     if (server.rdb_child_pid != -1) {
         addReplyError(c,"Background save already in progress");
     } else if (server.aof_child_pid != -1) {
-        addReplyError(c,"Can't BGSAVE while AOF log rewriting is in progress");
+        if (schedule) {
+            server.rdb_bgsave_scheduled = 1;
+            addReplyStatus(c,"Background saving scheduled");
+        } else {
+            addReplyError(c,
+                "An AOF log rewriting in progress: can't BGSAVE right now. "
+                "Use BGSAVE SCHEDULE in order to schedule a BGSAVE whenver "
+                "possible.");
+        }
     } else if (rdbSaveBackground(server.rdb_filename) == C_OK) {
         addReplyStatus(c,"Background saving started");
     } else {

src/rdb.h

@@ -38,16 +38,17 @@
 
 /* The current RDB version. When the format changes in a way that is no longer
  * backward compatible this number gets incremented. */
-#define RDB_VERSION 7
+#define RDB_VERSION 8
 
 /* Defines related to the dump file format. To store 32 bits lengths for short
  * keys requires a lot of space, so we check the most significant 2 bits of
  * the first byte to interpreter the length:
  *
- * 00|000000 => if the two MSB are 00 the len is the 6 bits of this byte
- * 01|000000 00000000 =>  01, the len is 14 byes, 6 bits + 8 bits of next byte
- * 10|000000 [32 bit integer] => if it's 01, a full 32 bit len will follow
- * 11|000000 this means: specially encoded object will follow. The six bits
+ * 00|XXXXXX => if the two MSB are 00 the len is the 6 bits of this byte
+ * 01|XXXXXX XXXXXXXX =>  01, the len is 14 byes, 6 bits + 8 bits of next byte
+ * 10|000000 [32 bit integer] => A full 32 bit len in net byte order will follow
+ * 10|000001 [64 bit integer] => A full 64 bit len in net byte order will follow
+ * 11|OBKIND this means: specially encoded object will follow. The six bits
  *           number specify the kind of object that follows.
  *           See the RDB_ENC_* defines.
  *
@@ -55,12 +56,13 @@
  * values, will fit inside. */
 #define RDB_6BITLEN 0
 #define RDB_14BITLEN 1
-#define RDB_32BITLEN 2
+#define RDB_32BITLEN 0x80
+#define RDB_64BITLEN 0x81
 #define RDB_ENCVAL 3
-#define RDB_LENERR UINT_MAX
+#define RDB_LENERR UINT64_MAX
 
 /* When a length of a string object stored on disk has the first two bits
- * set, the remaining two bits specify a special encoding for the object
+ * set, the remaining six bits specify a special encoding for the object
  * accordingly to the following defines: */
 #define RDB_ENC_INT8 0        /* 8 bit signed integer */
 #define RDB_ENC_INT16 1       /* 16 bit signed integer */
@@ -74,6 +76,8 @@
 #define RDB_TYPE_SET    2
 #define RDB_TYPE_ZSET   3
 #define RDB_TYPE_HASH   4
+#define RDB_TYPE_ZSET_2 5 /* ZSET version 2 with doubles stored in binary. */
+#define RDB_TYPE_MODULE 6
 /* NOTE: WHEN ADDING NEW RDB TYPE, UPDATE rdbIsObjectType() BELOW */
 
 /* Object types for encoded objects. */
@@ -86,7 +90,7 @@
 /* NOTE: WHEN ADDING NEW RDB TYPE, UPDATE rdbIsObjectType() BELOW */
 
 /* Test if a type is an object type. */
-#define rdbIsObjectType(t) ((t >= 0 && t <= 4) || (t >= 9 && t <= 14))
+#define rdbIsObjectType(t) ((t >= 0 && t <= 6) || (t >= 9 && t <= 14))
 
 /* Special RDB opcodes (saved/loaded with rdbSaveType/rdbLoadType). */
 #define RDB_OPCODE_AUX        250
@@ -96,12 +100,22 @@
 #define RDB_OPCODE_SELECTDB   254
 #define RDB_OPCODE_EOF        255
 
+/* rdbLoad...() functions flags. */
+#define RDB_LOAD_NONE   0
+#define RDB_LOAD_ENC    (1<<0)
+#define RDB_LOAD_PLAIN  (1<<1)
+#define RDB_LOAD_SDS    (1<<2)
+
+#define RDB_SAVE_NONE 0
+#define RDB_SAVE_AOF_PREAMBLE (1<<0)
+
 int rdbSaveType(rio *rdb, unsigned char type);
 int rdbLoadType(rio *rdb);
 int rdbSaveTime(rio *rdb, time_t t);
 time_t rdbLoadTime(rio *rdb);
-int rdbSaveLen(rio *rdb, uint32_t len);
-uint32_t rdbLoadLen(rio *rdb, int *isencoded);
+int rdbSaveLen(rio *rdb, uint64_t len);
+uint64_t rdbLoadLen(rio *rdb, int *isencoded);
+int rdbLoadLenByRef(rio *rdb, int *isencoded, uint64_t *lenptr);
 int rdbSaveObjectType(rio *rdb, robj *o);
 int rdbLoadObjectType(rio *rdb);
 int rdbLoad(char *filename);
@@ -115,5 +129,13 @@ robj *rdbLoadObject(int type, rio *rdb);
 void backgroundSaveDoneHandler(int exitcode, int bysignal);
 int rdbSaveKeyValuePair(rio *rdb, robj *key, robj *val, long long expiretime, long long now);
 robj *rdbLoadStringObject(rio *rdb);
+int rdbSaveStringObject(rio *rdb, robj *obj);
+ssize_t rdbSaveRawString(rio *rdb, unsigned char *s, size_t len);
+void *rdbGenericLoadStringObject(rio *rdb, int flags, size_t *lenptr);
+int rdbSaveBinaryDoubleValue(rio *rdb, double val);
+int rdbLoadBinaryDoubleValue(rio *rdb, double *val);
+int rdbSaveBinaryFloatValue(rio *rdb, float val);
+int rdbLoadBinaryFloatValue(rio *rdb, float *val);
+int rdbLoadRio(rio *rdb);
 
 #endif

src/redis-benchmark.c

@@ -65,6 +65,7 @@ static struct config {
     int randomkeys_keyspacelen;
     int keepalive;
     int pipeline;
+    int showerrors;
     long long start;
     long long totlatency;
     long long *latency;
@@ -212,6 +213,16 @@ static void readHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
                     exit(1);
                 }
 
+                if (config.showerrors) {
+                    static time_t lasterr_time = 0;
+                    time_t now = time(NULL);
+                    redisReply *r = reply;
+                    if (r->type == REDIS_REPLY_ERROR && lasterr_time != now) {
+                        lasterr_time = now;
+                        printf("Error from server: %s\n", r->str);
+                    }
+                }
+
                 freeReplyObject(reply);
                 /* This is an OK for prefix commands such as auth and select.*/
                 if (c->prefix_pending > 0) {
@@ -227,7 +238,7 @@ static void readHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
                             c->randptr[j] -= c->prefixlen;
                         c->prefixlen = 0;
                     }
-                    continue;                
+                    continue;
                 }
 
                 if (config.requests_finished < config.requests)
@@ -518,6 +529,8 @@ int parseOptions(int argc, const char **argv) {
             config.loop = 1;
         } else if (!strcmp(argv[i],"-I")) {
             config.idlemode = 1;
+        } else if (!strcmp(argv[i],"-e")) {
+            config.showerrors = 1;
         } else if (!strcmp(argv[i],"-t")) {
             if (lastarg) goto invalid;
             /* We get the list of tests to run as a string in the form
@@ -569,6 +582,8 @@ usage:
 "  is executed. Default tests use this to hit random keys in the\n"
 "  specified range.\n"
 " -P <numreq>        Pipeline <numreq> requests. Default 1 (no pipeline).\n"
+" -e                 If server replies with errors, show them on stdout.\n"
+"                    (no more than 1 error per second is displayed)\n"
 " -q                 Quiet. Just show query/sec values\n"
 " --csv              Output in CSV format\n"
 " -l                 Loop. Run the tests forever\n"
@@ -649,6 +664,7 @@ int main(int argc, const char **argv) {
     config.keepalive = 1;
     config.datasize = 3;
     config.pipeline = 1;
+    config.showerrors = 0;
     config.randomkeys = 0;
     config.randomkeys_keyspacelen = 0;
     config.quiet = 0;

src/redis-check-rdb.c

@@ -1,6 +1,5 @@
 /*
- * Copyright (c) 2009-2012, Pieter Noordhuis <pcnoordhuis at gmail dot com>
- * Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
+ * Copyright (c) 2016, Salvatore Sanfilippo <antirez at gmail dot com>
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -28,683 +27,324 @@
  * POSSIBILITY OF SUCH DAMAGE.
  */
 
-
 #include "server.h"
 #include "rdb.h"
-#include <stdlib.h>
-#include <stdio.h>
-#include <unistd.h>
-#include <fcntl.h>
-#include <sys/stat.h>
-#include <sys/mman.h>
-#include "lzf.h"
-#include "crc64.h"
 
-#define ERROR(...) { \
-    serverLog(LL_WARNING, __VA_ARGS__); \
-    exit(1); \
+#include <stdarg.h>
+
+void createSharedObjects(void);
+void rdbLoadProgressCallback(rio *r, const void *buf, size_t len);
+long long rdbLoadMillisecondTime(rio *rdb);
+int rdbCheckMode = 0;
+
+struct {
+    rio *rio;
+    robj *key;                      /* Current key we are reading. */
+    int key_type;                   /* Current key type if != -1. */
+    unsigned long keys;             /* Number of keys processed. */
+    unsigned long expires;          /* Number of keys with an expire. */
+    unsigned long already_expired;  /* Number of keys already expired. */
+    int doing;                      /* The state while reading the RDB. */
+    int error_set;                  /* True if error is populated. */
+    char error[1024];
+} rdbstate;
+
+/* At every loading step try to remember what we were about to do, so that
+ * we can log this information when an error is encountered. */
+#define RDB_CHECK_DOING_START 0
+#define RDB_CHECK_DOING_READ_TYPE 1
+#define RDB_CHECK_DOING_READ_EXPIRE 2
+#define RDB_CHECK_DOING_READ_KEY 3
+#define RDB_CHECK_DOING_READ_OBJECT_VALUE 4
+#define RDB_CHECK_DOING_CHECK_SUM 5
+#define RDB_CHECK_DOING_READ_LEN 6
+#define RDB_CHECK_DOING_READ_AUX 7
+
+char *rdb_check_doing_string[] = {
+    "start",
+    "read-type",
+    "read-expire",
+    "read-key",
+    "read-object-value",
+    "check-sum",
+    "read-len",
+    "read-aux"
+};
+
+char *rdb_type_string[] = {
+    "string",
+    "list-linked",
+    "set-hashtable",
+    "zset-v1",
+    "hash-hashtable",
+    "zset-v2",
+    "module-value",
+    "","",
+    "hash-zipmap",
+    "list-ziplist",
+    "set-intset",
+    "zset-ziplist",
+    "hash-ziplist",
+    "quicklist"
+};
+
+/* Show a few stats collected into 'rdbstate' */
+void rdbShowGenericInfo(void) {
+    printf("[info] %lu keys read\n", rdbstate.keys);
+    printf("[info] %lu expires\n", rdbstate.expires);
+    printf("[info] %lu already expired\n", rdbstate.already_expired);
 }
 
-/* data type to hold offset in file and size */
-typedef struct {
-    void *data;
-    size_t size;
-    size_t offset;
-} pos;
+/* Called on RDB errors. Provides details about the RDB and the offset
+ * we were when the error was detected. */
+void rdbCheckError(const char *fmt, ...) {
+    char msg[1024];
+    va_list ap;
 
-static unsigned char level = 0;
-static pos positions[16];
+    va_start(ap, fmt);
+    vsnprintf(msg, sizeof(msg), fmt, ap);
+    va_end(ap);
 
-#define CURR_OFFSET (positions[level].offset)
-
-/* Hold a stack of errors */
-typedef struct {
-    char error[16][1024];
-    size_t offset[16];
-    size_t level;
-} errors_t;
-static errors_t errors;
-
-#define SHIFT_ERROR(provided_offset, ...) { \
-    sprintf(errors.error[errors.level], __VA_ARGS__); \
-    errors.offset[errors.level] = provided_offset; \
-    errors.level++; \
+    printf("--- RDB ERROR DETECTED ---\n");
+    printf("[offset %llu] %s\n",
+        (unsigned long long) (rdbstate.rio ?
+            rdbstate.rio->processed_bytes : 0), msg);
+    printf("[additional info] While doing: %s\n",
+        rdb_check_doing_string[rdbstate.doing]);
+    if (rdbstate.key)
+        printf("[additional info] Reading key '%s'\n",
+            (char*)rdbstate.key->ptr);
+    if (rdbstate.key_type != -1)
+        printf("[additional info] Reading type %d (%s)\n",
+            rdbstate.key_type,
+            ((unsigned)rdbstate.key_type <
+             sizeof(rdb_type_string)/sizeof(char*)) ?
+                rdb_type_string[rdbstate.key_type] : "unknown");
+    rdbShowGenericInfo();
 }
 
-/* Data type to hold opcode with optional key name an success status */
-typedef struct {
-    char* key;
-    int type;
-    char success;
-} entry;
+/* Print informations during RDB checking. */
+void rdbCheckInfo(const char *fmt, ...) {
+    char msg[1024];
+    va_list ap;
 
-#define MAX_TYPES_NUM 256
-#define MAX_TYPE_NAME_LEN 16
-/* store string types for output */
-static char types[MAX_TYPES_NUM][MAX_TYPE_NAME_LEN];
+    va_start(ap, fmt);
+    vsnprintf(msg, sizeof(msg), fmt, ap);
+    va_end(ap);
 
-/* Return true if 't' is a valid object type. */
-static int rdbCheckType(unsigned char t) {
-    /* In case a new object type is added, update the following
-     * condition as necessary. */
-    return
-        (t >= RDB_TYPE_HASH_ZIPMAP && t <= RDB_TYPE_HASH_ZIPLIST) ||
-        t <= RDB_TYPE_HASH ||
-        t >= RDB_OPCODE_EXPIRETIME_MS;
+    printf("[offset %llu] %s\n",
+        (unsigned long long) (rdbstate.rio ?
+            rdbstate.rio->processed_bytes : 0), msg);
 }
 
-/* when number of bytes to read is negative, do a peek */
-static int readBytes(void *target, long num) {
-    char peek = (num < 0) ? 1 : 0;
-    num = (num < 0) ? -num : num;
+/* Used inside rdb.c in order to log specific errors happening inside
+ * the RDB loading internals. */
+void rdbCheckSetError(const char *fmt, ...) {
+    va_list ap;
 
-    pos p = positions[level];
-    if (p.offset + num > p.size) {
-        return 0;
-    } else {
-        memcpy(target, (void*)((size_t)p.data + p.offset), num);
-        if (!peek) positions[level].offset += num;
-    }
-    return 1;
+    va_start(ap, fmt);
+    vsnprintf(rdbstate.error, sizeof(rdbstate.error), fmt, ap);
+    va_end(ap);
+    rdbstate.error_set = 1;
 }
 
-int processHeader(void) {
-    char buf[10] = "_________";
-    int dump_version;
+/* During RDB check we setup a special signal handler for memory violations
+ * and similar conditions, so that we can log the offending part of the RDB
+ * if the crash is due to broken content. */
+void rdbCheckHandleCrash(int sig, siginfo_t *info, void *secret) {
+    UNUSED(sig);
+    UNUSED(info);
+    UNUSED(secret);
 
-    if (!readBytes(buf, 9)) {
-        ERROR("Cannot read header");
-    }
-
-    /* expect the first 5 bytes to equal REDIS */
-    if (memcmp(buf,"REDIS",5) != 0) {
-        ERROR("Wrong signature in header");
-    }
-
-    dump_version = (int)strtol(buf + 5, NULL, 10);
-    if (dump_version < 1 || dump_version > 6) {
-        ERROR("Unknown RDB format version: %d", dump_version);
-    }
-    return dump_version;
+    rdbCheckError("Server crash checking the specified RDB file!");
+    exit(1);
 }
 
-static int loadType(entry *e) {
-    uint32_t offset = CURR_OFFSET;
+void rdbCheckSetupSignals(void) {
+    struct sigaction act;
 
-    /* this byte needs to qualify as type */
-    unsigned char t;
-    if (readBytes(&t, 1)) {
-        if (rdbCheckType(t)) {
-            e->type = t;
-            return 1;
-        } else {
-            SHIFT_ERROR(offset, "Unknown type (0x%02x)", t);
-        }
-    } else {
-        SHIFT_ERROR(offset, "Could not read type");
-    }
-
-    /* failure */
-    return 0;
-}
-
-static int peekType() {
-    unsigned char t;
-    if (readBytes(&t, -1) && (rdbCheckType(t)))
-        return t;
-    return -1;
-}
-
-/* discard time, just consume the bytes */
-static int processTime(int type) {
-    uint32_t offset = CURR_OFFSET;
-    unsigned char t[8];
-    int timelen = (type == RDB_OPCODE_EXPIRETIME_MS) ? 8 : 4;
-
-    if (readBytes(t,timelen)) {
-        return 1;
-    } else {
-        SHIFT_ERROR(offset, "Could not read time");
-    }
-
-    /* failure */
-    return 0;
-}
-
-static uint32_t loadLength(int *isencoded) {
-    unsigned char buf[2];
-    uint32_t len;
-    int type;
-
-    if (isencoded) *isencoded = 0;
-    if (!readBytes(buf, 1)) return RDB_LENERR;
-    type = (buf[0] & 0xC0) >> 6;
-    if (type == RDB_6BITLEN) {
-        /* Read a 6 bit len */
-        return buf[0] & 0x3F;
-    } else if (type == RDB_ENCVAL) {
-        /* Read a 6 bit len encoding type */
-        if (isencoded) *isencoded = 1;
-        return buf[0] & 0x3F;
-    } else if (type == RDB_14BITLEN) {
-        /* Read a 14 bit len */
-        if (!readBytes(buf+1,1)) return RDB_LENERR;
-        return ((buf[0] & 0x3F) << 8) | buf[1];
-    } else {
-        /* Read a 32 bit len */
-        if (!readBytes(&len, 4)) return RDB_LENERR;
-        return (unsigned int)ntohl(len);
-    }
-}
-
-static char *loadIntegerObject(int enctype) {
-    uint32_t offset = CURR_OFFSET;
-    unsigned char enc[4];
-    long long val;
-
-    if (enctype == RDB_ENC_INT8) {
-        uint8_t v;
-        if (!readBytes(enc, 1)) return NULL;
-        v = enc[0];
-        val = (int8_t)v;
-    } else if (enctype == RDB_ENC_INT16) {
-        uint16_t v;
-        if (!readBytes(enc, 2)) return NULL;
-        v = enc[0]|(enc[1]<<8);
-        val = (int16_t)v;
-    } else if (enctype == RDB_ENC_INT32) {
-        uint32_t v;
-        if (!readBytes(enc, 4)) return NULL;
-        v = enc[0]|(enc[1]<<8)|(enc[2]<<16)|(enc[3]<<24);
-        val = (int32_t)v;
-    } else {
-        SHIFT_ERROR(offset, "Unknown integer encoding (0x%02x)", enctype);
-        return NULL;
-    }
-
-    /* convert val into string */
-    char *buf;
-    buf = zmalloc(sizeof(char) * 128);
-    sprintf(buf, "%lld", val);
-    return buf;
-}
-
-static char* loadLzfStringObject() {
-    unsigned int slen, clen;
-    char *c, *s;
-
-    if ((clen = loadLength(NULL)) == RDB_LENERR) return NULL;
-    if ((slen = loadLength(NULL)) == RDB_LENERR) return NULL;
-
-    c = zmalloc(clen);
-    if (!readBytes(c, clen)) {
-        zfree(c);
-        return NULL;
-    }
-
-    s = zmalloc(slen+1);
-    if (lzf_decompress(c,clen,s,slen) == 0) {
-        zfree(c); zfree(s);
-        return NULL;
-    }
-
-    zfree(c);
-    return s;
-}
-
-/* returns NULL when not processable, char* when valid */
-static char* loadStringObject() {
-    uint32_t offset = CURR_OFFSET;
-    int isencoded;
-    uint32_t len;
-
-    len = loadLength(&isencoded);
-    if (isencoded) {
-        switch(len) {
-        case RDB_ENC_INT8:
-        case RDB_ENC_INT16:
-        case RDB_ENC_INT32:
-            return loadIntegerObject(len);
-        case RDB_ENC_LZF:
-            return loadLzfStringObject();
-        default:
-            /* unknown encoding */
-            SHIFT_ERROR(offset, "Unknown string encoding (0x%02x)", len);
-            return NULL;
-        }
-    }
-
-    if (len == RDB_LENERR) return NULL;
-
-    char *buf = zmalloc(sizeof(char) * (len+1));
-    if (buf == NULL) return NULL;
-    buf[len] = '\0';
-    if (!readBytes(buf, len)) {
-        zfree(buf);
-        return NULL;
-    }
-    return buf;
-}
-
-static int processStringObject(char** store) {
-    unsigned long offset = CURR_OFFSET;
-    char *key = loadStringObject();
-    if (key == NULL) {
-        SHIFT_ERROR(offset, "Error reading string object");
-        zfree(key);
-        return 0;
-    }
-
-    if (store != NULL) {
-        *store = key;
-    } else {
-        zfree(key);
-    }
-    return 1;
-}
-
-static double* loadDoubleValue() {
-    char buf[256];
-    unsigned char len;
-    double* val;
-
-    if (!readBytes(&len,1)) return NULL;
-
-    val = zmalloc(sizeof(double));
-    switch(len) {
-    case 255: *val = R_NegInf;  return val;
-    case 254: *val = R_PosInf;  return val;
-    case 253: *val = R_Nan;     return val;
-    default:
-        if (!readBytes(buf, len)) {
-            zfree(val);
-            return NULL;
-        }
-        buf[len] = '\0';
-        sscanf(buf, "%lg", val);
-        return val;
-    }
-}
-
-static int processDoubleValue(double** store) {
-    unsigned long offset = CURR_OFFSET;
-    double *val = loadDoubleValue();
-    if (val == NULL) {
-        SHIFT_ERROR(offset, "Error reading double value");
-        zfree(val);
-        return 0;
-    }
-
-    if (store != NULL) {
-        *store = val;
-    } else {
-        zfree(val);
-    }
-    return 1;
-}
-
-static int loadPair(entry *e) {
-    uint32_t offset = CURR_OFFSET;
-    uint32_t i;
-
-    /* read key first */
-    char *key;
-    if (processStringObject(&key)) {
-        e->key = key;
-    } else {
-        SHIFT_ERROR(offset, "Error reading entry key");
-        return 0;
-    }
-
-    uint32_t length = 0;
-    if (e->type == RDB_TYPE_LIST ||
-        e->type == RDB_TYPE_SET  ||
-        e->type == RDB_TYPE_ZSET ||
-        e->type == RDB_TYPE_HASH) {
-        if ((length = loadLength(NULL)) == RDB_LENERR) {
-            SHIFT_ERROR(offset, "Error reading %s length", types[e->type]);
-            return 0;
-        }
-    }
-
-    switch(e->type) {
-    case RDB_TYPE_STRING:
-    case RDB_TYPE_HASH_ZIPMAP:
-    case RDB_TYPE_LIST_ZIPLIST:
-    case RDB_TYPE_SET_INTSET:
-    case RDB_TYPE_ZSET_ZIPLIST:
-    case RDB_TYPE_HASH_ZIPLIST:
-        if (!processStringObject(NULL)) {
-            SHIFT_ERROR(offset, "Error reading entry value");
-            return 0;
-        }
-    break;
-    case RDB_TYPE_LIST:
-    case RDB_TYPE_SET:
-        for (i = 0; i < length; i++) {
-            offset = CURR_OFFSET;
-            if (!processStringObject(NULL)) {
-                SHIFT_ERROR(offset, "Error reading element at index %d (length: %d)", i, length);
-                return 0;
-            }
-        }
-    break;
-    case RDB_TYPE_ZSET:
-        for (i = 0; i < length; i++) {
-            offset = CURR_OFFSET;
-            if (!processStringObject(NULL)) {
-                SHIFT_ERROR(offset, "Error reading element key at index %d (length: %d)", i, length);
-                return 0;
-            }
-            offset = CURR_OFFSET;
-            if (!processDoubleValue(NULL)) {
-                SHIFT_ERROR(offset, "Error reading element value at index %d (length: %d)", i, length);
-                return 0;
-            }
-        }
-    break;
-    case RDB_TYPE_HASH:
-        for (i = 0; i < length; i++) {
-            offset = CURR_OFFSET;
-            if (!processStringObject(NULL)) {
-                SHIFT_ERROR(offset, "Error reading element key at index %d (length: %d)", i, length);
-                return 0;
-            }
-            offset = CURR_OFFSET;
-            if (!processStringObject(NULL)) {
-                SHIFT_ERROR(offset, "Error reading element value at index %d (length: %d)", i, length);
-                return 0;
-            }
-        }
-    break;
-    default:
-        SHIFT_ERROR(offset, "Type not implemented");
-        return 0;
-    }
-    /* because we're done, we assume success */
-    e->success = 1;
-    return 1;
-}
-
-static entry loadEntry() {
-    entry e = { NULL, -1, 0 };
-    uint32_t length, offset[4];
-
-    /* reset error container */
-    errors.level = 0;
-
-    offset[0] = CURR_OFFSET;
-    if (!loadType(&e)) {
-        return e;
-    }
-
-    offset[1] = CURR_OFFSET;
-    if (e.type == RDB_OPCODE_SELECTDB) {
-        if ((length = loadLength(NULL)) == RDB_LENERR) {
-            SHIFT_ERROR(offset[1], "Error reading database number");
-            return e;
-        }
-        if (length > 63) {
-            SHIFT_ERROR(offset[1], "Database number out of range (%d)", length);
-            return e;
-        }
-    } else if (e.type == RDB_OPCODE_EOF) {
-        if (positions[level].offset < positions[level].size) {
-            SHIFT_ERROR(offset[0], "Unexpected EOF");
-        } else {
-            e.success = 1;
-        }
-        return e;
-    } else {
-        /* optionally consume expire */
-        if (e.type == RDB_OPCODE_EXPIRETIME ||
-            e.type == RDB_OPCODE_EXPIRETIME_MS) {
-            if (!processTime(e.type)) return e;
-            if (!loadType(&e)) return e;
-        }
-
-        offset[1] = CURR_OFFSET;
-        if (!loadPair(&e)) {
-            SHIFT_ERROR(offset[1], "Error for type %s", types[e.type]);
-            return e;
-        }
-    }
-
-    /* all entries are followed by a valid type:
-     * e.g. a new entry, SELECTDB, EXPIRE, EOF */
-    offset[2] = CURR_OFFSET;
-    if (peekType() == -1) {
-        SHIFT_ERROR(offset[2], "Followed by invalid type");
-        SHIFT_ERROR(offset[0], "Error for type %s", types[e.type]);
-        e.success = 0;
-    } else {
-        e.success = 1;
-    }
-
-    return e;
-}
-
-static void printCentered(int indent, int width, char* body) {
-    char head[256], tail[256];
-    memset(head, '\0', 256);
-    memset(tail, '\0', 256);
-
-    memset(head, '=', indent);
-    memset(tail, '=', width - 2 - indent - strlen(body));
-    serverLog(LL_WARNING, "%s %s %s", head, body, tail);
-}
-
-static void printValid(uint64_t ops, uint64_t bytes) {
-    char body[80];
-    sprintf(body, "Processed %llu valid opcodes (in %llu bytes)",
-        (unsigned long long) ops, (unsigned long long) bytes);
-    printCentered(4, 80, body);
-}
-
-static void printSkipped(uint64_t bytes, uint64_t offset) {
-    char body[80];
-    sprintf(body, "Skipped %llu bytes (resuming at 0x%08llx)",
-        (unsigned long long) bytes, (unsigned long long) offset);
-    printCentered(4, 80, body);
-}
-
-static void printErrorStack(entry *e) {
-    unsigned int i;
-    char body[64];
-
-    if (e->type == -1) {
-        sprintf(body, "Error trace");
-    } else if (e->type >= 253) {
-        sprintf(body, "Error trace (%s)", types[e->type]);
-    } else if (!e->key) {
-        sprintf(body, "Error trace (%s: (unknown))", types[e->type]);
-    } else {
-        char tmp[41];
-        strncpy(tmp, e->key, 40);
-
-        /* display truncation at the last 3 chars */
-        if (strlen(e->key) > 40) {
-            memset(&tmp[37], '.', 3);
-        }
-
-        /* display unprintable characters as ? */
-        for (i = 0; i < strlen(tmp); i++) {
-            if (tmp[i] <= 32) tmp[i] = '?';
-        }
-        sprintf(body, "Error trace (%s: %s)", types[e->type], tmp);
-    }
-
-    printCentered(4, 80, body);
-
-    /* display error stack */
-    for (i = 0; i < errors.level; i++) {
-        serverLog(LL_WARNING, "0x%08lx - %s",
-            (unsigned long) errors.offset[i], errors.error[i]);
-    }
-}
-
-void process(void) {
-    uint64_t num_errors = 0, num_valid_ops = 0, num_valid_bytes = 0;
-    entry entry = { NULL, -1, 0 };
-    int dump_version = processHeader();
-
-    /* Exclude the final checksum for RDB >= 5. Will be checked at the end. */
-    if (dump_version >= 5) {
-        if (positions[0].size < 8) {
-            serverLog(LL_WARNING, "RDB version >= 5 but no room for checksum.");
-            exit(1);
-        }
-        positions[0].size -= 8;
-    }
-
-    level = 1;
-    while(positions[0].offset < positions[0].size) {
-        positions[1] = positions[0];
-
-        entry = loadEntry();
-        if (!entry.success) {
-            printValid(num_valid_ops, num_valid_bytes);
-            printErrorStack(&entry);
-            num_errors++;
-            num_valid_ops = 0;
-            num_valid_bytes = 0;
-
-            /* search for next valid entry */
-            uint64_t offset = positions[0].offset + 1;
-            int i = 0;
-
-            while (!entry.success && offset < positions[0].size) {
-                positions[1].offset = offset;
-
-                /* find 3 consecutive valid entries */
-                for (i = 0; i < 3; i++) {
-                    entry = loadEntry();
-                    if (!entry.success) break;
-                }
-                /* check if we found 3 consecutive valid entries */
-                if (i < 3) {
-                    offset++;
-                }
-            }
-
-            /* print how many bytes we have skipped to find a new valid opcode */
-            if (offset < positions[0].size) {
-                printSkipped(offset - positions[0].offset, offset);
-            }
-
-            positions[0].offset = offset;
-        } else {
-            num_valid_ops++;
-            num_valid_bytes += positions[1].offset - positions[0].offset;
-
-            /* advance position */
-            positions[0] = positions[1];
-        }
-        zfree(entry.key);
-    }
-
-    /* because there is another potential error,
-     * print how many valid ops we have processed */
-    printValid(num_valid_ops, num_valid_bytes);
-
-    /* expect an eof */
-    if (entry.type != RDB_OPCODE_EOF) {
-        /* last byte should be EOF, add error */
-        errors.level = 0;
-        SHIFT_ERROR(positions[0].offset, "Expected EOF, got %s", types[entry.type]);
-
-        /* this is an EOF error so reset type */
-        entry.type = -1;
-        printErrorStack(&entry);
-
-        num_errors++;
-    }
-
-    /* Verify checksum */
-    if (dump_version >= 5) {
-        uint64_t crc = crc64(0,positions[0].data,positions[0].size);
-        uint64_t crc2;
-        unsigned char *p = (unsigned char*)positions[0].data+positions[0].size;
-        crc2 = ((uint64_t)p[0] << 0) |
-               ((uint64_t)p[1] << 8) |
-               ((uint64_t)p[2] << 16) |
-               ((uint64_t)p[3] << 24) |
-               ((uint64_t)p[4] << 32) |
-               ((uint64_t)p[5] << 40) |
-               ((uint64_t)p[6] << 48) |
-               ((uint64_t)p[7] << 56);
-        if (crc != crc2) {
-            SHIFT_ERROR(positions[0].offset, "RDB CRC64 does not match.");
-        } else {
-            serverLog(LL_WARNING, "CRC64 checksum is OK");
-        }
-    }
-
-    /* print summary on errors */
-    if (num_errors) {
-        serverLog(LL_WARNING, "Total unprocessable opcodes: %llu",
-            (unsigned long long) num_errors);
-    }
+    sigemptyset(&act.sa_mask);
+    act.sa_flags = SA_NODEFER | SA_RESETHAND | SA_SIGINFO;
+    act.sa_sigaction = rdbCheckHandleCrash;
+    sigaction(SIGSEGV, &act, NULL);
+    sigaction(SIGBUS, &act, NULL);
+    sigaction(SIGFPE, &act, NULL);
+    sigaction(SIGILL, &act, NULL);
 }
 
+/* Check the specified RDB file. Return 0 if the RDB looks sane, otherwise
+ * 1 is returned. */
 int redis_check_rdb(char *rdbfilename) {
-    int fd;
-    off_t size;
-    struct stat stat;
-    void *data;
+    uint64_t dbid;
+    int type, rdbver;
+    char buf[1024];
+    long long expiretime, now = mstime();
+    FILE *fp;
+    static rio rdb; /* Pointed by global struct riostate. */
 
-    fd = open(rdbfilename, O_RDONLY);
-    if (fd < 1) {
-        ERROR("Cannot open file: %s", rdbfilename);
+    if ((fp = fopen(rdbfilename,"r")) == NULL) return 1;
+
+    rioInitWithFile(&rdb,fp);
+    rdbstate.rio = &rdb;
+    rdb.update_cksum = rdbLoadProgressCallback;
+    if (rioRead(&rdb,buf,9) == 0) goto eoferr;
+    buf[9] = '\0';
+    if (memcmp(buf,"REDIS",5) != 0) {
+        rdbCheckError("Wrong signature trying to load DB from file");
+        return 1;
     }
-    if (fstat(fd, &stat) == -1) {
-        ERROR("Cannot stat: %s", rdbfilename);
-    } else {
-        size = stat.st_size;
+    rdbver = atoi(buf+5);
+    if (rdbver < 1 || rdbver > RDB_VERSION) {
+        rdbCheckError("Can't handle RDB format version %d",rdbver);
+        return 1;
     }
 
-    if (sizeof(size_t) == sizeof(int32_t) && size >= INT_MAX) {
-        ERROR("Cannot check dump files >2GB on a 32-bit platform");
+    startLoading(fp);
+    while(1) {
+        robj *key, *val;
+        expiretime = -1;
+
+        /* Read type. */
+        rdbstate.doing = RDB_CHECK_DOING_READ_TYPE;
+        if ((type = rdbLoadType(&rdb)) == -1) goto eoferr;
+
+        /* Handle special types. */
+        if (type == RDB_OPCODE_EXPIRETIME) {
+            rdbstate.doing = RDB_CHECK_DOING_READ_EXPIRE;
+            /* EXPIRETIME: load an expire associated with the next key
+             * to load. Note that after loading an expire we need to
+             * load the actual type, and continue. */
+            if ((expiretime = rdbLoadTime(&rdb)) == -1) goto eoferr;
+            /* We read the time so we need to read the object type again. */
+            rdbstate.doing = RDB_CHECK_DOING_READ_TYPE;
+            if ((type = rdbLoadType(&rdb)) == -1) goto eoferr;
+            /* the EXPIRETIME opcode specifies time in seconds, so convert
+             * into milliseconds. */
+            expiretime *= 1000;
+        } else if (type == RDB_OPCODE_EXPIRETIME_MS) {
+            /* EXPIRETIME_MS: milliseconds precision expire times introduced
+             * with RDB v3. Like EXPIRETIME but no with more precision. */
+            rdbstate.doing = RDB_CHECK_DOING_READ_EXPIRE;
+            if ((expiretime = rdbLoadMillisecondTime(&rdb)) == -1) goto eoferr;
+            /* We read the time so we need to read the object type again. */
+            rdbstate.doing = RDB_CHECK_DOING_READ_TYPE;
+            if ((type = rdbLoadType(&rdb)) == -1) goto eoferr;
+        } else if (type == RDB_OPCODE_EOF) {
+            /* EOF: End of file, exit the main loop. */
+            break;
+        } else if (type == RDB_OPCODE_SELECTDB) {
+            /* SELECTDB: Select the specified database. */
+            rdbstate.doing = RDB_CHECK_DOING_READ_LEN;
+            if ((dbid = rdbLoadLen(&rdb,NULL)) == RDB_LENERR)
+                goto eoferr;
+            rdbCheckInfo("Selecting DB ID %d", dbid);
+            continue; /* Read type again. */
+        } else if (type == RDB_OPCODE_RESIZEDB) {
+            /* RESIZEDB: Hint about the size of the keys in the currently
+             * selected data base, in order to avoid useless rehashing. */
+            uint64_t db_size, expires_size;
+            rdbstate.doing = RDB_CHECK_DOING_READ_LEN;
+            if ((db_size = rdbLoadLen(&rdb,NULL)) == RDB_LENERR)
+                goto eoferr;
+            if ((expires_size = rdbLoadLen(&rdb,NULL)) == RDB_LENERR)
+                goto eoferr;
+            continue; /* Read type again. */
+        } else if (type == RDB_OPCODE_AUX) {
+            /* AUX: generic string-string fields. Use to add state to RDB
+             * which is backward compatible. Implementations of RDB loading
+             * are requierd to skip AUX fields they don't understand.
+             *
+             * An AUX field is composed of two strings: key and value. */
+            robj *auxkey, *auxval;
+            rdbstate.doing = RDB_CHECK_DOING_READ_AUX;
+            if ((auxkey = rdbLoadStringObject(&rdb)) == NULL) goto eoferr;
+            if ((auxval = rdbLoadStringObject(&rdb)) == NULL) goto eoferr;
+
+            rdbCheckInfo("AUX FIELD %s = '%s'",
+                (char*)auxkey->ptr, (char*)auxval->ptr);
+            decrRefCount(auxkey);
+            decrRefCount(auxval);
+            continue; /* Read type again. */
+        } else {
+            if (!rdbIsObjectType(type)) {
+                rdbCheckError("Invalid object type: %d", type);
+                return 1;
+            }
+            rdbstate.key_type = type;
+        }
+
+        /* Read key */
+        rdbstate.doing = RDB_CHECK_DOING_READ_KEY;
+        if ((key = rdbLoadStringObject(&rdb)) == NULL) goto eoferr;
+        rdbstate.key = key;
+        rdbstate.keys++;
+        /* Read value */
+        rdbstate.doing = RDB_CHECK_DOING_READ_OBJECT_VALUE;
+        if ((val = rdbLoadObject(type,&rdb)) == NULL) goto eoferr;
+        /* Check if the key already expired. This function is used when loading
+         * an RDB file from disk, either at startup, or when an RDB was
+         * received from the master. In the latter case, the master is
+         * responsible for key expiry. If we would expire keys here, the
+         * snapshot taken by the master may not be reflected on the slave. */
+        if (server.masterhost == NULL && expiretime != -1 && expiretime < now)
+            rdbstate.already_expired++;
+        if (expiretime != -1) rdbstate.expires++;
+        rdbstate.key = NULL;
+        decrRefCount(key);
+        decrRefCount(val);
+        rdbstate.key_type = -1;
+    }
+    /* Verify the checksum if RDB version is >= 5 */
+    if (rdbver >= 5 && server.rdb_checksum) {
+        uint64_t cksum, expected = rdb.cksum;
+
+        rdbstate.doing = RDB_CHECK_DOING_CHECK_SUM;
+        if (rioRead(&rdb,&cksum,8) == 0) goto eoferr;
+        memrev64ifbe(&cksum);
+        if (cksum == 0) {
+            rdbCheckInfo("RDB file was saved with checksum disabled: no check performed.");
+        } else if (cksum != expected) {
+            rdbCheckError("RDB CRC error");
+        } else {
+            rdbCheckInfo("Checksum OK");
+        }
     }
 
-    data = mmap(NULL, size, PROT_READ, MAP_SHARED, fd, 0);
-    if (data == MAP_FAILED) {
-        ERROR("Cannot mmap: %s", rdbfilename);
-    }
-
-    /* Initialize static vars */
-    positions[0].data = data;
-    positions[0].size = size;
-    positions[0].offset = 0;
-    errors.level = 0;
-
-    /* Object types */
-    sprintf(types[RDB_TYPE_STRING], "STRING");
-    sprintf(types[RDB_TYPE_LIST], "LIST");
-    sprintf(types[RDB_TYPE_SET], "SET");
-    sprintf(types[RDB_TYPE_ZSET], "ZSET");
-    sprintf(types[RDB_TYPE_HASH], "HASH");
-
-    /* Object types only used for dumping to disk */
-    sprintf(types[RDB_OPCODE_EXPIRETIME], "EXPIRETIME");
-    sprintf(types[RDB_OPCODE_SELECTDB], "SELECTDB");
-    sprintf(types[RDB_OPCODE_EOF], "EOF");
-
-    process();
-
-    munmap(data, size);
-    close(fd);
+    fclose(fp);
     return 0;
+
+eoferr: /* unexpected end of file is handled here with a fatal exit */
+    if (rdbstate.error_set) {
+        rdbCheckError(rdbstate.error);
+    } else {
+        rdbCheckError("Unexpected EOF reading RDB file");
+    }
+    return 1;
 }
 
 /* RDB check main: called form redis.c when Redis is executed with the
- * redis-check-rdb alias. */
-int redis_check_rdb_main(char **argv, int argc) {
+ * redis-check-rdb alias.
+ *
+ * The function never returns, but exits with the status code according
+ * to success (RDB is sane) or error (RDB is corrupted). */
+int redis_check_rdb_main(int argc, char **argv) {
     if (argc != 2) {
         fprintf(stderr, "Usage: %s <rdb-file-name>\n", argv[0]);
         exit(1);
     }
-    serverLog(LL_WARNING, "Checking RDB file %s", argv[1]);
-    exit(redis_check_rdb(argv[1]));
-    return 0;
+    /* In order to call the loading functions we need to create the shared
+     * integer objects, however since this function may be called from
+     * an already initialized Redis instance, check if we really need to. */
+    if (shared.integers[0] == NULL)
+        createSharedObjects();
+    server.loading_process_events_interval_bytes = 0;
+    rdbCheckMode = 1;
+    rdbCheckInfo("Checking RDB file %s", argv[1]);
+    rdbCheckSetupSignals();
+    int retval = redis_check_rdb(argv[1]);
+    if (retval == 0) {
+        rdbCheckInfo("\\o/ RDB looks OK! \\o/");
+        rdbShowGenericInfo();
+    }
+    exit(retval);
 }

src/redis-cli.c

@@ -63,6 +63,8 @@
 #define REDIS_CLI_DEFAULT_PIPE_TIMEOUT 30 /* seconds */
 #define REDIS_CLI_HISTFILE_ENV "REDISCLI_HISTFILE"
 #define REDIS_CLI_HISTFILE_DEFAULT ".rediscli_history"
+#define REDIS_CLI_RCFILE_ENV "REDISCLI_RCFILE"
+#define REDIS_CLI_RCFILE_DEFAULT ".redisclirc"
 
 /* --latency-dist palettes. */
 int spectrum_palette_color_size = 19;
@@ -114,14 +116,21 @@ static struct config {
     int eval_ldb;
     int eval_ldb_sync;  /* Ask for synchronous mode of the Lua debugger. */
     int eval_ldb_end;   /* Lua debugging session ended. */
+    int enable_ldb_on_eval; /* Handle manual SCRIPT DEBUG + EVAL commands. */
     int last_cmd_type;
 } config;
 
+/* User preferences. */
+static struct pref {
+    int hints;
+} pref;
+
 static volatile sig_atomic_t force_cancel_loop = 0;
 static void usage(void);
 static void slaveMode(void);
 char *redisGitSHA1(void);
 char *redisGitDirty(void);
+static int cliConnect(int force);
 
 /*------------------------------------------------------------------------------
  * Utility functions
@@ -152,34 +161,41 @@ static void cliRefreshPrompt(void) {
         len = anetFormatAddr(config.prompt, sizeof(config.prompt),
                            config.hostip, config.hostport);
     /* Add [dbnum] if needed */
-    if (config.dbnum != 0 && config.last_cmd_type != REDIS_REPLY_ERROR)
+    if (config.dbnum != 0)
         len += snprintf(config.prompt+len,sizeof(config.prompt)-len,"[%d]",
             config.dbnum);
     snprintf(config.prompt+len,sizeof(config.prompt)-len,"> ");
 }
 
-static sds getHistoryPath() {
+/* Return the name of the dotfile for the specified 'dotfilename'.
+ * Normally it just concatenates user $HOME to the file specified
+ * in 'dotfilename'. However if the environment varialbe 'envoverride'
+ * is set, its value is taken as the path.
+ *
+ * The function returns NULL (if the file is /dev/null or cannot be
+ * obtained for some error), or an SDS string that must be freed by
+ * the user. */
+static sds getDotfilePath(char *envoverride, char *dotfilename) {
     char *path = NULL;
-    sds historyPath = NULL;
+    sds dotPath = NULL;
 
-    /* check the env for a histfile override */
-    path = getenv(REDIS_CLI_HISTFILE_ENV);
+    /* Check the env for a dotfile override. */
+    path = getenv(envoverride);
     if (path != NULL && *path != '\0') {
         if (!strcmp("/dev/null", path)) {
             return NULL;
         }
 
-        /* if the env is set, return it */
-        historyPath = sdscatprintf(sdsempty(), "%s", path);
+        /* If the env is set, return it. */
+        dotPath = sdsnew(path);
     } else {
         char *home = getenv("HOME");
         if (home != NULL && *home != '\0') {
-            /* otherwise, return the default */
-            historyPath = sdscatprintf(sdsempty(), "%s/%s", home, REDIS_CLI_HISTFILE_DEFAULT);
+            /* If no override is set use $HOME/<dotfilename>. */
+            dotPath = sdscatprintf(sdsempty(), "%s/%s", home, dotfilename);
         }
     }
-
-    return historyPath;
+    return dotPath;
 }
 
 /*------------------------------------------------------------------------------
@@ -223,11 +239,11 @@ static void cliInitHelp(void) {
     helpEntry tmp;
 
     helpEntriesLen = len = commandslen+groupslen;
-    helpEntries = malloc(sizeof(helpEntry)*len);
+    helpEntries = zmalloc(sizeof(helpEntry)*len);
 
     for (i = 0; i < groupslen; i++) {
         tmp.argc = 1;
-        tmp.argv = malloc(sizeof(sds));
+        tmp.argv = zmalloc(sizeof(sds));
         tmp.argv[0] = sdscatprintf(sdsempty(),"@%s",commandGroups[i]);
         tmp.full = tmp.argv[0];
         tmp.type = CLI_HELP_GROUP;
@@ -244,6 +260,61 @@ static void cliInitHelp(void) {
     }
 }
 
+/* cliInitHelp() setups the helpEntries array with the command and group
+ * names from the help.h file. However the Redis instance we are connecting
+ * to may support more commands, so this function integrates the previous
+ * entries with additional entries obtained using the COMMAND command
+ * available in recent versions of Redis. */
+static void cliIntegrateHelp(void) {
+    if (cliConnect(0) == REDIS_ERR) return;
+
+    redisReply *reply = redisCommand(context, "COMMAND");
+    if(reply == NULL || reply->type != REDIS_REPLY_ARRAY) return;
+
+    /* Scan the array reported by COMMAND and fill only the entries that
+     * don't already match what we have. */
+    for (size_t j = 0; j < reply->elements; j++) {
+        redisReply *entry = reply->element[j];
+        char *cmdname = entry->element[0]->str;
+        int i;
+
+        for (i = 0; i < helpEntriesLen; i++) {
+            helpEntry *he = helpEntries+i;
+            if (!strcasecmp(he->argv[0],cmdname))
+                break;
+        }
+        if (i != helpEntriesLen) continue;
+
+        helpEntriesLen++;
+        helpEntries = zrealloc(helpEntries,sizeof(helpEntry)*helpEntriesLen);
+        helpEntry *new = helpEntries+(helpEntriesLen-1);
+
+        new->argc = 1;
+        new->argv = zmalloc(sizeof(sds));
+        new->argv[0] = sdsnew(cmdname);
+        new->full = new->argv[0];
+        new->type = CLI_HELP_COMMAND;
+        sdstoupper(new->argv[0]);
+
+        struct commandHelp *ch = zmalloc(sizeof(*ch));
+        ch->name = new->argv[0];
+        ch->params = sdsempty();
+        int args = llabs(entry->element[1]->integer);
+        if (entry->element[3]->integer == 1) {
+            ch->params = sdscat(ch->params,"key ");
+            args--;
+        }
+        while(args--) ch->params = sdscat(ch->params,"arg ");
+        if (entry->element[1]->integer < 0)
+            ch->params = sdscat(ch->params,"...options...");
+        ch->summary = "Help not available";
+        ch->group = 0;
+        ch->since = "not known";
+        new->org = ch;
+    }
+    freeReplyObject(reply);
+}
+
 /* Output command help to stdout. */
 static void cliOutputCommandHelp(struct commandHelp *help, int group) {
     printf("\r\n  \x1b[1m%s\x1b[0m \x1b[90m%s\x1b[0m\r\n", help->name, help->params);
@@ -258,11 +329,17 @@ static void cliOutputCommandHelp(struct commandHelp *help, int group) {
 static void cliOutputGenericHelp(void) {
     sds version = cliVersion();
     printf(
-        "redis-cli %s\r\n"
-        "Type: \"help @<group>\" to get a list of commands in <group>\r\n"
-        "      \"help <command>\" for help on <command>\r\n"
-        "      \"help <tab>\" to get a list of possible help topics\r\n"
-        "      \"quit\" to exit\r\n",
+        "redis-cli %s\n"
+        "To get help about Redis commands type:\n"
+        "      \"help @<group>\" to get a list of commands in <group>\n"
+        "      \"help <command>\" for help on <command>\n"
+        "      \"help <tab>\" to get a list of possible help topics\n"
+        "      \"quit\" to exit\n"
+        "\n"
+        "To set redis-cli perferences:\n"
+        "      \":set hints\" enable online hints\n"
+        "      \":set nohints\" disable online hints\n"
+        "Set your preferences in ~/.redisclirc\n",
         version
     );
     sdsfree(version);
@@ -313,6 +390,7 @@ static void cliOutputHelp(int argc, char **argv) {
     printf("\r\n");
 }
 
+/* Linenoise completion callback. */
 static void completionCallback(const char *buf, linenoiseCompletions *lc) {
     size_t startpos = 0;
     int mask;
@@ -341,6 +419,58 @@ static void completionCallback(const char *buf, linenoiseCompletions *lc) {
     }
 }
 
+/* Linenoise hints callback. */
+static char *hintsCallback(const char *buf, int *color, int *bold) {
+    if (!pref.hints) return NULL;
+
+    int i, argc, buflen = strlen(buf);
+    sds *argv = sdssplitargs(buf,&argc);
+    int endspace = buflen && isspace(buf[buflen-1]);
+
+    /* Check if the argument list is empty and return ASAP. */
+    if (argc == 0) {
+        sdsfreesplitres(argv,argc);
+        return NULL;
+    }
+
+    for (i = 0; i < helpEntriesLen; i++) {
+        if (!(helpEntries[i].type & CLI_HELP_COMMAND)) continue;
+
+        if (strcasecmp(argv[0],helpEntries[i].full) == 0)
+        {
+            *color = 90;
+            *bold = 0;
+            sds hint = sdsnew(helpEntries[i].org->params);
+
+            /* Remove arguments from the returned hint to show only the
+             * ones the user did not yet typed. */
+            int toremove = argc-1;
+            while(toremove > 0 && sdslen(hint)) {
+                if (hint[0] == '[') break;
+                if (hint[0] == ' ') toremove--;
+                sdsrange(hint,1,-1);
+            }
+
+            /* Add an initial space if needed. */
+            if (!endspace) {
+                sds newhint = sdsnewlen(" ",1);
+                newhint = sdscatsds(newhint,hint);
+                sdsfree(hint);
+                hint = newhint;
+            }
+
+            sdsfreesplitres(argv,argc);
+            return hint;
+        }
+    }
+    sdsfreesplitres(argv,argc);
+    return NULL;
+}
+
+static void freeHintsCallback(void *ptr) {
+    sdsfree(ptr);
+}
+
 /*------------------------------------------------------------------------------
  * Networking / parsing
  *--------------------------------------------------------------------------- */
@@ -463,7 +593,7 @@ static sds cliFormatReplyTTY(redisReply *r, char *prefix) {
             _prefix = sdscat(sdsnew(prefix),_prefixlen);
 
             /* Setup prefix format for every entry */
-            snprintf(_prefixfmt,sizeof(_prefixfmt),"%%s%%%dd) ",idxlen);
+            snprintf(_prefixfmt,sizeof(_prefixfmt),"%%s%%%ud) ",idxlen);
 
             for (i = 0; i < r->elements; i++) {
                 /* Don't use the prefix for the first element, as the parent
@@ -551,6 +681,7 @@ static sds cliFormatReplyRaw(redisReply *r) {
 
             /* Detect the end of a debugging session. */
             if (strstr(r->str,"<endsession>") == r->str) {
+                config.enable_ldb_on_eval = 0;
                 config.eval_ldb = 0;
                 config.eval_ldb_end = 1; /* Signal the caller session ended. */
                 config.output = OUTPUT_STANDARD;
@@ -663,7 +794,7 @@ static int cliReadReply(int output_raw_strings) {
         p = strchr(s+1,' ');    /* MOVED[S]3999[P]127.0.0.1:6381 */
         *p = '\0';
         slot = atoi(s+1);
-        s = strchr(p+1,':');    /* MOVED 3999[P]127.0.0.1[S]6381 */
+        s = strrchr(p+1,':');    /* MOVED 3999[P]127.0.0.1[S]6381 */
         *s = '\0';
         sdsfree(config.hostip);
         config.hostip = sdsnew(p+1);
@@ -712,8 +843,10 @@ static int cliSendCommand(int argc, char **argv, int repeat) {
     output_raw = 0;
     if (!strcasecmp(command,"info") ||
         (argc >= 2 && !strcasecmp(command,"debug") &&
-                      (!strcasecmp(argv[1],"jemalloc") ||
-                       !strcasecmp(argv[1],"htstats"))) ||
+                       !strcasecmp(argv[1],"htstats")) ||
+        (argc >= 2 && !strcasecmp(command,"memory") &&
+                      (!strcasecmp(argv[1],"malloc-stats") ||
+                       !strcasecmp(argv[1],"doctor"))) ||
         (argc == 2 && !strcasecmp(command,"cluster") &&
                       (!strcasecmp(argv[1],"nodes") ||
                        !strcasecmp(argv[1],"info"))) ||
@@ -734,8 +867,26 @@ static int cliSendCommand(int argc, char **argv, int repeat) {
     if (!strcasecmp(command,"sync") ||
         !strcasecmp(command,"psync")) config.slave_mode = 1;
 
+    /* When the user manually calls SCRIPT DEBUG, setup the activation of
+     * debugging mode on the next eval if needed. */
+    if (argc == 3 && !strcasecmp(argv[0],"script") &&
+                     !strcasecmp(argv[1],"debug"))
+    {
+        if (!strcasecmp(argv[2],"yes") || !strcasecmp(argv[2],"sync")) {
+            config.enable_ldb_on_eval = 1;
+        } else {
+            config.enable_ldb_on_eval = 0;
+        }
+    }
+
+    /* Actually activate LDB on EVAL if needed. */
+    if (!strcasecmp(command,"eval") && config.enable_ldb_on_eval) {
+        config.eval_ldb = 1;
+        config.output = OUTPUT_RAW;
+    }
+
     /* Setup argument length */
-    argvlen = malloc(argc*sizeof(size_t));
+    argvlen = zmalloc(argc*sizeof(size_t));
     for (j = 0; j < argc; j++)
         argvlen[j] = sdslen(argv[j]);
 
@@ -758,16 +909,16 @@ static int cliSendCommand(int argc, char **argv, int repeat) {
             printf("Entering slave output mode...  (press Ctrl-C to quit)\n");
             slaveMode();
             config.slave_mode = 0;
-            free(argvlen);
+            zfree(argvlen);
             return REDIS_ERR;  /* Error = slaveMode lost connection to master */
         }
 
         if (cliReadReply(output_raw) != REDIS_OK) {
-            free(argvlen);
+            zfree(argvlen);
             return REDIS_ERR;
         } else {
             /* Store database number when SELECT was successfully executed. */
-            if (!strcasecmp(command,"select") && argc == 2) {
+            if (!strcasecmp(command,"select") && argc == 2 && config.last_cmd_type != REDIS_REPLY_ERROR) {
                 config.dbnum = atoi(argv[1]);
                 cliRefreshPrompt();
             } else if (!strcasecmp(command,"auth") && argc == 2) {
@@ -778,7 +929,7 @@ static int cliSendCommand(int argc, char **argv, int repeat) {
         fflush(stdout); /* Make it grep friendly */
     }
 
-    free(argvlen);
+    zfree(argvlen);
     return REDIS_OK;
 }
 
@@ -1003,7 +1154,8 @@ static void usage(void) {
 "  (Note: when using --eval the comma separates KEYS[] from ARGV[] items)\n"
 "\n"
 "When no command is given, redis-cli starts in interactive mode.\n"
-"Type \"help\" in interactive mode for information on available commands.\n"
+"Type \"help\" in interactive mode for information on available commands\n"
+"and settings.\n"
 "\n",
         version, REDIS_CLI_DEFAULT_PIPE_TIMEOUT);
     sdsfree(version);
@@ -1070,6 +1222,45 @@ static sds *cliSplitArgs(char *line, int *argc) {
     }
 }
 
+/* Set the CLI perferences. This function is invoked when an interactive
+ * ":command" is called, or when reading ~/.redisclirc file, in order to
+ * set user preferences. */
+void cliSetPreferences(char **argv, int argc, int interactive) {
+    if (!strcasecmp(argv[0],":set") && argc >= 2) {
+        if (!strcasecmp(argv[1],"hints")) pref.hints = 1;
+        else if (!strcasecmp(argv[1],"nohints")) pref.hints = 0;
+        else {
+            printf("%sunknown redis-cli preference '%s'\n",
+                interactive ? "" : ".redisclirc: ",
+                argv[1]);
+        }
+    } else {
+        printf("%sunknown redis-cli internal command '%s'\n",
+            interactive ? "" : ".redisclirc: ",
+            argv[0]);
+    }
+}
+
+/* Load the ~/.redisclirc file if any. */
+void cliLoadPreferences(void) {
+    sds rcfile = getDotfilePath(REDIS_CLI_RCFILE_ENV,REDIS_CLI_RCFILE_DEFAULT);
+    if (rcfile == NULL) return;
+    FILE *fp = fopen(rcfile,"r");
+    char buf[1024];
+
+    if (fp) {
+        while(fgets(buf,sizeof(buf),fp) != NULL) {
+            sds *argv;
+            int argc;
+
+            argv = sdssplitargs(buf,&argc);
+            if (argc > 0) cliSetPreferences(argv,argc,0);
+            sdsfreesplitres(argv,argc);
+        }
+    }
+    sdsfree(rcfile);
+}
+
 static void repl(void) {
     sds historyfile = NULL;
     int history = 0;
@@ -1080,14 +1271,17 @@ static void repl(void) {
     config.interactive = 1;
     linenoiseSetMultiLine(1);
     linenoiseSetCompletionCallback(completionCallback);
+    linenoiseSetHintsCallback(hintsCallback);
+    linenoiseSetFreeHintsCallback(freeHintsCallback);
 
-    /* Only use history when stdin is a tty. */
+    /* Only use history and load the rc file when stdin is a tty. */
     if (isatty(fileno(stdin))) {
-        historyfile = getHistoryPath();
+        historyfile = getDotfilePath(REDIS_CLI_HISTFILE_ENV,REDIS_CLI_HISTFILE_DEFAULT);
         if (historyfile != NULL) {
             history = 1;
             linenoiseHistoryLoad(historyfile);
         }
+        cliLoadPreferences();
     }
 
     cliRefreshPrompt();
@@ -1099,13 +1293,16 @@ static void repl(void) {
 
             if (argv == NULL) {
                 printf("Invalid argument(s)\n");
-                free(line);
+                linenoiseFree(line);
                 continue;
             } else if (argc > 0) {
                 if (strcasecmp(argv[0],"quit") == 0 ||
                     strcasecmp(argv[0],"exit") == 0)
                 {
                     exit(0);
+                } else if (argv[0][0] == ':') {
+                    cliSetPreferences(argv,argc,1);
+                    continue;
                 } else if (strcasecmp(argv[0],"restart") == 0) {
                     if (config.eval) {
                         config.eval_ldb = 1;
@@ -1155,7 +1352,7 @@ static void repl(void) {
             sdsfreesplitres(argv,argc);
         }
         /* linenoise() returns malloc-ed lines like readline() */
-        free(line);
+        linenoiseFree(line);
     }
     exit(0);
 }
@@ -2020,7 +2217,7 @@ static char *getInfoField(char *info, char *field) {
     n1 = strchr(p,'\r');
     n2 = strchr(p,',');
     if (n2 && n2 < n1) n1 = n2;
-    result = malloc(sizeof(char)*(n1-p)+1);
+    result = zmalloc(sizeof(char)*(n1-p)+1);
     memcpy(result,p,(n1-p));
     result[n1-p] = '\0';
     return result;
@@ -2034,7 +2231,7 @@ static long getLongInfoField(char *info, char *field) {
 
     if (!value) return LONG_MIN;
     l = strtol(value,NULL,10);
-    free(value);
+    zfree(value);
     return l;
 }
 
@@ -2050,7 +2247,7 @@ void bytesToHuman(char *s, long long n) {
     }
     if (n < 1024) {
         /* Bytes */
-        sprintf(s,"%lluB",n);
+        sprintf(s,"%lldB",n);
         return;
     } else if (n < (1024*1024)) {
         d = (double)n/(1024);
@@ -2207,7 +2404,7 @@ long long powerLawRand(long long min, long long max, double alpha) {
 /* Generates a key name among a set of lru_test_sample_size keys, using
  * an 80-20 distribution. */
 void LRUTestGenKey(char *buf, size_t buflen) {
-    snprintf(buf, buflen, "lru:%lld\n",
+    snprintf(buf, buflen, "lru:%lld",
         powerLawRand(1, config.lru_test_sample_size, 6.2));
 }
 
@@ -2229,8 +2426,11 @@ static void LRUTestMode(void) {
         while(mstime() - start_cycle < 1000) {
             /* Write cycle. */
             for (j = 0; j < LRU_CYCLE_PIPELINE_SIZE; j++) {
+                char val[6];
+                val[5] = '\0';
+                for (int i = 0; i < 5; i++) val[i] = 'A'+rand()%('z'-'A');
                 LRUTestGenKey(key,sizeof(key));
-                redisAppendCommand(context, "SET %s val",key);
+                redisAppendCommand(context, "SET %s %s",key,val);
             }
             for (j = 0; j < LRU_CYCLE_PIPELINE_SIZE; j++)
                 redisGetReply(context, (void**)&reply);
@@ -2331,7 +2531,7 @@ static void intrinsicLatencyMode(void) {
         }
 
         double avg_us = (double)run_time/runs;
-        double avg_ns = avg_us * 10e3;
+        double avg_ns = avg_us * 1e3;
         if (force_cancel_loop || end > test_end) {
             printf("\n%lld total runs "
                 "(avg latency: "
@@ -2383,8 +2583,11 @@ int main(int argc, char **argv) {
     config.eval_ldb = 0;
     config.eval_ldb_end = 0;
     config.eval_ldb_sync = 0;
+    config.enable_ldb_on_eval = 0;
     config.last_cmd_type = -1;
 
+    pref.hints = 1;
+
     spectrum_palette = spectrum_palette_color;
     spectrum_palette_size = spectrum_palette_color_size;
 
@@ -2393,12 +2596,16 @@ int main(int argc, char **argv) {
     else
         config.output = OUTPUT_STANDARD;
     config.mb_delim = sdsnew("\n");
-    cliInitHelp();
 
     firstarg = parseOptions(argc,argv);
     argc -= firstarg;
     argv += firstarg;
 
+    /* Initialize the help and, if possible, use the COMMAND command in order
+     * to retrieve missing entries. */
+    cliInitHelp();
+    cliIntegrateHelp();
+
     /* Latency mode */
     if (config.latency_mode) {
         if (cliConnect(0) == REDIS_ERR) exit(1);

src/redis-trib.rb

@@ -25,6 +25,11 @@ require 'rubygems'
 require 'redis'
 
 ClusterHashSlots = 16384
+MigrateDefaultTimeout = 60000
+MigrateDefaultPipeline = 10
+RebalanceDefaultThreshold = 2
+
+$verbose = false
 
 def xputs(s)
     case s[0..2]
@@ -32,6 +37,8 @@ def xputs(s)
         color="29;1"
     when "[ER"
         color="31;1"
+    when "[WA"
+        color="31;1"
     when "[OK"
         color="32"
     when "[FA","***"
@@ -49,7 +56,7 @@ end
 
 class ClusterNode
     def initialize(addr)
-        s = addr.split(":")
+        s = addr.split("@")[0].split(":")
         if s.length < 2
            puts "Invalid IP or Port (given as #{addr}) - use IP:Port format"
            exit 1
@@ -86,7 +93,7 @@ class ClusterNode
 
     def connect(o={})
         return if @r
-        print "Connecting to node #{self}: "
+        print "Connecting to node #{self}: " if $verbose
         STDOUT.flush
         begin
             @r = Redis.new(:host => @info[:host], :port => @info[:port], :timeout => 60)
@@ -96,7 +103,7 @@ class ClusterNode
             exit 1 if o[:abort]
             @r = nil
         end
-        xputs "OK"
+        xputs "OK" if $verbose
     end
 
     def assert_cluster
@@ -288,6 +295,7 @@ class RedisTrib
         @nodes = []
         @fix = false
         @errors = []
+        @timeout = MigrateDefaultTimeout
     end
 
     def check_arity(req_args, num_args)
@@ -302,11 +310,16 @@ class RedisTrib
         @nodes << node
     end
 
+    def reset_nodes
+        @nodes = []
+    end
+
     def cluster_error(msg)
         @errors << msg
         xputs msg
     end
 
+    # Return the node with the specified ID or Nil.
     def get_node_by_name(name)
         @nodes.each{|n|
             return n if n.info[:name] == name.downcase
@@ -314,6 +327,21 @@ class RedisTrib
         return nil
     end
 
+    # Like get_node_by_name but the specified name can be just the first
+    # part of the node ID as long as the prefix in unique across the
+    # cluster.
+    def get_node_by_abbreviated_name(name)
+        l = name.length
+        candidates = []
+        @nodes.each{|n|
+            if n.info[:name][0...l] == name.downcase
+                candidates << n
+            end
+        }
+        return nil if candidates.length != 1
+        candidates[0]
+    end
+
     # This function returns the master that has the least number of replicas
     # in the cluster. If there are multiple masters with the same smaller
     # number of replicas, one at random is returned.
@@ -325,14 +353,30 @@ class RedisTrib
         sorted[0]
     end
 
-    def check_cluster
+    def check_cluster(opt={})
         xputs ">>> Performing Cluster Check (using node #{@nodes[0]})"
-        show_nodes
+        show_nodes if !opt[:quiet]
         check_config_consistency
         check_open_slots
         check_slots_coverage
     end
 
+    def show_cluster_info
+        masters = 0
+        keys = 0
+        @nodes.each{|n|
+            if n.has_flag?("master")
+                puts "#{n} (#{n.info[:name][0...8]}...) -> #{n.r.dbsize} keys | #{n.slots.length} slots | "+
+                     "#{n.info[:replicas].length} slaves."
+                masters += 1
+                keys += n.r.dbsize
+            end
+        }
+        xputs "[OK] #{keys} keys in #{masters} masters."
+        keys_per_slot = sprintf("%.2f",keys/16384.0)
+        puts "#{keys_per_slot} keys per slot on average."
+    end
+
     # Merge slots of every known node. If the resulting slots are equal
     # to ClusterHashSlots, then all slots are served.
     def covered_slots
@@ -363,7 +407,8 @@ class RedisTrib
                 cluster_error \
                     "[WARNING] Node #{n} has slots in migrating state (#{n.info[:migrating].keys.join(",")})."
                 open_slots += n.info[:migrating].keys
-            elsif n.info[:importing].size > 0
+            end
+            if n.info[:importing].size > 0
                 cluster_error \
                     "[WARNING] Node #{n} has slots in importing state (#{n.info[:importing].keys.join(",")})."
                 open_slots += n.info[:importing].keys
@@ -381,6 +426,7 @@ class RedisTrib
     def nodes_with_keys_in_slot(slot)
         nodes = []
         @nodes.each{|n|
+            next if n.has_flag?("slave")
             nodes << n if n.r.cluster("getkeysinslot",slot,1).length > 0
         }
         nodes
@@ -399,7 +445,7 @@ class RedisTrib
         not_covered.each{|slot|
             nodes = nodes_with_keys_in_slot(slot)
             slots[slot] = nodes
-            xputs "Slot #{slot} has keys in #{nodes.length} nodes: #{nodes.join}"
+            xputs "Slot #{slot} has keys in #{nodes.length} nodes: #{nodes.join(", ")}"
         }
 
         none = slots.select {|k,v| v.length == 0}
@@ -435,26 +481,50 @@ class RedisTrib
             xputs multi.keys.join(",")
             yes_or_die "Fix these slots by moving keys into a single node?"
             multi.each{|slot,nodes|
-                xputs ">>> Covering slot #{slot} moving keys to #{nodes[0]}"
-                # TODO
-                # 1) Set all nodes as "MIGRATING" for this slot, so that we
-                # can access keys in the hash slot using ASKING.
-                # 2) Move everything to node[0]
-                # 3) Clear MIGRATING from nodes, and ADDSLOTS the slot to
-                # node[0].
-                raise "TODO: Work in progress"
+                target = get_node_with_most_keys_in_slot(nodes,slot)
+                xputs ">>> Covering slot #{slot} moving keys to #{target}"
+
+                target.r.cluster('addslots',slot)
+                target.r.cluster('setslot',slot,'stable')
+                nodes.each{|src|
+                    next if src == target
+                    # Set the source node in 'importing' state (even if we will
+                    # actually migrate keys away) in order to avoid receiving
+                    # redirections for MIGRATE.
+                    src.r.cluster('setslot',slot,'importing',target.info[:name])
+                    move_slot(src,target,slot,:dots=>true,:fix=>true,:cold=>true)
+                    src.r.cluster('setslot',slot,'stable')
+                }
             }
         end
     end
 
     # Return the owner of the specified slot
-    def get_slot_owner(slot)
+    def get_slot_owners(slot)
+        owners = []
         @nodes.each{|n|
+            next if n.has_flag?("slave")
             n.slots.each{|s,_|
-                return n if s == slot
+                owners << n if s == slot
             }
         }
-        nil
+        owners
+    end
+
+    # Return the node, among 'nodes' with the greatest number of keys
+    # in the specified slot.
+    def get_node_with_most_keys_in_slot(nodes,slot)
+        best = nil
+        best_numkeys = 0
+        @nodes.each{|n|
+            next if n.has_flag?("slave")
+            numkeys = n.r.cluster("countkeysinslot",slot)
+            if numkeys > best_numkeys || best == nil
+                best = n
+                best_numkeys = numkeys
+            end
+        }
+        return best
     end
 
     # Slot 'slot' was found to be in importing or migrating state in one or
@@ -465,16 +535,8 @@ class RedisTrib
 
         # Try to obtain the current slot owner, according to the current
         # nodes configuration.
-        owner = get_slot_owner(slot)
-
-        # If there is no slot owner, set as owner the slot with the biggest
-        # number of keys, among the set of migrating / importing nodes.
-        if !owner
-            xputs "*** Fix me, some work to do here."
-            # Select owner...
-            # Use ADDSLOTS to assign the slot.
-            exit 1
-        end
+        owners = get_slot_owners(slot)
+        owner = owners[0] if owners.length == 1
 
         migrating = []
         importing = []
@@ -492,10 +554,57 @@ class RedisTrib
         puts "Set as migrating in: #{migrating.join(",")}"
         puts "Set as importing in: #{importing.join(",")}"
 
+        # If there is no slot owner, set as owner the slot with the biggest
+        # number of keys, among the set of migrating / importing nodes.
+        if !owner
+            xputs ">>> Nobody claims ownership, selecting an owner..."
+            owner = get_node_with_most_keys_in_slot(@nodes,slot)
+
+            # If we still don't have an owner, we can't fix it.
+            if !owner
+                xputs "[ERR] Can't select a slot owner. Impossible to fix."
+                exit 1
+            end
+
+            # Use ADDSLOTS to assign the slot.
+            puts "*** Configuring #{owner} as the slot owner"
+            owner.r.cluster("setslot",slot,"stable")
+            owner.r.cluster("addslots",slot)
+            # Make sure this information will propagate. Not strictly needed
+            # since there is no past owner, so all the other nodes will accept
+            # whatever epoch this node will claim the slot with.
+            owner.r.cluster("bumpepoch")
+
+            # Remove the owner from the list of migrating/importing
+            # nodes.
+            migrating.delete(owner)
+            importing.delete(owner)
+        end
+
+        # If there are multiple owners of the slot, we need to fix it
+        # so that a single node is the owner and all the other nodes
+        # are in importing state. Later the fix can be handled by one
+        # of the base cases above.
+        #
+        # Note that this case also covers multiple nodes having the slot
+        # in migrating state, since migrating is a valid state only for
+        # slot owners.
+        if owners.length > 1
+            owner = get_node_with_most_keys_in_slot(owners,slot)
+            owners.each{|n|
+                next if n == owner
+                n.r.cluster('delslots',slot)
+                n.r.cluster('setslot',slot,'importing',owner.info[:name])
+                importing.delete(n) # Avoid duplciates
+                importing << n
+            }
+            owner.r.cluster('bumpepoch')
+        end
+
         # Case 1: The slot is in migrating state in one slot, and in
         #         importing state in 1 slot. That's trivial to address.
         if migrating.length == 1 && importing.length == 1
-            move_slot(migrating[0],importing[0],slot,:verbose=>true,:fix=>true)
+            move_slot(migrating[0],importing[0],slot,:dots=>true,:fix=>true)
         # Case 2: There are multiple nodes that claim the slot as importing,
         # they probably got keys about the slot after a restart so opened
         # the slot. In this case we just move all the keys to the owner
@@ -504,7 +613,7 @@ class RedisTrib
             xputs ">>> Moving all the #{slot} slot keys to its owner #{owner}"
             importing.each {|node|
                 next if node == owner
-                move_slot(node,owner,slot,:verbose=>true,:fix=>true,:cold=>true)
+                move_slot(node,owner,slot,:dots=>true,:fix=>true,:cold=>true)
                 xputs ">>> Setting #{slot} as STABLE in #{node}"
                 node.r.cluster("setslot",slot,"stable")
             }
@@ -802,62 +911,238 @@ class RedisTrib
     # Options:
     # :verbose -- Print a dot for every moved key.
     # :fix     -- We are moving in the context of a fix. Use REPLACE.
-    # :cold    -- Move keys without opening / reconfiguring the nodes.
+    # :cold    -- Move keys without opening slots / reconfiguring the nodes.
+    # :update  -- Update nodes.info[:slots] for source/target nodes.
+    # :quiet   -- Don't print info messages.
     def move_slot(source,target,slot,o={})
+        o = {:pipeline => MigrateDefaultPipeline}.merge(o)
+
         # We start marking the slot as importing in the destination node,
         # and the slot as migrating in the target host. Note that the order of
         # the operations is important, as otherwise a client may be redirected
         # to the target node that does not yet know it is importing this slot.
-        print "Moving slot #{slot} from #{source} to #{target}: "; STDOUT.flush
+        if !o[:quiet]
+            print "Moving slot #{slot} from #{source} to #{target}: "
+            STDOUT.flush
+        end
+
         if !o[:cold]
             target.r.cluster("setslot",slot,"importing",source.info[:name])
             source.r.cluster("setslot",slot,"migrating",target.info[:name])
         end
         # Migrate all the keys from source to target using the MIGRATE command
         while true
-            keys = source.r.cluster("getkeysinslot",slot,10)
+            keys = source.r.cluster("getkeysinslot",slot,o[:pipeline])
             break if keys.length == 0
-            keys.each{|key|
-                begin
-                    source.r.client.call(["migrate",target.info[:host],target.info[:port],key,0,15000])
-                rescue => e
-                    if o[:fix] && e.to_s =~ /BUSYKEY/
-                        xputs "*** Target key #{key} exists. Replacing it for FIX."
-                        source.r.client.call(["migrate",target.info[:host],target.info[:port],key,0,15000,:replace])
-                    else
-                        puts ""
-                        xputs "[ERR] #{e}"
-                        exit 1
-                    end
+            begin
+                source.r.client.call(["migrate",target.info[:host],target.info[:port],"",0,@timeout,:keys,*keys])
+            rescue => e
+                if o[:fix] && e.to_s =~ /BUSYKEY/
+                    xputs "*** Target key exists. Replacing it for FIX."
+                    source.r.client.call(["migrate",target.info[:host],target.info[:port],"",0,@timeout,:replace,:keys,*keys])
+                else
+                    puts ""
+                    xputs "[ERR] Calling MIGRATE: #{e}"
+                    exit 1
                 end
-                print "." if o[:verbose]
-                STDOUT.flush
-            }
+            end
+            print "."*keys.length if o[:dots]
+            STDOUT.flush
         end
 
-        puts
+        puts if !o[:quiet]
         # Set the new node as the owner of the slot in all the known nodes.
         if !o[:cold]
             @nodes.each{|n|
+                next if n.has_flag?("slave")
                 n.r.cluster("setslot",slot,"node",target.info[:name])
             }
         end
+
+        # Update the node logical config
+        if o[:update] then
+            source.info[:slots].delete(slot)
+            target.info[:slots][slot] = true
+        end
     end
 
-    # redis-trib subcommands implementations
+    # redis-trib subcommands implementations.
 
     def check_cluster_cmd(argv,opt)
         load_cluster_info_from_node(argv[0])
         check_cluster
     end
 
+    def info_cluster_cmd(argv,opt)
+        load_cluster_info_from_node(argv[0])
+        show_cluster_info
+    end
+
+    def rebalance_cluster_cmd(argv,opt)
+        opt = {
+            'pipeline' => MigrateDefaultPipeline,
+            'threshold' => RebalanceDefaultThreshold
+        }.merge(opt)
+
+        # Load nodes info before parsing options, otherwise we can't
+        # handle --weight.
+        load_cluster_info_from_node(argv[0])
+
+        # Options parsing
+        threshold = opt['threshold'].to_i
+        autoweights = opt['auto-weights']
+        weights = {}
+        opt['weight'].each{|w|
+            fields = w.split("=")
+            node = get_node_by_abbreviated_name(fields[0])
+            if !node || !node.has_flag?("master")
+                puts "*** No such master node #{fields[0]}"
+                exit 1
+            end
+            weights[node.info[:name]] = fields[1].to_f
+        } if opt['weight']
+        useempty = opt['use-empty-masters']
+
+       # Assign a weight to each node, and compute the total cluster weight.
+        total_weight = 0
+        nodes_involved = 0
+        @nodes.each{|n|
+            if n.has_flag?("master")
+                next if !useempty && n.slots.length == 0
+                n.info[:w] = weights[n.info[:name]] ? weights[n.info[:name]] : 1
+                total_weight += n.info[:w]
+                nodes_involved += 1
+            end
+        }
+
+        # Check cluster, only proceed if it looks sane.
+        check_cluster(:quiet => true)
+        if @errors.length != 0
+            puts "*** Please fix your cluster problems before rebalancing"
+            exit 1
+        end
+
+        # Calculate the slots balance for each node. It's the number of
+        # slots the node should lose (if positive) or gain (if negative)
+        # in order to be balanced.
+        threshold = opt['threshold'].to_f
+        threshold_reached = false
+        @nodes.each{|n|
+            if n.has_flag?("master")
+                next if !n.info[:w]
+                expected = ((ClusterHashSlots.to_f / total_weight) *
+                            n.info[:w]).to_i
+                n.info[:balance] = n.slots.length - expected
+                # Compute the percentage of difference between the
+                # expected number of slots and the real one, to see
+                # if it's over the threshold specified by the user.
+                over_threshold = false
+                if threshold > 0
+                    if n.slots.length > 0
+                        err_perc = (100-(100.0*expected/n.slots.length)).abs
+                        over_threshold = true if err_perc > threshold
+                    elsif expected > 0
+                        over_threshold = true
+                    end
+                end
+                threshold_reached = true if over_threshold
+            end
+        }
+        if !threshold_reached
+            xputs "*** No rebalancing needed! All nodes are within the #{threshold}% threshold."
+            return
+        end
+
+        # Only consider nodes we want to change
+        sn = @nodes.select{|n|
+            n.has_flag?("master") && n.info[:w]
+        }
+
+        # Because of rounding, it is possible that the balance of all nodes
+        # summed does not give 0. Make sure that nodes that have to provide
+        # slots are always matched by nodes receiving slots.
+        total_balance = sn.map{|x| x.info[:balance]}.reduce{|a,b| a+b}
+        while total_balance > 0
+            sn.each{|n|
+                if n.info[:balance] < 0 && total_balance > 0
+                    n.info[:balance] -= 1
+                    total_balance -= 1
+                end
+            }
+        end
+
+        # Sort nodes by their slots balance.
+        sn = sn.sort{|a,b|
+            a.info[:balance] <=> b.info[:balance]
+        }
+
+        xputs ">>> Rebalancing across #{nodes_involved} nodes. Total weight = #{total_weight}"
+
+        if $verbose
+            sn.each{|n|
+                puts "#{n} balance is #{n.info[:balance]} slots"
+            }
+        end
+
+        # Now we have at the start of the 'sn' array nodes that should get
+        # slots, at the end nodes that must give slots.
+        # We take two indexes, one at the start, and one at the end,
+        # incrementing or decrementing the indexes accordingly til we
+        # find nodes that need to get/provide slots.
+        dst_idx = 0
+        src_idx = sn.length - 1
+
+        while dst_idx < src_idx
+            dst = sn[dst_idx]
+            src = sn[src_idx]
+            numslots = [dst.info[:balance],src.info[:balance]].map{|n|
+                n.abs
+            }.min
+
+            if numslots > 0
+                puts "Moving #{numslots} slots from #{src} to #{dst}"
+
+                # Actaully move the slots.
+                reshard_table = compute_reshard_table([src],numslots)
+                if reshard_table.length != numslots
+                    xputs "*** Assertio failed: Reshard table != number of slots"
+                    exit 1
+                end
+                if opt['simulate']
+                    print "#"*reshard_table.length
+                else
+                    reshard_table.each{|e|
+                        move_slot(e[:source],dst,e[:slot],
+                            :quiet=>true,
+                            :dots=>false,
+                            :update=>true,
+                            :pipeline=>opt['pipeline'])
+                        print "#"
+                        STDOUT.flush
+                    }
+                end
+                puts
+            end
+
+            # Update nodes balance.
+            dst.info[:balance] += numslots
+            src.info[:balance] -= numslots
+            dst_idx += 1 if dst.info[:balance] == 0
+            src_idx -= 1 if src.info[:balance] == 0
+        end
+    end
+
     def fix_cluster_cmd(argv,opt)
         @fix = true
+        @timeout = opt['timeout'].to_i if opt['timeout']
+
         load_cluster_info_from_node(argv[0])
         check_cluster
     end
 
     def reshard_cluster_cmd(argv,opt)
+        opt = {'pipeline' => MigrateDefaultPipeline}.merge(opt)
+
         load_cluster_info_from_node(argv[0])
         check_cluster
         if @errors.length != 0
@@ -865,6 +1150,8 @@ class RedisTrib
             exit 1
         end
 
+        @timeout = opt['timeout'].to_i if opt['timeout'].to_i
+
         # Get number of slots
         if opt['slots']
             numslots = opt['slots'].to_i
@@ -968,7 +1255,9 @@ class RedisTrib
             exit(1) if (yesno != "yes")
         end
         reshard_table.each{|e|
-            move_slot(e[:source],target,e[:slot],:verbose=>true)
+            move_slot(e[:source],target,e[:slot],
+                :dots=>true,
+                :pipeline=>opt['pipeline'])
         }
     end
 
@@ -1016,6 +1305,11 @@ class RedisTrib
         sleep 1
         wait_cluster_join
         flush_nodes_config # Useful for the replicas
+        # Reset the node information, so that when the
+        # final summary is listed in check_cluster about the newly created cluster
+        # all the nodes would get properly listed as slaves or masters
+        reset_nodes
+        load_cluster_info_from_node(argv[0])
         check_cluster
     end
 
@@ -1186,7 +1480,7 @@ class RedisTrib
                 print "Migrating #{k} to #{target}: "
                 STDOUT.flush
                 begin
-                    cmd = ["migrate",target.info[:host],target.info[:port],k,0,15000]
+                    cmd = ["migrate",target.info[:host],target.info[:port],k,0,@timeout]
                     cmd << :copy if use_copy
                     cmd << :replace if use_replace
                     source.client.call(cmd)
@@ -1214,17 +1508,32 @@ class RedisTrib
             if ARGV[idx][0..1] == "--"
                 option = ARGV[idx][2..-1]
                 idx += 1
+
+                # --verbose is a global option
+                if option == "verbose"
+                    $verbose = true
+                    next
+                end
+
                 if ALLOWED_OPTIONS[cmd] == nil || ALLOWED_OPTIONS[cmd][option] == nil
                     puts "Unknown option '#{option}' for command '#{cmd}'"
                     exit 1
                 end
-                if ALLOWED_OPTIONS[cmd][option]
+                if ALLOWED_OPTIONS[cmd][option] != false
                     value = ARGV[idx]
                     idx += 1
                 else
                     value = true
                 end
-                options[option] = value
+
+                # If the option is set to [], it's a multiple arguments
+                # option. We just queue every new value into an array.
+                if ALLOWED_OPTIONS[cmd][option] == []
+                    options[option] = [] if !options[option]
+                    options[option] << value
+                else
+                    options[option] = value
+                end
             else
                 # Remaining arguments are not options.
                 break
@@ -1336,8 +1645,10 @@ end
 COMMANDS={
     "create"  => ["create_cluster_cmd", -2, "host1:port1 ... hostN:portN"],
     "check"   => ["check_cluster_cmd", 2, "host:port"],
+    "info"    => ["info_cluster_cmd", 2, "host:port"],
     "fix"     => ["fix_cluster_cmd", 2, "host:port"],
     "reshard" => ["reshard_cluster_cmd", 2, "host:port"],
+    "rebalance" => ["rebalance_cluster_cmd", -2, "host:port"],
     "add-node" => ["addnode_cluster_cmd", 3, "new_host:new_port existing_host:existing_port"],
     "del-node" => ["delnode_cluster_cmd", 3, "host:port node_id"],
     "set-timeout" => ["set_timeout_cluster_cmd", 3, "host:port milliseconds"],
@@ -1350,7 +1661,9 @@ ALLOWED_OPTIONS={
     "create" => {"replicas" => true},
     "add-node" => {"slave" => false, "master-id" => true},
     "import" => {"from" => :required, "copy" => false, "replace" => false},
-    "reshard" => {"from" => true, "to" => true, "slots" => true, "yes" => false}
+    "reshard" => {"from" => true, "to" => true, "slots" => true, "yes" => false, "timeout" => true, "pipeline" => true},
+    "rebalance" => {"weight" => [], "auto-weights" => false, "use-empty-masters" => false, "timeout" => true, "simulate" => false, "pipeline" => true, "threshold" => true},
+    "fix" => {"timeout" => MigrateDefaultTimeout},
 }
 
 def show_help

src/redismodule.h

@@ -0,0 +1,308 @@
+#ifndef REDISMODULE_H
+#define REDISMODULE_H
+
+#include <sys/types.h>
+#include <stdint.h>
+#include <stdio.h>
+
+/* ---------------- Defines common between core and modules --------------- */
+
+/* Error status return values. */
+#define REDISMODULE_OK 0
+#define REDISMODULE_ERR 1
+
+/* API versions. */
+#define REDISMODULE_APIVER_1 1
+
+/* API flags and constants */
+#define REDISMODULE_READ (1<<0)
+#define REDISMODULE_WRITE (1<<1)
+
+#define REDISMODULE_LIST_HEAD 0
+#define REDISMODULE_LIST_TAIL 1
+
+/* Key types. */
+#define REDISMODULE_KEYTYPE_EMPTY 0
+#define REDISMODULE_KEYTYPE_STRING 1
+#define REDISMODULE_KEYTYPE_LIST 2
+#define REDISMODULE_KEYTYPE_HASH 3
+#define REDISMODULE_KEYTYPE_SET 4
+#define REDISMODULE_KEYTYPE_ZSET 5
+#define REDISMODULE_KEYTYPE_MODULE 6
+
+/* Reply types. */
+#define REDISMODULE_REPLY_UNKNOWN -1
+#define REDISMODULE_REPLY_STRING 0
+#define REDISMODULE_REPLY_ERROR 1
+#define REDISMODULE_REPLY_INTEGER 2
+#define REDISMODULE_REPLY_ARRAY 3
+#define REDISMODULE_REPLY_NULL 4
+
+/* Postponed array length. */
+#define REDISMODULE_POSTPONED_ARRAY_LEN -1
+
+/* Expire */
+#define REDISMODULE_NO_EXPIRE -1
+
+/* Sorted set API flags. */
+#define REDISMODULE_ZADD_XX      (1<<0)
+#define REDISMODULE_ZADD_NX      (1<<1)
+#define REDISMODULE_ZADD_ADDED   (1<<2)
+#define REDISMODULE_ZADD_UPDATED (1<<3)
+#define REDISMODULE_ZADD_NOP     (1<<4)
+
+/* Hash API flags. */
+#define REDISMODULE_HASH_NONE       0
+#define REDISMODULE_HASH_NX         (1<<0)
+#define REDISMODULE_HASH_XX         (1<<1)
+#define REDISMODULE_HASH_CFIELDS    (1<<2)
+#define REDISMODULE_HASH_EXISTS     (1<<3)
+
+/* A special pointer that we can use between the core and the module to signal
+ * field deletion, and that is impossible to be a valid pointer. */
+#define REDISMODULE_HASH_DELETE ((RedisModuleString*)(long)1)
+
+/* Error messages. */
+#define REDISMODULE_ERRORMSG_WRONGTYPE "WRONGTYPE Operation against a key holding the wrong kind of value"
+
+#define REDISMODULE_POSITIVE_INFINITE (1.0/0.0)
+#define REDISMODULE_NEGATIVE_INFINITE (-1.0/0.0)
+
+/* ------------------------- End of common defines ------------------------ */
+
+#ifndef REDISMODULE_CORE
+
+typedef long long mstime_t;
+
+/* Incomplete structures for compiler checks but opaque access. */
+typedef struct RedisModuleCtx RedisModuleCtx;
+typedef struct RedisModuleKey RedisModuleKey;
+typedef struct RedisModuleString RedisModuleString;
+typedef struct RedisModuleCallReply RedisModuleCallReply;
+typedef struct RedisModuleIO RedisModuleIO;
+typedef struct RedisModuleType RedisModuleType;
+typedef struct RedisModuleDigest RedisModuleDigest;
+
+typedef int (*RedisModuleCmdFunc) (RedisModuleCtx *ctx, RedisModuleString **argv, int argc);
+
+typedef void *(*RedisModuleTypeLoadFunc)(RedisModuleIO *rdb, int encver);
+typedef void (*RedisModuleTypeSaveFunc)(RedisModuleIO *rdb, void *value);
+typedef void (*RedisModuleTypeRewriteFunc)(RedisModuleIO *aof, RedisModuleString *key, void *value);
+typedef void (*RedisModuleTypeDigestFunc)(RedisModuleDigest *digest, void *value);
+typedef void (*RedisModuleTypeFreeFunc)(void *value);
+
+#define REDISMODULE_GET_API(name) \
+    RedisModule_GetApi("RedisModule_" #name, ((void **)&RedisModule_ ## name))
+
+#define REDISMODULE_API_FUNC(x) (*x)
+
+
+void *REDISMODULE_API_FUNC(RedisModule_Alloc)(size_t bytes);
+void *REDISMODULE_API_FUNC(RedisModule_Realloc)(void *ptr, size_t bytes);
+void REDISMODULE_API_FUNC(RedisModule_Free)(void *ptr);
+void *REDISMODULE_API_FUNC(RedisModule_Calloc)(size_t nmemb, size_t size);
+char *REDISMODULE_API_FUNC(RedisModule_Strdup)(const char *str);
+int REDISMODULE_API_FUNC(RedisModule_GetApi)(const char *, void *);
+int REDISMODULE_API_FUNC(RedisModule_CreateCommand)(RedisModuleCtx *ctx, const char *name, RedisModuleCmdFunc cmdfunc, const char *strflags, int firstkey, int lastkey, int keystep);
+int REDISMODULE_API_FUNC(RedisModule_SetModuleAttribs)(RedisModuleCtx *ctx, const char *name, int ver, int apiver);
+int REDISMODULE_API_FUNC(RedisModule_WrongArity)(RedisModuleCtx *ctx);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithLongLong)(RedisModuleCtx *ctx, long long ll);
+int REDISMODULE_API_FUNC(RedisModule_GetSelectedDb)(RedisModuleCtx *ctx);
+int REDISMODULE_API_FUNC(RedisModule_SelectDb)(RedisModuleCtx *ctx, int newid);
+void *REDISMODULE_API_FUNC(RedisModule_OpenKey)(RedisModuleCtx *ctx, RedisModuleString *keyname, int mode);
+void REDISMODULE_API_FUNC(RedisModule_CloseKey)(RedisModuleKey *kp);
+int REDISMODULE_API_FUNC(RedisModule_KeyType)(RedisModuleKey *kp);
+size_t REDISMODULE_API_FUNC(RedisModule_ValueLength)(RedisModuleKey *kp);
+int REDISMODULE_API_FUNC(RedisModule_ListPush)(RedisModuleKey *kp, int where, RedisModuleString *ele);
+RedisModuleString *REDISMODULE_API_FUNC(RedisModule_ListPop)(RedisModuleKey *key, int where);
+RedisModuleCallReply *REDISMODULE_API_FUNC(RedisModule_Call)(RedisModuleCtx *ctx, const char *cmdname, const char *fmt, ...);
+const char *REDISMODULE_API_FUNC(RedisModule_CallReplyProto)(RedisModuleCallReply *reply, size_t *len);
+void REDISMODULE_API_FUNC(RedisModule_FreeCallReply)(RedisModuleCallReply *reply);
+int REDISMODULE_API_FUNC(RedisModule_CallReplyType)(RedisModuleCallReply *reply);
+long long REDISMODULE_API_FUNC(RedisModule_CallReplyInteger)(RedisModuleCallReply *reply);
+size_t REDISMODULE_API_FUNC(RedisModule_CallReplyLength)(RedisModuleCallReply *reply);
+RedisModuleCallReply *REDISMODULE_API_FUNC(RedisModule_CallReplyArrayElement)(RedisModuleCallReply *reply, size_t idx);
+RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateString)(RedisModuleCtx *ctx, const char *ptr, size_t len);
+RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateStringFromLongLong)(RedisModuleCtx *ctx, long long ll);
+RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateStringFromString)(RedisModuleCtx *ctx, const RedisModuleString *str);
+RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateStringPrintf)(RedisModuleCtx *ctx, const char *fmt, ...);
+void REDISMODULE_API_FUNC(RedisModule_FreeString)(RedisModuleCtx *ctx, RedisModuleString *str);
+const char *REDISMODULE_API_FUNC(RedisModule_StringPtrLen)(const RedisModuleString *str, size_t *len);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithError)(RedisModuleCtx *ctx, const char *err);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithSimpleString)(RedisModuleCtx *ctx, const char *msg);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithArray)(RedisModuleCtx *ctx, long len);
+void REDISMODULE_API_FUNC(RedisModule_ReplySetArrayLength)(RedisModuleCtx *ctx, long len);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithStringBuffer)(RedisModuleCtx *ctx, const char *buf, size_t len);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithString)(RedisModuleCtx *ctx, RedisModuleString *str);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithNull)(RedisModuleCtx *ctx);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithDouble)(RedisModuleCtx *ctx, double d);
+int REDISMODULE_API_FUNC(RedisModule_ReplyWithCallReply)(RedisModuleCtx *ctx, RedisModuleCallReply *reply);
+int REDISMODULE_API_FUNC(RedisModule_StringToLongLong)(const RedisModuleString *str, long long *ll);
+int REDISMODULE_API_FUNC(RedisModule_StringToDouble)(const RedisModuleString *str, double *d);
+void REDISMODULE_API_FUNC(RedisModule_AutoMemory)(RedisModuleCtx *ctx);
+int REDISMODULE_API_FUNC(RedisModule_Replicate)(RedisModuleCtx *ctx, const char *cmdname, const char *fmt, ...);
+int REDISMODULE_API_FUNC(RedisModule_ReplicateVerbatim)(RedisModuleCtx *ctx);
+const char *REDISMODULE_API_FUNC(RedisModule_CallReplyStringPtr)(RedisModuleCallReply *reply, size_t *len);
+RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateStringFromCallReply)(RedisModuleCallReply *reply);
+int REDISMODULE_API_FUNC(RedisModule_DeleteKey)(RedisModuleKey *key);
+int REDISMODULE_API_FUNC(RedisModule_StringSet)(RedisModuleKey *key, RedisModuleString *str);
+char *REDISMODULE_API_FUNC(RedisModule_StringDMA)(RedisModuleKey *key, size_t *len, int mode);
+int REDISMODULE_API_FUNC(RedisModule_StringTruncate)(RedisModuleKey *key, size_t newlen);
+mstime_t REDISMODULE_API_FUNC(RedisModule_GetExpire)(RedisModuleKey *key);
+int REDISMODULE_API_FUNC(RedisModule_SetExpire)(RedisModuleKey *key, mstime_t expire);
+int REDISMODULE_API_FUNC(RedisModule_ZsetAdd)(RedisModuleKey *key, double score, RedisModuleString *ele, int *flagsptr);
+int REDISMODULE_API_FUNC(RedisModule_ZsetIncrby)(RedisModuleKey *key, double score, RedisModuleString *ele, int *flagsptr, double *newscore);
+int REDISMODULE_API_FUNC(RedisModule_ZsetScore)(RedisModuleKey *key, RedisModuleString *ele, double *score);
+int REDISMODULE_API_FUNC(RedisModule_ZsetRem)(RedisModuleKey *key, RedisModuleString *ele, int *deleted);
+void REDISMODULE_API_FUNC(RedisModule_ZsetRangeStop)(RedisModuleKey *key);
+int REDISMODULE_API_FUNC(RedisModule_ZsetFirstInScoreRange)(RedisModuleKey *key, double min, double max, int minex, int maxex);
+int REDISMODULE_API_FUNC(RedisModule_ZsetLastInScoreRange)(RedisModuleKey *key, double min, double max, int minex, int maxex);
+int REDISMODULE_API_FUNC(RedisModule_ZsetFirstInLexRange)(RedisModuleKey *key, RedisModuleString *min, RedisModuleString *max);
+int REDISMODULE_API_FUNC(RedisModule_ZsetLastInLexRange)(RedisModuleKey *key, RedisModuleString *min, RedisModuleString *max);
+RedisModuleString *REDISMODULE_API_FUNC(RedisModule_ZsetRangeCurrentElement)(RedisModuleKey *key, double *score);
+int REDISMODULE_API_FUNC(RedisModule_ZsetRangeNext)(RedisModuleKey *key);
+int REDISMODULE_API_FUNC(RedisModule_ZsetRangePrev)(RedisModuleKey *key);
+int REDISMODULE_API_FUNC(RedisModule_ZsetRangeEndReached)(RedisModuleKey *key);
+int REDISMODULE_API_FUNC(RedisModule_HashSet)(RedisModuleKey *key, int flags, ...);
+int REDISMODULE_API_FUNC(RedisModule_HashGet)(RedisModuleKey *key, int flags, ...);
+int REDISMODULE_API_FUNC(RedisModule_IsKeysPositionRequest)(RedisModuleCtx *ctx);
+void REDISMODULE_API_FUNC(RedisModule_KeyAtPos)(RedisModuleCtx *ctx, int pos);
+unsigned long long REDISMODULE_API_FUNC(RedisModule_GetClientId)(RedisModuleCtx *ctx);
+void *REDISMODULE_API_FUNC(RedisModule_PoolAlloc)(RedisModuleCtx *ctx, size_t bytes);
+RedisModuleType *REDISMODULE_API_FUNC(RedisModule_CreateDataType)(RedisModuleCtx *ctx, const char *name, int encver, RedisModuleTypeLoadFunc rdb_load, RedisModuleTypeSaveFunc rdb_save, RedisModuleTypeRewriteFunc aof_rewrite, RedisModuleTypeDigestFunc digest, RedisModuleTypeFreeFunc free);
+int REDISMODULE_API_FUNC(RedisModule_ModuleTypeSetValue)(RedisModuleKey *key, RedisModuleType *mt, void *value);
+RedisModuleType *REDISMODULE_API_FUNC(RedisModule_ModuleTypeGetType)(RedisModuleKey *key);
+void *REDISMODULE_API_FUNC(RedisModule_ModuleTypeGetValue)(RedisModuleKey *key);
+void REDISMODULE_API_FUNC(RedisModule_SaveUnsigned)(RedisModuleIO *io, uint64_t value);
+uint64_t REDISMODULE_API_FUNC(RedisModule_LoadUnsigned)(RedisModuleIO *io);
+void REDISMODULE_API_FUNC(RedisModule_SaveSigned)(RedisModuleIO *io, int64_t value);
+int64_t REDISMODULE_API_FUNC(RedisModule_LoadSigned)(RedisModuleIO *io);
+void REDISMODULE_API_FUNC(RedisModule_EmitAOF)(RedisModuleIO *io, const char *cmdname, const char *fmt, ...);
+void REDISMODULE_API_FUNC(RedisModule_SaveString)(RedisModuleIO *io, RedisModuleString *s);
+void REDISMODULE_API_FUNC(RedisModule_SaveStringBuffer)(RedisModuleIO *io, const char *str, size_t len);
+RedisModuleString *REDISMODULE_API_FUNC(RedisModule_LoadString)(RedisModuleIO *io);
+char *REDISMODULE_API_FUNC(RedisModule_LoadStringBuffer)(RedisModuleIO *io, size_t *lenptr);
+void REDISMODULE_API_FUNC(RedisModule_SaveDouble)(RedisModuleIO *io, double value);
+double REDISMODULE_API_FUNC(RedisModule_LoadDouble)(RedisModuleIO *io);
+void REDISMODULE_API_FUNC(RedisModule_SaveFloat)(RedisModuleIO *io, float value);
+float REDISMODULE_API_FUNC(RedisModule_LoadFloat)(RedisModuleIO *io);
+void REDISMODULE_API_FUNC(RedisModule_Log)(RedisModuleCtx *ctx, const char *level, const char *fmt, ...);
+void REDISMODULE_API_FUNC(RedisModule_LogIOError)(RedisModuleIO *io, const char *levelstr, const char *fmt, ...);
+int REDISMODULE_API_FUNC(RedisModule_StringAppendBuffer)(RedisModuleCtx *ctx, RedisModuleString *str, const char *buf, size_t len);
+void REDISMODULE_API_FUNC(RedisModule_RetainString)(RedisModuleCtx *ctx, RedisModuleString *str);
+int REDISMODULE_API_FUNC(RedisModule_StringCompare)(RedisModuleString *a, RedisModuleString *b);
+RedisModuleCtx *REDISMODULE_API_FUNC(RedisModule_GetContextFromIO)(RedisModuleIO *io);
+
+/* This is included inline inside each Redis module. */
+static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int apiver) __attribute__((unused));
+static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int apiver) {
+    void *getapifuncptr = ((void**)ctx)[0];
+    RedisModule_GetApi = (int (*)(const char *, void *)) (unsigned long)getapifuncptr;
+    REDISMODULE_GET_API(Alloc);
+    REDISMODULE_GET_API(Calloc);
+    REDISMODULE_GET_API(Free);
+    REDISMODULE_GET_API(Realloc);
+    REDISMODULE_GET_API(Strdup);
+    REDISMODULE_GET_API(CreateCommand);
+    REDISMODULE_GET_API(SetModuleAttribs);
+    REDISMODULE_GET_API(WrongArity);
+    REDISMODULE_GET_API(ReplyWithLongLong);
+    REDISMODULE_GET_API(ReplyWithError);
+    REDISMODULE_GET_API(ReplyWithSimpleString);
+    REDISMODULE_GET_API(ReplyWithArray);
+    REDISMODULE_GET_API(ReplySetArrayLength);
+    REDISMODULE_GET_API(ReplyWithStringBuffer);
+    REDISMODULE_GET_API(ReplyWithString);
+    REDISMODULE_GET_API(ReplyWithNull);
+    REDISMODULE_GET_API(ReplyWithCallReply);
+    REDISMODULE_GET_API(ReplyWithDouble);
+    REDISMODULE_GET_API(ReplySetArrayLength);
+    REDISMODULE_GET_API(GetSelectedDb);
+    REDISMODULE_GET_API(SelectDb);
+    REDISMODULE_GET_API(OpenKey);
+    REDISMODULE_GET_API(CloseKey);
+    REDISMODULE_GET_API(KeyType);
+    REDISMODULE_GET_API(ValueLength);
+    REDISMODULE_GET_API(ListPush);
+    REDISMODULE_GET_API(ListPop);
+    REDISMODULE_GET_API(StringToLongLong);
+    REDISMODULE_GET_API(StringToDouble);
+    REDISMODULE_GET_API(Call);
+    REDISMODULE_GET_API(CallReplyProto);
+    REDISMODULE_GET_API(FreeCallReply);
+    REDISMODULE_GET_API(CallReplyInteger);
+    REDISMODULE_GET_API(CallReplyType);
+    REDISMODULE_GET_API(CallReplyLength);
+    REDISMODULE_GET_API(CallReplyArrayElement);
+    REDISMODULE_GET_API(CallReplyStringPtr);
+    REDISMODULE_GET_API(CreateStringFromCallReply);
+    REDISMODULE_GET_API(CreateString);
+    REDISMODULE_GET_API(CreateStringFromLongLong);
+    REDISMODULE_GET_API(CreateStringFromString);
+    REDISMODULE_GET_API(CreateStringPrintf);
+    REDISMODULE_GET_API(FreeString);
+    REDISMODULE_GET_API(StringPtrLen);
+    REDISMODULE_GET_API(AutoMemory);
+    REDISMODULE_GET_API(Replicate);
+    REDISMODULE_GET_API(ReplicateVerbatim);
+    REDISMODULE_GET_API(DeleteKey);
+    REDISMODULE_GET_API(StringSet);
+    REDISMODULE_GET_API(StringDMA);
+    REDISMODULE_GET_API(StringTruncate);
+    REDISMODULE_GET_API(GetExpire);
+    REDISMODULE_GET_API(SetExpire);
+    REDISMODULE_GET_API(ZsetAdd);
+    REDISMODULE_GET_API(ZsetIncrby);
+    REDISMODULE_GET_API(ZsetScore);
+    REDISMODULE_GET_API(ZsetRem);
+    REDISMODULE_GET_API(ZsetRangeStop);
+    REDISMODULE_GET_API(ZsetFirstInScoreRange);
+    REDISMODULE_GET_API(ZsetLastInScoreRange);
+    REDISMODULE_GET_API(ZsetFirstInLexRange);
+    REDISMODULE_GET_API(ZsetLastInLexRange);
+    REDISMODULE_GET_API(ZsetRangeCurrentElement);
+    REDISMODULE_GET_API(ZsetRangeNext);
+    REDISMODULE_GET_API(ZsetRangePrev);
+    REDISMODULE_GET_API(ZsetRangeEndReached);
+    REDISMODULE_GET_API(HashSet);
+    REDISMODULE_GET_API(HashGet);
+    REDISMODULE_GET_API(IsKeysPositionRequest);
+    REDISMODULE_GET_API(KeyAtPos);
+    REDISMODULE_GET_API(GetClientId);
+    REDISMODULE_GET_API(PoolAlloc);
+    REDISMODULE_GET_API(CreateDataType);
+    REDISMODULE_GET_API(ModuleTypeSetValue);
+    REDISMODULE_GET_API(ModuleTypeGetType);
+    REDISMODULE_GET_API(ModuleTypeGetValue);
+    REDISMODULE_GET_API(SaveUnsigned);
+    REDISMODULE_GET_API(LoadUnsigned);
+    REDISMODULE_GET_API(SaveSigned);
+    REDISMODULE_GET_API(LoadSigned);
+    REDISMODULE_GET_API(SaveString);
+    REDISMODULE_GET_API(SaveStringBuffer);
+    REDISMODULE_GET_API(LoadString);
+    REDISMODULE_GET_API(LoadStringBuffer);
+    REDISMODULE_GET_API(SaveDouble);
+    REDISMODULE_GET_API(LoadDouble);
+    REDISMODULE_GET_API(SaveFloat);
+    REDISMODULE_GET_API(LoadFloat);
+    REDISMODULE_GET_API(EmitAOF);
+    REDISMODULE_GET_API(Log);
+    REDISMODULE_GET_API(LogIOError);
+    REDISMODULE_GET_API(StringAppendBuffer);
+    REDISMODULE_GET_API(RetainString);
+    REDISMODULE_GET_API(StringCompare);
+    REDISMODULE_GET_API(GetContextFromIO);
+
+    RedisModule_SetModuleAttribs(ctx,name,ver,apiver);
+    return REDISMODULE_OK;
+}
+
+#else
+
+/* Things only defined for the modules core, not exported to modules
+ * including this file. */
+#define RedisModuleString robj
+
+#endif /* REDISMODULE_CORE */
+#endif /* REDISMOUDLE_H */

src/replication.c

@@ -41,12 +41,13 @@ void replicationDiscardCachedMaster(void);
 void replicationResurrectCachedMaster(int newfd);
 void replicationSendAck(void);
 void putSlaveOnline(client *slave);
+int cancelReplicationHandshake(void);
 
 /* --------------------------- Utility functions ---------------------------- */
 
 /* Return the pointer to a string representing the slave ip:listening_port
  * pair. Mostly useful for logging, since we want to log a slave using its
- * IP address and it's listening port which is more clear for the user, for
+ * IP address and its listening port which is more clear for the user, for
  * example: "Closing connection with slave 10.1.2.3:6380". */
 char *replicationGetSlaveName(client *c) {
     static char buf[NET_PEER_ID_LEN];
@@ -54,7 +55,12 @@ char *replicationGetSlaveName(client *c) {
 
     ip[0] = '\0';
     buf[0] = '\0';
-    if (anetPeerToString(c->fd,ip,sizeof(ip),NULL) != -1) {
+    if (c->slave_ip[0] != '\0' ||
+        anetPeerToString(c->fd,ip,sizeof(ip),NULL) != -1)
+    {
+        /* Note that the 'ip' buffer is always larger than 'c->slave_ip' */
+        if (c->slave_ip[0] != '\0') memcpy(ip,c->slave_ip,sizeof(c->slave_ip));
+
         if (c->slave_listening_port)
             anetFormatAddr(buf,sizeof(buf),ip,c->slave_listening_port);
         else
@@ -646,7 +652,7 @@ void syncCommand(client *c) {
         } else {
             /* No way, we need to wait for the next BGSAVE in order to
              * register differences. */
-            serverLog(LL_NOTICE,"Waiting for next BGSAVE for SYNC");
+            serverLog(LL_NOTICE,"Can't attach the slave to the current BGSAVE. Waiting for next BGSAVE for SYNC");
         }
 
     /* CASE 2: BGSAVE is in progress, with socket target. */
@@ -656,7 +662,7 @@ void syncCommand(client *c) {
         /* There is an RDB child process but it is writing directly to
          * children sockets. We need to wait for the next BGSAVE
          * in order to synchronize. */
-        serverLog(LL_NOTICE,"Waiting for next BGSAVE for SYNC");
+        serverLog(LL_NOTICE,"Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC");
 
     /* CASE 3: There is no BGSAVE is progress. */
     } else {
@@ -665,12 +671,18 @@ void syncCommand(client *c) {
              * replicationCron() since we want to delay its start a
              * few seconds to wait for more slaves to arrive. */
             if (server.repl_diskless_sync_delay)
-                serverLog(LL_NOTICE,"Delay next BGSAVE for SYNC");
+                serverLog(LL_NOTICE,"Delay next BGSAVE for diskless SYNC");
         } else {
             /* Target is disk (or the slave is not capable of supporting
              * diskless replication) and we don't have a BGSAVE in progress,
              * let's start one. */
-            if (startBgsaveForReplication(c->slave_capa) != C_OK) return;
+            if (server.aof_child_pid == -1) {
+                startBgsaveForReplication(c->slave_capa);
+            } else {
+                serverLog(LL_NOTICE,
+                    "No BGSAVE in progress, but an AOF rewrite is active. "
+                    "BGSAVE for replication delayed");
+            }
         }
     }
 
@@ -710,6 +722,15 @@ void replconfCommand(client *c) {
                     &port,NULL) != C_OK))
                 return;
             c->slave_listening_port = port;
+        } else if (!strcasecmp(c->argv[j]->ptr,"ip-address")) {
+            sds ip = c->argv[j+1]->ptr;
+            if (sdslen(ip) < sizeof(c->slave_ip)) {
+                memcpy(c->slave_ip,ip,sdslen(ip)+1);
+            } else {
+                addReplyErrorFormat(c,"REPLCONF ip-address provided by "
+                    "slave instance is too long: %zd bytes", sdslen(ip));
+                return;
+            }
         } else if (!strcasecmp(c->argv[j]->ptr,"capa")) {
             /* Ignore capabilities not understood by this master. */
             if (!strcasecmp(c->argv[j+1]->ptr,"eof"))
@@ -916,18 +937,6 @@ int slaveIsInHandshakeState(void) {
            server.repl_state <= REPL_STATE_RECEIVE_PSYNC;
 }
 
-/* Abort the async download of the bulk dataset while SYNC-ing with master */
-void replicationAbortSyncTransfer(void) {
-    serverAssert(server.repl_state == REPL_STATE_TRANSFER);
-
-    aeDeleteFileEvent(server.el,server.repl_transfer_s,AE_READABLE);
-    close(server.repl_transfer_s);
-    close(server.repl_transfer_fd);
-    unlink(server.repl_transfer_tmpfile);
-    zfree(server.repl_transfer_tmpfile);
-    server.repl_state = REPL_STATE_CONNECT;
-}
-
 /* Avoid the master to detect the slave is timing out while loading the
  * RDB file in initial synchronization. We send a single newline character
  * that is valid protocol but is guaranteed to either be sent entierly or
@@ -1053,7 +1062,7 @@ void readSyncBulkPayload(aeEventLoop *el, int fd, void *privdata, int mask) {
     if (nread <= 0) {
         serverLog(LL_WARNING,"I/O error trying to sync with MASTER: %s",
             (nread == -1) ? strerror(errno) : "connection lost");
-        replicationAbortSyncTransfer();
+        cancelReplicationHandshake();
         return;
     }
     server.stat_net_input_bytes += nread;
@@ -1113,7 +1122,7 @@ void readSyncBulkPayload(aeEventLoop *el, int fd, void *privdata, int mask) {
     if (eof_reached) {
         if (rename(server.repl_transfer_tmpfile,server.rdb_filename) == -1) {
             serverLog(LL_WARNING,"Failed trying to rename the temp DB into dump.rdb in MASTER <-> SLAVE synchronization: %s", strerror(errno));
-            replicationAbortSyncTransfer();
+            cancelReplicationHandshake();
             return;
         }
         serverLog(LL_NOTICE, "MASTER <-> SLAVE sync: Flushing old data");
@@ -1130,7 +1139,7 @@ void readSyncBulkPayload(aeEventLoop *el, int fd, void *privdata, int mask) {
         serverLog(LL_NOTICE, "MASTER <-> SLAVE sync: Loading DB in memory");
         if (rdbLoad(server.rdb_filename) != C_OK) {
             serverLog(LL_WARNING,"Failed trying to load the MASTER synchronization DB from disk");
-            replicationAbortSyncTransfer();
+            cancelReplicationHandshake();
             return;
         }
         /* Final setup of the connected slave <- master link */
@@ -1159,7 +1168,7 @@ void readSyncBulkPayload(aeEventLoop *el, int fd, void *privdata, int mask) {
     return;
 
 error:
-    replicationAbortSyncTransfer();
+    cancelReplicationHandshake();
     return;
 }
 
@@ -1467,7 +1476,8 @@ void syncWithMaster(aeEventLoop *el, int fd, void *privdata, int mask) {
     /* Set the slave port, so that Master's INFO command can list the
      * slave listening port correctly. */
     if (server.repl_state == REPL_STATE_SEND_PORT) {
-        sds port = sdsfromlonglong(server.port);
+        sds port = sdsfromlonglong(server.slave_announce_port ?
+            server.slave_announce_port : server.port);
         err = sendSynchronousCommand(SYNC_CMD_WRITE,fd,"REPLCONF",
                 "listening-port",port, NULL);
         sdsfree(port);
@@ -1487,6 +1497,37 @@ void syncWithMaster(aeEventLoop *el, int fd, void *privdata, int mask) {
                                 "REPLCONF listening-port: %s", err);
         }
         sdsfree(err);
+        server.repl_state = REPL_STATE_SEND_IP;
+    }
+
+    /* Skip REPLCONF ip-address if there is no slave-announce-ip option set. */
+    if (server.repl_state == REPL_STATE_SEND_IP &&
+        server.slave_announce_ip == NULL)
+    {
+            server.repl_state = REPL_STATE_SEND_CAPA;
+    }
+
+    /* Set the slave ip, so that Master's INFO command can list the
+     * slave IP address port correctly in case of port forwarding or NAT. */
+    if (server.repl_state == REPL_STATE_SEND_IP) {
+        err = sendSynchronousCommand(SYNC_CMD_WRITE,fd,"REPLCONF",
+                "ip-address",server.slave_announce_ip, NULL);
+        if (err) goto write_error;
+        sdsfree(err);
+        server.repl_state = REPL_STATE_RECEIVE_IP;
+        return;
+    }
+
+    /* Receive REPLCONF ip-address reply. */
+    if (server.repl_state == REPL_STATE_RECEIVE_IP) {
+        err = sendSynchronousCommand(SYNC_CMD_READ,fd,NULL);
+        /* Ignore the error if any, not all the Redis versions support
+         * REPLCONF listening-port. */
+        if (err[0] == '-') {
+            serverLog(LL_NOTICE,"(Non critical) Master does not understand "
+                                "REPLCONF ip-address: %s", err);
+        }
+        sdsfree(err);
         server.repl_state = REPL_STATE_SEND_CAPA;
     }
 
@@ -1639,16 +1680,26 @@ int connectWithMaster(void) {
 }
 
 /* This function can be called when a non blocking connection is currently
- * in progress to undo it. */
+ * in progress to undo it.
+ * Never call this function directly, use cancelReplicationHandshake() instead.
+ */
 void undoConnectWithMaster(void) {
     int fd = server.repl_transfer_s;
 
-    serverAssert(server.repl_state == REPL_STATE_CONNECTING ||
-                 slaveIsInHandshakeState());
     aeDeleteFileEvent(server.el,fd,AE_READABLE|AE_WRITABLE);
     close(fd);
     server.repl_transfer_s = -1;
-    server.repl_state = REPL_STATE_CONNECT;
+}
+
+/* Abort the async download of the bulk dataset while SYNC-ing with master.
+ * Never call this function directly, use cancelReplicationHandshake() instead.
+ */
+void replicationAbortSyncTransfer(void) {
+    serverAssert(server.repl_state == REPL_STATE_TRANSFER);
+    undoConnectWithMaster();
+    close(server.repl_transfer_fd);
+    unlink(server.repl_transfer_tmpfile);
+    zfree(server.repl_transfer_tmpfile);
 }
 
 /* This function aborts a non blocking replication attempt if there is one
@@ -1662,10 +1713,12 @@ void undoConnectWithMaster(void) {
 int cancelReplicationHandshake(void) {
     if (server.repl_state == REPL_STATE_TRANSFER) {
         replicationAbortSyncTransfer();
+        server.repl_state = REPL_STATE_CONNECT;
     } else if (server.repl_state == REPL_STATE_CONNECTING ||
                slaveIsInHandshakeState())
     {
         undoConnectWithMaster();
+        server.repl_state = REPL_STATE_CONNECT;
     } else {
         return 0;
     }
@@ -1780,12 +1833,16 @@ void roleCommand(client *c) {
         listRewind(server.slaves,&li);
         while((ln = listNext(&li))) {
             client *slave = ln->value;
-            char ip[NET_IP_STR_LEN];
+            char ip[NET_IP_STR_LEN], *slaveip = slave->slave_ip;
 
-            if (anetPeerToString(slave->fd,ip,sizeof(ip),NULL) == -1) continue;
+            if (slaveip[0] == '\0') {
+                if (anetPeerToString(slave->fd,ip,sizeof(ip),NULL) == -1)
+                    continue;
+                slaveip = ip;
+            }
             if (slave->replstate != SLAVE_STATE_ONLINE) continue;
             addReplyMultiBulkLen(c,3);
-            addReplyBulkCString(c,ip);
+            addReplyBulkCString(c,slaveip);
             addReplyBulkLongLong(c,slave->slave_listening_port);
             addReplyBulkLongLong(c,slave->repl_ack_off);
             slaves++;
@@ -2189,7 +2246,7 @@ void replicationCron(void) {
          (time(NULL)-server.repl_transfer_lastio) > server.repl_timeout)
     {
         serverLog(LL_WARNING,"Timeout connecting to the MASTER...");
-        undoConnectWithMaster();
+        cancelReplicationHandshake();
     }
 
     /* Bulk transfer I/O timeout? */
@@ -2197,7 +2254,7 @@ void replicationCron(void) {
         (time(NULL)-server.repl_transfer_lastio) > server.repl_timeout)
     {
         serverLog(LL_WARNING,"Timeout receiving bulk data from MASTER... If the problem persists try to set the 'repl-timeout' parameter in redis.conf to a larger value.");
-        replicationAbortSyncTransfer();
+        cancelReplicationHandshake();
     }
 
     /* Timed out master when we are an already connected slave? */
@@ -2306,13 +2363,12 @@ void replicationCron(void) {
         replicationScriptCacheFlush();
     }
 
-    /* If we are using diskless replication and there are slaves waiting
-     * in WAIT_BGSAVE_START state, check if enough seconds elapsed and
-     * start a BGSAVE.
+    /* Start a BGSAVE good for replication if we have slaves in
+     * WAIT_BGSAVE_START state.
      *
-     * This code is also useful to trigger a BGSAVE if the diskless
-     * replication was turned off with CONFIG SET, while there were already
-     * slaves in WAIT_BGSAVE_START state. */
+     * In case of diskless replication, we make sure to wait the specified
+     * number of seconds (according to configuration) so that other slaves
+     * have the time to arrive before we start streaming. */
     if (server.rdb_child_pid == -1 && server.aof_child_pid == -1) {
         time_t idle, max_idle = 0;
         int slaves_waiting = 0;
@@ -2332,9 +2388,13 @@ void replicationCron(void) {
             }
         }
 
-        if (slaves_waiting && max_idle > server.repl_diskless_sync_delay) {
-            /* Start a BGSAVE. Usually with socket target, or with disk target
-             * if there was a recent socket -> disk config change. */
+        if (slaves_waiting &&
+            (!server.repl_diskless_sync ||
+             max_idle > server.repl_diskless_sync_delay))
+        {
+            /* Start the BGSAVE. The called function may start a
+             * BGSAVE with socket target or disk target depending on the
+             * configuration and slaves capabilities. */
             startBgsaveForReplication(mincapa);
         }
     }

src/rio.c

@@ -163,7 +163,7 @@ void rioInitWithFile(rio *r, FILE *fp) {
  * The function returns success as long as we are able to correctly write
  * to at least one file descriptor.
  *
- * When buf is NULL adn len is 0, the function performs a flush operation
+ * When buf is NULL and len is 0, the function performs a flush operation
  * if there is some pending buffer, so this function is also used in order
  * to implement rioFdsetFlush(). */
 static size_t rioFdsetWrite(rio *r, const void *buf, size_t len) {
@@ -176,7 +176,7 @@ static size_t rioFdsetWrite(rio *r, const void *buf, size_t len) {
      * a given size, we actually write to the sockets. */
     if (len) {
         r->io.fdset.buf = sdscatlen(r->io.fdset.buf,buf,len);
-        len = 0; /* Prevent entering the while belove if we don't flush. */
+        len = 0; /* Prevent entering the while below if we don't flush. */
         if (sdslen(r->io.fdset.buf) > PROTO_IOBUF_LEN) doflush = 1;
     }
 
@@ -276,6 +276,7 @@ void rioInitWithFdset(rio *r, int *fds, int numfds) {
     r->io.fdset.buf = sdsempty();
 }
 
+/* release the rio stream. */
 void rioFreeFdset(rio *r) {
     zfree(r->io.fdset.fds);
     zfree(r->io.fdset.state);

src/rio.h

@@ -128,11 +128,16 @@ void rioInitWithFile(rio *r, FILE *fp);
 void rioInitWithBuffer(rio *r, sds s);
 void rioInitWithFdset(rio *r, int *fds, int numfds);
 
+void rioFreeFdset(rio *r);
+
 size_t rioWriteBulkCount(rio *r, char prefix, int count);
 size_t rioWriteBulkString(rio *r, const char *buf, size_t len);
 size_t rioWriteBulkLongLong(rio *r, long long l);
 size_t rioWriteBulkDouble(rio *r, double d);
 
+struct redisObject;
+int rioWriteBulkObject(rio *r, struct redisObject *obj);
+
 void rioGenericUpdateChecksum(rio *r, const void *buf, size_t len);
 void rioSetAutoSync(rio *r, off_t bytes);
 

src/scripting.c

@@ -463,7 +463,7 @@ int luaRedisGenericCommand(lua_State *lua, int raise_error) {
             luaPushError(lua,"Unknown Redis command called from Lua script");
         goto cleanup;
     }
-    c->cmd = cmd;
+    c->cmd = c->lastcmd = cmd;
 
     /* There are commands that are not allowed inside scripts. */
     if (cmd->flags & CMD_NOSCRIPT) {
@@ -512,8 +512,10 @@ int luaRedisGenericCommand(lua_State *lua, int raise_error) {
 
     /* If this is a Redis Cluster node, we need to make sure Lua is not
      * trying to access non-local keys, with the exception of commands
-     * received from our master. */
-    if (server.cluster_enabled && !(server.lua_caller->flags & CLIENT_MASTER)) {
+     * received from our master or when loading the AOF back in memory. */
+    if (server.cluster_enabled && !server.loading &&
+        !(server.lua_caller->flags & CLIENT_MASTER))
+    {
         /* Duplicate relevant flags in the lua client. */
         c->flags &= ~(CLIENT_READONLY|CLIENT_ASKING);
         c->flags |= server.lua_caller->flags & (CLIENT_READONLY|CLIENT_ASKING);
@@ -837,6 +839,8 @@ void luaLoadLibraries(lua_State *lua) {
 void luaRemoveUnsupportedFunctions(lua_State *lua) {
     lua_pushnil(lua);
     lua_setglobal(lua,"loadfile");
+    lua_pushnil(lua);
+    lua_setglobal(lua,"dofile");
 }
 
 /* This function installs metamethods in the global table _G that prevent
@@ -865,7 +869,7 @@ void scriptingEnableGlobalsProtection(lua_State *lua) {
     s[j++]="end\n";
     s[j++]="mt.__index = function (t, n)\n";
     s[j++]="  if dbg.getinfo(2) and dbg.getinfo(2, \"S\").what ~= \"C\" then\n";
-    s[j++]="    error(\"Script attempted to access unexisting global variable '\"..tostring(n)..\"'\", 2)\n";
+    s[j++]="    error(\"Script attempted to access nonexistent global variable '\"..tostring(n)..\"'\", 2)\n";
     s[j++]="  end\n";
     s[j++]="  return rawget(t, n)\n";
     s[j++]="end\n";
@@ -926,19 +930,19 @@ void scriptingInit(int setup) {
     lua_pushcfunction(lua,luaLogCommand);
     lua_settable(lua,-3);
 
-    lua_pushstring(lua,"LL_DEBUG");
+    lua_pushstring(lua,"LOG_DEBUG");
     lua_pushnumber(lua,LL_DEBUG);
     lua_settable(lua,-3);
 
-    lua_pushstring(lua,"LL_VERBOSE");
+    lua_pushstring(lua,"LOG_VERBOSE");
     lua_pushnumber(lua,LL_VERBOSE);
     lua_settable(lua,-3);
 
-    lua_pushstring(lua,"LL_NOTICE");
+    lua_pushstring(lua,"LOG_NOTICE");
     lua_pushnumber(lua,LL_NOTICE);
     lua_settable(lua,-3);
 
-    lua_pushstring(lua,"LL_WARNING");
+    lua_pushstring(lua,"LOG_WARNING");
     lua_pushnumber(lua,LL_WARNING);
     lua_settable(lua,-3);
 
@@ -1145,7 +1149,7 @@ int luaCreateFunction(client *c, lua_State *lua, char *funcname, robj *body) {
     funcdef = sdscatlen(funcdef,funcname,42);
     funcdef = sdscatlen(funcdef,"() ",3);
     funcdef = sdscatlen(funcdef,body->ptr,sdslen(body->ptr));
-    funcdef = sdscatlen(funcdef," end",4);
+    funcdef = sdscatlen(funcdef,"\nend",4);
 
     if (luaL_loadbuffer(lua,funcdef,sdslen(funcdef),"@user_script")) {
         addReplyErrorFormat(c,"Error compiling script (new function): %s\n",
@@ -1566,7 +1570,11 @@ void ldbSendLogs(void) {
         proto = sdscatlen(proto,"\r\n",2);
         listDelNode(ldb.logs,ln);
     }
-    write(ldb.fd,proto,sdslen(proto));
+    if (write(ldb.fd,proto,sdslen(proto)) == -1) {
+        /* Avoid warning. We don't check the return value of write()
+         * since the next read() will catch the I/O error and will
+         * close the debugging session. */
+    }
     sdsfree(proto);
 }
 
@@ -1842,9 +1850,13 @@ void ldbList(int around, int context) {
  *
  * The element is not automatically removed from the stack, nor it is
  * converted to a different type. */
-sds ldbCatStackValue(sds s, lua_State *lua, int idx) {
+#define LDB_MAX_VALUES_DEPTH (LUA_MINSTACK/2)
+sds ldbCatStackValueRec(sds s, lua_State *lua, int idx, int level) {
     int t = lua_type(lua,idx);
 
+    if (level++ == LDB_MAX_VALUES_DEPTH)
+        return sdscat(s,"<max recursion level reached! Nested table?>");
+
     switch(t) {
     case LUA_TSTRING:
         {
@@ -1879,12 +1891,13 @@ sds ldbCatStackValue(sds s, lua_State *lua, int idx) {
                  lua_tonumber(lua,-2) != expected_index)) is_array = 0;
             /* Stack now: table, key, value */
             /* Array repr. */
-            repr1 = ldbCatStackValue(repr1,lua,-1);
+            repr1 = ldbCatStackValueRec(repr1,lua,-1,level);
             repr1 = sdscatlen(repr1,"; ",2);
             /* Full repr. */
-            repr2 = ldbCatStackValue(repr2,lua,-2);
-            repr2 = sdscatlen(repr2,"=",1);
-            repr2 = ldbCatStackValue(repr2,lua,-1);
+            repr2 = sdscatlen(repr2,"[",1);
+            repr2 = ldbCatStackValueRec(repr2,lua,-2,level);
+            repr2 = sdscatlen(repr2,"]=",2);
+            repr2 = ldbCatStackValueRec(repr2,lua,-1,level);
             repr2 = sdscatlen(repr2,"; ",2);
             lua_pop(lua,1); /* Stack: table, key. Ready for next iteration. */
             expected_index++;
@@ -1911,16 +1924,22 @@ sds ldbCatStackValue(sds s, lua_State *lua, int idx) {
         else if (t == LUA_TUSERDATA) typename = "userdata";
         else if (t == LUA_TTHREAD) typename = "thread";
         else if (t == LUA_TLIGHTUSERDATA) typename = "light-userdata";
-        s = sdscatprintf(s,"%s@%p",typename,p);
+        s = sdscatprintf(s,"\"%s@%p\"",typename,p);
         }
         break;
     default:
-        s = sdscat(s,"<unknown-lua-type>");
+        s = sdscat(s,"\"<unknown-lua-type>\"");
         break;
     }
     return s;
 }
 
+/* Higher level wrapper for ldbCatStackValueRec() that just uses an initial
+ * recursion level of '0'. */
+sds ldbCatStackValue(sds s, lua_State *lua, int idx) {
+    return ldbCatStackValueRec(s,lua,idx,0);
+}
+
 /* Produce a debugger log entry representing the value of the Lua object
  * currently on the top of the stack. The element is ot popped nor modified.
  * Check ldbCatStackValue() for the actual implementation. */

src/sds.c

@@ -35,6 +35,7 @@
 #include <string.h>
 #include <ctype.h>
 #include <assert.h>
+#include <limits.h>
 #include "sds.h"
 #include "sdsalloc.h"
 
@@ -55,14 +56,16 @@ static inline int sdsHdrSize(char type) {
 }
 
 static inline char sdsReqType(size_t string_size) {
-    if (string_size < 32)
+    if (string_size < 1<<5)
         return SDS_TYPE_5;
-    if (string_size < 0xff)
+    if (string_size < 1<<8)
         return SDS_TYPE_8;
-    if (string_size < 0xffff)
+    if (string_size < 1<<16)
         return SDS_TYPE_16;
-    if (string_size < 0xffffffff)
+#if (LONG_MAX == LLONG_MAX)
+    if (string_size < 1ll<<32)
         return SDS_TYPE_32;
+#endif
     return SDS_TYPE_64;
 }
 

src/sentinel.c

@@ -518,7 +518,7 @@ sentinelAddr *createSentinelAddr(char *hostname, int port) {
     char ip[NET_IP_STR_LEN];
     sentinelAddr *sa;
 
-    if (port <= 0 || port > 65535) {
+    if (port < 0 || port > 65535) {
         errno = EINVAL;
         return NULL;
     }
@@ -1062,11 +1062,18 @@ int sentinelUpdateSentinelAddressInAllMasters(sentinelRedisInstance *ri) {
         sentinelRedisInstance *master = dictGetVal(de), *match;
         match = getSentinelRedisInstanceByAddrAndRunID(master->sentinels,
                                                        NULL,0,ri->runid);
-        if (match->link->disconnected == 0) {
+        /* If there is no match, this master does not know about this
+         * Sentinel, try with the next one. */
+        if (match == NULL) continue;
+
+        /* Disconnect the old links if connected. */
+        if (match->link->cc != NULL)
             instanceLinkCloseConnection(match->link,match->link->cc);
+        if (match->link->pc != NULL)
             instanceLinkCloseConnection(match->link,match->link->pc);
-        }
+
         if (match == ri) continue; /* Address already updated for it. */
+
         /* Update the address of the matching Sentinel by copying the address
          * of the Sentinel object that received the address update. */
         releaseSentinelAddr(match->addr);
@@ -1132,7 +1139,8 @@ void sentinelDisconnectCallback(const redisAsyncContext *c, int status) {
  * createSentinelAddr() function.
  *
  * The function may also fail and return NULL with errno set to EBUSY if
- * a master or slave with the same name already exists. */
+ * a master with the same name, a slave with the same address, or a sentinel
+ * with the same ID already exists. */
 
 sentinelRedisInstance *createSentinelRedisInstance(char *name, int flags, char *hostname, int port, int quorum, sentinelRedisInstance *master) {
     sentinelRedisInstance *ri;
@@ -1147,8 +1155,8 @@ sentinelRedisInstance *createSentinelRedisInstance(char *name, int flags, char *
     addr = createSentinelAddr(hostname,port);
     if (addr == NULL) return NULL;
 
-    /* For slaves and sentinel we use ip:port as name. */
-    if (flags & (SRI_SLAVE|SRI_SENTINEL)) {
+    /* For slaves use ip:port as name. */
+    if (flags & SRI_SLAVE) {
         anetFormatAddr(slavename, sizeof(slavename), hostname, port);
         name = slavename;
     }
@@ -1657,15 +1665,15 @@ char *sentinelHandleConfiguration(char **argv, int argc) {
                (argc == 4 || argc == 5)) {
         sentinelRedisInstance *si;
 
-        /* known-sentinel <name> <ip> <port> [runid] */
-        ri = sentinelGetMasterByName(argv[1]);
-        if (!ri) return "No such master with specified name.";
-        if ((si = createSentinelRedisInstance(NULL,SRI_SENTINEL,argv[2],
-                    atoi(argv[3]), ri->quorum, ri)) == NULL)
-        {
-            return "Wrong hostname or port for sentinel.";
-        }
-        if (argc == 5) {
+        if (argc == 5) { /* Ignore the old form without runid. */
+            /* known-sentinel <name> <ip> <port> [runid] */
+            ri = sentinelGetMasterByName(argv[1]);
+            if (!ri) return "No such master with specified name.";
+            if ((si = createSentinelRedisInstance(argv[4],SRI_SENTINEL,argv[2],
+                        atoi(argv[3]), ri->quorum, ri)) == NULL)
+            {
+                return "Wrong hostname or port for sentinel.";
+            }
             si->runid = sdsnew(argv[4]);
             sentinelTryConnectionSharing(si);
         }
@@ -1796,11 +1804,10 @@ void rewriteConfigSentinelOption(struct rewriteConfigState *state) {
         di2 = dictGetIterator(master->sentinels);
         while((de = dictNext(di2)) != NULL) {
             ri = dictGetVal(de);
+            if (ri->runid == NULL) continue;
             line = sdscatprintf(sdsempty(),
-                "sentinel known-sentinel %s %s %d%s%s",
-                master->name, ri->addr->ip, ri->addr->port,
-                ri->runid ? " " : "",
-                ri->runid ? ri->runid : "");
+                "sentinel known-sentinel %s %s %d %s",
+                master->name, ri->addr->ip, ri->addr->port, ri->runid);
             rewriteConfigRewriteLine(state,"sentinel",line,1);
         }
         dictReleaseIterator(di2);
@@ -1895,6 +1902,7 @@ void sentinelSetClientName(sentinelRedisInstance *ri, redisAsyncContext *c, char
  * one of the two links (commands and pub/sub) is missing. */
 void sentinelReconnectInstance(sentinelRedisInstance *ri) {
     if (ri->link->disconnected == 0) return;
+    if (ri->addr->port == 0) return; /* port == 0 means invalid address. */
     instanceLink *link = ri->link;
     mstime_t now = mstime();
 
@@ -1909,6 +1917,7 @@ void sentinelReconnectInstance(sentinelRedisInstance *ri) {
                 link->cc->errstr);
             instanceLinkCloseConnection(link,link->cc);
         } else {
+            link->pending_commands = 0;
             link->cc_conn_time = mstime();
             link->cc->data = link;
             redisAeAttach(server.el,link->cc);
@@ -2340,11 +2349,25 @@ void sentinelProcessHelloMessage(char *hello, int hello_len) {
             if (removed) {
                 sentinelEvent(LL_NOTICE,"+sentinel-address-switch",master,
                     "%@ ip %s port %d for %s", token[0],port,token[2]);
+            } else {
+                /* Check if there is another Sentinel with the same address this
+                 * new one is reporting. What we do if this happens is to set its
+                 * port to 0, to signal the address is invalid. We'll update it
+                 * later if we get an HELLO message. */
+                sentinelRedisInstance *other =
+                    getSentinelRedisInstanceByAddrAndRunID(
+                        master->sentinels, token[0],port,NULL);
+                if (other) {
+                    sentinelEvent(LL_NOTICE,"+sentinel-invalid-addr",other,"%@");
+                    other->addr->port = 0; /* It means: invalid address. */
+                    sentinelUpdateSentinelAddressInAllMasters(other);
+                }
             }
 
             /* Add the new sentinel. */
-            si = createSentinelRedisInstance(NULL,SRI_SENTINEL,
+            si = createSentinelRedisInstance(token[2],SRI_SENTINEL,
                             token[0],port,master->quorum,master);
+
             if (si) {
                 if (!removed) sentinelEvent(LL_NOTICE,"+sentinel",si,"%@");
                 /* The runid is NULL after a new instance creation and
@@ -2366,7 +2389,7 @@ void sentinelProcessHelloMessage(char *hello, int hello_len) {
         }
 
         /* Update master info if received configuration is newer. */
-        if (master->config_epoch < master_config_epoch) {
+        if (si && master->config_epoch < master_config_epoch) {
             master->config_epoch = master_config_epoch;
             if (master_port != master->addr->port ||
                 strcmp(master->addr->ip, token[5]))
@@ -2556,9 +2579,15 @@ void sentinelSendPeriodicCommands(sentinelRedisInstance *ri) {
     /* If this is a slave of a master in O_DOWN condition we start sending
      * it INFO every second, instead of the usual SENTINEL_INFO_PERIOD
      * period. In this state we want to closely monitor slaves in case they
-     * are turned into masters by another Sentinel, or by the sysadmin. */
+     * are turned into masters by another Sentinel, or by the sysadmin.
+     *
+     * Similarly we monitor the INFO output more often if the slave reports
+     * to be disconnected from the master, so that we can have a fresh
+     * disconnection time figure. */
     if ((ri->flags & SRI_SLAVE) &&
-        (ri->master->flags & (SRI_O_DOWN|SRI_FAILOVER_IN_PROGRESS))) {
+        ((ri->master->flags & (SRI_O_DOWN|SRI_FAILOVER_IN_PROGRESS)) ||
+         (ri->master_link_down_time != 0)))
+    {
         info_period = 1000;
     } else {
         info_period = SENTINEL_INFO_PERIOD;
@@ -3372,6 +3401,8 @@ void sentinelCheckSubjectivelyDown(sentinelRedisInstance *ri) {
 
     if (ri->link->act_ping_time)
         elapsed = mstime() - ri->link->act_ping_time;
+    else if (ri->link->disconnected)
+        elapsed = mstime() - ri->link->last_avail_time;
 
     /* Check if we are in need for a reconnection of one of the
      * links, because we are detecting low activity.
@@ -3609,15 +3640,15 @@ struct sentinelLeader {
 /* Helper function for sentinelGetLeader, increment the counter
  * relative to the specified runid. */
 int sentinelLeaderIncr(dict *counters, char *runid) {
-    dictEntry *de = dictFind(counters,runid);
+    dictEntry *existing, *de;
     uint64_t oldval;
 
-    if (de) {
-        oldval = dictGetUnsignedIntegerVal(de);
-        dictSetUnsignedIntegerVal(de,oldval+1);
+    de = dictAddRaw(counters,runid,&existing);
+    if (existing) {
+        oldval = dictGetUnsignedIntegerVal(existing);
+        dictSetUnsignedIntegerVal(existing,oldval+1);
         return oldval+1;
     } else {
-        de = dictAddRaw(counters,runid);
         serverAssert(de != NULL);
         dictSetUnsignedIntegerVal(de,1);
         return 1;
@@ -3643,7 +3674,7 @@ char *sentinelGetLeader(sentinelRedisInstance *master, uint64_t epoch) {
     serverAssert(master->flags & (SRI_O_DOWN|SRI_FAILOVER_IN_PROGRESS));
     counters = dictCreate(&leaderVotesDictType,NULL);
 
-    voters = dictSize(master->sentinels)+1; /* All the other sentinels and me. */
+    voters = dictSize(master->sentinels)+1; /* All the other sentinels and me.*/
 
     /* Count other sentinels votes */
     di = dictGetIterator(master->sentinels);
@@ -3857,11 +3888,11 @@ int compareSlavesForPromotion(const void *a, const void *b) {
         return (*sa)->slave_priority - (*sb)->slave_priority;
 
     /* If priority is the same, select the slave with greater replication
-     * offset (processed more data frmo the master). */
+     * offset (processed more data from the master). */
     if ((*sa)->slave_repl_offset > (*sb)->slave_repl_offset) {
         return -1; /* a < b */
     } else if ((*sa)->slave_repl_offset < (*sb)->slave_repl_offset) {
-        return 1; /* b > a */
+        return 1; /* a > b */
     }
 
     /* If the replication offset is the same select the slave with that has
@@ -3979,7 +4010,7 @@ void sentinelFailoverSendSlaveOfNoOne(sentinelRedisInstance *ri) {
     /* We can't send the command to the promoted slave if it is now
      * disconnected. Retry again and again with this state until the timeout
      * is reached, then abort the failover. */
-    if (ri->link->disconnected) {
+    if (ri->promoted_slave->link->disconnected) {
         if (mstime() - ri->failover_state_change_time > ri->failover_timeout) {
             sentinelEvent(LL_WARNING,"-failover-abort-slave-timeout",ri,"%@");
             sentinelAbortFailover(ri);

src/server.h

@@ -33,6 +33,7 @@
 #include "fmacros.h"
 #include "config.h"
 #include "solarisfixes.h"
+#include "rio.h"
 
 #include <stdio.h>
 #include <stdlib.h>
@@ -61,7 +62,7 @@ typedef long long mstime_t; /* millisecond time type. */
 #include "version.h" /* Version macro */
 #include "util.h"    /* Misc functions useful in many places */
 #include "latency.h" /* Latency monitor API */
-#include "sparkline.h" /* ASII graphs API */
+#include "sparkline.h" /* ASCII graphs API */
 #include "quicklist.h"
 
 /* Following includes allow test functions to be called from Redis main() */
@@ -92,6 +93,7 @@ typedef long long mstime_t; /* millisecond time type. */
 #define AOF_REWRITE_PERC  100
 #define AOF_REWRITE_MIN_SIZE (64*1024*1024)
 #define AOF_REWRITE_ITEMS_PER_CMD 64
+#define AOF_READ_DIFF_INTERVAL_BYTES (1024*10)
 #define CONFIG_DEFAULT_SLOWLOG_LOG_SLOWER_THAN 10000
 #define CONFIG_DEFAULT_SLOWLOG_MAX_LEN 128
 #define CONFIG_DEFAULT_MAX_CLIENTS 10000
@@ -108,9 +110,13 @@ typedef long long mstime_t; /* millisecond time type. */
 #define CONFIG_DEFAULT_PID_FILE "/var/run/redis.pid"
 #define CONFIG_DEFAULT_SYSLOG_IDENT "redis"
 #define CONFIG_DEFAULT_CLUSTER_CONFIG_FILE "nodes.conf"
+#define CONFIG_DEFAULT_CLUSTER_ANNOUNCE_IP NULL         /* Auto detect. */
+#define CONFIG_DEFAULT_CLUSTER_ANNOUNCE_PORT 0          /* Use server.port */
+#define CONFIG_DEFAULT_CLUSTER_ANNOUNCE_BUS_PORT 0      /* Use +10000 offset. */
 #define CONFIG_DEFAULT_DAEMONIZE 0
 #define CONFIG_DEFAULT_UNIX_SOCKET_PERM 0
-#define CONFIG_DEFAULT_TCP_KEEPALIVE 0
+#define CONFIG_DEFAULT_TCP_KEEPALIVE 300
+#define CONFIG_DEFAULT_PROTECTED_MODE 1
 #define CONFIG_DEFAULT_LOGFILE ""
 #define CONFIG_DEFAULT_SYSLOG_ENABLED 0
 #define CONFIG_DEFAULT_STOP_WRITES_ON_BGSAVE_ERROR 1
@@ -121,12 +127,17 @@ typedef long long mstime_t; /* millisecond time type. */
 #define CONFIG_DEFAULT_REPL_DISKLESS_SYNC_DELAY 5
 #define CONFIG_DEFAULT_SLAVE_SERVE_STALE_DATA 1
 #define CONFIG_DEFAULT_SLAVE_READ_ONLY 1
+#define CONFIG_DEFAULT_SLAVE_ANNOUNCE_IP NULL
+#define CONFIG_DEFAULT_SLAVE_ANNOUNCE_PORT 0
 #define CONFIG_DEFAULT_REPL_DISABLE_TCP_NODELAY 0
 #define CONFIG_DEFAULT_MAXMEMORY 0
 #define CONFIG_DEFAULT_MAXMEMORY_SAMPLES 5
+#define CONFIG_DEFAULT_LFU_LOG_FACTOR 10
+#define CONFIG_DEFAULT_LFU_DECAY_TIME 1
 #define CONFIG_DEFAULT_AOF_FILENAME "appendonly.aof"
 #define CONFIG_DEFAULT_AOF_NO_FSYNC_ON_REWRITE 0
 #define CONFIG_DEFAULT_AOF_LOAD_TRUNCATED 1
+#define CONFIG_DEFAULT_AOF_USE_RDB_PREAMBLE 0
 #define CONFIG_DEFAULT_ACTIVE_REHASHING 1
 #define CONFIG_DEFAULT_AOF_REWRITE_INCREMENTAL_FSYNC 1
 #define CONFIG_DEFAULT_MIN_SLAVES_TO_WRITE 0
@@ -160,7 +171,7 @@ typedef long long mstime_t; /* millisecond time type. */
 #define PROTO_REPLY_CHUNK_BYTES (16*1024) /* 16k output buffer */
 #define PROTO_INLINE_MAX_SIZE   (1024*64) /* Max size of inline reads */
 #define PROTO_MBULK_BIG_ARG     (1024*32)
-#define LONG_STR_SIZE      21          /* Bytes needed for long -> str */
+#define LONG_STR_SIZE      21          /* Bytes needed for long -> str + '\0' */
 #define AOF_AUTOSYNC_BYTES (1024*1024*32) /* fdatasync every 32MB */
 
 /* When configuring the server eventloop, we setup it so that the total number
@@ -174,47 +185,22 @@ typedef long long mstime_t; /* millisecond time type. */
 
 /* Command flags. Please check the command table defined in the redis.c file
  * for more information about the meaning of every flag. */
-#define CMD_WRITE 1                   /* "w" flag */
-#define CMD_READONLY 2                /* "r" flag */
-#define CMD_DENYOOM 4                 /* "m" flag */
-#define CMD_NOT_USED_1 8              /* no longer used flag */
-#define CMD_ADMIN 16                  /* "a" flag */
-#define CMD_PUBSUB 32                 /* "p" flag */
-#define CMD_NOSCRIPT  64              /* "s" flag */
-#define CMD_RANDOM 128                /* "R" flag */
-#define CMD_SORT_FOR_SCRIPT 256       /* "S" flag */
-#define CMD_LOADING 512               /* "l" flag */
-#define CMD_STALE 1024                /* "t" flag */
-#define CMD_SKIP_MONITOR 2048         /* "M" flag */
-#define CMD_ASKING 4096               /* "k" flag */
-#define CMD_FAST 8192                 /* "F" flag */
-
-/* Defines related to the dump file format. To store 32 bits lengths for short
- * keys requires a lot of space, so we check the most significant 2 bits of
- * the first byte to interpreter the length:
- *
- * 00|000000 => if the two MSB are 00 the len is the 6 bits of this byte
- * 01|000000 00000000 =>  01, the len is 14 byes, 6 bits + 8 bits of next byte
- * 10|000000 [32 bit integer] => if it's 10, a full 32 bit len will follow
- * 11|000000 this means: specially encoded object will follow. The six bits
- *           number specify the kind of object that follows.
- *           See the RDB_ENC_* defines.
- *
- * Lengths up to 63 are stored using a single byte, most DB keys, and may
- * values, will fit inside. */
-#define RDB_6BITLEN 0
-#define RDB_14BITLEN 1
-#define RDB_32BITLEN 2
-#define RDB_ENCVAL 3
-#define RDB_LENERR UINT_MAX
-
-/* When a length of a string object stored on disk has the first two bits
- * set, the remaining two bits specify a special encoding for the object
- * accordingly to the following defines: */
-#define RDB_ENC_INT8 0        /* 8 bit signed integer */
-#define RDB_ENC_INT16 1       /* 16 bit signed integer */
-#define RDB_ENC_INT32 2       /* 32 bit signed integer */
-#define RDB_ENC_LZF 3         /* string compressed with FASTLZ */
+#define CMD_WRITE (1<<0)            /* "w" flag */
+#define CMD_READONLY (1<<1)         /* "r" flag */
+#define CMD_DENYOOM (1<<2)          /* "m" flag */
+#define CMD_MODULE (1<<3)           /* Command exported by module. */
+#define CMD_ADMIN (1<<4)            /* "a" flag */
+#define CMD_PUBSUB (1<<5)           /* "p" flag */
+#define CMD_NOSCRIPT (1<<6)         /* "s" flag */
+#define CMD_RANDOM (1<<7)           /* "R" flag */
+#define CMD_SORT_FOR_SCRIPT (1<<8)  /* "S" flag */
+#define CMD_LOADING (1<<9)          /* "l" flag */
+#define CMD_STALE (1<<10)           /* "t" flag */
+#define CMD_SKIP_MONITOR (1<<11)    /* "M" flag */
+#define CMD_ASKING (1<<12)          /* "k" flag */
+#define CMD_FAST (1<<13)            /* "F" flag */
+#define CMD_MODULE_GETKEYS (1<<14)  /* Use the modules getkeys interface. */
+#define CMD_MODULE_NO_CLUSTER (1<<15) /* Deny on Redis Cluster. */
 
 /* AOF states */
 #define AOF_OFF 0             /* AOF is off */
@@ -252,6 +238,7 @@ typedef long long mstime_t; /* millisecond time type. */
 #define CLIENT_REPLY_SKIP (1<<24)  /* Don't send just this reply. */
 #define CLIENT_LUA_DEBUG (1<<25)  /* Run EVAL in debug mode. */
 #define CLIENT_LUA_DEBUG_SYNC (1<<26)  /* EVAL debugging without fork() */
+#define CLIENT_MODULE (1<<27) /* Non connected client used by some module. */
 
 /* Client block type (btype field in client structure)
  * if CLIENT_BLOCKED flag is set. */
@@ -284,13 +271,15 @@ typedef long long mstime_t; /* millisecond time type. */
 #define REPL_STATE_RECEIVE_AUTH 5 /* Wait for AUTH reply */
 #define REPL_STATE_SEND_PORT 6 /* Send REPLCONF listening-port */
 #define REPL_STATE_RECEIVE_PORT 7 /* Wait for REPLCONF reply */
-#define REPL_STATE_SEND_CAPA 8 /* Send REPLCONF capa */
-#define REPL_STATE_RECEIVE_CAPA 9 /* Wait for REPLCONF reply */
-#define REPL_STATE_SEND_PSYNC 10 /* Send PSYNC */
-#define REPL_STATE_RECEIVE_PSYNC 11 /* Wait for PSYNC reply */
+#define REPL_STATE_SEND_IP 8 /* Send REPLCONF ip-address */
+#define REPL_STATE_RECEIVE_IP 9 /* Wait for REPLCONF reply */
+#define REPL_STATE_SEND_CAPA 10 /* Send REPLCONF capa */
+#define REPL_STATE_RECEIVE_CAPA 11 /* Wait for REPLCONF reply */
+#define REPL_STATE_SEND_PSYNC 12 /* Send PSYNC */
+#define REPL_STATE_RECEIVE_PSYNC 13 /* Wait for PSYNC reply */
 /* --- End of handshake states --- */
-#define REPL_STATE_TRANSFER 12 /* Receiving .rdb from master */
-#define REPL_STATE_CONNECTED 13 /* Connected to master */
+#define REPL_STATE_TRANSFER 14 /* Receiving .rdb from master */
+#define REPL_STATE_CONNECTED 15 /* Connected to master */
 
 /* State of slaves from the POV of the master. Used in client->replstate.
  * In SEND_BULK and ONLINE state the slave receives new updates
@@ -360,13 +349,24 @@ typedef long long mstime_t; /* millisecond time type. */
 #define SET_OP_DIFF 1
 #define SET_OP_INTER 2
 
-/* Redis maxmemory strategies */
-#define MAXMEMORY_VOLATILE_LRU 0
-#define MAXMEMORY_VOLATILE_TTL 1
-#define MAXMEMORY_VOLATILE_RANDOM 2
-#define MAXMEMORY_ALLKEYS_LRU 3
-#define MAXMEMORY_ALLKEYS_RANDOM 4
-#define MAXMEMORY_NO_EVICTION 5
+/* Redis maxmemory strategies. Instead of using just incremental number
+ * for this defines, we use a set of flags so that testing for certain
+ * properties common to multiple policies is faster. */
+#define MAXMEMORY_FLAG_LRU (1<<0)
+#define MAXMEMORY_FLAG_LFU (1<<1)
+#define MAXMEMORY_FLAG_ALLKEYS (1<<2)
+#define MAXMEMORY_FLAG_NO_SHARED_INTEGERS \
+    (MAXMEMORY_FLAG_LRU|MAXMEMORY_FLAG_LFU)
+
+#define MAXMEMORY_VOLATILE_LRU ((0<<8)|MAXMEMORY_FLAG_LRU)
+#define MAXMEMORY_VOLATILE_LFU ((1<<8)|MAXMEMORY_FLAG_LFU)
+#define MAXMEMORY_VOLATILE_TTL (2<<8)
+#define MAXMEMORY_VOLATILE_RANDOM (3<<8)
+#define MAXMEMORY_ALLKEYS_LRU ((4<<8)|MAXMEMORY_FLAG_LRU|MAXMEMORY_FLAG_ALLKEYS)
+#define MAXMEMORY_ALLKEYS_LFU ((5<<8)|MAXMEMORY_FLAG_LFU|MAXMEMORY_FLAG_ALLKEYS)
+#define MAXMEMORY_ALLKEYS_RANDOM ((6<<8)|MAXMEMORY_FLAG_ALLKEYS)
+#define MAXMEMORY_NO_EVICTION (7<<8)
+
 #define CONFIG_DEFAULT_MAXMEMORY_POLICY MAXMEMORY_NO_EVICTION
 
 /* Scripting */
@@ -441,6 +441,93 @@ typedef long long mstime_t; /* millisecond time type. */
 #define OBJ_ZSET 3
 #define OBJ_HASH 4
 
+/* The "module" object type is a special one that signals that the object
+ * is one directly managed by a Redis module. In this case the value points
+ * to a moduleValue struct, which contains the object value (which is only
+ * handled by the module itself) and the RedisModuleType struct which lists
+ * function pointers in order to serialize, deserialize, AOF-rewrite and
+ * free the object.
+ *
+ * Inside the RDB file, module types are encoded as OBJ_MODULE followed
+ * by a 64 bit module type ID, which has a 54 bits module-specific signature
+ * in order to dispatch the loading to the right module, plus a 10 bits
+ * encoding version. */
+#define OBJ_MODULE 5
+
+/* Extract encver / signature from a module type ID. */
+#define REDISMODULE_TYPE_ENCVER_BITS 10
+#define REDISMODULE_TYPE_ENCVER_MASK ((1<<REDISMODULE_TYPE_ENCVER_BITS)-1)
+#define REDISMODULE_TYPE_ENCVER(id) (id & REDISMODULE_TYPE_ENCVER_MASK)
+#define REDISMODULE_TYPE_SIGN(id) ((id & ~((uint64_t)REDISMODULE_TYPE_ENCVER_MASK)) >>REDISMODULE_TYPE_ENCVER_BITS)
+
+struct RedisModule;
+struct RedisModuleIO;
+struct RedisModuleDigest;
+struct RedisModuleCtx;
+struct redisObject;
+
+/* Each module type implementation should export a set of methods in order
+ * to serialize and deserialize the value in the RDB file, rewrite the AOF
+ * log, create the digest for "DEBUG DIGEST", and free the value when a key
+ * is deleted. */
+typedef void *(*moduleTypeLoadFunc)(struct RedisModuleIO *io, int encver);
+typedef void (*moduleTypeSaveFunc)(struct RedisModuleIO *io, void *value);
+typedef void (*moduleTypeRewriteFunc)(struct RedisModuleIO *io, struct redisObject *key, void *value);
+typedef void (*moduleTypeDigestFunc)(struct RedisModuleDigest *digest, void *value);
+typedef void (*moduleTypeFreeFunc)(void *value);
+
+/* The module type, which is referenced in each value of a given type, defines
+ * the methods and links to the module exporting the type. */
+typedef struct RedisModuleType {
+    uint64_t id; /* Higher 54 bits of type ID + 10 lower bits of encoding ver. */
+    struct RedisModule *module;
+    moduleTypeLoadFunc rdb_load;
+    moduleTypeSaveFunc rdb_save;
+    moduleTypeRewriteFunc aof_rewrite;
+    moduleTypeDigestFunc digest;
+    moduleTypeFreeFunc free;
+    char name[10]; /* 9 bytes name + null term. Charset: A-Z a-z 0-9 _- */
+} moduleType;
+
+/* In Redis objects 'robj' structures of type OBJ_MODULE, the value pointer
+ * is set to the following structure, referencing the moduleType structure
+ * in order to work with the value, and at the same time providing a raw
+ * pointer to the value, as created by the module commands operating with
+ * the module type.
+ *
+ * So for example in order to free such a value, it is possible to use
+ * the following code:
+ *
+ *  if (robj->type == OBJ_MODULE) {
+ *      moduleValue *mt = robj->ptr;
+ *      mt->type->free(mt->value);
+ *      zfree(mt); // We need to release this in-the-middle struct as well.
+ *  }
+ */
+typedef struct moduleValue {
+    moduleType *type;
+    void *value;
+} moduleValue;
+
+/* This is a wrapper for the 'rio' streams used inside rdb.c in Redis, so that
+ * the user does not have to take the total count of the written bytes nor
+ * to care about error conditions. */
+typedef struct RedisModuleIO {
+    size_t bytes;       /* Bytes read / written so far. */
+    rio *rio;           /* Rio stream. */
+    moduleType *type;   /* Module type doing the operation. */
+    int error;          /* True if error condition happened. */
+    struct RedisModuleCtx *ctx; /* Optional context, see RM_GetContextFromIO()*/
+} RedisModuleIO;
+
+#define moduleInitIOContext(iovar,mtype,rioptr) do { \
+    iovar.rio = rioptr; \
+    iovar.type = mtype; \
+    iovar.bytes = 0; \
+    iovar.error = 0; \
+    iovar.ctx = NULL; \
+} while(0);
+
 /* Objects encoding. Some kind of objects like Strings and Hashes can be
  * internally represented in multiple ways. The 'encoding' field of the object
  * is set to one of this fields for this object. */
@@ -448,7 +535,7 @@ typedef long long mstime_t; /* millisecond time type. */
 #define OBJ_ENCODING_INT 1     /* Encoded as integer */
 #define OBJ_ENCODING_HT 2      /* Encoded as hash table */
 #define OBJ_ENCODING_ZIPMAP 3  /* Encoded as zipmap */
-#define OBJ_ENCODING_LINKEDLIST 4 /* Encoded as regular linked list */
+#define OBJ_ENCODING_LINKEDLIST 4 /* No longer used: old list encoding. */
 #define OBJ_ENCODING_ZIPLIST 5 /* Encoded as ziplist */
 #define OBJ_ENCODING_INTSET 6  /* Encoded as intset */
 #define OBJ_ENCODING_SKIPLIST 7  /* Encoded as skiplist */
@@ -463,7 +550,9 @@ typedef long long mstime_t; /* millisecond time type. */
 typedef struct redisObject {
     unsigned type:4;
     unsigned encoding:4;
-    unsigned lru:LRU_BITS; /* lru time (relative to server.lruclock) */
+    unsigned lru:LRU_BITS; /* LRU time (relative to server.lruclock) or
+                            * LFU data (least significant 8 bits frequency
+                            * and most significant 16 bits decreas time). */
     int refcount;
     void *ptr;
 } robj;
@@ -471,7 +560,7 @@ typedef struct redisObject {
 /* Macro used to obtain the current LRU clock.
  * If the current resolution is lower than the frequency we refresh the
  * LRU clock (as it should be in production servers) we return the
- * precomputed value, otherwise we need to resort to a function call. */
+ * precomputed value, otherwise we need to resort to a system call. */
 #define LRU_CLOCK() ((1000/server.hz <= LRU_CLOCK_RESOLUTION) ? server.lruclock : getLRUClock())
 
 /* Macro used to initialize a Redis object allocated on the stack.
@@ -483,20 +572,9 @@ typedef struct redisObject {
     _var.type = OBJ_STRING; \
     _var.encoding = OBJ_ENCODING_RAW; \
     _var.ptr = _ptr; \
-} while(0);
+} while(0)
 
-/* To improve the quality of the LRU approximation we take a set of keys
- * that are good candidate for eviction across freeMemoryIfNeeded() calls.
- *
- * Entries inside the eviciton pool are taken ordered by idle time, putting
- * greater idle times to the right (ascending order).
- *
- * Empty entries have the key pointer set to NULL. */
-#define MAXMEMORY_EVICTION_POOL_SIZE 16
-struct evictionPoolEntry {
-    unsigned long long idle;    /* Object idle time. */
-    sds key;                    /* Key name. */
-};
+struct evictionPoolEntry; /* Defined in evict.c */
 
 /* Redis database representation. There are multiple databases identified
  * by integers from 0 (the default database) up to the max configured
@@ -504,10 +582,9 @@ struct evictionPoolEntry {
 typedef struct redisDb {
     dict *dict;                 /* The keyspace for this DB */
     dict *expires;              /* Timeout of keys with a timeout set */
-    dict *blocking_keys;        /* Keys with clients waiting for data (BLPOP) */
+    dict *blocking_keys;        /* Keys with clients waiting for data (BLPOP)*/
     dict *ready_keys;           /* Blocked keys that received a PUSH */
     dict *watched_keys;         /* WATCHED keys for MULTI/EXEC CAS */
-    struct evictionPoolEntry *eviction_pool;    /* Eviction pool of keys */
     int id;                     /* Database ID */
     long long avg_ttl;          /* Average TTL, just for stats */
 } redisDb;
@@ -566,7 +643,6 @@ typedef struct client {
     uint64_t id;            /* Client incremental unique ID. */
     int fd;                 /* Client socket. */
     redisDb *db;            /* Pointer to currently SELECTed DB. */
-    int dictid;             /* ID of the currently SELECTed DB. */
     robj *name;             /* As set by CLIENT SETNAME. */
     sds querybuf;           /* Buffer we use to accumulate client queries. */
     size_t querybuf_peak;   /* Recent (100ms or more) peak of querybuf size. */
@@ -598,7 +674,8 @@ typedef struct client {
                                        copying this slave output buffer
                                        should use. */
     char replrunid[CONFIG_RUN_ID_SIZE+1]; /* Master run id if is a master. */
-    int slave_listening_port; /* As configured with: SLAVECONF listening-port */
+    int slave_listening_port; /* As configured with: REPLCONF listening-port */
+    char slave_ip[NET_IP_STR_LEN]; /* Optionally given by REPLCONF ip-address */
     int slave_capa;         /* Slave capabilities: SLAVE_CAPA_* bitwise OR. */
     multiState mstate;      /* MULTI/EXEC state */
     int btype;              /* Type of blocking op if CLIENT_BLOCKED. */
@@ -619,6 +696,12 @@ struct saveparam {
     int changes;
 };
 
+struct moduleLoadQueueEntry {
+    sds path;
+    int argc;
+    robj **argv;
+};
+
 struct sharedObjectsStruct {
     robj *crlf, *ok, *err, *emptybulk, *czero, *cone, *cnegone, *pong, *space,
     *colon, *nullbulk, *nullmultibulk, *queued,
@@ -689,6 +772,31 @@ typedef struct redisOpArray {
     int numops;
 } redisOpArray;
 
+/* This structure is returned by the getMemoryOverheadData() function in
+ * order to return memory overhead information. */
+struct redisMemOverhead {
+    size_t peak_allocated;
+    size_t total_allocated;
+    size_t startup_allocated;
+    size_t repl_backlog;
+    size_t clients_slaves;
+    size_t clients_normal;
+    size_t aof_buffer;
+    size_t overhead_total;
+    size_t dataset;
+    size_t total_keys;
+    size_t bytes_per_key;
+    float dataset_perc;
+    float peak_perc;
+    float fragmentation;
+    size_t num_dbs;
+    struct {
+        size_t dbid;
+        size_t overhead_ht_main;
+        size_t overhead_ht_expires;
+    } *db;
+};
+
 /*-----------------------------------------------------------------------------
  * Global server state
  *----------------------------------------------------------------------------*/
@@ -701,6 +809,10 @@ struct clusterState;
 #undef hz
 #endif
 
+#define CHILD_INFO_MAGIC 0xC17DDA7A12345678LL
+#define CHILD_INFO_TYPE_RDB 0
+#define CHILD_INFO_TYPE_AOF 1
+
 struct redisServer {
     /* General */
     pid_t pid;                  /* Main process pid. */
@@ -721,6 +833,10 @@ struct redisServer {
     int cronloops;              /* Number of times the cron function run */
     char runid[CONFIG_RUN_ID_SIZE+1];  /* ID always different at every exec. */
     int sentinel_mode;          /* True if this instance is a Sentinel. */
+    size_t initial_memory_usage; /* Bytes used after initialization. */
+    /* Modules */
+    dict *moduleapi;            /* Exported APIs dictionary for modules. */
+    list *loadmodule_queue;     /* List of modules to load at startup. */
     /* Networking */
     int port;                   /* TCP listening port */
     int tcp_backlog;            /* TCP listen() backlog */
@@ -743,6 +859,7 @@ struct redisServer {
     char neterr[ANET_ERR_LEN];   /* Error buffer for anet.c */
     dict *migrate_cached_sockets;/* MIGRATE cached sockets */
     uint64_t next_client_id;    /* Next client unique ID. Incremental. */
+    int protected_mode;         /* Don't accept external connections. */
     /* RDB / AOF loading information */
     int loading;                /* We are loading data from disk if true */
     off_t loading_total_bytes;
@@ -774,6 +891,8 @@ struct redisServer {
     size_t resident_set_size;       /* RSS sampled in serverCron(). */
     long long stat_net_input_bytes; /* Bytes read from network. */
     long long stat_net_output_bytes; /* Bytes written to network. */
+    size_t stat_rdb_cow_bytes;      /* Copy on write bytes during RDB saving. */
+    size_t stat_aof_cow_bytes;      /* Copy on write bytes during AOF rewrite. */
     /* The following two are used to track instantaneous metrics, like
      * number of operations per second, network traffic. */
     struct {
@@ -818,6 +937,7 @@ struct redisServer {
     int aof_last_write_status;      /* C_OK or C_ERR */
     int aof_last_write_errno;       /* Valid if aof_last_write_status is ERR */
     int aof_load_truncated;         /* Don't stop on unexpected AOF EOF. */
+    int aof_use_rdb_preamble;       /* Use RDB preamble on AOF rewrites. */
     /* AOF pipes used to communicate between parent and child during rewrite. */
     int aof_pipe_write_data_to_child;
     int aof_pipe_read_data_from_parent;
@@ -841,11 +961,19 @@ struct redisServer {
     time_t lastbgsave_try;          /* Unix time of last attempted bgsave */
     time_t rdb_save_time_last;      /* Time used by last RDB save run. */
     time_t rdb_save_time_start;     /* Current RDB save start time. */
+    int rdb_bgsave_scheduled;       /* BGSAVE when possible if true. */
     int rdb_child_type;             /* Type of save by active child. */
     int lastbgsave_status;          /* C_OK or C_ERR */
     int stop_writes_on_bgsave_err;  /* Don't allow writes if can't BGSAVE */
     int rdb_pipe_write_result_to_parent; /* RDB pipes used to return the state */
     int rdb_pipe_read_result_from_child; /* of each slave in diskless SYNC. */
+    /* Pipe and data structures for child -> parent info sharing. */
+    int child_info_pipe[2];         /* Pipe used to write the child_info_data. */
+    struct {
+        int process_type;           /* AOF or RDB child? */
+        size_t cow_size;            /* Copy on write size. */
+        unsigned long long magic;   /* Magic value to make sure data is valid. */
+    } child_info_data;
     /* Propagation of commands in AOF / replication */
     redisOpArray also_propagate;    /* Additional command to propagate. */
     /* Logging */
@@ -893,7 +1021,9 @@ struct redisServer {
     time_t repl_down_since; /* Unix time at which link with master went down */
     int repl_disable_tcp_nodelay;   /* Disable TCP_NODELAY after SYNC? */
     int slave_priority;             /* Reported in INFO and used by Sentinel. */
-    char repl_master_runid[CONFIG_RUN_ID_SIZE+1];  /* Master run id for PSYNC. */
+    int slave_announce_port;        /* Give the master this listening port. */
+    char *slave_announce_ip;        /* Give the master this ip address. */
+    char repl_master_runid[CONFIG_RUN_ID_SIZE+1];  /* Master run id for PSYNC.*/
     long long repl_master_initial_offset;         /* Master PSYNC offset. */
     int repl_slave_lazy_flush;          /* Lazy FLUSHALL before loading DB? */
     /* Replication script cache. */
@@ -908,6 +1038,8 @@ struct redisServer {
     unsigned long long maxmemory;   /* Max number of memory bytes to use */
     int maxmemory_policy;           /* Policy for key eviction */
     int maxmemory_samples;          /* Pricision of random sampling */
+    unsigned int lfu_log_factor;    /* LFU logarithmic counter factor. */
+    unsigned int lfu_decay_time;    /* LFU counter decay factor. */
     /* Blocked clients */
     unsigned int bpop_blocked_clients; /* Number of clients blocked by lists */
     list *unblocked_clients; /* list of clients to unblock before next loop */
@@ -945,6 +1077,9 @@ struct redisServer {
     int cluster_slave_validity_factor; /* Slave max data age for failover. */
     int cluster_require_full_coverage; /* If true, put the cluster down if
                                           there is at least an uncovered slot.*/
+    char *cluster_announce_ip;  /* IP address to announce on cluster bus. */
+    int cluster_announce_port;     /* base port to announce on cluster bus. */
+    int cluster_announce_bus_port; /* bus port to announce on cluster bus. */
     /* Scripting */
     lua_State *lua; /* The Lua interpreter. We use just one for all clients */
     client *lua_client;   /* The "fake client" to query Redis from Lua */
@@ -971,8 +1106,8 @@ struct redisServer {
     long long latency_monitor_threshold;
     dict *latency_events;
     /* Assert & bug reporting */
-    char *assert_failed;
-    char *assert_file;
+    const char *assert_failed;
+    const char *assert_file;
     int assert_line;
     int bug_report_start; /* True if bug report header was already logged. */
     int watchdog_period;  /* Software watchdog period in ms. 0 = off */
@@ -1077,11 +1212,21 @@ extern double R_Zero, R_PosInf, R_NegInf, R_Nan;
 extern dictType hashDictType;
 extern dictType replScriptCacheDictType;
 extern dictType keyptrDictType;
+extern dictType modulesDictType;
 
 /*-----------------------------------------------------------------------------
  * Functions prototypes
  *----------------------------------------------------------------------------*/
 
+/* Modules */
+void moduleInitModulesSystem(void);
+int moduleLoad(const char *path, void **argv, int argc);
+void moduleLoadFromQueue(void);
+int *moduleGetCommandKeysViaAPI(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
+moduleType *moduleTypeLookupModuleByID(uint64_t id);
+void moduleTypeNameByID(char *name, uint64_t moduleid);
+void moduleFreeContext(struct RedisModuleCtx *ctx);
+
 /* Utils */
 long long ustime(void);
 long long mstime(void);
@@ -1115,13 +1260,15 @@ void addReplyBulkSds(client *c, sds s);
 void addReplyError(client *c, const char *err);
 void addReplyStatus(client *c, const char *status);
 void addReplyDouble(client *c, double d);
+void addReplyHumanLongDouble(client *c, long double d);
 void addReplyLongLong(client *c, long long ll);
 void addReplyMultiBulkLen(client *c, long length);
 void copyClientOutputBuffer(client *dst, client *src);
+size_t sdsZmallocSize(sds s);
+size_t getStringObjectSdsUsedMemory(robj *o);
 void *dupClientReplyValue(void *o);
 void getClientsMaxBuffers(unsigned long *longest_output_list,
                           unsigned long *biggest_input_buffer);
-void formatPeerId(char *peerid, size_t peerid_len, char *ip, int port);
 char *getClientPeerId(client *client);
 sds catClientInfoString(sds s, client *client);
 sds getAllClientsInfoString(void);
@@ -1159,7 +1306,7 @@ void addReplyStatusFormat(client *c, const char *fmt, ...);
 void listTypeTryConversion(robj *subject, robj *value);
 void listTypePush(robj *subject, robj *value, int where);
 robj *listTypePop(robj *subject, int where);
-unsigned long listTypeLength(robj *subject);
+unsigned long listTypeLength(const robj *subject);
 listTypeIterator *listTypeInitIterator(robj *subject, long index, unsigned char direction);
 void listTypeReleaseIterator(listTypeIterator *li);
 int listTypeNext(listTypeIterator *li, listTypeEntry *entry);
@@ -1199,7 +1346,7 @@ robj *createObject(int type, void *ptr);
 robj *createStringObject(const char *ptr, size_t len);
 robj *createRawStringObject(const char *ptr, size_t len);
 robj *createEmbeddedStringObject(const char *ptr, size_t len);
-robj *dupStringObject(robj *o);
+robj *dupStringObject(const robj *o);
 int isSdsRepresentableAsLongLong(sds s, long long *llval);
 int isObjectRepresentableAsLongLong(robj *o, long long *llongval);
 robj *tryObjectEncoding(robj *o);
@@ -1214,10 +1361,12 @@ robj *createIntsetObject(void);
 robj *createHashObject(void);
 robj *createZsetObject(void);
 robj *createZsetZiplistObject(void);
+robj *createModuleObject(moduleType *mt, void *value);
 int getLongFromObjectOrReply(client *c, robj *o, long *target, const char *msg);
 int checkType(client *c, robj *o, int type);
 int getLongLongFromObjectOrReply(client *c, robj *o, long long *target, const char *msg);
 int getDoubleFromObjectOrReply(client *c, robj *o, double *target, const char *msg);
+int getDoubleFromObject(const robj *o, double *target);
 int getLongLongFromObject(robj *o, long long *target);
 int getLongDoubleFromObject(robj *o, long double *target);
 int getLongDoubleFromObjectOrReply(client *c, robj *o, long double *target, const char *msg);
@@ -1264,6 +1413,7 @@ void stopLoading(void);
 
 /* RDB persistence */
 #include "rdb.h"
+int rdbSaveRio(rio *rdb, int *error, int flags);
 
 /* AOF persistence */
 void flushAppendOnlyFile(int force);
@@ -1276,9 +1426,31 @@ int startAppendOnly(void);
 void backgroundRewriteDoneHandler(int exitcode, int bysignal);
 void aofRewriteBufferReset(void);
 unsigned long aofRewriteBufferSize(void);
+ssize_t aofReadDiffFromParent(void);
+
+/* Child info */
+void openChildInfoPipe(void);
+void closeChildInfoPipe(void);
+void sendChildInfo(int process_type);
+void receiveChildInfo(void);
 
 /* Sorted sets data type */
 
+/* Input flags. */
+#define ZADD_NONE 0
+#define ZADD_INCR (1<<0)    /* Increment the score instead of setting it. */
+#define ZADD_NX (1<<1)      /* Don't touch elements not already existing. */
+#define ZADD_XX (1<<2)      /* Only touch elements already exisitng. */
+
+/* Output flags. */
+#define ZADD_NOP (1<<3)     /* Operation not performed because of conditionals.*/
+#define ZADD_NAN (1<<4)     /* Only touch elements already exisitng. */
+#define ZADD_ADDED (1<<5)   /* The element was new and was added. */
+#define ZADD_UPDATED (1<<6) /* The element already existed, score updated. */
+
+/* Flags only used by the ZADD command but not by zsetAdd() API: */
+#define ZADD_CH (1<<16)      /* Return num of elements added or updated. */
+
 /* Struct to hold a inclusive/exclusive range spec by score comparison. */
 typedef struct {
     double min, max;
@@ -1301,10 +1473,29 @@ zskiplistNode *zslLastInRange(zskiplist *zsl, zrangespec *range);
 double zzlGetScore(unsigned char *sptr);
 void zzlNext(unsigned char *zl, unsigned char **eptr, unsigned char **sptr);
 void zzlPrev(unsigned char *zl, unsigned char **eptr, unsigned char **sptr);
-unsigned int zsetLength(robj *zobj);
+unsigned char *zzlFirstInRange(unsigned char *zl, zrangespec *range);
+unsigned char *zzlLastInRange(unsigned char *zl, zrangespec *range);
+unsigned int zsetLength(const robj *zobj);
 void zsetConvert(robj *zobj, int encoding);
+void zsetConvertToZiplistIfNeeded(robj *zobj, size_t maxelelen);
 int zsetScore(robj *zobj, sds member, double *score);
 unsigned long zslGetRank(zskiplist *zsl, double score, sds o);
+int zsetAdd(robj *zobj, double score, sds ele, int *flags, double *newscore);
+long zsetRank(robj *zobj, sds ele, int reverse);
+int zsetDel(robj *zobj, sds ele);
+sds ziplistGetObject(unsigned char *sptr);
+int zslValueGteMin(double value, zrangespec *spec);
+int zslValueLteMax(double value, zrangespec *spec);
+void zslFreeLexRange(zlexrangespec *spec);
+int zslParseLexRange(robj *min, robj *max, zlexrangespec *spec);
+unsigned char *zzlFirstInLexRange(unsigned char *zl, zlexrangespec *range);
+unsigned char *zzlLastInLexRange(unsigned char *zl, zlexrangespec *range);
+zskiplistNode *zslFirstInLexRange(zskiplist *zsl, zlexrangespec *range);
+zskiplistNode *zslLastInLexRange(zskiplist *zsl, zlexrangespec *range);
+int zzlLexValueGteMin(unsigned char *p, zlexrangespec *spec);
+int zzlLexValueLteMax(unsigned char *p, zlexrangespec *spec);
+int zslLexValueGteMin(sds value, zlexrangespec *spec);
+int zslLexValueLteMax(sds value, zlexrangespec *spec);
 
 /* Core functions */
 int freeMemoryIfNeeded(void);
@@ -1332,7 +1523,6 @@ void serverLogFromHandler(int level, const char *msg);
 void usage(void);
 void updateDictResizePolicy(void);
 int htNeedsResize(dict *dict);
-void oom(const char *msg);
 void populateCommandTable(void);
 void resetCommandTableStats(void);
 void adjustOpenFilesLimit(void);
@@ -1340,7 +1530,9 @@ void closeListeningSockets(int unlink_unix_socket);
 void updateCachedTime(void);
 void resetServerStats(void);
 unsigned int getLRUClock(void);
-const char *maxmemoryToString(void);
+const char *evictPolicyToString(void);
+struct redisMemOverhead *getMemoryOverheadData(void);
+void freeMemoryOverheadData(struct redisMemOverhead *mh);
 
 #define RESTART_SERVER_NONE 0
 #define RESTART_SERVER_GRACEFULLY (1<<0)     /* Do proper shutdown. */
@@ -1358,16 +1550,20 @@ int setTypeNext(setTypeIterator *si, sds *sdsele, int64_t *llele);
 sds setTypeNextObject(setTypeIterator *si);
 int setTypeRandomElement(robj *setobj, sds *sdsele, int64_t *llele);
 unsigned long setTypeRandomElements(robj *set, unsigned long count, robj *aux_set);
-unsigned long setTypeSize(robj *subject);
+unsigned long setTypeSize(const robj *subject);
 void setTypeConvert(robj *subject, int enc);
 
 /* Hash data type */
+#define HASH_SET_TAKE_FIELD (1<<0)
+#define HASH_SET_TAKE_VALUE (1<<1)
+#define HASH_SET_COPY 0
+
 void hashTypeConvert(robj *o, int enc);
 void hashTypeTryConversion(robj *subject, robj **argv, int start, int end);
 void hashTypeTryObjectEncoding(robj *subject, robj **o1, robj **o2);
 int hashTypeExists(robj *o, sds key);
 int hashTypeDelete(robj *o, sds key);
-unsigned long hashTypeLength(robj *o);
+unsigned long hashTypeLength(const robj *o);
 hashTypeIterator *hashTypeInitIterator(robj *subject);
 void hashTypeReleaseIterator(hashTypeIterator *hi);
 int hashTypeNext(hashTypeIterator *hi);
@@ -1380,6 +1576,7 @@ void hashTypeCurrentObject(hashTypeIterator *hi, int what, unsigned char **vstr,
 sds hashTypeCurrentObjectNewSds(hashTypeIterator *hi, int what);
 robj *hashTypeLookupWriteOrCreate(client *c, robj *key);
 robj *hashTypeGetValueObject(robj *o, sds field);
+int hashTypeSet(robj *o, sds field, sds value, int flags);
 
 /* Pub / Sub */
 int pubsubUnsubscribeAllChannels(client *c, int notify);
@@ -1407,11 +1604,14 @@ void propagateExpire(redisDb *db, robj *key, int lazy);
 int expireIfNeeded(redisDb *db, robj *key);
 long long getExpire(redisDb *db, robj *key);
 void setExpire(redisDb *db, robj *key, long long when);
-robj *lookupKey(redisDb *db, robj *key);
+robj *lookupKey(redisDb *db, robj *key, int flags);
 robj *lookupKeyRead(redisDb *db, robj *key);
 robj *lookupKeyWrite(redisDb *db, robj *key);
 robj *lookupKeyReadOrReply(client *c, robj *key, robj *reply);
 robj *lookupKeyWriteOrReply(client *c, robj *key, robj *reply);
+robj *lookupKeyReadWithFlags(redisDb *db, robj *key, int flags);
+#define LOOKUP_NONE 0
+#define LOOKUP_NOTOUCH (1<<0)
 void dbAdd(redisDb *db, robj *key, robj *val);
 void dbOverwrite(redisDb *db, robj *key, robj *val);
 void setKey(redisDb *db, robj *key, robj *val);
@@ -1448,6 +1648,7 @@ void getKeysFreeResult(int *result);
 int *zunionInterGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys);
 int *evalGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
 int *sortGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
+int *migrateGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
 
 /* Cluster */
 void clusterInit(void);
@@ -1467,7 +1668,7 @@ void sentinelIsRunning(void);
 
 /* redis-check-rdb */
 int redis_check_rdb(char *rdbfilename);
-int redis_check_rdb_main(char **argv, int argc);
+int redis_check_rdb_main(int argc, char **argv);
 
 /* Scripting */
 void scriptingInit(int setup);
@@ -1483,6 +1684,15 @@ void replyToBlockedClientTimedOut(client *c);
 int getTimeoutFromObjectOrReply(client *c, robj *object, mstime_t *timeout, int unit);
 void disconnectAllBlockedClients(void);
 
+/* expire.c -- Handling of expired keys */
+void activeExpireCycle(int type);
+
+/* evict.c -- maxmemory handling and LRU eviction. */
+void evictionPoolAlloc(void);
+#define LFU_INIT_VAL 5
+unsigned long LFUGetTimeInMinutes(void);
+uint8_t LFULogIncr(uint8_t value);
+
 /* Git SHA1 */
 char *redisGitSHA1(void);
 char *redisGitDirty(void);
@@ -1503,6 +1713,7 @@ void unlinkCommand(client *c);
 void existsCommand(client *c);
 void setbitCommand(client *c);
 void getbitCommand(client *c);
+void bitfieldCommand(client *c);
 void setrangeCommand(client *c);
 void getrangeCommand(client *c);
 void incrCommand(client *c);
@@ -1565,6 +1776,7 @@ void pexpireCommand(client *c);
 void pexpireatCommand(client *c);
 void getsetCommand(client *c);
 void ttlCommand(client *c);
+void touchCommand(client *c);
 void pttlCommand(client *c);
 void persistCommand(client *c);
 void slaveofCommand(client *c);
@@ -1633,6 +1845,7 @@ void readonlyCommand(client *c);
 void readwriteCommand(client *c);
 void dumpCommand(client *c);
 void objectCommand(client *c);
+void memoryCommand(client *c);
 void clientCommand(client *c);
 void evalCommand(client *c);
 void evalShaCommand(client *c);
@@ -1657,6 +1870,8 @@ void pfcountCommand(client *c);
 void pfmergeCommand(client *c);
 void pfdebugCommand(client *c);
 void latencyCommand(client *c);
+void moduleCommand(client *c);
+void securityWarningCommand(client *c);
 
 #if defined(__GNUC__)
 void *calloc(size_t count, size_t size) __attribute__ ((deprecated));
@@ -1666,17 +1881,18 @@ void *realloc(void *ptr, size_t size) __attribute__ ((deprecated));
 #endif
 
 /* Debugging stuff */
-void _serverAssertWithInfo(client *c, robj *o, char *estr, char *file, int line);
-void _serverAssert(char *estr, char *file, int line);
-void _serverPanic(char *msg, char *file, int line);
+void _serverAssertWithInfo(const client *c, const robj *o, const char *estr, const char *file, int line);
+void _serverAssert(const char *estr, const char *file, int line);
+void _serverPanic(const char *msg, const char *file, int line);
 void bugReportStart(void);
-void serverLogObjectDebugInfo(robj *o);
+void serverLogObjectDebugInfo(const robj *o);
 void sigsegvHandler(int sig, siginfo_t *info, void *secret);
 sds genRedisInfoString(char *section);
 void enableWatchdog(int period);
 void disableWatchdog(void);
 void watchdogScheduleSignal(int period);
 void serverLogHexDump(int level, char *descr, void *value, size_t len);
+int memtest_preserving_test(unsigned long *m, size_t bytes, int passes);
 
 #define redisDebug(fmt, ...) \
     printf("DEBUG %s:%d > " fmt "\n", __FILE__, __LINE__, __VA_ARGS__)

src/t_hash.c

@@ -308,13 +308,13 @@ int hashTypeDelete(robj *o, sds field) {
 }
 
 /* Return the number of elements in a hash. */
-unsigned long hashTypeLength(robj *o) {
+unsigned long hashTypeLength(const robj *o) {
     unsigned long length = ULONG_MAX;
 
     if (o->encoding == OBJ_ENCODING_ZIPLIST) {
         length = ziplistLen(o->ptr) / 2;
     } else if (o->encoding == OBJ_ENCODING_HT) {
-        length = dictSize((dict*)o->ptr);
+        length = dictSize((const dict*)o->ptr);
     } else {
         serverPanic("Unknown hash encoding");
     }

src/t_list.c

@@ -71,7 +71,7 @@ robj *listTypePop(robj *subject, int where) {
     return value;
 }
 
-unsigned long listTypeLength(robj *subject) {
+unsigned long listTypeLength(const robj *subject) {
     if (subject->encoding == OBJ_ENCODING_QUICKLIST) {
         return quicklistCount(subject->ptr);
     } else {
@@ -195,7 +195,7 @@ void listTypeConvert(robj *subject, int enc) {
  *----------------------------------------------------------------------------*/
 
 void pushGenericCommand(client *c, int where) {
-    int j, waiting = 0, pushed = 0;
+    int j, pushed = 0;
     robj *lobj = lookupKeyWrite(c->db,c->argv[1]);
 
     if (lobj && lobj->type != OBJ_LIST) {
@@ -204,7 +204,6 @@ void pushGenericCommand(client *c, int where) {
     }
 
     for (j = 2; j < c->argc; j++) {
-        c->argv[j] = tryObjectEncoding(c->argv[j]);
         if (!lobj) {
             lobj = createQuicklistObject();
             quicklistSetOptions(lobj->ptr, server.list_max_ziplist_size,
@@ -214,7 +213,7 @@ void pushGenericCommand(client *c, int where) {
         listTypePush(lobj,c->argv[j],where);
         pushed++;
     }
-    addReplyLongLong(c, waiting + (lobj ? listTypeLength(lobj) : 0));
+    addReplyLongLong(c, (lobj ? listTypeLength(lobj) : 0));
     if (pushed) {
         char *event = (where == LIST_HEAD) ? "lpush" : "rpush";
 
@@ -232,70 +231,80 @@ void rpushCommand(client *c) {
     pushGenericCommand(c,LIST_TAIL);
 }
 
-void pushxGenericCommand(client *c, robj *refval, robj *val, int where) {
+void pushxGenericCommand(client *c, int where) {
+    int j, pushed = 0;
+    robj *subject;
+
+    if ((subject = lookupKeyWriteOrReply(c,c->argv[1],shared.czero)) == NULL ||
+        checkType(c,subject,OBJ_LIST)) return;
+
+    for (j = 2; j < c->argc; j++) {
+        listTypePush(subject,c->argv[j],where);
+        pushed++;
+    }
+
+    addReplyLongLong(c,listTypeLength(subject));
+
+    if (pushed) {
+        char *event = (where == LIST_HEAD) ? "lpush" : "rpush";
+        signalModifiedKey(c->db,c->argv[1]);
+        notifyKeyspaceEvent(NOTIFY_LIST,event,c->argv[1],c->db->id);
+    }
+    server.dirty += pushed;
+}
+
+void lpushxCommand(client *c) {
+    pushxGenericCommand(c,LIST_HEAD);
+}
+
+void rpushxCommand(client *c) {
+    pushxGenericCommand(c,LIST_TAIL);
+}
+
+void linsertCommand(client *c) {
+    int where;
     robj *subject;
     listTypeIterator *iter;
     listTypeEntry entry;
     int inserted = 0;
 
+    if (strcasecmp(c->argv[2]->ptr,"after") == 0) {
+        where = LIST_TAIL;
+    } else if (strcasecmp(c->argv[2]->ptr,"before") == 0) {
+        where = LIST_HEAD;
+    } else {
+        addReply(c,shared.syntaxerr);
+        return;
+    }
+
     if ((subject = lookupKeyWriteOrReply(c,c->argv[1],shared.czero)) == NULL ||
         checkType(c,subject,OBJ_LIST)) return;
 
-    if (refval != NULL) {
-        /* Seek refval from head to tail */
-        iter = listTypeInitIterator(subject,0,LIST_TAIL);
-        while (listTypeNext(iter,&entry)) {
-            if (listTypeEqual(&entry,refval)) {
-                listTypeInsert(&entry,val,where);
-                inserted = 1;
-                break;
-            }
+    /* Seek pivot from head to tail */
+    iter = listTypeInitIterator(subject,0,LIST_TAIL);
+    while (listTypeNext(iter,&entry)) {
+        if (listTypeEqual(&entry,c->argv[3])) {
+            listTypeInsert(&entry,c->argv[4],where);
+            inserted = 1;
+            break;
         }
-        listTypeReleaseIterator(iter);
+    }
+    listTypeReleaseIterator(iter);
 
-        if (inserted) {
-            signalModifiedKey(c->db,c->argv[1]);
-            notifyKeyspaceEvent(NOTIFY_LIST,"linsert",
-                                c->argv[1],c->db->id);
-            server.dirty++;
-        } else {
-            /* Notify client of a failed insert */
-            addReply(c,shared.cnegone);
-            return;
-        }
-    } else {
-        char *event = (where == LIST_HEAD) ? "lpush" : "rpush";
-
-        listTypePush(subject,val,where);
+    if (inserted) {
         signalModifiedKey(c->db,c->argv[1]);
-        notifyKeyspaceEvent(NOTIFY_LIST,event,c->argv[1],c->db->id);
+        notifyKeyspaceEvent(NOTIFY_LIST,"linsert",
+                            c->argv[1],c->db->id);
         server.dirty++;
+    } else {
+        /* Notify client of a failed insert */
+        addReply(c,shared.cnegone);
+        return;
     }
 
     addReplyLongLong(c,listTypeLength(subject));
 }
 
-void lpushxCommand(client *c) {
-    c->argv[2] = tryObjectEncoding(c->argv[2]);
-    pushxGenericCommand(c,NULL,c->argv[2],LIST_HEAD);
-}
-
-void rpushxCommand(client *c) {
-    c->argv[2] = tryObjectEncoding(c->argv[2]);
-    pushxGenericCommand(c,NULL,c->argv[2],LIST_TAIL);
-}
-
-void linsertCommand(client *c) {
-    c->argv[4] = tryObjectEncoding(c->argv[4]);
-    if (strcasecmp(c->argv[2]->ptr,"after") == 0) {
-        pushxGenericCommand(c,c->argv[3],c->argv[4],LIST_TAIL);
-    } else if (strcasecmp(c->argv[2]->ptr,"before") == 0) {
-        pushxGenericCommand(c,c->argv[3],c->argv[4],LIST_HEAD);
-    } else {
-        addReply(c,shared.syntaxerr);
-    }
-}
-
 void llenCommand(client *c) {
     robj *o = lookupKeyReadOrReply(c,c->argv[1],shared.czero);
     if (o == NULL || checkType(c,o,OBJ_LIST)) return;
@@ -509,12 +518,17 @@ void lremCommand(client *c) {
     }
     listTypeReleaseIterator(li);
 
+    if (removed) {
+        signalModifiedKey(c->db,c->argv[1]);
+        notifyKeyspaceEvent(NOTIFY_GENERIC,"lrem",c->argv[1],c->db->id);
+    }
+
     if (listTypeLength(subject) == 0) {
         dbDelete(c->db,c->argv[1]);
+        notifyKeyspaceEvent(NOTIFY_GENERIC,"del",c->argv[1],c->db->id);
     }
 
     addReplyLongLong(c,removed);
-    if (removed) signalModifiedKey(c->db,c->argv[1]);
 }
 
 /* This is the semantic of this command:

src/t_set.c

@@ -53,7 +53,7 @@ int setTypeAdd(robj *subject, sds value) {
     long long llval;
     if (subject->encoding == OBJ_ENCODING_HT) {
         dict *ht = subject->ptr;
-        dictEntry *de = dictAddRaw(ht,value);
+        dictEntry *de = dictAddRaw(ht,value,NULL);
         if (de) {
             dictSetKey(ht,de,sdsdup(value));
             dictSetVal(ht,de,NULL);
@@ -219,11 +219,11 @@ int setTypeRandomElement(robj *setobj, sds *sdsele, int64_t *llele) {
     return setobj->encoding;
 }
 
-unsigned long setTypeSize(robj *subject) {
+unsigned long setTypeSize(const robj *subject) {
     if (subject->encoding == OBJ_ENCODING_HT) {
-        return dictSize((dict*)subject->ptr);
+        return dictSize((const dict*)subject->ptr);
     } else if (subject->encoding == OBJ_ENCODING_INTSET) {
-        return intsetLen((intset*)subject->ptr);
+        return intsetLen((const intset*)subject->ptr);
     } else {
         serverPanic("Unknown set encoding");
     }
@@ -351,9 +351,6 @@ void smoveCommand(client *c) {
         dbDelete(c->db,c->argv[1]);
         notifyKeyspaceEvent(NOTIFY_GENERIC,"del",c->argv[1],c->db->id);
     }
-    signalModifiedKey(c->db,c->argv[1]);
-    signalModifiedKey(c->db,c->argv[2]);
-    server.dirty++;
 
     /* Create the destination set when it doesn't exist */
     if (!dstset) {
@@ -361,6 +358,10 @@ void smoveCommand(client *c) {
         dbAdd(c->db,c->argv[2],dstset);
     }
 
+    signalModifiedKey(c->db,c->argv[1]);
+    signalModifiedKey(c->db,c->argv[2]);
+    server.dirty++;
+
     /* An extra key has changed when ele was successfully added to dstset */
     if (setTypeAdd(dstset,ele->ptr)) {
         server.dirty++;
@@ -547,6 +548,8 @@ void spopWithCountCommand(client *c) {
      * the alsoPropagate() API. */
     decrRefCount(propargv[0]);
     preventCommandPropagation(c);
+    signalModifiedKey(c->db,c->argv[1]);
+    server.dirty++;
 }
 
 void spopCommand(client *c) {

src/t_string.c

@@ -263,6 +263,10 @@ void getrangeCommand(client *c) {
     }
 
     /* Convert negative indexes */
+    if (start < 0 && end < 0 && start > end) {
+        addReply(c,shared.emptybulk);
+        return;
+    }
     if (start < 0) start = strlen+start;
     if (end < 0) end = strlen+end;
     if (start < 0) start = 0;

src/t_zset.c

@@ -63,8 +63,8 @@
  * Skiplist implementation of the low level API
  *----------------------------------------------------------------------------*/
 
-static int zslLexValueGteMin(sds value, zlexrangespec *spec);
-static int zslLexValueLteMax(sds value, zlexrangespec *spec);
+int zslLexValueGteMin(sds value, zlexrangespec *spec);
+int zslLexValueLteMax(sds value, zlexrangespec *spec);
 
 /* Create a skiplist node with the specified number of levels.
  * The SDS string 'ele' is referenced by the node after the call. */
@@ -244,7 +244,7 @@ int zslDelete(zskiplist *zsl, double score, sds ele, zskiplistNode **node) {
     return 0; /* not found */
 }
 
-static int zslValueGteMin(double value, zrangespec *spec) {
+int zslValueGteMin(double value, zrangespec *spec) {
     return spec->minex ? (value > spec->min) : (value >= spec->min);
 }
 
@@ -549,12 +549,12 @@ void zslFreeLexRange(zlexrangespec *spec) {
         spec->max != shared.maxstring) sdsfree(spec->max);
 }
 
-/* Populate the rangespec according to the objects min and max.
+/* Populate the lex rangespec according to the objects min and max.
  *
  * Return C_OK on success. On error C_ERR is returned.
  * When OK is returned the structure must be freed with zslFreeLexRange(),
  * otherwise no release is needed. */
-static int zslParseLexRange(robj *min, robj *max, zlexrangespec *spec) {
+int zslParseLexRange(robj *min, robj *max, zlexrangespec *spec) {
     /* The range can't be valid if objects are integer encoded.
      * Every item must start with ( or [. */
     if (min->encoding == OBJ_ENCODING_INT ||
@@ -580,13 +580,13 @@ int sdscmplex(sds a, sds b) {
     return sdscmp(a,b);
 }
 
-static int zslLexValueGteMin(sds value, zlexrangespec *spec) {
+int zslLexValueGteMin(sds value, zlexrangespec *spec) {
     return spec->minex ?
         (sdscmplex(value,spec->min) > 0) :
         (sdscmplex(value,spec->min) >= 0);
 }
 
-static int zslLexValueLteMax(sds value, zlexrangespec *spec) {
+int zslLexValueLteMax(sds value, zlexrangespec *spec) {
     return spec->maxex ?
         (sdscmplex(value,spec->max) < 0) :
         (sdscmplex(value,spec->max) <= 0);
@@ -852,14 +852,14 @@ unsigned char *zzlLastInRange(unsigned char *zl, zrangespec *range) {
     return NULL;
 }
 
-static int zzlLexValueGteMin(unsigned char *p, zlexrangespec *spec) {
+int zzlLexValueGteMin(unsigned char *p, zlexrangespec *spec) {
     sds value = ziplistGetObject(p);
     int res = zslLexValueGteMin(value,spec);
     sdsfree(value);
     return res;
 }
 
-static int zzlLexValueLteMax(unsigned char *p, zlexrangespec *spec) {
+int zzlLexValueLteMax(unsigned char *p, zlexrangespec *spec) {
     sds value = ziplistGetObject(p);
     int res = zslLexValueLteMax(value,spec);
     sdsfree(value);
@@ -1100,12 +1100,12 @@ unsigned char *zzlDeleteRangeByRank(unsigned char *zl, unsigned int start, unsig
  * Common sorted set API
  *----------------------------------------------------------------------------*/
 
-unsigned int zsetLength(robj *zobj) {
+unsigned int zsetLength(const robj *zobj) {
     int length = -1;
     if (zobj->encoding == OBJ_ENCODING_ZIPLIST) {
         length = zzlLength(zobj->ptr);
     } else if (zobj->encoding == OBJ_ENCODING_SKIPLIST) {
-        length = ((zset*)zobj->ptr)->zsl->length;
+        length = ((const zset*)zobj->ptr)->zsl->length;
     } else {
         serverPanic("Unknown sorted set encoding");
     }
@@ -1183,6 +1183,18 @@ void zsetConvert(robj *zobj, int encoding) {
     }
 }
 
+/* Convert the sorted set object into a ziplist if it is not already a ziplist
+ * and if the number of elements and the maximum element size is within the
+ * expected ranges. */
+void zsetConvertToZiplistIfNeeded(robj *zobj, size_t maxelelen) {
+    if (zobj->encoding == OBJ_ENCODING_ZIPLIST) return;
+    zset *zset = zobj->ptr;
+
+    if (zset->zsl->length <= server.zset_max_ziplist_entries &&
+        maxelelen <= server.zset_max_ziplist_value)
+            zsetConvert(zobj,OBJ_ENCODING_ZIPLIST);
+}
+
 /* Return (by reference) the score of the specified member of the sorted set
  * storing it into *score. If the element does not exist C_ERR is returned
  * otherwise C_OK is returned and *score is correctly populated.
@@ -1203,22 +1215,280 @@ int zsetScore(robj *zobj, sds member, double *score) {
     return C_OK;
 }
 
+/* Add a new element or update the score of an existing element in a sorted
+ * set, regardless of its encoding.
+ *
+ * The set of flags change the command behavior. They are passed with an integer
+ * pointer since the function will clear the flags and populate them with
+ * other flags to indicate different conditions.
+ *
+ * The input flags are the following:
+ *
+ * ZADD_INCR: Increment the current element score by 'score' instead of updating
+ *            the current element score. If the element does not exist, we
+ *            assume 0 as previous score.
+ * ZADD_NX:   Perform the operation only if the element does not exist.
+ * ZADD_XX:   Perform the operation only if the element already exist.
+ *
+ * When ZADD_INCR is used, the new score of the element is stored in
+ * '*newscore' if 'newscore' is not NULL.
+ *
+ * The returned flags are the following:
+ *
+ * ZADD_NAN:     The resulting score is not a number.
+ * ZADD_ADDED:   The element was added (not present before the call).
+ * ZADD_UPDATED: The element score was updated.
+ * ZADD_NOP:     No operation was performed because of NX or XX.
+ *
+ * Return value:
+ *
+ * The function returns 1 on success, and sets the appropriate flags
+ * ADDED or UPDATED to signal what happened during the operation (note that
+ * none could be set if we re-added an element using the same score it used
+ * to have, or in the case a zero increment is used).
+ *
+ * The function returns 0 on erorr, currently only when the increment
+ * produces a NAN condition, or when the 'score' value is NAN since the
+ * start.
+ *
+ * The commad as a side effect of adding a new element may convert the sorted
+ * set internal encoding from ziplist to hashtable+skiplist.
+ *
+ * Memory managemnet of 'ele':
+ *
+ * The function does not take ownership of the 'ele' SDS string, but copies
+ * it if needed. */
+int zsetAdd(robj *zobj, double score, sds ele, int *flags, double *newscore) {
+    /* Turn options into simple to check vars. */
+    int incr = (*flags & ZADD_INCR) != 0;
+    int nx = (*flags & ZADD_NX) != 0;
+    int xx = (*flags & ZADD_XX) != 0;
+    *flags = 0; /* We'll return our response flags. */
+    double curscore;
+
+    /* NaN as input is an error regardless of all the other parameters. */
+    if (isnan(score)) {
+        *flags = ZADD_NAN;
+        return 0;
+    }
+
+    /* Update the sorted set according to its encoding. */
+    if (zobj->encoding == OBJ_ENCODING_ZIPLIST) {
+        unsigned char *eptr;
+
+        if ((eptr = zzlFind(zobj->ptr,ele,&curscore)) != NULL) {
+            /* NX? Return, same element already exists. */
+            if (nx) {
+                *flags |= ZADD_NOP;
+                return 1;
+            }
+
+            /* Prepare the score for the increment if needed. */
+            if (incr) {
+                score += curscore;
+                if (isnan(score)) {
+                    *flags |= ZADD_NAN;
+                    return 0;
+                }
+                if (newscore) *newscore = score;
+            }
+
+            /* Remove and re-insert when score changed. */
+            if (score != curscore) {
+                zobj->ptr = zzlDelete(zobj->ptr,eptr);
+                zobj->ptr = zzlInsert(zobj->ptr,ele,score);
+                *flags |= ZADD_UPDATED;
+            }
+            return 1;
+        } else if (!xx) {
+            /* Optimize: check if the element is too large or the list
+             * becomes too long *before* executing zzlInsert. */
+            zobj->ptr = zzlInsert(zobj->ptr,ele,score);
+            if (zzlLength(zobj->ptr) > server.zset_max_ziplist_entries)
+                zsetConvert(zobj,OBJ_ENCODING_SKIPLIST);
+            if (sdslen(ele) > server.zset_max_ziplist_value)
+                zsetConvert(zobj,OBJ_ENCODING_SKIPLIST);
+            if (newscore) *newscore = score;
+            *flags |= ZADD_ADDED;
+            return 1;
+        } else {
+            *flags |= ZADD_NOP;
+            return 1;
+        }
+    } else if (zobj->encoding == OBJ_ENCODING_SKIPLIST) {
+        zset *zs = zobj->ptr;
+        zskiplistNode *znode;
+        dictEntry *de;
+
+        de = dictFind(zs->dict,ele);
+        if (de != NULL) {
+            /* NX? Return, same element already exists. */
+            if (nx) {
+                *flags |= ZADD_NOP;
+                return 1;
+            }
+            curscore = *(double*)dictGetVal(de);
+
+            /* Prepare the score for the increment if needed. */
+            if (incr) {
+                score += curscore;
+                if (isnan(score)) {
+                    *flags |= ZADD_NAN;
+                    return 0;
+                }
+                if (newscore) *newscore = score;
+            }
+
+            /* Remove and re-insert when score changes. */
+            if (score != curscore) {
+                zskiplistNode *node;
+                serverAssert(zslDelete(zs->zsl,curscore,ele,&node));
+                znode = zslInsert(zs->zsl,score,node->ele);
+                /* We reused the node->ele SDS string, free the node now
+                 * since zslInsert created a new one. */
+                node->ele = NULL;
+                zslFreeNode(node);
+                /* Note that we did not removed the original element from
+                 * the hash table representing the sorted set, so we just
+                 * update the score. */
+                dictGetVal(de) = &znode->score; /* Update score ptr. */
+                *flags |= ZADD_UPDATED;
+            }
+            return 1;
+        } else if (!xx) {
+            ele = sdsdup(ele);
+            znode = zslInsert(zs->zsl,score,ele);
+            serverAssert(dictAdd(zs->dict,ele,&znode->score) == DICT_OK);
+            *flags |= ZADD_ADDED;
+            if (newscore) *newscore = score;
+            return 1;
+        } else {
+            *flags |= ZADD_NOP;
+            return 1;
+        }
+    } else {
+        serverPanic("Unknown sorted set encoding");
+    }
+    return 0; /* Never reached. */
+}
+
+/* Delete the element 'ele' from the sorted set, returning 1 if the element
+ * existed and was deleted, 0 otherwise (the element was not there). */
+int zsetDel(robj *zobj, sds ele) {
+    if (zobj->encoding == OBJ_ENCODING_ZIPLIST) {
+        unsigned char *eptr;
+
+        if ((eptr = zzlFind(zobj->ptr,ele,NULL)) != NULL) {
+            zobj->ptr = zzlDelete(zobj->ptr,eptr);
+            return 1;
+        }
+    } else if (zobj->encoding == OBJ_ENCODING_SKIPLIST) {
+        zset *zs = zobj->ptr;
+        dictEntry *de;
+        double score;
+
+        de = dictUnlink(zs->dict,ele);
+        if (de != NULL) {
+            /* Get the score in order to delete from the skiplist later. */
+            score = *(double*)dictGetVal(de);
+
+            /* Delete from the hash table and later from the skiplist.
+             * Note that the order is important: deleting from the skiplist
+             * actually releases the SDS string representing the element,
+             * which is shared between the skiplist and the hash table, so
+             * we need to delete from the skiplist as the final step. */
+            dictFreeUnlinkedEntry(zs->dict,de);
+
+            /* Delete from skiplist. */
+            int retval = zslDelete(zs->zsl,score,ele,NULL);
+            serverAssert(retval);
+
+            if (htNeedsResize(zs->dict)) dictResize(zs->dict);
+            return 1;
+        }
+    } else {
+        serverPanic("Unknown sorted set encoding");
+    }
+    return 0; /* No such element found. */
+}
+
+/* Given a sorted set object returns the 0-based rank of the object or
+ * -1 if the object does not exist.
+ *
+ * For rank we mean the position of the element in the sorted collection
+ * of elements. So the first element has rank 0, the second rank 1, and so
+ * forth up to length-1 elements.
+ *
+ * If 'reverse' is false, the rank is returned considering as first element
+ * the one with the lowest score. Otherwise if 'reverse' is non-zero
+ * the rank is computed considering as element with rank 0 the one with
+ * the highest score. */
+long zsetRank(robj *zobj, sds ele, int reverse) {
+    unsigned long llen;
+    unsigned long rank;
+
+    llen = zsetLength(zobj);
+
+    if (zobj->encoding == OBJ_ENCODING_ZIPLIST) {
+        unsigned char *zl = zobj->ptr;
+        unsigned char *eptr, *sptr;
+
+        eptr = ziplistIndex(zl,0);
+        serverAssert(eptr != NULL);
+        sptr = ziplistNext(zl,eptr);
+        serverAssert(sptr != NULL);
+
+        rank = 1;
+        while(eptr != NULL) {
+            if (ziplistCompare(eptr,(unsigned char*)ele,sdslen(ele)))
+                break;
+            rank++;
+            zzlNext(zl,&eptr,&sptr);
+        }
+
+        if (eptr != NULL) {
+            if (reverse)
+                return llen-rank;
+            else
+                return rank-1;
+        } else {
+            return -1;
+        }
+    } else if (zobj->encoding == OBJ_ENCODING_SKIPLIST) {
+        zset *zs = zobj->ptr;
+        zskiplist *zsl = zs->zsl;
+        dictEntry *de;
+        double score;
+
+        de = dictFind(zs->dict,ele);
+        if (de != NULL) {
+            score = *(double*)dictGetVal(de);
+            rank = zslGetRank(zsl,score,ele);
+            /* Existing elements always have a rank. */
+            serverAssert(rank != 0);
+            if (reverse)
+                return llen-rank;
+            else
+                return rank-1;
+        } else {
+            return -1;
+        }
+    } else {
+        serverPanic("Unknown sorted set encoding");
+    }
+}
+
 /*-----------------------------------------------------------------------------
  * Sorted set commands
  *----------------------------------------------------------------------------*/
 
 /* This generic command implements both ZADD and ZINCRBY. */
-#define ZADD_NONE 0
-#define ZADD_INCR (1<<0)    /* Increment the score instead of setting it. */
-#define ZADD_NX (1<<1)      /* Don't touch elements not already existing. */
-#define ZADD_XX (1<<2)      /* Only touch elements already exisitng. */
-#define ZADD_CH (1<<3)      /* Return num of elements added or updated. */
 void zaddGenericCommand(client *c, int flags) {
     static char *nanerr = "resulting score is not a number (NaN)";
     robj *key = c->argv[1];
     robj *zobj;
     sds ele;
-    double score = 0, *scores = NULL, curscore = 0.0;
+    double score = 0, *scores = NULL;
     int j, elements;
     int scoreidx = 0;
     /* The following vars are used in order to track what the command actually
@@ -1299,91 +1569,22 @@ void zaddGenericCommand(client *c, int flags) {
     }
 
     for (j = 0; j < elements; j++) {
+        double newscore;
         score = scores[j];
+        int retflags = flags;
 
         ele = c->argv[scoreidx+1+j*2]->ptr;
-        if (zobj->encoding == OBJ_ENCODING_ZIPLIST) {
-            unsigned char *eptr;
-
-            if ((eptr = zzlFind(zobj->ptr,ele,&curscore)) != NULL) {
-                if (nx) continue;
-                if (incr) {
-                    score += curscore;
-                    if (isnan(score)) {
-                        addReplyError(c,nanerr);
-                        goto cleanup;
-                    }
-                }
-
-                /* Remove and re-insert when score changed. */
-                if (score != curscore) {
-                    zobj->ptr = zzlDelete(zobj->ptr,eptr);
-                    zobj->ptr = zzlInsert(zobj->ptr,ele,score);
-                    server.dirty++;
-                    updated++;
-                }
-                processed++;
-            } else if (!xx) {
-                /* Optimize: check if the element is too large or the list
-                 * becomes too long *before* executing zzlInsert. */
-                zobj->ptr = zzlInsert(zobj->ptr,ele,score);
-                if (zzlLength(zobj->ptr) > server.zset_max_ziplist_entries)
-                    zsetConvert(zobj,OBJ_ENCODING_SKIPLIST);
-                if (sdslen(ele) > server.zset_max_ziplist_value)
-                    zsetConvert(zobj,OBJ_ENCODING_SKIPLIST);
-                server.dirty++;
-                added++;
-                processed++;
-            }
-        } else if (zobj->encoding == OBJ_ENCODING_SKIPLIST) {
-            zset *zs = zobj->ptr;
-            zskiplistNode *znode;
-            dictEntry *de;
-
-            de = dictFind(zs->dict,ele);
-            if (de != NULL) {
-                if (nx) continue;
-                curscore = *(double*)dictGetVal(de);
-
-                if (incr) {
-                    score += curscore;
-                    if (isnan(score)) {
-                        addReplyError(c,nanerr);
-                        /* Don't need to check if the sorted set is empty
-                         * because we know it has at least one element. */
-                        goto cleanup;
-                    }
-                }
-
-                /* Remove and re-insert when score changes. */
-                if (score != curscore) {
-                    zskiplistNode *node;
-                    serverAssert(zslDelete(zs->zsl,curscore,ele,&node));
-                    znode = zslInsert(zs->zsl,score,node->ele);
-                    /* We reused the node->ele SDS string, free the node now
-                     * since zslInsert created a new one. */
-                    node->ele = NULL;
-                    zslFreeNode(node);
-                    /* Note that we did not removed the original element from
-                     * the hash table representing the sorted set, so we just
-                     * update the score. */
-                    dictGetVal(de) = &znode->score; /* Update score ptr. */
-                    server.dirty++;
-                    updated++;
-                }
-                processed++;
-            } else if (!xx) {
-                ele = sdsdup(ele);
-                znode = zslInsert(zs->zsl,score,ele);
-                serverAssert(dictAdd(zs->dict,ele,&znode->score) == DICT_OK);
-                server.dirty++;
-                added++;
-                processed++;
-            }
-        } else {
-            serverPanic("Unknown sorted set encoding");
+        int retval = zsetAdd(zobj, score, ele, &retflags, &newscore);
+        if (retval == 0) {
+            addReplyError(c,nanerr);
+            goto cleanup;
         }
+        if (retflags & ZADD_ADDED) added++;
+        if (retflags & ZADD_UPDATED) updated++;
+        if (!(retflags & ZADD_NOP)) processed++;
+        score = newscore;
     }
+    server.dirty += (added+updated);
 
 reply_to_client:
     if (incr) { /* ZINCRBY or INCR option. */
@@ -1420,56 +1621,13 @@ void zremCommand(client *c) {
     if ((zobj = lookupKeyWriteOrReply(c,key,shared.czero)) == NULL ||
         checkType(c,zobj,OBJ_ZSET)) return;
 
-    if (zobj->encoding == OBJ_ENCODING_ZIPLIST) {
-        unsigned char *eptr;
-
-        for (j = 2; j < c->argc; j++) {
-            if ((eptr = zzlFind(zobj->ptr,c->argv[j]->ptr,NULL)) != NULL) {
-                deleted++;
-                zobj->ptr = zzlDelete(zobj->ptr,eptr);
-                if (zzlLength(zobj->ptr) == 0) {
-                    dbDelete(c->db,key);
-                    keyremoved = 1;
-                    break;
-                }
-            }
+    for (j = 2; j < c->argc; j++) {
+        if (zsetDel(zobj,c->argv[j]->ptr)) deleted++;
+        if (zsetLength(zobj) == 0) {
+            dbDelete(c->db,key);
+            keyremoved = 1;
+            break;
         }
-    } else if (zobj->encoding == OBJ_ENCODING_SKIPLIST) {
-        zset *zs = zobj->ptr;
-        dictEntry *de;
-        double score;
-
-        for (j = 2; j < c->argc; j++) {
-            de = dictFind(zs->dict,c->argv[j]->ptr);
-            if (de != NULL) {
-                deleted++;
-
-                /* Get the score in order to delete from the skiplist later. */
-                score = *(double*)dictGetVal(de);
-
-                /* Delete from the hash table and later from the skiplist.
-                 * Note that the order is important: deleting from the skiplist
-                 * actually releases the SDS string representing the element,
-                 * which is shared between the skiplist and the hash table, so
-                 * we need to delete from the skiplist as the final step. */
-                int retval1 = dictDelete(zs->dict,c->argv[j]->ptr);
-
-                /* Delete from skiplist. */
-                int retval2 = zslDelete(zs->zsl,score,c->argv[j]->ptr,NULL);
-
-                serverAssertWithInfo(c,c->argv[j],
-                    retval1 == DICT_OK && retval2);
-
-                if (htNeedsResize(zs->dict)) dictResize(zs->dict);
-                if (dictSize(zs->dict) == 0) {
-                    dbDelete(c->db,key);
-                    keyremoved = 1;
-                    break;
-                }
-            }
-        }
-    } else {
-        serverPanic("Unknown sorted set encoding");
     }
 
     if (deleted) {
@@ -2103,7 +2261,7 @@ void zunionInterGenericCommand(client *c, robj *dstkey, int op) {
     } else if (op == SET_OP_UNION) {
         dict *accumulator = dictCreate(&setAccumulatorDictType,NULL);
         dictIterator *di;
-        dictEntry *de;
+        dictEntry *de, *existing;
         double score;
 
         if (setnum) {
@@ -2124,16 +2282,16 @@ void zunionInterGenericCommand(client *c, robj *dstkey, int op) {
                 if (isnan(score)) score = 0;
 
                 /* Search for this element in the accumulating dictionary. */
-                de = dictFind(accumulator,zuiSdsFromValue(&zval));
+                de = dictAddRaw(accumulator,zuiSdsFromValue(&zval),&existing);
                 /* If we don't have it, we need to create a new entry. */
-                if (de == NULL) {
+                if (!existing) {
                     tmp = zuiNewSdsFromValue(&zval);
                     /* Remember the longest single element encountered,
                      * to understand if it's possible to convert to ziplist
                      * at the end. */
                      if (sdslen(tmp) > maxelelen) maxelelen = sdslen(tmp);
-                    /* Add the element with its initial score. */
-                    de = dictAddRaw(accumulator,tmp);
+                    /* Update the element with its initial score. */
+                    dictSetKey(accumulator, de, tmp);
                     dictSetDoubleVal(de,score);
                 } else {
                     /* Update the score with the score of the new instance
@@ -2142,7 +2300,7 @@ void zunionInterGenericCommand(client *c, robj *dstkey, int op) {
                      * Here we access directly the dictEntry double
                      * value inside the union as it is a big speedup
                      * compared to using the getDouble/setDouble API. */
-                    zunionInterAggregate(&de->v.d,score,aggregate);
+                    zunionInterAggregate(&existing->v.d,score,aggregate);
                 }
             }
             zuiClearIterator(&src[i]);
@@ -2168,20 +2326,13 @@ void zunionInterGenericCommand(client *c, robj *dstkey, int op) {
         serverPanic("Unknown operator");
     }
 
-    if (dbDelete(c->db,dstkey)) {
-        signalModifiedKey(c->db,dstkey);
+    if (dbDelete(c->db,dstkey))
         touched = 1;
-        server.dirty++;
-    }
     if (dstzset->zsl->length) {
-        /* Convert to ziplist when in limits. */
-        if (dstzset->zsl->length <= server.zset_max_ziplist_entries &&
-            maxelelen <= server.zset_max_ziplist_value)
-                zsetConvert(dstobj,OBJ_ENCODING_ZIPLIST);
-
+        zsetConvertToZiplistIfNeeded(dstobj,maxelelen);
         dbAdd(c->db,dstkey,dstobj);
         addReplyLongLong(c,zsetLength(dstobj));
-        if (!touched) signalModifiedKey(c->db,dstkey);
+        signalModifiedKey(c->db,dstkey);
         notifyKeyspaceEvent(NOTIFY_ZSET,
             (op == SET_OP_UNION) ? "zunionstore" : "zinterstore",
             dstkey,c->db->id);
@@ -2189,8 +2340,11 @@ void zunionInterGenericCommand(client *c, robj *dstkey, int op) {
     } else {
         decrRefCount(dstobj);
         addReply(c,shared.czero);
-        if (touched)
+        if (touched) {
+            signalModifiedKey(c->db,dstkey);
             notifyKeyspaceEvent(NOTIFY_GENERIC,"del",dstkey,c->db->id);
+            server.dirty++;
+        }
     }
     zfree(src);
 }
@@ -2883,62 +3037,17 @@ void zrankGenericCommand(client *c, int reverse) {
     robj *key = c->argv[1];
     robj *ele = c->argv[2];
     robj *zobj;
-    unsigned long llen;
-    unsigned long rank;
+    long rank;
 
     if ((zobj = lookupKeyReadOrReply(c,key,shared.nullbulk)) == NULL ||
         checkType(c,zobj,OBJ_ZSET)) return;
-    llen = zsetLength(zobj);
 
     serverAssertWithInfo(c,ele,sdsEncodedObject(ele));
-
-    if (zobj->encoding == OBJ_ENCODING_ZIPLIST) {
-        unsigned char *zl = zobj->ptr;
-        unsigned char *eptr, *sptr;
-
-        eptr = ziplistIndex(zl,0);
-        serverAssertWithInfo(c,zobj,eptr != NULL);
-        sptr = ziplistNext(zl,eptr);
-        serverAssertWithInfo(c,zobj,sptr != NULL);
-
-        rank = 1;
-        while(eptr != NULL) {
-            if (ziplistCompare(eptr,ele->ptr,sdslen(ele->ptr)))
-                break;
-            rank++;
-            zzlNext(zl,&eptr,&sptr);
-        }
-
-        if (eptr != NULL) {
-            if (reverse)
-                addReplyLongLong(c,llen-rank);
-            else
-                addReplyLongLong(c,rank-1);
-        } else {
-            addReply(c,shared.nullbulk);
-        }
-    } else if (zobj->encoding == OBJ_ENCODING_SKIPLIST) {
-        zset *zs = zobj->ptr;
-        zskiplist *zsl = zs->zsl;
-        dictEntry *de;
-        double score;
-
-        ele = c->argv[2];
-        de = dictFind(zs->dict,ele->ptr);
-        if (de != NULL) {
-            score = *(double*)dictGetVal(de);
-            rank = zslGetRank(zsl,score,ele->ptr);
-            /* Existing elements always have a rank. */
-            serverAssertWithInfo(c,ele,rank);
-            if (reverse)
-                addReplyLongLong(c,llen-rank);
-            else
-                addReplyLongLong(c,rank-1);
-        } else {
-            addReply(c,shared.nullbulk);
-        }
+    rank = zsetRank(zobj,ele->ptr,reverse);
+    if (rank >= 0) {
+        addReplyLongLong(c,rank);
     } else {
-        serverPanic("Unknown sorted set encoding");
+        addReply(c,shared.nullbulk);
     }
 }
 

src/util.c

@@ -274,7 +274,7 @@ uint32_t sdigits10(int64_t v) {
  *
  * Modified in order to handle signed integers since the original code was
  * designed for unsigned integers. */
-int ll2string(char* dst, size_t dstlen, long long svalue) {
+int ll2string(char *dst, size_t dstlen, long long svalue) {
     static const char digits[201] =
         "0001020304050607080910111213141516171819"
         "2021222324252627282930313233343536373839"

src/zmalloc.c

@@ -72,7 +72,7 @@ void zlibc_free(void *ptr) {
     size_t _n = (__n); \
     if (_n&(sizeof(long)-1)) _n += sizeof(long)-(_n&(sizeof(long)-1)); \
     if (zmalloc_thread_safe) { \
-        atomicIncr(used_memory,__n,&used_memory_mutex); \
+        atomicIncr(used_memory,__n,used_memory_mutex); \
     } else { \
         used_memory += _n; \
     } \
@@ -82,7 +82,7 @@ void zlibc_free(void *ptr) {
     size_t _n = (__n); \
     if (_n&(sizeof(long)-1)) _n += sizeof(long)-(_n&(sizeof(long)-1)); \
     if (zmalloc_thread_safe) { \
-        atomicDecr(used_memory,__n,&used_memory_mutex); \
+        atomicDecr(used_memory,__n,used_memory_mutex); \
     } else { \
         used_memory -= _n; \
     } \
@@ -202,7 +202,7 @@ size_t zmalloc_used_memory(void) {
     size_t um;
 
     if (zmalloc_thread_safe) {
-        atomicGet(used_memory,um,&used_memory_mutex);
+        atomicGet(used_memory,um,used_memory_mutex);
     } else {
         um = used_memory;
     }
@@ -304,14 +304,26 @@ float zmalloc_get_fragmentation_ratio(size_t rss) {
  * /proc/self/smaps. The field must be specified with trailing ":" as it
  * apperas in the smaps output.
  *
- * Example: zmalloc_get_smap_bytes_by_field("Rss:");
+ * If a pid is specified, the information is extracted for such a pid,
+ * otherwise if pid is -1 the information is reported is about the
+ * current process.
+ *
+ * Example: zmalloc_get_smap_bytes_by_field("Rss:",-1);
  */
 #if defined(HAVE_PROC_SMAPS)
-size_t zmalloc_get_smap_bytes_by_field(char *field) {
+size_t zmalloc_get_smap_bytes_by_field(char *field, long pid) {
     char line[1024];
     size_t bytes = 0;
-    FILE *fp = fopen("/proc/self/smaps","r");
     int flen = strlen(field);
+    FILE *fp;
+
+    if (pid == -1) {
+        fp = fopen("/proc/self/smaps","r");
+    } else {
+        char filename[128];
+        snprintf(filename,sizeof(filename),"/proc/%ld/smaps",pid);
+        fp = fopen(filename,"r");
+    }
 
     if (!fp) return 0;
     while(fgets(line,sizeof(line),fp) != NULL) {
@@ -327,14 +339,15 @@ size_t zmalloc_get_smap_bytes_by_field(char *field) {
     return bytes;
 }
 #else
-size_t zmalloc_get_smap_bytes_by_field(char *field) {
+size_t zmalloc_get_smap_bytes_by_field(char *field, long pid) {
     ((void) field);
+    ((void) pid);
     return 0;
 }
 #endif
 
-size_t zmalloc_get_private_dirty(void) {
-    return zmalloc_get_smap_bytes_by_field("Private_Dirty:");
+size_t zmalloc_get_private_dirty(long pid) {
+    return zmalloc_get_smap_bytes_by_field("Private_Dirty:",pid);
 }
 
 /* Returns the size of physical memory (RAM) in bytes.

src/zmalloc.h

@@ -75,8 +75,8 @@ void zmalloc_enable_thread_safeness(void);
 void zmalloc_set_oom_handler(void (*oom_handler)(size_t));
 float zmalloc_get_fragmentation_ratio(size_t rss);
 size_t zmalloc_get_rss(void);
-size_t zmalloc_get_private_dirty(void);
-size_t zmalloc_get_smap_bytes_by_field(char *field);
+size_t zmalloc_get_private_dirty(long pid);
+size_t zmalloc_get_smap_bytes_by_field(char *field, long pid);
 size_t zmalloc_get_memory_size(void);
 void zlibc_free(void *ptr);
 

tests/cluster/tests/04-resharding.tcl

@@ -13,6 +13,24 @@ test "Cluster is up" {
     assert_cluster_state ok
 }
 
+test "Enable AOF in all the instances" {
+    foreach_redis_id id {
+        R $id config set appendonly yes
+        # We use "appendfsync no" because it's fast but also guarantees that
+        # write(2) is performed before replying to client.
+        R $id config set appendfsync no
+    }
+
+    foreach_redis_id id {
+        wait_for_condition 1000 500 {
+            [RI $id aof_rewrite_in_progress] == 0 &&
+            [RI $id aof_enabled] == 1
+        } else {
+            fail "Failed to enable AOF on instance #$id"
+        }
+    }
+}
+
 # Return nno-zero if the specified PID is about a process still in execution,
 # otherwise 0 is returned.
 proc process_is_running {pid} {
@@ -41,6 +59,7 @@ array set content {}
 set tribpid {}
 
 test "Cluster consistency during live resharding" {
+    set ele 0
     for {set j 0} {$j < $numops} {incr j} {
         # Trigger the resharding once we execute half the ops.
         if {$tribpid ne {} &&
@@ -68,7 +87,7 @@ test "Cluster consistency during live resharding" {
         # Write random data to random list.
         set listid [randomInt $numkeys]
         set key "key:$listid"
-        set ele [randomValue]
+        incr ele
         # We write both with Lua scripts and with plain commands.
         # This way we are able to stress Lua -> Redis command invocation
         # as well, that has tests to prevent Lua to write into wrong
@@ -97,6 +116,57 @@ test "Cluster consistency during live resharding" {
 test "Verify $numkeys keys for consistency with logical content" {
     # Check that the Redis Cluster content matches our logical content.
     foreach {key value} [array get content] {
-        assert {[$cluster lrange $key 0 -1] eq $value}
+        if {[$cluster lrange $key 0 -1] ne $value} {
+            fail "Key $key expected to hold '$value' but actual content is [$cluster lrange $key 0 -1]"
+        }
     }
 }
+
+test "Crash and restart all the instances" {
+    foreach_redis_id id {
+        kill_instance redis $id
+        restart_instance redis $id
+    }
+}
+
+test "Cluster should eventually be up again" {
+    assert_cluster_state ok
+}
+
+test "Verify $numkeys keys after the crash & restart" {
+    # Check that the Redis Cluster content matches our logical content.
+    foreach {key value} [array get content] {
+        if {[$cluster lrange $key 0 -1] ne $value} {
+            fail "Key $key expected to hold '$value' but actual content is [$cluster lrange $key 0 -1]"
+        }
+    }
+}
+
+test "Disable AOF in all the instances" {
+    foreach_redis_id id {
+        R $id config set appendonly no
+    }
+}
+
+test "Verify slaves consistency" {
+    set verified_masters 0
+    foreach_redis_id id {
+        set role [R $id role]
+        lassign $role myrole myoffset slaves
+        if {$myrole eq {slave}} continue
+        set masterport [get_instance_attrib redis $id port]
+        set masterdigest [R $id debug digest]
+        foreach_redis_id sid {
+            set srole [R $sid role]
+            if {[lindex $srole 0] eq {master}} continue
+            if {[lindex $srole 2] != $masterport} continue
+            wait_for_condition 1000 500 {
+                [R $sid debug digest] eq $masterdigest
+            } else {
+                fail "Master and slave data digest are different"
+            }
+            incr verified_masters
+        }
+    }
+    assert {$verified_masters >= 5}
+}

tests/cluster/tests/07-replica-migration.tcl

@@ -45,3 +45,59 @@ foreach_redis_id id {
         }
     }
 }
+
+# Now test the migration to a master which used to be a slave, after
+# a failver.
+
+source "../tests/includes/init-tests.tcl"
+
+# Create a cluster with 5 master and 10 slaves, so that we have 2
+# slaves for each master.
+test "Create a 5 nodes cluster" {
+    create_cluster 5 10
+}
+
+test "Cluster is up" {
+    assert_cluster_state ok
+}
+
+test "Kill slave #7 of master #2. Only slave left is #12 now" {
+    kill_instance redis 7
+}
+
+set current_epoch [CI 1 cluster_current_epoch]
+
+test "Killing master node #2, #12 should failover" {
+    kill_instance redis 2
+}
+
+test "Wait for failover" {
+    wait_for_condition 1000 50 {
+        [CI 1 cluster_current_epoch] > $current_epoch
+    } else {
+        fail "No failover detected"
+    }
+}
+
+test "Cluster should eventually be up again" {
+    assert_cluster_state ok
+}
+
+test "Cluster is writable" {
+    cluster_write_test 1
+}
+
+test "Instance 12 is now a master without slaves" {
+    assert {[RI 12 role] eq {master}}
+}
+
+# The remaining instance is now without slaves. Some other slave
+# should migrate to it.
+
+test "Master #12 should get at least one migrated replica" {
+    wait_for_condition 1000 50 {
+        [llength [lindex [R 12 role] 2]] >= 1
+    } else {
+        fail "Master #12 has no replicas"
+    }
+}

tests/cluster/tests/12-replica-migration-2.tcl

@@ -0,0 +1,64 @@
+# Replica migration test #2.
+#
+# Check that the status of master that can be targeted by replica migration
+# is acquired again, after being getting slots again, in a cluster where the
+# other masters have slaves.
+
+source "../tests/includes/init-tests.tcl"
+
+# Create a cluster with 5 master and 15 slaves, to make sure there are no
+# empty masters and make rebalancing simpler to handle during the test.
+test "Create a 5 nodes cluster" {
+    create_cluster 5 15
+}
+
+test "Cluster is up" {
+    assert_cluster_state ok
+}
+
+test "Each master should have at least two replicas attached" {
+    foreach_redis_id id {
+        if {$id < 5} {
+            wait_for_condition 1000 50 {
+                [llength [lindex [R 0 role] 2]] >= 2
+            } else {
+                fail "Master #$id does not have 2 slaves as expected"
+            }
+        }
+    }
+}
+
+set master0_id [dict get [get_myself 0] id]
+test "Resharding all the master #0 slots away from it" {
+    set output [exec \
+        ../../../src/redis-trib.rb rebalance \
+        --weight ${master0_id}=0 \
+        127.0.0.1:[get_instance_attrib redis 0 port] >@ stdout]
+}
+
+test "Master #0 should lose its replicas" {
+    wait_for_condition 1000 50 {
+        [llength [lindex [R 0 role] 2]] == 0
+    } else {
+        fail "Master #0 still has replicas"
+    }
+}
+
+test "Resharding back some slot to master #0" {
+    # Wait for the cluster config to propagate before attempting a
+    # new resharding.
+    after 10000
+    set output [exec \
+        ../../../src/redis-trib.rb rebalance \
+        --weight ${master0_id}=.01 \
+        --use-empty-masters \
+        127.0.0.1:[get_instance_attrib redis 0 port] >@ stdout]
+}
+
+test "Master #0 should re-acquire one or more replicas" {
+    wait_for_condition 1000 50 {
+        [llength [lindex [R 0 role] 2]] >= 1
+    } else {
+        fail "Master #0 has no has replicas"
+    }
+}

tests/instances.tcl

@@ -99,8 +99,25 @@ proc spawn_instance {type base_port count {conf {}}} {
     }
 }
 
+proc log_crashes {} {
+    set start_pattern {*REDIS BUG REPORT START*}
+    set logs [glob */log.txt]
+    foreach log $logs {
+        set fd [open $log]
+        set found 0
+        while {[gets $fd line] >= 0} {
+            if {[string match $start_pattern $line]} {
+                puts "\n*** Crash report found in $log ***"
+                set found 1
+            }
+            if {$found} {puts $line}
+        }
+    }
+}
+
 proc cleanup {} {
     puts "Cleaning up..."
+    log_crashes
     foreach pid $::pids {
         catch {exec kill -9 $pid}
     }
@@ -110,6 +127,7 @@ proc cleanup {} {
 }
 
 proc abort_sentinel_test msg {
+    incr ::failed
     puts "WARNING: Aborting the test."
     puts ">>>>>>>> $msg"
     if {$::pause_on_error} pause_on_error
@@ -136,6 +154,7 @@ proc parse_options {} {
             puts "--single <pattern>      Only runs tests specified by pattern."
             puts "--pause-on-error        Pause for manual inspection on error."
             puts "--fail                  Simulate a test failure."
+            puts "--valgrind              Run with valgrind."
             puts "--help                  Shows this help."
             exit 0
         } else {
@@ -248,6 +267,37 @@ proc test {descr code} {
     }
 }
 
+# Check memory leaks when running on OSX using the "leaks" utility.
+proc check_leaks instance_types {
+    if {[string match {*Darwin*} [exec uname -a]]} {
+        puts -nonewline "Testing for memory leaks..."; flush stdout
+        foreach type $instance_types {
+            foreach_instance_id [set ::${type}_instances] id {
+                if {[instance_is_killed $type $id]} continue
+                set pid [get_instance_attrib $type $id pid]
+                set output {0 leaks}
+                catch {exec leaks $pid} output
+                if {[string match {*process does not exist*} $output] ||
+                    [string match {*cannot examine*} $output]} {
+                    # In a few tests we kill the server process.
+                    set output "0 leaks"
+                } else {
+                    puts -nonewline "$type/$pid "
+                    flush stdout
+                }
+                if {![string match {*0 leaks*} $output]} {
+                    puts [colorstr red "=== MEMORY LEAK DETECTED ==="]
+                    puts "Instance type $type, ID $id:"
+                    puts $output
+                    puts "==="
+                    incr ::failed
+                }
+            }
+        }
+        puts ""
+    }
+}
+
 # Execute all the units inside the 'tests' directory.
 proc run_tests {} {
     set tests [lsort [glob ../tests/*]]
@@ -258,6 +308,7 @@ proc run_tests {} {
         if {[file isdirectory $test]} continue
         puts [colorstr yellow "Testing unit: [lindex [file split $test] end]"]
         source $test
+        check_leaks {redis sentinel}
     }
 }
 
@@ -444,5 +495,17 @@ proc restart_instance {type id} {
     set link [redis 127.0.0.1 $port]
     $link reconnect 1
     set_instance_attrib $type $id link $link
+
+    # Make sure the instance is not loading the dataset when this
+    # function returns.
+    while 1 {
+        catch {[$link ping]} retval
+        if {[string match {*LOADING*} $retval]} {
+            after 100
+            continue
+        } else {
+            break
+        }
+    }
 }
 

tests/integration/rdb.tcl

@@ -89,7 +89,7 @@ close $fd
 start_server_and_kill_it [list "dir" $server_path] {
     test {Server should not start if RDB is corrupted} {
         wait_for_condition 50 100 {
-            [string match {*RDB checksum*} \
+            [string match {*CRC error*} \
                 [exec tail -n10 < [dict get $srv stdout]]]
         } else {
             fail "Server started even if RDB was corrupted!"

tests/sentinel/tests/05-manual.tcl

@@ -6,7 +6,8 @@ test "Manual failover works" {
     set old_port [RI $master_id tcp_port]
     set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster]
     assert {[lindex $addr 1] == $old_port}
-    S 0 SENTINEL FAILOVER mymaster
+    catch {S 0 SENTINEL FAILOVER mymaster} reply
+    assert {$reply eq "OK"}
     foreach_sentinel_id id {
         wait_for_condition 1000 50 {
             [lindex [S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port

tests/sentinel/tests/07-down-conditions.tcl

@@ -0,0 +1,68 @@
+# Test conditions where an instance is considered to be down
+
+source "../tests/includes/init-tests.tcl"
+
+proc ensure_master_up {} {
+    wait_for_condition 1000 50 {
+        [dict get [S 4 sentinel master mymaster] flags] eq "master"
+    } else {
+        fail "Master flags are not just 'master'"
+    }
+}
+
+proc ensure_master_down {} {
+    wait_for_condition 1000 50 {
+        [string match *down* \
+            [dict get [S 4 sentinel master mymaster] flags]]
+    } else {
+        fail "Master is not flagged SDOWN"
+    }
+}
+
+test "Crash the majority of Sentinels to prevent failovers for this unit" {
+    for {set id 0} {$id < $quorum} {incr id} {
+        kill_instance sentinel $id
+    }
+}
+
+test "SDOWN is triggered by non-responding but not crashed instance" {
+    lassign [S 4 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] host port
+    ensure_master_up
+    exec ../../../src/redis-cli -h $host -p $port debug sleep 10 > /dev/null &
+    ensure_master_down
+    ensure_master_up
+}
+
+test "SDOWN is triggered by crashed instance" {
+    lassign [S 4 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] host port
+    ensure_master_up
+    kill_instance redis 0
+    ensure_master_down
+    restart_instance redis 0
+    ensure_master_up
+}
+
+test "SDOWN is triggered by masters advertising as slaves" {
+    ensure_master_up
+    R 0 slaveof 127.0.0.1 34567
+    ensure_master_down
+    R 0 slaveof no one
+    ensure_master_up
+}
+
+test "SDOWN is triggered by misconfigured instance repling with errors" {
+    ensure_master_up
+    set orig_dir [lindex [R 0 config get dir] 1]
+    set orig_save [lindex [R 0 config get save] 1]
+    # Set dir to / and filename to "tmp" to make sure it will fail.
+    R 0 config set dir /
+    R 0 config set dbfilename tmp
+    R 0 config set save "1000000 1000000"
+    R 0 bgsave
+    ensure_master_down
+    R 0 config set save $orig_save
+    R 0 config set dir $orig_dir
+    R 0 config set dbfilename dump.rdb
+    R 0 bgsave
+    ensure_master_up
+}

tests/support/cluster.tcl

@@ -58,7 +58,8 @@ proc ::redis_cluster::__method__refresh_nodes_map {id} {
     set idx 0; # Index of the node that will respond.
     set errmsg {}
     foreach start_node $::redis_cluster::startup_nodes($id) {
-        lassign [split $start_node :] start_host start_port
+        set ip_port [lindex [split $start_node @] 0]
+        lassign [split $ip_port :] start_host start_port
         if {[catch {
             set r {}
             set r [redis $start_host $start_port]
@@ -68,7 +69,7 @@ proc ::redis_cluster::__method__refresh_nodes_map {id} {
             if {$r ne {}} {catch {$r close}}
             incr idx
             if {[string length $errmsg] < 200} {
-                append errmsg " $start_node: $e"
+                append errmsg " $ip_port: $e"
             }
             continue ; # Try next.
         } else {
@@ -98,6 +99,7 @@ proc ::redis_cluster::__method__refresh_nodes_map {id} {
         set args [split $line " "]
         lassign $args nodeid addr flags slaveof pingsent pongrecv configepoch linkstate
         set slots [lrange $args 8 end]
+        set addr [lindex [split $addr @] 0]
         if {$addr eq {:0}} {
             set addr $start_host:$start_port
         }

tests/support/test.tcl

@@ -37,13 +37,7 @@ proc assert_error {pattern code} {
 }
 
 proc assert_encoding {enc key} {
-    # Swapped out values don't have an encoding, so make sure that
-    # the value is swapped in before checking the encoding.
     set dbg [r debug object $key]
-    while {[string match "* swapped at:*" $dbg]} {
-        r debug swapin $key
-        set dbg [r debug object $key]
-    }
     assert_match "* encoding:$enc *" $dbg
 }
 

tests/test_helper.tcl

@@ -46,9 +46,12 @@ set ::all_tests {
     unit/scripting
     unit/maxmemory
     unit/introspection
+    unit/introspection-2
     unit/limits
     unit/obuf-limits
     unit/bitops
+    unit/bitfield
+    unit/geo
     unit/memefficiency
     unit/hyperloglog
     unit/lazyfree

tests/unit/aofrw.tcl

@@ -4,60 +4,63 @@ start_server {tags {"aofrw"}} {
     r config set auto-aof-rewrite-percentage 0 ; # Disable auto-rewrite.
     waitForBgrewriteaof r
 
-    test {AOF rewrite during write load} {
-        # Start a write load for 10 seconds
-        set master [srv 0 client]
-        set master_host [srv 0 host]
-        set master_port [srv 0 port]
-        set load_handle0 [start_write_load $master_host $master_port 10]
-        set load_handle1 [start_write_load $master_host $master_port 10]
-        set load_handle2 [start_write_load $master_host $master_port 10]
-        set load_handle3 [start_write_load $master_host $master_port 10]
-        set load_handle4 [start_write_load $master_host $master_port 10]
+    foreach rdbpre {yes no} {
+        r config set aof-use-rdb-preamble $rdbpre
+        test "AOF rewrite during write load: RDB preamble=$rdbpre" {
+            # Start a write load for 10 seconds
+            set master [srv 0 client]
+            set master_host [srv 0 host]
+            set master_port [srv 0 port]
+            set load_handle0 [start_write_load $master_host $master_port 10]
+            set load_handle1 [start_write_load $master_host $master_port 10]
+            set load_handle2 [start_write_load $master_host $master_port 10]
+            set load_handle3 [start_write_load $master_host $master_port 10]
+            set load_handle4 [start_write_load $master_host $master_port 10]
 
-        # Make sure the instance is really receiving data
-        wait_for_condition 50 100 {
-            [r dbsize] > 0
-        } else {
-            fail "No write load detected."
+            # Make sure the instance is really receiving data
+            wait_for_condition 50 100 {
+                [r dbsize] > 0
+            } else {
+                fail "No write load detected."
+            }
+
+            # After 3 seconds, start a rewrite, while the write load is still
+            # active.
+            after 3000
+            r bgrewriteaof
+            waitForBgrewriteaof r
+
+            # Let it run a bit more so that we'll append some data to the new
+            # AOF.
+            after 1000
+
+            # Stop the processes generating the load if they are still active
+            stop_write_load $load_handle0
+            stop_write_load $load_handle1
+            stop_write_load $load_handle2
+            stop_write_load $load_handle3
+            stop_write_load $load_handle4
+
+            # Make sure that we remain the only connected client.
+            # This step is needed to make sure there are no pending writes
+            # that will be processed between the two "debug digest" calls.
+            wait_for_condition 50 100 {
+                [llength [split [string trim [r client list]] "\n"]] == 1
+            } else {
+                puts [r client list]
+                fail "Clients generating loads are not disconnecting"
+            }
+
+            # Get the data set digest
+            set d1 [r debug digest]
+
+            # Load the AOF
+            r debug loadaof
+            set d2 [r debug digest]
+
+            # Make sure they are the same
+            assert {$d1 eq $d2}
         }
-
-        # After 3 seconds, start a rewrite, while the write load is still
-        # active.
-        after 3000
-        r bgrewriteaof
-        waitForBgrewriteaof r
-
-        # Let it run a bit more so that we'll append some data to the new
-        # AOF.
-        after 1000
-
-        # Stop the processes generating the load if they are still active
-        stop_write_load $load_handle0
-        stop_write_load $load_handle1
-        stop_write_load $load_handle2
-        stop_write_load $load_handle3
-        stop_write_load $load_handle4
-
-        # Make sure that we remain the only connected client.
-        # This step is needed to make sure there are no pending writes
-        # that will be processed between the two "debug digest" calls.
-        wait_for_condition 50 100 {
-            [llength [split [string trim [r client list]] "\n"]] == 1
-        } else {
-            puts [r client list]
-            fail "Clients generating loads are not disconnecting"
-        }
-
-        # Get the data set digest
-        set d1 [r debug digest]
-
-        # Load the AOF
-        r debug loadaof
-        set d2 [r debug digest]
-
-        # Make sure they are the same
-        assert {$d1 eq $d2}
     }
 }
 

tests/unit/bitfield.tcl

@@ -0,0 +1,192 @@
+start_server {tags {"bitops"}} {
+    test {BITFIELD signed SET and GET basics} {
+        r del bits
+        set results {}
+        lappend results [r bitfield bits set i8 0 -100]
+        lappend results [r bitfield bits set i8 0 101]
+        lappend results [r bitfield bits get i8 0]
+        set results
+    } {0 -100 101}
+
+    test {BITFIELD unsigned SET and GET basics} {
+        r del bits
+        set results {}
+        lappend results [r bitfield bits set u8 0 255]
+        lappend results [r bitfield bits set u8 0 100]
+        lappend results [r bitfield bits get u8 0]
+        set results
+    } {0 255 100}
+
+    test {BITFIELD #<idx> form} {
+        r del bits
+        set results {}
+        r bitfield bits set u8 #0 65
+        r bitfield bits set u8 #1 66
+        r bitfield bits set u8 #2 67
+        r get bits
+    } {ABC}
+
+    test {BITFIELD basic INCRBY form} {
+        r del bits
+        set results {}
+        r bitfield bits set u8 #0 10
+        lappend results [r bitfield bits incrby u8 #0 100]
+        lappend results [r bitfield bits incrby u8 #0 100]
+        set results
+    } {110 210}
+
+    test {BITFIELD chaining of multiple commands} {
+        r del bits
+        set results {}
+        r bitfield bits set u8 #0 10
+        lappend results [r bitfield bits incrby u8 #0 100 incrby u8 #0 100]
+        set results
+    } {{110 210}}
+
+    test {BITFIELD unsigned overflow wrap} {
+        r del bits
+        set results {}
+        r bitfield bits set u8 #0 100
+        lappend results [r bitfield bits overflow wrap incrby u8 #0 257]
+        lappend results [r bitfield bits get u8 #0]
+        lappend results [r bitfield bits overflow wrap incrby u8 #0 255]
+        lappend results [r bitfield bits get u8 #0]
+    } {101 101 100 100}
+
+    test {BITFIELD unsigned overflow sat} {
+        r del bits
+        set results {}
+        r bitfield bits set u8 #0 100
+        lappend results [r bitfield bits overflow sat incrby u8 #0 257]
+        lappend results [r bitfield bits get u8 #0]
+        lappend results [r bitfield bits overflow sat incrby u8 #0 -255]
+        lappend results [r bitfield bits get u8 #0]
+    } {255 255 0 0}
+
+    test {BITFIELD signed overflow wrap} {
+        r del bits
+        set results {}
+        r bitfield bits set i8 #0 100
+        lappend results [r bitfield bits overflow wrap incrby i8 #0 257]
+        lappend results [r bitfield bits get i8 #0]
+        lappend results [r bitfield bits overflow wrap incrby i8 #0 255]
+        lappend results [r bitfield bits get i8 #0]
+    } {101 101 100 100}
+
+    test {BITFIELD signed overflow sat} {
+        r del bits
+        set results {}
+        r bitfield bits set u8 #0 100
+        lappend results [r bitfield bits overflow sat incrby i8 #0 257]
+        lappend results [r bitfield bits get i8 #0]
+        lappend results [r bitfield bits overflow sat incrby i8 #0 -255]
+        lappend results [r bitfield bits get i8 #0]
+    } {127 127 -128 -128}
+
+    test {BITFIELD overflow detection fuzzing} {
+        for {set j 0} {$j < 1000} {incr j} {
+            set bits [expr {[randomInt 64]+1}]
+            set sign [randomInt 2]
+            set range [expr {2**$bits}]
+            if {$bits == 64} {set sign 1} ; # u64 is not supported by BITFIELD.
+            if {$sign} {
+                set min [expr {-($range/2)}]
+                set type "i$bits"
+            } else {
+                set min 0
+                set type "u$bits"
+            }
+            set max [expr {$min+$range-1}]
+
+            # Compare Tcl vs Redis
+            set range2 [expr {$range*2}]
+            set value [expr {($min*2)+[randomInt $range2]}]
+            set increment [expr {($min*2)+[randomInt $range2]}]
+            if {$value > 9223372036854775807} {
+                set value 9223372036854775807
+            }
+            if {$value < -9223372036854775808} {
+                set value -9223372036854775808
+            }
+            if {$increment > 9223372036854775807} {
+                set increment 9223372036854775807
+            }
+            if {$increment < -9223372036854775808} {
+                set increment -9223372036854775808
+            }
+
+            set overflow 0
+            if {$value > $max || $value < $min} {set overflow 1}
+            if {($value + $increment) > $max} {set overflow 1}
+            if {($value + $increment) < $min} {set overflow 1}
+
+            r del bits
+            set res1 [r bitfield bits overflow fail set $type 0 $value]
+            set res2 [r bitfield bits overflow fail incrby $type 0 $increment]
+
+            if {$overflow && [lindex $res1 0] ne {} &&
+                             [lindex $res2 0] ne {}} {
+                fail "OW not detected where needed: $type $value+$increment"
+            }
+            if {!$overflow && ([lindex $res1 0] eq {} ||
+                               [lindex $res2 0] eq {})} {
+                fail "OW detected where NOT needed: $type $value+$increment"
+            }
+        }
+    }
+
+    test {BITFIELD overflow wrap fuzzing} {
+        for {set j 0} {$j < 1000} {incr j} {
+            set bits [expr {[randomInt 64]+1}]
+            set sign [randomInt 2]
+            set range [expr {2**$bits}]
+            if {$bits == 64} {set sign 1} ; # u64 is not supported by BITFIELD.
+            if {$sign} {
+                set min [expr {-($range/2)}]
+                set type "i$bits"
+            } else {
+                set min 0
+                set type "u$bits"
+            }
+            set max [expr {$min+$range-1}]
+
+            # Compare Tcl vs Redis
+            set range2 [expr {$range*2}]
+            set value [expr {($min*2)+[randomInt $range2]}]
+            set increment [expr {($min*2)+[randomInt $range2]}]
+            if {$value > 9223372036854775807} {
+                set value 9223372036854775807
+            }
+            if {$value < -9223372036854775808} {
+                set value -9223372036854775808
+            }
+            if {$increment > 9223372036854775807} {
+                set increment 9223372036854775807
+            }
+            if {$increment < -9223372036854775808} {
+                set increment -9223372036854775808
+            }
+
+            r del bits
+            r bitfield bits overflow wrap set $type 0 $value
+            r bitfield bits overflow wrap incrby $type 0 $increment
+            set res [lindex [r bitfield bits get $type 0] 0]
+
+            set expected 0
+            if {$sign} {incr expected [expr {$max+1}]}
+            incr expected $value
+            incr expected $increment
+            set expected [expr {$expected % $range}]
+            if {$sign} {incr expected $min}
+
+            if {$res != $expected} {
+                fail "WRAP error: $type $value+$increment = $res, should be $expected"
+            }
+        }
+    }
+
+    test {BITFIELD regression for #3221} {
+        r set bits 1
+        r bitfield bits get u1 0
+    } {0}
+}

tests/unit/bitops.tcl

@@ -1,4 +1,4 @@
-# Compare Redis commadns against Tcl implementations of the same commands.
+# Compare Redis commands against Tcl implementations of the same commands.
 proc count_bits s {
     binary scan $s b* bits
     string length [regsub -all {0} $bits {}]
@@ -43,6 +43,16 @@ start_server {tags {"bitops"}} {
         r bitcount no-key
     } 0
 
+    test {BITCOUNT returns 0 with out of range indexes} {
+        r set str "xxxx"
+        r bitcount str 4 10
+    } 0
+
+    test {BITCOUNT returns 0 with negative indexes where start > end} {
+        r set str "xxxx"
+        r bitcount str -6 -7
+    } 0
+
     catch {unset num}
     foreach vec [list "" "\xaa" "\x00\x00\xff" "foobar" "123"] {
         incr num
@@ -88,7 +98,7 @@ start_server {tags {"bitops"}} {
     } {ERR*syntax*}
 
     test {BITCOUNT regression test for github issue #582} {
-        r del str
+        r del foo
         r setbit foo 0 1
         if {[catch {r bitcount foo 0 4294967296} e]} {
             assert_match {*ERR*out of range*} $e

tests/unit/dump.tcl

@@ -217,4 +217,95 @@ start_server {tags {"dump"}} {
             assert_match {IOERR*} $e
         }
     }
+
+    test {MIGRATE can migrate multiple keys at once} {
+        set first [srv 0 client]
+        r set key1 "v1"
+        r set key2 "v2"
+        r set key3 "v3"
+        start_server {tags {"repl"}} {
+            set second [srv 0 client]
+            set second_host [srv 0 host]
+            set second_port [srv 0 port]
+
+            assert {[$first exists key1] == 1}
+            assert {[$second exists key1] == 0}
+            set ret [r -1 migrate $second_host $second_port "" 9 5000 keys key1 key2 key3]
+            assert {$ret eq {OK}}
+            assert {[$first exists key1] == 0}
+            assert {[$first exists key2] == 0}
+            assert {[$first exists key3] == 0}
+            assert {[$second get key1] eq {v1}}
+            assert {[$second get key2] eq {v2}}
+            assert {[$second get key3] eq {v3}}
+        }
+    }
+
+    test {MIGRATE with multiple keys must have empty key arg} {
+        catch {r MIGRATE 127.0.0.1 6379 NotEmpty 9 5000 keys a b c} e
+        set e
+    } {*empty string*}
+
+    test {MIGRATE with mutliple keys migrate just existing ones} {
+        set first [srv 0 client]
+        r set key1 "v1"
+        r set key2 "v2"
+        r set key3 "v3"
+        start_server {tags {"repl"}} {
+            set second [srv 0 client]
+            set second_host [srv 0 host]
+            set second_port [srv 0 port]
+
+            set ret [r -1 migrate $second_host $second_port "" 9 5000 keys nokey-1 nokey-2 nokey-2]
+            assert {$ret eq {NOKEY}}
+
+            assert {[$first exists key1] == 1}
+            assert {[$second exists key1] == 0}
+            set ret [r -1 migrate $second_host $second_port "" 9 5000 keys nokey-1 key1 nokey-2 key2 nokey-3 key3]
+            assert {$ret eq {OK}}
+            assert {[$first exists key1] == 0}
+            assert {[$first exists key2] == 0}
+            assert {[$first exists key3] == 0}
+            assert {[$second get key1] eq {v1}}
+            assert {[$second get key2] eq {v2}}
+            assert {[$second get key3] eq {v3}}
+        }
+    }
+
+    test {MIGRATE with multiple keys: stress command rewriting} {
+        set first [srv 0 client]
+        r flushdb
+        r mset a 1 b 2 c 3 d 4 c 5 e 6 f 7 g 8 h 9 i 10 l 11 m 12 n 13 o 14 p 15 q 16
+        start_server {tags {"repl"}} {
+            set second [srv 0 client]
+            set second_host [srv 0 host]
+            set second_port [srv 0 port]
+
+            set ret [r -1 migrate $second_host $second_port "" 9 5000 keys a b c d e f g h i l m n o p q]
+
+            assert {[$first dbsize] == 0}
+            assert {[$second dbsize] == 15}
+        }
+    }
+
+    test {MIGRATE with multiple keys: delete just ack keys} {
+        set first [srv 0 client]
+        r flushdb
+        r mset a 1 b 2 c 3 d 4 c 5 e 6 f 7 g 8 h 9 i 10 l 11 m 12 n 13 o 14 p 15 q 16
+        start_server {tags {"repl"}} {
+            set second [srv 0 client]
+            set second_host [srv 0 host]
+            set second_port [srv 0 port]
+
+            $second mset c _ d _; # Two busy keys and no REPLACE used
+
+            catch {r -1 migrate $second_host $second_port "" 9 5000 keys a b c d e f g h i l m n o p q} e
+
+            assert {[$first dbsize] == 2}
+            assert {[$second dbsize] == 15}
+            assert {[$first exists c] == 1}
+            assert {[$first exists d] == 1}
+        }
+    }
+
 }

tests/unit/expire.tcl

@@ -198,4 +198,10 @@ start_server {tags {"expire"}} {
         r set foo b
         lsort [r keys *]
     } {a e foo s t}
+
+    test {EXPIRE with empty string as TTL should report an error} {
+        r set foo bar
+        catch {r expire foo ""} e
+        set e
+    } {*not an integer*}
 }

tests/unit/geo.tcl

@@ -1,4 +1,4 @@
-# Helper functins to simulate search-in-radius in the Tcl side in order to
+# Helper functions to simulate search-in-radius in the Tcl side in order to
 # verify the Redis implementation with a fuzzy test.
 proc geo_degrad deg {expr {$deg*atan(1)*8/360}}
 
@@ -23,6 +23,44 @@ proc geo_random_point {lonvar latvar} {
     set lat [expr {-70 + rand()*140}]
 }
 
+# Return elements non common to both the lists.
+# This code is from http://wiki.tcl.tk/15489
+proc compare_lists {List1 List2} {
+   set DiffList {}
+   foreach Item $List1 {
+      if {[lsearch -exact $List2 $Item] == -1} {
+         lappend DiffList $Item
+      }
+   }
+   foreach Item $List2 {
+      if {[lsearch -exact $List1 $Item] == -1} {
+         if {[lsearch -exact $DiffList $Item] == -1} {
+            lappend DiffList $Item
+         }
+      }
+   }
+   return $DiffList
+}
+
+# The following list represents sets of random seed, search position
+# and radius that caused bugs in the past. It is used by the randomized
+# test later as a starting point. When the regression vectors are scanned
+# the code reverts to using random data.
+#
+# The format is: seed km lon lat
+set regression_vectors {
+    {1412 156 149.29737817929004 15.95807862745508}
+    {441574 143 59.235461856813856 66.269555127373678}
+    {160645 187 -101.88575239939883 49.061997951502917}
+    {750269 154 -90.187939661642517 66.615930412251487}
+    {342880 145 163.03472387745728 64.012747720821181}
+    {729955 143 137.86663517256579 63.986745399416776}
+    {939895 151 59.149620271823181 65.204186651485145}
+    {1412 156 149.29737817929004 15.95807862745508}
+    {564862 149 84.062063109158544 -65.685403922426232}
+}
+set rv_idx 0
+
 start_server {tags {"geo"}} {
     test {GEOADD create} {
         r geoadd nyc -73.9454966 40.747533 "lic market"
@@ -128,17 +166,80 @@ start_server {tags {"geo"}} {
         assert {$m eq {}}
     }
 
+    test {GEORADIUS STORE option: syntax error} {
+        r del points
+        r geoadd points 13.361389 38.115556 "Palermo" \
+                        15.087269 37.502669 "Catania"
+        catch {r georadius points 13.361389 38.115556 50 km store} e
+        set e
+    } {*ERR*syntax*}
+
+    test {GEORANGE STORE option: incompatible options} {
+        r del points
+        r geoadd points 13.361389 38.115556 "Palermo" \
+                        15.087269 37.502669 "Catania"
+        catch {r georadius points 13.361389 38.115556 50 km store points2 withdist} e
+        assert_match {*ERR*} $e
+        catch {r georadius points 13.361389 38.115556 50 km store points2 withhash} e
+        assert_match {*ERR*} $e
+        catch {r georadius points 13.361389 38.115556 50 km store points2 withcoords} e
+        assert_match {*ERR*} $e
+    }
+
+    test {GEORANGE STORE option: plain usage} {
+        r del points
+        r geoadd points 13.361389 38.115556 "Palermo" \
+                        15.087269 37.502669 "Catania"
+        r georadius points 13.361389 38.115556 500 km store points2
+        assert_equal [r zrange points 0 -1] [r zrange points2 0 -1]
+    }
+
+    test {GEORANGE STOREDIST option: plain usage} {
+        r del points
+        r geoadd points 13.361389 38.115556 "Palermo" \
+                        15.087269 37.502669 "Catania"
+        r georadius points 13.361389 38.115556 500 km storedist points2
+        set res [r zrange points2 0 -1 withscores]
+        assert {[lindex $res 1] < 1}
+        assert {[lindex $res 3] > 166}
+        assert {[lindex $res 3] < 167}
+    }
+
+    test {GEORANGE STOREDIST option: COUNT ASC and DESC} {
+        r del points
+        r geoadd points 13.361389 38.115556 "Palermo" \
+                        15.087269 37.502669 "Catania"
+        r georadius points 13.361389 38.115556 500 km storedist points2 asc count 1
+        assert {[r zcard points2] == 1}
+        set res [r zrange points2 0 -1 withscores]
+        assert {[lindex $res 0] eq "Palermo"}
+
+        r georadius points 13.361389 38.115556 500 km storedist points2 desc count 1
+        assert {[r zcard points2] == 1}
+        set res [r zrange points2 0 -1 withscores]
+        assert {[lindex $res 0] eq "Catania"}
+    }
+
     test {GEOADD + GEORANGE randomized test} {
-        set attempt 10
+        set attempt 20
         while {[incr attempt -1]} {
+            set rv [lindex $regression_vectors $rv_idx]
+            incr rv_idx
+
             unset -nocomplain debuginfo
             set srand_seed [randomInt 1000000]
+            if {$rv ne {}} {set srand_seed [lindex $rv 0]}
             lappend debuginfo "srand_seed is $srand_seed"
             expr {srand($srand_seed)} ; # If you need a reproducible run
             r del mypoints
             set radius_km [expr {[randomInt 200]+10}]
+            if {$rv ne {}} {set radius_km [lindex $rv 1]}
             set radius_m [expr {$radius_km*1000}]
             geo_random_point search_lon search_lat
+            if {$rv ne {}} {
+                set search_lon [lindex $rv 2]
+                set search_lat [lindex $rv 3]
+            }
             lappend debuginfo "Search area: $search_lon,$search_lat $radius_km km"
             set tcl_result {}
             set argv {}
@@ -154,10 +255,40 @@ start_server {tags {"geo"}} {
             set res [lsort [r georadius mypoints $search_lon $search_lat $radius_km km]]
             set res2 [lsort $tcl_result]
             set test_result OK
+
             if {$res != $res2} {
+                set rounding_errors 0
+                set diff [compare_lists $res $res2]
+                foreach place $diff {
+                    set mydist [geo_distance $lon $lat $search_lon $search_lat]
+                    set mydist [expr $mydist/1000]
+                    if {($mydist / $radius_km) > 0.999} {incr rounding_errors}
+                }
+                # Make sure this is a real error and not a rounidng issue.
+                if {[llength $diff] == $rounding_errors} {
+                    set res $res2; # Error silenced
+                }
+            }
+
+            if {$res != $res2} {
+                set diff [compare_lists $res $res2]
+                puts "*** Possible problem in GEO radius query ***"
                 puts "Redis: $res"
                 puts "Tcl  : $res2"
+                puts "Diff : $diff"
                 puts [join $debuginfo "\n"]
+                foreach place $diff {
+                    if {[lsearch -exact $res2 $place] != -1} {
+                        set where "(only in Tcl)"
+                    } else {
+                        set where "(only in Redis)"
+                    }
+                    lassign [lindex [r geopos mypoints $place] 0] lon lat
+                    set mydist [geo_distance $lon $lat $search_lon $search_lat]
+                    set mydist [expr $mydist/1000]
+                    puts "$place -> [r geopos mypoints $place] $mydist $where"
+                    if {($mydist / $radius_km) > 0.999} {incr rounding_errors}
+                }
                 set test_result FAIL
             }
             unset -nocomplain debuginfo

tests/unit/introspection-2.tcl

@@ -0,0 +1,23 @@
+start_server {tags {"introspection"}} {
+    test {TTL and TYPYE do not alter the last access time of a key} {
+        r set foo bar
+        after 3000
+        r ttl foo
+        r type foo
+        assert {[r object idletime foo] >= 2}
+    }
+
+    test {TOUCH alters the last access time of a key} {
+        r set foo bar
+        after 3000
+        r touch foo
+        assert {[r object idletime foo] < 2}
+    }
+
+    test {TOUCH returns the number of existing keys specified} {
+        r flushdb
+        r set key1 1
+        r set key2 2
+        r touch key0 key1 key2 key3
+    } 2
+}