Defrag: fix test false positive.

Apparently 1.4 is too low compared to what you get in certain setups (including mine). I raised it to 1.55 that hopefully is still enough to test that the fragmentation went down from 1.7 but without incurring in issues, however the test setup may be still fragile so certain times this may lead to false positives again, it's hard to test for these things in a determinsitic way. Related to #3786.
Redis 4.0.0-RC3 (3.9.103).
2017-04-22 13:23:27 +02:00 · 2017-04-22 13:16:41 +02:00 · 2017-04-22 13:16:01 +02:00 · 2017-04-22 13:12:42 +02:00 · 2017-04-20 07:58:24 +02:00 · 2017-04-20 07:58:22 +02:00
101 changed files with 10438 additions and 1429 deletions
@@ -4,3 +4,4 @@
 /*.so
 /*.dylib
 /*.a
+/*.pc
@@ -1,6 +1,39 @@
 language: c
+sudo: false
 compiler:
  - gcc
  - clang

-script: make && make check
+os:
+  - linux
+  - osx
+
+before_script:
+    - if [ "$TRAVIS_OS_NAME" == "osx" ] ; then brew update; brew install redis; fi
+
+addons:
+  apt:
+    packages:
+    - libc6-dbg
+    - libc6-dev
+    - libc6:i386
+    - libc6-dev-i386
+    - libc6-dbg:i386
+    - gcc-multilib
+    - valgrind
+
+env:
+    - CFLAGS="-Werror"
+    - PRE="valgrind --track-origins=yes --leak-check=full"
+    - TARGET="32bit" TARGET_VARS="32bit-vars" CFLAGS="-Werror"
+    - TARGET="32bit" TARGET_VARS="32bit-vars" PRE="valgrind --track-origins=yes --leak-check=full"
+
+matrix:
+  exclude:
+    - os: osx
+      env: PRE="valgrind --track-origins=yes --leak-check=full"
+
+    - os: osx
+      env: TARGET="32bit" TARGET_VARS="32bit-vars" PRE="valgrind --track-origins=yes --leak-check=full"
+
+script: make $TARGET CFLAGS="$CFLAGS" && make check PRE="$PRE" && make $TARGET_VARS hiredis-example
@@ -1,3 +1,128 @@
+### 1.0.0 (unreleased)
+
+**Fixes**:
+
+* Catch a buffer overflow when formatting the error message
+* Import latest upstream sds. This breaks applications that are linked against the old hiredis v0.13
+* Fix warnings, when compiled with -Wshadow
+* Make hiredis compile in Cygwin on Windows, now CI-tested
+
+**BREAKING CHANGES**:
+
+* Change `redisReply.len` to `size_t`, as it denotes the the size of a string
+
+User code should compare this to `size_t` values as well.
+If it was used to compare to other values, casting might be necessary or can be removed, if casting was applied before.
+
+* Remove backwards compatibility macro's
+
+This removes the following old function aliases, use the new name now:
+
+| Old                         | New                    |
+| --------------------------- | ---------------------- |
+| redisReplyReaderCreate      | redisReaderCreate      |
+| redisReplyReaderCreate      | redisReaderCreate      |
+| redisReplyReaderFree        | redisReaderFree        |
+| redisReplyReaderFeed        | redisReaderFeed        |
+| redisReplyReaderGetReply    | redisReaderGetReply    |
+| redisReplyReaderSetPrivdata | redisReaderSetPrivdata |
+| redisReplyReaderGetObject   | redisReaderGetObject   |
+| redisReplyReaderGetError    | redisReaderGetError    |
+
+* The `DEBUG` variable in the Makefile was renamed to `DEBUG_FLAGS`
+
+Previously it broke some builds for people that had `DEBUG` set to some arbitrary value,
+due to debugging other software.
+By renaming we avoid unintentional name clashes.
+
+Simply rename `DEBUG` to `DEBUG_FLAGS` in your environment to make it working again.
+
+### 0.13.3 (2015-09-16)
+
+* Revert "Clear `REDIS_CONNECTED` flag when connection is closed".
+* Make tests pass on FreeBSD (Thanks, Giacomo Olgeni)
+
+
+If the `REDIS_CONNECTED` flag is cleared,
+the async onDisconnect callback function will never be called.
+This causes problems as the disconnect is never reported back to the user.
+
+### 0.13.2 (2015-08-25)
+
+* Prevent crash on pending replies in async code (Thanks, @switch-st)
+* Clear `REDIS_CONNECTED` flag when connection is closed (Thanks, Jerry Jacobs)
+* Add MacOS X addapter (Thanks, @dizzus)
+* Add Qt adapter (Thanks, Pietro Cerutti)
+* Add Ivykis adapter (Thanks, Gergely Nagy)
+
+All adapters are provided as is and are only tested where possible.
+
+### 0.13.1 (2015-05-03)
+
+This is a bug fix release.
+The new `reconnect` method introduced new struct members, which clashed with pre-defined names in pre-C99 code.
+Another commit forced C99 compilation just to make it work, but of course this is not desirable for outside projects.
+Other non-C99 code can now use hiredis as usual again.
+Sorry for the inconvenience.
+
+* Fix memory leak in async reply handling (Salvatore Sanfilippo)
+* Rename struct member to avoid name clash with pre-c99 code (Alex Balashov, ncopa)
+
+### 0.13.0 (2015-04-16)
+
+This release adds a minimal Windows compatibility layer.
+The parser, standalone since v0.12.0, can now be compiled on Windows
+(and thus used in other client libraries as well)
+
+* Windows compatibility layer for parser code (tzickel)
+* Properly escape data printed to PKGCONF file (Dan Skorupski)
+* Fix tests when assert() undefined (Keith Bennett, Matt Stancliff)
+* Implement a reconnect method for the client context, this changes the structure of `redisContext` (Aaron Bedra)
+
+### 0.12.1 (2015-01-26)
+
+* Fix `make install`: DESTDIR support, install all required files, install PKGCONF in proper location
+* Fix `make test` as 32 bit build on 64 bit platform
+
+### 0.12.0 (2015-01-22)
+
+* Add optional KeepAlive support
+
+* Try again on EINTR errors
+
+* Add libuv adapter
+
+* Add IPv6 support
+
+* Remove possiblity of multiple close on same fd
+
+* Add ability to bind source address on connect
+
+* Add redisConnectFd() and redisFreeKeepFd()
+
+* Fix getaddrinfo() memory leak
+
+* Free string if it is unused (fixes memory leak)
+
+* Improve redisAppendCommandArgv performance 2.5x
+
+* Add support for SO_REUSEADDR
+
+* Fix redisvFormatCommand format parsing
+
+* Add GLib 2.0 adapter
+
+* Refactor reading code into read.c
+
+* Fix errno error buffers to not clobber errors
+
+* Generate pkgconf during build
+
+* Silence _BSD_SOURCE warnings
+
+* Improve digit counting for multibulk creation
+
+
 ### 0.11.0

 * Increase the maximum multi-bulk reply depth to 7.
@@ -3,13 +3,25 @@
 # Copyright (C) 2010-2011 Pieter Noordhuis <pcnoordhuis at gmail dot com>
 # This file is released under the BSD license, see the COPYING file

-OBJ=net.o hiredis.o sds.o async.o
-EXAMPLES=hiredis-example hiredis-example-libevent hiredis-example-libev
+OBJ=net.o hiredis.o sds.o async.o read.o
+EXAMPLES=hiredis-example hiredis-example-libevent hiredis-example-libev hiredis-example-glib
 TESTS=hiredis-test
 LIBNAME=libhiredis
+PKGCONFNAME=hiredis.pc

-HIREDIS_MAJOR=0
-HIREDIS_MINOR=11
+HIREDIS_MAJOR=$(shell grep HIREDIS_MAJOR hiredis.h | awk '{print $$3}')
+HIREDIS_MINOR=$(shell grep HIREDIS_MINOR hiredis.h | awk '{print $$3}')
+HIREDIS_PATCH=$(shell grep HIREDIS_PATCH hiredis.h | awk '{print $$3}')
+HIREDIS_SONAME=$(shell grep HIREDIS_SONAME hiredis.h | awk '{print $$3}')
+
+# Installation related variables and target
+PREFIX?=/usr/local
+INCLUDE_PATH?=include/hiredis
+LIBRARY_PATH?=lib
+PKGCONF_PATH?=pkgconfig
+INSTALL_INCLUDE_PATH= $(DESTDIR)$(PREFIX)/$(INCLUDE_PATH)
+INSTALL_LIBRARY_PATH= $(DESTDIR)$(PREFIX)/$(LIBRARY_PATH)
+INSTALL_PKGCONF_PATH= $(INSTALL_LIBRARY_PATH)/$(PKGCONF_PATH)

 # redis-server configuration used for testing
 REDIS_PORT=56379
@@ -25,15 +37,16 @@ export REDIS_TEST_CONFIG

 # Fallback to gcc when $CC is not in $PATH.
 CC:=$(shell sh -c 'type $(CC) >/dev/null 2>/dev/null && echo $(CC) || echo gcc')
+CXX:=$(shell sh -c 'type $(CXX) >/dev/null 2>/dev/null && echo $(CXX) || echo g++')
 OPTIMIZATION?=-O3
 WARNINGS=-Wall -W -Wstrict-prototypes -Wwrite-strings
-DEBUG?= -g -ggdb
-REAL_CFLAGS=$(OPTIMIZATION) -fPIC $(CFLAGS) $(WARNINGS) $(DEBUG) $(ARCH)
+DEBUG_FLAGS?= -g -ggdb
+REAL_CFLAGS=$(OPTIMIZATION) -fPIC $(CFLAGS) $(WARNINGS) $(DEBUG_FLAGS) $(ARCH)
 REAL_LDFLAGS=$(LDFLAGS) $(ARCH)

 DYLIBSUFFIX=so
 STLIBSUFFIX=a
-DYLIB_MINOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_MAJOR).$(HIREDIS_MINOR)
+DYLIB_MINOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_SONAME)
 DYLIB_MAJOR_NAME=$(LIBNAME).$(DYLIBSUFFIX).$(HIREDIS_MAJOR)
 DYLIBNAME=$(LIBNAME).$(DYLIBSUFFIX)
 DYLIB_MAKE_CMD=$(CC) -shared -Wl,-soname,$(DYLIB_MINOR_NAME) -o $(DYLIBNAME) $(LDFLAGS)
@@ -49,19 +62,20 @@ ifeq ($(uname_S),SunOS)
 endif
 ifeq ($(uname_S),Darwin)
  DYLIBSUFFIX=dylib
-  DYLIB_MINOR_NAME=$(LIBNAME).$(HIREDIS_MAJOR).$(HIREDIS_MINOR).$(DYLIBSUFFIX)
-  DYLIB_MAJOR_NAME=$(LIBNAME).$(HIREDIS_MAJOR).$(DYLIBSUFFIX)
+  DYLIB_MINOR_NAME=$(LIBNAME).$(HIREDIS_SONAME).$(DYLIBSUFFIX)
  DYLIB_MAKE_CMD=$(CC) -shared -Wl,-install_name,$(DYLIB_MINOR_NAME) -o $(DYLIBNAME) $(LDFLAGS)
 endif

-all: $(DYLIBNAME)
+all: $(DYLIBNAME) $(STLIBNAME) hiredis-test $(PKGCONFNAME)

 # Deps (use make dep to generate this)
-net.o: net.c fmacros.h net.h hiredis.h
-async.o: async.c async.h hiredis.h sds.h dict.c dict.h
-hiredis.o: hiredis.c fmacros.h hiredis.h net.h sds.h
+async.o: async.c fmacros.h async.h hiredis.h read.h sds.h net.h dict.c dict.h
+dict.o: dict.c fmacros.h dict.h
+hiredis.o: hiredis.c fmacros.h hiredis.h read.h sds.h net.h
+net.o: net.c fmacros.h net.h hiredis.h read.h sds.h
+read.o: read.c fmacros.h read.h sds.h
 sds.o: sds.c sds.h
-test.o: test.c hiredis.h
+test.o: test.c fmacros.h hiredis.h read.h sds.h

 $(DYLIBNAME): $(OBJ)
 	$(DYLIB_MAKE_CMD) $(OBJ)
@@ -79,6 +93,15 @@ hiredis-example-libevent: examples/example-libevent.c adapters/libevent.h $(STLI
 hiredis-example-libev: examples/example-libev.c adapters/libev.h $(STLIBNAME)
 	$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -lev $(STLIBNAME)

+hiredis-example-glib: examples/example-glib.c adapters/glib.h $(STLIBNAME)
+	$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) $(shell pkg-config --cflags --libs glib-2.0) -I. $< $(STLIBNAME)
+
+hiredis-example-ivykis: examples/example-ivykis.c adapters/ivykis.h $(STLIBNAME)
+	$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -livykis $(STLIBNAME)
+
+hiredis-example-macosx: examples/example-macosx.c adapters/macosx.h $(STLIBNAME)
+	$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. $< -framework CoreFoundation $(STLIBNAME)
+
 ifndef AE_DIR
 hiredis-example-ae:
 	@echo "Please specify AE_DIR (e.g. <redis repository>/src)"
@@ -94,7 +117,20 @@ hiredis-example-libuv:
 	@false
 else
 hiredis-example-libuv: examples/example-libuv.c adapters/libuv.h $(STLIBNAME)
-	$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. -I$(LIBUV_DIR)/include $< $(LIBUV_DIR)/.libs/libuv.a -lpthread $(STLIBNAME)
+	$(CC) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. -I$(LIBUV_DIR)/include $< $(LIBUV_DIR)/.libs/libuv.a -lpthread -lrt $(STLIBNAME)
+endif
+
+ifeq ($(and $(QT_MOC),$(QT_INCLUDE_DIR),$(QT_LIBRARY_DIR)),)
+hiredis-example-qt:
+	@echo "Please specify QT_MOC, QT_INCLUDE_DIR AND QT_LIBRARY_DIR"
+	@false
+else
+hiredis-example-qt: examples/example-qt.cpp adapters/qt.h $(STLIBNAME)
+	$(QT_MOC) adapters/qt.h -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore | \
+	    $(CXX) -x c++ -o qt-adapter-moc.o -c - $(REAL_CFLAGS) -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore
+	$(QT_MOC) examples/example-qt.h -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore | \
+	    $(CXX) -x c++ -o qt-example-moc.o -c - $(REAL_CFLAGS) -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore
+	$(CXX) -o examples/$@ $(REAL_CFLAGS) $(REAL_LDFLAGS) -I. -I$(QT_INCLUDE_DIR) -I$(QT_INCLUDE_DIR)/QtCore -L$(QT_LIBRARY_DIR) qt-adapter-moc.o qt-example-moc.o $< -pthread $(STLIBNAME) -lQtCore
 endif

 hiredis-example: examples/example.c $(STLIBNAME)
@@ -103,14 +139,16 @@ hiredis-example: examples/example.c $(STLIBNAME)
 examples: $(EXAMPLES)

 hiredis-test: test.o $(STLIBNAME)
-	$(CC) -o $@ $(REAL_LDFLAGS) $< $(STLIBNAME)
+
+hiredis-%: %.o $(STLIBNAME)
+	$(CC) $(REAL_CFLAGS) -o $@ $(REAL_LDFLAGS) $< $(STLIBNAME)

 test: hiredis-test
 	./hiredis-test

 check: hiredis-test
 	@echo "$$REDIS_TEST_CONFIG" | $(REDIS_SERVER) -
-	./hiredis-test -h 127.0.0.1 -p $(REDIS_PORT) -s /tmp/hiredis-test-redis.sock || \
+	$(PRE) ./hiredis-test -h 127.0.0.1 -p $(REDIS_PORT) -s /tmp/hiredis-test-redis.sock || \
 			( kill `cat /tmp/hiredis-test-redis.pid` && false )
 	kill `cat /tmp/hiredis-test-redis.pid`

@@ -118,29 +156,38 @@ check: hiredis-test
 	$(CC) -std=c99 -pedantic -c $(REAL_CFLAGS) $<

 clean:
-	rm -rf $(DYLIBNAME) $(STLIBNAME) $(TESTS) examples/hiredis-example* *.o *.gcda *.gcno *.gcov
+	rm -rf $(DYLIBNAME) $(STLIBNAME) $(TESTS) $(PKGCONFNAME) examples/hiredis-example* *.o *.gcda *.gcno *.gcov

 dep:
 	$(CC) -MM *.c

-# Installation related variables and target
-PREFIX?=/usr/local
-INSTALL_INCLUDE_PATH= $(PREFIX)/include/hiredis
-INSTALL_LIBRARY_PATH= $(PREFIX)/lib
-
 ifeq ($(uname_S),SunOS)
  INSTALL?= cp -r
 endif

 INSTALL?= cp -a

-install: $(DYLIBNAME) $(STLIBNAME)
+$(PKGCONFNAME): hiredis.h
+	@echo "Generating $@ for pkgconfig..."
+	@echo prefix=$(PREFIX) > $@
+	@echo exec_prefix=\$${prefix} >> $@
+	@echo libdir=$(PREFIX)/$(LIBRARY_PATH) >> $@
+	@echo includedir=$(PREFIX)/$(INCLUDE_PATH) >> $@
+	@echo >> $@
+	@echo Name: hiredis >> $@
+	@echo Description: Minimalistic C client library for Redis. >> $@
+	@echo Version: $(HIREDIS_MAJOR).$(HIREDIS_MINOR).$(HIREDIS_PATCH) >> $@
+	@echo Libs: -L\$${libdir} -lhiredis >> $@
+	@echo Cflags: -I\$${includedir} -D_FILE_OFFSET_BITS=64 >> $@
+
+install: $(DYLIBNAME) $(STLIBNAME) $(PKGCONFNAME)
 	mkdir -p $(INSTALL_INCLUDE_PATH) $(INSTALL_LIBRARY_PATH)
-	$(INSTALL) hiredis.h async.h adapters $(INSTALL_INCLUDE_PATH)
+	$(INSTALL) hiredis.h async.h read.h sds.h adapters $(INSTALL_INCLUDE_PATH)
 	$(INSTALL) $(DYLIBNAME) $(INSTALL_LIBRARY_PATH)/$(DYLIB_MINOR_NAME)
-	cd $(INSTALL_LIBRARY_PATH) && ln -sf $(DYLIB_MINOR_NAME) $(DYLIB_MAJOR_NAME)
-	cd $(INSTALL_LIBRARY_PATH) && ln -sf $(DYLIB_MAJOR_NAME) $(DYLIBNAME)
+	cd $(INSTALL_LIBRARY_PATH) && ln -sf $(DYLIB_MINOR_NAME) $(DYLIBNAME)
 	$(INSTALL) $(STLIBNAME) $(INSTALL_LIBRARY_PATH)
+	mkdir -p $(INSTALL_PKGCONF_PATH)
+	$(INSTALL) $(PKGCONFNAME) $(INSTALL_PKGCONF_PATH)

 32bit:
 	@echo ""
@@ -148,6 +195,10 @@ install: $(DYLIBNAME) $(STLIBNAME)
 	@echo ""
 	$(MAKE) CFLAGS="-m32" LDFLAGS="-m32"

+32bit-vars:
+	$(eval CFLAGS=-m32)
+	$(eval LDFLAGS=-m32)
+
 gprof:
 	$(MAKE) CFLAGS="-pg" LDFLAGS="-pg"

@@ -163,4 +214,4 @@ coverage: gcov
 noopt:
 	$(MAKE) OPTIMIZATION=""

-.PHONY: all test check clean dep install 32bit gprof gcov noopt
+.PHONY: all test check clean dep install 32bit 32bit-vars gprof gcov noopt
@@ -1,11 +1,13 @@
 [![Build Status](https://travis-ci.org/redis/hiredis.png)](https://travis-ci.org/redis/hiredis)

+**This Readme reflects the latest changed in the master branch. See [v0.13.3](https://github.com/redis/hiredis/tree/v0.13.3) for the Readme and documentation for the latest release.**
+
 # HIREDIS

 Hiredis is a minimalistic C client library for the [Redis](http://redis.io/) database.

 It is minimalistic because it just adds minimal support for the protocol, but
-at the same time it uses an high level printf-alike API in order to make it
+at the same time it uses a high level printf-alike API in order to make it
 much higher level than otherwise suggested by its minimal code base and the
 lack of explicit bindings for every Redis command.

@@ -20,7 +22,15 @@ Redis version >= 1.2.0.
 The library comes with multiple APIs. There is the
 *synchronous API*, the *asynchronous API* and the *reply parsing API*.

-## UPGRADING
+## Upgrading to `1.0.0`
+
+Version 1.0.0 marks a stable release of hiredis.
+It includes some minor breaking changes, mostly to make the exposed API more uniform and self-explanatory.
+It also bundles the updated `sds` library, to sync up with upstream and Redis.
+For most applications a recompile against the new hiredis should be enough.
+For code changes see the [Changelog](CHANGELOG.md).
+
+## Upgrading from `<0.9.0`

 Version 0.9.0 is a major overhaul of hiredis in every aspect. However, upgrading existing
 code using hiredis should not be a big pain. The key thing to keep in mind when
@@ -31,51 +41,62 @@ the stateless 0.0.1 that only has a file descriptor to work with.

 To consume the synchronous API, there are only a few function calls that need to be introduced:

-    redisContext *redisConnect(const char *ip, int port);
-    void *redisCommand(redisContext *c, const char *format, ...);
-    void freeReplyObject(void *reply);
+```c
+redisContext *redisConnect(const char *ip, int port);
+void *redisCommand(redisContext *c, const char *format, ...);
+void freeReplyObject(void *reply);
+```

 ### Connecting

 The function `redisConnect` is used to create a so-called `redisContext`. The
 context is where Hiredis holds state for a connection. The `redisContext`
-struct has an integer `err` field that is non-zero when an the connection is in
+struct has an integer `err` field that is non-zero when the connection is in
 an error state. The field `errstr` will contain a string with a description of
 the error. More information on errors can be found in the **Errors** section.
 After trying to connect to Redis using `redisConnect` you should
 check the `err` field to see if establishing the connection was successful:
-
-    redisContext *c = redisConnect("127.0.0.1", 6379);
-    if (c != NULL && c->err) {
+```c
+redisContext *c = redisConnect("127.0.0.1", 6379);
+if (c == NULL || c->err) {
+    if (c) {
        printf("Error: %s\n", c->errstr);
        // handle error
+    } else {
+        printf("Can't allocate redis context\n");
    }
+}
+```
+
+*Note: A `redisContext` is not thread-safe.*

 ### Sending commands

 There are several ways to issue commands to Redis. The first that will be introduced is
 `redisCommand`. This function takes a format similar to printf. In the simplest form,
 it is used like this:
-
-    reply = redisCommand(context, "SET foo bar");
+```c
+reply = redisCommand(context, "SET foo bar");
+```

 The specifier `%s` interpolates a string in the command, and uses `strlen` to
 determine the length of the string:
-
-    reply = redisCommand(context, "SET foo %s", value);
-
+```c
+reply = redisCommand(context, "SET foo %s", value);
+```
 When you need to pass binary safe strings in a command, the `%b` specifier can be
 used. Together with a pointer to the string, it requires a `size_t` length argument
 of the string:
-
-    reply = redisCommand(context, "SET foo %b", value, (size_t) valuelen);
-
+```c
+reply = redisCommand(context, "SET foo %b", value, (size_t) valuelen);
+```
 Internally, Hiredis splits the command in different arguments and will
 convert it to the protocol used to communicate with Redis.
 One or more spaces separates arguments, so you can use the specifiers
 anywhere in an argument:
-
-    reply = redisCommand(context, "SET key:%s %s", myid, value);
+```c
+reply = redisCommand(context, "SET key:%s %s", myid, value);
+```

 ### Using replies

@@ -114,11 +135,11 @@ was received:
      Redis may reply with nested arrays but this is fully supported.

 Replies should be freed using the `freeReplyObject()` function.
-Note that this function will take care of freeing sub-replies objects
+Note that this function will take care of freeing sub-reply objects
 contained in arrays and nested arrays, so there is no need for the user to
 free the sub replies (it is actually harmful and will corrupt the memory).

-**Important:** the current version of hiredis (0.10.0) free's replies when the
+**Important:** the current version of hiredis (0.10.0) frees replies when the
 asynchronous API is used. This means you should not call `freeReplyObject` when
 you use this API. The reply is cleaned up by hiredis _after_ the callback
 returns. This behavior will probably change in future releases, so make sure to
@@ -127,19 +148,19 @@ keep an eye on the changelog when upgrading (see issue #39).
 ### Cleaning up

 To disconnect and free the context the following function can be used:
-
-    void redisFree(redisContext *c);
-
-This function immediately closes the socket and then free's the allocations done in
+```c
+void redisFree(redisContext *c);
+```
+This function immediately closes the socket and then frees the allocations done in
 creating the context.

 ### Sending commands (cont'd)

 Together with `redisCommand`, the function `redisCommandArgv` can be used to issue commands.
 It has the following prototype:
-
-    void *redisCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);
-
+```c
+void *redisCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);
+```
 It takes the number of arguments `argc`, an array of strings `argv` and the lengths of the
 arguments `argvlen`. For convenience, `argvlen` may be set to `NULL` and the function will
 use `strlen(3)` on every argument to determine its length. Obviously, when any of the arguments
@@ -169,10 +190,10 @@ The function `redisGetReply` is exported as part of the Hiredis API and can be u
 is expected on the socket. To pipeline commands, the only things that needs to be done is
 filling up the output buffer. For this cause, two commands can be used that are identical
 to the `redisCommand` family, apart from not returning a reply:
-
-    void redisAppendCommand(redisContext *c, const char *format, ...);
-    void redisAppendCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);
-
+```c
+void redisAppendCommand(redisContext *c, const char *format, ...);
+void redisAppendCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen);
+```
 After calling either function one or more times, `redisGetReply` can be used to receive the
 subsequent replies. The return value for this function is either `REDIS_OK` or `REDIS_ERR`, where
 the latter means an error occurred while reading a reply. Just as with the other commands,
@@ -180,24 +201,24 @@ the `err` field in the context can be used to find out what the cause of this er

 The following examples shows a simple pipeline (resulting in only a single call to `write(2)` and
 a single call to `read(2)`):
-
-    redisReply *reply;
-    redisAppendCommand(context,"SET foo bar");
-    redisAppendCommand(context,"GET foo");
-    redisGetReply(context,&reply); // reply for SET
-    freeReplyObject(reply);
-    redisGetReply(context,&reply); // reply for GET
-    freeReplyObject(reply);
-
+```c
+redisReply *reply;
+redisAppendCommand(context,"SET foo bar");
+redisAppendCommand(context,"GET foo");
+redisGetReply(context,&reply); // reply for SET
+freeReplyObject(reply);
+redisGetReply(context,&reply); // reply for GET
+freeReplyObject(reply);
+```
 This API can also be used to implement a blocking subscriber:
-
-    reply = redisCommand(context,"SUBSCRIBE foo");
+```c
+reply = redisCommand(context,"SUBSCRIBE foo");
+freeReplyObject(reply);
+while(redisGetReply(context,&reply) == REDIS_OK) {
+    // consume message
    freeReplyObject(reply);
-    while(redisGetReply(context,&reply) == REDIS_OK) {
-        // consume message
-        freeReplyObject(reply);
-    }
-
+}
+```
 ### Errors

 When a function call is not successful, depending on the function either `NULL` or `REDIS_ERR` is
@@ -237,58 +258,62 @@ should be checked after creation to see if there were errors creating the connec
 Because the connection that will be created is non-blocking, the kernel is not able to
 instantly return if the specified host and port is able to accept a connection.

-    redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
-    if (c->err) {
-        printf("Error: %s\n", c->errstr);
-        // handle error
-    }
+*Note: A `redisAsyncContext` is not thread-safe.*
+
+```c
+redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
+if (c->err) {
+    printf("Error: %s\n", c->errstr);
+    // handle error
+}
+```

 The asynchronous context can hold a disconnect callback function that is called when the
 connection is disconnected (either because of an error or per user request). This function should
 have the following prototype:
-
-    void(const redisAsyncContext *c, int status);
-
+```c
+void(const redisAsyncContext *c, int status);
+```
 On a disconnect, the `status` argument is set to `REDIS_OK` when disconnection was initiated by the
 user, or `REDIS_ERR` when the disconnection was caused by an error. When it is `REDIS_ERR`, the `err`
 field in the context can be accessed to find out the cause of the error.

-The context object is always free'd after the disconnect callback fired. When a reconnect is needed,
+The context object is always freed after the disconnect callback fired. When a reconnect is needed,
 the disconnect callback is a good point to do so.

 Setting the disconnect callback can only be done once per context. For subsequent calls it will
 return `REDIS_ERR`. The function to set the disconnect callback has the following prototype:
-
-    int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);
-
+```c
+int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);
+```
 ### Sending commands and their callbacks

 In an asynchronous context, commands are automatically pipelined due to the nature of an event loop.
 Therefore, unlike the synchronous API, there is only a single way to send commands.
 Because commands are sent to Redis asynchronously, issuing a command requires a callback function
 that is called when the reply is received. Reply callbacks should have the following prototype:
-
-    void(redisAsyncContext *c, void *reply, void *privdata);
-
+```c
+void(redisAsyncContext *c, void *reply, void *privdata);
+```
 The `privdata` argument can be used to curry arbitrary data to the callback from the point where
 the command is initially queued for execution.

 The functions that can be used to issue commands in an asynchronous context are:
-
-    int redisAsyncCommand(
-      redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
-      const char *format, ...);
-    int redisAsyncCommandArgv(
-      redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
-      int argc, const char **argv, const size_t *argvlen);
-
+```c
+int redisAsyncCommand(
+  redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
+  const char *format, ...);
+int redisAsyncCommandArgv(
+  redisAsyncContext *ac, redisCallbackFn *fn, void *privdata,
+  int argc, const char **argv, const size_t *argvlen);
+```
 Both functions work like their blocking counterparts. The return value is `REDIS_OK` when the command
 was successfully added to the output buffer and `REDIS_ERR` otherwise. Example: when the connection
 is being disconnected per user-request, no new commands may be added to the output buffer and `REDIS_ERR` is
 returned on calls to the `redisAsyncCommand` family.

-If the reply for a command with a `NULL` callback is read, it is immediately free'd. When the callback
-for a command is non-`NULL`, the memory is free'd immediately following the callback: the reply is only
+If the reply for a command with a `NULL` callback is read, it is immediately freed. When the callback
+for a command is non-`NULL`, the memory is freed immediately following the callback: the reply is only
 valid for the duration of the callback.

 All pending callbacks are called with a `NULL` reply when the context encountered an error.
@@ -296,14 +321,14 @@ All pending callbacks are called with a `NULL` reply when the context encountere
 ### Disconnecting

 An asynchronous connection can be terminated using:
-
-    void redisAsyncDisconnect(redisAsyncContext *ac);
-
+```c
+void redisAsyncDisconnect(redisAsyncContext *ac);
+```
 When this function is called, the connection is **not** immediately terminated. Instead, new
 commands are no longer accepted and the connection is only terminated when all pending commands
 have been written to the socket, their respective replies have been read and their respective
 callbacks have been executed. After this, the disconnection callback is executed with the
-`REDIS_OK` status and the context object is free'd.
+`REDIS_OK` status and the context object is freed.

 ### Hooking it up to event library *X*

@@ -316,12 +341,12 @@ Hiredis comes with a reply parsing API that makes it easy for writing higher
 level language bindings.

 The reply parsing API consists of the following functions:
-
-    redisReader *redisReaderCreate(void);
-    void redisReaderFree(redisReader *reader);
-    int redisReaderFeed(redisReader *reader, const char *buf, size_t len);
-    int redisReaderGetReply(redisReader *reader, void **reply);
-
+```c
+redisReader *redisReaderCreate(void);
+void redisReaderFree(redisReader *reader);
+int redisReaderFeed(redisReader *reader, const char *buf, size_t len);
+int redisReaderGetReply(redisReader *reader, void **reply);
+```
 The same set of functions are used internally by hiredis when creating a
 normal Redis context, the above API just exposes it to the user for a direct
 usage.
@@ -361,7 +386,7 @@ Both when using the Reader API directly or when using it indirectly via a
 normal Redis context, the redisReader structure uses a buffer in order to
 accumulate data from the server.
 Usually this buffer is destroyed when it is empty and is larger than 16
-kb in order to avoid wasting memory in unused buffers
+KiB in order to avoid wasting memory in unused buffers

 However when working with very big payloads destroying the buffer may slow
 down performances considerably, so it is possible to modify the max size of
@@ -371,9 +396,9 @@ value for an idle buffer, so the buffer will never get freed.

 For instance if you have a normal Redis context you can set the maximum idle
 buffer to zero (unlimited) just with:
-
-    context->reader->maxbuf = 0;
-
+```c
+context->reader->maxbuf = 0;
+```
 This should be done only in order to maximize performances when working with
 large payloads. The context should be set back to `REDIS_READER_MAX_BUF` again
 as soon as possible in order to prevent allocation of useless memory.
@@ -381,4 +406,6 @@ as soon as possible in order to prevent allocation of useless memory.
 ## AUTHORS

 Hiredis was written by Salvatore Sanfilippo (antirez at gmail) and
-Pieter Noordhuis (pcnoordhuis at gmail) and is released under the BSD license.
+Pieter Noordhuis (pcnoordhuis at gmail) and is released under the BSD license.  
+Hiredis is currently maintained by Matt Stancliff (matt at genges dot com) and
+Jan-Erik Rediger (janerik at fnordig dot com)
@@ -0,0 +1,153 @@
+#ifndef __HIREDIS_GLIB_H__
+#define __HIREDIS_GLIB_H__
+
+#include <glib.h>
+
+#include "../hiredis.h"
+#include "../async.h"
+
+typedef struct
+{
+    GSource source;
+    redisAsyncContext *ac;
+    GPollFD poll_fd;
+} RedisSource;
+
+static void
+redis_source_add_read (gpointer data)
+{
+    RedisSource *source = (RedisSource *)data;
+    g_return_if_fail(source);
+    source->poll_fd.events |= G_IO_IN;
+    g_main_context_wakeup(g_source_get_context((GSource *)data));
+}
+
+static void
+redis_source_del_read (gpointer data)
+{
+    RedisSource *source = (RedisSource *)data;
+    g_return_if_fail(source);
+    source->poll_fd.events &= ~G_IO_IN;
+    g_main_context_wakeup(g_source_get_context((GSource *)data));
+}
+
+static void
+redis_source_add_write (gpointer data)
+{
+    RedisSource *source = (RedisSource *)data;
+    g_return_if_fail(source);
+    source->poll_fd.events |= G_IO_OUT;
+    g_main_context_wakeup(g_source_get_context((GSource *)data));
+}
+
+static void
+redis_source_del_write (gpointer data)
+{
+    RedisSource *source = (RedisSource *)data;
+    g_return_if_fail(source);
+    source->poll_fd.events &= ~G_IO_OUT;
+    g_main_context_wakeup(g_source_get_context((GSource *)data));
+}
+
+static void
+redis_source_cleanup (gpointer data)
+{
+    RedisSource *source = (RedisSource *)data;
+
+    g_return_if_fail(source);
+
+    redis_source_del_read(source);
+    redis_source_del_write(source);
+    /*
+     * It is not our responsibility to remove ourself from the
+     * current main loop. However, we will remove the GPollFD.
+     */
+    if (source->poll_fd.fd >= 0) {
+        g_source_remove_poll((GSource *)data, &source->poll_fd);
+        source->poll_fd.fd = -1;
+    }
+}
+
+static gboolean
+redis_source_prepare (GSource *source,
+                      gint    *timeout_)
+{
+    RedisSource *redis = (RedisSource *)source;
+    *timeout_ = -1;
+    return !!(redis->poll_fd.events & redis->poll_fd.revents);
+}
+
+static gboolean
+redis_source_check (GSource *source)
+{
+    RedisSource *redis = (RedisSource *)source;
+    return !!(redis->poll_fd.events & redis->poll_fd.revents);
+}
+
+static gboolean
+redis_source_dispatch (GSource      *source,
+                       GSourceFunc   callback,
+                       gpointer      user_data)
+{
+    RedisSource *redis = (RedisSource *)source;
+
+    if ((redis->poll_fd.revents & G_IO_OUT)) {
+        redisAsyncHandleWrite(redis->ac);
+        redis->poll_fd.revents &= ~G_IO_OUT;
+    }
+
+    if ((redis->poll_fd.revents & G_IO_IN)) {
+        redisAsyncHandleRead(redis->ac);
+        redis->poll_fd.revents &= ~G_IO_IN;
+    }
+
+    if (callback) {
+        return callback(user_data);
+    }
+
+    return TRUE;
+}
+
+static void
+redis_source_finalize (GSource *source)
+{
+    RedisSource *redis = (RedisSource *)source;
+
+    if (redis->poll_fd.fd >= 0) {
+        g_source_remove_poll(source, &redis->poll_fd);
+        redis->poll_fd.fd = -1;
+    }
+}
+
+static GSource *
+redis_source_new (redisAsyncContext *ac)
+{
+    static GSourceFuncs source_funcs = {
+        .prepare  = redis_source_prepare,
+        .check     = redis_source_check,
+        .dispatch = redis_source_dispatch,
+        .finalize = redis_source_finalize,
+    };
+    redisContext *c = &ac->c;
+    RedisSource *source;
+
+    g_return_val_if_fail(ac != NULL, NULL);
+
+    source = (RedisSource *)g_source_new(&source_funcs, sizeof *source);
+    source->ac = ac;
+    source->poll_fd.fd = c->fd;
+    source->poll_fd.events = 0;
+    source->poll_fd.revents = 0;
+    g_source_add_poll((GSource *)source, &source->poll_fd);
+
+    ac->ev.addRead = redis_source_add_read;
+    ac->ev.delRead = redis_source_del_read;
+    ac->ev.addWrite = redis_source_add_write;
+    ac->ev.delWrite = redis_source_del_write;
+    ac->ev.cleanup = redis_source_cleanup;
+    ac->ev.data = source;
+
+    return (GSource *)source;
+}
+
+#endif /* __HIREDIS_GLIB_H__ */
@@ -0,0 +1,81 @@
+#ifndef __HIREDIS_IVYKIS_H__
+#define __HIREDIS_IVYKIS_H__
+#include <iv.h>
+#include "../hiredis.h"
+#include "../async.h"
+
+typedef struct redisIvykisEvents {
+    redisAsyncContext *context;
+    struct iv_fd fd;
+} redisIvykisEvents;
+
+static void redisIvykisReadEvent(void *arg) {
+    redisAsyncContext *context = (redisAsyncContext *)arg;
+    redisAsyncHandleRead(context);
+}
+
+static void redisIvykisWriteEvent(void *arg) {
+    redisAsyncContext *context = (redisAsyncContext *)arg;
+    redisAsyncHandleWrite(context);
+}
+
+static void redisIvykisAddRead(void *privdata) {
+    redisIvykisEvents *e = (redisIvykisEvents*)privdata;
+    iv_fd_set_handler_in(&e->fd, redisIvykisReadEvent);
+}
+
+static void redisIvykisDelRead(void *privdata) {
+    redisIvykisEvents *e = (redisIvykisEvents*)privdata;
+    iv_fd_set_handler_in(&e->fd, NULL);
+}
+
+static void redisIvykisAddWrite(void *privdata) {
+    redisIvykisEvents *e = (redisIvykisEvents*)privdata;
+    iv_fd_set_handler_out(&e->fd, redisIvykisWriteEvent);
+}
+
+static void redisIvykisDelWrite(void *privdata) {
+    redisIvykisEvents *e = (redisIvykisEvents*)privdata;
+    iv_fd_set_handler_out(&e->fd, NULL);
+}
+
+static void redisIvykisCleanup(void *privdata) {
+    redisIvykisEvents *e = (redisIvykisEvents*)privdata;
+
+    iv_fd_unregister(&e->fd);
+    free(e);
+}
+
+static int redisIvykisAttach(redisAsyncContext *ac) {
+    redisContext *c = &(ac->c);
+    redisIvykisEvents *e;
+
+    /* Nothing should be attached when something is already attached */
+    if (ac->ev.data != NULL)
+        return REDIS_ERR;
+
+    /* Create container for context and r/w events */
+    e = (redisIvykisEvents*)malloc(sizeof(*e));
+    e->context = ac;
+
+    /* Register functions to start/stop listening for events */
+    ac->ev.addRead = redisIvykisAddRead;
+    ac->ev.delRead = redisIvykisDelRead;
+    ac->ev.addWrite = redisIvykisAddWrite;
+    ac->ev.delWrite = redisIvykisDelWrite;
+    ac->ev.cleanup = redisIvykisCleanup;
+    ac->ev.data = e;
+
+    /* Initialize and install read/write events */
+    IV_FD_INIT(&e->fd);
+    e->fd.fd = c->fd;
+    e->fd.handler_in = redisIvykisReadEvent;
+    e->fd.handler_out = redisIvykisWriteEvent;
+    e->fd.handler_err = NULL;
+    e->fd.cookie = e->context;
+
+    iv_fd_register(&e->fd);
+
+    return REDIS_OK;
+}
+#endif
@@ -30,13 +30,13 @@

 #ifndef __HIREDIS_LIBEVENT_H__
 #define __HIREDIS_LIBEVENT_H__
-#include <event.h>
+#include <event2/event.h>
 #include "../hiredis.h"
 #include "../async.h"

 typedef struct redisLibeventEvents {
    redisAsyncContext *context;
-    struct event rev, wev;
+    struct event *rev, *wev;
 } redisLibeventEvents;

 static void redisLibeventReadEvent(int fd, short event, void *arg) {
@@ -53,28 +53,28 @@ static void redisLibeventWriteEvent(int fd, short event, void *arg) {

 static void redisLibeventAddRead(void *privdata) {
    redisLibeventEvents *e = (redisLibeventEvents*)privdata;
-    event_add(&e->rev,NULL);
+    event_add(e->rev,NULL);
 }

 static void redisLibeventDelRead(void *privdata) {
    redisLibeventEvents *e = (redisLibeventEvents*)privdata;
-    event_del(&e->rev);
+    event_del(e->rev);
 }

 static void redisLibeventAddWrite(void *privdata) {
    redisLibeventEvents *e = (redisLibeventEvents*)privdata;
-    event_add(&e->wev,NULL);
+    event_add(e->wev,NULL);
 }

 static void redisLibeventDelWrite(void *privdata) {
    redisLibeventEvents *e = (redisLibeventEvents*)privdata;
-    event_del(&e->wev);
+    event_del(e->wev);
 }

 static void redisLibeventCleanup(void *privdata) {
    redisLibeventEvents *e = (redisLibeventEvents*)privdata;
-    event_del(&e->rev);
-    event_del(&e->wev);
+    event_del(e->rev);
+    event_del(e->wev);
    free(e);
 }

@@ -99,10 +99,10 @@ static int redisLibeventAttach(redisAsyncContext *ac, struct event_base *base) {
    ac->ev.data = e;

    /* Initialize and install read/write events */
-    event_set(&e->rev,c->fd,EV_READ,redisLibeventReadEvent,e);
-    event_set(&e->wev,c->fd,EV_WRITE,redisLibeventWriteEvent,e);
-    event_base_set(base,&e->rev);
-    event_base_set(base,&e->wev);
+    e->rev = event_new(base, c->fd, EV_READ, redisLibeventReadEvent, e);
+    e->wev = event_new(base, c->fd, EV_WRITE, redisLibeventWriteEvent, e);
+    event_add(e->rev, NULL);
+    event_add(e->wev, NULL);
    return REDIS_OK;
 }
 #endif
@@ -1,5 +1,6 @@
 #ifndef __HIREDIS_LIBUV_H__
 #define __HIREDIS_LIBUV_H__
+#include <stdlib.h>
 #include <uv.h>
 #include "../hiredis.h"
 #include "../async.h"
@@ -11,7 +12,6 @@ typedef struct redisLibuvEvents {
  int                events;
 } redisLibuvEvents;

-int redisLibuvAttach(redisAsyncContext*, uv_loop_t*);

 static void redisLibuvPoll(uv_poll_t* handle, int status, int events) {
  redisLibuvEvents* p = (redisLibuvEvents*)handle->data;
@@ -20,10 +20,10 @@ static void redisLibuvPoll(uv_poll_t* handle, int status, int events) {
    return;
  }

-  if (events & UV_READABLE) {
+  if (p->context != NULL && (events & UV_READABLE)) {
    redisAsyncHandleRead(p->context);
  }
-  if (events & UV_WRITABLE) {
+  if (p->context != NULL && (events & UV_WRITABLE)) {
    redisAsyncHandleWrite(p->context);
  }
 }
@@ -83,6 +83,7 @@ static void on_close(uv_handle_t* handle) {
 static void redisLibuvCleanup(void *privdata) {
  redisLibuvEvents* p = (redisLibuvEvents*)privdata;

+  p->context = NULL; // indicate that context might no longer exist
  uv_close((uv_handle_t*)&p->handle, on_close);
 }

@@ -0,0 +1,114 @@
+//
+//  Created by Дмитрий Бахвалов on 13.07.15.
+//  Copyright (c) 2015 Dmitry Bakhvalov. All rights reserved.
+//
+
+#ifndef __HIREDIS_MACOSX_H__
+#define __HIREDIS_MACOSX_H__
+
+#include <CoreFoundation/CoreFoundation.h>
+
+#include "../hiredis.h"
+#include "../async.h"
+
+typedef struct {
+    redisAsyncContext *context;
+    CFSocketRef socketRef;
+    CFRunLoopSourceRef sourceRef;
+} RedisRunLoop;
+
+static int freeRedisRunLoop(RedisRunLoop* redisRunLoop) {
+    if( redisRunLoop != NULL ) {
+        if( redisRunLoop->sourceRef != NULL ) {
+            CFRunLoopSourceInvalidate(redisRunLoop->sourceRef);
+            CFRelease(redisRunLoop->sourceRef);
+        }
+        if( redisRunLoop->socketRef != NULL ) {
+            CFSocketInvalidate(redisRunLoop->socketRef);
+            CFRelease(redisRunLoop->socketRef);
+        }
+        free(redisRunLoop);
+    }
+    return REDIS_ERR;
+}
+
+static void redisMacOSAddRead(void *privdata) {
+    RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
+    CFSocketEnableCallBacks(redisRunLoop->socketRef, kCFSocketReadCallBack);
+}
+
+static void redisMacOSDelRead(void *privdata) {
+    RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
+    CFSocketDisableCallBacks(redisRunLoop->socketRef, kCFSocketReadCallBack);
+}
+
+static void redisMacOSAddWrite(void *privdata) {
+    RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
+    CFSocketEnableCallBacks(redisRunLoop->socketRef, kCFSocketWriteCallBack);
+}
+
+static void redisMacOSDelWrite(void *privdata) {
+    RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
+    CFSocketDisableCallBacks(redisRunLoop->socketRef, kCFSocketWriteCallBack);
+}
+
+static void redisMacOSCleanup(void *privdata) {
+    RedisRunLoop *redisRunLoop = (RedisRunLoop*)privdata;
+    freeRedisRunLoop(redisRunLoop);
+}
+
+static void redisMacOSAsyncCallback(CFSocketRef __unused s, CFSocketCallBackType callbackType, CFDataRef __unused address, const void __unused *data, void *info) {
+    redisAsyncContext* context = (redisAsyncContext*) info;
+
+    switch (callbackType) {
+        case kCFSocketReadCallBack:
+            redisAsyncHandleRead(context);
+            break;
+
+        case kCFSocketWriteCallBack:
+            redisAsyncHandleWrite(context);
+            break;
+
+        default:
+            break;
+    }
+}
+
+static int redisMacOSAttach(redisAsyncContext *redisAsyncCtx, CFRunLoopRef runLoop) {
+    redisContext *redisCtx = &(redisAsyncCtx->c);
+
+    /* Nothing should be attached when something is already attached */
+    if( redisAsyncCtx->ev.data != NULL ) return REDIS_ERR;
+
+    RedisRunLoop* redisRunLoop = (RedisRunLoop*) calloc(1, sizeof(RedisRunLoop));
+    if( !redisRunLoop ) return REDIS_ERR;
+
+    /* Setup redis stuff */
+    redisRunLoop->context = redisAsyncCtx;
+
+    redisAsyncCtx->ev.addRead  = redisMacOSAddRead;
+    redisAsyncCtx->ev.delRead  = redisMacOSDelRead;
+    redisAsyncCtx->ev.addWrite = redisMacOSAddWrite;
+    redisAsyncCtx->ev.delWrite = redisMacOSDelWrite;
+    redisAsyncCtx->ev.cleanup  = redisMacOSCleanup;
+    redisAsyncCtx->ev.data     = redisRunLoop;
+
+    /* Initialize and install read/write events */
+    CFSocketContext socketCtx = { 0, redisAsyncCtx, NULL, NULL, NULL };
+
+    redisRunLoop->socketRef = CFSocketCreateWithNative(NULL, redisCtx->fd,
+                                                       kCFSocketReadCallBack | kCFSocketWriteCallBack,
+                                                       redisMacOSAsyncCallback,
+                                                       &socketCtx);
+    if( !redisRunLoop->socketRef ) return freeRedisRunLoop(redisRunLoop);
+
+    redisRunLoop->sourceRef = CFSocketCreateRunLoopSource(NULL, redisRunLoop->socketRef, 0);
+    if( !redisRunLoop->sourceRef ) return freeRedisRunLoop(redisRunLoop);
+
+    CFRunLoopAddSource(runLoop, redisRunLoop->sourceRef, kCFRunLoopDefaultMode);
+
+    return REDIS_OK;
+}
+
+#endif
+
@@ -0,0 +1,135 @@
+/*-
+ * Copyright (C) 2014 Pietro Cerutti <gahr@gahr.ch>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#ifndef __HIREDIS_QT_H__
+#define __HIREDIS_QT_H__
+#include <QSocketNotifier>
+#include "../async.h"
+
+static void RedisQtAddRead(void *);
+static void RedisQtDelRead(void *);
+static void RedisQtAddWrite(void *);
+static void RedisQtDelWrite(void *);
+static void RedisQtCleanup(void *);
+
+class RedisQtAdapter : public QObject {
+
+    Q_OBJECT
+
+    friend
+    void RedisQtAddRead(void * adapter) {
+        RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
+        a->addRead();
+    }
+
+    friend
+    void RedisQtDelRead(void * adapter) {
+        RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
+        a->delRead();
+    }
+
+    friend
+    void RedisQtAddWrite(void * adapter) {
+        RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
+        a->addWrite();
+    }
+
+    friend
+    void RedisQtDelWrite(void * adapter) {
+        RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
+        a->delWrite();
+    }
+
+    friend
+    void RedisQtCleanup(void * adapter) {
+        RedisQtAdapter * a = static_cast<RedisQtAdapter *>(adapter);
+        a->cleanup();
+    }
+
+    public:
+        RedisQtAdapter(QObject * parent = 0)
+            : QObject(parent), m_ctx(0), m_read(0), m_write(0) { }
+
+        ~RedisQtAdapter() {
+            if (m_ctx != 0) {
+                m_ctx->ev.data = NULL;
+            }
+        }
+
+        int setContext(redisAsyncContext * ac) {
+            if (ac->ev.data != NULL) {
+                return REDIS_ERR;
+            }
+            m_ctx = ac;
+            m_ctx->ev.data = this;
+            m_ctx->ev.addRead = RedisQtAddRead;
+            m_ctx->ev.delRead = RedisQtDelRead;
+            m_ctx->ev.addWrite = RedisQtAddWrite;
+            m_ctx->ev.delWrite = RedisQtDelWrite;
+            m_ctx->ev.cleanup = RedisQtCleanup;
+            return REDIS_OK;
+        }
+
+    private:
+        void addRead() {
+            if (m_read) return;
+            m_read = new QSocketNotifier(m_ctx->c.fd, QSocketNotifier::Read, 0);
+            connect(m_read, SIGNAL(activated(int)), this, SLOT(read()));
+        }
+
+        void delRead() {
+            if (!m_read) return;
+            delete m_read;
+            m_read = 0;
+        }
+
+        void addWrite() {
+            if (m_write) return;
+            m_write = new QSocketNotifier(m_ctx->c.fd, QSocketNotifier::Write, 0);
+            connect(m_write, SIGNAL(activated(int)), this, SLOT(write()));
+        }
+
+        void delWrite() {
+            if (!m_write) return;
+            delete m_write;
+            m_write = 0;
+        }
+
+        void cleanup() {
+            delRead();
+            delWrite();
+        }
+
+    private slots:
+        void read() { redisAsyncHandleRead(m_ctx); }
+        void write() { redisAsyncHandleWrite(m_ctx); }
+
+    private:
+        redisAsyncContext * m_ctx;
+        QSocketNotifier * m_read;
+        QSocketNotifier * m_write;
+};
+
+#endif /* !__HIREDIS_QT_H__ */
@@ -0,0 +1,36 @@
+# Appveyor configuration file for CI build of hiredis on Windows (under Cygwin)
+environment:
+    matrix:
+        - CYG_ROOT: C:\cygwin64
+          CYG_SETUP: setup-x86_64.exe
+          CYG_MIRROR: http://cygwin.mirror.constant.com
+          CYG_CACHE: C:\cygwin64\var\cache\setup
+          CYG_BASH: C:\cygwin64\bin\bash
+          CC: gcc
+        - CYG_ROOT: C:\cygwin
+          CYG_SETUP: setup-x86.exe
+          CYG_MIRROR: http://cygwin.mirror.constant.com
+          CYG_CACHE: C:\cygwin\var\cache\setup
+          CYG_BASH: C:\cygwin\bin\bash
+          CC: gcc
+          TARGET: 32bit
+          TARGET_VARS: 32bit-vars
+
+# Cache Cygwin files to speed up build
+cache:
+    - '%CYG_CACHE%'
+clone_depth: 1
+
+# Attempt to ensure we don't try to convert line endings to Win32 CRLF as this will cause build to fail
+init:
+    - git config --global core.autocrlf input
+
+# Install needed build dependencies
+install:
+    - ps: 'Start-FileDownload "http://cygwin.com/$env:CYG_SETUP" -FileName "$env:CYG_SETUP"'
+    - '%CYG_SETUP% --quiet-mode --no-shortcuts --only-site --root "%CYG_ROOT%" --site "%CYG_MIRROR%" --local-package-dir "%CYG_CACHE%" --packages automake,bison,gcc-core,libtool,make,gettext-devel,gettext,intltool,pkg-config,clang,llvm > NUL 2>&1'
+    - '%CYG_BASH% -lc "cygcheck -dc cygwin"'
+
+build_script:
+    - 'echo building...'
+    - '%CYG_BASH% -lc "cd $APPVEYOR_BUILD_FOLDER; exec 0</dev/null; make LDFLAGS=$LDFLAGS CC=$CC $TARGET CFLAGS=$CFLAGS && make LDFLAGS=$LDFLAGS CC=$CC $TARGET_VARS hiredis-example"'
@@ -58,7 +58,7 @@
    } while(0);

 /* Forward declaration of function in hiredis.c */
-void __redisAppendCommand(redisContext *c, char *cmd, size_t len);
+int __redisAppendCommand(redisContext *c, const char *cmd, size_t len);

 /* Functions managing dictionary of callbacks for pub/sub. */
 static unsigned int callbackHash(const void *key) {
@@ -142,6 +142,9 @@ static redisAsyncContext *redisAsyncInitialize(redisContext *c) {
 /* We want the error field to be accessible directly instead of requiring
 * an indirection to the redisContext struct. */
 static void __redisAsyncCopyError(redisAsyncContext *ac) {
+    if (!ac)
+        return;
+
    redisContext *c = &(ac->c);
    ac->err = c->err;
    ac->errstr = c->errstr;
@@ -173,6 +176,14 @@ redisAsyncContext *redisAsyncConnectBind(const char *ip, int port,
    return ac;
 }

+redisAsyncContext *redisAsyncConnectBindWithReuse(const char *ip, int port,
+                                                  const char *source_addr) {
+    redisContext *c = redisConnectBindNonBlockWithReuse(ip,port,source_addr);
+    redisAsyncContext *ac = redisAsyncInitialize(c);
+    __redisAsyncCopyError(ac);
+    return ac;
+}
+
 redisAsyncContext *redisAsyncConnectUnix(const char *path) {
    redisContext *c;
    redisAsyncContext *ac;
@@ -407,7 +418,8 @@ void redisProcessCallbacks(redisAsyncContext *ac) {
        if (reply == NULL) {
            /* When the connection is being disconnected and there are
             * no more replies, this is the cue to really disconnect. */
-            if (c->flags & REDIS_DISCONNECTING && sdslen(c->obuf) == 0) {
+            if (c->flags & REDIS_DISCONNECTING && sdslen(c->obuf) == 0
+                && ac->replies.head == NULL) {
                __redisAsyncDisconnect(ac);
                return;
            }
@@ -477,7 +489,7 @@ void redisProcessCallbacks(redisAsyncContext *ac) {
 }

 /* Internal helper function to detect socket status the first time a read or
- * write event fires. When connecting was not succesful, the connect callback
+ * write event fires. When connecting was not successful, the connect callback
 * is called with a REDIS_ERR status and the context is free'd. */
 static int __redisAsyncHandleConnect(redisAsyncContext *ac) {
    redisContext *c = &(ac->c);
@@ -551,8 +563,8 @@ void redisAsyncHandleWrite(redisAsyncContext *ac) {

 /* Sets a pointer to the first argument and its length starting at p. Returns
 * the number of bytes to skip to get to the following argument. */
-static char *nextArgument(char *start, char **str, size_t *len) {
-    char *p = start;
+static const char *nextArgument(const char *start, const char **str, size_t *len) {
+    const char *p = start;
    if (p[0] != '$') {
        p = strchr(p,'$');
        if (p == NULL) return NULL;
@@ -568,14 +580,15 @@ static char *nextArgument(char *start, char **str, size_t *len) {
 /* Helper function for the redisAsyncCommand* family of functions. Writes a
 * formatted command to the output buffer and registers the provided callback
 * function with the context. */
-static int __redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, char *cmd, size_t len) {
+static int __redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, const char *cmd, size_t len) {
    redisContext *c = &(ac->c);
    redisCallback cb;
    int pvariant, hasnext;
-    char *cstr, *astr;
+    const char *cstr, *astr;
    size_t clen, alen;
-    char *p;
+    const char *p;
    sds sname;
+    int ret;

    /* Don't accept new commands when the connection is about to be closed. */
    if (c->flags & (REDIS_DISCONNECTING | REDIS_FREEING)) return REDIS_ERR;
@@ -599,9 +612,11 @@ static int __redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void
        while ((p = nextArgument(p,&astr,&alen)) != NULL) {
            sname = sdsnewlen(astr,alen);
            if (pvariant)
-                dictReplace(ac->sub.patterns,sname,&cb);
+                ret = dictReplace(ac->sub.patterns,sname,&cb);
            else
-                dictReplace(ac->sub.channels,sname,&cb);
+                ret = dictReplace(ac->sub.channels,sname,&cb);
+
+            if (ret == 0) sdsfree(sname);
        }
    } else if (strncasecmp(cstr,"unsubscribe\r\n",13) == 0) {
        /* It is only useful to call (P)UNSUBSCRIBE when the context is
@@ -637,6 +652,11 @@ int redisvAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdat
    int len;
    int status;
    len = redisvFormatCommand(&cmd,format,ap);
+
+    /* We don't want to pass -1 or -2 to future functions as a length. */
+    if (len < 0)
+        return REDIS_ERR;
+
    status = __redisAsyncCommand(ac,fn,privdata,cmd,len);
    free(cmd);
    return status;
@@ -652,11 +672,16 @@ int redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata
 }

 int redisAsyncCommandArgv(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, int argc, const char **argv, const size_t *argvlen) {
-    char *cmd;
+    sds cmd;
    int len;
    int status;
-    len = redisFormatCommandArgv(&cmd,argc,argv,argvlen);
+    len = redisFormatSdsCommandArgv(&cmd,argc,argv,argvlen);
    status = __redisAsyncCommand(ac,fn,privdata,cmd,len);
-    free(cmd);
+    sdsfree(cmd);
+    return status;
+}
+
+int redisAsyncFormattedCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, const char *cmd, size_t len) {
+    int status = __redisAsyncCommand(ac,fn,privdata,cmd,len);
    return status;
 }
@@ -103,6 +103,8 @@ typedef struct redisAsyncContext {
 /* Functions that proxy to hiredis */
 redisAsyncContext *redisAsyncConnect(const char *ip, int port);
 redisAsyncContext *redisAsyncConnectBind(const char *ip, int port, const char *source_addr);
+redisAsyncContext *redisAsyncConnectBindWithReuse(const char *ip, int port,
+                                                  const char *source_addr);
 redisAsyncContext *redisAsyncConnectUnix(const char *path);
 int redisAsyncSetConnectCallback(redisAsyncContext *ac, redisConnectCallback *fn);
 int redisAsyncSetDisconnectCallback(redisAsyncContext *ac, redisDisconnectCallback *fn);
@@ -118,6 +120,7 @@ void redisAsyncHandleWrite(redisAsyncContext *ac);
 int redisvAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, const char *format, va_list ap);
 int redisAsyncCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, const char *format, ...);
 int redisAsyncCommandArgv(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, int argc, const char **argv, const size_t *argvlen);
+int redisAsyncFormattedCommand(redisAsyncContext *ac, redisCallbackFn *fn, void *privdata, const char *cmd, size_t len);

 #ifdef __cplusplus
 }
@@ -161,7 +161,7 @@ static int dictReplace(dict *ht, void *key, void *val) {
    dictEntry *entry, auxentry;

    /* Try to add the element. If the key
-     * does not exists dictAdd will suceed. */
+     * does not exists dictAdd will succeed. */
    if (dictAdd(ht, key, val) == DICT_OK)
        return 1;
    /* It already exists, get the entry */
@@ -293,7 +293,7 @@ static void dictReleaseIterator(dictIterator *iter) {

 /* Expand the hash table if needed */
 static int _dictExpandIfNeeded(dict *ht) {
-    /* If the hash table is empty expand it to the intial size,
+    /* If the hash table is empty expand it to the initial size,
     * if the table is "full" dobule its size. */
    if (ht->size == 0)
        return dictExpand(ht, DICT_HT_INITIAL_SIZE);
@@ -0,0 +1,73 @@
+#include <stdlib.h>
+
+#include <hiredis.h>
+#include <async.h>
+#include <adapters/glib.h>
+
+static GMainLoop *mainloop;
+
+static void
+connect_cb (const redisAsyncContext *ac G_GNUC_UNUSED,
+            int status)
+{
+    if (status != REDIS_OK) {
+        g_printerr("Failed to connect: %s\n", ac->errstr);
+        g_main_loop_quit(mainloop);
+    } else {
+        g_printerr("Connected...\n");
+    }
+}
+
+static void
+disconnect_cb (const redisAsyncContext *ac G_GNUC_UNUSED,
+               int status)
+{
+    if (status != REDIS_OK) {
+        g_error("Failed to disconnect: %s", ac->errstr);
+    } else {
+        g_printerr("Disconnected...\n");
+        g_main_loop_quit(mainloop);
+    }
+}
+
+static void
+command_cb(redisAsyncContext *ac,
+           gpointer r,
+           gpointer user_data G_GNUC_UNUSED)
+{
+    redisReply *reply = r;
+
+    if (reply) {
+        g_print("REPLY: %s\n", reply->str);
+    }
+
+    redisAsyncDisconnect(ac);
+}
+
+gint
+main (gint argc     G_GNUC_UNUSED,
+      gchar *argv[] G_GNUC_UNUSED)
+{
+    redisAsyncContext *ac;
+    GMainContext *context = NULL;
+    GSource *source;
+
+    ac = redisAsyncConnect("127.0.0.1", 6379);
+    if (ac->err) {
+        g_printerr("%s\n", ac->errstr);
+        exit(EXIT_FAILURE);
+    }
+
+    source = redis_source_new(ac);
+    mainloop = g_main_loop_new(context, FALSE);
+    g_source_attach(source, context);
+
+    redisAsyncSetConnectCallback(ac, connect_cb);
+    redisAsyncSetDisconnectCallback(ac, disconnect_cb);
+    redisAsyncCommand(ac, command_cb, NULL, "SET key 1234");
+    redisAsyncCommand(ac, command_cb, NULL, "GET key");
+
+    g_main_loop_run(mainloop);
+
+    return EXIT_SUCCESS;
+}
@@ -0,0 +1,58 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <signal.h>
+
+#include <hiredis.h>
+#include <async.h>
+#include <adapters/ivykis.h>
+
+void getCallback(redisAsyncContext *c, void *r, void *privdata) {
+    redisReply *reply = r;
+    if (reply == NULL) return;
+    printf("argv[%s]: %s\n", (char*)privdata, reply->str);
+
+    /* Disconnect after receiving the reply to GET */
+    redisAsyncDisconnect(c);
+}
+
+void connectCallback(const redisAsyncContext *c, int status) {
+    if (status != REDIS_OK) {
+        printf("Error: %s\n", c->errstr);
+        return;
+    }
+    printf("Connected...\n");
+}
+
+void disconnectCallback(const redisAsyncContext *c, int status) {
+    if (status != REDIS_OK) {
+        printf("Error: %s\n", c->errstr);
+        return;
+    }
+    printf("Disconnected...\n");
+}
+
+int main (int argc, char **argv) {
+    signal(SIGPIPE, SIG_IGN);
+
+    iv_init();
+
+    redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
+    if (c->err) {
+        /* Let *c leak for now... */
+        printf("Error: %s\n", c->errstr);
+        return 1;
+    }
+
+    redisIvykisAttach(c);
+    redisAsyncSetConnectCallback(c,connectCallback);
+    redisAsyncSetDisconnectCallback(c,disconnectCallback);
+    redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
+    redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
+
+    iv_main();
+
+    iv_deinit();
+
+    return 0;
+}
@@ -0,0 +1,66 @@
+//
+//  Created by Дмитрий Бахвалов on 13.07.15.
+//  Copyright (c) 2015 Dmitry Bakhvalov. All rights reserved.
+//
+
+#include <stdio.h>
+
+#include <hiredis.h>
+#include <async.h>
+#include <adapters/macosx.h>
+
+void getCallback(redisAsyncContext *c, void *r, void *privdata) {
+    redisReply *reply = r;
+    if (reply == NULL) return;
+    printf("argv[%s]: %s\n", (char*)privdata, reply->str);
+
+    /* Disconnect after receiving the reply to GET */
+    redisAsyncDisconnect(c);
+}
+
+void connectCallback(const redisAsyncContext *c, int status) {
+    if (status != REDIS_OK) {
+        printf("Error: %s\n", c->errstr);
+        return;
+    }
+    printf("Connected...\n");
+}
+
+void disconnectCallback(const redisAsyncContext *c, int status) {
+    if (status != REDIS_OK) {
+        printf("Error: %s\n", c->errstr);
+        return;
+    }
+    CFRunLoopStop(CFRunLoopGetCurrent());
+    printf("Disconnected...\n");
+}
+
+int main (int argc, char **argv) {
+    signal(SIGPIPE, SIG_IGN);
+
+    CFRunLoopRef loop = CFRunLoopGetCurrent();
+    if( !loop ) {
+        printf("Error: Cannot get current run loop\n");
+        return 1;
+    }
+
+    redisAsyncContext *c = redisAsyncConnect("127.0.0.1", 6379);
+    if (c->err) {
+        /* Let *c leak for now... */
+        printf("Error: %s\n", c->errstr);
+        return 1;
+    }
+
+    redisMacOSAttach(c, loop);
+
+    redisAsyncSetConnectCallback(c,connectCallback);
+    redisAsyncSetDisconnectCallback(c,disconnectCallback);
+
+    redisAsyncCommand(c, NULL, NULL, "SET key %b", argv[argc-1], strlen(argv[argc-1]));
+    redisAsyncCommand(c, getCallback, (char*)"end-1", "GET key");
+
+    CFRunLoopRun();
+
+    return 0;
+}
+
@@ -0,0 +1,46 @@
+#include <iostream>
+using namespace std;
+
+#include <QCoreApplication>
+#include <QTimer>
+
+#include "example-qt.h"
+
+void getCallback(redisAsyncContext *, void * r, void * privdata) {
+
+    redisReply * reply = static_cast<redisReply *>(r);
+    ExampleQt * ex = static_cast<ExampleQt *>(privdata);
+    if (reply == nullptr || ex == nullptr) return;
+
+    cout << "key: " << reply->str << endl;
+
+    ex->finish();
+}
+
+void ExampleQt::run() {
+
+    m_ctx = redisAsyncConnect("localhost", 6379);
+
+    if (m_ctx->err) {
+        cerr << "Error: " << m_ctx->errstr << endl;
+        redisAsyncFree(m_ctx);
+        emit finished();
+    }
+
+    m_adapter.setContext(m_ctx);
+
+    redisAsyncCommand(m_ctx, NULL, NULL, "SET key %s", m_value);
+    redisAsyncCommand(m_ctx, getCallback, this, "GET key");
+}
+
+int main (int argc, char **argv) {
+
+    QCoreApplication app(argc, argv);
+
+    ExampleQt example(argv[argc-1]);
+
+    QObject::connect(&example, SIGNAL(finished()), &app, SLOT(quit()));
+    QTimer::singleShot(0, &example, SLOT(run()));
+
+    return app.exec();
+}
@@ -0,0 +1,32 @@
+#ifndef __HIREDIS_EXAMPLE_QT_H
+#define __HIREDIS_EXAMPLE_QT_H
+
+#include <adapters/qt.h>
+
+class ExampleQt : public QObject {
+
+    Q_OBJECT
+
+    public:
+        ExampleQt(const char * value, QObject * parent = 0)
+            : QObject(parent), m_value(value) {}
+
+    signals:
+        void finished();
+
+    public slots:
+        void run();
+
+    private:
+        void finish() { emit finished(); }
+
+    private:
+        const char * m_value;
+        redisAsyncContext * m_ctx;
+        RedisQtAdapter m_adapter;
+
+    friend
+    void getCallback(redisAsyncContext *, void *, void *);
+};
+
+#endif /* !__HIREDIS_EXAMPLE_QT_H */
@@ -57,7 +57,7 @@ int main(int argc, char **argv) {
    for (j = 0; j < 10; j++) {
        char buf[64];

-        snprintf(buf,64,"%d",j);
+        snprintf(buf,64,"%u",j);
        reply = redisCommand(c,"LPUSH mylist element-%s", buf);
        freeReplyObject(reply);
    }
@@ -1,23 +1,24 @@
 #ifndef __HIREDIS_FMACRO_H
 #define __HIREDIS_FMACRO_H

-#if !defined(_BSD_SOURCE)
+#if defined(__linux__)
 #define _BSD_SOURCE
+#define _DEFAULT_SOURCE
 #endif

-#if defined(_AIX)
-#define _ALL_SOURCE
+#if defined(__CYGWIN__)
+#include <sys/cdefs.h>
 #endif

 #if defined(__sun__)
 #define _POSIX_C_SOURCE 200112L
-#elif defined(__linux__) || defined(__OpenBSD__) || defined(__NetBSD__)
-#define _XOPEN_SOURCE 600
 #else
-#define _XOPEN_SOURCE
+#if !(defined(__APPLE__) && defined(__MACH__))
+#define _XOPEN_SOURCE 600
+#endif
 #endif

-#if __APPLE__ && __MACH__
+#if defined(__APPLE__) && defined(__MACH__)
 #define _OSX
 #endif

@@ -1,6 +1,8 @@
 /*
 * Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
- * Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
+ * Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
+ * Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
+ *                     Jan-Erik Rediger <janerik at fnordig dot com>
 *
 * All rights reserved.
 *
@@ -73,6 +75,9 @@ void freeReplyObject(void *reply) {
    redisReply *r = reply;
    size_t j;

+    if (r == NULL)
+        return;
+
    switch(r->type) {
    case REDIS_REPLY_INTEGER:
        break; /* Nothing to free */
@@ -183,504 +188,23 @@ static void *createNilObject(const redisReadTask *task) {
    return r;
 }

-static void __redisReaderSetError(redisReader *r, int type, const char *str) {
-    size_t len;
-
-    if (r->reply != NULL && r->fn && r->fn->freeObject) {
-        r->fn->freeObject(r->reply);
-        r->reply = NULL;
-    }
-
-    /* Clear input buffer on errors. */
-    if (r->buf != NULL) {
-        sdsfree(r->buf);
-        r->buf = NULL;
-        r->pos = r->len = 0;
-    }
-
-    /* Reset task stack. */
-    r->ridx = -1;
-
-    /* Set error. */
-    r->err = type;
-    len = strlen(str);
-    len = len < (sizeof(r->errstr)-1) ? len : (sizeof(r->errstr)-1);
-    memcpy(r->errstr,str,len);
-    r->errstr[len] = '\0';
-}
-
-static size_t chrtos(char *buf, size_t size, char byte) {
-    size_t len = 0;
-
-    switch(byte) {
-    case '\\':
-    case '"':
-        len = snprintf(buf,size,"\"\\%c\"",byte);
-        break;
-    case '\n': len = snprintf(buf,size,"\"\\n\""); break;
-    case '\r': len = snprintf(buf,size,"\"\\r\""); break;
-    case '\t': len = snprintf(buf,size,"\"\\t\""); break;
-    case '\a': len = snprintf(buf,size,"\"\\a\""); break;
-    case '\b': len = snprintf(buf,size,"\"\\b\""); break;
-    default:
-        if (isprint(byte))
-            len = snprintf(buf,size,"\"%c\"",byte);
-        else
-            len = snprintf(buf,size,"\"\\x%02x\"",(unsigned char)byte);
-        break;
-    }
-
-    return len;
-}
-
-static void __redisReaderSetErrorProtocolByte(redisReader *r, char byte) {
-    char cbuf[8], sbuf[128];
-
-    chrtos(cbuf,sizeof(cbuf),byte);
-    snprintf(sbuf,sizeof(sbuf),
-        "Protocol error, got %s as reply type byte", cbuf);
-    __redisReaderSetError(r,REDIS_ERR_PROTOCOL,sbuf);
-}
-
-static void __redisReaderSetErrorOOM(redisReader *r) {
-    __redisReaderSetError(r,REDIS_ERR_OOM,"Out of memory");
-}
-
-static char *readBytes(redisReader *r, unsigned int bytes) {
-    char *p;
-    if (r->len-r->pos >= bytes) {
-        p = r->buf+r->pos;
-        r->pos += bytes;
-        return p;
-    }
-    return NULL;
-}
-
-/* Find pointer to \r\n. */
-static char *seekNewline(char *s, size_t len) {
-    int pos = 0;
-    int _len = len-1;
-
-    /* Position should be < len-1 because the character at "pos" should be
-     * followed by a \n. Note that strchr cannot be used because it doesn't
-     * allow to search a limited length and the buffer that is being searched
-     * might not have a trailing NULL character. */
-    while (pos < _len) {
-        while(pos < _len && s[pos] != '\r') pos++;
-        if (s[pos] != '\r') {
-            /* Not found. */
-            return NULL;
-        } else {
-            if (s[pos+1] == '\n') {
-                /* Found. */
-                return s+pos;
-            } else {
-                /* Continue searching. */
-                pos++;
-            }
-        }
-    }
-    return NULL;
-}
-
-/* Read a long long value starting at *s, under the assumption that it will be
- * terminated by \r\n. Ambiguously returns -1 for unexpected input. */
-static long long readLongLong(char *s) {
-    long long v = 0;
-    int dec, mult = 1;
-    char c;
-
-    if (*s == '-') {
-        mult = -1;
-        s++;
-    } else if (*s == '+') {
-        mult = 1;
-        s++;
-    }
-
-    while ((c = *(s++)) != '\r') {
-        dec = c - '0';
-        if (dec >= 0 && dec < 10) {
-            v *= 10;
-            v += dec;
-        } else {
-            /* Should not happen... */
-            return -1;
-        }
-    }
-
-    return mult*v;
-}
-
-static char *readLine(redisReader *r, int *_len) {
-    char *p, *s;
-    int len;
-
-    p = r->buf+r->pos;
-    s = seekNewline(p,(r->len-r->pos));
-    if (s != NULL) {
-        len = s-(r->buf+r->pos);
-        r->pos += len+2; /* skip \r\n */
-        if (_len) *_len = len;
-        return p;
-    }
-    return NULL;
-}
-
-static void moveToNextTask(redisReader *r) {
-    redisReadTask *cur, *prv;
-    while (r->ridx >= 0) {
-        /* Return a.s.a.p. when the stack is now empty. */
-        if (r->ridx == 0) {
-            r->ridx--;
-            return;
-        }
-
-        cur = &(r->rstack[r->ridx]);
-        prv = &(r->rstack[r->ridx-1]);
-        assert(prv->type == REDIS_REPLY_ARRAY);
-        if (cur->idx == prv->elements-1) {
-            r->ridx--;
-        } else {
-            /* Reset the type because the next item can be anything */
-            assert(cur->idx < prv->elements);
-            cur->type = -1;
-            cur->elements = -1;
-            cur->idx++;
-            return;
-        }
-    }
-}
-
-static int processLineItem(redisReader *r) {
-    redisReadTask *cur = &(r->rstack[r->ridx]);
-    void *obj;
-    char *p;
-    int len;
-
-    if ((p = readLine(r,&len)) != NULL) {
-        if (cur->type == REDIS_REPLY_INTEGER) {
-            if (r->fn && r->fn->createInteger)
-                obj = r->fn->createInteger(cur,readLongLong(p));
-            else
-                obj = (void*)REDIS_REPLY_INTEGER;
-        } else {
-            /* Type will be error or status. */
-            if (r->fn && r->fn->createString)
-                obj = r->fn->createString(cur,p,len);
-            else
-                obj = (void*)(size_t)(cur->type);
-        }
-
-        if (obj == NULL) {
-            __redisReaderSetErrorOOM(r);
-            return REDIS_ERR;
-        }
-
-        /* Set reply if this is the root object. */
-        if (r->ridx == 0) r->reply = obj;
-        moveToNextTask(r);
-        return REDIS_OK;
-    }
-
-    return REDIS_ERR;
-}
-
-static int processBulkItem(redisReader *r) {
-    redisReadTask *cur = &(r->rstack[r->ridx]);
-    void *obj = NULL;
-    char *p, *s;
-    long len;
-    unsigned long bytelen;
-    int success = 0;
-
-    p = r->buf+r->pos;
-    s = seekNewline(p,r->len-r->pos);
-    if (s != NULL) {
-        p = r->buf+r->pos;
-        bytelen = s-(r->buf+r->pos)+2; /* include \r\n */
-        len = readLongLong(p);
-
-        if (len < 0) {
-            /* The nil object can always be created. */
-            if (r->fn && r->fn->createNil)
-                obj = r->fn->createNil(cur);
-            else
-                obj = (void*)REDIS_REPLY_NIL;
-            success = 1;
-        } else {
-            /* Only continue when the buffer contains the entire bulk item. */
-            bytelen += len+2; /* include \r\n */
-            if (r->pos+bytelen <= r->len) {
-                if (r->fn && r->fn->createString)
-                    obj = r->fn->createString(cur,s+2,len);
-                else
-                    obj = (void*)REDIS_REPLY_STRING;
-                success = 1;
-            }
-        }
-
-        /* Proceed when obj was created. */
-        if (success) {
-            if (obj == NULL) {
-                __redisReaderSetErrorOOM(r);
-                return REDIS_ERR;
-            }
-
-            r->pos += bytelen;
-
-            /* Set reply if this is the root object. */
-            if (r->ridx == 0) r->reply = obj;
-            moveToNextTask(r);
-            return REDIS_OK;
-        }
-    }
-
-    return REDIS_ERR;
-}
-
-static int processMultiBulkItem(redisReader *r) {
-    redisReadTask *cur = &(r->rstack[r->ridx]);
-    void *obj;
-    char *p;
-    long elements;
-    int root = 0;
-
-    /* Set error for nested multi bulks with depth > 7 */
-    if (r->ridx == 8) {
-        __redisReaderSetError(r,REDIS_ERR_PROTOCOL,
-            "No support for nested multi bulk replies with depth > 7");
-        return REDIS_ERR;
-    }
-
-    if ((p = readLine(r,NULL)) != NULL) {
-        elements = readLongLong(p);
-        root = (r->ridx == 0);
-
-        if (elements == -1) {
-            if (r->fn && r->fn->createNil)
-                obj = r->fn->createNil(cur);
-            else
-                obj = (void*)REDIS_REPLY_NIL;
-
-            if (obj == NULL) {
-                __redisReaderSetErrorOOM(r);
-                return REDIS_ERR;
-            }
-
-            moveToNextTask(r);
-        } else {
-            if (r->fn && r->fn->createArray)
-                obj = r->fn->createArray(cur,elements);
-            else
-                obj = (void*)REDIS_REPLY_ARRAY;
-
-            if (obj == NULL) {
-                __redisReaderSetErrorOOM(r);
-                return REDIS_ERR;
-            }
-
-            /* Modify task stack when there are more than 0 elements. */
-            if (elements > 0) {
-                cur->elements = elements;
-                cur->obj = obj;
-                r->ridx++;
-                r->rstack[r->ridx].type = -1;
-                r->rstack[r->ridx].elements = -1;
-                r->rstack[r->ridx].idx = 0;
-                r->rstack[r->ridx].obj = NULL;
-                r->rstack[r->ridx].parent = cur;
-                r->rstack[r->ridx].privdata = r->privdata;
-            } else {
-                moveToNextTask(r);
-            }
-        }
-
-        /* Set reply if this is the root object. */
-        if (root) r->reply = obj;
-        return REDIS_OK;
-    }
-
-    return REDIS_ERR;
-}
-
-static int processItem(redisReader *r) {
-    redisReadTask *cur = &(r->rstack[r->ridx]);
-    char *p;
-
-    /* check if we need to read type */
-    if (cur->type < 0) {
-        if ((p = readBytes(r,1)) != NULL) {
-            switch (p[0]) {
-            case '-':
-                cur->type = REDIS_REPLY_ERROR;
-                break;
-            case '+':
-                cur->type = REDIS_REPLY_STATUS;
-                break;
-            case ':':
-                cur->type = REDIS_REPLY_INTEGER;
-                break;
-            case '$':
-                cur->type = REDIS_REPLY_STRING;
-                break;
-            case '*':
-                cur->type = REDIS_REPLY_ARRAY;
-                break;
-            default:
-                __redisReaderSetErrorProtocolByte(r,*p);
-                return REDIS_ERR;
-            }
-        } else {
-            /* could not consume 1 byte */
-            return REDIS_ERR;
-        }
-    }
-
-    /* process typed item */
-    switch(cur->type) {
-    case REDIS_REPLY_ERROR:
-    case REDIS_REPLY_STATUS:
-    case REDIS_REPLY_INTEGER:
-        return processLineItem(r);
-    case REDIS_REPLY_STRING:
-        return processBulkItem(r);
-    case REDIS_REPLY_ARRAY:
-        return processMultiBulkItem(r);
-    default:
-        assert(NULL);
-        return REDIS_ERR; /* Avoid warning. */
-    }
-}
-
-redisReader *redisReaderCreate(void) {
-    redisReader *r;
-
-    r = calloc(sizeof(redisReader),1);
-    if (r == NULL)
-        return NULL;
-
-    r->err = 0;
-    r->errstr[0] = '\0';
-    r->fn = &defaultFunctions;
-    r->buf = sdsempty();
-    r->maxbuf = REDIS_READER_MAX_BUF;
-    if (r->buf == NULL) {
-        free(r);
-        return NULL;
-    }
-
-    r->ridx = -1;
-    return r;
-}
-
-void redisReaderFree(redisReader *r) {
-    if (r->reply != NULL && r->fn && r->fn->freeObject)
-        r->fn->freeObject(r->reply);
-    if (r->buf != NULL)
-        sdsfree(r->buf);
-    free(r);
-}
-
-int redisReaderFeed(redisReader *r, const char *buf, size_t len) {
-    sds newbuf;
-
-    /* Return early when this reader is in an erroneous state. */
-    if (r->err)
-        return REDIS_ERR;
-
-    /* Copy the provided buffer. */
-    if (buf != NULL && len >= 1) {
-        /* Destroy internal buffer when it is empty and is quite large. */
-        if (r->len == 0 && r->maxbuf != 0 && sdsavail(r->buf) > r->maxbuf) {
-            sdsfree(r->buf);
-            r->buf = sdsempty();
-            r->pos = 0;
-
-            /* r->buf should not be NULL since we just free'd a larger one. */
-            assert(r->buf != NULL);
-        }
-
-        newbuf = sdscatlen(r->buf,buf,len);
-        if (newbuf == NULL) {
-            __redisReaderSetErrorOOM(r);
-            return REDIS_ERR;
-        }
-
-        r->buf = newbuf;
-        r->len = sdslen(r->buf);
-    }
-
-    return REDIS_OK;
-}
-
-int redisReaderGetReply(redisReader *r, void **reply) {
-    /* Default target pointer to NULL. */
-    if (reply != NULL)
-        *reply = NULL;
-
-    /* Return early when this reader is in an erroneous state. */
-    if (r->err)
-        return REDIS_ERR;
-
-    /* When the buffer is empty, there will never be a reply. */
-    if (r->len == 0)
-        return REDIS_OK;
-
-    /* Set first item to process when the stack is empty. */
-    if (r->ridx == -1) {
-        r->rstack[0].type = -1;
-        r->rstack[0].elements = -1;
-        r->rstack[0].idx = -1;
-        r->rstack[0].obj = NULL;
-        r->rstack[0].parent = NULL;
-        r->rstack[0].privdata = r->privdata;
-        r->ridx = 0;
-    }
-
-    /* Process items in reply. */
-    while (r->ridx >= 0)
-        if (processItem(r) != REDIS_OK)
-            break;
-
-    /* Return ASAP when an error occurred. */
-    if (r->err)
-        return REDIS_ERR;
-
-    /* Discard part of the buffer when we've consumed at least 1k, to avoid
-     * doing unnecessary calls to memmove() in sds.c. */
-    if (r->pos >= 1024) {
-        sdsrange(r->buf,r->pos,-1);
-        r->pos = 0;
-        r->len = sdslen(r->buf);
-    }
-
-    /* Emit a reply when there is one. */
-    if (r->ridx == -1) {
-        if (reply != NULL)
-            *reply = r->reply;
-        r->reply = NULL;
-    }
-    return REDIS_OK;
-}
-
-/* Calculate the number of bytes needed to represent an integer as string. */
-static int intlen(int i) {
-    int len = 0;
-    if (i < 0) {
-        len++;
-        i = -i;
-    }
-    do {
-        len++;
-        i /= 10;
-    } while(i);
-    return len;
+/* Return the number of digits of 'v' when converted to string in radix 10.
+ * Implementation borrowed from link in redis/src/util.c:string2ll(). */
+static uint32_t countDigits(uint64_t v) {
+  uint32_t result = 1;
+  for (;;) {
+    if (v < 10) return result;
+    if (v < 100) return result + 1;
+    if (v < 1000) return result + 2;
+    if (v < 10000) return result + 3;
+    v /= 10000U;
+    result += 4;
+  }
 }

 /* Helper that calculates the bulk length given a certain string length. */
 static size_t bulklen(size_t len) {
-    return 1+intlen(len)+2+len+2;
+    return 1+countDigits(len)+2+len+2;
 }

 int redisvFormatCommand(char **target, const char *format, va_list ap) {
@@ -692,6 +216,7 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
    char **curargv = NULL, **newargv = NULL;
    int argc = 0;
    int totlen = 0;
+    int error_type = 0; /* 0 = no error; -1 = memory error; -2 = format error */
    int j;

    /* Abort if there is not target to set */
@@ -708,19 +233,19 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
            if (*c == ' ') {
                if (touched) {
                    newargv = realloc(curargv,sizeof(char*)*(argc+1));
-                    if (newargv == NULL) goto err;
+                    if (newargv == NULL) goto memory_err;
                    curargv = newargv;
                    curargv[argc++] = curarg;
                    totlen += bulklen(sdslen(curarg));

                    /* curarg is put in argv so it can be overwritten. */
                    curarg = sdsempty();
-                    if (curarg == NULL) goto err;
+                    if (curarg == NULL) goto memory_err;
                    touched = 0;
                }
            } else {
                newarg = sdscatlen(curarg,c,1);
-                if (newarg == NULL) goto err;
+                if (newarg == NULL) goto memory_err;
                curarg = newarg;
                touched = 1;
            }
@@ -751,17 +276,14 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
                /* Try to detect printf format */
                {
                    static const char intfmts[] = "diouxX";
+                    static const char flags[] = "#0-+ ";
                    char _format[16];
                    const char *_p = c+1;
                    size_t _l = 0;
                    va_list _cpy;

                    /* Flags */
-                    if (*_p != '\0' && *_p == '#') _p++;
-                    if (*_p != '\0' && *_p == '0') _p++;
-                    if (*_p != '\0' && *_p == '-') _p++;
-                    if (*_p != '\0' && *_p == ' ') _p++;
-                    if (*_p != '\0' && *_p == '+') _p++;
+                    while (*_p != '\0' && strchr(flags,*_p) != NULL) _p++;

                    /* Field width */
                    while (*_p != '\0' && isdigit(*_p)) _p++;
@@ -829,7 +351,7 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {

                fmt_invalid:
                    va_end(_cpy);
-                    goto err;
+                    goto format_err;

                fmt_valid:
                    _l = (_p+1)-c;
@@ -848,7 +370,7 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
                }
            }

-            if (newarg == NULL) goto err;
+            if (newarg == NULL) goto memory_err;
            curarg = newarg;

            touched = 1;
@@ -860,7 +382,7 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
    /* Add the last argument if needed */
    if (touched) {
        newargv = realloc(curargv,sizeof(char*)*(argc+1));
-        if (newargv == NULL) goto err;
+        if (newargv == NULL) goto memory_err;
        curargv = newargv;
        curargv[argc++] = curarg;
        totlen += bulklen(sdslen(curarg));
@@ -872,11 +394,11 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
    curarg = NULL;

    /* Add bytes needed to hold multi bulk count */
-    totlen += 1+intlen(argc)+2;
+    totlen += 1+countDigits(argc)+2;

    /* Build the command at protocol level */
    cmd = malloc(totlen+1);
-    if (cmd == NULL) goto err;
+    if (cmd == NULL) goto memory_err;

    pos = sprintf(cmd,"*%d\r\n",argc);
    for (j = 0; j < argc; j++) {
@@ -894,20 +416,29 @@ int redisvFormatCommand(char **target, const char *format, va_list ap) {
    *target = cmd;
    return totlen;

-err:
-    while(argc--)
-        sdsfree(curargv[argc]);
-    free(curargv);
+format_err:
+    error_type = -2;
+    goto cleanup;

-    if (curarg != NULL)
-        sdsfree(curarg);
+memory_err:
+    error_type = -1;
+    goto cleanup;
+
+cleanup:
+    if (curargv) {
+        while(argc--)
+            sdsfree(curargv[argc]);
+        free(curargv);
+    }
+
+    sdsfree(curarg);

    /* No need to check cmd since it is the last statement that can fail,
     * but do it anyway to be as defensive as possible. */
    if (cmd != NULL)
        free(cmd);

-    return -1;
+    return error_type;
 }

 /* Format a command according to the Redis protocol. This function
@@ -928,9 +459,69 @@ int redisFormatCommand(char **target, const char *format, ...) {
    va_start(ap,format);
    len = redisvFormatCommand(target,format,ap);
    va_end(ap);
+
+    /* The API says "-1" means bad result, but we now also return "-2" in some
+     * cases.  Force the return value to always be -1. */
+    if (len < 0)
+        len = -1;
+
    return len;
 }

+/* Format a command according to the Redis protocol using an sds string and
+ * sdscatfmt for the processing of arguments. This function takes the
+ * number of arguments, an array with arguments and an array with their
+ * lengths. If the latter is set to NULL, strlen will be used to compute the
+ * argument lengths.
+ */
+int redisFormatSdsCommandArgv(sds *target, int argc, const char **argv,
+                              const size_t *argvlen)
+{
+    sds cmd;
+    unsigned long long totlen;
+    int j;
+    size_t len;
+
+    /* Abort on a NULL target */
+    if (target == NULL)
+        return -1;
+
+    /* Calculate our total size */
+    totlen = 1+countDigits(argc)+2;
+    for (j = 0; j < argc; j++) {
+        len = argvlen ? argvlen[j] : strlen(argv[j]);
+        totlen += bulklen(len);
+    }
+
+    /* Use an SDS string for command construction */
+    cmd = sdsempty();
+    if (cmd == NULL)
+        return -1;
+
+    /* We already know how much storage we need */
+    cmd = sdsMakeRoomFor(cmd, totlen);
+    if (cmd == NULL)
+        return -1;
+
+    /* Construct command */
+    cmd = sdscatfmt(cmd, "*%i\r\n", argc);
+    for (j=0; j < argc; j++) {
+        len = argvlen ? argvlen[j] : strlen(argv[j]);
+        cmd = sdscatfmt(cmd, "$%u\r\n", len);
+        cmd = sdscatlen(cmd, argv[j], len);
+        cmd = sdscatlen(cmd, "\r\n", sizeof("\r\n")-1);
+    }
+
+    assert(sdslen(cmd)==totlen);
+
+    *target = cmd;
+    return totlen;
+}
+
+void redisFreeSdsCommand(sds cmd) {
+    sdsfree(cmd);
+}
+
 /* Format a command according to the Redis protocol. This function takes the
 * number of arguments, an array with arguments and an array with their
 * lengths. If the latter is set to NULL, strlen will be used to compute the
@@ -942,8 +533,12 @@ int redisFormatCommandArgv(char **target, int argc, const char **argv, const siz
    size_t len;
    int totlen, j;

+    /* Abort on a NULL target */
+    if (target == NULL)
+        return -1;
+
    /* Calculate number of bytes needed for the command */
-    totlen = 1+intlen(argc)+2;
+    totlen = 1+countDigits(argc)+2;
    for (j = 0; j < argc; j++) {
        len = argvlen ? argvlen[j] : strlen(argv[j]);
        totlen += bulklen(len);
@@ -970,6 +565,10 @@ int redisFormatCommandArgv(char **target, int argc, const char **argv, const siz
    return totlen;
 }

+void redisFreeCommand(char *cmd) {
+    free(cmd);
+}
+
 void __redisSetError(redisContext *c, int type, const char *str) {
    size_t len;

@@ -982,10 +581,14 @@ void __redisSetError(redisContext *c, int type, const char *str) {
    } else {
        /* Only REDIS_ERR_IO may lack a description! */
        assert(type == REDIS_ERR_IO);
-        strerror_r(errno,c->errstr,sizeof(c->errstr));
+        __redis_strerror_r(errno, c->errstr, sizeof(c->errstr));
    }
 }

+redisReader *redisReaderCreate(void) {
+    return redisReaderCreateWithFunctions(&defaultFunctions);
+}
+
 static redisContext *redisContextInit(void) {
    redisContext *c;

@@ -997,24 +600,72 @@ static redisContext *redisContextInit(void) {
    c->errstr[0] = '\0';
    c->obuf = sdsempty();
    c->reader = redisReaderCreate();
+    c->tcp.host = NULL;
+    c->tcp.source_addr = NULL;
+    c->unix_sock.path = NULL;
+    c->timeout = NULL;
+
+    if (c->obuf == NULL || c->reader == NULL) {
+        redisFree(c);
+        return NULL;
+    }
+
    return c;
 }

 void redisFree(redisContext *c) {
+    if (c == NULL)
+        return;
    if (c->fd > 0)
        close(c->fd);
    if (c->obuf != NULL)
        sdsfree(c->obuf);
    if (c->reader != NULL)
        redisReaderFree(c->reader);
+    if (c->tcp.host)
+        free(c->tcp.host);
+    if (c->tcp.source_addr)
+        free(c->tcp.source_addr);
+    if (c->unix_sock.path)
+        free(c->unix_sock.path);
+    if (c->timeout)
+        free(c->timeout);
    free(c);
 }

 int redisFreeKeepFd(redisContext *c) {
-	int fd = c->fd;
-	c->fd = -1;
-	redisFree(c);
-	return fd;
+    int fd = c->fd;
+    c->fd = -1;
+    redisFree(c);
+    return fd;
+}
+
+int redisReconnect(redisContext *c) {
+    c->err = 0;
+    memset(c->errstr, '\0', strlen(c->errstr));
+
+    if (c->fd > 0) {
+        close(c->fd);
+    }
+
+    sdsfree(c->obuf);
+    redisReaderFree(c->reader);
+
+    c->obuf = sdsempty();
+    c->reader = redisReaderCreate();
+
+    if (c->connection_type == REDIS_CONN_TCP) {
+        return redisContextConnectBindTcp(c, c->tcp.host, c->tcp.port,
+                c->timeout, c->tcp.source_addr);
+    } else if (c->connection_type == REDIS_CONN_UNIX) {
+        return redisContextConnectUnix(c, c->unix_sock.path, c->timeout);
+    } else {
+        /* Something bad happened here and shouldn't have. There isn't
+           enough information in the context to reconnect. */
+        __redisSetError(c,REDIS_ERR_OTHER,"Not enough information to reconnect");
+    }
+
+    return REDIS_ERR;
 }

 /* Connect to a Redis instance. On error the field error in the returned
@@ -1064,6 +715,15 @@ redisContext *redisConnectBindNonBlock(const char *ip, int port,
    return c;
 }

+redisContext *redisConnectBindNonBlockWithReuse(const char *ip, int port,
+                                                const char *source_addr) {
+    redisContext *c = redisContextInit();
+    c->flags &= ~REDIS_BLOCK;
+    c->flags |= REDIS_REUSEADDR;
+    redisContextConnectBindTcp(c,ip,port,NULL,source_addr);
+    return c;
+}
+
 redisContext *redisConnectUnix(const char *path) {
    redisContext *c;

@@ -1162,10 +822,10 @@ int redisBufferRead(redisContext *c) {
 /* Write the output buffer to the socket.
 *
 * Returns REDIS_OK when the buffer is empty, or (a part of) the buffer was
- * succesfully written to the socket. When the buffer is empty after the
+ * successfully written to the socket. When the buffer is empty after the
 * write operation, "done" is set to 1 (if given).
 *
- * Returns REDIS_ERR if an error occured trying to write and sets
+ * Returns REDIS_ERR if an error occurred trying to write and sets
 * c->errstr to hold the appropriate error string.
 */
 int redisBufferWrite(redisContext *c, int *done) {
@@ -1274,6 +934,9 @@ int redisvAppendCommand(redisContext *c, const char *format, va_list ap) {
    if (len == -1) {
        __redisSetError(c,REDIS_ERR_OOM,"Out of memory");
        return REDIS_ERR;
+    } else if (len == -2) {
+        __redisSetError(c,REDIS_ERR_OTHER,"Invalid format string");
+        return REDIS_ERR;
    }

    if (__redisAppendCommand(c,cmd,len) != REDIS_OK) {
@@ -1296,21 +959,21 @@ int redisAppendCommand(redisContext *c, const char *format, ...) {
 }

 int redisAppendCommandArgv(redisContext *c, int argc, const char **argv, const size_t *argvlen) {
-    char *cmd;
+    sds cmd;
    int len;

-    len = redisFormatCommandArgv(&cmd,argc,argv,argvlen);
+    len = redisFormatSdsCommandArgv(&cmd,argc,argv,argvlen);
    if (len == -1) {
        __redisSetError(c,REDIS_ERR_OOM,"Out of memory");
        return REDIS_ERR;
    }

    if (__redisAppendCommand(c,cmd,len) != REDIS_OK) {
-        free(cmd);
+        sdsfree(cmd);
        return REDIS_ERR;
    }

-    free(cmd);
+    sdsfree(cmd);
    return REDIS_OK;
 }

@@ -1321,7 +984,7 @@ int redisAppendCommandArgv(redisContext *c, int argc, const char **argv, const s
 * context is non-blocking, the "reply" pointer will not be used and the
 * command is simply appended to the write buffer.
 *
- * Returns the reply when a reply was succesfully retrieved. Returns NULL
+ * Returns the reply when a reply was successfully retrieved. Returns NULL
 * otherwise. When NULL is returned in a blocking context, the error field
 * in the context will be set.
 */
@@ -1,6 +1,8 @@
 /*
 * Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
- * Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
+ * Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
+ * Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
+ *                     Jan-Erik Rediger <janerik at fnordig dot com>
 *
 * All rights reserved.
 *
@@ -31,26 +33,16 @@

 #ifndef __HIREDIS_H
 #define __HIREDIS_H
-#include <stdio.h> /* for size_t */
+#include "read.h"
 #include <stdarg.h> /* for va_list */
 #include <sys/time.h> /* for struct timeval */
+#include <stdint.h> /* uintXX_t, etc */
+#include "sds.h" /* for sds */

 #define HIREDIS_MAJOR 0
-#define HIREDIS_MINOR 11
-#define HIREDIS_PATCH 0
-
-#define REDIS_ERR -1
-#define REDIS_OK 0
-
-/* When an error occurs, the err flag in a context is set to hold the type of
- * error that occured. REDIS_ERR_IO means there was an I/O error and you
- * should use the "errno" variable to find out what is wrong.
- * For other values, the "errstr" field will hold a description. */
-#define REDIS_ERR_IO 1 /* Error in read or write */
-#define REDIS_ERR_EOF 3 /* End of file */
-#define REDIS_ERR_PROTOCOL 4 /* Protocol error */
-#define REDIS_ERR_OOM 5 /* Out of memory */
-#define REDIS_ERR_OTHER 2 /* Everything else... */
+#define HIREDIS_MINOR 13
+#define HIREDIS_PATCH 3
+#define HIREDIS_SONAME 0.13

 /* Connection type can be blocking or non-blocking and is set in the
 * least significant bit of the flags field in redisContext. */
@@ -79,17 +71,39 @@
 /* Flag that is set when monitor mode is active */
 #define REDIS_MONITORING 0x40

-#define REDIS_REPLY_STRING 1
-#define REDIS_REPLY_ARRAY 2
-#define REDIS_REPLY_INTEGER 3
-#define REDIS_REPLY_NIL 4
-#define REDIS_REPLY_STATUS 5
-#define REDIS_REPLY_ERROR 6
-
-#define REDIS_READER_MAX_BUF (1024*16)  /* Default max unused reader buffer. */
+/* Flag that is set when we should set SO_REUSEADDR before calling bind() */
+#define REDIS_REUSEADDR 0x80

 #define REDIS_KEEPALIVE_INTERVAL 15 /* seconds */

+/* number of times we retry to connect in the case of EADDRNOTAVAIL and
+ * SO_REUSEADDR is being used. */
+#define REDIS_CONNECT_RETRIES  10
+
+/* strerror_r has two completely different prototypes and behaviors
+ * depending on system issues, so we need to operate on the error buffer
+ * differently depending on which strerror_r we're using. */
+#ifndef _GNU_SOURCE
+/* "regular" POSIX strerror_r that does the right thing. */
+#define __redis_strerror_r(errno, buf, len)                                    \
+    do {                                                                       \
+        strerror_r((errno), (buf), (len));                                     \
+    } while (0)
+#else
+/* "bad" GNU strerror_r we need to clean up after. */
+#define __redis_strerror_r(errno, buf, len)                                    \
+    do {                                                                       \
+        char *err_str = strerror_r((errno), (buf), (len));                     \
+        /* If return value _isn't_ the start of the buffer we passed in,       \
+         * then GNU strerror_r returned an internal static buffer and we       \
+         * need to copy the result into our private buffer. */                 \
+        if (err_str != (buf)) {                                                \
+            strncpy((buf), err_str, ((len) - 1));                              \
+            buf[(len)-1] = '\0';                                               \
+        }                                                                      \
+    } while (0)
+#endif
+
 #ifdef __cplusplus
 extern "C" {
 #endif
@@ -98,61 +112,13 @@ extern "C" {
 typedef struct redisReply {
    int type; /* REDIS_REPLY_* */
    long long integer; /* The integer when type is REDIS_REPLY_INTEGER */
-    int len; /* Length of string */
+    size_t len; /* Length of string */
    char *str; /* Used for both REDIS_REPLY_ERROR and REDIS_REPLY_STRING */
    size_t elements; /* number of elements, for REDIS_REPLY_ARRAY */
    struct redisReply **element; /* elements vector for REDIS_REPLY_ARRAY */
 } redisReply;

-typedef struct redisReadTask {
-    int type;
-    int elements; /* number of elements in multibulk container */
-    int idx; /* index in parent (array) object */
-    void *obj; /* holds user-generated value for a read task */
-    struct redisReadTask *parent; /* parent task */
-    void *privdata; /* user-settable arbitrary field */
-} redisReadTask;
-
-typedef struct redisReplyObjectFunctions {
-    void *(*createString)(const redisReadTask*, char*, size_t);
-    void *(*createArray)(const redisReadTask*, int);
-    void *(*createInteger)(const redisReadTask*, long long);
-    void *(*createNil)(const redisReadTask*);
-    void (*freeObject)(void*);
-} redisReplyObjectFunctions;
-
-/* State for the protocol parser */
-typedef struct redisReader {
-    int err; /* Error flags, 0 when there is no error */
-    char errstr[128]; /* String representation of error when applicable */
-
-    char *buf; /* Read buffer */
-    size_t pos; /* Buffer cursor */
-    size_t len; /* Buffer length */
-    size_t maxbuf; /* Max length of unused buffer */
-
-    redisReadTask rstack[9];
-    int ridx; /* Index of current read task */
-    void *reply; /* Temporary reply pointer */
-
-    redisReplyObjectFunctions *fn;
-    void *privdata;
-} redisReader;
-
-/* Public API for the protocol parser. */
 redisReader *redisReaderCreate(void);
-void redisReaderFree(redisReader *r);
-int redisReaderFeed(redisReader *r, const char *buf, size_t len);
-int redisReaderGetReply(redisReader *r, void **reply);
-
-/* Backwards compatibility, can be removed on big version bump. */
-#define redisReplyReaderCreate redisReaderCreate
-#define redisReplyReaderFree redisReaderFree
-#define redisReplyReaderFeed redisReaderFeed
-#define redisReplyReaderGetReply redisReaderGetReply
-#define redisReplyReaderSetPrivdata(_r, _p) (int)(((redisReader*)(_r))->privdata = (_p))
-#define redisReplyReaderGetObject(_r) (((redisReader*)(_r))->reply)
-#define redisReplyReaderGetError(_r) (((redisReader*)(_r))->errstr)

 /* Function to free the reply objects hiredis returns by default. */
 void freeReplyObject(void *reply);
@@ -161,6 +127,14 @@ void freeReplyObject(void *reply);
 int redisvFormatCommand(char **target, const char *format, va_list ap);
 int redisFormatCommand(char **target, const char *format, ...);
 int redisFormatCommandArgv(char **target, int argc, const char **argv, const size_t *argvlen);
+int redisFormatSdsCommandArgv(sds *target, int argc, const char ** argv, const size_t *argvlen);
+void redisFreeCommand(char *cmd);
+void redisFreeSdsCommand(sds cmd);
+
+enum redisConnectionType {
+    REDIS_CONN_TCP,
+    REDIS_CONN_UNIX
+};

 /* Context for a connection to Redis */
 typedef struct redisContext {
@@ -170,16 +144,45 @@ typedef struct redisContext {
    int flags;
    char *obuf; /* Write buffer */
    redisReader *reader; /* Protocol reader */
+
+    enum redisConnectionType connection_type;
+    struct timeval *timeout;
+
+    struct {
+        char *host;
+        char *source_addr;
+        int port;
+    } tcp;
+
+    struct {
+        char *path;
+    } unix_sock;
+
 } redisContext;

 redisContext *redisConnect(const char *ip, int port);
 redisContext *redisConnectWithTimeout(const char *ip, int port, const struct timeval tv);
 redisContext *redisConnectNonBlock(const char *ip, int port);
-redisContext *redisConnectBindNonBlock(const char *ip, int port, const char *source_addr);
+redisContext *redisConnectBindNonBlock(const char *ip, int port,
+                                       const char *source_addr);
+redisContext *redisConnectBindNonBlockWithReuse(const char *ip, int port,
+                                                const char *source_addr);
 redisContext *redisConnectUnix(const char *path);
 redisContext *redisConnectUnixWithTimeout(const char *path, const struct timeval tv);
 redisContext *redisConnectUnixNonBlock(const char *path);
 redisContext *redisConnectFd(int fd);
+
+/**
+ * Reconnect the given context using the saved information.
+ *
+ * This re-uses the exact same connect options as in the initial connection.
+ * host, ip (or path), timeout and bind address are reused,
+ * flags are used unmodified from the existing context.
+ *
+ * Returns REDIS_OK on successful connect or REDIS_ERR otherwise.
+ */
+int redisReconnect(redisContext *c);
+
 int redisSetTimeout(redisContext *c, const struct timeval tv);
 int redisEnableKeepAlive(redisContext *c);
 void redisFree(redisContext *c);
@@ -1,7 +1,9 @@
 /* Extracted from anet.c to work properly with Hiredis error reporting.
 *
- * Copyright (c) 2006-2011, Salvatore Sanfilippo <antirez at gmail dot com>
- * Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
+ * Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
+ * Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
+ * Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
+ *                     Jan-Erik Rediger <janerik at fnordig dot com>
 *
 * All rights reserved.
 *
@@ -47,6 +49,7 @@
 #include <stdio.h>
 #include <poll.h>
 #include <limits.h>
+#include <stdlib.h>

 #include "net.h"
 #include "sds.h"
@@ -67,7 +70,7 @@ static void __redisSetErrorFromErrno(redisContext *c, int type, const char *pref

    if (prefix != NULL)
        len = snprintf(buf,sizeof(buf),"%s: ",prefix);
-    strerror_r(errno,buf+len,sizeof(buf)-len);
+    __redis_strerror_r(errno, (char *)(buf + len), sizeof(buf) - len);
    __redisSetError(c,type,buf);
 }

@@ -138,7 +141,7 @@ int redisKeepAlive(redisContext *c, int interval) {
        return REDIS_ERR;
    }
 #else
-#ifndef __sun
+#if defined(__GLIBC__) && !defined(__FreeBSD_kernel__)
    val = interval;
    if (setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE, &val, sizeof(val)) < 0) {
        __redisSetError(c,REDIS_ERR_OTHER,strerror(errno));
@@ -175,19 +178,15 @@ static int redisSetTcpNoDelay(redisContext *c) {

 #define __MAX_MSEC (((LONG_MAX) - 999) / 1000)

-static int redisContextWaitReady(redisContext *c, const struct timeval *timeout) {
-    struct pollfd   wfd[1];
-    long msec;
-
-    msec          = -1;
-    wfd[0].fd     = c->fd;
-    wfd[0].events = POLLOUT;
+static int redisContextTimeoutMsec(redisContext *c, long *result)
+{
+    const struct timeval *timeout = c->timeout;
+    long msec = -1;

    /* Only use timeout when not NULL. */
    if (timeout != NULL) {
        if (timeout->tv_usec > 1000000 || timeout->tv_sec > __MAX_MSEC) {
-            __redisSetErrorFromErrno(c, REDIS_ERR_IO, NULL);
-            redisContextCloseFd(c);
+            *result = msec;
            return REDIS_ERR;
        }

@@ -198,6 +197,16 @@ static int redisContextWaitReady(redisContext *c, const struct timeval *timeout)
        }
    }

+    *result = msec;
+    return REDIS_OK;
+}
+
+static int redisContextWaitReady(redisContext *c, long msec) {
+    struct pollfd   wfd[1];
+
+    wfd[0].fd     = c->fd;
+    wfd[0].events = POLLOUT;
+
    if (errno == EINPROGRESS) {
        int res;

@@ -256,10 +265,57 @@ int redisContextSetTimeout(redisContext *c, const struct timeval tv) {
 static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
                                   const struct timeval *timeout,
                                   const char *source_addr) {
-    int s, rv;
+    int s, rv, n;
    char _port[6];  /* strlen("65535"); */
    struct addrinfo hints, *servinfo, *bservinfo, *p, *b;
    int blocking = (c->flags & REDIS_BLOCK);
+    int reuseaddr = (c->flags & REDIS_REUSEADDR);
+    int reuses = 0;
+    long timeout_msec = -1;
+
+    servinfo = NULL;
+    c->connection_type = REDIS_CONN_TCP;
+    c->tcp.port = port;
+
+    /* We need to take possession of the passed parameters
+     * to make them reusable for a reconnect.
+     * We also carefully check we don't free data we already own,
+     * as in the case of the reconnect method.
+     *
+     * This is a bit ugly, but atleast it works and doesn't leak memory.
+     **/
+    if (c->tcp.host != addr) {
+        if (c->tcp.host)
+            free(c->tcp.host);
+
+        c->tcp.host = strdup(addr);
+    }
+
+    if (timeout) {
+        if (c->timeout != timeout) {
+            if (c->timeout == NULL)
+                c->timeout = malloc(sizeof(struct timeval));
+
+            memcpy(c->timeout, timeout, sizeof(struct timeval));
+        }
+    } else {
+        if (c->timeout)
+            free(c->timeout);
+        c->timeout = NULL;
+    }
+
+    if (redisContextTimeoutMsec(c, &timeout_msec) != REDIS_OK) {
+        __redisSetError(c, REDIS_ERR_IO, "Invalid timeout specified");
+        goto error;
+    }
+
+    if (source_addr == NULL) {
+        free(c->tcp.source_addr);
+        c->tcp.source_addr = NULL;
+    } else if (c->tcp.source_addr != source_addr) {
+        free(c->tcp.source_addr);
+        c->tcp.source_addr = strdup(source_addr);
+    }

    snprintf(_port, 6, "%d", port);
    memset(&hints,0,sizeof(hints));
@@ -271,7 +327,7 @@ static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
     * as this would add latency to every connect. Otherwise a more sensible
     * route could be: Use IPv6 if both addresses are available and there is IPv6
     * connectivity. */
-    if ((rv = getaddrinfo(addr,_port,&hints,&servinfo)) != 0) {
+    if ((rv = getaddrinfo(c->tcp.host,_port,&hints,&servinfo)) != 0) {
         hints.ai_family = AF_INET6;
         if ((rv = getaddrinfo(addr,_port,&hints,&servinfo)) != 0) {
            __redisSetError(c,REDIS_ERR_OTHER,gai_strerror(rv));
@@ -279,21 +335,31 @@ static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
        }
    }
    for (p = servinfo; p != NULL; p = p->ai_next) {
+addrretry:
        if ((s = socket(p->ai_family,p->ai_socktype,p->ai_protocol)) == -1)
            continue;

        c->fd = s;
        if (redisSetBlocking(c,0) != REDIS_OK)
            goto error;
-        if (source_addr) {
+        if (c->tcp.source_addr) {
            int bound = 0;
            /* Using getaddrinfo saves us from self-determining IPv4 vs IPv6 */
-            if ((rv = getaddrinfo(source_addr, NULL, &hints, &bservinfo)) != 0) {
+            if ((rv = getaddrinfo(c->tcp.source_addr, NULL, &hints, &bservinfo)) != 0) {
                char buf[128];
                snprintf(buf,sizeof(buf),"Can't get addr: %s",gai_strerror(rv));
                __redisSetError(c,REDIS_ERR_OTHER,buf);
                goto error;
            }
+
+            if (reuseaddr) {
+                n = 1;
+                if (setsockopt(s, SOL_SOCKET, SO_REUSEADDR, (char*) &n,
+                               sizeof(n)) < 0) {
+                    goto error;
+                }
+            }
+
            for (b = bservinfo; b != NULL; b = b->ai_next) {
                if (bind(s,b->ai_addr,b->ai_addrlen) != -1) {
                    bound = 1;
@@ -314,8 +380,15 @@ static int _redisContextConnectTcp(redisContext *c, const char *addr, int port,
                continue;
            } else if (errno == EINPROGRESS && !blocking) {
                /* This is ok. */
+            } else if (errno == EADDRNOTAVAIL && reuseaddr) {
+                if (++reuses >= REDIS_CONNECT_RETRIES) {
+                    goto error;
+                } else {
+                    redisContextCloseFd(c);
+                    goto addrretry;
+                }
            } else {
-                if (redisContextWaitReady(c,timeout) != REDIS_OK)
+                if (redisContextWaitReady(c,timeout_msec) != REDIS_OK)
                    goto error;
            }
        }
@@ -356,19 +429,40 @@ int redisContextConnectBindTcp(redisContext *c, const char *addr, int port,
 int redisContextConnectUnix(redisContext *c, const char *path, const struct timeval *timeout) {
    int blocking = (c->flags & REDIS_BLOCK);
    struct sockaddr_un sa;
+    long timeout_msec = -1;

    if (redisCreateSocket(c,AF_LOCAL) < 0)
        return REDIS_ERR;
    if (redisSetBlocking(c,0) != REDIS_OK)
        return REDIS_ERR;

+    c->connection_type = REDIS_CONN_UNIX;
+    if (c->unix_sock.path != path)
+        c->unix_sock.path = strdup(path);
+
+    if (timeout) {
+        if (c->timeout != timeout) {
+            if (c->timeout == NULL)
+                c->timeout = malloc(sizeof(struct timeval));
+
+            memcpy(c->timeout, timeout, sizeof(struct timeval));
+        }
+    } else {
+        if (c->timeout)
+            free(c->timeout);
+        c->timeout = NULL;
+    }
+
+    if (redisContextTimeoutMsec(c,&timeout_msec) != REDIS_OK)
+        return REDIS_ERR;
+
    sa.sun_family = AF_LOCAL;
    strncpy(sa.sun_path,path,sizeof(sa.sun_path)-1);
    if (connect(c->fd, (struct sockaddr*)&sa, sizeof(sa)) == -1) {
        if (errno == EINPROGRESS && !blocking) {
            /* This is ok. */
        } else {
-            if (redisContextWaitReady(c,timeout) != REDIS_OK)
+            if (redisContextWaitReady(c,timeout_msec) != REDIS_OK)
                return REDIS_ERR;
        }
    }
@@ -1,7 +1,9 @@
 /* Extracted from anet.c to work properly with Hiredis error reporting.
 *
- * Copyright (c) 2006-2011, Salvatore Sanfilippo <antirez at gmail dot com>
- * Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
+ * Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
+ * Copyright (c) 2010-2014, Pieter Noordhuis <pcnoordhuis at gmail dot com>
+ * Copyright (c) 2015, Matt Stancliff <matt at genges dot com>,
+ *                     Jan-Erik Rediger <janerik at fnordig dot com>
 *
 * All rights reserved.
 *
@@ -35,7 +37,7 @@

 #include "hiredis.h"

-#if defined(__sun) || defined(_AIX)
+#if defined(__sun)
 #define AF_LOCAL AF_UNIX
 #endif

@@ -0,0 +1,525 @@
+/*
+ * Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
+ * Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+
+#include "fmacros.h"
+#include <string.h>
+#include <stdlib.h>
+#ifndef _MSC_VER
+#include <unistd.h>
+#endif
+#include <assert.h>
+#include <errno.h>
+#include <ctype.h>
+
+#include "read.h"
+#include "sds.h"
+
+static void __redisReaderSetError(redisReader *r, int type, const char *str) {
+    size_t len;
+
+    if (r->reply != NULL && r->fn && r->fn->freeObject) {
+        r->fn->freeObject(r->reply);
+        r->reply = NULL;
+    }
+
+    /* Clear input buffer on errors. */
+    if (r->buf != NULL) {
+        sdsfree(r->buf);
+        r->buf = NULL;
+        r->pos = r->len = 0;
+    }
+
+    /* Reset task stack. */
+    r->ridx = -1;
+
+    /* Set error. */
+    r->err = type;
+    len = strlen(str);
+    len = len < (sizeof(r->errstr)-1) ? len : (sizeof(r->errstr)-1);
+    memcpy(r->errstr,str,len);
+    r->errstr[len] = '\0';
+}
+
+static size_t chrtos(char *buf, size_t size, char byte) {
+    size_t len = 0;
+
+    switch(byte) {
+    case '\\':
+    case '"':
+        len = snprintf(buf,size,"\"\\%c\"",byte);
+        break;
+    case '\n': len = snprintf(buf,size,"\"\\n\""); break;
+    case '\r': len = snprintf(buf,size,"\"\\r\""); break;
+    case '\t': len = snprintf(buf,size,"\"\\t\""); break;
+    case '\a': len = snprintf(buf,size,"\"\\a\""); break;
+    case '\b': len = snprintf(buf,size,"\"\\b\""); break;
+    default:
+        if (isprint(byte))
+            len = snprintf(buf,size,"\"%c\"",byte);
+        else
+            len = snprintf(buf,size,"\"\\x%02x\"",(unsigned char)byte);
+        break;
+    }
+
+    return len;
+}
+
+static void __redisReaderSetErrorProtocolByte(redisReader *r, char byte) {
+    char cbuf[8], sbuf[128];
+
+    chrtos(cbuf,sizeof(cbuf),byte);
+    snprintf(sbuf,sizeof(sbuf),
+        "Protocol error, got %s as reply type byte", cbuf);
+    __redisReaderSetError(r,REDIS_ERR_PROTOCOL,sbuf);
+}
+
+static void __redisReaderSetErrorOOM(redisReader *r) {
+    __redisReaderSetError(r,REDIS_ERR_OOM,"Out of memory");
+}
+
+static char *readBytes(redisReader *r, unsigned int bytes) {
+    char *p;
+    if (r->len-r->pos >= bytes) {
+        p = r->buf+r->pos;
+        r->pos += bytes;
+        return p;
+    }
+    return NULL;
+}
+
+/* Find pointer to \r\n. */
+static char *seekNewline(char *s, size_t len) {
+    int pos = 0;
+    int _len = len-1;
+
+    /* Position should be < len-1 because the character at "pos" should be
+     * followed by a \n. Note that strchr cannot be used because it doesn't
+     * allow to search a limited length and the buffer that is being searched
+     * might not have a trailing NULL character. */
+    while (pos < _len) {
+        while(pos < _len && s[pos] != '\r') pos++;
+        if (pos==_len) {
+            /* Not found. */
+            return NULL;
+        } else {
+            if (s[pos+1] == '\n') {
+                /* Found. */
+                return s+pos;
+            } else {
+                /* Continue searching. */
+                pos++;
+            }
+        }
+    }
+    return NULL;
+}
+
+/* Read a long long value starting at *s, under the assumption that it will be
+ * terminated by \r\n. Ambiguously returns -1 for unexpected input. */
+static long long readLongLong(char *s) {
+    long long v = 0;
+    int dec, mult = 1;
+    char c;
+
+    if (*s == '-') {
+        mult = -1;
+        s++;
+    } else if (*s == '+') {
+        mult = 1;
+        s++;
+    }
+
+    while ((c = *(s++)) != '\r') {
+        dec = c - '0';
+        if (dec >= 0 && dec < 10) {
+            v *= 10;
+            v += dec;
+        } else {
+            /* Should not happen... */
+            return -1;
+        }
+    }
+
+    return mult*v;
+}
+
+static char *readLine(redisReader *r, int *_len) {
+    char *p, *s;
+    int len;
+
+    p = r->buf+r->pos;
+    s = seekNewline(p,(r->len-r->pos));
+    if (s != NULL) {
+        len = s-(r->buf+r->pos);
+        r->pos += len+2; /* skip \r\n */
+        if (_len) *_len = len;
+        return p;
+    }
+    return NULL;
+}
+
+static void moveToNextTask(redisReader *r) {
+    redisReadTask *cur, *prv;
+    while (r->ridx >= 0) {
+        /* Return a.s.a.p. when the stack is now empty. */
+        if (r->ridx == 0) {
+            r->ridx--;
+            return;
+        }
+
+        cur = &(r->rstack[r->ridx]);
+        prv = &(r->rstack[r->ridx-1]);
+        assert(prv->type == REDIS_REPLY_ARRAY);
+        if (cur->idx == prv->elements-1) {
+            r->ridx--;
+        } else {
+            /* Reset the type because the next item can be anything */
+            assert(cur->idx < prv->elements);
+            cur->type = -1;
+            cur->elements = -1;
+            cur->idx++;
+            return;
+        }
+    }
+}
+
+static int processLineItem(redisReader *r) {
+    redisReadTask *cur = &(r->rstack[r->ridx]);
+    void *obj;
+    char *p;
+    int len;
+
+    if ((p = readLine(r,&len)) != NULL) {
+        if (cur->type == REDIS_REPLY_INTEGER) {
+            if (r->fn && r->fn->createInteger)
+                obj = r->fn->createInteger(cur,readLongLong(p));
+            else
+                obj = (void*)REDIS_REPLY_INTEGER;
+        } else {
+            /* Type will be error or status. */
+            if (r->fn && r->fn->createString)
+                obj = r->fn->createString(cur,p,len);
+            else
+                obj = (void*)(size_t)(cur->type);
+        }
+
+        if (obj == NULL) {
+            __redisReaderSetErrorOOM(r);
+            return REDIS_ERR;
+        }
+
+        /* Set reply if this is the root object. */
+        if (r->ridx == 0) r->reply = obj;
+        moveToNextTask(r);
+        return REDIS_OK;
+    }
+
+    return REDIS_ERR;
+}
+
+static int processBulkItem(redisReader *r) {
+    redisReadTask *cur = &(r->rstack[r->ridx]);
+    void *obj = NULL;
+    char *p, *s;
+    long len;
+    unsigned long bytelen;
+    int success = 0;
+
+    p = r->buf+r->pos;
+    s = seekNewline(p,r->len-r->pos);
+    if (s != NULL) {
+        p = r->buf+r->pos;
+        bytelen = s-(r->buf+r->pos)+2; /* include \r\n */
+        len = readLongLong(p);
+
+        if (len < 0) {
+            /* The nil object can always be created. */
+            if (r->fn && r->fn->createNil)
+                obj = r->fn->createNil(cur);
+            else
+                obj = (void*)REDIS_REPLY_NIL;
+            success = 1;
+        } else {
+            /* Only continue when the buffer contains the entire bulk item. */
+            bytelen += len+2; /* include \r\n */
+            if (r->pos+bytelen <= r->len) {
+                if (r->fn && r->fn->createString)
+                    obj = r->fn->createString(cur,s+2,len);
+                else
+                    obj = (void*)REDIS_REPLY_STRING;
+                success = 1;
+            }
+        }
+
+        /* Proceed when obj was created. */
+        if (success) {
+            if (obj == NULL) {
+                __redisReaderSetErrorOOM(r);
+                return REDIS_ERR;
+            }
+
+            r->pos += bytelen;
+
+            /* Set reply if this is the root object. */
+            if (r->ridx == 0) r->reply = obj;
+            moveToNextTask(r);
+            return REDIS_OK;
+        }
+    }
+
+    return REDIS_ERR;
+}
+
+static int processMultiBulkItem(redisReader *r) {
+    redisReadTask *cur = &(r->rstack[r->ridx]);
+    void *obj;
+    char *p;
+    long elements;
+    int root = 0;
+
+    /* Set error for nested multi bulks with depth > 7 */
+    if (r->ridx == 8) {
+        __redisReaderSetError(r,REDIS_ERR_PROTOCOL,
+            "No support for nested multi bulk replies with depth > 7");
+        return REDIS_ERR;
+    }
+
+    if ((p = readLine(r,NULL)) != NULL) {
+        elements = readLongLong(p);
+        root = (r->ridx == 0);
+
+        if (elements == -1) {
+            if (r->fn && r->fn->createNil)
+                obj = r->fn->createNil(cur);
+            else
+                obj = (void*)REDIS_REPLY_NIL;
+
+            if (obj == NULL) {
+                __redisReaderSetErrorOOM(r);
+                return REDIS_ERR;
+            }
+
+            moveToNextTask(r);
+        } else {
+            if (r->fn && r->fn->createArray)
+                obj = r->fn->createArray(cur,elements);
+            else
+                obj = (void*)REDIS_REPLY_ARRAY;
+
+            if (obj == NULL) {
+                __redisReaderSetErrorOOM(r);
+                return REDIS_ERR;
+            }
+
+            /* Modify task stack when there are more than 0 elements. */
+            if (elements > 0) {
+                cur->elements = elements;
+                cur->obj = obj;
+                r->ridx++;
+                r->rstack[r->ridx].type = -1;
+                r->rstack[r->ridx].elements = -1;
+                r->rstack[r->ridx].idx = 0;
+                r->rstack[r->ridx].obj = NULL;
+                r->rstack[r->ridx].parent = cur;
+                r->rstack[r->ridx].privdata = r->privdata;
+            } else {
+                moveToNextTask(r);
+            }
+        }
+
+        /* Set reply if this is the root object. */
+        if (root) r->reply = obj;
+        return REDIS_OK;
+    }
+
+    return REDIS_ERR;
+}
+
+static int processItem(redisReader *r) {
+    redisReadTask *cur = &(r->rstack[r->ridx]);
+    char *p;
+
+    /* check if we need to read type */
+    if (cur->type < 0) {
+        if ((p = readBytes(r,1)) != NULL) {
+            switch (p[0]) {
+            case '-':
+                cur->type = REDIS_REPLY_ERROR;
+                break;
+            case '+':
+                cur->type = REDIS_REPLY_STATUS;
+                break;
+            case ':':
+                cur->type = REDIS_REPLY_INTEGER;
+                break;
+            case '$':
+                cur->type = REDIS_REPLY_STRING;
+                break;
+            case '*':
+                cur->type = REDIS_REPLY_ARRAY;
+                break;
+            default:
+                __redisReaderSetErrorProtocolByte(r,*p);
+                return REDIS_ERR;
+            }
+        } else {
+            /* could not consume 1 byte */
+            return REDIS_ERR;
+        }
+    }
+
+    /* process typed item */
+    switch(cur->type) {
+    case REDIS_REPLY_ERROR:
+    case REDIS_REPLY_STATUS:
+    case REDIS_REPLY_INTEGER:
+        return processLineItem(r);
+    case REDIS_REPLY_STRING:
+        return processBulkItem(r);
+    case REDIS_REPLY_ARRAY:
+        return processMultiBulkItem(r);
+    default:
+        assert(NULL);
+        return REDIS_ERR; /* Avoid warning. */
+    }
+}
+
+redisReader *redisReaderCreateWithFunctions(redisReplyObjectFunctions *fn) {
+    redisReader *r;
+
+    r = calloc(sizeof(redisReader),1);
+    if (r == NULL)
+        return NULL;
+
+    r->err = 0;
+    r->errstr[0] = '\0';
+    r->fn = fn;
+    r->buf = sdsempty();
+    r->maxbuf = REDIS_READER_MAX_BUF;
+    if (r->buf == NULL) {
+        free(r);
+        return NULL;
+    }
+
+    r->ridx = -1;
+    return r;
+}
+
+void redisReaderFree(redisReader *r) {
+    if (r->reply != NULL && r->fn && r->fn->freeObject)
+        r->fn->freeObject(r->reply);
+    if (r->buf != NULL)
+        sdsfree(r->buf);
+    free(r);
+}
+
+int redisReaderFeed(redisReader *r, const char *buf, size_t len) {
+    sds newbuf;
+
+    /* Return early when this reader is in an erroneous state. */
+    if (r->err)
+        return REDIS_ERR;
+
+    /* Copy the provided buffer. */
+    if (buf != NULL && len >= 1) {
+        /* Destroy internal buffer when it is empty and is quite large. */
+        if (r->len == 0 && r->maxbuf != 0 && sdsavail(r->buf) > r->maxbuf) {
+            sdsfree(r->buf);
+            r->buf = sdsempty();
+            r->pos = 0;
+
+            /* r->buf should not be NULL since we just free'd a larger one. */
+            assert(r->buf != NULL);
+        }
+
+        newbuf = sdscatlen(r->buf,buf,len);
+        if (newbuf == NULL) {
+            __redisReaderSetErrorOOM(r);
+            return REDIS_ERR;
+        }
+
+        r->buf = newbuf;
+        r->len = sdslen(r->buf);
+    }
+
+    return REDIS_OK;
+}
+
+int redisReaderGetReply(redisReader *r, void **reply) {
+    /* Default target pointer to NULL. */
+    if (reply != NULL)
+        *reply = NULL;
+
+    /* Return early when this reader is in an erroneous state. */
+    if (r->err)
+        return REDIS_ERR;
+
+    /* When the buffer is empty, there will never be a reply. */
+    if (r->len == 0)
+        return REDIS_OK;
+
+    /* Set first item to process when the stack is empty. */
+    if (r->ridx == -1) {
+        r->rstack[0].type = -1;
+        r->rstack[0].elements = -1;
+        r->rstack[0].idx = -1;
+        r->rstack[0].obj = NULL;
+        r->rstack[0].parent = NULL;
+        r->rstack[0].privdata = r->privdata;
+        r->ridx = 0;
+    }
+
+    /* Process items in reply. */
+    while (r->ridx >= 0)
+        if (processItem(r) != REDIS_OK)
+            break;
+
+    /* Return ASAP when an error occurred. */
+    if (r->err)
+        return REDIS_ERR;
+
+    /* Discard part of the buffer when we've consumed at least 1k, to avoid
+     * doing unnecessary calls to memmove() in sds.c. */
+    if (r->pos >= 1024) {
+        sdsrange(r->buf,r->pos,-1);
+        r->pos = 0;
+        r->len = sdslen(r->buf);
+    }
+
+    /* Emit a reply when there is one. */
+    if (r->ridx == -1) {
+        if (reply != NULL)
+            *reply = r->reply;
+        r->reply = NULL;
+    }
+    return REDIS_OK;
+}
@@ -0,0 +1,111 @@
+/*
+ * Copyright (c) 2009-2011, Salvatore Sanfilippo <antirez at gmail dot com>
+ * Copyright (c) 2010-2011, Pieter Noordhuis <pcnoordhuis at gmail dot com>
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+
+#ifndef __HIREDIS_READ_H
+#define __HIREDIS_READ_H
+#include <stdio.h> /* for size_t */
+
+#define REDIS_ERR -1
+#define REDIS_OK 0
+
+/* When an error occurs, the err flag in a context is set to hold the type of
+ * error that occurred. REDIS_ERR_IO means there was an I/O error and you
+ * should use the "errno" variable to find out what is wrong.
+ * For other values, the "errstr" field will hold a description. */
+#define REDIS_ERR_IO 1 /* Error in read or write */
+#define REDIS_ERR_EOF 3 /* End of file */
+#define REDIS_ERR_PROTOCOL 4 /* Protocol error */
+#define REDIS_ERR_OOM 5 /* Out of memory */
+#define REDIS_ERR_OTHER 2 /* Everything else... */
+
+#define REDIS_REPLY_STRING 1
+#define REDIS_REPLY_ARRAY 2
+#define REDIS_REPLY_INTEGER 3
+#define REDIS_REPLY_NIL 4
+#define REDIS_REPLY_STATUS 5
+#define REDIS_REPLY_ERROR 6
+
+#define REDIS_READER_MAX_BUF (1024*16)  /* Default max unused reader buffer. */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+typedef struct redisReadTask {
+    int type;
+    int elements; /* number of elements in multibulk container */
+    int idx; /* index in parent (array) object */
+    void *obj; /* holds user-generated value for a read task */
+    struct redisReadTask *parent; /* parent task */
+    void *privdata; /* user-settable arbitrary field */
+} redisReadTask;
+
+typedef struct redisReplyObjectFunctions {
+    void *(*createString)(const redisReadTask*, char*, size_t);
+    void *(*createArray)(const redisReadTask*, int);
+    void *(*createInteger)(const redisReadTask*, long long);
+    void *(*createNil)(const redisReadTask*);
+    void (*freeObject)(void*);
+} redisReplyObjectFunctions;
+
+typedef struct redisReader {
+    int err; /* Error flags, 0 when there is no error */
+    char errstr[128]; /* String representation of error when applicable */
+
+    char *buf; /* Read buffer */
+    size_t pos; /* Buffer cursor */
+    size_t len; /* Buffer length */
+    size_t maxbuf; /* Max length of unused buffer */
+
+    redisReadTask rstack[9];
+    int ridx; /* Index of current read task */
+    void *reply; /* Temporary reply pointer */
+
+    redisReplyObjectFunctions *fn;
+    void *privdata;
+} redisReader;
+
+/* Public API for the protocol parser. */
+redisReader *redisReaderCreateWithFunctions(redisReplyObjectFunctions *fn);
+void redisReaderFree(redisReader *r);
+int redisReaderFeed(redisReader *r, const char *buf, size_t len);
+int redisReaderGetReply(redisReader *r, void **reply);
+
+#define redisReaderSetPrivdata(_r, _p) (int)(((redisReader*)(_r))->privdata = (_p))
+#define redisReaderGetObject(_r) (((redisReader*)(_r))->reply)
+#define redisReaderGetError(_r) (((redisReader*)(_r))->errstr)
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
@@ -89,9 +89,9 @@ sds sdsnewlen(const void *init, size_t initlen) {
    unsigned char *fp; /* flags pointer. */

    sh = s_malloc(hdrlen+initlen+1);
+    if (sh == NULL) return NULL;
    if (!init)
        memset(sh, 0, hdrlen+initlen+1);
-    if (sh == NULL) return NULL;
    s = (char*)sh+hdrlen;
    fp = ((unsigned char*)s)-1;
    switch(type) {
@@ -577,14 +577,12 @@ sds sdscatprintf(sds s, const char *fmt, ...) {
 * %% - Verbatim "%" character.
 */
 sds sdscatfmt(sds s, char const *fmt, ...) {
-    size_t initlen = sdslen(s);
    const char *f = fmt;
    int i;
    va_list ap;

    va_start(ap,fmt);
-    f = fmt;    /* Next format specifier byte to process. */
-    i = initlen; /* Position of the next byte to write to dest str. */
+    i = sdslen(s); /* Position of the next byte to write to dest str. */
    while(*f) {
        char next, *str;
        size_t l;
@@ -79,7 +79,7 @@ struct __attribute__ ((__packed__)) sdshdr64 {
 #define SDS_TYPE_64 4
 #define SDS_TYPE_MASK 7
 #define SDS_TYPE_BITS 3
-#define SDS_HDR_VAR(T,s) struct sdshdr##T *sh = (void*)((s)-(sizeof(struct sdshdr##T)));
+#define SDS_HDR_VAR(T,s) struct sdshdr##T *sh = (struct sdshdr##T *)((s)-(sizeof(struct sdshdr##T)));
 #define SDS_HDR(T,s) ((struct sdshdr##T *)((s)-(sizeof(struct sdshdr##T))))
 #define SDS_TYPE_5_LEN(f) ((f)>>SDS_TYPE_BITS)

@@ -1,6 +1,7 @@
 /* SDSLib 2.0 -- A C dynamic strings library
 *
 * Copyright (c) 2006-2015, Salvatore Sanfilippo <antirez at gmail dot com>
+ * Copyright (c) 2015, Oran Agra
 * Copyright (c) 2015, Redis Labs, Inc
 * All rights reserved.
 *
@@ -36,7 +37,6 @@
 * the include of your alternate allocator if needed (not needed in order
 * to use the default libc allocator). */

-#include "zmalloc.h"
-#define s_malloc zmalloc
-#define s_realloc zrealloc
-#define s_free zfree
+#define s_malloc malloc
+#define s_realloc realloc
+#define s_free free
@@ -11,6 +11,7 @@
 #include <limits.h>

 #include "hiredis.h"
+#include "net.h"

 enum connection_type {
    CONN_TCP,
@@ -29,7 +30,7 @@ struct config {

    struct {
        const char *path;
-    } unix;
+    } unix_sock;
 };

 /* The following lines make up our testing "framework" :) */
@@ -43,6 +44,13 @@ static long long usec(void) {
    return (((long long)tv.tv_sec)*1000000)+tv.tv_usec;
 }

+/* The assert() calls below have side effects, so we need assert()
+ * even if we are compiling without asserts (-DNDEBUG). */
+#ifdef NDEBUG
+#undef assert
+#define assert(e) (void)(e)
+#endif
+
 static redisContext *select_database(redisContext *c) {
    redisReply *reply;

@@ -51,7 +59,7 @@ static redisContext *select_database(redisContext *c) {
    assert(reply != NULL);
    freeReplyObject(reply);

-    /* Make sure the DB is empty */
+    /* Make sure the DB is emtpy */
    reply = redisCommand(c,"DBSIZE");
    assert(reply != NULL);
    if (reply->type == REDIS_REPLY_INTEGER && reply->integer == 0) {
@@ -89,10 +97,10 @@ static redisContext *connect(struct config config) {
    if (config.type == CONN_TCP) {
        c = redisConnect(config.tcp.host, config.tcp.port);
    } else if (config.type == CONN_UNIX) {
-        c = redisConnectUnix(config.unix.path);
+        c = redisConnectUnix(config.unix_sock.path);
    } else if (config.type == CONN_FD) {
        /* Create a dummy connection just to get an fd to inherit */
-        redisContext *dummy_ctx = redisConnectUnix(config.unix.path);
+        redisContext *dummy_ctx = redisConnectUnix(config.unix_sock.path);
        if (dummy_ctx) {
            int fd = disconnect(dummy_ctx, 1);
            printf("Connecting to inherited fd %d\n", fd);
@@ -107,6 +115,7 @@ static redisContext *connect(struct config config) {
        exit(1);
    } else if (c->err) {
        printf("Connection error: %s\n", c->errstr);
+        redisFree(c);
        exit(1);
    }

@@ -215,6 +224,22 @@ static void test_format_commands(void) {
    test_cond(strncmp(cmd,"*3\r\n$3\r\nSET\r\n$7\r\nfoo\0xxx\r\n$3\r\nbar\r\n",len) == 0 &&
        len == 4+4+(3+2)+4+(7+2)+4+(3+2));
    free(cmd);
+
+    sds sds_cmd;
+
+    sds_cmd = sdsempty();
+    test("Format command into sds by passing argc/argv without lengths: ");
+    len = redisFormatSdsCommandArgv(&sds_cmd,argc,argv,NULL);
+    test_cond(strncmp(sds_cmd,"*3\r\n$3\r\nSET\r\n$3\r\nfoo\r\n$3\r\nbar\r\n",len) == 0 &&
+        len == 4+4+(3+2)+4+(3+2)+4+(3+2));
+    sdsfree(sds_cmd);
+
+    sds_cmd = sdsempty();
+    test("Format command into sds by passing argc/argv with lengths: ");
+    len = redisFormatSdsCommandArgv(&sds_cmd,argc,argv,lens);
+    test_cond(strncmp(sds_cmd,"*3\r\n$3\r\nSET\r\n$7\r\nfoo\0xxx\r\n$3\r\nbar\r\n",len) == 0 &&
+        len == 4+4+(3+2)+4+(7+2)+4+(3+2));
+    sdsfree(sds_cmd);
 }

 static void test_append_formatted_commands(struct config config) {
@@ -318,16 +343,31 @@ static void test_reply_reader(void) {
    redisReaderFree(reader);
 }

+static void test_free_null(void) {
+    void *redisCtx = NULL;
+    void *reply = NULL;
+
+    test("Don't fail when redisFree is passed a NULL value: ");
+    redisFree(redisCtx);
+    test_cond(redisCtx == NULL);
+
+    test("Don't fail when freeReplyObject is passed a NULL value: ");
+    freeReplyObject(reply);
+    test_cond(reply == NULL);
+}
+
 static void test_blocking_connection_errors(void) {
    redisContext *c;

    test("Returns error when host cannot be resolved: ");
-    c = redisConnect((char*)"idontexist.local", 6379);
+    c = redisConnect((char*)"idontexist.test", 6379);
    test_cond(c->err == REDIS_ERR_OTHER &&
        (strcmp(c->errstr,"Name or service not known") == 0 ||
-         strcmp(c->errstr,"Can't resolve: idontexist.local") == 0 ||
+         strcmp(c->errstr,"Can't resolve: idontexist.test") == 0 ||
         strcmp(c->errstr,"nodename nor servname provided, or not known") == 0 ||
         strcmp(c->errstr,"No address associated with hostname") == 0 ||
+         strcmp(c->errstr,"Temporary failure in name resolution") == 0 ||
+         strcmp(c->errstr,"hostname nor servname provided, or not known") == 0 ||
         strcmp(c->errstr,"no address associated with name") == 0));
    redisFree(c);

@@ -337,7 +377,7 @@ static void test_blocking_connection_errors(void) {
        strcmp(c->errstr,"Connection refused") == 0);
    redisFree(c);

-    test("Returns error when the unix socket path doesn't accept connections: ");
+    test("Returns error when the unix_sock socket path doesn't accept connections: ");
    c = redisConnectUnix((char*)"/tmp/idontexist.sock");
    test_cond(c->err == REDIS_ERR_IO); /* Don't care about the message... */
    redisFree(c);
@@ -421,6 +461,52 @@ static void test_blocking_connection(struct config config) {
    disconnect(c, 0);
 }

+static void test_blocking_connection_timeouts(struct config config) {
+    redisContext *c;
+    redisReply *reply;
+    ssize_t s;
+    const char *cmd = "DEBUG SLEEP 3\r\n";
+    struct timeval tv;
+
+    c = connect(config);
+    test("Successfully completes a command when the timeout is not exceeded: ");
+    reply = redisCommand(c,"SET foo fast");
+    freeReplyObject(reply);
+    tv.tv_sec = 0;
+    tv.tv_usec = 10000;
+    redisSetTimeout(c, tv);
+    reply = redisCommand(c, "GET foo");
+    test_cond(reply != NULL && reply->type == REDIS_REPLY_STRING && memcmp(reply->str, "fast", 4) == 0);
+    freeReplyObject(reply);
+    disconnect(c, 0);
+
+    c = connect(config);
+    test("Does not return a reply when the command times out: ");
+    s = write(c->fd, cmd, strlen(cmd));
+    tv.tv_sec = 0;
+    tv.tv_usec = 10000;
+    redisSetTimeout(c, tv);
+    reply = redisCommand(c, "GET foo");
+    test_cond(s > 0 && reply == NULL && c->err == REDIS_ERR_IO && strcmp(c->errstr, "Resource temporarily unavailable") == 0);
+    freeReplyObject(reply);
+
+    test("Reconnect properly reconnects after a timeout: ");
+    redisReconnect(c);
+    reply = redisCommand(c, "PING");
+    test_cond(reply != NULL && reply->type == REDIS_REPLY_STATUS && strcmp(reply->str, "PONG") == 0);
+    freeReplyObject(reply);
+
+    test("Reconnect properly uses owned parameters: ");
+    config.tcp.host = "foo";
+    config.unix_sock.path = "foo";
+    redisReconnect(c);
+    reply = redisCommand(c, "PING");
+    test_cond(reply != NULL && reply->type == REDIS_REPLY_STATUS && strcmp(reply->str, "PONG") == 0);
+    freeReplyObject(reply);
+
+    disconnect(c, 0);
+}
+
 static void test_blocking_io_errors(struct config config) {
    redisContext *c;
    redisReply *reply;
@@ -444,7 +530,7 @@ static void test_blocking_io_errors(struct config config) {

    test("Returns I/O error when the connection is lost: ");
    reply = redisCommand(c,"QUIT");
-    if (major >= 2 && minor > 0) {
+    if (major > 2 || (major == 2 && minor > 0)) {
        /* > 2.0 returns OK on QUIT and read() should be issued once more
         * to know the descriptor is at EOF. */
        test_cond(strcasecmp(reply->str,"OK") == 0 &&
@@ -482,7 +568,8 @@ static void test_invalid_timeout_errors(struct config config) {

    c = redisConnectWithTimeout(config.tcp.host, config.tcp.port, config.tcp.timeout);

-    test_cond(c->err == REDIS_ERR_IO);
+    test_cond(c->err == REDIS_ERR_IO && strcmp(c->errstr, "Invalid timeout specified") == 0);
+    redisFree(c);

    test("Set error when an invalid timeout sec value is given to redisConnectWithTimeout: ");

@@ -491,8 +578,7 @@ static void test_invalid_timeout_errors(struct config config) {

    c = redisConnectWithTimeout(config.tcp.host, config.tcp.port, config.tcp.timeout);

-    test_cond(c->err == REDIS_ERR_IO);
-
+    test_cond(c->err == REDIS_ERR_IO && strcmp(c->errstr, "Invalid timeout specified") == 0);
    redisFree(c);
 }

@@ -666,7 +752,7 @@ int main(int argc, char **argv) {
            .host = "127.0.0.1",
            .port = 6379
        },
-        .unix = {
+        .unix_sock = {
            .path = "/tmp/redis.sock"
        }
    };
@@ -687,7 +773,7 @@ int main(int argc, char **argv) {
            cfg.tcp.port = atoi(argv[0]);
        } else if (argc >= 2 && !strcmp(argv[0],"-s")) {
            argv++; argc--;
-            cfg.unix.path = argv[0];
+            cfg.unix_sock.path = argv[0];
        } else if (argc >= 1 && !strcmp(argv[0],"--skip-throughput")) {
            throughput = 0;
        } else if (argc >= 1 && !strcmp(argv[0],"--skip-inherit-fd")) {
@@ -702,27 +788,31 @@ int main(int argc, char **argv) {
    test_format_commands();
    test_reply_reader();
    test_blocking_connection_errors();
+    test_free_null();

    printf("\nTesting against TCP connection (%s:%d):\n", cfg.tcp.host, cfg.tcp.port);
    cfg.type = CONN_TCP;
    test_blocking_connection(cfg);
+    test_blocking_connection_timeouts(cfg);
    test_blocking_io_errors(cfg);
    test_invalid_timeout_errors(cfg);
    test_append_formatted_commands(cfg);
    if (throughput) test_throughput(cfg);

-    printf("\nTesting against Unix socket connection (%s):\n", cfg.unix.path);
+    printf("\nTesting against Unix socket connection (%s):\n", cfg.unix_sock.path);
    cfg.type = CONN_UNIX;
    test_blocking_connection(cfg);
+    test_blocking_connection_timeouts(cfg);
    test_blocking_io_errors(cfg);
    if (throughput) test_throughput(cfg);

    if (test_inherit_fd) {
-        printf("\nTesting against inherited fd (%s):\n", cfg.unix.path);
+        printf("\nTesting against inherited fd (%s):\n", cfg.unix_sock.path);
        cfg.type = CONN_FD;
        test_blocking_connection(cfg);
    }

+
    if (fails) {
        printf("*** %d TESTS FAILED ***\n", fails);
        return 1;
@@ -0,0 +1,42 @@
+#ifndef _WIN32_HELPER_INCLUDE
+#define _WIN32_HELPER_INCLUDE
+#ifdef _MSC_VER
+
+#ifndef inline
+#define inline __inline
+#endif
+
+#ifndef va_copy
+#define va_copy(d,s) ((d) = (s))
+#endif
+
+#ifndef snprintf
+#define snprintf c99_snprintf
+
+__inline int c99_vsnprintf(char* str, size_t size, const char* format, va_list ap)
+{
+    int count = -1;
+
+    if (size != 0)
+        count = _vsnprintf_s(str, size, _TRUNCATE, format, ap);
+    if (count == -1)
+        count = _vscprintf(format, ap);
+
+    return count;
+}
+
+__inline int c99_snprintf(char* str, size_t size, const char* format, ...)
+{
+    int count;
+    va_list ap;
+
+    va_start(ap, format);
+    count = c99_vsnprintf(str, size, format, ap);
+    va_end(ap);
+
+    return count;
+}
+#endif
+
+#endif
+#endif
@@ -1,13 +0,0 @@
-/* Drop in replacement for zmalloc.h in order to just use libc malloc without
- * any wrappering. */
-
-#ifndef ZMALLOC_H
-#define ZMALLOC_H
-
-#define zmalloc malloc
-#define zrealloc realloc
-#define zcalloc(x) calloc(x,1)
-#define zfree free
-#define zstrdup strdup
-
-#endif
@@ -100,3 +100,7 @@
 #  define JEMALLOC_RESTRICT_RETURN
 #  define JEMALLOC_ALLOCATOR
 #endif
+
+/* This version of Jemalloc, modified for Redis, has the je_get_defrag_hint()
+ * function. */
+#define JEMALLOC_FRAG_HINT
@@ -2591,3 +2591,35 @@ jemalloc_postfork_child(void)
 }

 /******************************************************************************/
+
+/* Helps the application decide if a pointer is worth re-allocating in order to reduce fragmentation.
+ * returns 0 if the allocation is in the currently active run,
+ * or when it is not causing any frag issue (large or huge bin)
+ * returns the bin utilization and run utilization both in fixed point 16:16.
+ * If the application decides to re-allocate it should use MALLOCX_TCACHE_NONE when doing so. */
+JEMALLOC_EXPORT int JEMALLOC_NOTHROW
+je_get_defrag_hint(void* ptr, int *bin_util, int *run_util) {
+    int defrag = 0;
+    arena_chunk_t *chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(ptr);
+    if (likely(chunk != ptr)) { /* indication that this is not a HUGE alloc */
+        size_t pageind = ((uintptr_t)ptr - (uintptr_t)chunk) >> LG_PAGE;
+        size_t mapbits = arena_mapbits_get(chunk, pageind);
+        if (likely((mapbits & CHUNK_MAP_LARGE) == 0)) { /* indication that this is not a LARGE alloc */
+            arena_t *arena = extent_node_arena_get(&chunk->node);
+            size_t rpages_ind = pageind - arena_mapbits_small_runind_get(chunk, pageind);
+            arena_run_t *run = &arena_miscelm_get(chunk, rpages_ind)->run;
+            arena_bin_t *bin = &arena->bins[run->binind];
+            malloc_mutex_lock(&bin->lock);
+            /* runs that are in the same chunk in as the current chunk, are likely to be the next currun */
+            if (chunk != (arena_chunk_t *)CHUNK_ADDR2BASE(bin->runcur)) {
+                arena_bin_info_t *bin_info = &arena_bin_info[run->binind];
+                size_t availregs = bin_info->nregs * bin->stats.curruns;
+                *bin_util = (bin->stats.curregs<<16) / availregs;
+                *run_util = ((bin_info->nregs - run->nfree)<<16) / bin_info->nregs;
+                defrag = 1;
+            }
+            malloc_mutex_unlock(&bin->lock);
+        }
+    }
+    return defrag;
+}
@@ -185,6 +185,14 @@ logfile ""
 # dbid is a number between 0 and 'databases'-1
 databases 16

+# By default Redis shows an ASCII art logo only when started to log to the
+# standard output and if the standard output is a TTY. Basically this means
+# that normally a logo is displayed only in interactive sessions.
+#
+# However it is possible to force the pre-4.0 behavior and always show a
+# ASCII art logo in startup logs by setting the following option to yes.
+always-show-logo yes
+
 ################################ SNAPSHOTTING  ################################
 #
 # Save the DB on disk:
@@ -822,7 +830,7 @@ lua-time-limit 5000
 # A slave of a failing master will avoid to start a failover if its data
 # looks too old.
 #
-# There is no simple way for a slave to actually have a exact measure of
+# There is no simple way for a slave to actually have an exact measure of
 # its "data age", so the following two checks are performed:
 #
 # 1) If there are multiple slaves able to failover, they exchange messages
@@ -1220,3 +1228,66 @@ aof-rewrite-incremental-fsync yes
 #
 # lfu-log-factor 10
 # lfu-decay-time 1
+
+########################### ACTIVE DEFRAGMENTATION #######################
+#
+# WARNING THIS FEATURE IS EXPERIMENTAL. However it was stress tested
+# even in production and manually tested by multiple engineers for some
+# time.
+#
+# What is active defragmentation?
+# -------------------------------
+#
+# Active (online) defragmentation allows a Redis server to compact the
+# spaces left between small allocations and deallocations of data in memory,
+# thus allowing to reclaim back memory.
+#
+# Fragmentation is a natural process that happens with every allocator (but
+# less so with Jemalloc, fortunately) and certain workloads. Normally a server
+# restart is needed in order to lower the fragmentation, or at least to flush
+# away all the data and create it again. However thanks to this feature
+# implemented by Oran Agra for Redis 4.0 this process can happen at runtime
+# in an "hot" way, while the server is running.
+#
+# Basically when the fragmentation is over a certain level (see the
+# configuration options below) Redis will start to create new copies of the
+# values in contiguous memory regions by exploiting certain specific Jemalloc
+# features (in order to understand if an allocation is causing fragmentation
+# and to allocate it in a better place), and at the same time, will release the
+# old copies of the data. This process, repeated incrementally for all the keys
+# will cause the fragmentation to drop back to normal values.
+#
+# Important things to understand:
+#
+# 1. This feature is disabled by default, and only works if you compiled Redis
+#    to use the copy of Jemalloc we ship with the source code of Redis.
+#    This is the default with Linux builds.
+#
+# 2. You never need to enable this feature if you don't have fragmentation
+#    issues.
+#
+# 3. Once you experience fragmentation, you can enable this feature when
+#    needed with the command "CONFIG SET activedefrag yes".
+#
+# The configuration parameters are able to fine tune the behavior of the
+# defragmentation process. If you are not sure about what they mean it is
+# a good idea to leave the defaults untouched.
+
+# Enabled active defragmentation
+# activedefrag yes
+
+# Minimum amount of fragmentation waste to start active defrag
+# active-defrag-ignore-bytes 100mb
+
+# Minimum percentage of fragmentation to start active defrag
+# active-defrag-threshold-lower 10
+
+# Maximum percentage of fragmentation at which we use maximum effort
+# active-defrag-threshold-upper 100
+
+# Minimal effort for defrag in CPU percentage
+# active-defrag-cycle-min 25
+
+# Maximal effort for defrag in CPU percentage
+# active-defrag-cycle-max 75
+
@@ -14,6 +14,7 @@

 release_hdr := $(shell sh -c './mkreleasehdr.sh')
 uname_S := $(shell sh -c 'uname -s 2>/dev/null || echo not')
+uname_M := $(shell sh -c 'uname -m 2>/dev/null || echo not')
 OPTIMIZATION?=-O2
 DEPENDENCY_TARGETS=hiredis linenoise lua
 NODEPS:=clean distclean
@@ -27,11 +28,14 @@ PREFIX?=/usr/local
 INSTALL_BIN=$(PREFIX)/bin
 INSTALL=install

-# Default allocator
+# Default allocator defaults to Jemalloc if it's not an ARM
+MALLOC=libc
+ifneq ($(uname_M),armv6l)
+ifneq ($(uname_M),armv7l)
 ifeq ($(uname_S),Linux)
 	MALLOC=jemalloc
-else
-	MALLOC=libc
+endif
+endif
 endif

 # Backwards compatibility for selecting an allocator
@@ -61,6 +65,13 @@ DEBUG=-g -ggdb

 ifeq ($(uname_S),SunOS)
 	# SunOS
+        ifneq ($(@@),32bit)
+		CFLAGS+= -m64
+		LDFLAGS+= -m64
+	endif
+	DEBUG=-g
+	DEBUG_FLAGS=-g
+	export CFLAGS LDFLAGS DEBUG DEBUG_FLAGS
 	INSTALL=cp -pf
 	FINAL_CFLAGS+= -D__EXTENSIONS__ -D_XPG6
 	FINAL_LIBS+= -ldl -lnsl -lsocket -lresolv -lpthread -lrt
@@ -128,7 +139,7 @@ endif

 REDIS_SERVER_NAME=redis-server
 REDIS_SENTINEL_NAME=redis-sentinel
-REDIS_SERVER_OBJ=adlist.o quicklist.o ae.o anet.o dict.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o
+REDIS_SERVER_OBJ=adlist.o quicklist.o ae.o anet.o dict.o server.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o redis-check-rdb.o geo.o lazyfree.o module.o evict.o expire.o geohash.o geohash_helper.o childinfo.o defrag.o siphash.o rax.o
 REDIS_CLI_NAME=redis-cli
 REDIS_CLI_OBJ=anet.o adlist.o redis-cli.o zmalloc.o release.o anet.o ae.o crc64.o
 REDIS_BENCHMARK_NAME=redis-benchmark
@@ -203,8 +214,8 @@ $(REDIS_BENCHMARK_NAME): $(REDIS_BENCHMARK_OBJ)
 $(REDIS_CHECK_AOF_NAME): $(REDIS_CHECK_AOF_OBJ)
 	$(REDIS_LD) -o $@ $^ $(FINAL_LIBS)

-dict-benchmark: dict.c zmalloc.c sds.c
-	$(REDIS_CC) $(FINAL_CFLAGS) dict.c zmalloc.c sds.c -D DICT_BENCHMARK_MAIN -o dict-benchmark
+dict-benchmark: dict.c zmalloc.c sds.c siphash.c
+	$(REDIS_CC) $(FINAL_CFLAGS) $^ -D DICT_BENCHMARK_MAIN -o $@ $(FINAL_LIBS)

 # Because the jemalloc.h header is generated as a part of the jemalloc build,
 # building it should complete before building any other object. Instead of
@@ -380,8 +380,10 @@ int anetUnixGenericConnect(char *err, char *path, int flags)
    sa.sun_family = AF_LOCAL;
    strncpy(sa.sun_path,path,sizeof(sa.sun_path)-1);
    if (flags & ANET_CONNECT_NONBLOCK) {
-        if (anetNonBlock(err,s) != ANET_OK)
+        if (anetNonBlock(err,s) != ANET_OK) {
+            close(s);
            return ANET_ERR;
+        }
    }
    if (connect(s,(struct sockaddr*)&sa,sizeof(sa)) == -1) {
        if (errno == EINPROGRESS &&
@@ -462,7 +464,7 @@ static int anetV6Only(char *err, int s) {

 static int _anetTcpServer(char *err, int port, char *bindaddr, int af, int backlog)
 {
-    int s, rv;
+    int s = -1, rv;
    char _port[6];  /* strlen("65535") */
    struct addrinfo hints, *servinfo, *p;

@@ -491,6 +493,7 @@ static int _anetTcpServer(char *err, int port, char *bindaddr, int af, int backl
    }

 error:
+    if (s != -1) close(s);
    s = ANET_ERR;
 end:
    freeaddrinfo(servinfo);
@@ -115,6 +115,7 @@ void aofChildWriteDiffData(aeEventLoop *el, int fd, void *privdata, int mask) {
            if (nwritten <= 0) return;
            memmove(block->buf,block->buf+nwritten,block->used-nwritten);
            block->used -= nwritten;
+            block->free += nwritten;
        }
        if (block->used == 0) listDelNode(server.aof_rewrite_buf_blocks,ln);
    }
@@ -1357,6 +1358,7 @@ int rewriteAppendOnlyFileBackground(void) {
            serverLog(LL_WARNING,
                "Can't rewrite append only file in background: fork: %s",
                strerror(errno));
+            aofClosePipes();
            return C_ERR;
        }
        serverLog(LL_NOTICE,
@@ -51,7 +51,7 @@
 #ifndef __ATOMIC_VAR_H
 #define __ATOMIC_VAR_H

-#if defined(__ATOMIC_RELAXED) && (!defined(__clang__) || !defined(__APPLE__) || __apple_build_version__ > 4210057)
+#if defined(__ATOMIC_RELAXED) && !defined(__sun) && (!defined(__clang__) || !defined(__APPLE__) || __apple_build_version__ > 4210057)
 /* Implementation using __atomic macros. */

 #define atomicIncr(var,count,mutex) __atomic_add_fetch(&var,(count),__ATOMIC_RELAXED)
@@ -104,6 +104,7 @@ long redisBitpos(void *s, unsigned long count, int bit) {
    unsigned long skipval, word = 0, one;
    long pos = 0; /* Position of bit, to return to the caller. */
    unsigned long j;
+    int found;

    /* Process whole words first, seeking for first word that is not
     * all ones or all zeros respectively if we are lookig for zeros
@@ -117,21 +118,27 @@ long redisBitpos(void *s, unsigned long count, int bit) {
    /* Skip initial bits not aligned to sizeof(unsigned long) byte by byte. */
    skipval = bit ? 0 : UCHAR_MAX;
    c = (unsigned char*) s;
+    found = 0;
    while((unsigned long)c & (sizeof(*l)-1) && count) {
-        if (*c != skipval) break;
+        if (*c != skipval) {
+            found = 1;
+            break;
+        }
        c++;
        count--;
        pos += 8;
    }

    /* Skip bits with full word step. */
-    skipval = bit ? 0 : ULONG_MAX;
    l = (unsigned long*) c;
-    while (count >= sizeof(*l)) {
-        if (*l != skipval) break;
-        l++;
-        count -= sizeof(*l);
-        pos += sizeof(*l)*8;
+    if (!found) {
+        skipval = bit ? 0 : ULONG_MAX;
+        while (count >= sizeof(*l)) {
+            if (*l != skipval) break;
+            l++;
+            count -= sizeof(*l);
+            pos += sizeof(*l)*8;
+        }
    }

    /* Load bytes into "word" considering the first byte as the most significant
@@ -654,8 +661,11 @@ void bitopCommand(client *c) {

        /* Fast path: as far as we have data for all the input bitmaps we
         * can take a fast path that performs much better than the
-         * vanilla algorithm. */
+         * vanilla algorithm. On ARM we skip the fast path since it will
+         * result in GCC compiling the code using multiple-words load/store
+         * operations that are not supported even in ARM >= v6. */
        j = 0;
+        #ifndef USE_ALIGNED_ACCESS
        if (minlen >= sizeof(unsigned long)*4 && numkeys <= 16) {
            unsigned long *lp[16];
            unsigned long *lres = (unsigned long*) res;
@@ -716,6 +726,7 @@ void bitopCommand(client *c) {
                }
            }
        }
+        #endif

        /* j is set to the next byte to process by the previous loop. */
        for (; j < maxlen; j++) {
@@ -423,8 +423,11 @@ void clusterInit(void) {
    server.cluster->failover_auth_epoch = 0;
    server.cluster->cant_failover_reason = CLUSTER_CANT_FAILOVER_NONE;
    server.cluster->lastVoteEpoch = 0;
-    server.cluster->stats_bus_messages_sent = 0;
-    server.cluster->stats_bus_messages_received = 0;
+    for (int i = 0; i < CLUSTERMSG_TYPE_COUNT; i++) {
+        server.cluster->stats_bus_messages_sent[i] = 0;
+        server.cluster->stats_bus_messages_received[i] = 0;
+    }
+    server.cluster->stats_pfail_nodes = 0;
    memset(server.cluster->slots,0, sizeof(server.cluster->slots));
    clusterCloseAllSlots();

@@ -476,8 +479,10 @@ void clusterInit(void) {
        }
    }

-    /* The slots -> keys map is a sorted set. Init it. */
-    server.cluster->slots_to_keys = zslCreate();
+    /* The slots -> keys map is a radix tree. Initialize it here. */
+    server.cluster->slots_to_keys = raxNew();
+    memset(server.cluster->slots_keys_count,0,
+           sizeof(server.cluster->slots_keys_count));

    /* Set myself->port / cport to my listening ports, we'll just need to
     * discover the IP address via MEET messages. */
@@ -1350,6 +1355,28 @@ void clusterProcessGossipSection(clusterMsg *hdr, clusterLink *link) {
                }
            }

+            /* If from our POV the node is up (no failure flags are set),
+             * we have no pending ping for the node, nor we have failure
+             * reports for this node, update the last pong time with the
+             * one we see from the other nodes. */
+            if (!(flags & (CLUSTER_NODE_FAIL|CLUSTER_NODE_PFAIL)) &&
+                node->ping_sent == 0 &&
+                clusterNodeFailureReportsCount(node) == 0)
+            {
+                mstime_t pongtime = ntohl(g->pong_received);
+                pongtime *= 1000; /* Convert back to milliseconds. */
+
+                /* Replace the pong time with the received one only if
+                 * it's greater than our view but is not in the future
+                 * (with 500 milliseconds tolerance) from the POV of our
+                 * clock. */
+                if (pongtime <= (server.mstime+500) &&
+                    pongtime > node->pong_received)
+                {
+                    node->pong_received = pongtime;
+                }
+            }
+
            /* If we already know this node, but it is not reachable, and
             * we see a different address in the gossip section of a node that
             * can talk with this other node, update the address, disconnect
@@ -1581,7 +1608,8 @@ int clusterProcessPacket(clusterLink *link) {
    uint32_t totlen = ntohl(hdr->totlen);
    uint16_t type = ntohs(hdr->type);

-    server.cluster->stats_bus_messages_received++;
+    if (type < CLUSTERMSG_TYPE_COUNT)
+        server.cluster->stats_bus_messages_received[type]++;
    serverLog(LL_DEBUG,"--- Processing packet of type %d, %lu bytes",
        type, (unsigned long) totlen);

@@ -2128,7 +2156,12 @@ void clusterSendMessage(clusterLink *link, unsigned char *msg, size_t msglen) {
                    clusterWriteHandler,link);

    link->sndbuf = sdscatlen(link->sndbuf, msg, msglen);
-    server.cluster->stats_bus_messages_sent++;
+
+    /* Populate sent messages stats. */
+    clusterMsg *hdr = (clusterMsg*) msg;
+    uint16_t type = ntohs(hdr->type);
+    if (type < CLUSTERMSG_TYPE_COUNT)
+        server.cluster->stats_bus_messages_sent[type]++;
 }

 /* Send a message to all the nodes that are part of the cluster having
@@ -2229,6 +2262,33 @@ void clusterBuildMessageHdr(clusterMsg *hdr, int type) {
    /* For PING, PONG, and MEET, fixing the totlen field is up to the caller. */
 }

+/* Return non zero if the node is already present in the gossip section of the
+ * message pointed by 'hdr' and having 'count' gossip entries. Otherwise
+ * zero is returned. Helper for clusterSendPing(). */
+int clusterNodeIsInGossipSection(clusterMsg *hdr, int count, clusterNode *n) {
+    int j;
+    for (j = 0; j < count; j++) {
+        if (memcmp(hdr->data.ping.gossip[j].nodename,n->name,
+                CLUSTER_NAMELEN) == 0) break;
+    }
+    return j != count;
+}
+
+/* Set the i-th entry of the gossip section in the message pointed by 'hdr'
+ * to the info of the specified node 'n'. */
+void clusterSetGossipEntry(clusterMsg *hdr, int i, clusterNode *n) {
+    clusterMsgDataGossip *gossip;
+    gossip = &(hdr->data.ping.gossip[i]);
+    memcpy(gossip->nodename,n->name,CLUSTER_NAMELEN);
+    gossip->ping_sent = htonl(n->ping_sent/1000);
+    gossip->pong_received = htonl(n->pong_received/1000);
+    memcpy(gossip->ip,n->ip,sizeof(n->ip));
+    gossip->port = htons(n->port);
+    gossip->cport = htons(n->cport);
+    gossip->flags = htons(n->flags);
+    gossip->notused1 = 0;
+}
+
 /* Send a PING or PONG packet to the specified node, making sure to add enough
 * gossip informations. */
 void clusterSendPing(clusterLink *link, int type) {
@@ -2273,11 +2333,15 @@ void clusterSendPing(clusterLink *link, int type) {
    if (wanted < 3) wanted = 3;
    if (wanted > freshnodes) wanted = freshnodes;

+    /* Include all the nodes in PFAIL state, so that failure reports are
+     * faster to propagate to go from PFAIL to FAIL state. */
+    int pfail_wanted = server.cluster->stats_pfail_nodes;
+
    /* Compute the maxium totlen to allocate our buffer. We'll fix the totlen
     * later according to the number of gossip sections we really were able
     * to put inside the packet. */
    totlen = sizeof(clusterMsg)-sizeof(union clusterMsgData);
-    totlen += (sizeof(clusterMsgDataGossip)*wanted);
+    totlen += (sizeof(clusterMsgDataGossip)*(wanted+pfail_wanted));
    /* Note: clusterBuildMessageHdr() expects the buffer to be always at least
     * sizeof(clusterMsg) or more. */
    if (totlen < (int)sizeof(clusterMsg)) totlen = sizeof(clusterMsg);
@@ -2294,17 +2358,13 @@ void clusterSendPing(clusterLink *link, int type) {
    while(freshnodes > 0 && gossipcount < wanted && maxiterations--) {
        dictEntry *de = dictGetRandomKey(server.cluster->nodes);
        clusterNode *this = dictGetVal(de);
-        clusterMsgDataGossip *gossip;
-        int j;

        /* Don't include this node: the whole packet header is about us
         * already, so we just gossip about other nodes. */
        if (this == myself) continue;

-        /* Give a bias to FAIL/PFAIL nodes. */
-        if (maxiterations > wanted*2 &&
-            !(this->flags & (CLUSTER_NODE_PFAIL|CLUSTER_NODE_FAIL)))
-            continue;
+        /* PFAIL nodes will be added later. */
+        if (this->flags & CLUSTER_NODE_PFAIL) continue;

        /* In the gossip section don't include:
         * 1) Nodes in HANDSHAKE state.
@@ -2318,27 +2378,37 @@ void clusterSendPing(clusterLink *link, int type) {
            continue;
        }

-        /* Check if we already added this node */
-        for (j = 0; j < gossipcount; j++) {
-            if (memcmp(hdr->data.ping.gossip[j].nodename,this->name,
-                    CLUSTER_NAMELEN) == 0) break;
-        }
-        if (j != gossipcount) continue;
+        /* Do not add a node we already have. */
+        if (clusterNodeIsInGossipSection(hdr,gossipcount,this)) continue;

        /* Add it */
+        clusterSetGossipEntry(hdr,gossipcount,this);
        freshnodes--;
-        gossip = &(hdr->data.ping.gossip[gossipcount]);
-        memcpy(gossip->nodename,this->name,CLUSTER_NAMELEN);
-        gossip->ping_sent = htonl(this->ping_sent);
-        gossip->pong_received = htonl(this->pong_received);
-        memcpy(gossip->ip,this->ip,sizeof(this->ip));
-        gossip->port = htons(this->port);
-        gossip->cport = htons(this->cport);
-        gossip->flags = htons(this->flags);
-        gossip->notused1 = 0;
        gossipcount++;
    }

+    /* If there are PFAIL nodes, add them at the end. */
+    if (pfail_wanted) {
+        dictIterator *di;
+        dictEntry *de;
+
+        di = dictGetSafeIterator(server.cluster->nodes);
+        while((de = dictNext(di)) != NULL && pfail_wanted > 0) {
+            clusterNode *node = dictGetVal(de);
+            if (node->flags & CLUSTER_NODE_HANDSHAKE) continue;
+            if (node->flags & CLUSTER_NODE_NOADDR) continue;
+            if (!(node->flags & CLUSTER_NODE_PFAIL)) continue;
+            clusterSetGossipEntry(hdr,gossipcount,node);
+            freshnodes--;
+            gossipcount++;
+            /* We take the count of the slots we allocated, since the
+             * PFAIL stats may not match perfectly with the current number
+             * of PFAIL nodes. */
+            pfail_wanted--;
+        }
+        dictReleaseIterator(di);
+    }
+
    /* Ready to send... fix the totlen fiend and queue the message in the
     * output buffer. */
    totlen = sizeof(clusterMsg)-sizeof(union clusterMsgData);
@@ -2774,7 +2844,7 @@ void clusterHandleSlaveFailover(void) {
     * and wait for replies), and the failover retry time (the time to wait
     * before trying to get voted again).
     *
-     * Timeout is MIN(NODE_TIMEOUT*2,2000) milliseconds.
+     * Timeout is MAX(NODE_TIMEOUT*2,2000) milliseconds.
     * Retry is two times the Timeout.
     */
    auth_timeout = server.cluster_node_timeout*2;
@@ -3164,13 +3234,21 @@ void clusterCron(void) {
    handshake_timeout = server.cluster_node_timeout;
    if (handshake_timeout < 1000) handshake_timeout = 1000;

-    /* Check if we have disconnected nodes and re-establish the connection. */
+    /* Check if we have disconnected nodes and re-establish the connection.
+     * Also update a few stats while we are here, that can be used to make
+     * better decisions in other part of the code. */
    di = dictGetSafeIterator(server.cluster->nodes);
+    server.cluster->stats_pfail_nodes = 0;
    while((de = dictNext(di)) != NULL) {
        clusterNode *node = dictGetVal(de);

+        /* Not interested in reconnecting the link with myself or nodes
+         * for which we have no address. */
        if (node->flags & (CLUSTER_NODE_MYSELF|CLUSTER_NODE_NOADDR)) continue;

+        if (node->flags & CLUSTER_NODE_PFAIL)
+            server.cluster->stats_pfail_nodes++;
+
        /* A Node in HANDSHAKE state has a limited lifespan equal to the
         * configured node timeout. */
        if (nodeInHandshake(node) && now - node->ctime > handshake_timeout) {
@@ -3875,6 +3953,21 @@ sds clusterGenNodesDescription(int filter) {
 * CLUSTER command
 * -------------------------------------------------------------------------- */

+const char *clusterGetMessageTypeString(int type) {
+    switch(type) {
+    case CLUSTERMSG_TYPE_PING: return "ping";
+    case CLUSTERMSG_TYPE_PONG: return "pong";
+    case CLUSTERMSG_TYPE_MEET: return "meet";
+    case CLUSTERMSG_TYPE_FAIL: return "fail";
+    case CLUSTERMSG_TYPE_PUBLISH: return "publish";
+    case CLUSTERMSG_TYPE_FAILOVER_AUTH_REQUEST: return "auth-req";
+    case CLUSTERMSG_TYPE_FAILOVER_AUTH_ACK: return "auth-ack";
+    case CLUSTERMSG_TYPE_UPDATE: return "update";
+    case CLUSTERMSG_TYPE_MFSTART: return "mfstart";
+    }
+    return "unknown";
+}
+
 int getSlotOrReply(client *c, robj *o) {
    long long slot;

@@ -4206,8 +4299,6 @@ void clusterCommand(client *c) {
            "cluster_size:%d\r\n"
            "cluster_current_epoch:%llu\r\n"
            "cluster_my_epoch:%llu\r\n"
-            "cluster_stats_messages_sent:%lld\r\n"
-            "cluster_stats_messages_received:%lld\r\n"
            , statestr[server.cluster->state],
            slots_assigned,
            slots_ok,
@@ -4216,10 +4307,36 @@ void clusterCommand(client *c) {
            dictSize(server.cluster->nodes),
            server.cluster->size,
            (unsigned long long) server.cluster->currentEpoch,
-            (unsigned long long) myepoch,
-            server.cluster->stats_bus_messages_sent,
-            server.cluster->stats_bus_messages_received
+            (unsigned long long) myepoch
        );
+
+        /* Show stats about messages sent and received. */
+        long long tot_msg_sent = 0;
+        long long tot_msg_received = 0;
+
+        for (int i = 0; i < CLUSTERMSG_TYPE_COUNT; i++) {
+            if (server.cluster->stats_bus_messages_sent[i] == 0) continue;
+            tot_msg_sent += server.cluster->stats_bus_messages_sent[i];
+            info = sdscatprintf(info,
+                "cluster_stats_messages_%s_sent:%lld\r\n",
+                clusterGetMessageTypeString(i),
+                server.cluster->stats_bus_messages_sent[i]);
+        }
+        info = sdscatprintf(info,
+            "cluster_stats_messages_sent:%lld\r\n", tot_msg_sent);
+
+        for (int i = 0; i < CLUSTERMSG_TYPE_COUNT; i++) {
+            if (server.cluster->stats_bus_messages_received[i] == 0) continue;
+            tot_msg_received += server.cluster->stats_bus_messages_received[i];
+            info = sdscatprintf(info,
+                "cluster_stats_messages_%s_received:%lld\r\n",
+                clusterGetMessageTypeString(i),
+                server.cluster->stats_bus_messages_received[i]);
+        }
+        info = sdscatprintf(info,
+            "cluster_stats_messages_received:%lld\r\n", tot_msg_received);
+
+        /* Produce the reply protocol. */
        addReplySds(c,sdscatprintf(sdsempty(),"$%lu\r\n",
            (unsigned long)sdslen(info)));
        addReplySds(c,info);
@@ -4617,7 +4734,7 @@ void restoreCommand(client *c) {

    /* Create the key and set the TTL if any */
    dbAdd(c->db,c->argv[1],obj);
-    if (ttl) setExpire(c->db,c->argv[1],mstime()+ttl);
+    if (ttl) setExpire(c,c->db,c->argv[1],mstime()+ttl);
    signalModifiedKey(c->db,c->argv[1]);
    addReply(c,shared.ok);
    server.dirty++;
@@ -4750,13 +4867,13 @@ void migrateCommand(client *c) {
    int copy, replace, j;
    long timeout;
    long dbid;
-    long long ttl, expireat;
    robj **ov = NULL; /* Objects to migrate. */
    robj **kv = NULL; /* Key names. */
    robj **newargv = NULL; /* Used to rewrite the command as DEL ... keys ... */
    rio cmd, payload;
    int may_retry = 1;
    int write_error = 0;
+    int argv_rewritten = 0;

    /* To support the KEYS option we need the following additional state. */
    int first_key = 3; /* Argument index of the first key. */
@@ -4765,7 +4882,6 @@ void migrateCommand(client *c) {
    /* Initialization */
    copy = 0;
    replace = 0;
-    ttl = 0;

    /* Parse additional options */
    for (j = 6; j < c->argc; j++) {
@@ -4841,7 +4957,9 @@ try_again:

    /* Create RESTORE payload and generate the protocol to call the command. */
    for (j = 0; j < num_keys; j++) {
-        expireat = getExpire(c->db,kv[j]);
+        long long ttl = 0;
+        long long expireat = getExpire(c->db,kv[j]);
+
        if (expireat != -1) {
            ttl = expireat-mstime();
            if (ttl < 1) ttl = 1;
@@ -4939,12 +5057,20 @@ try_again:
        goto socket_err; /* A retry is guaranteed because of tested conditions.*/
    }

+    /* On socket errors, close the migration socket now that we still have
+     * the original host/port in the ARGV. Later the original command may be
+     * rewritten to DEL and will be too later. */
+    if (socket_error) migrateCloseSocket(c->argv[1],c->argv[2]);
+
    if (!copy) {
-        /* Translate MIGRATE as DEL for replication/AOF. */
+        /* Translate MIGRATE as DEL for replication/AOF. Note that we do
+         * this only for the keys for which we received an acknowledgement
+         * from the receiving Redis server, by using the del_idx index. */
        if (del_idx > 1) {
            newargv[0] = createStringObject("DEL",3);
            /* Note that the following call takes ownership of newargv. */
            replaceClientCommandVector(c,del_idx,newargv);
+            argv_rewritten = 1;
        } else {
            /* No key transfer acknowledged, no need to rewrite as DEL. */
            zfree(newargv);
@@ -4953,8 +5079,8 @@ try_again:
    }

    /* If we are here and a socket error happened, we don't want to retry.
-     * Just signal the problem to the client, but only do it if we don't
-     * already queued a different error reported by the destination server. */
+     * Just signal the problem to the client, but only do it if we did not
+     * already queue a different error reported by the destination server. */
    if (!error_from_target && socket_error) {
        may_retry = 0;
        goto socket_err;
@@ -4962,7 +5088,11 @@ try_again:

    if (!error_from_target) {
        /* Success! Update the last_dbid in migrateCachedSocket, so that we can
-         * avoid SELECT the next time if the target DB is the same. Reply +OK. */
+         * avoid SELECT the next time if the target DB is the same. Reply +OK.
+         *
+         * Note: If we reached this point, even if socket_error is true
+         * still the SELECT command succeeded (otherwise the code jumps to
+         * socket_err label. */
        cs->last_dbid = dbid;
        addReply(c,shared.ok);
    } else {
@@ -4972,7 +5102,6 @@ try_again:

    sdsfree(cmd.io.buffer.ptr);
    zfree(ov); zfree(kv); zfree(newargv);
-    if (socket_error) migrateCloseSocket(c->argv[1],c->argv[2]);
    return;

 /* On socket errors we try to close the cached socket and try again.
@@ -4982,7 +5111,12 @@ socket_err:
    /* Cleanup we want to perform in both the retry and no retry case.
     * Note: Closing the migrate socket will also force SELECT next time. */
    sdsfree(cmd.io.buffer.ptr);
-    migrateCloseSocket(c->argv[1],c->argv[2]);
+
+    /* If the command was rewritten as DEL and there was a socket error,
+     * we already closed the socket earlier. While migrateCloseSocket()
+     * is idempotent, the host/port arguments are now gone, so don't do it
+     * again. */
+    if (!argv_rewritten) migrateCloseSocket(c->argv[1],c->argv[2]);
    zfree(newargv);
    newargv = NULL; /* This will get reallocated on retry. */

@@ -73,6 +73,29 @@ typedef struct clusterLink {
 #define CLUSTER_CANT_FAILOVER_WAITING_VOTES 4
 #define CLUSTER_CANT_FAILOVER_RELOG_PERIOD (60*5) /* seconds. */

+/* clusterState todo_before_sleep flags. */
+#define CLUSTER_TODO_HANDLE_FAILOVER (1<<0)
+#define CLUSTER_TODO_UPDATE_STATE (1<<1)
+#define CLUSTER_TODO_SAVE_CONFIG (1<<2)
+#define CLUSTER_TODO_FSYNC_CONFIG (1<<3)
+
+/* Message types.
+ *
+ * Note that the PING, PONG and MEET messages are actually the same exact
+ * kind of packet. PONG is the reply to ping, in the exact format as a PING,
+ * while MEET is a special PING that forces the receiver to add the sender
+ * as a node (if it is not already in the list). */
+#define CLUSTERMSG_TYPE_PING 0          /* Ping */
+#define CLUSTERMSG_TYPE_PONG 1          /* Pong (reply to Ping) */
+#define CLUSTERMSG_TYPE_MEET 2          /* Meet "let's join" message */
+#define CLUSTERMSG_TYPE_FAIL 3          /* Mark node xxx as failing */
+#define CLUSTERMSG_TYPE_PUBLISH 4       /* Pub/Sub Publish propagation */
+#define CLUSTERMSG_TYPE_FAILOVER_AUTH_REQUEST 5 /* May I failover? */
+#define CLUSTERMSG_TYPE_FAILOVER_AUTH_ACK 6     /* Yes, you have my vote */
+#define CLUSTERMSG_TYPE_UPDATE 7        /* Another node slots configuration */
+#define CLUSTERMSG_TYPE_MFSTART 8       /* Pause clients for manual failover */
+#define CLUSTERMSG_TYPE_COUNT 9         /* Total number of message types. */
+
 /* This structure represent elements of node->fail_reports. */
 typedef struct clusterNodeFailReport {
    struct clusterNode *node;  /* Node reporting the failure condition. */
@@ -116,7 +139,8 @@ typedef struct clusterState {
    clusterNode *migrating_slots_to[CLUSTER_SLOTS];
    clusterNode *importing_slots_from[CLUSTER_SLOTS];
    clusterNode *slots[CLUSTER_SLOTS];
-    zskiplist *slots_to_keys;
+    uint64_t slots_keys_count[CLUSTER_SLOTS];
+    rax *slots_to_keys;
    /* The following fields are used to take the slave state on elections. */
    mstime_t failover_auth_time; /* Time of previous or next election. */
    int failover_auth_count;    /* Number of votes received so far. */
@@ -138,32 +162,15 @@ typedef struct clusterState {
    /* The followign fields are used by masters to take state on elections. */
    uint64_t lastVoteEpoch;     /* Epoch of the last vote granted. */
    int todo_before_sleep; /* Things to do in clusterBeforeSleep(). */
-    long long stats_bus_messages_sent;  /* Num of msg sent via cluster bus. */
-    long long stats_bus_messages_received; /* Num of msg rcvd via cluster bus.*/
+    /* Messages received and sent by type. */
+    long long stats_bus_messages_sent[CLUSTERMSG_TYPE_COUNT];
+    long long stats_bus_messages_received[CLUSTERMSG_TYPE_COUNT];
+    long long stats_pfail_nodes;    /* Number of nodes in PFAIL status,
+                                       excluding nodes without address. */
 } clusterState;

-/* clusterState todo_before_sleep flags. */
-#define CLUSTER_TODO_HANDLE_FAILOVER (1<<0)
-#define CLUSTER_TODO_UPDATE_STATE (1<<1)
-#define CLUSTER_TODO_SAVE_CONFIG (1<<2)
-#define CLUSTER_TODO_FSYNC_CONFIG (1<<3)
-
 /* Redis cluster messages header */

-/* Note that the PING, PONG and MEET messages are actually the same exact
- * kind of packet. PONG is the reply to ping, in the exact format as a PING,
- * while MEET is a special PING that forces the receiver to add the sender
- * as a node (if it is not already in the list). */
-#define CLUSTERMSG_TYPE_PING 0          /* Ping */
-#define CLUSTERMSG_TYPE_PONG 1          /* Pong (reply to Ping) */
-#define CLUSTERMSG_TYPE_MEET 2          /* Meet "let's join" message */
-#define CLUSTERMSG_TYPE_FAIL 3          /* Mark node xxx as failing */
-#define CLUSTERMSG_TYPE_PUBLISH 4       /* Pub/Sub Publish propagation */
-#define CLUSTERMSG_TYPE_FAILOVER_AUTH_REQUEST 5 /* May I failover? */
-#define CLUSTERMSG_TYPE_FAILOVER_AUTH_ACK 6     /* Yes, you have my vote */
-#define CLUSTERMSG_TYPE_UPDATE 7        /* Another node slots configuration */
-#define CLUSTERMSG_TYPE_MFSTART 8       /* Pause clients for manual failover */
-
 /* Initially we don't know our "name", but we'll find it once we connect
 * to the first node, using the getsockname() function. Then we'll use this
 * address for all the next messages. */
@@ -283,6 +283,10 @@ void loadServerConfigFromString(char *config) {
                }
                fclose(logfp);
            }
+        } else if (!strcasecmp(argv[0],"always-show-logo") && argc == 2) {
+            if ((server.always_show_logo = yesnotoi(argv[1])) == -1) {
+                err = "argument must be 'yes' or 'no'"; goto loaderr;
+            }
        } else if (!strcasecmp(argv[0],"syslog-enabled") && argc == 2) {
            if ((server.syslog_enabled = yesnotoi(argv[1])) == -1) {
                err = "argument must be 'yes' or 'no'"; goto loaderr;
@@ -419,6 +423,10 @@ void loadServerConfigFromString(char *config) {
            if ((server.repl_slave_lazy_flush = yesnotoi(argv[1])) == -1) {
                err = "argument must be 'yes' or 'no'"; goto loaderr;
            }
+        } else if (!strcasecmp(argv[0],"activedefrag") && argc == 2) {
+            if ((server.active_defrag_enabled = yesnotoi(argv[1])) == -1) {
+                err = "argument must be 'yes' or 'no'"; goto loaderr;
+            }
        } else if (!strcasecmp(argv[0],"daemonize") && argc == 2) {
            if ((server.daemonize = yesnotoi(argv[1])) == -1) {
                err = "argument must be 'yes' or 'no'"; goto loaderr;
@@ -495,6 +503,36 @@ void loadServerConfigFromString(char *config) {
            }
            zfree(server.rdb_filename);
            server.rdb_filename = zstrdup(argv[1]);
+        } else if (!strcasecmp(argv[0],"active-defrag-threshold-lower") && argc == 2) {
+            server.active_defrag_threshold_lower = atoi(argv[1]);
+            if (server.active_defrag_threshold_lower < 0) {
+                err = "active-defrag-threshold-lower must be 0 or greater";
+                goto loaderr;
+            }
+        } else if (!strcasecmp(argv[0],"active-defrag-threshold-upper") && argc == 2) {
+            server.active_defrag_threshold_upper = atoi(argv[1]);
+            if (server.active_defrag_threshold_upper < 0) {
+                err = "active-defrag-threshold-upper must be 0 or greater";
+                goto loaderr;
+            }
+        } else if (!strcasecmp(argv[0],"active-defrag-ignore-bytes") && argc == 2) {
+            server.active_defrag_ignore_bytes = memtoll(argv[1], NULL);
+            if (server.active_defrag_ignore_bytes <= 0) {
+                err = "active-defrag-ignore-bytes must above 0";
+                goto loaderr;
+            }
+        } else if (!strcasecmp(argv[0],"active-defrag-cycle-min") && argc == 2) {
+            server.active_defrag_cycle_min = atoi(argv[1]);
+            if (server.active_defrag_cycle_min < 1 || server.active_defrag_cycle_min > 99) {
+                err = "active-defrag-cycle-min must be between 1 and 99";
+                goto loaderr;
+            }
+        } else if (!strcasecmp(argv[0],"active-defrag-cycle-max") && argc == 2) {
+            server.active_defrag_cycle_max = atoi(argv[1]);
+            if (server.active_defrag_cycle_max < 1 || server.active_defrag_cycle_max > 99) {
+                err = "active-defrag-cycle-max must be between 1 and 99";
+                goto loaderr;
+            }
        } else if (!strcasecmp(argv[0],"hash-max-ziplist-entries") && argc == 2) {
            server.hash_max_ziplist_entries = memtoll(argv[1], NULL);
        } else if (!strcasecmp(argv[0],"hash-max-ziplist-value") && argc == 2) {
@@ -967,6 +1005,18 @@ void configSetCommand(client *c) {
      "slave-read-only",server.repl_slave_ro) {
    } config_set_bool_field(
      "activerehashing",server.activerehashing) {
+    } config_set_bool_field(
+      "activedefrag",server.active_defrag_enabled) {
+#ifndef HAVE_DEFRAG
+        if (server.active_defrag_enabled) {
+            server.active_defrag_enabled = 0;
+            addReplyError(c,
+                "Active defragmentation cannot be enabled: it requires a "
+                "Redis server compiled with a modified Jemalloc like the "
+                "one shipped by default with the Redis source distribution");
+            return;
+        }
+#endif
    } config_set_bool_field(
      "protected-mode",server.protected_mode) {
    } config_set_bool_field(
@@ -994,6 +1044,16 @@ void configSetCommand(client *c) {
      "lfu-decay-time",server.lfu_decay_time,0,LLONG_MAX) {
    } config_set_numerical_field(
      "timeout",server.maxidletime,0,LONG_MAX) {
+    } config_set_numerical_field(
+      "active-defrag-threshold-lower",server.active_defrag_threshold_lower,0,1000) {
+    } config_set_numerical_field(
+      "active-defrag-threshold-upper",server.active_defrag_threshold_upper,0,1000) {
+    } config_set_memory_field(
+      "active-defrag-ignore-bytes",server.active_defrag_ignore_bytes) {
+    } config_set_numerical_field(
+      "active-defrag-cycle-min",server.active_defrag_cycle_min,1,99) {
+    } config_set_numerical_field(
+      "active-defrag-cycle-max",server.active_defrag_cycle_max,1,99) {
    } config_set_numerical_field(
      "auto-aof-rewrite-percentage",server.aof_rewrite_perc,0,LLONG_MAX){
    } config_set_numerical_field(
@@ -1162,6 +1222,11 @@ void configGetCommand(client *c) {
    config_get_numerical_field("maxmemory",server.maxmemory);
    config_get_numerical_field("maxmemory-samples",server.maxmemory_samples);
    config_get_numerical_field("timeout",server.maxidletime);
+    config_get_numerical_field("active-defrag-threshold-lower",server.active_defrag_threshold_lower);
+    config_get_numerical_field("active-defrag-threshold-upper",server.active_defrag_threshold_upper);
+    config_get_numerical_field("active-defrag-ignore-bytes",server.active_defrag_ignore_bytes);
+    config_get_numerical_field("active-defrag-cycle-min",server.active_defrag_cycle_min);
+    config_get_numerical_field("active-defrag-cycle-max",server.active_defrag_cycle_max);
    config_get_numerical_field("auto-aof-rewrite-percentage",
            server.aof_rewrite_perc);
    config_get_numerical_field("auto-aof-rewrite-min-size",
@@ -1226,6 +1291,7 @@ void configGetCommand(client *c) {
    config_get_bool_field("rdbcompression", server.rdb_compression);
    config_get_bool_field("rdbchecksum", server.rdb_checksum);
    config_get_bool_field("activerehashing", server.activerehashing);
+    config_get_bool_field("activedefrag", server.active_defrag_enabled);
    config_get_bool_field("protected-mode", server.protected_mode);
    config_get_bool_field("repl-disable-tcp-nodelay",
            server.repl_disable_tcp_nodelay);
@@ -1357,7 +1423,7 @@ void configGetCommand(client *c) {
 /* We use the following dictionary type to store where a configuration
 * option is mentioned in the old configuration file, so it's
 * like "maxmemory" -> list of line numbers (first line is zero). */
-unsigned int dictSdsCaseHash(const void *key);
+uint64_t dictSdsCaseHash(const void *key);
 int dictSdsKeyCaseCompare(void *privdata, const void *key1, const void *key2);
 void dictSdsDestructor(void *privdata, void *val);
 void dictListDestructor(void *privdata, void *val);
@@ -1926,6 +1992,11 @@ int rewriteConfig(char *path) {
    rewriteConfigBytesOption(state,"maxmemory",server.maxmemory,CONFIG_DEFAULT_MAXMEMORY);
    rewriteConfigEnumOption(state,"maxmemory-policy",server.maxmemory_policy,maxmemory_policy_enum,CONFIG_DEFAULT_MAXMEMORY_POLICY);
    rewriteConfigNumericalOption(state,"maxmemory-samples",server.maxmemory_samples,CONFIG_DEFAULT_MAXMEMORY_SAMPLES);
+    rewriteConfigNumericalOption(state,"active-defrag-threshold-lower",server.active_defrag_threshold_lower,CONFIG_DEFAULT_DEFRAG_THRESHOLD_LOWER);
+    rewriteConfigNumericalOption(state,"active-defrag-threshold-upper",server.active_defrag_threshold_upper,CONFIG_DEFAULT_DEFRAG_THRESHOLD_UPPER);
+    rewriteConfigBytesOption(state,"active-defrag-ignore-bytes",server.active_defrag_ignore_bytes,CONFIG_DEFAULT_DEFRAG_IGNORE_BYTES);
+    rewriteConfigNumericalOption(state,"active-defrag-cycle-min",server.active_defrag_cycle_min,CONFIG_DEFAULT_DEFRAG_CYCLE_MIN);
+    rewriteConfigNumericalOption(state,"active-defrag-cycle-max",server.active_defrag_cycle_max,CONFIG_DEFAULT_DEFRAG_CYCLE_MAX);
    rewriteConfigYesNoOption(state,"appendonly",server.aof_state != AOF_OFF,0);
    rewriteConfigStringOption(state,"appendfilename",server.aof_filename,CONFIG_DEFAULT_AOF_FILENAME);
    rewriteConfigEnumOption(state,"appendfsync",server.aof_fsync,aof_fsync_enum,CONFIG_DEFAULT_AOF_FSYNC);
@@ -1952,6 +2023,7 @@ int rewriteConfig(char *path) {
    rewriteConfigNumericalOption(state,"zset-max-ziplist-value",server.zset_max_ziplist_value,OBJ_ZSET_MAX_ZIPLIST_VALUE);
    rewriteConfigNumericalOption(state,"hll-sparse-max-bytes",server.hll_sparse_max_bytes,CONFIG_DEFAULT_HLL_SPARSE_MAX_BYTES);
    rewriteConfigYesNoOption(state,"activerehashing",server.activerehashing,CONFIG_DEFAULT_ACTIVE_REHASHING);
+    rewriteConfigYesNoOption(state,"activedefrag",server.active_defrag_enabled,CONFIG_DEFAULT_ACTIVE_DEFRAG);
    rewriteConfigYesNoOption(state,"protected-mode",server.protected_mode,CONFIG_DEFAULT_PROTECTED_MODE);
    rewriteConfigClientoutputbufferlimitOption(state);
    rewriteConfigNumericalOption(state,"hz",server.hz,CONFIG_DEFAULT_HZ);
@@ -206,4 +206,22 @@ void setproctitle(const char *fmt, ...);
 #endif
 #endif

+/* Make sure we can test for ARM just checking for __arm__, since sometimes
+ * __arm is defined but __arm__ is not. */
+#if defined(__arm) && !defined(__arm__)
+#define __arm__
+#endif
+#if defined (__aarch64__) && !defined(__arm64__)
+#define __arm64__
+#endif
+
+/* Make sure we can test for SPARC just checking for __sparc__. */
+#if defined(__sparc) && !defined(__sparc__)
+#define __sparc__
+#endif
+
+#if defined(__sparc__) || defined(__arm__)
+#define USE_ALIGNED_ACCESS
+#endif
+
 #endif
@@ -190,7 +190,9 @@ void dbOverwrite(redisDb *db, robj *key, robj *val) {
 *
 * 1) The ref count of the value object is incremented.
 * 2) clients WATCHing for the destination key notified.
- * 3) The expire time of the key is reset (the key is made persistent). */
+ * 3) The expire time of the key is reset (the key is made persistent).
+ *
+ * All the new keys in the database should be craeted via this interface. */
 void setKey(redisDb *db, robj *key, robj *val) {
    if (lookupKeyWrite(db,key) == NULL) {
        dbAdd(db,key,val);
@@ -330,6 +332,7 @@ long long emptyDb(int dbnum, int flags, void(callback)(void*)) {
            slotToKeyFlush();
        }
    }
+    if (dbnum == -1) flushSlaveKeysWithExpireList();
    return removed;
 }

@@ -662,7 +665,7 @@ void scanGenericCommand(client *c, robj *o, unsigned long cursor) {
        privdata[0] = keys;
        privdata[1] = o;
        do {
-            cursor = dictScan(ht, cursor, scanCallback, privdata);
+            cursor = dictScan(ht, cursor, scanCallback, NULL, privdata);
        } while (cursor &&
              maxiterations-- &&
              listLength(keys) < (unsigned long)count);
@@ -851,7 +854,7 @@ void renameGenericCommand(client *c, int nx) {
        dbDelete(c->db,c->argv[2]);
    }
    dbAdd(c->db,c->argv[2],o);
-    if (expire != -1) setExpire(c->db,c->argv[2],expire);
+    if (expire != -1) setExpire(c,c->db,c->argv[2],expire);
    dbDelete(c->db,c->argv[1]);
    signalModifiedKey(c->db,c->argv[1]);
    signalModifiedKey(c->db,c->argv[2]);
@@ -917,7 +920,7 @@ void moveCommand(client *c) {
        return;
    }
    dbAdd(dst,c->argv[1],o);
-    if (expire != -1) setExpire(dst,c->argv[1],expire);
+    if (expire != -1) setExpire(c,dst,c->argv[1],expire);
    incrRefCount(o);

    /* OK! key moved, free the entry in the source DB */
@@ -1022,7 +1025,11 @@ int removeExpire(redisDb *db, robj *key) {
    return dictDelete(db->expires,key->ptr) == DICT_OK;
 }

-void setExpire(redisDb *db, robj *key, long long when) {
+/* Set an expire to the specified key. If the expire is set in the context
+ * of an user calling a command 'c' is the client, otherwise 'c' is set
+ * to NULL. The 'when' parameter is the absolute unix time in milliseconds
+ * after which the key will no longer be considered valid. */
+void setExpire(client *c, redisDb *db, robj *key, long long when) {
    dictEntry *kde, *de;

    /* Reuse the sds from the main dict in the expire dict */
@@ -1030,6 +1037,10 @@ void setExpire(redisDb *db, robj *key, long long when) {
    serverAssertWithInfo(NULL,key,kde != NULL);
    de = dictAddOrFind(db->expires,dictGetKey(kde));
    dictSetSignedIntegerVal(de,when);
+
+    int writable_slave = server.masterhost && server.repl_slave_ro == 0;
+    if (c && writable_slave && !(c->flags & CLIENT_MASTER))
+        rememberSlaveKeyWithExpire(db,key);
 }

 /* Return the expire time of the specified key, or -1 if no expire
@@ -1122,11 +1133,24 @@ int *getKeysUsingCommandTable(struct redisCommand *cmd,robj **argv, int argc, in
        *numkeys = 0;
        return NULL;
    }
+
    last = cmd->lastkey;
    if (last < 0) last = argc+last;
    keys = zmalloc(sizeof(int)*((last - cmd->firstkey)+1));
    for (j = cmd->firstkey; j <= last; j += cmd->keystep) {
-        serverAssert(j < argc);
+        if (j >= argc) {
+            /* Modules command do not have dispatch time arity checks, so
+             * we need to handle the case where the user passed an invalid
+             * number of arguments here. In this case we return no keys
+             * and expect the module command to report an arity error. */
+            if (cmd->flags & CMD_MODULE) {
+                zfree(keys);
+                *numkeys = 0;
+                return NULL;
+            } else {
+                serverPanic("Redis built-in command declared keys positions not matching the arity requirements.");
+            }
+        }
        keys[i++] = j;
    }
    *numkeys = i;
@@ -1290,90 +1314,85 @@ int *migrateGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkey

 /* Slot to Key API. This is used by Redis Cluster in order to obtain in
 * a fast way a key that belongs to a specified hash slot. This is useful
- * while rehashing the cluster. */
-void slotToKeyAdd(robj *key) {
+ * while rehashing the cluster and in other conditions when we need to
+ * understand if we have keys for a given hash slot. */
+void slotToKeyUpdateKey(robj *key, int add) {
    unsigned int hashslot = keyHashSlot(key->ptr,sdslen(key->ptr));
+    unsigned char buf[64];
+    unsigned char *indexed = buf;
+    size_t keylen = sdslen(key->ptr);

-    sds sdskey = sdsdup(key->ptr);
-    zslInsert(server.cluster->slots_to_keys,hashslot,sdskey);
+    server.cluster->slots_keys_count[hashslot] += add ? 1 : -1;
+    if (keylen+2 > 64) indexed = zmalloc(keylen+2);
+    indexed[0] = (hashslot >> 8) & 0xff;
+    indexed[1] = hashslot & 0xff;
+    memcpy(indexed+2,key->ptr,keylen);
+    if (add) {
+        raxInsert(server.cluster->slots_to_keys,indexed,keylen+2,NULL,NULL);
+    } else {
+        raxRemove(server.cluster->slots_to_keys,indexed,keylen+2,NULL);
+    }
+    if (indexed != buf) zfree(indexed);
+}
+
+void slotToKeyAdd(robj *key) {
+    slotToKeyUpdateKey(key,1);
 }

 void slotToKeyDel(robj *key) {
-    unsigned int hashslot = keyHashSlot(key->ptr,sdslen(key->ptr));
-    zslDelete(server.cluster->slots_to_keys,hashslot,key->ptr,NULL);
+    slotToKeyUpdateKey(key,0);
 }

 void slotToKeyFlush(void) {
-    zslFree(server.cluster->slots_to_keys);
-    server.cluster->slots_to_keys = zslCreate();
+    raxFree(server.cluster->slots_to_keys);
+    server.cluster->slots_to_keys = raxNew();
+    memset(server.cluster->slots_keys_count,0,
+           sizeof(server.cluster->slots_keys_count));
 }

 /* Pupulate the specified array of objects with keys in the specified slot.
 * New objects are returned to represent keys, it's up to the caller to
 * decrement the reference count to release the keys names. */
 unsigned int getKeysInSlot(unsigned int hashslot, robj **keys, unsigned int count) {
-    zskiplistNode *n;
-    zrangespec range;
+    raxIterator iter;
    int j = 0;
+    unsigned char indexed[2];

-    range.min = range.max = hashslot;
-    range.minex = range.maxex = 0;
-
-    n = zslFirstInRange(server.cluster->slots_to_keys, &range);
-    while(n && n->score == hashslot && count--) {
-        keys[j++] = createStringObject(n->ele,sdslen(n->ele));
-        n = n->level[0].forward;
+    indexed[0] = (hashslot >> 8) & 0xff;
+    indexed[1] = hashslot & 0xff;
+    raxStart(&iter,server.cluster->slots_to_keys);
+    raxSeek(&iter,">=",indexed,2);
+    while(count-- && raxNext(&iter)) {
+        if (iter.key[0] != indexed[0] || iter.key[1] != indexed[1]) break;
+        keys[j++] = createStringObject((char*)iter.key+2,iter.key_len-2);
    }
+    raxStop(&iter);
    return j;
 }

 /* Remove all the keys in the specified hash slot.
 * The number of removed items is returned. */
 unsigned int delKeysInSlot(unsigned int hashslot) {
-    zskiplistNode *n;
-    zrangespec range;
+    raxIterator iter;
    int j = 0;
+    unsigned char indexed[2];

-    range.min = range.max = hashslot;
-    range.minex = range.maxex = 0;
+    indexed[0] = (hashslot >> 8) & 0xff;
+    indexed[1] = hashslot & 0xff;
+    raxStart(&iter,server.cluster->slots_to_keys);
+    while(server.cluster->slots_keys_count[hashslot]) {
+        raxSeek(&iter,">=",indexed,2);
+        raxNext(&iter);

-    n = zslFirstInRange(server.cluster->slots_to_keys, &range);
-    while(n && n->score == hashslot) {
-        sds sdskey = n->ele;
-        robj *key = createStringObject(sdskey,sdslen(sdskey));
-        n = n->level[0].forward; /* Go to the next item before freeing it. */
+        robj *key = createStringObject((char*)iter.key+2,iter.key_len-2);
        dbDelete(&server.db[0],key);
        decrRefCount(key);
        j++;
    }
+    raxStop(&iter);
    return j;
 }

 unsigned int countKeysInSlot(unsigned int hashslot) {
-    zskiplist *zsl = server.cluster->slots_to_keys;
-    zskiplistNode *zn;
-    zrangespec range;
-    int rank, count = 0;
-
-    range.min = range.max = hashslot;
-    range.minex = range.maxex = 0;
-
-    /* Find first element in range */
-    zn = zslFirstInRange(zsl, &range);
-
-    /* Use rank of first element, if any, to determine preliminary count */
-    if (zn != NULL) {
-        rank = zslGetRank(zsl, zn->score, zn->ele);
-        count = (zsl->length - (rank - 1));
-
-        /* Find last element in range */
-        zn = zslLastInRange(zsl, &range);
-
-        /* Use rank of last element, if any, to determine the actual count */
-        if (zn != NULL) {
-            rank = zslGetRank(zsl, zn->score, zn->ele);
-            count -= (zsl->length - rank);
-        }
-    }
-    return count;
+    return server.cluster->slots_keys_count[hashslot];
 }
@@ -126,7 +126,7 @@ void computeDatasetDigest(unsigned char *final) {
        redisDb *db = server.db+j;

        if (dictSize(db->dict) == 0) continue;
-        di = dictGetIterator(db->dict);
+        di = dictGetSafeIterator(db->dict);

        /* hash the DB id, so the same dataset moved in a different
         * DB will lead to a different digest */
@@ -266,6 +266,8 @@ void debugCommand(client *c) {
        blen++; addReplyStatus(c,
        "segfault -- Crash the server with sigsegv.");
        blen++; addReplyStatus(c,
+        "panic -- Crash the server simulating a panic.");
+        blen++; addReplyStatus(c,
        "restart  -- Graceful restart: save config, db, restart.");
        blen++; addReplyStatus(c,
        "crash-and-recovery <milliseconds> -- Hard crash and restart after <milliseconds> delay.");
@@ -280,7 +282,9 @@ void debugCommand(client *c) {
        blen++; addReplyStatus(c,
        "sdslen <key> -- Show low level SDS string info representing key and value.");
        blen++; addReplyStatus(c,
-        "populate <count> [prefix] -- Create <count> string keys named key:<num>. If a prefix is specified is used instead of the 'key' prefix.");
+        "ziplist <key> -- Show low level info about the ziplist encoding.");
+        blen++; addReplyStatus(c,
+        "populate <count> [prefix] [size] -- Create <count> string keys named key:<num>. If a prefix is specified is used instead of the 'key' prefix.");
        blen++; addReplyStatus(c,
        "digest   -- Outputs an hex signature representing the current DB content.");
        blen++; addReplyStatus(c,
@@ -298,6 +302,8 @@ void debugCommand(client *c) {
        setDeferredMultiBulkLength(c,blenp,blen);
    } else if (!strcasecmp(c->argv[1]->ptr,"segfault")) {
        *((char*)-1) = 'x';
+    } else if (!strcasecmp(c->argv[1]->ptr,"panic")) {
+        serverPanic("DEBUG PANIC called at Unix time %ld", time(NULL));
    } else if (!strcasecmp(c->argv[1]->ptr,"restart") ||
               !strcasecmp(c->argv[1]->ptr,"crash-and-recover"))
    {
@@ -418,8 +424,20 @@ void debugCommand(client *c) {
                (long long) sdsavail(val->ptr),
                (long long) getStringObjectSdsUsedMemory(val));
        }
+    } else if (!strcasecmp(c->argv[1]->ptr,"ziplist") && c->argc == 3) {
+        robj *o;
+
+        if ((o = objectCommandLookupOrReply(c,c->argv[2],shared.nokeyerr))
+                == NULL) return;
+
+        if (o->encoding != OBJ_ENCODING_ZIPLIST) {
+            addReplyError(c,"Not an sds encoded string.");
+        } else {
+            ziplistRepr(o->ptr);
+            addReplyStatus(c,"Ziplist structure printed on stdout");
+        }
    } else if (!strcasecmp(c->argv[1]->ptr,"populate") &&
-               (c->argc == 3 || c->argc == 4)) {
+               c->argc >= 3 && c->argc <= 5) {
        long keys, j;
        robj *key, *val;
        char buf[128];
@@ -428,15 +446,25 @@ void debugCommand(client *c) {
            return;
        dictExpand(c->db->dict,keys);
        for (j = 0; j < keys; j++) {
+            long valsize = 0;
            snprintf(buf,sizeof(buf),"%s:%lu",
                (c->argc == 3) ? "key" : (char*)c->argv[3]->ptr, j);
            key = createStringObject(buf,strlen(buf));
+            if (c->argc == 5)
+                if (getLongFromObjectOrReply(c, c->argv[4], &valsize, NULL) != C_OK)
+                    return;
            if (lookupKeyWrite(c->db,key) != NULL) {
                decrRefCount(key);
                continue;
            }
            snprintf(buf,sizeof(buf),"value:%lu",j);
-            val = createStringObject(buf,strlen(buf));
+            if (valsize==0)
+                val = createStringObject(buf,strlen(buf));
+            else {
+                int buflen = strlen(buf);
+                val = createStringObject(NULL,valsize);
+                memcpy(val->ptr, buf, valsize<=buflen? valsize: buflen);
+            }
            dbAdd(c->db,key,val);
            signalModifiedKey(c->db,key);
            decrRefCount(key);
@@ -591,11 +619,17 @@ void _serverAssertWithInfo(const client *c, const robj *o, const char *estr, con
    _serverAssert(estr,file,line);
 }

-void _serverPanic(const char *msg, const char *file, int line) {
+void _serverPanic(const char *file, int line, const char *msg, ...) {
+    va_list ap;
+    va_start(ap,msg);
+    char fmtmsg[256];
+    vsnprintf(fmtmsg,sizeof(fmtmsg),msg,ap);
+    va_end(ap);
+
    bugReportStart();
    serverLog(LL_WARNING,"------------------------------------------------");
    serverLog(LL_WARNING,"!!! Software Failure. Press left mouse button to continue");
-    serverLog(LL_WARNING,"Guru Meditation: %s #%s:%d",msg,file,line);
+    serverLog(LL_WARNING,"Guru Meditation: %s #%s:%d",fmtmsg,file,line);
 #ifdef HAVE_BACKTRACE
    serverLog(LL_WARNING,"(forcing SIGSEGV in order to print the stack trace)");
 #endif
@@ -995,8 +1029,6 @@ void sigsegvHandler(int sig, siginfo_t *info, void *secret) {
    /* Log INFO and CLIENT LIST */
    serverLogRaw(LL_WARNING|LL_RAW, "\n------ INFO OUTPUT ------\n");
    infostring = genRedisInfoString("all");
-    infostring = sdscatprintf(infostring, "hash_init_value: %u\n",
-        dictGetHashFunctionSeed());
    serverLogRaw(LL_WARNING|LL_RAW, infostring);
    serverLogRaw(LL_WARNING|LL_RAW, "\n------ CLIENT LIST OUTPUT ------\n");
    clients = getAllClientsInfoString();
@@ -0,0 +1,579 @@
+/* 
+ * Active memory defragmentation
+ * Try to find key / value allocations that need to be re-allocated in order 
+ * to reduce external fragmentation.
+ * We do that by scanning the keyspace and for each pointer we have, we can try to
+ * ask the allocator if moving it to a new address will help reduce fragmentation.
+ *
+ * Copyright (c) 2017, Oran Agra
+ * Copyright (c) 2017, Redis Labs, Inc
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include "server.h"
+#include <time.h>
+#include <assert.h>
+#include <stddef.h>
+
+#ifdef HAVE_DEFRAG
+
+/* this method was added to jemalloc in order to help us understand which
+ * pointers are worthwhile moving and which aren't */
+int je_get_defrag_hint(void* ptr, int *bin_util, int *run_util);
+
+/* Defrag helper for generic allocations.
+ *
+ * returns NULL in case the allocatoin wasn't moved.
+ * when it returns a non-null value, the old pointer was already released
+ * and should NOT be accessed. */
+void* activeDefragAlloc(void *ptr) {
+    int bin_util, run_util;
+    size_t size;
+    void *newptr;
+    if(!je_get_defrag_hint(ptr, &bin_util, &run_util)) {
+        server.stat_active_defrag_misses++;
+        return NULL;
+    }
+    /* if this run is more utilized than the average utilization in this bin
+     * (or it is full), skip it. This will eventually move all the allocations
+     * from relatively empty runs into relatively full runs. */
+    if (run_util > bin_util || run_util == 1<<16) {
+        server.stat_active_defrag_misses++;
+        return NULL;
+    }
+    /* move this allocation to a new allocation.
+     * make sure not to use the thread cache. so that we don't get back the same
+     * pointers we try to free */
+    size = zmalloc_size(ptr);
+    newptr = zmalloc_no_tcache(size);
+    memcpy(newptr, ptr, size);
+    zfree_no_tcache(ptr);
+    return newptr;
+}
+
+/*Defrag helper for sds strings
+ *
+ * returns NULL in case the allocatoin wasn't moved.
+ * when it returns a non-null value, the old pointer was already released
+ * and should NOT be accessed. */
+sds activeDefragSds(sds sdsptr) {
+    void* ptr = sdsAllocPtr(sdsptr);
+    void* newptr = activeDefragAlloc(ptr);
+    if (newptr) {
+        size_t offset = sdsptr - (char*)ptr;
+        sdsptr = (char*)newptr + offset;
+        return sdsptr;
+    }
+    return NULL;
+}
+
+/* Defrag helper for robj and/or string objects
+ *
+ * returns NULL in case the allocatoin wasn't moved.
+ * when it returns a non-null value, the old pointer was already released
+ * and should NOT be accessed. */
+robj *activeDefragStringOb(robj* ob, int *defragged) {
+    robj *ret = NULL;
+    if (ob->refcount!=1)
+        return NULL;
+
+    /* try to defrag robj (only if not an EMBSTR type (handled below). */
+    if (ob->type!=OBJ_STRING || ob->encoding!=OBJ_ENCODING_EMBSTR) {
+        if ((ret = activeDefragAlloc(ob))) {
+            ob = ret;
+            (*defragged)++;
+        }
+    }
+
+    /* try to defrag string object */
+    if (ob->type == OBJ_STRING) {
+        if(ob->encoding==OBJ_ENCODING_RAW) {
+            sds newsds = activeDefragSds((sds)ob->ptr);
+            if (newsds) {
+                ob->ptr = newsds;
+                (*defragged)++;
+            }
+        } else if (ob->encoding==OBJ_ENCODING_EMBSTR) {
+            /* The sds is embedded in the object allocation, calculate the
+             * offset and update the pointer in the new allocation. */
+            long ofs = (intptr_t)ob->ptr - (intptr_t)ob;
+            if ((ret = activeDefragAlloc(ob))) {
+                ret->ptr = (void*)((intptr_t)ret + ofs);
+                (*defragged)++;
+            }
+        } else if (ob->encoding!=OBJ_ENCODING_INT) {
+            serverPanic("Unknown string encoding");
+        }
+    }
+    return ret;
+}
+
+/* Defrag helper for dictEntries to be used during dict iteration (called on
+ * each step). Teturns a stat of how many pointers were moved. */
+int dictIterDefragEntry(dictIterator *iter) {
+    /* This function is a little bit dirty since it messes with the internals
+     * of the dict and it's iterator, but the benefit is that it is very easy
+     * to use, and require no other chagnes in the dict. */
+    int defragged = 0;
+    dictht *ht;
+    /* Handle the next entry (if there is one), and update the pointer in the
+     * current entry. */
+    if (iter->nextEntry) {
+        dictEntry *newde = activeDefragAlloc(iter->nextEntry);
+        if (newde) {
+            defragged++;
+            iter->nextEntry = newde;
+            iter->entry->next = newde;
+        }
+    }
+    /* handle the case of the first entry in the hash bucket. */
+    ht = &iter->d->ht[iter->table];
+    if (ht->table[iter->index] == iter->entry) {
+        dictEntry *newde = activeDefragAlloc(iter->entry);
+        if (newde) {
+            iter->entry = newde;
+            ht->table[iter->index] = newde;
+            defragged++;
+        }
+    }
+    return defragged;
+}
+
+/* Defrag helper for dict main allocations (dict struct, and hash tables).
+ * receives a pointer to the dict* and implicitly updates it when the dict
+ * struct itself was moved. Returns a stat of how many pointers were moved. */
+int dictDefragTables(dict** dictRef) {
+    dict *d = *dictRef;
+    dictEntry **newtable;
+    int defragged = 0;
+    /* handle the dict struct */
+    dict *newd = activeDefragAlloc(d);
+    if (newd)
+        defragged++, *dictRef = d = newd;
+    /* handle the first hash table */
+    newtable = activeDefragAlloc(d->ht[0].table);
+    if (newtable)
+        defragged++, d->ht[0].table = newtable;
+    /* handle the second hash table */
+    if (d->ht[1].table) {
+        newtable = activeDefragAlloc(d->ht[1].table);
+        if (newtable)
+            defragged++, d->ht[1].table = newtable;
+    }
+    return defragged;
+}
+
+/* Internal function used by zslDefrag */
+void zslUpdateNode(zskiplist *zsl, zskiplistNode *oldnode, zskiplistNode *newnode, zskiplistNode **update) {
+    int i;
+    for (i = 0; i < zsl->level; i++) {
+        if (update[i]->level[i].forward == oldnode)
+            update[i]->level[i].forward = newnode;
+    }
+    serverAssert(zsl->header!=oldnode);
+    if (newnode->level[0].forward) {
+        serverAssert(newnode->level[0].forward->backward==oldnode);
+        newnode->level[0].forward->backward = newnode;
+    } else {
+        serverAssert(zsl->tail==oldnode);
+        zsl->tail = newnode;
+    }
+}
+
+/* Defrag helper for sorted set.
+ * Update the robj pointer, defrag the skiplist struct and return the new score
+ * reference. We may not access oldele pointer (not even the pointer stored in
+ * the skiplist), as it was already freed. Newele may be null, in which case we
+ * only need to defrag the skiplist, but not update the obj pointer.
+ * When return value is non-NULL, it is the score reference that must be updated
+ * in the dict record. */
+double *zslDefrag(zskiplist *zsl, double score, sds oldele, sds newele) {
+    zskiplistNode *update[ZSKIPLIST_MAXLEVEL], *x, *newx;
+    int i;
+    sds ele = newele? newele: oldele;
+
+    /* find the skiplist node referring to the object that was moved,
+     * and all pointers that need to be updated if we'll end up moving the skiplist node. */
+    x = zsl->header;
+    for (i = zsl->level-1; i >= 0; i--) {
+        while (x->level[i].forward &&
+            x->level[i].forward->ele != oldele && /* make sure not to access the
+                                                     ->obj pointer if it matches
+                                                     oldele */
+            (x->level[i].forward->score < score ||
+                (x->level[i].forward->score == score &&
+                sdscmp(x->level[i].forward->ele,ele) < 0)))
+            x = x->level[i].forward;
+        update[i] = x;
+    }
+
+    /* update the robj pointer inside the skip list record. */
+    x = x->level[0].forward;
+    serverAssert(x && score == x->score && x->ele==oldele);
+    if (newele)
+        x->ele = newele;
+
+    /* try to defrag the skiplist record itself */
+    newx = activeDefragAlloc(x);
+    if (newx) {
+        zslUpdateNode(zsl, x, newx, update);
+        return &newx->score;
+    }
+    return NULL;
+}
+
+/* Utility function that replaces an old key pointer in the dictionary with a
+ * new pointer. Additionally, we try to defrag the dictEntry in that dict.
+ * Oldkey mey be a dead pointer and should not be accessed (we get a
+ * pre-calculated hash value). Newkey may be null if the key pointer wasn't
+ * moved. Return value is the the dictEntry if found, or NULL if not found.
+ * NOTE: this is very ugly code, but it let's us avoid the complication of
+ * doing a scan on another dict. */
+dictEntry* replaceSateliteDictKeyPtrAndOrDefragDictEntry(dict *d, sds oldkey, sds newkey, unsigned int hash, int *defragged) {
+    dictEntry **deref = dictFindEntryRefByPtrAndHash(d, oldkey, hash);
+    if (deref) {
+        dictEntry *de = *deref;
+        dictEntry *newde = activeDefragAlloc(de);
+        if (newde) {
+            de = *deref = newde;
+            (*defragged)++;
+        }
+        if (newkey)
+            de->key = newkey;
+        return de;
+    }
+    return NULL;
+}
+
+/* for each key we scan in the main dict, this function will attempt to defrag
+ * all the various pointers it has. Returns a stat of how many pointers were
+ * moved. */
+int defragKey(redisDb *db, dictEntry *de) {
+    sds keysds = dictGetKey(de);
+    robj *newob, *ob;
+    unsigned char *newzl;
+    dict *d;
+    dictIterator *di;
+    int defragged = 0;
+    sds newsds;
+
+    /* Try to defrag the key name. */
+    newsds = activeDefragSds(keysds);
+    if (newsds)
+        defragged++, de->key = newsds;
+    if (dictSize(db->expires)) {
+         /* Dirty code:
+          * I can't search in db->expires for that key after i already released
+          * the pointer it holds it won't be able to do the string compare */
+        unsigned int hash = dictGetHash(db->dict, de->key);
+        replaceSateliteDictKeyPtrAndOrDefragDictEntry(db->expires, keysds, newsds, hash, &defragged);
+    }
+
+    /* Try to defrag robj and / or string value. */
+    ob = dictGetVal(de);
+    if ((newob = activeDefragStringOb(ob, &defragged))) {
+        de->v.val = newob;
+        ob = newob;
+    }
+
+    if (ob->type == OBJ_STRING) {
+        /* Already handled in activeDefragStringOb. */
+    } else if (ob->type == OBJ_LIST) {
+        if (ob->encoding == OBJ_ENCODING_QUICKLIST) {
+            quicklist *ql = ob->ptr, *newql;
+            quicklistNode *node = ql->head, *newnode;
+            if ((newql = activeDefragAlloc(ql)))
+                defragged++, ob->ptr = ql = newql;
+            while (node) {
+                if ((newnode = activeDefragAlloc(node))) {
+                    if (newnode->prev)
+                        newnode->prev->next = newnode;
+                    else
+                        ql->head = newnode;
+                    if (newnode->next)
+                        newnode->next->prev = newnode;
+                    else
+                        ql->tail = newnode;
+                    node = newnode;
+                    defragged++;
+                }
+                if ((newzl = activeDefragAlloc(node->zl)))
+                    defragged++, node->zl = newzl;
+                node = node->next;
+            }
+        } else if (ob->encoding == OBJ_ENCODING_ZIPLIST) {
+            if ((newzl = activeDefragAlloc(ob->ptr)))
+                defragged++, ob->ptr = newzl;
+        } else {
+            serverPanic("Unknown list encoding");
+        }
+    } else if (ob->type == OBJ_SET) {
+        if (ob->encoding == OBJ_ENCODING_HT) {
+            d = ob->ptr;
+            di = dictGetIterator(d);
+            while((de = dictNext(di)) != NULL) {
+                sds sdsele = dictGetKey(de);
+                if ((newsds = activeDefragSds(sdsele)))
+                    defragged++, de->key = newsds;
+                defragged += dictIterDefragEntry(di);
+            }
+            dictReleaseIterator(di);
+            dictDefragTables((dict**)&ob->ptr);
+        } else if (ob->encoding == OBJ_ENCODING_INTSET) {
+            intset *is = ob->ptr;
+            intset *newis = activeDefragAlloc(is);
+            if (newis)
+                defragged++, ob->ptr = newis;
+        } else {
+            serverPanic("Unknown set encoding");
+        }
+    } else if (ob->type == OBJ_ZSET) {
+        if (ob->encoding == OBJ_ENCODING_ZIPLIST) {
+            if ((newzl = activeDefragAlloc(ob->ptr)))
+                defragged++, ob->ptr = newzl;
+        } else if (ob->encoding == OBJ_ENCODING_SKIPLIST) {
+            zset *zs = (zset*)ob->ptr;
+            zset *newzs;
+            zskiplist *newzsl;
+            struct zskiplistNode *newheader;
+            if ((newzs = activeDefragAlloc(zs)))
+                defragged++, ob->ptr = zs = newzs;
+            if ((newzsl = activeDefragAlloc(zs->zsl)))
+                defragged++, zs->zsl = newzsl;
+            if ((newheader = activeDefragAlloc(zs->zsl->header)))
+                defragged++, zs->zsl->header = newheader;
+            d = zs->dict;
+            di = dictGetIterator(d);
+            while((de = dictNext(di)) != NULL) {
+                double* newscore;
+                sds sdsele = dictGetKey(de);
+                if ((newsds = activeDefragSds(sdsele)))
+                    defragged++, de->key = newsds;
+                newscore = zslDefrag(zs->zsl, *(double*)dictGetVal(de), sdsele, newsds);
+                if (newscore) {
+                    dictSetVal(d, de, newscore);
+                    defragged++;
+                }
+                defragged += dictIterDefragEntry(di);
+            }
+            dictReleaseIterator(di);
+            dictDefragTables(&zs->dict);
+        } else {
+            serverPanic("Unknown sorted set encoding");
+        }
+    } else if (ob->type == OBJ_HASH) {
+        if (ob->encoding == OBJ_ENCODING_ZIPLIST) {
+            if ((newzl = activeDefragAlloc(ob->ptr)))
+                defragged++, ob->ptr = newzl;
+        } else if (ob->encoding == OBJ_ENCODING_HT) {
+            d = ob->ptr;
+            di = dictGetIterator(d);
+            while((de = dictNext(di)) != NULL) {
+                sds sdsele = dictGetKey(de);
+                if ((newsds = activeDefragSds(sdsele)))
+                    defragged++, de->key = newsds;
+                sdsele = dictGetVal(de);
+                if ((newsds = activeDefragSds(sdsele)))
+                    defragged++, de->v.val = newsds;
+                defragged += dictIterDefragEntry(di);
+            }
+            dictReleaseIterator(di);
+            dictDefragTables((dict**)&ob->ptr);
+        } else {
+            serverPanic("Unknown hash encoding");
+        }
+    } else if (ob->type == OBJ_MODULE) {
+        /* Currently defragmenting modules private data types
+         * is not supported. */
+    } else {
+        serverPanic("Unknown object type");
+    }
+    return defragged;
+}
+
+/* Defrag scan callback for the main db dictionary. */
+void defragScanCallback(void *privdata, const dictEntry *de) {
+    int defragged = defragKey((redisDb*)privdata, (dictEntry*)de);
+    server.stat_active_defrag_hits += defragged;
+    if(defragged)
+        server.stat_active_defrag_key_hits++;
+    else
+        server.stat_active_defrag_key_misses++;
+}
+
+/* Defrag scan callback for for each hash table bicket,
+ * used in order to defrag the dictEntry allocations. */
+void defragDictBucketCallback(void *privdata, dictEntry **bucketref) {
+    UNUSED(privdata);
+    while(*bucketref) {
+        dictEntry *de = *bucketref, *newde;
+        if ((newde = activeDefragAlloc(de))) {
+            *bucketref = newde;
+        }
+        bucketref = &(*bucketref)->next;
+    }
+}
+
+/* Utility function to get the fragmentation ratio from jemalloc.
+ * It is critical to do that by comparing only heap maps that belown to
+ * jemalloc, and skip ones the jemalloc keeps as spare. Since we use this
+ * fragmentation ratio in order to decide if a defrag action should be taken
+ * or not, a false detection can cause the defragmenter to waste a lot of CPU
+ * without the possibility of getting any results. */
+float getAllocatorFragmentation(size_t *out_frag_bytes) {
+    size_t epoch = 1, allocated = 0, resident = 0, active = 0, sz = sizeof(size_t);
+    /* Update the statistics cached by mallctl. */
+    je_mallctl("epoch", &epoch, &sz, &epoch, sz);
+    /* Unlike RSS, this does not include RSS from shared libraries and other non
+     * heap mappings. */
+    je_mallctl("stats.resident", &resident, &sz, NULL, 0);
+    /* Unlike resident, this doesn't not include the pages jemalloc reserves
+     * for re-use (purge will clean that). */
+    je_mallctl("stats.active", &active, &sz, NULL, 0);
+    /* Unlike zmalloc_used_memory, this matches the stats.resident by taking
+     * into account all allocations done by this process (not only zmalloc). */
+    je_mallctl("stats.allocated", &allocated, &sz, NULL, 0);
+    float frag_pct = ((float)active / allocated)*100 - 100;
+    size_t frag_bytes = active - allocated;
+    float rss_pct = ((float)resident / allocated)*100 - 100;
+    size_t rss_bytes = resident - allocated;
+    if(out_frag_bytes)
+        *out_frag_bytes = frag_bytes;
+    serverLog(LL_DEBUG,
+        "allocated=%zu, active=%zu, resident=%zu, frag=%.0f%% (%.0f%% rss), frag_bytes=%zu (%zu%% rss)",
+        allocated, active, resident, frag_pct, rss_pct, frag_bytes, rss_bytes);
+    return frag_pct;
+}
+
+#define INTERPOLATE(x, x1, x2, y1, y2) ( (y1) + ((x)-(x1)) * ((y2)-(y1)) / ((x2)-(x1)) )
+#define LIMIT(y, min, max) ((y)<(min)? min: ((y)>(max)? max: (y)))
+
+/* Perform incremental defragmentation work from the serverCron.
+ * This works in a similar way to activeExpireCycle, in the sense that
+ * we do incremental work across calls. */
+void activeDefragCycle(void) {
+    static int current_db = -1;
+    static unsigned long cursor = 0;
+    static redisDb *db = NULL;
+    static long long start_scan, start_stat;
+    unsigned int iterations = 0;
+    unsigned long long defragged = server.stat_active_defrag_hits;
+    long long start, timelimit;
+
+    if (server.aof_child_pid!=-1 || server.rdb_child_pid!=-1)
+        return; /* Defragging memory while there's a fork will just do damage. */
+
+    /* Once a second, check if we the fragmentation justfies starting a scan
+     * or making it more aggressive. */
+    run_with_period(1000) {
+        size_t frag_bytes;
+        float frag_pct = getAllocatorFragmentation(&frag_bytes);
+        /* If we're not already running, and below the threshold, exit. */
+        if (!server.active_defrag_running) {
+            if(frag_pct < server.active_defrag_threshold_lower || frag_bytes < server.active_defrag_ignore_bytes)
+                return;
+        }
+
+        /* Calculate the adaptive aggressiveness of the defrag */
+        int cpu_pct = INTERPOLATE(frag_pct,
+                server.active_defrag_threshold_lower,
+                server.active_defrag_threshold_upper,
+                server.active_defrag_cycle_min,
+                server.active_defrag_cycle_max);
+        cpu_pct = LIMIT(cpu_pct,
+                server.active_defrag_cycle_min,
+                server.active_defrag_cycle_max);
+         /* We allow increasing the aggressiveness during a scan, but don't
+          * reduce it. */
+        if (!server.active_defrag_running ||
+            cpu_pct > server.active_defrag_running)
+        {
+            server.active_defrag_running = cpu_pct;
+            serverLog(LL_VERBOSE,
+                "Starting active defrag, frag=%.0f%%, frag_bytes=%zu, cpu=%d%%",
+                frag_pct, frag_bytes, cpu_pct);
+        }
+    }
+    if (!server.active_defrag_running)
+        return;
+
+    /* See activeExpireCycle for how timelimit is handled. */
+    start = ustime();
+    timelimit = 1000000*server.active_defrag_running/server.hz/100;
+    if (timelimit <= 0) timelimit = 1;
+
+    do {
+        if (!cursor) {
+            /* Move on to next database, and stop if we reached the last one. */
+            if (++current_db >= server.dbnum) {
+                long long now = ustime();
+                size_t frag_bytes;
+                float frag_pct = getAllocatorFragmentation(&frag_bytes);
+                serverLog(LL_VERBOSE,
+                    "Active defrag done in %dms, reallocated=%d, frag=%.0f%%, frag_bytes=%zu",
+                    (int)((now - start_scan)/1000), (int)(server.stat_active_defrag_hits - start_stat), frag_pct, frag_bytes);
+
+                start_scan = now;
+                current_db = -1;
+                cursor = 0;
+                db = NULL;
+                server.active_defrag_running = 0;
+                return;
+            }
+            else if (current_db==0) {
+                /* Start a scan from the first database. */
+                start_scan = ustime();
+                start_stat = server.stat_active_defrag_hits;
+            }
+
+            db = &server.db[current_db];
+            cursor = 0;
+        }
+
+        do {
+            cursor = dictScan(db->dict, cursor, defragScanCallback, defragDictBucketCallback, db);
+            /* Once in 16 scan iterations, or 1000 pointer reallocations
+             * (if we have a lot of pointers in one hash bucket), check if we
+             * reached the tiem limit. */
+            if (cursor && (++iterations > 16 || server.stat_active_defrag_hits - defragged > 1000)) {
+                if ((ustime() - start) > timelimit) {
+                    return;
+                }
+                iterations = 0;
+                defragged = server.stat_active_defrag_hits;
+            }
+        } while(cursor);
+    } while(1);
+}
+
+#else /* HAVE_DEFRAG */
+
+void activeDefragCycle(void) {
+    /* Not implemented yet. */
+}
+
+#endif
@@ -37,11 +37,11 @@

 #include <stdio.h>
 #include <stdlib.h>
+#include <stdint.h>
 #include <string.h>
 #include <stdarg.h>
 #include <limits.h>
 #include <sys/time.h>
-#include <ctype.h>

 #include "dict.h"
 #include "zmalloc.h"
@@ -71,77 +71,28 @@ static int _dictInit(dict *ht, dictType *type, void *privDataPtr);

 /* -------------------------- hash functions -------------------------------- */

-static uint32_t dict_hash_function_seed = 5381;
+static uint8_t dict_hash_function_seed[16];

-void dictSetHashFunctionSeed(uint32_t seed) {
-    dict_hash_function_seed = seed;
+void dictSetHashFunctionSeed(uint8_t *seed) {
+    memcpy(dict_hash_function_seed,seed,sizeof(dict_hash_function_seed));
 }

-uint32_t dictGetHashFunctionSeed(void) {
+uint8_t *dictGetHashFunctionSeed(void) {
    return dict_hash_function_seed;
 }

-/* MurmurHash2, by Austin Appleby
- * Note - This code makes a few assumptions about how your machine behaves -
- * 1. We can read a 4-byte value from any address without crashing
- * 2. sizeof(int) == 4
- *
- * And it has a few limitations -
- *
- * 1. It will not work incrementally.
- * 2. It will not produce the same results on little-endian and big-endian
- *    machines.
- */
-unsigned int dictGenHashFunction(const void *key, int len) {
-    /* 'm' and 'r' are mixing constants generated offline.
-     They're not really 'magic', they just happen to work well.  */
-    uint32_t seed = dict_hash_function_seed;
-    const uint32_t m = 0x5bd1e995;
-    const int r = 24;
+/* The default hashing function uses SipHash implementation
+ * in siphash.c. */

-    /* Initialize the hash to a 'random' value */
-    uint32_t h = seed ^ len;
+uint64_t siphash(const uint8_t *in, const size_t inlen, const uint8_t *k);
+uint64_t siphash_nocase(const uint8_t *in, const size_t inlen, const uint8_t *k);

-    /* Mix 4 bytes at a time into the hash */
-    const unsigned char *data = (const unsigned char *)key;
-
-    while(len >= 4) {
-        uint32_t k = *(uint32_t*)data;
-
-        k *= m;
-        k ^= k >> r;
-        k *= m;
-
-        h *= m;
-        h ^= k;
-
-        data += 4;
-        len -= 4;
-    }
-
-    /* Handle the last few bytes of the input array  */
-    switch(len) {
-    case 3: h ^= data[2] << 16;
-    case 2: h ^= data[1] << 8;
-    case 1: h ^= data[0]; h *= m;
-    };
-
-    /* Do a few final mixes of the hash to ensure the last few
-     * bytes are well-incorporated. */
-    h ^= h >> 13;
-    h *= m;
-    h ^= h >> 15;
-
-    return (unsigned int)h;
+uint64_t dictGenHashFunction(const void *key, int len) {
+    return siphash(key,len,dict_hash_function_seed);
 }

-/* And a case insensitive hash function (based on djb hash) */
-unsigned int dictGenCaseHashFunction(const unsigned char *buf, int len) {
-    unsigned int hash = (unsigned int)dict_hash_function_seed;
-
-    while (len--)
-        hash = ((hash << 5) + hash) + (tolower(*buf++)); /* hash * 33 + c */
-    return hash;
+uint64_t dictGenCaseHashFunction(const unsigned char *buf, int len) {
+    return siphash_nocase(buf,len,dict_hash_function_seed);
 }

 /* ----------------------------- API implementation ------------------------- */
@@ -885,6 +836,7 @@ static unsigned long rev(unsigned long v) {
 unsigned long dictScan(dict *d,
                       unsigned long v,
                       dictScanFunction *fn,
+                       dictScanBucketFunction* bucketfn,
                       void *privdata)
 {
    dictht *t0, *t1;
@@ -898,6 +850,7 @@ unsigned long dictScan(dict *d,
        m0 = t0->sizemask;

        /* Emit entries at cursor */
+        if (bucketfn) bucketfn(privdata, &t0->table[v & m0]);
        de = t0->table[v & m0];
        while (de) {
            next = de->next;
@@ -919,6 +872,7 @@ unsigned long dictScan(dict *d,
        m1 = t1->sizemask;

        /* Emit entries at cursor */
+        if (bucketfn) bucketfn(privdata, &t0->table[v & m0]);
        de = t0->table[v & m0];
        while (de) {
            next = de->next;
@@ -930,6 +884,7 @@ unsigned long dictScan(dict *d,
         * of the index pointed to by the cursor in the smaller table */
        do {
            /* Emit entries at cursor */
+            if (bucketfn) bucketfn(privdata, &t1->table[v & m1]);
            de = t1->table[v & m1];
            while (de) {
                next = de->next;
@@ -1040,6 +995,35 @@ void dictDisableResize(void) {
    dict_can_resize = 0;
 }

+unsigned int dictGetHash(dict *d, const void *key) {
+    return dictHashKey(d, key);
+}
+
+/* Finds the dictEntry reference by using pointer and pre-calculated hash.
+ * oldkey is a dead pointer and should not be accessed.
+ * the hash value should be provided using dictGetHash.
+ * no string / key comparison is performed.
+ * return value is the reference to the dictEntry if found, or NULL if not found. */
+dictEntry **dictFindEntryRefByPtrAndHash(dict *d, const void *oldptr, unsigned int hash) {
+    dictEntry *he, **heref;
+    unsigned int idx, table;
+
+    if (d->ht[0].used + d->ht[1].used == 0) return NULL; /* dict is empty */
+    for (table = 0; table <= 1; table++) {
+        idx = hash & d->ht[table].sizemask;
+        heref = &d->ht[table].table[idx];
+        he = *heref;
+        while(he) {
+            if (oldptr==he->key)
+                return heref;
+            heref = &he->next;
+            he = *heref;
+        }
+        if (!dictIsRehashing(d)) return NULL;
+    }
+    return NULL;
+}
+
 /* ------------------------------- Debugging ---------------------------------*/

 #define DICT_STATS_VECTLEN 50
@@ -1125,7 +1109,7 @@ void dictGetStats(char *buf, size_t bufsize, dict *d) {

 #include "sds.h"

-unsigned int hashCallback(const void *key) {
+uint64_t hashCallback(const void *key) {
    return dictGenHashFunction((unsigned char*)key, sdslen((char*)key));
 }

@@ -56,7 +56,7 @@ typedef struct dictEntry {
 } dictEntry;

 typedef struct dictType {
-    unsigned int (*hashFunction)(const void *key);
+    uint64_t (*hashFunction)(const void *key);
    void *(*keyDup)(void *privdata, const void *key);
    void *(*valDup)(void *privdata, const void *obj);
    int (*keyCompare)(void *privdata, const void *key1, const void *key2);
@@ -95,6 +95,7 @@ typedef struct dictIterator {
 } dictIterator;

 typedef void (dictScanFunction)(void *privdata, const dictEntry *de);
+typedef void (dictScanBucketFunction)(void *privdata, dictEntry **bucketref);

 /* This is the initial size of every hash table */
 #define DICT_HT_INITIAL_SIZE     4
@@ -167,16 +168,18 @@ void dictReleaseIterator(dictIterator *iter);
 dictEntry *dictGetRandomKey(dict *d);
 unsigned int dictGetSomeKeys(dict *d, dictEntry **des, unsigned int count);
 void dictGetStats(char *buf, size_t bufsize, dict *d);
-unsigned int dictGenHashFunction(const void *key, int len);
-unsigned int dictGenCaseHashFunction(const unsigned char *buf, int len);
+uint64_t dictGenHashFunction(const void *key, int len);
+uint64_t dictGenCaseHashFunction(const unsigned char *buf, int len);
 void dictEmpty(dict *d, void(callback)(void*));
 void dictEnableResize(void);
 void dictDisableResize(void);
 int dictRehash(dict *d, int n);
 int dictRehashMilliseconds(dict *d, int ms);
-void dictSetHashFunctionSeed(unsigned int initval);
-unsigned int dictGetHashFunctionSeed(void);
-unsigned long dictScan(dict *d, unsigned long v, dictScanFunction *fn, void *privdata);
+void dictSetHashFunctionSeed(uint8_t *seed);
+uint8_t *dictGetHashFunctionSeed(void);
+unsigned long dictScan(dict *d, unsigned long v, dictScanFunction *fn, dictScanBucketFunction *bucketfn, void *privdata);
+unsigned int dictGetHash(dict *d, const void *key);
+dictEntry **dictFindEntryRefByPtrAndHash(dict *d, const void *oldptr, unsigned int hash);

 /* Hash table types */
 extern dictType dictTypeHeapStringCopyKey;
@@ -336,11 +336,34 @@ unsigned long LFUDecrAndReturn(robj *o) {
 * server when there is data to add in order to make space if needed.
 * --------------------------------------------------------------------------*/

+/* We don't want to count AOF buffers and slaves output buffers as
+ * used memory: the eviction should use mostly data size. This function
+ * returns the sum of AOF and slaves buffer. */
+size_t freeMemoryGetNotCountedMemory(void) {
+    size_t overhead = 0;
+    int slaves = listLength(server.slaves);
+
+    if (slaves) {
+        listIter li;
+        listNode *ln;
+
+        listRewind(server.slaves,&li);
+        while((ln = listNext(&li))) {
+            client *slave = listNodeValue(ln);
+            overhead += getClientOutputBufferMemoryUsage(slave);
+        }
+    }
+    if (server.aof_state != AOF_OFF) {
+        overhead += sdslen(server.aof_buf)+aofRewriteBufferSize();
+    }
+    return overhead;
+}
+
 int freeMemoryIfNeeded(void) {
    size_t mem_reported, mem_used, mem_tofree, mem_freed;
-    int slaves = listLength(server.slaves);
    mstime_t latency, eviction_latency;
    long long delta;
+    int slaves = listLength(server.slaves);

    /* Check if we are over the memory usage limit. If we are not, no need
     * to subtract the slaves output buffers. We can just return ASAP. */
@@ -350,24 +373,8 @@ int freeMemoryIfNeeded(void) {
    /* Remove the size of slaves output buffers and AOF buffer from the
     * count of used memory. */
    mem_used = mem_reported;
-    if (slaves) {
-        listIter li;
-        listNode *ln;
-
-        listRewind(server.slaves,&li);
-        while((ln = listNext(&li))) {
-            client *slave = listNodeValue(ln);
-            unsigned long obuf_bytes = getClientOutputBufferMemoryUsage(slave);
-            if (obuf_bytes > mem_used)
-                mem_used = 0;
-            else
-                mem_used -= obuf_bytes;
-        }
-    }
-    if (server.aof_state != AOF_OFF) {
-        mem_used -= sdslen(server.aof_buf);
-        mem_used -= aofRewriteBufferSize();
-    }
+    size_t overhead = freeMemoryGetNotCountedMemory();
+    mem_used = (mem_used > overhead) ? mem_used-overhead : 0;

    /* Check if we are still over the memory limit. */
    if (mem_used <= server.maxmemory) return C_OK;
@@ -498,6 +505,22 @@ int freeMemoryIfNeeded(void) {
             * deliver data to the slaves fast enough, so we force the
             * transmission here inside the loop. */
            if (slaves) flushSlavesOutputBuffers();
+
+            /* Normally our stop condition is the ability to release
+             * a fixed, pre-computed amount of memory. However when we
+             * are deleting objects in another thread, it's better to
+             * check, from time to time, if we already reached our target
+             * memory, since the "mem_freed" amount is computed only
+             * across the dbAsyncDelete() call, while the thread can
+             * release the memory all the time. */
+            if (server.lazyfree_lazy_eviction && !(keys_freed % 16)) {
+                overhead = freeMemoryGetNotCountedMemory();
+                mem_used = zmalloc_used_memory();
+                mem_used = (mem_used > overhead) ? mem_used-overhead : 0;
+                if (mem_used <= server.maxmemory) {
+                    mem_freed = mem_tofree;
+                }
+            }
        }

        if (!keys_freed) {
@@ -92,7 +92,7 @@ int activeExpireCycleTryExpire(redisDb *db, dictEntry *de, long long now) {
 *
 * If type is ACTIVE_EXPIRE_CYCLE_SLOW, that normal expire cycle is
 * executed, where the time limit is a percentage of the REDIS_HZ period
- * as specified by the REDIS_EXPIRELOOKUPS_TIME_PERC define. */
+ * as specified by the ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC define. */

 void activeExpireCycle(int type) {
    /* This function has some global state in order to continue the work
@@ -216,6 +216,154 @@ void activeExpireCycle(int type) {
    }
 }

+/*-----------------------------------------------------------------------------
+ * Expires of keys created in writable slaves
+ *
+ * Normally slaves do not process expires: they wait the masters to synthesize
+ * DEL operations in order to retain consistency. However writable slaves are
+ * an exception: if a key is created in the slave and an expire is assigned
+ * to it, we need a way to expire such a key, since the master does not know
+ * anything about such a key.
+ *
+ * In order to do so, we track keys created in the slave side with an expire
+ * set, and call the expireSlaveKeys() function from time to time in order to
+ * reclaim the keys if they already expired.
+ *
+ * Note that the use case we are trying to cover here, is a popular one where
+ * slaves are put in writable mode in order to compute slow operations in
+ * the slave side that are mostly useful to actually read data in a more
+ * processed way. Think at sets intersections in a tmp key, with an expire so
+ * that it is also used as a cache to avoid intersecting every time.
+ *
+ * This implementation is currently not perfect but a lot better than leaking
+ * the keys as implemented in 3.2.
+ *----------------------------------------------------------------------------*/
+
+/* The dictionary where we remember key names and database ID of keys we may
+ * want to expire from the slave. Since this function is not often used we
+ * don't even care to initialize the database at startup. We'll do it once
+ * the feature is used the first time, that is, when rememberSlaveKeyWithExpire()
+ * is called.
+ *
+ * The dictionary has an SDS string representing the key as the hash table
+ * key, while the value is a 64 bit unsigned integer with the bits corresponding
+ * to the DB where the keys may exist set to 1. Currently the keys created
+ * with a DB id > 63 are not expired, but a trivial fix is to set the bitmap
+ * to the max 64 bit unsigned value when we know there is a key with a DB
+ * ID greater than 63, and check all the configured DBs in such a case. */
+dict *slaveKeysWithExpire = NULL;
+
+/* Check the set of keys created by the master with an expire set in order to
+ * check if they should be evicted. */
+void expireSlaveKeys(void) {
+    if (slaveKeysWithExpire == NULL ||
+        dictSize(slaveKeysWithExpire) == 0) return;
+
+    int cycles = 0, noexpire = 0;
+    mstime_t start = mstime();
+    while(1) {
+        dictEntry *de = dictGetRandomKey(slaveKeysWithExpire);
+        sds keyname = dictGetKey(de);
+        uint64_t dbids = dictGetUnsignedIntegerVal(de);
+        uint64_t new_dbids = 0;
+
+        /* Check the key against every database corresponding to the
+         * bits set in the value bitmap. */
+        int dbid = 0;
+        while(dbids && dbid < server.dbnum) {
+            if ((dbids & 1) != 0) {
+                redisDb *db = server.db+dbid;
+                dictEntry *expire = dictFind(db->expires,keyname);
+                int expired = 0;
+
+                if (expire &&
+                    activeExpireCycleTryExpire(server.db+dbid,expire,start))
+                {
+                    expired = 1;
+                }
+
+                /* If the key was not expired in this DB, we need to set the
+                 * corresponding bit in the new bitmap we set as value.
+                 * At the end of the loop if the bitmap is zero, it means we
+                 * no longer need to keep track of this key. */
+                if (expire && !expired) {
+                    noexpire++;
+                    new_dbids |= (uint64_t)1 << dbid;
+                }
+            }
+            dbid++;
+            dbids >>= 1;
+        }
+
+        /* Set the new bitmap as value of the key, in the dictionary
+         * of keys with an expire set directly in the writable slave. Otherwise
+         * if the bitmap is zero, we no longer need to keep track of it. */
+        if (new_dbids)
+            dictSetUnsignedIntegerVal(de,new_dbids);
+        else
+            dictDelete(slaveKeysWithExpire,keyname);
+
+        /* Stop conditions: found 3 keys we cna't expire in a row or
+         * time limit was reached. */
+        cycles++;
+        if (noexpire > 3) break;
+        if ((cycles % 64) == 0 && mstime()-start > 1) break;
+        if (dictSize(slaveKeysWithExpire) == 0) break;
+    }
+}
+
+/* Track keys that received an EXPIRE or similar command in the context
+ * of a writable slave. */
+void rememberSlaveKeyWithExpire(redisDb *db, robj *key) {
+    if (slaveKeysWithExpire == NULL) {
+        static dictType dt = {
+            dictSdsHash,                /* hash function */
+            NULL,                       /* key dup */
+            NULL,                       /* val dup */
+            dictSdsKeyCompare,          /* key compare */
+            dictSdsDestructor,          /* key destructor */
+            NULL                        /* val destructor */
+        };
+        slaveKeysWithExpire = dictCreate(&dt,NULL);
+    }
+    if (db->id > 63) return;
+
+    dictEntry *de = dictAddOrFind(slaveKeysWithExpire,key->ptr);
+    /* If the entry was just created, set it to a copy of the SDS string
+     * representing the key: we don't want to need to take those keys
+     * in sync with the main DB. The keys will be removed by expireSlaveKeys()
+     * as it scans to find keys to remove. */
+    if (de->key == key->ptr) {
+        de->key = sdsdup(key->ptr);
+        dictSetUnsignedIntegerVal(de,0);
+    }
+
+    uint64_t dbids = dictGetUnsignedIntegerVal(de);
+    dbids |= (uint64_t)1 << db->id;
+    dictSetUnsignedIntegerVal(de,dbids);
+}
+
+/* Return the number of keys we are tracking. */
+size_t getSlaveKeyWithExpireCount(void) {
+    if (slaveKeysWithExpire == NULL) return 0;
+    return dictSize(slaveKeysWithExpire);
+}
+
+/* Remove the keys in the hash table. We need to do that when data is
+ * flushed from the server. We may receive new keys from the master with
+ * the same name/db and it is no longer a good idea to expire them.
+ *
+ * Note: technically we should handle the case of a single DB being flushed
+ * but it is not worth it since anyway race conditions using the same set
+ * of key names in a wriatable slave and in its master will lead to
+ * inconsistencies. This is just a best-effort thing we do. */
+void flushSlaveKeysWithExpireList(void) {
+    if (slaveKeysWithExpire) {
+        dictRelease(slaveKeysWithExpire);
+        slaveKeysWithExpire = NULL;
+    }
+}
+
 /*-----------------------------------------------------------------------------
 * Expires Commands
 *----------------------------------------------------------------------------*/
@@ -265,7 +413,7 @@ void expireGenericCommand(client *c, long long basetime, int unit) {
        addReply(c, shared.cone);
        return;
    } else {
-        setExpire(c->db,key,when);
+        setExpire(c,c->db,key,when);
        addReply(c,shared.cone);
        signalModifiedKey(c->db,key);
        notifyKeyspaceEvent(NOTIFY_GENERIC,"expire",key,c->db->id);
@@ -161,7 +161,7 @@ double extractDistanceOrReply(client *c, robj **argv,
        addReplyError(c,"radius cannot be negative");
        return -1;
    }
-    
+
    double to_meters = extractUnitOrReply(c,argv[1]);
    if (to_meters < 0) {
        return -1;
@@ -326,6 +326,7 @@ int membersOfGeoHashBox(robj *zobj, GeoHashBits hash, geoArray *ga, double lon,
 int membersOfAllNeighbors(robj *zobj, GeoHashRadius n, double lon, double lat, double radius, geoArray *ga) {
    GeoHashBits neighbors[9];
    unsigned int i, count = 0, last_processed = 0;
+    int debugmsg = 0;

    neighbors[0] = n.hash;
    neighbors[1] = n.neighbors.north;
@@ -340,8 +341,26 @@ int membersOfAllNeighbors(robj *zobj, GeoHashRadius n, double lon, double lat, d
    /* For each neighbor (*and* our own hashbox), get all the matching
     * members and add them to the potential result list. */
    for (i = 0; i < sizeof(neighbors) / sizeof(*neighbors); i++) {
-        if (HASHISZERO(neighbors[i]))
+        if (HASHISZERO(neighbors[i])) {
+            if (debugmsg) D("neighbors[%d] is zero",i);
            continue;
+        }
+
+        /* Debugging info. */
+        if (debugmsg) {
+            GeoHashRange long_range, lat_range;
+            geohashGetCoordRange(&long_range,&lat_range);
+            GeoHashArea myarea = {{0}};
+            geohashDecode(long_range, lat_range, neighbors[i], &myarea);
+
+            /* Dump center square. */
+            D("neighbors[%d]:\n",i);
+            D("area.longitude.min: %f\n", myarea.longitude.min);
+            D("area.longitude.max: %f\n", myarea.longitude.max);
+            D("area.latitude.min: %f\n", myarea.latitude.min);
+            D("area.latitude.max: %f\n", myarea.latitude.max);
+            D("\n");
+        }

        /* When a huge Radius (in the 5000 km range or more) is used,
         * adjacent neighbors can be the same, leading to duplicated
@@ -350,7 +369,11 @@ int membersOfAllNeighbors(robj *zobj, GeoHashRadius n, double lon, double lat, d
        if (last_processed &&
            neighbors[i].bits == neighbors[last_processed].bits &&
            neighbors[i].step == neighbors[last_processed].step)
+        {
+            if (debugmsg)
+                D("Skipping processing of %d, same as previous\n",i);
            continue;
+        }
        count += membersOfGeoHashBox(zobj, neighbors[i], ga, lon, lat, radius);
        last_processed = i;
    }
@@ -661,16 +684,15 @@ void geohashCommand(client *c) {
    int j;

    /* Look up the requested zset */
-    robj *zobj = NULL;
-    if ((zobj = lookupKeyReadOrReply(c, c->argv[1], shared.emptymultibulk))
-        == NULL || checkType(c, zobj, OBJ_ZSET)) return;
+    robj *zobj = lookupKeyRead(c->db, c->argv[1]);
+    if (zobj && checkType(c, zobj, OBJ_ZSET)) return;

    /* Geohash elements one after the other, using a null bulk reply for
     * missing elements. */
    addReplyMultiBulkLen(c,c->argc-2);
    for (j = 2; j < c->argc; j++) {
        double score;
-        if (zsetScore(zobj, c->argv[j]->ptr, &score) == C_ERR) {
+        if (!zobj || zsetScore(zobj, c->argv[j]->ptr, &score) == C_ERR) {
            addReply(c,shared.nullbulk);
        } else {
            /* The internal format we use for geocoding is a bit different
@@ -715,16 +737,15 @@ void geoposCommand(client *c) {
    int j;

    /* Look up the requested zset */
-    robj *zobj = NULL;
-    if ((zobj = lookupKeyReadOrReply(c, c->argv[1], shared.emptymultibulk))
-        == NULL || checkType(c, zobj, OBJ_ZSET)) return;
+    robj *zobj = lookupKeyRead(c->db, c->argv[1]);
+    if (zobj && checkType(c, zobj, OBJ_ZSET)) return;

    /* Report elements one after the other, using a null bulk reply for
     * missing elements. */
    addReplyMultiBulkLen(c,c->argc-2);
    for (j = 2; j < c->argc; j++) {
        double score;
-        if (zsetScore(zobj, c->argv[j]->ptr, &score) == C_ERR) {
+        if (!zobj || zsetScore(zobj, c->argv[j]->ptr, &score) == C_ERR) {
            addReply(c,shared.nullmultibulk);
        } else {
            /* Decode... */
@@ -759,7 +780,7 @@ void geodistCommand(client *c) {

    /* Look up the requested zset */
    robj *zobj = NULL;
-    if ((zobj = lookupKeyReadOrReply(c, c->argv[1], shared.emptybulk))
+    if ((zobj = lookupKeyReadOrReply(c, c->argv[1], shared.nullbulk))
        == NULL || checkType(c, zobj, OBJ_ZSET)) return;

    /* Get the scores. We need both otherwise NULL is returned. */
@@ -82,30 +82,18 @@ uint8_t geohashEstimateStepsByRadius(double range_meters, double lat) {
    return step;
 }

+/* Return the bounding box of the search area centered at latitude,longitude
+ * having a radius of radius_meter. bounds[0] - bounds[2] is the minimum
+ * and maxium longitude, while bounds[1] - bounds[3] is the minimum and
+ * maximum latitude. */
 int geohashBoundingBox(double longitude, double latitude, double radius_meters,
                       double *bounds) {
    if (!bounds) return 0;

-    double lonr, latr;
-    lonr = deg_rad(longitude);
-    latr = deg_rad(latitude);
-
-    if (radius_meters > EARTH_RADIUS_IN_METERS)
-        radius_meters = EARTH_RADIUS_IN_METERS;
-    double distance = radius_meters / EARTH_RADIUS_IN_METERS;
-    double min_latitude = latr - distance;
-    double max_latitude = latr + distance;
-
-    /* Note: we're being lazy and not accounting for coordinates near poles */
-    double min_longitude, max_longitude;
-    double difference_longitude = asin(sin(distance) / cos(latr));
-    min_longitude = lonr - difference_longitude;
-    max_longitude = lonr + difference_longitude;
-
-    bounds[0] = rad_deg(min_longitude);
-    bounds[1] = rad_deg(min_latitude);
-    bounds[2] = rad_deg(max_longitude);
-    bounds[3] = rad_deg(max_latitude);
+    bounds[0] = longitude - rad_deg(radius_meters/EARTH_RADIUS_IN_METERS/cos(deg_rad(latitude)));
+    bounds[2] = longitude + rad_deg(radius_meters/EARTH_RADIUS_IN_METERS/cos(deg_rad(latitude)));
+    bounds[1] = latitude - rad_deg(radius_meters/EARTH_RADIUS_IN_METERS);
+    bounds[3] = latitude + rad_deg(radius_meters/EARTH_RADIUS_IN_METERS);
    return 1;
 }

@@ -158,35 +146,13 @@ GeoHashRadius geohashGetAreasByRadius(double longitude, double latitude, double
            < radius_meters) decrease_step = 1;
    }

-    if (decrease_step) {
+    if (steps > 1 && decrease_step) {
        steps--;
        geohashEncode(&long_range,&lat_range,longitude,latitude,steps,&hash);
        geohashNeighbors(&hash,&neighbors);
        geohashDecode(long_range,lat_range,hash,&area);
    }

-    /* Example debug info. This turns to be very useful every time there is
-     * to investigate radius search potential bugs. So better to leave it
-     * here. */
-    if (0) {
-        GeoHashArea myarea = {{0}};
-        geohashDecode(long_range, lat_range, neighbors.west, &myarea);
-
-        /* Dump West. */
-        D("Neighbors");
-        D("area.longitude.min: %f\n", myarea.longitude.min);
-        D("area.longitude.max: %f\n", myarea.longitude.max);
-        D("area.latitude.min: %f\n", myarea.latitude.min);
-        D("area.latitude.max: %f\n", myarea.latitude.max);
-
-        /* Dump center square. */
-        D("Area");
-        D("area.longitude.min: %f\n", area.longitude.min);
-        D("area.longitude.max: %f\n", area.longitude.max);
-        D("area.latitude.min: %f\n", area.latitude.min);
-        D("area.latitude.max: %f\n", area.latitude.max);
-    }
-
    /* Exclude the search areas that are useless. */
    if (area.latitude.min < min_lat) {
        GZERO(neighbors.south);
@@ -401,7 +401,11 @@ uint64_t MurmurHash64A (const void * key, int len, unsigned int seed) {
        uint64_t k;

 #if (BYTE_ORDER == LITTLE_ENDIAN)
+	#ifdef USE_ALIGNED_ACCESS
+	memcpy(&k,data,sizeof(uint64_t));
+	#else
        k = *((uint64_t*)data);
+	#endif
 #else
        k = (uint64_t) data[0];
        k |= (uint64_t) data[1] << 8;
@@ -994,32 +998,21 @@ uint64_t hllCount(struct hllhdr *hdr, int *invalid) {
        serverPanic("Unknown HyperLogLog encoding in hllCount()");
    }

-    /* Muliply the inverse of E for alpha_m * m^2 to have the raw estimate. */
-    E = (1/E)*alpha*m*m;
+    /* Apply loglog-beta to the raw estimate. See:
+     * "LogLog-Beta and More: A New Algorithm for Cardinality Estimation
+     * Based on LogLog Counting" Jason Qin, Denys Kim, Yumei Tung
+     * arXiv:1612.02284 */
+    double zl = log(ez + 1);
+    double beta = -0.370393911*ez +
+                   0.070471823*zl +
+                   0.17393686*pow(zl,2) +
+                   0.16339839*pow(zl,3) +
+                  -0.09237745*pow(zl,4) +
+                   0.03738027*pow(zl,5) +
+                  -0.005384159*pow(zl,6) +
+                   0.00042419*pow(zl,7);

-    /* Use the LINEARCOUNTING algorithm for small cardinalities.
-     * For larger values but up to 72000 HyperLogLog raw approximation is
-     * used since linear counting error starts to increase. However HyperLogLog
-     * shows a strong bias in the range 2.5*16384 - 72000, so we try to
-     * compensate for it. */
-    if (E < m*2.5 && ez != 0) {
-        E = m*log(m/ez); /* LINEARCOUNTING() */
-    } else if (m == 16384 && E < 72000) {
-        /* We did polynomial regression of the bias for this range, this
-         * way we can compute the bias for a given cardinality and correct
-         * according to it. Only apply the correction for P=14 that's what
-         * we use and the value the correction was verified with. */
-        double bias = 5.9119*1.0e-18*(E*E*E*E)
-                      -1.4253*1.0e-12*(E*E*E)+
-                      1.2940*1.0e-7*(E*E)
-                      -5.2921*1.0e-3*E+
-                      83.3216;
-        E -= E*(bias/100);
-    }
-    /* We don't apply the correction for E > 1/30 of 2^32 since we use
-     * a 64 bit function and 6 bit counters. To apply the correction for
-     * 1/30 of 2^64 is not needed since it would require a huge set
-     * to approach such a value. */
+    E  = llroundl(alpha*m*(m-ez)*(1/(E+beta)));
    return (uint64_t) E;
 }

@@ -261,7 +261,7 @@ int64_t intsetRandom(intset *is) {
    return _intsetGet(is,rand()%intrev32ifbe(is->length));
 }

-/* Sets the value to the value at the given position. When this position is
+/* Get the value at the given position. When this position is
 * out of range the function returns 0, when in range it returns 1. */
 uint8_t intsetGet(intset *is, uint32_t pos, int64_t *value) {
    if (pos < intrev32ifbe(is->length)) {
@@ -41,7 +41,7 @@ int dictStringKeyCompare(void *privdata, const void *key1, const void *key2) {
    return strcmp(key1,key2) == 0;
 }

-unsigned int dictStringHash(const void *key) {
+uint64_t dictStringHash(const void *key) {
    return dictGenHashFunction(key, strlen(key));
 }

@@ -97,11 +97,14 @@ void emptyDbAsync(redisDb *db) {
 /* Empty the slots-keys map of Redis CLuster by creating a new empty one
 * and scheduiling the old for lazy freeing. */
 void slotToKeyFlushAsync(void) {
-    zskiplist *oldsl = server.cluster->slots_to_keys;
-    server.cluster->slots_to_keys = zslCreate();
-    atomicIncr(lazyfree_objects,oldsl->length,
+    rax *old = server.cluster->slots_to_keys;
+
+    server.cluster->slots_to_keys = raxNew();
+    memset(server.cluster->slots_keys_count,0,
+           sizeof(server.cluster->slots_keys_count));
+    atomicIncr(lazyfree_objects,old->numele,
        lazyfree_objects_mutex);
-    bioCreateBackgroundJob(BIO_LAZY_FREE,NULL,NULL,oldsl);
+    bioCreateBackgroundJob(BIO_LAZY_FREE,NULL,NULL,old);
 }

 /* Release objects from the lazyfree thread. It's just decrRefCount()
@@ -125,8 +128,8 @@ void lazyfreeFreeDatabaseFromBioThread(dict *ht1, dict *ht2) {

 /* Release the skiplist mapping Redis Cluster keys to slots in the
 * lazyfree thread. */
-void lazyfreeFreeSlotsMapFromBioThread(zskiplist *sl) {
-    size_t len = sl->length;
-    zslFree(sl);
+void lazyfreeFreeSlotsMapFromBioThread(rax *rt) {
+    size_t len = rt->numele;
+    raxFree(rt);
    atomicDecr(lazyfree_objects,len,lazyfree_objects_mutex);
 }
@@ -1342,7 +1342,7 @@ int RM_SetExpire(RedisModuleKey *key, mstime_t expire) {
        return REDISMODULE_ERR;
    if (expire != REDISMODULE_NO_EXPIRE) {
        expire += mstime();
-        setExpire(key->db,key->key,expire);
+        setExpire(key->ctx->client,key->db,key->key,expire);
    } else {
        removeExpire(key->db,key->key);
    }
@@ -1858,12 +1858,12 @@ int RM_ZsetRangeNext(RedisModuleKey *key) {
        } else {
            /* Are we still within the range? */
            if (key->ztype == REDISMODULE_ZSET_RANGE_SCORE &&
-                !zslValueLteMax(ln->score,&key->zrs))
+                !zslValueLteMax(next->score,&key->zrs))
            {
                key->zer = 1;
                return 0;
            } else if (key->ztype == REDISMODULE_ZSET_RANGE_LEX) {
-                if (!zslLexValueLteMax(ln->ele,&key->zlrs)) {
+                if (!zslLexValueLteMax(next->ele,&key->zlrs)) {
                    key->zer = 1;
                    return 0;
                }
@@ -1921,7 +1921,7 @@ int RM_ZsetRangePrev(RedisModuleKey *key) {
        } else {
            /* Are we still within the range? */
            if (key->ztype == REDISMODULE_ZSET_RANGE_SCORE &&
-                !zslValueGteMin(ln->score,&key->zrs))
+                !zslValueGteMin(prev->score,&key->zrs))
            {
                key->zer = 1;
                return 0;
@@ -2286,8 +2286,10 @@ void RM_FreeCallReply_Rec(RedisModuleCallReply *reply, int freenested){
 * to have the first level function to return on nested replies, but only
 * if called by the module API. */
 void RM_FreeCallReply(RedisModuleCallReply *reply) {
+
+    RedisModuleCtx *ctx = reply->ctx;
    RM_FreeCallReply_Rec(reply,0);
-    autoMemoryFreed(reply->ctx,REDISMODULE_AM_REPLY,reply);
+    autoMemoryFreed(ctx,REDISMODULE_AM_REPLY,reply);
 }

 /* Return the reply type. */
@@ -2741,8 +2743,8 @@ moduleType *RM_CreateDataType(RedisModuleCtx *ctx, const char *name, int encver,
        moduleTypeLoadFunc rdb_load;
        moduleTypeSaveFunc rdb_save;
        moduleTypeRewriteFunc aof_rewrite;
-        moduleTypeDigestFunc digest;
        moduleTypeMemUsageFunc mem_usage;
+        moduleTypeDigestFunc digest;
        moduleTypeFreeFunc free;
    } *tms = (struct typemethods*) typemethods_ptr;

@@ -3106,6 +3108,17 @@ void RM_LogIOError(RedisModuleIO *io, const char *levelstr, const char *fmt, ...
 * Blocking clients from modules
 * -------------------------------------------------------------------------- */

+/* Readable handler for the awake pipe. We do nothing here, the awake bytes
+ * will be actually read in a more appropriate place in the
+ * moduleHandleBlockedClients() function that is where clients are actually
+ * served. */
+void moduleBlockedClientPipeReadable(aeEventLoop *el, int fd, void *privdata, int mask) {
+    UNUSED(el);
+    UNUSED(fd);
+    UNUSED(mask);
+    UNUSED(privdata);
+}
+
 /* This is called from blocked.c in order to unblock a client: may be called
 * for multiple reasons while the client is in the middle of being blocked
 * because the client is terminated, but is also called for cleanup when a
@@ -3169,6 +3182,9 @@ int RM_UnblockClient(RedisModuleBlockedClient *bc, void *privdata) {
    pthread_mutex_lock(&moduleUnblockedClientsMutex);
    bc->privdata = privdata;
    listAddNodeTail(moduleUnblockedClients,bc);
+    if (write(server.module_blocked_pipe[1],"A",1) != 1) {
+        /* Ignore the error, this is best-effort. */
+    }
    pthread_mutex_unlock(&moduleUnblockedClientsMutex);
    return REDISMODULE_OK;
 }
@@ -3193,6 +3209,10 @@ void moduleHandleBlockedClients(void) {
    RedisModuleBlockedClient *bc;

    pthread_mutex_lock(&moduleUnblockedClientsMutex);
+    /* Here we unblock all the pending clients blocked in modules operations
+     * so we can read every pending "awake byte" in the pipe. */
+    char buf[1];
+    while (read(server.module_blocked_pipe[0],buf,1) == 1);
    while (listLength(moduleUnblockedClients)) {
        ln = listFirst(moduleUnblockedClients);
        bc = ln->value;
@@ -3215,8 +3235,11 @@ void moduleHandleBlockedClients(void) {
        }
        if (bc->privdata && bc->free_privdata)
            bc->free_privdata(bc->privdata);
-        zfree(bc);
        if (c != NULL) unblockClient(c);
+        /* Free 'bc' only after unblocking the client, since it is
+         * referenced in the client blocking context, and must be valid
+         * when calling unblockClient(). */
+        zfree(bc);

        /* Lock again before to iterate the loop. */
        pthread_mutex_lock(&moduleUnblockedClientsMutex);
@@ -3262,7 +3285,7 @@ void *RM_GetBlockedClientPrivateData(RedisModuleCtx *ctx) {
 /* server.moduleapi dictionary type. Only uses plain C strings since
 * this gets queries from modules. */

-unsigned int dictCStringKeyHash(const void *key) {
+uint64_t dictCStringKeyHash(const void *key) {
    return dictGenHashFunction((unsigned char*)key, strlen((char*)key));
 }

@@ -3296,6 +3319,16 @@ void moduleInitModulesSystem(void) {
    server.loadmodule_queue = listCreate();
    modules = dictCreate(&modulesDictType,NULL);
    moduleRegisterCoreAPI();
+    if (pipe(server.module_blocked_pipe) == -1) {
+        serverLog(LL_WARNING,
+            "Can't create the pipe for module blocking commands: %s",
+            strerror(errno));
+        exit(1);
+    }
+    /* Make the pipe non blocking. This is just a best effort aware mechanism
+     * and we do not want to block not in the read nor in the write half. */
+    anetNonBlock(NULL,server.module_blocked_pipe[0]);
+    anetNonBlock(NULL,server.module_blocked_pipe[1]);
 }

 /* Load all the modules in the server.loadmodule_queue list, which is
@@ -7,7 +7,16 @@
 Use like malloc(). Memory allocated with this function is reported in
 Redis INFO memory, used for keys eviction according to maxmemory settings
 and in general is taken into account as memory allocated by Redis.
-You should avoid to use malloc().
+You should avoid using malloc().
+
+## `RM_Calloc`
+
+    void *RM_Calloc(size_t nmemb, size_t size);
+
+Use like calloc(). Memory allocated with this function is reported in
+Redis INFO memory, used for keys eviction according to maxmemory settings
+and in general is taken into account as memory allocated by Redis.
+You should avoid using calloc() directly.

 ## `RM_Realloc`

@@ -150,6 +159,12 @@ Called by `RM_Init()` to setup the `ctx->module` structure.
 This is an internal function, Redis modules developers don't need
 to use it.

+## `RM_Milliseconds`
+
+    long long RM_Milliseconds(void);
+
+Return the current UNIX time in milliseconds.
+
 ## `RM_AutoMemory`

    void RM_AutoMemory(RedisModuleCtx *ctx);
@@ -169,6 +184,16 @@ with `RedisModule_FreeString()`, unless automatic memory is enabled.
 The string is created by copying the `len` bytes starting
 at `ptr`. No reference is retained to the passed buffer.

+## `RM_CreateStringPrintf`
+
+    RedisModuleString *RM_CreateStringPrintf(RedisModuleCtx *ctx, const char *fmt, ...);
+
+Create a new module string object from a printf format and arguments.
+The returned string must be freed with `RedisModule_FreeString()`, unless
+automatic memory is enabled.
+
+The string is created using the sds formatter function sdscatvprintf().
+
 ## `RM_CreateStringFromLongLong`

    RedisModuleString *RM_CreateStringFromLongLong(RedisModuleCtx *ctx, long long ll);
@@ -183,7 +208,7 @@ enabling automatic memory management.

    RedisModuleString *RM_CreateStringFromString(RedisModuleCtx *ctx, const RedisModuleString *str);

-Like `RedisModule_CreatString()`, but creates a string starting from an existing
+Like `RedisModule_CreatString()`, but creates a string starting from another
 RedisModuleString.

 The returned string must be released with `RedisModule_FreeString()` or by
@@ -200,9 +225,36 @@ It is possible to call this function even when automatic memory management
 is enabled. In that case the string will be released ASAP and removed
 from the pool of string to release at the end.

+## `RM_RetainString`
+
+    void RM_RetainString(RedisModuleCtx *ctx, RedisModuleString *str);
+
+Every call to this function, will make the string 'str' requiring
+an additional call to `RedisModule_FreeString()` in order to really
+free the string. Note that the automatic freeing of the string obtained
+enabling modules automatic memory management counts for one
+`RedisModule_FreeString()` call (it is just executed automatically).
+
+Normally you want to call this function when, at the same time
+the following conditions are true:
+
+1) You have automatic memory management enabled.
+2) You want to create string objects.
+3) Those string objects you create need to live *after* the callback
+   function(for example a command implementation) creating them returns.
+
+Usually you want this in order to store the created string object
+into your own data structure, for example when implementing a new data
+type.
+
+Note that when memory management is turned off, you don't need
+any call to RetainString() since creating a string will always result
+into a string that lives after the callback function returns, if
+no FreeString() call is performed.
+
 ## `RM_StringPtrLen`

-    const char *RM_StringPtrLen(RedisModuleString *str, size_t *len);
+    const char *RM_StringPtrLen(const RedisModuleString *str, size_t *len);

 Given a string module object, this function returns the string pointer
 and length of the string. The returned pointer and length should only
@@ -210,7 +262,7 @@ be used for read only accesses and never modified.

 ## `RM_StringToLongLong`

-    int RM_StringToLongLong(RedisModuleString *str, long long *ll);
+    int RM_StringToLongLong(const RedisModuleString *str, long long *ll);

 Convert the string into a long long integer, storing it at `*ll`.
 Returns `REDISMODULE_OK` on success. If the string can't be parsed
@@ -219,12 +271,28 @@ is returned.

 ## `RM_StringToDouble`

-    int RM_StringToDouble(RedisModuleString *str, double *d);
+    int RM_StringToDouble(const RedisModuleString *str, double *d);

 Convert the string into a double, storing it at `*d`.
 Returns `REDISMODULE_OK` on success or `REDISMODULE_ERR` if the string is
 not a valid string representation of a double value.

+## `RM_StringCompare`
+
+    int RM_StringCompare(RedisModuleString *a, RedisModuleString *b);
+
+Compare two string objects, returning -1, 0 or 1 respectively if
+a < b, a == b, a > b. Strings are compared byte by byte as two
+binary blobs without any encoding care / collation attempt.
+
+## `RM_StringAppendBuffer`
+
+    int RM_StringAppendBuffer(RedisModuleCtx *ctx, RedisModuleString *str, const char *buf, size_t len);
+
+Append the specified buffere to the string 'str'. The string must be a
+string created by the user that is referenced only a single time, otherwise
+`REDISMODULE_ERR` is returend and the operation is not performed.
+
 ## `RM_WrongArity`

    int RM_WrongArity(RedisModuleCtx *ctx);
@@ -951,11 +1019,11 @@ that returned the reply object.

 ## `RM_CreateDataType`

-    moduleType *RM_CreateDataType(RedisModuleCtx *ctx, const char *name, int encver, moduleTypeLoadFunc rdb_load, moduleTypeSaveFunc rdb_save, moduleTypeRewriteFunc aof_rewrite, moduleTypeDigestFunc digest, moduleTypeFreeFunc free);
+    moduleType *RM_CreateDataType(RedisModuleCtx *ctx, const char *name, int encver, void *typemethods_ptr);

 Register a new data type exported by the module. The parameters are the
 following. Please for in depth documentation check the modules API
-documentation, especially the INTRO.md file.
+documentation, especially the TYPES.md file.

 * **name**: A 9 characters data type name that MUST be unique in the Redis
  Modules ecosystem. Be creative... and there will be no collisions. Use
@@ -974,12 +1042,32 @@ documentation, especially the INTRO.md file.
  still load old data produced by an older version if the rdb_load
  callback is able to check the encver value and act accordingly.
  The encver must be a positive value between 0 and 1023.
+* **typemethods_ptr** is a pointer to a RedisModuleTypeMethods structure
+  that should be populated with the methods callbacks and structure
+  version, like in the following example:
+
+     RedisModuleTypeMethods tm = {
+         .version = `REDISMODULE_TYPE_METHOD_VERSION`,
+         .rdb_load = myType_RDBLoadCallBack,
+         .rdb_save = myType_RDBSaveCallBack,
+         .aof_rewrite = myType_AOFRewriteCallBack,
+         .free = myType_FreeCallBack,
+
+         // Optional fields
+         .digest = myType_DigestCallBack,
+         .mem_usage = myType_MemUsageCallBack,
+     }
+
 * **rdb_load**: A callback function pointer that loads data from RDB files.
 * **rdb_save**: A callback function pointer that saves data to RDB files.
 * **aof_rewrite**: A callback function pointer that rewrites data as commands.
 * **digest**: A callback function pointer that is used for `DEBUG DIGEST`.
+* **mem_usage**: A callback function pointer that is used for `MEMORY`.
 * **free**: A callback function pointer that can free a type value.

+The **digest* and **mem_usage** methods should currently be omitted since
+they are not yet implemented inside the Redis modules core.
+
 Note: the module name "AAAAAAAAA" is reserved and produces an error, it
 happens to be pretty lame as well.

@@ -1115,6 +1203,21 @@ It is possible to load back the value with `RedisModule_LoadDouble()`.
 In the context of the rdb_save method of a module data type, loads back the
 double value saved by `RedisModule_SaveDouble()`.

+## `RM_SaveFloat`
+
+    void RM_SaveFloat(RedisModuleIO *io, float value);
+
+In the context of the rdb_save method of a module data type, saves a float 
+value to the RDB file. The float can be a valid number, a NaN or infinity.
+It is possible to load back the value with `RedisModule_LoadFloat()`.
+
+## `RM_LoadFloat`
+
+    float RM_LoadFloat(RedisModuleIO *io);
+
+In the context of the rdb_save method of a module data type, loads back the
+float value saved by `RedisModule_SaveFloat()`.
+
 ## `RM_EmitAOF`

    void RM_EmitAOF(RedisModuleIO *io, const char *cmdname, const char *fmt, ...);
@@ -1125,10 +1228,20 @@ by a module. The command works exactly like `RedisModule_Call()` in the way
 the parameters are passed, but it does not return anything as the error
 handling is performed by Redis itself.

+## `RM_LogRaw`
+
+    void RM_LogRaw(RedisModule *module, const char *levelstr, const char *fmt, va_list ap);
+
+This is the low level function implementing both:
+
+ `RM_Log()`
+ `RM_LogIOError()`
+
 ## `RM_Log`

    void RM_Log(RedisModuleCtx *ctx, const char *levelstr, const char *fmt, ...);

+/*
 Produces a log message to the standard Redis log, the format accepts
 printf-alike specifiers, while level is a string describing the log
 level to use when emitting the log, and must be one of the following:
@@ -1143,3 +1256,74 @@ There is a fixed limit to the length of the log line this function is able
 to emit, this limti is not specified but is guaranteed to be more than
 a few lines of text.

+## `RM_LogIOError`
+
+    void RM_LogIOError(RedisModuleIO *io, const char *levelstr, const char *fmt, ...);
+
+Log errors from RDB / AOF serialization callbacks.
+
+This function should be used when a callback is returning a critical
+error to the caller since cannot load or save the data for some
+critical reason.
+
+## `RM_BlockClient`
+
+    RedisModuleBlockedClient *RM_BlockClient(RedisModuleCtx *ctx, RedisModuleCmdFunc reply_callback, RedisModuleCmdFunc timeout_callback, void (*free_privdata)(void*), long long timeout_ms);
+
+Block a client in the context of a blocking command, returning an handle
+which will be used, later, in order to block the client with a call to
+`RedisModule_UnblockClient()`. The arguments specify callback functions
+and a timeout after which the client is unblocked.
+
+The callbacks are called in the following contexts:
+
+reply_callback:  called after a successful `RedisModule_UnblockClient()` call
+                 in order to reply to the client and unblock it.
+reply_timeout:   called when the timeout is reached in order to send an
+                 error to the client.
+free_privdata:   called in order to free the privata data that is passed
+                 by `RedisModule_UnblockClient()` call.
+
+## `RM_UnblockClient`
+
+    int RM_UnblockClient(RedisModuleBlockedClient *bc, void *privdata);
+
+Unblock a client blocked by ``RedisModule_BlockedClient``. This will trigger
+the reply callbacks to be called in order to reply to the client.
+The 'privdata' argument will be accessible by the reply callback, so
+the caller of this function can pass any value that is needed in order to
+actually reply to the client.
+
+A common usage for 'privdata' is a thread that computes something that
+needs to be passed to the client, included but not limited some slow
+to compute reply or some reply obtained via networking.
+
+Note: this function can be called from threads spawned by the module.
+
+## `RM_AbortBlock`
+
+    int RM_AbortBlock(RedisModuleBlockedClient *bc);
+
+Abort a blocked client blocking operation: the client will be unblocked
+without firing the reply callback.
+
+## `RM_IsBlockedReplyRequest`
+
+    int RM_IsBlockedReplyRequest(RedisModuleCtx *ctx);
+
+Return non-zero if a module command was called in order to fill the
+reply for a blocked client.
+
+## `RM_IsBlockedTimeoutRequest`
+
+    int RM_IsBlockedTimeoutRequest(RedisModuleCtx *ctx);
+
+Return non-zero if a module command was called in order to fill the
+reply for a blocked client that timed out.
+
+## `RM_GetBlockedClientPrivateData`
+
+    void *RM_GetBlockedClientPrivateData(RedisModuleCtx *ctx);
+
+Get the privata data set by `RedisModule_UnblockClient()`
+
@@ -47,23 +47,25 @@ be stored in the global variable.
    #define MYTYPE_ENCODING_VERSION 0

    int RedisModule_OnLoad(RedisModuleCtx *ctx) {
-        MyType = RedisModule_CreateDataType("MyType-AZ", MYTYPE_ENCODING_VERSION,
-            MyTypeRDBLoad, MyTypeRDBSave, MyTypeAOFRewrite, MyTypeDigest,
-            MyTypeFree);
+	RedisModuleTypeMethods tm = {
+	    .version = REDISMODULE_TYPE_METHOD_VERSION,
+	    .rdb_load = MyTypeRDBLoad,
+	    .rdb_save = MyTypeRDBSave,
+	    .aof_rewrite = MyTypeAOFRewrite,
+	    .free = MyTypeFree
+	};
+
+        MyType = RedisModule_CreateDataType(ctx, "MyType-AZ",
+		MYTYPE_ENCODING_VERSION, &tm);
        if (MyType == NULL) return REDISMODULE_ERR;
    }

 As you can see from the example above, a single API call is needed in order to
 register the new type. However a number of function pointers are passed as
-arguments. The prototype of `RedisModule_CreateDataType` is the following:
-
-    moduleType *RedisModule_CreateDataType(RedisModuleCtx *ctx,
-               const char *name, int encver,
-               moduleTypeLoadFunc rdb_load,
-               moduleTypeSaveFunc rdb_save,
-               moduleTypeRewriteFunc aof_rewrite,
-               moduleTypeDigestFunc digest,
-               moduleTypeFreeFunc free);
+arguments. Certain are optionals while some are mandatory. The above set
+of methods *must* be passed, while `.digest` and `.mem_usage` are optional
+and are currently not actually supported by the modules internals, so for
+now you can just ignore them.

 The `ctx` argument is the context that we receive in the `OnLoad` function.
 The type `name` is a 9 character name in the character set that includes
@@ -74,6 +76,9 @@ ecosystem, so be creative, use both lower-case and upper case if it makes
 sense, and try to use the convention of mixing the type name with the name
 of the author of the module, to create a 9 character unique name.

+**NOTE:** It is very important that the name is exactly 9 chars or the
+registration of the type will fail. Read more to understand why.
+
 For example if I'm building a *b-tree* data structure and my name is *antirez*
 I'll call my type **btree1-az**. The name, converted to a 64 bit integer,
 is stored inside the RDB file when saving the type, and will be used when the
@@ -95,12 +100,14 @@ there is data found for a different encoding version (and the encoding version
 is passed as argument to `rdb_load`), so that the module can still load old
 RDB files.

-The remaining arguments `rdb_load`, `rdb_save`, `aof_rewrite`, `digest` and
-`free` are all callbacks with the following prototypes and uses:
+The last argument is a structure used in order to pass the type methods to the
+registration function: `rdb_load`, `rdb_save`, `aof_rewrite`, `digest` and
+`free` and `mem_usage` are all callbacks with the following prototypes and uses:

    typedef void *(*RedisModuleTypeLoadFunc)(RedisModuleIO *rdb, int encver);
    typedef void (*RedisModuleTypeSaveFunc)(RedisModuleIO *rdb, void *value);
    typedef void (*RedisModuleTypeRewriteFunc)(RedisModuleIO *aof, RedisModuleString *key, void *value);
+    typedef size_t (*RedisModuleTypeMemUsageFunc)(void *value);
    typedef void (*RedisModuleTypeDigestFunc)(RedisModuleDigest *digest, void *value);
    typedef void (*RedisModuleTypeFreeFunc)(void *value);

@@ -108,6 +115,7 @@ The remaining arguments `rdb_load`, `rdb_save`, `aof_rewrite`, `digest` and
 * `rdb_save` is called when saving data to the RDB file.
 * `aof_rewrite` is called when the AOF is being rewritten, and the module needs to tell Redis what is the sequence of commands to recreate the content of a given key.
 * `digest` is called when `DEBUG DIGEST` is executed and a key holding this module type is found. Currently this is not yet implemented so the function ca be left empty.
+* `mem_usage` is called when the `MEMORY` command asks for the total memory consumed by a specific key, and is used in order to get the amount of bytes used by the module value.
 * `free` is called when a key with the module native type is deleted via `DEL` or in any other mean, in order to let the module reclaim the memory associated with such a value.

 Ok, but *why* modules types require a 9 characters name?
@@ -226,10 +226,12 @@ void HelloTypeAofRewrite(RedisModuleIO *aof, RedisModuleString *key, void *value
    }
 }

-void HelloTypeDigest(RedisModuleDigest *digest, void *value) {
-    REDISMODULE_NOT_USED(digest);
-    REDISMODULE_NOT_USED(value);
-    /* TODO: The DIGEST module interface is yet not implemented. */
+/* The goal of this function is to return the amount of memory used by
+ * the HelloType value. */
+size_t HelloTypeMemUsage(const void *value) {
+    const struct HelloTypeObject *hto = value;
+    struct HelloTypeNode *node = hto->head;
+    return sizeof(*hto) + sizeof(*node)*hto->len;
 }

 void HelloTypeFree(void *value) {
@@ -250,6 +252,7 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
        .rdb_load = HelloTypeRdbLoad,
        .rdb_save = HelloTypeRdbSave,
        .aof_rewrite = HelloTypeAofRewrite,
+        .mem_usage = HelloTypeMemUsage,
        .free = HelloTypeFree
    };

@@ -93,6 +93,7 @@ client *createClient(int fd) {
    c->name = NULL;
    c->bufpos = 0;
    c->querybuf = sdsempty();
+    c->pending_querybuf = sdsempty();
    c->querybuf_peak = 0;
    c->reqtype = 0;
    c->argc = 0;
@@ -107,6 +108,7 @@ client *createClient(int fd) {
    c->replstate = REPL_STATE_NONE;
    c->repl_put_online_on_ack = 0;
    c->reploff = 0;
+    c->read_reploff = 0;
    c->repl_ack_off = 0;
    c->repl_ack_time = 0;
    c->slave_listening_port = 0;
@@ -796,6 +798,7 @@ void freeClient(client *c) {

    /* Free the query buffer */
    sdsfree(c->querybuf);
+    sdsfree(c->pending_querybuf);
    c->querybuf = NULL;

    /* Deallocate structures used to block on blocking ops. */
@@ -1027,6 +1030,13 @@ void resetClient(client *c) {
    }
 }

+/* Like processMultibulkBuffer(), but for the inline protocol instead of RESP,
+ * this function consumes the client query buffer and creates a command ready
+ * to be executed inside the client structure. Returns C_OK if the command
+ * is ready to be executed, or C_ERR if there is still protocol to read to
+ * have a well formed command. The function also returns C_ERR when there is
+ * a protocol error: in such a case the client structure is setup to reply
+ * with the error and close the connection. */
 int processInlineBuffer(client *c) {
    char *newline;
    int argc, j;
@@ -1119,6 +1129,17 @@ static void setProtocolError(const char *errstr, client *c, int pos) {
    sdsrange(c->querybuf,pos,-1);
 }

+/* Process the query buffer for client 'c', setting up the client argument
+ * vector for command execution. Returns C_OK if after running the function
+ * the client has a well-formed ready to be processed command, otherwise
+ * C_ERR if there is still to read more buffer to get the full command.
+ * The function also returns C_ERR when there is a protocol error: in such a
+ * case the client structure is setup to reply with the error and close
+ * the connection.
+ *
+ * This function is called if processInputBuffer() detects that the next
+ * command is in RESP format, so the first byte in the command is found
+ * to be '*'. Otherwise for inline commands processInlineBuffer() is called. */
 int processMultibulkBuffer(client *c) {
    char *newline = NULL;
    int pos = 0, ok;
@@ -1253,10 +1274,14 @@ int processMultibulkBuffer(client *c) {
    /* We're done when c->multibulk == 0 */
    if (c->multibulklen == 0) return C_OK;

-    /* Still not read to process the command */
+    /* Still not ready to process the command */
    return C_ERR;
 }

+/* This function is called every time, in the client structure 'c', there is
+ * more query buffer to process, because we read more data from the socket
+ * or because a client was blocked and later reactivated, so there could be
+ * pending query buffer, already representing a full command, to process. */
 void processInputBuffer(client *c) {
    server.current_client = c;
    /* Keep processing while there is something in the input buffer */
@@ -1296,8 +1321,13 @@ void processInputBuffer(client *c) {
            resetClient(c);
        } else {
            /* Only reset the client when the command was executed. */
-            if (processCommand(c) == C_OK)
+            if (processCommand(c) == C_OK) {
+                if (c->flags & CLIENT_MASTER && !(c->flags & CLIENT_MULTI)) {
+                    /* Update the applied replication offset of our master. */
+                    c->reploff = c->read_reploff - sdslen(c->querybuf);
+                }
                resetClient(c);
+            }
            /* freeMemoryIfNeeded may flush slave output buffers. This may result
             * into a slave, that may be the active client, to be freed. */
            if (server.current_client == NULL) break;
@@ -1344,15 +1374,17 @@ void readQueryFromClient(aeEventLoop *el, int fd, void *privdata, int mask) {
        serverLog(LL_VERBOSE, "Client closed connection");
        freeClient(c);
        return;
+    } else if (c->flags & CLIENT_MASTER) {
+        /* Append the query buffer to the pending (not applied) buffer
+         * of the master. We'll use this buffer later in order to have a
+         * copy of the string applied by the last command executed. */
+        c->pending_querybuf = sdscatlen(c->pending_querybuf,
+                                        c->querybuf+qblen,nread);
    }

    sdsIncrLen(c->querybuf,nread);
    c->lastinteraction = server.unixtime;
-    if (c->flags & CLIENT_MASTER) {
-        c->reploff += nread;
-        replicationFeedSlavesFromMasterStream(server.slaves,
-                c->querybuf+qblen,nread);
-    }
+    if (c->flags & CLIENT_MASTER) c->read_reploff += nread;
    server.stat_net_input_bytes += nread;
    if (sdslen(c->querybuf) > server.client_max_querybuf_len) {
        sds ci = catClientInfoString(sdsempty(),c), bytes = sdsempty();
@@ -1364,7 +1396,25 @@ void readQueryFromClient(aeEventLoop *el, int fd, void *privdata, int mask) {
        freeClient(c);
        return;
    }
-    processInputBuffer(c);
+
+    /* Time to process the buffer. If the client is a master we need to
+     * compute the difference between the applied offset before and after
+     * processing the buffer, to understand how much of the replication stream
+     * was actually applied to the master state: this quantity, and its
+     * corresponding part of the replication stream, will be propagated to
+     * the sub-slaves and to the replication backlog. */
+    if (!(c->flags & CLIENT_MASTER)) {
+        processInputBuffer(c);
+    } else {
+        size_t prev_offset = c->reploff;
+        processInputBuffer(c);
+        size_t applied = c->reploff - prev_offset;
+        if (applied) {
+            replicationFeedSlavesFromMasterStream(server.slaves,
+                    c->pending_querybuf, applied);
+            sdsrange(c->pending_querybuf,applied,-1);
+        }
+    }
 }

 void getClientsMaxBuffers(unsigned long *longest_output_list,
@@ -246,11 +246,9 @@ void freeStringObject(robj *o) {
 }

 void freeListObject(robj *o) {
-    switch (o->encoding) {
-    case OBJ_ENCODING_QUICKLIST:
+    if (o->encoding == OBJ_ENCODING_QUICKLIST) {
        quicklistRelease(o->ptr);
-        break;
-    default:
+    } else {
        serverPanic("Unknown list encoding type");
    }
 }
@@ -786,6 +784,14 @@ size_t objectComputeSize(robj *o, size_t sample_size) {
        } else {
            serverPanic("Unknown hash encoding");
        }
+    } else if (o->type == OBJ_MODULE) {
+        moduleValue *mv = o->ptr;
+        moduleType *mt = mv->type;
+        if (mt->mem_usage != NULL) {
+            asize = mt->mem_usage(mv->value);
+        } else {
+            asize = 0;
+        }
    } else {
        serverPanic("Unknown object type");
    }
@@ -945,7 +951,7 @@ sds getMemoryDoctorReport(void) {
        }

        /* Slaves using more than 10 MB each? */
-        if (mh->clients_slaves / numslaves > (1024*1024*10)) {
+        if (numslaves > 0 && mh->clients_slaves / numslaves > (1024*1024*10)) {
            big_slave_buf = 1;
            num_reports++;
        }
@@ -955,14 +961,14 @@ sds getMemoryDoctorReport(void) {
    if (num_reports == 0) {
        s = sdsnew(
        "Hi Sam, I can't find any memory issue in your instance. "
-        "I can only account for what occurs on this base.");
+        "I can only account for what occurs on this base.\n");
    } else if (empty == 1) {
        s = sdsnew(
        "Hi Sam, this instance is empty or is using very little memory, "
        "my issues detector can't be used in these conditions. "
        "Please, leave for your mission on Earth and fill it with some data. "
        "The new Sam and I will be back to our programming as soon as I "
-        "finished rebooting.");
+        "finished rebooting.\n");
    } else {
        s = sdsnew("Sam, I detected a few issues in this Redis instance memory implants:\n\n");
        if (big_peak) {
@@ -0,0 +1,160 @@
+#ifndef RAX_H
+#define RAX_H
+
+#include <stdint.h>
+
+/* Representation of a radix tree as implemented in this file, that contains
+ * the strings "foo", "foobar" and "footer" after the insertion of each
+ * word. When the node represents a key inside the radix tree, we write it
+ * between [], otherwise it is written between ().
+ *
+ * This is the vanilla representation:
+ *
+ *              (f) ""
+ *                \
+ *                (o) "f"
+ *                  \
+ *                  (o) "fo"
+ *                    \
+ *                  [t   b] "foo"
+ *                  /     \
+ *         "foot" (e)     (a) "foob"
+ *                /         \
+ *      "foote" (r)         (r) "fooba"
+ *              /             \
+ *    "footer" []             [] "foobar"
+ *
+ * However, this implementation implements a very common optimization where
+ * successive nodes having a single child are "compressed" into the node
+ * itself as a string of characters, each representing a next-level child,
+ * and only the link to the node representing the last character node is
+ * provided inside the representation. So the above representation is turend
+ * into:
+ *
+ *                  ["foo"] ""
+ *                     |
+ *                  [t   b] "foo"
+ *                  /     \
+ *        "foot" ("er")    ("ar") "foob"
+ *                 /          \
+ *       "footer" []          [] "foobar"
+ *
+ * However this optimization makes the implementation a bit more complex.
+ * For instance if a key "first" is added in the above radix tree, a
+ * "node splitting" operation is needed, since the "foo" prefix is no longer
+ * composed of nodes having a single child one after the other. This is the
+ * above tree and the resulting node splitting after this event happens:
+ *
+ *
+ *                    (f) ""
+ *                    /
+ *                 (i o) "f"
+ *                 /   \
+ *    "firs"  ("rst")  (o) "fo"
+ *              /        \
+ *    "first" []       [t   b] "foo"
+ *                     /     \
+ *           "foot" ("er")    ("ar") "foob"
+ *                    /          \
+ *          "footer" []          [] "foobar"
+ *
+ * Similarly after deletion, if a new chain of nodes having a single child
+ * is created (the chain must also not include nodes that represent keys),
+ * it must be compressed back into a single node.
+ *
+ */
+
+#define RAX_NODE_MAX_SIZE ((1<<29)-1)
+typedef struct raxNode {
+    uint32_t iskey:1;     /* Does this node contain a key? */
+    uint32_t isnull:1;    /* Associated value is NULL (don't store it). */
+    uint32_t iscompr:1;   /* Node is compressed. */
+    uint32_t size:29;     /* Number of children, or compressed string len. */
+    /* Data layout is as follows:
+     *
+     * If node is not compressed we have 'size' bytes, one for each children
+     * character, and 'size' raxNode pointers, point to each child node.
+     * Note how the character is not stored in the children but in the
+     * edge of the parents:
+     *
+     * [header strlen=0][abc][a-ptr][b-ptr][c-ptr](value-ptr?)
+     *
+     * if node is compressed (strlen != 0) the node has 1 children.
+     * In that case the 'size' bytes of the string stored immediately at
+     * the start of the data section, represent a sequence of successive
+     * nodes linked one after the other, for which only the last one in
+     * the sequence is actually represented as a node, and pointed to by
+     * the current compressed node.
+     *
+     * [header strlen=3][xyz][z-ptr](value-ptr?)
+     *
+     * Both compressed and not compressed nodes can represent a key
+     * with associated data in the radix tree at any level (not just terminal
+     * nodes).
+     *
+     * If the node has an associated key (iskey=1) and is not NULL
+     * (isnull=0), then after the raxNode pointers poiting to the
+     * childen, an additional value pointer is present (as you can see
+     * in the representation above as "value-ptr" field).
+     */
+    unsigned char data[];
+} raxNode;
+
+typedef struct rax {
+    raxNode *head;
+    uint64_t numele;
+    uint64_t numnodes;
+} rax;
+
+/* Stack data structure used by raxLowWalk() in order to, optionally, return
+ * a list of parent nodes to the caller. The nodes do not have a "parent"
+ * field for space concerns, so we use the auxiliary stack when needed. */
+#define RAX_STACK_STATIC_ITEMS 32
+typedef struct raxStack {
+    void **stack; /* Points to static_items or an heap allocated array. */
+    size_t items, maxitems; /* Number of items contained and total space. */
+    /* Up to RAXSTACK_STACK_ITEMS items we avoid to allocate on the heap
+     * and use this static array of pointers instead. */
+    void *static_items[RAX_STACK_STATIC_ITEMS];
+    int oom; /* True if pushing into this stack failed for OOM at some point. */
+} raxStack;
+
+/* Radix tree iterator state is encapsulated into this data structure. */
+#define RAX_ITER_STATIC_LEN 128
+#define RAX_ITER_JUST_SEEKED (1<<0) /* Iterator was just seeked. Return current
+                                       element for the first iteration and
+                                       clear the flag. */
+#define RAX_ITER_EOF (1<<1)    /* End of iteration reached. */
+#define RAX_ITER_SAFE (1<<2)   /* Safe iterator, allows operations while
+                                  iterating. But it is slower. */
+typedef struct raxIterator {
+    int flags;
+    rax *rt;                /* Radix tree we are iterating. */
+    unsigned char *key;     /* The current string. */
+    void *data;             /* Data associated to this key. */
+    size_t key_len;         /* Current key length. */
+    size_t key_max;         /* Max key len the current key buffer can hold. */
+    unsigned char key_static_string[RAX_ITER_STATIC_LEN];
+    raxNode *node;          /* Current node. Only for unsafe iteration. */
+    raxStack stack;         /* Stack used for unsafe iteration. */
+} raxIterator;
+
+/* A special pointer returned for not found items. */
+extern void *raxNotFound;
+
+/* Exported API. */
+rax *raxNew(void);
+int raxInsert(rax *rax, unsigned char *s, size_t len, void *data, void **old);
+int raxRemove(rax *rax, unsigned char *s, size_t len, void **old);
+void *raxFind(rax *rax, unsigned char *s, size_t len);
+void raxFree(rax *rax);
+void raxStart(raxIterator *it, rax *rt);
+int raxSeek(raxIterator *it, const char *op, unsigned char *ele, size_t len);
+int raxNext(raxIterator *it);
+int raxPrev(raxIterator *it);
+int raxRandomWalk(raxIterator *it, size_t steps);
+int raxCompare(raxIterator *iter, const char *op, unsigned char *key, size_t key_len);
+void raxStop(raxIterator *it);
+void raxShow(rax *rax);
+
+#endif
@@ -0,0 +1,44 @@
+/* Rax -- A radix tree implementation.
+ *
+ * Copyright (c) 2017, Salvatore Sanfilippo <antirez at gmail dot com>
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ *   * Redistributions of source code must retain the above copyright notice,
+ *     this list of conditions and the following disclaimer.
+ *   * Redistributions in binary form must reproduce the above copyright
+ *     notice, this list of conditions and the following disclaimer in the
+ *     documentation and/or other materials provided with the distribution.
+ *   * Neither the name of Redis nor the names of its contributors may be used
+ *     to endorse or promote products derived from this software without
+ *     specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/* Allocator selection.
+ *
+ * This file is used in order to change the Rax allocator at compile time.
+ * Just define the following defines to what you want to use. Also add
+ * the include of your alternate allocator if needed (not needed in order
+ * to use the default libc allocator). */
+
+#ifndef RAX_ALLOC_H
+#define RAX_ALLOC_H
+#include "zmalloc.h"
+#define rax_malloc zmalloc
+#define rax_realloc zrealloc
+#define rax_free zfree
+#endif
@@ -704,23 +704,30 @@ ssize_t rdbSaveObject(rio *rdb, robj *o) {
            nwritten += n;
        } else if (o->encoding == OBJ_ENCODING_SKIPLIST) {
            zset *zs = o->ptr;
-            dictIterator *di = dictGetIterator(zs->dict);
-            dictEntry *de;
+            zskiplist *zsl = zs->zsl;

-            if ((n = rdbSaveLen(rdb,dictSize(zs->dict))) == -1) return -1;
+            if ((n = rdbSaveLen(rdb,zsl->length)) == -1) return -1;
            nwritten += n;

-            while((de = dictNext(di)) != NULL) {
-                sds ele = dictGetKey(de);
-                double *score = dictGetVal(de);
-
-                if ((n = rdbSaveRawString(rdb,(unsigned char*)ele,sdslen(ele)))
-                    == -1) return -1;
+            /* We save the skiplist elements from the greatest to the smallest
+             * (that's trivial since the elements are already ordered in the
+             * skiplist): this improves the load process, since the next loaded
+             * element will always be the smaller, so adding to the skiplist
+             * will always immediately stop at the head, making the insertion
+             * O(1) instead of O(log(N)). */
+            zskiplistNode *zn = zsl->tail;
+            while (zn != NULL) {
+                if ((n = rdbSaveRawString(rdb,
+                    (unsigned char*)zn->ele,sdslen(zn->ele))) == -1)
+                {
+                    return -1;
+                }
                nwritten += n;
-                if ((n = rdbSaveBinaryDoubleValue(rdb,*score)) == -1) return -1;
+                if ((n = rdbSaveBinaryDoubleValue(rdb,zn->score)) == -1)
+                    return -1;
                nwritten += n;
+                zn = zn->backward;
            }
-            dictReleaseIterator(di);
        } else {
            serverPanic("Unknown sorted set encoding");
        }
@@ -1552,7 +1559,7 @@ int rdbLoadRio(rio *rdb, rdbSaveInfo *rsi) {
        dbAdd(db,key,val);

        /* Set the expire time if needed */
-        if (expiretime != -1) setExpire(db,key,expiretime);
+        if (expiretime != -1) setExpire(NULL,db,key,expiretime);

        decrRefCount(key);
    }
@@ -1939,7 +1946,7 @@ void bgsaveCommand(client *c) {
        } else {
            addReplyError(c,
                "An AOF log rewriting in progress: can't BGSAVE right now. "
-                "Use BGSAVE SCHEDULE in order to schedule a BGSAVE whenver "
+                "Use BGSAVE SCHEDULE in order to schedule a BGSAVE whenever "
                "possible.");
        }
    } else if (rdbSaveBackground(server.rdb_filename,NULL) == C_OK) {
@@ -565,7 +565,7 @@ invalid:

 usage:
    printf(
-"Usage: redis-benchmark [-h <host>] [-p <port>] [-c <clients>] [-n <requests]> [-k <boolean>]\n\n"
+"Usage: redis-benchmark [-h <host>] [-p <port>] [-c <clients>] [-n <requests>] [-k <boolean>]\n\n"
 " -h <hostname>      Server hostname (default 127.0.0.1)\n"
 " -p <port>          Server port (default 6379)\n"
 " -s <socket>        Server socket (overrides host and port)\n"
@@ -275,6 +275,10 @@ static void cliIntegrateHelp(void) {
     * don't already match what we have. */
    for (size_t j = 0; j < reply->elements; j++) {
        redisReply *entry = reply->element[j];
+        if (entry->type != REDIS_REPLY_ARRAY || entry->elements < 4 ||
+            entry->element[0]->type != REDIS_REPLY_STRING ||
+            entry->element[1]->type != REDIS_REPLY_INTEGER ||
+            entry->element[3]->type != REDIS_REPLY_INTEGER) return;
        char *cmdname = entry->element[0]->str;
        int i;

@@ -1269,6 +1273,11 @@ static void repl(void) {
    int argc;
    sds *argv;

+    /* Initialize the help and, if possible, use the COMMAND command in order
+     * to retrieve missing entries. */
+    cliInitHelp();
+    cliIntegrateHelp();
+
    config.interactive = 1;
    linenoiseSetMultiLine(1);
    linenoiseSetCompletionCallback(completionCallback);
@@ -2602,11 +2611,6 @@ int main(int argc, char **argv) {
    argc -= firstarg;
    argv += firstarg;

-    /* Initialize the help and, if possible, use the COMMAND command in order
-     * to retrieve missing entries. */
-    cliInitHelp();
-    cliIntegrateHelp();
-
    /* Latency mode */
    if (config.latency_mode) {
        if (cliConnect(0) == REDIS_ERR) exit(1);
@@ -41,7 +41,9 @@
 #include <unistd.h> /* for _exit() */

 #define assert(_e) ((_e)?(void)0 : (_serverAssert(#_e,__FILE__,__LINE__),_exit(1)))
+#define panic(...) _serverPanic(__FILE__,__LINE__,__VA_ARGS__),_exit(1)

 void _serverAssert(char *estr, char *file, int line);
+void _serverPanic(const char *file, int line, const char *msg, ...);

 #endif
@@ -91,7 +91,7 @@ typedef int (*RedisModuleCmdFunc) (RedisModuleCtx *ctx, RedisModuleString **argv
 typedef void *(*RedisModuleTypeLoadFunc)(RedisModuleIO *rdb, int encver);
 typedef void (*RedisModuleTypeSaveFunc)(RedisModuleIO *rdb, void *value);
 typedef void (*RedisModuleTypeRewriteFunc)(RedisModuleIO *aof, RedisModuleString *key, void *value);
-typedef size_t (*RedisModuleTypeMemUsageFunc)(void *value);
+typedef size_t (*RedisModuleTypeMemUsageFunc)(const void *value);
 typedef void (*RedisModuleTypeDigestFunc)(RedisModuleDigest *digest, void *value);
 typedef void (*RedisModuleTypeFreeFunc)(void *value);

@@ -102,7 +102,7 @@ typedef struct RedisModuleTypeMethods {
    RedisModuleTypeSaveFunc rdb_save;
    RedisModuleTypeRewriteFunc aof_rewrite;
    RedisModuleTypeMemUsageFunc mem_usage;
-    RedisModuleTypeRewriteFunc digest;
+    RedisModuleTypeDigestFunc digest;
    RedisModuleTypeFreeFunc free;
 } RedisModuleTypeMethods;

@@ -122,7 +122,7 @@ void freeReplicationBacklog(void) {
 /* Add data to the replication backlog.
 * This function also increments the global replication offset stored at
 * server.master_repl_offset, because there is no case where we want to feed
- * the backlog without incrementing the buffer. */
+ * the backlog without incrementing the offset. */
 void feedReplicationBacklog(void *ptr, size_t len) {
    unsigned char *p = ptr;

@@ -1078,6 +1078,7 @@ void replicationCreateMasterClient(int fd, int dbid) {
    server.master->flags |= CLIENT_MASTER;
    server.master->authenticated = 1;
    server.master->reploff = server.master_initial_offset;
+    server.master->read_reploff = server.master->reploff;
    memcpy(server.master->replid, server.master_replid,
        sizeof(server.master_replid));
    /* If master offset is set to -1, this master is old and is not
@@ -1087,6 +1088,18 @@ void replicationCreateMasterClient(int fd, int dbid) {
    if (dbid != -1) selectDb(server.master,dbid);
 }

+void restartAOF() {
+    int retry = 10;
+    while (retry-- && startAppendOnly() == C_ERR) {
+        serverLog(LL_WARNING,"Failed enabling the AOF after successful master synchronization! Trying it again in one second.");
+        sleep(1);
+    }
+    if (!retry) {
+        serverLog(LL_WARNING,"FATAL: this slave instance finished the synchronization with its master, but the AOF can't be turned on. Exiting now.");
+        exit(1);
+    }
+}
+
 /* Asynchronously read the SYNC payload we receive from a master */
 #define REPL_MAX_WRITTEN_BEFORE_FSYNC (1024*1024*8) /* 8 MB */
 void readSyncBulkPayload(aeEventLoop *el, int fd, void *privdata, int mask) {
@@ -1228,12 +1241,17 @@ void readSyncBulkPayload(aeEventLoop *el, int fd, void *privdata, int mask) {
    }

    if (eof_reached) {
+        int aof_is_enabled = server.aof_state != AOF_OFF;
+
        if (rename(server.repl_transfer_tmpfile,server.rdb_filename) == -1) {
            serverLog(LL_WARNING,"Failed trying to rename the temp DB into dump.rdb in MASTER <-> SLAVE synchronization: %s", strerror(errno));
            cancelReplicationHandshake();
            return;
        }
        serverLog(LL_NOTICE, "MASTER <-> SLAVE sync: Flushing old data");
+        /* We need to stop any AOFRW fork before flusing and parsing
+         * RDB, otherwise we'll create a copy-on-write disaster. */
+        if(aof_is_enabled) stopAppendOnly();
        signalFlushedDb(-1);
        emptyDb(
            -1,
@@ -1249,6 +1267,9 @@ void readSyncBulkPayload(aeEventLoop *el, int fd, void *privdata, int mask) {
        if (rdbLoad(server.rdb_filename,&rsi) != C_OK) {
            serverLog(LL_WARNING,"Failed trying to load the MASTER synchronization DB from disk");
            cancelReplicationHandshake();
+            /* Re-enable the AOF if we disabled it earlier, in order to restore
+             * the original configuration. */
+            if (aof_is_enabled) restartAOF();
            return;
        }
        /* Final setup of the connected slave <- master link */
@@ -1272,21 +1293,8 @@ void readSyncBulkPayload(aeEventLoop *el, int fd, void *privdata, int mask) {
        /* Restart the AOF subsystem now that we finished the sync. This
         * will trigger an AOF rewrite, and when done will start appending
         * to the new file. */
-        if (server.aof_state != AOF_OFF) {
-            int retry = 10;
-
-            stopAppendOnly();
-            while (retry-- && startAppendOnly() == C_ERR) {
-                serverLog(LL_WARNING,"Failed enabling the AOF after successful master synchronization! Trying it again in one second.");
-                sleep(1);
-            }
-            if (!retry) {
-                serverLog(LL_WARNING,"FATAL: this slave instance finished the synchronization with its master, but the AOF can't be turned on. Exiting now.");
-                exit(1);
-            }
-        }
+        if (aof_is_enabled) restartAOF();
    }
-
    return;

 error:
@@ -1561,7 +1569,7 @@ int slaveTryPartialResynchronization(int fd, int read_reply) {
 * establish a connection with the master. */
 void syncWithMaster(aeEventLoop *el, int fd, void *privdata, int mask) {
    char tmpfile[256], *err = NULL;
-    int dfd, maxtries = 5;
+    int dfd = -1, maxtries = 5;
    int sockerr = 0, psync_result;
    socklen_t errlen = sizeof(sockerr);
    UNUSED(el);
@@ -1825,6 +1833,7 @@ void syncWithMaster(aeEventLoop *el, int fd, void *privdata, int mask) {

 error:
    aeDeleteFileEvent(server.el,fd,AE_READABLE|AE_WRITABLE);
+    if (dfd != -1) close(dfd);
    close(fd);
    server.repl_transfer_s = -1;
    server.repl_state = REPL_STATE_CONNECT;
@@ -2110,6 +2119,14 @@ void replicationCacheMaster(client *c) {
    /* Unlink the client from the server structures. */
    unlinkClient(c);

+    /* Fix the master specific fields: we want to discard to non processed
+     * query buffers and non processed offsets, including pending
+     * transactions. */
+    sdsclear(server.master->querybuf);
+    sdsclear(server.master->pending_querybuf);
+    server.master->read_reploff = server.master->reploff;
+    if (c->flags & CLIENT_MULTI) discardTransaction(c);
+
    /* Save the master. Server.master will be set to null later by
     * replicationHandleMasterDisconnection(). */
    server.cached_master = server.master;
@@ -900,7 +900,6 @@ void scriptingInit(int setup) {
        server.lua_caller = NULL;
        server.lua_timedout = 0;
        server.lua_always_replicate_commands = 0; /* Only DEBUG can change it.*/
-        server.lua_time_limit = LUA_SCRIPT_TIME_LIMIT;
        ldbInit();
    }

@@ -379,7 +379,7 @@ void sentinelSimFailureCrash(void);

 /* ========================= Dictionary types =============================== */

-unsigned int dictSdsHash(const void *key);
+uint64_t dictSdsHash(const void *key);
 int dictSdsKeyCompare(void *privdata, const void *key1, const void *key2);
 void releaseSentinelRedisInstance(sentinelRedisInstance *ri);

@@ -482,16 +482,16 @@ int dictObjKeyCompare(void *privdata, const void *key1,
    return dictSdsKeyCompare(privdata,o1->ptr,o2->ptr);
 }

-unsigned int dictObjHash(const void *key) {
+uint64_t dictObjHash(const void *key) {
    const robj *o = key;
    return dictGenHashFunction(o->ptr, sdslen((sds)o->ptr));
 }

-unsigned int dictSdsHash(const void *key) {
+uint64_t dictSdsHash(const void *key) {
    return dictGenHashFunction((unsigned char*)key, sdslen((char*)key));
 }

-unsigned int dictSdsCaseHash(const void *key) {
+uint64_t dictSdsCaseHash(const void *key) {
    return dictGenCaseHashFunction((unsigned char*)key, sdslen((char*)key));
 }

@@ -513,7 +513,7 @@ int dictEncObjKeyCompare(void *privdata, const void *key1,
    return cmp;
 }

-unsigned int dictEncObjHash(const void *key) {
+uint64_t dictEncObjHash(const void *key) {
    robj *o = (robj*) key;

    if (sdsEncodedObject(o)) {
@@ -526,7 +526,7 @@ unsigned int dictEncObjHash(const void *key) {
            len = ll2string(buf,32,(long)o->ptr);
            return dictGenHashFunction((unsigned char*)buf, len);
        } else {
-            unsigned int hash;
+            uint64_t hash;

            o = getDecodedObject(o);
            hash = dictGenHashFunction(o->ptr, sdslen((sds)o->ptr));
@@ -870,8 +870,15 @@ void clientsCron(void) {
 void databasesCron(void) {
    /* Expire keys by random sampling. Not required for slaves
     * as master will synthesize DELs for us. */
-    if (server.active_expire_enabled && server.masterhost == NULL)
+    if (server.active_expire_enabled && server.masterhost == NULL) {
        activeExpireCycle(ACTIVE_EXPIRE_CYCLE_SLOW);
+    } else if (server.masterhost != NULL) {
+        expireSlaveKeys();
+    }
+
+    /* Defrag keys gradually. */
+    if (server.active_defrag_enabled)
+        activeDefragCycle();

    /* Perform hash tables rehashing if needed, but only if there are no
     * other processes saving the DB on disk. Otherwise rehashing is bad
@@ -1329,6 +1336,12 @@ void initServerConfig(void) {
    server.maxidletime = CONFIG_DEFAULT_CLIENT_TIMEOUT;
    server.tcpkeepalive = CONFIG_DEFAULT_TCP_KEEPALIVE;
    server.active_expire_enabled = 1;
+    server.active_defrag_enabled = CONFIG_DEFAULT_ACTIVE_DEFRAG;
+    server.active_defrag_ignore_bytes = CONFIG_DEFAULT_DEFRAG_IGNORE_BYTES;
+    server.active_defrag_threshold_lower = CONFIG_DEFAULT_DEFRAG_THRESHOLD_LOWER;
+    server.active_defrag_threshold_upper = CONFIG_DEFAULT_DEFRAG_THRESHOLD_UPPER;
+    server.active_defrag_cycle_min = CONFIG_DEFAULT_DEFRAG_CYCLE_MIN;
+    server.active_defrag_cycle_max = CONFIG_DEFAULT_DEFRAG_CYCLE_MAX;
    server.client_max_querybuf_len = PROTO_MAX_QUERYBUF_LEN;
    server.saveparams = NULL;
    server.loading = 0;
@@ -1365,6 +1378,7 @@ void initServerConfig(void) {
    server.rdb_checksum = CONFIG_DEFAULT_RDB_CHECKSUM;
    server.stop_writes_on_bgsave_err = CONFIG_DEFAULT_STOP_WRITES_ON_BGSAVE_ERROR;
    server.activerehashing = CONFIG_DEFAULT_ACTIVE_REHASHING;
+    server.active_defrag_running = 0;
    server.notify_keyspace_events = 0;
    server.maxclients = CONFIG_DEFAULT_MAX_CLIENTS;
    server.bpop_blocked_clients = 0;
@@ -1397,6 +1411,8 @@ void initServerConfig(void) {
    server.lazyfree_lazy_eviction = CONFIG_DEFAULT_LAZYFREE_LAZY_EVICTION;
    server.lazyfree_lazy_expire = CONFIG_DEFAULT_LAZYFREE_LAZY_EXPIRE;
    server.lazyfree_lazy_server_del = CONFIG_DEFAULT_LAZYFREE_LAZY_SERVER_DEL;
+    server.always_show_logo = CONFIG_DEFAULT_ALWAYS_SHOW_LOGO;
+    server.lua_time_limit = LUA_SCRIPT_TIME_LIMIT;

    server.lruclock = getLRUClock();
    resetServerSaveParams();
@@ -1572,9 +1588,12 @@ void adjustOpenFilesLimit(void) {
            if (bestlimit < oldlimit) bestlimit = oldlimit;

            if (bestlimit < maxfiles) {
-                int old_maxclients = server.maxclients;
+                unsigned int old_maxclients = server.maxclients;
                server.maxclients = bestlimit-CONFIG_MIN_RESERVED_FDS;
-                if (server.maxclients < 1) {
+                /* maxclients is unsigned so may overflow: in order
+                 * to check if maxclients is now logically less than 1
+                 * we test indirectly via bestlimit. */
+                if (bestlimit <= CONFIG_MIN_RESERVED_FDS) {
                    serverLog(LL_WARNING,"Your current 'ulimit -n' "
                        "of %llu is not enough for the server to start. "
                        "Please increase your open file limit to at least "
@@ -1711,6 +1730,10 @@ void resetServerStats(void) {
    server.stat_evictedkeys = 0;
    server.stat_keyspace_misses = 0;
    server.stat_keyspace_hits = 0;
+    server.stat_active_defrag_hits = 0;
+    server.stat_active_defrag_misses = 0;
+    server.stat_active_defrag_key_hits = 0;
+    server.stat_active_defrag_key_misses = 0;
    server.stat_fork_time = 0;
    server.stat_fork_rate = 0;
    server.stat_rejected_conn = 0;
@@ -1848,6 +1871,16 @@ void initServer(void) {
    if (server.sofd > 0 && aeCreateFileEvent(server.el,server.sofd,AE_READABLE,
        acceptUnixHandler,NULL) == AE_ERR) serverPanic("Unrecoverable error creating server.sofd file event.");

+
+    /* Register a readable event for the pipe used to awake the event loop
+     * when a blocked client in a module needs attention. */
+    if (aeCreateFileEvent(server.el, server.module_blocked_pipe[0], AE_READABLE,
+        moduleBlockedClientPipeReadable,NULL) == AE_ERR) {
+            serverPanic(
+                "Error registering the readable event for the module "
+                "blocked clients subsystem.");
+    }
+
    /* Open the AOF file if needed. */
    if (server.aof_state == AOF_ON) {
        server.aof_fd = open(server.aof_filename,
@@ -2866,6 +2899,7 @@ sds genRedisInfoString(char *section) {
            "maxmemory_policy:%s\r\n"
            "mem_fragmentation_ratio:%.2f\r\n"
            "mem_allocator:%s\r\n"
+            "active_defrag_running:%d\r\n"
            "lazyfree_pending_objects:%zu\r\n",
            zmalloc_used,
            hmem,
@@ -2887,6 +2921,7 @@ sds genRedisInfoString(char *section) {
            evict_policy,
            mh->fragmentation,
            ZMALLOC_LIB,
+            server.active_defrag_running,
            lazyfreeGetPendingObjectsCount()
        );
        freeMemoryOverheadData(mh);
@@ -3005,7 +3040,12 @@ sds genRedisInfoString(char *section) {
            "pubsub_channels:%ld\r\n"
            "pubsub_patterns:%lu\r\n"
            "latest_fork_usec:%lld\r\n"
-            "migrate_cached_sockets:%ld\r\n",
+            "migrate_cached_sockets:%ld\r\n"
+            "slave_expires_tracked_keys:%zu\r\n"
+            "active_defrag_hits:%lld\r\n"
+            "active_defrag_misses:%lld\r\n"
+            "active_defrag_key_hits:%lld\r\n"
+            "active_defrag_key_misses:%lld\r\n",
            server.stat_numconnections,
            server.stat_numcommands,
            getInstantaneousMetric(STATS_METRIC_COMMAND),
@@ -3024,7 +3064,12 @@ sds genRedisInfoString(char *section) {
            dictSize(server.pubsub_channels),
            listLength(server.pubsub_patterns),
            server.stat_fork_time,
-            dictSize(server.migrate_cached_sockets));
+            dictSize(server.migrate_cached_sockets),
+            getSlaveKeyWithExpireCount(),
+            server.stat_active_defrag_hits,
+            server.stat_active_defrag_misses,
+            server.stat_active_defrag_key_hits,
+            server.stat_active_defrag_key_misses);
    }

    /* Replication */
@@ -3327,15 +3372,18 @@ void redisAsciiArt(void) {
    else if (server.sentinel_mode) mode = "sentinel";
    else mode = "standalone";

-    if (server.syslog_enabled) {
+    /* Show the ASCII logo if: log file is stdout AND stdout is a
+     * tty AND syslog logging is disabled. Also show logo if the user
+     * forced us to do so via redis.conf. */
+    int show_logo = ((!server.syslog_enabled &&
+                      server.logfile[0] == '\0' &&
+                      isatty(fileno(stdout))) ||
+                     server.always_show_logo);
+
+    if (!show_logo) {
        serverLog(LL_NOTICE,
-            "Redis %s (%s/%d) %s bit, %s mode, port %d, pid %ld ready to start.",
-            REDIS_VERSION,
-            redisGitSHA1(),
-            strtol(redisGitDirty(),NULL,10) > 0,
-            (sizeof(long) == 8) ? "64" : "32",
-            mode, server.port,
-            (long) getpid()
+            "Running mode=%s, port=%d.",
+            mode, server.port
        );
    } else {
        snprintf(buf,1024*16,ascii_logo,
@@ -3602,7 +3650,9 @@ int main(int argc, char **argv) {
    zmalloc_set_oom_handler(redisOutOfMemoryHandler);
    srand(time(NULL)^getpid());
    gettimeofday(&tv,NULL);
-    dictSetHashFunctionSeed(tv.tv_sec^tv.tv_usec^getpid());
+    char hashseed[16];
+    getRandomHexChars(hashseed,sizeof(hashseed));
+    dictSetHashFunctionSeed((uint8_t*)hashseed);
    server.sentinel_mode = checkForSentinelMode(argc,argv);
    initServerConfig();
    moduleInitModulesSystem();
@@ -3692,8 +3742,21 @@ int main(int argc, char **argv) {
        resetServerSaveParams();
        loadServerConfig(configfile,options);
        sdsfree(options);
-    } else {
+    }
+
+    serverLog(LL_WARNING, "oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo");
+    serverLog(LL_WARNING,
+        "Redis version=%s, bits=%d, commit=%s, modified=%d, pid=%d, just started",
+            REDIS_VERSION,
+            (sizeof(long) == 8) ? 64 : 32,
+            redisGitSHA1(),
+            strtol(redisGitDirty(),NULL,10) > 0,
+            (int)getpid());
+
+    if (argc == 1) {
        serverLog(LL_WARNING, "Warning: no config file specified, using the default config. In order to specify a config file use %s /path/to/%s.conf", argv[0], server.sentinel_mode ? "sentinel" : "redis");
+    } else {
+        serverLog(LL_WARNING, "Configuration loaded");
    }

    server.supervised = redisIsSupervised(server.supervised_mode);
@@ -3708,7 +3771,7 @@ int main(int argc, char **argv) {

    if (!server.sentinel_mode) {
        /* Things not needed when running in Sentinel mode. */
-        serverLog(LL_WARNING,"Server started, Redis version " REDIS_VERSION);
+        serverLog(LL_WARNING,"Server initialized");
    #ifdef __linux__
        linuxMemoryWarnings();
    #endif
@@ -3723,7 +3786,7 @@ int main(int argc, char **argv) {
            }
        }
        if (server.ipfd_count > 0)
-            serverLog(LL_NOTICE,"The server is now ready to accept connections on port %d", server.port);
+            serverLog(LL_NOTICE,"Ready to accept connections");
        if (server.sofd > 0)
            serverLog(LL_NOTICE,"The server is now ready to accept connections at %s", server.unixsocket);
    } else {
@@ -63,7 +63,9 @@ typedef long long mstime_t; /* millisecond time type. */
 #include "util.h"    /* Misc functions useful in many places */
 #include "latency.h" /* Latency monitor API */
 #include "sparkline.h" /* ASCII graphs API */
-#include "quicklist.h"
+#include "quicklist.h"  /* Lists are encoded as linked lists of
+                           N-elements flat arrays */
+#include "rax.h"     /* Radix tree */

 /* Following includes allow test functions to be called from Redis main() */
 #include "zipmap.h"
@@ -151,6 +153,13 @@ typedef long long mstime_t; /* millisecond time type. */
 #define CONFIG_DEFAULT_LAZYFREE_LAZY_EVICTION 0
 #define CONFIG_DEFAULT_LAZYFREE_LAZY_EXPIRE 0
 #define CONFIG_DEFAULT_LAZYFREE_LAZY_SERVER_DEL 0
+#define CONFIG_DEFAULT_ALWAYS_SHOW_LOGO 0
+#define CONFIG_DEFAULT_ACTIVE_DEFRAG 0
+#define CONFIG_DEFAULT_DEFRAG_THRESHOLD_LOWER 10 /* don't defrag when fragmentation is below 10% */
+#define CONFIG_DEFAULT_DEFRAG_THRESHOLD_UPPER 100 /* maximum defrag force at 100% fragmentation */
+#define CONFIG_DEFAULT_DEFRAG_IGNORE_BYTES (100<<20) /* don't defrag if frag overhead is below 100mb */
+#define CONFIG_DEFAULT_DEFRAG_CYCLE_MIN 25 /* 25% CPU min (at lower threshold) */
+#define CONFIG_DEFAULT_DEFRAG_CYCLE_MAX 75 /* 75% CPU max (at upper threshold) */

 #define ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP 20 /* Loopkups per loop. */
 #define ACTIVE_EXPIRE_CYCLE_FAST_DURATION 1000 /* Microseconds */
@@ -428,7 +437,7 @@ typedef long long mstime_t; /* millisecond time type. */
 /* We can print the stacktrace, so our assert is defined this way: */
 #define serverAssertWithInfo(_c,_o,_e) ((_e)?(void)0 : (_serverAssertWithInfo(_c,_o,#_e,__FILE__,__LINE__),_exit(1)))
 #define serverAssert(_e) ((_e)?(void)0 : (_serverAssert(#_e,__FILE__,__LINE__),_exit(1)))
-#define serverPanic(_e) _serverPanic(#_e,__FILE__,__LINE__),_exit(1)
+#define serverPanic(...) _serverPanic(__FILE__,__LINE__,__VA_ARGS__),_exit(1)

 /*-----------------------------------------------------------------------------
 * Data types
@@ -476,7 +485,7 @@ typedef void *(*moduleTypeLoadFunc)(struct RedisModuleIO *io, int encver);
 typedef void (*moduleTypeSaveFunc)(struct RedisModuleIO *io, void *value);
 typedef void (*moduleTypeRewriteFunc)(struct RedisModuleIO *io, struct redisObject *key, void *value);
 typedef void (*moduleTypeDigestFunc)(struct RedisModuleDigest *digest, void *value);
-typedef size_t (*moduleTypeMemUsageFunc)(void *value);
+typedef size_t (*moduleTypeMemUsageFunc)(const void *value);
 typedef void (*moduleTypeFreeFunc)(void *value);

 /* The module type, which is referenced in each value of a given type, defines
@@ -654,6 +663,9 @@ typedef struct client {
    redisDb *db;            /* Pointer to currently SELECTed DB. */
    robj *name;             /* As set by CLIENT SETNAME. */
    sds querybuf;           /* Buffer we use to accumulate client queries. */
+    sds pending_querybuf;   /* If this is a master, this buffer represents the
+                               yet not applied replication stream that we
+                               are receiving from the master. */
    size_t querybuf_peak;   /* Recent (100ms or more) peak of querybuf size. */
    int argc;               /* Num of arguments of current command. */
    robj **argv;            /* Arguments of current command. */
@@ -676,7 +688,8 @@ typedef struct client {
    off_t repldboff;        /* Replication DB file offset. */
    off_t repldbsize;       /* Replication DB file size. */
    sds replpreamble;       /* Replication DB preamble. */
-    long long reploff;      /* Replication offset if this is our master. */
+    long long read_reploff; /* Read replication offset if this is a master. */
+    long long reploff;      /* Applied replication offset if this is a master. */
    long long repl_ack_off; /* Replication ack offset, if this is a slave. */
    long long repl_ack_time;/* Replication ack time, if this is a slave. */
    long long psync_initial_offset; /* FULLRESYNC reply offset other slaves
@@ -856,6 +869,7 @@ struct redisServer {
    unsigned lruclock:LRU_BITS; /* Clock for LRU eviction */
    int shutdown_asap;          /* SHUTDOWN needed ASAP */
    int activerehashing;        /* Incremental rehash in serverCron() */
+    int active_defrag_running;  /* Active defragmentation running (holds current scan aggressiveness) */
    char *requirepass;          /* Pass for AUTH command, or NULL */
    char *pidfile;              /* PID file path */
    int arch_bits;              /* 32 or 64 depending on sizeof(long) */
@@ -863,9 +877,13 @@ struct redisServer {
    char runid[CONFIG_RUN_ID_SIZE+1];  /* ID always different at every exec. */
    int sentinel_mode;          /* True if this instance is a Sentinel. */
    size_t initial_memory_usage; /* Bytes used after initialization. */
+    int always_show_logo;       /* Show logo even for non-stdout logging. */
    /* Modules */
    dict *moduleapi;            /* Exported APIs dictionary for modules. */
    list *loadmodule_queue;     /* List of modules to load at startup. */
+    int module_blocked_pipe[2]; /* Pipe used to awake the event loop if a
+                                   client blocked on a module command needs
+                                   to be processed. */
    /* Networking */
    int port;                   /* TCP listening port */
    int tcp_backlog;            /* TCP listen() backlog */
@@ -906,6 +924,10 @@ struct redisServer {
    long long stat_evictedkeys;     /* Number of evicted keys (maxmemory) */
    long long stat_keyspace_hits;   /* Number of successful lookups of keys */
    long long stat_keyspace_misses; /* Number of failed lookups of keys */
+    long long stat_active_defrag_hits;      /* number of allocations moved */
+    long long stat_active_defrag_misses;    /* number of allocations scanned but not moved */
+    long long stat_active_defrag_key_hits;  /* number of keys with moved allocations */
+    long long stat_active_defrag_key_misses;/* number of keys scanned and not moved */
    size_t stat_peak_memory;        /* Max used memory record */
    long long stat_fork_time;       /* Time needed to perform latest fork() */
    double stat_fork_rate;          /* Fork rate in GB/sec. */
@@ -935,6 +957,12 @@ struct redisServer {
    int maxidletime;                /* Client timeout in seconds */
    int tcpkeepalive;               /* Set SO_KEEPALIVE if non-zero. */
    int active_expire_enabled;      /* Can be disabled for testing purposes. */
+    int active_defrag_enabled;
+    size_t active_defrag_ignore_bytes; /* minimum amount of fragmentation waste to start active defrag */
+    int active_defrag_threshold_lower; /* minimum percentage of fragmentation to start active defrag */
+    int active_defrag_threshold_upper; /* maximum percentage of fragmentation at which we use maximum effort */
+    int active_defrag_cycle_min;       /* minimal effort for defrag in CPU percentage */
+    int active_defrag_cycle_max;       /* maximal effort for defrag in CPU percentage */
    size_t client_max_querybuf_len; /* Limit for client query buffer length */
    int dbnum;                      /* Total number of configured DBs */
    int supervised;                 /* 1 if supervised, 0 otherwise. */
@@ -1265,6 +1293,7 @@ void moduleFreeContext(struct RedisModuleCtx *ctx);
 void unblockClientFromModule(client *c);
 void moduleHandleBlockedClients(void);
 void moduleBlockedClientTimedOut(client *c);
+void moduleBlockedClientPipeReadable(aeEventLoop *el, int fd, void *privdata, int mask);

 /* Utils */
 long long ustime(void);
@@ -1574,6 +1603,7 @@ void adjustOpenFilesLimit(void);
 void closeListeningSockets(int unlink_unix_socket);
 void updateCachedTime(void);
 void resetServerStats(void);
+void activeDefragCycle(void);
 unsigned int getLRUClock(void);
 const char *evictPolicyToString(void);
 struct redisMemOverhead *getMemoryOverheadData(void);
@@ -1648,13 +1678,15 @@ int removeExpire(redisDb *db, robj *key);
 void propagateExpire(redisDb *db, robj *key, int lazy);
 int expireIfNeeded(redisDb *db, robj *key);
 long long getExpire(redisDb *db, robj *key);
-void setExpire(redisDb *db, robj *key, long long when);
+void setExpire(client *c, redisDb *db, robj *key, long long when);
 robj *lookupKey(redisDb *db, robj *key, int flags);
 robj *lookupKeyRead(redisDb *db, robj *key);
 robj *lookupKeyWrite(redisDb *db, robj *key);
 robj *lookupKeyReadOrReply(client *c, robj *key, robj *reply);
 robj *lookupKeyWriteOrReply(client *c, robj *key, robj *reply);
 robj *lookupKeyReadWithFlags(redisDb *db, robj *key, int flags);
+robj *objectCommandLookup(client *c, robj *key);
+robj *objectCommandLookupOrReply(client *c, robj *key, robj *reply);
 #define LOOKUP_NONE 0
 #define LOOKUP_NOTOUCH (1<<0)
 void dbAdd(redisDb *db, robj *key, robj *val);
@@ -1731,6 +1763,10 @@ void disconnectAllBlockedClients(void);

 /* expire.c -- Handling of expired keys */
 void activeExpireCycle(int type);
+void expireSlaveKeys(void);
+void rememberSlaveKeyWithExpire(redisDb *db, robj *key);
+void flushSlaveKeysWithExpireList(void);
+size_t getSlaveKeyWithExpireCount(void);

 /* evict.c -- maxmemory handling and LRU eviction. */
 void evictionPoolAlloc(void);
@@ -1738,6 +1774,11 @@ void evictionPoolAlloc(void);
 unsigned long LFUGetTimeInMinutes(void);
 uint8_t LFULogIncr(uint8_t value);

+/* Keys hashing / comparison functions for dict.c hash tables. */
+uint64_t dictSdsHash(const void *key);
+int dictSdsKeyCompare(void *privdata, const void *key1, const void *key2);
+void dictSdsDestructor(void *privdata, void *val);
+
 /* Git SHA1 */
 char *redisGitSHA1(void);
 char *redisGitDirty(void);
@@ -1929,7 +1970,7 @@ void *realloc(void *ptr, size_t size) __attribute__ ((deprecated));
 /* Debugging stuff */
 void _serverAssertWithInfo(const client *c, const robj *o, const char *estr, const char *file, int line);
 void _serverAssert(const char *estr, const char *file, int line);
-void _serverPanic(const char *msg, const char *file, int line);
+void _serverPanic(const char *file, int line, const char *msg, ...);
 void bugReportStart(void);
 void serverLogObjectDebugInfo(const robj *o);
 void sigsegvHandler(int sig, siginfo_t *info, void *secret);
@@ -0,0 +1,360 @@
+/*
+   SipHash reference C implementation
+
+   Copyright (c) 2012-2016 Jean-Philippe Aumasson
+   <jeanphilippe.aumasson@gmail.com>
+   Copyright (c) 2012-2014 Daniel J. Bernstein <djb@cr.yp.to>
+   Copyright (c) 2017 Salvatore Sanfilippo <antirez@gmail.com>
+
+   To the extent possible under law, the author(s) have dedicated all copyright
+   and related and neighboring rights to this software to the public domain
+   worldwide. This software is distributed without any warranty.
+
+   You should have received a copy of the CC0 Public Domain Dedication along
+   with this software. If not, see
+   <http://creativecommons.org/publicdomain/zero/1.0/>.
+
+   ----------------------------------------------------------------------------
+
+   This version was modified by Salvatore Sanfilippo <antirez@gmail.com>
+   in the following ways:
+
+   1. We use SipHash 1-2. This is not believed to be as strong as the
+      suggested 2-4 variant, but AFAIK there are not trivial attacks
+      against this reduced-rounds version, and it runs at the same speed
+      as Murmurhash2 that we used previously, why the 2-4 variant slowed
+      down Redis by a 4% figure more or less.
+   2. Hard-code rounds in the hope the compiler can optimize it more
+      in this raw from. Anyway we always want the standard 2-4 variant.
+   3. Modify the prototype and implementation so that the function directly
+      returns an uint64_t value, the hash itself, instead of receiving an
+      output buffer. This also means that the output size is set to 8 bytes
+      and the 16 bytes output code handling was removed.
+   4. Provide a case insensitive variant to be used when hashing strings that
+      must be considered identical by the hash table regardless of the case.
+      If we don't have directly a case insensitive hash function, we need to
+      perform a text transformation in some temporary buffer, which is costly.
+   5. Remove debugging code.
+   6. Modified the original test.c file to be a stand-alone function testing
+      the function in the new form (returing an uint64_t) using just the
+      relevant test vector.
+ */
+#include <assert.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <ctype.h>
+
+/* Fast tolower() alike function that does not care about locale
+ * but just returns a-z insetad of A-Z. */
+int siptlw(int c) {
+    if (c >= 'A' && c <= 'Z') {
+        return c+('a'-'A');
+    } else {
+        return c;
+    }
+}
+
+/* Test of the CPU is Little Endian and supports not aligned accesses.
+ * Two interesting conditions to speedup the function that happen to be
+ * in most of x86 servers. */
+#if defined(__X86_64__) || defined(__x86_64__) || defined (__i386__)
+#define UNALIGNED_LE_CPU
+#endif
+
+#define ROTL(x, b) (uint64_t)(((x) << (b)) | ((x) >> (64 - (b))))
+
+#define U32TO8_LE(p, v)                                                        \
+    (p)[0] = (uint8_t)((v));                                                   \
+    (p)[1] = (uint8_t)((v) >> 8);                                              \
+    (p)[2] = (uint8_t)((v) >> 16);                                             \
+    (p)[3] = (uint8_t)((v) >> 24);
+
+#define U64TO8_LE(p, v)                                                        \
+    U32TO8_LE((p), (uint32_t)((v)));                                           \
+    U32TO8_LE((p) + 4, (uint32_t)((v) >> 32));
+
+#ifdef UNALIGNED_LE_CPU
+#define U8TO64_LE(p) (*((uint64_t*)(p)))
+#else
+#define U8TO64_LE(p)                                                           \
+    (((uint64_t)((p)[0])) | ((uint64_t)((p)[1]) << 8) |                        \
+     ((uint64_t)((p)[2]) << 16) | ((uint64_t)((p)[3]) << 24) |                 \
+     ((uint64_t)((p)[4]) << 32) | ((uint64_t)((p)[5]) << 40) |                 \
+     ((uint64_t)((p)[6]) << 48) | ((uint64_t)((p)[7]) << 56))
+#endif
+
+#define U8TO64_LE_NOCASE(p)                                                    \
+    (((uint64_t)(siptlw((p)[0]))) |                                           \
+     ((uint64_t)(siptlw((p)[1])) << 8) |                                      \
+     ((uint64_t)(siptlw((p)[2])) << 16) |                                     \
+     ((uint64_t)(siptlw((p)[3])) << 24) |                                     \
+     ((uint64_t)(siptlw((p)[4])) << 32) |                                              \
+     ((uint64_t)(siptlw((p)[5])) << 40) |                                              \
+     ((uint64_t)(siptlw((p)[6])) << 48) |                                              \
+     ((uint64_t)(siptlw((p)[7])) << 56))
+
+#define SIPROUND                                                               \
+    do {                                                                       \
+        v0 += v1;                                                              \
+        v1 = ROTL(v1, 13);                                                     \
+        v1 ^= v0;                                                              \
+        v0 = ROTL(v0, 32);                                                     \
+        v2 += v3;                                                              \
+        v3 = ROTL(v3, 16);                                                     \
+        v3 ^= v2;                                                              \
+        v0 += v3;                                                              \
+        v3 = ROTL(v3, 21);                                                     \
+        v3 ^= v0;                                                              \
+        v2 += v1;                                                              \
+        v1 = ROTL(v1, 17);                                                     \
+        v1 ^= v2;                                                              \
+        v2 = ROTL(v2, 32);                                                     \
+    } while (0)
+
+uint64_t siphash(const uint8_t *in, const size_t inlen, const uint8_t *k) {
+#ifndef UNALIGNED_LE_CPU
+    uint64_t hash;
+    uint8_t *out = (uint8_t*) &hash;
+#endif
+    uint64_t v0 = 0x736f6d6570736575ULL;
+    uint64_t v1 = 0x646f72616e646f6dULL;
+    uint64_t v2 = 0x6c7967656e657261ULL;
+    uint64_t v3 = 0x7465646279746573ULL;
+    uint64_t k0 = U8TO64_LE(k);
+    uint64_t k1 = U8TO64_LE(k + 8);
+    uint64_t m;
+    const uint8_t *end = in + inlen - (inlen % sizeof(uint64_t));
+    const int left = inlen & 7;
+    uint64_t b = ((uint64_t)inlen) << 56;
+    v3 ^= k1;
+    v2 ^= k0;
+    v1 ^= k1;
+    v0 ^= k0;
+
+    for (; in != end; in += 8) {
+        m = U8TO64_LE(in);
+        v3 ^= m;
+
+        SIPROUND;
+
+        v0 ^= m;
+    }
+
+    switch (left) {
+    case 7: b |= ((uint64_t)in[6]) << 48;
+    case 6: b |= ((uint64_t)in[5]) << 40;
+    case 5: b |= ((uint64_t)in[4]) << 32;
+    case 4: b |= ((uint64_t)in[3]) << 24;
+    case 3: b |= ((uint64_t)in[2]) << 16;
+    case 2: b |= ((uint64_t)in[1]) << 8;
+    case 1: b |= ((uint64_t)in[0]); break;
+    case 0: break;
+    }
+
+    v3 ^= b;
+
+    SIPROUND;
+
+    v0 ^= b;
+    v2 ^= 0xff;
+
+    SIPROUND;
+    SIPROUND;
+
+    b = v0 ^ v1 ^ v2 ^ v3;
+#ifndef UNALIGNED_LE_CPU
+    U64TO8_LE(out, b);
+    return hash;
+#else
+    return b;
+#endif
+}
+
+uint64_t siphash_nocase(const uint8_t *in, const size_t inlen, const uint8_t *k)
+{
+#ifndef UNALIGNED_LE_CPU
+    uint64_t hash;
+    uint8_t *out = (uint8_t*) &hash;
+#endif
+    uint64_t v0 = 0x736f6d6570736575ULL;
+    uint64_t v1 = 0x646f72616e646f6dULL;
+    uint64_t v2 = 0x6c7967656e657261ULL;
+    uint64_t v3 = 0x7465646279746573ULL;
+    uint64_t k0 = U8TO64_LE(k);
+    uint64_t k1 = U8TO64_LE(k + 8);
+    uint64_t m;
+    const uint8_t *end = in + inlen - (inlen % sizeof(uint64_t));
+    const int left = inlen & 7;
+    uint64_t b = ((uint64_t)inlen) << 56;
+    v3 ^= k1;
+    v2 ^= k0;
+    v1 ^= k1;
+    v0 ^= k0;
+
+    for (; in != end; in += 8) {
+        m = U8TO64_LE_NOCASE(in);
+        v3 ^= m;
+
+        SIPROUND;
+
+        v0 ^= m;
+    }
+
+    switch (left) {
+    case 7: b |= ((uint64_t)siptlw(in[6])) << 48;
+    case 6: b |= ((uint64_t)siptlw(in[5])) << 40;
+    case 5: b |= ((uint64_t)siptlw(in[4])) << 32;
+    case 4: b |= ((uint64_t)siptlw(in[3])) << 24;
+    case 3: b |= ((uint64_t)siptlw(in[2])) << 16;
+    case 2: b |= ((uint64_t)siptlw(in[1])) << 8;
+    case 1: b |= ((uint64_t)siptlw(in[0])); break;
+    case 0: break;
+    }
+
+    v3 ^= b;
+
+    SIPROUND;
+
+    v0 ^= b;
+    v2 ^= 0xff;
+
+    SIPROUND;
+    SIPROUND;
+
+    b = v0 ^ v1 ^ v2 ^ v3;
+#ifndef UNALIGNED_LE_CPU
+    U64TO8_LE(out, b);
+    return hash;
+#else
+    return b;
+#endif
+}
+
+
+/* --------------------------------- TEST ------------------------------------ */
+
+#ifdef SIPHASH_TEST
+
+const uint8_t vectors_sip64[64][8] = {
+    { 0x31, 0x0e, 0x0e, 0xdd, 0x47, 0xdb, 0x6f, 0x72, },
+    { 0xfd, 0x67, 0xdc, 0x93, 0xc5, 0x39, 0xf8, 0x74, },
+    { 0x5a, 0x4f, 0xa9, 0xd9, 0x09, 0x80, 0x6c, 0x0d, },
+    { 0x2d, 0x7e, 0xfb, 0xd7, 0x96, 0x66, 0x67, 0x85, },
+    { 0xb7, 0x87, 0x71, 0x27, 0xe0, 0x94, 0x27, 0xcf, },
+    { 0x8d, 0xa6, 0x99, 0xcd, 0x64, 0x55, 0x76, 0x18, },
+    { 0xce, 0xe3, 0xfe, 0x58, 0x6e, 0x46, 0xc9, 0xcb, },
+    { 0x37, 0xd1, 0x01, 0x8b, 0xf5, 0x00, 0x02, 0xab, },
+    { 0x62, 0x24, 0x93, 0x9a, 0x79, 0xf5, 0xf5, 0x93, },
+    { 0xb0, 0xe4, 0xa9, 0x0b, 0xdf, 0x82, 0x00, 0x9e, },
+    { 0xf3, 0xb9, 0xdd, 0x94, 0xc5, 0xbb, 0x5d, 0x7a, },
+    { 0xa7, 0xad, 0x6b, 0x22, 0x46, 0x2f, 0xb3, 0xf4, },
+    { 0xfb, 0xe5, 0x0e, 0x86, 0xbc, 0x8f, 0x1e, 0x75, },
+    { 0x90, 0x3d, 0x84, 0xc0, 0x27, 0x56, 0xea, 0x14, },
+    { 0xee, 0xf2, 0x7a, 0x8e, 0x90, 0xca, 0x23, 0xf7, },
+    { 0xe5, 0x45, 0xbe, 0x49, 0x61, 0xca, 0x29, 0xa1, },
+    { 0xdb, 0x9b, 0xc2, 0x57, 0x7f, 0xcc, 0x2a, 0x3f, },
+    { 0x94, 0x47, 0xbe, 0x2c, 0xf5, 0xe9, 0x9a, 0x69, },
+    { 0x9c, 0xd3, 0x8d, 0x96, 0xf0, 0xb3, 0xc1, 0x4b, },
+    { 0xbd, 0x61, 0x79, 0xa7, 0x1d, 0xc9, 0x6d, 0xbb, },
+    { 0x98, 0xee, 0xa2, 0x1a, 0xf2, 0x5c, 0xd6, 0xbe, },
+    { 0xc7, 0x67, 0x3b, 0x2e, 0xb0, 0xcb, 0xf2, 0xd0, },
+    { 0x88, 0x3e, 0xa3, 0xe3, 0x95, 0x67, 0x53, 0x93, },
+    { 0xc8, 0xce, 0x5c, 0xcd, 0x8c, 0x03, 0x0c, 0xa8, },
+    { 0x94, 0xaf, 0x49, 0xf6, 0xc6, 0x50, 0xad, 0xb8, },
+    { 0xea, 0xb8, 0x85, 0x8a, 0xde, 0x92, 0xe1, 0xbc, },
+    { 0xf3, 0x15, 0xbb, 0x5b, 0xb8, 0x35, 0xd8, 0x17, },
+    { 0xad, 0xcf, 0x6b, 0x07, 0x63, 0x61, 0x2e, 0x2f, },
+    { 0xa5, 0xc9, 0x1d, 0xa7, 0xac, 0xaa, 0x4d, 0xde, },
+    { 0x71, 0x65, 0x95, 0x87, 0x66, 0x50, 0xa2, 0xa6, },
+    { 0x28, 0xef, 0x49, 0x5c, 0x53, 0xa3, 0x87, 0xad, },
+    { 0x42, 0xc3, 0x41, 0xd8, 0xfa, 0x92, 0xd8, 0x32, },
+    { 0xce, 0x7c, 0xf2, 0x72, 0x2f, 0x51, 0x27, 0x71, },
+    { 0xe3, 0x78, 0x59, 0xf9, 0x46, 0x23, 0xf3, 0xa7, },
+    { 0x38, 0x12, 0x05, 0xbb, 0x1a, 0xb0, 0xe0, 0x12, },
+    { 0xae, 0x97, 0xa1, 0x0f, 0xd4, 0x34, 0xe0, 0x15, },
+    { 0xb4, 0xa3, 0x15, 0x08, 0xbe, 0xff, 0x4d, 0x31, },
+    { 0x81, 0x39, 0x62, 0x29, 0xf0, 0x90, 0x79, 0x02, },
+    { 0x4d, 0x0c, 0xf4, 0x9e, 0xe5, 0xd4, 0xdc, 0xca, },
+    { 0x5c, 0x73, 0x33, 0x6a, 0x76, 0xd8, 0xbf, 0x9a, },
+    { 0xd0, 0xa7, 0x04, 0x53, 0x6b, 0xa9, 0x3e, 0x0e, },
+    { 0x92, 0x59, 0x58, 0xfc, 0xd6, 0x42, 0x0c, 0xad, },
+    { 0xa9, 0x15, 0xc2, 0x9b, 0xc8, 0x06, 0x73, 0x18, },
+    { 0x95, 0x2b, 0x79, 0xf3, 0xbc, 0x0a, 0xa6, 0xd4, },
+    { 0xf2, 0x1d, 0xf2, 0xe4, 0x1d, 0x45, 0x35, 0xf9, },
+    { 0x87, 0x57, 0x75, 0x19, 0x04, 0x8f, 0x53, 0xa9, },
+    { 0x10, 0xa5, 0x6c, 0xf5, 0xdf, 0xcd, 0x9a, 0xdb, },
+    { 0xeb, 0x75, 0x09, 0x5c, 0xcd, 0x98, 0x6c, 0xd0, },
+    { 0x51, 0xa9, 0xcb, 0x9e, 0xcb, 0xa3, 0x12, 0xe6, },
+    { 0x96, 0xaf, 0xad, 0xfc, 0x2c, 0xe6, 0x66, 0xc7, },
+    { 0x72, 0xfe, 0x52, 0x97, 0x5a, 0x43, 0x64, 0xee, },
+    { 0x5a, 0x16, 0x45, 0xb2, 0x76, 0xd5, 0x92, 0xa1, },
+    { 0xb2, 0x74, 0xcb, 0x8e, 0xbf, 0x87, 0x87, 0x0a, },
+    { 0x6f, 0x9b, 0xb4, 0x20, 0x3d, 0xe7, 0xb3, 0x81, },
+    { 0xea, 0xec, 0xb2, 0xa3, 0x0b, 0x22, 0xa8, 0x7f, },
+    { 0x99, 0x24, 0xa4, 0x3c, 0xc1, 0x31, 0x57, 0x24, },
+    { 0xbd, 0x83, 0x8d, 0x3a, 0xaf, 0xbf, 0x8d, 0xb7, },
+    { 0x0b, 0x1a, 0x2a, 0x32, 0x65, 0xd5, 0x1a, 0xea, },
+    { 0x13, 0x50, 0x79, 0xa3, 0x23, 0x1c, 0xe6, 0x60, },
+    { 0x93, 0x2b, 0x28, 0x46, 0xe4, 0xd7, 0x06, 0x66, },
+    { 0xe1, 0x91, 0x5f, 0x5c, 0xb1, 0xec, 0xa4, 0x6c, },
+    { 0xf3, 0x25, 0x96, 0x5c, 0xa1, 0x6d, 0x62, 0x9f, },
+    { 0x57, 0x5f, 0xf2, 0x8e, 0x60, 0x38, 0x1b, 0xe5, },
+    { 0x72, 0x45, 0x06, 0xeb, 0x4c, 0x32, 0x8a, 0x95, },
+};
+
+
+/* Test siphash using a test vector. Returns 0 if the function passed
+ * all the tests, otherwise 1 is returned.
+ *
+ * IMPORTANT: The test vector is for SipHash 2-4. Before running
+ * the test revert back the siphash() function to 2-4 rounds since
+ * now it uses 1-2 rounds. */
+int siphash_test(void) {
+    uint8_t in[64], k[16];
+    int i;
+    int fails = 0;
+
+    for (i = 0; i < 16; ++i)
+        k[i] = i;
+
+    for (i = 0; i < 64; ++i) {
+        in[i] = i;
+        uint64_t hash = siphash(in, i, k);
+        const uint8_t *v = NULL;
+        v = (uint8_t *)vectors_sip64;
+        if (memcmp(&hash, v + (i * 8), 8)) {
+            /* printf("fail for %d bytes\n", i); */
+            fails++;
+        }
+    }
+
+    /* Run a few basic tests with the case insensitive version. */
+    uint64_t h1, h2;
+    h1 = siphash((uint8_t*)"hello world",11,(uint8_t*)"1234567812345678");
+    h2 = siphash_nocase((uint8_t*)"hello world",11,(uint8_t*)"1234567812345678");
+    if (h1 != h2) fails++;
+
+    h1 = siphash((uint8_t*)"hello world",11,(uint8_t*)"1234567812345678");
+    h2 = siphash_nocase((uint8_t*)"HELLO world",11,(uint8_t*)"1234567812345678");
+    if (h1 != h2) fails++;
+
+    h1 = siphash((uint8_t*)"HELLO world",11,(uint8_t*)"1234567812345678");
+    h2 = siphash_nocase((uint8_t*)"HELLO world",11,(uint8_t*)"1234567812345678");
+    if (h1 == h2) fails++;
+
+    if (!fails) return 0;
+    return 1;
+}
+
+int main(void) {
+    if (siphash_test() == 0) {
+        printf("SipHash test: OK\n");
+        return 0;
+    } else {
+        printf("SipHash test: FAILED\n");
+        return 1;
+    }
+}
+
+#endif
@@ -85,7 +85,7 @@ void setGenericCommand(client *c, int flags, robj *key, robj *val, robj *expire,
    }
    setKey(c->db,key,val);
    server.dirty++;
-    if (expire) setExpire(c->db,key,mstime()+milliseconds);
+    if (expire) setExpire(c,c->db,key,mstime()+milliseconds);
    notifyKeyspaceEvent(NOTIFY_STRING,"set",key,c->db->id);
    if (expire) notifyKeyspaceEvent(NOTIFY_GENERIC,
        "expire",key,c->db->id);
@@ -1521,7 +1521,7 @@ void zaddGenericCommand(client *c, int flags) {
    /* After the options, we expect to have an even number of args, since
     * we expect any number of score-element pairs. */
    elements = c->argc-scoreidx;
-    if (elements % 2) {
+    if (elements % 2 || !elements) {
        addReply(c,shared.syntaxerr);
        return;
    }
@@ -2110,7 +2110,7 @@ inline static void zunionInterAggregate(double *target, double val, int aggregat
    }
 }

-unsigned int dictSdsHash(const void *key);
+uint64_t dictSdsHash(const void *key);
 int dictSdsKeyCompare(void *privdata, const void *key1, const void *key2);

 dictType setAccumulatorDictType = {
@@ -1 +1 @@
-#define REDIS_VERSION "999.999.999"
+#define REDIS_VERSION "3.9.103"
@@ -8,72 +8,150 @@
 *
 * ----------------------------------------------------------------------------
 *
- * ZIPLIST OVERALL LAYOUT:
+ * ZIPLIST OVERALL LAYOUT
+ * ======================
+ *
 * The general layout of the ziplist is as follows:
- * <zlbytes><zltail><zllen><entry><entry><zlend>
 *
- * <zlbytes> is an unsigned integer to hold the number of bytes that the
- * ziplist occupies. This value needs to be stored to be able to resize the
- * entire structure without the need to traverse it first.
+ * <zlbytes> <zltail> <zllen> <entry> <entry> ... <entry> <zlend>
 *
- * <zltail> is the offset to the last entry in the list. This allows a pop
- * operation on the far side of the list without the need for full traversal.
+ * NOTE: all fields are stored in little endian, if not specified otherwise.
 *
- * <zllen> is the number of entries.When this value is larger than 2**16-2,
- * we need to traverse the entire list to know how many items it holds.
+ * <uint32_t zlbytes> is an unsigned integer to hold the number of bytes that
+ * the ziplist occupies, including the four bytes of the zlbytes field itself.
+ * This value needs to be stored to be able to resize the entire structure
+ * without the need to traverse it first.
 *
- * <zlend> is a single byte special value, equal to 255, which indicates the
- * end of the list.
+ * <uint32_t zltail> is the offset to the last entry in the list. This allows
+ * a pop operation on the far side of the list without the need for full
+ * traversal.
 *
- * ZIPLIST ENTRIES:
- * Every entry in the ziplist is prefixed by a header that contains two pieces
+ * <uint16_t zllen> is the number of entries. When there are more than
+ * 2^16-2 entires, this value is set to 2^16-1 and we need to traverse the
+ * entire list to know how many items it holds.
+ *
+ * <uint8_t zlend> is a special entry representing the end of the ziplist.
+ * Is encoded as a single byte equal to 255. No other normal entry starts
+ * with a byte set to the value of 255.
+ *
+ * ZIPLIST ENTRIES
+ * ===============
+ *
+ * Every entry in the ziplist is prefixed by metadata that contains two pieces
 * of information. First, the length of the previous entry is stored to be
- * able to traverse the list from back to front. Second, the encoding with an
- * optional string length of the entry itself is stored.
+ * able to traverse the list from back to front. Second, the entry encoding is
+ * provided. It represents the entry type, integer or string, and in the case
+ * of strings it also represents the length of the string payload.
+ * So a complete entry is stored like this:
 *
- * The length of the previous entry is encoded in the following way:
- * If this length is smaller than 254 bytes, it will only consume a single
- * byte that takes the length as value. When the length is greater than or
- * equal to 254, it will consume 5 bytes. The first byte is set to 254 to
- * indicate a larger value is following. The remaining 4 bytes take the
- * length of the previous entry as value.
+ * <prevlen> <encoding> <entry-data>
 *
- * The other header field of the entry itself depends on the contents of the
- * entry. When the entry is a string, the first 2 bits of this header will hold
- * the type of encoding used to store the length of the string, followed by the
- * actual length of the string. When the entry is an integer the first 2 bits
- * are both set to 1. The following 2 bits are used to specify what kind of
- * integer will be stored after this header. An overview of the different
- * types and encodings is as follows:
+ * Sometimes the encoding represents the entry itself, like for small integers
+ * as we'll see later. In such a case the <entry-data> part is missing, and we
+ * could have just:
+ *
+ * <prevlen> <encoding>
+ *
+ * The length of the previous entry, <prevlen>, is encoded in the following way:
+ * If this length is smaller than 255 bytes, it will only consume a single
+ * byte representing the length as an unsinged 8 bit integer. When the length
+ * is greater than or equal to 255, it will consume 5 bytes. The first byte is
+ * set to 255 (FF) to indicate a larger value is following. The remaining 4
+ * bytes take the length of the previous entry as value.
+ *
+ * So practically an entry is encoded in the following way:
+ *
+ * <prevlen from 0 to 254> <encoding> <entry>
+ *
+ * Or alternatively if the previous entry length is greater than 254 bytes
+ * the following encoding is used:
+ *
+ * 0xFF <4 bytes unsigned little endian prevlen> <encoding> <entry>
+ *
+ * The encoding field of the entry depends on the content of the
+ * entry. When the entry is a string, the first 2 bits of the encoding first
+ * byte will hold the type of encoding used to store the length of the string,
+ * followed by the actual length of the string. When the entry is an integer
+ * the first 2 bits are both set to 1. The following 2 bits are used to specify
+ * what kind of integer will be stored after this header. An overview of the
+ * different types and encodings is as follows. The first byte is always enough
+ * to determine the kind of entry.
 *
 * |00pppppp| - 1 byte
 *      String value with length less than or equal to 63 bytes (6 bits).
+ *      "pppppp" represents the unsigned 6 bit length.
 * |01pppppp|qqqqqqqq| - 2 bytes
 *      String value with length less than or equal to 16383 bytes (14 bits).
- * |10______|qqqqqqqq|rrrrrrrr|ssssssss|tttttttt| - 5 bytes
+ *      IMPORTANT: The 14 bit number is stored in big endian.
+ * |10000000|qqqqqqqq|rrrrrrrr|ssssssss|tttttttt| - 5 bytes
 *      String value with length greater than or equal to 16384 bytes.
- * |11000000| - 1 byte
+ *      Only the 4 bytes following the first byte represents the length
+ *      up to 32^2-1. The 6 lower bits of the first byte are not used and
+ *      are set to zero.
+ *      IMPORTANT: The 32 bit number is stored in big endian.
+ * |11000000| - 3 bytes
 *      Integer encoded as int16_t (2 bytes).
- * |11010000| - 1 byte
+ * |11010000| - 5 bytes
 *      Integer encoded as int32_t (4 bytes).
- * |11100000| - 1 byte
+ * |11100000| - 9 bytes
 *      Integer encoded as int64_t (8 bytes).
- * |11110000| - 1 byte
+ * |11110000| - 4 bytes
 *      Integer encoded as 24 bit signed (3 bytes).
- * |11111110| - 1 byte
+ * |11111110| - 2 bytes
 *      Integer encoded as 8 bit signed (1 byte).
 * |1111xxxx| - (with xxxx between 0000 and 1101) immediate 4 bit integer.
 *      Unsigned integer from 0 to 12. The encoded value is actually from
 *      1 to 13 because 0000 and 1111 can not be used, so 1 should be
 *      subtracted from the encoded 4 bit value to obtain the right value.
- * |11111111| - End of ziplist.
+ * |11111111| - End of ziplist special entry.
 *
- * All the integers are represented in little endian byte order.
+ * Like for the ziplist header, all the integers are represented in little
+ * endian byte order, even when this code is compiled in big endian systems.
+ *
+ * EXAMPLES OF ACTUAL ZIPLISTS
+ * ===========================
+ *
+ * The following is a ziplist containing the two elements representing
+ * the strings "2" and "5". It is composed of 15 bytes, that we visually
+ * split into sections:
+ *
+ *  [0f 00 00 00] [0c 00 00 00] [02 00] [00 f3] [02 f6] [ff]
+ *        |             |          |       |       |     |
+ *     zlbytes        zltail    entries   "2"     "5"   end
+ *
+ * The first 4 bytes represent the number 15, that is the number of bytes
+ * the whole ziplist is composed of. The second 4 bytes are the offset
+ * at which the last ziplist entry is found, that is 12, in fact the
+ * last entry, that is "5", is at offset 12 inside the ziplist.
+ * The next 16 bit integer represents the number of elements inside the
+ * ziplist, its value is 2 since there are just two elements inside.
+ * Finally "00 f3" is the first entry representing the number 2. It is
+ * composed of the previous entry length, which is zero because this is
+ * our first entry, and the byte F3 which corresponds to the encoding
+ * |1111xxxx| with xxxx between 0001 and 1101. We need to remove the "F"
+ * higher order bits 1111, and subtract 1 from the "3", so the entry value
+ * is "2". The next entry has a prevlen of 02, since the first entry is
+ * composed of exactly two bytes. The entry itself, F6, is encoded exactly
+ * like the first entry, and 6-1 = 5, so the value of the entry is 5.
+ * Finally the special entry FF signals the end of the ziplist.
+ *
+ * Adding another element to the above string with the value "Hello World"
+ * allows us to show how the ziplist encodes small strings. We'll just show
+ * the hex dump of the entry itself. Imagine the bytes as following the
+ * entry that stores "5" in the ziplist above:
+ *
+ * [02] [0b] [48 65 6c 6c 6f 20 57 6f 72 6c 64]
+ *
+ * The first byte, 02, is the length of the previous entry. The next
+ * byte represents the encoding in the pattern |00pppppp| that means
+ * that the entry is a string of length <pppppp>, so 0B means that
+ * an 11 bytes string follows. From the third byte (48) to the last (64)
+ * there are just the ASCII characters for "Hello World".
 *
 * ----------------------------------------------------------------------------
 *
 * Copyright (c) 2009-2012, Pieter Noordhuis <pcnoordhuis at gmail dot com>
- * Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
+ * Copyright (c) 2009-2017, Salvatore Sanfilippo <antirez at gmail dot com>
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
@@ -112,8 +190,13 @@
 #include "endianconv.h"
 #include "redisassert.h"

-#define ZIP_END 255
-#define ZIP_BIGLEN 254
+#define ZIP_END 255         /* Special "end of ziplist" entry. */
+#define ZIP_BIG_PREVLEN 254 /* Max number of bytes of the previous entry, for
+                               the "prevlen" field prefixing each entry, to be
+                               represented with just a single byte. Otherwise
+                               it is represented as FF AA BB CC DD, where
+                               AA BB CC DD are a 4 bytes unsigned integer
+                               representing the previous entry len. */

 /* Different encoding/length possibilities */
 #define ZIP_STR_MASK 0xc0
@@ -126,41 +209,83 @@
 #define ZIP_INT_64B (0xc0 | 2<<4)
 #define ZIP_INT_24B (0xc0 | 3<<4)
 #define ZIP_INT_8B 0xfe
-/* 4 bit integer immediate encoding */
-#define ZIP_INT_IMM_MASK 0x0f
+
+/* 4 bit integer immediate encoding |1111xxxx| with xxxx between
+ * 0001 and 1101. */
+#define ZIP_INT_IMM_MASK 0x0f   /* Mask to extract the 4 bits value. To add
+                                   one is needed to reconstruct the value. */
 #define ZIP_INT_IMM_MIN 0xf1    /* 11110001 */
 #define ZIP_INT_IMM_MAX 0xfd    /* 11111101 */
-#define ZIP_INT_IMM_VAL(v) (v & ZIP_INT_IMM_MASK)

 #define INT24_MAX 0x7fffff
 #define INT24_MIN (-INT24_MAX - 1)

-/* Macro to determine type */
+/* Macro to determine if the entry is a string. String entries never start
+ * with "11" as most significant bits of the first byte. */
 #define ZIP_IS_STR(enc) (((enc) & ZIP_STR_MASK) < ZIP_STR_MASK)

-/* Utility macros */
+/* Utility macros.*/
+
+/* Return total bytes a ziplist is composed of. */
 #define ZIPLIST_BYTES(zl)       (*((uint32_t*)(zl)))
+
+/* Return the offset of the last item inside the ziplist. */
 #define ZIPLIST_TAIL_OFFSET(zl) (*((uint32_t*)((zl)+sizeof(uint32_t))))
+
+/* Return the length of a ziplist, or UINT16_MAX if the length cannot be
+ * determined without scanning the whole ziplist. */
 #define ZIPLIST_LENGTH(zl)      (*((uint16_t*)((zl)+sizeof(uint32_t)*2)))
+
+/* The size of a ziplist header: two 32 bit integers for the total
+ * bytes count and last item offset. One 16 bit integer for the number
+ * of items field. */
 #define ZIPLIST_HEADER_SIZE     (sizeof(uint32_t)*2+sizeof(uint16_t))
+
+/* Size of the "end of ziplist" entry. Just one byte. */
 #define ZIPLIST_END_SIZE        (sizeof(uint8_t))
+
+/* Return the pointer to the first entry of a ziplist. */
 #define ZIPLIST_ENTRY_HEAD(zl)  ((zl)+ZIPLIST_HEADER_SIZE)
+
+/* Return the pointer to the last entry of a ziplist, using the
+ * last entry offset inside the ziplist header. */
 #define ZIPLIST_ENTRY_TAIL(zl)  ((zl)+intrev32ifbe(ZIPLIST_TAIL_OFFSET(zl)))
+
+/* Return the pointer to the last byte of a ziplist, which is, the
+ * end of ziplist FF entry. */
 #define ZIPLIST_ENTRY_END(zl)   ((zl)+intrev32ifbe(ZIPLIST_BYTES(zl))-1)

-/* We know a positive increment can only be 1 because entries can only be
- * pushed one at a time. */
+/* Increment the number of items field in the ziplist header. Note that this
+ * macro should never overflow the unsigned 16 bit integer, since entires are
+ * always pushed one at a time. When UINT16_MAX is reached we want the count
+ * to stay there to signal that a full scan is needed to get the number of
+ * items inside the ziplist. */
 #define ZIPLIST_INCR_LENGTH(zl,incr) { \
    if (ZIPLIST_LENGTH(zl) < UINT16_MAX) \
        ZIPLIST_LENGTH(zl) = intrev16ifbe(intrev16ifbe(ZIPLIST_LENGTH(zl))+incr); \
 }

+/* We use this function to receive information about a ziplist entry.
+ * Note that this is not how the data is actually encoded, is just what we
+ * get filled by a function in order to operate more easily. */
 typedef struct zlentry {
-    unsigned int prevrawlensize, prevrawlen;
-    unsigned int lensize, len;
-    unsigned int headersize;
-    unsigned char encoding;
-    unsigned char *p;
+    unsigned int prevrawlensize; /* Bytes used to encode the previos entry len*/
+    unsigned int prevrawlen;     /* Previous entry len. */
+    unsigned int lensize;        /* Bytes used to encode this entry type/len.
+                                    For example strings have a 1, 2 or 5 bytes
+                                    header. Integers always use a single byte.*/
+    unsigned int len;            /* Bytes used to represent the actual entry.
+                                    For strings this is just the string length
+                                    while for integers it is 1, 2, 3, 4, 8 or
+                                    0 (for 4 bit immediate) depending on the
+                                    number range. */
+    unsigned int headersize;     /* prevrawlensize + lensize. */
+    unsigned char encoding;      /* Set to ZIP_STR_* or ZIP_INT_* depending on
+                                    the entry encoding. However for 4 bits
+                                    immediate integers this can assume a range
+                                    of values and must be range-checked. */
+    unsigned char *p;            /* Pointer to the very start of the entry, that
+                                    is, this points to prev-entry-len field. */
 } zlentry;

 #define ZIPLIST_ENTRY_ZERO(zle) { \
@@ -171,31 +296,40 @@ typedef struct zlentry {
 }

 /* Extract the encoding from the byte pointed by 'ptr' and set it into
- * 'encoding'. */
+ * 'encoding' field of the zlentry structure. */
 #define ZIP_ENTRY_ENCODING(ptr, encoding) do {  \
    (encoding) = (ptr[0]); \
    if ((encoding) < ZIP_STR_MASK) (encoding) &= ZIP_STR_MASK; \
 } while(0)

-void ziplistRepr(unsigned char *zl);
-
-/* Return bytes needed to store integer encoded by 'encoding' */
-static unsigned int zipIntSize(unsigned char encoding) {
+/* Return bytes needed to store integer encoded by 'encoding'. */
+unsigned int zipIntSize(unsigned char encoding) {
    switch(encoding) {
    case ZIP_INT_8B:  return 1;
    case ZIP_INT_16B: return 2;
    case ZIP_INT_24B: return 3;
    case ZIP_INT_32B: return 4;
    case ZIP_INT_64B: return 8;
-    default: return 0; /* 4 bit immediate */
    }
-    assert(NULL);
+    if (encoding >= ZIP_INT_IMM_MIN && encoding <= ZIP_INT_IMM_MAX)
+        return 0; /* 4 bit immediate */
+    panic("Invalid integer encoding 0x%02X", encoding);
    return 0;
 }

-/* Encode the length 'rawlen' writing it in 'p'. If p is NULL it just returns
- * the amount of bytes required to encode such a length. */
-static unsigned int zipEncodeLength(unsigned char *p, unsigned char encoding, unsigned int rawlen) {
+/* Write the encoidng header of the entry in 'p'. If p is NULL it just returns
+ * the amount of bytes required to encode such a length. Arguments:
+ *
+ * 'encoding' is the encoding we are using for the entry. It could be
+ * ZIP_INT_* or ZIP_STR_* or between ZIP_INT_IMM_MIN and ZIP_INT_IMM_MAX
+ * for single-byte small immediate integers.
+ *
+ * 'rawlen' is only used for ZIP_STR_* encodings and is the length of the
+ * srting that this entry represents.
+ *
+ * The function returns the number of bytes used by the encoding/length
+ * header stored in 'p'. */
+unsigned int zipStoreEntryEncoding(unsigned char *p, unsigned char encoding, unsigned int rawlen) {
    unsigned char len = 1, buf[5];

    if (ZIP_IS_STR(encoding)) {
@@ -224,15 +358,16 @@ static unsigned int zipEncodeLength(unsigned char *p, unsigned char encoding, un
        buf[0] = encoding;
    }

-    /* Store this length at p */
+    /* Store this length at p. */
    memcpy(p,buf,len);
    return len;
 }

-/* Decode the length encoded in 'ptr'. The 'encoding' variable will hold the
- * entries encoding, the 'lensize' variable will hold the number of bytes
- * required to encode the entries length, and the 'len' variable will hold the
- * entries length. */
+/* Decode the entry encoding type and data length (string length for strings,
+ * number of bytes used for the integer for integer entries) encoded in 'ptr'.
+ * The 'encoding' variable will hold the entry encoding, the 'lensize'
+ * variable will hold the number of bytes required to encode the entry
+ * length, and the 'len' variable will hold the entry length. */
 #define ZIP_DECODE_LENGTH(ptr, encoding, lensize, len) do {                    \
    ZIP_ENTRY_ENCODING((ptr), (encoding));                                     \
    if ((encoding) < ZIP_STR_MASK) {                                           \
@@ -242,14 +377,14 @@ static unsigned int zipEncodeLength(unsigned char *p, unsigned char encoding, un
        } else if ((encoding) == ZIP_STR_14B) {                                \
            (lensize) = 2;                                                     \
            (len) = (((ptr)[0] & 0x3f) << 8) | (ptr)[1];                       \
-        } else if (encoding == ZIP_STR_32B) {                                  \
+        } else if ((encoding) == ZIP_STR_32B) {                                \
            (lensize) = 5;                                                     \
            (len) = ((ptr)[1] << 24) |                                         \
                    ((ptr)[2] << 16) |                                         \
                    ((ptr)[3] <<  8) |                                         \
                    ((ptr)[4]);                                                \
        } else {                                                               \
-            assert(NULL);                                                      \
+            panic("Invalid string encoding 0x%02X", (encoding));               \
        }                                                                      \
    } else {                                                                   \
        (lensize) = 1;                                                         \
@@ -257,45 +392,49 @@ static unsigned int zipEncodeLength(unsigned char *p, unsigned char encoding, un
    }                                                                          \
 } while(0);

+/* Encode the length of the previous entry and write it to "p". This only
+ * uses the larger encoding (required in __ziplistCascadeUpdate). */
+int zipStorePrevEntryLengthLarge(unsigned char *p, unsigned int len) {
+    if (p != NULL) {
+        p[0] = ZIP_BIG_PREVLEN;
+        memcpy(p+1,&len,sizeof(len));
+        memrev32ifbe(p+1);
+    }
+    return 1+sizeof(len);
+}
+
 /* Encode the length of the previous entry and write it to "p". Return the
 * number of bytes needed to encode this length if "p" is NULL. */
-static unsigned int zipPrevEncodeLength(unsigned char *p, unsigned int len) {
+unsigned int zipStorePrevEntryLength(unsigned char *p, unsigned int len) {
    if (p == NULL) {
-        return (len < ZIP_BIGLEN) ? 1 : sizeof(len)+1;
+        return (len < ZIP_BIG_PREVLEN) ? 1 : sizeof(len)+1;
    } else {
-        if (len < ZIP_BIGLEN) {
+        if (len < ZIP_BIG_PREVLEN) {
            p[0] = len;
            return 1;
        } else {
-            p[0] = ZIP_BIGLEN;
-            memcpy(p+1,&len,sizeof(len));
-            memrev32ifbe(p+1);
-            return 1+sizeof(len);
+            return zipStorePrevEntryLengthLarge(p,len);
        }
    }
 }

-/* Encode the length of the previous entry and write it to "p". This only
- * uses the larger encoding (required in __ziplistCascadeUpdate). */
-static void zipPrevEncodeLengthForceLarge(unsigned char *p, unsigned int len) {
-    if (p == NULL) return;
-    p[0] = ZIP_BIGLEN;
-    memcpy(p+1,&len,sizeof(len));
-    memrev32ifbe(p+1);
-}
-
-/* Decode the number of bytes required to store the length of the previous
- * element, from the perspective of the entry pointed to by 'ptr'. */
+/* Return the number of bytes used to encode the length of the previous
+ * entry. The length is returned by setting the var 'prevlensize'. */
 #define ZIP_DECODE_PREVLENSIZE(ptr, prevlensize) do {                          \
-    if ((ptr)[0] < ZIP_BIGLEN) {                                               \
+    if ((ptr)[0] < ZIP_BIG_PREVLEN) {                                          \
        (prevlensize) = 1;                                                     \
    } else {                                                                   \
        (prevlensize) = 5;                                                     \
    }                                                                          \
 } while(0);

-/* Decode the length of the previous element, from the perspective of the entry
- * pointed to by 'ptr'. */
+/* Return the length of the previous element, and the number of bytes that
+ * are used in order to encode the previous element length.
+ * 'ptr' must point to the prevlen prefix of an entry (that encodes the
+ * length of the previos entry in order to navigate the elements backward).
+ * The length of the previous entry is stored in 'prevlen', the number of
+ * bytes needed to encode the previous entry length are stored in
+ * 'prevlensize'. */
 #define ZIP_DECODE_PREVLEN(ptr, prevlensize, prevlen) do {                     \
    ZIP_DECODE_PREVLENSIZE(ptr, prevlensize);                                  \
    if ((prevlensize) == 1) {                                                  \
@@ -307,16 +446,29 @@ static void zipPrevEncodeLengthForceLarge(unsigned char *p, unsigned int len) {
    }                                                                          \
 } while(0);

-/* Return the difference in number of bytes needed to store the length of the
- * previous element 'len', in the entry pointed to by 'p'. */
-static int zipPrevLenByteDiff(unsigned char *p, unsigned int len) {
+/* Given a pointer 'p' to the prevlen info that prefixes an entry, this
+ * function returns the difference in number of bytes needed to encode
+ * the prevlen if the previous entry changes of size.
+ *
+ * So if A is the number of bytes used right now to encode the 'prevlen'
+ * field.
+ *
+ * And B is the number of bytes that are needed in order to encode the
+ * 'prevlen' if the previous element will be updated to one of size 'len'.
+ *
+ * Then the function returns B - A
+ *
+ * So the function returns a positive number if more space is needed,
+ * a negative number if less space is needed, or zero if the same space
+ * is needed. */
+int zipPrevLenByteDiff(unsigned char *p, unsigned int len) {
    unsigned int prevlensize;
    ZIP_DECODE_PREVLENSIZE(p, prevlensize);
-    return zipPrevEncodeLength(NULL, len) - prevlensize;
+    return zipStorePrevEntryLength(NULL, len) - prevlensize;
 }

 /* Return the total number of bytes used by the entry pointed to by 'p'. */
-static unsigned int zipRawEntryLength(unsigned char *p) {
+unsigned int zipRawEntryLength(unsigned char *p) {
    unsigned int prevlensize, encoding, lensize, len;
    ZIP_DECODE_PREVLENSIZE(p, prevlensize);
    ZIP_DECODE_LENGTH(p + prevlensize, encoding, lensize, len);
@@ -325,7 +477,7 @@ static unsigned int zipRawEntryLength(unsigned char *p) {

 /* Check if string pointed to by 'entry' can be encoded as an integer.
 * Stores the integer value in 'v' and its encoding in 'encoding'. */
-static int zipTryEncoding(unsigned char *entry, unsigned int entrylen, long long *v, unsigned char *encoding) {
+int zipTryEncoding(unsigned char *entry, unsigned int entrylen, long long *v, unsigned char *encoding) {
    long long value;

    if (entrylen >= 32 || entrylen == 0) return 0;
@@ -352,7 +504,7 @@ static int zipTryEncoding(unsigned char *entry, unsigned int entrylen, long long
 }

 /* Store integer 'value' at 'p', encoded as 'encoding' */
-static void zipSaveInteger(unsigned char *p, int64_t value, unsigned char encoding) {
+void zipSaveInteger(unsigned char *p, int64_t value, unsigned char encoding) {
    int16_t i16;
    int32_t i32;
    int64_t i64;
@@ -382,7 +534,7 @@ static void zipSaveInteger(unsigned char *p, int64_t value, unsigned char encodi
 }

 /* Read integer encoded as 'encoding' from 'p' */
-static int64_t zipLoadInteger(unsigned char *p, unsigned char encoding) {
+int64_t zipLoadInteger(unsigned char *p, unsigned char encoding) {
    int16_t i16;
    int32_t i32;
    int64_t i64, ret = 0;
@@ -414,7 +566,7 @@ static int64_t zipLoadInteger(unsigned char *p, unsigned char encoding) {
 }

 /* Return a struct with all information about an entry. */
-static void zipEntry(unsigned char *p, zlentry *e) {
+void zipEntry(unsigned char *p, zlentry *e) {

    ZIP_DECODE_PREVLEN(p, e->prevrawlensize, e->prevrawlen);
    ZIP_DECODE_LENGTH(p + e->prevrawlensize, e->encoding, e->lensize, e->len);
@@ -434,7 +586,7 @@ unsigned char *ziplistNew(void) {
 }

 /* Resize the ziplist. */
-static unsigned char *ziplistResize(unsigned char *zl, unsigned int len) {
+unsigned char *ziplistResize(unsigned char *zl, unsigned int len) {
    zl = zrealloc(zl,len);
    ZIPLIST_BYTES(zl) = intrev32ifbe(len);
    zl[len-1] = ZIP_END;
@@ -449,8 +601,8 @@ static unsigned char *ziplistResize(unsigned char *zl, unsigned int len) {
 * causes a realloc and memmove). However, encoding the prevlen may require
 * that this entry is grown as well. This effect may cascade throughout
 * the ziplist when there are consecutive entries with a size close to
- * ZIP_BIGLEN, so we need to check that the prevlen can be encoded in every
- * consecutive entry.
+ * ZIP_BIG_PREVLEN, so we need to check that the prevlen can be encoded in
+ * every consecutive entry.
 *
 * Note that this effect can also happen in reverse, where the bytes required
 * to encode the prevlen field can shrink. This effect is deliberately ignored,
@@ -461,7 +613,7 @@ static unsigned char *ziplistResize(unsigned char *zl, unsigned int len) {
 *
 * The pointer "p" points to the first entry that does NOT need to be
 * updated, i.e. consecutive fields MAY need an update. */
-static unsigned char *__ziplistCascadeUpdate(unsigned char *zl, unsigned char *p) {
+unsigned char *__ziplistCascadeUpdate(unsigned char *zl, unsigned char *p) {
    size_t curlen = intrev32ifbe(ZIPLIST_BYTES(zl)), rawlen, rawlensize;
    size_t offset, noffset, extra;
    unsigned char *np;
@@ -470,7 +622,7 @@ static unsigned char *__ziplistCascadeUpdate(unsigned char *zl, unsigned char *p
    while (p[0] != ZIP_END) {
        zipEntry(p, &cur);
        rawlen = cur.headersize + cur.len;
-        rawlensize = zipPrevEncodeLength(NULL,rawlen);
+        rawlensize = zipStorePrevEntryLength(NULL,rawlen);

        /* Abort if there is no next entry. */
        if (p[rawlen] == ZIP_END) break;
@@ -501,7 +653,7 @@ static unsigned char *__ziplistCascadeUpdate(unsigned char *zl, unsigned char *p
            memmove(np+rawlensize,
                np+next.prevrawlensize,
                curlen-noffset-next.prevrawlensize-1);
-            zipPrevEncodeLength(np,rawlen);
+            zipStorePrevEntryLength(np,rawlen);

            /* Advance the cursor */
            p += rawlen;
@@ -510,9 +662,9 @@ static unsigned char *__ziplistCascadeUpdate(unsigned char *zl, unsigned char *p
            if (next.prevrawlensize > rawlensize) {
                /* This would result in shrinking, which we want to avoid.
                 * So, set "rawlen" in the available bytes. */
-                zipPrevEncodeLengthForceLarge(p+rawlen,rawlen);
+                zipStorePrevEntryLengthLarge(p+rawlen,rawlen);
            } else {
-                zipPrevEncodeLength(p+rawlen,rawlen);
+                zipStorePrevEntryLength(p+rawlen,rawlen);
            }

            /* Stop here, as the raw length of "next" has not changed. */
@@ -523,7 +675,7 @@ static unsigned char *__ziplistCascadeUpdate(unsigned char *zl, unsigned char *p
 }

 /* Delete "num" entries, starting at "p". Returns pointer to the ziplist. */
-static unsigned char *__ziplistDelete(unsigned char *zl, unsigned char *p, unsigned int num) {
+unsigned char *__ziplistDelete(unsigned char *zl, unsigned char *p, unsigned int num) {
    unsigned int i, totlen, deleted = 0;
    size_t offset;
    int nextdiff = 0;
@@ -535,7 +687,7 @@ static unsigned char *__ziplistDelete(unsigned char *zl, unsigned char *p, unsig
        deleted++;
    }

-    totlen = p-first.p;
+    totlen = p-first.p; /* Bytes taken by the element(s) to delete. */
    if (totlen > 0) {
        if (p[0] != ZIP_END) {
            /* Storing `prevrawlen` in this entry may increase or decrease the
@@ -543,8 +695,13 @@ static unsigned char *__ziplistDelete(unsigned char *zl, unsigned char *p, unsig
             * There always is room to store this, because it was previously
             * stored by an entry that is now being deleted. */
            nextdiff = zipPrevLenByteDiff(p,first.prevrawlen);
+
+            /* Note that there is always space when p jumps backward: if
+             * the new previous entry is large, one of the deleted elements
+             * had a 5 bytes prevlen header, so there is for sure at least
+             * 5 bytes free and we need just 4. */
            p -= nextdiff;
-            zipPrevEncodeLength(p,first.prevrawlen);
+            zipStorePrevEntryLength(p,first.prevrawlen);

            /* Update offset for tail */
            ZIPLIST_TAIL_OFFSET(zl) =
@@ -583,7 +740,7 @@ static unsigned char *__ziplistDelete(unsigned char *zl, unsigned char *p, unsig
 }

 /* Insert item at "p". */
-static unsigned char *__ziplistInsert(unsigned char *zl, unsigned char *p, unsigned char *s, unsigned int slen) {
+unsigned char *__ziplistInsert(unsigned char *zl, unsigned char *p, unsigned char *s, unsigned int slen) {
    size_t curlen = intrev32ifbe(ZIPLIST_BYTES(zl)), reqlen;
    unsigned int prevlensize, prevlen = 0;
    size_t offset;
@@ -609,19 +766,24 @@ static unsigned char *__ziplistInsert(unsigned char *zl, unsigned char *p, unsig
        /* 'encoding' is set to the appropriate integer encoding */
        reqlen = zipIntSize(encoding);
    } else {
-        /* 'encoding' is untouched, however zipEncodeLength will use the
+        /* 'encoding' is untouched, however zipStoreEntryEncoding will use the
         * string length to figure out how to encode it. */
        reqlen = slen;
    }
    /* We need space for both the length of the previous entry and
     * the length of the payload. */
-    reqlen += zipPrevEncodeLength(NULL,prevlen);
-    reqlen += zipEncodeLength(NULL,encoding,slen);
+    reqlen += zipStorePrevEntryLength(NULL,prevlen);
+    reqlen += zipStoreEntryEncoding(NULL,encoding,slen);

    /* When the insert position is not equal to the tail, we need to
     * make sure that the next entry can hold this entry's length in
     * its prevlen field. */
+    int forcelarge = 0;
    nextdiff = (p[0] != ZIP_END) ? zipPrevLenByteDiff(p,reqlen) : 0;
+    if (nextdiff == -4 && reqlen < 4) {
+        nextdiff = 0;
+        forcelarge = 1;
+    }

    /* Store offset because a realloc may change the address of zl. */
    offset = p-zl;
@@ -634,7 +796,10 @@ static unsigned char *__ziplistInsert(unsigned char *zl, unsigned char *p, unsig
        memmove(p+reqlen,p-nextdiff,curlen-offset-1+nextdiff);

        /* Encode this entry's raw length in the next entry. */
-        zipPrevEncodeLength(p+reqlen,reqlen);
+        if (forcelarge)
+            zipStorePrevEntryLengthLarge(p+reqlen,reqlen);
+        else
+            zipStorePrevEntryLength(p+reqlen,reqlen);

        /* Update offset for tail */
        ZIPLIST_TAIL_OFFSET(zl) =
@@ -662,8 +827,8 @@ static unsigned char *__ziplistInsert(unsigned char *zl, unsigned char *p, unsig
    }

    /* Write the entry */
-    p += zipPrevEncodeLength(p,prevlen);
-    p += zipEncodeLength(p,encoding,slen);
+    p += zipStorePrevEntryLength(p,prevlen);
+    p += zipStoreEntryEncoding(p,encoding,slen);
    if (ZIP_IS_STR(encoding)) {
        memcpy(p,s,slen);
    } else {
@@ -1029,7 +1194,7 @@ void ziplistRepr(unsigned char *zl) {

    printf(
        "{total bytes %d} "
-        "{length %u}\n"
+        "{num entries %u}\n"
        "{tail offset %u}\n",
        intrev32ifbe(ZIPLIST_BYTES(zl)),
        intrev16ifbe(ZIPLIST_LENGTH(zl)),
@@ -1038,16 +1203,15 @@ void ziplistRepr(unsigned char *zl) {
    while(*p != ZIP_END) {
        zipEntry(p, &entry);
        printf(
-            "{"
-                "addr 0x%08lx, "
-                "index %2d, "
-                "offset %5ld, "
-                "rl: %5u, "
-                "hs %2u, "
-                "pl: %5u, "
-                "pls: %2u, "
-                "payload %5u"
-            "} ",
+            "{\n"
+                "\taddr 0x%08lx,\n"
+                "\tindex %2d,\n"
+                "\toffset %5ld,\n"
+                "\thdr+entry len: %5u,\n"
+                "\thdr len%2u,\n"
+                "\tprevrawlen: %5u,\n"
+                "\tprevrawlensize: %2u,\n"
+                "\tpayload %5u\n",
            (long unsigned)p,
            index,
            (unsigned long) (p-zl),
@@ -1056,8 +1220,14 @@ void ziplistRepr(unsigned char *zl) {
            entry.prevrawlen,
            entry.prevrawlensize,
            entry.len);
+        printf("\tbytes: ");
+        for (unsigned int i = 0; i < entry.headersize+entry.len; i++) {
+            printf("%02x|",p[i]);
+        }
+        printf("\n");
        p += entry.headersize;
        if (ZIP_IS_STR(entry.encoding)) {
+            printf("\t[str]");
            if (entry.len > 40) {
                if (fwrite(p,40,1,stdout) == 0) perror("fwrite");
                printf("...");
@@ -1066,9 +1236,9 @@ void ziplistRepr(unsigned char *zl) {
                    fwrite(p,entry.len,1,stdout) == 0) perror("fwrite");
            }
        } else {
-            printf("%lld", (long long) zipLoadInteger(p,entry.encoding));
+            printf("\t[int]%lld", (long long) zipLoadInteger(p,entry.encoding));
        }
-        printf("\n");
+        printf("\n}\n");
        p += entry.len;
        index++;
    }
@@ -48,6 +48,7 @@ unsigned int ziplistCompare(unsigned char *p, unsigned char *s, unsigned int sle
 unsigned char *ziplistFind(unsigned char *p, unsigned char *vstr, unsigned int vlen, unsigned int skip);
 unsigned int ziplistLen(unsigned char *zl);
 size_t ziplistBlobLen(unsigned char *zl);
+void ziplistRepr(unsigned char *zl);

 #ifdef REDIS_TEST
 int ziplistTest(int argc, char *argv[]);
@@ -66,6 +66,8 @@ void zlibc_free(void *ptr) {
 #define calloc(count,size) je_calloc(count,size)
 #define realloc(ptr,size) je_realloc(ptr,size)
 #define free(ptr) je_free(ptr)
+#define mallocx(size,flags) je_mallocx(size,flags)
+#define dallocx(ptr,flags) je_dallocx(ptr,flags)
 #endif

 #define update_zmalloc_stat_alloc(__n) do { \
@@ -115,6 +117,24 @@ void *zmalloc(size_t size) {
 #endif
 }

+/* Allocation and free functions that bypass the thread cache
+ * and go straight to the allocator arena bins.
+ * Currently implemented only for jemalloc. Used for online defragmentation. */
+#ifdef HAVE_DEFRAG
+void *zmalloc_no_tcache(size_t size) {
+    void *ptr = mallocx(size+PREFIX_SIZE, MALLOCX_TCACHE_NONE);
+    if (!ptr) zmalloc_oom_handler(size);
+    update_zmalloc_stat_alloc(zmalloc_size(ptr));
+    return ptr;
+}
+
+void zfree_no_tcache(void *ptr) {
+    if (ptr == NULL) return;
+    update_zmalloc_stat_free(zmalloc_size(ptr));
+    dallocx(ptr, MALLOCX_TCACHE_NONE);
+}
+#endif
+
 void *zcalloc(size_t size) {
    void *ptr = calloc(1, size+PREFIX_SIZE);

@@ -398,8 +418,9 @@ size_t zmalloc_get_memory_size(void) {
    if (sysctl(mib, 2, &size, &len, NULL, 0) == 0)
        return (size_t)size;
    return 0L;          /* Failed? */
-#endif /* sysctl and sysconf variants */
-
+#else
+    return 0L;          /* Unknown method to get the data. */
+#endif
 #else
    return 0L;          /* Unknown OS. */
 #endif
@@ -65,6 +65,13 @@
 #define ZMALLOC_LIB "libc"
 #endif

+/* We can enable the Redis defrag capabilities only if we are using Jemalloc
+ * and the version used is our special version modified for Redis having
+ * the ability to return per-allocation fragmentation hints. */
+#if defined(USE_JEMALLOC) && defined(JEMALLOC_FRAG_HINT)
+#define HAVE_DEFRAG
+#endif
+
 void *zmalloc(size_t size);
 void *zcalloc(size_t size);
 void *zrealloc(void *ptr, size_t size);
@@ -80,6 +87,11 @@ size_t zmalloc_get_smap_bytes_by_field(char *field, long pid);
 size_t zmalloc_get_memory_size(void);
 void zlibc_free(void *ptr);

+#ifdef HAVE_DEFRAG
+void zfree_no_tcache(void *ptr);
+void *zmalloc_no_tcache(size_t size);
+#endif
+
 #ifndef HAVE_MALLOC_SIZE
 size_t zmalloc_size(void *ptr);
 #endif
@@ -1,5 +1,6 @@
 # Redis configuration for testing.

+always-show-logo yes
 notify-keyspace-events KEA
 daemonize no
 pidfile /var/run/redis.pid
@@ -88,7 +88,7 @@ tags {"aof"} {
            set pattern "*Bad file format reading the append only file*"
            set retry 10
            while {$retry} {
-                set result [exec tail -n1 < [dict get $srv stdout]]
+                set result [exec tail -1 < [dict get $srv stdout]]
                if {[string match $pattern $result]} {
                    break
                }
@@ -113,7 +113,7 @@ tags {"aof"} {
            set pattern "*Unexpected end of file reading the append only file*"
            set retry 10
            while {$retry} {
-                set result [exec tail -n1 < [dict get $srv stdout]]
+                set result [exec tail -1 < [dict get $srv stdout]]
                if {[string match $pattern $result]} {
                    break
                }
@@ -137,7 +137,7 @@ tags {"aof"} {
            set pattern "*Unexpected end of file reading the append only file*"
            set retry 10
            while {$retry} {
-                set result [exec tail -n1 < [dict get $srv stdout]]
+                set result [exec tail -1 < [dict get $srv stdout]]
                if {[string match $pattern $result]} {
                    break
                }
@@ -66,7 +66,7 @@ if {!$isroot} {
        test {Server should not start if RDB file can't be open} {
            wait_for_condition 50 100 {
                [string match {*Fatal error loading*} \
-                    [exec tail -n1 < [dict get $srv stdout]]]
+                    [exec tail -1 < [dict get $srv stdout]]]
            } else {
                fail "Server started even if RDB was unreadable!"
            }
@@ -90,7 +90,7 @@ start_server_and_kill_it [list "dir" $server_path] {
    test {Server should not start if RDB is corrupted} {
        wait_for_condition 50 100 {
            [string match {*CRC error*} \
-                [exec tail -n10 < [dict get $srv stdout]]]
+                [exec tail -10 < [dict get $srv stdout]]]
        } else {
            fail "Server started even if RDB was corrupted!"
        }
@@ -30,6 +30,18 @@ start_server {tags {"repl"}} {
            }
            assert_equal [r debug digest] [r -1 debug digest]
        }
+
+        test {Slave is able to evict keys created in writable slaves} {
+            r -1 select 5
+            assert {[r -1 dbsize] == 0}
+            r -1 config set slave-read-only no
+            r -1 set key1 1 ex 5
+            r -1 set key2 2 ex 5
+            r -1 set key3 3 ex 5
+            assert {[r -1 dbsize] == 3}
+            after 6000
+            r -1 dbsize
+        } {0}
    }
 }

@@ -47,7 +47,7 @@ proc test_psync {descr duration backlog_size backlog_ttl delay cond diskless rec

            # Check that the background clients are actually writing.
            test {Detect write load to master} {
-                wait_for_condition 50 100 {
+                wait_for_condition 50 1000 {
                    [$master dbsize] > 100
                } else {
                    fail "Can't detect write load from background clients."
@@ -110,7 +110,7 @@ foreach diskless {no yes} {
    test_psync {no reconnection, just sync} 6 1000000 3600 0 {
    } $diskless 0

-    test_psync {ok psync} 6 1000000 3600 0 {
+    test_psync {ok psync} 6 100000000 3600 0 {
        assert {[s -1 sync_partial_ok] > 0}
    } $diskless 1

@@ -278,7 +278,7 @@ proc start_server {options {code undefined}} {

        while 1 {
            # check that the server actually started and is ready for connections
-            if {[exec grep "ready to accept" | wc -l < $stdout] > 0} {
+            if {[exec grep -i "Ready to accept" | wc -l < $stdout] > 0} {
                break
            }
            after 10
@@ -73,7 +73,7 @@ start_server {tags {"aofrw"}} {
        r config set appendonly no
        r exec
        wait_for_condition 50 100 {
-            [string match {*Killing*AOF*child*} [exec tail -n5 < [srv 0 stdout]]]
+            [string match {*Killing*AOF*child*} [exec tail -5 < [srv 0 stdout]]]
        } else {
            fail "Can't find 'Killing AOF child' into recent logs"
        }
@@ -221,18 +221,26 @@ start_server {tags {"geo"}} {
    }

    test {GEOADD + GEORANGE randomized test} {
-        set attempt 20
+        set attempt 30
        while {[incr attempt -1]} {
            set rv [lindex $regression_vectors $rv_idx]
            incr rv_idx

            unset -nocomplain debuginfo
-            set srand_seed [randomInt 1000000]
+            set srand_seed [clock milliseconds]
            if {$rv ne {}} {set srand_seed [lindex $rv 0]}
            lappend debuginfo "srand_seed is $srand_seed"
            expr {srand($srand_seed)} ; # If you need a reproducible run
            r del mypoints
-            set radius_km [expr {[randomInt 200]+10}]
+
+            if {[randomInt 10] == 0} {
+                # From time to time use very big radiuses
+                set radius_km [expr {[randomInt 50000]+10}]
+            } else {
+                # Normally use a few - ~200km radiuses to stress
+                # test the code the most in edge cases.
+                set radius_km [expr {[randomInt 200]+10}]
+            }
            if {$rv ne {}} {set radius_km [lindex $rv 1]}
            set radius_m [expr {$radius_km*1000}]
            geo_random_point search_lon search_lat
@@ -246,10 +254,11 @@ start_server {tags {"geo"}} {
            for {set j 0} {$j < 20000} {incr j} {
                geo_random_point lon lat
                lappend argv $lon $lat "place:$j"
-                if {[geo_distance $lon $lat $search_lon $search_lat] < $radius_m} {
+                set distance [geo_distance $lon $lat $search_lon $search_lat]
+                if {$distance < $radius_m} {
                    lappend tcl_result "place:$j"
-                    lappend debuginfo "place:$j $lon $lat [expr {[geo_distance $lon $lat $search_lon $search_lat]/1000}] km"
                }
+                lappend debuginfo "place:$j $lon $lat [expr {$distance/1000}] km"
            }
            r geoadd mypoints {*}$argv
            set res [lsort [r georadius mypoints $search_lon $search_lat $radius_km km]]
@@ -24,7 +24,7 @@ start_server {tags {"maxmemory"}} {
    }

    foreach policy {
-        allkeys-random allkeys-lru volatile-lru volatile-random volatile-ttl
+        allkeys-random allkeys-lru allkeys-lfu volatile-lru volatile-lfu volatile-random volatile-ttl
    } {
        test "maxmemory - is the memory limit honoured? (policy $policy)" {
            # make sure to start with a blank instance
@@ -98,7 +98,7 @@ start_server {tags {"maxmemory"}} {
    }

    foreach policy {
-        volatile-lru volatile-random volatile-ttl
+        volatile-lru volatile-lfu volatile-random volatile-ttl
    } {
        test "maxmemory - policy $policy should only remove volatile keys." {
            # make sure to start with a blank instance
@@ -35,3 +35,49 @@ start_server {tags {"memefficiency"}} {
        }
    }
 }
+
+start_server {tags {"defrag"}} {
+    if {[string match {*jemalloc*} [s mem_allocator]]} {
+        test "Active defrag" {
+            r config set activedefrag no
+            r config set active-defrag-threshold-lower 5
+            r config set active-defrag-ignore-bytes 2mb
+            r config set maxmemory 100mb
+            r config set maxmemory-policy allkeys-lru
+            r debug populate 700000 asdf 150
+            r debug populate 170000 asdf 300
+            set frag [s mem_fragmentation_ratio]
+            assert {$frag >= 1.7}
+            r config set activedefrag yes
+            after 1500 ;# active defrag tests the status once a second.
+            set hits [s active_defrag_hits]
+
+            # wait for the active defrag to stop working
+            set tries 0
+            while { True } {
+                incr tries
+                after 500
+                set prev_hits $hits
+                set hits [s active_defrag_hits]
+                if {$hits == $prev_hits} {
+                    break
+                }
+                assert {$tries < 100}
+            }
+
+            # TODO: we need to expose more accurate fragmentation info
+            # i.e. the allocator used and active pages
+            # instead we currently look at RSS so we need to ask for purge
+            r memory purge
+
+            # Test the the fragmentation is lower and that the defragger
+            # stopped working
+            set frag [s mem_fragmentation_ratio]
+            assert {$frag < 1.55}
+            set misses [s active_defrag_misses]
+            after 500
+            set misses2 [s active_defrag_misses]
+            assert {$misses2 == $misses}
+        }
+    }
+}
@@ -525,7 +525,7 @@ start_server {tags {"hash"}} {
    # 1.23 cannot be represented correctly with 64 bit doubles, so we skip
    # the test, since we are only testing pretty printing here and is not
    # a bug if the program outputs things like 1.299999...
-    if {!$::valgrind || ![string match *x86_64* [exec uname -a]]} {
+    if {!$::valgrind && [string match *x86_64* [exec uname -a]]} {
        test {Test HINCRBYFLOAT for correct float representation (issue #2846)} {
            r del myhash
            assert {[r hincrbyfloat myhash float 1.23] eq {1.23}}
@@ -86,6 +86,12 @@ then
    exit 0
 fi

+if [ "$1" == "clean-logs" ]
+then
+    rm -rf *.log
+    exit 0
+fi
+
 echo "Usage: $0 [start|create|stop|watch|tail|clean]"
 echo "start       -- Launch Redis Cluster instances."
 echo "create      -- Create a cluster using redis-trib create."
@@ -93,3 +99,4 @@ echo "stop        -- Stop Redis Cluster instances."
 echo "watch       -- Show CLUSTER NODES output (first 30 lines) of first node."
 echo "tail <id>   -- Run tail -f of instance at base port + ID."
 echo "clean       -- Remove all instances data, logs, configs."
+echo "clean-logs  -- Remove just instances logs."
@@ -30,7 +30,7 @@ def run_experiment(r,seed,max,step)
            elements << ele
            i += 1
        }
-        r.pfadd('hll',*elements)
+        r.pfadd('hll',elements)
        approx = r.pfcount('hll')
        err = approx-i
        rel_err = 100.to_f*err/i
--- a/Show More
+++ b/Show More